bug#45054: 27.1; Can get point into the middle of a run of characters with a replacing display spec

bug#45054: 27.1; Can get point into the middle of a run of characters with a replacing display spec

[Zack Weinberg]

I'm implementing a minor mode that "folds" certain strings, causing them
to be displayed as the actual single characters that they are notation
for.  For purposes of this bug report, I will use the running example of
displaying C-style backslash escapes for Unicode characters as their
actual characters, e.g. \u00B6 is displayed as ¶ when the mode is
active.

The mode does this by adding a font-lock keywords rule to the buffer,
that applies display properties that make each string look like it's a
single character.  The documentation says that Emacs will not let point
get into the middle of a run of text that's hidden this way, and that's
mostly true, but I found a way to do it.

Starting from ‘emacs -Q’:

In the *scratch* buffer, evaluate the following minor mode definition;
this is a cut-down version of the real mode, that only replaces \u00B6.

(define-minor-mode fold-demo-mode
  "Folding bug demo"
  :init-value nil
  :lighter " FD"
  :keymap nil
  (make-variable-buffer-local 'font-lock-extra-managed-props)
  (cl-flet ((refontify ()
             (save-excursion
               (goto-char (point-min))
               (while (re-search-forward "\\\\u00[bB]6" nil t)
                 (save-excursion
                   (font-lock-fontify-region
                    (match-beginning 0) (match-end 0)))))))
    (let ((case-fold-search nil)
          (demo-fl-rule '(("\\\\u00[bB]6" 0
                           '(face font-lock-string-face
                                  display "¶")))))
      (if fold-demo-mode
          (progn
            (push 'display font-lock-extra-managed-props)
            (font-lock-add-keywords nil demo-fl-rule)
            (refontify))
        (font-lock-remove-keywords nil demo-fl-rule)
        (refontify)
        (setq font-lock-extra-managed-props
              (cl-remove 'display font-lock-extra-managed-props
                         :count 1))))))

Then, in any convenient buffer, on an initially blank line, type the
following.  I’ve written the recipe out as it might appear in a
keyboard-macro editing buffer, but instead of command names, the
annotations show what you see at each stage.  The *visible cursor* is
indicated with a |; all other characters are real.

\                    ;; \|
u00B6                ;; \u00B6|
C-a                  ;; |\u00B6
M-x fold-demo-mode   ;; |¶
C-d                  ;; |u00B6
\                    ;; ¶|
x                    ;; \x|u00B6

The bug happens when you type \ for the second time.  The font-lock rule
matches again, “\u00B6” is converted to “¶”, and the *visible cursor* is
displaced to after the “¶”, but *point* is still where it was, between
the invisible \ and u characters.  Any self-inserting character, like
the ‘x’ shown in the recipe, will be inserted at the actual location of
point, instead of the position of the visible cursor.  In the demo, this
causes the font-lock rule to stop matching again.

Any cursor-motion command while Emacs is in this state, will put point
back in sync with the visible cursor; this can lead to odd but harmless
phenomena, for instance a single C-f doesn’t appear to do anything,
instead of moving the cursor to the next line like it normally would.
(Because point moves within the hidden run of characters, and then gets
displaced to the end of the run, where the visible cursor already is.)

Questions:

1. Is this indeed a bug in Emacs?
2. Assuming it is, can you suggest a workaround?  I’m hoping for a
   viable solution within my code that’s compatible at least as far back
   as Emacs 24.

3. I think it would be better UI if the C-d in the recipe deleted the
   entire run of text that’s getting hidden (more generally, deletion
   commands should behave as if the ¶ is really what’s in the buffer).
   I can’t figure out how to implement that, can you suggest anything?

4. Is there a better way to do this sort of visual replacement of runs
   of text?  This is the first time I’ve done any serious elisp
   programming, I might have missed something.

Thanks for your attention,
zw

In GNU Emacs 27.1 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.23,
cairo version 1.16.0) of 2020-11-07, modified by Debian built on
x86-ubc-01 Windowing system distributor 'The X.Org Foundation', version
11.0.12009000 System Description: Debian GNU/Linux bullseye/sid

Configured using:
 'configure --build x86_64-linux-gnu --prefix=/usr
 --sharedstatedir=/var/lib --libexecdir=/usr/lib
 --localstatedir=/var/lib --infodir=/usr/share/info
 --mandir=/usr/share/man --enable-libsystemd --with-pop=yes
 --enable-locallisppath=/etc/emacs:/usr/local/share/emacs/27.1/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/27.1/site-lisp:/usr/share/emacs/site-lisp
 --with-sound=alsa --without-gconf --with-mailutils --build
 x86_64-linux-gnu --prefix=/usr --sharedstatedir=/var/lib
 --libexecdir=/usr/lib --localstatedir=/var/lib
 --infodir=/usr/share/info --mandir=/usr/share/man --enable-libsystemd
 --with-pop=yes
 --enable-locallisppath=/etc/emacs:/usr/local/share/emacs/27.1/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/27.1/site-lisp:/usr/share/emacs/site-lisp
 --with-sound=alsa --without-gconf --with-mailutils --with-cairo
 --with-x=yes --with-x-toolkit=gtk3 --with-toolkit-scroll-bars
 'CFLAGS=-g -O2
 -fdebug-prefix-map=/build/emacs-6jKC2B/emacs-27.1+1=.
-fstack-protector-strong -Wformat -Werror=format-security -Wall'
'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2' LDFLAGS=-Wl,-z,relro'

Configured features:
XPM JPEG TIFF GIF PNG RSVG CAIRO SOUND GPM DBUS GSETTINGS GLIB NOTIFY
INOTIFY ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE HARFBUZZ M17N_FLT LIBOTF
ZLIB TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM MODULES THREADS LIBSYSTEMD
JSON PDUMPER LCMS2 GMP

Important settings:
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Features:
(shadow sort mail-extr emacsbug message rmc puny rfc822 mml mml-sec epa
derived epg epg-config gnus-util rmail rmail-loaddefs
text-property-search mm-decode mm-bodies mm-encode mail-parse rfc2231
mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums
mm-util mail-prsvr mail-utils help-mode server visual-regexp-steroids
visual-regexp tramp tramp-loaddefs trampver tramp-integration files-x
tramp-compat parse-time iso8601 time-date ls-lisp dired dired-loaddefs
dbus xml solarized-light-theme solarized color dash typo paren grep
cus-start cus-load erlang-start ess-site ess-toolbar ess-mouse mouseme
ess-swv ess-noweb ess-noweb-font-lock-mode ess-jags-d ess-bugs-l
essd-els ess-xls-d ess-vst-d ess-stata-mode ess-stata-lang cc-vars
cc-defs make-regexp ess-sp6w-d ess-sp5-d ess-sp4-d ess-sas-d ess-sas-l
ess-sas-a ess-s4-d ess-s3-d ess-omg-d ess-omg-l ess-arc-d ess-lsp-l
ess-sp6-d ess-dde ess-sp3-d ess-julia julia-mode cl ess-r-mode
ess-r-flymake flymake-proc flymake warnings thingatpt ess-r-xref xref
project ess-trns ess-r-package shell pcomplete ess-r-syntax
ess-r-completion ess-roxy ess-rd essddr noutline outline easy-mmode
hideshow ess-s-lang ess-help info ess-mode ess ess-noweb-mode ess-inf
ess-tracebug advice format-spec ess-generics compile ess-utils ido
ess-custom executable comint ansi-color ring go-mode-autoloads package
easymenu browse-url url-handlers url-parse auth-source cl-seq eieio
eieio-core cl-macs eieio-loaddefs password-cache json subr-x map
url-vars seq byte-opt gv bytecomp byte-compile cconv cl-loaddefs cl-lib
tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode elisp-mode
lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow isearch
timer select scroll-bar mouse jit-lock font-lock syntax facemenu
font-core term/tty-colors frame minibuffer cl-generic cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote threads dbusbind
inotify lcms2 dynamic-setting system-font-setting font-render-setting
cairo move-toolbar gtk x-toolkit x multi-tty make-network-process emacs)

bug#45054: 27.1; Can get point into the middle of a run of characters with a replacing display spec

[Zack Weinberg]

I forgot to mention that nothing changes if I also have font-lock
apply the 'cursor-intangible property to the hidden run of text.

bug#45054: 27.1; Can get point into the middle of a run of characters with a replacing display spec

[Eli Zaretskii]

> From: Zack Weinberg <zackw@panix.com>
> Date: Sat, 5 Dec 2020 10:54:57 -0500
> 
> \                    ;; \|
> u00B6                ;; \u00B6|
> C-a                  ;; |\u00B6
> M-x fold-demo-mode   ;; |¶
> C-d                  ;; |u00B6
> \                    ;; ¶|
> x                    ;; \x|u00B6
> 
> The bug happens when you type \ for the second time.  The font-lock rule
> matches again, “\u00B6” is converted to “¶”, and the *visible cursor* is
> displaced to after the “¶”, but *point* is still where it was, between
> the invisible \ and u characters.  Any self-inserting character, like
> the ‘x’ shown in the recipe, will be inserted at the actual location of
> point, instead of the position of the visible cursor.  In the demo, this
> causes the font-lock rule to stop matching again.
> 
> Any cursor-motion command while Emacs is in this state, will put point
> back in sync with the visible cursor; this can lead to odd but harmless
> phenomena, for instance a single C-f doesn’t appear to do anything,
> instead of moving the cursor to the next line like it normally would.
> (Because point moves within the hidden run of characters, and then gets
> displaced to the end of the run, where the visible cursor already is.)
> 
> Questions:
> 
> 1. Is this indeed a bug in Emacs?

Probably a bug in adjust_point_for_property.  Try your recipe after
setting global-disable-point-adjustment non-nil.

> 2. Assuming it is, can you suggest a workaround?  I’m hoping for a
>    viable solution within my code that’s compatible at least as far back
>    as Emacs 24.

A workaround in Lisp?  Try making the affected text, or some of it,
invisible, i.e. give it the 'invisible' text property as well.  (I
didn't try to test this solution, so I cannot promise it will work.)

> 3. I think it would be better UI if the C-d in the recipe deleted the
>    entire run of text that’s getting hidden (more generally, deletion
>    commands should behave as if the ¶ is really what’s in the buffer).
>    I can’t figure out how to implement that, can you suggest anything?

Why can't you delete every character around point that has this
special property?

> 4. Is there a better way to do this sort of visual replacement of runs
>    of text?  This is the first time I’ve done any serious elisp
>    programming, I might have missed something.

Sounds the right way to me (modulo making the text invisible).

bug#45054: 27.1; Can get point into the middle of a run of characters with a replacing display spec

[Zack Weinberg]

On Sat, Dec 5, 2020 at 11:18 AM Eli Zaretskii <eliz@gnu.org> wrote:
> > 1. Is this indeed a bug in Emacs?
>
> Probably a bug in adjust_point_for_property.  Try your recipe after
> setting global-disable-point-adjustment non-nil.

No change, but set-variable doesn't recognize that variable (so I used
setq instead)  Is this variable possibly only available in development
trunk and/or in an Emacs compiled with internal debugging enabled?
I'm using 27.1 as packaged by Debian, right now.

> > 2. Assuming it is, can you suggest a workaround?  I’m hoping for a
> >    viable solution within my code that’s compatible at least as far back
> >    as Emacs 24.
>
> A workaround in Lisp?  Try making the affected text, or some of it,
> invisible, i.e. give it the 'invisible' text property as well.  (I
> didn't try to test this solution, so I cannot promise it will work.)

That makes the situation worse: with the mode enabled, I can separate
point from the visible cursor just by attempting to move the visible
cursor over the displayed ¶.  Also, if the ¶ is at the beginning of a
line and point is at the end of that line, C-a moves only to after the
(invisible) 6.  And redisplay gets screwed up if I delete one of the
invisible characters.

> > 3. I think it would be better UI if the C-d in the recipe deleted the
> >    entire run of text that’s getting hidden (more generally, deletion
> >    commands should behave as if the ¶ is really what’s in the buffer).
> >    I can’t figure out how to implement that, can you suggest anything?
>
> Why can't you delete every character around point that has this
> special property?

I don't know how to do that.  My best guess is this would be done in a
change hook, but I'm not sure of that, and I don't understand how to
write change hooks, either.  Is there a good example or guide-level
documentation you could point me at?

zw

bug#45054: 27.1; Can get point into the middle of a run of characters with a replacing display spec

[Eli Zaretskii]

> From: Zack Weinberg <zackw@panix.com>
> Date: Sat, 5 Dec 2020 15:00:30 -0500
> Cc: 45054@debbugs.gnu.org
> 
> On Sat, Dec 5, 2020 at 11:18 AM Eli Zaretskii <eliz@gnu.org> wrote:
> > > 1. Is this indeed a bug in Emacs?
> >
> > Probably a bug in adjust_point_for_property.  Try your recipe after
> > setting global-disable-point-adjustment non-nil.
> 
> No change, but set-variable doesn't recognize that variable (so I used
> setq instead)  Is this variable possibly only available in development
> trunk and/or in an Emacs compiled with internal debugging enabled?

No, it just isn't a user option, just a variable.

> > A workaround in Lisp?  Try making the affected text, or some of it,
> > invisible, i.e. give it the 'invisible' text property as well.  (I
> > didn't try to test this solution, so I cannot promise it will work.)
> 
> That makes the situation worse: with the mode enabled, I can separate
> point from the visible cursor just by attempting to move the visible
> cursor over the displayed ¶.

Then I guess your method of putting the display property really freaks
out the mechanism of adjusting point to move it out of this and other
similar properties.

> > > 3. I think it would be better UI if the C-d in the recipe deleted the
> > >    entire run of text that’s getting hidden (more generally, deletion
> > >    commands should behave as if the ¶ is really what’s in the buffer).
> > >    I can’t figure out how to implement that, can you suggest anything?
> >
> > Why can't you delete every character around point that has this
> > special property?
> 
> I don't know how to do that.

What part is missing?  Implement a command that uses
next-property-change and previous-property-change to find where the
property starts and ends, then delete all those characters.  Am I
missing something?

bug#45054: 27.1; Can get point into the middle of a run of characters with a replacing display spec

[Eli Zaretskii]

> Date: Sat, 05 Dec 2020 18:18:15 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 45054@debbugs.gnu.org
> 
> > 4. Is there a better way to do this sort of visual replacement of runs
> >    of text?  This is the first time I’ve done any serious elisp
> >    programming, I might have missed something.
> 
> Sounds the right way to me (modulo making the text invisible).

As an alternative, you could try the technique used by
prettify-symbols-mode.

bug#45054: 27.1; Can get point into the middle of a run of characters with a replacing display spec

[Zack Weinberg]

On Sat, Dec 5, 2020 at 4:54 PM Zack Weinberg <zackw@panix.com> wrote:
> As a matter of fact, for the larger goal, I think I can just ... use
> prettify-symbols-mode. :)  Thanks for the pointer, I missed that that
> had been added.

prettify-symbols-mode exhibits the same bug!

Simpler reproduction recipe:

type "ambda" in a lisp-interaction-mode buffer
place point immediately before the first "a"
M-x prettify-symbols-mode
type "l"; the word "lambda" will vanish and a λ will appear in its
place; the visible cursor will be just after the λ
type "x"; the λ will disappear and "lxambda" will appear, the visible
cursor will be just after the "x".

This might mean that the bug also affects writing ordinary text in
scripts where composite.el is necessary for correct appearance; I'm
not familiar enough with any of those to say for sure.

zw

bug#45054: 27.1; Can get point into the middle of a run of characters with a replacing display spec

[Eli Zaretskii]

> From: Zack Weinberg <zackw@panix.com>
> Date: Sat, 5 Dec 2020 17:12:35 -0500
> 
> type "ambda" in a lisp-interaction-mode buffer
> place point immediately before the first "a"
> M-x prettify-symbols-mode
> type "l"; the word "lambda" will vanish and a λ will appear in its
> place; the visible cursor will be just after the λ
> type "x"; the λ will disappear and "lxambda" will appear, the visible
> cursor will be just after the "x".
> 
> This might mean that the bug also affects writing ordinary text in
> scripts where composite.el is necessary for correct appearance; I'm
> not familiar enough with any of those to say for sure.

The adjustment of point to move it outside of the "forbidden" region
is triggered by point movements, while your examples don't move point.

bug#45054: 27.1; Can get point into the middle of a run of characters with a replacing display spec

[Zack Weinberg]

Are you saying that this is not a bug?

bug#45054: 27.1; Can get point into the middle of a run of characters with a replacing display spec

[Eli Zaretskii]

> From: Zack Weinberg <zackw@panix.com>
> Date: Wed, 9 Dec 2020 13:56:28 -0500
> Cc: 45054@debbugs.gnu.org
> 
> Are you saying that this is not a bug?

Did I say that?

bug#45054: 27.1; Can get point into the middle of a run of characters with a replacing display spec

[Zack Weinberg]

On Wed, Dec 9, 2020 at 2:14 PM Eli Zaretskii <eliz@gnu.org> wrote:
> > From: Zack Weinberg <zackw@panix.com>
> > Date: Wed, 9 Dec 2020 13:56:28 -0500
> > Cc: 45054@debbugs.gnu.org
> >
> > Are you saying that this is not a bug?
>
> Did I say that?

I don't know, that's why I ask. You said that "The adjustment of point
to move it outside of the "forbidden" region is triggered by point
movements, while your examples don't move point" and that could be
read as "this is working as intended".

zw

bug#45054: 27.1; Can get point into the middle of a run of characters with a replacing display spec

[Eli Zaretskii]

> From: Zack Weinberg <zackw@panix.com>
> Date: Wed, 9 Dec 2020 14:20:15 -0500
> Cc: 45054@debbugs.gnu.org
> 
> On Wed, Dec 9, 2020 at 2:14 PM Eli Zaretskii <eliz@gnu.org> wrote:
> > > From: Zack Weinberg <zackw@panix.com>
> > > Date: Wed, 9 Dec 2020 13:56:28 -0500
> > > Cc: 45054@debbugs.gnu.org
> > >
> > > Are you saying that this is not a bug?
> >
> > Did I say that?
> 
> I don't know, that's why I ask. You said that "The adjustment of point
> to move it outside of the "forbidden" region is triggered by point
> movements, while your examples don't move point" and that could be
> read as "this is working as intended".

It could also be read as "that's why it doesn't work as intended".