bug#39390: 28.0.50; prettify-symbols-mode confuses display when start/end faces are different

bug#39390: 28.0.50; prettify-symbols-mode confuses display when start/end faces are different

[Raphael 'kena' Poss]

When I am using the following substitution with prettify-symbols-mode,
everything works fine and the word "err" is substituted with "⊙":

(add-hook 'go-mode-hook
 (lambda ()
   (push '("error" . ?⊙) prettify-symbols-alist)
   ;; (push '("err != nil" . "⊙?") prettify-symbols-alist)
 ))

However if I then uncomment the second substitution for "err != nil":
all hell breaks loose: moving the cursor up and down around a source
line containing this text will mess up the display of the lines that
follow in a way that is sometimes irrecoverable.

The display bug is exacerbated (and thus easier to recognize/reproduce)
when global-hl-line-mode is set.

I have traced this down to substitutions where the font-lock face at the
beginning and the end of the symbol composition is different:

- replacing "String" is OK, replacing ".String()" is not
- replacing "func" is OK, replacing "func(" is not
- replacing "Fatal" is OK, replacing "t.Fatal" is not

This probably needs to be fixed somehow, either by preventing the
problem or by documenting the pitfall.

I would like to know if a workaround is available?




In GNU Emacs 28.0.50 (build 1, amd64-portbld-freebsd13.0)
Repository revision: e31287e
Repository branch: master
System Description: 13.0-CURRENT

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.

Configured using:
 'configure --disable-build-details --localstatedir=/var --with-sound=no
 --with-x-toolkit=no --without-cairo --without-dbus --without-gconf
 --without-gif --without-gsettings --without-imagemagick --without-jpeg
 --without-lcms2 --without-libotf --without-m17n-flt --without-png
 --without-rsvg --without-tiff --without-toolkit-scroll-bars --without-x
 --without-xim --without-xpm --without-xwidgets --enable-acl
 --without-cairo --without-dbus --without-gconf --without-gif
 --with-gnutls --without-gsettings --without-harfbuzz --without-jpeg
 --with-json --with-file-notification=kqueue --without-lcms2
 --without-m17n-flt --without-imagemagick --with-mailutils
 --with-modules --without-libotf --without-png
 --without-toolkit-scroll-bars --without-rsvg --with-threads
 --without-tiff --without-xft --without-xim --with-xml2 --without-xpm
 --without-xwidgets --with-x-toolkit=no --prefix=/usr/local
 --mandir=/usr/local/man --disable-silent-rules
 --infodir=/usr/local/share/emacs/info/
 --build=amd64-portbld-freebsd13.0 'CFLAGS=-O2 -pipe
 -fstack-protector-strong -isystem /usr/local/include
 -fno-strict-aliasing ' 'CPPFLAGS=-isystem /usr/local/include' 'LDFLAGS=
 -fstack-protector-strong -L/usr/local/lib ''

Configured features:
NOTIFY KQUEUE ACL GNUTLS LIBXML2 ZLIB MODULES THREADS JSON PDUMPER GMP

Important settings:
  value of $LC_COLLATE: C
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Go

Minor modes in effect:
  lsp-ui-mode: t
  lsp-ui-doc-mode: t
  lsp-ui-sideline-mode: t
  global-pretty-mode: t
  global-hl-line-mode: t
  show-paren-mode: t
  electric-pair-mode: t
  go-guru-hl-identifier-mode: t
  lsp-managed-mode: t
  lsp-mode: t
  flymake-mode: t
  yas-global-mode: t
  yas-minor-mode: t
  company-mode: t
  helm-mode: t
  helm--remap-mouse-mode: t
  projectile-mode: t
  global-magit-file-mode: t
  magit-file-mode: t
  magit-auto-revert-mode: t
  auto-revert-mode: t
  global-git-commit-mode: t
  async-bytecomp-package-mode: t
  shell-dirtrack-mode: t
  global-flycheck-mode: t
  flycheck-mode: t
  global-whitespace-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  electric-indent-mode: t
  global-prettify-symbols-mode: t
  prettify-symbols-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  global-visual-line-mode: t
  visual-line-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug sendmail winner helm-command helm-elisp
helm-eval edebug backtrace helm-info mule-util lsp-ui lsp-ui-flycheck
lsp-ui-doc goto-addr lsp-ui-imenu lsp-ui-peek lsp-ui-sideline view
lsp-clients lsp-eslint lsp-verilog lsp-json lsp-csharp gnutls lsp-pwsh
lsp-terraform lsp-yaml lsp-vhdl lsp-haxe lsp-erlang lsp-fsharp
lsp-metals lsp-elm lsp-dart lsp-clojure lsp-go lsp-xml lsp-css
lsp-intelephense lsp-vetur lsp-html lsp-solargraph lsp-rust lsp-pyls lsp
company-oddmuse company-keywords company-etags company-gtags
company-dabbrev-code company-dabbrev company-files company-cmake
company-xcode company-clang company-semantic company-eclim
company-template company-bbdb company-capf company-anaconda
anaconda-mode pythonic python tramp-sh tramp tramp-loaddefs trampver
tramp-integration files-x tramp-compat parse-time iso8601 ls-lisp
term/tmux term/xterm xterm pretty-mode cl swiper ivy flx delsel colir
ivy-overlay ido hl-line paren elec-pair go-projectile vc-git go-rename
go-guru go-eldoc go-mode url url-proxy url-privacy url-expand
url-methods url-history url-cookie url-domsuf mailcap find-file ffap
etags fileloop generator company-lsp lsp-mode xref url-util tree-widget
wid-edit spinner network-stream nsm markdown-mode color noutline outline
lv inline ht f s ewoc em-glob esh-util dash-functional bindat
flymake-proc flymake mwheel warnings project yasnippet-snippets
yasnippet company pcase helm-mode helm-projectile helm-files helm-tags
helm-buffers helm-occur helm-grep helm-regexp helm-utils helm-locate
helm-help helm-types helm-config helm-easymenu helm helm-source
eieio-compat helm-multi-match helm-lib edmacro kmacro projectile grep
compile ibuf-ext ibuffer ibuffer-loaddefs thingatpt magit-submodule
magit-obsolete magit-blame magit-stash magit-reflog magit-bisect
magit-push magit-pull magit-fetch magit-clone magit-remote magit-commit
magit-sequence magit-notes magit-worktree magit-tag magit-merge
magit-branch magit-reset magit-files magit-refs magit-status magit
magit-repos magit-apply magit-wip magit-log which-func imenu magit-diff
smerge-mode diff diff-mode magit-core magit-autorevert autorevert
filenotify magit-margin magit-transient magit-process magit-mode
git-commit transient magit-git magit-section magit-utils crm log-edit
easy-mmode message rmc puny dired dired-loaddefs format-spec rfc822 mml
mml-sec epa derived epg epg-config gnus-util rmail rmail-loaddefs
text-property-search time-date mm-decode mm-bodies mm-encode mail-parse
rfc2231 rfc2047 rfc2045 mm-util ietf-drums mail-prsvr mailabbrev
mail-utils gmm-utils mailheader pcvs-util add-log with-editor cl-extra
async-bytecomp advice async shell pcomplete comint ring server flycheck
regexp-opt ansi-color find-func help-mode rx dash disp-table whitespace
info tool-bar package easymenu browse-url url-handlers url-parse
auth-source cl-seq eieio eieio-core cl-macs eieio-loaddefs
password-cache json subr-x map url-vars seq byte-opt gv bytecomp
byte-compile cconv cl-loaddefs cl-lib tooltip eldoc electric uniquify
ediff-hook vc-hooks lisp-float-type tabulated-list replace newcomment
text-mode elisp-mode lisp-mode prog-mode register page tab-bar menu-bar
rfn-eshadow isearch timer select mouse jit-lock font-lock syntax
facemenu font-core term/tty-colors frame minibuffer cl-generic cham
georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote threads kqueue
multi-tty make-network-process emacs)

Memory information:
((conses 16 382129 48543)
 (symbols 48 30567 1)
 (strings 32 115055 9348)
 (string-bytes 1 3699508)
 (vectors 16 54044)
 (vector-slots 8 769489 36090)
 (floats 8 286 445)
 (intervals 56 16026 8087)
 (buffers 1000 20))

-- 
Raphael 'kena' Poss

bug#39390: 28.0.50; prettify-symbols-mode confuses display when start/end faces are different

[Eli Zaretskii]

> From: Raphael 'kena' Poss <knz@thaumogen.net>
> Date: Sun, 2 Feb 2020 15:45:46 +0100
> 
> (add-hook 'go-mode-hook
>  (lambda ()
>    (push '("error" . ?⊙) prettify-symbols-alist)
>    ;; (push '("err != nil" . "⊙?") prettify-symbols-alist)
>  ))
> 
> However if I then uncomment the second substitution for "err != nil":
> all hell breaks loose: moving the cursor up and down around a source
> line containing this text will mess up the display of the lines that
> follow in a way that is sometimes irrecoverable.
> 
> The display bug is exacerbated (and thus easier to recognize/reproduce)
> when global-hl-line-mode is set.
> 
> I have traced this down to substitutions where the font-lock face at the
> beginning and the end of the symbol composition is different:
> 
> - replacing "String" is OK, replacing ".String()" is not
> - replacing "func" is OK, replacing "func(" is not
> - replacing "Fatal" is OK, replacing "t.Fatal" is not
> 
> This probably needs to be fixed somehow, either by preventing the
> problem or by documenting the pitfall.
> 
> I would like to know if a workaround is available?

You've bumped into a fundamental limitation of the feature, called
"character composition", that prettify-symbols-mode piggy-backs to do
its thing: Emacs can only compose characters that all have the same
face.  This restriction is imposed on a very low level of the display
code.

So I don't think there can be a workaround, except by "fixing" the
font-lock faces to use the same face on all the characters you want to
substitute.

bug#39390: 28.0.50; prettify-symbols-mode confuses display when start/end faces are different

[Raphael 'kena' Poss]

On 02-02-2020 18:18, Eli Zaretskii wrote:
> So I don't think there can be a workaround, except by "fixing" the
> font-lock faces to use the same face on all the characters you want to
> substitute.
> 

I don't mind this at all! Can you explain how to do this?

-- 
Raphael 'kena' Poss

bug#39390: 28.0.50; prettify-symbols-mode confuses display when start/end faces are different

[Eli Zaretskii]

> Cc: 39390@debbugs.gnu.org
> From: Raphael 'kena' Poss <knz@thaumogen.net>
> Date: Sun, 2 Feb 2020 18:56:35 +0100
> 
> On 02-02-2020 18:18, Eli Zaretskii wrote:
> > So I don't think there can be a workaround, except by "fixing" the
> > font-lock faces to use the same face on all the characters you want to
> > substitute.
> > 
> 
> I don't mind this at all! Can you explain how to do this?

I can only explain it in general terms, since I'm not familiar with
go-mode.

The idea is to modify the font-lock definitions of the mode such that
the characters you want to substitute all get the same face.  To see
what faces a character gets, go to that character and type

   M-x describe-text-properties RET

HTH

bug#39390: 28.0.50; prettify-symbols-mode confuses display when start/end faces are different

[Raphael 'kena' Poss]

Hi Eli,

thanks for your fast answer.

On 02-02-2020 19:03, Eli Zaretskii wrote:
> I can only explain it in general terms, since I'm not familiar with
> go-mode.

Look the issue is not particular to go-mode.

Also note that I am using prettify-symbols-mode which is a standard
function from prog-mode.el shipped in core Emacs. This is not an
extension and it is not my own custom code.

I am looking for advice to either:

1. enhance my *configuration* of prettify-symbols-alist, so as to avoid
the problem when using the standard code prog-mode.el

2. or, a diff for prog-mode.el to fix/workaround the problem in a way
independent of the particular lang major mode.

Your advice:

> The idea is to modify the font-lock definitions of the mode such that
> the characters you want to substitute all get the same face.

The substitution is performed by prog-mode.el
(prettify-symbols--compose-symbol) not by my own code. I would like to
know what I need to do to make that code use the same face for all of
the result of the substitution.

Preferably without modifying prog-mode.el (which I don't control), but
I'll take a diff against that if that's my only option.

I think the salient part from prog-mode.el is this:

        ;; That's a symbol alright, so add the composition.
        (with-silent-modifications
          (compose-region start end (cdr (assoc match alist)))
          (add-text-properties
           start end
           `(prettify-symbols-start ,start prettify-symbols-end ,end)))

from here:
https://git.savannah.gnu.org/cgit/emacs.git/tree/lisp/progmodes/prog-mode.el#n134

Is there a way to tweak either the configuration
(prettify-symbols-alist) or prog-mode.el to trick it into emitting the
same face for the substitution?

If that is not possible, is there a way to add an additional font-lock
keyword rule that applies _after_ the substitution performed by
prettify-symbols-mode and "clean up" the face mix-up?

(My opinion is that this bug exists in prog-mode.el and thus a
workaround/fix should be provided at that level, not in a per-mode or
per-user customization.)


-- 
Raphael 'kena' Poss

bug#39390: 28.0.50; prettify-symbols-mode confuses display when start/end faces are different

[Eli Zaretskii]

> Cc: 39390@debbugs.gnu.org
> From: Raphael 'kena' Poss <knz@thaumogen.net>
> Date: Sun, 2 Feb 2020 19:28:24 +0100
> 
> Is there a way to tweak either the configuration
> (prettify-symbols-alist) or prog-mode.el to trick it into emitting the
> same face for the substitution?

It sounds like there's a misunderstanding.  prettify-symbols-mode
doesn't emit any faces, it "composes" several characters into one or
more symbols.  The composition process needs that all of the
characters to be composed have the same face.  That face is put on the
characters by font-lock mode, not by prettify-symbols-mode.  That is
why I suggested to amend the font-lock definitions for the major
mode(s) you are using.  Without all the characters to be replaced
having the same font-lock face, you will not be able to "convince"
prettify-symbols-mode to replace them, because the underlying
mechanism that examines the characters to be composed stops when the
face property changes.

> (My opinion is that this bug exists in prog-mode.el and thus a
> workaround/fix should be provided at that level, not in a per-mode or
> per-user customization.)

It isn't a bug, it is a fundamental limitation of the method used by
prettify-symbols-mode to replace text with symbols.

bug#39390: 28.0.50; prettify-symbols-mode confuses display when start/end faces are different

[Raphael 'kena' Poss]

[-- Attachment #1: Type: text/plain, Size: 1170 bytes --]

On 02-02-2020 20:20, Eli Zaretskii wrote:
> The composition process needs that all of the
> characters to be composed have the same face.  That face is put on the
> characters by font-lock mode, not by prettify-symbols-mode. [...]
> Without all the characters to be replaced
> having the same font-lock face, you will not be able to "convince"
> prettify-symbols-mode to replace them

I think this wasn't clear in my original bug report: the replacement
*does* occur -- the display shows the result of the substitution upon
first display. (see first attached screenshot).

It even properly switches back and forth between the composition and
original form with unprettify-at-point. (see second attachment)

The problem is that the *redisplays* (e.g. as triggered by moving the
highlight around with hl-line-mode) mess up the lines around the
replacement. See third and fourth screenshots.

If you are telling me that (compose-region) should never have agreed to
operate on a range of symbols with different faces, then I have
counter-evidence that it does. If you are telling me that it shouldn't,
then maybe that's where the bug really lies.



-- 
Raphael 'kena' Poss

[-- Attachment #2: Screenshot_2020-02-02_20-58-58.png --]
[-- Type: image/png, Size: 21665 bytes --]

[-- Attachment #3: Screenshot_2020-02-02_20-59-18.png --]
[-- Type: image/png, Size: 31679 bytes --]

[-- Attachment #4: Screenshot_2020-02-02_20-59-39.png --]
[-- Type: image/png, Size: 29112 bytes --]

[-- Attachment #5: Screenshot_2020-02-02_20-59-58.png --]
[-- Type: image/png, Size: 28522 bytes --]

bug#39390: 28.0.50; prettify-symbols-mode confuses display when start/end faces are different

[Eli Zaretskii]

> Cc: 39390@debbugs.gnu.org
> From: Raphael 'kena' Poss <knz@thaumogen.net>
> Date: Sun, 2 Feb 2020 21:00:53 +0100
> 
> If you are telling me that (compose-region) should never have agreed to
> operate on a range of symbols with different faces, then I have
> counter-evidence that it does. If you are telling me that it shouldn't,
> then maybe that's where the bug really lies.

I'm telling that it cannot work well in these circumstances, and you
shouldn't expect it to.

bug#39390: 28.0.50; prettify-symbols-mode confuses display when start/end faces are different

[Raphael 'kena' Poss]

All right, so I take from our conversation so far that I should add
font-lock rules that ensure that the symbols that I am substituting have
a common face as per font-lock before the substitution by
prettify-symbols-mode takes place.

I have tried this:

(font-lock-add-keywords
 'go-mode
 '(("func(" . font-lock-keyword-face)))

(add-hook
 'go-mode-hook
 (lambda ()
   (push '("func(" . "λ(") prettify-symbols-alist)))

I have checked manually (by first disabling my prettify-symbols-alist
customization) that the font-lock customization indeed applies the same
face to all occurences of "func(" in the source.

However once the prettify-symbols substitution is active, the display
becomes messed up again.

Is it possible that this occurs because go-mode already pre-defines a
rule to apply keyword-face to "func", before my additional rule kicks
in? Do you reckon there is a way to remove the native rule defined by
go-mode so that mine remains the only that parses "func"?

-- 
Raphael 'kena' Poss

bug#39390: 28.0.50; prettify-symbols-mode confuses display when start/end faces are different

[Eli Zaretskii]

> Cc: 39390@debbugs.gnu.org
> From: Raphael 'kena' Poss <knz@thaumogen.net>
> Date: Sun, 2 Feb 2020 21:26:32 +0100
> 
> (font-lock-add-keywords
>  'go-mode
>  '(("func(" . font-lock-keyword-face)))
> 
> (add-hook
>  'go-mode-hook
>  (lambda ()
>    (push '("func(" . "λ(") prettify-symbols-alist)))
> 
> I have checked manually (by first disabling my prettify-symbols-alist
> customization) that the font-lock customization indeed applies the same
> face to all occurences of "func(" in the source.
> 
> However once the prettify-symbols substitution is active, the display
> becomes messed up again.
> 
> Is it possible that this occurs because go-mode already pre-defines a
> rule to apply keyword-face to "func", before my additional rule kicks
> in? Do you reckon there is a way to remove the native rule defined by
> go-mode so that mine remains the only that parses "func"?

Sorry, I don't know.

Would it be possible for you to prepare a reproducing recipe, starting
from "emacs -Q", and loading the minimum number of packages/features
required to show the problem?  Then I'll try to look at what happens
in the display code.

In general, static character composition used by prettify-symbols is a
deprecated feature, which doesn't fully support the current Emacs
display capabilities, so I'm not surprised there are problems when it
is used in "creative" ways.  But maybe there's an easy solution, who
knows.

Thanks.

bug#39390: 28.0.50; prettify-symbols-mode confuses display when start/end faces are different

[Raphael 'kena' Poss]

On 03-02-2020 16:50, Eli Zaretskii wrote:
> Would it be possible for you to prepare a reproducing recipe, starting
> from "emacs -Q", and loading the minimum number of packages/features
> required to show the problem?

Absolutely, there is not even a single package needed. The following
test file is sufficient to reproduce (run emacs -Q without argument then
copy/paste the entire text into the buffer before starting evaluation):

;; 1) evaluate the following:
(setq prettify-symbols-unprettify-at-point t)
(prettify-symbols-mode 1)
;; 2) observe: the substitution produces (setq abc (λ () t)) as
;;    expected.
;; 3) observe: moving the cursor in and out of the lambda signal
;;    expands the substitutions and back again.  (so far so good)
(setq abc (lambda () t))
;; 3) evaluate the following:
(push '("setq abc" . "@@") prettify-symbols-alist)
(prettify-symbols-mode 0)
(prettify-symbols-mode 1)
;; 4) observe immediately: the substitution has produced (@@ (λ () t))
;;    as expected.
;; 5) move the cursor into the line containing lambda, then around the
;;    substituted keyword then up and/or down.
;; 6) observe:
;;    - the opening parenthesis between "@@" and "λ" is
;;      non-deterministically rendered
;;    - the second "@" is improperly cleaned up when moving the cursor
;;      "into" the substitution






-- 
Raphael 'kena' Poss

bug#39390: 28.0.50; prettify-symbols-mode confuses display when start/end faces are different

[Eli Zaretskii]

> Cc: 39390@debbugs.gnu.org
> From: Raphael 'kena' Poss <knz@thaumogen.net>
> Date: Tue, 4 Feb 2020 23:07:07 +0100
> 
> On 03-02-2020 16:50, Eli Zaretskii wrote:
> > Would it be possible for you to prepare a reproducing recipe, starting
> > from "emacs -Q", and loading the minimum number of packages/features
> > required to show the problem?
> 
> Absolutely, there is not even a single package needed. The following
> test file is sufficient to reproduce (run emacs -Q without argument then
> copy/paste the entire text into the buffer before starting evaluation):

Sorry for the delay, I finally got to looking into this.

> ;; 1) evaluate the following:
> (setq prettify-symbols-unprettify-at-point t)
> (prettify-symbols-mode 1)
> ;; 2) observe: the substitution produces (setq abc (λ () t)) as
> ;;    expected.
> ;; 3) observe: moving the cursor in and out of the lambda signal
> ;;    expands the substitutions and back again.  (so far so good)
> (setq abc (lambda () t))

OK, I see that, too.

> ;; 3) evaluate the following:
> (push '("setq abc" . "@@") prettify-symbols-alist)
> (prettify-symbols-mode 0)
> (prettify-symbols-mode 1)
> ;; 4) observe immediately: the substitution has produced (@@ (λ () t))
> ;;    as expected.

Here' I see (@@ (λ () t)) instead (only one @).

> ;; 5) move the cursor into the line containing lambda, then around the
> ;;    substituted keyword then up and/or down.
> ;; 6) observe:
> ;;    - the opening parenthesis between "@@" and "λ" is
> ;;      non-deterministically rendered
> ;;    - the second "@" is improperly cleaned up when moving the cursor
> ;;      "into" the substitution

And I don't see any of these problems.  Not sure if this is related to
the fact that only one @ is displayed on my system, not 2.

I tested this in today's master branch; could it be that your build is
too old?

bug#39390: 28.0.50; prettify-symbols-mode confuses display when start/end faces are different

[Raphael 'kena' Poss]

[-- Attachment #1: Type: text/plain, Size: 599 bytes --]

On 16-02-2020 12:46, Eli Zaretskii wrote:

>> ;; 4) observe immediately: the substitution has produced (@@ (λ () t))
>> ;;    as expected.
> 
> Here' I see (@@ (λ () t)) instead (only one @).
> And I don't see any of these problems.  Not sure if this is related to
> the fact that only one @ is displayed on my system, not 2.

Look at what you wrote above: "You see (@@ ...) ... (only one @)"

See the attached screenshot.
Do you see one or two?


> I tested this in today's master branch; could it be that your build is
> too old?

Seems unlikely; it's from last week.



-- 
Raphael 'kena' Poss

[-- Attachment #2: Screenshot_2020-02-16_13-37-31.png --]
[-- Type: image/png, Size: 10852 bytes --]

bug#39390: 28.0.50; prettify-symbols-mode confuses display when start/end faces are different

[Eli Zaretskii]

> Cc: 39390@debbugs.gnu.org
> From: Raphael 'kena' Poss <knz@thaumogen.net>
> Date: Sun, 16 Feb 2020 13:37:49 -0500
> 
> > Here' I see (@@ (λ () t)) instead (only one @).
> > And I don't see any of these problems.  Not sure if this is related to
> > the fact that only one @ is displayed on my system, not 2.
> 
> Look at what you wrote above: "You see (@@ ...) ... (only one @)"
> 
> See the attached screenshot.
> Do you see one or two?

One.  Copy/paste reveals that there are two in the buffer, but they
display as one.

What did I miss?

bug#39390: 28.0.50; prettify-symbols-mode confuses display when start/end faces are different

[Eli Zaretskii]

> Date: Sun, 16 Feb 2020 21:29:50 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 39390@debbugs.gnu.org
> 
> One.  Copy/paste reveals that there are two in the buffer, but they
> display as one.

Actually, I take that back: what's in the buffer is "abc", and it
displays as a single @.  I simply copy/pasted from your message and
forgot to delete one @.  Sorry for the confusion.

bug#39390: 28.0.50; prettify-symbols-mode confuses display when start/end faces are different

[Raphael 'kena' Poss]

On 16-02-2020 14:34, Eli Zaretskii wrote:

> Actually, I take that back: what's in the buffer is "abc", and it
> displays as a single @.  I simply copy/pasted from your message and
> forgot to delete one @.  Sorry for the confusion.

1) so you agree there's a discrepancy between the configuration (which
should generate a _double_ @) and the display (which displays just one
@). I am expecting the result of the composition to display two @'s. I
would like to understand where this discrepancy comes from.

2) after we settle on what's the expected _display_ you can still see
the display bug as per my further instructions. I am still observing
inconsistent redraws, and I'd like to understand where they come from.

3) My original report was that _regardless of the intended result
display_ (either one or two chars or more), there is a refresh/redisplay
bug when the cursor moves across the composition. Instead of focusing on
the number of characters on the result of composition, I'd like to focus
on the display bug. If you do not observe the display bug with this
particular example config, I can produce a dozen alternate
configurations that demonstrate the bug, with screen recordings of each.
Let me know if that's useful.

Regards,

-- 
Raphael 'kena' Poss

bug#39390: 28.0.50; prettify-symbols-mode confuses display when start/end faces are different

[Eli Zaretskii]

> Cc: 39390@debbugs.gnu.org
> From: Raphael 'kena' Poss <knz@thaumogen.net>
> Date: Sun, 16 Feb 2020 14:40:04 -0500
> 
> 1) so you agree there's a discrepancy between the configuration (which
> should generate a _double_ @) and the display (which displays just one
> @). I am expecting the result of the composition to display two @'s. I
> would like to understand where this discrepancy comes from.

Your original report was about display problems with cursor motion, so
I didn't look into the issue of 1 vs 2 @'s, mainly because I wasn't
sure I reproduce the correct situation.

> 2) after we settle on what's the expected _display_ you can still see
> the display bug as per my further instructions. I am still observing
> inconsistent redraws, and I'd like to understand where they come from.

I don't see any display problems.

> 3) My original report was that _regardless of the intended result
> display_ (either one or two chars or more), there is a refresh/redisplay
> bug when the cursor moves across the composition. Instead of focusing on
> the number of characters on the result of composition, I'd like to focus
> on the display bug. If you do not observe the display bug with this
> particular example config

I don't.

> I can produce a dozen alternate configurations that demonstrate the
> bug, with screen recordings of each.

Please do, and thanks.

bug#39390: 28.0.50; prettify-symbols-mode confuses display when start/end faces are different

[Raphael 'kena' Poss]

On 16-02-2020 15:23, Eli Zaretskii wrote:

> Please do, and thanks.

My pleasure:

https://asciinema.org/a/3nOh1zCJYHOeQ4uXk0rQ9akv8



-- 
Raphael 'kena' Poss

bug#39390: 28.0.50; prettify-symbols-mode confuses display when start/end faces are different

[Tassilo Horn]

Raphael 'kena' Poss <knz@thaumogen.net> writes:

> https://asciinema.org/a/3nOh1zCJYHOeQ4uXk0rQ9akv8

No doubt you get strange display effects in your example, but I don't
know if that can be considered a bug given that your configuration is
illegal anyway.  See the docs of prettify-symbols-alist:

,----[ C-h v prettify-symbols-alist RET ]
| prettify-symbols-alist is a variable defined in ‘prog-mode.el’.
| Its value is nil
| 
|   Automatically becomes buffer-local when set.
| 
| Documentation:
| Alist of symbol prettifications.
| Each element looks like (SYMBOL . CHARACTER), where the symbol
| matching SYMBOL (a string, not a regexp) will be shown as
| CHARACTER instead.
| 
| CHARACTER can be a character, or it can be a list or vector, in
| which case it will be used to compose the new symbol as per the
| third argument of ‘compose-region’.
`----

I.e., the second element of each pair in `prettify-symbols-alist' is a
simple replacement glyph only in the case were it is a character.  You
provide strings instead, so the outcome can only be determined by
checking what `compose-region' does with that input.

AFAIK, `prettify-symbols-mode's intent is to replace programming
language symbols (like "lambda" in Lisp or >>= in Haskell) with a fancy
single Unicode character each, not to replace any text with any
arbitrary other text.  (Also, using multi-word matches like "setq abc"
or unbalanced expressions like "(push" is definitely asking for
trouble.)

Bye,
Tassilo

bug#39390: 28.0.50; prettify-symbols-mode confuses display when start/end faces are different

[Raphael 'kena' Poss]

Hi Tassilo,

thank you for joining the conversation.

to summarize your standpoint is that:

1) configurations that specify a string are invalid.
2) multi-word matches are "asking for trouble".


Regarding point (1). First of all the function compose-region does
accept a string as per its documentation. Separately if there is
consensus that this type of configuration is invalid, the function
should be programmed to reject it.

Finally, let us not distract ourselves with this detail. The bug is
readily reproducible with a single character:

https://asciinema.org/a/IGDZhOMnmF7sAWJYEjRqPdboA


Regarding point (2). Multiple characters in the output actually works
fine most of the time. Please double check the title of this e-mail
thread and the reason why I reported the issue in the first place:

- replacing either a single character or multiple characters in the
buffer, even separated by blanks, is fine as long as they display with
just 1 face.

- replacing anything with either a single or multiple characters in the
result of the composition is fine as long as the input uses just 1 face.


There is an error in the display code when compose-region composes over
multiple faces, and I'd like us to focus on that. I'm pretty sure that
pretty-symbols-mode is just one of multiple ways one can trigger this bug.

-- 
Raphael 'kena' Poss

bug#39390: 28.0.50; prettify-symbols-mode confuses display when start/end faces are different

[Tassilo Horn]

Raphael 'kena' Poss <knz@thaumogen.net> writes:

> Finally, let us not distract ourselves with this detail. The bug is
> readily reproducible with a single character:
>
> https://asciinema.org/a/IGDZhOMnmF7sAWJYEjRqPdboA
>
> Regarding point (2). Multiple characters in the output actually works
> fine most of the time. Please double check the title of this e-mail
> thread and the reason why I reported the issue in the first place:
>
> - replacing either a single character or multiple characters in the
> buffer, even separated by blanks, is fine as long as they display with
> just 1 face.
>
> - replacing anything with either a single or multiple characters in
> the result of the composition is fine as long as the input uses just 1
> face.
>
> There is an error in the display code when compose-region composes
> over multiple faces, and I'd like us to focus on that.  I'm pretty
> sure that pretty-symbols-mode is just one of multiple ways one can
> trigger this bug.

Indeed, that's the real problem.  I guess that there's an implicit
assumption that composition always takes place in one word or symbol
which is almost always fontified with just one face.  Your example
invalidates that assumption.

But then the question is how the composition should be displayed?  In
your example where you replace "setq abc" with the LAST QUARTER MOON
WITH FACE Unicode character, should that have font-lock-keyword-face
(like setq) or the default face (like abc)?

Bye,
Tassilo

bug#39390: 28.0.50; prettify-symbols-mode confuses display when start/end faces are different

[Raphael 'kena' Poss]

On 17-02-2020 10:59, Tassilo Horn wrote:

> But then the question is how the composition should be displayed?  In
> your example where you replace "setq abc" with the LAST QUARTER MOON
> WITH FACE Unicode character, should that have font-lock-keyword-face
> (like setq) or the default face (like abc)?

I honestly don't have a preference. I don't mind either the first face,
the last one, or some other in the middle. I don't want to be picky
here, the first good next step would be to ensure that the display
refreshes are not erratic and do not corrupt the display.

Regards


-- 
Raphael 'kena' Poss

bug#39390: 28.0.50; prettify-symbols-mode confuses display when start/end faces are different

[Eli Zaretskii]

> From: Tassilo Horn <tsdh@gnu.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  39390@debbugs.gnu.org
> Date: Mon, 17 Feb 2020 16:59:38 +0100
> 
> > There is an error in the display code when compose-region composes
> > over multiple faces, and I'd like us to focus on that.  I'm pretty
> > sure that pretty-symbols-mode is just one of multiple ways one can
> > trigger this bug.
> 
> Indeed, that's the real problem.  I guess that there's an implicit
> assumption that composition always takes place in one word or symbol
> which is almost always fontified with just one face.  Your example
> invalidates that assumption.

As I explained at the very beginning of this discussion, this is not
an assumption, this is part of the basic design of the Emacs display
engine: it _never_ considers characters in different faces together,
and thus can never compose them.  We could perhaps lift this
limitation when faces differ only in colors (thus we will be able to
support character compositions when part of them are inside a
highlighted region), but a face in general specifies also the font,
its dimensions, weight, slant, etc., and we cannot possibly combine
glyphs that come from different fonts.  So this limitation is not
really arbitrary, and can only be lifted by a thorough redesign of how
the display engine traverses the text it is about to display, and what
it does when it meets a composition that crosses face boundaries.

Let me also remind you that character composition was introduced for
purposes very different from what pretty-symbols-mode does; it
definitely wasn't supposed to support arbitrary display of one text
as another: for that you have display strings and overlays.

> But then the question is how the composition should be displayed?  In
> your example where you replace "setq abc" with the LAST QUARTER MOON
> WITH FACE Unicode character, should that have font-lock-keyword-face
> (like setq) or the default face (like abc)?

Imagine that font-lock-keyword-face specifies bold-italic typeface, or
maybe even a different size of the font, for example.  There be
dragons.

bug#39390: 28.0.50; prettify-symbols-mode confuses display when start/end faces are different

[Raphael 'kena' Poss]

On 17-02-2020 12:30, Eli Zaretskii wrote:
> So this limitation is not
> really arbitrary, and can only be lifted by a thorough redesign of how
> the display engine traverses the text it is about to display, and what
> it does when it meets a composition that crosses face boundaries.

My humble opinion on this is that either the compose function should
report an exception / error, or pick some arbitrary behavior (e.g. just
use the 1st face for the entire substitution) and use that.

It seems strange to me to accept the current non-deterministic,
display-corrupting behavior as a reasonable alternative.

-- 
Raphael 'kena' Poss

bug#39390: 28.0.50; prettify-symbols-mode confuses display when start/end faces are different

[Eli Zaretskii]

> Cc: 39390@debbugs.gnu.org
> From: Raphael 'kena' Poss <knz@thaumogen.net>
> Date: Mon, 17 Feb 2020 13:37:07 -0500
> 
> On 17-02-2020 12:30, Eli Zaretskii wrote:
> > So this limitation is not
> > really arbitrary, and can only be lifted by a thorough redesign of how
> > the display engine traverses the text it is about to display, and what
> > it does when it meets a composition that crosses face boundaries.
> 
> My humble opinion on this is that either the compose function should
> report an exception / error, or pick some arbitrary behavior (e.g. just
> use the 1st face for the entire substitution) and use that.

We cannot raise an exception inside redisplay, because that would
produce an infinite sequence of error messages (each error message
requires a redisplay cycle -- to display the message -- which then
again raises the same exception).

> It seems strange to me to accept the current non-deterministic,
> display-corrupting behavior as a reasonable alternative.

We are talking about 2 different things.  I was talking about
composing characters across face changes, whereas you were talking
about handling invalid composition rules, such as the one you tried to
use.

bug#39390: 28.0.50; prettify-symbols-mode confuses display when start/end faces are different

[Raphael 'kena' Poss]

On 17-02-2020 14:24, Eli Zaretskii wrote:

>> My humble opinion on this is that either the compose function should
>> report an exception / error, or pick some arbitrary behavior (e.g. just
>> use the 1st face for the entire substitution) and use that.
> 
> We cannot raise an exception inside redisplay, because that would
> produce an infinite sequence of error messages (each error message
> requires a redisplay cycle -- to display the message -- which then
> again raises the same exception).

Yes I understand that but I was thinking that `compose-region` would be
the place to check validity.

>> It seems strange to me to accept the current non-deterministic,
>> display-corrupting behavior as a reasonable alternative.
> 
> We are talking about 2 different things.  I was talking about
> composing characters across face changes, whereas you were talking
> about handling invalid composition rules, such as the one you tried to
> use.

Look there is nothing else that determines what is "valid" and "invalid"
than what the docs say and what emacs allow me to configure. At this
point neither the doc of `compose-region` nor its code prevent me from
using this configuration, so from the user's perspective (mine) the
configuration is valid -- and instead I find a bug in the redisplay code.

If you want to hold a position that the configuration is invalid, Emacs
(or some function) needs to be first taught to refuse it.

Meanwhile if the configuration can be deemed valid but with reduced
functionality (as in, the behavior is limited to only use one face, the
first one found in the composition), that could also be documented.

-- 
Raphael 'kena' Poss

bug#39390: 28.0.50; prettify-symbols-mode confuses display when start/end faces are different

[Eli Zaretskii]

> Cc: tsdh@gnu.org, 39390@debbugs.gnu.org
> From: Raphael 'kena' Poss <knz@thaumogen.net>
> Date: Mon, 17 Feb 2020 14:28:09 -0500
> 
> > We are talking about 2 different things.  I was talking about
> > composing characters across face changes, whereas you were talking
> > about handling invalid composition rules, such as the one you tried to
> > use.
> 
> Look there is nothing else that determines what is "valid" and "invalid"
> than what the docs say and what emacs allow me to configure. At this
> point neither the doc of `compose-region` nor its code prevent me from
> using this configuration, so from the user's perspective (mine) the
> configuration is valid -- and instead I find a bug in the redisplay code.
> 
> If you want to hold a position that the configuration is invalid, Emacs
> (or some function) needs to be first taught to refuse it.

If someone wants to submit patches to detect that this is indeed
invalid, those patches will be welcome.  But I must note that it will
not be easy, because compose-region _can_ accept strings, it just
assigns special meaning to the string's characters, as described in
the doc string.  IOW, your string is "valid" by being a string, but
"invalid" in that it causes incorrect display.

bug#39390: 28.0.50; prettify-symbols-mode confuses display when start/end faces are different

[Eli Zaretskii]

> From: Tassilo Horn <tsdh@gnu.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  39390@debbugs.gnu.org
> Date: Mon, 17 Feb 2020 12:17:19 +0100
> 
> Raphael 'kena' Poss <knz@thaumogen.net> writes:
> 
> > https://asciinema.org/a/3nOh1zCJYHOeQ4uXk0rQ9akv8
> 
> No doubt you get strange display effects in your example, but I don't
> know if that can be considered a bug given that your configuration is
> illegal anyway.

You mean "invalid", right? ;-)

bug#39390: 28.0.50; prettify-symbols-mode confuses display when start/end faces are different

[Eli Zaretskii]

> Cc: 39390@debbugs.gnu.org
> From: Raphael 'kena' Poss <knz@thaumogen.net>
> Date: Sun, 16 Feb 2020 21:47:43 -0500
> 
> On 16-02-2020 15:23, Eli Zaretskii wrote:
> 
> > Please do, and thanks.
> 
> My pleasure:
> 
> https://asciinema.org/a/3nOh1zCJYHOeQ4uXk0rQ9akv8

Ah, "emacs -Q -nw"!  You never said that before, and your recipe
explicitly says "emacs -Q".

With -nw, I do see two @@ as you say, and also some irregularities in
the display.  But your data is not according to documentation, so I
guess that's why the result misbehaves.