bug#57214: 29.0.50; Incorrect rendering of Emoji (grapheme cluster) in Org heading

bug#57214: 29.0.50; Incorrect rendering of Emoji (grapheme cluster) in Org heading

[Protesilaos Stavrou]

[-- Attachment #1: Type: text/plain, Size: 2081 bytes --]

Dear maintainers,

I have encountered a bug where grapheme clusters are not rendered
properly when used inside of a folded heading in Org.

Steps to reproduce with 'emacs -Q':

1. Evaluate:

      (setq org-pretty-entties t
            org-ellipsis " ☀️")

2. Either visit an existing Org file or use the attached sample.

3. Fold a heading in Org by pressing TAB with point somewhere on the
   heading.

4. Notice the incorrectly rendered sun emoji at the end of the folded
   heading.

If I change 'org-ellipsis' to a single character emoji, such as the
generic smile, the problem no longer occurs:

* Evaluate:

      (setq org-ellipsis " 😀")

* Then in the Org file do 'M-x org-mode-restart'.

The smile emoji renders properly at the end of the folded heading's
line.

The attached screenshots show the different results.

All the best,
Protesilaos (or simply "Prot")

* * *

In GNU Emacs 29.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.34, cairo version 1.17.6)
 of 2022-08-14 built on kronos
Repository revision: b93e14fa0fd5833adbdd88ec86fccc4b59172302
Repository branch: master
System Description: Arch Linux

Configured using:
 'configure --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib
 --localstatedir=/var --mandir=/usr/share/man --with-gameuser=:games
 --without-libotf --without-m17n-flt --without-gconf
 --with-native-compilation --with-pgtk --with-sound=no --without-gpm
 --without-compress-install
 '--program-transform-name=s/\([ec]tags\)/\1.emacs/'
 'CFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions
 -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security
 -fstack-clash-protection -fcf-protection'
 LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG JSON
LCMS2 LIBSYSTEMD LIBXML2 MODULES NATIVE_COMP NOTIFY INOTIFY PDUMPER PGTK
PNG RSVG SECCOMP SQLITE3 THREADS TIFF TOOLKIT_SCROLL_BARS WEBP XIM GTK3
ZLIB

-- 
Protesilaos Stavrou
https://protesilaos.com

[-- Attachment #2: sample.org --]
[-- Type: application/vnd.lotus-organizer, Size: 65 bytes --]

[-- Attachment #3: Screenshot from 2022-08-14 21-39-19.png --]
[-- Type: image/png, Size: 48974 bytes --]

[-- Attachment #4: Screenshot from 2022-08-14 21-39-44.png --]
[-- Type: image/png, Size: 51849 bytes --]

bug#57214: 29.0.50; Incorrect rendering of Emoji (grapheme cluster) in Org heading

[Eli Zaretskii]

> Cc: summeremacs@gmail.com
> From: Protesilaos Stavrou <info@protesilaos.com>
> Date: Sun, 14 Aug 2022 21:47:21 +0300
> 
> I have encountered a bug where grapheme clusters are not rendered
> properly when used inside of a folded heading in Org.
> 
> Steps to reproduce with 'emacs -Q':
> 
> 1. Evaluate:
> 
>       (setq org-pretty-entties t
>             org-ellipsis " ☀️")
> 
> 2. Either visit an existing Org file or use the attached sample.
> 
> 3. Fold a heading in Org by pressing TAB with point somewhere on the
>    heading.
> 
> 4. Notice the incorrectly rendered sun emoji at the end of the folded
>    heading.
> 
> If I change 'org-ellipsis' to a single character emoji, such as the
> generic smile, the problem no longer occurs:

This is a limitation of how the ellipsis display is designed and
implemented in Emacs: we display the ellipsis via the buffer's
display-table, and character compositions aren't applied to glyphs
that come from the display-table.  You can clearly see this from how
org.el prepares the display-table:

  (when (and (stringp org-ellipsis) (not (equal "" org-ellipsis)))
    (unless org-display-table
      (setq org-display-table (make-display-table)))
    (set-display-table-slot
     org-display-table 4
     (vconcat (mapcar (lambda (c) (make-glyph-code c 'org-ellipsis))
		      org-ellipsis)))
    (setq buffer-display-table org-display-table))

As you see, the code puts each individual glyph of the Emoji sequence,
marked with its face, into a vector.  When the display engine uses the
glyphs from the display-table, it doesn't interpret them in any way.

So the only way this can work is if the font used for Emoji has a
precomposed glyph for the sequence you want, and that precomposed
glyph has a character code you could use instead of the string.

bug#57214: 29.0.50; Incorrect rendering of Emoji (grapheme cluster) in Org heading

[Protesilaos Stavrou]

> From: Eli Zaretskii <eliz@gnu.org>
> Date: Sun, 14 Aug 2022 22:12:11 +0300
>
> [... 25 lines elided]
>
> This is a limitation of how the ellipsis display is designed and
> implemented in Emacs: we display the ellipsis via the buffer's
> display-table, and character compositions aren't applied to glyphs
> that come from the display-table.  You can clearly see this from how
> org.el prepares the display-table:

I see.  Thank you Eli for your time!

Should we close this or there is potential to change things?

-- 
Protesilaos Stavrou
https://protesilaos.com

bug#57214: 29.0.50; Incorrect rendering of Emoji (grapheme cluster) in Org heading

[Eli Zaretskii]

> From: Protesilaos Stavrou <info@protesilaos.com>
> Cc: 57214@debbugs.gnu.org, summeremacs@gmail.com
> Date: Mon, 15 Aug 2022 10:37:17 +0300
> 
> > From: Eli Zaretskii <eliz@gnu.org>
> > Date: Sun, 14 Aug 2022 22:12:11 +0300
> >
> > [... 25 lines elided]
> >
> > This is a limitation of how the ellipsis display is designed and
> > implemented in Emacs: we display the ellipsis via the buffer's
> > display-table, and character compositions aren't applied to glyphs
> > that come from the display-table.  You can clearly see this from how
> > org.el prepares the display-table:
> 
> I see.  Thank you Eli for your time!
> 
> Should we close this or there is potential to change things?

The potential always exists, although the implementation will not be
trivial, because we'll need to invent a new kind of "display vector"
to hold the font glyph codes (or maybe just the composition ID?), and
implement the code to support its display.

Is this an important feature to have?

bug#57214: 29.0.50; Incorrect rendering of Emoji (grapheme cluster) in Org heading

[Protesilaos Stavrou]

> From: Eli Zaretskii <eliz@gnu.org>
> Date: Mon, 15 Aug 2022 14:43:25 +0300
>
> [... 17 lines elided]
>
>> Should we close this or there is potential to change things?
>
> The potential always exists, although the implementation will not be
> trivial, because we'll need to invent a new kind of "display vector"
> to hold the font glyph codes (or maybe just the composition ID?), and
> implement the code to support its display.

Understood.

> Is this an important feature to have?

No.  I just wanted to be sure that it is not trivial to change.

Closing the issue now.  Thank you for your time!

-- 
Protesilaos Stavrou
https://protesilaos.com