* bug#70000: 29.2; Grapheme handling incorrect @ 2024-03-25 18:45 Phillip Susi 2024-03-25 19:35 ` Eli Zaretskii 0 siblings, 1 reply; 5+ messages in thread From: Phillip Susi @ 2024-03-25 18:45 UTC (permalink / raw) To: 70000 I had some terminal breakage the other day when browsing email with notmuch. Now a ways down the rabbit hole, it seems this is because emacs does not correctly handle graphemes. I found this article here: https://mitchellh.com/writing/grapheme-clusters-in-terminals If I paste that gramehe into GUI emacs, it is displayed as two separate characters, each two columns wide, instead of the correct way: as a single double wide character. C-f and C-b move over the character as if it were one, however, backspace deletes only the second, leaving both the first and the zero width joiner. If C-f and C-b treat it as one, then so should backspace. Under recent versions of the foot terminal emulator, this character is displayed as a single, double wide character, but emacs assumes it still is 4 colums wide, leading to terminal breakage. Emacs needs to not assume the width of graphemes are what wcwidth() reports, but instead need to query the cursor position after printing one to find out how wide the terminal actually dispalyed it as. In GNU Emacs 29.2 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.39, cairo version 1.18.0) of 2024-02-26 built on localhost System Description: Gentoo Linux Configured using: 'configure --prefix=/usr --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu --mandir=/usr/share/man --infodir=/usr/share/info --datadir=/usr/share --sysconfdir=/etc --localstatedir=/var/lib --datarootdir=/usr/share --disable-silent-rules --docdir=/usr/share/doc/emacs-29.2-r1 --htmldir=/usr/share/doc/emacs-29.2-r1/html --libdir=/usr/lib64 --program-suffix=-emacs-29 --includedir=/usr/include/emacs-29 --infodir=/usr/share/info/emacs-29 --localstatedir=/var --enable-locallisppath=/etc/emacs:/usr/share/emacs/site-lisp --without-compress-install --without-hesiod --without-pop --with-file-notification=inotify --with-pdumper --enable-acl --with-dbus --with-modules --without-gameuser --with-libgmp --with-gpm --with-native-compilation=aot --without-json --without-kerberos --without-kerberos5 --with-lcms2 --without-xml2 --without-mailutils --without-selinux --without-sqlite3 --with-gnutls --with-libsystemd --with-threads --with-tree-sitter --without-wide-int --with-sound=alsa --with-zlib --with-pgtk --without-x --without-ns --with-toolkit-scroll-bars --without-gconf --without-gsettings --without-harfbuzz --without-libotf --without-m17n-flt --without-xwidgets --with-gif --with-jpeg --with-png --with-rsvg --with-tiff --without-webp --without-imagemagick --with-dumping=pdumper 'CFLAGS=-march=native -O2 -pipe' 'LDFLAGS=-Wl,-O1 -Wl,--as-needed'' Configured features: ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM JPEG LCMS2 LIBSYSTEMD MODULES NATIVE_COMP NOTIFY INOTIFY PDUMPER PGTK PNG RSVG SECCOMP SOUND THREADS TIFF TOOLKIT_SCROLL_BARS TREE_SITTER XIM GTK3 ZLIB Important settings: value of $LANG: en_US.UTF-8 locale-coding-system: utf-8-unix Major mode: Lisp Interaction Minor modes in effect: tooltip-mode: t global-eldoc-mode: t eldoc-mode: t show-paren-mode: t electric-indent-mode: t mouse-wheel-mode: t menu-bar-mode: t file-name-shadow-mode: t global-font-lock-mode: t font-lock-mode: t blink-cursor-mode: t column-number-mode: t line-number-mode: t indent-tabs-mode: t transient-mark-mode: t auto-composition-mode: t auto-encryption-mode: t auto-compression-mode: t Load-path shadows: None found. Features: (shadow sort mail-extr emacsbug message yank-media puny dired dired-loaddefs rfc822 mml mml-sec epa derived epg rfc6068 epg-config gnus-util text-property-search time-date mm-decode mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils cus-start cus-load wid-edit descr-text enriched disp-table facemenu comp comp-cstr warnings icons rx cl-extra help-mode manoj-dark-theme site-gentoo ranger-autoloads scopeline-autoloads package browse-url url url-proxy url-privacy url-expand url-methods url-history url-cookie generate-lisp-file url-domsuf url-util mailcap url-handlers url-parse auth-source cl-seq eieio eieio-core cl-macs password-cache json subr-x map byte-opt gv bytecomp byte-compile url-vars cl-loaddefs cl-lib rmc iso-transl tooltip cconv eldoc paren electric uniquify ediff-hook vc-hooks lisp-float-type elisp-mode mwheel term/pgtk-win pgtk-win term/common-win pgtk-dnd tool-bar dnd fontset image regexp-opt fringe tabulated-list replace newcomment text-mode lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu timer select scroll-bar mouse jit-lock font-lock syntax font-core term/tty-colors frame minibuffer nadvice seq simple cl-generic indonesian philippine cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european ethiopic indian cyrillic chinese composite emoji-zwj charscript charprop case-table epa-hook jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs theme-loaddefs faces cus-face macroexp files window text-properties overlay sha1 md5 base64 format env code-pages mule custom widget keymap hashtable-print-readable backquote threads dbusbind inotify dynamic-setting font-render-setting cairo gtk pgtk lcms2 multi-tty make-network-process native-compile emacs) Memory information: ((conses 16 121243 14450) (symbols 48 22924 0) (strings 32 87992 2869) (string-bytes 1 2065634) (vectors 16 27491) (vector-slots 8 1623278 223666) (floats 8 58 48) (intervals 56 908 0) (buffers 984 13)) ^ permalink raw reply [flat|nested] 5+ messages in thread
* bug#70000: 29.2; Grapheme handling incorrect 2024-03-25 18:45 bug#70000: 29.2; Grapheme handling incorrect Phillip Susi @ 2024-03-25 19:35 ` Eli Zaretskii 2024-03-27 14:11 ` Phillip Susi 0 siblings, 1 reply; 5+ messages in thread From: Eli Zaretskii @ 2024-03-25 19:35 UTC (permalink / raw) To: Phillip Susi; +Cc: 70000 tags 70000 notabug thanks > From: Phillip Susi <phill@thesusis.net> > Date: Mon, 25 Mar 2024 14:45:48 -0400 > > I had some terminal breakage the other day when browsing email with > notmuch. Now a ways down the rabbit hole, it seems this is because > emacs does not correctly handle graphemes. I found this article here: > > https://mitchellh.com/writing/grapheme-clusters-in-terminals > > If I paste that gramehe into GUI emacs, it is displayed as two separate > characters, each two columns wide, instead of the correct way: as a > single double wide character. First, the above blog talks about text-mode terminals (a.k.a. "TTYs"), so it is not relevant to GUI Emacs session. And second, how that particular sequence of codepoints is displayed on GUI frames depends on how your Emacs was built. According to the list of features included in your report, viz.: Configured features: ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM JPEG LCMS2 LIBSYSTEMD MODULES NATIVE_COMP NOTIFY INOTIFY PDUMPER PGTK PNG RSVG SECCOMP SOUND THREADS TIFF TOOLKIT_SCROLL_BARS TREE_SITTER XIM GTK3 ZLIB your Emacs is built without HarfBuzz, which I think explains why your Emacs displays the above sequences as 2 separate characters. Furthermore, the appearance depends on the fonts you have installed; specifically, Emoji sequences need a font that has a good support of the Emoji Unicode blocks. In my Emacs, which does use HarfBuzz, I see a single grapheme cluster. > C-f and C-b move over the character as if > it were one, however, backspace deletes only the second, leaving both > the first and the zero width joiner. If C-f and C-b treat it as one, > then so should backspace. That Backspace deletes a single codepoint is a feature: it allows easier editing of composable character sequences, such as Emoji. E.g., imagine you want to make a slight change to the Emoji by modifying just the second of the two characters composed into a grapheme cluster. Emacs supports deletion of the entire grapheme cluster with the command delete-forward-char, by default bound to the <Delete> function key. > Under recent versions of the foot terminal emulator, this character is > displayed as a single, double wide character, but emacs assumes it still > is 4 colums wide, leading to terminal breakage. Emacs cannot know what the terminal does with these characters, because there's no widely-accepted protocol for accessing that information. Different terminal emulators behave differently, and some even have options to modify their behavior via the various settings. > Emacs needs to not assume the width of graphemes are what wcwidth() > reports, but instead need to query the cursor position after > printing one to find out how wide the terminal actually dispalyed it > as. Querying the cursor position won't help in this case because it is Emacs that moves the cursor when you type C-f, not the terminal. I see no Emacs bug here. Until we have standard ways of querying text-mode terminals about their processing of composable character sequences into grapheme clusters, there's no way for Emacs to behave correctly with all such terminal emulators. Sorry. ^ permalink raw reply [flat|nested] 5+ messages in thread
* bug#70000: 29.2; Grapheme handling incorrect 2024-03-25 19:35 ` Eli Zaretskii @ 2024-03-27 14:11 ` Phillip Susi 2024-03-27 17:17 ` Eli Zaretskii 0 siblings, 1 reply; 5+ messages in thread From: Phillip Susi @ 2024-03-27 14:11 UTC (permalink / raw) To: Eli Zaretskii; +Cc: 70000 Eli Zaretskii <eliz@gnu.org> writes: > Querying the cursor position won't help in this case because it is > Emacs that moves the cursor when you type C-f, not the terminal. I'm not talking about C-f, but simply displaying the characters on the screen. Emacs assumes the width is 4 when it prints this character, and so it thinks that the cursor moved over 4 places. When the terminal actually only moves the cursor over 2 spaces, emacs gets out of sync with the terminal, and massive breakage occurs. By reading back the cursor position from the terminal after displaying a grapheme cluster, it would learn how the terminal displayed it and update its idea of where the cursor is correctly. I originally ran into this problem not with a ZWJ, but with an emoji followed by alternate selector 16 that someone used in a subject line of an email, and when browsing my inbox with notmuch, the terminal went FUBAR. ^ permalink raw reply [flat|nested] 5+ messages in thread
* bug#70000: 29.2; Grapheme handling incorrect 2024-03-27 14:11 ` Phillip Susi @ 2024-03-27 17:17 ` Eli Zaretskii 2024-03-28 16:16 ` Phillip Susi 0 siblings, 1 reply; 5+ messages in thread From: Eli Zaretskii @ 2024-03-27 17:17 UTC (permalink / raw) To: Phillip Susi; +Cc: 70000 > From: Phillip Susi <phill@thesusis.net> > Cc: 70000@debbugs.gnu.org > Date: Wed, 27 Mar 2024 10:11:30 -0400 > > Eli Zaretskii <eliz@gnu.org> writes: > > > Querying the cursor position won't help in this case because it is > > Emacs that moves the cursor when you type C-f, not the terminal. > > I'm not talking about C-f, but simply displaying the characters on the > screen. Emacs assumes the width is 4 when it prints this character, and > so it thinks that the cursor moved over 4 places. When the terminal > actually only moves the cursor over 2 spaces, emacs gets out of sync > with the terminal, and massive breakage occurs. I understand what you are saying, but this is not how Emacs display code works. It needs to know the width of every character displayed on the screen, and it needs to be able to determine that even without actually displaying the character. When Emacs is about to redraw some portion of the screen, it moves the cursor to that place. To be able to move the cursor there, it needs to be able to compute the coordinates on the screen of every character that is currently shown, so it can construct the command for the terminal driver to move cursor to that place. If Emacs were to rely on displaying characters for that, it would have needed to constantly redraw large portions of the screen, and that would both be much slower and cause unpleasant flickering of the display, due to redrawing of screen portions that don't actually change. So this technique is out of the question for Emacs. > By reading back the cursor position from the terminal after displaying a > grapheme cluster, it would learn how the terminal displayed it and > update its idea of where the cursor is correctly. I understand. But Emacs needs this information also long after the characters were already drawn. For example, imagine that Emacs displays these characters on the screen, and then leaves most of the screen intact and periodically redraws some small portion of the screen, like updating current time in the lower-right corner of the screen when Emacs is otherwise idle. To do that, Emacs needs to move the cursor from its current position somewhere on the screen to the lower-right corner, redraw the time there, then move the cursor back to where it was. These cursor moves are based on the ability to calculate the geometry of each character on display without actually writing the characters to the screen. In addition, if Emacs had to query the cursor position after each written character, its redisplay would be much slower than it is now. > I originally ran into this problem not with a ZWJ, but with an emoji > followed by alternate selector 16 that someone used in a subject line of > an email, and when browsing my inbox with notmuch, the terminal went > FUBAR. Yes, that's a known issue with some of the terminal emulators that compose Emoji and other similar character sequences into grapheme clusters, while ignoring the width that is expected from the result. I'm not aware of any good solution, unfortunately. Sometimes, disabling auto-composition-mode helps, but even that cannot solve all the problems, especially when each of the characters composed by the terminal into a single grapheme cluster has non-zero width according to the Unicode tables. (If only the first character in the composed sequence has non-zero width and the rest are zero-width, disabling auto-composition-mode might produce a correct display.) The bottom line is what I said at the beginning: we need some protocol by which a terminal emulator could be queried about whether it supports character composition, and if so, what is the screen width of a given sequence of codepoints that will be composed, without actually displaying them. Better yet, some standard table of such widths could be accepted by complying terminal emulators, and then Emacs could use such a table to know the width in advance (similarly to how it knows that from the Unicode data files). Until such protocols or tables exist, Emacs will be unable to produce correct display on these terminal emulators. ^ permalink raw reply [flat|nested] 5+ messages in thread
* bug#70000: 29.2; Grapheme handling incorrect 2024-03-27 17:17 ` Eli Zaretskii @ 2024-03-28 16:16 ` Phillip Susi 0 siblings, 0 replies; 5+ messages in thread From: Phillip Susi @ 2024-03-28 16:16 UTC (permalink / raw) To: Eli Zaretskii; +Cc: 70000 Eli Zaretskii <eliz@gnu.org> writes: > I understand. But Emacs needs this information also long after the > characters were already drawn. For example, imagine that Emacs Yes, it would have to learn the width the first time it displays each grapheme and build a list of known widths to remember for future use. > In addition, if Emacs had to query the cursor position after each > written character, its redisplay would be much slower than it is now. It would only need to query when printing a grapheme cluster, and only the first time. After that, it could remeber. ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2024-03-28 16:16 UTC | newest] Thread overview: 5+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2024-03-25 18:45 bug#70000: 29.2; Grapheme handling incorrect Phillip Susi 2024-03-25 19:35 ` Eli Zaretskii 2024-03-27 14:11 ` Phillip Susi 2024-03-27 17:17 ` Eli Zaretskii 2024-03-28 16:16 ` Phillip Susi
Code repositories for project(s) associated with this public inbox https://git.savannah.gnu.org/cgit/emacs.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).