bug#52055: 29.0.50; emoji-insert garbles terminal display on tsdh-dark theme

bug#52055: 29.0.50; emoji-insert garbles terminal display on tsdh-dark theme

[bird]

Hello,

The display of emacs becomes weird when using emoji-insert on a terminal
emulator.

To reproduce:
emacs -Q -nw
M-x customize-themes
select tsdh-dark and return to *scratch* buffer
M-x emoji-insert (this should already show some problems)
select smileys (s) (it's garbled even more)

I can reproduce this with st terminal with lucid toolkit.


In GNU Emacs 29.0.50 (build 4, x86_64-pc-linux-musl, X toolkit, cairo version 1.16.0, Xaw scroll bars)
 of 2021-11-20 built on laptop
Repository revision: 560a6c35cb274af6b79f89eddbdbb2fd6c00e2b0
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12013000
System Description: void

Configured using:
 'configure --with-imagemagick --with-cairo --with-modules
 --without-dbus --without-gconf --without-gsettings
 --with-x-toolkit=lucid --prefix=/usr'

Configured features:
ACL CAIRO FREETYPE GIF GLIB GMP GNUTLS HARFBUZZ IMAGEMAGICK JPEG JSON
LCMS2 LIBOTF LIBXML2 MODULES NOTIFY INOTIFY PDUMPER PNG RSVG SECCOMP
SOUND THREADS TIFF TOOLKIT_SCROLL_BARS X11 XDBE XIM XPM LUCID ZLIB

Important settings:
  value of $LC_COLLATE: C
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  pdf-occur-global-minor-mode: t
  global-display-line-numbers-mode: t
  display-line-numbers-mode: t
  repeat-mode: t
  fido-vertical-mode: t
  icomplete-vertical-mode: t
  icomplete-mode: t
  fido-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort flyspell ispell mail-extr emacsbug sendmail nndraft nnmh
nnmaildir gnus-agent gnus-srvr gnus-score score-mode nnvirtual gnus-msg
gnus-art mm-uu mml2015 mm-view mml-smime smime dig nntp gnus-cache
gnus-sum shr kinsoku svg dom gnus-group gnus-undo gnus-start gnus-dbus
dbus xml gnus-cloud nnimap nnmail mail-source utf7 netrc nnoo parse-time
gnus-spec gnus-int gnus-range message yank-media rmc puny rfc822 mml
mml-sec epa derived epg rfc6068 epg-config mm-decode mm-bodies mm-encode
mail-parse rfc2231 mailabbrev gmm-utils mailheader gnus-win gnus
nnheader gnus-util rmail rmail-loaddefs rfc2047 rfc2045 ietf-drums
mail-utils mm-util mail-prsvr tsdh-dark-theme org-tempo org ob ob-tangle
ob-ref ob-lob ob-table ob-exp org-macro org-footnote org-src ob-comint
org-pcomplete pcomplete org-list org-faces org-entities noutline outline
easy-mmode org-version ob-emacs-lisp ob-core ob-eval org-table oc-basic
bibtex iso8601 time-date ol rx org-keys oc org-compat org-macs
org-loaddefs cal-menu calendar cal-loaddefs tempo pdf-occur ibuf-ext
ibuffer ibuffer-loaddefs tablist tablist-filter semantic/wisent/comp
semantic/wisent semantic/wisent/wisent semantic/util-modes semantic/util
semantic semantic/tag semantic/lex semantic/fw mode-local find-func
cedet pdf-isearch let-alist pdf-misc imenu pdf-tools compile comint
ansi-color ring cus-edit cus-load wid-edit pdf-view bookmark
text-property-search pp jka-compr pdf-cache pdf-info tq pdf-util advice
format-spec pdf-macs image-mode dired dired-loaddefs exif cl-extra
use-package-ensure use-package-core display-line-numbers repeat
icomplete help-mode info package browse-url url url-proxy url-privacy
url-expand url-methods url-history url-cookie url-domsuf url-util
mailcap url-handlers url-parse auth-source cl-seq eieio eieio-core
cl-macs eieio-loaddefs password-cache json map url-vars seq gv subr-x
byte-opt bytecomp byte-compile cconv cl-loaddefs cl-lib iso-transl
tooltip eldoc paren electric uniquify ediff-hook vc-hooks
lisp-float-type elisp-mode mwheel term/x-win x-win term/common-win x-dnd
tool-bar dnd fontset image regexp-opt fringe tabulated-list replace
newcomment text-mode lisp-mode prog-mode register page tab-bar menu-bar
rfn-eshadow isearch easymenu timer select scroll-bar mouse jit-lock
font-lock syntax font-core term/tty-colors frame minibuffer cl-generic
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite emoji-zwj charscript
charprop case-table epa-hook jka-cmpr-hook help simple abbrev obarray
cl-preloaded nadvice button loaddefs faces cus-face macroexp files
window text-properties overlay sha1 md5 base64 format env code-pages
mule custom widget keymap hashtable-print-readable backquote threads
inotify lcms2 dynamic-setting font-render-setting cairo x-toolkit x
multi-tty make-network-process emacs)

Memory information:
((conses 16 232851 18573)
 (symbols 48 24578 2)
 (strings 32 84285 4561)
 (string-bytes 1 2819364)
 (vectors 16 43845)
 (vector-slots 8 496582 38838)
 (floats 8 305 469)
 (intervals 56 364 0)
 (buffers 992 11))

bug#52055: 29.0.50; emoji-insert garbles terminal display on tsdh-dark theme

[Eli Zaretskii]

> From: bird <birdsite@airmail.cc>
> Date: Tue, 23 Nov 2021 10:29:14 +0000
> Cc: bird <birdsite@airmail.cc>
> 
> The display of emacs becomes weird when using emoji-insert on a terminal
> emulator.
> 
> To reproduce:
> emacs -Q -nw
> M-x customize-themes
> select tsdh-dark and return to *scratch* buffer
> M-x emoji-insert (this should already show some problems)
> select smileys (s) (it's garbled even more)
> 
> I can reproduce this with st terminal with lucid toolkit.

I suspect that the terminal emulator does things that Emacs doesn't
expect.  There's an entry in etc/PROBLEMS about problematic behavior
of another emulator (search for "Kitty"): perhaps yours does something
similar?  First thing to try is to disable auto-composition-mode.

If nothing helps, please post screenshots showing the problematic
display.

bug#52055: 29.0.50; emoji-insert garbles terminal display on tsdh-dark theme

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> I suspect that the terminal emulator does things that Emacs doesn't
> expect.  There's an entry in etc/PROBLEMS about problematic behavior
> of another emulator (search for "Kitty"): perhaps yours does something
> similar?  First thing to try is to disable auto-composition-mode.
>
> If nothing helps, please post screenshots showing the problematic
> display.

I'm wondering whether we should just disable the `C-x 8 e e' command on
non-GUI -- it relies on presenting things in a tabular fashion, and we
just don't have a way to interrogate the terminals about whether they
support grapheme clusters like 🧏🏾 or not.

The other commands (like `C-x 8 e s') would work.

Or perhaps we should have a user option that disables all the emoji
commands on non-graphical displays, and then people that know that their
terminals support it can switch it on.  Because otherwise I think we're
going to be getting a lot of these bug reports about this.

Or...  on non-graphical displays, the commands could say "These commands
are likely to produce odd display results in most terminals; continue?"

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#52055: 29.0.50; emoji-insert garbles terminal display on tsdh-dark theme

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: bird <birdsite@airmail.cc>,  52055@debbugs.gnu.org
> Date: Tue, 23 Nov 2021 14:43:20 +0100
> 
> I'm wondering whether we should just disable the `C-x 8 e e' command on
> non-GUI -- it relies on presenting things in a tabular fashion, and we
> just don't have a way to interrogate the terminals about whether they
> support grapheme clusters like 🧏🏾 or not.

Would it work to display a single list on TTY frames, not two-column
list?

As for disabling: I'm not sure yet we should disable right away,
because it isn't clear to me how many terminals have such problems and
which problems can be solved.  We do know the type of the terminal, so
if there are simple enough solutions that produce reasonable display,
I'd hate denying those users the feature or parts thereof.

So I think we should collect data and possible solutions before we
decide.  (On the Linux console auto-composition-mode is already
disabled by default.)

> Or perhaps we should have a user option that disables all the emoji
> commands on non-graphical displays, and then people that know that their
> terminals support it can switch it on.  Because otherwise I think we're
> going to be getting a lot of these bug reports about this.
> 
> Or...  on non-graphical displays, the commands could say "These commands
> are likely to produce odd display results in most terminals; continue?"

Those are possible solutions, yes.  But I think we don't have enough
data yet to decide what's best.

bug#52055: 29.0.50; emoji-insert garbles terminal display on tsdh-dark theme

[Bird]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> I suspect that the terminal emulator does things that Emacs doesn't
>> expect.  There's an entry in etc/PROBLEMS about problematic behavior
>> of another emulator (search for "Kitty"): perhaps yours does something
>> similar?

forgot to mention in previous mail, but i don't think st does anything
special. it's focus is to do as minimal as possible and let users add
functionality they want.

>>First thing to try is to disable auto-composition-mode.
>>
>> If nothing helps, please post screenshots showing the problematic
>> display.
>
> I'm wondering whether we should just disable the `C-x 8 e e' command on
> non-GUI -- it relies on presenting things in a tabular fashion, and we
> just don't have a way to interrogate the terminals about whether they
> support grapheme clusters like 🧏🏾 or not.
>
> The other commands (like `C-x 8 e s') would work.

Could something "barebones" for terminals like

s Smilieys smiley-emojis \t t travel & places-emojis

work? basically, it'd only be a simple display similar to normal text
but groups are tab separated (maybe with a tab width of 8 in-case
tab-width is something else). 

-- 
Show respect for age.  Drink good Scotch for a change.

bug#52055: 29.0.50; emoji-insert garbles terminal display on tsdh-dark theme

[Eli Zaretskii]

> From: Bird <birdsite@airmail.cc>
> Cc: Eli Zaretskii <eliz@gnu.org>,  bird <birdsite@airmail.cc>,
>   52055@debbugs.gnu.org
> Date: Tue, 23 Nov 2021 17:22:07 +0000
> 
> forgot to mention in previous mail, but i don't think st does anything
> special.

Of course, it does: it displays Emoji with color fonts, and interprets
Emoji sequences (like Emacs does on GUI frames).  The problem is that
it doesn't pad the Emoji to 2-column width, but instead uses the width
of the font glyphs as-is.

bug#52055: 29.0.50; emoji-insert garbles terminal display on tsdh-dark theme

[Lars Ingebrigtsen]

Bird <birdsite@airmail.cc> writes:

> Could something "barebones" for terminals like
>
> s Smilieys smiley-emojis \t t travel & places-emojis
>
> work? basically, it'd only be a simple display similar to normal text
> but groups are tab separated (maybe with a tab width of 8 in-case
> tab-width is something else). 

I don't think that'd be practical.  But `C-x 8 e l' should work, I
think?

I'm unable to test this myself, because apparently st doesn't work at
all on Debian/bookworm:

larsi@xo:~/src/emacs/trunk$ st
property '142' doesn't exist, you need to specify its type and format


-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#52055: 29.0.50; emoji-insert garbles terminal display on tsdh-dark theme

[Bird]

[-- Attachment #1: Type: text/plain, Size: 485 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> I can reproduce this with st terminal with lucid toolkit.
>
> I suspect that the terminal emulator does things that Emacs doesn't
> expect.

> First thing to try is to disable auto-composition-mode.

>
> If nothing helps, please post screenshots showing the problematic
> display.
I disabled auto-composition-mode (with M-x auto-composition-mode) in
emacs -Q -nw and then selected the theme.
This is the display after emoji-insert (C-x 8 e i)


[-- Attachment #2: Type: image/png, Size: 67845 bytes --]

[-- Attachment #3: Type: text/plain, Size: 32 bytes --]


After pressing s for smileys,


[-- Attachment #4: Type: image/png, Size: 70493 bytes --]

[-- Attachment #5: Type: text/plain, Size: 34 bytes --]


and after pressing C-g on above


[-- Attachment #6: Type: image/png, Size: 74150 bytes --]

[-- Attachment #7: Type: text/plain, Size: 15 bytes --]


one more C-g


[-- Attachment #8: Type: image/png, Size: 9413 bytes --]

[-- Attachment #9: Type: text/plain, Size: 387 bytes --]



St does double buffering i heard (idk what that means tho 😜) but that
shouldn't be problematic as other terminals do that too.

(unrelated: does copying to xorg's clipboard with emacsclient -nw on a terminal
emulater work? don't think it works for me now so just wanted to ask
to confirm) 
-- 
The part of the world that people find most puzzling is the part called "Me".

bug#52055: 29.0.50; emoji-insert garbles terminal display on tsdh-dark theme

[Eli Zaretskii]

> From: Bird <birdsite@airmail.cc>
> Cc: bird <birdsite@airmail.cc>,  52055@debbugs.gnu.org
> Date: Tue, 23 Nov 2021 16:59:54 +0000
> 
> > If nothing helps, please post screenshots showing the problematic
> > display.
> I disabled auto-composition-mode (with M-x auto-composition-mode) in
> emacs -Q -nw and then selected the theme.
> This is the display after emoji-insert (C-x 8 e i)

Is this better or worse than with auto-composition-mode enabled?

> St does double buffering i heard (idk what that means tho 😜) but that
> shouldn't be problematic as other terminals do that too.

Double buffering has nothing to do with it.  The problem is that Emacs
expects each Emoji character to take 2 columns, but the terminal
emulator doesn't necessarily behave like that, as can be clearly seen
from the images.  And that is the problem.  (With
auto-composition-mode Emacs expects each Emoji to take just 1 column,
so it's probably even worse.)

> (unrelated: does copying to xorg's clipboard with emacsclient -nw on a terminal
> emulater work? don't think it works for me now so just wanted to ask
> to confirm) 

How do you copy to the clipboard from a -nw session?

bug#52055: 29.0.50; emoji-insert garbles terminal display on tsdh-dark theme

[Bird]

Eli Zaretskii <eliz@gnu.org> writes:

>> > If nothing helps, please post screenshots showing the problematic
>> > display.
>> I disabled auto-composition-mode (with M-x auto-composition-mode) in
>> emacs -Q -nw and then selected the theme.
>> This is the display after emoji-insert (C-x 8 e i)
>
> Is this better or worse than with auto-composition-mode enabled?

It's nearly the same, i think.

> The problem is that Emacs expects each Emoji character to take 2
> columns, but the terminal emulator doesn't necessarily behave like
> that, as can be clearly seen from the images.  And that is the
> problem.  (With auto-composition-mode Emacs expects each Emoji to take
> just 1 column, so it's probably even worse.)

Ohh, could it be related to libXft? i mean, that's what terminals use
and libXft dosen't handle emoji at all.

A lot of terminal emulators workaround this problem by handling emojis
manually. st dosen't do that and hence i use a patched libXft that
handles emojis too.
The patch is at:
https://gitlab.freedesktop.org/xorg/lib/libxft/merge_requests/1.patch

If you try to display emojis on st without patched libXft, it just
crashes.

>
>> (unrelated: does copying to xorg's clipboard with emacsclient -nw on a terminal
>> emulater work? don't think it works for me now so just wanted to ask
>> to confirm) 
>
> How do you copy to the clipboard from a -nw session?

By normal copy commands (C-w and M-w). i think it worked before. it even works on
android (by launching emacs on termux). but now it can't access
clipboard or set it. I'm using an emacs-daemon if that matters.

i'll download 27.2 to see if it worked.

-- 
The hearing ear is always found close to the speaking tongue, a custom
whereof the memory of man runneth not howsomever to the contrary, nohow.

bug#52055: 29.0.50; emoji-insert garbles terminal display on tsdh-dark theme

[Eli Zaretskii]

> From: Bird <birdsite@airmail.cc>
> Cc: Bird <birdsite@airmail.cc>,  52055@debbugs.gnu.org
> Date: Tue, 23 Nov 2021 17:34:48 +0000
> 
> >> (unrelated: does copying to xorg's clipboard with emacsclient -nw on a terminal
> >> emulater work? don't think it works for me now so just wanted to ask
> >> to confirm) 
> >
> > How do you copy to the clipboard from a -nw session?
> 
> By normal copy commands (C-w and M-w). i think it worked before. it even works on
> android (by launching emacs on termux). but now it can't access
> clipboard or set it. I'm using an emacs-daemon if that matters.

I think xterm supports that, but st doesn't, at least by default.  See
the variables xterm-st-extra-capabilities and xterm-extra-capabilities.