bug#54032: 29.0.50; Emoji display on Linux console switched to hexadecimal output

[Eli Zaretskii <eliz@gnu.org>]

> From: Aura Kelloniemi <kaura.dev@sange.fi>
> Date: Thu, 17 Feb 2022 08:57:40 +0200
> 
> on recent Emacs development repository builds, emoji characters are no more
> displayed on Linux console. Instead Emacs prints \UABCDEF hexadecimal codes.
> 
> This is due to the commit 10c680551e899805a6de7360e9b65986fd87df72 which
> probably makes things better on some terminals. Reverting this commit fixed
> the issue for me, and emojis are again displayed as usual.

Thanks, but I think we need more detailed information to understand
the problem.  First, when you say "display emoji characters", which
characters exactly does that allude to?  Can you show specific
examples of text that includes Emoji, which displayed the Emoji glyphs
before the above commit, but not after it?  In particular, are we
talking about single codepoints in the Emoji block, or are we talking
about Emoji sequences that involve more than one codepoint (and are
supposed to display like a single Emoji glyph)?  For each example,
please show both the text and what you see on display for that text.

Also, what is the value of auto-composition-mode?  I think it should
be the string "linux", in which case please try setting it to t and
see if the display becomes better or worse.

> Linux console is (sort of) capable of displaying emojis. Console font can be
> configured so that it has glyphs for emojis exactly the same way as for other
> characters.

If that is the case, why did Emacs think the terminal cannot display
these characters?  Can you step with GDB inside terminal_glyph_code,
when it is called for the first time in the Emacs session, and see
whether the ioctl call we issue in calculate_glyph_code_table returns
valid values for the Emoji codepoints?

> Also, blind users using refreshable braile displays use Linux
> console to access Emacs. The braille terminal driver is able to detect the
> correct character code points even when Linux itself is not able to display
> them properly on the screen. This detection is done using the /dev/vcsu
> (virtual console screen unicode) character devices.

Are you saying that the braille terminal driver will not respond to
the ioctl call we issue in calculate_glyph_code_table?  Is there any
other method of knowing which characters are supported in that case?

> For these reasons it is important that emacs outputs the real
> characters to the terminal on Linux console.

Outputting codepoints for which there are no glyphs produced
unreadable display, since (AFAIU) the console displays them all as the
"diamond" replacement character.  Detecting the fact that a codepoint
cannot be displayed allows us to produce something that at least can
be interpreted, and allows the user to install optional features (such
as those provided by latin1-disp.el) which will replace the characters
that cannot be displayed by equivalent strings, for example ASCII
strings.

So we would like to keep the automatic detection of whether a given
character can be displayed by the console, although it sounds like the
current solution should be made more flexible and sophisticated in
some way.

> Linux console has a terminal type string of "linux". lisp/term/linux.el
> contains already some Linux terminal specific code (which unfortunately
> assumes though that Linux has a default character set of Latin-1, which has
> never been true).

That's just the default.  We attempt to detect which characters can be
displayed later on, when the functions I mentioned above are called
during startup.

> My preferred solution to this problem would be to add and document a way to
> configure character display logic on TTYs more precisely. It would be great to
> be able to control the terminal output of Unicode on grapheme cluster
> precision – i.e. allow the user to define a function which translates code
> points/grapheme clusters into something that their terminal can display.

I'm not yet sure something like that would be needed.  It will
certainly slow down the display on the console, which is undesirable
for obvious reasons.  It is also too complex (not every Emacs user can
write Lisp programs that play sophisticated games with characters and
glyphs).

I think we don't yet have a detailed enough understanding of the issue
to discuss solutions, so I suggest to postpone this discussion until
the questions I asked above are answered, and we have a good
understanding of what is going on.

The commit to which you point out was made based on reports from
another user of the Linux console (albeit not about Emoji), and in
that case the change had a positive effect.  So the issue is not
simple, and we need a good understanding of it before we devise a
solution.

Thanks.