Display of undisplayable characters: \U01F3A8 instead of diamond

Display of undisplayable characters: \U01F3A8 instead of diamond

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

The text-based Linux console is unable to display many Unicode
characters.  In Emacs 29, display of undisplayable characters has
changed.  It used to show them with a diamond.  Now it shows the
unicode character code as hex, preceded by \U.

I find that change quite inconvenient.  It makes the text harder to
read.  Showing the codes does no good, since I don't know these codes,
not even for characters I am familiar with.  To find out what
character a code represents, I have to use C-u C-x =, just as I did
before.

I last built the sources in May.  Has this changed in a significant
way since then?

I could not find, in NEWS, anything about this change -- it ought to
be in NEWS .  Nor a way to go back to the old
behavior.  Is there a way?  If not, would people please create a way?

As a separate question, do users generally like this change?
Would it be better to return to the old diamond method as the default?


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Richard Stallman wrote:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]

https://dataswamp.org/~incal/terror-3/img/terror.jpg

> The text-based Linux console is unable to display many
> Unicode characters. In Emacs 29, display of undisplayable
> characters has changed. It used to show them with a diamond.
> Now it shows the unicode character code as hex, preceded by
> \U.
>
> I find that change quite inconvenient. It makes the text
> harder to read.

100% Annoying, serves no purpose what I can see and messes
up alignment/indentation, even.

> Showing the codes does no good, since I don't know these
> codes, not even for characters I am familiar with. To find
> out what character a code represents, I have to use C-u C-x
> =, just as I did before.

I have some code, here

  https://dataswamp.org/~incal/emacs-init/char.el

which doesn't bring up the *Help* buffer but echoes the char's
names (new and old, often, e.g. / is

  solidus (old: slash) (at point: 1308)

so yes, we're all old-school I think ...

Anyway ... one could bind that to just a single key,
optimally, or even do with the idle timer (it could be
instructed so if the key is showable in the Linux VT it
wouldn't echo anything).

Also, those chars shouldn't be everywhere anyway, right?

But I agree diamonds were much neater and low-key ...

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Date: Thu, 25 Aug 2022 23:33:57 -0400
> 
> The text-based Linux console is unable to display many Unicode
> characters.  In Emacs 29, display of undisplayable characters has
> changed.  It used to show them with a diamond.  Now it shows the
> unicode character code as hex, preceded by \U.
> 
> I find that change quite inconvenient.  It makes the text harder to
> read.  Showing the codes does no good, since I don't know these codes,
> not even for characters I am familiar with.  To find out what
> character a code represents, I have to use C-u C-x =, just as I did
> before.

This is a standard Emacs way of displaying unsupported characters on
TTY frames.  I fail to see how this is worse than displaying the same
diamond glyph for any unsupported character, because with the
character codes you at least know when the characters are different,
even without "C-u C-x =".

You can have some (many) of the characters show in a more readable
form by invoking "M-x latin1-display-ucs-per-lynx RET".  The
particular character you mentioned is not affected by that, though.

Is this specifically about that U+1F3A8 character?  Or is this more
general?

> I last built the sources in May.  Has this changed in a significant
> way since then?

I don't think so, but I don't have access to the Linux console to
check.

> I could not find, in NEWS, anything about this change -- it ought to
> be in NEWS .

From my POV, it's a bugfix, and we don't call out those in NEWS.  What
you saw before was a bug in the way Emacs detected which characters
the Linux console is capable of displaying.

> Nor a way to go back to the old
> behavior.  Is there a way?  If not, would people please create a way?

What would you want Emacs to do instead?  Writing a code of an
unsupported character to a text-mode terminal is out of the question:
it can easily mess up the entire display.  What else is possible that
would satisfy you?

> As a separate question, do users generally like this change?
> Would it be better to return to the old diamond method as the default?

I don't use the Linux console, so I cannot answer that.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Eli Zaretskii wrote:

>> Nor a way to go back to the old behavior. Is there a way?
>> If not, would people please create a way?
>
> What would you want Emacs to do instead? Writing a code of
> an unsupported character to a text-mode terminal is out of
> the question: it can easily mess up the entire display.
> What else is possible that would satisfy you?

The old way was better as it was less disruptive in size and
color and didn't mess up alignment/indentation.

As for what char it was, it looked like these two - and they
look the same in a Linux VT -

\u2666 ♦	BLACK DIAMOND SUIT
\u25c8 ◈	WHITE DIAMOND CONTAINING BLACK SMALL DIAMOND

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

> As for what char it was, it looked like these two - and they
> look the same in a Linux VT -
>
> \u2666 ♦ BLACK DIAMOND SUIT
> \u25c8 ◈ WHITE DIAMOND CONTAINING BLACK SMALL DIAMOND

But it wasn't that but probably

  \ufffd � REPLACEMENT CHARACTER

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Yuri Khan]

On Fri, 26 Aug 2022 at 15:24, Emanuel Berg <incal@dataswamp.org> wrote:


> As for what char it was, it looked like these two - and they
> look the same in a Linux VT -
>
> \u2666 ♦        BLACK DIAMOND SUIT
> \u25c8 ◈        WHITE DIAMOND CONTAINING BLACK SMALL DIAMOND

Linux console works over (an emulation of?) the VGA text mode, which
descends from IBM CGA text mode, whose set of displayable characters
was defined by code page 437. Its codes 0x03 through 0x06 encoded
playing card suits — hearts, diamonds, clubs, spades, in this order.
In Unicode, these became U+2665, U+2666, U+2663, U+2660, respectively.

I do not know the history of replacing an undisplayable character with
a diamond. The U+FFFD Replacement Character glyph looks similar, but
the cause/consequence link might be in either direction.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Yuri Khan wrote:

> I do not know the history of replacing an undisplayable
> character with a diamond. The U+FFFD Replacement Character
> glyph looks similar, but the cause/consequence link might be
> in either direction.

Can't we check this with the Emacs source if that's where the
change happened?

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Yuri Khan]

On Fri, 26 Aug 2022 at 16:14, Emanuel Berg <incal@dataswamp.org> wrote:

> > I do not know the history of replacing an undisplayable
> > character with a diamond. The U+FFFD Replacement Character
> > glyph looks similar, but the cause/consequence link might be
> > in either direction.
>
> Can't we check this with the Emacs source if that's where the
> change happened?

No, because the replacement with diamond happens in the Linux console
no matter if Emacs or something else is running.

    bash$ echo $'\U0001F4A9'
    ♦

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Yuri Khan wrote:

>>> I do not know the history of replacing an undisplayable
>>> character with a diamond. The U+FFFD Replacement Character
>>> glyph looks similar, but the cause/consequence link might
>>> be in either direction.
>>
>> Can't we check this with the Emacs source if that's where
>> the change happened?
>
> No, because the replacement with diamond happens in the
> Linux console no matter if Emacs or something else
> is running.
>
>     bash$ echo $'\U0001F4A9'
>     ♦

???

(insert-char #x1f3a8) 🎨
                      ^^^^^^^^
                      not a diamond anymore!
                      (i.e., the fffd: �)
                      but instead 8 chars, \U01F3A8

Wait, I'll post a screenshot ...

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

> Wait, I'll post a screenshot ...

Observe:

  https://dataswamp.org/~incal/char/emacs-char.png

  https://dataswamp.org/~incal/char/tty-char.png

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Yuri Khan]

On Fri, 26 Aug 2022 at 16:49, Emanuel Berg <incal@dataswamp.org> wrote:

> > Wait, I'll post a screenshot ...
>
> Observe:
>
>   https://dataswamp.org/~incal/char/emacs-char.png
>
>   https://dataswamp.org/~incal/char/tty-char.png

That’s Emacs no longer passing the undisplayable Unicode character to
the TTY, but actual
backslash-uniform-zero-one-foxtrot-three-alpha-eight sequence. As
stated elsewhere upthread, it was considered a bugfix in undisplayable
character detection.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Yuri Khan wrote:

>>> Wait, I'll post a screenshot ...
>>
>> Observe:
>>
>>   https://dataswamp.org/~incal/char/emacs-char.png
>>   https://dataswamp.org/~incal/char/tty-char.png
>
> That’s Emacs no longer passing the undisplayable Unicode
> character to the TTY, but actual
> backslash-uniform-zero-one-foxtrot-three-alpha-eight
> sequence. As stated elsewhere upthread, it was considered
> a bugfix in undisplayable character detection.

And that's the bugfix we want to fix!

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

[-- Attachment #1: Type: text/plain, Size: 591 bytes --]

>>>> I do not know the history of replacing an undisplayable
>>>> character with a diamond. The U+FFFD Replacement
>>>> Character glyph looks similar, but the cause/consequence
>>>> link might be in either direction.
>>>
>>> Can't we check this with the Emacs source if that's where
>>> the change happened?
>>
>> No, because the replacement with diamond happens in the
>> Linux console no matter if Emacs or something else
>> is running.
>>
>>     bash$ echo $'\U0001F4A9'
>>     ♦
>
> ???

https://dataswamp.org/~incal/char/emacs-char.png
https://dataswamp.org/~incal/char/tty-char.png


[-- Attachment #2: emacs-char.png --]
[-- Type: image/png, Size: 799 bytes --]

[-- Attachment #3: tty-char.png --]
[-- Type: image/png, Size: 651 bytes --]

[-- Attachment #4: Type: text/plain, Size: 61 bytes --]


-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Is this specifically about that U+1F3A8 character?  Or is this more
  > general?

This applies to all characters that display that way.  I don't know
what U+1F3A8 stands for; to me, it is simply "some undisplayable character."

  > This is a standard Emacs way of displaying unsupported characters on
  > TTY frames.  I fail to see how this is worse than displaying the same
  > diamond glyph for any unsupported character, because with the
  > character codes you at least know when the characters are different,
  > even without "C-u C-x =".

This change makes it harder to see and recognize the displayable
characters in the line.

When a line contains many nondisplayable characters, this change makes it
much longer, so the line gets continued.

In practice, I can't recognize when the same \U sequence appears more
than once on the screen.  I can't see what the hex digits are just
with a glance; only if I concentrate and read one.  If I do that, I
can't recognize other instances of the same sequence without
concentrating on each of them to read it.

  >   What
  > you saw before was a bug in the way Emacs detected which characters
  > the Linux console is capable of displaying.

I am surprised.  I thought of the diamond display as a feature; ISTR
we discussed it as such.  But maybe I'm mistaken.

Either way, I'd much rather have the diamonds.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Sun, 28 Aug 2022 00:04:51 -0400
> 
>   > Is this specifically about that U+1F3A8 character?  Or is this more
>   > general?
> 
> This applies to all characters that display that way.  I don't know
> what U+1F3A8 stands for; to me, it is simply "some undisplayable character."

In that case, you can significantly improve the situation by typing

   M-x latin1-display-ucs-per-lynx RET

or by using "M-x customize-variable" to customize the variable
latin1-display-ucs-per-lynx to a non-nil value.  Then many non-ASCII
characters will be displayed as ASCII emulations.  Some characters
will inevitably be still not displayable; for those you can define
their ASCII representation like this:

  (set-char-table-extra-slot glyphless-char-display 0 "*")

Then the characters that cannot be displayed will be represented by
the string that is the last argument to set-char-table-extra-slot,
with a special face designed to make those stand out.  The string must
be ASCII, though, because nothing else is safe to serve as fallback.

(With the latest master branch, if the string is a single character,
it will not be enclosed in [..], so you can effectively use that
single character as a replacement character for those your terminal
cannot display.  It still has to be an ASCII character.)

>   > This is a standard Emacs way of displaying unsupported characters on
>   > TTY frames.  I fail to see how this is worse than displaying the same
>   > diamond glyph for any unsupported character, because with the
>   > character codes you at least know when the characters are different,
>   > even without "C-u C-x =".
> 
> This change makes it harder to see and recognize the displayable
> characters in the line.

I don't think I understand how this is possible.  The \Unnnn thing has
a distinct face.  Maybe you need to customize that face to make it
stand out on your terminal?

> When a line contains many nondisplayable characters, this change makes it
> much longer, so the line gets continued.

Only if the line is already long, which is not necessarily true.

Let's not exaggerate problems too much, okay?

> In practice, I can't recognize when the same \U sequence appears more
> than once on the screen.  I can't see what the hex digits are just
> with a glance; only if I concentrate and read one.  If I do that, I
> can't recognize other instances of the same sequence without
> concentrating on each of them to read it.

This still sounds like an exaggeration to me.  If you care about those
characters, you should not have any difficulties realizing which are
identical and which aren't.  For several instances of the same
character, you need to type "C-u C-x =" just once.  So this is an
improvement vs the identical display of all the undisplayable
characters, where you must type "C-u C-x =" for each one of them.

If you don't care what those characters are, then you just don't need
to waste any time thinking what they could be and which ones are
different.  So here, there's no disadvantage to the current display.

>   >   What
>   > you saw before was a bug in the way Emacs detected which characters
>   > the Linux console is capable of displaying.
> 
> I am surprised.  I thought of the diamond display as a feature; ISTR
> we discussed it as such.  But maybe I'm mistaken.

You are mistaken.  It was never a feature.

> Either way, I'd much rather have the diamonds.

You can have any ASCII character (see above) or a string of characters
instead.  But not the diamond, which is a non-ASCII character.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Eli Zaretskii wrote:

> In that case, you can significantly improve the situation by typing
>
>    M-x latin1-display-ucs-per-lynx RET

Not a command or even function but a variable?

> or by using "M-x customize-variable" to customize the variable
> latin1-display-ucs-per-lynx to a non-nil value.

Yeah, it says in the help you can't set just it. Not good ...

  Setting this variable directly does not take effect; use
  either M-x customize or the function ‘latin1-display’.

Don't understand how `latin1-display' is supposed to be
used ...

> (set-char-table-extra-slot glyphless-char-display 0 "*")

But that will be [*] - three chars.

This

  (set-char-table-extra-slot glyphless-char-display 0 'thin-space)

is better then since it won't prolong the line where it
happens ...

> (With the latest master branch, if the string is a single character,
> it will not be enclosed in [..]

So it can, great ... I'll get it now!

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

>> In that case, you can significantly improve the situation by typing
>>
>>    M-x latin1-display-ucs-per-lynx RET
>
> Not a command or even function but a variable?

No, you have to upgrade to get that as well!

>> (With the latest master branch, if the string is a single
>> character, it will not be enclosed in [..]
>
> So it can, great ... I'll get it now!

Indeed, this

  (set-char-table-extra-slot glyphless-char-display 0 "%")

don't get the [ and ] now and the face has also changed, it is
just yellow now (ANSI 3), not bright yellow ...

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

>>> M-x latin1-display-ucs-per-lynx RET
>>
>> Not a command or even function but a variable?
>
> No, you have to upgrade to get that as well!

Okay, now I have it in

  GNU Emacs 29.0.50 (build 1, x86_64-pc-linux-gnu, cairo version 1.16.0)
  of 2022-08-28 [commit df2f6fb7fc4b79834ae40db8be2ccdc1e4a273f1]

So like this?

(latin1-display-ucs-per-lynx 1)
(set-char-table-extra-slot glyphless-char-display 0 "%")
;; test: (insert-char #x1f3ab)

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Sun, 28 Aug 2022 09:04:09 +0200
> 
> >> In that case, you can significantly improve the situation by typing
> >>
> >>    M-x latin1-display-ucs-per-lynx RET
> >
> > Not a command or even function but a variable?
> 
> No, you have to upgrade to get that as well!

It was always a command, it just wasn't autoloaded.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Eli Zaretskii wrote:

>>>> In that case, you can significantly improve the situation by typing
>>>>
>>>>   M-x latin1-display-ucs-per-lynx RET
>>>
>>> Not a command or even function but a variable?
>> 
>> No, you have to upgrade to get that as well!
>
> It was always a command, it just wasn't autoloaded.

Makes sense to do if you are supposed to set the variable that
way ...

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Kévin Le Gouguec]

Richard Stallman <rms@gnu.org> writes:

> The text-based Linux console is unable to display many Unicode
> characters.  In Emacs 29, display of undisplayable characters has
> changed.  It used to show them with a diamond.  Now it shows the
> unicode character code as hex, preceded by \U.
>
> I find that change quite inconvenient.  It makes the text harder to
> read.  Showing the codes does no good, since I don't know these codes,
> not even for characters I am familiar with.  To find out what
> character a code represents, I have to use C-u C-x =, just as I did
> before.
>
> I last built the sources in May.  Has this changed in a significant
> way since then?
>
> I could not find, in NEWS, anything about this change -- it ought to
> be in NEWS .  Nor a way to go back to the old
> behavior.  Is there a way?  If not, would people please create a way?
>
> As a separate question, do users generally like this change?
> Would it be better to return to the old diamond method as the default?

That brings to mind a discussion in January, where (IIRC; apologies for
any inaccuracies) you reported that Emacs degraded to displaying
diamonds when it had no glyphs for precomposed Unicode codepoints:

<E1nCZ9R-0005I0-5n@fencepost.gnu.org>

While one subthread went into displaying those codepoints with sequences
of ASCII characters, another subthread dealt with
glyphless-char-display-control; the default value for characters which
fall in the 'no-font group were supposed to be handled with the hex-code
method, but weren't.  Eli fixed that inconsistency:

<837dalfp4t.fsf@gnu.org>

IIUC this was considered a bugfix, which might explain why NEWS does not
mention it?

I can't find a way to tweak glyphless-char-display-control to let TTY
frames show diamonds again…  maybe a new value display method would be
needed for that purpose?

The 'acronym method could display something useful IIUC, though I don't
know what would be the smartest way to fill in char-acronym-table.

(Hope I'm not confused and that thread was indeed the source of the
change you mention; apologies for the noise if not)

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> From: Kévin Le Gouguec <kevin.legouguec@gmail.com>
> Date: Fri, 26 Aug 2022 08:52:05 +0200
> 
> I can't find a way to tweak glyphless-char-display-control to let TTY
> frames show diamonds again…  maybe a new value display method would be
> needed for that purpose?

The diamond is a non-ASCII character, so to display it by default,
we'd need to make sure the terminal is capable of displaying it.

Also, do you really mean diamond, or do you mean U+FFFD REPLACEMENT
CHARACTER (which on GUI display is shown as a diamond with a question
mark inside)?

And why the diamond, of all the possible characters? why not simply
'?', for example?

(FWIW, providing such an option makes very little sense to me, for the
reasons I explained in my previous message.  But if people insist, and
if we can find a good and safe replacement character to show instead
of the hex code, I won't object adding such an option.)

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Eli Zaretskii wrote:

> The diamond is a non-ASCII character, so to display it by default,
> we'd need to make sure the terminal is capable of displaying it.

You can print it in a Linux VT with

  printf -v tmp %04x 128
  printf "\u${tmp}\n"

> Also, do you really mean diamond, or do you mean U+FFFD
> REPLACEMENT CHARACTER

  M-x insert-char RET fffd RET

Indeed, looks like a diamond to me ...

> And why the diamond, of all the possible characters? why not
> simply '?', for example?

Because then we would have to change the QUESTION MARK

  M-x insert-char RET 3f RET

for the diamond instead ...

> (FWIW, providing such an option makes very little sense to
> me, for the reasons I explained in my previous message.
> But if people insist, and if we can find a good and safe
> replacement character to show instead of the hex code,
> I won't object adding such an option.)

Why isn't U+FFFD (�) safe?

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Eli Zaretskii wrote:

> The diamond is a non-ASCII character, so to display it by
> default, we'd need to make sure the terminal is capable of
> displaying it [...] Also, do you really mean diamond, or do
> you mean U+FFFD REPLACEMENT CHARACTER

I think "diamond" simply was a description how the char
looked. But U+FFFD also looks like a diamond so if you think
that's what it is, that sounds reasonable given its
name, right?

In a Linux VT running an Emacs instance, insert this char with

  (insert-char ?\ufffd)

In a Linux VT just running zsh (also tested with bash; works),
do

  $ printf -v rc %04x 0xfffd
  $ printf "\u${rc}\n"

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

This seems to be the best attempt so far to print all chars.

Isn't the Linux VT supposed to be "512K"? But not all chars
show up ... maybe there are yet others to be found?

They show up in xterm tho, not the non-printables obviously,
but a lot more than in the console.

#! /bin/zsh

all-chars () {
    local i
    local chr
    for (( i=0; i < 512; i++)); do
        printf -v chr %04x $i
        printf "\u${chr}\n" 2> /dev/null
    done
}

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Andreas Schwab]

On Aug 26 2022, Emanuel Berg wrote:

> In a Linux VT just running zsh (also tested with bash; works),
> do
>
>   $ printf -v rc %04x 0xfffd
>   $ printf "\u${rc}\n"

aka $ printf '\ufffd\n'

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Andreas Schwab wrote:

>> In a Linux VT just running zsh (also tested with bash;
>> works), do
>>
>>   $ printf -v rc %04x 0xfffd
>>   $ printf "\u${rc}\n"
>
> aka $ printf '\ufffd\n'

Okay, that's another way, I'm not saying it's wrong ...

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Kévin Le Gouguec]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Kévin Le Gouguec <kevin.legouguec@gmail.com>
>> Date: Fri, 26 Aug 2022 08:52:05 +0200
>> 
>> I can't find a way to tweak glyphless-char-display-control to let TTY
>> frames show diamonds again…  maybe a new value display method would be
>> needed for that purpose?
>
> The diamond is a non-ASCII character, so to display it by default,
> we'd need to make sure the terminal is capable of displaying it.
>
> Also, do you really mean diamond, or do you mean U+FFFD REPLACEMENT
> CHARACTER (which on GUI display is shown as a diamond with a question
> mark inside)?

Based on that past discussion, my assumption was that when Richard said
in his OP:

>              In Emacs 29, display of undisplayable characters has
> changed.  It used to show them with a diamond.

… he was referring to the diamond-like symbol that the Linux console
shows for glyphless characters (so not U+FFFD), and that Emacs (in a
TTY) used to show as well, until your patch which made these characters
fall into the 'no-font group, which means they are now displayed with
the 'hex-code method.

I tried to find out precisely what Unicode codepoint that symbol is
supposed to correspond to, if any, but haven't reached any conclusion.

> And why the diamond, of all the possible characters? why not simply
> '?', for example?

I'll let Richard answer that one, but I suspect any single-column
character would be fine: the problem, as I understand it, is that (1)
they are no more informative than a single placeholder char:

>                                              To find out what
> character a code represents, I have to use C-u C-x =, just as I did
> before.

… and (2) they make reading more difficult:

> I find that change quite inconvenient.  It makes the text harder to
> read.

I'm guessing Richard is faced with mostly-legible ASCII paragraphs with
just a couple of glyphless chars strewn within; then I'd imagine the hex
codes impair the reading experience more than single-column characters
(diamonds, '?', what-have-you).

*shrug* I don't live in a TTY, so I don't have an informed opinion on
what the most useful representation would be.  Off the top of my head,
I see

* a minimalist approach: add a new 'char method that lets you pick your
  favourite placeholder character for the 'no-font group,

* a maximalist approach: embark on a quest to add legible and
  informative display methods for every category that might be
  susceptible to lack glyphs[1].  (And allow to express these categories
  in glyphless-char-display-control… perhaps allowing GROUP to be a
  script symbol, like set-fontset-font's CHARACTERS argument?)

Can't speak for Richard, but I'm getting the sense the minimalist
approach would suit him just fine.


[1] E.g.

  * 'phonetic for international alphabets (sort of the opposite of an
    input method like cyrillic-translit, a table that would map ?я to
    "ja", ?ш to "sh"),
  * 'cldr for emoji (?😀 → "*grinning face*"), or 'smiley (?🙌 → \o/),
  * 'tex for symbols (?↦ → "\mapsto"), …

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Kévin Le Gouguec wrote:

> *shrug* I don't live in a TTY, so I don't have an informed
> opinion on what the most useful representation would be.
> Off the top of my head, I see

Here in tty land sayings are you are correct in everything you
say, you could have added that the font color - not an Emacs
face it seems - it is the ANSI Bright Yellow color
(\u001b[33;1m) - this makes the char code stick out so they
are what you notice first and most while it should be the
other way around.

> a minimalist approach: add a new 'char method that lets
> you pick your favourite placeholder character for the
> 'no-font group,

+1 (minimalist?)

When you stay with point on the char in the echo area it would
say, for 😀 for example, indeed, "grinning face".

> * a maximalist approach: embark on a quest to add legible
>   and informative display methods [...]

(setq expand-glyphless-method nil)

Show 😀 as 😀. (default; same as now)

(setq expand-glyphless-method #xfffd)

Show 😀 as �.

(setq expand-glyphless-method 'codepoint)

Show 😀 as \U01F600.

(setq expand-glyphless-method 'human)

Show 😀 as [grinning-face].

[smiling-face-saying-goodbye]

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]


  > The diamond is a non-ASCII character, so to display it by default,
  > we'd need to make sure the terminal is capable of displaying it.

  > Also, do you really mean diamond, or do you mean U+FFFD REPLACEMENT
  > CHARACTER (which on GUI display is shown as a diamond with a question
  > mark inside)?

I don't know the difference.
I'm talking about what appears on my screen.

These two lines

> \u2666 ♦        BLACK DIAMOND SUIT
> \u25c8 ◈        WHITE DIAMOND CONTAINING BLACK SMALL DIAMOND

display on my Linux console with a diamond.  The same diamond glyph
that user to represent undisplayable characters.

I like this diamond better than ?.  Maybe that is because ? really
appears in text.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Richard Stallman wrote:

> These two lines
>
>> \u2666 ♦        BLACK DIAMOND SUIT
>> \u25c8 ◈        WHITE DIAMOND CONTAINING BLACK SMALL DIAMOND
>
> display on my Linux console with a diamond. The same diamond
> glyph that used to represent undisplayable characters.

Indeed ...

> I like this diamond better than ?. Maybe that is because ?
> really appears in text.

Ha ha :) Obviously we can't have that ...

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Gregory Heytings]

>
> The text-based Linux console is unable to display many Unicode 
> characters.
>

Did you try to use fbterm?  Most Unicode should be displayed correctly.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Gregory Heytings wrote:

>> The text-based Linux console is unable to display many
>> Unicode characters.
>
> Did you try to use fbterm? Most Unicode should be
> displayed correctly.

For the record not this one, that is \U01F3A8.

But even so, then one has to redo everything, even the cursor
underscore starts blinking again ...

That could be interesting of course but why can't we just have
an option with Emacs of one prefers the old
"all-diamond" bahvior?

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Gregory Heytings]

>>> The text-based Linux console is unable to display many Unicode 
>>> characters.
>>
>> Did you try to use fbterm? Most Unicode should be displayed correctly.
>
> For the record not this one, that is \U01F3A8.
>

I said "most".  Indeed color emojis are not displayed correctly, but they 
aren't in most terminal emulators either.

>
> But even so, then one has to redo everything, even the cursor underscore 
> starts blinking again ...
>

Set cursor-interval=0 in your ~/.fbtermrc.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Gregory Heytings wrote:

>> But even so, then one has to redo everything, even the
>> cursor underscore starts blinking again ...
>
> Set cursor-interval=0 in your ~/.fbtermrc.

Okay then, so you got any heads-up what other fundamental
building-block software of my computer stack that is expected
to be expected to be replaced to compensate for coming Emacs
development efforts?

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Gregory Heytings]

>>> But even so, then one has to redo everything, even the cursor 
>>> underscore starts blinking again ...
>>
>> Set cursor-interval=0 in your ~/.fbtermrc.
>
> Okay then, so you got any heads-up what other fundamental building-block 
> software of my computer stack that is expected to be expected to be 
> replaced to compensate for coming Emacs development efforts?
>

I cannot parse this, sorry.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Alan Mackenzie]

Hello, Gregory.

On Fri, Aug 26, 2022 at 09:58:23 +0000, Gregory Heytings wrote:

> > The text-based Linux console is unable to display many Unicode 
> > characters.

> Did you try to use fbterm?  Most Unicode should be displayed correctly.

A problem with this terminal emulator (along, it seems, with most
others) is that it steals key sequnces which Emacs needs.

Amongst those which fbterm's author thought would never be used by a
user are C-<space> and C-M-k.

Maybe fbterm has a way of freeing up these key sequences (I didn't get
that far into its FAQ), but I doubt it.  That would make it not very
useful for running Emacs on.

Shame.  If only these terminal emulators could restrict themselves to
"standard" key sequences, I suspect they would have become more
prevelant than they have.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Gregory Heytings]

>
> A problem with this terminal emulator (along, it seems, with most 
> others) is that it steals key sequnces which Emacs needs.
>
> Amongst those which fbterm's author thought would never be used by a 
> user are C-<space> and C-M-k.
>
> Maybe fbterm has a way of freeing up these key sequences (I didn't get 
> that far into its FAQ), but I doubt it.
>

In fact the default fbterm configuration (on Debian at least) does not 
steal those key sequences.  They are stealed only if you manually add 
either a setuid bit or a capability on fbterm's binary.  If the setuid bit 
or capability is set by default in other distributions, removing it should 
be enough.  (And if you're annoyed by the warning that fbterm prints 
because it cannot steal those key sequences, you can add a "clear" in you 
shell startup script.)

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Alan Mackenzie]

Hello, Gregory.

On Fri, Aug 26, 2022 at 13:29:30 +0000, Gregory Heytings wrote:


> > A problem with this terminal emulator (along, it seems, with most 
> > others) is that it steals key sequnces which Emacs needs.

> > Amongst those which fbterm's author thought would never be used by a 
> > user are C-<space> and C-M-k.

> > Maybe fbterm has a way of freeing up these key sequences (I didn't get 
> > that far into its FAQ), but I doubt it.


> In fact the default fbterm configuration (on Debian at least) does not 
> steal those key sequences.  They are stealed only if you manually add 
> either a setuid bit or a capability on fbterm's binary.  If the setuid bit 
> or capability is set by default in other distributions, removing it should 
> be enough.  (And if you're annoyed by the warning that fbterm prints 
> because it cannot steal those key sequences, you can add a "clear" in you 
> shell startup script.)

I've tried fbterm out, thanks!  It comes up in a font which is too small
for me, but that is easily remedied in the configuration file ~/.fbterm.

Indeed it doesn't steal C-<space> or C-M-k, but it does take some other
bindings like C-<left>/C-<right>.  I don't think there's an option in
fbterm to leave all key sequences alone.

And it displays these pesky unicode punctuation characters correctly.
:-)

At the moment, I'm not sure whether I will end up using it or not.
Probably not.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> Date: Sun, 28 Aug 2022 10:35:22 +0000
> Cc: Richard Stallman <rms@gnu.org>, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> I've tried fbterm out, thanks!  It comes up in a font which is too small
> for me, but that is easily remedied in the configuration file ~/.fbterm.
> 
> Indeed it doesn't steal C-<space> or C-M-k, but it does take some other
> bindings like C-<left>/C-<right>.  I don't think there's an option in
> fbterm to leave all key sequences alone.

No text terminal can support all the key combinations a GUI display
can.  Many terminals don't support C-<left> either (xterm is one such
prominent example).

So this shouldn't prevent you from using this console.

> At the moment, I'm not sure whether I will end up using it or not.
> Probably not.

Why not?  The Linux console is abysmally inadequate for Emacs, from
where I stand.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 567 bytes --]


>> At the moment, I'm not sure whether I will end up using it or not. 
>> Probably not.
>
> Why not?  The Linux console is abysmally inadequate for Emacs, from 
> where I stand.
>

Indeed.  It's not clear to me why we are having this discussion.  With 
fbterm, the Linux console is on the par with most terminal emulators. 
FTR, here are two screenshots of Emacs' HELLO file, one with Emacs running 
in a raw console, and one with Emacs running inside fbterm.

Eli, is the attached patch okay?  Or is there a better place for that 
information (efaq.texi, perhaps?)?

[-- Attachment #2: raw.png --]
[-- Type: image/png, Size: 66695 bytes --]

[-- Attachment #3: fbterm.png --]
[-- Type: image/png, Size: 360039 bytes --]

[-- Attachment #4: etc-PROBLEMS-Recommend-using-fbterm-in-the-Linux-con.patch --]
[-- Type: text/x-diff, Size: 1127 bytes --]

From 1b7dc043fee520da99e545101bc5be69be1fb3cb Mon Sep 17 00:00:00 2001
From: Gregory Heytings <gregory@heytings.org>
Date: Sun, 28 Aug 2022 12:53:42 +0000
Subject: [PATCH] * etc/PROBLEMS: Recommend using fbterm in the Linux console.

---
 etc/PROBLEMS | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/etc/PROBLEMS b/etc/PROBLEMS
index 6624f747c8..dbc7e90a06 100644
--- a/etc/PROBLEMS
+++ b/etc/PROBLEMS
@@ -2267,6 +2267,16 @@ term/xterm.el) for more details.
 
 *** Linux console problems with double-width characters
 
+If possible, it is recommended to run, in a Linux console, Emacs
+inside fbterm, with the following command line:
+
+fbterm -- emacs
+
+Most Unicode characters should then be displayed correctly.
+
+If that is not possible, the following may be useful to alleviate the
+problem of displaying Unicode characters in a raw console.
+
 The Linux console declares UTF-8 encoding, but supports only a limited
 number of Unicode characters, and can cause Emacs produce corrupted or
 garbled display with some unusual characters and sequences.  Emacs 28
-- 
2.35.1

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> Date: Sun, 28 Aug 2022 13:00:44 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: Alan Mackenzie <acm@muc.de>, rms@gnu.org, emacs-devel@gnu.org
> 
> > Why not?  The Linux console is abysmally inadequate for Emacs, from 
> > where I stand.
> 
> Indeed.  It's not clear to me why we are having this discussion.  With 
> fbterm, the Linux console is on the par with most terminal emulators. 
> FTR, here are two screenshots of Emacs' HELLO file, one with Emacs running 
> in a raw console, and one with Emacs running inside fbterm.
> 
> Eli, is the attached patch okay?  Or is there a better place for that 
> information (efaq.texi, perhaps?)?

I think both?

For PROBLEMS, the entry is OK, but please change "it is recommended"
to say "we recommend".

Btw, does fbterm warrant or need a file in lisp/term/ ?

And what does tty-type return when running Emacs on fbterm?

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Gregory Heytings]

>
> I think both?
>
> For PROBLEMS, the entry is OK, but please change "it is recommended" to 
> say "we recommend".
>

Okay, I'll do that.

>
> Btw, does fbterm warrant or need a file in lisp/term/ ?
>

I don't know (but I think not).

>
> And what does tty-type return when running Emacs on fbterm?
>

"linux"

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> Date: Sun, 28 Aug 2022 14:44:41 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: acm@muc.de, rms@gnu.org, emacs-devel@gnu.org
> 
> > And what does tty-type return when running Emacs on fbterm?
> 
> "linux"

Ouch! that's bed -- it means the change I installed on the release
branch will affect fbterm as well, while it ideally shouldn't.

Any idea whether there's an easy way of distinguishing between those
two?

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Gregory Heytings]

>>> And what does tty-type return when running Emacs on fbterm?
>>
>> "linux"
>
> Ouch! that's bed -- it means the change I installed on the release 
> branch will affect fbterm as well, while it ideally shouldn't.
>
> Any idea whether there's an easy way of distinguishing between those 
> two?
>

I don't know, I did not (yet?) find a way to distinguish them.  One 
difference is that the "tty" command returns "/dev/ttyN" in a raw terminal 
and "/dev/pts/N" inside fbterm, but that's not reflected in Emacs' 
(frame-parameters): (cdr (assq 'tty (frame-parameters))) is "/dev/tty" in 
both cases.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> Date: Sun, 28 Aug 2022 15:18:12 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: acm@muc.de, rms@gnu.org, emacs-devel@gnu.org
> 
> >> "linux"
> >
> > Ouch! that's bed -- it means the change I installed on the release 
> > branch will affect fbterm as well, while it ideally shouldn't.
> >
> > Any idea whether there's an easy way of distinguishing between those 
> > two?
> 
> I don't know, I did not (yet?) find a way to distinguish them.  One 
> difference is that the "tty" command returns "/dev/ttyN" in a raw terminal 
> and "/dev/pts/N" inside fbterm, but that's not reflected in Emacs' 
> (frame-parameters): (cdr (assq 'tty (frame-parameters))) is "/dev/tty" in 
> both cases.

Well, maybe the situation is not that bad, actually: we'll just waste
some cycles at info.el startup, to find that all the necessary
characters are supported if the terminal is fbterm.

Thanks.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Gregory Heytings]

>
> Well, maybe the situation is not that bad, actually: we'll just waste 
> some cycles at info.el startup, to find that all the necessary 
> characters are supported if the terminal is fbterm.
>

Okay.  Do you perhaps have a recipe with which I could check that it's 
indeed the case?  It's unclear to me what problem the patch is fixing.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> Date: Sun, 28 Aug 2022 15:53:27 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: acm@muc.de, rms@gnu.org, emacs-devel@gnu.org
> 
> > Well, maybe the situation is not that bad, actually: we'll just waste 
> > some cycles at info.el startup, to find that all the necessary 
> > characters are supported if the terminal is fbterm.
> 
> Okay.  Do you perhaps have a recipe with which I could check that it's 
> indeed the case?

Start Info in a new session and see how much it takes.  Compare the
times on the release branch and on master (the change was installed on
the branch and was not yet merged to master).

> It's unclear to me what problem the patch is fixing.

It fixes the problem of the Linux console, which is unable to display
the non-ASCII symbols used by Texinfo although terminal-coding-system
is UTF-8.  For the list of the symbols, see
info-symbols-and-replacements.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Gregory Heytings]

>> Okay.  Do you perhaps have a recipe with which I could check that it's 
>> indeed the case?
>
> Start Info in a new session and see how much it takes.  Compare the 
> times on the release branch and on master (the change was installed on 
> the branch and was not yet merged to master).
>

There is no visible difference here, starting Info takes ~60 ms with and 
without the patch.

>> It's unclear to me what problem the patch is fixing.
>
> It fixes the problem of the Linux console, which is unable to display 
> the non-ASCII symbols used by Texinfo although terminal-coding-system is 
> UTF-8.  For the list of the symbols, see info-symbols-and-replacements.
>

Thanks, I see curly quotes and quotation marks inside Info with the patch, 
so it works as expected.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> Date: Sun, 28 Aug 2022 16:07:54 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: acm@muc.de, rms@gnu.org, emacs-devel@gnu.org
> 
> > Start Info in a new session and see how much it takes.  Compare the 
> > times on the release branch and on master (the change was installed on 
> > the branch and was not yet merged to master).
> 
> There is no visible difference here, starting Info takes ~60 ms with and 
> without the patch.
> 
> >> It's unclear to me what problem the patch is fixing.
> >
> > It fixes the problem of the Linux console, which is unable to display 
> > the non-ASCII symbols used by Texinfo although terminal-coding-system is 
> > UTF-8.  For the list of the symbols, see info-symbols-and-replacements.
> 
> Thanks, I see curly quotes and quotation marks inside Info with the patch, 
> so it works as expected.

OK, thanks.  I guess my worries about the change are unjustified,
then.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 243 bytes --]


>> I think both?
>> 
>> For PROBLEMS, the entry is OK, but please change "it is recommended" to 
>> say "we recommend".
>
> Okay, I'll do that.
>

See the attached (documentation-only) patch.  Should it go to master or to 
the release branch?

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Recommend-using-fbterm-in-the-Linux-console.patch --]
[-- Type: text/x-diff; name=Recommend-using-fbterm-in-the-Linux-console.patch, Size: 4071 bytes --]

From e9010023e6d33ba84dca4b7aea284c9d804b0b8c Mon Sep 17 00:00:00 2001
From: Gregory Heytings <gregory@heytings.org>
Date: Sun, 28 Aug 2022 20:04:11 +0000
Subject: [PATCH] Recommend using fbterm in the Linux console.

* doc/misc/efaq.texi (Emacs in a Linux console): New node.
(Common requests): Entry for the new node.

* etc/PROBLEMS (Linux console problems...): Mention the new FAQ node.
---
 doc/misc/efaq.texi | 65 ++++++++++++++++++++++++++++++++++++++++++++++
 etc/PROBLEMS       |  7 +++++
 2 files changed, 72 insertions(+)

diff --git a/doc/misc/efaq.texi b/doc/misc/efaq.texi
index a3459abd04..5fa8080f4b 100644
--- a/doc/misc/efaq.texi
+++ b/doc/misc/efaq.texi
@@ -1602,6 +1602,7 @@ Common requests
 * Filling paragraphs with a single space::
 * Escape sequences in shell output::
 * Fullscreen mode on MS-Windows::
+* Emacs in a Linux console::
 @end menu
 
 @node Setting up a customization file
@@ -3031,6 +3032,70 @@ Fullscreen mode on MS-Windows
 Emacs frame and then evaluate @code{(frame-height)} and
 @code{(frame-width)} with @kbd{M-:}.
 
+@node Emacs in a Linux console
+@section How can I alleviate the limitations of the Linux console?
+@cindex Console, TTY
+
+If possible, we recommend to run, in a Linux console, Emacs inside
+fbterm.  This brings the Linux console on par with most terminal
+emulators under X.  To do this, install fbterm, for example with the
+package manager of your GNU/Linux distribution, and execute the
+command
+
+@example
+$ fbterm
+@end example
+
+This will create a sample configuration file @file{~/.fbtermrc} in
+your home directory.  Edit that file and change the options
+@code{font-names} and @code{font-size} if necessary.  For the former,
+you can choose one or more of the lines in the output of the following
+command, separated by commas:
+
+@example
+$ fc-list :spacing=mono family | sed 's/ /\\ /g'
+@end example
+
+Now you can start Emacs inside fbterm with the command
+
+@example
+$ fbterm -- emacs
+@end example
+
+You may want to add an alias for that command in your shell
+configuration file.  For example, if you use Bash, you can add the
+following line to @file{~/.bashrc}:
+
+@example
+alias emacs="fbterm -- emacs"
+@end example
+
+@noindent
+or, if you use Emacs both in the Linux console and under X:
+
+@example
+[[ "$(tty)" =~ "/dev/tty" ]] && alias emacs="fbterm -- emacs"
+@end example
+
+The fbterm terminal emulator may use a number of key bindings for its
+own use, some of which conflict with those that Emacs uses.  Execute
+the following two commands as root to ensure that fbterm does not use
+these key bindings:
+
+@example
+# chmod a-s `which fbterm`
+# setcap cap_sys_tty_config=-ep `which fbterm`
+@end example
+
+If you use Emacs as a root user, the above is not enough, because the
+root user has all privileges, and you can use the following command to
+start Emacs inside fbterm as a root user while ensuring that fbterm
+does not use these key bindings:
+
+@example
+# capsh --drop=cap_sys_tty_config -- -c 'fbterm -- emacs'
+@end example
+
 @c ------------------------------------------------------------
 @node Bugs and problems
 @chapter Bugs and problems
diff --git a/etc/PROBLEMS b/etc/PROBLEMS
index 6624f747c8..3816919d3b 100644
--- a/etc/PROBLEMS
+++ b/etc/PROBLEMS
@@ -2267,6 +2267,13 @@ term/xterm.el) for more details.
 
 *** Linux console problems with double-width characters
 
+If possible, we recommended to run, in a Linux console, Emacs inside
+fbterm (see the node "Emacs in a Linux console" in the Emacs FAQ).
+Most Unicode characters should then be displayed correctly.
+
+If that is not possible, the following may be useful to alleviate the
+problem of displaying Unicode characters in a raw console.
+
 The Linux console declares UTF-8 encoding, but supports only a limited
 number of Unicode characters, and can cause Emacs produce corrupted or
 garbled display with some unusual characters and sequences.  Emacs 28
-- 
2.35.1

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> Date: Sun, 28 Aug 2022 20:07:27 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: acm@muc.de, rms@gnu.org, emacs-devel@gnu.org
> 
> See the attached (documentation-only) patch.  Should it go to master or to 
> the release branch?

The latter.

A few minor comments below:

> +@node Emacs in a Linux console
> +@section How can I alleviate the limitations of the Linux console?
> +@cindex Console, TTY

I'd add here a @cindex entries for fbterm and "Linux console".

> +If possible, we recommend to run, in a Linux console, Emacs inside
> +fbterm.  This brings the Linux console on par with most terminal

Since fbterm is a program, it should have the @command markup (here
and elsewhere, except in @example's).

Thanks.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Gregory Heytings]

>> See the attached (documentation-only) patch.  Should it go to master or 
>> to the release branch?
>
> The latter.
>

Thanks, now done, with a few other minor improvements (commit 2d9a391727).

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Alan Mackenzie]

Hello, Eli.

On Sun, Aug 28, 2022 at 17:18:25 +0300, Eli Zaretskii wrote:
> > Date: Sun, 28 Aug 2022 13:00:44 +0000
> > From: Gregory Heytings <gregory@heytings.org>
> > cc: Alan Mackenzie <acm@muc.de>, rms@gnu.org, emacs-devel@gnu.org

> > > Why not?  The Linux console is abysmally inadequate for Emacs, from 
> > > where I stand.

> > Indeed.  It's not clear to me why we are having this discussion.  With 
> > fbterm, the Linux console is on the par with most terminal emulators. 
> > FTR, here are two screenshots of Emacs' HELLO file, one with Emacs running 
> > in a raw console, and one with Emacs running inside fbterm.

> > Eli, is the attached patch okay?  Or is there a better place for that 
> > information (efaq.texi, perhaps?)?

> I think both?

> For PROBLEMS, the entry is OK, but please change "it is recommended"
> to say "we recommend".

> Btw, does fbterm warrant or need a file in lisp/term/ ?

> And what does tty-type return when running Emacs on fbterm?

On my system, it returns "fbterm".

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> Date: Sun, 28 Aug 2022 15:39:09 +0000
> Cc: Gregory Heytings <gregory@heytings.org>, rms@gnu.org, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > And what does tty-type return when running Emacs on fbterm?
> 
> On my system, it returns "fbterm".

If on some systems it returns a string that's different from "linux",
then starting Info for the first time will be faster on those systems.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Alan Mackenzie]

Hello, Gregory.

On Sun, Aug 28, 2022 at 13:00:44 +0000, Gregory Heytings wrote:

> >> At the moment, I'm not sure whether I will end up using it or not. 
> >> Probably not.

> > Why not?  The Linux console is abysmally inadequate for Emacs, from 
> > where I stand.

> Indeed.  It's not clear to me why we are having this discussion.  With 
> fbterm, the Linux console is on the par with most terminal emulators. 

In my copy of fbterm C-M-e aborts the fbterm session without saving
anything.  C-M-c and C-M-d are similarly stolen, making this version of
fbterm unusable for Emacs.

Have you tried these key combinations?  Do they really get through to
Emacs on your system?

> FTR, here are two screenshots of Emacs' HELLO file, one with Emacs running 
> in a raw console, and one with Emacs running inside fbterm.

> Eli, is the attached patch okay?  Or is there a better place for that 
> information (efaq.texi, perhaps?)?

I think recommending fbterm is premature.  The problems I've run into
with it need resolving, first.

[ .... ]

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> Date: Sun, 28 Aug 2022 15:35:22 +0000
> Cc: Eli Zaretskii <eliz@gnu.org>, rms@gnu.org, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> I think recommending fbterm is premature.

No, it isn't.  Whatever the problems you found, they are far less
serious than what you get on the default Linux console.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Gregory Heytings]

>
> In my copy of fbterm C-M-e aborts the fbterm session without saving 
> anything.  C-M-c and C-M-d are similarly stolen, making this version of 
> fbterm unusable for Emacs.
>

That means either that you're running fbterm and Emacs as root, or that 
fbterm is installed with a setuid bit or has the SYS_TTY_CONFIG 
capability, which you can remove (see upthread).

>
> Have you tried these key combinations?  Do they really get through to 
> Emacs on your system?
>

Yes I tried, and yes they did.  I've been using fbterm for quite some time 
in fact (but unlike you I don't use the console most of the time).

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Alan Mackenzie]

Hello, Gregory.

On Sun, Aug 28, 2022 at 15:46:23 +0000, Gregory Heytings wrote:


> > In my copy of fbterm C-M-e aborts the fbterm session without saving 
> > anything.  C-M-c and C-M-d are similarly stolen, making this version of 
> > fbterm unusable for Emacs.


> That means either that you're running fbterm and Emacs as root, or that 
> fbterm is installed with a setuid bit or has the SYS_TTY_CONFIG 
> capability, which you can remove (see upthread).

All these reservations surely need to go into your entry in the PROBLEMS
file.  People run Emacs as root, e.g. for editing configuration files.
Do we really expect users to be able to flip setuid bits, or remove
SYS_TTY_CONFIG capability?  Myself, I don't even know what the second of
these means, though I will examine the thread to try and find out.

> > Have you tried these key combinations?  Do they really get through to 
> > Emacs on your system?


> Yes I tried, and yes they did.  I've been using fbterm for quite some time 
> in fact (but unlike you I don't use the console most of the time).

OK.  There remains the mystery of why M-: (tty-type) returns "linux" on
your system, but "fbterm" on mine.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Gregory Heytings]

>
> All these reservations surely need to go into your entry in the PROBLEMS 
> file.  People run Emacs as root, e.g. for editing configuration files. 
> Do we really expect users to be able to flip setuid bits, or remove 
> SYS_TTY_CONFIG capability?  Myself, I don't even know what the second of 
> these means, though I will examine the thread to try and find out.
>

I'll add all these details in the FAQ entry.

>
> OK.  There remains the mystery of why M-: (tty-type) returns "linux" on 
> your system, but "fbterm" on mine.
>

I guess you're using some modified version of fbterm, some distros provide 
patched versions of the programs they distribute.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Stefan Monnier]

> That means either that you're running fbterm and Emacs as root, or that
> fbterm is installed with a setuid bit or has the SYS_TTY_CONFIG capability,
> which you can remove (see upthread).

Isn't there some way to explicitly request the "unpriviledged" mode of
operation (e.g. with a command line arg, config file setting, you name
it) so it also applies when the application does have the needed
privilege (e.g. when running as root)?


        Stefan

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Stefan Monnier]

>> That means either that you're running fbterm and Emacs as root, or that
>> fbterm is installed with a setuid bit or has the SYS_TTY_CONFIG capability,
>> which you can remove (see upthread).
> Isn't there some way to explicitly request the "unpriviledged" mode of
> operation (e.g. with a command line arg, config file setting, you name
> it) so it also applies when the application does have the needed
> privilege (e.g. when running as root)?

Or better yet: some way to change those keyboard shortcuts so they don't
conflict with those of Emacs?


        Stefan

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Alan Mackenzie]

Hello, Stefan.

On Sun, Aug 28, 2022 at 14:27:22 -0400, Stefan Monnier wrote:
> > That means either that you're running fbterm and Emacs as root, or that
> > fbterm is installed with a setuid bit or has the SYS_TTY_CONFIG capability,
> > which you can remove (see upthread).

> Isn't there some way to explicitly request the "unprivileged" mode of
> operation (e.g. with a command line arg, config file setting, you name
> it) so it also applies when the application does have the needed
> privilege (e.g. when running as root)?

No, there isn't.  I've been looking at fbterm's source code.  The
"unprivileged" mode is actually a malfunction due to the user having
insufficient privilege to execute certain ioctl commands in the kernel
which configure the keyboard.

When fbterm executes as intended, all the "stolen" key combinations
(with the execption of C-<space> and C-M-k, which have, in effect, been
commented out in the key table) get remapped.  This includes critical
Emacs key sequences such as C-M-e and C-M-c.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Gregory Heytings]

>>> That means either that you're running fbterm and Emacs as root, or 
>>> that fbterm is installed with a setuid bit or has the SYS_TTY_CONFIG 
>>> capability, which you can remove (see upthread).
>>
>> Isn't there some way to explicitly request the "unprivileged" mode of 
>> operation (e.g. with a command line arg, config file setting, you name 
>> it) so it also applies when the application does have the needed 
>> privilege (e.g. when running as root)?
>
> No, there isn't.
>

Of course there is.  All details are in the new FAQ node that will be 
installed after being reviewed.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Alan Mackenzie]

Hello, Gregory.

On Sun, Aug 28, 2022 at 20:09:28 +0000, Gregory Heytings wrote:

> >>> That means either that you're running fbterm and Emacs as root, or 
> >>> that fbterm is installed with a setuid bit or has the SYS_TTY_CONFIG 
> >>> capability, which you can remove (see upthread).

> >> Isn't there some way to explicitly request the "unprivileged" mode of 
> >> operation (e.g. with a command line arg, config file setting, you name 
> >> it) so it also applies when the application does have the needed 
> >> privilege (e.g. when running as root)?

> > No, there isn't.


> Of course there is.  All details are in the new FAQ node that will be 
> installed after being reviewed.

OK, I was wrong.  Thanks for the education!

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Gregory Heytings wrote:

> Indeed. It's not clear to me why we are having this
> discussion. With fbterm, the Linux console is on the par
> with most terminal emulators. FTR, here are two screenshots
> of Emacs' HELLO file, one with Emacs running in a raw
> console, and one with Emacs running inside fbterm.

Sure ... but there are a lot of things to do if one is to
migrate, fonts, colors, keys.

In particular keys

  https://dataswamp.org/~incal/conf/vt/remap.inc

But maybe that's easy to do? Cmp. mplayer vs mpv ...

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Alan Mackenzie]

Hello, Eli.

On Sun, Aug 28, 2022 at 13:41:28 +0300, Eli Zaretskii wrote:
> > Date: Sun, 28 Aug 2022 10:35:22 +0000
> > Cc: Richard Stallman <rms@gnu.org>, emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > I've tried fbterm out, thanks!  It comes up in a font which is too small
> > for me, but that is easily remedied in the configuration file ~/.fbterm.

> > Indeed it doesn't steal C-<space> or C-M-k, but it does take some other
> > bindings like C-<left>/C-<right>.  I don't think there's an option in
> > fbterm to leave all key sequences alone.

> No text terminal can support all the key combinations a GUI display
> can.

I think it can, though that involves enhancing the keyboard layout.
loadkeys is my friend, here.  There is no key combination I've wanted
that I've been unable to create with loadkeys.

X-Windows is a different matter, just as a matter of interest.  To get
C-M-S-<F1> etc. working, I had to amend the source of some keyboard
thing (I can't remember exactly what, any more).

> Many terminals don't support C-<left> either (xterm is one such
> prominent example).

My keyboard certainly supports C-<left>, etc.

> So this shouldn't prevent you from using this console.

fbterm appears to respect the loadkeys setting mostly, anyway.

> > At the moment, I'm not sure whether I will end up using it or not.
> > Probably not.

> Why not?  The Linux console is abysmally inadequate for Emacs, from
> where I stand.

I probably use a more boring selection of languages to write in than you
do, just a couple of West European ones, so the Linux console's
limitations will affect me a lot less.

Partly, it is a good thing for there to be a Linux console user on the
development team, since this helps maintain Emacs's usability on it.
People do use Emacs in this environment.

Also, I haven't found out how to modify the fonts in use on fbterm, and
I don't like the default one supplied.  Maybe I'll find that out.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Gregory Heytings]

>
> Also, I haven't found out how to modify the fonts in use on fbterm, and 
> I don't like the default one supplied.
>

The first two user options in ~/.fbtermrc are "font-names" and 
"font-size".  For the former, you can use any fonts returned by "fc-list 
:spacing=mono".  Note that it's a list separated by commas, in which the 
next ones are fallbacks when the previous ones do not contain a glyph for 
a given character.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Eli Zaretskii wrote:

> No text terminal can support all the key combinations a GUI
> display can.

Why ever not?

> The Linux console is abysmally inadequate for Emacs, from
> where I stand.

In terms of text, what's missing?

But it is atypical for Linux because it isn't feature-rich,
they don't try to make it better, and they don't add stuff,
this is a policy, even.

But if one uses feature-rich systems on top of it or with it,
for example tmux which is very common, here Emacs as we have
heard, zsh, etc, I don't see how they should be limited
by it, really?

But it is possible/likely that fbterm, if that is the
feature-rich option then, that the other posted suggested in
this thread, that that is indeed more powerful and offers
a more consistent configuration and extension interface,
active development and all that ...

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Sun, 28 Aug 2022 22:05:28 +0200
> 
> Eli Zaretskii wrote:
> 
> > No text terminal can support all the key combinations a GUI
> > display can.
> 
> Why ever not?

Because not every combination can be expressed as a character code.

> > The Linux console is abysmally inadequate for Emacs, from
> > where I stand.
> 
> In terms of text, what's missing?

Gobs of characters without glyphs.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Yuri Khan]

On Mon, 29 Aug 2022 at 18:00, Eli Zaretskii <eliz@gnu.org> wrote:

> > > No text terminal can support all the key combinations a GUI
> > > display can.
> >
> > Why ever not?
>
> Because not every combination can be expressed as a character code.

Kitty manages to represent a lot of key combinations in its
[comprehensive key protocol][1]. Someone™ could implement that in
fbterm, too.

(The current iteration of the comprehensive key protocol is a lot
easier to work with in Emacs than the previous one, too. This one does
not require one to filter out key release events.)

[1]: https://sw.kovidgoyal.net/kitty/keyboard-protocol/

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> From: Yuri Khan <yuri.v.khan@gmail.com>
> Date: Mon, 29 Aug 2022 18:26:00 +0700
> Cc: Emanuel Berg <incal@dataswamp.org>, emacs-devel@gnu.org
> 
> On Mon, 29 Aug 2022 at 18:00, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > > > No text terminal can support all the key combinations a GUI
> > > > display can.
> > >
> > > Why ever not?
> >
> > Because not every combination can be expressed as a character code.
> 
> Kitty manages to represent a lot of key combinations in its
> [comprehensive key protocol][1]. Someone™ could implement that in
> fbterm, too.

Kitty isn't a console program, it's a terminal emulator running on X.
That's not what I described, nor is it what we are discussing here.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Alan Mackenzie]

Hello, Eli.

On Mon, Aug 29, 2022 at 14:37:43 +0300, Eli Zaretskii wrote:
> > From: Yuri Khan <yuri.v.khan@gmail.com>
> > Date: Mon, 29 Aug 2022 18:26:00 +0700
> > Cc: Emanuel Berg <incal@dataswamp.org>, emacs-devel@gnu.org

> > On Mon, 29 Aug 2022 at 18:00, Eli Zaretskii <eliz@gnu.org> wrote:

> > > > > No text terminal can support all the key combinations a GUI
> > > > > display can.

> > > > Why ever not?

> > > Because not every combination can be expressed as a character code.

> > Kitty manages to represent a lot of key combinations in its
> > [comprehensive key protocol][1]. Someone™ could implement that in
> > fbterm, too.

> Kitty isn't a console program, it's a terminal emulator running on X.
> That's not what I described, nor is it what we are discussing here.

On a Linux console essentially every key combination of shift keys plus a
non-shift key CAN be represented.  It takes some work, but that work was
done over 20 years ago by Kalle Niemitalo.  The keyboard layout needs to
be extended, and some Lisp code needs to be added to the function-key-map
to interpret the new codes.  It works well.

fbterm uses the key layout configured outside fbterm (except for the
"special" keys, had Gregory not masked them).

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Gregory Heytings]

>
> Indeed it doesn't steal C-<space> or C-M-k, but it does take some other 
> bindings like C-<left>/C-<right>.  I don't think there's an option in 
> fbterm to leave all key sequences alone.
>

Hmm...  I'm not sure what you did, here fbterm does not (by default) steal 
a single key sequence.  The only key sequences that are not sent to Emacs 
are those that are predefined, like C-M-F1 or M-<left> and M-<right>. 
(It should be possible to patch it so that all key sequences, including 
the predefined ones, are sent to Emacs.)

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Alan Mackenzie]

Hello, Richard.

On Thu, Aug 25, 2022 at 23:33:57 -0400, Richard Stallman wrote:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]

> The text-based Linux console is unable to display many Unicode
> characters.  In Emacs 29, display of undisplayable characters has
> changed.  It used to show them with a diamond.  Now it shows the
> unicode character code as hex, preceded by \U.

Yes.  On my character bitmap, I see a reversed `?'.  I'm pretty sure
this is \Ufffd, and that your diamond is the same character.

> I find that change quite inconvenient.  It makes the text harder to
> read.  Showing the codes does no good, since I don't know these codes,
> not even for characters I am familiar with.  To find out what
> character a code represents, I have to use C-u C-x =, just as I did
> before.

As somebody who uses solely the Linux console for Emacs (apart,
sometimes, from when testing), I agree.

It disturbs me most whilst reading info.  Modern versions of makeinfo
generate Unicode punctuation characters, and having them displayed on
the screen as \Uxxxx completely screws up the alignment these characters
are meant to achieve (for example).

> I last built the sources in May.  Has this changed in a significant
> way since then?

My feeling is that the change is older than three months.  It's one of
these changes that one gradually becomes aware of, and which irritates,
but isn't quite serious enough to do anything about.  :-(

> I could not find, in NEWS, anything about this change -- it ought to
> be in NEWS .  Nor a way to go back to the old
> behavior.  Is there a way?  If not, would people please create a way?

> As a separate question, do users generally like this change?
> Would it be better to return to the old diamond method as the default?

If there were an option to use \Ufffd, I would surely set that option
that way.  Particularly in Info mode.

> -- 
> Dr Richard Stallman (https://stallman.org)
> Chief GNUisance of the GNU Project (https://gnu.org)
> Founder, Free Software Foundation (https://fsf.org)
> Internet Hall-of-Famer (https://internethalloffame.org)

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Alan Mackenzie wrote:

> My feeling is that the change is older than three months.
> It's one of these changes that one gradually becomes aware
> of, and which irritates, but isn't quite serious enough to
> do anything about. :-(

I think we should just try to do Emacs as good as possible
everywhere we can and not think too much what is serious enough or
not enough, that's difficult to say, also.

And in particular because we had the old style already so we
should know how to do that already, and we know how to do the
new style as well, so we just need a little interface so the
user can set up what way to branch to the prefered one from
there ...

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Sat, 27 Aug 2022 00:56:31 +0200
> 
> And in particular because we had the old style already so we
> should know how to do that already

No, we didn't "have the old style".  We had a bug: we sent to a
terminal a character code that it cannot display.  That should never
happen in Emacs.  That bug is now fixed, and such characters are
considered as not displayable by the terminal, as they should be.

We are not going back to the situation where we sent to the Linux
console character codes for which it couldn't produce valid glyphs.
People who suggest sending U+FFFD to the Linux console should first
verify whether the console is capable of displaying that character.
One way of verifying this is to type

   C-x 8 RET fffd RET

in an Emacs session which displays on the Linux console, and see if
that shows the diamond-like glyph or the \Unnnn thing.  If it's the
latter, it means the console doesn't support that character, and we
cannot send its code from Emacs.  We should instead send something
else (provided that someone submits patches to send that "something").

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Visuwesh]

[சனி ஆகஸ்ட் 27, 2022] Eli Zaretskii wrote:

> We are not going back to the situation where we sent to the Linux
> console character codes for which it couldn't produce valid glyphs.
> People who suggest sending U+FFFD to the Linux console should first
> verify whether the console is capable of displaying that character.
> One way of verifying this is to type
>
>    C-x 8 RET fffd RET
>
> in an Emacs session which displays on the Linux console, and see if
> that shows the diamond-like glyph or the \Unnnn thing.  If it's the
> latter, it means the console doesn't support that character, and we
> cannot send its code from Emacs.  We should instead send something
> else (provided that someone submits patches to send that "something").

It is displayed as a diamond in emacs -Q -nw in a Linux tty here.  I'm
on Debian GNU/Linux bookworm/sid.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> From: Visuwesh <visuweshm@gmail.com>
> Cc: Emanuel Berg <incal@dataswamp.org>,  emacs-devel@gnu.org
> Date: Sat, 27 Aug 2022 12:52:19 +0530
> 
> [சனி ஆகஸ்ட் 27, 2022] Eli Zaretskii wrote:
> 
> > We are not going back to the situation where we sent to the Linux
> > console character codes for which it couldn't produce valid glyphs.
> > People who suggest sending U+FFFD to the Linux console should first
> > verify whether the console is capable of displaying that character.
> > One way of verifying this is to type
> >
> >    C-x 8 RET fffd RET
> >
> > in an Emacs session which displays on the Linux console, and see if
> > that shows the diamond-like glyph or the \Unnnn thing.  If it's the
> > latter, it means the console doesn't support that character, and we
> > cannot send its code from Emacs.  We should instead send something
> > else (provided that someone submits patches to send that "something").
> 
> It is displayed as a diamond in emacs -Q -nw in a Linux tty here.  I'm
> on Debian GNU/Linux bookworm/sid.

In what Emacs version?

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Visuwesh]

[சனி ஆகஸ்ட் 27, 2022] Eli Zaretskii wrote:

>
>> From: Visuwesh <visuweshm@gmail.com>
>> Cc: Emanuel Berg <incal@dataswamp.org>,  emacs-devel@gnu.org
>> Date: Sat, 27 Aug 2022 12:52:19 +0530
>> 
>> [சனி ஆகஸ்ட் 27, 2022] Eli Zaretskii wrote:
>> 
>> > We are not going back to the situation where we sent to the Linux
>> > console character codes for which it couldn't produce valid glyphs.
>> > People who suggest sending U+FFFD to the Linux console should first
>> > verify whether the console is capable of displaying that character.
>> > One way of verifying this is to type
>> >
>> >    C-x 8 RET fffd RET
>> >
>> > in an Emacs session which displays on the Linux console, and see if
>> > that shows the diamond-like glyph or the \Unnnn thing.  If it's the
>> > latter, it means the console doesn't support that character, and we
>> > cannot send its code from Emacs.  We should instead send something
>> > else (provided that someone submits patches to send that "something").
>> 
>> It is displayed as a diamond in emacs -Q -nw in a Linux tty here.  I'm
>> on Debian GNU/Linux bookworm/sid.
>
> In what Emacs version?

Master.  emacs-repository-version is "ae2199258f6eba3a8a066616dad1a7bf666ec8be".

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > It disturbs me most whilst reading info.  Modern versions of makeinfo
  > generate Unicode punctuation characters, and having them displayed on
  > the screen as \Uxxxx completely screws up the alignment these characters
  > are meant to achieve (for example).

Can we modify info.el so as to display those characters
as the curresponding ASCII punctuation characters?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Sun, 28 Aug 2022 00:04:53 -0400
> 
>   > It disturbs me most whilst reading info.  Modern versions of makeinfo
>   > generate Unicode punctuation characters, and having them displayed on
>   > the screen as \Uxxxx completely screws up the alignment these characters
>   > are meant to achieve (for example).
> 
> Can we modify info.el so as to display those characters
> as the curresponding ASCII punctuation characters?

It already does, where an ASCII equivalent exists.  See
info-symbols-and-replacements.  Alan didn't report any details, so I
don't have any idea why it doesn't work for him (if indeed it doesn't).

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Alan Mackenzie]

Hello, Eli.

On Sun, Aug 28, 2022 at 08:22:02 +0300, Eli Zaretskii wrote:
> > From: Richard Stallman <rms@gnu.org>
> > Cc: emacs-devel@gnu.org
> > Date: Sun, 28 Aug 2022 00:04:53 -0400

> >   > It disturbs me most whilst reading info.  Modern versions of makeinfo
> >   > generate Unicode punctuation characters, and having them displayed on
> >   > the screen as \Uxxxx completely screws up the alignment these characters
> >   > are meant to achieve (for example).

> > Can we modify info.el so as to display those characters
> > as the curresponding ASCII punctuation characters?

> It already does, where an ASCII equivalent exists.  See
> info-symbols-and-replacements.  Alan didn't report any details, so I
> don't have any idea why it doesn't work for him (if indeed it doesn't).

I didn't actually know about that feature, which you added in October
2019 (commit 8dd18bbb6f3c09a4988cf2e06378aa24b098fb85).  It fails on my
Linux console because of a guarding clause in the `unless' which
surrounds the initialisation of buffer-display-table:

  (unless (or (display-multi-font-p)
              (coding-system-equal                 <====================
               (coding-system-base (terminal-coding-system)) <==========
               'utf-8))                            <====================
    (dolist (elt info-symbols-and-replacements)
      (let ((ch (car elt))
            (repl (cdr elt)))
        (or (char-displayable-p ch)
            (aset (or buffer-display-table
                      (setq buffer-display-table (make-display-table)))
                  ch (vconcat (mapcar (lambda (c)
                                        (make-glyph-code c 'homoglyph))
                                      repl)))))))

On my setup that (coding-system-equal .... 'utf-8) returns t, so
buffer-display-table never gets initialised.

Why is that coding-system-equal form there?  I think the mechanism would
work on my system if it weren't (though I haven't tried it, yet).

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> Date: Sun, 28 Aug 2022 10:07:00 +0000
> Cc: rms@gnu.org, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > > Can we modify info.el so as to display those characters
> > > as the curresponding ASCII punctuation characters?
> 
> > It already does, where an ASCII equivalent exists.  See
> > info-symbols-and-replacements.  Alan didn't report any details, so I
> > don't have any idea why it doesn't work for him (if indeed it doesn't).
> 
> I didn't actually know about that feature, which you added in October
> 2019 (commit 8dd18bbb6f3c09a4988cf2e06378aa24b098fb85).  It fails on my
> Linux console because of a guarding clause in the `unless' which
> surrounds the initialisation of buffer-display-table:
> 
>   (unless (or (display-multi-font-p)
>               (coding-system-equal                 <====================
>                (coding-system-base (terminal-coding-system)) <==========
>                'utf-8))                            <====================
>     (dolist (elt info-symbols-and-replacements)
>       (let ((ch (car elt))
>             (repl (cdr elt)))
>         (or (char-displayable-p ch)
>             (aset (or buffer-display-table
>                       (setq buffer-display-table (make-display-table)))
>                   ch (vconcat (mapcar (lambda (c)
>                                         (make-glyph-code c 'homoglyph))
>                                       repl)))))))
> 
> On my setup that (coding-system-equal .... 'utf-8) returns t, so
> buffer-display-table never gets initialised.
> 
> Why is that coding-system-equal form there?

It was written based on the assumption that a terminal that supports
the UTF-8 encoding can also display the original characters reasonably
enough.  Are you saying that the Linux console lies about that, and
those characters are displayed as hex codes?  If so, please try the
patch below.

diff --git a/lisp/info.el b/lisp/info.el
index 739116c..5655e3f 100644
--- a/lisp/info.el
+++ b/lisp/info.el
@@ -4446,9 +4446,12 @@ Info-mode
   (setq buffer-read-only t)
   (setq Info-tag-table-marker (make-marker))
   (unless (or (display-multi-font-p)
-              (coding-system-equal
-               (coding-system-base (terminal-coding-system))
-               'utf-8))
+              (and (coding-system-equal
+                    (coding-system-base (terminal-coding-system))
+                    'utf-8)
+                   ;; The Linux console has limited character
+                   ;; repertoire even when its encoding is UTF-8.
+                   (not (equal (tty-type) "linux"))))
     (dolist (elt info-symbols-and-replacements)
       (let ((ch (car elt))
             (repl (cdr elt)))

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Alan Mackenzie]

Hello, Eli.

On Sun, Aug 28, 2022 at 13:29:13 +0300, Eli Zaretskii wrote:
> > Date: Sun, 28 Aug 2022 10:07:00 +0000
> > Cc: rms@gnu.org, emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > > > Can we modify info.el so as to display those characters
> > > > as the curresponding ASCII punctuation characters?

> > > It already does, where an ASCII equivalent exists.  See
> > > info-symbols-and-replacements.  Alan didn't report any details, so I
> > > don't have any idea why it doesn't work for him (if indeed it doesn't).

> > I didn't actually know about that feature, which you added in October
> > 2019 (commit 8dd18bbb6f3c09a4988cf2e06378aa24b098fb85).  It fails on my
> > Linux console because of a guarding clause in the `unless' which
> > surrounds the initialisation of buffer-display-table:

> >   (unless (or (display-multi-font-p)
> >               (coding-system-equal                 <====================
> >                (coding-system-base (terminal-coding-system)) <==========
> >                'utf-8))                            <====================
> >     (dolist (elt info-symbols-and-replacements)
> >       (let ((ch (car elt))
> >             (repl (cdr elt)))
> >         (or (char-displayable-p ch)
> >             (aset (or buffer-display-table
> >                       (setq buffer-display-table (make-display-table)))
> >                   ch (vconcat (mapcar (lambda (c)
> >                                         (make-glyph-code c 'homoglyph))
> >                                       repl)))))))

> > On my setup that (coding-system-equal .... 'utf-8) returns t, so
> > buffer-display-table never gets initialised.

> > Why is that coding-system-equal form there?

> It was written based on the assumption that a terminal that supports
> the UTF-8 encoding can also display the original characters reasonably
> enough.  Are you saying that the Linux console lies about that, and
> those characters are displayed as hex codes?

Well, I think "lies" is a bit strong, here.  The Linux console uses UTF-8
characters, but is restricted to an arbitrary selection of 256 of them.
An attempt to display any other character on the screen gets \ufffd
displayed instead.  Possibly, double-width characters aren't correctly
handled, I don't know.  My console is configured to use Latin-1, and it
also includes some line graphic characters, but not the punctuation
characters used by makeinfo.

Those characters were indeed displayed as hex codes, until I applied the
patch below.

> If so, please try the patch below.

> diff --git a/lisp/info.el b/lisp/info.el
> index 739116c..5655e3f 100644
> --- a/lisp/info.el
> +++ b/lisp/info.el
> @@ -4446,9 +4446,12 @@ Info-mode
>    (setq buffer-read-only t)
>    (setq Info-tag-table-marker (make-marker))
>    (unless (or (display-multi-font-p)
> -              (coding-system-equal
> -               (coding-system-base (terminal-coding-system))
> -               'utf-8))
> +              (and (coding-system-equal
> +                    (coding-system-base (terminal-coding-system))
> +                    'utf-8)
> +                   ;; The Linux console has limited character
> +                   ;; repertoire even when its encoding is UTF-8.
> +                   (not (equal (tty-type) "linux"))))
>      (dolist (elt info-symbols-and-replacements)
>        (let ((ch (car elt))
>              (repl (cdr elt)))

This works for me.  :-)

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> Date: Sun, 28 Aug 2022 11:07:24 +0000
> Cc: rms@gnu.org, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > > Why is that coding-system-equal form there?
> 
> > It was written based on the assumption that a terminal that supports
> > the UTF-8 encoding can also display the original characters reasonably
> > enough.  Are you saying that the Linux console lies about that, and
> > those characters are displayed as hex codes?
> 
> Well, I think "lies" is a bit strong, here.  The Linux console uses UTF-8
> characters, but is restricted to an arbitrary selection of 256 of them.
> An attempt to display any other character on the screen gets \ufffd
> displayed instead.  Possibly, double-width characters aren't correctly
> handled, I don't know.  My console is configured to use Latin-1, and it
> also includes some line graphic characters, but not the punctuation
> characters used by makeinfo.

If the console is configured for Latin-1, how come the terminal's
encoding is UTF-8?

> Those characters were indeed displayed as hex codes, until I applied the
> patch below.

Thanks, installed on the release branch.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Alan Mackenzie]

Hello, Eli.

On Sun, Aug 28, 2022 at 14:44:03 +0300, Eli Zaretskii wrote:
> > Date: Sun, 28 Aug 2022 11:07:24 +0000
> > Cc: rms@gnu.org, emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > > > Why is that coding-system-equal form there?

> > > It was written based on the assumption that a terminal that supports
> > > the UTF-8 encoding can also display the original characters reasonably
> > > enough.  Are you saying that the Linux console lies about that, and
> > > those characters are displayed as hex codes?

> > Well, I think "lies" is a bit strong, here.  The Linux console uses UTF-8
> > characters, but is restricted to an arbitrary selection of 256 of them.
> > An attempt to display any other character on the screen gets \ufffd
> > displayed instead.  Possibly, double-width characters aren't correctly
> > handled, I don't know.  My console is configured to use Latin-1, and it
> > also includes some line graphic characters, but not the punctuation
> > characters used by makeinfo.

> If the console is configured for Latin-1, how come the terminal's
> encoding is UTF-8?

I think I meant that portion of UTF-8 which is based on Latin-1.

> > Those characters were indeed displayed as hex codes, until I applied the
> > patch below.

> Thanks, installed on the release branch.

Thanks!  I look forward to the patch arriving in master.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Akib Azmain Turja]

[-- Attachment #1: Type: text/plain, Size: 2063 bytes --]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> The text-based Linux console is unable to display many Unicode
> characters.  In Emacs 29, display of undisplayable characters has
> changed.  It used to show them with a diamond.  Now it shows the
> unicode character code as hex, preceded by \U.
>
> I find that change quite inconvenient.  It makes the text harder to
> read.  Showing the codes does no good, since I don't know these codes,
> not even for characters I am familiar with.  To find out what
> character a code represents, I have to use C-u C-x =, just as I did
> before.
>
> I last built the sources in May.  Has this changed in a significant
> way since then?
>
> I could not find, in NEWS, anything about this change -- it ought to
> be in NEWS .  Nor a way to go back to the old
> behavior.  Is there a way?  If not, would people please create a way?
>
> As a separate question, do users generally like this change?
> Would it be better to return to the old diamond method as the default?

I don't think this is a very good change.  It's easy to mistake those \U
codes as text.  If someone really want to see the code, they can use M-x
describe-char.  All terminal emulators broke after this change.  Anyway,
I workarounded it by modifying glyphless-char-display with the following
in terminal buffers:

--8<---------------cut here---------------start------------->8---
(set-char-table-extra-slot glyphless-char-display 0 'thin-space)
--8<---------------cut here---------------end--------------->8---

This shows a single space character.  Though ambitious, at least it save
the terminals.

-- 
Akib Azmain Turja

Find me on Mastodon at @akib@hostux.social.

This message is signed by me with my GnuPG key.  Its fingerprint is:

    7001 8CE5 819F 17A3 BBA6  66AF E74F 0EFA 922A E7F5

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Andreas Schwab]

On Aug 27 2022, Akib Azmain Turja wrote:

> I don't think this is a very good change.  It's easy to mistake those \U
> codes as text.

Just like the \123 replacement.  Nothing new here.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: Richard Stallman <rms@gnu.org>,  emacs-devel@gnu.org
> Date: Sat, 27 Aug 2022 09:29:01 +0200
> 
> On Aug 27 2022, Akib Azmain Turja wrote:
> 
> > I don't think this is a very good change.  It's easy to mistake those \U
> > codes as text.
> 
> Just like the \123 replacement.  Nothing new here.

It is also displayed in a distinct face, designed to make this stand
out.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Eli Zaretskii wrote:

> It is also displayed in a distinct face, designed to make
> this stand out.

It looks like it is Bright Yellow (ANSI 3) but it's not of any
Emacs face that my tools detect so that must be handled in
some special way if so ...

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Sat, 27 Aug 2022 19:24:39 +0200
> 
> Eli Zaretskii wrote:
> 
> > It is also displayed in a distinct face, designed to make
> > this stand out.
> 
> It looks like it is Bright Yellow (ANSI 3) but it's not of any
> Emacs face that my tools detect so that must be handled in
> some special way if so ...

There's nothing special about its face.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Eli Zaretskii wrote:

>>> It is also displayed in a distinct face, designed to make
>>> this stand out.
>> 
>> It looks like it is Bright Yellow (ANSI 3) but it's not of
>> any Emacs face that my tools detect so that must be handled
>> in some special way if so ...
>
> There's nothing special about its face.

Try do

  M-x describe-face RET RET

or

  (get-char-property (point) 'face)
  
or

  (get-char-property (point) 'read-cf-name)

nothing!

Unless there is a face already, but then it still don't have
that color ...

Even when no face is reported, it isn't the default color
either (again, it is bright yellow). But I can set to red like
this ...

(defun set-all-faces (fg &optional bg weight)
  (let ((backg (or bg     "black"))
        (wght  (or weight 'normal))
        (faces) )
    (mapatoms (lambda (s)
                (when (facep s)
                  (push (symbol-name s) faces) )))
    (dolist (f faces)
      (set-face-attribute (intern f) nil
                          :foreground fg)
                          :background backg
                          :weight     wght
                          :italic     nil) ))
;; (set-all-faces "red")

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

[-- Attachment #1: Type: text/plain, Size: 388 bytes --]

> Unless there is a face already, but then it still don't have
> that color ...

Here we see in ERC how the good Python bot echos the title of
some YouTube video just mentioned

  https://dataswamp.org/~incal/error/char-face.png

It is the actually in `erc-default-face', all chars, but as
you see there is also a mysterious orange face that just
pretends to be the ERC default face ...


[-- Attachment #2: char-face.png --]
[-- Type: image/png, Size: 7440 bytes --]

[-- Attachment #3: Type: text/plain, Size: 61 bytes --]


-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Sat, 27 Aug 2022 23:50:42 +0200
> 
> Eli Zaretskii wrote:
> 
> > There's nothing special about its face.
> 
> Try do
> 
>   M-x describe-face RET RET
> 
> or
> 
>   (get-char-property (point) 'face)
>   
> or
> 
>   (get-char-property (point) 'read-cf-name)
> 
> nothing!

I don't see the significance of these findings.  Faces in Emacs can be
applied by means other than text or overlay properties.  An example is
how we show control characters (try C-q C-m, for example).

The face is still a normal Emacs face, and can be customized with
"M-x customize-face RET".

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Eli Zaretskii wrote:

>> Try do
>> 
>>   M-x describe-face RET RET
>> 
>> or
>> 
>>   (get-char-property (point) 'face)
>>   
>> or
>> 
>>   (get-char-property (point) 'read-cf-name)
>> 
>> nothing!
>
> I don't see the significance of these findings. Faces in
> Emacs can be applied by means other than text or overlay
> properties.

Okay, so it's the mean then, not the face per se.

> The face is still a normal Emacs face, and can be customized
> with "M-x customize-face RET".

I get `erc-default-face' with that as well ...

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Akib Azmain Turja wrote:

> (set-char-table-extra-slot glyphless-char-display 0
> 'thin-space)
>
> This shows a single space character. Though ambitious, at
> least it save the terminals.

No, that's a lot better already.

But can we set it to something so that it will still be
visible, and still just one char wide?

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Anyway,
  > I workarounded it by modifying glyphless-char-display with the following
  > in terminal buffers:

  > --8<---------------cut here---------------start------------->8---
  > (set-char-table-extra-slot glyphless-char-display 0 'thin-space)
  > --8<---------------cut here---------------end--------------->8---

  > This shows a single space character.  

Could a variant of that display a diamond instead of a space?

Eli wrote

  > One way of verifying this is to type

  >    C-x 8 RET fffd RET

  > in an Emacs session which displays on the Linux console, and see if
  > that shows the diamond-like glyph or the \Unnnn thing.  If it's the
  > latter, it means the console doesn't support that character, and we
  > cannot send its code from Emacs.

but what is so bad about sending a glyph that the terminal can't
display?  What bad results does it cause?

Or how about fixing it by sending an official code for that diamond glyph?


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Sun, 28 Aug 2022 23:36:01 -0400
> 
>   > --8<---------------cut here---------------start------------->8---
>   > (set-char-table-extra-slot glyphless-char-display 0 'thin-space)
>   > --8<---------------cut here---------------end--------------->8---
> 
>   > This shows a single space character.  
> 
> Could a variant of that display a diamond instead of a space?

No, because displaying a non-ASCII character on a text-mode terminal
requires to encode it.  And since the extra-slot of that char-table is
used for fallback display of characters for which the usual encoding
doesn't work, nothing but ASCII characters can be safely put there.

But you can have a different ASCII character there, like '*', for
example (it will be displayed with a special face, to make it stand
out).

> but what is so bad about sending a glyph that the terminal can't
> display?  What bad results does it cause?

It could mess up the display, because sending commands to the terminal
driver also involves sequences of unprintable bytes.  We don't send
unprintable characters to the terminal for that reason.

> Or how about fixing it by sending an official code for that diamond glyph?

I don't see a reason to have a general-purpose feature in Emacs
display like that.  For starters, I don't think terminals in general
use the U+FFFD "diamond" in such cases.  It sounds like your personal
preference, specifically for the Linux console.  In which case I can
suggest a simple customization to do this in your Emacs sessions.  Put
the following into your init file:

  (defun setup-diamonds ()
    (let ((table (or standard-display-table
		     (setq standard-display-table (make-display-table)))))
      (set-char-table-range table '(#x100 . #x10FFFF)
			    (vconcat
			     (list (make-glyph-code #xFFFD 'homoglyph))))))

This arranges for all the characters beyond 255 to be displayed as the
diamond with a special face.  (You can, of course, adjust the range of
characters for which this is done according to your needs, or have
several disjoint ranges instead of just one.)

Would that be good enough for your use patterns?

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > No, because displaying a non-ASCII character on a text-mode terminal
  > requires to encode it.

Maybe the existing mechamisms will insist on encoring it, but I think
it won't be hard to bypass that step.  The user could specify the
encoded sequence to send, by putting that sequence in a variable as a
string or vector.

Then redisplay simply has to send the sequence of characters
from the vector, _as if_ they were the result of encoding something.

  > > but what is so bad about sending a glyph that the terminal can't
  > > display?  What bad results does it cause?

  > It could mess up the display, because sending commands to the terminal
  > driver also involves sequences of unprintable bytes.

In some cases, maybe there is a risk it could do that.  But if the
user uses the feature I propose just above, perse will be responsible
for specifying a sequence that displays as a character.  So that won't
mess up the display.

  > This arranges for all the characters beyond 255 to be displayed as the
  > diamond with a special face.  (You can, of course, adjust the range of
  > characters for which this is done according to your needs, or have
  > several disjoint ranges instead of just one.)

  > Would that be good enough for your use patterns?

With 255 specified, it would not be adequate.  The Linux console handles
the Greek alphabet, the Russian alphabet, and I am not sure what else.

Using several disjoint ranges could do the job, I suppose,
if someone can tell me how to determine what those ranges should be.

Could this be figured out automatically by calling char-displayable-p
on each character code?  Up to what number would that loop have to run?


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Tue, 30 Aug 2022 00:12:02 -0400
> 
>   > No, because displaying a non-ASCII character on a text-mode terminal
>   > requires to encode it.
> 
> Maybe the existing mechamisms will insist on encoring it, but I think
> it won't be hard to bypass that step.  The user could specify the
> encoded sequence to send, by putting that sequence in a variable as a
> string or vector.
> 
> Then redisplay simply has to send the sequence of characters
> from the vector, _as if_ they were the result of encoding something.

This is possible technically, but it isn't safe, because figuring out
the correct encoding of the terminal and producing a suitably-encoded
byte sequence is tricky at best, for users to get right.  And this is
going to misfire if the user ever uses a terminal with a different
encoding.

More importantly, we are talking about a fallback feature, used for
characters that fail to display in any other way we know.  What you
suggest is inappropriate for such fallback methods, which must be 110%
safe.

Using a display-table is therefore a much better way of adapting Emacs
to the specific desires of the user in this case.  For starters, the
character codes emitted by the display-table are encoded as usual,
when written to the terminal.

>   > This arranges for all the characters beyond 255 to be displayed as the
>   > diamond with a special face.  (You can, of course, adjust the range of
>   > characters for which this is done according to your needs, or have
>   > several disjoint ranges instead of just one.)
> 
>   > Would that be good enough for your use patterns?
> 
> With 255 specified, it would not be adequate.  The Linux console handles
> the Greek alphabet, the Russian alphabet, and I am not sure what else.

Then maybe change #x100 to #x530 as the first approximation.

> Using several disjoint ranges could do the job, I suppose,
> if someone can tell me how to determine what those ranges should be.
> 
> Could this be figured out automatically by calling char-displayable-p
> on each character code?

Such a code can be written, and can use char-displayable-p, indeed.

> Up to what number would that loop have to run?

Up to (and including) #x10FFFF.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> Date: Tue, 30 Aug 2022 14:41:49 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> > Using several disjoint ranges could do the job, I suppose,
> > if someone can tell me how to determine what those ranges should be.
> > 
> > Could this be figured out automatically by calling char-displayable-p
> > on each character code?
> 
> Such a code can be written, and can use char-displayable-p, indeed.
> 
> > Up to what number would that loop have to run?
> 
> Up to (and including) #x10FFFF.

That's the theory, of course.  In practice, with the Linux console,
you could only run the loop till #xFFFF, I guess, and just assume the
rest of the characters are not displayable without checking.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Gregory Heytings]

>
> With 255 specified, it would not be adequate.  The Linux console handles 
> the Greek alphabet, the Russian alphabet, and I am not sure what else.
>
> Using several disjoint ranges could do the job, I suppose, if someone 
> can tell me how to determine what those ranges should be.
>

I don't understand why you want to use such a complicated mechanism, when 
a much simpler and better one is available.  Is there a reason why you 
cannot (or do not want to?) use fbterm?

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

This

  (latin1-display-ucs-per-lynx 1)
  (set-char-table-extra-slot glyphless-char-display 0 "%")
  ;; test: (insert-char #x1f3ab)

is good enough for me.

I thought the diamond was better still since then one
immediately saw it was a non-printable char, but hey, that
shouldn't happen all the time anyway, right?

With this it doesn't screw up anything and with the yellow
face I think it'll be pretty obvious or obvious enough what's
going on ...

Not sure what `latin1-display-ucs-per-lynx' is suppose to
do tho?

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Wed, 31 Aug 2022 02:34:04 +0200
> 
> Not sure what `latin1-display-ucs-per-lynx' is suppose to
> do tho?

Try typing some non-ASCII characters, like ⊞ or ⋙ or ｛.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Eli Zaretskii wrote:

>> Not sure what `latin1-display-ucs-per-lynx' is suppose to
>> do tho?
>
> Try typing some non-ASCII characters, like ⊞ or ⋙ or ｛.

Okay ... no, not sure I like that?

Goes against ... something.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Wed, 31 Aug 2022 05:00:09 +0200
> 
> Eli Zaretskii wrote:
> 
> >> Not sure what `latin1-display-ucs-per-lynx' is suppose to
> >> do tho?
> >
> > Try typing some non-ASCII characters, like ⊞ or ⋙ or ｛.
> 
> Okay ... no, not sure I like that?

If you don't like that, just don't use latin1-display-ucs-per-lynx:
it's an optional feature for those who do like the results.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I don't understand why you want to use such a complicated mechanism, when 
  > a much simpler and better one is available.

I don't see how it could be simpler than using a Linux console,
since that is as simple as could be.  I have 8 of them, and to use
one, I just type the character that switches.

When you claim it is simpler, that has to be a misundersanding.

  >   Is there a reason why you 
  > cannot (or do not want to?) use fbterm?

1. It will be different
2. I don't have any idea how to use fbterm.
3. I odn't know what new inconveniences it would cause.

ISTR seeing a discussion about whether fbterm "steals characters".
I did not follow it because I don't use fbterm.

I would like Emacs to handle nondisplayable characters on Linux
consoles the way it did a year ago.  I am sure that can't be hard.



-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Richard Stallman wrote:

>> I don't understand why you want to use such a complicated
>> mechanism, when a much simpler and better one is available.
>
> I don't see how it could be simpler than using a Linux
> console, since that is as simple as could be. I have 8 of
> them, and to use one, I just type the character
> that switches.
>
> When you claim it is simpler, that has to be
> a misundersanding.

Actually it is too simple, for example it doesn't support
scrolling which is why people use tmux on top of it.
But people do that in xterm as well, maybe for other reasons
tho. (?)

>> Is there a reason why you 
>> cannot (or do not want to?) use fbterm?
>
> 1. It will be different
> 2. I don't have any idea how to use fbterm.
> 3. I odn't know what new inconveniences it would cause.

Yes.

> I would like Emacs to handle nondisplayable characters on
> Linux consoles the way it did a year ago. I am sure that
> can't be hard.

Try this, it's not that bad:

  (set-char-table-extra-slot glyphless-char-display 0 "%")

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

> Try this, it's not that bad:
>
>   (set-char-table-extra-slot glyphless-char-display 0 "%")

With the latest change so you don't get the square brackets
for a single char.

GNU Emacs 29.0.50 (build 1, x86_64-pc-linux-gnu, cairo version 1.16.0)
of 2022-08-28 [commit df2f6fb7fc4b79834ae40db8be2ccdc1e4a273f1]

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: eliz@gnu.org, emacs-devel@gnu.org
> Date: Thu, 01 Sep 2022 19:32:01 -0400
> 
>   >   Is there a reason why you 
>   > cannot (or do not want to?) use fbterm?
> 
> 1. It will be different
> 2. I don't have any idea how to use fbterm.
> 3. I odn't know what new inconveniences it would cause.
> 
> ISTR seeing a discussion about whether fbterm "steals characters".
> I did not follow it because I don't use fbterm.
> 
> I would like Emacs to handle nondisplayable characters on Linux
> consoles the way it did a year ago.  I am sure that can't be hard.

I hear you, but I'd suggest that you give fbterm a try.  Who knows,
maybe you will find that you like it.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Akib Azmain Turja]

[-- Attachment #1: Type: text/plain, Size: 1732 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Richard Stallman <rms@gnu.org>
>> Cc: eliz@gnu.org, emacs-devel@gnu.org
>> Date: Thu, 01 Sep 2022 19:32:01 -0400
>> 
>>   >   Is there a reason why you 
>>   > cannot (or do not want to?) use fbterm?
>> 
>> 1. It will be different
>> 2. I don't have any idea how to use fbterm.
>> 3. I odn't know what new inconveniences it would cause.
>> 
>> ISTR seeing a discussion about whether fbterm "steals characters".
>> I did not follow it because I don't use fbterm.
>> 
>> I would like Emacs to handle nondisplayable characters on Linux
>> consoles the way it did a year ago.  I am sure that can't be hard.
>
> I hear you, but I'd suggest that you give fbterm a try.  Who knows,
> maybe you will find that you like it.
>

Or even better, install X and a terminal emulator and maximize the
terminal emulator window.  There are many options, st, XTerm, Kitty,
GNOME Terminal, Konsole, etc.

Or make Emacs your X Window Manager.  There is a GNU ELPA package EXWM
for doing that.

How about GNOME or KDE?  Or even GNUStep?

Even better, emulate Linux TTY on GNOME Terminal running on GNOME
desktop rendered using Sixels on XTerm.

Or the best option, don't use computers at all.  There would be no
software in your life, so you'll also have no proprietary software in
your life.

... OK, just kidding.  Now let's come to the point, why switch to fbterm
when the Linux console works?  The buggy program (looks like it's Emacs
here) should be fixed.

-- 
Akib Azmain Turja

Find me on Mastodon at @akib@hostux.social.

This message is signed by me with my GnuPG key.  Its fingerprint is:

    7001 8CE5 819F 17A3 BBA6  66AF E74F 0EFA 922A E7F5

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> From: Akib Azmain Turja <akib@disroot.org>
> Cc: rms@gnu.org,  gregory@heytings.org,  emacs-devel@gnu.org
> Date: Fri, 02 Sep 2022 18:21:11 +0600
> 
> > I hear you, but I'd suggest that you give fbterm a try.  Who knows,
> > maybe you will find that you like it.
> 
> Or even better, install X and a terminal emulator and maximize the
> terminal emulator window.  There are many options, st, XTerm, Kitty,
> GNOME Terminal, Konsole, etc.
> 
> Or make Emacs your X Window Manager.  There is a GNU ELPA package EXWM
> for doing that.
> 
> How about GNOME or KDE?  Or even GNUStep?
> 
> Even better, emulate Linux TTY on GNOME Terminal running on GNOME
> desktop rendered using Sixels on XTerm.
> 
> Or the best option, don't use computers at all.

Your sarcasm is misplaced.  Suggesting to use an alternative console
terminal program is hardly something that needs to be laughed about.

> Now let's come to the point, why switch to fbterm when the Linux
> console works?

Because fbterm is a better console terminal program?  Because it
supports more characters OOTB?

> The buggy program (looks like it's Emacs here) should be fixed.

If you can explain what are the bugs to which you allude, sure, we do
want to fix bugs in Emacs.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Gregory Heytings]

>
> Or even better, install X and a terminal emulator and maximize the 
> terminal emulator window.
>

If all you need is a terminal, you won't get significantly better results 
with a terminal emulator under X than with fbterm.

>
> Now let's come to the point, why switch to fbterm when the Linux console 
> works?
>

Because it doesn't work?

>
> The buggy program (looks like it's Emacs here) should be fixed.
>

Nonsense.  The "buggy" program here is clearly not Emacs.  The raw Linux 
console is simply too limited, by design, and there's little that can be 
done on the Emacs side to alleviate these limitations.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Or even better, install X and a terminal emulator and maximize the
  > terminal emulator window.  There are many options, st, XTerm, Kitty,
  > GNOME Terminal, Konsole, etc.

Right now, I switch to the Linux console I want using M-F<n>.
Is something that easy possible with an X terminal?  If not,
it would be much worse for me.

Please stop assuming that what's good for you is good for other
people.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Richard Stallman wrote:

>> Or even better, install X and a terminal emulator and
>> maximize the terminal emulator window. There are many
>> options, st, XTerm, Kitty, GNOME Terminal, Konsole, etc.
>
> Right now, I switch to the Linux console I want using
> M-F<n>. Is something that easy possible with an X terminal?
> If not, it would be much worse for me.

Well, yes!

But for the record ...

From X to the VTs one can use chvt(1), first

  $ sudo chmod +s /bin/chvt

then, for example with the cwm window manager

  bind-key M-e "chvt 1"

In the VTs one can also use chvt of course but also the
remap.inc file, for example
  
  alt keycode 38 = Incr_Console # M-l, cycle right
  alt keycode 36 = Decr_Console # M-j, cycle left

  alt keycode 18 = Console_1    # M-e, tty/Emacs
  alt keycode 22 = Console_6    # M-u, X/xterm/tmux/zsh

https://dataswamp.org/~incal/conf/.cwmrc
https://dataswamp.org/~incal/conf/vt/remap.inc

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Akib Azmain Turja]

[-- Attachment #1: Type: text/plain, Size: 1013 bytes --]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > Or even better, install X and a terminal emulator and maximize the
>   > terminal emulator window.  There are many options, st, XTerm, Kitty,
>   > GNOME Terminal, Konsole, etc.
>
> Right now, I switch to the Linux console I want using M-F<n>.
> Is something that easy possible with an X terminal?  If not,
> it would be much worse for me.
>
> Please stop assuming that what's good for you is good for other
> people.

I didn't mean that everything is good for everyone.  I just tried to say
how people make things complex.

-- 
Akib Azmain Turja

Find me on Mastodon at @akib@hostux.social.

This message is signed by me with my GnuPG key.  Its fingerprint is:

    7001 8CE5 819F 17A3 BBA6  66AF E74F 0EFA 922A E7F5

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Gregory Heytings]

>
> Right now, I switch to the Linux console I want using M-F<n>.
>
> Is something that easy possible with an X terminal?
>

Is the possibility to use multiple consoles the only reason you use the 
console?  Or do you have other reasons?

Yes, something that easy (or even easier!) is possible with an X terminal. 
But it's an unusual requirement (most people use a single X terminal), so 
a certain amount of configuration is necessary.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Gregory Heytings wrote:

>> Right now, I switch to the Linux console I want using
>> M-F<n>.
>>
>> Is something that easy possible with an X terminal?
>
> Is the possibility to use multiple consoles the only reason
> you use the console? Or do you have other reasons?

It doesn't matter what _other software_ you or someone else
might prefer or what stuff you may or may not do with
that software.

If we have a problem in Emacs we should fix it in Emacs, not
telling people who experience the problem to use some
other software!

Don't you agree? Or are you suggesting we say Emacs isn't
supported on the Linux VTs because of the existence of other
terminal emulators?

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Gregory Heytings]

>
> If we have a problem in Emacs we should fix it in Emacs
>

Once again, it's not a problem in Emacs, it's an inherent limitation of 
the Linux console.  So it cannot be "fixed in Emacs".  Just like the fact 
that terminal emulators do not support variable width fonts, which cannot 
be "fixed in Emacs".  Or like the fact that (most) terminal emulators do 
not support color emojis, which cannot be "fixed in Emacs".  Or like the 
fact that keys like C-. do not work in terminal emulators, which cannot be 
"fixed in Emacs".  And so forth.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Gregory Heytings wrote:

>> If we have a problem in Emacs we should fix it in Emacs
>
> Once again, it's not a problem in Emacs, it's an inherent
> limitation of the Linux console. So it cannot be "fixed in
> Emacs".

Why can't we have an option what char should be used instead
of those bulky \U01F3A8 strings with the problems they bring
which have been mentioned? So if someone wants the old diamond
that can be arranged by a simple line of configuration?

> Just like the fact that terminal emulators do not support
> variable width fonts, which cannot be "fixed in Emacs".
> Or like the fact that (most) terminal emulators do not
> support color emojis, which cannot be "fixed in Emacs".
> Or like the fact that keys like C-. do not work in terminal
> emulators, which cannot be "fixed in Emacs". And so forth.

-1

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Sat, 03 Sep 2022 11:04:47 +0200
> 
> Why can't we have an option what char should be used instead
> of those bulky \U01F3A8 strings with the problems they bring
> which have been mentioned? So if someone wants the old diamond
> that can be arranged by a simple line of configuration?

Of course, we have such an option with a simple configuration.
Here:

  (set-char-table-range standard-display-table nil
    (vconcat (list (make-glyph-code #xFFFD 'homoglyph))))

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Eli Zaretskii wrote:

>> Why can't we have an option what char should be used
>> instead of those bulky \U01F3A8 strings with the problems
>> they bring which have been mentioned? So if someone wants
>> the old diamond that can be arranged by a simple line
>> of configuration?
>
> Of course, we have such an option with a simple
> configuration. Here:
>
>   (set-char-table-range standard-display-table nil
>     (vconcat (list (make-glyph-code #xFFFD 'homoglyph))))

���� ������ ���

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Sat, 03 Sep 2022 11:42:30 +0200
> 
> Eli Zaretskii wrote:
> 
> >> Why can't we have an option what char should be used
> >> instead of those bulky \U01F3A8 strings with the problems
> >> they bring which have been mentioned? So if someone wants
> >> the old diamond that can be arranged by a simple line
> >> of configuration?
> >
> > Of course, we have such an option with a simple
> > configuration. Here:
> >
> >   (set-char-table-range standard-display-table nil
> >     (vconcat (list (make-glyph-code #xFFFD 'homoglyph))))
> 
> ���� ������ ���

��������

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Akib Azmain Turja]

[-- Attachment #1: Type: text/plain, Size: 949 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Emanuel Berg <incal@dataswamp.org>
>> Date: Sat, 03 Sep 2022 11:42:30 +0200
>> 
>> Eli Zaretskii wrote:
>> 
>> >> Why can't we have an option what char should be used
>> >> instead of those bulky \U01F3A8 strings with the problems
>> >> they bring which have been mentioned? So if someone wants
>> >> the old diamond that can be arranged by a simple line
>> >> of configuration?
>> >
>> > Of course, we have such an option with a simple
>> > configuration. Here:
>> >
>> >   (set-char-table-range standard-display-table nil
>> >     (vconcat (list (make-glyph-code #xFFFD 'homoglyph))))
>> 
>> ���� ������ ���
>
> ��������
>

What?

-- 
Akib Azmain Turja

Find me on Mastodon at @akib@hostux.social.

This message is signed by me with my GnuPG key.  Its fingerprint is:

    7001 8CE5 819F 17A3 BBA6  66AF E74F 0EFA 922A E7F5

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I recommend making the output of standard-display-by-replacement-char
use 

  (let ((tbl (or standard-display-table
		 (setq standard-display-table (make-display-table)))))

to avoid an error.

Perhaps with the current sources standard-display-table is never nil.
But once I added this generated code to my .emacs file, it ran also on
April's build, and got an error here.

I recommend generating a comment at the start of the output:

  ;; This code was output by standard-display-by-replacement-char.

or perhaps showing a sexp to call it with the specific args that were
used to generate this output.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: incal@dataswamp.org, emacs-devel@gnu.org
> Date: Wed, 07 Sep 2022 22:56:39 -0400
> 
> I recommend making the output of standard-display-by-replacement-char
> use 
> 
>   (let ((tbl (or standard-display-table
> 		 (setq standard-display-table (make-display-table)))))
> 
> to avoid an error.

I was under impression that disp-table.el is always loaded in TTY
session.  But it doesn't do any harm to add that, so done.

> I recommend generating a comment at the start of the output:
> 
>   ;; This code was output by standard-display-by-replacement-char.
> 
> or perhaps showing a sexp to call it with the specific args that were
> used to generate this output.

I added just the one line stating the command name.  People who are
interested can read the command's doc string.

Thanks.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Gregory Heytings]

I cannot understand why you used so much of your time and energy to 
restore a half-broken behavior (diamonds instead of undisplayable 
characters), when it would not have taken you more than fifteen minutes to 
get a correct behavior (no undisplayable characters anymore, or very few 
undisplayable characters).

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I cannot understand why you used so much of your time and energy to 
  > restore a half-broken behavior (diamonds instead of undisplayable 
  > characters), when it would not have taken you more than fifteen minutes to 
  > get a correct behavior (no undisplayable characters anymore, or very few 
  > undisplayable characters).

I have to expect that switching to a different kind of terminal
will reveal other problems.
-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Gregory Heytings]

>> I cannot understand why you used so much of your time and energy to 
>> restore a half-broken behavior (diamonds instead of undisplayable 
>> characters), when it would not have taken you more than fifteen minutes 
>> to get a correct behavior (no undisplayable characters anymore, or very 
>> few undisplayable characters).
>
> I have to expect that switching to a different kind of terminal will 
> reveal other problems.
>

That's unlikely (the only known problem is for users who use a mouse, 
which you said you don't), but even if that were the case, that's how 
progress goes: leaving a stable position with known problems to reach a 
position in which some of these problems are solved, while taking the risk 
of facing other yet unknown problems.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Po Lu]

Gregory Heytings <gregory@heytings.org> writes:

> That's unlikely (the only known problem is for users who use a mouse,
> which you said you don't), but even if that were the case, that's how
> progress goes: leaving a stable position with known problems to reach
> a position in which some of these problems are solved, while taking
> the risk of facing other yet unknown problems.

The worst act a software developer can commit is to break things for a
user who is already comfortable.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Po Lu wrote:

>> That's unlikely (the only known problem is for users who
>> use a mouse, which you said you don't), but even if that
>> were the case, that's how progress goes: leaving a stable
>> position with known problems to reach a position in which
>> some of these problems are solved, while taking the risk of
>> facing other yet unknown problems.
>
> The worst act a software developer can commit is to break
> things for a user who is already comfortable.

A wise man once said ...

  <mchehab@redhat.com> wrote:

  % Are you saying that pulseaudio is entering on some weird
  % loop if the returned value is not -EINVAL? That seems a bug
  % at pulseaudio.

  Mauro, SHUT THE FUCK UP!

  It's a bug alright - in the kernel. How long have you been
  a maintainer? And you *still* haven't learnt the first rule
  of kernel maintenance?

  If a change results in user programs breaking, it's a bug in
  the kernel. We never EVER blame the user programs. How hard
  can this be to understand?

https://linuxreviews.org/WE_DO_NOT_BREAK_USERSPACE

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Gregory Heytings]

>
> A wise man once said ...
>

Emacs is not a kernel, so the analogy is broken.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Gregory Heytings wrote:

>> A wise man once said ...
>
> Emacs is not a kernel

Emacs has a KERNEL thread. But scheduling is non-preemptive,
it isn't a POSIX system. It's called ... collaborative
or cooperative?

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Fri, 09 Sep 2022 12:25:21 +0200
> 
> Gregory Heytings wrote:
> 
> >> A wise man once said ...
> >
> > Emacs is not a kernel
> 
> Emacs has a KERNEL thread. But scheduling is non-preemptive,
> it isn't a POSIX system. It's called ... collaborative
> or cooperative?

Please stop sending off-topic messages to this list.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Fri, 09 Sep 2022 12:08:07 +0200
> 
> Po Lu wrote:
> 
> > The worst act a software developer can commit is to break
> > things for a user who is already comfortable.
> 
> A wise man once said ...

"Wise man (n): Someone who agrees with Emanuel Berg."

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Po Lu wrote:

>> That's unlikely (the only known problem is for users who
>> use a mouse, which you said you don't), but even if that
>> were the case, that's how progress goes: leaving a stable
>> position with known problems to reach a position in which
>> some of these problems are solved, while taking the risk of
>> facing other yet unknown problems.
>
> The worst act a software developer can commit is to break
> things for a user who is already comfortable.

I once as a kid saw a magician performing on the street, he
took a bill from one guy in the audience and pulled it in two
halfs. The guy got angry and instinctively reached for it but
then it was gone and he was totally confused, everyone laughed
thinking what a great performer. But when he made the bill
reappear and gave it back to the dude it was *still* in two
halfs. ???

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Fri, 09 Sep 2022 12:13:21 +0200
> 
> > The worst act a software developer can commit is to break
> > things for a user who is already comfortable.
> 
> I once as a kid saw a magician performing on the street, he
> took a bill from one guy in the audience and pulled it in two
> halfs. The guy got angry and instinctively reached for it but
> then it was gone and he was totally confused, everyone laughed
> thinking what a great performer. But when he made the bill
> reappear and gave it back to the dude it was *still* in two
> halfs. ???

Please stop those off-topic posts on this list.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: Richard Stallman <rms@gnu.org>,  emacs-devel@gnu.org
> Date: Fri, 09 Sep 2022 16:10:07 +0800
> 
> Gregory Heytings <gregory@heytings.org> writes:
> 
> > That's unlikely (the only known problem is for users who use a mouse,
> > which you said you don't), but even if that were the case, that's how
> > progress goes: leaving a stable position with known problems to reach
> > a position in which some of these problems are solved, while taking
> > the risk of facing other yet unknown problems.
> 
> The worst act a software developer can commit is to break things for a
> user who is already comfortable.

If a user is comfortable with a side effect of a bug, breaking that is
inevitable, as long as we want to fix a bug.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Eli Zaretskii wrote:

>>> That's unlikely (the only known problem is for users who
>>> use a mouse, which you said you don't), but even if that
>>> were the case, that's how progress goes: leaving a stable
>>> position with known problems to reach a position in which
>>> some of these problems are solved, while taking the risk
>>> of facing other yet unknown problems.
>> 
>> The worst act a software developer can commit is to break
>> things for a user who is already comfortable.
>
> If a user is comfortable with a side effect of a bug,
> breaking that is inevitable, as long as we want to fix
> a bug.

Still, one shouldn't burn down the house to kill the rats ...

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Gregory Heytings]

>
> The worst act a software developer can commit is to break things for a 
> user who is already comfortable.
>

Please define "comfortable".

Here's a (real) story.  A program for a certain task is awfully slow, it 
takes ten to fifteen minutes to finish that task, and the computer on 
which it runs becomes almost unusable during that time.  It is made 
faster, and now takes at most a few seconds for that same task.  Its end 
users are not at all happy with that improvement: they were comfortable 
with the slow program, because it was a good reason/excuse to take yet 
another break in their workday.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Po Lu]

Gregory Heytings <gregory@heytings.org> writes:

> Please define "comfortable".

In this case, with diamonds being displayed instead of escape sequences.
But that also applies to anything, so there is nothing to define here.

We should never change behavior in a way that makes a previously
comfortable user become uncomfortable.  Especially when the procedure to
undo the effects of the change is so convoluted that it necessitates a
long thread on emacs-devel.

> Here's a (real) story.  A program for a certain task is awfully slow,
> it takes ten to fifteen minutes to finish that task, and the computer
> on which it runs becomes almost unusable during that time.  It is made
> faster, and now takes at most a few seconds for that same task.  Its
> end users are not at all happy with that improvement: they were
> comfortable with the slow program, because it was a good reason/excuse
> to take yet another break in their workday.

I don't see why you have to conflate clearly unreasonable "comfort" with
reasonable behavior, such as letting the Linux console display
undisplayable characters as diamonds.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Gregory Heytings]

>
> We should never change behavior in a way that makes a previously 
> comfortable user become uncomfortable.
>

I fear that's not a useful rule, because users develop Stockholm syndromes 
with the limitations of the software they use.

>
> I don't see why you have to conflate clearly unreasonable "comfort" with 
> reasonable behavior
>

The point is that everyone has their own definition/views of what is 
"reasonable" and "comfortable".

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Gregory Heytings wrote:

>> We should never change behavior in a way that makes
>> a previously comfortable user become uncomfortable.
>
> I fear that's not a useful rule, because users develop
> Stockholm syndromes with the limitations of the software
> they use.

We are suggesting a more advanced Emacs where this would be
configurable - as easy as anything else - to get the desired
behavior, including the diamonds if that what (some)
people want.

If the bug did it, it's possible for Emacs to do as well!

>> I don't see why you have to conflate clearly unreasonable
>> "comfort" with reasonable behavior
>
> The point is that everyone has their own definition/views of
> what is "reasonable" and "comfortable".

Everyone that uses it have (or could have at least) their own
views what makes up a good Emacs. This multitude is the
strength of the FOSS world and just in general, what an awful
world it would be if everyone was the same.

We are not against you using or pushing for your software, on
the contrary. Maybe it is better, as you say. But that's not
the issue here, really.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: Richard Stallman <rms@gnu.org>,  emacs-devel@gnu.org
> Date: Fri, 09 Sep 2022 21:47:12 +0800
> 
> Gregory Heytings <gregory@heytings.org> writes:
> 
> > Please define "comfortable".
> 
> In this case, with diamonds being displayed instead of escape sequences.

I already explained why diamonds are _less_ comfortable than hex
escapes.

Of course, individual users can legitimately disagree, so we now have
a new feature specifically for those users.

I see nothing more to argue about here.  This list has high enough
noise level even without that.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Po Lu wrote:

> In this case, with diamonds being displayed instead of
> escape sequences. But that also applies to anything, so
> there is nothing to define here.
>
> We should never change behavior in a way that makes
> a previously comfortable user become uncomfortable.

100% correct.

And if that happens anyway and a user says so, the equivalence
of "I thought it was better before", we should never say "use
program X instead", that should instead be handled within
Emacs since that's what caused the change.

The bug should of course not be re-introduced, but its
solution should be transparent so the user experience will be,
if desired, unchanged.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Gregory Heytings wrote:

> Here's a (real) story. A program for a certain task is
> awfully slow, it takes ten to fifteen minutes to finish that
> task, and the computer on which it runs becomes almost
> unusable during that time. It is made faster, and now takes
> at most a few seconds for that same task. Its end users are
> not at all happy with that improvement: they were
> comfortable with the slow program, because it was a good
> reason/excuse to take yet another break in their workday.

???

What does this refer to, did the method that showed the
diamonds in the Linux VTs do that for certain chars?

If so, I never experienced it, but good that you fixed
it obviously.

But please explain why is it so difficult to setup the mere
_interface_ aspect, which is the representation of
undisplayable chars, so that it can be configured to non-ASCII
chars as well?

  (setq undisplayable-char-char CHAR)

Char?

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Gregory Heytings wrote:

> That's unlikely (the only known problem is for users who use
> a mouse, which you said you don't), but even if that were
> the case, that's how progress goes: leaving a stable
> position with known problems ...

But the change wasn't in the Linux VTs but in Emacs. So maybe
we should leave Emacs then?

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Yuri Khan]

On Sat, 3 Sept 2022 at 13:56, Gregory Heytings <gregory@heytings.org> wrote:

> Yes, something that easy (or even easier!) is possible with an X terminal.
> But it's an unusual requirement (most people use a single X terminal)

[citation needed].

People use multiple workspaces/virtual desktops on a single X display,
multiple windows on a workspace, some of those windows are terminal
emulators, some terminal emulators support multiple tabs and/or panes,
and people use those.

So:

* One can switch from an X display to a Linux TTY with
Ctrl+Alt+F[1-6], and back to X with Alt+F7 (usually).
* One can switch workspaces with a mouse click or a configurable set of keys.
* One can cycle windows in a single workspace with [Shift+]Alt+Tab or
a mouse click.
* One can cycle tabs in a single window with an application-specific
shortcut or a mouse click.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 1236 bytes --]


>> Yes, something that easy (or even easier!) is possible with an X 
>> terminal. But it's an unusual requirement (most people use a single X 
>> terminal)
>
> [citation needed].
>
> People use multiple workspaces/virtual desktops on a single X display, 
> multiple windows on a workspace, some of those windows are terminal 
> emulators, some terminal emulators support multiple tabs and/or panes, 
> and people use those.
>

[citation needed] 😉

>
> So:
>
> * One can switch from an X display to a Linux TTY with Ctrl+Alt+F[1-6], 
> and back to X with Alt+F7 (usually).
>
> * One can switch workspaces with a mouse click or a configurable set of 
> keys.
>
> * One can cycle windows in a single workspace with [Shift+]Alt+Tab or a 
> mouse click.
>
> * One can cycle tabs in a single window with an application-specific 
> shortcut or a mouse click.
>

All that is nice, but it's not what Richard asked AFAIU.  He comes from a 
multiple terminal background, each of which is completely separated from 
the other ones, and what comes closest to that is to have multiple X 
servers running in parallel, and switching between them with, say, M-F<n>. 
At least, that's how I understood his question.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Gregory Heytings wrote:

> He comes from a multiple terminal background, each of which
> is completely separated from the other ones, and what comes
> closest to that is to have multiple X servers [...]

?

Why is that closer than the Linux VTs or the BSD console for
that matter?

And How do we measure the distance from a "multiple terminal
background" even? It feels pretty close to me TBH ...

  https://www.youtube.com/watch?v=0hRcPZ8kN8I

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Yuri Khan wrote:

>> Yes, something that easy (or even easier!) is possible with
>> an X terminal. But it's an unusual requirement (most people
>> use a single X terminal)
>
> [citation needed].
>
> People use multiple workspaces/virtual desktops on a single
> X display, multiple windows on a workspace, some of those
> windows are terminal emulators, some terminal emulators
> support multiple tabs and/or panes, and people use those.

Also, most people use a terminal multiplexer (tmux most
often) on top of whatever terminal emulator they use ...

> One can switch from an X display to a Linux TTY with
> Ctrl+Alt+F[1-6], and back to X with Alt+F7 (usually).

As for keys, keys in X that is, that can be setup with
X window managers (e.g., cwm) or X software specifically for
that (xbindkeys).

But: do most people do that?

Or is it just us console Emacs users?

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Yuri Khan]

On Sat, 3 Sept 2022 at 16:32, Emanuel Berg <incal@dataswamp.org> wrote:

> > One can switch from an X display to a Linux TTY with
> > Ctrl+Alt+F[1-6], and back to X with Alt+F7 (usually).
>
> As for keys, keys in X that is, that can be setup with
> X window managers (e.g., cwm) or X software specifically for
> that (xbindkeys).
>
> But: do most people do that?

Most people™ only use the Linux console to fix X when it breaks.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Yuri Khan wrote:

>>> One can switch from an X display to a Linux TTY with
>>> Ctrl+Alt+F[1-6], and back to X with Alt+F7 (usually).
>>
>> As for keys, keys in X that is, that can be setup with
>> X window managers (e.g., cwm) or X software specifically
>> for that (xbindkeys).
>>
>> But: do most people do that?
>
> Most people™ only use the Linux console to fix X when
> it breaks.

I know right? Why aren't they using fbterm like any sensible
person would?! It's so stupid ...

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

>>>> One can switch from an X display to a Linux TTY with
>>>> Ctrl+Alt+F[1-6], and back to X with Alt+F7 (usually).
>>>
>>> As for keys, keys in X that is, that can be setup with
>>> X window managers (e.g., cwm) or X software specifically
>>> for that (xbindkeys).
>>>
>>> But: do most people do that?
>>
>> Most people™ only use the Linux console to fix X when
>> it breaks.
>
> I know right? Why aren't they using fbterm like any sensible
> person would?! It's so stupid ...

Hey, this whole discussion and attitude just gave me an idea!

Why don't we turn the whole thing around, and instead suggest
that EVERYONE, all Emacs users, use the Linux console!
And that would be the ONLY way to use Emacs!

Just imagine what that would do, we could drop support for not
just X, there would be no more porting Emacs to Windows or Mac
OS or even the BSDs, there would just be no end to all the
problems that we could just ignore and instead say, "Stop
whining you techno-imbecile, and use the Linux console like
everyone else goddammit"

@&%#$!

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Akib Azmain Turja]

[-- Attachment #1: Type: text/plain, Size: 1403 bytes --]

Emanuel Berg <incal@dataswamp.org> writes:

>>>>> One can switch from an X display to a Linux TTY with
>>>>> Ctrl+Alt+F[1-6], and back to X with Alt+F7 (usually).
>>>>
>>>> As for keys, keys in X that is, that can be setup with
>>>> X window managers (e.g., cwm) or X software specifically
>>>> for that (xbindkeys).
>>>>
>>>> But: do most people do that?
>>>
>>> Most people™ only use the Linux console to fix X when
>>> it breaks.
>>
>> I know right? Why aren't they using fbterm like any sensible
>> person would?! It's so stupid ...
>
> Hey, this whole discussion and attitude just gave me an idea!
>
> Why don't we turn the whole thing around, and instead suggest
> that EVERYONE, all Emacs users, use the Linux console!
> And that would be the ONLY way to use Emacs!
>
> Just imagine what that would do, we could drop support for not
> just X, there would be no more porting Emacs to Windows or Mac
> OS or even the BSDs, there would just be no end to all the
> problems that we could just ignore and instead say, "Stop
> whining you techno-imbecile, and use the Linux console like
> everyone else goddammit"
>
> @&%#$!

That would degrade Emacs to Vi level.

-- 
Akib Azmain Turja

Find me on Mastodon at @akib@hostux.social.

This message is signed by me with my GnuPG key.  Its fingerprint is:

    7001 8CE5 819F 17A3 BBA6  66AF E74F 0EFA 922A E7F5

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Akib Azmain Turja wrote:

> That would degrade Emacs to Vi level.

Here are the top 25 channels right now on Libera. As you see
#emacs (position 19) just barely beats vim (22) and is
severely under-gunned by neovim (10).

#matrix                2068
#linux                 2050
#libera                1673
#archlinux-cn          1654
#python                1600
#fedora                1493
#archlinux             1402
#ubuntu                1080
#archlinux-cn-offtopic 1054
#neovim                 958
#networking             900
#ansible                897
##rust                  871
#c                      842
#git                    830
#postgresql             820
#security               804
#bash                   801
#emacs                  798
#gentoo                 796
#debian                 760
#vim                    735
##programming           732
#go-nuts                723
#haskell                691

-- 
underground experts united
https://dataswamp.org/~incal

enum (was: Re: Display of undisplayable characters: \U01F3A8 instead of diamond)

[Emanuel Berg]

>> That would degrade Emacs to Vi level.
>
> Here are the top 25 channels right now on Libera. As you see
> #emacs (position 19) just barely beats vim (22) and is
> severely under-gunned by neovim (10).
>
> #matrix                2068
> #linux                 2050
> #libera                1673
> [...]

Or, using

;;; -*- lexical-binding: t -*-
;;
;; this file:
;;   https://dataswamp.org/~incal/emacs-init/enum.el

(require 'cl-lib)
(require 'subr-x)

(defun enum (beg end)
  (interactive "r")
  (goto-char beg)
  (let*((lines   (count-lines beg end))
        (pad-len (length (number-to-string lines))) )
    (cl-loop
     for line from 1 to lines
     do (goto-char (line-beginning-position))
     (insert
      (format "%s. " (string-pad (number-to-string line) pad-len nil t)))
     (forward-line) )))

 1. #matrix                2068
 2. #linux                 2050
 3. #libera                1673
 4. #archlinux-cn          1654
 5. #python                1600
 6. #fedora                1493
 7. #archlinux             1402
 8. #ubuntu                1080
 9. #archlinux-cn-offtopic 1054
10. #neovim                 958
11. #networking             900
12. #ansible                897
13. ##rust                  871
14. #c                      842
15. #git                    830
16. #postgresql             820
17. #security               804
18. #bash                   801
19. #emacs                  798
20. #gentoo                 796
21. #debian                 760
22. #vim                    735
23. ##programming           732
24. #go-nuts                723
25. #haskell                691

-- 
underground experts united
https://dataswamp.org/~incal

Re: enum

[Emanuel Berg]

;;; -*- lexical-binding: t -*-
;;
;; this file:
;;   https://dataswamp.org/~incal/emacs-init/enum.el

(require 'cl-lib)
(require 'subr-x)

(defun enum (&optional beg end suf)
  "Enumerate each line counting from one from BEG to END.
Use SUF as a suffix to the digits inserted.
BEG defaults to the beginning of the buffer,
END to the end of it; and
SUF to \". \""
  (interactive
   `(,@(if (region-active-p)
           (list (region-beginning) (region-end))
         (list nil nil) )
     ,(when current-prefix-arg
        (read-string "suffix: ") )))
  (or beg (setq beg (point-min)))
  (or end (setq end (point-max)))
  (or suf (setq suf ". "))
  (goto-char beg)
  (let*((lines   (count-lines beg end))
        (pad-len (length (number-to-string lines))) )
    (cl-loop
      for line from 1 to lines do
      (goto-char (line-beginning-position))
      (insert
       (format "%s%s" (string-pad (number-to-string line) pad-len nil t) suf) )
      (forward-line) )))

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Is the possibility to use multiple consoles the only reason you use the 
  > console?  Or do you have other reasons?

Other reasons include:

* On the Linux console, there is no mouse pointer, so I don't have to
worry that I will accidentally move it.

* Ehatever I run is full screen, automatically.  No need to adjust the
size or position of anything.
-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Richard Stallman wrote:

>> Is the possibility to use multiple consoles the only reason
>> you use the console? Or do you have other reasons?
>
> Other reasons include:
>
> * On the Linux console, there is no mouse pointer, so
> I don't have to worry that I will accidentally move it.
>
> * Ehatever I run is full screen, automatically. No need to
> adjust the size or position of anything.

Indeed, it is also very fast and easy to setup. It is stable,
moving configuration files between computers and Linux distros
never brakes anything.

If you are a keyboard oriented person the keyboard
configuration is very good and and easy (e.g., the compose
key, navigating between VTs), and after it's done it works
ever after and uniformly across applications.

https://dataswamp.org/~incal/conf/vt/console-setup
https://dataswamp.org/~incal/conf/vt/keyboard
https://dataswamp.org/~incal/conf/vt/remap.inc

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I hear you, but I'd suggest that you give fbterm a try.  Who knows,
  > maybe you will find that you like it.

I will try to do that some day in my copious spare time ;-{.
I found that apt-get installed it, that's one step with no difficulty.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Gregory Heytings]

>
> I don't see how it could be simpler than using a Linux console, since 
> that is as simple as could be.  I have 8 of them, and to use one, I just 
> type the character that switches.
>

That won't change.

>
> ISTR seeing a discussion about whether fbterm "steals characters".
>

It doesn't, when configured properly.

>> Is there a reason why you cannot (or do not want to?) use fbterm?
>
> 1. It will be different
> 2. I don't have any idea how to use fbterm.
> 3. I odn't know what new inconveniences it would cause.
>

How to install and configure it is documented in efaq.texi (on latest 
master and emacs-28), around line 3050.  Doing that will take you less 
time that you've already spent on trying to find a way to change the way 
Emacs handles nondisplayable characters on Linux consoles.  If you need 
help, feel free to ask, either here or privately.

What you'll get is:

1. Full UTF-8 support, i.e. no "undisplayable" characters anymore.

2. 256 colors instead of 8.

3. Modern fonts instead of pixelated ones.

There are no known inconveniences.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Alan Mackenzie]

Hello, Gregory.

On Fri, Sep 02, 2022 at 07:28:51 +0000, Gregory Heytings wrote:

> >
> > I don't see how it could be simpler than using a Linux console, since 
> > that is as simple as could be.  I have 8 of them, and to use one, I just 
> > type the character that switches.
> >

> That won't change.

That is true.

> >
> > ISTR seeing a discussion about whether fbterm "steals characters".
> >

> It doesn't, when configured properly.

Though it has to be said that this configuration is actually disabling
part of fbterm.  It is a hack.

> >> Is there a reason why you cannot (or do not want to?) use fbterm?

> > 1. It will be different
> > 2. I don't have any idea how to use fbterm.
> > 3. I odn't know what new inconveniences it would cause.

> How to install and configure it is documented in efaq.texi (on latest 
> master and emacs-28), around line 3050.  Doing that will take you less 
> time that you've already spent on trying to find a way to change the way 
> Emacs handles nondisplayable characters on Linux consoles.  If you need 
> help, feel free to ask, either here or privately.

> What you'll get is:

> 1. Full UTF-8 support, i.e. no "undisplayable" characters anymore.

Yes.

> 2. 256 colors instead of 8.

Well, actually instead of 16, since all the colours come in a normal and
a bright version.  But these 256 colours will be getting used, and that
means work configuring faces that are not satisfactory to the user in
the 256 colour version.

> 3. Modern fonts instead of pixelated ones.

Here's the rub.  The "pixelated" fonts were designed to be used in, say,
an 8x16 matrix, and work very well indeed.  The "modern" fonts were
designed to be used in X, and work less well in fbterm on a pixelated
grid.

All of the "modern" fonts available for fbterm on my machine are
"spidery" single pixel thick fonts.  On the lat1-16 font I've been using
in consoles for decades, the lines are two pixels thick.  I find this
much more readable.

On some of the fonts I tried, full-frame windows were only 61 rows high
rather than the normal 65.  Some of them were so dark as to be
effectively unreadable.  None of them were really satisfactory.

> There are no known inconveniences.

There are several.  As mentioned above, you have to configure faces,
possibly a lot of them.  There are all the problems with fonts.  You've
got to remember to hack the fbterm binary each time a new version of
fbterm comes out (unless you've got a system like Gentoo, and arrange
for this to happen automatically).

There will likely be people who use an unhacked version of fbterm, with
its key sequences, and they cannot run Emacs on such.  They will
possibly get confused if they need both hacked and unhacked versions of
fbterm together on one machine.

I had problems with fbterm working on my ISP's server over ssh.  The
program at the far end didn't recognise TERM=fbterm and gave up.  I had
to call ssh as $ TERM=linux ssh .... to get it to work.

There are lots of itty-bitty things you've got to get used to to use
Emacs in fbterm, and it is more inconvenient than the plain Linux
console where you need merely to log on and type $ emacs.

Likely there will be more little problems with fbterm turning up.

I will probably continue using the plain Linux console, for all these
reasons.

I agree with Richard that there should be an option for displaying
characters outside of the current font as "�" rather than "\u2022".  It
is a feature of the Linux console that all such characters are displayed
that way.  There is no possibility of any character causing an undefined
action.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> Date: Fri, 2 Sep 2022 11:12:32 +0000
> Cc: Richard Stallman <rms@gnu.org>, eliz@gnu.org, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> I will probably continue using the plain Linux console, for all these
> reasons.

So you don't like fbterm.  That doesn't mean Richard won't.

The fact that fbterm uses hacks and kludges shouldn't be significant,
since many a software on GNU/Linux does.  Including Emacs, btw.  The
question is: does the user like the end result?

> I agree with Richard that there should be an option for displaying
> characters outside of the current font as "�" rather than "\u2022".

I already explained how to set that up, so what exactly does the
"should be" part here want to say?  There is already such a way in
Emacs.  Just use it, if that's what you want.

> It is a feature of the Linux console that all such characters are
> displayed that way.

That feature provides no way of using it via standard terminfo
interfaces.  Heck, even the way to ask the console which characters it
supports is a kludge specific to that console!

> There is no possibility of any character causing an undefined
> action.

Famous last words.  (And we don't send characters to the console, we
send bytes.)

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Alan Mackenzie]

Hello, Eli.

On Fri, Sep 02, 2022 at 14:29:12 +0300, Eli Zaretskii wrote:
> > Date: Fri, 2 Sep 2022 11:12:32 +0000
> > Cc: Richard Stallman <rms@gnu.org>, eliz@gnu.org, emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

[ .... ]

> > I agree with Richard that there should be an option for displaying
> > characters outside of the current font as "�" rather than "\u2022".

> I already explained how to set that up, so what exactly does the
> "should be" part here want to say?

The way you explained involves hacking Lisp and finding out precise
character ranges.  That's a lot different from being able just to set an
option.

> There is already such a way in Emacs.  Just use it, if that's what you
> want.

There appears to be no easy way to get the old behaviour back, where
characters undisplayable on the console got displayed with \ufffd
instead.  You've characterised this old behaviour as a bug, but in the
preferences of two actual Linux console users (Richard and myself), the
solution is not better than the bug.

> > It is a feature of the Linux console that all such characters are
> > displayed that way.

> That feature provides no way of using it via standard terminfo
> interfaces.  Heck, even the way to ask the console which characters it
> supports is a kludge specific to that console!

Yes, I remember looking into that to try to find a way.  Somebody
brighter than me clearly managed it.  :-)

> > There is no possibility of any character causing an undefined
> > action.

> Famous last words.  (And we don't send characters to the console, we
> send bytes.)

OK, bytes.  Have there been any untoward effects reported, by sending
valid Unicode strings of bytes to the Linux console?

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Gregory Heytings]

>
> There appears to be no easy way to get the old behaviour back, where 
> characters undisplayable on the console got displayed with \ufffd 
> instead.
>

I have good news for you, there's a way: use fbterm.  Characters that are 
not displayable with the fonts you selected get displayed in with \ufffd 
instead.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> Date: Fri, 2 Sep 2022 13:39:51 +0000
> Cc: gregory@heytings.org, rms@gnu.org, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > > I agree with Richard that there should be an option for displaying
> > > characters outside of the current font as "�" rather than "\u2022".
> 
> > I already explained how to set that up, so what exactly does the
> > "should be" part here want to say?
> 
> The way you explained involves hacking Lisp

It involves writing some Lisp code and running it on your system,
yes.  How is that a problem?

> and finding out precise character ranges.  That's a lot different
> from being able just to set an option.

How can Emacs know which characters does your Linux console actually
support, out of all the Unicode range?  And how can Emacs know which
ones of those you want to see as their ASCII "emulations" (per
latin1-disp.el) and which you want to see as U+FFFD replacements?

So yes, this requires you, the user, to tell Emacs which ones you want
to see in what manner.  There are a lot of Unicode characters, so it
could be a large job, if the characters actually supported by the
console are all in distinct Unicode blocks.  (But if you only want to
see Latin-1 characters, I've shown in this thread a one-liner to do
that.  I guess you will reject that as well.)

> > There is already such a way in Emacs.  Just use it, if that's what you
> > want.
> 
> There appears to be no easy way to get the old behaviour back, where
> characters undisplayable on the console got displayed with \ufffd
> instead.  You've characterised this old behaviour as a bug, but in the
> preferences of two actual Linux console users (Richard and myself), the
> solution is not better than the bug.

I don't understand why you are consistently rejecting every solution
that was suggested.  Including, amazingly enough, a way to actually
produce the same "diamond" glyphs on display under the same
circumstances.  Since when do veteran Emacs hackers such as you and
Richard consider some Lisp coding a problem so grave as to justify
flatly rejecting it?

Please understand: you are _really_ using Emacs on a terminal that is
unsuitable for it.  You _should_ expect problems.  And when solutions
are pointed out that are just a few lines of Lisp away, I'd expect you
to embrace them, not flatly reject them.

> Have there been any untoward effects reported, by sending valid
> Unicode strings of bytes to the Linux console?

Sorry, but I don't care.  We will not send unsupported byte sequences
to any terminal, not if I have anything to say about that.  That's
simply bad engineering.  We have plenty of better, safer solutions.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[chad]

[-- Attachment #1: Type: text/plain, Size: 478 bytes --]

I understand (academically) that once someone has adapted themselves to a
particular set of bugs, shortcomings, and limitations, then that adaptation
tends to produce a feeling of home and safety, but it's hard not to see the
current emacs-devel conversation as a real-life recreation of this bit from
XKCD:

  https://xkcd.com/1172/

I'm sharing this here in the hopes that it adds some levity to someone
else's day. Thanks (again) to you all for making emacs (better).

~Chad

[-- Attachment #2: Type: text/html, Size: 677 bytes --]

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 71 bytes --]


> 
> https://xkcd.com/1172/
>

😃 I had that XKCD in mind, too!

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

chad wrote:

> I understand (academically) that once someone has adapted
> themselves to a particular set of bugs, shortcomings, and
> limitations

But here the bug was actually better from our perspective ...

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Gregory Heytings]

>> I understand (academically) that once someone has adapted themselves to 
>> a particular set of bugs, shortcomings, and limitations
>
> But here the bug was actually better from our perspective ...
>

In which case you still have at least two possibilities:

1. Use an earlier version of Emacs in which that bug is present.

2. Build your own patched Emacs in which that bug is restored.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Gregory Heytings wrote:

>>> I understand (academically) that once someone has adapted
>>> themselves to a particular set of bugs, shortcomings, and
>>> limitations
>>
>> But here the bug was actually better from our perspective ...
>
> In which case you still have at least two possibilities:
>
> 1. Use an earlier version of Emacs in which that bug
> is present.
>
> 2. Build your own patched Emacs in which that bug
> is restored.

With this

  (set-char-table-extra-slot glyphless-char-display 0 "%")

and the latest change to not get the brackets for a single
char its pretty equivalent but the diamond was slightly better
since the percent sign (%) is a common char that is used not
all the time maybe but often enough. The with the yellow face
it isn't a problem, really.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[chad]

[-- Attachment #1: Type: text/plain, Size: 1549 bytes --]

On Fri, Sep 2, 2022 at 3:25 PM Emanuel Berg <incal@dataswamp.org> wrote:

> > I understand (academically) that once someone has adapted
> > themselves to a particular set of bugs, shortcomings, and
> > limitations
>
> But here the bug was actually better from our perspective ...
>

The bug was better in that the undefined behavior from sending known-bad
data to the console hasn't yet caused you trouble that you've identified.
Everyone (who's looked at the code) acknowledges that it was doing the
wrong thing. The fact that the bug didn't hurt you and you got used to it
is exactly what I meant by "adapted themselves".

What the other user (RMS, in this case) _wanted_ to do was to use a console
(not window system) emacs to look at a range of characters that extends
beyond ASCII. The specific implementations he was using did that right some
of the time and wrong some of the time. When it was wrong, it failed in a
certain way. He adapted himself to that failure.

The alternative that emacs-devel is trying to establish (via experiments,
etc/PROBLEMS changes, and perhaps code patches) will make the system fail
less often -- that is, do what the user wants more often. The argument in
question is basically "Don't make the software do what the user wants more
often if it changes away from the bugs that are already familiar to me",
especially when that argument is expressed *in the middle of fixing the
problem*, as a discouragement from fixing the problem for all users, then
we've arrived at "That's horrifying." ala XKCD 1172.

~Chad

[-- Attachment #2: Type: text/html, Size: 2039 bytes --]

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

chad wrote:

> The bug was better in that the undefined behavior from
> sending known-bad data to the console hasn't yet caused you
> trouble that you've identified. Everyone (who's looked at
> the code) acknowledges that it was doing the wrong thing.
> The fact that the bug didn't hurt you and you got used to it
> is exactly what I meant by "adapted themselves".

-1

> What the other user (RMS, in this case) _wanted_ to do was
> to use a console (not window system) emacs to look at
> a range of characters that extends beyond ASCII.
> The specific implementations he was using did that right
> some of the time and wrong some of the time. When it was
> wrong, it failed in a certain way. He adapted himself to
> that failure.

-2

> The alternative that emacs-devel is trying to establish (via
> experiments, etc/PROBLEMS changes, and perhaps code patches)
> will make the system fail less often -- that is, do what the
> user wants more often. The argument in question is basically
> "Don't make the software do what the user wants more often
> if it changes away from the bugs that are already familiar
> to me", especially when that argument is expressed *in the
> middle of fixing the problem*, as a discouragement from
> fixing the problem for all users, then we've arrived at
> "That's horrifying." ala XKCD 1172.

-3

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Alan Mackenzie]

Hello, Chad.

On Fri, Sep 02, 2022 at 16:39:29 -0400, chad wrote:
> On Fri, Sep 2, 2022 at 3:25 PM Emanuel Berg <incal@dataswamp.org> wrote:

> > > I understand (academically) that once someone has adapted
> > > themselves to a particular set of bugs, shortcomings, and
> > > limitations

> > But here the bug was actually better from our perspective ...


> The bug was better in that the undefined behavior from sending known-bad
> data to the console hasn't yet caused you trouble that you've identified.

We weren't talking about bad data.  We were talking about sending valid
UTF-8 sequences to the Linux console.  This console is programmed to
handle these correctly, whether or not there is a matching font entry
for that UTF-8 sequence.

> Everyone (who's looked at the code) acknowledges that it was doing the
> wrong thing.

This is untrue.  I've looked at the code, and consider it was doing the
right thing.  In fact, I've perused the Linux console code too, and
vaguely remember its handling of glyph-less outputs.

> The fact that the bug didn't hurt you and you got used to it is
> exactly what I meant by "adapted themselves".

What bug?  I can't see a bug.  But I've already half-volunteered to work
on it.  Maybe other console types don't handle random UTF-8 byte
sequences well.  I don't know.  But Linux does.

> What the other user (RMS, in this case) _wanted_ to do was to use a console
> (not window system) emacs to look at a range of characters that extends
> beyond ASCII.

Of course.  So do I.  The Linux console is designed and implemented to
support characters beyond ASCII.  It's restriction is to 256 distinct
character glyphs.

> The specific implementations he was using did that right some of the
> time and wrong some of the time.

Too many pronouns.  What are you saying?  To what are you referring by
"specific implementations"?  Implementations of what?  You're saying
these implementations did "that" right some of the time.  What does
"that" refer to?  What do you mean by wrong?  

> When it was wrong, it failed in a certain way. He adapted himself to
> that failure.

I don't see any failure.  Richard's complaint was that characters
without glyphs were getting displayed as long hex strings rather than
the "diamond" that they were previously displayed as.  I think the
complaint has merit.  It seems to me to be a classic case for a user
option.

> The alternative that emacs-devel is trying to establish (via experiments,
> etc/PROBLEMS changes, and perhaps code patches) will make the system fail
> less often -- that is, do what the user wants more often. The argument in
> question is basically "Don't make the software do what the user wants more
> often if it changes away from the bugs that are already familiar to me",
> especially when that argument is expressed *in the middle of fixing the
> problem*, as a discouragement from fixing the problem for all users, then
> we've arrived at "That's horrifying." ala XKCD 1172.

????

> ~Chad

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 1472 bytes --]


>
> I don't see any failure.  Richard's complaint was that characters 
> without glyphs were getting displayed as long hex strings rather than 
> the "diamond" that they were previously displayed as.  I think the 
> complaint has merit.  It seems to me to be a classic case for a user 
> option.
>

You forget to mention that the context in which this bug was fixed in 
January is that RMS complained that ligatures such as "fi" showed up as a 
diamond (see [1]).  At that time what he wanted is that Emacs would 
automatically replace such ligatures by two letters in his Linux console, 
and that's what he got: Eli improved the latin1-display-ucs-per-lynx 
function in fd42ba3adb.  At that time diamonds were considered 
"unhelpful", and he said for example: "I doubt any user wants to see a 
diamond instead of `fi'."

During that discussion, the bug that is now objected against was also 
fixed, by 10c680551e.

I, too, doubt any user wants to see a diamond instead of an actual 
character!  Why are these diamonds suddenly useful again?  How comes that 
a proposed solution with which ligatures such as "fi" are actually 
displayed as "fi", with which UTF-8 is actually supported instead of 
having to resort to ugly hacks such as latin1-display-ucs-per-lynx, and 
with which missing glyphs are in fact again displayed with these same 
diamonds, is criticized?

[1] https://lists.gnu.org/archive/html/emacs-devel/2022-01/msg01176.html

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Gregory Heytings wrote:

>> I don't see any failure. Richard's complaint was that
>> characters without glyphs were getting displayed as long
>> hex strings rather than the "diamond" that they were
>> previously displayed as. I think the complaint has merit.
>> It seems to me to be a classic case for a user option.
>
> You forget to mention that the context in which this bug was
> fixed in January is that RMS complained that ligatures such
> as "fi" showed up as a diamond [...] At that time what he
> wanted is that Emacs would automatically replace such
> ligatures by two letters in his Linux console, and that's
> what he got: Eli improved the latin1-display-ucs-per-lynx
> function in fd42ba3adb. At that time diamonds were
> considered "unhelpful", and he said for example: "I doubt
> any user wants to see a diamond instead of `fi'."
>
> During that discussion, the bug that is now objected against
> was also fixed, by 10c680551e.
>
> I, too, doubt any user wants to see a diamond instead of an
> actual character! Why are these diamonds suddenly useful
> again? How comes that a proposed solution with which
> ligatures such as "fi" are actually displayed as "fi", with
> which UTF-8 is actually supported instead of having to
> resort to ugly hacks such as latin1-display-ucs-per-lynx,
> and with which missing glyphs are in fact again displayed
> with these same diamonds, is criticized?

Guys, we are not fighting a holy war over in what order to
make the Sign of the Cross here, right?

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Alan Mackenzie]

Hello, Eli.

On Fri, Sep 02, 2022 at 16:59:48 +0300, Eli Zaretskii wrote:
> > Date: Fri, 2 Sep 2022 13:39:51 +0000
> > Cc: gregory@heytings.org, rms@gnu.org, emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > > > I agree with Richard that there should be an option for displaying
> > > > characters outside of the current font as "�" rather than "\u2022".

> > > I already explained how to set that up, so what exactly does the
> > > "should be" part here want to say?

> > The way you explained involves hacking Lisp

> It involves writing some Lisp code and running it on your system,
> yes.  How is that a problem?

It's not a problem for me, but might well be for other Linux console
users.  It really needs documenting somewhere, which might well be more
work than just supplying an option.

> > and finding out precise character ranges.  That's a lot different
> > from being able just to set an option.

> How can Emacs know which characters does your Linux console actually
> support, out of all the Unicode range?

The console supports ALL characters in the Unicode range.  Most of them
it displays with the replacement character glyph, but it supports them
all in the important sense of not crashing, or going unresponsive, or
anything else like that.

> And how can Emacs know which ones of those you want to see as their
> ASCII "emulations" (per latin1-disp.el) and which you want to see as
> U+FFFD replacements?

It could examine the currently loaded font, if needed.  But I was more
thinking about allowing the console itself to do the replacement with
\ufffd.

> So yes, this requires you, the user, to tell Emacs which ones you want
> to see in what manner.  There are a lot of Unicode characters, so it
> could be a large job, if the characters actually supported by the
> console are all in distinct Unicode blocks.  (But if you only want to
> see Latin-1 characters, I've shown in this thread a one-liner to do
> that.  I guess you will reject that as well.)

Some of the characters in my font lie outside the Latin-1 range.  Doing
this job properly requires writing some sort of script to examine the
loaded font.  I don't know how easy that would be.

> > > There is already such a way in Emacs.  Just use it, if that's what you
> > > want.

> > There appears to be no easy way to get the old behaviour back, where
> > characters undisplayable on the console got displayed with \ufffd
> > instead.  You've characterised this old behaviour as a bug, but in
> > the preferences of two actual Linux console users (Richard and
> > myself), the solution is not better than the bug.

> I don't understand why you are consistently rejecting every solution
> that was suggested.

Every solution suggested is either a lot of work, or is less good than
how things were.

> Including, amazingly enough, a way to actually produce the same
> "diamond" glyphs on display under the same circumstances.  Since when
> do veteran Emacs hackers such as you and Richard consider some Lisp
> coding a problem so grave as to justify flatly rejecting it?

It doesn't scale.  Without some sort of script, such as I suggested
above, it has to be hacked individually for each setup.

> Please understand: you are _really_ using Emacs on a terminal that is
> unsuitable for it.

For me, there is no better setup that I'm aware of.  If there were, I
would use it.

> You _should_ expect problems.  And when solutions are pointed out that
> are just a few lines of Lisp away, I'd expect you to embrace them, not
> flatly reject them.

I would prefer a solution that would work for all Linux console users,
including those who can't or don't want to hack Lisp.

[ .... ]

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> Date: Fri, 2 Sep 2022 16:12:12 +0000
> Cc: gregory@heytings.org, rms@gnu.org, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > It involves writing some Lisp code and running it on your system,
> > yes.  How is that a problem?
> 
> It's not a problem for me, but might well be for other Linux console
> users.

I'm not sure there are other users.

> It really needs documenting somewhere, which might well be more
> work than just supplying an option.

The display-tables are documented already.  And we use them in quite a
few situations already, including in latin1-disp.el.

> > How can Emacs know which characters does your Linux console actually
> > support, out of all the Unicode range?
> 
> The console supports ALL characters in the Unicode range.  Most of them
> it displays with the replacement character glyph

That's not what we call "support".

> > And how can Emacs know which ones of those you want to see as their
> > ASCII "emulations" (per latin1-disp.el) and which you want to see as
> > U+FFFD replacements?
> 
> It could examine the currently loaded font, if needed.

I'd welcome patches to do that, TIA.

> But I was more thinking about allowing the console itself to do the
> replacement with \ufffd.

There's no need to do that, since Emacs is capable of emitting that
character by itself.

> > So yes, this requires you, the user, to tell Emacs which ones you want
> > to see in what manner.  There are a lot of Unicode characters, so it
> > could be a large job, if the characters actually supported by the
> > console are all in distinct Unicode blocks.  (But if you only want to
> > see Latin-1 characters, I've shown in this thread a one-liner to do
> > that.  I guess you will reject that as well.)
> 
> Some of the characters in my font lie outside the Latin-1 range.  Doing
> this job properly requires writing some sort of script to examine the
> loaded font.  I don't know how easy that would be.

It's just a loop calling char-displayable-p, that's all.  What's left
is for you to examine the results.

> > Including, amazingly enough, a way to actually produce the same
> > "diamond" glyphs on display under the same circumstances.  Since when
> > do veteran Emacs hackers such as you and Richard consider some Lisp
> > coding a problem so grave as to justify flatly rejecting it?
> 
> It doesn't scale.  Without some sort of script, such as I suggested
> above, it has to be hacked individually for each setup.

Just once.  So its scaling is O(1).  Literally.

> > You _should_ expect problems.  And when solutions are pointed out that
> > are just a few lines of Lisp away, I'd expect you to embrace them, not
> > flatly reject them.
> 
> I would prefer a solution that would work for all Linux console users,
> including those who can't or don't want to hack Lisp.

For that, someone will have to figure out how to do that and submit
patches.  I don't know what else to suggest, since nothing seems to be
good enough.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Tomas Hlavaty]

On Fri 02 Sep 2022 at 19:26, Eli Zaretskii <eliz@gnu.org> wrote:
>> It's not a problem for me, but might well be for other Linux console
>> users.
>
> I'm not sure there are other users.

there are

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Tomas Hlavaty wrote:

>>> It's not a problem for me, but might well be for other
>>> Linux console users.
>>
>> I'm not sure there are other users.
>
> there are

Either a joke or provocation, of course there are ...

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

>>>> It's not a problem for me, but might well be for other
>>>> Linux console users.
>>>
>>> I'm not sure there are other users.
>>
>> there are
>
> Either a joke or provocation, of course there are ...

For every one (1) person here saying something moronic, in
reality it is just a matter of how many zeroes one should add
to that digit to express how many morons there are thinking
the same thing ...

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Please do not call anyone in this discussion a moron.
It injects bad feeling into the discussion.

See https://gnu.org/philosophy/kind-communication.html.
-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Andreas Schwab]

On Sep 02 2022, Eli Zaretskii wrote:

>> The console supports ALL characters in the Unicode range.  Most of them
>> it displays with the replacement character glyph
>
> That's not what we call "support".

It shares that with all other terminal emulators, depending on the
configured font.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Gregory Heytings]

>
> Some of the characters in my font lie outside the Latin-1 range.  Doing 
> this job properly requires writing some sort of script to examine the 
> loaded font.  I don't know how easy that would be.
>

I'm not sure, but I think what you want for that job is

fc-match --format='%{charset}' <font name>

which displays the Unicode ranges for which that font has a glyph.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > How can Emacs know which characters does your Linux console actually
  > support, out of all the Unicode range?

A few days ago you said it could use char-displayable-p
to determine this.  If that works, it solves the problem of finding out
which characters to display as diamond.

As for how, it would be asy to add a variable whose value should be a
vector containing integers, or a single-byte string, saying what to
send.  This will be adequate for a Linux console, and it doesn't need
to handle anything else.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: acm@muc.de, gregory@heytings.org, emacs-devel@gnu.org
> Date: Fri, 02 Sep 2022 22:51:27 -0400
> 
>   > How can Emacs know which characters does your Linux console actually
>   > support, out of all the Unicode range?
> 
> A few days ago you said it could use char-displayable-p
> to determine this.  If that works, it solves the problem of finding out
> which characters to display as diamond.

It works, but looping over all the Unicode characters is very slow, so
doing that at each startup would be unacceptable.  If someone needs, I
can show a simple code to be run by the user, once, on the Linux
console, that would allow then to determine the codepoint ranges for
which the U+FFFD (or any other) replacements are needed, so that users
could run that code, and then put such customizations in their init
files.  We could even have a special command to do that, if people
think it's important and helpful.

But all I hear till now is flat rejection of that solution, which is a
very weak incentive (to say the least) to work on such a command.

> As for how, it would be asy to add a variable whose value should be a
> vector containing integers, or a single-byte string, saying what to
> send.  This will be adequate for a Linux console, and it doesn't need
> to handle anything else.

You are describing what display-tables provide.  That's the solution I
recommend if you really want to see those U+FFFD "diamonds" instead of
the characters that cannot be displayed.  I see no reason for yet
another solution for a problem for which we already have two
(glyphless-char-display and display-tables).

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Alan Mackenzie]

Hello, Eli.

On Sat, Sep 03, 2022 at 09:44:46 +0300, Eli Zaretskii wrote:
> > From: Richard Stallman <rms@gnu.org>
> > Cc: acm@muc.de, gregory@heytings.org, emacs-devel@gnu.org
> > Date: Fri, 02 Sep 2022 22:51:27 -0400

> >   > How can Emacs know which characters does your Linux console actually
> >   > support, out of all the Unicode range?

> > A few days ago you said it could use char-displayable-p
> > to determine this.  If that works, it solves the problem of finding out
> > which characters to display as diamond.

> It works, but looping over all the Unicode characters is very slow, so
> doing that at each startup would be unacceptable.  If someone needs, I
> can show a simple code to be run by the user, once, on the Linux
> console, that would allow then to determine the codepoint ranges for
> which the U+FFFD (or any other) replacements are needed, so that users
> could run that code, and then put such customizations in their init
> files.  We could even have a special command to do that, if people
> think it's important and helpful.

> But all I hear till now is flat rejection of that solution, which is a
> very weak incentive (to say the least) to work on such a command.

I will start working on this soon (if nobody else does in the meantime).

> > As for how, it would be asy to add a variable whose value should be a
> > vector containing integers, or a single-byte string, saying what to
> > send.  This will be adequate for a Linux console, and it doesn't need
> > to handle anything else.

> You are describing what display-tables provide.  That's the solution I
> recommend if you really want to see those U+FFFD "diamonds" instead of
> the characters that cannot be displayed.  I see no reason for yet
> another solution for a problem for which we already have two
> (glyphless-char-display and display-tables).

I envisage having a minor mode specific to the Linux console, which when
enabled for the first time will do the loop over all Unicode characters
testing them for glyps, and create the display table.  This can then be
stored in .emacs in the customisation section.  When the mode is
enabled, this display table will be used to output the \ufffd's.  When
disabled, the glyphless characters will get printed as hex strings, as
currently.  Possibly this will involve having two display tables which
will get swapped when the mode is en/disabled.  Or something like that.

This should work well for a user who doesn't frequently change her Linux
console font settings.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> Date: Sun, 4 Sep 2022 11:23:43 +0000
> Cc: rms@gnu.org, gregory@heytings.org, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > > A few days ago you said it could use char-displayable-p
> > > to determine this.  If that works, it solves the problem of finding out
> > > which characters to display as diamond.
> 
> > It works, but looping over all the Unicode characters is very slow, so
> > doing that at each startup would be unacceptable.  If someone needs, I
> > can show a simple code to be run by the user, once, on the Linux
> > console, that would allow then to determine the codepoint ranges for
> > which the U+FFFD (or any other) replacements are needed, so that users
> > could run that code, and then put such customizations in their init
> > files.  We could even have a special command to do that, if people
> > think it's important and helpful.
> 
> > But all I hear till now is flat rejection of that solution, which is a
> > very weak incentive (to say the least) to work on such a command.
> 
> I will start working on this soon (if nobody else does in the meantime).

No need, it's already done.  See the new command
standard-display-by-replacement-char.

> I envisage having a minor mode specific to the Linux console, which when
> enabled for the first time will do the loop over all Unicode characters
> testing them for glyps, and create the display table.

This loop takes a long time (25 sec in my unoptimized build, likely 5
sec in optimized ones), and needs to be run only once on a given
terminal.  So instead the command I added produces a buffer with Lisp
code that you could eval and also add to your init file, to be run
every startup.

> This can then be
> stored in .emacs in the customisation section.  When the mode is
> enabled, this display table will be used to output the \ufffd's.  When
> disabled, the glyphless characters will get printed as hex strings, as
> currently.  Possibly this will involve having two display tables which
> will get swapped when the mode is en/disabled.  Or something like that.

I don't really see a reason to reset the display table to its default
state, especially since it isn't clear what is that default.  E.g.,
the startup code could add a few entries in the display-table (to show
the Help buffers), as does info.el.  You don't want to lose those
setting, and there's no easy way of discerning them from the ones
produced by the command we are discussing.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Alan Mackenzie]

Hello, Eli.

On Sun, Sep 04, 2022 at 16:01:53 +0300, Eli Zaretskii wrote:
> > Date: Sun, 4 Sep 2022 11:23:43 +0000
> > Cc: rms@gnu.org, gregory@heytings.org, emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > > > A few days ago you said it could use char-displayable-p
> > > > to determine this.  If that works, it solves the problem of finding out
> > > > which characters to display as diamond.

> > > It works, but looping over all the Unicode characters is very slow, so
> > > doing that at each startup would be unacceptable.  If someone needs, I
> > > can show a simple code to be run by the user, once, on the Linux
> > > console, that would allow then to determine the codepoint ranges for
> > > which the U+FFFD (or any other) replacements are needed, so that users
> > > could run that code, and then put such customizations in their init
> > > files.  We could even have a special command to do that, if people
> > > think it's important and helpful.

> > > But all I hear till now is flat rejection of that solution, which is a
> > > very weak incentive (to say the least) to work on such a command.

> > I will start working on this soon (if nobody else does in the meantime).

> No need, it's already done.  See the new command
> standard-display-by-replacement-char.

Thanks, that works well.

> > I envisage having a minor mode specific to the Linux console, which when
> > enabled for the first time will do the loop over all Unicode characters
> > testing them for glyps, and create the display table.

> This loop takes a long time (25 sec in my unoptimized build, likely 5
> sec in optimized ones), and needs to be run only once on a given
> terminal.  So instead the command I added produces a buffer with Lisp
> code that you could eval and also add to your init file, to be run
> every startup.

The second time I tried it (having rebooted the machine between times)
standard-display-by-replacement-char ran in around 0.077 seconds, which
scarcely counts as slow.  I don't know why it is fast on my machine, but
slower on others'.  Still, the code it produced is now firmly in my
..emacs.

> > This can then be
> > stored in .emacs in the customisation section.  When the mode is
> > enabled, this display table will be used to output the \ufffd's.  When
> > disabled, the glyphless characters will get printed as hex strings, as
> > currently.  Possibly this will involve having two display tables which
> > will get swapped when the mode is en/disabled.  Or something like that.

> I don't really see a reason to reset the display table to its default
> state, especially since it isn't clear what is that default.  E.g.,
> the startup code could add a few entries in the display-table (to show
> the Help buffers), as does info.el.  You don't want to lose those
> setting, and there's no easy way of discerning them from the ones
> produced by the command we are discussing.

OK.  I suppose that people only rarely change the fonts they are using
(more precisely, the character ranges of those fonts).

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Thanks for implementing this feature.  I hope I can try it soon.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I envisage having a minor mode specific to the Linux console, which when
  > enabled for the first time will do the loop over all Unicode characters
  > testing them for glyps, and create the display table.  This can then be
  > stored in .emacs in the customisation section.  When the mode is
  > enabled, this display table will be used to output the \ufffd's.

Thank you.

If using display tables is a convenient way to implement this feature,
I have no complaints about using that.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]


  > It works, but looping over all the Unicode characters is very slow, so
  > doing that at each startup would be unacceptable.  If someone needs, I
  > can show a simple code to be run by the user, once, on the Linux
  > console, that would allow then to determine the codepoint ranges for
  > which the U+FFFD (or any other) replacements are needed, so that users
  > could run that code, and then put such customizations in their init
  > files.

That's the way I'd like it to work.  I would not mind running this
again if the capabilities of the Linux console ever change,

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Richard Stallman wrote:

>> It works, but looping over all the Unicode characters is
>> very slow, so doing that at each startup would be
>> unacceptable. If someone needs, I can show a simple code to
>> be run by the user, once, on the Linux console, that would
>> allow then to determine the codepoint ranges for which the
>> U+FFFD (or any other) replacements are needed, so that
>> users could run that code, and then put such customizations
>> in their init files.
>
> That's the way I'd like it to work. I would not mind running
> this again if the capabilities of the Linux console ever
> change,

Last time I heard anything about it work on it was
disencouraged, I don't know if this was because they wanted to
make Linux stronger on the desktop (this was before the
smartphone era) or if they thought working on it would
increase complexity and introduce bugs, and that would be bad
since, as we heard in another post already, it is relied upon
not just by Emacs powerusers but also for debugging when
X fails ...

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Mon, 05 Sep 2022 00:05:32 -0400
> 
>   > It works, but looping over all the Unicode characters is very slow, so
>   > doing that at each startup would be unacceptable.  If someone needs, I
>   > can show a simple code to be run by the user, once, on the Linux
>   > console, that would allow then to determine the codepoint ranges for
>   > which the U+FFFD (or any other) replacements are needed, so that users
>   > could run that code, and then put such customizations in their init
>   > files.
> 
> That's the way I'd like it to work.  I would not mind running this
> again if the capabilities of the Linux console ever change,

Great, then please update from Git's master branch and use the new
command standard-display-by-replacement-char, which I added a couple
of days ago.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Eli Zaretskii wrote:

> Great, then please update from Git's master branch and use
> the new command standard-display-by-replacement-char, which
> I added a couple of days ago.

I just updated Emacs so I got it now, but what are you
supposed to do with it exactly?

The Linux console is even mentioned in the docstring so it
must have something to do with it ...

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

On September 6, 2022 7:20:03 AM GMT+03:00, Emanuel Berg <incal@dataswamp.org> wrote:
> Eli Zaretskii wrote:
> 
> > Great, then please update from Git's master branch and use
> > the new command standard-display-by-replacement-char, which
> > I added a couple of days ago.
> 
> I just updated Emacs so I got it now, but what are you
> supposed to do with it exactly?
> 
> The Linux console is even mentioned in the docstring so it
> must have something to do with it ...
> 

The command has a doc string.  Did you read it?  If so, and if the doc string didn't explain that, please ask specific questions.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Eli Zaretskii wrote:

> The command has a doc string. Did you read it? If so, and if
> the doc string didn't explain that, please ask
> specific questions.

Don't we need an interface that actually executes the wishes
of the user as specified with the provided argument(s) so that
middle level of using "produced code" can be
skipped altogether? Something like

  (set-unprintable-char-char chr)

And CHR can be optional and default to #xfffd if that's the
"Unicode replacement character".

However ... if it is, why should that have to be done at all?

  (setq use-default-values t)

?

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Tue, 06 Sep 2022 12:38:28 +0200
> 
> Eli Zaretskii wrote:
> 
> > The command has a doc string. Did you read it? If so, and if
> > the doc string didn't explain that, please ask
> > specific questions.
> 
> Don't we need an interface that actually executes the wishes
> of the user as specified with the provided argument(s) so that
> middle level of using "produced code" can be
> skipped altogether?

The doc string says:

  This function produces a buffer with code to set up `standard-display-table'
  such that characters that cannot be displayed by the terminal, and
  don't already have their display set up in `standard-display-table', will
  be represented by a replacement character.  You can evaluate the produced
  code to use the setup for the current Emacs session, or copy the code
  into your init file, to make Emacs use it for subsequent sessions.

The last sentence says how to cause the code to be "executed".

> Something like
> 
>   (set-unprintable-char-char chr)

That would require Emacs to detect all the "unprintable" characters on
each startup, which would slow down startup, sometimes to unacceptable
lengths of time.

If you followed the discussion, this was explained and unanimously
rejected.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Eli Zaretskii wrote:

>> Something like
>> 
>>   (set-unprintable-char-char chr)
>
> That would require Emacs to detect all the "unprintable"
> characters on each startup, which would slow down startup,
> sometimes to unacceptable lengths of time.
>
> If you followed the discussion, this was explained and
> unanimously rejected.

Yes, but there is surely a way to get around that, a way that
is better, since it's possible to do it's merely a question of
streamlining the interface ...

All this programmer stuff that are common all around Emacs and
the Emacs help, all that should be gradually reduced, not
added to.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Wed, 07 Sep 2022 03:03:22 +0200
> 
> Eli Zaretskii wrote:
> 
> >> Something like
> >> 
> >>   (set-unprintable-char-char chr)
> >
> > That would require Emacs to detect all the "unprintable"
> > characters on each startup, which would slow down startup,
> > sometimes to unacceptable lengths of time.
> >
> > If you followed the discussion, this was explained and
> > unanimously rejected.
> 
> Yes, but there is surely a way to get around that, a way that
> is better

If you know how to get around that, please tell the details.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Eli Zaretskii wrote:

>>> If you followed the discussion, this was explained and
>>> unanimously rejected.
>> 
>> Yes, but there is surely a way to get around that, a way
>> that is better
>
> If you know how to get around that, please tell the details.

In general ...

Compute f(x) = y 

Store f = ((x y) ... ) somewhere.

Next time don't compute, search for x, find y?

Maybe x can be detected automatically, as well? Then the whole
process is automated, and invisible.

No addition to the user's init files necessary, unless s/he
explicitely wants to recompute, "switch x" to something else,
redefine it, yada yada, that will then use whatever you used,
to do steps 1-2 ... and then THAT will be used with the same
functions because only the data (mapping) has changed.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Wed, 07 Sep 2022 05:21:26 +0200
> 
> Eli Zaretskii wrote:
> 
> >>> If you followed the discussion, this was explained and
> >>> unanimously rejected.
> >> 
> >> Yes, but there is surely a way to get around that, a way
> >> that is better
> >
> > If you know how to get around that, please tell the details.
> 
> In general ...
> 
> Compute f(x) = y 
> 
> Store f = ((x y) ... ) somewhere.
> 
> Next time don't compute, search for x, find y?
> 
> Maybe x can be detected automatically, as well? Then the whole
> process is automated, and invisible.
> 
> No addition to the user's init files necessary, unless s/he
> explicitely wants to recompute, "switch x" to something else,
> redefine it, yada yada, that will then use whatever you used,
> to do steps 1-2 ... and then THAT will be used with the same
> functions because only the data (mapping) has changed.

I don't think I follow.  What is "x" and what is "y" in this case?

And where "somewhere" to store this, except on the user init file?

And what is meant by "detect x automatically, as well"?  If that is
the inability to display certain characters, then that's exactly what
the command does, so what is the "automatic" alternative you have in
mind?  Invoking the command automatically every startup is
unacceptably slow, as was already explained.  What other ideas are for
automatically detecting this?

IOW, do you have concrete ideas, or are these just general
aspirations?  If the latter, then I already considered the possible
concrete implementations, and concluded that they will be worse than
what was installed.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Yuri Khan]

On Wed, 7 Sept 2022 at 18:04, Eli Zaretskii <eliz@gnu.org> wrote:

> > >>> If you followed the discussion, this was explained and
> > >>> unanimously rejected.
> > >>

> > Compute f(x) = y
> >
> > Store f = ((x y) ... ) somewhere.
> >
> > Next time don't compute, search for x, find y?

I believe Emanuel is alluding to the memoization pattern “if a
computation (e.g. of the set of all displayable characters) is too
expensive to perform every time its result is needed, cache the result
in some storage that survives between the times it is needed (e.g. a
data file in .emacs.d/var/)”.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> From: Yuri Khan <yuri.v.khan@gmail.com>
> Date: Wed, 7 Sep 2022 20:12:40 +0700
> Cc: Emanuel Berg <incal@dataswamp.org>, emacs-devel@gnu.org
> 
> On Wed, 7 Sept 2022 at 18:04, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > > >>> If you followed the discussion, this was explained and
> > > >>> unanimously rejected.
> > > >>
> 
> > > Compute f(x) = y
> > >
> > > Store f = ((x y) ... ) somewhere.
> > >
> > > Next time don't compute, search for x, find y?
> 
> I believe Emanuel is alluding to the memoization pattern “if a
> computation (e.g. of the set of all displayable characters) is too
> expensive to perform every time its result is needed, cache the result
> in some storage that survives between the times it is needed (e.g. a
> data file in .emacs.d/var/)”.

That's exactly what the instructions produced with the code suggest...

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Yuri Khan]

On Wed, 7 Sept 2022 at 20:34, Eli Zaretskii <eliz@gnu.org> wrote:

> > I believe Emanuel is alluding to the memoization pattern “if a
> > computation (e.g. of the set of all displayable characters) is too
> > expensive to perform every time its result is needed, cache the result
> > in some storage that survives between the times it is needed (e.g. a
> > data file in .emacs.d/var/)”.
>
> That's exactly what the instructions produced with the code suggest...

Um, no? This is code generation, a different pattern where slow code
generates fast code that needs to be included in the actual program
(in this case, user’s init file).

Code generation:

* M-x standard-display-by-replacement-char RET.
* It pauses for a while, gives you some comments and code.
* At a minimum, you have to read and understand the comment.
* Then you copy and paste that code in your init file and optionally
evaluate it for immediate effect.

Compare with caching/memoization:

* M-x some-hypothetical-other-command RET.
* It pauses for a while, stores a data file, has an immediate effect.
* You see the effect, decide that it is good, put
(some-hypothetical-other-command) in your init file.
* On the next startup, it reads the file (which is fast), applies the
data (which is supposedly also fast), has the same effect.

The file name could include some relevant factors such as the terminal
type on which the data depends.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> From: Yuri Khan <yuri.v.khan@gmail.com>
> Date: Wed, 7 Sep 2022 20:58:53 +0700
> Cc: incal@dataswamp.org, emacs-devel@gnu.org
> 
> * M-x some-hypothetical-other-command RET.
> * It pauses for a while, stores a data file, has an immediate effect.
> * You see the effect, decide that it is good, put
> (some-hypothetical-other-command) in your init file.
> * On the next startup, it reads the file (which is fast), applies the
> data (which is supposedly also fast), has the same effect.
> 
> The file name could include some relevant factors such as the terminal
> type on which the data depends.

You suggest semi-automatically saving the generated code in the init
file, instead of the user copying it there manually?  Then I have two
comments:

  . IME users don't particularly like commands writing into their
    files
  . the difference between these two patterns is in this case barely
    perceptible

(And yes, it has to be the init file, because if users dislike us
writing into the init file, they dislike us writing to additional
files even more.)

Bottom line: sounds like a tempest in a teapot.  Especially since the
two users who requested this already said they were satisfied.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Tomas Hlavaty]

On Wed 07 Sep 2022 at 17:20, Eli Zaretskii <eliz@gnu.org> wrote:
>> * You see the effect, decide that it is good, put
>> (some-hypothetical-other-command) in your init file.
>> * On the next startup, it reads the file (which is fast), applies the
>> data (which is supposedly also fast), has the same effect.
>> 
>> The file name could include some relevant factors such as the terminal
>> type on which the data depends.
>
> You suggest semi-automatically saving the generated code in the init
> file, instead of the user copying it there manually?

No, he said that the generated code is saved automatically in a file and
if the user puts (some-hypothetical-other-command) in his .emacs
manually, it will load the saved file.  Something like:

(defun some-hypothetical-other-command ()
  (or (let ((f "~/.emacs.d/cache/some-hypothetical-other-command.el"))
        (when (file-exists-p f)
          (load f)))
      (generate-and-save-that-slow-code)))

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Stefan Monnier]

> No, he said that the generated code is saved automatically in a file and
> if the user puts (some-hypothetical-other-command) in his .emacs
> manually, it will load the saved file.  Something like:
>
> (defun some-hypothetical-other-command ()
>   (or (let ((f "~/.emacs.d/cache/some-hypothetical-other-command.el"))
>         (when (file-exists-p f)
>           (load f)))
>       (generate-and-save-that-slow-code)))

The problem of course being how to decide when to flush the cache.
By making the system less automatic, we expose enough of it that the
user will obviously know how to "flush the cache" and will hopefully
realize when it might need to be "flushed".


        Stefan

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Tomas Hlavaty]

On Wed 07 Sep 2022 at 12:48, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>> No, he said that the generated code is saved automatically in a file and
>> if the user puts (some-hypothetical-other-command) in his .emacs
>> manually, it will load the saved file.  Something like:
>>
>> (defun some-hypothetical-other-command ()
>>   (or (let ((f "~/.emacs.d/cache/some-hypothetical-other-command.el"))
>>         (when (file-exists-p f)
>>           (load f)))
>>       (generate-and-save-that-slow-code)))
>
> The problem of course being how to decide when to flush the cache.
> By making the system less automatic, we expose enough of it that the
> user will obviously know how to "flush the cache" and will hopefully
> realize when it might need to be "flushed".

What does "less automatic" mean?

There are two separate issues:

1) How to update the cache:
   a) rm ~/.emacs.d/cache/some-hypothetical-other-command.el
   b) or M-x generate-and-save-that-slow-code
2) When to update the cache.  I think this has already been discussed:
   when the user chooses font with different charset.  It seems enough
   to let the user decide.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Stefan Monnier wrote:

> The problem of course being how to decide when to flush the
> cache. By making the system less automatic, we expose enough
> of it that the user will obviously know how to "flush the
> cache" and will hopefully realize when it might need to be
> "flushed".

But when should that happen to begin with, when the user has
reconfigured the console so that suddenly new chars are (or
aren't) visible, or what?

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> From: Tomas Hlavaty <tom@logand.com>
> Cc: incal@dataswamp.org, emacs-devel@gnu.org
> Date: Wed, 07 Sep 2022 17:48:58 +0200
> 
> On Wed 07 Sep 2022 at 17:20, Eli Zaretskii <eliz@gnu.org> wrote:
> >> * You see the effect, decide that it is good, put
> >> (some-hypothetical-other-command) in your init file.
> >> * On the next startup, it reads the file (which is fast), applies the
> >> data (which is supposedly also fast), has the same effect.
> >> 
> >> The file name could include some relevant factors such as the terminal
> >> type on which the data depends.
> >
> > You suggest semi-automatically saving the generated code in the init
> > file, instead of the user copying it there manually?
> 
> No, he said that the generated code is saved automatically in a file and
> if the user puts (some-hypothetical-other-command) in his .emacs
> manually, it will load the saved file.  Something like:

Yes.  And I explained why I thought that was not the best idea, in the
parts that you elided.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Eli Zaretskii wrote:

>>> You suggest semi-automatically saving the generated code
>>> in the init file, instead of the user copying it
>>> there manually?
>> 
>> No, he said that the generated code is saved automatically
>> in a file and if the user puts
>> (some-hypothetical-other-command) in his .emacs manually,
>> it will load the saved file. Something like:
>
> Yes. And I explained why I thought that was not the best
> idea, in the parts that you elided.

...

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Tomas Hlavaty wrote:

>>> * You see the effect, decide that it is good, put
>>>   (some-hypothetical-other-command) in your init file.
>>> * On the next startup, it reads the file (which is fast), applies the
>>>   data (which is supposedly also fast), has the same effect.
>>> 
>>> The file name could include some relevant factors such as the terminal
>>> type on which the data depends.
>>
>> You suggest semi-automatically saving the generated code in
>> the init file, instead of the user copying it
>> there manually?
>
> No, he said that the generated code is saved automatically
> in a file and if the user puts
> (some-hypothetical-other-command) in his .emacs manually, it
> will load the saved file. Something like:

*Nothing like :)

> (defun some-hypothetical-other-command ()
>   (or (let ((f "~/.emacs.d/cache/some-hypothetical-other-command.el"))
>         (when (file-exists-p f)
>           (load f)))
>       (generate-and-save-that-slow-code)))

Again, the result, not the code ...

I have this [last] which I know works but I don't know what
the prefered format or idiomatic Elisp ways are to store and
retrieve data to and from the filesystem.

So yeah, what do we we have to do that fast and secure and for
arbitrary (nested) data structures?

;;; -*- lexical-binding: t -*-
;;
;; this file:
;;   https://dataswamp.org/~incal/emacs-init/file-write-to.el

(require 'subr-x)

;; write

(defun write-to-file (file str)
  (write-region str nil file) )
(defalias 'write-to-file-string #'write-to-file)

(defun write-to-file-integer (file int)
  (write-region (number-to-string int) nil file) )

;; read

(defun file-to-string (file)
  "A string with the contents of FILE."
  (interactive "Ffile: ")
  (with-temp-buffer
    (insert-file-contents file)
    (string-trim
     (buffer-substring-no-properties (point-min) (point-max)) )))

(defun file-to-integer (file)
  (string-to-number (file-to-string file)) )

(defmacro file-to-variable (file var)
  `(setq ,var ,(file-to-string file)) )

(defmacro file-to-variable-integer (file var)
  `(setq ,var ,(string-to-number (file-to-string file))) )

;; (write-to-file         "~/5ifth.txt"   "Leeloo Dallas Multipass")
;; (write-to-file-integer "~/element.txt" 5)
;;
;; (file-to-string   "~/5ifth.txt")                         ; Leeloo ...
;; (file-to-variable "~/5ifth.txt" string-value)            ; Leeloo ...
;; string-value                                             ; Leeloo ...
;; (file-to-variable-integer "~/element.txt" integer-value) ; 5
;; integer-value                                            ; 5

(provide 'file-write-to)

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Yuri Khan]

On Wed, 7 Sept 2022 at 21:20, Eli Zaretskii <eliz@gnu.org> wrote:

> You suggest semi-automatically saving the generated code in the init
> file, instead of the user copying it there manually?

Does there have to be generated code involved? I was thinking it could
be static code acting on generated and cached data.

> (And yes, it has to be the init file, because if users dislike us
> writing into the init file, they dislike us writing to additional
> files even more.)

Do they really? My ~/.emacs.d/var contains many files and I am not
bothered with them and I know I can afford to lose them.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> From: Yuri Khan <yuri.v.khan@gmail.com>
> Date: Thu, 8 Sep 2022 00:29:55 +0700
> Cc: incal@dataswamp.org, emacs-devel@gnu.org
> 
> On Wed, 7 Sept 2022 at 21:20, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > You suggest semi-automatically saving the generated code in the init
> > file, instead of the user copying it there manually?
> 
> Does there have to be generated code involved? I was thinking it could
> be static code acting on generated and cached data.

Why does it matter?  In Lisp, code is data, so how does it matter
whether you generate code to call set-char-table-range for a range of
characters, or generate just the range of characters that some code
should then read and call set-char-table-range?

> > (And yes, it has to be the init file, because if users dislike us
> > writing into the init file, they dislike us writing to additional
> > files even more.)
> 
> Do they really? My ~/.emacs.d/var contains many files and I am not
> bothered with them and I know I can afford to lose them.

I said users, not all of the users including Yuri Khan.  Just witness
the frequent discussions regarding whether customizations should be
written to this or that file.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Eli Zaretskii wrote:

>> Does there have to be generated code involved? I was
>> thinking it could be static code acting on generated and
>> cached data.
>
> Why does it matter?

Observe ...

Situation: You want to do this, f(x) = y.

Problem: f(x) is expensive to compute.

Solution: Compute it ONCE, then store f = ((x y) ...) in
a cache data file, so next time f(x) = y is needed, instead of
computing f(x) again search the data file - search, which
_isn't_ expensive.

>> Do they really? My ~/.emacs.d/var contains many files and
>> I am not bothered with them and I know I can afford to
>> lose them.
>
> I said users, not all of the users including Yuri Khan.
> Just witness the frequent discussions regarding whether
> customizations should be written to this or that file.

People don't want auto-insertions by Emacs into files they
edit or view, some very short data file somewhere on the disk
where they never go they don't know/care about, even.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Fri, 09 Sep 2022 11:56:00 +0200
> 
> Eli Zaretskii wrote:
> 
> >> Does there have to be generated code involved? I was
> >> thinking it could be static code acting on generated and
> >> cached data.
> >
> > Why does it matter?
> 
> Observe ...
> 
> Situation: You want to do this, f(x) = y.
> 
> Problem: f(x) is expensive to compute.
> 
> Solution: Compute it ONCE, then store f = ((x y) ...) in
> a cache data file, so next time f(x) = y is needed, instead of
> computing f(x) again search the data file - search, which
> _isn't_ expensive.

And I ask again: why does it matter whether you record a program or
its data, when the program is in Lisp, where data and code are the
same?

> People don't want auto-insertions by Emacs into files they
> edit or view, some very short data file somewhere on the disk
> where they never go they don't know/care about, even.

That's not what I conclude from user reaction to features that write
non-temporary files.

Anyway, this discussion has exhausted itself, so I'll bow out of it.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Eli Zaretskii wrote:

>>> Why does it matter?
>> 
>> Observe ...
>> 
>> Situation: You want to do this, f(x) = y.
>> 
>> Problem: f(x) is expensive to compute.
>> 
>> Solution: Compute it ONCE, then store f = ((x y) ...) in
>> a cache data file, so next time f(x) = y is needed, instead of
>> computing f(x) again search the data file - search, which
>> _isn't_ expensive.
>
> And I ask again: why does it matter whether you record
> a program or its data

If you store the result of running the program you don't need
to execute the program again next time you need the result,
instead you search for it and retrieve it, simple as that.

Only if it cannot be found do you execute (and then insert the
computed value), so optimally and in practice often execution
only has to be done once.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Sat, 10 Sep 2022 06:41:32 +0200
> 
> Eli Zaretskii wrote:
> 
> >> Situation: You want to do this, f(x) = y.
> >> 
> >> Problem: f(x) is expensive to compute.
> >> 
> >> Solution: Compute it ONCE, then store f = ((x y) ...) in
> >> a cache data file, so next time f(x) = y is needed, instead of
> >> computing f(x) again search the data file - search, which
> >> _isn't_ expensive.
> >
> > And I ask again: why does it matter whether you record
> > a program or its data
> 
> If you store the result of running the program you don't need
> to execute the program again next time you need the result,
> instead you search for it and retrieve it, simple as that.

That's what the current implementation does.  It just stores the
results as a (much faster) program to be run at Emacs startup time,
that's all.  AFAIU, your proposal is to store just the ranges of
problematic characters, which would require to run, at Emacs startup
time, another program to read those ranges and issue the calls that
the current implementation embeds in the stored results.

Which is why I think there's no real difference between the two
approaches, and I consider the current implementation be slightly
better, since the risk of getting the startup calls wrong is somewhat
lower.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Eli Zaretskii wrote:

>>>> Situation: You want to do this, f(x) = y.
>>>> 
>>>> Problem: f(x) is expensive to compute.
>>>> 
>>>> Solution: Compute it ONCE, then store f = ((x y) ...) in
>>>> a cache data file, so next time f(x) = y is needed,
>>>> instead of computing f(x) again search the data file -
>>>> search, which _isn't_ expensive.
>>>
>>> And I ask again: why does it matter whether you record
>>> a program or its data
>> 
>> If you store the result of running the program you don't
>> need to execute the program again next time you need the
>> result, instead you search for it and retrieve it, simple
>> as that.
>
> That's what the current implementation does. It just stores
> the results as a (much faster) program to be run at Emacs
> startup time, that's all. AFAIU, your proposal is to store
> just the ranges of problematic characters, which would
> require to run, at Emacs startup time, another program to
> read those ranges and issue the calls that the current
> implementation embeds in the stored results.

Yes, of course, you need to be able to read the stored result.

But don't we have that already, and isn't that much less
expensive than to compute that, optimized or not?

Also, isn't it easier for the user to have just a

  (setup-chars)

in the init which would

  1. look for stored data
  2. (a) use it, if found
  2. (b) compute it, if not found, and store it

I don't see how that ever can be worse than generating code
the user has to put into her/his file to compute the same
thing every time?

A cache is better in terms of computing resources, and
requires a less complicated interface for us humans.

You don't agree?

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Sat, 10 Sep 2022 17:19:44 +0200
> 
> Eli Zaretskii wrote:
> 
> >> If you store the result of running the program you don't
> >> need to execute the program again next time you need the
> >> result, instead you search for it and retrieve it, simple
> >> as that.
> >
> > That's what the current implementation does. It just stores
> > the results as a (much faster) program to be run at Emacs
> > startup time, that's all. AFAIU, your proposal is to store
> > just the ranges of problematic characters, which would
> > require to run, at Emacs startup time, another program to
> > read those ranges and issue the calls that the current
> > implementation embeds in the stored results.
> 
> Yes, of course, you need to be able to read the stored result.
> 
> But don't we have that already

Have what?

> and isn't that much less
> expensive than to compute that, optimized or not?
> 
> Also, isn't it easier for the user to have just a
> 
>   (setup-chars)
> 
> in the init which would
> 
>   1. look for stored data
>   2. (a) use it, if found
>   2. (b) compute it, if not found, and store it

What would be the purpose of adding a function for code that's
supposed to go into the user's init file?

> I don't see how that ever can be worse than generating code
> the user has to put into her/his file to compute the same
> thing every time?

The code must be added to the init file, once, anyway.

> A cache is better in terms of computing resources, and
> requires a less complicated interface for us humans.

It isn't a cache.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Eli Zaretskii wrote:

>> and isn't that much less
>> expensive than to compute that, optimized or not?
>> 
>> Also, isn't it easier for the user to have just a
>> 
>>   (setup-chars)
>> 
>> in the init which would
>> 
>>   1. look for stored data
>>   2. (a) use it, if found
>>   2. (b) compute it, if not found, and store it
>
> What would be the purpose of adding a function for code that's
> supposed to go into the user's init file?

... ??

If this is an attempt at provocation, it's not working.

The simple and well-known method that I describe has been
explained many times now by 2 people.

If you truly don't understand how it is completely different
from your code-generation idea, so be it.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

> The simple and well-known method [...]

Wait, don't say anything!

I'll just code it and show you!

I didn't think of it ...

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

>> The simple and well-known method [...]
>
> Wait, don't say anything!
>
> I'll just code it and show you!
>
> I didn't think of it ...

Couldn't find a standard way of storing arbitrary Elisp data
so just wrote it on instinct.

At one point I didn't know what to do but it turned out my
subconscious had already that problem solved.

;;; -*- lexical-binding: t -*-
;;
;; this file:
;;   https://dataswamp.org/~incal/emacs-init/cache.el

(let ((cache-file (format "%svar/data-cache.el" user-emacs-directory)))
  (defun save-char-table ()
    (with-temp-buffer
      (insert "(setq standard-display-table\n\n")
      (prin1 standard-display-table (current-buffer))
      (insert "\n\n)")
      (write-file cache-file) ))
  (declare-function save-char-table nil)

  (defun load-char-table ()
    (when (file-exists-p cache-file)
      (load-file cache-file) ))
  (declare-function load-char-table nil) )

(defun set-replacement-char-table ()
  (unless (load-char-table)
    (let ((tbl standard-display-table)) ;; WARNING, ugly code ahead!
      (set-char-table-range tbl '(#x102 . #x103)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x108 . #x10b)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x10e . #x111)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x114 . #x115)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x11a . #x121)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x124 . #x129)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x12c . #x12d)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x130 . #x135)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x138 . #x13a)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x13d . #x140)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x147 . #x14b)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x14e . #x155)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x158 . #x159)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x15c . #x15f)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x162 . #x169)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x16c . #x171)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x174 . #x178)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x17f . #x2c6)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x2c8 . #x2c8)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x2ca . #x2d8)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x2da . #x2da)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x2dc . #x390)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x393 . #x394)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x398 . #x398)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x39b . #x39b)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x39e . #x39e)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x3a0 . #x3a0)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x3a2 . #x3a3)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x3a5 . #x3a6)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x3a8 . #x3bb)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x3bd . #x3bf)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x3c1 . #x400)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x402 . #x404)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x407 . #x407)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x409 . #x40f)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x411 . #x411)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x413 . #x414)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x416 . #x419)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x41b . #x41b)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x41f . #x41f)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x423 . #x424)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x426 . #x42f)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x431 . #x434)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x436 . #x43d)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x43f . #x43f)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x442 . #x442)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x444 . #x444)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x446 . #x450)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x452 . #x454)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x457 . #x457)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x459 . #x4ad)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x4af . #x1fff)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x200b . #x200f)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x2016 . #x2017)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x201b . #x201b)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x201f . #x201f)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x2023 . #x2025)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x2027 . #x202e)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x2031 . #x2038)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x203b . #x20ab)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x20ad . #x2115)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x2117 . #x2121)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x2123 . #x2129)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x212c . #x218f)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x2194 . #x2211)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x2213 . #x2247)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x2249 . #x225f)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x2261 . #x2263)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x2266 . #x2269)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x226c . #x2294)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x2297 . #x2297)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x229a . #x229a)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x229d . #x245f)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x2469 . #x24b5)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x24eb . #x24ff)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x2580 . #x2587)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x2589 . #x2590)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x2593 . #x259f)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x25a1 . #x25ad)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x25af . #x25b1)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x25b3 . #x25b3)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x25b5 . #x25b5)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x25b7 . #x25b7)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x25b9 . #x25bb)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x25bd . #x25bd)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x25bf . #x25bf)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x25c1 . #x25c1)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x25c3 . #x25c7)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x25c9 . #x25ce)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x25d0 . #x260f)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x2613 . #x2665)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x2667 . #x2716)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x2719 . #x2bbc)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#x2bbe . #xfffc)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))
      (set-char-table-range tbl '(#xfffe . #x10ffff)
                            (vconcat (list (make-glyph-code #xfffd 'homoglyph))))

      (save-char-table) )))

(set-replacement-char-table)

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Yuri Khan wrote:

> Does there have to be generated code involved?

Ikr?

> I was thinking it could be static code acting on generated
> and cached data.

Indeed ...

>> (And yes, it has to be the init file, because if users
>> dislike us writing into the init file, they dislike us
>> writing to additional files even more.)
>
> Do they really?

No.

> My ~/.emacs.d/var contains many files and I am not bothered
> with them and I know I can afford to lose them.

Exactly.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Eli Zaretskii wrote:

>> * M-x some-hypothetical-other-command RET.
>> * It pauses for a while, stores a data file, has an immediate effect.
>> * You see the effect, decide that it is good, put
>>   (some-hypothetical-other-command) in your init file.
>> * On the next startup, it reads the file (which is fast), applies the
>>   data (which is supposedly also fast), has the same effect.
>> 
>> The file name could include some relevant factors such as
>> the terminal type on which the data depends.
>
> You suggest semi-automatically saving the generated code [...]

The result.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Yuri Khan wrote:

>>> I believe Emanuel is alluding to the memoization pattern
>>> "if a computation (e.g. of the set of all displayable
>>> characters) is too expensive to perform every time its
>>> result is needed, cache the result in some storage that
>>> survives between the times it is needed (e.g. a data file
>>> in .emacs.d/var/)".
>>
>> That's exactly what the instructions produced with the code
>> suggest...
>
> Um, no? This is code generation, a different pattern where
> slow code generates fast code that needs to be included in
> the actual program (in this case, user’s init file).
>
> Code generation:
>
> * M-x standard-display-by-replacement-char RET.
> * It pauses for a while, gives you some comments and code.
> * At a minimum, you have to read and understand the comment.
> * Then you copy and paste that code in your init file and optionally
> evaluate it for immediate effect.
>
> Compare with caching/memoization:
>
> * M-x some-hypothetical-other-command RET.
> * It pauses for a while, stores a data file, has an immediate effect.
> * You see the effect, decide that it is good, put
>   (some-hypothetical-other-command) in your init file.
> * On the next startup, it reads the file (which is fast), applies the
> data (which is supposedly also fast), has the same effect.

Yes, this is exactly what I mean.

> The file name could include some relevant factors such as
> the terminal type on which the data depends.

Good idea, base that on tty(1) perhaps?

  (info "(coreutils) tty invocation")

Also see ttyname(3).

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Yuri Khan wrote:

>> Compute f(x) = y
>>
>> Store f = ((x y) ... ) somewhere.
>>
>> Next time don't compute, search for x, find y [...]
>
> I believe Emanuel is alluding to the memoization pattern "if
> a computation [...] is too expensive to perform every time
> its result is needed, cache the result in some storage that
> survives between the times it is needed

Here we again see the benefit of that Soviet Union math
education :)

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Thu, 08 Sep 2022 03:32:12 +0200
> 
> Yuri Khan wrote:
> 
> > I believe Emanuel is alluding to the memoization pattern "if
> > a computation [...] is too expensive to perform every time
> > its result is needed, cache the result in some storage that
> > survives between the times it is needed
> 
> Here we again see the benefit of that Soviet Union math
> education :)

Your assumptions/guesses regarding who around here was educated where
are (as usual) off the mark.  May I suggest keeping such remarks off
this list (or any public forum)?  You'll be doing favor to yourself if
you do.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Eli Zaretskii wrote:

>>> I believe Emanuel is alluding to the memoization pattern
>>> "if a computation [...] is too expensive to perform every
>>> time its result is needed, cache the result in some
>>> storage that survives between the times it is needed
>> 
>> Here we again see the benefit of that Soviet Union math
>> education :)
>
> Your assumptions/guesses regarding who around here was
> educated where are (as usual) off the mark.

You are saying it was after 1991? Okay, Russian Federation
then! Seems to have done the job anyway :)

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Fri, 09 Sep 2022 11:59:06 +0200
> 
> Eli Zaretskii wrote:
> 
> >>> I believe Emanuel is alluding to the memoization pattern
> >>> "if a computation [...] is too expensive to perform every
> >>> time its result is needed, cache the result in some
> >>> storage that survives between the times it is needed
> >> 
> >> Here we again see the benefit of that Soviet Union math
> >> education :)
> >
> > Your assumptions/guesses regarding who around here was
> > educated where are (as usual) off the mark.
> 
> You are saying it was after 1991?

No, I'm saying you really have no idea who around here was educated
where.

And this sub-thread is off-topic.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > >> Here we again see the benefit of that Soviet Union math
  > >> education :)
  > >
  > > Your assumptions/guesses regarding who around here was
  > > educated where are (as usual) off the mark.

  > You are saying it was after 1991? Okay, Russian Federation
  > then! Seems to have done the job anyway :)

Would all of you please take this personal squabble off the list?
Aside from being unkind in tone, it's 100% tangential to our work.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Great, then please update from Git's master branch and use the new
  > command standard-display-by-replacement-char, which I added a couple
  > of days ago.

Thanks very much.  It seems to work.

I suggest adding comments at the start of the output explaining the
way to use the output.  And moving point to the start, so the user
sees the comments.  For example:

;; Evaluate the Lisp code below to make Emacs show the standard
;; replacement character as a substitute for each undisplayable character.
;; One way to do that is with C-x h M-x eval-region RET.
;; Normally you would put this code in your Emacs initialization file,
;; perhaps conditionally based on the type of terminal.
;; See <more info>.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Tue, 06 Sep 2022 22:19:09 -0400
> 
>   > Great, then please update from Git's master branch and use the new
>   > command standard-display-by-replacement-char, which I added a couple
>   > of days ago.
> 
> Thanks very much.  It seems to work.
> 
> I suggest adding comments at the start of the output explaining the
> way to use the output.  And moving point to the start, so the user
> sees the comments.  For example:

Thanks, done.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  >   You've characterised this old behaviour as a bug, but in the
  > preferences of two actual Linux console users (Richard and myself), the
  > solution is not better than the bug.

With user interface questions, what users want is the key to what
behavior is good or bad.

Eli said it would be difficult to reliably display undisplayable
characters as diamond _on all consoles_.  But I contend that it is not
hard to implement an option to do this _on Linux consoles_.
Eli, is there something difficult about that?

This option doesn't need to go through terminfo, if that would make
it difficult.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Richard Stallman wrote:

>> You've characterised this old behaviour as a bug, but in
>> the preferences of two actual Linux console users (Richard
>> and myself), the solution is not better than the bug.
>
> With user interface questions, what users want is the key to
> what behavior is good or bad.

With this

  (set-char-table-extra-slot glyphless-char-display 0 "%")

and

  GNU Emacs 29.0.50 (build 1, x86_64-pc-linux-gnu, cairo
   version 1.16.0) of 2022-08-28 [commit
   df2f6fb7fc4b79834ae40db8be2ccdc1e4a273f1]

it's pretty equivalent IMO. The diamond, from a user
perspective, was slightly better since % is an actual char
that one use in programming and the diamond wasn't.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: eliz@gnu.org, gregory@heytings.org, emacs-devel@gnu.org
> Date: Fri, 02 Sep 2022 22:51:26 -0400
> 
> Eli said it would be difficult to reliably display undisplayable
> characters as diamond _on all consoles_.  But I contend that it is not
> hard to implement an option to do this _on Linux consoles_.
> Eli, is there something difficult about that?

Yes: I know of no way of identifying such a console except by its
name.  And hard-coding a name of a specific console into our code is
inelegant at best.  Worse, AFAIU, when people use fbterm, which
doesn't have these problems, the name of the console reported to Emacs
is sometimes also "linux", so we already see, even in this relatively
short discussion, the difficulties with using the name as the unique
identifier for this purpose.

More generally, I have no reason to believe that the "feature" of the
Linux console whereby it accepts _any_ UTF-8 encoded character and
displays the U+FFFD "diamond" replacement for those it has no glyphs,
is rock-solid enough.  Someone said that the Linux console only knows
about Unicode 5.0 and older; we are going to see Unicode 15.0 in a few
days, where there are many more characters.  To say nothing of the
Emacs's own extensions of the Unicode code-space.  These are shaky
grounds to build upon, since none of the people who are talk,ing here
in favor of such a "solution" are actual maintainers of the Linux
console, and none of actual maintainers have spoken in this
discussion.

Moreover, a safer method to get the same effect exists, and was
described here.  It just needs some user configuration, which
unfortunately only users can do, because the Linux console (AFAIU) can
be configured to use different fonts, and thus the characters it is
capable of displaying correctly cannot be known by Emacs in advance.

> This option doesn't need to go through terminfo, if that would make
> it difficult.

This kind of kludgey features leads to code that is harder to
maintain, so I'd like to avoid doing so if better ways exist.  And
they do in this case.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Eli said it would be difficult to reliably display undisplayable
  > > characters as diamond _on all consoles_.  But I contend that it is not
  > > hard to implement an option to do this _on Linux consoles_.
  > > Eli, is there something difficult about that?

  > Yes: I know of no way of identifying such a console except by its
  > name.  And hard-coding a name of a specific console into our code is
  > inelegant at best.

We are having two miscommunications here.

I did not have in mind that Emacs should automatically identify
Linux consoles or distinguish them from other text consoles.  Rather, the
feature should _reliably display diamonds_ on Linux consoles.  I can easily
make my .emacs file determine whether Emacs is on a Linux console in a way
that is reliable _in my case_, and enable this feature only in that case.

  > More generally, I have no reason to believe that the "feature" of the
  > Linux console whereby it accepts _any_ UTF-8 encoded character and
  > displays the U+FFFD "diamond" replacement for those it has no glyphs,
  > is rock-solid enough.

There is no need to set the requirement that high.  If it works for
user U, that's good enough.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Richard Stallman wrote:

> I did not have in mind that Emacs should automatically
> identify Linux consoles or distinguish them from other text
> consoles. Rather, the feature should _reliably display
> diamonds_ on Linux consoles.

Haha, give us the diamonds back! Or else ...

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Haha, give us the diamonds back! Or else ...

I laughted out loud at that.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Gregory Heytings]

>
> Well, actually instead of 16, since all the colours come in a normal and 
> a bright version.
>

Are these "bright" versions used by Emacs?  I don't think so.

>
> But these 256 colours will be getting used, and that means work 
> configuring faces that are not satisfactory to the user in the 256 
> colour version.
>

What do you mean?  It's not more necessary to configure faces in a 
terminal than to configure faces under X.  If you don't like the defaults, 
you can configure them, that's all.

>
> Here's the rub.  The "pixelated" fonts were designed to be used in, say, 
> an 8x16 matrix, and work very well indeed.  The "modern" fonts were 
> designed to be used in X, and work less well in fbterm on a pixelated 
> grid.
>

Not if you use monospaced fonts, which were designed to be used in 
terminals.

>
> All of the "modern" fonts available for fbterm on my machine are 
> "spidery" single pixel thick fonts.  On the lat1-16 font I've been using 
> in consoles for decades, the lines are two pixels thick.  I find this 
> much more readable.
>

Well, install another font?  I'd recommend DejaVu Sans Mono.

>
> There are several.  As mentioned above, you have to configure faces, 
> possibly a lot of them.
>

That's incorrect.  You don't have to configure any faces, they come 
preconfigured, just like they are preconfigured under X and for a raw 
Linux console.

>
> There are all the problems with fonts.  You've got to remember to hack 
> the fbterm binary each time a new version of fbterm comes out (unless 
> you've got a system like Gentoo, and arrange for this to happen 
> automatically).
>

If you follow the instructions in the FAQ entry, you won't have to 
remember any such hacks.

>
> I will probably continue using the plain Linux console, for all these 
> reasons.
>

You are of course free to do what you want, but you should not discourage 
others to try it.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> Date: Fri, 02 Sep 2022 11:39:04 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: Richard Stallman <rms@gnu.org>, eliz@gnu.org, emacs-devel@gnu.org
> 
> > Well, actually instead of 16, since all the colours come in a normal and 
> > a bright version.
> 
> Are these "bright" versions used by Emacs?  I don't think so.

They are, see xterm-standard-colors.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Gregory Heytings]

>>> Well, actually instead of 16, since all the colours come in a normal 
>>> and a bright version.
>>
>> Are these "bright" versions used by Emacs?  I don't think so.
>
> They are, see xterm-standard-colors.
>

Sorry, my reply was not precise enough.  I meant:

Are these "bright" versions used by Emacs running in a raw Linux console? 
I don't think so.

I just checked again, and list-colors-display only lists eight colors in a 
raw Linux console.  Under fbterm those "bright" versions become available. 
Perhaps there's a trick to activate them in a raw Linux console, though; I 
don't know.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> Date: Fri, 02 Sep 2022 12:44:19 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: acm@muc.de, rms@gnu.org, emacs-devel@gnu.org
> 
> Are these "bright" versions used by Emacs running in a raw Linux console? 
> I don't think so.

If you set the Linux console so that Emacs knows it has 16 colors,
they will be used, yes.  xterm--init does that if display-color-cells
returns 16.

> I just checked again, and list-colors-display only lists eight colors in a 
> raw Linux console.

What does display-color-cells return in that case?

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Gregory Heytings]

>> I just checked again, and list-colors-display only lists eight colors 
>> in a raw Linux console.
>
> What does display-color-cells return in that case?
>

16777216

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> Date: Fri, 02 Sep 2022 14:15:28 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: acm@muc.de, rms@gnu.org, emacs-devel@gnu.org
> 
> 
> >> I just checked again, and list-colors-display only lists eight colors 
> >> in a raw Linux console.
> >
> > What does display-color-cells return in that case?
> >
> 
> 16777216

Then I guess linux.el should call the same color setup code as
xterm.el does in this case.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Gregory Heytings]

>>>> I just checked again, and list-colors-display only lists eight colors 
>>>> in a raw Linux console.
>>>
>>> What does display-color-cells return in that case?
>>
>> 16777216
>
> Then I guess linux.el should call the same color setup code as xterm.el 
> does in this case.
>

That's the value I get on my system, is it guaranteed to be the same on 
other systems?

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Gregory Heytings]

>>>>> I just checked again, and list-colors-display only lists eight 
>>>>> colors in a raw Linux console.
>>>> 
>>>> What does display-color-cells return in that case?
>>> 
>>> 16777216
>> 
>> Then I guess linux.el should call the same color setup code as xterm.el 
>> does in this case.
>
> That's the value I get on my system, is it guaranteed to be the same on 
> other systems?
>

Answering my own question: no.  I just tried on another computer, with a 
similar kernel (5.17 instead of 5.18), and there display-color-cells 
returns 8.  So I guess we use the most conservative value here.  We could 
make the (xterm-register-default-colors xterm-standard-colors) call 
conditional to the return value of display-color-cells, though.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 1140 bytes --]


>>>>>> I just checked again, and list-colors-display only lists eight 
>>>>>> colors in a raw Linux console.
>>>>> 
>>>>> What does display-color-cells return in that case?
>>>> 
>>>> 16777216
>>> 
>>> Then I guess linux.el should call the same color setup code as 
>>> xterm.el does in this case.
>> 
>> That's the value I get on my system, is it guaranteed to be the same on 
>> other systems?
>
> Answering my own question: no.  I just tried on another computer, with a 
> similar kernel (5.17 instead of 5.18), and there display-color-cells 
> returns 8.  So I guess we use the most conservative value here.  We 
> could make the (xterm-register-default-colors xterm-standard-colors) 
> call conditional to the return value of display-color-cells, though.
>

Answering my own suggestion: no.  I just tried, and for some reason it 
doesn't work.  Here's a screenshot of list-colors-display with 
(xterm-register-default-colors xterm-standard-colors) in 
lisp/term/linux.el.  As you can see, bright colors and non-bright colors 
are displayed in the same way, and other colors are displayed wrongly. 
Perhaps there's another way, though.

[-- Attachment #2: list-colors-display.png --]
[-- Type: image/png, Size: 46492 bytes --]

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> Date: Fri, 02 Sep 2022 15:45:19 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: acm@muc.de, rms@gnu.org, emacs-devel@gnu.org
> 
> Answering my own suggestion: no.  I just tried, and for some reason it 
> doesn't work.  Here's a screenshot of list-colors-display with 
> (xterm-register-default-colors xterm-standard-colors) in 
> lisp/term/linux.el.  As you can see, bright colors and non-bright colors 
> are displayed in the same way, and other colors are displayed wrongly. 

Then I don't understand how come display-color-cells returns such a
large number.  Does terminfo returns correct color commands?

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> Date: Fri, 02 Sep 2022 19:06:44 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: acm@muc.de, rms@gnu.org, emacs-devel@gnu.org
> 
> > Answering my own suggestion: no.  I just tried, and for some reason it 
> > doesn't work.  Here's a screenshot of list-colors-display with 
> > (xterm-register-default-colors xterm-standard-colors) in 
> > lisp/term/linux.el.  As you can see, bright colors and non-bright colors 
> > are displayed in the same way, and other colors are displayed wrongly. 
> 
> Then I don't understand how come display-color-cells returns such a
> large number.  Does terminfo returns correct color commands?

More generally, what are the color commands which the Linux console
supports for more than 8 standard ANSI colors?  Does it support any of
the commands we can use, as you can see in term.c around lines
4130-4195?

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Gregory Heytings]

>> Answering my own suggestion: no.  I just tried, and for some reason it 
>> doesn't work.  Here's a screenshot of list-colors-display with 
>> (xterm-register-default-colors xterm-standard-colors) in 
>> lisp/term/linux.el.  As you can see, bright colors and non-bright 
>> colors are displayed in the same way, and other colors are displayed 
>> wrongly.
>
> Then I don't understand how come display-color-cells returns such a 
> large number.  Does terminfo returns correct color commands?
>

infocmp tells me that there are 8 colors.  So it's likely some kind of bug 
in Emacs, I guess.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Andreas Schwab]

On Sep 02 2022, Gregory Heytings wrote:

> infocmp tells me that there are 8 colors.  So it's likely some kind of bug
> in Emacs, I guess.

Unless you have overridden the color mode with the tty-color-mode frame
parameter.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Gregory Heytings]

>> infocmp tells me that there are 8 colors.  So it's likely some kind of 
>> bug in Emacs, I guess.
>
> Unless you have overridden the color mode with the tty-color-mode frame 
> parameter.
>

I didn't, I just started emacs -Q.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> Date: Fri, 02 Sep 2022 16:56:55 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: Eli Zaretskii <eliz@gnu.org>, acm@muc.de, rms@gnu.org, emacs-devel@gnu.org
> 
> >> infocmp tells me that there are 8 colors.  So it's likely some kind of 
> >> bug in Emacs, I guess.
> >
> > Unless you have overridden the color mode with the tty-color-mode frame 
> > parameter.
> 
> I didn't, I just started emacs -Q.

Is the "RGB" variable set in the environment?

More generally, can you step through the code in init_tty that
computes the TN_max_colors terminal parameter, and see why we set it
to 16777216?

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> Date: Fri, 02 Sep 2022 21:19:30 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: schwab@linux-m68k.org, acm@muc.de, rms@gnu.org,
> 	emacs-devel@gnu.org
> 
> Is the "RGB" variable set in the environment?

Sorry, I meant "COLORTERM".

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Gregory Heytings]

>> Is the "RGB" variable set in the environment?
>
> Sorry, I meant "COLORTERM".
>

That was the culprit.  I was a bug in my ~/.bashrc, not in Emacs.  I 
forgot that I originally had COLORTERM set there when TERM != linux, and 
that I changed the condition a week ago or so to TERM != fbterm.  Sorry 
for the noise.

So the conclusion is that TERM=linux implies 8 colors.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> Date: Fri, 02 Sep 2022 15:34:51 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: acm@muc.de, rms@gnu.org, emacs-devel@gnu.org
> 
> > That's the value I get on my system, is it guaranteed to be the same on 
> > other systems?
> 
> Answering my own question: no.  I just tried on another computer, with a 
> similar kernel (5.17 instead of 5.18), and there display-color-cells 
> returns 8.  So I guess we use the most conservative value here.

We should use the actual value, not the most conservative one.

> We could make the (xterm-register-default-colors
> xterm-standard-colors) call conditional to the return value of
> display-color-cells, though.

If the value is >= 16, I believe that code does already do that, no?

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> Date: Fri, 02 Sep 2022 15:27:24 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: acm@muc.de, rms@gnu.org, emacs-devel@gnu.org
> 
> 
> >>>> I just checked again, and list-colors-display only lists eight colors 
> >>>> in a raw Linux console.
> >>>
> >>> What does display-color-cells return in that case?
> >>
> >> 16777216
> >
> > Then I guess linux.el should call the same color setup code as xterm.el 
> > does in this case.
> >
> 
> That's the value I get on my system, is it guaranteed to be the same on 
> other systems?

As long as display-color-cells returns a value greater or equal to 16,
I think the result will be correct for the number of colors actually
supported?  If display-color-cells says 8, then we don't need to do
anything.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Alan Mackenzie]

Hello, Gregory.

On Fri, Sep 02, 2022 at 11:39:04 +0000, Gregory Heytings wrote:


> > Well, actually instead of 16, since all the colours come in a normal and 
> > a bright version.


> Are these "bright" versions used by Emacs?  I don't think so.

Yes they are, by activating the bold weight in the face customisation.
On XEmacs, there were 16 distinct colours.

> > But these 256 colours will be getting used, and that means work 
> > configuring faces that are not satisfactory to the user in the 256 
> > colour version.


> What do you mean?  It's not more necessary to configure faces in a 
> terminal than to configure faces under X.  If you don't like the defaults, 
> you can configure them, that's all.

Nobody likes all the defaults, and it's a continual annoyance coming
across faces you don't like, and having to change them one by one.
That's my experience, anyway.

> > Here's the rub.  The "pixelated" fonts were designed to be used in, say, 
> > an 8x16 matrix, and work very well indeed.  The "modern" fonts were 
> > designed to be used in X, and work less well in fbterm on a pixelated 
> > grid.

> Not if you use monospaced fonts, which were designed to be used in 
> terminals.

In my experience, they work less well than the fonts for the Linux
console.

> > All of the "modern" fonts available for fbterm on my machine are 
> > "spidery" single pixel thick fonts.  On the lat1-16 font I've been using 
> > in consoles for decades, the lines are two pixels thick.  I find this 
> > much more readable.


> Well, install another font?  I'd recommend DejaVu Sans Mono.

That was the first one I tried this morning.  I didn't like it.  It's a
"spidery" font like all the other ones I have available.

[ .... ]

> > There are all the problems with fonts.  You've got to remember to hack 
> > the fbterm binary each time a new version of fbterm comes out (unless 
> > you've got a system like Gentoo, and arrange for this to happen 
> > automatically).

> If you follow the instructions in the FAQ entry, you won't have to 
> remember any such hacks.

Surely that depends on the package manager you use?  What am I missing
here?

> > I will probably continue using the plain Linux console, for all these 
> > reasons.

I just came across another problem.  The GPM mouse utility won't transfer
text between an fbterm console and another console, not even a second
fbterm.

> You are of course free to do what you want, but you should not discourage 
> others to try it.

I think you were being too positive about the topic, and felt I should
express the other side of the argument.  By sharing my experience, I
might be saving time for other people.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Gregory Heytings]

>
> In my experience, they work less well than the fonts for the Linux 
> console.
>

Your experience is rather short, isn't it?  You objected to that solution 
as soon as it was suggested.  Please try again with a better font.  What 
you see under fbterm is exactly (with respect to fonts) what you would see 
in a terminal emulator under X, including anti-aliasing.

>
> That was the first one I tried this morning.  I didn't like it.  It's a 
> "spidery" font like all the other ones I have available.
>

Can you provide a screenshot of Emacs under fbterm with that font and one 
of Emacs in the raw Linux console?  I'm not sure I understant what you 
mean by "spidery".

>
> I just came across another problem.  The GPM mouse utility won't 
> transfer text between an fbterm console and another console, not even a 
> second fbterm.
>

I don't know.  In any case, that seems to be a very minor problem.

>
> I think you were being too positive about the topic, and felt I should 
> express the other side of the argument.
>

I don't think you are expressing the other side of the argument.  You are 
listing very minor problems (and non-problems, such as when you said that 
"you have to configure faces").

By the way, if you don't want 256 colors and want to use only 8 colors as 
you did before, just don't set TERM=fbterm, or set TERM=linux.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Alan Mackenzie]

Hello, Gregory.

On Fri, Sep 02, 2022 at 13:41:14 +0000, Gregory Heytings wrote:


> > In my experience, they work less well than the fonts for the Linux 
> > console.


> Your experience is rather short, isn't it?  You objected to that solution 
> as soon as it was suggested.  Please try again with a better font.

Suggest one to me, please.  I'm not an expert on X-Windows fonts.

> What you see under fbterm is exactly (with respect to fonts) what you
> would see in a terminal emulator under X, including anti-aliasing.

I detest anti-aliasing.  It makes everything go blurred for me.  Maybe
that's a reason why I don't like these fonts.

> > That was the first one I tried this morning.  I didn't like it.  It's a 
> > "spidery" font like all the other ones I have available.


> Can you provide a screenshot of Emacs under fbterm with that font and one 
> of Emacs in the raw Linux console?  I'm not sure I understand what you 
> mean by "spidery".

No, I don't know how to take a screen shot of a console.  But the utility
psf2txt displays the bitmaps of all characters in a console font.  The
image for C in lat1-16.psfu looks like:

// Character 67
Bitmap: -------- \
        -------- \
        --####-- \
        -##--##- \
        ##----#- \
        ##------ \
        ##------ \
        ##------ \
        ##------ \
        ##----#- \
        -##--##- \
        --####-- \
        -------- \
        -------- \
        -------- \
        --------
Unicode: [00000043];
%

Note that the main body of the character is two pixels thick.  This
doesn't appear to be the case for any of the fonts I've tried in fbterm,
which appear to be just one pixel thick; or to use anti-aliasing or other
tricks.  That one pixel thickness is what I mean by "spidery", in the
sense of "as thin as a spider's web".

> > I just came across another problem.  The GPM mouse utility won't 
> > transfer text between an fbterm console and another console, not even a 
> > second fbterm.

> I don't know.  In any case, that seems to be a very minor problem.

It's not minor to me.  Without GPM, I'd have to alter my workflow
considerably.

> > I think you were being too positive about the topic, and felt I should 
> > express the other side of the argument.

> I don't think you are expressing the other side of the argument.  You are 
> listing very minor problems (and non-problems, such as when you said that 
> "you have to configure faces").

I think you said earlier on in the thread that there were "no known
inconveniences" with fbterm.  I have listed some inconveniences, even if
I've called them problems.  I think you were mistaken on this point.

> By the way, if you don't want 256 colors and want to use only 8 colors as 
> you did before, just don't set TERM=fbterm, or set TERM=linux.

Well, it's not just as before - as I said earlier, hi-green comes out as
brown.  That's the sort of thing I meant when I said that you had to
configure faces.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Gregory Heytings]

>
> I detest anti-aliasing.  It makes everything go blurred for me.  Maybe 
> that's a reason why I don't like these fonts.
>

Well, you can easily turn of anti-aliasing if that's what you prefer:

font-names=DejaVu\ Sans\ Mono:antialias=false

>
> No, I don't know how to take a screen shot of a console.
>

You can use fbcat.

>> I don't know.  In any case, that seems to be a very minor problem.
>
> It's not minor to me.  Without GPM, I'd have to alter my workflow 
> considerably.
>

Sorry, I don't use GPM.  I'm pretty sure it's not hard to fix, however, 
the manpage of fbterm says "copy/past selected text between windows with 
mouse when gpm server is running", so it's something that should be 
feasible somehow.

>
> I think you said earlier on in the thread that there were "no known 
> inconveniences" with fbterm.
>

Yes, at that time there were no known inconveniences.  Now there is 
(perhaps) a minor one with GPM.

>> By the way, if you don't want 256 colors and want to use only 8 colors 
>> as you did before, just don't set TERM=fbterm, or set TERM=linux.
>
> Well, it's not just as before - as I said earlier, hi-green comes out as 
> brown.  That's the sort of thing I meant when I said that you had to 
> configure faces.
>

I don't know what you do exactly, but here (with the latest master or 
emacs-28):

fbterm -- env TERM=fbterm emacs

uses 256 colors, and

fbterm -- env TERM=linux emacs

uses 8 colors.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Stefan Monnier]

>> I detest anti-aliasing.  It makes everything go blurred for me.
>> Maybe that's a reason why I don't like these fonts.
> Well, you can easily turn of anti-aliasing if that's what you prefer:
>
> font-names=DejaVu\ Sans\ Mono:antialias=false

AFAIK you can control the fonts just like you do under X11.  But it can
be difficult to find a good alternative font when you grew fond
a particular choice, indeed (I'm still using the bitmapped
misc-fixed-semicondensed here :-)

>>> I don't know.  In any case, that seems to be a very minor problem.
>> It's not minor to me.  Without GPM, I'd have to alter my
>> workflow considerably.
> Sorry, I don't use GPM.  I'm pretty sure it's not hard to fix, however, the
> manpage of fbterm says "copy/past selected text between windows with mouse
> when gpm server is running", so it's something that should be
> feasible somehow.

I suspect that this text means you can do copy&paste between different
"windows" of a single `fbterm` (which gives you something similar to
Linux's multiple consoles but uses a single console) but not between
different `fbterm` running in different consoles.


        Stefan

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Yuri Khan]

On Fri, 2 Sept 2022 at 22:39, Alan Mackenzie <acm@muc.de> wrote:

> > Your experience is rather short, isn't it?  You objected to that solution
> > as soon as it was suggested.  Please try again with a better font.
>
> Suggest one to me, please.  I'm not an expert on X-Windows fonts.

Based on your dislikes stated in this thread, you might like fonts
distributed from <https://int10h.org/oldschool-pc-fonts/>, also
packaged in Debian and derivatives as fonts-pc and fonts-pc-extra. Do
note that they are meant to be used at a few exact point sizes.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Tomas Hlavaty]

On Fri 02 Sep 2022 at 15:38, Alan Mackenzie <acm@muc.de> wrote:
>> Can you provide a screenshot of Emacs under fbterm with that font and one 
>> of Emacs in the raw Linux console?  I'm not sure I understand what you 
>> mean by "spidery".
>
> No, I don't know how to take a screen shot of a console.

One option is something like:

   cat /dev/fb0 >/tmp/a

Another option is the function framebuffer-screenshot in
emacs-framebuffer
https://logand.com/sw/emacs-framebuffer/file/emacs-framebuffer.el.html#l754

>> > I just came across another problem.  The GPM mouse utility won't 
>> > transfer text between an fbterm console and another console, not even a 
>> > second fbterm.
>
>> I don't know.  In any case, that seems to be a very minor problem.
>
> It's not minor to me.  Without GPM, I'd have to alter my workflow
> considerably.

And those dismissed minor problems add up.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Jean Louis]

* Tomas Hlavaty <tom@logand.com> [2022-09-03 08:36]:
> On Fri 02 Sep 2022 at 15:38, Alan Mackenzie <acm@muc.de> wrote:
> >> Can you provide a screenshot of Emacs under fbterm with that font and one 
> >> of Emacs in the raw Linux console?  I'm not sure I understand what you 
> >> mean by "spidery".
> >
> > No, I don't know how to take a screen shot of a console.

In FBTerm you can run utility `screen' and then follow instructions:

https://www.linuxquestions.org/questions/linux-server-73/can-i-take-a-text-screenshot-of-a-gnu-screen-session-932703/

Something like:

screen -S SESSIONAME -X -p0 hardcopy

That way you will send command "hardcopy" and get the file hardcopy.0
with the text inside. It may no be what you want in this case.

Otherwise use:

Description     : A framebuffer screenshot grabber
URL             : https://github.com/GunnarMonell/fbgrab
Licenses        : GPL



-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Emanuel Berg]

Jean Louis wrote:

>>>> Can you provide a screenshot of Emacs under fbterm with
>>>> that font and one of Emacs in the raw Linux console?
>>>> I'm not sure I understand what you mean by "spidery".
>>>
>>> No, I don't know how to take a screen shot of a console.
>
> In FBTerm you can run utility `screen' and then follow
> instructions [...]

You can but there's no need for that.

Linux VTs from Emacs: [here, dump = screenshot]

(defun dump (name)
  (interactive "sname: ")
  (message nil)
  (let*((file (format "~/%s.png" name))
        (cmd  (format "fbgrab %s" file)) )
    (shell-command cmd) ))

Linux VTs from Linux VTs:

  $ sudo fbgrab -c VT NAME

https://dataswamp.org/~incal/conf/.zsh/dump
https://dataswamp.org/~incal/emacs-init/vt.el

-- 
underground experts united
https://dataswamp.org/~incal

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Gregory Heytings]

>
> And those dismissed minor problems add up.
>

That's unfair.  A significant amount of time and energy has been spent in 
this thread to solve each reported problem.  If you experience other minor 
(or major!) problems, tell us what they are, instead of telling that 
problems "add up" without corresponding solutions.

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Akib Azmain Turja]

[-- Attachment #1: Type: text/plain, Size: 2783 bytes --]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > Anyway,
>   > I workarounded it by modifying glyphless-char-display with the following
>   > in terminal buffers:
>
>   > --8<---------------cut here---------------start------------->8---
>   > (set-char-table-extra-slot glyphless-char-display 0 'thin-space)
>   > --8<---------------cut here---------------end--------------->8---
>
>   > This shows a single space character.  
>
> Could a variant of that display a diamond instead of a space?

I couldn't find one.  I tried the following, but it didn't work:

--8<---------------cut here---------------start------------->8---
(set-char-table-extra-slot glyphless-char-display 0 "�")
--8<---------------cut here---------------end--------------->8---

It didn't show the diamond (or, in my case, a character similar to a
question mark in inverse video), which is the expected behavior
according to the manual, but it didn't even show the diamond in a box
(or square brackets), instead just a empty box appeared, which is
unexpected.

>
> Eli wrote
>
>   > One way of verifying this is to type
>
>   >    C-x 8 RET fffd RET
>
>   > in an Emacs session which displays on the Linux console, and see if
>   > that shows the diamond-like glyph or the \Unnnn thing.  If it's the
>   > latter, it means the console doesn't support that character, and we
>   > cannot send its code from Emacs.

How do Emacs know that determine whether a character is supported or
not?  Using the encoding used for output?

>
> but what is so bad about sending a glyph that the terminal can't
> display?  What bad results does it cause?

As long as the character can be encoded to the encoding the terminal
understand, AFAIK it's fine.  But if the character can't be encoding,
it's impossible, because you don't know how to send it and the terminal
doesn't know how to read it.

>
> Or how about fixing it by sending an official code for that diamond glyph?

I think it's better.  But the format of glyphless-char-display should be
changed to allow something like the following too, which won't put the
text in a box:

--8<---------------cut here---------------start------------->8---
(set-char-table-extra-slot glyphless-char-display 0 (verbatim . "�"))
--8<---------------cut here---------------end--------------->8---

-- 
Akib Azmain Turja

Find me on Mastodon at @akib@hostux.social.

This message is signed by me with my GnuPG key.  Its fingerprint is:

    7001 8CE5 819F 17A3 BBA6  66AF E74F 0EFA 922A E7F5

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 644 bytes --]


>
> I couldn't find one.  I tried the following, but it didn't work:
>
> (set-char-table-extra-slot glyphless-char-display 0 "�")
>
> It didn't show the diamond (or, in my case, a character similar to a 
> question mark in inverse video), which is the expected behavior 
> according to the manual, but it didn't even show the diamond in a box 
> (or square brackets), instead just a empty box appeared, which is 
> unexpected.
>

Eli already explained (twice IIRC) that you can only use ASCII characters 
there, and that's also what the docstring of glyphless-char-display says: 
"its value should be an ASCII acronym string".

Re: Display of undisplayable characters: \U01F3A8 instead of diamond

[Eli Zaretskii]

> From: Akib Azmain Turja <akib@disroot.org>
> Cc: emacs-devel@gnu.org
> Date: Fri, 02 Sep 2022 17:47:39 +0600
> 
> >   > (set-char-table-extra-slot glyphless-char-display 0 'thin-space)
> >   > --8<---------------cut here---------------end--------------->8---
> >
> >   > This shows a single space character.  
> >
> > Could a variant of that display a diamond instead of a space?
> 
> I couldn't find one.  I tried the following, but it didn't work:
> 
> --8<---------------cut here---------------start------------->8---
> (set-char-table-extra-slot glyphless-char-display 0 "�")
> --8<---------------cut here---------------end--------------->8---
> 
> It didn't show the diamond (or, in my case, a character similar to a
> question mark in inverse video), which is the expected behavior
> according to the manual, but it didn't even show the diamond in a box
> (or square brackets), instead just a empty box appeared, which is
> unexpected.

It shouldn't work, indeed.  See the doc string of
glyphless-char-display.

> >   > One way of verifying this is to type
> >
> >   >    C-x 8 RET fffd RET
> >
> >   > in an Emacs session which displays on the Linux console, and see if
> >   > that shows the diamond-like glyph or the \Unnnn thing.  If it's the
> >   > latter, it means the console doesn't support that character, and we
> >   > cannot send its code from Emacs.
> 
> How do Emacs know that determine whether a character is supported or
> not?  Using the encoding used for output?

Emacs uses char-displayable-p for that purpose.

> > but what is so bad about sending a glyph that the terminal can't
> > display?  What bad results does it cause?
> 
> As long as the character can be encoded to the encoding the terminal
> understand, AFAIK it's fine.

That's insufficient.  The Linux console uses UTF-8 encoding, but it
doesn't support all the Unicode characters, although they all can be
encoded by UTF-8.  AFAIR, the Linux console supports at most 256
characters.

> > Or how about fixing it by sending an official code for that diamond glyph?
> 
> I think it's better.

If that is what is wanted, we already have in Emacs the means to set
the display to do it, and I already explained how to set that up.

> But the format of glyphless-char-display should be
> changed to allow something like the following too, which won't put the
> text in a box:
> 
> --8<---------------cut here---------------start------------->8---
> (set-char-table-extra-slot glyphless-char-display 0 (verbatim . "�"))
> --8<---------------cut here---------------end--------------->8---

It won't, because the extra slot is the fallback, the last resort.  So
it cannot use any fancy non-ASCII characters.  Which I also explained
here.