On the masking of undisplayable characters

On the masking of undisplayable characters

[bruce.connor.am]

[-- Attachment #1: Type: text/plain, Size: 730 bytes --]

I just noticed we have a feature where we use display tables to mask
characters that cannot be displayed.

When this happens, it caused me quite a bit of confusion, as I tried to
type one thing and something else got inserted. I eventually thought of
investigating the character (which revealed it was being masked by the
display table), but many users wouldn't know to do this.

Could we have a message to let the user know when this happens? Something
like "undisplayable char being displayed as X".

I know this is meant to be helpful, but displaying something different from
what's in the file is a pretty big deal. I'm not entirely sure it should be
done without consent, but it definitely shouldn't be done silently.

Artur

[-- Attachment #2: Type: text/html, Size: 850 bytes --]

Re: On the masking of undisplayable characters

[Eli Zaretskii]

> Date: Mon, 6 Jul 2015 18:38:13 +0100
> From: <bruce.connor.am@gmail.com>
> 
> I just noticed we have a feature where we use display tables to mask characters
> that cannot be displayed. 

What characters are you talking about, specifically?

> When this happens, it caused me quite a bit of confusion, as I tried to type
> one thing and something else got inserted. I eventually thought of
> investigating the character (which revealed it was being masked by the display
> table), but many users wouldn't know to do this. 
> 
> Could we have a message to let the user know when this happens? Something like
> "undisplayable char being displayed as X". 
> 
> I know this is meant to be helpful, but displaying something different from
> what's in the file is a pretty big deal. I'm not entirely sure it should be
> done without consent, but it definitely shouldn't be done silently. 

It's hard to respond without knowing the details.  Most of the uses of
display tables I know of are there for ages, so adding some kind of
message about that now would seem inappropriate.  (And when exactly do
we display that message? when the character is first displayed?)

Then again, I think you might be over-reacting a bit: Emacs always
displayed special characters specially.  Think about the ^X control
characters, the glyphless-char-display feature, composed characters,
even the newline.  There's nothing new here.

Re: On the masking of undisplayable characters

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 1967 bytes --]

> What characters are you talking about, specifically?

Round quotes were displayed as hard quotes. The reason this happened on my
pc was a bug, but I understand that for systems that really can't display
them this is what will happen.

I didn't check whether this applies to any other characters. I was just
surprised to find out the file was not being displayed as-is without so
much as a notice.

> It's hard to respond without knowing the details.  Most of the uses of
> display tables I know of are there for ages, so adding some kind of
> message about that now would seem inappropriate.  (And when exactly do
> we display that message? when the character is first displayed?)

When visiting the file and when the character is typed for the first time,
would be an option.

> Then again, I think you might be over-reacting a bit:

Perhaps. I didn't mean to sound dramatic.
If this only happens for quotes it's less of an issue, but it could grow
with time.

I'm saying that I (and maybe other people) expect that the contents of a
buffer are reflected in what's displayed. I'm perfectly fine with modes and
options that change that (like prettify-symbols, for instance), but having
this masking as a default surprised me quite a bit.

> Emacs always
> displayed special characters specially.  Think about the ^X control
> characters,

That's fine. When I see a blue ^X, I know that's a control character, it's
very clear that it's not a ^ with an X.

> the glyphless-char-display feature,

I don't know which characters are affected by this.

> composed characters,

You mean like a non-spacing ` and a looking like an à? The non-spacing
characters are specifically designed to do that, it's not the same as
displaying one quote as another quote.

> even the newline.

You mean that \n and \r\n are displayed the same way? Yeah, I've been bit
several times by it, but at least it's indicated in the mode line.

[-- Attachment #2: Type: text/html, Size: 2329 bytes --]

Re: On the masking of undisplayable characters

[Juri Linkov]

>> Emacs always
>> displayed special characters specially.  Think about the ^X control
>> characters,
>
> That's fine. When I see a blue ^X, I know that's a control character, it's
> very clear that it's not a ^ with an X.

Highlighted using the ‘escape-glyph’ face.

>> the glyphless-char-display feature,

The ‘glyphless-char’ face.

>> composed characters,
>
> You mean like a non-spacing ` and a looking like an à? The non-spacing
> characters are specifically designed to do that, it's not the same as
> displaying one quote as another quote.

Another face intended to highlight confusable characters is ‘nobreak-space’.
But note how ‘gnus-article-mode’ uses

  (set (make-local-variable 'nobreak-char-display) nil)

to prevent Gnus from displaying non-break space with `nobreak-space' face
because usually in view modes users don't want displaying special characters
specially.

>> even the newline.
>
> You mean that \n and \r\n are displayed the same way? Yeah, I've been bit
> several times by it, but at least it's indicated in the mode line.

I've been bit more by indicating different codings in the mode line
with the same code letter, e.g. using the same letter ‘U’ for
utf-8, utf-16le, utf-16be, etc.  Better would be to display coding systems
as full names but this makes the mode line wider.

Re: On the masking of undisplayable characters

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel <emacs-devel@gnu.org>
> Date: Tue, 07 Jul 2015 02:07:36 +0300
> 
> >> the glyphless-char-display feature,
> 
> The ‘glyphless-char’ face.

It's not just the face, the feature is much more complex than just
showing characters in a particular face.

Re: On the masking of undisplayable characters

[Eli Zaretskii]

> Date: Tue, 07 Jul 2015 05:44:13 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: bruce.connor.am@gmail.com, emacs-devel@gnu.org
> 
> > From: Juri Linkov <juri@linkov.net>
> > Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel <emacs-devel@gnu.org>
> > Date: Tue, 07 Jul 2015 02:07:36 +0300
> > 
> > >> the glyphless-char-display feature,
> > 
> > The ‘glyphless-char’ face.
> 
> It's not just the face, the feature is much more complex than just
> showing characters in a particular face.

Of course, if people would favor that, we could make these quotes
display with some distinct face, since display tables support that
(see 'make-glyph-code').

Re: On the masking of undisplayable characters

[Eli Zaretskii]

> Date: Mon, 6 Jul 2015 20:57:08 +0100
> From: Artur Malabarba <bruce.connor.am@gmail.com>
> Cc: emacs-devel <emacs-devel@gnu.org>
> 
> > What characters are you talking about, specifically?
> 
> Round quotes were displayed as hard quotes. The reason this happened on my pc
> was a bug, but I understand that for systems that really can't display them
> this is what will happen. 
> 
> I didn't check whether this applies to any other characters. I was just
> surprised to find out the file was not being displayed as-is without so much as
> a notice.

It should happen only with quotes, and presumably only in
documentation-related buffers.  The rationale is that we cannot afford
showing illegible documentation to users just because their terminal
is incapable of displaying these important characters.

I suggested to limit this to windows showing *Help* buffers, but it
was decided to wait until someone complains.

> > It's hard to respond without knowing the details. Most of the uses of
> > display tables I know of are there for ages, so adding some kind of
> > message about that now would seem inappropriate. (And when exactly do
> > we display that message? when the character is first displayed?)
> 
> When visiting the file and when the character is typed for the first time,
> would be an option.

Not really: the display engine doesn't examine any of those
characters, until they are on or very near to the visible portion of
the buffer.

> > the glyphless-char-display feature,
> 
> I don't know which characters are affected by this.

Those that cannot be displayed by any fonts.

Re: On the masking of undisplayable characters

[Artur Malabarba]

> It should happen only with quotes, and presumably only in
> documentation-related buffers.  The rationale is that we cannot afford
> showing illegible documentation to users just because their terminal
> is incapable of displaying these important characters.
>
> I suggested to limit this to windows showing *Help* buffers, but it
> was decided to wait until someone complains.

I would be much more confortable if this only turned on by default in
buffers where we _know_ that's ok, such as *help* buffers, info
buffers, and maybe Emacs sources.
What bothers me is that it'll happen on any buffer without
confirmation or notification.

Let's say the user is reading some code designed to read a stream and
do something with quotation marks. Or maybe the user is reading prose
in a context where typography is important.
Usually they'll have a font to display these round quotations, but if
they start emacs without a graphical frame for one reason or another
(or, god forbid, they've misconfigured something) then it would a
problem that they see " when they actually have “ or ”.

Now, those are not very likely scenarios, that's why it bothers me a
little less if it's just quotation marks. But I wanted to make sure
we're aware of this, specially if this feature starts spreading to
other characters.

> Not really: the display engine doesn't examine any of those
> characters, until they are on or very near to the visible portion of
> the buffer.

Yeah, I see how that would be complicated.

I guess I'll just step down from soap box now.

Re: On the masking of undisplayable characters

[Yuri Khan]

On Tue, Jul 7, 2015 at 3:32 PM, Artur Malabarba
<bruce.connor.am@gmail.com> wrote:

> Let's say the user is reading some code designed to read a stream and
> do something with quotation marks. Or maybe the user is reading prose
> in a context where typography is important.

That description sounds like me, and in my view the context includes:

* any text-based files meant to be read by humans;
* any user-directed text strings in program code;
* any comments in program code that are meant to be read by humans (as
opposed to compiler pragma directives).

> Usually they'll have a font to display these round quotations, but if
> they start emacs without a graphical frame for one reason or another
> (or, god forbid, they've misconfigured something) then it would a
> problem that they see " when they actually have “ or ”.

If I have misconfigured something and see straight quotes instead of
curly quotes in an X frame, I will notice it right away and start
investigating.

The same applies to a terminal frame running in an X-based terminal emulator.

If I am on a Linux terminal, typography will be the least of my
concern. I will be busy getting my X display up and running again, or
getting my system working, or resolving whatever other disaster that
forced me out of X. Heck, even my native alphabet is not displayed
properly on the Linux tty.

On a Windows machine in a graphical frame, the situation is analogous to X.

On a Windows console with a proper vector font, the situation is
analogous to an X-based terminal.

On a Windows console with a bitmap font, I would expect degraded
typography, and either use C-x = on each suspect quote, or take my
time migrating to a better environment.

(Side note: Two editors that I sometimes use beside Emacs display the
code of the character under cursor in their status lines. This is very
convenient when checking for unusual characters. Emacs can be set up
to do the same by adding “(:eval (format "%06x" (following-char)))” to
the mode-line-format variable.)

Re: On the masking of undisplayable characters

[Artur Malabarba]

> If I have misconfigured something and see straight quotes instead of
> curly quotes in an X frame, I will notice it right away and start
> investigating.

Yep, that's what I did too. I'm wondering how long it might take a
less Emacs-savy user to learn that he can `C-x ='.

Re: On the masking of undisplayable characters

[Yuri Khan]

On Tue, Jul 7, 2015 at 6:29 PM, Artur Malabarba
<bruce.connor.am@gmail.com> wrote:
>> If I have misconfigured something and see straight quotes instead of
>> curly quotes in an X frame, I will notice it right away and start
>> investigating.
>
> Yep, that's what I did too. I'm wondering how long it might take a
> less Emacs-savy user to learn that he can `C-x ='.

I imagine there are more Emacs users who know about “C-x =” than there
are Emacs users who care about curly quotes.

Re: On the masking of undisplayable characters

[Artur Malabarba]

> I imagine there are more Emacs users who know about “C-x =” than there
> are Emacs users who care about curly quotes.

And I have no idea either way. :)
So I suppose we'll wait and see.

Re: On the masking of undisplayable characters

[N. Jackson]

At 09:29 -0300 on Tuesday 2015-07-07, Artur Malabarba wrote:

>> If I have misconfigured something and see straight quotes instead of
>> curly quotes in an X frame, I will notice it right away and start
>> investigating.
>
> Yep, that's what I did too. I'm wondering how long it might take a
> less Emacs-savy user to learn that he can `C-x ='.

FWIW, it took me, your "less Emacs-savy user", about ten years to
discover `C-x ='. (I found M-x describe-char only slightly sooner.)
Before that, if I was suspicious about characters being displayed I'd
open the file in a hex editor. (It took me until today to discover Hexl
Mode in Emacs!)

While I'm here, I may as well say that I'm not happy with Emacs changing
/ misdisplaying characters on me (/ for me) without some sort of visual
indication or message, and it would be nice to have a user option to
tell Emacs not to make such substitutions. An empty box (or one of those
boxes with the number of the codepoint in them) would be fine for me if
none of my available fonts can display the character.

Re: On the masking of undisplayable characters

[Stefan Monnier]

> I would be much more confortable if this only turned on by default in
> buffers where we _know_ that's ok, such as *help* buffers, info

Agreed (unless we use an escape-glyph kind of thing to bring it to the
user's attention).


        Stefan