bug#27544: 25.1; Visualization of Unicode bidirectional marks

bug#27544: 25.1; Visualization of Unicode bidirectional marks

[Itai Berli]

Emacs supports 12 Unicode bidirectional marks (ALM, RLM, LRM, LRE,
RLE, LRO, RLO, PDF, FSI, LRI, RLI, and PDI), each of which displays as
a very thin space. This raises two problems.

1. On the one hand, the fact that these inherently invisible
marks manifest, by default, as thin spaces undermines attempts at
precise alignment and positioning. Moreover, in the case of LRM, RLM
and ALM, this behavior contradicts explicit directions given in the
Unicode
Bidirectional Algorithm 8.0.0 specifications (section 2.6 Implicit
Directional Marks):
> they do not appear in the display
(To my understanding, this is meant to apply to all bidi marks, even
if only stated explicitly for LRM, RLM and ALM.)

2. On the other hand, the fact that these spaces are so thin as to be
barely noticeable, and the fact that
they are indistinguishable from one another makes it difficult to debug
and resolve strange and/or erroneous behavior that can happen in a
bidi document, an example of which is given below.

The solution to both problems is to make the bidi marks visible in
`whitespace` mode only, and to give them glyphs that are (a) easy to
notice, (b) distinguishable from other whitespace visualization glyphs, (c)
distinct from one another.

The following example exhibits strange behavior that can arise due to
the use of bidi marks. This behavior is difficult to debug
without visualizing the bidi marks.

Consider the following paragraph.

ILLUSTRATION #1: An English sentence that is formatted from right to left.
http://imgur.com/is1OBtM

The paragraph is entirely in English, then why is it formatted from right
to left? Without visible bidi marks, it's hard to tell; however a savvy
Unicode-aware person would realize that this must indicate the presence
of a Right-To-Left Mark (U+200F). Therefore, if we position the cursor
at the beginning of the paragraph (`C-a`), and delete the following
character (`C-d`), the sentence should display normally.

ILLUSTRATION #2: Deleting the first Right-To-Left mark at the
beginning of the paragraph has no effect.
http://imgur.com/eBVpdZA

Against our expectations, nothing appears to have changed. There must
be another Right-To-Left mark at the beginning of the paragraph. Let's
delete it as
well. (`C-d`)

ILLUSTRATION #3: Deleting the second Right-To-Left Mark left-aligns
the paragraph, but leave the comma misplaced.
http://imgur.com/Klj3lZC

The paragraph is now aligned to the left, as it should, and everything
looks normal, except for the comma, which appears in the beginning of
the paragraph. But this can be easily remedied: let's delete the comma
and then retype it in its proper place. We position the cursor at the
beginning of the paragraph (`C-a`) and delete the following character (`C-d`).

ILLUSTRATION #4: After trying to delete the comma, the paragraph is
finally displayed correctly.
http://imgur.com/3w73MxM

Instead of deleting the comma, this has shifted the comma to it's
correct position.

If we were able to visualize the whitespace, we would have realized from
the beginning that the sequence of characters in this paragraph was, from
left to right:

RTL-RTL-RLO-RLO-H-e-l-l-o-PDO-,-PDO-SPACE-w-o-r-l-d-!

Thus, our first three actions removed the first three characters, leaving us
with:

RLO-H-e-l-l-o-PDO-,-PDO-SPACE-w-o-r-l-d-!

We now realize that even the final, correct form, is in fact littered
with bidi errors and potential landmines!


In GNU Emacs 25.1.1 (x86_64-apple-darwin13.4.0, NS appkit-1265.21
Version 10.9.5 (Build 13F1911))
 of 2016-09-21 built on builder10-9.porkrind.org
Windowing system distributor 'Apple', version 10.3.1504
Configured using:
 'configure --with-ns '--enable-locallisppath=/Library/Application
 Support/Emacs/${version}/site-lisp:/Library/Application
 Support/Emacs/site-lisp' --with-modules'

Configured features:
NOTIFY ACL GNUTLS LIBXML2 ZLIB TOOLKIT_SCROLL_BARS NS MODULES

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: TeX/P

Minor modes in effect:
  diff-auto-refine-mode: t
  TeX-PDF-mode: t
  ivy-mode: t
  shell-dirtrack-mode: t
  projectile-mode: t
  helm-descbinds-mode: t
  async-bytecomp-package-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent messages:
Applying style hooks... done
Mark set
C-> is undefined
Mark set [2 times]
Saving file /Users/itaiberli/Documents/GitHub/Thesis/test22.tex...
Wrote /Users/itaiberli/Documents/GitHub/Thesis/test22.tex
Undo!
Auto-saving...done
repeat-complex-command: There are no previous complex commands to repeat
delete-backward-char: Text is read-only

Load-path shadows:
/Users/itaiberli/.emacs.d/elpa/seq-2.20/seq hides
/Applications/Emacs.app/Contents/Resources/lisp/emacs-lisp/seq

Features:
(shadow sort mail-extr emacsbug message rfc822 mml mml-sec epg mm-decode
mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
sendmail rfc2047 rfc2045 ietf-drums mail-utils vc-git diff-mode tex-bar
toolbar-x font-latex plain-tex tex-buf latex tex-ispell tex-style tex
crm tex-mode latexenc colir color counsel jka-compr esh-util etags xref
project swiper reftex reftex-vars two-column ivy delsel ivy-overlay
helm-projectile helm-files rx image-dired tramp tramp-compat
tramp-loaddefs trampver shell pcomplete format-spec dired-x dired-aux
ffap helm-tags helm-bookmark helm-adaptive helm-info bookmark pp
helm-external helm-net browse-url xml url url-proxy url-privacy
url-expand url-methods url-history url-cookie url-domsuf url-util
url-parse auth-source gnus-util mm-util help-fns mail-prsvr
password-cache url-vars mailcap helm-buffers helm-grep helm-regexp
helm-utils helm-locate helm-help helm-types projectile grep compile
comint ansi-color ring ibuf-ext ibuffer thingatpt helm-descbinds helm
easy-mmode helm-source cl-seq eieio-compat eieio eieio-core
helm-multi-match helm-lib dired helm-config helm-easymenu cl-macs
async-bytecomp async advice edmacro kmacro finder-inf tex-site info
package epg-config seq byte-opt gv bytecomp byte-compile cl-extra
help-mode easymenu cconv cl-loaddefs pcase cl-lib time-date mule-util
tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel ns-win ucs-normalize term/common-win tool-bar dnd fontset image
regexp-opt fringe tabulated-list newcomment elisp-mode lisp-mode
prog-mode register page menu-bar rfn-eshadow timer select scroll-bar
mouse jit-lock font-lock syntax facemenu font-core frame cl-generic cham
georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese charscript case-table epa-hook
jka-cmpr-hook help simple abbrev minibuffer cl-preloaded nadvice
loaddefs button faces cus-face macroexp files text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote kqueue cocoa ns multi-tty
make-network-process emacs)

Memory information:
((conses 16 359730 16119)
 (symbols 48 34262 0)
 (miscs 40 100 221)
 (strings 32 65306 15883)
 (string-bytes 1 1997869)
 (vectors 16 60314)
 (vector-slots 8 1721804 214602)
 (floats 8 589 398)
 (intervals 56 269 0)
 (buffers 976 19))

bug#27544: 25.1; Visualization of Unicode bidirectional marks

[Eli Zaretskii]

> From: Itai Berli <itai.berli@gmail.com>
> Date: Sat, 1 Jul 2017 12:58:28 +0300
> 
> Emacs supports 12 Unicode bidirectional marks (ALM, RLM, LRM, LRE,
> RLE, LRO, RLO, PDF, FSI, LRI, RLI, and PDI), each of which displays as
> a very thin space. This raises two problems.
> 
> 1. On the one hand, the fact that these inherently invisible
> marks manifest, by default, as thin spaces undermines attempts at
> precise alignment and positioning. Moreover, in the case of LRM, RLM
> and ALM, this behavior contradicts explicit directions given in the
> Unicode
> Bidirectional Algorithm 8.0.0 specifications (section 2.6 Implicit
> Directional Marks):
> > they do not appear in the display
> (To my understanding, this is meant to apply to all bidi marks, even
> if only stated explicitly for LRM, RLM and ALM.)
> 
> 2. On the other hand, the fact that these spaces are so thin as to be
> barely noticeable, and the fact that
> they are indistinguishable from one another makes it difficult to debug
> and resolve strange and/or erroneous behavior that can happen in a
> bidi document, an example of which is given below.

The above is the default way these control characters are displayed.
This default was chosen so as to, on the one hand avoid making them
entirely invisible, as doing that was deemed un-Emacsy, and OTOH make
them barely visible, so that they won't disrupt the legibility of the
displayed text.

However, Emacs being Emacs, this is just the default, and it can be
changed.  The visual appearance of these (and other similar)
characters can be customized via the variable
'glyphless-char-display-control', which is described in the Emacs
manual, and in more detail in the ELisp manual.

> The solution to both problems is to make the bidi marks visible in
> `whitespace` mode only, and to give them glyphs that are (a) easy to
> notice, (b) distinguishable from other whitespace visualization glyphs, (c)
> distinct from one another.

You can do that using 'glyphless-char-display-control'.  If that is
somehow not enough, you could also define a display-table entry for
these characters, specifically for whitespace-mode.  Patches to that
effect are welcome (I think this should be a user option, if we want
such a feature).

> If we were able to visualize the whitespace, we would have realized from
> the beginning that the sequence of characters in this paragraph was, from
> left to right:
> 
> RTL-RTL-RLO-RLO-H-e-l-l-o-PDO-,-PDO-SPACE-w-o-r-l-d-!
> 
> Thus, our first three actions removed the first three characters, leaving us
> with:
> 
> RLO-H-e-l-l-o-PDO-,-PDO-SPACE-w-o-r-l-d-!
> 
> We now realize that even the final, correct form, is in fact littered
> with bidi errors and potential landmines!

Overriding the bidi attributes with the likes of RLO can indeed lead
to confusing display.  Emacs has functions that Lisp applications can
use to discover these confusing situations, where the application
would like to warn users.  See the description of
bidi-find-overridden-directionality in the ELisp manual.

bug#27544: 25.1; Visualization of Unicode bidirectional marks

[Itai Berli]

A default should be something that aligns with the expectations of a
casual user; you cannot expect a casual user to start writing elisp
code or tinkering with `glyphless-char-display-control`.

And how can we know what a casual user would expect?

1. We can follow established standards, particularly if we claim to
conform to them, as Emacs does to the Unicode standard. The Unicode
bidi algorithm 8.0.0 specifications state the following about LRM, RLM
and ALM (section 2.6 Implicit Directional Marks):
> they do not display or have any other semantic effect
and again, in the same section:
>  they do not appear in the display

2. We can see how other text editors behave. The following is a sample
of text editors that are bidi-aware, and none of them displays
explicit bidi marks (they only support RLM and LRM), not even as a
very thing space: Gmail, Google Docs, Libre Writer, MS-Word, Pages,
and TextEdit. By the way, the Unicode bidi algorithm was written by
three editors, two of whom work for Google, and the other one - for
Microsoft, so seeing how Google docs and MS-Word implement
bidirectionally is a good way of seeing how the standard should be
implemented.

3. We can employ common sense. Either the Emacs user will notice the
spaces introduced by the bidi marks, or he won't.

If he does, then it will be visually annoying for him, and will thwart
attempts to realign and position the text precisely.

If he doesn't notice the spaces, then he won't understand, for
instance, why he needs to press the left arrow four times to position
the cursor past the first visible character of a paragraph that begins
with three bidi marks.

In both cases, he won't be able to see the difference between the
various marks, which is a cause for confusion, frustration, and
errors.

4. We can see how Emacs implements similar constructs. The bidi marks
are control sequences that can be inserted into the editor using `C-x
8 RET <Unicode hex number>`. Let's see how other control sequences are
handled by this mechanism. If you try to insert, say, the characters
Null character (U+0000), Bell character (U+0007) and Escape character
(U+001B) using this method, we'll get:

ILLUSTRATION: The characters Null character (U+0000), Bell character
(U+0007) and Escape character (U+001B), in this order.
http://imgur.com/bsJoblk

We see that the output satisfies the three condition I specified in my
original post: (a) Easy to notice, (b) Distinguishable from other
whitespace visualization glyphs, (c) Distinct from one another.

However, in light of to this example I should state that, in my
opinion, unlike other control characters, Emacs should not display the
bidi marks unless in `whitespace` mode. The difference is that control
characters are rarely used by casual writers of alphabetic text,
whereas explicit bidirectional marks are quite commonly used.

bug#27544: 25.1; Visualization of Unicode bidirectional marks

[Eli Zaretskii]

> From: Itai Berli <itai.berli@gmail.com>
> Date: Sat, 1 Jul 2017 14:48:21 +0300
> 
> A default should be something that aligns with the expectations of a
> casual user; you cannot expect a casual user to start writing elisp
> code or tinkering with `glyphless-char-display-control`.
> 
> And how can we know what a casual user would expect?

Thank you for describing your expectations.  However, to change the
defaults, we'd need to hear from more than one user.  If we find that
the majority thinks the default should be changed, we will eventually
do that, as we do with any other default.

> 1. We can follow established standards, particularly if we claim to
> conform to them, as Emacs does to the Unicode standard. The Unicode
> bidi algorithm 8.0.0 specifications state the following about LRM, RLM
> and ALM (section 2.6 Implicit Directional Marks):
> > they do not display or have any other semantic effect
> and again, in the same section:
> >  they do not appear in the display

The UBA also allows to retain these characters, including display
them, see section 5.2 there.  This is how Emacs behaves, again in line
with general Emacs feature of allowing users to see what's in the
buffer.

> 4. We can see how Emacs implements similar constructs. The bidi marks
> are control sequences that can be inserted into the editor using `C-x
> 8 RET <Unicode hex number>`. Let's see how other control sequences are
> handled by this mechanism. If you try to insert, say, the characters
> Null character (U+0000), Bell character (U+0007) and Escape character
> (U+001B) using this method, we'll get:

There are control characters other than those which you tried, for
which Emacs does use the same display method as for bidi controls: try
ZWJ (U+200D) or ARABIC NUMBER SIGN (U+0600) -- all the characters of
Unicode General Category "Cf" that don't have an established graphic
image are by default displayed as thin spaces.  This is explained in
the doc string of glyphless-char-display-control.

bug#27544: 25.1; Visualization of Unicode bidirectional marks

[Itai Berli]

> This is how Emacs behaves, again in line with general Emacs feature of allowing users to see what's in the buffer.

Except the current display doesn't allow the user to see what's in the buffer.

 > The UBA also allows to retain these characters, including display
them, see section 5.2 there.

Well, section 5.2 of the UBA explains:

> retention of these formatting characters and BNs is important to users who need to display a graphic representation of hidden characters, and who thus need to obtain their visual positions for display.

Emacs users who need to display a graphic representation of the bidi
marks are ill-served by the current implementation. And so are those
who don't need to display a graphic representation of them. The
current implementation is the worse of both worlds.

bug#27544: 25.1; Visualization of Unicode bidirectional marks

[Eli Zaretskii]

> From: Itai Berli <itai.berli@gmail.com>
> Date: Sat, 1 Jul 2017 15:36:39 +0300
> 
> > This is how Emacs behaves, again in line with general Emacs feature of allowing users to see what's in the buffer.
> 
> Except the current display doesn't allow the user to see what's in the buffer.

I think it does: it shows the bidi controls instead of completely
hiding them.

>  > The UBA also allows to retain these characters, including display
> them, see section 5.2 there.
> 
> Well, section 5.2 of the UBA explains:
> 
> > retention of these formatting characters and BNs is important to users who need to display a graphic representation of hidden characters, and who thus need to obtain their visual positions for display.
> 
> Emacs users who need to display a graphic representation of the bidi
> marks are ill-served by the current implementation. And so are those
> who don't need to display a graphic representation of them. The
> current implementation is the worse of both worlds.

Well, I obviously disagree.

In any case, Emacs provides customizations to change the display of
these controls to your liking, and modes which have such needs can do
that in the buffers under the mode.  the Emacs implementation allows
all that.  If we hear enough report from people who would like the
defaults to change, we could consider that in the future.

bug#27544: 25.1; Visualization of Unicode bidirectional marks

[Lars Ingebrigtsen]

Itai Berli <itai.berli@gmail.com> writes:

> Emacs supports 12 Unicode bidirectional marks (ALM, RLM, LRM, LRE,
> RLE, LRO, RLO, PDF, FSI, LRI, RLI, and PDI), each of which displays as
> a very thin space. This raises two problems.

Perhaps there should be a minor mode to display these characters more
clearly, because they can be pretty confusing to deal with.

What would be the best way to implement such a mode?  Change
`glyphless-char-display' buffer-locally?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#27544: 25.1; Visualization of Unicode bidirectional marks

[Eli Zaretskii]

On December 1, 2021 6:55:22 AM GMT+02:00, Lars Ingebrigtsen <larsi@gnus.org> wrote:
> Itai Berli <itai.berli@gmail.com> writes:
> 
> > Emacs supports 12 Unicode bidirectional marks (ALM, RLM, LRM, LRE,
> > RLE, LRO, RLO, PDF, FSI, LRI, RLI, and PDI), each of which displays as
> > a very thin space. This raises two problems.
> 
> Perhaps there should be a minor mode to display these characters more
> clearly, because they can be pretty confusing to deal with.
> 
> What would be the best way to implement such a mode?  Change
> `glyphless-char-display' buffer-locally?

Yes, I think so.

bug#27544: [External] : bug#27544: 25.1; Visualization of Unicode bidirectional marks

[Drew Adams]

> > > Emacs supports 12 Unicode bidirectional marks (ALM, RLM, LRM, LRE,
> > > RLE, LRO, RLO, PDF, FSI, LRI, RLI, and PDI), each of which displays as
> > > a very thin space. This raises two problems.
> >
> > Perhaps there should be a minor mode to display these characters more
> > clearly, because they can be pretty confusing to deal with.
> >
> > What would be the best way to implement such a mode?  Change
> > `glyphless-char-display' buffer-locally?
> 
> Yes, I think so.

(Not following this bug thread.  Ignore if not helpful.)

Would something like what's in `modeline-char.el' help?

There's a minor mode that shows, in the mode line, the
char  after point, together with its hex Unicode code
point.  E.g., for char `e' it shows e=000065.

* If you click `mouse-1' on the lighter then full
  information about the char is shown in `*Help*',
  including the font and faces used for the char.

* If you click `mouse-2' on the lighter then the
  char is copied to the secondary selection.

* If you click `mouse-3' on the lighter then the
  char is shown enlarged in a tooltip.

https://www.emacswiki.org/emacs/download/modeline-char.el

For this bug, maybe some indication like `FSI' would
help - instead of or in addition to the code point.

bug#27544: 25.1; Visualization of Unicode bidirectional marks

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> What would be the best way to implement such a mode?  Change
>> `glyphless-char-display' buffer-locally?
>
> Yes, I think so.

OK; I've now added this under the name `glyphless-display-mode', and it
seems to work fine in my test case (which is TUTORIAL.he).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#27544: 25.1; Visualization of Unicode bidirectional marks

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Itai Berli <itai.berli@gmail.com>,  27544@debbugs.gnu.org
> Date: Wed, 01 Dec 2021 16:53:41 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> What would be the best way to implement such a mode?  Change
> >> `glyphless-char-display' buffer-locally?
> >
> > Yes, I think so.
> 
> OK; I've now added this under the name `glyphless-display-mode', and it
> seems to work fine in my test case (which is TUTORIAL.he).

Thanks.  But wouldn't glyphless-hex-display-mode be a better name?
glyphless-display-mode doesn't really say what it does to glyphless
characters.

bug#27544: 25.1; Visualization of Unicode bidirectional marks

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Thanks.  But wouldn't glyphless-hex-display-mode be a better name?
> glyphless-display-mode doesn't really say what it does to glyphless
> characters.

It doesn't hexify the glyphs, it uses `acronym'.  But I thought we might
allow that to be customised if somebody asks for it, which is why it's
not called `glyphless-acronym-display-mode'.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#27544: 25.1; Visualization of Unicode bidirectional marks

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: itai.berli@gmail.com,  27544@debbugs.gnu.org
> Date: Wed, 01 Dec 2021 18:36:56 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Thanks.  But wouldn't glyphless-hex-display-mode be a better name?
> > glyphless-display-mode doesn't really say what it does to glyphless
> > characters.
> 
> It doesn't hexify the glyphs, it uses `acronym'.  But I thought we might
> allow that to be customised if somebody asks for it, which is why it's
> not called `glyphless-acronym-display-mode'.

But glyphless-display-mode has no mnemonic value.  We already display
glyphless characters, so this name doesn't help at all.

Does anyone have suggestions for a better name?

bug#27544: 25.1; Visualization of Unicode bidirectional marks

[Juri Linkov]

>> > Thanks.  But wouldn't glyphless-hex-display-mode be a better name?
>> > glyphless-display-mode doesn't really say what it does to glyphless
>> > characters.
>>
>> It doesn't hexify the glyphs, it uses `acronym'.  But I thought we might
>> allow that to be customised if somebody asks for it, which is why it's
>> not called `glyphless-acronym-display-mode'.
>
> But glyphless-display-mode has no mnemonic value.  We already display
> glyphless characters, so this name doesn't help at all.
>
> Does anyone have suggestions for a better name?

"glyphless-display-mode" looks fine - I read it as to mean
a mode that defines how to display glyphless chars.
It could also provide a defcustom with a choice from
glyphless-char-display-method.

Where I see the problem is that the whole new file was created in
lisp/textmodes/glyphless-mode.el for just glyphless-display-mode,
instead of adding it to lisp/international/characters.el
where this feature is defined with all its functions
glyphless-char-display-control, update-glyphless-char-display,
and more.

bug#27544: 25.1; Visualization of Unicode bidirectional marks

[Filipp Gunbin]

On 02/12/2021 11:04 +0200, Juri Linkov wrote:

>>> > Thanks.  But wouldn't glyphless-hex-display-mode be a better name?
>>> > glyphless-display-mode doesn't really say what it does to glyphless
>>> > characters.
>>>
>>> It doesn't hexify the glyphs, it uses `acronym'.  But I thought we might
>>> allow that to be customised if somebody asks for it, which is why it's
>>> not called `glyphless-acronym-display-mode'.
>>
>> But glyphless-display-mode has no mnemonic value.  We already display
>> glyphless characters, so this name doesn't help at all.
>>
>> Does anyone have suggestions for a better name?
>
> "glyphless-display-mode" looks fine - I read it as to mean
> a mode that defines how to display glyphless chars.
> It could also provide a defcustom with a choice from
> glyphless-char-display-method.

Maybe glyphless-char-display-mode?  I'm sure that was considered, and
dropped, but glyphless-display-mode is just too broad...  Is it a mode
with no glyphs displayed?