Message headers in right-to-left mode in Emacs notmuch-show-mode

Message headers in right-to-left mode in Emacs notmuch-show-mode

[Teemu Likonen]

[-- Attachment #1.1.1: Type: text/plain, Size: 4109 bytes --]

Emacs uses setting bidi-paragraph-direction=nil by default which means
that paragraph's text direction can change automatically, for example:
right-to-left. A quote from the variable's description:

    If this is nil (the default), the direction of each paragraph is
    determined by the first strong directional character of its text.

So, in notmuch-show-mode the message header paragraph is switched to
right-to-left mode if headers contain a "strong directional"
right-to-left character.

The result can be seen in the attached jpg image which displays part of
the message <id:20200614013343.GB1808@freeshell.de>
<https://lists.debian.org/debian-devel/2020/06/msg00099.html>.

I think message headers should always be in left-to-right mode but
message contents should follow the Emacs default (choose automatically
from the content). I believe the header part can be switched to
left-to-right mode by using function bidi-string-mark-left-to-right.
Below is a relevant quote from Emacs Lisp reference manual.

Unfortunately I don't know Notmuch Emacs code (at least not yet). So
this message is mainly for documenting the current behavior.


Emacs Lisp reference manual, section "Bidirectional Display".

https://www.gnu.org/software/emacs/manual/html_node/elisp/Bidirectional-Display.html#Bidirectional-Display


    Bidirectional reordering can have surprising and unpleasant effects
    when two strings with bidirectional content are juxtaposed in a
    buffer, or otherwise programmatically concatenated into a string of
    text. A typical problematic case is when a buffer consists of
    sequences of text fields separated by whitespace or punctuation
    characters, like Buffer Menu mode or Rmail Summary Mode. Because the
    punctuation characters used as separators have “weak
    directionality”, they take on the directionality of surrounding
    text. As result, a numeric field that follows a field with
    bidirectional content can be displayed _to the left_ of the
    preceding field, messing up the expected layout. There are several
    ways to avoid this problem:

       − Append the special character U+200E LEFT-TO-RIGHT MARK, or LRM,
         to the end of each field that may have bidirectional content,
         or prepend it to the beginning of the following field. The
         function ‘bidi-string-mark-left-to-right’, described below,
         comes in handy for this purpose. (In a right-to-left paragraph,
         use U+200F RIGHT-TO-LEFT MARK, or RLM, instead.) This is one of
         the solutions recommended by the UBA.

       − Include the tab character in the field separator. The tab
         character plays the role of “segment separator” in
         bidirectional reordering, causing the text on either side to be
         reordered separately.

       − Separate fields with a ‘display’ property or overlay with a
         property value of the form ‘(space . PROPS)’ (*note Specified
         Space::). Emacs treats this display specification as a
         “paragraph separator”, and reorders the text on either side
         separately.

     -- Function: bidi-string-mark-left-to-right string

         This function returns its argument STRING, possibly modified,
         such that the result can be safely concatenated with another
         string, or juxtaposed with another string in a buffer, without
         disrupting the relative layout of this string and the next one
         on display. If the string returned by this function is
         displayed as part of a left-to-right paragraph, it will always
         appear on display to the left of the text that follows it. The
         function works by examining the characters of its argument, and
         if any of those characters could cause reordering on display,
         the function appends the LRM character to the string. The
         appended LRM character is made invisible by giving it an
         ‘invisible’ text property of ‘t’ (*note Invisible Text::).



[-- Attachment #1.1.2: notmuch-show-right-to-left.jpg --]
[-- Type: image/jpeg, Size: 79264 bytes --]

[-- Attachment #1.1.3: Type: text/plain, Size: 110 bytes --]


--
/// Teemu Likonen - .-.. http://www.iki.fi/tlikonen/
// OpenPGP: 4E1055DC84E9DFF613D78557719D69D324539450

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 251 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Message headers in right-to-left mode in Emacs notmuch-show-mode

[Teemu Likonen]

[-- Attachment #1.1.1: Type: text/plain, Size: 370 bytes --]

* 2020-08-03 09:13:35+03, Teemu Likonen wrote:

> I believe the header part can be switched to left-to-right mode by
> using function bidi-string-mark-left-to-right.

It seems that adding just a single U+200E LEFT-TO-RIGHT MARK character
(possibly with invisible text property) in the beginning of message
header paragraph will do. The attached (inline) patch does it.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.1.2: 0001-Emacs-Force-left-to-right-display-for-message-header.patch --]
[-- Type: text/x-diff, Size: 1303 bytes --]

From 4264a1b3561c132343fe6a74e8cf2548e5127c3e Mon Sep 17 00:00:00 2001
From: Teemu Likonen <tlikonen@iki.fi>
Date: Mon, 3 Aug 2020 10:25:27 +0300
Subject: [PATCH] Emacs: Force left-to-right display for message headers

Insert U+200E LEFT-TO-RIGHT MARK at the beginning of message headers.
It forces message headers to display as left-to-right text even if
there are strong directional characters in header's values.

See Emacs Lisp reference manual section "(elisp) Bidirectional
Display" for more info.
---
 emacs/notmuch-show.el | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index c9170466..87f68bc8 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -467,7 +467,8 @@ unchanged ADDRESS if parsing fails."
   "Insert a notmuch style headerline based on HEADERS for a
 message at DEPTH in the current thread."
   (let ((start (point)))
-    (insert (notmuch-show-spaces-n (* notmuch-show-indent-messages-width depth))
+    (insert (propertize (string #x200e) 'invisible t) ; U+200E LEFT-TO-RIGHT MARK
+	    (notmuch-show-spaces-n (* notmuch-show-indent-messages-width depth))
 	    (notmuch-sanitize
 	     (notmuch-show-clean-address (plist-get headers :From)))
 	    " ("
-- 
2.20.1


[-- Attachment #1.1.3: Type: text/plain, Size: 115 bytes --]


-- 
/// Teemu Likonen - .-.. http://www.iki.fi/tlikonen/
// OpenPGP: 4E1055DC84E9DFF613D78557719D69D324539450

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 251 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

[PATCH v2] Emacs: Force left-to-right display for message headers

[Teemu Likonen]

[-- Attachment #1.1: Type: text/plain, Size: 1894 bytes --]

* 2020-08-03 10:37:11+03, Teemu Likonen wrote:

> From 4264a1b3561c132343fe6a74e8cf2548e5127c3e Mon Sep 17 00:00:00 2001
> From: Teemu Likonen <tlikonen@iki.fi>
> Date: Mon, 3 Aug 2020 10:25:27 +0300
> Subject: [PATCH] Emacs: Force left-to-right display for message headers

Here's slightly better patch with (string ?\x200e) instead of (string
#x200e) because STRING function wants characters as argument.
Conceptually:

  ?\x200e  = character
  #x200e   = integer

In the current Emacs they are both integers (8206).

This message can be applied with "git am --scissors".

--- 8< ----
Insert invisible U+200E LEFT-TO-RIGHT MARK at the beginning of message
headers. It forces message headers to display as left-to-right text even
if there are strong directional characters in header's values.

See Emacs Lisp reference manual section "(elisp) Bidirectional
Display" for more info.
---
 emacs/notmuch-show.el | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index c9170466..c45db57d 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -466,9 +466,10 @@ unchanged ADDRESS if parsing fails."
 (defun notmuch-show-insert-headerline (headers date tags depth)
   "Insert a notmuch style headerline based on HEADERS for a
 message at DEPTH in the current thread."
   (let ((start (point)))
-    (insert (notmuch-show-spaces-n (* notmuch-show-indent-messages-width depth))
+    (insert (propertize (string ?\x200e) 'invisible t) ; U+200E LEFT-TO-RIGHT MARK
+	    (notmuch-show-spaces-n (* notmuch-show-indent-messages-width depth))
 	    (notmuch-sanitize
 	     (notmuch-show-clean-address (plist-get headers :From)))
 	    " ("
 	    date
-- 
2.20.1




-- 
/// Teemu Likonen - .-.. http://www.iki.fi/tlikonen/
// OpenPGP: 4E1055DC84E9DFF613D78557719D69D324539450

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 251 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

[PATCH v3] Emacs: Force left-to-right display for message headers

[Teemu Likonen]

Insert invisible U+200E LEFT-TO-RIGHT MARK at the beginning of message
headers. It forces message headers to display as left-to-right text
even if there are strong directional characters in header's values.

See Emacs Lisp reference manual section "(elisp) Bidirectional
Display" for more info.
---

> This message can be applied with "git am --scissors".

Sorry, it doesn't apply with "git am". Maybe PGP/MIME or
quoted-printable encoding messed it up. This time I try with "git
send-email".


 emacs/notmuch-show.el | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index c9170466..c45db57d 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -466,9 +466,10 @@ unchanged ADDRESS if parsing fails."
 (defun notmuch-show-insert-headerline (headers date tags depth)
   "Insert a notmuch style headerline based on HEADERS for a
 message at DEPTH in the current thread."
   (let ((start (point)))
-    (insert (notmuch-show-spaces-n (* notmuch-show-indent-messages-width depth))
+    (insert (propertize (string ?\x200e) 'invisible t) ; U+200E LEFT-TO-RIGHT MARK
+	    (notmuch-show-spaces-n (* notmuch-show-indent-messages-width depth))
 	    (notmuch-sanitize
 	     (notmuch-show-clean-address (plist-get headers :From)))
 	    " ("
 	    date
-- 
2.20.1

Re: [PATCH v3] Emacs: Force left-to-right display for message headers

[David Edmondson]

On Tuesday, 2020-08-04 at 21:02:25 +03, Teemu Likonen wrote:

> Insert invisible U+200E LEFT-TO-RIGHT MARK at the beginning of message
> headers. It forces message headers to display as left-to-right text
> even if there are strong directional characters in header's values.
>
> See Emacs Lisp reference manual section "(elisp) Bidirectional
> Display" for more info.
> ---
>
>> This message can be applied with "git am --scissors".
>
> Sorry, it doesn't apply with "git am". Maybe PGP/MIME or
> quoted-printable encoding messed it up. This time I try with "git
> send-email".

I've no idea if this is the appropriate approach to addressing this, but
the resulting behaviour is obviously an improvement over what happens
now.

It would make sense to add some commentary to the code as well as the
commit message explaining the reason for inserting the seemingly
arbitrary character.

>  emacs/notmuch-show.el | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
> index c9170466..c45db57d 100644
> --- a/emacs/notmuch-show.el
> +++ b/emacs/notmuch-show.el
> @@ -466,9 +466,10 @@ unchanged ADDRESS if parsing fails."
>  (defun notmuch-show-insert-headerline (headers date tags depth)
>    "Insert a notmuch style headerline based on HEADERS for a
>  message at DEPTH in the current thread."
>    (let ((start (point)))
> -    (insert (notmuch-show-spaces-n (* notmuch-show-indent-messages-width depth))
> +    (insert (propertize (string ?\x200e) 'invisible t) ; U+200E LEFT-TO-RIGHT MARK
> +	    (notmuch-show-spaces-n (* notmuch-show-indent-messages-width depth))
>  	    (notmuch-sanitize
>  	     (notmuch-show-clean-address (plist-get headers :From)))
>  	    " ("
>  	    date
> -- 
> 2.20.1
> _______________________________________________
> notmuch mailing list -- notmuch@notmuchmail.org
> To unsubscribe send an email to notmuch-leave@notmuchmail.org

dme.
-- 
Too much information, running through my brain.

Re: [PATCH v3] Emacs: Force left-to-right display for message headers

[Teemu Likonen]

[-- Attachment #1.1: Type: text/plain, Size: 1283 bytes --]

* 2020-08-05 09:45:23+01, David Edmondson wrote:

> I've no idea if this is the appropriate approach to addressing this, but
> the resulting behaviour is obviously an improvement over what happens
> now.

I think there are two options:

 1. Add a string like "From: " in the beginning of notmuch-show header
    paragraph so that the paragraph always starts with left-to-right
    characters (those latin letters "From").

 2. Add invisible characters that force left-to-right text for the
    paragraph. Character U+200E LEFT-TO-RIGHT MARK is meant for
    controlling exactly that.

My patch implements the option 2 and...

> It would make sense to add some commentary to the code as well as the
> commit message explaining the reason for inserting the seemingly
> arbitrary character.

...it has at least comment

    ; U+200E LEFT-TO-RIGHT MARK

in the code. I think that explains the purpose quite well. More verbose
explanation could be something like this:

    ;; Add invisible U+200E LEFT-TO-RIGHT MARK character to force the
    ;; header paragraph as left-to-right text even if some header's
    ;; content is right-to-left.

-- 
/// Teemu Likonen - .-.. http://www.iki.fi/tlikonen/
// OpenPGP: 4E1055DC84E9DFF613D78557719D69D324539450

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 251 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: [PATCH v3] Emacs: Force left-to-right display for message headers

[Teemu Likonen]

[-- Attachment #1.1: Type: text/plain, Size: 817 bytes --]

* 2020-08-05 12:40:06+03, Teemu Likonen wrote:

> I think there are two options:
>
>  1. Add a string like "From: " in the beginning of notmuch-show header
>     paragraph so that the paragraph always starts with left-to-right
>     characters (those latin letters "From").
>
>  2. Add invisible characters that force left-to-right text for the
>     paragraph. Character U+200E LEFT-TO-RIGHT MARK is meant for
>     controlling exactly that.

 3. Bad option: (setq bidi-paragraph-direction 'left-to-right) for the
    whole notmuch-show buffer. This is not good idea because also the
    message body is forced to left-to-right direction (even if it
    contained right-to-left text).

-- 
/// Teemu Likonen - .-.. http://www.iki.fi/tlikonen/
// OpenPGP: 4E1055DC84E9DFF613D78557719D69D324539450

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 251 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: [PATCH v3] Emacs: Force left-to-right display for message headers

[David Edmondson]

On Wednesday, 2020-08-05 at 12:40:06 +03, Teemu Likonen wrote:

> * 2020-08-05 09:45:23+01, David Edmondson wrote:
>
>> I've no idea if this is the appropriate approach to addressing this, but
>> the resulting behaviour is obviously an improvement over what happens
>> now.
>
> I think there are two options:
>
>  1. Add a string like "From: " in the beginning of notmuch-show header
>     paragraph so that the paragraph always starts with left-to-right
>     characters (those latin letters "From").
>
>  2. Add invisible characters that force left-to-right text for the
>     paragraph. Character U+200E LEFT-TO-RIGHT MARK is meant for
>     controlling exactly that.
>
> My patch implements the option 2 and...

It seems fine.

Reviewed-by: David Edmondson <dme@dme.org>

>> It would make sense to add some commentary to the code as well as the
>> commit message explaining the reason for inserting the seemingly
>> arbitrary character.
>
> ...it has at least comment
>
>     ; U+200E LEFT-TO-RIGHT MARK
>
> in the code. I think that explains the purpose quite well. More verbose
> explanation could be something like this:
>
>     ;; Add invisible U+200E LEFT-TO-RIGHT MARK character to force the
>     ;; header paragraph as left-to-right text even if some header's
>     ;; content is right-to-left.

This is much more useful to someone looking at the code (I know that the
explanation is in the commit, but when things get moved around a lot,
chasing back through commit messages can be time consuming and
difficult).

dme.
-- 
It's gettin', it's gettin', it's gettin' kinda hectic.

[PATCH v4] Emacs: Force left-to-right display for message headers

[Teemu Likonen]

Insert invisible U+200E LEFT-TO-RIGHT MARK character at the beginning
of message header paragraph in notmuch-show buffer. The U+200E
character forces the header paragraph as left-to-right text even if
the header content started with right-to-left characters.

See Emacs Lisp reference manual section "(elisp) Bidirectional
Display" for more info.

Reviewed-by: David Edmondson <dme@dme.org>
---
 emacs/notmuch-show.el | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index c9170466..6548891f 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -466,9 +466,14 @@ unchanged ADDRESS if parsing fails."
 (defun notmuch-show-insert-headerline (headers date tags depth)
   "Insert a notmuch style headerline based on HEADERS for a
 message at DEPTH in the current thread."
   (let ((start (point)))
-    (insert (notmuch-show-spaces-n (* notmuch-show-indent-messages-width depth))
+    (insert (propertize (string ?\x200e) 'invisible t)
+	    ;; Add invisible U+200E LEFT-TO-RIGHT MARK character (see
+	    ;; above) to force the header paragraph as left-to-right
+	    ;; text even if the header content started with
+	    ;; right-to-left characters.
+	    (notmuch-show-spaces-n (* notmuch-show-indent-messages-width depth))
 	    (notmuch-sanitize
 	     (notmuch-show-clean-address (plist-get headers :From)))
 	    " ("
 	    date
-- 
2.20.1

Re: [PATCH v4] Emacs: Force left-to-right display for message headers

[David Bremner]

Teemu Likonen <tlikonen@iki.fi> writes:

> Insert invisible U+200E LEFT-TO-RIGHT MARK character at the beginning
> of message header paragraph in notmuch-show buffer. The U+200E
> character forces the header paragraph as left-to-right text even if
> the header content started with right-to-left characters.
>
> See Emacs Lisp reference manual section "(elisp) Bidirectional
> Display" for more info.

This causes 10 tests to fail for me. At a guess, the added LEFT-TO-RIGHT MARK
should probably be stripped out in the test framework. Either that or
added to test output files. The latter sounds easy to miss when editing.

Re: [PATCH v4] Emacs: Force left-to-right display for message headers

[Teemu Likonen]

[-- Attachment #1.1: Type: text/plain, Size: 3048 bytes --]

* 2020-08-06 09:04:50-03, David Bremner wrote:

> This causes 10 tests to fail for me. At a guess, the added
> LEFT-TO-RIGHT MARK should probably be stripped out in the test
> framework. Either that or added to test output files. The latter
> sounds easy to miss when editing.

For the first time I ran the tests and got over 60 test fails. :-) I'm
probably doing something wrong and have to study the test framework
better.

I don't know which tests are related to the U+200E LEFT-TO-RIGHT MARK
patch but if test output files are representing the content of
notmuch-show-mode buffer then I think U+200E belongs in those expected
output files, even if the character is invisible.

But here is another idea for the whole thing: When displaying a message
in notmuch-show buffer check if message's From header has any
right-to-left characters and only if it does add invisible U+200E
character at the beginning, otherwise don't bother. This way those tests
probably won't be affected. What do you think?

Below is a quick try on the top of my previous (v4) patch. I'll do a
proper patch later.


diff --git i/emacs/notmuch-show.el w/emacs/notmuch-show.el
index 6548891f..6b7d70d9 100644
--- i/emacs/notmuch-show.el
+++ w/emacs/notmuch-show.el
@@ -465,22 +465,23 @@ unchanged ADDRESS if parsing fails."
 
 (defun notmuch-show-insert-headerline (headers date tags depth)
   "Insert a notmuch style headerline based on HEADERS for a
 message at DEPTH in the current thread."
-  (let ((start (point)))
-    (insert (propertize (string ?\x200e) 'invisible t)
-	    ;; Add invisible U+200E LEFT-TO-RIGHT MARK character (see
-	    ;; above) to force the header paragraph as left-to-right
-	    ;; text even if the header content started with
-	    ;; right-to-left characters.
-	    (notmuch-show-spaces-n (* notmuch-show-indent-messages-width depth))
-	    (notmuch-sanitize
-	     (notmuch-show-clean-address (plist-get headers :From)))
-	    " ("
-	    date
-	    ") ("
-	    (notmuch-tag-format-tags tags tags)
-	    ")\n")
+  (let ((start (point))
+        (from (notmuch-sanitize
+               (notmuch-show-clean-address (plist-get headers :From)))))
+    (insert (when (string-match "\\cR" from)
+              ;; If the From header has a right-to-left character add
+              ;; invisible U+200E LEFT-TO-RIGHT MARK character which
+              ;; forces the header paragraph as left-to-right text.
+              (propertize (string ?\x200e) 'invisible t))
+            (notmuch-show-spaces-n (* notmuch-show-indent-messages-width depth))
+            from
+            " ("
+            date
+            ") ("
+            (notmuch-tag-format-tags tags tags)
+            ")\n")
     (overlay-put (make-overlay start (point)) 'face 'notmuch-message-summary-face)))
 
 (defun notmuch-show-insert-header (header header-value)
   "Insert a single header."


-- 
/// Teemu Likonen - .-.. http://www.iki.fi/tlikonen/
// OpenPGP: 4E1055DC84E9DFF613D78557719D69D324539450

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 251 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: [PATCH v4] Emacs: Force left-to-right display for message headers

[Teemu Likonen]

[-- Attachment #1.1: Type: text/plain, Size: 1715 bytes --]

* 2020-08-06 17:50:36+03, Teemu Likonen wrote:

> But here is another idea for the whole thing: When displaying a message
> in notmuch-show buffer check if message's From header has any
> right-to-left characters and only if it does add invisible U+200E
> character at the beginning, otherwise don't bother. This way those tests
> probably won't be affected. What do you think?
>
> Below is a quick try on the top of my previous (v4) patch. I'll do a
> proper patch later.

Better version which is not based on any patches but the Git version:


diff --git c/emacs/notmuch-show.el w/emacs/notmuch-show.el
index c9170466..0eb27e33 100644
--- c/emacs/notmuch-show.el
+++ w/emacs/notmuch-show.el
@@ -466,10 +466,16 @@ unchanged ADDRESS if parsing fails."
 (defun notmuch-show-insert-headerline (headers date tags depth)
   "Insert a notmuch style headerline based on HEADERS for a
 message at DEPTH in the current thread."
-  (let ((start (point)))
+  (let ((start (point))
+	(from (notmuch-sanitize
+	       (notmuch-show-clean-address (plist-get headers :From)))))
+    (when (string-match "\\cR" from)
+      ;; If the From header has a right-to-left character add
+      ;; invisible U+200E LEFT-TO-RIGHT MARK character which forces
+      ;; the header paragraph as left-to-right text.
+      (insert (propertize (string ?\x200e) 'invisible t)))
     (insert (notmuch-show-spaces-n (* notmuch-show-indent-messages-width depth))
-	    (notmuch-sanitize
-	     (notmuch-show-clean-address (plist-get headers :From)))
+	    from
 	    " ("
 	    date
 	    ") ("

-- 
/// Teemu Likonen - .-.. http://www.iki.fi/tlikonen/
// OpenPGP: 4E1055DC84E9DFF613D78557719D69D324539450

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 251 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

[PATCH v5] Emacs: Ensure left-to-right display for message headers

[Teemu Likonen]

In notmuch-show buffer insert invisible U+200E LEFT-TO-RIGHT MARK
character at the beginning of message header paragraph if the From
header contains a right-to-left character. This ensures that the
header paragraph is always rendered in left-to-right mode.

See Emacs Lisp reference manual section "(elisp) Bidirectional
Display" for more info.
---
 emacs/notmuch-show.el | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)


As the commit description says this version inserts U+200E
LEFT-TO-RIGHT MARK only if the first header line (From header)
contains a right-to-left character.

This version is probably friendlier to the current test files which
don't expect to see U+200E LEFT-TO-RIGHT MARK in the output.


diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index c9170466..0eb27e33 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -466,10 +466,16 @@ unchanged ADDRESS if parsing fails."
 (defun notmuch-show-insert-headerline (headers date tags depth)
   "Insert a notmuch style headerline based on HEADERS for a
 message at DEPTH in the current thread."
-  (let ((start (point)))
+  (let ((start (point))
+	(from (notmuch-sanitize
+	       (notmuch-show-clean-address (plist-get headers :From)))))
+    (when (string-match "\\cR" from)
+      ;; If the From header has a right-to-left character add
+      ;; invisible U+200E LEFT-TO-RIGHT MARK character which forces
+      ;; the header paragraph as left-to-right text.
+      (insert (propertize (string ?\x200e) 'invisible t)))
     (insert (notmuch-show-spaces-n (* notmuch-show-indent-messages-width depth))
-	    (notmuch-sanitize
-	     (notmuch-show-clean-address (plist-get headers :From)))
+	    from
 	    " ("
 	    date
 	    ") ("
-- 
2.20.1

Re: [PATCH v5] Emacs: Ensure left-to-right display for message headers

[David Edmondson]

On Friday, 2020-08-07 at 07:46:41 +03, Teemu Likonen wrote:

> In notmuch-show buffer insert invisible U+200E LEFT-TO-RIGHT MARK
> character at the beginning of message header paragraph if the From
> header contains a right-to-left character. This ensures that the
> header paragraph is always rendered in left-to-right mode.
>
> See Emacs Lisp reference manual section "(elisp) Bidirectional
> Display" for more info.

Reviewed-by: David Edmondson <dme@dme.org>

> ---
>  emacs/notmuch-show.el | 12 +++++++++---
>  1 file changed, 9 insertions(+), 3 deletions(-)
>
>
> As the commit description says this version inserts U+200E
> LEFT-TO-RIGHT MARK only if the first header line (From header)
> contains a right-to-left character.
>
> This version is probably friendlier to the current test files which
> don't expect to see U+200E LEFT-TO-RIGHT MARK in the output.
>
>
> diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
> index c9170466..0eb27e33 100644
> --- a/emacs/notmuch-show.el
> +++ b/emacs/notmuch-show.el
> @@ -466,10 +466,16 @@ unchanged ADDRESS if parsing fails."
>  (defun notmuch-show-insert-headerline (headers date tags depth)
>    "Insert a notmuch style headerline based on HEADERS for a
>  message at DEPTH in the current thread."
> -  (let ((start (point)))
> +  (let ((start (point))
> +	(from (notmuch-sanitize
> +	       (notmuch-show-clean-address (plist-get headers :From)))))
> +    (when (string-match "\\cR" from)
> +      ;; If the From header has a right-to-left character add
> +      ;; invisible U+200E LEFT-TO-RIGHT MARK character which forces
> +      ;; the header paragraph as left-to-right text.
> +      (insert (propertize (string ?\x200e) 'invisible t)))
>      (insert (notmuch-show-spaces-n (* notmuch-show-indent-messages-width depth))
> -	    (notmuch-sanitize
> -	     (notmuch-show-clean-address (plist-get headers :From)))
> +	    from
>  	    " ("
>  	    date
>  	    ") ("
> -- 
> 2.20.1

dme.
-- 
At least they're not lonely.

Re: [PATCH v5] Emacs: Ensure left-to-right display for message headers

[David Bremner]

Teemu Likonen <tlikonen@iki.fi> writes:

> In notmuch-show buffer insert invisible U+200E LEFT-TO-RIGHT MARK
> character at the beginning of message header paragraph if the From
> header contains a right-to-left character. This ensures that the
> header paragraph is always rendered in left-to-right mode.
>
> See Emacs Lisp reference manual section "(elisp) Bidirectional
> Display" for more info.

v5 applied to master.

d

Re: [PATCH v5] Emacs: Ensure left-to-right display for message headers

[=?utf-8?Q?T=E2=80=AEomi?= Ollila]

How about this =D

(Sorry how the headers might look...)

From: contains U+202E (LEFT-TO-RIGHT OVERRIDE) (in =?utf-8?Q?T=E2=80=AEomi?=)

https://www.fileformat.info/info/unicode/char/202e/index.htm

Tomi

---

(top-posting on purpose, all rest is for reference only)

On Fri, Aug 07 2020, Teemu Likonen wrote:

> In notmuch-show buffer insert invisible U+200E LEFT-TO-RIGHT MARK
> character at the beginning of message header paragraph if the From
> header contains a right-to-left character. This ensures that the
> header paragraph is always rendered in left-to-right mode.
>
> See Emacs Lisp reference manual section "(elisp) Bidirectional
> Display" for more info.
> ---
>  emacs/notmuch-show.el | 12 +++++++++---
>  1 file changed, 9 insertions(+), 3 deletions(-)
>
>
> As the commit description says this version inserts U+200E
> LEFT-TO-RIGHT MARK only if the first header line (From header)
> contains a right-to-left character.
>
> This version is probably friendlier to the current test files which
> don't expect to see U+200E LEFT-TO-RIGHT MARK in the output.
>
>
> diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
> index c9170466..0eb27e33 100644
> --- a/emacs/notmuch-show.el
> +++ b/emacs/notmuch-show.el
> @@ -466,10 +466,16 @@ unchanged ADDRESS if parsing fails."
>  (defun notmuch-show-insert-headerline (headers date tags depth)
>    "Insert a notmuch style headerline based on HEADERS for a
>  message at DEPTH in the current thread."
> -  (let ((start (point)))
> +  (let ((start (point))
> +	(from (notmuch-sanitize
> +	       (notmuch-show-clean-address (plist-get headers :From)))))
> +    (when (string-match "\\cR" from)
> +      ;; If the From header has a right-to-left character add
> +      ;; invisible U+200E LEFT-TO-RIGHT MARK character which forces
> +      ;; the header paragraph as left-to-right text.
> +      (insert (propertize (string ?\x200e) 'invisible t)))
>      (insert (notmuch-show-spaces-n (* notmuch-show-indent-messages-width depth))
> -	    (notmuch-sanitize
> -	     (notmuch-show-clean-address (plist-get headers :From)))
> +	    from
>  	    " ("
>  	    date
>  	    ") ("
> -- 
> 2.20.1

Re: [PATCH v5] Emacs: Ensure left-to-right display for message headers

[David Edmondson]

On Sunday, 2020-08-09 at 23:12:28 +03, =?utf-8?Q?T=E2=80=AEomi?= Ollila wrote:

> How about this =D

Your point being that the code sanitising the displayed header could do
better?

> (Sorry how the headers might look...)
>
> From: contains U+202E (LEFT-TO-RIGHT OVERRIDE) (in =?utf-8?Q?T=E2=80=AEomi?=)
>
> https://www.fileformat.info/info/unicode/char/202e/index.htm
>
> Tomi
>
> ---
>
> (top-posting on purpose, all rest is for reference only)
>
> On Fri, Aug 07 2020, Teemu Likonen wrote:
>
>> In notmuch-show buffer insert invisible U+200E LEFT-TO-RIGHT MARK
>> character at the beginning of message header paragraph if the From
>> header contains a right-to-left character. This ensures that the
>> header paragraph is always rendered in left-to-right mode.
>>
>> See Emacs Lisp reference manual section "(elisp) Bidirectional
>> Display" for more info.
>> ---
>>  emacs/notmuch-show.el | 12 +++++++++---
>>  1 file changed, 9 insertions(+), 3 deletions(-)
>>
>>
>> As the commit description says this version inserts U+200E
>> LEFT-TO-RIGHT MARK only if the first header line (From header)
>> contains a right-to-left character.
>>
>> This version is probably friendlier to the current test files which
>> don't expect to see U+200E LEFT-TO-RIGHT MARK in the output.
>>
>>
>> diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
>> index c9170466..0eb27e33 100644
>> --- a/emacs/notmuch-show.el
>> +++ b/emacs/notmuch-show.el
>> @@ -466,10 +466,16 @@ unchanged ADDRESS if parsing fails."
>>  (defun notmuch-show-insert-headerline (headers date tags depth)
>>    "Insert a notmuch style headerline based on HEADERS for a
>>  message at DEPTH in the current thread."
>> -  (let ((start (point)))
>> +  (let ((start (point))
>> +	(from (notmuch-sanitize
>> +	       (notmuch-show-clean-address (plist-get headers :From)))))
>> +    (when (string-match "\\cR" from)
>> +      ;; If the From header has a right-to-left character add
>> +      ;; invisible U+200E LEFT-TO-RIGHT MARK character which forces
>> +      ;; the header paragraph as left-to-right text.
>> +      (insert (propertize (string ?\x200e) 'invisible t)))
>>      (insert (notmuch-show-spaces-n (* notmuch-show-indent-messages-width depth))
>> -	    (notmuch-sanitize
>> -	     (notmuch-show-clean-address (plist-get headers :From)))
>> +	    from
>>  	    " ("
>>  	    date
>>  	    ") ("
>> -- 
>> 2.20.1
> _______________________________________________
> notmuch mailing list -- notmuch@notmuchmail.org
> To unsubscribe send an email to notmuch-leave@notmuchmail.org

dme.
-- 
Our President's crazy, did you hear what he said?

Re: [PATCH v5] Emacs: Ensure left-to-right display for message headers

[Teemu Likonen]

[-- Attachment #1.1: Type: text/plain, Size: 1377 bytes --]

* 2020-08-09 23:12:28+03, utf wrote:

> How about this =D

> From: contains U+202E (LEFT-TO-RIGHT OVERRIDE) (in
> =?utf-8?Q?T=E2=80=AEomi?=)

Indeed message's header fields can contain such override characters. The
override mode should be terminated with U+202C POP DIRECTIONAL
FORMATTING within the same header field. That POP character pops the
override mode from the direction mode stack and returns to the previous
mode. Those characters can mess any text badly when not used in
controlled pairs of "push" and "pop".

I'll write "abc abc abc" series but the middle "abc" have RIGHT-TO-LEFT
OVERRIDE before "a" and POP DIRECTIONAL FORMATTING after "c". In Emacs
try using C-f and C-b commands to move the cursor above the text:

    abc ‮abc‬ abc

If we wanted to clean message headers from possible unpaired overrides
we should clean all these:

    U+202A LEFT-TO-RIGHT EMBEDDING (push)
    U+202B RIGHT-TO-LEFT EMBEDDING (push)
    U+202C POP DIRECTIONAL FORMATTING (pop)
    U+202D LEFT-TO-RIGHT OVERRIDE (push)
    U+202E RIGHT-TO-LEFT OVERRIDE (push)

Or we could even try to be clever and count those characters and then
insert or remove some of them so that there are as many "push"
characters as "pop" characters.

-- 
/// Teemu Likonen - .-.. http://www.iki.fi/tlikonen/
// OpenPGP: 4E1055DC84E9DFF613D78557719D69D324539450

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 251 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Sanitize bidi control chars

[Teemu Likonen]

[-- Attachment #1.1: Type: text/plain, Size: 2324 bytes --]

* 2020-08-10 19:45:11+03, Teemu Likonen wrote:

> If we wanted to clean message headers from possible unpaired overrides
> we should clean all these:
>
>     U+202A LEFT-TO-RIGHT EMBEDDING (push)
>     U+202B RIGHT-TO-LEFT EMBEDDING (push)
>     U+202C POP DIRECTIONAL FORMATTING (pop)
>     U+202D LEFT-TO-RIGHT OVERRIDE (push)
>     U+202E RIGHT-TO-LEFT OVERRIDE (push)
>
> Or we could even try to be clever and count those characters and then
> insert or remove some of them so that there are as many "push"
> characters as "pop" characters.

Below is an example Emacs Lisp function to balance those "push" and
"pop" bidi control chars. This kind of code could be used to sanitize
message headers or any arbitrary text coming from user.

I'm not even sure if such thing should be done in Emacs or in lower
level Notmuch code. Anyway, I tried to add it to notmuch-sanitize
function. Now Tomi's message didn't switch direction of other text
anymore (in notmuch-search-mode buffer).


(defun notmuch-balance-bidi-ctrl-chars (string)
  (let ((new nil)
        (stack-count 0))

    (cl-flet ((push-char-p (c)
                ;; U+202A LEFT-TO-RIGHT EMBEDDING
                ;; U+202B RIGHT-TO-LEFT EMBEDDING
                ;; U+202D LEFT-TO-RIGHT OVERRIDE
                ;; U+202E RIGHT-TO-LEFT OVERRIDE
                (cl-find c '(?\x202a ?\x202b ?\x202d ?\x202e)))
              (pop-char-p (c)
                ;; U+202C POP DIRECTIONAL FORMATTING
                (eql c ?\x202c)))

      (cl-loop
       for char across string
       do (cond ((push-char-p char)
                 (cl-incf stack-count)
                 (push char new))
                ((and (pop-char-p char)
                      (cl-plusp stack-count))
                 (cl-decf stack-count)
                 (push char new))
                ((and (pop-char-p char)
                      (not (cl-plusp stack-count)))
                 ;; The stack is empty. Ignore this pop char.
                 )
                (t (push char new)))))

    ;; Add missing pops.
    (cl-loop
     repeat stack-count
     do (push ?\x202c new))

    (seq-into (nreverse new) 'string)))



-- 
/// Teemu Likonen - .-.. http://www.iki.fi/tlikonen/
// OpenPGP: 4E1055DC84E9DFF613D78557719D69D324539450

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 251 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]