bug#68446: 29.1.90; Bidi right-to-left paragraphs missing text in Org mode

bug#68446: 29.1.90; Bidi right-to-left paragraphs missing text in Org mode

[Thamer Mahmoud]

Hi,

1. In emacs -Q, create an empty buffer with Org mode active and type:

a [[link]]

2. Evaluate: (setq bidi-paragraph-direction 'right-to-left)
3. Note the "a" and link are no longer visible.

In GNU Emacs 29.1.90 (build 1, x86_64-pc-linux-gnu, GTK+ Version
  3.24.39, cairo version 1.18.0)
Repository branch: emacs-29
Windowing system distributor 'The X.Org Foundation', version 11.0.12101009
System Description: Debian GNU/Linux trixie/sid

Configured features:
CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG
LIBSELINUX LIBXML2 MODULES NATIVE_COMP NOTIFY INOTIFY PDUMPER PNG RSVG
SECCOMP SOUND SQLITE3 THREADS TIFF TOOLKIT_SCROLL_BARS TREE_SITTER WEBP
X11 XDBE XIM XINPUT2 XPM GTK3 ZLIB

-- 
Regards,
Thamer

bug#68446: 29.1.90; Bidi right-to-left paragraphs missing text in Org mode

[Eli Zaretskii]

> From: Thamer Mahmoud <thamer.mahmoud@gmail.com>
> Date: Sun, 14 Jan 2024 13:21:57 +0300
> 
> 1. In emacs -Q, create an empty buffer with Org mode active and type:
> 
> a [[link]]
> 
> 2. Evaluate: (setq bidi-paragraph-direction 'right-to-left)
> 3. Note the "a" and link are no longer visible.

I think it's an Org bug: it should prevent bidi reordering inside the
"[[link]]" string.  For example, wrap the "[[link]]" thing in LRO..PDF
bidi controls.  Because without that, the brackets can be mirrored by
bidi reordering and the BPA algorithm, and the link is no longer in
the form that Org expects.  The result is that the entire text becomes
invisible.

A work-around is to do one of the following:

  . insert one or more L2R characters after the "[[link]]", or
  . set bidi-inhibit-bpa to a non-nil value

I don't see an Emacs bug here, surprising as it may sound.  Lisp
programs that depend on particular sequence of characters on display
should be aware that bidi reordering can affect that.

Adding Ihor to the discussion.

bug#68446: 29.1.90; Bidi right-to-left paragraphs missing text in Org mode

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> 1. In emacs -Q, create an empty buffer with Org mode active and type:
>> 
>> a [[link]]
>> 
>> 2. Evaluate: (setq bidi-paragraph-direction 'right-to-left)
>> 3. Note the "a" and link are no longer visible.
>
> I think it's an Org bug: it should prevent bidi reordering inside the
> "[[link]]" string.  For example, wrap the "[[link]]" thing in LRO..PDF
> bidi controls.  Because without that, the brackets can be mirrored by
> bidi reordering and the BPA algorithm, and the link is no longer in
> the form that Org expects.  The result is that the entire text becomes
> invisible.

May you please elaborate? I do not see anything unexpected in the text
properties that Org mode applies to the link and the letter "a".

In fact, after following the reproducer, and enabling visible-mode, M-x
describe-text-properties on "a" yields

    Text content at position 101:
    
    
    There are text properties here:
      fontified            t

Yet, "a" is rendered invisible.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#68446: 29.1.90; Bidi right-to-left paragraphs missing text in Org mode

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: Thamer Mahmoud <thamer.mahmoud@gmail.com>, 68446@debbugs.gnu.org
> Date: Sun, 14 Jan 2024 12:11:03 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I think it's an Org bug: it should prevent bidi reordering inside the
> > "[[link]]" string.  For example, wrap the "[[link]]" thing in LRO..PDF
> > bidi controls.  Because without that, the brackets can be mirrored by
> > bidi reordering and the BPA algorithm, and the link is no longer in
> > the form that Org expects.  The result is that the entire text becomes
> > invisible.
> 
> May you please elaborate? I do not see anything unexpected in the text
> properties that Org mode applies to the link and the letter "a".

(For some value of "expected".  Me, I don't understand more than half
of the properties there, wrt their expected effect on the text and its
display.  Are you surprised?)

Anyway, after performing the recipe, do this:

  . M-<
  . C-x =
  . C-f
  . C-x =
  . C-f
  . C-x =

etc.  IOW, move 1 buffer position forward at a time from BOB, and each
time show the character at that position.  Do you see something
strange?

Then do the same in a buffer under Fundamental mode with the same text
and the same value of bidi-paragraph-direction.  Can you explain what
Org does to cause the difference, which makes some characters be
"skipped" by C-f (due to point-adjustment feature), and probably also
makes all the characters invisible as result?

I can tell you what I see when stepping through the display code: it
skips all the characters as invisible.  I see that org-link is in the
buffer-invisibility-spec, but that's probably just the tip of the
iceberg, so I'm asking you below to describe what are all those faces
and properties for, and how they are related to the invisibility.

> In fact, after following the reproducer, and enabling visible-mode, M-x
> describe-text-properties on "a" yields
> 
>     Text content at position 101:
>     
>     
>     There are text properties here:
>       fontified            t
> 
> Yet, "a" is rendered invisible.

When you invoke visible-mode, the text is magically shown as expected.
Can you explain why is that?

If you invoke describe-text-properties on any character in the buffer,
there's no character with 'invisible' face, either.  And yet they are
invisible.  How come?

If you describe in enough details how the [[link]] thing is handled by
Org, maybe I could elaborate more than the general observations and
questions above.

To summarize, my point is simple: the Emacs bidirectional display is
responsible for reordering plain text with faces, and while doing so,
it more-or-less ignores invisible text.  Anything fancier: overlays,
images, text properties that affect display in some other ways,
etc. -- all these are not guaranteed to behave in reordered text as
they do in strict L2R text, and Lisp programs which create visual
effects using those fancier features should look out for problems, and
should use bidi formatting controls judiciously to prevent such
disasters.

(My guess is that the brackets in this case are reordered, and as
result cause everything to become invisible.  But that's just a guess,
and I don't expect to understand why without a lot more background and
help from Org experts.)

bug#68446: 29.1.90; Bidi right-to-left paragraphs missing text in Org mode

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> May you please elaborate? I do not see anything unexpected in the text
>> properties that Org mode applies to the link and the letter "a".
>
> ...
> If you describe in enough details how the [[link]] thing is handled by
> Org, maybe I could elaborate more than the general observations and
> questions above.

Let's simplify the reproducer, getting rid of Org mode:

(let ((str (concat "a " (propertize "[" 'invisible t) "b" (propertize "]" 'invisible t))))
  (with-temp-buffer
    (display-buffer (current-buffer))
    (insert str)
    (read-char "'a b' is visible. Press any key.")
    (setq bidi-paragraph-direction 'right-to-left)
    (read-char "'a b' is invisible. Press any key.")))

Is this the above expected?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#68446: 29.1.90; Bidi right-to-left paragraphs missing text in Org mode

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: thamer.mahmoud@gmail.com, 68446@debbugs.gnu.org
> Date: Sun, 14 Jan 2024 14:41:21 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> May you please elaborate? I do not see anything unexpected in the text
> >> properties that Org mode applies to the link and the letter "a".
> >
> > ...
> > If you describe in enough details how the [[link]] thing is handled by
> > Org, maybe I could elaborate more than the general observations and
> > questions above.

Actually, I think I see the reason, and the problem is indeed in the
display code's logic in this case.

> Let's simplify the reproducer, getting rid of Org mode:
> 
> (let ((str (concat "a " (propertize "[" 'invisible t) "b" (propertize "]" 'invisible t))))
>   (with-temp-buffer
>     (display-buffer (current-buffer))
>     (insert str)
>     (read-char "'a b' is visible. Press any key.")
>     (setq bidi-paragraph-direction 'right-to-left)
>     (read-char "'a b' is invisible. Press any key.")))
> 
> Is this the above expected?

No.

bug#68446: 29.1.90; Bidi right-to-left paragraphs missing text in Org mode

[Eli Zaretskii]

> Cc: 68446@debbugs.gnu.org, thamer.mahmoud@gmail.com
> Date: Sun, 14 Jan 2024 16:58:43 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > From: Ihor Radchenko <yantar92@posteo.net>
> > Cc: thamer.mahmoud@gmail.com, 68446@debbugs.gnu.org
> > Date: Sun, 14 Jan 2024 14:41:21 +0000
> > 
> > Eli Zaretskii <eliz@gnu.org> writes:
> > 
> > >> May you please elaborate? I do not see anything unexpected in the text
> > >> properties that Org mode applies to the link and the letter "a".
> > >
> > > ...
> > > If you describe in enough details how the [[link]] thing is handled by
> > > Org, maybe I could elaborate more than the general observations and
> > > questions above.
> 
> Actually, I think I see the reason, and the problem is indeed in the
> display code's logic in this case.

There was an algorithmic flaw in handling invisible property when the
invisible text starts inside embedding level above the base paragraph
level: the code failed to account for the "backward" iteration which
is part of the bidi reordering, and instead always looked forward for
the first visible position.

> > Let's simplify the reproducer, getting rid of Org mode:
> > 
> > (let ((str (concat "a " (propertize "[" 'invisible t) "b" (propertize "]" 'invisible t))))
> >   (with-temp-buffer
> >     (display-buffer (current-buffer))
> >     (insert str)
> >     (read-char "'a b' is visible. Press any key.")
> >     (setq bidi-paragraph-direction 'right-to-left)
> >     (read-char "'a b' is invisible. Press any key.")))
> > 
> > Is this the above expected?
> 
> No.

Should be fixed now on the master branch.

bug#68446: 29.1.90; Bidi right-to-left paragraphs missing text in Org mode

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> Should be fixed now on the master branch.

I confirm that I can no longer reproduce the problem following the
original steps and my reproducer.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#68446: 29.1.90; Bidi right-to-left paragraphs missing text in Org mode

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: thamer.mahmoud@gmail.com, 68446@debbugs.gnu.org
> Date: Sun, 04 Feb 2024 13:10:14 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Should be fixed now on the master branch.
> 
> I confirm that I can no longer reproduce the problem following the
> original steps and my reproducer.

Thanks, I'm therefore closing this bug.  We can reopen if someone
reports some issues or left-overs.