New bidi problems with org-mode (Arabic, Hebrew,..)

New bidi problems with org-mode (Arabic, Hebrew,..)

[Fr Ml]

[-- Attachment #1: Type: text/html, Size: 2507 bytes --]

[-- Attachment #2: bidi_org_problem2.png --]
[-- Type: image/png, Size: 48361 bytes --]

[-- Attachment #3: bidi_org_problem1.png --]
[-- Type: image/png, Size: 59462 bytes --]

Re: New bidi problems with org-mode (Arabic, Hebrew,..)

[Samim Pezeshki]

[-- Attachment #1: Type: text/plain, Size: 2055 bytes --]

Hi,

I have no problem. I could not reproduce the issue. I tried Persian
language only.
Can the problem be of your theme? Have you tried other themes?
Please also try reproducing this with emacs -q (load Emacs with no
configuration).
Does this happen when the line exceeds a certain number of characters?
Does this happen on all org heading levels?

Regards,
Samim

Excerpts from Fr Ml's message of November 3, 2020 8:01 pm:
> https://emacs.stackexchange.com/questions/61494/
> new-bidi-problems-with-org-mode-arabic-hebrew
>
> Hallo,
>
> Some time ago (maybe 3 months) I've noticed some problems with the bidi support
> (right-to-left languages) in org-mode.
>
> One problem I can't reproduce: The characters have then a strange form but If I
> change the text scale (larger or smaller) it becomes normal.
>
> The second problem I can reproduce, you can see it in the two images:
>
> In this image: I have the option `bidi-paragraph-direction: nil`, to get the
> right direction for  Hebrew and Arabic. As you see the headline color isn't
> consistently brown. The English text don't have this problem.
>
> [![The color of the headline changes, gets black. The direction is right to
> left][1]][1]
>
> In this image without the option: Everything is ok:
> [![Here it's ok, the direction is left to right][2]][2]
>
> In both examples my init file is empty. And I've tried many fonts. My org
> version is 9.3 and emacs version is 27.1.
>
> Actually, the first problem is bigger for me but I can't reproduce it; I think
> that both problems appeared a the same time. So maybe solving the second will
> solve the first one too(?)
>
> EDIT: I can reproduce the problem with the colors also on a Windows machine
> with latest version: emacs 27.1 and the included org 9.3. With the older
> version: emacs 26.3 and the included org-version 9.1.7 I don't have the problem
> - at least I didn't see it.
>
>   [1]: https://i.stack.imgur.com/NVUMY.png
>   [2]: https://i.stack.imgur.com/emHDx.png
>
>
>
>
>

[-- Attachment #2: Type: application/pgp-signature, Size: 488 bytes --]

Re: New bidi problems with org-mode (Arabic, Hebrew,..)

[Samim Pezeshki]

[-- Attachment #1: Type: text/plain, Size: 3351 bytes --]

The attached screenshot is how I see the file you sent on my screen
using emacs -q
I see no problem in it. Please correct me if I am wrong.

Emacs 27.1
Archlinux

Regards,
Samim

Excerpts from Fr Ml's message of November 7, 2020 10:03 pm:
> Hi Samim,
>
> thanks for your answer.
>
> I tested it again with the -q option: Still have the problem (of course if I
> have in the first line: -*- bidi-paragraph-direction: nil; truncate-lines: nil;
> -*-)
>
> - It happens only at line breaks. I think always if a word is divided. But also
> in some other cases (the new line doesn't start with space). I don't think it
> depends of the length: If I enlarge the text size it changes.
>
> - Yes, it happens on all levels. And it happens also bei the underscore '_'
> (part3).
>
> As I noticed in the additional edit: It happens also on windows.
>
> Regards,
> Frank
>
> P.S. I've attached the org file too with some pictures of it.
>
>
> On 11/6/20 6:22 PM, Samim Pezeshki wrote:
>
>     Hi,
>
>     I have no problem. I could not reproduce the issue. I tried Persian
>     language only.
>     Can the problem be of your theme? Have you tried other themes?
>     Please also try reproducing this with emacs -q (load Emacs with no
>     configuration).
>     Does this happen when the line exceeds a certain number of characters?
>     Does this happen on all org heading levels?
>
>     Regards,
>     Samim
>
>     Excerpts from Fr Ml's message of November 3, 2020 8:01 pm:
>
>         https://emacs.stackexchange.com/questions/61494/
>         new-bidi-problems-with-org-mode-arabic-hebrew
>
>         Hallo,
>
>         Some time ago (maybe 3 months) I've noticed some problems with the bidi support
>         (right-to-left languages) in org-mode.
>
>         One problem I can't reproduce: The characters have then a strange form but If I
>         change the text scale (larger or smaller) it becomes normal.
>
>         The second problem I can reproduce, you can see it in the two images:
>
>         In this image: I have the option `bidi-paragraph-direction: nil`, to get the
>         right direction for  Hebrew and Arabic. As you see the headline color isn't
>         consistently brown. The English text don't have this problem.
>
>         [![The color of the headline changes, gets black. The direction is right to
>         left][1]][1]
>
>         In this image without the option: Everything is ok:
>         [![Here it's ok, the direction is left to right][2]][2]
>
>         In both examples my init file is empty. And I've tried many fonts. My org
>         version is 9.3 and emacs version is 27.1.
>
>         Actually, the first problem is bigger for me but I can't reproduce it; I think
>         that both problems appeared a the same time. So maybe solving the second will
>         solve the first one too(?)
>
>         EDIT: I can reproduce the problem with the colors also on a Windows machine
>         with latest version: emacs 27.1 and the included org 9.3. With the older
>         version: emacs 26.3 and the included org-version 9.1.7 I don't have the problem
>         - at least I didn't see it.
>
>           [1]: https://i.stack.imgur.com/NVUMY.png
>           [2]: https://i.stack.imgur.com/emHDx.png
>
>
>
>
>
>
>

[-- Attachment #2: bla.org --]
[-- Type: text/plain, Size: 1542 bytes --]

-*- bidi-paragraph-direction: nil; truncate-lines: nil; -*-

* سلام سلام سلام سلام سلام سلام سلام سلام سلام سلام سلام سلام سلام سلام سلام سلام سلام سلام سلام سلام 
** שלום שלום שלום שלום שלום שלום שלום שלום שלום שלום שלום שלום שלום שלום שלום שלום שלום שלום שלום שלום 
*** سلام سلام سلام سلام سلام سلام سلام سلام سلام سلام سلام سلام سلام سلام سلام سلام سلام سلام سلام سلام 
**** שלום שלום שלום שלום שלום שלום שלום שלום שלום שלום שלום שלום שלום שלום שלום שלום שלום שלום שלום שלום 
***** سلام سلام سلام سلام سلام سلام سلام سلام سلام سلام سلام سلام سلام سلام سلام سلام سلام سلام سلام سلام 
****** שלום שלום שלום שלום שלום שלום שלום שלום שלום שלום שלום שלום שלום שלום שלום שלום שלום שלום שלום שלום 


_سلام سلام سلام سلام سلام سلام سلام سلام سلام سلام سلام سلام سلام سلام سلام سلام سلام سلام سلام سلام_
_שלום שלום שלום שלום שלום שלום שלום שלום שלום שלום שלום שלום שלום שלום שלום שלום שלום שלום שלום שלום_ 


[-- Attachment #3: blah.png --]
[-- Type: image/png, Size: 42919 bytes --]

[-- Attachment #4: Type: application/pgp-signature, Size: 488 bytes --]

bug#44528: New bidi problems with outline-mode (Arabic, Hebrew,..)

[Fr Ml]

[-- Attachment #1: Type: text/html, Size: 2943 bytes --]

[-- Attachment #2: bidi_org_problem2.png --]
[-- Type: image/png, Size: 48361 bytes --]

[-- Attachment #3: bidi_org_problem1.png --]
[-- Type: image/png, Size: 59462 bytes --]

bug#44528: New bidi problems with outline-mode (Arabic, Hebrew, ..)

[Eli Zaretskii]

> From: Fr Ml <fr_ml@t-online.de>
> Date: Mon, 9 Nov 2020 07:10:38 +0100
> 
> Some time ago (maybe 3 months) I've noticed some problems with the bidi support (right-to-left languages)
> in outline-mode and org-mode.
> 
> One problem I can't reproduce: The characters have then a strange form but If I change the text scale
> (larger or smaller) it becomes normal.
> 
> The second problem I can reproduce, you can see it in the two images:
> 
> In this image: I have the option `bidi-paragraph-direction: nil` (the same if I put in the first line: `-*- mode:
> outline -*-`), to get the right direction for  Hebrew and Arabic. As you see the headline color isn't consistently
> brown. The English text don't have this problem.
> 
> [![The color of the headline changes, gets black. The direction is right to left][1]][1]

This is a known bug#43363, which was already fixed for the upcoming
Emacs 27.2.

> Actually, the first problem is bigger for me but I can't reproduce it; I think that both problems appeared a the
> same time. So maybe solving the second will solve the first one too(?)

Unlikely.  But you didn't say almost anything about this problem:
which characters take a "strange form", what is that strange form,
etc.  It's possible that an obscure bug related to Arabic shaping,
which was fixed just today (see bug#44521), is what you see.  (Do you
use Ivy or some similar add-on?)

bug#44528: New bidi problems with outline-mode (Arabic, Hebrew,..)

[Eli Zaretskii]

[-- Attachment #1: Type: text/plain, Size: 1465 bytes --]

[Please use Reply All to CC the bug tracker as well.]

> From: Fr Ml <fr_ml@t-online.de>
> Date: Mon, 9 Nov 2020 18:34:07 +0100
> Thanks for the answer. Good news  that one problem is solved.
> 
> I didn't mention details because I assumed that's the same problem and it wasn't my main question there. I
> chose the reproducable proble. 
> 
> But now some details:
> - I don't use ivy or ido
> - The problem happens after "some time" of working with Emacs, usually not directly after starting Emacs.   
>   Example : the 3 attached pictures are of the same text file (not org or outline), but with different text sizes. I
> have just the Arabic word العربية in it. And I'm using a monospaced Font: Dejavu Sans Mono.
>   In 'text8correct.png' everything is fine, "text size": +8 word , if I decrease to +7, the shape is not correct the
> glyphs are too dense. It remains incorrect until the size +5. At +4 it becomes correct again.
>   But if I open the same file with a new Emacs process I don't have any problem!

I can only hope this is due to the same problem as was fixed in
bug#44521.  If not, let's hope someone will come up with a
reproducible recipe in the future.

> Also what happens:
> - Sometimes the letters don't have the correct form: a middle form for an ending letter 
> - Sometimes the letters are disconnected as in bug#44521.

I attach below the images you sent, so that they are stored with the
discussion:


[-- Attachment #2: text4correct.png --]
[-- Type: image/png, Size: 18587 bytes --]

[-- Attachment #3: text8correct.png --]
[-- Type: image/png, Size: 27129 bytes --]

[-- Attachment #4: text7incorrect.png --]
[-- Type: image/png, Size: 22796 bytes --]

bug#44528: New bidi problems with outline-mode (Arabic, Hebrew,..)

[Fr Ml]

[-- Attachment #1: Type: text/html, Size: 3243 bytes --]

bug#44528: New bidi problems with outline-mode (Arabic, Hebrew,..)

[Eli Zaretskii]

> Cc: 44528@debbugs.gnu.org
> From: Fr Ml <fr_ml@t-online.de>
> Date: Thu, 12 Nov 2020 08:14:30 +0100
> 
> in the INSTALL instruction of Emacs we read:
> (On GNU and Unix systems, Emacs needs the optional libraries "m17n-db",
> "libm17n-flt", "libotf" to correctly display such complex scripts as
> Indic and Khmer, and also for scripts that require Arabic shaping
> support (Arabic and Farsi).)
> 
> But I noticed that libm17n-flt isn't installed and it's not a package in the repository of Archlinux . I didn't need
> to install it before.
> 
> Can it be the problem? Or is it included in the other packages (m17n-db, libotf)?

This text is outdated: we nowadays prefer HarfBuzz, if available, as
our shaping engine.  Is your build using HarfBuzz?  You can tell by
looking at the value of the variable 'system-configuration-features'.

If your build doesn't use HarfBuzz, I suggest to install one that
does.

bug#44528: New bidi problems with outline-mode (Arabic, Hebrew,..)

[Fr Ml]

[-- Attachment #1: Type: text/html, Size: 1757 bytes --]

bug#44528: New bidi problems with outline-mode (Arabic, Hebrew,..)

[Fr Ml]

[-- Attachment #1: Type: text/html, Size: 2016 bytes --]

bug#44528: New bidi problems with outline-mode (Arabic, Hebrew,..)

[Eli Zaretskii]

> Cc: 44528@debbugs.gnu.org
> From: Fr Ml <fr_ml@t-online.de>
> Date: Tue, 17 Nov 2020 11:04:05 +0100
> 
> I've made a new installation of Emacs (from the repository with pacman) and I've noticed that m17n-db and
> libotf are installed implicitly (because of dependency) with Emacs.
> Could this cause a problem with Harfbuzz (which is also installed implicitly with Emacs)?

I don't think so.

Start Emacs as "emacs -Q", then type "C-u C-x =" with point on some
character in the *scratch* buffer, and look at the line which says

              display: by this font (glyph code)

What does the following line say?

bug#44528: New bidi problems with outline-mode (Arabic, Hebrew,..)

[Fr Ml]

[-- Attachment #1: Type: text/html, Size: 1710 bytes --]

bug#44528: New bidi problems with outline-mode (Arabic, Hebrew,..)

[Eli Zaretskii]

> Cc: 44528@debbugs.gnu.org
> From: Fr Ml <fr_ml@t-online.de>
> Date: Wed, 18 Nov 2020 11:58:56 +0100
> 
> For 'l' (latin) and 'ع' (arabic):
> 
> l:     ftcrhb:-ADBO-Source Code Pro-normal-normal-normal-*-13-*-*-*-m-0-iso10646-1 (#x27)
> ع:     ftcrhb:-1ASC-Droid Serif-normal-normal-normal-*-13-*-*-*-*-0-iso10646-1 (#x1A)

The "hb" part of "ftcrhb" means that you are using HarfBuzz as the
shaping engine, so m17n libraries are only there as fallbacks (which
will probably never used).

IOW, your configuration is as good as it gets.

bug#44528: New bidi problems with outline-mode (Arabic, Hebrew, ..)

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> I can only hope this is due to the same problem as was fixed in
> bug#44521.  If not, let's hope someone will come up with a
> reproducible recipe in the future.

(I'm going through old bug reports that unfortunately weren't resolved
at the time.)

Fr Ml, are you still seeing these problems in recent Emacs versions?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#44528: New bidi problems with outline-mode (Arabic, Hebrew, ..)

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Fr Ml, are you still seeing these problems in recent Emacs versions?

More information was requested, but no response was given within a
month, so I'm closing this bug report.  If the problem still exists,
please respond to this email and we'll reopen the bug report.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no