bug#41005: problem with rendering Persian text in Emacs 27

[Pip Cet <pipcet@gmail.com>]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Pip Cet <pipcet@gmail.com>
>> Cc: hossein valizadeh <valizadeh.ho@gmail.com>,  41005@debbugs.gnu.org,
>>   nicholasdrozd@gmail.com
>> Date: Fri, 05 Jun 2020 13:05:43 +0000
>> 
>> I think the attached patch is a fairly minimal fix; it's against master,
>> applies to emacs-27 but I haven't tested it there.
>
> Thanks, it LGTM.
>
> I think we should put this on emacs-27, because this is a regression
> caused by Emacs 27's support for HarfBuzz as the default shaping
> engine.  The other shapers didn't want us to provide the direction,
> they determined it internally.  We added the DIRECTION argument as
> part of integrating HarfBuzz.

Okay, will do that unless someone objects.

> We could do better than your patch by actually computing the resolved
> bidi level there, which would require start_display followed by
> move_it_to, in which case we probably won't need to call
> composition_reseat_it by hand at all, and could just pick up the
> result produced by move_it_to.  Or maybe we should just use
> Fvertical_motion instead (which does all that internally).  But these
> ideas are for the master branch, not for emacs-27.

That sounds good.

>> Given these two bugs, I wonder whether it wouldn't be more reasonable
>> always to let HarfBuzz guess the direction, at least for Emacs-27:
>> scripts which change direction, if they are supported by HarfBuzz, won't
>> work anyway.
>
> Please explain "scripts that change direction" and "won't work
> anyway", I don't think I understand that part.

I think your example (RLO..PDF in RTL text) is better: that won't work
anyway, right now, because if, for example, you type

<HEBREW LETTER SHIN> <RIGHT-TO-LEFT OVERRIDE> f i

and have set the char table to treat "fi" as a ligature, the result will
(at least sometimes) be an "fi" ligature, but it should look like the
word "if".

> The reason we don't let HarfBuzz guess in all cases is because the
> resolved bidi level, when we have it, is a more accurate indication of
> the required direction.

Yes, but we'll still cache the wrong direction. If we let HarfBuzz guess
in all cases, output will be consistent and usually correct

> For example, if you have RTL characters
> inside the LRO..PDF embedding, it would be wrong to let the shaper
> guess, because it could (and usually will) guess wrongly that the
> direction is R2L.  It is true that these are rare and unusual use
> cases, but they do exist, and Emacs does want to support them,
> including with scripts that must use the shaping engine.

As I described, I don't think RLO..PDF works with shaping right now,
because other code might have already cached the non-overridden glyph
string.