bug#20140: 24.4; M17n shaper output rejected

[Eli Zaretskii <eliz@gnu.org>]

> Date: Sat, 5 Feb 2022 22:52:51 +0000
> From: Richard Wordingham <richard.wordingham@ntlworld.com>
> Cc: Lars Ingebrigtsen <larsi@gnus.org>, 20140@debbugs.gnu.org
> 
> I'm currently using the vanilla emacs on Ubuntu Focal, which is
> described as 'GNU Emacs 26.3 (build 2, x86_64-pc-linux-gnu, GTK+
> Version 3.24.14) of 2020-03-26, modified by Debian'.  The key good news
> is that the commands forward-char-intrusive and backward-char-intrusive
> are now standard, so I can position the cursor by dead-reckoning.  You
> can reasonably mark the issue as solved.

I don't see the commands forward-char-intrusive and
backward-char-intrusive anywhere in Emacs, so I guess they are your
local changes, based on the code posted by Handa-san in this
discussion?

> > The most important change is that we now use HarfBuzz by default.
> 
> Isn't that only true for Emacs 27.1 and above?

That's true, but Emacs 26 is ancient history; Emacs 28.1 is about to
be released.  So from our perspective, HarfBuzz is the default shaping
engine, and since it's available on all the supported platforms we
care about, we are phasing out m17n-flt shapers.

> > Richard didn't contribute the Tai Tham composition rules to us
> > (AFAIR), so I cannot test what happens now in Emacs with HarfBuzz.
> > Maybe we should revisit this issue, but first I hope Richard could
> > tell whether the issue still exists, and if so, what composition rules
> > he uses or suggests to use for Tai Tham.
> 
> Sad to see that Khaled Hosny's suggestion not to use composition rules
> seems not to have been taken.

You mean, to pass all the text via HarfBuzz instead?  That makes the
Emacs redisplay painfully slow, and would require a complete redesign
of how we render text to be bearable.  So as long as such a redesign
is not available, we cannot use that advice.

> You're welcome to include my composition rules.

Thanks.

> They're complicated by the facts that the 'regular expressions' are
> not interpreted as regular expressions and they are not interpreted
> as closed under canonical equivalence.  I therefore calculate the
> regular expression.

I'm not sure I understand the issue: what you do seems to be very
similar to what we do for the Indic scripts in indian.el, so what kind
of complications are you talking about here?

Also, your rules seem to follow the description in the "Structuring
Tai Tham Unicode" document (Revision 7), a.k.a. "L2/19-365", dated Oct
2019, is that right?  Is that document the latest word on shaping Tai
Tham, or are there any additional sources?

> There are some deficiencies; I've a feeling there may be a problem with
> adding ZWNJ and CGJ as marks; ZWJ should also be added for
> completeness.

These are barely mentioned in the L2/19-365 document, and not
mentioned at all in the Tai Tham section of the Unicode Standard.
Does it mean they are not very important in contemporary Tai Tham
texts?

> I need ZWNJ to write 4-column ᨴᩣᩴᨶ᩠ᩅ‌ᩣ᩠ᨿ as opposed to
> 3-column ᨴᩣᩴᨶ᩠ᩅᩣ᩠ᨿ, and even with my font, HarfBuzz will need CGJ for
> the suppression of jack-booted dotted circles. Additionally, for
> didactic text, what can I do for U+25CC for explicit display of marks
> and their equivalents on a dotted circle, and for that matter, for
> display on NBSP?

At least for the dotted circle case, Emacs has a general composition
rule; see compose-gstring-for-dotted-circle and the corresponding rule
in composite.c.  So I'm not sure we need anything specific to Tai Tham
there.

Can you recommend good fonts for Tai Tham?  Are they free fonts?

Thanks.