Adding Tai Tham Script to GNU/Linux Distribution's Version of Emacs

Adding Tai Tham Script to GNU/Linux Distribution's Version of Emacs

[Richard Wordingham]

I am currently using a distribution's latest version of Emacs.
The distribution is Ubuntu 12.04, precise pangolin, and the version of
Emacs it currently delivers is 23.3.1.  Is it 'simple' to add complex
text layout (CTL) for the Tai Tham script on non-Windows systems?

I hope that all I have to do is to:

1) Modify fontset.el to add the script.
2) Replace fontset.elc
in /usr/share/emacs/23.3/lisp/international/fontset.elc by the modified
version of fontset.el from the source description.
3) Augment the m17n library to use a HarfBuzz-compatible Tai Tham
font.  I have such a font, and it works fine with Emacs when CTL is not
needed.

One of my worries is that I may have to update characters.el and
recompile Emacs.

I already have a working Emacs input method for the Tai Tham script.

Richard.

Re: Adding Tai Tham Script to GNU/Linux Distribution's Version of Emacs

[Eli Zaretskii]

> Date: Sun, 15 Mar 2015 12:32:20 +0000
> From: Richard Wordingham <richard.wordingham@ntlworld.com>
> 
> I am currently using a distribution's latest version of Emacs.
> The distribution is Ubuntu 12.04, precise pangolin, and the version of
> Emacs it currently delivers is 23.3.1.  Is it 'simple' to add complex
> text layout (CTL) for the Tai Tham script on non-Windows systems?

I recommend to get Emacs 24, as its support of Unicode is much better.

> I hope that all I have to do is to:
> 
> 1) Modify fontset.el to add the script.
> 2) Replace fontset.elc
> in /usr/share/emacs/23.3/lisp/international/fontset.elc by the modified
> version of fontset.el from the source description.
> 3) Augment the m17n library to use a HarfBuzz-compatible Tai Tham
> font.  I have such a font, and it works fine with Emacs when CTL is not
> needed.
> 
> One of my worries is that I may have to update characters.el and
> recompile Emacs.
> 
> I already have a working Emacs input method for the Tai Tham script.

Only this should be necessary:

  1) Modify fontset.el to add the script.

There are examples in the Emacs manual's node "Modifying Fontsets", go
by them.  AFAIU, either the 1st or the last one should fix the issue
for you, assuming you have a suitable font.

Caveat: I only tried these examples with Emacs 24, not with 23.3.

Re: Adding Tai Tham Script to GNU/Linux Distribution's Version of Emacs

[Eli Zaretskii]

> Date: Sun, 15 Mar 2015 18:56:31 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> 
> Only this should be necessary:
> 
>   1) Modify fontset.el to add the script.
> 
> There are examples in the Emacs manual's node "Modifying Fontsets", go
> by them.  AFAIU, either the 1st or the last one should fix the issue
> for you, assuming you have a suitable font.
> 
> Caveat: I only tried these examples with Emacs 24, not with 23.3.

Sorry, I see now that Emacs 23.3 doesn't know about this script.  So
the first example in the manual will probably not work, but maybe the
last will.

And, as I wrote previously, you are advised to install Emacs 24.
Version 23 is really far behind on these issues.

Re: Adding Tai Tham Script to GNU/Linux Distribution's Version of Emacs

[Richard Wordingham]

Eli Zaretskii wrote:

> Only this should be necessary:
>
>  1) Modify fontset.el to add the script.

Is this meant to be relevant to the use of Emacs 24?  In Emacs 24, Tai
Tham is already listed in fontset.el.

The examples in 'modifying fontsets' seem to be irrelevant, and
certainly didn't help.  I may be missing something subtle, but I was
already picking up the right font, and adding it to the default fontset
does not seem to make any difference, in either Emacs 23.3.1 or Emacs
24.

The rendering is different - in Emacs 24, Tai Tham non-spacing marks are
rendered as spacing marks.

Might the character categories be relevant?  They don't seem to be set
for Tai Tham characters, though they are set for Tai Viet characters,
which are slightly younger.  I also notice that a sequence of Thai
consonants does not prompy any m17n action, but that m17n action occurs
as soon as a combining character (vowel, tone mark etc.) is entered.

Richard.

Re: Adding Tai Tham Script to GNU/Linux Distribution's Version of Emacs

[Eli Zaretskii]

> Date: Sun, 15 Mar 2015 23:15:56 +0000
> From: Richard Wordingham <richard.wordingham@ntlworld.com>
> 
> Eli Zaretskii wrote:
> 
> > Only this should be necessary:
> >
> >  1) Modify fontset.el to add the script.
> 
> Is this meant to be relevant to the use of Emacs 24?  In Emacs 24, Tai
> Tham is already listed in fontset.el.

Sorry, you are right.  I've misread that line to mean you want to
change the setup of the default fontset, not that you want to add the
script (which is already recognized by default).

> The examples in 'modifying fontsets' seem to be irrelevant, and
> certainly didn't help.  I may be missing something subtle, but I was
> already picking up the right font, and adding it to the default fontset
> does not seem to make any difference, in either Emacs 23.3.1 or Emacs
> 24.

Then what exactly is/was your problem with that script?  Your question
said:

> Is it 'simple' to add complex text layout (CTL) for the Tai Tham
> script on non-Windows systems?

Emacs does CTL on Unix by using 2 m17l libraries.  Here's a citation
from INSTALL:

  * Complex Text Layout support libraries

  On GNU and Unix systems, Emacs needs the optional libraries "m17n-db",
  "libm17n-flt", "libotf" to correctly display such complex scripts as
  Indic and Khmer, and also for scripts that require Arabic shaping
  support (Arabic and Farsi).  On some systems, particularly GNU/Linux,
  these libraries may be already present or available as additional
  packages.  Note that if there is a separate `dev' or `devel' package,
  for use at compilation time rather than run time, you will need that
  as well as the corresponding run time package; typically the dev
  package will contain header files and a library archive.  Otherwise,
  you can download the libraries from <http://www.nongnu.org/m17n/>.

  Note that Emacs cannot support complex scripts on a TTY, unless the
  terminal includes such a support.

But you seem to have these libraries already, so I'm unsure what
exactly is the problem you are having.

> Might the character categories be relevant?  They don't seem to be set
> for Tai Tham characters, though they are set for Tai Viet characters,
> which are slightly younger.

Which categories did you have in mind?  "Category" is too general a
term here.

> I also notice that a sequence of Thai consonants does not prompy any
> m17n action, but that m17n action occurs as soon as a combining
> character (vowel, tone mark etc.) is entered.

If the problem is with character compositions (is it?), then perhaps
the corresponding rules are not defined yet, and should be added.  Or
maybe the versions of those libraries you have don't yet have support
for Tai Tham.

Bottom line, I think I'm confused about the problems you are trying to
solve, so please elaborate on them.

Adding Tai Tham Script to GNU/Linux Distribution's Version of Emacs

[Richard Wordingham]

> Date: Mon, 16 Mar 2015 05:43:07 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> To: help-gnu-emacs@gnu.org
> Subject: Re: Adding Tai Tham Script to GNU/Linux Distribution's
> 	Version of	Emacs
> Message-ID: <83d2491ub8.fsf@gnu.org>

> Then what exactly is/was your problem with that script?  Your question
> said:
> 
> > Is it 'simple' to add complex text layout (CTL) for the Tai Tham
> > script on non-Windows systems?
> 
> Emacs does CTL on Unix by using 2 m17l libraries.  Here's a citation
> from INSTALL:
> 
>   * Complex Text Layout support libraries
> 
>   On GNU and Unix systems, Emacs needs the optional libraries
> "m17n-db", "libm17n-flt", "libotf" ...

> But you seem to have these libraries already, so I'm unsure what
> exactly is the problem you are having.

The primary problem is that m17n currently does not have support for the
Tai Tham script, not even at Version 1.7.0.  I therefore need to add
this to my copy of the m17n database, and, if I do an acceptable job,
offer this support for inclusion in the public m17n database.  I've
added a first attempt to my copy, but I see no evidence that Emacs is
attempting to use it. (I've enabled full m17n debugging by setting
environment variable MDEBUG_ALL, and get lots of output to the terminal
I launch Emacs from, including a record of my Tai Tham shaping rules
being read in.) Unfortunately, writing a test bed for m17n does not seem
simple - I had hoped to use Emacs as the test bed, as the rest of the system is
often the quickest test bed to build. (It's for Emacs that I want m17n
support - HarfBuzz provides support for browsers and word processing.)
I've asked for advice on purely m17n matters on the m17n help list.

> > Might the character categories be relevant?  They don't seem to be
> > set for Tai Tham characters, though they are set for Tai Viet
> > characters, which are slightly younger in Unicode.

> Which categories did you have in mind?  "Category" is too general a
> term here.

I meant category as being set by calls of modify-category-entry in
characters.el.  (The lisp function is defined in file category.c.)  The
clue is that somehow Emacs knows that text shaping is not needed for a
sequence of Thai consonants, and it is also true that it is not needed
for a sequence of Tai Tham consonants.  However, I don't know where the
triggering logic is.  It *could* be something like 'invoke shaping if
there is a combining character or Indian character present'.  I'm not
even sure what an 'Indian character' (category codes 'i' and 'I') is.
It might merely be a non-ASCII character supported by ISCII.  Another
clue is that shaping seems to be invoked for lone Tibetan consonants. 

Richard.

Re: Adding Tai Tham Script to GNU/Linux Distribution's Version of Emacs

[Eli Zaretskii]

> Date: Mon, 16 Mar 2015 07:47:34 +0000
> From: Richard Wordingham <richard.wordingham@ntlworld.com>
> 
> The primary problem is that m17n currently does not have support for the
> Tai Tham script, not even at Version 1.7.0.  I therefore need to add
> this to my copy of the m17n database, and, if I do an acceptable job,
> offer this support for inclusion in the public m17n database.  I've
> added a first attempt to my copy, but I see no evidence that Emacs is
> attempting to use it. (I've enabled full m17n debugging by setting
> environment variable MDEBUG_ALL, and get lots of output to the terminal
> I launch Emacs from, including a record of my Tai Tham shaping rules
> being read in.) Unfortunately, writing a test bed for m17n does not seem
> simple - I had hoped to use Emacs as the test bed, as the rest of the system is
> often the quickest test bed to build. (It's for Emacs that I want m17n
> support - HarfBuzz provides support for browsers and word processing.)
> I've asked for advice on purely m17n matters on the m17n help list.

Yes, questions about m17n libraries are best asked there.

> > > Might the character categories be relevant?  They don't seem to be
> > > set for Tai Tham characters, though they are set for Tai Viet
> > > characters, which are slightly younger in Unicode.
> 
> > Which categories did you have in mind?  "Category" is too general a
> > term here.
> 
> I meant category as being set by calls of modify-category-entry in
> characters.el.  (The lisp function is defined in file category.c.)  The
> clue is that somehow Emacs knows that text shaping is not needed for a
> sequence of Thai consonants, and it is also true that it is not needed
> for a sequence of Tai Tham consonants.  However, I don't know where the
> triggering logic is.  It *could* be something like 'invoke shaping if
> there is a combining character or Indian character present'.  I'm not
> even sure what an 'Indian character' (category codes 'i' and 'I') is.
> It might merely be a non-ASCII character supported by ISCII.  Another
> clue is that shaping seems to be invoked for lone Tibetan consonants.

OK, now I understand the issue well enough to actually try talking
intelligently about it ;-)  Thanks for taking time to explain it, and
sorry I didn't catch that earlier.

First, a caveat: I'm not enough of an expert on these matters in
Emacs, so please take what's below with a grain of salt, and expect to
have to experiment to at least some extent.  Also, please excuse if I
describe below features you already know about.

That said, I don't think character categories are the root cause of
your problem, at least not directly.  Instead, please take a look at
the language-specific files in lisp/language/, e.g. tibetan.el,
thai.el, and thai-util.el: this is where Emacs defines data and code
required for CTL of these languages.

In Emacs parlance, CTL is called "character composition".  The rules
for character composition are stored in composition-function-table,
whose doc string describes its contents.  Using these rules, Emacs
calls the shaping engine with more than one character when they need
to be shaped as a single entity (a.k.a. "grapheme cluster").  The
shaping engine then returns one or more glyphs for Emacs to display.

The default composition rules are defined in lisp/composite.el
(towards the second half of the file), and files in lisp/language/ add
rules for specific languages.  (Those rules sometimes use character
categories, and sometimes even invent new categories; see thai-util.el
as an example.)  I believe you will see there the data that allows
Emacs to perform CTL for Thai and Tibetan.

My guess is that you will have to write a tai-tham.el file with
similar data for Tai Tham, and then load it into Emacs.  All the rest
should "just work".

Now, if all of the above doesn't help, then I suggest to write to
Kenichi Handa, who wrote most of the related code in Emacs, and who I
believe is also reading the m17n lists (he is one of the developers of
those libraries).

HTH

Re: Adding Tai Tham Script to GNU/Linux Distribution's Version of Emacs

[Richard Wordingham]

On Mon, 16 Mar 2015 18:34:03 +0200
Eli Zaretskii <eliz@gnu.org> wrote:

> The default composition rules are defined in lisp/composite.el
> (towards the second half of the file), and files in lisp/language/ add
> rules for specific languages.  (Those rules sometimes use character
> categories, and sometimes even invent new categories; see thai-util.el
> as an example.)  I believe you will see there the data that allows
> Emacs to perform CTL for Thai and Tibetan.
> 
> My guess is that you will have to write a tai-tham.el file with
> similar data for Tai Tham, and then load it into Emacs.  All the rest
> should "just work".

I've done as you suggested and am getting what seem to be evidence of
communication, Emacs to m17n, for Tai Tham text.  Everything was broken
at the first attempt - infinite loop in Emacs, and I think m17n is
giving up on my instructions.  I'll just have to approach the full
design bit by bit. I aped burmese.el for tai-tham.el.

Having been forced to recompile emacs for myself, it will be less
awkward to put monitoring in.  I nearly missed the need to obtain the
m17n and libotf developer packages - it was only the display of
configuration options selected by configure that alerted me.

Richard.

Re: Adding Tai Tham Script to GNU/Linux Distribution's Version of Emacs

[Eli Zaretskii]

> Date: Tue, 17 Mar 2015 00:23:03 +0000
> From: Richard Wordingham <richard.wordingham@ntlworld.com>
> 
> I've done as you suggested and am getting what seem to be evidence of
> communication, Emacs to m17n, for Tai Tham text.  Everything was broken
> at the first attempt - infinite loop in Emacs, and I think m17n is
> giving up on my instructions.  I'll just have to approach the full
> design bit by bit. I aped burmese.el for tai-tham.el.
> 
> Having been forced to recompile emacs for myself, it will be less
> awkward to put monitoring in.  I nearly missed the need to obtain the
> m17n and libotf developer packages - it was only the display of
> configuration options selected by configure that alerted me.

Please don't hesitate to submit bug reports for whatever problems you
find (infloops etc.), using "M-x report-emacs-bug".  Also, when
tai-tham.el is ready, please consider donating it to Emacs.

TIA

Re: Adding Tai Tham Script to GNU/Linux Distribution's Version of Emacs

[Richard Wordingham]

On Mon, 16 Mar 2015 18:34:03 +0200
Eli Zaretskii <eliz@gnu.org> wrote:

> My guess is that you will have to write a tai-tham.el file with
> similar data for Tai Tham, and then load it into Emacs.  All the rest
> should "just work".

It's now mostly working, but with some strange anomalies.  For
example, where a syllable is to render as spacing glyph, non-spacing
glyph, spacing glyph, m17n delivers the correct glyphs, in the right
order, to emacs, but then emacs chooses not to display the derived
glyphs!  Highly analogous sequences where the third glyph is
non-spacing (just by having a different final consonant) render
correctly.  I'm actively investigating the anomalies.

Re: Adding Tai Tham Script to GNU/Linux Distribution's Version of Emacs

[Richard Wordingham]

On Wed, 18 Mar 2015 08:27:37 +0000
Richard Wordingham <richard.wordingham@ntlworld.com> wrote:

> On Mon, 16 Mar 2015 18:34:03 +0200
> Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > My guess is that you will have to write a tai-tham.el file with
> > similar data for Tai Tham, and then load it into Emacs.  All the
> > rest should "just work".

> It's now mostly working, but with some strange anomalies.

The problem is that function font-shape-gstring, defined in file
font.c, which *appears* to be the correct composition function, does not
grok the m17n FLT module, or at least not at Version 1.6.3.  A work
around appears to be declare all the strings passed to m17n by Emacs as
grapheme clusters, which can be done by the special symbols < > in the
FLT generator commands.  I'm preparing a bug report.  Without the < >
symbols, the 'composition' of swapping consonant and vowel round to
convert from logical and phonetic order to visual order is declared to
be a shaper error!

I detest grapheme clusters.  I refuse to believe that in a Tai
language, the users consider every combination of consonant, vowel and
tone as a single letter.  Indeed, there were howls of outrage when the
practical effects of Unicode declaring a Thai preposed vowel plus
consonant to be a grapheme cluster struck, and the standard was
rapidly reversed.

Richard.

Re: Adding Tai Tham Script to GNU/Linux Distribution's Version of Emacs

[Eli Zaretskii]

> Date: Wed, 18 Mar 2015 21:33:22 +0000
> From: Richard Wordingham <richard.wordingham@ntlworld.com>
> 
> The problem is that function font-shape-gstring, defined in file
> font.c, which *appears* to be the correct composition function, does not
> grok the m17n FLT module, or at least not at Version 1.6.3.

That sounds pretty strange.  Have you tried to discuss this with m17n
FLT developers?  Emacs is supposed to fit m17n very well, since the
same developers contributed to both projects.

Re: Adding Tai Tham Script to GNU/Linux Distribution's Version of Emacs

[Richard Wordingham]

On Thu, 19 Mar 2015 05:41:16 +0200
Eli Zaretskii <eliz@gnu.org> wrote:

> > Date: Wed, 18 Mar 2015 21:33:22 +0000
> > From: Richard Wordingham <richard.wordingham@ntlworld.com>
> > 
> > The problem is that function font-shape-gstring, defined in file
> > font.c, which *appears* to be the correct composition function,
> > does not grok the m17n FLT module, or at least not at Version 1.6.3.
> 
> That sounds pretty strange.  Have you tried to discuss this with m17n
> FLT developers?  Emacs is supposed to fit m17n very well, since the
> same developers contributed to both projects.

If one follows the Microsoft example of making an Indic orthographic
syllable a grapheme cluster, the problem largely (entirely?) goes away.
Moreover, apart from the swapping round of consonant and vowel or
dependent RA glyphs, the problem examples (bug 20140) all include U+1A60
TAI THAM SIGN SAKOT, which was quietly given a non-zero canonical
combining class despite the explicit recommendation in the encoding
proposal that it be given a canonical combining class of 0.  Its low,
positive canonical combining class (9 for virama) causes no end of minor
problems.

The main issue with m17n itself is that the from and to fields don't
handle discontiguous substrings.  Discontiguous substrings, typically
but not necessarily associated with canonical equivalence, are
generally not handled well.

There might be some more unusual (from, to) sequences once I get the
<NA, AA> ligature working.  It's not forming under m17n for some reason,
but Emacs is innocent.

Richard.

Re: Adding Tai Tham Script to GNU/Linux Distribution's Version of Emacs

[Richard Wordingham]

On Thu, 19 Mar 2015 07:37:16 +0000
Richard Wordingham <richard.wordingham@ntlworld.com> wrote:

> There might be some more unusual (from, to) sequences once I get the
> <NA, AA> ligature working.  It's not forming under m17n for some
> reason, but Emacs is innocent.

Libotf stands accused - https://savannah.nongnu.org/bugs/?44584.  The
quick fix for me was to modify my font.

Richard.