utf-8 thai character display issue?

utf-8 thai character display issue?

[scole_mail]

I am sometimes having trouble with displaying and entering a specific
thai character ำ "THAI CHARACTER SARA AM".

If that character follows some other thai characters or I try to type it
in directly following some other thai character it won't display
properly in the buffer.  It looks like there is no character there.

I know actually it is in the buffer because I can get the properties of
it even though it is not being displayed.  Here are 2 example words that
have issues on my system

กำลัง    (ing)
น้ำ      (water)

The 2nd word is always not displayed correctly. The first word is
sometimes displayed correctly depending on what other thai text files I
may have visited early.  I have tried using and switching various thai
fonts and sizes, but it seems like once the issue appears changing or
resizing the fonts never helps.

My emacs version is
GNU Emacs 26.3 (build 1, x86_64--netbsd, GTK+ Version 3.24.14) of
2020-05-31
I also use this version of emacs built on macos and it doesn't have the
problem.

Here is the char info for it when it didn't display:

C-x C-u = gives this:
             position: 285 of 365 (78%), restriction: <36-366>, column: 10
            character: ำ (displayed as ำ) (codepoint 3635, #o7063, #xe33)
              charset: unicode (Unicode (ISO10646))
code point in charset: 0x0E33
               script: thai
               syntax: w 	which means: word
             category: .:Base, 1:base vowel, L:Left-to-right (strong), t:Thai
             to input: type "e" with thai-kesmanee input method
          buffer code: #xE0 #xB8 #xB3
            file code: #xE0 #xB8 #xB3 (encoded by coding system utf-8-emacs)
              display: by this font (glyph code)
    xft:-PfEd-Garuda-normal-normal-normal-*-13-*-*-*-*-0-iso10646-1 (#x104)

Character code properties: customize what to show
  name: THAI CHARACTER SARA AM
  old-name: THAI VOWEL SIGN SARA AM
  general-category: Lo (Letter, Other)
  decomposition: (compat 3661 3634) (compat 'ํ' 'า')

There are text properties here:
  fontified            t

[back]

Here's for one time it was being display correctly:

             position: 50 of 83 (59%), column: 7
            character: ำ (displayed as ำ) (codepoint 3635, #o7063, #xe33)
              charset: unicode (Unicode (ISO10646))
code point in charset: 0x0E33
               script: thai
               syntax: w 	which means: word
             category: .:Base, 1:base vowel, L:Left-to-right (strong), t:Thai
             to input: type "C-x 8 RET e33" or "C-x 8 RET THAI CHARACTER SARA AM"
          buffer code: #xE0 #xB8 #xB3
            file code: #xE0 #xB8 #xB3 (encoded by coding system utf-8-unix)
              display: composed to form "ำ" (see below)

Composed using this font:
  xft:-PfEd-Garuda-normal-normal-normal-*-13-*-*-*-*-0-iso10646-1
by these glyphs:
  [1 1 3634 259 6 0 5 8 0 nil]

Character code properties: customize what to show
  name: THAI CHARACTER SARA AM
  old-name: THAI VOWEL SIGN SARA AM
  general-category: Lo (Letter, Other)
  decomposition: (compat 3661 3634) (compat 'ํ' 'า')

There are text properties here:
  field                output
  fontified            t
  front-sticky         (field inhibit-line-move-field-capture)
  inhibit-line-move-field-capture t
  rear-nonsticky       t

Thank you for any suggestions

Re: utf-8 thai character display issue?

[Eli Zaretskii]

> From: scole_mail <scole_mail@gmx.com>
> Date: Fri, 05 Jun 2020 11:05:18 -0700
> 
> I am sometimes having trouble with displaying and entering a specific
> thai character ำ "THAI CHARACTER SARA AM".
> 
> If that character follows some other thai characters or I try to type it
> in directly following some other thai character it won't display
> properly in the buffer.  It looks like there is no character there.
> 
> I know actually it is in the buffer because I can get the properties of
> it even though it is not being displayed.  Here are 2 example words that
> have issues on my system
> 
> กำลัง    (ing)
> น้ำ      (water)
> 
> The 2nd word is always not displayed correctly. The first word is
> sometimes displayed correctly depending on what other thai text files I
> may have visited early.  I have tried using and switching various thai
> fonts and sizes, but it seems like once the issue appears changing or
> resizing the fonts never helps.
> 
> My emacs version is
> GNU Emacs 26.3 (build 1, x86_64--netbsd, GTK+ Version 3.24.14) of
> 2020-05-31
> I also use this version of emacs built on macos and it doesn't have the
> problem.

This sounds like a problem with the text shaping engine your build is
using on NetBSD.  (you didn't give the details of the build, so I
cannot know which shape engine is being used, but I'm guessing its
libflt.)  Try upgrading to Emacs 27, and make sure it is built with
HarfBuzz support (the latest pretest can be downloaded from
alpha.gnu.org).

Re: utf-8 thai character display issue?

[scole_mail]

Eli Zaretskii <eliz@gnu.org> writes:
>
> This sounds like a problem with the text shaping engine your build is
> using on NetBSD.  (you didn't give the details of the build, so I
> cannot know which shape engine is being used, but I'm guessing its
> libflt.)  Try upgrading to Emacs 27, and make sure it is built with
> HarfBuzz support (the latest pretest can be downloaded from
> alpha.gnu.org).

Just for posterity, I was using the pkgsrc/editors/emacs26 from
http://pkgsrc.org/, which I think is actually emacs26.3.  As best as I
could tell, it was not using harfbuzz?  Here were some of its
dependencies:
 libotf>=0.9.13nb2:../../graphics/libotf
 m17n-lib>=1.8.0nb1:../../devel/m17n-lib
 libXft>=2.3.2nb2:../../x11/libXft

Luckily, there already existed a package in
pkgsrc/editors/emacs-snapshot that is emacs28.0.50.  It had almost the
same dependencies plus
 harfbuzz>=2.6.4nb2
and a few others.  It seems to be working now for what I need.

Thank you for the information.

Re: utf-8 thai character display issue?

[Eli Zaretskii]

> From: scole_mail <scole_mail@gmx.com>
> Cc: Eli Zaretskii <eliz@gnu.org>
> Date: Sat, 06 Jun 2020 09:24:09 -0700
> 
> > This sounds like a problem with the text shaping engine your build is
> > using on NetBSD.  (you didn't give the details of the build, so I
> > cannot know which shape engine is being used, but I'm guessing its
> > libflt.)  Try upgrading to Emacs 27, and make sure it is built with
> > HarfBuzz support (the latest pretest can be downloaded from
> > alpha.gnu.org).
> 
> Just for posterity, I was using the pkgsrc/editors/emacs26 from
> http://pkgsrc.org/, which I think is actually emacs26.3.  As best as I
> could tell, it was not using harfbuzz?

No, HarfBuzz was not supported before Emacs 27.

> Luckily, there already existed a package in
> pkgsrc/editors/emacs-snapshot that is emacs28.0.50.  It had almost the
> same dependencies plus
>  harfbuzz>=2.6.4nb2
> and a few others.  It seems to be working now for what I need.

Thanks, so I think my guess was correct, and the problem was with the
shaping engine.