bug#17739: 24.3; Dotted circle does not combine with Thai vowel markers

bug#17739: 24.3; Dotted circle does not combine with Thai vowel markers

[Brady Garvin]

1. Start Emacs:
    $ emacs -Q
2. Type "◌ึ" (U+25CC followed by U+E36)

Expected behavior: the vowel marker appears over the dotted circle, per
the usual convention for presenting Thai vowels without preceding
consonants.

Actual behavior: the characters do not combine; a dotted circle is shown
followed by the vowel over empty space.

See also: http://en.wikipedia.org/wiki/Dotted_circle

Emacs-specific on my system.

Apologies if this is a duplicate; I could not find a prior report.


In GNU Emacs 24.3.1 (x86_64-pc-linux-gnu, GTK+ Version 3.10.7)
 of 2014-03-07 on lamiak, modified by Debian
Windowing system distributor `The X.Org Foundation', version 11.0.11501000
System Description:	Ubuntu 14.04 LTS

Configured using:
 `configure '--build' 'x86_64-linux-gnu' '--build' 'x86_64-linux-gnu'
 '--prefix=/usr' '--sharedstatedir=/var/lib' '--libexecdir=/usr/lib'
 '--localstatedir=/var/lib' '--infodir=/usr/share/info'
 '--mandir=/usr/share/man' '--with-pop=yes'
 '--enable-locallisppath=/etc/emacs24:/etc/emacs:/usr/local/share/emacs/24.3/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/24.3/site-lisp:/usr/share/emacs/site-lisp'
 '--with-crt-dir=/usr/lib/x86_64-linux-gnu' '--with-x=yes'
 '--with-x-toolkit=gtk3' '--with-toolkit-scroll-bars'
 'build_alias=x86_64-linux-gnu' 'CFLAGS=-g -O2 -fstack-protector
 --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wall'
 'LDFLAGS=-Wl,-Bsymbolic-functions -Wl,-z,relro'
 'CPPFLAGS=-D_FORTIFY_SOURCE=2''

Important settings:
  value of $LC_TIME: en_GB.UTF-8
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix
  default enable-multibyte-characters: t

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
<help-echo> M-x r e p <tab> o <tab> r <tab> <retur
n>

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
Making completion list... [2 times]

Load-path shadows:
None found.

Features:
(shadow sort gnus-util mail-extr emacsbug message format-spec rfc822 mml
mml-sec mm-decode mm-bodies mm-encode mail-parse rfc2231 mailabbrev
gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums mm-util
mail-prsvr mail-utils help-mode easymenu time-date tooltip ediff-hook
vc-hooks lisp-float-type mwheel x-win x-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list newcomment lisp-mode register page
menu-bar rfn-eshadow timer select scroll-bar mouse jit-lock font-lock
syntax facemenu font-core frame cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
case-table epa-hook jka-cmpr-hook help simple abbrev minibuffer loaddefs
button faces cus-face macroexp files text-properties overlay sha1 md5
base64 format env code-pages mule custom widget hashtable-print-readable
backquote make-network-process dbusbind dynamic-setting
system-font-setting font-render-setting move-toolbar gtk x-toolkit x
multi-tty emacs)

bug#17739: 24.3; Dotted circle does not combine with Thai vowel markers

[Andreas Schwab]

Brady Garvin <bgarvin@acm.org> writes:

> Actual behavior: the characters do not combine; a dotted circle is shown
> followed by the vowel over empty space.

Emacs can only combine characters if they are displayed by the same
font.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#17739: 24.3; Dotted circle does not combine with Thai vowel markers

[Brady Garvin]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1167 bytes --]

Thanks.  However, I see the same problem even putting both characters
in the same font.  With S-mouse-1 "Change Buffer Font..." I select
Norasi.  M-x describe-char then reports

         character: ◌ (9676, #o22714, #x25cc)
preferred charset: unicode-bmp
 		   (Unicode Basic Multilingual Plane (U+0000..U+FFFF))
        code point: 0x25CC
            syntax: w 	which means: word
          category: .:Base
       buffer code: #xE2 #x97 #x8C
         file code: #xE2 #x97 #x8C (encoded by coding system utf-8-unix)
           display: by this font (glyph code)
     xft:-unknown-Norasi-normal-normal-normal-*-13-*-*-*-*-0-iso10646-1 (#x146)

and

         character: ึ (3638, #o7066, #xe36)
preferred charset: unicode-bmp
 		   (Unicode Basic Multilingual Plane (U+0000..U+FFFF))
        code point: 0x0E36
            syntax: w 	which means: word
          category: 2:upper diacritic, t:Thai
       buffer code: #xE0 #xB8 #xB6
         file code: #xE0 #xB8 #xB6 (encoded by coding system utf-8-unix)
           display: by this font (glyph code)
     xft:-unknown-Norasi-normal-normal-normal-*-13-*-*-*-*-0-iso10646-1 (#x109)

The characters remain uncombined.

bug#17739: 24.3; Dotted circle does not combine with Thai vowel markers

[Eli Zaretskii]

> Date: Sun, 8 Jun 2014 16:37:55 -0500 (CDT)
> From: Brady Garvin <bgarvin@cse.unl.edu>
> 
> Thanks.  However, I see the same problem even putting both characters
> in the same font.  With S-mouse-1 "Change Buffer Font..." I select
> Norasi.  M-x describe-char then reports
> 
>          character: ◌ (9676, #o22714, #x25cc)
> preferred charset: unicode-bmp
>  		   (Unicode Basic Multilingual Plane (U+0000..U+FFFF))
>         code point: 0x25CC
>             syntax: w 	which means: word
>           category: .:Base
>        buffer code: #xE2 #x97 #x8C
>          file code: #xE2 #x97 #x8C (encoded by coding system utf-8-unix)
>            display: by this font (glyph code)
>      xft:-unknown-Norasi-normal-normal-normal-*-13-*-*-*-*-0-iso10646-1 (#x146)
> 
> and
> 
>          character: ึ (3638, #o7066, #xe36)
> preferred charset: unicode-bmp
>  		   (Unicode Basic Multilingual Plane (U+0000..U+FFFF))
>         code point: 0x0E36
>             syntax: w 	which means: word
>           category: 2:upper diacritic, t:Thai
>        buffer code: #xE0 #xB8 #xB6
>          file code: #xE0 #xB8 #xB6 (encoded by coding system utf-8-unix)
>            display: by this font (glyph code)
>      xft:-unknown-Norasi-normal-normal-normal-*-13-*-*-*-*-0-iso10646-1 (#x109)
> 
> The characters remain uncombined.

Check out the character composition rules in thai-util.el, perhaps
something is missing there.

bug#17739: 24.3; Dotted circle does not combine with Thai vowel markers

[Brady Garvin]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1027 bytes --]

>
> Check out the character composition rules in thai-util.el, perhaps
> something is missing there.
>

Thanks for the pointer.  With

58c58,59
< (let ((l '((?ก consonant "LETTER KO KAI")				; 0xA1
---
> (let ((l '((?◌ consonant "DOTTED CIRCLE")
> 	   (?ก consonant "LETTER KO KAI")				; 0xA1

and a reevaluation of the relevant s-expression, thai-compose-region
and thai-compose-buffer give me almost correct results for that test
case.  The vowel does combine, but renders oddly:

When the cursor is over the dotted circle, the vowel, which should
appear roughly centered above, is shown above and to the left.

When the cursor is just to the right of the dotted circle, the vowel
is shown in the same place, but the character to the left of the
dotted circle is occluded by another dotted circle.

When the cursor is just to the left of the dotted circle, the vowel is
shown above and to the right.

When the cursor is elsewhere, the rendering is unchanged from its most
recent state.

Perhaps that's a separate issue?

bug#17739: 24.3; Dotted circle does not combine with Thai vowel markers

[Eli Zaretskii]

> Date: Sun, 8 Jun 2014 22:17:49 -0500 (CDT)
> From: Brady Garvin <bgarvin@cse.unl.edu>
> 
> Thanks for the pointer.  With
> 
> 58c58,59
> < (let ((l '((?ก consonant "LETTER KO KAI")				; 0xA1
> ---
> > (let ((l '((?◌ consonant "DOTTED CIRCLE")
> > 	   (?ก consonant "LETTER KO KAI")				; 0xA1
> 
> and a reevaluation of the relevant s-expression, thai-compose-region
> and thai-compose-buffer give me almost correct results for that test
> case.  The vowel does combine, but renders oddly:
> 
> When the cursor is over the dotted circle, the vowel, which should
> appear roughly centered above, is shown above and to the left.
> 
> When the cursor is just to the right of the dotted circle, the vowel
> is shown in the same place, but the character to the left of the
> dotted circle is occluded by another dotted circle.
> 
> When the cursor is just to the left of the dotted circle, the vowel is
> shown above and to the right.
> 
> When the cursor is elsewhere, the rendering is unchanged from its most
> recent state.
> 
> Perhaps that's a separate issue?

The first suspect is your font driver.  As described in INSTALL, you
need to build Emacs with libotf, m17n-db, and libm17n-flt libraries to
correctly display complex scripts.  Did you build Emacs with those?

bug#17739: 24.3; Dotted circle does not combine with Thai vowel markers

[Brady Garvin]

>
> The first suspect is your font driver.  As described in INSTALL, you
> need to build Emacs with libotf, m17n-db, and libm17n-flt libraries to
> correctly display complex scripts.  Did you build Emacs with those?
>

Not sure; I'm using the Ubuntu package emacs24.  I'll try a fresh
build after work and, if that fixes it, send a report to Canonical.

bug#17739: 24.3; Dotted circle does not combine with Thai vowel markers

[K. Handa]

In article <838up63j1x.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> Check out the character composition rules in thai-util.el, perhaps
> something is missing there.

The function for composing Thai is thai-composition-function
in thai-util.el and which surely doesn't handle
dotted-circle.  I think what we need is a general function
to compose dotted-circle with the following character
artifically.  When you eval the following code:

(defun compose-dotted-circle (gstring)
  (let* ((dotted-circle-glyph (lgstring-glyph gstring 0))
	 (following-glyph (lgstring-glyph gstring 1))
	 (from (lglyph-from dotted-circle-glyph))
	 (to (lglyph-to following-glyph)))
    (lglyph-set-from-to dotted-circle-glyph from to)
    (lglyph-set-from-to following-glyph from to)
    gstring))

(aset composition-function-table ?◌ '(["◌[^◌ ]" 0 compose-dotted-circle]))

and Thai characters and dotted circle use the same Norasi
font, I think you get a good result.  Could you please try?

But, the above function is not yet perfect.  In some fonts,
a glyhp of dotted-circle may have positive width, a glyph of
a character following dotted-circle may be narrower than the
glyph of dotted-circle.  The function should handle those
cases gracefully.  I'm now working on it.

---
Kenichi Handa
handa@gnu.org

bug#17739: 24.3; Dotted circle does not combine with Thai vowel markers

[K. Handa]

In article <877g4n5sww.fsf@gnu.org>, handa@gnu.org (K. Handa) writes:

> But, the above function is not yet perfect.  In some fonts,
> a glyhp of dotted-circle may have positive width, a glyph of
> a character following dotted-circle may be narrower than the
> glyph of dotted-circle.  The function should handle those
> cases gracefully.  I'm now working on it.

I've just committed a function to handle dotted-circle
properly to the trunk.

---
Kenichi Handa
handa@gnu.org

bug#17739: 24.3; Dotted circle does not combine with Thai vowel markers

[Stefan Kangas]

handa@gnu.org (K. Handa) writes:

> In article <877g4n5sww.fsf@gnu.org>, handa@gnu.org (K. Handa) writes:
>
>> But, the above function is not yet perfect.  In some fonts,
>> a glyhp of dotted-circle may have positive width, a glyph of
>> a character following dotted-circle may be narrower than the
>> glyph of dotted-circle.  The function should handle those
>> cases gracefully.  I'm now working on it.
>
> I've just committed a function to handle dotted-circle
> properly to the trunk.

Since there have been no further updates in 5 years, I'm going to
assume that the fix worked as expected and close this bug.  If that is
incorrect, please report back and we can reopen the bug report.

Best regards,
Stefan Kangas

bug#17739: 24.3; Dotted circle does not combine with Thai vowel markers

[Eli Zaretskii]

> From: Stefan Kangas <stefan@marxist.se>
> Date: Sat, 21 Sep 2019 01:04:37 +0200
> Cc: bgarvin@cse.unl.edu, 17739-done@debbugs.gnu.org, 
> 	Eli Zaretskii <eliz@gnu.org>
> 
> handa@gnu.org (K. Handa) writes:
> 
> > In article <877g4n5sww.fsf@gnu.org>, handa@gnu.org (K. Handa) writes:
> >
> >> But, the above function is not yet perfect.  In some fonts,
> >> a glyhp of dotted-circle may have positive width, a glyph of
> >> a character following dotted-circle may be narrower than the
> >> glyph of dotted-circle.  The function should handle those
> >> cases gracefully.  I'm now working on it.
> >
> > I've just committed a function to handle dotted-circle
> > properly to the trunk.
> 
> Since there have been no further updates in 5 years, I'm going to
> assume that the fix worked as expected and close this bug.  If that is
> incorrect, please report back and we can reopen the bug report.

The original recipe still doesn't display correctly on the current
master, so I think something is still missing.  Not sure what exactly
is missing.

bug#17739: 24.3; Dotted circle does not combine with Thai vowel markers

[Stefan Kangas]

[-- Attachment #1: Type: text/plain, Size: 3772 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

> > > I've just committed a function to handle dotted-circle
> > > properly to the trunk.
> >
> > Since there have been no further updates in 5 years, I'm going to
> > assume that the fix worked as expected and close this bug.  If that is
> > incorrect, please report back and we can reopen the bug report.
>
> The original recipe still doesn't display correctly on the current
> master, so I think something is still missing.  Not sure what exactly
> is missing.

OK, you're right.  Reopening.

However, it works properly for me using my init file AFAICT.  I'm
using the following recipe:

1. C-x 8 RET
2. DOTTED CIRCLE RET
3. C-x 8 RET
4. THAI CHARACTER SARA UE RET

1. Using my init file, the characters compose and M-x describe-char gives:

             position: 300 of 3559 (8%), column: 9
            character: ◌ (displayed as ◌) (codepoint 9676, #o22714, #x25cc)
              charset: unicode (Unicode (ISO10646))
code point in charset: 0x25CC
               script: symbol
               syntax: w     which means: word
             category: .:Base
             to input: type "C-x 8 RET 25cc" or "C-x 8 RET DOTTED CIRCLE"
          buffer code: #xE2 #x97 #x8C
            file code: #xE2 #x97 #x8C (encoded by coding system utf-8-unix)
              display: composed to form "◌ึ" (see below)

Composed with the following character(s) "ึ" using this font:
  x:-mutt-clearlyu-medium-r-normal--17-120-100-100-p-123-iso10646-1
by these glyphs:
  [0 1 9676 9676 14 2 13 12 -1 nil]
  [0 1 3638 3638 0 -7 0 14 -9 [0 -3 0]]

Character code properties: customize what to show
  name: DOTTED CIRCLE
  general-category: So (Symbol, Other)
  decomposition: (9676) ('◌')

I've attached a screenshot of the above.

2. When I run under emacs -Q, I see two characters next to each other
and M-x describe-char says:

a)

             position: 146 of 147 (99%), column: 0
            character: ◌ (displayed as ◌) (codepoint 9676, #o22714, #x25cc)
              charset: unicode (Unicode (ISO10646))
code point in charset: 0x25CC
               script: symbol
               syntax: w     which means: word
             category: .:Base
             to input: type "C-x 8 RET 25cc" or "C-x 8 RET DOTTED CIRCLE"
          buffer code: #xE2 #x97 #x8C
            file code: #xE2 #x97 #x8C (encoded by coding system utf-8-unix)
              display: by this font (glyph code)
    xfthb:-PfEd-DejaVu Sans
Mono-normal-normal-normal-*-16-*-*-*-m-0-iso10646-1 (#xA23)

Character code properties: customize what to show
  name: DOTTED CIRCLE
  general-category: So (Symbol, Other)
  decomposition: (9676) ('◌')

b)

             position: 147 of 147 (99%), column: 1
            character: ึ (displayed as ึ) (codepoint 3638, #o7066, #xe36)
              charset: unicode (Unicode (ISO10646))
code point in charset: 0x0E36
               script: thai
               syntax: w     which means: word
             category: 2:upper diacritic, ^:Combining, t:Thai
             to input: type "C-x 8 RET e36" or "C-x 8 RET THAI
CHARACTER SARA UE"
          buffer code: #xE0 #xB8 #xB6
            file code: #xE0 #xB8 #xB6 (encoded by coding system utf-8-unix)
              display: by this font (glyph code)
    xfthb:-GOOG-Noto Sans Thai
UI-normal-normal-normal-*-16-*-*-*-*-0-iso10646-1 (#x77)

Character code properties: customize what to show
  name: THAI CHARACTER SARA UE
  old-name: THAI VOWEL SIGN SARA UE
  general-category: Mn (Mark, Nonspacing)
  decomposition: (3638) ('ึ')

I don't know much of anything about this, but wouldn't that suggest
that the bug is related to the font?

Best regards,
Stefan Kangas

[-- Attachment #2: bug-17739-sk.png --]
[-- Type: image/png, Size: 4495 bytes --]

bug#17739: 24.3; Dotted circle does not combine with Thai vowel markers

[Eli Zaretskii]

> From: Stefan Kangas <stefan@marxist.se>
> Date: Sat, 21 Sep 2019 11:36:54 +0200
> Cc: handa@gnu.org, bgarvin@cse.unl.edu, 17739@debbugs.gnu.org
> 
> I don't know much of anything about this, but wouldn't that suggest
> that the bug is related to the font?

It is in your case.  But in my case, even if I start Emacs with
"emacs -Q -fn FONT", where FONT is a font that can display both the
dotted circle and the Thai character, I still see them not combining,
for some reason.

bug#17739: 24.3; Dotted circle does not combine with Thai vowel markers

[Stefan Kangas]

Stefan Kangas <stefan@marxist.se> writes:

> I don't know much of anything about this, but wouldn't that suggest
> that the bug is related to the font?

Also, could this be related to Bug#17261?

Best regards,
Stefan Kangas

bug#17739: 24.3; Dotted circle does not combine with Thai vowel markers

[Eli Zaretskii]

> From: Stefan Kangas <stefan@marxist.se>
> Date: Sat, 21 Sep 2019 11:53:04 +0200
> Cc: handa@gnu.org, bgarvin@cse.unl.edu, 17739@debbugs.gnu.org
> 
> Stefan Kangas <stefan@marxist.se> writes:
> 
> > I don't know much of anything about this, but wouldn't that suggest
> > that the bug is related to the font?
> 
> Also, could this be related to Bug#17261?

No.