bug#44784: Combining accent is not combining

bug#44784: Combining accent is not combining

[Juri Linkov]

[-- Attachment #1: Type: text/plain, Size: 258 bytes --]

Sorry that I noticed this regression only now,
but it's a pretty bad-looking regression:

0. emacs -Q
1. insert to the *scratch* buffer:

ä́  і́  Я́ 

In 26.3 everything is correct, and COMBINING ACUTE ACCENT
is combined above its previous character:


[-- Attachment #2: comb26.png --]
[-- Type: image/png, Size: 2983 bytes --]

[-- Attachment #3: Type: text/plain, Size: 590 bytes --]


Composed with the following character(s) "́" using this font:
  xft:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-22-*-*-*-m-0-iso10646-1
by these glyphs:
  [0 1 1071 878 13 0 13 21 0 nil]
  [0 1 769 649 0 5 11 18 -14 [-15 -4 0]]
In 26.3 (x86_64-pc-linux-gnu, GTK+ Version 3.24.14)
Configured features:
XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GSETTINGS GLIB
NOTIFY ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB
TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM THREADS LIBSYSTEMD LCMS2

But in 27.1 COMBINING ACUTE ACCENT is not combined with its previous character:


[-- Attachment #4: comb27.png --]
[-- Type: image/png, Size: 2958 bytes --]

[-- Attachment #5: Type: text/plain, Size: 792 bytes --]


Composed with the following character(s) "́" using this font:
  xfthb:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-22-*-*-*-m-0-iso10646-1
by these glyphs:
  [0 1 1071 878 13 0 13 16 0 nil]
  [0 1 769 649 13 4 11 17 -13 [0 0 0]]
In 27.1 (x86_64-pc-linux-gnu, GTK+ Version 3.24.20)
Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND GPM DBUS GSETTINGS GLIB NOTIFY INOTIFY
ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE HARFBUZZ M17N_FLT LIBOTF XFT ZLIB
TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM MODULES THREADS LIBSYSTEMD JSON
PDUMPER LCMS2 GMP

It seems in 26.3 it's correct because it uses XFT, but the problem in 27.1
is with HarfBuzz .  Maybe HarfBuzz doesn't support combining accents on
some characters?  It fails with the scripts above, but still has no problems
with plain Latin characters.

bug#44784: Combining accent is not combining

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Date: Sat, 21 Nov 2020 21:17:09 +0200
> 
> 0. emacs -Q
> 1. insert to the *scratch* buffer:
> 
> ä́  і́  Я́ 
> 
> In 26.3 everything is correct, and COMBINING ACUTE ACCENT
> is combined above its previous character:
> 
> 
> But in 27.1 COMBINING ACUTE ACCENT is not combined with its previous character:

On my system, this displays the same in both Emacs 26 and 27, the
latter using HarfBuzz.

> It seems in 26.3 it's correct because it uses XFT, but the problem in 27.1
> is with HarfBuzz .  Maybe HarfBuzz doesn't support combining accents on
> some characters?  It fails with the scripts above, but still has no problems
> with plain Latin characters.

That cannot be the reason.  Please try with other fonts, for
starters.  DejaVu Sans Mono is known to have problems with combining
accents, see this discussion:

  https://lists.freedesktop.org/archives/harfbuzz/2019-August/007422.html

bug#44784: Combining accent is not combining

[Andreas Schwab]

On Nov 21 2020, Juri Linkov wrote:

> But in 27.1 COMBINING ACUTE ACCENT is not combined with its previous character:

Do they come from the same font?

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

bug#44784: Combining accent is not combining

[Eli Zaretskii]

Btw, the report by Emacs 27's "C-u C-x =", viz.:

  Composed with the following character(s) "́" using this font:
    xfthb:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-22-*-*-*-m-0-iso10646-1
  by these glyphs:
    [0 1 1071 878 13 0 13 16 0 nil]
    [0 1 769 649 13 4 11 17 -13 [0 0 0]]

explicitly says that character composition _did_ happen here, it's
just the display that is broken.  And I guess that moving point to
these characters will show a "fat" cursor that includes both the base
character and the accent, once again showing that Emacs did compose
the accent with the preceding character.

bug#44784: Combining accent is not combining

[Juri Linkov]

>> But in 27.1 COMBINING ACUTE ACCENT is not combined with its previous character:
>
> Do they come from the same font?

Yes, both use the same font:

Composed with the following character(s) "́" using this font:
  ftcrhb:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-13-*-*-*-m-0-iso10646-1

The component character(s) are displayed by these fonts (glyph codes):
 ́: ftcrhb:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-13-*-*-*-m-0-iso10646-1 (#x289)

bug#44784: Combining accent is not combining

[Juri Linkov]

> Btw, the report by Emacs 27's "C-u C-x =", viz.:
>
>   Composed with the following character(s) "́" using this font:
>     xfthb:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-22-*-*-*-m-0-iso10646-1
>   by these glyphs:
>     [0 1 1071 878 13 0 13 16 0 nil]
>     [0 1 769 649 13 4 11 17 -13 [0 0 0]]
>
> explicitly says that character composition _did_ happen here, it's
> just the display that is broken.  And I guess that moving point to
> these characters will show a "fat" cursor that includes both the base
> character and the accent, once again showing that Emacs did compose
> the accent with the preceding character.

Actually, separate cursor is shown on a character and separate on its
combining character, so it seems they are not composed.

>> It seems in 26.3 it's correct because it uses XFT, but the problem in 27.1
>> is with HarfBuzz .  Maybe HarfBuzz doesn't support combining accents on
>> some characters?
>
> That cannot be the reason.  Please try with other fonts, for
> starters.  DejaVu Sans Mono is known to have problems with combining
> accents, see this discussion:

Yes, it displays correctly with other fonts, e.g. Noto Sans.
It would be a pity to switch to another font.  So this bug report
could be closed if nothing can be done in Emacs.

bug#44784: Combining accent is not combining

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: 44784@debbugs.gnu.org
> Date: Sat, 21 Nov 2020 22:35:11 +0200
> 
> > Btw, the report by Emacs 27's "C-u C-x =", viz.:
> >
> >   Composed with the following character(s) "́" using this font:
> >     xfthb:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-22-*-*-*-m-0-iso10646-1
> >   by these glyphs:
> >     [0 1 1071 878 13 0 13 16 0 nil]
> >     [0 1 769 649 13 4 11 17 -13 [0 0 0]]
> >
> > explicitly says that character composition _did_ happen here, it's
> > just the display that is broken.  And I guess that moving point to
> > these characters will show a "fat" cursor that includes both the base
> > character and the accent, once again showing that Emacs did compose
> > the accent with the preceding character.
> 
> Actually, separate cursor is shown on a character and separate on its
> combining character, so it seems they are not composed.

The 2 vectors you show above, let alone the "composed with" part,
explicitly say they _were_ composed.  So something else is going on
here.

bug#44784: Combining accent is not combining

[Stephen Berman]

On Sun, 22 Nov 2020 05:35:30 +0200 Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Juri Linkov <juri@linkov.net>
>> Cc: 44784@debbugs.gnu.org
>> Date: Sat, 21 Nov 2020 22:35:11 +0200
>> 
>> > Btw, the report by Emacs 27's "C-u C-x =", viz.:
>> >
>> >   Composed with the following character(s) "́" using this font:
>> >     xfthb:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-22-*-*-*-m-0-iso10646-1
>> >   by these glyphs:
>> >     [0 1 1071 878 13 0 13 16 0 nil]
>> >     [0 1 769 649 13 4 11 17 -13 [0 0 0]]
>> >
>> > explicitly says that character composition _did_ happen here, it's
>> > just the display that is broken.  And I guess that moving point to
>> > these characters will show a "fat" cursor that includes both the base
>> > character and the accent, once again showing that Emacs did compose
>> > the accent with the preceding character.
>> 
>> Actually, separate cursor is shown on a character and separate on its
>> combining character, so it seems they are not composed.
>
> The 2 vectors you show above, let alone the "composed with" part,
> explicitly say they _were_ composed.  So something else is going on
> here.

I think it's a display problem with DejaVu Sans Mono, as you already
noted.  I also see what Juri sees, but when the cursor appears to be
over the separately displayed accent, `C-u C-x =' says the character at
point is SPC.

Steve Berman

bug#44784: Combining accent is not combining

[Eli Zaretskii]

> From: Stephen Berman <stephen.berman@gmx.net>
> Cc: Juri Linkov <juri@linkov.net>,  44784@debbugs.gnu.org
> Date: Sun, 22 Nov 2020 11:12:03 +0100
> 
> > The 2 vectors you show above, let alone the "composed with" part,
> > explicitly say they _were_ composed.  So something else is going on
> > here.
> 
> I think it's a display problem with DejaVu Sans Mono, as you already
> noted.  I also see what Juri sees, but when the cursor appears to be
> over the separately displayed accent, `C-u C-x =' says the character at
> point is SPC.

Right.  I think the discussion on the HarfBuzz list to which I pointed
explains what happens: the problematic fonts don't provide the
information necessary to correctly locate the accent over some
character (specifically, precomposed characters that are already
accented).  The [0 0 0] part of the second glyph in the composition
info is a telltale sign of lack of that information: if you try the
same with a font where display is correct, you will see that the first
2 components are non-zero (notably, the first component should be
negative).

bug#44784: Combining accent is not combining

[Stephen Berman]

On Sun, 22 Nov 2020 17:28:08 +0200 Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Stephen Berman <stephen.berman@gmx.net>
>> Cc: Juri Linkov <juri@linkov.net>,  44784@debbugs.gnu.org
>> Date: Sun, 22 Nov 2020 11:12:03 +0100
>> 
>> > The 2 vectors you show above, let alone the "composed with" part,
>> > explicitly say they _were_ composed.  So something else is going on
>> > here.
>> 
>> I think it's a display problem with DejaVu Sans Mono, as you already
>> noted.  I also see what Juri sees, but when the cursor appears to be
>> over the separately displayed accent, `C-u C-x =' says the character at
>> point is SPC.
>
> Right.  I think the discussion on the HarfBuzz list to which I pointed
> explains what happens: the problematic fonts don't provide the
> information necessary to correctly locate the accent over some
> character (specifically, precomposed characters that are already
> accented).  The [0 0 0] part of the second glyph in the composition
> info is a telltale sign of lack of that information: if you try the
> same with a font where display is correct, you will see that the first
> 2 components are non-zero (notably, the first component should be
> negative).

FTR, here's what `C-u C-x =' shows for the correct and incorrect
displays, respectively:

Composed with the following character(s) "́" using this font:
  ftcrhb:-PfEd-DejaVu Sans-normal-normal-normal-*-13-*-*-*-*-0-iso10646-1
by these glyphs:
  [0 1 1110 243 4 1 3 7 0 nil]
  [0 1 769 690 0 -5 -1 11 -7 [1 0 0]]
with these character(s):
  ́ (#x301) COMBINING ACUTE ACCENT

Composed with the following character(s) "́" using this font:
  ftcrhb:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-13-*-*-*-m-0-iso10646-1
by these glyphs:
  [0 1 1110 917 8 1 8 10 0 nil]
  [0 1 769 649 8 3 8 11 -8 [0 0 0]]
with these character(s):
  ́ (#x301) COMBINING ACUTE ACCENT

Steve Berman

bug#44784: Combining accent is not combining

[Eli Zaretskii]

> From: Stephen Berman <stephen.berman@gmx.net>
> Cc: juri@linkov.net,  44784@debbugs.gnu.org
> Date: Sun, 22 Nov 2020 17:28:09 +0100
> 
> FTR, here's what `C-u C-x =' shows for the correct and incorrect
> displays, respectively:
> 
> Composed with the following character(s) "́" using this font:
>   ftcrhb:-PfEd-DejaVu Sans-normal-normal-normal-*-13-*-*-*-*-0-iso10646-1
> by these glyphs:
>   [0 1 1110 243 4 1 3 7 0 nil]
>   [0 1 769 690 0 -5 -1 11 -7 [1 0 0]]
> with these character(s):
>   ́ (#x301) COMBINING ACUTE ACCENT
> 
> Composed with the following character(s) "́" using this font:
>   ftcrhb:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-13-*-*-*-m-0-iso10646-1
> by these glyphs:
>   [0 1 1110 917 8 1 8 10 0 nil]
>   [0 1 769 649 8 3 8 11 -8 [0 0 0]]
> with these character(s):
>   ́ (#x301) COMBINING ACUTE ACCENT

OK, zero WIDTH and negative LBEARING/RBEARING is another way to place
the accent correctly.

bug#44784: Combining accent is not combining

[Lars Ingebrigtsen]

[-- Attachment #1: Type: text/plain, Size: 384 bytes --]

Juri Linkov <juri@linkov.net> writes:

> Yes, it displays correctly with other fonts, e.g. Noto Sans.
> It would be a pity to switch to another font.  So this bug report
> could be closed if nothing can be done in Emacs.

Skimming this bug report, it's not clear whether this was a problem with
the fonts or in Emacs.  If I do emacs -Q -fn "Noto Sans" with the
current trunk, I get:


[-- Attachment #2: Type: image/png, Size: 1258 bytes --]

[-- Attachment #3: Type: text/plain, Size: 224 bytes --]


So the combining accent ̷ is combining with the o correctly here.

So is there anything more to be done here?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#44784: Combining accent is not combining

[Juri Linkov]

[-- Attachment #1: Type: text/plain, Size: 525 bytes --]

>> Yes, it displays correctly with other fonts, e.g. Noto Sans.
>> It would be a pity to switch to another font.  So this bug report
>> could be closed if nothing can be done in Emacs.
>
> Skimming this bug report, it's not clear whether this was a problem with
> the fonts or in Emacs.  If I do emacs -Q -fn "Noto Sans" with the
> current trunk, I get:
>
> So the combining accent ̷ is combining with the o correctly here.
>
> So is there anything more to be done here?

I confirm that "DejaVu Sans Mono" is still broken:


[-- Attachment #2: noncombining.png --]
[-- Type: image/png, Size: 1758 bytes --]

[-- Attachment #3: Type: text/plain, Size: 65 bytes --]


whereas other fonts can correctly handle combining characters:


[-- Attachment #4: combining.png --]
[-- Type: image/png, Size: 2045 bytes --]

[-- Attachment #5: Type: text/plain, Size: 98 bytes --]


Please close this bug report if it's not possible
to implement some workaround for broken fonts.

bug#44784: Combining accent is not combining

[Robert Pluim]

>>>>> On Thu, 10 Feb 2022 10:40:55 +0200, Juri Linkov <juri@linkov.net> said:

    Juri> Please close this bug report if it's not possible
    Juri> to implement some workaround for broken fonts.

Itʼs possible, since we do something similar for certain emoji
sequences. I question whether itʼs worth doing, though, it seems like
very much a rare condition, which is easily fixed by switching fonts.

Robert
-- 

bug#44784: Combining accent is not combining

[Lars Ingebrigtsen]

Robert Pluim <rpluim@gmail.com> writes:

> Itʼs possible, since we do something similar for certain emoji
> sequences. I question whether itʼs worth doing, though, it seems like
> very much a rare condition, which is easily fixed by switching fonts.

I guess so.  I'm surprised that DejaVu Sans Mono is broken, though, so
perhaps there is something else going on.

This reminds me of a related thing I've wondered about: We don't combine
glyphs from different fonts, because this may look very ugly.  But has
anybody tried doing it anyway?  It may not look worse than what some
people are seeing today...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#44784: Combining accent is not combining

[Eli Zaretskii]

> Resent-From: Lars Ingebrigtsen <larsi@gnus.org>
> Original-Sender: "Debbugs-submit" <debbugs-submit-bounces@debbugs.gnu.org>
> Resent-CC: bug-gnu-emacs@gnu.org
> Resent-Sender: help-debbugs@gnu.org
> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Thu, 10 Feb 2022 12:19:11 +0100
> Cc: 44784@debbugs.gnu.org, Juri Linkov <juri@linkov.net>
> 
> Robert Pluim <rpluim@gmail.com> writes:
> 
> > Itʼs possible, since we do something similar for certain emoji
> > sequences. I question whether itʼs worth doing, though, it seems like
> > very much a rare condition, which is easily fixed by switching fonts.
> 
> I guess so.  I'm surprised that DejaVu Sans Mono is broken, though, so
> perhaps there is something else going on.
> 
> This reminds me of a related thing I've wondered about: We don't combine
> glyphs from different fonts, because this may look very ugly.  But has
> anybody tried doing it anyway?  It may not look worse than what some
> people are seeing today...

I think it depends on the two fonts and the characters to compose.  It
_could_ be much worse, there's no doubt about that.  Different fonts
use very different styles for the same glyphs.