bug#39554: 27.0.50; cairo not composing sequences

bug#39554: 27.0.50; cairo not composing sequences

[James Cloos]

Sequences like 0̸ fail to display composed in master --with-cairo but do
when usin xft.

In a version w/o cairo I get:

Composed with the following character(s) "̸" using this font:
  xft:-unknown-DejaVu Sans Mono-normal-normal-normal-*-22-*-*-*-m-0-iso10646-1
by these glyphs:

and the single char takes up the same width as any ascii letter.

W/ cair i get:

Composed with the following character(s) "̸" using this font:
  ftcrhb:-unknown-DejaVu Sans Mono-normal-normal-normal-*-22-*-*-*-m-0-iso10646-1
by these glyphs:
  [0 1 48 19 13 1 12 16 0 nil]
  [0 1 824 704 13 0 13 17 1 nil]

and the single char takes twice the expected width, but still works as a
sing;e char.  OTOH, in the *Help* buffer '"̸"' is three separate chars.
Buth with xft '"̸"' displays with the slash overlaying the first ".
As it should.

The ftcrhb: code needs to display the combining chars over the base
chars like the earlier code does.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 0x997A9F17ED7DAEA6

bug#39554: 27.0.50; cairo not composing sequences

[Eli Zaretskii]

> From: James Cloos <cloos@jhcloos.com>
> Date: Mon, 10 Feb 2020 15:53:32 -0500
> 
> Sequences like 0̸ fail to display composed in master --with-cairo but do
> when usin xft.

Please show a complete reproducing recipe for this problem.

> In a version w/o cairo I get:
> 
> Composed with the following character(s) "̸" using this font:
>   xft:-unknown-DejaVu Sans Mono-normal-normal-normal-*-22-*-*-*-m-0-iso10646-1
> by these glyphs:
> 
> and the single char takes up the same width as any ascii letter.
> 
> W/ cair i get:
> 
> Composed with the following character(s) "̸" using this font:
>   ftcrhb:-unknown-DejaVu Sans Mono-normal-normal-normal-*-22-*-*-*-m-0-iso10646-1
> by these glyphs:
>   [0 1 48 19 13 1 12 16 0 nil]
>   [0 1 824 704 13 0 13 17 1 nil]
> 
> and the single char takes twice the expected width, but still works as a
> sing;e char.  OTOH, in the *Help* buffer '"̸"' is three separate chars.
> Buth with xft '"̸"' displays with the slash overlaying the first ".
> As it should.

This means that the font backend couldn't produce a single glyph for
the character combination, for some reason, so it displayed the
original glyphs as a single grapheme cluster.  IOW, character
composition did work, it just didn't find a precomposed glyph in the
font, or maybe the precomposed glyph was rejected for some reason.

We need the detailed use case to investigate.

Please also tell what is your version of HarfBuzz, in case this
matters.

And in the XFT case, what was the shaping engine? was it libflt?

Thanks.

bug#39554: 27.0.50; cairo not composing sequences

[Eli Zaretskii]

> From: James Cloos <cloos@jhcloos.com>
> Date: Mon, 10 Feb 2020 15:53:32 -0500
> 
> Sequences like 0̸ fail to display composed in master --with-cairo but do
> when usin xft.

And one more thing: if your master branch is at version 27.0.50, then
it is quite old.  Please try the latest master or the emacs-27 branch.

bug#39554: 27.0.50; cairo not composing sequences

[James Cloos]

>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:

EZ> And one more thing: if your master branch is at version 27.0.50, then
EZ> it is quite old.  Please try the latest master or the emacs-27 branch.

as i noted in my followup, master this month (at least) breaks gnus, so i
must still use december's compile for mail.  but the tests were on master
of the day i sent them.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 0x997A9F17ED7DAEA6

bug#39554: 27.0.50; cairo not composing sequences

[Eli Zaretskii]

[Please keep the bug number on the CC list.]

> From: James Cloos <cloos@jhcloos.com>
> Date: Wed, 12 Feb 2020 13:51:53 -0500
> 
> >>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Sequences like 0̸ fail to display composed in master --with-cairo but do
> >> when usin xft.
> 
> EZ> Please show a complete reproducing recipe for this problem.
> 
> The 0̸ in thequoted line is one.

Sorry, I failed to realize that ̸ was a combining accent, not an ASCII
slash.

In an Emacs built with HarfBuzz on MS-Windows, if I use a font that
has support for ̸, I do see these two characters composed into a single
glyph whose width is as that of a single character.  But if I use
DejaVu Sans Mono, I indeed see a double-width grapheme cluster.  So I
think this might be related to font selection somehow.  Can you try
different monospaced fonts and see if the results in the Cairo build
are better with other fonts?

> EZ> This means that the font backend couldn't produce a single glyph for
> EZ> the character combination, for some reason, so it displayed the
> EZ> original glyphs as a single grapheme cluster.  IOW, character
> EZ> composition did work, it just didn't find a precomposed glyph in the
> EZ> font, or maybe the precomposed glyph was rejected for some reason.
> 
> It is not supposed to be looking for precomposed glyphs.  It is supposed
> to be rendering each combining glyph on top of the base glyph.  Just
> like xft does.

The way character composition works in Emacs, we first ask the font
for precomposed glyphs, and display them if the font has them.  If
that fails, then we combine the separate glyphs ourselves.  See
compose-gstring-for-graphic.

bug#39554: 27.0.50; cairo not composing sequences

[James Cloos]

It does work correctly when i turn on variable-pitch-mode, which here is
set to use DejaVu Serif.

But xft does fine with DejaVu Sans Mono.

This bug remains a regression and requires a fix.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 0x997A9F17ED7DAEA6

bug#39554: 27.0.50; cairo not composing sequences

[Eli Zaretskii]

> From: James Cloos <cloos@jhcloos.com>
> Cc: 39554@debbugs.gnu.org
> Date: Thu, 13 Feb 2020 15:16:39 -0500
> 
> It does work correctly when i turn on variable-pitch-mode, which here is
> set to use DejaVu Serif.
> 
> But xft does fine with DejaVu Sans Mono.
> 
> This bug remains a regression and requires a fix.

I don't think I agree.  This font has problems with combining accents,
see this discussion:

  https://lists.freedesktop.org/archives/harfbuzz/2019-August/007422.html

bug#39554: 27.0.50; cairo not composing sequences

[James Cloos]

>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:

EZ> I don't think I agree.  This font has problems with combining accents,

if xft is to be depricated in favour of cairo, then anything which works
in xft but fails in cairo is be definition a regression.

and ir works perfectly irrespective of font with xft.

so the fact that --with-cairo gets it wrong is indisputably a regression
in emacs on x11.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 0x997A9F17ED7DAEA6

bug#39554: 27.0.50; cairo not composing sequences

[YAMAMOTO Mitsuharu]

On Fri, 14 Feb 2020 09:58:05 +0900,
James Cloos wrote:
> 
> >>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:
> 
> EZ> I don't think I agree.  This font has problems with combining accents,
> 
> if xft is to be depricated in favour of cairo, then anything which works
> in xft but fails in cairo is be definition a regression.
> 
> and ir works perfectly irrespective of font with xft.
> 
> so the fact that --with-cairo gets it wrong is indisputably a regression
> in emacs on x11.

I guess the difference comes from with vs. without HarfBuzz rather
than cairo vs. xft.  Could you check if that is the case on your
environment?

				     YAMAMOTO Mitsuharu
				mituharu@math.s.chiba-u.ac.jp

bug#39554: 27.0.50; cairo not composing sequences

[James Cloos]

>>>>> "YM" == YAMAMOTO Mitsuharu <mituharu@math.s.chiba-u.ac.jp> writes:

YM> I guess the difference comes from with vs. without HarfBuzz rather
YM> than cairo vs. xft.

makes sense.

YM> Could you check if that is the case on your environment?

all i know is that w/o cairo, describe-char reports:

    xft:-unknown-DejaVu Sans Mono-normal-normal-normal-*-22-*-*-*-m-0-iso10646-1
by these glyphs:
  [0 1 48 19 13 0 13 17 1 nil]
  [0 1 824 704 0 0 13 17 1 [-13 0 0]]

and w/ cairo it reports:

    ftcrhb:-unknown-DejaVu Serif-normal-normal-normal-*-22-*-*-*-*-0-iso10646-1
by these glyphs:
  [0 1 48 19 14 1 12 16 0 nil]
  [0 1 824 741 0 -17 -1 18 1 nil]

I presume that the '0 0 13 17 1 [-13 0 0]' vs '0 -17 -1 18 1 nil'
represents the failed overlay.

and that ftcrhb expands to freetype-cairo-harfbuzz.

it is a pain to dig deep w/o the left hand, but i can try w/ some
leading suggestions.  i haven't looked at the src since before my last
stroke, much less since this one.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 0x997A9F17ED7DAEA6

bug#39554: 27.0.50; cairo not composing sequences

[Robert Pluim]

>>>>> On Thu, 13 Feb 2020 23:50:51 -0500, James Cloos <cloos@jhcloos.com> said:

>>>>> "YM" == YAMAMOTO Mitsuharu <mituharu@math.s.chiba-u.ac.jp> writes:
    YM> I guess the difference comes from with vs. without HarfBuzz rather
    YM> than cairo vs. xft.

    James> makes sense.

    YM> Could you check if that is the case on your environment?

    James> all i know is that w/o cairo, describe-char reports:

    James>     xft:-unknown-DejaVu Sans Mono-normal-normal-normal-*-22-*-*-*-m-0-iso10646-1
    James> by these glyphs:
    James>   [0 1 48 19 13 0 13 17 1 nil]
    James>   [0 1 824 704 0 0 13 17 1 [-13 0 0]]

Thatʼs XFT not using harfbuzz

    James> and w/ cairo it reports:

    James>     ftcrhb:-unknown-DejaVu Serif-normal-normal-normal-*-22-*-*-*-*-0-iso10646-1
    James> by these glyphs:
    James>   [0 1 48 19 14 1 12 16 0 nil]
    James>   [0 1 824 741 0 -17 -1 18 1 nil]

And this is Cairo using harfbuzz.

    James> I presume that the '0 0 13 17 1 [-13 0 0]' vs '0 -17 -1 18 1 nil'
    James> represents the failed overlay.

    James> and that ftcrhb expands to freetype-cairo-harfbuzz.

    James> it is a pain to dig deep w/o the left hand, but i can try w/ some
    James> leading suggestions.  i haven't looked at the src since before my last
    James> stroke, much less since this one.

I donʼt see the issue with DejaVu Sans Mono in an Xft build, and
can reproduce it in an Xft+Harfbuzz build, so Cairo is not to blame
here.

Robert

bug#39554: 27.0.50; cairo not composing sequences

[Lars Ingebrigtsen]

Robert Pluim <rpluim@gmail.com> writes:

> I donʼt see the issue with DejaVu Sans Mono in an Xft build, and
> can reproduce it in an Xft+Harfbuzz build, so Cairo is not to blame
> here.

(I'm going through old bug reports that unfortunately weren't resolved
at the time.)

This looks very similar to the problem in bug#44784, and the issue there
was that Emacs was choosing a font for ̷ that's not the same as the font
used for 0 -- in that case Emacs doesn't combine chars.  So it's a
matter of choosing a font that has wider coverage, I think.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#39554: 27.0.50; cairo not composing sequences

[James Cloos]

>>>>> "RP" == Robert Pluim <rpluim@gmail.com> writes:
>>>>> "LI" == Lars Ingebrigtsen <larsi@gnus.org> writes:

RP>> I donʼt see the issue with DejaVu Sans Mono in an Xft build, and can
RP>> reproduce it in an Xft+Harfbuzz build, so Cairo is not to blame here.

LI> in that case Emacs doesn't combine chars.  So it's a
LI> matter of choosing a font that has wider coverage, I think.

no.

in this case one font is used.

w/ m17lib+libotf the two glyphs are combined and displayed one atop the other
(ie, z-axis stacking).

w/ harfbuzz the two glyphs still are combined but displayed next to each other.
(ie, x-axis stacking).

the first way is the correct way.  the latter is a bug.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 0x997A9F17ED7DAEA6

bug#39554: 27.0.50; cairo not composing sequences

[James Cloos]

Oh, i forgot to add:

the Noto fonts (at least Serif and Sans Mono)  also screws up 0̸.

but differently than dejavu sans mono does.

in noto there is a (very) tiny bit of overlap.  It looks like a percent
w/o the lower-right circle.  and the combining slash overlaps the
subsequent glyph.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 0x997A9F17ED7DAEA6

bug#39554: 27.0.50; cairo not composing sequences

[Eli Zaretskii]

> From: James Cloos <cloos@jhcloos.com>
> Date: Fri, 11 Feb 2022 15:40:19 -0500
> Cc: Robert Pluim <rpluim@gmail.com>, 39554@debbugs.gnu.org
> 
> in this case one font is used.
> 
> w/ m17lib+libotf the two glyphs are combined and displayed one atop the other
> (ie, z-axis stacking).
> 
> w/ harfbuzz the two glyphs still are combined but displayed next to each other.
> (ie, x-axis stacking).
> 
> the first way is the correct way.  the latter is a bug.

_If_ it's a bug, it's either in the font or in HarfBuzz.  Not in Emacs.