bug#57066: 29.0.50; Complex emoji overlaps with text

bug#57066: 29.0.50; Complex emoji overlaps with text

[Protesilaos Stavrou]

[-- Attachment #1: Type: text/plain, Size: 1862 bytes --]

Dear maintainers,

I just noticed a bug with some complex emoji (grapheme clusters?) where
the emoji does not have clear character boundaries but instead overlaps
with text adjacent to it.

Steps to reproduce on 'emacs -Q':

* In the scratch buffer type "Hello "
* C-x 8 e s
* Search for "sun" and type RET
* Now you have: Hello ☀️
* C-e
* Type something else, like "test".
* Notice how "test" is written over the emoji (see attached screenshots).

Please let me know if you need me to test anything else on my side or
provide additional information.

All the best,
Protesilaos (or simply "Prot")

* * *

In GNU Emacs 29.0.50 (build 3, x86_64-pc-linux-gnu, GTK+ Version 3.24.34, cairo version 1.17.6)
 of 2022-08-09 built on kronos
Repository revision: 97a54d132732131fb7635d91a45cf2088e98b60b
Repository branch: makepkg
System Description: Arch Linux

Configured using:
 'configure --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib
 --localstatedir=/var --mandir=/usr/share/man --with-gameuser=:games
 --without-libotf --without-m17n-flt --without-gconf
 --with-native-compilation --with-pgtk --with-sound=no --without-gpm
 --without-compress-install
 '--program-transform-name=s/\([ec]tags\)/\1.emacs/'
 'CFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions
 -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security
 -fstack-clash-protection -fcf-protection'
 LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG JSON
LCMS2 LIBSYSTEMD LIBXML2 MODULES NATIVE_COMP NOTIFY INOTIFY PDUMPER PGTK
PNG RSVG SECCOMP SQLITE3 THREADS TIFF TOOLKIT_SCROLL_BARS WEBP XIM GTK3
ZLIB

Important settings:
  value of $LANG: C
  locale-coding-system: nil

-- 
Protesilaos Stavrou
https://protesilaos.com

[-- Attachment #2: Screenshot from 2022-08-09 12-07-53.png --]
[-- Type: image/png, Size: 35219 bytes --]

[-- Attachment #3: Screenshot from 2022-08-09 12-08-20.png --]
[-- Type: image/png, Size: 35676 bytes --]

bug#57066: 29.0.50; Complex emoji overlaps with text

[Eli Zaretskii]

> From: Protesilaos Stavrou <info@protesilaos.com>
> Date: Tue, 09 Aug 2022 12:17:40 +0300
> 
> * In the scratch buffer type "Hello "
> * C-x 8 e s
> * Search for "sun" and type RET
> * Now you have: Hello ☀️
> * C-e
> * Type something else, like "test".
> * Notice how "test" is written over the emoji (see attached screenshots).

I cannot reproduce this.

Please go to the Emoji, type "C-u C-x =" and post here everything that
Emacs shows in the *Help* buffer it pops up.

bug#57066: 29.0.50; Complex emoji overlaps with text

[Fernando de Morais]

[-- Attachment #1: Type: text/plain, Size: 643 bytes --]

Hello,

I've been experiencing the same issue on Emacs 28.1[1] since upgrading
HarfBuzz to 5.1.0 (with the previous version, 5.0.1, composed sequences
were not displayed properly, as reported in bug#56789[2]).

I couldn't reproduce exactly the same steps listed, because the command
C-x 8 e s is not available, but follows attached a screenshot to
illustrate the same problem faced by Protesilaos.

Thanks.

Footnotes:
[1] GNU Emacs 28.1 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.33,
     cairo version 1.17.6) of 2022-04-27

[2]  https://mail.gnu.org/archive/html/bug-gnu-emacs/2022-07/msg02241.html

-- 
Regards,
Fernando de Morais.


[-- Attachment #2: Print. --]
[-- Type: image/png, Size: 268323 bytes --]

bug#57066: 29.0.50; Complex emoji overlaps with text

[Protesilaos Stavrou]

[-- Attachment #1: Type: text/plain, Size: 1163 bytes --]

> From: Eli Zaretskii <eliz@gnu.org>
> Date: Tue, 09 Aug 2022 14:14:54 +0300
>
>> From: Protesilaos Stavrou <info@protesilaos.com>
>> Date: Tue, 09 Aug 2022 12:17:40 +0300
>> 
>> * In the scratch buffer type "Hello "
>> * C-x 8 e s
>> * Search for "sun" and type RET
>> * Now you have: Hello ☀️
>> * C-e
>> * Type something else, like "test".
>> * Notice how "test" is written over the emoji (see attached screenshots).
>
> I cannot reproduce this.
>
> Please go to the Emoji, type "C-u C-x =" and post here everything that
> Emacs shows in the *Help* buffer it pops up.

Right after inserting the "sun" emoji, the 'describe-char' does not
work.  The echo area shows:

    point=154 of 153 (EOB) column=7

Moving the point one character back and retrying returns:

    cl--assertion-failed: Assertion failed: (not (multibyte-string-p str))

Apparently this happens with all emoji.  Even when I insert the first
smiling emoji with the C-x 8 e e s s a sequence I cannot get a Help
buffer out of 'describe-char'.  The point is on the emoji as shown in
the attached screenshot.

-- 
Protesilaos Stavrou
https://protesilaos.com

[-- Attachment #2: Screenshot from 2022-08-09 14-26-59.png --]
[-- Type: image/png, Size: 51192 bytes --]

bug#57066: 29.0.50; Complex emoji overlaps with text

[Eli Zaretskii]

> Cc: Protesilaos Stavrou <info@protesilaos.com>
> From: Fernando de Morais <fernandodemorais.jf@gmail.com>
> Date: Tue, 09 Aug 2022 08:23:32 -0300
> 
> I've been experiencing the same issue on Emacs 28.1[1] since upgrading
> HarfBuzz to 5.1.0 (with the previous version, 5.0.1, composed sequences
> were not displayed properly, as reported in bug#56789[2]).

If this is related to upgrading HarfBuzz, it sounds like a HarfBuzz
problem, not an Emacs problem.  Did you try reporting this to the
HarfBuzz folks?

bug#57066: 29.0.50; Complex emoji overlaps with text

[Eli Zaretskii]

> From: Protesilaos Stavrou <info@protesilaos.com>
> Cc: 57066@debbugs.gnu.org
> Date: Tue, 09 Aug 2022 14:28:52 +0300
> 
> > Please go to the Emoji, type "C-u C-x =" and post here everything that
> > Emacs shows in the *Help* buffer it pops up.
> 
> Right after inserting the "sun" emoji, the 'describe-char' does not
> work.  The echo area shows:
> 
>     point=154 of 153 (EOB) column=7

Are you saying that what describe-char does at EOB means it doesn't
work?  The above is exactly the expected result at EOB.

> Moving the point one character back and retrying returns:
> 
>     cl--assertion-failed: Assertion failed: (not (multibyte-string-p str))

I cannot reproduce that, either.  Could you run this under Edebug and
show the value of 'str' in the above assertion?

Someone else reported that this started happening after upgrading
HarfBuzz to v5.1.0 -- is that the version you are using?

bug#57066: 29.0.50; Complex emoji overlaps with text

[Fernando de Morais]

Eli Zaretskii <eliz@gnu.org> writes:

> If this is related to upgrading HarfBuzz, it sounds like a HarfBuzz
> problem, not an Emacs problem.  Did you try reporting this to the
> HarfBuzz folks?

Hello Eli,

Not yet, but I will and will let you know in case I get any answers
there.

Thanks.

-- 
Regards,
Fernando de Morais.

bug#57066: 29.0.50; Complex emoji overlaps with text

[Protesilaos Stavrou]

> From: Eli Zaretskii <eliz@gnu.org>
> Date: Tue, 09 Aug 2022 15:01:44 +0300
>
>> From: Protesilaos Stavrou <info@protesilaos.com>
>> Cc: 57066@debbugs.gnu.org
>> Date: Tue, 09 Aug 2022 14:28:52 +0300
>> 
>> > Please go to the Emoji, type "C-u C-x =" and post here everything that
>> > Emacs shows in the *Help* buffer it pops up.
>> 
>> Right after inserting the "sun" emoji, the 'describe-char' does not
>> work.  The echo area shows:
>> 
>>     point=154 of 153 (EOB) column=7
>
> Are you saying that what describe-char does at EOB means it doesn't
> work?  The above is exactly the expected result at EOB.

Sorry, I meant that I could not produce a Help buffer.

>> Moving the point one character back and retrying returns:
>> 
>>     cl--assertion-failed: Assertion failed: (not (multibyte-string-p str))
>
> I cannot reproduce that, either.  Could you run this under Edebug and
> show the value of 'str' in the above assertion?

I am not sure how to do it with Edebug.  With (setq debug-on-error t) I
get this:

Debugger entered--Lisp error: (cl-assertion-failed ((not (multibyte-string-p str)) nil))
  cl--assertion-failed((not (multibyte-string-p str)))
  encoded-string-description(#("😀" 0 1 (charset unicode)) nil)
  describe-char(146)
  what-cursor-position((4))
  funcall-interactively(what-cursor-position (4))
  call-interactively(what-cursor-position nil nil)
  command-execute(what-cursor-position)

> Someone else reported that this started happening after upgrading
> HarfBuzz to v5.1.0 -- is that the version you are using?

Yes, I am on HarfBuzz v5.1.0 as well.  I will check that thread as well.

-- 
Protesilaos Stavrou
https://protesilaos.com

bug#57066: 29.0.50; Complex emoji overlaps with text

[Eli Zaretskii]

> From: Protesilaos Stavrou <info@protesilaos.com>
> Cc: 57066@debbugs.gnu.org
> Date: Tue, 09 Aug 2022 15:50:10 +0300
> 
> >> Right after inserting the "sun" emoji, the 'describe-char' does not
> >> work.  The echo area shows:
> >> 
> >>     point=154 of 153 (EOB) column=7
> >
> > Are you saying that what describe-char does at EOB means it doesn't
> > work?  The above is exactly the expected result at EOB.
> 
> Sorry, I meant that I could not produce a Help buffer.

That's expected: there's no character at EOB.

> >>     cl--assertion-failed: Assertion failed: (not (multibyte-string-p str))
> >
> > I cannot reproduce that, either.  Could you run this under Edebug and
> > show the value of 'str' in the above assertion?
> 
> I am not sure how to do it with Edebug.

I tried to explain that below.

> With (setq debug-on-error t) I get this:
> 
> Debugger entered--Lisp error: (cl-assertion-failed ((not (multibyte-string-p str)) nil))
>   cl--assertion-failed((not (multibyte-string-p str)))
>   encoded-string-description(#("😀" 0 1 (charset unicode)) nil)
>   describe-char(146)

There are 2 calls to encoded-string-description in describe-char, and
it's important to know which one causes this and why.

To run this under Edebug, do the following

  emacs -Q
  M-x load-library RET descr-text.el RET
  C-x C-f /path/to/lisp/international/descr-text.el RET
  C-u 385 M-g g
  M-x edebug-defun RET

Note that you must load-library with an explicit .el extension.

Line 385 in descr-text.el is where describe-char starts.

Then repeat the recipe, and when you call describe-char, Edebug will
kick in.  Now step through the code and try to figure out which call
to encoded-string-description is the one that barfs.  My guess is that
it's the second one, in which case it is important to know the value
of the variable 'charset' here:

      ("file code"
       ,@(if multibyte-p
	     (let* ((coding buffer-file-coding-system)
		    (encoded (encode-coding-char char coding charset))) <<<<<<
	       (if encoded
		   (list (encoded-string-description encoded coding)
			 (format "(encoded by coding system %S)"
				 coding))

Thanks.

bug#57066: 29.0.50; Complex emoji overlaps with text

[Protesilaos Stavrou]

> From: Eli Zaretskii <eliz@gnu.org>
> Date: Tue, 09 Aug 2022 19:05:06 +0300
>
> There are 2 calls to encoded-string-description in describe-char, and
> it's important to know which one causes this and why.
>
> To run this under Edebug, do the following
>
>   emacs -Q
>   M-x load-library RET descr-text.el RET
>   C-x C-f /path/to/lisp/international/descr-text.el RET
>   C-u 385 M-g g
>   M-x edebug-defun RET

Thank you!  I followed the instructions.  Though note that descr-text.el
was in lisp/descr-text.el without the "international" part.

> Note that you must load-library with an explicit .el extension.
>
> Line 385 in descr-text.el is where describe-char starts.
>
> Then repeat the recipe, and when you call describe-char, Edebug will
> kick in.  Now step through the code and try to figure out which call
> to encoded-string-description is the one that barfs.  My guess is that
> it's the second one, in which case it is important to know the value
> of the variable 'charset' here:
>
>       ("file code"
>        ,@(if multibyte-p
> 	     (let* ((coding buffer-file-coding-system)
> 		    (encoded (encode-coding-char char coding charset))) <<<<<<
> 	       (if encoded
> 		   (list (encoded-string-description encoded coding)
> 			 (format "(encoded by coding system %S)"
> 				 coding))

It is as you surmised about the seond call.  The charset itself gives:

    Result: unicode

While the form (encode-coding-char char coding charset) returns:

    Result: #("😀" 0 1 (charset unicode))

Then the error is in (encoded-string-description encoded coding):

    Assertion failed: ((not (multibyte-string-p str)) nil)

The reason is that (coding buffer-file-coding-system) returns nil.

-- 
Protesilaos Stavrou
https://protesilaos.com

bug#57066: 29.0.50; Complex emoji overlaps with text

[Eli Zaretskii]

> From: Protesilaos Stavrou <info@protesilaos.com>
> Cc: 57066@debbugs.gnu.org
> Date: Tue, 09 Aug 2022 20:17:12 +0300
> 
> While the form (encode-coding-char char coding charset) returns:
> 
>     Result: #("😀" 0 1 (charset unicode))
> 
> Then the error is in (encoded-string-description encoded coding):
> 
>     Assertion failed: ((not (multibyte-string-p str)) nil)
> 
> The reason is that (coding buffer-file-coding-system) returns nil.

Why is buffer-file-coding-system nil?  Isn't that in *scratch*?

bug#57066: 29.0.50; Complex emoji overlaps with text

[Protesilaos Stavrou]

> From: Eli Zaretskii <eliz@gnu.org>
> Date: Tue, 09 Aug 2022 20:31:58 +0300
>
>> The reason is that (coding buffer-file-coding-system) returns nil.
>
> Why is buffer-file-coding-system nil?  Isn't that in *scratch*?

Yes, it is *scratch*.

- emacs -Q
- Insert buffer-file-coding-system
- C-j
- Returns 'nil'

-- 
Protesilaos Stavrou
https://protesilaos.com

bug#57066: 29.0.50; Complex emoji overlaps with text

[Eli Zaretskii]

> From: Protesilaos Stavrou <info@protesilaos.com>
> Cc: 57066@debbugs.gnu.org
> Date: Tue, 09 Aug 2022 20:41:42 +0300
> 
> > Why is buffer-file-coding-system nil?  Isn't that in *scratch*?
> 
> Yes, it is *scratch*.
> 
> - emacs -Q
> - Insert buffer-file-coding-system
> - C-j
> - Returns 'nil'

<Shrug> strange.  What is the default value?  That is

   M-: (default-value 'buffer-file-coding-system) RET

bug#57066: 29.0.50; Complex emoji overlaps with text

[Lars Ingebrigtsen]

Fernando de Morais <fernandodemorais.jf@gmail.com> writes:

> I've been experiencing the same issue on Emacs 28.1[1] since upgrading
> HarfBuzz to 5.1.0 (with the previous version, 5.0.1, composed sequences
> were not displayed properly, as reported in bug#56789[2]).

As an aside, since Emacs is relying so heavily on HarfBuzz these days,
and since there's been a couple of these bugs surfacing, it might be
nice if somebody volunteers to build development versions of HarfBuzz
(and use that with Emacs) so that bugs like this can be reported to the
HarfBuzz people before they make releases.

(I'm not volunteering myself.  😀)

bug#57066: 29.0.50; Complex emoji overlaps with text

[Protesilaos Stavrou]

> From: Eli Zaretskii <eliz@gnu.org>
> Date: Tue, 09 Aug 2022 20:54:16 +0300
>
>> From: Protesilaos Stavrou <info@protesilaos.com>
>> Cc: 57066@debbugs.gnu.org
>> Date: Tue, 09 Aug 2022 20:41:42 +0300
>> 
>> > Why is buffer-file-coding-system nil?  Isn't that in *scratch*?
>> 
>> Yes, it is *scratch*.
>> 
>> - emacs -Q
>> - Insert buffer-file-coding-system
>> - C-j
>> - Returns 'nil'
>
> <Shrug> strange.  What is the default value?  That is
>
>    M-: (default-value 'buffer-file-coding-system) RET

It is nil.

-- 
Protesilaos Stavrou
https://protesilaos.com

bug#57066: 29.0.50; Complex emoji overlaps with text

[Protesilaos Stavrou]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Tue, 09 Aug 2022 20:28:33 +0200
>
> Fernando de Morais <fernandodemorais.jf@gmail.com> writes:
>
>> I've been experiencing the same issue on Emacs 28.1[1] since upgrading
>> HarfBuzz to 5.1.0 (with the previous version, 5.0.1, composed sequences
>> were not displayed properly, as reported in bug#56789[2]).
>
> As an aside, since Emacs is relying so heavily on HarfBuzz these days,
> and since there's been a couple of these bugs surfacing, it might be
> nice if somebody volunteers to build development versions of HarfBuzz
> (and use that with Emacs) so that bugs like this can be reported to the
> HarfBuzz people before they make releases.
>
> (I'm not volunteering myself.  😀)

I will give it a try later today.  Assuming it all goes well, I will
report back to you.

-- 
Protesilaos Stavrou
https://protesilaos.com

bug#57066: 29.0.50; Complex emoji overlaps with text

[Eli Zaretskii]

> From: Protesilaos Stavrou <info@protesilaos.com>
> Cc: 57066@debbugs.gnu.org
> Date: Wed, 10 Aug 2022 06:15:15 +0300
> 
> > From: Eli Zaretskii <eliz@gnu.org>
> > Date: Tue, 09 Aug 2022 20:54:16 +0300
> >
> >> From: Protesilaos Stavrou <info@protesilaos.com>
> >> Cc: 57066@debbugs.gnu.org
> >> Date: Tue, 09 Aug 2022 20:41:42 +0300
> >> 
> >> > Why is buffer-file-coding-system nil?  Isn't that in *scratch*?
> >> 
> >> Yes, it is *scratch*.
> >> 
> >> - emacs -Q
> >> - Insert buffer-file-coding-system
> >> - C-j
> >> - Returns 'nil'
> >
> > <Shrug> strange.  What is the default value?  That is
> >
> >    M-: (default-value 'buffer-file-coding-system) RET
> 
> It is nil.

But of course.  Should be fixed now, thanks.

bug#57066: 29.0.50; Complex emoji overlaps with text

[Protesilaos Stavrou]

[-- Attachment #1: Type: text/plain, Size: 1760 bytes --]

> From: Eli Zaretskii <eliz@gnu.org>
> Date: Wed, 10 Aug 2022 15:33:20 +0300

>> > <Shrug> strange.  What is the default value?  That is
>> >
>> >    M-: (default-value 'buffer-file-coding-system) RET
>> 
>> It is nil.
>
> But of course.  Should be fixed now, thanks.

Thank you!  The C-u C-x = works now.

Back to my original recipe then:

- C-x 8 e s
- Type "sun" and RET
- C-b
- C-u C-x =

This gives me the following Help buffer:

                 position: 152 of 153 (99%), column: 6
                character: ☀ (displayed as ☀) (codepoint 9728, #o23000, #x2600)
                  charset: unicode (Unicode (ISO10646))
    code point in charset: 0x2600
                   script: symbol
                   syntax: w 	which means: word
                 category: .:Base, j:Japanese
                 to input: type "C-x 8 RET 2600" or "C-x 8 RET BLACK SUN WITH RAYS"
              buffer code: #xE2 #x98 #x80
                file code: not encodable by coding system nil
                  display: composed to form "☀️" (see below)

    Composed with the following character(s) "️" using this font:
      ftcrhb:-GOOG-Noto Color Emoji-regular-normal-normal-*-13-*-*-*-m-0-iso10646-1
    by these glyphs:
      [0 1 9728 59 16 0 17 13 4 [0 0 2]]
    with these character(s):
      ️ (#xfe0f) VARIATION SELECTOR-16

    Character code properties: customize what to show
      name: BLACK SUN WITH RAYS
      general-category: So (Symbol, Other)
      decomposition: (9728) ('☀')

    There are text properties here:
      fontified            t


I also attach a screenshot where the coloured sun overlaps with a double
quotes character.

-- 
Protesilaos Stavrou
https://protesilaos.com

[-- Attachment #2: Screenshot from 2022-08-11 07-01-44.png --]
[-- Type: image/png, Size: 127140 bytes --]

bug#57066: 29.0.50; Complex emoji overlaps with text

[YAMAMOTO Mitsuharu]

On Thu, 11 Aug 2022 13:07:03 +0900,
Protesilaos Stavrou wrote:
> 
> > From: Eli Zaretskii <eliz@gnu.org>
> > Date: Wed, 10 Aug 2022 15:33:20 +0300
> 
> >> > <Shrug> strange.  What is the default value?  That is
> >> >
> >> >    M-: (default-value 'buffer-file-coding-system) RET
> >> 
> >> It is nil.
> >
> > But of course.  Should be fixed now, thanks.
> 
> Thank you!  The C-u C-x = works now.
> 
> Back to my original recipe then:
> 
> - C-x 8 e s
> - Type "sun" and RET
> - C-b
> - C-u C-x =
> 
> This gives me the following Help buffer:
> 
>                  position: 152 of 153 (99%), column: 6
>                 character: ☀ (displayed as ☀) (codepoint 9728, #o23000, #x2600)
>                   charset: unicode (Unicode (ISO10646))
>     code point in charset: 0x2600
>                    script: symbol
>                    syntax: w 	which means: word
>                  category: .:Base, j:Japanese
>                  to input: type "C-x 8 RET 2600" or "C-x 8 RET BLACK SUN WITH RAYS"
>               buffer code: #xE2 #x98 #x80
>                 file code: not encodable by coding system nil
>                   display: composed to form "☀️" (see below)
> 
>     Composed with the following character(s) "️" using this font:
>       ftcrhb:-GOOG-Noto Color Emoji-regular-normal-normal-*-13-*-*-*-m-0-iso10646-1
>     by these glyphs:
>       [0 1 9728 59 16 0 17 13 4 [0 0 2]]
>     with these character(s):
>       ️ (#xfe0f) VARIATION SELECTOR-16
> 
>     Character code properties: customize what to show
>       name: BLACK SUN WITH RAYS
>       general-category: So (Symbol, Other)
>       decomposition: (9728) ('☀')
> 
>     There are text properties here:
>       fontified            t
> 
> 
> I also attach a screenshot where the coloured sun overlaps with a double
> quotes character.
> 
> -- 
> Protesilaos Stavrou
> https://protesilaos.com

It seems that position unit adjustment for bitmap fonts is no longer
necessary on HarfBuzz 5:

  https://github.com/harfbuzz/harfbuzz/issues/489

Could you test if the following patch works?

				     YAMAMOTO Mitsuharu
				mituharu@math.s.chiba-u.ac.jp

diff --git a/src/ftcrfont.c b/src/ftcrfont.c
index 39ea068556..42610ccd9d 100644
--- a/src/ftcrfont.c
+++ b/src/ftcrfont.c
@@ -273,7 +273,8 @@ ftcrfont_open (struct frame *f, Lisp_Object entity, int pixel_size)
 #ifdef HAVE_HARFBUZZ
   ftcrfont_info->hb_font = NULL;
 #endif	/* HAVE_HARFBUZZ */
-  if (ft_face->units_per_EM)
+  if (hb_version_atleast (5, 0, 0)
+      || ft_face->units_per_EM)
     ftcrfont_info->bitmap_position_unit = 0;
   else
     ftcrfont_info->bitmap_position_unit = (extents.height
diff --git a/src/ftfont.h b/src/ftfont.h
index f771dc159b..5248aef649 100644
--- a/src/ftfont.h
+++ b/src/ftfont.h
@@ -66,7 +66,9 @@ #define EMACS_FTFONT_H
   cairo_scaled_font_t *cr_scaled_font;
   /* Scale factor from the bitmap strike metrics in 1/64 pixels, used
      as the hb_position_t value in HarfBuzz, to those in (scaled)
-     pixels.  The value is 0 for scalable fonts.  */
+     pixels.  The value is 0 for scalable fonts.  This is only
+     necessary for HarfBuzz prior to version 5, and the value is
+     always 0 on later versions.  */
   double bitmap_position_unit;
   /* Font metrics cache.  */
   struct font_metrics **metrics;

bug#57066: 29.0.50; Complex emoji overlaps with text

[YAMAMOTO Mitsuharu]

On Thu, 11 Aug 2022 15:01:31 +0900,
YAMAMOTO Mitsuharu wrote:
> 
> It seems that position unit adjustment for bitmap fonts is no longer
> necessary on HarfBuzz 5:
> 
>   https://github.com/harfbuzz/harfbuzz/issues/489
> 
> Could you test if the following patch works?
 
Oops, it breaks non-HarfBuzz builds.  Pleae try below instead.

				     YAMAMOTO Mitsuharu
				mituharu@math.s.chiba-u.ac.jp

diff --git a/src/ftcrfont.c b/src/ftcrfont.c
index 39ea068556..d5d3e440af 100644
--- a/src/ftcrfont.c
+++ b/src/ftcrfont.c
@@ -568,7 +568,11 @@ ftcrhbfont_begin_hb_font (struct font *font, double *position_unit)
 
   ftcrfont_info->ft_size = ft_face->size;
   hb_font_t *hb_font = fthbfont_begin_hb_font (font, position_unit);
-  if (ftcrfont_info->bitmap_position_unit)
+  /* HarfBuzz 5 correctly scales bitmap-only fonts without position
+     unit adjustment.
+     (https://github.com/harfbuzz/harfbuzz/issues/489) */
+  if (!hb_version_atleast (5, 0, 0)
+      && ftcrfont_info->bitmap_position_unit)
     *position_unit = ftcrfont_info->bitmap_position_unit;
 
   return hb_font;

bug#57066: 29.0.50; Complex emoji overlaps with text

[Protesilaos Stavrou]

> From: YAMAMOTO Mitsuharu <mituharu@math.s.chiba-u.ac.jp>
> Date: Thu, 11 Aug 2022 15:24:38 +0900
>
> Oops, it breaks non-HarfBuzz builds.  Pleae try below instead.
>
> 				     YAMAMOTO Mitsuharu
> 				mituharu@math.s.chiba-u.ac.jp
>
> diff --git a/src/ftcrfont.c b/src/ftcrfont.c
> index 39ea068556..d5d3e440af 100644
> --- a/src/ftcrfont.c
> +++ b/src/ftcrfont.c
> @@ -568,7 +568,11 @@ ftcrhbfont_begin_hb_font (struct font *font, double *position_unit)
>  
>    ftcrfont_info->ft_size = ft_face->size;
>    hb_font_t *hb_font = fthbfont_begin_hb_font (font, position_unit);
> -  if (ftcrfont_info->bitmap_position_unit)
> +  /* HarfBuzz 5 correctly scales bitmap-only fonts without position
> +     unit adjustment.
> +     (https://github.com/harfbuzz/harfbuzz/issues/489) */
> +  if (!hb_version_atleast (5, 0, 0)
> +      && ftcrfont_info->bitmap_position_unit)
>      *position_unit = ftcrfont_info->bitmap_position_unit;
>  
>    return hb_font;

Thank you!  I just tried it and it works on my end.

I only tested it with a regular 'make'.  Do you want me to use other
build options as well?

-- 
Protesilaos Stavrou
https://protesilaos.com

bug#57066: 29.0.50; Complex emoji overlaps with text

[Eli Zaretskii]

> Date: Thu, 11 Aug 2022 15:24:38 +0900
> From: YAMAMOTO Mitsuharu <mituharu@math.s.chiba-u.ac.jp>
> Cc: Eli Zaretskii <eliz@gnu.org>,
> 	57066@debbugs.gnu.org
> 
> On Thu, 11 Aug 2022 15:01:31 +0900,
> YAMAMOTO Mitsuharu wrote:
> > 
> > It seems that position unit adjustment for bitmap fonts is no longer
> > necessary on HarfBuzz 5:
> > 
> >   https://github.com/harfbuzz/harfbuzz/issues/489
> > 
> > Could you test if the following patch works?
>  
> Oops, it breaks non-HarfBuzz builds.  Pleae try below instead.

Thanks!

Do other font backends that use HarfBuzz need something similar, or is
this specific to the ftcrhb backend?

bug#57066: 29.0.50; Complex emoji overlaps with text

[YAMAMOTO Mitsuharu]

On Thu, 11 Aug 2022 15:49:10 +0900,
Eli Zaretskii wrote:
> 
> Do other font backends that use HarfBuzz need something similar, or is
> this specific to the ftcrhb backend?

The xfthb font backend driver does not need the similar change since
it always uses 1.0 / (1 << 6) as the position unit, just like the
patched ftcrhb code for HarfBuzz 5 and later.

I'm not familiar with the harfbuzz driver on W32.  But at least, the
bug#489 on HarfBuzz is about hb-ft.cc.  So probably the
uniscribe-specific part is not affected.

				     YAMAMOTO Mitsuharu
				mituharu@math.s.chiba-u.ac.jp

bug#57066: 29.0.50; Complex emoji overlaps with text

[Eli Zaretskii]

> Date: Thu, 11 Aug 2022 16:23:47 +0900
> From: YAMAMOTO Mitsuharu <mituharu@math.s.chiba-u.ac.jp>
> Cc: 	info@protesilaos.com,
> 	57066@debbugs.gnu.org
> 
> On Thu, 11 Aug 2022 15:49:10 +0900,
> Eli Zaretskii wrote:
> > 
> > Do other font backends that use HarfBuzz need something similar, or is
> > this specific to the ftcrhb backend?
> 
> The xfthb font backend driver does not need the similar change since
> it always uses 1.0 / (1 << 6) as the position unit, just like the
> patched ftcrhb code for HarfBuzz 5 and later.
> 
> I'm not familiar with the harfbuzz driver on W32.  But at least, the
> bug#489 on HarfBuzz is about hb-ft.cc.  So probably the
> uniscribe-specific part is not affected.

OK, thanks.

I think the fix should be installed on the emacs-28 release branch.
It seems to be quite safe.  Do you agree?

bug#57066: 29.0.50; Complex emoji overlaps with text

[YAMAMOTO Mitsuharu]

On Thu, 11 Aug 2022 16:43:53 +0900,
Eli Zaretskii wrote:
> 
> > Date: Thu, 11 Aug 2022 16:23:47 +0900
> > From: YAMAMOTO Mitsuharu <mituharu@math.s.chiba-u.ac.jp>
> > Cc: 	info@protesilaos.com,
> > 	57066@debbugs.gnu.org
> > 
> > On Thu, 11 Aug 2022 15:49:10 +0900,
> > Eli Zaretskii wrote:
> > > 
> > > Do other font backends that use HarfBuzz need something similar, or is
> > > this specific to the ftcrhb backend?
> > 
> > The xfthb font backend driver does not need the similar change since
> > it always uses 1.0 / (1 << 6) as the position unit, just like the
> > patched ftcrhb code for HarfBuzz 5 and later.
> > 
> > I'm not familiar with the harfbuzz driver on W32.  But at least, the
> > bug#489 on HarfBuzz is about hb-ft.cc.  So probably the
> > uniscribe-specific part is not affected.
> 
> OK, thanks.
> 
> I think the fix should be installed on the emacs-28 release branch.
> It seems to be quite safe.  Do you agree?

Yes.  The function hb_version_atleast was introduced in HarfBuzz
version 0.9.30, and Emacs already requires 0.9.42 on non-W32 systems.

				     YAMAMOTO Mitsuharu
				mituharu@math.s.chiba-u.ac.jp

bug#57066: 29.0.50; Complex emoji overlaps with text

[Eli Zaretskii]

> Date: Thu, 11 Aug 2022 17:57:53 +0900
> From: YAMAMOTO Mitsuharu <mituharu@math.s.chiba-u.ac.jp>
> Cc: info@protesilaos.com,
> 	57066@debbugs.gnu.org
> 
> On Thu, 11 Aug 2022 16:43:53 +0900,
> Eli Zaretskii wrote:
> > 
> > > Date: Thu, 11 Aug 2022 16:23:47 +0900
> > > From: YAMAMOTO Mitsuharu <mituharu@math.s.chiba-u.ac.jp>
> > > Cc: 	info@protesilaos.com,
> > > 	57066@debbugs.gnu.org
> > > 
> > > On Thu, 11 Aug 2022 15:49:10 +0900,
> > > Eli Zaretskii wrote:
> > > > 
> > > > Do other font backends that use HarfBuzz need something similar, or is
> > > > this specific to the ftcrhb backend?
> > > 
> > > The xfthb font backend driver does not need the similar change since
> > > it always uses 1.0 / (1 << 6) as the position unit, just like the
> > > patched ftcrhb code for HarfBuzz 5 and later.
> > > 
> > > I'm not familiar with the harfbuzz driver on W32.  But at least, the
> > > bug#489 on HarfBuzz is about hb-ft.cc.  So probably the
> > > uniscribe-specific part is not affected.
> > 
> > OK, thanks.
> > 
> > I think the fix should be installed on the emacs-28 release branch.
> > It seems to be quite safe.  Do you agree?
> 
> Yes.  The function hb_version_atleast was introduced in HarfBuzz
> version 0.9.30, and Emacs already requires 0.9.42 on non-W32 systems.

Great, thanks.  So please install this on the release branch.

bug#57066: 29.0.50; Complex emoji overlaps with text

[YAMAMOTO Mitsuharu]

On Thu, 11 Aug 2022 18:35:51 +0900,
Eli Zaretskii wrote:
> 
> > > 
> > > I think the fix should be installed on the emacs-28 release branch.
> > > It seems to be quite safe.  Do you agree?
> > 
> > Yes.  The function hb_version_atleast was introduced in HarfBuzz
> > version 0.9.30, and Emacs already requires 0.9.42 on non-W32 systems.
> 
> Great, thanks.  So please install this on the release branch.

Done.  Closing.

				     YAMAMOTO Mitsuharu
				mituharu@math.s.chiba-u.ac.jp