bug#56695: A new and old font problem with Arabic

bug#56695: A new and old font problem with Arabic

[fr_ml]

[-- Attachment #1.1: Type: text/plain, Size: 864 bytes --]

 
Hello,
Since Emacs 28 I have a new problem with Arabic fonts.
1. I can't change the Arabic characters to the font of DejaVu Sans
Mono (or others). This wasn't a problem before.
2. The old one: As you can see the glyphs aren't correct, the '-'
after س is not correctly placed. It's in the place of the glyph of
س. Sometimes I have the opposite problem: gaps between the
characters. It depends usually of 'text-scale-adjust', if I change
the scale the problem can disappear.

(Of course I started Emacs with the '-Q' option.)
Attachment: To see the difference: a screenshot in Word/LibreOffice Writer:
In LibreOffice:
1. The font is correct (also for the Arabic characters)
2. The '-' is at the correct place
Can Harfbuzz be a problem? In the older emacs version (maybe 3 or 4 years 
ago?) I never had these problems.

Kind regards
Frank


[-- Attachment #1.2: Type: text/html, Size: 2865 bytes --]

[-- Attachment #2: with_emacs.png --]
[-- Type: image/png, Size: 76771 bytes --]

[-- Attachment #3: with_writer.png --]
[-- Type: image/png, Size: 26747 bytes --]

bug#56695: A new and old font problem with Arabic

[Eli Zaretskii]

> Date: Fri, 22 Jul 2022 10:11:16 +0200 (CEST)
> From: "fr_ml@t-online.de" <fr_ml@t-online.de>
> 
> 1. I can't change the Arabic characters to the font of DejaVu Sans
> Mono (or others). This wasn't a problem before.

What do you mean by "cannot change"?  What happens if you try?  Please
show the exact sequence of commands or Lisp code you used, and any
error messages or incorrect effects you see when you do that.

> 2. The old one: As you can see the glyphs aren't correct, the '-'
> after س is not correctly placed. It's in the place of the glyph of
> س. Sometimes I have the opposite problem: gaps between the
> characters. It depends usually of 'text-scale-adjust', if I change
> the scale the problem can disappear.
> 
> (Of course I started Emacs with the '-Q' option.)
> 
> Attachment: To see the difference: a screenshot in Word/LibreOffice Writer:
> In LibreOffice:
> 1. The font is correct (also for the Arabic characters)
> 2. The '-' is at the correct place
> 
> Can Harfbuzz be a problem? In the older emacs version (maybe 3 or 4 years ago?) I never had these
> problems.

I don't know, because critical information is missing:

  . on what OS is that?
  . what is the version of HarfBuzz you have installed?
  . what is the text that is not displayed correctly? (please provide
    it as plain text that can be copied and examined, not as an image)
  . what does Emacs say when you go to the characters involved in this
    display and type "C-u C-x ="?

In general DejaVu Sans is known as a problematic font for complex text
shaping (which is required for Arabic), so my advice is to use some
other font as the default.

Thanks.

bug#56695: A new and old font problem with Arabic

[Eli Zaretskii]

[Please use Reply All to keep the bug tracker CC'ed.]

> Date: Fri, 22 Jul 2022 15:41:05 +0200 (CEST)
> From: "fr_ml@t-online.de" <fr_ml@t-online.de>
> 
> > 1. I can't change the Arabic characters to the font of DejaVu Sans
> > > Mono (or others). This wasn't a problem before.
> > 
> > What do you mean by "cannot change"?  What happens if you try?  Please
> > show the exact sequence of commands or Lisp code you used, and any
> > error messages or incorrect effects you see when you do that.
> I use the mouse with left click to chose  'Change Buffer Font'.
> I don't get any error message. Just: "Buffer-Face mode enabled".
> The Latin letters get their new font, but the Arabic ones don't.

This is not a bug: if the font you choose cannot support Arabic, Emacs
will fall back to the setup in the current fontset.  In general,
selecting a font only affects ASCII characters; it affects non-ASCII
characters only if the font can display them properly.

> > > Can Harfbuzz be a problem? In the older emacs version (maybe 3 or 4 years ago?) I never had these
> > > problems.
> > 
> > I don't know, because critical information is missing:
> > 
> >   . on what OS is that?
> Linux. I've tried Ubuntu and Arch.
> >   . what is the version of HarfBuzz you have installed?
> harfbuzz 4.4.1-1 
> >   . what is the text that is not displayed correctly? (please provide
> >     it as plain text that can be copied and examined, not as an image)
> Sorry. I forgot:
> (
> a-b-c-d
> 1234567
> س-س-س-س
> 1234567
> )

I don't think HarfBuzz is a problem in your case, but the font
probably is.

> >   . what does Emacs say when you go to the characters involved in this
> >     display and type "C-u C-x ="?
> Before changing the font:
> 
> position: 23 of 32 (69%), column: 6
> character: س‎ (displayed as س‎) (codepoint 1587, #o3063, #x633)
> charset: unicode (Unicode (ISO10646))
> code point in charset: 0x0633
> script: arabic
> syntax: w 	which means: word
> category: .:Base, R:Strong R2L, b:Arabic
> to input: type "C-x 8 RET 633" or "C-x 8 RET ARABIC LETTER SEEN"
> buffer code: #xD8 #xB3
> file code: #xD8 #xB3 (encoded by coding system utf-8-unix)
> display: by this font (glyph code):
> ftcrhb:-1ASC-Droid Serif-normal-normal-normal-*-13-*-*-*-*-0-iso10646-1 (#x0E)

So Emacs uses the Droid Serif font for Arabic, and I guess that's the
problem.

> 
> Character code properties: customize what to show
>   name: ARABIC LETTER SEEN
>   general-category: Lo (Letter, Other)
>   decomposition: (1587) ('س')
> ----
> After changing, or trying to change it:I get exactly the same output.
> 
> > 
> > In general DejaVu Sans is known as a problematic font for complex text
> > shaping (which is required for Arabic), so my advice is to use some
> > other font as the default.
> DejaVu Sans Mono was always ok. And often I need a monospaced font - do you know another one for Arabic?

Try the Amiri font, I'm told it is OK.  But you need to set that up in
the fontset; do not use Amiri as your default font.

bug#56695: AW: bug#56695: A new and old font problem with Arabic

[fr_ml]

[-- Attachment #1: Type: text/plain, Size: 4012 bytes --]

Try the Amiri font, I'm told it is OK.  But you need to set that up in
the fontset; do not use Amiri as your default font.

I've installed Amiri. And it was used as default for Arabic ('emacs -Q'). Is this normal? 

Anyway, I get the same positioning problem of the characters. But as I said I can't reproduce it. If I copy the same text into another frame the characters are displayed correctly.  

[I've added in this email the last 2 image files ('org_plus6_textsize.png' and 'org_plus6_textsize.png').]

About the other problem:
Is it possible to force Emacs to use 'DejaVu Sans Mono' for Arabic? As I said, it was always without any problems in Emacs.
(Amiri isn't a monospace font.)



-----Original-Nachricht-----
Betreff: Re: bug#56695: A new and old font problem with Arabic
Datum: 2022-07-22T16:18:59+0200
Von: "Eli Zaretskii" <eliz@gnu.org>
An: "fr_ml@t-online.de" <fr_ml@t-online.de>

[Please use Reply All to keep the bug tracker CC'ed.]

> Date: Fri, 22 Jul 2022 15:41:05 +0200 (CEST)
> From: "fr_ml@t-online.de" <fr_ml@t-online.de>
> 
> > 1. I can't change the Arabic characters to the font of DejaVu Sans
> > > Mono (or others). This wasn't a problem before.
> > 
> > What do you mean by "cannot change"?  What happens if you try?  Please
> > show the exact sequence of commands or Lisp code you used, and any
> > error messages or incorrect effects you see when you do that.
> I use the mouse with left click to chose  'Change Buffer Font'.
> I don't get any error message. Just: "Buffer-Face mode enabled".
> The Latin letters get their new font, but the Arabic ones don't.

This is not a bug: if the font you choose cannot support Arabic, Emacs
will fall back to the setup in the current fontset.  In general,
selecting a font only affects ASCII characters; it affects non-ASCII
characters only if the font can display them properly.

> > > Can Harfbuzz be a problem? In the older emacs version (maybe 3 or 4 years ago?) I never had these
> > > problems.
> > 
> > I don't know, because critical information is missing:
> > 
> >   . on what OS is that?
> Linux. I've tried Ubuntu and Arch.
> >   . what is the version of HarfBuzz you have installed?
> harfbuzz 4.4.1-1 
> >   . what is the text that is not displayed correctly? (please provide
> >     it as plain text that can be copied and examined, not as an image)
> Sorry. I forgot:
> (
> a-b-c-d
> 1234567
> س-س-س-س
> 1234567
> )

I don't think HarfBuzz is a problem in your case, but the font
probably is.

> >   . what does Emacs say when you go to the characters involved in this
> >     display and type "C-u C-x ="?
> Before changing the font:
> 
> position: 23 of 32 (69%), column: 6
> character: س‎ (displayed as س‎) (codepoint 1587, #o3063, #x633)
> charset: unicode (Unicode (ISO10646))
> code point in charset: 0x0633
> script: arabic
> syntax: w 	which means: word
> category: .:Base, R:Strong R2L, b:Arabic
> to input: type "C-x 8 RET 633" or "C-x 8 RET ARABIC LETTER SEEN"
> buffer code: #xD8 #xB3
> file code: #xD8 #xB3 (encoded by coding system utf-8-unix)
> display: by this font (glyph code):
> ftcrhb:-1ASC-Droid Serif-normal-normal-normal-*-13-*-*-*-*-0-iso10646-1 (#x0E)

So Emacs uses the Droid Serif font for Arabic, and I guess that's the
problem.

> 
> Character code properties: customize what to show
>   name: ARABIC LETTER SEEN
>   general-category: Lo (Letter, Other)
>   decomposition: (1587) ('س')
> ----
> After changing, or trying to change it:I get exactly the same output.
> 
> > 
> > In general DejaVu Sans is known as a problematic font for complex text
> > shaping (which is required for Arabic), so my advice is to use some
> > other font as the default.
> DejaVu Sans Mono was always ok. And often I need a monospaced font - do you know another one for Arabic?

Try the Amiri font, I'm told it is OK.  But you need to set that up in
the fontset; do not use Amiri as your default font.


[-- Attachment #2: org_plus6_textsize.png --]
[-- Type: image/png, Size: 9809 bytes --]

[-- Attachment #3: org_plus5_textsize.png --]
[-- Type: image/png, Size: 7382 bytes --]

bug#56695: A new and old font problem with Arabic

[Eli Zaretskii]

> Date: Fri, 22 Jul 2022 17:58:34 +0200 (CEST)
> From: "fr_ml@t-online.de" <fr_ml@t-online.de>
> Cc: "56695@debbugs.gnu.org" <56695@debbugs.gnu.org>
> 
> Is it possible to force Emacs to use 'DejaVu Sans Mono' for Arabic? As I said, it was always without any problems in Emacs.

Yes, it's possible: you need to use set-fontset-font.

bug#56695: AW: bug#56695: A new and old font problem with Arabic

[fr_ml]

(set-fontset-font "fontset-default" 'arabic "DejaVu Sans Mono") das scheint zu gehen!

I tested it with:
--
(set-fontset-font "fontset-default" 'arabic "DejaVu Sans Mono")
--
in the init file. (I hope this is right way?)

It works, Emacs uses the font correctly.
And it seems that the positioning is then much better. The old problem is still there but not so often.

What I also noticed:
When I change the font with the same function 'set-fontset-font' but without restarting Emacs I get the old positioning problem immediately or after some typing and testing.


bug#56695: A new and old font problem with Arabic

[Eli Zaretskii]

> Date: Sun, 24 Jul 2022 09:24:45 +0200 (CEST)
> From: "fr_ml@t-online.de" <fr_ml@t-online.de>
> Cc: "56695@debbugs.gnu.org" <56695@debbugs.gnu.org>
> 
> (set-fontset-font "fontset-default" 'arabic "DejaVu Sans Mono") das scheint zu gehen!
> 
> I tested it with:
> --
> (set-fontset-font "fontset-default" 'arabic "DejaVu Sans Mono")
> --
> in the init file. (I hope this is right way?)

Yes, this is the right way.

> It works, Emacs uses the font correctly.
> And it seems that the positioning is then much better. The old problem is still there but not so often.

Does this mean we can close this bug?

> What I also noticed:
> When I change the font with the same function 'set-fontset-font' but without restarting Emacs I get the old positioning problem immediately or after some typing and testing.

This is expected: Emacs caches the results of previous text-shaping.
You should invoke

  M-: (clear-composition-cache) RET

to force Emacs to forget the old text-shaping results.

bug#56695: AW: bug#56695: A new and old font problem with Arabic

[fr_ml]

> This is expected: Emacs caches the results of previous text-shaping.
> You should invoke
> 
>   M-: (clear-composition-cache) RET
> 
> to force Emacs to forget the old text-shaping results.
If I have the positioning problem, this command doesn't solve it.

> > It works, Emacs uses the font correctly.
> > And it seems that the positioning is then much better. The old problem is still there but not so often.
> 
> Does this mean we can close this bug?
It's better but it's strange. And unfortunately  some years ago this problem wasn't there at all. But it's ok. 
We can close it.
Many thanks.

bug#56695: A new and old font problem with Arabic

[Eli Zaretskii]

> Date: Mon, 25 Jul 2022 17:29:23 +0200 (CEST)
> From: "fr_ml@t-online.de" <fr_ml@t-online.de>
> Cc: "56695@debbugs.gnu.org" <56695@debbugs.gnu.org>
> 
> > This is expected: Emacs caches the results of previous text-shaping.
> > You should invoke
> > 
> >   M-: (clear-composition-cache) RET
> > 
> > to force Emacs to forget the old text-shaping results.
> If I have the positioning problem, this command doesn't solve it.

It doesn't always work, indeed.

> > > It works, Emacs uses the font correctly.
> > > And it seems that the positioning is then much better. The old problem is still there but not so often.
> > 
> > Does this mean we can close this bug?
> It's better but it's strange. And unfortunately  some years ago this
> problem wasn't there at all.

I'm guessing "some years ago" you used different versions of the same
fonts.  A font that was OK in the past could regress later.

For correct complex shaping, it is very important to find good fonts.
I suggest to look for a good Arabic font that is fixed-pitch, and use
it instead of DejaVu Sans.

> But it's ok.  We can close it.  Many thanks.

Done.

bug#56695: AW: bug#56695: A new and old font problem with Arabic

[fr_ml]

> I'm guessing "some years ago" you used different versions of the same
> fonts.  A font that was OK in the past could regress later.
It was, I think, when harfbuzz was introduced.

> For correct complex shaping, it is very important to find good fonts.
> I suggest to look for a good Arabic font that is fixed-pitch, and use
> it instead of DejaVu Sans.
I never had any problem with any font. "Dejavu Sans Mono" is fixed-pitch,



bug#56695: AW: bug#56695: A new and old font problem with Arabic

[fr_ml]

Unfortunately I have another question.

Is it possible to force emacs to re-position the characters/glyphs?

It doesn't help to re-open the file.
The command:
(set-fontset-font "fontset-default" 'arabic "ANOTHER FONT") 
helps only for one-time. After some typing re-using the command doesn't help.

This command:
M-: (clear-composition-cache) RET
seems to change something but it doesn't help.

Somehow Emacs seems to remember the false positioning of the characters.
If I a start a new Emacs the problem disappears (for some time, as I said).

So how can I let Emacs "forget" the false positioning and make a new one?



bug#56695: A new and old font problem with Arabic

[Eli Zaretskii]

> Date: Sat, 13 Aug 2022 23:58:45 +0200 (CEST)
> From: "fr_ml@t-online.de" <fr_ml@t-online.de>
> Cc: "56695-done@debbugs.gnu.org" <56695-done@debbugs.gnu.org>
> 
> 
> Unfortunately I have another question.
> 
> Is it possible to force emacs to re-position the characters/glyphs?
> 
> It doesn't help to re-open the file.
> The command:
> (set-fontset-font "fontset-default" 'arabic "ANOTHER FONT") 
> helps only for one-time. After some typing re-using the command doesn't help.
> 
> This command:
> M-: (clear-composition-cache) RET
> seems to change something but it doesn't help.
> 
> Somehow Emacs seems to remember the false positioning of the characters.
> If I a start a new Emacs the problem disappears (for some time, as I said).
> 
> So how can I let Emacs "forget" the false positioning and make a new one?

If clear-composition-cache doesn't help, I don't know any way short of
restarting Emacs.  Sorry.

bug#56695: AW: bug#56695: A new and old font problem with Arabic

[fr_ml]

Ok. Thanks.