Emacs 28: Specific TTF font gets loaded with font-backend x instead of ftcrhb

Emacs 28: Specific TTF font gets loaded with font-backend x instead of ftcrhb

[Tassilo Horn]

Hi all,

I use the current Emacs from the master branch.  It is configured with
cairo and HarfBuzz support.

  (frame-parameter nil 'font-backend) ;=> (ftcrhb x)

I don't configure fonts inside emacs but it takes my default monospaced
font from GNOME settings.  Right now, that is set to PragmataPro, so
describe-char says something like

    display: by this font (glyph code)
    ftcrhb:-FSD -PragmataPro-normal-normal-normal-*-15-*-*-*-*-0-iso10646-1 (#x46)

So far, so good.  When I now switch my GNOME monospace font setting to
the JetBrains Mono font [1] and fire up a new frame, it seems like a
bitmap variant of that font is used.  describe-char says:

    display: by this font (glyph code)
    x:-misc-jetbrains mono-medium-r-normal--15-108-100-100-m-90-iso8859-1 (#x6F)

So indeed, here the font-backend is x rather than ftcrhb.  But both
PragmataPro and JetBrainsMono are normal TTF fonts.  fc-list knows them
both:

--8<---------------cut here---------------start------------->8---
➜ fc-list|egrep '(Jet|Pragmata)'|sort
/usr/share/fonts/TTF/FSD - PragmataPro Bold Italic.ttf: PragmataPro:style=Bold Italic
/usr/share/fonts/TTF/FSD - PragmataPro Bold.ttf: PragmataPro:style=Bold
/usr/share/fonts/TTF/FSD - PragmataPro Italic.ttf: PragmataPro:style=Italic
/usr/share/fonts/TTF/FSD - PragmataPro Liga Bold Italic.ttf: PragmataPro Liga:style=Bold Italic
/usr/share/fonts/TTF/FSD - PragmataPro Liga Bold.ttf: PragmataPro Liga:style=Bold
/usr/share/fonts/TTF/FSD - PragmataPro Liga Italic.ttf: PragmataPro Liga:style=Italic
/usr/share/fonts/TTF/FSD - PragmataPro Liga Regular.ttf: PragmataPro Liga:style=Regular
/usr/share/fonts/TTF/FSD - PragmataPro Mono Bold Italic.ttf: PragmataPro Mono:style=Bold Italic
/usr/share/fonts/TTF/FSD - PragmataPro Mono Bold.ttf: PragmataPro Mono:style=Bold
/usr/share/fonts/TTF/FSD - PragmataPro Mono Italic.ttf: PragmataPro Mono:style=Italic
/usr/share/fonts/TTF/FSD - PragmataPro Mono Liga Bold Italic.ttf: PragmataPro Mono Liga:style=Bold Italic
/usr/share/fonts/TTF/FSD - PragmataPro Mono Liga Bold.ttf: PragmataPro Mono Liga:style=Bold
/usr/share/fonts/TTF/FSD - PragmataPro Mono Liga Italic.ttf: PragmataPro Mono Liga:style=Italic
/usr/share/fonts/TTF/FSD - PragmataPro Mono Liga Regular.ttf: PragmataPro Mono Liga:style=Regular
/usr/share/fonts/TTF/FSD - PragmataPro Mono Regular.ttf: PragmataPro Mono:style=Regular
/usr/share/fonts/TTF/FSD - PragmataPro Regular.ttf: PragmataPro:style=Regular
/usr/share/fonts/TTF/JetBrainsMono-Bold-Italic.ttf: JetBrains Mono:style=Bold Italic
/usr/share/fonts/TTF/JetBrainsMono-Bold.ttf: JetBrains Mono:style=Bold
/usr/share/fonts/TTF/JetBrainsMono-ExtraBold-Italic.ttf: JetBrains Mono,JetBrains Mono ExtraBold:style=Extra Bold Italic,Italic
/usr/share/fonts/TTF/JetBrainsMono-ExtraBold.ttf: JetBrains Mono,JetBrains Mono ExtraBold:style=Extra Bold,Regular
/usr/share/fonts/TTF/JetBrainsMono-Italic.ttf: JetBrains Mono:style=Italic
/usr/share/fonts/TTF/JetBrainsMono-Medium-Italic.ttf: JetBrains Mono,JetBrains Mono Medium:style=Medium Italic,Italic
/usr/share/fonts/TTF/JetBrainsMono-Medium.ttf: JetBrains Mono,JetBrains Mono Medium:style=Medium,Regular
/usr/share/fonts/TTF/JetBrainsMono-Regular.ttf: JetBrains Mono:style=Regular
--8<---------------cut here---------------end--------------->8---

I've tried other fonts as well (like DejaVu Mono) and they all work just
fine in Emacs.  It's just that single one which doesn't work in emacs
but it does work in other apps (like all GNOME apps).

Any idea how to debug what's the problem with that font?

Thanks,
  Tassilo

[1] https://github.com/JetBrains/JetBrainsMono

Re: Emacs 28: Specific TTF font gets loaded with font-backend x instead of ftcrhb

[Robert Pluim]

>>>>> On Tue, 04 Feb 2020 12:12:13 +0100, Tassilo Horn <tsdh@gnu.org> said:

    Tassilo> Hi all,
    Tassilo> I use the current Emacs from the master branch.  It is configured with
    Tassilo> cairo and HarfBuzz support.

    Tassilo>   (frame-parameter nil 'font-backend) ;=> (ftcrhb x)

    Tassilo> I don't configure fonts inside emacs but it takes my default monospaced
    Tassilo> font from GNOME settings.  Right now, that is set to PragmataPro, so
    Tassilo> describe-char says something like

    Tassilo>     display: by this font (glyph code)
    Tassilo>     ftcrhb:-FSD -PragmataPro-normal-normal-normal-*-15-*-*-*-*-0-iso10646-1 (#x46)

    Tassilo> So far, so good.  When I now switch my GNOME monospace font setting to
    Tassilo> the JetBrains Mono font [1] and fire up a new frame, it seems like a
    Tassilo> bitmap variant of that font is used.  describe-char says:

    Tassilo>     display: by this font (glyph code)
    Tassilo>     x:-misc-jetbrains mono-medium-r-normal--15-108-100-100-m-90-iso8859-1 (#x6F)

This is from 'emacs -Q' as well? One thing you could do is run

FC_DEBUG=15 emacs -Q

which will output debug on what FontConfig is trying to do. The
jetbrains font is using iso8859-1 rather than iso10646-1, perhaps
there is some influence from locale settings?

FWIW, I canʼt reproduce this. Which distribution is this on?

Robert

Re: Emacs 28: Specific TTF font gets loaded with font-backend x instead of ftcrhb

[Robert Pluim]

>>>>> On Tue, 04 Feb 2020 13:15:48 +0100, Robert Pluim <rpluim@gmail.com> said:
    Robert> This is from 'emacs -Q' as well? One thing you could do is run

    Robert> FC_DEBUG=15 emacs -Q

    Robert> which will output debug on what FontConfig is trying to do. The
    Robert> jetbrains font is using iso8859-1 rather than iso10646-1, perhaps
    Robert> there is some influence from locale settings?

One other thing to try is M-: (setq font-log nil) before you create
the second frame, and then look at the value of font-log afterwards.

Robert

Re: Emacs 28: Specific TTF font gets loaded with font-backend x instead of ftcrhb

[Tassilo Horn]

[-- Attachment #1: Type: text/plain, Size: 1298 bytes --]

Robert Pluim <rpluim@gmail.com> writes:

Hi Robert,

>>>>>> On Tue, 04 Feb 2020 13:15:48 +0100, Robert Pluim <rpluim@gmail.com> said:
>     Robert> This is from 'emacs -Q' as well? One thing you could do is run

Yes, emacs -Q is the same.

>     Robert> FC_DEBUG=15 emacs -Q

Wow, that's large.  I uploaded it here:

  https://send.firefox.com/download/c5f0f5318f936032/#OGuNHBWhZCZHTCW0TV2YDA

>     Robert> which will output debug on what FontConfig is trying to do. The
>     Robert> jetbrains font is using iso8859-1 rather than iso10646-1, perhaps
>     Robert> there is some influence from locale settings?

I don't know.  I use a en_US.UTF-8 locale with some German unit
exceptions here.

➜ locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC=de_DE.UTF-8
LC_TIME=de_DE.UTF-8
LC_COLLATE="en_US.UTF-8"
LC_MONETARY=de_DE.UTF-8
LC_MESSAGES="en_US.UTF-8"
LC_PAPER=de_DE.UTF-8
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT=de_DE.UTF-8
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

> One other thing to try is M-: (setq font-log nil) before you create
> the second frame, and then look at the value of font-log afterwards.

That's the value after changing the font, setting it to nil and opening
a new frame.

[-- Attachment #2: font-log.gz --]
[-- Type: application/gzip, Size: 2314 bytes --]

[-- Attachment #3: Type: text/plain, Size: 481 bytes --]


> FWIW, I canʼt reproduce this. Which distribution is this on?

This is on Arch Linux where the JetBrainsMono font is installed as a
system package in version 1.0.1.  I've seen that the current version is
1.0.2, so maybe the issue is fixed upstream alread...

So I've just downloaded the JetBrainsMono-1.0.2.zip and copied over the
new TTF files to /usr/share/fonts/TTF/ overwriting the existing ones and
ran "sudo fc-cache".  Therafter, my emacs shows mostly boxes:


[-- Attachment #4: Screenshot-2020-02-04_135307-692x348.png --]
[-- Type: image/png, Size: 18523 bytes --]

[-- Attachment #5: Type: text/plain, Size: 54 bytes --]


Gedit still looks good with that font.

Bye,
Tassilo

Re: Emacs 28: Specific TTF font gets loaded with font-backend x instead of ftcrhb

[Robert Pluim]

>>>>> On Tue, 04 Feb 2020 13:58:12 +0100, Tassilo Horn <tsdh@gnu.org> said:

    Tassilo> Robert Pluim <rpluim@gmail.com> writes:
    Tassilo> Hi Robert,

    >>>>>>> On Tue, 04 Feb 2020 13:15:48 +0100, Robert Pluim <rpluim@gmail.com> said:
    Robert> This is from 'emacs -Q' as well? One thing you could do is run

    Tassilo> Yes, emacs -Q is the same.

    Robert> FC_DEBUG=15 emacs -Q

    Tassilo> Wow, that's large.  I uploaded it here:

    Tassilo>   https://send.firefox.com/download/c5f0f5318f936032/#OGuNHBWhZCZHTCW0TV2YDA

Yes, FontConfig can be kind of verbose. I didnʼt notice anything
untoward there.

    Tassilo> That's the value after changing the font, setting it to nil and opening
    Tassilo> a new frame.

Something is causing Emacs to look for -misc-jetbrains mono- ,
and itʼs failing to find the TTF version.

    >> FWIW, I canʼt reproduce this. Which distribution is this on?

    Tassilo> This is on Arch Linux where the JetBrainsMono font is installed as a
    Tassilo> system package in version 1.0.1.  I've seen that the current version is
    Tassilo> 1.0.2, so maybe the issue is fixed upstream alread...

    Tassilo> So I've just downloaded the JetBrainsMono-1.0.2.zip and copied over the
    Tassilo> new TTF files to /usr/share/fonts/TTF/ overwriting the existing ones and
    Tassilo> ran "sudo fc-cache".  Therafter, my emacs shows mostly boxes:

Does Arch have some kind of 'fallback to bitmap' configuration? What
does 'C-u C-x =' say about which font is being used?

Robert

Re: Emacs 28: Specific TTF font gets loaded with font-backend x instead of ftcrhb

[Tassilo Horn]

Robert Pluim <rpluim@gmail.com> writes:

>     >> FWIW, I canʼt reproduce this. Which distribution is this on?
>
>     Tassilo> This is on Arch Linux where the JetBrainsMono font is installed as a
>     Tassilo> system package in version 1.0.1.  I've seen that the current version is
>     Tassilo> 1.0.2, so maybe the issue is fixed upstream alread...
>
>     Tassilo> So I've just downloaded the JetBrainsMono-1.0.2.zip and copied over the
>     Tassilo> new TTF files to /usr/share/fonts/TTF/ overwriting the existing ones and
>     Tassilo> ran "sudo fc-cache".  Therafter, my emacs shows mostly boxes:
>
> Does Arch have some kind of 'fallback to bitmap' configuration?

Not that I know of but I'm no font config expert.  Usually, it just
works...

> What does 'C-u C-x =' say about which font is being used?

That's with emacs -Q and the working PragmataPro GNOME setting on the
first char in *scratch*.

--8<---------------cut here---------------start------------->8---
             position: 1 of 145 (0%), column: 0
            character: ; (displayed as ;) (codepoint 59, #o73, #x3b)
              charset: ascii (ASCII (ISO646 IRV))
code point in charset: 0x3B
               script: latin
               syntax: < 	which means: comment
             category: .:Base, a:ASCII, l:Latin, r:Roman
             to input: type "C-x 8 RET 3b" or "C-x 8 RET SEMICOLON"
          buffer code: #x3B
            file code: #x3B (encoded by coding system utf-8-unix)
              display: by this font (glyph code)
    ftcrhb:-FSD -PragmataPro-normal-normal-normal-*-15-*-*-*-*-0-iso10646-1 (#x1C)

Character code properties: customize what to show
  name: SEMICOLON
  general-category: Po (Punctuation, Other)
  decomposition: (59) (';')

There are text properties here:
  face                 font-lock-comment-delimiter-face
  fontified            nil
--8<---------------cut here---------------end--------------->8---

And that's the same after changing the font to JetBrains Mono.

--8<---------------cut here---------------start------------->8---
             position: 1 of 145 (0%), column: 0
            character: ; (displayed as ;) (codepoint 59, #o73, #x3b)
              charset: ascii (ASCII (ISO646 IRV))
code point in charset: 0x3B
               script: latin
               syntax: < 	which means: comment
             category: .:Base, a:ASCII, l:Latin, r:Roman
             to input: type "C-x 8 RET 3b" or "C-x 8 RET SEMICOLON"
          buffer code: #x3B
            file code: #x3B (encoded by coding system utf-8-unix)
              display: by this font (glyph code)
    x:-misc-jetbrains mono-medium-r-normal--13-94-100-100-m-80-iso8859-1 (#x3B)

Character code properties: customize what to show
  name: SEMICOLON
  general-category: Po (Punctuation, Other)
  decomposition: (59) (';')

There are text properties here:
  face                 font-lock-comment-delimiter-face
  fontified            nil
--8<---------------cut here---------------end--------------->8---

Bye,
Tassilo

Re: Emacs 28: Specific TTF font gets loaded with font-backend x instead of ftcrhb

[Robert Pluim]

>>>>> On Tue, 04 Feb 2020 15:21:10 +0100, Tassilo Horn <tsdh@gnu.org> said:

    Tassilo> Robert Pluim <rpluim@gmail.com> writes:
    >> >> FWIW, I canʼt reproduce this. Which distribution is this on?
    >> 
    Tassilo> This is on Arch Linux where the JetBrainsMono font is installed as a
    Tassilo> system package in version 1.0.1.  I've seen that the current version is
    Tassilo> 1.0.2, so maybe the issue is fixed upstream alread...
    >> 
    Tassilo> So I've just downloaded the JetBrainsMono-1.0.2.zip and copied over the
    Tassilo> new TTF files to /usr/share/fonts/TTF/ overwriting the existing ones and
    Tassilo> ran "sudo fc-cache".  Therafter, my emacs shows mostly boxes:
    >> 
    >> Does Arch have some kind of 'fallback to bitmap' configuration?

Could you try uninstalling the system package, and installing the
1.0.2 version using the fonts tool? Just open the font(s) from your file
manager, and there will be an 'install' button at the top right which
will copy it somewhere under ~/.local/ rather than the system directories.

    Tassilo> Not that I know of but I'm no font config expert.  Usually, it just
    Tassilo> works...

I took a look at the Arch jetbrainsmono-1.0.1 package, and it wants to
install 'xorg-bdftopcf', which is used for converting fonts from
bitmap formats, which is somewhat strange for a TTF font.

Robert

Re: Emacs 28: Specific TTF font gets loaded with font-backend x instead of ftcrhb

[Tassilo Horn]

Robert Pluim <rpluim@gmail.com> writes:

Hi Robert,

I've now tried at home with my own computer (also running Arch) and got
the same issue so it is at least not machine specific.

> Could you try uninstalling the system package, and installing the
> 1.0.2 version using the fonts tool? Just open the font(s) from your
> file manager, and there will be an 'install' button at the top right
> which will copy it somewhere under ~/.local/ rather than the system
> directories.

I did that, and now it works just fine.  Is there something different
(except for the location) when installing a font by just copying the ttf
files to /usr/share/fonts/TTF/ and installing them with the Fonts tool?
I mean, now it works with the very same ttf files I've previously copied
to /usr/share/fonts/TTF/ (replacing the files installed by the Arch
package).

Hm, another minor thing: when I compare the rendering of the above
paragraph in gedit and emacs, there are slight differences for some
glyphs.  For example, the F in TTF is slightly higher than the Ts, and
the y is slightly lower than, e.g., an a or o.  In gedit, their height
is equal.

>     Tassilo> Not that I know of but I'm no font config expert.
>     Tassilo> Usually, it just works...
>
> I took a look at the Arch jetbrainsmono-1.0.1 package, and it wants to
> install 'xorg-bdftopcf', which is used for converting fonts from
> bitmap formats, which is somewhat strange for a TTF font.

I think that's an indirect dependency of xorg-font-utils but I don't
know why that is needed.  I think, it isn't.  At least the installation
recipe just says "copy the TTF files to /usr/share/fonts/TTF/ and make
them readable".

  https://git.archlinux.org/svntogit/community.git/tree/trunk/PKGBUILD?h=packages/ttf-jetbrains-mono

Bye,
Tassilo

Re: Emacs 28: Specific TTF font gets loaded with font-backend x instead of ftcrhb

[Robert Pluim]

>>>>> On Tue, 04 Feb 2020 19:32:08 +0100, Tassilo Horn <tsdh@gnu.org> said:

    Tassilo> Robert Pluim <rpluim@gmail.com> writes:
    Tassilo> Hi Robert,

    Tassilo> I've now tried at home with my own computer (also running Arch) and got
    Tassilo> the same issue so it is at least not machine specific.

I spun up an Arch VM, and the system jetbrains mono package works fine
there.

    Tassilo> I did that, and now it works just fine.  Is there something different
    Tassilo> (except for the location) when installing a font by just copying the ttf
    Tassilo> files to /usr/share/fonts/TTF/ and installing them with the Fonts tool?
    Tassilo> I mean, now it works with the very same ttf files I've previously copied
    Tassilo> to /usr/share/fonts/TTF/ (replacing the files installed by the Arch
    Tassilo> package).

Not that I know of, as far as I know it just does the moral equivalent of
'fc-cache' somewhere.

    Tassilo> Hm, another minor thing: when I compare the rendering of the above
    Tassilo> paragraph in gedit and emacs, there are slight differences for some
    Tassilo> glyphs.  For example, the F in TTF is slightly higher than the Ts, and
    Tassilo> the y is slightly lower than, e.g., an a or o.  In gedit, their height
    Tassilo> is equal.

That sounds like there are hinting differences. Cairo doesnʼt always
obey the Gnome Tweaks hinting settings. The following might help:
<https://www.kilobitspersecond.com/2009/04/17/ubuntu-font-hinting-you-a-cautionary-tale/>

Robert

Re: Emacs 28: Specific TTF font gets loaded with font-backend x instead of ftcrhb

[Tassilo Horn]

Robert Pluim <rpluim@gmail.com> writes:

Hi Robert,

> Not that I know of, as far as I know it just does the moral equivalent
> of 'fc-cache' somewhere.

Seems so.  On my work computer I removed the system package and then
just copied the JetBrainsMono ttf files to /usr/share/fonts/TTF/, and
then it worked just fine...

>     Tassilo> Hm, another minor thing: when I compare the rendering of
>     Tassilo> the above paragraph in gedit and emacs, there are slight
>     Tassilo> differences for some glyphs.  For example, the F in TTF
>     Tassilo> is slightly higher than the Ts, and the y is slightly
>     Tassilo> lower than, e.g., an a or o.  In gedit, their height is
>     Tassilo> equal.
>
> That sounds like there are hinting differences. Cairo doesnʼt always
> obey the Gnome Tweaks hinting settings. The following might help:
> <https://www.kilobitspersecond.com/2009/04/17/ubuntu-font-hinting-you-a-cautionary-tale/>

Hm, I've tried creating the ~/.config/fontconfig/fonts.conf below which
looks like the equivalent of the GNOME settings (I tested with and
without the lcdfilter because that's not visible there) but the issue
persists.

--8<---------------cut here---------------start------------->8---
<?xml version="1.0"?>
<!DOCTYPE fontconfig SYSTEM "fonts.dtd">
<fontconfig>
  <match target="font">
    <edit name="antialias" mode="assign">
      <bool>true</bool>
    </edit>
    <edit name="hinting" mode="assign">
      <bool>true</bool>
    </edit>
    <edit name="hintstyle" mode="assign">
      <const>hintfull</const>
    </edit>
    <edit name="rgba" mode="assign">
      <const>rgb</const>
    </edit>
    <edit name="lcdfilter" mode="assign">
      <const>lcddefault</const>
    </edit>
  </match>
</fontconfig>

<!-- Local Variables: -->
<!-- mode: nxml -->
<!-- End: -->
--8<---------------cut here---------------end--------------->8---

Bye,
  Tassilo

Re: Emacs 28: Specific TTF font gets loaded with font-backend x instead of ftcrhb

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Date: Tue, 04 Feb 2020 13:25:00 +0100
> 
> One other thing to try is M-: (setq font-log nil) before you create
> the second frame, and then look at the value of font-log afterwards.

It is best to look at (pp (reverse font-log)), so that you see it top
to bottom, not the other way around, and also nicely formatted.

Re: Emacs 28: Specific TTF font gets loaded with font-backend x instead of ftcrhb

[Tassilo Horn]

Eli Zaretskii <eliz@gnu.org> writes:

Hi Eli,

>> One other thing to try is M-: (setq font-log nil) before you create
>> the second frame, and then look at the value of font-log afterwards.
>
> It is best to look at (pp (reverse font-log)), so that you see it top
> to bottom, not the other way around, and also nicely formatted.

I'll try to remember it for the next time.

While we are at a related topic: I think that I've read a message of
yours where you said that prettify-symbols-mode is a candidate for
deprecation because there was a better way for (real) ligatures, at
least, that's what I understood.

Is "the new way" the thing with composition-function-table?  Something
like the Fira Code font emacs setup shown here?

  https://github.com/tonsky/FiraCode/wiki/Emacs-instructions#using-composition-char-table

I've tried that code although with the PragmataPro font in the hope it
would work at least partly given that both fonts offer a quite large
common subset of ligatures but couldn't see any difference, e.g., -> is
not composed to an arrow.

Is there any documentation I could read?

Bye,
Tassilo

Re: Emacs 28: Specific TTF font gets loaded with font-backend x instead of ftcrhb

[Eli Zaretskii]

> From: Tassilo Horn <tsdh@gnu.org>
> Cc: help-gnu-emacs@gnu.org
> Date: Tue, 04 Feb 2020 19:43:38 +0100
> 
> Is "the new way" the thing with composition-function-table?

Yes.  We should use the so-called "automatic composition", which works
via composition-function-table (as opposed to "static composition"
currently used by prettify-symbols-mode).

> Something like the Fira Code font emacs setup shown here?
> 
>   https://github.com/tonsky/FiraCode/wiki/Emacs-instructions#using-composition-char-table

Something like that, yes.

> I've tried that code although with the PragmataPro font in the hope it
> would work at least partly given that both fonts offer a quite large
> common subset of ligatures but couldn't see any difference, e.g., -> is
> not composed to an arrow.

That probably means PragmataPro doesn't support ligatures, at least
not the one for ->.

> Is there any documentation I could read?

This area is notoriously under-documented.  What we have (and it's
abysmally insufficient) is this:

 . the code in composite.el, including the basic composition rules for
   accented characters
 . the code which sets composition-function-table in the various files
   under lisp/language/, where language-specific composition rules are
   provided
 . the doc string of composition-get-gstring (the only place where we
   document the form of the vectors shown by "C-u C-x =" for composed
   character sequences)

Re: Emacs 28: Specific TTF font gets loaded with font-backend x instead of ftcrhb

[Tassilo Horn]

Eli Zaretskii <eliz@gnu.org> writes:

Hi Eli,

>> Is "the new way" the thing with composition-function-table?
>
> Yes.  We should use the so-called "automatic composition", which works
> via composition-function-table (as opposed to "static composition"
> currently used by prettify-symbols-mode).

Ok, thanks for the confirmation.

>> Something like the Fira Code font emacs setup shown here?
>> 
>>   https://github.com/tonsky/FiraCode/wiki/Emacs-instructions#using-composition-char-table
>
> Something like that, yes.
>
>> I've tried that code although with the PragmataPro font in the hope
>> it would work at least partly given that both fonts offer a quite
>> large common subset of ligatures but couldn't see any difference,
>> e.g., -> is not composed to an arrow.
>
> That probably means PragmataPro doesn't support ligatures, at least
> not the one for ->.

It does, at least I get an -> arrow glyph in gedit.

>> Is there any documentation I could read?
>
> This area is notoriously under-documented.  What we have (and it's
> abysmally insufficient) is this:
>
>  . the code in composite.el, including the basic composition rules for
>    accented characters
>  . the code which sets composition-function-table in the various files
>    under lisp/language/, where language-specific composition rules are
>    provided
>  . the doc string of composition-get-gstring (the only place where we
>    document the form of the vectors shown by "C-u C-x =" for composed
>    character sequences)

Ok, I'll tinker with it when I find some spare time.

Bye,
Tassilo

Re: Emacs 28: Specific TTF font gets loaded with font-backend x instead of ftcrhb

[Eli Zaretskii]

> From: Tassilo Horn <tsdh@gnu.org>
> Cc: help-gnu-emacs@gnu.org
> Date: Wed, 05 Feb 2020 17:44:59 +0100
> 
> >> I've tried that code although with the PragmataPro font in the hope
> >> it would work at least partly given that both fonts offer a quite
> >> large common subset of ligatures but couldn't see any difference,
> >> e.g., -> is not composed to an arrow.
> >
> > That probably means PragmataPro doesn't support ligatures, at least
> > not the one for ->.
> 
> It does, at least I get an -> arrow glyph in gedit.

Can you show the code you tried to use, and also tell in which version
of Emacs did you try that?

> Ok, I'll tinker with it when I find some spare time.

Feel free to ask any questions or point out unclear areas, we should
at least make the effort of clarifying the existing docs to the extent
that we are able.

Re: Emacs 28: Specific TTF font gets loaded with font-backend x instead of ftcrhb

[Tassilo Horn]

Eli Zaretskii <eliz@gnu.org> writes:

Hi Eli,

>> > That probably means PragmataPro doesn't support ligatures, at least
>> > not the one for ->.
>> 
>> It does, at least I get an -> arrow glyph in gedit.
>
> Can you show the code you tried to use, and also tell in which version
> of Emacs did you try that?

I've tried it at home maybe two weeks ago with the current master at
that time.  I've just tried anew at work with today's master and the
JetBrains Mono font, and now it works just fine:

--8<---------------cut here---------------start------------->8---
(let ((alist '((33 . ".\\(?:\\(?:==\\|!!\\)\\|[!=]\\)")
               (35 . ".\\(?:###\\|##\\|_(\\|[#(?[_{]\\)")
               (36 . ".\\(?:>\\)")
               (37 . ".\\(?:\\(?:%%\\)\\|%\\)")
               (38 . ".\\(?:\\(?:&&\\)\\|&\\)")
               (42 . ".\\(?:\\(?:\\*\\*/\\)\\|\\(?:\\*[*/]\\)\\|[*/>]\\)")
               (43 . ".\\(?:\\(?:\\+\\+\\)\\|[+>]\\)")
               (45 . ".\\(?:\\(?:-[>-]\\|<<\\|>>\\)\\|[<>}~-]\\)")
               (46 . ".\\(?:\\(?:\\.[.<]\\)\\|[.=-]\\)")
               (47 . ".\\(?:\\(?:\\*\\*\\|//\\|==\\)\\|[*/=>]\\)")
               (48 . ".\\(?:x[a-zA-Z]\\)")
               (58 . ".\\(?:::\\|[:=]\\)")
               (59 . ".\\(?:;;\\|;\\)")
               (60 . ".\\(?:\\(?:!--\\)\\|\\(?:~~\\|->\\|\\$>\\|\\*>\\|\\+>\\|--\\|<[<=-]\\|=[<=>]\\||>\\)\\|[*$+~/<=>|-]\\)")
               (61 . ".\\(?:\\(?:/=\\|:=\\|<<\\|=[=>]\\|>>\\)\\|[<=>~]\\)")
               (62 . ".\\(?:\\(?:=>\\|>[=>-]\\)\\|[=>-]\\)")
               (63 . ".\\(?:\\(\\?\\?\\)\\|[:=?]\\)")
               (91 . ".\\(?:]\\)")
               (92 . ".\\(?:\\(?:\\\\\\\\\\)\\|\\\\\\)")
               (94 . ".\\(?:=\\)")
               (119 . ".\\(?:ww\\)")
               (123 . ".\\(?:-\\)")
               (124 . ".\\(?:\\(?:|[=|]\\)\\|[=>|]\\)")
               (126 . ".\\(?:~>\\|~~\\|[>=@~-]\\)"))))
  (dolist (char-regexp alist)
    (set-char-table-range composition-function-table (car char-regexp)
                          `([,(cdr char-regexp) 0 font-shape-gstring]))))
--8<---------------cut here---------------end--------------->8---

After evaling that, -> becomes an arrow and ~> becomes an arrow with
curvy stroke.  So either there's some issue with PragmataPro or my
computer at home.

>> Ok, I'll tinker with it when I find some spare time.
>
> Feel free to ask any questions or point out unclear areas, we should
> at least make the effort of clarifying the existing docs to the extent
> that we are able.

Do I understand it correctly that "overshooting" with the above rules
has no bad effect, i.e., if there is a composition rule for -> but the
font has no ligature glyph for that, then it just stays the way it is?
So there could be some ligature-minor-mode which just adds all possible
composition rules?  (Just speaking naively, I guess there are several
distinct categories of ligatures which you would want to enable/disable
on a per-mode basis.)

Bye,
Tassilo

Re: Emacs 28: Specific TTF font gets loaded with font-backend x instead of ftcrhb

[Eli Zaretskii]

> From: Tassilo Horn <tsdh@gnu.org>
> Cc: help-gnu-emacs@gnu.org
> Date: Thu, 06 Feb 2020 08:12:06 +0100
> 
> After evaling that, -> becomes an arrow and ~> becomes an arrow with
> curvy stroke.  So either there's some issue with PragmataPro or my
> computer at home.

Well, it would be interesting to hear what happens with that other
machine and/or the font.  Be sure to try that in an Emacs built with
the HarfBuzz library, as it should have the most advanced ligature
support there is.

> Do I understand it correctly that "overshooting" with the above rules
> has no bad effect, i.e., if there is a composition rule for -> but the
> font has no ligature glyph for that, then it just stays the way it is?

Not exactly.  In Emacs built with HarfBuzz, you will see the original
ASCII characters displayed, but handled as a single grapheme cluster,
i.e. the cursor will be "widened" to include all of them, and a single
C-f will move across all of them.  However, I believe this is what
users should expect when the font doesn't include some of the
ligatures, no?

> So there could be some ligature-minor-mode which just adds all possible
> composition rules?  (Just speaking naively, I guess there are several
> distinct categories of ligatures which you would want to enable/disable
> on a per-mode basis.)

The part in parentheses is exactly the non-trivial part: we should
figure out which ligatures should be in effect in what major modes,
and probably also as function of some user preferences (such as the
language used in the buffer).  IOW, we need to design the UI for
specifying what classes of ligatures to activate.

Burt if you only care about ligatures in programming languages, the
job becomes much simpler, I think.  Although I'd still expect the
ligatures in effect to depend on the programming language of the
current buffer.  Which means composition-function-table needs to be
buffer-local, and we should make sure making it buffer-local does TRT.

Re: Emacs 28: Specific TTF font gets loaded with font-backend x instead of ftcrhb

[Tassilo Horn]

[Sorry for the second mail, I guess my first is in the moderation queue
because of a too large image attachment which I shouldn't have added
in the first place.  Now replaced with link.]

On Thu, 2020-02-06 at 20:15 +0200, Eli Zaretskii wrote:
> > After evaling that, -> becomes an arrow and ~> becomes an arrow with
> > curvy stroke.  So either there's some issue with PragmataPro or my
> > computer at home.
> 
> Well, it would be interesting to hear what happens with that other
> machine and/or the font.  Be sure to try that in an Emacs built with
> the HarfBuzz library, as it should have the most advanced ligature
> support there is.

I'm pretty sure I had built with HarfBuzz but tried again anyway.  This
time it worked just fine.  Here is a screenshot showing two emacs
frames with ligatures in action.  The left one is with the JetBrains
Mono font, the right one is with the PragmataPro Liga font.

https://imagebin.ca/v/5BUpzkwkgR1k

> > Do I understand it correctly that "overshooting" with the above
> > rules has no bad effect, i.e., if there is a composition rule for ->
> > but the font has no ligature glyph for that, then it just stays the
> > way it is?
> 
> Not exactly.  In Emacs built with HarfBuzz, you will see the original
> ASCII characters displayed, but handled as a single grapheme cluster,
> i.e. the cursor will be "widened" to include all of them, and a single
> C-f will move across all of them.

That doesn't seem to happen.  forward-char moves inside ligature
sequences no matter if the font has a ligature or not.  I.e., even with
a ligature ~= which gets composed to an equal sign with curvy upper line
point move half-by-half.

> However, I believe this is what users should expect when the font
> doesn't include some of the ligatures, no?

It doesn't really matter to me.  Maybe it would be a bit awkward for
longer sequences like =/= if you could edit them only from the
beginning/end.

> > So there could be some ligature-minor-mode which just adds all
> > possible composition rules?  (Just speaking naively, I guess there
> > are several distinct categories of ligatures which you would want to
> > enable/disable on a per-mode basis.)
> 
> The part in parentheses is exactly the non-trivial part: we should
> figure out which ligatures should be in effect in what major modes,
> and probably also as function of some user preferences (such as the
> language used in the buffer).  IOW, we need to design the UI for
> specifying what classes of ligatures to activate.

I've tried categorizing them a bit but this feels like quite a hard
task, and I've just done programming ligatures.  Does -- fit in an
operators category (because of C, Java, ...) or a comment category
(because of SQL) or better forget about concrete language syntaxes and
have a dashes category?

> Burt if you only care about ligatures in programming languages, the
> job becomes much simpler, I think.  Although I'd still expect the
> ligatures in effect to depend on the programming language of the
> current buffer.

Right now I've just enabled anything.  To me it seems like there is some
guideline like "if ligature X is not applicable in programming language
Y then the characted sequence it is composed from won't appear there
anyway".

But one thing which comes to mind is that one might want to suppress
ligature composition inside strings...

> Which means composition-function-table needs to be buffer-local, and
> we should make sure making it buffer-local does TRT.

This doesn't seem to work right now.  See the FIXME at the bottom of
below code.

--8<---------------cut here---------------start------------->8---
(defgroup ligature nil
  "Support for font ligatures"
  :version "28.1"
  :prefix "ligature-")

(defcustom ligature-arrows
  (list "-->" "<!--" "->>" "<<-" "->" "<-"
	"<-<" ">>-" ">-" "<~>" "-<" "-<<"
	"<=>" "=>" "<=<" "<<=" "<==" "<==>" "==>" "=>>" ">=>" ">>="
	"<-|" "<=|"  "|=>" "|->" "<~~" "<~"  "~~>"
	"~>" "<->")
  "Arrow ligatures."
  :type '(repeat string))

(defcustom ligature-misc
  (list "..<"  "~-" "-~" "~@"  "-|" "_|_" "|-" "||-" "|=" "||="
	".?" "?="  "<|>" "<:" ":<" ":>" ">:"
	".=" ".-" "__" "<<<" ">>>" "<<" ">>" "~~"
	"<$>" "<$" "$>" "<+>" "<+" "+>" "<*>" "<*" "*>" "</" "</>" "/>"
	"|}" "{|" "[<" ">]" ":?>" ":?"  "[||]" "?:" "?."
	"|>" "<|" "||>" "<||" "|||>" "<|||::=" "|]" "[|"
	"#{" "#[" "]#" "#(" "#?" "#_" "#_(" "#:" "#!" "#=")
  "Miscellaneous ligatures."
  :type '(repeat string))

(defcustom ligature-relations
  (list "==" "!=" "<=" ">=" "=:=" "!==" "===" "<>" "/==" "=!=" "=/=" "~=" ":="
	"/=" "^=")
  "Relation ligatures."
  :type '(repeat string))

(defcustom ligature-operators
  (list "&&" "&&&" "||" "++" "--" "!!" "::" "+++" "??" ":::" "***" "---"
	"/\\" "\\/")
  "Operator ligatures."
  :type '(repeat string))

(defcustom ligature-comments-c-like
  (list "//" "///" "/**" "/*" "*/")
  "Ligatures for comments in C-like languages."
  :type '(repeat string))

(defcustom ligature-comments-xml-like
  (list "<!--" "-->")
  "Ligatures for comments in XML-like languages."
  :type '(repeat string))

(defcustom ligature-hashes
  (list "##" "###" "####")
  "Ligatures for comments in languages with # being the comment character."
  :type '(repeat string))

(defcustom ligature-dots
  (list "..." "..")
  "Dot ligatures."
  :type '(repeat string))

(defcustom ligature-semicolons
  (list ";;" ";;;")
  "Ligatures for comments in lisp languages."
  :type '(repeat string))

(defun ligature--get-all ()
  (append ligature-arrows
	  ligature-relations
	  ligature-operators
	  ligature-misc
	  ligature-dots
	  ligature-comments-c-like
	  ligature-comments-xml-like
	  ligature-hashes
	  ligature-semicolons))

(defun ligature--apply (ligatures)
  (let ((groups (seq-group-by #'string-to-char ligatures)))
    (dolist (group groups)
      (let ((c (car group))
	    (rx (regexp-opt (mapcar (lambda (s) (substring s 1))
				    (cdr group)))))
	(set-char-table-range composition-function-table
			      c `([,(concat "." rx) 0 font-shape-gstring]))))))

(defun ligature-enable-globally ()
  (interactive)
  (ligature--apply (ligature--get-all)))

(define-minor-mode ligature-minor-mode
  "A mode for font ligatures."
  nil "" nil
  (if ligature-minor-mode
      (progn
	;; FIXME: This doesn't work.  When enabled in ~/.emacs, there won't be
	;; a local variable.  When enabled in a buffer manually, there will be
	;; a local variable but the global value is the same (and also includes
	;; the ligature composition rules).
	(make-local-variable 'composition-function-table)
	(ligature--apply (ligature--get-all)))
    ;; FIXME: Even if the above worked, this could remove much more than this
    ;; mode added itself.
    (kill-local-variable 'composition-function-table)))

(provide 'ligature)
--8<---------------cut here---------------end--------------->8---

Bye,
Tassilo

Re: Emacs 28: Specific TTF font gets loaded with font-backend x instead of ftcrhb

[Eli Zaretskii]

> From: Tassilo Horn <tsdh@gnu.org>
> Date: Fri, 07 Feb 2020 09:57:36 +0100
> 
> On Thu, 2020-02-06 at 20:15 +0200, Eli Zaretskii wrote:
> > > After evaling that, -> becomes an arrow and ~> becomes an arrow with
> > > curvy stroke.  So either there's some issue with PragmataPro or my
> > > computer at home.
> > 
> > Well, it would be interesting to hear what happens with that other
> > machine and/or the font.  Be sure to try that in an Emacs built with
> > the HarfBuzz library, as it should have the most advanced ligature
> > support there is.
> 
> I'm pretty sure I had built with HarfBuzz but tried again anyway.  This
> time it worked just fine.  I'll attach a screenshot showing two emacs
> frames with ligatures in action.  The left one is with the JetBrains
> Mono font, the right one is with the PragmataPro Liga font.

So the conclusion is that this works with both fonts?  (I cannot be
sure because the two displays look somewhat differently).

> > > Do I understand it correctly that "overshooting" with the above
> > > rules has no bad effect, i.e., if there is a composition rule for ->
> > > but the font has no ligature glyph for that, then it just stays the
> > > way it is?
> > 
> > Not exactly.  In Emacs built with HarfBuzz, you will see the original
> > ASCII characters displayed, but handled as a single grapheme cluster,
> > i.e. the cursor will be "widened" to include all of them, and a single
> > C-f will move across all of them.
> 
> That doesn't seem to happen.  forward-char moves inside ligature
> sequences no matter if the font has a ligature or not.  I.e., even with
> a ligature ~= which gets composed to an equal sign with curvy upper line
> point move half-by-half.

I think that's because you use font-shape-gstring directly.  You
should use compose-gstring-for-graphic instead.

> > > So there could be some ligature-minor-mode which just adds all
> > > possible composition rules?  (Just speaking naively, I guess there
> > > are several distinct categories of ligatures which you would want to
> > > enable/disable on a per-mode basis.)
> > 
> > The part in parentheses is exactly the non-trivial part: we should
> > figure out which ligatures should be in effect in what major modes,
> > and probably also as function of some user preferences (such as the
> > language used in the buffer).  IOW, we need to design the UI for
> > specifying what classes of ligatures to activate.
> 
> I've tried categorizing them a bit but this feels like quite a hard
> task, and I've just done programming ligatures.  Does -- fit in an
> operators category (because of C, Java, ...) or a comment category
> (because of SQL) or better forget about concrete language syntaxes and
> have a dashes category?

Yes, this is non-trivial.

> > Burt if you only care about ligatures in programming languages, the
> > job becomes much simpler, I think.  Although I'd still expect the
> > ligatures in effect to depend on the programming language of the
> > current buffer.
> 
> Right now I've just enabled anything.

Not really everything, IMO, as there are also ligatures relevant only
to human-readable text.  For example, see this URL:

  https://en.wikipedia.org/wiki/Orthographic_ligature#Ligatures_in_Unicode_(Latin_alphabets)

> To me it seems like there is some guideline like "if ligature X is
> not applicable in programming language Y then the characted sequence
> it is composed from won't appear there anyway".

This is probably correct for programming-language major modes, yes.

> But one thing which comes to mind is that one might want to suppress
> ligature composition inside strings...

Which probably means we'd need some text property to disable
composition there.

> > Which means composition-function-table needs to be buffer-local, and
> > we should make sure making it buffer-local does TRT.
> 
> This doesn't seem to work right now.  See the FIXME at the bottom of
> below code.

We need to fix that.  We do have buffer-local char-tables, for example
buffer-display-table.  We should probably define a buffer-local
composition-function-table in the same way, and have a global table as
its parent or something (for language-specific compositions, like
those for accented letters).

Re: Emacs 28: Specific TTF font gets loaded with font-backend x instead of ftcrhb

[Tassilo Horn]

[-- Attachment #1: Type: text/plain, Size: 3166 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> I'm pretty sure I had built with HarfBuzz but tried again anyway.
>> This time it worked just fine.  I'll attach a screenshot showing two
>> emacs frames with ligatures in action.  The left one is with the
>> JetBrains Mono font, the right one is with the PragmataPro Liga font.
>
> So the conclusion is that this works with both fonts?  (I cannot be
> sure because the two displays look somewhat differently).

I'd say, yes.  There are some slight differences in which ligatures
those two fonts support, of course.

>> > Not exactly.  In Emacs built with HarfBuzz, you will see the
>> > original ASCII characters displayed, but handled as a single
>> > grapheme cluster, i.e. the cursor will be "widened" to include all
>> > of them, and a single C-f will move across all of them.
>> 
>> That doesn't seem to happen.  forward-char moves inside ligature
>> sequences no matter if the font has a ligature or not.  I.e., even
>> with a ligature ~= which gets composed to an equal sign with curvy
>> upper line point move half-by-half.
>
> I think that's because you use font-shape-gstring directly.  You
> should use compose-gstring-for-graphic instead.

Ah, then it works as you've described.  I actually like this behavior.

>> > Burt if you only care about ligatures in programming languages, the
>> > job becomes much simpler, I think.  Although I'd still expect the
>> > ligatures in effect to depend on the programming language of the
>> > current buffer.
>> 
>> Right now I've just enabled anything.
>
> Not really everything, IMO, as there are also ligatures relevant only
> to human-readable text.  For example, see this URL:
>
>   https://en.wikipedia.org/wiki/Orthographic_ligature#Ligatures_in_Unicode_(Latin_alphabets)

Yes, right.  I meant "all that I've listed" which are just those I could
copy & paste from the JetBrains Mono homepage and those are basically
just the typical programming ligatures.

>> But one thing which comes to mind is that one might want to suppress
>> ligature composition inside strings...
>
> Which probably means we'd need some text property to disable
> composition there.

Yes, something like that.

>> > Which means composition-function-table needs to be buffer-local,
>> > and we should make sure making it buffer-local does TRT.
>> 
>> This doesn't seem to work right now.  See the FIXME at the bottom of
>> below code.
>
> We need to fix that.  We do have buffer-local char-tables, for example
> buffer-display-table.  We should probably define a buffer-local
> composition-function-table in the same way, and have a global table as
> its parent or something (for language-specific compositions, like
> those for accented letters).

Sounds about right.

By the way, if I enable ligatures, I get reproducible infloops or
something alike.  The visual bell starts ringing/blinking, C-g won't
help, and even with debug-on-{error,quit} I'm unable to get into the
debugger.  I can reproduce that simply by starting Gnus and trying to
enter a group but I got that also once with pushing my commits in Magit
and sometimes when quitting emacs with C-x C-c.

Attached is a GDB backtrace.

[-- Attachment #2: gdb.log.gz --]
[-- Type: application/gzip, Size: 3642 bytes --]

[-- Attachment #3: Type: text/plain, Size: 14 bytes --]


Bye,
Tassilo

Re: Emacs 28: Specific TTF font gets loaded with font-backend x instead of ftcrhb

[Eli Zaretskii]

> From: Tassilo Horn <tsdh@gnu.org>
> Cc: help-gnu-emacs@gnu.org
> Date: Fri, 07 Feb 2020 11:41:57 +0100
> 
> By the way, if I enable ligatures, I get reproducible infloops or
> something alike.  The visual bell starts ringing/blinking, C-g won't
> help, and even with debug-on-{error,quit} I'm unable to get into the
> debugger.  I can reproduce that simply by starting Gnus and trying to
> enter a group but I got that also once with pushing my commits in Magit
> and sometimes when quitting emacs with C-x C-c.
> 
> Attached is a GDB backtrace.

Some Lisp error happens during command processing.  We need to know
which error.  Try this:

  (gdb) frame 16
  (gdb) pp data

The number 16 may need to be changed, it should be the number of this
call-stack frame:

  #16 0x000055555566d191 in cmd_error (data=XIL(0x555558f70713)) at keyboard.c:953

The command "pp" is defined in src/.gdbinit that comes with Emacs, so
if GDB says it doesn't recognize that command, do this:

  (gdb) source /path/to/emacs/src/.gdbinit

and then repeat the command.

If the above doesn't provide enough of a clue, I will need a
reproduction recipe (and please report this as a bug then).

Re: Emacs 28: Specific TTF font gets loaded with font-backend x instead of ftcrhb

[Tassilo Horn]

Eli Zaretskii <eliz@gnu.org> writes:

>> By the way, if I enable ligatures, I get reproducible infloops or
>> something alike.  The visual bell starts ringing/blinking, C-g won't
>> help, and even with debug-on-{error,quit} I'm unable to get into the
>> debugger.  I can reproduce that simply by starting Gnus and trying to
>> enter a group but I got that also once with pushing my commits in
>> Magit and sometimes when quitting emacs with C-x C-c.
>> 
>> Attached is a GDB backtrace.
>
> Some Lisp error happens during command processing.  We need to know
> which error.  Try this:
>
>   (gdb) frame 16
>   (gdb) pp data
>
> The number 16 may need to be changed, it should be the number of this
> call-stack frame:
>
>   #16 0x000055555566d191 in cmd_error (data=XIL(0x555558f70713)) at
> keyboard.c:953

(gdb) frame 16
#16 0x000055555566d16e in cmd_error (data=XIL(0x555559b53a43))
    at keyboard.c:953
953	  cmd_error_internal (data, macroerror);
(gdb) pp data
(error "Attempt to shape unibyte text")

That's signalled in composite.c's Fcomposition_get_gstring function,
either line 1738 or 1749.  I've set beakpoints there, and now I have an
emacs instance halted at 1749.

Thread 1 "emacs" hit Breakpoint 4, Fcomposition_get_gstring (
    from=<optimized out>, to=<optimized out>, font_object=XIL(0x5555562346e5), 
    string=XIL(0x555556a34ec4)) at composite.c:1749
1749		error ("Attempt to shape unibyte text");

The backtrace is this:

--8<---------------cut here---------------start------------->8---
(gdb) bt
#0  0x000055555572f850 in Fcomposition_get_gstring
    (from=<optimized out>, to=<optimized out>, font_object=XIL(0x5555562346e5), string=XIL(0x555556a34ec4)) at composite.c:1749
#1  0x000055555572fd75 in autocmp_chars
    (rule=XIL(0x555556899e35), charpos=<optimized out>, bytepos=0, limit=2, win=0x55555621c3e0, face=0x5555588fa5d0, string=XIL(0x555556a34ec4), direction=XIL(0x25b0)) at lisp.h:1147
#2  0x00005555557331ea in composition_reseat_it
    (cmp_it=0x7fffffff9340, charpos=0, bytepos=0, endpos=2, w=0x55555621c3e0, bidi_level=<optimized out>, face=0x5555588fa5d0, string=XIL(0x555556a34ec4))
    at composite.c:1222
#3  0x00005555555bcdba in next_element_from_string (it=0x7fffffff8ac0)
    at xdisp.c:8408
#4  0x00005555555bcbcd in next_element_from_string (it=0x7fffffff8ac0)
    at xdisp.c:8316
#5  0x00005555555bf267 in get_next_display_element (it=0x7fffffff8ac0)
    at xdisp.c:7518
#6  0x00005555555c50b9 in display_string
    (string=<optimized out>, lisp_string=<optimized out>, face_string=<optimized out>, face_string_pos=<optimized out>, start=<optimized out>, it=0x7fffffff8ac0, field_width=<optimized out>, precision=<optimized out>, max_x=871, multibyte=<optimized out>) at xdisp.c:26721
#7  0x00005555555c567a in display_mode_element
    (it=0x7fffffff8ac0, depth=3, field_width=0, precision=-3, elt=XIL(0x555556a3--Type <RET> for more, q to quit, c to continue without paging--c
4ec4), props=XIL(0), risky=false) at lisp.h:1541
#8  0x00005555555c6cd0 in display_mode_element (it=0x7fffffff8ac0, depth=1, field_width=0, precision=0, elt=XIL(0x555558f75af3), props=XIL(0), risky=<optimized out>) at lisp.h:1449
#9  0x00005555555c7590 in display_mode_line (w=<optimized out>, face_id=MODE_LINE_FACE_ID, format=XIL(0x555558f75b33)) at xdisp.c:25065
#10 0x00005555555c785b in display_mode_lines (w=0x55555621c3e0) at lisp.h:730
#11 0x00005555555e2894 in redisplay_window (window=XIL(0x55555621c3e5), just_this_one_p=<optimized out>) at xdisp.c:18861
#12 0x00005555555e3d77 in redisplay_window_0 (window=<optimized out>) at xdisp.c:16194
#13 0x00005555556dac06 in internal_condition_case_1 (bfun=0x5555555e3d48 <redisplay_window_0>, arg=XIL(0x55555621c3e5), handlers=<optimized out>, hfun=0x5555555a9fa8 <redisplay_window_error>) at eval.c:1379
#14 0x00005555555ad7d6 in redisplay_windows (window=XIL(0x55555621c3e5)) at xdisp.c:16174
#15 0x00005555555d1b0e in redisplay_internal () at xdisp.c:15642
#16 0x00005555555d2b5f in redisplay () at xdisp.c:14869
#17 0x0000555555673e35 in read_char (commandflag=1, map=XIL(0x555559798473), prev_event=XIL(0), used_mouse_menu=0x7fffffffe08b, end_time=0x0) at keyboard.c:2493
#18 0x0000555555677272 in read_key_sequence (keybuf=0x7fffffffe1c0, prompt=XIL(0), dont_downcase_last=<optimized out>, can_return_switch_frame=true, fix_current_buffer=true, prevent_redisplay=false) at keyboard.c:9553
#19 0x0000555555677dd5 in command_loop_1 () at keyboard.c:1350
#20 0x00005555556dab90 in internal_condition_case (bfun=0x555555677bf8 <command_loop_1>, handlers=<optimized out>, hfun=0x55555566d058 <cmd_error>) at eval.c:1355
#21 0x0000555555669db5 in command_loop_2 (ignore=<optimized out>) at lisp.h:1032
#22 0x00005555556dab07 in internal_catch (tag=<optimized out>, func=0x555555669d99 <command_loop_2>, arg=XIL(0)) at eval.c:1116
#23 0x0000555555669acb in command_loop () at keyboard.c:1070
#24 0x000055555566cd2e in recursive_edit_1 () at keyboard.c:714
#25 0x000055555566cfb1 in Frecursive_edit () at keyboard.c:786
#26 0x0000555555668cc7 in main (argc=1, argv=0x7fffffffe608) at emacs.c:2037

Lisp Backtrace:
"redisplay_internal (C function)" (0x0)
--8<---------------cut here---------------end--------------->8---

I keep this instance open for inspection (but it is easy for me to get
there again anyway).  Just tell me what frames are of interest to you
and what I should inspect there.  (I guessed that maybe frame 6 could be
of interest to get the string but that's optimized out [should I requild
-O0?], and frame 14 to get the window but "p window" just prints $5 =
XIL(0x55555621c3e5)).

Bye,
Tassilo

Re: Emacs 28: Specific TTF font gets loaded with font-backend x instead of ftcrhb

[Tassilo Horn]

Tassilo Horn <tsdh@gnu.org> writes:

> That's signalled in composite.c's Fcomposition_get_gstring function,
> either line 1738 or 1749.  I've set beakpoints there, and now I have an
> emacs instance halted at 1749.
>
> Thread 1 "emacs" hit Breakpoint 4, Fcomposition_get_gstring (
>     from=<optimized out>, to=<optimized out>, font_object=XIL(0x5555562346e5), 
>     string=XIL(0x555556a34ec4)) at composite.c:1749
> 1749		error ("Attempt to shape unibyte text");

Oh, of course I've forgotten the most obvious part, i.e., frame 0:

(gdb) pp string
"--"
(gdb) pp font_object
#<font-object "-JB  -JetBrains Mono-normal-normal-normal-*-15-*-*-*-m-0-iso10646-1">

And indeed, the mode-line in a Gnus Summary buffer starts with "U:---".

--8<---------------cut here---------------start------------->8---
mode-line-format is a variable defined in ‘C source code’.
Its value is
("%e" mode-line-front-space mode-line-mule-info mode-line-client mode-line-modified mode-line-remote mode-line-frame-identification mode-line-buffer-identification "   " mode-line-position
 (vc-mode vc-mode)
 "  " mode-line-modes mode-line-misc-info mode-line-end-spaces)
Local in buffer *Summary nnimap+Fastmail:mailinglists/help-gnu-emacs*; global value is 
("%e" mode-line-front-space mode-line-mule-info mode-line-client mode-line-modified mode-line-remote mode-line-frame-identification mode-line-buffer-identification "   " mode-line-position
 (vc-mode vc-mode)
 "  " mode-line-modes mode-line-misc-info mode-line-end-spaces)
--8<---------------cut here---------------end--------------->8---

Bye,
Tassilo

Re: Emacs 28: Specific TTF font gets loaded with font-backend x instead of ftcrhb

[Eli Zaretskii]

> From: Tassilo Horn <tsdh@gnu.org>
> Cc: help-gnu-emacs@gnu.org
> Date: Sat, 08 Feb 2020 10:52:13 +0100
> 
> Tassilo Horn <tsdh@gnu.org> writes:
> 
> > That's signalled in composite.c's Fcomposition_get_gstring function,
> > either line 1738 or 1749.  I've set beakpoints there, and now I have an
> > emacs instance halted at 1749.
> >
> > Thread 1 "emacs" hit Breakpoint 4, Fcomposition_get_gstring (
> >     from=<optimized out>, to=<optimized out>, font_object=XIL(0x5555562346e5), 
> >     string=XIL(0x555556a34ec4)) at composite.c:1749
> > 1749		error ("Attempt to shape unibyte text");
> 
> Oh, of course I've forgotten the most obvious part, i.e., frame 0:
> 
> (gdb) pp string
> "--"
> (gdb) pp font_object
> #<font-object "-JB  -JetBrains Mono-normal-normal-normal-*-15-*-*-*-m-0-iso10646-1">
> 
> And indeed, the mode-line in a Gnus Summary buffer starts with "U:---".

We should not signal this error when the unibyte string is
pure-ASCII.  Let me think of a solution.

Re: Emacs 28: Specific TTF font gets loaded with font-backend x instead of ftcrhb

[Eli Zaretskii]

> Date: Sat, 08 Feb 2020 12:16:48 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > > Thread 1 "emacs" hit Breakpoint 4, Fcomposition_get_gstring (
> > >     from=<optimized out>, to=<optimized out>, font_object=XIL(0x5555562346e5), 
> > >     string=XIL(0x555556a34ec4)) at composite.c:1749
> > > 1749		error ("Attempt to shape unibyte text");
> > 
> > Oh, of course I've forgotten the most obvious part, i.e., frame 0:
> > 
> > (gdb) pp string
> > "--"
> > (gdb) pp font_object
> > #<font-object "-JB  -JetBrains Mono-normal-normal-normal-*-15-*-*-*-m-0-iso10646-1">
> > 
> > And indeed, the mode-line in a Gnus Summary buffer starts with "U:---".
> 
> We should not signal this error when the unibyte string is
> pure-ASCII.  Let me think of a solution.

Does the below help?  If it does, you should see the "--" in the mode
line replaced with the ligature (whether this is the desired behavior
is another question).

diff --git a/src/composite.c b/src/composite.c
index 53e6930..05365cf 100644
--- a/src/composite.c
+++ b/src/composite.c
@@ -1746,7 +1746,18 @@ Otherwise (for terminal display), FONT-OBJECT must be a terminal ID, a
       CHECK_STRING (string);
       validate_subarray (string, from, to, SCHARS (string), &frompos, &topos);
       if (! STRING_MULTIBYTE (string))
-	error ("Attempt to shape unibyte text");
+	{
+	  ptrdiff_t i;
+
+	  for (i = SBYTES (string) - 1; i >= 0; i--)
+	    if (!ASCII_CHAR_P (SREF (string, i)))
+	      error ("Attempt to shape unibyte text");
+	  /* STRING is a pure-ASCII string, so we can convert it (or,
+	     rather, its copy) to multibyte and use that thereafter.  */
+	  Lisp_Object string_copy = Fconcat (1, &string);
+	  STRING_SET_MULTIBYTE (string_copy);
+	  string = string_copy;
+	}
       frombyte = string_char_to_byte (string, frompos);
     }
 

Re: Emacs 28: Specific TTF font gets loaded with font-backend x instead of ftcrhb

[Tassilo Horn]

Eli Zaretskii <eliz@gnu.org> writes:

>> We should not signal this error when the unibyte string is
>> pure-ASCII.  Let me think of a solution.
>
> Does the below help?

Yes, it does help.  The error is gone.

> If it does, you should see the "--" in the mode line replaced with the
> ligature (whether this is the desired behavior is another question).

Exactly, and here I'd say it is definitively not the right behavior
because because both chars have a separate meaning (buffer writable?
and buffer modified?) and help-echo which aren't accessible anymore.

I don't know how to cope with such situations.  Maybe composition should
only consider strings with equal text properties.  Or, as you already
said, there should be some text property inhibiting composition which
the mode-line could use.

Bye,
Tassilo

Re: Emacs 28: Specific TTF font gets loaded with font-backend x instead of ftcrhb

[Eli Zaretskii]

> From: Tassilo Horn <tsdh@gnu.org>
> Cc: help-gnu-emacs@gnu.org
> Date: Sat, 08 Feb 2020 13:29:47 +0100
> 
> >> We should not signal this error when the unibyte string is
> >> pure-ASCII.  Let me think of a solution.
> >
> > Does the below help?
> 
> Yes, it does help.  The error is gone.

Thanks, I installed it on master, since even though such compositions
are undesirable, they are better then an infinite sequence of errors
during redisplay.

> > If it does, you should see the "--" in the mode line replaced with the
> > ligature (whether this is the desired behavior is another question).
> 
> Exactly, and here I'd say it is definitively not the right behavior
> because because both chars have a separate meaning (buffer writable?
> and buffer modified?) and help-echo which aren't accessible anymore.
> 
> I don't know how to cope with such situations.

We could insert a ZWNJ character between the two characters, but that
is not a general solution, since mode-line strings can be dynamically
consed by applications, and at least the input-method indicator on the
mode line must be able to display composed characters.  I guess this
is something else we should figure out before we can freely use
ligatures in Emacs.

> Maybe composition should only consider strings with equal text
> properties.

That already happens, at least with face properties.

> Or, as you already said, there should be some text property
> inhibiting composition which the mode-line could use.

Yes, maybe.  Although that, too, could have problems with mode-line
strings produced by Lisp programs.