Suggest installing more fonts?

Suggest installing more fonts?

[Lars Ingebrigtsen]

This is just a stray thought:

I think, over the years, the most common question users have that has an
easy-to-fix solution is: "Why is Emacs displaying boxes for some of
these characters I'm seeing?"  The solution is "install some more
fonts", of course.

But could Emacs be more helpful here?  That is, could we somehow,
unobtrusively, tell the users this when Emacs is trying to display a
character that it has no fonts for?

For added helpfulness, would it be possible for Emacs to be even more
specific?  Like, say "sudo apt install fonts-noto-color-emoji
fonts-symbola", or whatever, depending on the system.

Fonts change over the years, so this would be an added maintenance
burden...  but they don't change a lot: New general-use fonts with good
coverage aren't created very often.

(Perhaps distribution packagers could also push more fonts as part of
the Emacs installation, but that'd probably be more controversial.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Suggest installing more fonts?

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Fri, 16 Oct 2020 12:54:13 +0200
> 
> I think, over the years, the most common question users have that has an
> easy-to-fix solution is: "Why is Emacs displaying boxes for some of
> these characters I'm seeing?"  The solution is "install some more
> fonts", of course.
> 
> But could Emacs be more helpful here?  That is, could we somehow,
> unobtrusively, tell the users this when Emacs is trying to display a
> character that it has no fonts for?

That box with the hex code is our way of telling the users that.  What
else can be better, given that there could be more than one such
character?

> For added helpfulness, would it be possible for Emacs to be even more
> specific?  Like, say "sudo apt install fonts-noto-color-emoji
> fonts-symbola", or whatever, depending on the system.

For that, you'd need a database of fonts per OS and per release (and I
think on GNU systems this will have to depend on the distro as well?),
and you'd need to maintain that database.  Who around here knows
enough about fonts on different systems to maintain such a database?

> Fonts change over the years, so this would be an added maintenance
> burden...  but they don't change a lot: New general-use fonts with good
> coverage aren't created very often.

IME, the fonts do change quite a lot between releases.

> (Perhaps distribution packagers could also push more fonts as part of
> the Emacs installation, but that'd probably be more controversial.)

Yes, it's the responsibility of the distro to make sure the necessary
fonts are installed when Emacs is installed.  If MS-Windows can have
all the fonts to cover the supported scripts, so can GNU/Linux
distros.  They should make sure each Emacs user has fonts to cover all
the scripts known to Emacs.

I think the best we can do here is (a) add more fonts to the default
fontset (it isn't trivial, as quite a few good fonts aren't free); and
(b) have a command to report which of the fonts mentioned in the
default fontset aren't available.

Re: Suggest installing more fonts?

[Gregory Heytings via Emacs development discussions.]

>> But could Emacs be more helpful here?  That is, could we somehow, 
>> unobtrusively, tell the users this when Emacs is trying to display a 
>> character that it has no fonts for?
>
> That box with the hex code is our way of telling the users that.  What 
> else can be better, given that there could be more than one such 
> character?
>

A suggestion: would it not be better to include a version of GNU Unifont 
in Emacs?  Instead of a box with a hex code, users would see what the 
character is.  It is bitmapped, so users would at the same time see that 
the character is missing in their installed fonts.

Re: Suggest installing more fonts?

[Eli Zaretskii]

> Date: Fri, 16 Oct 2020 11:30:40 +0000
> From: Gregory Heytings <ghe@sdf.org>
> cc: Lars Ingebrigtsen <larsi@gnus.org>, emacs-devel@gnu.org
> 
> A suggestion: would it not be better to include a version of GNU Unifont 
> in Emacs?

Including a font with Emacs is problematic, since installing a font
requires system-wide actions.

(Personally, I think Unifont is an ugly font, with many character
glyphs completely illegible, and distributing is with Emacs will only
cause us look unprofessional.  But that's my personal view.)

Re: Suggest installing more fonts?

[Gregory Heytings via Emacs development discussions.]

>> A suggestion: would it not be better to include a version of GNU 
>> Unifont in Emacs?
>
> Including a font with Emacs is problematic, since installing a font 
> requires system-wide actions.
>

PDF readers use the fonts embedded in PDF files, so I would guess that 
Emacs could use a built-in font (without installing it).

>
> (Personally, I think Unifont is an ugly font, with many character glyphs 
> completely illegible, and distributing is with Emacs will only cause us 
> look unprofessional.  But that's my personal view.)
>

It is an ugly font indeed, the point is only to use it in place of the 
(even more ugly) boxes with a hex code.  I think it is ugly "by design", 
it aims at a very broad coverage without becoming too large.

Re: Suggest installing more fonts?

[Eli Zaretskii]

> Date: Fri, 16 Oct 2020 12:59:47 +0000
> From: Gregory Heytings <ghe@sdf.org>
> cc: larsi@gnus.org, emacs-devel@gnu.org
> 
> >> A suggestion: would it not be better to include a version of GNU 
> >> Unifont in Emacs?
> >
> > Including a font with Emacs is problematic, since installing a font 
> > requires system-wide actions.
> 
> PDF readers use the fonts embedded in PDF files, so I would guess that 
> Emacs could use a built-in font (without installing it).

This is not currently supported, AFAIK: Emacs uses the likes of
Fontconfig facilities to search for suitable fonts given a character.

> > (Personally, I think Unifont is an ugly font, with many character glyphs 
> > completely illegible, and distributing is with Emacs will only cause us 
> > look unprofessional.  But that's my personal view.)
> 
> It is an ugly font indeed, the point is only to use it in place of the 
> (even more ugly) boxes with a hex code.  I think it is ugly "by design", 
> it aims at a very broad coverage without becoming too large.

You know it and I know it, but most users won't.  They will just see
the ugly glyphs.  Whether that is less or more ugly than the "tofu" we
show is in the eyes of the beholder; my personal opinion is above.

Re: Suggest installing more fonts?

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> For added helpfulness, would it be possible for Emacs to be even more
>> specific?  Like, say "sudo apt install fonts-noto-color-emoji
>> fonts-symbola", or whatever, depending on the system.
>
> For that, you'd need a database of fonts per OS and per release (and I
> think on GNU systems this will have to depend on the distro as well?),
> and you'd need to maintain that database.  Who around here knows
> enough about fonts on different systems to maintain such a database?

To have perfect coverage, we'd need to do a lot of work.  But we don't
have to have that, so we don't need to do that much work.

Having a list of nice fonts with good coverage for, say, the five most
significant operating systems, should be a list of less than thirty
font packages.

>> Fonts change over the years, so this would be an added maintenance
>> burden...  but they don't change a lot: New general-use fonts with good
>> coverage aren't created very often.
>
> IME, the fonts do change quite a lot between releases.

The fonts change, but the number of fonts we care about doesn't.

> I think the best we can do here is (a) add more fonts to the default
> fontset (it isn't trivial, as quite a few good fonts aren't free); and
> (b) have a command to report which of the fonts mentioned in the
> default fontset aren't available.

There's still the issue of letting them know what the issue is.
Clearly, showing them tofu isn't enough, because the issue comes up
again and again.

Popping up a warning buffer (at the first time tofu is displayed per
session) may be OK, but could be annoying?  However, our warning buffers
now have buttons to easily disable the warning permanently, so it might
be OK.

Displaying a message in the echo area (again, only the first time we
tofu in a session) saying something like "Unable to display character;
type `C-h TO-BE-DETERMINED' for more information" might also be an
option.

Or...  adding help echo text props to tofu?  We give non-character bytes
in a buffer a special font, so that would be kinda analogous.

I think there's probably plenty of opportunity here to be more helpful
to the users.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Suggest installing more fonts?

[Lars Ingebrigtsen]

Gregory Heytings <ghe@sdf.org> writes:

> A suggestion: would it not be better to include a version of GNU
> Unifont in Emacs? 

The Unifont is so bad that I think that's even worse than showing tofu.
:-/

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Suggest installing more fonts?

[Rudolf Schlatte]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> I think the best we can do here is (a) add more fonts to the default
>> fontset (it isn't trivial, as quite a few good fonts aren't free); and
>> (b) have a command to report which of the fonts mentioned in the
>> default fontset aren't available.
>
> There's still the issue of letting them know what the issue is.
> Clearly, showing them tofu isn't enough, because the issue comes up
> again and again.
>
> Popping up a warning buffer (at the first time tofu is displayed per
> session) may be OK, but could be annoying?  However, our warning buffers
> now have buttons to easily disable the warning permanently, so it might
> be OK.
>
> Displaying a message in the echo area (again, only the first time we
> tofu in a session) saying something like "Unable to display character;
> type `C-h TO-BE-DETERMINED' for more information" might also be an
> option.
>
> Or...  adding help echo text props to tofu?  We give non-character bytes
> in a buffer a special font, so that would be kinda analogous.

Adding information about which font carries the missing glyph to the
screen displayed with ‘C-u C-x =’ would be a possibility.

Rudi

Re: Suggest installing more fonts?

[Gregory Heytings via Emacs development discussions.]

>> A suggestion: would it not be better to include a version of GNU 
>> Unifont in Emacs?
>
> The Unifont is so bad that I think that's even worse than showing tofu. 
> :-/
>

Okay, you and Eli both think that it's "just too ugly".

Perhaps there is another font that covers (at least) the complete BMP that 
could be included in Emacs?

About "showing tofu" vs "ugly font", I think that giving users a clear 
indication about the language is useful.  Something with which they can 
see for example "it's chinese, so if I want to see this I need to install 
a font with chinese characters".

Re: Suggest installing more fonts?

[Gregory Heytings via Emacs development discussions.]

>>> (Personally, I think Unifont is an ugly font, with many character 
>>> glyphs completely illegible, and distributing is with Emacs will only 
>>> cause us look unprofessional.  But that's my personal view.)
>>
>> It is an ugly font indeed, the point is only to use it in place of the 
>> (even more ugly) boxes with a hex code.  I think it is ugly "by 
>> design", it aims at a very broad coverage without becoming too large.
>
> You know it and I know it, but most users won't.  They will just see the 
> ugly glyphs.  Whether that is less or more ugly than the "tofu" we show 
> is in the eyes of the beholder; my personal opinion is above.
>

Another thought on this: to avoid "looking unprofessional", a help-echo 
property could be added on these characters, saying something like 
"Warning: no font installed for this character".  So the user would see an 
ugly (but often recognizable) character, and at the same time would be 
warned that they need to do something to see it better.

Re: Suggest installing more fonts?

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Fri, 16 Oct 2020 15:24:00 +0200
> 
> > For that, you'd need a database of fonts per OS and per release (and I
> > think on GNU systems this will have to depend on the distro as well?),
> > and you'd need to maintain that database.  Who around here knows
> > enough about fonts on different systems to maintain such a database?
> 
> To have perfect coverage, we'd need to do a lot of work.  But we don't
> have to have that, so we don't need to do that much work.
> 
> Having a list of nice fonts with good coverage for, say, the five most
> significant operating systems, should be a list of less than thirty
> font packages.

I doubt it's as few as 30, assuming we want to cover every living
script and some popular dead ones.  But the only way to be sure how
many is to do the work and come up with the list of such fonts.  then
we will see.

> >> Fonts change over the years, so this would be an added maintenance
> >> burden...  but they don't change a lot: New general-use fonts with good
> >> coverage aren't created very often.
> >
> > IME, the fonts do change quite a lot between releases.
> 
> The fonts change, but the number of fonts we care about doesn't.

Since the fonts change, we will need to change the list.

> Popping up a warning buffer (at the first time tofu is displayed per
> session) may be OK

That's not doable, because you cannot pop up anything in the middle of
redisplay.  We could perhaps use the delayed-warnings mechanism, but
is that sufficient?

> Displaying a message in the echo area (again, only the first time we
> tofu in a session) saying something like "Unable to display character;
> type `C-h TO-BE-DETERMINED' for more information" might also be an
> option.

Not from redisplay, we cannot do that.

> Or...  adding help echo text props to tofu?

To each one of them?

Re: Suggest installing more fonts?

[Eli Zaretskii]

> Date: Fri, 16 Oct 2020 14:08:40 +0000
> From: Gregory Heytings <ghe@sdf.org>
> cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> 
> Perhaps there is another font that covers (at least) the complete BMP that 
> could be included in Emacs?

The BMP is not enough these days.  E.g., many Emoji is outside of it,
as are many newly introduced scripts.

Unifont supports much more than the BMP.

It is unusual for a single font to support many different scripts,
certainly not the amount that Unifont supports.

Re: Suggest installing more fonts?

[Eli Zaretskii]

> Date: Fri, 16 Oct 2020 14:38:04 +0000
> From: Gregory Heytings <ghe@sdf.org>
> cc: larsi@gnus.org, emacs-devel@gnu.org
> 
> Another thought on this: to avoid "looking unprofessional", a help-echo 
> property could be added on these characters, saying something like 
> "Warning: no font installed for this character".  So the user would see an 
> ugly (but often recognizable) character, and at the same time would be 
> warned that they need to do something to see it better.

Yes, Lars suggested that as well.  But we have to be careful: too many
such properties or overlays will slow down redisplay.  Also, we'd need
to see how to do this technically, since characters that have no fonts
are discovered during redisplay.  Perhaps some jit-lock registered
function could detect them after redisplay, and put the properties on
some of them (which ones?).

Re: Suggest installing more fonts?

[Gregory Heytings via Emacs development discussions.]

>> Perhaps there is another font that covers (at least) the complete BMP 
>> that could be included in Emacs?
>
> The BMP is not enough these days.  E.g., many Emoji is outside of it, as 
> are many newly introduced scripts.
>
> Unifont supports much more than the BMP.
>

Yes, I know: I wrote "at least" ;-)

>
> It is unusual for a single font to support many different scripts, 
> certainly not the amount that Unifont supports.
>

Hence the idea of Unifont + help-echo warning.  The users see something 
they can decypher, and are warned that they need to do something to see it 
better.

Re: Suggest installing more fonts?

[Gregory Heytings via Emacs development discussions.]

>> Another thought on this: to avoid "looking unprofessional", a help-echo 
>> property could be added on these characters, saying something like 
>> "Warning: no font installed for this character".  So the user would see 
>> an ugly (but often recognizable) character, and at the same time would 
>> be warned that they need to do something to see it better.
>
> Yes, Lars suggested that as well.  But we have to be careful: too many 
> such properties or overlays will slow down redisplay.  Also, we'd need 
> to see how to do this technically, since characters that have no fonts 
> are discovered during redisplay.  Perhaps some jit-lock registered 
> function could detect them after redisplay, and put the properties on 
> some of them (which ones?).
>

I think I see what you mean: characters that have no fonts are discovered 
during redisplay, and at that moment it's too late to add text properties, 
which are not supposed to change anymore.  It might be possible to add a 
special help-echo like property that would be added at the moment these 
characters are discovered, for example in produce_glyphless_glyph().

I don't think there would be "too many" such properties, it seems safe to 
assume that in general such characters follow each other and form a 
"block", and a single text property would be enough for one such block.

Re: Suggest installing more fonts?

[Stefan Monnier]

> I think, over the years, the most common question users have that has an
> easy-to-fix solution is: "Why is Emacs displaying boxes for some of
> these characters I'm seeing?"  The solution is "install some more
> fonts", of course.

[ FWIW, I don't think replacing those tofu chars with some "ugly
  default" is a good option: it's more work with very little benefit if
  any, since the problem still remains largely unchanged.  ]

I think to solve this problem we need to have a clear idea of what the
users do when they're faced with this:

- Does the problem only affect Emacs and not other applications?
  If so why?  If not, then what do other applications do about it?
- Do users immediately run to search the internet for answers without
  even trying to interact any further with their Emacs instance to
  investigate the problem first?
- If they do interact with Emacs, what do they do?  Do they move the
  mouse over the tofu?  Do they look for some help some other way?

The potential elements of solutions can be:
A- Make the users understand that the problem is a lack of fonts
B- Steer the users to the appropriate fonts
C- Fix the problem that makes Emacs fail to find the fonts when other
   applications deal with it just fine.

We could arrange for some kind of tooltip to show up when you wave the
mouse over those tofu chars, but that won't help if the users don't pass
the mouse over them.

We could have the redisplay code record the tofu chars and their
location in some global var, and then use something like
`post-command-hook` to popup a help/warning buffer explaining the
problem and its potential solutions.
That could be annoying for experienced users, tho, so it should come
with a "don't warn me again" button.


        Stefan

Re: Suggest installing more fonts?

[Eli Zaretskii]

> Date: Fri, 16 Oct 2020 16:09:35 +0000
> From: Gregory Heytings <ghe@sdf.org>
> cc: larsi@gnus.org, emacs-devel@gnu.org
> 
> I think I see what you mean: characters that have no fonts are discovered 
> during redisplay, and at that moment it's too late to add text properties, 
> which are not supposed to change anymore.  It might be possible to add a 
> special help-echo like property that would be added at the moment these 
> characters are discovered, for example in produce_glyphless_glyph().

I don't recommend this, it is tricky and might cause problems.

Instead, it is much easier to have a feature implemented in Lisp that
would detect characters with no font glyphs and put help-echo on them.
We have infrastructure for such features in jit-lock.el.

> I don't think there would be "too many" such properties, it seems safe to 
> assume that in general such characters follow each other and form a 
> "block", and a single text property would be enough for one such block.

It is okay to assume, but I'd still prefer that we also check that the
assumption holds, and if so, do something to keep the number of
properties in check.

Re: Suggest installing more fonts?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Fri, 16 Oct 2020 14:48:00 -0400
> Cc: emacs-devel@gnu.org
> 
> - Does the problem only affect Emacs and not other applications?
>   If so why?

AFAIK, "most other applications", at least those I could investigate,
come with fixed lists of fonts to use for each script.  These lists
are maintained for each OS and are adjusted as the OS version and the
fonts change.  This method has its advantages and also its
disadvantages; I think both are quite clear.

> - Do users immediately run to search the internet for answers without
>   even trying to interact any further with their Emacs instance to
>   investigate the problem first?
> - If they do interact with Emacs, what do they do?  Do they move the
>   mouse over the tofu?  Do they look for some help some other way?

The rational thing to do is to use "C-u C-x =", see the script of the
problematic character(s), and then find and install some font that
covers those characters.

Re: Suggest installing more fonts?

[Stefan Monnier]

>> - Does the problem only affect Emacs and not other applications?
>>   If so why?
> AFAIK, "most other applications", at least those I could investigate,
> come with fixed lists of fonts to use for each script.  These lists
> are maintained for each OS and are adjusted as the OS version and the
> fonts change.  This method has its advantages and also its
> disadvantages; I think both are quite clear.

But then it means that "installing more fonts" is not really the solution
that the user needs to follow, right?

>> - Do users immediately run to search the internet for answers without
>>   even trying to interact any further with their Emacs instance to
>>   investigate the problem first?
>> - If they do interact with Emacs, what do they do?  Do they move the
>>   mouse over the tofu?  Do they look for some help some other way?
> The rational thing to do is to use "C-u C-x =", see the script of the
> problematic character(s), and then find and install some font that
> covers those characters.

But from what you said above it's not just a question of installing
fonts (since what you wrote before suggests that the same problem
doesn't affect other applications on the same system).

Also I suspect that this requires non-trivial knowledge:
- `C-u C-x =`, which most beginning Emacs users won't know.
- The notion of "script" and how to find it out.
- How to know which font covers which script.
- Potentially how to tell Emacs which font to use for which script.


        Stefan

Re: Suggest installing more fonts?

[Stefan Monnier]

> Instead, it is much easier to have a feature implemented in Lisp that
> would detect characters with no font glyphs and put help-echo on them.
> We have infrastructure for such features in jit-lock.el.

Indeed.  And we can make this very efficient if the redisplay stashes
in some Lisp var the chars-with-locations of the tofu it generates.


        Stefan

Re: Suggest installing more fonts?

[Gregory Heytings via Emacs development discussions.]

[-- Attachment #1: Type: text/plain, Size: 1633 bytes --]


>> I think, over the years, the most common question users have that has 
>> an easy-to-fix solution is: "Why is Emacs displaying boxes for some of 
>> these characters I'm seeing?"  The solution is "install some more 
>> fonts", of course.
>
> [ FWIW, I don't think replacing those tofu chars with some "ugly 
> default" is a good option: it's more work with very little benefit if 
> any, since the problem still remains largely unchanged. ]
>

FWIW, I do not agree with this.  It is much better from a end-user 
viewpoint to see something they can decypher, even if it has a non-optimal 
rendering, than to see a hex code in a box.  For example, I use the DejaVu 
Sans font.  I just checked, and among the twelve "smiling face" emojis, 
three are missing, for example "smiling face with smiling eyes and three 
hearts".  I would find it much better to have Emacs displaying the 
attached bitmap (taken from the Unicode font) instead of a box with 
"01F970" (for the default value of glyphless-char-display).  At least I 
see what the character is.

[I know that an experienced user can do M-x describe-char or C-u C-x =, or 
(setq what-cursor-show-names t) and type C-x =.  But that's for 
experienced users, and in any case much less user-friendly than a bitmap.]

>
> - Does the problem only affect Emacs and not other applications? If so 
> why?  If not, then what do other applications do about it?
>

This problem affects all applications.  If happens whenever an application 
tries to display a character for which the font in use has no glyph. 
What they do varies, most display a question mark in a box, or an empty 
box.

[-- Attachment #2: Type: image/png, Size: 113 bytes --]

Re: Suggest installing more fonts?

[Stefan Monnier]

>> [ FWIW, I don't think replacing those tofu chars with some "ugly default"
>> is a good option: it's more work with very little benefit if any, since
>> the problem still remains largely unchanged. ]
> FWIW, I do not agree with this.

I knid of guessed ;-)

> It is much better from a end-user viewpoint to see something they can
> decypher, even if it has a non-optimal rendering, than to see a hex
> code in a box.

[ What you describe is the "very little benefit" I referred to.  ]

I think most users won't be happy with the result in both cases,
so they'll still want to find a fix for the problem [ This is "the
problem still remains largely unchanged" that I referred to.  ]

>> - Does the problem only affect Emacs and not other applications? If so
>> why?  If not, then what do other applications do about it?
> This problem affects all applications.  If happens whenever an application
> tries to display a character for which the font in use has no glyph. What
> they do varies, most display a question mark in a box, or an empty box.

I have seen several questions about Emacs showing tofu where the user
points out that such-and-such other application displays that same text
just fine on the same system.

So, I know that it sometimes "only affects Emacs".


        Stefan

Re: Suggest installing more fonts?

[Gregory Heytings via Emacs development discussions.]

[-- Attachment #1: Type: text/plain, Size: 3323 bytes --]


>>> [ FWIW, I don't think replacing those tofu chars with some "ugly 
>>> default" is a good option: it's more work with very little benefit if 
>>> any, since the problem still remains largely unchanged. ]
>>
>> FWIW, I do not agree with this.
>
> I knid of guessed ;-)
>

Life would be boring with quick agreements ;-)

>> It is much better from a end-user viewpoint to see something they can 
>> decypher, even if it has a non-optimal rendering, than to see a hex 
>> code in a box.
>
> [ What you describe is the "very little benefit" I referred to.  ]
>

I don't understand why you consider this as a "very little benefit".  The 
Unifont is not unreadable, it is certainly not a nice modern font with 
antialiasing etc. , because it is on purpose as small as possible (8x16 or 
16x16 black and white bitmaps), but it is useable.

>
> I think most users won't be happy with the result in both cases, so 
> they'll still want to find a fix for the problem [ This is "the problem 
> still remains largely unchanged" that I referred to. ]
>

Here I agree with you: the problem remains largely unchanged, but at least 
Emacs would have done its best (at a reasonable cost) to help the user. 
And Emacs would do better than most other editors, which just show an 
empty box or a question mark in that case.

>>> - Does the problem only affect Emacs and not other applications? If so 
>>> why?  If not, then what do other applications do about it?
>>
>> This problem affects all applications.  If happens whenever an 
>> application tries to display a character for which the font in use has 
>> no glyph. What they do varies, most display a question mark in a box, 
>> or an empty box.
>
> I have seen several questions about Emacs showing tofu where the user 
> points out that such-and-such other application displays that same text 
> just fine on the same system.
>

This depends on many parameters, most of which are outside Emacs' control. 
And you will probably not see questions about the opposite situation, 
where other applications show tofus when Emacs displays the text just fine 
;-)  So I'll give you two examples (and I'm sure there are many, many 
more):

- On my computer, with a recent version of Chromium, the sentence "Can we 
see the full text of your speech before Tuesday?" 
(我们能在周二前看到你演讲的全文吗？) on 
https://dictionary.cambridge.org/dictionary/english-chinese-simplified/text 
contains three tofus (们讲吗), which Chromium displays as empty boxes. 
The same three tofus appear on Firefox, but Firefox displays them with 
four hex digits in a box.  Emacs displays that sentence fine, both in a 
GUI because I happen to have a certain "-isas-song 
ti-medium-r-normal--24-240-72-72-c-240-gb2312.1980-0" font installed, and 
in a terminal because my default font is DejaVu Sans, which contains all 
these characters.

- On my computer, two of the three missing "smiling face" emojis in DejaVu 
Sans are displayed fine by VS Code, because I happen to have the 
"fonts-noto-color-emoji" Debian package installed.  But the third one 
(0x1f972) is displayed as a tofu (an empty box) by VS Code, even with that 
package installed.  If I remove that package, VS Code displays these three 
emojis as tofus, as Emacs does.

Re: Suggest installing more fonts?

[Stefan Monnier]

>> I have seen several questions about Emacs showing tofu where the user
>> points out that such-and-such other application displays that same text
>> just fine on the same system.
> This depends on many parameters, most of which are outside Emacs' control.

I know.  I was just pointing out that there are several problems
(missing fonts is one of them, but not the only one), so before trying
to solve "the problem", we should be clear about which one it is and try
and figure out whether it's indeed the most common.


        Stefan

Re: Suggest installing more fonts?

[Gregory Heytings via Emacs development discussions.]

>>> I have seen several questions about Emacs showing tofu where the user
>>> points out that such-and-such other application displays that same text
>>> just fine on the same system.
>>
>> This depends on many parameters, most of which are outside Emacs' 
>> control.
>
> I know.  I was just pointing out that there are several problems 
> (missing fonts is one of them, but not the only one), so before trying 
> to solve "the problem", we should be clear about which one it is and try 
> and figure out whether it's indeed the most common.
>

A user has (usually) no idea of these "several problems", and knows even 
less how to fix them.  For such users the problem is simply "Emacs 
displays gibberish".  That is "the problem" IMO.  Everything else are 
technicalities that very few know/understand/have the patience to deal 
with.

So IMO for these users it would be much better to display characters with 
a low-resolution font, together with a warning (and perhaps a pointer to a 
short guide) (as a help-echo and/or in the echo area), instead of a tofu.

Re: Suggest installing more fonts?

[Lars Ingebrigtsen]

Rudolf Schlatte <rudi@constantly.at> writes:

> Adding information about which font carries the missing glyph to the
> screen displayed with ‘C-u C-x =’ would be a possibility.

Yes, that's a good idea.

If we want to warn users using `message' (only once per session), the
message could then be "Couldn't find font, use `C-u C-x =' on the
character for more information" or something.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Suggest installing more fonts?

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> I doubt it's as few as 30, assuming we want to cover every living
> script and some popular dead ones.

I don't think we do, really.  Listing the major fonts in a help text
like this would be sufficient.

>> Popping up a warning buffer (at the first time tofu is displayed per
>> session) may be OK
>
> That's not doable, because you cannot pop up anything in the middle of
> redisplay.  We could perhaps use the delayed-warnings mechanism, but
> is that sufficient?

I think so.

>> Displaying a message in the echo area (again, only the first time we
>> tofu in a session) saying something like "Unable to display character;
>> type `C-h TO-BE-DETERMINED' for more information" might also be an
>> option.
>
> Not from redisplay, we cannot do that.

Not directly, but we could add a mechanism for making Emacs do that.
I.e., as with delayed-warnings, we could add a list of actions to be
performed after redisplay is over...  I.e., generalise delayed-warnings
somewhat. 

>> Or...  adding help echo text props to tofu?
>
> To each one of them?

That's probably overkill.  Say, the first 100 of those in a buffer?
Only the first?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Suggest installing more fonts?

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> C- Fix the problem that makes Emacs fail to find the fonts when other
>    applications deal with it just fine.

I agree with everything in this post, but just one comment here:

There's two philosophies here, depending on the "class" of the
application.  The applications that are primarily concerned with
displaying documents (Chrome, Firefox, Libreoffice) put a lot of work
into this, and both have giant lists of fonts preinstalled, and are
distributed with (some) fonts.

Other apps, that are more bare-bones (like Terminal) just look at the
default font and then tufo if the font doesn't have that glyph.

Emacs is somewhere in the middle here, as usual.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Suggest installing more fonts?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: larsi@gnus.org,  emacs-devel@gnu.org
> Date: Fri, 16 Oct 2020 15:37:40 -0400
> 
> >> - Does the problem only affect Emacs and not other applications?
> >>   If so why?
> > AFAIK, "most other applications", at least those I could investigate,
> > come with fixed lists of fonts to use for each script.  These lists
> > are maintained for each OS and are adjusted as the OS version and the
> > fonts change.  This method has its advantages and also its
> > disadvantages; I think both are quite clear.
> 
> But then it means that "installing more fonts" is not really the solution
> that the user needs to follow, right?

I'm not sure how you reached this conclusion.  Are you talking about
Emacs or are you talking about other applications?

If an application expects to see some font that is not installed, the
user should go out and install it, otherwise the characters will not
be shown.  Happened to me with applications other than Emacs a few
times.

The main difference between what Emacs does and what the other
applications do is that the other applications specify proprietary
fonts usually available on the target system, while we don't.  the
disadvantage of what we do is that Emacs might find a font that is not
the best one for supporting a given block of Unicode codepoints.

> >> - Do users immediately run to search the internet for answers without
> >>   even trying to interact any further with their Emacs instance to
> >>   investigate the problem first?
> >> - If they do interact with Emacs, what do they do?  Do they move the
> >>   mouse over the tofu?  Do they look for some help some other way?
> > The rational thing to do is to use "C-u C-x =", see the script of the
> > problematic character(s), and then find and install some font that
> > covers those characters.
> 
> But from what you said above it's not just a question of installing
> fonts (since what you wrote before suggests that the same problem
> doesn't affect other applications on the same system).

See above.

> Also I suspect that this requires non-trivial knowledge:
> - `C-u C-x =`, which most beginning Emacs users won't know.
> - The notion of "script" and how to find it out.
> - How to know which font covers which script.
> - Potentially how to tell Emacs which font to use for which script.

Which is why we are having this conversation.

The idea behind what Emacs does is that the system is already
configured to display the script that the user wants to use in Emacs.
Where this is not true -- and I'm not sure I have a clear picture of
why this could happen -- what we do is insufficient.

Btw, one problem I think makes this issue more severe than it has to
be is that there are many fontset customizations floating out there
which people copy into their init files without understanding what
they do, and those customizations are either dangerous or plain
incorrect, and get users into trouble wrt fonts.  Another related
problem is that people install all kinds of fonts that could claim
support for some script, but in fact don't have a decent support for
it.  These two issues probably call for a "M-x fontset-analyze"
command, if we have anyone aboard capable of writing it.

Re: Suggest installing more fonts?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Gregory Heytings <ghe@sdf.org>,  larsi@gnus.org,  emacs-devel@gnu.org
> Date: Fri, 16 Oct 2020 15:40:11 -0400
> 
> > Instead, it is much easier to have a feature implemented in Lisp that
> > would detect characters with no font glyphs and put help-echo on them.
> > We have infrastructure for such features in jit-lock.el.
> 
> Indeed.  And we can make this very efficient if the redisplay stashes
> in some Lisp var the chars-with-locations of the tofu it generates.

I think relying on such a trace variable would be unreliable, because
you can never predict which parts of the window will be actually
redisplayed, and the display engine can only leave the trace about
those parts where it tried to find a font glyph for a character.

Emacs already knows not to search the available fonts for a character
that it already tried to display and failed to find a suitable font.
We should probably reuse the information which says a character
doesn't have a font for finding such characters and marking them.  It
cannot be too expensive, I think.

Re: Suggest installing more fonts?

[Eli Zaretskii]

> Date: Fri, 16 Oct 2020 23:02:37 +0000
> cc: Lars Ingebrigtsen <larsi@gnus.org>, emacs-devel@gnu.org
> From: Gregory Heytings via "Emacs development discussions." <emacs-devel@gnu.org>
> 
> > I know.  I was just pointing out that there are several problems 
> > (missing fonts is one of them, but not the only one), so before trying 
> > to solve "the problem", we should be clear about which one it is and try 
> > and figure out whether it's indeed the most common.
> 
> A user has (usually) no idea of these "several problems", and knows even 
> less how to fix them.  For such users the problem is simply "Emacs 
> displays gibberish".  That is "the problem" IMO.  Everything else are 
> technicalities that very few know/understand/have the patience to deal 
> with.
> 
> So IMO for these users it would be much better to display characters with 
> a low-resolution font, together with a warning (and perhaps a pointer to a 
> short guide) (as a help-echo and/or in the echo area), instead of a tofu.

Then the problem will become "Emacs uses an ugly font where other apps
don't", and we are none the wiser.

IMO, we should try fixing whatever problems are there (and AFAIU
there's more than one) without using ugly fonts.  For example,
problems with displaying Emoji are due to some missing infrastructure
(see a separate discussion about that), and once we add that, the
result will be much better than anything you could get with Unifont.

IOW, using Unifont is simply a kind of admission of defeat: we don't
really know why Emacs doesn't find a good font, so we take the easy
path of providing _some_ font that will always work, albeit show ugly
glyphs.  I see no need to declare defeat, as we have facilities in
Emacs to solve these problems in a better way, once we understand
them.  We should therefore try to understand the problems better, and
once understood, solve them properly.

Re: Suggest installing more fonts?

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Sat, 17 Oct 2020 08:23:22 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I doubt it's as few as 30, assuming we want to cover every living
> > script and some popular dead ones.
> 
> I don't think we do, really.  Listing the major fonts in a help text
> like this would be sufficient.

Then I really don't understand the goal here, because I think the
problems people report are never about "the major fonts" or the
"usual" scripts -- those are well covered by our default fontset.

Do you have any evidence that we have such fundamental problems with
our fontset setup?  If so, can you point me to the relevant reports
and discussions?

> >> Popping up a warning buffer (at the first time tofu is displayed per
> >> session) may be OK
> >
> > That's not doable, because you cannot pop up anything in the middle of
> > redisplay.  We could perhaps use the delayed-warnings mechanism, but
> > is that sufficient?
> 
> I think so.

Before we implement something like that, we should be reasonably
certain it will not annoy most users (not only power users).  It is
entirely legitimate to leave some of the Unicode blocks uncovered by
the fonts you have installed, and having Emacs display warnings about
that might be annoying.  OTOH, such a feature must be turned on by
default, otherwise it will not do its job.  So knowing whether users
will welcome this is important, IMO.

> >> Or...  adding help echo text props to tofu?
> >
> > To each one of them?
> 
> That's probably overkill.  Say, the first 100 of those in a buffer?
> Only the first?

It cannot be the first in a buffer, it should be in the window,
because otherwise it might not be visible.

Re: Suggest installing more fonts?

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Sat, 17 Oct 2020 08:31:33 +0200
> Cc: emacs-devel@gnu.org
> 
> There's two philosophies here, depending on the "class" of the
> application.  The applications that are primarily concerned with
> displaying documents (Chrome, Firefox, Libreoffice) put a lot of work
> into this, and both have giant lists of fonts preinstalled, and are
> distributed with (some) fonts.
> 
> Other apps, that are more bare-bones (like Terminal) just look at the
> default font and then tufo if the font doesn't have that glyph.
> 
> Emacs is somewhere in the middle here, as usual.

A significant difference between Emacs and other apps is that Emacs
doesn't use fixed lists of fonts, but instead relies on system font
configurations to provide the fonts suitable for displaying some
character.  Thus, Emacs allows the users to install _any_ font that
supports a character, and get that displayed, even if the font is not
one of those known to the application in advance.  I believe this
makes Emacs more flexible and future-proof, but as usual, that power
comes with a price.

For that reason, I think it doesn't help too much to look at other
applications, because their philosophy in this department is very
different.  We should instead identify the issues Emacs users
encounter wrt fonts, and design and implement solutions for those
issues, whether by improving documentation or by enhancing our font
search and matching infrastructures.  For example, the issues with
Emoji are due to several fundamental issues we already identified (the
fact that Emoji belong to a large and very heterogeneous "script" we
call "symbol", for example) and are working on a solution.

Re: Suggest installing more fonts?

[Gregory Heytings via Emacs development discussions.]

[-- Attachment #1: Type: text/plain, Size: 4031 bytes --]


>>> I know.  I was just pointing out that there are several problems 
>>> (missing fonts is one of them, but not the only one), so before trying 
>>> to solve "the problem", we should be clear about which one it is and 
>>> try and figure out whether it's indeed the most common.
>>
>> A user has (usually) no idea of these "several problems", and knows 
>> even less how to fix them.  For such users the problem is simply "Emacs 
>> displays gibberish".  That is "the problem" IMO.  Everything else are 
>> technicalities that very few know/understand/have the patience to deal 
>> with.
>>
>> So IMO for these users it would be much better to display characters 
>> with a low-resolution font, together with a warning (and perhaps a 
>> pointer to a short guide) (as a help-echo and/or in the echo area), 
>> instead of a tofu.
>
> Then the problem will become "Emacs uses an ugly font where other apps 
> don't", and we are none the wiser.
>

_All_ apps have that problem, it is _not_ an Emacs-specific problem.  I 
provided two examples with apps that have a lot more available manpower 
than Emacs (Chromium and VS Code) in which that problem also exists. 
What _all_ apps do in that case is to display a tofu.

>
> IMO, we should try fixing whatever problems are there (and AFAIU there's 
> more than one) without using ugly fonts.  For example, problems with 
> displaying Emoji are due to some missing infrastructure (see a separate 
> discussion about that), and once we add that, the result will be much 
> better than anything you could get with Unifont.
>

It's a very complex problem, which is why it exists in all apps.  I don't 
know a single app in which that problem does not exist.

FWIW, here is my solution for Emojis (on Debian, after installing the 
fonts-noto-color-emoji package):

fc-match --format='%{charset}\n' NotoColorEmoji | tr ' ' '\n' | grep '^[^-][^-][^-][^-]' | grep -v '^200d$' | sed 's/^/#x/;s/-/ . #x/;s/^\(#[^#]*#[^#]*\)/!(\1)/;s/^\(.*\)$/(set-fontset-font "fontset-default" \1 "NotoColorEmoji")/' | tr '!' "'"

This creates 154 calls to "set-fontset-font", and as a result 
https://unicode.org/Public/UNIDATA/emoji/emoji-data.txt is displayed 
nicely by Emacs.  I'm sure you will tell me that you don't like that 
solution ;-)  But I doubt it is possible to do really better, because 
Emojis are scattered across the whole Unicode code space.

>
> IOW, using Unifont is simply a kind of admission of defeat: we don't 
> really know why Emacs doesn't find a good font, so we take the easy path 
> of providing _some_ font that will always work, albeit show ugly glyphs. 
> I see no need to declare defeat, as we have facilities in Emacs to solve 
> these problems in a better way, once we understand them.  We should 
> therefore try to understand the problems better, and once understood, 
> solve them properly.
>

The meaning of my proposal was, in fact, the exact opposite of an 
admission of defeat, it was to make Emacs better in that respect than all 
other apps.  Displaying a tofu is an admission of defeat, and it's what 
all other apps do.  They say to the user: "I don't know how to display 
that character, so I just give up without giving you any clue." 
Displaying a bitmap glyph with a warning message is doing something 
better: the users see the characters and get an indication on what they 
could do to see them better.

Another example: I have no fonts to display tamil characters on my 
computer.  Which means every app, including Emacs, displays tamil 
characters as a sequence of tofus.  I attach three pictures of the same 
short tamil text displayed by Emacs, one with its current behavior 
(tamil-no-font.png), one if it used Unifont as a fallback 
(tamil-unifont.png), and one if a proper tamil font (the Debian package 
fonts-lohit-taml) is installed (tamil-good-font.png).  I do agree (of 
course) that tamil-good-font.png is better than tamil-unifont.png, but I 
really can't understand how tamil-no-font.png could be considered better 
than tamil-unifont.png.

[-- Attachment #2: Type: image/png, Size: 1618 bytes --]

[-- Attachment #3: Type: image/png, Size: 690 bytes --]

[-- Attachment #4: Type: image/png, Size: 1866 bytes --]

Re: Suggest installing more fonts?

[Eli Zaretskii]

> Date: Sat, 17 Oct 2020 11:21:52 +0000
> From: Gregory Heytings <ghe@sdf.org>
> cc: larsi@gnus.org, monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> >> So IMO for these users it would be much better to display characters 
> >> with a low-resolution font, together with a warning (and perhaps a 
> >> pointer to a short guide) (as a help-echo and/or in the echo area), 
> >> instead of a tofu.
> >
> > Then the problem will become "Emacs uses an ugly font where other apps 
> > don't", and we are none the wiser.
> 
> _All_ apps have that problem, it is _not_ an Emacs-specific problem.

Let's keep the discussion in context, okay?  The context was the
complaints that Emacs displays tofu where other applications display
characters from installed fonts (which aren't GNU Unifont).  It is
those use cases that I was talking about.  Where Emacs behaves no
worse than other apps is in a different category.  More about that
below.

> FWIW, here is my solution for Emojis (on Debian, after installing the 
> fonts-noto-color-emoji package):
> 
> fc-match --format='%{charset}\n' NotoColorEmoji | tr ' ' '\n' | grep '^[^-][^-][^-][^-]' | grep -v '^200d$' | sed 's/^/#x/;s/-/ . #x/;s/^\(#[^#]*#[^#]*\)/!(\1)/;s/^\(.*\)$/(set-fontset-font "fontset-default" \1 "NotoColorEmoji")/' | tr '!' "'"
> 
> This creates 154 calls to "set-fontset-font", and as a result 
> https://unicode.org/Public/UNIDATA/emoji/emoji-data.txt is displayed 
> nicely by Emacs.  I'm sure you will tell me that you don't like that 
> solution ;-)  But I doubt it is possible to do really better, because 
> Emojis are scattered across the whole Unicode code space.

The Emoji problem was already analyzed and a solution is in the works
which will not look anywhere like the above (setting up the fontset is
not enough, btw).  Let's move on to problems for which we don't yet
have a solution, okay?

> > IOW, using Unifont is simply a kind of admission of defeat: we don't 
> > really know why Emacs doesn't find a good font, so we take the easy path 
> > of providing _some_ font that will always work, albeit show ugly glyphs. 
> > I see no need to declare defeat, as we have facilities in Emacs to solve 
> > these problems in a better way, once we understand them.  We should 
> > therefore try to understand the problems better, and once understood, 
> > solve them properly.
> 
> The meaning of my proposal was, in fact, the exact opposite of an 
> admission of defeat, it was to make Emacs better in that respect than all 
> other apps.  Displaying a tofu is an admission of defeat, and it's what 
> all other apps do.

Using Unifont is a defeat in the sense that we don't do better by
finding a good font.  The Unifont "solution" is already in Emacs, see
the setting of fontset-default.  So if you are willing to settle for
Unifont, we already do that.  IMO, we should try to find ways to do
better than that.

> Another example: I have no fonts to display tamil characters on my 
> computer.  Which means every app, including Emacs, displays tamil 
> characters as a sequence of tofus.  I attach three pictures of the same 
> short tamil text displayed by Emacs, one with its current behavior 
> (tamil-no-font.png), one if it used Unifont as a fallback 
> (tamil-unifont.png), and one if a proper tamil font (the Debian package 
> fonts-lohit-taml) is installed (tamil-good-font.png).  I do agree (of 
> course) that tamil-good-font.png is better than tamil-unifont.png, but I 
> really can't understand how tamil-no-font.png could be considered better 
> than tamil-unifont.png.

Do you read the Tamil script (I don't)?  If not, we should ask someone
who does what they think about the Unifont glyphs.  (To my un-expert
eyes, the right-most glyph looks completely unrecognizable with
Unifont, but that's me.)

In any case, it is never a bad idea to try find better ways of
handling this situation than we already have.  For starters, I'm not
sure I understand what kinds of situation is it that the user wants to
read Tamil text, but the system isn't ready for that wrt installed
fonts.  When we understand the relevant use case, we might be able to
come up with useful features for them.  For example, perhaps we should
have a command to report on fonts suitable to display a certain script
or a certain block of Unicode codepoints -- it should be easy to write
such a command, and it could be advertised as something users should
run when they are about to start using a new script or range of
characters.  We could then provide some specific advice in the text
displayed by that command if no suitable fonts were found.

Re: Suggest installing more fonts?

[Gregory Heytings via Emacs development discussions.]

>
> Let's keep the discussion in context, okay?  The context was the 
> complaints that Emacs displays tofu where other applications display 
> characters from installed fonts (which aren't GNU Unifont).  It is those 
> use cases that I was talking about.  Where Emacs behaves no worse than 
> other apps is in a different category.  More about that below.
>

That was not the meaning of Lars' message 24 hours ago:

>
> I think, over the years, the most common question users have that has an 
> easy-to-fix solution is: "Why is Emacs displaying boxes for some of 
> these characters I'm seeing?"  The solution is "install some more 
> fonts", of course.
>
> But could Emacs be more helpful here?  That is, could we somehow, 
> unobtrusively, tell the users this when Emacs is trying to display a 
> character that it has no fonts for?
>
> For added helpfulness, would it be possible for Emacs to be even more 
> specific?  Like, say "sudo apt install fonts-noto-color-emoji 
> fonts-symbola", or whatever, depending on the system.
>

About the use cases you are talking about, I haven't seen cases where 
"Emacs displays tofu where other applications display characters from 
installed fonts".  My experience is that Emacs behaves better than many 
other applications in this respect (except for the particular case of 
Emojis).

>
> The Emoji problem was already analyzed and a solution is in the works 
> which will not look anywhere like the above (setting up the fontset is 
> not enough, btw).  Let's move on to problems for which we don't yet have 
> a solution, okay?
>

I was not aware that you were working on a solution to the Emoji problem 
before I read your last email.  My (admittedly hacky) solution works fine, 
in fact it makes Emacs behave better than any other program (browser or 
editor) on my computer, so I don't understand why you write that it's not 
enough.  I'll continue to use it, until the better solution arrives.

>> The meaning of my proposal was, in fact, the exact opposite of an 
>> admission of defeat, it was to make Emacs better in that respect than 
>> all other apps.  Displaying a tofu is an admission of defeat, and it's 
>> what all other apps do.
>
> Using Unifont is a defeat in the sense that we don't do better by 
> finding a good font.  The Unifont "solution" is already in Emacs, see 
> the setting of fontset-default.  So if you are willing to settle for 
> Unifont, we already do that.  IMO, we should try to find ways to do 
> better than that.
>

My proposal was for the cases where _no_ suitable font exists.  In that 
case it is by definition impossible to find a good font.  And no, the 
Unifont solution is not already _in_ Emacs, because it requires to install 
a particular additional package/font (Unifont), which is something most 
users will not do.  The meaning of my proposal was only to include it _in_ 
Emacs, in short, to use, when it exists, the Unifont glyph in 
produce_glyphless_glyph() instead of creating a tofu with a hexcode. 
(For example, C-h h should IMO not display a single tofu.)

>
> Do you read the Tamil script (I don't)?  If not, we should ask someone 
> who does what they think about the Unifont glyphs.  (To my un-expert 
> eyes, the right-most glyph looks completely unrecognizable with Unifont, 
> but that's me.)
>

I do not read the Tamil script, but when I compare this to the variety of 
fonts in the languages I read, I doubt a Tamil reader would be unable to 
understand that text because that right-most glyph is not exactly 
positioned as it should be.

>
> In any case, it is never a bad idea to try find better ways of handling 
> this situation than we already have.  For starters, I'm not sure I 
> understand what kinds of situation is it that the user wants to read 
> Tamil text, but the system isn't ready for that wrt installed fonts.
>

This does happen in practice.  For example when opening a source file with 
comments written in a foreign language.  In that case I'm not necessarily 
interested in actually reading that text, but I'd like to see at least 
what these characters are/look like.  Another case when this does happen 
in practice is (as Lars mentioned in his message) for Emojis, in which 
case I'd like to have at least an idea of what the Emoji represents.  Yet 
another case would be when visiting a website written in a foreign 
language with eww, again I'm probably not interested in reading the text 
but seeing tofus is just ugly.  And so forth.

Re: Suggest installing more fonts?

[Eli Zaretskii]

> Date: Sat, 17 Oct 2020 13:09:58 +0000
> From: Gregory Heytings <ghe@sdf.org>
> cc: larsi@gnus.org, monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> > Let's keep the discussion in context, okay?  The context was the 
> > complaints that Emacs displays tofu where other applications display 
> > characters from installed fonts (which aren't GNU Unifont).  It is those 
> > use cases that I was talking about.  Where Emacs behaves no worse than 
> > other apps is in a different category.  More about that below.
> 
> That was not the meaning of Lars' message 24 hours ago:

That was not the only message in the thread.

> About the use cases you are talking about, I haven't seen cases where 
> "Emacs displays tofu where other applications display characters from 
> installed fonts".  My experience is that Emacs behaves better than many 
> other applications in this respect (except for the particular case of 
> Emojis).

I invite you to read the various font- and fontset-related bug reports
we have, and also relevant posts on Reddit and elsewhere out there.

> > The Emoji problem was already analyzed and a solution is in the works 
> > which will not look anywhere like the above (setting up the fontset is 
> > not enough, btw).  Let's move on to problems for which we don't yet have 
> > a solution, okay?
> 
> I was not aware that you were working on a solution to the Emoji problem 

I'm not; Robert Pluim is.  See bug#44020.

> The meaning of my proposal was only to include it _in_ Emacs, in
> short, to use, when it exists, the Unifont glyph in
> produce_glyphless_glyph() instead of creating a tofu with a hexcode.

As I wrote, this is not easily done, as font installation is a
system-wide action, as the font needs to be known by the system-wide
utilities and libraries we use for font searching.

And I personally wouldn't do this for Unifont.  It's UGLY.

But we are repeating ourselves, so let's agree to disagree on this.

Re: Suggest installing more fonts?

[Stefan Monnier]

>> > AFAIK, "most other applications", at least those I could investigate,
>> > come with fixed lists of fonts to use for each script.  These lists
>> > are maintained for each OS and are adjusted as the OS version and the
>> > fonts change.  This method has its advantages and also its
>> > disadvantages; I think both are quite clear.
>> But then it means that "installing more fonts" is not really the solution
>> that the user needs to follow, right?
> I'm not sure how you reached this conclusion.

I understood what you wrote above as saying that the only extra info
other apps have is a "fixed lists of fonts to use for each script".
So if they can display the text correctly with that extra list, it
follows that installing more fonts is not necessary for Emacs to also
display the text properly.  I guess you're right that it doesn't imply
that installing more fonts isn't also a solution, tho.


        Stefan

Re: Suggest installing more fonts?

[Stefan Monnier]

> Emacs already knows not to search the available fonts for a character
> that it already tried to display and failed to find a suitable font.
> We should probably reuse the information which says a character
> doesn't have a font for finding such characters and marking them.

I assumed that Emacs knows it's choosing "tofu" for any particular
glyph, regardless if that's the result of a long search or the reuse of
some previous search.

If it doesn't (e.g. it just reuses "the same glyph as last time this
char was used with this face"), then indeed it might be a bit more
tricky, but I'm sure it can be made acceptably cheap (both in terms of
memory&CPU resources and in terms of code complexity).


        Stefan

Re: Suggest installing more fonts?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: ghe@sdf.org,  larsi@gnus.org,  emacs-devel@gnu.org
> Date: Sat, 17 Oct 2020 09:49:09 -0400
> 
> > Emacs already knows not to search the available fonts for a character
> > that it already tried to display and failed to find a suitable font.
> > We should probably reuse the information which says a character
> > doesn't have a font for finding such characters and marking them.
> 
> I assumed that Emacs knows it's choosing "tofu" for any particular
> glyph, regardless if that's the result of a long search or the reuse of
> some previous search.

The glyph information (assuming you mean 'struct glyph') is ephemeral,
and is not recorded between redisplay cycles.  It stays stored in the
"current" glyph matrix, but Emacs almost never consults that for
anything except comparison with "desired" matrix, because the current
matrix can easily become outdated.

> If it doesn't (e.g. it just reuses "the same glyph as last time this
> char was used with this face"), then indeed it might be a bit more
> tricky, but I'm sure it can be made acceptably cheap (both in terms of
> memory&CPU resources and in terms of code complexity).

AFAIR, this is indeed based on face information (and the fact that
there's no font is recorded in the face struct), but these faces
aren't exposed to Lisp, they are created and used internally.  Still,
the information should be reachable, I think.

Re: Suggest installing more fonts?

[Stefan Monnier]

> The glyph information (assuming you mean 'struct glyph') is ephemeral,
> and is not recorded between redisplay cycles.  It stays stored in the
> "current" glyph matrix, but Emacs almost never consults that for
> anything except comparison with "desired" matrix, because the current
> matrix can easily become outdated.

I didn't mean "struct glyph", no.  I was thinking about putting a test
in the place where we *draw* a tofu char.  I can't find that code right
now, tho, so I don't know if that would be workable.


        Stefan

Re: Suggest installing more fonts?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: ghe@sdf.org,  larsi@gnus.org,  emacs-devel@gnu.org
> Date: Sat, 17 Oct 2020 11:04:23 -0400
> 
> I didn't mean "struct glyph", no.  I was thinking about putting a test
> in the place where we *draw* a tofu char.  I can't find that code right
> now, tho, so I don't know if that would be workable.

The code is in xdisp.c:produce_glyphless_glyph.

Re: Suggest installing more fonts?

[Gregory Heytings via Emacs development discussions.]

>> About the use cases you are talking about, I haven't seen cases where 
>> "Emacs displays tofu where other applications display characters from 
>> installed fonts".  My experience is that Emacs behaves better than many 
>> other applications in this respect (except for the particular case of 
>> Emojis).
>
> I invite you to read the various font- and fontset-related bug reports 
> we have, and also relevant posts on Reddit and elsewhere out there.

Alas, I could not find anything with your indications.

>> The meaning of my proposal was only to include it _in_ Emacs, in short, 
>> to use, when it exists, the Unifont glyph in produce_glyphless_glyph() 
>> instead of creating a tofu with a hexcode.
>
> As I wrote, this is not easily done, as font installation is a 
> system-wide action, as the font needs to be known by the system-wide 
> utilities and libraries we use for font searching.
>

Apparently it's not clear, so I'll say one last time that the feature I 
propose does _not_ require to install Unifont system-wide.  It is to 
include Unifont in Emacs (say in etc/unifont), and to offer it as an 
additional option for glyphless-char-display, along with hex-code, 
empty-box, thin-space, and zero-width.  The bitmap data would be used to 
draw the glyph in produce_glyphless_glyph() .

FYI, that feature is not "already in Emacs", as you said: when Unifont is 
installed, it takes precedence over some (but not all, I'm not sure why) 
of the better-looking available fonts.  One example: DejaVu Sans has 
hebrew characters, but with Unifont installed hebrew characters are 
displayed with Unifont.

By the way, while doing various experiments around that question, I found 
a bug in ftcrfont.c.  If a buffer is displayed with a certain font, and 
that font is removed, Emacs segfaults when it redisplays that buffer. 
The bug is in ftcrfont_open, on line 237:

ft_face = cairo_ft_scaled_font_lock_face (scaled_font);

This line should be followed by a:

if (!ft_face)
   {
     unblock_input ();
     /* further cleanup? */
     return Qnil;
   }

Re: Suggest installing more fonts?

[Eli Zaretskii]

> Date: Sat, 17 Oct 2020 16:09:09 +0000
> From: Gregory Heytings <ghe@sdf.org>
> cc: larsi@gnus.org, monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> > I invite you to read the various font- and fontset-related bug reports 
> > we have, and also relevant posts on Reddit and elsewhere out there.
> 
> Alas, I could not find anything with your indications.

Nonetheless, it's there.

> >> The meaning of my proposal was only to include it _in_ Emacs, in short, 
> >> to use, when it exists, the Unifont glyph in produce_glyphless_glyph() 
> >> instead of creating a tofu with a hexcode.
> >
> > As I wrote, this is not easily done, as font installation is a 
> > system-wide action, as the font needs to be known by the system-wide 
> > utilities and libraries we use for font searching.
> 
> Apparently it's not clear, so I'll say one last time that the feature I 
> propose does _not_ require to install Unifont system-wide.  It is to 
> include Unifont in Emacs (say in etc/unifont), and to offer it as an 
> additional option for glyphless-char-display, along with hex-code, 
> empty-box, thin-space, and zero-width.  The bitmap data would be used to 
> draw the glyph in produce_glyphless_glyph() .

I don't understand this proposal.  Are you saying Emacs can already
use a font that is not installed?  If so, can you tell how to do that?

The way Emacs uses fonts is by using various system libraries, such as
Fontconfig, to find fonts that match certain criteria (script,
encoding, character codepoint, size, slant, etc.).  How do you propose
to do that if, for example, Fontconfig knows nothing about a font?

> FYI, that feature is not "already in Emacs", as you said: when Unifont is 
> installed, it takes precedence over some (but not all, I'm not sure why) 
> of the better-looking available fonts.  One example: DejaVu Sans has 
> hebrew characters, but with Unifont installed hebrew characters are 
> displayed with Unifont.

This actually means that Unifont is more "in Emacs" than I thought. ;-)

Anyway, did you check the coverage of Hebrew by DejaVu Sans? does it
cover Hebrew?  In any case, Emacs uses Fontconfig to tell which fonts
cover what characters, so I'm guessing the answer is in what
Fontconfig thinks.  (Of course, this is a separate issue, and I'm not
sure we should discuss it in this thread.)

> By the way, while doing various experiments around that question, I found 
> a bug in ftcrfont.c.  If a buffer is displayed with a certain font, and 
> that font is removed, Emacs segfaults when it redisplays that buffer. 
> The bug is in ftcrfont_open, on line 237:
> 
> ft_face = cairo_ft_scaled_font_lock_face (scaled_font);
> 
> This line should be followed by a:
> 
> if (!ft_face)
>    {
>      unblock_input ();
>      /* further cleanup? */
>      return Qnil;
>    }

We had already such a patch submitted, but not installed for some
reason I cannot understand, and it was just recently mentioned on
bug-gnu-emacs (together with a few more places where such a test is
needed).

RE: Suggest installing more fonts?

[Drew Adams]

[Once again, had to add emacs-devel manually.
 `Reply All' wasn't good enough.]

> I attach three pictures of the same
> short tamil text displayed by Emacs, one with its current behavior
> (tamil-no-font.png), one if it used Unifont as a fallback
> (tamil-unifont.png), and one if a proper tamil font (the Debian package
> fonts-lohit-taml) is installed (tamil-good-font.png).  I do agree (of
> course) that tamil-good-font.png is better than tamil-unifont.png, but I
> really can't understand how tamil-no-font.png could be considered better
> than tamil-unifont.png.

(Caveat: Not following the details of this thread.)

I think, at least for the case shown by those images,
that showing the less-than-ideal, but presumably
readable, chars is better than just showing tofus.

However, in that case, it would be good if Emacs
could also (optionally, in case notification would
bother some users) notify you that you didn't have
a font supporting some chars, and so Emacs used font
Unifont as a fallback.

If Emacs could further highlight or otherwise show
which chars (or some of them) it's talking about,
that would be even more helpful.

IOW, if Emacs could (1) make it possible to read the
text, showing more/all chars, even if with a poor
font, AND (2) tell users that some chars underwent
this display degradation because they don't have a
font installed for them, AND (3) indicate which chars
underwent the possible degradation, that would be an
improvement.

Other things being equal, that is.  E.g., if UI or
performance would be strongly negatively influenced,
then trying to help this way might not be worth it
in some cases.

Re: Suggest installing more fonts?

[Gregory Heytings via Emacs development discussions.]

>> Apparently it's not clear, so I'll say one last time that the feature I 
>> propose does _not_ require to install Unifont system-wide.  It is to 
>> include Unifont in Emacs (say in etc/unifont), and to offer it as an 
>> additional option for glyphless-char-display, along with hex-code, 
>> empty-box, thin-space, and zero-width.  The bitmap data would be used 
>> to draw the glyph in produce_glyphless_glyph() .
>
> I don't understand this proposal.  Are you saying Emacs can already use 
> a font that is not installed?  If so, can you tell how to do that?
>

I don't (yet) know, but I'd be surprised if this could not be done. 
Emacs already creates glyphs dynamically for tofus, and already uses 
black-and-white bitmaps at least in fringes.

>
> The way Emacs uses fonts is by using various system libraries, such as 
> Fontconfig, to find fonts that match certain criteria (script, encoding, 
> character codepoint, size, slant, etc.).  How do you propose to do that 
> if, for example, Fontconfig knows nothing about a font?
>

The proposal is not to change anything to the way Emacs uses fonts, but to 
change something to the way Emacs behaves when it does not find an 
appropriate font to display a character.  In that case Emacs would display 
a "degraded" glyph (from Unifont), and would issue a warning that the user 
should install another font.

>> FYI, that feature is not "already in Emacs", as you said: when Unifont 
>> is installed, it takes precedence over some (but not all, I'm not sure 
>> why) of the better-looking available fonts.  One example: DejaVu Sans 
>> has hebrew characters, but with Unifont installed hebrew characters are 
>> displayed with Unifont.
>
> This actually means that Unifont is more "in Emacs" than I thought. ;-)
>
> Anyway, did you check the coverage of Hebrew by DejaVu Sans? does it 
> cover Hebrew?
>

Yes, DejaVu Sans covers Hebrew (100% coverage).

Re: Suggest installing more fonts?

[Eli Zaretskii]

> Date: Sat, 17 Oct 2020 17:37:30 +0000
> From: Gregory Heytings <ghe@sdf.org>
> cc: larsi@gnus.org, monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> >> Apparently it's not clear, so I'll say one last time that the feature I 
> >> propose does _not_ require to install Unifont system-wide.  It is to 
> >> include Unifont in Emacs (say in etc/unifont), and to offer it as an 
> >> additional option for glyphless-char-display, along with hex-code, 
> >> empty-box, thin-space, and zero-width.  The bitmap data would be used 
> >> to draw the glyph in produce_glyphless_glyph() .
> >
> > I don't understand this proposal.  Are you saying Emacs can already use 
> > a font that is not installed?  If so, can you tell how to do that?
> 
> I don't (yet) know, but I'd be surprised if this could not be done. 
> Emacs already creates glyphs dynamically for tofus

No, Emacs doesn't create any glyphs dynamically for tofus, it simply
uses a smaller font in a box it itself draws.  See
xterm.c:x_draw_glyphless_glyph_string_foreground for how this is done
on X (w32 and ns do it very similarly).

> > The way Emacs uses fonts is by using various system libraries, such as 
> > Fontconfig, to find fonts that match certain criteria (script, encoding, 
> > character codepoint, size, slant, etc.).  How do you propose to do that 
> > if, for example, Fontconfig knows nothing about a font?
> 
> The proposal is not to change anything to the way Emacs uses fonts, but to 
> change something to the way Emacs behaves when it does not find an 
> appropriate font to display a character.  In that case Emacs would display 
> a "degraded" glyph (from Unifont), and would issue a warning that the user 
> should install another font.

I don't see how this can be done without serious changes in the code
that finds and uses fonts.  We _need_ the Fontconfig functionality.

> > Anyway, did you check the coverage of Hebrew by DejaVu Sans? does it 
> > cover Hebrew?
> 
> Yes, DejaVu Sans covers Hebrew (100% coverage).

And this is in "emacs -Q" and with DejaVu Sans the default font?

Re: Suggest installing more fonts?

[Gregory Heytings via Emacs development discussions.]

>>>> Apparently it's not clear, so I'll say one last time that the feature 
>>>> I propose does _not_ require to install Unifont system-wide.  It is 
>>>> to include Unifont in Emacs (say in etc/unifont), and to offer it as 
>>>> an additional option for glyphless-char-display, along with hex-code, 
>>>> empty-box, thin-space, and zero-width.  The bitmap data would be used 
>>>> to draw the glyph in produce_glyphless_glyph() .
>>>
>>> I don't understand this proposal.  Are you saying Emacs can already 
>>> use a font that is not installed?  If so, can you tell how to do that?
>>
>> I don't (yet) know, but I'd be surprised if this could not be done. 
>> Emacs already creates glyphs dynamically for tofus
>
> No, Emacs doesn't create any glyphs dynamically for tofus, it simply 
> uses a smaller font in a box it itself draws.  See 
> xterm.c:x_draw_glyphless_glyph_string_foreground for how this is done on 
> X (w32 and ns do it very similarly).
>

Yes, but Emacs draws the box itself, and I suppose that if it can draw a 
box, it could instead draw a small black-and-white (8x16 or 16x16) bitmap. 
Emacs already does this for fringe bitmaps AFAIU.

>>> The way Emacs uses fonts is by using various system libraries, such as 
>>> Fontconfig, to find fonts that match certain criteria (script, 
>>> encoding, character codepoint, size, slant, etc.).  How do you propose 
>>> to do that if, for example, Fontconfig knows nothing about a font?
>>
>> The proposal is not to change anything to the way Emacs uses fonts, but 
>> to change something to the way Emacs behaves when it does not find an 
>> appropriate font to display a character.  In that case Emacs would 
>> display a "degraded" glyph (from Unifont), and would issue a warning 
>> that the user should install another font.
>
> I don't see how this can be done without serious changes in the code 
> that finds and uses fonts.  We _need_ the Fontconfig functionality.
>

With the proposal, there is no need to find the font, it is already 
included, and only used as a fallback when no appropriate font has been 
found.  Unifont is just a long sequence of small bitmaps, Emacs only has 
to pick the bitmap corresponding to a given character in that list and to 
draw it.  For example, the character "A" in Unifont is the 8x16 bitmap 
"0000000018242442427E424242420000" which occupies only 16 bytes.

Re: Suggest installing more fonts?

[Eli Zaretskii]

> Date: Sat, 17 Oct 2020 18:36:47 +0000
> From: Gregory Heytings <ghe@sdf.org>
> cc: larsi@gnus.org, monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> >> I don't (yet) know, but I'd be surprised if this could not be done. 
> >> Emacs already creates glyphs dynamically for tofus
> >
> > No, Emacs doesn't create any glyphs dynamically for tofus, it simply 
> > uses a smaller font in a box it itself draws.  See 
> > xterm.c:x_draw_glyphless_glyph_string_foreground for how this is done on 
> > X (w32 and ns do it very similarly).
> 
> Yes, but Emacs draws the box itself, and I suppose that if it can draw a 
> box, it could instead draw a small black-and-white (8x16 or 16x16) bitmap. 
> Emacs already does this for fringe bitmaps AFAIU.

But the "bitmap" in this case is a font glyph.  So to get at that
glyph, we would need to have the code that is already in the library
we use for accessing fonts, is that what you have in mind?  So we'd
need to have the code which understands how font files are structured,
and how to find in them the glyph(s) for a certain character given
that character's codepoint, and how to know whether the font has a
glyph for that character to begin with?

For example, look at xftfont_open, which is one implementation of how
to open a font.  Count the number of functions and variables whose
names begin with "Fc" -- those are references to the Fontconfig
library.  Under your proposal, we'd need to implement all that in our
own code, is that right?

> >> The proposal is not to change anything to the way Emacs uses fonts, but 
> >> to change something to the way Emacs behaves when it does not find an 
> >> appropriate font to display a character.  In that case Emacs would 
> >> display a "degraded" glyph (from Unifont), and would issue a warning 
> >> that the user should install another font.
> >
> > I don't see how this can be done without serious changes in the code 
> > that finds and uses fonts.  We _need_ the Fontconfig functionality.
> 
> With the proposal, there is no need to find the font, it is already 
> included, and only used as a fallback when no appropriate font has been 
> found.  Unifont is just a long sequence of small bitmaps, Emacs only has 
> to pick the bitmap corresponding to a given character in that list and to 
> draw it.  For example, the character "A" in Unifont is the 8x16 bitmap 
> "0000000018242442427E424242420000" which occupies only 16 bytes.

So we will code a special font "backend" that is tailored to what
Unifont has in its files now, and on top of that support this only for
a single size of the default font, i.e. no support for "C-x +", no
support for specifying a different size in default-frame-alist, etc.?
And then we will have to maintain this code when Unifont changes
something in its files?  I'm sorry, but this makes very little sense
to me, especially with an ugly font such as Unifont.

Re: Suggest installing more fonts?

[Eli Zaretskii]

> Date: Sat, 17 Oct 2020 21:50:43 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: larsi@gnus.org, monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> For example, look at xftfont_open, which is one implementation of how
> to open a font.  Count the number of functions and variables whose
> names begin with "Fc" -- those are references to the Fontconfig
> library.  Under your proposal, we'd need to implement all that in our
> own code, is that right?

Correction: not only the Fc* functions (from Fontconfig), but also the
Xft* functions (from libXft) -- the latter actually access the font
files and extract data from them.

Re: Suggest installing more fonts?

[Gregory Heytings via Emacs development discussions.]

> But the "bitmap" in this case is a font glyph.  So to get at that glyph, 
> we would need to have the code that is already in the library we use for 
> accessing fonts, is that what you have in mind?  So we'd need to have 
> the code which understands how font files are structured, and how to 
> find in them the glyph(s) for a certain character given that character's 
> codepoint, and how to know whether the font has a glyph for that 
> character to begin with?
>
> For example, look at xftfont_open, which is one implementation of how to 
> open a font.  Count the number of functions and variables whose names 
> begin with "Fc" -- those are references to the Fontconfig library. 
> Under your proposal, we'd need to implement all that in our own code, is 
> that right?
>

I don't understand why you think we would need something complex.  We 
don't need it.  Unifont has the simplest format you can imagine: a 
sequence of bitmaps.

>> With the proposal, there is no need to find the font, it is already 
>> included, and only used as a fallback when no appropriate font has been 
>> found.  Unifont is just a long sequence of small bitmaps, Emacs only 
>> has to pick the bitmap corresponding to a given character in that list 
>> and to draw it.  For example, the character "A" in Unifont is the 8x16 
>> bitmap "0000000018242442427E424242420000" which occupies only 16 bytes.
>
> So we will code a special font "backend" that is tailored to what 
> Unifont has in its files now, and on top of that support this only for a 
> single size of the default font, i.e. no support for "C-x +", no support 
> for specifying a different size in default-frame-alist, etc.? And then 
> we will have to maintain this code when Unifont changes something in its 
> files?  I'm sorry, but this makes very little sense to me, especially 
> with an ugly font such as Unifont.
>

The format of Unifont hasn't changed since it was created, about 25 years 
ago, and it will not change in the future (because this format is part of 
its design principles).

But you are so reluctant to this idea that I don't think it is useful to 
continue this discussion.  I only tried to propose something to improve 
Emacs, so that it would behave in a more user-friendly way when it is 
asked to display a character for which it has no fonts.

Re: Suggest installing more fonts?

[Eli Zaretskii]

> Date: Sat, 17 Oct 2020 19:18:25 +0000
> From: Gregory Heytings <ghe@sdf.org>
> cc: larsi@gnus.org, monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> 
> > But the "bitmap" in this case is a font glyph.  So to get at that glyph, 
> > we would need to have the code that is already in the library we use for 
> > accessing fonts, is that what you have in mind?  So we'd need to have 
> > the code which understands how font files are structured, and how to 
> > find in them the glyph(s) for a certain character given that character's 
> > codepoint, and how to know whether the font has a glyph for that 
> > character to begin with?
> >
> > For example, look at xftfont_open, which is one implementation of how to 
> > open a font.  Count the number of functions and variables whose names 
> > begin with "Fc" -- those are references to the Fontconfig library. 
> > Under your proposal, we'd need to implement all that in our own code, is 
> > that right?
> 
> I don't understand why you think we would need something complex.  We 
> don't need it.  Unifont has the simplest format you can imagine: a 
> sequence of bitmaps.

Even for such a simple file, and even if we believe the file will not
change, there's a lot of code to write, maintain and test, code that
already exists in the libraries we use.  For example, how do you know
which characters are in Unifont and which aren't? this is a very basic
capability that any font backend used by Emacs is required to have.
Even if we want to write code to access Unifont "by hand", we still
need to do that within the framework of how Emacs deals with fonts.

> > So we will code a special font "backend" that is tailored to what 
> > Unifont has in its files now, and on top of that support this only for a 
> > single size of the default font, i.e. no support for "C-x +", no support 
> > for specifying a different size in default-frame-alist, etc.? And then 
> > we will have to maintain this code when Unifont changes something in its 
> > files?  I'm sorry, but this makes very little sense to me, especially 
> > with an ugly font such as Unifont.
> 
> The format of Unifont hasn't changed since it was created, about 25 years 
> ago, and it will not change in the future (because this format is part of 
> its design principles).

What about the contents -- do they also not change?  If they do, we
will need to know how to read the data and how to adapt to changes in
the contents (e.g., when glyphs are added/removed from the font).

> But you are so reluctant to this idea that I don't think it is useful to 
> continue this discussion.

I'm reluctant to reinvent the wheel that others already invented, and
to maintain code that is already maintained by experts in this area,
yes.  It is not an efficient use of our scarce resources.

> I only tried to propose something to improve Emacs, so that it would
> behave in a more user-friendly way when it is asked to display a
> character for which it has no fonts.

I appreciate the motivation, but the way to improve Emacs in this area
is not by using Unifont, let alone "manually" handling it.  Much
better solutions exist, we just need to understand the problems well
enough, find their solutions, and improve/extend Emacs to incorporate
those solutions.  Computer typography is a fast-advancing field of
technology, we just need to take advantage of those advances in a
timely manner.  One reason for migrating our main font backends to
HarfBuzz was that it allows us to follow those advances much more
easily.

Re: Suggest installing more fonts?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > A user has (usually) no idea of these "several problems", and knows even 
  > less how to fix them.  For such users the problem is simply "Emacs 
  > displays gibberish".

If we can reproduce that case, we might be able to implement changes
that would detect some of the causes of the gibberish, and take action
either to (1) provide a fallback substitute for something or (2)
give a meaningful and useful error message.

That would be a step forward, at least; and after that we could
see what problems users still encounter.

  > So IMO for these users it would be much better to display characters with 
  > a low-resolution font, together with a warning (and perhaps a pointer to a 
  > short guide) (as a help-echo and/or in the echo area), instead of a tofu.

I don't know what you mean by "tofu", but your idea of displaying the
right character in the wrong font, rather than gibberish, sounds
useful.

The possible challenge would be deciding when to do this.
When would users prefer this, and
when would users prefer what Emacs does now?

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Suggest installing more fonts?

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Then I really don't understand the goal here, because I think the
> problems people report are never about "the major fonts" or the
> "usual" scripts -- those are well covered by our default fontset.

I was just thinking about listing fonts like DejaVu Sans and Noto (with
a wide support for many scripts), not smaller language-specific fonts.
And Symbola?

> Before we implement something like that, we should be reasonably
> certain it will not annoy most users (not only power users).  It is
> entirely legitimate to leave some of the Unicode blocks uncovered by
> the fonts you have installed, and having Emacs display warnings about
> that might be annoying.  OTOH, such a feature must be turned on by
> default, otherwise it will not do its job.  So knowing whether users
> will welcome this is important, IMO.

We display a button in warning buffers saying "Click here to switch the
warning off", so I think the annoyance factor would be somewhat limited.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Suggest installing more fonts?

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> if (!ft_face)
>>    {
>>      unblock_input ();
>>      /* further cleanup? */
>>      return Qnil;
>>    }
>
> We had already such a patch submitted, but not installed for some
> reason I cannot understand, and it was just recently mentioned on
> bug-gnu-emacs (together with a few more places where such a test is
> needed).

I seem to recall that the reason was that the patch doesn't cover the
other places where the test is needed?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Suggest installing more fonts?

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Date: Sun, 18 Oct 2020 00:12:25 -0400
> Cc: larsi@gnus.org, monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
>   > So IMO for these users it would be much better to display characters with 
>   > a low-resolution font, together with a warning (and perhaps a pointer to a 
>   > short guide) (as a help-echo and/or in the echo area), instead of a tofu.
> 
> I don't know what you mean by "tofu", but your idea of displaying the
> right character in the wrong font, rather than gibberish, sounds
> useful.

"Tofu" is the code of the character in hex, displayed in a box -- this
is how we by default show on GUI frames characters for which Emacs
couldn't find a suitable font.

But that isn't gibberish, in any shape or form.  It is one of the
accepted ways of showing a character for which there's no font; other
applications do the same (and don't bother to show any message about
that, btw).

Re: Suggest installing more fonts?

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Gregory Heytings <ghe@sdf.org>,  monnier@iro.umontreal.ca,
>   emacs-devel@gnu.org
> Date: Sun, 18 Oct 2020 10:06:48 +0200
> 
> >> if (!ft_face)
> >>    {
> >>      unblock_input ();
> >>      /* further cleanup? */
> >>      return Qnil;
> >>    }
> >
> > We had already such a patch submitted, but not installed for some
> > reason I cannot understand, and it was just recently mentioned on
> > bug-gnu-emacs (together with a few more places where such a test is
> > needed).
> 
> I seem to recall that the reason was that the patch doesn't cover the
> other places where the test is needed?

The other places were identified in the discussion, and are quite easy
to find by looking at the code.  I'd write the code myself, but I have
no access to a system where I can run the result, so I prefer that
someone else does that.

Re: Suggest installing more fonts?

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Sun, 18 Oct 2020 10:03:25 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Then I really don't understand the goal here, because I think the
> > problems people report are never about "the major fonts" or the
> > "usual" scripts -- those are well covered by our default fontset.
> 
> I was just thinking about listing fonts like DejaVu Sans and Noto (with
> a wide support for many scripts), not smaller language-specific fonts.
> And Symbola?
> 
> > Before we implement something like that, we should be reasonably
> > certain it will not annoy most users (not only power users).  It is
> > entirely legitimate to leave some of the Unicode blocks uncovered by
> > the fonts you have installed, and having Emacs display warnings about
> > that might be annoying.  OTOH, such a feature must be turned on by
> > default, otherwise it will not do its job.  So knowing whether users
> > will welcome this is important, IMO.
> 
> We display a button in warning buffers saying "Click here to switch the
> warning off", so I think the annoyance factor would be somewhat limited.

So what is the proposal in more practical terms?  We want redisplay to
leave some information in a buffer-local variable regarding some
buffer positions where no font could be found, and then we will
display a message saying something like "Characters detected for which
there is no font (buffer positions NN, MM, XX, YY); suggest to install
additional fonts"?  Is that the proposal.

If so, will we also mention specific fonts, and if so, which ones?
Should we also make sure those fonts are not already installed?

Btw, I've looked at browsers, and they just display tofu or an empty
rectangle without any message in this case.

Re: Suggest installing more fonts?

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> So what is the proposal in more practical terms?  We want redisplay to
> leave some information in a buffer-local variable regarding some
> buffer positions where no font could be found, and then we will
> display a message saying something like "Characters detected for which
> there is no font (buffer positions NN, MM, XX, YY); suggest to install
> additional fonts"?  Is that the proposal.

Yes, or "Characters detected for which there is no font; `C-h C-F' for
more information", and that would be a new command that'd look for tofu
in the buffer.  (In which case we don't need the variable, but just a
delayed warning, because the new command could do the analysis,
perhaps.)

> If so, will we also mention specific fonts, and if so, which ones?
> Should we also make sure those fonts are not already installed?

If we can, that'd be nice.  We do already know (based on the Unicode
data) what scripts the characters are from?  So we could at least list
that, and then expand the help, if we have the stamina, to also include
some fonts.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Suggest installing more fonts?

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Mon, 19 Oct 2020 10:31:29 +0200
> 
> > If so, will we also mention specific fonts, and if so, which ones?
> > Should we also make sure those fonts are not already installed?
> 
> If we can, that'd be nice.  We do already know (based on the Unicode
> data) what scripts the characters are from?

Yes, evaluate (aref char-script-table CHAR) for any CHAR.

> So we could at least list that, and then expand the help, if we have
> the stamina, to also include some fonts.

This I don't think I understand, at least not fully.  We can show the
script and suggest to install some font that covers that script, this
is clear.  But what did you mean by "and include some fonts"? name
some specific fonts? how can we do that when the fonts aren't
installed?

Re: Suggest installing more fonts?

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> So we could at least list that, and then expand the help, if we have
>> the stamina, to also include some fonts.
>
> This I don't think I understand, at least not fully.  We can show the
> script and suggest to install some font that covers that script, this
> is clear.  But what did you mean by "and include some fonts"? name
> some specific fonts? how can we do that when the fonts aren't
> installed?

That's where the stamina comes in -- we'd have to maintain a list of
fonts that we think are nice for the scripts in question.  How
extensive, of course, depends on how much work we want to put into it.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Suggest installing more fonts?

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Tue, 20 Oct 2020 12:28:14 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> So we could at least list that, and then expand the help, if we have
> >> the stamina, to also include some fonts.
> >
> > This I don't think I understand, at least not fully.  We can show the
> > script and suggest to install some font that covers that script, this
> > is clear.  But what did you mean by "and include some fonts"? name
> > some specific fonts? how can we do that when the fonts aren't
> > installed?
> 
> That's where the stamina comes in -- we'd have to maintain a list of
> fonts that we think are nice for the scripts in question.  How
> extensive, of course, depends on how much work we want to put into it.

That brings us back to the issue of how many fonts for how many
different system configurations will we need to know about and have
up-to-date information.

I think nowadays it should be enough to say something like "we
recommend to install a font for displaying the TaiViet script", it is
easy to find fonts with this information.

Re: Suggest installing more fonts?

[Stephen Leake]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> [ FWIW, I don't think replacing those tofu chars with some "ugly default"
>>> is a good option: it's more work with very little benefit if any, since
>>> the problem still remains largely unchanged. ]
>> FWIW, I do not agree with this.
>
> I kind of guessed ;-)
>
>> It is much better from a end-user viewpoint to see something they can
>> decypher, even if it has a non-optimal rendering, than to see a hex
>> code in a box.
>
> [ What you describe is the "very little benefit" I referred to.  ]
>
> I think most users won't be happy with the result in both cases,
> so they'll still want to find a fix for the problem [ This is "the
> problem still remains largely unchanged" that I referred to.  ]

I would be happier with an ugly but readable solution.

This problem occurs most often for me when reading email (this list
especially); the message is refering to some specific character that
is involved in some problem.

If I knew what the character was, I would not bother to track down the
appropriate script/font.

Apparently I can type C-x C-8 = to see what the character is by Unicode
name, so that's a partial workaround.

>>> - Does the problem only affect Emacs and not other applications? If so
>>> why?  If not, then what do other applications do about it?

I'll have to remember to view a problematic message via Firefox, to see
if it can display the character.

-- 
Written from unceded ancestral homelands of the Huichiun, an Ohlone
people,for which I pay <a href=

Re: Suggest installing more fonts?

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> I think nowadays it should be enough to say something like "we
> recommend to install a font for displaying the TaiViet script", it is
> easy to find fonts with this information.

Well, yes and no.

"apt search font | grep viet" gives me no matches on this Debian system.

I've been frustrated by this before, and I've resorted to saying "apt
install fonts-*' or something, which isn't really ideal, so if Emacs
could provide some guidance here (without us having to maintain a
never-ending list of fonts), I think that would be helpful.

I just looked at the HELLO file on this system, and these are the fonts
that Emacs has chosen (there's no tofu in the buffer):

Unifont
DejaVu Sans Mono
Various things from the Noto Serif/Noto Sans family of fonts -- I think
  they're all from the fonts-noto-core and fonts-noto-extra packages
Symbola
FreeSerif

Plus daewoo, for Korean.

So that'd be just about all the fonts (and possibly packages) we'd have
to mention on the font help page.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Suggest installing more fonts?

[Michael Albinus]

Lars Ingebrigtsen <larsi@gnus.org> writes:

Hi Lars,

> I've been frustrated by this before, and I've resorted to saying "apt
> install fonts-*' or something, which isn't really ideal, so if Emacs
> could provide some guidance here (without us having to maintain a
> never-ending list of fonts), I think that would be helpful.
>
> I just looked at the HELLO file on this system, and these are the fonts
> that Emacs has chosen (there's no tofu in the buffer):

See recent discussion in bug#43887. In order to see all symbols in
etc/HELLO, I had to apply on Ubuntu 20.04

$ sudo apt-get install fonts-noto-core

On Fedora 32, I have applied

$ sudo yum install google-noto-sans-javanese-fonts google-noto-sans-tai-viet-fonts

Likely, this works also for the other Debian and Red Hat derivates.

However, I don't know whether we want to recommend Google Noto fonts
<https://www.google.com/get/noto/>. They use the Open Font License 1.1
<https://github.com/googlefonts/noto-fonts/>, but I don't know whether
this license counts as free.

In Emacs, the file INSTALL recommends to install
intlfonts-VERSION.tar.gz. Might be sufficient, but without further
instruction I doubt every Emacs user will be able to follow.

> So that'd be just about all the fonts (and possibly packages) we'd have
> to mention on the font help page.

And/or better instructions in the INSTALL file.

Best regards, Michael.

Re: Suggest installing more fonts?

[Stefan Kangas]

Michael Albinus <michael.albinus@gmx.de> writes:

> However, I don't know whether we want to recommend Google Noto fonts
> <https://www.google.com/get/noto/>. They use the Open Font License 1.1
> <https://github.com/googlefonts/noto-fonts/>, but I don't know whether
> this license counts as free.

The full name of its license is SIL Open Font License 1.1, and:

"The Open Font License (including its original release, version 1.0)
is a free copyleft license for fonts. Its only unusual requirement
is that when selling the font, you must redistribute it bundled with
some software, rather than alone. Since a simple Hello World program
will satisfy the requirement, it is harmless. Neither we nor SIL
recommend the use of this license for anything other than fonts."

https://www.gnu.org/licenses/license-list.html#SILOFL

Re: Suggest installing more fonts?

[Basil L. Contovounesios]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> "apt search font | grep viet" gives me no matches on this Debian system.
                        ^^^
                        -i

gives

  fonts-sil-taiheritagepro - typeface reflecting the traditional
  hand-written style of the Tai Viet script

-- 
Basil

Re: Suggest installing more fonts?

[Lars Ingebrigtsen]

"Basil L. Contovounesios" <contovob@tcd.ie> writes:

> Lars Ingebrigtsen <larsi@gnus.org> writes:
>
>> "apt search font | grep viet" gives me no matches on this Debian system.
>                         ^^^
>                         -i
>
> gives
>
>   fonts-sil-taiheritagepro - typeface reflecting the traditional
>   hand-written style of the Tai Viet script

Well, it gives

larsi@xo:~/src/emacs/trunk$ apt search font | grep -i viet

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

  typeface reflecting the traditional hand-written style of the Tai Viet script

here, but point taken.  I'm guessing this isn't the font people would be
looking for, though, when Emacs isn't able to display most Vietnamese
text.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Suggest installing more fonts?

[Stephen Berman]

On Wed, 21 Oct 2020 14:05:41 +0200 Lars Ingebrigtsen <larsi@gnus.org> wrote:

> "Basil L. Contovounesios" <contovob@tcd.ie> writes:
>
>> Lars Ingebrigtsen <larsi@gnus.org> writes:
>>
>>> "apt search font | grep viet" gives me no matches on this Debian system.
>>                         ^^^
>>                         -i
>>
>> gives
>>
>>   fonts-sil-taiheritagepro - typeface reflecting the traditional
>>   hand-written style of the Tai Viet script
>
> Well, it gives
>
> larsi@xo:~/src/emacs/trunk$ apt search font | grep -i viet
>
> WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
>
>   typeface reflecting the traditional hand-written style of the Tai Viet script
>
> here, but point taken.  I'm guessing this isn't the font people would be
> looking for, though, when Emacs isn't able to display most Vietnamese
> text.

Tai Viet is not Vietnamese:

"The Vietnamese alphabet (Vietnamese: chữ Quốc ngữ; literally "National
language script") is the modern writing system for the Vietnamese
language. It uses the Latin script based on Romance languages,[4] in
particular, the Portuguese alphabet,[1] with some digraphs and the
addition of nine accent marks or diacritics – four of them to create
sounds, and the other five to indicate tone. These many diacritics,
often two on the same vowel, make written Vietnamese recognizable among
localized variants of Latin alphabets.[5]"
(from https://en.wikipedia.org/wiki/Vietnamese_alphabet)

and:

"The Tai Viet script (Tai Dam: Xư Tay (Tai Script), Vietnamese: Chữ Thái
Việt) is a Brahmic script used by the Tai Dam people and various other
Tai peoples in Vietnam.
[...]
According to Thai authors, the writing system is probably derived from
the old Thai writing of the kingdom of Sukhotai."
(from https://en.wikipedia.org/wiki/Tai_Viet_script)

Steve Berman

Re: Suggest installing more fonts?

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Wed, 21 Oct 2020 12:46:55 +0200
> 
> I just looked at the HELLO file on this system, and these are the fonts
> that Emacs has chosen (there's no tofu in the buffer):
> 
> Unifont
> DejaVu Sans Mono
> Various things from the Noto Serif/Noto Sans family of fonts -- I think
>   they're all from the fonts-noto-core and fonts-noto-extra packages
> Symbola
> FreeSerif
> 
> Plus daewoo, for Korean.
> 
> So that'd be just about all the fonts (and possibly packages) we'd have
> to mention on the font help page.

Uninstall Unifont (which I thought we agreed shouldn't be used), then
try again.

And are we okay with telling people to install Noto fonts?  There are
a lot of fonts in that collection, I'm not sure they are really
needed.  For example, do modern GNU/Linux systems really come without
CJK fonts?

Also, if this is the way we want to go, we should have separate advice
for MS-Windows users, where there are several system specific issues
regarding font search by Emacs, and many users are unable to install
fonts system-wide due to fascist corporation policies.

And finally, HELLO is not necessarily the right instrument to judge
this issue, because it doesn't pretend to cover many codepoints.
Users with special needs might very well need rarely-used characters
or even special-purpose fonts (e.g., think about scholars who need to
read text in some ancient script, or someone whose hobby is to write
Klingon).  As Unicode advances, more and more specialized scripts are
added to it, and HELLO explicitly avoids having too many ancient
"dead" scripts, for practical reasons.  Users who need such rare
characters and scripts will bump into missing glyphs whether you want
it or not, and we will never be able in practice to have every such
font in a list that we can name explicitly.

That is why I would propose a general advice that doesn't name
specific fonts.  It is easy to find fonts for a known script nowadays.

Re: Suggest installing more fonts?

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
> Date: Wed, 21 Oct 2020 13:31:01 +0200
> 
> In Emacs, the file INSTALL recommends to install
> intlfonts-VERSION.tar.gz. Might be sufficient, but without further
> instruction I doubt every Emacs user will be able to follow.

intlfonts are bitmapped (BDF) fonts, so I don't think nowadays they
will be considered as a satisfactory solution.  They are GNU software,
but otherwise I see no reason to advertise them today.

> And/or better instructions in the INSTALL file.

Yes, for those who still read those files.

tofu-help-mode (was: Suggest installing more fonts?)

[Stefan Monnier]

BTW, for those interested, here's a proof-of-concept.

I don't intend to work much more on this (except maybe for the
`post-redisplay-functions` part which might be useful on its own), so
I'd encourage someone else to take it over from here if they'd like to
see it turn into something usable.


        Stefan


diff --git a/lisp/simple.el b/lisp/simple.el
index ef519aa2cb..647ffd7320 100644
--- a/lisp/simple.el
+++ b/lisp/simple.el
@@ -6342,6 +6342,48 @@ redisplay--pre-redisplay-functions
 (add-function :before pre-redisplay-function
               #'redisplay--pre-redisplay-functions)
 
+(defvar post-redisplay-functions nil
+  "Hook run just after redisplay.
+It is called in each window that has been redisplayed.  It takes one argument,
+which is the window that was redisplayed.  When run, the `current-buffer'
+is set to the buffer displayed in that window.")
+
+(defun redisplay--post-redisplay-functions (windows)
+  (with-demoted-errors "redisplay--post-redisplay-functions: %S"
+    (if (null windows)
+        (with-current-buffer (window-buffer (selected-window))
+          (run-hook-with-args 'post-redisplay-functions (selected-window)))
+      (dolist (win (if (listp windows) windows (window-list-1 nil nil t)))
+        (with-current-buffer (window-buffer win)
+          (run-hook-with-args 'post-redisplay-functions win))))))
+
+(add-function :before post-redisplay-function
+              #'redisplay--post-redisplay-functions)
+
+(defun tofu-help--post-redisplay (_window)
+  (declare-function describe-char-display "descr-text" (pos char))
+  (while (consp redisplay-tofu)
+    (let* ((pos (pop redisplay-tofu))
+           (char (if (< pos (point-max)) (char-after pos))))
+      ;; Don't hide pre-existing help-echo.
+      (unless (or (null char)
+                  (get-text-property pos 'help-echo)
+                  (describe-char-display pos char))
+        (with-silent-modifications
+          (put-text-property pos (1+ pos) 'help-echo
+                             (or (get-char-code-property char 'name)
+		                 (get-char-code-property char 'old-name)
+		                 "This is unknown TOFU!")))))))
+
+(define-minor-mode tofu-help-mode
+  "Add help-echo on TOFU chars."
+  :lighter nil
+  (kill-local-variable 'redisplay-tofu)
+  (remove-hook 'post-redisplay-functions #'tofu-help--post-redisplay t)
+  (when tofu-help-mode
+    (require 'descr-text)               ;For describe-char-display
+    (setq-local redisplay-tofu nil)
+    (add-hook 'post-redisplay-functions #'tofu-help--post-redisplay nil t)))
 
 (defvar-local mark-ring nil
   "The list of former marks of the current buffer, most recent first.")
diff --git a/src/xdisp.c b/src/xdisp.c
index 4086eef9d0..e5559774b8 100644
--- a/src/xdisp.c
+++ b/src/xdisp.c
@@ -7333,6 +7333,18 @@ #define CHAR_COMPOSED_P(IT,CHARPOS,BYTEPOS,END_CHARPOS)			\
 						       (IT)->face_id),	\
 				 (IT)->string)))
 
+static void
+record_tofu (struct it *it)
+{
+  if (!EQ (Vredisplay_tofu, Qt))
+    {
+      Lisp_Object pos = make_fixnum (IT_CHARPOS (*it));
+      /* FIXME: If the user sets `redisplay-tofu' to a value that's
+         neither t nor a proper list, this signals an error!  */
+      if (NILP (Fmemq (pos, Vredisplay_tofu)))
+        Vredisplay_tofu = Fcons (pos, Vredisplay_tofu);
+    }
+}
 
 /* Lookup the char-table Vglyphless_char_display for character C (-1
    if we want information for no-font case), and return the display
@@ -7382,7 +7394,10 @@ lookup_glyphless_char_display (int c, struct it *it)
   else if (EQ (glyphless_method, Qempty_box))
     it->glyphless_method = GLYPHLESS_DISPLAY_EMPTY_BOX;
   else if (EQ (glyphless_method, Qhex_code))
-    it->glyphless_method = GLYPHLESS_DISPLAY_HEX_CODE;
+    {
+      record_tofu (it);
+      it->glyphless_method = GLYPHLESS_DISPLAY_HEX_CODE;
+    }
   else if (STRINGP (glyphless_method))
     it->glyphless_method = GLYPHLESS_DISPLAY_ACRONYM;
   else
@@ -15514,6 +15529,7 @@ #define RESUME_POLLING					\
 static int redisplay_window_wcounter;
 static int redisplay_window_bcounter;
 static int redisplay_window_fcounter;
+static Lisp_Object redisplayed_windows;
 
 /* Perhaps in the future avoid recentering windows if it
    is not necessary; currently that causes some problems.  */
@@ -16016,9 +16032,13 @@ #define AINC(a,i)							\
   redisplay_window_bcounter = 0;
   redisplay_window_fcounter = 0;
   redisplay_window_wcounter = 0;
+  redisplayed_windows = Qt;
 
   if (consider_all_windows_p)
     {
+      if (REDISPLAY_SOME_P ())
+        redisplayed_windows = Qnil;
+
       FOR_EACH_FRAME (tail, frame)
 	XFRAME (frame)->updated_p = false;
 
@@ -16328,6 +16348,10 @@ #define AINC(a,i)							\
     request_sigio ();
   RESUME_POLLING;
 
+  if (FUNCTIONP (Vpost_redisplay_function))
+    safe__call1 (true, Vpost_redisplay_function,
+                 consider_all_windows_p ? redisplayed_windows : Qnil);
+
   /* If a frame has become visible which was not before, redisplay
      again, so that we display it.  Expose events for such a frame
      (which it gets when becoming visible) don't call the parts of
@@ -18422,6 +18446,9 @@ redisplay_window (Lisp_Object window, bool just_this_one_p)
       && BUF_PT (buffer) == w->last_point)
     return;
 
+  if (!EQ (redisplayed_windows, Qt))
+    redisplayed_windows = Fcons (window, redisplayed_windows);
+
   if (w->redisplay)
     redisplay_window_wcounter++;
   if (f->redisplay)
@@ -29696,7 +29723,7 @@ compute_relative_width (struct it *it, Lisp_Object prop)
 
   it2 = *it;
   if (it->multibyte_p)
-    it2.c = it2.char_to_display = string_char_and_length (p, it2.len);
+    it2.c = it2.char_to_display = string_char_and_length (p, &it2.len);
   else
     {
       it2.c = it2.char_to_display = *p, it2.len = 1;
@@ -34683,6 +34710,9 @@ syms_of_xdisp (void)
   Vmessage_stack = Qnil;
   staticpro (&Vmessage_stack);
 
+  redisplayed_windows = Qt;
+  staticpro (&redisplayed_windows);
+
   /* Non-nil means don't actually do any redisplay.  */
   DEFSYM (Qinhibit_redisplay, "inhibit-redisplay");
 
@@ -35544,6 +35574,13 @@ syms_of_xdisp (void)
 or t (meaning all windows).  */);
   Vpre_redisplay_function = intern ("ignore");
 
+  DEFVAR_LISP ("post-redisplay-function", Vpost_redisplay_function,
+	       doc: /* Function run just after redisplay.
+It is called with one argument, which is the set of windows that have been
+redisplayed.  This set can be nil (meaning, only the selected window),
+or t (meaning all windows).  */);
+  Vpost_redisplay_function = intern ("ignore");
+
   /* Symbol for the purpose of Vglyphless_char_display.  */
   DEFSYM (Qglyphless_char_display, "glyphless-char-display");
   Fput (Qglyphless_char_display, Qchar_table_extra_slots, make_fixnum (1));
@@ -35614,6 +35651,10 @@ syms_of_xdisp (void)
 	       doc: /*  */);
   Vredisplay__touched_fcounts = Fmake_hash_table (0, NULL);
 
+  DEFVAR_LISP ("redisplay-tofu", Vredisplay_tofu,
+               doc: /*  */);
+  Vredisplay_tofu = Qt;
+
   DEFVAR_BOOL ("redisplay--inhibit-bidi", redisplay__inhibit_bidi,
      doc: /* Non-nil means it is not safe to attempt bidi reordering for display.  */);
   /* Initialize to t, since we need to disable reordering until

Re: Suggest installing more fonts?

[Juri Linkov]

>> I've been frustrated by this before, and I've resorted to saying "apt
>> install fonts-*' or something, which isn't really ideal, so if Emacs
>> could provide some guidance here (without us having to maintain a
>> never-ending list of fonts), I think that would be helpful.
>>
> See recent discussion in bug#43887. In order to see all symbols in
> etc/HELLO, I had to apply on Ubuntu 20.04
>
> $ sudo apt-get install fonts-noto-core

I found this package by running:

  apt-cache search viet | grep font

that outputs:

  fonts-noto-core - "No Tofu" font families with large Unicode coverage (core)

The same result is with:

  apt-cache search javanese | grep font

I have no idea why `apt-cache search` searches in package
descriptions whereas `apt search` searches only in package names.
These apt command names have no indications that
they search in different places!

Re: Suggest installing more fonts?

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Michael Albinus <michael.albinus@gmx.de>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
>> Date: Wed, 21 Oct 2020 13:31:01 +0200
>>
>> In Emacs, the file INSTALL recommends to install
>> intlfonts-VERSION.tar.gz. Might be sufficient, but without further
>> instruction I doubt every Emacs user will be able to follow.
>
> intlfonts are bitmapped (BDF) fonts, so I don't think nowadays they
> will be considered as a satisfactory solution.  They are GNU software,
> but otherwise I see no reason to advertise them today.
>
>> And/or better instructions in the INSTALL file.
>
> Yes, for those who still read those files.

Installation of intlfonts is described in efaq.texi. If we still want to
mention them in INSTALL, we better add a reference?

I've seen that you're not in favor to mention Google Noto fonts here. But
maybe there could be a hint in efaq.texi? On
<https://www.google.com/get/noto/help/install/>, there are also
instructions how to install them for Windows and macOS.

Best regards, Michael.

Re: Suggest installing more fonts?

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: larsi@gnus.org,  emacs-devel@gnu.org
> Date: Wed, 21 Oct 2020 19:46:11 +0200
> 
> >> And/or better instructions in the INSTALL file.
> >
> > Yes, for those who still read those files.
> 
> Installation of intlfonts is described in efaq.texi. If we still want to
> mention them in INSTALL, we better add a reference?

??? It's already in INSTALL.

> I've seen that you're not in favor to mention Google Noto fonts here. But
> maybe there could be a hint in efaq.texi? On
> <https://www.google.com/get/noto/help/install/>, there are also
> instructions how to install them for Windows and macOS.

Any reason why we should prefer Google Noto to other capable fonts?

Re: Suggest installing more fonts?

[Rasmus]

Eli Zaretskii <eliz@gnu.org> writes:

> Also, if this is the way we want to go, we should have separate 
> advice for MS-Windows users, where there are several system 
> specific issues regarding font search by Emacs, and many users 
> are unable to install fonts system-wide due to fascist 
> corporation policies. 

Is there a way to make fonts available to Emacs without installing 
them on Windows?  On my work laptop I can’t install fonts, but 
would like to have broader unicode coverage in Emacs.

Kind regards,
Rasmus

-- 
Slowly unravels in a ball of yarn and the devil collects it

Re: Suggest installing more fonts?

[Rasmus]

Eli Zaretskii <eliz@gnu.org> writes:

> Also, if this is the way we want to go, we should have separate 
> advice for MS-Windows users, where there are several system 
> specific issues regarding font search by Emacs, and many users 
> are unable to install fonts system-wide due to fascist 
> corporation policies. 

Is there a way to make fonts available to Emacs without installing 
them on Windows?  On my work laptop I can’t install fonts, but 
would like to have broader unicode coverage in Emacs.

Kind regards,
Rasmus

-- 
Slowly unravels in a ball of yarn and the devil collects it

Re: Suggest installing more fonts?

[Eli Zaretskii]

> From: Rasmus <rasmus@gmx.us>
> Date: Wed, 21 Oct 2020 22:19:14 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Also, if this is the way we want to go, we should have separate 
> > advice for MS-Windows users, where there are several system 
> > specific issues regarding font search by Emacs, and many users 
> > are unable to install fonts system-wide due to fascist 
> > corporation policies. 
> 
> Is there a way to make fonts available to Emacs without installing 
> them on Windows?  On my work laptop I can’t install fonts, but 
> would like to have broader unicode coverage in Emacs.

"Can't install" even only for your user?

Re: Suggest installing more fonts?

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

Hi Eli,

>> Installation of intlfonts is described in efaq.texi. If we still want to
>> mention them in INSTALL, we better add a reference?
>
> ??? It's already in INSTALL.

I haven't see any reference to efaq.texi in INSTALL.

>> I've seen that you're not in favor to mention Google Noto fonts here. But
>> maybe there could be a hint in efaq.texi? On
>> <https://www.google.com/get/noto/help/install/>, there are also
>> instructions how to install them for Windows and macOS.
>
> Any reason why we should prefer Google Noto to other capable fonts?

No reason. I just wanted to show how to install such fonts for major
GNU/Linux distros, using apt or dnf. And maybe also for other target systems.

Best regards, Michael.

Re: Suggest installing more fonts?

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Uninstall Unifont (which I thought we agreed shouldn't be used), then
> try again.

And now the HELLO file looks even nicer.  :-)

> And are we okay with telling people to install Noto fonts?  There are
> a lot of fonts in that collection, I'm not sure they are really
> needed.  For example, do modern GNU/Linux systems really come without
> CJK fonts?

"Install noto-core" is simple advice to give to users, and will fix
their problems, so I don't see why not.

> Also, if this is the way we want to go, we should have separate advice
> for MS-Windows users, where there are several system specific issues
> regarding font search by Emacs, and many users are unable to install
> fonts system-wide due to fascist corporation policies.

Sure.  Like I said, it depends how much work we want to put into this.
We don't need to have a perfect list of all scripts/systems, just
some...  guidance...  for the systems we decide to care about.

> That is why I would propose a general advice that doesn't name
> specific fonts.  It is easy to find fonts for a known script nowadays.

Not my experience on Debian systems, really.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Suggest installing more fonts?

[Gregory Heytings via Emacs development discussions.]

>> That is why I would propose a general advice that doesn't name specific 
>> fonts.  It is easy to find fonts for a known script nowadays.
>
> Not my experience on Debian systems, really.
>

I don't understand this, "apt-cache search <script> font" or "apt search 
<script> font" work for (almost?) every <script>, including tagbanwa, 
sinhala, lepcha, bhaiksuki... which I'd never heard about.

Re: tofu-help-mode

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> BTW, for those interested, here's a proof-of-concept.
>
> I don't intend to work much more on this (except maybe for the
> `post-redisplay-functions` part which might be useful on its own), so
> I'd encourage someone else to take it over from here if they'd like to
> see it turn into something usable.

Reading the code, it looks like everything is there, except expanding on
this:

> +          (put-text-property pos (1+ pos) 'help-echo
> +                             (or (get-char-code-property char 'name)
> +		                 (get-char-code-property char 'old-name)
> +		                 "This is unknown TOFU!")))))))

And adding some documentation and NEWS and stuff, of course.  If you
push this to the trunk, I can take it from there...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Suggest installing more fonts?

[Lars Ingebrigtsen]

Gregory Heytings <ghe@sdf.org> writes:

> I don't understand this, "apt-cache search <script> font" or "apt
> search <script> font" work for (almost?) every <script>, including
> tagbanwa, sinhala, lepcha, bhaiksuki... which I'd never heard about.

larsi@xo:~/src/emacs/trunk$ apt-cache search vietnamese | grep font
fonts-pecita - OpenType hand-written font whose letters are connected
emacs-intl-fonts - fonts to allow multilingual PostScript printing from Emacs
xfonts-intl-asian - international fonts for X - (south-east) Asian
fonts-texgyre - OpenType fonts based on URW Fonts
fonts-ibm-plex - extensive typeface family designed by IBM

None of these are actually the package you'd want to install.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Suggest installing more fonts?

[Gregory Heytings via Emacs development discussions.]

>> I don't understand this, "apt-cache search <script> font" or "apt 
>> search <script> font" work for (almost?) every <script>, including 
>> tagbanwa, sinhala, lepcha, bhaiksuki... which I'd never heard about.
>
> larsi@xo:~/src/emacs/trunk$ apt-cache search vietnamese | grep font
> fonts-pecita - OpenType hand-written font whose letters are connected
> emacs-intl-fonts - fonts to allow multilingual PostScript printing from Emacs
> xfonts-intl-asian - international fonts for X - (south-east) Asian
> fonts-texgyre - OpenType fonts based on URW Fonts
> fonts-ibm-plex - extensive typeface family designed by IBM
>
> None of these are actually the package you'd want to install.
>

I don't know, I don't read/write vietnamese.  But I don't understand why 
you write that "none of these are the package you'd want to install", 
fonts-ibm-plex seems a good candidate.  And from what I see in HELLO, 
Vietnamese uses the latin script, with added diacritics, so I guess 
there's in fact no need to install additional fonts to read/write 
Vietnamese.  Viet is something else, it uses the tai-viet script, and:

$ apt-cache search viet font
fonts-pecita - OpenType hand-written font whose letters are connected
emacs-intl-fonts - fonts to allow multilingual PostScript printing from Emacs
xfonts-intl-asian - international fonts for X - (south-east) Asian
fonts-texgyre - OpenType fonts based on URW Fonts
fonts-noto-core - "No Tofu" font families with large Unicode coverage (core)
fonts-sil-taiheritagepro - typeface reflecting the traditional hand-written style of the Tai Viet script
texlive-lang-other - TeX Live: Other languages
fonts-ibm-plex - extensive typeface family designed by IBM

in which you find fonts-noto-core.

Re: Suggest installing more fonts?

[Rasmus]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Rasmus <rasmus@gmx.us> Date: Wed, 21 Oct 2020 22:19:14 
>> +0200  
> 
>> Eli Zaretskii <eliz@gnu.org> writes:  
>> > Also, if this is the way we want to go, we should have 
>> > separate  advice for MS-Windows users, where there are 
>> > several system  specific issues regarding font search by 
>> > Emacs, and many users  are unable to install fonts 
>> > system-wide due to fascist  corporation policies.  
>>  Is there a way to make fonts available to Emacs without 
>> installing  them on Windows?  On my work laptop I can’t install 
>> fonts, but  would like to have broader unicode coverage in 
>> Emacs. 
> 
> "Can't install" even only for your user? 

I have not found a way on Win7, which seems to still be pretty 
widespread on corporate laptops.  I understand that Win10 might 
have user-fonts.  (I am not a Windows expert, though).

Rasmus

-- 
Don't slow down Johnny, leave the Cadillac runnin'

Re: Suggest installing more fonts?

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: larsi@gnus.org,  emacs-devel@gnu.org
> Date: Thu, 22 Oct 2020 09:27:38 +0200
> 
> >> Installation of intlfonts is described in efaq.texi. If we still want to
> >> mention them in INSTALL, we better add a reference?
> >
> > ??? It's already in INSTALL.
> 
> I haven't see any reference to efaq.texi in INSTALL.

Sorry, I thought you meant intlfonts, not efaq.  So now could you
elaborate on what kind of reference did you have in mind and to what
information?

> >> I've seen that you're not in favor to mention Google Noto fonts here. But
> >> maybe there could be a hint in efaq.texi? On
> >> <https://www.google.com/get/noto/help/install/>, there are also
> >> instructions how to install them for Windows and macOS.
> >
> > Any reason why we should prefer Google Noto to other capable fonts?
> 
> No reason. I just wanted to show how to install such fonts for major
> GNU/Linux distros, using apt or dnf. And maybe also for other target systems.

We could also use a neutral name like "apt get FONT ...", couldn't we?

Re: Suggest installing more fonts?

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Thu, 22 Oct 2020 13:32:09 +0200
> Cc: emacs-devel@gnu.org
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Uninstall Unifont (which I thought we agreed shouldn't be used), then
> > try again.
> 
> And now the HELLO file looks even nicer.  :-)

But how many fonts were loaded for that?

AFAIK, an OpenType font can have at most 64K glyphs, so it is very
unusual for a high-quality font to cover too many Unicode blocks.
Even Unifont only supports the BMP and little else.

Re: Suggest installing more fonts?

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
> Date: Thu, 22 Oct 2020 14:08:47 +0200
> 
> Gregory Heytings <ghe@sdf.org> writes:
> 
> > I don't understand this, "apt-cache search <script> font" or "apt
> > search <script> font" work for (almost?) every <script>, including
> > tagbanwa, sinhala, lepcha, bhaiksuki... which I'd never heard about.
> 
> larsi@xo:~/src/emacs/trunk$ apt-cache search vietnamese | grep font

There's no "vietnamese" script.

And why don't you use the most efficient tool there is?  Try

  M-s M-w vietnamese font RET

and the first several hits are what you need.

> None of these are actually the package you'd want to install.

Why would someone need to install anything for Vietnamese?  It uses
the Latin script, so should be supported OOTB.

Re: Suggest installing more fonts?

[Eli Zaretskii]

> From: Rasmus <rasmus@gmx.us>
> Cc: emacs-devel@gnu.org
> Date: Thu, 22 Oct 2020 14:34:36 +0200
> 
> > "Can't install" even only for your user? 
> 
> I have not found a way on Win7, which seems to still be pretty 
> widespread on corporate laptops.  I understand that Win10 might 
> have user-fonts.  (I am not a Windows expert, though).

For Windows 7 you will need some non-standard program.  There are a
few floating around.

Re: Suggest installing more fonts?

[Gregory Heytings via Emacs development discussions.]

>>> Uninstall Unifont (which I thought we agreed shouldn't be used), then 
>>> try again.
>>
>> And now the HELLO file looks even nicer.  :-)
>
> But how many fonts were loaded for that?
>

DejaVu Sans Mono is a single font, but fonts-noto-core alone contains 162 
fonts.

>
> Even Unifont only supports the BMP and little else.
>

That's wrong.  Unifont supports 100% of the BMP (including many scripts 
for fictional scripts such as Klingon, which are not offically part of 
Unicode), most of the SMP (the exceptions are cuneiform and hieroglyphs), 
a few glyphs of SIP.

Re: Suggest installing more fonts?

[Eli Zaretskii]

> Date: Thu, 22 Oct 2020 13:40:35 +0000
> From: Gregory Heytings <ghe@sdf.org>
> cc: Lars Ingebrigtsen <larsi@gnus.org>, emacs-devel@gnu.org
> 
> > Even Unifont only supports the BMP and little else.
> >
> 
> That's wrong.  Unifont supports 100% of the BMP (including many scripts 
> for fictional scripts such as Klingon, which are not offically part of 
> Unicode), most of the SMP (the exceptions are cuneiform and hieroglyphs), 
> a few glyphs of SIP.

Which is essentially what I said.

Re: Suggest installing more fonts?

[Gregory Heytings via Emacs development discussions.]

>>> Even Unifont only supports the BMP and little else.
>>
>> That's wrong.  Unifont supports 100% of the BMP (including many scripts 
>> for fictional scripts such as Klingon, which are not offically part of 
>> Unicode), most of the SMP (the exceptions are cuneiform and 
>> hieroglyphs), a few glyphs of SIP.
>
> Which is essentially what I said.
>

Aha.  So the 80%-90% coverage of the SMP (the exceptions being cuneiform 
and hieroglyphs) by Unifont is essentially nothing?  And its ~8000 glyphs 
for fictional scripts is essentially nothing?  All in all, Unifont 
contains all glyphs needed by about 99.95% of the world population.  It's 
more than clear that you dislike Unifont, you are entitled to your 
opinions, but I believe it is important to do justice to what others did.

Re: tofu-help-mode

[Stefan Monnier]

>> I don't intend to work much more on this (except maybe for the
>> `post-redisplay-functions` part which might be useful on its own), so
>> I'd encourage someone else to take it over from here if they'd like to
>> see it turn into something usable.
> Reading the code, it looks like everything is there, except expanding on
> this:

No: the code I sent only offers a buffer-local minor mode and more
changes will be required if we want it to work as a global minor mode
(which I'd expect is what you'd usually want).


        Stefan

Re: Suggest installing more fonts?

[Eli Zaretskii]

> Date: Thu, 22 Oct 2020 14:20:37 +0000
> From: Gregory Heytings <ghe@sdf.org>
> cc: larsi@gnus.org, emacs-devel@gnu.org
> 
> >>> Even Unifont only supports the BMP and little else.
> >>
> >> That's wrong.  Unifont supports 100% of the BMP (including many scripts 
> >> for fictional scripts such as Klingon, which are not offically part of 
> >> Unicode), most of the SMP (the exceptions are cuneiform and 
> >> hieroglyphs), a few glyphs of SIP.
> >
> > Which is essentially what I said.
> 
> Aha.  So the 80%-90% coverage of the SMP (the exceptions being cuneiform 
> and hieroglyphs) by Unifont is essentially nothing?

It isn't nothing, but it is a small fraction of the existing
codepoints, let alone those which could/will be defined in the future.
A single font cannot support more than 64K glyphs, which is a small
number compared to the Unicode range of codes.

Re: Suggest installing more fonts?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Why would someone need to install anything for Vietnamese?  It uses
  > the Latin script, so should be supported OOTB.

I don't know whether it is, but I do know that Vietnamese
uses diacritical marks that other languages don't use.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Suggest installing more fonts?

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: larsi@gnus.org, ghe@sdf.org, emacs-devel@gnu.org
> Date: Thu, 22 Oct 2020 23:49:53 -0400
> 
>   > Why would someone need to install anything for Vietnamese?  It uses
>   > the Latin script, so should be supported OOTB.
> 
> I don't know whether it is, but I do know that Vietnamese
> uses diacritical marks that other languages don't use.

Fonts support scripts, not languages.  A font that supports the Latin
script will normally support the diacriticals needed for Vietnamese.

Re: Suggest installing more fonts?

[Andreas Schwab]

On Okt 23 2020, Eli Zaretskii wrote:

> Fonts support scripts, not languages.  A font that supports the Latin
> script will normally support the diacriticals needed for Vietnamese.

But they may not support the combining character for the NFD form.  I
often see non-combined diacriticals in NFD text because the combining
character comes from a different font than the default one.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Suggest installing more fonts?

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> There's no "vietnamese" script.
>
> And why don't you use the most efficient tool there is?  Try
>
>   M-s M-w vietnamese font RET
>
> and the first several hits are what you need.

None of them say anything about how to fix the problem on a Debian
system, as far as I can tell.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: tofu-help-mode

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> No: the code I sent only offers a buffer-local minor mode and more
> changes will be required if we want it to work as a global minor mode
> (which I'd expect is what you'd usually want).

Isn't that just a small matter of programming?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Suggest installing more fonts?

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: rms@gnu.org,  ghe@sdf.org,  larsi@gnus.org,  emacs-devel@gnu.org
> Date: Fri, 23 Oct 2020 10:28:50 +0200
> 
> On Okt 23 2020, Eli Zaretskii wrote:
> 
> > Fonts support scripts, not languages.  A font that supports the Latin
> > script will normally support the diacriticals needed for Vietnamese.
> 
> But they may not support the combining character for the NFD form.  I
> often see non-combined diacriticals in NFD text because the combining
> character comes from a different font than the default one.

Any specific examples?  Does that happen with modern fonts with recent
enough version?

Re: Suggest installing more fonts?

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: ghe@sdf.org,  emacs-devel@gnu.org
> Date: Fri, 23 Oct 2020 12:45:49 +0200
> 
> >   M-s M-w vietnamese font RET
> >
> > and the first several hits are what you need.
> 
> None of them say anything about how to fix the problem on a Debian
> system, as far as I can tell.

You mean, a Debian user cannot simply download font files and install
them?  Are Debian users forced to use "apt get" etc.?  I'm surprised.

Re: Suggest installing more fonts?

[Stefan Monnier]

> You mean, a Debian user cannot simply download font files and install
> them?  Are Debian users forced to use "apt get" etc.?  I'm surprised.

As a Debian user, I do think that I can "download font files and install
them" and there's a good chance I can even do that "simply", but I have
never needed to do that and would have to search for info about how to
do it and I wouldn't be satisfied with the result because now those
fonts wouldn't be kept up-to-date by the usual mechanisms.

So, while I'm not "forced" to use `apt`, I would first and foremost want
to use `apt` and only resort to another solution if there's no
other solution.


        Stefan

Re: Suggest installing more fonts?

[Andreas Schwab]

On Okt 23 2020, Eli Zaretskii wrote:

> Any specific examples?

ñ

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Suggest installing more fonts?

[Stefan Monnier]

>> Any specific examples?
> ñ

Funny: here I see it non-composed (i.e. occupies 2 columns) when
auto-composition-mode is enabled and I see it composed when
auto-composition-mode is disabled!


        Stefan

Re: Suggest installing more fonts?

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

Hi Eli,

>> >> Installation of intlfonts is described in efaq.texi. If we still want to
>> >> mention them in INSTALL, we better add a reference?
>> >
>> > ??? It's already in INSTALL.
>>
>> I haven't see any reference to efaq.texi in INSTALL.
>
> Sorry, I thought you meant intlfonts, not efaq.  So now could you
> elaborate on what kind of reference did you have in mind and to what
> information?

Something like

See the Emacs Frequently Asked Questions manual "(efaq) How to add
fonts" for installation instructions.

>> No reason. I just wanted to show how to install such fonts for major
>> GNU/Linux distros, using apt or dnf. And maybe also for other target systems.
>
> We could also use a neutral name like "apt get FONT ...", couldn't we?

Hmm. The problem is, that the majority of users doesn't know what to
replace FONT with.

Best regards, Michael.

Re: Suggest installing more fonts?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Lars Ingebrigtsen <larsi@gnus.org>,  ghe@sdf.org,  emacs-devel@gnu.org
> Date: Fri, 23 Oct 2020 10:59:02 -0400
> 
> > You mean, a Debian user cannot simply download font files and install
> > them?  Are Debian users forced to use "apt get" etc.?  I'm surprised.
> 
> As a Debian user, I do think that I can "download font files and install
> them" and there's a good chance I can even do that "simply", but I have
> never needed to do that and would have to search for info about how to
> do it and I wouldn't be satisfied with the result because now those
> fonts wouldn't be kept up-to-date by the usual mechanisms.
> 
> So, while I'm not "forced" to use `apt`, I would first and foremost want
> to use `apt` and only resort to another solution if there's no
> other solution.

So Debian users are even less "free" in this regard than MS-Windows
users?  That's sad, isn't it?

Re: Suggest installing more fonts?

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: rms@gnu.org,  ghe@sdf.org,  larsi@gnus.org,  emacs-devel@gnu.org
> Date: Fri, 23 Oct 2020 17:50:37 +0200
> 
> On Okt 23 2020, Eli Zaretskii wrote:
> 
> > Any specific examples?
> 
> ñ

Which Latin fonts on modern systems don't support the COMBINING TILDE?

Re: Suggest installing more fonts?

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: larsi@gnus.org,  emacs-devel@gnu.org
> Date: Fri, 23 Oct 2020 18:53:17 +0200
> 
> > Sorry, I thought you meant intlfonts, not efaq.  So now could you
> > elaborate on what kind of reference did you have in mind and to what
> > information?
> 
> Something like
> 
> See the Emacs Frequently Asked Questions manual "(efaq) How to add
> fonts" for installation instructions.

OK, except "manual" is wrong here.  The FAQ is not a manual.

> >> No reason. I just wanted to show how to install such fonts for major
> >> GNU/Linux distros, using apt or dnf. And maybe also for other target systems.
> >
> > We could also use a neutral name like "apt get FONT ...", couldn't we?
> 
> Hmm. The problem is, that the majority of users doesn't know what to
> replace FONT with.

But then the instructions you show cannot be applied if the name of
the font changes?

Re: Suggest installing more fonts?

[Stefan Monnier]

>> As a Debian user, I do think that I can "download font files and install
>> them" and there's a good chance I can even do that "simply", but I have
>> never needed to do that and would have to search for info about how to
>> do it and I wouldn't be satisfied with the result because now those
>> fonts wouldn't be kept up-to-date by the usual mechanisms.
>> 
>> So, while I'm not "forced" to use `apt`, I would first and foremost want
>> to use `apt` and only resort to another solution if there's no
>> other solution.
>
> So Debian users are even less "free" in this regard than MS-Windows users?

Not sure what you mean by that.

Basically, what I was saying is that "simply downloading and installing"
(the fact that it was applied to fonts doesn't make much difference in
this respect) sounds to me like pulling my own teeth: I'm "free" to do
it, but I would look for other solutions first.


        Stefan

Re: Suggest installing more fonts?

[Gregory Heytings via Emacs development discussions.]

>>> You mean, a Debian user cannot simply download font files and install 
>>> them?  Are Debian users forced to use "apt get" etc.?  I'm surprised.
>>
>> As a Debian user, I do think that I can "download font files and 
>> install them" and there's a good chance I can even do that "simply", 
>> but I have never needed to do that and would have to search for info 
>> about how to do it and I wouldn't be satisfied with the result because 
>> now those fonts wouldn't be kept up-to-date by the usual mechanisms.
>>
>> So, while I'm not "forced" to use `apt`, I would first and foremost 
>> want to use `apt` and only resort to another solution if there's no 
>> other solution.
>
> So Debian users are even less "free" in this regard than MS-Windows 
> users?  That's sad, isn't it?
>

Of course Debian users can simply download font files and install them: it 
suffices to put them in the /usr/share/fonts/truetype or 
/usr/share/fonts/opentype directory .  Fonts can also be added with KDE 
Font Management or GNOME Font Manager.  As Stefan said, the drawback of 
this solution (be it with GNU/Linux, Windows or macOS) is that "those 
fonts aren't kept up-to-date by the usual mechanisms".

Re: Suggest installing more fonts?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: larsi@gnus.org,  ghe@sdf.org,  emacs-devel@gnu.org
> Date: Fri, 23 Oct 2020 14:13:24 -0400
> 
> >> So, while I'm not "forced" to use `apt`, I would first and foremost want
> >> to use `apt` and only resort to another solution if there's no
> >> other solution.
> >
> > So Debian users are even less "free" in this regard than MS-Windows users?
> 
> Not sure what you mean by that.
> 
> Basically, what I was saying is that "simply downloading and installing"
> (the fact that it was applied to fonts doesn't make much difference in
> this respect) sounds to me like pulling my own teeth: I'm "free" to do
> it, but I would look for other solutions first.

Which is sub-optimal, since it sounds like the facilities to search
the DB of installable packages is limited, whereas the facilities we
have for finding fonts on the Internet are very powerful and accurate.
So the ability to download a font and install it sounds very useful.

Re: Suggest installing more fonts?

[Stefan Monnier]

> Which is sub-optimal, since it sounds like the facilities to search
> the DB of installable packages is limited, whereas the facilities we
> have for finding fonts on the Internet are very powerful and accurate.
> So the ability to download a font and install it sounds very useful.

Oh, the escape hatch exists.  It's just that Debian users normally
prefer to use the other, more controlled (and hence limited) way.


        Stefan

Re: Suggest installing more fonts?

[Robert Pluim]

>>>>> On Fri, 23 Oct 2020 21:03:48 +0300, Eli Zaretskii <eliz@gnu.org> said:

    >> From: Andreas Schwab <schwab@linux-m68k.org>
    >> Cc: rms@gnu.org,  ghe@sdf.org,  larsi@gnus.org,  emacs-devel@gnu.org
    >> Date: Fri, 23 Oct 2020 17:50:37 +0200
    >> 
    >> On Okt 23 2020, Eli Zaretskii wrote:
    >> 
    >> > Any specific examples?
    >> 
    >> ñ

    Eli> Which Latin fonts on modern systems don't support the COMBINING TILDE?

Jetbrains Mono for one (although I haven't updated it in a while)

Robert
-- 

Re: Suggest installing more fonts?

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: Andreas Schwab <schwab@linux-m68k.org>,  ghe@sdf.org,  larsi@gnus.org,
>   rms@gnu.org,  emacs-devel@gnu.org
> Date: Fri, 23 Oct 2020 20:43:29 +0200
> 
>     >> ñ
> 
>     Eli> Which Latin fonts on modern systems don't support the COMBINING TILDE?
> 
> Jetbrains Mono for one (although I haven't updated it in a while)

Is that font likely to be used by Emacs by default?

Re: Suggest installing more fonts?

[Robert Pluim]

>>>>> On Fri, 23 Oct 2020 22:31:26 +0300, Eli Zaretskii <eliz@gnu.org> said:

    >> From: Robert Pluim <rpluim@gmail.com>
    >> Cc: Andreas Schwab <schwab@linux-m68k.org>,  ghe@sdf.org,  larsi@gnus.org,
    >> rms@gnu.org,  emacs-devel@gnu.org
    >> Date: Fri, 23 Oct 2020 20:43:29 +0200
    >> 
    >> >> ñ
    >> 
    Eli> Which Latin fonts on modern systems don't support the COMBINING TILDE?
    >> 
    >> Jetbrains Mono for one (although I haven't updated it in a while)

    Eli> Is that font likely to be used by Emacs by default?

No. Itʼs fairly popular, but users need to decide to install and use
it themselves.

Robert
-- 

Re: Suggest installing more fonts?

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

Hi Eli,

>> See the Emacs Frequently Asked Questions manual "(efaq) How to add
>> fonts" for installation instructions.
>
> OK, except "manual" is wrong here.  The FAQ is not a manual.

I've added it slightly rephrased to the INSTALL file. Pushed to the
emacs-27 branch.

>> >> No reason. I just wanted to show how to install such fonts for major
>> >> GNU/Linux distros, using apt or dnf. And maybe also for other
>> >> target systems.
>> >
>> > We could also use a neutral name like "apt get FONT ...", couldn't we?
>>
>> Hmm. The problem is, that the majority of users doesn't know what to
>> replace FONT with.
>
> But then the instructions you show cannot be applied if the name of
> the font changes?

Yes, we can only give examples of font packages for Debian and Red Hat
(and maybe other major distributions). Likely, this is too special for
the INSTALL file, and we shall add this information to efaq.texi?

Best regards, Michael.

Re: Suggest installing more fonts?

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: larsi@gnus.org,  emacs-devel@gnu.org
> Date: Sun, 25 Oct 2020 12:47:40 +0100
> 
> >> > We could also use a neutral name like "apt get FONT ...", couldn't we?
> >>
> >> Hmm. The problem is, that the majority of users doesn't know what to
> >> replace FONT with.
> >
> > But then the instructions you show cannot be applied if the name of
> > the font changes?
> 
> Yes, we can only give examples of font packages for Debian and Red Hat
> (and maybe other major distributions). Likely, this is too special for
> the INSTALL file, and we shall add this information to efaq.texi?

Yes, I think so.