displaying missing glyphs

displaying missing glyphs

[Leo Butler]

I use `emacs -nw` inside of screen inside of uxterm. Unfortunately, many
unicode glyphs are not displayed correctly (although they are if I
attach the screen session in gnome-terminal, for example).

In emacs/elisp, how might I override the default empty box to display
something more informative?

TIA,
Leo

Re: displaying missing glyphs

[Eli Zaretskii]

> From: Leo Butler <leo.butler@umanitoba.ca>
> Date: Fri, 09 Apr 2021 11:32:25 -0500
> 
> I use `emacs -nw` inside of screen inside of uxterm. Unfortunately, many
> unicode glyphs are not displayed correctly (although they are if I
> attach the screen session in gnome-terminal, for example).
> 
> In emacs/elisp, how might I override the default empty box to display
> something more informative?

Not enough information to answer your questions.  Does your terminal
support UTF-8 encoding?  What is your locale?  And what do you mean by
"not displayed correctly"?

Re: displaying missing glyphs

[Andreas Eder]

On Fr 09 Apr 2021 at 11:32, Leo Butler <leo.butler@umanitoba.ca> wrote:

> I use `emacs -nw` inside of screen inside of uxterm. Unfortunately, many
> unicode glyphs are not displayed correctly (although they are if I
> attach the screen session in gnome-terminal, for example).
>
> In emacs/elisp, how might I override the default empty box to display
> something more informative?

Well, uxterm is unicode capable and wether inside if screen or not
shouldn't be of any concern.
Anyway, your combination works perfectly well here.
So - I guess = the only reason it works insode gnome term, but not with
uxterm os the fonts used. Zou should make sure that uxterm uses a
unicode capable font with glyphs for the language you want to display.
Try the same one for uxterm that is used by gnome-terminal and it
should work.

'Andreas

Re: displaying missing glyphs

[Andreas Eder]

On Fr 09 Apr 2021 at 11:32, Leo Butler <leo.butler@umanitoba.ca> wrote:

> I use `emacs -nw` inside of screen inside of uxterm. Unfortunately, many
> unicode glyphs are not displayed correctly (although they are if I
> attach the screen session in gnome-terminal, for example).
>
> In emacs/elisp, how might I override the default empty box to display
> something more informative?

The problem is - most likely - a font that is not unicode capable.
If you set uxterm to ise the same font as gnome-terminal then it should
work.
The same combination (uxterm, screen and emacs) works perfectly well
here.

'Andreas

Re: displaying missing glyphs

[Leo Butler]

[-- Attachment #1: Type: text/plain, Size: 1220 bytes --]

Andreas Eder <a_eder_muc@web.de> writes:

> On Fr 09 Apr 2021 at 11:32, Leo Butler <leo.butler@umanitoba.ca> wrote:
>
>> I use `emacs -nw` inside of screen inside of uxterm. Unfortunately, many
>> unicode glyphs are not displayed correctly (although they are if I
>> attach the screen session in gnome-terminal, for example).
>>
>> In emacs/elisp, how might I override the default empty box to display
>> something more informative?
>
> The problem is - most likely - a font that is not unicode capable.
> If you set uxterm to ise the same font as gnome-terminal then it should
> work.
> The same combination (uxterm, screen and emacs) works perfectly well
> here.

Thanks for the suggestion. I have attached a marked-up screen shot of an
xterm (left) and gnome-terminal running `emacsclient -nw` and showing
the same buffer. You can see there is a noticeable clipping of some of
the characters in the xterm.

According to lsof, gnome-terminal is using

 /usr/share/fonts/truetype/dejavu/DejaVuSansMono.ttf

so the xterm has been run using

xterm -fa 'DejaVuSansMono' -fs 9

(and all font-related options are commented out in ~/.Xdefaults).

FWIW, this is on a debian testing system with XTERM_LOCALE=en_US.UTF-8.

Leo


[-- Attachment #2: clipping.png --]
[-- Type: image/png, Size: 203043 bytes --]

Re: displaying missing glyphs

[2QdxY4RzWzUUiLuE]

On 2021-04-12 at 12:08:08 -0500,
Regarding "Re: displaying missing glyphs,"
Leo Butler <leo.butler@umanitoba.ca> wrote:

> Andreas Eder <a_eder_muc@web.de> writes:
> 
> > On Fr 09 Apr 2021 at 11:32, Leo Butler <leo.butler@umanitoba.ca> wrote:
> >
> >> I use `emacs -nw` inside of screen inside of uxterm. Unfortunately, many
> >> unicode glyphs are not displayed correctly (although they are if I
> >> attach the screen session in gnome-terminal, for example).
> >>
> >> In emacs/elisp, how might I override the default empty box to display
> >> something more informative?
> >
> > The problem is - most likely - a font that is not unicode capable.
> > If you set uxterm to ise the same font as gnome-terminal then it should
> > work.
> > The same combination (uxterm, screen and emacs) works perfectly well
> > here.
> 
> Thanks for the suggestion. I have attached a marked-up screen shot of an
> xterm (left) and gnome-terminal running `emacsclient -nw` and showing
> the same buffer. You can see there is a noticeable clipping of some of
> the characters in the xterm.
> 
> According to lsof, gnome-terminal is using
> 
>  /usr/share/fonts/truetype/dejavu/DejaVuSansMono.ttf
> 
> so the xterm has been run using
> 
> xterm -fa 'DejaVuSansMono' -fs 9
> 
> (and all font-related options are commented out in ~/.Xdefaults).
> 
> FWIW, this is on a debian testing system with XTERM_LOCALE=en_US.UTF-8.

At startup time, both programs have to determine the size of a character
cell.  They do so by applying some algorithm to the font(s) involved
(the maximum width of all glyphs?  the average width of selected glyphs?
something else?).  Evidently, xterm's algorithm doesn't account for all
the glyphs, and ends up clipping some of them, whereas gnome-terminal
has a different algorithm.

You clipped the xterm and gnome-terminal windows.  Are they the same
size?  Does the gnome-terminal contain more pixels (because it accounts
for the wider glyphs)?

Re: displaying missing glyphs

[Leo Butler]

<2QdxY4RzWzUUiLuE@potatochowder.com> writes:

>
> On 2021-04-12 at 12:08:08 -0500,
> Regarding "Re: displaying missing glyphs,"
> Leo Butler <leo.butler@umanitoba.ca> wrote:
>
>> Andreas Eder <a_eder_muc@web.de> writes:
>> 
>> > On Fr 09 Apr 2021 at 11:32, Leo Butler <leo.butler@umanitoba.ca> wrote:
>> >
>> >> I use `emacs -nw` inside of screen inside of uxterm. Unfortunately, many
>> >> unicode glyphs are not displayed correctly (although they are if I
>> >> attach the screen session in gnome-terminal, for example).
>> >>
>> >> In emacs/elisp, how might I override the default empty box to display
>> >> something more informative?
>> >
>> > The problem is - most likely - a font that is not unicode capable.
>> > If you set uxterm to ise the same font as gnome-terminal then it should
>> > work.
>> > The same combination (uxterm, screen and emacs) works perfectly well
>> > here.
>> 
>> Thanks for the suggestion. I have attached a marked-up screen shot of an
>> xterm (left) and gnome-terminal running `emacsclient -nw` and showing
>> the same buffer. You can see there is a noticeable clipping of some of
>> the characters in the xterm.
>> 
>> According to lsof, gnome-terminal is using
>> 
>>  /usr/share/fonts/truetype/dejavu/DejaVuSansMono.ttf
>> 
>> so the xterm has been run using
>> 
>> xterm -fa 'DejaVuSansMono' -fs 9
>> 
>> (and all font-related options are commented out in ~/.Xdefaults).
>> 
>> FWIW, this is on a debian testing system with XTERM_LOCALE=en_US.UTF-8.
>
> At startup time, both programs have to determine the size of a character
> cell.  They do so by applying some algorithm to the font(s) involved
> (the maximum width of all glyphs?  the average width of selected glyphs?
> something else?).  Evidently, xterm's algorithm doesn't account for all
> the glyphs, and ends up clipping some of them, whereas gnome-terminal
> has a different algorithm.

That seems like a reasonable answer, although, because I am ignorant of
all things font-related, I would have thought the font would have
contained this information.

>
> You clipped the xterm and gnome-terminal windows.  Are they the same
> size?

The two windows share 50% of the width of my screen. I used gimp to
select the whole screen, but I may have missed a few pixels on the
margins. Also, the WM (fluxbox) seems to create a 2-3 pixel-wide overlap
for reasons that escape me and this is consistent across applications.

>   Does the gnome-terminal contain more pixels (because it accounts
> for the wider glyphs)?

xwininfo shows the left window is 2 pixels wider than the right (800 v
798). If I give each window 799 pixels, I still see the same behaviour.

Leo

Re: displaying missing glyphs

[2QdxY4RzWzUUiLuE]

On 2021-04-12 at 13:45:20 -0500,
Regarding "Re: displaying missing glyphs,"
Leo Butler <leo.butler@umanitoba.ca> wrote:

> <2QdxY4RzWzUUiLuE@potatochowder.com> writes:
> 
> >
> > On 2021-04-12 at 12:08:08 -0500,
> > Regarding "Re: displaying missing glyphs,"
> > Leo Butler <leo.butler@umanitoba.ca> wrote:
> >
> >> Andreas Eder <a_eder_muc@web.de> writes:
> >> 
> >> > On Fr 09 Apr 2021 at 11:32, Leo Butler <leo.butler@umanitoba.ca> wrote:
> >> >
> >> >> I use `emacs -nw` inside of screen inside of uxterm. Unfortunately, many
> >> >> unicode glyphs are not displayed correctly (although they are if I
> >> >> attach the screen session in gnome-terminal, for example).
> >> >>
> >> >> In emacs/elisp, how might I override the default empty box to display
> >> >> something more informative?
> >> >
> >> > The problem is - most likely - a font that is not unicode capable.
> >> > If you set uxterm to ise the same font as gnome-terminal then it should
> >> > work.
> >> > The same combination (uxterm, screen and emacs) works perfectly well
> >> > here.
> >> 
> >> Thanks for the suggestion. I have attached a marked-up screen shot of an
> >> xterm (left) and gnome-terminal running `emacsclient -nw` and showing
> >> the same buffer. You can see there is a noticeable clipping of some of
> >> the characters in the xterm.
> >> 
> >> According to lsof, gnome-terminal is using
> >> 
> >>  /usr/share/fonts/truetype/dejavu/DejaVuSansMono.ttf
> >> 
> >> so the xterm has been run using
> >> 
> >> xterm -fa 'DejaVuSansMono' -fs 9
> >> 
> >> (and all font-related options are commented out in ~/.Xdefaults).
> >> 
> >> FWIW, this is on a debian testing system with XTERM_LOCALE=en_US.UTF-8.
> >
> > At startup time, both programs have to determine the size of a character
> > cell.  They do so by applying some algorithm to the font(s) involved
> > (the maximum width of all glyphs?  the average width of selected glyphs?
> > something else?).  Evidently, xterm's algorithm doesn't account for all
> > the glyphs, and ends up clipping some of them, whereas gnome-terminal
> > has a different algorithm.
> 
> That seems like a reasonable answer, although, because I am ignorant of
> all things font-related, I would have thought the font would have
> contained this information.
> 
> >
> > You clipped the xterm and gnome-terminal windows.  Are they the same
> > size?
> 
> The two windows share 50% of the width of my screen. I used gimp to
> select the whole screen, but I may have missed a few pixels on the
> margins. Also, the WM (fluxbox) seems to create a 2-3 pixel-wide overlap
> for reasons that escape me and this is consistent across applications.
> 
> >   Does the gnome-terminal contain more pixels (because it accounts
> > for the wider glyphs)?
> 
> xwininfo shows the left window is 2 pixels wider than the right (800 v
> 798). If I give each window 799 pixels, I still see the same behaviour.

Okay, the windows are the same size in pixels, but look at the number of
characters on a line.  The xterm window includes more characters, which
means that the gnome-terminal window includes more pixels per character
(the gnome-terminal window cuts your command off in the middle of the
closing parenthesis of the call to format).

I don't think emacs has anything to do with it.

Re: displaying missing glyphs

[Leo Butler]

[-- Attachment #1: Type: text/plain, Size: 3929 bytes --]

<2QdxY4RzWzUUiLuE@potatochowder.com> writes:

> On 2021-04-12 at 13:45:20 -0500,
> Regarding "Re: displaying missing glyphs,"
> Leo Butler <leo.butler@umanitoba.ca> wrote:
>
>> <2QdxY4RzWzUUiLuE@potatochowder.com> writes:
>> 
>> >
>> > On 2021-04-12 at 12:08:08 -0500,
>> > Regarding "Re: displaying missing glyphs,"
>> > Leo Butler <leo.butler@umanitoba.ca> wrote:
>> >
>> >> Andreas Eder <a_eder_muc@web.de> writes:
>> >> 
>> >> > On Fr 09 Apr 2021 at 11:32, Leo Butler <leo.butler@umanitoba.ca> wrote:
>> >> >
>> >> >> I use `emacs -nw` inside of screen inside of uxterm. Unfortunately, many
>> >> >> unicode glyphs are not displayed correctly (although they are if I
>> >> >> attach the screen session in gnome-terminal, for example).
>> >> >>
>> >> >> In emacs/elisp, how might I override the default empty box to display
>> >> >> something more informative?
>> >> >
>> >> > The problem is - most likely - a font that is not unicode capable.
>> >> > If you set uxterm to ise the same font as gnome-terminal then it should
>> >> > work.
>> >> > The same combination (uxterm, screen and emacs) works perfectly well
>> >> > here.
>> >> 
>> >> Thanks for the suggestion. I have attached a marked-up screen shot of an
>> >> xterm (left) and gnome-terminal running `emacsclient -nw` and showing
>> >> the same buffer. You can see there is a noticeable clipping of some of
>> >> the characters in the xterm.
>> >> 
>> >> According to lsof, gnome-terminal is using
>> >> 
>> >>  /usr/share/fonts/truetype/dejavu/DejaVuSansMono.ttf
>> >> 
>> >> so the xterm has been run using
>> >> 
>> >> xterm -fa 'DejaVuSansMono' -fs 9
>> >> 
>> >> (and all font-related options are commented out in ~/.Xdefaults).
>> >> 
>> >> FWIW, this is on a debian testing system with XTERM_LOCALE=en_US.UTF-8.
>> >
>> > At startup time, both programs have to determine the size of a character
>> > cell.  They do so by applying some algorithm to the font(s) involved
>> > (the maximum width of all glyphs?  the average width of selected glyphs?
>> > something else?).  Evidently, xterm's algorithm doesn't account for all
>> > the glyphs, and ends up clipping some of them, whereas gnome-terminal
>> > has a different algorithm.
>> 
>> That seems like a reasonable answer, although, because I am ignorant of
>> all things font-related, I would have thought the font would have
>> contained this information.
>> 
>> >
>> > You clipped the xterm and gnome-terminal windows.  Are they the same
>> > size?
>> 
>> The two windows share 50% of the width of my screen. I used gimp to
>> select the whole screen, but I may have missed a few pixels on the
>> margins. Also, the WM (fluxbox) seems to create a 2-3 pixel-wide overlap
>> for reasons that escape me and this is consistent across applications.
>> 
>> >   Does the gnome-terminal contain more pixels (because it accounts
>> > for the wider glyphs)?
>> 
>> xwininfo shows the left window is 2 pixels wider than the right (800 v
>> 798). If I give each window 799 pixels, I still see the same behaviour.
>
> Okay, the windows are the same size in pixels, but look at the number of
> characters on a line.  The xterm window includes more characters, which
> means that the gnome-terminal window includes more pixels per character
> (the gnome-terminal window cuts your command off in the middle of the
> closing parenthesis of the call to format).

I don't think what you are saying is true. Open the file in gimp and cut
out the two lines you mention. You will see the character density is the
same (p2 of attachment).

I think that the attachment shows what Yuri said about gnome-terminal
vs. xterm's painting of the glyph is accurate (and gnome-terminal is
making an effort to scale the glyphs to fit into the allotted box,
whereas xterm does not seem to be.

>
> I don't think emacs has anything to do with it.

Yes, as I said, I was wondering how/if one can overcome this deficiency
using elisp.

Leo


[-- Attachment #2: xterm-clipping-2.pdf --]
[-- Type: application/pdf, Size: 213771 bytes --]

Re: displaying missing glyphs

[Yuri Khan]

On Tue, 13 Apr 2021 at 00:24, Leo Butler <leo.butler@umanitoba.ca> wrote:

> Thanks for the suggestion. I have attached a marked-up screen shot of an
> xterm (left) and gnome-terminal running `emacsclient -nw` and showing
> the same buffer. You can see there is a noticeable clipping of some of
> the characters in the xterm.
>
> According to lsof, gnome-terminal is using
>
>  /usr/share/fonts/truetype/dejavu/DejaVuSansMono.ttf

The DejaVu Sans Mono font does not even contain glyphs for
MATHEMATICAL BOLD CAPITAL [A-Z] (on my Ubuntu 20.04, checked via
gucharmap(1)). The glyphs are taken from some other font that does
have them, possibly FreeSerif (which is not a monospace font, and even
if it were, it would not have the same metrics as DejaVu Sans Mono).

The difference in rendering stems from the way the two terminals paint
each cell. You will observe it more evidently if you enclose each
glyph in some delimiters, e.g. (insert (format "\n%s\n%x\t[%c]" x y
y)). Xterm seems to paint the next cell (space) opaquely, partially
erasing the MATHEMATICAL BOLD CAPITAL A. GNOME Terminal paints it
transparently so the rightmost part of the letter is still visible
under the closing bracket. Another terminal emulator, Kitty,
dynamically reduces the font size for the MATHEMATICAL BOLD CAPITAL A
so its width does not exceed the cell width.

Emacs running as a text-only application in a terminal emulator does
not, cannot, and should not know the font(s) it’s being rendered with
or any terminal-emulator-specific rendering peculiarities, or do
anything to improve the way it’s rendered.

Font fallback is hard, especially when you want character cells to
align perfectly (which, in case of a terminal emulator, you always
do). You might try filing a bug against Xterm.

Re: displaying missing glyphs

[Leo Butler]

Yuri Khan <yuri.v.khan@gmail.com> writes:

> On Tue, 13 Apr 2021 at 00:24, Leo Butler <leo.butler@umanitoba.ca> wrote:
>
>> Thanks for the suggestion. I have attached a marked-up screen shot of an
>> xterm (left) and gnome-terminal running `emacsclient -nw` and showing
>> the same buffer. You can see there is a noticeable clipping of some of
>> the characters in the xterm.
>>
>> According to lsof, gnome-terminal is using
>>
>>  /usr/share/fonts/truetype/dejavu/DejaVuSansMono.ttf
>
> The DejaVu Sans Mono font does not even contain glyphs for
> MATHEMATICAL BOLD CAPITAL [A-Z] (on my Ubuntu 20.04, checked via
> gucharmap(1)). The glyphs are taken from some other font that does
> have them, possibly FreeSerif (which is not a monospace font, and even
> if it were, it would not have the same metrics as DejaVu Sans Mono).
>
> The difference in rendering stems from the way the two terminals paint
> each cell. You will observe it more evidently if you enclose each
> glyph in some delimiters, e.g. (insert (format "\n%s\n%x\t[%c]" x y
> y)). 

Yuri, thanks for that suggestion. After doing so, I can see that
gnome-terminal is scaling each glyph to fit in a fixed-size box, while
xterm is allowing it to overflow.

> Xterm seems to paint the next cell (space) opaquely, partially
> erasing the MATHEMATICAL BOLD CAPITAL A. GNOME Terminal paints it
> transparently so the rightmost part of the letter is still visible
> under the closing bracket. Another terminal emulator, Kitty,
> dynamically reduces the font size for the MATHEMATICAL BOLD CAPITAL A
> so its width does not exceed the cell width.

Hmm, my gnome-terminal largely behaves the way that you say Kitty does...

>
> Emacs running as a text-only application in a terminal emulator does
> not, cannot, and should not know the font(s) it’s being rendered with
> or any terminal-emulator-specific rendering peculiarities, or do
> anything to improve the way it’s rendered.

Yes, the thread started by my asking how I could fix the problem with
elisp. I was thinking of possibly using an overlay or something like
that.

>
> Font fallback is hard, especially when you want character cells to
> align perfectly (which, in case of a terminal emulator, you always
> do). You might try filing a bug against Xterm.

I will file a bug against the debian xterm package.

Leo