bug#65803: 29.1; Noto Sans Mono CJK JP has doubled-width on Windows

bug#65803: 29.1; Noto Sans Mono CJK JP has doubled-width on Windows

[Shingo Tanaka]

(set-face-attribute 'default nil :font "Noto Sans Mono CJK JP")
makes frame pixel width 2x larger than ascii char width on Windows
10/11.  This also makes tabulated-list unaligned as you can see in
(list-buffers).

Emacs Binary: emacs-29.1_2-installer.exe

Font: https://github.com/notofonts/noto-cjk
Sans/Variable/TTF/Mono/NotoSansMonoCJKjp-VF.ttf

OS: Windows 10, Windows 11

Regards,
Shingo

bug#65803: 29.1; Noto Sans Mono CJK JP has doubled-width on Windows

[Eli Zaretskii]

> From: Shingo Tanaka <shingo.fg8@gmail.com>
> Date: Thu, 7 Sep 2023 22:38:32 +0900
> 
> (set-face-attribute 'default nil :font "Noto Sans Mono CJK JP")
> makes frame pixel width 2x larger than ascii char width on Windows
> 10/11.  This also makes tabulated-list unaligned as you can see in
> (list-buffers).

Thanks, but what do you mean by "ascii char width"?  Do you mean that
character glyphs on Noto Sans Mono CJK JP are two times wider than
ASCII characters of some other font?  If so, why is it a problem that
glyphs of one font are wider than glyphs of another?

Or maybe I misunderstand the issue?  Could you post a screenshot of
the display with mis-aligned buffer list, so we could see what you are
talking about?

bug#65803: 29.1; Noto Sans Mono CJK JP has doubled-width on Windows

[Shingo Tanaka]

[-- Attachment #1: Type: text/plain, Size: 1502 bytes --]

Please find attached 2 images.  The frame pixel width with
frame-width=40 & Noto Font is the same as the one with frame-width=80
& Windows Font "ＭＳ ゴシック", meaning the pixel width of Noto Font is
recognized 2x wider than Windows Font, although the actual ascii
characters pixel widths of the two are the same as you can see.  And
you can also see that the (list-buffers) buffer is unaligned around
"Size" column and "Mode" column with Noto Font, which should probably
the wrong pixel width recognition as you can see the space between
"Buffer" column and "Size" column is too long.

If there is any further information needed, Please let me know.

2023年9月7日(木) 22:50 Eli Zaretskii <eliz@gnu.org>:
>
> > From: Shingo Tanaka <shingo.fg8@gmail.com>
> > Date: Thu, 7 Sep 2023 22:38:32 +0900
> >
> > (set-face-attribute 'default nil :font "Noto Sans Mono CJK JP")
> > makes frame pixel width 2x larger than ascii char width on Windows
> > 10/11.  This also makes tabulated-list unaligned as you can see in
> > (list-buffers).
>
> Thanks, but what do you mean by "ascii char width"?  Do you mean that
> character glyphs on Noto Sans Mono CJK JP are two times wider than
> ASCII characters of some other font?  If so, why is it a problem that
> glyphs of one font are wider than glyphs of another?
>
> Or maybe I misunderstand the issue?  Could you post a screenshot of
> the display with mis-aligned buffer list, so we could see what you are
> talking about?

[-- Attachment #2: Win Font.jpg --]
[-- Type: image/jpeg, Size: 57398 bytes --]

[-- Attachment #3: Noto Font.jpg --]
[-- Type: image/jpeg, Size: 66945 bytes --]

bug#65803: 29.1; Noto Sans Mono CJK JP has doubled-width on Windows

[Eli Zaretskii]

> From: Shingo Tanaka <shingo.fg8@gmail.com>
> Date: Thu, 7 Sep 2023 23:24:55 +0900
> Cc: 65803@debbugs.gnu.org
> 
> Please find attached 2 images.  The frame pixel width with
> frame-width=40 & Noto Font is the same as the one with frame-width=80
> & Windows Font "ＭＳ ゴシック", meaning the pixel width of Noto Font is
> recognized 2x wider than Windows Font, although the actual ascii
> characters pixel widths of the two are the same as you can see.  And
> you can also see that the (list-buffers) buffer is unaligned around
> "Size" column and "Mode" column with Noto Font, which should probably
> the wrong pixel width recognition as you can see the space between
> "Buffer" column and "Size" column is too long.

It looks like the Noto font is not a fixed-pitch font, even though it
has "Mono" in its name.

What does "M-: (frame-char-width) RET" produce in each of these two
cases?

bug#65803: 29.1; Noto Sans Mono CJK JP has doubled-width on Windows

[Eli Zaretskii]

> Cc: 65803@debbugs.gnu.org
> Date: Thu, 07 Sep 2023 17:40:29 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> It looks like the Noto font is not a fixed-pitch font, even though it
> has "Mono" in its name.

There seems to be quite a few issues about this font family and its
width aspects:

  https://github.com/adobe-fonts/source-han-code-jp/issues/14
  https://github.com/notofonts/noto-cjk/issues/122
  https://github.com/notofonts/latin-greek-cyrillic/issues/196

I'm not an expert on fonts, but I must ask: why do you use this font
as the _default_ font? isn't it for CJK characters?  If so, why not
use it only for CJK scripts, not for ASCII?

Also, did anything changed wrt this font in Emacs 29?  That is, did
the problem you describe not exist in Emacs 28?

bug#65803: 29.1; Noto Sans Mono CJK JP has doubled-width on Windows

[Shingo Tanaka]

[-- Attachment #1: Type: text/plain, Size: 1156 bytes --]

I'm not an expert on fonts either, but the reason why I reported is because
I don't see this issue on Ubuntu as attached with exactly the same font
(NotoSansMonoCJKjp-VF.ttf).  So the difference looks like coming from OS
difference but not emacs version difference.

2023年9月7日(木) 23:57 Eli Zaretskii <eliz@gnu.org>:

>
> > Cc: 65803@debbugs.gnu.org
> > Date: Thu, 07 Sep 2023 17:40:29 +0300
> > From: Eli Zaretskii <eliz@gnu.org>
> >
> > It looks like the Noto font is not a fixed-pitch font, even though it
> > has "Mono" in its name.
>
> There seems to be quite a few issues about this font family and its
> width aspects:
>
>   https://github.com/adobe-fonts/source-han-code-jp/issues/14
>   https://github.com/notofonts/noto-cjk/issues/122
>   https://github.com/notofonts/latin-greek-cyrillic/issues/196
>
> I'm not an expert on fonts, but I must ask: why do you use this font
> as the _default_ font? isn't it for CJK characters?  If so, why not
> use it only for CJK scripts, not for ASCII?
>
> Also, did anything changed wrt this font in Emacs 29?  That is, did
> the problem you describe not exist in Emacs 28?

[-- Attachment #2: Screenshot from 2023-09-08 08-04-53.png --]
[-- Type: image/png, Size: 135302 bytes --]

bug#65803: 29.1; Noto Sans Mono CJK JP has doubled-width on Windows

[Eli Zaretskii]

> From: Shingo Tanaka <shingo.fg8@gmail.com>
> Date: Fri, 8 Sep 2023 08:19:29 +0900
> Cc: 65803@debbugs.gnu.org
> 
> I'm not an expert on fonts either, but the reason why I reported is because
> I don't see this issue on Ubuntu as attached with exactly the same font
> (NotoSansMonoCJKjp-VF.ttf).  So the difference looks like coming from OS
> difference but not emacs version difference.

Could be: the way Emacs handles fonts on Windows is very different
from what we do on GNU/Linux.  For example, it seemed to me that the
session using this font on Windows actually used the bold variant of
the font, which might have something to do with the very partial
support for font families on MS-Windows.

Please also tell what does (frame-char-width) return with each of the
two fonts on Windows.

Is your Ubuntu build with Cairo, btw?

bug#65803: 29.1; Noto Sans Mono CJK JP has doubled-width on Windows

[Shingo Tanaka]

> Please also tell what does (frame-char-width) return with each of the
> two fonts on Windows.

Here are the results including the one on Ubuntu.
Obviously, the 2nd one (Noto on Windows) returns a doubled-width size
which is unexpected.

;; On Windows - No issue
(progn
  (set-face-attribute 'default nil :font "ＭＳ ゴシック")
  (frame-char-width))
10
(string-pixel-width "A")
10
(string-pixel-width "あ")
20

;; On Windows - Wrong frame-char-width
(progn
  (set-face-attribute 'default nil :font "Noto Sans Mono CJK JP")
  (frame-char-width))
20
(string-pixel-width "A")
10
(string-pixel-width "あ")
20

;; On Ubuntu (w/Cairo) - No issue
(progn
  (set-face-attribute 'default nil :font "Noto Sans Mono CJK JP")
  (frame-char-width))
13
(string-pixel-width "A")
13
(string-pixel-width "あ")
26

> Is your Ubuntu build with Cairo, btw?

Yes, the Emacs on Ubuntu is the latest Snap version which is compiled
with Cairo, as I can double check it by seeing colored Emoji.

2023年9月8日(金) 15:25 Eli Zaretskii <eliz@gnu.org>:
>
> > From: Shingo Tanaka <shingo.fg8@gmail.com>
> > Date: Fri, 8 Sep 2023 08:19:29 +0900
> > Cc: 65803@debbugs.gnu.org
> >
> > I'm not an expert on fonts either, but the reason why I reported is because
> > I don't see this issue on Ubuntu as attached with exactly the same font
> > (NotoSansMonoCJKjp-VF.ttf).  So the difference looks like coming from OS
> > difference but not emacs version difference.
>
> Could be: the way Emacs handles fonts on Windows is very different
> from what we do on GNU/Linux.  For example, it seemed to me that the
> session using this font on Windows actually used the bold variant of
> the font, which might have something to do with the very partial
> support for font families on MS-Windows.
>
> Please also tell what does (frame-char-width) return with each of the
> two fonts on Windows.
>
> Is your Ubuntu build with Cairo, btw?

bug#65803: 29.1; Noto Sans Mono CJK JP has doubled-width on Windows

[Eli Zaretskii]

> From: Shingo Tanaka <shingo.fg8@gmail.com>
> Date: Fri, 8 Sep 2023 07:26:18 +0900
> Cc: 65803@debbugs.gnu.org
> 
> > Please also tell what does (frame-char-width) return with each of the
> > two fonts on Windows.
> 
> Here are the results including the one on Ubuntu.
> Obviously, the 2nd one (Noto on Windows) returns a doubled-width size
> which is unexpected.
> 
> ;; On Windows - No issue
> (progn
>   (set-face-attribute 'default nil :font "ＭＳ ゴシック")
>   (frame-char-width))
> 10
> (string-pixel-width "A")
> 10
> (string-pixel-width "あ")
> 20
> 
> ;; On Windows - Wrong frame-char-width
> (progn
>   (set-face-attribute 'default nil :font "Noto Sans Mono CJK JP")
>   (frame-char-width))
> 20
> (string-pixel-width "A")
> 10
> (string-pixel-width "あ")
> 20
> 
> ;; On Ubuntu (w/Cairo) - No issue
> (progn
>   (set-face-attribute 'default nil :font "Noto Sans Mono CJK JP")
>   (frame-char-width))
> 13
> (string-pixel-width "A")
> 13
> (string-pixel-width "あ")
> 26

This is strange.  AFAIU, frame-char-width returns the "average width"
attribute of the font, so why do we get different results on Windows
and on X?  Po Lu, can you help?  Do font backends on X perform some
trickery on the font's average_width attribute that we don't do on
Windows?

> > Is your Ubuntu build with Cairo, btw?
> 
> Yes, the Emacs on Ubuntu is the latest Snap version which is compiled
> with Cairo, as I can double check it by seeing colored Emoji.

bug#65803: 29.1; Noto Sans Mono CJK JP has doubled-width on Windows

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

> This is strange.  AFAIU, frame-char-width returns the "average width"
> attribute of the font, so why do we get different results on Windows
> and on X?  Po Lu, can you help?  Do font backends on X perform some
> trickery on the font's average_width attribute that we don't do on
> Windows?

For a nominally monospace font, the ft*font backends infer the average
from the width of the space glyph, instead of giving undue credence to
its reported ``average width''.  CJK fonts customarily contain tens of
thousands of glyphs, of which only a small subset represent ASCII
``monospace'' characters relevant to Emacs, but the `tmAveCharWidth'
field, which is derived from the font's OS/2 table:

  https://developer.apple.com/fonts/TrueType-Reference-Manual/RM06/Chap6OS2.html

represents the average width of all the glyphs within the font, and is
ergo naturally biased towards the large number of CJK glyphs
incorporated within the font.  The W32 backend should either infer the
average width itself, or ground it upon on the width of the space glyph.
Refer to this code in ftfont_open:

      font->min_width = font->average_width = font->space_width = 0;
      for (i = 32, n = 0; i < 127; i++)
	if (FT_Load_Char (ft_face, i, FT_LOAD_DEFAULT) == 0)
	  {
	    int this_width = ft_face->glyph->metrics.horiAdvance >> 6;

	    if (this_width > 0
		&& (! font->min_width || font->min_width > this_width))
	      font->min_width = this_width;
	    if (i == 32)
	      font->space_width = this_width;
	    font->average_width += this_width;
	    n++;
	  }
      if (n > 0)
	font->average_width /= n;

bug#65803: 29.1; Noto Sans Mono CJK JP has doubled-width on Windows

[Werner LEMBERG]

> [...]  CJK fonts customarily contain tens of
> thousands of glyphs, of which only a small subset represent ASCII
> ``monospace'' characters relevant to Emacs, but the `tmAveCharWidth'
> field, which is derived from the font's OS/2 table:
> 
>   https://developer.apple.com/fonts/TrueType-Reference-Manual/RM06/Chap6OS2.html
>
> [...]

In general I recommend to refer to the OpenType specification

  https://learn.microsoft.com/en-us/typography/opentype/spec/

which is an ISO standard (ISO/IEC 14496-22 “Open Font Format”),
instead of the Apple-only TrueType reference.  There are significant
differences, especially for modern fonts.


    Werner

bug#65803: 29.1; Noto Sans Mono CJK JP has doubled-width on Windows

[Eli Zaretskii]

> Date: Fri, 08 Sep 2023 14:49:11 +0000 (UTC)
> Cc: eliz@gnu.org, 65803@debbugs.gnu.org, shingo.fg8@gmail.com
> From: Werner LEMBERG <wl@gnu.org>
> 
> > [...]  CJK fonts customarily contain tens of
> > thousands of glyphs, of which only a small subset represent ASCII
> > ``monospace'' characters relevant to Emacs, but the `tmAveCharWidth'
> > field, which is derived from the font's OS/2 table:
> > 
> >   https://developer.apple.com/fonts/TrueType-Reference-Manual/RM06/Chap6OS2.html
> >
> > [...]
> 
> In general I recommend to refer to the OpenType specification
> 
>   https://learn.microsoft.com/en-us/typography/opentype/spec/
> 
> which is an ISO standard (ISO/IEC 14496-22 “Open Font Format”),
> instead of the Apple-only TrueType reference.  There are significant
> differences, especially for modern fonts.

How does one know, using the OpenType specification info, whether a
given font is fixed-pitch or proportional?  I seem to be unable to
find this in the spec, but maybe I need new glasses.

bug#65803: 29.1; Noto Sans Mono CJK JP has doubled-width on Windows

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

> How does one know, using the OpenType specification info, whether a
> given font is fixed-pitch or proportional?  I seem to be unable to
> find this in the spec, but maybe I need new glasses.

This information is not available within the font file, at least in the
TrueType specification which is the basis for OpenType.  Programs which
read TrueType fonts are obliged to judge for themselves, customarily by
taking measurements of each font's glyphs, or by searching for ``Mono''
within the font's family name.  I don't know which approach Windows
employs.

bug#65803: 29.1; Noto Sans Mono CJK JP has doubled-width on Windows

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: Werner LEMBERG <wl@gnu.org>,  65803@debbugs.gnu.org,  shingo.fg8@gmail.com
> Date: Sat, 09 Sep 2023 21:42:14 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > How does one know, using the OpenType specification info, whether a
> > given font is fixed-pitch or proportional?  I seem to be unable to
> > find this in the spec, but maybe I need new glasses.
> 
> This information is not available within the font file, at least in the
> TrueType specification which is the basis for OpenType.  Programs which
> read TrueType fonts are obliged to judge for themselves, customarily by
> taking measurements of each font's glyphs, or by searching for ``Mono''
> within the font's family name.  I don't know which approach Windows
> employs.

MS-Windows seems to report it in the data it holds about the font.
See the lfPitchAndFimily attribute in the LOGFONT structure:

  https://learn.microsoft.com/en-us/windows/win32/api/wingdi/ns-wingdi-logfontw

and the tmPitchAndFamily attribute of the TEXTMETRIC structure:

  https://learn.microsoft.com/en-us/windows/win32/api/wingdi/ns-wingdi-textmetricw

I have no idea how these attributes are determined by Windows.

bug#65803: 29.1; Noto Sans Mono CJK JP has doubled-width on Windows

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: Shingo Tanaka <shingo.fg8@gmail.com>,  65803@debbugs.gnu.org
> Date: Fri, 08 Sep 2023 21:42:37 +0800
> 
> For a nominally monospace font, the ft*font backends infer the average
> from the width of the space glyph, instead of giving undue credence to
> its reported ``average width''.  CJK fonts customarily contain tens of
> thousands of glyphs, of which only a small subset represent ASCII
> ``monospace'' characters relevant to Emacs, but the `tmAveCharWidth'
> field, which is derived from the font's OS/2 table:
> 
>   https://developer.apple.com/fonts/TrueType-Reference-Manual/RM06/Chap6OS2.html
> 
> represents the average width of all the glyphs within the font, and is
> ergo naturally biased towards the large number of CJK glyphs
> incorporated within the font.  The W32 backend should either infer the
> average width itself, or ground it upon on the width of the space glyph.
> Refer to this code in ftfont_open:
> 
>       font->min_width = font->average_width = font->space_width = 0;
>       for (i = 32, n = 0; i < 127; i++)
> 	if (FT_Load_Char (ft_face, i, FT_LOAD_DEFAULT) == 0)
> 	  {
> 	    int this_width = ft_face->glyph->metrics.horiAdvance >> 6;
> 
> 	    if (this_width > 0
> 		&& (! font->min_width || font->min_width > this_width))
> 	      font->min_width = this_width;
> 	    if (i == 32)
> 	      font->space_width = this_width;
> 	    font->average_width += this_width;
> 	    n++;
> 	  }
>       if (n > 0)
> 	font->average_width /= n;

Thanks, but the above snippet in ftfont.c is only done for
proportional fonts, not for fixed-pitch fonts.  Is the font in
question, Noto Sans Mono CJK JP, a proportional font?  That is, does
it not set the fixed-pitch attribute?

bug#65803: 29.1; Noto Sans Mono CJK JP has doubled-width on Windows

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

> Thanks, but the above snippet in ftfont.c is only done for
> proportional fonts, not for fixed-pitch fonts.  Is the font in
> question, Noto Sans Mono CJK JP, a proportional font?  That is, does
> it not set the fixed-pitch attribute?

There's no spacing attribute in TrueType fonts, so that is contingent
upon how the MS Windows font scaler detects fixed pitch fonts.  Here's
how ftfont.c calculates the average width for fonts that Fontconfig
deems fixed pitch:

    font->min_width = font->average_width = font->space_width
      = (scalable ? ft_face->max_advance_width * size / upEM + 0.5
	 : ft_face->size->metrics.max_advance >> 6);

That aside, Fontconfig does not judge Noto Sans Mono CJK JP a fixed
pitch font on my system.

bug#65803: 29.1; Noto Sans Mono CJK JP has doubled-width on Windows

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: shingo.fg8@gmail.com,  65803@debbugs.gnu.org
> Date: Sat, 09 Sep 2023 21:38:32 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Thanks, but the above snippet in ftfont.c is only done for
> > proportional fonts, not for fixed-pitch fonts.  Is the font in
> > question, Noto Sans Mono CJK JP, a proportional font?  That is, does
> > it not set the fixed-pitch attribute?
> 
> There's no spacing attribute in TrueType fonts, so that is contingent
> upon how the MS Windows font scaler detects fixed pitch fonts.  Here's
> how ftfont.c calculates the average width for fonts that Fontconfig
> deems fixed pitch:
> 
>     font->min_width = font->average_width = font->space_width
>       = (scalable ? ft_face->max_advance_width * size / upEM + 0.5
> 	 : ft_face->size->metrics.max_advance >> 6);

What is metrics.max_advance, in terms of the attributes recorded in
the font file?

> That aside, Fontconfig does not judge Noto Sans Mono CJK JP a fixed
> pitch font on my system.

OK, that might explain part of the issue, thanks.

bug#65803: 29.1; Noto Sans Mono CJK JP has doubled-width on Windows

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: shingo.fg8@gmail.com,  65803@debbugs.gnu.org
> Date: Sat, 09 Sep 2023 21:38:32 +0800
> 
>     font->min_width = font->average_width = font->space_width
>       = (scalable ? ft_face->max_advance_width * size / upEM + 0.5
> 	 : ft_face->size->metrics.max_advance >> 6);
> 
> That aside, Fontconfig does not judge Noto Sans Mono CJK JP a fixed
> pitch font on my system.

Basically, calculating our own estimate of the average width means we
discard the attribute reported by the font, in effect backing up on
the change made in OpenType spec v3, which deprecated the previous
requirement to compute the average width based only on ASCII
characters.  This seems to be justified only because Emacs uses the
average width of the font for one purpose only: to calculate the
default column width of a frame.  So this calculation is only relevant
for when a font is used as the default face's font.  If we ever decide
to use the average width for anything else, we might be bitten by
this.

So I think a cleaner solution would be to leave the average width
attribute as the font reports it, and introduce a new attribute for
the average width of the ASCII characters.  Not sure how urgent this
is, but we should at least describe this subtlety in the comments.

bug#65803: 29.1; Noto Sans Mono CJK JP has doubled-width on Windows

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Po Lu <luangruo@yahoo.com>
>> Cc: shingo.fg8@gmail.com,  65803@debbugs.gnu.org
>> Date: Sat, 09 Sep 2023 21:38:32 +0800
>> 
>>     font->min_width = font->average_width = font->space_width
>>       = (scalable ? ft_face->max_advance_width * size / upEM + 0.5
>> 	 : ft_face->size->metrics.max_advance >> 6);
>> 
>> That aside, Fontconfig does not judge Noto Sans Mono CJK JP a fixed
>> pitch font on my system.
>
> Basically, calculating our own estimate of the average width means we
> discard the attribute reported by the font, in effect backing up on
> the change made in OpenType spec v3, which deprecated the previous
> requirement to compute the average width based only on ASCII
> characters.  This seems to be justified only because Emacs uses the
> average width of the font for one purpose only: to calculate the
> default column width of a frame.  So this calculation is only relevant
> for when a font is used as the default face's font.  If we ever decide
> to use the average width for anything else, we might be bitten by
> this.
>
> So I think a cleaner solution would be to leave the average width
> attribute as the font reports it, and introduce a new attribute for
> the average width of the ASCII characters.  Not sure how urgent this
> is, but we should at least describe this subtlety in the comments.

However, the only function of the average width property is to provide
the average width of ASCII characters; the W32 font driver is AFAIK the
only exception to this rule.  Moreover, the average width attribute in
older TrueType fonts is that of each ASCII glyph, and several fonts
provide no average width attribute at all (given that an OS/2 table need
not be supplied in fonts that aren't designed to function under
MS-Windows), in which case calculating its value for each glyph at
load-time will prove prohibitively expensive.

bug#65803: 29.1; Noto Sans Mono CJK JP has doubled-width on Windows

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: shingo.fg8@gmail.com,  65803@debbugs.gnu.org
> Date: Sun, 10 Sep 2023 09:00:45 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Basically, calculating our own estimate of the average width means we
> > discard the attribute reported by the font, in effect backing up on
> > the change made in OpenType spec v3, which deprecated the previous
> > requirement to compute the average width based only on ASCII
> > characters.  This seems to be justified only because Emacs uses the
> > average width of the font for one purpose only: to calculate the
> > default column width of a frame.  So this calculation is only relevant
> > for when a font is used as the default face's font.  If we ever decide
> > to use the average width for anything else, we might be bitten by
> > this.
> >
> > So I think a cleaner solution would be to leave the average width
> > attribute as the font reports it, and introduce a new attribute for
> > the average width of the ASCII characters.  Not sure how urgent this
> > is, but we should at least describe this subtlety in the comments.
> 
> However, the only function of the average width property is to provide
> the average width of ASCII characters

AFAICT, we never use this for anything but FRAME_COLUMN_WIDTH.  So
when you talk about "average width of ASCII characters", I don't think
I understand what is that property, since we never call it like that
and never use it for ASCII characters.

> Moreover, the average width attribute in
> older TrueType fonts is that of each ASCII glyph, and several fonts
> provide no average width attribute at all (given that an OS/2 table need
> not be supplied in fonts that aren't designed to function under
> MS-Windows), in which case calculating its value for each glyph at
> load-time will prove prohibitively expensive.

I don't understand what you are trying to say here.  Who suggested to
calculate the value of the average width for each glyph in the font at
load time?

bug#65803: 29.1; Noto Sans Mono CJK JP has doubled-width on Windows

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

> AFAICT, we never use this for anything but FRAME_COLUMN_WIDTH.  So
> when you talk about "average width of ASCII characters", I don't think
> I understand what is that property, since we never call it like that
> and never use it for ASCII characters.

That is the purpose of FRAME_COLUMN_WIDTH, and also what it is set to in
every font driver except for the MS-Windows one, which is the only
backend to consult the font's own average width information.

> I don't understand what you are trying to say here.  Who suggested to
> calculate the value of the average width for each glyph in the font at
> load time?

My point is, we don't need a new property; the W32 port should simply
compute font->average_width using the widths of each ASCII glyph,
disregarding tmAveCharWidth.

bug#65803: 29.1; Noto Sans Mono CJK JP has doubled-width on Windows

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: shingo.fg8@gmail.com,  65803@debbugs.gnu.org
> Date: Sun, 10 Sep 2023 13:36:01 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > AFAICT, we never use this for anything but FRAME_COLUMN_WIDTH.  So
> > when you talk about "average width of ASCII characters", I don't think
> > I understand what is that property, since we never call it like that
> > and never use it for ASCII characters.
> 
> That is the purpose of FRAME_COLUMN_WIDTH

No, the purpose of FRAME_COLUMN_WIDTH is much more than just "the
width of ASCII characters".  It is used as the canonical character
width of the frame, for gazillion purposes.  One example which
triggered this bug is :align-to display spec, something utterly
unrelated to ASCII characters.

> > I don't understand what you are trying to say here.  Who suggested to
> > calculate the value of the average width for each glyph in the font at
> > load time?
> 
> My point is, we don't need a new property; the W32 port should simply
> compute font->average_width using the widths of each ASCII glyph,
> disregarding tmAveCharWidth.

But other font back-ends don't compute average_width for fixed-pitch
fonts, so are you only talking about proportional fonts here?

bug#65803: 29.1; Noto Sans Mono CJK JP has doubled-width on Windows

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

> No, the purpose of FRAME_COLUMN_WIDTH is much more than just "the
> width of ASCII characters".  It is used as the canonical character
> width of the frame, for gazillion purposes.  One example which
> triggered this bug is :align-to display spec, something utterly
> unrelated to ASCII characters.

However, the column width has hitherto been defined to the average width
of the frame font's ASCII characters.  At least outside W32, that is.

> But other font back-ends don't compute average_width for fixed-pitch
> fonts, so are you only talking about proportional fonts here?

I'm talking about fonts in general: since fixed pitch fonts are meant to
incorporate uniformly sized glyphs, the width of the space glyph should
represent the average width of any subset of the font's glyphs.  In this
particular case, Fontconfig doesn't deem the font in question a fixed
pitch font, and thus Emacs measures the average width of each ASCII
character itself.

bug#65803: 29.1; Noto Sans Mono CJK JP has doubled-width on Windows

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: shingo.fg8@gmail.com,  65803@debbugs.gnu.org
> Date: Sun, 10 Sep 2023 13:55:47 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > No, the purpose of FRAME_COLUMN_WIDTH is much more than just "the
> > width of ASCII characters".  It is used as the canonical character
> > width of the frame, for gazillion purposes.  One example which
> > triggered this bug is :align-to display spec, something utterly
> > unrelated to ASCII characters.
> 
> However, the column width has hitherto been defined to the average width
> of the frame font's ASCII characters.  At least outside W32, that is.

No!  Once again, for fixed-pitch fonts the average width is taken from
the font.  Here, from ftfont.c:

    if (spacing != FC_PROPORTIONAL
  #ifdef FC_DUAL
	&& spacing != FC_DUAL
  #endif	/* FC_DUAL */
	)
      font->min_width = font->average_width = font->space_width
	= (scalable ? ft_face->max_advance_width * size / upEM + 0.5
	   : ft_face->size->metrics.max_advance >> 6);
    else
      {
	int n;

	font->min_width = font->average_width = font->space_width = 0;
	for (i = 32, n = 0; i < 127; i++)
	  if (FT_Load_Char (ft_face, i, FT_LOAD_DEFAULT) == 0)
	    {
	      int this_width = ft_face->glyph->metrics.horiAdvance >> 6;

	      if (this_width > 0
		  && (! font->min_width || font->min_width > this_width))
		font->min_width = this_width;
	      if (i == 32)
		font->space_width = this_width;
	      font->average_width += this_width;
	      n++;
	    }
	if (n > 0)
	  font->average_width /= n;
      }

This clearly only calculates the average width for proportional fonts,
and otherwise takes the average width from the font's max_advance
width without calculating anything.  Or what am I missing?

> > But other font back-ends don't compute average_width for fixed-pitch
> > fonts, so are you only talking about proportional fonts here?
> 
> I'm talking about fonts in general: since fixed pitch fonts are meant to
> incorporate uniformly sized glyphs, the width of the space glyph should
> represent the average width of any subset of the font's glyphs.  In this
> particular case, Fontconfig doesn't deem the font in question a fixed
> pitch font, and thus Emacs measures the average width of each ASCII
> character itself.

See above: other backends only calculate the average width for
proportional fonts.  So what you say doesn't fit my reading of the
code.

bug#65803: 29.1; Noto Sans Mono CJK JP has doubled-width on Windows

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Po Lu <luangruo@yahoo.com>
>> Cc: shingo.fg8@gmail.com,  65803@debbugs.gnu.org
>> Date: Sun, 10 Sep 2023 13:55:47 +0800
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > No, the purpose of FRAME_COLUMN_WIDTH is much more than just "the
>> > width of ASCII characters".  It is used as the canonical character
>> > width of the frame, for gazillion purposes.  One example which
>> > triggered this bug is :align-to display spec, something utterly
>> > unrelated to ASCII characters.
>> 
>> However, the column width has hitherto been defined to the average width
>> of the frame font's ASCII characters.  At least outside W32, that is.
>
> No!  Once again, for fixed-pitch fonts the average width is taken from
> the font.  Here, from ftfont.c:
>
>     if (spacing != FC_PROPORTIONAL
>   #ifdef FC_DUAL
> 	&& spacing != FC_DUAL
>   #endif	/* FC_DUAL */
> 	)
>       font->min_width = font->average_width = font->space_width
> 	= (scalable ? ft_face->max_advance_width * size / upEM + 0.5
> 	   : ft_face->size->metrics.max_advance >> 6);
>     else
>       {
> 	int n;
>
> 	font->min_width = font->average_width = font->space_width = 0;
> 	for (i = 32, n = 0; i < 127; i++)
> 	  if (FT_Load_Char (ft_face, i, FT_LOAD_DEFAULT) == 0)
> 	    {
> 	      int this_width = ft_face->glyph->metrics.horiAdvance >> 6;
>
> 	      if (this_width > 0
> 		  && (! font->min_width || font->min_width > this_width))
> 		font->min_width = this_width;
> 	      if (i == 32)
> 		font->space_width = this_width;
> 	      font->average_width += this_width;
> 	      n++;
> 	    }
> 	if (n > 0)
> 	  font->average_width /= n;
>       }
>
> This clearly only calculates the average width for proportional fonts,
> and otherwise takes the average width from the font's max_advance
> width without calculating anything.  Or what am I missing?
>
>> > But other font back-ends don't compute average_width for fixed-pitch
>> > fonts, so are you only talking about proportional fonts here?
>> 
>> I'm talking about fonts in general: since fixed pitch fonts are meant to
>> incorporate uniformly sized glyphs, the width of the space glyph should
>> represent the average width of any subset of the font's glyphs.  In this
>> particular case, Fontconfig doesn't deem the font in question a fixed
>> pitch font, and thus Emacs measures the average width of each ASCII
>> character itself.
>
> See above: other backends only calculate the average width for
> proportional fonts.  So what you say doesn't fit my reading of the
> code.

Because if spacing is not FC_PROPORTIONAL or FC_DUAL, we know in advance
that max_advance_width or max_advance are identical to the average of
all ASCII glyphs.  Such special treatment is an optimization, nothing
more.  max_advance_width is the advance width (in em space) of the
widest glyph when the font is scalable, and max_advance is that in pixel
space if not.

bug#65803: 29.1; Noto Sans Mono CJK JP has doubled-width on Windows

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: shingo.fg8@gmail.com,  65803@debbugs.gnu.org
> Date: Sun, 10 Sep 2023 15:31:36 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > See above: other backends only calculate the average width for
> > proportional fonts.  So what you say doesn't fit my reading of the
> > code.
> 
> Because if spacing is not FC_PROPORTIONAL or FC_DUAL, we know in advance
> that max_advance_width or max_advance are identical to the average of
> all ASCII glyphs.  Such special treatment is an optimization, nothing
> more.  max_advance_width is the advance width (in em space) of the
> widest glyph when the font is scalable, and max_advance is that in pixel
> space if not.

So you think it's okay to do the same in the w32 font backend,
i.e. take the average width from the font when the font is known to be
fixed-pitch?  If not, please elaborate, because that's what I
understand from what you wrote above.

bug#65803: 29.1; Noto Sans Mono CJK JP has doubled-width on Windows

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Po Lu <luangruo@yahoo.com>
>> Cc: shingo.fg8@gmail.com,  65803@debbugs.gnu.org
>> Date: Sun, 10 Sep 2023 15:31:36 +0800
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > See above: other backends only calculate the average width for
>> > proportional fonts.  So what you say doesn't fit my reading of the
>> > code.
>> 
>> Because if spacing is not FC_PROPORTIONAL or FC_DUAL, we know in advance
>> that max_advance_width or max_advance are identical to the average of
>> all ASCII glyphs.  Such special treatment is an optimization, nothing
>> more.  max_advance_width is the advance width (in em space) of the
>> widest glyph when the font is scalable, and max_advance is that in pixel
>> space if not.
>
> So you think it's okay to do the same in the w32 font backend,
> i.e. take the average width from the font when the font is known to be
> fixed-pitch?  If not, please elaborate, because that's what I
> understand from what you wrote above.

I don't think it's okay, because the W32 font backend judges fonts that
are not fixed pitch to be so; Noto Sans Mono CJK JP, for example.

bug#65803: 29.1; Noto Sans Mono CJK JP has doubled-width on Windows

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: shingo.fg8@gmail.com,  65803@debbugs.gnu.org
> Date: Sun, 10 Sep 2023 15:55:56 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > So you think it's okay to do the same in the w32 font backend,
> > i.e. take the average width from the font when the font is known to be
> > fixed-pitch?  If not, please elaborate, because that's what I
> > understand from what you wrote above.
> 
> I don't think it's okay, because the W32 font backend judges fonts that
> are not fixed pitch to be so; Noto Sans Mono CJK JP, for example.

How do you know that Noto Sans Mono CJK JP is handled as fixed-pitch
font by the w32 backend?  If you actually tried that with the
MS-Windows build, please tell how you saw it being handled as
fixed-pitch.

bug#65803: 29.1; Noto Sans Mono CJK JP has doubled-width on Windows

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

> How do you know that Noto Sans Mono CJK JP is handled as fixed-pitch
> font by the w32 backend?

Wasn't that established earlier by the OP?  If not, then would Shingo
please tell us whether the font is considered fixed-pitch?  (Type M-x
describe-font RET, and send us the output.)

bug#65803: 29.1; Noto Sans Mono CJK JP has doubled-width on Windows

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: shingo.fg8@gmail.com,  65803@debbugs.gnu.org
> Date: Sun, 10 Sep 2023 16:08:24 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > How do you know that Noto Sans Mono CJK JP is handled as fixed-pitch
> > font by the w32 backend?
> 
> Wasn't that established earlier by the OP?

No, I don't think so.

> If not, then would Shingo please tell us whether the font is
> considered fixed-pitch?  (Type M-x describe-font RET, and send us
> the output.)

I don't see the fact of the font being fixed-pitch or proportional
mentioned in the output of describe-font.  Am I missing something?

bug#65803: 29.1; Noto Sans Mono CJK JP has doubled-width on Windows

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

> I don't see the fact of the font being fixed-pitch or proportional
> mentioned in the output of describe-font.  Am I missing something?

Strange.  Here, I see:

full name: Source Code Pro:pixelsize=13:foundry=ADBO:weight=regular:slant=normal:width=normal:spacing=100:scalable=true

where `spacing=100' indicates the font is monospace.

bug#65803: 29.1; Noto Sans Mono CJK JP has doubled-width on Windows

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: shingo.fg8@gmail.com,  65803@debbugs.gnu.org
> Date: Sun, 10 Sep 2023 16:31:57 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I don't see the fact of the font being fixed-pitch or proportional
> > mentioned in the output of describe-font.  Am I missing something?
> 
> Strange.  Here, I see:
> 
> full name: Source Code Pro:pixelsize=13:foundry=ADBO:weight=regular:slant=normal:width=normal:spacing=100:scalable=true
> 
> where `spacing=100' indicates the font is monospace.

That's specific to Fontconfig, I think.  On Windows I see:

       full name: Courier New-10.0

bug#65803: 29.1; Noto Sans Mono CJK JP has doubled-width on Windows

[Eli Zaretskii]

> Cc: 65803@debbugs.gnu.org, shingo.fg8@gmail.com
> Date: Sun, 10 Sep 2023 11:50:26 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > From: Po Lu <luangruo@yahoo.com>
> > Cc: shingo.fg8@gmail.com,  65803@debbugs.gnu.org
> > Date: Sun, 10 Sep 2023 16:31:57 +0800
> > 
> > Eli Zaretskii <eliz@gnu.org> writes:
> > 
> > > I don't see the fact of the font being fixed-pitch or proportional
> > > mentioned in the output of describe-font.  Am I missing something?
> > 
> > Strange.  Here, I see:
> > 
> > full name: Source Code Pro:pixelsize=13:foundry=ADBO:weight=regular:slant=normal:width=normal:spacing=100:scalable=true
> > 
> > where `spacing=100' indicates the font is monospace.
> 
> That's specific to Fontconfig, I think.  On Windows I see:
> 
>        full name: Courier New-10.0

Anyway, I tried to write the code to compute the font average width,
and found that it's impossible to do reliably on MS-Windows: all the
font-related functions that return glyph metrics require a "device
context" argument, which cannot be provided as long as we don't have
at least one w32 frame.  So invocations like

  emacs -fn "Arial Unicode MS"

will not work, and that means invoking Emacs with variable-pitch fonts
as the default will not be able to take advantage of this improvement,
which basically makes all this change useless.

Maybe it's possible to pull this trick anyway, but it will take
someone who knows the Windows GUI programming better than myself,
perhaps by reading the font data directly (like the Windows port of
Freetype does?).

Sorry.  I guess the conclusion is, unfortunately, that fonts like Noto
Sans Mono CJK JP cannot be used on Windows as the default font, only
as font for the CJK scripts (configured via the fontset).

bug#65803: 29.1; Noto Sans Mono CJK JP has doubled-width on Windows

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

> Anyway, I tried to write the code to compute the font average width,
> and found that it's impossible to do reliably on MS-Windows: all the
> font-related functions that return glyph metrics require a "device
> context" argument, which cannot be provided as long as we don't have
> at least one w32 frame.  So invocations like
>
>   emacs -fn "Arial Unicode MS"
>
> will not work, and that means invoking Emacs with variable-pitch fonts
> as the default will not be able to take advantage of this improvement,
> which basically makes all this change useless.

If I'm not mistaken, the font*_open functions take a single frame
argument F, and the average width property doesn't need to be computed
until then.  Can't you obtain the DC from that frame?

TIA.

bug#65803: 29.1; Noto Sans Mono CJK JP has doubled-width on Windows

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: 65803@debbugs.gnu.org,  shingo.fg8@gmail.com
> Date: Sun, 10 Sep 2023 19:29:45 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Anyway, I tried to write the code to compute the font average width,
> > and found that it's impossible to do reliably on MS-Windows: all the
> > font-related functions that return glyph metrics require a "device
> > context" argument, which cannot be provided as long as we don't have
> > at least one w32 frame.  So invocations like
> >
> >   emacs -fn "Arial Unicode MS"
> >
> > will not work, and that means invoking Emacs with variable-pitch fonts
> > as the default will not be able to take advantage of this improvement,
> > which basically makes all this change useless.
> 
> If I'm not mistaken, the font*_open functions take a single frame
> argument F, and the average width property doesn't need to be computed
> until then.  Can't you obtain the DC from that frame?

That's what I tried.  The problem is, first time we open the fonts
(which is when it's important to have the metrics of the default
face's font), we are called from x-create-frame for the initial frame,
and the frame's w32-specific attributes are not yet set.  The
window-system for the frame is still "initial", and get_frame_dc
aborts.

bug#65803: 29.1; Noto Sans Mono CJK JP has doubled-width on Windows

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

> That's what I tried.  The problem is, first time we open the fonts
> (which is when it's important to have the metrics of the default
> face's font), we are called from x-create-frame for the initial frame,
> and the frame's w32-specific attributes are not yet set.  The
> window-system for the frame is still "initial", and get_frame_dc
> aborts.

Can't font initialization be moved to a location slightly later in the
frame initialization process?  xfont_open uses the frame's X display
without ill effect.

Thanks.

bug#65803: 29.1; Noto Sans Mono CJK JP has doubled-width on Windows

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: 65803@debbugs.gnu.org,  shingo.fg8@gmail.com
> Date: Sun, 10 Sep 2023 20:09:08 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > That's what I tried.  The problem is, first time we open the fonts
> > (which is when it's important to have the metrics of the default
> > face's font), we are called from x-create-frame for the initial frame,
> > and the frame's w32-specific attributes are not yet set.  The
> > window-system for the frame is still "initial", and get_frame_dc
> > aborts.
> 
> Can't font initialization be moved to a location slightly later in the
> frame initialization process?

It is needed for determining the frame geometry, since it provides us
with the column width and line height.  Maybe something could be done
about that, for example using the current approximation initially,
then correcting this with another iteration or somesuch.  But that
requires a better understanding of the fine details of w32font.c and
w32uniscribe.c than what I have.  For example, there's a cache of
metrics there, which would need to be discarded somehow.

> xfont_open uses the frame's X display without ill effect.

xfont_open gets the font metrics from Xlib calls, and its X display
argument is also something returned by Xlib.  With w32, I simply don't
know what minimum amount of setup is needed to make a frame a capable
enough w32 frame that can support get_frame_dc and the calls like
SelectObject we need for this particular job.  Feel free to
experiment, you or someone else; I simply don't have enough time for
that, sorry.

bug#65803: 29.1; Noto Sans Mono CJK JP has doubled-width on Windows

[Shingo Tanaka]

> Wasn't that established earlier by the OP?  If not, then would Shingo
> please tell us whether the font is considered fixed-pitch?  (Type M-x
> describe-font RET, and send us the output.)

On Windows 11:
runemacs.exe --no-init-file

In *scratch* buffer:
(set-face-attribute 'default nil :font "Noto Sans Mono CJK JP")
nil

M-x describe-font RET

Output in *help* buffer:
name (opened by): -outline-Noto Sans Mono CJK
JP-regular-normal-normal-mono-20-*-*-*-p-*-iso8859-1
       full name: Noto Sans Mono CJK JP-10.0
            size: 20
          height: 29
 baseline-offset:  0
relative-compose:  0
  default-ascent: 23
          ascent: 23
         descent:  6
   average-width: 20
     space-width: 20
       max-width: 79

Regards,
Shingo

2023年9月10日(日) 17:08 Po Lu <luangruo@yahoo.com>:
>
> Eli Zaretskii <eliz@gnu.org> writes:
>
> > How do you know that Noto Sans Mono CJK JP is handled as fixed-pitch
> > font by the w32 backend?
>
> Wasn't that established earlier by the OP?  If not, then would Shingo
> please tell us whether the font is considered fixed-pitch?  (Type M-x
> describe-font RET, and send us the output.)