bug#29078: 25.2; font issue with FreeType 2.8; should not use the rounded ascender and descender

bug#29078: 25.2; font issue with FreeType 2.8; should not use the rounded ascender and descender

[Vincent Lefevre]

For TrueType fonts, FreeType 2.8 uses different rounding rules for
values in the FT_Size_Metrics structure, which is apparently used
by Emacs (since there are visible changes). In particular, one can
now have ascender + descender > height. The consequences are:

1. Tables look a bit ugly.

2. Windows (with the same number of text lines) are unnecessarily
   higher than before.

3. This can break existing configurations, where Emacs would be
   opened at start up, with an expected window size.

I could notice changes at least with the default Mono font, which
appears to be DejaVu Sans Mono.

Upstream now recommends to use the values from the FT_Face structure
and scale them manually:

------------------------------------------------------------------------

Global size metrics values in the `FT_Size_Metrics' structure can be
different for TrueType fonts. Reason is that in older FreeType
versions the metrics were rounded differently to integer pixels
compared to all other font formats, yielding an inconsistent behaviour
if you used non-native hinting. Starting with this version, global
size metrics for TrueType fonts are handled the same as other font
formats: `ascender' gets rounded up, `descender' getsrounded down,
`height' gets normally rounded, and `max_advance' gets normally
rounded, too.

If you need more precise values of (global) ascender, descender,
height, or `max_advance', please take the corresponding values from
the `FT_Face' structure and scale them manually.

------------------------------------------------------------------------

See the discussion:
  https://savannah.nongnu.org/bugs/?52165



In GNU Emacs 25.2.2 (x86_64-pc-linux-gnu, GTK+ Version 3.22.20)
 of 2017-09-12, modified by Debian built on trouble
Windowing system distributor 'The X.Org Foundation', version 11.0.11905000
System Description:	Debian GNU/Linux stable-updates (sid)

Configured using:
 'configure --build x86_64-linux-gnu --prefix=/usr
 --sharedstatedir=/var/lib --libexecdir=/usr/lib
 --localstatedir=/var/lib --infodir=/usr/share/info
 --mandir=/usr/share/man --with-pop=yes
 --enable-locallisppath=/etc/emacs25:/etc/emacs:/usr/local/share/emacs/25.2/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/25.2/site-lisp:/usr/share/emacs/site-lisp
 --with-sound=alsa --without-gconf --build x86_64-linux-gnu
 --prefix=/usr --sharedstatedir=/var/lib --libexecdir=/usr/lib
 --localstatedir=/var/lib --infodir=/usr/share/info
 --mandir=/usr/share/man --with-pop=yes
 --enable-locallisppath=/etc/emacs25:/etc/emacs:/usr/local/share/emacs/25.2/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/25.2/site-lisp:/usr/share/emacs/site-lisp
 --with-sound=alsa --without-gconf --with-x=yes --with-x-toolkit=gtk3
 --with-toolkit-scroll-bars 'CFLAGS=-g -O2
 -fdebug-prefix-map=/build/emacs25-XrMyQe/emacs25-25.2+1=.
 -fstack-protector-strong -Wformat -Werror=format-security -Wall'
 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2' LDFLAGS=-Wl,-z,relro'

Configured features:
XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GSETTINGS NOTIFY
ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB
TOOLKIT_SCROLL_BARS GTK3 X11

Important settings:
  value of $LC_COLLATE: POSIX
  value of $LC_CTYPE: en_US.UTF-8
  value of $LC_TIME: en_DK
  value of $LANG: POSIX
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  display-time-mode: t
  show-paren-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent messages:
Loading /etc/emacs/site-start.d/50latex-cjk-common.el (source)...done
Loading /etc/emacs/site-start.d/50latex-cjk-thai.el (source)...done
Loading /etc/emacs/site-start.d/50psvn.el (source)...done
Loading /etc/emacs/site-start.d/50rnc-mode.el (source)...done
Loading /etc/emacs/site-start.d/50texlive-lang-english.el (source)...done
Loading /etc/emacs/site-start.d/50w3m-el.el (source)...done
Loading /home/vlefevre/share/emacs/site-lisp/mutteditor.el (source)...done
Loading time...done
For information about GNU Emacs and the GNU system, type C-h C-a.
Making completion list...

Load-path shadows:
/usr/share/emacs25/site-lisp/cmake-data/cmake-mode hides /usr/share/emacs/site-lisp/cmake-mode
/usr/share/emacs/25.2/site-lisp/debian-startup hides /usr/share/emacs/site-lisp/debian-startup
/usr/share/emacs25/site-lisp/flim/md4 hides /usr/share/emacs/25.2/lisp/md4
/usr/share/emacs25/site-lisp/flim/hex-util hides /usr/share/emacs/25.2/lisp/hex-util
/usr/share/emacs25/site-lisp/flim/sasl-cram hides /usr/share/emacs/25.2/lisp/net/sasl-cram
/usr/share/emacs25/site-lisp/flim/hmac-md5 hides /usr/share/emacs/25.2/lisp/net/hmac-md5
/usr/share/emacs25/site-lisp/flim/hmac-def hides /usr/share/emacs/25.2/lisp/net/hmac-def
/usr/share/emacs25/site-lisp/flim/sasl-digest hides /usr/share/emacs/25.2/lisp/net/sasl-digest
/usr/share/emacs25/site-lisp/flim/sasl hides /usr/share/emacs/25.2/lisp/net/sasl
/usr/share/emacs25/site-lisp/flim/sasl-ntlm hides /usr/share/emacs/25.2/lisp/net/sasl-ntlm
/usr/share/emacs25/site-lisp/flim/ntlm hides /usr/share/emacs/25.2/lisp/net/ntlm
/usr/share/emacs25/site-lisp/latex-cjk-thai/thai-word hides /usr/share/emacs/25.2/lisp/language/thai-word

Features:
(shadow sort mail-extr warnings emacsbug message dired format-spec
rfc822 mml mml-sec password-cache epg epg-config gnus-util mm-decode
mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
sendmail rfc2047 rfc2045 ietf-drums mm-util help-fns help-mode easymenu
mail-prsvr mail-utils time cus-start cus-load paren cc-styles cc-align
cc-engine cc-vars cc-defs edmacro kmacro cl-loaddefs pcase cl-lib
w3m-load time-date mule-util tooltip eldoc electric uniquify ediff-hook
vc-hooks lisp-float-type mwheel x-win term/common-win x-dnd tool-bar dnd
fontset image regexp-opt fringe tabulated-list newcomment elisp-mode
lisp-mode prog-mode register page menu-bar rfn-eshadow timer select
scroll-bar mouse jit-lock font-lock syntax facemenu font-core frame
cl-generic cham georgian utf-8-lang misc-lang vietnamese tibetan thai
tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek romanian
slovak czech european ethiopic indian cyrillic chinese charscript
case-table epa-hook jka-cmpr-hook help simple abbrev minibuffer
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote dbusbind inotify
dynamic-setting system-font-setting font-render-setting move-toolbar gtk
x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 107514 7364)
 (symbols 48 22500 0)
 (miscs 40 55 113)
 (strings 32 20823 4597)
 (string-bytes 1 613432)
 (vectors 16 12942)
 (vector-slots 8 444085 4241)
 (floats 8 171 38)
 (intervals 56 277 12)
 (buffers 976 19))

bug#29078: 25.2; font issue with FreeType 2.8; should not use the rounded ascender and descender

[Robert Pluim]

Vincent Lefevre <vincent@vinc17.net> writes:

> For TrueType fonts, FreeType 2.8 uses different rounding rules for
> values in the FT_Size_Metrics structure, which is apparently used
> by Emacs (since there are visible changes). In particular, one can
> now have ascender + descender > height. The consequences are:
>
> 1. Tables look a bit ugly.
>
> 2. Windows (with the same number of text lines) are unnecessarily
>    higher than before.
>
> 3. This can break existing configurations, where Emacs would be
>    opened at start up, with an expected window size.
>
> I could notice changes at least with the default Mono font, which
> appears to be DejaVu Sans Mono.
>
> Upstream now recommends to use the values from the FT_Face structure
> and scale them manually:
>

I don't have FreeType2.8 to test, but you're saying we need to do this?

diff --git a/src/ftfont.c b/src/ftfont.c
index 35f5923376..d16bf09a1e 100644
--- a/src/ftfont.c
+++ b/src/ftfont.c
@@ -1153,18 +1153,9 @@ ftfont_open2 (struct frame *f,
   upEM = ft_face->units_per_EM;
   scalable = (INTEGERP (AREF (entity, FONT_AVGWIDTH_INDEX))
 	      && XINT (AREF (entity, FONT_AVGWIDTH_INDEX)) == 0);
-  if (scalable)
-    {
-      font->ascent = ft_face->ascender * size / upEM + 0.5;
-      font->descent = - ft_face->descender * size / upEM + 0.5;
-      font->height = ft_face->height * size / upEM + 0.5;
-    }
-  else
-    {
-      font->ascent = ft_face->size->metrics.ascender >> 6;
-      font->descent = - ft_face->size->metrics.descender >> 6;
-      font->height = ft_face->size->metrics.height >> 6;
-    }
+  font->ascent = ft_face->ascender * size / upEM + 0.5;
+  font->descent = - ft_face->descender * size / upEM + 0.5;
+  font->height = ft_face->height * size / upEM + 0.5;
   if (INTEGERP (AREF (entity, FONT_SPACING_INDEX)))
     spacing = XINT (AREF (entity, FONT_SPACING_INDEX));
   else

bug#29078: 25.2; font issue with FreeType 2.8; should not use the rounded ascender and descender

[Werner LEMBERG]

>> For TrueType fonts, FreeType 2.8 uses different rounding rules for
>> values in the FT_Size_Metrics structure, which is apparently used
>> by Emacs (since there are visible changes). In particular, one can
>> now have ascender + descender > height. The consequences are:
>>
>> 1. Tables look a bit ugly.
>>
>> 2. Windows (with the same number of text lines) are unnecessarily
>>    higher than before.
>>
>> 3. This can break existing configurations, where Emacs would be
>>    opened at start up, with an expected window size.
>>
>> I could notice changes at least with the default Mono font, which
>> appears to be DejaVu Sans Mono.
>>
>> Upstream now recommends to use the values from the FT_Face
>> structure and scale them manually:
> 
> I don't have FreeType2.8 to test, but you're saying we need to do this?
> 
> diff --git a/src/ftfont.c b/src/ftfont.c
> index 35f5923376..d16bf09a1e 100644
> --- a/src/ftfont.c
> +++ b/src/ftfont.c
> @@ -1153,18 +1153,9 @@ ftfont_open2 (struct frame *f,
>    upEM = ft_face->units_per_EM;
>    scalable = (INTEGERP (AREF (entity, FONT_AVGWIDTH_INDEX))
>  	      && XINT (AREF (entity, FONT_AVGWIDTH_INDEX)) == 0);
> -  if (scalable)
> -    {
> -      font->ascent = ft_face->ascender * size / upEM + 0.5;
> -      font->descent = - ft_face->descender * size / upEM + 0.5;
> -      font->height = ft_face->height * size / upEM + 0.5;
> -    }
> -  else
> -    {
> -      font->ascent = ft_face->size->metrics.ascender >> 6;
> -      font->descent = - ft_face->size->metrics.descender >> 6;
> -      font->height = ft_face->size->metrics.height >> 6;
> -    }
> +  font->ascent = ft_face->ascender * size / upEM + 0.5;
> +  font->descent = - ft_face->descender * size / upEM + 0.5;
> +  font->height = ft_face->height * size / upEM + 0.5;
>    if (INTEGERP (AREF (entity, FONT_SPACING_INDEX)))
>      spacing = XINT (AREF (entity, FONT_SPACING_INDEX));
>    else

No.  The `scalable' branch must make a distinction between TrueType
and non-TrueType fonts if the font gets fully hinted (i.e., if the
TrueType bytecode gets interpreted), something like

  if (scalable)
    {
      if (use_truetype_bytecode_hinting(font))
        {
          /* use TrueType rules for rounding */
          font->ascent = ROUND(ft_face->ascender * size / upEM)
          font->descent = ROUND(-ft_face->descender * size / upEM);
          font->height = font->ascent + font->descent;
        }
      else
        {
          font->ascent = CEIL(ft_face->ascender * size / upEM);
          font->descent = FLOOR(-ft_face->descender * size / upEM);
          font->height = ROUND(ft_face->height * size / upEM);
        }
    }
  ...


    Werner

bug#29078: 25.2; font issue with FreeType 2.8; should not use the rounded ascender and descender

[Vincent Lefevre]

On 2017-10-31 12:22:14 +0100, Werner LEMBERG wrote:
> No.  The `scalable' branch must make a distinction between TrueType
> and non-TrueType fonts if the font gets fully hinted (i.e., if the
> TrueType bytecode gets interpreted), something like
> 
>   if (scalable)
>     {
>       if (use_truetype_bytecode_hinting(font))
>         {
>           /* use TrueType rules for rounding */
>           font->ascent = ROUND(ft_face->ascender * size / upEM)
>           font->descent = ROUND(-ft_face->descender * size / upEM);
>           font->height = font->ascent + font->descent;
>         }
>       else
>         {
>           font->ascent = CEIL(ft_face->ascender * size / upEM);
>           font->descent = FLOOR(-ft_face->descender * size / upEM);
>           font->height = ROUND(ft_face->height * size / upEM);
>         }
>     }
>   ...

FLOOR, CEIL and ROUND round integers to multiples of 64. Don't
you mean floor(), ceil() and round() from <math.h>?

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

bug#29078: 25.2; font issue with FreeType 2.8; should not use the rounded ascender and descender

[Werner LEMBERG]

> FLOOR, CEIL and ROUND round integers to multiples of 64. Don't
> you mean floor(), ceil() and round() from <math.h>?

Whatever :-)  It's just some quick, untested pseudo-code.


    Werner

bug#29078: 25.2; font issue with FreeType 2.8; should not use the rounded ascender and descender

[Robert Pluim]

Werner LEMBERG <wl@gnu.org> writes:

>
> No.  The `scalable' branch must make a distinction between TrueType
> and non-TrueType fonts if the font gets fully hinted (i.e., if the
> TrueType bytecode gets interpreted), something like
>
>   if (scalable)
>     {
>       if (use_truetype_bytecode_hinting(font))

Hmm, ok. Then it becomes a question of determining how to define
'use_truetype_bytecode_hinting', which is not immediately obvious to
me.

Robert

bug#29078: 25.2; font issue with FreeType 2.8; should not use the rounded ascender and descender

[Werner LEMBERG]

>>   if (scalable)
>>     {
>>       if (use_truetype_bytecode_hinting(font))
> 
> Hmm, ok. Then it becomes a question of determining how to define
> 'use_truetype_bytecode_hinting', which is not immediately obvious to
> me.

FreeType decides at glyph loading time (i.e., while calling
`FT_Load_Glyph' and its siblings) whether native bytecode hinting gets
used.[*]

Or to say it differently: It's up to Emacs which hinting mode gets
used.  Currently, Emacs uses the FT_LOAD_DEFAULT flag almost
everywhere; this flag implies native TrueType hinting first.  Only if
the TTF doesn't contain bytecode FreeType tries auto-hinting.  So
maybe a simpler alternative is just to test whether we have a TrueType
font:

  if (scalable)
    {
      if (!strcmp(FT_Get_X11_Font_Format(ft_face), "TrueType")
          && !(ft_load_glyph_mode & FT_LOAD_NO_HINTING)
          && !(ft_load_glyph_mode & FT_LOAD_FORCE_AUTOHINT))

Note that this is true for direct FreeType access.  I don't know how
Emacs interacts with GTK, Pango, Cairo, whatever, behaves.  I also
wonder whether Emacs obeys fontconfig settings, which are normally
used to globally set the hinting mode on a GNU/Linux box...


    Werner


[*] I meanwhile consider this late selection of the hinting mode a
    design error in FreeType, which I can't change due to backward
    compatibility.

bug#29078: 25.2; font issue with FreeType 2.8; should not use the rounded ascender and descender

[Lars Ingebrigtsen]

Vincent Lefevre <vincent@vinc17.net> writes:

> For TrueType fonts, FreeType 2.8 uses different rounding rules for
> values in the FT_Size_Metrics structure, which is apparently used
> by Emacs (since there are visible changes). In particular, one can
> now have ascender + descender > height. The consequences are:
>
> 1. Tables look a bit ugly.
>
> 2. Windows (with the same number of text lines) are unnecessarily
>    higher than before.
>
> 3. This can break existing configurations, where Emacs would be
>    opened at start up, with an expected window size.
>
> I could notice changes at least with the default Mono font, which
> appears to be DejaVu Sans Mono.

The Emacs font rendering has changed some in Emacs 27 -- are you still
seeing these problems there?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#29078: 25.2; font issue with FreeType 2.8; should not use the rounded ascender and descender

[Vincent Lefevre]

On 2019-11-17 09:02:50 +0100, Lars Ingebrigtsen wrote:
> Vincent Lefevre <vincent@vinc17.net> writes:
> 
> > For TrueType fonts, FreeType 2.8 uses different rounding rules for
> > values in the FT_Size_Metrics structure, which is apparently used
> > by Emacs (since there are visible changes). In particular, one can
> > now have ascender + descender > height. The consequences are:
> >
> > 1. Tables look a bit ugly.
> >
> > 2. Windows (with the same number of text lines) are unnecessarily
> >    higher than before.
> >
> > 3. This can break existing configurations, where Emacs would be
> >    opened at start up, with an expected window size.
> >
> > I could notice changes at least with the default Mono font, which
> > appears to be DejaVu Sans Mono.
> 
> The Emacs font rendering has changed some in Emacs 27 -- are you still
> seeing these problems there?

With a snapsnot from 23 August 2019,  I'm still seeing them.

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

bug#29078: 25.2; font issue with FreeType 2.8; should not use the rounded ascender and descender

[Lars Ingebrigtsen]

Vincent Lefevre <vincent@vinc17.net> writes:

> With a snapsnot from 23 August 2019,  I'm still seeing them.

OK; thanks for checking.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#29078: 25.2; font issue with FreeType 2.8; should not use the rounded ascender and descender

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Mon, 18 Nov 2019 09:30:42 +0100
> Cc: 29078@debbugs.gnu.org
> 
> Vincent Lefevre <vincent@vinc17.net> writes:
> 
> > With a snapsnot from 23 August 2019,  I'm still seeing them.
> 
> OK; thanks for checking.

The discussion of this issue includes several proposed patches, so
maybe we can dust off one of them and install something similar?

(I don't really understand the issue, so I'm unable to say anything
more intelligent, sorry.)

bug#29078: 25.2; font issue with FreeType 2.8; should not use the rounded ascender and descender

[Robert Pluim]

[-- Attachment #1: Type: text/plain, Size: 999 bytes --]

>>>>> On Mon, 18 Nov 2019 18:05:52 +0200, Eli Zaretskii <eliz@gnu.org> said:

    >> From: Lars Ingebrigtsen <larsi@gnus.org>
    >> Date: Mon, 18 Nov 2019 09:30:42 +0100
    >> Cc: 29078@debbugs.gnu.org
    >> 
    >> Vincent Lefevre <vincent@vinc17.net> writes:
    >> 
    >> > With a snapsnot from 23 August 2019,  I'm still seeing them.
    >> 
    >> OK; thanks for checking.

    Eli> The discussion of this issue includes several proposed patches, so
    Eli> maybe we can dust off one of them and install something similar?

My proposed patch was wrong, so we can remove that from consideration.

    Eli> (I don't really understand the issue, so I'm unable to say anything
    Eli> more intelligent, sorry.)

Iʼll note that Vincent's sample file displays wrong on macOS as well
(this is with
mac-ct:-*-Hack-normal-normal-normal-*-14-*-*-*-m-0-iso10646-1 (#x4C7),
although Menlo shows similar artifacts, and Inconsolata gets the
character widths wrong as well).


[-- Attachment #2: box-drawing-hack.png --]
[-- Type: image/png, Size: 20569 bytes --]

bug#29078: 25.2; font issue with FreeType 2.8; should not use the rounded ascender and descender

[Vincent Lefevre]

On 2019-11-19 11:07:20 +0100, Robert Pluim wrote:
> Iʼll note that Vincent's sample file displays wrong on macOS as well
> (this is with
> mac-ct:-*-Hack-normal-normal-normal-*-14-*-*-*-m-0-iso10646-1 (#x4C7),
> although Menlo shows similar artifacts, and Inconsolata gets the
> character widths wrong as well).

It seems different: no gap, but line joining is a bit wrong.
And artifacts are also visible at column joining.

I would tend to say that this is a completely different issue:
something buggy in the rendering algorithm rather than the
interpretation of the metrics.

BTW, note that my sample file is an excerpt from Markus Kuhn's UTF-8
demo at <https://www.cl.cam.ac.uk/~mgk25/ucs/examples/UTF-8-demo.txt>.

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

bug#29078: 25.2; font issue with FreeType 2.8; should not use the rounded ascender and descender

[Stefan Kangas]

Vincent Lefevre <vincent@vinc17.net> writes:

> BTW, note that my sample file is an excerpt from Markus Kuhn's UTF-8
> demo at <https://www.cl.cam.ac.uk/~mgk25/ucs/examples/UTF-8-demo.txt>.

I tried loading that file up in Emacs on macOS, and from a quick
glance I get better results with the aligning in Emacs than in
Firefox.  I don't see any vertical gaps in the box drawing.

The only issue I see is that "Mathematics and sciences" is
horizontally misaligned.  I think the offending characters are ∮, ≪,
⟦, ⟧, ⎷ and ⎳.  They are rendered in a different font though, so I
guess this is expected?  (I also see a vertical gap on one line but
not the others.)

Best regards,
Stefan Kangas

bug#29078: 25.2; font issue with FreeType 2.8; should not use the rounded ascender and descender

[Eli Zaretskii]

> From: Stefan Kangas <stefan@marxist.se>
> Date: Tue, 19 Nov 2019 17:41:18 +0100
> Cc: Robert Pluim <rpluim@gmail.com>, Lars Ingebrigtsen <larsi@gnus.org>,
>  29078@debbugs.gnu.org
> 
> I tried loading that file up in Emacs on macOS, and from a quick
> glance I get better results with the aligning in Emacs than in
> Firefox.  I don't see any vertical gaps in the box drawing.

You should try different font sizes, sometimes the gaps appear for
specific sizes.

> The only issue I see is that "Mathematics and sciences" is
> horizontally misaligned.  I think the offending characters are ∮, ≪,
> ⟦, ⟧, ⎷ and ⎳.  They are rendered in a different font though, so I
> guess this is expected?  (I also see a vertical gap on one line but
> not the others.)

If that other font is variable-pitch, yes.

bug#29078: 25.2; font issue with FreeType 2.8; should not use the rounded ascender and descender

[Eli Zaretskii]

> Date: Mon, 18 Nov 2019 08:47:14 +0100
> From: Vincent Lefevre <vincent@vinc17.net>
> Cc: 29078@debbugs.gnu.org
> 
> With a snapsnot from 23 August 2019,  I'm still seeing them.

Btw, I think it'd be nice to have a screenshot, so we'd know what this
looks like.  I looked through the linked bug discussion, and couldn't
find any images showing the problem, definitely not in Emacs.

TIA

bug#29078: 25.2; font issue with FreeType 2.8; should not use the rounded ascender and descender

[Vincent Lefevre]

[-- Attachment #1: Type: text/plain, Size: 1071 bytes --]

On 2019-11-18 17:36:34 +0200, Eli Zaretskii wrote:
> Btw, I think it'd be nice to have a screenshot, so we'd know what this
> looks like.  I looked through the linked bug discussion, and couldn't
> find any images showing the problem, definitely not in Emacs.

I've attached screenshots I had done on 2017-10-31 (it seems that
I forgot to send them after my bug report).

This is a GNU Emacs 25 window obtained with

   emacs -Q -fn Mono-10 --geometry 80x16 box-drawing.txt

with libfreetype6 2.6.3-3.2 (emacs-263.png, what is expected) and
2.8.1-0.1 (emacs-281.png, buggy). The behavior with a GNU Emacs
27.0.50 snapshot from August 2019 and libfreetype6 2.10.1-2 seems
identical to emacs-281.png (i.e. nothing has improved).

According to fc-match, Mono-10 corresponds to

DejaVuSansMono.ttf: "DejaVu Sans Mono" "Book"

Other fonts show the same issue.

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

[-- Attachment #2: emacs-263.png --]
[-- Type: image/png, Size: 25652 bytes --]

[-- Attachment #3: emacs-281.png --]
[-- Type: image/png, Size: 27288 bytes --]

[-- Attachment #4: box-drawing.txt --]
[-- Type: text/plain, Size: 1545 bytes --]

Box drawing alignment tests:                                          █
                                                                      ▉
  ╔══╦══╗  ┌──┬──┐  ╭──┬──╮  ╭──┬──╮  ┏━━┳━━┓  ┎┒┏┑   ╷  ╻ ┏┯┓ ┌┰┐    ▊ ╱╲╱╲╳╳╳
  ║┌─╨─┐║  │╔═╧═╗│  │╒═╪═╕│  │╓─╁─╖│  ┃┌─╂─┐┃  ┗╃╄┙  ╶┼╴╺╋╸┠┼┨ ┝╋┥    ▋ ╲╱╲╱╳╳╳
  ║│╲ ╱│║  │║   ║│  ││ │ ││  │║ ┃ ║│  ┃│ ╿ │┃  ┍╅╆┓   ╵  ╹ ┗┷┛ └┸┘    ▌ ╱╲╱╲╳╳╳
  ╠╡ ╳ ╞╣  ├╢   ╟┤  ├┼─┼─┼┤  ├╫─╂─╫┤  ┣┿╾┼╼┿┫  ┕┛┖┚     ┌┄┄┐ ╎ ┏┅┅┓ ┋ ▍ ╲╱╲╱╳╳╳
  ║│╱ ╲│║  │║   ║│  ││ │ ││  │║ ┃ ║│  ┃│ ╽ │┃  ░░▒▒▓▓██ ┊  ┆ ╎ ╏  ┇ ┋ ▎
  ║└─╥─┘║  │╚═╤═╝│  │╘═╪═╛│  │╙─╀─╜│  ┃└─╂─┘┃  ░░▒▒▓▓██ ┊  ┆ ╎ ╏  ┇ ┋ ▏
  ╚══╩══╝  └──┴──┘  ╰──┴──╯  ╰──┴──╯  ┗━━┻━━┛  ▗▄▖▛▀▜   └╌╌┘ ╎ ┗╍╍┛ ┋  ▁▂▃▄▅▆▇█
                                               ▝▀▘▙▄▟

$Id: box-drawing.txt 103168 2017-10-31 12:17:05Z vinc17/cventin $

bug#29078: 25.2; font issue with FreeType 2.8; should not use the rounded ascender and descender

[Eli Zaretskii]

> Date: Mon, 18 Nov 2019 17:04:41 +0100
> From: Vincent Lefevre <vincent@vinc17.net>
> Cc: larsi@gnus.org, 29078@debbugs.gnu.org
> 
> I've attached screenshots I had done on 2017-10-31 (it seems that
> I forgot to send them after my bug report).
> 
> This is a GNU Emacs 25 window obtained with
> 
>    emacs -Q -fn Mono-10 --geometry 80x16 box-drawing.txt
> 
> with libfreetype6 2.6.3-3.2 (emacs-263.png, what is expected) and
> 2.8.1-0.1 (emacs-281.png, buggy). The behavior with a GNU Emacs
> 27.0.50 snapshot from August 2019 and libfreetype6 2.10.1-2 seems
> identical to emacs-281.png (i.e. nothing has improved).
> 
> According to fc-match, Mono-10 corresponds to
> 
> DejaVuSansMono.ttf: "DejaVu Sans Mono" "Book"
> 
> Other fonts show the same issue.

Thanks.

But now I'm not sure there's a bug here.  You evidently expect the
lines of the box-drawing characters to join without gaps, but I'm not
sure what this expectation is based on.  For example, Emacs on
MS-Windows doesn't use FreeType, but with a completely different font
I see the gaps here as well.  The fact that older versions of FreeType
produced different display doesn't yet mean that display was correct
and the current one isn't.

So I'm inclined to argue that there's no bug here.  What am I missing?

bug#29078: 25.2; font issue with FreeType 2.8; should not use the rounded ascender and descender

[Vincent Lefevre]

On 2019-11-18 19:09:37 +0200, Eli Zaretskii wrote:
> But now I'm not sure there's a bug here.  You evidently expect the
> lines of the box-drawing characters to join without gaps, but I'm not
> sure what this expectation is based on.  For example, Emacs on
> MS-Windows doesn't use FreeType, but with a completely different font
> I see the gaps here as well.  The fact that older versions of FreeType
> produced different display doesn't yet mean that display was correct
> and the current one isn't.
> 
> So I'm inclined to argue that there's no bug here.  What am I missing?

AFAIK, from the font description (which does not involve rounding),
there is no gap (that's why the FreeType developers suggested not
to use the rounded values). Moreover, it does not make much sense
to insert a gap. This looks ugly and this wastes screen space.

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

bug#29078: 25.2; font issue with FreeType 2.8; should not use the rounded ascender and descender

[Eli Zaretskii]

> Date: Mon, 18 Nov 2019 18:23:33 +0100
> From: Vincent Lefevre <vincent@vinc17.net>
> Cc: larsi@gnus.org, 29078@debbugs.gnu.org
> 
> AFAIK, from the font description (which does not involve rounding),
> there is no gap (that's why the FreeType developers suggested not
> to use the rounded values). Moreover, it does not make much sense
> to insert a gap. This looks ugly and this wastes screen space.

Emacs doesn't insert any gaps, it just computes the line metrics from
all the characters displayed on that line.

I've read the discussions of the FreeType developers, but couldn't
understand what they were talking about, as the discussion was in
terms of internals of FreeType, and I don't know enough to map that to
what the Emacs display engine does.

My point is that this seems to have nothing to do with the FreeType
library or its version, since I see the same on a system that uses
neither FreeType nor this particular font.  IOW, this looks to me
"normal", i.e. Emacs always worked like that.

bug#29078: 25.2; font issue with FreeType 2.8; should not use the rounded ascender and descender

[Eli Zaretskii]

> Date: Mon, 18 Nov 2019 19:50:09 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: larsi@gnus.org, 29078@debbugs.gnu.org
> 
> My point is that this seems to have nothing to do with the FreeType
> library or its version, since I see the same on a system that uses
> neither FreeType nor this particular font.

And indeed the reason here seems to be that some of the box-drawing
characters are taken from a different font.

Could it be that the same happens on your system?  If you press and
hold the right arrow key on the first line with the boxes (the 3rd
line of the file, starting from top), do you see the block cursor
sometimes becoming slightly larger?  If so, the characters where it
becomes larger come from a different font, which is slightly taller
than the default font; delete those taller characters, and the gaps
should disappear.  They did here.

bug#29078: 25.2; font issue with FreeType 2.8; should not use the rounded ascender and descender

[Vincent Lefevre]

On 2019-11-18 19:58:57 +0200, Eli Zaretskii wrote:
> > Date: Mon, 18 Nov 2019 19:50:09 +0200
> > From: Eli Zaretskii <eliz@gnu.org>
> > Cc: larsi@gnus.org, 29078@debbugs.gnu.org
> > 
> > My point is that this seems to have nothing to do with the FreeType
> > library or its version, since I see the same on a system that uses
> > neither FreeType nor this particular font.
> 
> And indeed the reason here seems to be that some of the box-drawing
> characters are taken from a different font.
> 
> Could it be that the same happens on your system?  If you press and
> hold the right arrow key on the first line with the boxes (the 3rd
> line of the file, starting from top), do you see the block cursor
> sometimes becoming slightly larger?

No, the block cursor still seems to have the same size.

> If so, the characters where it becomes larger come from a different
> font, which is slightly taller than the default font; delete those
> taller characters, and the gaps should disappear. They did here.

Even if I take just

╔══╦══╗
║┌─╨─┐║

I still see the gap.

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

bug#29078: 25.2; font issue with FreeType 2.8; should not use the rounded ascender and descender

[Eli Zaretskii]

> Date: Mon, 18 Nov 2019 19:20:49 +0100
> From: Vincent Lefevre <vincent@vinc17.net>
> Cc: larsi@gnus.org, 29078@debbugs.gnu.org
> 
> > Could it be that the same happens on your system?  If you press and
> > hold the right arrow key on the first line with the boxes (the 3rd
> > line of the file, starting from top), do you see the block cursor
> > sometimes becoming slightly larger?
> 
> No, the block cursor still seems to have the same size.
> 
> > If so, the characters where it becomes larger come from a different
> > font, which is slightly taller than the default font; delete those
> > taller characters, and the gaps should disappear. They did here.
> 
> Even if I take just
> 
> ╔══╦══╗
> ║┌─╨─┐║
> 
> I still see the gap.

OK, so the next step is for someone to step through the code in
xdisp.c that computes the line metrics, and see what are the metrics
of these characters.  We can then compare with a build that doesn't
use FreeType, and take it from there.

Or maybe some FreeType expert could look at our code and suggest what
to do, because I don't have a clue.

bug#29078: 25.2; font issue with FreeType 2.8; should not use the rounded ascender and descender

[Vincent Lefevre]

[-- Attachment #1: Type: text/plain, Size: 2074 bytes --]

On 2019-11-18 19:50:09 +0200, Eli Zaretskii wrote:
> > Date: Mon, 18 Nov 2019 18:23:33 +0100
> > From: Vincent Lefevre <vincent@vinc17.net>
> > Cc: larsi@gnus.org, 29078@debbugs.gnu.org
> > 
> > AFAIK, from the font description (which does not involve rounding),
> > there is no gap (that's why the FreeType developers suggested not
> > to use the rounded values). Moreover, it does not make much sense
> > to insert a gap. This looks ugly and this wastes screen space.
> 
> Emacs doesn't insert any gaps, it just computes the line metrics from
> all the characters displayed on that line.

There are different ways to do the computation, and due to the rounded
values, they give different results. Compared to the cell height,
there is a gap. When I analyzed the issue with xterm, I could find
that with the rounded metrics, height < ascent + descent (while this
is an equality with the unrounded metrics).

> I've read the discussions of the FreeType developers, but couldn't
> understand what they were talking about, as the discussion was in
> terms of internals of FreeType, and I don't know enough to map that to
> what the Emacs display engine does.
> 
> My point is that this seems to have nothing to do with the FreeType
> library or its version,

The issue appeared just after upgrading FreeType from 2.6 to 2.8,
and the FreeType developers said this was because they changed the
rounding rules for ttf fonts.

> since I see the same on a system that uses neither FreeType nor this
> particular font. IOW, this looks to me "normal", i.e. Emacs always
> worked like that.

Of course, if you change the font, you may see something different.

I've attached what I get with the fixed bitmap font. As you can see,
there is no gap, and this is the expected rendering (with bitmap fonts,
there are no rounding issues, thus what is shown is as designed).

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

[-- Attachment #2: fixed.png --]
[-- Type: image/png, Size: 7995 bytes --]

bug#29078: 25.2; font issue with FreeType 2.8; should not use the rounded ascender and descender

[Lars Ingebrigtsen]

[-- Attachment #1: Type: text/plain, Size: 540 bytes --]

Vincent Lefevre <vincent@vinc17.net> writes:

> with libfreetype6 2.6.3-3.2 (emacs-263.png, what is expected) and
> 2.8.1-0.1 (emacs-281.png, buggy). The behavior with a GNU Emacs
> 27.0.50 snapshot from August 2019 and libfreetype6 2.10.1-2 seems
> identical to emacs-281.png (i.e. nothing has improved).

Using

    ftcrhb:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-25-*-*-*-m-0-iso10646-1 (#x957)

and

libfreetype6/stable,now 2.9.1-3+deb10u1

(although I'm not sure that's what Emacs uses), I don't get a buggy
display in Emacs 27:


[-- Attachment #2: screenzMqTG9.jpg --]
[-- Type: image/jpeg, Size: 4705 bytes --]

[-- Attachment #3: Type: text/plain, Size: 156 bytes --]


That is, there are no gaps here.  (Debian Stable.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#29078: 25.2; font issue with FreeType 2.8; should not use the rounded ascender and descender

[Vincent Lefevre]

On 2019-11-18 19:32:41 +0100, Lars Ingebrigtsen wrote:
> Using
> 
>     ftcrhb:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-25-*-*-*-m-0-iso10646-1 (#x957)

Under Debian/unstable, I don't get it either. Neither with

  ftcrhb:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-15-*-*-*-m-0-iso10646-1

But the bug occurs with

  ftcrhb:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-18-*-*-*-m-0-iso10646-1

That's for the rendering. I don't know whether the cell height is
honored in all these cases.

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

bug#29078: 25.2; font issue with FreeType 2.8; should not use the rounded ascender and descender

[Lars Ingebrigtsen]

Vincent Lefevre <vincent@vinc17.net> writes:

> On 2019-11-18 19:32:41 +0100, Lars Ingebrigtsen wrote:
>> Using
>> 
>>     ftcrhb:-PfEd-DejaVu Sans
>> Mono-normal-normal-normal-*-25-*-*-*-m-0-iso10646-1 (#x957)
>
> Under Debian/unstable, I don't get it either. Neither with
>
>   ftcrhb:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-15-*-*-*-m-0-iso10646-1
>
> But the bug occurs with
>
>   ftcrhb:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-18-*-*-*-m-0-iso10646-1
>
> That's for the rendering. I don't know whether the cell height is
> honored in all these cases.

Ah, yes.  If I say

emacs -Q -fn "DejaVu Sans Mono:size=30"

and vary the size, I get the gaps in the lines for some sizes (like 23)
but not others (like 25).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#29078: 25.2; font issue with FreeType 2.8; should not use the rounded ascender and descender

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  29078@debbugs.gnu.org
> Date: Mon, 18 Nov 2019 21:31:43 +0100
> 
> Vincent Lefevre <vincent@vinc17.net> writes:
> 
> > On 2019-11-18 19:32:41 +0100, Lars Ingebrigtsen wrote:
> >> Using
> >> 
> >>     ftcrhb:-PfEd-DejaVu Sans
> >> Mono-normal-normal-normal-*-25-*-*-*-m-0-iso10646-1 (#x957)
> >
> > Under Debian/unstable, I don't get it either. Neither with
> >
> >   ftcrhb:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-15-*-*-*-m-0-iso10646-1
> >
> > But the bug occurs with
> >
> >   ftcrhb:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-18-*-*-*-m-0-iso10646-1
> >
> > That's for the rendering. I don't know whether the cell height is
> > honored in all these cases.
> 
> Ah, yes.  If I say
> 
> emacs -Q -fn "DejaVu Sans Mono:size=30"
> 
> and vary the size, I get the gaps in the lines for some sizes (like 23)
> but not others (like 25).

Strangely, I see small (1- or 2-pixel) gaps for Courier New on
MS-Windows, and only for a couple of size values.  Moreover, with
DejaVu Sans Mono, I don't see any gaps for all sizes between 9 and 30.

FWIW, the font-related APIs on MS-Windows report glyph metrics as
integer values, so Emacs doesn't round anywhere.  So either MS-Windows
does a better job with that font (unlikely) or there's something else
at work here.

I guess we will have to wait until someone tells us what to do here.