bug#20628: 25.0.50; Incorrect line height for some fonts

bug#20628: 25.0.50; Incorrect line height for some fonts

[Clément Pit--Claudel]

Hi,

Some Unicode characters incorrectly increase the height of the line on which
they are inserted, by an incorrect amount (typically 4/5 times the normal line
height). This is due to Emacs falling back to a font for which line height
calculations are incorrect. The problem can generally be reproduced just by
inputing the following characters:

  (𝓝𝓟)

Alternatively, the problem can be reproduced by switching to certain
specific fonts. For example:

  (set-frame-font "-unknown-Latin Modern Math-normal-normal-normal-*-*-*-*-*-*-0-iso10646-1" nil nil)

This problem is not specific to 25.0.50. It is discussed on stackexchange [1]
and the effect can be observed in [2]. It only occurs with specific fonts. It
particularly impacts packages that rely on prettify-symbols-mode to display math
symbols; when users install the package, some lines in the buffer start being 4
or 5 times taller than other lines, although no characters on the affected lines
stand out. For this reason, even if this is likely a problem in the way the
fonts are packaged, it would be nice to have a workaround at the Emacs level.

Emacs is the only program on my system that displays this behaviour; typing the
same text in gedit or switching gedit to one of the misbehaving fonts, for
example, does not affect the line height.

[1] http://emacs.stackexchange.com/questions/251/
[2] https://cloud.githubusercontent.com/assets/2506825/7760973/67ceaaea-ffd5-11e4-8bf6-d796aa162b0e.png

---

In GNU Emacs 25.0.50.4 (x86_64-unknown-linux-gnu, GTK+ Version 3.10.8)
 of 2015-05-14 on c-mint
Windowing system distributor `The X.Org Foundation', version 11.0.11501000
System Description:	Linux Mint 17.1 Rebecca

Configured features:
XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GCONF GSETTINGS
NOTIFY ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB

Important settings:
  value of $LC_TIME: en_DK.UTF-8
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.

Load-path shadows:
None found.

Features:
(shadow sort gnus-util mail-extr emacsbug message dired format-spec
rfc822 mml easymenu mml-sec mm-decode mm-bodies mm-encode mail-parse
rfc2231 mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045
ietf-drums mm-util help-fns mail-prsvr mail-utils mule-util time-date
tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel x-win term/common-win x-dnd tool-bar dnd fontset image regexp-opt
fringe tabulated-list newcomment elisp-mode lisp-mode prog-mode register
page menu-bar rfn-eshadow timer select scroll-bar mouse jit-lock
font-lock syntax facemenu font-core frame cl-generic cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese case-table epa-hook jka-cmpr-hook help simple abbrev
minibuffer cl-preloaded nadvice loaddefs button faces cus-face macroexp
files text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote dbusbind gfilenotify
dynamic-setting system-font-setting font-render-setting move-toolbar gtk
x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 76472 5652)
 (symbols 48 18496 0)
 (miscs 40 87 99)
 (strings 32 11421 4141)
 (string-bytes 1 314062)
 (vectors 16 9839)
 (vector-slots 8 396279 11643)
 (floats 8 102 46)
 (intervals 56 177 0)
 (buffers 976 11)
 (heap 1024 40850 1039))

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> Date: Thu, 21 May 2015 23:02:06 -0400
> From: Clément Pit--Claudel
>  <clement.pitclaudel@live.com>
> 
> Some Unicode characters incorrectly increase the height of the line on which
> they are inserted, by an incorrect amount (typically 4/5 times the normal line
> height). This is due to Emacs falling back to a font for which line height
> calculations are incorrect. The problem can generally be reproduced just by
> inputing the following characters:
> 
>   (𝓝𝓟)
> 
> Alternatively, the problem can be reproduced by switching to certain
> specific fonts. For example:
> 
>   (set-frame-font "-unknown-Latin Modern Math-normal-normal-normal-*-*-*-*-*-*-0-iso10646-1" nil nil)
> 
> This problem is not specific to 25.0.50. It is discussed on stackexchange [1]
> and the effect can be observed in [2]. It only occurs with specific fonts. It
> particularly impacts packages that rely on prettify-symbols-mode to display math
> symbols; when users install the package, some lines in the buffer start being 4
> or 5 times taller than other lines, although no characters on the affected lines
> stand out. For this reason, even if this is likely a problem in the way the
> fonts are packaged, it would be nice to have a workaround at the Emacs level.
> 
> Emacs is the only program on my system that displays this behaviour; typing the
> same text in gedit or switching gedit to one of the misbehaving fonts, for
> example, does not affect the line height.

I'm sorry, but I don't see the bug here, at least not a bug I know how
to fix.  Emacs obeys the information about the glyph sizes that the
font supplies; if the font says the font glyphs need more vertical
space than in the font used for the default face, then Emacs will
enlarge line height.  What else can it do?

In particular, the ugly effect in
https://cloud.githubusercontent.com/assets/2506825/7760973/67ceaaea-ffd5-11e4-8bf6-d796aa162b0e.png
is most probably caused by using a font whose Math Alphanumerics block
specifies a very large vertical space, to accommodate for
superscripts, power, integrals, etc.  Just switch to a different font
which covers the same block without this adverse side effect.

So the only way to fix this (if it really can be fixed) is for someone
who knows enough about these fonts to explain (a) why the metrics
provided by these fonts are incorrect, and (b) what metrics to use
instead and how to compute them.  Failing that, there's nothing that
can be done here, as Emacs behaves as required and as designed.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Oleh Krehel]

Eli Zaretskii <eliz@gnu.org> writes:

> In particular, the ugly effect in
> https://cloud.githubusercontent.com/assets/2506825/7760973/67ceaaea-ffd5-11e4-8bf6-d796aa162b0e.png
> is most probably caused by using a font whose Math Alphanumerics block
> specifies a very large vertical space, to accommodate for
> superscripts, power, integrals, etc.  Just switch to a different font
> which covers the same block without this adverse side effect.
>
> So the only way to fix this (if it really can be fixed) is for someone
> who knows enough about these fonts to explain (a) why the metrics
> provided by these fonts are incorrect, and (b) what metrics to use
> instead and how to compute them.  Failing that, there's nothing that
> can be done here, as Emacs behaves as required and as designed.

I have the same problem with DejaVu:

    -unknown-DejaVu Sans Mono-normal-normal-normal-*-15-*-*-*-m-0-fontset-auto2

which is basically the most popular free font. And yet when I paste a
problematic char, like 𝝹 into gedit, configured also with DejaVu Sans Mono,
there's no issue. So it can be solved, since gedit does it.

Oleh

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> From: Oleh Krehel <ohwoeowho@gmail.com>
> Cc: Clément Pit--Claudel <clement.pitclaudel@live.com>,
>   20628@debbugs.gnu.org
> Date: Fri, 22 May 2015 12:11:40 +0200
> 
> I have the same problem with DejaVu:
> 
>     -unknown-DejaVu Sans Mono-normal-normal-normal-*-15-*-*-*-m-0-fontset-auto2
> 
> which is basically the most popular free font. And yet when I paste a
> problematic char, like 𝝹 into gedit, configured also with DejaVu Sans Mono,
> there's no issue. So it can be solved, since gedit does it.

If someone tells how to fix that, that would be welcome.  Failing
that, I will claim that gedit has a bug (did you try all the glyphs in
that font, to make sure none of them come out cropped in some way?).

bug#20628: 25.0.50; Incorrect line height for some fonts

[Oleh Krehel]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Oleh Krehel <ohwoeowho@gmail.com>
>> Cc: Clément Pit--Claudel <clement.pitclaudel@live.com>,
>>   20628@debbugs.gnu.org
>> Date: Fri, 22 May 2015 12:11:40 +0200
>> 
>> I have the same problem with DejaVu:
>> 
>>     -unknown-DejaVu Sans Mono-normal-normal-normal-*-15-*-*-*-m-0-fontset-auto2
>> 
>> which is basically the most popular free font. And yet when I paste a
>> problematic char, like 𝝹 into gedit, configured also with DejaVu Sans Mono,
>> there's no issue. So it can be solved, since gedit does it.
>
> If someone tells how to fix that, that would be welcome.  Failing
> that, I will claim that gedit has a bug (did you try all the glyphs in
> that font, to make sure none of them come out cropped in some way?).

The test file with contents:
    asdf 𝞳

is displayed fine with "emacs -nw" when inside a GNOME Terminal.  Other
applications like Firefox, Gedit, and Chromium also display it fine.

Could you point me to a code position in Emacs where the line pixel
height is determined. I'll try to play around with it, maybe I can fix
it at least for GTK.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> Date: Fri, 22 May 2015 15:26:47 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: clement.pitclaudel@live.com, 20628@debbugs.gnu.org
> 
> > From: Oleh Krehel <ohwoeowho@gmail.com>
> > Cc: Clément Pit--Claudel <clement.pitclaudel@live.com>,
> >   20628@debbugs.gnu.org
> > Date: Fri, 22 May 2015 12:11:40 +0200
> > 
> > I have the same problem with DejaVu:
> > 
> >     -unknown-DejaVu Sans Mono-normal-normal-normal-*-15-*-*-*-m-0-fontset-auto2
> > 
> > which is basically the most popular free font. And yet when I paste a
> > problematic char, like 𝝹 into gedit, configured also with DejaVu Sans Mono,
> > there's no issue. So it can be solved, since gedit does it.
> 
> If someone tells how to fix that, that would be welcome.  Failing
> that, I will claim that gedit has a bug (did you try all the glyphs in
> that font, to make sure none of them come out cropped in some way?).

Btw, in the version of DejaVu Sans Mono I just downloaded from their
site doesn't cover the U+01D779 character.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> From: Oleh Krehel <ohwoeowho@gmail.com>
> Cc: clement.pitclaudel@live.com,  20628@debbugs.gnu.org
> Date: Fri, 22 May 2015 14:49:16 +0200
> 
> The test file with contents:
>     asdf 𝞳
> 
> is displayed fine with "emacs -nw" when inside a GNOME Terminal.

In "emacs -nw", Emacs doesn't perform layout, it just writes the
characters using stdio functions.  It's the terminal emulator that
performs layout.

> Other applications like Firefox, Gedit, and Chromium also display it
> fine.

Knowing that doesn't help in fixing Emacs behavior.

> Could you point me to a code position in Emacs where the line pixel
> height is determined.

It's in xdisp.c:x_produce_glyphs.  Emacs updates the line's ascent and
descent as it processes each character for display, based on the data
it gets from the font used by the face of that character.  Later in
display_line, the height of each screen line is determined:

	  row->ascent = max (row->ascent, it->max_ascent);
	  row->height = max (row->height, it->max_ascent + it->max_descent);
	  row->phys_ascent = max (row->phys_ascent, it->max_phys_ascent);
	  row->phys_height = max (row->phys_height,
				  it->max_phys_ascent + it->max_phys_descent);
	  row->extra_line_spacing = max (row->extra_line_spacing,
					 it->max_extra_line_spacing);

> I'll try to play around with it, maybe I can fix it at least for
> GTK.

This code is device independent, so GTK doesn't come into play here.

Thanks in advance for any solutions you might find and/or suggest.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Oleh Krehel]

Eli Zaretskii <eliz@gnu.org> writes:

> Btw, in the version of DejaVu Sans Mono I just downloaded from their
> site doesn't cover the U+01D779 character.

It appears to re-use the system fixed width font (Ubuntu Mono 13) in
case the selected font (DejaVu Sans Mono) doesn't support a char.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> From: Oleh Krehel <ohwoeowho@gmail.com>
> Cc: clement.pitclaudel@live.com,  20628@debbugs.gnu.org
> Date: Fri, 22 May 2015 14:49:16 +0200
> 
> The test file with contents:
>     asdf 𝞳
> 
> is displayed fine with "emacs -nw" when inside a GNOME Terminal.

In "emacs -nw", Emacs doesn't perform layout, it just writes the
characters using stdio functions.  It's the terminal emulator that
performs layout.

> Other applications like Firefox, Gedit, and Chromium also display it
> fine.

Knowing that doesn't help in fixing Emacs behavior.

> Could you point me to a code position in Emacs where the line pixel
> height is determined.

It's in xdisp.c:x_produce_glyphs.  Emacs updates the line's ascent and
descent as it processes each character for display, based on the data
it gets from the font used by the face of that character.  Later in
display_line, the height of each screen line is determined:

	  row->ascent = max (row->ascent, it->max_ascent);
	  row->height = max (row->height, it->max_ascent + it->max_descent);
	  row->phys_ascent = max (row->phys_ascent, it->max_phys_ascent);
	  row->phys_height = max (row->phys_height,
				  it->max_phys_ascent + it->max_phys_descent);
	  row->extra_line_spacing = max (row->extra_line_spacing,
					 it->max_extra_line_spacing);

> I'll try to play around with it, maybe I can fix it at least for
> GTK.

The code mentioned above is device independent, so GTK doesn't come
into play here.

Thanks in advance for any solutions you might find and/or suggest.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

> Btw, in the version of DejaVu Sans Mono I just downloaded from their
> site doesn't cover the U+01D779 character.

On my system Emacs choses TeX Gyre Pagella Math to display 𝝹. 

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#20628: 25.0.50; Incorrect line height for some fonts

[Oleh Krehel]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Oleh Krehel <ohwoeowho@gmail.com>
>> Cc: clement.pitclaudel@live.com,  20628@debbugs.gnu.org
>> Date: Fri, 22 May 2015 15:15:45 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > Btw, in the version of DejaVu Sans Mono I just downloaded from their
>> > site doesn't cover the U+01D779 character.
>> 
>> It appears to re-use the system fixed width font (Ubuntu Mono 13) in
>> case the selected font (DejaVu Sans Mono) doesn't support a char.
>
> So that means the problem is not with DejaVu Sans Mono, it's with the
> fallback font, right?

Right.

I'm looking at xdisp.c now. When I set this:

    it->max_ascent = 0;
    it->max_descent = 0;

the problem disappears. Of course, it causes a problem in places where
ascent and descent are actually used, like for displaying images. I just
need to figure out how it->max_ascent gets computed from glyph->descent.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> From: Oleh Krehel <ohwoeowho@gmail.com>
> Cc: clement.pitclaudel@live.com,  20628@debbugs.gnu.org
> Date: Fri, 22 May 2015 15:15:45 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Btw, in the version of DejaVu Sans Mono I just downloaded from their
> > site doesn't cover the U+01D779 character.
> 
> It appears to re-use the system fixed width font (Ubuntu Mono 13) in
> case the selected font (DejaVu Sans Mono) doesn't support a char.

So that means the problem is not with DejaVu Sans Mono, it's with the
fallback font, right?

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> From: Oleh Krehel <ohwoeowho@gmail.com>
> Cc: clement.pitclaudel@live.com,  20628@debbugs.gnu.org
> Date: Fri, 22 May 2015 15:54:21 +0200
> 
> I'm looking at xdisp.c now. When I set this:
> 
>     it->max_ascent = 0;
>     it->max_descent = 0;
> 
> the problem disappears.

Of course, it does: you've just made Emacs ignore characters which
have non-zero ascent and descent.

> Of course, it causes a problem in places where ascent and descent
> are actually used, like for displaying images.

More importantly, it will display characters with ascent or descent
incorrectly clipped.

> I just need to figure out how it->max_ascent gets computed from
> glyph->descent.

Like this:

  it->max_ascent = max (it->max_ascent, it->ascent);
  it->max_descent = max (it->max_descent, it->descent);

IOW, it's the max value of ascent and descent of all the characters on
that screen line.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Oleh Krehel]

Eli Zaretskii <eliz@gnu.org> writes:

> Like this:
>
>   it->max_ascent = max (it->max_ascent, it->ascent);
>   it->max_descent = max (it->max_descent, it->descent);
>
> IOW, it's the max value of ascent and descent of all the characters on
> that screen line.

OK, I got this far:

    p FACE_FROM_ID(it->f,it->face_id)->font->ascent
    $18 = 15
    (gdb) p FACE_FROM_ID(it->f,it->face_id)->font->descent
    $19 = 4

15 and 4 are the eventual (wrong) values of it->max_ascent and
it->max_descent. But I don't know how and where the font structure is
initialized and how the current glyph actually is factored here: I just
see a reference to a frame and a face, no reference to the current char.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> From: Oleh Krehel <ohwoeowho@gmail.com>
> Cc: clement.pitclaudel@live.com,  20628@debbugs.gnu.org
> Date: Fri, 22 May 2015 16:20:31 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Like this:
> >
> >   it->max_ascent = max (it->max_ascent, it->ascent);
> >   it->max_descent = max (it->max_descent, it->descent);
> >
> > IOW, it's the max value of ascent and descent of all the characters on
> > that screen line.
> 
> OK, I got this far:
> 
>     p FACE_FROM_ID(it->f,it->face_id)->font->ascent
>     $18 = 15
>     (gdb) p FACE_FROM_ID(it->f,it->face_id)->font->descent
>     $19 = 4
> 
> 15 and 4 are the eventual (wrong) values of it->max_ascent and
> it->max_descent.

Why do you think they are wrong?

> But I don't know how and where the font structure is initialized and
> how the current glyph actually is factored here: I just see a
> reference to a frame and a face, no reference to the current char.

AFAIK, they are initialized from the font data.  Here's what ftfont.c
does in ftfont_open:

  scalable = (INTEGERP (AREF (entity, FONT_AVGWIDTH_INDEX))
	      && XINT (AREF (entity, FONT_AVGWIDTH_INDEX)) == 0);
  if (scalable)
    {
      font->ascent = ft_face->ascender * size / upEM;
      font->descent = - ft_face->descender * size / upEM;
      font->height = ft_face->height * size / upEM;
    }
  else
    {
      font->ascent = ft_face->size->metrics.ascender >> 6;
      font->descent = - ft_face->size->metrics.descender >> 6;
      font->height = ft_face->size->metrics.height >> 6;
    }

And the fields of ft_face seem to be set by FreeType library, via the
call to FT_Set_Pixel_Sizes, a few lines before the above snippet.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Oleh Krehel]

Eli Zaretskii <eliz@gnu.org> writes:

> Why do you think they are wrong?

Never mind, I made a bad assumption.

Here's what I've found by looking at:

    ;; equivalent:
    (format "%c" #x01d779)
    (format "%c" 120697)

    ;; what gets passed to x_produce_glyphs:
    (format "%c" 120755)

So now, if I put this check at the end of x_produce_glyphs:

    if (it->char_to_display == 120755)
    {
      it->max_ascent = 0;
      it->max_descent = 0;
    }

The problem is solved for this one char. So now the question is how
120697 got translated into 120755?

Also, for some reason I can't evaluate this in gdb:

   p get_char_glyph_code (it->char_to_display, font, &char2b)

It says that get_char_glyph_code isn't defined. In any case, for this
character it returns FONT_INVALID_CODE.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> Date: Fri, 22 May 2015 17:49:16 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: clement.pitclaudel@live.com, 20628@debbugs.gnu.org
> 
> AFAIK, they are initialized from the font data.  Here's what ftfont.c
> does in ftfont_open:
> 
>   scalable = (INTEGERP (AREF (entity, FONT_AVGWIDTH_INDEX))
> 	      && XINT (AREF (entity, FONT_AVGWIDTH_INDEX)) == 0);
>   if (scalable)
>     {
>       font->ascent = ft_face->ascender * size / upEM;
>       font->descent = - ft_face->descender * size / upEM;
>       font->height = ft_face->height * size / upEM;
>     }
>   else
>     {
>       font->ascent = ft_face->size->metrics.ascender >> 6;
>       font->descent = - ft_face->size->metrics.descender >> 6;
>       font->height = ft_face->size->metrics.height >> 6;
>     }
> 
> And the fields of ft_face seem to be set by FreeType library, via the
> call to FT_Set_Pixel_Sizes, a few lines before the above snippet.

Forgot to tell that similar code you can find in xfont.c and
xftfont.c (and w32font.c for Windows).  Which of these is actually
used in your Emacs depends on how it was configured and against which
libraries it was linked.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Rasmus]

Clément Pit--Claudel <clement.pitclaudel@live.com> writes:

> Some Unicode characters incorrectly increase the height of the line on which
> they are inserted, by an incorrect amount (typically 4/5 times the normal line
> height). This is due to Emacs falling back to a font for which line height
> calculations are incorrect. The problem can generally be reproduced just by
> inputing the following characters:
>
>   (𝓝𝓟)
>
> Alternatively, the problem can be reproduced by switching to certain
> specific fonts. For example:
>
>   (set-frame-font "-unknown-Latin Modern Math-normal-normal-normal-*-*-*-*-*-*-0-iso10646-1" nil nil)

It would be great if this "bug" could be fixed somehow.

As a workaround, I have found that XITS Math behaves nicely.  I have
something like this in my init (much simplified).

(mapc (lambda (set)
        (set-fontset-font set 'mathematical  (font-spec :family "XITS Math") nil 'append)
        (set-fontset-font set 'symbol  (font-spec :family "DejaVu Sans Mono") nil 'prepend))
      '("fontset-startup" "fontset-default"))

—Rasmus

-- 
I almost cut my hair, it happened just the other day

bug#20628: 25.0.50; Incorrect line height for some fonts

[Stefan Monnier]

> And the fields of ft_face seem to be set by FreeType library, via the
> call to FT_Set_Pixel_Sizes, a few lines before the above snippet.

Could it be that the difference between Emacs and things like GEdit is
that we request the ascent/descent of the whole font, whereas GEdit
requests this data on a glyph-by-glyph basis?


        Stefan

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> From: Oleh Krehel <ohwoeowho@gmail.com>
> Cc: clement.pitclaudel@live.com,  20628@debbugs.gnu.org
> Date: Fri, 22 May 2015 17:03:29 +0200
> 
>     ;; equivalent:
>     (format "%c" #x01d779)
>     (format "%c" 120697)
> 
>     ;; what gets passed to x_produce_glyphs:
>     (format "%c" 120755)
> 
> So now, if I put this check at the end of x_produce_glyphs:
> 
>     if (it->char_to_display == 120755)
>     {
>       it->max_ascent = 0;
>       it->max_descent = 0;
>     }
> 
> The problem is solved for this one char. So now the question is how
> 120697 got translated into 120755?

I can't reproduce this: I only see 120697.

I'd suggest to put a by-location watchpoint on it->char_to_display,
conditioned on the value being greater than 120000, and see which code
sets it to 120755.

> Also, for some reason I can't evaluate this in gdb:
> 
>    p get_char_glyph_code (it->char_to_display, font, &char2b)
> 
> It says that get_char_glyph_code isn't defined.

Is your build optimized, per chance?  It works for me, FWIW.

> In any case, for this character it returns FONT_INVALID_CODE.

Not sure how this is relevant; and it should return either 'true' or
'false'.

Anyway, on my system, this character enlarges the it->max_ascent value
by 1 pixel, and leaves the it->max_descent value unchanged.  The font
used to display it is Code2001, and the default font is Courier New.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Andreas Schwab]

Oleh Krehel <ohwoeowho@gmail.com> writes:

> Also, for some reason I can't evaluate this in gdb:
>
>    p get_char_glyph_code (it->char_to_display, font, &char2b)
>
> It says that get_char_glyph_code isn't defined.

It's probably inlined, since it's called only once.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Oleh Krehel <ohwoeowho@gmail.com>,  clement.pitclaudel@live.com,  20628@debbugs.gnu.org
> Date: Fri, 22 May 2015 11:27:45 -0400
> 
> > And the fields of ft_face seem to be set by FreeType library, via the
> > call to FT_Set_Pixel_Sizes, a few lines before the above snippet.
> 
> Could it be that the difference between Emacs and things like GEdit is
> that we request the ascent/descent of the whole font, whereas GEdit
> requests this data on a glyph-by-glyph basis?

_Is_ there such a thing as glyph-specific ascent and descent values?

bug#20628: 25.0.50; Incorrect line height for some fonts

[Oleh Krehel]

Eli Zaretskii <eliz@gnu.org> writes:

> Is your build optimized, per chance?  It works for me, FWIW.

How do I set it to non-optimized? So far I've tracked the bug to
xftfont_open, where instead of the usual

    font ascent: 14, descent: 5

I get for the 120755 char:

    font ascent: 54, descent: 46

However, a lot of the stuff in xftfont_open is optimized out, so I need
to figure out how to re-make with -O0.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> From: Oleh Krehel <ohwoeowho@gmail.com>
> Cc: clement.pitclaudel@live.com,  20628@debbugs.gnu.org
> Date: Fri, 22 May 2015 18:05:25 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Is your build optimized, per chance?  It works for me, FWIW.
> 
> How do I set it to non-optimized?

Reconfigure like this:

  CFLAGS='-O0 -g3' ./configure ...

and then re-run "make".

> So far I've tracked the bug to xftfont_open, where instead of the
> usual
> 
>     font ascent: 14, descent: 5
> 
> I get for the 120755 char:
> 
>     font ascent: 54, descent: 46

How is this strange character displayed, eventually?  Is it displayed
as a character or as a box with its hex code?

bug#20628: 25.0.50; Incorrect line height for some fonts

[Oleh Krehel]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Oleh Krehel <ohwoeowho@gmail.com>
>> Cc: clement.pitclaudel@live.com,  20628@debbugs.gnu.org
>> Date: Fri, 22 May 2015 18:05:25 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > Is your build optimized, per chance?  It works for me, FWIW.
>> 
>> How do I set it to non-optimized?
>
> Reconfigure like this:
>
>   CFLAGS='-O0 -g3' ./configure ...
>
> and then re-run "make".

Thanks, I'll try that.

>> So far I've tracked the bug to xftfont_open, where instead of the
>> usual
>> 
>>     font ascent: 14, descent: 5
>> 
>> I get for the 120755 char:
>> 
>>     font ascent: 54, descent: 46
>
> How is this strange character displayed, eventually?  Is it displayed
> as a character or as a box with its hex code?

It's displayed just the way gedit displays it, only with a huge (3 line
height) ascent and a huge (3 line height) descent. So the line with the
char is effectively 7 times taller than a regular line.

Also, I've tracked the font family of the char to be "Latin Modern
Mathfont" in xftfont_open.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> From: Oleh Krehel <ohwoeowho@gmail.com>
> Cc: clement.pitclaudel@live.com,  20628@debbugs.gnu.org
> Date: Fri, 22 May 2015 18:15:28 +0200
> 
> >> So far I've tracked the bug to xftfont_open, where instead of the
> >> usual
> >> 
> >>     font ascent: 14, descent: 5
> >> 
> >> I get for the 120755 char:
> >> 
> >>     font ascent: 54, descent: 46

According to what I see here 120755 (#x1d7b3) is the same kappa symbol
as #x1d779, just in italic form.  Not sure why Emacs on your system
displays the former when you request the latter, but at least it's the
same letter.  Perhaps the font driver substitutes one for the other?

> > How is this strange character displayed, eventually?  Is it displayed
> > as a character or as a box with its hex code?
> 
> It's displayed just the way gedit displays it, only with a huge (3 line
> height) ascent and a huge (3 line height) descent. So the line with the
> char is effectively 7 times taller than a regular line.
> 
> Also, I've tracked the font family of the char to be "Latin Modern
> Mathfont" in xftfont_open.

Math fonts are notorious for requesting huge ascent and descent
values.  I always disable them using fontsets.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Oleh Krehel]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Oleh Krehel <ohwoeowho@gmail.com>
>> Cc: clement.pitclaudel@live.com,  20628@debbugs.gnu.org
>> Date: Fri, 22 May 2015 18:15:28 +0200
>> 
>> >> So far I've tracked the bug to xftfont_open, where instead of the
>> >> usual
>> >> 
>> >>     font ascent: 14, descent: 5
>> >> 
>> >> I get for the 120755 char:
>> >> 
>> >>     font ascent: 54, descent: 46
>
> According to what I see here 120755 (#x1d7b3) is the same kappa symbol
> as #x1d779, just in italic form.  Not sure why Emacs on your system
> displays the former when you request the latter, but at least it's the
> same letter.  Perhaps the font driver substitutes one for the other?

Could be. I don't mind that really, just the ascent/descent.

>> > How is this strange character displayed, eventually?  Is it displayed
>> > as a character or as a box with its hex code?
>> 
>> It's displayed just the way gedit displays it, only with a huge (3 line
>> height) ascent and a huge (3 line height) descent. So the line with the
>> char is effectively 7 times taller than a regular line.
>> 
>> Also, I've tracked the font family of the char to be "Latin Modern
>> Mathfont" in xftfont_open.
>
> Math fonts are notorious for requesting huge ascent and descent
> values.  I always disable them using fontsets.

Can you explain what are fontsets and how to use the to disable
ascent/descent in math? Is it possible for Emacs to do so by default?

In any case, here's more data for the bad font that gets created with
XftFontOpenPattern.

Family:
    $ p (FcChar8 *) SDATA (SYMBOL_NAME (val))
    "Latin Modern Math"

Foundry:
    $ p (FcChar8 *) SDATA (SYMBOL_NAME (val))
    "unknown"

Filename:

    $ p (FcChar8 *) SDATA (filename)
    "/usr/share/texmf/fonts/opentype/public/lm-math/latinmodernmath-regular.otf"

So it might be that I could just patch latinmodernmath-regular.otf to
fix the problem. Or maybe Emacs could fix the problem specifically for
this file / font family. That's what gedit does, I guess.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Oleh Krehel]

[-- Attachment #1: Type: text/plain, Size: 226 bytes --]


I attach a patch that sets ascent/descent to 0 specifically for the
problematic font family "Latin Modern Math".

This way, the problem is solved for me for all 1152 chars with "math" in
their name that `ucs-names' returns.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Set-font-ascent-descent-to-zero-for-Latin-Modern-Mat.patch --]
[-- Type: text/x-diff, Size: 1219 bytes --]

From 802b84adc05a92650ee882ddd25e6860bacde457 Mon Sep 17 00:00:00 2001
From: Oleh Krehel <ohwoeowho@gmail.com>
Date: Fri, 22 May 2015 18:46:51 +0200
Subject: [PATCH] Set font ascent/descent to zero for "Latin Modern Math"

* src/xftfont.c (xftfont_open): Update.
---
 src/xftfont.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/src/xftfont.c b/src/xftfont.c
index 0e8b876..a09f7fc 100644
--- a/src/xftfont.c
+++ b/src/xftfont.c
@@ -289,6 +289,7 @@ xftfont_open (struct frame *f, Lisp_Object entity, int pixel_size)
   FcPatternAddInteger (pat, FC_WIDTH, FONT_WIDTH_NUMERIC (entity));
   FcPatternAddDouble (pat, FC_PIXEL_SIZE, pixel_size);
   val = AREF (entity, FONT_FAMILY_INDEX);
+  int mathp = strcmp("Latin Modern Math", SDATA(SYMBOL_NAME(val))) == 0;
   if (! NILP (val))
     FcPatternAddString (pat, FC_FAMILY, (FcChar8 *) SDATA (SYMBOL_NAME (val)));
   val = AREF (entity, FONT_FOUNDRY_INDEX);
@@ -332,6 +333,11 @@ xftfont_open (struct frame *f, Lisp_Object entity, int pixel_size)
       XftPatternDestroy (match);
       return Qnil;
     }
+  if (mathp)
+    {
+      xftfont->ascent = 0;
+      xftfont->descent = 0;
+    }
   ft_face = XftLockFace (xftfont);
   unblock_input ();
 
-- 
1.8.4

bug#20628: 25.0.50; Incorrect line height for some fonts

[Andreas Schwab]

Oleh Krehel <ohwoeowho@gmail.com> writes:

> I attach a patch that sets ascent/descent to 0 specifically for the
> problematic font family "Latin Modern Math".

Why only this family?

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#20628: 25.0.50; Incorrect line height for some fonts

[Oleh Krehel]

Andreas Schwab <schwab@linux-m68k.org> writes:

> Oleh Krehel <ohwoeowho@gmail.com> writes:
>
>> I attach a patch that sets ascent/descent to 0 specifically for the
>> problematic font family "Latin Modern Math".
>
> Why only this family?

That's the one that causes problems on my machine. I don't know any
others. Of course, the list can be extended.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> From: Oleh Krehel <ohwoeowho@gmail.com>
> Cc: clement.pitclaudel@live.com,  20628@debbugs.gnu.org
> Date: Fri, 22 May 2015 18:33:43 +0200
> 
> > Math fonts are notorious for requesting huge ascent and descent
> > values.  I always disable them using fontsets.
> 
> Can you explain what are fontsets

Type "i fontset RET" in the ELisp manual and "i fontsets RET" in the
Emacs User manual, and you will be able to read about that.  The
latter includes (in the section next to the one where the above
command lands you) several examples of how to set up your fontset to
display certain ranges of characters with specific fonts.

> and how to use the to disable ascent/descent in math?

You cannot disable that.  What you can do is find a font that covers
the same range of characters, but does not specify such huge ascend
and descent values.

> Is it possible for Emacs to do so by default?

You mean, have the default fontset set up to avoid the problem?  The
difficulty with that is that the fonts we'd need to put into the
default fontset are not free, and there's an understandable reluctance
to advertise them.

> maybe Emacs could fix the problem specifically for this file / font
> family. That's what gedit does, I guess.

It would be nice if someone could look at gedit sources and describe
what it does to avoid the problem.  Finding such a solution is what
this discussion is all about, no?

bug#20628: 25.0.50; Incorrect line height for some fonts

[Oleh Krehel]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Oleh Krehel <ohwoeowho@gmail.com>
>> Cc: clement.pitclaudel@live.com,  20628@debbugs.gnu.org
>> Date: Fri, 22 May 2015 18:54:17 +0200
>> 
>> I attach a patch that sets ascent/descent to 0 specifically for the
>> problematic font family "Latin Modern Math".
>
> That's hardly TRT.

Why not? "Latin Modern Math" has likely been unchanged in ages and isn't
getting changed soon; and we know that it causes a bug.

> But what I'd really like to do is find out how do other applications
> avoid this problem.  I thought that's what we wanted to do in this
> thread.

Would be great to know this. Unfortunately, I can't do much to help. I
barely found how Emacs sets the font, thanks to you. I don't know if
it's easy to do with gedit.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> From: Oleh Krehel <ohwoeowho@gmail.com>
> Cc: clement.pitclaudel@live.com,  20628@debbugs.gnu.org
> Date: Fri, 22 May 2015 18:54:17 +0200
> 
> I attach a patch that sets ascent/descent to 0 specifically for the
> problematic font family "Latin Modern Math".

That's hardly TRT.  Even if we'd want to "fix" some fonts, setting
ascent/descent to zero can only be justified if someone would examine
all of the characters of that font and verify that none of then needs
a non-zero value of ascent/descent, or tell us which (smaller) values
are actually needed.

But what I'd really like to do is find out how do other applications
avoid this problem.  I thought that's what we wanted to do in this
thread.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> From: Oleh Krehel <ohwoeowho@gmail.com>
> Cc: clement.pitclaudel@live.com,  20628@debbugs.gnu.org
> Date: Fri, 22 May 2015 20:18:17 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Oleh Krehel <ohwoeowho@gmail.com>
> >> Cc: clement.pitclaudel@live.com,  20628@debbugs.gnu.org
> >> Date: Fri, 22 May 2015 18:54:17 +0200
> >> 
> >> I attach a patch that sets ascent/descent to 0 specifically for the
> >> problematic font family "Latin Modern Math".
> >
> > That's hardly TRT.
> 
> Why not?

Because you assume no one will want to see that font in its original
metrics, and don't provide any way for users that might want that to
get that back.  There's no "fire escape".

> "Latin Modern Math" has likely been unchanged in ages and isn't
> getting changed soon; and we know that it causes a bug.

Isn't it much easier (and cleaner) to set up Emacs to not use that
font for these characters?  There are quite a few good fonts out there
that support these symbols without such huge ascent/descent values,
just use them instead.

> > But what I'd really like to do is find out how do other applications
> > avoid this problem.  I thought that's what we wanted to do in this
> > thread.
> 
> Would be great to know this. Unfortunately, I can't do much to help.

Well, I hope someone does.  Thanks for your efforts, regardless.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Oleh Krehel]

Eli Zaretskii <eliz@gnu.org> writes:

> Isn't it much easier (and cleaner) to set up Emacs to not use that
> font for these characters?  There are quite a few good fonts out there
> that support these symbols without such huge ascent/descent values,
> just use them instead.

It may be, but I've noticed this problem a month ago with no idea how to
fix it. And even now, I still have to research some more: find a good
font and write down the needed setting. Most users won't go to such
lengths to fix a visual bug, they'll just be silently annoyed and
unhappy.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Clément Pit--Claudel]

On 05/22/2015 02:15 PM, Eli Zaretskii wrote:
>> From: Oleh Krehel <ohwoeowho@gmail.com>
>> Cc: clement.pitclaudel@live.com,  20628@debbugs.gnu.org
>> Date: Fri, 22 May 2015 18:33:43 +0200
>>
>>> Math fonts are notorious for requesting huge ascent and descent
>>> values.  I always disable them using fontsets.
>>
> 
> It would be nice if someone could look at gedit sources and describe
> what it does to avoid the problem.  Finding such a solution is what
> this discussion is all about, no?

I don't know much about gedit's source code, nor how fonts are handled. However, I'm not sure if gedit can really give us much information: indeed, Emacs is the only application that behaves this way on my system. LibreOffice and Thunderbird, for example, display these characters fine.

Here is an hypothesis. When I open Latin Modern in FontForge, I see two types of ascent and descent values: the ones in the "General" tab are 806 and 194, and the ones in the OS/2 tab, in particular Win Ascent and Win Descent, are 3560 and 3060. Such a discrepancy does not seem to exist in the few well-behaved fonts that I checked.

Could it be that most applications use the first set of values, but Emacs relies on the second?

Clément.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Clément Pit--Claudel]

On 05/22/2015 02:15 PM, Eli Zaretskii wrote:
>> From: Oleh Krehel <ohwoeowho@gmail.com>
>> Cc: clement.pitclaudel@live.com,  20628@debbugs.gnu.org
>> Date: Fri, 22 May 2015 18:33:43 +0200
>>
>>> Math fonts are notorious for requesting huge ascent and descent
>>> values.  I always disable them using fontsets.
>>
>> Can you explain what are fontsets
> 
> Type "i fontset RET" in the ELisp manual and "i fontsets RET" in the
> Emacs User manual, and you will be able to read about that.  The
> latter includes (in the section next to the one where the above
> command lands you) several examples of how to set up your fontset to
> display certain ranges of characters with specific fonts.

I believe you have something like the following line in mind:

  (set-fontset-font fontset 'unicode (font-spec :name "Symbola") nil 'append)

Indeed, this fixes the problem. Unfortunately, this problem makes it hard for package developers to make use of prettify-symbols-mode. Indeed, programming languages like Agda or Gallina (Coq) would gain a lot from heavy prettification, but since the default fallback font tends to be one of these badly behaved TeX fonts, users of Adga and Coq packages will often run into this problem if we enable prettification by default at the package level the package level. IOW, it's currently hard to come up with a workaround that does not involve user intervention at the moment.

> (...)
>> Is it possible for Emacs to do so by default?
> 
> You mean, have the default fontset set up to avoid the problem?  The
> difficulty with that is that the fonts we'd need to put into the
> default fontset are not free, and there's an understandable reluctance
> to advertise them.

Symbola (in package ttf-ancient-fonts in Debian) actually seems to have good coverage for the kind of math symbols that triggers fallback to TeX fonts.

Clément.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Clément Pit--Claudel]

On 05/22/2015 02:53 PM, Oleh Krehel wrote:
> Eli Zaretskii <eliz@gnu.org> writes:
> 
>> Isn't it much easier (and cleaner) to set up Emacs to not use that
>> font for these characters?  There are quite a few good fonts out there
>> that support these symbols without such huge ascent/descent values,
>> just use them instead.
> 
> It may be, but I've noticed this problem a month ago with no idea how to
> fix it. And even now, I still have to research some more: find a good
> font and write down the needed setting. Most users won't go to such
> lengths to fix a visual bug, they'll just be silently annoyed and
> unhappy.

I agree with this statement. For the record, the correct invocation is

  (set-fontset-font fontset 'unicode (font-spec :name "Symbola") nil 'append)

and Symbola is in package ttf-ancient-fonts in Debian.

Clément.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Clément Pit--Claudel]

On 05/22/2015 02:43 PM, Eli Zaretskii wrote:
>> From: Oleh Krehel <ohwoeowho@gmail.com>
>> Cc: clement.pitclaudel@live.com,  20628@debbugs.gnu.org
>> Date: Fri, 22 May 2015 20:18:17 +0200
>>
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>>>> From: Oleh Krehel <ohwoeowho@gmail.com>
>>>> Cc: clement.pitclaudel@live.com,  20628@debbugs.gnu.org
>>>> Date: Fri, 22 May 2015 18:54:17 +0200
>>>>
>>>> I attach a patch that sets ascent/descent to 0 specifically for the
>>>> problematic font family "Latin Modern Math".
>>>
>>> That's hardly TRT.
>>
>> Why not?
> 
> Because you assume no one will want to see that font in its original
> metrics, and don't provide any way for users that might want that to
> get that back.  There's no "fire escape".

If we don't find what causes this strange ascent computation, and as a slightly more general approach that the patch above, what about including an ascent-override-alist at the Lisp level, which could by default contain ("Latin Modern Math" . 0) and maybe other entries for a number of other Latin Modern variants?

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> Date: Fri, 22 May 2015 14:57:01 -0400
> From: Clément Pit--Claudel
>  <clement.pitclaudel@live.com>
> CC: 20628@debbugs.gnu.org
> 
> Here is an hypothesis. When I open Latin Modern in FontForge, I see two types of ascent and descent values: the ones in the "General" tab are 806 and 194, and the ones in the OS/2 tab, in particular Win Ascent and Win Descent, are 3560 and 3060. Such a discrepancy does not seem to exist in the few well-behaved fonts that I checked.
> 
> Could it be that most applications use the first set of values, but Emacs relies on the second?

Maybe.  I know nothing about these internals of the fonts, sorry.  I
hope someone more knowledgeable could chime in and tell us what, if
anything, to do with this discrepancy.  If you can find such person(s)
on some relevant forum, please invite them to join this discussion and
help us fix this problem.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> From: Oleh Krehel <ohwoeowho@gmail.com>
> Cc: clement.pitclaudel@live.com,  20628@debbugs.gnu.org
> Date: Fri, 22 May 2015 20:53:05 +0200
> 
> I've noticed this problem a month ago with no idea how to
> fix it. And even now, I still have to research some more: find a good
> font and write down the needed setting. Most users won't go to such
> lengths to fix a visual bug, they'll just be silently annoyed and
> unhappy.

If we document the solution in the user manual, they could read it
there and act accordingly.  Or they can discover this discussion by
googling.

As for good fonts, try these: Styx (but not Styx Math!), Symbola, or
Quivira.  I think they all cover that block.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> Date: Fri, 22 May 2015 15:03:24 -0400
> From: Clément Pit--Claudel
>  <clement.pitclaudel@live.com>
> CC: 20628@debbugs.gnu.org
> 
> I believe you have something like the following line in mind:
> 
>   (set-fontset-font fontset 'unicode (font-spec :name "Symbola") nil 'append)

That's too radical.  You could be more selective, e.g.:

  (set-fontset-font "fontset-default"
                    '(#x1d400 . #x1d7ff)
		    "Symbola")

That's because you may wish using other fonts for other Unicode
blocks.

> Indeed, this fixes the problem. Unfortunately, this problem makes it hard for package developers to make use of prettify-symbols-mode. Indeed, programming languages like Agda or Gallina (Coq) would gain a lot from heavy prettification, but since the default fallback font tends to be one of these badly behaved TeX fonts, users of Adga and Coq packages will often run into this problem if we enable prettification by default at the package level the package level. IOW, it's currently hard to come up with a workaround that does not involve user intervention at the moment.

Couldn't those package developers recommend fontset settings, of even
include ready-to-use .emacs snippets as part of the package?

> > (...)
> >> Is it possible for Emacs to do so by default?
> > 
> > You mean, have the default fontset set up to avoid the problem?  The
> > difficulty with that is that the fonts we'd need to put into the
> > default fontset are not free, and there's an understandable reluctance
> > to advertise them.
> 
> Symbola (in package ttf-ancient-fonts in Debian) actually seems to have good coverage for the kind of math symbols that triggers fallback to TeX fonts.

Yes, and there are others (I mentioned them in my other message).

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> Date: Fri, 22 May 2015 15:09:11 -0400
> From: Clément Pit--Claudel
>  <clement.pitclaudel@live.com>
> CC: 20628@debbugs.gnu.org
> 
> If we don't find what causes this strange ascent computation, and as a slightly more general approach that the patch above, what about including an ascent-override-alist at the Lisp level, which could by default contain ("Latin Modern Math" . 0) and maybe other entries for a number of other Latin Modern variants?
> 

I'd rather tell users how to avoid these fonts entirely.  Making such
ad-hoc changes to font metrics sounds too much for my palate, unless
some expert tells us this is how everybody else does it.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Oleh Krehel]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Oleh Krehel <ohwoeowho@gmail.com>
>> Cc: clement.pitclaudel@live.com,  20628@debbugs.gnu.org
>> Date: Fri, 22 May 2015 18:54:17 +0200
>> 
>> I attach a patch that sets ascent/descent to 0 specifically for the
>> problematic font family "Latin Modern Math".
>
> That's hardly TRT.  Even if we'd want to "fix" some fonts, setting
> ascent/descent to zero can only be justified if someone would examine
> all of the characters of that font and verify that none of then needs
> a non-zero value of ascent/descent, or tell us which (smaller) values
> are actually needed.

Actually, I just found out that "Latin Modern Math" is inherently
completely broken. If I set it in gedit, even the ASCII chars have
ridiculous ascent/descent. So "Latin Modern Math" is basically unusable
unless a program specifically modifies ascent/descent, like I did.

The reason gedit was working fine previously is that it never used
"Latin Modern Math", it used some other font.

So it makes sense to "fix" a font that is common and inherently broken.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Clément Pit--Claudel]

On 05/22/2015 03:35 PM, Eli Zaretskii wrote:
>> Date: Fri, 22 May 2015 15:03:24 -0400
>> From: Clément Pit--Claudel
>>  <clement.pitclaudel@live.com>
>> CC: 20628@debbugs.gnu.org
>>
>> I believe you have something like the following line in mind:
>>
>>   (set-fontset-font fontset 'unicode (font-spec :name "Symbola") nil 'append)
> 
> That's too radical.  You could be more selective, e.g.:
> 
>   (set-fontset-font "fontset-default"
>                     '(#x1d400 . #x1d7ff)
> 		    "Symbola")
> 
> That's because you may wish using other fonts for other Unicode
> blocks.

Note that I added the 'append parameter at the end of that line, so if it is executed last (and if I understand correctly!) then it should not override any other preferences.

>> Indeed, this fixes the problem. Unfortunately, this problem makes it hard for package developers to make use of prettify-symbols-mode. Indeed, programming languages like Agda or Gallina (Coq) would gain a lot from heavy prettification, but since the default fallback font tends to be one of these badly behaved TeX fonts, users of Adga and Coq packages will often run into this problem if we enable prettification by default at the package level the package level. IOW, it's currently hard to come up with a workaround that does not involve user intervention at the moment.
> 
> Couldn't those package developers recommend fontset settings, of even
> include ready-to-use .emacs snippets as part of the package?

Indeed. In fact, that's what I already do with company-coq ( https://github.com/cpitclaudel/company-coq/#troubleshooting ). However, people are still reporting this as a bug.

I am not sure what I can do as a package developer to make it work "out of the box". An option to override the line height would be helpful, since I could do it at the package level; advanced users could then tweak things further (eg. by installing better fonts), but basic users would still benefit from decent defaults.

This is particularly relevant in the Coq case, since Emacs (as part of the Proof General package) is one of the main IDEs for Coq, and a number of users have no experience with Emacs when they start using it.

Clément.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Werner LEMBERG]

>> Here is an hypothesis. When I open Latin Modern in FontForge, I see
>> two types of ascent and descent values: the ones in the "General"
>> tab are 806 and 194, and the ones in the OS/2 tab, in particular
>> Win Ascent and Win Descent, are 3560 and 3060.  Such a discrepancy
>> does not seem to exist in the few well-behaved fonts that I
>> checked.  Could it be that most applications use the first set of
>> values, but Emacs relies on the second?

Actually, there are *three* sets of font-wide ascender and descender
values in TrueType fonts:

  (1) From the `hhea' table: The `ascent' and `descent' fields,
      together with `linegap'.  Used by Apple, cf.

        https://developer.apple.com/fonts/TrueType-Reference-Manual/RM06/Chap6hhea.html

      These values are normally set by the font developer; there is no
      relation to the actual ascender and descender values of
      individual glyphs.

  (2) From the `OS/2' table: The `usWinAscent' and `usWinDescent'
      fields, for Windows.  Originally, those values are the ymax and
      ymin values from all characters in the Windows ANSI character
      set.  Today, however, it is often set to the ymax and ymin
      values of all glyphs in a font to avoid nasty clipping on (some?
      older?)  Windows applications.

  (3) From the `OS/2' table: The `sTypoAscender' and `sTypoDescender'
      fields, together with `sTypoLinegap'.  For Windows.  These
      values are normally set by the font developer; there is no
      relation to the actual ascender and descender values of
      individual glyphs.

Note that (1) and (3) are defined differently.  Mac fonts often miss
an `OS/2' table, making (2) and (3) unavailable.  Additionally, many
fonts have incompatible or erroneous values for any of the fields.
It's really a mess, unfortunately.

IMHO the bes solution is to completely ignore font-wide ascender and
descender values.  Instead, use the TeX approach: set the line gap to
the current size of the font, multiplied by a factor of 1.2 (and make
this configurable on a font-by-font basis in case it isn't already),
and increase the linegap if individual glyphs need it.


    Werner

bug#20628: 25.0.50; Incorrect line height for some fonts

[Stefan Monnier]

> The reason gedit was working fine previously is that it never used
> "Latin Modern Math", it used some other font.

IOW Emacs doesn't seem to behave differently from other applications
w.r.t this font, except for the fact that it ends up selecting up while
other apps select another font instead.

So the question becomes: why does Emacs select this font and how could
we change ti so it selects something else.

BTW, I think that using something like

    (set-fontset-font fontset 'unicode (font-spec :name "Symbola") nil 'append)
or
    (set-fontset-font "fontset-default" '(#x1d400 . #x1d7ff) "Symbola")

just sucks: we don't want to say "use Symbola", but we instead want to
say something like "avoid Latin Modern Math" or "ignore Latin Modern
Math's ascent/descent information".


        Stefan

bug#20628: 25.0.50; Incorrect line height for some fonts

[Clément Pit--Claudel]

[-- Attachment #1: Type: text/plain, Size: 1643 bytes --]

On 05/22/2015 06:43 PM, Stefan Monnier wrote:
>> The reason gedit was working fine previously is that it never used
>> "Latin Modern Math", it used some other font.
> 
> IOW Emacs doesn't seem to behave differently from other applications
> w.r.t this font, except for the fact that it ends up selecting up while
> other apps select another font instead.

No, I don't think that's correct. On my machine, explicitly selecting Latin Modern Math on Emacs yields very tall lines. Selecting it in gedit or LibreOffice or Tomboy (the notes application) does not. See the attached screenshots.

> So the question becomes: why does Emacs select this font and how could
> we change ti so it selects something else.

If we don't have a way to detect misbehaving fonts dynamically, then I don't think that this works. Indeed, Latin Modern Math is not the only misbehaving font.

> BTW, I think that using something like
> 
>     (set-fontset-font fontset 'unicode (font-spec :name "Symbola") nil 'append)
> or
>     (set-fontset-font "fontset-default" '(#x1d400 . #x1d7ff) "Symbola")
> 
> just sucks: we don't want to say "use Symbola", but we instead want to
> say something like "avoid Latin Modern Math" or "ignore Latin Modern
> Math's ascent/descent information".

I don't think so; this forces us to maintain a list of misbehaving fonts. If we just say "Avoid Latin Modern Math" and the next selected font is also broken, then the problem remains (Asana Math, for example, is broken too, albeit a bit less). Ideally, we would also want to be able to use Latin Modern Math: ignoring the height issue, it's a nice font for maths symbols.

Clément.


[-- Attachment #2: Screenshot from 2015-05-22 19:54:35.png --]
[-- Type: image/png, Size: 173163 bytes --]

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> Date: Fri, 22 May 2015 23:49:59 +0200 (CEST)
> Cc: clement.pitclaudel@live.com, ohwoeowho@gmail.com, 20628@debbugs.gnu.org
> From: Werner LEMBERG <wl@gnu.org>
> 
> IMHO the bes solution is to completely ignore font-wide ascender and
> descender values.  Instead, use the TeX approach: set the line gap to
> the current size of the font, multiplied by a factor of 1.2 (and make
> this configurable on a font-by-font basis in case it isn't already),
> and increase the linegap if individual glyphs need it.

Could you perhaps look at the Emacs sources and suggest how to change
the *_open functions in the *font.c back-ends, to do what you suggest
above?  Or at least tell how to get "the current size of the font"
from the back-ends we use, which are Freetype, Fontconfig, and XLib's
XLoadQueryFont?  The relevant source files are xfont.c, ftfont.c, and
xftfont.c.

Also, how to know from the glyph metrics, in their Emacs incarnation,
that an individual glyph needs an increase of the vertical space?

Thanks.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> From: Oleh Krehel <ohwoeowho@gmail.com>
> Cc: clement.pitclaudel@live.com,  20628@debbugs.gnu.org
> Date: Fri, 22 May 2015 22:08:34 +0200
> 
> Actually, I just found out that "Latin Modern Math" is inherently
> completely broken. If I set it in gedit, even the ASCII chars have
> ridiculous ascent/descent. So "Latin Modern Math" is basically unusable
> unless a program specifically modifies ascent/descent, like I did.
> 
> The reason gedit was working fine previously is that it never used
> "Latin Modern Math", it used some other font.
> 
> So it makes sense to "fix" a font that is common and inherently broken.

IMO, it makes even more sense to avoid using it.  Patches to introduce
a list of fonts to ignore while searching for a suitable font, and
allow users to configure that list, will be welcome.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> Date: Fri, 22 May 2015 16:25:05 -0400
> From: Clément Pit--Claudel
>  <clement.pitclaudel@live.com>
> CC: ohwoeowho@gmail.com, 20628@debbugs.gnu.org
> 
> > Couldn't those package developers recommend fontset settings, of even
> > include ready-to-use .emacs snippets as part of the package?
> 
> Indeed. In fact, that's what I already do with company-coq ( https://github.com/cpitclaudel/company-coq/#troubleshooting ). However, people are still reporting this as a bug.
> 
> I am not sure what I can do as a package developer to make it work "out of the box". An option to override the line height would be helpful, since I could do it at the package level; advanced users could then tweak things further (eg. by installing better fonts), but basic users would still benefit from decent defaults.

How about setting the list of fonts to be ignored (assuming Emacs
acquires such a capability) -- would that be useful for packages?

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  clement.pitclaudel@live.com,  20628@debbugs.gnu.org
> Date: Fri, 22 May 2015 18:43:56 -0400
> 
> > The reason gedit was working fine previously is that it never used
> > "Latin Modern Math", it used some other font.
> 
> IOW Emacs doesn't seem to behave differently from other applications
> w.r.t this font, except for the fact that it ends up selecting up while
> other apps select another font instead.
> 
> So the question becomes: why does Emacs select this font and how could
> we change ti so it selects something else.

Emacs selects that font because it's available, and claims support for
the particular character Emacs needs to display.

The only mechanism we currently have for tailoring the fonts used for
specific ranges of characters is by defining the standard fontsets.
The ones we provide out of the box are set up on fontset.el, which
see.

> BTW, I think that using something like
> 
>     (set-fontset-font fontset 'unicode (font-spec :name "Symbola") nil 'append)
> or
>     (set-fontset-font "fontset-default" '(#x1d400 . #x1d7ff) "Symbola")
> 
> just sucks: we don't want to say "use Symbola", but we instead want to
> say something like "avoid Latin Modern Math" or "ignore Latin Modern
> Math's ascent/descent information".

We don't have such a feature, AFAIK.  At least not a documented one.

And I don't agree with the "just sucks" part: there are use cases when
the user might prefer a specific font for valid reason, e.g. the
quality of the glyphs.  If there are more than a few fonts on the
system that support the same range of characters, it is easier to
prefer one than to un-prefer the rest.

Also, there are fonts that claim support for a specific block, but in
fact support that block only partially, in some extreme cases just a
few characters.  Emacs's naive (due to efficiency considerations) way
of looking up suitable fonts might yield a negative result, where a
font is chosen that claims support, and then turns out not to have a
glyph for the specific character we need to display.  In this
situation, making a fontset that names a font with good coverage for
some block is the way to go.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> Date: Fri, 22 May 2015 19:55:24 -0400
> From: Clément Pit--Claudel
>  <clement.pitclaudel@live.com>
> CC: Eli Zaretskii <eliz@gnu.org>, 20628@debbugs.gnu.org
> 
> > BTW, I think that using something like
> > 
> >     (set-fontset-font fontset 'unicode (font-spec :name "Symbola") nil 'append)
> > or
> >     (set-fontset-font "fontset-default" '(#x1d400 . #x1d7ff) "Symbola")
> > 
> > just sucks: we don't want to say "use Symbola", but we instead want to
> > say something like "avoid Latin Modern Math" or "ignore Latin Modern
> > Math's ascent/descent information".
> 
> I don't think so; this forces us to maintain a list of misbehaving fonts. If we just say "Avoid Latin Modern Math" and the next selected font is also broken, then the problem remains (Asana Math, for example, is broken too, albeit a bit less). Ideally, we would also want to be able to use Latin Modern Math: ignoring the height issue, it's a nice font for maths symbols.

But the proposed patch, to which you agreed, did precisely that: it
singled out a particular font where the ascent/descent information was
to be ignored (and suggested to extend the list if needed).  How's
that different from what Stefan proposes?  And what's wrong with
maintaining a list of fonts that are known to misbehave?  There a lot
of broken fonts out there, and so far we relied on users configuring
the fonts on their machines to avoid negative effects, something
that's not always possible.

bug#20628: 25.0.50; Incorrect line height for some fonts

[YAMAMOTO Mitsuharu]

>>>>> On Sat, 23 May 2015 10:10:06 +0300, Eli Zaretskii <eliz@gnu.org> said:

> Patches to introduce a list of fonts to ignore while searching for a
> suitable font, and allow users to configure that list, will be
> welcome.

face-ignored-fonts ? 

  DEFVAR_LISP ("face-ignored-fonts", Vface_ignored_fonts,
	       doc: /* List of ignored fonts.
Each element is a regular expression that matches names of fonts to
ignore.  */);
  Vface_ignored_fonts = Qnil;

The regexps to specify are against font names in the XLFD format.  The
documentation is not so clear about this, probably because it was
written for older font code that predates the current font backend.

				     YAMAMOTO Mitsuharu
				mituharu@math.s.chiba-u.ac.jp

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> Date: Fri, 22 May 2015 16:25:05 -0400
> From: Clément Pit--Claudel
>  <clement.pitclaudel@live.com>
> CC: ohwoeowho@gmail.com, 20628@debbugs.gnu.org
> 
> On 05/22/2015 03:35 PM, Eli Zaretskii wrote:
> >> Date: Fri, 22 May 2015 15:03:24 -0400
> >> From: Clément Pit--Claudel
> >>  <clement.pitclaudel@live.com>
> >> CC: 20628@debbugs.gnu.org
> >>
> >> I believe you have something like the following line in mind:
> >>
> >>   (set-fontset-font fontset 'unicode (font-spec :name "Symbola") nil 'append)
> > 
> > That's too radical.  You could be more selective, e.g.:
> > 
> >   (set-fontset-font "fontset-default"
> >                     '(#x1d400 . #x1d7ff)
> > 		    "Symbola")
> > 
> > That's because you may wish using other fonts for other Unicode
> > blocks.
> 
> Note that I added the 'append parameter at the end of that line, so if it is executed last (and if I understand correctly!) then it should not override any other preferences.

When you use 'append' you can never know whether that would fix the
problem, because there could be other specs in the fontset that
override your addition.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> Date: Sat, 23 May 2015 16:47:56 +0900
> From: YAMAMOTO Mitsuharu <mituharu@math.s.chiba-u.ac.jp>
> Cc: Oleh Krehel <ohwoeowho@gmail.com>,
> 	clement.pitclaudel@live.com,
> 	20628@debbugs.gnu.org
> 
> >>>>> On Sat, 23 May 2015 10:10:06 +0300, Eli Zaretskii <eliz@gnu.org> said:
> 
> > Patches to introduce a list of fonts to ignore while searching for a
> > suitable font, and allow users to configure that list, will be
> > welcome.
> 
> face-ignored-fonts ? 

Yep, looks like it, thanks.  And completely undocumented in any
manual.

So, Oleh, does this variable help to avoid using the offending font?

bug#20628: 25.0.50; Incorrect line height for some fonts

[Werner LEMBERG]

>> IMHO the bes solution is to completely ignore font-wide ascender
>> and descender values.  Instead, use the TeX approach: set the line
>> gap to the current size of the font, multiplied by a factor of 1.2
>> (and make this configurable on a font-by-font basis in case it
>> isn't already), and increase the linegap if individual glyphs need
>> it.
> 
> Could you perhaps look at the Emacs sources and suggest how to
> change the *_open functions in the *font.c back-ends, to do what you
> suggest above?  Or at least tell how to get "the current size of the
> font" from the back-ends we use, which are Freetype, Fontconfig, and
> XLib's XLoadQueryFont?  The relevant source files are xfont.c,
> ftfont.c, and xftfont.c.

Sorry, no time.  However, with `current size' I mean the pixels per EM
value computed in the standard way:

  ppem = size * DPI / 72

where `size' is given in points and `DPI' the screen resolution.  This
should be completely independent of the back-end.  The idea is that
font designers have a vital interest that a font rendered at, say,
10pt (more or less) fits other fonts drawn at 10pt, regardless what
the font metrics say.  An exception to that are math fonts, of course,
since those need a real two-dimensional layout instead of positioning
glyphs in lines.

> Also, how to know from the glyph metrics, in their Emacs
> incarnation, that an individual glyph needs an increase of the
> vertical space?

Again no time to check this, sorry.  Assuming that Emacs somehow
provides the maximum descender of the glyphs in the previous line
together with a linegap value, simply check that the maximum ascender
of the glyphs in the current line doesn't collide, shifting the line
downwards if necessary.  AFAIK, Emacs does this already.

As a corollary, the only question is how to compute a proper default
linegap value without relying on quirks caused by incompatible font
formats and font metric data.


    Werner


PS: If you want it especially nifty, implement a skyline algorithm to
    check whether the ascender of the glyph at a given horizontal
    position collides with the descender of the glyph(s) at the same
    horizontal position one line above.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Oleh Krehel]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Sat, 23 May 2015 16:47:56 +0900
>> From: YAMAMOTO Mitsuharu <mituharu@math.s.chiba-u.ac.jp>
>> Cc: Oleh Krehel <ohwoeowho@gmail.com>,
>> 	clement.pitclaudel@live.com,
>> 	20628@debbugs.gnu.org
>> 
>> >>>>> On Sat, 23 May 2015 10:10:06 +0300, Eli Zaretskii <eliz@gnu.org> said:
>> 
>> > Patches to introduce a list of fonts to ignore while searching for a
>> > suitable font, and allow users to configure that list, will be
>> > welcome.
>> 
>> face-ignored-fonts ? 
>
> Yep, looks like it, thanks.  And completely undocumented in any
> manual.
>
> So, Oleh, does this variable help to avoid using the offending font?

Yes! It works great. How strange that we haven't seen this before? Only
363 variables matching "font", all I had to do was add "ignore" to the query:)

bug#20628: 25.0.50; Incorrect line height for some fonts

[Werner LEMBERG]

> IMHO the bes solution is to completely ignore font-wide ascender and
> descender values.  Instead, use the TeX approach: set the line gap
> to the current size of the font, multiplied by a factor of 1.2 (and
> make this configurable on a font-by-font basis in case it isn't
> already), and increase the linegap if individual glyphs need it.

I have to correct myself: The linegap should apply to all fonts, of
course.  To make various fonts optically fit a per-font scaling factor
should be used (what Emacs already provides, I think).

> As a corollary, the only question is how to compute a proper default
> linegap value without relying on quirks caused by incompatible font
> formats and font metric data.

And for this default linegap value I suggest the abovementioned value:

  linegap_default = 1.2 * font_size


    Werner

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> From: Oleh Krehel <ohwoeowho@gmail.com>
> Cc: YAMAMOTO Mitsuharu <mituharu@math.s.chiba-u.ac.jp>,  clement.pitclaudel@live.com,  20628@debbugs.gnu.org
> Date: Sat, 23 May 2015 11:56:38 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> face-ignored-fonts ? 
> >
> > Yep, looks like it, thanks.  And completely undocumented in any
> > manual.
> >
> > So, Oleh, does this variable help to avoid using the offending font?
> 
> Yes! It works great. How strange that we haven't seen this before? Only
> 363 variables matching "font", all I had to do was add "ignore" to the query:)

For completeness, would you please post the customization you used?
If nothing else, we should have that in the manual.

Thanks.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> Date: Sat, 23 May 2015 11:50:22 +0200 (CEST)
> Cc: clement.pitclaudel@live.com, ohwoeowho@gmail.com, 20628@debbugs.gnu.org
> From: Werner LEMBERG <wl@gnu.org>
> 
> 
> >> IMHO the bes solution is to completely ignore font-wide ascender
> >> and descender values.  Instead, use the TeX approach: set the line
> >> gap to the current size of the font, multiplied by a factor of 1.2
> >> (and make this configurable on a font-by-font basis in case it
> >> isn't already), and increase the linegap if individual glyphs need
> >> it.
> > 
> > Could you perhaps look at the Emacs sources and suggest how to
> > change the *_open functions in the *font.c back-ends, to do what you
> > suggest above?  Or at least tell how to get "the current size of the
> > font" from the back-ends we use, which are Freetype, Fontconfig, and
> > XLib's XLoadQueryFont?  The relevant source files are xfont.c,
> > ftfont.c, and xftfont.c.
> 
> Sorry, no time.  However, with `current size' I mean the pixels per EM
> value computed in the standard way:
> 
>   ppem = size * DPI / 72
> 
> where `size' is given in points and `DPI' the screen resolution.

I guess you mean font->pixel_size, something we have already.

> > Also, how to know from the glyph metrics, in their Emacs
> > incarnation, that an individual glyph needs an increase of the
> > vertical space?
> 
> Again no time to check this, sorry.  Assuming that Emacs somehow
> provides the maximum descender of the glyphs in the previous line
> together with a linegap value, simply check that the maximum ascender
> of the glyphs in the current line doesn't collide, shifting the line
> downwards if necessary.  AFAIK, Emacs does this already.

Emacs indeed does that already, but it uses the font's ascent and
descent values, not values specific to each glyph.  I was under the
impression that you said we could access and use the ascent/descent
values of each glyph in a font, and I was asking how to do that,
i.e. which metrics express these per-glyph values.

> As a corollary, the only question is how to compute a proper default
> linegap value without relying on quirks caused by incompatible font
> formats and font metric data.

What do you mean by "linegap"?  the vertical gap between two screen
lines in Emacs is the sum of the line height, computed as a sum of its
max_ascent and max_descent values, plus the value of line-spacing.

> PS: If you want it especially nifty, implement a skyline algorithm to
>     check whether the ascender of the glyph at a given horizontal
>     position collides with the descender of the glyph(s) at the same
>     horizontal position one line above.

This again goes back to the question how to access the ascender of a
glyph.

Thanks.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Oleh Krehel]

Eli Zaretskii <eliz@gnu.org> writes:

>> > So, Oleh, does this variable help to avoid using the offending font?
>> 
>> Yes! It works great. How strange that we haven't seen this before? Only
>> 363 variables matching "font", all I had to do was add "ignore" to the query:)
>
> For completeness, would you please post the customization you used?
> If nothing else, we should have that in the manual.

(setq face-ignored-fonts '("Latin Modern Math"))

Thanks for the help in investigating this. I still think it would be
nice to fix Latin Modern Math for the hypothetical poor user that either
has no other font or doesn't know about `face-ignored-fonts'. Maybe:

(setq face-fonts-descent-accent-alist '("Latin Modern Math" . (0 . 0)))

But anyway, my problem is solved, so I'm not that inclined to fix it
further.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Werner LEMBERG]

>> However, with `current size' I mean the pixels per EM value
>> computed in the standard way:
>>
>>   ppem = size * DPI / 72
>>
>> where `size' is given in points and `DPI' the screen resolution.
>
> I guess you mean font->pixel_size, something we have already.

Sounds right.

>> Assuming that Emacs somehow provides the maximum descender of the
>> glyphs in the previous line together with a linegap value, simply
>> check that the maximum ascender of the glyphs in the current line
>> doesn't collide, shifting the line downwards if necessary.  AFAIK,
>> Emacs does this already.
>
> Emacs indeed does that already, but it uses the font's ascent and
> descent values, not values specific to each glyph.

I suggest to change that, not relying on the font's ascent and descent
value, but deriving this value from the font size instead.

> I was under the impression that you said we could access and use the
> ascent/descent values of each glyph in a font, and I was asking how
> to do that, i.e. which metrics express these per-glyph values.

Well, a quick search for FreeType functions brings me to function
`ftfont_text_extents', which fills `font_metrics' structures for
individual glyphs.  So I guess the answer are the fields `ascent' and
`descent' of Emacs's `font_metrics' structure.

>> As a corollary, the only question is how to compute a proper
>> default linegap value without relying on quirks caused by
>> incompatible font formats and font metric data.
>
> What do you mean by "linegap"?  the vertical gap between two screen
> lines in Emacs is the sum of the line height, computed as a sum of
> its max_ascent and max_descent values, plus the value of
> line-spacing.

Yes, I've meant the baseline-to-baseline distance.  Sorry for the
confusion.


    Werner

bug#20628: 25.0.50; Incorrect line height for some fonts

[Stefan Monnier]

>> So the question becomes: why does Emacs select this font and how could
>> we change ti so it selects something else.
> Emacs selects that font because it's available, and claims support for
> the particular character Emacs needs to display.

That doesn't explain why it selects that font instead of some other font
(say Symbola).

> And I don't agree with the "just sucks" part: there are use cases when
> the user might prefer a specific font for valid reason, e.g. the
> quality of the glyphs.

Oh, I'm not saying that specifying the font you want sucks.  I'm just
saying that having to specify a good font sucks when what you want to do
is to specify which font is bad instead (and you may not even know
which font is good, and/or that font may not always be the same).


        Stefan

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: ohwoeowho@gmail.com,  clement.pitclaudel@live.com,  20628@debbugs.gnu.org
> Date: Sat, 23 May 2015 09:27:52 -0400
> 
> >> So the question becomes: why does Emacs select this font and how could
> >> we change ti so it selects something else.
> > Emacs selects that font because it's available, and claims support for
> > the particular character Emacs needs to display.
> 
> That doesn't explain why it selects that font instead of some other font
> (say Symbola).

Most probably because it came up first in the search.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Clément Pit--Claudel]

On 05/23/2015 03:12 AM, Eli Zaretskii wrote:
>> Date: Fri, 22 May 2015 16:25:05 -0400
>> From: Clément Pit--Claudel
>>  <clement.pitclaudel@live.com>
>> CC: ohwoeowho@gmail.com, 20628@debbugs.gnu.org
>>
>>> Couldn't those package developers recommend fontset settings, of even
>>> include ready-to-use .emacs snippets as part of the package?
>>
>> Indeed. In fact, that's what I already do with company-coq ( https://github.com/cpitclaudel/company-coq/#troubleshooting ). However, people are still reporting this as a bug.
>>
>> I am not sure what I can do as a package developer to make it work "out of the box". An option to override the line height would be helpful, since I could do it at the package level; advanced users could then tweak things further (eg. by installing better fonts), but basic users would still benefit from decent defaults.
> 
> How about setting the list of fonts to be ignored (assuming Emacs
> acquires such a capability) -- would that be useful for packages?

Maybe. The problem is that if a good font is not installed yet, ignoring Latin Modern Math means replacing huge lines with square boxes. I'm not sure which one is best. Ignoring Latin Modern Math may be better because fixing the square boxes then only requires installing a new font (while fixing the line heights requires installing a new font /and/ setting up fontsets in Emacs). But ignoring Latin Modern Math is also more confusing (the symbols will display fine in user's web browsers, but not in Emacs). The unearthing of `face-ignored-fonts' likely makes this an easy change; do you have an opinion either way?

bug#20628: 25.0.50; Incorrect line height for some fonts

[Clément Pit--Claudel]

On 05/22/2015 05:49 PM, Werner LEMBERG wrote:
> 
>>> Here is an hypothesis. When I open Latin Modern in FontForge, I see
>>> two types of ascent and descent values: the ones in the "General"
>>> tab are 806 and 194, and the ones in the OS/2 tab, in particular
>>> Win Ascent and Win Descent, are 3560 and 3060.  Such a discrepancy
>>> does not seem to exist in the few well-behaved fonts that I
>>> checked.  Could it be that most applications use the first set of
>>> values, but Emacs relies on the second?
> 
> Actually, there are *three* sets of font-wide ascender and descender
> values in TrueType fonts:
> 
>   (1) From the `hhea' table: The `ascent' and `descent' fields,
>       together with `linegap'.  Used by Apple, cf.
> 
>         https://developer.apple.com/fonts/TrueType-Reference-Manual/RM06/Chap6hhea.html
> 
>       These values are normally set by the font developer; there is no
>       relation to the actual ascender and descender values of
>       individual glyphs.
> 
>   (2) From the `OS/2' table: The `usWinAscent' and `usWinDescent'
>       fields, for Windows.  Originally, those values are the ymax and
>       ymin values from all characters in the Windows ANSI character
>       set.  Today, however, it is often set to the ymax and ymin
>       values of all glyphs in a font to avoid nasty clipping on (some?
>       older?)  Windows applications.
> 
>   (3) From the `OS/2' table: The `sTypoAscender' and `sTypoDescender'
>       fields, together with `sTypoLinegap'.  For Windows.  These
>       values are normally set by the font developer; there is no
>       relation to the actual ascender and descender values of
>       individual glyphs.
> 
> Note that (1) and (3) are defined differently.  Mac fonts often miss
> an `OS/2' table, making (2) and (3) unavailable.  Additionally, many
> fonts have incompatible or erroneous values for any of the fields.
> It's really a mess, unfortunately.
> 
> IMHO the bes solution is to completely ignore font-wide ascender and
> descender values.  Instead, use the TeX approach: set the line gap to
> the current size of the font, multiplied by a factor of 1.2 (and make
> this configurable on a font-by-font basis in case it isn't already),
> and increase the linegap if individual glyphs need it.

Thanks for this detailed description! 

I looked a bit more into how other programs handle this, and it seems that they take the simple approach of relying on font-wide metrics. Only, they don't seem to use the same ones as Emacs. This causes LibreOffice and vim-gtk to display "∫" as slightly truncated when using Latin Modern Math (it's taller than the line height). Emacs on the other hand displays it fine (at the cost of having a huge line height.

I wonder if changing the set of metrics used in Emacs would be easy.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Clément Pit--Claudel]

On 05/23/2015 06:56 AM, Oleh Krehel wrote:
> Thanks for the help in investigating this. I still think it would be
> nice to fix Latin Modern Math for the hypothetical poor user that either
> has no other font or doesn't know about `face-ignored-fonts'. Maybe:
> 
> (setq face-fonts-descent-accent-alist '("Latin Modern Math" . (0 . 0)))

I agree with this (Latin Modern Math is commonly available, and not many other fonts cover math symbols like "𝓟").

bug#20628: 25.0.50; Incorrect line height for some fonts

[Clément Pit--Claudel]

On 05/23/2015 03:24 AM, Eli Zaretskii wrote:
>> Date: Fri, 22 May 2015 19:55:24 -0400 From: Clément Pit--Claudel 
>> <clement.pitclaudel@live.com> CC: Eli Zaretskii <eliz@gnu.org>,
>> 20628@debbugs.gnu.org
>> 
>>> BTW, I think that using something like
>>> 
>>> (set-fontset-font fontset 'unicode (font-spec :name "Symbola")
>>> nil 'append) or (set-fontset-font "fontset-default" '(#x1d400 .
>>> #x1d7ff) "Symbola")
>>> 
>>> just sucks: we don't want to say "use Symbola", but we instead
>>> want to say something like "avoid Latin Modern Math" or "ignore
>>> Latin Modern Math's ascent/descent information".
>> 
>> I don't think so; this forces us to maintain a list of misbehaving 
>> fonts. If we just say "Avoid Latin Modern Math" and the next 
>> selected font is also broken, then the problem remains (Asana
>> Math, for example, is broken too, albeit a bit less). Ideally, we
>> would also want to be able to use Latin Modern Math: ignoring the
>> height issue, it's a nice font for maths symbols.
> 
> But the proposed patch, to which you agreed, did precisely that: it 
> singled out a particular font where the ascent/descent information
> was to be ignored (and suggested to extend the list if needed).

I meant to disagree with the part regarding ignoring a font entirely. Maintaining a list of broken fonts in a way that still allows us to use them (for example by overriding line height information) is not the most robust solution (ideally, we'd like to support these fonts just like virtually every other program out there), but it's still an improvement over the current situation. 

On the other hand, adding Latin Modern Math to a list of ignored fonts means that:
* Emacs suddenly seems to not support one of the fonts that users can use elsewhere without problems (e.g. in LibreOffice). 
* If Latin Modern Math is the only font that covers a particular character, we've changed "incorrect line height" into "square boxes in place of symbols", which is hardly an improvement over the current behaviour.
* If users install a fixed version of Latin Modern Math, is still won't be used unless they also remove Latin Modern from the ignore list.

> How's that different from what Stefan proposes?  And what's wrong
> with maintaining a list of fonts that are known to misbehave?  There
> a lot of broken fonts out there, and so far we relied on users
> configuring the fonts on their machines to avoid negative effects,
> something that's not always possible.
> 
> 

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> Date: Sun, 24 May 2015 04:20:32 -0400
> From: Clément Pit--Claudel
>  <clement.pitclaudel@live.com>
> CC: ohwoeowho@gmail.com, 20628@debbugs.gnu.org
> 
> I looked a bit more into how other programs handle this, and it seems that they take the simple approach of relying on font-wide metrics. Only, they don't seem to use the same ones as Emacs. This causes LibreOffice and vim-gtk to display "∫" as slightly truncated when using Latin Modern Math (it's taller than the line height). Emacs on the other hand displays it fine (at the cost of having a huge line height.
> 
> I wonder if changing the set of metrics used in Emacs would be easy.

Changing how?  Which metrics to use instead?  We need details to
discuss our options.

And why is it TRT to make a change that clearly results in incorrect
display of some of the glyphs in that font?  It seems we have just
found the answer to the question how do those other programs "avoid"
the problem: they do it at a price of producing incorrect display of
tall glyphs.  I don't think Emacs should go that way.

A better alternative would be to detect such tall glyphs when they are
needed for display, and enlarge the line height only then.  But I
don't yet see the data of these glyphs that could tell us to do that,
any help will be appreciated.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> Date: Sun, 24 May 2015 04:20:22 -0400
> From: Clément Pit--Claudel
>  <clement.pitclaudel@live.com>
> CC: ohwoeowho@gmail.com, 20628@debbugs.gnu.org
> 
> > How about setting the list of fonts to be ignored (assuming Emacs
> > acquires such a capability) -- would that be useful for packages?
> 
> Maybe. The problem is that if a good font is not installed yet, ignoring Latin Modern Math means replacing huge lines with square boxes. I'm not sure which one is best. Ignoring Latin Modern Math may be better because fixing the square boxes then only requires installing a new font (while fixing the line heights requires installing a new font /and/ setting up fontsets in Emacs). But ignoring Latin Modern Math is also more confusing (the symbols will display fine in user's web browsers, but not in Emacs). The unearthing of `face-ignored-fonts' likely makes this an easy change; do you have an opinion either way?

I think it is better top ignore fonts that produce display which is
unacceptable to users.  You can include in the installation
instructions advice on which fonts to install.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Werner LEMBERG]

> A better alternative would be to detect such tall glyphs when they are
> needed for display, and enlarge the line height only then.

Yep.

> But I don't yet see the data of these glyphs that could tell us to
> do that, any help will be appreciated.

Have you seen my message w.r.t. the `ascent' and `descent' fields in
the `font_metrics' structure of Emacs?  Does this help?


    Werner

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> Date: Sun, 24 May 2015 04:20:39 -0400
> From: Clément Pit--Claudel
>  <clement.pitclaudel@live.com>
> CC: 20628@debbugs.gnu.org
> 
> I agree with this (Latin Modern Math is commonly available, and not many other fonts cover math symbols like "𝓟").

Actually, quite a few do.  I mentioned them in this thread.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> Date: Sun, 24 May 2015 04:20:50 -0400
> From: Clément Pit--Claudel
>  <clement.pitclaudel@live.com>
> CC: monnier@iro.umontreal.ca, ohwoeowho@gmail.com, 20628@debbugs.gnu.org
> 
> On the other hand, adding Latin Modern Math to a list of ignored fonts means that:
> * Emacs suddenly seems to not support one of the fonts that users can use elsewhere without problems (e.g. in LibreOffice). 

As you yourself described, using that font is not "without problems":
some glyphs come out truncated.

> * If Latin Modern Math is the only font that covers a particular character, we've changed "incorrect line height" into "square boxes in place of symbols", which is hardly an improvement over the current behaviour.

It's an improvement if using Latin Modern Math produces results that
are unacceptable.  This bug report says loud and clear that the
results are indeed unacceptable.

> * If users install a fixed version of Latin Modern Math, is still won't be used unless they also remove Latin Modern from the ignore list.

Does such a fixed version exist?  I doubt that.  I see several fonts
whose names match ".* Math" which all have this problem.  So it sounds
like this was done on purpose, and the font designers will not fix
that.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> Date: Sun, 24 May 2015 11:32:03 +0200 (CEST)
> Cc: clement.pitclaudel@live.com, ohwoeowho@gmail.com, 20628@debbugs.gnu.org
> From: Werner LEMBERG <wl@gnu.org>
> 
> 
> > A better alternative would be to detect such tall glyphs when they are
> > needed for display, and enlarge the line height only then.
> 
> Yep.
> 
> > But I don't yet see the data of these glyphs that could tell us to
> > do that, any help will be appreciated.
> 
> Have you seen my message w.r.t. the `ascent' and `descent' fields in
> the `font_metrics' structure of Emacs?  Does this help?

Yes, I've seen it; and yes, it helps.  I need to try the solution
based on that.

Thanks.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> Date: Sun, 24 May 2015 12:48:46 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: clement.pitclaudel@live.com, ohwoeowho@gmail.com, 20628@debbugs.gnu.org
> 
> > Have you seen my message w.r.t. the `ascent' and `descent' fields in
> > the `font_metrics' structure of Emacs?  Does this help?
> 
> Yes, I've seen it; and yes, it helps.  I need to try the solution
> based on that.

Actually, I need help with this, as I don't know enough about fonts,
and don't have access to systems where per-character ascent/descent
values are currently available in Emacs.

So I'm going to describe here the information that should allow those
who have access to affected systems to propose a patch.

The relevant place in the display engine where these factors are taken
into consideration is around line 26388 in xdisp.c, which is part of
the function x_produce_glyphs.  There you will find a call to the
function get_per_char_metric, which in turn calls the font driver's
text_extents method.  The metrics returned by get_per_char_metric
include ascent and descent, but I think they are not in pixel units.
We currently assign these values to the phys_ascent and phys_descent
of 'struct it', the iterator object used to walk the visible portion
of the buffer and produce glyphs for display.  I'm not sure what we do
with phys_ascent and phys_descent values once we compute them, but you
can search for them in xdisp.c to get the idea.  By contrast, the
height of the screen line is computed by summing it->ascent and
it->descent, which currently are set using the font's ascent and
descent values.

The change we look for should:

  . set the initial values for it->ascent and it->descent using some
    heuristics based on pixel_size of the default face's font;

  . update it->ascent and it->descent based on ascent/descent values
    returned by get_per_char_metric in the per-character metrics data

If people who want to work on this have questions about the display
engine, please don't hesitate to ask.

Thanks in advance!

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> Date: Sun, 24 May 2015 13:06:57 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 20628@debbugs.gnu.org
> 
> The relevant place in the display engine where these factors are taken
> into consideration is around line 26388 in xdisp.c, which is part of
> the function x_produce_glyphs.  There you will find a call to the
> function get_per_char_metric, which in turn calls the font driver's
> text_extents method.  The metrics returned by get_per_char_metric
> include ascent and descent, but I think they are not in pixel units.
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
This part seems to be incorrect, as I see elsewhere in the code that
phys_ascent and ascent are freely added and subtracted, and the same
for phys_descent and descent.  IOW, the per-character ascent/descent
values are returned in pixel units.

Sorry for any confusion I caused.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Rasmus]

YAMAMOTO Mitsuharu <mituharu@math.s.chiba-u.ac.jp> writes:

>>>>>> On Sat, 23 May 2015 10:10:06 +0300, Eli Zaretskii <eliz@gnu.org> said:
>
>> Patches to introduce a list of fonts to ignore while searching for a
>> suitable font, and allow users to configure that list, will be
>> welcome.
>
> face-ignored-fonts ? 
>
>   DEFVAR_LISP ("face-ignored-fonts", Vface_ignored_fonts,
> 	       doc: /* List of ignored fonts.
> Each element is a regular expression that matches names of fonts to
> ignore.  */);
>   Vface_ignored_fonts = Qnil;
>
> The regexps to specify are against font names in the XLFD format.  The
> documentation is not so clear about this, probably because it was
> written for older font code that predates the current font backend.

Thanks for pointing this variable out!  It is insanely helpful and would
have solved my old woes:

     http://thread.gmane.org/gmane.emacs.help/93832

[Note to self: apropos!]

It should definitely be documented prominently in the documentation
preferably close to the fontset variable!

Rasmus

-- 
ツ

bug#20628: 25.0.50; Incorrect line height for some fonts

[Drew Adams]

> >   DEFVAR_LISP ("face-ignored-fonts", Vface_ignored_fonts,
> > 	       doc: /* List of ignored fonts.
> > Each element is a regular expression that matches names of fonts
> > to ignore.  */);

Why does it have `face' in the name, let alone as the prefix?

The value doesn't refer to fonts to be ignored for any specific
face, right?  It might be easier to discover, as well as easier
to understand, if it started with `font' and didn't include `face':
`fonts-to-ignore'.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> Date: Mon, 25 May 2015 06:24:40 -0700 (PDT)
> From: Drew Adams <drew.adams@oracle.com>
> 
> > >   DEFVAR_LISP ("face-ignored-fonts", Vface_ignored_fonts,
> > > 	       doc: /* List of ignored fonts.
> > > Each element is a regular expression that matches names of fonts
> > > to ignore.  */);
> 
> Why does it have `face' in the name, let alone as the prefix?

Historical reasons.

> The value doesn't refer to fonts to be ignored for any specific
> face, right?

Emacs cannot use any font, unless it has a face that specifies that
font.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Drew Adams]

> Emacs cannot use any font, unless it has a face that specifies
> that font.

Irrelevant to the question of `face-' prefixing the name.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> Date: Sun, 24 May 2015 13:06:57 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 20628@debbugs.gnu.org
> 
> Actually, I need help with this, as I don't know enough about fonts,
> and don't have access to systems where per-character ascent/descent
> values are currently available in Emacs.

Since no one volunteered, I've pushed an experimental branch
scratch/large-fonts, which attempts to deal with the problem.

After looking into the relevant code and trying a few things, I
concluded that I don't want to stop using the font-global height
information for fonts that provide reasonable values.  That's because
the current code works well enough in those cases, and it would be a
shame to throw it out the window.  Also, it turns out that the
assumptions about using font-global values are all over the place, and
in some of them the per-glyph information is not even available.

So instead I added a heuristic test intended to detect the problematic
fonts, and code to fall back on the per-glyph data when such fonts are
detected.  I also fixed a few issues exposed by this change, mainly
with the hollow cursor.

The changes are in platform-independent code, but I could only test
them on MS-Windows.  So users of Unix and GNU systems who are affected
by this problem are encouraged to try the branch and report their
experiences, whether good or bad.

I will look into merging the branch to master in a few days, barring
any grave bugs.

TIA

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> Date: Wed, 27 May 2015 18:20:02 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 20628@debbugs.gnu.org
> 
> The changes are in platform-independent code, but I could only test
> them on MS-Windows.  So users of Unix and GNU systems who are affected
> by this problem are encouraged to try the branch and report their
> experiences, whether good or bad.
> 
> I will look into merging the branch to master in a few days, barring
> any grave bugs.
> 
> TIA

A bug reported that generated 70 messages describing how important it
was to fix it, and yet no one -- not a single soul -- says anything
when the solution seems to be at hand?  Isn't that strange?

Would people who urged us to fix this please try the
scratch/large-fonts branch, and see if it's good enough to be merged
to master?  I can post the diffs here, if that would make things
easier for someone.

Once again, the Unix-specific portions of the changes are untested,
and need at least to be verified to do a reasonable job with the
offending fonts.

If there are no comments in a few days, I will proceed with the merge.

TIA

bug#20628: 25.0.50; Incorrect line height for some fonts

[Clément Pit--Claudel]

On 05/29/2015 01:20 AM, Eli Zaretskii wrote:
>> Date: Wed, 27 May 2015 18:20:02 +0300
>> From: Eli Zaretskii <eliz@gnu.org>
>> Cc: 20628@debbugs.gnu.org
>>
>> The changes are in platform-independent code, but I could only test
>> them on MS-Windows.  So users of Unix and GNU systems who are affected
>> by this problem are encouraged to try the branch and report their
>> experiences, whether good or bad.
>>
>> I will look into merging the branch to master in a few days, barring
>> any grave bugs.
>>
>> TIA
> 
> A bug reported that generated 70 messages describing how important it
> was to fix it, and yet no one -- not a single soul -- says anything
> when the solution seems to be at hand?  Isn't that strange?

Hi Eli,

Thanks a lot for taking the time to put this patch together! I've been pretty busy these last few days, and just got around testing your patch (on Linux Mint). The approach of only activating per-glyph metrics when the font provides absurd ascent and descent values seems reasonable. 

> Would people who urged us to fix this please try the
> scratch/large-fonts branch, and see if it's good enough to be merged
> to master?  I can post the diffs here, if that would make things
> easier for someone.

The patch partially solves the problem for me, but I noticed a few problems after running (set-frame-font "Latin Modern Math")
* When the cursor is at the end of the file, on an empty line, it has a very height.
* The modeline is still very tall
* The fix seems to only apply to certain characters. The line that I mentioned in my original email, in particular, is still very tall. In other words, when trying the following three lines in a Latin Modern Math buffer, the last line is much too tall:

𝓝  ;; This is still very tall
∏∑∫ ;; This is a bit taller than the normal height, which is great
ABC ;; This has the normal height, which is also great

> 
> Once again, the Unix-specific portions of the changes are untested,
> and need at least to be verified to do a reasonable job with the
> offending fonts.
> 
> If there are no comments in a few days, I will proceed with the merge.
> 
> TIA
> 
> 

bug#20628: 25.0.50; Incorrect line height for some fonts

[Oleh Krehel]

Clément Pit--Claudel <clement.pitclaudel@live.com> writes:

> The patch partially solves the problem for me, but I noticed a few
> problems after running (set-frame-font "Latin Modern Math")
> * When the cursor is at the end of the file, on an empty line, it has a very height.
> * The modeline is still very tall
> * The fix seems to only apply to certain characters. The line that I
> mentioned in my original email, in particular, is still very tall. In
> other words, when trying the following three lines in a Latin Modern
> Math buffer, the last line is much too tall:
>
> 𝓝  ;; This is still very tall
> ∏∑∫ ;; This is a bit taller than the normal height, which is great
> ABC ;; This has the normal height, which is also great

Same issues for me on Ubuntu. The most outstanding one is the mode line
height (of 7 lines instead of 1). And the cursor at eof is very tall as
well.

Additionally, these chars are bad: \\|, λ.
And all lines in Buffer-menu-mode are super-tall, just like the mode
line. And the header line is tall.

Oleh

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> From: Oleh Krehel <ohwoeowho@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  wl@gnu.org,  20628@debbugs.gnu.org
> Date: Fri, 29 May 2015 11:30:55 +0200
> 
> Clément Pit--Claudel <clement.pitclaudel@live.com> writes:
> 
> > The patch partially solves the problem for me, but I noticed a few
> > problems after running (set-frame-font "Latin Modern Math")
> > * When the cursor is at the end of the file, on an empty line, it has a very height.
> > * The modeline is still very tall
> > * The fix seems to only apply to certain characters. The line that I
> > mentioned in my original email, in particular, is still very tall. In
> > other words, when trying the following three lines in a Latin Modern
> > Math buffer, the last line is much too tall:
> >
> > 𝓝  ;; This is still very tall
> > ∏∑∫ ;; This is a bit taller than the normal height, which is great
> > ABC ;; This has the normal height, which is also great
> 
> Same issues for me on Ubuntu. The most outstanding one is the mode line
> height (of 7 lines instead of 1). And the cursor at eof is very tall as
> well.

I will need snapshots of all you see, because I cannot even begin
debugging without seeing the exact pictures.

Mode line is not handled in the patches I made, so it's little wonder
you see what you see.  (Does someone actually use Latin Modern Math as
the default font? it's an ugly font, AFAICS, and I expect it to be
used only for math stuff that the main font doesn't cover.)

> Additionally, these chars are bad: \\|, λ.

Any special character you see look bad, please make sure with "C-u C-x ="
with which font it's displayed.

> And all lines in Buffer-menu-mode are super-tall, just like the mode
> line.

Recipe, please.  Starting from "emacs -Q", as usual.

> And the header line is tall.

That's the same code as for mode line, so it's expected.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Oleh Krehel]

Eli Zaretskii <eliz@gnu.org> writes:

> I will need snapshots of all you see, because I cannot even begin
> debugging without seeing the exact pictures.
>
> Mode line is not handled in the patches I made, so it's little wonder
> you see what you see.  (Does someone actually use Latin Modern Math as
> the default font? it's an ugly font, AFAICS, and I expect it to be
> used only for math stuff that the main font doesn't cover.)

Of course I don't use it. I've added it to `face-ignored-fonts'. But
since you want to make a better fix, I did
(set-frame-font "Latin Modern Math") to check everything.

>> Additionally, these chars are bad: \\|, λ.
>
> Any special character you see look bad, please make sure with "C-u C-x ="
> with which font it's displayed.

OK, they are: 0x2228 and 0x03BB.

>
>> And all lines in Buffer-menu-mode are super-tall, just like the mode
>> line.
>
> Recipe, please.  Starting from "emacs -Q", as usual.

Eval (set-frame-font "Latin Modern Math"), and "C-x C-b".

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> From: Oleh Krehel <ohwoeowho@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  wl@gnu.org,  20628@debbugs.gnu.org
> Date: Fri, 29 May 2015 11:30:55 +0200
> 
> Clément Pit--Claudel <clement.pitclaudel@live.com> writes:
> 
> > The patch partially solves the problem for me, but I noticed a few
> > problems after running (set-frame-font "Latin Modern Math")
> > * When the cursor is at the end of the file, on an empty line, it has a very height.
                                                                            ^^^^^^^^^^^

"Very WHAT height"? "very small" or "very large"?

> > * The fix seems to only apply to certain characters. The line that I
> > mentioned in my original email, in particular, is still very tall. In
> > other words, when trying the following three lines in a Latin Modern
> > Math buffer, the last line is much too tall:
> >
> > 𝓝  ;; This is still very tall

I don't see this on my system.  Here, it makes the line slightly
higher, and that's all.  Perhaps you have a different version of the
font installed.  But in any case, if that character still shows up as
too large (a screenshot would be nice), and Emacs uses the Latin
Modern Math font to display it (make sure with "C-u C-x ="), then
there's nothing that can be done about that, since it means the
metrics of the glyph itself, as reported by the font/font driver, are
screwed.  If you want to make sure that's the reason, put a breakpoint
on line 26425 of xdisp.c, after the call to get_per_char_metric, as
shown below:

	  if (get_char_glyph_code (it->char_to_display, font, &char2b))
	    {
	      pcm = get_per_char_metric (font, &char2b);
	      if (pcm->width == 0    <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
		  && pcm->rbearing == 0 && pcm->lbearing == 0)
		pcm = NULL;
	    }

and see what ascent and descent values are reported for that character
in pcm->ascent and pcm->descent.

> Additionally, these chars are bad: \\|, λ.

What do you mean by "bad"?  Here, they just make the line slightly
higher (by a few pixels), again due to the metrics of the glyphs, but
nowhere near the original height before my changes.  Isn't that what
you see?

> And all lines in Buffer-menu-mode are super-tall, just like the mode
> line.

That's because Buffer-menu-mode uses 'space' display property, which
is another place where we use font dimensions.  I will see if that can
be fixed.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Oleh Krehel]

Eli Zaretskii <eliz@gnu.org> writes:

>> Additionally, these chars are bad: ∨, λ.
>
> What do you mean by "bad"?  Here, they just make the line slightly
> higher (by a few pixels), again due to the metrics of the glyphs, but
> nowhere near the original height before my changes.  Isn't that what
> you see?

I see 7x line height caused by any of these chars. Which was exactly the
initial issue for all chars. Now most of them are fine, but these two
are not. Actually now, I notice that they are fine in this buffer. This
issue happens if I use them for `prettify-symbols-mode' stuff, like:

(font-lock-add-keywords
 'emacs-lisp-mode
 '(("\\\\\\\\|"
    (0 font-lock-keyword-face t)
    (0
     (prog1
         (compose-region
          (match-beginning 0)
          (match-end 0)
          "∨")
       nil)))))

bug#20628: 25.0.50; Incorrect line height for some fonts

[Stefan Monnier]

>> Any special character you see look bad, please make sure with "C-u C-x ="
>> with which font it's displayed.
> OK, they are: 0x2228 and 0x03BB.

That's only the char.  C-u C-x = should give you a crapload more info,
including the actual font being used.


        Stefan

bug#20628: 25.0.50; Incorrect line height for some fonts

[Oleh Krehel]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> Any special character you see look bad, please make sure with "C-u C-x ="
>>> with which font it's displayed.
>> OK, they are: 0x2228 and 0x03BB.
>
> That's only the char.  C-u C-x = should give you a crapload more info,
> including the actual font being used.

Here's the rest of the info for "lambda" being composed into "lambda":

             position: 14095 of 224875 (6%), column: 4
            character: l (displayed as l) (codepoint 108, #o154, #x6c)
    preferred charset: ascii (ASCII (ISO646 IRV))
code point in charset: 0x6C
               script: latin
               syntax: w 	which means: word
             category: .:Base, L:Left-to-right (strong), a:ASCII, l:Latin, r:Roman
             to input: type "C-x 8 RET HEX-CODEPOINT" or "C-x 8 RET NAME"
          buffer code: #x6C
            file code: #x6C (encoded by coding system utf-8-unix)
              display: composed to form "lambda" (see below)

Composed with the following character(s) "ambda" by the rule:
	(?λ)
The component character(s) are displayed by these fonts (glyph codes):
 λ: xft:-unknown-Latin Modern Math-normal-normal-normal-*-15-*-*-*-*-0-iso10646-1 (#xF6F)
See the variable `reference-point-alist' for the meaning of the rule.

Character code properties: customize what to show
  name: LATIN SMALL LETTER L
  general-category: Ll (Letter, Lowercase)
  decomposition: (108) ('l')

There are text properties here:
  composition          [Show]
  face                 font-lock-keyword-face
  fontified            nil

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> From: Oleh Krehel <ohwoeowho@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  clement.pitclaudel@live.com,  20628@debbugs.gnu.org
> Date: Fri, 29 May 2015 16:49:41 +0200
> 
> Here's the rest of the info for "lambda" being composed into "lambda":
> 
>              position: 14095 of 224875 (6%), column: 4
>             character: l (displayed as l) (codepoint 108, #o154, #x6c)
>     preferred charset: ascii (ASCII (ISO646 IRV))
> code point in charset: 0x6C
>                script: latin
>                syntax: w 	which means: word
>              category: .:Base, L:Left-to-right (strong), a:ASCII, l:Latin, r:Roman
>              to input: type "C-x 8 RET HEX-CODEPOINT" or "C-x 8 RET NAME"
>           buffer code: #x6C
>             file code: #x6C (encoded by coding system utf-8-unix)
>               display: composed to form "lambda" (see below)
> 
> Composed with the following character(s) "ambda" by the rule:
> 	(?λ)
> The component character(s) are displayed by these fonts (glyph codes):
>  λ: xft:-unknown-Latin Modern Math-normal-normal-normal-*-15-*-*-*-*-0-iso10646-1 (#xF6F)
> See the variable `reference-point-alist' for the meaning of the rule.

The key part is "Composed with" etc.  My changes didn't touch the
composed characters, so it's small wonder you see this.

I will try to have a look, once I get the other issues covered, but
don't hold your breath: I don't really understand all the complex
juggling we do with ascent and descent for composed characters, and
unless I find an easy and safe solution, I'd prefer to tell people not
to set these fonts as default fonts than to break character
composition, which is much more central to Emacs display capabilities.

Thanks.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> From: Oleh Krehel <ohwoeowho@gmail.com>
> Cc: clement.pitclaudel@live.com,  wl@gnu.org,  20628@debbugs.gnu.org
> Date: Fri, 29 May 2015 15:16:44 +0200
> 
> Actually now, I notice that they are fine in this buffer. This
> issue happens if I use them for `prettify-symbols-mode' stuff, like:
> 
> (font-lock-add-keywords
>  'emacs-lisp-mode
>  '(("\\\\\\\\|"
>     (0 font-lock-keyword-face t)
>     (0
>      (prog1
>          (compose-region
>           (match-beginning 0)
>           (match-end 0)
>           "∨")
>        nil)))))

That explains the difference in what we see: compose-region causes the
display engine to take a different path through the code.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Clément Pit--Claudel]

On 05/29/2015 06:15 AM, Eli Zaretskii wrote:
>> From: Oleh Krehel <ohwoeowho@gmail.com>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  wl@gnu.org,  20628@debbugs.gnu.org
>> Date: Fri, 29 May 2015 11:30:55 +0200
>>
>> Clément Pit--Claudel <clement.pitclaudel@live.com> writes:
>>
>>> The patch partially solves the problem for me, but I noticed a few
>>> problems after running (set-frame-font "Latin Modern Math")
>>> * When the cursor is at the end of the file, on an empty line, it has a very height.
>                                                                             ^^^^^^^^^^^
> 
> "Very WHAT height"? "very small" or "very large"?

Woops, sorry. Very large height.

>>> * The fix seems to only apply to certain characters. The line that I
>>> mentioned in my original email, in particular, is still very tall. In
>>> other words, when trying the following three lines in a Latin Modern
>>> Math buffer, the last line is much too tall:
>>>
>>> 𝓝  ;; This is still very tall
> 
> I don't see this on my system.  Here, it makes the line slightly
> higher, and that's all. Perhaps you have a different version of the
> font installed.  But in any case, if that character still shows up as
> too large (a screenshot would be nice), and Emacs uses the Latin
> Modern Math font to display it (make sure with "C-u C-x ="), then
> there's nothing that can be done about that, since it means the
> metrics of the glyph itself, as reported by the font/font driver, are
> screwed.  If you want to make sure that's the reason, put a breakpoint
> on line 26425 of xdisp.c, after the call to get_per_char_metric, as
> shown below:
> 
> 	  if (get_char_glyph_code (it->char_to_display, font, &char2b))
> 	    {
> 	      pcm = get_per_char_metric (font, &char2b);
> 	      if (pcm->width == 0    <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
> 		  && pcm->rbearing == 0 && pcm->lbearing == 0)
> 		pcm = NULL;
> 	    }
> 
> and see what ascent and descent values are reported for that character
> in pcm->ascent and pcm->descent.

Thanks for the detailed explanation! You were right, the height is fine if the character itself is inserted. The problem is due to prettify-symbols-mode.

Thanks for your work on this, it's very cool! This is a bug that had annoyed me 4 or 5 years ago when I first used Emacs, but at that time I didn't really know how to report bugs.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> From: Oleh Krehel <ohwoeowho@gmail.com>
> Cc: clement.pitclaudel@live.com,  wl@gnu.org,  20628@debbugs.gnu.org
> Date: Fri, 29 May 2015 15:16:44 +0200
> 
> I see 7x line height caused by any of these chars. Which was exactly the
> initial issue for all chars. Now most of them are fine, but these two
> are not. Actually now, I notice that they are fine in this buffer. This
> issue happens if I use them for `prettify-symbols-mode' stuff, like:
> 
> (font-lock-add-keywords
>  'emacs-lisp-mode
>  '(("\\\\\\\\|"
>     (0 font-lock-keyword-face t)
>     (0
>      (prog1
>          (compose-region
>           (match-beginning 0)
>           (match-end 0)
>           "∨")
>        nil)))))

I pushed a few more fixes to the branch, please try it.  The problem
in Buffer-menu-mode should be fixed now.

I don't see the problem with prettify-symbols-mode, though.  Is it
enough to "M-x prettify-symbols-mode RET" in *scratch*, and then type
"(lambda ", to reproduce the bad display?  If not, what else is
needed?  Can you show a screenshot?

I also don't see the problem with mode line and header line.  If I
evaluate '(set-frame-font "Latin Modern Math")', I get a very tall
frame (expected), but the mode line has reasonable dimensions, and if
I enter Info, the header line is also OK.  Is there any step I'm
missing?  Do you still see the problem with the current branch head?

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> Date: Fri, 29 May 2015 22:21:29 -0700
> From: Clément Pit--Claudel
>  <clement.pitclaudel@live.com>
> CC: wl@gnu.org, 20628@debbugs.gnu.org
> 
> On 05/29/2015 06:15 AM, Eli Zaretskii wrote:
> >> From: Oleh Krehel <ohwoeowho@gmail.com>
> >> Cc: Eli Zaretskii <eliz@gnu.org>,  wl@gnu.org,  20628@debbugs.gnu.org
> >> Date: Fri, 29 May 2015 11:30:55 +0200
> >>
> >> Clément Pit--Claudel <clement.pitclaudel@live.com> writes:
> >>
> >>> The patch partially solves the problem for me, but I noticed a few
> >>> problems after running (set-frame-font "Latin Modern Math")
> >>> * When the cursor is at the end of the file, on an empty line, it has a very height.
> >                                                                             ^^^^^^^^^^^
> > 
> > "Very WHAT height"? "very small" or "very large"?
> 
> Woops, sorry. Very large height.

Do you still see the problem with the current branch head?

> Thanks for the detailed explanation! You were right, the height is fine if the character itself is inserted. The problem is due to prettify-symbols-mode.

Do you still see the problem with the current branch head?  If I just
turn on prettify-symbols-mode (the one included in the Emacs
repository) in *scratch*, I see no problems with the current branch
head.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Oleh Krehel]

Eli Zaretskii <eliz@gnu.org> writes:

> Do you still see the problem with the current branch head?  If I just
> turn on prettify-symbols-mode (the one included in the Emacs
> repository) in *scratch*, I see no problems with the current branch
> head.

I still see the problem with the current branch head. Here are my settings:

(font-lock-add-keywords
 'emacs-lisp-mode
 `((,"\\\\\\\\|"
    (0 font-lock-keyword-face t)
    (0
     (prog1
         (compose-region
          (match-beginning 0)
          (match-end 0)
          ,"∨")
       nil)))))

Also, while the mode line became normal height, the echo area became x7
height instead. The buffer menu is fine.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> From: Oleh Krehel <ohwoeowho@gmail.com>
> Cc: Clément Pit--Claudel <clement.pitclaudel@live.com>,
>   20628@debbugs.gnu.org
> Date: Sat, 30 May 2015 15:00:52 +0200
> 
> I still see the problem with the current branch head. Here are my settings:
> 
> (font-lock-add-keywords
>  'emacs-lisp-mode
>  `((,"\\\\\\\\|"
>     (0 font-lock-keyword-face t)
>     (0
>      (prog1
>          (compose-region
>           (match-beginning 0)
>           (match-end 0)
>           ,"∨")
>        nil)))))

Sorry, I don't understand: what do you do _after_ evaluating the
above, to show the problem?

IOW, I'm guessing that your recipe is this:

  emacs -Q
  M-: (set-frame-font "Latin Modern Math") RET
  "M-:" to eval the above form to set up font-lock-keyword
  and then ...?

If the above is correct, then what is the last part required to
produce some problematic display?

> the echo area became x7 height instead.

This is expected, if you use set-frame-font: Emacs reserves for each
window the space in pixels that is derived from the frame font's size.
This happens very early in redisplay cycle, where Emacs cannot yet
override these dimensions.  Fixing that would not be simple; I don't
see a reason for trying, as people should not have any plausible
reasons to set frame's font to one of the offending fonts discussed in
this thread.

But if you set only the font of the buffer (e.g., by clicking
S-mouse-1 in the echo area, when the minibuffer is active), then the
echo area height remains almost the same, whereas it wasn't before
these changes.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Oleh Krehel]

Eli Zaretskii <eliz@gnu.org> writes:

>> I still see the problem with the current branch head. Here are my settings:
>> 
>> (font-lock-add-keywords
>>  'emacs-lisp-mode
>>  `((,"\\\\\\\\|"
>>     (0 font-lock-keyword-face t)
>>     (0
>>      (prog1
>>          (compose-region
>>           (match-beginning 0)
>>           (match-end 0)
>>           ,"∨")
>>        nil)))))
>
> Sorry, I don't understand: what do you do _after_ evaluating the
> above, to show the problem?
>
> IOW, I'm guessing that your recipe is this:
>
>   emacs -Q
>   M-: (set-frame-font "Latin Modern Math") RET
>   "M-:" to eval the above form to set up font-lock-keyword
>   and then ...?

Insert this string into an `emacs-lisp-mode' buffer:

    "^\\|\\s-\\|\\[\\|[(`'#@~_%,]"

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> From: Oleh Krehel <ohwoeowho@gmail.com>
> Cc: clement.pitclaudel@live.com,  20628@debbugs.gnu.org
> Date: Sat, 30 May 2015 16:32:26 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> I still see the problem with the current branch head. Here are my settings:
> >> 
> >> (font-lock-add-keywords
> >>  'emacs-lisp-mode
> >>  `((,"\\\\\\\\|"
> >>     (0 font-lock-keyword-face t)
> >>     (0
> >>      (prog1
> >>          (compose-region
> >>           (match-beginning 0)
> >>           (match-end 0)
> >>           ,"∨")
> >>        nil)))))
> >
> > Sorry, I don't understand: what do you do _after_ evaluating the
> > above, to show the problem?
> >
> > IOW, I'm guessing that your recipe is this:
> >
> >   emacs -Q
> >   M-: (set-frame-font "Latin Modern Math") RET
> >   "M-:" to eval the above form to set up font-lock-keyword
> >   and then ...?
> 
> Insert this string into an `emacs-lisp-mode' buffer:
> 
>     "^\\|\\s-\\|\\[\\|[(`'#@~_%,]"

Thanks.  I don't really see what you say, but I saw something similar,
and fix that.  Could you see if your problem is fixed with the latest
branch tip?

bug#20628: 25.0.50; Incorrect line height for some fonts

[Oleh Krehel]

Eli Zaretskii <eliz@gnu.org> writes:

> Thanks.  I don't really see what you say, but I saw something similar,
> and fix that.  Could you see if your problem is fixed with the latest
> branch tip?

Thanks, I just tried. It looked fine, but then suddenly I got a segfault:

Fatal error 11: Segmentation fault
Backtrace:
newemacs[0x4fed82]
newemacs[0x4e67d9]
newemacs[0x4fdbae]
newemacs[0x4fddb3]
newemacs[0x4fde2f]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10340)[0x7ff73ea44340]
newemacs[0x44e61b]
newemacs[0x452186]
newemacs[0x45e37c]
newemacs[0x46232e]
newemacs[0x554eb3]
newemacs[0x4502d8]
newemacs[0x4f03db]
newemacs[0x4f2cd3]
newemacs[0x4f4891]
newemacs[0x554d8b]
newemacs[0x4e6d0c]
newemacs[0x554c93]
newemacs[0x4e6cc7]
newemacs[0x4eb248]
newemacs[0x4eb565]
newemacs[0x415a48]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7ff73e690ec5]
newemacs[0x416556]
Segmentation fault (core dumped)

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> From: Oleh Krehel <ohwoeowho@gmail.com>
> Cc: clement.pitclaudel@live.com,  20628@debbugs.gnu.org
> Date: Sat, 30 May 2015 18:59:25 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Thanks.  I don't really see what you say, but I saw something similar,
> > and fix that.  Could you see if your problem is fixed with the latest
> > branch tip?
> 
> Thanks, I just tried. It looked fine, but then suddenly I got a segfault:

Can you convert this to file names and line numbers, or reproduce the
crash under GDB and show the backtrace?  Without that, I cannot do
much with this information.

Thanks.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Oleh Krehel]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Oleh Krehel <ohwoeowho@gmail.com>
>> Cc: clement.pitclaudel@live.com,  20628@debbugs.gnu.org
>> Date: Sat, 30 May 2015 18:59:25 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > Thanks.  I don't really see what you say, but I saw something similar,
>> > and fix that.  Could you see if your problem is fixed with the latest
>> > branch tip?
>> 
>> Thanks, I just tried. It looked fine, but then suddenly I got a segfault:
>
> Can you convert this to file names and line numbers, or reproduce the
> crash under GDB and show the backtrace?  Without that, I cannot do
> much with this information.

I get dropped into GDB at line 3135 in xdisp.c. Here's bt:

#0  init_from_display_pos (it=it@entry=0x7fffffff7bc0, w=w@entry=0x11d5eb0, pos=pos@entry=0x163c580) at xdisp.c:3135
#1  0x0000000000452186 in init_to_row_start (row=0x163c530, w=0x11d5eb0, it=0x7fffffff7bc0) at xdisp.c:3205
#2  try_window_reusing_current_matrix (w=w@entry=0x11d5eb0) at xdisp.c:17245
#3  0x000000000045e37c in redisplay_window (window=18702005, just_this_one_p=just_this_one_p@entry=true) at xdisp.c:16577
#4  0x000000000046232e in redisplay_window_1 (window=window@entry=18702005) at xdisp.c:14203
#5  0x0000000000554eb3 in internal_condition_case_1 (bfun=bfun@entry=0x462300 <redisplay_window_1>, arg=18702005, handlers=<optimized out>, hfun=hfun@entry=0x429930 <redisplay_window_error>) at eval.c:1372
#6  0x00000000004502d8 in redisplay_internal () at xdisp.c:13846
#7  0x0000000000451775 in redisplay () at xdisp.c:13030
#8  0x00000000004f03db in read_char (commandflag=commandflag@entry=1, map=map@entry=48583475, prev_event=0, used_mouse_menu=used_mouse_menu@entry=0x7fffffffcf6b, end_time=end_time@entry=0x0) at keyboard.c:2542
#9  0x00000000004f2cd3 in read_key_sequence (keybuf=keybuf@entry=0x7fffffffd040, prompt=prompt@entry=0, dont_downcase_last=dont_downcase_last@entry=false, can_return_switch_frame=can_return_switch_frame@entry=true, fix_current_buffer=fix_current_buffer@entry=true, prevent_redisplay=prevent_redisplay@entry=false, bufsize=30) at keyboard.c:9156
#10 0x00000000004f4891 in command_loop_1 () at keyboard.c:1407
#11 0x0000000000554d8b in internal_condition_case (bfun=bfun@entry=0x4f4660 <command_loop_1>, handlers=handlers@entry=19872, hfun=hfun@entry=0x4eb610 <cmd_error>) at eval.c:1348
#12 0x00000000004e6d0c in command_loop_2 (ignore=ignore@entry=0) at keyboard.c:1139
#13 0x0000000000554c93 in internal_catch (tag=tag@entry=47328, func=func@entry=0x4e6cf0 <command_loop_2>, arg=arg@entry=0) at eval.c:1108
#14 0x00000000004e6cc7 in command_loop () at keyboard.c:1118
#15 0x00000000004eb248 in recursive_edit_1 () at keyboard.c:728
#16 0x00000000004eb565 in Frecursive_edit () at keyboard.c:799
#17 0x0000000000415a48 in main (argc=1, argv=0x7fffffffd3b8) at emacs.c:1626

bug#20628: 25.0.50; Incorrect line height for some fonts

[Clément Pit--Claudel]

On 05/30/2015 02:42 AM, Eli Zaretskii wrote:
>> Date: Fri, 29 May 2015 22:21:29 -0700
>> From: Clément Pit--Claudel
>>  <clement.pitclaudel@live.com>
>> CC: wl@gnu.org, 20628@debbugs.gnu.org
>>
>> On 05/29/2015 06:15 AM, Eli Zaretskii wrote:
>>>> From: Oleh Krehel <ohwoeowho@gmail.com>
>>>> Cc: Eli Zaretskii <eliz@gnu.org>,  wl@gnu.org,  20628@debbugs.gnu.org
>>>> Date: Fri, 29 May 2015 11:30:55 +0200
>>>>
>>>> Clément Pit--Claudel <clement.pitclaudel@live.com> writes:
>>>>
>>>>> The patch partially solves the problem for me, but I noticed a few
>>>>> problems after running (set-frame-font "Latin Modern Math")
>>>>> * When the cursor is at the end of the file, on an empty line, it has a very height.
>>>                                                                             ^^^^^^^^^^^
>>>
>>> "Very WHAT height"? "very small" or "very large"?
>>
>> Woops, sorry. Very large height.
> 
> Do you still see the problem with the current branch head?
> 
>> Thanks for the detailed explanation! You were right, the height is fine if the character itself is inserted. The problem is due to prettify-symbols-mode.
> 
> Do you still see the problem with the current branch head?  If I just
> turn on prettify-symbols-mode (the one included in the Emacs
> repository) in *scratch*, I see no problems with the current branch
> head.

This works beautifully for me! It's very nice to see this long-standing issue come to a resolution :)

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> From: Oleh Krehel <ohwoeowho@gmail.com>
> Cc: clement.pitclaudel@live.com,  20628@debbugs.gnu.org
> Date: Sat, 30 May 2015 20:57:59 +0200
> 
> >> Thanks, I just tried. It looked fine, but then suddenly I got a segfault:
> >
> > Can you convert this to file names and line numbers, or reproduce the
> > crash under GDB and show the backtrace?  Without that, I cannot do
> > much with this information.
> 
> I get dropped into GDB at line 3135 in xdisp.c. Here's bt:
> 
> #0  init_from_display_pos (it=it@entry=0x7fffffff7bc0, w=w@entry=0x11d5eb0, pos=pos@entry=0x163c580) at xdisp.c:3135

Thanks.  I think I know what the problem is.  If you still have this
in GDB, could you show the output of the following commands:

 (gdb) frame 2
 (gdb) p first_row_to_display->y
 (gdb) p yb
 (gdb) p window_text_bottom_y (w)

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> Date: Sat, 30 May 2015 11:58:29 -0700
> From: Clément Pit--Claudel
>  <clement.pitclaudel@live.com>
> CC: ohwoeowho@gmail.com, wl@gnu.org, 20628@debbugs.gnu.org
> 
> This works beautifully for me! It's very nice to see this long-standing issue come to a resolution :)

Thanks for testing.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Oleh Krehel]

Eli Zaretskii <eliz@gnu.org> writes:

> Thanks.  I think I know what the problem is.  If you still have this
> in GDB, could you show the output of the following commands:
>
>  (gdb) frame 2
>  (gdb) p first_row_to_display->y
>  (gdb) p yb
>  (gdb) p window_text_bottom_y (w)

I closed it, but here it is from the new one:

bt
#0  init_from_display_pos (it=it@entry=0x7fffffff7bc0, w=w@entry=0x11d5eb0, pos=pos@entry=0x1b71180) at xdisp.c:3135
#1  0x0000000000452186 in init_to_row_start (row=0x1b71130, w=0x11d5eb0, it=0x7fffffff7bc0) at xdisp.c:3205
#2  try_window_reusing_current_matrix (w=w@entry=0x11d5eb0) at xdisp.c:17245
#3  0x000000000045e37c in redisplay_window (window=18702005, just_this_one_p=just_this_one_p@entry=true) at xdisp.c:16577
#4  0x000000000046232e in redisplay_window_1 (window=window@entry=18702005) at xdisp.c:14203
#5  0x0000000000554eb3 in internal_condition_case_1 (bfun=bfun@entry=0x462300 <redisplay_window_1>, arg=18702005, handlers=<optimized out>, hfun=hfun@entry=0x429930 <redisplay_window_error>) at eval.c:1372
#6  0x00000000004502d8 in redisplay_internal () at xdisp.c:13846
#7  0x0000000000451775 in redisplay () at xdisp.c:13030
#8  0x00000000004f03db in read_char (commandflag=commandflag@entry=1, map=map@entry=30649395, prev_event=0, used_mouse_menu=used_mouse_menu@entry=0x7fffffffcf6b, end_time=end_time@entry=0x0) at keyboard.c:2542
#9  0x00000000004f2cd3 in read_key_sequence (keybuf=keybuf@entry=0x7fffffffd040, prompt=prompt@entry=0, dont_downcase_last=dont_downcase_last@entry=false, can_return_switch_frame=can_return_switch_frame@entry=true, fix_current_buffer=fix_current_buffer@entry=true, prevent_redisplay=prevent_redisplay@entry=false, bufsize=30) at keyboard.c:9156
#10 0x00000000004f4891 in command_loop_1 () at keyboard.c:1407
#11 0x0000000000554d8b in internal_condition_case (bfun=bfun@entry=0x4f4660 <command_loop_1>, handlers=handlers@entry=19872, hfun=hfun@entry=0x4eb610 <cmd_error>) at eval.c:1348
#12 0x00000000004e6d0c in command_loop_2 (ignore=ignore@entry=0) at keyboard.c:1139
#13 0x0000000000554c93 in internal_catch (tag=tag@entry=47328, func=func@entry=0x4e6cf0 <command_loop_2>, arg=arg@entry=0) at eval.c:1108
#14 0x00000000004e6cc7 in command_loop () at keyboard.c:1118
#15 0x00000000004eb248 in recursive_edit_1 () at keyboard.c:728
#16 0x00000000004eb565 in Frecursive_edit () at keyboard.c:799
#17 0x0000000000415a48 in main (argc=1, argv=0x7fffffffd3b8) at emacs.c:1626

frame 2
#2  try_window_reusing_current_matrix (w=w@entry=0x11d5eb0) at xdisp.c:17245
17245	      init_to_row_start (&it, w, first_row_to_display);

p first_row_to_display->y
$1 = 909

p yb
$2 = <optimized out>

p window_text_bottom_y (w)
$3 = 909

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> From: Oleh Krehel <ohwoeowho@gmail.com>
> Cc: clement.pitclaudel@live.com,  20628@debbugs.gnu.org
> Date: Sat, 30 May 2015 21:26:03 +0200
> 
> frame 2
> #2  try_window_reusing_current_matrix (w=w@entry=0x11d5eb0) at xdisp.c:17245
> 17245	      init_to_row_start (&it, w, first_row_to_display);
> 
> p first_row_to_display->y
> $1 = 909
> 
> p yb
> $2 = <optimized out>
> 
> p window_text_bottom_y (w)
> $3 = 909

Thanks, this confirms my suspicions: we are writing beyond the
window's glyph matrix.  I will see what can be done about that.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> Date: Sat, 30 May 2015 22:52:13 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: clement.pitclaudel@live.com, 20628@debbugs.gnu.org
> 
> > From: Oleh Krehel <ohwoeowho@gmail.com>
> > Cc: clement.pitclaudel@live.com,  20628@debbugs.gnu.org
> > Date: Sat, 30 May 2015 21:26:03 +0200
> > 
> > frame 2
> > #2  try_window_reusing_current_matrix (w=w@entry=0x11d5eb0) at xdisp.c:17245
> > 17245	      init_to_row_start (&it, w, first_row_to_display);
> > 
> > p first_row_to_display->y
> > $1 = 909
> > 
> > p yb
> > $2 = <optimized out>
> > 
> > p window_text_bottom_y (w)
> > $3 = 909
> 
> Thanks, this confirms my suspicions: we are writing beyond the
> window's glyph matrix.  I will see what can be done about that.

I couldn't actually reproduce such crashes on my system, but I made a
few changes that hopefully will prevent them.  Could you please see if
the latest branch tip still crashes under some circumstances?

Thanks.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Oleh Krehel]

Eli Zaretskii <eliz@gnu.org> writes:

> I couldn't actually reproduce such crashes on my system, but I made a
> few changes that hopefully will prevent them.  Could you please see if
> the latest branch tip still crashes under some circumstances?

Thanks, the crash no longer occurs with your latest change.

 

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> From: Oleh Krehel <ohwoeowho@gmail.com>
> Cc: clement.pitclaudel@live.com,  20628@debbugs.gnu.org
> Date: Mon, 01 Jun 2015 12:54:28 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I couldn't actually reproduce such crashes on my system, but I made a
> > few changes that hopefully will prevent them.  Could you please see if
> > the latest branch tip still crashes under some circumstances?
> 
> Thanks, the crash no longer occurs with your latest change.

Great!

I will wait a few more days, in the hope that more people could check
out the branch and provide feedback, and will merge it then, barring
any new problems.

(Could someone please test this on NS?)

Thank you and Clement for your help in testing these non-trivial
changes.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Werner LEMBERG]

> The relevant place in the display engine where these factors are
> taken into consideration is around line 26388 in xdisp.c, [...]

I'm completely lost in this huuuge file.

> The change we look for should:
> 
>   . set the initial values for it->ascent and it->descent using some
>     heuristics based on pixel_size of the default face's font;

Where are the initial values set?

>   . update it->ascent and it->descent based on ascent/descent values
>     returned by get_per_char_metric in the per-character metrics
>     data

I think this already happens, right?  At least the line height
increases on my GNU/Linux version of Emacs if I use a font with larger
ascent/descent values...  So no additional action is needed for this,
it seems.


    Werner

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> Date: Wed, 03 Jun 2015 11:44:17 +0200 (CEST)
> Cc: clement.pitclaudel@live.com, ohwoeowho@gmail.com, 20628@debbugs.gnu.org
> From: Werner LEMBERG <wl@gnu.org>
> 
> > The relevant place in the display engine where these factors are
> > taken into consideration is around line 26388 in xdisp.c, [...]
> 
> I'm completely lost in this huuuge file.

Which is why I mentioned the line number explicitly...

> > The change we look for should:
> > 
> >   . set the initial values for it->ascent and it->descent using some
> >     heuristics based on pixel_size of the default face's font;
> 
> Where are the initial values set?

They are not set anywhere in the original display engine, because it
was not needed.  The values were set to the ascent and descent of each
produced glyph in x_produce_glyphs and its subroutines.  For character
glyphs, these were taken from the font-global values, since the
assumption was that those global values are reasonable.

But I was somewhat wrong in the above description: the important
values are not it->ascent and it->descent, but rather it->max_ascent
and it->max_descent.  These accumulate the maximum values seen while
producing a single screen line, and when the screen line ends, they
determine the line's height (see compute_line_metrics):

	  row->height = it->max_ascent + it->max_descent;

> >   . update it->ascent and it->descent based on ascent/descent values
> >     returned by get_per_char_metric in the per-character metrics
> >     data

> I think this already happens, right?  At least the line height
> increases on my GNU/Linux version of Emacs if I use a font with larger
> ascent/descent values...  So no additional action is needed for this,
> it seems.

Actually, since I wrote the message to which you respond, I think I
already solved the problem, you can see the solution in the
scratch/large-fonts branch of the Emacs Git repository.  If you have
time, I'd appreciate your opinion about the heuristics I use there.
Specifically, the following macro is used to detect fonts that claim
preposterously large ascent/descent values:

 #define FONT_TOO_HIGH(ft) ((ft)->ascent + (ft)->descent > 3*(ft)->pixel_size)

The pixel_size member of the struct font object is the pixel size with
which we opened the font, see the 'open' method of the font drivers
(e.g., 'xftfont_open' in xftfont.c).  The number 3 above is the
heuristics.

The other part of the heuristics is in how I compute "reasonable"
values of ascent and descent to be used instead of font's global
values.  The code is in the function 'normal_char_ascent_descent'
defined in xdisp.c.  This is necessary because, as I expected, the
font's global values are used all over the place in display-related
code, and so just computing line height from character metrics was not
enough to countermand all the negative effects of such preposterously
large font height values.  As just 2 examples, the font height was
used to allocate pixel screen estate for the mode line, and for
calculating line height when a 'display' property or a line-height
property specifies height as a fraction of the "normal" text height.

Thanks.

bug#20628: 25.0.50; Incorrect line height for some fonts

[Eli Zaretskii]

> Date: Mon, 01 Jun 2015 17:49:17 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: clement.pitclaudel@live.com, 20628@debbugs.gnu.org
> 
> > From: Oleh Krehel <ohwoeowho@gmail.com>
> > Cc: clement.pitclaudel@live.com,  20628@debbugs.gnu.org
> > Date: Mon, 01 Jun 2015 12:54:28 +0200
> > 
> > Eli Zaretskii <eliz@gnu.org> writes:
> > 
> > > I couldn't actually reproduce such crashes on my system, but I made a
> > > few changes that hopefully will prevent them.  Could you please see if
> > > the latest branch tip still crashes under some circumstances?
> > 
> > Thanks, the crash no longer occurs with your latest change.
> 
> Great!
> 
> I will wait a few more days, in the hope that more people could check
> out the branch and provide feedback, and will merge it then, barring
> any new problems.

The changes are now merged with master.

I'm marking this bug "done".

Thanks again to everyone for their help.