bug#25945: Emacs aborts while calling FT_Load_Glyph

bug#25945: Emacs aborts while calling FT_Load_Glyph

[Werner LEMBERG]

[f5388ba8a7f3970afd0e2bcc52c834ae56178442, 2017-Mar-03]

For me, Emacs aborts at ftfont.c:1550 while `mew' tries to display an
e-mail.

  1549  if (FT_Load_Glyph (ft_face, g->g.code, FT_LOAD_DEFAULT) != 0)
  1550    emacs_abort ();

Examining `ft_face' and `g->g.code' I see that the font in question is
`Padauk Book Bold' (PadaukBook-Bold.ttf), glyph 376.  Examining this
font further with `ftview' I see that bytecode of this font is broken,
and that the font can only be displayed successfully without bytecode.
[This is version 3.002 of the font, taken from the current TeXLive
repository.]

I think there is no reason that Emacs aborts for such broken fonts.
Instead I suggest that (a) Emacs tries to load the glyph again without
hinting, and (b) if that fails, it should display a missing glyph,
using the standard rectangle with hex digits in it.

New code for (a) is quite simple:

  if (FT_Load_Glyph (ft_face, g->g.code, FT_LOAD_DEFAULT) != 0)
    if (FT_Load_Glyph (ft_face, g->g.code, FT_LOAD_NO_HINTING) != 0)
      ...

[The code might be further improved by implementing a per-font glyph
 load mode, to be initialized with `FT_LOAD_DEFAULT'.  If, say, 100
 calls using `FT_LOAD_DEFAULT' for a given font fails, and
 `FT_LOAD_NO_HINTING' is successful, the default loading behaviour for
 this font could be switched to `FT_LOAD_NO_HINTING'.  Given that such
 broken fonts are rare my suggestion is probably overkill, however.]

My knowledge of Emacs internals is too small to provide an
implementation for (b).


    Werner


======================================================================


In GNU Emacs 26.0.50 (build 1, x86_64-unknown-linux-gnu, GTK+ Version 3.16.7)
 of 2017-03-03 built on linux.suse
Repository revision: f5388ba8a7f3970afd0e2bcc52c834ae56178442
Windowing system distributor 'The X.Org Foundation', version 11.0.11702000
System Description:	openSUSE Leap 42.1 (x86_64)

Recent messages:
Auto-saving...done
Auto-saving...done
next-line: End of buffer [4 times]
Mark set
Making completion list...
Quit
Type C-x 1 to remove help window.  
Mark set
scroll-up-command: End of buffer
Making completion list...

Configured using:
 'configure MAKEINFO=/usr/bin/makeinfo --with-x-toolkit=gtk
 --enable-checking=yes,glyphs --enable-check-lisp-object-type
 'CFLAGS=-O0 -g3 -gdwarf-4''

Configured features:
XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND DBUS GSETTINGS NOTIFY
GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB TOOLKIT_SCROLL_BARS
GTK3 X11

Important settings:
  value of $LC_MONETARY: de_AT.UTF-8
  value of $LC_TIME: de_AT.UTF-8
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=none
  locale-coding-system: utf-8-unix

Major mode: Summary

Minor modes in effect:
  TeX-PDF-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  column-number-mode: t
  transient-mark-mode: (only . t)

Load-path shadows:
/usr/local/share/emacs/site-lisp/thai-word hides /usr/local/share/emacs/26.0.50/lisp/language/thai-word

Features:
(shadow emacsbug message puny dired dired-loaddefs format-spec rfc822
mml mml-sec epa derived epg gnus-util rmail rmail-loaddefs mm-decode
mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
sendmail rfc2047 rfc2045 ietf-drums mail-utils apropos pp mew-varsx
mew-unix cal-menu calendar cal-loaddefs mew-auth mew-config mew-imap2
mew-imap mew-nntp2 mew-nntp mew-pop mew-smtp mew-ssl mew-ssh mew-net
mew-highlight mew-sort mew-fib mew-ext mew-refile mew-demo mew-attach
mew-draft mew-message mew-thread mew-virtual mew-summary4 mew-summary3
mew-summary2 mew-summary mew-search mew-pick mew-passwd mew-scan
mew-syntax mew-bq mew-smime mew-pgp mew-header mew-exec mew-mark
mew-mime mew-edit mew-decode mew-encode mew-cache mew-minibuf
mew-complete mew-addrbook mew-local mew-vars3 mew-vars2 mew-vars
mew-env mew-mule3 mew-mule mew-gemacs mew-key mew-func mew-blvs
mew-const mew tex dbus xml crm advice tex-site auto-loads quail
cjktilde mm-util mail-prsvr disp-table finder-inf package epg-config
url-handlers url-parse auth-source cl-seq eieio eieio-core cl-macs
eieio-loaddefs password-cache url-vars seq byte-opt subr-x gv bytecomp
byte-compile cl-extra help-mode easymenu cconv cl-loaddefs pcase
cl-lib time-date mule-util tooltip eldoc electric uniquify ediff-hook
vc-hooks lisp-float-type mwheel term/x-win x-win term/common-win x-dnd
tool-bar dnd fontset image regexp-opt fringe tabulated-list replace
newcomment text-mode elisp-mode lisp-mode prog-mode register page
menu-bar rfn-eshadow isearch timer select scroll-bar mouse jit-lock
font-lock syntax facemenu font-core term/tty-colors frame cl-generic
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet
lao korean japanese eucjp-ms cp51932 hebrew greek romanian slovak
czech european ethiopic indian cyrillic chinese composite charscript
case-table epa-hook jka-cmpr-hook help simple abbrev obarray
minibuffer cl-preloaded nadvice loaddefs button faces cus-face
macroexp files text-properties overlay sha1 md5 base64 format env
code-pages mule custom widget hashtable-print-readable backquote
dbusbind inotify dynamic-setting system-font-setting
font-render-setting move-toolbar gtk x-toolkit x multi-tty
make-network-process emacs)

Memory information:
((conses 16 237202 13045)
 (symbols 48 31618 2)
 (miscs 40 316 279)
 (strings 32 66908 9104)
 (string-bytes 1 1731569)
 (vectors 16 27986)
 (vector-slots 8 753851 6763)
 (floats 8 78 291)
 (intervals 56 1837 147)
 (buffers 976 14)
 (heap 1024 44161 1292))

bug#25945: Emacs aborts while calling FT_Load_Glyph

[Eli Zaretskii]

> Date: Fri, 03 Mar 2017 08:04:29 +0100 (CET)
> From: Werner LEMBERG <wl@gnu.org>
> 
> For me, Emacs aborts at ftfont.c:1550 while `mew' tries to display an
> e-mail.
> 
>   1549  if (FT_Load_Glyph (ft_face, g->g.code, FT_LOAD_DEFAULT) != 0)
>   1550    emacs_abort ();
> 
> Examining `ft_face' and `g->g.code' I see that the font in question is
> `Padauk Book Bold' (PadaukBook-Bold.ttf), glyph 376.  Examining this
> font further with `ftview' I see that bytecode of this font is broken,
> and that the font can only be displayed successfully without bytecode.
> [This is version 3.002 of the font, taken from the current TeXLive
> repository.]
> 
> I think there is no reason that Emacs aborts for such broken fonts.
> Instead I suggest that (a) Emacs tries to load the glyph again without
> hinting, and (b) if that fails, it should display a missing glyph,
> using the standard rectangle with hex digits in it.
> 
> New code for (a) is quite simple:
> 
>   if (FT_Load_Glyph (ft_face, g->g.code, FT_LOAD_DEFAULT) != 0)
>     if (FT_Load_Glyph (ft_face, g->g.code, FT_LOAD_NO_HINTING) != 0)
>       ...

This should probably be accompanied by a suitable FONT_ADD_LOG call,
to mention that this fallback was taken.

> [The code might be further improved by implementing a per-font glyph
>  load mode, to be initialized with `FT_LOAD_DEFAULT'.  If, say, 100
>  calls using `FT_LOAD_DEFAULT' for a given font fails, and
>  `FT_LOAD_NO_HINTING' is successful, the default loading behaviour for
>  this font could be switched to `FT_LOAD_NO_HINTING'.  Given that such
>  broken fonts are rare my suggestion is probably overkill, however.]
> 
> My knowledge of Emacs internals is too small to provide an
> implementation for (b).

I think it's too late for (b) when we discover this problem in
ftfont_get_metrics.  To do (b) we should have discovered this in
ftfont_has_char, or thereabouts.

I'm CC'ing Handa-san in the hope that he will have insights and
comments on this.

Thanks.

bug#25945: Emacs aborts while calling FT_Load_Glyph

[Werner LEMBERG]

>> New code for (a) is quite simple:
>> 
>>   if (FT_Load_Glyph (ft_face, g->g.code, FT_LOAD_DEFAULT) != 0)
>>     if (FT_Load_Glyph (ft_face, g->g.code, FT_LOAD_NO_HINTING) != 0)
>>       ...
> 
> This should probably be accompanied by a suitable FONT_ADD_LOG call,
> to mention that this fallback was taken.

Yes, perhaps.  However, if all glyphs are broken you will get a huuge
logfile...

>> My knowledge of Emacs internals is too small to provide an
>> implementation for (b).
> 
> I think it's too late for (b) when we discover this problem in
> ftfont_get_metrics.  To do (b) we should have discovered this in
> ftfont_has_char, or thereabouts.

Interesting.  How comes that Emacs aborts right there?

A note regarding the Padauk font: The problem is partly due to
FreeType 2.7.1, which has a stricter looping limit for TrueType
bytecode to detect endless loops – for this font, however, the limit
is a bit too strict; I will fix this in the next FreeType release.
Regardless of that, the bytecode in Padauk *is* buggy, and I've
already contacted the maintainers, asking for a new release using a
new, fixed ttfautohint version.

  https://github.com/silnrsi/font-padauk/issues/12


     Werner

bug#25945: Emacs aborts while calling FT_Load_Glyph

[Eli Zaretskii]

> Date: Fri, 03 Mar 2017 09:32:17 +0100 (CET)
> Cc: handa@gnu.org, 25945@debbugs.gnu.org
> From: Werner LEMBERG <wl@gnu.org>
> 
> > This should probably be accompanied by a suitable FONT_ADD_LOG call,
> > to mention that this fallback was taken.
> 
> Yes, perhaps.  However, if all glyphs are broken you will get a huuge
> logfile...

It's not a file, it's a buffer.  And the log is actually only added if
a special variable is non-nil, which only happens when one wants to
debug font issues.

> >> My knowledge of Emacs internals is too small to provide an
> >> implementation for (b).
> > 
> > I think it's too late for (b) when we discover this problem in
> > ftfont_get_metrics.  To do (b) we should have discovered this in
> > ftfont_has_char, or thereabouts.
> 
> Interesting.  How comes that Emacs aborts right there?

Sorry, I don't understand the question.  If you are asking why there's
a call to emacs_abort if FT_Load_Glyph fails, then I guess it''s
because this is unexpected and we have no code capable of coping with
such a calamity.

> A note regarding the Padauk font: The problem is partly due to
> FreeType 2.7.1, which has a stricter looping limit for TrueType
> bytecode to detect endless loops – for this font, however, the limit
> is a bit too strict; I will fix this in the next FreeType release.
> Regardless of that, the bytecode in Padauk *is* buggy, and I've
> already contacted the maintainers, asking for a new release using a
> new, fixed ttfautohint version.
> 
>   https://github.com/silnrsi/font-padauk/issues/12

Thanks for the info.

bug#25945: Emacs aborts while calling FT_Load_Glyph

[Werner LEMBERG]

>> > I think it's too late for (b) when we discover this problem in
>> > ftfont_get_metrics.  To do (b) we should have discovered this in
>> > ftfont_has_char, or thereabouts.
>> 
>> Interesting.  How comes that Emacs aborts right there?
> 
> Sorry, I don't understand the question.  If you are asking why
> there's a call to emacs_abort if FT_Load_Glyph fails, then I guess
> it''s because this is unexpected and we have no code capable of
> coping with such a calamity.

I mean: There's more than one place where FT_Load_Glyph is called with
`FT_LOAD_DEFAULT'.  You say that it is `too late'; this implies that
FT_Load_Glyph' has already been called earlier for a given glyph (with
`FT_LOAD_DEFAULT'), apparently without any fatal causes.  There's just
a single `emacs_abort' related to the calls to FT_Load_Glyph, and this
looks strange to me – and not justified.


    Werner

bug#25945: Emacs aborts while calling FT_Load_Glyph

[Eli Zaretskii]

> Date: Fri, 03 Mar 2017 11:37:06 +0100 (CET)
> Cc: handa@gnu.org, 25945@debbugs.gnu.org
> From: Werner LEMBERG <wl@gnu.org>
> 
> >> > I think it's too late for (b) when we discover this problem in
> >> > ftfont_get_metrics.  To do (b) we should have discovered this in
> >> > ftfont_has_char, or thereabouts.
> >> 
> >> Interesting.  How comes that Emacs aborts right there?
> > 
> > Sorry, I don't understand the question.  If you are asking why
> > there's a call to emacs_abort if FT_Load_Glyph fails, then I guess
> > it''s because this is unexpected and we have no code capable of
> > coping with such a calamity.
> 
> I mean: There's more than one place where FT_Load_Glyph is called with
> `FT_LOAD_DEFAULT'.  You say that it is `too late'; this implies that
> FT_Load_Glyph' has already been called earlier for a given glyph (with
> `FT_LOAD_DEFAULT'), apparently without any fatal causes.

No, it's "too late" because by the time ftfont_get_metrics is called,
Emacs has already established that the particular character is
supported by this font, and ftfont_get_metrics doesn't provide a way
to tell the caller that the character is not supported.  IOW, all the
fallbacks that look for an alternative font don't assume (AFAIK) that
a failure to display a character by some font can be discovered that
late.

> There's just a single `emacs_abort' related to the calls to
> FT_Load_Glyph, and this looks strange to me – and not justified.

It could be a problem, indeed.  But it could also be that all of those
other calls are either (a) after this particular one, so they will
never be made if this one fails, or (b) they have a way of
communicating a failure to their caller.

IOW, one must analyze these calls in the context of the control flow
when accessing a font for displaying a character, to understand
whether this single call to emacs_abort is enough.

bug#25945: Emacs aborts while calling FT_Load_Glyph

[Werner LEMBERG]

>> I mean: There's more than one place where FT_Load_Glyph is called
>> with `FT_LOAD_DEFAULT'.  You say that it is `too late'; this
>> implies that FT_Load_Glyph' has already been called earlier for a
>> given glyph (with `FT_LOAD_DEFAULT'), apparently without any fatal
>> causes.
> 
> No, it's "too late" because by the time ftfont_get_metrics is
> called, Emacs has already established that the particular character
> is supported by this font, and ftfont_get_metrics doesn't provide a
> way to tell the caller that the character is not supported.  IOW,
> all the fallbacks that look for an alternative font don't assume
> (AFAIK) that a failure to display a character by some font can be
> discovered that late.

OK, then my additional code line is definitely a good thing, since the
it makes FreeType try harder to load the glyph, so to say, and it
causes zero harm (speaking as the FreeType maintainer).

>> There's just a single `emacs_abort' related to the calls to
>> FT_Load_Glyph, and this looks strange to me – and not justified.
> 
> It could be a problem, indeed.  But it could also be that all of
> those other calls are either (a) after this particular one, so they
> will never be made if this one fails, or (b) they have a way of
> communicating a failure to their caller.
> 
> IOW, one must analyze these calls in the context of the control flow
> when accessing a font for displaying a character, to understand
> whether this single call to emacs_abort is enough.

Yeah.  However, I still suggest to apply my quick fix – this makes
Emacs crash one time less :-)  It works for me just fine, BTW.


    Werner

bug#25945: Emacs aborts while calling FT_Load_Glyph

[Eli Zaretskii]

> Date: Sat, 04 Mar 2017 06:39:03 +0100 (CET)
> Cc: handa@gnu.org, 25945@debbugs.gnu.org
> From: Werner LEMBERG <wl@gnu.org>
> 
> I still suggest to apply my quick fix – this makes Emacs crash one
> time less :-) It works for me just fine, BTW.

We most probably will, unless Handa-san will suggest a better idea

Thanks.

bug#25945: Emacs aborts while calling FT_Load_Glyph

[Eli Zaretskii]

Ping!

> Date: Fri, 03 Mar 2017 10:08:42 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 25945@debbugs.gnu.org
> 
> > Date: Fri, 03 Mar 2017 08:04:29 +0100 (CET)
> > From: Werner LEMBERG <wl@gnu.org>
> > 
> > For me, Emacs aborts at ftfont.c:1550 while `mew' tries to display an
> > e-mail.
> > 
> >   1549  if (FT_Load_Glyph (ft_face, g->g.code, FT_LOAD_DEFAULT) != 0)
> >   1550    emacs_abort ();
> > 
> > Examining `ft_face' and `g->g.code' I see that the font in question is
> > `Padauk Book Bold' (PadaukBook-Bold.ttf), glyph 376.  Examining this
> > font further with `ftview' I see that bytecode of this font is broken,
> > and that the font can only be displayed successfully without bytecode.
> > [This is version 3.002 of the font, taken from the current TeXLive
> > repository.]
> > 
> > I think there is no reason that Emacs aborts for such broken fonts.
> > Instead I suggest that (a) Emacs tries to load the glyph again without
> > hinting, and (b) if that fails, it should display a missing glyph,
> > using the standard rectangle with hex digits in it.
> > 
> > New code for (a) is quite simple:
> > 
> >   if (FT_Load_Glyph (ft_face, g->g.code, FT_LOAD_DEFAULT) != 0)
> >     if (FT_Load_Glyph (ft_face, g->g.code, FT_LOAD_NO_HINTING) != 0)
> >       ...
> 
> This should probably be accompanied by a suitable FONT_ADD_LOG call,
> to mention that this fallback was taken.
> 
> > [The code might be further improved by implementing a per-font glyph
> >  load mode, to be initialized with `FT_LOAD_DEFAULT'.  If, say, 100
> >  calls using `FT_LOAD_DEFAULT' for a given font fails, and
> >  `FT_LOAD_NO_HINTING' is successful, the default loading behaviour for
> >  this font could be switched to `FT_LOAD_NO_HINTING'.  Given that such
> >  broken fonts are rare my suggestion is probably overkill, however.]
> > 
> > My knowledge of Emacs internals is too small to provide an
> > implementation for (b).
> 
> I think it's too late for (b) when we discover this problem in
> ftfont_get_metrics.  To do (b) we should have discovered this in
> ftfont_has_char, or thereabouts.
> 
> I'm CC'ing Handa-san in the hope that he will have insights and
> comments on this.
> 
> Thanks.

bug#25945: Emacs aborts while calling FT_Load_Glyph

[Eli Zaretskii]

> Date: Fri, 03 Mar 2017 08:04:29 +0100 (CET)
> From: Werner LEMBERG <wl@gnu.org>
> 
> For me, Emacs aborts at ftfont.c:1550 while `mew' tries to display an
> e-mail.
> 
>   1549  if (FT_Load_Glyph (ft_face, g->g.code, FT_LOAD_DEFAULT) != 0)
>   1550    emacs_abort ();
> 
> Examining `ft_face' and `g->g.code' I see that the font in question is
> `Padauk Book Bold' (PadaukBook-Bold.ttf), glyph 376.  Examining this
> font further with `ftview' I see that bytecode of this font is broken,
> and that the font can only be displayed successfully without bytecode.
> [This is version 3.002 of the font, taken from the current TeXLive
> repository.]
> 
> I think there is no reason that Emacs aborts for such broken fonts.
> Instead I suggest that (a) Emacs tries to load the glyph again without
> hinting, and (b) if that fails, it should display a missing glyph,
> using the standard rectangle with hex digits in it.
> 
> New code for (a) is quite simple:
> 
>   if (FT_Load_Glyph (ft_face, g->g.code, FT_LOAD_DEFAULT) != 0)
>     if (FT_Load_Glyph (ft_face, g->g.code, FT_LOAD_NO_HINTING) != 0)
>       ...

Thanks, I've now pushed this change to the Emacs master branch, and
I'm marking this bug done.