bug#39892: 28.0.50; Crash when running async command

bug#39892: 28.0.50; Crash when running async command

[Ravine Var]

Emacs crashes when the output of 'gpg -d' is processed
in the '*Async Shell Command*' buffer. This happens when the
"Noto Sans Symbols2" font is used. To reproduce:

* Inside the emacs git tree, do:
   "gzip configure"
   "gpg -c configure.gz"

* Start 'emacs -Q'

* Eval this: (set-fontset-font t 'unicode "Noto Sans Symbols2" nil 'append)

* In dired, run dired-do-async-shell-command (&) on configure.gz.gpg and
  give 'gpg -d'. gpg now sends the decrypted output to stdout.

* At this point emacs crashes.

Backtrace:

Program received signal SIGSEGV, Segmentation fault.
0x00005555559705bf in AREF (array=0x0, idx=1) at lisp.h:1853
1853      return XVECTOR (array)->contents[idx];

(gdb) bt
#0  0x00005555559705bf in AREF (array=0x0, idx=1) at lisp.h:1853
#1  0x00005555559738cf in reorder_font_vector (font_group=0x5555569695e3, font=0x55555619a030) at fontset.c:403
#2  0x0000555555973f98 in fontset_find_font (fontset=0x555556250f25, c=59896, face=0x555556047d10, charset_id=-1, fallback=false) at fontset.c:565
#3  0x00005555559749b7 in fontset_font (fontset=0x55555687b0e5, c=59896, face=0x555556047d10, id=-1) at fontset.c:775
#4  0x00005555559751f2 in face_for_char (f=0x55555612d6e0, face=0x555556047d10, c=59896, pos=2280, object=0x0) at fontset.c:989
#5  0x00005555555bca03 in FACE_FOR_CHAR (f=0x55555612d6e0, face=0x555556047d10, character=59896, pos=2280, object=0x0) at dispextern.h:1891
#6  0x00005555555d523d in get_next_display_element (it=0x7fffffff92d0) at xdisp.c:7566
#7  0x00005555555d8aad in move_it_in_display_line_to (it=0x7fffffff92d0, to_charpos=2572, to_x=-1, op=MOVE_TO_POS) at xdisp.c:9054
#8  0x00005555555dc31a in move_it_to (it=0x7fffffff92d0, to_charpos=2572, to_x=-1, to_y=-1, to_vpos=-1, op=8) at xdisp.c:9804
#9  0x00005555555dcce7 in move_it_vertically_backward (it=0x7fffffffa6d0, dy=935) at xdisp.c:9998
#10 0x000055555564de4c in Frecenter (arg=0xfffffffffffffffe, redisplay=0x0) at window.c:6560
#11 0x000055555586b2ee in funcall_subr (subr=0x555555e719a0 <Srecenter>, numargs=1, args=0x7fffffffbb60) at eval.c:2869
#12 0x000055555586ae83 in Ffuncall (nargs=2, args=0x7fffffffbb58) at eval.c:2794
#13 0x00005555558d90ca in exec_byte_code (bytestr=0x5555560590e4, vector=0x55555673b2d5, maxdepth=0x32, args_template=0x406, nargs=1, args=0x7fffffffc268) at bytecode.c:633
#14 0x000055555586b987 in funcall_lambda (fun=0x55555673b3b5, nargs=1, arg_vector=0x7fffffffc260) at eval.c:2989
#15 0x000055555586aec7 in Ffuncall (nargs=2, args=0x7fffffffc258) at eval.c:2796
#16 0x000055555586a146 in funcall_nil (nargs=2, args=0x7fffffffc258) at eval.c:2435
#17 0x000055555586a575 in run_hook_with_args (nargs=2, args=0x7fffffffc258, funcall=0x55555586a123 <funcall_nil>) at eval.c:2612
#18 0x000055555586a1ce in Frun_hook_with_args (nargs=2, args=0x7fffffffc258) at eval.c:2477
#19 0x000055555586b1ce in funcall_subr (subr=0x555555e7cc60 <Srun_hook_with_args>, numargs=2, args=0x7fffffffc258) at eval.c:2847
#20 0x000055555586ae83 in Ffuncall (nargs=3, args=0x7fffffffc250) at eval.c:2794
#21 0x00005555558d90ca in exec_byte_code (bytestr=0x555556058fe4, vector=0x55555673b005, maxdepth=0x3a, args_template=0x80a, nargs=2, args=0x7fffffffc838) at bytecode.c:633
#22 0x000055555586b987 in funcall_lambda (fun=0x55555673b155, nargs=2, arg_vector=0x7fffffffc828) at eval.c:2989
#23 0x000055555586aec7 in Ffuncall (nargs=3, args=0x7fffffffc820) at eval.c:2796
#24 0x000055555586a0f9 in Fapply (nargs=2, args=0x7fffffffc8d0) at eval.c:2424
#25 0x000055555586a6ac in apply1 (fn=0x2aaa9e4cecf0, arg=0x555556956033) at eval.c:2640
#26 0x00005555558efa68 in read_process_output_call (fun_and_args=0x555556956043) at process.c:5988
#27 0x00005555558675d0 in internal_condition_case_1 (bfun=0x5555558efa3a <read_process_output_call>, arg=0x555556956043, handlers=0x90,
    hfun=0x5555558efa6a <read_process_output_error_handler>) at eval.c:1379
#28 0x00005555558f0353 in read_and_dispose_of_process_output (p=0x555556143950, chars=0x7fffffffc9e0 "\037\213\b\b\246\020_^", nbytes=4095, coding=0x55555674a590)
    at process.c:6209
#29 0x00005555558eff5d in read_process_output (proc=0x555556143955, channel=5) at process.c:6120
#30 0x00005555558ef28c in wait_reading_process_output (time_limit=30, nsecs=0, read_kbd=-1, do_display=true, wait_for_cell=0x0, wait_proc=0x0, just_wait_proc=0)
    at process.c:5807
#31 0x000055555559e9da in sit_for (timeout=0x7a, reading=true, display_option=1) at dispnew.c:6052
#32 0x0000555555748c67 in read_char (commandflag=1, map=0x555556956cd3, prev_event=0x0, used_mouse_menu=0x7fffffffe271, end_time=0x0) at keyboard.c:2738
#33 0x0000555555757aac in read_key_sequence (keybuf=0x7fffffffe4a0, prompt=0x0, dont_downcase_last=false, can_return_switch_frame=true, fix_current_buffer=true,
    prevent_redisplay=false) at keyboard.c:9549
#34 0x00005555557450b1 in command_loop_1 () at keyboard.c:1350
#35 0x0000555555867529 in internal_condition_case (bfun=0x555555744c4d <command_loop_1>, handlers=0x90, hfun=0x5555557443c2 <cmd_error>) at eval.c:1355
#36 0x0000555555744913 in command_loop_2 (ignore=0x0) at keyboard.c:1091
#37 0x0000555555866d9d in internal_catch (tag=0xcc60, func=0x5555557448e6 <command_loop_2>, arg=0x0) at eval.c:1116
#38 0x00005555557448b1 in command_loop () at keyboard.c:1070
--Type <RET> for more, q to quit, c to continue without paging--
#39 0x0000555555743f9a in recursive_edit_1 () at keyboard.c:714
#40 0x000055555574411c in Frecursive_edit () at keyboard.c:786
#41 0x000055555573c694 in main (argc=2, argv=0x7fffffffe928) at emacs.c:2036

In GNU Emacs 28.0.50 (build 1, x86_64-pc-linux-gnu, cairo version 1.17.3)
 of 2020-03-04 built on comp-lnx
Repository revision: 21ebfa1dd8129420c832031d055c708075aec02c
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12007000
System Description: Arch Linux

Configured using:
 'configure --prefix=/usr/local --without-rsvg --with-x-toolkit=no
 --without-lcms2 --without-libsystemd --without-dbus --without-gsettings
 --without-selinux --with-sound=no --enable-link-time-optimization
 'CFLAGS=-O3 -march=native''

Configured features:
XPM JPEG TIFF GIF PNG CAIRO NOTIFY INOTIFY ACL GNUTLS LIBXML2 FREETYPE
HARFBUZZ ZLIB OLDXMENU X11 XDBE XIM MODULES THREADS PDUMPER GMP

Important settings:
  value of $LANG: en_IN.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Dired by name

Minor modes in effect:
  iswitchb-mode: t
  save-place-mode: t
  savehist-mode: t
  dired-omit-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  column-number-mode: 1
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow face-remap mule-util emacsbug cl-extra help-mode dired-aux
edmacro kmacro emms-browser sort emms-playlist-sort emms-last-played
emms-cache emms-info-libtag emms-info later-do emms-player-simple
emms-playlist-mode emms-source-playlist emms-source-file locate emms
emms-compat iswitchb saveplace savehist visual-fill-column mu4e-contrib
eshell esh-cmd esh-ext esh-opt esh-proc esh-io esh-arg esh-module
esh-groups esh-util bookmark pp mu4e desktop frameset mu4e-org org ob
ob-tangle ob-ref ob-lob ob-table ob-exp org-macro org-footnote org-src
ob-comint org-pcomplete pcomplete org-list org-faces org-entities
noutline outline easy-mmode org-version ob-emacs-lisp ob-core ob-eval
org-table ol org-keys org-compat advice org-macs org-loaddefs find-func
mu4e-main mu4e-view cal-menu calendar cal-loaddefs browse-url comint
ansi-color mu4e-headers mu4e-compose mu4e-context mu4e-draft
mu4e-actions ido rfc2368 smtpmail auth-source eieio eieio-core cl-macs
eieio-loaddefs json map sendmail mu4e-mark mu4e-message flow-fill
mu4e-proc mu4e-utils doc-view jka-compr image-mode exif mu4e-lists
cl-seq hl-line mu4e-vars message rmc puny format-spec rfc822 mml mml-sec
password-cache epa derived epg epg-config gnus-util rmail rmail-loaddefs
text-property-search time-date subr-x seq byte-opt gv bytecomp
byte-compile cconv mm-decode mm-bodies mm-encode mail-parse rfc2231
rfc2047 rfc2045 mm-util ietf-drums mail-prsvr mailabbrev mail-utils
gmm-utils mailheader cl-loaddefs cl-lib mu4e-meta dictionary link
connection cycbuf dired-x dired dired-loaddefs xcscope ring easymenu
thingatpt tooltip eldoc electric uniquify ediff-hook vc-hooks
lisp-float-type mwheel term/x-win x-win term/common-win x-dnd tool-bar
dnd fontset image regexp-opt fringe tabulated-list replace newcomment
text-mode elisp-mode lisp-mode prog-mode register page tab-bar menu-bar
rfn-eshadow isearch timer select scroll-bar mouse jit-lock font-lock
syntax facemenu font-core term/tty-colors frame minibuffer cl-generic
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote threads inotify
dynamic-setting font-render-setting cairo x multi-tty
make-network-process emacs)

Memory information:
((conses 16 137708 9076)
 (symbols 48 15722 2)
 (strings 32 51174 3076)
 (string-bytes 1 1720398)
 (vectors 16 27082)
 (vector-slots 8 292203 16378)
 (floats 8 137 87)
 (intervals 56 483 4)
 (buffers 1000 12))

bug#39892: 28.0.50; Crash when running async command

[Robert Pluim]

>>>>> On Wed, 04 Mar 2020 08:07:21 +0530, Ravine Var <ravine.var@gmail.com> said:

    Ravine> Emacs crashes when the output of 'gpg -d' is processed
    Ravine> in the '*Async Shell Command*' buffer. This happens when the
    Ravine> "Noto Sans Symbols2" font is used. To reproduce:

    Ravine> * Inside the emacs git tree, do:
    Ravine>    "gzip configure"
    Ravine>    "gpg -c configure.gz"

    Ravine> * Start 'emacs -Q'

    Ravine> * Eval this: (set-fontset-font t 'unicode "Noto Sans Symbols2" nil 'append)

    Ravine> * In dired, run dired-do-async-shell-command (&) on configure.gz.gpg and
    Ravine>   give 'gpg -d'. gpg now sends the decrypted output to stdout.

    Ravine> * At this point emacs crashes.

Indeed. A simpler recipe is to do the set-fontset-font and then

'C-x 8 RET e9f8'

The relevant code is:

  for (i = 0; i < size; i++)
    {
      Lisp_Object rfont_def = AREF (vec, i);
      Lisp_Object font_def = RFONT_DEF_FONT_DEF (rfont_def);

'vec' at that point is

[
[nil [#<font-spec nil nil nil nil iso10646-1 nil nil nil nil nil nil nil nil> 144 nil] nil 0]
nil
[nil [#<font-spec nil nil Noto\ Sans\ Symbols2 nil nil nil nil nil nil nil nil nil ((:name . "Noto Sans Symbols2"))> 0 0] nil 2]
]

and i == 1, so rfont_def ends up as nil.

How did that 'nil' get in there?

Robert

bug#39892: 28.0.50; Crash when running async command

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Date: Wed, 04 Mar 2020 11:01:57 +0100
> Cc: 39892@debbugs.gnu.org
> 
>     Ravine> * Start 'emacs -Q'
> 
>     Ravine> * Eval this: (set-fontset-font t 'unicode "Noto Sans Symbols2" nil 'append)
> 
>     Ravine> * In dired, run dired-do-async-shell-command (&) on configure.gz.gpg and
>     Ravine>   give 'gpg -d'. gpg now sends the decrypted output to stdout.
> 
>     Ravine> * At this point emacs crashes.
> 
> Indeed. A simpler recipe is to do the set-fontset-font and then
> 
> 'C-x 8 RET e9f8'

In fact, _any_ font seems to cause that, as long as the other
arguments of set-fontset-font are as above.

> The relevant code is:
> 
>   for (i = 0; i < size; i++)
>     {
>       Lisp_Object rfont_def = AREF (vec, i);
>       Lisp_Object font_def = RFONT_DEF_FONT_DEF (rfont_def);
> 
> 'vec' at that point is
> 
> [
> [nil [#<font-spec nil nil nil nil iso10646-1 nil nil nil nil nil nil nil nil> 144 nil] nil 0]
> nil
> [nil [#<font-spec nil nil Noto\ Sans\ Symbols2 nil nil nil nil nil nil nil nil nil ((:name . "Noto Sans Symbols2"))> 0 0] nil 2]
> ]
> 
> and i == 1, so rfont_def ends up as nil.
> 
> How did that 'nil' get in there?

I don't know.  The code seems to expect one nil as the last element of
the vector, but not in the middle.  Note that the commentary at the
beginning of fontset.c explains when RFONT_DEF may be nil.

I will look into this soon if no one beats me to it.

Btw, U+E9F8 is a PUA character; how did it get into a context where we
are trying to display it?

bug#39892: 28.0.50; Crash when running async command

[Eli Zaretskii]

> Date: Wed, 04 Mar 2020 18:13:38 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: ravine.var@gmail.com, 39892@debbugs.gnu.org
> 
> >   for (i = 0; i < size; i++)
> >     {
> >       Lisp_Object rfont_def = AREF (vec, i);
> >       Lisp_Object font_def = RFONT_DEF_FONT_DEF (rfont_def);
> > 
> > 'vec' at that point is
> > 
> > [
> > [nil [#<font-spec nil nil nil nil iso10646-1 nil nil nil nil nil nil nil nil> 144 nil] nil 0]
> > nil
> > [nil [#<font-spec nil nil Noto\ Sans\ Symbols2 nil nil nil nil nil nil nil nil nil ((:name . "Noto Sans Symbols2"))> 0 0] nil 2]
> > ]
> > 
> > and i == 1, so rfont_def ends up as nil.
> > 
> > How did that 'nil' get in there?
> 
> I don't know.  The code seems to expect one nil as the last element of
> the vector, but not in the middle.  Note that the commentary at the
> beginning of fontset.c explains when RFONT_DEF may be nil.

How about the patch below?  It works for me, and seems to be TRT in
general, since it makes the code more defensive in the face of
possible nil values in that vector, which are legitimate according to
my reading of the commentary at the beginning of fontset.c.

CC'ing Handa-san in the hope that he might have comments or
suggestions.

Btw, in general using such fontset customizations make little sense,
since you are in effect telling Emacs this font covers all of the
Unicode, which cannot be true.  Not specifying the registry is also
not recommended, definitely not for such large ranges of codepoints.

> Btw, U+E9F8 is a PUA character; how did it get into a context where we
> are trying to display it?

I'd still like to know the answer to this.  It seems like something
fishy is going on in the original use case that requires us to try to
display this PUA codepoint.

Here's the patch I propose to install on the emacs-27 branch:

diff --git a/src/fontset.c b/src/fontset.c
index bca9452..c2bb8b2 100644
--- a/src/fontset.c
+++ b/src/fontset.c
@@ -367,8 +367,14 @@ fontset_add (Lisp_Object fontset, Lisp_Object range, Lisp_Object elt, Lisp_Objec
 static int
 fontset_compare_rfontdef (const void *val1, const void *val2)
 {
-  return (RFONT_DEF_SCORE (*(Lisp_Object *) val1)
-	  - RFONT_DEF_SCORE (*(Lisp_Object *) val2));
+  Lisp_Object v1 = *(Lisp_Object *) val1, v2 = *(Lisp_Object *) val2;
+  if (NILP (v1) && NILP (v2))
+    return 0;
+  else if (NILP (v1))
+    return INT_MIN;
+  else if (NILP (v2))
+    return INT_MAX;
+  return (RFONT_DEF_SCORE (v1) - RFONT_DEF_SCORE (v2));
 }
 
 /* Update a cons cell which has this form:
@@ -400,6 +406,8 @@ reorder_font_vector (Lisp_Object font_group, struct font *font)
   for (i = 0; i < size; i++)
     {
       Lisp_Object rfont_def = AREF (vec, i);
+      if (NILP (rfont_def))
+	continue;
       Lisp_Object font_def = RFONT_DEF_FONT_DEF (rfont_def);
       Lisp_Object font_spec = FONT_DEF_SPEC (font_def);
       int score = RFONT_DEF_SCORE (rfont_def) & 0xFF;

bug#39892: 28.0.50; Crash when running async command

[Robert Pluim]

>>>>> On Thu, 05 Mar 2020 11:46:59 +0200, Eli Zaretskii <eliz@gnu.org> said:

    Eli> How about the patch below?  It works for me, and seems to be TRT in
    Eli> general, since it makes the code more defensive in the face of
    Eli> possible nil values in that vector, which are legitimate according to
    Eli> my reading of the commentary at the beginning of fontset.c.

That patch works for me.

    >> Btw, U+E9F8 is a PUA character; how did it get into a context where we
    >> are trying to display it?

    Eli> I'd still like to know the answer to this.  It seems like something
    Eli> fishy is going on in the original use case that requires us to try to
    Eli> display this PUA codepoint.

Itʼs essentially doing 'cat' on a .gz file. Thereʼs no guarantee
whatsoever that the result is valid. And in fact doing exactly that
crashes the same way (without the patch).

Interestingly, under macOS it displays a proper glyph using
"mac-ct:-*-Hannotate TC-normal-normal-normal-*-14-*-*-*-p-0-iso10646-1
(#x933F)", so somebody has decided that codepoint is good for something.

Robert

bug#39892: 28.0.50; Crash when running async command

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: ravine.var@gmail.com,  39892@debbugs.gnu.org,  Kenichi Handa
>  <handa@gnu.org>
> Date: Thu, 05 Mar 2020 11:57:24 +0100
> 
> >>>>> On Thu, 05 Mar 2020 11:46:59 +0200, Eli Zaretskii <eliz@gnu.org> said:
> 
>     Eli> How about the patch below?  It works for me, and seems to be TRT in
>     Eli> general, since it makes the code more defensive in the face of
>     Eli> possible nil values in that vector, which are legitimate according to
>     Eli> my reading of the commentary at the beginning of fontset.c.
> 
> That patch works for me.

Thanks for testing.  I will wait for Ravine to confirm it fixes the
original use case.

>     >> Btw, U+E9F8 is a PUA character; how did it get into a context where we
>     >> are trying to display it?
> 
>     Eli> I'd still like to know the answer to this.  It seems like something
>     Eli> fishy is going on in the original use case that requires us to try to
>     Eli> display this PUA codepoint.
> 
> Itʼs essentially doing 'cat' on a .gz file.

I see.  But I guess there was some real-life use case behind all that?
I mean, why would someone want to dump a Gzip-compressed text to the
screen, and inside Emacs on top of that?

> Interestingly, under macOS it displays a proper glyph using
> "mac-ct:-*-Hannotate TC-normal-normal-normal-*-14-*-*-*-p-0-iso10646-1
> (#x933F)", so somebody has decided that codepoint is good for something.

Yes, some fonts usurp the PUA for displaying characters not (yet) in
Unicode.  This being a font for Chinese, I guess they use it for
characters in GB-18030 that don't map to Unicode, or somesuch.

bug#39892: 28.0.50; Crash when running async command

[Ravine Var]

Eli Zaretskii <eliz@gnu.org> writes:
> Thanks for testing.  I will wait for Ravine to confirm it fixes the
> original use case.

I tested the patch and there is no crash now.

> I see.  But I guess there was some real-life use case behind all that?
> I mean, why would someone want to dump a Gzip-compressed text to the
> screen, and inside Emacs on top of that?

It was simply a mistake. :-)

The '-o' option needs to be given to gpg to specify an output file.
I forgot to mention it and gpg sent its output to stdout, and then
emacs crashed.

bug#39892: 28.0.50; Crash when running async command

[Eli Zaretskii]

> From: Ravine Var <ravine.var@gmail.com>
> Cc: Robert Pluim <rpluim@gmail.com>, 39892@debbugs.gnu.org, handa@gnu.org
> Date: Thu, 05 Mar 2020 20:51:55 +0530
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> > Thanks for testing.  I will wait for Ravine to confirm it fixes the
> > original use case.
> 
> I tested the patch and there is no crash now.

Thanks, I installed the patch on the emacs-27 branch, and I'm closing
the bug report.

bug#39892: 28.0.50; Crash when running async command

[Ravine Var]

Eli Zaretskii <eliz@gnu.org> writes:
> Btw, in general using such fontset customizations make little sense,
> since you are in effect telling Emacs this font covers all of the
> Unicode, which cannot be true.  Not specifying the registry is also
> not recommended, definitely not for such large ranges of codepoints.

I was searching for some fancy characters for emms, like this:

(setq emms-browser-info-title-format "%i𝅘𝅥𝅯 %t")
(setq emms-browser-info-artist-format    "%i🞶 %n")
(setq emms-browser-info-album-format     "%i🟅 %n")

But you are right, the customization needs to be fixed.

bug#39892: 28.0.50; Crash when running async command

[Eli Zaretskii]

> From: Ravine Var <ravine.var@gmail.com>
> Cc: rpluim@gmail.com, 39892@debbugs.gnu.org, Kenichi Handa <handa@gnu.org>
> Date: Thu, 05 Mar 2020 20:54:50 +0530
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> > Btw, in general using such fontset customizations make little sense,
> > since you are in effect telling Emacs this font covers all of the
> > Unicode, which cannot be true.  Not specifying the registry is also
> > not recommended, definitely not for such large ranges of codepoints.
> 
> I was searching for some fancy characters for emms, like this:
> 
> (setq emms-browser-info-title-format "%i𝅘𝅥𝅯 %t")
> (setq emms-browser-info-artist-format    "%i🞶 %n")
> (setq emms-browser-info-album-format     "%i🟅 %n")

I think the Emacs defaults will allow you to see them, and if they
don't (try "emacs -Q" to see) then just install the Symbola font.