bug#39075: 28.0.50; Emacs hangs on 100% CPU and grows beyond bounds in shell-mode

bug#39075: 28.0.50; Emacs hangs on 100% CPU and grows beyond bounds in shell-mode

[Pieter van Oostrum]

1) Emacs -Q
2) M-x shell
3) type some command, and use some filename completions on the way
(using TAB).
4) Type a semicolon (;)

Now Emacs hangs, the ; does not appear, and it doesn't react to a C-g typed.
It uses 100% CPU and memory grows beyond bounds, eventually just making
my whole computer unresponsive.

Typing a whole series of C-g's might stop the loop (with the ; appearing
then) or it might crash Emacs:
Fatal error 11: Segmentation fault
Abort trap: 6

This report is typed in a session where the sequence of C-g's did stop
the loop.
The C-g caused this message:
Error in post-command-hook (completion-in-region--postch): (quit)
Although the option Enter debugger on Quit/C-g was set, no backtrace was
generated.

This emacs was compiled from master today, but it also happens in earlier versions.


In GNU Emacs 28.0.50 (build 1, i686-apple-darwin10.0.0, NS appkit-1561.61 Version 10.13.6 (Build 17G10021))
 of 2020-01-10 built on cochabamba.vanoostrum.org
Repository revision: 17cfd708575c351d030f8b05c5921d1867028d79
Repository branch: fix
Windowing system distributor 'Apple', version 10.3.1561
System Description:  Mac OS X 10.13.6

Recent messages:
Quit
No match [2 times]
~ 
Error in post-command-hook (completion-in-region--postch): (quit)
Quit [2 times]
Complete, but not unique
Making completion list...
Complete, but not unique
Error in post-command-hook (completion-in-region--postch): (quit)
Quit
Quit
Configured using:
 'configure --build i686-apple-darwin10.0.0 --without-dbus --with-ns
 build_alias=i686-apple-darwin10.0.0 'CFLAGS=-pipe -march=nocona'
 PKG_CONFIG_PATH=/opt/local/lib/pkgconfig/:/usr/X11R6/pkgconfig/:/usr/local/lib/pkgconfig/:/usr/lib/pkgconfig/'

Configured features:
RSVG GLIB NOTIFY KQUEUE ACL GNUTLS LIBXML2 ZLIB TOOLKIT_SCROLL_BARS XIM
NS MODULES THREADS PDUMPER LCMS2

Important settings:
  value of $LC_CTYPE: UTF-8
  value of $LANG: en_GB.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Shell

Minor modes in effect:
  shell-dirtrack-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny dired dired-loaddefs
format-spec rfc822 mml easymenu mml-sec password-cache epa derived epg
epg-config gnus-util rmail rmail-loaddefs text-property-search time-date
subr-x seq byte-opt gv bytecomp byte-compile cconv mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader cl-loaddefs
cl-lib sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils
shell pcomplete comint ansi-color ring tooltip eldoc electric uniquify
ediff-hook vc-hooks lisp-float-type mwheel term/ns-win ns-win
ucs-normalize mule-util term/common-win tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode elisp-mode
lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow isearch
timer select scroll-bar mouse jit-lock font-lock syntax facemenu
font-core term/tty-colors frame minibuffer cl-generic cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote threads kqueue cocoa ns
lcms2 multi-tty make-network-process emacs)

Memory information:
((conses 16 55230 57689)
 (symbols 48 7130 11)
 (strings 32 18286 5181)
 (string-bytes 1 600859)
 (vectors 16 11939)
 (vector-slots 8 157593 69152)
 (floats 8 19 126)
 (intervals 56 1161 97)
 (buffers 1000 14))

-- 
Pieter van Oostrum <pieter@vanoostrum.org>
www: http://pieter.vanoostrum.org/
PGP key: [8DAE142BE17999C4]


-- 
Pieter van Oostrum <pieter@vanoostrum.org>
www: http://pieter.vanoostrum.org/
PGP key: [8DAE142BE17999C4]

bug#39075: 28.0.50; Emacs hangs on 100% CPU and grows beyond bounds in shell-mode

[Eli Zaretskii]

> Date: Fri, 10 Jan 2020 22:16:58 +0100
> From: Pieter van Oostrum <pieter@vanoostrum.org>
> 
> 1) Emacs -Q
> 2) M-x shell
> 3) type some command, and use some filename completions on the way
> (using TAB).
> 4) Type a semicolon (;)
> 
> Now Emacs hangs, the ; does not appear, and it doesn't react to a C-g typed.
> It uses 100% CPU and memory grows beyond bounds, eventually just making
> my whole computer unresponsive.

I cannot reproduce this on GNU/Linux, so it's probably macOS-specific.

bug#39075: 28.0.50; Emacs hangs on 100% CPU and grows beyond bounds in shell-mode

[Pieter van Oostrum]

[-- Attachment #1: Type: text/plain, Size: 1012 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Fri, 10 Jan 2020 22:16:58 +0100
>> From: Pieter van Oostrum <pieter@vanoostrum.org>
>> 
>> 1) Emacs -Q
>> 2) M-x shell
>> 3) type some command, and use some filename completions on the way
>> (using TAB).
>> 4) Type a semicolon (;)
>> 
>> Now Emacs hangs, the ; does not appear, and it doesn't react to a C-g typed.
>> It uses 100% CPU and memory grows beyond bounds, eventually just making
>> my whole computer unresponsive.
>
> I cannot reproduce this on GNU/Linux, so it's probably macOS-specific.

It's not so clear what could be MacOS-specific.

I ran it under gdb, and interrupted it several times with C-z in gdb. Most of the stack traces were in the garbage collector, suggesting that it is collecting like crazy. This doesn't surprise me, as it is constantly allocating new memory. The rest of this stack trace doesn't have useful information.

I managed to get a stack trace where it is processing. I haven't analysed these yet, but I include both here.


[-- Attachment #2: stack traces --]
[-- Type: application/octet-stream, Size: 12646 bytes --]

^Z
Thread 3 received signal SIGTSTP, Stopped (user).
0x000000010029c66c in PSEUDOVECTORP (a=XIL(0x1061000bd), code=26)
    at ./lisp.h:1723
1723	}
(gdb) bt
#0  0x000000010029c66c in PSEUDOVECTORP (a=XIL(0x1061000bd), code=26)
    at ./lisp.h:1723
#1  0x000000010029c57a in SUB_CHAR_TABLE_P (a=XIL(0x1061000bd))
    at ./lisp.h:2018
#2  0x000000010029c41b in CHAR_TABLE_REF_ASCII (ct=XIL(0x1060f4b9d), idx=59)
    at ./lisp.h:2036
#3  0x000000010029c275 in CHAR_TABLE_REF (ct=XIL(0x1060f4b9d), idx=59)
    at ./lisp.h:2052
#4  0x00000001002893a4 in char_table_translate (obj=XIL(0x1060f4b9d), ch=59)
    at ./character.h:701
#5  0x000000010028a510 in re_match_2_internal (bufp=0x100a07e28, 
    string1=0x10705f200 "bash-3.2$ (cd ~/TEST/LATEX/FANCYHDR/;", size1=37, 
    string2=0x10705f9e0 "latex", size2=5, pos=36, regs=0x100991098, stop=42)
    at regex-emacs.c:4226
#6  0x0000000100290966 in rpl_re_match_2 (bufp=0x100a07e28, 
    string1=0x10705f200 "bash-3.2$ (cd ~/TEST/LATEX/FANCYHDR/;", size1=37, 
    string2=0x10705f9e0 "latex", size2=5, pos=36, regs=0x100991098, stop=42)
    at regex-emacs.c:3850
#7  0x0000000100277960 in looking_at_1 (string=XIL(0x10ba43fc4), posix=false)
    at search.c:316
#8  0x00000001002774d7 in Flooking_at (regexp=XIL(0x10ba43fc4)) at search.c:352
#9  0x0000000100316094 in funcall_subr (subr=0x100556578, numargs=1, 
    args=0x7ffeefbf7348) at eval.c:2867
#10 0x000000010031469e in Ffuncall (nargs=2, args=0x7ffeefbf7340)
    at eval.c:2794
#11 0x00000001003a50bf in exec_byte_code (bytestr=XIL(0x10ba43f84), 
    vector=XIL(0x105978755), maxdepth=make_fixnum(10), 
    args_template=make_fixnum(0), nargs=0, args=0x7ffeefbf8398)
    at bytecode.c:633
#12 0x0000000100316785 in funcall_lambda (fun=XIL(0x1059787f5), nargs=0, 
    arg_vector=0x7ffeefbf8398) at eval.c:2989
#13 0x00000001003146ee in Ffuncall (nargs=1, args=0x7ffeefbf8390)
    at eval.c:2796
#14 0x00000001003a50bf in exec_byte_code (bytestr=XIL(0x10ba42df4), 
    vector=XIL(0x10497d6d5), maxdepth=make_fixnum(13), 
    args_template=make_fixnum(256), nargs=1, args=0x7ffeefbf9448)
    at bytecode.c:633
#15 0x0000000100316785 in funcall_lambda (fun=XIL(0x10497d7a5), nargs=1, 
    arg_vector=0x7ffeefbf9440) at eval.c:2989
#16 0x00000001003146ee in Ffuncall (nargs=2, args=0x7ffeefbf9438)
    at eval.c:2796
#17 0x00000001003a50bf in exec_byte_code (bytestr=XIL(0x10ba43264), 
    vector=XIL(0x10497dd15), maxdepth=make_fixnum(5), 
    args_template=make_fixnum(0), nargs=0, args=0x7ffeefbfa420)
    at bytecode.c:633
#18 0x0000000100316785 in funcall_lambda (fun=XIL(0x10497dda5), nargs=0, 
    arg_vector=0x7ffeefbfa420) at eval.c:2989
#19 0x00000001003146ee in Ffuncall (nargs=1, args=0x7ffeefbfa418)
--Type <RET> for more, q to quit, c to continue without paging--
    at eval.c:2796
#20 0x00000001003a50bf in exec_byte_code (bytestr=XIL(0x10ba42244), 
    vector=XIL(0x10599d7d5), maxdepth=make_fixnum(15), 
    args_template=make_fixnum(0), nargs=0, args=0x7ffeefbfb898)
    at bytecode.c:633
#21 0x0000000100316785 in funcall_lambda (fun=XIL(0x10599d905), nargs=0, 
    arg_vector=0x7ffeefbfb898) at eval.c:2989
#22 0x00000001003146ee in Ffuncall (nargs=1, args=0x7ffeefbfb890)
    at eval.c:2796
#23 0x0000000100315046 in run_hook_with_args (nargs=1, args=0x7ffeefbfb890, 
    funcall=0x1003144b0 <Ffuncall>) at eval.c:2612
#24 0x0000000100315134 in Frun_hook_with_args_until_success (nargs=1, 
    args=0x7ffeefbfb890) at eval.c:2498
#25 0x0000000100315f76 in funcall_subr (subr=0x1005596c8, numargs=1, 
    args=0x7ffeefbfb890) at eval.c:2847
#26 0x000000010031469e in Ffuncall (nargs=2, args=0x7ffeefbfb888)
    at eval.c:2794
#27 0x00000001003a50bf in exec_byte_code (bytestr=XIL(0x10ba2c794), 
    vector=XIL(0x1059a2965), maxdepth=make_fixnum(2), 
    args_template=make_fixnum(0), nargs=0, args=0x7ffeefbfc820)
    at bytecode.c:633
#28 0x0000000100316785 in funcall_lambda (fun=XIL(0x1059a2985), nargs=0, 
    arg_vector=0x7ffeefbfc820) at eval.c:2989
#29 0x00000001003146ee in Ffuncall (nargs=1, args=0x7ffeefbfc818)
    at eval.c:2796
#30 0x00000001003a50bf in exec_byte_code (bytestr=XIL(0x106228994), 
    vector=XIL(0x104be1815), maxdepth=make_fixnum(3), 
    args_template=make_fixnum(0), nargs=0, args=0x7ffeefbfd7c0)
    at bytecode.c:633
#31 0x0000000100316785 in funcall_lambda (fun=XIL(0x104be1835), nargs=0, 
    arg_vector=0x7ffeefbfd7c0) at eval.c:2989
#32 0x00000001003146ee in Ffuncall (nargs=1, args=0x7ffeefbfd7b8)
    at eval.c:2796
#33 0x00000001003a50bf in exec_byte_code (bytestr=XIL(0x1061948b4), 
    vector=XIL(0x1061947b5), maxdepth=make_fixnum(2), 
    args_template=make_fixnum(0), nargs=0, args=0x7ffeefbfe778)
    at bytecode.c:633
#34 0x0000000100316785 in funcall_lambda (fun=XIL(0x10619478d), nargs=0, 
    arg_vector=0x7ffeefbfe778) at eval.c:2989
#35 0x00000001003146ee in Ffuncall (nargs=1, args=0x7ffeefbfe770)
    at eval.c:2796
#36 0x000000010031574f in call0 (fn=XIL(0x575b7d8)) at eval.c:2647
#37 0x00000001001e9fa5 in safe_run_hooks_1 (nargs=2, args=0x7ffeefbfe858)
    at keyboard.c:1775
#38 0x000000010030d3ca in internal_condition_case_n (
    bfun=0x1001e9f50 <safe_run_hooks_1>, nargs=2, args=0x7ffeefbfe858, 
    handlers=XIL(0x30), hfun=0x1001e9fc0 <safe_run_hooks_error>) at eval.c:1435
#39 0x00000001001ccd35 in safe_run_hook_funcall (nargs=2, args=0x7ffeefbfe9f8)
--Type <RET> for more, q to quit, c to continue without paging--
    at keyboard.c:1822
#40 0x0000000100315046 in run_hook_with_args (nargs=2, args=0x7ffeefbfe9f8, 
    funcall=0x1001ccc90 <safe_run_hook_funcall>) at eval.c:2612
#41 0x00000001001c7ba3 in safe_run_hooks (hook=XIL(0xa170)) at keyboard.c:1838
#42 0x00000001001c71b6 in command_loop_1 () at keyboard.c:1477
#43 0x000000010030d0af in internal_condition_case (
    bfun=0x1001c63a0 <command_loop_1>, handlers=XIL(0x90), 
    hfun=0x1001e9a40 <cmd_error>) at eval.c:1355
#44 0x00000001001e9921 in command_loop_2 (ignore=XIL(0)) at keyboard.c:1091
#45 0x000000010030c1e8 in internal_catch (tag=XIL(0xc480), 
    func=0x1001e98f0 <command_loop_2>, arg=XIL(0)) at eval.c:1116
#46 0x00000001001c5415 in command_loop () at keyboard.c:1070
#47 0x00000001001c51e7 in recursive_edit_1 () at keyboard.c:714
#48 0x00000001001c5696 in Frecursive_edit () at keyboard.c:786
#49 0x00000001001c21fe in main (argc=2, argv=0x7ffeefbff648) at emacs.c:2054

Lisp Backtrace:
Cannot access memory at address 0x56e7998
(gdb) 


========================================================================

^Z
Thread 3 received signal SIGTSTP, Stopped (user).
0x00000001002b062d in mem_find (start=0x19533c6c0) at alloc.c:4025
4025	  while (start < p->start || start >= p->end)
(gdb) bt
#0  0x00000001002b062d in mem_find (start=0x19533c6c0) at alloc.c:4025
#1  0x00000001002b353f in mark_object (arg=XIL(0x19b11b7a3)) at alloc.c:6685
#2  0x00000001002b004c in mark_maybe_object (obj=XIL(0x19b11b7a3))
    at alloc.c:4642
#3  0x00000001002b0153 in mark_memory (start=0x7ffeefbf7030, 
    end=0x7ffeefbff608) at alloc.c:4788
#4  0x00000001002b009d in mark_stack (bottom=0x7ffeefbff608 "", 
    end=0x7ffeefbf7030 "@p\277\357\376\177") at alloc.c:4995
#5  0x0000000100421f61 in mark_one_thread (thread=0x100991000) at thread.c:630
#6  0x0000000100420763 in mark_threads_callback (ignore=0x0) at thread.c:661
#7  0x00000001002b01b4 in flush_stack_call_func (
    func=0x1004206c0 <mark_threads_callback>, arg=0x0) at alloc.c:5022
#8  0x00000001004206b4 in mark_threads () at thread.c:668
#9  0x00000001002b20a6 in garbage_collect () at alloc.c:6012
#10 0x00000001002b1d3d in maybe_garbage_collect () at alloc.c:5918
#11 0x000000010030f0ca in maybe_gc () at ./lisp.h:5066
#12 0x000000010031458b in Ffuncall (nargs=2, args=0x7ffeefbf7340)
    at eval.c:2778
#13 0x00000001003a50bf in exec_byte_code (bytestr=XIL(0x10ba43f84), 
    vector=XIL(0x105978755), maxdepth=make_fixnum(10), 
    args_template=make_fixnum(0), nargs=0, args=0x7ffeefbf8398)
    at bytecode.c:633
#14 0x0000000100316785 in funcall_lambda (fun=XIL(0x1059787f5), nargs=0, 
    arg_vector=0x7ffeefbf8398) at eval.c:2989
#15 0x00000001003146ee in Ffuncall (nargs=1, args=0x7ffeefbf8390)
    at eval.c:2796
#16 0x00000001003a50bf in exec_byte_code (bytestr=XIL(0x10ba42df4), 
    vector=XIL(0x10497d6d5), maxdepth=make_fixnum(13), 
    args_template=make_fixnum(256), nargs=1, args=0x7ffeefbf9448)
    at bytecode.c:633
#17 0x0000000100316785 in funcall_lambda (fun=XIL(0x10497d7a5), nargs=1, 
    arg_vector=0x7ffeefbf9440) at eval.c:2989
#18 0x00000001003146ee in Ffuncall (nargs=2, args=0x7ffeefbf9438)
    at eval.c:2796
#19 0x00000001003a50bf in exec_byte_code (bytestr=XIL(0x10ba43264), 
    vector=XIL(0x10497dd15), maxdepth=make_fixnum(5), 
    args_template=make_fixnum(0), nargs=0, args=0x7ffeefbfa420)
    at bytecode.c:633
#20 0x0000000100316785 in funcall_lambda (fun=XIL(0x10497dda5), nargs=0, 
    arg_vector=0x7ffeefbfa420) at eval.c:2989
#21 0x00000001003146ee in Ffuncall (nargs=1, args=0x7ffeefbfa418)
    at eval.c:2796
#22 0x00000001003a50bf in exec_byte_code (bytestr=XIL(0x10ba42244), 
    vector=XIL(0x10599d7d5), maxdepth=make_fixnum(15), 
    args_template=make_fixnum(0), nargs=0, args=0x7ffeefbfb898)
    at bytecode.c:633
#23 0x0000000100316785 in funcall_lambda (fun=XIL(0x10599d905), nargs=0, 
    arg_vector=0x7ffeefbfb898) at eval.c:2989
--Type <RET> for more, q to quit, c to continue without paging--
#24 0x00000001003146ee in Ffuncall (nargs=1, args=0x7ffeefbfb890)
    at eval.c:2796
#25 0x0000000100315046 in run_hook_with_args (nargs=1, args=0x7ffeefbfb890, 
    funcall=0x1003144b0 <Ffuncall>) at eval.c:2612
#26 0x0000000100315134 in Frun_hook_with_args_until_success (nargs=1, 
    args=0x7ffeefbfb890) at eval.c:2498
#27 0x0000000100315f76 in funcall_subr (subr=0x1005596c8, numargs=1, 
    args=0x7ffeefbfb890) at eval.c:2847
#28 0x000000010031469e in Ffuncall (nargs=2, args=0x7ffeefbfb888)
    at eval.c:2794
#29 0x00000001003a50bf in exec_byte_code (bytestr=XIL(0x10ba2c794), 
    vector=XIL(0x1059a2965), maxdepth=make_fixnum(2), 
    args_template=make_fixnum(0), nargs=0, args=0x7ffeefbfc820)
    at bytecode.c:633
#30 0x0000000100316785 in funcall_lambda (fun=XIL(0x1059a2985), nargs=0, 
    arg_vector=0x7ffeefbfc820) at eval.c:2989
#31 0x00000001003146ee in Ffuncall (nargs=1, args=0x7ffeefbfc818)
    at eval.c:2796
#32 0x00000001003a50bf in exec_byte_code (bytestr=XIL(0x106228994), 
    vector=XIL(0x104be1815), maxdepth=make_fixnum(3), 
    args_template=make_fixnum(0), nargs=0, args=0x7ffeefbfd7c0)
    at bytecode.c:633
#33 0x0000000100316785 in funcall_lambda (fun=XIL(0x104be1835), nargs=0, 
    arg_vector=0x7ffeefbfd7c0) at eval.c:2989
#34 0x00000001003146ee in Ffuncall (nargs=1, args=0x7ffeefbfd7b8)
    at eval.c:2796
#35 0x00000001003a50bf in exec_byte_code (bytestr=XIL(0x1061948b4), 
    vector=XIL(0x1061947b5), maxdepth=make_fixnum(2), 
    args_template=make_fixnum(0), nargs=0, args=0x7ffeefbfe778)
    at bytecode.c:633
#36 0x0000000100316785 in funcall_lambda (fun=XIL(0x10619478d), nargs=0, 
    arg_vector=0x7ffeefbfe778) at eval.c:2989
#37 0x00000001003146ee in Ffuncall (nargs=1, args=0x7ffeefbfe770)
    at eval.c:2796
#38 0x000000010031574f in call0 (fn=XIL(0x575b7d8)) at eval.c:2647
#39 0x00000001001e9fa5 in safe_run_hooks_1 (nargs=2, args=0x7ffeefbfe858)
    at keyboard.c:1775
#40 0x000000010030d3ca in internal_condition_case_n (
    bfun=0x1001e9f50 <safe_run_hooks_1>, nargs=2, args=0x7ffeefbfe858, 
    handlers=XIL(0x30), hfun=0x1001e9fc0 <safe_run_hooks_error>) at eval.c:1435
#41 0x00000001001ccd35 in safe_run_hook_funcall (nargs=2, args=0x7ffeefbfe9f8)
    at keyboard.c:1822
#42 0x0000000100315046 in run_hook_with_args (nargs=2, args=0x7ffeefbfe9f8, 
    funcall=0x1001ccc90 <safe_run_hook_funcall>) at eval.c:2612
#43 0x00000001001c7ba3 in safe_run_hooks (hook=XIL(0xa170)) at keyboard.c:1838
#44 0x00000001001c71b6 in command_loop_1 () at keyboard.c:1477
#45 0x000000010030d0af in internal_condition_case (
    bfun=0x1001c63a0 <command_loop_1>, handlers=XIL(0x90), 
--Type <RET> for more, q to quit, c to continue without paging--
    hfun=0x1001e9a40 <cmd_error>) at eval.c:1355
#46 0x00000001001e9921 in command_loop_2 (ignore=XIL(0)) at keyboard.c:1091
#47 0x000000010030c1e8 in internal_catch (tag=XIL(0xc480), 
    func=0x1001e98f0 <command_loop_2>, arg=XIL(0)) at eval.c:1116
#48 0x00000001001c5415 in command_loop () at keyboard.c:1070
#49 0x00000001001c51e7 in recursive_edit_1 () at keyboard.c:714
#50 0x00000001001c5696 in Frecursive_edit () at keyboard.c:786
#51 0x00000001001c21fe in main (argc=2, argv=0x7ffeefbff648) at emacs.c:2054

Lisp Backtrace:
Cannot access memory at address 0x291
(gdb) 

[-- Attachment #3: Type: text/plain, Size: 87 bytes --]


-- 
Pieter van Oostrum
www: http://pieter.vanoostrum.org/
PGP key: [8DAE142BE17999C4]

bug#39075: 28.0.50; Emacs hangs on 100% CPU and grows beyond bounds in shell-mode

[Eli Zaretskii]

> From: Pieter van Oostrum <pieter-l@vanoostrum.org>
> Cc: 39075@debbugs.gnu.org
> Date: Sun, 12 Jan 2020 18:11:22 +0100
> 
> I ran it under gdb, and interrupted it several times with C-z in gdb. Most of the stack traces were in the garbage collector, suggesting that it is collecting like crazy. This doesn't surprise me, as it is constantly allocating new memory. The rest of this stack trace doesn't have useful information.
> 
> I managed to get a stack trace where it is processing. I haven't analysed these yet, but I include both here.

Thanks, this was very useful.  It turns out to reproduce one must do
this at the shell's prompt, after "M-x shell":

  $ cd /some/directory/;

The /some/directory/ part should be a real directory.  Once one types
the semi-colon, Emacs hangs.  Here's the Lisp backtrace:

  "Automatic GC" (0x0)
  "looking-at" (0x766f5fc8)
  "shell--parse-pcomplete-arguments" (0x766f64f8)
  "pcomplete-parse-arguments" (0x766f6a90)
  "pcomplete-completions" (0x766f6f60)
  "pcomplete-completions-at-point" (0x766f7698)
  "run-hook-with-args-until-success" (0x766f7690)
  "comint-completion-at-point" (0x766f7b10)
  0x317f7b0 PVEC_COMPILED
  "completion-in-region--postch" (0x766f8450)

So I think shell--parse-pcomplete-arguments infloops in this case.

bug#39075: 28.0.50; Emacs hangs on 100% CPU and grows beyond bounds in shell-mode

[Pieter van Oostrum]

Eli Zaretskii <eliz@gnu.org> writes:

> Thanks, this was very useful.  It turns out to reproduce one must do
> this at the shell's prompt, after "M-x shell":
>
>   $ cd /some/directory/;
>
> The /some/directory/ part should be a real directory.  Once one types
> the semi-colon, Emacs hangs.  Here's the Lisp backtrace:
>
>   "Automatic GC" (0x0)
>   "looking-at" (0x766f5fc8)
>   "shell--parse-pcomplete-arguments" (0x766f64f8)
>   "pcomplete-parse-arguments" (0x766f6a90)
>   "pcomplete-completions" (0x766f6f60)
>   "pcomplete-completions-at-point" (0x766f7698)
>   "run-hook-with-args-until-success" (0x766f7690)
>   "comint-completion-at-point" (0x766f7b10)
>   0x317f7b0 PVEC_COMPILED
>   "completion-in-region--postch" (0x766f8450)
>
> So I think shell--parse-pcomplete-arguments infloops in this case.

Yes, I have traced it.

shell--parse-pcomplete-arguments splits the line into chunk, each chunk being either
1) a sequence of chars not containing white space, \ " ' ;
2) a sequence of chars between apostrophes (') not containing ', maybe final one missing
3) a sequence of chars between quotes (") not containing unescaped ", maybe final one missing
4) a backslash (\) possibly followed by a char

It uses a regexp for that.
It collects a list of these chunks.
It skips over white space between the chunks.

The problem is that a semicolon ; is not covered by this regexp. In the case that caused the error the end position was after the semicolon but the match loop stopped just before the semicolon, and would never advance. It kept going in an infinite loop, continually pushing empty strings on the result list, thereby exhausting memory, and using 100% CPU time.
I don't know why it wouldn't react to C-g, but I guess because after some time it would be mostly in the garbage collector.

Originally case 1) did not have the semicolon.
It was introduced in commit eaeeece92da51b517097667f13d580aa92ad5d59 on Dec 4, 2018 18:39:47 +0100.
There was a test case added in test/lisp/shell-tests.el, but it cheated by positioning point before the semicolon.

I think the simplest solution is to add a semicolon to the part where it skips over white space, i.e. treat the semicolon like white space. But I am not wholly sure that the caller doesn't want to see the semicolon. Otherwise the semicolon should be pushed to the result.

diff -u /Users/pieter/TEMP/shell.el.\~1\~ /Users/pieter/TEMP/shell.el
--- /Users/pieter/TEMP/shell.el.~1~	2020-01-13 14:37:40.000000000 +0100
+++ /Users/pieter/TEMP/shell.el	2020-01-13 14:38:07.000000000 +0100
@@ -428,7 +428,7 @@
     (save-excursion
       (goto-char begin)
       (while (< (point) end)
-	(skip-chars-forward " \t\n")
+	(skip-chars-forward " \t\n;")
 	(push (point) begins)
         (let ((arg ()))
           (while (looking-at

Diff finished.  Mon Jan 13 14:38:22 2020

-- 
Pieter van Oostrum
www: http://pieter.vanoostrum.org/
PGP key: [8DAE142BE17999C4]

bug#39075: 28.0.50; Emacs hangs on 100% CPU and grows beyond bounds in shell-mode

[Michael Welsh Duggan]

This seems very similar to the bug I reported at bug#38549.  Could you
see if this also fixes that recipe and, if so, merge the bugs?

-- 
Michael Welsh Duggan
(md5i@md5i.com)

bug#39075: 28.0.50; Emacs hangs on 100% CPU and grows beyond bounds in shell-mode

[Pieter van Oostrum]

Michael Welsh Duggan <mwd@md5i.com> writes:

> This seems very similar to the bug I reported at bug#38549.  Could you
> see if this also fixes that recipe and, if so, merge the bugs?
>
Yes, the same fix solves that bug also. It is the same bug.
-- 
Pieter van Oostrum
www: http://pieter.vanoostrum.org/
PGP key: [8DAE142BE17999C4]

bug#39075: 28.0.50; Emacs hangs on 100% CPU and grows beyond bounds in shell-mode

[Pieter van Oostrum]

[-- Attachment #1: Type: text/plain, Size: 594 bytes --]

Pieter van Oostrum <pieter-l@vanoostrum.org> writes:

> Michael Welsh Duggan <mwd@md5i.com> writes:
>
>> This seems very similar to the bug I reported at bug#38549.  Could you
>> see if this also fixes that recipe and, if so, merge the bugs?
>>
> Yes, the same fix solves that bug also. It is the same bug.

I propose to amend the test also, so that it would have caught this error (by hanging). I add both patches here now. I have tested that all other things worked the same and found no problems.

Is there an official procedure for requesting the change, or is posting it here sufficient?


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: shell.patch --]
[-- Type: text/x-patch, Size: 867 bytes --]

diff --git a/lisp/shell.el b/lisp/shell.el
index 98e830ee49..ecebf937e2 100644
--- a/lisp/shell.el
+++ b/lisp/shell.el
@@ -428,7 +428,7 @@ shell--parse-pcomplete-arguments
     (save-excursion
       (goto-char begin)
       (while (< (point) end)
-	(skip-chars-forward " \t\n")
+	(skip-chars-forward " \t\n;")
 	(push (point) begins)
         (let ((arg ()))
           (while (looking-at
diff --git a/test/lisp/shell-tests.el b/test/lisp/shell-tests.el
index 6d262f8e7c..7113cb941c 100644
--- a/test/lisp/shell-tests.el
+++ b/test/lisp/shell-tests.el
@@ -34,8 +34,7 @@ shell-tests-completion-before-semi
   (with-temp-buffer
     (shell-mode)
     (insert "cd ba;")
-    (forward-char -1)
     (should (equal (shell--parse-pcomplete-arguments)
-                   '(("cd" "ba") 1 4)))))
+                   '(("cd" "ba" "") 1 4)))))
 
 ;;; shell-tests.el ends here

[-- Attachment #3: Type: text/plain, Size: 87 bytes --]


-- 
Pieter van Oostrum
www: http://pieter.vanoostrum.org/
PGP key: [8DAE142BE17999C4]

bug#39075: 28.0.50; Emacs hangs on 100% CPU and grows beyond bounds in shell-mode

[Pieter van Oostrum]

[-- Attachment #1: Type: text/plain, Size: 3260 bytes --]

Pieter van Oostrum <pieter-l@vanoostrum.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> Thanks, this was very useful.  It turns out to reproduce one must do
>> this at the shell's prompt, after "M-x shell":
>>
>>   $ cd /some/directory/;
>>
>> The /some/directory/ part should be a real directory.  Once one types
>> the semi-colon, Emacs hangs.  Here's the Lisp backtrace:
>>
>>   "Automatic GC" (0x0)
>>   "looking-at" (0x766f5fc8)
>>   "shell--parse-pcomplete-arguments" (0x766f64f8)
>>   "pcomplete-parse-arguments" (0x766f6a90)
>>   "pcomplete-completions" (0x766f6f60)
>>   "pcomplete-completions-at-point" (0x766f7698)
>>   "run-hook-with-args-until-success" (0x766f7690)
>>   "comint-completion-at-point" (0x766f7b10)
>>   0x317f7b0 PVEC_COMPILED
>>   "completion-in-region--postch" (0x766f8450)
>>
>> So I think shell--parse-pcomplete-arguments infloops in this case.
>
> Yes, I have traced it.
>
> shell--parse-pcomplete-arguments splits the line into chunk, each chunk being either
> 1) a sequence of chars not containing white space, \ " ' ;
> 2) a sequence of chars between apostrophes (') not containing ', maybe final one missing
> 3) a sequence of chars between quotes (") not containing unescaped ", maybe final one missing
> 4) a backslash (\) possibly followed by a char
>
> It uses a regexp for that.
> It collects a list of these chunks.
> It skips over white space between the chunks.
>
> The problem is that a semicolon ; is not covered by this regexp. In the case that caused the error the end position was after the semicolon but the match loop stopped just before the semicolon, and would never advance. It kept going in an infinite loop, continually pushing empty strings on the result list, thereby exhausting memory, and using 100% CPU time.
> I don't know why it wouldn't react to C-g, but I guess because after some time it would be mostly in the garbage collector.
>
> Originally case 1) did not have the semicolon.
> It was introduced in commit eaeeece92da51b517097667f13d580aa92ad5d59 on Dec 4, 2018 18:39:47 +0100.
> There was a test case added in test/lisp/shell-tests.el, but it cheated by positioning point before the semicolon.
>
> I think the simplest solution is to add a semicolon to the part where it skips over white space, i.e. treat the semicolon like white space. But I am not wholly sure that the caller doesn't want to see the semicolon. Otherwise the semicolon should be pushed to the result.
>
> diff -u /Users/pieter/TEMP/shell.el.\~1\~ /Users/pieter/TEMP/shell.el
> --- /Users/pieter/TEMP/shell.el.~1~	2020-01-13 14:37:40.000000000 +0100
> +++ /Users/pieter/TEMP/shell.el	2020-01-13 14:38:07.000000000 +0100
> @@ -428,7 +428,7 @@
>      (save-excursion
>        (goto-char begin)
>        (while (< (point) end)
> -	(skip-chars-forward " \t\n")
> +	(skip-chars-forward " \t\n;")
>  	(push (point) begins)
>          (let ((arg ()))
>            (while (looking-at
>
> Diff finished.  Mon Jan 13 14:38:22 2020


I propose to amend the test also, so that it would have caught this
error (by hanging). I add both patches here now. I have tested that all
other things worked the same and found no problems.

Is there an official procedure for requesting the change, or is posting it here sufficient?


[-- Attachment #2: patch --]
[-- Type: text/plain, Size: 867 bytes --]

diff --git a/lisp/shell.el b/lisp/shell.el
index 98e830ee49..ecebf937e2 100644
--- a/lisp/shell.el
+++ b/lisp/shell.el
@@ -428,7 +428,7 @@ shell--parse-pcomplete-arguments
     (save-excursion
       (goto-char begin)
       (while (< (point) end)
-	(skip-chars-forward " \t\n")
+	(skip-chars-forward " \t\n;")
 	(push (point) begins)
         (let ((arg ()))
           (while (looking-at
diff --git a/test/lisp/shell-tests.el b/test/lisp/shell-tests.el
index 6d262f8e7c..7113cb941c 100644
--- a/test/lisp/shell-tests.el
+++ b/test/lisp/shell-tests.el
@@ -34,8 +34,7 @@ shell-tests-completion-before-semi
   (with-temp-buffer
     (shell-mode)
     (insert "cd ba;")
-    (forward-char -1)
     (should (equal (shell--parse-pcomplete-arguments)
-                   '(("cd" "ba") 1 4)))))
+                   '(("cd" "ba" "") 1 4)))))
 
 ;;; shell-tests.el ends here

[-- Attachment #3: Type: text/plain, Size: 87 bytes --]


-- 
Pieter van Oostrum
www: http://pieter.vanoostrum.org/
PGP key: [8DAE142BE17999C4]

bug#39075: 28.0.50; Emacs hangs on 100% CPU and grows beyond bounds in shell-mode

[Eli Zaretskii]

> From: Pieter van Oostrum <pieter-l@vanoostrum.org>
> Date: Thu, 16 Jan 2020 20:21:37 +0100
> Cc: 39075@debbugs.gnu.org
> 
> Pieter van Oostrum <pieter-l@vanoostrum.org> writes:
> 
> > Michael Welsh Duggan <mwd@md5i.com> writes:
> >
> >> This seems very similar to the bug I reported at bug#38549.  Could you
> >> see if this also fixes that recipe and, if so, merge the bugs?
> >>
> > Yes, the same fix solves that bug also. It is the same bug.
> 
> I propose to amend the test also, so that it would have caught this error (by hanging). I add both patches here now. I have tested that all other things worked the same and found no problems.

Thanks, pushed to the release branch.

> Is there an official procedure for requesting the change, or is posting it here sufficient?

It is sufficient to post here, but:

 . please in the future provide a ChangeLog-style commit log message
   (see CONTRIBUTE for more about this);
 . it looks like the disclaimer of your employer has expired several
   years ago, so if you want to continue contributing to Emacs, I
   suggest to start/renew your legal paperwork.

bug#39075: 28.0.50; Emacs hangs on 100% CPU and grows beyond bounds in shell-mode

[Glenn Morris]

This causes a test failure for me on CentOS 8.1.
(BTW, the bug# in the commit log has a typo, but obviosuly nothing can
be done about that.)


Test shell-tests-completion-before-semi backtrace:
  signal(ert-test-failed (((should (equal (shell--parse-pcomplete-argu
  ert-fail(((should (equal (shell--parse-pcomplete-arguments) '(("cd" 
  #f(compiled-function () #<bytecode 0x45cfa9>)()
  ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test
  ert-run-test(#s(ert-test :name shell-tests-completion-before-semi :d
  ert-run-or-rerun-test(#s(ert--stats :selector (not (or (tag :expensi
  ert-run-tests((not (or (tag :expensive-test) (tag :unstable))) #f(co
  ert-run-tests-batch((not (or (tag :expensive-test) (tag :unstable)))
  ert-run-tests-batch-and-exit((not (or (tag :expensive-test) (tag :un
  eval((ert-run-tests-batch-and-exit '(not (or (tag :expensive-test) (
  command-line-1(("-L" ":." "-l" "ert" "-l" "lisp/shell-tests" "--eval
  command-line()
  normal-top-level()
Test shell-tests-completion-before-semi condition:
    (ert-test-failed
     ((should
       (equal
	(shell--parse-pcomplete-arguments)
	'...))
      :form
      (equal
       (("cd" "ba" "")
	1 4 7)
       (("cd" "ba" "")
	1 4))
      :value nil :explanation
      (proper-lists-of-different-length 4 3
					(("cd" "ba" "")
					 1 4 7)
					(("cd" "ba" "")
					 1 4)
					first-mismatch-at 3)))
   FAILED  1/2  shell-tests-completion-before-semi (0.000774 sec)

bug#39075: 28.0.50; Emacs hangs on 100% CPU and grows beyond bounds in shell-mode

[Glenn Morris]

Pieter van Oostrum wrote:

> I propose to amend the test also, so that it would have caught this
> error (by hanging).

I haven't read anything else, so take this with a grain of salt, but
tests that hang (rather than just fail) are a PITA for automated testing.

bug#39075: 28.0.50; Emacs hangs on 100% CPU and grows beyond bounds in shell-mode

[Pieter van Oostrum]

Glenn Morris <rgm@gnu.org> writes:

> Pieter van Oostrum wrote:
>
>> I propose to amend the test also, so that it would have caught this
>> error (by hanging).
>
> I haven't read anything else, so take this with a grain of salt, but
> tests that hang (rather than just fail) are a PITA for automated testing.

That is true, but the test is there to test the fix, which makes it no longer hang. And as these are in the same commit, the test should never hang, unless somebody breaks the fix later. But then hanging could occur with any bug.
-- 
Pieter van Oostrum
www: http://pieter.vanoostrum.org/
PGP key: [8DAE142BE17999C4]

bug#39075: 28.0.50; Emacs hangs on 100% CPU and grows beyond bounds in shell-mode

[Pieter van Oostrum]

Glenn Morris <rgm@gnu.org> writes:

> This causes a test failure for me on CentOS 8.1.
> (BTW, the bug# in the commit log has a typo, but obviosuly nothing can
> be done about that.)
>
>
> Test shell-tests-completion-before-semi backtrace:
>   signal(ert-test-failed (((should (equal (shell--parse-pcomplete-argu
>   ert-fail(((should (equal (shell--parse-pcomplete-arguments) '(("cd" 
>   #f(compiled-function () #<bytecode 0x45cfa9>)()
>   ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test
>   ert-run-test(#s(ert-test :name shell-tests-completion-before-semi :d
>   ert-run-or-rerun-test(#s(ert--stats :selector (not (or (tag :expensi
>   ert-run-tests((not (or (tag :expensive-test) (tag :unstable))) #f(co
>   ert-run-tests-batch((not (or (tag :expensive-test) (tag :unstable)))
>   ert-run-tests-batch-and-exit((not (or (tag :expensive-test) (tag :un
>   eval((ert-run-tests-batch-and-exit '(not (or (tag :expensive-test) (
>   command-line-1(("-L" ":." "-l" "ert" "-l" "lisp/shell-tests" "--eval
>   command-line()
>   normal-top-level()
> Test shell-tests-completion-before-semi condition:
>     (ert-test-failed
>      ((should
>        (equal
> 	(shell--parse-pcomplete-arguments)
> 	'...))
>       :form
>       (equal
>        (("cd" "ba" "")
> 	1 4 7)
>        (("cd" "ba" "")
> 	1 4))
>       :value nil :explanation
>       (proper-lists-of-different-length 4 3
> 					(("cd" "ba" "")
> 					 1 4 7)
> 					(("cd" "ba" "")
> 					 1 4)
> 					first-mismatch-at 3)))
>    FAILED  1/2  shell-tests-completion-before-semi (0.000774 sec)

Aahh! My bad. It should have been 1 4 7. I made a copying error.
-- 
Pieter van Oostrum
www: http://pieter.vanoostrum.org/
PGP key: [8DAE142BE17999C4]

bug#39075: 28.0.50; Emacs hangs on 100% CPU and grows beyond bounds in shell-mode

[Michael Albinus]

Pieter van Oostrum <pieter-l@vanoostrum.org> writes:

>>> I propose to amend the test also, so that it would have caught this
>>> error (by hanging).
>>
>> I haven't read anything else, so take this with a grain of salt, but
>> tests that hang (rather than just fail) are a PITA for automated testing.
>
> That is true, but the test is there to test the fix, which makes it no
> longer hang. And as these are in the same commit, the test should
> never hang, unless somebody breaks the fix later. But then hanging
> could occur with any bug.

I haven't checked your test case, but in Tramp I try to avoid hanging by
wrapping process related tests with a timer.

Experience has taught me, that I'm able to break the tests all the time
by new code :-(

Best regards, Michael.

bug#39075: 28.0.50; Emacs hangs on 100% CPU and grows beyond bounds in shell-mode

[Pieter van Oostrum]

Michael Albinus <michael.albinus@gmx.de> writes:

> Pieter van Oostrum <pieter-l@vanoostrum.org> writes:
>
>>>> I propose to amend the test also, so that it would have caught this
>>>> error (by hanging).
>>>
>>> I haven't read anything else, so take this with a grain of salt, but
>>> tests that hang (rather than just fail) are a PITA for automated testing.
>>
>> That is true, but the test is there to test the fix, which makes it no
>> longer hang. And as these are in the same commit, the test should
>> never hang, unless somebody breaks the fix later. But then hanging
>> could occur with any bug.
>
> I haven't checked your test case, but in Tramp I try to avoid hanging by
> wrapping process related tests with a timer.
>
> Experience has taught me, that I'm able to break the tests all the time
> by new code :-(
>
If you know a better test to check for this particular fix, I would be happy if you could let me know. I don't know if some timer code would help.
-- 
Pieter van Oostrum
www: http://pieter.vanoostrum.org/
PGP key: [8DAE142BE17999C4]

bug#39075: 28.0.50; Emacs hangs on 100% CPU and grows beyond bounds in shell-mode

[Michael Albinus]

Pieter van Oostrum <pieter-l@vanoostrum.org> writes:

>>> That is true, but the test is there to test the fix, which makes it no
>>> longer hang. And as these are in the same commit, the test should
>>> never hang, unless somebody breaks the fix later. But then hanging
>>> could occur with any bug.
>>
>> I haven't checked your test case, but in Tramp I try to avoid hanging by
>> wrapping process related tests with a timer.
>>
>> Experience has taught me, that I'm able to break the tests all the time
>> by new code :-(
>>
> If you know a better test to check for this particular fix, I would be
> happy if you could let me know. I don't know if some timer code would
> help.

Which test do you mean? There is shell-tests-completion-before-semi, but
this doesn't use a process.

Best regards, Michael.

bug#39075: 28.0.50; Emacs hangs on 100% CPU and grows beyond bounds in shell-mode

[Mattias Engdegård]

At the very least, a test named 'shell-tests-completion-before-semi' should not be changed into testing completion after a semicolon. I took the liberty to fixing that, and the erroneous test value that made the changed test fail.