bug#44809: One very long line of <中文> XML tags puts emacs out of business

bug#44809: One very long line of <中文> XML tags puts emacs out of business

[積丹尼 Dan Jacobson]

1. Visit https://data.gov.tw/dataset/34315 .
2. Download e.g., B臺中市 .
3. Unzip, and open one of the more whopping XML files,

$ ls -ogS Downloads/B_XML/ | head -n 5
-rw-r--r-- 1 3091875 2020-04-13  b_4602.xml
-rw-r--r-- 1 3087978 2020-04-13  b_0755.xml
-rw-r--r-- 1 2874539 2020-04-13  b_5709.xml
-rw-r--r-- 1 2829385 2020-04-13  b_0530.xml

Press M-> [end-of-buffer]. Now press M-<.

You'll see the combination of one very long line, UTF-8 tags (<國有>),
etc. causes emacs to, well, only be able to process C-g's and beep.

Before long one can only put it out of its misery with
$ killall -1 emacs.
emacs-version "27.1"

bug#44818: 27.0.91; wedged

[Devon Sean McCullough]

Having fat-fingered it in dired, I inadvertently opened a large file
with no newlines.  That Emacs instance has been burning 100% CPU all
day.  I can interrupt and single step it in llbd from another Emacs.
Is there any way to unwedge Emacs?  E.g., would forcing read_char to
return Qnil, Qt or something cause corruption?  Would invoking, say,
Fbury_buffer_internal (Fcurrent_buffer ()) regain control or blow it
up?  I'll leave it overnight in case it reads the ^G^G^Xk^M I typed.

		Peace
			--Devon

P.S. Obviously, long stretches of non-newlines wedge Emacs for ages,
because redisplay assumes there are no long lines.  Perhaps the docs
mention some workaround I missed?  Redisplay has been buggy for over
a year now, glitchy blank windows, etc., but that's not today's bug.

(lldb) process attach --pid 1361

Process 1361 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
     frame #0: 0x00007fff7eedf21a libsystem_kernel.dylib`mach_msg_trap + 10
libsystem_kernel.dylib`mach_msg_trap:
->  0x7fff7eedf21a <+10>: retq
     0x7fff7eedf21b <+11>: nop

libsystem_kernel.dylib`mach_msg_overwrite_trap:
     0x7fff7eedf21c <+0>:  movq   %rcx, %r10
     0x7fff7eedf21f <+3>:  movl   $0x1000020, %eax          ; imm = 
0x1000020
Target 0: (Emacs-x86_64-10_14) stopped.

Executable module set to 
"/Applications/Emacs-27.0.91.app/Contents/MacOS/Emacs-x86_64-10_14".
Architecture set to: x86_64h-apple-macosx-.
(lldb) Process 1361 resuming
(lldb) error: Process is running.  Use 'process interrupt' to pause 
execution.
(lldb) error: Process is running.  Use 'process interrupt' to pause 
execution.
(lldb) process interrupt
(lldb) Process 1361 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
     frame #0: 0x00000001037bb087 
Emacs-x86_64-10_14`bidi_find_bracket_pairs + 1319
Emacs-x86_64-10_14`bidi_find_bracket_pairs:
->  0x1037bb087 <+1319>: leaq   0x5d6862(%rip), %rax      ; globals
     0x1037bb08e <+1326>: movb   0xe2a(%rax), %al
     0x1037bb094 <+1332>: testb  %al, %al
     0x1037bb096 <+1334>: jne    0x1037bacf0               ; <+400>
Target 0: (Emacs-x86_64-10_14) stopped.
bt
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
   * frame #0: 0x00000001037bb087 
Emacs-x86_64-10_14`bidi_find_bracket_pairs + 1319
     frame #1: 0x00000001037ba8ff 
Emacs-x86_64-10_14`bidi_resolve_brackets + 959
     frame #2: 0x00000001037b8236 
Emacs-x86_64-10_14`bidi_level_of_next_char + 358
     frame #3: 0x00000001037b7a07 
Emacs-x86_64-10_14`bidi_move_to_visually_next + 439
     frame #4: 0x000000010372fb4f 
Emacs-x86_64-10_14`set_iterator_to_next + 703
     frame #5: 0x000000010373110b 
Emacs-x86_64-10_14`move_it_in_display_line_to + 3835
     frame #6: 0x000000010372e60f Emacs-x86_64-10_14`move_it_to + 783
     frame #7: 0x00000001037371b9 
Emacs-x86_64-10_14`move_it_vertically_backward + 1273
     frame #8: 0x0000000103769fe7 Emacs-x86_64-10_14`redisplay_window + 
17735
     frame #9: 0x0000000103765a96 Emacs-x86_64-10_14`redisplay_window_0 + 38
     frame #10: 0x000000010385bbef 
Emacs-x86_64-10_14`internal_condition_case_1 + 271
     frame #11: 0x000000010376516c Emacs-x86_64-10_14`redisplay_windows 
+ 140
     frame #12: 0x000000010373b706 Emacs-x86_64-10_14`redisplay_internal 
+ 4886
     frame #13: 0x00000001037d8b15 Emacs-x86_64-10_14`read_char + 2213
     frame #14: 0x00000001037d66da Emacs-x86_64-10_14`read_key_sequence 
+ 1722
     frame #15: 0x00000001037d4edc Emacs-x86_64-10_14`command_loop_1 + 1340
     frame #16: 0x000000010385bab7 
Emacs-x86_64-10_14`internal_condition_case + 263
     frame #17: 0x00000001037e5050 Emacs-x86_64-10_14`command_loop_2 + 48
     frame #18: 0x000000010385b2db Emacs-x86_64-10_14`internal_catch + 267
     frame #19: 0x0000000103927355 
Emacs-x86_64-10_14`command_loop.cold.1 + 69
     frame #20: 0x00000001037d3fa3 Emacs-x86_64-10_14`command_loop + 131
     frame #21: 0x00000001037d3ed3 Emacs-x86_64-10_14`recursive_edit_1 + 115
     frame #22: 0x00000001037d412b Emacs-x86_64-10_14`Frecursive_edit + 347
     frame #23: 0x00000001037d2d07 Emacs-x86_64-10_14`main + 7431
     frame #24: 0x00007fff7edaa3d5 libdyld.dylib`start + 1
     frame #25: 0x00007fff7edaa3d5 libdyld.dylib`start + 1
(lldb) continue
Process 1361 resuming
(lldb) process interrupt
(lldb) Process 1361 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
     frame #0: 0x00000001039152a0 
Emacs-x86_64-10_14`macfont_get_glyph_for_character + 32
Emacs-x86_64-10_14`macfont_get_glyph_for_character:
->  0x1039152a0 <+32>: movq   %rax, -0x30(%rbp)
     0x1039152a4 <+36>: movq   0xd8(%rdi), %r15
     0x1039152ab <+43>: movq   0xf0(%rdi), %r12
     0x1039152b2 <+50>: cmpl   $0xd800, %esi             ; imm = 0xD800
Target 0: (Emacs-x86_64-10_14) stopped.
bt
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
   * frame #0: 0x00000001039152a0 
Emacs-x86_64-10_14`macfont_get_glyph_for_character + 32
     frame #1: 0x0000000103911094 Emacs-x86_64-10_14`macfont_encode_char 
+ 20
     frame #2: 0x0000000103749bdc Emacs-x86_64-10_14`gui_produce_glyphs 
+ 2428
     frame #3: 0x00000001037308c5 
Emacs-x86_64-10_14`move_it_in_display_line_to + 1717
     frame #4: 0x000000010372e60f Emacs-x86_64-10_14`move_it_to + 783
     frame #5: 0x00000001037371b9 
Emacs-x86_64-10_14`move_it_vertically_backward + 1273
     frame #6: 0x0000000103769fe7 Emacs-x86_64-10_14`redisplay_window + 
17735
     frame #7: 0x0000000103765a96 Emacs-x86_64-10_14`redisplay_window_0 + 38
     frame #8: 0x000000010385bbef 
Emacs-x86_64-10_14`internal_condition_case_1 + 271
     frame #9: 0x000000010376516c Emacs-x86_64-10_14`redisplay_windows + 140
     frame #10: 0x000000010373b706 Emacs-x86_64-10_14`redisplay_internal 
+ 4886
     frame #11: 0x00000001037d8b15 Emacs-x86_64-10_14`read_char + 2213
     frame #12: 0x00000001037d66da Emacs-x86_64-10_14`read_key_sequence 
+ 1722
     frame #13: 0x00000001037d4edc Emacs-x86_64-10_14`command_loop_1 + 1340
     frame #14: 0x000000010385bab7 
Emacs-x86_64-10_14`internal_condition_case + 263
     frame #15: 0x00000001037e5050 Emacs-x86_64-10_14`command_loop_2 + 48
     frame #16: 0x000000010385b2db Emacs-x86_64-10_14`internal_catch + 267
     frame #17: 0x0000000103927355 
Emacs-x86_64-10_14`command_loop.cold.1 + 69
     frame #18: 0x00000001037d3fa3 Emacs-x86_64-10_14`command_loop + 131
     frame #19: 0x00000001037d3ed3 Emacs-x86_64-10_14`recursive_edit_1 + 115
     frame #20: 0x00000001037d412b Emacs-x86_64-10_14`Frecursive_edit + 347
     frame #21: 0x00000001037d2d07 Emacs-x86_64-10_14`main + 7431
     frame #22: 0x00007fff7edaa3d5 libdyld.dylib`start + 1
     frame #23: 0x00007fff7edaa3d5 libdyld.dylib`start + 1
(lldb) continue
Process 1361 resuming
(lldb)

In GNU Emacs 27.0.91 (build 1, x86_64-apple-darwin18.7.0, NS 
appkit-1671.60 Version 10.14.6 (Build 18G6042))
  of 2020-11-22 built on la
Windowing system distributor 'Apple', version 10.3.1671
System Description:  Mac OS X 10.14.6

Recent messages:
History item: 0
History item: 1
History item: 2
History item: 3
History item: 4
History item: 5
History item: 6
History item: 7
History item: 8
History item: 9

Configured using:
  'configure --with-ns '--enable-locallisppath=/Library/Application
  Support/Emacs/${version}/site-lisp:/Library/Application
  Support/Emacs/site-lisp' --with-modules'

Configured features:
NOTIFY KQUEUE ACL GNUTLS LIBXML2 ZLIB TOOLKIT_SCROLL_BARS NS MODULES
THREADS PDUMPER

Important settings:
   value of $LANG: en_BE@currency=USD.UTF-8
   locale-coding-system: utf-8-unix

Major mode: Lisp

Minor modes in effect:
   display-time-mode: t
   slime-trace-dialog-minor-mode: t
   slime-autodoc-mode: t
   slime-mode: t
   gud-tooltip-mode: t
   which-function-mode: t
   shell-dirtrack-mode: t
   tooltip-mode: t
   global-eldoc-mode: t
   eldoc-mode: t
   electric-indent-mode: t
   mouse-wheel-mode: t
   menu-bar-mode: t
   file-name-shadow-mode: t
   global-font-lock-mode: t
   font-lock-mode: t
   blink-cursor-mode: t
   auto-composition-mode: t
   auto-encryption-mode: t
   auto-compression-mode: t
   temp-buffer-resize-mode: t
   line-number-mode: t
   transient-mark-mode: t

Load-path shadows:
/Users/devon/quicklisp/dists/quicklisp/software/slime-v2.24/slime-autoloads 
hides /Users/devon/.emacs.d/elpa/slime-20200711.1419/slime-autoloads
/Users/devon/quicklisp/dists/quicklisp/software/slime-v2.24/slime-tests 
hides /Users/devon/.emacs.d/elpa/slime-20200711.1419/slime-tests
/Users/devon/quicklisp/dists/quicklisp/software/slime-v2.24/slime hides 
/Users/devon/.emacs.d/elpa/slime-20200711.1419/slime
/Applications/Emacs-27.0.91.app/Contents/Resources/lisp/frame hides 
/Users/devon/emacs/frame
/Applications/Emacs-27.0.91.app/Contents/Resources/lisp/emacs-lisp/cl-macs 
hides /Users/devon/emacs/cl-macs
/Applications/Emacs-27.0.91.app/Contents/Resources/lisp/mail/sendmail 
hides /Users/devon/emacs/sendmail
/Applications/Emacs-27.0.91.app/Contents/Resources/lisp/forms hides 
/Users/devon/emacs/forms
/Applications/Emacs-27.0.91.app/Contents/Resources/lisp/mail/hashcash 
hides /Users/devon/emacs/hashcash
/Applications/Emacs-27.0.91.app/Contents/Resources/lisp/progmodes/inf-lisp 
hides /Users/devon/emacs/inf-lisp
/Applications/Emacs-27.0.91.app/Contents/Resources/lisp/dired-aux hides 
/Users/devon/emacs/dired-aux
/Applications/Emacs-27.0.91.app/Contents/Resources/lisp/json hides 
/Users/devon/emacs/json
/Applications/Emacs-27.0.91.app/Contents/Resources/lisp/net/shr hides 
/Users/devon/emacs/shr
/Applications/Emacs-27.0.91.app/Contents/Resources/lisp/replace hides 
/Users/devon/emacs/replace
/Applications/Emacs-27.0.91.app/Contents/Resources/lisp/textmodes/sgml-mode 
hides /Users/devon/emacs/sgml-mode
/Users/devon/quicklisp/dists/quicklisp/software/slime-v2.24/slime hides 
/Users/devon/emacs/slime
/Applications/Emacs-27.0.91.app/Contents/Resources/lisp/comint hides 
/Users/devon/emacs/comint
/Applications/Emacs-27.0.91.app/Contents/Resources/lisp/play/morse hides 
/Users/devon/emacs/morse
/Users/devon/.emacs.d/elpa/slime-repl-ansi-color-20190426.1414/slime-repl-ansi-color 
hides /Users/devon/emacs/slime-repl-ansi-color
/Applications/Emacs-27.0.91.app/Contents/Resources/lisp/xml hides 
/Users/devon/emacs/xml
/Applications/Emacs-27.0.91.app/Contents/Resources/lisp/emacs-lisp/regexp-opt 
hides /Users/devon/emacs/regexp-opt
/Applications/Emacs-27.0.91.app/Contents/Resources/lisp/emacs-lisp/advice 
hides /Users/devon/emacs/advice
/Applications/Emacs-27.0.91.app/Contents/Resources/lisp/emacs-lisp/lisp 
hides /Users/devon/emacs/lisp
/Users/devon/.emacs.d/elpa/csv-mode-1.7/csv-mode hides 
/Users/devon/emacs/csv-mode
/Applications/Emacs-27.0.91.app/Contents/Resources/lisp/textmodes/picture 
hides /Users/devon/emacs/picture
/Applications/Emacs-27.0.91.app/Contents/Resources/lisp/progmodes/xref 
hides /Users/devon/emacs/xref
/Applications/Emacs-27.0.91.app/Contents/Resources/lisp/rect hides 
/Users/devon/emacs/rect
/Applications/Emacs-27.0.91.app/Contents/Resources/lisp/lpr hides 
/Users/devon/emacs/lpr
/Applications/Emacs-27.0.91.app/Contents/Resources/lisp/net/tramp-ftp 
hides /Users/devon/emacs/tramp-ftp

Features:
(shadow mail-extr emacsbug message rfc822 mml mml-sec epa derived epg
epg-config gnus-util rmail rmail-loaddefs text-property-search mm-decode
mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils
pcmpl-unix eieio-opt speedbar sb-image ezimage dframe lisp-cycle
jka-compr gcl-info server tabify man macrostep-c cmacexp libgl-doc
cc-mode cc-fonts cc-guess cc-menus cc-cmds cc-styles cc-align cc-engine
cc-vars cc-defs tramp-cache tramp-sh tramp tramp-loaddefs trampver
tramp-integration files-x tramp-compat parse-time iso8601 ls-lisp
format-spec cl-indent vc-git markdown-mode network-stream puny nsm rmc
slime-media slime-repl-ansi-color slime-fancy slime-trace-dialog
slime-fontifying-fu slime-package-fu slime-references
slime-compiler-notes-tree slime-scratch slime-presentations bridge
slime-macrostep macrostep slime-mdot-fu slime-enclosing-context
slime-fuzzy slime-fancy-trace slime-fancy-inspector slime-c-p-c
slime-editing-commands slime-autodoc slime-repl elp slime-parse slime
lisp-mnt gud apropos compile arc-mode archive-mode noutline outline pp
hyperspec view color rect shell pcomplete bucky macosx sort etags
fileloop generator xref project compare-w diff-mode easy-mmode paren
advice sgml-mode dom info-look ispell disp-table edmacro kmacro supersub
rx comint ansi-color ring dired-aux dired dired-loaddefs time-date
misearch multi-isearch cl-extra thingatpt help-fns radix-tree cl-print
debug finder-inf backtrace help-mode find-func appt diary-lib
diary-loaddefs cal-menu calendar cal-loaddefs cl info slime-autoloads
package easymenu browse-url url-handlers url-parse auth-source cl-seq
eieio eieio-core cl-macs eieio-loaddefs password-cache json subr-x map
url-vars seq byte-opt gv bytecomp byte-compile cconv cl-loaddefs cl-lib
tooltip cus-start eldoc electric uniquify ediff-hook vc-hooks
lisp-float-type mwheel term/ns-win ns-win ucs-normalize mule-util
term/common-win tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame minibuffer cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite charscript charprop case-table epa-hook
jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice loaddefs
button faces cus-face macroexp files text-properties overlay sha1 md5
base64 format env code-pages mule custom widget hashtable-print-readable
backquote threads kqueue cocoa ns multi-tty make-network-process emacs)

Memory information:
((conses 16 908857 112639)
  (symbols 48 44923 2)
  (strings 32 191673 14229)
  (string-bytes 1 6084677)
  (vectors 16 79173)
  (vector-slots 8 1780939 139134)
  (floats 8 539 306)
  (intervals 56 47437 1195)
  (buffers 1000 76))

bug#44818: 27.0.91; wedged

[Eli Zaretskii]

> From: Devon Sean McCullough <Devon2020@jovi.net>
> Date: Mon, 23 Nov 2020 00:07:10 -0500
> 
> Having fat-fingered it in dired, I inadvertently opened a large file
> with no newlines.  That Emacs instance has been burning 100% CPU all
> day.  I can interrupt and single step it in llbd from another Emacs.
> Is there any way to unwedge Emacs?  E.g., would forcing read_char to
> return Qnil, Qt or something cause corruption?  Would invoking, say,
> Fbury_buffer_internal (Fcurrent_buffer ()) regain control or blow it
> up?  I'll leave it overnight in case it reads the ^G^G^Xk^M I typed.

Try this:

  C-g M-<

This will probably take some time to come through, but once it does,
you will see the very beginning of the file, and should be able to
kill the buffer with "C-x k RET".

> P.S. Obviously, long stretches of non-newlines wedge Emacs for ages,
> because redisplay assumes there are no long lines.  Perhaps the docs
> mention some workaround I missed?  Redisplay has been buggy for over
> a year now, glitchy blank windows, etc., but that's not today's bug.

When you visit such a long file in Emacs 27.1, you should see a
suggestion to visit it literally; take it.

A more general solution is to turn on so-long mode.

You seem to be running a pretest of Emacs 27.1, so maybe these don't
work in your version.

bug#44818: 27.0.91; wedged

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> When you visit such a long file in Emacs 27.1, you should see a
> suggestion to visit it literally; take it.
>
> A more general solution is to turn on so-long mode.

This is such a general problem that people bump into all the time that I
think Emacs should have an even more general solution, and it should be
on by default.

The long-file warning isn't sufficient -- there's plenty of smaller
files that have this problem, too.

Why doesn't Emacs just check for long lines (in the C code, for speed)
when opening files, and offer the "visit literally?" if it detects them?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#44809: One very long line of <中文> XML tags puts emacs out of business

[Lars Ingebrigtsen]

積丹尼 Dan Jacobson <jidanni@jidanni.org> writes:

> Before long one can only put it out of its misery with
> $ killall -1 emacs.

Yes, this is a general Emacs problem.  Consider switching so-long mode
on.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#44818: 27.0.91; wedged

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Devon Sean McCullough <Devon2020@jovi.net>,  44818@debbugs.gnu.org
> Date: Tue, 24 Nov 2020 08:04:16 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > When you visit such a long file in Emacs 27.1, you should see a
> > suggestion to visit it literally; take it.
> >
> > A more general solution is to turn on so-long mode.
> 
> This is such a general problem that people bump into all the time that I
> think Emacs should have an even more general solution, and it should be
> on by default.

We generally intended to turn on so-long-mode by default.  We just
didn't get our act together in time for Emacs 27, as there are a few
bits to get straight before we could make it the default.

> The long-file warning isn't sufficient -- there's plenty of smaller
> files that have this problem, too.

so-long-mode is supposed to detect those cases reliably, not just by
file size, but also by line size and other indications.

> Why doesn't Emacs just check for long lines (in the C code, for speed)
> when opening files, and offer the "visit literally?" if it detects them?

That's what so-long-mode does, AFAIR.  It was designed to solve these
use cases, and AFAIR it does that well enough to not need to invent
yet another similar wheel.

We should try to finish the bits that aren't yet finalized, and turn
it on by default as soon as we can.

bug#44818: 27.0.91; wedged

[Devon Sean McCullough]

On 23/11/2020 10:51, Eli Zaretskii wrote:
> Try this:
> 
>    C-g M-<
> 

After 48 hours of 100% CPU, Emacs finally returns from read_char.
Rather than wait 48 more hours for the next read_char, I disabled
breakpoints, tried

	(lldb) print (long)Fgoto_char((long)Fpoint_min())
	(long) $12 = 6
	(lldb) continue

and was able to kill the buffer manually.  Thanks, Eli!

		Peace
			--Devon

P.S.  Could a watchdog timer safely abort such wedgitude?
Reading a file is not the only way to lose here.

bug#44818: 27.0.91; wedged

[Eli Zaretskii]

> Cc: 44818@debbugs.gnu.org
> From: Devon Sean McCullough <Emacs-hacker2018@jovi.net>
> Date: Tue, 24 Nov 2020 13:42:03 -0500
> 
> P.S.  Could a watchdog timer safely abort such wedgitude?

That would have to run in a separate thread, right?

But what timeout to set it to? some jobs might legitimately take some
time.  And then what to do when the timer expires?

bug#44818: 27.0.91; wedged

[Devon Sean McCullough]

On 24/11/2020 13:48, Eli Zaretskii wrote:
>> Cc: 44818@debbugs.gnu.org
>> From: Devon Sean McCullough <Emacs-hacker2018@jovi.net>
>> Date: Tue, 24 Nov 2020 13:42:03 -0500
>>
>> P.S.  Could a watchdog timer safely abort such wedgitude?
> 
> That would have to run in a separate thread, right?
> 
> But what timeout to set it to? some jobs might legitimately take some
> time.  And then what to do when the timer expires?

The job of redisplay cannot legitimately take > 50 milliseconds.
When the user repeatedly tries ^G quit but Emacs is unresponsive
because of redisplay, such redisplay must be prevented while the
user is offered options to regain control.

		Peace
			--Devon

P.S. Could an invisibility overlay offer temporary relief?
Perhaps a red screen of redisplay death with a short menu?

bug#44818: 27.0.91; wedged

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> Why doesn't Emacs just check for long lines (in the C code, for speed)
>> when opening files, and offer the "visit literally?" if it detects them?
>
> That's what so-long-mode does, AFAIR.  It was designed to solve these
> use cases, and AFAIR it does that well enough to not need to invent
> yet another similar wheel.
>
> We should try to finish the bits that aren't yet finalized, and turn
> it on by default as soon as we can.

I didn't know that so-long was going to be switched on by default -- in
that case, that should indeed solve these problems.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#44818: 27.0.91; wedged

[Andreas Schwab]

On Nov 24 2020, Devon Sean McCullough wrote:

> The job of redisplay cannot legitimately take > 50 milliseconds.

Unless Emacs is busy looking up fonts.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

bug#44818: 27.0.91; wedged

[Devon Sean McCullough]

On 25/11/2020 03:34, Andreas Schwab wrote:
> On Nov 24 2020, Devon Sean McCullough wrote:
> 
>> The job of redisplay cannot legitimately take > 50 milliseconds.
> 
> Unless Emacs is busy looking up fonts.
> If font lookup obstructs timely redisplay, the correct action is to 
display some blurry placeholder and sharpen it as soon as possible.

		Peace
			--Devon

bug#44818: 27.0.91; wedged

[Eli Zaretskii]

> Cc: 44818@debbugs.gnu.org
> From: Devon Sean McCullough <Emacs-hacker2018@jovi.net>
> Date: Tue, 24 Nov 2020 20:35:00 -0500
> 
> On 24/11/2020 13:48, Eli Zaretskii wrote:
> >> Cc: 44818@debbugs.gnu.org
> >> From: Devon Sean McCullough <Emacs-hacker2018@jovi.net>
> >> Date: Tue, 24 Nov 2020 13:42:03 -0500
> >>
> >> P.S.  Could a watchdog timer safely abort such wedgitude?
> > 
> > That would have to run in a separate thread, right?
> > 
> > But what timeout to set it to? some jobs might legitimately take some
> > time.  And then what to do when the timer expires?
> 
> The job of redisplay cannot legitimately take > 50 milliseconds.

It can, and it does.  Although this is quite rare, it does happen.
Here's a simple example: visit src/xdisp.c, then type M-> and measure
the time.

> When the user repeatedly tries ^G quit but Emacs is unresponsive
> because of redisplay, such redisplay must be prevented while the
> user is offered options to regain control.

I asked "what to do when the timer expires" because the practical
meaning of "redisplay must be prevented" is not at all clear.

> P.S. Could an invisibility overlay offer temporary relief?
> Perhaps a red screen of redisplay death with a short menu?

Anything that has to be done by the display code will suffer from the
same problem.

bug#44818: 27.0.91; wedged

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Devon2020@jovi.net,  44818@debbugs.gnu.org
> Date: Wed, 25 Nov 2020 07:57:10 +0100
> 
> > We should try to finish the bits that aren't yet finalized, and turn
> > it on by default as soon as we can.
> 
> I didn't know that so-long was going to be switched on by default -- in
> that case, that should indeed solve these problems.

That was the plan, AFAIR.

Phil, would it be possible to finalize all the minor bits we've
identified last time this was discussed, and then turn so-long on by
default?

bug#44818: 27.0.91; wedged

[Devon Sean McCullough]

Happy Thanksgiving, hackers in the USA!

Will "so long" catch spammed shells?

What if four or more consecutive ^G quits were to select a
"safe" full frame window or dialog offering a few choices,
e.g., ignore the quits and restore the saved window state,
kill the buffer, select another buffer, read the tutorial,
add overlays to appease redisplay or (eek) quit Emacs?

Would a ^G quit discard unprocessed input?  Is it possible
to time out when unprocessed input languishes for > 50 ms,
in a situation where ^G quit is either ignored or fails to
put the user back in the driver's seat?

Perhaps there's no way to detect when the editor is wedged
and the user has lost control.  One clue might be when the
user types ^G quit many times, but was it a sound check of
the beeper's audio output, or a frustrated user attempting
to regain control of a rogue editor?

		Peace
			--Devon

P.S.  Some jobs involve a noticeable delay — not ok for redisplay!

(find-file-noselect "src/xdisp.c")	; 480.5 ms to read the file
(find-file "src/xdisp.c")	; 1.4 ms to display the buffer

Dead, comatose or entranced?  Best show some sign of life!
E.g., tramp is slower than most remote file interfaces but
users are used it.  Its progress reports should be updated
every second, either a timer or the traditional ETA.

bug#44818: 27.0.91; wedged

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Will "so long" catch spammed shells?

Could you explain the problem you're worried about?
What is the scenario?

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#44818: 27.0.91; wedged

[Michael Albinus]

Devon Sean McCullough <Emacs-hacker2018@jovi.net> writes:

Hi,

> Dead, comatose or entranced?  Best show some sign of life!
> E.g., tramp is slower than most remote file interfaces but
> users are used it.  Its progress reports should be updated
> every second, either a timer or the traditional ETA.

Tramp uses Emacs' progress-reporter. However, this is based on timers,
which are fired only at dedicated point, for example during
accept-process-output. If there's no process output, no timer call, no
progress report.

(IIRC, there were problems with Tramp's progress reporter, which have
been fixed beginning of this year in the master branch)

Best regards, Michael.

bug#44809: Warn that current file needs so-long mode

[積丹尼 Dan Jacobson]

retitle 44809 Say "Consider switching so-long mode on" when detecting long line files
thanks

>>>>> "LI" == Lars Ingebrigtsen <larsi@gnus.org> writes:

>> Before long one can only put it out of its misery with
>> $ killall -1 emacs.

LI> Yes, this is a general Emacs problem.  Consider switching so-long mode
LI> on.

OK. SO LONG as emacs also says that automatically,
when detecting such files.
Else the user will never know.
And ends up reporting more bugs than necessary.

Then at least the user will know something is up and won't try ESC >
etc. locking up emacs.

bug#44818: Looks like I retitled both bugs

[積丹尼 Dan Jacobson]

Hmmm, looks like I retitled both bugs. Well, feel free to reretitle them.

bug#44809: Warn that current file needs so-long mode

[Lars Ingebrigtsen]

積丹尼 Dan Jacobson <jidanni@jidanni.org> writes:

> retitle 44809 Say "Consider switching so-long mode on" when detecting
> long line files

The plan is to have so-long mode on by default in Emacs 28, but there
are some things that have to be tweaked a bit first, if I understand
things correctly.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#44818: 27.0.91; wedged

[Devon Sean McCullough]

On 27/11/2020 00:40, Richard Stallman wrote:
> 
>    > Will "so long" catch spammed shells?
> 
> Could you explain the problem you're worried about?
> What is the scenario?

REPL & shell output sometimes wedge redisplay
when lines are very long, same as with files.

		Peace
			--Devon

P.S.  Redisplay should never take > 50 ms.
Perhaps it would have completed in 51 ms
but usually it takes minutes, hours,
days or weeks to complete, running
at 100% CPU with fans roaring.

Rather than set a fifty millisecond timer,
maybe users should be allowed to ^G quit
into a degraded state.  The only other
choice is to force kill the editor
and hope for auto save files.

P.P.S.  When Emacs becomes unresponsive,
i.e., user input is ignored for a while,
could some helpful options be offered,
e.g., redisplay-friendly overlays?

P.P.P.S.  I instrumented a typical scenario.  Emacs went dead
for only 5.6 minutes with point at EoB but when at the middle
of the file, it was unresponsively burning CPU for an hour or
so until I got tired of the fan noise and force-quit the app:
     emacs-version "27.1"
	For information about GNU Emacs and the GNU system, type C-h C-a.
	2841.861010 ms to evaluate (insert-file-contents-literally file)
	0.010967 ms to evaluate (goto-char (point-max)) => 248867275
	0.025988 ms to evaluate (switch-to-buffer buffer)
	336605.978966 ms to evaluate (redisplay 1) => t
	0.002861 ms to evaluate (backward-char) => nil
	0.086069 ms to evaluate (redisplay 2) => t
	0.002861 ms to evaluate (backward-char) => nil
	0.063896 ms to evaluate (redisplay 3) => t
⋮

bug#44818: 27.0.91; wedged

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > P.S.  Redisplay should never take > 50 ms.
  > Perhaps it would have completed in 51 ms
  > but usually it takes minutes, hours,
  > days or weeks to complete, running
  > at 100% CPU with fans roaring.

This might be a good idea.  But the simple way is not an adequate fix.
If all Emacs does is interrupt redisplay, the next redisplay
will run into the same problem and get stuck again.

I wonder if it is possible to detect that a single line has taken too
long, and set a flag to truncate long lines in that buffer.
Perhaps set truncate-lines.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#44818: 27.0.91; wedged

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Date: Tue, 01 Dec 2020 23:32:05 -0500
> Cc: 44818@debbugs.gnu.org
> 
> I wonder if it is possible to detect that a single line has taken too
> long, and set a flag to truncate long lines in that buffer.
> Perhaps set truncate-lines.

That could be too drastic: Emacs becomes painfully slow long after the
number of characters exceeds 80 or 130 or 250 or any other reasonable
line width.  So setting truncate-lines would hide too much from the
view.

Also, setting truncate-lines does not make Emacs fast in all cases,
when very long lines are involved.

bug#44818: 27.0.91; wedged

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > I wonder if it is possible to detect that a single line has taken too
  > > long, and set a flag to truncate long lines in that buffer.
  > > Perhaps set truncate-lines.

  > That could be too drastic: Emacs becomes painfully slow long after the
  > number of characters exceeds 80 or 130 or 250 or any other reasonable
  > line width.  So setting truncate-lines would hide too much from the
  > view.

Sorry, I am don't follow you.  What exactly is the problem?  What action
do you recommend, or discourage?

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#44818: 27.0.91; wedged

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: Emacs-hacker2018@jovi.net, 44818@debbugs.gnu.org
> Date: Thu, 03 Dec 2020 00:29:04 -0500
> 
>   > > I wonder if it is possible to detect that a single line has taken too
>   > > long, and set a flag to truncate long lines in that buffer.
>   > > Perhaps set truncate-lines.
> 
>   > That could be too drastic: Emacs becomes painfully slow long after the
>   > number of characters exceeds 80 or 130 or 250 or any other reasonable
>   > line width.  So setting truncate-lines would hide too much from the
>   > view.
> 
> Sorry, I am don't follow you.  What exactly is the problem?  What action
> do you recommend, or discourage?

You suggested to behave as if truncate-lines is turned on when we
discover a line whose rendering takes too long.  I'm saying that doing
so will prevent users from seeing more than a hundred or so characters
of every line, and hide the rest.  And that might be too drastic a
measure, because Emacs becomes slow on lines much longer than 100 or
200 characters; a single very long line will only make it slow for
that single window-full.  IOW, the punishment is too severe, IMO.

bug#44818: 27.0.91; wedged

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > You suggested to behave as if truncate-lines is turned on when we
  > discover a line whose rendering takes too long.  I'm saying that doing
  > so will prevent users from seeing more than a hundred or so characters
  > of every line, and hide the rest.

Yes, it would.  Wouldn't that be better than what happens now?

  > And that might be too drastic a
  > measure, because Emacs becomes slow on lines much longer than 100 or
  > 200 characters; a single very long line will only make it slow for
  > that single window-full.

I agree. different changes might give a superior result.  I'm not
arguing against that.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#44818: 27.0.91; wedged

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: Emacs-hacker2018@jovi.net, 44818@debbugs.gnu.org
> Date: Fri, 04 Dec 2020 01:01:10 -0500
> 
>   > You suggested to behave as if truncate-lines is turned on when we
>   > discover a line whose rendering takes too long.  I'm saying that doing
>   > so will prevent users from seeing more than a hundred or so characters
>   > of every line, and hide the rest.
> 
> Yes, it would.  Wouldn't that be better than what happens now?

Not in all case, IME.

bug#44818: Say "Consider switching so-long mode on" when detecting long line files

[Lars Ingebrigtsen]

These issues have been fixed by Gregory in Emacs 29 -- Emacs no longer
needs to rely on so-long, but works fine out of the box with these sorts
of files, so I'm closing this bug report.