bug#47589: [PATCH] 27.1; Freeze on long lines in compilation-shell-minor-mode

bug#47589: [PATCH] 27.1; Freeze on long lines in compilation-shell-minor-mode

[Anton Tayanovskyy]

To reproduce the issue of Emacs freezing on long lines, do the
following:

```
M-x shell
M-x compilation-shell-minor-mode
$ python3 -c 'print("x"*1024*8)'
```

Depending on the machine configuration, make your lines longer. 1024*8
is enough to freeze my MacBook Pro 2019 visibly but I need more
characters to slow down an XPS-13 running Ubuntu.

In the real world the issue comes up when I'm enjoying my
compilation-shell-minor-mode buffer but accidentally cat or print some
data with very long lines. Then Emacs freezes and I experience one of my
very rare moments of unhappiness with the editor.

Using `M-x profiler-start, profiler-stop, profiler-report` the culprit
is `compilation-parse-errors`. The function loops over many (about 50)
patterns of potential error output and scans the buffer for them
repeatedly.

The patch edits the function to use a limited version of
`re-search-forward`. Instead of finding the pattern everywhere, it only
looks for the pattern in the first 1024 chars of every line. It seems
plausible that real-world compiler warnings would sit close to the
beginning of lines.

With the patch the experience (in combination with global-so-long-mode)
becomes tolerable; while not lightning fast, Emacs is a lot more
responsive in this situation and I as a user retain control to fix the
problem, such as comint-clear-buffer etc.

FWIW see also the PR https://github.com/emacs-mirror/emacs/pull/24/files


---
 lisp/progmodes/compile.el | 24 +++++++++++++++++++++++-
 1 file changed, 23 insertions(+), 1 deletion(-)

diff --git a/lisp/progmodes/compile.el b/lisp/progmodes/compile.el
index 7a02c3a896..45c69f330c 100644
--- a/lisp/progmodes/compile.el
+++ b/lisp/progmodes/compile.el
@@ -1535,7 +1535,7 @@ to `compilation-error-regexp-alist' if RULES is nil."
           (error "HYPERLINK should be an integer: %s" (nth 5 item)))

         (goto-char start)
-        (while (re-search-forward pat end t)
+        (while (compilation--re-search-forward-limited pat end 1024)
           (when (setq props (compilation-error-properties
                              file line end-line col end-col
                              (or type 2) fmt rule))
@@ -1597,6 +1597,28 @@ to `compilation-error-regexp-alist' if RULES is nil."
                (match-beginning mn) (match-end mn)
                'font-lock-face (cadr props)))))))))

+(defun compilation--re-search-forward-limited (regexp bound n)
+  "Like 're-search-forward limited to the first N chars per line.
+This avoids Emacs performance degradation on scanning excessively
+long lines of text.  It is reasonable when scanning for compiler
+warnings to expect to find them early on each line.  REGEXP and
+BOUND are as in 're-search-forward."
+  (let ((inhibit-field-text-motion t)
+        (found nil)
+        (orig-point (point)))
+    (while (and (null found)
+                (< (point) bound)
+                (not (eobp)))
+      (setq found (re-search-forward regexp
+                                     (max (point)
+                                          (min bound (+ (point-at-bol) n)))
+                                     t))
+      (when (null found)
+        (forward-line 1)))
+    (when (null found)
+      (goto-char orig-point))
+    found))
+
 (defvar-local compilation--parsed -1)

 (defun compilation--ensure-parse (limit)
-- 
2.28.0


--------------------------------------------------------------------------------

In GNU Emacs 27.1 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.21,
cairo version 1.16.0)
Windowing system distributor 'The X.Org Foundation', version 11.0.12009000
System Description: Ubuntu 20.04.2 LTS

Configured using:
 'configure
 --prefix=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-emacs-27.1
 --disable-build-details --with-modules --with-x-toolkit=gtk3 --with-xft
 --with-cairo CFLAGS=-DMAC_OS_X_VERSION_MAX_ALLOWED=101200'

Configured features:
XPM JPEG TIFF GIF PNG RSVG CAIRO SOUND DBUS GSETTINGS GLIB NOTIFY
INOTIFY LIBSELINUX GNUTLS LIBXML2 FREETYPE HARFBUZZ M17N_FLT LIBOTF ZLIB
TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM MODULES THREADS LIBSYSTEMD JSON
PDUMPER GMP

bug#47589: [PATCH] 27.1; Freeze on long lines in compilation-shell-minor-mode

[Lars Ingebrigtsen]

Anton Tayanovskyy <anton.tayanovskyy@gmail.com> writes:

> The patch edits the function to use a limited version of
> `re-search-forward`. Instead of finding the pattern everywhere, it only
> looks for the pattern in the first 1024 chars of every line. It seems
> plausible that real-world compiler warnings would sit close to the
> beginning of lines.
>
> With the patch the experience (in combination with global-so-long-mode)
> becomes tolerable; while not lightning fast, Emacs is a lot more
> responsive in this situation and I as a user retain control to fix the
> problem, such as comint-clear-buffer etc.

I think that's an interesting approach to this problem.  I seem to
recall there being some work in this area, though, and I'm not able to
see the slowdowns in the example given here (even if I increase the
number hugely).

Would it be possible for you to test Emacs 28 and see whether you still
see the problem there?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#47589: [PATCH] 27.1; Freeze on long lines in compilation-shell-minor-mode

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I think that's an interesting approach to this problem.  I seem to
> recall there being some work in this area, though, and I'm not able to
> see the slowdowns in the example given here (even if I increase the
> number hugely).
>
> Would it be possible for you to test Emacs 28 and see whether you still
> see the problem there?

More information was requested, but no response was given within a
month, so I'm closing this bug report.  If the problem still exists,
please respond to this email and we'll reopen the bug report.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#47589: not reproducing on emacs 28

[Anton Tayanovskyy]

[-- Attachment #1: Type: text/plain, Size: 1718 bytes --]

Thanks for looking into it.

I have tried to reproduce the slowdown on Ubuntu with Emacs 28 and 27.2 and
compared against my current patched 27.1 Emacs. The conclusion is that
something improved in a recent version of Emacs and the patch is not moving
the needle on this.

There is still a super-liner cost somewhere. But as a user I only hit this
when accidentally displaying a large file, and current behavior is
interactive enough that I can kill the process running in the shell.

Once I get a chance I would also like to try on Mac OS X where the problem
was more pronounced.

Thanks,

Anton

|--------------------+---------------------------------------+----------------|
| GNU Emacs 27.2     | time python3 -c 'print("x"*1024*256)' | real
0m20.805s |
| GNU Emacs 28.0.50  | time python3 -c 'print("x"*1024*32)'  | real
0m0.315s  |
| GNU Emacs 28.0.50  | time python3 -c 'print("x"*1024*64)'  | real
0m1.151s  |
| GNU Emacs 28.0.50  | time python3 -c 'print("x"*1024*128)' | real
0m6.168s  |
| GNU Emacs 28.0.50  | time python3 -c 'print("x"*1024*256)' | real
0m22.153s |
| GNU Emacs 27.1*    | time python3 -c 'print("x"*1024*32)'  | real
0m0.237s  |
| GNU Emacs 27.1*    | time python3 -c 'print("x"*1024*64)'  | real
0m1.108s  |
| GNU Emacs 27.1*    | time python3 -c 'print("x"*1024*128)' | real
0m5.172s  |
| GNU Emacs 27.1*    | time python3 -c 'print("x"*1024*256)' | real
0m24.369s |
| GNU Emacs 28.0.50* | time python3 -c 'print("x"*1024*32)'  | real
0m0.234s  |
| GNU Emacs 28.0.50* | time python3 -c 'print("x"*1024*64)'  | real
0m0.978s  |
| GNU Emacs 28.0.50* | time python3 -c 'print("x"*1024*128)' | real
0m4.716s  |
| GNU Emacs 28.0.50* | time python3 -c 'print("x"*1024*256)' | real
0m19.047s |

[-- Attachment #2: Type: text/html, Size: 2220 bytes --]