bug#56271: lisp/progmodes/python.el; unmatched quotes cause infinite loop freezing emacs

bug#56271: lisp/progmodes/python.el; unmatched quotes cause infinite loop freezing emacs

[Tom Gillespie]

Using git bisect shows 85db21b94b23b4701930892d1eecc9a1167ed968 is the
source of this issue, all commits following it on master show the same
issue. Reverting that commit fixes the issue.

While editing python files if a single quote occurs in the file during
the course of editing then emacs will enter an infinite loop. The
exact location of the single quote is important, but the pattern that
triggers it happens frequently in python files.

You can get a traceback pinpointing the issue by sending SIGUSR2 to
the hung emacs process.

Below is a minimal python file that will trigger the issue.
Run with emacs -Q evil-python-file.py
#+begin_verbatim
def
    =''
 '
""""""
 # there must be at least one space on this line, comment not needed
''
#+end_verbatim

The full pattern that causes the issue is something like the following.

#+begin_verbatim
<keyword>
<exactly-python-indent-offset-number-of-spaces>=<zero-or-more-not-quotes>''
<at-least-one-space>'
<zero-or-more-lines-with-any-contents-except-single-quote-marks*>
<zero-or-more-spaces><triple-double-quote-string><any-but-single-quote>
<at-least-one-space><any-but-single-quote>
<single-quote-string>
#+end_verbatim

Any keyword should work I have tested with class, def, and with.

The zero or more lines at any indentation can also have single quote
marks that are paired, but there is some weirdness when you have 4 on
a line that makes it possible to avoid the issue.

I can fix the issue by reverting to the cl-assert pattern using unless
to raise an error (unless (>= string-start last-string-end) (error
"oops")) and running the code after that unconditionally. This
prevents jit-lock from trying to run infinitely on a malformed buffer.

That said, this may simply reintroduce bug#54996, and the change
unmasked some other issue with what is going on during the call to
jit-lock-function which needed that error to be throw by cl-assert to
avoid infinite loops/retries.

bug#56271: lisp/progmodes/python.el; unmatched quotes cause infinite loop freezing emacs

[Tom Gillespie]

[-- Attachment #1: Type: text/plain, Size: 534 bytes --]

I was able to hunt down the issue.

The problem is in python-nav-end-of-block where there is an implicit
assumption that python-nav-end-of-statement always makes forward
progress, which is violated when the buffer contains e.g. a single
quote.

I have attached a patch the resolves this issue but there may be other
lurking issues due to the removal of cl-assert because there are now
all sorts of code paths that will run that never ran before because
they would hit that assertion and bail out, usually all the way back
to the top.

[-- Attachment #2: 0001-lisp-progmodes-python.el-python-nav-end-of-block-pre.patch --]
[-- Type: text/x-patch, Size: 2524 bytes --]

From 0b37e78d633af66d96e49655e3bb8418433a6164 Mon Sep 17 00:00:00 2001
From: Tom Gillespie <tgbugs@gmail.com>
Date: Tue, 28 Jun 2022 19:28:05 -0700
Subject: [PATCH] lisp/progmodes/python.el (python-nav-end-of-block): prevent
 infinite loop

lisp/progmodes/python.el (python-nav-end-of-block): Fix a bad
assumption that python-nav-end-of-statement always makes forward
progress by testing that it actually does. If this check is not made
then it is possible for python-nav-end-of-block to enter an infinite
loop. (bug#56271)
---
 lisp/progmodes/python.el | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index e0c937d7ce..16cdf58611 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -1636,11 +1636,15 @@ python-nav-end-of-statement
     (while (and (or noend (goto-char (line-end-position)))
                 (not (eobp))
                 (cond ((setq string-start (python-syntax-context 'string))
-                       ;; The assertion can only fail if syntax table
+                       ;; The condition can be nil if syntax table
                        ;; text properties and the `syntax-ppss' cache
                        ;; are somehow out of whack.  This has been
                        ;; observed when using `syntax-ppss' during
                        ;; narrowing.
+                       ;; It can also fail in cases where the buffer is in
+                       ;; the process of being modified, e.g. when creating
+                       ;; a string with `electric-pair-mode' disabled such
+                       ;; that there can be an unmatched single quote
                        (when (>= string-start last-string-end)
                          (goto-char string-start)
                          (if (python-syntax-context 'paren)
@@ -1723,7 +1727,10 @@ python-nav-end-of-block
       (while (and (forward-line 1)
                   (not (eobp))
                   (or (and (> (current-indentation) block-indentation)
-                           (or (python-nav-end-of-statement) t))
+                           (let ((start (point)))
+                             (python-nav-end-of-statement)
+                             ;; must move forward otherwise infinite loop
+                             (> (point) start)))
                       (python-info-current-line-comment-p)
                       (python-info-current-line-empty-p))))
       (python-util-forward-comment -1)
-- 
2.35.1

bug#56271: lisp/progmodes/python.el; unmatched quotes cause infinite loop freezing emacs

[Tom Gillespie]

[-- Attachment #1: Type: text/plain, Size: 55 bytes --]

Patch to add a test using the evil file as an example.

[-- Attachment #2: 0001-test-lisp-progmodes-python-tests.el-add-test-for-nav.patch --]
[-- Type: text/x-patch, Size: 1157 bytes --]

From 2c6eb7793b8938d2cce413499a62716a87163a29 Mon Sep 17 00:00:00 2001
From: Tom Gillespie <tgbugs@gmail.com>
Date: Tue, 28 Jun 2022 19:55:31 -0700
Subject: [PATCH] test/lisp/progmodes/python-tests.el: add test for nav end of
 block

Add test for python-nav-end-of-block to prevent regression of bug#56271.
---
 test/lisp/progmodes/python-tests.el | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/test/lisp/progmodes/python-tests.el b/test/lisp/progmodes/python-tests.el
index e17bc0df92..c59a2e7953 100644
--- a/test/lisp/progmodes/python-tests.el
+++ b/test/lisp/progmodes/python-tests.el
@@ -2565,6 +2565,18 @@ python-nav-end-of-block-1
                 (python-tests-look-at "print 'After f(*args)'")
                 (line-end-position))))))
 
+(ert-deftest python-nav-end-of-block-2 ()
+  "Ensure that `python-nav-end-of-block' does not enter an infinite loop."
+  (python-tests-with-temp-buffer
+   "def
+    =''
+ '
+\"\"\"\"\"\"
+ #
+''
+"
+   (python-nav-end-of-block)))
+
 (ert-deftest python-nav-forward-block-1 ()
   "This also accounts as a test for `python-nav-backward-block'."
   (python-tests-with-temp-buffer
-- 
2.35.1

bug#56271: lisp/progmodes/python.el; unmatched quotes cause infinite loop freezing emacs

[Lars Ingebrigtsen]

Tom Gillespie <tgbugs@gmail.com> writes:

> I have attached a patch the resolves this issue but there may be other
> lurking issues due to the removal of cl-assert because there are now
> all sorts of code paths that will run that never ran before because
> they would hit that assertion and bail out, usually all the way back
> to the top.

Tom Gillespie <tgbugs@gmail.com> writes:

> Patch to add a test using the evil file as an example.

Thanks; pushed to Emacs 29.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#56271: lisp/progmodes/python.el; unmatched quotes cause infinite loop freezing emacs

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Thanks; pushed to Emacs 29.

But then I apparently forgot to close this bug report, so I'm doing that
now.