bug#30964: python-mode failing "overlapping strings" assertion lately

bug#30964: python-mode failing "overlapping strings" assertion lately

[dancol]

Has something changed in master that might be breaking python-mode? We
keep hitting a parsing assertion.

The surrounding code in python.el claims that

                       ;; The assertion can only fail if syntax table
                       ;; text properties and the `syntax-ppss' cache
                       ;; are somehow out of whack.  This has been
                       ;; observed when using `syntax-ppss' during
                       ;; narrowing.
                       (cl-assert (> string-start last-string-end)
                                  :show-args
                                  "\
Overlapping strings detected (start=%d, last-end=%d)")

I can _sometimes_ repro with (while (not (eobp))
(python-nav-forward-statement)), but I haven't found a reliable test case.

bug#30964: python-mode failing "overlapping strings" assertion lately

[Noam Postavsky]

dancol@dancol.org writes:

> Has something changed in master that might be breaking python-mode? We
> keep hitting a parsing assertion.
>
> The surrounding code in python.el claims that
>
>                        ;; The assertion can only fail if syntax table
>                        ;; text properties and the `syntax-ppss' cache
>                        ;; are somehow out of whack.  This has been
>                        ;; observed when using `syntax-ppss' during
>                        ;; narrowing.
>                        (cl-assert (> string-start last-string-end)
>                                   :show-args
>                                   "\
> Overlapping strings detected (start=%d, last-end=%d)")
>
> I can _sometimes_ repro with (while (not (eobp))
> (python-nav-forward-statement)), but I haven't found a reliable test case.

How recent is "lately"?  Could it be related to [1: 14b9558752]?

[1: 14b9558752]: 2017-12-12 23:03:00 -0500
  * src/syntax.c (find_defun_start): Use syntax-ppss
  https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=14b95587520959c5b54356547a0a69932a9bb480

bug#30964: python-mode failing "overlapping strings" assertion lately

[Charles A. Roelli]

> Date: Tue, 27 Mar 2018 11:09:38 -0700
> From: dancol@dancol.org
> 
> Has something changed in master that might be breaking python-mode? We
> keep hitting a parsing assertion.
> 
> The surrounding code in python.el claims that
> 
>                        ;; The assertion can only fail if syntax table
>                        ;; text properties and the `syntax-ppss' cache
>                        ;; are somehow out of whack.  This has been
>                        ;; observed when using `syntax-ppss' during
>                        ;; narrowing.
>                        (cl-assert (> string-start last-string-end)
>                                   :show-args
>                                   "\
> Overlapping strings detected (start=%d, last-end=%d)")
> 
> I can _sometimes_ repro with (while (not (eobp))
> (python-nav-forward-statement)), but I haven't found a reliable test case.

I saw it with the pretest (26.0.91) today, with no narrowing,
font-lock off, and after typing M-3 " to open a docstring.  I didn't
find a reproducible case either, though.

bug#30964: python-mode failing "overlapping strings" assertion lately

[Dmitry Gutov]

On 4/3/18 9:12 PM, Charles A. Roelli wrote:

>> I can _sometimes_ repro with (while (not (eobp))
>> (python-nav-forward-statement)), but I haven't found a reliable test case.

You could try calling (syntax-ppss-flush-cache (point-min)) before that, 
for instance, and see if that changes something.

Anyway, an example file should help, even if the problem shows up only 
intermittently.

bug#30964: python-mode failing "overlapping strings" assertion lately

[Clément Pit-Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 1164 bytes --]

(Hoping that emailing the bug address is enough to respond to an existing bug, and that I didn't miss recent follow-ups)

(Resending, since I apparently emailed the wrong bug number at first)

Here's a reliable repro:

def f():
    "

def g():
    """That's a docstring"""
    return ''

Place the point after the unclosed quote in f, and run (python-info-current-defun).  You get this:

Debugger entered--Lisp error: (cl-assertion-failed ((> string-start last-string-end) "Overlapping strings detected (start=%d, last-end=%d)" 67 67))
  cl--assertion-failed((> string-start last-string-end) "Overlapping strings detected (start=%d, last-end=%d)" (67 67) nil)
  python-nav-end-of-statement()
  python-nav-end-of-defun()
  python-info-current-defun()
  eval((python-info-current-defun) nil)
  eval-expression((python-info-current-defun) nil nil 127)
  funcall-interactively(eval-expression (python-info-current-defun) nil nil 127)
  call-interactively(eval-expression nil nil)
  command-execute(eval-expression)

This error triggers in the wild for me due to which-function-mode, which calls python-info-current-defun.

Clément.




[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

bug#30964: python-mode failing "overlapping strings" assertion lately

[Andreas Röhler]

On 18.05.2018 17:46, Clément Pit-Claudel wrote:

> This error triggers in the wild for me due to which-function-mode, which calls python-info-current-defun.
> 
> Clément.
> 
> 
> 

Can't reproduce with GNU Emacs 26.1 (build 1, i686-pc-linux-gnu, GTK+ 
Version 3.14.5) of 2018-04-10

which-function-mode displays [f]

bug#30964: python-mode failing "overlapping strings" assertion lately

[Noam Postavsky]

Andreas Röhler <andreas.roehler@easy-emacs.de> writes:

> On 18.05.2018 17:46, Clément Pit-Claudel wrote:
>
>> This error triggers in the wild for me due to which-function-mode, which calls python-info-current-defun.
>>
>> Clément.
>>
>>
>>
>
> Can't reproduce with GNU Emacs 26.1 (build 1, i686-pc-linux-gnu, GTK+
> Version 3.14.5) of 2018-04-10

I can reproduce, if and only if I make sure to put a newline on the last
line.

bug#30964: python-mode failing "overlapping strings" assertion lately

[Andreas Röhler]

[-- Attachment #1: Type: text/plain, Size: 1544 bytes --]

On 19.05.2018 03:49, Noam Postavsky wrote:
> Andreas Röhler <andreas.roehler@easy-emacs.de> writes:
> 
>> On 18.05.2018 17:46, Clément Pit-Claudel wrote:
>>
>>> This error triggers in the wild for me due to which-function-mode, which calls python-info-current-defun.
>>>
>>> Clément.
>>>
>>>
>>>
>>
>> Can't reproduce with GNU Emacs 26.1 (build 1, i686-pc-linux-gnu, GTK+
>> Version 3.14.5) of 2018-04-10
> 
> I can reproduce, if and only if I make sure to put a newline on the last
> line.
> 

Okay, got the error at some point when inserting a newline. However a 
newline must not trigger it - see attachment.

Below some reports which might be of interest:

Which-Function mode enabled
next-line: End of buffer
Which-Function mode disabled
next-line: End of bufferError during redisplay: (jit-lock-function 3905) 
signaled (error "Overlapping strings detected")
Error during redisplay: (jit-lock-function 3905) signaled (error 
"Overlapping strings detected")
Which-Function mode enabled
Error during redisplay: (jit-lock-function 3919) signaled (error 
"Overlapping strings detected") [2 times]
Which-Function mode disabled
Error during redisplay: (jit-lock-function 3905) signaled (error 
"Overlapping strings detected") [2 times]
Which-Function mode enabled
C-c e is undefined
M-+ is undefined
C-x c is undefined
Which-Function mode disabled
Which-Function mode enabled
Mark set [4 times]
scroll-up-command: End of buffer
Mark set [2 times]
Saved text from "Which-Function mode disabled
next-line: "
Auto-saving...
Mark set [2 times]


[-- Attachment #2: which-function-mode.png --]
[-- Type: image/png, Size: 61216 bytes --]

bug#30964: python-mode failing "overlapping strings" assertion lately

[Noam Postavsky]

[-- Attachment #1: Type: text/plain, Size: 208 bytes --]

Clément Pit-Claudel <clement.pitclaudel@live.com> writes:

> Here's a reliable repro:
>
> def f():
>     "
>
> def g():
>     """That's a docstring"""
>     return ''

It can be reduced a bit:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: bug reproducer --]
[-- Type: text/x-python, Size: 14 bytes --]

'
docstring''

[-- Attachment #3: Type: text/plain, Size: 2034 bytes --]


Debugger entered--Lisp error: (cl-assertion-failed ((> string-start last-string-end) "Overlapping strings detected" 13 13))
  cl--assertion-failed((> string-start last-string-end) "Overlapping strings detected" (13 13) nil)
  python-nav-end-of-statement()
  python-info-end-of-statement-p()
  python-info-end-of-block-p()
  python-nav--forward-sexp(-1 nil nil)
  python-nav-forward-sexp(-1 nil nil)
  python-nav-backward-sexp()
  python-info-docstring-p((0 nil nil t nil nil 0 nil 1 nil nil))
  python-font-lock-syntactic-face-function((0 nil nil t nil nil 0 nil 1 nil nil))
  font-lock-fontify-syntactically-region(1 15 nil)
  font-lock-default-fontify-region(1 15 nil)
  font-lock-fontify-region(1 15)
  font-lock-fontify-block(nil)
  funcall-interactively(font-lock-fontify-block nil)
  call-interactively(font-lock-fontify-block nil nil)
  command-execute(font-lock-fontify-block)

I think the problem is that the assertion is off by 1, it currently
asserts that there is at least one space between strings (because the
end position goes *after* the string, while the start position is part
of the string).

I see the assertion was added in [1: 4fbd330fae].  Phillip, does my
proposed change make sense to you?  ('make -C test python-tests' still
passes.)

--- i/lisp/progmodes/python.el
+++ w/lisp/progmodes/python.el
@@ -1506,7 +1506,7 @@ python-nav-end-of-statement
                        ;; are somehow out of whack.  This has been
                        ;; observed when using `syntax-ppss' during
                        ;; narrowing.
-                       (cl-assert (> string-start last-string-end)
+                       (cl-assert (>= string-start last-string-end)
                                   :show-args
                                   "Overlapping strings detected")
                        (goto-char string-start)

[1: 4fbd330fae]: 2017-03-23 23:05:19 +0100
  Protect against an infloop in python-mode
  https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=4fbd330fae54a9c45d4a717127aa86d75e9938d5

bug#30964: python-mode failing "overlapping strings" assertion lately

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 2772 bytes --]

Noam Postavsky <npostavs@gmail.com> schrieb am Mi., 30. Mai 2018 um
02:29 Uhr:

> Clément Pit-Claudel <clement.pitclaudel@live.com> writes:
>
> > Here's a reliable repro:
> >
> > def f():
> >     "
> >
> > def g():
> >     """That's a docstring"""
> >     return ''
>
> It can be reduced a bit:
>
>
> Debugger entered--Lisp error: (cl-assertion-failed ((> string-start
> last-string-end) "Overlapping strings detected" 13 13))
>   cl--assertion-failed((> string-start last-string-end) "Overlapping
> strings detected" (13 13) nil)
>   python-nav-end-of-statement()
>   python-info-end-of-statement-p()
>   python-info-end-of-block-p()
>   python-nav--forward-sexp(-1 nil nil)
>   python-nav-forward-sexp(-1 nil nil)
>   python-nav-backward-sexp()
>   python-info-docstring-p((0 nil nil t nil nil 0 nil 1 nil nil))
>   python-font-lock-syntactic-face-function((0 nil nil t nil nil 0 nil 1
> nil nil))
>   font-lock-fontify-syntactically-region(1 15 nil)
>   font-lock-default-fontify-region(1 15 nil)
>   font-lock-fontify-region(1 15)
>   font-lock-fontify-block(nil)
>   funcall-interactively(font-lock-fontify-block nil)
>   call-interactively(font-lock-fontify-block nil nil)
>   command-execute(font-lock-fontify-block)
>
> I think the problem is that the assertion is off by 1, it currently
> asserts that there is at least one space between strings (because the
> end position goes *after* the string, while the start position is part
> of the string).
>
> I see the assertion was added in [1: 4fbd330fae].  Phillip, does my
> proposed change make sense to you?


Maybe :-)
Off-by-one errors are always a bit subtle, so it might definitely be the
case that there is one. If so, the problem could be further reduced to
' '' '
or similar. Could you add a unit test using such input and make sure that
it breaks without your patch?



>   ('make -C test python-tests' still
> passes.)
>
> --- i/lisp/progmodes/python.el
> +++ w/lisp/progmodes/python.el
> @@ -1506,7 +1506,7 @@ python-nav-end-of-statement
>                         ;; are somehow out of whack.  This has been
>                         ;; observed when using `syntax-ppss' during
>                         ;; narrowing.
> -                       (cl-assert (> string-start last-string-end)
> +                       (cl-assert (>= string-start last-string-end)
>                                    :show-args
>                                    "Overlapping strings detected")
>                         (goto-char string-start)
>
> [1: 4fbd330fae]: 2017-03-23 23:05:19 +0100
>   Protect against an infloop in python-mode
>
> https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=4fbd330fae54a9c45d4a717127aa86d75e9938d5
>
>

[-- Attachment #2: Type: text/html, Size: 3801 bytes --]

bug#30964: python-mode failing "overlapping strings" assertion lately

[Noam Postavsky]

[-- Attachment #1: Type: text/plain, Size: 735 bytes --]


Philipp Stephani <p.stephani2@gmail.com> writes:

>> I think the problem is that the assertion is off by 1, it currently
>> asserts that there is at least one space between strings (because the
>> end position goes *after* the string, while the start position is part
>> of the string).
>>
>> I see the assertion was added in [1: 4fbd330fae].  Phillip, does my
>> proposed change make sense to you?
>
>
> Maybe :-)
> Off-by-one errors are always a bit subtle, so it might definitely be the
> case that there is one. If so, the problem could be further reduced to
> ' '' '
> or similar. Could you add a unit test using such input and make sure that
> it breaks without your patch?

Yeah, the newlines are the important part it seems.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: patch --]
[-- Type: text/x-diff, Size: 2017 bytes --]

From 209506955227d3b4de030f817c087bf27e5a7426 Mon Sep 17 00:00:00 2001
From: Noam Postavsky <npostavs@gmail.com>
Date: Sat, 2 Jun 2018 16:22:17 -0400
Subject: [PATCH] Fix off by one error in python-mode assertion (Bug#30964)

* lisp/progmodes/python.el (python-nav-end-of-statement): Don't assert
that string-start is strictly greater than last-string-end, because
the string end is a position outside of the string and may therefore
be the same as the following string's start.
* test/lisp/progmodes/python-tests.el (python-nav-end-of-statement-2):
New test.
---
 lisp/progmodes/python.el            | 2 +-
 test/lisp/progmodes/python-tests.el | 6 ++++++
 2 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index c7bb2d97c8..87d2f08fee 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -1506,7 +1506,7 @@ python-nav-end-of-statement
                        ;; are somehow out of whack.  This has been
                        ;; observed when using `syntax-ppss' during
                        ;; narrowing.
-                       (cl-assert (> string-start last-string-end)
+                       (cl-assert (>= string-start last-string-end)
                                   :show-args
                                   "Overlapping strings detected")
                        (goto-char string-start)
diff --git a/test/lisp/progmodes/python-tests.el b/test/lisp/progmodes/python-tests.el
index 4955da02a2..375bd9588d 100644
--- a/test/lisp/progmodes/python-tests.el
+++ b/test/lisp/progmodes/python-tests.el
@@ -2004,6 +2004,12 @@ python-tests-visible-string
                 (python-util-forward-comment -1)
                 (point))))))
 
+(ert-deftest python-nav-end-of-statement-2 ()
+  "Test the string overlap assertion (Bug#30964)."
+  (python-tests-with-temp-buffer
+   "'\n''\n"
+   (python-nav-end-of-statement)))
+
 (ert-deftest python-nav-forward-statement-1 ()
   (python-tests-with-temp-buffer
    "
-- 
2.11.0

bug#30964: python-mode failing "overlapping strings" assertion lately

[Daniel Colascione]

So it turns out that it wasn't an off-by-one problem at all.  Instead,
python was using (re-search-forward (rx (syntax string-delimiter))), which
was matching incorrectly because re-search-forward didn't update syntax as
it matched. It didn't do that because regex matching in general wasn't
reentrant, and it wasn't reentrant because regex bytecode programs are
self-modifying.

I checked in some work that makes regex searching update syntax as needed.

bug#30964: python-mode failing "overlapping strings" assertion lately

[Noam Postavsky]

"Daniel Colascione" <dancol@dancol.org> writes:

> So it turns out that it wasn't an off-by-one problem at all.  Instead,
> python was using (re-search-forward (rx (syntax string-delimiter))), which
> was matching incorrectly because re-search-forward didn't update syntax as
> it matched. It didn't do that because regex matching in general wasn't
> reentrant, and it wasn't reentrant because regex bytecode programs are
> self-modifying.
>
> I checked in some work that makes regex searching update syntax as needed.

It seems there were two separate problems.  The off-by-one error still
happens after your changes, so I pushed my patch to master.

[1: 05345babc9]: 2018-06-16 18:34:19 -0400
  Fix off by one error in python-mode assertion (Bug#30964)
  https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=05345babc988060cca540770599282102c34f2a7

bug#30964: python-mode failing "overlapping strings" assertion lately

[Andreas Röhler]

On 17.06.2018 00:36, Noam Postavsky wrote:
> "Daniel Colascione" <dancol@dancol.org> writes:
> 
>> So it turns out that it wasn't an off-by-one problem at all.  Instead,
>> python was using (re-search-forward (rx (syntax string-delimiter))), which
>> was matching incorrectly because re-search-forward didn't update syntax as
>> it matched. It didn't do that because regex matching in general wasn't
>> reentrant, and it wasn't reentrant because regex bytecode programs are
>> self-modifying.
>>
>> I checked in some work that makes regex searching update syntax as needed.
> 
> It seems there were two separate problems.  The off-by-one error still
> happens after your changes, so I pushed my patch to master.
> 
> [1: 05345babc9]: 2018-06-16 18:34:19 -0400
>    Fix off by one error in python-mode assertion (Bug#30964)
>    https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=05345babc988060cca540770599282102c34f2a7
> 
> 
> 

Hi folks,

sorry, but as long as syntax-ppss has fundamentally issues, all fixing 
here will be vain. For the very same reason don't be surprised having 
not "found a reliable test case".

Cheers,
Andreas