* bug#30964: python-mode failing "overlapping strings" assertion lately @ 2018-03-27 18:09 dancol 2018-04-02 1:11 ` Noam Postavsky ` (2 more replies) 0 siblings, 3 replies; 14+ messages in thread From: dancol @ 2018-03-27 18:09 UTC (permalink / raw) To: 30964 Has something changed in master that might be breaking python-mode? We keep hitting a parsing assertion. The surrounding code in python.el claims that ;; The assertion can only fail if syntax table ;; text properties and the `syntax-ppss' cache ;; are somehow out of whack. This has been ;; observed when using `syntax-ppss' during ;; narrowing. (cl-assert (> string-start last-string-end) :show-args "\ Overlapping strings detected (start=%d, last-end=%d)") I can _sometimes_ repro with (while (not (eobp)) (python-nav-forward-statement)), but I haven't found a reliable test case. ^ permalink raw reply [flat|nested] 14+ messages in thread
* bug#30964: python-mode failing "overlapping strings" assertion lately 2018-03-27 18:09 bug#30964: python-mode failing "overlapping strings" assertion lately dancol @ 2018-04-02 1:11 ` Noam Postavsky 2018-04-03 18:12 ` Charles A. Roelli 2018-05-18 15:46 ` Clément Pit-Claudel 2 siblings, 0 replies; 14+ messages in thread From: Noam Postavsky @ 2018-04-02 1:11 UTC (permalink / raw) To: dancol; +Cc: 30964, Stefan Monnier dancol@dancol.org writes: > Has something changed in master that might be breaking python-mode? We > keep hitting a parsing assertion. > > The surrounding code in python.el claims that > > ;; The assertion can only fail if syntax table > ;; text properties and the `syntax-ppss' cache > ;; are somehow out of whack. This has been > ;; observed when using `syntax-ppss' during > ;; narrowing. > (cl-assert (> string-start last-string-end) > :show-args > "\ > Overlapping strings detected (start=%d, last-end=%d)") > > I can _sometimes_ repro with (while (not (eobp)) > (python-nav-forward-statement)), but I haven't found a reliable test case. How recent is "lately"? Could it be related to [1: 14b9558752]? [1: 14b9558752]: 2017-12-12 23:03:00 -0500 * src/syntax.c (find_defun_start): Use syntax-ppss https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=14b95587520959c5b54356547a0a69932a9bb480 ^ permalink raw reply [flat|nested] 14+ messages in thread
* bug#30964: python-mode failing "overlapping strings" assertion lately 2018-03-27 18:09 bug#30964: python-mode failing "overlapping strings" assertion lately dancol 2018-04-02 1:11 ` Noam Postavsky @ 2018-04-03 18:12 ` Charles A. Roelli 2018-04-03 22:06 ` Dmitry Gutov 2018-05-18 15:46 ` Clément Pit-Claudel 2 siblings, 1 reply; 14+ messages in thread From: Charles A. Roelli @ 2018-04-03 18:12 UTC (permalink / raw) To: dancol; +Cc: 30964 > Date: Tue, 27 Mar 2018 11:09:38 -0700 > From: dancol@dancol.org > > Has something changed in master that might be breaking python-mode? We > keep hitting a parsing assertion. > > The surrounding code in python.el claims that > > ;; The assertion can only fail if syntax table > ;; text properties and the `syntax-ppss' cache > ;; are somehow out of whack. This has been > ;; observed when using `syntax-ppss' during > ;; narrowing. > (cl-assert (> string-start last-string-end) > :show-args > "\ > Overlapping strings detected (start=%d, last-end=%d)") > > I can _sometimes_ repro with (while (not (eobp)) > (python-nav-forward-statement)), but I haven't found a reliable test case. I saw it with the pretest (26.0.91) today, with no narrowing, font-lock off, and after typing M-3 " to open a docstring. I didn't find a reproducible case either, though. ^ permalink raw reply [flat|nested] 14+ messages in thread
* bug#30964: python-mode failing "overlapping strings" assertion lately 2018-04-03 18:12 ` Charles A. Roelli @ 2018-04-03 22:06 ` Dmitry Gutov 0 siblings, 0 replies; 14+ messages in thread From: Dmitry Gutov @ 2018-04-03 22:06 UTC (permalink / raw) To: Charles A. Roelli, dancol; +Cc: 30964 On 4/3/18 9:12 PM, Charles A. Roelli wrote: >> I can _sometimes_ repro with (while (not (eobp)) >> (python-nav-forward-statement)), but I haven't found a reliable test case. You could try calling (syntax-ppss-flush-cache (point-min)) before that, for instance, and see if that changes something. Anyway, an example file should help, even if the problem shows up only intermittently. ^ permalink raw reply [flat|nested] 14+ messages in thread
* bug#30964: python-mode failing "overlapping strings" assertion lately 2018-03-27 18:09 bug#30964: python-mode failing "overlapping strings" assertion lately dancol 2018-04-02 1:11 ` Noam Postavsky 2018-04-03 18:12 ` Charles A. Roelli @ 2018-05-18 15:46 ` Clément Pit-Claudel 2018-05-18 19:18 ` Andreas Röhler 2018-05-30 0:29 ` Noam Postavsky 2 siblings, 2 replies; 14+ messages in thread From: Clément Pit-Claudel @ 2018-05-18 15:46 UTC (permalink / raw) To: 30964 [-- Attachment #1.1: Type: text/plain, Size: 1164 bytes --] (Hoping that emailing the bug address is enough to respond to an existing bug, and that I didn't miss recent follow-ups) (Resending, since I apparently emailed the wrong bug number at first) Here's a reliable repro: def f(): " def g(): """That's a docstring""" return '' Place the point after the unclosed quote in f, and run (python-info-current-defun). You get this: Debugger entered--Lisp error: (cl-assertion-failed ((> string-start last-string-end) "Overlapping strings detected (start=%d, last-end=%d)" 67 67)) cl--assertion-failed((> string-start last-string-end) "Overlapping strings detected (start=%d, last-end=%d)" (67 67) nil) python-nav-end-of-statement() python-nav-end-of-defun() python-info-current-defun() eval((python-info-current-defun) nil) eval-expression((python-info-current-defun) nil nil 127) funcall-interactively(eval-expression (python-info-current-defun) nil nil 127) call-interactively(eval-expression nil nil) command-execute(eval-expression) This error triggers in the wild for me due to which-function-mode, which calls python-info-current-defun. Clément. [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 819 bytes --] ^ permalink raw reply [flat|nested] 14+ messages in thread
* bug#30964: python-mode failing "overlapping strings" assertion lately 2018-05-18 15:46 ` Clément Pit-Claudel @ 2018-05-18 19:18 ` Andreas Röhler 2018-05-19 1:49 ` Noam Postavsky 2018-05-30 0:29 ` Noam Postavsky 1 sibling, 1 reply; 14+ messages in thread From: Andreas Röhler @ 2018-05-18 19:18 UTC (permalink / raw) To: 30964 On 18.05.2018 17:46, Clément Pit-Claudel wrote: > This error triggers in the wild for me due to which-function-mode, which calls python-info-current-defun. > > Clément. > > > Can't reproduce with GNU Emacs 26.1 (build 1, i686-pc-linux-gnu, GTK+ Version 3.14.5) of 2018-04-10 which-function-mode displays [f] ^ permalink raw reply [flat|nested] 14+ messages in thread
* bug#30964: python-mode failing "overlapping strings" assertion lately 2018-05-18 19:18 ` Andreas Röhler @ 2018-05-19 1:49 ` Noam Postavsky 2018-05-19 6:49 ` Andreas Röhler 0 siblings, 1 reply; 14+ messages in thread From: Noam Postavsky @ 2018-05-19 1:49 UTC (permalink / raw) To: Andreas Röhler; +Cc: 30964 Andreas Röhler <andreas.roehler@easy-emacs.de> writes: > On 18.05.2018 17:46, Clément Pit-Claudel wrote: > >> This error triggers in the wild for me due to which-function-mode, which calls python-info-current-defun. >> >> Clément. >> >> >> > > Can't reproduce with GNU Emacs 26.1 (build 1, i686-pc-linux-gnu, GTK+ > Version 3.14.5) of 2018-04-10 I can reproduce, if and only if I make sure to put a newline on the last line. ^ permalink raw reply [flat|nested] 14+ messages in thread
* bug#30964: python-mode failing "overlapping strings" assertion lately 2018-05-19 1:49 ` Noam Postavsky @ 2018-05-19 6:49 ` Andreas Röhler 0 siblings, 0 replies; 14+ messages in thread From: Andreas Röhler @ 2018-05-19 6:49 UTC (permalink / raw) To: Noam Postavsky; +Cc: 30964 [-- Attachment #1: Type: text/plain, Size: 1544 bytes --] On 19.05.2018 03:49, Noam Postavsky wrote: > Andreas Röhler <andreas.roehler@easy-emacs.de> writes: > >> On 18.05.2018 17:46, Clément Pit-Claudel wrote: >> >>> This error triggers in the wild for me due to which-function-mode, which calls python-info-current-defun. >>> >>> Clément. >>> >>> >>> >> >> Can't reproduce with GNU Emacs 26.1 (build 1, i686-pc-linux-gnu, GTK+ >> Version 3.14.5) of 2018-04-10 > > I can reproduce, if and only if I make sure to put a newline on the last > line. > Okay, got the error at some point when inserting a newline. However a newline must not trigger it - see attachment. Below some reports which might be of interest: Which-Function mode enabled next-line: End of buffer Which-Function mode disabled next-line: End of bufferError during redisplay: (jit-lock-function 3905) signaled (error "Overlapping strings detected") Error during redisplay: (jit-lock-function 3905) signaled (error "Overlapping strings detected") Which-Function mode enabled Error during redisplay: (jit-lock-function 3919) signaled (error "Overlapping strings detected") [2 times] Which-Function mode disabled Error during redisplay: (jit-lock-function 3905) signaled (error "Overlapping strings detected") [2 times] Which-Function mode enabled C-c e is undefined M-+ is undefined C-x c is undefined Which-Function mode disabled Which-Function mode enabled Mark set [4 times] scroll-up-command: End of buffer Mark set [2 times] Saved text from "Which-Function mode disabled next-line: " Auto-saving... Mark set [2 times] [-- Attachment #2: which-function-mode.png --] [-- Type: image/png, Size: 61216 bytes --] ^ permalink raw reply [flat|nested] 14+ messages in thread
* bug#30964: python-mode failing "overlapping strings" assertion lately 2018-05-18 15:46 ` Clément Pit-Claudel 2018-05-18 19:18 ` Andreas Röhler @ 2018-05-30 0:29 ` Noam Postavsky 2018-06-02 19:03 ` Philipp Stephani 1 sibling, 1 reply; 14+ messages in thread From: Noam Postavsky @ 2018-05-30 0:29 UTC (permalink / raw) To: Clément Pit-Claudel; +Cc: Philipp Stephani, 30964 [-- Attachment #1: Type: text/plain, Size: 208 bytes --] Clément Pit-Claudel <clement.pitclaudel@live.com> writes: > Here's a reliable repro: > > def f(): > " > > def g(): > """That's a docstring""" > return '' It can be reduced a bit: [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: bug reproducer --] [-- Type: text/x-python, Size: 14 bytes --] ' docstring'' [-- Attachment #3: Type: text/plain, Size: 2034 bytes --] Debugger entered--Lisp error: (cl-assertion-failed ((> string-start last-string-end) "Overlapping strings detected" 13 13)) cl--assertion-failed((> string-start last-string-end) "Overlapping strings detected" (13 13) nil) python-nav-end-of-statement() python-info-end-of-statement-p() python-info-end-of-block-p() python-nav--forward-sexp(-1 nil nil) python-nav-forward-sexp(-1 nil nil) python-nav-backward-sexp() python-info-docstring-p((0 nil nil t nil nil 0 nil 1 nil nil)) python-font-lock-syntactic-face-function((0 nil nil t nil nil 0 nil 1 nil nil)) font-lock-fontify-syntactically-region(1 15 nil) font-lock-default-fontify-region(1 15 nil) font-lock-fontify-region(1 15) font-lock-fontify-block(nil) funcall-interactively(font-lock-fontify-block nil) call-interactively(font-lock-fontify-block nil nil) command-execute(font-lock-fontify-block) I think the problem is that the assertion is off by 1, it currently asserts that there is at least one space between strings (because the end position goes *after* the string, while the start position is part of the string). I see the assertion was added in [1: 4fbd330fae]. Phillip, does my proposed change make sense to you? ('make -C test python-tests' still passes.) --- i/lisp/progmodes/python.el +++ w/lisp/progmodes/python.el @@ -1506,7 +1506,7 @@ python-nav-end-of-statement ;; are somehow out of whack. This has been ;; observed when using `syntax-ppss' during ;; narrowing. - (cl-assert (> string-start last-string-end) + (cl-assert (>= string-start last-string-end) :show-args "Overlapping strings detected") (goto-char string-start) [1: 4fbd330fae]: 2017-03-23 23:05:19 +0100 Protect against an infloop in python-mode https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=4fbd330fae54a9c45d4a717127aa86d75e9938d5 ^ permalink raw reply [flat|nested] 14+ messages in thread
* bug#30964: python-mode failing "overlapping strings" assertion lately 2018-05-30 0:29 ` Noam Postavsky @ 2018-06-02 19:03 ` Philipp Stephani 2018-06-02 20:35 ` Noam Postavsky 0 siblings, 1 reply; 14+ messages in thread From: Philipp Stephani @ 2018-06-02 19:03 UTC (permalink / raw) To: Noam Postavsky; +Cc: Clément Pit-Claudel, 30964 [-- Attachment #1: Type: text/plain, Size: 2772 bytes --] Noam Postavsky <npostavs@gmail.com> schrieb am Mi., 30. Mai 2018 um 02:29 Uhr: > Clément Pit-Claudel <clement.pitclaudel@live.com> writes: > > > Here's a reliable repro: > > > > def f(): > > " > > > > def g(): > > """That's a docstring""" > > return '' > > It can be reduced a bit: > > > Debugger entered--Lisp error: (cl-assertion-failed ((> string-start > last-string-end) "Overlapping strings detected" 13 13)) > cl--assertion-failed((> string-start last-string-end) "Overlapping > strings detected" (13 13) nil) > python-nav-end-of-statement() > python-info-end-of-statement-p() > python-info-end-of-block-p() > python-nav--forward-sexp(-1 nil nil) > python-nav-forward-sexp(-1 nil nil) > python-nav-backward-sexp() > python-info-docstring-p((0 nil nil t nil nil 0 nil 1 nil nil)) > python-font-lock-syntactic-face-function((0 nil nil t nil nil 0 nil 1 > nil nil)) > font-lock-fontify-syntactically-region(1 15 nil) > font-lock-default-fontify-region(1 15 nil) > font-lock-fontify-region(1 15) > font-lock-fontify-block(nil) > funcall-interactively(font-lock-fontify-block nil) > call-interactively(font-lock-fontify-block nil nil) > command-execute(font-lock-fontify-block) > > I think the problem is that the assertion is off by 1, it currently > asserts that there is at least one space between strings (because the > end position goes *after* the string, while the start position is part > of the string). > > I see the assertion was added in [1: 4fbd330fae]. Phillip, does my > proposed change make sense to you? Maybe :-) Off-by-one errors are always a bit subtle, so it might definitely be the case that there is one. If so, the problem could be further reduced to ' '' ' or similar. Could you add a unit test using such input and make sure that it breaks without your patch? > ('make -C test python-tests' still > passes.) > > --- i/lisp/progmodes/python.el > +++ w/lisp/progmodes/python.el > @@ -1506,7 +1506,7 @@ python-nav-end-of-statement > ;; are somehow out of whack. This has been > ;; observed when using `syntax-ppss' during > ;; narrowing. > - (cl-assert (> string-start last-string-end) > + (cl-assert (>= string-start last-string-end) > :show-args > "Overlapping strings detected") > (goto-char string-start) > > [1: 4fbd330fae]: 2017-03-23 23:05:19 +0100 > Protect against an infloop in python-mode > > https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=4fbd330fae54a9c45d4a717127aa86d75e9938d5 > > [-- Attachment #2: Type: text/html, Size: 3801 bytes --] ^ permalink raw reply [flat|nested] 14+ messages in thread
* bug#30964: python-mode failing "overlapping strings" assertion lately 2018-06-02 19:03 ` Philipp Stephani @ 2018-06-02 20:35 ` Noam Postavsky 2018-06-16 20:55 ` Daniel Colascione 0 siblings, 1 reply; 14+ messages in thread From: Noam Postavsky @ 2018-06-02 20:35 UTC (permalink / raw) To: Philipp Stephani; +Cc: Clément Pit-Claudel, 30964 [-- Attachment #1: Type: text/plain, Size: 735 bytes --] Philipp Stephani <p.stephani2@gmail.com> writes: >> I think the problem is that the assertion is off by 1, it currently >> asserts that there is at least one space between strings (because the >> end position goes *after* the string, while the start position is part >> of the string). >> >> I see the assertion was added in [1: 4fbd330fae]. Phillip, does my >> proposed change make sense to you? > > > Maybe :-) > Off-by-one errors are always a bit subtle, so it might definitely be the > case that there is one. If so, the problem could be further reduced to > ' '' ' > or similar. Could you add a unit test using such input and make sure that > it breaks without your patch? Yeah, the newlines are the important part it seems. [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: patch --] [-- Type: text/x-diff, Size: 2017 bytes --] From 209506955227d3b4de030f817c087bf27e5a7426 Mon Sep 17 00:00:00 2001 From: Noam Postavsky <npostavs@gmail.com> Date: Sat, 2 Jun 2018 16:22:17 -0400 Subject: [PATCH] Fix off by one error in python-mode assertion (Bug#30964) * lisp/progmodes/python.el (python-nav-end-of-statement): Don't assert that string-start is strictly greater than last-string-end, because the string end is a position outside of the string and may therefore be the same as the following string's start. * test/lisp/progmodes/python-tests.el (python-nav-end-of-statement-2): New test. --- lisp/progmodes/python.el | 2 +- test/lisp/progmodes/python-tests.el | 6 ++++++ 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el index c7bb2d97c8..87d2f08fee 100644 --- a/lisp/progmodes/python.el +++ b/lisp/progmodes/python.el @@ -1506,7 +1506,7 @@ python-nav-end-of-statement ;; are somehow out of whack. This has been ;; observed when using `syntax-ppss' during ;; narrowing. - (cl-assert (> string-start last-string-end) + (cl-assert (>= string-start last-string-end) :show-args "Overlapping strings detected") (goto-char string-start) diff --git a/test/lisp/progmodes/python-tests.el b/test/lisp/progmodes/python-tests.el index 4955da02a2..375bd9588d 100644 --- a/test/lisp/progmodes/python-tests.el +++ b/test/lisp/progmodes/python-tests.el @@ -2004,6 +2004,12 @@ python-tests-visible-string (python-util-forward-comment -1) (point)))))) +(ert-deftest python-nav-end-of-statement-2 () + "Test the string overlap assertion (Bug#30964)." + (python-tests-with-temp-buffer + "'\n''\n" + (python-nav-end-of-statement))) + (ert-deftest python-nav-forward-statement-1 () (python-tests-with-temp-buffer " -- 2.11.0 ^ permalink raw reply related [flat|nested] 14+ messages in thread
* bug#30964: python-mode failing "overlapping strings" assertion lately 2018-06-02 20:35 ` Noam Postavsky @ 2018-06-16 20:55 ` Daniel Colascione 2018-06-16 22:36 ` Noam Postavsky 0 siblings, 1 reply; 14+ messages in thread From: Daniel Colascione @ 2018-06-16 20:55 UTC (permalink / raw) To: Noam Postavsky; +Cc: Clément Pit-Claudel, Philipp Stephani, 30964 So it turns out that it wasn't an off-by-one problem at all. Instead, python was using (re-search-forward (rx (syntax string-delimiter))), which was matching incorrectly because re-search-forward didn't update syntax as it matched. It didn't do that because regex matching in general wasn't reentrant, and it wasn't reentrant because regex bytecode programs are self-modifying. I checked in some work that makes regex searching update syntax as needed. ^ permalink raw reply [flat|nested] 14+ messages in thread
* bug#30964: python-mode failing "overlapping strings" assertion lately 2018-06-16 20:55 ` Daniel Colascione @ 2018-06-16 22:36 ` Noam Postavsky 2018-06-17 7:33 ` Andreas Röhler 0 siblings, 1 reply; 14+ messages in thread From: Noam Postavsky @ 2018-06-16 22:36 UTC (permalink / raw) To: Daniel Colascione; +Cc: Clément Pit-Claudel, Philipp Stephani, 30964 "Daniel Colascione" <dancol@dancol.org> writes: > So it turns out that it wasn't an off-by-one problem at all. Instead, > python was using (re-search-forward (rx (syntax string-delimiter))), which > was matching incorrectly because re-search-forward didn't update syntax as > it matched. It didn't do that because regex matching in general wasn't > reentrant, and it wasn't reentrant because regex bytecode programs are > self-modifying. > > I checked in some work that makes regex searching update syntax as needed. It seems there were two separate problems. The off-by-one error still happens after your changes, so I pushed my patch to master. [1: 05345babc9]: 2018-06-16 18:34:19 -0400 Fix off by one error in python-mode assertion (Bug#30964) https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=05345babc988060cca540770599282102c34f2a7 ^ permalink raw reply [flat|nested] 14+ messages in thread
* bug#30964: python-mode failing "overlapping strings" assertion lately 2018-06-16 22:36 ` Noam Postavsky @ 2018-06-17 7:33 ` Andreas Röhler 0 siblings, 0 replies; 14+ messages in thread From: Andreas Röhler @ 2018-06-17 7:33 UTC (permalink / raw) To: 30964 On 17.06.2018 00:36, Noam Postavsky wrote: > "Daniel Colascione" <dancol@dancol.org> writes: > >> So it turns out that it wasn't an off-by-one problem at all. Instead, >> python was using (re-search-forward (rx (syntax string-delimiter))), which >> was matching incorrectly because re-search-forward didn't update syntax as >> it matched. It didn't do that because regex matching in general wasn't >> reentrant, and it wasn't reentrant because regex bytecode programs are >> self-modifying. >> >> I checked in some work that makes regex searching update syntax as needed. > > It seems there were two separate problems. The off-by-one error still > happens after your changes, so I pushed my patch to master. > > [1: 05345babc9]: 2018-06-16 18:34:19 -0400 > Fix off by one error in python-mode assertion (Bug#30964) > https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=05345babc988060cca540770599282102c34f2a7 > > > Hi folks, sorry, but as long as syntax-ppss has fundamentally issues, all fixing here will be vain. For the very same reason don't be surprised having not "found a reliable test case". Cheers, Andreas ^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2018-06-17 7:33 UTC | newest] Thread overview: 14+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2018-03-27 18:09 bug#30964: python-mode failing "overlapping strings" assertion lately dancol 2018-04-02 1:11 ` Noam Postavsky 2018-04-03 18:12 ` Charles A. Roelli 2018-04-03 22:06 ` Dmitry Gutov 2018-05-18 15:46 ` Clément Pit-Claudel 2018-05-18 19:18 ` Andreas Röhler 2018-05-19 1:49 ` Noam Postavsky 2018-05-19 6:49 ` Andreas Röhler 2018-05-30 0:29 ` Noam Postavsky 2018-06-02 19:03 ` Philipp Stephani 2018-06-02 20:35 ` Noam Postavsky 2018-06-16 20:55 ` Daniel Colascione 2018-06-16 22:36 ` Noam Postavsky 2018-06-17 7:33 ` Andreas Röhler
Code repositories for project(s) associated with this external index https://git.savannah.gnu.org/cgit/emacs.git https://git.savannah.gnu.org/cgit/emacs/org-mode.git This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.