all messages for Emacs-related lists mirrored at yhetil.org
 help / color / mirror / code / Atom feed
* [PATCH] Protect against an infloop in python-mode
@ 2017-02-28 21:31 Philipp Stephani
  2017-03-23 22:08 ` Philipp Stephani
  0 siblings, 1 reply; 4+ messages in thread
From: Philipp Stephani @ 2017-02-28 21:31 UTC (permalink / raw)
  To: emacs-devel; +Cc: Philipp Stephani

There appears to be an edge case caused by using `syntax-ppss' in a
narrowed buffer during JIT lock inside of Python triple-quote strings.
Unfortunately it is impossible to reproduce without manually
destroying the syntactic information in the Python buffer, but it has
been observed in practice.  In that case it can happen that the syntax
caches get sufficiently out of whack so that there appear to be
overlapping strings in the buffer.  As Python has no nested strings,
this situation is impossible and leads to an infloop in
`python-nav-end-of-statement'.  Protect against this by checking
whether the search for the end of the current string makes progress.

* python.el (python-nav-end-of-statement): Protect against infloop.
* progmodes/python-tests.el
(python-tests--python-nav-end-of-statement--infloop): Add unit test.
---
 lisp/progmodes/python.el            | 16 +++++++++++++---
 test/lisp/progmodes/python-tests.el | 17 +++++++++++++++++
 2 files changed, 30 insertions(+), 3 deletions(-)

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index 90b5e4e0dc..71f4298fa5 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -1491,10 +1491,18 @@ python-nav-end-of-statement
 jump to the end of line when moving forward searching for the end
 of the statement."
   (interactive "^")
-  (let (string-start bs-pos)
+  (let (string-start bs-pos (last-string-end 0))
     (while (and (or noend (goto-char (line-end-position)))
                 (not (eobp))
                 (cond ((setq string-start (python-syntax-context 'string))
+                       ;; The assertion can only fail if syntax table
+                       ;; text properties and the `syntax-ppss' cache
+                       ;; are somehow out of whack.  This has been
+                       ;; observed when using `syntax-ppss' during
+                       ;; narrowing.
+                       (cl-assert (> string-start last-string-end)
+                                  :show-args
+                                  "Overlapping strings detected")
                        (goto-char string-start)
                        (if (python-syntax-context 'paren)
                            ;; Ended up inside a paren, roll again.
@@ -1504,8 +1512,10 @@ python-nav-end-of-statement
                          (goto-char (+ (point)
                                        (python-syntax-count-quotes
                                         (char-after (point)) (point))))
-                         (or (re-search-forward (rx (syntax string-delimiter)) nil t)
-                             (goto-char (point-max)))))
+                         (setq last-string-end
+                               (or (re-search-forward
+                                    (rx (syntax string-delimiter)) nil t)
+                                   (goto-char (point-max))))))
                       ((python-syntax-context 'paren)
                        ;; The statement won't end before we've escaped
                        ;; at least one level of parenthesis.
diff --git a/test/lisp/progmodes/python-tests.el b/test/lisp/progmodes/python-tests.el
index 1e6b867d30..893efca0a0 100644
--- a/test/lisp/progmodes/python-tests.el
+++ b/test/lisp/progmodes/python-tests.el
@@ -5314,6 +5314,23 @@ python-tests-shell-interpreter
       (or enabled (hs-minor-mode -1)))))
 
 
+(ert-deftest python-tests--python-nav-end-of-statement--infloop ()
+  "Checks that `python-nav-end-of-statement' doesn't infloop in a
+buffer with overlapping strings."
+  (python-tests-with-temp-buffer "''' '\n''' ' '\n"
+    (syntax-propertize (point-max))
+    ;; Create a situation where strings nominally overlap.  This
+    ;; shouldn't happen in practice, but apparently it can happen when
+    ;; a package calls `syntax-ppss' in a narrowed buffer during JIT
+    ;; lock.
+    (put-text-property 4 5 'syntax-table (string-to-syntax "|"))
+    (remove-text-properties 8 9 '(syntax-table nil))
+    (goto-char 4)
+    (setq-local syntax-propertize-function nil)
+    ;; The next form should not infloop.
+    (should-error (python-nav-end-of-statement))
+    (should (eolp))))
+
 
 (provide 'python-tests)
 
-- 
2.11.1




^ permalink raw reply related	[flat|nested] 4+ messages in thread

* Re: [PATCH] Protect against an infloop in python-mode
  2017-02-28 21:31 [PATCH] Protect against an infloop in python-mode Philipp Stephani
@ 2017-03-23 22:08 ` Philipp Stephani
  2017-03-24 17:53   ` Alan Mackenzie
  0 siblings, 1 reply; 4+ messages in thread
From: Philipp Stephani @ 2017-03-23 22:08 UTC (permalink / raw)
  To: emacs-devel; +Cc: Philipp Stephani

[-- Attachment #1: Type: text/plain, Size: 968 bytes --]

Philipp Stephani <p.stephani2@gmail.com> schrieb am Di., 28. Feb. 2017 um
22:32 Uhr:

> There appears to be an edge case caused by using `syntax-ppss' in a
> narrowed buffer during JIT lock inside of Python triple-quote strings.
> Unfortunately it is impossible to reproduce without manually
> destroying the syntactic information in the Python buffer, but it has
> been observed in practice.  In that case it can happen that the syntax
> caches get sufficiently out of whack so that there appear to be
> overlapping strings in the buffer.  As Python has no nested strings,
> this situation is impossible and leads to an infloop in
> `python-nav-end-of-statement'.  Protect against this by checking
> whether the search for the end of the current string makes progress.
>
> * python.el (python-nav-end-of-statement): Protect against infloop.
> * progmodes/python-tests.el
> (python-tests--python-nav-end-of-statement--infloop): Add unit test.
>

Pushed as 4fbd330fae.

[-- Attachment #2: Type: text/html, Size: 1560 bytes --]

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [PATCH] Protect against an infloop in python-mode
  2017-03-23 22:08 ` Philipp Stephani
@ 2017-03-24 17:53   ` Alan Mackenzie
  2017-03-26 18:49     ` Philipp Stephani
  0 siblings, 1 reply; 4+ messages in thread
From: Alan Mackenzie @ 2017-03-24 17:53 UTC (permalink / raw)
  To: Philipp Stephani; +Cc: Philipp Stephani, emacs-devel

Hello, Philipp.

On Thu, Mar 23, 2017 at 22:08:15 +0000, Philipp Stephani wrote:
> Philipp Stephani <p.stephani2@gmail.com> schrieb am Di., 28. Feb. 2017 um
> 22:32 Uhr:

> > There appears to be an edge case caused by using `syntax-ppss' in a
> > narrowed buffer during JIT lock inside of Python triple-quote strings.
> > Unfortunately it is impossible to reproduce without manually
> > destroying the syntactic information in the Python buffer, but it has
> > been observed in practice.  In that case it can happen that the syntax
> > caches get sufficiently out of whack so that there appear to be
> > overlapping strings in the buffer.  As Python has no nested strings,
> > this situation is impossible and leads to an infloop in
> > `python-nav-end-of-statement'.  Protect against this by checking
> > whether the search for the end of the current string makes progress.

I think it's fair to say that syntax-ppss doesn't work in narrowed
buffers.  This has been known for a long time now - I raised bug #22983
about this topic over a year ago, yet despite repeated requests from
myself, there are no signs this is going to be fixed any time soon.

I "welcome" your misfortune, as it shows that the flaws in syntax-ppss
in narrowed buffers occur in real situations, not just test cases.
Presumably, diagnosing the bug took up quite a lot of your valuable
time.

Let's get bug #22983 fixed.

> > * python.el (python-nav-end-of-statement): Protect against infloop.
> > * progmodes/python-tests.el
> > (python-tests--python-nav-end-of-statement--infloop): Add unit test.


> Pushed as 4fbd330fae.

-- 
Alan Mackenzie (Nuremberg, Germany).



^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [PATCH] Protect against an infloop in python-mode
  2017-03-24 17:53   ` Alan Mackenzie
@ 2017-03-26 18:49     ` Philipp Stephani
  0 siblings, 0 replies; 4+ messages in thread
From: Philipp Stephani @ 2017-03-26 18:49 UTC (permalink / raw)
  To: Alan Mackenzie; +Cc: Philipp Stephani, emacs-devel

[-- Attachment #1: Type: text/plain, Size: 1635 bytes --]

Alan Mackenzie <acm@muc.de> schrieb am Fr., 24. März 2017 um 18:53 Uhr:

> Hello, Philipp.
>
> On Thu, Mar 23, 2017 at 22:08:15 +0000, Philipp Stephani wrote:
> > Philipp Stephani <p.stephani2@gmail.com> schrieb am Di., 28. Feb. 2017
> um
> > 22:32 Uhr:
>
> > > There appears to be an edge case caused by using `syntax-ppss' in a
> > > narrowed buffer during JIT lock inside of Python triple-quote strings.
> > > Unfortunately it is impossible to reproduce without manually
> > > destroying the syntactic information in the Python buffer, but it has
> > > been observed in practice.  In that case it can happen that the syntax
> > > caches get sufficiently out of whack so that there appear to be
> > > overlapping strings in the buffer.  As Python has no nested strings,
> > > this situation is impossible and leads to an infloop in
> > > `python-nav-end-of-statement'.  Protect against this by checking
> > > whether the search for the end of the current string makes progress.
>
> I think it's fair to say that syntax-ppss doesn't work in narrowed
> buffers.  This has been known for a long time now - I raised bug #22983
> about this topic over a year ago, yet despite repeated requests from
> myself, there are no signs this is going to be fixed any time soon.
>
> I "welcome" your misfortune, as it shows that the flaws in syntax-ppss
> in narrowed buffers occur in real situations, not just test cases.
> Presumably, diagnosing the bug took up quite a lot of your valuable
> time.
>
> Let's get bug #22983 fixed.
>
>
Agreed. I'm not 100% positive that it's the same bug, but it's very likely.

[-- Attachment #2: Type: text/html, Size: 2671 bytes --]

^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2017-03-26 18:49 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2017-02-28 21:31 [PATCH] Protect against an infloop in python-mode Philipp Stephani
2017-03-23 22:08 ` Philipp Stephani
2017-03-24 17:53   ` Alan Mackenzie
2017-03-26 18:49     ` Philipp Stephani

Code repositories for project(s) associated with this external index

	https://git.savannah.gnu.org/cgit/emacs.git
	https://git.savannah.gnu.org/cgit/emacs/org-mode.git

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.