bug#41195: 27.0.91; Using syntax-ppss-flush-cache in syntax-propertize-function infloops

bug#41195: 27.0.91; Using syntax-ppss-flush-cache in syntax-propertize-function infloops

[Dmitry Gutov]

In theory, syntax-propertize-function implementations shouldn't change
buffer syntax the way that would affect the current syntax-ppss cache.
Or if they do, they should clear the cache accordingly.

Unfortunately, using syntax-ppss-flush-cache for that purpose doesn't
work: it makes Emacs infloop.

To reproduce, apply this patch:

diff --git a/lisp/progmodes/js.el b/lisp/progmodes/js.el
index 5ec3e94275..efe38049ab 100644
--- a/lisp/progmodes/js.el
+++ b/lisp/progmodes/js.el
@@ -2362,6 +2362,7 @@ js-syntax-propertize
                             (eval-when-compile (append "=({[,:;" 
'(nil))))))
             (put-text-property (match-beginning 1) (match-end 1)
                                'syntax-table (string-to-syntax "\"/"))
+           (syntax-ppss-flush-cache (match-beginning 0))
             (js-syntax-propertize-regexp end)))))
      ("\`\(#\)!" (1 "< b"))
      ("<" (0 (ignore

and then edit a file with js-mode containing

   a = /xyz/

bug#41195: 27.0.91; Using syntax-ppss-flush-cache in syntax-propertize-function infloops

[Stefan Monnier]

Version: 28.1

> In theory, syntax-propertize-function implementations shouldn't change
> buffer syntax the way that would affect the current syntax-ppss cache.
> Or if they do, they should clear the cache accordingly.
> Unfortunately, using syntax-ppss-flush-cache for that purpose doesn't
> work: it makes Emacs infloop.

I thought at first that the fix should go into `emacs-27`, but then you
correctly pointed out that the docstring of `syntax-propertize-function`
warned that `syntax-ppss-flush-cache` should not be called, so this bug
is really a known and documented limitation.

But indeed, it's a limitation that we should lift, so I installed the
patch below into `master`.


        Stefan


diff --git a/lisp/emacs-lisp/syntax.el b/lisp/emacs-lisp/syntax.el
index 3294378754a..46dc8d9ade8 100644
--- a/lisp/emacs-lisp/syntax.el
+++ b/lisp/emacs-lisp/syntax.el
@@ -63,9 +63,10 @@ syntax-propertize-function
 cannot be handled just by the buffer's syntax-table.
 
 The specified function may call `syntax-ppss' on any position
-before END, but it should not call `syntax-ppss-flush-cache',
-which means that it should not call `syntax-ppss' on some
-position and later modify the buffer on some earlier position.")
+before END, but if it calls `syntax-ppss' on some
+position and later modifies the buffer on some earlier position,
+then it is its responsability to call `syntax-ppss-flush-cache' to flush
+the now obsolete ppss info from the cache.")
 
 (defvar syntax-propertize-chunk-size 500)
 
@@ -320,6 +321,11 @@ syntax-propertize-via-font-lock
 (defvar-local syntax-ppss-table nil
   "Syntax-table to use during `syntax-ppss', if any.")
 
+(defvar-local syntax-propertize--inhibit-flush nil
+  "If non-nil, `syntax-ppss-flush-cache' only flushes the ppss cache.
+Otherwise it flushes both the ppss cache and the properties
+set by `syntax-propertize'")
+
 (defun syntax-propertize (pos)
   "Ensure that syntax-table properties are set until POS (a buffer point)."
   (when (< syntax-propertize--done pos)
@@ -375,8 +381,13 @@ syntax-propertize
               ;; (message "syntax-propertizing from %s to %s" start end)
               (remove-text-properties start end
                                       '(syntax-table nil syntax-multiline nil))
-              ;; Avoid recursion!
-              (let ((syntax-propertize--done most-positive-fixnum))
+              ;; Make sure we only let-bind it buffer-locally.
+              (make-local-variable 'syntax-propertize--inhibit-flush)
+              ;; Let-bind `syntax-propertize--done' to avoid infinite recursion!
+              (let ((syntax-propertize--done most-positive-fixnum)
+                    ;; Let `syntax-propertize-function' call
+                    ;; `syntax-ppss-flush-cache' without worries.
+                    (syntax-propertize--inhibit-flush t))
                 (funcall syntax-propertize-function start end)))))))))
 
 ;;; Link syntax-propertize with syntax.c.
@@ -455,7 +466,8 @@ 'syntax-ppss-after-change-function
 (defun syntax-ppss-flush-cache (beg &rest ignored)
   "Flush the cache of `syntax-ppss' starting at position BEG."
   ;; Set syntax-propertize to refontify anything past beg.
-  (setq syntax-propertize--done (min beg syntax-propertize--done))
+  (unless syntax-propertize--inhibit-flush
+    (setq syntax-propertize--done (min beg syntax-propertize--done)))
   ;; Flush invalid cache entries.
   (dolist (cell (list syntax-ppss-wide syntax-ppss-narrow))
     (pcase cell

bug#41195: 27.0.91; Using syntax-ppss-flush-cache in syntax-propertize-function infloops

[Dmitry Gutov]

On 12.05.2020 23:55, Stefan Monnier wrote:
> I thought at first that the fix should go into `emacs-27`, but then you
> correctly pointed out that the docstring of `syntax-propertize-function`
> warned that `syntax-ppss-flush-cache` should not be called, so this bug
> is really a known and documented limitation.
> 
> But indeed, it's a limitation that we should lift, so I installed the
> patch below into `master`.

OK. Thank you.