bug#68445: [PATCH] Problem with python--treesit-syntax-propertize

[Dmitry Gutov <dmitry@gutov.dev>]

Hi!

On 21/01/2024 16:47, kobarity wrote:
> 
> I am resending my mail, as I made a mistake in X-Debbugs-CC.

Was it supposed to appear in the bug's thread? I don't see it anywhere.

> I wrote:
>> Hi,
>>
>> I found a problem with python--treesit-syntax-propertize recently
>> introduced by the Bug#67977 patch.
>>
>> 1. emacs -Q
>> 2. Open a file in python-ts-mode with the following contents:
>>
>> #+begin_src python
>> """Docstring.
>>
>> test.
>> """
>> S = """string."""
>> #+end_src
>>
>> 3. Locate the point on the third line.
>> 4. M-q
>> 5. An empty line will be inserted.
>> 6. M-q
>> 7. The string literal on the last line will be split as follows:
>>
>> S = ""
>>
>> "string."""
>>
>> This problem does not occur in python-mode.
>>
>> The direct cause of this problem is that the string-delimiter property
>> set in the docstring is removed.  python--treesit-syntax-propertize is
>> called to set the property, but it fails to set it properly.  Here is
>> the trace of python--treesit-syntax-propertize from step 4 above.
>>
>> ======================================================================
>> 1 -> (python--treesit-syntax-propertize 1 45)
>> 1 <- python--treesit-syntax-propertize: nil
>> ======================================================================
>> 1 -> (python--treesit-syntax-propertize 16 45)
>> 1 <- python--treesit-syntax-propertize: nil
>>
>> python--treesit-syntax-propertize is called with argument START 16.
>> This is the position inside the docstring.
>>
>> It seems to me that python--treesit-syntax-propertize assumes that the
>> START argument is outside the triple-quoted string.  So one solution
>> might be to change START to the start of the string if it is within a
>> string, as in the attached patch.  However, I'm not sure this is the
>> right approach.

Sounds good to me. I don't see the patch, though, or where to read it.

>> Should we use
>> syntax-propertize-extend-region-functions?

That's another option, but it shouldn't be necessary. After all, the 
absence of a notification from the parser (which would extend the range) 
should mean that the node before position 16 is untouched, so there's no 
real need to clear the properties there.

I think there is also another approach--handle two different types of 
nodes separately, instead of just string_content, so we don't have to 
start from the beginning of the literal. Like this:

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index e2f614f52c2..4f8b0cb9473 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -1361,13 +1361,15 @@ python--treesit-syntax-propertize
      (while (re-search-forward (rx (or "\"\"\"" "'''")) end t)
        (let ((node (treesit-node-at (point))))
          ;; The triple quotes surround a non-empty string.
-        (when (equal (treesit-node-type node) "string_content")
-          (let ((start (treesit-node-start node))
-                (end (treesit-node-end node)))
-            (put-text-property (1- start) start
-                               'syntax-table (string-to-syntax "|"))
-            (put-text-property end (min (1+ end) (point-max))
-                               'syntax-table (string-to-syntax "|"))))))))
+        (cond
+         ((equal (treesit-node-type node) "string_content")
+          (put-text-property (1- (treesit-node-start node))
+                             (treesit-node-start node)
+                             'syntax-table (string-to-syntax "|")))
+         ((and (equal (treesit-node-type node) "string_end")
+               (= (treesit-node-start node) (- (point) 3)))
+          (put-text-property (- (point) 3) (- (point) 2)
+                             'syntax-table (string-to-syntax "|"))))))))

  \f
  ;;; Indentation