* bug#68445: [PATCH] Problem with python--treesit-syntax-propertize
[not found] <eke7il3w6ztw.wl-kobarity@gmail.com>
@ 2024-01-21 14:47 ` kobarity
2024-01-21 18:09 ` Dmitry Gutov
0 siblings, 1 reply; 6+ messages in thread
From: kobarity @ 2024-01-21 14:47 UTC (permalink / raw)
To: Yuan Fu, Dmitry Gutov; +Cc: 68445
I am resending my mail, as I made a mistake in X-Debbugs-CC.
I wrote:
> Hi,
>
> I found a problem with python--treesit-syntax-propertize recently
> introduced by the Bug#67977 patch.
>
> 1. emacs -Q
> 2. Open a file in python-ts-mode with the following contents:
>
> #+begin_src python
> """Docstring.
>
> test.
> """
> S = """string."""
> #+end_src
>
> 3. Locate the point on the third line.
> 4. M-q
> 5. An empty line will be inserted.
> 6. M-q
> 7. The string literal on the last line will be split as follows:
>
> S = ""
>
> "string."""
>
> This problem does not occur in python-mode.
>
> The direct cause of this problem is that the string-delimiter property
> set in the docstring is removed. python--treesit-syntax-propertize is
> called to set the property, but it fails to set it properly. Here is
> the trace of python--treesit-syntax-propertize from step 4 above.
>
> ======================================================================
> 1 -> (python--treesit-syntax-propertize 1 45)
> 1 <- python--treesit-syntax-propertize: nil
> ======================================================================
> 1 -> (python--treesit-syntax-propertize 16 45)
> 1 <- python--treesit-syntax-propertize: nil
>
> python--treesit-syntax-propertize is called with argument START 16.
> This is the position inside the docstring.
>
> It seems to me that python--treesit-syntax-propertize assumes that the
> START argument is outside the triple-quoted string. So one solution
> might be to change START to the start of the string if it is within a
> string, as in the attached patch. However, I'm not sure this is the
> right approach. Should we use
> syntax-propertize-extend-region-functions?
>
> --
> In GNU Emacs 30.0.50 (build 5, x86_64-pc-linux-gnu, X toolkit, cairo
> version 1.16.0, Xaw scroll bars) of 2024-01-13 built on ubuntu
> Repository revision: 106cd9aafe8248ef91d7e89161adc5f912ea54eb
> Repository branch: master
> System Description: Ubuntu 22.04.3 LTS
I appreciate your comments.
^ permalink raw reply [flat|nested] 6+ messages in thread
* bug#68445: [PATCH] Problem with python--treesit-syntax-propertize
2024-01-21 14:47 ` bug#68445: [PATCH] Problem with python--treesit-syntax-propertize kobarity
@ 2024-01-21 18:09 ` Dmitry Gutov
2024-01-22 15:44 ` kobarity
0 siblings, 1 reply; 6+ messages in thread
From: Dmitry Gutov @ 2024-01-21 18:09 UTC (permalink / raw)
To: kobarity, Yuan Fu; +Cc: 68445
Hi!
On 21/01/2024 16:47, kobarity wrote:
>
> I am resending my mail, as I made a mistake in X-Debbugs-CC.
Was it supposed to appear in the bug's thread? I don't see it anywhere.
> I wrote:
>> Hi,
>>
>> I found a problem with python--treesit-syntax-propertize recently
>> introduced by the Bug#67977 patch.
>>
>> 1. emacs -Q
>> 2. Open a file in python-ts-mode with the following contents:
>>
>> #+begin_src python
>> """Docstring.
>>
>> test.
>> """
>> S = """string."""
>> #+end_src
>>
>> 3. Locate the point on the third line.
>> 4. M-q
>> 5. An empty line will be inserted.
>> 6. M-q
>> 7. The string literal on the last line will be split as follows:
>>
>> S = ""
>>
>> "string."""
>>
>> This problem does not occur in python-mode.
>>
>> The direct cause of this problem is that the string-delimiter property
>> set in the docstring is removed. python--treesit-syntax-propertize is
>> called to set the property, but it fails to set it properly. Here is
>> the trace of python--treesit-syntax-propertize from step 4 above.
>>
>> ======================================================================
>> 1 -> (python--treesit-syntax-propertize 1 45)
>> 1 <- python--treesit-syntax-propertize: nil
>> ======================================================================
>> 1 -> (python--treesit-syntax-propertize 16 45)
>> 1 <- python--treesit-syntax-propertize: nil
>>
>> python--treesit-syntax-propertize is called with argument START 16.
>> This is the position inside the docstring.
>>
>> It seems to me that python--treesit-syntax-propertize assumes that the
>> START argument is outside the triple-quoted string. So one solution
>> might be to change START to the start of the string if it is within a
>> string, as in the attached patch. However, I'm not sure this is the
>> right approach.
Sounds good to me. I don't see the patch, though, or where to read it.
>> Should we use
>> syntax-propertize-extend-region-functions?
That's another option, but it shouldn't be necessary. After all, the
absence of a notification from the parser (which would extend the range)
should mean that the node before position 16 is untouched, so there's no
real need to clear the properties there.
I think there is also another approach--handle two different types of
nodes separately, instead of just string_content, so we don't have to
start from the beginning of the literal. Like this:
diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index e2f614f52c2..4f8b0cb9473 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -1361,13 +1361,15 @@ python--treesit-syntax-propertize
(while (re-search-forward (rx (or "\"\"\"" "'''")) end t)
(let ((node (treesit-node-at (point))))
;; The triple quotes surround a non-empty string.
- (when (equal (treesit-node-type node) "string_content")
- (let ((start (treesit-node-start node))
- (end (treesit-node-end node)))
- (put-text-property (1- start) start
- 'syntax-table (string-to-syntax "|"))
- (put-text-property end (min (1+ end) (point-max))
- 'syntax-table (string-to-syntax "|"))))))))
+ (cond
+ ((equal (treesit-node-type node) "string_content")
+ (put-text-property (1- (treesit-node-start node))
+ (treesit-node-start node)
+ 'syntax-table (string-to-syntax "|")))
+ ((and (equal (treesit-node-type node) "string_end")
+ (= (treesit-node-start node) (- (point) 3)))
+ (put-text-property (- (point) 3) (- (point) 2)
+ 'syntax-table (string-to-syntax "|"))))))))
\f
;;; Indentation
^ permalink raw reply related [flat|nested] 6+ messages in thread
* bug#68445: [PATCH] Problem with python--treesit-syntax-propertize
2024-01-21 18:09 ` Dmitry Gutov
@ 2024-01-22 15:44 ` kobarity
2024-01-22 18:52 ` Dmitry Gutov
0 siblings, 1 reply; 6+ messages in thread
From: kobarity @ 2024-01-22 15:44 UTC (permalink / raw)
To: Dmitry Gutov; +Cc: Yuan Fu, 68445
Hi,
Dmitry Gutov wrote:
> On 21/01/2024 16:47, kobarity wrote:
> > I am resending my mail, as I made a mistake in X-Debbugs-CC.
> Was it supposed to appear in the bug's thread? I don't see it anywhere.
My first mail was registered as Bug#68445, and my patch is there.
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=68445
It says:
Report forwarded to casouri <at> gmail.com, dmitry@.gutov.dev, bug-gnu-emacs <at> gnu.org:
The extra period is my mistake and it may have caused the problem.
I'm sorry for the confusion.
> I think there is also another approach--handle two different types of
> nodes separately, instead of just string_content, so we don't have to
> start from the beginning of the literal. Like this:
>
> diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
> index e2f614f52c2..4f8b0cb9473 100644
> --- a/lisp/progmodes/python.el
> +++ b/lisp/progmodes/python.el
> @@ -1361,13 +1361,15 @@ python--treesit-syntax-propertize
> (while (re-search-forward (rx (or "\"\"\"" "'''")) end t)
> (let ((node (treesit-node-at (point))))
> ;; The triple quotes surround a non-empty string.
> - (when (equal (treesit-node-type node) "string_content")
> - (let ((start (treesit-node-start node))
> - (end (treesit-node-end node)))
> - (put-text-property (1- start) start
> - 'syntax-table (string-to-syntax "|"))
> - (put-text-property end (min (1+ end) (point-max))
> - 'syntax-table (string-to-syntax "|"))))))))
> + (cond
> + ((equal (treesit-node-type node) "string_content")
> + (put-text-property (1- (treesit-node-start node))
> + (treesit-node-start node)
> + 'syntax-table (string-to-syntax "|")))
> + ((and (equal (treesit-node-type node) "string_end")
> + (= (treesit-node-start node) (- (point) 3)))
> + (put-text-property (- (point) 3) (- (point) 2)
> + 'syntax-table (string-to-syntax "|"))))))))
>
> \f
> ;;; Indentation
>
This approach seems better than my patch, but it does not seem to
address the following special case.
#+begin_src python
"""a""""""b"""
#+end_src
^ permalink raw reply [flat|nested] 6+ messages in thread
* bug#68445: [PATCH] Problem with python--treesit-syntax-propertize
2024-01-22 15:44 ` kobarity
@ 2024-01-22 18:52 ` Dmitry Gutov
2024-01-23 14:14 ` kobarity
0 siblings, 1 reply; 6+ messages in thread
From: Dmitry Gutov @ 2024-01-22 18:52 UTC (permalink / raw)
To: kobarity; +Cc: Yuan Fu, 68445
On 22/01/2024 17:44, kobarity wrote:
> Hi,
>
> Dmitry Gutov wrote:
>> On 21/01/2024 16:47, kobarity wrote:
>>> I am resending my mail, as I made a mistake in X-Debbugs-CC.
>> Was it supposed to appear in the bug's thread? I don't see it anywhere.
>
> My first mail was registered as Bug#68445, and my patch is there.
>
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=68445
>
> It says:
>
> Report forwarded to casouri <at> gmail.com, dmitry@.gutov.dev, bug-gnu-emacs <at> gnu.org:
>
> The extra period is my mistake and it may have caused the problem.
> I'm sorry for the confusion.
Yeah, but even so that's odd: I'm subscribed to the bug tracker, so the
email should have at least arrived in my inbox, but it did not.
>> I think there is also another approach--handle two different types of
>> nodes separately, instead of just string_content, so we don't have to
>> start from the beginning of the literal. Like this:
>>
>> diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
>> index e2f614f52c2..4f8b0cb9473 100644
>> --- a/lisp/progmodes/python.el
>> +++ b/lisp/progmodes/python.el
>> @@ -1361,13 +1361,15 @@ python--treesit-syntax-propertize
>> (while (re-search-forward (rx (or "\"\"\"" "'''")) end t)
>> (let ((node (treesit-node-at (point))))
>> ;; The triple quotes surround a non-empty string.
>> - (when (equal (treesit-node-type node) "string_content")
>> - (let ((start (treesit-node-start node))
>> - (end (treesit-node-end node)))
>> - (put-text-property (1- start) start
>> - 'syntax-table (string-to-syntax "|"))
>> - (put-text-property end (min (1+ end) (point-max))
>> - 'syntax-table (string-to-syntax "|"))))))))
>> + (cond
>> + ((equal (treesit-node-type node) "string_content")
>> + (put-text-property (1- (treesit-node-start node))
>> + (treesit-node-start node)
>> + 'syntax-table (string-to-syntax "|")))
>> + ((and (equal (treesit-node-type node) "string_end")
>> + (= (treesit-node-start node) (- (point) 3)))
>> + (put-text-property (- (point) 3) (- (point) 2)
>> + 'syntax-table (string-to-syntax "|"))))))))
>>
>> \f
>> ;;; Indentation
>>
>
> This approach seems better than my patch, but it does not seem to
> address the following special case.
>
> #+begin_src python
> """a""""""b"""
> #+end_src
All right, try the patch below, please. It also covers the case of the
empty literal.
I've tried to find a case where it would behave poorly (e.g. by
misdetecting three quotes from a combination of some other string
literals), but couldn't. E.g.,
s = '''asdasd'
is not a concatenation. It's always an error, at least according to the
TS grammar.
diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index e2f614f52c2..41f612c8b1c 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -1359,15 +1359,15 @@ python--treesit-syntax-propertize
(save-excursion
(goto-char start)
(while (re-search-forward (rx (or "\"\"\"" "'''")) end t)
- (let ((node (treesit-node-at (point))))
- ;; The triple quotes surround a non-empty string.
- (when (equal (treesit-node-type node) "string_content")
- (let ((start (treesit-node-start node))
- (end (treesit-node-end node)))
- (put-text-property (1- start) start
- 'syntax-table (string-to-syntax "|"))
- (put-text-property end (min (1+ end) (point-max))
- 'syntax-table (string-to-syntax "|"))))))))
+ (let ((node (treesit-node-at (- (point) 3))))
+ ;; Handle triple-quoted strings.
+ (pcase (treesit-node-type node)
+ ("string_start"
+ (put-text-property (1- (point)) (point)
+ 'syntax-table (string-to-syntax "|")))
+ ("string_end"
+ (put-text-property (- (point) 3) (- (point) 2)
+ 'syntax-table (string-to-syntax "|"))))))))
\f
;;; Indentation
^ permalink raw reply related [flat|nested] 6+ messages in thread
* bug#68445: [PATCH] Problem with python--treesit-syntax-propertize
2024-01-22 18:52 ` Dmitry Gutov
@ 2024-01-23 14:14 ` kobarity
2024-01-26 1:15 ` Dmitry Gutov
0 siblings, 1 reply; 6+ messages in thread
From: kobarity @ 2024-01-23 14:14 UTC (permalink / raw)
To: Dmitry Gutov; +Cc: Yuan Fu, 68445
Dmitry Gutov wrote:
>
> On 22/01/2024 17:44, kobarity wrote:
> > Hi,
> >
> > Dmitry Gutov wrote:
> >> On 21/01/2024 16:47, kobarity wrote:
> >>> I am resending my mail, as I made a mistake in X-Debbugs-CC.
> >> Was it supposed to appear in the bug's thread? I don't see it anywhere.
> >
> > My first mail was registered as Bug#68445, and my patch is there.
> >
> > https://debbugs.gnu.org/cgi/bugreport.cgi?bug=68445
> >
> > It says:
> >
> > Report forwarded to casouri <at> gmail.com, dmitry@.gutov.dev, bug-gnu-emacs <at> gnu.org:
> >
> > The extra period is my mistake and it may have caused the problem.
> > I'm sorry for the confusion.
>
> Yeah, but even so that's odd: I'm subscribed to the bug tracker, so
> the email should have at least arrived in my inbox, but it did not.
I agree. I can't find my first mail in the bug-gnu-emacs archive.
> >> I think there is also another approach--handle two different types of
> >> nodes separately, instead of just string_content, so we don't have to
> >> start from the beginning of the literal. Like this:
> >>
> >> diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
> >> index e2f614f52c2..4f8b0cb9473 100644
> >> --- a/lisp/progmodes/python.el
> >> +++ b/lisp/progmodes/python.el
> >> @@ -1361,13 +1361,15 @@ python--treesit-syntax-propertize
> >> (while (re-search-forward (rx (or "\"\"\"" "'''")) end t)
> >> (let ((node (treesit-node-at (point))))
> >> ;; The triple quotes surround a non-empty string.
> >> - (when (equal (treesit-node-type node) "string_content")
> >> - (let ((start (treesit-node-start node))
> >> - (end (treesit-node-end node)))
> >> - (put-text-property (1- start) start
> >> - 'syntax-table (string-to-syntax "|"))
> >> - (put-text-property end (min (1+ end) (point-max))
> >> - 'syntax-table (string-to-syntax "|"))))))))
> >> + (cond
> >> + ((equal (treesit-node-type node) "string_content")
> >> + (put-text-property (1- (treesit-node-start node))
> >> + (treesit-node-start node)
> >> + 'syntax-table (string-to-syntax "|")))
> >> + ((and (equal (treesit-node-type node) "string_end")
> >> + (= (treesit-node-start node) (- (point) 3)))
> >> + (put-text-property (- (point) 3) (- (point) 2)
> >> + 'syntax-table (string-to-syntax "|"))))))))
> >>
> >> \f
> >> ;;; Indentation
> >>
> >
> > This approach seems better than my patch, but it does not seem to
> > address the following special case.
> >
> > #+begin_src python
> > """a""""""b"""
> > #+end_src
>
> All right, try the patch below, please. It also covers the case of the
> empty literal.
Thanks, it looks good to me.
> I've tried to find a case where it would behave poorly (e.g. by
> misdetecting three quotes from a combination of some other string
> literals), but couldn't. E.g.,
>
> s = '''asdasd'
>
> is not a concatenation. It's always an error, at least according to
> the TS grammar.
I think the TS grammar is correct, because this example is also an
error according to the Python interpreter.
^ permalink raw reply [flat|nested] 6+ messages in thread
* bug#68445: [PATCH] Problem with python--treesit-syntax-propertize
2024-01-23 14:14 ` kobarity
@ 2024-01-26 1:15 ` Dmitry Gutov
0 siblings, 0 replies; 6+ messages in thread
From: Dmitry Gutov @ 2024-01-26 1:15 UTC (permalink / raw)
To: kobarity; +Cc: Yuan Fu, 68445-done
On 23/01/2024 16:14, kobarity wrote:
>
> Dmitry Gutov wrote:
>>
>> On 22/01/2024 17:44, kobarity wrote:
>>> Hi,
>>>
>>> Dmitry Gutov wrote:
>>>> On 21/01/2024 16:47, kobarity wrote:
>>>>> I am resending my mail, as I made a mistake in X-Debbugs-CC.
>>>> Was it supposed to appear in the bug's thread? I don't see it anywhere.
>>>
>>> My first mail was registered as Bug#68445, and my patch is there.
>>>
>>> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=68445
>>>
>>> It says:
>>>
>>> Report forwarded to casouri <at> gmail.com, dmitry@.gutov.dev, bug-gnu-emacs <at> gnu.org:
>>>
>>> The extra period is my mistake and it may have caused the problem.
>>> I'm sorry for the confusion.
>>
>> Yeah, but even so that's odd: I'm subscribed to the bug tracker, so
>> the email should have at least arrived in my inbox, but it did not.
>
> I agree. I can't find my first mail in the bug-gnu-emacs archive.
>
>>>> I think there is also another approach--handle two different types of
>>>> nodes separately, instead of just string_content, so we don't have to
>>>> start from the beginning of the literal. Like this:
>>>>
>>>> diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
>>>> index e2f614f52c2..4f8b0cb9473 100644
>>>> --- a/lisp/progmodes/python.el
>>>> +++ b/lisp/progmodes/python.el
>>>> @@ -1361,13 +1361,15 @@ python--treesit-syntax-propertize
>>>> (while (re-search-forward (rx (or "\"\"\"" "'''")) end t)
>>>> (let ((node (treesit-node-at (point))))
>>>> ;; The triple quotes surround a non-empty string.
>>>> - (when (equal (treesit-node-type node) "string_content")
>>>> - (let ((start (treesit-node-start node))
>>>> - (end (treesit-node-end node)))
>>>> - (put-text-property (1- start) start
>>>> - 'syntax-table (string-to-syntax "|"))
>>>> - (put-text-property end (min (1+ end) (point-max))
>>>> - 'syntax-table (string-to-syntax "|"))))))))
>>>> + (cond
>>>> + ((equal (treesit-node-type node) "string_content")
>>>> + (put-text-property (1- (treesit-node-start node))
>>>> + (treesit-node-start node)
>>>> + 'syntax-table (string-to-syntax "|")))
>>>> + ((and (equal (treesit-node-type node) "string_end")
>>>> + (= (treesit-node-start node) (- (point) 3)))
>>>> + (put-text-property (- (point) 3) (- (point) 2)
>>>> + 'syntax-table (string-to-syntax "|"))))))))
>>>>
>>>> \f
>>>> ;;; Indentation
>>>>
>>>
>>> This approach seems better than my patch, but it does not seem to
>>> address the following special case.
>>>
>>> #+begin_src python
>>> """a""""""b"""
>>> #+end_src
>>
>> All right, try the patch below, please. It also covers the case of the
>> empty literal.
>
> Thanks, it looks good to me.
>
>> I've tried to find a case where it would behave poorly (e.g. by
>> misdetecting three quotes from a combination of some other string
>> literals), but couldn't. E.g.,
>>
>> s = '''asdasd'
>>
>> is not a concatenation. It's always an error, at least according to
>> the TS grammar.
>
> I think the TS grammar is correct, because this example is also an
> error according to the Python interpreter.
Thanks for testing! Installed, and closing.
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2024-01-26 1:15 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <eke7il3w6ztw.wl-kobarity@gmail.com>
2024-01-21 14:47 ` bug#68445: [PATCH] Problem with python--treesit-syntax-propertize kobarity
2024-01-21 18:09 ` Dmitry Gutov
2024-01-22 15:44 ` kobarity
2024-01-22 18:52 ` Dmitry Gutov
2024-01-23 14:14 ` kobarity
2024-01-26 1:15 ` Dmitry Gutov
Code repositories for project(s) associated with this external index
https://git.savannah.gnu.org/cgit/emacs.git
https://git.savannah.gnu.org/cgit/emacs/org-mode.git
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.