bug#59679: 29.0.50; python + tree-sitter: Syntax highlighting is sometimes wrong with multi line docstrings

bug#59679: 29.0.50; python + tree-sitter: Syntax highlighting is sometimes wrong with multi line docstrings

[Matthias Paulmier]

Steps to reproduce:
- emacs -Q
- find-file "foo.py"
- write the following file

```
def test():
      '''This is a function

      It just does nothing and it's ok.
      '''
      pass
```
- at this point, syntax highlighting should be good
- kill-buffer
- find-file "foo.py"
- syntax highlighting doesn't show properly (comment ends after "it'"
and pass is shown with the `font-lock-doc-face')


In GNU Emacs 29.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version
3.24.20, cairo version 1.16.0) of 2022-11-23 built on 20045CLT490
Repository revision: 43060910874f46579fa190d9b5534508e4ea058e
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12013000
System Description: Ubuntu 20.04.5 LTS

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LCMS2 LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 M17N_FLT MODULES NOTIFY
INOTIFY PDUMPER PNG RSVG SECCOMP SOUND SQLITE3 THREADS TIFF
TOOLKIT_SCROLL_BARS X11 XDBE XIM XINPUT2 XPM GTK3 ZLIB

Important settings:
value of $LC_ALL: fr_FR.UTF-8
value of $LC_CTYPE: fr_FR.UTF-8
value of $LANG: fr_FR
locale-coding-system: utf-8-unix

Major mode: Python

Minor modes in effect:
tooltip-mode: t
global-eldoc-mode: t
eldoc-mode: t
show-paren-mode: t
electric-indent-mode: t
mouse-wheel-mode: t
tool-bar-mode: t
menu-bar-mode: t
file-name-shadow-mode: t
global-font-lock-mode: t
font-lock-mode: t
blink-cursor-mode: t
line-number-mode: t
transient-mark-mode: t
auto-composition-mode: t
auto-encryption-mode: t
auto-compression-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message mailcap yank-media puny dired
dired-loaddefs rfc822 mml mml-sec password-cache epa derived epg rfc6068
epg-config gnus-util text-property-search time-date subr-x mm-decode
mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils vc-hg
vc-git diff-mode vc-bzr vc-src vc-sccs vc-svn vc-cvs vc-rcs log-view
easy-mmode pcvs-util vc vc-dispatcher python rx project byte-opt gv
bytecomp byte-compile pcase treesit cl-seq comint ansi-osc ring
cl-loaddefs cl-lib ansi-color rmc iso-transl tooltip cconv eldoc paren
electric uniquify ediff-hook vc-hooks lisp-float-type elisp-mode mwheel
term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu
timer select scroll-bar mouse jit-lock font-lock syntax font-core
term/tty-colors frame minibuffer nadvice seq simple cl-generic
indonesian philippine cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs
theme-loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads dbusbind inotify lcms2
dynamic-setting system-font-setting font-render-setting cairo
move-toolbar gtk x-toolkit xinput2 x multi-tty make-network-process
emacs)

Memory information:
((conses 16 67878 7612)
(symbols 48 8488 0)
(strings 32 25282 1967)
(string-bytes 1 742048)
(vectors 16 16094)
(vector-slots 8 211336 10774)
(floats 8 30 47)
(intervals 56 389 0)
(buffers 984 13))

bug#59679: 29.0.50; python + tree-sitter: Syntax highlighting is sometimes wrong with multi line docstrings

[Yuan Fu]

Matthias Paulmier <hello@mattplm.fr> writes:

> Steps to reproduce:
> - emacs -Q
> - find-file "foo.py"
> - write the following file
>
> ```
> def test():
>      '''This is a function
>
>      It just does nothing and it's ok.
>      '''
>      pass
> ```
> - at this point, syntax highlighting should be good
> - kill-buffer
> - find-file "foo.py"
> - syntax highlighting doesn't show properly (comment ends after "it'"
> and pass is shown with the `font-lock-doc-face')

Hmmm, it seems the tabs in the code throws tree-sitter off and makes it
parse incorrectly. But I’m not sure if the tabs are in your code or they
are added by the email provider. Do you expect the code to contain tabs?

Yuan

bug#59679: 29.0.50; python + tree-sitter: Syntax highlighting is sometimes wrong with multi line docstrings

[Matthias Paulmier]

> Hmmm, it seems the tabs in the code throws tree-sitter off and makes it
> parse incorrectly. But I’m not sure if the tabs are in your code or they
> are added by the email provider. Do you expect the code to contain tabs?

Hello Yuan,

Thank you for looking at the issue.

The actual file doesn't contain any tabs (they are 4 spaces indent), 
must be my email client trying to format it.

After reflecting a bit, I'm not sure it is related in any way to 
treesitter. I'm not even sure python + treesitter is a thing in emacs in 
29.0.50 (not according to etc/NEWS at least).

I can just confirm that versions <28.2 don't show this issue, that's why 
I thought about treesitter.

--
Matthias

bug#59679: 29.0.50; python + tree-sitter: Syntax highlighting is sometimes wrong with multi line docstrings

[Matthias Paulmier]

On 30/11/2022 12:06, Matthias Paulmier wrote:
>
>> Hmmm, it seems the tabs in the code throws tree-sitter off and makes it
>> parse incorrectly. But I’m not sure if the tabs are in your code or they
>> are added by the email provider. Do you expect the code to contain tabs?
>
> Hello Yuan,
>
> Thank you for looking at the issue.
>
> The actual file doesn't contain any tabs (they are 4 spaces indent), 
> must be my email client trying to format it.
>
> After reflecting a bit, I'm not sure it is related in any way to 
> treesitter. I'm not even sure python + treesitter is a thing in emacs 
> in 29.0.50 (not according to etc/NEWS at least).
>
> I can just confirm that versions <28.2 don't show this issue, that's 
> why I thought about treesitter.
>
> -- 
> Matthias


Hello again.

Turns out I had a slightly out of date version (from last week). I 
recompiled and don't have this problem anymore.

Seems to have been resolved in fc9d7b3d04.

Sorry for the confusion.

--

Matthias

bug#59679: 29.0.50; python + tree-sitter: Syntax highlighting is sometimes wrong with multi line docstrings

[Yuan Fu]

Matthias Paulmier <hello@mattplm.fr> writes:

> On 30/11/2022 12:06, Matthias Paulmier wrote:
>>
>>> Hmmm, it seems the tabs in the code throws tree-sitter off and makes it
>>> parse incorrectly. But I’m not sure if the tabs are in your code or they
>>> are added by the email provider. Do you expect the code to contain tabs?
>>
>> Hello Yuan,
>>
>> Thank you for looking at the issue.
>>
>> The actual file doesn't contain any tabs (they are 4 spaces indent),
>> must be my email client trying to format it.
>>
>> After reflecting a bit, I'm not sure it is related in any way to
>> treesitter. I'm not even sure python + treesitter is a thing in
>> emacs in 29.0.50 (not according to etc/NEWS at least).
>>
>> I can just confirm that versions <28.2 don't show this issue, that's
>> why I thought about treesitter.
>>
>> -- Matthias
>
>
> Hello again.
>
> Turns out I had a slightly out of date version (from last week). I
> recompiled and don't have this problem anymore.
>
> Seems to have been resolved in fc9d7b3d04.
>
> Sorry for the confusion.

Don’t be sorry :-) Thanks for reporting the bug! I’m closing this report.

Yuan