Tree-sitter integration in python.el

Tree-sitter integration in python.el

[Yuan Fu]

Hi, 

I’ve added tree-sitter version for font-lock and which-func in python.el. And I’d love to hear some feedback from python.el maintainers. Specifically, does it look right and which other part of python.el could actually benefit from a parse tree? I wrote a tree-sitter imenu indexer for python and it performed worse than the current one, presumably because it traverses the whole parse tree whereas the current one only scans the buffer once or so and do some regex matching.

Here is the commit: https://git.savannah.gnu.org/cgit/emacs.git/commit/?h=feature/tree-sitter&id=1cdb24fe35a9ff2e4f92c5acc93a5a5b0e70d93f

Yuan

Re: Tree-sitter integration in python.el

[Jostein Kjønigsen]

[-- Attachment #1: Type: text/plain, Size: 2295 bytes --]

On 22.09.2022 20:42, Yuan Fu wrote:
> Hi, 
> 
> I’ve added tree-sitter version for font-lock and which-func in python.el. And I’d love to hear some feedback from python.el maintainers. Specifically, does it look right and which other part of python.el could actually benefit from a parse tree? I wrote a tree-sitter imenu indexer for python and it performed worse than the current one, presumably because it traverses the whole parse tree whereas the current one only scans the buffer once or so and do some regex matching.
> 
> Here is the commit: https://git.savannah.gnu.org/cgit/emacs.git/commit/?h=feature/tree-sitter&id=1cdb24fe35a9ff2e4f92c5acc93a5a5b0e70d93f
> 
> Yuan
> 
> 
Hey Yuan.

Thanks for putting in all this work into tree-sitter in Emacs!

Trying the latest Emacs-version in feature/tree-sitter I got an error I tend to get "a lot" with tree-sitter based modes, which I hoped bundling things with Emacs would solve, namely obtaining the original shared-object containing the compiled grammer.

Like for your python-mode, I get this:

File mode specification error: (treesit-load-language-error python (/home/jostein/.emacs.d/tree-sitter/libtree-sitter-python: cannot open shared object file: No such file or directory /home/jostein/.emacs.d/tree-sitter/libtree-sitter-python.so: cannot open shared object file: No such file or directory libtree-sitter-python: cannot open shared object file: No such file or directory libtree-sitter-python.so: cannot open shared object file: No such file or directory))

I realize third-party modes are on their own, but when tree-sitter is compiled with Emacs, I would at least expect the Emacs-build to also produce these .so-files.

What are your thoughts on how we can best, across the Emacs-verse, provide these libraries? Or at least for the modes which are bundled with Emacs itself?

Being a tree-sitter based-developer myself, I know where I can go to get this compiled to make the mode runnable, but surely that's not how we can deploy this en-masse. Most people will be stuck at this point, and we will need to come up with a better answer.

-- 
 Kind regards
*Jostein Kjønigsen*

jostein@kjonigsen.net 🍵 jostein@gmail.com
https://jostein.kjønigsen.no <https://jostein.xn--kjnigsen-64a.no/>

[-- Attachment #2: Type: text/html, Size: 3638 bytes --]

Re: Tree-sitter integration in python.el

[Yuan Fu]

> On Sep 26, 2022, at 12:10 PM, Jostein Kjønigsen <jostein@secure.kjonigsen.net> wrote:
> 
> On 22.09.2022 20:42, Yuan Fu wrote:
>> Hi, 
>> 
>> I’ve added tree-sitter version for font-lock and which-func in python.el. And I’d love to hear some feedback from python.el maintainers. Specifically, does it look right and which other part of python.el could actually benefit from a parse tree? I wrote a tree-sitter imenu indexer for python and it performed worse than the current one, presumably because it traverses the whole parse tree whereas the current one only scans the buffer once or so and do some regex matching.
>> 
>> Here is the commit: 
>> https://git.savannah.gnu.org/cgit/emacs.git/commit/?h=feature/tree-sitter&id=1cdb24fe35a9ff2e4f92c5acc93a5a5b0e70d93f
>> 
>> 
>> Yuan
>> 
>> 
>> 
> Hey Yuan.
> 
> Thanks for putting in all this work into tree-sitter in Emacs!
> 
> Trying the latest Emacs-version in feature/tree-sitter I got an error I tend to get "a lot" with tree-sitter based modes, which I hoped bundling things with Emacs would solve, namely obtaining the original shared-object containing the compiled grammer.
> 
> Like for your python-mode, I get this:
> 
> File mode specification error: (treesit-load-language-error python (/home/jostein/.emacs.d/tree-sitter/libtree-sitter-python: cannot open shared object file: No such file or directory /home/jostein/.emacs.d/tree-sitter/libtree-sitter-python.so: cannot open shared object file: No such file or directory libtree-sitter-python: cannot open shared object file: No such file or directory libtree-sitter-python.so: cannot open shared object file: No such file or directory))
> 
> I realize third-party modes are on their own, but when tree-sitter is compiled with Emacs, I would at least expect the Emacs-build to also produce these .so-files.
> 
> What are your thoughts on how we can best, across the Emacs-verse, provide these libraries? Or at least for the modes which are bundled with Emacs itself?
> 
> Being a tree-sitter based-developer myself, I know where I can go to get this compiled to make the mode runnable, but surely that's not how we can deploy this en-masse. Most people will be stuck at this point, and we will need to come up with a better answer.
> 
> -- 
>  Kind regards
> Jostein Kjønigsen
> 
> jostein@kjonigsen.net 🍵 jostein@gmail.com
> https://jostein.kjønigsen.no

This has been discussed before and our conclusion is to treat language definitions like other dynamic libraries, expecting package manager to install them. There are a couple of reasons. Tree-sitter library and language definitions must be on the same “protocol” version to work. I expect this version to change slowly, but in principle we don’t want to bundle a language definition that could conflict with the tree-sitter library provided by the system. Also, as a dynamic object file, it is machine-dependent. I don’t know if we bundle anything machine-dependent in Emacs distribution, but at any rate it is unusual. 

I do have some ideas to make it easier for users, like hosting them on ELPA or NONGNU ELPA, and user can install them by package-install. Eg, a package tree-sitter-installer-linux, and M-x tree-sitter-installer-linux RET c RET will install C language definition for you. Stefan, WDYT?

Yuan

Re: Tree-sitter integration in python.el

[Matthias Meulien]

Yuan Fu <casouri@gmail.com> writes:

> I’ve added tree-sitter version for font-lock and which-func in
> python.el. And I’d love to hear some feedback from python.el
> maintainers.

I'd like to test your integration but... I am new to tree-sitter.

I've cloned the Git tree-sitter repository, make, make install, call to
ldconfig.  The generated shared object is known to the linker and I
successfully build the feature/tree-sitter branch of emacs (it starts
succesfully and I see symbols under the treesit- prefix).

I also cloned tree-sitter-python but I've no idea of what should be done
with the grammar file.  I've understood that I am suppposed to convert
the grammar to a shared object but I don't know how...

(I've tree-sitter-cli installed and in path, but "tree-sitter generate
tree-sitter-python/grammar.js" seems to generate rust and node bindings;
Am I supposed to compile the src/parser.c file?  I don't see any
Makefile...).
-- 
Matthias

Re: Tree-sitter integration in python.el

[Eli Zaretskii]

> From: Matthias Meulien <orontee@gmail.com>
> Cc: emacs-devel <emacs-devel@gnu.org>
> Date: Mon, 03 Oct 2022 20:07:44 +0200
> 
> I also cloned tree-sitter-python but I've no idea of what should be done
> with the grammar file.  I've understood that I am suppposed to convert
> the grammar to a shared object but I don't know how...
> 
> (I've tree-sitter-cli installed and in path, but "tree-sitter generate
> tree-sitter-python/grammar.js" seems to generate rust and node bindings;
> Am I supposed to compile the src/parser.c file?  I don't see any
> Makefile...).

Some of these lack the Makefile.  But the Makefile is standard and can
be taken from any other language module, for example I see on in
tree-sitter-ruby.  So just clone that as well, copy its Makefile into
tree-sitter-python, and run "make".

Re: Tree-sitter integration in python.el

[Matthias Meulien]

[-- Attachment #1: Type: text/plain, Size: 1542 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Matthias Meulien <orontee@gmail.com>
>> Cc: emacs-devel <emacs-devel@gnu.org>
>> Date: Mon, 03 Oct 2022 20:07:44 +0200
>> 
>> I also cloned tree-sitter-python but I've no idea of what should be done
>> with the grammar file.  I've understood that I am suppposed to convert
>> the grammar to a shared object but I don't know how...
>> 
>> (I've tree-sitter-cli installed and in path, but "tree-sitter generate
>> tree-sitter-python/grammar.js" seems to generate rust and node bindings;
>> Am I supposed to compile the src/parser.c file?  I don't see any
>> Makefile...).
>
> Some of these lack the Makefile.  But the Makefile is standard and can
> be taken from any other language module, for example I see on in
> tree-sitter-ruby.  So just clone that as well, copy its Makefile into
> tree-sitter-python, and run "make".

Thank you, Eli.  It wasn't as easy as you said but I managed to compile
a shared object using a modified version of the Makefile found in the
tree-sitter-ruby (using the file as is fails when building
tree-sitter-{ruby,python} due to a missing -Wl, before -soname usage).

At first sight, moving commands and imenu seems to work as
expected.  Fontification too.

I saw small differences between buffers fontified with
`python-use-tree-sitter' equal to nil and t.  Attached is a screenshot
where one can see two such differences (the third tall buffer shows the
output of "tree-sitter highlight"):

- line 284: The self attribute

- line 296: The variable in the f-string


[-- Attachment #2: Capture d’écran de 2022-10-04 00-07-27.png --]
[-- Type: image/png, Size: 599187 bytes --]

[-- Attachment #3: Type: text/plain, Size: 76 bytes --]


I'll keep using this branch and collect unexpected behaviors.
-- 
Matthias

Re: Tree-sitter integration in python.el

[Yuan Fu]

> On Oct 3, 2022, at 3:19 PM, Matthias Meulien <orontee@gmail.com> wrote:
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
>>> From: Matthias Meulien <orontee@gmail.com>
>>> Cc: emacs-devel <emacs-devel@gnu.org>
>>> Date: Mon, 03 Oct 2022 20:07:44 +0200
>>> 
>>> I also cloned tree-sitter-python but I've no idea of what should be done
>>> with the grammar file.  I've understood that I am suppposed to convert
>>> the grammar to a shared object but I don't know how...
>>> 
>>> (I've tree-sitter-cli installed and in path, but "tree-sitter generate
>>> tree-sitter-python/grammar.js" seems to generate rust and node bindings;
>>> Am I supposed to compile the src/parser.c file?  I don't see any
>>> Makefile...).
>> 
>> Some of these lack the Makefile.  But the Makefile is standard and can
>> be taken from any other language module, for example I see on in
>> tree-sitter-ruby.  So just clone that as well, copy its Makefile into
>> tree-sitter-python, and run "make".
> 
> Thank you, Eli.  It wasn't as easy as you said but I managed to compile
> a shared object using a modified version of the Makefile found in the
> tree-sitter-ruby (using the file as is fails when building
> tree-sitter-{ruby,python} due to a missing -Wl, before -soname usage).
> 
> At first sight, moving commands and imenu seems to work as
> expected.  Fontification too.

I didn’t modify and move commands. There are a lot of them and I don’t know if tree-sitter would bring any improvement. Imenu and font-lock are indeed powered by tree-sitter.

> 
> I saw small differences between buffers fontified with
> `python-use-tree-sitter' equal to nil and t.  Attached is a screenshot
> where one can see two such differences (the third tall buffer shows the
> output of "tree-sitter highlight"):
> 
> - line 284: The self attribute

Why is the _url attribute highlighted? Other attributes don’t seem to be highlighted.

> 
> - line 296: The variable in the f-string

That is expected? I added that to add a bit of flair to tree-sitter font-lock ;-)

> 
> <Capture d’écran de 2022-10-04 00-07-27.png>
> I'll keep using this branch and collect unexpected behaviors.

Thanks!

Yuan

Re: Tree-sitter integration in python.el

[Matthias Meulien]

Yuan Fu <casouri@gmail.com> writes:

>> On Oct 3, 2022, at 3:19 PM, Matthias Meulien <orontee@gmail.com> wrote:

>> (...)  At first sight, moving commands and imenu seems to work as
>> expected.  Fontification too.
>
> I didn’t modify and move commands. There are a lot of them and I don’t
> know if tree-sitter would bring any improvement. Imenu and font-lock
> are indeed powered by tree-sitter.

Ok.  Thanks for clarifying.

>> I saw small differences between buffers fontified with
>> `python-use-tree-sitter' equal to nil and t.  Attached is a screenshot
>> where one can see two such differences (the third tall buffer shows the
>> output of "tree-sitter highlight"):
>> 
>> - line 284: The self attribute
>
> Why is the _url attribute highlighted? Other attributes don’t seem to
> be highlighted.

I guess it's because it is on the left hand side of an assignment.

>> - line 296: The variable in the f-string
>
> That is expected? I added that to add a bit of flair to tree-sitter
> font-lock ;-)

I like it.
-- 
Matthias

Re: Tree-sitter integration in python.el

[Yuan Fu]

> On Oct 3, 2022, at 3:47 PM, Matthias Meulien <orontee@gmail.com> wrote:
> 
> Yuan Fu <casouri@gmail.com> writes:
> 
>>> On Oct 3, 2022, at 3:19 PM, Matthias Meulien <orontee@gmail.com> wrote:
> 
>>> (...)  At first sight, moving commands and imenu seems to work as
>>> expected.  Fontification too.
>> 
>> I didn’t modify and move commands. There are a lot of them and I don’t
>> know if tree-sitter would bring any improvement. Imenu and font-lock
>> are indeed powered by tree-sitter.
> 
> Ok.  Thanks for clarifying.
> 
>>> I saw small differences between buffers fontified with
>>> `python-use-tree-sitter' equal to nil and t.  Attached is a screenshot
>>> where one can see two such differences (the third tall buffer shows the
>>> output of "tree-sitter highlight"):
>>> 
>>> - line 284: The self attribute
>> 
>> Why is the _url attribute highlighted? Other attributes don’t seem to
>> be highlighted.
> 
> I guess it's because it is on the left hand side of an assignment.
> 

I’ve fixed fontification discrepancies you found and pushed the change.

Yuan

Re: Tree-sitter integration in python.el

[Matthias Meulien]

Yuan Fu <casouri@gmail.com> writes:

>>> (...)  Why is the _url attribute highlighted? Other attributes don’t
>>> seem to be highlighted.
>> 
>> I guess it's because it is on the left hand side of an assignment.
>> 
>
> I’ve fixed fontification discrepancies you found and pushed the change.

Thanks!

Happy to see that master was merged into feature/tree-sitter branch.
Unfortunately on my side it doesn't compile anymore.  Am I the only one
with that problem?

matthias@carbon:~/Sources/emacs (feature/tree-sitter ✕$ =)
↳ make clean && make distclean && ./configure --with-native-compilation
--with-x-toolkit=no && make -j6 bootstrap

In toplevel form:
mh-e/mh-folder.el:513:2: Error: Doc string is missing
make[4]: *** [Makefile:327 : mh-e/mh-folder.elc] Erreur 1
make[4]: *** Attente des tâches non terminées....
make[4] : on quitte le répertoire « /home/matthias/Sources/emacs/lisp »
make[3]: *** [Makefile:366 : compile-main] Erreur 2
make[3] : on quitte le répertoire « /home/matthias/Sources/emacs/lisp »
make[2]: *** [Makefile:531 : lisp] Erreur 2
make[2] : on quitte le répertoire « /home/matthias/Sources/emacs »
make[1]: *** [Makefile:1253 : actual-bootstrap] Erreur 2
make[1] : on quitte le répertoire « /home/matthias/Sources/emacs »
make[1] : on entre dans le répertoire « /home/matthias/Sources/emacs »
***
*** "make bootstrap" failed with exit status 2.


-- 
Matthias

Re: Tree-sitter integration in python.el

[Yuan Fu]

> On Oct 6, 2022, at 12:18 AM, Matthias Meulien <orontee@gmail.com> wrote:
> 
> Yuan Fu <casouri@gmail.com> writes:
> 
>>>> (...)  Why is the _url attribute highlighted? Other attributes don’t
>>>> seem to be highlighted.
>>> 
>>> I guess it's because it is on the left hand side of an assignment.
>>> 
>> 
>> I’ve fixed fontification discrepancies you found and pushed the change.
> 
> Thanks!
> 
> Happy to see that master was merged into feature/tree-sitter branch.
> Unfortunately on my side it doesn't compile anymore.  Am I the only one
> with that problem?
> 
> matthias@carbon:~/Sources/emacs (feature/tree-sitter ✕$ =)
> ↳ make clean && make distclean && ./configure --with-native-compilation
> --with-x-toolkit=no && make -j6 bootstrap
> 
> In toplevel form:
> mh-e/mh-folder.el:513:2: Error: Doc string is missing
> make[4]: *** [Makefile:327 : mh-e/mh-folder.elc] Erreur 1
> make[4]: *** Attente des tâches non terminées....
> make[4] : on quitte le répertoire « /home/matthias/Sources/emacs/lisp »
> make[3]: *** [Makefile:366 : compile-main] Erreur 2
> make[3] : on quitte le répertoire « /home/matthias/Sources/emacs/lisp »
> make[2]: *** [Makefile:531 : lisp] Erreur 2
> make[2] : on quitte le répertoire « /home/matthias/Sources/emacs »
> make[1]: *** [Makefile:1253 : actual-bootstrap] Erreur 2
> make[1] : on quitte le répertoire « /home/matthias/Sources/emacs »
> make[1] : on entre dans le répertoire « /home/matthias/Sources/emacs »
> ***
> *** "make bootstrap" failed with exit status 2.

Oops, I merged the master again and ran git clean -xf followed by make bootstrap, which compiled successfully. Something in the tree-sitter branch requires a git clean to build after merging master, it seems.

Yuan

Re: Tree-sitter integration in python.el

[Matthias Meulien]

Yuan Fu <casouri@gmail.com> writes:

> (...)  Oops, I merged the master again and ran git clean -xf followed
> by make bootstrap, which compiled successfully. Something in the
> tree-sitter branch requires a git clean to build after merging master,
> it seems.

Thank you, compilation is fixed.

I confirm that you also fixed the fontification problems I saw!

Would the integration of tree-sitter help to implement an option to have
all types fontified?  Now that it's becoming more popular to uses type
hints in Python code, it may improve code readability to have all types
fontified.
-- 
Matthias

Re: Tree-sitter integration in python.el

[Yuan Fu]

> On Oct 6, 2022, at 1:53 PM, Matthias Meulien <orontee@gmail.com> wrote:
> 
> Yuan Fu <casouri@gmail.com> writes:
> 
>> (...)  Oops, I merged the master again and ran git clean -xf followed
>> by make bootstrap, which compiled successfully. Something in the
>> tree-sitter branch requires a git clean to build after merging master,
>> it seems.
> 
> Thank you, compilation is fixed.
> 
> I confirm that you also fixed the fontification problems I saw!
> 
> Would the integration of tree-sitter help to implement an option to have
> all types fontified?  Now that it's becoming more popular to uses type
> hints in Python code, it may improve code readability to have all types
> fontified.
> -- 
> Matthias

Yeah, with tree-sitter, fortifying types is trivial. In fact all types should be fortified already. (I tested with some simple examples.) Should we provide some variables to toggle fontification for different things? Like python-fontify-type/f-string/assignment/built-in/etc. 

Yuan

Re: Tree-sitter integration in python.el

[Augusto Stoffel]

On Fri,  7 Oct 2022 at 01:25, Yuan Fu wrote:

> Yeah, with tree-sitter, fortifying types is trivial. In fact all types
> should be fortified already. (I tested with some simple examples.)
> Should we provide some variables to toggle fontification for different
> things? Like python-fontify-type/f-string/assignment/built-in/etc.

Looking at the screenshots posted a few messages back, which are VERY
busy, I would really appreciate an option to disable a few fontification
rules or, conversely, disable all but a few of them.  Ideally, this
should be done through a generic mechanism that works across major
modes.

Have you seen the new `font-lock-ignore' option?  Tree-sitter could
provide something similar (and much better/less hacky).

Re: Tree-sitter integration in python.el

[chad]

[-- Attachment #1: Type: text/plain, Size: 1116 bytes --]

On Fri, Oct 7, 2022 at 6:07 AM Augusto Stoffel <arstoffel@gmail.com> wrote:

> Looking at the screenshots posted a few messages back, which are VERY
> busy, I would really appreciate an option to disable a few fontification
> rules or, conversely, disable all but a few of them.  Ideally, this
> should be done through a generic mechanism that works across major
> modes.
>

I believe that history suggests that a better approach is "fontify
everything and provide a setting that gives most of them the same
appearance", assuming that the combination of tree-sitter and fontification
is performant enough. This also enables some interesting features that were
tried way back in the days of hilit19 like "transiently highlight
everything of *this* type, but not all types". Somewhat like how isearch
(occur, et al) can highlight literal duplicates, but based on shared
semantic properties instead.

I couldn't tell you offhand if the current tree-sitter performance for
fontification is strong enough for this sort of thing or not, and I suspect
the noverlay branch also has some impact here.

Hope that helps,
~Chad

[-- Attachment #2: Type: text/html, Size: 1552 bytes --]

Re: Tree-sitter integration in python.el

[Eli Zaretskii]

> From: chad <yandros@gmail.com>
> Date: Fri, 7 Oct 2022 13:53:57 -0400
> Cc: Yuan Fu <casouri@gmail.com>, Matthias Meulien <orontee@gmail.com>,
>  Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> 
> I couldn't tell you offhand if the current tree-sitter performance for fontification is strong enough for this sort
> of thing or not, and I suspect the noverlay branch also has some impact here.

It shouldn't be, because fontifications don't use overlays.

Re: Tree-sitter integration in python.el

[Yuan Fu]

> I believe that history suggests that a better approach is "fontify everything and provide a setting that gives most of them the same appearance", assuming that the combination of tree-sitter and fontification is performant enough.

I think toggling on/off is easier for the users to understand and configure.

> This also enables some interesting features that were tried way back in the days of hilit19 like "transiently highlight everything of *this* type, but not all types". Somewhat like how isearch (occur, et al) can highlight literal duplicates, but based on shared semantic properties instead.

You mean “transiently highlight all variable names”? What would be a use-case of that? I’m only aware of “highlight all appearance of _this_ variable”, which is provided by IDEs and LSP.

> I couldn't tell you offhand if the current tree-sitter performance for fontification is strong enough for this sort of thing or not, and I suspect the noverlay branch also has some impact here.

Tree-sitter is pretty fast and we don’t need to worry about “too-much fontification”.

Yuan

Re: Tree-sitter integration in python.el

[Yuan Fu]

> On Oct 7, 2022, at 3:03 AM, Augusto Stoffel <arstoffel@gmail.com> wrote:
> 
> On Fri,  7 Oct 2022 at 01:25, Yuan Fu wrote:
> 
>> Yeah, with tree-sitter, fortifying types is trivial. In fact all types
>> should be fortified already. (I tested with some simple examples.)
>> Should we provide some variables to toggle fontification for different
>> things? Like python-fontify-type/f-string/assignment/built-in/etc.
> 
> Looking at the screenshots posted a few messages back, which are VERY
> busy, I would really appreciate an option to disable a few fontification
> rules or, conversely, disable all but a few of them.  Ideally, this
> should be done through a generic mechanism that works across major
> modes.
> 
> Have you seen the new `font-lock-ignore' option?  Tree-sitter could
> provide something similar (and much better/less hacky).

The complaint for font-lock-maximum-decoration is that it’s obscure and too corse-grained. So my idea is for each major mode to provide fined-grained controls like python-fontify-type/f-string/assignment/built-in/etc. And tree-sitter makes it easy to implement this kind of toggle. But I guess a global control is also nice, I can make tree-sitter respect font-lock-maximum-decoration, in addition to the fined grained local-control.

Since we are designing a new system, I don’t think we need to resort to the likes of font-lock-ignore.

Yuan

Re: Tree-sitter integration in python.el

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Fri, 7 Oct 2022 15:10:10 -0700
> Cc: Matthias Meulien <orontee@gmail.com>,
>  Eli Zaretskii <eliz@gnu.org>,
>  emacs-devel@gnu.org
> 
> The complaint for font-lock-maximum-decoration is that it’s obscure and too corse-grained. So my idea is for each major mode to provide fined-grained controls like python-fontify-type/f-string/assignment/built-in/etc. And tree-sitter makes it easy to implement this kind of toggle. But I guess a global control is also nice, I can make tree-sitter respect font-lock-maximum-decoration, in addition to the fined grained local-control.

I think having tree-sitter respect font-lock-maximum-decoration would
be good, because it allows a major-mode agnostic way of controlling
fontifications.  With tree-sitter in mind, we'd need to agree on what
kind of syntactic entities are included in each level (which is also a
Good Thing, because currently what is level N of font-lock is entirely
up to the major-mode, AFAIU).

Re: Tree-sitter integration in python.el

[Yuan Fu]

> On Oct 7, 2022, at 11:30 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Yuan Fu <casouri@gmail.com>
>> Date: Fri, 7 Oct 2022 15:10:10 -0700
>> Cc: Matthias Meulien <orontee@gmail.com>,
>> Eli Zaretskii <eliz@gnu.org>,
>> emacs-devel@gnu.org
>> 
>> The complaint for font-lock-maximum-decoration is that it’s obscure and too corse-grained. So my idea is for each major mode to provide fined-grained controls like python-fontify-type/f-string/assignment/built-in/etc. And tree-sitter makes it easy to implement this kind of toggle. But I guess a global control is also nice, I can make tree-sitter respect font-lock-maximum-decoration, in addition to the fined grained local-control.
> 
> I think having tree-sitter respect font-lock-maximum-decoration would
> be good, because it allows a major-mode agnostic way of controlling
> fontifications.  With tree-sitter in mind, we'd need to agree on what
> kind of syntactic entities are included in each level (which is also a
> Good Thing, because currently what is level N of font-lock is entirely
> up to the major-mode, AFAIU).

I think it is difficult to define syntactic entities for each level such that it is generally enough to include all kinds of major mode out there, and specific enough to be useful. It is easy for common programming languages, but hard for others like html, css, prolog, etc.

My impression of the levels are 1 for absolute minimum, 2 for moderate, and 3 for maximum. I don’t know if any major modes uses more than 3 levels. Perhaps rough guidelines like this could be more helpful than specifying syntactic entities for each level.

Yuan

Re: Tree-sitter integration in python.el

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Sat, 8 Oct 2022 13:57:19 -0700
> Cc: arstoffel@gmail.com,
>  orontee@gmail.com,
>  emacs-devel@gnu.org
> 
> > I think having tree-sitter respect font-lock-maximum-decoration would
> > be good, because it allows a major-mode agnostic way of controlling
> > fontifications.  With tree-sitter in mind, we'd need to agree on what
> > kind of syntactic entities are included in each level (which is also a
> > Good Thing, because currently what is level N of font-lock is entirely
> > up to the major-mode, AFAIU).
> 
> I think it is difficult to define syntactic entities for each level such that it is generally enough to include all kinds of major mode out there, and specific enough to be useful. It is easy for common programming languages, but hard for others like html, css, prolog, etc.

I don't think it should be hard.  At worst, we will find that HTML,
CSS, etc. have fewer syntactic entities than programming languages, so
maybe they will have fewer meaningful levels.

> My impression of the levels are 1 for absolute minimum, 2 for moderate, and 3 for maximum.

Right.

> Perhaps rough guidelines like this could be more helpful than specifying syntactic entities for each level.

We could come up with such guidelines, yes.

Re: Tree-sitter integration in python.el

[Stefan Monnier]

>>> The complaint for font-lock-maximum-decoration is that it’s obscure and
>>> too corse-grained. So my idea is for each major mode to provide
>>> fined-grained controls like
>>> python-fontify-type/f-string/assignment/built-in/etc. And tree-sitter
>>> makes it easy to implement this kind of toggle. But I guess a global
>>> control is also nice, I can make tree-sitter respect
>>> font-lock-maximum-decoration, in addition to the fined grained
>>> local-control.
>> 
>> I think having tree-sitter respect font-lock-maximum-decoration would
>> be good, because it allows a major-mode agnostic way of controlling
>> fontifications.  With tree-sitter in mind, we'd need to agree on what
>> kind of syntactic entities are included in each level (which is also a
>> Good Thing, because currently what is level N of font-lock is entirely
>> up to the major-mode, AFAIU).
>
> I think it is difficult to define syntactic entities for each level such
> that it is generally enough to include all kinds of major mode out there,
> and specific enough to be useful. It is easy for common programming
> languages, but hard for others like html, css, prolog, etc.

How 'bout:

- Major modes provide a list of available fontification "features"
  in a buffer-local variable like `treesit-font-lock-features`
  e.g. `(type f-string assignment built-in ...)`.
- the list of features is *ordered* so that
  the user can control the fontification via a Customizable
  `treesit-font-lock-level` that can be set between 0 and 100 to
  select the corresponding proportion of features.
- Users can additionally enable/disable those features individually via
  some `treesit-font-lock-control` variable.


        Stefan

Re: Tree-sitter integration in python.el

[Yuan Fu]

> On Oct 11, 2022, at 3:15 PM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> 
>>>> The complaint for font-lock-maximum-decoration is that it’s obscure and
>>>> too corse-grained. So my idea is for each major mode to provide
>>>> fined-grained controls like
>>>> python-fontify-type/f-string/assignment/built-in/etc. And tree-sitter
>>>> makes it easy to implement this kind of toggle. But I guess a global
>>>> control is also nice, I can make tree-sitter respect
>>>> font-lock-maximum-decoration, in addition to the fined grained
>>>> local-control.
>>> 
>>> I think having tree-sitter respect font-lock-maximum-decoration would
>>> be good, because it allows a major-mode agnostic way of controlling
>>> fontifications.  With tree-sitter in mind, we'd need to agree on what
>>> kind of syntactic entities are included in each level (which is also a
>>> Good Thing, because currently what is level N of font-lock is entirely
>>> up to the major-mode, AFAIU).
>> 
>> I think it is difficult to define syntactic entities for each level such
>> that it is generally enough to include all kinds of major mode out there,
>> and specific enough to be useful. It is easy for common programming
>> languages, but hard for others like html, css, prolog, etc.
> 
> How 'bout:
> 
> - Major modes provide a list of available fontification "features"
>  in a buffer-local variable like `treesit-font-lock-features`
>  e.g. `(type f-string assignment built-in ...)`.
> - the list of features is *ordered* so that
>  the user can control the fontification via a Customizable
>  `treesit-font-lock-level` that can be set between 0 and 100 to
>  select the corresponding proportion of features.
> - Users can additionally enable/disable those features individually via
>  some `treesit-font-lock-control` variable.

One problem I can see is that the same level could give very different busyness across modes. That would defeat the purpose of having a single setting. Say mode 1 has (A B C D E F G H I), where ABC are very basic, DEF moderate, and GHI fancy, mode 2 has (A B C D E F G), A basic, BCDEF moderate, G fancy. Then 80 would give a fancy font-lock in mode 1 and a moderate font-lock in mode 2. You get the idea.

Currently tree-sitter supports both :level and :toggle. When defining queries you can say :toggle python-fontify-types, and python-fontify-types can control this query. :level would be the global rough settings, :toggle would be the local, fine-grained setting. Instead of variables, :toggle could be changed to use symbols as you suggested, so it doesn’t create a gazillion variables. But WDYT of the general design?

Yuan

Re: Tree-sitter integration in python.el

[Stefan Monnier]

> One problem I can see is that the same level could give very different
> busyness across modes. That would defeat the purpose of having a single
> setting. Say mode 1 has (A B C D E F G H I), where ABC are very basic, DEF
> moderate, and GHI fancy, mode 2 has (A B C D E F G), A basic, BCDEF
> moderate, G fancy. Then 80 would give a fancy font-lock in mode 1 and
> a moderate font-lock in mode 2. You get the idea.

Yup.  We could be more explicit, then, e.g. use things like ((A B C) (D
E F) (G H I)) for the first and ((A) (B C D E F) (G)) for the second.

> Currently tree-sitter supports both :level and :toggle. When defining
> queries you can say :toggle python-fontify-types, and python-fontify-types
> can control this query. :level would be the global rough settings, :toggle
> would be the local, fine-grained setting. Instead of variables, :toggle
> could be changed to use symbols as you suggested, so it doesn’t create
> a gazillion variables. But WDYT of the general design?

I definitely prefer symbols to variables: this way they can use their
own namespace and don't need the long package prefix and can be
shared/reused between different modes without conflict.

And I would prefer not to include :level in the rules themselves and
instead rely on a separate mapping between features and levels.


        Stefan

Re: Tree-sitter integration in python.el

[Yuan Fu]

> On Oct 12, 2022, at 10:52 AM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> 
>> One problem I can see is that the same level could give very different
>> busyness across modes. That would defeat the purpose of having a single
>> setting. Say mode 1 has (A B C D E F G H I), where ABC are very basic, DEF
>> moderate, and GHI fancy, mode 2 has (A B C D E F G), A basic, BCDEF
>> moderate, G fancy. Then 80 would give a fancy font-lock in mode 1 and
>> a moderate font-lock in mode 2. You get the idea.
> 
> Yup.  We could be more explicit, then, e.g. use things like ((A B C) (D
> E F) (G H I)) for the first and ((A) (B C D E F) (G)) for the second.
> 
>> Currently tree-sitter supports both :level and :toggle. When defining
>> queries you can say :toggle python-fontify-types, and python-fontify-types
>> can control this query. :level would be the global rough settings, :toggle
>> would be the local, fine-grained setting. Instead of variables, :toggle
>> could be changed to use symbols as you suggested, so it doesn’t create
>> a gazillion variables. But WDYT of the general design?
> 
> I definitely prefer symbols to variables: this way they can use their
> own namespace and don't need the long package prefix and can be
> shared/reused between different modes without conflict.
> 
> And I would prefer not to include :level in the rules themselves and
> instead rely on a separate mapping between features and levels.

Sounds good! I can do that.

Yuan

Re: Tree-sitter integration in python.el

[Yuan Fu]

>> 
>> I definitely prefer symbols to variables: this way they can use their
>> own namespace and don't need the long package prefix and can be
>> shared/reused between different modes without conflict.
>> 
>> And I would prefer not to include :level in the rules themselves and
>> instead rely on a separate mapping between features and levels.
> 
> Sounds good! I can do that.
> 
> Yuan

What are some suggestions for kind names? We could provide the common ones in the doctoring so major modes use the same names for them. I can think of: function, type, keyword, constant, builtin, string-interpolation, escape, assignment-lhs, macro, preprocessor, string, comment, hmmm… this list is extremely similar to font-lock-faces…

Yuan

Re: [SPAM UNSURE] Re: Tree-sitter integration in python.el

[Stephen Leake]

Yuan Fu <casouri@gmail.com> writes:

> What are some suggestions for kind names? We could provide the common
> ones in the doctoring so major modes use the same names for them. I
> can think of: function, type, keyword, constant, builtin,
> string-interpolation, escape, assignment-lhs, macro, preprocessor,
> string, comment, hmmm… this list is extremely similar to
> font-lock-faces…

It's also similar to LSP SemanticTokens:

https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#textDocument_semanticTokens


-- 
-- Stephe

Re: Tree-sitter integration in python.el

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Wed, 12 Oct 2022 16:43:30 -0700
> Cc: Eli Zaretskii <eliz@gnu.org>,
>  Augusto Stoffel <arstoffel@gmail.com>,
>  Matthias Meulien <orontee@gmail.com>,
>  emacs-devel@gnu.org
> 
> What are some suggestions for kind names? We could provide the common ones in the doctoring so major modes use the same names for them. I can think of: function, type, keyword, constant, builtin, string-interpolation, escape, assignment-lhs, macro, preprocessor, string, comment, hmmm… this list is extremely similar to font-lock-faces…

Exactly.  So how about just taking the list from the font-lock-faces?
We can always add some more later, if we find it necessary.

Re: Tree-sitter integration in python.el

[Yuan Fu]

> On Oct 12, 2022, at 10:55 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Yuan Fu <casouri@gmail.com>
>> Date: Wed, 12 Oct 2022 16:43:30 -0700
>> Cc: Eli Zaretskii <eliz@gnu.org>,
>> Augusto Stoffel <arstoffel@gmail.com>,
>> Matthias Meulien <orontee@gmail.com>,
>> emacs-devel@gnu.org
>> 
>> What are some suggestions for kind names? We could provide the common ones in the doctoring so major modes use the same names for them. I can think of: function, type, keyword, constant, builtin, string-interpolation, escape, assignment-lhs, macro, preprocessor, string, comment, hmmm… this list is extremely similar to font-lock-faces…
> 
> Exactly.  So how about just taking the list from the font-lock-faces?
> We can always add some more later, if we find it necessary.

I have implemented this, but renamed :kind to :feature. See treesit-font-lock-rules and treesit-font-lock-feature-list.

Yuan

Re: Tree-sitter integration in python.el

[Augusto Stoffel]

On Fri,  7 Oct 2022 at 15:10, Yuan Fu wrote:

>> On Oct 7, 2022, at 3:03 AM, Augusto Stoffel <arstoffel@gmail.com> wrote:
>> 
>> On Fri,  7 Oct 2022 at 01:25, Yuan Fu wrote:
>> 
>>> Yeah, with tree-sitter, fortifying types is trivial. In fact all types
>>> should be fortified already. (I tested with some simple examples.)
>>> Should we provide some variables to toggle fontification for different
>>> things? Like python-fontify-type/f-string/assignment/built-in/etc.
>> 
>> Looking at the screenshots posted a few messages back, which are VERY
>> busy, I would really appreciate an option to disable a few fontification
>> rules or, conversely, disable all but a few of them.  Ideally, this
>> should be done through a generic mechanism that works across major
>> modes.
>> 
>> Have you seen the new `font-lock-ignore' option?  Tree-sitter could
>> provide something similar (and much better/less hacky).
>
> The complaint for font-lock-maximum-decoration is that it’s obscure
> and too corse-grained.

To me, the biggest problem with font-lock-maximum-decoration is that few
major modes bothered to implement levels.

> So my idea is for each major mode to provide fined-grained controls
> like python-fontify-type/f-string/assignment/built-in/etc.  And
> tree-sitter makes it easy to implement this kind of toggle.

Given the lack of success of font-lock-maximum-decoration, I don't see
this being implemented by many major modes.  Also, if the idea does take
traction, it will lead to a proliferation of user options that is hard
to use effectively -- if someone doesn't want to fontify built-ins in
Python, they probably don't want it in other languages either, so they
need to set a similar option for N languages.

> But I guess a global control is also nice, I can make tree-sitter
> respect font-lock-maximum-decoration, in addition to the fined grained
> local-control.
>
> Since we are designing a new system, I don’t think we need to resort
> to the likes of font-lock-ignore.

It's exactly the opposite: since you are designing a new systems, you
can create a much nicer customization mechanism on the lines of
font-lock-ignore.  For instance, one could select fontification rules
based on the affected node type.

The “decoration levels” feature can then build up on this, with the
advantage that it would be consistent across languages and require no
extra effort from the major mode developer.

RE: [External] : Re: Tree-sitter integration in python.el

[Drew Adams]

> To me, the biggest problem with font-lock-maximum-decoration 
> is that few major modes bothered to implement levels.

That's a problem with the non-few modes that
don't do that.

Some Elisp code has less than stellar doc
strings, and even a lack of doc strings in
some cases.

And much Elisp code uses only rudimentary
defcustom :type specs (e.g. just `sexp'),
instead of specifying just what type of
Elisp values are appropriate, so excluding
inappropriate types.

These are only reasons to encourage _more_
attention to helping users with better doc
etc.  They're by no means reasons not to
bother with font-lock levels, doc, or :type.

(I'm not saying you actually suggested
giving up on font-lock levels, BTW.)

If Emacs's own code hardly bothers with
font-lock levels then it shouldn't be much
of a surprise that 3rd-party code doesn't
bother with them either.  One might even
bet that many who write 3rd-party code are
completely _unaware_ of font-lock levels.

> Given the lack of success of font-lock-maximum-decoration,

See above.  Lack of awareness of it.  Lack
of encouragement to make use of it.

> I don't see this being implemented by many major modes.

Ignorance of Emacs-Lisp coding conventions
is combatted by maintainers reminding about
them, encouraging their respect, and even
requiring their respect for contributions
to Emacs.  Awareness is probably the first
hurdle to putting font-lock levels to use.

One simple analogy - any number of others
could be given:

Just because electoral participation is
limited in some countries (e.g. the USA),
that's not a reason to say that "given the
lack of success" of voluntary electoral
participation we might as well not bother
having elections.

> Also, if the idea does take traction, it will lead
> to a proliferation of user options that is hard
> to use effectively 

Every user option (and face) has a _default_
value, which should be what the designers
think is a good value for most users.

There's nothing wrong with a library
providing options and faces for easy
customization.

Providing a default value has the same effect
as hard-coding the behavior - you get what
you get, OOTB - EXCEPT that you _can_ easily
get other behaviors.  The mere existence of
an option cannot possibly be inferior to
hard-coding the behavior it lets you modify.

That users have many options to easily
change behavior here and there isn't a bad
thing - a limitation.  It's a good thing.

(The only downside to a plethora of options
is that their names show up with completion,
apropos, etc.  And if users start to pay
attention to them, by reading their doc, that
attention takes time.)

> -- if someone doesn't want to fontify built-ins in
> Python, they probably don't want it in other
> languages either, so they need to set a similar
> option for N languages.

And if they don't have those options,
then what?  How do they then get behavior
they want across those N languages?

Nothing prevents a library (or Emacs)
from providing ways to affect multiple
such options together, across your N
languages.

That's not hard to do.  And it lets users
decide for _which N_ of the N+M existing
languages with font-lock levels they really
want to override the default value.

> > Since we are designing a new system, I don’t
> > think we need to resort to the likes of font-lock-ignore.
> 
> It's exactly the opposite: since you are designing
> a new systems, you can create a much nicer
> customization mechanism on the lines of
> font-lock-ignore.  For instance, one could select 
> fontification rules based on the affected node type.
> 
> The “decoration levels” feature can then build up on this, with the
> advantage that it would be consistent across languages and require no
> extra effort from the major mode developer.

See https://lists.gnu.org/archive/html/emacs-devel/2022-04/msg00066.html

I'm not against giving users and library
authors multiple-grain, selective ways to
override/prevent font-locking.  I proposed
this long ago.

I still haven't found the thread where the
current `font-lock-ignore' is introduced, so
it's hard to comment on it.  I asked about
it, but was never pointed to it.

But is it really related to whether it can
be useful to provide more than one font-lock
level?

Both fine-grained and coarse-grained control
over an existing set of font-lock rules and
faces are useful, by both "end" users and
library authors.

Font-lock levels can provide one kind of such
(coarse-grained) control.

Why should the `font-lock-ignore' feature
being discussed (for which I haven't found
any spec or code, the only thread I found for
it starting with Eli's comments [1] on some
previous thread (somewhere?) about "master
5c70ff9") obviate the usefulness of font-lock
levels?  Why does the one preclude use of the
other?
___

[1] https://lists.gnu.org/archive/html/emacs-devel/2022-04/msg00041.html

Re: [External] : Re: Tree-sitter integration in python.el

[Augusto Stoffel]

On Sat,  8 Oct 2022 at 16:20, Drew Adams wrote:

> Why should the `font-lock-ignore' feature
> being discussed (for which I haven't found
> any spec or code, the only thread I found for
> it starting with Eli's comments [1] on some
> previous thread (somewhere?) about "master
> 5c70ff9") obviate the usefulness of font-lock
> levels?  Why does the one preclude use of the
> other?

The font-lock-ignore variable is available in the master branch.  It has
a docstring and a manual entry.  I think nobody said it precludes the
need for a more coarse-grained customization option; I only said in
passing that in alternative universe it could have been used as the
basis to implement font-lock-maximum-decoration.

Re: Tree-sitter integration in python.el

[Yuan Fu]

> On Oct 8, 2022, at 1:03 AM, Augusto Stoffel <arstoffel@gmail.com> wrote:
> 
> On Fri,  7 Oct 2022 at 15:10, Yuan Fu wrote:
> 
>>> On Oct 7, 2022, at 3:03 AM, Augusto Stoffel <arstoffel@gmail.com> wrote:
>>> 
>>> On Fri,  7 Oct 2022 at 01:25, Yuan Fu wrote:
>>> 
>>>> Yeah, with tree-sitter, fortifying types is trivial. In fact all types
>>>> should be fortified already. (I tested with some simple examples.)
>>>> Should we provide some variables to toggle fontification for different
>>>> things? Like python-fontify-type/f-string/assignment/built-in/etc.
>>> 
>>> Looking at the screenshots posted a few messages back, which are VERY
>>> busy, I would really appreciate an option to disable a few fontification
>>> rules or, conversely, disable all but a few of them.  Ideally, this
>>> should be done through a generic mechanism that works across major
>>> modes.
>>> 
>>> Have you seen the new `font-lock-ignore' option?  Tree-sitter could
>>> provide something similar (and much better/less hacky).
>> 
>> The complaint for font-lock-maximum-decoration is that it’s obscure
>> and too corse-grained.
> 
> To me, the biggest problem with font-lock-maximum-decoration is that few
> major modes bothered to implement levels.

That’s their choice. If no one complains it’s not a problem ;-)

> 
>> So my idea is for each major mode to provide fined-grained controls
>> like python-fontify-type/f-string/assignment/built-in/etc.  And
>> tree-sitter makes it easy to implement this kind of toggle.
> 
> Given the lack of success of font-lock-maximum-decoration, I don't see
> this being implemented by many major modes.  Also, if the idea does take
> traction, it will lead to a proliferation of user options that is hard
> to use effectively -- if someone doesn't want to fontify built-ins in
> Python, they probably don't want it in other languages either, so they
> need to set a similar option for N languages.

It sounds nice, but (1) such generalization breaks down for not-c-like-general-programming-languages, like html, css, texinfo, etc, and (2) maybe one wants builtins in C but not in Python.

> 
>> But I guess a global control is also nice, I can make tree-sitter
>> respect font-lock-maximum-decoration, in addition to the fined grained
>> local-control.
>> 
>> Since we are designing a new system, I don’t think we need to resort
>> to the likes of font-lock-ignore.
> 
> It's exactly the opposite: since you are designing a new systems, you
> can create a much nicer customization mechanism on the lines of
> font-lock-ignore.  For instance, one could select fontification rules
> based on the affected node type.
> 
> The “decoration levels” feature can then build up on this, with the
> advantage that it would be consistent across languages and require no
> extra effort from the major mode developer.

It’s a nice idea, but in tree-sitter, different languages tag differently, it could be “function_definition” in python but “function_declaration” in C, etc. So it is not consistent across language. Also, it is often more complicated than a single tag like “function_identifier”, but rather a nested structure like (function_definition (identifier)).

Yuan

Re: Tree-sitter integration in python.el

[Augusto Stoffel]

On Sat,  8 Oct 2022 at 14:06, Yuan Fu wrote:

>> To me, the biggest problem with font-lock-maximum-decoration is that few
>> major modes bothered to implement levels.
>
> That’s their choice. If no one complains it’s not a problem ;-)

Well, I'm complaining now and I think you can solve this :-).

>> It's exactly the opposite: since you are designing a new systems, you
>> can create a much nicer customization mechanism on the lines of
>> font-lock-ignore.  For instance, one could select fontification rules
>> based on the affected node type.
>> 
>> The “decoration levels” feature can then build up on this, with the
>> advantage that it would be consistent across languages and require no
>> extra effort from the major mode developer.
>
> It’s a nice idea, but in tree-sitter, different languages tag
> differently, it could be “function_definition” in python but
> “function_declaration” in C, etc. So it is not consistent across
> language. Also, it is often more complicated than a single tag like
> “function_identifier”, but rather a nested structure like
> (function_definition (identifier)).

One suggestion then is to allow attaching some “metadata” to the
fontification rules in `treesit-font-lock-rules'.  For instance,
declare that a rule has :kind function or :level 2.

Re: Tree-sitter integration in python.el

[Yuan Fu]

> On Oct 10, 2022, at 12:16 AM, Augusto Stoffel <arstoffel@gmail.com> wrote:
> 
> On Sat,  8 Oct 2022 at 14:06, Yuan Fu wrote:
> 
>>> To me, the biggest problem with font-lock-maximum-decoration is that few
>>> major modes bothered to implement levels.
>> 
>> That’s their choice. If no one complains it’s not a problem ;-)
> 
> Well, I'm complaining now and I think you can solve this :-).

Of course, within my reach I try to stratify font-lock into levels. I meant major modes not in core. 

> 
>>> It's exactly the opposite: since you are designing a new systems, you
>>> can create a much nicer customization mechanism on the lines of
>>> font-lock-ignore.  For instance, one could select fontification rules
>>> based on the affected node type.
>>> 
>>> The “decoration levels” feature can then build up on this, with the
>>> advantage that it would be consistent across languages and require no
>>> extra effort from the major mode developer.
>> 
>> It’s a nice idea, but in tree-sitter, different languages tag
>> differently, it could be “function_definition” in python but
>> “function_declaration” in C, etc. So it is not consistent across
>> language. Also, it is often more complicated than a single tag like
>> “function_identifier”, but rather a nested structure like
>> (function_definition (identifier)).
> 
> One suggestion then is to allow attaching some “metadata” to the
> fontification rules in `treesit-font-lock-rules'.  For instance,
> declare that a rule has :kind function or :level 2.

I’ve added support for :level in treesit-font-lock-rules, yes ;-)

Yuan

Re: Tree-sitter integration in python.el

[Augusto Stoffel]

On Mon, 10 Oct 2022 at 08:10, Yuan Fu wrote:

> I’ve added support for :level in treesit-font-lock-rules, yes ;-)

Nice, thanks.

Now, concerning font-lock-ignore: why do you say you think something
like it is not needed in tree-sitter?  Do you see no need for a
fine-grained customization option for font lock, or do you have a better
implementation of fine-grained customization in mind?

I'd say a fine-grained customization option is very much needed.
Requiring the user to advice `treesit-font-lock-rules' is to high of a
bar.  `font-lock-ignore' may not be incredibly easy to use (the user
still has to look at the font-lock rule definitions), but it's flexible,
has a simple implementation, and imposes no extra work on the major-mode
developer.

You can probably just stick a call to font-lock--filter-keywords
somewhere in treesit.el and call it a day.

Re: Tree-sitter integration in python.el

[Yuan Fu]

> 
> Now, concerning font-lock-ignore: why do you say you think something
> like it is not needed in tree-sitter?  Do you see no need for a
> fine-grained customization option for font lock, or do you have a better
> implementation of fine-grained customization in mind?
> 
> I'd say a fine-grained customization option is very much needed.
> Requiring the user to advice `treesit-font-lock-rules' is to high of a
> bar.  `font-lock-ignore' may not be incredibly easy to use (the user
> still has to look at the font-lock rule definitions), but it's flexible,
> has a simple implementation, and imposes no extra work on the major-mode
> developer.
> 
> You can probably just stick a call to font-lock--filter-keywords
> somewhere in treesit.el and call it a day.

Currently we have :toggle variable, which let variable control whether to enable a query. Major modes could sever font-lock into categories, each controlled by a variable. Eg, python-font-lock-type, python-font-lock-assignment, etc. I think that should be more than enough as long as developers bother to create these toggles.

Yuan

Re: Tree-sitter integration in python.el

[Stefan Monnier]

>> One suggestion then is to allow attaching some “metadata” to the
>> fontification rules in `treesit-font-lock-rules'.  For instance,
>> declare that a rule has :kind function or :level 2.

Exactly.

> I’ve added support for :level in treesit-font-lock-rules, yes ;-)

I'd *much* prefer `:kind` over `:level` because it provides
more information.

I suggested in my other message how that `:kind` info could be (re)mapped
to a level so users can provide a kind of baseline busy-ness level.


        Stefan

Re: Tree-sitter integration in python.el

[Eli Zaretskii]

> From: Matthias Meulien <orontee@gmail.com>
> Cc: casouri@gmail.com,  emacs-devel@gnu.org
> Date: Tue, 04 Oct 2022 00:19:43 +0200
> 
> > Some of these lack the Makefile.  But the Makefile is standard and can
> > be taken from any other language module, for example I see on in
> > tree-sitter-ruby.  So just clone that as well, copy its Makefile into
> > tree-sitter-python, and run "make".
> 
> Thank you, Eli.  It wasn't as easy as you said but I managed to compile
> a shared object using a modified version of the Makefile found in the
> tree-sitter-ruby (using the file as is fails when building
> tree-sitter-{ruby,python} due to a missing -Wl, before -soname usage).

Would you please share the changes you made, FTR?  Then others who
face the same problem could easily find the solution.  TIA.

> I'll keep using this branch and collect unexpected behaviors.

Thanks!

Re: Tree-sitter integration in python.el

[Matthias Meulien]

Eli Zaretskii <eliz@gnu.org> writes:

> (...)  Would you please share the changes you made, FTR?  Then others
> who face the same problem could easily find the solution.  TIA.

Sure, but let me make clear that it's an unclean hack and using
https://github.com/casouri/tree-sitter-module may prove to be more
robust.

diff --unified tree-sitter-ruby/Makefile tree-sitter-python/Makefile
--- tree-sitter-ruby/Makefile	2022-10-03 23:01:34.963750322 +0200
+++ tree-sitter-python/Makefile	2022-10-04 13:06:46.014817832 +0200
@@ -31,7 +31,7 @@
 ifeq (, $(CPPSRC))
 	ADDITIONALLIBS := 
 else
-	ADDITIONALLIBS := -lc++
+	ADDITIONALLIBS :=
 endif
 
 # collect sources
@@ -71,14 +71,14 @@
 ifneq (,$(filter $(shell uname),FreeBSD NetBSD DragonFly))
 	PCLIBDIR := $(PREFIX)/libdata/pkgconfig
 endif
-				
-all: libtree-sitter-$(PARSER_NAME).a libtree-sitter-$(PARSER_NAME).$(SOEXTVER) bindings/c/$(PARSER_NAME).h bindings/c/tree-sitter-$(PARSER_NAME).pc
+
+all: libtree-sitter-$(PARSER_NAME).a libtree-sitter-$(PARSER_NAME).$(SOEXTVER)
 
 libtree-sitter-$(PARSER_NAME).a: $(OBJ)
 	$(AR) rcs $@ $^
 
 libtree-sitter-$(PARSER_NAME).$(SOEXTVER): $(OBJ)
-	$(CC) $(LDFLAGS) $(LINKSHARED) $^ $(LDLIBS) -o $@
+	g++ $(LDFLAGS) $(LINKSHARED) $^ $(LDLIBS) -o $@
 	ln -sf $@ libtree-sitter-$(PARSER_NAME).$(SOEXT)
 	ln -sf $@ libtree-sitter-$(PARSER_NAME).$(SOEXTVER_MAJOR)
 
@@ -102,10 +102,6 @@
 	install -m755 libtree-sitter-$(PARSER_NAME).$(SOEXTVER) '$(DESTDIR)$(LIBDIR)'/libtree-sitter-$(PARSER_NAME).$(SOEXTVER)
 	ln -sf libtree-sitter-$(PARSER_NAME).$(SOEXTVER) '$(DESTDIR)$(LIBDIR)'/libtree-sitter-$(PARSER_NAME).$(SOEXTVER_MAJOR)
 	ln -sf libtree-sitter-$(PARSER_NAME).$(SOEXTVER) '$(DESTDIR)$(LIBDIR)'/libtree-sitter-$(PARSER_NAME).$(SOEXT)
-	install -d '$(DESTDIR)$(INCLUDEDIR)'/tree_sitter
-	install -m644 bindings/c/$(PARSER_NAME).h '$(DESTDIR)$(INCLUDEDIR)'/tree_sitter/
-	install -d '$(DESTDIR)$(PCLIBDIR)'
-	install -m644 bindings/c/tree-sitter-$(PARSER_NAME).pc '$(DESTDIR)$(PCLIBDIR)'/
 
 clean:
 	rm -f $(OBJ) libtree-sitter-$(PARSER_NAME).a libtree-sitter-$(PARSER_NAME).$(SOEXT) libtree-sitter-$(PARSER_NAME).$(SOEXTVER_MAJOR) libtree-sitter-$(PARSER_NAME).$(SOEXTVER)
-- 
Matthias

Re: Tree-sitter integration in python.el

[Eli Zaretskii]

> From: Matthias Meulien <orontee@gmail.com>
> Cc: casouri@gmail.com,  emacs-devel@gnu.org
> Date: Tue, 04 Oct 2022 13:21:45 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > (...)  Would you please share the changes you made, FTR?  Then others
> > who face the same problem could easily find the solution.  TIA.
> 
> Sure, but let me make clear that it's an unclean hack and using
> https://github.com/casouri/tree-sitter-module may prove to be more
> robust.

Understood, thanks.

> diff --unified tree-sitter-ruby/Makefile tree-sitter-python/Makefile
> --- tree-sitter-ruby/Makefile	2022-10-03 23:01:34.963750322 +0200
> +++ tree-sitter-python/Makefile	2022-10-04 13:06:46.014817832 +0200
> @@ -31,7 +31,7 @@
>  ifeq (, $(CPPSRC))
>  	ADDITIONALLIBS := 
>  else
> -	ADDITIONALLIBS := -lc++
> +	ADDITIONALLIBS :=
>  endif

Any reason why you needed to use the C++ compiler?

Re: Tree-sitter integration in python.el

[Matthias Meulien]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Matthias Meulien <orontee@gmail.com>
>> Cc: casouri@gmail.com,  emacs-devel@gnu.org
>> Date: Tue, 04 Oct 2022 13:21:45 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > (...)  Would you please share the changes you made, FTR?  Then others
>> > who face the same problem could easily find the solution.  TIA.
>> 
>> Sure, but let me make clear that it's an unclean hack and using
>> https://github.com/casouri/tree-sitter-module may prove to be more
>> robust.
>
> Understood, thanks.
>
>> diff --unified tree-sitter-ruby/Makefile tree-sitter-python/Makefile
>> --- tree-sitter-ruby/Makefile	2022-10-03 23:01:34.963750322 +0200
>> +++ tree-sitter-python/Makefile	2022-10-04 13:06:46.014817832 +0200
>> @@ -31,7 +31,7 @@
>>  ifeq (, $(CPPSRC))
>>  	ADDITIONALLIBS := 
>>  else
>> -	ADDITIONALLIBS := -lc++
>> +	ADDITIONALLIBS :=
>>  endif
>
> Any reason why you needed to use the C++ compiler?

Well, I saw the C++ file tree-sitter-python/src/parser.cc
(cf. https://github.com/tree-sitter/tree-sitter-python/blob/master/src/scanner.cc#L1)
and also got a linker error when using the original Makefile where gcc
use option -lc++.  I didn't try to understand that error, I am used to
compile C++ using g++ so I just changed the Makefile to fix an error
I've not truly understood.

Note that
https://github.com/casouri/tree-sitter-module/blob/master/build.sh#L46
switch to using c++ compiler when a parser.cc file is found.
-- 
Matthias

Re: Tree-sitter integration in python.el

[Yuan Fu]

> On Oct 3, 2022, at 11:07 AM, Matthias Meulien <orontee@gmail.com> wrote:
> 
> 
> I also cloned tree-sitter-python but I've no idea of what should be done
> with the grammar file.  I've understood that I am suppposed to convert
> the grammar to a shared object but I don't know how...
> 
> (I've tree-sitter-cli installed and in path, but "tree-sitter generate
> tree-sitter-python/grammar.js" seems to generate rust and node bindings;
> Am I supposed to compile the src/parser.c file?  I don't see any
> Makefile...).
> -- 
> Matthias

Eli has pointed out how to compile the language definition. Alternatively, you can use my script here to download and compile them:

https://github.com/casouri/tree-sitter-module

Yuan

Re: Tree-sitter integration in python.el

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Mon, 3 Oct 2022 15:25:29 -0700
> Cc: emacs-devel <emacs-devel@gnu.org>
> 
> > On Oct 3, 2022, at 11:07 AM, Matthias Meulien <orontee@gmail.com> wrote:
> > 
> Eli has pointed out how to compile the language definition. Alternatively, you can use my script here to download and compile them:
> 
> https://github.com/casouri/tree-sitter-module

That script seems to build an Emacs module?  Is that really required?
AFAIU, the tree-sitter branch loads the language modules directly, not
via the emacs-module interface.  So why would we need to produce an
Emacs module from each language definition library?

Re: Tree-sitter integration in python.el

[Yuan Fu]

> On Oct 16, 2022, at 12:31 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Yuan Fu <casouri@gmail.com>
>> Date: Mon, 3 Oct 2022 15:25:29 -0700
>> Cc: emacs-devel <emacs-devel@gnu.org>
>> 
>>> On Oct 3, 2022, at 11:07 AM, Matthias Meulien <orontee@gmail.com> wrote:
>>> 
>> Eli has pointed out how to compile the language definition. Alternatively, you can use my script here to download and compile them:
>> 
>> https://github.com/casouri/tree-sitter-module
> 
> That script seems to build an Emacs module?  Is that really required?
> AFAIU, the tree-sitter branch loads the language modules directly, not
> via the emacs-module interface.  So why would we need to produce an
> Emacs module from each language definition library?

I forgot to remove old files from that repo. The script should build an ordinary dynamic library.

Yuan

Re: Tree-sitter integration in python.el

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Sun, 16 Oct 2022 01:15:46 -0700
> Cc: orontee@gmail.com,
>  emacs-devel@gnu.org
> 
> >> https://github.com/casouri/tree-sitter-module
> > 
> > That script seems to build an Emacs module?  Is that really required?
> > AFAIU, the tree-sitter branch loads the language modules directly, not
> > via the emacs-module interface.  So why would we need to produce an
> > Emacs module from each language definition library?
> 
> I forgot to remove old files from that repo. The script should build an ordinary dynamic library.

OK, thanks.

Re: Tree-sitter integration in python.el

[Matthias Meulien]

[-- Attachment #1: Type: text/plain, Size: 818 bytes --]

Yuan Fu <casouri@gmail.com> writes:

> (...)  I’ve added tree-sitter version for font-lock and which-func in
> python.el. And I’d love to hear some feedback from python.el
> maintainers. Specifically, does it look right and which other part of
> python.el could actually benefit from a parse tree? I wrote a
> tree-sitter imenu indexer for python and it performed worse than the
> current one, presumably because it traverses the whole parse tree
> whereas the current one only scans the buffer once or so and do some
> regex matching.
>
> Here is the commit: https://git.savannah.gnu.org/cgit/emacs.git/commit/?h=feature/tree-sitter&id=1cdb24fe35a9ff2e4f92c5acc93a5a5b0e70d93f
>
> Yuan

Boolean and comparison operators like in, is, and, not aren't fontified
anymore.  See attached screenshot.


[-- Attachment #2: Capture d’écran du 2022-10-04 00-27-24.png --]
[-- Type: image/png, Size: 166432 bytes --]

[-- Attachment #3: Type: text/plain, Size: 15 bytes --]



-- 
Matthias

Re: Tree-sitter integration in python.el

[lkg.ch@pm.me]

[-- Attachment #1: Type: text/plain, Size: 120 bytes --]

Usually

gcc -shared -fPIC parse.c scanner.c -ltree-sitter -o libtree-sitter-$lang.so

can generate the so file you need

[-- Attachment #2: Type: text/html, Size: 781 bytes --]