Re: Turing on tree-sitter

Re: Turing on tree-sitter

[Drew Adams]

(Looking to see what Turing has to say
about tree-sitter...)

Re: Turing on tree-sitter

[Zhang Haijun]

[-- Attachment #1: Type: text/plain, Size: 130 bytes --]


Maybe there will be another one like the tree-sitter in the future? Then there will be 3 or more choices for every major-mode?

[-- Attachment #2: Type: text/html, Size: 1775 bytes --]

Re: Turing on tree-sitter

[Daniel Martín]

Zhang Haijun <ccsmile2008@outlook.com> writes:

> Maybe there will be another one like the tree-sitter in the future? Then there will be 3 or more choices for every major-mode?

Yes, I think a centralized way to enable it would reduce the
proliferation of variables (not only per major mode, I expect other
specialized modes to want to use Tree-sitter as well).

Call for volunteers: add tree-sitter support to major modes

[Eli Zaretskii]

The Emacs 29 release branch is planned to be cut in about 1.5 months,
and we want tree-sitter support to be in Emacs 29.  The Emacs Git
repository has a feature branch that integrates tree-sitter with
Emacs.  That branch is being prepared for landing on master.  However,
currently there's only one major-mode which can use tree-sitter
support on the branch: Python mode.  We need to add tree-sitter
integration to more major modes, so that Emacs 29 could have
reasonably useful tree-sitter support options in at least the popular
major modes.

This is a call for volunteers to work on adding tree-sitter support to
major modes beyond Python, and on improving the existing support in
Python mode.

Some of the popular major modes for which we would like to see
tree-sitter support in Emacs 29 are:

 c-mode
 c++-mode
 emacs-lisp-mode
 js-mode
 js-json-mode
 ruby-mode
 shell-script-mode

We probably won't have all of them by Emacs 29, but we hope to have at
least some.

So if you are interested in enhancing your favorite major-mode with
tree-sitter support, and by doing that making Emacs 29 much better,
please checkout the feature/tree-sitter branch, and start working on
some mode(s).  Tree-sitter support for a major-mode should include
enhancement of one or more of the following Emacs features:

 - font-lock
 - indentation
 - navigation (beginning-of-defun-function etc.)
 - imenu
 - which-func
 - thing-at-point

We think that font-lock is the absolute minimum, followed by
indentation.

To help with this effort, Yuan Fu, who did most of the development on
this feature branch, wrote a document with guidance notes which aim to
help you implement tree-sitter support for the above features.  You
will find that document and some other helpful documentation in the
admin/notes/tree-sitter directory and its subdirectories on the
branch.  That directory also includes scripts for easy compilation of
tree-sitter language modules (which you will need to download from the
tree-sitter site).

We hope that the existing support for Python can serve as working
example of how to add tree-sitter support for other major modes, and
together with the guidance document it will allow you to add support
for more modes with minimum effort.  Don't hesitate to ask questions
if something is unclear or you need more advice.

We hope volunteers will step forward in time to make tree-sitter
integration in Emacs 29 functional and useful.  To avoid waste of
effort, please announce here on which major mode(s) you decided to
work.

Thanks in advance.

Re: Call for volunteers: add tree-sitter support to major modes

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

> Some of the popular major modes for which we would like to see
> tree-sitter support in Emacs 29 are:
>

[...]

>  emacs-lisp-mode

How much of a priority is using tree sitter for Emacs Lisp?  It appears
that this[0] is the grammar implementation, and it currently doesn't
support the following:

    Autoload cookies
    Definitions (e.g. defun, defvar, defmacro)

As far as I understand, this means that from your list of enhanced
features, the following will either be restricted or not work at all:

 - font-lock
 - indentation
 - imenu
 - which-func

Also, when taking a look at the current implementation for Python[1][2],
I cannot but wonder if a few high-level functions/macros should be added
first to make it easier.

One more thing, it looks like the usage of tree-sitter is enabled for
python by toggling `python-use-tree-sitter'.  Will every mode now have
this kind of an option?  I would hope not...

[0] https://github.com/Wilfred/tree-sitter-elisp
[1] https://git.savannah.gnu.org/cgit/emacs.git/tree/lisp/progmodes/python.el?h=feature/tree-sitter#n953
[2] https://git.savannah.gnu.org/cgit/emacs.git/tree/lisp/progmodes/python.el?h=feature/tree-sitter#n5324

Re: Call for volunteers: add tree-sitter support to major modes

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: emacs-devel@gnu.org, Yuan Fu <casouri@gmail.com>
> Date: Sun, 09 Oct 2022 12:26:09 +0000
> 
> >  emacs-lisp-mode
> 
> How much of a priority is using tree sitter for Emacs Lisp?  It appears
> that this[0] is the grammar implementation, and it currently doesn't
> support the following:
> 
>     Autoload cookies
>     Definitions (e.g. defun, defvar, defmacro)
> 
> As far as I understand, this means that from your list of enhanced
> features, the following will either be restricted or not work at all:
> 
>  - font-lock
>  - indentation
>  - imenu
>  - which-func

If emacs-lisp-mode cannot be reasonably supported, we can omit it for
now.

> Also, when taking a look at the current implementation for Python[1][2],
> I cannot but wonder if a few high-level functions/macros should be added
> first to make it easier.
> 
> One more thing, it looks like the usage of tree-sitter is enabled for
> python by toggling `python-use-tree-sitter'.  Will every mode now have
> this kind of an option?  I would hope not...
> 
> [0] https://github.com/Wilfred/tree-sitter-elisp
> [1] https://git.savannah.gnu.org/cgit/emacs.git/tree/lisp/progmodes/python.el?h=feature/tree-sitter#n953
> [2] https://git.savannah.gnu.org/cgit/emacs.git/tree/lisp/progmodes/python.el?h=feature/tree-sitter#n5324

That's a separate discussion, so please start a new thread.

Re: Turing on tree-sitter (was: Call for volunteers: add tree-sitter support to major modes)

[Eli Zaretskii]

> Date: Sun, 09 Oct 2022 16:25:17 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org, casouri@gmail.com
> 
> > From: Philip Kaludercic <philipk@posteo.net>
> > Cc: emacs-devel@gnu.org, Yuan Fu <casouri@gmail.com>
> > Date: Sun, 09 Oct 2022 12:26:09 +0000
> > 
> > One more thing, it looks like the usage of tree-sitter is enabled for
> > python by toggling `python-use-tree-sitter'.  Will every mode now have
> > this kind of an option?  I would hope not...
> 
> That's a separate discussion, so please start a new thread.

So here I am, starting that separate discussion.

Since tree-sitter support is per-language, I do think it makes sense
for each mode to have its tree-sitter support turned on separately.
But we could also have a global "turn on tree-sitter support in every
mode that has it" option, which will turn this on for those modes
which can be supported by tree-sitter, if the appropriate
language-support library is installed.

Comments?

Re: Turing on tree-sitter (was: Call for volunteers: add tree-sitter support to major modes)

[Theodor Thornhill]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Sun, 09 Oct 2022 16:25:17 +0300
>> From: Eli Zaretskii <eliz@gnu.org>
>> Cc: emacs-devel@gnu.org, casouri@gmail.com
>> 
>> > From: Philip Kaludercic <philipk@posteo.net>
>> > Cc: emacs-devel@gnu.org, Yuan Fu <casouri@gmail.com>
>> > Date: Sun, 09 Oct 2022 12:26:09 +0000
>> > 
>> > One more thing, it looks like the usage of tree-sitter is enabled for
>> > python by toggling `python-use-tree-sitter'.  Will every mode now have
>> > this kind of an option?  I would hope not...
>> 
>> That's a separate discussion, so please start a new thread.
>
> So here I am, starting that separate discussion.
>
> Since tree-sitter support is per-language, I do think it makes sense
> for each mode to have its tree-sitter support turned on separately.
> But we could also have a global "turn on tree-sitter support in every
> mode that has it" option, which will turn this on for those modes
> which can be supported by tree-sitter, if the appropriate
> language-support library is installed.
>
> Comments?

+1 for this, but also keep the per-modes customs.  Then we can eat the
cake and have it too.

Such a mechanism can be made later, I think?

Theo

Re: Turing on tree-sitter (was: Call for volunteers: add tree-sitter support to major modes)

[Yuan Fu]

> On Oct 9, 2022, at 11:46 PM, Theodor Thornhill <theo@thornhill.no> wrote:
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
>>> Date: Sun, 09 Oct 2022 16:25:17 +0300
>>> From: Eli Zaretskii <eliz@gnu.org>
>>> Cc: emacs-devel@gnu.org, casouri@gmail.com
>>> 
>>>> From: Philip Kaludercic <philipk@posteo.net>
>>>> Cc: emacs-devel@gnu.org, Yuan Fu <casouri@gmail.com>
>>>> Date: Sun, 09 Oct 2022 12:26:09 +0000
>>>> 
>>>> One more thing, it looks like the usage of tree-sitter is enabled for
>>>> python by toggling `python-use-tree-sitter'.  Will every mode now have
>>>> this kind of an option?  I would hope not...
>>> 
>>> That's a separate discussion, so please start a new thread.
>> 
>> So here I am, starting that separate discussion.
>> 
>> Since tree-sitter support is per-language, I do think it makes sense
>> for each mode to have its tree-sitter support turned on separately.
>> But we could also have a global "turn on tree-sitter support in every
>> mode that has it" option, which will turn this on for those modes
>> which can be supported by tree-sitter, if the appropriate
>> language-support library is installed.
>> 
>> Comments?
> 
> +1 for this, but also keep the per-modes customs.  Then we can eat the
> cake and have it too.
> 
> Such a mechanism can be made later, I think?
> 
> Theo

+1 for this, but also add a third possible value to per-mode customs that keeps tree-sitter always off for the mode. That way someone can enable tree-sitter for all but that mode. Are there precedents for such ternary variables? nil, t, and 'never?

Yuan

Re: Turing on tree-sitter

[Philip Kaludercic]

Yuan Fu <casouri@gmail.com> writes:

>> On Oct 9, 2022, at 11:46 PM, Theodor Thornhill <theo@thornhill.no> wrote:
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>>>> Date: Sun, 09 Oct 2022 16:25:17 +0300
>>>> From: Eli Zaretskii <eliz@gnu.org>
>>>> Cc: emacs-devel@gnu.org, casouri@gmail.com
>>>> 
>>>>> From: Philip Kaludercic <philipk@posteo.net>
>>>>> Cc: emacs-devel@gnu.org, Yuan Fu <casouri@gmail.com>
>>>>> Date: Sun, 09 Oct 2022 12:26:09 +0000
>>>>> 
>>>>> One more thing, it looks like the usage of tree-sitter is enabled for
>>>>> python by toggling `python-use-tree-sitter'.  Will every mode now have
>>>>> this kind of an option?  I would hope not...
>>>> 
>>>> That's a separate discussion, so please start a new thread.
>>> 
>>> So here I am, starting that separate discussion.

Sorry, I missed the message.

>>> Since tree-sitter support is per-language, I do think it makes sense
>>> for each mode to have its tree-sitter support turned on separately.
>>> But we could also have a global "turn on tree-sitter support in every
>>> mode that has it" option, which will turn this on for those modes
>>> which can be supported by tree-sitter, if the appropriate
>>> language-support library is installed.
>>> 
>>> Comments?
>> 
>> +1 for this, but also keep the per-modes customs.  Then we can eat the
>> cake and have it too.
>> 
>> Such a mechanism can be made later, I think?
>> 
>> Theo
>
> +1 for this, but also add a third possible value to per-mode customs
> that keeps tree-sitter always off for the mode. That way someone can
> enable tree-sitter for all but that mode. Are there precedents for
> such ternary variables? nil, t, and 'never?

Is there a reason we can't use a minor mode?  Something like

        (add-hook 'python-mode-hook #'treesit-mode)

or a list

        (add-to-list 'treesit-modes 'python-mode)

?

> Yuan

Re: Turing on tree-sitter

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: Theodor Thornhill <theo@thornhill.no>,  Eli Zaretskii <eliz@gnu.org>,
>   emacs-devel@gnu.org
> Date: Mon, 10 Oct 2022 07:26:55 +0000
> 
> > +1 for this, but also add a third possible value to per-mode customs
> > that keeps tree-sitter always off for the mode. That way someone can
> > enable tree-sitter for all but that mode. Are there precedents for
> > such ternary variables? nil, t, and 'never?
> 
> Is there a reason we can't use a minor mode?  Something like
> 
>         (add-hook 'python-mode-hook #'treesit-mode)
> 
> or a list
> 
>         (add-to-list 'treesit-modes 'python-mode)
> 
> ?

We could, if a minor mode is justified.  When this was previously
brought up, someone said the justification for a minor mode was too
weak in most cases.  But maybe we should revisit that idea.

Re: Turing on tree-sitter

[Stefan Monnier]

>> Is there a reason we can't use a minor mode?  Something like
>> 
>>         (add-hook 'python-mode-hook #'treesit-mode)
>> 
>> or a list
>> 
>>         (add-to-list 'treesit-modes 'python-mode)
>> 
>> ?
>
> We could, if a minor mode is justified.  When this was previously
> brought up, someone said the justification for a minor mode was too
> weak in most cases.  But maybe we should revisit that idea.

I think a buffer-local `treesit-mode` plus a `global-treesit-mode` would
make a lot of sense, from a user's perspective.  This way they don't
have to hunt for the name of the boolean variable that their mode
decided to use to control the use of treesitter: all modes use the same
boolean variable called `treesit-mode`.

Then again, to me a minor mode is something so cheap that the idea that
"justification for a minor mode was too weak" is rather hard to grasp.


        Stefan

Re: Turing on tree-sitter

[Daniel Martín]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>
> I think a buffer-local `treesit-mode` plus a `global-treesit-mode` would
> make a lot of sense, from a user's perspective.  This way they don't
> have to hunt for the name of the boolean variable that their mode
> decided to use to control the use of treesitter: all modes use the same
> boolean variable called `treesit-mode`.

I agree.  A buffer-local and a global minor modes for Tree-sitter would
feel more natural to users.

Re: Turing on tree-sitter

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Philip Kaludercic <philipk@posteo.net>,  casouri@gmail.com,
>   theo@thornhill.no,  emacs-devel@gnu.org
> Date: Mon, 10 Oct 2022 10:52:21 -0400
> 
> I think a buffer-local `treesit-mode` plus a `global-treesit-mode` would
> make a lot of sense, from a user's perspective.  This way they don't
> have to hunt for the name of the boolean variable that their mode
> decided to use to control the use of treesitter: all modes use the same
> boolean variable called `treesit-mode`.
> 
> Then again, to me a minor mode is something so cheap that the idea that
> "justification for a minor mode was too weak" is rather hard to grasp.

That's okay, I was the one who proposed a minor mode to begin with...

Re: Turing on tree-sitter

[Yuan Fu]

> On Oct 10, 2022, at 7:52 AM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> 
>>> Is there a reason we can't use a minor mode?  Something like
>>> 
>>>        (add-hook 'python-mode-hook #'treesit-mode)
>>> 
>>> or a list
>>> 
>>>        (add-to-list 'treesit-modes 'python-mode)
>>> 
>>> ?
>> 
>> We could, if a minor mode is justified.  When this was previously
>> brought up, someone said the justification for a minor mode was too
>> weak in most cases.  But maybe we should revisit that idea.
> 
> I think a buffer-local `treesit-mode` plus a `global-treesit-mode` would
> make a lot of sense, from a user's perspective.  This way they don't
> have to hunt for the name of the boolean variable that their mode
> decided to use to control the use of treesitter: all modes use the same
> boolean variable called `treesit-mode`.

Usually users set buffer-local variables in a major mode hook, which runs after the major mode is loaded, no? But major modes need to know whether to use tree-sitter up-front.

I don’t think xxx-mode-use-tree-sitter would be hard to find if every mode uses this pattern. Though now we are discussing adding separate major modes which wouldn’t use this variable. I think that indeed could be confusing. (How do I know if xxx language uses xxx-mode-use-tree-sitter or a separate major mode xxx-tree-sitter-mode?)

> 
> Then again, to me a minor mode is something so cheap that the idea that
> "justification for a minor mode was too weak" is rather hard to grasp.

I don’t think there is too much problem using minor modes, but minor modes wouldn’t fit very well with separate major modes. IMO it would be weird if turning on tree-sitter-mode changes the major mode.

Yuan

Re: Turing on tree-sitter

[Philip Kaludercic]

Yuan Fu <casouri@gmail.com> writes:

>> On Oct 10, 2022, at 7:52 AM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>> 
>>>> Is there a reason we can't use a minor mode?  Something like
>>>> 
>>>>        (add-hook 'python-mode-hook #'treesit-mode)
>>>> 
>>>> or a list
>>>> 
>>>>        (add-to-list 'treesit-modes 'python-mode)
>>>> 
>>>> ?
>>> 
>>> We could, if a minor mode is justified.  When this was previously
>>> brought up, someone said the justification for a minor mode was too
>>> weak in most cases.  But maybe we should revisit that idea.
>> 
>> I think a buffer-local `treesit-mode` plus a `global-treesit-mode` would
>> make a lot of sense, from a user's perspective.  This way they don't
>> have to hunt for the name of the boolean variable that their mode
>> decided to use to control the use of treesitter: all modes use the same
>> boolean variable called `treesit-mode`.
>
> Usually users set buffer-local variables in a major mode hook, which
> runs after the major mode is loaded, no? But major modes need to know
> whether to use tree-sitter up-front.

What if the hook is a noop and the mode definition just checks if the
function is a member?  Or is there no way to re-fontify a buffer?

> I don’t think xxx-mode-use-tree-sitter would be hard to find if every
> mode uses this pattern. Though now we are discussing adding separate
> major modes which wouldn’t use this variable. I think that indeed
> could be confusing. (How do I know if xxx language uses
> xxx-mode-use-tree-sitter or a separate major mode
> xxx-tree-sitter-mode?)

I think the issue here is not so much discoverability, but consistency.
Having to a different option for every mode can get annoying and is
difficult to automate.  It is also unnecessarily decentralised.  That is
why I think that having a single list like `treesit-modes' (by whatever
name) would be nice to have.  Using the easy customisation interface
you'd represent it as a set/radio menu where you get a list of supported
languages you can tick.

>> Then again, to me a minor mode is something so cheap that the idea that
>> "justification for a minor mode was too weak" is rather hard to grasp.
>
> I don’t think there is too much problem using minor modes, but minor
> modes wouldn’t fit very well with separate major modes. IMO it would
> be weird if turning on tree-sitter-mode changes the major mode.

I don't get why a `tree-sitter-mode' should change the major mode?

> Yuan

Re: Turing on tree-sitter

[Yuan Fu]

> On Oct 10, 2022, at 9:24 AM, Philip Kaludercic <philipk@posteo.net> wrote:
> 
> Yuan Fu <casouri@gmail.com> writes:
> 
>>> On Oct 10, 2022, at 7:52 AM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>>> 
>>>>> Is there a reason we can't use a minor mode?  Something like
>>>>> 
>>>>>       (add-hook 'python-mode-hook #'treesit-mode)
>>>>> 
>>>>> or a list
>>>>> 
>>>>>       (add-to-list 'treesit-modes 'python-mode)
>>>>> 
>>>>> ?
>>>> 
>>>> We could, if a minor mode is justified.  When this was previously
>>>> brought up, someone said the justification for a minor mode was too
>>>> weak in most cases.  But maybe we should revisit that idea.
>>> 
>>> I think a buffer-local `treesit-mode` plus a `global-treesit-mode` would
>>> make a lot of sense, from a user's perspective.  This way they don't
>>> have to hunt for the name of the boolean variable that their mode
>>> decided to use to control the use of treesitter: all modes use the same
>>> boolean variable called `treesit-mode`.
>> 
>> Usually users set buffer-local variables in a major mode hook, which
>> runs after the major mode is loaded, no? But major modes need to know
>> whether to use tree-sitter up-front.
> 
> What if the hook is a noop and the mode definition just checks if the
> function is a member?  Or is there no way to re-fontify a buffer?

IMO that’s a bit hacky, we have plenty better options.

> 
>> I don’t think xxx-mode-use-tree-sitter would be hard to find if every
>> mode uses this pattern. Though now we are discussing adding separate
>> major modes which wouldn’t use this variable. I think that indeed
>> could be confusing. (How do I know if xxx language uses
>> xxx-mode-use-tree-sitter or a separate major mode
>> xxx-tree-sitter-mode?)
> 
> I think the issue here is not so much discoverability, but consistency.
> Having to a different option for every mode can get annoying and is
> difficult to automate.  It is also unnecessarily decentralised.  That is
> why I think that having a single list like `treesit-modes' (by whatever
> name) would be nice to have.  Using the easy customisation interface
> you'd represent it as a set/radio menu where you get a list of supported
> languages you can tick.

That’s nice, but it wouldn’t make much sense to add A to the list when A-mode doesn’t actually support tree-sitter. I think of tree-sitter as something at major mode level.

> 
>>> Then again, to me a minor mode is something so cheap that the idea that
>>> "justification for a minor mode was too weak" is rather hard to grasp.
>> 
>> I don’t think there is too much problem using minor modes, but minor
>> modes wouldn’t fit very well with separate major modes. IMO it would
>> be weird if turning on tree-sitter-mode changes the major mode.
> 
> I don't get why a `tree-sitter-mode' should change the major mode?

My bad, I thought Theodor was saying he would add separate major modes alongside non-tree-sitter major modes for the same language. Reading his message I think he is only adding the tree-sitter powered mode into core and leaving the non-tree-sitter version out.

So IIUC we will have (a) major modes that supports both tree-sitter and non-tree-sitter, and (b) major modes that require tree-sitter to work.

Again, a M-x treesit-enable in a mode that requires tree-sitter to work doesn’t make much sense. I think having individual major-mode level toggle variables and a global toggle variable is conceptually more fit?

Yuan

Re: Turing on tree-sitter

[Theodor Thornhill]

>
>My bad, I thought Theodor was saying he would add separate major modes alongside non-tree-sitter major modes for the same language. Reading his message I think he is only adding the tree-sitter powered mode into core and leaving the non-tree-sitter version out.
>
>So IIUC we will have (a) major modes that supports both tree-sitter and non-tree-sitter, and (b) major modes that require tree-sitter to work.
>
>Again, a M-x treesit-enable in a mode that requires tree-sitter to work doesn’t make much sense. I think having individual major-mode level toggle variables and a global toggle variable is conceptually more fit?
>

Agreed!

Re: Turing on tree-sitter

[Daniel Martín]

Yuan Fu <casouri@gmail.com> writes:

>
> Again, a M-x treesit-enable in a mode that requires tree-sitter to
> work doesn’t make much sense. I think having individual major-mode
> level toggle variables and a global toggle variable is conceptually
> more fit?
>

Right now, changing the major mode variables to enable/disable
Tree-sitter requires a restart of the major mode to have Tree-sitter
manage the buffer.  I think making the variables a command/minor mode so
that all this preparatory work is done for users would be useful.

Re: Turing on tree-sitter

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> But we could also have a global "turn on tree-sitter support in every
> mode that has it" option, which will turn this on for those modes
> which can be supported by tree-sitter, if the appropriate
> language-support library is installed.

Yes, a user option that's a (derived-)mode predicate would be nice for
this.