Re: Tree-sitter maturity

[Philip Kaludercic <philipk@posteo.net>]

[Re-adding Emacs-Devel]

Daniel Colascione <dancol@dancol.org> writes:

> On December 31, 2024 8:00:17 AM EST, Philip Kaludercic <philipk@posteo.net> wrote:
>>Daniel Colascione <dancol@dancol.org> writes:
>>
>>[...]
>>
>>>>> There is also the general point of helping to realise software freedom,
>>>>> where a -ts-mode makes it much more difficult (though of course not
>>>>> impossible) to adjust a grammar.  Wasn't there some complication when
>>>>> trying to reload a grammar?  The additional dependencies and the
>>>>> indirect effect of changes compared with Elisp is something we should be
>>>>> concerned about when trying to maintain "the spirit of Emacs" (which of
>>>>> course means different things to different people).
>>>>>
>>>>> Vendoring might help to reproduce builds if that turns out to be a big
>>>>> issue, but I am not a fan of the additional hurdles in making use of the
>>>>> source code.  Does anyone know of alternative, less invested
>>>>> build-chain the re-uses the libtree-sitter.so library.
>>>>
>>>>Oh, and another point I have been reminded of while writing this: The
>>>>recent addition of more and more -ts-modes without "regular" -modes has
>>>>been slightly concerning.  While I understand that re-implementing a
>>>>"lua-mode" or "php-mode" from scratch is not an effort one wants to
>>>>impose on anymore, simpler files such as dockerfile-mode or go-mod-mode
>>>>/without/ Tree Sitter would be a nice thing to have for people on
>>>>systems without Tree Sitter, or without the ability to download and
>>>>build code from GitHub (e.g. missing internet access, without Git/GCC,
>>>>without the necessary development libraries).  Even if the experience
>>>>were degraded, just re-using the keywords to provide some basic
>>>>highlighting would be a nice fallback.
>>>
>>> I think it's inevitable that, long term, tree sitter becomes a
>>> mandatory dependency and we delete the bespoke Emacs modes. It just
>>> doesn't make sense to replicate per-language work the community is
>>> already doing. There's no economic or software freedom benefit in
>>> doing so. 
>>
>>As long as (live-)modifying a Tree Sitter grammar is significantly more
>>difficult than with the regular major modes, I think that -ts-modes have
>>a significant advantage -- especially for Emacs.
>
> You meant non-ts? 

No, I meant that modifying the grammar of a -ts-mode is more difficult,
as changes to the JS files requires a special toolchain and IIUC cannot
be reloaded inside of Emacs.  Or am I wrong?

>                   Anyway, I've already described ways to make modify
> TS easy. (For example, not introducing unnecessary complexity to the
> distribution process.) It'd be better to make TS customization easy
> than redundantly write and maintain TS and non-TS modes under the
> theory that the latter can be more easily customized.

I haven't been able to follow the mailing list for the past few days, I
would appreciate i you could give me the message ID where you elaborated
on this.

>>>           If we try, we'll just confuse users and ship modes that fall
>>> further and further behind those of other systems.
>>
>>Ideally, it should have been possible to hide the difference between
>>tree sitter and non-tree sitter modes.  From what I understand that was
>>not possible with the existing framework.  After all, most of the time
>>if tree sitter works, you don't notice it directly.  I *do* notice it if
>>something doesn't work as expected (e.g. `mark-sexp' not selecting the
>>right regions is something that has been a deal breaker for me).
>
> That's not my ideal.

To clarify, I meant that ideally the user shouldn't notice Tree Sitter,
not the implementer.  If everything works, it just improves indentation,
structural navigation, syntax highlighting, etc.  These are the effects
of tree sitter, not tree sitter itself.  This is different from other
editor-generic systems like LSP, as Eglot exposes some functionality
that people use directly, such as eglot-rename or eglot-code-actions.

> Making modes work with or without tree sitter would involve doing a
> lot of hairy language specific work twice, once to work with TS and
> once without. This duplication of effort just doesn't make any sense
> to me. How is anyone better off for having done this redundant work
> instead of something else?

Most of the time, the non-TS modes already exist and have accumulated a
lot of functionality over time: REPL support, integration with Flymake,
Eldoc, CAPF, custom user options, etc.  What irks me about some of the
new -ts-modes is that they provide a significant degradation in these
aspects for now.  We have to replicate all of this for -ts-modes.

> Even if we could: why confuse users with a choice they're not prepared
> to make? Why distract them with the difference between TS and non-TS
> (which I'll start calling "legacy modes", come to think of it)? Simple
> is better, especially for users.

I agree that it would be better if we could hide the distinction.  But
tree sitter modes are not in a position to replace the traditional modes
right now.  Especially if using them at all requires fetching source
code from GitHub.

One thing one can do remedy the concrete problem that we have -ts-modes
without corresponding non-ts-modes is to fall back onto
`define-generic-mode' and at least provide some basic syntax
highlighting for keywords -- which all -ts-modes have to declare anyway.
That way languages like dockerfile(-ts)-mode or go-mod(-ts)-mode would
at least do something, given that the information is already there.