Re: Tree-sitter maturity

[Daniel Colascione <dancol@dancol.org>]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Sun, 29 Dec 2024 05:01:23 -0500
>> From: Daniel Colascione <dancol@dancol.org>
>> CC: casouri@gmail.com, emacs-devel@gnu.org
>> 
>> 
>> 
>> On December 29, 2024 4:24:11 AM EST, Eli Zaretskii <eliz@gnu.org> wrote:
>> >> Date: Sun, 29 Dec 2024 04:14:37 -0500
>> >> From: Daniel Colascione <dancol@dancol.org>
>> >> CC: emacs-devel@gnu.org
>> >> 
>> >> The grammars don't make any backwards compatibility
>> >> guarantees. There have been multiple Emacs bugs arising from
>> >> grammars unilaterally changing terminal names and such. ISTM the
>> >> only way to guarantee compatibility is to vendor the whole stack.
>> >
>> >I hope that, as the use of tree-sitter and the grammars increases, the
>> >distros will work with the grammar developers so that the latter get
>> >their act together and start producing more dependable releases with
>> >clear versioning and library compatibilities.
>> 
>> Is hope a strategy?
>
> No, just hope.

I think grammars and are too tightly coupled to AST consumers to ever
have stable interfaces.  If the "API" of the system is the precise set
of terminals and non-terminals a grammar recognizes in a stream of text,
then, when you enlarge the recognized language, you break old consumers,
who have no idea what to do with AST nodes that didn't even exist when
they were written. Besides: augmenting a grammar with support for a new
language feature sometimes requires changing the parse trees for even
old code!

That's why I come down so strongly in favor of bundling grammars with
code that consumes them.  If you do so, then at worst, your combined
grammar-and-Emacs-mode might not recognize new language features, but
it's not going to suddenly choke on language features that
haven't changed!

Any kind of stable language-analysis library would have to work at a
higher level than tree-sitter.  Such a library would have to provide
interfaces for highlighting, indentation, completion, and so on: these
concepts, unlike AST node identity, can be stable across
language revisions.  It'd look a lot like LSP.

(AFAIK, there's no general-purpose implementation of this concept.
LSP has some support for highlighting, but it's meant as a helper for
built-in highlighting, not a total replacement.)

Tree sitter is great, but I'd much rather treat it more like flex and
bison than as some externally-shipped thing.  Nobody objects to a
program having a .y file and generating the .c at build time.
Nobody insists that we dynamically link against some separately-shipped
Bison parser output.  We don't have, e.g. GCC link against some kind of
libcparser.so and expect the latter to provide a long-term stable
interface.  GCC just does its own parsing, and there's no expectation
that GCC's internal parser present a stable interface to a different
part of the same GCC program.

It's because of this dynamic, BTW, that I really want c-ts-mode *not* to
provide indentation customization based on raw tree-sitter AST queries
but instead emulate cc-mode's higher level customization.