Re: Tree-sitter maturity

[Yuan Fu <casouri@gmail.com>]

> On Dec 29, 2024, at 11:06 AM, Björn Bidar <bjorn.bidar@thaodan.de> wrote:
> 
> Daniel Colascione <dancol@dancol.org> writes:
> 
>> On December 29, 2024 11:02:47 AM EST, "Björn Bidar" <bjorn.bidar@thaodan.de> wrote:
>>> Daniel Colascione <dancol@dancol.org> writes:
>>> 
>>>> On December 29, 2024 10:05:26 AM EST, "Björn Bidar"
>>>> <bjorn.bidar@thaodan.de> wrote:
>>>>> Daniel Colascione <dancol@dancol.org> writes:
>>>>> 
>>>>>> On December 27, 2024 9:59:14 AM EST, Eli Zaretskii <eliz@gnu.org> wrote:
>>>>>>>> Date: Fri, 27 Dec 2024 08:46:06 -0500
>>>>>>>> From: Daniel Colascione <dancol@dancol.org>
>>>>>>>> CC: rms@gnu.org, manphiz@gmail.com
>>>>>>>> 
>>>>>>>>>> It might take a while for that to happen, which is why I still
>>>>>>>>>> believe
>>>>>>>>>> it would be better if tree-sitter major modes would populate
>>>>>>>>>> `treesit-language-source-alist' on their own, and point to the
>>>>>>>>>> specific
>>>>>>>>>> checkouts that the major mode developer tested their implementation
>>>>>>>>>> against.
>>>>>>>>> 
>>>>>>>>> We could have done that, but there's no way we could keep the value of
>>>>>>>>> treesit-language-source-alist up-to-date, because the grammar
>>>>>>>>> libraries put out new versions much more frequently than Emacs
>>>>>>>>> releases, especially if you consider libraries that have no official
>>>>>>>>> versions at all (in which case we can only point to some revision in
>>>>>>>>> their repository).
>>>>>>>>> 
>>>>>>>>> The question that bothers me is how useful is it to have
>>>>>>>>> treesit-language-source-alist that is outdated?  What do we expect the
>>>>>>>>> users to do with such an outdated value?
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> Why not just vendor all the grammars with the Emacs modes that
>>>>>>>> use them?
>>>>>>> 
>>>>>>> We'd need to ask their developers to agree to this. 
>>>>>> 
>>>>>> Why? They're free software. For copyright assignment? Seems like an
>>>>>> exception would make sense here.
>>>>>> 
>>>>>>> Other than that,
>>>>>>> I don't see how is that different from pointing to a specific version
>>>>>>> of each grammar: both will be outdated a short time after we point to
>>>>>>> the version or release Emacs with that version.
>>>>>>> 
>>>>>>> So why do you think this is better?
>>>>>> 
>>>>>> Vendoring enables building a full featured Emacs without a network
>>>>>> connection and guarantees build reproducibility in perpetuity.
>>>>> 
>>>>> Did you think of the long term consequences?
>>>>> 
>>>>> The embedded dependencies would have to be maintained first by Emacs and
>>>>> later by packagers.
>>>>> 
>>>>> All the infrastructure around syncing of grammars is time spend that
>>>>> could spend on more long term efforts such as stabilizing the
>>>>> tree-sitter based modes to not break as easy on grammar changes or to
>>>>> improve tree-sitter it self.
>>>> 
>>>> I've vendored plenty of things. Works fine in practice. Big programs
>>>> like Firefox vendor the world too, and they work fine. It's really not
>>>> that much work. It eliminates entire classes of problem. It's going to
>>>> take more time to deal with the problems of taking a dependency and
>>>> the headaches of not having a stable interface than it would to set up
>>>> a few git subtrees or submodules and invoke their build system from
>>>> that of Emacs.
>>> 
>>> Big programs like Firefox vendor the world only for packagers to have
>>> to revert those. 
>> 
>> These packagers are wrong, FWIW. Unbundling is needless and often
>> introduces bugs.
> 
> It introduces bugs in software with generally unstable APIs/ABIs or software
> using unfinished/unreleased versions of software. Partially the problem
> we are facing here right now.
> 
> The rat tail of issues this can entrail can be long.
> 
> FWIW You called over half of the Unix community wrong where bundled
> dependencies are frowned or forbidden upon.
> 
>> In the mobile world, popular OSes seldom provide
>> libraries. Apps are expected to bundle their dependencies. The sky
>> doesn't fall. In fact, the mobile app ecosystem is healthier and more
>> secure than the desktop one precisely because it isn't burdened by
>> ideas that no longer make sense in a modern technical context,
>> e.g. that apps should casually share libraries.
> 
> I work on mobile operating systems, what you describe is double edged
> sword. The applications size's increase and the party to blame for
> security issues moves from the os to the application developer. The
> mobile OS would had plenty of issues from this practice notably for
> example the log4you debacle.
> 
> Most mobile operating systems provide their own set of available
> libraries, apps are not expected to bundle dependencies unless they are
> not available for that OS.
> 
> Part of the issue is that library dependencies moving faster than many
> operating systems can or with stable APIs. The end result of such lets
> bundled everything approach is that you have use the exact chain of
> dependencies to build a software which is awesome if you like the fire
> and forget approach of software development.
> 
> 
> To me what you write reads like mobile operating systems = JavaScript/AI developers.
> 
>>> Vendoring only works long enough until the dependencies
>>> you have vendored are not out of date. 
>> 
>> It doesn't matter whether the dependency is out of date so long as
>> it's in sync with code that interacts with it. It's even worse when
>> the dependency doesn't make any compatibility guarantees. IMHO, the
>> only reasonable way to consume a dependency with an unstable interface
>> is to bundle or hash-lock or outright vendor it.
> 
> If you have software which has short life cycles this can work but I
> don't think this works for Emacs.
> Further bundled dependencies require to patch the software bundling the
> dependency to fix bugs and security issue. Bugfixes are not really an
> issue with grammars but with libtree-sitter which Emacs depend on.
> Putting bundled grammars and libtree-sitter in this equation makes it
> harder to maintain the Emacs package since I have to watch to not break
> the embedded grammars when updating tree-sitter or it's dependencies.
> 
>>> It is something which only works
>>> in projects who control most of their dependency chain and/or have a
>>> fire and forget approach of software development.
>>> 
>>>> It's not even the precise mechanics: pulling down a grammar by hash is
>>>> tantamount to just checking in the grammar, but with more moving
>>>> parts. You still pair one to one the grammar and the Lisp code meant
>>>> to use it so you don't end up chasing down weird compatibility
>>>> issues. IMHO, since they're tightly coupled anyway, we might as well
>>>> distribute them together.
>>>> 
>>>> As for changing TS grammars not to break: why do you think that would
>>>> be feasible? So far TS grammar authors haven't felt particularly
>>>> obligated to maintain compatibility.
>>> 
>>> I don't know exactly to be honst but I don't think we are alone with
>>> this issue. If we are we should check out it is handled in other
>>> editors.
>> 
>> You're right that this is a problem everyone should be hitting.
> 
> Maybe there's a way around it. The only way is to reach out to other
> projects using the library or upstream.

Nvim vendors grammars, it also has a “database” repo that pins grammars (https://github.com/nvim-treesitter/nvim-treesitter), that repo also provides a command to install grammars.

IIUC Pulsar (community-supported Atom successor) vendors grammars too, because that’s what Atom did originally.

Helix I think only provide a command to install grammars (plus their database pinning grammar versions).

For context, Emacs also has a command to install grammars, but we don’t provide a database nor version pinning.

Nvim and Pulsar have plans to move towards using wasm grammars. Tree-sitter recently gained the ability to compile grammars into wasm object files and load wasm grammars. Pulsar is built on electron so naturally has access to wasm, nvim is adding a wasm runtime as an optional dependency.

Yuan