Re: Tree-sitter maturity

[Daniel Colascione <dancol@dancol.org>]

On December 27, 2024 9:19:12 AM EST, Philip Kaludercic <philipk@posteo.net> wrote:
>Daniel Colascione <dancol@dancol.org> writes:
>
>> On December 27, 2024 7:40:19 AM EST, Eli Zaretskii <eliz@gnu.org> wrote:
>>>> From: Philip Kaludercic <philipk@posteo.net>
>>>> Cc: Xiyue Deng <manphiz@gmail.com>,  emacs-devel@gnu.org
>>>> Date: Fri, 27 Dec 2024 10:54:29 +0000
>>>> 
>>>> Richard Stallman <rms@gnu.org> writes:
>>>> 
>>>> > If we add something like this to Emacs, there is an issue we need to
>>>> > take care about: to make carefully sure that it does not install
>>>> > any nonfree grammars.  I don't know how those grammars are released,
>>>> > ir by whom, or how much they care about free software.  We can't
>>>> > take for granted that they do.
>>>> >
>>>> > Perhaps we could check automatically that the grammar found is properly
>>>> > licenses, and disregard any grammars that are not free.
>>>> >
>>>> > By contrast, if grammars are going to be packaged and released for
>>>> > distros, and chosen for installation by users, then it is the user's
>>>> > responsibility, not Emacs's responsibility, to reject the nonfree ones
>>>> > (and the GNU/Linux distro might insist on that).
>>>> 
>>>> It might take a while for that to happen, which is why I still believe
>>>> it would be better if tree-sitter major modes would populate
>>>> `treesit-language-source-alist' on their own, and point to the specific
>>>> checkouts that the major mode developer tested their implementation
>>>> against.
>>>
>>>We could have done that, but there's no way we could keep the value of
>>>treesit-language-source-alist up-to-date, because the grammar
>>>libraries put out new versions much more frequently than Emacs
>>>releases, especially if you consider libraries that have no official
>>>versions at all (in which case we can only point to some revision in
>>>their repository).
>>>
>>>The question that bothers me is how useful is it to have
>>>treesit-language-source-alist that is outdated?  What do we expect the
>>>users to do with such an outdated value?
>>>
>>
>> Why not just vendor all the grammars with the Emacs modes that use them?
>
>I am guessing part of the reason is that TS grammars are not fun to
>build.  IIRC they are specified in a Javascript DSL (that used to
>require node.js but AFAIU works with other implementations as well),
>that a program written in Rust translates to C code.  So do we vendor
>the DSL and depend on the TreeSitter toolchain or do we vender the
>generated code?

It's a shame there's no way to write TS grammars in plain elisp. I figure vendoring both the source and the generated code would be best, as it'd allow building Emacs anywhere but still make it convenient on systems with needed tools (JS runtime, Rust, etc.) to update and modify the grammar. As with any scheme involving checking in generated outputs, the source and output can get out of sync, but I think there are build time guardrails we can build to make sure it doesn't happen.