Re: Tree-sitter maturity

[Daniel Colascione <dancol@dancol.org>]

On January 4, 2025 3:57:48 PM EST, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Sat, 04 Jan 2025 15:46:49 -0500
>> From: Daniel Colascione <dancol@dancol.org>
>> CC: owinebar@gmail.com, bjorn.bidar@thaodan.de, philipk@posteo.net,
>>  emacs-devel@gnu.org, rms@gnu.org, manphiz@gmail.com
>> 
>> >> And it creates the need to do code distribution in a bespoke way. How is that a net win?
>> >
>> >It's _our_ win.  The problem is shifted to the distros, where I think
>> >it belongs.
>> 
>> No, it isn't, because distros *can't* do a good job of this.
>
>Then they need to get their act together and improve.

They cannot and will not do major software engineering to work around compatibility problems that should not exist. Do you expect a Fedora packager to notice or care when the distro version of some grammar is subtly incompatible with foo-ts-mode.el in Emacs? And patch the grammar or Emacs or both? And for the Debian and Arch packagers and so to do that too? Probably with subtly different bugs? We ought to have learned after the openssl security debacle not to let distros make nontrivial code changes.

>> Besides: plenty of people (I'd guess a majority going by how the industry is set up) were using Emacs outside the context of a distro. If we weren't allergic to telemetry, we'd know.
>
>People who do that (I'm one of them, btw) can build their grammars.  I
>do.

Yeah it's totally normal to download a macOS or Windows package and then have to set up a development tool chain to make the program they just installed actually work. Nothing user hostile about that.

It's really hard to see how this "rely on the distro" model is supposed to work in the real world. Besides, some of the antique platforms you insist we support, like MS-DOS and Windows 98, have nothing resembling a package manager, packages, or any way to automatically get some compatible version of a grammar. Not including the batteries means the toy doesn't work.

>
>> >> Nobody has been able to describe these legal aspects. Grammars are free software. GPL compatible, too.
>> >> That means we can put them in Emacs. That's what software freedom means.
>> >
>> >I did explain that, and don't want to repeat.  You can consider those
>> >reasons unimportant, but I don't (and cannot, really: I don't call the
>> >shots in this particular game).
>> 
>> No, you didn't explain. You asserted.
>> 
>> You said that we would need to get written permission from grammar projects to include their code in Emacs. When I asked where this requirement comes from, you said it had always been this way and that RMS might have more information. He appears not to.
>
>RMS explicitly asked me to do that for every package we admit into
>Emacs.

Is RMS a judge? The assertion is that there is a *legal requirement* to get permission to include something in Emacs. I don't think any such requirement exists. RMS may impose that requirement, but he's not the law, and unlike the law, his policy can change.

>> >You are missing the point.  My point is that several branches means we
>> >need to match a different version of the library to each branch
>> 
>> If you check the grammar into the tree, the version in each branch matches that branch automatically. That's what a version control system is. It's the essence of what it does. Same goes for submodules, which are just checked-in pointers.
>
>Again, you are missing the point: we need to find an appropriate
>version for each branch, and we need to track the versions of the
>grammar so ass to be able to know, for each of our branches, the
>compatible version of the grammar.  The maintenance burden is
>multiplied by the branches.

The way it should work is that you have a checked in foo-ts-mode.el and right alongside it a foo.js file that describes the tree sitter grammar that foo-ts-mode uses. When you cut a branch, you snapshot both files. No extra overhead.

Want to update foo.js? Just sync it from upstream and check it in, just like you'd check in any modified foo-ts-mode.el. You only have to do that when you want to update the grammar, and that happens whenever the mode author feels like doing it --- *exactly* the way it works today when you have a standalone foo-mode.el.

>> Can you give me a concrete example of how checking in code burdens branch maintenance? Your position isn't making any sense to me in context of how version control systems work.
>
>Finding the matching version and testing whether it matches does.

You don't do that when you cut a branch. You do that when you update the mode or its grammar, and no matter how you associate modes with their grammars, you have to test the combination.

>> >> >The difference is that the RI changes.  And that's not something to
>> >> >ignore, from where I stand.
>> >> 
>> >> Huh? In what possible way could a bespoke downloader be a better engineering choice than submodules?
>> >
>> >I didn't say anything about engineering, I was talking about the
>> >responsibility.
>> 
>> Good engineering minimizes the number of moving parts and needless failure points in a system.
>
>Responsibility has nothing to do with engineering or moving parts.

I have no idea what you're talking about. There's responsibility to make a worse engineering choice and make a custom downloader? What's the nature of this responsibility?