Re: Suggesting that feature/tree-sitter be merged (was Re: Tree-sitter and major mode inheritance)

[Philip Kaludercic <philipk@posteo.net>]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: jostein@secure.kjonigsen.net,  casouri@gmail.com,  emacs-devel@gnu.org,
>>   theo@thornhill.no,  jostein@kjonigsen.net
>> Date: Sat, 19 Nov 2022 10:46:41 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > From where I stand, it makes very little sense to release Emacs 29
>> > with tree-sitter support that is limited to primitives and some
>> > minimal Lisp glue on top of that.  Tree-sitter was added to Emacs to
>> > allow major modes provide better support for editing program source
>> > code, so having tree-sitter "support" in Emacs 29 that didn't include
>> > at least several major modes using it would be disappointing at best.
>> > It would mean we ourselves have no idea how to make major modes use
>> > the feature.  Moreover, adding those few major modes on the branch
>> > exposed several deficiencies in the original design and
>> > implementation, and required changes to make the integration better;
>> > releasing Emacs 29 with those issues unresolved (and unknown) would
>> > require significant, sometimes incompatible changes in the future,
>> > which is another reason why it would be wrong.
>> >
>> > Basically, my firm belief is that adding to Emacs infrastructure
>> > without user-level applications built on that infrastructure is wrong
>> > and runs the risk of producing features that are not used or need deep
>> > surgery before they become useful.  We should avoid doing that as much
>> > as possible.
>> 
>> My question is, do these user-level applications have to be distributed
>> along with Emacs, or could they be made to be "explicitly" opt-in by
>> installing them from ELPA.
>
> It depends, the decision should be on a case by case basis, IMO.  For
> functionality that is part of what we want Emacs to have, yes, it
> should be distributed with Emacs.
>
>> In-core appears to usually bring a commitment to maintain a library,
>> and deprecating can take years.  If Emacs 29 lays the technical
>> foundations, the low-level API for treesitter to work, we can have
>> packages on ELPA experiment with the higher-level abstractions.
>> Whatever is the most successful approach, can be added to Emacs
>> later on.
>
> Emacs cannot come without support for important programming languages,
> that would make no sense.  If we want to move towards tree-sitter as
> the basis for some aspects of such major modes, we must have this in
> core.  Having such important parts of Emacs in ELPA when we don't have
> a way of bundling ELPA packages with an Emacs release tarball means a
> deficiency in released versions of Emacs, so we should not go that
> way.
>
> I don't see why what was done on the branch with introducing
> tree-sitter capabilities into major modes should be considered
> "experiments", let alone not "successful".  What parts of that
> concretely do you think belong to these categories, and why?
>
> More generally, why should we be afraid of including new stuff in
> Emacs, and instead designate it "experimental" and put it on ELPA?  We
> never did that in the past, and I don't see why would we want that now
> or in the future.  ELPA is not a platform for "experiments" in Emacs
> development; the master branch and the feature branches are that
> platform.
>
>> >> As an aside: This might also be a good opportunity to clean up some of
>> >> the current major mode implementations and make them more consistent.
>> >> The issue with custom options to enable tree-sitter for every major mode
>> >> has revealed an inherent duplication of features.  There are other
>> >> inconsistencies, especially regarding bindings for equivalent operations
>> >> (e.g. in interpreted language with a repl, how to load function into the
>> >> current session: Lisp, Prolog, Python all differ in minor details).
>> >
>> > Cleaning up major modes is a Good Thing that needs no opportunities.
>> > We should do that whenever we know and agree how.
>> 
>> Fair enough, but just as above I think these kinds of experiments are
>> better made outside of the core, in ELPA, to avoid committing to
>> mistakes.  If it works out, it can be added.
>
> No, we should do that on feature branches, not on ELPA.  Certainly so
> for changes that require changes on the C level.
>
> Again, ELPA is not a place where we should develop Emacs.
>
>> >> The current branch has major modes, should these be deleted before
>> >> merging?
>> >
>> > Definitely not!  These modes are there because we want Emacs 29 to
>> > have them, and we want users to use them and report back.
>> 
>> IIUC these modes aren't ripe yet, or at least aren't satisfying
>> replacements for the existing modes.
>
> What concretely isn't ripe?

Jostein said:

  Me and Theodor faced these same issues with "our" C# and TypeScript
  major-modes, and the only "clean" way we agreed we could make this
  work was to create wholly new implementations. I can come up with many
  good, objective reasons for this, but I think Theodor has already
  represented this view fairly well.

> And please note that Emacs 29 won't be released tomorrow or the next
> week.  We have the whole release cycle ahead of us to figure out what
> is not yet ripe for a release and either fix that or (in extreme
> cases) remove that from Emacs.  I see no reason to make these
> decisions today.  We used the feature branch for initial shakeup and
> stabilization, and we now think the tree-sitter support is mature
> enough to let more people use it and provide their feedback.

Naturally, I didn't understand this to be a discussion on an immediate
decision.

>> If tree-sitter were not to be merged for that reason, that would
>> delay the ability to use tree-sitter on a widespread basis for at
>> least another release.  My proposal above would make it possible,
>> and encourage users to report on their experience, while allowing
>> for the flexibility to make the right decisions in the long term.
>
> If that was your reasoning, then I think you are three steps ahead of
> where we are, and you are trying to find solutions for problems that
> don't necessarily exist.  We should see what concrete problems are
> left after the merge, and take it from there.  We have ample time for
> figuring that out and fixing whatever will need fixing.

You are right, I'll have to test the branch more seriously.