Re: Suggesting that feature/tree-sitter be merged (was Re: Tree-sitter and major mode inheritance)

[Eli Zaretskii <eliz@gnu.org>]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: jostein@secure.kjonigsen.net,  casouri@gmail.com,  emacs-devel@gnu.org,
>   theo@thornhill.no,  jostein@kjonigsen.net
> Date: Sat, 19 Nov 2022 10:46:41 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > From where I stand, it makes very little sense to release Emacs 29
> > with tree-sitter support that is limited to primitives and some
> > minimal Lisp glue on top of that.  Tree-sitter was added to Emacs to
> > allow major modes provide better support for editing program source
> > code, so having tree-sitter "support" in Emacs 29 that didn't include
> > at least several major modes using it would be disappointing at best.
> > It would mean we ourselves have no idea how to make major modes use
> > the feature.  Moreover, adding those few major modes on the branch
> > exposed several deficiencies in the original design and
> > implementation, and required changes to make the integration better;
> > releasing Emacs 29 with those issues unresolved (and unknown) would
> > require significant, sometimes incompatible changes in the future,
> > which is another reason why it would be wrong.
> >
> > Basically, my firm belief is that adding to Emacs infrastructure
> > without user-level applications built on that infrastructure is wrong
> > and runs the risk of producing features that are not used or need deep
> > surgery before they become useful.  We should avoid doing that as much
> > as possible.
> 
> My question is, do these user-level applications have to be distributed
> along with Emacs, or could they be made to be "explicitly" opt-in by
> installing them from ELPA.

It depends, the decision should be on a case by case basis, IMO.  For
functionality that is part of what we want Emacs to have, yes, it
should be distributed with Emacs.

> In-core appears to usually bring a commitment to maintain a library,
> and deprecating can take years.  If Emacs 29 lays the technical
> foundations, the low-level API for treesitter to work, we can have
> packages on ELPA experiment with the higher-level abstractions.
> Whatever is the most successful approach, can be added to Emacs
> later on.

Emacs cannot come without support for important programming languages,
that would make no sense.  If we want to move towards tree-sitter as
the basis for some aspects of such major modes, we must have this in
core.  Having such important parts of Emacs in ELPA when we don't have
a way of bundling ELPA packages with an Emacs release tarball means a
deficiency in released versions of Emacs, so we should not go that
way.

I don't see why what was done on the branch with introducing
tree-sitter capabilities into major modes should be considered
"experiments", let alone not "successful".  What parts of that
concretely do you think belong to these categories, and why?

More generally, why should we be afraid of including new stuff in
Emacs, and instead designate it "experimental" and put it on ELPA?  We
never did that in the past, and I don't see why would we want that now
or in the future.  ELPA is not a platform for "experiments" in Emacs
development; the master branch and the feature branches are that
platform.

> >> As an aside: This might also be a good opportunity to clean up some of
> >> the current major mode implementations and make them more consistent.
> >> The issue with custom options to enable tree-sitter for every major mode
> >> has revealed an inherent duplication of features.  There are other
> >> inconsistencies, especially regarding bindings for equivalent operations
> >> (e.g. in interpreted language with a repl, how to load function into the
> >> current session: Lisp, Prolog, Python all differ in minor details).
> >
> > Cleaning up major modes is a Good Thing that needs no opportunities.
> > We should do that whenever we know and agree how.
> 
> Fair enough, but just as above I think these kinds of experiments are
> better made outside of the core, in ELPA, to avoid committing to
> mistakes.  If it works out, it can be added.

No, we should do that on feature branches, not on ELPA.  Certainly so
for changes that require changes on the C level.

Again, ELPA is not a place where we should develop Emacs.

> >> The current branch has major modes, should these be deleted before
> >> merging?
> >
> > Definitely not!  These modes are there because we want Emacs 29 to
> > have them, and we want users to use them and report back.
> 
> IIUC these modes aren't ripe yet, or at least aren't satisfying
> replacements for the existing modes.

What concretely isn't ripe?

And please note that Emacs 29 won't be released tomorrow or the next
week.  We have the whole release cycle ahead of us to figure out what
is not yet ripe for a release and either fix that or (in extreme
cases) remove that from Emacs.  I see no reason to make these
decisions today.  We used the feature branch for initial shakeup and
stabilization, and we now think the tree-sitter support is mature
enough to let more people use it and provide their feedback.

> If tree-sitter were not to be merged for that reason, that would
> delay the ability to use tree-sitter on a widespread basis for at
> least another release.  My proposal above would make it possible,
> and encourage users to report on their experience, while allowing
> for the flexibility to make the right decisions in the long term.

If that was your reasoning, then I think you are three steps ahead of
where we are, and you are trying to find solutions for problems that
don't necessarily exist.  We should see what concrete problems are
left after the merge, and take it from there.  We have ample time for
figuring that out and fixing whatever will need fixing.