Re: Unifying "foo-mode"s and "foo-ts-mode"s

[Eli Zaretskii <eliz@gnu.org>]

> From: Richard Stallman <rms@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Sat, 31 Dec 2022 22:03:55 -0500
> 
> The use of tree-sitter logically ought to be a minor mode.

We tried to do it this way at first.  It didn't work well and had
significant difficulties, so we decided a separate major mode is a
better idea.  See below for some details; they were all discussed here
at the time (in October).

> We don't define c-mode and c-fl-mode to specify whether to enable Font
> Lock mode in a file of C code.  Rather, We have c-mode, plus another
> orthogonal command, font-lock-mode.

That is not the whole picture, as you know.  font-lock-mode just
activates the fontifications; the definitions of what to fontify and
how are the responsibility of the major mode.  These definitions are
completely different for tree-sitter based fontifications, as they are
not based on regexps and syntax tables.

And the features based on tree-sitter are not limited to
fontifications, they also support indentation, navigation by defun and
other language-dependent program components, and Imenu indices of
functions, structures, etc.  Each one of these needs to change quite a
few functions and variables of the particular major mode to switch
between tree-sitter and the "traditional" ways, and we also needed to
introduce a lot of new data structures that are not needed for the
"traditional" modes.

Adding all that to the existing modes would mean significant
complication of those existing modes (thus risking destabilizing
them), and would require no less than enthusiastic support from the
maintainers of those modes to get this done cleanly, safely, and in a
reasonable time.  I considered that to be "a bridge too far", and
instead decided to go with a safer, simpler alternative that we have
now.  This allowed us to add tree-sitter supported modes for quite a
few languages and file types in a very short time, and to do that
relatively cleanly.  Most importantly, those additions almost never
touch the modes for the same languages that existed before, thus
allowing us to introduce this new technology in a smooth way, leaving
it to the users to decide whether and when they want to try these new
modes.  This opt-in way is very important given the fact that distros
seem to not yet be organized well enough for providing the relevant
libraries for end-user installation.

> Likewise, the Emacs user interface ought to have another minor-mode
> command, tree-sitter-mode.
> 
> How it "really works" under the hood is another issue.
> 
> Users will be able to control the default for enabling tree-sitter in
> various major modes with the existing general mechanism that is used
> to control whether to enable Font Lodk mode and other minor modes: the
> major mode's mode hook.

This idealistic vision turned out not to work well enough in practice,
especially for large and complex major modes such as CC mode.  Many
settings and sub-features of the original mode make n o sense when
tree-sitter is used, and a huge amount of code based on regexps,
syntax-ppss, and ad-hoc algorithms needs to be bypassed or switched
off with tree-sitter.  It is impractical to do all that in a way that
both takes some reasonably short time and avoids a high risk of
breaking modes that have been working reasonably well for decades,
thus annoying users who may not even want to use tree-sitter.  I
couldn't allow such risky changes in Emacs 29, and I didn't want to
delay this to Emacs 30, which means another year r two.

So instead of turning on a local minor mode, users will have to invoke
a command that actually turns on a major mode.  FWIW, I don't see the
significant difference between these two alternatives, for the user
POV.  And it is quite possible that we will eventually decide to make
these minor modes, if we find a good way of combining them with the
major mode for the specific programming language.  But this is in the
future; we need first to collect user and maintenance experience,
without which there are too many questions and issues (some of which
are being raised already on this list) for which we have no good,
solid answers, only opinions based on information and experience
outside of Emacs, which from my POV are not necessarily valid for us.