Tree-sitter and major mode inheritance

Tree-sitter and major mode inheritance

[Yuan Fu]

So I’m trying to merge css-ts-mode with css-mode. Scss-mode inherits css-mode, but if user enables tree-sitter for css-mode, scss-mode will inherit all that tree-sitter setup, and lose all the native css setup. Then if a user doesn’t want to enable tree-sitter in scss-mode, too bad: scss-mode breaks.

Essentially scss-mode needs to be able to control which parts of css-mode’s setup it wants to inherit—native setup or tree-sitter setup—regardless of whether css-mode enables tree-sitter or not.

I wonder if we can do something like this:

             css-mode
                 |
       +---------+-----+-----------+
       |               |           |
css-native-mode   css-ts-mode  scss-mode
                                   |
                              +----+------------+
                              |                 |
                        scss-native-mode   scss-ts-mode

css-mode: a virtual mode, only sets up basic things that both native and tree-sitter mode needs, like comment-start.
css-native-mode: native setup
css-ts-mode: tree-sitter setup

scss-mode: a virtual mode, inherits css-mode
scss-native-mode: native setup
scss-ts-mode: tree-sitter setup

And user could use major-mode-remap-alist to choose which mode they want:

(css-mode . css-ts-mode) for enabling tree-sitter
(css-mode . css-native-mode) for not enabling tree-sitter

This could also used for other modes, like c-mode: c-mode, c-native-mode (cc-mode), c-ts-mode.

Yuan

Suggesting that feature/tree-sitter be merged (was Re: Tree-sitter and major mode inheritance)

[Jostein Kjønigsen]

[-- Attachment #1: Type: text/plain, Size: 4662 bytes --]

Hey everyone.

I know this has been said before, by people which by far has been 
contributing much more than me... But I still don't think it's a good 
idea to replace the implementation in existing major-modes with 
tree-sitter implementations, nor selectively activate tree-sitter in 
major-modes prone to inhetitence and derivation.

Me and Theodor faced these same issues with "our" C# and TypeScript 
major-modes, and the only "clean" way we agreed we could make this work 
was to create wholly new implementations. I can come up with many good, 
objective reasons for this, but I think Theodor has already represented 
this view fairly well.

While for the sake of brevity, I'll not diving deeply into this 
particular thing, I will say this: A new tree-sitter based major-mode 
free of compatibility concerns allowed us to create entirely new 
major-modes fixing most of our existing bugs, faster than we before 
would be able to fix a single bug. My personal view is that mixing 
existing major-modes with tree-sitter represents absolutely the worst of 
all worlds. It maintains all existing complexities, provides us with 
very few benefits, and at the same time adds complexities we didn't use 
to have. To me, that's a net negative.

Somewhat surprising to me, I see this is a somewhat controversial point 
of view and not as clear cut a matter as I would have expected it to be. 
I realize and respect that final decisions in these matter might take 
some time to mature. Time which given our current approach detracts from 
the momentum tree-sitter has been having.

At this point poor Yuan Fu here has spent quite a bit of time and effort 
getting a core tree-sitter interface into Emacs. I would really hate to 
see all that effort be for nothing, because we end up conflating a 
creating a core tree-sitter feature with how this feature should best be 
employed in subsequent major-modes.

So in the name of pragmatism, I propose a compromise of sorts.

Instead of waiting for "every" major-mode to be re-implemented into a 
tree-sitter derivative in the feature/tree-sitter branch before we 
merge... How about we just accept the current "core" tree-sitter 
implementation as good enough, and consider merging that to git master 
as is.

This will allow us to land one important mile-stone, while giving us the 
head-room to further discuss how we should best 
implement/reimplement/"upgrade" existing major-modes to take advantage 
of tree-sitter.

It will also allow third-party packages to make use of tree-sitter in 
Emacs core instead coming up with its own implementation, beginning a 
defacto standardization of this new API (which I may note already has a 
competing implementation in MELPA).

That is to say: We should land the core library, with holding it hostage 
to all its possible consumers being implemented.

How about it? Are there any good arguments for NOT merging 
feature/tree-sitter at this point? :)

--
Kind regards
*Jostein Kjønigsen*

jostein@kjonigsen.net 🍵 jostein@gmail.com
https://jostein.kjønigsen.no <https://jostein.kjønigsen.no>
On 16.11.2022 21:45, Yuan Fu wrote:
> So I’m trying to merge css-ts-mode with css-mode. Scss-mode inherits css-mode, but if user enables tree-sitter for css-mode, scss-mode will inherit all that tree-sitter setup, and lose all the native css setup. Then if a user doesn’t want to enable tree-sitter in scss-mode, too bad: scss-mode breaks.
>
> Essentially scss-mode needs to be able to control which parts of css-mode’s setup it wants to inherit—native setup or tree-sitter setup—regardless of whether css-mode enables tree-sitter or not.
>
> I wonder if we can do something like this:
>
>               css-mode
>                   |
>         +---------+-----+-----------+
>         |               |           |
> css-native-mode   css-ts-mode  scss-mode
>                                     |
>                                +----+------------+
>                                |                 |
>                          scss-native-mode   scss-ts-mode
>
> css-mode: a virtual mode, only sets up basic things that both native and tree-sitter mode needs, like comment-start.
> css-native-mode: native setup
> css-ts-mode: tree-sitter setup
>
> scss-mode: a virtual mode, inherits css-mode
> scss-native-mode: native setup
> scss-ts-mode: tree-sitter setup
>
> And user could use major-mode-remap-alist to choose which mode they want:
>
> (css-mode . css-ts-mode) for enabling tree-sitter
> (css-mode . css-native-mode) for not enabling tree-sitter
>
> This could also used for other modes, like c-mode: c-mode, c-native-mode (cc-mode), c-ts-mode.
>
> Yuan

[-- Attachment #2: Type: text/html, Size: 5607 bytes --]

Re: Suggesting that feature/tree-sitter be merged (was Re: Tree-sitter and major mode inheritance)

[Philip Kaludercic]

Jostein Kjønigsen <jostein@secure.kjonigsen.net> writes:

> Instead of waiting for "every" major-mode to be re-implemented into a
> tree-sitter derivative in the feature/tree-sitter branch before we
> merge... How about we just accept the current "core" tree-sitter
> implementation as good enough, and consider merging that to git master
> as is.

I think this sounds like a good idea -- as someone who has mostly just
been following the discussions.  The core bindings and major modes that
are based on these are separate issues, with a clear dependency linked
them.

As an aside: This might also be a good opportunity to clean up some of
the current major mode implementations and make them more consistent.
The issue with custom options to enable tree-sitter for every major mode
has revealed an inherent duplication of features.  There are other
inconsistencies, especially regarding bindings for equivalent operations
(e.g. in interpreted language with a repl, how to load function into the
current session: Lisp, Prolog, Python all differ in minor details).

I can imagine a more specialised `define-generic-mode' could be of use
here, along with more "abstract" major modes for various types of
programming languages (using `prog-mode' as a base to add
`compiled-prog-mode' that has generic commands for building program,
`interpreted-prog-mode' that has generic commands for REPL
communication, ...), where the tree-sitter configuration would be one of
the attributes these modes would specify.

[...]

> How about it? Are there any good arguments for NOT merging
> feature/tree-sitter at this point? :)

The current branch has major modes, should these be deleted before
merging?

Re: Suggesting that feature/tree-sitter be merged (was Re: Tree-sitter and major mode inheritance)

[Yuan Fu]

> On Nov 18, 2022, at 2:34 PM, Philip Kaludercic <philipk@posteo.net> wrote:
> 
> Jostein Kjønigsen <jostein@secure.kjonigsen.net> writes:
> 
>> Instead of waiting for "every" major-mode to be re-implemented into a
>> tree-sitter derivative in the feature/tree-sitter branch before we
>> merge... How about we just accept the current "core" tree-sitter
>> implementation as good enough, and consider merging that to git master
>> as is.
> 
> I think this sounds like a good idea -- as someone who has mostly just
> been following the discussions.  The core bindings and major modes that
> are based on these are separate issues, with a clear dependency linked
> them.
> 
> As an aside: This might also be a good opportunity to clean up some of
> the current major mode implementations and make them more consistent.
> The issue with custom options to enable tree-sitter for every major mode
> has revealed an inherent duplication of features.  There are other
> inconsistencies, especially regarding bindings for equivalent operations
> (e.g. in interpreted language with a repl, how to load function into the
> current session: Lisp, Prolog, Python all differ in minor details).

I’ve though of this too, other things are indent level, and documentation. I wrote ghelp[1] to get a uniform interface for getting documentation in different major modes (because I don’t have the heart to understand and modify help.el). A builtin, unified documentation system would be nice, like eldoc. But eldoc is for at-point short and quick signature/doc more than for full-fledged documentation like help.el.


> I can imagine a more specialised `define-generic-mode' could be of use
> here, along with more "abstract" major modes for various types of
> programming languages (using `prog-mode' as a base to add
> `compiled-prog-mode' that has generic commands for building program,
> `interpreted-prog-mode' that has generic commands for REPL
> communication, ...), where the tree-sitter configuration would be one of
> the attributes these modes would specify.

Sounds nice. Though what do you mean by “one of the attributes”?

>> How about it? Are there any good arguments for NOT merging
>> feature/tree-sitter at this point? :)
> 
> The current branch has major modes, should these be deleted before
> merging?

I think they can stay, we’ll work on them and improve them before branch is cut.

Yuan

Re: Suggesting that feature/tree-sitter be merged (was Re: Tree-sitter and major mode inheritance)

[Stefan Monnier]

> I’ve though of this too, other things are indent level, and
> documentation. I wrote ghelp[1] to get a uniform interface for getting
> documentation in different major modes (because I don’t have the heart to
> understand and modify help.el). A builtin, unified documentation system
> would be nice, like eldoc. But eldoc is for at-point short and quick
> signature/doc more than for full-fledged documentation like help.el.

help.el does it specifically for ELisp and doesn't try to present
a language-independent interface for other languages.  We could/should
change that, but I agree that it's not easy.  We may want to extract
some of that code to make it available for other uses, but I suspect it
will be a small part of it.

In that vicinity there is also `info-lookup-symbol` (aka `C-h S`) and
`xref.el`.


        Stefan

Re: Suggesting that feature/tree-sitter be merged (was Re: Tree-sitter and major mode inheritance)

[Philip Kaludercic]

Yuan Fu <casouri@gmail.com> writes:

>> On Nov 18, 2022, at 2:34 PM, Philip Kaludercic <philipk@posteo.net> wrote:
>> 
>> Jostein Kjønigsen <jostein@secure.kjonigsen.net> writes:
>> 
>>> Instead of waiting for "every" major-mode to be re-implemented into a
>>> tree-sitter derivative in the feature/tree-sitter branch before we
>>> merge... How about we just accept the current "core" tree-sitter
>>> implementation as good enough, and consider merging that to git master
>>> as is.
>> 
>> I think this sounds like a good idea -- as someone who has mostly just
>> been following the discussions.  The core bindings and major modes that
>> are based on these are separate issues, with a clear dependency linked
>> them.
>> 
>> As an aside: This might also be a good opportunity to clean up some of
>> the current major mode implementations and make them more consistent.
>> The issue with custom options to enable tree-sitter for every major mode
>> has revealed an inherent duplication of features.  There are other
>> inconsistencies, especially regarding bindings for equivalent operations
>> (e.g. in interpreted language with a repl, how to load function into the
>> current session: Lisp, Prolog, Python all differ in minor details).
>
> I’ve though of this too, other things are indent level, and
> documentation. I wrote ghelp[1] to get a uniform interface for getting
> documentation in different major modes (because I don’t have the heart
> to understand and modify help.el). A builtin, unified documentation
> system would be nice, like eldoc. But eldoc is for at-point short and
> quick signature/doc more than for full-fledged documentation like
> help.el.

I suppose you forgot the link: https://github.com/casouri/ghelp.
Perhaps it could be added to ELPA, and one day to the core?

>> I can imagine a more specialised `define-generic-mode' could be of use
>> here, along with more "abstract" major modes for various types of
>> programming languages (using `prog-mode' as a base to add
>> `compiled-prog-mode' that has generic commands for building program,
>> `interpreted-prog-mode' that has generic commands for REPL
>> communication, ...), where the tree-sitter configuration would be one of
>> the attributes these modes would specify.
>
> Sounds nice. Though what do you mean by “one of the attributes”?

If we think of this as a declarative block, something like

   (define-prog-mode foo
    :type 'compiled
    :syntax (tree-sitter-syntax 'foo)
    :doc-func #'foo-get-docs
    ...)

would have a list of attributes (what kind of a programming language,
how to indent, how to fetch documentation, ...), one of which would be
how syntax and fontification is calculated.

>>> How about it? Are there any good arguments for NOT merging
>>> feature/tree-sitter at this point? :)
>> 
>> The current branch has major modes, should these be deleted before
>> merging?
>
> I think they can stay, we’ll work on them and improve them before branch is cut.

Ok, sounds good.

> Yuan

Standardized access to a REPL (was: Suggesting that feature/tree-sitter be merged)

[Stefan Monnier]

[-- Attachment #1: Type: text/plain, Size: 933 bytes --]

>>> `compiled-prog-mode' that has generic commands for building program,
>>> `interpreted-prog-mode' that has generic commands for REPL
[...]
>    (define-prog-mode foo
>     :type 'compiled

<soapbox>
Let me point out that the idea that some languages are compiled and
others are interpreted is bogus.  This is a property of a language's
*implementation* and not of the language per se.
And of course, here we don't even really care about this facet of the
implementation: you're using those terms as a proxy for whether we use
a REPL or a batch-compiler.
</soapbox>

Many languages have both REPLs and batch compilers (like, say ELisp), so
a major mode needs to be able to offer access to both functionalities at
the same time.

Regarding the original suggestion to provide a uniform access to a REPL,
I started on this a long time ago, but never got it "finished" :-(
I attached what I still have of that effort.


        Stefan

[-- Attachment #2: prog-proc.el --]
[-- Type: application/emacs-lisp, Size: 13137 bytes --]

Re: Standardized access to a REPL

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>> `compiled-prog-mode' that has generic commands for building program,
>>>> `interpreted-prog-mode' that has generic commands for REPL
> [...]
>>    (define-prog-mode foo
>>     :type 'compiled
>
> <soapbox>
> Let me point out that the idea that some languages are compiled and
> others are interpreted is bogus.  This is a property of a language's
> *implementation* and not of the language per se.
> And of course, here we don't even really care about this facet of the
> implementation: you're using those terms as a proxy for whether we use
> a REPL or a batch-compiler.
> </soapbox>
>
> Many languages have both REPLs and batch compilers (like, say ELisp), so
> a major mode needs to be able to offer access to both functionalities at
> the same time.

Good point, that also indicates that that using `derive-major-mode' is
not the right approach, because (AFAIK) major modes span a tree, not a
DAG.

Perhaps it would be better to extend prog-mode and bind keys in the
default map that call methods with the current major mode.  These can
then either implement the methods or indicate that the feature
(evaluating in a REPL when using C) is not supported.

> Regarding the original suggestion to provide a uniform access to a REPL,
> I started on this a long time ago, but never got it "finished" :-(
> I attached what I still have of that effort.

Sounds interesting, I'll take a look at it.

Re: Standardized access to a REPL

[Stefan Monnier]

> Perhaps it would be better to extend prog-mode and bind keys in the
> default map that call methods with the current major mode.

Agreed.
[ And my muscle memory says that `C-c C-c` should be used for "send this
  buffer to its natural destination".  ]


        Stefan

Re: Standardized access to a REPL

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Perhaps it would be better to extend prog-mode and bind keys in the
>> default map that call methods with the current major mode.
>
> Agreed.
> [ And my muscle memory says that `C-c C-c` should be used for "send this
>   buffer to its natural destination".  ]

I don't know if some a thing is even practical, but I'd want to bind C-c
C-c to a generalised `do-what-i-mean' command that is context sensitive,
and even bound globally.

Re: Standardized access to a REPL

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: Yuan Fu <casouri@gmail.com>,  Jostein Kjønigsen
>  <jostein@secure.kjonigsen.net>,  emacs-devel <emacs-devel@gnu.org>,
>   Theodor Thornhill <theo@thornhill.no>,  Eli Zaretskii <eliz@gnu.org>,
>   jostein@kjonigsen.net
> Date: Sat, 19 Nov 2022 16:10:58 +0000
> 
> I don't know if some a thing is even practical, but I'd want to bind C-c
> C-c to a generalised `do-what-i-mean' command that is context sensitive,
> and even bound globally.

"C-c C-c" doesn't mean DWIM, it means "I'm done with whatever I was
doing, use the result as appropriate".

Re: Standardized access to a REPL

[Stefan Monnier]

> "C-c C-c" doesn't mean DWIM, it means "I'm done with whatever I was
> doing, use the result as appropriate".

I agree, and I suspect Philip agrees as well :-)


        Stefan

Re: Standardized access to a REPL

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> "C-c C-c" doesn't mean DWIM, it means "I'm done with whatever I was
>> doing, use the result as appropriate".
>
> I agree, and I suspect Philip agrees as well :-)

Yes, I didn't phrase that perfectly.  What I had in mind was Org's usage
of C-c C-c that is context sensitive.

Re: Suggesting that feature/tree-sitter be merged (was Re: Tree-sitter and major mode inheritance)

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: Yuan Fu <casouri@gmail.com>,  emacs-devel <emacs-devel@gnu.org>,
>   Theodor Thornhill <theo@thornhill.no>,  Eli Zaretskii <eliz@gnu.org>,
>   jostein@kjonigsen.net
> Date: Fri, 18 Nov 2022 22:34:13 +0000
> 
> Jostein Kjønigsen <jostein@secure.kjonigsen.net> writes:
> 
> > Instead of waiting for "every" major-mode to be re-implemented into a
> > tree-sitter derivative in the feature/tree-sitter branch before we
> > merge... How about we just accept the current "core" tree-sitter
> > implementation as good enough, and consider merging that to git master
> > as is.
> 
> I think this sounds like a good idea -- as someone who has mostly just
> been following the discussions.  The core bindings and major modes that
> are based on these are separate issues, with a clear dependency linked
> them.

From where I stand, it makes very little sense to release Emacs 29
with tree-sitter support that is limited to primitives and some
minimal Lisp glue on top of that.  Tree-sitter was added to Emacs to
allow major modes provide better support for editing program source
code, so having tree-sitter "support" in Emacs 29 that didn't include
at least several major modes using it would be disappointing at best.
It would mean we ourselves have no idea how to make major modes use
the feature.  Moreover, adding those few major modes on the branch
exposed several deficiencies in the original design and
implementation, and required changes to make the integration better;
releasing Emacs 29 with those issues unresolved (and unknown) would
require significant, sometimes incompatible changes in the future,
which is another reason why it would be wrong.

Basically, my firm belief is that adding to Emacs infrastructure
without user-level applications built on that infrastructure is wrong
and runs the risk of producing features that are not used or need deep
surgery before they become useful.  We should avoid doing that as much
as possible.

> As an aside: This might also be a good opportunity to clean up some of
> the current major mode implementations and make them more consistent.
> The issue with custom options to enable tree-sitter for every major mode
> has revealed an inherent duplication of features.  There are other
> inconsistencies, especially regarding bindings for equivalent operations
> (e.g. in interpreted language with a repl, how to load function into the
> current session: Lisp, Prolog, Python all differ in minor details).

Cleaning up major modes is a Good Thing that needs no opportunities.
We should do that whenever we know and agree how.

> The current branch has major modes, should these be deleted before
> merging?

Definitely not!  These modes are there because we want Emacs 29 to
have them, and we want users to use them and report back.

Re: Suggesting that feature/tree-sitter be merged (was Re: Tree-sitter and major mode inheritance)

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: Yuan Fu <casouri@gmail.com>,  emacs-devel <emacs-devel@gnu.org>,
>>   Theodor Thornhill <theo@thornhill.no>,  Eli Zaretskii <eliz@gnu.org>,
>>   jostein@kjonigsen.net
>> Date: Fri, 18 Nov 2022 22:34:13 +0000
>> 
>> Jostein Kjønigsen <jostein@secure.kjonigsen.net> writes:
>> 
>> > Instead of waiting for "every" major-mode to be re-implemented into a
>> > tree-sitter derivative in the feature/tree-sitter branch before we
>> > merge... How about we just accept the current "core" tree-sitter
>> > implementation as good enough, and consider merging that to git master
>> > as is.
>> 
>> I think this sounds like a good idea -- as someone who has mostly just
>> been following the discussions.  The core bindings and major modes that
>> are based on these are separate issues, with a clear dependency linked
>> them.
>
> From where I stand, it makes very little sense to release Emacs 29
> with tree-sitter support that is limited to primitives and some
> minimal Lisp glue on top of that.  Tree-sitter was added to Emacs to
> allow major modes provide better support for editing program source
> code, so having tree-sitter "support" in Emacs 29 that didn't include
> at least several major modes using it would be disappointing at best.
> It would mean we ourselves have no idea how to make major modes use
> the feature.  Moreover, adding those few major modes on the branch
> exposed several deficiencies in the original design and
> implementation, and required changes to make the integration better;
> releasing Emacs 29 with those issues unresolved (and unknown) would
> require significant, sometimes incompatible changes in the future,
> which is another reason why it would be wrong.
>
> Basically, my firm belief is that adding to Emacs infrastructure
> without user-level applications built on that infrastructure is wrong
> and runs the risk of producing features that are not used or need deep
> surgery before they become useful.  We should avoid doing that as much
> as possible.

My question is, do these user-level applications have to be distributed
along with Emacs, or could they be made to be "explicitly" opt-in by
installing them from ELPA.  In-core appears to usually bring a
commitment to maintain a library, and deprecating can take years.  If
Emacs 29 lays the technical foundations, the low-level API for
treesitter to work, we can have packages on ELPA experiment with the
higher-level abstractions.  Whatever is the most successful approach,
can be added to Emacs later on.

>> As an aside: This might also be a good opportunity to clean up some of
>> the current major mode implementations and make them more consistent.
>> The issue with custom options to enable tree-sitter for every major mode
>> has revealed an inherent duplication of features.  There are other
>> inconsistencies, especially regarding bindings for equivalent operations
>> (e.g. in interpreted language with a repl, how to load function into the
>> current session: Lisp, Prolog, Python all differ in minor details).
>
> Cleaning up major modes is a Good Thing that needs no opportunities.
> We should do that whenever we know and agree how.

Fair enough, but just as above I think these kinds of experiments are
better made outside of the core, in ELPA, to avoid committing to
mistakes.  If it works out, it can be added.

>> The current branch has major modes, should these be deleted before
>> merging?
>
> Definitely not!  These modes are there because we want Emacs 29 to
> have them, and we want users to use them and report back.

IIUC these modes aren't ripe yet, or at least aren't satisfying
replacements for the existing modes.  If tree-sitter were not to be
merged for that reason, that would delay the ability to use tree-sitter
on a widespread basis for at least another release.  My proposal above
would make it possible, and encourage users to report on their
experience, while allowing for the flexibility to make the right
decisions in the long term.

Re: Suggesting that feature/tree-sitter be merged (was Re: Tree-sitter and major mode inheritance)

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: jostein@secure.kjonigsen.net,  casouri@gmail.com,  emacs-devel@gnu.org,
>   theo@thornhill.no,  jostein@kjonigsen.net
> Date: Sat, 19 Nov 2022 10:46:41 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > From where I stand, it makes very little sense to release Emacs 29
> > with tree-sitter support that is limited to primitives and some
> > minimal Lisp glue on top of that.  Tree-sitter was added to Emacs to
> > allow major modes provide better support for editing program source
> > code, so having tree-sitter "support" in Emacs 29 that didn't include
> > at least several major modes using it would be disappointing at best.
> > It would mean we ourselves have no idea how to make major modes use
> > the feature.  Moreover, adding those few major modes on the branch
> > exposed several deficiencies in the original design and
> > implementation, and required changes to make the integration better;
> > releasing Emacs 29 with those issues unresolved (and unknown) would
> > require significant, sometimes incompatible changes in the future,
> > which is another reason why it would be wrong.
> >
> > Basically, my firm belief is that adding to Emacs infrastructure
> > without user-level applications built on that infrastructure is wrong
> > and runs the risk of producing features that are not used or need deep
> > surgery before they become useful.  We should avoid doing that as much
> > as possible.
> 
> My question is, do these user-level applications have to be distributed
> along with Emacs, or could they be made to be "explicitly" opt-in by
> installing them from ELPA.

It depends, the decision should be on a case by case basis, IMO.  For
functionality that is part of what we want Emacs to have, yes, it
should be distributed with Emacs.

> In-core appears to usually bring a commitment to maintain a library,
> and deprecating can take years.  If Emacs 29 lays the technical
> foundations, the low-level API for treesitter to work, we can have
> packages on ELPA experiment with the higher-level abstractions.
> Whatever is the most successful approach, can be added to Emacs
> later on.

Emacs cannot come without support for important programming languages,
that would make no sense.  If we want to move towards tree-sitter as
the basis for some aspects of such major modes, we must have this in
core.  Having such important parts of Emacs in ELPA when we don't have
a way of bundling ELPA packages with an Emacs release tarball means a
deficiency in released versions of Emacs, so we should not go that
way.

I don't see why what was done on the branch with introducing
tree-sitter capabilities into major modes should be considered
"experiments", let alone not "successful".  What parts of that
concretely do you think belong to these categories, and why?

More generally, why should we be afraid of including new stuff in
Emacs, and instead designate it "experimental" and put it on ELPA?  We
never did that in the past, and I don't see why would we want that now
or in the future.  ELPA is not a platform for "experiments" in Emacs
development; the master branch and the feature branches are that
platform.

> >> As an aside: This might also be a good opportunity to clean up some of
> >> the current major mode implementations and make them more consistent.
> >> The issue with custom options to enable tree-sitter for every major mode
> >> has revealed an inherent duplication of features.  There are other
> >> inconsistencies, especially regarding bindings for equivalent operations
> >> (e.g. in interpreted language with a repl, how to load function into the
> >> current session: Lisp, Prolog, Python all differ in minor details).
> >
> > Cleaning up major modes is a Good Thing that needs no opportunities.
> > We should do that whenever we know and agree how.
> 
> Fair enough, but just as above I think these kinds of experiments are
> better made outside of the core, in ELPA, to avoid committing to
> mistakes.  If it works out, it can be added.

No, we should do that on feature branches, not on ELPA.  Certainly so
for changes that require changes on the C level.

Again, ELPA is not a place where we should develop Emacs.

> >> The current branch has major modes, should these be deleted before
> >> merging?
> >
> > Definitely not!  These modes are there because we want Emacs 29 to
> > have them, and we want users to use them and report back.
> 
> IIUC these modes aren't ripe yet, or at least aren't satisfying
> replacements for the existing modes.

What concretely isn't ripe?

And please note that Emacs 29 won't be released tomorrow or the next
week.  We have the whole release cycle ahead of us to figure out what
is not yet ripe for a release and either fix that or (in extreme
cases) remove that from Emacs.  I see no reason to make these
decisions today.  We used the feature branch for initial shakeup and
stabilization, and we now think the tree-sitter support is mature
enough to let more people use it and provide their feedback.

> If tree-sitter were not to be merged for that reason, that would
> delay the ability to use tree-sitter on a widespread basis for at
> least another release.  My proposal above would make it possible,
> and encourage users to report on their experience, while allowing
> for the flexibility to make the right decisions in the long term.

If that was your reasoning, then I think you are three steps ahead of
where we are, and you are trying to find solutions for problems that
don't necessarily exist.  We should see what concrete problems are
left after the merge, and take it from there.  We have ample time for
figuring that out and fixing whatever will need fixing.

Re: Suggesting that feature/tree-sitter be merged (was Re: Tree-sitter and major mode inheritance)

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: jostein@secure.kjonigsen.net,  casouri@gmail.com,  emacs-devel@gnu.org,
>>   theo@thornhill.no,  jostein@kjonigsen.net
>> Date: Sat, 19 Nov 2022 10:46:41 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > From where I stand, it makes very little sense to release Emacs 29
>> > with tree-sitter support that is limited to primitives and some
>> > minimal Lisp glue on top of that.  Tree-sitter was added to Emacs to
>> > allow major modes provide better support for editing program source
>> > code, so having tree-sitter "support" in Emacs 29 that didn't include
>> > at least several major modes using it would be disappointing at best.
>> > It would mean we ourselves have no idea how to make major modes use
>> > the feature.  Moreover, adding those few major modes on the branch
>> > exposed several deficiencies in the original design and
>> > implementation, and required changes to make the integration better;
>> > releasing Emacs 29 with those issues unresolved (and unknown) would
>> > require significant, sometimes incompatible changes in the future,
>> > which is another reason why it would be wrong.
>> >
>> > Basically, my firm belief is that adding to Emacs infrastructure
>> > without user-level applications built on that infrastructure is wrong
>> > and runs the risk of producing features that are not used or need deep
>> > surgery before they become useful.  We should avoid doing that as much
>> > as possible.
>> 
>> My question is, do these user-level applications have to be distributed
>> along with Emacs, or could they be made to be "explicitly" opt-in by
>> installing them from ELPA.
>
> It depends, the decision should be on a case by case basis, IMO.  For
> functionality that is part of what we want Emacs to have, yes, it
> should be distributed with Emacs.
>
>> In-core appears to usually bring a commitment to maintain a library,
>> and deprecating can take years.  If Emacs 29 lays the technical
>> foundations, the low-level API for treesitter to work, we can have
>> packages on ELPA experiment with the higher-level abstractions.
>> Whatever is the most successful approach, can be added to Emacs
>> later on.
>
> Emacs cannot come without support for important programming languages,
> that would make no sense.  If we want to move towards tree-sitter as
> the basis for some aspects of such major modes, we must have this in
> core.  Having such important parts of Emacs in ELPA when we don't have
> a way of bundling ELPA packages with an Emacs release tarball means a
> deficiency in released versions of Emacs, so we should not go that
> way.
>
> I don't see why what was done on the branch with introducing
> tree-sitter capabilities into major modes should be considered
> "experiments", let alone not "successful".  What parts of that
> concretely do you think belong to these categories, and why?
>
> More generally, why should we be afraid of including new stuff in
> Emacs, and instead designate it "experimental" and put it on ELPA?  We
> never did that in the past, and I don't see why would we want that now
> or in the future.  ELPA is not a platform for "experiments" in Emacs
> development; the master branch and the feature branches are that
> platform.
>
>> >> As an aside: This might also be a good opportunity to clean up some of
>> >> the current major mode implementations and make them more consistent.
>> >> The issue with custom options to enable tree-sitter for every major mode
>> >> has revealed an inherent duplication of features.  There are other
>> >> inconsistencies, especially regarding bindings for equivalent operations
>> >> (e.g. in interpreted language with a repl, how to load function into the
>> >> current session: Lisp, Prolog, Python all differ in minor details).
>> >
>> > Cleaning up major modes is a Good Thing that needs no opportunities.
>> > We should do that whenever we know and agree how.
>> 
>> Fair enough, but just as above I think these kinds of experiments are
>> better made outside of the core, in ELPA, to avoid committing to
>> mistakes.  If it works out, it can be added.
>
> No, we should do that on feature branches, not on ELPA.  Certainly so
> for changes that require changes on the C level.
>
> Again, ELPA is not a place where we should develop Emacs.
>
>> >> The current branch has major modes, should these be deleted before
>> >> merging?
>> >
>> > Definitely not!  These modes are there because we want Emacs 29 to
>> > have them, and we want users to use them and report back.
>> 
>> IIUC these modes aren't ripe yet, or at least aren't satisfying
>> replacements for the existing modes.
>
> What concretely isn't ripe?

Jostein said:

  Me and Theodor faced these same issues with "our" C# and TypeScript
  major-modes, and the only "clean" way we agreed we could make this
  work was to create wholly new implementations. I can come up with many
  good, objective reasons for this, but I think Theodor has already
  represented this view fairly well.

> And please note that Emacs 29 won't be released tomorrow or the next
> week.  We have the whole release cycle ahead of us to figure out what
> is not yet ripe for a release and either fix that or (in extreme
> cases) remove that from Emacs.  I see no reason to make these
> decisions today.  We used the feature branch for initial shakeup and
> stabilization, and we now think the tree-sitter support is mature
> enough to let more people use it and provide their feedback.

Naturally, I didn't understand this to be a discussion on an immediate
decision.

>> If tree-sitter were not to be merged for that reason, that would
>> delay the ability to use tree-sitter on a widespread basis for at
>> least another release.  My proposal above would make it possible,
>> and encourage users to report on their experience, while allowing
>> for the flexibility to make the right decisions in the long term.
>
> If that was your reasoning, then I think you are three steps ahead of
> where we are, and you are trying to find solutions for problems that
> don't necessarily exist.  We should see what concrete problems are
> left after the merge, and take it from there.  We have ample time for
> figuring that out and fixing whatever will need fixing.

You are right, I'll have to test the branch more seriously.

Re: Suggesting that feature/tree-sitter be merged (was Re: Tree-sitter and major mode inheritance)

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: jostein@secure.kjonigsen.net,  casouri@gmail.com,  emacs-devel@gnu.org,
>   theo@thornhill.no,  jostein@kjonigsen.net
> Date: Sat, 19 Nov 2022 12:15:10 +0000
> 
> >> IIUC these modes aren't ripe yet, or at least aren't satisfying
> >> replacements for the existing modes.
> >
> > What concretely isn't ripe?
> 
> Jostein said:
> 
>   Me and Theodor faced these same issues with "our" C# and TypeScript
>   major-modes, and the only "clean" way we agreed we could make this
>   work was to create wholly new implementations. I can come up with many
>   good, objective reasons for this, but I think Theodor has already
>   represented this view fairly well.

That's Jostein's opinion.  We've heard it before, and AFAIU the branch
addresses these problems in ways that at least to me look adequate and
consistent with how I'd like to see this feature in Emacs 29.  In
particular, the TypeScript mode in the branch is indeed a separate
mode.

Any other problems that will be flagged before the release we will
handle when they pop up.
> > If that was your reasoning, then I think you are three steps ahead of
> > where we are, and you are trying to find solutions for problems that
> > don't necessarily exist.  We should see what concrete problems are
> > left after the merge, and take it from there.  We have ample time for
> > figuring that out and fixing whatever will need fixing.
> 
> You are right, I'll have to test the branch more seriously.

Thanks.

Re: Suggesting that feature/tree-sitter be merged (was Re: Tree-sitter and major mode inheritance)

[Dmitry Gutov]

On 19.11.2022 12:46, Philip Kaludercic wrote:
> My question is, do these user-level applications have to be distributed
> along with Emacs, or could they be made to be "explicitly" opt-in by
> installing them from ELPA.  In-core appears to usually bring a
> commitment to maintain a library, and deprecating can take years.  If
> Emacs 29 lays the technical foundations, the low-level API for
> treesitter to work, we can have packages on ELPA experiment with the
> higher-level abstractions.  Whatever is the most successful approach,
> can be added to Emacs later on.

Wasn't https://github.com/emacs-tree-sitter/elisp-tree-sitter that way 
for us to experiment with approaches before bring TS to the core?

Re: Suggesting that feature/tree-sitter be merged (was Re: Tree-sitter and major mode inheritance)

[Yuan Fu]

> On Nov 18, 2022, at 1:54 PM, Jostein Kjønigsen <jostein@secure.kjonigsen.net> wrote:
> 
> Hey everyone.
> 
> I know this has been said before, by people which by far has been contributing much more than me... But I still don't think it's a good idea to replace the implementation in existing major-modes with tree-sitter implementations, nor selectively activate tree-sitter in major-modes prone to inhetitence and derivation.
> 
> Me and Theodor faced these same issues with "our" C# and TypeScript major-modes, and the only "clean" way we agreed we could make this work was to create wholly new implementations. I can come up with many good, objective reasons for this, but I think Theodor has already represented this view fairly well.
> 
> While for the sake of brevity, I'll not diving deeply into this particular thing, I will say this: A new tree-sitter based major-mode free of compatibility concerns allowed us to create entirely new major-modes fixing most of our existing bugs, faster than we before would be able to fix a single bug. My personal view is that mixing existing major-modes with tree-sitter represents absolutely the worst of all worlds. It maintains all existing complexities, provides us with very few benefits, and at the same time adds complexities we didn't use to have. To me, that's a net negative.
> 
> Somewhat surprising to me, I see this is a somewhat controversial point of view and not as clear cut a matter as I would have expected it to be. I realize and respect that final decisions in these matter might take some time to mature. Time which given our current approach detracts from the momentum tree-sitter has been having.

Hey Jostein,

Thank you very much for your thoughtful input! I originally thought of tree-sitter as merely a mean/tool that major modes could use to enhance their features. But in reality it seems that tree-sitter replaces a large enough chunk of a major-mode’s responsibility, which has caused all these compatibility problems we’ve encountered. Examples include undoing cc-mode’s setup when turning on tree-sitter, invalidating many existing major-mode variables, and the major mode inheritance problem I presented in the previous message. 

These difficulties has nudged us to decouple the tree-sitter and non-tree-sitter version further and further, from trying to make tree-sitter a feature that enables/disabled in a major mode by a command, to a major mode configuration that causes the major mode to setup differently, to my suggestion of using separate major mode but share a single virtual parent mode.

I think using separate major mode but share a single virtual parent mode gets the benefit of both worlds:
1. Tree-sitter and non-tree-sitter are separated in different modes, meaning it gets all the benefit Jostein and Theo described.
2. Backward-compatible. Existing configuration to the non-tree-sitter version works as they do before. (Here I’m mainly thinking about hooks: Hooks added to c-mode-hook still runs in c-native-mode.)
3. Solves the inheritance problem I described.
4. Minimal changes to the existing modes.

> At this point poor Yuan Fu here has spent quite a bit of time and effort getting a core tree-sitter interface into Emacs. I would really hate to see all that effort be for nothing, because we end up conflating a creating a core tree-sitter feature with how this feature should best be employed in subsequent major-modes.
> 
> So in the name of pragmatism, I propose a compromise of sorts.
> 
> Instead of waiting for "every" major-mode to be re-implemented into a tree-sitter derivative in the feature/tree-sitter branch before we merge... How about we just accept the current "core" tree-sitter implementation as good enough, and consider merging that to git master as is.
> 
> This will allow us to land one important mile-stone, while giving us the head-room to further discuss how we should best implement/reimplement/"upgrade" existing major-modes to take advantage of tree-sitter.
> 
> It will also allow third-party packages to make use of tree-sitter in Emacs core instead coming up with its own implementation, beginning a defacto standardization of this new API (which I may note already has a competing implementation in MELPA).
> 
> That is to say: We should land the core library, with holding it hostage to all its possible consumers being implemented.
> 
> How about it? Are there any good arguments for NOT merging feature/tree-sitter at this point? :)

The good news is, feature/tree-sitter is merging in a few days! And we’ll make further improvements on the master. So rest assured! :-) I think we can at least get C, Python, Javascript, Typescript, Bash, JONS and CSS to go with the up coming release, largely thanks to Theo’s (and tree-sitter’s) productivity. I personally don’t know enough of C++ and Java to polish them, but they have a good chance too.

Taking a step back, I think developing major mode support with the tree-sitter API is a good idea. We found countless shortcomings of the API when developing major modes with it.

Yuan

Re: Suggesting that feature/tree-sitter be merged (was Re: Tree-sitter and major mode inheritance)

[Theodor Thornhill]

>The good news is, feature/tree-sitter is merging in a few days! And we’ll make further improvements on the master. So rest assured! :-) I think we can at least get C, Python, Javascript, Typescript, Bash, JONS and CSS to go with the up coming release, largely thanks to Theo’s (and tree-sitter’s) productivity. I personally don’t know enough of C++ and Java to polish them, but they have a good chance too.

Java should be good to - modulo some tweaks. I'm using it daily at work already :) 

>
>Taking a step back, I think developing major mode support with the tree-sitter API is a good idea. We found countless shortcomings of the API when developing major modes with it.
>
>Yuan

I don't think I have much more to add to this discussion than what I've already have been yapping on about for weeks ;)

Looking forward to the merge!

Theo

Re: Suggesting that feature/tree-sitter be merged (was Re: Tree-sitter and major mode inheritance)

[Eli Zaretskii]

> Date: Sat, 19 Nov 2022 06:21:12 +0100
> From: Theodor Thornhill <theo@thornhill.no>
> CC: emacs-devel <emacs-devel@gnu.org>, Eli Zaretskii <eliz@gnu.org>
> 
> >The good news is, feature/tree-sitter is merging in a few days! And we’ll make further improvements on the master. So rest assured! :-) I think we can at least get C, Python, Javascript, Typescript, Bash, JONS and CSS to go with the up coming release, largely thanks to Theo’s (and tree-sitter’s) productivity. I personally don’t know enough of C++ and Java to polish them, but they have a good chance too.
> 
> Java should be good to - modulo some tweaks. I'm using it daily at work already :) 

Btw, should we add C# to c-ts-mode.el?  Or did we already discuss that and
decided against?  I don't remember, sorry.

Re: Suggesting that feature/tree-sitter be merged (was Re: Tree-sitter and major mode inheritance)

[Theodor Thornhill]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Sat, 19 Nov 2022 06:21:12 +0100
>> From: Theodor Thornhill <theo@thornhill.no>
>> CC: emacs-devel <emacs-devel@gnu.org>, Eli Zaretskii <eliz@gnu.org>
>> 
>> >The good news is, feature/tree-sitter is merging in a few days! And
>> >we’ll make further improvements on the master. So rest assured! :-)
>> >I think we can at least get C, Python, Javascript, Typescript, Bash,
>> >JONS and CSS to go with the up coming release, largely thanks to
>> >Theo’s (and tree-sitter’s) productivity. I personally don’t know
>> >enough of C++ and Java to polish them, but they have a good chance
>> >too.
>> 
>> Java should be good to - modulo some tweaks. I'm using it daily at work already :) 
>
> Btw, should we add C# to c-ts-mode.el?  Or did we already discuss that and
> decided against?  I don't remember, sorry.

I don't think we decided against, and we didn't really discuss it. IIRC
your "challenge" was for the cc modes already included in emacs, and C#
is not that.  But seeing how there's a functioning cc-based c#-mode, I
could tweak that to include both.  The Cc mode variant is very stable
and have been for some time already.  There's no need to maintain the
one in ELPA, and as I'm the author of it I think we can merge both?  So
we can have in-tree support for c# whether or not you have tree-sitter
enabled?  I can whip up a patch for that if you want, or we could just
add the tree-sitter variant.  In any case, I think c#-mode should
probably not be inside of c-ts-mode, considering that it's not a
superset of C, like C++, but its own entity.

 What do you think?

Re: Suggesting that feature/tree-sitter be merged (was Re: Tree-sitter and major mode inheritance)

[Eli Zaretskii]

> From: Theodor Thornhill <theo@thornhill.no>
> Cc: casouri@gmail.com, jostein@kjonigsen.net, emacs-devel@gnu.org
> Date: Sat, 19 Nov 2022 19:46:07 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Btw, should we add C# to c-ts-mode.el?  Or did we already discuss that and
> > decided against?  I don't remember, sorry.
> 
> I don't think we decided against, and we didn't really discuss it. IIRC
> your "challenge" was for the cc modes already included in emacs, and C#
> is not that.  But seeing how there's a functioning cc-based c#-mode, I
> could tweak that to include both.  The Cc mode variant is very stable
> and have been for some time already.  There's no need to maintain the
> one in ELPA, and as I'm the author of it I think we can merge both?  So
> we can have in-tree support for c# whether or not you have tree-sitter
> enabled?  I can whip up a patch for that if you want, or we could just
> add the tree-sitter variant.  In any case, I think c#-mode should
> probably not be inside of c-ts-mode, considering that it's not a
> superset of C, like C++, but its own entity.
> 
>  What do you think?

It's fine with me to have C# support both with and without tree-sitter, if it's
indeed easy.

Thanks.

Re: Suggesting that feature/tree-sitter be merged (was Re: Tree-sitter and major mode inheritance)

[Theodor Thornhill]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Theodor Thornhill <theo@thornhill.no>
>> Cc: casouri@gmail.com, jostein@kjonigsen.net, emacs-devel@gnu.org
>> Date: Sat, 19 Nov 2022 19:46:07 +0100
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > Btw, should we add C# to c-ts-mode.el?  Or did we already discuss that and
>> > decided against?  I don't remember, sorry.
>> 
>> I don't think we decided against, and we didn't really discuss it. IIRC
>> your "challenge" was for the cc modes already included in emacs, and C#
>> is not that.  But seeing how there's a functioning cc-based c#-mode, I
>> could tweak that to include both.  The Cc mode variant is very stable
>> and have been for some time already.  There's no need to maintain the
>> one in ELPA, and as I'm the author of it I think we can merge both?  So
>> we can have in-tree support for c# whether or not you have tree-sitter
>> enabled?  I can whip up a patch for that if you want, or we could just
>> add the tree-sitter variant.  In any case, I think c#-mode should
>> probably not be inside of c-ts-mode, considering that it's not a
>> superset of C, like C++, but its own entity.
>> 
>>  What do you think?
>
> It's fine with me to have C# support both with and without tree-sitter, if it's
> indeed easy.

Gotcha.  The few commits that has happened after I rewrote the cc mode
variant should be compliant assignment-wise, but I'll double-check.
Most are very, very small.

@jostein, could we just iterate on it the next couple of days? I can do
the coding, but I'm not bathing in C# these days, so it would be nice
with some real life usage too :-)

Theo

Re: Suggesting that feature/tree-sitter be merged (was Re: Tree-sitter and major mode inheritance)

[Stefan Kangas]

Yuan Fu <casouri@gmail.com> writes:

> The good news is, feature/tree-sitter is merging in a few days! And
> we’ll make further improvements on the master. So rest assured! :-) I
> think we can at least get C, Python, Javascript, Typescript, Bash,
> JONS and CSS to go with the up coming release, largely thanks to
> Theo’s (and tree-sitter’s) productivity. I personally don’t know
> enough of C++ and Java to polish them, but they have a good chance
> too.

That's very exciting news!  Thanks Yuan, Theodor and all others who
contributed.

There seems to be a whole lot of figuring out left to do with both the
API and how the major modes should be implemented.  Would it make sense
to ship the `tree-sitter' feature (and related modes) as explicitly
"experimental", to allow us to make changes in Emacs 30 without overly
worrying about backwards-compatibility?

Re: Suggesting that feature/tree-sitter be merged (was Re: Tree-sitter and major mode inheritance)

[Eli Zaretskii]

> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Fri, 18 Nov 2022 23:36:30 -0800
> Cc: emacs-devel <emacs-devel@gnu.org>, Theodor Thornhill <theo@thornhill.no>, Eli Zaretskii <eliz@gnu.org>
> 
> Yuan Fu <casouri@gmail.com> writes:
> 
> > The good news is, feature/tree-sitter is merging in a few days! And
> > we’ll make further improvements on the master. So rest assured! :-) I
> > think we can at least get C, Python, Javascript, Typescript, Bash,
> > JONS and CSS to go with the up coming release, largely thanks to
> > Theo’s (and tree-sitter’s) productivity. I personally don’t know
> > enough of C++ and Java to polish them, but they have a good chance
> > too.
> 
> That's very exciting news!  Thanks Yuan, Theodor and all others who
> contributed.
> 
> There seems to be a whole lot of figuring out left to do with both the
> API and how the major modes should be implemented.  Would it make sense
> to ship the `tree-sitter' feature (and related modes) as explicitly
> "experimental", to allow us to make changes in Emacs 30 without overly
> worrying about backwards-compatibility?

The intent is for Emacs 29 to include several modes based on
tree-sitter, and several others to have optional features based on
tree-sitter.  Based on the state of the soon-to-be-merged branch, I
see no reason to declare its support as experimental.

As for backwards-compatibility, you will have to be more specific.
Which features or APIs you see on the branch are in your opinion
likely to change significantly after the release of Emacs 29?  We need
to consider these on a case by case basis.  Personally, I think that
we had this figured out in a way that won't create incompatibilities
and yet allow changes, but maybe I'm missing something.

Re: Suggesting that feature/tree-sitter be merged (was Re: Tree-sitter and major mode inheritance)

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

> The intent is for Emacs 29 to include several modes based on
> tree-sitter, and several others to have optional features based on
> tree-sitter.  Based on the state of the soon-to-be-merged branch, I
> see no reason to declare its support as experimental.

My comment was more general, as IIUC we are still seeing quite a bit of
movement even in the low-level fundamentals on that branch, as recently
as the last week or two.  But if you and others are happy to declare our
tree-sitter support stable, so much the better.

Re: Suggesting that feature/tree-sitter be merged (was Re: Tree-sitter and major mode inheritance)

[Theodor Thornhill]

Stefan Kangas <stefankangas@gmail.com> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> The intent is for Emacs 29 to include several modes based on
>> tree-sitter, and several others to have optional features based on
>> tree-sitter.  Based on the state of the soon-to-be-merged branch, I
>> see no reason to declare its support as experimental.
>
> My comment was more general, as IIUC we are still seeing quite a bit of
> movement even in the low-level fundamentals on that branch, as recently
> as the last week or two.  But if you and others are happy to declare our
> tree-sitter support stable, so much the better.

Only one of the modes is auto-enabled, and that one is because it is
missing support altogether natively in Emacs.  The others are
second-class to their older counterparts, and likely will be for years
to come.  My personal suggestion is to just keep them separate, as
interested parties will find and enable these anyways.  If at some point
tree-sitter is so ubiquitous that it being second class doesn't make
sense anymore we can make it the default.  That's why I don't think we
need to make it experimental: people that will be early adopters will
likely compile emacs from master and get improvements incrementally,
though they should be usable enough for others to use, should a package
manager provide the integration within the stable Emacs 29 package.

I agree with Eli in that providing only the api and "glue-code" would
guarantee that people will just create their own, external variants
that'll be hard to remove should we provide our own down the line.

Re: Suggesting that feature/tree-sitter be merged (was Re: Tree-sitter and major mode inheritance)

[Eli Zaretskii]

> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Sat, 19 Nov 2022 03:25:11 -0800
> Cc: casouri@gmail.com, jostein@kjonigsen.net, emacs-devel@gnu.org, 
> 	theo@thornhill.no
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > The intent is for Emacs 29 to include several modes based on
> > tree-sitter, and several others to have optional features based on
> > tree-sitter.  Based on the state of the soon-to-be-merged branch, I
> > see no reason to declare its support as experimental.
> 
> My comment was more general, as IIUC we are still seeing quite a bit of
> movement even in the low-level fundamentals on that branch, as recently
> as the last week or two.

We will merge the tree-sitter branch to master, but Emacs 29 release
is still several moons ahead, and the master branch is used for
developing Emacs, including low-level stuff, all the time.

IOW, the fact that the tree-sitter support is still in flux is normal
and should not preclude us from landing it on master.  And if we want
it to be part of Emacs 29, we cannot wait any longer.

> But if you and others are happy to declare our tree-sitter support
> stable, so much the better.

It is stable enough to have it on master, yes.  Not stable enough to
release Emacs with it, but then Emacs 29 as a whole is nowhere near
that status, either.

Re: Suggesting that feature/tree-sitter be merged (was Re: Tree-sitter and major mode inheritance)

[Eli Zaretskii]

> Date: Fri, 18 Nov 2022 22:54:49 +0100
> From: Jostein Kjønigsen <jostein@secure.kjonigsen.net>
> 
> I know this has been said before, by people which by far has been contributing much more than me... But I
> still don't think it's a good idea to replace the implementation in existing major-modes with tree-sitter
> implementations, nor selectively activate tree-sitter in major-modes prone to inhetitence and derivation.
> 
> Me and Theodor faced these same issues with "our" C# and TypeScript major-modes, and the only "clean"
> way we agreed we could make this work was to create wholly new implementations. I can come up with
> many good, objective reasons for this, but I think Theodor has already represented this view fairly well.
> 
> While for the sake of brevity, I'll not diving deeply into this particular thing, I will say this: A new tree-sitter
> based major-mode free of compatibility concerns allowed us to create entirely new major-modes fixing most
> of our existing bugs, faster than we before would be able to fix a single bug. My personal view is that mixing
> existing major-modes with tree-sitter represents absolutely the worst of all worlds. It maintains all existing
> complexities, provides us with very few benefits, and at the same time adds complexities we didn't use to
> have. To me, that's a net negative.
> 
> Somewhat surprising to me, I see this is a somewhat controversial point of view and not as clear cut a
> matter as I would have expected it to be. I realize and respect that final decisions in these matter might take
> some time to mature. Time which given our current approach detracts from the momentum tree-sitter has
> been having.

You are looking at this from the POV of developing these features.
But we have another vantage point to consider: that of our users.
From their POV, we cannot replace existing modes with completely new
and separate implementations, we must provide a migration path which
will allow users to decide when and whether they want to switch to the
tree-sitter based implementation.  This will also allow us to improve
the tree-sitter support of the modes by collecting user feedback
sooner rather than later.

So we decided to have a hybrid approach: in some modes to provide
separate implementations, and in others to provide optional features
that users can selectively switch on.

> Instead of waiting for "every" major-mode to be re-implemented into a tree-sitter derivative in the
> feature/tree-sitter branch before we merge... How about we just accept the current "core" tree-sitter
> implementation as good enough, and consider merging that to git master as is.

Here you are preaching to the choir, since the decision to merge soon
was already made.  And it cannot be otherwise, since the time of
cutting the emacs-29 release branch is closing up, and we said 2
months ago that we intend to release Emacs 29 with tree-sitter
support.

Re: Tree-sitter and major mode inheritance

[Yuan Fu]

> On Nov 16, 2022, at 12:45 PM, Yuan Fu <casouri@gmail.com> wrote:
> 
> So I’m trying to merge css-ts-mode with css-mode. Scss-mode inherits css-mode, but if user enables tree-sitter for css-mode, scss-mode will inherit all that tree-sitter setup, and lose all the native css setup. Then if a user doesn’t want to enable tree-sitter in scss-mode, too bad: scss-mode breaks.
> 
> Essentially scss-mode needs to be able to control which parts of css-mode’s setup it wants to inherit—native setup or tree-sitter setup—regardless of whether css-mode enables tree-sitter or not.
> 
> I wonder if we can do something like this:
> 
>             css-mode
>                 |
>       +---------+-----+-----------+
>       |               |           |
> css-native-mode   css-ts-mode  scss-mode
>                                   |
>                              +----+------------+
>                              |                 |
>                        scss-native-mode   scss-ts-mode
> 
> css-mode: a virtual mode, only sets up basic things that both native and tree-sitter mode needs, like comment-start.
> css-native-mode: native setup
> css-ts-mode: tree-sitter setup
> 
> scss-mode: a virtual mode, inherits css-mode
> scss-native-mode: native setup
> scss-ts-mode: tree-sitter setup
> 
> And user could use major-mode-remap-alist to choose which mode they want:
> 
> (css-mode . css-ts-mode) for enabling tree-sitter
> (css-mode . css-native-mode) for not enabling tree-sitter
> 
> This could also used for other modes, like c-mode: c-mode, c-native-mode (cc-mode), c-ts-mode.
> 
> Yuan

Anyway, does anyone think this is a good/bad idea? Should I go implement this on css, js, c, etc? It can also be the other way around: instead of having c-mode being the virtual mode, we can leave c-mode as-is, and have a c-base-mode inherited by c-mode and c-ts-mode. And similarly rss-base-mode, rss-mode, and rss-ts-mode.

Yuan

Re: Tree-sitter and major mode inheritance

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Sat, 19 Nov 2022 01:41:47 -0800
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,
>  Eli Zaretskii <eliz@gnu.org>,
>  Theodor Thornhill <theo@thornhill.no>
> 
> Anyway, does anyone think this is a good/bad idea? Should I go implement this on css, js, c, etc? It can also be the other way around: instead of having c-mode being the virtual mode, we can leave c-mode as-is, and have a c-base-mode inherited by c-mode and c-ts-mode. And similarly rss-base-mode, rss-mode, and rss-ts-mode.

I'd prefer leaving the original modes as-is.  That should cause less
compatibility problems, I think.

Stefan, any thoughts?

Re: Tree-sitter and major mode inheritance

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> I'd prefer leaving the original modes as-is.  That should cause less
> compatibility problems, I think.

+1

Re: Tree-sitter and major mode inheritance

[Stefan Monnier]

>> Anyway, does anyone think this is a good/bad idea? Should I go implement
>> this on css, js, c, etc? It can also be the other way around: instead of
>> having c-mode being the virtual mode, we can leave c-mode as-is, and have
>> a c-base-mode inherited by c-mode and c-ts-mode. And similarly
>> rss-base-mode, rss-mode, and rss-ts-mode.
>
> I'd prefer leaving the original modes as-is.  That should cause less
> compatibility problems, I think.
>
> Stefan, any thoughts?

To the extent that Emacs-29's new `major-mode-remap-alist` can be used
to select which mode to use, we can indeed leave the original modes
as-is.

Another argument in favor is that it's a bit tricky to make `<foo>-mode`
both the parent mode and the standard entry point: we do that for
`tex-mode` but the implementation is ugly.
[ If it weren't for this implementation problem, it would be my
  favorite choice.  So maybe the better option is to add specific support
  for that in `define-derived-mode`, where we could implement it
  cleanly and thus also fix the ugly gymnastics of `tex-mode`.  ]

OTOH it's a bit jarring to have the generic term `<foo>-mode` refer to
a specific implementation.

For that reason, my preference is for:

- `<foo>-<abstract/parent/base/common>-mode` as the shared parent.
- `<foo>-mode` as a dispatch function that calls the appropriate specific
  major mode which could be `<foo>-ts-mode`, or `cc-<foo>-mode`, or
  `<foo>-with-JSX-mode`, or ...


        Stefan

Re: Tree-sitter and major mode inheritance

[Yuan Fu]

> On Nov 19, 2022, at 7:19 AM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> 
>>> Anyway, does anyone think this is a good/bad idea? Should I go implement
>>> this on css, js, c, etc? It can also be the other way around: instead of
>>> having c-mode being the virtual mode, we can leave c-mode as-is, and have
>>> a c-base-mode inherited by c-mode and c-ts-mode. And similarly
>>> rss-base-mode, rss-mode, and rss-ts-mode.
>> 
>> I'd prefer leaving the original modes as-is.  That should cause less
>> compatibility problems, I think.
>> 
>> Stefan, any thoughts?
> 
> To the extent that Emacs-29's new `major-mode-remap-alist` can be used
> to select which mode to use, we can indeed leave the original modes
> as-is.
> 
> Another argument in favor is that it's a bit tricky to make `<foo>-mode`
> both the parent mode and the standard entry point: we do that for
> `tex-mode` but the implementation is ugly.
> [ If it weren't for this implementation problem, it would be my
>  favorite choice.  So maybe the better option is to add specific support
>  for that in `define-derived-mode`, where we could implement it
>  cleanly and thus also fix the ugly gymnastics of `tex-mode`.  ]
> 
> OTOH it's a bit jarring to have the generic term `<foo>-mode` refer to
> a specific implementation.
> 
> For that reason, my preference is for:
> 
> - `<foo>-<abstract/parent/base/common>-mode` as the shared parent.
> - `<foo>-mode` as a dispatch function that calls the appropriate specific
>  major mode which could be `<foo>-ts-mode`, or `cc-<foo>-mode`, or
>  `<foo>-with-JSX-mode`, or …

If we are already renaming existing modes (cc-<foo>-mode), why don’t we use the generic name <foo>-mode for the virtual parent mode? It would be nicer if the generic mode (<foo>-mode) is an actual mode, with mode hooks, keycaps, etc, rather than simply a dispatch function.

Plus, if we make <foo>-mode a command rather, all existing configuration of <foo>-mode breaks: there is no such major mode anymore, users need to use either <foo>-base-mode or one of <foo>-cc/ts-mode.

Yuan

Re: Tree-sitter and major mode inheritance

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Sat, 19 Nov 2022 09:17:11 -0800
> Cc: Eli Zaretskii <eliz@gnu.org>,
>  emacs-devel@gnu.org,
>  theo@thornhill.no
> 
> If we are already renaming existing modes (cc-<foo>-mode), why don’t we use the generic name <foo>-mode for the virtual parent mode? It would be nicer if the generic mode (<foo>-mode) is an actual mode, with mode hooks, keycaps, etc, rather than simply a dispatch function.

I already said that I prefer not to rename existing modes.  Such
renaming will break too many init files and other Lisp programs.

Re: Tree-sitter and major mode inheritance

[Yuan Fu]

> On Nov 19, 2022, at 9:52 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Yuan Fu <casouri@gmail.com>
>> Date: Sat, 19 Nov 2022 09:17:11 -0800
>> Cc: Eli Zaretskii <eliz@gnu.org>,
>> emacs-devel@gnu.org,
>> theo@thornhill.no
>> 
>> If we are already renaming existing modes (cc-<foo>-mode), why don’t we use the generic name <foo>-mode for the virtual parent mode? It would be nicer if the generic mode (<foo>-mode) is an actual mode, with mode hooks, keycaps, etc, rather than simply a dispatch function.
> 
> I already said that I prefer not to rename existing modes.  Such
> renaming will break too many init files and other Lisp programs.

I guess we can at least try it for a bit? Because hook, keymaps, etc, should just work, that’s the point of major mode inheritance, after all. C-native-mode will run all the setup for c-mode, plus setup for c-native-mode.

Yuan

Re: Tree-sitter and major mode inheritance

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Sat, 19 Nov 2022 13:45:41 -0800
> Cc: monnier@iro.umontreal.ca,
>  emacs-devel@gnu.org,
>  theo@thornhill.no
> 
> 
> 
> > On Nov 19, 2022, at 9:52 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> > 
> >> From: Yuan Fu <casouri@gmail.com>
> >> Date: Sat, 19 Nov 2022 09:17:11 -0800
> >> Cc: Eli Zaretskii <eliz@gnu.org>,
> >> emacs-devel@gnu.org,
> >> theo@thornhill.no
> >> 
> >> If we are already renaming existing modes (cc-<foo>-mode), why don’t we use the generic name <foo>-mode for the virtual parent mode? It would be nicer if the generic mode (<foo>-mode) is an actual mode, with mode hooks, keycaps, etc, rather than simply a dispatch function.
> > 
> > I already said that I prefer not to rename existing modes.  Such
> > renaming will break too many init files and other Lisp programs.
> 
> I guess we can at least try it for a bit?

"Try" in what way?

> Because hook, keymaps, etc, should just work, that’s the point of major mode inheritance, after all. C-native-mode will run all the setup for c-mode, plus setup for c-native-mode.

Renaming public symbols is BAAAAD!  It causes breakage for many users and
Lisp programs.  It will either cause large-scale renaming of hooks, keymaps,
etc., which will break user init files; or it will cause confusion (because
a hook for cc-FOO-mode will be called cc-mode-hook and not the expected
cc-FOO-mode-hook).  And that is just the tip of the iceberg.

Why cannot we have the solution we already discussed and agreed upon:

  . modes that didn't exist before Emacs 29 will require tree-sitter
  . modes that existed before Emacs 29 will either
    - offer tree-sitter support as an optional feature, via a minor mode or
      a defcustom; or
    - add a completely new major mode with a different name that requires
      tree-sitter

If you need to add a FOO-base-mode to make it easier to share between
tree-sitter and non-tree-sitter modes features that are common to both, it's
fine with me to add such *-base-modes, but they should not be in any
auto-mode alist, and should generally be only an implementation detail
mostly hidden from users.

What is the problem with the above?  I thought we already agreed on that, so
how come this issue pops up time and again?

Re: Tree-sitter and major mode inheritance

[Po Lu]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> To the extent that Emacs-29's new `major-mode-remap-alist` can be used
> to select which mode to use, we can indeed leave the original modes
> as-is.

Yes, but you seem to contradict that preference below:

> - `<foo>-mode` as a dispatch function that calls the appropriate specific
>   major mode which could be `<foo>-ts-mode`, or `cc-<foo>-mode`, or
>   `<foo>-with-JSX-mode`, or ...

Why can't c-ts-mode be a separate mode, unrelated to c-mode in any way?
AFAIU it's supposed to be an optional feature users are supposed to turn
on for themselves.

Re: Tree-sitter and major mode inheritance

[Dmitry Gutov]

On 19.11.2022 12:26, Eli Zaretskii wrote:
>> From: Yuan Fu<casouri@gmail.com>
>> Date: Sat, 19 Nov 2022 01:41:47 -0800
>> Cc: Stefan Monnier<monnier@iro.umontreal.ca>,
>>   Eli Zaretskii<eliz@gnu.org>,
>>   Theodor Thornhill<theo@thornhill.no>
>>
>> Anyway, does anyone think this is a good/bad idea? Should I go implement this on css, js, c, etc? It can also be the other way around: instead of having c-mode being the virtual mode, we can leave c-mode as-is, and have a c-base-mode inherited by c-mode and c-ts-mode. And similarly rss-base-mode, rss-mode, and rss-ts-mode.
> I'd prefer leaving the original modes as-is.  That should cause less
> compatibility problems, I think.

Eli, what's your solution for the problem, then?

E.g. js-mode enables tree-sitter, and installs some stuff based on it.

But js2-mode inherits from js-mode (meaning, it will run the same setup 
code, and then some of its own), yet it has its own parser. Which will 
cause all sorts of conflicts with tree-sitter.

Re: Tree-sitter and major mode inheritance

[Yuan Fu]

> On Nov 19, 2022, at 1:39 PM, Dmitry Gutov <dgutov@yandex.ru> wrote:
> 
> On 19.11.2022 12:26, Eli Zaretskii wrote:
>>> From: Yuan Fu<casouri@gmail.com>
>>> Date: Sat, 19 Nov 2022 01:41:47 -0800
>>> Cc: Stefan Monnier<monnier@iro.umontreal.ca>,
>>>  Eli Zaretskii<eliz@gnu.org>,
>>>  Theodor Thornhill<theo@thornhill.no>
>>> 
>>> Anyway, does anyone think this is a good/bad idea? Should I go implement this on css, js, c, etc? It can also be the other way around: instead of having c-mode being the virtual mode, we can leave c-mode as-is, and have a c-base-mode inherited by c-mode and c-ts-mode. And similarly rss-base-mode, rss-mode, and rss-ts-mode.
>> I'd prefer leaving the original modes as-is.  That should cause less
>> compatibility problems, I think.
> 
> Eli, what's your solution for the problem, then?
> 
> E.g. js-mode enables tree-sitter, and installs some stuff based on it.
> 
> But js2-mode inherits from js-mode (meaning, it will run the same setup code, and then some of its own), yet it has its own parser. Which will cause all sorts of conflicts with tree-sitter.

Actually, that’s evidence supporting his preference: js-mode will remain to be the native implementation, so inheriting from it is exactly as before. Js-ts-mode will install tree-sitter stuff. And js-base-mode wouldn’t do much.


Yuan

Re: Tree-sitter and major mode inheritance

[Dmitry Gutov]

On 19.11.2022 23:49, Yuan Fu wrote:
> Actually, that’s evidence supporting his preference: js-mode will remain to be the native implementation, so inheriting from it is exactly as before. Js-ts-mode will install tree-sitter stuff. And js-base-mode wouldn’t do much.

But js-base-mode will be used in auto-mode-alist?

That should work, I think.

Re: Tree-sitter and major mode inheritance

[Dmitry Gutov]

On 20.11.2022 00:03, Dmitry Gutov wrote:
> On 19.11.2022 23:49, Yuan Fu wrote:
>> Actually, that’s evidence supporting his preference: js-mode will 
>> remain to be the native implementation, so inheriting from it is 
>> exactly as before. Js-ts-mode will install tree-sitter stuff. And 
>> js-base-mode wouldn’t do much.
> 
> But js-base-mode will be used in auto-mode-alist?
> 
> That should work, I think.

Could we make the dispatcher "modes" regular functions, though? Keeping 
them out of the inheritance chain.

That would make (derived-mode-p 'js-base-mode) always fail, of course, 
but if we are talking about existing code, there will be checks like 
(derived-mode-p 'js-mode) which are going to fail anyway now because 
js-js-mode isn't going to derive from js-mode. Could this be solvable 
through major-mode-remap-alist?

And if they (base modes) are not real modes, call it something like 
js-mode-dispatch or js-mode-virtual. Or js-mode-choose, etc, something 
with a verb at the end might do a better signal that it's not a "mode" 
and there is no point in inheriting or doing derived-mode-p checks on it.

Re: Tree-sitter and major mode inheritance

[Yuan Fu]

> On Nov 19, 2022, at 2:36 PM, Dmitry Gutov <dgutov@yandex.ru> wrote:
> 
> On 20.11.2022 00:03, Dmitry Gutov wrote:
>> On 19.11.2022 23:49, Yuan Fu wrote:
>>> Actually, that’s evidence supporting his preference: js-mode will remain to be the native implementation, so inheriting from it is exactly as before. Js-ts-mode will install tree-sitter stuff. And js-base-mode wouldn’t do much.
>> But js-base-mode will be used in auto-mode-alist?
>> That should work, I think.
> 
> Could we make the dispatcher "modes" regular functions, though? Keeping them out of the inheritance chain.
> 
> That would make (derived-mode-p 'js-base-mode) always fail, of course, but if we are talking about existing code, there will be checks like (derived-mode-p 'js-mode) which are going to fail anyway now because js-js-mode isn't going to derive from js-mode. Could this be solvable through major-mode-remap-alist?
> 
> And if they (base modes) are not real modes, call it something like js-mode-dispatch or js-mode-virtual. Or js-mode-choose, etc, something with a verb at the end might do a better signal that it's not a "mode" and there is no point in inheriting or doing derived-mode-p checks on it.

If we keep js-mode as-is, and add js-base-mode and js-ts-mode, (derived-mode-p ‘js-mode) should keep working as before, or maybe I’m msiunderstanding your question?

Yuan

Re: Tree-sitter and major mode inheritance

[Dmitry Gutov]

On 20.11.2022 01:36, Yuan Fu wrote:
> If we keep js-mode as-is, and add js-base-mode and js-ts-mode, (derived-mode-p ‘js-mode) should keep working as before, or maybe I’m msiunderstanding your question?

(derived-mode-p 'js-mode) will return nil in js-ts-mode.

Which could be a problem when this call is used as a substitute for a 
file type check (e.g. "are we editing a JavaScript file?"), which is one 
of its common uses.

Re: Tree-sitter and major mode inheritance

[Eli Zaretskii]

> Date: Sun, 20 Nov 2022 01:42:04 +0200
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org,
>  monnier@iro.umontreal.ca, theo@thornhill.no
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 20.11.2022 01:36, Yuan Fu wrote:
> > If we keep js-mode as-is, and add js-base-mode and js-ts-mode, (derived-mode-p ‘js-mode) should keep working as before, or maybe I’m msiunderstanding your question?
> 
> (derived-mode-p 'js-mode) will return nil in js-ts-mode.
> 
> Which could be a problem when this call is used as a substitute for a 
> file type check (e.g. "are we editing a JavaScript file?"), which is one 
> of its common uses.

This test can only work on the assumption that there's a single parent mode
for all the modes which support a given programming language.  This is a
fragile assumption, so code which is based on it is broken and should be
fixed.

IOW, I don't think we should feel ourselves bound by this fragile
assumption (assuming the solution we decide on breaks it).

Re: Tree-sitter and major mode inheritance

[Dmitry Gutov]

On 20.11.2022 09:28, Eli Zaretskii wrote:
>> Date: Sun, 20 Nov 2022 01:42:04 +0200
>> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org,
>>   monnier@iro.umontreal.ca, theo@thornhill.no
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>> On 20.11.2022 01:36, Yuan Fu wrote:
>>> If we keep js-mode as-is, and add js-base-mode and js-ts-mode, (derived-mode-p ‘js-mode) should keep working as before, or maybe I’m msiunderstanding your question?
>>
>> (derived-mode-p 'js-mode) will return nil in js-ts-mode.
>>
>> Which could be a problem when this call is used as a substitute for a
>> file type check (e.g. "are we editing a JavaScript file?"), which is one
>> of its common uses.
> 
> This test can only work on the assumption that there's a single parent mode
> for all the modes which support a given programming language.

Technically correct, the best kind of correct.

> This is a
> fragile assumption, so code which is based on it is broken and should be
> fixed.

Okay then, but then we'll need to learn another way to ask that 
question. Previously, we did try to ensure (not always successfully) a 
single inheritance chain between such modes.

Re: Tree-sitter and major mode inheritance

[Eli Zaretskii]

> Date: Sun, 20 Nov 2022 15:22:40 +0200
> Cc: casouri@gmail.com, emacs-devel@gnu.org, monnier@iro.umontreal.ca,
>  theo@thornhill.no
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> > This is a
> > fragile assumption, so code which is based on it is broken and should be
> > fixed.
> 
> Okay then, but then we'll need to learn another way to ask that 
> question.

I guess.  I admit I didn't know derived-mode-p was being used for such
tests.

Would it make sense to use alternatives, as in

  (or (derived-mode-p 'js-mode)
      (derived-mode-p 'js-ts-mode))

?

Or maybe we should add a new predicate, which will take a LANGUAGE argument,
and use some database of known modes internally to call derived-mode-p as
above?

Re: Tree-sitter and major mode inheritance

[Dmitry Gutov]

On 20.11.2022 16:49, Eli Zaretskii wrote:
>> Date: Sun, 20 Nov 2022 15:22:40 +0200
>> Cc: casouri@gmail.com, emacs-devel@gnu.org, monnier@iro.umontreal.ca,
>>   theo@thornhill.no
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>>> This is a
>>> fragile assumption, so code which is based on it is broken and should be
>>> fixed.
>>
>> Okay then, but then we'll need to learn another way to ask that
>> question.
> 
> I guess.  I admit I didn't know derived-mode-p was being used for such
> tests.
> 
> Would it make sense to use alternatives, as in
> 
>    (or (derived-mode-p 'js-mode)
>        (derived-mode-p 'js-ts-mode))
> 
> ?

Those kind of lists are going to be inherently non-exhaustive.

> Or maybe we should add a new predicate, which will take a LANGUAGE argument,
> and use some database of known modes internally to call derived-mode-p as
> above?

Some kind of new registry could be the answer, but it would be nice to 
manage using the existing tools/variables somehow.

Stefan has touched on this issue in https://debbugs.gnu.org/58075

Re: Tree-sitter and major mode inheritance

[Eli Zaretskii]

> Date: Sun, 20 Nov 2022 00:03:35 +0200
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org,
>  monnier@iro.umontreal.ca, theo@thornhill.no
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 19.11.2022 23:49, Yuan Fu wrote:
> > Actually, that’s evidence supporting his preference: js-mode will remain to be the native implementation, so inheriting from it is exactly as before. Js-ts-mode will install tree-sitter stuff. And js-base-mode wouldn’t do much.
> 
> But js-base-mode will be used in auto-mode-alist?

NO!!!  auto-mode-alist should keep using js-mode, as it does today.

js-base-mode, if we need it, should just be a vehicle for easy sharing of
common stuff between several modes that pertain to the same or similar
languages.  It should NOT be visible to users, so should not appear in any
variables users are likely to customize.

Re: Tree-sitter and major mode inheritance

[Yuan Fu]

> On Nov 19, 2022, at 11:11 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> Date: Sun, 20 Nov 2022 00:03:35 +0200
>> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org,
>> monnier@iro.umontreal.ca, theo@thornhill.no
>> From: Dmitry Gutov <dgutov@yandex.ru>
>> 
>> On 19.11.2022 23:49, Yuan Fu wrote:
>>> Actually, that’s evidence supporting his preference: js-mode will remain to be the native implementation, so inheriting from it is exactly as before. Js-ts-mode will install tree-sitter stuff. And js-base-mode wouldn’t do much.
>> 
>> But js-base-mode will be used in auto-mode-alist?
> 
> NO!!!  auto-mode-alist should keep using js-mode, as it does today.
> 
> js-base-mode, if we need it, should just be a vehicle for easy sharing of
> common stuff between several modes that pertain to the same or similar
> languages.  It should NOT be visible to users, so should not appear in any
> variables users are likely to customize.

Alright, changing js-mode to ja-native-mode is indeed a bad idea. So this is what I did: for eg, js-mode, I created js-ts-mode and js-base-mode. Js-mode and js-ts-mode inherits js-base-mode. Auto-mode-alist has javascript-mode (that’s what’s in the list right now, I didn’t change it).

If a user wants to use tree-sitter for javascript, they can add (javascript-mode . js-ts-mode) into major-mode-remap-alist.

If someone wants js-mode and js-ts-mode to share the same configuration, he needs to configure js-base-mode. So js-base-mode isn’t completely invisible. 

Yuan

Re: Tree-sitter and major mode inheritance

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Sun, 20 Nov 2022 01:19:13 -0800
> Cc: Dmitry Gutov <dgutov@yandex.ru>,
>  emacs-devel@gnu.org,
>  monnier@iro.umontreal.ca,
>  theo@thornhill.no
> 
> 
> 
> > On Nov 19, 2022, at 11:11 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> > 
> >> Date: Sun, 20 Nov 2022 00:03:35 +0200
> >> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org,
> >> monnier@iro.umontreal.ca, theo@thornhill.no
> >> From: Dmitry Gutov <dgutov@yandex.ru>
> >> 
> >> On 19.11.2022 23:49, Yuan Fu wrote:
> >>> Actually, that’s evidence supporting his preference: js-mode will remain to be the native implementation, so inheriting from it is exactly as before. Js-ts-mode will install tree-sitter stuff. And js-base-mode wouldn’t do much.
> >> 
> >> But js-base-mode will be used in auto-mode-alist?
> > 
> > NO!!!  auto-mode-alist should keep using js-mode, as it does today.
> > 
> > js-base-mode, if we need it, should just be a vehicle for easy sharing of
> > common stuff between several modes that pertain to the same or similar
> > languages.  It should NOT be visible to users, so should not appear in any
> > variables users are likely to customize.
> 
> Alright, changing js-mode to ja-native-mode is indeed a bad idea. So this is what I did: for eg, js-mode, I created js-ts-mode and js-base-mode. Js-mode and js-ts-mode inherits js-base-mode. Auto-mode-alist has javascript-mode (that’s what’s in the list right now, I didn’t change it).
> 
> If a user wants to use tree-sitter for javascript, they can add (javascript-mode . js-ts-mode) into major-mode-remap-alist.

Or manually turn on js-ts-mode.  Or customize auto-mode-alist to have
js-ts-mode be used in preference to js-mode.  Right?

> If someone wants js-mode and js-ts-mode to share the same configuration, he needs to configure js-base-mode. So js-base-mode isn’t completely invisible. 

We should not advertise js-base-mode.  If people want to share
configurations between two modes, they should do that explicitly via their
own hook function, or via copying the configurations.  (And some of the
configurations cannot be shared anyway, because the two modes use different
features for font-lock and indentation.)

Re: Tree-sitter and major mode inheritance

[Yuan Fu]

> On Nov 20, 2022, at 2:02 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Yuan Fu <casouri@gmail.com>
>> Date: Sun, 20 Nov 2022 01:19:13 -0800
>> Cc: Dmitry Gutov <dgutov@yandex.ru>,
>> emacs-devel@gnu.org,
>> monnier@iro.umontreal.ca,
>> theo@thornhill.no
>> 
>> 
>> 
>>> On Nov 19, 2022, at 11:11 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>>> 
>>>> Date: Sun, 20 Nov 2022 00:03:35 +0200
>>>> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org,
>>>> monnier@iro.umontreal.ca, theo@thornhill.no
>>>> From: Dmitry Gutov <dgutov@yandex.ru>
>>>> 
>>>> On 19.11.2022 23:49, Yuan Fu wrote:
>>>>> Actually, that’s evidence supporting his preference: js-mode will remain to be the native implementation, so inheriting from it is exactly as before. Js-ts-mode will install tree-sitter stuff. And js-base-mode wouldn’t do much.
>>>> 
>>>> But js-base-mode will be used in auto-mode-alist?
>>> 
>>> NO!!!  auto-mode-alist should keep using js-mode, as it does today.
>>> 
>>> js-base-mode, if we need it, should just be a vehicle for easy sharing of
>>> common stuff between several modes that pertain to the same or similar
>>> languages.  It should NOT be visible to users, so should not appear in any
>>> variables users are likely to customize.
>> 
>> Alright, changing js-mode to ja-native-mode is indeed a bad idea. So this is what I did: for eg, js-mode, I created js-ts-mode and js-base-mode. Js-mode and js-ts-mode inherits js-base-mode. Auto-mode-alist has javascript-mode (that’s what’s in the list right now, I didn’t change it).
>> 
>> If a user wants to use tree-sitter for javascript, they can add (javascript-mode . js-ts-mode) into major-mode-remap-alist.
> 
> Or manually turn on js-ts-mode.  Or customize auto-mode-alist to have
> js-ts-mode be used in preference to js-mode.  Right?

Right.

> 
>> If someone wants js-mode and js-ts-mode to share the same configuration, he needs to configure js-base-mode. So js-base-mode isn’t completely invisible. 
> 
> We should not advertise js-base-mode.  If people want to share
> configurations between two modes, they should do that explicitly via their
> own hook function, or via copying the configurations.  (And some of the
> configurations cannot be shared anyway, because the two modes use different
> features for font-lock and indentation.)

Right.

Yuan

Re: Tree-sitter and major mode inheritance

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Sat, 19 Nov 2022 13:49:50 -0800
> Cc: Eli Zaretskii <eliz@gnu.org>,
>  emacs-devel@gnu.org,
>  monnier@iro.umontreal.ca,
>  theo@thornhill.no
> 
> 
> 
> > On Nov 19, 2022, at 1:39 PM, Dmitry Gutov <dgutov@yandex.ru> wrote:
> > 
> > On 19.11.2022 12:26, Eli Zaretskii wrote:
> >>> From: Yuan Fu<casouri@gmail.com>
> >>> Date: Sat, 19 Nov 2022 01:41:47 -0800
> >>> Cc: Stefan Monnier<monnier@iro.umontreal.ca>,
> >>>  Eli Zaretskii<eliz@gnu.org>,
> >>>  Theodor Thornhill<theo@thornhill.no>
> >>> 
> >>> Anyway, does anyone think this is a good/bad idea? Should I go implement this on css, js, c, etc? It can also be the other way around: instead of having c-mode being the virtual mode, we can leave c-mode as-is, and have a c-base-mode inherited by c-mode and c-ts-mode. And similarly rss-base-mode, rss-mode, and rss-ts-mode.
> >> I'd prefer leaving the original modes as-is.  That should cause less
> >> compatibility problems, I think.
> > 
> > Eli, what's your solution for the problem, then?
> > 
> > E.g. js-mode enables tree-sitter, and installs some stuff based on it.
> > 
> > But js2-mode inherits from js-mode (meaning, it will run the same setup code, and then some of its own), yet it has its own parser. Which will cause all sorts of conflicts with tree-sitter.
> 
> Actually, that’s evidence supporting his preference: js-mode will remain to be the native implementation, so inheriting from it is exactly as before. Js-ts-mode will install tree-sitter stuff. And js-base-mode wouldn’t do much.

Why do we need js-base-mode? what will it do?

Re: Tree-sitter and major mode inheritance

[Dmitry Gutov]

On 20.11.2022 09:05, Eli Zaretskii wrote:
> Why do we need js-base-mode? what will it do?

I was under the impression that it would dispatch either to js-mode or 
to js-ts-mode based on the value of some global defcustom like 
treesit-global-modes (similar in structure for font-lock-global-modes).

But I guess not. If that choice is going to be made instead through 
auto-mode-alist and major-mode-remap-alist.

Re: Tree-sitter and major mode inheritance

[Eli Zaretskii]

> Date: Sun, 20 Nov 2022 19:12:27 +0200
> Cc: emacs-devel@gnu.org, monnier@iro.umontreal.ca, theo@thornhill.no
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 20.11.2022 09:05, Eli Zaretskii wrote:
> > Why do we need js-base-mode? what will it do?
> 
> I was under the impression that it would dispatch either to js-mode or 
> to js-ts-mode based on the value of some global defcustom like 
> treesit-global-modes (similar in structure for font-lock-global-modes).
> 
> But I guess not. If that choice is going to be made instead through 
> auto-mode-alist and major-mode-remap-alist.

Yes, I prefer to hide the *-base-mode modes from users as much as possible,
or not have them at all.  I think it will prevent confusion.

Re: Tree-sitter and major mode inheritance

[Eli Zaretskii]

> Date: Sat, 19 Nov 2022 23:39:37 +0200
> Cc: emacs-devel@gnu.org, monnier@iro.umontreal.ca, theo@thornhill.no
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 19.11.2022 12:26, Eli Zaretskii wrote:
> >> From: Yuan Fu<casouri@gmail.com>
> >> Date: Sat, 19 Nov 2022 01:41:47 -0800
> >> Cc: Stefan Monnier<monnier@iro.umontreal.ca>,
> >>   Eli Zaretskii<eliz@gnu.org>,
> >>   Theodor Thornhill<theo@thornhill.no>
> >>
> >> Anyway, does anyone think this is a good/bad idea? Should I go implement this on css, js, c, etc? It can also be the other way around: instead of having c-mode being the virtual mode, we can leave c-mode as-is, and have a c-base-mode inherited by c-mode and c-ts-mode. And similarly rss-base-mode, rss-mode, and rss-ts-mode.
> > I'd prefer leaving the original modes as-is.  That should cause less
> > compatibility problems, I think.
> 
> Eli, what's your solution for the problem, then?

I don't think I understand the question.  Several (3, AFAIU) solutions were
proposed, one of them leaves the original modes intact and either adds
opt-in features to the original modes to turn on tree-sitter support, or
adds an entirely new mode which requires tree-sitter.  This is the solution
I prefer.

> E.g. js-mode enables tree-sitter, and installs some stuff based on it.

Only if tree-sitter is available, AFAICT.  Btw, if that happens
automatically, then it isn't what I meant -- I meant tree-sitter to be an
explicitly opt-in feature in modes which existed before Emacs 29 and worked
without tree-sitter.

> But js2-mode inherits from js-mode (meaning, it will run the same setup 
> code, and then some of its own), yet it has its own parser. Which will 
> cause all sorts of conflicts with tree-sitter.

js2-mode is not in Emacs, so I cannot control what it does.  Ideally, it
will need only minor adjustments (like making sure it doesn't turn on
tree-sitter if it doesn't want to) or none at all.

If our changes somehow break js2-mode, we should discuss the details and try
to fix the breakage as much as is reasonable from our side.  The details
aren't important from where I stand; what is important is that users of
js-mode can still use the mode even if they don't have tree-sitter installed
or don't want to use it even if it is installed.

Re: Tree-sitter and major mode inheritance

[Dmitry Gutov]

On 20.11.2022 08:51, Eli Zaretskii wrote:
>> But js2-mode inherits from js-mode (meaning, it will run the same setup
>> code, and then some of its own), yet it has its own parser. Which will
>> cause all sorts of conflicts with tree-sitter.
> js2-mode is not in Emacs, so I cannot control what it does.  Ideally, it
> will need only minor adjustments (like making sure it doesn't turn on
> tree-sitter if it doesn't want to) or none at all.

That was my question: how would those "minor adjustments" look if not 
the way Yuan proposed things.

> If our changes somehow break js2-mode, we should discuss the details and try
> to fix the breakage as much as is reasonable from our side.  The details
> aren't important from where I stand; what is important is that users of
> js-mode can still use the mode even if they don't have tree-sitter installed
> or don't want to use it even if it is installed.

I agree with that goal, of course.

Re: Tree-sitter and major mode inheritance

[Eli Zaretskii]

> Date: Sun, 20 Nov 2022 14:45:59 +0200
> Cc: casouri@gmail.com, emacs-devel@gnu.org, monnier@iro.umontreal.ca,
>  theo@thornhill.no
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 20.11.2022 08:51, Eli Zaretskii wrote:
> >> But js2-mode inherits from js-mode (meaning, it will run the same setup
> >> code, and then some of its own), yet it has its own parser. Which will
> >> cause all sorts of conflicts with tree-sitter.
> > js2-mode is not in Emacs, so I cannot control what it does.  Ideally, it
> > will need only minor adjustments (like making sure it doesn't turn on
> > tree-sitter if it doesn't want to) or none at all.
> 
> That was my question: how would those "minor adjustments" look if not 
> the way Yuan proposed things.

I don't know, sorry.  First, Yuan didn't yet finish adjusting the code to
what we have said here today.  And on top of that, I'm not familiar with
js2-mode, so I'd be grateful if someone else could tell what would be needed
there (when that is clear).