Extending define-derived-mode

Extending define-derived-mode

[Yuan Fu]

When we were adding tree-sitter modes a couple of month ago, it was
clear that the current major mode model needs some upgrade, I’d like
to discuss the things we need and how can we address them.

Features we want:

1. Fallback modes: user enables xxx-ts-mode, but there’s no
   tree-sitter grammar for xxx, so Emacs falls back to xxx-mode
   instead. This feature is also desirable for some non-tree-sitter
   modes, like TeX modes. Ideally the dispatch should happen before
   major mode does anything.

2. Inheritance:

2.1 For xxx-mode and xxx-ts-mode, there should be shared hook. More
    generally, we want to be able to have a shared hook for similar
    modes (same language, similar language, etc).

2.2 For xxx-mode and xxx-ts-mode, they should be able to share some
    setup, and diverge on some other.

2.3 Right now we have a little inconsistency regarding hook
    inheritance. js-json-mode inherits from js-mode, json-ts-mode
    inherits from js-ts-mode. So there isn’t a shared hook that runs
    in both js-json-mode and json-ts-mode.

    More generally, if there is a language X and a derived language Y,
    and we have x-mode, x-ts-mode, y-mode, y-ts-mode, how should
    inheritance of code and hooks works among them? y-mode probably
    wants to inherit from x-mode, and y-ts-mode probably wants to
    inherit hook (but not necessarily code [1]) from x-ts-mode.

[1] If Y is different enough from X such that it has its own
tree-sitter grammar, y-ts-mode cannot inherit x-ts-mode’s code because
they target different grammar.

3. Unrelated to tree-sitter, here’s something I personally want:
   it would be nice if every major mode can
   have a hook that’s not run by its derived modes. Use case:
   sage-mode inherits python-mode. I have eglot-ensure in
   python-mode-hook but don’t want it when using sage-mode. Right now
   I have to wrap eglot-ensure with a lambda function so that it only
   runs when major-mode = python-mode.


Implementation wise, here are my thoughts regarding each feature:

1. Shouldn’t be hard to add some facility to define-derived-mode that
   runs before everything and dispatches to different major modes.
   Maybe define-derive-mode can have a :dispatch keyword which takes a
   arbitrary function; and if the function returns nil, rest of the
   major mode setup are skipped.

2. For those who don’t know, right now we define a xxx-base-mode,
   which is then inherited by xxx-mode and xxx-ts-mode. My
   understanding is that xxx-base-mode is considered a temporary
   solution and will be replaced. Personally I don’t mind
   xxx-base-mode, why can’t we keep it?

   From what I can see, major mode inheritance is used for three
   purposes: a) inheriting from a “virtual” mode (eg, prog-mode) for
   hooks and keymaps; b) y-mode deriving from x-mode, where Y is a
   derived language from X; c) y-2-mode deriving from y-mode as an
   extension, where y-2-mode can be a user’s personal customization,
   or distributed as packages (js2-mode).

   I think the current model serves a) and c) just fine. But for b),
   and for when there are multiple major modes for the same language,
   the single-inheritance model can be a bit lacking. I don’t have a
   clear vision right now, perhaps we want multiple inheritance for
   hooks, perhaps we can simply allow define-derived-mode to specify
   additional hooks to run, and create some language hooks that are
   shared by every mode serving that language. I’d like to hear your
   thoughts.

3. Like 1., shouldn’t be hard to modify define-derived-mode to add a
   hook that’s not run by its derived modes.

Yuan

Re: Extending define-derived-mode

[Theodor Thornhill]

On 30 May 2023 07:16:49 CEST, Yuan Fu <casouri@gmail.com> wrote:
>When we were adding tree-sitter modes a couple of month ago, it was
>clear that the current major mode model needs some upgrade, I’d like
>to discuss the things we need and how can we address them.

Thanks for bringing this up!

I'll quickly just add one more thing, and respond more thoroughly in a different mail:


Maybe we can consider another abstraction to cover the following:

 - configure what set of modes to use for a given language
 - set choice of diagnostic tool (flymake/flycheck etc)
 - set LSP client implementation with config to use
 - project based configs
 - separate mode name and implementation

I posted some time ago a simple idea covering this, and maybe the time is now to revisit this idea? People seem to like doom Emacs/spacemacs etc, so their idea of "layers" may be a nice route to consider.

We can supply some DSL with preconfigured settings, and add extensible functionalities.

Maybe something like:

(define-layer 'javascript-mode
  :mode 'treesit
  :enabled-in '(".js", ".mjs")
  :diagnostic-backend 'flymake
  :lsp-client 'lsp-mode ;; default is 'eglot
  :project '(:root-function #'some-rootfinder-fn)
  :other-stuff '(eslint json-something nvm))

This may seem only tangentially related to your email, but I believe it is, in that mode inheritance touches the "the first language mode implementation owns the namespace"-problem.
 
WDYT?

Theo

Re: Extending define-derived-mode

[Yuan Fu]

> On May 29, 2023, at 10:51 PM, Theodor Thornhill <theo@thornhill.no> wrote:
> 
> 
> 
> On 30 May 2023 07:16:49 CEST, Yuan Fu <casouri@gmail.com> wrote:
>> When we were adding tree-sitter modes a couple of month ago, it was
>> clear that the current major mode model needs some upgrade, I’d like
>> to discuss the things we need and how can we address them.
> 
> Thanks for bringing this up!
> 
> I'll quickly just add one more thing, and respond more thoroughly in a different mail:
> 
> 
> Maybe we can consider another abstraction to cover the following:
> 
> - configure what set of modes to use for a given language
> - set choice of diagnostic tool (flymake/flycheck etc)
> - set LSP client implementation with config to use
> - project based configs
> - separate mode name and implementation
> 
> I posted some time ago a simple idea covering this, and maybe the time is now to revisit this idea? People seem to like doom Emacs/spacemacs etc, so their idea of "layers" may be a nice route to consider.
> 
> We can supply some DSL with preconfigured settings, and add extensible functionalities.
> 
> Maybe something like:
> 
> (define-layer 'javascript-mode
>  :mode 'treesit
>  :enabled-in '(".js", ".mjs")
>  :diagnostic-backend 'flymake
>  :lsp-client 'lsp-mode ;; default is 'eglot
>  :project '(:root-function #'some-rootfinder-fn)
>  :other-stuff '(eslint json-something nvm))
> 
> This may seem only tangentially related to your email, but I believe it is, in that mode inheritance touches the "the first language mode implementation owns the namespace"-problem.

Yeah this look interesting. From your example it looks more like a configuration oriented, like a more integrated use-package. I feel like we should solve the "the first language mode implementation owns the namespace”-problem in a more direct and standalone way.

Yuan

Re: Extending define-derived-mode

[Theodor Thornhill]

Yuan Fu <casouri@gmail.com> writes:

>> On May 29, 2023, at 10:51 PM, Theodor Thornhill <theo@thornhill.no> wrote:
>> 
>> 
>> 
>> On 30 May 2023 07:16:49 CEST, Yuan Fu <casouri@gmail.com> wrote:
>>> When we were adding tree-sitter modes a couple of month ago, it was
>>> clear that the current major mode model needs some upgrade, I’d like
>>> to discuss the things we need and how can we address them.
>> 
>> Thanks for bringing this up!
>> 
>> I'll quickly just add one more thing, and respond more thoroughly in a different mail:
>> 
>> 
>> Maybe we can consider another abstraction to cover the following:
>> 
>> - configure what set of modes to use for a given language
>> - set choice of diagnostic tool (flymake/flycheck etc)
>> - set LSP client implementation with config to use
>> - project based configs
>> - separate mode name and implementation
>> 
>> I posted some time ago a simple idea covering this, and maybe the time is now to revisit this idea? People seem to like doom Emacs/spacemacs etc, so their idea of "layers" may be a nice route to consider.
>> 
>> We can supply some DSL with preconfigured settings, and add extensible functionalities.
>> 
>> Maybe something like:
>> 
>> (define-layer 'javascript-mode
>>  :mode 'treesit
>>  :enabled-in '(".js", ".mjs")
>>  :diagnostic-backend 'flymake
>>  :lsp-client 'lsp-mode ;; default is 'eglot
>>  :project '(:root-function #'some-rootfinder-fn)
>>  :other-stuff '(eslint json-something nvm))
>> 
>> This may seem only tangentially related to your email, but I believe it is, in that mode inheritance touches the "the first language mode implementation owns the namespace"-problem.
>
> Yeah this look interesting. From your example it looks more like a configuration oriented, like a more integrated use-package. I feel like we should solve the "the first language mode implementation owns the namespace”-problem in a more direct and standalone way.
>
> Yuan


Yeah, I agree. This example would piggyback on such a solution. 

Re: Extending define-derived-mode

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Mon, 29 May 2023 22:16:49 -0700
> Cc: Mickey Petersen <mickey@masteringemacs.org>,
>  Theodor Thornhill <theo@thornhill.no>, Dmitry Gutov <dgutov@yandex.ru>
> 
> When we were adding tree-sitter modes a couple of month ago, it was
> clear that the current major mode model needs some upgrade, I’d like
> to discuss the things we need and how can we address them.

You are basically talking about different modes that support the same
programming language or file format.  We never had this in Emacs, with
(AFAIK) the single exception of perl-mode vs cperl-mode, which never
played well.  I think some thought was invested in trying to reconcile
them (Stefan, am I right?), but we never came up with a good solution.
Not sure if that is a general problem or not.

> 1. Fallback modes: user enables xxx-ts-mode, but there’s no
>    tree-sitter grammar for xxx, so Emacs falls back to xxx-mode
>    instead. This feature is also desirable for some non-tree-sitter
>    modes, like TeX modes. Ideally the dispatch should happen before
>    major mode does anything.

This fallback must be user-controlled.  At the very least, users
should be able to say whether they want the fallback, and if they do,
whether the fallback should be silent or vociferous.

> 2. Inheritance:
> 
> 2.1 For xxx-mode and xxx-ts-mode, there should be shared hook. More
>     generally, we want to be able to have a shared hook for similar
>     modes (same language, similar language, etc).
> 
> 2.2 For xxx-mode and xxx-ts-mode, they should be able to share some
>     setup, and diverge on some other.
> 
> 2.3 Right now we have a little inconsistency regarding hook
>     inheritance. js-json-mode inherits from js-mode, json-ts-mode
>     inherits from js-ts-mode. So there isn’t a shared hook that runs
>     in both js-json-mode and json-ts-mode.
> 
>     More generally, if there is a language X and a derived language Y,
>     and we have x-mode, x-ts-mode, y-mode, y-ts-mode, how should
>     inheritance of code and hooks works among them? y-mode probably
>     wants to inherit from x-mode, and y-ts-mode probably wants to
>     inherit hook (but not necessarily code [1]) from x-ts-mode.

This is not simple in practice.  Ask yourself why don't the
FOO-ts-mode's simply run the (existing) hooks of FOO-mode?  Look at
the code, and you will understand: the stuff people put in these hooks
usually customizes aspects that are very different in each of the
modes: font-lock, indentation, syntax tables, supporting "electric"
sub-modes, styles of comments and other things, etc.  The few places
where the new TS modes tried to run the same hooks and share most of
the initialization code (for example, bash-ts-mode and sh-mode,
python-ts-mode and python-mode) introduced subtle bugs which were only
recently fixed.  One problem with inheriting from the same base mode
is that doing initialization there could be problematic because the
mode stuff that gets automatically instantiated and initialized by
define-derived-mode, such as the mode's map and variable, is not yet
set up when the base mode's initialization runs, so you cannot do some
of the things we are used to do there, like setting up the menu-bar
menus.

So it would seem that, for efficient and convenient facilities of this
kind, we need some way of defining stuff in the base mode, which will
be actually run later, when the child mode is set up.

> 3. Unrelated to tree-sitter, here’s something I personally want:
>    it would be nice if every major mode can
>    have a hook that’s not run by its derived modes. Use case:
>    sage-mode inherits python-mode. I have eglot-ensure in
>    python-mode-hook but don’t want it when using sage-mode. Right now
>    I have to wrap eglot-ensure with a lambda function so that it only
>    runs when major-mode = python-mode.

What is wrong with that solution?  Except for the maybe minor
inconvenience of having to use a lambda-function, what different way
of doing this would you envision except telling define-derived-mode to
run some hook only under this-and-that condition?

> 1. Shouldn’t be hard to add some facility to define-derived-mode that
>    runs before everything and dispatches to different major modes.
>    Maybe define-derive-mode can have a :dispatch keyword which takes a
>    arbitrary function; and if the function returns nil, rest of the
>    major mode setup are skipped.

What kind of code would run before the dispatch?  I suggest to take a
couple of examples of related modes, and examine their common parts,
so you have a clear idea of the potential issues.

> 3. Like 1., shouldn’t be hard to modify define-derived-mode to add a
>    hook that’s not run by its derived modes.

If this is a frequent need, then yes, adding that would be easy.

Re: Extending define-derived-mode

[Stefan Monnier]

> You are basically talking about different modes that support the same
> programming language or file format.  We never had this in Emacs, with
> (AFAIK) the single exception of perl-mode vs cperl-mode, which never
> played well.

Not quite: within Emacs there's at least pascal.el vs opascal.el and
sgml-mode vs nxml-mode, and depending on how you define it there's also
postscript-mode vs doc-view-mode, as well as c-mode vs image-mode (for
XPM files).

And if we consider packages not bundled with Emacs, there are many more
case:s such as latex-mode vs LaTeX-mode, python.el vs python-mode.el,
js.el vs js2.el vs js3.el, octave-mode vs matlab-mode.

> I think some thought was invested in trying to reconcile
> them (Stefan, am I right?), but we never came up with a good solution.
> Not sure if that is a general problem or not.

We don't really have a good answer yet, no.
`major-mode-remap-alist` is aimed at this problem, but brand new so it's
not clear how useful it will be for that and it's definitely not
a complete solution.

>> 1. Fallback modes: user enables xxx-ts-mode, but there’s no
>>    tree-sitter grammar for xxx, so Emacs falls back to xxx-mode
>>    instead. This feature is also desirable for some non-tree-sitter
>>    modes, like TeX modes. Ideally the dispatch should happen before
>>    major mode does anything.
>
> This fallback must be user-controlled.

I don't see this as a big problem, actually (there are already several
mechanisms that can do that).  The question of how "user enables
xxx-ts-mode" is probably harder.

>> 2.1 For xxx-mode and xxx-ts-mode, there should be shared hook. More
>>     generally, we want to be able to have a shared hook for similar
>>     modes (same language, similar language, etc).

As Eli explains, this is not always desirable.  And at other times it
*is* desirable: the users's hook function can set/use features that are
specific to one of the alternative, but they can also set/use features
that are shared between the alternatives :-(

Most users use only one of the alternatives, tho, so it's usually not
a big problem (other than introducing incompatibilities when Emacs's
defaults change from one alternative to another).

It can be more annoying for `.dir-locals.el` per-mode settings.

>>     More generally, if there is a language X and a derived language Y,
>>     and we have x-mode, x-ts-mode, y-mode, y-ts-mode, how should
>>     inheritance of code and hooks works among them? y-mode probably
>>     wants to inherit from x-mode, and y-ts-mode probably wants to
>>     inherit hook (but not necessarily code [1]) from x-ts-mode.

`y-ts-mode` can explicitly run `y-mode-hook` (or `x-ts-mode-hook`).

We may more generally want to extend our notion of "derived" mode to
allow "multiple inheritance".  For the actual activation code part,
multiple inheritance is a hard problem that we probably don't want to
tackle, but we can easily run several parent hooks, setup multiple
keymap karents, and make `derived-mode-p` support multiple parents.

>> 3. Unrelated to tree-sitter, here’s something I personally want:
>>    it would be nice if every major mode can
>>    have a hook that’s not run by its derived modes. Use case:
>>    sage-mode inherits python-mode. I have eglot-ensure in
>>    python-mode-hook but don’t want it when using sage-mode. Right now
>>    I have to wrap eglot-ensure with a lambda function so that it only
>>    runs when major-mode = python-mode.
>
> What is wrong with that solution?  Except for the maybe minor
> inconvenience of having to use a lambda-function, what different way
> of doing this would you envision except telling define-derived-mode to
> run some hook only under this-and-that condition?

While I tend to agree that it's not a big deal, I also agree that it's
arguably cleaner if parent modes are kept "abstract", so rather than
have `c++-mode` inherit from `c-mode`, you make them both inherit from
a `c-base-mode`.  It tends to smell of overkill, tho.


        Stefan

Re: Extending define-derived-mode

[Yuan Fu]

> On May 30, 2023, at 7:16 AM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> 
>> You are basically talking about different modes that support the same
>> programming language or file format.  We never had this in Emacs, with
>> (AFAIK) the single exception of perl-mode vs cperl-mode, which never
>> played well.
> 
> Not quite: within Emacs there's at least pascal.el vs opascal.el and
> sgml-mode vs nxml-mode, and depending on how you define it there's also
> postscript-mode vs doc-view-mode, as well as c-mode vs image-mode (for
> XPM files).
> 
> And if we consider packages not bundled with Emacs, there are many more
> case:s such as latex-mode vs LaTeX-mode, python.el vs python-mode.el,
> js.el vs js2.el vs js3.el, octave-mode vs matlab-mode.
> 
>> I think some thought was invested in trying to reconcile
>> them (Stefan, am I right?), but we never came up with a good solution.
>> Not sure if that is a general problem or not.
> 
> We don't really have a good answer yet, no.
> `major-mode-remap-alist` is aimed at this problem, but brand new so it's
> not clear how useful it will be for that and it's definitely not
> a complete solution.
> 
>>> 1. Fallback modes: user enables xxx-ts-mode, but there’s no
>>>   tree-sitter grammar for xxx, so Emacs falls back to xxx-mode
>>>   instead. This feature is also desirable for some non-tree-sitter
>>>   modes, like TeX modes. Ideally the dispatch should happen before
>>>   major mode does anything.
>> 
>> This fallback must be user-controlled.
> 
> I don't see this as a big problem, actually (there are already several
> mechanisms that can do that).  The question of how "user enables
> xxx-ts-mode" is probably harder.

Couldn’t they use major-mode-remap-alist?

There are mechanisms to do that, yes, if you are talking about the advices for latex-mode and bash-ts-mode. But I’d love to see a standard, cleaner way to do it.

> 
>>> 2.1 For xxx-mode and xxx-ts-mode, there should be shared hook. More
>>>    generally, we want to be able to have a shared hook for similar
>>>    modes (same language, similar language, etc).
> 
> As Eli explains, this is not always desirable.  And at other times it
> *is* desirable: the users's hook function can set/use features that are
> specific to one of the alternative, but they can also set/use features
> that are shared between the alternatives :-(

For sure, those that aren’t sharable should go into the not-shared hooks. I’m mainly saying that there should be a shared hook, so users _can_ share some of the configs.

> 
> Most users use only one of the alternatives, tho, so it's usually not
> a big problem (other than introducing incompatibilities when Emacs's
> defaults change from one alternative to another).

Keep in mind that when people try out tree-sitter modes, they are unlikely to just throw away their config for the old mode; also since tree-sitter and grammars aren’t the easiest to install, people working on multiple machines probably want both tree-sitter and no-tree-sitter modes configured and ready to go. So I think we’ll see a lot of people having config for both modes (me included).

> 
> It can be more annoying for `.dir-locals.el` per-mode settings.

And in general, any configuration that takes a major-mode symbol as the key. There are quite a few of them in Emacs. I think this is a big motivation for having multiple inheritance for derived-mode-p, and sharing a base mode.

> 
>>>    More generally, if there is a language X and a derived language Y,
>>>    and we have x-mode, x-ts-mode, y-mode, y-ts-mode, how should
>>>    inheritance of code and hooks works among them? y-mode probably
>>>    wants to inherit from x-mode, and y-ts-mode probably wants to
>>>    inherit hook (but not necessarily code [1]) from x-ts-mode.
> 
> `y-ts-mode` can explicitly run `y-mode-hook` (or `x-ts-mode-hook`).
> 
> We may more generally want to extend our notion of "derived" mode to
> allow "multiple inheritance".  For the actual activation code part,
> multiple inheritance is a hard problem that we probably don't want to
> tackle, but we can easily run several parent hooks, setup multiple
> keymap karents, and make `derived-mode-p` support multiple parents.

I agree that we don’t want multiple-inheritance for activation code. Also, as Juri pointed out, we can encapsulate code into functions and call functions in major mode body. Multiple-inheritance for hooks and maps has the potential disadvantage of being confusing. Right now it’s clear what hooks are run when a major mode turns on, but with multiple-inheritance it may not be. (I know I brought up the multiple-inheritance thing in the first place, I’m just writing whatever comes to my mind :-)

How do you setup multiple keymap parents? I thought a keymap can only have one parent?

Here’s another wild idea: we keep single-inheritance for define-derived-mode; major modes for the same language inherits from the same base mode; add a feature where xxx-base-mode is automatically defined when someone defines a major mode with xxx-base-mode as parent, so we don’t need to pre-define base-modes for every possible language; separate to major modes, we add a tag/category system, where modes can adopt multiple tags/categories, and a function like mode-has-category-p can work similarly to derived-mode-p.

> 
>>> 3. Unrelated to tree-sitter, here’s something I personally want:
>>>   it would be nice if every major mode can
>>>   have a hook that’s not run by its derived modes. Use case:
>>>   sage-mode inherits python-mode. I have eglot-ensure in
>>>   python-mode-hook but don’t want it when using sage-mode. Right now
>>>   I have to wrap eglot-ensure with a lambda function so that it only
>>>   runs when major-mode = python-mode.
>> 
>> What is wrong with that solution?  Except for the maybe minor
>> inconvenience of having to use a lambda-function, what different way
>> of doing this would you envision except telling define-derived-mode to
>> run some hook only under this-and-that condition?
> 
> While I tend to agree that it's not a big deal, I also agree that it's
> arguably cleaner if parent modes are kept "abstract", so rather than
> have `c++-mode` inherit from `c-mode`, you make them both inherit from
> a `c-base-mode`.  It tends to smell of overkill, tho.

Maybe defien-derived-mode can additionally define a function that only runs the major mode body but doesn’t setup anything that are autogenerated (eg, keymap, hooks, etc). This way another major mode is free to reuse other mode’s setup while not inheriting from that mode.

Yuan

Re: Extending define-derived-mode

[Stefan Monnier]

>> I don't see this as a big problem, actually (there are already several
>> mechanisms that can do that).  The question of how "user enables
>> xxx-ts-mode" is probably harder.
> Couldn’t they use major-mode-remap-alist?

Yes, that's one way.  With its pros and cons.

> For sure, those that aren’t sharable should go into the not-shared
> hooks.  I’m mainly saying that there should be a shared hook, so users
> _can_ share some of the configs.

Ideally, I agree, tho it's not terribly hard for the user to share code
between hooks, so it's not absolutely indispensable.

>> Most users use only one of the alternatives, tho, so it's usually not
>> a big problem (other than introducing incompatibilities when Emacs's
>> defaults change from one alternative to another).
>
> Keep in mind that when people try out tree-sitter modes, they are unlikely
> to just throw away their config for the old mode; also since tree-sitter and
> grammars aren’t the easiest to install, people working on multiple machines
> probably want both tree-sitter and no-tree-sitter modes configured and ready
> to go. So I think we’ll see a lot of people having config for both modes (me
> included).

Good point.

> And in general, any configuration that takes a major-mode symbol as
> the key.  There are quite a few of them in Emacs.  I think this is
> a big motivation for having multiple inheritance for derived-mode-p,
> and sharing a base mode.

I think the case for support of multiple inheritance in `derived-mode-p`
is fairly compelling, indeed.

> I agree that we don’t want multiple-inheritance for activation code.
> Also, as Juri pointed out, we can encapsulate code into functions and
> call functions in major mode body.  Multiple-inheritance for hooks and
> maps has the potential disadvantage of being confusing.  Right now
> it’s clear what hooks are run when a major mode turns on, but with
> multiple-inheritance it may not be.

Normally, `define-derived-mode makes sure that the docstring states it.

> How do you setup multiple keymap parents? I thought a keymap can only have one parent?

The single parent can be a composite map (i.e. using `make-composed-keymap`).

> Here’s another wild idea: we keep single-inheritance for
> define-derived-mode; major modes for the same language inherits from
> the same base mode; add a feature where xxx-base-mode is automatically
> defined when someone defines a major mode with xxx-base-mode as
> parent, so we don’t need to pre-define base-modes for every possible
> language;

Sounds hackish.  E.g. what would the `xxx-mode` docstring say about
which hooks are run?

> Maybe defien-derived-mode can additionally define a function that only runs
> the major mode body but doesn’t setup anything that are autogenerated (eg,
> keymap, hooks, etc). This way another major mode is free to reuse other
> mode’s setup while not inheriting from that mode.

Maybe we could offer a way to call "the mode's setup code", tho it's not
completely clear what that would do.  E.g. would that run `:after-hook`?
Oh wait, so maybe we should expose a major mode as a kind of "struct"
with accessors to get its keymap, hook, setup-code-function, :after-hook
function, ... maybe that could be useful.


        Stefan

Re: Extending define-derived-mode

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org,
>   mickey@masteringemacs.org,  theo@thornhill.no,  dgutov@yandex.ru
> Date: Thu, 01 Jun 2023 00:06:54 -0400
> 
> > Keep in mind that when people try out tree-sitter modes, they are unlikely
> > to just throw away their config for the old mode; also since tree-sitter and
> > grammars aren’t the easiest to install, people working on multiple machines
> > probably want both tree-sitter and no-tree-sitter modes configured and ready
> > to go. So I think we’ll see a lot of people having config for both modes (me
> > included).
> 
> Good point.

It's a good point, indeed, but with some mode pairs is very hard
(read: impossible) to achieve.  A notable example is c-mode and
c-ts-mode: the former has a lot of mode-specific commands and features
that cannot be used with the latter, because there's no equivalent
infrastructure that supports the same interfaces, and sometimes
because the feature makes no sense in a TS-based mode.  We try very
hard to use the same key bindings and variable names where it does
make sense, but the group of features where that is possible is very
small.  For example, all the enormous set of features we have in CC
mode around indentation and its customization cannot be "ported" to
c-ts-mode and c++-ts-mode, because the latter is built on completely
different analysis of the text.  Another example is the ad-hoc support
for some frequently-use macro names that CC mode has.

Re: Extending define-derived-mode

[Yuan Fu]

> On May 31, 2023, at 11:39 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org,
>>  mickey@masteringemacs.org,  theo@thornhill.no,  dgutov@yandex.ru
>> Date: Thu, 01 Jun 2023 00:06:54 -0400
>> 
>>> Keep in mind that when people try out tree-sitter modes, they are unlikely
>>> to just throw away their config for the old mode; also since tree-sitter and
>>> grammars aren’t the easiest to install, people working on multiple machines
>>> probably want both tree-sitter and no-tree-sitter modes configured and ready
>>> to go. So I think we’ll see a lot of people having config for both modes (me
>>> included).
>> 
>> Good point.
> 
> It's a good point, indeed, but with some mode pairs is very hard
> (read: impossible) to achieve.  A notable example is c-mode and
> c-ts-mode: the former has a lot of mode-specific commands and features
> that cannot be used with the latter, because there's no equivalent
> infrastructure that supports the same interfaces, and sometimes
> because the feature makes no sense in a TS-based mode.  We try very
> hard to use the same key bindings and variable names where it does
> make sense, but the group of features where that is possible is very
> small.  For example, all the enormous set of features we have in CC
> mode around indentation and its customization cannot be "ported" to
> c-ts-mode and c++-ts-mode, because the latter is built on completely
> different analysis of the text.  Another example is the ad-hoc support
> for some frequently-use macro names that CC mode has.

I fully agree. Note thought that I wasn’t saying ts and non-ts modes should accept the same set of configs. I know all-too-well that’s impossible (for the majority of modes). I was saying that sometimes there are configs that can be shared (enabling electric-quote-local-mode, for example), and it would be nice to put them in a single hook rather than duplicating the code in two hooks. 

Yuan

Re: Extending define-derived-mode

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Fri, 2 Jun 2023 00:50:07 -0700
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,
>  emacs-devel@gnu.org,
>  mickey@masteringemacs.org,
>  theo@thornhill.no,
>  dgutov@yandex.ru
> 
> > It's a good point, indeed, but with some mode pairs is very hard
> > (read: impossible) to achieve.  A notable example is c-mode and
> > c-ts-mode: the former has a lot of mode-specific commands and features
> > that cannot be used with the latter, because there's no equivalent
> > infrastructure that supports the same interfaces, and sometimes
> > because the feature makes no sense in a TS-based mode.  We try very
> > hard to use the same key bindings and variable names where it does
> > make sense, but the group of features where that is possible is very
> > small.  For example, all the enormous set of features we have in CC
> > mode around indentation and its customization cannot be "ported" to
> > c-ts-mode and c++-ts-mode, because the latter is built on completely
> > different analysis of the text.  Another example is the ad-hoc support
> > for some frequently-use macro names that CC mode has.
> 
> I fully agree. Note thought that I wasn’t saying ts and non-ts modes should accept the same set of configs. I know all-too-well that’s impossible (for the majority of modes). I was saying that sometimes there are configs that can be shared (enabling electric-quote-local-mode, for example), and it would be nice to put them in a single hook rather than duplicating the code in two hooks. 

For users to be able to share stuff like electric-quote-local-mode
we'd need to rewrite those supporting modes to allow that.  When one
of the two modes uses regexps and syntax tables, whereas the other
uses treesit-based parsers, this is not a trivial task.  I invite you
to audit the various electric modes we have and see how many of them
can be shared with minimum effort between non-TS and TS modes.

Re: Extending define-derived-mode

[Yuan Fu]

> On Jun 2, 2023, at 4:54 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Yuan Fu <casouri@gmail.com>
>> Date: Fri, 2 Jun 2023 00:50:07 -0700
>> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,
>> emacs-devel@gnu.org,
>> mickey@masteringemacs.org,
>> theo@thornhill.no,
>> dgutov@yandex.ru
>> 
>>> It's a good point, indeed, but with some mode pairs is very hard
>>> (read: impossible) to achieve.  A notable example is c-mode and
>>> c-ts-mode: the former has a lot of mode-specific commands and features
>>> that cannot be used with the latter, because there's no equivalent
>>> infrastructure that supports the same interfaces, and sometimes
>>> because the feature makes no sense in a TS-based mode.  We try very
>>> hard to use the same key bindings and variable names where it does
>>> make sense, but the group of features where that is possible is very
>>> small.  For example, all the enormous set of features we have in CC
>>> mode around indentation and its customization cannot be "ported" to
>>> c-ts-mode and c++-ts-mode, because the latter is built on completely
>>> different analysis of the text.  Another example is the ad-hoc support
>>> for some frequently-use macro names that CC mode has.
>> 
>> I fully agree. Note thought that I wasn’t saying ts and non-ts modes should accept the same set of configs. I know all-too-well that’s impossible (for the majority of modes). I was saying that sometimes there are configs that can be shared (enabling electric-quote-local-mode, for example), and it would be nice to put them in a single hook rather than duplicating the code in two hooks. 
> 
> For users to be able to share stuff like electric-quote-local-mode
> we'd need to rewrite those supporting modes to allow that.  When one
> of the two modes uses regexps and syntax tables, whereas the other
> uses treesit-based parsers, this is not a trivial task.  I invite you
> to audit the various electric modes we have and see how many of them
> can be shared with minimum effort between non-TS and TS modes.

I’m talking about sharing the sharable config. Electric modes already work the same in ts and non-ts modes: electric-pair-mode and electric-quote-mode inserts matching pairs and quotes and aren’t affected by tree-sitter, electric-indent-mode uses the standard indent-line-function which both ts and non-ts modes confront to. Is there any other electric modes?

Also, ts modes generally have the same syntax table as non-ts modes. So if some package uses the syntax table they are largely not affected either.

Yuan

Re: Extending define-derived-mode

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Mon, 5 Jun 2023 00:31:02 -0700
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,
>  emacs-devel <emacs-devel@gnu.org>,
>  Mickey Petersen <mickey@masteringemacs.org>,
>  Theodor Thornhill <theo@thornhill.no>,
>  dgutov@yandex.ru
> 
> > For users to be able to share stuff like electric-quote-local-mode
> > we'd need to rewrite those supporting modes to allow that.  When one
> > of the two modes uses regexps and syntax tables, whereas the other
> > uses treesit-based parsers, this is not a trivial task.  I invite you
> > to audit the various electric modes we have and see how many of them
> > can be shared with minimum effort between non-TS and TS modes.
> 
> I’m talking about sharing the sharable config. Electric modes already work the same in ts and non-ts modes: electric-pair-mode and electric-quote-mode inserts matching pairs and quotes and aren’t affected by tree-sitter, electric-indent-mode uses the standard indent-line-function which both ts and non-ts modes confront to. Is there any other electric modes?

For one prominent example, see the node "Electric Keys" in the CC Mode
manual, and the minor modes described in "Minor Modes" there.

> Also, ts modes generally have the same syntax table as non-ts modes. So if some package uses the syntax table they are largely not affected either.

But it makes little sense to me to have part of a mode use our syntax
tables, and another part use the results of parsing the code by a
parser library.  I can guarantee inconsistencies.

Re: Extending define-derived-mode

[Yuan Fu]

> On Jun 5, 2023, at 4:33 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Yuan Fu <casouri@gmail.com>
>> Date: Mon, 5 Jun 2023 00:31:02 -0700
>> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,
>> emacs-devel <emacs-devel@gnu.org>,
>> Mickey Petersen <mickey@masteringemacs.org>,
>> Theodor Thornhill <theo@thornhill.no>,
>> dgutov@yandex.ru
>> 
>>> For users to be able to share stuff like electric-quote-local-mode
>>> we'd need to rewrite those supporting modes to allow that.  When one
>>> of the two modes uses regexps and syntax tables, whereas the other
>>> uses treesit-based parsers, this is not a trivial task.  I invite you
>>> to audit the various electric modes we have and see how many of them
>>> can be shared with minimum effort between non-TS and TS modes.
>> 
>> I’m talking about sharing the sharable config. Electric modes already work the same in ts and non-ts modes: electric-pair-mode and electric-quote-mode inserts matching pairs and quotes and aren’t affected by tree-sitter, electric-indent-mode uses the standard indent-line-function which both ts and non-ts modes confront to. Is there any other electric modes?
> 
> For one prominent example, see the node "Electric Keys" in the CC Mode
> manual, and the minor modes described in "Minor Modes" there.

Ok, but they are CC mode’s features, not Emacs-wide features like electric-quote/pair/indent-mode.

> 
>> Also, ts modes generally have the same syntax table as non-ts modes. So if some package uses the syntax table they are largely not affected either.
> 
> But it makes little sense to me to have part of a mode use our syntax
> tables, and another part use the results of parsing the code by a
> parser library.  I can guarantee inconsistencies.

Not really used by the mode itself, but some generic third-party package. Having correct information in the syntax table is better than not. Also some ts modes enhances the syntax table information, making it more accurate, by adding appropriate syntax-table text property on some chars, like the angle brackets in C++ and Rust.

Yuan

Re: Extending define-derived-mode

[Yuan Fu]

> On May 31, 2023, at 9:06 PM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> 
>>> I don't see this as a big problem, actually (there are already several
>>> mechanisms that can do that).  The question of how "user enables
>>> xxx-ts-mode" is probably harder.
>> Couldn’t they use major-mode-remap-alist?
> 
> Yes, that's one way.  With its pros and cons.

What do you consider it’s cons? To me it’s more-or-less just a nicer auto-mode-alist without needing to fiddle with regular expression.

> 
>> For sure, those that aren’t sharable should go into the not-shared
>> hooks.  I’m mainly saying that there should be a shared hook, so users
>> _can_ share some of the configs.
> 
> Ideally, I agree, tho it's not terribly hard for the user to share code
> between hooks, so it's not absolutely indispensable.

Right, a nice-to-have.

> 
>>> Most users use only one of the alternatives, tho, so it's usually not
>>> a big problem (other than introducing incompatibilities when Emacs's
>>> defaults change from one alternative to another).
>> 
>> Keep in mind that when people try out tree-sitter modes, they are unlikely
>> to just throw away their config for the old mode; also since tree-sitter and
>> grammars aren’t the easiest to install, people working on multiple machines
>> probably want both tree-sitter and no-tree-sitter modes configured and ready
>> to go. So I think we’ll see a lot of people having config for both modes (me
>> included).
> 
> Good point.
> 
>> And in general, any configuration that takes a major-mode symbol as
>> the key.  There are quite a few of them in Emacs.  I think this is
>> a big motivation for having multiple inheritance for derived-mode-p,
>> and sharing a base mode.
> 
> I think the case for support of multiple inheritance in `derived-mode-p`
> is fairly compelling, indeed.
> 
>> I agree that we don’t want multiple-inheritance for activation code.
>> Also, as Juri pointed out, we can encapsulate code into functions and
>> call functions in major mode body.  Multiple-inheritance for hooks and
>> maps has the potential disadvantage of being confusing.  Right now
>> it’s clear what hooks are run when a major mode turns on, but with
>> multiple-inheritance it may not be.
> 
> Normally, `define-derived-mode makes sure that the docstring states it.

Makes sense, normally there won’t be more than 3 levels of inheritance anyway so it should be fine.

> 
>> How do you setup multiple keymap parents? I thought a keymap can only have one parent?
> 
> The single parent can be a composite map (i.e. using `make-composed-keymap`).

But IIUC that creates a new map instead of pointing to the parent maps, so any change in the parent map are not reflected in the child map, which is kind of the point of inheriting maps.

> 
>> Here’s another wild idea: we keep single-inheritance for
>> define-derived-mode; major modes for the same language inherits from
>> the same base mode; add a feature where xxx-base-mode is automatically
>> defined when someone defines a major mode with xxx-base-mode as
>> parent, so we don’t need to pre-define base-modes for every possible
>> language;
> 
> Sounds hackish.  E.g. what would the `xxx-mode` docstring say about
> which hooks are run?

xxx-base-mode-hook, and…uh, right, we need to specify which mode xxx-base-mode inherits, hmm.

Yuan

Re: Extending define-derived-mode

[Stefan Monnier]

>>>> I don't see this as a big problem, actually (there are already several
>>>> mechanisms that can do that).  The question of how "user enables
>>>> xxx-ts-mode" is probably harder.
>>> Couldn’t they use major-mode-remap-alist?
>> Yes, that's one way.  With its pros and cons.
> What do you consider it’s cons? To me it’s more-or-less just a nicer
> auto-mode-alist without needing to fiddle with regular expression.

It's "one mode at a time", whereas I'd expect some users might be
looking for a more "global" setting such as one that causes TS modes to
be used whenever possible (i.e. when there's a TS-using mode and
the corresponding grammar is installed).

>>> How do you setup multiple keymap parents? I thought a keymap can only have one parent?
>> The single parent can be a composite map (i.e. using `make-composed-keymap`).
>
> But IIUC that creates a new map instead of pointing to the parent maps, so
> any change in the parent map are not reflected in the child map, which is
> kind of the point of inheriting maps.

It creates a new map but it doesn't *copy* anything, it just keeps
references to the maps included in the composite map, so changes to the
parent maps *are* reflected.

IOW, we can implement XEmacs's multiple inheritance with

    (defun set-keymap-parents (keymap parents)
      (set-keymap-parent keymap
                         ;; Aka (cons 'keymap parents).
                         (make-composed-map parents)))

    (defun keymap-parents (keymap)
      (let ((parent (keymap-parent keymap)))
        (if (and (eq 'keymap (car-safe parent))
                 (proper-list-p parent)
                 (seq-every-p #'keymapp (cdr parent)))
            (cdr parent)
          (if parent (list parent)))))

>>> Here’s another wild idea: we keep single-inheritance for
>>> define-derived-mode; major modes for the same language inherits from
>>> the same base mode; add a feature where xxx-base-mode is automatically
>>> defined when someone defines a major mode with xxx-base-mode as
>>> parent, so we don’t need to pre-define base-modes for every possible
>>> language;
>> Sounds hackish.  E.g. what would the `xxx-mode` docstring say about
>> which hooks are run?
> xxx-base-mode-hook,

But IIUC you're suggesting that this hook would only exist "when someone
defines a major mode with xxx-base-mode as parent", so it's
dynamic/conditional.


        Stefan

Re: Extending define-derived-mode

[Yuan Fu]

> On Jun 2, 2023, at 9:46 AM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> 
>>>>> I don't see this as a big problem, actually (there are already several
>>>>> mechanisms that can do that).  The question of how "user enables
>>>>> xxx-ts-mode" is probably harder.
>>>> Couldn’t they use major-mode-remap-alist?
>>> Yes, that's one way.  With its pros and cons.
>> What do you consider it’s cons? To me it’s more-or-less just a nicer
>> auto-mode-alist without needing to fiddle with regular expression.
> 
> It's "one mode at a time", whereas I'd expect some users might be
> looking for a more "global" setting such as one that causes TS modes to
> be used whenever possible (i.e. when there's a TS-using mode and
> the corresponding grammar is installed).
> 
>>>> How do you setup multiple keymap parents? I thought a keymap can only have one parent?
>>> The single parent can be a composite map (i.e. using `make-composed-keymap`).
>> 
>> But IIUC that creates a new map instead of pointing to the parent maps, so
>> any change in the parent map are not reflected in the child map, which is
>> kind of the point of inheriting maps.
> 
> It creates a new map but it doesn't *copy* anything, it just keeps
> references to the maps included in the composite map, so changes to the
> parent maps *are* reflected.
> 
> IOW, we can implement XEmacs's multiple inheritance with
> 
>    (defun set-keymap-parents (keymap parents)
>      (set-keymap-parent keymap
>                         ;; Aka (cons 'keymap parents).
>                         (make-composed-map parents)))
> 
>    (defun keymap-parents (keymap)
>      (let ((parent (keymap-parent keymap)))
>        (if (and (eq 'keymap (car-safe parent))
>                 (proper-list-p parent)
>                 (seq-every-p #'keymapp (cdr parent)))
>            (cdr parent)
>          (if parent (list parent)))))

Oh great! I didn’t know that.

> 
>>>> Here’s another wild idea: we keep single-inheritance for
>>>> define-derived-mode; major modes for the same language inherits from
>>>> the same base mode; add a feature where xxx-base-mode is automatically
>>>> defined when someone defines a major mode with xxx-base-mode as
>>>> parent, so we don’t need to pre-define base-modes for every possible
>>>> language;
>>> Sounds hackish.  E.g. what would the `xxx-mode` docstring say about
>>> which hooks are run?
>> xxx-base-mode-hook,
> 
> But IIUC you're suggesting that this hook would only exist "when someone
> defines a major mode with xxx-base-mode as parent", so it's
> dynamic/conditional.

IIUC you can add functions to a hook regardless whether it’s defined or not, so it should be fine?


BTW, I want to clarify one thing: how are we feeling about base modes? Are they fine? Are they staying?

Yuan

Re: Extending define-derived-mode

[Stefan Monnier]

>> But IIUC you're suggesting that this hook would only exist "when someone
>> defines a major mode with xxx-base-mode as parent", so it's
>> dynamic/conditional.
> IIUC you can add functions to a hook regardless whether it’s defined
> or not, so it should be fine?

The question is not whether the variable is `boundp` or not but whether
it's run or not.  I think making it dynamic/conditional on the existence
of some other major mode that refers to it will likely cause surprises.

> BTW, I want to clarify one thing: how are we feeling about base modes?
> Are they fine?  Are they staying?

Good question.
To me they're halfway between The Right Thing and Overkill :-(


        Stefan

Re: Extending define-derived-mode

[Yuan Fu]

> 
>> 2. Inheritance:
>> 
>> 2.1 For xxx-mode and xxx-ts-mode, there should be shared hook. More
>>    generally, we want to be able to have a shared hook for similar
>>    modes (same language, similar language, etc).
>> 
>> 2.2 For xxx-mode and xxx-ts-mode, they should be able to share some
>>    setup, and diverge on some other.
>> 
>> 2.3 Right now we have a little inconsistency regarding hook
>>    inheritance. js-json-mode inherits from js-mode, json-ts-mode
>>    inherits from js-ts-mode. So there isn’t a shared hook that runs
>>    in both js-json-mode and json-ts-mode.
>> 
>>    More generally, if there is a language X and a derived language Y,
>>    and we have x-mode, x-ts-mode, y-mode, y-ts-mode, how should
>>    inheritance of code and hooks works among them? y-mode probably
>>    wants to inherit from x-mode, and y-ts-mode probably wants to
>>    inherit hook (but not necessarily code [1]) from x-ts-mode.
> 
> This is not simple in practice.  Ask yourself why don't the
> FOO-ts-mode's simply run the (existing) hooks of FOO-mode?  Look at
> the code, and you will understand: the stuff people put in these hooks
> usually customizes aspects that are very different in each of the
> modes: font-lock, indentation, syntax tables, supporting "electric"
> sub-modes, styles of comments and other things, etc.  The few places
> where the new TS modes tried to run the same hooks and share most of
> the initialization code (for example, bash-ts-mode and sh-mode,
> python-ts-mode and python-mode) introduced subtle bugs which were only
> recently fixed.  One problem with inheriting from the same base mode
> is that doing initialization there could be problematic because the
> mode stuff that gets automatically instantiated and initialized by
> define-derived-mode, such as the mode's map and variable, is not yet
> set up when the base mode's initialization runs, so you cannot do some
> of the things we are used to do there, like setting up the menu-bar
> menus.
> 
> So it would seem that, for efficient and convenient facilities of this
> kind, we need some way of defining stuff in the base mode, which will
> be actually run later, when the child mode is set up.

I don’t entirely understand the example. Say I have a xxx-base-mode, and I want to setup menu-bar menus in it to be shared by xxx-mode and xxx-ts-mode. What is the problem that I’m gonna run into? Couldn’t you just use the base-mode’s keymap?

Yuan

Re: Extending define-derived-mode

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Wed, 31 May 2023 13:48:56 -0700
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,
>  emacs-devel@gnu.org,
>  mickey@masteringemacs.org,
>  theo@thornhill.no,
>  dgutov@yandex.ru
> 
> > This is not simple in practice.  Ask yourself why don't the
> > FOO-ts-mode's simply run the (existing) hooks of FOO-mode?  Look at
> > the code, and you will understand: the stuff people put in these hooks
> > usually customizes aspects that are very different in each of the
> > modes: font-lock, indentation, syntax tables, supporting "electric"
> > sub-modes, styles of comments and other things, etc.  The few places
> > where the new TS modes tried to run the same hooks and share most of
> > the initialization code (for example, bash-ts-mode and sh-mode,
> > python-ts-mode and python-mode) introduced subtle bugs which were only
> > recently fixed.  One problem with inheriting from the same base mode
> > is that doing initialization there could be problematic because the
> > mode stuff that gets automatically instantiated and initialized by
> > define-derived-mode, such as the mode's map and variable, is not yet
> > set up when the base mode's initialization runs, so you cannot do some
> > of the things we are used to do there, like setting up the menu-bar
> > menus.
> > 
> > So it would seem that, for efficient and convenient facilities of this
> > kind, we need some way of defining stuff in the base mode, which will
> > be actually run later, when the child mode is set up.
> 
> I don’t entirely understand the example. Say I have a xxx-base-mode, and I want to setup menu-bar menus in it to be shared by xxx-mode and xxx-ts-mode. What is the problem that I’m gonna run into? Couldn’t you just use the base-mode’s keymap?

A base mode usually doesn't have a map, only the actually used derived
modes do.

Re: Extending define-derived-mode

[Yuan Fu]

> On May 31, 2023, at 10:47 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Yuan Fu <casouri@gmail.com>
>> Date: Wed, 31 May 2023 13:48:56 -0700
>> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,
>> emacs-devel@gnu.org,
>> mickey@masteringemacs.org,
>> theo@thornhill.no,
>> dgutov@yandex.ru
>> 
>>> This is not simple in practice.  Ask yourself why don't the
>>> FOO-ts-mode's simply run the (existing) hooks of FOO-mode?  Look at
>>> the code, and you will understand: the stuff people put in these hooks
>>> usually customizes aspects that are very different in each of the
>>> modes: font-lock, indentation, syntax tables, supporting "electric"
>>> sub-modes, styles of comments and other things, etc.  The few places
>>> where the new TS modes tried to run the same hooks and share most of
>>> the initialization code (for example, bash-ts-mode and sh-mode,
>>> python-ts-mode and python-mode) introduced subtle bugs which were only
>>> recently fixed.  One problem with inheriting from the same base mode
>>> is that doing initialization there could be problematic because the
>>> mode stuff that gets automatically instantiated and initialized by
>>> define-derived-mode, such as the mode's map and variable, is not yet
>>> set up when the base mode's initialization runs, so you cannot do some
>>> of the things we are used to do there, like setting up the menu-bar
>>> menus.
>>> 
>>> So it would seem that, for efficient and convenient facilities of this
>>> kind, we need some way of defining stuff in the base mode, which will
>>> be actually run later, when the child mode is set up.
>> 
>> I don’t entirely understand the example. Say I have a xxx-base-mode, and I want to setup menu-bar menus in it to be shared by xxx-mode and xxx-ts-mode. What is the problem that I’m gonna run into? Couldn’t you just use the base-mode’s keymap?
> 
> A base mode usually doesn't have a map, only the actually used derived
> modes do.

We can create a map for the base mode, and the child mode’s map will inherit from it.

Yuan

Re: Extending define-derived-mode

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Fri, 2 Jun 2023 00:45:31 -0700
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,
>  emacs-devel@gnu.org,
>  mickey@masteringemacs.org,
>  theo@thornhill.no,
>  dgutov@yandex.ru
> 
> >> I don’t entirely understand the example. Say I have a xxx-base-mode, and I want to setup menu-bar menus in it to be shared by xxx-mode and xxx-ts-mode. What is the problem that I’m gonna run into? Couldn’t you just use the base-mode’s keymap?
> > 
> > A base mode usually doesn't have a map, only the actually used derived
> > modes do.
> 
> We can create a map for the base mode, and the child mode’s map will inherit from it.

We can, but it won't solve the basic problem.

This sub-thread comes from me explaining why this presents a problem
for, e.g., defining menu-bar items, remember?  So, if we examine the
issue in that context, you will see that creating a map for the base
mode has the following downsides:

 . It can only have bindings for the commands that are identical in
   all child modes, something that normally isn't the case.  For
   example "C-c C-q" is bound in both c-mode and c-ts-mode, but to
   different commands.
 . The menu items you can define in the base mode can therefore only
   reference commands bound in the base mode's map, which once again
   means only identical commands.  For example, compare the menu items
   in the "C/C++" menu-bar menu of c-ts-mode with those in the "C"
   menu of c-mode, and you will see what I mean.

This is why it makes sense to define the bindings and menus in the
child modes, not in the common base mode: in many cases the same key
sequences and menu items will be bound to different commands.

Likewise with buffer-local variables: in many cases they will have
different values for different modes, and sometimes even different
names, although their functions and effects could be very similar or
identical.

Bottom line: based on our experience to this point, it sounds like at
least in some cases the differences between the non-TS mode and TS
mode supporting the same language is significant enough to make the
idea of inheriting from a common base mode problematic if not
impractical.

Re: Extending define-derived-mode

[Juri Linkov]

>    I don’t have a clear vision right now, perhaps we want
>    multiple inheritance for hooks, perhaps we can simply allow
>    define-derived-mode to specify additional hooks to run, and create
>    some language hooks that are shared by every mode serving that language.
>    I’d like to hear your thoughts.

A well-known alternative to multiple inheritance would be to use mixins.
Implementation-wise this means extracting common parts to separate functions,
then calling e.g. 'js-base-setup' from both 'js-mode' and 'js-ts-mode'.

Re: Extending define-derived-mode

[Philip Kaludercic]

Yuan Fu <casouri@gmail.com> writes:

> When we were adding tree-sitter modes a couple of month ago, it was
> clear that the current major mode model needs some upgrade, I’d like
> to discuss the things we need and how can we address them.

Perhaps this is a stupid suggestion (since I am permanently behind on
developments wrt. tree-sitter), but might it be that the notions of
"major modes" just aren't strong enough of an abstraction to talk about
languages?  Just like we have user options that have a relation to
regular variables, wouldn't it make sense to provide something like
"language modes" that would provide a higher-level and a more
specialised vocabulary.

This might also tie into the issue of inconsistent bindings for
semantically "analogous" operations between major modes (compiling,
opening a REPL, requesting documentation, etc.) or the issues of
duplicate configurations options that all raise the barrier when trying
out a new major mode or switching between languages.