Need for "-ts-mode" modes

Need for "-ts-mode" modes

[Philip Kaludercic]

The issue has been discussed before, but I failed to understand the
point of duplicating a lot of modes, or creating new modes that depend
on tree-sitter and that don't work or even try to provide some fall-back
if the library is not available.

Or the re-phrase the question, why can't tree-sitter support be
implemented by extending `define-derived-mode', ideally in such a way
that can be translated to some kind of font-lock rules for basic syntax
if the library is not installed.

Re: Need for "-ts-mode" modes

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Date: Thu, 29 Dec 2022 17:08:09 +0000
> 
> The issue has been discussed before, but I failed to understand the
> point of duplicating a lot of modes

Because this was the simplest way, that created no complications,
neither to users nor to the code, and could be added in
backward-compatible ways.

> or creating new modes that depend
> on tree-sitter and that don't work or even try to provide some fall-back
> if the library is not available.

I fail to see what's wrong with that.  It is entirely legitimate to
have a major mode which depends on an optional feature and doesn't
work without it.  Besides, we've lived this far without major modes
for those languages, so adding a tree-sitter based one cannot possibly
do any harm, it can only make things better for Emacs.

> Or the re-phrase the question, why can't tree-sitter support be
> implemented by extending `define-derived-mode', ideally in such a way
> that can be translated to some kind of font-lock rules for basic syntax
> if the library is not installed.

Because it doesn't work.  Tree-sitter supported modes are not derived
modes, they are completely different modes with different settings
which make no sense when tree-sitter is not used.

Re: Need for "-ts-mode" modes

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 2881 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Date: Thu, 29 Dec 2022 17:08:09 +0000
>> 
>> The issue has been discussed before, but I failed to understand the
>> point of duplicating a lot of modes
>
> Because this was the simplest way, that created no complications,
> neither to users nor to the code, and could be added in
> backward-compatible ways.

What backwards-compatibility are we taking about here?

>> or creating new modes that depend
>> on tree-sitter and that don't work or even try to provide some fall-back
>> if the library is not available.
>
> I fail to see what's wrong with that.  It is entirely legitimate to
> have a major mode which depends on an optional feature and doesn't
> work without it.  

Naturally, if there is no way around this.  My argument is that there
is.  What prompted me to ask this question was when I just opened a
Dockerfile and a warning popped up, telling me that I was missing the
right libraries.  I took a peak inside of dockerfile-ts-mode, and found
the following

--8<---------------cut here---------------start------------->8---
(defvar dockerfile-ts-mode--keywords
  '("ADD" "ARG" "AS" "CMD" "COPY" "CROSS_BUILD" "ENTRYPOINT" "ENV"
    "EXPOSE" "FROM" "HEALTHCHECK" "LABEL" "MAINTAINER" "ONBUILD" "RUN"
    "SHELL" "STOPSIGNAL" "USER" "VOLUME" "WORKDIR")
  "Dockerfile keywords for tree-sitter font-locking.")
--8<---------------cut here---------------end--------------->8---

For me, using these keywords would provide all the "functionality" I
could need from a Dockerfile.  Sure, if I have the parser installed use
that, but otherwise it would be preferable if instead of giving up with
a warning the keyword list is extracted from
`dockerfile-ts-mode--font-lock-settings' and converted into something
that is useful even if I don't have tree-sitter configured.  And this
doesn't just apply to this mode.

>                   Besides, we've lived this far without major modes
> for those languages, so adding a tree-sitter based one cannot possibly
> do any harm, it can only make things better for Emacs.

With the exception that a warning is generated, popping up a new window,
which is distracting.

>> Or the re-phrase the question, why can't tree-sitter support be
>> implemented by extending `define-derived-mode', ideally in such a way
>> that can be translated to some kind of font-lock rules for basic syntax
>> if the library is not installed.
>
> Because it doesn't work.  Tree-sitter supported modes are not derived
> modes, they are completely different modes with different settings
> which make no sense when tree-sitter is not used.

I am not talking about deriving foo-ts-mode from foo-mode, but adding a
keyword to `define-derived-mode' that would generate code to try to make
use of tree-sitter if available, and fall back to traditional font
locking if not.


[-- Attachment #2: Type: text/plain, Size: 1978 bytes --]

diff --git a/lisp/progmodes/dockerfile-ts-mode.el b/lisp/progmodes/dockerfile-ts-mode.el
index 40d90cc2df..99fea26c94 100644
--- a/lisp/progmodes/dockerfile-ts-mode.el
+++ b/lisp/progmodes/dockerfile-ts-mode.el
@@ -141,36 +141,24 @@ dockerfile-ts-mode--imenu-1
 ;;;###autoload
 (define-derived-mode dockerfile-ts-mode prog-mode "Dockerfile"
   "Major mode for editing Dockerfiles, powered by tree-sitter."
+  :tree-sitter-settings treesit-font-lock-settings
+  :tree-sitter-features '((comment)
+                          (keyword string)
+                          (image-spec number)
+                          (bracket delimiter error operator))
+  :tree-sitter-indent dockerfile-ts-mode--indent-rules
   :group 'dockerfile
   :syntax-table dockerfile-ts-mode--syntax-table
 
-  (when (treesit-ready-p 'dockerfile)
-    (treesit-parser-create 'dockerfile)
-
-    ;; Comments.
-    (setq-local comment-start "# ")
-    (setq-local comment-end "")
-    (setq-local comment-start-skip (rx "#" (* (syntax whitespace))))
-
-    ;; Imenu.
-    (setq-local imenu-create-index-function
-                #'dockerfile-ts-mode--imenu)
-    (setq-local which-func-functions nil)
-
-    ;; Indent.
-    (setq-local treesit-simple-indent-rules
-                dockerfile-ts-mode--indent-rules)
-
-    ;; Font-lock.
-    (setq-local treesit-font-lock-settings
-                dockerfile-ts-mode--font-lock-settings)
-    (setq-local treesit-font-lock-feature-list
-                '((comment)
-                  (keyword string)
-                  (image-spec number)
-                  (bracket delimiter error operator)))
-
-    (treesit-major-mode-setup)))
+  ;; Comments.
+  (setq-local comment-start "# ")
+  (setq-local comment-end "")
+  (setq-local comment-start-skip (rx "#" (* (syntax whitespace))))
+
+  ;; Imenu
+  (setq-local imenu-create-index-function
+              #'dockerfile-ts-mode--imenu)
+  (setq-local which-func-functions nil))
 
 (provide 'dockerfile-ts-mode)
 

[-- Attachment #3: Type: text/plain, Size: 366 bytes --]


Perhaps it would be even better to bundle all the tree-sitter related
options into a single object, define it as a defvar and pass that via a
keyword.

What I want to know is, if this is viable (as I plan to find out in the
next few days), would it be possible to change the current route before
29.1 is released, so as to avoid committing to the current strategy.

Re: Need for "-ts-mode" modes

[Yuan Fu]

> 
>>> or creating new modes that depend
>>> on tree-sitter and that don't work or even try to provide some fall-back
>>> if the library is not available.
>> 
>> I fail to see what's wrong with that.  It is entirely legitimate to
>> have a major mode which depends on an optional feature and doesn't
>> work without it.  
> 
> Naturally, if there is no way around this.  My argument is that there
> is.  What prompted me to ask this question was when I just opened a
> Dockerfile and a warning popped up, telling me that I was missing the
> right libraries. 

I think we should not enable docker-ts-mode in auto-mode-alist if (treesit-language-available-p 'dockerfile) is nil. This way you don’t get the warning. In Emacs 30 we can upgrade the auto-mode facility to use different modes based on some condition.

Yuan

Re: Need for "-ts-mode" modes

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Thu, 29 Dec 2022 11:27:41 -0800
> Cc: Eli Zaretskii <eliz@gnu.org>,
>  emacs-devel@gnu.org
> 
> I think we should not enable docker-ts-mode in auto-mode-alist if (treesit-language-available-p 'dockerfile) is nil.

I don't see how this could be accomplished, since whether three-sitter
is available for a language is only known at run time, and
auto-mode-alist is in files.el, which is preloaded.

I also don't see any disaster in having an error message in a new mode
which we didn't have before.  It is not a regression.

Re: Need for "-ts-mode" modes

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > I think we should not enable docker-ts-mode in auto-mode-alist if (treesit-language-available-p 'dockerfile) is nil.

  > I don't see how this could be accomplished, since whether three-sitter
  > is available for a language is only known at run time, and
  > auto-mode-alist is in files.el, which is preloaded.

It would be possible for `docker-ts-mode' to check whether tree-sitter
is available for the Docker language, and DTRT if not.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Need for "-ts-mode" modes

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Thu, 29 Dec 2022 23:05:24 -0500
> 
>   > > I think we should not enable docker-ts-mode in auto-mode-alist if (treesit-language-available-p 'dockerfile) is nil.
> 
>   > I don't see how this could be accomplished, since whether three-sitter
>   > is available for a language is only known at run time, and
>   > auto-mode-alist is in files.el, which is preloaded.
> 
> It would be possible for `docker-ts-mode' to check whether tree-sitter
> is available for the Docker language, and DTRT if not.

It does: it displays a warning (when you visit a file that should
trigger this mode).  Like this:

 ■  Warning (treesit): Cannot activate tree-sitter, because tree-sitter library\
 is not compiled with Emacs

Re: Need for "-ts-mode" modes

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: emacs-devel@gnu.org
> Date: Thu, 29 Dec 2022 18:26:07 +0000
> 
> > Because this was the simplest way, that created no complications,
> > neither to users nor to the code, and could be added in
> > backward-compatible ways.
> 
> What backwards-compatibility are we taking about here?

Let the users decide whether and when they want to turn this on or
off, and keep their init files and use habits working.

> > I fail to see what's wrong with that.  It is entirely legitimate to
> > have a major mode which depends on an optional feature and doesn't
> > work without it.  
> 
> Naturally, if there is no way around this.  My argument is that there
> is.  What prompted me to ask this question was when I just opened a
> Dockerfile and a warning popped up, telling me that I was missing the
> right libraries.  I took a peak inside of dockerfile-ts-mode, and found
> the following
> 
> --8<---------------cut here---------------start------------->8---
> (defvar dockerfile-ts-mode--keywords
>   '("ADD" "ARG" "AS" "CMD" "COPY" "CROSS_BUILD" "ENTRYPOINT" "ENV"
>     "EXPOSE" "FROM" "HEALTHCHECK" "LABEL" "MAINTAINER" "ONBUILD" "RUN"
>     "SHELL" "STOPSIGNAL" "USER" "VOLUME" "WORKDIR")
>   "Dockerfile keywords for tree-sitter font-locking.")
> --8<---------------cut here---------------end--------------->8---
> 
> For me, using these keywords would provide all the "functionality" I
> could need from a Dockerfile.

If you want to provide a dockerfile mode without tree-sitter, please
do.  I'm saying that having just a TS-supported mode is better than
not having any mode.

> With the exception that a warning is generated, popping up a new window,
> which is distracting.

Not a disaster, IMO.

> >> Or the re-phrase the question, why can't tree-sitter support be
> >> implemented by extending `define-derived-mode', ideally in such a way
> >> that can be translated to some kind of font-lock rules for basic syntax
> >> if the library is not installed.
> >
> > Because it doesn't work.  Tree-sitter supported modes are not derived
> > modes, they are completely different modes with different settings
> > which make no sense when tree-sitter is not used.
> 
> I am not talking about deriving foo-ts-mode from foo-mode, but adding a
> keyword to `define-derived-mode' that would generate code to try to make
> use of tree-sitter if available, and fall back to traditional font
> locking if not.

The *-ts mode provide more than just fontifications.  They provide
indentation, navigation, and Imenu support.  And they do that in ways
that are radically different from the "traditional" ways we did this
stuff, with data structures and algorithms that are very different.
If you think you know how to extend define-derived-mode so that we
could treat tree-sitter as some enable-FOO style variable, please show
the code, and let's see whether it's clean enough for our palate.  But
please do this for a serious major mode, like Python or C/C++, not for
dockerfiles, because that's where we bumped into problems that
eventually led to what we have now.  We did try to do something like
that before.

> Perhaps it would be even better to bundle all the tree-sitter related
> options into a single object, define it as a defvar and pass that via a
> keyword.

Feel free to propose how to do that.  We tried several different ideas
and each one of them either had problems or was not cleaner than what
we have.

> What I want to know is, if this is viable (as I plan to find out in the
> next few days), would it be possible to change the current route before
> 29.1 is released, so as to avoid committing to the current strategy.

You can try.  I would like to start a full feature freeze in a day or
two, so I'm not sure you will have enough time.  And it isn't like we
didn't try various approaches during the past two months, so frankly I
don't think that a better way even exists.  But if you come up with
some very bright idea, who knows?

Re: Need for "-ts-mode" modes

[Eli Zaretskii]

> Date: Thu, 29 Dec 2022 21:57:12 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> > What I want to know is, if this is viable (as I plan to find out in the
> > next few days), would it be possible to change the current route before
> > 29.1 is released, so as to avoid committing to the current strategy.
> 
> You can try.  I would like to start a full feature freeze in a day or
> two, so I'm not sure you will have enough time.  And it isn't like we
> didn't try various approaches during the past two months, so frankly I
> don't think that a better way even exists.  But if you come up with
> some very bright idea, who knows?

And, btw, we are not really "committed to the current strategy".  What
the FOO-ts-mode commands do is entirely up to us, and if we find a way
of doing that as some kind of minor customization that is part of the
"traditional" FOO-mode, we can easily do it in the future.  So I
really don't see a need to hurry with alternative strategies, as we
don't yet have enough data to validate any strategies for this
purpose.  What we have now is our best shot, as the alternatives we
considered, including something similar to what you propose, each one
had either significant disadvantages or not enough advantages.

Unifying "foo-mode"s and "foo-ts-mode"s

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 2605 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

> You can try.  I would like to start a full feature freeze in a day or
> two, so I'm not sure you will have enough time.  And it isn't like we
> didn't try various approaches during the past two months, so frankly I
> don't think that a better way even exists.  But if you come up with
> some very bright idea, who knows?

I have attached a sketch of my proposal with support for Python.
Instead of a separate python-ts-mode, we regulate tree-sitter support
using a user option `treesit-enabled-modes'.  It can either be a list

--8<---------------cut here---------------start------------->8---
  (setq treesit-enabled-modes '(python-mode c-mode))
--8<---------------cut here---------------end--------------->8---

or generally enable tree-sitter

--8<---------------cut here---------------start------------->8---
  (setq treesit-enabled-modes t)
--8<---------------cut here---------------end--------------->8---

All a major modes has to do is pass a parser configuration

--8<---------------cut here---------------start------------->8---
  (define-derived-mode python-mode prog-mode "Python"
    "Major mode for editing Python files.

  \\{python-mode-map}"
    :syntax-table python-mode-syntax-table
    :parser-conf python-mode--treesit-conf
   ...
--8<---------------cut here---------------end--------------->8---

that expands to

--8<---------------cut here---------------start------------->8---
  (when-let
      ((conf python-mode--treesit-conf)
       ((cond
         ((listp treesit-enabled-modes)
          (memq 'python-mode treesit-enabled-modes))
         ((eq treesit-enabled-modes t))))
       ((treesit-ready-p
         (nth 0 conf)))
       (parser
        (treesit-parser-create
         (nth 0 conf))))
    (setq-local treesit-font-lock-feature-list
                (nth 1 conf)
                treesit-font-lock-settings
                (nth 2 conf)
                treesit-defun-name-function
                (nth 3 conf)
                treesit-defun-type-regexp
                (nth 4 conf)
                imenu-create-index-function
                (nth 5 conf))
    (treesit-major-mode-setup))
--8<---------------cut here---------------end--------------->8---

at *the end* of the major mode definition.  Note that if no parser
configuration was parsed, the entire expression is byte-compiled away,
so there is no run-time overhead for other modes.

The parser configuration is currently a list but if might as well be a
vector, a structure or anything else.  This is just a rushed proposal to
meet the deadline.  How does it look like?


[-- Attachment #2: Type: text/plain, Size: 15287 bytes --]

diff --git a/lisp/emacs-lisp/derived.el b/lisp/emacs-lisp/derived.el
index 260fc3bf47..d6a1f4af00 100644
--- a/lisp/emacs-lisp/derived.el
+++ b/lisp/emacs-lisp/derived.el
@@ -143,6 +143,8 @@ define-derived-mode
            :interactive BOOLEAN
                    Whether the derived mode should be `interactive' or not.
                    The default is t.
+           :parser-conf CONF
+                   A tree-sitter parser configuration.
 
 BODY:      forms to execute just before running the
            hooks for the new mode.  Do not use `interactive' here.
@@ -192,6 +194,7 @@ define-derived-mode
 	(hook (derived-mode-hook-name child))
 	(group nil)
         (interactive t)
+        (parser-conf nil)
         (after-hook nil))
 
     ;; Process the keyword args.
@@ -202,6 +205,7 @@ define-derived-mode
 	(:syntax-table (setq syntax (pop body)) (setq declare-syntax nil))
         (:after-hook (setq after-hook (pop body)))
         (:interactive (setq interactive (pop body)))
+        (:parser-conf (setq parser-conf (pop body)))
 	(_ (pop body))))
 
     (setq docstring (derived-mode-make-docstring
@@ -285,7 +289,22 @@ define-derived-mode
 	  ,(when abbrev `(setq local-abbrev-table ,abbrev))
 					; Splice in the body (if any).
 	  ,@body
-	  )
+
+          ;; Activate tree-sitter if requested and available.
+          (when-let ((conf ,parser-conf)
+                      ((cond
+                        ((listp treesit-enabled-modes)
+                         (memq ',child treesit-enabled-modes))
+                        ((eq treesit-enabled-modes t))))
+                      ((treesit-ready-p (nth 0 conf)))
+                      (parser (treesit-parser-create (nth 0 conf))))
+             (setq-local
+              treesit-font-lock-feature-list (nth 1 conf)
+              treesit-font-lock-settings     (nth 2 conf)
+              treesit-defun-name-function    (nth 3 conf)
+              treesit-defun-type-regexp      (nth 4 conf)
+              imenu-create-index-function    (nth 5 conf))
+             (treesit-major-mode-setup)))
 	 ,@(when after-hook
 	     `((push (lambda () ,after-hook) delayed-after-hook-functions)))
 	 ;; Run the hooks (and delayed-after-hook-functions), if any.
diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index 9a6f807f4f..5ad90e6d71 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -405,9 +405,6 @@ python-mode-map
     map)
   "Keymap for `python-mode'.")
 
-(defvar python-ts-mode-map (copy-keymap python-mode-map)
-  "Keymap for `(copy-keymap python-mode-map)'.")
-
 \f
 ;;; Python specialized rx
 
@@ -1072,124 +1069,6 @@ python--treesit-fontify-string
     (treesit-fontify-with-override
      string-beg string-end face override start end)))
 
-(defvar python--treesit-settings
-  (treesit-font-lock-rules
-   :feature 'comment
-   :language 'python
-   '((comment) @font-lock-comment-face)
-
-   :feature 'string
-   :language 'python
-   '((string) @python--treesit-fontify-string)
-
-   :feature 'string-interpolation
-   :language 'python
-   :override t
-   '((interpolation (identifier) @font-lock-variable-name-face))
-
-   :feature 'definition
-   :language 'python
-   '((function_definition
-      name: (identifier) @font-lock-function-name-face)
-     (class_definition
-      name: (identifier) @font-lock-type-face))
-
-   :feature 'function
-   :language 'python
-   '((function_definition
-      name: (identifier) @font-lock-function-name-face)
-     (call function: (identifier) @font-lock-function-name-face)
-     (call function: (attribute
-                      attribute: (identifier) @font-lock-function-name-face)))
-
-   :feature 'keyword
-   :language 'python
-   `([,@python--treesit-keywords] @font-lock-keyword-face
-     ((identifier) @font-lock-keyword-face
-      (:match "^self$" @font-lock-keyword-face)))
-
-   :feature 'builtin
-   :language 'python
-   `(((identifier) @font-lock-builtin-face
-      (:match ,(rx-to-string
-                `(seq bol
-                      (or ,@python--treesit-builtins
-                          ,@python--treesit-special-attributes)
-                      eol))
-              @font-lock-builtin-face)))
-
-   :feature 'constant
-   :language 'python
-   '([(true) (false) (none)] @font-lock-constant-face)
-
-   :feature 'assignment
-   :language 'python
-   `(;; Variable names and LHS.
-     (assignment left: (identifier)
-                 @font-lock-variable-name-face)
-     (assignment left: (attribute
-                        attribute: (identifier)
-                        @font-lock-property-face))
-     (pattern_list (identifier)
-                   @font-lock-variable-name-face)
-     (tuple_pattern (identifier)
-                    @font-lock-variable-name-face)
-     (list_pattern (identifier)
-                   @font-lock-variable-name-face)
-     (list_splat_pattern (identifier)
-                         @font-lock-variable-name-face))
-
-   :feature 'decorator
-   :language 'python
-   '((decorator "@" @font-lock-type-face)
-     (decorator (call function: (identifier) @font-lock-type-face))
-     (decorator (identifier) @font-lock-type-face))
-
-   :feature 'type
-   :language 'python
-   `(((identifier) @font-lock-type-face
-      (:match ,(rx-to-string
-                `(seq bol (or ,@python--treesit-exceptions)
-                      eol))
-              @font-lock-type-face))
-     (type (identifier) @font-lock-type-face))
-
-   :feature 'escape-sequence
-   :language 'python
-   :override t
-   '((escape_sequence) @font-lock-escape-face)
-
-   :feature 'number
-   :language 'python
-   '([(integer) (float)] @font-lock-number-face)
-
-   :feature 'property
-   :language 'python
-   '((attribute
-      attribute: (identifier) @font-lock-property-face)
-     (class_definition
-      body: (block
-             (expression_statement
-              (assignment left:
-                          (identifier) @font-lock-property-face)))))
-
-   :feature 'operator
-   :language 'python
-   `([,@python--treesit-operators] @font-lock-operator-face)
-
-   :feature 'bracket
-   :language 'python
-   '(["(" ")" "[" "]" "{" "}"] @font-lock-bracket-face)
-
-   :feature 'delimiter
-   :language 'python
-   '(["," "." ":" ";" (ellipsis)] @font-lock-delimiter-face)
-
-   :feature 'variable
-   :language 'python
-   '((identifier) @python--treesit-fontify-variable))
-  "Tree-sitter font-lock settings.")
-
 (defun python--treesit-variable-p (node)
   "Check whether NODE is a variable.
 NODE's type should be \"identifier\"."
@@ -6541,13 +6420,145 @@ python-electric-pair-string-delimiter
 (defvar electric-indent-inhibit)
 (defvar prettify-symbols-alist)
 
+(defvar python-mode--treesit-conf
+  (list
+   'python
+   ;; font-lock feature list
+   '(( comment definition)
+     ( keyword string type)
+     ( assignment builtin constant decorator
+       escape-sequence number property string-interpolation )
+     ( bracket delimiter function operator variable))
+   ;; font-lock settings
+   (treesit-font-lock-rules
+    :feature 'comment
+    :language 'python
+    '((comment) @font-lock-comment-face)
+
+    :feature 'string
+    :language 'python
+    '((string) @python--treesit-fontify-string)
+
+    :feature 'string-interpolation
+    :language 'python
+    :override t
+    '((interpolation (identifier) @font-lock-variable-name-face))
+
+    :feature 'definition
+    :language 'python
+    '((function_definition
+       name: (identifier) @font-lock-function-name-face)
+      (class_definition
+       name: (identifier) @font-lock-type-face))
+
+    :feature 'function
+    :language 'python
+    '((function_definition
+       name: (identifier) @font-lock-function-name-face)
+      (call function: (identifier) @font-lock-function-name-face)
+      (call function: (attribute
+                       attribute: (identifier) @font-lock-function-name-face)))
+
+    :feature 'keyword
+    :language 'python
+    `([,@python--treesit-keywords] @font-lock-keyword-face
+      ((identifier) @font-lock-keyword-face
+       (:match "^self$" @font-lock-keyword-face)))
+
+    :feature 'builtin
+    :language 'python
+    `(((identifier) @font-lock-builtin-face
+       (:match ,(rx-to-string
+                 `(seq bol
+                       (or ,@python--treesit-builtins
+                           ,@python--treesit-special-attributes)
+                       eol))
+               @font-lock-builtin-face)))
+
+    :feature 'constant
+    :language 'python
+    '([(true) (false) (none)] @font-lock-constant-face)
+
+    :feature 'assignment
+    :language 'python
+    `(;; Variable names and LHS.
+      (assignment left: (identifier)
+                  @font-lock-variable-name-face)
+      (assignment left: (attribute
+                         attribute: (identifier)
+                         @font-lock-property-face))
+      (pattern_list (identifier)
+                    @font-lock-variable-name-face)
+      (tuple_pattern (identifier)
+                     @font-lock-variable-name-face)
+      (list_pattern (identifier)
+                    @font-lock-variable-name-face)
+      (list_splat_pattern (identifier)
+                          @font-lock-variable-name-face))
+
+    :feature 'decorator
+    :language 'python
+    '((decorator "@" @font-lock-type-face)
+      (decorator (call function: (identifier) @font-lock-type-face))
+      (decorator (identifier) @font-lock-type-face))
+
+    :feature 'type
+    :language 'python
+    `(((identifier) @font-lock-type-face
+       (:match ,(rx-to-string
+                 `(seq bol (or ,@python--treesit-exceptions)
+                       eol))
+               @font-lock-type-face))
+      (type (identifier) @font-lock-type-face))
+
+    :feature 'escape-sequence
+    :language 'python
+    :override t
+    '((escape_sequence) @font-lock-escape-face)
+
+    :feature 'number
+    :language 'python
+    '([(integer) (float)] @font-lock-number-face)
+
+    :feature 'property
+    :language 'python
+    '((attribute
+       attribute: (identifier) @font-lock-property-face)
+      (class_definition
+       body: (block
+              (expression_statement
+               (assignment left:
+                           (identifier) @font-lock-property-face)))))
+
+    :feature 'operator
+    :language 'python
+    `([,@python--treesit-operators] @font-lock-operator-face)
+
+    :feature 'bracket
+    :language 'python
+    '(["(" ")" "[" "]" "{" "}"] @font-lock-bracket-face)
+
+    :feature 'delimiter
+    :language 'python
+    '(["," "." ":" ";" (ellipsis)] @font-lock-delimiter-face)
+
+    :feature 'variable
+    :language 'python
+    '((identifier) @python--treesit-fontify-variable))
+   ;; defun name function
+   #'python--treesit-defun-name
+   ;; defun regexp
+   (rx (or "function" "class") "_definition")
+   ;; imenu function
+   #'python-imenu-create-index))
+
 ;;;###autoload
-(define-derived-mode python-base-mode prog-mode "Python"
-  "Generic major mode for editing Python files.
+(define-derived-mode python-mode prog-mode "Python"
+  "Major mode for editing Python files.
 
-This is a generic major mode intended to be inherited by
-concrete implementations.  Currently there are two concrete
-implementations: `python-mode' and `python-ts-mode'."
+\\{python-mode-map}"
+  :syntax-table python-mode-syntax-table
+  :parser-conf python-mode--treesit-conf
   (setq-local tab-width 8)
   (setq-local indent-tabs-mode nil)
 
@@ -6603,20 +6614,19 @@ python-base-mode
                       #'python-eldoc-function))))
 
   ;; TODO: Use tree-sitter to figure out the block in `python-ts-mode'.
-  (dolist (mode '(python-mode python-ts-mode))
-    (add-to-list
-     'hs-special-modes-alist
-     `(,mode
-       ,python-nav-beginning-of-block-regexp
-       ;; Use the empty string as end regexp so it doesn't default to
-       ;; "\\s)".  This way parens at end of defun are properly hidden.
-       ""
-       "#"
-       python-hideshow-forward-sexp-function
-       nil
-       python-nav-beginning-of-block
-       python-hideshow-find-next-block
-       python-info-looking-at-beginning-of-block)))
+  (add-to-list
+   'hs-special-modes-alist
+   `(python-mode
+     ,python-nav-beginning-of-block-regexp
+     ;; Use the empty string as end regexp so it doesn't default to
+     ;; "\\s)".  This way parens at end of defun are properly hidden.
+     ""
+     "#"
+     python-hideshow-forward-sexp-function
+     nil
+     python-nav-beginning-of-block
+     python-hideshow-find-next-block
+     python-info-looking-at-beginning-of-block))
 
   (setq-local outline-regexp (python-rx (* space) block-start))
   (setq-local outline-level
@@ -6630,13 +6640,8 @@ python-base-mode
 
   (make-local-variable 'python-shell-internal-buffer)
 
-  (add-hook 'flymake-diagnostic-functions #'python-flymake nil t))
+  (add-hook 'flymake-diagnostic-functions #'python-flymake nil t)
 
-;;;###autoload
-(define-derived-mode python-mode python-base-mode "Python"
-  "Major mode for editing Python files.
-
-\\{python-mode-map}"
   (setq-local font-lock-defaults
               `(,python-font-lock-keywords
                 nil nil nil nil
@@ -6654,32 +6659,6 @@ python-mode
   (when python-indent-guess-indent-offset
     (python-indent-guess-indent-offset)))
 
-;;;###autoload
-(define-derived-mode python-ts-mode python-base-mode "Python"
-  "Major mode for editing Python files, using tree-sitter library.
-
-\\{python-ts-mode-map}"
-  :syntax-table python-mode-syntax-table
-  (when (treesit-ready-p 'python)
-    (treesit-parser-create 'python)
-    (setq-local treesit-font-lock-feature-list
-                '(( comment definition)
-                  ( keyword string type)
-                  ( assignment builtin constant decorator
-                    escape-sequence number property string-interpolation )
-                  ( bracket delimiter function operator variable)))
-    (setq-local treesit-font-lock-settings python--treesit-settings)
-    (setq-local imenu-create-index-function
-                #'python-imenu-treesit-create-index)
-    (setq-local treesit-defun-type-regexp (rx (or "function" "class")
-                                              "_definition"))
-    (setq-local treesit-defun-name-function
-                #'python--treesit-defun-name)
-    (treesit-major-mode-setup)
-
-    (when python-indent-guess-indent-offset
-      (python-indent-guess-indent-offset))))
-
 ;;; Completion predicates for M-x
 ;; Commands that only make sense when editing Python code
 (dolist (sym '(python-add-import
diff --git a/lisp/treesit.el b/lisp/treesit.el
index 2130cd0061..ba38a7d9b2 100644
--- a/lisp/treesit.el
+++ b/lisp/treesit.el
@@ -99,6 +99,15 @@ treesit
   :group 'tools
   :version "29.1")
 
+(defcustom treesit-enabled-modes nil
+  "List of modes to enable tree-sitter support if available.
+When initialising a major mode with potential tree-sitter
+support, this variable is consulted.  The special value t will
+enable tree-sitter support whenever possible."
+  :type '(choice (const :tag "Whenever possible" t)
+                 (repeat :tag "Specific modes" function))
+  :version "29.1")
+
 (defcustom treesit-max-buffer-size
   (let ((mb (* 1024 1024)))
     ;; 40MB for 64-bit systems, 15 for 32-bit.

Re: Unifying "foo-mode"s and "foo-ts-mode"s

[Theodor Thornhill]

Philip Kaludercic <philipk@posteo.net> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> You can try.  I would like to start a full feature freeze in a day or
>> two, so I'm not sure you will have enough time.  And it isn't like we
>> didn't try various approaches during the past two months, so frankly I
>> don't think that a better way even exists.  But if you come up with
>> some very bright idea, who knows?
>
> I have attached a sketch of my proposal with support for Python.
> Instead of a separate python-ts-mode, we regulate tree-sitter support
> using a user option `treesit-enabled-modes'.  It can either be a list
>

[...]

IIUC this will make all other config run before the treesit-related
code?  In that case I think this cannot work, because we _don't_ want to
set all the before/after-change functions many modes set, for example.

Theo

Re: Unifying "foo-mode"s and "foo-ts-mode"s

[Philip Kaludercic]

Theodor Thornhill <theo@thornhill.no> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>>> You can try.  I would like to start a full feature freeze in a day or
>>> two, so I'm not sure you will have enough time.  And it isn't like we
>>> didn't try various approaches during the past two months, so frankly I
>>> don't think that a better way even exists.  But if you come up with
>>> some very bright idea, who knows?
>>
>> I have attached a sketch of my proposal with support for Python.
>> Instead of a separate python-ts-mode, we regulate tree-sitter support
>> using a user option `treesit-enabled-modes'.  It can either be a list
>>
>
> [...]
>
> IIUC this will make all other config run before the treesit-related
> code?  

If that is the problem, that we can solve that by re-adjusting the order
in which the expanded code occurs. 

>        In that case I think this cannot work, because we _don't_ want to
> set all the before/after-change functions many modes set, for example.

What exactly is the issue here?  Can't we overwrite it again if
necessary?

> Theo

Re: Unifying "foo-mode"s and "foo-ts-mode"s

[Theodor Thornhill]

Philip Kaludercic <philipk@posteo.net> writes:

> Theodor Thornhill <theo@thornhill.no> writes:
>
>> Philip Kaludercic <philipk@posteo.net> writes:
>>
>>> Eli Zaretskii <eliz@gnu.org> writes:
>>>
>>>> You can try.  I would like to start a full feature freeze in a day or
>>>> two, so I'm not sure you will have enough time.  And it isn't like we
>>>> didn't try various approaches during the past two months, so frankly I
>>>> don't think that a better way even exists.  But if you come up with
>>>> some very bright idea, who knows?
>>>
>>> I have attached a sketch of my proposal with support for Python.
>>> Instead of a separate python-ts-mode, we regulate tree-sitter support
>>> using a user option `treesit-enabled-modes'.  It can either be a list
>>>
>>
>> [...]
>>
>> IIUC this will make all other config run before the treesit-related
>> code?  
>
> If that is the problem, that we can solve that by re-adjusting the order
> in which the expanded code occurs. 
>
>>        In that case I think this cannot work, because we _don't_ want to
>> set all the before/after-change functions many modes set, for example.
>
> What exactly is the issue here?  Can't we overwrite it again if
> necessary?
>

For example the CC modes set up lots of functions in the mode init, many
of which override things like '*-function' variables, that if either not
overriden explicitly by a treesit alternative or removed before mode
init will impact performance.  There are some modes that will be worse
in this regard than others, but one of my earlier suggestions was to
just:

(define-derived-mode foo ........

  (cond
    (use-treesit-p
     (init-all-the-treesit-stuff))
    (use-hypothetical-future-thing
     (init-all-the-hypothetical-future-stuff))
    (t
     (init-all-the-other-stuff))))

In this case we don't let any code bleed in between the modes, which IMO
is necessary.  At least we should be very careful with _when_ it is ok
for such settings to bleed in.  Things like comment-start/end etc can
bleed in just fine, but stuff like

```
  (c-init-language-vars js-mode)
  (setq-local indent-line-function #'js-indent-line)
  (setq-local beginning-of-defun-function #'js-beginning-of-defun)
  (setq-local end-of-defun-function #'js-end-of-defun)
  (setq-local open-paren-in-column-0-is-defun-start nil)
  (setq-local font-lock-defaults
              (list js--font-lock-keywords nil nil nil nil
                    '(font-lock-syntactic-face-function
                      . js-font-lock-syntactic-face-function)))
  (setq-local syntax-propertize-function #'js-syntax-propertize)
  (add-hook 'syntax-propertize-extend-region-functions
            #'syntax-propertize-multiline 'append 'local)
  (add-hook 'syntax-propertize-extend-region-functions
            #'js--syntax-propertize-extend-region 'append 'local)
  (setq-local prettify-symbols-alist js--prettify-symbols-alist)

  (setq-local parse-sexp-ignore-comments t)
  (setq-local which-func-imenu-joiner-function #'js--which-func-joiner)
```

Should absolutely not.

Does that make sense?  I don't think this is impossible, but my biggest
argument was that we need to keep things distinct, or at least be very
explicit on when we share code.

Theo

Re: Unifying "foo-mode"s and "foo-ts-mode"s

[Philip Kaludercic]

Theodor Thornhill <theo@thornhill.no> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> Theodor Thornhill <theo@thornhill.no> writes:
>>
>>> Philip Kaludercic <philipk@posteo.net> writes:
>>>
>>>> Eli Zaretskii <eliz@gnu.org> writes:
>>>>
>>>>> You can try.  I would like to start a full feature freeze in a day or
>>>>> two, so I'm not sure you will have enough time.  And it isn't like we
>>>>> didn't try various approaches during the past two months, so frankly I
>>>>> don't think that a better way even exists.  But if you come up with
>>>>> some very bright idea, who knows?
>>>>
>>>> I have attached a sketch of my proposal with support for Python.
>>>> Instead of a separate python-ts-mode, we regulate tree-sitter support
>>>> using a user option `treesit-enabled-modes'.  It can either be a list
>>>>
>>>
>>> [...]
>>>
>>> IIUC this will make all other config run before the treesit-related
>>> code?  
>>
>> If that is the problem, that we can solve that by re-adjusting the order
>> in which the expanded code occurs. 
>>
>>>        In that case I think this cannot work, because we _don't_ want to
>>> set all the before/after-change functions many modes set, for example.
>>
>> What exactly is the issue here?  Can't we overwrite it again if
>> necessary?
>>
>
> For example the CC modes set up lots of functions in the mode init, many
> of which override things like '*-function' variables, that if either not
> overriden explicitly by a treesit alternative or removed before mode
> init will impact performance.  There are some modes that will be worse
> in this regard than others, but one of my earlier suggestions was to
> just:
>
> (define-derived-mode foo ........
>
>   (cond
>     (use-treesit-p
>      (init-all-the-treesit-stuff))
>     (use-hypothetical-future-thing
>      (init-all-the-hypothetical-future-stuff))
>     (t
>      (init-all-the-other-stuff))))

This also looks good.

> In this case we don't let any code bleed in between the modes, which IMO
> is necessary.  At least we should be very careful with _when_ it is ok
> for such settings to bleed in.  Things like comment-start/end etc can
> bleed in just fine, but stuff like
>
> ```
>   (c-init-language-vars js-mode)
>   (setq-local indent-line-function #'js-indent-line)
>   (setq-local beginning-of-defun-function #'js-beginning-of-defun)
>   (setq-local end-of-defun-function #'js-end-of-defun)
>   (setq-local open-paren-in-column-0-is-defun-start nil)
>   (setq-local font-lock-defaults
>               (list js--font-lock-keywords nil nil nil nil
>                     '(font-lock-syntactic-face-function
>                       . js-font-lock-syntactic-face-function)))
>   (setq-local syntax-propertize-function #'js-syntax-propertize)
>   (add-hook 'syntax-propertize-extend-region-functions
>             #'syntax-propertize-multiline 'append 'local)
>   (add-hook 'syntax-propertize-extend-region-functions
>             #'js--syntax-propertize-extend-region 'append 'local)
>   (setq-local prettify-symbols-alist js--prettify-symbols-alist)
>
>   (setq-local parse-sexp-ignore-comments t)
>   (setq-local which-func-imenu-joiner-function #'js--which-func-joiner)
> ```
>
> Should absolutely not.
>
> Does that make sense?  I don't think this is impossible, but my biggest
> argument was that we need to keep things distinct, or at least be very
> explicit on when we share code.

Yes, I do understand this point, yet my impression has been that this
was not always necessary.  The relative complexity of cc-mode might
necessitate a separate mode, but I don't see why that should be the rule
instead of an exception?

Re: Unifying "foo-mode"s and "foo-ts-mode"s

[Theodor Thornhill]

On 30 December 2022 16:02:37 CET, Philip Kaludercic <philipk@posteo.net> wrote:
>Theodor Thornhill <theo@thornhill.no> writes:
>
>> Philip Kaludercic <philipk@posteo.net> writes:
>>
>>> Theodor Thornhill <theo@thornhill.no> writes:
>>>
>>>> Philip Kaludercic <philipk@posteo.net> writes:
>>>>
>>>>> Eli Zaretskii <eliz@gnu.org> writes:
>>>>>
>>>>>> You can try.  I would like to start a full feature freeze in a day or
>>>>>> two, so I'm not sure you will have enough time.  And it isn't like we
>>>>>> didn't try various approaches during the past two months, so frankly I
>>>>>> don't think that a better way even exists.  But if you come up with
>>>>>> some very bright idea, who knows?
>>>>>
>>>>> I have attached a sketch of my proposal with support for Python.
>>>>> Instead of a separate python-ts-mode, we regulate tree-sitter support
>>>>> using a user option `treesit-enabled-modes'.  It can either be a list
>>>>>
>>>>
>>>> [...]
>>>>
>>>> IIUC this will make all other config run before the treesit-related
>>>> code?  
>>>
>>> If that is the problem, that we can solve that by re-adjusting the order
>>> in which the expanded code occurs. 
>>>
>>>>        In that case I think this cannot work, because we _don't_ want to
>>>> set all the before/after-change functions many modes set, for example.
>>>
>>> What exactly is the issue here?  Can't we overwrite it again if
>>> necessary?
>>>
>>
>> For example the CC modes set up lots of functions in the mode init, many
>> of which override things like '*-function' variables, that if either not
>> overriden explicitly by a treesit alternative or removed before mode
>> init will impact performance.  There are some modes that will be worse
>> in this regard than others, but one of my earlier suggestions was to
>> just:
>>
>> (define-derived-mode foo ........
>>
>>   (cond
>>     (use-treesit-p
>>      (init-all-the-treesit-stuff))
>>     (use-hypothetical-future-thing
>>      (init-all-the-hypothetical-future-stuff))
>>     (t
>>      (init-all-the-other-stuff))))
>
>This also looks good.
>
>> In this case we don't let any code bleed in between the modes, which IMO
>> is necessary.  At least we should be very careful with _when_ it is ok
>> for such settings to bleed in.  Things like comment-start/end etc can
>> bleed in just fine, but stuff like
>>
>> ```
>>   (c-init-language-vars js-mode)
>>   (setq-local indent-line-function #'js-indent-line)
>>   (setq-local beginning-of-defun-function #'js-beginning-of-defun)
>>   (setq-local end-of-defun-function #'js-end-of-defun)
>>   (setq-local open-paren-in-column-0-is-defun-start nil)
>>   (setq-local font-lock-defaults
>>               (list js--font-lock-keywords nil nil nil nil
>>                     '(font-lock-syntactic-face-function
>>                       . js-font-lock-syntactic-face-function)))
>>   (setq-local syntax-propertize-function #'js-syntax-propertize)
>>   (add-hook 'syntax-propertize-extend-region-functions
>>             #'syntax-propertize-multiline 'append 'local)
>>   (add-hook 'syntax-propertize-extend-region-functions
>>             #'js--syntax-propertize-extend-region 'append 'local)
>>   (setq-local prettify-symbols-alist js--prettify-symbols-alist)
>>
>>   (setq-local parse-sexp-ignore-comments t)
>>   (setq-local which-func-imenu-joiner-function #'js--which-func-joiner)
>> ```
>>
>> Should absolutely not.
>>
>> Does that make sense?  I don't think this is impossible, but my biggest
>> argument was that we need to keep things distinct, or at least be very
>> explicit on when we share code.
>
>Yes, I do understand this point, yet my impression has been that this
>was not always necessary.  The relative complexity of cc-mode might
>necessitate a separate mode, but I don't see why that should be the rule
>instead of an exception?

IIRC it was mostly because we wanted to start creating things instead of bikeshedding over these details. 

My thought process was to create separate modes now, and make a facility to map a language to a mode implementation. For example, imo it doesn't make sense for "the first" implementation to own a language's namespace. c-mode should be able to leverage cc-mode _or_ c-ts-mode, rather than the other way around.

At least during these months that seemed smart, because we have many contributions now due to the simple nature of creating a foo-ts-mode. 

Surgically injecting tree sitter into existing modes is prone to error, and requires quite deep knowledge of each existing mode's inner working.

So I'd just hold off and maybe create a nice facility for Emacs 30.

Something like

(setq major-mode-backend
  '((c . cc-mode)
    (c++ . treesit)))

Then M-x c-mode would trigger cc, and c++-mode would trigger c++-ts-mode.

And we keep the old implementations as the default until we know tree-sitter, has no disadvantages and swap the default. No need to deprecate anything or require config changes, imo.

Theo

Re: Unifying "foo-mode"s and "foo-ts-mode"s

[Philip Kaludercic]

Theodor Thornhill <theo@thornhill.no> writes:

> On 30 December 2022 16:02:37 CET, Philip Kaludercic <philipk@posteo.net> wrote:
>>Theodor Thornhill <theo@thornhill.no> writes:
>>
>>> Philip Kaludercic <philipk@posteo.net> writes:
>>>
>>>> Theodor Thornhill <theo@thornhill.no> writes:
>>>>
>>>>> Philip Kaludercic <philipk@posteo.net> writes:
>>>>>
>>>>>> Eli Zaretskii <eliz@gnu.org> writes:
>>>>>>
>>>>>>> You can try.  I would like to start a full feature freeze in a day or
>>>>>>> two, so I'm not sure you will have enough time.  And it isn't like we
>>>>>>> didn't try various approaches during the past two months, so frankly I
>>>>>>> don't think that a better way even exists.  But if you come up with
>>>>>>> some very bright idea, who knows?
>>>>>>
>>>>>> I have attached a sketch of my proposal with support for Python.
>>>>>> Instead of a separate python-ts-mode, we regulate tree-sitter support
>>>>>> using a user option `treesit-enabled-modes'.  It can either be a list
>>>>>>
>>>>>
>>>>> [...]
>>>>>
>>>>> IIUC this will make all other config run before the treesit-related
>>>>> code?  
>>>>
>>>> If that is the problem, that we can solve that by re-adjusting the order
>>>> in which the expanded code occurs. 
>>>>
>>>>>        In that case I think this cannot work, because we _don't_ want to
>>>>> set all the before/after-change functions many modes set, for example.
>>>>
>>>> What exactly is the issue here?  Can't we overwrite it again if
>>>> necessary?
>>>>
>>>
>>> For example the CC modes set up lots of functions in the mode init, many
>>> of which override things like '*-function' variables, that if either not
>>> overriden explicitly by a treesit alternative or removed before mode
>>> init will impact performance.  There are some modes that will be worse
>>> in this regard than others, but one of my earlier suggestions was to
>>> just:
>>>
>>> (define-derived-mode foo ........
>>>
>>>   (cond
>>>     (use-treesit-p
>>>      (init-all-the-treesit-stuff))
>>>     (use-hypothetical-future-thing
>>>      (init-all-the-hypothetical-future-stuff))
>>>     (t
>>>      (init-all-the-other-stuff))))
>>
>>This also looks good.
>>
>>> In this case we don't let any code bleed in between the modes, which IMO
>>> is necessary.  At least we should be very careful with _when_ it is ok
>>> for such settings to bleed in.  Things like comment-start/end etc can
>>> bleed in just fine, but stuff like
>>>
>>> ```
>>>   (c-init-language-vars js-mode)
>>>   (setq-local indent-line-function #'js-indent-line)
>>>   (setq-local beginning-of-defun-function #'js-beginning-of-defun)
>>>   (setq-local end-of-defun-function #'js-end-of-defun)
>>>   (setq-local open-paren-in-column-0-is-defun-start nil)
>>>   (setq-local font-lock-defaults
>>>               (list js--font-lock-keywords nil nil nil nil
>>>                     '(font-lock-syntactic-face-function
>>>                       . js-font-lock-syntactic-face-function)))
>>>   (setq-local syntax-propertize-function #'js-syntax-propertize)
>>>   (add-hook 'syntax-propertize-extend-region-functions
>>>             #'syntax-propertize-multiline 'append 'local)
>>>   (add-hook 'syntax-propertize-extend-region-functions
>>>             #'js--syntax-propertize-extend-region 'append 'local)
>>>   (setq-local prettify-symbols-alist js--prettify-symbols-alist)
>>>
>>>   (setq-local parse-sexp-ignore-comments t)
>>>   (setq-local which-func-imenu-joiner-function #'js--which-func-joiner)
>>> ```
>>>
>>> Should absolutely not.
>>>
>>> Does that make sense?  I don't think this is impossible, but my biggest
>>> argument was that we need to keep things distinct, or at least be very
>>> explicit on when we share code.
>>
>>Yes, I do understand this point, yet my impression has been that this
>>was not always necessary.  The relative complexity of cc-mode might
>>necessitate a separate mode, but I don't see why that should be the rule
>>instead of an exception?
>
> IIRC it was mostly because we wanted to start creating things instead of bikeshedding over these details. 
>
> My thought process was to create separate modes now, and make a
> facility to map a language to a mode implementation. For example, imo
> it doesn't make sense for "the first" implementation to own a
> language's namespace. c-mode should be able to leverage cc-mode _or_
> c-ts-mode, rather than the other way around.

This is a good point, but as you say below this might be something that
we should aim towards for Emacs 30.

> At least during these months that seemed smart, because we have many
> contributions now due to the simple nature of creating a foo-ts-mode.

Some of which have been very superficial, leading to the dilemma that
users will have themselves with:  Do they want tree-sitter support or
the features of a more complete and mature mode.

> Surgically injecting tree sitter into existing modes is prone to
> error, and requires quite deep knowledge of each existing mode's inner
> working.
>
> So I'd just hold off and maybe create a nice facility for Emacs 30.
>
> Something like
>
> (setq major-mode-backend
>   '((c . cc-mode)
>     (c++ . treesit)))
>
> Then M-x c-mode would trigger cc, and c++-mode would trigger c++-ts-mode.
>
> And we keep the old implementations as the default until we know
> tree-sitter, has no disadvantages and swap the default. No need to
> deprecate anything or require config changes, imo.
>
> Theo

Re: Unifying "foo-mode"s and "foo-ts-mode"s

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org,  Yuan Fu
>  <casouri@gmail.com>
> Date: Fri, 30 Dec 2022 15:02:37 +0000
> 
> Yes, I do understand this point, yet my impression has been that this
> was not always necessary.  The relative complexity of cc-mode might
> necessitate a separate mode, but I don't see why that should be the rule
> instead of an exception?

CC Mode means C, C++, and Java already.  If these 3 are separate
modes, how to make your proposal work for them and for the others as
well, in a way that is (1) convenient and easily understood by users,
and (2) simple and safe enough to implement so that we don't
inadvertently screw existing modes?

Re: Unifying "foo-mode"s and "foo-ts-mode"s

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org,  Yuan Fu
>>  <casouri@gmail.com>
>> Date: Fri, 30 Dec 2022 15:02:37 +0000
>> 
>> Yes, I do understand this point, yet my impression has been that this
>> was not always necessary.  The relative complexity of cc-mode might
>> necessitate a separate mode, but I don't see why that should be the rule
>> instead of an exception?
>
> CC Mode means C, C++, and Java already.  If these 3 are separate
> modes, how to make your proposal work for them and for the others as
> well, in a way that is (1) convenient and easily understood by users,
> and (2) simple and safe enough to implement so that we don't
> inadvertently screw existing modes?

In some other response I have said that cc-mode seems too complicated
for now.  I agree that it is a better approach to take some time and
develop the right infrastructure going forward towards Emacs 30.  But
would this mean that the "...-ts-mode"s are to be regarded as
experimental demonstrations?

Re: Unifying "foo-mode"s and "foo-ts-mode"s

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: theo@thornhill.no,  emacs-devel@gnu.org,  casouri@gmail.com
> Date: Fri, 30 Dec 2022 15:57:59 +0000
> 
> would this mean that the "...-ts-mode"s are to be regarded as
> experimental demonstrations?

I don't understand the question.  We provide to users _features_, not
their implementations in terms of files and variables/functions.  So
the symbols *-ts-mode are here to stay, and are not experimental
demonstrations in any sense of that term, but what they do under the
hood is up to us.

Re: Unifying "foo-mode"s and "foo-ts-mode"s

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: theo@thornhill.no,  emacs-devel@gnu.org,  casouri@gmail.com
>> Date: Fri, 30 Dec 2022 15:57:59 +0000
>> 
>> would this mean that the "...-ts-mode"s are to be regarded as
>> experimental demonstrations?
>
> I don't understand the question.  We provide to users _features_, not
> their implementations in terms of files and variables/functions.  So
> the symbols *-ts-mode are here to stay, and are not experimental
> demonstrations in any sense of that term, but what they do under the
> hood is up to us.

That answers my question.  The point is whether the usage of ...-ts-mode
major modes is recommended or provided with a "we reserve the right to
change anything"-like caveat, so one shouldn't rely too much on their
public interface.

Re: Unifying "foo-mode"s and "foo-ts-mode"s

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: theo@thornhill.no,  emacs-devel@gnu.org,  casouri@gmail.com
> Date: Fri, 30 Dec 2022 16:39:18 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I don't understand the question.  We provide to users _features_, not
> > their implementations in terms of files and variables/functions.  So
> > the symbols *-ts-mode are here to stay, and are not experimental
> > demonstrations in any sense of that term, but what they do under the
> > hood is up to us.
> 
> That answers my question.  The point is whether the usage of ...-ts-mode
> major modes is recommended or provided with a "we reserve the right to
> change anything"-like caveat, so one shouldn't rely too much on their
> public interface.

I guess it depends on who "one" is for this purpose.  Which code wants
to rely on these symbols, and why?

Re: Unifying "foo-mode"s and "foo-ts-mode"s

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: theo@thornhill.no,  emacs-devel@gnu.org,  casouri@gmail.com
>> Date: Fri, 30 Dec 2022 16:39:18 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > I don't understand the question.  We provide to users _features_, not
>> > their implementations in terms of files and variables/functions.  So
>> > the symbols *-ts-mode are here to stay, and are not experimental
>> > demonstrations in any sense of that term, but what they do under the
>> > hood is up to us.
>> 
>> That answers my question.  The point is whether the usage of ...-ts-mode
>> major modes is recommended or provided with a "we reserve the right to
>> change anything"-like caveat, so one shouldn't rely too much on their
>> public interface.
>
> I guess it depends on who "one" is for this purpose.  Which code wants
> to rely on these symbols, and why?

Both emacs-internal packages and external packages that depend on Emacs
29.

Re: Unifying "foo-mode"s and "foo-ts-mode"s

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: theo@thornhill.no,  emacs-devel@gnu.org,  casouri@gmail.com
> Date: Sat, 31 Dec 2022 00:13:03 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> That answers my question.  The point is whether the usage of ...-ts-mode
> >> major modes is recommended or provided with a "we reserve the right to
> >> change anything"-like caveat, so one shouldn't rely too much on their
> >> public interface.
> >
> > I guess it depends on who "one" is for this purpose.  Which code wants
> > to rely on these symbols, and why?
> 
> Both emacs-internal packages and external packages that depend on Emacs
> 29.

Are there any concrete examples of those?  If not, let's delay this
discussion until such concrete cases emerge, because discussing a
theoretical issue is many times not very productive, as it's easy to
talk past each other when there's nothing concrete in sight.

Re: Unifying "foo-mode"s and "foo-ts-mode"s

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: emacs-devel@gnu.org, Yuan Fu <casouri@gmail.com>
> Date: Fri, 30 Dec 2022 10:58:39 +0000
> 
> I have attached a sketch of my proposal with support for Python.
> Instead of a separate python-ts-mode, we regulate tree-sitter support
> using a user option `treesit-enabled-modes'.  It can either be a list
> 
> --8<---------------cut here---------------start------------->8---
>   (setq treesit-enabled-modes '(python-mode c-mode))
> --8<---------------cut here---------------end--------------->8---
> 
> or generally enable tree-sitter
> 
> --8<---------------cut here---------------start------------->8---
>   (setq treesit-enabled-modes t)
> --8<---------------cut here---------------end--------------->8---

I think we want to let the users say, for every single mode, whether
they want to use the treesit-enabled variant or not, and also to be
able to go back to the non-treesit mode later in the session (e.g., if
they don't like the results).  A list is not a convenient means for
doing so.

> 
> All a major modes has to do is pass a parser configuration
> 
> --8<---------------cut here---------------start------------->8---
>   (define-derived-mode python-mode prog-mode "Python"
>     "Major mode for editing Python files.
> 
>   \\{python-mode-map}"
>     :syntax-table python-mode-syntax-table
>     :parser-conf python-mode--treesit-conf
>    ...
> --8<---------------cut here---------------end--------------->8---
> 
> that expands to
> 
> --8<---------------cut here---------------start------------->8---
>   (when-let
>       ((conf python-mode--treesit-conf)
>        ((cond
>          ((listp treesit-enabled-modes)
>           (memq 'python-mode treesit-enabled-modes))
>          ((eq treesit-enabled-modes t))))
>        ((treesit-ready-p
>          (nth 0 conf)))
>        (parser
>         (treesit-parser-create
>          (nth 0 conf))))
>     (setq-local treesit-font-lock-feature-list
>                 (nth 1 conf)
>                 treesit-font-lock-settings
>                 (nth 2 conf)
>                 treesit-defun-name-function
>                 (nth 3 conf)
>                 treesit-defun-type-regexp
>                 (nth 4 conf)
>                 imenu-create-index-function
>                 (nth 5 conf))
>     (treesit-major-mode-setup))
> --8<---------------cut here---------------end--------------->8---
> 
> at *the end* of the major mode definition.  Note that if no parser
> configuration was parsed, the entire expression is byte-compiled away,
> so there is no run-time overhead for other modes.

What does this mean in user-facing behavior?  Does it mean that if
tree-sitter is not available, or the Python grammar fails to load for
some reason, Emacs will silently fall back to the "traditional"
python-mode?  If so, I don't think this is what we want.  The failure
for loading tree-sitter support should not be silent.

These are exactly the aspects of the behavior we discussed a month
ago, and what we have now is the result of those discussions.

Re: Unifying "foo-mode"s and "foo-ts-mode"s

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: emacs-devel@gnu.org, Yuan Fu <casouri@gmail.com>
>> Date: Fri, 30 Dec 2022 10:58:39 +0000
>> 
>> I have attached a sketch of my proposal with support for Python.
>> Instead of a separate python-ts-mode, we regulate tree-sitter support
>> using a user option `treesit-enabled-modes'.  It can either be a list
>> 
>> --8<---------------cut here---------------start------------->8---
>>   (setq treesit-enabled-modes '(python-mode c-mode))
>> --8<---------------cut here---------------end--------------->8---
>> 
>> or generally enable tree-sitter
>> 
>> --8<---------------cut here---------------start------------->8---
>>   (setq treesit-enabled-modes t)
>> --8<---------------cut here---------------end--------------->8---
>
> I think we want to let the users say, for every single mode, whether
> they want to use the treesit-enabled variant or not, and also to be
> able to go back to the non-treesit mode later in the session (e.g., if
> they don't like the results).  A list is not a convenient means for
> doing so.

How come?  When presented in the customise interface we could make it
out to be a set where users get to pick what modes they want.  And
updating the value works fine whenever a mode is re-applied.

>> 
>> All a major modes has to do is pass a parser configuration
>> 
>> --8<---------------cut here---------------start------------->8---
>>   (define-derived-mode python-mode prog-mode "Python"
>>     "Major mode for editing Python files.
>> 
>>   \\{python-mode-map}"
>>     :syntax-table python-mode-syntax-table
>>     :parser-conf python-mode--treesit-conf
>>    ...
>> --8<---------------cut here---------------end--------------->8---
>> 
>> that expands to
>> 
>> --8<---------------cut here---------------start------------->8---
>>   (when-let
>>       ((conf python-mode--treesit-conf)
>>        ((cond
>>          ((listp treesit-enabled-modes)
>>           (memq 'python-mode treesit-enabled-modes))
>>          ((eq treesit-enabled-modes t))))
>>        ((treesit-ready-p
>>          (nth 0 conf)))
>>        (parser
>>         (treesit-parser-create
>>          (nth 0 conf))))
>>     (setq-local treesit-font-lock-feature-list
>>                 (nth 1 conf)
>>                 treesit-font-lock-settings
>>                 (nth 2 conf)
>>                 treesit-defun-name-function
>>                 (nth 3 conf)
>>                 treesit-defun-type-regexp
>>                 (nth 4 conf)
>>                 imenu-create-index-function
>>                 (nth 5 conf))
>>     (treesit-major-mode-setup))
>> --8<---------------cut here---------------end--------------->8---
>> 
>> at *the end* of the major mode definition.  Note that if no parser
>> configuration was parsed, the entire expression is byte-compiled away,
>> so there is no run-time overhead for other modes.
>
> What does this mean in user-facing behavior?  Does it mean that if
> tree-sitter is not available, or the Python grammar fails to load for
> some reason, Emacs will silently fall back to the "traditional"
> python-mode?  If so, I don't think this is what we want.  The failure
> for loading tree-sitter support should not be silent.

I am not sure why?  Tree-sitter is an improvement in that it allows
Emacs to provide better highlighting and knowledge of the syntax, but in
the end it isn't something you think about actively -- or even should
have to think about.This is all backend stuff that doesn't interest the
casual user.  I strongly believe that the principle of "graceful
degradation" is the right approach here.

And in the end, if the tree-sitter support is hidden behind new modes, I
know already that most people (who don't use starter packs) will never
notice their existence and won't make use of the support.  There are
people still using linum-mode, even though display-line-numbers-mode has
been around for a while.

> These are exactly the aspects of the behavior we discussed a month
> ago, and what we have now is the result of those discussions.

Could you point me to the thread(s)?  I did not have the time to follow
the threads in detail a month ago.

Re: Unifying "foo-mode"s and "foo-ts-mode"s

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: emacs-devel@gnu.org,  casouri@gmail.com
> Date: Fri, 30 Dec 2022 15:20:35 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I think we want to let the users say, for every single mode, whether
> > they want to use the treesit-enabled variant or not, and also to be
> > able to go back to the non-treesit mode later in the session (e.g., if
> > they don't like the results).  A list is not a convenient means for
> > doing so.
> 
> How come?  When presented in the customise interface we could make it
> out to be a set where users get to pick what modes they want.  And
> updating the value works fine whenever a mode is re-applied.

The need to manipulate a list when all I want is to turn on or off a
single feature is a nuisance.

> > What does this mean in user-facing behavior?  Does it mean that if
> > tree-sitter is not available, or the Python grammar fails to load for
> > some reason, Emacs will silently fall back to the "traditional"
> > python-mode?  If so, I don't think this is what we want.  The failure
> > for loading tree-sitter support should not be silent.
> 
> I am not sure why?  Tree-sitter is an improvement in that it allows
> Emacs to provide better highlighting and knowledge of the syntax, but in
> the end it isn't something you think about actively -- or even should
> have to think about.This is all backend stuff that doesn't interest the
> casual user.  I strongly believe that the principle of "graceful
> degradation" is the right approach here.

You assume a lot here.  First you assume that everyone will see
tree-sitter as an improvement -- that is not a given, and moreover can
vary from language to language.  Second, you assume that whether
tree-sitter is used doesn't interest the user -- again, not a given.
Especially since we currently ask users to install the grammar
libraries.

Your strong beliefs in this matter are noted, but I don't share them,
and think that we should collect actual user experience before we make
such significant decisions.

Btw, you were previously expressing concerns about the risk of
committing to some strategy too early -- well, I think what you
propose will commit us to some strategy much more than what we have
now.

> And in the end, if the tree-sitter support is hidden behind new modes, I
> know already that most people (who don't use starter packs) will never
> notice their existence and won't make use of the support.  There are
> people still using linum-mode, even though display-line-numbers-mode has
> been around for a while.

There's a world of difference between tree-sitter support and a minor
feature such as display-line-numbers-mode.  Just look at Reddit as one
example.

I'm sure tree-sitter will be very visible to users.

> > These are exactly the aspects of the behavior we discussed a month
> > ago, and what we have now is the result of those discussions.
> 
> Could you point me to the thread(s)?  I did not have the time to follow
> the threads in detail a month ago.

Sorry, I don't have them ready and cannot afford looking them up: too
many things on my plate.  Perhaps someone else will do that.

Re: Unifying "foo-mode"s and "foo-ts-mode"s

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: emacs-devel@gnu.org,  casouri@gmail.com
>> Date: Fri, 30 Dec 2022 15:20:35 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > I think we want to let the users say, for every single mode, whether
>> > they want to use the treesit-enabled variant or not, and also to be
>> > able to go back to the non-treesit mode later in the session (e.g., if
>> > they don't like the results).  A list is not a convenient means for
>> > doing so.
>> 
>> How come?  When presented in the customise interface we could make it
>> out to be a set where users get to pick what modes they want.  And
>> updating the value works fine whenever a mode is re-applied.
>
> The need to manipulate a list when all I want is to turn on or off a
> single feature is a nuisance.

This shouldn't have been an issue.  It is simple to write a command that
toggles tree-sitter support for the major mode in the current buffer.

>> > What does this mean in user-facing behavior?  Does it mean that if
>> > tree-sitter is not available, or the Python grammar fails to load for
>> > some reason, Emacs will silently fall back to the "traditional"
>> > python-mode?  If so, I don't think this is what we want.  The failure
>> > for loading tree-sitter support should not be silent.
>> 
>> I am not sure why?  Tree-sitter is an improvement in that it allows
>> Emacs to provide better highlighting and knowledge of the syntax, but in
>> the end it isn't something you think about actively -- or even should
>> have to think about.This is all backend stuff that doesn't interest the
>> casual user.  I strongly believe that the principle of "graceful
>> degradation" is the right approach here.
>
> You assume a lot here.  First you assume that everyone will see
> tree-sitter as an improvement -- that is not a given, and moreover can
> vary from language to language.  Second, you assume that whether
> tree-sitter is used doesn't interest the user -- again, not a given.
> Especially since we currently ask users to install the grammar
> libraries.

To me the ideal would still be that tree-sitter support is provided by
the distribution, just like how package managers have started bundling
libgccjit for native compilation, instead of requiring the user enable
it themselves.

> Your strong beliefs in this matter are noted, but I don't share them,
> and think that we should collect actual user experience before we make
> such significant decisions.
>
> Btw, you were previously expressing concerns about the risk of
> committing to some strategy too early -- well, I think what you
> propose will commit us to some strategy much more than what we have
> now.

It is a bit tricky for me to argue this point now that I have agreed it
shouldn't be done now, but I don't think this is a greater commitment,
since the user interface would be reduce from a number of major modes to
a single user option (another design principle I stand behind).  The
only other commitment would be that :parser-conf would be exposed as
part of `define-derived-mode'.

>> And in the end, if the tree-sitter support is hidden behind new modes, I
>> know already that most people (who don't use starter packs) will never
>> notice their existence and won't make use of the support.  There are
>> people still using linum-mode, even though display-line-numbers-mode has
>> been around for a while.
>
> There's a world of difference between tree-sitter support and a minor
> feature such as display-line-numbers-mode.  Just look at Reddit as one
> example.

I am not sure what you are referring to here.  My point was just that
features that have to be enabled manually are usually never enabled by
non-enthusiasts (which I believe are the majority, even for Emacs).

> I'm sure tree-sitter will be very visible to users.

Maybe I am in a minority here, but I know I am not alone, in having to
double-check if tree-sitter is enabled or noticing what it even changes.
I am not quite saying it is a placebo, there are certainly tricky C++
fontification cases where the difference is obvious, but simpler
languages like JSON, CSS or to a degree even C the (visual) difference
is rather minor.

>> > These are exactly the aspects of the behavior we discussed a month
>> > ago, and what we have now is the result of those discussions.
>> 
>> Could you point me to the thread(s)?  I did not have the time to follow
>> the threads in detail a month ago.
>
> Sorry, I don't have them ready and cannot afford looking them up: too
> many things on my plate.  Perhaps someone else will do that.

No problem, I'll try to find them myself.  Thanks.

Re: Unifying "foo-mode"s and "foo-ts-mode"s

[Stefan Monnier]

> I am not sure what you are referring to here.  My point was just that
> features that have to be enabled manually are usually never enabled by
> non-enthusiasts (which I believe are the majority, even for Emacs).

FWIW, I think it's perfectly OK if that's how it plays out in Emacs-29.
That gives us a freer hand to improve the integration in Emacs>29.


        Stefan

Re: Unifying "foo-mode"s and "foo-ts-mode"s

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org,  casouri@gmail.com
> Date: Fri, 30 Dec 2022 11:30:44 -0500
> 
> > I am not sure what you are referring to here.  My point was just that
> > features that have to be enabled manually are usually never enabled by
> > non-enthusiasts (which I believe are the majority, even for Emacs).
> 
> FWIW, I think it's perfectly OK if that's how it plays out in Emacs-29.
> That gives us a freer hand to improve the integration in Emacs>29.

Agreed.  AFAIR, that was one of the reasons we eventually decided to
go this way in Emacs 29.

Re: Unifying "foo-mode"s and "foo-ts-mode"s

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

The use of tree-sitter logically ought to be a minor mode.

We don't define c-mode and c-fl-mode to specify whether to enable Font
Lock mode in a file of C code.  Rather, We have c-mode, plus another
orthogonal command, font-lock-mode.

Likewise, the Emacs user interface ought to have another minor-mode
command, tree-sitter-mode.

How it "really works" under the hood is another issue.

Users will be able to control the default for enabling tree-sitter in
various major modes with the existing general mechanism that is used
to control whether to enable Font Lodk mode and other minor modes: the
major mode's mode hook.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Unifying "foo-mode"s and "foo-ts-mode"s

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Sat, 31 Dec 2022 22:03:55 -0500
> 
> The use of tree-sitter logically ought to be a minor mode.

We tried to do it this way at first.  It didn't work well and had
significant difficulties, so we decided a separate major mode is a
better idea.  See below for some details; they were all discussed here
at the time (in October).

> We don't define c-mode and c-fl-mode to specify whether to enable Font
> Lock mode in a file of C code.  Rather, We have c-mode, plus another
> orthogonal command, font-lock-mode.

That is not the whole picture, as you know.  font-lock-mode just
activates the fontifications; the definitions of what to fontify and
how are the responsibility of the major mode.  These definitions are
completely different for tree-sitter based fontifications, as they are
not based on regexps and syntax tables.

And the features based on tree-sitter are not limited to
fontifications, they also support indentation, navigation by defun and
other language-dependent program components, and Imenu indices of
functions, structures, etc.  Each one of these needs to change quite a
few functions and variables of the particular major mode to switch
between tree-sitter and the "traditional" ways, and we also needed to
introduce a lot of new data structures that are not needed for the
"traditional" modes.

Adding all that to the existing modes would mean significant
complication of those existing modes (thus risking destabilizing
them), and would require no less than enthusiastic support from the
maintainers of those modes to get this done cleanly, safely, and in a
reasonable time.  I considered that to be "a bridge too far", and
instead decided to go with a safer, simpler alternative that we have
now.  This allowed us to add tree-sitter supported modes for quite a
few languages and file types in a very short time, and to do that
relatively cleanly.  Most importantly, those additions almost never
touch the modes for the same languages that existed before, thus
allowing us to introduce this new technology in a smooth way, leaving
it to the users to decide whether and when they want to try these new
modes.  This opt-in way is very important given the fact that distros
seem to not yet be organized well enough for providing the relevant
libraries for end-user installation.

> Likewise, the Emacs user interface ought to have another minor-mode
> command, tree-sitter-mode.
> 
> How it "really works" under the hood is another issue.
> 
> Users will be able to control the default for enabling tree-sitter in
> various major modes with the existing general mechanism that is used
> to control whether to enable Font Lodk mode and other minor modes: the
> major mode's mode hook.

This idealistic vision turned out not to work well enough in practice,
especially for large and complex major modes such as CC mode.  Many
settings and sub-features of the original mode make n o sense when
tree-sitter is used, and a huge amount of code based on regexps,
syntax-ppss, and ad-hoc algorithms needs to be bypassed or switched
off with tree-sitter.  It is impractical to do all that in a way that
both takes some reasonably short time and avoids a high risk of
breaking modes that have been working reasonably well for decades,
thus annoying users who may not even want to use tree-sitter.  I
couldn't allow such risky changes in Emacs 29, and I didn't want to
delay this to Emacs 30, which means another year r two.

So instead of turning on a local minor mode, users will have to invoke
a command that actually turns on a major mode.  FWIW, I don't see the
significant difference between these two alternatives, for the user
POV.  And it is quite possible that we will eventually decide to make
these minor modes, if we find a good way of combining them with the
major mode for the specific programming language.  But this is in the
future; we need first to collect user and maintenance experience,
without which there are too many questions and issues (some of which
are being raised already on this list) for which we have no good,
solid answers, only opinions based on information and experience
outside of Emacs, which from my POV are not necessarily valid for us.

Re: Unifying "foo-mode"s and "foo-ts-mode"s

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > And the features based on tree-sitter are not limited to
  > fontifications, they also support indentation, navigation by defun and
  > other language-dependent program components, and Imenu indices of
  > functions, structures, etc.  Each one of these needs to change quite a
  > few functions and variables of the particular major mode to switch
  > between tree-sitter and the "traditional" ways, and we also needed to
  > introduce a lot of new data structures that are not needed for the
  > "traditional" modes.

I know.  I don't think that precludes having tree-sitter be a minor
mode.  The major mode command could set some variables that are used
only when tree-sitter is off, and other variables that are used only
when tree-sitter is on.

  >   I considered that to be "a bridge too far", and
  > instead decided to go with a safer, simpler alternative that we have
  > now.

It sounds like you consider the current design as a _temporary_
expedient.  If that's what it is, maybe writing this first was a good
idea.

I'm talking about how the code should work after there is time to take
stock and clean it up.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Unifying "foo-mode"s and "foo-ts-mode"s

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> Date: Mon, 02 Jan 2023 23:07:49 -0500
> 
>   > And the features based on tree-sitter are not limited to
>   > fontifications, they also support indentation, navigation by defun and
>   > other language-dependent program components, and Imenu indices of
>   > functions, structures, etc.  Each one of these needs to change quite a
>   > few functions and variables of the particular major mode to switch
>   > between tree-sitter and the "traditional" ways, and we also needed to
>   > introduce a lot of new data structures that are not needed for the
>   > "traditional" modes.
> 
> I know.  I don't think that precludes having tree-sitter be a minor
> mode.  The major mode command could set some variables that are used
> only when tree-sitter is off, and other variables that are used only
> when tree-sitter is on.

In principle, yes.  We just haven't yet found a way of doing this
everywhere in a way that is both convenient and safe enough to release
without a prolonged testing and fixing period.

>   >   I considered that to be "a bridge too far", and
>   > instead decided to go with a safer, simpler alternative that we have
>   > now.
> 
> It sounds like you consider the current design as a _temporary_
> expedient.  If that's what it is, maybe writing this first was a good
> idea.
> 
> I'm talking about how the code should work after there is time to take
> stock and clean it up.

IMO, we currently lack not only time, but also (and more importantly)
the necessary experience and user feedback to decide how to do it
correctly.  We need to collect that experience before we decide on the
best arrangement.  Of course, if the minor-mode idea turns out to be
workable, we will seriously consider it; it's not like we rejected it
for good.

Re: Unifying "foo-mode"s and "foo-ts-mode"s

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > IMO, we currently lack not only time, but also (and more importantly)
  > the necessary experience and user feedback to decide how to do it
  > correctly.  We need to collect that experience before we decide on the
  > best arrangement.

I'm not saying you should do this by any particular deadline.  All
along I've been talking about how this interface should end up.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Unifying "foo-mode"s and "foo-ts-mode"s

[Gregory Heytings]

>
> To me the ideal would still be that tree-sitter support is provided by 
> the distribution, just like how package managers have started bundling 
> libgccjit for native compilation, instead of requiring the user enable 
> it themselves.
>

FWIW, I don't think this will happen.  Each of the three other editors I 
know of that already use tree-sitter (Atom, Helix, Neovim) bundle 
tree-sitter library files and/or provide a way to install them on demand 
from their repositories.