Make all tree-sitter modes optional

Make all tree-sitter modes optional

[Eli Zaretskii]

This came out of the discussion in bug#60559 and other related
discussions, where I said:

> So here's a suggestion for such a solution: we make all the
> *-ts-mode's optional.  That is, we don't add any of them to
> auto-mode-alist unless the file *-ts-mode.el is loaded, and we
> document them all in NEWS and the user manual as optional.  users who
> want them will have to manually activate them.  This way, the original
> use case that started this bug report is automatically solved, and the
> other use case, where the user intends to activate one of these modes,
> is also served by showing the warning, which in that case is perfectly
> justified: the user asked for something that we cannot do, so we warn
> him/her.
> 
> This is a retreat of sorts, but I think it strikes a better balance
> wrt user expectations, assuming not everyone will build with
> tree-sitter.

The proposed change to the current emacs-29 branch is below.  You will
see that where possible, just loading a TS mode modifies
auto-mode-alist if the tree-sitter support for that mode is available,
whereas for other modes auto-mode-alist is modified only when the mode
is actually turned on successfully for the first time.  This is
because some of the TS modes have their own *.el files, whereas others
share a .el file with other modes, and so loading that file doesn't
necessarily means the user wants to use the tree-sitter based mode.

With these changes, users are supposed to turn the mode manually when
they want to use it, or customize auto-mode-alist in their init files
to turn those modes automatically for certain files.  Modes that have
their own *.el files can be just 'require'd in the init file to modify
auto-mode-alist.

This arrangement is perhaps not ideal, but I think it is safe enough
and simple enough to go into Emacs 29.

Comments? objections? ideas for improvements?  I will install in a few
days, barring any serious problems.

Here's the patch:

diff --git a/etc/NEWS b/etc/NEWS
index d1ddd01..996134c 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -34,13 +34,14 @@ This feature existed in Emacs 28.1, but was less easy to request.
 
 +++
 ** Emacs can be built with the tree-sitter parsing library.
-This library, together with grammar libraries, provides incremental
-parsing capabilities for several popular programming languages and
-other formatted files.  Emacs built with this library offers major
-modes, described elsewhere in this file, that are based on the
-tree-sitter's parsers.  If you have the tree-sitter library
-installed, the configure script will automatically include it in the
-build; use '--without-tree-sitter' at configure time to disable that.
+This library, together with separate grammar libraries for each
+language, provides incremental parsing capabilities for several
+popular programming languages and other formatted files.  Emacs built
+with this library offers major modes, described elsewhere in this
+file, that are based on the tree-sitter's parsers.  If you have the
+tree-sitter library installed, the configure script will automatically
+include it in the build; use '--without-tree-sitter' at configure time
+to disable that.
 
 Emacs modes based on the tree-sitter library require an additional
 grammar library for each mode.  These grammar libraries provide the
@@ -3181,19 +3182,10 @@ indentation, and navigation by defuns based on parsing the buffer text
 by a tree-sitter parser.  Some major modes also offer support for
 Imenu and 'which-func'.
 
-Where major modes already exist in Emacs for editing certain kinds of
-files, the new modes based on tree-sitter are for now entirely
-optional, and you must turn them on manually, or customize
-'auto-mode-alist' to turn them on automatically.
-
-Where no major modes previously existed in Emacs for editing the kinds
-of files for which Emacs now provides a tree-sitter based mode, Emacs
-will now try to enable these new modes automatically when you visit
-such files, and will display a warning if the tree-sitter library or
-the parser grammar library is not available.  To prevent the warnings,
-either build Emacs with tree-sitter and install the grammar libraries,
-or customize 'auto-mode-alist' to specify some other major mode (or
-even 'fundamental-mode') for those kinds of files.
+The new modes based on tree-sitter are for now entirely optional, and
+you must turn them on manually, or load them in your init file, or
+customize 'auto-mode-alist' to turn them on automatically for certain
+files.
 
 Each major mode based on tree-sitter needs a language grammar library,
 usually named "libtree-sitter-LANG.so" ("libtree-sitter-LANG.dll" on
@@ -3210,20 +3202,18 @@ We recommend to install these libraries in one of the standard system
 locations (the last place in the above list).
 
 If a language grammar library required by a mode is not found in any
-of the above places, the mode will signal an error when you try to
+of the above places, the mode will display a warning when you try to
 turn it on.
 
 +++
 *** New major mode 'typescript-ts-mode'.
 A major mode based on the tree-sitter library for editing programs
-in the TypeScript language.  This mode is auto-enabled for files with
-the ".ts" extension.
+in the TypeScript language.
 
 +++
 *** New major mode 'tsx-ts-mode'.
 A major mode based on the tree-sitter library for editing programs
-in the TypeScript language, with support for TSX.  This mode is
-auto-enabled for files with the ".tsx" extension.
+in the TypeScript language, with support for TSX.
 
 +++
 *** New major mode 'c-ts-mode'.
@@ -3268,15 +3258,12 @@ Bash shell scripts.
 +++
 *** New major mode 'dockerfile-ts-mode'.
 A major mode based on the tree-sitter library for editing
-Dockerfiles.  This mode is auto-enabled for files which are named
-"Dockerfile", have the "Dockerfile." prefix, or have the ".dockerfile"
-extension.
+Dockerfiles.
 
 +++
 *** New major mode 'cmake-ts-mode'.
 A major mode based on the tree-sitter library for editing CMake files.
-It is auto-enabled for files whose name is "CMakeLists.txt" or whose
-extension is ".cmake".
+
 
 +++
 *** New major mode 'toml-ts-mode'.
@@ -3286,23 +3273,22 @@ files written in TOML, a format for writing configuration files.
 +++
 *** New major mode 'go-ts-mode'.
 A major mode based on the tree-sitter library for editing programs in
-the Go language.  It is auto-enabled for files with the ".go" extension.
+the Go language.
 
 +++
 *** New major mode 'go-mod-ts-mode'.
 A major mode based on the tree-sitter library for editing "go.mod"
-files.  It is auto-enabled for files which are named "go.mod".
+files.
 
 +++
 *** New major mode 'yaml-ts-mode'.
 A major mode based on the tree-sitter library for editing files
-written in YAML.  It is auto-enabled for files with the ".yaml" or
-".yml" extensions.
+written in YAML.
 
 +++
 *** New major mode 'rust-ts-mode'.
 A major mode based on the tree-sitter library for editing programs in
-the Rust language.  It is auto-enabled for files with the ".rs" extension.
+the Rust language.
 
 ---
 *** New major mode 'ruby-ts-mode'.
diff --git a/lisp/progmodes/c-ts-mode.el b/lisp/progmodes/c-ts-mode.el
index 89a08a6..d004920 100644
--- a/lisp/progmodes/c-ts-mode.el
+++ b/lisp/progmodes/c-ts-mode.el
@@ -972,6 +972,19 @@ c++-ts-mode
     (setq-local treesit-font-lock-settings (c-ts-mode--font-lock-settings 'cpp))
     (treesit-major-mode-setup)))
 
+;; The entries for C++ must come first to prevent *.c files be taken
+;; as C++ on case-insensitive filesystems, since *.C files are C++,
+;; not C.
+(if (treesit-ready-p 'cpp)
+    (add-to-list 'auto-mode-alist
+                 '("\\(\\.ii\\|\\.\\(CC?\\|HH?\\)\\|\\.[ch]\\(pp\\|xx\\|\\+\\+\\)\\|\\.\\(cc\\|hh\\)\\)\\'"
+                   . c++-ts-mode)))
+
+(if (treesit-ready-p 'c)
+    (add-to-list 'auto-mode-alist
+                 '("\\(\\.[chi]\\|\\.lex\\|\\.y\\(acc\\)?\\|\\.x[bp]m\\)\\'"
+                   . c-ts-mode)))
+
 (provide 'c-ts-mode)
 
 ;;; c-ts-mode.el ends here
diff --git a/lisp/progmodes/cmake-ts-mode.el b/lisp/progmodes/cmake-ts-mode.el
index a31250f..c241a28 100644
--- a/lisp/progmodes/cmake-ts-mode.el
+++ b/lisp/progmodes/cmake-ts-mode.el
@@ -195,10 +195,6 @@ cmake-ts-mode--imenu-1
       `((,name . ,marker))))))
 
 ;;;###autoload
-(add-to-list 'auto-mode-alist
-             '("\\(?:CMakeLists\\.txt\\|\\.cmake\\)\\'" . cmake-ts-mode))
-
-;;;###autoload
 (define-derived-mode cmake-ts-mode prog-mode "CMake"
   "Major mode for editing CMake files, powered by tree-sitter."
   :group 'cmake
@@ -229,6 +225,10 @@ cmake-ts-mode
 
     (treesit-major-mode-setup)))
 
+(if (treesit-ready-p 'cmake)
+    (add-to-list 'auto-mode-alist
+                 '("\\(?:CMakeLists\\.txt\\|\\.cmake\\)\\'" . cmake-ts-mode)))
+
 (provide 'cmake-ts-mode)
 
 ;;; cmake-ts-mode.el ends here
diff --git a/lisp/progmodes/csharp-mode.el b/lisp/progmodes/csharp-mode.el
index 81ce416..04f7f22 100644
--- a/lisp/progmodes/csharp-mode.el
+++ b/lisp/progmodes/csharp-mode.el
@@ -884,9 +884,6 @@ csharp-ts-mode--defun-name
       t))))
 
 ;;;###autoload
-(add-to-list 'auto-mode-alist '("\\.cs\\'" . csharp-mode))
-
-;;;###autoload
 (define-derived-mode csharp-mode prog-mode "C#"
   "Major mode for editing Csharp code.
 
@@ -941,7 +938,9 @@ csharp-ts-mode
                 ("Struct" "\\`struct_declaration\\'" nil nil)
                 ("Method" "\\`method_declaration\\'" nil nil)))
 
-  (treesit-major-mode-setup))
+  (treesit-major-mode-setup)
+
+  (add-to-list 'auto-mode-alist '("\\.cs\\'" . csharp-ts-mode)))
 
 (provide 'csharp-mode)
 
diff --git a/lisp/progmodes/dockerfile-ts-mode.el b/lisp/progmodes/dockerfile-ts-mode.el
index 3f8766e..2a295e8 100644
--- a/lisp/progmodes/dockerfile-ts-mode.el
+++ b/lisp/progmodes/dockerfile-ts-mode.el
@@ -133,12 +133,6 @@ dockerfile-ts-mode--imenu-1
       `((,name . ,marker))))))
 
 ;;;###autoload
-(add-to-list 'auto-mode-alist
-             ;; NOTE: We can't use `rx' here, as it breaks bootstrap.
-             '("\\(?:Dockerfile\\(?:\\..*\\)?\\|\\.[Dd]ockerfile\\)\\'"
-               . dockerfile-ts-mode))
-
-;;;###autoload
 (define-derived-mode dockerfile-ts-mode prog-mode "Dockerfile"
   "Major mode for editing Dockerfiles, powered by tree-sitter."
   :group 'dockerfile
@@ -172,6 +166,12 @@ dockerfile-ts-mode
 
     (treesit-major-mode-setup)))
 
+(if (treesit-ready-p 'dockerfile)
+    (add-to-list 'auto-mode-alist
+                 ;; NOTE: We can't use `rx' here, as it breaks bootstrap.
+                 '("\\(?:Dockerfile\\(?:\\..*\\)?\\|\\.[Dd]ockerfile\\)\\'"
+                   . dockerfile-ts-mode)))
+
 (provide 'dockerfile-ts-mode)
 
 ;;; dockerfile-ts-mode.el ends here
diff --git a/lisp/progmodes/go-ts-mode.el b/lisp/progmodes/go-ts-mode.el
index 64e761d..d552e13 100644
--- a/lisp/progmodes/go-ts-mode.el
+++ b/lisp/progmodes/go-ts-mode.el
@@ -175,9 +175,6 @@ go-ts-mode--font-lock-settings
   "Tree-sitter font-lock settings for `go-ts-mode'.")
 
 ;;;###autoload
-(add-to-list 'auto-mode-alist '("\\.go\\'" . go-ts-mode))
-
-;;;###autoload
 (define-derived-mode go-ts-mode prog-mode "Go"
   "Major mode for editing Go, powered by tree-sitter."
   :group 'go
@@ -226,6 +223,9 @@ go-ts-mode
 
     (treesit-major-mode-setup)))
 
+(if (treesit-ready-p 'go)
+    (add-to-list 'auto-mode-alist '("\\.go\\'" . go-ts-mode)))
+
 (defun go-ts-mode--defun-name (node)
   "Return the defun name of NODE.
 Return nil if there is no name or if NODE is not a defun node."
@@ -346,9 +346,6 @@ go-mod-ts-mode--font-lock-settings
   "Tree-sitter font-lock settings for `go-mod-ts-mode'.")
 
 ;;;###autoload
-(add-to-list 'auto-mode-alist '("/go\\.mod\\'" . go-mod-ts-mode))
-
-;;;###autoload
 (define-derived-mode go-mod-ts-mode prog-mode "Go Mod"
   "Major mode for editing go.mod files, powered by tree-sitter."
   :group 'go
@@ -376,6 +373,9 @@ go-mod-ts-mode
 
     (treesit-major-mode-setup)))
 
+(if (treesit-ready-p 'gomod)
+    (add-to-list 'auto-mode-alist '("/go\\.mod\\'" . go-mod-ts-mode)))
+
 (provide 'go-ts-mode)
 
 ;;; go-ts-mode.el ends here
diff --git a/lisp/progmodes/java-ts-mode.el b/lisp/progmodes/java-ts-mode.el
index d29fcd8..d909a36 100644
--- a/lisp/progmodes/java-ts-mode.el
+++ b/lisp/progmodes/java-ts-mode.el
@@ -331,6 +331,9 @@ java-ts-mode
                 ("Method" "\\`method_declaration\\'" nil nil)))
   (treesit-major-mode-setup))
 
+(if (treesit-ready-p 'java)
+    (add-to-list 'auto-mode-alist '("\\.java\\'" . java-ts-mode)))
+
 (provide 'java-ts-mode)
 
 ;;; java-ts-mode.el ends here
diff --git a/lisp/progmodes/js.el b/lisp/progmodes/js.el
index 058c890..4428672 100644
--- a/lisp/progmodes/js.el
+++ b/lisp/progmodes/js.el
@@ -3840,7 +3840,10 @@ js-ts-mode
                                         "method_definition")
                                 eos)
                    nil nil)))
-    (treesit-major-mode-setup)))
+    (treesit-major-mode-setup)
+
+    (add-to-list 'auto-mode-alist
+                 '("\\(\\.js[mx]\\|\\.har\\)\\'" . js-ts-mode))))
 
 ;;;###autoload
 (define-derived-mode js-json-mode js-mode "JSON"
diff --git a/lisp/progmodes/json-ts-mode.el b/lisp/progmodes/json-ts-mode.el
index fbcda22..f54d018 100644
--- a/lisp/progmodes/json-ts-mode.el
+++ b/lisp/progmodes/json-ts-mode.el
@@ -160,6 +160,10 @@ json-ts-mode
 
   (treesit-major-mode-setup))
 
+(if (treesit-ready-p 'json)
+    (add-to-list 'auto-mode-alist
+                 '("\\.json\\'" . json-ts-mode)))
+
 (provide 'json-ts-mode)
 
 ;;; json-ts-mode.el ends here
diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index 21d16db..a869cdc 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -6713,7 +6713,10 @@ python-ts-mode
     (treesit-major-mode-setup)
 
     (when python-indent-guess-indent-offset
-      (python-indent-guess-indent-offset))))
+      (python-indent-guess-indent-offset))
+
+    (add-to-list 'auto-mode-alist
+                 '("\\.py[iw]?\\'\\|python[0-9.]*" . python-ts-mode))))
 
 ;;; Completion predicates for M-x
 ;; Commands that only make sense when editing Python code
diff --git a/lisp/progmodes/ruby-ts-mode.el b/lisp/progmodes/ruby-ts-mode.el
index d68b579..f798f09 100644
--- a/lisp/progmodes/ruby-ts-mode.el
+++ b/lisp/progmodes/ruby-ts-mode.el
@@ -986,6 +986,10 @@ ruby-ts-mode
 
   (treesit-major-mode-setup))
 
+(if (treesit-ready-p 'ruby)
+    (add-to-list 'auto-mode-alist
+                 '("\\(?:\\.\\(?:rbw?\\|ru\\|rake\\|thor\\|jbuilder\\|rabl\\|gemspec\\|podspec\\)\\|/\\(?:Gem\\|Rake\\|Cap\\|Thor\\|Puppet\\|Berks\\|Brew\\|Vagrant\\|Guard\\|Pod\\)file\\)\\'" . ruby-ts-mode)))
+
 (provide 'ruby-ts-mode)
 
 ;;; ruby-ts-mode.el ends here
diff --git a/lisp/progmodes/rust-ts-mode.el b/lisp/progmodes/rust-ts-mode.el
index 7536726..08590ae 100644
--- a/lisp/progmodes/rust-ts-mode.el
+++ b/lisp/progmodes/rust-ts-mode.el
@@ -276,9 +276,6 @@ rust-ts-mode--defun-name
       (treesit-node-child-by-field-name node "name") t))))
 
 ;;;###autoload
-(add-to-list 'auto-mode-alist '("\\.rs\\'" . rust-ts-mode))
-
-;;;###autoload
 (define-derived-mode rust-ts-mode prog-mode "Rust"
   "Major mode for editing Rust, powered by tree-sitter."
   :group 'rust
@@ -322,6 +319,9 @@ rust-ts-mode
 
     (treesit-major-mode-setup)))
 
+(if (treesit-ready-p 'rust)
+    (add-to-list 'auto-mode-alist '("\\.rs\\'" . rust-ts-mode)))
+
 (provide 'rust-ts-mode)
 
 ;;; rust-ts-mode.el ends here
diff --git a/lisp/progmodes/typescript-ts-mode.el b/lisp/progmodes/typescript-ts-mode.el
index 037d5c8..0b0bbe1 100644
--- a/lisp/progmodes/typescript-ts-mode.el
+++ b/lisp/progmodes/typescript-ts-mode.el
@@ -313,12 +313,6 @@ typescript-ts-mode--font-lock-settings
    '((escape_sequence) @font-lock-escape-face)))
 
 ;;;###autoload
-(add-to-list 'auto-mode-alist '("\\.ts\\'" . typescript-ts-mode))
-
-;;;###autoload
-(add-to-list 'auto-mode-alist '("\\.tsx\\'" . tsx-ts-mode))
-
-;;;###autoload
 (define-derived-mode typescript-ts-base-mode prog-mode "TypeScript"
   "Major mode for editing TypeScript."
   :group 'typescript
@@ -373,6 +367,9 @@ typescript-ts-mode
 
     (treesit-major-mode-setup)))
 
+(if (treesit-ready-p 'typescript)
+    (add-to-list 'auto-mode-alist '("\\.ts\\'" . typescript-ts-mode)))
+
 ;;;###autoload
 (define-derived-mode tsx-ts-mode typescript-ts-base-mode "TypeScript[TSX]"
   "Major mode for editing TypeScript."
@@ -408,6 +405,9 @@ tsx-ts-mode
 
     (treesit-major-mode-setup)))
 
+(if (treesit-ready-p 'tsx)
+    (add-to-list 'auto-mode-alist '("\\.tsx\\'" . tsx-ts-mode)))
+
 (provide 'typescript-ts-mode)
 
 ;;; typescript-ts-mode.el ends here
diff --git a/lisp/textmodes/css-mode.el b/lisp/textmodes/css-mode.el
index 8991610..a1d7d4b 100644
--- a/lisp/textmodes/css-mode.el
+++ b/lisp/textmodes/css-mode.el
@@ -1827,7 +1827,9 @@ css-ts-mode
     (setq-local treesit-simple-imenu-settings
                 `(( nil ,(rx bos (or "rule_set" "media_statement") eos)
                     nil nil)))
-    (treesit-major-mode-setup)))
+    (treesit-major-mode-setup)
+
+    (add-to-list 'auto-mode-alist '("\\.css\\'" . css-ts-mode))))
 
 ;;;###autoload
 (define-derived-mode css-mode css-base-mode "CSS"
diff --git a/lisp/textmodes/toml-ts-mode.el b/lisp/textmodes/toml-ts-mode.el
index 2430c5f..4165420 100644
--- a/lisp/textmodes/toml-ts-mode.el
+++ b/lisp/textmodes/toml-ts-mode.el
@@ -117,8 +117,6 @@ toml-ts-mode--defun-name
      (or (treesit-node-text (treesit-node-child node 1) t)
          "Root table"))))
 
-(add-to-list 'auto-mode-alist '("\\.toml\\'" . toml-ts-mode))
-
 ;;;###autoload
 (define-derived-mode toml-ts-mode text-mode "TOML"
   "Major mode for editing TOML, powered by tree-sitter."
@@ -155,6 +153,9 @@ toml-ts-mode
 
     (treesit-major-mode-setup)))
 
+(if (treesit-ready-p 'toml)
+    (add-to-list 'auto-mode-alist '("\\.toml\\'" . toml-ts-mode)))
+
 (provide 'toml-ts-mode)
 
 ;;; toml-ts-mode.el ends here
diff --git a/lisp/textmodes/yaml-ts-mode.el b/lisp/textmodes/yaml-ts-mode.el
index 8c61ee0..a25230e 100644
--- a/lisp/textmodes/yaml-ts-mode.el
+++ b/lisp/textmodes/yaml-ts-mode.el
@@ -118,9 +118,6 @@ yaml-ts-mode--font-lock-settings
   "Tree-sitter font-lock settings for `yaml-ts-mode'.")
 
 ;;;###autoload
-(add-to-list 'auto-mode-alist '("\\.ya?ml\\'" . yaml-ts-mode))
-
-;;;###autoload
 (define-derived-mode yaml-ts-mode text-mode "YAML"
   "Major mode for editing YAML, powered by tree-sitter."
   :group 'yaml
@@ -146,6 +143,9 @@ yaml-ts-mode
 
     (treesit-major-mode-setup)))
 
+(if (treesit-ready-p 'yaml)
+    (add-to-list 'auto-mode-alist '("\\.ya?ml\\'" . yaml-ts-mode)))
+
 (provide 'yaml-ts-mode)
 
 ;;; yaml-ts-mode.el ends here

Re: Make all tree-sitter modes optional

[Dmitry Gutov]

On 15/01/2023 16:01, Eli Zaretskii wrote:
> You will
> see that where possible, just loading a TS mode modifies
> auto-mode-alist if the tree-sitter support for that mode is available,
> whereas for other modes auto-mode-alist is modified only when the mode
> is actually turned on successfully for the first time.  This is
> because some of the TS modes have their own *.el files, whereas others
> share a .el file with other modes, and so loading that file doesn't
> necessarily means the user wants to use the tree-sitter based mode.

If we *are* going to do this (make all ts mode strictly optional), I 
don't think either of this is a good idea: for a given foo-ts-mode, the 
user might already have an auto-mode-alist entry configured with another 
mode (third-party or not), and they will likely 'M-x foo-ts-mode' to try 
how well it works (or doesn't).

Having auto-mode-alist modified automatically can come as a surprise 
either way.

Note the this is different to having the auto-mode-alist entries set up 
from the outset, because the user's alterations override those.

We can drop the auto-mode-alist alterations from ts modes altogether, to 
bring them back when we do decide to enable them by default.

Re: Make all tree-sitter modes optional

[Theodor Thornhill]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 15/01/2023 16:01, Eli Zaretskii wrote:
>> You will
>> see that where possible, just loading a TS mode modifies
>> auto-mode-alist if the tree-sitter support for that mode is available,
>> whereas for other modes auto-mode-alist is modified only when the mode
>> is actually turned on successfully for the first time.  This is
>> because some of the TS modes have their own *.el files, whereas others
>> share a .el file with other modes, and so loading that file doesn't
>> necessarily means the user wants to use the tree-sitter based mode.
>
> If we *are* going to do this (make all ts mode strictly optional), I 
> don't think either of this is a good idea: for a given foo-ts-mode, the 
> user might already have an auto-mode-alist entry configured with another 
> mode (third-party or not), and they will likely 'M-x foo-ts-mode' to try 
> how well it works (or doesn't).
>
> Having auto-mode-alist modified automatically can come as a surprise 
> either way.
>
> Note the this is different to having the auto-mode-alist entries set up 
> from the outset, because the user's alterations override those.
>
> We can drop the auto-mode-alist alterations from ts modes altogether, to 
> bring them back when we do decide to enable them by default.

I'm fine with either, but I understood your proposal to just remove all
the auto-mode-alist additions for emacs-29, then let users config that
themselves.

Theo

Re: Make all tree-sitter modes optional

[Dmitry Gutov]

On 16/01/2023 09:43, Theodor Thornhill wrote:
> Dmitry Gutov<dgutov@yandex.ru>  writes:
> 
>> On 15/01/2023 16:01, Eli Zaretskii wrote:
>>> You will
>>> see that where possible, just loading a TS mode modifies
>>> auto-mode-alist if the tree-sitter support for that mode is available,
>>> whereas for other modes auto-mode-alist is modified only when the mode
>>> is actually turned on successfully for the first time.  This is
>>> because some of the TS modes have their own *.el files, whereas others
>>> share a .el file with other modes, and so loading that file doesn't
>>> necessarily means the user wants to use the tree-sitter based mode.
>> If we*are*  going to do this (make all ts mode strictly optional), I
>> don't think either of this is a good idea: for a given foo-ts-mode, the
>> user might already have an auto-mode-alist entry configured with another
>> mode (third-party or not), and they will likely 'M-x foo-ts-mode' to try
>> how well it works (or doesn't).
>>
>> Having auto-mode-alist modified automatically can come as a surprise
>> either way.
>>
>> Note the this is different to having the auto-mode-alist entries set up
>> from the outset, because the user's alterations override those.
>>
>> We can drop the auto-mode-alist alterations from ts modes altogether, to
>> bring them back when we do decide to enable them by default.
> I'm fine with either, but I understood your proposal to just remove all
> the auto-mode-alist additions for emacs-29, then let users config that
> themselves.

My preferred course of action was to keep things as-is, but since that's 
not in the cards, the proposed alternative for emacs-29 was to indeed 
remove the auto-mode-alist additions entirely, to keep things simple.

Re: Make all tree-sitter modes optional

[Eli Zaretskii]

> Date: Mon, 16 Jan 2023 01:39:18 +0200
> Cc: emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 15/01/2023 16:01, Eli Zaretskii wrote:
> > You will
> > see that where possible, just loading a TS mode modifies
> > auto-mode-alist if the tree-sitter support for that mode is available,
> > whereas for other modes auto-mode-alist is modified only when the mode
> > is actually turned on successfully for the first time.  This is
> > because some of the TS modes have their own *.el files, whereas others
> > share a .el file with other modes, and so loading that file doesn't
> > necessarily means the user wants to use the tree-sitter based mode.
> 
> If we *are* going to do this (make all ts mode strictly optional), I 
> don't think either of this is a good idea: for a given foo-ts-mode, the 
> user might already have an auto-mode-alist entry configured with another 
> mode (third-party or not), and they will likely 'M-x foo-ts-mode' to try 
> how well it works (or doesn't).
> 
> Having auto-mode-alist modified automatically can come as a surprise 
> either way.
> 
> Note the this is different to having the auto-mode-alist entries set up 
> from the outset, because the user's alterations override those.
> 
> We can drop the auto-mode-alist alterations from ts modes altogether, to 
> bring them back when we do decide to enable them by default.

Like I said: this is not ideal.  So I'm not surprised that whatever we
do, there can be usage scenarios where what we do will annoy someone.

That said:

  . The proposed patch made many changes of auto-mode-alist
    conditional where they previously were _un_conditional.  I submit
    that this is less annoyance in the important use case where the
    tree-sitter or the grammar library is not available.  There are
    indications that this situation will be frequent enough when Emacs
    29.1 hits the street.

  . I don't buy the assumption that customizations of auto-mode-alist
    are frequent enough to make that an important factor in these
    decisions, let alone suggest that users should always do that if
    they want to try the *-ts-* modes seriously:

      - IME, auto-mode-alist is relatively rarely customized for modes
        that are included in Emacs (e.g., I don't customize entries of
        any such modes), for the simple reason that it is very rarely
        needed.
      - Customizing auto-mode-alist is not the easiest task, it
        requires good knowledge of Emacs regexps and alists.  So
        asking anyone who wants to try using the tree-sitter modes to
        do that is not the best idea from the POV of user-friendliness.
      - OTOH, I'm quite sure that people who do already customize
        auto-mode-alist for built-in modes are more advanced users and
        will be able to overcome any problems that could be caused by
        modifying auto-mode-alist as side effect of activating the
        mode.

  . Last, but not least: I think someone who turns on a tree-sitter
    mode in some buffer is much more likely to want to use that mode
    in more than that single buffer than the other way around.  And if
    for some reason they are disappointed soon enough, just restarting
    the Emacs session will get them back the old behavior.  Which
    again tells me that if we accept your proposal, we will annoy more
    users (with the need to modify auto-mode-alist) than under my
    proposal.

So, on balance, I think what I proposed is better than what you
propose.

Re: Make all tree-sitter modes optional

[Dmitry Gutov]

On 16/01/2023 14:34, Eli Zaretskii wrote:
>> Date: Mon, 16 Jan 2023 01:39:18 +0200
>> Cc: emacs-devel@gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>> On 15/01/2023 16:01, Eli Zaretskii wrote:
>>> You will
>>> see that where possible, just loading a TS mode modifies
>>> auto-mode-alist if the tree-sitter support for that mode is available,
>>> whereas for other modes auto-mode-alist is modified only when the mode
>>> is actually turned on successfully for the first time.  This is
>>> because some of the TS modes have their own *.el files, whereas others
>>> share a .el file with other modes, and so loading that file doesn't
>>> necessarily means the user wants to use the tree-sitter based mode.
>>
>> If we *are* going to do this (make all ts mode strictly optional), I
>> don't think either of this is a good idea: for a given foo-ts-mode, the
>> user might already have an auto-mode-alist entry configured with another
>> mode (third-party or not), and they will likely 'M-x foo-ts-mode' to try
>> how well it works (or doesn't).
>>
>> Having auto-mode-alist modified automatically can come as a surprise
>> either way.
>>
>> Note the this is different to having the auto-mode-alist entries set up
>> from the outset, because the user's alterations override those.
>>
>> We can drop the auto-mode-alist alterations from ts modes altogether, to
>> bring them back when we do decide to enable them by default.
> 
> Like I said: this is not ideal.  So I'm not surprised that whatever we
> do, there can be usage scenarios where what we do will annoy someone.
> 
> That said:
> 
>    . The proposed patch made many changes of auto-mode-alist
>      conditional where they previously were _un_conditional.

It also added some where there were none.

> I submit
>      that this is less annoyance in the important use case where the
>      tree-sitter or the grammar library is not available.  There are
>      indications that this situation will be frequent enough when Emacs
>      29.1 hits the street.

Like I said, either making them unconditional, or removing them, would 
lead to better, more predictable behavior.

>    . I don't buy the assumption that customizations of auto-mode-alist
>      are frequent enough to make that an important factor in these
>      decisions, let alone suggest that users should always do that if
>      they want to try the *-ts-* modes seriously:

Why wouldn't they be frequent? It's the only way for the user to have 
file format supported, where we don't support it OOTB.

The customization of auto-mode-alist might also happen for the user 
automatically. When the user installs a third-party package, such as 
json-mode, from ELPA, that modified auto-mode-alist through package 
autoloads.

Having it modified again by 'M-x yaml-ts-mode', but only for the 
duration of the current session, would be surprising and odd. If I were 
a new user, I would be questioning both why the mode association 
changed, and why it didn't persist between sessions.

>        - IME, auto-mode-alist is relatively rarely customized for modes
>          that are included in Emacs (e.g., I don't customize entries of
>          any such modes), for the simple reason that it is very rarely
>          needed.

I'm talking about modes which are not included in Emacs.

But having the association for .rb files rewritten when somebody invokes 
'M-x ruby-ts-mode' would be weird too. It will make it a pain in the ass 
for me to test ruby-ts-mode, for one thing.

Or js-mode -> js-ts-mode, or python-mode -> python-ts-mode. People 
should be allowed to experiment without having to figure out how one can 
undo an automated change like that.

>        - Customizing auto-mode-alist is not the easiest task, it
>          requires good knowledge of Emacs regexps and alists.  So
>          asking anyone who wants to try using the tree-sitter modes to
>          do that is not the best idea from the POV of user-friendliness.

They will have to learn how to do that anyway, because the 
auto-mode-alist alterations from your patch won't persist between sessions.

To make it easier, we could put examples in the Commentary of each ts 
mode. That's something that has been a common practice with third-party 
mode, so there are a lot of examples of this code out there anyway.

>        - OTOH, I'm quite sure that people who do already customize
>          auto-mode-alist for built-in modes are more advanced users and
>          will be able to overcome any problems that could be caused by
>          modifying auto-mode-alist as side effect of activating the
>          mode.

Adding elements to auto-mode-alist is easier than removing them.

>    . Last, but not least: I think someone who turns on a tree-sitter
>      mode in some buffer is much more likely to want to use that mode
>      in more than that single buffer than the other way around.  And if
>      for some reason they are disappointed soon enough, just restarting
>      the Emacs session will get them back the old behavior.  Which
>      again tells me that if we accept your proposal, we will annoy more
>      users (with the need to modify auto-mode-alist) than under my
>      proposal.

Someone who turns it on for the first time doesn't know yet whether they 
want to use it or not. It hard to gauge the percentage of those who will.

And again, those who will want to, will have to learn how to deal with 
auto-mode-alist anyway. For some modes that could be avoided with e.g. 
(require 'yaml-ts-mode), but for others (e.g. python-ts-mode) it could 
not. Why proliferate special cases?

Re: Make all tree-sitter modes optional

[Eli Zaretskii]

> Date: Mon, 16 Jan 2023 15:06:15 +0200
> Cc: casouri@gmail.com, monnier@iro.umontreal.ca, larsi@gnus.org,
>  theo@thornhill.no, jostein@secure.kjonigsen.net, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> >    . The proposed patch made many changes of auto-mode-alist
> >      conditional where they previously were _un_conditional.
> 
> It also added some where there were none.

Yes, so that we do this consistently.  Inconsistency would lead to
more confusion and difficulties in documentation.

> > I submit
> >      that this is less annoyance in the important use case where the
> >      tree-sitter or the grammar library is not available.  There are
> >      indications that this situation will be frequent enough when Emacs
> >      29.1 hits the street.
> 
> Like I said, either making them unconditional, or removing them, would 
> lead to better, more predictable behavior.

OK, so let's disagree about this.

> >    . I don't buy the assumption that customizations of auto-mode-alist
> >      are frequent enough to make that an important factor in these
> >      decisions, let alone suggest that users should always do that if
> >      they want to try the *-ts-* modes seriously:
> 
> Why wouldn't they be frequent? It's the only way for the user to have 
> file format supported, where we don't support it OOTB.

I was talking about modes included in Emacs.

> The customization of auto-mode-alist might also happen for the user 
> automatically. When the user installs a third-party package, such as 
> json-mode, from ELPA, that modified auto-mode-alist through package 
> autoloads.
> 
> Having it modified again by 'M-x yaml-ts-mode', but only for the 
> duration of the current session, would be surprising and odd. If I were 
> a new user, I would be questioning both why the mode association 
> changed, and why it didn't persist between sessions.

We disagree again.  I guess we will have to wait and see who is right.

> >        - IME, auto-mode-alist is relatively rarely customized for modes
> >          that are included in Emacs (e.g., I don't customize entries of
> >          any such modes), for the simple reason that it is very rarely
> >          needed.
> 
> I'm talking about modes which are not included in Emacs.

Not relevant to this discussion.

Re: Make all tree-sitter modes optional

[Dmitry Gutov]

On 16/01/2023 16:20, Eli Zaretskii wrote:
> We disagree again.  I guess we will have to wait and see who is right.

How are we going to determine that?

If you're waiting for bug reports, I might as well file one now.

Given that you're making this change on the basis of just one complaint, 
that would make it 1:1.

Re: Make all tree-sitter modes optional

[Eli Zaretskii]

> Date: Mon, 16 Jan 2023 16:50:07 +0200
> Cc: casouri@gmail.com, monnier@iro.umontreal.ca, larsi@gnus.org,
>  theo@thornhill.no, jostein@secure.kjonigsen.net, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 16/01/2023 16:20, Eli Zaretskii wrote:
> > We disagree again.  I guess we will have to wait and see who is right.
> 
> How are we going to determine that?
> 
> If you're waiting for bug reports, I might as well file one now.

You can file a bug report if you want, but it won't count because it
won't tell anything we don't know already.

This isn't a parliament, so the number of votes doesn't necessarily
count.

> Given that you're making this change on the basis of just one complaint, 
> that would make it 1:1.

I'm making this change because that one complaint told me something I
didn't know and didn't take into consideration before.  IOW, I've
changed my mind, and that's why I made these changes.

Re: Make all tree-sitter modes optional

[Dmitry Gutov]

On 16/01/2023 16:59, Eli Zaretskii wrote:
> You can file a bug report if you want, but it won't count because it
> won't tell anything we don't know already.

The fact that it will make it a pain for me to test ruby-ts-mode in 
"regular" sessions was something you'd been aware of already?

Re: Make all tree-sitter modes optional

[Eli Zaretskii]

> Date: Tue, 17 Jan 2023 14:59:06 +0200
> Cc: casouri@gmail.com, monnier@iro.umontreal.ca, larsi@gnus.org,
>  theo@thornhill.no, jostein@secure.kjonigsen.net, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 16/01/2023 16:59, Eli Zaretskii wrote:
> > You can file a bug report if you want, but it won't count because it
> > won't tell anything we don't know already.
> 
> The fact that it will make it a pain for me to test ruby-ts-mode in 
> "regular" sessions was something you'd been aware of already?

Please tell more details about this aspect of the issue.

Re: Make all tree-sitter modes optional

[Dmitry Gutov]

On 17/01/2023 15:47, Eli Zaretskii wrote:
>> Date: Tue, 17 Jan 2023 14:59:06 +0200
>> Cc: casouri@gmail.com, monnier@iro.umontreal.ca, larsi@gnus.org,
>>   theo@thornhill.no, jostein@secure.kjonigsen.net, emacs-devel@gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>> On 16/01/2023 16:59, Eli Zaretskii wrote:
>>> You can file a bug report if you want, but it won't count because it
>>> won't tell anything we don't know already.
>>
>> The fact that it will make it a pain for me to test ruby-ts-mode in
>> "regular" sessions was something you'd been aware of already?
> 
> Please tell more details about this aspect of the issue.

I'm using ruby-mode, at least for now, while all the wrinkles with 
indentation haven't been ironed out (and we'll probably not manage to 
get them all 100% right before the 29 release), and while ts modes don't 
support show-paren-mode like SMIE does. No proper sexp navigation, etc.

So here I am, in my normal Emacs, working. Suppose a bug report comes 
in, I switch to a new buffer (maybe visiting a pre-existing file, maybe 
not), turn on ruby-ts-mode, reproduce it. Then try to fix it.

 From that moment on, my current session has a different major mode 
associated with Ruby files, and I have to deal with that somehow.

Or a different scenario:

Recently I had to investigate worse font-lock performance of 
ruby-ts-mode compared to ruby-mode. How did I test that? I started a new 
session and visited a bunch of existing files. First I run the benchmark 
in ruby-mode (which it's associated with by default), then I switch to 
ruby-ts-mode, repeat the benchmark, compare the results. And I do that 
for a number of files.

With your change, the first file will start with ruby-mode, but the 
second file and the rest will start in ruby-ts-mode. And I would somehow 
need to remember that change and account for it when evaluating the results.

What's your recommendation?

I suppose I could add these two forms to the init script:

   (require 'ruby-ts-mode)
   (add-to-list 'auto-mode-alist <...huge regexp from ruby-mode.el...>)

...and then update it over the years if new entries are added there over 
the years -- by the way, having a separate regexp in ruby-ts-mode.el is 
an unfortunate duplication.

But that will only affect scenario 1. The second scenario is fairly 
likely to start with 'emacs -Q' (to try to eliminate the effect of user 
customizations).

Also try to imagine some prospective ruby-ts-mode contributor who is not 
one of us, but who's also trying to test and compare the two modes.

Re: Make all tree-sitter modes optional

[Eli Zaretskii]

> Date: Tue, 17 Jan 2023 16:32:30 +0200
> Cc: casouri@gmail.com, monnier@iro.umontreal.ca, larsi@gnus.org,
>  theo@thornhill.no, jostein@secure.kjonigsen.net, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> I'm using ruby-mode, at least for now, while all the wrinkles with 
> indentation haven't been ironed out (and we'll probably not manage to 
> get them all 100% right before the 29 release), and while ts modes don't 
> support show-paren-mode like SMIE does. No proper sexp navigation, etc.
> 
> So here I am, in my normal Emacs, working. Suppose a bug report comes 
> in, I switch to a new buffer (maybe visiting a pre-existing file, maybe 
> not), turn on ruby-ts-mode, reproduce it. Then try to fix it.
> 
>  From that moment on, my current session has a different major mode 
> associated with Ruby files, and I have to deal with that somehow.
> 
> Or a different scenario:
> 
> Recently I had to investigate worse font-lock performance of 
> ruby-ts-mode compared to ruby-mode. How did I test that? I started a new 
> session and visited a bunch of existing files. First I run the benchmark 
> in ruby-mode (which it's associated with by default), then I switch to 
> ruby-ts-mode, repeat the benchmark, compare the results. And I do that 
> for a number of files.
> 
> With your change, the first file will start with ruby-mode, but the 
> second file and the rest will start in ruby-ts-mode. And I would somehow 
> need to remember that change and account for it when evaluating the results.
> 
> What's your recommendation?

My recommendation is to use separate, throw-away Emacs sessions for
any such testing.  I'm doing that myself for a long time, for reasons
unrelated to the issue at hand -- simply because (a) my production
sessions are too customized for clean-room testing, and (b) because my
production sessions are too valuable to me to risk loading non-trivial
packages that change the defaults.

I'm frankly surprised that this is not what you do in your testing.  I
was quite sure that everyone who does any serious development or
maintenance work in Emacs does something like that.  How else is it
possible to, e.g., load some obscure package someone says is necessary
for reproducing a problem?

So in your case, when I'm done with testing whatever I need to test in
ruby-ts-mode, I ether shut down that session, or start another one in
parallel if I need to compare it with, say, ruby-mode.

> I suppose I could add these two forms to the init script:
> 
>    (require 'ruby-ts-mode)
>    (add-to-list 'auto-mode-alist <...huge regexp from ruby-mode.el...>)
> 
> ...and then update it over the years if new entries are added there over 
> the years -- by the way, having a separate regexp in ruby-ts-mode.el is 
> an unfortunate duplication.

If this is about the difference between the regexps, we can fix that,
I don't object to have both modes handle the same files.

> Also try to imagine some prospective ruby-ts-mode contributor who is not 
> one of us, but who's also trying to test and compare the two modes.

My suggestion to anyone who does such testing is to use separate
sessions.

Re: Make all tree-sitter modes optional

[Dmitry Gutov]

On 17/01/2023 16:52, Eli Zaretskii wrote:
>> Date: Tue, 17 Jan 2023 16:32:30 +0200
>> Cc: casouri@gmail.com, monnier@iro.umontreal.ca, larsi@gnus.org,
>>   theo@thornhill.no, jostein@secure.kjonigsen.net, emacs-devel@gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>> I'm using ruby-mode, at least for now, while all the wrinkles with
>> indentation haven't been ironed out (and we'll probably not manage to
>> get them all 100% right before the 29 release), and while ts modes don't
>> support show-paren-mode like SMIE does. No proper sexp navigation, etc.
>>
>> So here I am, in my normal Emacs, working. Suppose a bug report comes
>> in, I switch to a new buffer (maybe visiting a pre-existing file, maybe
>> not), turn on ruby-ts-mode, reproduce it. Then try to fix it.
>>
>>   From that moment on, my current session has a different major mode
>> associated with Ruby files, and I have to deal with that somehow.
>>
>> Or a different scenario:
>>
>> Recently I had to investigate worse font-lock performance of
>> ruby-ts-mode compared to ruby-mode. How did I test that? I started a new
>> session and visited a bunch of existing files. First I run the benchmark
>> in ruby-mode (which it's associated with by default), then I switch to
>> ruby-ts-mode, repeat the benchmark, compare the results. And I do that
>> for a number of files.
>>
>> With your change, the first file will start with ruby-mode, but the
>> second file and the rest will start in ruby-ts-mode. And I would somehow
>> need to remember that change and account for it when evaluating the results.
>>
>> What's your recommendation?
> 
> My recommendation is to use separate, throw-away Emacs sessions for
> any such testing.  I'm doing that myself for a long time, for reasons
> unrelated to the issue at hand -- simply because (a) my production
> sessions are too customized for clean-room testing, and (b) because my
> production sessions are too valuable to me to risk loading non-trivial
> packages that change the defaults.

I just mentioned in the previous message that scenario 2 is likely to 
start with 'emacs -Q'.

That doesn't solve the described problem in that scenario.

> I'm frankly surprised that this is not what you do in your testing.  I
> was quite sure that everyone who does any serious development or
> maintenance work in Emacs does something like that.  How else is it
> possible to, e.g., load some obscure package someone says is necessary
> for reproducing a problem?
> 
> So in your case, when I'm done with testing whatever I need to test in
> ruby-ts-mode, I ether shut down that session, or start another one in
> parallel if I need to compare it with, say, ruby-mode.

That's untenable.

Running a benchmark is evaluating a form like

   (benchmark-run 1000 (progn (font-lock-mode -1) (font-lock-mode 1) 
(font-lock-ensure)))

I can start a new session for an investigation, but I'm not going to 
restart Emacs every time I evaluate a form.

Doing the benchmarks in a different order (e.g. go through the files 
with one mode and then restart and go with another) is also only an 
option if I were to note the numbers on e.g. a piece of paper. I rarely 
do that; that would also slow me down compared to the current practice.

And also speaking of using 'emacs -Q', that's well-suited to testing 
some classes of bugs, and not so much for others.

>> I suppose I could add these two forms to the init script:
>>
>>     (require 'ruby-ts-mode)
>>     (add-to-list 'auto-mode-alist <...huge regexp from ruby-mode.el...>)
>>
>> ...and then update it over the years if new entries are added there over
>> the years -- by the way, having a separate regexp in ruby-ts-mode.el is
>> an unfortunate duplication.
> 
> If this is about the difference between the regexps, we can fix that,
> I don't object to have both modes handle the same files.

A common constant would help, with that particular aspect.

Re: Make all tree-sitter modes optional

[Eli Zaretskii]

> Date: Tue, 17 Jan 2023 17:22:05 +0200
> Cc: casouri@gmail.com, monnier@iro.umontreal.ca, larsi@gnus.org,
>  theo@thornhill.no, jostein@secure.kjonigsen.net, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> > I'm frankly surprised that this is not what you do in your testing.  I
> > was quite sure that everyone who does any serious development or
> > maintenance work in Emacs does something like that.  How else is it
> > possible to, e.g., load some obscure package someone says is necessary
> > for reproducing a problem?
> > 
> > So in your case, when I'm done with testing whatever I need to test in
> > ruby-ts-mode, I ether shut down that session, or start another one in
> > parallel if I need to compare it with, say, ruby-mode.
> 
> That's untenable.
> 
> Running a benchmark is evaluating a form like
> 
>    (benchmark-run 1000 (progn (font-lock-mode -1) (font-lock-mode 1) 
> (font-lock-ensure)))
> 
> I can start a new session for an investigation, but I'm not going to 
> restart Emacs every time I evaluate a form.

Why not?  It's easy and quick and solves all the problems you
mentioned (and then some).  Like I said: I'm using this myself all the
time.

> Doing the benchmarks in a different order (e.g. go through the files 
> with one mode and then restart and go with another) is also only an 
> option if I were to note the numbers on e.g. a piece of paper. I rarely 
> do that; that would also slow me down compared to the current practice.

No need for paper: just M-w the data and yank into your production
session (which stays up and running all the time).  Again, I'm doing
this all the time in my work on Emacs.

> And also speaking of using 'emacs -Q', that's well-suited to testing 
> some classes of bugs, and not so much for others.

You can start from "emacs -Q" and load whatever is needed.  You can
make an ad-hoc init file that loads everything you need automatically,
to save manual typing.  I'm doing this all the time when the setup is
complicated.

> >>     (require 'ruby-ts-mode)
> >>     (add-to-list 'auto-mode-alist <...huge regexp from ruby-mode.el...>)
> >>
> >> ...and then update it over the years if new entries are added there over
> >> the years -- by the way, having a separate regexp in ruby-ts-mode.el is
> >> an unfortunate duplication.
> > 
> > If this is about the difference between the regexps, we can fix that,
> > I don't object to have both modes handle the same files.
> 
> A common constant would help, with that particular aspect.

Consider it done.

Re: Make all tree-sitter modes optional

[Dmitry Gutov]

On 17/01/2023 19:02, Eli Zaretskii wrote:
>> Date: Tue, 17 Jan 2023 17:22:05 +0200
>> Cc: casouri@gmail.com, monnier@iro.umontreal.ca, larsi@gnus.org,
>>   theo@thornhill.no, jostein@secure.kjonigsen.net, emacs-devel@gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>>> I'm frankly surprised that this is not what you do in your testing.  I
>>> was quite sure that everyone who does any serious development or
>>> maintenance work in Emacs does something like that.  How else is it
>>> possible to, e.g., load some obscure package someone says is necessary
>>> for reproducing a problem?
>>>
>>> So in your case, when I'm done with testing whatever I need to test in
>>> ruby-ts-mode, I ether shut down that session, or start another one in
>>> parallel if I need to compare it with, say, ruby-mode.
>>
>> That's untenable.
>>
>> Running a benchmark is evaluating a form like
>>
>>     (benchmark-run 1000 (progn (font-lock-mode -1) (font-lock-mode 1)
>> (font-lock-ensure)))
>>
>> I can start a new session for an investigation, but I'm not going to
>> restart Emacs every time I evaluate a form.
> 
> Why not?  It's easy and quick and solves all the problems you
> mentioned (and then some).

That would increase the time and effort required for such an 
investigation ~5x. (*)

 > Like I said: I'm using this myself all the
 > time.

I'm reasonably certain that, for the tasks that I do, my workflows are 
better optimized.

>> Doing the benchmarks in a different order (e.g. go through the files
>> with one mode and then restart and go with another) is also only an
>> option if I were to note the numbers on e.g. a piece of paper. I rarely
>> do that; that would also slow me down compared to the current practice.
> 
> No need for paper: just M-w the data and yank into your production
> session (which stays up and running all the time).  Again, I'm doing
> this all the time in my work on Emacs.

Likewise (*), but with a smaller multiplier.

>> And also speaking of using 'emacs -Q', that's well-suited to testing
>> some classes of bugs, and not so much for others.
> 
> You can start from "emacs -Q" and load whatever is needed.  You can
> make an ad-hoc init file that loads everything you need automatically,
> to save manual typing.  I'm doing this all the time when the setup is
> complicated.

And now more people will have to, in less complicated situations, which 
previously required no such preparation.

Re: Make all tree-sitter modes optional

[Eli Zaretskii]

> Date: Tue, 17 Jan 2023 19:10:45 +0200
> Cc: casouri@gmail.com, monnier@iro.umontreal.ca, larsi@gnus.org,
>  theo@thornhill.no, jostein@secure.kjonigsen.net, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> >> I can start a new session for an investigation, but I'm not going to
> >> restart Emacs every time I evaluate a form.
> > 
> > Why not?  It's easy and quick and solves all the problems you
> > mentioned (and then some).
> 
> That would increase the time and effort required for such an 
> investigation ~5x. (*)

But testing in a session that is not clean is not recommended.  For
reasons more important than the issue at hand...

However, I'm the last person to tell others how to organize their
workflows, so I'll leave it at that.

> > You can start from "emacs -Q" and load whatever is needed.  You can
> > make an ad-hoc init file that loads everything you need automatically,
> > to save manual typing.  I'm doing this all the time when the setup is
> > complicated.
> 
> And now more people will have to, in less complicated situations, which 
> previously required no such preparation.

Maybe.  Like I said, the solution I proposed is not ideal, I just
don't see a clearly better one.

Re: Make all tree-sitter modes optional

[Dmitry Gutov]

On 17/01/2023 19:38, Eli Zaretskii wrote:
>> Date: Tue, 17 Jan 2023 19:10:45 +0200
>> Cc: casouri@gmail.com, monnier@iro.umontreal.ca, larsi@gnus.org,
>>   theo@thornhill.no, jostein@secure.kjonigsen.net, emacs-devel@gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>>>> I can start a new session for an investigation, but I'm not going to
>>>> restart Emacs every time I evaluate a form.
>>>
>>> Why not?  It's easy and quick and solves all the problems you
>>> mentioned (and then some).
>>
>> That would increase the time and effort required for such an
>> investigation ~5x. (*)
> 
> But testing in a session that is not clean is not recommended.  For
> reasons more important than the issue at hand...

With some experience, one can usually sort issues into those that 
require clean environment, those than might use it, and those that most 
likely don't. In the end, that saves time.

> However, I'm the last person to tell others how to organize their
> workflows, so I'll leave it at that.

Sure.

>>> You can start from "emacs -Q" and load whatever is needed.  You can
>>> make an ad-hoc init file that loads everything you need automatically,
>>> to save manual typing.  I'm doing this all the time when the setup is
>>> complicated.
>>
>> And now more people will have to, in less complicated situations, which
>> previously required no such preparation.
> 
> Maybe.  Like I said, the solution I proposed is not ideal, I just
> don't see a clearly better one.

Every approach we can take will annoy somebody, like like in SPC xkcd.

But dropping the auto-mode-alist modifications looks perfectly in line 
with the conservative approach we have used in the past, where features 
are rolled out gradually.

I also don't fully understand the benefits of your proposal. Suppose we 
apply it. You talked about how easier it will be to document the new 
behaviors if all ts modes are consistent. Okay. What are we going to say 
in that documentation?

Let's say there are two users, Bob and Alice. Bob has tried out 
yaml-ts-mode and wants to use it regularly. Alice has tried out 
js-ts-mode and also wants to use it from now on. What will be our 
recommendations for them to make that happen?

Re: Make all tree-sitter modes optional

[Eli Zaretskii]

> Date: Tue, 17 Jan 2023 19:59:04 +0200
> Cc: casouri@gmail.com, monnier@iro.umontreal.ca, larsi@gnus.org,
>  theo@thornhill.no, jostein@secure.kjonigsen.net, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> I also don't fully understand the benefits of your proposal. Suppose we 
> apply it. You talked about how easier it will be to document the new 
> behaviors if all ts modes are consistent. Okay. What are we going to say 
> in that documentation?
> 
> Let's say there are two users, Bob and Alice. Bob has tried out 
> yaml-ts-mode and wants to use it regularly. Alice has tried out 
> js-ts-mode and also wants to use it from now on. What will be our 
> recommendations for them to make that happen?

That's already in the patch that I posted.  Suggestions for improving
it are welcome.

Re: Make all tree-sitter modes optional

[Dmitry Gutov]

On 17/01/2023 20:04, Eli Zaretskii wrote:
>> Date: Tue, 17 Jan 2023 19:59:04 +0200
>> Cc:casouri@gmail.com,monnier@iro.umontreal.ca,larsi@gnus.org,
>>   theo@thornhill.no,jostein@secure.kjonigsen.net,emacs-devel@gnu.org
>> From: Dmitry Gutov<dgutov@yandex.ru>
>>
>> I also don't fully understand the benefits of your proposal. Suppose we
>> apply it. You talked about how easier it will be to document the new
>> behaviors if all ts modes are consistent. Okay. What are we going to say
>> in that documentation?
>>
>> Let's say there are two users, Bob and Alice. Bob has tried out
>> yaml-ts-mode and wants to use it regularly. Alice has tried out
>> js-ts-mode and also wants to use it from now on. What will be our
>> recommendations for them to make that happen?
> That's already in the patch that I posted.  Suggestions for improving
> it are welcome.

This part, right?

+The new modes based on tree-sitter are for now entirely optional, and
+you must turn them on manually, or load them in your init file, or
+customize 'auto-mode-alist' to turn them on automatically for certain
+files.

I thought there would be something more in the docs, given that the new 
behavior is unusual.

But I see you are referring to auto-mode-alist here, modifying which 
will still be necessary for js-ts-mode and python-ts-mode. Which will 
touch a lot of users, possibly even the majority of tree-sitter 
enthusiasts, given that JS and Python are some of the most popular 
languages these days.

And yet you rejected my counter-proposal claiming (if I got your 
position right) that modying auto-mode-alist is difficult/annoying/etc 
for an average user. To quote:

       - Customizing auto-mode-alist is not the easiest task, it
         requires good knowledge of Emacs regexps and alists.  So
         asking anyone who wants to try using the tree-sitter modes to
         do that is not the best idea from the POV of user-friendliness.

So which is it?

To "try out" tree-sitter modes, the users can 'M-x js-ts-mode' or etc 
either way. But to switch to the said mode, they will need to deal with 
auto-mode-alist, again, without any alternative.

Re: Make all tree-sitter modes optional

[Eli Zaretskii]

> Date: Tue, 17 Jan 2023 20:21:18 +0200
> Cc: casouri@gmail.com, monnier@iro.umontreal.ca, larsi@gnus.org,
>  theo@thornhill.no, jostein@secure.kjonigsen.net, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> +The new modes based on tree-sitter are for now entirely optional, and
> +you must turn them on manually, or load them in your init file, or
> +customize 'auto-mode-alist' to turn them on automatically for certain
> +files.
> 
> I thought there would be something more in the docs, given that the new 
> behavior is unusual.

What do you suggest to add?

> But I see you are referring to auto-mode-alist here, modifying which 
> will still be necessary for js-ts-mode and python-ts-mode. Which will 
> touch a lot of users, possibly even the majority of tree-sitter 
> enthusiasts, given that JS and Python are some of the most popular 
> languages these days.
> 
> And yet you rejected my counter-proposal claiming (if I got your 
> position right) that modying auto-mode-alist is difficult/annoying/etc 
> for an average user. To quote:
> 
>        - Customizing auto-mode-alist is not the easiest task, it
>          requires good knowledge of Emacs regexps and alists.  So
>          asking anyone who wants to try using the tree-sitter modes to
>          do that is not the best idea from the POV of user-friendliness.
> 
> So which is it?

Both.  I mention auto-mode-alist as the last alternative, for those
who are fine with going that way.  It isn't black-and-white.

> To "try out" tree-sitter modes, the users can 'M-x js-ts-mode' or etc 
> either way. But to switch to the said mode, they will need to deal with 
> auto-mode-alist, again, without any alternative.

That's specific to js-ts-mode (and one other, I think) because they
share the .el file with a non-tree-sitter mode.  If there's a
reasonable way to give them separate files, things would be easier.

But this is not a documentation issue, first and foremost.

Re: Make all tree-sitter modes optional

[Dmitry Gutov]

On 17/01/2023 20:40, Eli Zaretskii wrote:
>> Date: Tue, 17 Jan 2023 20:21:18 +0200
>> Cc: casouri@gmail.com, monnier@iro.umontreal.ca, larsi@gnus.org,
>>   theo@thornhill.no, jostein@secure.kjonigsen.net, emacs-devel@gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>> +The new modes based on tree-sitter are for now entirely optional, and
>> +you must turn them on manually, or load them in your init file, or
>> +customize 'auto-mode-alist' to turn them on automatically for certain
>> +files.
>>
>> I thought there would be something more in the docs, given that the new
>> behavior is unusual.
> 
> What do you suggest to add?

The design doesn't really make sense to me, so it would be hard for me 
to explain it to a user. So no proposed docs from me, sorry.

>> But I see you are referring to auto-mode-alist here, modifying which
>> will still be necessary for js-ts-mode and python-ts-mode. Which will
>> touch a lot of users, possibly even the majority of tree-sitter
>> enthusiasts, given that JS and Python are some of the most popular
>> languages these days.
>>
>> And yet you rejected my counter-proposal claiming (if I got your
>> position right) that modying auto-mode-alist is difficult/annoying/etc
>> for an average user. To quote:
>>
>>         - Customizing auto-mode-alist is not the easiest task, it
>>           requires good knowledge of Emacs regexps and alists.  So
>>           asking anyone who wants to try using the tree-sitter modes to
>>           do that is not the best idea from the POV of user-friendliness.
>>
>> So which is it?
> 
> Both.  I mention auto-mode-alist as the last alternative, for those
> who are fine with going that way.  It isn't black-and-white.

But it's either that, or "turn them on manually", right? That would also 
work with my proposal.

>> To "try out" tree-sitter modes, the users can 'M-x js-ts-mode' or etc
>> either way. But to switch to the said mode, they will need to deal with
>> auto-mode-alist, again, without any alternative.
> 
> That's specific to js-ts-mode (and one other, I think) because they
> share the .el file with a non-tree-sitter mode.  If there's a
> reasonable way to give them separate files, things would be easier.

Right. If they were in separate files, then we could write the doc with 
suggestions to merely violate the standard recommendation of avoiding 
'require' in the init script.

> But this is not a documentation issue, first and foremost.

Inconsistencies in design often turn up as documentation issues later.

Re: Make all tree-sitter modes optional

[Eli Zaretskii]

> Date: Tue, 17 Jan 2023 20:49:59 +0200
> Cc: casouri@gmail.com, monnier@iro.umontreal.ca, larsi@gnus.org,
>  theo@thornhill.no, jostein@secure.kjonigsen.net, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> >> But I see you are referring to auto-mode-alist here, modifying which
> >> will still be necessary for js-ts-mode and python-ts-mode. Which will
> >> touch a lot of users, possibly even the majority of tree-sitter
> >> enthusiasts, given that JS and Python are some of the most popular
> >> languages these days.
> >>
> >> And yet you rejected my counter-proposal claiming (if I got your
> >> position right) that modying auto-mode-alist is difficult/annoying/etc
> >> for an average user. To quote:
> >>
> >>         - Customizing auto-mode-alist is not the easiest task, it
> >>           requires good knowledge of Emacs regexps and alists.  So
> >>           asking anyone who wants to try using the tree-sitter modes to
> >>           do that is not the best idea from the POV of user-friendliness.
> >>
> >> So which is it?
> > 
> > Both.  I mention auto-mode-alist as the last alternative, for those
> > who are fine with going that way.  It isn't black-and-white.
> 
> But it's either that, or "turn them on manually", right? That would also 
> work with my proposal.

Your proposal makes it harder by making it necessary to mess with
auto-mode-alist.  I hope we can make it easier, at least in many/most
cases.  That is a non-negligible advantage from my POV.

> > That's specific to js-ts-mode (and one other, I think) because they
> > share the .el file with a non-tree-sitter mode.  If there's a
> > reasonable way to give them separate files, things would be easier.
> 
> Right. If they were in separate files, then we could write the doc with 
> suggestions to merely violate the standard recommendation of avoiding 
> 'require' in the init script.
> 
> > But this is not a documentation issue, first and foremost.
> 
> Inconsistencies in design often turn up as documentation issues later.

I didn't design js.el that way.  I just assumed there were good
reasons for that and tried to adapt as best I could.

Re: Make all tree-sitter modes optional

[Dmitry Gutov]

On 17/01/2023 21:03, Eli Zaretskii wrote:

>>> Both.  I mention auto-mode-alist as the last alternative, for those
>>> who are fine with going that way.  It isn't black-and-white.
>>
>> But it's either that, or "turn them on manually", right? That would also
>> work with my proposal.
> 
> Your proposal makes it harder by making it necessary to mess with
> auto-mode-alist.  I hope we can make it easier, at least in many/most
> cases.  That is a non-negligible advantage from my POV.

The userbase I'm familiar with has been messing with auto-mode-alist for 
decades. To make it easier, people have been adding the necessary lines 
to documentation: either to the repository's README, or to package's 
Commentary. So the user could take it from there.

Similarly, we can add such line to the Commentary of every ts mode, or 
even to the major mode docstrings.

We also have merged use-package to Emacs. It makes the same operation a 
little easier, though not by much:

(use-package js
   :mode ("\\.js\\'" . js-ts-mode))

Either way, there are a lot of examples of this kind of configuration 
out there.

>>> That's specific to js-ts-mode (and one other, I think) because they
>>> share the .el file with a non-tree-sitter mode.  If there's a
>>> reasonable way to give them separate files, things would be easier.
>>
>> Right. If they were in separate files, then we could write the doc with
>> suggestions to merely violate the standard recommendation of avoiding
>> 'require' in the init script.
>>
>>> But this is not a documentation issue, first and foremost.
>>
>> Inconsistencies in design often turn up as documentation issues later.
> 
> I didn't design js.el that way.  I just assumed there were good
> reasons for that and tried to adapt as best I could.

I think the design is fine, but whatever solution we choose should be 
compatible with it.

Re: Make all tree-sitter modes optional

[Yuan Fu]

> 
> The userbase I'm familiar with has been messing with auto-mode-alist for decades. To make it easier, people have been adding the necessary lines to documentation: either to the repository's README, or to package's Commentary. So the user could take it from there.
> 
> Similarly, we can add such line to the Commentary of every ts mode, or even to the major mode docstrings

IF we do that, it’d be better to instruct users to use major-mode-remap-alist, since it’s easier to understand and use IMO.

Yuan

Re: Make all tree-sitter modes optional

[Dmitry Gutov]

On 18/01/2023 03:11, Yuan Fu wrote:
>> The userbase I'm familiar with has been messing with auto-mode-alist for decades. To make it easier, people have been adding the necessary lines to documentation: either to the repository's README, or to package's Commentary. So the user could take it from there.
>>
>> Similarly, we can add such line to the Commentary of every ts mode, or even to the major mode docstrings
> IF we do that, it’d be better to instruct users to use major-mode-remap-alist, since it’s easier to understand and use IMO.

Sure. This way we also avoid duplicating the file name regexp.

Re: Make all tree-sitter modes optional

[Eli Zaretskii]

> Date: Wed, 18 Jan 2023 03:23:57 +0200
> Cc: Eli Zaretskii <eliz@gnu.org>, monnier@iro.umontreal.ca, larsi@gnus.org,
>  theo@thornhill.no, jostein@secure.kjonigsen.net, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 18/01/2023 03:11, Yuan Fu wrote:
> >> The userbase I'm familiar with has been messing with auto-mode-alist for decades. To make it easier, people have been adding the necessary lines to documentation: either to the repository's README, or to package's Commentary. So the user could take it from there.
> >>
> >> Similarly, we can add such line to the Commentary of every ts mode, or even to the major mode docstrings
> > IF we do that, it’d be better to instruct users to use major-mode-remap-alist, since it’s easier to understand and use IMO.
> 
> Sure. This way we also avoid duplicating the file name regexp.

The purpose of major-mode-remap-alist doesn't fit such use, so this
proposal is a no-go.

Again, we've been through this already, at least twice.

Re: Make all tree-sitter modes optional

[Dmitry Gutov]

On 18/01/2023 05:34, Eli Zaretskii wrote:
>>>> Similarly, we can add such line to the Commentary of every ts mode, or even to the major mode docstrings
>>> IF we do that, it’d be better to instruct users to use major-mode-remap-alist, since it’s easier to understand and use IMO.
>> Sure. This way we also avoid duplicating the file name regexp.
> The purpose of major-mode-remap-alist doesn't fit such use, so this
> proposal is a no-go.

Its NEWS entry says:

** New user option 'major-mode-remap-alist' to specify favorite major modes.
This user option lets you remap the default modes (e.g. 'perl-mode' or
'latex-mode') to your favorite ones (e.g. 'cperl-mode' or
'LaTeX-mode') without having to use 'defalias', which can have
undesirable side effects.

If we leave it to the user to add an entry to that variable, we don't 
have to worry whether tree-sitter support is compiled and the respective 
grammar is available -- the user can ensure that. So there will be no 
need for a treesit-available-p check.

Seems like a perfect fit.

Re: Make all tree-sitter modes optional

[Eli Zaretskii]

> Date: Wed, 18 Jan 2023 05:52:14 +0200
> Cc: casouri@gmail.com, monnier@iro.umontreal.ca, larsi@gnus.org,
>  theo@thornhill.no, jostein@secure.kjonigsen.net, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 18/01/2023 05:34, Eli Zaretskii wrote:
> >>>> Similarly, we can add such line to the Commentary of every ts mode, or even to the major mode docstrings
> >>> IF we do that, it’d be better to instruct users to use major-mode-remap-alist, since it’s easier to understand and use IMO.
> >> Sure. This way we also avoid duplicating the file name regexp.
> > The purpose of major-mode-remap-alist doesn't fit such use, so this
> > proposal is a no-go.
> 
> Its NEWS entry says:
> 
> ** New user option 'major-mode-remap-alist' to specify favorite major modes.
> This user option lets you remap the default modes (e.g. 'perl-mode' or
> 'latex-mode') to your favorite ones (e.g. 'cperl-mode' or
> 'LaTeX-mode') without having to use 'defalias', which can have
> undesirable side effects.
> 
> If we leave it to the user to add an entry to that variable, we don't 
> have to worry whether tree-sitter support is compiled and the respective 
> grammar is available -- the user can ensure that. So there will be no 
> need for a treesit-available-p check.

We can _suggest_ using that variable for this particular purpose, but
we cannot make it the only or even the main means of dealing with the
issue: this is a _user_ variable, and users can have needs or reasons
to do stuff with it that are incompatible with the kind of usage
proposed here.

Re: Make all tree-sitter modes optional

[Dmitry Gutov]

On 18/01/2023 14:06, Eli Zaretskii wrote:
> We can_suggest_  using that variable for this particular purpose, but
> we cannot make it the only or even the main means of dealing with the
> issue: this is a_user_  variable, and users can have needs or reasons
> to do stuff with it that are incompatible with the kind of usage
> proposed here.

Since the recommendation will be in the docs, the user can tweak the 
form as convenient, won't they?

I'm not sure which particular stuff you are thinking of, though.

Re: Make all tree-sitter modes optional

[Eli Zaretskii]

> Date: Wed, 18 Jan 2023 16:00:47 +0200
> Cc: casouri@gmail.com, monnier@iro.umontreal.ca, larsi@gnus.org,
>  theo@thornhill.no, jostein@secure.kjonigsen.net, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 18/01/2023 14:06, Eli Zaretskii wrote:
> > We can_suggest_  using that variable for this particular purpose, but
> > we cannot make it the only or even the main means of dealing with the
> > issue: this is a_user_  variable, and users can have needs or reasons
> > to do stuff with it that are incompatible with the kind of usage
> > proposed here.
> 
> Since the recommendation will be in the docs, the user can tweak the 
> form as convenient, won't they?
> 
> I'm not sure which particular stuff you are thinking of, though.

I will hopefully install this in a day or two, and will mention that
possibility in NEWS when I do.  Let's take it from there.

Re: Make all tree-sitter modes optional

[Stefan Monnier]

>> > IF we do that, it’d be better to instruct users to use
>> > major-mode-remap-alist, since it’s easier to understand and use IMO.
>> 
>> Sure. This way we also avoid duplicating the file name regexp.
>
> The purpose of major-mode-remap-alist doesn't fit such use, so this
> proposal is a no-go.

Maybe I missed something (I'm having trouble keeping up with
emacs-devel these days, sorry), but I don't see why you think so.
It seems perfectly appropriate (and indeed fits the original goal) for
`Commentary:` to say something like:

    ;; If you want to use this mode by default, you can add the following
    to your init file:
    ;;
    ;;     (add-to-list 'major-mode-remap-alist '(ruby-mode . ruby-ts-mode))


-- Stefan

treesit-forward-sexp (was: Make all tree-sitter modes optional)

[Juri Linkov]

> I'm using ruby-mode, at least for now, while all the wrinkles with
> indentation haven't been ironed out (and we'll probably not manage to get
> them all 100% right before the 29 release), and while ts modes don't
> support show-paren-mode like SMIE does. No proper sexp navigation, etc.

BTW, how ruby-ts-mode is intended to be used without proper sexp navigation?

I see that forward-sentence support for tree sitter was added recently
with treesit-forward-sentence.  There are also treesit-transpose-sexps,
treesit-beginning-of-defun and treesit-end-of-defun.

Are there any plans to add treesit-forward-sexp as well?

Currently I'm using such workaround:

```
(with-eval-after-load 'ruby-ts-mode
  (add-hook 'ruby-ts-mode-hook
            (lambda ()
              (smie-setup ruby-smie-grammar #'ruby-smie-rules
                          :forward-token  #'ruby-smie--forward-token
                          :backward-token #'ruby-smie--backward-token))))
```

Maybe something like this should be added to emacs-29
until treesit-forward-sexp is implemented in master?

Re: treesit-forward-sexp (was: Make all tree-sitter modes optional)

[Theodor Thornhill]

On 17 January 2023 18:34:04 CET, Juri Linkov <juri@linkov.net> wrote:
>> I'm using ruby-mode, at least for now, while all the wrinkles with
>> indentation haven't been ironed out (and we'll probably not manage to get
>> them all 100% right before the 29 release), and while ts modes don't
>> support show-paren-mode like SMIE does. No proper sexp navigation, etc.
>
>BTW, how ruby-ts-mode is intended to be used without proper sexp navigation?
>
>I see that forward-sentence support for tree sitter was added recently
>with treesit-forward-sentence.  There are also treesit-transpose-sexps,
>treesit-beginning-of-defun and treesit-end-of-defun.
>
>Are there any plans to add treesit-forward-sexp as well?

Yes, working on it now :)

>
>Currently I'm using such workaround:
>
>```
>(with-eval-after-load 'ruby-ts-mode
>  (add-hook 'ruby-ts-mode-hook
>            (lambda ()
>              (smie-setup ruby-smie-grammar #'ruby-smie-rules
>                          :forward-token  #'ruby-smie--forward-token
>                          :backward-token #'ruby-smie--backward-token))))
>```
>
>Maybe something like this should be added to emacs-29
>until treesit-forward-sexp is implemented in master?

Theo

Re: treesit-forward-sexp

[Juri Linkov]

>> I see that forward-sentence support for tree sitter was added recently
>> with treesit-forward-sentence.  There are also treesit-transpose-sexps,
>> treesit-beginning-of-defun and treesit-end-of-defun.
>>
>> Are there any plans to add treesit-forward-sexp as well?
>
> Yes, working on it now :)

Good news!  Thanks.

Re: treesit-forward-sexp (was: Make all tree-sitter modes optional)

[Dmitry Gutov]

On 17/01/2023 19:34, Juri Linkov wrote:
>> I'm using ruby-mode, at least for now, while all the wrinkles with
>> indentation haven't been ironed out (and we'll probably not manage to get
>> them all 100% right before the 29 release), and while ts modes don't
>> support show-paren-mode like SMIE does. No proper sexp navigation, etc.
> 
> BTW, how ruby-ts-mode is intended to be used without proper sexp navigation?
> 
> I see that forward-sentence support for tree sitter was added recently
> with treesit-forward-sentence.  There are also treesit-transpose-sexps,
> treesit-beginning-of-defun and treesit-end-of-defun.
> 
> Are there any plans to add treesit-forward-sexp as well?
 >
> Currently I'm using such workaround:
> 
> ```
> (with-eval-after-load 'ruby-ts-mode
>    (add-hook 'ruby-ts-mode-hook
>              (lambda ()
>                (smie-setup ruby-smie-grammar #'ruby-smie-rules
>                            :forward-token  #'ruby-smie--forward-token
>                            :backward-token #'ruby-smie--backward-token))))
> ```

I have also been toying with the idea of using SMIE for indentation with 
ruby-ts-mode, keeping tree-sitter for font-lock and other facilities 
which are easier to support.

But for now I'm working on improving tree-sitter based indentation. 
After all, the underlying data it's using is by definition more correct.

> Maybe something like this should be added to emacs-29
> until treesit-forward-sexp is implemented in master?

I don't know if everybody will agree with that. ruby-ts-mode's author in 
particular.

So it's probably better suited for personal customizations, like yours 
above. BTW note that smie-setup also sets indent-line-function.

Re: treesit-forward-sexp (was: Make all tree-sitter modes optional)

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: Eli Zaretskii <eliz@gnu.org>,  casouri@gmail.com,
>   monnier@iro.umontreal.ca,  larsi@gnus.org,  theo@thornhill.no,
>   jostein@secure.kjonigsen.net,  emacs-devel@gnu.org
> Date: Tue, 17 Jan 2023 19:34:04 +0200
> 
> ```
> (with-eval-after-load 'ruby-ts-mode
>   (add-hook 'ruby-ts-mode-hook
>             (lambda ()
>               (smie-setup ruby-smie-grammar #'ruby-smie-rules
>                           :forward-token  #'ruby-smie--forward-token
>                           :backward-token #'ruby-smie--backward-token))))
> ```
> 
> Maybe something like this should be added to emacs-29
> until treesit-forward-sexp is implemented in master?

Sorry, we are done adding new features to emacs-29.

Re: Make all tree-sitter modes optional

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

>> So here's a suggestion for such a solution: we make all the
>> *-ts-mode's optional.  That is, we don't add any of them to
>> auto-mode-alist unless the file *-ts-mode.el is loaded, and we
>> document them all in NEWS and the user manual as optional.  users who
>> want them will have to manually activate them.  This way, the original
>> use case that started this bug report is automatically solved, and the
>> other use case, where the user intends to activate one of these modes,
>> is also served by showing the warning, which in that case is perfectly
>> justified: the user asked for something that we cannot do, so we warn
>> him/her.
>> 
>> This is a retreat of sorts, but I think it strikes a better balance
>> wrt user expectations, assuming not everyone will build with
>> tree-sitter.
>
> The proposed change to the current emacs-29 branch is below.  You will
> see that where possible, just loading a TS mode modifies
> auto-mode-alist if the tree-sitter support for that mode is available,
> whereas for other modes auto-mode-alist is modified only when the mode
> is actually turned on successfully for the first time.  This is
> because some of the TS modes have their own *.el files, whereas others
> share a .el file with other modes, and so loading that file doesn't
> necessarily means the user wants to use the tree-sitter based mode.
>
> With these changes, users are supposed to turn the mode manually when
> they want to use it, or customize auto-mode-alist in their init files
> to turn those modes automatically for certain files.  Modes that have
> their own *.el files can be just 'require'd in the init file to modify
> auto-mode-alist.
>
> This arrangement is perhaps not ideal, but I think it is safe enough
> and simple enough to go into Emacs 29.
>
> Comments? objections? ideas for improvements?  I will install in a few
> days, barring any serious problems.

Thank you.  I think this is a very reasonable arragement, certainly
better than what we had before.

Re: Make all tree-sitter modes optional

[Juri Linkov]

>> This came out of the discussion in bug#60559 and other related discussions
>
> I addressed that in the discussion on emacs-devel, see
>   https://lists.gnu.org/archive/html/emacs-devel/2023-01/msg00278.html
> (I suggest to discuss this there, not here.)

Ok, moving to emacs-devel from bug#60176.

>> >> >> >> (defcustom treesit-enable-modes nil
>> >> >> >>   :type '(repeat
>> >> >> >>           (choice (function-item c-ts-mode)
>> >> >> >>                   (function-item c++-ts-mode)
>> >> >> >>                   (function-item c-or-c++-ts-mode)
>> >> >> >>                   ...
>> >> >> >>              ('c-ts-mode
>> >> >> >>               (when (treesit-ready-p 'c t)
>> >> >> >>                 (add-to-list 'major-mode-remap-alist '(c-mode . c-ts-mode))))
>> >> >> >
>> >> >> > And this bit is completely unacceptable, from where I stand: it
>> >> >> > basically means that the user activated a certain major mode he/she
>> >> >> > wanted to use, but the result could be that an entirely different mode
>> >> >> > was silently activated instead.  What kind of UX is that, and for a
>> >> >> > shining new feature at that??
>> >> >>
>> >> >> It could update 'auto-mode-alist' instead of 'major-mode-remap-alist'.
>> >> >> From the user's point of view this would be more manageable than
>> >> >> what you proposed on emacs-devel with some obscure logic of activating
>> >> >> ts modes when the package is loaded or when the mode is enabled first time.
>> >> >
>> >> > I don't understand the "obscure" part: the logic was exactly as above:
>> >> > test that treesit-ready-p returns non-nil for the mode's language.
>> >> >
>> >> > Other than that, my proposal does exactly what you say here: it
>> >> > updates auto-mode-alist.  So it sounds like we are in violent agreement.
>> >>
>> >> The difference is that an explicit option is more controllable by the user.
>> >> When the user needs to use some ts-mode then it's easier just to customize
>> >> the option instead of tweaking 'auto-mode-alist' when the user want to
>> >> start using that mode without first loading its package or calling it
>> >> the first time that modifies 'auto-mode-alist' as the side effect.
>> >
>> > With the changes I proposed, there's no need to tweak
>> > auto-mode-alist.  A simple load or require of the mode will install
>> > the mode in auto-mode-alist.  What can be easier and simpler?
>>
>> This doesn't address the problems mentioned above and below.
>
> Which parts "above" were not addressed?
>
> As for below:
>
>> >> Or when the user wants to remove the mode from 'auto-mode-alist' after
>> >> accidentally loading the corresponding ts package.

It fails for many scenarios:

1. The user needs to use e.g. js-ts-mode.  Your patch requires that
users first visit a js file in js-mode, then they need to type

  M-x js-ts-mode RET

then for the rest of the session they can use js-ts-mode.
But for a new session they need to repeat the same again.

2. For other ts modes adding something like (require 'c-ts-mode)
to the user's init file is not a clean way to enable such modes.
Also takes additional space even when users don't intend to use them
in the current session, thus negating the benefits of autoloading.

3. There is no simple way to disable a ts mode after loading
the corresponding ts file.

Re: Make all tree-sitter modes optional

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: casouri@gmail.com,  monnier@iro.umontreal.ca,  larsi@gnus.org,
>   theo@thornhill.no,  jostein@secure.kjonigsen.net,  emacs-devel@gnu.org
> Date: Tue, 17 Jan 2023 19:30:16 +0200
> 
> It fails for many scenarios:
> 
> 1. The user needs to use e.g. js-ts-mode.  Your patch requires that
> users first visit a js file in js-mode, then they need to type
> 
>   M-x js-ts-mode RET
> 
> then for the rest of the session they can use js-ts-mode.
> But for a new session they need to repeat the same again.

Or customize auto-mode-alist.

This is a consequence of the fact that js-ts-mode doesn't have a
separate .el file.  If you have better ideas for that, I'm all ears.

> 2. For other ts modes adding something like (require 'c-ts-mode)
> to the user's init file is not a clean way to enable such modes.

Why not?

> Also takes additional space even when users don't intend to use them
> in the current session, thus negating the benefits of autoloading.

Once again, if the above is somehow too annoying, they can customize
auto-mode-alist manually.  Or even use use-package (and have full
benefit of autoloading).  In any case, the fact that a mode is loaded
and not used is not a problem large enough to reject this simple
arrangement.  Moreover, if someone puts the require into their init
file, they probably want to use the mode quite a lot, so it will not
remain ballast for too long.

> 3. There is no simple way to disable a ts mode after loading
> the corresponding ts file.

Yes, there is: restart the session.  IMO, adequate enough for
something you want to try, and then find not to like.

Bottom line: if you have better, simpler solutions, please tell.  The
ones you proposed until now are significantly more complex, and it is
not reasonable to ask users to go to such lengths to try a new mode.
But of course, if someone wants to do it in a more complex and more
flexible way, they still can.

Re: Make all tree-sitter modes optional

[Juri Linkov]

>> 1. The user needs to use e.g. js-ts-mode.  Your patch requires that
>> users first visit a js file in js-mode, then they need to type
>>
>>   M-x js-ts-mode RET
>>
>> then for the rest of the session they can use js-ts-mode.
>> But for a new session they need to repeat the same again.
>
> Or customize auto-mode-alist.
>
> This is a consequence of the fact that js-ts-mode doesn't have a
> separate .el file.  If you have better ideas for that, I'm all ears.

A better idea is a simple customizable list of enabled ts modes.

Re: Make all tree-sitter modes optional

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: casouri@gmail.com,  monnier@iro.umontreal.ca,  larsi@gnus.org,
>   theo@thornhill.no,  jostein@secure.kjonigsen.net,  emacs-devel@gnu.org
> Date: Tue, 17 Jan 2023 20:19:44 +0200
> 
> >> 1. The user needs to use e.g. js-ts-mode.  Your patch requires that
> >> users first visit a js file in js-mode, then they need to type
> >>
> >>   M-x js-ts-mode RET
> >>
> >> then for the rest of the session they can use js-ts-mode.
> >> But for a new session they need to repeat the same again.
> >
> > Or customize auto-mode-alist.
> >
> > This is a consequence of the fact that js-ts-mode doesn't have a
> > separate .el file.  If you have better ideas for that, I'm all ears.
> 
> A better idea is a simple customizable list of enabled ts modes.

Yes, you suggested that already.  But I disagree that it's simple, and
therefore don't agree that it's better.

Re: Make all tree-sitter modes optional

[Alan Mackenzie]

Hello, Eli.

I missed this thread back in January, due to being away from Emacs
development for a time, and I think the current state of this feature is
so far away from ideal as to make some design and implementation
advisable.

On Tue, Jan 17, 2023 at 19:58:40 +0200, Eli Zaretskii wrote:
> > From: Juri Linkov <juri@linkov.net>
> > Cc: casouri@gmail.com,  monnier@iro.umontreal.ca,  larsi@gnus.org,
> >   theo@thornhill.no,  jostein@secure.kjonigsen.net,  emacs-devel@gnu.org
> > Date: Tue, 17 Jan 2023 19:30:16 +0200

> > It fails for many scenarios:

> > 1. The user needs to use e.g. js-ts-mode.  Your patch requires that
> > users first visit a js file in js-mode, then they need to type

> >   M-x js-ts-mode RET

> > then for the rest of the session they can use js-ts-mode.
> > But for a new session they need to repeat the same again.

> Or customize auto-mode-alist.

> This is a consequence of the fact that js-ts-mode doesn't have a
> separate .el file.  If you have better ideas for that, I'm all ears.

> > 2. For other ts modes adding something like (require 'c-ts-mode)
> > to the user's init file is not a clean way to enable such modes.

> Why not?

> > Also takes additional space even when users don't intend to use them
> > in the current session, thus negating the benefits of autoloading.

> Once again, if the above is somehow too annoying, they can customize
> auto-mode-alist manually.  Or even use use-package (and have full
> benefit of autoloading).  In any case, the fact that a mode is loaded
> and not used is not a problem large enough to reject this simple
> arrangement.  Moreover, if someone puts the require into their init
> file, they probably want to use the mode quite a lot, so it will not
> remain ballast for too long.

> > 3. There is no simple way to disable a ts mode after loading
> > the corresponding ts file.

> Yes, there is: restart the session.  IMO, adequate enough for
> something you want to try, and then find not to like.

No, it is not adequate.  It is horrible.  It forces the user to discard
all the state in his Emacs session, and restart, which is s..l..o..w..,
particularly if said user has a large desktop file.  In my experiments
with c-ts-mode, I switched backwards and forwards several times between C
Mode and c-ts-mode.  This now corrupts my Emacs session, in that C Mode
is no longer the default for C files.  I got caught out this evening when
I opened a C file, and something failed to work on it because it opened
in c-ts-mode.

This seems to be undocumented in NEWS.

I think lots of users are going to hate having the new -ts- modes forced
upon them in this manner.  Normally, when a new feature is introduced
into an Emacs version, it is fully optional, and defaults to disabled.
Why do the -ts- modes not conform to this well established practice?

There appears to be no way, apart from editing the Lisp source files, to
make C Mode the default for .c files when there are also c-ts-mode
buffers in the session.  This is a Bad Thing.

> Bottom line: if you have better, simpler solutions, please tell.

How about commands c-make-ts-default-mode and c-make-ts-undefault-mode
(and analogous functions for the other ts-modes)?  The first of these
would push the new entries onto auto-mode-alist and the second would
remove them again.  Or we could have a customisable option,
c-default-c-mode which users could set when they are ready.  Either of
these would allow the user to try out the new modes freely without being
coerced against their will to use the new -ts- modes.

> The ones you proposed until now are significantly more complex, and it
> is not reasonable to ask users to go to such lengths to try a new mode.

Any reasonable solution is going to be more complicated than the current
code.  I don't think it reasonable silently to force users into setting the
new modes as defaults.  They should be able to try out a new mode simply
by doing M-x c-ts-mode, or the like, then go back to their normal way of
working.

> But of course, if someone wants to do it in a more complex and more
> flexible way, they still can.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Make all tree-sitter modes optional

[Eli Zaretskii]

> Date: Tue, 14 Feb 2023 19:08:42 +0000
> Cc: Juri Linkov <juri@linkov.net>, casouri@gmail.com,
>   monnier@iro.umontreal.ca, larsi@gnus.org, theo@thornhill.no,
>   jostein@secure.kjonigsen.net, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> Hello, Eli.
> 
> I missed this thread back in January, due to being away from Emacs
> development for a time, and I think the current state of this feature is
> so far away from ideal as to make some design and implementation
> advisable.

It's too bad you are raising this up only now, because it's too late
for any significant changes, even if I were to agree with what you
propose.

Some things you miss cannot be brought back, sorry.  You have to be
around and participate in the relevant discussions, let alone be part
of implementing the features, to have your opinions taken into
considerations when it matters.

> No, it is not adequate.  It is horrible.

Not a very kind remark, to say the least.

> How about commands c-make-ts-default-mode and c-make-ts-undefault-mode

I'm okay with adding the latter, if it turns out easy enough and safe
enough (of which I'm not sure at all), and if such a command will be
implemented for all the *-ts-modes which have non-ts siblings, but I
see no reason for the former, since there are several simple ways to
cause the same effect, and they are all documented in NEWS.

Please note, however, that a pretest is hopefully very close, and so
there isn't much time for adding such commands.  I would say that for
that reason your proposal is not practical.

> The first of these would push the new entries onto auto-mode-alist
> and the second would remove them again.

I think you have a very simplistic idea of what loading a *-ts-mode
does, but if you can come up with a simple and safe implementation, I
won't object adding such a command -- it cannot do any harm by just
being there, and if it turns to be not what users want, we can always
advise them not to use it.

> Or we could have a customisable option, c-default-c-mode which users
> could set when they are ready.

That was considered, and was found to be either too complicated in
practice or, if implemented simply, not to work well enough.  So let's
not go there.

> Either of these would allow the user to try out the new modes freely
> without being coerced against their will to use the new -ts- modes.

"Try out" is not what I had in mind for users who'd like to use these
modes.

> > The ones you proposed until now are significantly more complex, and it
> > is not reasonable to ask users to go to such lengths to try a new mode.
> 
> Any reasonable solution is going to be more complicated than the current
> code.

Then they aren't "reasonable" at this time.  Maybe later, maybe on
master.  As I said several times, we have no good idea yet how users
will react to what we have.  Maybe, after we hear from them, we decide
to implement such switches, who knows.

> I don't think it reasonable silently to force users into setting the
> new modes as defaults.

That is so far from what we have that I suspect you haven't read the
code closely enough.  NEWS explains the situation.

Re: Make all tree-sitter modes optional

[Alan Mackenzie]

Hello, Eli.

On Tue, Feb 14, 2023 at 21:29:01 +0200, Eli Zaretskii wrote:
> > Date: Tue, 14 Feb 2023 19:08:42 +0000
> > Cc: Juri Linkov <juri@linkov.net>, casouri@gmail.com,
> >   monnier@iro.umontreal.ca, larsi@gnus.org, theo@thornhill.no,
> >   jostein@secure.kjonigsen.net, emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

[ .... ]

> > No, it is not adequate.  It is horrible.

> Not a very kind remark, to say the least.

Sorry, I should have written "It is horrible for me." - which is true.

> > How about commands c-make-ts-default-mode and c-make-ts-undefault-mode

> I'm okay with adding the latter, if it turns out easy enough and safe
> enough (of which I'm not sure at all), and if such a command will be
> implemented for all the *-ts-modes which have non-ts siblings, but I
> see no reason for the former, since there are several simple ways to
> cause the same effect, and they are all documented in NEWS.

OK, Try this (so far only on c-ts-mode.):


diff --git a/etc/NEWS b/etc/NEWS
index 2d15e39036a..0a745d7cde9 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -3239,10 +3239,13 @@ for which a "built-in" mode would be turned on.  For example:
 
     (add-to-list 'major-mode-remap-alist '(ruby-mode . ruby-ts-mode))
 
-If you try these modes and don't like them, you can go back to the
-"built-in" modes by restarting Emacs.  But please tell us why you
-didn't like the tree-sitter based modes, so that we could try
-improving them.
+Normally, the loading of one of the new modes amends 'auto-mode-alist'
+such that future visiting of the same type of file will continue to
+use that new mode.  If this is not what you want, do M-x
+<mode>-make-ts-undefault-mode.  For a more permanent effect, put, for
+example, the following into your initialization file:
+
+    (eval-after-load 'c-ts-mode '(c-make-ts-undefault-mode))
 
 Each major mode based on tree-sitter needs a language grammar library,
 usually named "libtree-sitter-LANG.so" ("libtree-sitter-LANG.dll" on
diff --git a/lisp/progmodes/c-ts-mode.el b/lisp/progmodes/c-ts-mode.el
index 5093c3980b6..d6ea95a2980 100644
--- a/lisp/progmodes/c-ts-mode.el
+++ b/lisp/progmodes/c-ts-mode.el
@@ -904,6 +904,20 @@ c-or-c++-ts-mode
          (treesit-ready-p 'c))
     (add-to-list 'auto-mode-alist '("\\.h\\'" . c-or-c++-ts-mode)))
 
+(defun c-make-ts-undefault-mode ()
+  "Make the older C and C++ Modes the default major modes for C(++) files."
+  (interactive)
+  (setq auto-mode-alist (delete '("\\.h\\'" . c-or-c++-ts-mode)
+                                auto-mode-alist))
+  (setq auto-mode-alist
+        (delete '("\\(\\.[chi]\\|\\.lex\\|\\.y\\(acc\\)?\\|\\.x[bp]m\\)\\'" . c-ts-mode)
+		auto-mode-alist))
+  (setq auto-mode-alist
+        (delete
+	 '("\\(\\.ii\\|\\.\\(CC?\\|HH?\\)\\|\\.[ch]\\(pp\\|xx\\|\\+\\+\\)\\|\\.\\(cc\\|hh\\)\\)\\'"
+	   . c++-ts-mode)
+	 auto-mode-alist)))
+
 (provide 'c-ts-mode)
 
 ;;; c-ts-mode.el ends here

[ .... ]

> I think you have a very simplistic idea of what loading a *-ts-mode
> does, but if you can come up with a simple and safe implementation, I
> won't object adding such a command -- it cannot do any harm by just
> being there, and if it turns to be not what users want, we can always
> advise them not to use it.

[ .... ]

> > Either of these would allow the user to try out the new modes freely
> > without being coerced against their will to use the new -ts- modes.

> "Try out" is not what I had in mind for users who'd like to use these
> modes.

Some will want to try them out first, before definitively committing to
them.  I am such a user.

[ .... ]

> Then they [proposed amendments] aren't "reasonable" at this time.
> Maybe later, maybe on master.

That will be too late, the damage will have been done.  It is the first
experience people have of the new modes which will create their long term
impressions of them.  I remember something similar happening in Emacs
21.1, when the new fringes were not made optional.  Lots of users
complained loudly and bitterly.

> As I said several times, we have no good idea yet how users will react
> to what we have.  Maybe, after we hear from them, we decide to
> implement such switches, who knows.

We are ourselves all users, too.  We know how we have reacted, and it is
reasonable to try to prevent bad experiences for users similar to
ourselves.

[ .... ]

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Make all tree-sitter modes optional

[Eli Zaretskii]

> Date: Tue, 14 Feb 2023 21:02:25 +0000
> Cc: juri@linkov.net, casouri@gmail.com, monnier@iro.umontreal.ca,
>   larsi@gnus.org, theo@thornhill.no, jostein@secure.kjonigsen.net,
>   emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > I'm okay with adding the latter, if it turns out easy enough and safe
> > enough (of which I'm not sure at all), and if such a command will be
> > implemented for all the *-ts-modes which have non-ts siblings, but I
> > see no reason for the former, since there are several simple ways to
> > cause the same effect, and they are all documented in NEWS.
> 
> OK, Try this (so far only on c-ts-mode.):
> 
> 
> diff --git a/etc/NEWS b/etc/NEWS
> index 2d15e39036a..0a745d7cde9 100644
> --- a/etc/NEWS
> +++ b/etc/NEWS
> @@ -3239,10 +3239,13 @@ for which a "built-in" mode would be turned on.  For example:
>  
>      (add-to-list 'major-mode-remap-alist '(ruby-mode . ruby-ts-mode))
>  
> -If you try these modes and don't like them, you can go back to the
> -"built-in" modes by restarting Emacs.  But please tell us why you
> -didn't like the tree-sitter based modes, so that we could try
> -improving them.
> +Normally, the loading of one of the new modes amends 'auto-mode-alist'
> +such that future visiting of the same type of file will continue to
> +use that new mode.  If this is not what you want, do M-x
> +<mode>-make-ts-undefault-mode.  For a more permanent effect, put, for
> +example, the following into your initialization file:
> +
> +    (eval-after-load 'c-ts-mode '(c-make-ts-undefault-mode))

Please don't delete the text in NEWS, it is a result of many
discussions and a lot of thought.  Your proposal is yet another way of
going back to the non-ts modes, so simply _adding_ that to what's
already in NEWS should be much better.

> +(defun c-make-ts-undefault-mode ()
> +  "Make the older C and C++ Modes the default major modes for C(++) files."
> +  (interactive)
> +  (setq auto-mode-alist (delete '("\\.h\\'" . c-or-c++-ts-mode)
> +                                auto-mode-alist))
> +  (setq auto-mode-alist
> +        (delete '("\\(\\.[chi]\\|\\.lex\\|\\.y\\(acc\\)?\\|\\.x[bp]m\\)\\'" . c-ts-mode)
> +		auto-mode-alist))
> +  (setq auto-mode-alist
> +        (delete
> +	 '("\\(\\.ii\\|\\.\\(CC?\\|HH?\\)\\|\\.[ch]\\(pp\\|xx\\|\\+\\+\\)\\|\\.\\(cc\\|hh\\)\\)\\'"
> +	   . c++-ts-mode)
> +	 auto-mode-alist)))
> +

So you revert auto-mode-alist to its original shape, but leave the
buffers already under c-ts-mode in that mode?  Is that what the users
would expect, you think?

Also, this won't work if the user customized auto-mode-alist in some
way wrt some of those file-name extensions.

> > Then they [proposed amendments] aren't "reasonable" at this time.
> > Maybe later, maybe on master.
> 
> That will be too late, the damage will have been done.

What "damage"? why do you call "damage" changes made by others in
Emacs as part of Emacs development?

> It is the first experience people have of the new modes which will
> create their long term impressions of them.

Please leave that to people.  We are introducing new technology to
Emacs, and try doing that in the least intrusive way.  If you don't
want to help that effort (a stance that is frankly very
disappointing), at least don't bad-mouth it.

> I remember something similar happening in Emacs 21.1, when the new
> fringes were not made optional.  Lots of users complained loudly and
> bitterly.

Well, that's exactly why these new modes are entirely optional.

> > As I said several times, we have no good idea yet how users will react
> > to what we have.  Maybe, after we hear from them, we decide to
> > implement such switches, who knows.
> 
> We are ourselves all users, too.  We know how we have reacted, and it is
> reasonable to try to prevent bad experiences for users similar to
> ourselves.

For you and me as users, having to restart Emacs, or having to use a
separate session for such experiments, is an entirely reasonable and
simple alternative, one which should eliminate any need for undoing
the "damage" of c-ts-mode.

Re: Make all tree-sitter modes optional

[Alan Mackenzie]

Hello, Eli.

On Wed, Feb 15, 2023 at 17:35:16 +0200, Eli Zaretskii wrote:
> > Date: Tue, 14 Feb 2023 21:02:25 +0000
> > Cc: juri@linkov.net, casouri@gmail.com, monnier@iro.umontreal.ca,
> >   larsi@gnus.org, theo@thornhill.no, jostein@secure.kjonigsen.net,
> >   emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > > I'm okay with adding the latter, if it turns out easy enough and safe
> > > enough (of which I'm not sure at all), and if such a command will be
> > > implemented for all the *-ts-modes which have non-ts siblings, but I
> > > see no reason for the former, since there are several simple ways to
> > > cause the same effect, and they are all documented in NEWS.

> > OK, Try this (so far only on c-ts-mode.):


> > diff --git a/etc/NEWS b/etc/NEWS
> > index 2d15e39036a..0a745d7cde9 100644
> > --- a/etc/NEWS
> > +++ b/etc/NEWS
> > @@ -3239,10 +3239,13 @@ for which a "built-in" mode would be turned on.  For example:

> >      (add-to-list 'major-mode-remap-alist '(ruby-mode . ruby-ts-mode))

> > -If you try these modes and don't like them, you can go back to the
> > -"built-in" modes by restarting Emacs.  But please tell us why you
> > -didn't like the tree-sitter based modes, so that we could try
> > -improving them.
> > +Normally, the loading of one of the new modes amends 'auto-mode-alist'
> > +such that future visiting of the same type of file will continue to
> > +use that new mode.  If this is not what you want, do M-x
> > +<mode>-make-ts-undefault-mode.  For a more permanent effect, put, for
> > +example, the following into your initialization file:
> > +
> > +    (eval-after-load 'c-ts-mode '(c-make-ts-undefault-mode))

> Please don't delete the text in NEWS, it is a result of many
> discussions and a lot of thought.  Your proposal is yet another way of
> going back to the non-ts modes, so simply _adding_ that to what's
> already in NEWS should be much better.

OK, I'll try to combine them.

> > +(defun c-make-ts-undefault-mode ()
> > +  "Make the older C and C++ Modes the default major modes for C(++) files."
> > +  (interactive)
> > +  (setq auto-mode-alist (delete '("\\.h\\'" . c-or-c++-ts-mode)
> > +                                auto-mode-alist))
> > +  (setq auto-mode-alist
> > +        (delete '("\\(\\.[chi]\\|\\.lex\\|\\.y\\(acc\\)?\\|\\.x[bp]m\\)\\'" . c-ts-mode)
> > +		auto-mode-alist))
> > +  (setq auto-mode-alist
> > +        (delete
> > +	 '("\\(\\.ii\\|\\.\\(CC?\\|HH?\\)\\|\\.[ch]\\(pp\\|xx\\|\\+\\+\\)\\|\\.\\(cc\\|hh\\)\\)\\'"
> > +	   . c++-ts-mode)
> > +	 auto-mode-alist)))
> > +

> So you revert auto-mode-alist to its original shape, but leave the
> buffers already under c-ts-mode in that mode?  Is that what the users
> would expect, you think?

I think so, yes.  The scenario I am envisaging is the user tentatively
trying c-ts-mode on one, or a few buffers, then doing C-x C-f foo.c to
carry on with her work using C Mode.

> Also, this won't work if the user customized auto-mode-alist in some
> way wrt some of those file-name extensions.

OK.  How about something like the following instead (untested)?

(defun c-make-ts-undefault-mode ()
  "<Doc string>"
  (interactive)
  (let (c)
    (while (setq c (rassq 'c-ts-mode auto-mode-alist))
      (setq auto-mode-alist (remq c auto-mode-alist)))))

> > > Then they [proposed amendments] aren't "reasonable" at this time.
> > > Maybe later, maybe on master.

> > That will be too late, the damage will have been done.

> What "damage"? why do you call "damage" changes made by others in
> Emacs as part of Emacs development?

The damage I'm talking about is the disruption in users' work flow which
is likely to occur.  Being perfectly blunt, c-ts-mode is not yet as
capable as CC Mode in some areas, and in any case its configuration is
wholly different (for good reasons).  Converting to the use of c-ts-mode
is a project, not something which can just happen.  The current code is
likely to confuse and anger users when, after trying out c-ts-mode, it
gradually dawns on them that the reason C Mode no longer works is that
their newly opened files aren't in C Mode at all.  That is what has
happened to me several times.

I'm not calling others' changes damage.  I'm calling the disruptive
effect on users' work flow damage.  That disruption, once it's happened,
cannot ever be undone.

> > It is the first experience people have of the new modes which will
> > create their long term impressions of them.

> Please leave that to people.  We are introducing new technology to
> Emacs, and try doing that in the least intrusive way.  If you don't
> want to help that effort (a stance that is frankly very
> disappointing), at least don't bad-mouth it.

I'm doing my best to help.  If you don't like me using "damage", perhaps
you could suggest a more acceptable synonym expressing the disadvantage
about to be suffered by some of our users.

> > I remember something similar happening in Emacs 21.1, when the new
> > fringes were not made optional.  Lots of users complained loudly and
> > bitterly.

> Well, that's exactly why these new modes are entirely optional.

They're not entirely optional, that's the whole point of my posts over
the last couple of days.  One can opt in to using c-ts-mode, but opting
out again is unreasonably difficult.  Even restarting Emacs (which to me
is a drastic operation) won't opt out if there are still buffers in
c-ts-mode in the desktop.  I don't think the current NEWS item makes this
clear enough.

> > > As I said several times, we have no good idea yet how users will react
> > > to what we have.  Maybe, after we hear from them, we decide to
> > > implement such switches, who knows.

> > We are ourselves all users, too.  We know how we have reacted, and it is
> > reasonable to try to prevent bad experiences for users similar to
> > ourselves.

> For you and me as users, having to restart Emacs, or having to use a
> separate session for such experiments, is an entirely reasonable and
> simple alternative, ....

Having to restart Emacs is NOT at all reasonable for me, even if it may
be for you.  Neither is having to use a separate session.  We all use
Emacs differently, with different expectations.

> .... one which should eliminate any need for undoing the "damage" of
> c-ts-mode.

As I noted above, such restarting will not clear the state of c-ts-mode
being default whilst there are still c-ts-mode buffers in the desktop.
Anyhow, there is no mention of using a separate session in NEWS, and
restarting Emacs is incompletely documented (as already noted).

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Make all tree-sitter modes optional

[Eli Zaretskii]

> Date: Wed, 15 Feb 2023 17:57:15 +0000
> Cc: juri@linkov.net, casouri@gmail.com, monnier@iro.umontreal.ca,
>   larsi@gnus.org, theo@thornhill.no, jostein@secure.kjonigsen.net,
>   emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > So you revert auto-mode-alist to its original shape, but leave the
> > buffers already under c-ts-mode in that mode?  Is that what the users
> > would expect, you think?
> 
> I think so, yes.  The scenario I am envisaging is the user tentatively
> trying c-ts-mode on one, or a few buffers, then doing C-x C-f foo.c to
> carry on with her work using C Mode.

And when going back, they don't want their C/C++ buffers revert to C
mode?  I'd be surprised.

> > Also, this won't work if the user customized auto-mode-alist in some
> > way wrt some of those file-name extensions.
> 
> OK.  How about something like the following instead (untested)?
> 
> (defun c-make-ts-undefault-mode ()
>   "<Doc string>"
>   (interactive)
>   (let (c)
>     (while (setq c (rassq 'c-ts-mode auto-mode-alist))
>       (setq auto-mode-alist (remq c auto-mode-alist)))))

Shouldn't you add the elements of C mode back, in case they were
removed?

> > > > Then they [proposed amendments] aren't "reasonable" at this time.
> > > > Maybe later, maybe on master.
> 
> > > That will be too late, the damage will have been done.
> 
> > What "damage"? why do you call "damage" changes made by others in
> > Emacs as part of Emacs development?
> 
> The damage I'm talking about is the disruption in users' work flow which
> is likely to occur.  Being perfectly blunt, c-ts-mode is not yet as
> capable as CC Mode in some areas, and in any case its configuration is
> wholly different (for good reasons).  Converting to the use of c-ts-mode
> is a project, not something which can just happen.  The current code is
> likely to confuse and anger users when, after trying out c-ts-mode, it
> gradually dawns on them that the reason C Mode no longer works is that
> their newly opened files aren't in C Mode at all.  That is what has
> happened to me several times.

This can happen with any new feature.  There's nothing we can do about
this, so please just stop worrying about it.

> I'm not calling others' changes damage.  I'm calling the disruptive
> effect on users' work flow damage.  That disruption, once it's happened,
> cannot ever be undone.

With the implied assumption that the effect will necessarily be
"disruptive" in many cases.  Why assume that?

> I'm doing my best to help.

Actually, no, you aren't.  "Help" would be to actively partake in the
development of c/c++-ts-mode.  You are our best expert on supporting
these languages, so who better than you to do at least part of this
job, if not coordinate and guide the few brave souls who are motivated
enough to do that in record time.  I'm extremely disappointed that you
completely removed yourself from that effort.  I think we could have
ended up with much better ts modes if you took part in that these last
weeks.

Instead, you only speak up to describe the "disadvantages" of these
new modes, and suggest ways to turn them off.

> > Well, that's exactly why these new modes are entirely optional.
> 
> They're not entirely optional, that's the whole point of my posts over
> the last couple of days.  One can opt in to using c-ts-mode, but opting
> out again is unreasonably difficult.

That's an unusual way of interpreting "opt out".  One doesn't need to
"opt out" of an optional behavior; instead, one should avoid turning
it on, and that's all!

> Even restarting Emacs (which to me is a drastic operation) won't opt
> out if there are still buffers in c-ts-mode in the desktop.

Many people restart Emacs all the time.  And those who use desktop
know how to overcome such problems, which aren't unique to these
modes.

> I don't think the current NEWS item makes this clear enough.

The current NEWS already goes way beyond its purpose and scope in
presenting these new modes and the related issues.  And I object to
using NEWS to tell users in too much detail how NOT to use our new
features: that is like shooting ourselves in the foot.

> > For you and me as users, having to restart Emacs, or having to use a
> > separate session for such experiments, is an entirely reasonable and
> > simple alternative, ....
> 
> Having to restart Emacs is NOT at all reasonable for me, even if it may
> be for you.  Neither is having to use a separate session.  We all use
> Emacs differently, with different expectations.
> 
> > .... one which should eliminate any need for undoing the "damage" of
> > c-ts-mode.
> 
> As I noted above, such restarting will not clear the state of c-ts-mode
> being default whilst there are still c-ts-mode buffers in the desktop.
> Anyhow, there is no mention of using a separate session in NEWS, and
> restarting Emacs is incompletely documented (as already noted).

Sounds like you run your full customized production environment in
test runs of Emacs.  The prudent thing to do is instead to either use
"emacs -Q" or to have a special user/home directory which you use for
such jobs.  Then restarting and/or deleting the desktop is not an
issue at all.  I'm surprised I need to explain that, I though
everybody who is involved in Emacs maintenance does that.

Re: Make all tree-sitter modes optional

[Alan Mackenzie]

Hello, Eli.

On Wed, Feb 15, 2023 at 20:33:49 +0200, Eli Zaretskii wrote:
> > Date: Wed, 15 Feb 2023 17:57:15 +0000
> > Cc: juri@linkov.net, casouri@gmail.com, monnier@iro.umontreal.ca,
> >   larsi@gnus.org, theo@thornhill.no, jostein@secure.kjonigsen.net,
> >   emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

[ .... ]

> > I'm doing my best to help.

> Actually, no, you aren't.  "Help" would be to actively partake in the
> development of c/c++-ts-mode.  You are our best expert on supporting
> these languages, so who better than you to do at least part of this
> job, if not coordinate and guide the few brave souls who are motivated
> enough to do that in record time.  I'm extremely disappointed that you
> completely removed yourself from that effort.  I think we could have
> ended up with much better ts modes if you took part in that these last
> weeks.

I think it's worth answering this point separately.

There's plenty in CC Mode which isn't good, and a danger of me actively
participating in the development of the -ts- modes is that some of this
bad stuff will get transferred.  I'm surely not the best judge of which
bits the bad bits are.

In the very nature of things, there's bound to be some competition
between CC Mode and the new modes, and me being on both "sides" would be
a psychologically uncomfortable position to be in.  In the long run, of
course, CC Mode cannot "win", and I accept that.

But you've got to allow me to have mixed feelings when the project I've
put so much into over ~20 years is facing redundancy.

On top of these considerations, there was a deluge of bugs reported in CC
Mode over the autumn.  I felt my priority had to be the maintenance of CC
Mode.

> Instead, you only speak up to describe the "disadvantages" of these
> new modes, ....

Yes, in the spirit of improving them, much as I have reported bugs in
other parts of Emacs.

> .... and suggest ways to turn them off.

Yes.  Plenty of people like CC Mode, but I don't think you are among
them.  These people are likely to want to carry on using CC Mode, at
least in the short term, and to do this they need a way to switch off the
new modes.  I think you must agree with me there.  Where we differ is
that I want to make this way of restoring CC Mode's position in
auto-mode-alist easy, whereas you seem to be content that it remain
difficult.  I confess I don't understand why.

[ .... ]

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Make all tree-sitter modes optional

[tomas]

[-- Attachment #1: Type: text/plain, Size: 286 bytes --]

On Wed, Feb 15, 2023 at 08:31:46PM +0000, Alan Mackenzie wrote:
> Hello, Eli.

[...]

> Yes.  Plenty of people like CC Mode [...]

FWIW, I'm one of them. I'll sure try out ts-mode at some point,
but definitely not now (priorities...). And I doubt I'd switch.

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Make all tree-sitter modes optional

[Eli Zaretskii]

> Date: Wed, 15 Feb 2023 20:31:46 +0000
> Cc: juri@linkov.net, casouri@gmail.com, monnier@iro.umontreal.ca,
>   larsi@gnus.org, theo@thornhill.no, jostein@secure.kjonigsen.net,
>   emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> But you've got to allow me to have mixed feelings when the project I've
> put so much into over ~20 years is facing redundancy.

It isn't facing redundancy, it's nowhere near that.  You don't need to
worry about that.  I expect a significant proportion of users to wish
to stay with CC Mode, for several good reasons:

 . the use cases it handles better that c-ts-mode (cpp stuff etc.)
 . the plethora of minor conveniences it offers that c-ts-mode
   doesn't, at least not yet, such as much more elaborate
   customizations of indentation and electric behavior

I'm not even sure yet whether I myself will switch.  I will give the
c-ts-mode a lot of leeway and credit, but I don't know yet what will
be the outcome.

> > .... and suggest ways to turn them off.
> 
> Yes.  Plenty of people like CC Mode, but I don't think you are among
> them.

How can I not like it?  I use it every day, for several decades.  I
have several non-trivial customizations of it, some of which I will
probably miss with c-ts-mode, which doesn't yet support them, and
maybe never will.

> These people are likely to want to carry on using CC Mode, at
> least in the short term, and to do this they need a way to switch off the
> new modes.

The new modes don't need to be switched off because they are off to
begin with.  People who don't like them don't need to change their
configuration even a single bit.  I specifically and explicitly took
care of that, over the hesitations, bewilderment, and downright
objections of several good people here.  You have just heard them
again, and they still disagree with me over the result, with valid
arguments.  So I find it ironic that all that effort is now seen as
insufficient, by you of all people.  Maybe you should try looking at
this issue from my POV.

> I think you must agree with me there.  Where we differ is
> that I want to make this way of restoring CC Mode's position in
> auto-mode-alist easy, whereas you seem to be content that it remain
> difficult.  I confess I don't understand why.

"Difficult"?  Restarting Emacs is not difficult.  Many users do that
all the time (watch the posts about how fast Emacs should start up),
even though we recommend not to do that.  And if the user finds he or
she doesn't like the new mode, that restart is a single event, no need
to do it more than once.  That's not my notion of "difficult".

Re: Make all tree-sitter modes optional

[Yuan Fu]

> On Feb 15, 2023, at 11:27 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> Date: Wed, 15 Feb 2023 20:31:46 +0000
>> Cc: juri@linkov.net, casouri@gmail.com, monnier@iro.umontreal.ca,
>>  larsi@gnus.org, theo@thornhill.no, jostein@secure.kjonigsen.net,
>>  emacs-devel@gnu.org
>> From: Alan Mackenzie <acm@muc.de>
>> 
>> But you've got to allow me to have mixed feelings when the project I've
>> put so much into over ~20 years is facing redundancy.
> 
> It isn't facing redundancy, it's nowhere near that.  You don't need to
> worry about that.  I expect a significant proportion of users to wish
> to stay with CC Mode, for several good reasons:
> 
> . the use cases it handles better that c-ts-mode (cpp stuff etc.)
> . the plethora of minor conveniences it offers that c-ts-mode
>   doesn't, at least not yet, such as much more elaborate
>   customizations of indentation and electric behavior
> 
> I'm not even sure yet whether I myself will switch.  I will give the
> c-ts-mode a lot of leeway and credit, but I don't know yet what will
> be the outcome.

+1. It’ll be a long way until c-ts-mode can be comparable to c-mode.

Yuan

Re: Make all tree-sitter modes optional

[Eli Zaretskii]

> From: dick <dick.r.chiang@gmail.com>
> Cc: Alan Mackenzie <acm@muc.de>,  juri@linkov.net,  casouri@gmail.com,
>   monnier@iro.umontreal.ca,  larsi@gnus.org,  theo@thornhill.no,
>   jostein@secure.kjonigsen.net,  emacs-devel@gnu.org
> Date: Wed, 15 Feb 2023 15:11:05 -0500
> 
> EZ> Instead, you only ... suggest ways to turn them off.
> 
> Why the continued pretense of surprise at cc-mode guy's naysaying?
> 
> If your reliable but decades-old module were aggressively being unseated,
> you'd react with equal hostility.  Nearly all middle-aged programmers,
> myself among them, would.

Will you ever learn that when someone says something that seems
trivial and maybe ridiculous to you, it could be for reasons other
than stupidity or ignorance?  And that you should think harder to try
to figure out what you might be missing here, rather than assume that
you are the only bearer of wisdom around?

Re: Make all tree-sitter modes optional

[Eli Zaretskii]

> From: dick <dick.r.chiang@gmail.com>
> Cc: acm@muc.de,  juri@linkov.net,  casouri@gmail.com,
>   monnier@iro.umontreal.ca,  larsi@gnus.org,  theo@thornhill.no,
>   jostein@secure.kjonigsen.net,  emacs-devel@gnu.org
> Date: Wed, 15 Feb 2023 16:58:39 -0500
> 
> > it could be for reasons other than stupidity?

The full citation was:

> > Will you ever learn that when someone says something that seems
> > trivial and maybe ridiculous to you, it could be for reasons other
> > than stupidity or ignorance?

> Unfalsifiability is your rhetorical crutch of choice.  Examples include,
> "You don't understand the code", and "I have my reasons."

I guess the answer to my question above is "not yet".

> Unlike you, I work on emacs full-time, so consider my opinion a
> professional one.  That your initial reaction to Bug#60559 was
> "notabug wontfix" suggests you could use it.

That bug is fixed, by yours truly, btw.

> It will be years before any *working* C programmer would default to
> c-ts-mode over cc-mode.

You may be right here.  We shall see.

Re: Make all tree-sitter modes optional

[Po Lu]

People, when replying to dick, it would be easier to simply say:

      *plonk*

perform the associated action, and then let that kook and/or troll shout
his head off in peace. :-)

Re: Make all tree-sitter modes optional

[Alan Mackenzie]

Hello, Eli.

On Wed, Feb 15, 2023 at 20:33:49 +0200, Eli Zaretskii wrote:
> > Date: Wed, 15 Feb 2023 17:57:15 +0000
> > Cc: juri@linkov.net, casouri@gmail.com, monnier@iro.umontreal.ca,
> >   larsi@gnus.org, theo@thornhill.no, jostein@secure.kjonigsen.net,
> >   emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > > So you revert auto-mode-alist to its original shape, but leave the
> > > buffers already under c-ts-mode in that mode?  Is that what the users
> > > would expect, you think?

> > I think so, yes.  The scenario I am envisaging is the user tentatively
> > trying c-ts-mode on one, or a few buffers, then doing C-x C-f foo.c to
> > carry on with her work using C Mode.

> And when going back, they don't want their C/C++ buffers revert to C
> mode?  I'd be surprised.

I wouldn't.  If I type M-x foo-mode in a buffer, I would be surprised if
something changed that without asking me.

> > > Also, this won't work if the user customized auto-mode-alist in some
> > > way wrt some of those file-name extensions.

> > OK.  How about something like the following instead (untested)?

> > (defun c-make-ts-undefault-mode ()
> >   "<Doc string>"
> >   (interactive)
> >   (let (c)
> >     (while (setq c (rassq 'c-ts-mode auto-mode-alist))
> >       (setq auto-mode-alist (remq c auto-mode-alist)))))

> Shouldn't you add the elements of C mode back, in case they were
> removed?

Possibly.  But we're looking for something simple and reliable at the
moment.  Adding code to check for the presence of c-mode in
auto-mode-alist would add some complexity, and may not really be needed.

> > > > > Then they [proposed amendments] aren't "reasonable" at this time.
> > > > > Maybe later, maybe on master.

> > > > That will be too late, the damage will have been done.

> > > What "damage"? why do you call "damage" changes made by others in
> > > Emacs as part of Emacs development?

> > The damage I'm talking about is the disruption in users' work flow which
> > is likely to occur.  Being perfectly blunt, c-ts-mode is not yet as
> > capable as CC Mode in some areas, and in any case its configuration is
> > wholly different (for good reasons).  Converting to the use of c-ts-mode
> > is a project, not something which can just happen.  The current code is
> > likely to confuse and anger users when, after trying out c-ts-mode, it
> > gradually dawns on them that the reason C Mode no longer works is that
> > their newly opened files aren't in C Mode at all.  That is what has
> > happened to me several times.

> This can happen with any new feature.  There's nothing we can do about
> this, so please just stop worrying about it.

I think we can do things about it, hence my patch from yesterday evening
and this discussion.

> > I'm not calling others' changes damage.  I'm calling the disruptive
> > effect on users' work flow damage.  That disruption, once it's happened,
> > cannot ever be undone.

> With the implied assumption that the effect will necessarily be
> "disruptive" in many cases.  Why assume that?

Because it's unexpected to the users, something which throws them out of
their rhythm, and forces them to expend time and energy getting back to
where they wanted to be.  Not all users, but likely enough of them to
matter.

[ .... ]

> > > Well, that's exactly why these new modes are entirely optional.

> > They're not entirely optional, that's the whole point of my posts over
> > the last couple of days.  One can opt in to using c-ts-mode, but opting
> > out again is unreasonably difficult.

> That's an unusual way of interpreting "opt out".  One doesn't need to
> "opt out" of an optional behavior; instead, one should avoid turning
> it on, and that's all!

Again, we differ here.  You're picturing a process where a user one day
makes a definitive decision to switch, and does so permanently.  Maybe
that's how you work, it's not how I work.  I would find myself with a
spare few minutes, experiment a bit with the new mode, then carry on with
my work, maybe experiment a bit more.  I would subconsciously expect my
Emacs state would not be changed by such experimentation.  But it is, in
a way which is difficult to reverse.  Again, I don't understand why
you're content that such difficulty remain.

> > Even restarting Emacs (which to me is a drastic operation) won't opt
> > out if there are still buffers in c-ts-mode in the desktop.

> Many people restart Emacs all the time.  And those who use desktop
> know how to overcome such problems, which aren't unique to these
> modes.

Probably most users start Emacs in the morning and shut it down in the
evening.  I certainly do.  I think a user who gets into this problem is
going to have to spend, perhaps, between 1 and 2 hours getting it sorted
out, or on the other hand, will continue dissatisfied at having to type
M-x c-mode after loading each and every C file.  Why do you not see this
as a bad thing?

> > I don't think the current NEWS item makes this clear enough.

> The current NEWS already goes way beyond its purpose and scope in
> presenting these new modes and the related issues.  And I object to
> using NEWS to tell users in too much detail how NOT to use our new
> features: that is like shooting ourselves in the foot.

NEWS has always introduced new features, and amongst the description is
(almost) always some text telling users how to restore the old behaviour.
Why do we not do that here?

> > > For you and me as users, having to restart Emacs, or having to use a
> > > separate session for such experiments, is an entirely reasonable and
> > > simple alternative, ....

> > Having to restart Emacs is NOT at all reasonable for me, even if it may
> > be for you.  Neither is having to use a separate session.  We all use
> > Emacs differently, with different expectations.

> > > .... one which should eliminate any need for undoing the "damage" of
> > > c-ts-mode.

> > As I noted above, such restarting will not clear the state of c-ts-mode
> > being default whilst there are still c-ts-mode buffers in the desktop.
> > Anyhow, there is no mention of using a separate session in NEWS, and
> > restarting Emacs is incompletely documented (as already noted).

> Sounds like you run your full customized production environment in
> test runs of Emacs.  The prudent thing to do is instead to either use
> "emacs -Q" or to have a special user/home directory which you use for
> such jobs.  Then restarting and/or deleting the desktop is not an
> issue at all.  I'm surprised I need to explain that, I thought
> everybody who is involved in Emacs maintenance does that.

Everybody uses Emacs differently.  Even (especially) amongst Emacs
developers.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Make all tree-sitter modes optional

[Alan Mackenzie]

Hello, Eli.

On Wed, Feb 15, 2023 at 20:33:49 +0200, Eli Zaretskii wrote:
> > Date: Wed, 15 Feb 2023 17:57:15 +0000
> > Cc: juri@linkov.net, casouri@gmail.com, monnier@iro.umontreal.ca,
> >   larsi@gnus.org, theo@thornhill.no, jostein@secure.kjonigsen.net,
> >   emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

[ .... ]

> This [severe inconvenience to users who wish to stop using a new
> feature] can happen with any new feature.  There's nothing we can do about
> this, ....

There is; we're competent hackers.

> .... so please just stop worrying about it.

I continue to be unhappy about the state of this change.  If we are going
to insist on users restarting Emacs simply to remove three entries from
auto-mode-alist, we should at least ensure that such restarting will
work.  In the current NEWS, the directions are incomplete and misleading.

I propose the following to correct this; it adds text to what is
currently there rather than replacing it, as you requested a couple of
days ago:


diff --git a/etc/NEWS b/etc/NEWS
index 2d15e39036a..dbe6517e78f 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -3239,10 +3239,13 @@ for which a "built-in" mode would be turned on.  For example:
 
     (add-to-list 'major-mode-remap-alist '(ruby-mode . ruby-ts-mode))
 
-If you try these modes and don't like them, you can go back to the
-"built-in" modes by restarting Emacs.  But please tell us why you
-didn't like the tree-sitter based modes, so that we could try
-improving them.
+Loading one of the new modes amends 'auto-mode-alist' such that
+visiting the same type of file in the future will continue to use that
+new mode.  If you try these modes and don't like them, you can go back
+to the "built-in" modes by restarting Emacs, but first, if you use
+desktop-save-mode, make sure no buffers using the new mode will get
+entered into your .desktop file.  But please tell us why you didn't
+like the tree-sitter based modes, so that we can try improving them.
 
 Each major mode based on tree-sitter needs a language grammar library,
 usually named "libtree-sitter-LANG.so" ("libtree-sitter-LANG.dll" on


-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Make all tree-sitter modes optional

[Po Lu]

Alan Mackenzie <acm@muc.de> writes:

> Hello, Eli.
>
> On Wed, Feb 15, 2023 at 20:33:49 +0200, Eli Zaretskii wrote:
>> > Date: Wed, 15 Feb 2023 17:57:15 +0000
>> > Cc: juri@linkov.net, casouri@gmail.com, monnier@iro.umontreal.ca,
>> >   larsi@gnus.org, theo@thornhill.no, jostein@secure.kjonigsen.net,
>> >   emacs-devel@gnu.org
>> > From: Alan Mackenzie <acm@muc.de>
>
> [ .... ]
>
>> This [severe inconvenience to users who wish to stop using a new
>> feature] can happen with any new feature.  There's nothing we can do about
>> this, ....
>
> There is; we're competent hackers.
>
>> .... so please just stop worrying about it.
>
> I continue to be unhappy about the state of this change.  If we are going
> to insist on users restarting Emacs simply to remove three entries from
> auto-mode-alist, we should at least ensure that such restarting will
> work.  In the current NEWS, the directions are incomplete and misleading.
>
> I propose the following to correct this; it adds text to what is
> currently there rather than replacing it, as you requested a couple of
> days ago:
>
> diff --git a/etc/NEWS b/etc/NEWS
> index 2d15e39036a..dbe6517e78f 100644
> --- a/etc/NEWS
> +++ b/etc/NEWS
> @@ -3239,10 +3239,13 @@ for which a "built-in" mode would be turned on.  For example:
>  
>      (add-to-list 'major-mode-remap-alist '(ruby-mode . ruby-ts-mode))
>  
> -If you try these modes and don't like them, you can go back to the
> -"built-in" modes by restarting Emacs.  But please tell us why you
> -didn't like the tree-sitter based modes, so that we could try
> -improving them.
> +Loading one of the new modes amends 'auto-mode-alist' such that
> +visiting the same type of file in the future will continue to use that
> +new mode.  If you try these modes and don't like them, you can go back
> +to the "built-in" modes by restarting Emacs, but first, if you use
> +desktop-save-mode, make sure no buffers using the new mode will get
> +entered into your .desktop file.  But please tell us why you didn't
> +like the tree-sitter based modes, so that we can try improving them.
>  
>  Each major mode based on tree-sitter needs a language grammar library,
>  usually named "libtree-sitter-LANG.so" ("libtree-sitter-LANG.dll" on

Let me ask what I asked earlier, again?
Does c-ts-mode make itself default upon being loaded, or does it make
itself the default upon being first enabled?

The former has been a Bad Thing for as long as I can remember.
The latter is not particularly problematic as long as we document the
caveat.

Thanks.

Re: Make all tree-sitter modes optional

[Stefan Monnier]

> Does c-ts-mode make itself default upon being loaded, or does it make
> itself the default upon being first enabled?

Upon being loaded.

> The former has been a Bad Thing for as long as I can remember.

Yup.

> The latter is not particularly problematic as long as we document
> the caveat.

I can live with that one, yes.


        Stefan

Re: Make all tree-sitter modes optional

[Po Lu]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Does c-ts-mode make itself default upon being loaded, or does it make
>> itself the default upon being first enabled?
>
> Upon being loaded.

That is extremely nasty, especially for people who preload all (or most)
Lisp libraries.

c-ts-mode should be fixed before the pretest starts.  It is not some
small triviality that can be dismissed.

Alan Mackenzie <acm@muc.de> writes:

>> Let me ask what I asked earlier, again?
>> Does c-ts-mode make itself default upon being loaded, or does it make
>> itself the default upon being first enabled?
>
> It makes itself default upon loading, even if that loading is only in
> response to C-h C-f c-ts-mode.
>
>> The former has been a Bad Thing for as long as I can remember.
>
> Indeed.

This means that if we release Emacs 29 as-is, the resulting Emacs will
be broken.

There is NO REASON asking Emacs to describe something should result in
changes to auto-mode-alist.

Re: Make all tree-sitter modes optional

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: Alan Mackenzie <acm@muc.de>,  Eli Zaretskii <eliz@gnu.org>,
>   juri@linkov.net,  casouri@gmail.com,  larsi@gnus.org,  theo@thornhill.no,
>   jostein@secure.kjonigsen.net,  emacs-devel@gnu.org
> Date: Fri, 17 Feb 2023 22:16:07 +0800
> 
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
> 
> >> Does c-ts-mode make itself default upon being loaded, or does it make
> >> itself the default upon being first enabled?
> >
> > Upon being loaded.
> 
> That is extremely nasty, especially for people who preload all (or most)
> Lisp libraries.
> 
> c-ts-mode should be fixed before the pretest starts.

It won't be.

Please, everybody, stop pushing for these changes.  I understand that
you disagree, but I respectfully request that you-all assume that
either I'm right here (in that I consider the alternatives to be
worse), or if I'm wrong, it is for me to make this mistake and learn
from it.  Please give me some minimal credit that I have thought long
and hard about the possible alternatives, and didn't arrive at this
conclusion easily, and that I'm also fully aware that changing the
behavior by loading a package is not good, in and of itself.

TIA

Re: Make all tree-sitter modes optional

[Dmitry Gutov]

On 17/02/2023 16:40, Eli Zaretskii wrote:
> I understand that
> you disagree, but I respectfully request that you-all assume that
> either I'm right here (in that I consider the alternatives to be
> worse), or if I'm wrong, it is for me to make this mistake and learn
> from it.  Please give me some minimal credit that I have thought long
> and hard about the possible alternatives, and didn't arrive at this
> conclusion easily, and that I'm also fully aware that changing the
> behavior by loading a package is not good, in and of itself.

I suppose there is not much to add here, and the mistake (if it is one, 
respectfully) is yours to make.

I just don't understand what's your plan here regarding Emacs 29. What's 
going to happen next? What kind of feedback will you be looking for?

What I think will happen, is people will try out the new modes, some 
will suffer the inconveniences we warned about here and possibly think 
less of Emacs as a result; others will avoid those problems by accident; 
yet a lot more users will never try these new modes and thus avoid the 
problems as well.

If we're lucky, we get a couple of new bug reports associated with it, 
maybe 1-6 months after the release: a lot of users don't report 
problems, much less these less obvious ones, where the behavior doesn't 
end up in a "error" written somewhere. The reports will likely repeat 
some of what's already been said.

At what point does this turn into some kind of conclusion, and a 
teaching moment, so to speak?

Re: Make all tree-sitter modes optional

[Eli Zaretskii]

> Date: Fri, 17 Feb 2023 16:56:51 +0200
> Cc: monnier@iro.umontreal.ca, acm@muc.de, juri@linkov.net, casouri@gmail.com,
>  larsi@gnus.org, theo@thornhill.no, jostein@secure.kjonigsen.net,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> I just don't understand what's your plan here regarding Emacs 29. What's 
> going to happen next? What kind of feedback will you be looking for?

Whatever feedback we will get.  I don't know what we will hear, and
I'm not sure why I should try guessing.

> What I think will happen, is people will try out the new modes, some 
> will suffer the inconveniences we warned about here and possibly think 
> less of Emacs as a result; others will avoid those problems by accident; 
> yet a lot more users will never try these new modes and thus avoid the 
> problems as well.
> 
> If we're lucky, we get a couple of new bug reports associated with it, 
> maybe 1-6 months after the release: a lot of users don't report 
> problems, much less these less obvious ones, where the behavior doesn't 
> end up in a "error" written somewhere. The reports will likely repeat 
> some of what's already been said.

Something like that, yes.  Except that the 6-month figurer could be
better (or worse), and some of the reports might actually tell us
something that wasn't yet said or proposed.

> At what point does this turn into some kind of conclusion, and a 
> teaching moment, so to speak?

It depends on what we hear.  It is quite possible that a single report
will show the light.  Or a significant number of reports expressing a
particular opinion will change the weight of that opinion.  Or
something else.  (Or I step down, or am overrun by a bus.)

Re:

[Alan Mackenzie]

Hello, Dick.

On Fri, Feb 17, 2023 at 10:24:43 -0500, dick wrote:
> EZ> Or I step down

> This wouldn't be the first time you've threatened that when you were on
> the wrong side of history.  Examples include bzr-to-git and pdumper.

> Perhaps now is the time to make good.  I know at least one person
> who'd like to take the reins.

Please refrain from making such unhelpful, and unnecessarily personl
insinuations.  They go against the respectful tenor which prevails on
this mailing list and, I think, are unwanted by everybody here.

Thanks.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Make all tree-sitter modes optional

[Stefan Kangas]

Dmitry Gutov <dgutov@yandex.ru> writes:

> I just don't understand what's your plan here regarding Emacs 29. What's
> going to happen next? What kind of feedback will you be looking for?
>
> What I think will happen, is people will try out the new modes, some
> will suffer the inconveniences we warned about here and possibly think
> less of Emacs as a result;

Changing Emacs' behavior upon loading a file is very bad.  We already
know this (we even put it at the top of our coding conventions), and the
fix is trivial.  So from my point of view, there is no need to wait for
users to tell us about it; we can act on it immediately.

In comparison, I consider changing `auto-mode-alist' upon enabling a
major mode a mere annoyance.  I'll be able to hack my way around it
myself, of course, but I think it will make for a bad user experience.
Especially since it breaks our existing conventions: users are not
trained to know that enabling a mode in one buffer will change
`auto-mode-alist' globally.  It is surprising, to say the least.

Restarting Emacs is not a solution at all.  It breaks my workflow, and I
know that it breaks that of many users (e.g. those that start Emacs as a
daemon when logging in).

Re: Make all tree-sitter modes optional

[Morgan Willcock]

Stefan Kangas <stefankangas@gmail.com> writes:

> Dmitry Gutov <dgutov@yandex.ru> writes:
>
>> I just don't understand what's your plan here regarding Emacs 29. What's
>> going to happen next? What kind of feedback will you be looking for?
>>
>> What I think will happen, is people will try out the new modes, some
>> will suffer the inconveniences we warned about here and possibly think
>> less of Emacs as a result;
>
> Changing Emacs' behavior upon loading a file is very bad.  We already
> know this (we even put it at the top of our coding conventions), and the
> fix is trivial.  So from my point of view, there is no need to wait for
> users to tell us about it; we can act on it immediately.
>
> In comparison, I consider changing `auto-mode-alist' upon enabling a
> major mode a mere annoyance.  I'll be able to hack my way around it
> myself, of course, but I think it will make for a bad user experience.
> Especially since it breaks our existing conventions: users are not
> trained to know that enabling a mode in one buffer will change
> `auto-mode-alist' globally.  It is surprising, to say the least.
>
> Restarting Emacs is not a solution at all.  It breaks my workflow, and I
> know that it breaks that of many users (e.g. those that start Emacs as a
> daemon when logging in).

And for someone using EXWM, requiring a restart is asking the user to
logout and then login again.

A slower and more stable path sounds more appealing to me.

Morgan

Re: Make all tree-sitter modes optional

[Gregory Heytings]

May I suggest the following: enclose the

(if (treesit-ready-p ...)
     (add-to-list 'auto-mode-alist ...))

statements in a conditional, say (when treesit-add-to-auto-mode-alist 
...), where treesit-add-to-auto-mode-alist would be a defcustom defaulting 
to t?

This would make it possible for users who don't want that behavior to turn 
it off, and would make everyone happy (I hope).

Re: Make all tree-sitter modes optional

[Alan Mackenzie]

Hello, Po.

On Fri, Feb 17, 2023 at 21:37:59 +0800, Po Lu wrote:
> Alan Mackenzie <acm@muc.de> writes:

> > On Wed, Feb 15, 2023 at 20:33:49 +0200, Eli Zaretskii wrote:
> >> > Date: Wed, 15 Feb 2023 17:57:15 +0000
> >> > Cc: juri@linkov.net, casouri@gmail.com, monnier@iro.umontreal.ca,
> >> >   larsi@gnus.org, theo@thornhill.no, jostein@secure.kjonigsen.net,
> >> >   emacs-devel@gnu.org
> >> > From: Alan Mackenzie <acm@muc.de>

[ .... ]

> Let me ask what I asked earlier, again?
> Does c-ts-mode make itself default upon being loaded, or does it make
> itself the default upon being first enabled?

It makes itself default upon loading, even if that loading is only in
response to C-h C-f c-ts-mode.

> The former has been a Bad Thing for as long as I can remember.

Indeed.

> The latter is not particularly problematic as long as we document the
> caveat.

I don't quite agree.  An Emacs user should retain full control over her
Emacs session.  Silently changing the entries in auto-mode-alist removes
that control and causes confusion.

> Thanks.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Make all tree-sitter modes optional

[Po Lu]

(I copied the list back in since this is quite relevant to the thread.)

Dmitry Gutov <dgutov@yandex.ru> writes:

> I understand there is an issue of a possible language barrier, but
> isn't that something you could have found out in ten seconds by trying
> it out? (require 'c-ts-mode) alters auto-mode-alist.

But then it could've been a bug, not intentional behavior.

> And about a month ago you called this "a very reasonable arrangement":
>
> https://lists.gnu.org/archive/html/emacs-devel/2023-01/msg00267.html
>
> Here's a couple of quotes from the message you responded to:
>
>> The proposed change to the current emacs-29 branch is below.  You will
>> see that where possible, just loading a TS mode modifies
>> auto-mode-alist if the tree-sitter support for that mode is available,
>> whereas for other modes auto-mode-alist is modified only when the mode
>> is actually turned on successfully for the first time.
>
> ...
>
>> Modes that have
>> their own *.el files can be just 'require'd in the init file to modify
>> auto-mode-alist.

At that time I thought in both paragraphs the `require' and ``loading''
parts referred to enabling the major mode itself, since there is no
other reasonable behavior.

I didn't even bother to ask about what I thought was a typo, since I
wasn't expecting anyone to come up with any other behavior.

Re: Make all tree-sitter modes optional

[Eli Zaretskii]

> Date: Fri, 17 Feb 2023 13:30:58 +0000
> Cc: juri@linkov.net, casouri@gmail.com, monnier@iro.umontreal.ca,
>   larsi@gnus.org, theo@thornhill.no, jostein@secure.kjonigsen.net,
>   emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> I continue to be unhappy about the state of this change.  If we are going
> to insist on users restarting Emacs simply to remove three entries from
> auto-mode-alist, we should at least ensure that such restarting will
> work.  In the current NEWS, the directions are incomplete and misleading.
> 
> I propose the following to correct this; it adds text to what is
> currently there rather than replacing it, as you requested a couple of
> days ago:

Thanks, I used a slightly different text to that effect.

Re: Make all tree-sitter modes optional

[Alan Mackenzie]

Hello, Eli.

On Fri, Feb 17, 2023 at 16:58:42 +0200, Eli Zaretskii wrote:
> > Date: Fri, 17 Feb 2023 13:30:58 +0000
> > Cc: juri@linkov.net, casouri@gmail.com, monnier@iro.umontreal.ca,
> >   larsi@gnus.org, theo@thornhill.no, jostein@secure.kjonigsen.net,
> >   emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > I continue to be unhappy about the state of this change.  If we are going
> > to insist on users restarting Emacs simply to remove three entries from
> > auto-mode-alist, we should at least ensure that such restarting will
> > work.  In the current NEWS, the directions are incomplete and misleading.

> > I propose the following to correct this; it adds text to what is
> > currently there rather than replacing it, as you requested a couple of
> > days ago:

> Thanks, I used a slightly different text to that effect.

Thanks!

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Make all tree-sitter modes optional

[Stefan Monnier]

> out again is unreasonably difficult.  Even restarting Emacs (which to me
> is a drastic operation) won't opt out if there are still buffers in
> c-ts-mode in the desktop.

Sounds like a bug, indeed.
The `add-to-list` should not happen just because `c-ts-mode` is loaded.
I think it should only happen if `c-ts-mode` is called explicitly
(i.e. interactively).

Maybe something like the patch below.

We could even add a `y-or-n-p` test in `c-ts--activate` to avoid
changing config vars without the user's consent.


        Stefan


diff --git a/lisp/progmodes/c-ts-mode.el b/lisp/progmodes/c-ts-mode.el
index 6db28459c32..e4148649822 100644
--- a/lisp/progmodes/c-ts-mode.el
+++ b/lisp/progmodes/c-ts-mode.el
@@ -740,7 +740,6 @@ c-ts-base-mode-map
   "C-c C-q" #'c-ts-mode-indent-defun
   "C-c ." #'c-ts-mode-set-style)
 
-;;;###autoload
 (define-derived-mode c-ts-base-mode prog-mode "C"
   "Major mode for editing C, powered by tree-sitter.
 
@@ -856,6 +855,7 @@ c-ts-mode
   :group 'c
 
   (when (treesit-ready-p 'c)
+    (when (called-interactively-p 'any) (c-ts--activate))
     (treesit-parser-create 'c)
     ;; Comments.
     (setq-local comment-start "/* ")
@@ -888,6 +888,7 @@ c++-ts-mode
   :group 'c++
 
   (when (treesit-ready-p 'cpp)
+    (when (called-interactively-p 'any) (c-ts--activate))
     (setq-local treesit-text-type-regexp
                 (regexp-opt '("comment"
                               "raw_string_literal")))
@@ -942,6 +943,7 @@ c-or-c++-ts-mode
 the code is C or C++ and based on that chooses whether to enable
 `c-ts-mode' or `c++-ts-mode'."
   (interactive)
+  (when (called-interactively-p 'any) (c-ts--activate))
   (if (save-excursion
         (save-restriction
           (save-match-data ; Why `save-match-data'?
@@ -950,22 +952,30 @@ c-or-c++-ts-mode
             (re-search-forward c-ts-mode--c-or-c++-regexp nil t))))
       (c++-ts-mode)
     (c-ts-mode)))
-;; The entries for C++ must come first to prevent *.c files be taken
-;; as C++ on case-insensitive filesystems, since *.C files are C++,
-;; not C.
-(if (treesit-ready-p 'cpp)
-    (add-to-list 'auto-mode-alist
-                 '("\\(\\.ii\\|\\.\\(CC?\\|HH?\\)\\|\\.[ch]\\(pp\\|xx\\|\\+\\+\\)\\|\\.\\(cc\\|hh\\)\\)\\'"
-                   . c++-ts-mode)))
-
-(if (treesit-ready-p 'c)
-    (add-to-list 'auto-mode-alist
-                 '("\\(\\.[chi]\\|\\.lex\\|\\.y\\(acc\\)?\\|\\.x[bp]m\\)\\'"
-                   . c-ts-mode)))
-
-(if (and (treesit-ready-p 'cpp)
-         (treesit-ready-p 'c))
-    (add-to-list 'auto-mode-alist '("\\.h\\'" . c-or-c++-ts-mode)))
+
+(defun c-ts--activate ()
+  (unless (or (rassq 'c++-ts-mode auto-mode-alist)
+              (rassq 'c-ts-mode auto-mode-alist)
+              (rassq 'c-or-c++-ts-mode auto-mode-alist)
+              (rassq 'c++-ts-mode major-mode-remap-alist)
+              (rassq 'c-ts-mode major-mode-remap-alist)
+              (rassq 'c-or-c++-ts-mode major-mode-remap-alist))
+    ;; The entries for C++ must come first to prevent *.c files be taken
+    ;; as C++ on case-insensitive filesystems, since *.C files are C++,
+    ;; not C.
+    (if (treesit-ready-p 'cpp)
+        (add-to-list 'auto-mode-alist
+                     '("\\(\\.ii\\|\\.\\(CC?\\|HH?\\)\\|\\.[ch]\\(pp\\|xx\\|\\+\\+\\)\\|\\.\\(cc\\|hh\\)\\)\\'"
+                       . c++-ts-mode)))
+
+    (if (treesit-ready-p 'c)
+        (add-to-list 'auto-mode-alist
+                     '("\\(\\.[chi]\\|\\.lex\\|\\.y\\(acc\\)?\\|\\.x[bp]m\\)\\'"
+                       . c-ts-mode)))
+
+    (if (and (treesit-ready-p 'cpp)
+             (treesit-ready-p 'c))
+        (add-to-list 'auto-mode-alist '("\\.h\\'" . c-or-c++-ts-mode)))))
 
 (provide 'c-ts-mode)
 

Re: Make all tree-sitter modes optional

[Dmitry Gutov]

On 15/02/2023 20:34, Stefan Monnier wrote:
>> out again is unreasonably difficult.  Even restarting Emacs (which to me
>> is a drastic operation) won't opt out if there are still buffers in
>> c-ts-mode in the desktop.
> Sounds like a bug, indeed.
> The `add-to-list` should not happen just because `c-ts-mode` is loaded.
> I think it should only happen if `c-ts-mode` is called explicitly
> (i.e. interactively).

That would make this disruptive behavior even less useful, though, 
because one of the suggested ways to use it was to add (require 
'c-ts-mode) to the user's init script -- to have the auto-mode-alist 
settings applied.

Re: Make all tree-sitter modes optional

[Stefan Monnier]

>>> out again is unreasonably difficult.  Even restarting Emacs (which to me
>>> is a drastic operation) won't opt out if there are still buffers in
>>> c-ts-mode in the desktop.
>> Sounds like a bug, indeed.
>> The `add-to-list` should not happen just because `c-ts-mode` is loaded.
>> I think it should only happen if `c-ts-mode` is called explicitly
>> (i.e. interactively).
>
> That would make this disruptive behavior even less useful, though, because
> one of the suggested ways to use it was to add (require 'c-ts-mode) to the
> user's init script -- to have the auto-mode-alist settings applied.

We should never recommend (require <foo>) as a way to change Emacs's
behavior, since it directly conflicts with our convention that loading
a file should not significantly alter Emacs's behavior.

OTOH we can expose&preload `c-ts--activate` and tell users to use

    (cs-ts-activate)

in their `.emacs` (or use another name for it, of course).


        Stefan

Re: Make all tree-sitter modes optional

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Alan Mackenzie <acm@muc.de>,  Eli Zaretskii <eliz@gnu.org>,
>   juri@linkov.net,  casouri@gmail.com,  larsi@gnus.org,  theo@thornhill.no,
>   jostein@secure.kjonigsen.net,  emacs-devel@gnu.org
> Date: Wed, 15 Feb 2023 14:26:25 -0500
> 
> > That would make this disruptive behavior even less useful, though, because
> > one of the suggested ways to use it was to add (require 'c-ts-mode) to the
> > user's init script -- to have the auto-mode-alist settings applied.
> 
> We should never recommend (require <foo>) as a way to change Emacs's
> behavior, since it directly conflicts with our convention that loading
> a file should not significantly alter Emacs's behavior.

We never do -- except in this case.  Practical considerations trump
elegance, at least sometimes.

Re: Make all tree-sitter modes optional

[Stefan Monnier]

>> We should never recommend (require <foo>) as a way to change Emacs's
>> behavior, since it directly conflicts with our convention that loading
>> a file should not significantly alter Emacs's behavior.
>
> We never do -- except in this case.  Practical considerations trump
> elegance, at least sometimes.

The convention is not one of elegance but one of pragmatic concerns,
because files can get loaded unexpectedly.

(c-ts-activate) is no harder to write than (require 'c-ts-mode), so
I really can't see the "practical consideration" that justifies
such a decision.


        Stefan

Re: Make all tree-sitter modes optional

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: dgutov@yandex.ru,  acm@muc.de,  juri@linkov.net,  casouri@gmail.com,
>   larsi@gnus.org,  theo@thornhill.no,  jostein@secure.kjonigsen.net,
>   emacs-devel@gnu.org
> Date: Wed, 15 Feb 2023 14:53:09 -0500
> 
> > We never do -- except in this case.  Practical considerations trump
> > elegance, at least sometimes.
> 
> The convention is not one of elegance but one of pragmatic concerns,
> because files can get loaded unexpectedly.
> 
> (c-ts-activate) is no harder to write than (require 'c-ts-mode)

This is not about harder at all.

> I really can't see the "practical consideration" that justifies
> such a decision.

Well, I do.  And I explained this several times already in the past,
so from my POV this issue is closed for Emacs 29.  After the release
we should revisit this and related stuff, based on what we hear from
the users.

Re: Make all tree-sitter modes optional

[Stefan Monnier]

>> I really can't see the "practical consideration" that justifies
>> such a decision.
> Well, I do.  And I explained this several times already in the past,

Could you point to those explanations because I can't remember seeing
them in the long discussion(s).

Overriding the user's `auto-mode-alist` setting when merely loading
`c-ts-mode.el` is pretty drastic in my view, so the argument in favor of
this should be very strong and clear.

E.g. just `C-h o c-ts-mode RET` or `C-h o c-ts TAB` is enough to load that file.

In which use-case is it more "practical" to do as we currently do than
to do it in a way like the one I proposed (or various other ways: as
Dmitry points out, there are many alternatives, and coding them is not
the problem).


        Stefan

Re: Make all tree-sitter modes optional

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: dgutov@yandex.ru,  acm@muc.de,  juri@linkov.net,  casouri@gmail.com,
>   larsi@gnus.org,  theo@thornhill.no,  jostein@secure.kjonigsen.net,
>   emacs-devel@gnu.org
> Date: Wed, 15 Feb 2023 16:04:23 -0500
> 
> >> I really can't see the "practical consideration" that justifies
> >> such a decision.
> > Well, I do.  And I explained this several times already in the past,
> 
> Could you point to those explanations because I can't remember seeing
> them in the long discussion(s).

Sorry, no can do.  I don't have the time required to look them up,
what with the search of the emacs-devel archives still badly broken.

> Overriding the user's `auto-mode-alist` setting when merely loading
> `c-ts-mode.el` is pretty drastic in my view, so the argument in favor of
> this should be very strong and clear.

The argument is simplicity for the users to try these modes and turn
them on in their customizations.

> E.g. just `C-h o c-ts-mode RET` or `C-h o c-ts TAB` is enough to load that file.

That's an existing problem that should somehow be fixed, if we
consider "C-h o" a popular command used by many users (I don't
consider it that, and fail to see why someone would even think about
doing that).

Re: Make all tree-sitter modes optional

[Basil L. Contovounesios]

Eli Zaretskii [2023-02-16 09:42 +0200] wrote:

>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Cc: dgutov@yandex.ru,  acm@muc.de,  juri@linkov.net,  casouri@gmail.com,
>>   larsi@gnus.org,  theo@thornhill.no,  jostein@secure.kjonigsen.net,
>>   emacs-devel@gnu.org
>> Date: Wed, 15 Feb 2023 16:04:23 -0500
>> 
>> E.g. just `C-h o c-ts-mode RET` or `C-h o c-ts TAB` is enough to load that file.
>
> That's an existing problem that should somehow be fixed, if we
> consider "C-h o" a popular command used by many users (I don't
> consider it that, and fail to see why someone would even think about
> doing that).

C-h o is just a convenient union of C-h v, C-h f, etc.

The same problem exists with C-h f:

0. emacs -Q
1. C-h f c-ts-mode RET
   [ or C-h f c-ts TAB C-g ]
2. M-: (car auto-mode-alist) RET
   => ("\\.h\\'" . c-or-c++-ts-mode)

I understand that this situation compromises between various drawbacks
that I do not fully grasp, and I know how I'd work around this issue if
it bothered me, but I still find it unfortunate that an opt-in feature
is only sometimes opt-in for the average user.

Thanks,

-- 
Basil

Re: Make all tree-sitter modes optional

[Eli Zaretskii]

> From: "Basil L. Contovounesios" <contovob@tcd.ie>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,  dgutov@yandex.ru,
>  acm@muc.de,  juri@linkov.net,  casouri@gmail.com,  larsi@gnus.org,
>  theo@thornhill.no,  jostein@secure.kjonigsen.net,  emacs-devel@gnu.org
> Date: Thu, 16 Feb 2023 09:49:15 +0000
> 
> 0. emacs -Q
> 1. C-h f c-ts-mode RET
>    [ or C-h f c-ts TAB C-g ]
> 2. M-: (car auto-mode-alist) RET
>    => ("\\.h\\'" . c-or-c++-ts-mode)
> 
> I understand that this situation compromises between various drawbacks
> that I do not fully grasp, and I know how I'd work around this issue if
> it bothered me, but I still find it unfortunate that an opt-in feature
> is only sometimes opt-in for the average user.

I agree, but I had to choose the less evil alternative, out of those
that were at all practical given the schedule.

Re: Make all tree-sitter modes optional

[Stefan Monnier]

>> Overriding the user's `auto-mode-alist` setting when merely loading
>> `c-ts-mode.el` is pretty drastic in my view, so the argument in favor of
>> this should be very strong and clear.
>
> The argument is simplicity for the users to try these modes and turn
> them on in their customizations.

But with my patch, trying the modes is exactly the same (just `M-x
c-ts-mode`) and turning them on in their customization is no harder
(since `(c-ts-activate)` is no harder to type than `(require
'c-ts-mode)`; we could also make it a global minor mode so it can be
done via Custom if it's considered important).

I spent a lot of time educating ELisp package maintainers about the need
to make sure that merely loading a file doesn't change Emacs's behavior,
and that `require` should basically never be needed in `.emacs` file.

There's *always* a good way to solve the problem without changing
Emacs's behavior just by loading a file (with the sole exception of the
`.emacs` file itself, for which changing Emacs's behavior is the sole
purpose).

>> E.g. just `C-h o c-ts-mode RET` or `C-h o c-ts TAB` is enough to load that file.
>
> That's an existing problem that should somehow be fixed, if we
> consider "C-h o" a popular command used by many users (I don't
> consider it that, and fail to see why someone would even think about
> doing that).

Replace `C-h o` with `C-h f` and the same holds.

[ I only ever use `C-h o` nowadays.  I too often used `C-h f` to look
  for a var or `C-h v` to look for a function.  :-) ]


        Stefan

Re: Make all tree-sitter modes optional

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: dgutov@yandex.ru,  acm@muc.de,  juri@linkov.net,  casouri@gmail.com,
>  larsi@gnus.org,  theo@thornhill.no,  jostein@secure.kjonigsen.net,
>  emacs-devel@gnu.org
> Date: Thu, 16 Feb 2023 09:41:20 -0500
> 
> But with my patch, trying the modes is exactly the same (just `M-x
> c-ts-mode`) and turning them on in their customization is no harder
> (since `(c-ts-activate)` is no harder to type than `(require
> 'c-ts-mode)`

Sorry, I don't buy this argument.

> we could also make it a global minor mode so it can be
> done via Custom if it's considered important).

This was considered already, but had its own issues.  And we have ran
out of time needed to look for better solutions.  (I personally don't
believe there are any that weren't already proposed.)

> I spent a lot of time educating ELisp package maintainers about the need
> to make sure that merely loading a file doesn't change Emacs's behavior,
> and that `require` should basically never be needed in `.emacs` file.

We'll have to make this one exception to the rule.  The situation
itself is exceptional and probably won't happen again soon, if ever.

> Replace `C-h o` with `C-h f` and the same holds.

So be it.  (And restarting Emacs solves that as well.)

Re: Make all tree-sitter modes optional

[Stefan Monnier]

>> But with my patch, trying the modes is exactly the same (just `M-x
>> c-ts-mode`) and turning them on in their customization is no harder
>> (since `(c-ts-activate)` is no harder to type than `(require
>> 'c-ts-mode)`
> Sorry, I don't buy this argument.

Care to expand a bit more.

>> we could also make it a global minor mode so it can be
>> done via Custom if it's considered important).
> This was considered already, but had its own issues.

Can you mention at least one?

> And we have ran out of time needed to look for better solutions.  (I
> personally don't believe there are any that weren't already proposed.)

Giving at least one downside of the patch I propose compared to the
current code would help me understand your position.

> We'll have to make this one exception to the rule.  The situation
> itself is exceptional and probably won't happen again soon, if ever.

I understand there's time pressure.  But I think this would argue
towards making the code more conservative (e.g. not change the defaults
at all, not even after enabling `c-ts-mode`) since that's much less
likely to bring problems now and in the future.

E.g. if users do as (require 'c-ts-mode) as you suggest, they'll bump
into a regression when they move on to Emacs-30 where we will have
hopefully fixed this misfeature.

I think if we want a quick&dirty short-term way to encourage the use of
tree-sitter by enthusiastic users, we should provide a "one stop shop"
function which redirects all applicable major modes to their tree-sitter
variant.  It's easy to write, it's non-intrusive, and future-proof (it
should be easy to preserve that function's approximate behavior in the
future, regardless of how tree-sitter's integration evolves).

The current code will bring bad surprises to some of our users when they
try Emacs-29, and it will bring further bad surprises later when we move
to a different solution.

The fix is trivial enough that the urgency doesn't pose any problem.


        Stefan

Re: Make all tree-sitter modes optional

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: dgutov@yandex.ru,  acm@muc.de,  juri@linkov.net,  casouri@gmail.com,
>   larsi@gnus.org,  theo@thornhill.no,  jostein@secure.kjonigsen.net,
>   emacs-devel@gnu.org
> Date: Thu, 16 Feb 2023 11:45:30 -0500
> 
> >> But with my patch, trying the modes is exactly the same (just `M-x
> >> c-ts-mode`) and turning them on in their customization is no harder
> >> (since `(c-ts-activate)` is no harder to type than `(require
> >> 'c-ts-mode)`
> > Sorry, I don't buy this argument.
> 
> Care to expand a bit more.

I don't agree that calling (c-ts-activate) in an init file is no
harder than loading a package.  For you and me, maybe, but not for
users.

> >> we could also make it a global minor mode so it can be
> >> done via Custom if it's considered important).
> > This was considered already, but had its own issues.
> 
> Can you mention at least one?

Sorry, I remember only the conclusion.  You will have to look up the
discussions.  (And you were here when they happened.)

> > We'll have to make this one exception to the rule.  The situation
> > itself is exceptional and probably won't happen again soon, if ever.
> 
> I understand there's time pressure.  But I think this would argue
> towards making the code more conservative (e.g. not change the defaults
> at all, not even after enabling `c-ts-mode`) since that's much less
> likely to bring problems now and in the future.

That'd make it significantly harder for users to try these modes, so I
cannot agree to that.

> E.g. if users do as (require 'c-ts-mode) as you suggest, they'll bump
> into a regression when they move on to Emacs-30 where we will have
> hopefully fixed this misfeature.

If Emacs 30 will require different ways of activating these modes, we
will document that in NEWS.  Right now, the chances for such a change
are very high anyhow, so I already took this into account.

> I think if we want a quick&dirty short-term way to encourage the use of
> tree-sitter by enthusiastic users, we should provide a "one stop shop"
> function which redirects all applicable major modes to their tree-sitter
> variant.

That, too, was suggested a couple of months ago.  It has a
disadvantage of blindly assuming that a given user will either want
all of the TS modes or none of them.  It also assumes that a given
user has all of the grammar libraries installed.  Both assumptions are
not necessarily true, and so I decided that we must allow the users to
make separate and independent decisions for each such case.

> The current code will bring bad surprises to some of our users when they
> try Emacs-29, and it will bring further bad surprises later when we move
> to a different solution.

I understand and agree it could happen, but again: I see no better
way.  Every alternative that was proposed, including those proposed
now, had its own disadvantages, and I think what we have now is the
least of all evils.

> The fix is trivial enough that the urgency doesn't pose any problem.

The fix is not trivial at all.  If it were, we would have found it
already, as we are debating this for the past two or three months, and
it's not like we lack people here who can come up with bright ideas.

Re: Make all tree-sitter modes optional

[Dmitry Gutov]

On 16/02/2023 19:05, Eli Zaretskii wrote:
>>>> But with my patch, trying the modes is exactly the same (just `M-x
>>>> c-ts-mode`) and turning them on in their customization is no harder
>>>> (since `(c-ts-activate)` is no harder to type than `(require
>>>> 'c-ts-mode)`
>>> Sorry, I don't buy this argument.
>> Care to expand a bit more.
> I don't agree that calling (c-ts-activate) in an init file is no
> harder than loading a package.  For you and me, maybe, but not for
> users.

Loading a package is easy. But what happens after the user decides they 
like the new major mode and want to use it in every session? They will 
need to add something to their init script. And at that point,

(require 'c-ts-mode)

is not particularly easier for a newbie than

(treesit-setup-mode 'c-ts-mode)

Or (treesit-setup-mode 'c), or (c-ts-activate), or whatever.

>>>> we could also make it a global minor mode so it can be
>>>> done via Custom if it's considered important).
>>> This was considered already, but had its own issues.
>> Can you mention at least one?
> Sorry, I remember only the conclusion.  You will have to look up the
> discussions.  (And you were here when they happened.)

It has been my impression that you made up your mind pretty early on, so 
I didn't even try to get into the discussions for the technical 
solutions, but (*)

>>> We'll have to make this one exception to the rule.  The situation
>>> itself is exceptional and probably won't happen again soon, if ever.
>> I understand there's time pressure.  But I think this would argue
>> towards making the code more conservative (e.g. not change the defaults
>> at all, not even after enabling `c-ts-mode`) since that's much less
>> likely to bring problems now and in the future.
> That'd make it significantly harder for users to try these modes, so I
> cannot agree to that.

To try a major mode like that, the user can 'M-x c-ts-mode' either way, 
no matter the solution we choose.

>> I think if we want a quick&dirty short-term way to encourage the use of
>> tree-sitter by enthusiastic users, we should provide a "one stop shop"
>> function which redirects all applicable major modes to their tree-sitter
>> variant.
> That, too, was suggested a couple of months ago.  It has a
> disadvantage of blindly assuming that a given user will either want
> all of the TS modes or none of them.  It also assumes that a given
> user has all of the grammar libraries installed.  Both assumptions are
> not necessarily true, and so I decided that we must allow the users to
> make separate and independent decisions for each such case.

(*)

If you prefer the users enable the modes one-by-one, we can provide a 
helper to do just that. Like in the example above, it could be called 
'treesit-setup-mode' and could accept as an argument either the major 
mode name, or a language, and set up the necessary auto-mode-alist or 
major-mode-alist-alist entries.

Re: Make all tree-sitter modes optional

[Stefan Monnier]

>> > Sorry, I don't buy this argument.
>> Care to expand a bit more.
> I don't agree that calling (c-ts-activate) in an init file is no
> harder than loading a package.  For you and me, maybe, but not for
> users.

AFAIK users do it by searching for advice (in our docs when we're
lucky, or on the web more often) and copying the code snippet they find.
The specific content of the snippet doesn't matter very much in this regard.

>> > We'll have to make this one exception to the rule.  The situation
>> > itself is exceptional and probably won't happen again soon, if ever.
>> I understand there's time pressure.  But I think this would argue
>> towards making the code more conservative (e.g. not change the defaults
>> at all, not even after enabling `c-ts-mode`) since that's much less
>> likely to bring problems now and in the future.
> That'd make it significantly harder for users to try these modes, so I
> cannot agree to that.

To temporarily try the modes, the easiest is `M-x <THEMODE>` in all cases.
To try it more lastingly, "look for the snippet, copy it to `.emacs`" is
also the same in all cases.

>> I think if we want a quick&dirty short-term way to encourage the use of
>> tree-sitter by enthusiastic users, we should provide a "one stop shop"
>> function which redirects all applicable major modes to their tree-sitter
>> variant.
> That, too, was suggested a couple of months ago.  It has a
> disadvantage of blindly assuming that a given user will either want
> all of the TS modes or none of them.  It also assumes that a given
> user has all of the grammar libraries installed.  Both assumptions are
> not necessarily true, and so I decided that we must allow the users to
> make separate and independent decisions for each such case.

The point is that it's super easy to write those helper functions.
We can write one per mode, one per group of modes and an
overarching one in less time than this discussion.  And none of those
impose any surprising behavior that breaks the conventions we preach
(and on which other code relies).

IOW, what I'm proposing is a safe and simple solution.  It's probably
not the best solution in the long term either, but it obey the
Hypocratic oath which the current code doesn't, and it will to less
suffering of the users in the long run as well because we much more
easily keep supporting it in the future.

>> The current code will bring bad surprises to some of our users when they
>> try Emacs-29, and it will bring further bad surprises later when we move
>> to a different solution.
> I understand and agree it could happen,

It's not "could" it's "will".

> but again: I see no better way.  Every alternative that was proposed,
> including those proposed now, had its own disadvantages, and I think
> what we have now is the least of all evils.

The current solution has some evil because it breaks the "no effect on
load" convention (that's the reason I keep opposing it).
I don't see what's evil about the solution I advocate.
So I don't see how the current solution is "the least of all evils".
I'm not claiming my solution is better overall, but I do think it is
"the least of all evils".

>> The fix is trivial enough that the urgency doesn't pose any problem.
> The fix is not trivial at all.

What's not trivial about my solution?

> If it were, we would have found it already,

We have :-)

> as we are debating this for the past two or three months, and
> it's not like we lack people here who can come up with bright ideas.

That debate was focused a lot more on whether modes should be merged or
not and how things like that.  This here debate is strictly about
changing Emacs configuration when we load a file.


        Stefan

Re: Make all tree-sitter modes optional

[tomas]

[-- Attachment #1: Type: text/plain, Size: 277 bytes --]

On Wed, Feb 15, 2023 at 10:06:35PM +0200, Eli Zaretskii wrote:

[...]

> Well, I do.  And I explained this several times already in the past,

I don't understand that. To me it feels like pushing people to bump
into ts-modes whether they want or not.

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Make all tree-sitter modes optional

[Eli Zaretskii]

> Date: Thu, 16 Feb 2023 06:45:13 +0100
> From: <tomas@tuxteam.de>
> 
> On Wed, Feb 15, 2023 at 10:06:35PM +0200, Eli Zaretskii wrote:
> 
> > Well, I do.  And I explained this several times already in the past,
> 
> I don't understand that. To me it feels like pushing people to bump
> into ts-modes whether they want or not.

These modes are completely optional, turned off by default.  Users
need to turn them on, in one of the described ways, for them to take
any effect.  None of the described ways of turning on the modes
happens automatically.  I'm bewildered how this can be regarded as
"pushing people to bump into" these modes.  Are you sure we are
talking about the same editor?

Re: Make all tree-sitter modes optional

[Alan Mackenzie]

Hello, Eli.

On Thu, Feb 16, 2023 at 10:26:00 +0200, Eli Zaretskii wrote:
> > Date: Thu, 16 Feb 2023 06:45:13 +0100
> > From: <tomas@tuxteam.de>

> > On Wed, Feb 15, 2023 at 10:06:35PM +0200, Eli Zaretskii wrote:

> > > Well, I do.  And I explained this several times already in the past,

> > I don't understand that. To me it feels like pushing people to bump
> > into ts-modes whether they want or not.

> These modes are completely optional, turned off by default.  Users
> need to turn them on, in one of the described ways, for them to take
> any effect.  None of the described ways of turning on the modes
> happens automatically.  I'm bewildered how this can be regarded as
> "pushing people to bump into" these modes.  Are you sure we are
> talking about the same editor?

I'm a little surprised that you don't appreciate how other people (such
as me) work, and that you seem to regard restarting Emacs as a perfectly
acceptable way of reversing M-x c-ts-mode.

(Some) users will not be satisfied with a single switch to c-ts-mode,
they will want to move backwards and forwards to and from CC Mode IN THE
SAME EMACS SESSION, and will resent continually having to restart Emacs
to do so.

This "moving backwards and forward" is asymmetric.  It is far, far
easier to switch into c-ts-mode than switch out of it, and I feel this
is not right.  It is why I proposed c-make-ts-undefault-mode.

Also, restarting Emacs will NOT restore auto-mode-alist, unless the user
doesn't use desktop, or somehow knows (how?) he must clear his desktop
of c-ts-mode buffers before restarting.  I foresee much wasted time and
frustration resulting from this.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Make all tree-sitter modes optional

[Po Lu]

Alan Mackenzie <acm@muc.de> writes:

> Hello, Eli.
>
> On Thu, Feb 16, 2023 at 10:26:00 +0200, Eli Zaretskii wrote:
>> > Date: Thu, 16 Feb 2023 06:45:13 +0100
>> > From: <tomas@tuxteam.de>
>
>> > On Wed, Feb 15, 2023 at 10:06:35PM +0200, Eli Zaretskii wrote:
>
>> > > Well, I do.  And I explained this several times already in the past,
>
>> > I don't understand that. To me it feels like pushing people to bump
>> > into ts-modes whether they want or not.
>
>> These modes are completely optional, turned off by default.  Users
>> need to turn them on, in one of the described ways, for them to take
>> any effect.  None of the described ways of turning on the modes
>> happens automatically.  I'm bewildered how this can be regarded as
>> "pushing people to bump into" these modes.  Are you sure we are
>> talking about the same editor?
>
> I'm a little surprised that you don't appreciate how other people (such
> as me) work, and that you seem to regard restarting Emacs as a perfectly
> acceptable way of reversing M-x c-ts-mode.
>
> (Some) users will not be satisfied with a single switch to c-ts-mode,
> they will want to move backwards and forwards to and from CC Mode IN THE
> SAME EMACS SESSION, and will resent continually having to restart Emacs
> to do so.
>
> This "moving backwards and forward" is asymmetric.  It is far, far
> easier to switch into c-ts-mode than switch out of it, and I feel this
> is not right.  It is why I proposed c-make-ts-undefault-mode.
>
> Also, restarting Emacs will NOT restore auto-mode-alist, unless the user
> doesn't use desktop, or somehow knows (how?) he must clear his desktop
> of c-ts-mode buffers before restarting.  I foresee much wasted time and
> frustration resulting from this.

Is c-ts-mode made default immediately after the file is loaded, or after
c-ts-mode is first enabled?

The first is at least considered a Bad Thing, and that wisdom is written
down somewhere in the Lisp reference manual.  (elisp)Major Mode
Conventions perhaps?

Re: Make all tree-sitter modes optional

[Dmitry Gutov]

On 16/02/2023 14:38, Po Lu wrote:
> The first is at least considered a Bad Thing, and that wisdom is written
> down somewhere in the Lisp reference manual.  (elisp)Major Mode
> Conventions perhaps?

It's the first in the list of Emacs Lisp Coding Conventions, Stefan 
mentioned it in this thread.

doc/lispref/tips.texi:51

Re: Make all tree-sitter modes optional

[Dmitry Gutov]

On 15/02/2023 21:26, Stefan Monnier wrote:
> OTOH we can expose&preload `c-ts--activate` and tell users to use
> 
>      (cs-ts-activate)

A bunch of similar solutions have been proposed (global modes, etc; I 
can whip up one more in a few minutes as well), but they unfortunately 
failed to convince.

Re: Make all tree-sitter modes optional

[Eli Zaretskii]

> Date: Wed, 15 Feb 2023 22:24:49 +0200
> Cc: Alan Mackenzie <acm@muc.de>, Eli Zaretskii <eliz@gnu.org>,
>  juri@linkov.net, casouri@gmail.com, larsi@gnus.org, theo@thornhill.no,
>  jostein@secure.kjonigsen.net, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 15/02/2023 21:26, Stefan Monnier wrote:
> > OTOH we can expose&preload `c-ts--activate` and tell users to use
> > 
> >      (cs-ts-activate)
> 
> A bunch of similar solutions have been proposed (global modes, etc; I 
> can whip up one more in a few minutes as well), but they unfortunately 
> failed to convince.

Indeed, I'm still not convinced.  We should decide based on user
reactions and requests once Emacs 29 is out.

Re: Make all tree-sitter modes optional

[Theodor Thornhill]

On 16 February 2023 08:05:44 CET, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Wed, 15 Feb 2023 22:24:49 +0200
>> Cc: Alan Mackenzie <acm@muc.de>, Eli Zaretskii <eliz@gnu.org>,
>>  juri@linkov.net, casouri@gmail.com, larsi@gnus.org, theo@thornhill.no,
>>  jostein@secure.kjonigsen.net, emacs-devel@gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>> 
>> On 15/02/2023 21:26, Stefan Monnier wrote:
>> > OTOH we can expose&preload `c-ts--activate` and tell users to use
>> > 
>> >      (cs-ts-activate)
>> 
>> A bunch of similar solutions have been proposed (global modes, etc; I 
>> can whip up one more in a few minutes as well), but they unfortunately 
>> failed to convince.
>
>Indeed, I'm still not convinced.  We should decide based on user
>reactions and requests once Emacs 29 is out.

To me the last point here is the important one. We kinda "rushed" the modes in so that the treesit backend would have anything to show for in emacs-29. We've tried many times to devise a mechanism for it to be unintrusive, and definitely so for the cc mode equivalents. Yes they are far behind them in some respects, so there was never a point to make them the default for the foreseeable future. 

Also remember that whatever mechanism we make now won't be part of Emacs 30+, as there are clear shortcomings in all directions. Let's rather focus on how we can improve the situation for Emacs 30.

I'm thinking some devise like a "language layer", where major and minor modes are pluggable.

Let's say you want to program in JavaScript.

Then you can for example do something like:

(make-language-layer 'js-layer
  :major-mode 'js-ts-mode
  :lsp 'eglot
  :dagnostics 'flymake)


Then

(add-to-list auto-mode-alist ".js" 'js-layer)

And so forth. In this case a person can swap out flymake for flycheck, eglot for lsp-mode, js-ts-mode for js-mode etc. Then no implementation "owns" the language namespace, and we hopefully don't step on anyone's toes.

What do you think?

Theo

Re: Make all tree-sitter modes optional

[Eli Zaretskii]

> Date: Thu, 16 Feb 2023 08:53:56 +0100
> From: Theodor Thornhill <theo@thornhill.no>
> CC: monnier@iro.umontreal.ca, acm@muc.de, juri@linkov.net, casouri@gmail.com,
>  larsi@gnus.org, jostein@secure.kjonigsen.net, emacs-devel@gnu.org
> 
> I'm thinking some devise like a "language layer", where major and minor modes are pluggable.
> 
> Let's say you want to program in JavaScript.
> 
> Then you can for example do something like:
> 
> (make-language-layer 'js-layer
>   :major-mode 'js-ts-mode
>   :lsp 'eglot
>   :dagnostics 'flymake)
> 
> 
> Then
> 
> (add-to-list auto-mode-alist ".js" 'js-layer)
> 
> And so forth. In this case a person can swap out flymake for flycheck, eglot for lsp-mode, js-ts-mode for js-mode etc. Then no implementation "owns" the language namespace, and we hopefully don't step on anyone's toes.
> 
> What do you think?

I don't know yet.  We should definitely think about this more.  Emacs
never had more than a single major mode per programming language, so
for us "language" and "major mode" were always synonyms.  (Perl mode
and CPerl mode is the only exception I know of, and it did cause us
grief.)

So yes, this is a new situation that could call for some new concepts
in Emacs.

Re: Make all tree-sitter modes optional

[Theodor Thornhill]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Thu, 16 Feb 2023 08:53:56 +0100
>> From: Theodor Thornhill <theo@thornhill.no>
>> CC: monnier@iro.umontreal.ca, acm@muc.de, juri@linkov.net, casouri@gmail.com,
>>  larsi@gnus.org, jostein@secure.kjonigsen.net, emacs-devel@gnu.org
>> 
>> I'm thinking some devise like a "language layer", where major and minor modes are pluggable.
>> 
>> Let's say you want to program in JavaScript.
>> 
>> Then you can for example do something like:
>> 
>> (make-language-layer 'js-layer
>>   :major-mode 'js-ts-mode
>>   :lsp 'eglot
>>   :dagnostics 'flymake)
>> 
>> 
>> Then
>> 
>> (add-to-list auto-mode-alist ".js" 'js-layer)
>> 
>> And so forth. In this case a person can swap out flymake for flycheck, eglot for lsp-mode, js-ts-mode for js-mode etc. Then no implementation "owns" the language namespace, and we hopefully don't step on anyone's toes.
>> 
>> What do you think?
>
> I don't know yet.  We should definitely think about this more.  Emacs
> never had more than a single major mode per programming language, so
> for us "language" and "major mode" were always synonyms.  (Perl mode
> and CPerl mode is the only exception I know of, and it did cause us
> grief.)
>
> So yes, this is a new situation that could call for some new concepts
> in Emacs.

Yes, absolutely.  I just wanted to put out there a concrete thought that
hopefully moves the discussion forward.

Theo

Re: Make all tree-sitter modes optional

[Dmitry Gutov]

On 16/02/2023 10:34, Eli Zaretskii wrote:
> I don't know yet.  We should definitely think about this more.  Emacs
> never had more than a single major mode per programming language, so
> for us "language" and "major mode" were always synonyms.  (Perl mode
> and CPerl mode is the only exception I know of, and it did cause us
> grief.)
> 
> So yes, this is a new situation that could call for some new concepts
> in Emacs.

The relevant helps could make use of major-mode-remap-alist somehow. 
According to NEWS, it was introduced more or less for this purpose.

Re: Make all tree-sitter modes optional

[Dmitry Gutov]

On 16/02/2023 09:53, Theodor Thornhill wrote:
> To me the last point here is the important one. We kinda "rushed" the modes in so that the treesit backend would have anything to show for in emacs-29. We've tried many times to devise a mechanism for it to be unintrusive, and definitely so for the cc mode equivalents. Yes they are far behind them in some respects, so there was never a point to make them the default for the foreseeable future.

Nobody is arguing about whether the modes should be default at the moment.

> (make-language-layer 'js-layer
>    :major-mode 'js-ts-mode
>    :lsp 'eglot
>    :dagnostics 'flymake)

If we do something like that, I'd rather we try for a scheme where we 
don't need to enumerate the "swappable" features in advance -- aside 
from the major mode, of course. But Eglot and friends plug into a 
language through indirection.

Other than that, the user still needs to 'M-x eglot' or 'M-x lsp', or 
enable global-flycheck-mode, so the declarations like above seem redundant.

To clarify, the example above looks nice, but there are a lot more 
programming related minor modes than the LSP clients and Flymake/Flycheck.

Re: Make all tree-sitter modes optional

[Theodor Thornhill via Emacs development discussions.]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 16/02/2023 09:53, Theodor Thornhill wrote:
>> To me the last point here is the important one. We kinda "rushed" the modes in so that the treesit backend would have anything to show for in emacs-29. We've tried many times to devise a mechanism for it to be unintrusive, and definitely so for the cc mode equivalents. Yes they are far behind them in some respects, so there was never a point to make them the default for the foreseeable future.
>
> Nobody is arguing about whether the modes should be default at the moment.
>
>> (make-language-layer 'js-layer
>>    :major-mode 'js-ts-mode
>>    :lsp 'eglot
>>    :dagnostics 'flymake)
>
> If we do something like that, I'd rather we try for a scheme where we 
> don't need to enumerate the "swappable" features in advance -- aside 
> from the major mode, of course. But Eglot and friends plug into a 
> language through indirection.
>
> Other than that, the user still needs to 'M-x eglot' or 'M-x lsp', or 
> enable global-flycheck-mode, so the declarations like above seem redundant.
>
> To clarify, the example above looks nice, but there are a lot more 
> programming related minor modes than the LSP clients and Flymake/Flycheck.

Yeah, absolutely.  This was just an example written on my phone in a
hurry.  My point is simply that from a users perspective you'd want the
whole "ide" experience many times, but not necessarily always, and some
construct like this would allow for that.  In addition, a user opening a
project in a language never used before may be pleasantly surprised to
see diagnostics, autocomplete etc without setup, at least after an
optional "enable language layer for Rust? [yn]" or something like that.

and sure, there are many things that should be added, :completions,
:project etc, where settings could be company, corfu, project could be
project.el or projectile and so forth.  And obviously some :finally that
runs an arbitrary form of sorts. Maybe?

Theo

Re: Make all tree-sitter modes optional

[Dmitry Gutov]

On 16/02/2023 16:48, Theodor Thornhill via Emacs development 
discussions. wrote:
> and sure, there are many things that should be added, :completions,
> :project etc, where settings could be company, corfu, project could be
> project.el or projectile and so forth.  And obviously some :finally that
> runs an arbitrary form of sorts. Maybe?

My point was, we'd rather not have to enumerate them all (or any) in 
advance.

I get your point about offering an IDE experience OOTB, though.

Re: Make all tree-sitter modes optional

[Theodor Thornhill]

On 16 February 2023 15:56:23 CET, Dmitry Gutov <dgutov@yandex.ru> wrote:
>On 16/02/2023 16:48, Theodor Thornhill via Emacs development discussions. wrote:
>> and sure, there are many things that should be added, :completions,
>> :project etc, where settings could be company, corfu, project could be
>> project.el or projectile and so forth.  And obviously some :finally that
>> runs an arbitrary form of sorts. Maybe?
>
>My point was, we'd rather not have to enumerate them all (or any) in advance.
>
>I get your point about offering an IDE experience OOTB, though.

Then we agree, though I think some are clearer than others, so could be offered by default, not necessarily required

Theo

Re: Make all tree-sitter modes optional

[Lynn Winebarger]

On Wed, Feb 15, 2023 at 2:27 PM Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> We should never recommend (require <foo>) as a way to change Emacs's
> behavior, since it directly conflicts with our convention that loading
> a file should not significantly alter Emacs's behavior.

As someone who preloads the (elisp) world, I'd love to see this maxim
enshrined somewhere.

Re: Make all tree-sitter modes optional

[Stefan Monnier]

Lynn Winebarger [2023-02-15 18:48:50] wrote:

> On Wed, Feb 15, 2023 at 2:27 PM Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>> We should never recommend (require <foo>) as a way to change Emacs's
>> behavior, since it directly conflicts with our convention that loading
>> a file should not significantly alter Emacs's behavior.
>
> As someone who preloads the (elisp) world, I'd love to see this maxim
> enshrined somewhere.

It's in doc/lispref/tips.texi:

    @section Emacs Lisp Coding Conventions
    
    @cindex coding conventions in Emacs Lisp
      Here are conventions that you should follow when writing Emacs Lisp
    code intended for widespread use:
    
    @itemize @bullet
    @item
    Simply loading a package should not change Emacs's editing behavior.
    Include a command or commands to enable and disable the feature,
    or to invoke it.
    
    This convention is mandatory for any file that includes custom
    definitions.  If fixing such a file to follow this convention requires
    an incompatible change, go ahead and make the incompatible change;
    don't postpone it.


-- Stefan

Re: Make all tree-sitter modes optional

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  juri@linkov.net,  casouri@gmail.com,
>   larsi@gnus.org,  theo@thornhill.no,  jostein@secure.kjonigsen.net,
>   emacs-devel@gnu.org
> Date: Wed, 15 Feb 2023 13:34:02 -0500
> 
> > out again is unreasonably difficult.  Even restarting Emacs (which to me
> > is a drastic operation) won't opt out if there are still buffers in
> > c-ts-mode in the desktop.
> 
> Sounds like a bug, indeed.
> The `add-to-list` should not happen just because `c-ts-mode` is loaded.
> I think it should only happen if `c-ts-mode` is called explicitly
> (i.e. interactively).
> 
> Maybe something like the patch below.

Sorry, no.  The fact that add-to-list happens also when loading
c-ts-mode non-interactively is deliberate, so that users could
'require' or 'load' them in their init files.  Less clean, maybe, but
more practical and easier to use.

Re: Make all tree-sitter modes optional

[Stefan Monnier]

> Sorry, no.  The fact that add-to-list happens also when loading
> c-ts-mode non-interactively is deliberate, so that users could
> 'require' or 'load' them in their init files.  Less clean, maybe, but
> more practical and easier to use.

In my book, this is plain harmful.
Just say no.


        Stefan

Re: Make all tree-sitter modes optional

[Basil L. Contovounesios]

Eli Zaretskii [2023-02-15 21:07 +0200] wrote:

> Sorry, no.  The fact that add-to-list happens also when loading
> c-ts-mode non-interactively is deliberate, so that users could
> 'require' or 'load' them in their init files.  Less clean, maybe, but
> more practical and easier to use.

Is it a problem that this happens inconsistently, AFAICT, across
different *-ts-modes?

Added to auto-mode-alist at load time:
- c++-ts-mode
- c-or-c++-ts-mode
- c-ts-mode
- cmake-ts-mode
- dockerfile-ts-mode
- go-mod-ts-mode
- go-ts-mode
- html-ts-mode
- java-ts-mode
- json-ts-mode
- ruby-ts-mode
- rust-ts-mode
- toml-ts-mode
- tsx-ts-mode
- typescript-ts-mode
- yaml-ts-mode

Added to auto-mode-alist at call time:
- csharp-ts-mode
- css-ts-mode
- js-ts-mode
- python-ts-mode

Never added to auto-mode-alist:
- bash-ts-mode

Or is this intentional?

Thanks,

-- 
Basil

Re: Make all tree-sitter modes optional

[Eli Zaretskii]

> From: "Basil L. Contovounesios" <contovob@tcd.ie>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,  acm@muc.de,
>   juri@linkov.net,  casouri@gmail.com,  larsi@gnus.org,  theo@thornhill.no,
>   jostein@secure.kjonigsen.net,  emacs-devel@gnu.org
> Date: Wed, 15 Feb 2023 21:06:21 +0000
> 
> Or is this intentional?

It's intentional.  Some *-ts-modes are defined on the same file as
non-ts modes, and bash-ts-mode handles only Bash scripts.

Re: Make all tree-sitter modes optional

[Juri Linkov]

> How about commands c-make-ts-default-mode and c-make-ts-undefault-mode
> (and analogous functions for the other ts-modes)?  The first of these
> would push the new entries onto auto-mode-alist and the second would
> remove them again.  Or we could have a customisable option,
> c-default-c-mode which users could set when they are ready.  Either of
> these would allow the user to try out the new modes freely without being
> coerced against their will to use the new -ts- modes.

The proposed defcustom 'treesit-enable-modes' was supposed to have such
:set that would compare the old with new list of customized modes, then
for added modes like 'c-ts-mode' it could call (treesit-enable 'c-ts-mode),
and for removed modes (treesit-disable 'c-ts-mode).

Then for modes whose entries already exist in auto-mode-alist,
it will add/remove a mapping in major-mode-remap-alist.
For modes without an entry in auto-mode-alist, it will
update auto-mode-alist.