bug#59662: 29.0.50; [PATCH] Add treesit--indent-defun

[Theodor Thornhill via "Bug reports for GNU Emacs, the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>]

[-- Attachment #1: Type: text/plain, Size: 1124 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Thu, 08 Dec 2022 22:21:35 +0100
>> From: Theodor Thornhill <theo@thornhill.no>
>> CC: dgutov@yandex.ru, larsi@gnus.org, 59662@debbugs.gnu.org,
>>  monnier@iro.umontreal.ca, casouri@gmail.com
>> 
>> >> Ok, will do.  In preparation for that function, we need that variable on
>> >> the emacs-29 branch, right?
>> >
>> >Why?  Why cannot we introduce the variable only on master?
>> 
>> Mostly because I want to use it for other stuff, and for merge conflict reasons, but you decide!
>> 
>> I can revise the patch to go on master and introduce the function there.
>
> Yes, please do that on master.
>
>> By the way, what is the deciding factor for what goes where wrt treesit going forward?
>
> In general, anything that doesn't need to support the minimal treesit
> modes we have now on emacs-29 should go to master.  That includes any
> new features, even if they are supported by treesit.  IOW, mot
> everything related to tree-sitter should automatically go to the
> release branch.
>
> And when in doubt, ask, as always.

Ok!  Here's the patch.  What do you all think?

Theo


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Add-prog-fill-reindent-defun-bug-59664.patch --]
[-- Type: text/x-diff, Size: 15906 bytes --]

From 6ee9c0caa6677b1047f07235682f948e123f740e Mon Sep 17 00:00:00 2001
From: Theodor Thornhill <theo@thornhill.no>
Date: Thu, 8 Dec 2022 20:06:09 +0100
Subject: [PATCH] Add prog-fill-reindent-defun (bug#59664)

Introduce a new command that aims to reindent code in a defun, or fill
a paragraph of text.  The command uses treesit.el when available,
otherwise falls back to using syntax-ppss and regexps.  Treesit.el
needs a new variable that is intended to be set by the major modes so
that this and other future functions can know what kind of node we are
looking at.

* doc/emacs/programs.texi: Mention the new command.
* etc/NEWS: Mention the new command.
* lisp/progmodes/c-ts-mode.el (c-ts-mode--font-lock-settings): Add
raw_string_literal for C++.
(c-ts-mode, c++-ts-mode): Add regexp for the new variable.
* lisp/progmodes/csharp-mode.el (csharp-ts-mode): Add regexp for the
new variable.
* lisp/progmodes/java-ts-mode.el (java-ts-mode--indent-rules): Add
text_block as no-indent.
(java-ts-mode--font-lock-settings): Add text_block font-lock-rule.
(java-ts-mode): Add regexp for the new variable.
* lisp/progmodes/js.el (js-ts-mode): Add regexp for the new variable.
* lisp/progmodes/json-ts-mode.el (json-ts-mode--font-lock-settings):
Add comment font-lock-rule.
(json-ts-mode): Add regexp for the new variable, and use the new
comment feature.
* list/progmodes/prog-mode.el (prog-mode-map): Bind the new command by
default.
(prog-fill-reindent-defun): New command.
* lisp/progmodes/python.el (python-ts-mode): Add regexp for the new
variable.
* lisp/progmodes/sh-script.el (bash-ts-mode): Add regexp for the new
variable.
* lisp/progmodes/typescript-ts-mode.el (typescript-ts-base-mode): Add
regexp for the new variable.
(typescript-ts-mode--indent-rules): Add class_body.
* lisp/textmodes/css-mode.el (css-ts-mode): Add regexp for the new
variable.
* lisp/treesit.el (treesit-text-type-regexp): New variable.
---
 doc/emacs/programs.texi              |  6 +++++
 etc/NEWS                             |  7 ++++++
 lisp/progmodes/c-ts-mode.el          | 12 +++++++++-
 lisp/progmodes/csharp-mode.el        |  5 ++++
 lisp/progmodes/java-ts-mode.el       | 19 +++++++++++++--
 lisp/progmodes/js.el                 |  6 +++++
 lisp/progmodes/json-ts-mode.el       |  8 ++++++-
 lisp/progmodes/prog-mode.el          | 35 ++++++++++++++++++++++++++--
 lisp/progmodes/python.el             |  2 ++
 lisp/progmodes/sh-script.el          |  4 ++++
 lisp/progmodes/typescript-ts-mode.el |  6 +++++
 lisp/textmodes/css-mode.el           |  2 ++
 lisp/treesit.el                      |  9 +++++++
 13 files changed, 115 insertions(+), 6 deletions(-)

diff --git a/doc/emacs/programs.texi b/doc/emacs/programs.texi
index ba8475e86a..f9a15e819b 100644
--- a/doc/emacs/programs.texi
+++ b/doc/emacs/programs.texi
@@ -409,6 +409,12 @@ Multi-line Indent
 @table @kbd
 @item C-M-q
 Reindent all the lines within one parenthetical grouping.
+@item M-q
+Major modes that derive from @code{prog-mode} can reindent all the
+lines within a defun at point using the command @kbd{M-q}
+(@code{prog-mode-fill-reindent-defun}).  The bounds of of a defun is
+decided by @code{beginning-of-defun-function} and
+@code{end-of-defun-function}.
 @item C-u @key{TAB}
 Shift an entire parenthetical grouping rigidly sideways so that its
 first line is properly indented.
diff --git a/etc/NEWS b/etc/NEWS
index 61f813568f..db1ed2a474 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -61,6 +61,13 @@ using this new option.  (Or set 'display-buffer-alist' directly.)
 After manually editing 'eshell-aliases-file', you can use
 'M-x eshell-read-aliases-list' to load the edited aliases.
 
+** Prog Mode
++++
+*** New command 'prog-fill-reindent-defun'
+This command uses treesit and syntax-ppss to either fill a textual
+element in a defun, such as a doc-string or a comment.  If point is
+inside normal code, it (re)indents the surrounding defun.  Bound by
+default in 'prog-mode-map' to 'M-q'.
 \f
 * New Modes and Packages in Emacs 30.1
 
diff --git a/lisp/progmodes/c-ts-mode.el b/lisp/progmodes/c-ts-mode.el
index 08b03d5666..f0728f6fda 100644
--- a/lisp/progmodes/c-ts-mode.el
+++ b/lisp/progmodes/c-ts-mode.el
@@ -248,7 +248,9 @@ c-ts-mode--font-lock-settings
    :language mode
    :feature 'string
    `((string_literal) @font-lock-string-face
-     (system_lib_string) @font-lock-string-face)
+     (system_lib_string) @font-lock-string-face
+     ,@(when (eq mode 'cpp)
+         '((raw_string_literal) @font-lock-string-face)))
 
    :language mode
    :feature 'literal
@@ -567,6 +569,10 @@ c-ts-mode
   (setq-local comment-start-skip (rx (or (seq "/" (+ "/"))
                                          (seq "/" (+ "*")))
                                      (* (syntax whitespace))))
+
+  (setq-local treesit-text-type-regexp
+              (regexp-opt '("comment")))
+
   (setq-local comment-end-skip
               (rx (* (syntax whitespace))
                   (group (or (syntax comment-end)
@@ -603,6 +609,10 @@ c++-ts-mode
                   (group (or (syntax comment-end)
                              (seq (+ "*") "/")))))
 
+  (setq-local treesit-text-type-regexp
+              (regexp-opt '("comment"
+                            "raw_string_literal")))
+
   (treesit-parser-create 'cpp)
 
   (setq-local treesit-simple-indent-rules
diff --git a/lisp/progmodes/csharp-mode.el b/lisp/progmodes/csharp-mode.el
index 8ab5fbc91d..3faeb54fc5 100644
--- a/lisp/progmodes/csharp-mode.el
+++ b/lisp/progmodes/csharp-mode.el
@@ -911,6 +911,11 @@ csharp-ts-mode
                   (group (or (syntax comment-end)
                              (seq (+ "*") "/")))))
 
+  (setq-local treesit-text-type-regexp
+              (regexp-opt '("comment"
+                            "verbatim_string-literal"
+                            "interpolated_verbatim_string-text")))
+
   ;; Indent.
   (setq-local treesit-simple-indent-rules csharp-ts-mode--indent-rules)
 
diff --git a/lisp/progmodes/java-ts-mode.el b/lisp/progmodes/java-ts-mode.el
index 96e0d5244c..f29445195a 100644
--- a/lisp/progmodes/java-ts-mode.el
+++ b/lisp/progmodes/java-ts-mode.el
@@ -68,6 +68,7 @@ java-ts-mode--indent-rules
      ((node-is "]") parent-bol 0)
      ((and (parent-is "comment") comment-end) comment-start -1)
      ((parent-is "comment") comment-start-skip 0)
+     ((parent-is "text_block") no-indent)
      ((parent-is "class_body") parent-bol java-ts-mode-indent-offset)
      ((parent-is "interface_body") parent-bol java-ts-mode-indent-offset)
      ((parent-is "constructor_body") parent-bol java-ts-mode-indent-offset)
@@ -157,7 +158,8 @@ java-ts-mode--font-lock-settings
    :language 'java
    :override t
    :feature 'string
-   `((string_literal) @font-lock-string-face)
+   `((string_literal) @font-lock-string-face
+     (text_block) @font-lock-string-face)
    :language 'java
    :override t
    :feature 'literal
@@ -308,6 +310,11 @@ java-ts-mode
                   (group (or (syntax comment-end)
                              (seq (+ "*") "/")))))
 
+  (setq-local treesit-text-type-regexp
+              (regexp-opt '("line_comment"
+                            "block_comment"
+                            "text_block")))
+
   ;; Indent.
   (setq-local treesit-simple-indent-rules java-ts-mode--indent-rules)
 
@@ -316,7 +323,15 @@ java-ts-mode
               (append "{}():;," electric-indent-chars))
 
   ;; Navigation.
-  (setq-local treesit-defun-type-regexp "declaration")
+  (setq-local treesit-defun-type-regexp
+              (regexp-opt '("method_declaration"
+                            "class_declaration"
+                            "record_declaration"
+                            "interface_declaration"
+                            "enum_declaration"
+                            "import_declaration"
+                            "package_declaration"
+                            "module_declaration")))
 
   ;; Font-lock.
   (setq-local treesit-font-lock-settings java-ts-mode--font-lock-settings)
diff --git a/lisp/progmodes/js.el b/lisp/progmodes/js.el
index 45dfef372c..02cc558366 100644
--- a/lisp/progmodes/js.el
+++ b/lisp/progmodes/js.el
@@ -3857,6 +3857,11 @@ js-ts-mode
                     (group (or (syntax comment-end)
                                (seq (+ "*") "/")))))
     (setq-local comment-multi-line t)
+
+    (setq-local treesit-text-type-regexp
+                (regexp-opt '("comment"
+                              "template_string")))
+
     ;; Electric-indent.
     (setq-local electric-indent-chars
 	        (append "{}():;," electric-indent-chars)) ;FIXME: js2-mode adds "[]*".
@@ -3868,6 +3873,7 @@ js-ts-mode
     ;; Indent.
     (setq-local treesit-simple-indent-rules js--treesit-indent-rules)
     ;; Navigation.
+    (setq-local treesit-defun-prefer-top-level t)
     (setq-local treesit-defun-type-regexp
                 (rx (or "class_declaration"
                         "method_definition"
diff --git a/lisp/progmodes/json-ts-mode.el b/lisp/progmodes/json-ts-mode.el
index 0a0113d1d8..441e25c287 100644
--- a/lisp/progmodes/json-ts-mode.el
+++ b/lisp/progmodes/json-ts-mode.el
@@ -72,6 +72,9 @@ json-ts--indent-rules
 
 (defvar json-ts-mode--font-lock-settings
   (treesit-font-lock-rules
+   :language 'json
+   :feature 'comment
+   '((comment) @font-lock-comment-face)
    :language 'json
    :feature 'bracket
    '((["[" "]" "{" "}"]) @font-lock-bracket-face)
@@ -156,10 +159,13 @@ json-ts-mode
   (setq-local treesit-defun-type-regexp
               (rx (or "pair" "object")))
 
+  (setq-local treesit-text-type-regexp
+              (regexp-opt '("comment")))
+
   ;; Font-lock.
   (setq-local treesit-font-lock-settings json-ts-mode--font-lock-settings)
   (setq-local treesit-font-lock-feature-list
-              '((constant number pair string)
+              '((comment constant number pair string)
                 (escape-sequence)
                 (bracket delimiter error)))
 
diff --git a/lisp/progmodes/prog-mode.el b/lisp/progmodes/prog-mode.el
index 58cb48f182..c29efa3a21 100644
--- a/lisp/progmodes/prog-mode.el
+++ b/lisp/progmodes/prog-mode.el
@@ -30,7 +30,11 @@
 ;;; Code:
 
 (eval-when-compile (require 'cl-lib)
-                   (require 'subr-x))
+                   (require 'subr-x)
+                   (require 'treesit))
+
+(declare-function treesit-parser-list "treesit.c")
+(declare-function treesit-node-type "treesit.c")
 
 (defgroup prog-mode nil
   "Generic programming mode, from which others derive."
@@ -102,7 +106,8 @@ prog-context-menu
 
 (defvar-keymap prog-mode-map
   :doc "Keymap used for programming modes."
-  "C-M-q" #'prog-indent-sexp)
+  "C-M-q" #'prog-indent-sexp
+  "M-q" #'prog-fill-reindent-defun)
 
 (defvar prog-indentation-context nil
   "When non-nil, provides context for indenting embedded code chunks.
@@ -140,6 +145,32 @@ prog-indent-sexp
 	  (end (progn (forward-sexp 1) (point))))
       (indent-region start end nil))))
 
+(defun prog-fill-reindent-defun (&optional argument)
+  "Refill paragraph or reindent the definition that the point is on.
+
+If the point is in a string, or in a comment, or there is a
+comment on the current line, fill the paragraph that the point is
+in or is on the same line.
+
+Otherwise, reindent the definition around or below point."
+  (interactive "P")
+  (save-excursion
+    (let ((treesit-text-node
+           (and (treesit-parser-list)
+                (string-match-p
+                 treesit-text-type-regexp
+                 (treesit-node-type (treesit-node-at (point)))))))
+      (if (or treesit-text-node
+              (nth 8 (syntax-ppss))
+              (re-search-forward comment-start-skip (line-end-position) t))
+          (if (memq fill-paragraph-function '(t nil))
+              (lisp-fill-paragraph argument)
+            (funcall fill-paragraph-function argument))
+        (beginning-of-defun)
+        (let ((start (point)))
+          (end-of-defun)
+          (indent-region start (point) nil))))))
+
 (defun prog-first-column ()
   "Return the indentation column normally used for top-level constructs."
   (or (car prog-indentation-context) 0))
diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index ebee703499..f678e10fdf 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -6631,6 +6631,8 @@ python-ts-mode
                 #'python-imenu-treesit-create-index)
     (setq-local treesit-defun-type-regexp (rx (or "function" "class")
                                               "_definition"))
+    (setq-local treesit-text-type-regexp
+                (regexp-opt '("comment")))
     (treesit-major-mode-setup)
 
   (when python-indent-guess-indent-offset
diff --git a/lisp/progmodes/sh-script.el b/lisp/progmodes/sh-script.el
index e170d18afe..1605e40347 100644
--- a/lisp/progmodes/sh-script.el
+++ b/lisp/progmodes/sh-script.el
@@ -1619,6 +1619,10 @@ bash-ts-mode
                   ( bracket delimiter misc-punctuation operator)))
     (setq-local treesit-font-lock-settings
                 sh-mode--treesit-settings)
+    (setq-local treesit-text-type-regexp
+                (regexp-opt '("comment"
+                              "heredoc_start"
+                              "heredoc_body")))
     (treesit-major-mode-setup)))
 
 (advice-add 'bash-ts-mode :around #'sh--redirect-bash-ts-mode
diff --git a/lisp/progmodes/typescript-ts-mode.el b/lisp/progmodes/typescript-ts-mode.el
index 20916eaf37..74462bd232 100644
--- a/lisp/progmodes/typescript-ts-mode.el
+++ b/lisp/progmodes/typescript-ts-mode.el
@@ -74,6 +74,7 @@ typescript-ts-mode--indent-rules
      ((parent-is "member_expression") parent-bol typescript-ts-mode-indent-offset)
      ((parent-is "named_imports") parent-bol typescript-ts-mode-indent-offset)
      ((parent-is "statement_block") parent-bol typescript-ts-mode-indent-offset)
+     ((parent-is "class_body") parent-bol typescript-ts-mode-indent-offset)
      ((parent-is "type_arguments") parent-bol typescript-ts-mode-indent-offset)
      ((parent-is "variable_declarator") parent-bol typescript-ts-mode-indent-offset)
      ((parent-is "arguments") parent-bol typescript-ts-mode-indent-offset)
@@ -323,6 +324,11 @@ typescript-ts-base-mode
                   (group (or (syntax comment-end)
                              (seq (+ "*") "/")))))
 
+  (setq-local treesit-defun-prefer-top-level t)
+  (setq-local treesit-text-type-regexp
+              (regexp-opt '("comment"
+                            "template_string")))
+
   ;; Electric
   (setq-local electric-indent-chars
               (append "{}():;," electric-indent-chars))
diff --git a/lisp/textmodes/css-mode.el b/lisp/textmodes/css-mode.el
index 8a66986dc6..f5b7c3ce3d 100644
--- a/lisp/textmodes/css-mode.el
+++ b/lisp/textmodes/css-mode.el
@@ -1834,6 +1834,8 @@ css-ts-mode
     (treesit-parser-create 'css)
     (setq-local treesit-simple-indent-rules css--treesit-indent-rules)
     (setq-local treesit-defun-type-regexp "rule_set")
+    (setq-local treesit-text-type-regexp
+                (regexp-opt '("comment")))
     (setq-local treesit-font-lock-settings css--treesit-settings)
     (setq-local treesit-font-lock-feature-list
                 '((selector comment query keyword)
diff --git a/lisp/treesit.el b/lisp/treesit.el
index dbbf7ec18c..11fb699180 100644
--- a/lisp/treesit.el
+++ b/lisp/treesit.el
@@ -1626,6 +1626,15 @@ treesit-end-of-defun
     (when top
       (goto-char (treesit-node-end top)))))
 
+(defvar-local treesit-text-type-regexp nil
+  "A regexp that matches the node type of textual nodes.
+
+A textual node is a node that is not normal code, such as
+comments and multiline string literals.  For example,
+\"(line|block)_comment\" in the case of a comment, or
+\"text_block\" in the case of a string.  This is used by
+`prog-fill-reindent-defun' and friends.")
+
 ;;; Activating tree-sitter
 
 (defun treesit-ready-p (language &optional quiet)
-- 
2.34.1