bug#60623: 30.0.50; Add forward-sentence with tree sitter support

bug#60623: 30.0.50; Add forward-sentence with tree sitter support

[Theodor Thornhill via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[-- Attachment #1: Type: text/plain, Size: 448 bytes --]


Hi all!

This patch tweaks the forward-sentence function to be usable with
tree-sitter.

It follows the same style as the recent change in transpose-sexps, so I
hope it isn't too controversial.

What exact node types do you consider useful for sentence movement?

I added an example value in java-ts-mode and c-ts-mode, but that is
merely a suggestion.  Let's decide on some heuristic to decide the
proper nodes to use.

What do you think?

Theo


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Add-forward-sentence-with-tree-sitter-support.patch --]
[-- Type: text/x-patch, Size: 8243 bytes --]

From 32eb3c3401f2232c349843e21269ff88e982945e Mon Sep 17 00:00:00 2001
From: Theodor Thornhill <theo@thornhill.no>
Date: Sat, 7 Jan 2023 12:44:14 +0100
Subject: [PATCH] Add forward-sentence with tree sitter support

* etc/NEWS: Mention the new changes.
* lisp/progmodes/c-ts-mode.el (c-ts-base-mode): Set the variable.
* lisp/progmodes/java-ts-mode.el (java-ts-mode): Set the variable.
* lisp/textmodes/paragraphs.el (forward-sentence-function): Move old
implementation to a new defvar.
(forward-sentence): Use the variable in this function unconditionally.
* lisp/treesit.el (treesit-sentence-type-regexp): New defvar.
(treesit-forward-sentence): New defun.
(treesit-major-mode-setup): Conditionally set
forward-sentence-function.
---
 etc/NEWS                       | 12 ++++++
 lisp/progmodes/c-ts-mode.el    |  8 ++++
 lisp/progmodes/java-ts-mode.el |  6 +++
 lisp/textmodes/paragraphs.el   | 71 +++++++++++++++++++---------------
 lisp/treesit.el                | 18 +++++++++
 5 files changed, 83 insertions(+), 32 deletions(-)

diff --git a/etc/NEWS b/etc/NEWS
index a2924201267..5b4533b0522 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -57,6 +57,18 @@ treesit.el now unconditionally sets 'transpose-sexps-function' for all
 Tree-sitter modes.  This functionality utilizes the new
 'transpose-sexps-function'.
 
+** New defvar-local forward-sentence-function.
+The previous implementation of 'forward-sentence' is moved into this
+variable, which can be set to customize the sentece movement behavior.
+
+** New defvar-local 'treesit-sentence-type-regexp.
+Similarly to 'treesit-defun-type-regexp', this variable is used to
+navigate sentences in Tree-sitter enabled modes.
+
+** New function 'treesit-forward-sentence'.
+treesit.el now conditionally sets 'forward-sentence-function' for all
+Tree-sitter modes that sets 'treesit-sentence-type-regexp'.
+
 \f
 * Changes in Specialized Modes and Packages in Emacs 30.1
 ---
diff --git a/lisp/progmodes/c-ts-mode.el b/lisp/progmodes/c-ts-mode.el
index e76966e7660..3794286162e 100644
--- a/lisp/progmodes/c-ts-mode.el
+++ b/lisp/progmodes/c-ts-mode.el
@@ -735,6 +735,14 @@ c-ts-base-mode
   :syntax-table c-ts-mode--syntax-table
 
   ;; Navigation.
+  (setq-local treesit-sentence-type-regexp
+              (regexp-opt '("statement"
+                            "expression"
+                            "definition"
+                            "specifier"
+                            "declaration"
+                            "comment"
+                            "preproc")))
   (setq-local treesit-defun-type-regexp
               (cons (regexp-opt '("function_definition"
                                   "type_definition"
diff --git a/lisp/progmodes/java-ts-mode.el b/lisp/progmodes/java-ts-mode.el
index 87a4e2b90f8..0e3381e219a 100644
--- a/lisp/progmodes/java-ts-mode.el
+++ b/lisp/progmodes/java-ts-mode.el
@@ -296,6 +296,12 @@ java-ts-mode
               (append "{}():;," electric-indent-chars))
 
   ;; Navigation.
+  (setq-local treesit-sentence-type-regexp
+              (regexp-opt '("statement"
+                            "expression"
+                            "parameters"
+                            "list"
+                            "comment")))
   (setq-local treesit-defun-type-regexp
               (regexp-opt '("method_declaration"
                             "class_declaration"
diff --git a/lisp/textmodes/paragraphs.el b/lisp/textmodes/paragraphs.el
index 73abb155aaa..5bbb77c1cb6 100644
--- a/lisp/textmodes/paragraphs.el
+++ b/lisp/textmodes/paragraphs.el
@@ -441,44 +441,51 @@ end-of-paragraph-text
 	  (if (< (point) (point-max))
 	      (end-of-paragraph-text))))))
 
-(defun forward-sentence (&optional arg)
+(defvar forward-sentence-function
+  (lambda (&optional arg)
+    (let ((opoint (point))
+          (sentence-end (sentence-end)))
+      (while (< arg 0)
+        (let ((pos (point))
+	      par-beg par-text-beg)
+	  (save-excursion
+	    (start-of-paragraph-text)
+	    ;; Start of real text in the paragraph.
+	    ;; We move back to here if we don't see a sentence-end.
+	    (setq par-text-beg (point))
+	    ;; Start of the first line of the paragraph.
+	    ;; We use this as the search limit
+	    ;; to allow sentence-end to match if it is anchored at
+	    ;; BOL and the paragraph starts indented.
+	    (beginning-of-line)
+	    (setq par-beg (point)))
+	  (if (and (re-search-backward sentence-end par-beg t)
+		   (or (< (match-end 0) pos)
+		       (re-search-backward sentence-end par-beg t)))
+	      (goto-char (match-end 0))
+	    (goto-char par-text-beg)))
+        (setq arg (1+ arg)))
+      (while (> arg 0)
+        (let ((par-end (save-excursion (end-of-paragraph-text) (point))))
+	  (if (re-search-forward sentence-end par-end t)
+	      (skip-chars-backward " \t\n")
+	    (goto-char par-end)))
+        (setq arg (1- arg)))
+      (let ((npoint (constrain-to-field nil opoint t)))
+        (not (= npoint opoint)))))
   "Move forward to next end of sentence.  With argument, repeat.
 When ARG is negative, move backward repeatedly to start of sentence.
 
 The variable `sentence-end' is a regular expression that matches ends of
-sentences.  Also, every paragraph boundary terminates sentences as well."
+sentences.  Also, every paragraph boundary terminates sentences as well.")
+
+(defun forward-sentence (&optional arg)
+  "Move forward to next end of sentence.  With argument, repeat.
+When ARG is negative, move backward repeatedly to start of sentence.
+Delegates its work to `forward-sentence-function'."
   (interactive "^p")
   (or arg (setq arg 1))
-  (let ((opoint (point))
-        (sentence-end (sentence-end)))
-    (while (< arg 0)
-      (let ((pos (point))
-	    par-beg par-text-beg)
-	(save-excursion
-	  (start-of-paragraph-text)
-	  ;; Start of real text in the paragraph.
-	  ;; We move back to here if we don't see a sentence-end.
-	  (setq par-text-beg (point))
-	  ;; Start of the first line of the paragraph.
-	  ;; We use this as the search limit
-	  ;; to allow sentence-end to match if it is anchored at
-	  ;; BOL and the paragraph starts indented.
-	  (beginning-of-line)
-	  (setq par-beg (point)))
-	(if (and (re-search-backward sentence-end par-beg t)
-		 (or (< (match-end 0) pos)
-		     (re-search-backward sentence-end par-beg t)))
-	    (goto-char (match-end 0))
-	  (goto-char par-text-beg)))
-      (setq arg (1+ arg)))
-    (while (> arg 0)
-      (let ((par-end (save-excursion (end-of-paragraph-text) (point))))
-	(if (re-search-forward sentence-end par-end t)
-	    (skip-chars-backward " \t\n")
-	  (goto-char par-end)))
-      (setq arg (1- arg)))
-    (let ((npoint (constrain-to-field nil opoint t)))
-      (not (= npoint opoint)))))
+  (funcall forward-sentence-function arg))
 
 (defun count-sentences (start end)
   "Count sentences in current buffer from START to END."
diff --git a/lisp/treesit.el b/lisp/treesit.el
index a7f453a8899..ec4d711bba8 100644
--- a/lisp/treesit.el
+++ b/lisp/treesit.el
@@ -1783,6 +1783,22 @@ treesit-end-of-defun
     (when treesit-defun-skipper
       (funcall treesit-defun-skipper))))
 
+(defvar-local treesit-sentence-type-regexp ""
+  "A regexp that matches the node type of sentence nodes.
+
+A sentence node is a node that is bigger than a sexp, and
+delimits larger statements in the source code.  It is, however,
+smaller in scope than defuns.  This is used by
+`treesit-forward-sentence' and friends.")
+
+(defun treesit-forward-sentence (&optional arg)
+  "Tree-sitter `forward-sentence-function' function.
+ARG is the same as in `forward-sentence'.  Behavior of this
+function depends on `treesit-sentence-type-regexp'."
+  (funcall
+   (if (> arg 0) #'treesit-end-of-thing #'treesit-beginning-of-thing)
+   treesit-sentence-type-regexp (abs arg)))
+
 (defvar-local treesit-text-type-regexp "\\`comment\\'"
   "A regexp that matches the node type of textual nodes.
 
@@ -2256,6 +2272,8 @@ treesit-major-mode-setup
                 #'treesit-add-log-current-defun))
 
   (setq-local transpose-sexps-function #'treesit-transpose-sexps)
+  (when treesit-sentence-type-regexp
+    (setq-local forward-sentence-function #'treesit-forward-sentence))
 
   ;; Imenu.
   (when treesit-simple-imenu-settings
-- 
2.34.1

bug#60623: 30.0.50; Add forward-sentence with tree sitter support

[Daniel Martín via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Theodor Thornhill via "Bug reports for GNU Emacs, the Swiss army knife
of text editors" <bug-gnu-emacs@gnu.org> writes:

> Hi all!
>
> This patch tweaks the forward-sentence function to be usable with
> tree-sitter.
>
> It follows the same style as the recent change in transpose-sexps, so I
> hope it isn't too controversial.

Thanks.

>
> What exact node types do you consider useful for sentence movement?
>

I haven't thought much about your proposed nodes, I initially thought
that sentences in a programming language are just "statements".

As a suggestion, treesit-forward-sentence could navigate by textual
sentences when point is inside comments or strings.

> +** New defvar-local forward-sentence-function.
> +The previous implementation of 'forward-sentence' is moved into this
> +variable, which can be set to customize the sentece movement behavior.
                                               ^^^^^^^
                                               sentence

Also, this feature probably needs an update to the Info documentation to
mention that Tree-sitter can specialize sentence commands in programming
modes.

bug#60623: 30.0.50; Add forward-sentence with tree sitter support

[Theodor Thornhill via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[-- Attachment #1: Type: text/plain, Size: 2037 bytes --]

Daniel Martín <mardani29@yahoo.es> writes:

> Theodor Thornhill via "Bug reports for GNU Emacs, the Swiss army knife
> of text editors" <bug-gnu-emacs@gnu.org> writes:
>
>> Hi all!
>>
>> This patch tweaks the forward-sentence function to be usable with
>> tree-sitter.
>>
>> It follows the same style as the recent change in transpose-sexps, so I
>> hope it isn't too controversial.
>
> Thanks.
>
>>
>> What exact node types do you consider useful for sentence movement?
>>
>
> I haven't thought much about your proposed nodes, I initially thought
> that sentences in a programming language are just "statements".

They aren't really proper propsals.  Mostly some example values to show
that the code works.  The problem with just stating "statements" is that
the names are different across parsers. So in java one would call

```
void foo() {
  var foo = 5;  // <-- This thing
}

```

A "local_variable_declaration" or something like that.  But it would
make sense for M-e to move across that whole line.  So this is language
dependent, I believe.

>
> As a suggestion, treesit-forward-sentence could navigate by textual
> sentences when point is inside comments or strings.
>

Yeah, this is a good idea - added in following patch.

>> +** New defvar-local forward-sentence-function.
>> +The previous implementation of 'forward-sentence' is moved into this
>> +variable, which can be set to customize the sentece movement behavior.
>                                                ^^^^^^^
>                                                sentence
>

Thanks - fixed.

> Also, this feature probably needs an update to the Info documentation to
> mention that Tree-sitter can specialize sentence commands in programming
> modes.

Yes, likely.  I will add this a bit later, when we agree on its behavior
fully :)

@Eli, what doc changes do you see as needed here?

@Juri: I added a change with how I understood what you meant.  Is that
in your line of reasoning?


Theo


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Add-forward-sentence-with-tree-sitter-support-bug-60.patch --]
[-- Type: text/x-patch, Size: 6758 bytes --]

From 62d6f162ba7646783675dc87199ca31691164bf4 Mon Sep 17 00:00:00 2001
From: Theodor Thornhill <theo@thornhill.no>
Date: Sat, 7 Jan 2023 12:44:14 +0100
Subject: [PATCH] Add forward-sentence with tree sitter support (bug#60623)

* etc/NEWS: Mention the new changes.
* lisp/progmodes/c-ts-mode.el (c-ts-base-mode): Set the variable.
* lisp/progmodes/java-ts-mode.el (java-ts-mode): Set the variable.
* lisp/textmodes/paragraphs.el (forward-sentence-default-function):
Move old implementation to its own function.
(forward-sentence-function): New defvar defaulting to old behavior.
(forward-sentence): Use the variable in this function unconditionally.
* lisp/treesit.el (treesit-sentence-type-regexp): New defvar.
(treesit-forward-sentence): New defun.
(treesit-major-mode-setup): Conditionally set
forward-sentence-function.
---
 etc/NEWS                       | 13 +++++++++++++
 lisp/progmodes/c-ts-mode.el    |  8 ++++++++
 lisp/progmodes/java-ts-mode.el |  6 ++++++
 lisp/textmodes/paragraphs.el   | 16 +++++++++++++---
 lisp/treesit.el                | 28 ++++++++++++++++++++++++++++
 5 files changed, 68 insertions(+), 3 deletions(-)

diff --git a/etc/NEWS b/etc/NEWS
index 690e9c3faa9..98f5d71cc90 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -57,6 +57,19 @@ treesit.el now unconditionally sets 'transpose-sexps-function' for all
 Tree-sitter modes.  This functionality utilizes the new
 'transpose-sexps-function'.
 
+** New defvar-local forward-sentence-function.
+The previous implementation of 'forward-sentence' is moved into this
+variable, which can be set to customize the sentence movement
+behavior.
+
+** New defvar-local 'treesit-sentence-type-regexp.
+Similarly to 'treesit-defun-type-regexp', this variable is used to
+navigate sentences in Tree-sitter enabled modes.
+
+** New function 'treesit-forward-sentence'.
+treesit.el now conditionally sets 'forward-sentence-function' for all
+Tree-sitter modes that sets 'treesit-sentence-type-regexp'.
+
 \f
 * Changes in Specialized Modes and Packages in Emacs 30.1
 ---
diff --git a/lisp/progmodes/c-ts-mode.el b/lisp/progmodes/c-ts-mode.el
index a22f1f3c44f..dec866f762f 100644
--- a/lisp/progmodes/c-ts-mode.el
+++ b/lisp/progmodes/c-ts-mode.el
@@ -736,6 +736,14 @@ c-ts-base-mode
   :syntax-table c-ts-mode--syntax-table
 
   ;; Navigation.
+  (setq-local treesit-sentence-type-regexp
+              (regexp-opt '("statement"
+                            "expression"
+                            "definition"
+                            "specifier"
+                            "declaration"
+                            "comment"
+                            "preproc")))
   (setq-local treesit-defun-type-regexp
               (cons (regexp-opt '("function_definition"
                                   "type_definition"
diff --git a/lisp/progmodes/java-ts-mode.el b/lisp/progmodes/java-ts-mode.el
index 87a4e2b90f8..0e3381e219a 100644
--- a/lisp/progmodes/java-ts-mode.el
+++ b/lisp/progmodes/java-ts-mode.el
@@ -296,6 +296,12 @@ java-ts-mode
               (append "{}():;," electric-indent-chars))
 
   ;; Navigation.
+  (setq-local treesit-sentence-type-regexp
+              (regexp-opt '("statement"
+                            "expression"
+                            "parameters"
+                            "list"
+                            "comment")))
   (setq-local treesit-defun-type-regexp
               (regexp-opt '("method_declaration"
                             "class_declaration"
diff --git a/lisp/textmodes/paragraphs.el b/lisp/textmodes/paragraphs.el
index 73abb155aaa..4580be3450d 100644
--- a/lisp/textmodes/paragraphs.el
+++ b/lisp/textmodes/paragraphs.el
@@ -441,14 +441,12 @@ end-of-paragraph-text
 	  (if (< (point) (point-max))
 	      (end-of-paragraph-text))))))
 
-(defun forward-sentence (&optional arg)
+(defun forward-sentence-default-function (&optional arg)
   "Move forward to next end of sentence.  With argument, repeat.
 When ARG is negative, move backward repeatedly to start of sentence.
 
 The variable `sentence-end' is a regular expression that matches ends of
 sentences.  Also, every paragraph boundary terminates sentences as well."
-  (interactive "^p")
-  (or arg (setq arg 1))
   (let ((opoint (point))
         (sentence-end (sentence-end)))
     (while (< arg 0)
@@ -480,6 +478,18 @@ forward-sentence
     (let ((npoint (constrain-to-field nil opoint t)))
       (not (= npoint opoint)))))
 
+(defvar forward-sentence-function #'forward-sentence-default-function
+  "Function to be used to calculate sentence movements.
+See `forward-sentence-default-function' for behavior description.")
+
+(defun forward-sentence (&optional arg)
+  "Move forward to next end of sentence.  With argument, repeat.
+When ARG is negative, move backward repeatedly to start of sentence.
+Delegates its work to `forward-sentence-function'."
+  (interactive "^p")
+  (or arg (setq arg 1))
+  (funcall forward-sentence-function arg))
+
 (defun count-sentences (start end)
   "Count sentences in current buffer from START to END."
   (let ((sentences 0)
diff --git a/lisp/treesit.el b/lisp/treesit.el
index a7f453a8899..0681e758b37 100644
--- a/lisp/treesit.el
+++ b/lisp/treesit.el
@@ -1783,6 +1783,32 @@ treesit-end-of-defun
     (when treesit-defun-skipper
       (funcall treesit-defun-skipper))))
 
+(defvar-local treesit-sentence-type-regexp ""
+  "A regexp that matches the node type of sentence nodes.
+
+A sentence node is a node that is bigger than a sexp, and
+delimits larger statements in the source code.  It is, however,
+smaller in scope than defuns.  This is used by
+`treesit-forward-sentence' and friends.")
+
+(defun treesit-forward-sentence (&optional arg)
+  "Tree-sitter `forward-sentence-function' function.
+
+ARG is the same as in `forward-sentence-function'.
+
+If inside comment or other nodes described in
+`treesit-sentence-type-regexp', use
+`forward-sentence-default-function', else move across nodes as
+described by `treesit-sentence-type-regexp'."
+
+  (if (string-match-p
+       treesit-text-type-regexp
+       (treesit-node-type (treesit-node-at (point))))
+      (funcall #'forward-sentence-default-function arg)
+    (funcall
+     (if (> arg 0) #'treesit-end-of-thing #'treesit-beginning-of-thing)
+     treesit-sentence-type-regexp (abs arg))))
+
 (defvar-local treesit-text-type-regexp "\\`comment\\'"
   "A regexp that matches the node type of textual nodes.
 
@@ -2256,6 +2282,8 @@ treesit-major-mode-setup
                 #'treesit-add-log-current-defun))
 
   (setq-local transpose-sexps-function #'treesit-transpose-sexps)
+  (when treesit-sentence-type-regexp
+    (setq-local forward-sentence-function #'treesit-forward-sentence))
 
   ;; Imenu.
   (when treesit-simple-imenu-settings
-- 
2.34.1

bug#60623: 30.0.50; Add forward-sentence with tree sitter support

[Eli Zaretskii]

> From: Theodor Thornhill <theo@thornhill.no>
> Cc: 60623@debbugs.gnu.org, casouri@gmail.com, eliz@gnu.org,
>  monnier@iro.umontreal.ca,Juri Linkov <juri@linkov.net>
> Date: Sun, 08 Jan 2023 14:29:08 +0100
> 
> @Eli, what doc changes do you see as needed here?

More or less.  They need some polishing, like a few words about what
does "sentence" mean in the tree-sitter context.  But we can make
these changes after this is in the repository.

Thanks.

bug#60623: 30.0.50; Add forward-sentence with tree sitter support

[Theodor Thornhill via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[-- Attachment #1: Type: text/plain, Size: 791 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Theodor Thornhill <theo@thornhill.no>
>> Cc: 60623@debbugs.gnu.org, casouri@gmail.com, eliz@gnu.org,
>>  monnier@iro.umontreal.ca,Juri Linkov <juri@linkov.net>
>> Date: Sun, 08 Jan 2023 14:29:08 +0100
>> 
>> @Eli, what doc changes do you see as needed here?
>
> More or less.  They need some polishing, like a few words about what
> does "sentence" mean in the tree-sitter context.  But we can make
> these changes after this is in the repository.
>
> Thanks.


Ok, so in other words, this patch is good to go?

I omitted the additions to java-ts-mode and c-ts-mode.  I can make a
separate commit to add some values that makes sense for multiple modes
after?  Will the changes to the manual lie in "26.2 Sentences"? in the
Emacs manual?

Theo


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Add-forward-sentence-with-tree-sitter-support-bug-60.patch --]
[-- Type: text/x-patch, Size: 5180 bytes --]

From 558382b5c712431f0a19262c6e1fb67fdedd908b Mon Sep 17 00:00:00 2001
From: Theodor Thornhill <theo@thornhill.no>
Date: Sun, 8 Jan 2023 20:28:02 +0100
Subject: [PATCH] Add forward-sentence with tree sitter support (bug#60623)

* etc/NEWS: Mention the new changes.
* lisp/textmodes/paragraphs.el (forward-sentence-default-function):
Move old implementation to its own function.
(forward-sentence-function): New defvar defaulting to old behavior.
(forward-sentence): Use the variable in this function unconditionally.
* lisp/treesit.el (treesit-sentence-type-regexp): New defvar.
(treesit-forward-sentence): New defun.
(treesit-major-mode-setup): Conditionally set
forward-sentence-function.
---
 etc/NEWS                     | 16 ++++++++++++++++
 lisp/textmodes/paragraphs.el | 15 +++++++++++++--
 lisp/treesit.el              | 28 ++++++++++++++++++++++++++++
 3 files changed, 57 insertions(+), 2 deletions(-)

diff --git a/etc/NEWS b/etc/NEWS
index 690e9c3faa9..a0b2de056d8 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -57,6 +57,22 @@ treesit.el now unconditionally sets 'transpose-sexps-function' for all
 Tree-sitter modes.  This functionality utilizes the new
 'transpose-sexps-function'.
 
+** New defvar forward-sentence-function.
+Emacs now can set this variable to customize the behavior of the
+'forward-sentence' function.
+
+** New defun forward-sentence-default-function.
+The previous implementation of 'forward-sentence' is moved into its
+own function, to be bound by 'forward-sentence-function'.
+
+** New defvar-local 'treesit-sentence-type-regexp.
+Similarly to 'treesit-defun-type-regexp', this variable is used to
+navigate sentences in Tree-sitter enabled modes.
+
+** New function 'treesit-forward-sentence'.
+treesit.el now conditionally sets 'forward-sentence-function' for all
+Tree-sitter modes that sets 'treesit-sentence-type-regexp'.
+
 \f
 * Changes in Specialized Modes and Packages in Emacs 30.1
 ---
diff --git a/lisp/textmodes/paragraphs.el b/lisp/textmodes/paragraphs.el
index 73abb155aaa..fd2d83eeebf 100644
--- a/lisp/textmodes/paragraphs.el
+++ b/lisp/textmodes/paragraphs.el
@@ -441,13 +441,12 @@ end-of-paragraph-text
 	  (if (< (point) (point-max))
 	      (end-of-paragraph-text))))))
 
-(defun forward-sentence (&optional arg)
+(defun forward-sentence-default-function (&optional arg)
   "Move forward to next end of sentence.  With argument, repeat.
 When ARG is negative, move backward repeatedly to start of sentence.
 
 The variable `sentence-end' is a regular expression that matches ends of
 sentences.  Also, every paragraph boundary terminates sentences as well."
-  (interactive "^p")
   (or arg (setq arg 1))
   (let ((opoint (point))
         (sentence-end (sentence-end)))
@@ -480,6 +479,18 @@ forward-sentence
     (let ((npoint (constrain-to-field nil opoint t)))
       (not (= npoint opoint)))))
 
+(defvar forward-sentence-function #'forward-sentence-default-function
+  "Function to be used to calculate sentence movements.
+See `forward-sentence' for a description of its behavior.")
+
+(defun forward-sentence (&optional arg)
+  "Move forward to next end of sentence.  With argument, repeat.
+When ARG is negative, move backward repeatedly to start of sentence.
+Delegates its work to `forward-sentence-function'."
+  (interactive "^p")
+  (or arg (setq arg 1))
+  (funcall forward-sentence-function arg))
+
 (defun count-sentences (start end)
   "Count sentences in current buffer from START to END."
   (let ((sentences 0)
diff --git a/lisp/treesit.el b/lisp/treesit.el
index a7f453a8899..0681e758b37 100644
--- a/lisp/treesit.el
+++ b/lisp/treesit.el
@@ -1783,6 +1783,32 @@ treesit-end-of-defun
     (when treesit-defun-skipper
       (funcall treesit-defun-skipper))))
 
+(defvar-local treesit-sentence-type-regexp ""
+  "A regexp that matches the node type of sentence nodes.
+
+A sentence node is a node that is bigger than a sexp, and
+delimits larger statements in the source code.  It is, however,
+smaller in scope than defuns.  This is used by
+`treesit-forward-sentence' and friends.")
+
+(defun treesit-forward-sentence (&optional arg)
+  "Tree-sitter `forward-sentence-function' function.
+
+ARG is the same as in `forward-sentence-function'.
+
+If inside comment or other nodes described in
+`treesit-sentence-type-regexp', use
+`forward-sentence-default-function', else move across nodes as
+described by `treesit-sentence-type-regexp'."
+
+  (if (string-match-p
+       treesit-text-type-regexp
+       (treesit-node-type (treesit-node-at (point))))
+      (funcall #'forward-sentence-default-function arg)
+    (funcall
+     (if (> arg 0) #'treesit-end-of-thing #'treesit-beginning-of-thing)
+     treesit-sentence-type-regexp (abs arg))))
+
 (defvar-local treesit-text-type-regexp "\\`comment\\'"
   "A regexp that matches the node type of textual nodes.
 
@@ -2256,6 +2282,8 @@ treesit-major-mode-setup
                 #'treesit-add-log-current-defun))
 
   (setq-local transpose-sexps-function #'treesit-transpose-sexps)
+  (when treesit-sentence-type-regexp
+    (setq-local forward-sentence-function #'treesit-forward-sentence))
 
   ;; Imenu.
   (when treesit-simple-imenu-settings
-- 
2.34.1

bug#60623: 30.0.50; Add forward-sentence with tree sitter support

[Eli Zaretskii]

> From: Theodor Thornhill <theo@thornhill.no>
> Cc: mardani29@yahoo.es, bug-gnu-emacs@gnu.org, 60623@debbugs.gnu.org,
>  casouri@gmail.com, monnier@iro.umontreal.ca, juri@linkov.net
> Date: Sun, 08 Jan 2023 20:35:58 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Theodor Thornhill <theo@thornhill.no>
> >> Cc: 60623@debbugs.gnu.org, casouri@gmail.com, eliz@gnu.org,
> >>  monnier@iro.umontreal.ca,Juri Linkov <juri@linkov.net>
> >> Date: Sun, 08 Jan 2023 14:29:08 +0100
> >> 
> >> @Eli, what doc changes do you see as needed here?
> >
> > More or less.  They need some polishing, like a few words about what
> > does "sentence" mean in the tree-sitter context.  But we can make
> > these changes after this is in the repository.
> >
> > Thanks.
> 
> 
> Ok, so in other words, this patch is good to go?

Yes, I think so.

> I omitted the additions to java-ts-mode and c-ts-mode.  I can make a
> separate commit to add some values that makes sense for multiple modes
> after?

SGTM.

> Will the changes to the manual lie in "26.2 Sentences"? in the Emacs
> manual?

No, because these are not really sentences in some human-readable
language, these are program parts.  As such they should be somewhere
under "27 Programs", possibly in "Defuns".

However, "Sentences" might mention that programming modes have their
own interpretation of "sentence" and corresponding movement commands.

bug#60623: 30.0.50; Add forward-sentence with tree sitter support

[Theodor Thornhill via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Theodor Thornhill <theo@thornhill.no>
>> Cc: mardani29@yahoo.es, bug-gnu-emacs@gnu.org, 60623@debbugs.gnu.org,
>>  casouri@gmail.com, monnier@iro.umontreal.ca, juri@linkov.net
>> Date: Sun, 08 Jan 2023 20:35:58 +0100
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> From: Theodor Thornhill <theo@thornhill.no>
>> >> Cc: 60623@debbugs.gnu.org, casouri@gmail.com, eliz@gnu.org,
>> >>  monnier@iro.umontreal.ca,Juri Linkov <juri@linkov.net>
>> >> Date: Sun, 08 Jan 2023 14:29:08 +0100
>> >> 
>> >> @Eli, what doc changes do you see as needed here?
>> >
>> > More or less.  They need some polishing, like a few words about what
>> > does "sentence" mean in the tree-sitter context.  But we can make
>> > these changes after this is in the repository.
>> >
>> > Thanks.
>> 
>> 
>> Ok, so in other words, this patch is good to go?
>
> Yes, I think so.
>

Great!

>> I omitted the additions to java-ts-mode and c-ts-mode.  I can make a
>> separate commit to add some values that makes sense for multiple modes
>> after?
>
> SGTM.
>

Nice.  Will you install this for me?

>> Will the changes to the manual lie in "26.2 Sentences"? in the Emacs
>> manual?
>
> No, because these are not really sentences in some human-readable
> language, these are program parts.  As such they should be somewhere
> under "27 Programs", possibly in "Defuns".
>
> However, "Sentences" might mention that programming modes have their
> own interpretation of "sentence" and corresponding movement commands.

Yeah, that makes sense.  Should I make an attempt at such formulations,
or will you do it at a later time?

Theo

bug#60623: 30.0.50; Add forward-sentence with tree sitter support

[Eli Zaretskii]

> From: Theodor Thornhill <theo@thornhill.no>
> Cc: mardani29@yahoo.es, 60623@debbugs.gnu.org, casouri@gmail.com,
>  monnier@iro.umontreal.ca, juri@linkov.net
> Date: Sun, 08 Jan 2023 21:07:21 +0100
> 
> >> Ok, so in other words, this patch is good to go?
> >
> > Yes, I think so.
> >
> 
> Great!
> 
> >> I omitted the additions to java-ts-mode and c-ts-mode.  I can make a
> >> separate commit to add some values that makes sense for multiple modes
> >> after?
> >
> > SGTM.
> >
> 
> Nice.  Will you install this for me?

I'm under the impression that this is still being discussed?

> >> Will the changes to the manual lie in "26.2 Sentences"? in the Emacs
> >> manual?
> >
> > No, because these are not really sentences in some human-readable
> > language, these are program parts.  As such they should be somewhere
> > under "27 Programs", possibly in "Defuns".
> >
> > However, "Sentences" might mention that programming modes have their
> > own interpretation of "sentence" and corresponding movement commands.
> 
> Yeah, that makes sense.  Should I make an attempt at such formulations,
> or will you do it at a later time?

It is better that you try, if only to gain experience ;-)

bug#60623: 30.0.50; Add forward-sentence with tree sitter support

[Theodor Thornhill via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Theodor Thornhill <theo@thornhill.no>
>> Cc: mardani29@yahoo.es, 60623@debbugs.gnu.org, casouri@gmail.com,
>>  monnier@iro.umontreal.ca, juri@linkov.net
>> Date: Sun, 08 Jan 2023 21:07:21 +0100
>> 
>> >> Ok, so in other words, this patch is good to go?
>> >
>> > Yes, I think so.
>> >
>> 
>> Great!
>> 
>> >> I omitted the additions to java-ts-mode and c-ts-mode.  I can make a
>> >> separate commit to add some values that makes sense for multiple modes
>> >> after?
>> >
>> > SGTM.
>> >
>> 
>> Nice.  Will you install this for me?
>
> I'm under the impression that this is still being discussed?
>

Hmm - I thought I'd addressed all comments.  I believe we discussed the
transpose-sexps-function equivalent change, but sure - no hurry :)

>> >> Will the changes to the manual lie in "26.2 Sentences"? in the Emacs
>> >> manual?
>> >
>> > No, because these are not really sentences in some human-readable
>> > language, these are program parts.  As such they should be somewhere
>> > under "27 Programs", possibly in "Defuns".
>> >
>> > However, "Sentences" might mention that programming modes have their
>> > own interpretation of "sentence" and corresponding movement commands.
>> 
>> Yeah, that makes sense.  Should I make an attempt at such formulations,
>> or will you do it at a later time?
>
> It is better that you try, if only to gain experience ;-)

Will do!

Thanks,
Theo

bug#60623: 30.0.50; Add forward-sentence with tree sitter support

[Theodor Thornhill via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[-- Attachment #1: Type: text/plain, Size: 1303 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Theodor Thornhill <theo@thornhill.no>
>> Cc: mardani29@yahoo.es, 60623@debbugs.gnu.org, casouri@gmail.com,
>>  monnier@iro.umontreal.ca, juri@linkov.net
>> Date: Sun, 08 Jan 2023 21:07:21 +0100
>> 
>> >> Ok, so in other words, this patch is good to go?
>> >
>> > Yes, I think so.
>> >
>> 
>> Great!
>> 
>> >> I omitted the additions to java-ts-mode and c-ts-mode.  I can make a
>> >> separate commit to add some values that makes sense for multiple modes
>> >> after?
>> >
>> > SGTM.
>> >
>> 
>> Nice.  Will you install this for me?
>
> I'm under the impression that this is still being discussed?
>
>> >> Will the changes to the manual lie in "26.2 Sentences"? in the Emacs
>> >> manual?
>> >
>> > No, because these are not really sentences in some human-readable
>> > language, these are program parts.  As such they should be somewhere
>> > under "27 Programs", possibly in "Defuns".
>> >
>> > However, "Sentences" might mention that programming modes have their
>> > own interpretation of "sentence" and corresponding movement commands.
>> 
>> Yeah, that makes sense.  Should I make an attempt at such formulations,
>> or will you do it at a later time?
>
> It is better that you try, if only to gain experience ;-)

How about this for starter?

Theo


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Add-forward-sentence-with-tree-sitter-support-bug-60.patch --]
[-- Type: text/x-patch, Size: 9780 bytes --]

From 7b954b4f16d1dc832733932ca58f6e906ef0705d Mon Sep 17 00:00:00 2001
From: Theodor Thornhill <theo@thornhill.no>
Date: Sun, 8 Jan 2023 20:28:02 +0100
Subject: [PATCH] Add forward-sentence with tree sitter support (bug#60623)

* etc/NEWS: Mention the new changes.
* lisp/textmodes/paragraphs.el (forward-sentence-default-function):
Move old implementation to its own function.
(forward-sentence-function): New defvar defaulting to old behavior.
(forward-sentence): Use the variable in this function unconditionally.
* lisp/treesit.el (treesit-sentence-type-regexp): New defvar.
(treesit-forward-sentence): New defun.
(treesit-major-mode-setup): Conditionally set
forward-sentence-function.
* doc/emacs/programs.texi (Defuns): Add new subsection.
(Moving by Sentences): Add some documentation with xrefs to the elisp
manual and related nodes.
* doc/lispref/positions.texi (List Motion): Mention
treesit-sentence-type-regexp and describe how to enable this
functionality.
---
 doc/emacs/programs.texi      | 37 ++++++++++++++++++++++++++++++++++++
 doc/emacs/text.texi          |  8 ++++++++
 doc/lispref/positions.texi   | 14 ++++++++++++++
 etc/NEWS                     | 16 ++++++++++++++++
 lisp/textmodes/paragraphs.el | 15 +++++++++++++--
 lisp/treesit.el              | 27 ++++++++++++++++++++++++++
 6 files changed, 115 insertions(+), 2 deletions(-)

diff --git a/doc/emacs/programs.texi b/doc/emacs/programs.texi
index 44cad5a148e..f7cdd99fa2b 100644
--- a/doc/emacs/programs.texi
+++ b/doc/emacs/programs.texi
@@ -163,6 +163,7 @@ Defuns
 * Left Margin Paren::   An open-paren or similar opening delimiter
                           starts a defun if it is at the left margin.
 * Moving by Defuns::    Commands to move over or mark a major definition.
+* Moving by Sentences:: Commands to move over certain definitions in code.
 * Imenu::               Making buffer indexes as menus.
 * Which Function::      Which Function mode shows which function you are in.
 @end menu
@@ -254,6 +255,42 @@ Moving by Defuns
 language.  Other major modes may replace any or all of these key
 bindings for that purpose.
 
+@node Moving by Sentences
+@subsection Moving by Sentences
+
+  These commands move point or set up the region based on definitions,
+also called @dfn{sentences}.  Even though sentences is usually
+considered when writing human languages, Emacs can use the same
+commands to move over certain constructs in programming languages
+(@pxref{Sentences}, @pxref{Moving by Defuns}).  In a programming
+language a sentence is usually a complete language construct smaller
+than defuns, but larger than sexps (@pxref{List Motion,,, elisp, The
+Emacs Lisp Reference Manual}).
+
+@table @kbd
+@item M-a
+Move to beginning of current or preceding sentence
+(@code{backward-sentence}).
+@item M-e
+Move to end of current or following sentence (@code{forward-sentence}).
+@end table
+
+@cindex move to beginning or end of sentence
+@cindex sentence, move to beginning or end
+@kindex M-a
+@kindex M-e
+@findex backward-sentence
+@findex forward-sentence
+  The commands to move to the beginning and end of the current
+sentence are @kbd{M-a} (@code{backward-sentence}) and @kbd{M-e}
+(@code{forward-sentence}).  If you repeat one of these commands, or
+use a positive numeric argument, each repetition moves to the next
+sentence in the direction of motion.
+
+  @kbd{M-a} with a negative argument @minus{}@var{n} moves forward
+@var{n} times to the next end of a sentence.  Likewise, @kbd{M-e} with
+a negative argument moves back to a start of a sentence.
+
 @node Imenu
 @subsection Imenu
 @cindex index of buffer definitions
diff --git a/doc/emacs/text.texi b/doc/emacs/text.texi
index 8fbf731a4f7..373582a93a4 100644
--- a/doc/emacs/text.texi
+++ b/doc/emacs/text.texi
@@ -253,6 +253,14 @@ Sentences
 of a sentence.  Set the variable @code{sentence-end-without-period} to
 @code{t} in such cases.
 
+  Even though the above mentioned sentence movement commands are based
+on human languages, other Emacs modes can set these command to get
+similar functionality.  What exactly a sentence is in a non-human
+language is dependent on the target language, but usually it is
+complete statements, such as a variable definition and initialization,
+or a conditional statement (@pxref{Moving by Sentences,,, emacs, The
+extensible self-documenting text editor}).
+
 @node Paragraphs
 @section Paragraphs
 @cindex paragraphs
diff --git a/doc/lispref/positions.texi b/doc/lispref/positions.texi
index f3824436246..639d0d8025e 100644
--- a/doc/lispref/positions.texi
+++ b/doc/lispref/positions.texi
@@ -858,6 +858,20 @@ List Motion
 recognize nested defuns.
 @end defvar
 
+@defvar treesit-sentence-type-regexp
+The value of this variable is a regexp matching the node type of sentence
+nodes.  (For ``node'' and ``node type'', @pxref{Parsing Program Source}.)
+
+@findex treesit-forward-sentence
+@findex forward-sentence
+@findex backward-sentence
+If Emacs is compiled with tree-sitter, it can use the tree-sitter
+parser information to move across syntax constructs.  Since what
+exactly is considered a sentence varies between languages, a major mode
+should set @code{treesit-sentence-type-regexp} to determine that.  Then
+the mode can get navigation-by-sentence functionality for free, by using
+@code{forward-sentence} and @code{backward-sentence}.
+
 @node Skipping Characters
 @subsection Skipping Characters
 @cindex skipping characters
diff --git a/etc/NEWS b/etc/NEWS
index 3aa8f2abb77..af15a9b4545 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -66,6 +66,22 @@ treesit.el now unconditionally sets 'transpose-sexps-function' for all
 Tree-sitter modes.  This functionality utilizes the new
 'transpose-sexps-function'.
 
+** New defvar forward-sentence-function.
+Emacs now can set this variable to customize the behavior of the
+'forward-sentence' function.
+
+** New defun forward-sentence-default-function.
+The previous implementation of 'forward-sentence' is moved into its
+own function, to be bound by 'forward-sentence-function'.
+
+** New defvar-local 'treesit-sentence-type-regexp.
+Similarly to 'treesit-defun-type-regexp', this variable is used to
+navigate sentences in Tree-sitter enabled modes.
+
+** New function 'treesit-forward-sentence'.
+treesit.el now conditionally sets 'forward-sentence-function' for all
+Tree-sitter modes that sets 'treesit-sentence-type-regexp'.
+
 \f
 * Changes in Specialized Modes and Packages in Emacs 30.1
 ---
diff --git a/lisp/textmodes/paragraphs.el b/lisp/textmodes/paragraphs.el
index 73abb155aaa..fd2d83eeebf 100644
--- a/lisp/textmodes/paragraphs.el
+++ b/lisp/textmodes/paragraphs.el
@@ -441,13 +441,12 @@ end-of-paragraph-text
 	  (if (< (point) (point-max))
 	      (end-of-paragraph-text))))))
 
-(defun forward-sentence (&optional arg)
+(defun forward-sentence-default-function (&optional arg)
   "Move forward to next end of sentence.  With argument, repeat.
 When ARG is negative, move backward repeatedly to start of sentence.
 
 The variable `sentence-end' is a regular expression that matches ends of
 sentences.  Also, every paragraph boundary terminates sentences as well."
-  (interactive "^p")
   (or arg (setq arg 1))
   (let ((opoint (point))
         (sentence-end (sentence-end)))
@@ -480,6 +479,18 @@ forward-sentence
     (let ((npoint (constrain-to-field nil opoint t)))
       (not (= npoint opoint)))))
 
+(defvar forward-sentence-function #'forward-sentence-default-function
+  "Function to be used to calculate sentence movements.
+See `forward-sentence' for a description of its behavior.")
+
+(defun forward-sentence (&optional arg)
+  "Move forward to next end of sentence.  With argument, repeat.
+When ARG is negative, move backward repeatedly to start of sentence.
+Delegates its work to `forward-sentence-function'."
+  (interactive "^p")
+  (or arg (setq arg 1))
+  (funcall forward-sentence-function arg))
+
 (defun count-sentences (start end)
   "Count sentences in current buffer from START to END."
   (let ((sentences 0)
diff --git a/lisp/treesit.el b/lisp/treesit.el
index a7f453a8899..95f0fec739f 100644
--- a/lisp/treesit.el
+++ b/lisp/treesit.el
@@ -1792,6 +1792,31 @@ treesit-text-type-regexp
 \"text_block\" in the case of a string.  This is used by
 `prog-fill-reindent-defun' and friends.")
 
+(defvar-local treesit-sentence-type-regexp ""
+  "A regexp that matches the node type of sentence nodes.
+
+A sentence node is a node that is bigger than a sexp, and
+delimits larger statements in the source code.  It is, however,
+smaller in scope than defuns.  This is used by
+`treesit-forward-sentence' and friends.")
+
+(defun treesit-forward-sentence (&optional arg)
+  "Tree-sitter `forward-sentence-function' function.
+
+ARG is the same as in `forward-sentence-function'.
+
+If inside comment or other nodes described in
+`treesit-sentence-type-regexp', use
+`forward-sentence-default-function', else move across nodes as
+described by `treesit-sentence-type-regexp'."
+  (if (string-match-p
+       treesit-text-type-regexp
+       (treesit-node-type (treesit-node-at (point))))
+      (funcall #'forward-sentence-default-function arg)
+    (funcall
+     (if (> arg 0) #'treesit-end-of-thing #'treesit-beginning-of-thing)
+     treesit-sentence-type-regexp (abs arg))))
+
 (defun treesit-default-defun-skipper ()
   "Skips spaces after navigating a defun.
 This function tries to move to the beginning of a line, either by
@@ -2256,6 +2281,8 @@ treesit-major-mode-setup
                 #'treesit-add-log-current-defun))
 
   (setq-local transpose-sexps-function #'treesit-transpose-sexps)
+  (when treesit-sentence-type-regexp
+    (setq-local forward-sentence-function #'treesit-forward-sentence))
 
   ;; Imenu.
   (when treesit-simple-imenu-settings
-- 
2.34.1

bug#60623: 30.0.50; Add forward-sentence with tree sitter support

[Eli Zaretskii]

> From: Theodor Thornhill <theo@thornhill.no>
> Cc: mardani29@yahoo.es, 60623@debbugs.gnu.org, casouri@gmail.com,
>  monnier@iro.umontreal.ca, juri@linkov.net
> Date: Tue, 10 Jan 2023 09:37:26 +0100
> 
> >> > No, because these are not really sentences in some human-readable
> >> > language, these are program parts.  As such they should be somewhere
> >> > under "27 Programs", possibly in "Defuns".
> >> >
> >> > However, "Sentences" might mention that programming modes have their
> >> > own interpretation of "sentence" and corresponding movement commands.
> >> 
> >> Yeah, that makes sense.  Should I make an attempt at such formulations,
> >> or will you do it at a later time?
> >
> > It is better that you try, if only to gain experience ;-)
> 
> How about this for starter?

Very good, thank you very much.  A few comments below.

> --- a/doc/emacs/programs.texi
> +++ b/doc/emacs/programs.texi
> @@ -163,6 +163,7 @@ Defuns
>  * Left Margin Paren::   An open-paren or similar opening delimiter
>                            starts a defun if it is at the left margin.
>  * Moving by Defuns::    Commands to move over or mark a major definition.
> +* Moving by Sentences:: Commands to move over certain definitions in code.
                                                         ^^^^^^^^^^^
I'd use "code units" or "units of code" here.

Also, should we perhaps name the section "Moving by Statements"? or
would it be too inaccurate?

> +  These commands move point or set up the region based on definitions,
> +also called @dfn{sentences}.  Even though sentences is usually

Each @dfn in a manual should have an index entry, so that readers
could easily find it.  in this case, the index entry should qualify
the "sentences" term by the fact that we are talking about units of
code.  So:

  @cindex sentences, in programming languages

> +considered when writing human languages, Emacs can use the same
> +commands to move over certain constructs in programming languages
> +(@pxref{Sentences}, @pxref{Moving by Defuns}).  In a programming
> +language a sentence is usually a complete language construct smaller
> +than defuns, but larger than sexps (@pxref{List Motion,,, elisp, The
> +Emacs Lisp Reference Manual}).

A couple of examples from two different languages could be a great
help here.  Otherwise this text sounds a bit too abstract.

> +@kindex M-a
> +@kindex M-e

Since we already have M-e elsewhere in the manual, I suggest to
qualify the key bindings here:

  @kindex M-a @r{(programming modes)}

and similarly for M-e.  The @r{..} thingy is necessary to reset to the
default typeface, since key index is implicitly typeset in @code.

> +@findex backward-sentence
> +@findex forward-sentence

Likewise with these two @findex entries: qualify them, since we have
the same commands documented elsewhere under "Sentences".

> --- a/doc/emacs/text.texi
> +++ b/doc/emacs/text.texi
> @@ -253,6 +253,14 @@ Sentences
>  of a sentence.  Set the variable @code{sentence-end-without-period} to
>  @code{t} in such cases.
>  
> +  Even though the above mentioned sentence movement commands are based
> +on human languages, other Emacs modes can set these command to get
> +similar functionality.  What exactly a sentence is in a non-human
> +language is dependent on the target language, but usually it is
> +complete statements, such as a variable definition and initialization,
> +or a conditional statement (@pxref{Moving by Sentences,,, emacs, The
> +extensible self-documenting text editor}).

The last sentence should be in "Moving by Sentences", since it
describes the commands documented there.  Also, please add a
cross-reference here to "Moving by Sentences", since you mention that
in the text (and rightfully so).

> +@defvar treesit-sentence-type-regexp
> +The value of this variable is a regexp matching the node type of sentence
> +nodes.  (For ``node'' and ``node type'', @pxref{Parsing Program Source}.)
> +
> +@findex treesit-forward-sentence
> +@findex forward-sentence
> +@findex backward-sentence
> +If Emacs is compiled with tree-sitter, it can use the tree-sitter
> +parser information to move across syntax constructs.  Since what
> +exactly is considered a sentence varies between languages, a major mode
> +should set @code{treesit-sentence-type-regexp} to determine that.  Then
> +the mode can get navigation-by-sentence functionality for free, by using
> +@code{forward-sentence} and @code{backward-sentence}.

Here please also add a cross-reference to the "Moving by Sentences"
node in the Emacs manual, so that people could understand what kind of
"sentences" this is talking about.

> +** New defvar forward-sentence-function.
      ^^^^^^^^^^
"New variable"

> +Emacs now can set this variable to customize the behavior of the
> +'forward-sentence' function.

Not "Emacs", but "major modes".

> +** New defun forward-sentence-default-function.
      ^^^^^^^^^
"New function"

> +The previous implementation of 'forward-sentence' is moved into its
> +own function, to be bound by 'forward-sentence-function'.
> +
> +** New defvar-local 'treesit-sentence-type-regexp.
> +Similarly to 'treesit-defun-type-regexp', this variable is used to
> +navigate sentences in Tree-sitter enabled modes.
> +
> +** New function 'treesit-forward-sentence'.
> +treesit.el now conditionally sets 'forward-sentence-function' for all
> +Tree-sitter modes that sets 'treesit-sentence-type-regexp'.

Please make these related items sub-headings of a common heading,
something like "Commands and variables to move by program statements".

> +
>  \f
>  * Changes in Specialized Modes and Packages in Emacs 30.1
>  ---
> diff --git a/lisp/textmodes/paragraphs.el b/lisp/textmodes/paragraphs.el
> index 73abb155aaa..fd2d83eeebf 100644
> --- a/lisp/textmodes/paragraphs.el
> +++ b/lisp/textmodes/paragraphs.el
> @@ -441,13 +441,12 @@ end-of-paragraph-text
>  	  (if (< (point) (point-max))
>  	      (end-of-paragraph-text))))))
>  
> -(defun forward-sentence (&optional arg)
> +(defun forward-sentence-default-function (&optional arg)
>    "Move forward to next end of sentence.  With argument, repeat.
>  When ARG is negative, move backward repeatedly to start of sentence.
>  
>  The variable `sentence-end' is a regular expression that matches ends of
>  sentences.  Also, every paragraph boundary terminates sentences as well."
> -  (interactive "^p")
>    (or arg (setq arg 1))
>    (let ((opoint (point))
>          (sentence-end (sentence-end)))
> @@ -480,6 +479,18 @@ forward-sentence
>      (let ((npoint (constrain-to-field nil opoint t)))
>        (not (= npoint opoint)))))
>  
> +(defvar forward-sentence-function #'forward-sentence-default-function
> +  "Function to be used to calculate sentence movements.
> +See `forward-sentence' for a description of its behavior.")
> +
> +(defun forward-sentence (&optional arg)
> +  "Move forward to next end of sentence.  With argument, repeat.
                                             ^^^^^^^^^^^^^^^^^^^^^
"With argument ARG, repeat."  The doc string should reference the
arguments where possible.

> +When ARG is negative, move backward repeatedly to start of sentence.
   ^^^^
"If", not "When".

> +(defvar-local treesit-sentence-type-regexp ""
> +  "A regexp that matches the node type of sentence nodes.

Why is the default an empty regexp? wouldn't nil be better?

bug#60623: 30.0.50; Add forward-sentence with tree sitter support

[Theodor Thornhill via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[-- Attachment #1: Type: text/plain, Size: 8059 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Theodor Thornhill <theo@thornhill.no>
>> Cc: mardani29@yahoo.es, 60623@debbugs.gnu.org, casouri@gmail.com,
>>  monnier@iro.umontreal.ca, juri@linkov.net
>> Date: Tue, 10 Jan 2023 09:37:26 +0100
>> 
>> >> > No, because these are not really sentences in some human-readable
>> >> > language, these are program parts.  As such they should be somewhere
>> >> > under "27 Programs", possibly in "Defuns".
>> >> >
>> >> > However, "Sentences" might mention that programming modes have their
>> >> > own interpretation of "sentence" and corresponding movement commands.
>> >> 
>> >> Yeah, that makes sense.  Should I make an attempt at such formulations,
>> >> or will you do it at a later time?
>> >
>> > It is better that you try, if only to gain experience ;-)
>> 
>> How about this for starter?
>
> Very good, thank you very much.  A few comments below.
>
>> --- a/doc/emacs/programs.texi
>> +++ b/doc/emacs/programs.texi
>> @@ -163,6 +163,7 @@ Defuns
>>  * Left Margin Paren::   An open-paren or similar opening delimiter
>>                            starts a defun if it is at the left margin.
>>  * Moving by Defuns::    Commands to move over or mark a major definition.
>> +* Moving by Sentences:: Commands to move over certain definitions in code.
>                                                          ^^^^^^^^^^^
> I'd use "code units" or "units of code" here.

Done.

>
> Also, should we perhaps name the section "Moving by Statements"? or
> would it be too inaccurate?
>

I'm not sure.  I think that maybe because the commands involved, and the
ones that implicitly will be impacted, such as kill-sentence and friends
it is best to stay with Sentences?  But a statement is the better term
wrt programming languages of course.  I hold no strong opinions here.

>> +  These commands move point or set up the region based on definitions,
>> +also called @dfn{sentences}.  Even though sentences is usually
>
> Each @dfn in a manual should have an index entry, so that readers
> could easily find it.  in this case, the index entry should qualify
> the "sentences" term by the fact that we are talking about units of
> code.  So:
>
>   @cindex sentences, in programming languages
>

Done.

>> +considered when writing human languages, Emacs can use the same
>> +commands to move over certain constructs in programming languages
>> +(@pxref{Sentences}, @pxref{Moving by Defuns}).  In a programming
>> +language a sentence is usually a complete language construct smaller
>> +than defuns, but larger than sexps (@pxref{List Motion,,, elisp, The
>> +Emacs Lisp Reference Manual}).
>
> A couple of examples from two different languages could be a great
> help here.  Otherwise this text sounds a bit too abstract.
>

Something like this?

>> +@kindex M-a
>> +@kindex M-e
>
> Since we already have M-e elsewhere in the manual, I suggest to
> qualify the key bindings here:
>
>   @kindex M-a @r{(programming modes)}
>
> and similarly for M-e.  The @r{..} thingy is necessary to reset to the
> default typeface, since key index is implicitly typeset in @code.
>
>> +@findex backward-sentence
>> +@findex forward-sentence
>
> Likewise with these two @findex entries: qualify them, since we have
> the same commands documented elsewhere under "Sentences".
>

Done.

>> --- a/doc/emacs/text.texi
>> +++ b/doc/emacs/text.texi
>> @@ -253,6 +253,14 @@ Sentences
>>  of a sentence.  Set the variable @code{sentence-end-without-period} to
>>  @code{t} in such cases.
>>  
>> +  Even though the above mentioned sentence movement commands are based
>> +on human languages, other Emacs modes can set these command to get
>> +similar functionality.  What exactly a sentence is in a non-human
>> +language is dependent on the target language, but usually it is
>> +complete statements, such as a variable definition and initialization,
>> +or a conditional statement (@pxref{Moving by Sentences,,, emacs, The
>> +extensible self-documenting text editor}).
>
> The last sentence should be in "Moving by Sentences", since it
> describes the commands documented there.  Also, please add a
> cross-reference here to "Moving by Sentences", since you mention that
> in the text (and rightfully so).
>

Is something like this what you meant?

>> +@defvar treesit-sentence-type-regexp
>> +The value of this variable is a regexp matching the node type of sentence
>> +nodes.  (For ``node'' and ``node type'', @pxref{Parsing Program Source}.)
>> +
>> +@findex treesit-forward-sentence
>> +@findex forward-sentence
>> +@findex backward-sentence
>> +If Emacs is compiled with tree-sitter, it can use the tree-sitter
>> +parser information to move across syntax constructs.  Since what
>> +exactly is considered a sentence varies between languages, a major mode
>> +should set @code{treesit-sentence-type-regexp} to determine that.  Then
>> +the mode can get navigation-by-sentence functionality for free, by using
>> +@code{forward-sentence} and @code{backward-sentence}.
>
> Here please also add a cross-reference to the "Moving by Sentences"
> node in the Emacs manual, so that people could understand what kind of
> "sentences" this is talking about.
>
>> +** New defvar forward-sentence-function.
>       ^^^^^^^^^^
> "New variable"
>
>> +Emacs now can set this variable to customize the behavior of the
>> +'forward-sentence' function.
>
> Not "Emacs", but "major modes".
>
>> +** New defun forward-sentence-default-function.
>       ^^^^^^^^^
> "New function"
>
>> +The previous implementation of 'forward-sentence' is moved into its
>> +own function, to be bound by 'forward-sentence-function'.
>> +
>> +** New defvar-local 'treesit-sentence-type-regexp.
>> +Similarly to 'treesit-defun-type-regexp', this variable is used to
>> +navigate sentences in Tree-sitter enabled modes.
>> +
>> +** New function 'treesit-forward-sentence'.
>> +treesit.el now conditionally sets 'forward-sentence-function' for all
>> +Tree-sitter modes that sets 'treesit-sentence-type-regexp'.
>
> Please make these related items sub-headings of a common heading,
> something like "Commands and variables to move by program statements".
>

Done.

>> +
>>  \f
>>  * Changes in Specialized Modes and Packages in Emacs 30.1
>>  ---
>> diff --git a/lisp/textmodes/paragraphs.el b/lisp/textmodes/paragraphs.el
>> index 73abb155aaa..fd2d83eeebf 100644
>> --- a/lisp/textmodes/paragraphs.el
>> +++ b/lisp/textmodes/paragraphs.el
>> @@ -441,13 +441,12 @@ end-of-paragraph-text
>>  	  (if (< (point) (point-max))
>>  	      (end-of-paragraph-text))))))
>>  
>> -(defun forward-sentence (&optional arg)
>> +(defun forward-sentence-default-function (&optional arg)
>>    "Move forward to next end of sentence.  With argument, repeat.
>>  When ARG is negative, move backward repeatedly to start of sentence.
>>  
>>  The variable `sentence-end' is a regular expression that matches ends of
>>  sentences.  Also, every paragraph boundary terminates sentences as well."
>> -  (interactive "^p")
>>    (or arg (setq arg 1))
>>    (let ((opoint (point))
>>          (sentence-end (sentence-end)))
>> @@ -480,6 +479,18 @@ forward-sentence
>>      (let ((npoint (constrain-to-field nil opoint t)))
>>        (not (= npoint opoint)))))
>>  
>> +(defvar forward-sentence-function #'forward-sentence-default-function
>> +  "Function to be used to calculate sentence movements.
>> +See `forward-sentence' for a description of its behavior.")
>> +
>> +(defun forward-sentence (&optional arg)
>> +  "Move forward to next end of sentence.  With argument, repeat.
>                                              ^^^^^^^^^^^^^^^^^^^^^
> "With argument ARG, repeat."  The doc string should reference the
> arguments where possible.
>

Thanks, done.

>> +When ARG is negative, move backward repeatedly to start of sentence.
>    ^^^^
> "If", not "When".
>

Done

>> +(defvar-local treesit-sentence-type-regexp ""
>> +  "A regexp that matches the node type of sentence nodes.
>
> Why is the default an empty regexp? wouldn't nil be better?

Indeed it will, done.

How about this?

Theo


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Add-forward-sentence-with-tree-sitter-support-bug-60.patch --]
[-- Type: text/x-patch, Size: 10322 bytes --]

From 05b880680f70825d56b624b55f39b9114ba22c8d Mon Sep 17 00:00:00 2001
From: Theodor Thornhill <theo@thornhill.no>
Date: Sun, 8 Jan 2023 20:28:02 +0100
Subject: [PATCH] Add forward-sentence with tree sitter support (bug#60623)

* etc/NEWS: Mention the new changes.
* lisp/textmodes/paragraphs.el (forward-sentence-default-function):
Move old implementation to its own function.
(forward-sentence-function): New defvar defaulting to old behavior.
(forward-sentence): Use the variable in this function unconditionally.
* lisp/treesit.el (treesit-sentence-type-regexp): New defvar.
(treesit-forward-sentence): New defun.
(treesit-major-mode-setup): Conditionally set
forward-sentence-function.
* doc/emacs/programs.texi (Defuns): Add new subsection.
(Moving by Sentences): Add some documentation with xrefs to the elisp
manual and related nodes.
* doc/lispref/positions.texi (List Motion): Mention
treesit-sentence-type-regexp and describe how to enable this
functionality.
---
 doc/emacs/programs.texi      | 56 ++++++++++++++++++++++++++++++++++++
 doc/emacs/text.texi          |  5 ++++
 doc/lispref/positions.texi   | 17 +++++++++++
 etc/NEWS                     | 18 ++++++++++++
 lisp/textmodes/paragraphs.el | 15 ++++++++--
 lisp/treesit.el              | 27 +++++++++++++++++
 6 files changed, 136 insertions(+), 2 deletions(-)

diff --git a/doc/emacs/programs.texi b/doc/emacs/programs.texi
index 44cad5a148e..a2cdf6c6eb9 100644
--- a/doc/emacs/programs.texi
+++ b/doc/emacs/programs.texi
@@ -163,6 +163,7 @@ Defuns
 * Left Margin Paren::   An open-paren or similar opening delimiter
                           starts a defun if it is at the left margin.
 * Moving by Defuns::    Commands to move over or mark a major definition.
+* Moving by Sentences:: Commands to move over certain code units.
 * Imenu::               Making buffer indexes as menus.
 * Which Function::      Which Function mode shows which function you are in.
 @end menu
@@ -254,6 +255,61 @@ Moving by Defuns
 language.  Other major modes may replace any or all of these key
 bindings for that purpose.
 
+@node Moving by Sentences
+@subsection Moving by Sentences
+@cindex sentences, in programming languages
+
+  These commands move point or set up the region based on units of
+code, also called @dfn{sentences}.  Even though sentences are usually
+considered when writing human languages, Emacs can use the same
+commands to move over certain constructs in programming languages
+(@pxref{Sentences}, @pxref{Moving by Defuns}).  In a programming
+language a sentence is usually a complete language construct smaller
+than defuns, but larger than sexps (@pxref{List Motion,,, elisp, The
+Emacs Lisp Reference Manual}).  What exactly a sentence is in a
+non-human language is dependent on the target language, but usually it
+is complete statements, such as a variable definition and
+initialization, or a conditional statement.  An example of a sentence
+in the C language could be
+
+@example
+int x = 5;
+@end example
+
+or in the JavaScript language it could look like
+
+@example
+const thing = () => console.log("Hi");
+
+const foo = [1] == '1'
+  ? "No way"
+  : "...";
+@end example
+
+@table @kbd
+@item M-a
+Move to beginning of current or preceding sentence
+(@code{backward-sentence}).
+@item M-e
+Move to end of current or following sentence (@code{forward-sentence}).
+@end table
+
+@cindex move to beginning or end of sentence
+@cindex sentence, move to beginning or end
+@kindex M-a @r{(programming modes)}
+@kindex M-e @r{(programming modes)}
+@findex backward-sentence @r{(programming modes)}
+@findex forward-sentence @r{(programming modes)}
+  The commands to move to the beginning and end of the current
+sentence are @kbd{M-a} (@code{backward-sentence}) and @kbd{M-e}
+(@code{forward-sentence}).  If you repeat one of these commands, or
+use a positive numeric argument, each repetition moves to the next
+sentence in the direction of motion.
+
+  @kbd{M-a} with a negative argument @minus{}@var{n} moves forward
+@var{n} times to the next end of a sentence.  Likewise, @kbd{M-e} with
+a negative argument moves back to a start of a sentence.
+
 @node Imenu
 @subsection Imenu
 @cindex index of buffer definitions
diff --git a/doc/emacs/text.texi b/doc/emacs/text.texi
index 8fbf731a4f7..acd3bb21c29 100644
--- a/doc/emacs/text.texi
+++ b/doc/emacs/text.texi
@@ -253,6 +253,11 @@ Sentences
 of a sentence.  Set the variable @code{sentence-end-without-period} to
 @code{t} in such cases.
 
+  Even though the above mentioned sentence movement commands are based
+on human languages, other Emacs modes can set these command to get
+similar functionality (@pxref{Moving by Sentences,,, emacs, The
+extensible self-documenting text editor}).
+
 @node Paragraphs
 @section Paragraphs
 @cindex paragraphs
diff --git a/doc/lispref/positions.texi b/doc/lispref/positions.texi
index f3824436246..8d95ecee7ab 100644
--- a/doc/lispref/positions.texi
+++ b/doc/lispref/positions.texi
@@ -858,6 +858,23 @@ List Motion
 recognize nested defuns.
 @end defvar
 
+@defvar treesit-sentence-type-regexp
+The value of this variable is a regexp matching the node type of sentence
+nodes.  (For ``node'' and ``node type'', @pxref{Parsing Program Source}.)
+@end defvar
+
+@findex treesit-forward-sentence
+@findex forward-sentence
+@findex backward-sentence
+If Emacs is compiled with tree-sitter, it can use the tree-sitter
+parser information to move across syntax constructs.  Since what
+exactly is considered a sentence varies between languages, a major
+mode should set @code{treesit-sentence-type-regexp} to determine that.
+Then the mode can get navigation-by-sentence functionality for free,
+by using @code{forward-sentence} and
+@code{backward-sentence}(@pxref{Moving by Sentences,,, emacs, The
+extensible self-documenting text editor}).
+
 @node Skipping Characters
 @subsection Skipping Characters
 @cindex skipping characters
diff --git a/etc/NEWS b/etc/NEWS
index 3aa8f2abb77..0c782eeaee8 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -66,6 +66,24 @@ treesit.el now unconditionally sets 'transpose-sexps-function' for all
 Tree-sitter modes.  This functionality utilizes the new
 'transpose-sexps-function'.
 
+** Commands and variables to move by program statements
+
+*** New variable 'forward-sentence-function'.
+Major modes now can set this variable to customize the behavior of the
+'forward-sentence' function.
+
+*** New function 'forward-sentence-default-function'.
+The previous implementation of 'forward-sentence' is moved into its
+own function, to be bound by 'forward-sentence-function'.
+
+*** New defvar-local 'treesit-sentence-type-regexp.
+Similarly to 'treesit-defun-type-regexp', this variable is used to
+navigate sentences in Tree-sitter enabled modes.
+
+*** New function 'treesit-forward-sentence'.
+treesit.el now conditionally sets 'forward-sentence-function' for all
+Tree-sitter modes that sets 'treesit-sentence-type-regexp'.
+
 \f
 * Changes in Specialized Modes and Packages in Emacs 30.1
 ---
diff --git a/lisp/textmodes/paragraphs.el b/lisp/textmodes/paragraphs.el
index 73abb155aaa..bf249fdcdfb 100644
--- a/lisp/textmodes/paragraphs.el
+++ b/lisp/textmodes/paragraphs.el
@@ -441,13 +441,12 @@ end-of-paragraph-text
 	  (if (< (point) (point-max))
 	      (end-of-paragraph-text))))))
 
-(defun forward-sentence (&optional arg)
+(defun forward-sentence-default-function (&optional arg)
   "Move forward to next end of sentence.  With argument, repeat.
 When ARG is negative, move backward repeatedly to start of sentence.
 
 The variable `sentence-end' is a regular expression that matches ends of
 sentences.  Also, every paragraph boundary terminates sentences as well."
-  (interactive "^p")
   (or arg (setq arg 1))
   (let ((opoint (point))
         (sentence-end (sentence-end)))
@@ -480,6 +479,18 @@ forward-sentence
     (let ((npoint (constrain-to-field nil opoint t)))
       (not (= npoint opoint)))))
 
+(defvar forward-sentence-function #'forward-sentence-default-function
+  "Function to be used to calculate sentence movements.
+See `forward-sentence' for a description of its behavior.")
+
+(defun forward-sentence (&optional arg)
+  "Move forward to next end of sentence.  With argument ARG, repeat.
+If ARG is negative, move backward repeatedly to start of
+sentence.  Delegates its work to `forward-sentence-function'."
+  (interactive "^p")
+  (or arg (setq arg 1))
+  (funcall forward-sentence-function arg))
+
 (defun count-sentences (start end)
   "Count sentences in current buffer from START to END."
   (let ((sentences 0)
diff --git a/lisp/treesit.el b/lisp/treesit.el
index a7f453a8899..4c01a8db281 100644
--- a/lisp/treesit.el
+++ b/lisp/treesit.el
@@ -1792,6 +1792,31 @@ treesit-text-type-regexp
 \"text_block\" in the case of a string.  This is used by
 `prog-fill-reindent-defun' and friends.")
 
+(defvar-local treesit-sentence-type-regexp nil
+  "A regexp that matches the node type of sentence nodes.
+
+A sentence node is a node that is bigger than a sexp, and
+delimits larger statements in the source code.  It is, however,
+smaller in scope than defuns.  This is used by
+`treesit-forward-sentence' and friends.")
+
+(defun treesit-forward-sentence (&optional arg)
+  "Tree-sitter `forward-sentence-function' function.
+
+ARG is the same as in `forward-sentence'.
+
+If inside comment or other nodes described in
+`treesit-sentence-type-regexp', use
+`forward-sentence-default-function', else move across nodes as
+described by `treesit-sentence-type-regexp'."
+  (if (string-match-p
+       treesit-text-type-regexp
+       (treesit-node-type (treesit-node-at (point))))
+      (funcall #'forward-sentence-default-function arg)
+    (funcall
+     (if (> arg 0) #'treesit-end-of-thing #'treesit-beginning-of-thing)
+     treesit-sentence-type-regexp (abs arg))))
+
 (defun treesit-default-defun-skipper ()
   "Skips spaces after navigating a defun.
 This function tries to move to the beginning of a line, either by
@@ -2256,6 +2281,8 @@ treesit-major-mode-setup
                 #'treesit-add-log-current-defun))
 
   (setq-local transpose-sexps-function #'treesit-transpose-sexps)
+  (when treesit-sentence-type-regexp
+    (setq-local forward-sentence-function #'treesit-forward-sentence))
 
   ;; Imenu.
   (when treesit-simple-imenu-settings
-- 
2.34.1

bug#60623: 30.0.50; Add forward-sentence with tree sitter support

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>>> +* Moving by Sentences:: Commands to move over certain definitions in code.
>>                                                          ^^^^^^^^^^^
>> I'd use "code units" or "units of code" here.
>
> Done.
>
>>
>> Also, should we perhaps name the section "Moving by Statements"? or
>> would it be too inaccurate?
>>
>
> I'm not sure.  I think that maybe because the commands involved, and the
> ones that implicitly will be impacted, such as kill-sentence and friends
> it is best to stay with Sentences?  But a statement is the better term
> wrt programming languages of course.  I hold no strong opinions here.

FWIW, while it may correspond to "statements" for some languages, it
will correspond to other things in other languages (e.g. those that
don't have a notion of "statement"), so it's probably best to stick to
"sentence" here and then in the doc explain how that notion is expected
to be mapped to notions that make sense for a given language.


        Stefan

bug#60623: 30.0.50; Add forward-sentence with tree sitter support

[Theodor Thornhill via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>> +* Moving by Sentences:: Commands to move over certain definitions in code.
>>>                                                          ^^^^^^^^^^^
>>> I'd use "code units" or "units of code" here.
>>
>> Done.
>>
>>>
>>> Also, should we perhaps name the section "Moving by Statements"? or
>>> would it be too inaccurate?
>>>
>>
>> I'm not sure.  I think that maybe because the commands involved, and the
>> ones that implicitly will be impacted, such as kill-sentence and friends
>> it is best to stay with Sentences?  But a statement is the better term
>> wrt programming languages of course.  I hold no strong opinions here.
>
> FWIW, while it may correspond to "statements" for some languages, it
> will correspond to other things in other languages (e.g. those that
> don't have a notion of "statement"), so it's probably best to stick to
> "sentence" here and then in the doc explain how that notion is expected
> to be mapped to notions that make sense for a given language.
>
>
>         Stefan


Yeah, that makes sense.

Theo

bug#60623: 30.0.50; Add forward-sentence with tree sitter support

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>>>>> +* Moving by Sentences:: Commands to move over certain definitions in code.
[...]
>> FWIW, while it may correspond to "statements" for some languages, it
>> will correspond to other things in other languages (e.g. those that
>> don't have a notion of "statement"), so it's probably best to stick to
>> "sentence" here and then in the doc explain how that notion is expected
>> to be mapped to notions that make sense for a given language.
> Yeah, that makes sense.

Another option is to use a mix, as in:

    Moving by Sentences:: Commands to move over statement-like code chunks.


-- Stefan

bug#60623: 30.0.50; Add forward-sentence with tree sitter support

[Drew Adams]

> > I'm not sure.  I think that maybe because the commands involved, and the
> > ones that implicitly will be impacted, such as kill-sentence and friends
> > it is best to stay with Sentences?  But a statement is the better term
> > wrt programming languages of course.  I hold no strong opinions here.
> 
> FWIW, while it may correspond to "statements" for some languages, it
> will correspond to other things in other languages (e.g. those that
> don't have a notion of "statement"), so it's probably best to stick to
> "sentence" here and then in the doc explain how that notion is expected
> to be mapped to notions that make sense for a given language.

I'm not following this thread.  But neither
"statement" nor "sentence" sounds appropriate for
what you're apparently talking about.  Both would
seem to be misleading.  Maybe you should add a new
THING type for just what you mean?

bug#60623: 30.0.50; Add forward-sentence with tree sitter support

[Eli Zaretskii]

> From: Theodor Thornhill <theo@thornhill.no>
> Cc: mardani29@yahoo.es, 60623@debbugs.gnu.org, casouri@gmail.com,
>  monnier@iro.umontreal.ca, juri@linkov.net
> Date: Tue, 10 Jan 2023 20:33:52 +0100
> 
> How about this?

LGTM, thanks.  Just one gotcha:

> --- a/doc/emacs/text.texi
> +++ b/doc/emacs/text.texi
> @@ -253,6 +253,11 @@ Sentences
>  of a sentence.  Set the variable @code{sentence-end-without-period} to
>  @code{t} in such cases.
>  
> +  Even though the above mentioned sentence movement commands are based
> +on human languages, other Emacs modes can set these command to get
> +similar functionality (@pxref{Moving by Sentences,,, emacs, The
> +extensible self-documenting text editor}).

This is a cross-reference to the same manual, so you don't need the
full 5-argument form of @pxref; just the node name will suffice.

I think I can give you write access to the tree now, so please request
access on the Savannah page (and create a user if you haven't
already), and then you can install this yourself.

bug#60623: 30.0.50; Add forward-sentence with tree sitter support

[Theodor Thornhill via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Theodor Thornhill <theo@thornhill.no>
>> Cc: mardani29@yahoo.es, 60623@debbugs.gnu.org, casouri@gmail.com,
>>  monnier@iro.umontreal.ca, juri@linkov.net
>> Date: Tue, 10 Jan 2023 20:33:52 +0100
>> 
>> How about this?
>
> LGTM, thanks.  Just one gotcha:
>
>> --- a/doc/emacs/text.texi
>> +++ b/doc/emacs/text.texi
>> @@ -253,6 +253,11 @@ Sentences
>>  of a sentence.  Set the variable @code{sentence-end-without-period} to
>>  @code{t} in such cases.
>>  
>> +  Even though the above mentioned sentence movement commands are based
>> +on human languages, other Emacs modes can set these command to get
>> +similar functionality (@pxref{Moving by Sentences,,, emacs, The
>> +extensible self-documenting text editor}).
>
> This is a cross-reference to the same manual, so you don't need the
> full 5-argument form of @pxref; just the node name will suffice.

Right, thanks!

>
> I think I can give you write access to the tree now, so please request
> access on the Savannah page (and create a user if you haven't
> already), and then you can install this yourself.

Wow thanks!  Request sent :-)

Theo

bug#60623: 30.0.50; Add forward-sentence with tree sitter support

[Juri Linkov]

> @Juri: I added a change with how I understood what you meant.  Is that
> in your line of reasoning?

Thanks, now this is more maintainable.

bug#60623: 30.0.50; Add forward-sentence with tree sitter support

[Juri Linkov]

> -(defun forward-sentence (&optional arg)
> +(defvar forward-sentence-function
> +  (lambda (&optional arg)

A good practice is to name such a function e.g. forward-sentence-default-function.

bug#60623: 30.0.50; Add forward-sentence with tree sitter support

[Theodor Thornhill via Bug reports for GNU Emacs, the Swiss army knife of text editors]

On 8 January 2023 09:36:42 CET, Juri Linkov <juri@linkov.net> wrote:
>> -(defun forward-sentence (&optional arg)
>> +(defvar forward-sentence-function
>> +  (lambda (&optional arg)
>
>A good practice is to name such a function e.g. forward-sentence-default-function.

Is this practice used anywhere else? Iirc forward-sexp-function doesn't follow that practice.

Thanks,
Theo

bug#60623: 30.0.50; Add forward-sentence with tree sitter support

[Drew Adams]

> >A good practice is to name such a function
> >e.g. forward-sentence-default-function.
> 
> Is this practice used anywhere else? Iirc
> forward-sexp-function doesn't follow that
> practice.

Exactly.  IMO, if the variable can have a
nonfunction value, especially nil, then
there's no need (nothing gained, and even
possible confusion/misunderstanding added)
by adding "-default-" to the name.

On the other hand, if the value must always
be a function, then having "-default-" in
the name makes sense.

Following this guideline helps users know
(guess) what the variable's value can be
even without consulting its doc.

bug#60623: 30.0.50; Add forward-sentence with tree sitter support

[Theodor Thornhill via Bug reports for GNU Emacs, the Swiss army knife of text editors]

On 8 January 2023 17:41:05 CET, Drew Adams <drew.adams@oracle.com> wrote:
>> >A good practice is to name such a function
>> >e.g. forward-sentence-default-function.
>> 
>> Is this practice used anywhere else? Iirc
>> forward-sexp-function doesn't follow that
>> practice.
>
>Exactly.  IMO, if the variable can have a
>nonfunction value, especially nil, then
>there's no need (nothing gained, and even
>possible confusion/misunderstanding added)
>by adding "-default-" to the name.
>
>On the other hand, if the value must always
>be a function, then having "-default-" in
>the name makes sense.
>
>Following this guideline helps users know
>(guess) what the variable's value can be
>even without consulting its doc.

So is this to be considered an improvement to forward-sexp too, then?

bug#60623: 30.0.50; Add forward-sentence with tree sitter support

[Juri Linkov]

>>> >A good practice is to name such a function
>>> >e.g. forward-sentence-default-function.
>>>
>>> Is this practice used anywhere else? Iirc
>>> forward-sexp-function doesn't follow that
>>> practice.
>>
>>On the other hand, if the value must always
>>be a function, then having "-default-" in
>>the name makes sense.
>
> So is this to be considered an improvement to forward-sexp too, then?

Sorry, I can't find where a lambda is set to forward-sexp-function.
I only see this:

  (defvar forward-sexp-function nil

But if it will be set to a function later, it would be nice
to define a default function as well.

bug#60623: 30.0.50; Add forward-sentence with tree sitter support

[Theodor Thornhill via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Juri Linkov <juri@linkov.net> writes:

>>>> >A good practice is to name such a function
>>>> >e.g. forward-sentence-default-function.
>>>>
>>>> Is this practice used anywhere else? Iirc
>>>> forward-sexp-function doesn't follow that
>>>> practice.
>>>
>>>On the other hand, if the value must always
>>>be a function, then having "-default-" in
>>>the name makes sense.
>>
>> So is this to be considered an improvement to forward-sexp too, then?
>
> Sorry, I can't find where a lambda is set to forward-sexp-function.
> I only see this:
>
>   (defvar forward-sexp-function nil
>
> But if it will be set to a function later, it would be nice
> to define a default function as well.

I meant the way we did with 'transpose-sexps', where there now is a
'transpose-sexps-function' variable containing the factored-out earlier
implementation.  And by if "this is an improvement" I meant declaring a
specific defun as the default value for the defvar in question.  Maybe I
should add the same change which is now developing here there too.

What do you think?

Theo

bug#60623: 30.0.50; Add forward-sentence with tree sitter support

[Juri Linkov]

>>> So is this to be considered an improvement to forward-sexp too, then?
>>
>> Sorry, I can't find where a lambda is set to forward-sexp-function.
>> I only see this:
>>
>>   (defvar forward-sexp-function nil
>>
>> But if it will be set to a function later, it would be nice
>> to define a default function as well.
>
> I meant the way we did with 'transpose-sexps', where there now is a
> 'transpose-sexps-function' variable containing the factored-out earlier
> implementation.  And by if "this is an improvement" I meant declaring a
> specific defun as the default value for the defvar in question.  Maybe I
> should add the same change which is now developing here there too.
>
> What do you think?

I agree that 'transpose-sexps-function' could benefit from the same improvement.

bug#60623: 30.0.50; Add forward-sentence with tree sitter support

[Theodor Thornhill via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Juri Linkov <juri@linkov.net> writes:

>>>> So is this to be considered an improvement to forward-sexp too, then?
>>>
>>> Sorry, I can't find where a lambda is set to forward-sexp-function.
>>> I only see this:
>>>
>>>   (defvar forward-sexp-function nil
>>>
>>> But if it will be set to a function later, it would be nice
>>> to define a default function as well.
>>
>> I meant the way we did with 'transpose-sexps', where there now is a
>> 'transpose-sexps-function' variable containing the factored-out earlier
>> implementation.  And by if "this is an improvement" I meant declaring a
>> specific defun as the default value for the defvar in question.  Maybe I
>> should add the same change which is now developing here there too.
>>
>> What do you think?
>
> I agree that 'transpose-sexps-function' could benefit from the same improvement.

Yeah, added such a change in bug#60654, as it was already reported :-)

I appreciate your feedback!

Theo

bug#60623: 30.0.50; Add forward-sentence with tree sitter support

[Drew Adams]

> So is this to be considered an improvement to forward-sexp too, then?

I don't understand the question.  What is "this"?

Variable `forward-sexp-function' already has an
appropriate name, IMO, since its value can be nil
(not a function).

bug#60623: 30.0.50; Add forward-sentence with tree sitter support

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Exactly.  IMO, if the variable can have a
> nonfunction value, especially nil, then

Bad idea.  That precludes changing the value of the variable with
`add-function`, whereas `add-function` is often the best way for
a major/minor mode to change that variable (actually, the "only" way to
do it with some hope that it will interact correctly with other modes
that may change it as well).

> there's no need (nothing gained, and even possible
> confusion/misunderstanding added) by adding "-default-" to the name.

Instead we should strive to make the name of the default function simply
irrelevant.  If all goes well, nobody should ever need to explicitly
call "the default value" of that variable.  Instead the function they
added via `add-function` will receive the "previous" value as argument.

> On the other hand, if the value must always be a function, then having
> "-default-" in the name makes sense.

Agreed.  It's also helpful when you look at the var's value, it lets you
know that it hasn't been modified.


        Stefan

bug#60623: 30.0.50; Add forward-sentence with tree sitter support

[Drew Adams]

> > Exactly.  IMO, if the variable can have
                   ^^
> > a nonfunction value, especially nil, then
> 
> Bad idea.

I know you think so. ;-)

> That precludes changing the value of the
> variable with `add-function`,

Yes, it does, at least blindly and ignoring
its current value.  And that's _appropriate_
IF the var can have a non-function value.

But even in that case the var value can be
tested to see if it's a function, and if/when
so, advising it can make sense.

> whereas `add-function` is often the best way for
> a major/minor mode to change that variable (actually, the "only" way to
> do it with some hope that it will interact correctly with other modes
> that may change it as well).

A legitimate argument.  But it doesn't apply to
a variable already defined so that it "can have
a nonfunction value."  Did you perhaps miss that
"if"?

I also don't agree that that (_good_) reason you
give is all-deciding.

I'd say that _other things being equal_, yes,
you can take advantage of that good reason, IF
the variable's value can be ensured to always be
a function, or sometimes even if it just is
currently a function.

IOW, you give one (good) reason for one (good)
practice, which, yes, can sometimes make sense.

> > there's no need (nothing gained, and even
> > possible confusion/misunderstanding added) by
> > adding "-default-" to the name.
> >
> > On the other hand, if the value must always
> > be a function, then having "-default-" in the
> > name makes sense.
> 
> Agreed.  It's also helpful when you look at the var's
> value, it lets you know that it hasn't been modified.