bug#60856: 29.0.60; C-M-f in python-ts-mode does not work properly with docstrings

bug#60856: 29.0.60; C-M-f in python-ts-mode does not work properly with docstrings

[Apostolis Bessas]

In python-ts-mode executing C-M-f (forward-sexp) when at the beginning
of a docstring does not go to the end of the docstring (as happens in
python-mode), but at the end of the file.

However, if I switch to python-mode and then back to python-ts-mode,
C-M-f works correctly. See the following example:

- Start emacs -Q
- Execute:

(add-to-list 'auto-mode-alist '("\\.py\\'" . python-ts-mode))

- Create the file test.py with the contents:

"""Some docstring.

The docstring has multiple lines.
"""

x = 1

- Go to the first character in the file (beginning of docstring) and
  press C-M-f. The point is moved to the end of the file (after the line
  `x = 1`). Instead, the point should have been moved to the end of the
  docstring (line 4, last '"' character).

- Switch to python-mode and then back to python-ts-mode and repeat the
  above step. The point is moved to the end of the docstring.

In GNU Emacs 29.0.60 (build 1, x86_64-pc-linux-gnu, GTK+ Version
 3.24.36, cairo version 1.17.6) of 2023-01-16 built on galileo
Repository revision: 352e41016bcaab8566347091b6b61908a9b57b57
Repository branch: emacs-29
System Description: Arch Linux

Configured using:
 'configure --with-pgtk --with-json --with-native-compilation
 --with-native-compilation=aot --with-tree-sitter --with-imagemagick
 --with-xwidgets'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ
IMAGEMAGICK JPEG JSON LCMS2 LIBOTF LIBSYSTEMD LIBXML2 MODULES
NATIVE_COMP NOTIFY INOTIFY PDUMPER PGTK PNG RSVG SECCOMP SOUND SQLITE3
THREADS TIFF TOOLKIT_SCROLL_BARS TREE_SITTER WEBP XIM XWIDGETS GTK3 ZLIB

Important settings:
  value of $LC_MONETARY: el_GR.UTF-8
  value of $LC_NUMERIC: el_GR.UTF-8
  value of $LC_TIME: el_GR.UTF-8
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  yas-minor-mode: t
  hl-todo-mode: t
  flyspell-mode: t
  TeX-PDF-mode: t
  server-mode: t
  global-company-mode: t
  company-mode: t
  editorconfig-mode: t
  csv-field-index-mode: t
  global-auto-revert-mode: t
  consult-notes-denote-mode: t
  recentf-mode: t
  override-global-mode: t
  marginalia-mode: t
  savehist-mode: t
  vertico-mode: t
  whole-line-or-region-global-mode: t
  whole-line-or-region-local-mode: t
  shell-dirtrack-mode: t
  global-so-long-mode: t
  global-flycheck-mode: t
  flycheck-mode: t
  save-place-mode: t
  pixel-scroll-precision-mode: t
  apheleia-global-mode: t
  apheleia-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tab-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
/home/apos/.emacs.d/elpa/transient-20221202.1727/transient hides /usr/local/share/emacs/29.0.60/lisp/transient

Features:
(shadow cl-print mailalias textsec uni-scripts idna-mapping
ucs-normalize uni-confusable textsec-check mail-extr misearch
multi-isearch magit-base magit-section compat-27 compat-26 shortdoc
help-fns radix-tree oc-basic ol-eww ol-rmail ol-mhe ol-irc ol-info
ol-gnus nnselect gnus-art mm-uu mml2015 gnus-sum gnus-group gnus-undo
gnus-start gnus-dbus gnus-cloud nnimap nnmail mail-source utf7 nnoo
gnus-spec gnus-int gnus-range gnus-win ol-docview doc-view ol-bibtex
bibtex ol-bbdb ol-w3m ol-doi org-link-doi vc-svn vc-git emacsbug
mule-util company-oddmuse company-keywords company-etags company-gtags
company-files company-clang company-capf company-cmake company-semantic
company-template company-bbdb yasnippet-snippets yasnippet hl-todo
flyspell clipetty tex dbus texmathp pdf-tools pdf-view jka-compr
pdf-cache pdf-info tq pdf-util pdf-macs image-mode exif pass
password-store auth-source-pass with-editor server company-terraform
company-terraform-data terraform-mode hcl-mode flycheck-ledger
ledger-mode ledger-check ledger-texi ledger-test ledger-sort
ledger-report ledger-reconcile ledger-occur ledger-fonts ledger-fontify
ledger-state ledger-complete ledger-schedule ledger-init ledger-xact
ledger-post ledger-exec ledger-navigate eshell esh-cmd esh-ext esh-opt
esh-proc esh-io esh-arg esh-module esh-groups esh-util ledger-context
ledger-commodities ledger-regex gnus-alias ol-notmuch notmuch
notmuch-tree notmuch-jump notmuch-hello notmuch-show notmuch-print
notmuch-crypto notmuch-mua notmuch-message notmuch-draft
notmuch-maildir-fcc notmuch-address notmuch-company notmuch-parser
notmuch-wash diff-mode coolj goto-addr icalendar diary-lib
diary-loaddefs notmuch-tag crm notmuch-lib notmuch-compat hl-line
mm-view mml-smime smime dig org-mime message sendmail yank-media rfc822
mml mml-sec epa derived epg rfc6068 epg-config mm-decode mm-bodies
mm-encode mailabbrev gmm-utils mailheader ox-org org-web-tools
esxml-query org-make-toc org-tree-slide org-timer org-clock
ob-restclient restclient ox-md ox-odt rng-loc rng-uri rng-parse
rng-match rng-dt rng-util rng-pttrn nxml-parse nxml-ns nxml-enc xmltok
nxml-util ox-latex ox-icalendar org-agenda ox-html table ox-ascii
ox-publish ox org-attach org-superstar org-element org-persist org-id
org-refile avl-tree ob-plantuml ob-js ob-ruby ob-python python ob-sql
ob-shell org ob ob-tangle ob-ref ob-lob ob-table ob-exp org-macro
org-src ob-comint org-pcomplete org-list org-footnote org-faces
org-entities ob-emacs-lisp ob-core ob-eval org-cycle org-table ol
org-fold org-fold-core org-keys oc org-loaddefs org-version org-compat
org-macs rego-mode reformatter jsonnet-mode msgpack yaml-mode
markdown-mode noutline outline alchemist alchemist-macroexpand
alchemist-company alchemist-help alchemist-complete company-dabbrev-code
company-dabbrev alchemist-refcard alchemist-phoenix alchemist-compile
alchemist-iex alchemist-message alchemist-hooks alchemist-hex
alchemist-mix alchemist-info alchemist-goto alchemist-scope
alchemist-eval alchemist-interact alchemist-server alchemist-execute
alchemist-report alchemist-test-mode alchemist-project alchemist-file
alchemist-key alchemist-utils elixir-mode elixir-format elixir-smie
rust-utils rust-mode rust-rustfmt rust-playpen rust-compile rust-cargo
go-mode find-file ffap etags fileloop generator ruby-electric ruby-mode
nginx-mode css-mode sgml-mode facemenu eww url-queue mm-url gnus
nnheader gnus-util mail-utils range typescript-mode python-info
python-docstring company pcase devdocs shr pixel-fill kinsoku url-file
svg dom apheleia plantuml-mode xml iedit iedit-lib protobuf-mode cc-mode
cc-fonts cc-guess cc-menus cc-cmds cc-styles cc-align cc-engine cc-vars
cc-defs graphviz-dot-mode dockerfile-mode sh-script smie treesit
executable autoinsert deadgrep spinner rg vc vc-dispatcher rg-info-hack
advice rg-menu transient rg-ibuffer rg-result wgrep-rg wgrep rg-history
rg-header ibuf-ext ibuffer ibuffer-loaddefs grep comp comp-cstr warnings
editorconfig editorconfig-core editorconfig-core-handle
editorconfig-fnmatch git-link thingatpt ghub-graphql treepy gsexp ghub
url-http mail-parse rfc2231 rfc2047 rfc2045 mm-util ietf-drums
mail-prsvr url-gw nsm url-auth let-alist gnutls puny cal-menu calendar
cal-loaddefs vterm face-remap compile color vterm-module term/xterm
xterm eterm-256color f f-shortdoc s term disp-table ehelp xterm-color
ispell imenu-list hideshow imenu csv-mode sort htmlize writeroom-mode
typo visual-fill-column autorevert filenotify all-the-icons
all-the-icons-faces data-material data-weathericons data-octicons
data-fileicons data-faicons data-alltheicons modus-operandi-tinted-theme
modus-themes consult-notes-denote denote xdg dired dired-loaddefs xref
project consult-notes consult-vertico consult compat-28 recentf
tree-widget bookmark text-property-search edmacro kmacro
use-package-bind-key bind-key marginalia orderless savehist vertico
whole-line-or-region tramp tramp-loaddefs trampver tramp-integration
cus-edit pp cus-load wid-edit files-x tramp-compat shell pcomplete
comint ansi-osc parse-time iso8601 time-date ls-lisp format-spec so-long
flycheck ansi-color find-func dash use-package-diminish diminish
exec-path-from-shell saveplace use-package-ensure no-littering compat
compat-macs use-package-core pixel-scroll cua-base ring cl-extra
help-mode icons finder-inf org-mime-autoloads gcmh-autoloads
org-roam-autoloads rust-mode-autoloads go-mode-autoloads
web-mode-autoloads apheleia-autoloads easy-mmode deadgrep-autoloads
editorconfig-autoloads modus-themes-autoloads consult-notes-autoloads
consult-autoloads marginalia-autoloads orderless-autoloads
vertico-autoloads no-littering-autoloads clipetty-autoloads
auctex-autoloads tex-site pdf-tools-autoloads tablist-autoloads
pass-autoloads password-store-otp-autoloads password-store-autoloads
elfeed-autoloads company-terraform-autoloads terraform-mode-autoloads
hcl-mode-autoloads flycheck-ledger-autoloads ledger-mode-autoloads
gnus-alias-autoloads ol-notmuch-autoloads notmuch-autoloads
denote-autoloads org-web-tools-autoloads esxml-autoloads kv-autoloads
request-autoloads org-make-toc-autoloads org-tree-slide-autoloads
ob-restclient-autoloads restclient-autoloads org-superstar-autoloads
ox-gfm-autoloads rego-mode-autoloads dhall-mode-autoloads
reformatter-autoloads jsonnet-mode-autoloads msgpack-autoloads
yaml-mode-autoloads alchemist-autoloads elixir-mode-autoloads
ruby-electric-autoloads nginx-mode-autoloads json-mode-autoloads rx
json-snatcher-autoloads typescript-mode-autoloads rjsx-mode-autoloads
js2-mode-autoloads python-info-autoloads sphinx-doc-autoloads
python-docstring-autoloads py-isort-autoloads company-autoloads
devdocs-autoloads plantuml-mode-autoloads iedit-autoloads
protobuf-mode-autoloads graphviz-dot-mode-autoloads
dockerfile-mode-autoloads spinner-autoloads rg-autoloads wgrep-autoloads
hl-todo-autoloads format-all-autoloads inheritenv-autoloads
language-id-autoloads yasnippet-snippets-autoloads yasnippet-autoloads
git-link-autoloads forge-autoloads markdown-mode-autoloads
closql-autoloads emacsql-sqlite-autoloads emacsql-autoloads
yaml-autoloads ghub-autoloads treepy-autoloads magit-autoloads
git-commit-autoloads magit-section-autoloads transient-autoloads
with-editor-autoloads vterm-autoloads eterm-256color-autoloads
f-autoloads xterm-color-autoloads imenu-list-autoloads
csv-mode-autoloads htmlize-autoloads writeroom-mode-autoloads
typo-autoloads visual-fill-column-autoloads all-the-icons-autoloads
compat-autoloads amx-autoloads s-autoloads
whole-line-or-region-autoloads flycheck-autoloads info dash-autoloads
pkg-info-autoloads epl-autoloads diminish-autoloads
exec-path-from-shell-autoloads package browse-url url url-proxy
url-privacy url-expand url-methods url-history url-cookie
generate-lisp-file url-domsuf url-util mailcap url-handlers url-parse
auth-source cl-seq eieio eieio-core cl-macs password-cache json subr-x
map byte-opt gv bytecomp byte-compile url-vars cl-loaddefs cl-lib rmc
iso-transl tooltip cconv eldoc paren electric uniquify ediff-hook
vc-hooks lisp-float-type elisp-mode mwheel term/pgtk-win pgtk-win
term/common-win pgtk-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode lisp-mode prog-mode register
page tab-bar menu-bar rfn-eshadow isearch easymenu timer select
scroll-bar mouse jit-lock font-lock syntax font-core term/tty-colors
frame minibuffer nadvice seq simple cl-generic indonesian philippine
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite emoji-zwj charscript
charprop case-table epa-hook jka-cmpr-hook help abbrev obarray oclosure
cl-preloaded button loaddefs theme-loaddefs faces cus-face macroexp
files window text-properties overlay sha1 md5 base64 format env
code-pages mule custom widget keymap hashtable-print-readable backquote
threads xwidget-internal dbusbind inotify dynamic-setting
system-font-setting font-render-setting cairo gtk pgtk lcms2 multi-tty
make-network-process native-compile emacs)

Memory information:
((conses 16 1106012 278003)
 (symbols 48 59332 151)
 (strings 32 310702 84951)
 (string-bytes 1 10724974)
 (vectors 16 132391)
 (vector-slots 8 3190054 557840)
 (floats 8 1140 1213)
 (intervals 56 1645 1742)
 (buffers 984 21))

bug#60856: 29.0.60; C-M-f in python-ts-mode does not work properly with docstrings

[Theodor Thornhill via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Hi there!

Apostolis Bessas <apostolis@bessas.me> writes:

> In python-ts-mode executing C-M-f (forward-sexp) when at the beginning
> of a docstring does not go to the end of the docstring (as happens in
> python-mode), but at the end of the file.
>
> However, if I switch to python-mode and then back to python-ts-mode,
> C-M-f works correctly. See the following example:
>
> - Start emacs -Q
> - Execute:
>
> (add-to-list 'auto-mode-alist '("\\.py\\'" . python-ts-mode))
>
> - Create the file test.py with the contents:
>
> """Some docstring.
>
> The docstring has multiple lines.
> """
>
> x = 1
>
> - Go to the first character in the file (beginning of docstring) and
>   press C-M-f. The point is moved to the end of the file (after the line
>   `x = 1`). Instead, the point should have been moved to the end of the
>   docstring (line 4, last '"' character).
>
> - Switch to python-mode and then back to python-ts-mode and repeat the
>   above step. The point is moved to the end of the docstring.
>

Yeah, I can reproduce this as well.  What happens is that python.el
redefines and remaps lots of commands, and doesn't set the functions.
In addition it uses some internal stuff that if I'm not mistaken isn't
really enabled in python-ts-mode.  If you try to apply the supplied
patch, then execute these steps:

1. M-< ;; go to beginning of buffer
2. M-e
3. M-<
4. M-x forward-sentence

You should see the behavior diverges, right?  That's because the
'treesit-major-mode-setup' sets up 'forward-sentence-function' which
python doesn't set.  A fix could be either to set the functions rather
than remapping, or make sure we at least create equivalents for
python-ts-mode.  I think we inherit too much from the python-base-mode,
so it is a little hard to reason about where functionalities come from.
I added the maintainers of treesit.el and python.el to CC for them to
provide some input.

Theo


diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index 21d16db287..973e405e8f 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -6710,6 +6710,10 @@ python-ts-mode
                                               "_definition"))
     (setq-local treesit-defun-name-function
                 #'python--treesit-defun-name)
+
+    (setq-local treesit-sentence-type-regexp
+                '"statement")
+
     (treesit-major-mode-setup)
 
     (when python-indent-guess-indent-offset

bug#60856: 29.0.60; C-M-f in python-ts-mode does not work properly with docstrings

[Theodor Thornhill via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[-- Attachment #1: Type: text/plain, Size: 718 bytes --]

Apostolis Bessas <apostolis@bessas.me> writes:

> In python-ts-mode executing C-M-f (forward-sexp) when at the beginning
> of a docstring does not go to the end of the docstring (as happens in
> python-mode), but at the end of the file.
>
> However, if I switch to python-mode and then back to python-ts-mode,
> C-M-f works correctly. See the following example:
>
> - Start emacs -Q
> - Execute:
>
> (add-to-list 'auto-mode-alist '("\\.py\\'" . python-ts-mode))
>
> - Create the file test.py with the contents:
>
> """Some docstring.
>
> The docstring has multiple lines.
> """
>
> x = 1
>

Can you test this patch?  More nodes are needed, but this is a start, I
think.  Must be applied to the master branch :)

Theo


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Add-sentence-and-sexp-movement-to-python-ts-mode.patch --]
[-- Type: text/x-patch, Size: 1293 bytes --]

From bec1839da51e1fe6c399e636d1cf476fe76682c9 Mon Sep 17 00:00:00 2001
From: Theodor Thornhill <theo@thornhill.no>
Date: Sun, 22 Jan 2023 11:27:37 +0100
Subject: [PATCH] Add sentence and sexp movement to python-ts-mode

* lisp/progmodes/python.el (python-ts-mode): Add relevant node types.
---
 lisp/progmodes/python.el | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index 21d16db287..2a8fb5fcf5 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -6710,6 +6710,20 @@ python-ts-mode
                                               "_definition"))
     (setq-local treesit-defun-name-function
                 #'python--treesit-defun-name)
+
+    (setq-local treesit-sentence-type-regexp
+                (regexp-opt '("statement"
+                              "clause")))
+
+    (setq-local treesit-sexp-type-regexp
+                (regexp-opt '("expression"
+                              "string"
+                              "call"
+                              "operator"
+                              "identifier"
+                              "integer"
+                              "float")))
+
     (treesit-major-mode-setup)
 
     (when python-indent-guess-indent-offset
-- 
2.34.1

bug#60856: 29.0.60; C-M-f in python-ts-mode does not work properly with docstrings

[Apostolis Bessas]

Hello,

On Sun, Jan 22, 2023 at 11:30, Theodor Thornhill <theo@thornhill.no> wrote:
> Can you test this patch?  More nodes are needed, but this is a start, I
> think.  Must be applied to the master branch :)

Yup, the patch fixes the issue for me, when applied specifically to the master branch.

Thanks,
Apostolis

bug#60856: 29.0.60; C-M-f in python-ts-mode does not work properly with docstrings

[Stefan Kangas]

Apostolis Bessas <apostolis@bessas.me> writes:

> On Sun, Jan 22, 2023 at 11:30, Theodor Thornhill <theo@thornhill.no> wrote:
>> Can you test this patch?  More nodes are needed, but this is a start, I
>> think.  Must be applied to the master branch :)
>
> Yup, the patch fixes the issue for me, when applied specifically to the master branch.

It seems like this patch was never installed?

bug#60856: 29.0.60; C-M-f in python-ts-mode does not work properly with docstrings

[Stefan Kangas]

Version: 31.1

Stefan Kangas <stefankangas@gmail.com> writes:

> Apostolis Bessas <apostolis@bessas.me> writes:
>
>> On Sun, Jan 22, 2023 at 11:30, Theodor Thornhill <theo@thornhill.no> wrote:
>>> Can you test this patch?  More nodes are needed, but this is a start, I
>>> think.  Must be applied to the master branch :)
>>
>> Yup, the patch fixes the issue for me, when applied specifically to the master
>> branch.
>
> It seems like this patch was never installed?

Now installed on master (commit c9e30e8c77d).  Closing.