bug#37452: 26.3; skip-syntax-forward doesn't work in python-mode

bug#37452: 26.3; skip-syntax-forward doesn't work in python-mode

[Amai Kinono]

[-- Attachment #1: Type: text/plain, Size: 8913 bytes --]

Here's how to reproduce:

- Run `emacs -Q`

- Define a command like this and eval it

  ``` emacs-lisp
  (defun skip-syntax ()
    (interactive)
    (skip-syntax-forward (char-to-string (char-syntax (char-after)))))
  ```

  It gets the syntax of the next char and skip chars with the same syntax.

- M-x python-mode, clear the buffer, type

  ``` python
  print(|'hello')
  ```

  `|` stands for the cursor position.

- Call skip-syntax. The expected behavior is:

  ``` python
  print('|hello')
  ```

  but actually the cursor doesn't move.


I've tried some other modes, and the above command can skip the
quotation mark normally, so this should be a problem of Python mode.

In GNU Emacs 26.3 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.10)
 of 2019-08-30 built on juergen
Windowing system distributor 'The X.Org Foundation', version 11.0.12005000
System Description: Manjaro Linux

Recent messages:
uncompressing python.el.gz...
uncompressing python.el.gz...done
Note: file is write protected
Quit [2 times]
M-SPC r is undefined

Configured using:
 'configure --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib
 --localstatedir=/var --with-x-toolkit=gtk3 --with-xft --with-modules
 'CFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fno-plt'
 CPPFLAGS=-D_FORTIFY_SOURCE=2
 LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now'

Configured features:
XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GSETTINGS GLIB
NOTIFY ACL GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB
TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM MODULES THREADS LIBSYSTEMD LCMS2

Important settings:
  value of $LC_MONETARY: zh_CN.UTF-8
  value of $LC_NUMERIC: zh_CN.UTF-8
  value of $LC_TIME: zh_CN.UTF-8
  value of $LANG: zh_CN.utf8
  value of $XMODIFIERS: @im=fcitx
  locale-coding-system: utf-8-unix

Major mode: Emacs-Lisp

Minor modes in effect:
  smartparens-global-mode: t
  smartparens-strict-mode: t
  smartparens-mode: t
  diff-auto-refine-mode: t
  which-key-mode: t
  global-whitespace-cleanup-mode: t
  whitespace-cleanup-mode: t
  global-whitespace-mode: t
  global-company-mode: t
  company-mode: t
  show-paren-mode: t
  winum-mode: t
  awesome-tab-mode: t
  ivy-rich-mode: t
  evil-escape-mode: t
  global-undo-tree-mode: t
  undo-tree-mode: t
  shell-dirtrack-mode: t
  evil-mode: t
  evil-local-mode: t
  global-subword-mode: t
  subword-mode: t
  global-auto-revert-mode: t
  awesome-tray-mode: t
  general-override-mode: t
  ivy-mode: t
  delete-selection-mode: t
  straight-use-package-mode: t
  straight-package-neutering-mode: t
  straight-live-modifications-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
/home/kino/.emacs.d/straight/build/seq/seq hides
/usr/share/emacs/26.3/lisp/emacs-lisp/seq
/home/kino/.emacs.d/straight/build/let-alist/let-alist hides
/usr/share/emacs/26.3/lisp/emacs-lisp/let-alist
/home/kino/.emacs.d/straight/build/cl-lib/cl-lib hides
/usr/share/emacs/26.3/lisp/emacs-lisp/cl-lib
~/.emacs.d/modules/ui/appearance/config hides
~/.emacs.d/modules/ui/file/config
~/.emacs.d/modules/ui/appearance/config hides
~/.emacs.d/modules/ui/edit/config
~/.emacs.d/modules/ui/appearance/config hides
~/.emacs.d/modules/ui/evil/config
~/.emacs.d/modules/ui/appearance/config hides
~/.emacs.d/modules/ui/sexp/config
~/.emacs.d/modules/ui/appearance/config hides
~/.emacs.d/modules/ui/window-buffer/config
~/.emacs.d/modules/ui/appearance/config hides
~/.emacs.d/modules/ide/completion/config
~/.emacs.d/modules/ui/appearance/config hides
~/.emacs.d/modules/ide/checker/config
~/.emacs.d/modules/ui/appearance/config hides
~/.emacs.d/modules/ide/lsp/config
~/.emacs.d/modules/ui/appearance/config hides
~/.emacs.d/modules/ide/help/config
~/.emacs.d/modules/ui/appearance/config hides
~/.emacs.d/modules/ide/ctags/config
~/.emacs.d/modules/ui/appearance/config hides
~/.emacs.d/modules/lang/dart/config
~/.emacs.d/modules/ui/appearance/config hides
~/.emacs.d/modules/lang/web/config
~/.emacs.d/modules/ui/appearance/config hides
~/.emacs.d/modules/lang/elisp/config
~/.emacs.d/modules/ui/appearance/config hides
~/.emacs.d/modules/app/qrcode/config
~/.emacs.d/modules/ui/appearance/config hides
~/.emacs.d/modules/app/esup/config

Features:
(shadow sort mail-extr emacsbug message rmc puny format-spec rfc822 mml
mml-sec password-cache epa derived epg epg-config gnus-util rmail
rmail-loaddefs mm-decode mm-bodies mm-encode mail-parse rfc2231
mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums
mm-util mail-prsvr mail-utils two-column smartparens-config
smartparens-text smartparens elec-pair vc-mtn vc-hg vc-git diff-mode
vc-bzr vc-src vc-sccs vc-svn vc-cvs vc-rcs vc vc-dispatcher jka-compr
cursor-sensor which-key whitespace-cleanup-mode disp-table whitespace
company-oddmuse company-keywords company-etags company-gtags
company-dabbrev-code company-dabbrev company-files company-capf
company-cmake company-xcode company-clang company-semantic company-eclim
company-template company-bbdb company pcase flycheck json map find-func
rx seq seq-25 paren winum dash awesome-tab easy-mmode which-func imenu
counsel-etags ivy-rich counsel xdg dired dired-loaddefs compile swiper
etags xref project eieio eieio-core eieio-loaddefs evil-escape evil
evil-integration undo-tree diff evil-maps evil-commands reveal flyspell
ispell evil-jumps evil-command-window evil-types evil-search evil-ex
shell pcomplete comint ansi-color evil-macros evil-repeat evil-states
evil-core byte-opt advice evil-common windmove thingatpt rect
evil-digraphs evil-vars toki-app/esup esup-autoloads toki-app/qrcode
toki-lang/elisp toki-lang/web web-mode-autoloads toki-lang/dart
dart-mode-autoloads toki-ide/ctags counsel-etags-autoloads toki-ide/help
toki-ide/lsp lsp-mode-autoloads markdown-mode-autoloads
spinner-autoloads ht-autoloads f-autoloads s-autoloads
dash-functional-autoloads toki-ide/checker flycheck-autoloads
let-alist-autoloads pkg-info-autoloads epl-autoloads toki-ide/completion
company-autoloads toki-ui/window-buffer awesome-tab-autoloads
winum-autoloads toki-ui/sexp smartparens-autoloads dash-autoloads
toki-ui/evil term-cursor-autoloads evil-escape-autoloads evil-autoloads
goto-chg-autoloads toki-ui/edit whitespace-cleanup-mode-autoloads
avy-autoloads cap-words superword subword undo-tree-autoloads
toki-ui/file autorevert filenotify toki-ui/appearance beacon-autoloads
seq-autoloads awesome-tray awesome-tray-autoloads all-the-icons
all-the-icons-faces data-material data-weathericons data-octicons
data-fileicons data-faicons data-alltheicons memoize
all-the-icons-autoloads memoize-autoloads doom-gruvbox-theme
doom-themes-ext-org doom-themes doom-themes-base doom-themes-autoloads
ns-auto-titlebar-autoloads cl toki-core toki-core-keybind hydra lv
hydra-autoloads lv-autoloads which-key-autoloads edmacro kmacro general
general-autoloads toki-core-os exec-path-from-shell-autoloads
osx-clipboard-autoloads xclip-autoloads toki-core-packages
ivy-rich-autoloads counsel-autoloads swiper-autoloads ivy delsel ring
colir color ivy-overlay ivy-autoloads no-littering
no-littering-autoloads cl-lib-autoloads cl-seq use-package-core
use-package-autoloads bind-key-autoloads straight-autoloads info
cl-extra help-mode easymenu straight subr-x bytecomp byte-compile cconv
toki-core-defaults toki-core-helpers cl-macs gv cl-loaddefs cl-lib
time-date mule-util china-util tooltip eldoc electric uniquify
ediff-hook vc-hooks lisp-float-type mwheel term/x-win x-win
term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page menu-bar rfn-eshadow isearch timer select
scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame cl-generic cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932
hebrew greek romanian slovak czech european ethiopic indian cyrillic
chinese composite charscript charprop case-table epa-hook jka-cmpr-hook
help simple abbrev obarray minibuffer cl-preloaded nadvice loaddefs
button faces cus-face macroexp files text-properties overlay sha1 md5
base64 format env code-pages mule custom widget hashtable-print-readable
backquote threads dbusbind inotify lcms2 dynamic-setting
system-font-setting font-render-setting move-toolbar gtk x-toolkit x
multi-tty make-network-process emacs)

Memory information:
((conses 16 324283 285642)
 (symbols 48 36062 15)
 (miscs 40 111 410)
 (strings 32 82193 35731)
 (string-bytes 1 2537074)
 (vectors 16 34096)
 (vector-slots 8 746877 160936)
 (floats 8 768 1220)
 (intervals 56 3869 1417)
 (buffers 992 14))

[-- Attachment #2: Type: text/html, Size: 9706 bytes --]

bug#37452: 26.3; skip-syntax-forward doesn't work in python-mode

[Noam Postavsky]

Amai Kinono <amaikinono@gmail.com> writes:

>   (defun skip-syntax ()
>     (interactive)
>     (skip-syntax-forward (char-to-string (char-syntax (char-after)))))

If you look at char-syntax's docstring, you should see

    If you’re trying to determine the syntax of characters in the buffer,
    this is probably the wrong function to use, because it can’t take
    ‘syntax-table’ text properties into account.  Consider using
    ‘syntax-after’ instead.

Which is exactly the situation you've encountered here: python-mode sets
the syntax of the quote to string fence ("|") using a text property.
This version of skip-syntax should work:

(defun syntax-class-to-char (syntax-class)
  (aref " .w_()'\"$\\/<>@!|" syntax-class))

(defun skip-syntax ()
  (interactive)
  (skip-syntax-forward (char-to-string
                        (syntax-class-to-char
                         (syntax-class (syntax-after (point)))))))

Although I'd say that the function syntax-class-to-char should be added
to Emacs (in syntax.c, so that it can reuse syntax_code_spec).

bug#37452: Fwd: bug#37452: 26.3; skip-syntax-forward doesn't work in python-mode

[Amai Kinono]

[-- Attachment #1: Type: text/plain, Size: 1641 bytes --]

---------- Forwarded message ---------
发件人： Amai Kinono <amaikinono@gmail.com>
Date: 2019年9月20日周五 下午9:39
Subject: Re: bug#37452: 26.3; skip-syntax-forward doesn't work in
python-mode
To: Noam Postavsky <npostavs@gmail.com>


I just found that the `forward-same-syntax` command in subr.el did the same
thing, so it doesn't work in my example too. Maybe this should be fixed?

Noam Postavsky <npostavs@gmail.com> 于2019年9月19日周四 上午6:37写道：

> Amai Kinono <amaikinono@gmail.com> writes:
>
> >   (defun skip-syntax ()
> >     (interactive)
> >     (skip-syntax-forward (char-to-string (char-syntax (char-after)))))
>
> If you look at char-syntax's docstring, you should see
>
>     If you’re trying to determine the syntax of characters in the buffer,
>     this is probably the wrong function to use, because it can’t take
>     ‘syntax-table’ text properties into account.  Consider using
>     ‘syntax-after’ instead.
>
> Which is exactly the situation you've encountered here: python-mode sets
> the syntax of the quote to string fence ("|") using a text property.
> This version of skip-syntax should work:
>
> (defun syntax-class-to-char (syntax-class)
>   (aref " .w_()'\"$\\/<>@!|" syntax-class))
>
> (defun skip-syntax ()
>   (interactive)
>   (skip-syntax-forward (char-to-string
>                         (syntax-class-to-char
>                          (syntax-class (syntax-after (point)))))))
>
> Although I'd say that the function syntax-class-to-char should be added
> to Emacs (in syntax.c, so that it can reuse syntax_code_spec).
>

[-- Attachment #2: Type: text/html, Size: 2443 bytes --]

bug#37452: 26.3; skip-syntax-forward doesn't work in python-mode

[Lars Ingebrigtsen]

Noam Postavsky <npostavs@gmail.com> writes:

> (defun syntax-class-to-char (syntax-class)
>   (aref " .w_()'\"$\\/<>@!|" syntax-class))
>
> (defun skip-syntax ()
>   (interactive)
>   (skip-syntax-forward (char-to-string
>                         (syntax-class-to-char
>                          (syntax-class (syntax-after (point)))))))
>
> Although I'd say that the function syntax-class-to-char should be added
> to Emacs (in syntax.c, so that it can reuse syntax_code_spec).

Makes sense.  I've now done this in Emacs 28.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no