bug#54131: 29.0.50; Flyspell incorrectly reports first word in Python f-string

bug#54131: 29.0.50; Flyspell incorrectly reports first word in Python f-string

[Philipp Stephani]

Given this Python file:

$ cat /tmp/a.py 
print(f'hello world')

Visit it and enable Flyspell:

$ emacs -Q /tmp/a.py -f flyspell-prog-mode

Flyspell then marks the string "f'hello" as incorrect, thinking it's a
misspelling of "hello".  But it shouldn't cross the string boundary.


In GNU Emacs 29.0.50 (build 59, x86_64-pc-linux-gnu, GTK+ Version 3.24.31, cairo version 1.16.0)
 of 2022-02-23
Repository revision: 85ad8616007e286c237bb2906d1928bb551462e7
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12014000
System Description: Debian GNU/Linux rodete

Configured using:
 'configure --enable-gcc-warnings=warn-only
 --enable-gtk-deprecation-warnings --without-pop --with-mailutils
 --enable-checking=all --enable-check-lisp-object-type --with-modules
 'CFLAGS=-O0 -ggdb3''

Configured features:
CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG JSON
LIBSELINUX LIBSYSTEMD LIBXML2 MODULES NOTIFY INOTIFY PDUMPER PNG SECCOMP
SOUND THREADS TIFF TOOLKIT_SCROLL_BARS X11 XDBE XIM XPM GTK3 ZLIB

Important settings:
  value of $LC_TIME: en_DK.utf8
  value of $LANG: en_US.utf8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug sendmail phst skeleton pcase ffap
thingatpt url url-proxy url-privacy url-expand url-methods url-history
url-cookie url-domsuf url-util url-parse auth-source cl-seq eieio
eieio-core cl-macs eieio-loaddefs json map url-vars rx message mailcap
yank-media rmc dired dired-loaddefs rfc822 mml mml-sec password-cache
epa derived epg rfc6068 epg-config gnus-util time-date mm-decode
mm-bodies mm-encode mail-parse rfc2231 rfc2047 rfc2045 mm-util
ietf-drums mail-prsvr mailabbrev mail-utils gmm-utils mailheader gnutls
puny elp dbus xml seq gv subr-x byte-opt bytecomp byte-compile cconv
compile text-property-search comint ansi-color ring cl-loaddefs cl-lib
iso-transl tooltip eldoc paren electric uniquify ediff-hook vc-hooks
lisp-float-type elisp-mode mwheel term/x-win x-win term/common-win x-dnd
tool-bar dnd fontset image regexp-opt fringe tabulated-list replace
newcomment text-mode lisp-mode prog-mode register page tab-bar menu-bar
rfn-eshadow isearch easymenu timer select scroll-bar mouse jit-lock
font-lock syntax font-core term/tty-colors frame minibuffer cl-generic
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite emoji-zwj charscript
charprop case-table epa-hook jka-cmpr-hook help simple abbrev obarray
cl-preloaded nadvice button loaddefs faces cus-face macroexp files
window text-properties overlay sha1 md5 base64 format env code-pages
mule custom widget keymap hashtable-print-readable backquote threads
dbusbind inotify dynamic-setting system-font-setting font-render-setting
cairo move-toolbar gtk x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 67449 9159)
 (symbols 48 8204 1)
 (strings 32 23683 2078)
 (string-bytes 1 760952)
 (vectors 16 15596)
 (vector-slots 8 208549 49248)
 (floats 8 28 30)
 (intervals 56 230 0)
 (buffers 992 11))

-- 
Google Germany GmbH
Erika-Mann-Straße 33
80636 München

Geschäftsführer: Paul Manicle, Liana Sebastian
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg

Diese E-Mail ist vertraulich.  Falls Sie diese fälschlicherweise erhalten haben
sollten, leiten Sie diese bitte nicht an jemand anderes weiter, löschen Sie
alle Kopien und Anhänge davon und lassen Sie mich bitte wissen, dass die E-Mail
an die falsche Person gesendet wurde.

This e-mail is confidential.  If you received this communication by mistake,
please don’t forward it to anyone else, please erase all copies and
attachments, and please let me know that it has gone to the wrong person.

bug#54131: 29.0.50; Flyspell incorrectly reports first word in Python f-string

[Lars Ingebrigtsen]

Philipp Stephani <p.stephani2@gmail.com> writes:

> Given this Python file:
>
> $ cat /tmp/a.py 
> print(f'hello world')
>
> Visit it and enable Flyspell:
>
> $ emacs -Q /tmp/a.py -f flyspell-prog-mode
>
> Flyspell then marks the string "f'hello" as incorrect, thinking it's a
> misspelling of "hello".  But it shouldn't cross the string boundary.

Hm, yes.  In this case, the mode knows that f isn't part of the
expression, but I guess we have no way of communicating that to ispell?

Skimming ispell-get-word, it looks like it uses a regexp to determine
what the word at point is, so we'd need to make some sort of framework
to allow modes to say where a string begins and ends?  Or if we want to
just do something hackish, we could make that function check
the face for font-lock-string-face and then limit based on that.  (Which
sounds simple enough.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#54131: 29.0.50; Flyspell incorrectly reports first word in Python f-string

[Philipp Stephani]

Am Mi., 23. Feb. 2022 um 21:13 Uhr schrieb Lars Ingebrigtsen <larsi@gnus.org>:
>
> Philipp Stephani <p.stephani2@gmail.com> writes:
>
> > Given this Python file:
> >
> > $ cat /tmp/a.py
> > print(f'hello world')
> >
> > Visit it and enable Flyspell:
> >
> > $ emacs -Q /tmp/a.py -f flyspell-prog-mode
> >
> > Flyspell then marks the string "f'hello" as incorrect, thinking it's a
> > misspelling of "hello".  But it shouldn't cross the string boundary.
>
> Hm, yes.  In this case, the mode knows that f isn't part of the
> expression, but I guess we have no way of communicating that to ispell?
>
> Skimming ispell-get-word, it looks like it uses a regexp to determine
> what the word at point is, so we'd need to make some sort of framework
> to allow modes to say where a string begins and ends?

Like the syntax table? (nth 8 (syntax-ppss)) gives you the beginning
of the string, and that seems to give correct results even for
f-strings.

bug#54131: 29.0.50; Flyspell incorrectly reports first word in Python f-string

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Wed, 23 Feb 2022 21:13:27 +0100
> Cc: 54131@debbugs.gnu.org
> 
> Skimming ispell-get-word, it looks like it uses a regexp to determine
> what the word at point is, so we'd need to make some sort of framework
> to allow modes to say where a string begins and ends?  Or if we want to
> just do something hackish, we could make that function check
> the face for font-lock-string-face and then limit based on that.  (Which
> sounds simple enough.)

Our spell-checking modes aren't supposed to work on code, so they
don't pay any attention to the buffer's syntax table, and instead use
regular expressions specific to the proof-reading language to know
what can and cannot be in a word.

For program sources, flyspell-prog-mode relies on font-lock faces to
tell it where strings and comments are, so perhaps this part doesn't
work for some reason (and perhaps the root cause is in font-lock of
python-mode, not in flyspell.el per se).

bug#54131: 29.0.50; Flyspell incorrectly reports first word in Python f-string

[Eli Zaretskii]

> Resent-From: Philipp Stephani <p.stephani2@gmail.com>
> Original-Sender: "Debbugs-submit" <debbugs-submit-bounces@debbugs.gnu.org>
> Resent-CC: bug-gnu-emacs@gnu.org
> Resent-Sender: help-debbugs@gnu.org
> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Wed, 23 Feb 2022 21:27:04 +0100
> Cc: 54131@debbugs.gnu.org
> 
> Am Mi., 23. Feb. 2022 um 21:13 Uhr schrieb Lars Ingebrigtsen <larsi@gnus.org>:
> >
> > Philipp Stephani <p.stephani2@gmail.com> writes:
> >
> > > Given this Python file:
> > >
> > > $ cat /tmp/a.py
> > > print(f'hello world')
> > >
> > > Visit it and enable Flyspell:
> > >
> > > $ emacs -Q /tmp/a.py -f flyspell-prog-mode
> > >
> > > Flyspell then marks the string "f'hello" as incorrect, thinking it's a
> > > misspelling of "hello".  But it shouldn't cross the string boundary.
> >
> > Hm, yes.  In this case, the mode knows that f isn't part of the
> > expression, but I guess we have no way of communicating that to ispell?
> >
> > Skimming ispell-get-word, it looks like it uses a regexp to determine
> > what the word at point is, so we'd need to make some sort of framework
> > to allow modes to say where a string begins and ends?
> 
> Like the syntax table? (nth 8 (syntax-ppss)) gives you the beginning
> of the string, and that seems to give correct results even for
> f-strings.

That's not how flyspell-prog-mode works, see my other message.  I
guess it doesn't want to run syntax analysis functions on the fly, as
that could be too slow (flyspell being on post-command-hook), but
instead relies on font lock that is run by the display engine anyway.

bug#54131: 29.0.50; Flyspell incorrectly reports first word in Python f-string

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> For program sources, flyspell-prog-mode relies on font-lock faces to
> tell it where strings and comments are, so perhaps this part doesn't
> work for some reason (and perhaps the root cause is in font-lock of
> python-mode, not in flyspell.el per se).

I didn't know about flyspell-prog-mode, but it does indeed do the right
thing here.  (I've now mentioned the mode in the flyspell-mode doc
string.)

But `M-$' does not do the right thing -- if you `M-$' on f'hello, it
tries to correct the whole thing.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#54131: 29.0.50; Flyspell incorrectly reports first word in Python f-string

[martin rudalics]

 > But `M-$' does not do the right thing -- if you `M-$' on f'hello, it
 > tries to correct the whole thing.

Try with 'ispell-comment-or-string-at-point'.

martin

bug#54131: 29.0.50; Flyspell incorrectly reports first word in Python f-string

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: p.stephani2@gmail.com,  54131@debbugs.gnu.org
> Date: Thu, 24 Feb 2022 10:12:58 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > For program sources, flyspell-prog-mode relies on font-lock faces to
> > tell it where strings and comments are, so perhaps this part doesn't
> > work for some reason (and perhaps the root cause is in font-lock of
> > python-mode, not in flyspell.el per se).
> 
> I didn't know about flyspell-prog-mode, but it does indeed do the right
> thing here.  (I've now mentioned the mode in the flyspell-mode doc
> string.)

That's strange, because the OP explicitly invoked flyspell-prog-mode,
so how come it didn't work for Philipp, but did for you?

> But `M-$' does not do the right thing -- if you `M-$' on f'hello, it
> tries to correct the whole thing.

Maybe we should have a key binding for
ispell-comment-or-string-at-point, and/or maybe
ispell-comments-and-strings and flyspell-prog-mode should rebind M-$
to ispell-comment-or-string-at-point.

bug#54131: 29.0.50; Flyspell incorrectly reports first word in Python f-string

[Visuwesh]

[வியாழன், பிப்ரவரி 24 2022] Eli Zaretskii wrote:

>> From: Lars Ingebrigtsen <larsi@gnus.org>
>> Cc: p.stephani2@gmail.com,  54131@debbugs.gnu.org
>> Date: Thu, 24 Feb 2022 10:12:58 +0100
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > For program sources, flyspell-prog-mode relies on font-lock faces to
>> > tell it where strings and comments are, so perhaps this part doesn't
>> > work for some reason (and perhaps the root cause is in font-lock of
>> > python-mode, not in flyspell.el per se).
>> 
>> I didn't know about flyspell-prog-mode, but it does indeed do the right
>> thing here.  (I've now mentioned the mode in the flyspell-mode doc
>> string.)
>
> That's strange, because the OP explicitly invoked flyspell-prog-mode,
> so how come it didn't work for Philipp, but did for you?
>
>> But `M-$' does not do the right thing -- if you `M-$' on f'hello, it
>> tries to correct the whole thing.
>
> Maybe we should have a key binding for
> ispell-comment-or-string-at-point, and/or maybe
> ispell-comments-and-strings and flyspell-prog-mode should rebind M-$
> to ispell-comment-or-string-at-point.

I misspell parts of a symbol quite often so rebinding M-$ to a command
that NOOPs when the point is not over a comment/string would end up
annoying, IMO.  I am not sure how common this usecase is though.

bug#54131: 29.0.50; Flyspell incorrectly reports first word in Python f-string

[Lars Ingebrigtsen]

Visuwesh <visuweshm@gmail.com> writes:

> I misspell parts of a symbol quite often so rebinding M-$ to a command
> that NOOPs when the point is not over a comment/string would end up
> annoying, IMO.  I am not sure how common this usecase is though.

I think it's pretty common.  I think a new command that does what
ispell-comment-or-string-at-point does if we're in a comment/string and
what ispell-word does otherwise might be nice.

And flyspell-prog-mode could rebind `M-$', as Eli suggests, but to that
new command.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no