bug#49060: Python-mode sporadically stops recognising single quote '-strings

bug#49060: Python-mode sporadically stops recognising single quote '-strings

[Konstantin Kharlamov]

[-- Attachment #1: Type: text/plain, Size: 9294 bytes --]

Given a python code like `foo = 'text'`, python-mode sometimes starts thinking the content inside single quotes is a code, and rather than apply font-lock-string-face to it, it starts applying various code fontifications.

It's an old bug, I think I've seen it for an year or more. Unfortunately, I never found steps to reproduce it, which is ultimately why I never reported it. It just spontaneously happens inside some python buffer, sometimes following days of using it, with no visible trigger. The only way to fix it AFAIK is to close the buffer, and reopen the file anew.

I'm hoping someone might give hints for debugging that. Perhaps, is there anything useful I can look into, once that problem appears?

I attached a screenshot of the bug, and it can also be viewed online at https://i.imgur.com/6H105Ob.png In the screenshot, on the left is a normal buffer, and on the right is one where the bug got triggered, both buffers has same content (ignore identifiers highlight, that's color-identifiers-mode).

As you can see, double-quote strings are unaffected. I tried finding the difference between processing single and double quote strings in python.el, but found none. At this point I have no idea what could be causing it.

Emacs version: master built from `f572735c51:       Fix the etags test suite` with one malloc_trim() commit on top. Below is an information from M-x report-emacs-bug

--------------

In GNU Emacs 28.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.29, cairo
version 1.17.4)
 of 2021-05-18 built on constantine-N61Ja
Repository revision: 222269b33e841a1909bfadc7e21da084e50ed4ff
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12011000
System Description: Arch Linux

Configured using:
 'configure --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib
 --localstatedir=/var --mandir=/usr/share/man --with-gameuser=:games
 --with-sound=alsa --with-modules --without-gconf --without-gsettings
 --enable-link-time-optimization --with-x-toolkit=gtk3 --without-xaw3d
 --without-compress-install
 '--program-transform-name=s/\([ec]tags\)/\1.emacs/' 'CFLAGS=-flto=2
 -march=native -O3 -pipe -fno-stack-protector -fweb
 -fmerge-all-constants -fno-plt -fcommon -flto -fuse-linker-plugin -flto
 -fuse-linker-plugin' 'LDFLAGS=-flto=2 -O3 -march=native -fweb
 -fmerge-all-constants -floop-nest-optimize -Wl,--sort-common,-z,relro
 -fno-plt -fcommon''

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM HARFBUZZ JPEG JSON LCMS2
LIBOTF LIBSYSTEMD LIBXML2 M17N_FLT MODULES NOTIFY INOTIFY PDUMPER PNG
RSVG SECCOMP SOUND THREADS TIFF TOOLKIT_SCROLL_BARS X11 XDBE XIM XPM
GTK3 ZLIB

Important settings:
  value of $LC_TIME: ru_RU.UTF-8
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Python

Minor modes in effect:
  flycheck-mode: t
  lsp-modeline-workspace-status-mode: t
  magit-auto-revert-mode: t
  global-git-commit-mode: t
  async-bytecomp-package-mode: t
  auto-revert-mode: t
  yas-global-mode: t
  yas-minor-mode: t
  global-color-identifiers-mode: t
  color-identifiers-mode: t
  delete-selection-mode: t
  winum-mode: t
  symbol-overlay-mode: t
  global-symbol-overlay-mode: t
  global-highlight-parentheses-mode: t
  highlight-parentheses-mode: t
  show-smartparens-global-mode: t
  show-smartparens-mode: t
  smartparens-global-mode: t
  smartparens-mode: t
  highlight-numbers-mode: t
  evil-goggles-mode: t
  global-evil-surround-mode: t
  evil-surround-mode: t
  shell-dirtrack-mode: t
  evil-mode: t
  evil-local-mode: t
  global-company-mode: t
  company-mode: t
  override-global-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
/home/constantine/.emacs.d/lisp/idomenu hides
/home/constantine/.emacs.d/elpa/idomenu-20141123.1320/idomenu
/home/constantine/.emacs.d/lisp/popup hides
/home/constantine/.emacs.d/elpa/popup-20160709.729/popup
/home/constantine/.emacs.d/elpa/cmake-mode-20180104.1137/cmake-mode hides
/usr/share/emacs/site-lisp/cmake-mode
/home/constantine/.emacs.d/elpa/transient-20201114.1825/transient hides
/usr/share/emacs/28.0.50/lisp/transient

Features:
(shadow mail-extr emacsbug sendmail cl-print debug backtrace eieio-opt
speedbar ezimage dframe shortdoc descr-text sort pp vc-mtn vc-hg vc-bzr
vc-src vc-sccs vc-svn vc-cvs vc-rcs vc hideif goto-chg help-fns
radix-tree find-dired ffap grep tabify cua-base undo-fu misearch
multi-isearch flycheck find-func lsp-diagnostics lsp-modeline view
lsp-steep lsp-svelte lsp-sqls lsp-yaml lsp-xml lsp-vimscript lsp-vhdl
lsp-vetur lsp-html lsp-verilog lsp-vala lsp-terraform lsp-tex lsp-sorbet
lsp-solargraph lsp-rust lsp-rf lsp-r lsp-purescript lsp-pyls lsp-pwsh
lsp-php lsp-perl lsp-ocaml lsp-nix lsp-nim lsp-lua lsp-kotlin lsp-json
lsp-javascript lsp-haxe lsp-groovy lsp-hack lsp-go lsp-completion
lsp-gdscript lsp-fsharp lsp-fortran lsp-eslint lsp-erlang lsp-elixir
lsp-elm lsp-dockerfile lsp-dhall lsp-css lsp-csharp gnutls lsp-crystal
lsp-cmake lsp-clojure lsp-clangd dom lsp-bash lsp-angular lsp-ada
lsp-mode lsp-protocol tree-widget wid-edit spinner network-stream nsm
smartparens-markdown markdown-mode noutline outline lv inline ht f s
ewoc dash-functional compile bindat dabbrev smartparens-c cc-mode
cc-fonts cc-guess cc-menus cc-cmds cc-styles cc-align cc-engine cc-vars
cc-defs quail evil-magit magit-submodule magit-obsolete magit-popup
magit-blame magit-stash magit-reflog magit-bisect magit-push magit-pull
magit-fetch magit-clone magit-remote magit-commit magit-sequence
magit-notes magit-worktree magit-tag magit-merge magit-branch
magit-reset magit-files magit-refs magit-status magit magit-repos
magit-apply magit-wip magit-log which-func imenu magit-diff smerge-mode
magit-core magit-autorevert magit-margin magit-transient magit-process
magit-mode git-commit transient magit-git magit-section magit-utils crm
log-edit message rmc puny dired dired-loaddefs rfc822 mml mml-sec epa
derived epg epg-config gnus-util rmail rmail-loaddefs
text-property-search mm-decode mm-bodies mm-encode mail-parse rfc2231
rfc2047 rfc2045 mm-util ietf-drums mail-prsvr mailabbrev mail-utils
gmm-utils mailheader pcvs-util add-log with-editor async-bytecomp async
server make-mode sh-script smie executable conf-mode autorevert
filenotify vc-git diff-mode vc-dispatcher yaml-mode desktop frameset
smex yasnippet-snippets yasnippet face-remap color-identifiers-mode
smartparens-python python tramp-sh tramp tramp-loaddefs trampver
tramp-integration files-x tramp-compat parse-time iso8601 time-date
ls-lisp format-spec delsel winum symbol-overlay highlight-parentheses
sp-sublimetext-like smartparens-config smartparens-text smartparens dash
highlight-numbers parent-mode ido evil-goggles pulse color evil-surround
emvil evil evil-integration undo-tree diff evil-maps evil-commands
flyspell ispell evil-jumps evil-command-window evil-types evil-search
evil-ex shell pcomplete comint ansi-color evil-macros evil-repeat
evil-states evil-core advice evil-common windmove thingatpt rect
evil-digraphs evil-vars avy company-oddmuse company-keywords
company-etags etags fileloop generator xref project ring company-gtags
company-dabbrev-code company-dabbrev company-files company-capf
company-cmake company-xcode company-clang company-semantic company-eclim
company-template company-bbdb company pcase cl-extra help-mode
use-package-bind-key use-package-core edmacro kmacro bind-key easy-mmode
rx info package browse-url url url-proxy url-privacy url-expand
url-methods url-history url-cookie url-domsuf url-util mailcap
url-handlers url-parse auth-source cl-seq eieio eieio-core cl-macs
eieio-loaddefs password-cache json subr-x map url-vars seq byte-opt gv
bytecomp byte-compile cconv cl-loaddefs cl-lib iso-transl tooltip eldoc
electric uniquify ediff-hook vc-hooks lisp-float-type mwheel term/x-win
x-win term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu
timer select scroll-bar mouse jit-lock font-lock syntax font-core
term/tty-colors frame minibuffer cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite charscript charprop case-table epa-hook
jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice button
loaddefs faces cus-face macroexp files window text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote threads dbusbind inotify lcms2
dynamic-setting font-render-setting cairo move-toolbar gtk x-toolkit x
multi-tty make-network-process emacs)

Memory information:
((conses 16 833973 402991)
 (symbols 48 50378 1)
 (strings 32 235267 26373)
 (string-bytes 1 6628245)
 (vectors 16 104698)
 (vector-slots 8 2468565 767967)
 (floats 8 351 1239)
 (intervals 56 50806 1258)
 (buffers 992 80))


[-- Attachment #2: 6H105Ob.png --]
[-- Type: image/png, Size: 13649 bytes --]

bug#49060: Acknowledgement (Python-mode sporadically stops recognising single quote '-strings)

[Konstantin Kharlamov]

I allocated some time to this problem today. To no avail, but commenting here just not to repeat this work next time:

1. python-mode is derived from prog-mode, but the latter has no code for working with single quotes nor has any mention of string-face. So the problem is unlikely to be there.
2. I did lots of editing in attempts to trigger the bug. Specifically, I was creating single-quoted strings and f-strings; first manually, then with a keyboard macro. Again, I couldn't trigger it.
    That makes me thinking, the problem may be some race condition of python fontification with another mode. Although, even if that been true, I don't get why would that influence the whole file. It would likely have been local damage, which could've been fixed by adding another singular quote, thus forcing refontification. So, I'm not sure that theory really holds up.

> Perhaps, is there anything useful I can look into, once that problem appears?

Next time it happens, gotta add debug prints to `python--font-lock-f-strings` function, and see the difference between its execution in the file when the problem is triggered compared to when it's not.