bug#45679: 28.0.50; python-mode variable font locking is inconsistent

bug#45679: 28.0.50; python-mode variable font locking is inconsistent

[Doug Davis]

In python-mode I'm seeing inconsistent variable font locking in class
and function definitions. The first variable in a class keeps the
default face while the remaining variables get the variable face. In
functions it's a bit more inconsistent. If I modify the text around a
bugged variable the face is corrected. Below is a small example which
recreated the behavior. I see it with both my full configuration loaded
and with emacs -Q. I can reproduce on both GNU/Linux and macOS.

#+begin_src python
@dataclass
class Student:
    """Described a student.

    Attributes
    ----------
    name : str
        Full name
    idnum : int
        Identification number

    """
    name: str = ""   ## <--- 'name' is not given the variable face (keeps default)
    idnum: int = 0
#+end_src

In GNU Emacs 28.0.50 (build 1, x86_64-apple-darwin20.2.0, NS appkit-2022.20 Version 11.1 (Build 20C69))
 of 2021-01-02 built on top.local
Repository revision: f14869cd70e61b1908ec88a5e3d4bf21c7d538a0
Repository branch: master
Windowing system distributor 'Apple', version 10.3.2022
System Description:  macOS 11.1

Configured using:
 'configure --with-ns --with-json --with-xml2 --with-gnutls
 --with-modules --with-rsvg 'CFLAGS=-mtune=native -march=native -O3''

Configured features:
JPEG TIFF GIF PNG RSVG GLIB NOTIFY KQUEUE ACL GNUTLS LIBXML2 ZLIB
TOOLKIT_SCROLL_BARS NS MODULES THREADS JSON PDUMPER LCMS2

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Org

Minor modes in effect:
  flyspell-mode: t
  global-git-commit-mode: t
  eros-mode: t
  pdf-occur-global-minor-mode: t
  yas-global-mode: t
  yas-minor-mode: t
  which-key-mode: t
  async-bytecomp-package-mode: t
  projectile-mode: t
  marginalia-mode: t
  selectrum-prescient-mode: t
  selectrum-mode: t
  recentf-mode: t
  show-paren-mode: t
  shell-dirtrack-mode: t
  global-auto-revert-mode: t
  whitespace-mode: t
  global-display-line-numbers-mode: t
  display-line-numbers-mode: t
  override-global-mode: t
  straight-use-package-mode: t
  straight-package-neutering-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
/Users/ddavis/.emacs.d/straight/build/jsonrpc/jsonrpc hides /Applications/Emacs.app/Contents/Resources/lisp/jsonrpc
/Users/ddavis/.emacs.d/straight/build/xref/xref hides /Applications/Emacs.app/Contents/Resources/lisp/progmodes/xref
/Users/ddavis/.emacs.d/straight/build/project/project hides /Applications/Emacs.app/Contents/Resources/lisp/progmodes/project
/Users/ddavis/.emacs.d/straight/build/flymake/flymake hides /Applications/Emacs.app/Contents/Resources/lisp/progmodes/flymake
/Users/ddavis/.emacs.d/straight/build/let-alist/let-alist hides /Applications/Emacs.app/Contents/Resources/lisp/emacs-lisp/let-alist
/Users/ddavis/.emacs.d/straight/build/eldoc/eldoc hides /Applications/Emacs.app/Contents/Resources/lisp/emacs-lisp/eldoc

Features:
(shadow sort mail-extr emacsbug blacken python tramp-sh flyspell ispell
org-element avl-tree ol-eww eww xdg url-queue mm-url ol-rmail ol-mhe
ol-irc ol-info ol-gnus nnselect gnus-search eieio-opt speedbar ezimage
dframe ol-docview ol-bibtex bibtex ol-bbdb ol-w3m gnus-topic nndraft
nnmh utf-7 gnutls nnfolder nnnil gnus-agent gnus-srvr gnus-score
score-mode nnvirtual gnus-msg gnus-art mm-uu mml2015 mm-view mml-smime
smime dig nntp gnus-cache gnus-sum shr kinsoku svg dom gnus-group
gnus-undo gnus-start gnus-dbus dbus xml gnus-cloud nnimap nnmail
mail-source utf7 netrc nnoo gnus-spec gnus-int gnus-range gnus-win gnus
nnheader mu4e desktop frameset mu4e-org mu4e-main mu4e-view mu4e-headers
mu4e-compose mu4e-context mu4e-draft mu4e-actions ido rfc2368 smtpmail
mu4e-mark mu4e-message flow-fill mu4e-proc mu4e-utils mu4e-lists hl-line
mu4e-vars mu4e-meta magit-extras magit-bookmark magit-submodule
magit-obsolete magit-blame magit-stash magit-reflog magit-bisect
magit-push magit-pull magit-fetch magit-clone magit-remote magit-commit
magit-sequence magit-notes magit-worktree magit-tag magit-merge
magit-branch magit-reset magit-files magit-refs magit-status magit
magit-repos magit-apply magit-wip magit-log which-func magit-diff
smerge-mode diff magit-core magit-autorevert magit-margin
magit-transient magit-process magit-mode git-commit magit-git
magit-section magit-utils log-edit pcvs-util add-log with-editor vc-git
diff-mode char-fold eros company-oddmuse company-keywords company-etags
etags fileloop generator company-gtags company-dabbrev-code
company-dabbrev company-files company-clang company-cmake
company-semantic company-template company-bbdb company-capf
rainbow-delimiters init sendmail message rfc822 mml mml-sec gnus-util
rmail rmail-loaddefs mm-decode mm-bodies mm-encode mail-parse rfc2231
rfc2047 rfc2045 mm-util ietf-drums mail-prsvr mailabbrev mail-utils
gmm-utils mailheader w3m doc-view timezone w3m-hist w3m-fb bookmark-w3m
w3m-ems w3m-favicon w3m-image tab-line w3m-proc w3m-util w3m-autoloads
ox-hugo-autoloads pdf-occur tablist tablist-filter semantic/wisent/comp
semantic/wisent semantic/wisent/wisent semantic/util-modes semantic/util
semantic semantic/tag semantic/lex semantic/fw mode-local cedet
pdf-isearch let-alist pdf-misc pdf-tools package url-handlers pdf-view
jka-compr pdf-cache pdf-info tq pdf-util image-mode exif
pdf-tools-autoloads tablist-autoloads lsp-latex lsp-latex-autoloads
auctex-autoloads tex-site erc-hl-nicks-autoloads auth-source-pass
circe-autoloads elfeed-autoloads gruvbox-theme gruvbox autothemer
gruvbox-autoloads autothemer-autoloads helpful-autoloads
elisp-refs-autoloads ace-window-autoloads avy-autoloads
yaml-mode-autoloads iedit-autoloads yasnippet-snippets-autoloads
yasnippet-snippets yasnippet yasnippet-autoloads which-key
which-key-autoloads rainbow-delimiters-autoloads cider-autoloads
sesman-autoloads queue-autoloads parseedn-autoloads parseclj-autoloads
a-autoloads clojure-mode-autoloads blacken-autoloads pyvenv eshell
pyvenv-autoloads modern-cpp-font-lock-autoloads clang-format-autoloads
eldoc-box-autoloads eglot-autoloads project-autoloads xref-autoloads
flymake-autoloads eldoc-autoloads jsonrpc-autoloads lsp-ui
lsp-ui-flycheck lsp-ui-doc goto-addr lsp-ui-imenu lsp-ui-peek
lsp-ui-sideline face-remap lsp-ui-autoloads lsp-pyls lsp-mode
lsp-protocol xref spinner network-stream puny nsm rmc markdown-mode
color lv inline ht f s ewoc dash-functional bindat lsp-mode-autoloads
lv-autoloads markdown-mode-autoloads spinner-autoloads ht-autoloads
f-autoloads s-autoloads dash-functional-autoloads flycheck rx dash
flycheck-autoloads let-alist-autoloads rg rg-info-hack rg-menu transient
async-bytecomp async rg-ibuffer rg-result wgrep-rg wgrep rg-history
rg-header cus-edit cus-start cus-load rg-autoloads wgrep-autoloads
magit-autoloads git-commit-autoloads with-editor-autoloads
transient-autoloads dash-autoloads async-autoloads company pcase
company-autoloads projectile grep ibuf-ext ibuffer ibuffer-loaddefs
projectile-autoloads pkg-info-autoloads epl-autoloads consult-selectrum
consult-selectrum-autoloads consult imenu compile bookmark
text-property-search pp consult-autoloads marginalia
marginalia-autoloads selectrum-prescient prescient
selectrum-prescient-autoloads prescient-autoloads selectrum
minibuf-eldef crm selectrum-autoloads visual-fill-column
visual-fill-column-autoloads crux tramp tramp-loaddefs trampver
tramp-integration files-x tramp-compat parse-time iso8601 ls-lisp
thingatpt crux-autoloads all-the-icons-dired-autoloads
all-the-icons-autoloads memoize-autoloads exec-path-from-shell
exec-path-from-shell-autoloads finder-inf eros-autoloads recentf
tree-widget wid-edit cc-mode cc-fonts cc-guess cc-menus cc-cmds
cc-styles cc-align cc-engine cc-vars cc-defs sh-script smie executable
browse-url url url-proxy url-privacy url-expand url-methods url-history
url-cookie url-domsuf url-util url-parse url-vars mailcap paren dired
dired-loaddefs vc vc-dispatcher epa-file epa derived epg epg-config
auth-source eieio eieio-core eieio-loaddefs password-cache json map term
disp-table shell ehelp esh-mode esh-cmd esh-ext esh-opt esh-proc esh-io
esh-arg esh-module esh-groups esh-util autorevert filenotify whitespace
project org ob ob-tangle ob-ref ob-lob ob-table ob-exp org-macro
org-footnote org-src ob-comint org-pcomplete pcomplete comint ansi-color
ring org-list org-faces org-entities time-date noutline outline
org-version ob-emacs-lisp ob-core ob-eval org-table ol org-keys
org-compat advice org-macs org-loaddefs format-spec find-func cal-menu
calendar cal-loaddefs display-line-numbers edmacro kmacro server
straight-x cl-seq use-package use-package-ensure use-package-delight
use-package-diminish use-package-bind-key bind-key easy-mmode
use-package-core use-package-autoloads bind-key-autoloads
straight-autoloads info cl-extra help-mode easymenu seq byte-opt
straight subr-x cl-macs gv cl-loaddefs cl-lib bytecomp byte-compile
cconv early-init iso-transl tooltip eldoc electric uniquify ediff-hook
vc-hooks lisp-float-type mwheel term/ns-win ns-win ucs-normalize
mule-util term/common-win tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame minibuffer cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite charscript charprop case-table epa-hook
jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice button
loaddefs faces cus-face macroexp files window text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote threads kqueue cocoa ns lcms2
multi-tty make-network-process emacs)

Memory information:
((conses 16 623239 653268)
 (symbols 48 54792 837)
 (strings 32 192401 97666)
 (string-bytes 1 6740188)
 (vectors 16 90452)
 (vector-slots 8 1093967 823574)
 (floats 8 579 2234)
 (intervals 56 690 334)
 (buffers 984 18))

bug#45679: 28.0.50; python-mode variable font locking is inconsistent

[Doug Davis]

Doug Davis <ddavis@ddavis.io> writes:

> In python-mode I'm seeing inconsistent variable font locking in class
> and function definitions. The first variable in a class keeps the
> default face while the remaining variables get the variable face. In
> functions it's a bit more inconsistent. If I modify the text around a
> bugged variable the face is corrected. Below is a small example which
> recreated the behavior. I see it with both my full configuration loaded
> and with emacs -Q. I can reproduce on both GNU/Linux and macOS.

Just to expand on this a bit-- the font locking appears to cycle between
the variable name face and the default face. Making changes inside of a
function is triggering some variables to rotate between the two faces. I
don't yet see an obvious pattern.

bug#45679: 28.0.50; python-mode variable font locking is inconsistent

[Doug Davis]

My guess is b2ce94fa5e introduced the bug.

bug#45679: 28.0.50; python-mode variable font locking is inconsistent

[Doug Davis]

Doug Davis <ddavis@ddavis.io> writes:

> My guess is b2ce94fa5e introduced the bug.

My first guess was wrong. I tested python.el after a few commits over
the last few months and it was after 296e4dd and e7b4f46 where the bug
pops up.

bug#45679: 28.0.50; python-mode variable font locking is inconsistent

[Doug Davis]

Its seems that the issue pops up when a function signature includes the
trailing return type int:

#+begin_src python
def func(a, b) -> None:
    x = 5
    y = 6
#+end_src

has bad assignment font locking for "x = 5"; while this version:

#+begin_src
def func(a, b):
    x = 5
    y = 6
#+end_src

appears to have expected variable assignment font locking.

bug#45679: 28.0.50; python-mode variable font locking is inconsistent

[Doug Davis]

Doug Davis <ddavis@ddavis.io> writes:

Spoke too soon again; types for individual function arguments _also_
triggers the issue. Perhaps some recent changes
(296e4dd15e3754fdddf70d33f2d630462d4b3309 and
e7b4f465e8b2ec1c8d54a611db7eca44ae350711) associated with font locking
(which appear to have been meant to add better support for the existence
of type hits) should be reverted for now.

bug#45679: 28.0.50; python-mode variable font locking is inconsistent

[Doug Davis]

Doug Davis <ddavis@ddavis.io> writes:

> Spoke too soon again; types for individual function arguments _also_
> triggers the issue. Perhaps some recent changes
> (296e4dd15e3754fdddf70d33f2d630462d4b3309 and
> e7b4f465e8b2ec1c8d54a611db7eca44ae350711) associated with font locking
> (which appear to have been meant to add better support for the existence
> of type hits) should be reverted for now.

And this is another commit that touches the font locking code (actually
overwriting the two commits above):
b2ce94fa5eecee0afd0e6237956cfb2b02b8bb0b

bug#45679: 28.0.50; python-mode variable font locking is inconsistent

[Lars Ingebrigtsen]

Doug Davis <ddavis@ddavis.io> writes:

> Its seems that the issue pops up when a function signature includes the
> trailing return type int:
>
> #+begin_src python
> def func(a, b) -> None:
>     x = 5
>     y = 6
> #+end_src
>
> has bad assignment font locking for "x = 5"; while this version:
>
> #+begin_src
> def func(a, b):
>     x = 5
>     y = 6
> #+end_src
>
> appears to have expected variable assignment font locking.

This should now be fixed (in a patch for bug#51362) in Emacs 29.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no