bug#71784: 31.0.50; Inconsistent fontification for field_identifier in c++-ts-mode

bug#71784: 31.0.50; Inconsistent fontification for field_identifier in c++-ts-mode

[Ergus via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Hi:

Using the c++-ts-mode I found that there is some inconsistent
fontification for the `fields_identifier`:

See the fontification in this example with `emacs -Q`.

```test.cpp

std::string key;
bool inserted;

struct name_t {
	std::string key;
	bool inserted;
};

name_t keys = {"aaa", true};

keys.inserted = false;
bool a = keys.inserted;
```

1. The `keys.inserted` values are shown differently before or after the
= (the inserted word is fontified is some cases, but not in all)

2. The variable names are fontified differently outside or
inside the struct.

3. The escape sequence (\t) is fontified differently to the rest of the
text inside the string. I don't know if that is intentional or not. If
it is intentional, just ignore this comment.

The inconsistencies 1 and 2 are not only different to c++-mode but they
are semantically incorrect.

Best,
Ergus


In GNU Emacs 31.0.50 (build 3, x86_64-pc-linux-gnu, GTK+ Version
 3.24.42, cairo version 1.18.0) of 2024-06-26 built on RTX
Repository revision: d800d6b3bdaa927e031e003e761e623147e812e6
Repository branch: project
System Description: Arch Linux

Configured using:
 'configure --prefix=/home/ergo/.local/ --with-mailutils --with-pgtk
 --with-modules --with-cairo --with-harfbuzz
 --with-native-compilation=aot
 '--program-transform-name=s/^ctags$/ctags.emacs/''

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
LCMS2 LIBSYSTEMD LIBXML2 MODULES NATIVE_COMP NOTIFY INOTIFY PDUMPER PGTK
PNG RSVG SECCOMP SOUND SQLITE3 THREADS TIFF TOOLKIT_SCROLL_BARS
TREE_SITTER WEBP XIM GTK3 ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: C++//

Minor modes in effect:
  fancy-compilation-mode: t
  windmove-mode: t
  global-auto-revert-mode: t
  electric-pair-mode: t
  whitespace-mode: t
  flyspell-mode: t
  completion-preview-mode: t
  diff-hl-margin-local-mode: t
  diff-hl-margin-mode: t
  diff-hl-mode: t
  corfu-terminal-mode: t
  global-corfu-mode: t
  corfu-mode: t
  project-multi-mode: t
  gtags-mode: t
  repeat-mode: t
  xterm-mouse-mode: t
  xclip-mode: t
  override-global-mode: t
  winner-mode: t
  save-place-mode: t
  delete-selection-mode: t
  savehist-mode: t
  global-display-fill-column-indicator-mode: t
  display-fill-column-indicator-mode: t
  global-display-line-numbers-mode: t
  display-line-numbers-mode: t
  which-key-mode: t
  show-paren-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  context-menu-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  minibuffer-regexp-mode: t
  size-indication-mode: t
  column-number-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
/mnt/casa/gits/emacs_clones/cuda-mode/cuda-mode hides /home/ergo/.config/emacs/elpa/cuda-mode-20201013.2230/cuda-mode
/mnt/casa/gits/emacs_clones/gtags-mode/gtags-mode hides /home/ergo/.config/emacs/elpa/gtags-mode-1.6/gtags-mode
/home/ergo/.config/emacs/elpa/transient-20240626.947/transient hides /home/ergo/.local/share/emacs/31.0.50/lisp/transient

Features:
(shadow sort mail-extr fancy-compilation compile comint ansi-osc
ansi-color comp-run comp-common emacsbug message mailcap yank-media puny
rfc822 mml mml-sec epa derived epg rfc6068 epg-config gnus-util
text-property-search mm-decode mm-bodies mm-encode mail-parse rfc2231
mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums
mm-util mail-prsvr mail-utils help-fns radix-tree cl-print debug
backtrace find-func c++-ts-mode c-ts-mode c-ts-common treesit time-date
windmove autorevert filenotify ffap thingatpt url-parse auth-source
eieio eieio-core icons password-cache json map url-vars elec-pair
whitespace flyspell-correct flyspell ispell completion-preview
diff-hl-margin diff-hl-dired citre-lang-fileref dired-x dired
dired-loaddefs diff-hl log-view pcvs-util vc-dir ewoc vc vc-dispatcher
diff-mode track-changes corfu-terminal popon corfu project-multi-mode
gtags-mode cl-macs files-x xref modern-cpp-font-lock cap-words superword
subword citre-lang-c citre-tags citre-ctags citre-readtags
citre-readtags-tables citre-backend-interface citre-common-tag rx
citre-common-util subr-x project cc-mode cc-fonts cc-guess cc-menus
cc-cmds cc-styles cc-align cc-engine cc-vars cc-defs term/tmux
term/xterm xterm init repeat cape compat use-package-ensure
use-package-diminish xt-mouse xclip edmacro kmacro byte-opt gv
use-package-bind-key bind-key cl-extra help-mode simple-16-theme winner
ring saveplace delsel savehist easy-mmode display-fill-column-indicator
display-line-numbers diminish which-key cl-seq use-package-core
cl-loaddefs cl-lib bytecomp byte-compile disp-table info
ac-emoji-autoloads ac-haskell-process-autoloads ac-html-autoloads
arduino-cli-mode-autoloads auctex-autoloads tex-site
auto-complete-autoloads avy-zap-autoloads avy-autoloads
better-shell-autoloads caml-autoloads cape-autoloads citre-autoloads
clang-format-autoloads cobol-mode-autoloads compile-multi-autoloads
corfu-terminal-autoloads corfu-autoloads crdt-autoloads
csv-mode-autoloads cuda-mode-autoloads d-mode-autoloads
deadgrep-autoloads debbugs-autoloads diff-hl-autoloads
diminish-autoloads dired-sidebar-autoloads dired-subtree-autoloads
dired-hacks-utils-autoloads dumb-jump-autoloads e2ansi-autoloads
emamux-autoloads esup-autoloads evil-collection-autoloads
annalist-autoloads evil-leader-autoloads evil-autoloads
face-explorer-autoloads fancy-compilation-autoloads flx-autoloads
flycheck-julia-autoloads flycheck-rust-autoloads flycheck-autoloads
flymake-nasm-autoloads flymake-quickdef-autoloads
flyspell-correct-autoloads git-modes-autoloads git-timemachine-autoloads
gnuplot-autoloads google-c-style-autoloads goto-chg-autoloads
groovy-mode-autoloads gtags-mode-autoloads haskell-mode-autoloads
highlight-indent-guides-autoloads i3wm-config-mode-autoloads
ibuffer-sidebar-autoloads iedit-autoloads imenu-list-autoloads
julia-ts-mode-autoloads julia-mode-autoloads languagetool-autoloads
lice-autoloads lorem-ipsum-autoloads lua-mode-autoloads magit-autoloads
git-commit-autoloads magit-section-autoloads markdown-mode-autoloads
modern-cpp-font-lock-autoloads move-dup-autoloads
multiple-cursors-autoloads mutt-mode-autoloads nasm-mode-autoloads
neotree-autoloads nftables-mode-autoloads nginx-mode-autoloads
notmuch-autoloads objed-autoloads opencl-mode-autoloads
paradox-autoloads phi-search-autoloads pkg-info-autoloads epl-autoloads
pkgbuild-mode-autoloads platformio-mode-autoloads async-autoloads
popon-autoloads popup-autoloads projectile-autoloads
projection-autoloads protobuf-mode-autoloads protobuf-ts-mode-autoloads
ptemplate-templates-autoloads ptemplate-autoloads scopeline-autoloads
shell-command+-autoloads slime-autoloads macrostep-autoloads
sphinx-mode-autoloads f-autoloads dash-autoloads s-autoloads
spinner-autoloads ssh-config-mode-autoloads string-inflection-autoloads
sudo-edit-autoloads systemd-autoloads tmux-mode-autoloads
transient-autoloads tsc-autoloads urgrep-autoloads vdiff-autoloads
hydra-autoloads lv-autoloads vterm-toggle-autoloads vterm-autoloads
vundo-autoloads with-editor-autoloads xclip-autoloads
yasnippet-snippets-autoloads yasnippet-autoloads early-init rmc
iso-transl tooltip cconv eldoc paren electric uniquify ediff-hook
vc-hooks lisp-float-type elisp-mode mwheel term/pgtk-win pgtk-win
term/common-win touch-screen pgtk-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu
timer select scroll-bar mouse jit-lock font-lock syntax font-core
term/tty-colors frame minibuffer nadvice seq simple cl-generic
indonesian philippine cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs
theme-loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads dbusbind inotify
dynamic-setting system-font-setting font-render-setting cairo gtk pgtk
lcms2 multi-tty move-toolbar make-network-process native-compile emacs)

Memory information:
((conses 16 227196 51858) (symbols 48 17144 0)
 (strings 32 58441 11185) (string-bytes 1 2123450) (vectors 16 22552)
 (vector-slots 8 264028 8898) (floats 8 109 235) (intervals 56 1774 0)
 (buffers 992 15))

bug#71784: 31.0.50; Inconsistent fontification for field_identifier in c++-ts-mode

[Eli Zaretskii]

> Date: Wed, 26 Jun 2024 16:13:47 +0200
> From:  Ergus via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> Using the c++-ts-mode I found that there is some inconsistent
> fontification for the `fields_identifier`:
> 
> See the fontification in this example with `emacs -Q`.
> 
> ```test.cpp
> 
> std::string key;
> bool inserted;
> 
> struct name_t {
> 	std::string key;
> 	bool inserted;
> };
> 
> name_t keys = {"aaa", true};
> 
> keys.inserted = false;
> bool a = keys.inserted;
> ```
> 
> 1. The `keys.inserted` values are shown differently before or after the
> = (the inserted word is fontified is some cases, but not in all)
> 
> 2. The variable names are fontified differently outside or
> inside the struct.
> 
> 3. The escape sequence (\t) is fontified differently to the rest of the
> text inside the string. I don't know if that is intentional or not. If
> it is intentional, just ignore this comment.
> 
> The inconsistencies 1 and 2 are not only different to c++-mode but they
> are semantically incorrect.

What does treesit-explore-mode tell you about these instances of
keys.inserted?

bug#71784: 31.0.50; Inconsistent fontification for field_identifier in c++-ts-mode

[Ergus via Bug reports for GNU Emacs, the Swiss army knife of text editors]

On Wed, Jun 26, 2024 at 06:46:04PM GMT, Eli Zaretskii wrote:
>> Date: Wed, 26 Jun 2024 16:13:47 +0200
>> From:  Ergus via "Bug reports for GNU Emacs,
>>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
>>
>> Using the c++-ts-mode I found that there is some inconsistent
>> fontification for the `fields_identifier`:
>>
>> See the fontification in this example with `emacs -Q`.
>>
>> ```test.cpp
>>
>> std::string key;
>> bool inserted;
>>
>> struct name_t {
>> 	std::string key;
>> 	bool inserted;
>> };
>>
>> name_t keys = {"aaa", true};
>>
>> keys.inserted = false;
>> bool a = keys.inserted;
>> ```
>>
>> 1. The `keys.inserted` values are shown differently before or after the
>> = (the inserted word is fontified is some cases, but not in all)
>>
>> 2. The variable names are fontified differently outside or
>> inside the struct.
>>
>> 3. The escape sequence (\t) is fontified differently to the rest of the
>> text inside the string. I don't know if that is intentional or not. If
>> it is intentional, just ignore this comment.
>>
>> The inconsistencies 1 and 2 are not only different to c++-mode but they
>> are semantically incorrect.
>
>What does treesit-explore-mode tell you about these instances of
>keys.inserted?

This is the whole explorer buffer for the example code:

(translation_unit
  (declaration
   type: (qualified_identifier scope: (namespace_identifier) :: name: (type_identifier))
   declarator: (identifier) ;)
  (declaration type: (primitive_type) declarator: (identifier) ;)
  (struct_specifier struct name: (type_identifier)
   body: 
    (field_declaration_list {
     (field_declaration
      type: (qualified_identifier scope: (namespace_identifier) :: name: (type_identifier))
      declarator: (field_identifier) ;)
     (field_declaration type: (primitive_type) declarator: (field_identifier) ;)
     }))
  ;
  (declaration type: (type_identifier)
   declarator: 
    (init_declarator declarator: (identifier) =
     value: 
      (initializer_list {
       (string_literal " (string_content) ")
       , (true) }))
   ;)
  (expression_statement
   (assignment_expression
    left: (field_expression argument: (identifier) operator: . field: (field_identifier))
    operator: = right: (false))
   ;)
  (declaration type: (primitive_type)
   declarator: 
    (init_declarator declarator: (identifier) =
     value: (field_expression argument: (identifier) operator: . field: (field_identifier)))
   ;))

The faces are:

1. Inside the struct insert has: font-lock-property-name-face
    It says `declarator: (field_identifier)` and I thin is applying the
    function c-ts-mode--fontify-declarator.

2. In `keys.inserted = false;` the `insert` words has: font-lock-property-use-face
    It says `field: (field_identifier)` and applies (I think) :feature 'property

3. In `bool a = keys.inserted;` is not fontified.
    But it says `field: (field_identifier)` like in 2.


Hope this helps.
Ergus

bug#71784: 31.0.50; Inconsistent fontification for field_identifier in c++-ts-mode

[Yuan Fu]

> On Jun 26, 2024, at 7:13 AM, Ergus via Bug reports for GNU Emacs, the Swiss army knife of text editors <bug-gnu-emacs@gnu.org> wrote:
> 
> 
> Hi:
> 
> Using the c++-ts-mode I found that there is some inconsistent
> fontification for the `fields_identifier`:
> 
> See the fontification in this example with `emacs -Q`.
> 
> ```test.cpp
> 
> std::string key;
> bool inserted;
> 
> struct name_t {
> std::string key;
> bool inserted;
> };
> 
> name_t keys = {"aaa", true};
> 
> keys.inserted = false;
> bool a = keys.inserted;
> ```
> 
> 1. The `keys.inserted` values are shown differently before or after the
> = (the inserted word is fontified is some cases, but not in all)

What’s the value of treesit-font-lock-level for you? If it’s 4, they should be fontified the same. On level 3, only LHS is fontified.

> 
> 2. The variable names are fontified differently outside or
> inside the struct.

I mean, the “variable name” inside a structure is a field, not a variable, so it makes sense that they are fontified differently. Variable has font-lock-variable-name-face, field has font-lock-field-name-face. Also variable and field face are the same in the default theme, so they should look the same nevertheless.

> 
> 3. The escape sequence (\t) is fontified differently to the rest of the
> text inside the string. I don't know if that is intentional or not. If
> it is intentional, just ignore this comment.

Yeah it’s intentional.

> 
> The inconsistencies 1 and 2 are not only different to c++-mode but they
> are semantically incorrect.

Yuan

bug#71784: 31.0.50; Inconsistent fontification for field_identifier in c++-ts-mode

[Ergus via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Hi Yuan:

Very thanks for replying

On Thu, Jun 27, 2024 at 12:16:13AM GMT, Yuan Fu wrote:
>
>
>> On Jun 26, 2024, at 7:13 AM, Ergus via Bug reports for GNU Emacs, the Swiss army knife of text editors <bug-gnu-emacs@gnu.org> wrote:
>>
>>
>> Hi:
>>
>> Using the c++-ts-mode I found that there is some inconsistent
>> fontification for the `fields_identifier`:
>>
>> See the fontification in this example with `emacs -Q`.
>>
>> ```test.cpp
>>
>> std::string key;
>> bool inserted;
>>
>> struct name_t {
>> std::string key;
>> bool inserted;
>> };
>>
>> name_t keys = {"aaa", true};
>>
>> keys.inserted = false;
>> bool a = keys.inserted;
>> ```
>>
>> 1. The `keys.inserted` values are shown differently before or after the
>> = (the inserted word is fontified is some cases, but not in all)
>
>What’s the value of treesit-font-lock-level for you? If it’s 4, they
>should be fontified the same. On level 3, only LHS is fontified.
>
You are right; it is 3 in my system.

However I would expect that treesit-font-lock-level will be equivalent
somehow to using font-lock-maximum-decoration with similar value.

I think it is confusing having two different fontifications for the same
variable due to their position. The colors are supposed to be a sort of
hint or help for the programmer eyes; not just a decoration right?

>>
>> 2. The variable names are fontified differently outside or
>> inside the struct.
>
>I mean, the “variable name” inside a structure is a field, not a
>variable, so it makes sense that they are fontified
>differently. Variable has font-lock-variable-name-face, field has
>font-lock-field-name-face. Also variable and field face are the same in
>the default theme, so they should look the same nevertheless.
>
Probably what annoys me is the difference with the previous behavior in
this case. A field is also a variable in some sense for C++. There is
not much difference with a variable in a namespace or a static variable
in a class... 

Does it makes sense not to colorize these "field" and LHS on level 3
(like it used to be in c++-mode)? But put the new fontifications all
together in level 4? In that way everything will be fontified in level 4
and will become immediately consistent.

>>
>> 3. The escape sequence (\t) is fontified differently to the rest of the
>> text inside the string. I don't know if that is intentional or not. If
>> it is intentional, just ignore this comment.
>
>Yeah it’s intentional.
>
Ok, good! Again, (just as a suggestion) it makes sense to move this new
fontification to level 4 as well?

>>
>> The inconsistencies 1 and 2 are not only different to c++-mode but they
>> are semantically incorrect.
>
>Yuan


Just to mention: I am not wondering about the match/compatibility with
c++-mode. I am only concerned about the semantic coherence of the
fontification; which is supposed to be somehow helpful, not confusing.

Thanks again,
Best
Ergus