bug#63390: 29.0.90; c-ts-mode fails to recognize functions in xterm.c

bug#63390: 29.0.90; c-ts-mode fails to recognize functions in xterm.c

[Eli Zaretskii]

To reproduce:

  emacs -Q
  M-x load-library RET c-ts-mode RET
  C-x C-f src/xterm.c
  C-u 8290 M-g g

Observe that the name of the function x_draw_glyph_string_foreground
is not fontified in font-lock-function-name-face, but in the default
face.

Starting treesit-explore-mode seems to indicate that tree-sitter
interprets this as a function declaration, not a function definition:

  (function_declarator declarator: (identifier)
   parameters: 
    (parameter_list (
     (parameter_declaration
      type: (struct_specifier struct name: (type_identifier))
      declarator: (pointer_declarator * declarator: (identifier)))
     )))

Same with the next function, x_draw_composite_glyph_string_foreground.
But the function after that, x_draw_glyphless_glyph_string_foreground,
is again recognized as function definition.  I wonder if the
preprocessor conditionals around there have something to do with that.

Btw, function declarations in a header file are recognized as such,
but the names of the functions there are still correctly fontified.

In GNU Emacs 29.0.90 (build 88, i686-pc-mingw32) of 2023-05-09 built on
 HOME-C4E4A596F7
Repository revision: 387ddc0ccc1b21f612b9106bafec63170ede30e6
Repository branch: emacs-29
Windowing system distributor 'Microsoft Corp.', version 5.1.2600
System Description: Microsoft Windows XP Service Pack 3 (v5.1.0.2600)

Configured using:
 'configure -C --prefix=/d/usr --with-wide-int
 --enable-checking=yes,glyphs 'CFLAGS=-O0 -gdwarf-4 -g3''

Configured features:
ACL GIF GMP GNUTLS HARFBUZZ JPEG JSON LCMS2 LIBXML2 MODULES NOTIFY
W32NOTIFY PDUMPER PNG RSVG SOUND SQLITE3 THREADS TIFF
TOOLKIT_SCROLL_BARS TREE_SITTER WEBP XPM ZLIB

Important settings:
  value of $LANG: ENU
  locale-coding-system: cp1255

Major mode: C/*

Minor modes in effect:
  bug-reference-prog-mode: t
  treesit-explore-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message mailcap yank-media puny dired
dired-loaddefs rfc822 mml mml-sec password-cache epa derived epg rfc6068
epg-config gnus-util text-property-search time-date subr-x mm-decode
mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils vc-git
diff-mode easy-mmode vc-dispatcher bug-reference byte-opt gv bytecomp
byte-compile c-ts-mode c-ts-common treesit cl-seq thingatpt cl-loaddefs
cl-lib find-func rmc iso-transl tooltip cconv eldoc paren electric
uniquify ediff-hook vc-hooks lisp-float-type elisp-mode mwheel dos-w32
ls-lisp disp-table term/w32-win w32-win w32-vars term/common-win
tool-bar dnd fontset image regexp-opt fringe tabulated-list replace
newcomment text-mode lisp-mode prog-mode register page tab-bar menu-bar
rfn-eshadow isearch easymenu timer select scroll-bar mouse jit-lock
font-lock syntax font-core term/tty-colors frame minibuffer nadvice seq
simple cl-generic indonesian philippine cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite emoji-zwj charscript charprop case-table
epa-hook jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button
loaddefs theme-loaddefs faces cus-face macroexp files window
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget keymap hashtable-print-readable backquote threads
w32notify w32 lcms2 multi-tty make-network-process emacs)

Memory information:
((conses 16 97345 9002)
 (symbols 48 7923 0)
 (strings 16 22587 2046)
 (string-bytes 1 609997)
 (vectors 16 12738)
 (vector-slots 8 179418 12186)
 (floats 8 31 52)
 (intervals 40 6159 809)
 (buffers 888 12))

bug#63390: 29.0.90; c-ts-mode fails to recognize functions in xterm.c

[Eli Zaretskii]

Yuan, could you please look into this?

> Date: Tue, 09 May 2023 15:03:08 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> To reproduce:
> 
>   emacs -Q
>   M-x load-library RET c-ts-mode RET
>   C-x C-f src/xterm.c
>   C-u 8290 M-g g
> 
> Observe that the name of the function x_draw_glyph_string_foreground
> is not fontified in font-lock-function-name-face, but in the default
> face.
> 
> Starting treesit-explore-mode seems to indicate that tree-sitter
> interprets this as a function declaration, not a function definition:
> 
>   (function_declarator declarator: (identifier)
>    parameters: 
>     (parameter_list (
>      (parameter_declaration
>       type: (struct_specifier struct name: (type_identifier))
>       declarator: (pointer_declarator * declarator: (identifier)))
>      )))
> 
> Same with the next function, x_draw_composite_glyph_string_foreground.
> But the function after that, x_draw_glyphless_glyph_string_foreground,
> is again recognized as function definition.  I wonder if the
> preprocessor conditionals around there have something to do with that.
> 
> Btw, function declarations in a header file are recognized as such,
> but the names of the functions there are still correctly fontified.
> 
> In GNU Emacs 29.0.90 (build 88, i686-pc-mingw32) of 2023-05-09 built on
>  HOME-C4E4A596F7
> Repository revision: 387ddc0ccc1b21f612b9106bafec63170ede30e6
> Repository branch: emacs-29
> Windowing system distributor 'Microsoft Corp.', version 5.1.2600
> System Description: Microsoft Windows XP Service Pack 3 (v5.1.0.2600)
> 
> Configured using:
>  'configure -C --prefix=/d/usr --with-wide-int
>  --enable-checking=yes,glyphs 'CFLAGS=-O0 -gdwarf-4 -g3''
> 
> Configured features:
> ACL GIF GMP GNUTLS HARFBUZZ JPEG JSON LCMS2 LIBXML2 MODULES NOTIFY
> W32NOTIFY PDUMPER PNG RSVG SOUND SQLITE3 THREADS TIFF
> TOOLKIT_SCROLL_BARS TREE_SITTER WEBP XPM ZLIB
> 
> Important settings:
>   value of $LANG: ENU
>   locale-coding-system: cp1255
> 
> Major mode: C/*
> 
> Minor modes in effect:
>   bug-reference-prog-mode: t
>   treesit-explore-mode: t
>   tooltip-mode: t
>   global-eldoc-mode: t
>   show-paren-mode: t
>   electric-indent-mode: t
>   mouse-wheel-mode: t
>   tool-bar-mode: t
>   menu-bar-mode: t
>   file-name-shadow-mode: t
>   global-font-lock-mode: t
>   font-lock-mode: t
>   blink-cursor-mode: t
>   line-number-mode: t
>   indent-tabs-mode: t
>   transient-mark-mode: t
>   auto-composition-mode: t
>   auto-encryption-mode: t
>   auto-compression-mode: t
> 
> Load-path shadows:
> None found.
> 
> Features:
> (shadow sort mail-extr emacsbug message mailcap yank-media puny dired
> dired-loaddefs rfc822 mml mml-sec password-cache epa derived epg rfc6068
> epg-config gnus-util text-property-search time-date subr-x mm-decode
> mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
> sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils vc-git
> diff-mode easy-mmode vc-dispatcher bug-reference byte-opt gv bytecomp
> byte-compile c-ts-mode c-ts-common treesit cl-seq thingatpt cl-loaddefs
> cl-lib find-func rmc iso-transl tooltip cconv eldoc paren electric
> uniquify ediff-hook vc-hooks lisp-float-type elisp-mode mwheel dos-w32
> ls-lisp disp-table term/w32-win w32-win w32-vars term/common-win
> tool-bar dnd fontset image regexp-opt fringe tabulated-list replace
> newcomment text-mode lisp-mode prog-mode register page tab-bar menu-bar
> rfn-eshadow isearch easymenu timer select scroll-bar mouse jit-lock
> font-lock syntax font-core term/tty-colors frame minibuffer nadvice seq
> simple cl-generic indonesian philippine cham georgian utf-8-lang
> misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
> cp51932 hebrew greek romanian slovak czech european ethiopic indian
> cyrillic chinese composite emoji-zwj charscript charprop case-table
> epa-hook jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button
> loaddefs theme-loaddefs faces cus-face macroexp files window
> text-properties overlay sha1 md5 base64 format env code-pages mule
> custom widget keymap hashtable-print-readable backquote threads
> w32notify w32 lcms2 multi-tty make-network-process emacs)
> 
> Memory information:
> ((conses 16 97345 9002)
>  (symbols 48 7923 0)
>  (strings 16 22587 2046)
>  (string-bytes 1 609997)
>  (vectors 16 12738)
>  (vector-slots 8 179418 12186)
>  (floats 8 31 52)
>  (intervals 40 6159 809)
>  (buffers 888 12))
> 
> 
> 
> 

bug#63390: 29.0.90; c-ts-mode fails to recognize functions in xterm.c

[Yuan Fu]

> On May 12, 2023, at 4:11 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> Yuan, could you please look into this?
> 
>> Date: Tue, 09 May 2023 15:03:08 +0300
>> From: Eli Zaretskii <eliz@gnu.org>
>> 
>> To reproduce:
>> 
>>  emacs -Q
>>  M-x load-library RET c-ts-mode RET
>>  C-x C-f src/xterm.c
>>  C-u 8290 M-g g
>> 
>> Observe that the name of the function x_draw_glyph_string_foreground
>> is not fontified in font-lock-function-name-face, but in the default
>> face.
>> 
>> Starting treesit-explore-mode seems to indicate that tree-sitter
>> interprets this as a function declaration, not a function definition:
>> 
>>  (function_declarator declarator: (identifier)
>>   parameters: 
>>    (parameter_list (
>>     (parameter_declaration
>>      type: (struct_specifier struct name: (type_identifier))
>>      declarator: (pointer_declarator * declarator: (identifier)))
>>     )))
>> 
>> Same with the next function, x_draw_composite_glyph_string_foreground.
>> But the function after that, x_draw_glyphless_glyph_string_foreground,
>> is again recognized as function definition.  I wonder if the
>> preprocessor conditionals around there have something to do with that.

Ok, so that’s because there are ifdef’s inside the function, which cuts the function into pieces and tree-sitter can’t make out a function_definition, which is what we use to fontify the function name. A function_declarator alone can be used in many places, like in an argument list for function pointers, I think?

I can fix this by fontifying top-level function_declaration, I think a top-level function_declaration should always be a function definition?

>> 
>> Btw, function declarations in a header file are recognized as such,
>> but the names of the functions there are still correctly fontified.

They are fine because there’s a semicolon in the end, so the function_decalration is wrapped in a declaration node, which we (the fontification rules) recognize.

Yuan

bug#63390: 29.0.90; c-ts-mode fails to recognize functions in xterm.c

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Sun, 14 May 2023 22:47:06 -0700
> Cc: 63390@debbugs.gnu.org
> 
> >>  (function_declarator declarator: (identifier)
> >>   parameters: 
> >>    (parameter_list (
> >>     (parameter_declaration
> >>      type: (struct_specifier struct name: (type_identifier))
> >>      declarator: (pointer_declarator * declarator: (identifier)))
> >>     )))
> >> 
> >> Same with the next function, x_draw_composite_glyph_string_foreground.
> >> But the function after that, x_draw_glyphless_glyph_string_foreground,
> >> is again recognized as function definition.  I wonder if the
> >> preprocessor conditionals around there have something to do with that.
> 
> Ok, so that’s because there are ifdef’s inside the function, which cuts the function into pieces and tree-sitter can’t make out a function_definition, which is what we use to fontify the function name. A function_declarator alone can be used in many places, like in an argument list for function pointers, I think?

Does this mean that on master movement by defuns will be broken around
those functions?

> I can fix this by fontifying top-level function_declaration, I think a top-level function_declaration should always be a function definition?

Yes, I think this would be better.

> >> Btw, function declarations in a header file are recognized as such,
> >> but the names of the functions there are still correctly fontified.
> 
> They are fine because there’s a semicolon in the end, so the function_decalration is wrapped in a declaration node, which we (the fontification rules) recognize.

Thanks for explaining this.

bug#63390: 29.0.90; c-ts-mode fails to recognize functions in xterm.c

[Yuan Fu]

> On May 15, 2023, at 4:13 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Yuan Fu <casouri@gmail.com>
>> Date: Sun, 14 May 2023 22:47:06 -0700
>> Cc: 63390@debbugs.gnu.org
>> 
>>>> (function_declarator declarator: (identifier)
>>>>  parameters: 
>>>>   (parameter_list (
>>>>    (parameter_declaration
>>>>     type: (struct_specifier struct name: (type_identifier))
>>>>     declarator: (pointer_declarator * declarator: (identifier)))
>>>>    )))
>>>> 
>>>> Same with the next function, x_draw_composite_glyph_string_foreground.
>>>> But the function after that, x_draw_glyphless_glyph_string_foreground,
>>>> is again recognized as function definition.  I wonder if the
>>>> preprocessor conditionals around there have something to do with that.
>> 
>> Ok, so that’s because there are ifdef’s inside the function, which cuts the function into pieces and tree-sitter can’t make out a function_definition, which is what we use to fontify the function name. A function_declarator alone can be used in many places, like in an argument list for function pointers, I think?
> 
> Does this mean that on master movement by defuns will be broken around
> those functions?

Yeah, unfortunately, I’ll try accommodate for it. 

> 
>> I can fix this by fontifying top-level function_declaration, I think a top-level function_declaration should always be a function definition?
> 
> Yes, I think this would be better.
> 
>>>> Btw, function declarations in a header file are recognized as such,
>>>> but the names of the functions there are still correctly fontified.
>> 
>> They are fine because there’s a semicolon in the end, so the function_decalration is wrapped in a declaration node, which we (the fontification rules) recognize.
> 
> Thanks for explaining this.

Should I fix this on emacs-29 or master? Sorry for the delay, I was having some distractions lately ;-)

Yuan

bug#63390: 29.0.90; c-ts-mode fails to recognize functions in xterm.c

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Wed, 17 May 2023 23:19:22 -0700
> Cc: 63390@debbugs.gnu.org
> 
> >> Ok, so that’s because there are ifdef’s inside the function, which cuts the function into pieces and tree-sitter can’t make out a function_definition, which is what we use to fontify the function name. A function_declarator alone can be used in many places, like in an argument list for function pointers, I think?
> > 
> > Does this mean that on master movement by defuns will be broken around
> > those functions?
> 
> Yeah, unfortunately, I’ll try accommodate for it. 
> 
> > 
> >> I can fix this by fontifying top-level function_declaration, I think a top-level function_declaration should always be a function definition?
> > 
> > Yes, I think this would be better.
> > 
> >>>> Btw, function declarations in a header file are recognized as such,
> >>>> but the names of the functions there are still correctly fontified.
> >> 
> >> They are fine because there’s a semicolon in the end, so the function_decalration is wrapped in a declaration node, which we (the fontification rules) recognize.
> > 
> > Thanks for explaining this.
> 
> Should I fix this on emacs-29 or master? Sorry for the delay, I was having some distractions lately ;-)

No need to apologize.  We all have our lives, with their "disasters".

If the fix is relatively simple and safe, I'd prefer this to be fixed
on emacs-29.  But if not, we can fix it later; after all, on emacs-29
this is a relatively rare issue, since we don't use tree-sitter for
movement by defuns there.

Thanks.

bug#63390: 29.0.90; c-ts-mode fails to recognize functions in xterm.c

[Yuan Fu]

> On May 17, 2023, at 11:56 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Yuan Fu <casouri@gmail.com>
>> Date: Wed, 17 May 2023 23:19:22 -0700
>> Cc: 63390@debbugs.gnu.org
>> 
>>>> Ok, so that’s because there are ifdef’s inside the function, which cuts the function into pieces and tree-sitter can’t make out a function_definition, which is what we use to fontify the function name. A function_declarator alone can be used in many places, like in an argument list for function pointers, I think?
>>> 
>>> Does this mean that on master movement by defuns will be broken around
>>> those functions?
>> 
>> Yeah, unfortunately, I’ll try accommodate for it. 
>> 
>>> 
>>>> I can fix this by fontifying top-level function_declaration, I think a top-level function_declaration should always be a function definition?
>>> 
>>> Yes, I think this would be better.
>>> 
>>>>>> Btw, function declarations in a header file are recognized as such,
>>>>>> but the names of the functions there are still correctly fontified.
>>>> 
>>>> They are fine because there’s a semicolon in the end, so the function_decalration is wrapped in a declaration node, which we (the fontification rules) recognize.
>>> 
>>> Thanks for explaining this.
>> 
>> Should I fix this on emacs-29 or master? Sorry for the delay, I was having some distractions lately ;-)
> 
> No need to apologize.  We all have our lives, with their "disasters".
> 
> If the fix is relatively simple and safe, I'd prefer this to be fixed
> on emacs-29.  But if not, we can fix it later; after all, on emacs-29
> this is a relatively rare issue, since we don't use tree-sitter for
> movement by defuns there.
> 
> Thanks.

I pushed a fix for it to emacs-29.

Yuan

bug#63390: 29.0.90; c-ts-mode fails to recognize functions in xterm.c

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Fri, 19 May 2023 16:13:36 -0700
> Cc: 63390-done@debbugs.gnu.org
> 
> > On May 17, 2023, at 11:56 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> > 
> > If the fix is relatively simple and safe, I'd prefer this to be fixed
> > on emacs-29.  But if not, we can fix it later; after all, on emacs-29
> > this is a relatively rare issue, since we don't use tree-sitter for
> > movement by defuns there.
> > 
> > Thanks.
> 
> I pushed a fix for it to emacs-29.

Thanks, but this leads to:

    ELC      progmodes/c-ts-mode.elc

  In end of data:
  progmodes/c-ts-mode.el:770:11: Warning: the function `treesit-node-match-p' is not known to be defined.

The function treesit-node-match-p is only available on master, AFAICT.

bug#63390: 29.0.90; c-ts-mode fails to recognize functions in xterm.c

[Yuan Fu]

> On May 19, 2023, at 10:42 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Yuan Fu <casouri@gmail.com>
>> Date: Fri, 19 May 2023 16:13:36 -0700
>> Cc: 63390-done@debbugs.gnu.org
>> 
>>> On May 17, 2023, at 11:56 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>>> 
>>> If the fix is relatively simple and safe, I'd prefer this to be fixed
>>> on emacs-29.  But if not, we can fix it later; after all, on emacs-29
>>> this is a relatively rare issue, since we don't use tree-sitter for
>>> movement by defuns there.
>>> 
>>> Thanks.
>> 
>> I pushed a fix for it to emacs-29.
> 
> Thanks, but this leads to:
> 
>    ELC      progmodes/c-ts-mode.elc
> 
>  In end of data:
>  progmodes/c-ts-mode.el:770:11: Warning: the function `treesit-node-match-p' is not known to be defined.
> 
> The function treesit-node-match-p is only available on master, AFAICT.

Ah! My bad, now fixed.

Yuan

bug#63390: 29.0.90; c-ts-mode fails to recognize functions in xterm.c

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Sat, 20 May 2023 01:17:37 -0700
> Cc: 63390@debbugs.gnu.org
> 
> >    ELC      progmodes/c-ts-mode.elc
> > 
> >  In end of data:
> >  progmodes/c-ts-mode.el:770:11: Warning: the function `treesit-node-match-p' is not known to be defined.
> > 
> > The function treesit-node-match-p is only available on master, AFAICT.
> 
> Ah! My bad, now fixed.

Thanks.