bug#63840: 29.0.91; c-ts-mode fails to fontify common C extension

bug#63840: 29.0.91; c-ts-mode fails to fontify common C extension

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

A common extension to the C language is the use of the `$' sign inside
external identifier names.

This is permitted by GNU CC on most systems it supports, and also
supported by CC Mode.  However, it does not work on c-ts-mode:

sl$loadarena ()
{
  struct FAB fab;
  register int status;
  extern char *sl$dfname;

  fab = cc$rms_fab;
  fab.fab$b_fac = FAB$M_BIO | FAB$M_GET;
  fab.fab$l_fna = sl$dfname;
  fab.fab$b_fns = strlen (sl$dfname);
  status = sys$open (&fab);
  if (status != RMS$_NORMAL)
    return status;
  /* More code below.  */
}

in the identifier name of this function declarator, `sl' is fontified as
a type.  `$' is not fontified at all.  `loadarena' is fontified as an
identifier name, as it should be.  In:

  extern char *sl$defname;

only `dfname' is fontified as a variable name; in CC Mode, all
fontification is correct.

This should be a simple change in c-ts-mode's identifier fontifying
regexp to include `$', right?  I don't know enough about tree-sitter to
fix it myself.

Thanks.

bug#63840: 29.0.91; c-ts-mode fails to fontify common C extension

[Eli Zaretskii]

> Date: Fri, 02 Jun 2023 20:11:33 +0800
> From:  Po Lu via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> A common extension to the C language is the use of the `$' sign inside
> external identifier names.
> 
> This is permitted by GNU CC on most systems it supports, and also
> supported by CC Mode.  However, it does not work on c-ts-mode:
> 
> sl$loadarena ()
> {
>   struct FAB fab;
>   register int status;
>   extern char *sl$dfname;
> 
>   fab = cc$rms_fab;
>   fab.fab$b_fac = FAB$M_BIO | FAB$M_GET;
>   fab.fab$l_fna = sl$dfname;
>   fab.fab$b_fns = strlen (sl$dfname);
>   status = sys$open (&fab);
>   if (status != RMS$_NORMAL)
>     return status;
>   /* More code below.  */
> }
> 
> in the identifier name of this function declarator, `sl' is fontified as
> a type.  `$' is not fontified at all.  `loadarena' is fontified as an
> identifier name, as it should be.  In:
> 
>   extern char *sl$defname;
> 
> only `dfname' is fontified as a variable name; in CC Mode, all
> fontification is correct.
> 
> This should be a simple change in c-ts-mode's identifier fontifying
> regexp to include `$', right?  I don't know enough about tree-sitter to
> fix it myself.

I'm not sure this can be fixed unless the tree-sitter's C grammar
supports such identifiers, but if we can fix it in our code, we
should.

Yuan and Theo, can you please look into this?

bug#63840: 29.0.91; c-ts-mode fails to fontify common C extension

[Yuan Fu]

> On Jun 2, 2023, at 5:51 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> Date: Fri, 02 Jun 2023 20:11:33 +0800
>> From:  Po Lu via "Bug reports for GNU Emacs,
>> the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
>> 
>> A common extension to the C language is the use of the `$' sign inside
>> external identifier names.
>> 
>> This is permitted by GNU CC on most systems it supports, and also
>> supported by CC Mode.  However, it does not work on c-ts-mode:
>> 
>> sl$loadarena ()
>> {
>>  struct FAB fab;
>>  register int status;
>>  extern char *sl$dfname;
>> 
>>  fab = cc$rms_fab;
>>  fab.fab$b_fac = FAB$M_BIO | FAB$M_GET;
>>  fab.fab$l_fna = sl$dfname;
>>  fab.fab$b_fns = strlen (sl$dfname);
>>  status = sys$open (&fab);
>>  if (status != RMS$_NORMAL)
>>    return status;
>>  /* More code below.  */
>> }
>> 
>> in the identifier name of this function declarator, `sl' is fontified as
>> a type.  `$' is not fontified at all.  `loadarena' is fontified as an
>> identifier name, as it should be.  In:
>> 
>>  extern char *sl$defname;
>> 
>> only `dfname' is fontified as a variable name; in CC Mode, all
>> fontification is correct.
>> 
>> This should be a simple change in c-ts-mode's identifier fontifying
>> regexp to include `$', right?  I don't know enough about tree-sitter to
>> fix it myself.
> 
> I'm not sure this can be fixed unless the tree-sitter's C grammar
> supports such identifiers, but if we can fix it in our code, we
> should.
> 
> Yuan and Theo, can you please look into this?

Yeah. So is this the only way $ is used in C? Could it appear in contexts other than identifiers?

Yuan

bug#63840: 29.0.91; c-ts-mode fails to fontify common C extension

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Yuan Fu <casouri@gmail.com> writes:

> Yeah. So is this the only way $ is used in C?

Yes.

> Could it appear in contexts other than identifiers?

No.

bug#63840: 29.0.91; c-ts-mode fails to fontify common C extension

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Theodor Thornhill <theo@thornhill.no>,
>   63840@debbugs.gnu.org
> Date: Thu, 08 Jun 2023 16:50:21 +0800
> 
> Yuan Fu <casouri@gmail.com> writes:
> 
> > Yeah. So is this the only way $ is used in C?
> 
> Yes.
> 
> > Could it appear in contexts other than identifiers?
> 
> No.

Ping!  Any progress with this?

bug#63840: 29.0.91; c-ts-mode fails to fontify common C extension

[Yuan Fu]

> On Jun 14, 2023, at 10:56 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Po Lu <luangruo@yahoo.com>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  Theodor Thornhill <theo@thornhill.no>,
>>  63840@debbugs.gnu.org
>> Date: Thu, 08 Jun 2023 16:50:21 +0800
>> 
>> Yuan Fu <casouri@gmail.com> writes:
>> 
>>> Yeah. So is this the only way $ is used in C?
>> 
>> Yes.
>> 
>>> Could it appear in contexts other than identifiers?
>> 
>> No.
> 
> Ping!  Any progress with this?

The best option is to ask tree-sitter-c to recognize $. I see a PR for it but the author closed it shortly after posting, not sure what was going on[1].

Fixing it ourselves involves checking every identifier during fontification. Is that something we want to do?

[1] https://github.com/tree-sitter/tree-sitter-c/pull/114

Yuan

bug#63840: 29.0.91; c-ts-mode fails to fontify common C extension

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Thu, 15 Jun 2023 22:05:33 -0700
> Cc: Theodor Thornhill <theo@thornhill.no>,
>  Po Lu <luangruo@yahoo.com>,
>  63840@debbugs.gnu.org
> 
> > Ping!  Any progress with this?
> 
> The best option is to ask tree-sitter-c to recognize $. I see a PR for it but the author closed it shortly after posting, not sure what was going on[1].

Any hope of re-opening it, or at least asking why it was closed
without fixing?

> Fixing it ourselves involves checking every identifier during fontification. Is that something we want to do?

How would that work?  Can you describe how could this be implemented?
It is hard to decide whether it's something we want to do without
knowing the implications.

bug#63840: 29.0.91; c-ts-mode fails to fontify common C extension

[Yuan Fu]

> On Jun 15, 2023, at 11:31 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Yuan Fu <casouri@gmail.com>
>> Date: Thu, 15 Jun 2023 22:05:33 -0700
>> Cc: Theodor Thornhill <theo@thornhill.no>,
>> Po Lu <luangruo@yahoo.com>,
>> 63840@debbugs.gnu.org
>> 
>>> Ping!  Any progress with this?
>> 
>> The best option is to ask tree-sitter-c to recognize $. I see a PR for it but the author closed it shortly after posting, not sure what was going on[1].
> 
> Any hope of re-opening it, or at least asking why it was closed
> without fixing?

I asked on GitHub.

> 
>> Fixing it ourselves involves checking every identifier during fontification. Is that something we want to do?
> 
> How would that work?  Can you describe how could this be implemented?
> It is hard to decide whether it's something we want to do without
> knowing the implications.

I was going to write a demo implementation, but that actually would change quite some lines. So let me just describe it for now. In c-ts-mode--font-lock-settings, I would replace all the different types of faces used for all the identifiers to a function. (If you search for identifier in c-ts-mode--font-lock-settings, there are quite a few of them.) This function would obviously receive the beg and end of the identifier tree-sitter detects, and it would check if there are $ right before or after the given range, if there is, it would extend the range fontified.

For example, for a code like abc$de, tree-sitter might consider abc as an identifier, and $ de to be error. The function I mentioned would look at abc’s beg and end, and see that c is immediately followed by a $, so it would extend the fontification range to include abc$de.

Yuan

bug#63840: 29.0.91; c-ts-mode fails to fontify common C extension

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Sun, 18 Jun 2023 23:43:51 -0700
> Cc: Theodor Thornhill <theo@thornhill.no>,
>  Po Lu <luangruo@yahoo.com>,
>  63840@debbugs.gnu.org
> 
> > On Jun 15, 2023, at 11:31 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> > 
> >> From: Yuan Fu <casouri@gmail.com>
> >> Date: Thu, 15 Jun 2023 22:05:33 -0700
> >> Cc: Theodor Thornhill <theo@thornhill.no>,
> >> Po Lu <luangruo@yahoo.com>,
> >> 63840@debbugs.gnu.org
> >> 
> >>> Ping!  Any progress with this?
> >> 
> >> The best option is to ask tree-sitter-c to recognize $. I see a PR for it but the author closed it shortly after posting, not sure what was going on[1].
> > 
> > Any hope of re-opening it, or at least asking why it was closed
> > without fixing?
> 
> I asked on GitHub.
> 
> > 
> >> Fixing it ourselves involves checking every identifier during fontification. Is that something we want to do?
> > 
> > How would that work?  Can you describe how could this be implemented?
> > It is hard to decide whether it's something we want to do without
> > knowing the implications.
> 
> I was going to write a demo implementation, but that actually would change quite some lines. So let me just describe it for now. In c-ts-mode--font-lock-settings, I would replace all the different types of faces used for all the identifiers to a function. (If you search for identifier in c-ts-mode--font-lock-settings, there are quite a few of them.) This function would obviously receive the beg and end of the identifier tree-sitter detects, and it would check if there are $ right before or after the given range, if there is, it would extend the range fontified.
> 
> For example, for a code like abc$de, tree-sitter might consider abc as an identifier, and $ de to be error. The function I mentioned would look at abc’s beg and end, and see that c is immediately followed by a $, so it would extend the fontification range to include abc$de.

Thanks.  This sounds like a lot of hair, so let's first see how the
developers of the tree-sitter-c grammar respond.

bug#63840: 29.0.91; c-ts-mode fails to fontify common C extension

[Yuan Fu]

> On Jun 19, 2023, at 9:23 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Yuan Fu <casouri@gmail.com>
>> Date: Sun, 18 Jun 2023 23:43:51 -0700
>> Cc: Theodor Thornhill <theo@thornhill.no>,
>> Po Lu <luangruo@yahoo.com>,
>> 63840@debbugs.gnu.org
>> 
>>> On Jun 15, 2023, at 11:31 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>>> 
>>>> From: Yuan Fu <casouri@gmail.com>
>>>> Date: Thu, 15 Jun 2023 22:05:33 -0700
>>>> Cc: Theodor Thornhill <theo@thornhill.no>,
>>>> Po Lu <luangruo@yahoo.com>,
>>>> 63840@debbugs.gnu.org
>>>> 
>>>>> Ping!  Any progress with this?
>>>> 
>>>> The best option is to ask tree-sitter-c to recognize $. I see a PR for it but the author closed it shortly after posting, not sure what was going on[1].
>>> 
>>> Any hope of re-opening it, or at least asking why it was closed
>>> without fixing?
>> 
>> I asked on GitHub.
>> 
>>> 
>>>> Fixing it ourselves involves checking every identifier during fontification. Is that something we want to do?
>>> 
>>> How would that work?  Can you describe how could this be implemented?
>>> It is hard to decide whether it's something we want to do without
>>> knowing the implications.
>> 
>> I was going to write a demo implementation, but that actually would change quite some lines. So let me just describe it for now. In c-ts-mode--font-lock-settings, I would replace all the different types of faces used for all the identifiers to a function. (If you search for identifier in c-ts-mode--font-lock-settings, there are quite a few of them.) This function would obviously receive the beg and end of the identifier tree-sitter detects, and it would check if there are $ right before or after the given range, if there is, it would extend the range fontified.
>> 
>> For example, for a code like abc$de, tree-sitter might consider abc as an identifier, and $ de to be error. The function I mentioned would look at abc’s beg and end, and see that c is immediately followed by a $, so it would extend the fontification range to include abc$de.
> 
> Thanks.  This sounds like a lot of hair, so let's first see how the
> developers of the tree-sitter-c grammar respond.

Cool. (Not surprisingly) I didn’t get any reply on GitHub as for now. It would be nice if tree-sitter-c (and other grammars) can have a separate maintainer other than tree-sitter’s author. He seems to be heavily overloaded right now (tree-sitter, a dozen grammars, Zed editor).

Yuan

bug#63840: 29.0.91; c-ts-mode fails to fontify common C extension

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Mon, 26 Jun 2023 22:54:16 -0700
> Cc: Theodor Thornhill <theo@thornhill.no>,
>  Po Lu <luangruo@yahoo.com>,
>  63840@debbugs.gnu.org
> 
> 
> 
> > On Jun 19, 2023, at 9:23 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> > 
> >> From: Yuan Fu <casouri@gmail.com>
> >> Date: Sun, 18 Jun 2023 23:43:51 -0700
> >> Cc: Theodor Thornhill <theo@thornhill.no>,
> >> Po Lu <luangruo@yahoo.com>,
> >> 63840@debbugs.gnu.org
> >> 
> >>> On Jun 15, 2023, at 11:31 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> >>> 
> >>>> From: Yuan Fu <casouri@gmail.com>
> >>>> Date: Thu, 15 Jun 2023 22:05:33 -0700
> >>>> Cc: Theodor Thornhill <theo@thornhill.no>,
> >>>> Po Lu <luangruo@yahoo.com>,
> >>>> 63840@debbugs.gnu.org
> >>>> 
> >>>>> Ping!  Any progress with this?
> >>>> 
> >>>> The best option is to ask tree-sitter-c to recognize $. I see a PR for it but the author closed it shortly after posting, not sure what was going on[1].
> >>> 
> >>> Any hope of re-opening it, or at least asking why it was closed
> >>> without fixing?
> >> 
> >> I asked on GitHub.
> >> 
> >>> 
> >>>> Fixing it ourselves involves checking every identifier during fontification. Is that something we want to do?
> >>> 
> >>> How would that work?  Can you describe how could this be implemented?
> >>> It is hard to decide whether it's something we want to do without
> >>> knowing the implications.
> >> 
> >> I was going to write a demo implementation, but that actually would change quite some lines. So let me just describe it for now. In c-ts-mode--font-lock-settings, I would replace all the different types of faces used for all the identifiers to a function. (If you search for identifier in c-ts-mode--font-lock-settings, there are quite a few of them.) This function would obviously receive the beg and end of the identifier tree-sitter detects, and it would check if there are $ right before or after the given range, if there is, it would extend the range fontified.
> >> 
> >> For example, for a code like abc$de, tree-sitter might consider abc as an identifier, and $ de to be error. The function I mentioned would look at abc’s beg and end, and see that c is immediately followed by a $, so it would extend the fontification range to include abc$de.
> > 
> > Thanks.  This sounds like a lot of hair, so let's first see how the
> > developers of the tree-sitter-c grammar respond.
> 
> Cool. (Not surprisingly) I didn’t get any reply on GitHub as for now. It would be nice if tree-sitter-c (and other grammars) can have a separate maintainer other than tree-sitter’s author. He seems to be heavily overloaded right now (tree-sitter, a dozen grammars, Zed editor).

Would it be possible for you or someone else of the interested Emacs
users to submit a patch for the grammar library to support $, either
as an option or as an extension?  IOW, is it hard to modify the
TS grammar definitions for such relatively simple and straightforward
extensions?

bug#63840: 29.0.91; c-ts-mode fails to fontify common C extension

[Yuan Fu]

> On Jun 27, 2023, at 4:05 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Yuan Fu <casouri@gmail.com>
>> Date: Mon, 26 Jun 2023 22:54:16 -0700
>> Cc: Theodor Thornhill <theo@thornhill.no>,
>> Po Lu <luangruo@yahoo.com>,
>> 63840@debbugs.gnu.org
>> 
>> 
>> 
>>> On Jun 19, 2023, at 9:23 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>>> 
>>>> From: Yuan Fu <casouri@gmail.com>
>>>> Date: Sun, 18 Jun 2023 23:43:51 -0700
>>>> Cc: Theodor Thornhill <theo@thornhill.no>,
>>>> Po Lu <luangruo@yahoo.com>,
>>>> 63840@debbugs.gnu.org
>>>> 
>>>>> On Jun 15, 2023, at 11:31 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>>>>> 
>>>>>> From: Yuan Fu <casouri@gmail.com>
>>>>>> Date: Thu, 15 Jun 2023 22:05:33 -0700
>>>>>> Cc: Theodor Thornhill <theo@thornhill.no>,
>>>>>> Po Lu <luangruo@yahoo.com>,
>>>>>> 63840@debbugs.gnu.org
>>>>>> 
>>>>>>> Ping!  Any progress with this?
>>>>>> 
>>>>>> The best option is to ask tree-sitter-c to recognize $. I see a PR for it but the author closed it shortly after posting, not sure what was going on[1].
>>>>> 
>>>>> Any hope of re-opening it, or at least asking why it was closed
>>>>> without fixing?
>>>> 
>>>> I asked on GitHub.
>>>> 
>>>>> 
>>>>>> Fixing it ourselves involves checking every identifier during fontification. Is that something we want to do?
>>>>> 
>>>>> How would that work?  Can you describe how could this be implemented?
>>>>> It is hard to decide whether it's something we want to do without
>>>>> knowing the implications.
>>>> 
>>>> I was going to write a demo implementation, but that actually would change quite some lines. So let me just describe it for now. In c-ts-mode--font-lock-settings, I would replace all the different types of faces used for all the identifiers to a function. (If you search for identifier in c-ts-mode--font-lock-settings, there are quite a few of them.) This function would obviously receive the beg and end of the identifier tree-sitter detects, and it would check if there are $ right before or after the given range, if there is, it would extend the range fontified.
>>>> 
>>>> For example, for a code like abc$de, tree-sitter might consider abc as an identifier, and $ de to be error. The function I mentioned would look at abc’s beg and end, and see that c is immediately followed by a $, so it would extend the fontification range to include abc$de.
>>> 
>>> Thanks.  This sounds like a lot of hair, so let's first see how the
>>> developers of the tree-sitter-c grammar respond.
>> 
>> Cool. (Not surprisingly) I didn’t get any reply on GitHub as for now. It would be nice if tree-sitter-c (and other grammars) can have a separate maintainer other than tree-sitter’s author. He seems to be heavily overloaded right now (tree-sitter, a dozen grammars, Zed editor).
> 
> Would it be possible for you or someone else of the interested Emacs
> users to submit a patch for the grammar library to support $, either
> as an option or as an extension?  IOW, is it hard to modify the
> TS grammar definitions for such relatively simple and straightforward
> extensions?

There is a PR for it on GitHub (under which I left a comment). Judging from the PR, it’s simple to add support for $.

Yuan

bug#63840: 29.0.91; c-ts-mode fails to fontify common C extension

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Tue, 27 Jun 2023 18:46:24 -0700
> Cc: Theodor Thornhill <theo@thornhill.no>,
>  Po Lu <luangruo@yahoo.com>,
>  63840@debbugs.gnu.org
> 
> > Would it be possible for you or someone else of the interested Emacs
> > users to submit a patch for the grammar library to support $, either
> > as an option or as an extension?  IOW, is it hard to modify the
> > TS grammar definitions for such relatively simple and straightforward
> > extensions?
> 
> There is a PR for it on GitHub (under which I left a comment). Judging from the PR, it’s simple to add support for $.

If so, perhaps just pinging about it from time to time is all that
should be done.

bug#63840: 29.0.91; c-ts-mode fails to fontify common C extension

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Yuan Fu <casouri@gmail.com>
>> Date: Tue, 27 Jun 2023 18:46:24 -0700
>> Cc: Theodor Thornhill <theo@thornhill.no>,
>>  Po Lu <luangruo@yahoo.com>,
>>  63840@debbugs.gnu.org
>>
>> > Would it be possible for you or someone else of the interested Emacs
>> > users to submit a patch for the grammar library to support $, either
>> > as an option or as an extension?  IOW, is it hard to modify the
>> > TS grammar definitions for such relatively simple and straightforward
>> > extensions?
>>
>> There is a PR for it on GitHub (under which I left a comment). Judging from the PR, it’s simple to add support for $.
>
> If so, perhaps just pinging about it from time to time is all that
> should be done.

Does it make sense to keep this bug open on our side if we only intend
to wait for upstream support?  Or should we close it as wontfix?

bug#63840: 29.0.91; c-ts-mode fails to fontify common C extension

[Eli Zaretskii]

> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Sun, 1 Oct 2023 14:10:15 -0700
> Cc: luangruo@yahoo.com, 63840@debbugs.gnu.org, theo@thornhill.no
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Yuan Fu <casouri@gmail.com>
> >> Date: Tue, 27 Jun 2023 18:46:24 -0700
> >> Cc: Theodor Thornhill <theo@thornhill.no>,
> >>  Po Lu <luangruo@yahoo.com>,
> >>  63840@debbugs.gnu.org
> >>
> >> > Would it be possible for you or someone else of the interested Emacs
> >> > users to submit a patch for the grammar library to support $, either
> >> > as an option or as an extension?  IOW, is it hard to modify the
> >> > TS grammar definitions for such relatively simple and straightforward
> >> > extensions?
> >>
> >> There is a PR for it on GitHub (under which I left a comment). Judging from the PR, it’s simple to add support for $.
> >
> > If so, perhaps just pinging about it from time to time is all that
> > should be done.
> 
> Does it make sense to keep this bug open on our side if we only intend
> to wait for upstream support?  Or should we close it as wontfix?

We could indeed close it.

bug#63840: 29.0.91; c-ts-mode fails to fontify common C extension

[Stefan Kangas]

tags 63840 notabug
close 63840
thanks

Eli Zaretskii <eliz@gnu.org> writes:

>> Does it make sense to keep this bug open on our side if we only intend
>> to wait for upstream support?  Or should we close it as wontfix?
>
> We could indeed close it.

Done.