bug#64204: C/C++ local variables not syntax highlighted if they start with $

bug#64204: C/C++ local variables not syntax highlighted if they start with $

[Joseph Garvin]

[-- Attachment #1: Type: text/plain, Size: 1204 bytes --]

Start emacs with `emacs -Q`, press C-x C-f and type in a filename that
ends in .cpp like "foo.cpp". Then type in:

int main()
{
    int a = 0;
    int b = 0;
    int $ = 0;
    int d$ = 0;
    int $e = 0;
}

Yes it compiles! GCC, Clang, and MSVC all have a non-standard
extension that allows $ signs to appear in C/C++ variable and type
names: https://gcc.gnu.org/onlinedocs/gcc/Dollar-Signs.html

Then a,b,d$ will be highlighted correctly as local vars, but not $ or
$e. The highlighting doesn't appear to work whenever $ is the first or
only character. Since emacs is accepting of the $ sign anywhere else in
the variable name, I suspect supporting $ was intended, even though it's
non-standard. Even if not intended, it's a popular syntax
extension supported by the major current compilers and whose history
goes back to the DEC compiler, so support would be nice.

In GNU Emacs 28.0.50 (build 1, x86_64-pc-linux-gnu, X toolkit, cairo
version 1.16.0, Xaw3d scroll bars)
 of 2021-05-14 built on eruv
Repository revision: efc24f1e0bb127481bcc9c3054c57c4cd3c99aa9
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12013000
System Description: Ubuntu 20.04.6 LTS

[-- Attachment #2: Type: text/html, Size: 1443 bytes --]

bug#64204: C/C++ local variables not syntax highlighted if they start with $

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Joseph Garvin <k04jg02@gmail.com> writes:

> Start emacs with `emacs -Q`, press C-x C-f and type in a filename that
> ends in .cpp like "foo.cpp". Then type in:
>
> int main()
> {
>     int a = 0;
>     int b = 0;
>     int $ = 0;
>     int d$ = 0;
>     int $e = 0;
> }
>
> Yes it compiles! GCC, Clang, and MSVC all have a non-standard
> extension that allows $ signs to appear in C/C++ variable and type
> names: https://gcc.gnu.org/onlinedocs/gcc/Dollar-Signs.html
>
> Then a,b,d$ will be highlighted correctly as local vars, but not $ or
> $e. The highlighting doesn't appear to work whenever $ is the first or
> only character. Since emacs is accepting of the $ sign anywhere else in
> the variable name, I suspect supporting $ was intended, even though it's
> non-standard. Even if not intended, it's a popular syntax
> extension supported by the major current compilers and whose history
> goes back to the DEC compiler, so support would be nice.

This is a bug indeed.  Alan, would you please look into this?

bug#64204: C/C++ local variables not syntax highlighted if they start with $

[Alan Mackenzie]

Hello, Joseph and Po.

On Wed, Jun 21, 2023 at 19:51:50 +0800, Po Lu wrote:
> Joseph Garvin <k04jg02@gmail.com> writes:

> > Start emacs with `emacs -Q`, press C-x C-f and type in a filename that
> > ends in .cpp like "foo.cpp". Then type in:

> > int main()
> > {
> >     int a = 0;
> >     int b = 0;
> >     int $ = 0;
> >     int d$ = 0;
> >     int $e = 0;
> > }

> > Yes it compiles! GCC, Clang, and MSVC all have a non-standard
> > extension that allows $ signs to appear in C/C++ variable and type
> > names: https://gcc.gnu.org/onlinedocs/gcc/Dollar-Signs.html

> > Then a,b,d$ will be highlighted correctly as local vars, but not $ or
> > $e. The highlighting doesn't appear to work whenever $ is the first or
> > only character. Since emacs is accepting of the $ sign anywhere else in
> > the variable name, I suspect supporting $ was intended, even though it's
> > non-standard. Even if not intended, it's a popular syntax
> > extension supported by the major current compilers and whose history
> > goes back to the DEC compiler, so support would be nice.

> This is a bug indeed.  Alan, would you please look into this?

OK, I've amended C Mode, C++ Mode and Objective C Mode to fontify
identifiers starting with a $.

Please apply the patch below to .../lisp/progmodes/cc-langs.el, and
(since Lisp macros have been changed) recompile the entire CC Mode files
and reload them.  (Joseph, if you want any help with the patching or
compiling, feel free to send me private email.)  Then please try out the
new version on your real code, and confirm to me that the bug has been
fixed, or tell me what's still wrong.

Thanks!



diff -r c45cc2208ed4 cc-langs.el
--- a/cc-langs.el	Wed Jun 21 14:58:04 2023 +0000
+++ b/cc-langs.el	Wed Jun 21 19:14:40 2023 +0000
@@ -819,8 +819,9 @@
 keyword.  It's unspecified how far it matches.  Does not contain a \\|
 operator at the top level."
   t    (concat "[" c-alpha "_]")
+  (c c++) (concat "[" c-alpha "_$]")
   java (concat "[" c-alpha "_@]")
-  objc (concat "[" c-alpha "@]")
+  objc (concat "[" c-alpha "@$]")
   pike (concat "[" c-alpha "_`]"))
 (c-lang-defvar c-symbol-start (c-lang-const c-symbol-start))
 
@@ -844,9 +845,10 @@
   t    (concat (c-lang-const c-symbol-start)
 	       "[" (c-lang-const c-symbol-chars) "]\\{,1000\\}")
   pike (concat
-	;; Use the value from C here since the operator backquote is
+	;; Use the value from AWK here since the operator backquote is
 	;; covered by the other alternative.
-	(c-lang-const c-symbol-key c)
+	(c-lang-const c-symbol-key awk
+		      )
 	"\\|"
 	(c-make-keywords-re nil
 	  (c-lang-const c-overloadable-operators))))



-- 
Alan Mackenzie (Nuremberg, Germany).

bug#64204: C/C++ local variables not syntax highlighted if they start with $

[Alan Mackenzie]

[-- Attachment #1: Type: text/plain, Size: 1012 bytes --]

Hello, Po.

Apologies, the patch I sent earlier was defective, not applying cleanly
to the Emacs master branch.  I also should have sent the patch as an
attachment.

I'm now sending the patch as attachment.  It should apply cleanly to the
master branch or emacs-29.


On Wed, Jun 21, 2023 at 19:25:41 +0000, Alan Mackenzie wrote:
> On Wed, Jun 21, 2023 at 19:51:50 +0800, Po Lu wrote:
> > Joseph Garvin <k04jg02@gmail.com> writes:

[ .... ]

> OK, I've amended C Mode, C++ Mode and Objective C Mode to fontify
> identifiers starting with a $.

> Please apply the patch below to .../lisp/progmodes/cc-langs.el, and
> (since Lisp macros have been changed) recompile the entire CC Mode files
> and reload them.  (Joseph, if you want any help with the patching or
> compiling, feel free to send me private email.)  Then please try out the
> new version on your real code, and confirm to me that the bug has been
> fixed, or tell me what's still wrong.

> Thanks!

[ .... ]

-- 
Alan Mackenzie (Nuremberg, Germany).


[-- Attachment #2: diff.20230621b.diff --]
[-- Type: text/plain, Size: 996 bytes --]

diff -r c45cc2208ed4 cc-langs.el
--- a/cc-langs.el	Wed Jun 21 14:58:04 2023 +0000
+++ b/cc-langs.el	Wed Jun 21 19:14:40 2023 +0000
@@ -819,8 +819,9 @@
 keyword.  It's unspecified how far it matches.  Does not contain a \\|
 operator at the top level."
   t    (concat "[" c-alpha "_]")
+  (c c++) (concat "[" c-alpha "_$]")
   java (concat "[" c-alpha "_@]")
-  objc (concat "[" c-alpha "_@]")
+  objc (concat "[" c-alpha "_@$]")
   pike (concat "[" c-alpha "_`]"))
 (c-lang-defvar c-symbol-start (c-lang-const c-symbol-start))
 
@@ -844,9 +845,10 @@
   t    (concat (c-lang-const c-symbol-start)
 	       "[" (c-lang-const c-symbol-chars) "]\\{,1000\\}")
   pike (concat
-	;; Use the value from C here since the operator backquote is
+	;; Use the value from AWK here since the operator backquote is
 	;; covered by the other alternative.
-	(c-lang-const c-symbol-key c)
+	(c-lang-const c-symbol-key awk
+		      )
 	"\\|"
 	(c-make-keywords-re nil
 	  (c-lang-const c-overloadable-operators))))

bug#64204: C/C++ local variables not syntax highlighted if they start with $

[Eli Zaretskii]

> Cc: acm@muc.de, 64204@debbugs.gnu.org
> Date: Wed, 21 Jun 2023 21:06:37 +0000
> From: Alan Mackenzie <acm@muc.de>
> 
> Apologies, the patch I sent earlier was defective, not applying cleanly
> to the Emacs master branch.  I also should have sent the patch as an
> attachment.
> 
> I'm now sending the patch as attachment.  It should apply cleanly to the
> master branch or emacs-29.

Thanks, but shouldn't this be optional behavior, which users should be
able to control?  Standard C disallows '$' in identifiers AFAIK, and
some users might wish to follow the Standard.

bug#64204: C/C++ local variables not syntax highlighted if they start with $

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

> Thanks, but shouldn't this be optional behavior, which users should be
> able to control?  Standard C disallows '$' in identifiers AFAIK, and
> some users might wish to follow the Standard.

There is no other use for the character $, so this is not necessary:
it's either an identifier or a syntax error, and we don't try to fontify
syntax errors.

bug#64204: C/C++ local variables not syntax highlighted if they start with $

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: Alan Mackenzie <acm@muc.de>,  64204@debbugs.gnu.org
> Date: Thu, 22 Jun 2023 13:44:29 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Thanks, but shouldn't this be optional behavior, which users should be
> > able to control?  Standard C disallows '$' in identifiers AFAIK, and
> > some users might wish to follow the Standard.
> 
> There is no other use for the character $, so this is not necessary:
> it's either an identifier or a syntax error, and we don't try to fontify
> syntax errors.

Sorry, I disagree, for the reasons I explained above.  (I don't mind
this feature to be ON by default, though.)

bug#64204: C/C++ local variables not syntax highlighted if they start with $

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

> Sorry, I disagree, for the reasons I explained above.  (I don't mind
> this feature to be ON by default, though.)

Previously, this syntax (_only_ the syntax where an identifier starts
with $; CC Mode has always supported identifiers containing $, which are
also non-standard) would not be fontified at all, or would be fontified
randomly.  Why would that be desirable, and how would it help anyone
follow the Standard?

My objection to adding unnecessary options to CC Mode is because, for
some reason, these options end up requiring counter-intuitive steps to
modify, and slow down CC Mode by a tiny (but not negligible) amount.

For an example, see c-noise-macro-names or c-font-lock-extra-types, both
of which can only be set as file-local variables or before C Mode is
enabled.

bug#64204: C/C++ local variables not syntax highlighted if they start with $

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: acm@muc.de,  64204@debbugs.gnu.org
> Date: Thu, 22 Jun 2023 14:18:37 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Sorry, I disagree, for the reasons I explained above.  (I don't mind
> > this feature to be ON by default, though.)
> 
> Previously, this syntax (_only_ the syntax where an identifier starts
> with $; CC Mode has always supported identifiers containing $, which are
> also non-standard) would not be fontified at all, or would be fontified
> randomly.  Why would that be desirable, and how would it help anyone
> follow the Standard?

All the uses of '$' in identifiers should be controlled by that
option.

> My objection to adding unnecessary options to CC Mode is because, for
> some reason, these options end up requiring counter-intuitive steps to
> modify, and slow down CC Mode by a tiny (but not negligible) amount.

I disagree that this option is unnecessary.

bug#64204: C/C++ local variables not syntax highlighted if they start with $

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

> All the uses of '$' in identifiers should be controlled by that
> option.

CC Mode doesn't make use of identifiers containing $.  No syntax
features are affected, I think, only fontification.

bug#64204: C/C++ local variables not syntax highlighted if they start with $

[Alan Mackenzie]

Hello, Eli.

On Thu, Jun 22, 2023 at 07:50:47 +0300, Eli Zaretskii wrote:
> > Cc: acm@muc.de, 64204@debbugs.gnu.org
> > Date: Wed, 21 Jun 2023 21:06:37 +0000
> > From: Alan Mackenzie <acm@muc.de>

> > Apologies, the patch I sent earlier was defective, not applying cleanly
> > to the Emacs master branch.  I also should have sent the patch as an
> > attachment.

> > I'm now sending the patch as attachment.  It should apply cleanly to the
> > master branch or emacs-29.

> Thanks, but shouldn't this be optional behavior, which users should be
> able to control?  Standard C disallows '$' in identifiers AFAIK, and
> some users might wish to follow the Standard.

The thing is, such an option would be difficult to implement.  At the
moment, c-symbol-start and c-symbol-key (regular expressions which match
the start of an identifier and all of one) are calculated in cc-langs.el
at compilation time.  They are widely used (~40 times) in the indentation
engine and fontification.

It would be necessary to change these constants to variables, probably
doubling the number of relevant c-lang-const's in cc-langs.el, and to add
a mechanism to toggle between them at run time.

I don't feel that the benefit from this new option would really be worth
the work it would take to implement.  I don't think people are going to
be putting $ signs at the start of identifiers by accident.  CC Mode has
always fontified $s in the middle (or at the end) of identifiers, and
this doesn't seem to have caused problems.

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#64204: C/C++ local variables not syntax highlighted if they start with $

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: acm@muc.de,  64204@debbugs.gnu.org
> Date: Thu, 22 Jun 2023 15:15:34 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > All the uses of '$' in identifiers should be controlled by that
> > option.
> 
> CC Mode doesn't make use of identifiers containing $.  No syntax
> features are affected, I think, only fontification.

I _am_ talking about fontifications.  One of the main purposes of
fontifying code is to give the programmer feedback regarding the
syntax and semantics of the code.  Flagging invalid syntax is
therefore important, and when $ are invalid in identifiers, showing
that with fontifications is important.

bug#64204: C/C++ local variables not syntax highlighted if they start with $

[Eli Zaretskii]

> Date: Thu, 22 Jun 2023 09:11:38 +0000
> Cc: luangruo@yahoo.com, 64204@debbugs.gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > Thanks, but shouldn't this be optional behavior, which users should be
> > able to control?  Standard C disallows '$' in identifiers AFAIK, and
> > some users might wish to follow the Standard.
> 
> The thing is, such an option would be difficult to implement.  At the
> moment, c-symbol-start and c-symbol-key (regular expressions which match
> the start of an identifier and all of one) are calculated in cc-langs.el
> at compilation time.  They are widely used (~40 times) in the indentation
> engine and fontification.
> 
> It would be necessary to change these constants to variables, probably
> doubling the number of relevant c-lang-const's in cc-langs.el, and to add
> a mechanism to toggle between them at run time.
> 
> I don't feel that the benefit from this new option would really be worth
> the work it would take to implement.

I do feel it's worth the effort.  If we value correct fontification,
of course.  I don't know what my opinions on this mean for you, but
there you have it.  I cannot force you make this change, of course, I
can only try to convince.

bug#64204: C/C++ local variables not syntax highlighted if they start with $

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

> I _am_ talking about fontifications.  One of the main purposes of
> fontifying code is to give the programmer feedback regarding the
> syntax and semantics of the code.  Flagging invalid syntax is
> therefore important, and when $ are invalid in identifiers, showing
> that with fontifications is important.

But CC Mode fontification has never been intended to flag invalid
syntax: instead of not being fontified at all, invalid syntax is simply
fontified every which way.  This is not a problem one option can solve;
it will require large scale redesign of CC Mode, and detecting invalid
syntax is more the forte of c-ts-mode anyway.

Thanks.

bug#64204: C/C++ local variables not syntax highlighted if they start with $

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: acm@muc.de,  64204@debbugs.gnu.org
> Date: Thu, 22 Jun 2023 19:46:41 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I _am_ talking about fontifications.  One of the main purposes of
> > fontifying code is to give the programmer feedback regarding the
> > syntax and semantics of the code.  Flagging invalid syntax is
> > therefore important, and when $ are invalid in identifiers, showing
> > that with fontifications is important.
> 
> But CC Mode fontification has never been intended to flag invalid
> syntax: instead of not being fontified at all, invalid syntax is simply
> fontified every which way.

It is fontified the way the syntax is interpreted, which is exactly
what I'm talking about.

> This is not a problem one option can solve; it will require large
> scale redesign of CC Mode, and detecting invalid syntax is more the
> forte of c-ts-mode anyway.

No redesign is needed.

bug#64204: C/C++ local variables not syntax highlighted if they start with $

[Alan Mackenzie]

Hello, Eli.

On Thu, Jun 22, 2023 at 13:13:59 +0300, Eli Zaretskii wrote:
> > Date: Thu, 22 Jun 2023 09:11:38 +0000
> > Cc: luangruo@yahoo.com, 64204@debbugs.gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > > Thanks, but shouldn't this be optional behavior, which users should be
> > > able to control?  Standard C disallows '$' in identifiers AFAIK, and
> > > some users might wish to follow the Standard.

> > The thing is, such an option would be difficult to implement.  At the
> > moment, c-symbol-start and c-symbol-key (regular expressions which match
> > the start of an identifier and all of one) are calculated in cc-langs.el
> > at compilation time.  They are widely used (~40 times) in the indentation
> > engine and fontification.

> > It would be necessary to change these constants to variables, probably
> > doubling the number of relevant c-lang-const's in cc-langs.el, and to add
> > a mechanism to toggle between them at run time.

> > I don't feel that the benefit from this new option would really be worth
> > the work it would take to implement.

> I do feel it's worth the effort.  If we value correct fontification,
> of course.  I don't know what my opinions on this mean for you, but
> there you have it.  I cannot force you make this change, of course, I
> can only try to convince.

Having thought about this some more, maybe it's not such a big problem.
The strategy I'm thinking about would not attempt to analyse identifiers
with $ in them.  Instead, it would amend only the fontification, with a
new ad-hoc entry in font-lock-keywords which would go over all
identifiers at the end of the fontification, and splat those containing a
$ with font-lock-warning-face.

I can see more problems arising when people start wanting to use
identifiers with random Unicode characters in them.  :-(

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#64204: C/C++ local variables not syntax highlighted if they start with $

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

> It is fontified the way the syntax is interpreted, which is exactly
> what I'm talking about.

The problem is that:

  int $lsp;

is not interpreted as an ``error'', just as some syntax that cannot be
fontified.  CC Mode is not in any position to determine whether or not
syntax is valid.

bug#64204: C/C++ local variables not syntax highlighted if they start with $

[Alan Mackenzie]

Hello, Joseph and Eli.

On Thu, Jun 22, 2023 at 13:13:59 +0300, Eli Zaretskii wrote:
> > Date: Thu, 22 Jun 2023 09:11:38 +0000
> > Cc: luangruo@yahoo.com, 64204@debbugs.gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > > Thanks, but shouldn't this be optional behavior, which users should be
> > > able to control?  Standard C disallows '$' in identifiers AFAIK, and
> > > some users might wish to follow the Standard.

> > The thing is, such an option would be difficult to implement.  At the
> > moment, c-symbol-start and c-symbol-key (regular expressions which match
> > the start of an identifier and all of one) are calculated in cc-langs.el
> > at compilation time.  They are widely used (~40 times) in the indentation
> > engine and fontification.

> > It would be necessary to change these constants to variables, probably
> > doubling the number of relevant c-lang-const's in cc-langs.el, and to add
> > a mechanism to toggle between them at run time.

> > I don't feel that the benefit from this new option would really be worth
> > the work it would take to implement.

> I do feel it's worth the effort.  If we value correct fontification,
> of course.  I don't know what my opinions on this mean for you, but
> there you have it.  I cannot force you make this change, of course, I
> can only try to convince.

:-)  OK, the fontification of identifiers with $ is now working.  By
default they are fontified as normal identifiers (i.e. with
font-lock-type-face, font-lock-variable-name-face, or
font-lock-function-name-face), but if the (new) customisable option
c-warn-ids-with-dollar is set to non-nil, font-lock-warning-face is used
instead.  The above applies only to C, C++, and Objective C Modes, and to
any derived modes which enable the mechanism.

So, I'm closing the bug now with this post.

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#64204: C/C++ local variables not syntax highlighted if they start with $

[Eli Zaretskii]

> Date: Wed, 28 Jun 2023 18:45:09 +0000
> Cc: luangruo@yahoo.com, 64204-done@debbugs.gnu.org, acm@muc.de
> From: Alan Mackenzie <acm@muc.de>
> 
> > I do feel it's worth the effort.  If we value correct fontification,
> > of course.  I don't know what my opinions on this mean for you, but
> > there you have it.  I cannot force you make this change, of course, I
> > can only try to convince.
> 
> :-)  OK, the fontification of identifiers with $ is now working.  By
> default they are fontified as normal identifiers (i.e. with
> font-lock-type-face, font-lock-variable-name-face, or
> font-lock-function-name-face), but if the (new) customisable option
> c-warn-ids-with-dollar is set to non-nil, font-lock-warning-face is used
> instead.  The above applies only to C, C++, and Objective C Modes, and to
> any derived modes which enable the mechanism.

Thanks!

bug#64204: C/C++ local variables not syntax highlighted if they start with $

[Joseph Garvin]

[-- Attachment #1: Type: text/plain, Size: 995 bytes --]

Thanks, appreciate the work!

On Wed, Jun 28, 2023, 1:50 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > Date: Wed, 28 Jun 2023 18:45:09 +0000
> > Cc: luangruo@yahoo.com, 64204-done@debbugs.gnu.org, acm@muc.de
> > From: Alan Mackenzie <acm@muc.de>
> >
> > > I do feel it's worth the effort.  If we value correct fontification,
> > > of course.  I don't know what my opinions on this mean for you, but
> > > there you have it.  I cannot force you make this change, of course, I
> > > can only try to convince.
> >
> > :-)  OK, the fontification of identifiers with $ is now working.  By
> > default they are fontified as normal identifiers (i.e. with
> > font-lock-type-face, font-lock-variable-name-face, or
> > font-lock-function-name-face), but if the (new) customisable option
> > c-warn-ids-with-dollar is set to non-nil, font-lock-warning-face is used
> > instead.  The above applies only to C, C++, and Objective C Modes, and to
> > any derived modes which enable the mechanism.
>
> Thanks!
>

[-- Attachment #2: Type: text/html, Size: 1668 bytes --]