bug#24377: 25.1; CC-Mode (C++) uses font-lock-type-face for ordinary variable

bug#24377: 25.1; CC-Mode (C++) uses font-lock-type-face for ordinary variable

[Bastian Beischer]

This piece of C++ code is wrongly fontified by CC-Mode:

int b = 5;
int a = (b*3);

Reproduce as follows:

1) emacs -Q
2) Open file "test.C"
3) Paste snippet in buffer
4) Place cursor over "b" in second line
5) M-x describe-face -> font-lock-type-face

This appears to be correlated with the presence of the opening
parentheses and the '*' character, because:

1) int a = b*3;
does not have this problem

2) "b" gets fontified incorrectly after entering the '*' character:
int a = (b*



In GNU Emacs 25.1.2 (x86_64-unknown-linux-gnu, GTK+ Version 2.24.30)
 of 2016-08-30 built on beischer-w520
Repository revision: 17197d0ae9eac100b56078b2a48964b29710b76e
Windowing system distributor 'The X.Org Foundation', version 11.0.11804000
Configured using:
 'configure --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib
 --localstatedir=/var --mandir=/usr/share/man --with-gameuser=:games
 --with-sound=alsa --with-xft --with-modules --with-x-toolkit=gtk2
 --with-gconf --without-gsettings 'CFLAGS=-march=native -O2 -pipe
 -fstack-protector-strong' CPPFLAGS=-D_FORTIFY_SOURCE=2
 LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro'

Configured features:
XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GCONF NOTIFY ACL
GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB TOOLKIT_SCROLL_BARS
GTK2 X11 MODULES

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: C++/l

Minor modes in effect:
  diff-auto-refine-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t
  abbrev-mode: t

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
Followed link to /home/beischer/.vc/.emacs
Mark saved where search started
"Bastian Beischer"
"bastian.beischer@gmail.com"
"gmail.com"
Quit

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message dired format-spec rfc822 mml
mml-sec password-cache epg epg-config gnus-util mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail
rfc2047 rfc2045 ietf-drums mm-util help-fns mail-prsvr mail-utils
cl-extra help-mode cc-mode cc-fonts cc-guess cc-menus cc-cmds cc-styles
cc-align cc-engine cc-vars cc-defs cl-loaddefs pcase cl-lib misearch
multi-isearch vc-git diff-mode easymenu easy-mmode time-date mule-util
tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel x-win term/common-win x-dnd tool-bar dnd fontset image regexp-opt
fringe tabulated-list newcomment elisp-mode lisp-mode prog-mode register
page menu-bar rfn-eshadow timer select scroll-bar mouse jit-lock
font-lock syntax facemenu font-core frame cl-generic cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese charscript case-table epa-hook
jka-cmpr-hook help simple abbrev minibuffer cl-preloaded nadvice
loaddefs button faces cus-face macroexp files text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote dbusbind inotify dynamic-setting
system-font-setting font-render-setting move-toolbar gtk x-toolkit x
multi-tty make-network-process emacs)

Memory information:
((conses 16 110659 3314)
 (symbols 48 22270 0)
 (miscs 40 57 143)
 (strings 32 21604 4932)
 (string-bytes 1 732025)
 (vectors 16 14743)
 (vector-slots 8 464634 4487)
 (floats 8 171 60)
 (intervals 56 497 7)
 (buffers 976 21))

bug#24377: 25.1; CC-Mode (C++) uses font-lock-type-face for ordinary variable

[Alan Mackenzie]

Hello, Bastian.

In article <mailman.1977.1473362155.22741.bug-gnu-emacs@gnu.org> you wrote:

> This piece of C++ code is wrongly fontified by CC-Mode:

> int b = 5;
> int a = (b*3);

> Reproduce as follows:

> 1) emacs -Q
> 2) Open file "test.C"
> 3) Paste snippet in buffer
> 4) Place cursor over "b" in second line
> 5) M-x describe-face -> font-lock-type-face

> This appears to be correlated with the presence of the opening
> parentheses and the '*' character, because:

> 1) int a = b*3;
> does not have this problem

> 2) "b" gets fontified incorrectly after entering the '*' character:
> int a = (b*

Thanks for this bug report, and thanks even more for taking the trouble
to reduce the test case to just two lines.

> In GNU Emacs 25.1.2 (x86_64-unknown-linux-gnu, GTK+ Version 2.24.30)
>  of 2016-08-30 built on beischer-w520
> Repository revision: 17197d0ae9eac100b56078b2a48964b29710b76e
> Windowing system distributor 'The X.Org Foundation', version 11.0.11804000
> Configured using:
>  'configure --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib
>  --localstatedir=/var --mandir=/usr/share/man --with-gameuser=:games
>  --with-sound=alsa --with-xft --with-modules --with-x-toolkit=gtk2
>  --with-gconf --without-gsettings 'CFLAGS=-march=native -O2 -pipe
>  -fstack-protector-strong' CPPFLAGS=-D_FORTIFY_SOURCE=2
>  LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro'

[ .... ]

Yes, the CC Mode fontification code was confusing the "*" with an
indirection operator, such as you might get in "int (*b) (....);"

The solution seems to be to recognise the case where an arithmetic
operator (including "=") precedes the open parenthesis.  This is what the
following patch does.  Would you please try out this patch on real code,
and let me know whether the problem is completely fixed, and if not,
what's still going wrong.  Thanks!

After applying the patch, plese recompile all of CC Mode, since the
change is partly in Lisp macros, which need to propagate to other files.

Here's the patch:



diff -r 9ff65a2d07bd cc-fonts.el
--- a/cc-fonts.el	Sun Sep 11 20:46:35 2016 +0000
+++ b/cc-fonts.el	Tue Sep 13 18:22:17 2016 +0000
@@ -1310,6 +1310,13 @@
 		     ;; multiline declaration.
 		     (c-put-char-property (1- match-pos)
 					  'c-type 'c-decl-arg-start))
+		    ;; Got an open paren preceded by an arith operator.
+		    ((and (eq (char-before match-pos) ?\()
+			  (save-excursion
+			    (and (zerop (c-backward-token-2 2))
+				 (looking-at c-arithmetic-op-regexp))))
+		     (setq context nil
+			   c-restricted-<>-arglists nil))
 		    (t (setq context 'arglist
 			     c-restricted-<>-arglists t))))
 
diff -r 9ff65a2d07bd cc-langs.el
--- a/cc-langs.el	Sun Sep 11 20:46:35 2016 +0000
+++ b/cc-langs.el	Tue Sep 13 18:22:17 2016 +0000
@@ -1231,6 +1231,22 @@
 (c-lang-defvar c-assignment-op-regexp
   (c-lang-const c-assignment-op-regexp))
 
+(c-lang-defconst c-arithmetic-operators
+  "List of all arithmetic operators, including \"+=\", etc."
+  ;; Note: in the following, there are too many operators for AWK and IDL.
+  t (append (c-lang-const c-assignment-operators)
+	    '("+" "-" "*" "/" "%"
+	      "<<" ">>"
+	      "<" ">" "<=" ">="
+	      "==" "!="
+	      "&" "^" "|"
+	      "&&" "||")))
+
+(c-lang-defconst c-arithmetic-op-regexp
+  t (c-make-keywords-re nil
+      (c-lang-const c-arithmetic-operators)))
+(c-lang-defvar c-arithmetic-op-regexp (c-lang-const c-arithmetic-op-regexp))
+
 (c-lang-defconst c-:$-multichar-token-regexp
   ;; Regexp matching all tokens ending in ":" which are longer than one char.
   ;; Currently (2016-01-07) only used in C++ Mode.



-- 
Alan Mackenzie (Nuremberg, Germany).

bug#24377: 25.1; CC-Mode (C++) uses font-lock-type-face for ordinary variable

[Bastian Beischer]

Hello Alan,

First of all, thanks for the patch - it's appreciated. My comments
below:

Alan Mackenzie <acm@muc.de> writes:

> Hello, Bastian.
>
> In article <mailman.1977.1473362155.22741.bug-gnu-emacs@gnu.org> you wrote:
>
>> This piece of C++ code is wrongly fontified by CC-Mode:
>
>> int b = 5;
>> int a = (b*3);
>
>> Reproduce as follows:
>
>> 1) emacs -Q
>> 2) Open file "test.C"
>> 3) Paste snippet in buffer
>> 4) Place cursor over "b" in second line
>> 5) M-x describe-face -> font-lock-type-face
>
>> This appears to be correlated with the presence of the opening
>> parentheses and the '*' character, because:
>
>> 1) int a = b*3;
>> does not have this problem
>
>> 2) "b" gets fontified incorrectly after entering the '*' character:
>> int a = (b*
>
> Thanks for this bug report, and thanks even more for taking the trouble
> to reduce the test case to just two lines.
>
>> In GNU Emacs 25.1.2 (x86_64-unknown-linux-gnu, GTK+ Version 2.24.30)
>>  of 2016-08-30 built on beischer-w520
>> Repository revision: 17197d0ae9eac100b56078b2a48964b29710b76e
>> Windowing system distributor 'The X.Org Foundation', version 11.0.11804000
>> Configured using:
>>  'configure --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib
>>  --localstatedir=/var --mandir=/usr/share/man --with-gameuser=:games
>>  --with-sound=alsa --with-xft --with-modules --with-x-toolkit=gtk2
>>  --with-gconf --without-gsettings 'CFLAGS=-march=native -O2 -pipe
>>  -fstack-protector-strong' CPPFLAGS=-D_FORTIFY_SOURCE=2
>>  LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro'
>
> [ .... ]
>
> Yes, the CC Mode fontification code was confusing the "*" with an
> indirection operator, such as you might get in "int (*b) (....);"
>
> The solution seems to be to recognise the case where an arithmetic
> operator (including "=") precedes the open parenthesis.  This is what the
> following patch does.  Would you please try out this patch on real code,
> and let me know whether the problem is completely fixed, and if not,
> what's still going wrong.  Thanks!
>

Yes this helps in the example above, but I'm not sure this is a fully
correct solution. One can write something like this:

void f(int arg)  {
     printf("%d\n", arg)
}

f(a*5);

Another example:

int d(a*5)

In both cases a is also wrongly fontified (and still is after applying
the patch).

> After applying the patch, plese recompile all of CC Mode, since the
> change is partly in Lisp macros, which need to propagate to other files.
>
> Here's the patch:
>

Thanks a lot again!

On Tue, Sep 13, 2016 at 8:35 PM, Alan Mackenzie <acm@muc.de> wrote:
> Hello, Bastian.
>
> In article <mailman.1977.1473362155.22741.bug-gnu-emacs@gnu.org> you wrote:
>
>> This piece of C++ code is wrongly fontified by CC-Mode:
>
>> int b = 5;
>> int a = (b*3);
>
>> Reproduce as follows:
>
>> 1) emacs -Q
>> 2) Open file "test.C"
>> 3) Paste snippet in buffer
>> 4) Place cursor over "b" in second line
>> 5) M-x describe-face -> font-lock-type-face
>
>> This appears to be correlated with the presence of the opening
>> parentheses and the '*' character, because:
>
>> 1) int a = b*3;
>> does not have this problem
>
>> 2) "b" gets fontified incorrectly after entering the '*' character:
>> int a = (b*
>
> Thanks for this bug report, and thanks even more for taking the trouble
> to reduce the test case to just two lines.
>
>> In GNU Emacs 25.1.2 (x86_64-unknown-linux-gnu, GTK+ Version 2.24.30)
>>  of 2016-08-30 built on beischer-w520
>> Repository revision: 17197d0ae9eac100b56078b2a48964b29710b76e
>> Windowing system distributor 'The X.Org Foundation', version 11.0.11804000
>> Configured using:
>>  'configure --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib
>>  --localstatedir=/var --mandir=/usr/share/man --with-gameuser=:games
>>  --with-sound=alsa --with-xft --with-modules --with-x-toolkit=gtk2
>>  --with-gconf --without-gsettings 'CFLAGS=-march=native -O2 -pipe
>>  -fstack-protector-strong' CPPFLAGS=-D_FORTIFY_SOURCE=2
>>  LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro'
>
> [ .... ]
>
> Yes, the CC Mode fontification code was confusing the "*" with an
> indirection operator, such as you might get in "int (*b) (....);"
>
> The solution seems to be to recognise the case where an arithmetic
> operator (including "=") precedes the open parenthesis.  This is what the
> following patch does.  Would you please try out this patch on real code,
> and let me know whether the problem is completely fixed, and if not,
> what's still going wrong.  Thanks!
>
> After applying the patch, plese recompile all of CC Mode, since the
> change is partly in Lisp macros, which need to propagate to other files.
>
> Here's the patch:
>
>
>
> diff -r 9ff65a2d07bd cc-fonts.el
> --- a/cc-fonts.el       Sun Sep 11 20:46:35 2016 +0000
> +++ b/cc-fonts.el       Tue Sep 13 18:22:17 2016 +0000
> @@ -1310,6 +1310,13 @@
>                      ;; multiline declaration.
>                      (c-put-char-property (1- match-pos)
>                                           'c-type 'c-decl-arg-start))
> +                   ;; Got an open paren preceded by an arith operator.
> +                   ((and (eq (char-before match-pos) ?\()
> +                         (save-excursion
> +                           (and (zerop (c-backward-token-2 2))
> +                                (looking-at c-arithmetic-op-regexp))))
> +                    (setq context nil
> +                          c-restricted-<>-arglists nil))
>                     (t (setq context 'arglist
>                              c-restricted-<>-arglists t))))
>
> diff -r 9ff65a2d07bd cc-langs.el
> --- a/cc-langs.el       Sun Sep 11 20:46:35 2016 +0000
> +++ b/cc-langs.el       Tue Sep 13 18:22:17 2016 +0000
> @@ -1231,6 +1231,22 @@
>  (c-lang-defvar c-assignment-op-regexp
>    (c-lang-const c-assignment-op-regexp))
>
> +(c-lang-defconst c-arithmetic-operators
> +  "List of all arithmetic operators, including \"+=\", etc."
> +  ;; Note: in the following, there are too many operators for AWK and IDL.
> +  t (append (c-lang-const c-assignment-operators)
> +           '("+" "-" "*" "/" "%"
> +             "<<" ">>"
> +             "<" ">" "<=" ">="
> +             "==" "!="
> +             "&" "^" "|"
> +             "&&" "||")))
> +
> +(c-lang-defconst c-arithmetic-op-regexp
> +  t (c-make-keywords-re nil
> +      (c-lang-const c-arithmetic-operators)))
> +(c-lang-defvar c-arithmetic-op-regexp (c-lang-const c-arithmetic-op-regexp))
> +
>  (c-lang-defconst c-:$-multichar-token-regexp
>    ;; Regexp matching all tokens ending in ":" which are longer than one char.
>    ;; Currently (2016-01-07) only used in C++ Mode.
>
>
>
> --
> Alan Mackenzie (Nuremberg, Germany).
>



-- 
Bastian Beischer
RWTH Aachen University of Technology

@CERN
Office: Bdg 32-4-B12
Phone: +41-22-76-75750
E-mail: bastian.beischer@cern.ch
Address: CERN, CH-1211 Geneve 23

@RWTH Aachen
Office: 28 C 203
Phone: +49-241-80-27205
E-mail: beischer@physik.rwth-aachen.de
Address: I. Physikalisches Institut B, Sommerfeldstr. 14, D-52074 Aachen

bug#24377: 25.1; CC-Mode (C++) uses font-lock-type-face for ordinary variable

[Alan Mackenzie]

Hello, Bastian.

On Tue, Sep 13, 2016 at 10:24:40PM +0200, Bastian Beischer wrote:
> Hello Alan,

> First of all, thanks for the patch - it's appreciated. My comments
> below:

> Alan Mackenzie <acm@muc.de> writes:

> > In article <mailman.1977.1473362155.22741.bug-gnu-emacs@gnu.org> you wrote:

> >> This piece of C++ code is wrongly fontified by CC-Mode:

> >> int b = 5;
> >> int a = (b*3);

> >> Reproduce as follows:

> >> 1) emacs -Q
> >> 2) Open file "test.C"
> >> 3) Paste snippet in buffer
> >> 4) Place cursor over "b" in second line
> >> 5) M-x describe-face -> font-lock-type-face

> >> This appears to be correlated with the presence of the opening
> >> parentheses and the '*' character, because:

> >> 1) int a = b*3;
> >> does not have this problem

> >> 2) "b" gets fontified incorrectly after entering the '*' character:
> >> int a = (b*

> > Thanks for this bug report, and thanks even more for taking the trouble
> > to reduce the test case to just two lines.

> >> In GNU Emacs 25.1.2 (x86_64-unknown-linux-gnu, GTK+ Version 2.24.30)
> >>  of 2016-08-30 built on beischer-w520
> >> Repository revision: 17197d0ae9eac100b56078b2a48964b29710b76e
> >> Windowing system distributor 'The X.Org Foundation', version 11.0.11804000
> >> Configured using:
> >>  'configure --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib
> >>  --localstatedir=/var --mandir=/usr/share/man --with-gameuser=:games
> >>  --with-sound=alsa --with-xft --with-modules --with-x-toolkit=gtk2
> >>  --with-gconf --without-gsettings 'CFLAGS=-march=native -O2 -pipe
> >>  -fstack-protector-strong' CPPFLAGS=-D_FORTIFY_SOURCE=2
> >>  LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro'

> > [ .... ]

> > Yes, the CC Mode fontification code was confusing the "*" with an
> > indirection operator, such as you might get in "int (*b) (....);"

> > The solution seems to be to recognise the case where an arithmetic
> > operator (including "=") precedes the open parenthesis.  This is what the
> > following patch does.  Would you please try out this patch on real code,
> > and let me know whether the problem is completely fixed, and if not,
> > what's still going wrong.  Thanks!


> Yes this helps in the example above, but I'm not sure this is a fully
> correct solution. One can write something like this:

> void f(int arg)  {
>      printf("%d\n", arg)
> }

> f(a*5);

> Another example:

> int d(a*5)

> In both cases a is also wrongly fontified (and still is after applying
> the patch).

Sorry, but in the end I wasn't able to fix this problem.

There is now just too much ambiguity in C++'s syntax.  The "int d(a*5)"
is a sort of special case variant of "int d(a*b)".  This could either be
a declaration of d which is initialised to a*b, or the declaration of a
function d taking one parameter b whose type is a pointer to a.

C++ Mode, being an almost purely syntactic editor has no way to resolve
these ambiguities.  It seems not really worthwhile to handle the special
case of a literal integer following the "*".  So I have left that
construct being fontified as the function declaration, which is probably
the more common case.

So, I have committed the patch as it was when I last sent it to you for
review in September, and I am closing this bug.  Sorry, again.

> Thanks a lot again!


> -- 
> Bastian Beischer
> RWTH Aachen University of Technology

-- 
Alan Mackenzie (Nuremberg, Germany).