CC-mode highlight change between 24.5 and 25

CC-mode highlight change between 24.5 and 25

[Oleh Krehel]

Hi all,

I just noticed a difference in the highlight of this minimal file:

    #include <fstream>
    
    int test (const char *file_name) {
      std::ofstream fout (file_name);
    }

In version 24.5 the variable `fout' is properly highlighted with
`font-lock-variable-name-face'.  But in 25 it's highlighted with
`font-lock-function-name-face'.

Sorry if this was reported before. I just hope it can be fixed.

regards,
Oleh

Re: CC-mode highlight change between 24.5 and 25

[Alan Mackenzie]

Hello, Oleh.

On Fri, Sep 02, 2016 at 01:58:15PM +0200, Oleh Krehel wrote:
> Hi all,

> I just noticed a difference in the highlight of this minimal file:

> 1    #include <fstream>
> 2    
> 3    int test (const char *file_name) {
> 4      std::ofstream fout (file_name);
> 5    }

> In version 24.5 the variable `fout' is properly highlighted with
> `font-lock-variable-name-face'.  But in 25 it's highlighted with
> `font-lock-function-name-face'.

Yes, some change (I can't remember exactly when or what) was made to
make a function declaration fontify properly.

The trouble is that L4 looks _exactly_ like a function declaration, the
function `fout' returning something of type `std::ofstream', and taking
a single parameter of type `file_name'.

> Sorry if this was reported before. I just hope it can be fixed.

I hope so, too.  ;-)

Can you help me out here, please - by what criterion is the C++ compiler
deciding that `fout' is a variable initialised to `file_name' rather
than a function declaration?  The only syntactic thing I can see for
this is that `fout' is declared inside a function rather than at the top
level (or directly inside a class, struct, or namespace).

At the moment, CC Mode's fontification functions don't keep track of
whether the "current position" is inside a function or not, and it would
be a fair amount of work to make it track this.

If the compiler makes the decision based on semantic things (i.e. that
`file_name' is already known to be a variable rather than a type),
things would be very difficult indeed for CC Mode, which isn't a
compiler.

> regards,
> Oleh

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: CC-mode highlight change between 24.5 and 25

[Yuri Khan]

On Fri, Sep 2, 2016 at 8:27 PM, Alan Mackenzie <acm@muc.de> wrote:

>> 4      std::ofstream fout (file_name);

> The trouble is that L4 looks _exactly_ like a function declaration, the
> function `fout' returning something of type `std::ofstream', and taking
> a single parameter of type `file_name'.

> Can you help me out here, please - by what criterion is the C++ compiler
> deciding that `fout' is a variable initialised to `file_name' rather
> than a function declaration?

There is no such criterion in the syntax alone.

The more famous version of this ambiguity is called “C++’s most vexing
parse”. Lifting an example from Scott Meyers, Effective STL, chapter 1
Item 6:

    list<int> data(istream_iterator<int>(dataFile),  // warning! this doesn't do
                   istream_iterator<int>());         // what you think it does

This looks like initialization of a variable “data” of type
“list<int>” with a pair of iterators which are constructed on the
spot, but it can alternatively be parsed as a declaration of a
function named “data” that returns a list<int> and accepts two
arguments, the first of which is named “dataFile” and has type
“istream_iterator<int>”, and the second of which is unnamed and has
type “function of no arguments returning istream_iterator<int>”.

The C++ syntax requires the compiler to resolve the ambiguity in favor
of a declaration.

The canonical workaround is to add parentheses around arguments so
that they no longer fit the syntax of a function argument:

    list<int> data((istream_iterator<int>(dataFile)),
                   istream_iterator<int>());

In the original “fout” example, the ambiguity does not arise because
the compiler knows “file_name” to be a value, not a type, and because
there is no implicit int in C++.

Re: CC-mode highlight change between 24.5 and 25

[Oleh Krehel]

Alan Mackenzie <acm@muc.de> writes:

> Can you help me out here, please - by what criterion is the C++ compiler
> deciding that `fout' is a variable initialised to `file_name' rather
> than a function declaration?  The only syntactic thing I can see for
> this is that `fout' is declared inside a function rather than at the top
> level (or directly inside a class, struct, or namespace).

I think it very uncommon to declare a function inside another function
in C++: it's done in top-level instead.

I would be happy enough with the heuristic of parsing all indented
(i.e. not top-level) declarations/definitions as (variable) definitions.

    Oleh

Re: CC-mode highlight change between 24.5 and 25

[Alan Mackenzie]

Hello, Oleh.

On Fri, Sep 02, 2016 at 04:18:28PM +0200, Oleh Krehel wrote:
> Alan Mackenzie <acm@muc.de> writes:

> > Can you help me out here, please - by what criterion is the C++ compiler
> > deciding that `fout' is a variable initialised to `file_name' rather
> > than a function declaration?  The only syntactic thing I can see for
> > this is that `fout' is declared inside a function rather than at the top
> > level (or directly inside a class, struct, or namespace).

> I think it very uncommon to declare a function inside another function
> in C++: it's done in top-level instead.

Is it even valid C++?

> I would be happy enough with the heuristic of parsing all indented
> (i.e. not top-level) declarations/definitions as (variable) definitions.

Hmm.  Functions are also declared within class definitions, and are
frequently indented there.  Likely also within namepaces, at least
sometimes.

At the moment, I'm looking at somehow tracking whether or not a possible
declaration is at "top-level" or not (where "top-level" includes being
directly inside a class or namespace), by counting braces and noting the
pertinent keywords ("class", "struct", "namespace", and maybe one or two
others).  There is already a function c-at-toplevel-p which determines
this, but that would be ruinously slow in the sort of tight loop we're
dealing with here.

>     Oleh

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: CC-mode highlight change between 24.5 and 25

[Alan Mackenzie]

Hello, Yuri.

On Fri, Sep 02, 2016 at 08:10:33PM +0600, Yuri Khan wrote:
> On Fri, Sep 2, 2016 at 8:27 PM, Alan Mackenzie <acm@muc.de> wrote:

> >> 4      std::ofstream fout (file_name);

> > The trouble is that L4 looks _exactly_ like a function declaration, the
> > function `fout' returning something of type `std::ofstream', and taking
> > a single parameter of type `file_name'.

> > Can you help me out here, please - by what criterion is the C++ compiler
> > deciding that `fout' is a variable initialised to `file_name' rather
> > than a function declaration?

> There is no such criterion in the syntax alone.

I feared as much.  I've often wondered why the C++ language developers
have had such a low regard for unambiguous syntax, such as most other
(computer) languages have.  I've cursed them for it on more than one
occasion.

> The more famous version of this ambiguity is called “C++’s most vexing
> parse”. Lifting an example from Scott Meyers, Effective STL, chapter 1
> Item 6:

>     list<int> data(istream_iterator<int>(dataFile),  // warning! this doesn't do
>                    istream_iterator<int>());         // what you think it does

> This looks like initialization of a variable “data” of type
> “list<int>” with a pair of iterators which are constructed on the
> spot, but it can alternatively be parsed as a declaration of a
> function named “data” that returns a list<int> and accepts two
> arguments, the first of which is named “dataFile” and has type
> “istream_iterator<int>”, and the second of which is unnamed and has
> type “function of no arguments returning istream_iterator<int>”.

> The C++ syntax requires the compiler to resolve the ambiguity in favor
> of a declaration.

> The canonical workaround is to add parentheses around arguments so
> that they no longer fit the syntax of a function argument:

>     list<int> data((istream_iterator<int>(dataFile)),
>                    istream_iterator<int>());

> In the original “fout” example, the ambiguity does not arise because
> the compiler knows “file_name” to be a value, not a type, and because
> there is no implicit int in C++.

In other words, only semantics determine that the line 4 is a variable
declaration rather than a function declaration.  CC Mode doesn't do
semantics, except to a _very_ limited degree.  So we're reduced, yet
again, to difficult and unreliable heuristics.  Curse the C++ language
developers!

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: CC-mode highlight change between 24.5 and 25

[Davis Herring]

>> I think it very uncommon to declare a function inside another function
>> in C++: it's done in top-level instead.
>
> Is it even valid C++?

Yes.  It is a GCC extension to _define_ a function inside another 
(unless the inner function is nested inside a class defined in the outer 
function), but it is strictly conforming to declare one in C or in C++. 
(The function so-declared is indistinguishable from one declared in the 
enclosing file/namespace scope, except that its name is invisible 
outside the function containing the declaration.)

Davis

-- 
This product is sold by volume, not by mass.  If it appears too dense or 
too sparse, it is because mass-energy conversion has occurred during 
shipping.

Re: CC-mode highlight change between 24.5 and 25

[Davis Herring]

> I feared as much.  I've often wondered why the C++ language developers
> have had such a low regard for unambiguous syntax, such as most other
> (computer) languages have.  I've cursed them for it on more than one
> occasion.

Blame Ritchie (if we may speak ill of the dead), for similar vagueness 
dates to C:

/* Choose one: */
/* typedef int foo; */
/* void foo(int); */

int arg;

void f() {foo(arg);}

Depending on which of the declarations you include, "foo(arg)" is a 
declaration of a local variable "arg" or is a call to the function foo. 
(Obviously the declarations would typically be hidden in a header.)

Part of the reason for the introduction of uniform initialization in 
C++11 is to avoid the most-vexing-parse (though it does nothing for the 
above).

> In other words, only semantics determine that the line 4 is a variable
> declaration rather than a function declaration.  CC Mode doesn't do
> semantics, except to a _very_ limited degree.  So we're reduced, yet
> again, to difficult and unreliable heuristics.  Curse the C++ language
> developers!

At least here the heuristic of "out of a function, it's a function; 
inside a function, it's a variable" works pretty well and both ways 
(since most people avoid global variables with constructors).

Davis

PS - If you like, I'll be sure to mention your curse in Issaquah 
(https://isocpp.org/std/meetings-and-participation/upcoming-meetings). 
(Just be sure to tell me on how many occasions you have cursed WG21; 
curses queue like POSIX real-time signals.)

-- 
This product is sold by volume, not by mass.  If it appears too dense or 
too sparse, it is because mass-energy conversion has occurred during 
shipping.

Re: CC-mode highlight change between 24.5 and 25

[Eli Zaretskii]

> From: Davis Herring <herring@lanl.gov>
> Date: Fri, 2 Sep 2016 11:39:54 -0600
> Cc: Oleh Krehel <oleh@oremacs.com>, Emacs developers <emacs-devel@gnu.org>,
> 	Yuri Khan <yuri.v.khan@gmail.com>
> 
> Blame Ritchie (if we may speak ill of the dead), for similar vagueness 
> dates to C:
> 
> /* Choose one: */
> /* typedef int foo; */
> /* void foo(int); */
> 
> int arg;
> 
> void f() {foo(arg);}
> 
> Depending on which of the declarations you include, "foo(arg)" is a 
> declaration of a local variable "arg" or is a call to the function foo. 

Except that nobody writes declarations as in

  int (c);

Re: CC-mode highlight change between 24.5 and 25

[Alan Mackenzie]

Hello, Oleh.

On Fri, Sep 02, 2016 at 04:18:28PM +0200, Oleh Krehel wrote:
> Alan Mackenzie <acm@muc.de> writes:

> > Can you help me out here, please - by what criterion is the C++ compiler
> > deciding that `fout' is a variable initialised to `file_name' rather
> > than a function declaration?  The only syntactic thing I can see for
> > this is that `fout' is declared inside a function rather than at the top
> > level (or directly inside a class, struct, or namespace).

> I think it very uncommon to declare a function inside another function
> in C++: it's done in top-level instead.

> I would be happy enough with the heuristic of parsing all indented
> (i.e. not top-level) declarations/definitions as (variable) definitions.

OK.  Here's a first approximation to a solution, which I would be
grateful if you could try out on real code.  Please let me know how well
it works, and if it introduces any nasty looking bugs.

What I've done is to count nesting depth of braces inside a class or
namespace, etc.  When that depth is 1, we're at the top level, and
anything looking like a function is fontified as one.  When the depth is
more than 1, we're not at top level, and anything looking like a
function is fontified as a uniform initialisation.

The following patch should apply OK to the savannah master branch:



diff --git a/lisp/progmodes/cc-engine.el b/lisp/progmodes/cc-engine.el
index 2cad2d0..4622398 100644
--- a/lisp/progmodes/cc-engine.el
+++ b/lisp/progmodes/cc-engine.el
@@ -422,6 +422,15 @@ c-forward-to-cpp-define-body
 \f
 ;;; Basic utility functions.
 
+(defmacro c-pull-open-brace (ps)
+  ;; Pull the next open brace from PS (which has the form of paren-state),
+  ;; skipping over any brace pairs.  Returns NIL when PS is exhausted.
+  `(progn
+     (while (consp (car ,ps))
+       (setq ,ps (cdr ,ps)))
+     (prog1 (car ,ps)
+       (setq ,ps (cdr ,ps)))))
+
 (defun c-delq-from-dotted-list (elt dlist)
   ;; If ELT is a member of the (possibly dotted) list DLIST, remove all
   ;; occurrences of it (except for any in the last cdr of DLIST).
@@ -5109,6 +5118,123 @@ c-debug-remove-decl-spot-faces
        (c-debug-remove-face ,beg ,end 'c-debug-decl-spot-face)
        (c-debug-remove-face ,beg ,end 'c-debug-decl-sws-face))))
 
+;;;; NEW STOUGH, 2016-09-02, and ...-03
+(defconst c-bs-interval 5000)
+(defvar c-bs-cache nil)
+(make-variable-buffer-local 'c-bs-cache)
+(defvar c-bs-cache-limit 1)
+(make-variable-buffer-local 'c-bs-cache-limit)
+(defvar c-bs-prev-pos most-positive-fixnum)
+(make-variable-buffer-local 'c-bs-prev-pos)
+(defvar c-bs-prev-stack nil)
+(make-variable-buffer-local 'c-bs-prev-stack)
+
+(defun c-init-bs-cache ()
+  ;; Initialize the cache in `c-bs-cache' and related variables.
+  (setq c-bs-cache nil
+	c-bs-cache-limit 1
+	c-bs-prev-pos most-positive-fixnum
+	c-bs-prev-stack nil))
+
+(defun c-truncate-bs-cache (pos &rest _ignore)
+  ;; Truncate the upper bound of the cache `c-bs-cache' to POS, if it is
+  ;; higher than that position.  This is called as either a before- or
+  ;; after-change-function.
+  (setq c-bs-cache-limit
+	(min c-bs-cache-limit pos)))
+
+(defun c-update-brace-stack (stack from to)
+  ;; Give a brace-stack which has the value STACK at position FROM, update it
+  ;; to it's value at position TO, where TO is after (or equal to) FROM.
+  ;; Return this new value.
+  (let (match kwd-sym (prev-match-pos 1)
+	      )
+    (save-excursion
+      (goto-char from)
+      (while (and (< (point) to)
+		  (c-syntactic-re-search-forward c-brace-stack-thing-key to 'bound)
+		  (> (point) prev-match-pos))
+	(setq prev-match-pos (point))
+	(setq match (match-string-no-properties 1)
+	      kwd-sym (c-keyword-sym match))
+	(cond
+	 ((and (equal match "{")
+	       (progn (backward-char)
+		      (prog1 (looking-at "\\s(")
+			(forward-char)))		      
+	  (setq stack (if stack
+			  (cons (1+ (car stack)) (cdr stack))
+			(list 1)))))
+	 ((and (equal match "}")
+	       (progn (backward-char)
+		      (prog1 (looking-at "\\s)")
+			(forward-char))))		
+	  (setq stack
+		(cond
+		 ((and stack (> (car stack) 1))
+		  (cons (1- (car stack)) (cdr stack)))
+		 ((and (cdr stack) (eq (car stack) 1))
+		  (cdr stack))
+		 (t stack))))
+	 ((equal match ";")
+	  (when (and stack (cdr stack) (eq (car stack) 0))
+	    (setq stack (cdr stack))))
+	 ((c-keyword-member kwd-sym 'c-flat-decl-block-kwds)
+	  (push 0 stack))))
+      stack)))
+
+(defun c-brace-stack-at (here)
+  ;; Given a buffer position HERE, Return the value of the brace stack there.
+  (save-excursion
+    (save-restriction
+      (widen)
+      (let ((c c-bs-cache)
+	    (can-use-prev (<= c-bs-prev-pos c-bs-cache-limit))
+	    elt stack pos npos high-elt)
+	;; Trim the cache to take account of buffer changes.
+	(while (and c
+		    (> (caar c) c-bs-cache-limit))
+	  (setq c (cdr c)))
+	(setq c-bs-cache c)
+
+	(while (and c
+		    (> (caar c) here))
+	  (setq high-elt (car c))
+	  (setq c (cdr c)))
+	(setq pos (or (and c (caar c))
+		      (point-min)))
+
+	(setq elt (if c
+		      (car c)
+		    (cons (point-min) (list 1))))
+	(when (not high-elt)
+	  (setq stack (cdr elt))
+	  (while
+	      ;; Add an element to `c-state-semi-nonlit-pos-cache' each iteration.
+	      (<= (setq npos (+ pos c-bs-interval)) here)
+	    (setq stack (c-update-brace-stack stack pos npos))
+	    (setq elt (cons npos stack))
+	    (setq c-bs-cache (cons elt c-bs-cache))
+	    (setq pos npos)))
+
+	(if (> pos c-bs-cache-limit)
+	    (setq c-bs-cache-limit pos))
+
+	;; Can we just use the previous value?
+	(if (and can-use-prev
+		 (<= c-bs-prev-pos here)
+		 (> c-bs-prev-pos (car elt)))
+	    (setq pos c-bs-prev-pos
+		  stack c-bs-prev-stack)
+	  (setq pos (car elt)
+		stack (cdr elt)))
+	(if (> here c-bs-cache-limit)
+	    (setq c-bs-cache-limit here))
+	(setq c-bs-prev-pos here
+	      c-bs-prev-stack (c-update-brace-stack stack pos here))))))
+
+;;;; END OF NEW STOUGH, 2016-09-02, and ...-03, and ...-04
+
 (defmacro c-find-decl-prefix-search ()
   ;; Macro used inside `c-find-decl-spots'.  It ought to be a defun,
   ;; but it contains lots of free variables that refer to things
@@ -5202,6 +5328,9 @@ c-find-decl-prefix-search
 	       cfd-re-match nil)
        (setq cfd-match-pos cfd-prop-match
 	     cfd-prop-match nil))
+     (let ((stack (c-brace-stack-at cfd-match-pos)))
+       (setq cfd-top-level (or (null stack)
+			       (<= (car stack) 1))))
 
      (goto-char cfd-match-pos)
 
@@ -5300,8 +5429,12 @@ c-find-decl-spots
 	;; comments.
 	(cfd-token-pos 0)
 	;; The end position of the last entered macro.
-	(cfd-macro-end 0))
-
+	(cfd-macro-end 0)
+	;; Whether the last position returned from `c-find-decl-prefix-search'
+	;; is at the top-level (including directly in a class or namespace,
+	;; etc.).
+	cfd-top-level)
+    
     ;; Initialize by finding a syntactically relevant start position
     ;; before the point, and do the first `c-decl-prefix-or-start-re'
     ;; search unless we're at bob.
@@ -5608,7 +5741,7 @@ c-find-decl-spots
 		   nil))))		; end of when condition
 
 	(c-debug-put-decl-spot-faces cfd-match-pos (point))
-	(if (funcall cfd-fun cfd-match-pos (/= cfd-macro-end 0))
+	(if (funcall cfd-fun cfd-match-pos (/= cfd-macro-end 0) cfd-top-level)
 	    (setq cfd-prop-match nil))
 
 	(when (/= cfd-macro-end 0)
@@ -7349,15 +7482,6 @@ c-forward-annotation
 	    t))
      (progn (goto-char pos) nil))))
 
-(defmacro c-pull-open-brace (ps)
-  ;; Pull the next open brace from PS (which has the form of paren-state),
-  ;; skipping over any brace pairs.  Returns NIL when PS is exhausted.
-  `(progn
-     (while (consp (car ,ps))
-       (setq ,ps (cdr ,ps)))
-     (prog1 (car ,ps)
-       (setq ,ps (cdr ,ps)))))
-
 (defun c-back-over-compound-identifier ()
   ;; Point is putatively just after a "compound identifier", i.e. something
   ;; looking (in C++) like this "FQN::of::base::Class".  Move to the start of
@@ -7555,7 +7679,7 @@ c-forward-declarator
   ;; array/struct initialization) or "=" or terminating delimiter
   ;; (e.g. "," or ";" or "}").
   (let ((here (point))
-	id-start id-end brackets-after-id paren-depth)
+	id-start id-end brackets-after-id paren-depth paren-pos)
     (or limit (setq limit (point-max)))
     (if	(and
 	 (< (point) limit)
@@ -7597,7 +7721,8 @@ c-forward-declarator
 			   t))
 		    (if (eq (char-after) ?\()
 			(progn
-			  (setq paren-depth (1+ paren-depth))
+			  (setq paren-depth (1+ paren-depth)
+				paren-pos (or paren-pos (point)))
 			  (forward-char))
 		      (goto-char (match-end 1)))
 		    (c-forward-syntactic-ws)
@@ -7725,6 +7850,9 @@ c-forward-decl-or-cast-1
   ;;           inside a function declaration arglist).
   ;; '<>       In an angle bracket arglist.
   ;; 'arglist  Some other type of arglist.
+  ;; 'top      Some other context and point is at the top-level (either
+  ;;           outside any braces or directly inside a class or namespace,
+  ;;           etc.)
   ;; nil       Some other context or unknown context.  Includes
   ;;           within the parens of an if, for, ... construct.
   ;; 'not-decl This value is never supplied to this function.  It
@@ -8133,7 +8261,7 @@ c-forward-decl-or-cast-1
 			      maybe-typeless
 			      backup-maybe-typeless
 			      (when c-recognize-typeless-decls
-				(and (not context)
+				(and (memq context '(nil top))
 				     ;; Deal with C++11's "copy-initialization"
 				     ;; where we have <type>(<constant>), by
 				     ;; contrasting with a typeless
@@ -8196,7 +8324,7 @@ c-forward-decl-or-cast-1
 
 	 (setq at-decl-end
 	       (looking-at (cond ((eq context '<>) "[,>]")
-				 (context "[,)]")
+				 ((not (memq context '(nil top))) "[,\)]")
 				 (t "[,;]"))))
 
 	 ;; Now we've collected info about various characteristics of
@@ -8325,7 +8453,7 @@ c-forward-decl-or-cast-1
 
 	   (if (and got-parens
 		    (not got-prefix)
-		    (not context)
+		    (memq context '(nil top))
 		    (not (eq at-type t))
 		    (or backup-at-type
 			maybe-typeless
@@ -8375,6 +8503,18 @@ c-forward-decl-or-cast-1
 	       ;; instantiation expression).
 	       (throw 'at-decl-or-cast nil))))
 
+	 ;; CASE 9.5
+	 (when (and (not context)	; i.e. not at top level.
+		    (c-major-mode-is 'c++-mode)
+		    (eq at-decl-or-cast 'ids)
+		    after-paren-pos)
+	   ;; We've got something like "foo bar (...)" in C++ which isn't at
+	   ;; the top level.  This is probably a uniform initialization of bar
+	   ;; to the contents of the parens.  In this case the declarator ends
+	   ;; at the open paren.
+	   (goto-char (1- after-paren-pos))
+	   (throw 'at-decl-or-cast t))
+
 	 ;; CASE 10
 	 (when at-decl-or-cast
 	   ;; By now we've located the type in the declaration that we know
@@ -8383,7 +8523,7 @@ c-forward-decl-or-cast-1
 
 	 ;; CASE 11
 	 (when (and got-identifier
-		    (not context)
+		    (memq context '(nil top))
 		    (looking-at c-after-suffixed-type-decl-key)
 		    (if (and got-parens
 			     (not got-prefix)
@@ -8479,7 +8619,7 @@ c-forward-decl-or-cast-1
 	       (when (and got-prefix-before-parens
 			  at-type
 			  (or at-decl-end (looking-at "=[^=]"))
-			  (not context)
+			  (memq context '(nil top))
 			  (or (not got-suffix)
 			      at-decl-start))
 		 ;; Got something like "foo * bar;".  Since we're not inside
@@ -8505,7 +8645,7 @@ c-forward-decl-or-cast-1
 		 (throw 'at-decl-or-cast t)))
 
 	   ;; CASE 18
-	   (when (and context
+	   (when (and (not (memq context '(nil top)))
 		      (or got-prefix
 			  (and (eq context 'decl)
 			       (not c-recognize-paren-inits)
diff --git a/lisp/progmodes/cc-fonts.el b/lisp/progmodes/cc-fonts.el
index 735108f..38362b2 100644
--- a/lisp/progmodes/cc-fonts.el
+++ b/lisp/progmodes/cc-fonts.el
@@ -991,7 +991,7 @@ c-font-lock-<>-arglists
 	  (goto-char pos)))))
   nil)
 
-(defun c-font-lock-declarators (limit list types)
+(defun c-font-lock-declarators (limit list types &optional not-top)
   ;; Assuming the point is at the start of a declarator in a declaration,
   ;; fontify the identifier it declares.  (If TYPES is set, it does this via
   ;; the macro `c-fontify-types-and-refs'.)
@@ -1001,7 +1001,9 @@ c-font-lock-declarators
   ;; additionally, mark the commas with c-type property 'c-decl-id-start or
   ;; 'c-decl-type-start (according to TYPES).  Stop at LIMIT.
   ;;
-  ;; If TYPES is non-nil, fontify all identifiers as types.
+  ;; If TYPES is non-nil, fontify all identifiers as types.  If NOT-TOP is
+  ;; non-nil, we are not at the top-level ("top-level" includes being directly
+  ;; inside a class or namespace, etc.).
   ;;
   ;; Nil is always returned.  The function leaves point at the delimiter after
   ;; the last declarator it processes.
@@ -1025,6 +1027,11 @@ c-font-lock-declarators
       (setq next-pos (point)
 	    id-start (car decl-res)
 	    id-face (if (and (eq (char-after) ?\()
+			     (or (not not-top)
+				 (save-excursion
+				   (forward-char)
+				   (c-forward-syntactic-ws)
+				   (looking-at "[*&]")))
 			     (not (car (cddr decl-res))) ; brackets-after-id
 			     (or (not (c-major-mode-is 'c++-mode))
 				 (save-excursion
@@ -1199,7 +1206,7 @@ c-font-lock-declarations
        c-decl-start-re
        (eval c-maybe-decl-faces)
 
-       (lambda (match-pos inside-macro)
+       (lambda (match-pos inside-macro &optional toplev)
 	 ;; Note to maintainers: don't use `limit' inside this lambda form;
 	 ;; c-find-decl-spots sometimes narrows to less than `limit'.
 	 (setq start-pos (point))
@@ -1223,7 +1230,7 @@ c-font-lock-declarations
 	    (let ((type (and (> match-pos (point-min))
 			     (c-get-char-property (1- match-pos) 'c-type))))
 	      (cond ((not (memq (char-before match-pos) '(?\( ?, ?\[ ?< ?{)))
-		     (setq context nil
+		     (setq context (and toplev 'top)
 			   c-restricted-<>-arglists nil))
 		    ;; A control flow expression or a decltype
 		    ((and (eq (char-before match-pos) ?\()
@@ -1273,7 +1280,7 @@ c-font-lock-declarations
 					  'c-not-decl))
 		    ;; We're inside an "ordinary" open brace.
 		    ((eq (char-before match-pos) ?{)
-		     (setq context nil
+		     (setq context (and toplev 'top)
 			   c-restricted-<>-arglists nil))
 		    ;; Inside an angle bracket arglist.
 		    ((or (eq type 'c-<>-arg-sep)
@@ -1437,7 +1444,8 @@ c-font-lock-declarations
 						 'c-decl-id-start)))))
 
 		    (c-font-lock-declarators
-		     (min limit (point-max)) decl-list (cadr decl-or-cast)))
+		     (min limit (point-max)) decl-list
+		     (cadr decl-or-cast) (not toplev)))
 
 		  ;; A declaration has been successfully identified, so do all the
 		  ;; fontification of types and refs that've been recorded.
@@ -2413,7 +2421,7 @@ c-font-lock-objc-methods
      limit
      "[-+]"
      nil
-     (lambda (match-pos inside-macro)
+     (lambda (match-pos inside-macro &optional top-level)
        (forward-char)
        (c-font-lock-objc-method))))
   nil)
diff --git a/lisp/progmodes/cc-langs.el b/lisp/progmodes/cc-langs.el
index ae6e6a3..b8c4420 100644
--- a/lisp/progmodes/cc-langs.el
+++ b/lisp/progmodes/cc-langs.el
@@ -479,10 +479,12 @@ c-populate-syntax-table
 	c-before-change-check-<>-operators
 	c-depropertize-CPP
 	c-before-after-change-digit-quote
-	c-invalidate-macro-cache)
+	c-invalidate-macro-cache
+	c-truncate-bs-cache)
   (c objc) '(c-extend-region-for-CPP
 	     c-depropertize-CPP
-	     c-invalidate-macro-cache)
+	     c-invalidate-macro-cache
+	     c-truncate-bs-cache)
   ;; java 'c-before-change-check-<>-operators
   awk 'c-awk-record-region-clear-NL)
 (c-lang-defvar c-get-state-before-change-functions
@@ -2588,6 +2590,26 @@ 'c-opt-op-identitier-prefix
 (c-lang-defvar c-opt-inexpr-brace-list-key
   (c-lang-const c-opt-inexpr-brace-list-key))
 
+(c-lang-defconst c-flat-decl-block-kwds
+  ;; Keywords that can introduce another declaration level, i.e. where a
+  ;; following "{" isn't a function block or brace list.  Note that, for
+  ;; historical reasons, `c-decl-block-key' is NOT constructed from this lang
+  ;; const.
+  t (c--delete-duplicates
+     (append (c-lang-const c-class-decl-kwds)
+	     (c-lang-const c-other-block-decl-kwds)
+	     (c-lang-const c-inexpr-class-kwds))
+     :test 'string-equal))
+
+(c-lang-defconst c-brace-stack-thing-key
+  ;; Regexp matching any keyword or operator relevant to the brace stack (see
+  ;; `c-update-brace-stack' in cc-engine.el).
+  t (c-make-keywords-re 'appendable
+      (append
+       (c-lang-const c-flat-decl-block-kwds)
+       '("{" "}" ";"))))
+(c-lang-defvar c-brace-stack-thing-key (c-lang-const c-brace-stack-thing-key))
+	   
 (c-lang-defconst c-decl-block-key
   ;; Regexp matching keywords in any construct that contain another
   ;; declaration level, i.e. that isn't followed by a function block
diff --git a/lisp/progmodes/cc-mode.el b/lisp/progmodes/cc-mode.el
index f2c6256..5b0679a 100644
--- a/lisp/progmodes/cc-mode.el
+++ b/lisp/progmodes/cc-mode.el
@@ -557,6 +557,8 @@ c-basic-common-init
 
   ;; Initialize the cache of brace pairs, and opening braces/brackets/parens.
   (c-state-cache-init)
+  ;; Initialize the "brace stack" cache.
+  (c-init-bs-cache)
 
   (when (or c-recognize-<>-arglists
 	    (c-major-mode-is 'awk-mode)


>     Oleh

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: CC-mode highlight change between 24.5 and 25

[Alan Mackenzie]

Hello, Davis.

On Fri, Sep 02, 2016 at 11:39:54AM -0600, Davis Herring wrote:
> > I feared as much.  I've often wondered why the C++ language developers
> > have had such a low regard for unambiguous syntax, such as most other
> > (computer) languages have.  I've cursed them for it on more than one
> > occasion.

> Blame Ritchie (if we may speak ill of the dead), for similar vagueness 
> dates to C:

> /* Choose one: */
> /* typedef int foo; */
> /* void foo(int); */

> int arg;

> void f() {foo(arg);}

> Depending on which of the declarations you include, "foo(arg)" is a 
> declaration of a local variable "arg" or is a call to the function foo. 
> (Obviously the declarations would typically be hidden in a header.)

That's true, but it's vanishingly uncommon to declare a variable with
parentheses round it (unless there's good reason like in

    void (*foo) (void);

).

> Part of the reason for the introduction of uniform initialization in 
> C++11 is to avoid the most-vexing-parse (though it does nothing for the 
> above).

OK.

> > In other words, only semantics determine that the line 4 is a variable
> > declaration rather than a function declaration.  CC Mode doesn't do
> > semantics, except to a _very_ limited degree.  So we're reduced, yet
> > again, to difficult and unreliable heuristics.  Curse the C++ language
> > developers!

> At least here the heuristic of "out of a function, it's a function; 
> inside a function, it's a variable" works pretty well and both ways 
> (since most people avoid global variables with constructors).

Thanks!  That's what I've been coding up over the weekend.  It seems to
work reasonably well.

> Davis

> PS - If you like, I'll be sure to mention your curse in Issaquah 
> (https://isocpp.org/std/meetings-and-participation/upcoming-meetings). 
> (Just be sure to tell me on how many occasions you have cursed WG21; 
> curses queue like POSIX real-time signals.)

Maybe.  But what's done is done and is not going to get undone in future
C++ standards.  But if you could point out that (near) pure syntactic
tools like Emacs do exist, and that every ambiguity introduced into the
language causes massive amounts of work for the maintainers of those
tools (like me), that would be helpful.

For example, I don't get the impression that the C++ developers were
aware of the ambiguities caused by using "less than" an "greater than"
as template delimiters.  Unique operators, perhaps something like "[<"
and ">]" could have been chosen without detriment to the language.  "<"
and ">" have caused, probably, scores (if not hundreds) of hours of work
in CC Mode, and likely in other syntax dependent tools.

At the moment, we have rules like "semicolons and braces can not appear
within template delimiters".  PLEASE don't do anything to destroy the
utility of such rules in future C++ versions.  If something is currently
syntactically clear, PLEASE make sure it remains so.

> -- 
> This product is sold by volume, not by mass.  If it appears too dense or 
> too sparse, it is because mass-energy conversion has occurred during 
> shipping.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: CC-mode highlight change between 24.5 and 25

[Oleh Krehel]

Hi Alan,

Alan Mackenzie <acm@muc.de> writes:

> OK.  Here's a first approximation to a solution, which I would be
> grateful if you could try out on real code.  Please let me know how well
> it works, and if it introduces any nasty looking bugs.
>
> What I've done is to count nesting depth of braces inside a class or
> namespace, etc.  When that depth is 1, we're at the top level, and
> anything looking like a function is fontified as one.  When the depth is
> more than 1, we're not at top level, and anything looking like a
> function is fontified as a uniform initialisation.
>
> The following patch should apply OK to the savannah master branch:

Thanks for the patch. I'm testing it now. It works fine with the example
that I initally gave. However, this one does not work:

    template <class T> void barf (T t, const char *file_name) {
      std::ofstream fout (file_name);
      fout << t;
      fout.close ();
    }

Here, "template <class T>" is what confuses the font-lock. In C++, these
angle braces can be nested to an arbitrary depth.

Oleh

Re: CC-mode highlight change between 24.5 and 25

[Alan Mackenzie]

Hello, Oleh.

On Mon, Sep 05, 2016 at 11:27:40AM +0200, Oleh Krehel wrote:
> Hi Alan,

> Alan Mackenzie <acm@muc.de> writes:

[ .... ]

> Thanks for the patch. I'm testing it now. It works fine with the example
> that I initally gave. However, this one does not work:

>     template <class T> void barf (T t, const char *file_name) {
>       std::ofstream fout (file_name);
>       fout << t;
>       fout.close ();
>     }

> Here, "template <class T>" is what confuses the font-lock. In C++, these
> angle braces can be nested to an arbitrary depth.

Oh, don't I know it!  ;-(

I was a little over confident in my brace counting.  The code was
landing at the "class" inside the template delimiters, and moving
forward over the "{", then saying "I'm now at the top level inside a
class".  Rubbish!

I've (hopefully) corrected that in the patch that follows which, again,
is the full patch which should apply to the savannah master.  (Please
just discard the patch from yesterday.)

So, please try out the following, and let me know, again, how it copes
with real C++ code.  Thanks!



diff --git a/lisp/progmodes/cc-engine.el b/lisp/progmodes/cc-engine.el
index 2cad2d0..e72c31a 100644
--- a/lisp/progmodes/cc-engine.el
+++ b/lisp/progmodes/cc-engine.el
@@ -422,6 +422,15 @@ c-forward-to-cpp-define-body
 \f
 ;;; Basic utility functions.
 
+(defmacro c-pull-open-brace (ps)
+  ;; Pull the next open brace from PS (which has the form of paren-state),
+  ;; skipping over any brace pairs.  Returns NIL when PS is exhausted.
+  `(progn
+     (while (consp (car ,ps))
+       (setq ,ps (cdr ,ps)))
+     (prog1 (car ,ps)
+       (setq ,ps (cdr ,ps)))))
+
 (defun c-delq-from-dotted-list (elt dlist)
   ;; If ELT is a member of the (possibly dotted) list DLIST, remove all
   ;; occurrences of it (except for any in the last cdr of DLIST).
@@ -5109,6 +5118,141 @@ c-debug-remove-decl-spot-faces
        (c-debug-remove-face ,beg ,end 'c-debug-decl-spot-face)
        (c-debug-remove-face ,beg ,end 'c-debug-decl-sws-face))))
 
+;;;; NEW STOUGH, 2016-09-02, and ...-03
+(defconst c-bs-interval 5000)
+(defvar c-bs-cache nil)
+(make-variable-buffer-local 'c-bs-cache)
+(defvar c-bs-cache-limit 1)
+(make-variable-buffer-local 'c-bs-cache-limit)
+(defvar c-bs-prev-pos most-positive-fixnum)
+(make-variable-buffer-local 'c-bs-prev-pos)
+(defvar c-bs-prev-stack nil)
+(make-variable-buffer-local 'c-bs-prev-stack)
+
+(defun c-init-bs-cache ()
+  ;; Initialize the cache in `c-bs-cache' and related variables.
+  (setq c-bs-cache nil
+	c-bs-cache-limit 1
+	c-bs-prev-pos most-positive-fixnum
+	c-bs-prev-stack nil))
+
+(defun c-truncate-bs-cache (pos &rest _ignore)
+  ;; Truncate the upper bound of the cache `c-bs-cache' to POS, if it is
+  ;; higher than that position.  This is called as either a before- or
+  ;; after-change-function.
+  (setq c-bs-cache-limit
+	(min c-bs-cache-limit pos)))
+
+(defun c-update-brace-stack (stack from to)
+  ;; Give a brace-stack which has the value STACK at position FROM, update it
+  ;; to it's value at position TO, where TO is after (or equal to) FROM.
+  ;; Return this new value.
+  (let (match kwd-sym (prev-match-pos 1)
+	      (stack-1 (cdr stack))
+	      (bound-<> (car stack))
+	      )
+    (save-excursion
+      (cond
+       ((and bound-<> (<= to bound-<>))
+	(goto-char to))			; Nothing to do.
+       (bound-<>
+	(goto-char bound-<>)
+	(setq bound-<> nil))
+       (t (goto-char from)))
+      (while (and (< (point) to)
+		  (c-syntactic-re-search-forward c-brace-stack-thing-key to 'bound)
+		  (> (point) prev-match-pos))
+	(setq prev-match-pos (point))
+	(setq match (match-string-no-properties 1)
+	      kwd-sym (c-keyword-sym match))
+	(cond
+	 ((and (equal match "{")
+	       (progn (backward-char)
+		      (prog1 (looking-at "\\s(")
+			(forward-char))))
+	  (setq stack-1 (if stack-1
+			    (cons (1+ (car stack-1)) (cdr stack-1))
+			  (list 1))))
+	 ((and (equal match "}")
+	       (progn (backward-char)
+		      (prog1 (looking-at "\\s)")
+			(forward-char))))
+	  (setq stack-1
+		(cond
+		 ((and stack-1 (> (car stack-1) 1))
+		  (cons (1- (car stack-1)) (cdr stack-1)))
+		 ((and (cdr stack-1) (eq (car stack-1) 1))
+		  (cdr stack-1))
+		 (t stack-1))))
+	 ((and (equal match "<")
+	       (progn (backward-char)
+		      (prog1 (looking-at "\\s(")
+			(forward-char))))
+	  (backward-char)
+	  (if (c-forward-<>-arglist nil) ; Should always work.
+	      (when (> (point) to)
+		(setq bound-<> (point)))
+	    (forward-char)))
+	 ((equal match ";")
+	  (when (and stack-1 (cdr stack-1) (eq (car stack-1) 0))
+	    (setq stack-1 (cdr stack-1))))
+	 ((c-keyword-member kwd-sym 'c-flat-decl-block-kwds)
+	  (push 0 stack-1))))
+      (cons bound-<> stack-1))))
+
+(defun c-brace-stack-at (here)
+  ;; Given a buffer position HERE, Return the value of the brace stack there.
+  (save-excursion
+    (save-restriction
+      (widen)
+      (let ((c c-bs-cache)
+	    (can-use-prev (<= c-bs-prev-pos c-bs-cache-limit))
+	    elt stack pos npos high-elt)
+	;; Trim the cache to take account of buffer changes.
+	(while (and c
+		    (> (caar c) c-bs-cache-limit))
+	  (setq c (cdr c)))
+	(setq c-bs-cache c)
+
+	(while (and c
+		    (> (caar c) here))
+	  (setq high-elt (car c))
+	  (setq c (cdr c)))
+	(setq pos (or (and c (caar c))
+		      (point-min)))
+
+	(setq elt (if c
+		      (car c)
+		    (cons (point-min)
+			  (cons nil (list 1)))))
+	(when (not high-elt)
+	  (setq stack (cdr elt))
+	  (while
+	      ;; Add an element to `c-state-semi-nonlit-pos-cache' each iteration.
+	      (<= (setq npos (+ pos c-bs-interval)) here)
+	    (setq stack (c-update-brace-stack stack pos npos))
+	    (setq elt (cons npos stack))
+	    (setq c-bs-cache (cons elt c-bs-cache))
+	    (setq pos npos)))
+
+	(if (> pos c-bs-cache-limit)
+	    (setq c-bs-cache-limit pos))
+
+	;; Can we just use the previous value?
+	(if (and can-use-prev
+		 (<= c-bs-prev-pos here)
+		 (> c-bs-prev-pos (car elt)))
+	    (setq pos c-bs-prev-pos
+		  stack c-bs-prev-stack)
+	  (setq pos (car elt)
+		stack (cdr elt)))
+	(if (> here c-bs-cache-limit)
+	    (setq c-bs-cache-limit here))
+	(setq c-bs-prev-pos here
+	      c-bs-prev-stack (c-update-brace-stack stack pos here))))))
+
+;;;; END OF NEW STOUGH, 2016-09-02, and ...-03, and ...-04
+
 (defmacro c-find-decl-prefix-search ()
   ;; Macro used inside `c-find-decl-spots'.  It ought to be a defun,
   ;; but it contains lots of free variables that refer to things
@@ -5202,6 +5346,9 @@ c-find-decl-prefix-search
 	       cfd-re-match nil)
        (setq cfd-match-pos cfd-prop-match
 	     cfd-prop-match nil))
+     (let ((stack (c-brace-stack-at cfd-match-pos)))
+       (setq cfd-top-level (or (null stack)
+			       (<= (cadr stack) 1))))
 
      (goto-char cfd-match-pos)
 
@@ -5300,7 +5447,11 @@ c-find-decl-spots
 	;; comments.
 	(cfd-token-pos 0)
 	;; The end position of the last entered macro.
-	(cfd-macro-end 0))
+	(cfd-macro-end 0)
+	;; Whether the last position returned from `c-find-decl-prefix-search'
+	;; is at the top-level (including directly in a class or namespace,
+	;; etc.).
+	cfd-top-level)
 
     ;; Initialize by finding a syntactically relevant start position
     ;; before the point, and do the first `c-decl-prefix-or-start-re'
@@ -5608,7 +5759,7 @@ c-find-decl-spots
 		   nil))))		; end of when condition
 
 	(c-debug-put-decl-spot-faces cfd-match-pos (point))
-	(if (funcall cfd-fun cfd-match-pos (/= cfd-macro-end 0))
+	(if (funcall cfd-fun cfd-match-pos (/= cfd-macro-end 0) cfd-top-level)
 	    (setq cfd-prop-match nil))
 
 	(when (/= cfd-macro-end 0)
@@ -7349,15 +7500,6 @@ c-forward-annotation
 	    t))
      (progn (goto-char pos) nil))))
 
-(defmacro c-pull-open-brace (ps)
-  ;; Pull the next open brace from PS (which has the form of paren-state),
-  ;; skipping over any brace pairs.  Returns NIL when PS is exhausted.
-  `(progn
-     (while (consp (car ,ps))
-       (setq ,ps (cdr ,ps)))
-     (prog1 (car ,ps)
-       (setq ,ps (cdr ,ps)))))
-
 (defun c-back-over-compound-identifier ()
   ;; Point is putatively just after a "compound identifier", i.e. something
   ;; looking (in C++) like this "FQN::of::base::Class".  Move to the start of
@@ -7533,10 +7675,12 @@ c-forward-declarator
   ;; Assuming point is at the start of a declarator, move forward over it,
   ;; leaving point at the next token after it (e.g. a ) or a ; or a ,).
   ;;
-  ;; Return a list (ID-START ID-END BRACKETS-AFTER-ID GOT-INIT), where ID-START and
-  ;; ID-END are the bounds of the declarator's identifier, and
-  ;; BRACKETS-AFTER-ID is non-nil if a [...] pair is present after the id.
-  ;; GOT-INIT is non-nil when the declarator is followed by "=" or "(".
+  ;; Return a list (ID-START ID-END BRACKETS-AFTER-ID GOT-INIT DECORATED),
+  ;; where ID-START and ID-END are the bounds of the declarator's identifier,
+  ;; and BRACKETS-AFTER-ID is non-nil if a [...] pair is present after the id.
+  ;; GOT-INIT is non-nil when the declarator is followed by "=" or "(",
+  ;; DECORATED it non-nil when the identifier is embellished by an operator,
+  ;; like "*x", or "(*x)".
   ;;
   ;; If ACCEPT-ANON is non-nil, move forward over any "anonymous declarator",
   ;; i.e. something like the (*) in int (*), such as might be found in a
@@ -7555,7 +7699,7 @@ c-forward-declarator
   ;; array/struct initialization) or "=" or terminating delimiter
   ;; (e.g. "," or ";" or "}").
   (let ((here (point))
-	id-start id-end brackets-after-id paren-depth)
+	id-start id-end brackets-after-id paren-depth decorated)
     (or limit (setq limit (point-max)))
     (if	(and
 	 (< (point) limit)
@@ -7595,6 +7739,8 @@ c-forward-declarator
 				 (setq got-identifier t)
 				 nil))
 			   t))
+		    (if (looking-at c-type-decl-operator-prefix-key)
+			(setq decorated t))
 		    (if (eq (char-after) ?\()
 			(progn
 			  (setq paren-depth (1+ paren-depth))
@@ -7649,7 +7795,7 @@ c-forward-declarator
 	     (setq brackets-after-id t))
 	   (backward-char)
 	   found))
-	(list id-start id-end brackets-after-id (match-beginning 1))
+	(list id-start id-end brackets-after-id (match-beginning 1) decorated)
 
       (goto-char here)
       nil)))
@@ -7725,6 +7871,9 @@ c-forward-decl-or-cast-1
   ;;           inside a function declaration arglist).
   ;; '<>       In an angle bracket arglist.
   ;; 'arglist  Some other type of arglist.
+  ;; 'top      Some other context and point is at the top-level (either
+  ;;           outside any braces or directly inside a class or namespace,
+  ;;           etc.)
   ;; nil       Some other context or unknown context.  Includes
   ;;           within the parens of an if, for, ... construct.
   ;; 'not-decl This value is never supplied to this function.  It
@@ -8133,7 +8282,7 @@ c-forward-decl-or-cast-1
 			      maybe-typeless
 			      backup-maybe-typeless
 			      (when c-recognize-typeless-decls
-				(and (not context)
+				(and (memq context '(nil top))
 				     ;; Deal with C++11's "copy-initialization"
 				     ;; where we have <type>(<constant>), by
 				     ;; contrasting with a typeless
@@ -8196,7 +8345,7 @@ c-forward-decl-or-cast-1
 
 	 (setq at-decl-end
 	       (looking-at (cond ((eq context '<>) "[,>]")
-				 (context "[,)]")
+				 ((not (memq context '(nil top))) "[,)]")
 				 (t "[,;]"))))
 
 	 ;; Now we've collected info about various characteristics of
@@ -8325,7 +8474,7 @@ c-forward-decl-or-cast-1
 
 	   (if (and got-parens
 		    (not got-prefix)
-		    (not context)
+		    (memq context '(nil top))
 		    (not (eq at-type t))
 		    (or backup-at-type
 			maybe-typeless
@@ -8375,6 +8524,18 @@ c-forward-decl-or-cast-1
 	       ;; instantiation expression).
 	       (throw 'at-decl-or-cast nil))))
 
+	 ;; CASE 9.5
+	 (when (and (not context)	; i.e. not at top level.
+		    (c-major-mode-is 'c++-mode)
+		    (eq at-decl-or-cast 'ids)
+		    after-paren-pos)
+	   ;; We've got something like "foo bar (...)" in C++ which isn't at
+	   ;; the top level.  This is probably a uniform initialization of bar
+	   ;; to the contents of the parens.  In this case the declarator ends
+	   ;; at the open paren.
+	   (goto-char (1- after-paren-pos))
+	   (throw 'at-decl-or-cast t))
+
 	 ;; CASE 10
 	 (when at-decl-or-cast
 	   ;; By now we've located the type in the declaration that we know
@@ -8383,7 +8544,7 @@ c-forward-decl-or-cast-1
 
 	 ;; CASE 11
 	 (when (and got-identifier
-		    (not context)
+		    (memq context '(nil top))
 		    (looking-at c-after-suffixed-type-decl-key)
 		    (if (and got-parens
 			     (not got-prefix)
@@ -8479,7 +8640,7 @@ c-forward-decl-or-cast-1
 	       (when (and got-prefix-before-parens
 			  at-type
 			  (or at-decl-end (looking-at "=[^=]"))
-			  (not context)
+			  (memq context '(nil top))
 			  (or (not got-suffix)
 			      at-decl-start))
 		 ;; Got something like "foo * bar;".  Since we're not inside
@@ -8505,7 +8666,7 @@ c-forward-decl-or-cast-1
 		 (throw 'at-decl-or-cast t)))
 
 	   ;; CASE 18
-	   (when (and context
+	   (when (and (not (memq context '(nil top)))
 		      (or got-prefix
 			  (and (eq context 'decl)
 			       (not c-recognize-paren-inits)
diff --git a/lisp/progmodes/cc-fonts.el b/lisp/progmodes/cc-fonts.el
index 735108f..da9d226 100644
--- a/lisp/progmodes/cc-fonts.el
+++ b/lisp/progmodes/cc-fonts.el
@@ -991,7 +991,7 @@ c-font-lock-<>-arglists
 	  (goto-char pos)))))
   nil)
 
-(defun c-font-lock-declarators (limit list types)
+(defun c-font-lock-declarators (limit list types &optional not-top)
   ;; Assuming the point is at the start of a declarator in a declaration,
   ;; fontify the identifier it declares.  (If TYPES is set, it does this via
   ;; the macro `c-fontify-types-and-refs'.)
@@ -1001,7 +1001,9 @@ c-font-lock-declarators
   ;; additionally, mark the commas with c-type property 'c-decl-id-start or
   ;; 'c-decl-type-start (according to TYPES).  Stop at LIMIT.
   ;;
-  ;; If TYPES is non-nil, fontify all identifiers as types.
+  ;; If TYPES is non-nil, fontify all identifiers as types.  If NOT-TOP is
+  ;; non-nil, we are not at the top-level ("top-level" includes being directly
+  ;; inside a class or namespace, etc.).
   ;;
   ;; Nil is always returned.  The function leaves point at the delimiter after
   ;; the last declarator it processes.
@@ -1025,6 +1027,14 @@ c-font-lock-declarators
       (setq next-pos (point)
 	    id-start (car decl-res)
 	    id-face (if (and (eq (char-after) ?\()
+			     (or (not (c-major-mode-is 'c++-mode))
+				 (not not-top)
+				 (car (cddr (cddr decl-res))) ; Id is in
+							      ; parens, etc.
+				 (save-excursion
+				   (forward-char)
+				   (c-forward-syntactic-ws)
+				   (looking-at "[*&]")))
 			     (not (car (cddr decl-res))) ; brackets-after-id
 			     (or (not (c-major-mode-is 'c++-mode))
 				 (save-excursion
@@ -1199,7 +1209,7 @@ c-font-lock-declarations
        c-decl-start-re
        (eval c-maybe-decl-faces)
 
-       (lambda (match-pos inside-macro)
+       (lambda (match-pos inside-macro &optional toplev)
 	 ;; Note to maintainers: don't use `limit' inside this lambda form;
 	 ;; c-find-decl-spots sometimes narrows to less than `limit'.
 	 (setq start-pos (point))
@@ -1223,7 +1233,7 @@ c-font-lock-declarations
 	    (let ((type (and (> match-pos (point-min))
 			     (c-get-char-property (1- match-pos) 'c-type))))
 	      (cond ((not (memq (char-before match-pos) '(?\( ?, ?\[ ?< ?{)))
-		     (setq context nil
+		     (setq context (and toplev 'top)
 			   c-restricted-<>-arglists nil))
 		    ;; A control flow expression or a decltype
 		    ((and (eq (char-before match-pos) ?\()
@@ -1273,7 +1283,7 @@ c-font-lock-declarations
 					  'c-not-decl))
 		    ;; We're inside an "ordinary" open brace.
 		    ((eq (char-before match-pos) ?{)
-		     (setq context nil
+		     (setq context (and toplev 'top)
 			   c-restricted-<>-arglists nil))
 		    ;; Inside an angle bracket arglist.
 		    ((or (eq type 'c-<>-arg-sep)
@@ -1411,7 +1421,7 @@ c-font-lock-declarations
 		  (goto-char (car decl-or-cast))
 
 		  (let ((decl-list
-			 (if context
+			 (if (not (memq context '(nil top)))
 			     ;; Should normally not fontify a list of
 			     ;; declarators inside an arglist, but the first
 			     ;; argument in the ';' separated list of a "for"
@@ -1437,7 +1447,8 @@ c-font-lock-declarations
 						 'c-decl-id-start)))))
 
 		    (c-font-lock-declarators
-		     (min limit (point-max)) decl-list (cadr decl-or-cast)))
+		     (min limit (point-max)) decl-list
+		     (cadr decl-or-cast) (not toplev)))
 
 		  ;; A declaration has been successfully identified, so do all the
 		  ;; fontification of types and refs that've been recorded.
@@ -2413,7 +2424,7 @@ c-font-lock-objc-methods
      limit
      "[-+]"
      nil
-     (lambda (match-pos inside-macro)
+     (lambda (match-pos inside-macro &optional top-level)
        (forward-char)
        (c-font-lock-objc-method))))
   nil)
diff --git a/lisp/progmodes/cc-langs.el b/lisp/progmodes/cc-langs.el
index ae6e6a3..d50aba9 100644
--- a/lisp/progmodes/cc-langs.el
+++ b/lisp/progmodes/cc-langs.el
@@ -479,10 +479,12 @@ c-populate-syntax-table
 	c-before-change-check-<>-operators
 	c-depropertize-CPP
 	c-before-after-change-digit-quote
-	c-invalidate-macro-cache)
+	c-invalidate-macro-cache
+	c-truncate-bs-cache)
   (c objc) '(c-extend-region-for-CPP
 	     c-depropertize-CPP
-	     c-invalidate-macro-cache)
+	     c-invalidate-macro-cache
+	     c-truncate-bs-cache)
   ;; java 'c-before-change-check-<>-operators
   awk 'c-awk-record-region-clear-NL)
 (c-lang-defvar c-get-state-before-change-functions
@@ -2588,6 +2590,28 @@ 'c-opt-op-identitier-prefix
 (c-lang-defvar c-opt-inexpr-brace-list-key
   (c-lang-const c-opt-inexpr-brace-list-key))
 
+(c-lang-defconst c-flat-decl-block-kwds
+  ;; Keywords that can introduce another declaration level, i.e. where a
+  ;; following "{" isn't a function block or brace list.  Note that, for
+  ;; historical reasons, `c-decl-block-key' is NOT constructed from this lang
+  ;; const.
+  t (c--delete-duplicates
+     (append (c-lang-const c-class-decl-kwds)
+	     (c-lang-const c-other-block-decl-kwds)
+	     (c-lang-const c-inexpr-class-kwds))
+     :test 'string-equal))
+
+(c-lang-defconst c-brace-stack-thing-key
+  ;; Regexp matching any keyword or operator relevant to the brace stack (see
+  ;; `c-update-brace-stack' in cc-engine.el).
+  t (c-make-keywords-re 'appendable
+      (append
+       (c-lang-const c-flat-decl-block-kwds)
+       (if (c-lang-const c-recognize-<>-arglists)
+	   '("{" "}" ";" "<")
+	 '("{" "}" ";")))))
+(c-lang-defvar c-brace-stack-thing-key (c-lang-const c-brace-stack-thing-key))
+
 (c-lang-defconst c-decl-block-key
   ;; Regexp matching keywords in any construct that contain another
   ;; declaration level, i.e. that isn't followed by a function block
@@ -3031,6 +3055,28 @@ 'c-opt-op-identitier-prefix
 (c-lang-defvar c-type-decl-prefix-key (c-lang-const c-type-decl-prefix-key)
   'dont-doc)
 
+(c-lang-defconst c-type-decl-operator-prefix-key
+  "Regexp matching any declarator operator which isn't a keyword
+that might precede the identifier in a declaration, e.g. the
+\"*\" in \"char *argv\".  The end of the first submatch is taken
+as the end of the operator.  Identifier syntax is in effect when
+this is matched \(see `c-identifier-syntax-table')."
+  t ;; Default to a regexp that never matches.
+    "\\<\\>"
+  ;; Check that there's no "=" afterwards to avoid matching tokens
+  ;; like "*=".
+  (c objc) (concat "\\(\\*\\)"
+		   "\\([^=]\\|$\\)")
+  c++  (concat "\\("
+	       "\\.\\.\\."
+	       "\\|"
+	       "\\*"
+	       "\\)"
+	       "\\([^=]\\|$\\)")
+  pike "\\(\\*\\)\\([^=]\\|$\\)")
+(c-lang-defvar c-type-decl-operator-prefix-key
+  (c-lang-const c-type-decl-operator-prefix-key))
+
 (c-lang-defconst c-type-decl-suffix-key
   "Regexp matching the declarator operators that might follow after the
 identifier in a declaration, e.g. the \"[\" in \"char argv[]\".  This
diff --git a/lisp/progmodes/cc-mode.el b/lisp/progmodes/cc-mode.el
index f2c6256..5b0679a 100644
--- a/lisp/progmodes/cc-mode.el
+++ b/lisp/progmodes/cc-mode.el
@@ -557,6 +557,8 @@ c-basic-common-init
 
   ;; Initialize the cache of brace pairs, and opening braces/brackets/parens.
   (c-state-cache-init)
+  ;; Initialize the "brace stack" cache.
+  (c-init-bs-cache)
 
   (when (or c-recognize-<>-arglists
 	    (c-major-mode-is 'awk-mode)


> Oleh

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: CC-mode highlight change between 24.5 and 25

[Oleh Krehel]

[-- Attachment #1: Type: text/plain, Size: 1050 bytes --]

Hi Alan,

Alan Mackenzie <acm@muc.de> writes:

> I was a little over confident in my brace counting.  The code was
> landing at the "class" inside the template delimiters, and moving
> forward over the "{", then saying "I'm now at the top level inside a
> class".  Rubbish!
>
> I've (hopefully) corrected that in the patch that follows which, again,
> is the full patch which should apply to the savannah master.  (Please
> just discard the patch from yesterday.)
>
> So, please try out the following, and let me know, again, how it copes
> with real C++ code.  Thanks!

I'm testing the updated code. Looks much better. But I attach an example
where it still highlights a variable as a function. It's a strange
example as well: the comments and previous statements are significant. I
reduced the code to around 10 lines and it's not easy to reduce further.

Here's how I test it:

    emacs -Q cctest.cc

The variable `gfs' is highlighted as a function. The highlight is quite
volatile: adding a newline before the line removes the highlight.

Oleh


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: cctest.cc --]
[-- Type: text/x-c++src, Size: 552 bytes --]

template<class GV>
void example01b_Q2 (const GV& gv)
{
  // <<<1>>> Choose domain and range field type
  typedef typename GV::Grid::ctype Coord;
  typedef double Real;

  // <<<2>>> Make grid function space
  typedef Dune::PDELab::QkLocalFiniteElementMap<GV,Coord,Real,2> FEM;
  FEM fem(gv);
  // typedef Dune::PDELab::NoConstraints CON;
  typedef Dune::PDELab::ConformingDirichletConstraints CON; // constraints class
  typedef Dune::PDELab::ISTLVectorBackend<> VBE;
  typedef Dune::PDELab::GridFunctionSpace<GV,FEM,CON,VBE> GFS;
  GFS gfs(gv,fem);
}

Re: CC-mode highlight change between 24.5 and 25

[Alan Mackenzie]

Hello, Oleh.

On Tue, Sep 06, 2016 at 03:37:55PM +0200, Oleh Krehel wrote:
> Hi Alan,

> Alan Mackenzie <acm@muc.de> writes:

[ .... ]

> > So, please try out the following, and let me know, again, how it copes
> > with real C++ code.  Thanks!

> I'm testing the updated code. Looks much better. But I attach an example
> where it still highlights a variable as a function. It's a strange
> example as well: the comments and previous statements are significant. I
> reduced the code to around 10 lines and it's not easy to reduce further.

Thanks for taking the trouble to reduce it.  It really does help!

> Here's how I test it:

>     emacs -Q cctest.cc

> The variable `gfs' is highlighted as a function. The highlight is quite
> volatile: adding a newline before the line removes the highlight.

I've tracked the cause of this down.  The line with "gfs" is at the
start of a (500 byte) jit-lock fontification hunk.  So CC Mode checks
whether the first form in the hunk is really a sub-form of what just
came before.

It then calls a critical function `c-font-lock-declarators', but
_without_ the &optional new argument `not-top' which should be t to
signify "not at the top level".  So this identifier gets fontified as
though it were at the top level, i.e. as a function.

Note that if you fontify the line with M-o M-o, for example, the above
scenario doesn't occur, and the id gets font-lock-variable-name-face.

There are quite a few calls to `c-font-lock-declarators', and I'm going
to have to attend to each one of them individually.  So, give me a
little time, and then I'll send you patch version 3.

> Oleh

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: CC-mode highlight change between 24.5 and 25

[Alan Mackenzie]

Hello, Oleh.

On Tue, Sep 06, 2016 at 05:02:17PM +0000, Alan Mackenzie wrote:
> On Tue, Sep 06, 2016 at 03:37:55PM +0200, Oleh Krehel wrote:
> > Hi Alan,

> > Alan Mackenzie <acm@muc.de> writes:

> [ .... ]

> > > So, please try out the following, and let me know, again, how it copes
> > > with real C++ code.  Thanks!

> > I'm testing the updated code. Looks much better. But I attach an example
> > where it still highlights a variable as a function. It's a strange
> > example as well: the comments and previous statements are significant. I
> > reduced the code to around 10 lines and it's not easy to reduce further.

> Thanks for taking the trouble to reduce it.  It really does help!

> > Here's how I test it:

> >     emacs -Q cctest.cc

> > The variable `gfs' is highlighted as a function. The highlight is quite
> > volatile: adding a newline before the line removes the highlight.

> I've tracked the cause of this down.  The line with "gfs" is at the
> start of a (500 byte) jit-lock fontification hunk.  So CC Mode checks
> whether the first form in the hunk is really a sub-form of what just
> came before.

> It then calls a critical function `c-font-lock-declarators', but
> _without_ the &optional new argument `not-top' which should be t to
> signify "not at the top level".  So this identifier gets fontified as
> though it were at the top level, i.e. as a function.

> Note that if you fontify the line with M-o M-o, for example, the above
> scenario doesn't occur, and the id gets font-lock-variable-name-face.

> There are quite a few calls to `c-font-lock-declarators', and I'm going
> to have to attend to each one of them individually.  So, give me a
> little time, and then I'll send you patch version 3.

OK, time for a fresh patch.  We're getting there, slowly.

This patch should fix the problems with cctest.cc, as described in my
last post.  It also fixes a problem in the "top-level" counting where
the comma in "class foo : bar, baz { ...." caused the code to think that
the "...." weren't at top level.

Again, please try this out on your real C++ code, and let me know how
well it fares.  Thanks!

> > Oleh



diff --git a/lisp/progmodes/cc-engine.el b/lisp/progmodes/cc-engine.el
index 259f8a0..1c667e8 100644
--- a/lisp/progmodes/cc-engine.el
+++ b/lisp/progmodes/cc-engine.el
@@ -422,6 +422,15 @@ c-forward-to-cpp-define-body
 \f
 ;;; Basic utility functions.
 
+(defmacro c-pull-open-brace (ps)
+  ;; Pull the next open brace from PS (which has the form of paren-state),
+  ;; skipping over any brace pairs.  Returns NIL when PS is exhausted.
+  `(progn
+     (while (consp (car ,ps))
+       (setq ,ps (cdr ,ps)))
+     (prog1 (car ,ps)
+       (setq ,ps (cdr ,ps)))))
+
 (defun c-delq-from-dotted-list (elt dlist)
   ;; If ELT is a member of the (possibly dotted) list DLIST, remove all
   ;; occurrences of it (except for any in the last cdr of DLIST).
@@ -5128,6 +5137,200 @@ c-debug-remove-decl-spot-faces
        (c-debug-remove-face ,beg ,end 'c-debug-decl-spot-face)
        (c-debug-remove-face ,beg ,end 'c-debug-decl-sws-face))))
 
+;;;; NEW STOUGH, 2016-09-02, and ...-03
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+;; Machinery for determining when we're at top level (this including being
+;; directly inside a class or namespace, etc.)
+;;
+;; We maintain a stack of brace depths in structures like classes and
+;; namespaces.  The car of this structure, when non-nil, indicates that the
+;; associated position is within a template (etc.) structure, and the value is
+;; the position where the (outermost) template ends.  The other elements in
+;; the structure are stacked elements, one each for each enclosing structure.
+;;
+;; At the very outermost level, the value of the stack would be (nil 1), the
+;; "1" indicating an enclosure in a notional all-enclosing block.  After
+;; passing a keyword such as "namespace", the value would become (nil 0 1).
+;; At this point, passing a semicolon would cause the 0 to be dropped from the
+;; stack (at any other time, a semicolon is ignored).  Alternatively, on
+;; passing an opening brace, the stack would become (nil 1 1).  Each opening
+;; brace passed causes the cadr to be incremented, and passing closing braces
+;; causes it to be decremented until it reaches 1.  On passing a closing brace
+;; when the cadr of the stack is at 1, this causes it to be removed from the
+;; stack, the corresponding namespace (etc.) structure having been closed.
+;;
+;; When the car of the stack is non-nil, i.e. when we're in a template (etc.)
+;; structure, braces are not counted.  The counting resumes only after passing
+;; the template's closing position, which is recorded in the car of the stack.
+;;
+;; The test for being at top level consists of the cadr being 0 or 1.
+;;
+;; The values of this stack throughout a buffer are cached in a simple linear
+;; cache, every 5000 characters.
+;;
+;; Note to maintainers: This cache mechanism is MUCH faster than recalculating
+;; the stack at every entry to `c-find-decl-spots' using `c-at-toplevel-p' or
+;; the like.
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+(defconst c-bs-interval 5000)
+(defvar c-bs-cache nil)
+(make-variable-buffer-local 'c-bs-cache)
+(defvar c-bs-cache-limit 1)
+(make-variable-buffer-local 'c-bs-cache-limit)
+(defvar c-bs-prev-pos most-positive-fixnum)
+(make-variable-buffer-local 'c-bs-prev-pos)
+(defvar c-bs-prev-stack nil)
+(make-variable-buffer-local 'c-bs-prev-stack)
+
+(defun c-init-bs-cache ()
+  ;; Initialize the cache in `c-bs-cache' and related variables.
+  (setq c-bs-cache nil
+	c-bs-cache-limit 1
+	c-bs-prev-pos most-positive-fixnum
+	c-bs-prev-stack nil))
+
+(defun c-truncate-bs-cache (pos &rest _ignore)
+  ;; Truncate the upper bound of the cache `c-bs-cache' to POS, if it is
+  ;; higher than that position.  This is called as either a before- or
+  ;; after-change-function.
+  (setq c-bs-cache-limit
+	(min c-bs-cache-limit pos)))
+
+(defun c-update-brace-stack (stack from to)
+  ;; Give a brace-stack which has the value STACK at position FROM, update it
+  ;; to it's value at position TO, where TO is after (or equal to) FROM.
+  ;; Return a cons of TO and this new value.  If TO is inside a literal, we
+  ;; replace it in the return value with the next position outside the
+  ;; literal.
+  (let (match kwd-sym (prev-match-pos 1)
+	      (s (cdr stack))
+	      (bound-<> (car stack))
+	      )
+    (save-excursion
+      (cond
+       ((and bound-<> (<= to bound-<>))
+	(goto-char to))			; Nothing to do.
+       (bound-<>
+	(goto-char bound-<>)
+	(setq bound-<> nil))
+       (t (goto-char from)))
+      (while (and (< (point) to)
+		  (c-syntactic-re-search-forward
+		   (if (<= (car s) 0)
+		       c-brace-stack-thing-key
+		     c-brace-stack-no-semi-key)
+		   to 'after-literal)
+		  (> (point) prev-match-pos)) ; prevent infinite loop.
+	(setq prev-match-pos (point))
+	(setq match (match-string-no-properties 1)
+	      kwd-sym (c-keyword-sym match))
+	(cond
+	 ((and (equal match "{")
+	       (progn (backward-char)
+		      (prog1 (looking-at "\\s(")
+			(forward-char))))
+	  (setq s (if s
+		      (cons (if (<= (car s) 0)
+				1
+			      (1+ (car s)))
+			    (cdr s))
+		    (list 1))))
+	 ((and (equal match "}")
+	       (progn (backward-char)
+		      (prog1 (looking-at "\\s)")
+			(forward-char))))
+	  (setq s
+		(cond
+		 ((and s (> (car s) 1))
+		  (cons (1- (car s)) (cdr s)))
+		 ((and (cdr s) (eq (car s) 1))
+		  (cdr s))
+		 (t s))))
+	 ((and (equal match "<")
+	       (progn (backward-char)
+		      (prog1 (looking-at "\\s(")
+			(forward-char))))
+	  (backward-char)
+	  (if (c-forward-<>-arglist nil) ; Should always work.
+	      (when (> (point) to)
+		(setq bound-<> (point)))
+	    (forward-char)))
+	 ((and (equal match ":")
+	       s
+	       (eq (car s) 0))
+	  (setq s (cons -1 (cdr s))))
+	 ((and (equal match ",")
+	       (eq (car s) -1)))	; at "," in "class foo : bar, ..."
+	 ((member match '(";" "," ")"))
+	  (when (and s (cdr s) (<= (car s) 0))
+	    (setq s (cdr s))))
+	 ((c-keyword-member kwd-sym 'c-flat-decl-block-kwds)
+	  (push 0 s))))
+      (cons (point)
+	    (cons bound-<> s)))))
+
+(defun c-brace-stack-at (here)
+  ;; Given a buffer position HERE, Return the value of the brace stack there.
+  (save-excursion
+    (save-restriction
+      (widen)
+      (let ((c c-bs-cache)
+	    (can-use-prev (<= c-bs-prev-pos c-bs-cache-limit))
+	    elt stack pos npos high-elt)
+	;; Trim the cache to take account of buffer changes.
+	(while (and c
+		    (> (caar c) c-bs-cache-limit))
+	  (setq c (cdr c)))
+	(setq c-bs-cache c)
+
+	(while (and c
+		    (> (caar c) here))
+	  (setq high-elt (car c))
+	  (setq c (cdr c)))
+	(setq pos (or (and c (caar c))
+		      (point-min)))
+
+	(setq elt (if c
+		      (car c)
+		    (cons (point-min)
+			  (cons nil (list 1)))))
+	(when (not high-elt)
+	  (setq stack (cdr elt))
+	  (while
+	      ;; Add an element to `c-state-semi-nonlit-pos-cache' each iteration.
+	      (<= (setq npos (+ pos c-bs-interval)) here)
+	    (setq elt (c-update-brace-stack stack pos npos))
+	    (setq npos (car elt))
+	    (setq stack (cdr elt))
+	    (unless (eq npos (point-max)) ; NPOS could be in a literal at EOB.
+	      (setq c-bs-cache (cons elt c-bs-cache)))
+	    (setq pos npos)))
+
+	(if (> pos c-bs-cache-limit)
+	    (setq c-bs-cache-limit pos))
+
+	;; Can we just use the previous value?
+	(if (and can-use-prev
+		 (<= c-bs-prev-pos here)
+		 (> c-bs-prev-pos (car elt)))
+	    (setq pos c-bs-prev-pos
+		  stack c-bs-prev-stack)
+	  (setq pos (car elt)
+		stack (cdr elt)))
+	(if (> here c-bs-cache-limit)
+	    (setq c-bs-cache-limit here))
+	(setq elt (c-update-brace-stack stack pos here)
+	      c-bs-prev-pos (car elt)
+	      c-bs-prev-stack (cdr elt))))))
+
+(defun c-bs-at-toplevel-p (here)
+  ;; Is position HERE at the top level, as indicated by the brace stack?
+  (let ((stack (c-brace-stack-at here)))
+    (or (null stack)			; Probably unnecessary.
+	(<= (cadr stack) 1))))
+
+;;;; END OF NEW STOUGH, 2016-09-02, and ...-03, and ...-04 - -06.
+
 (defmacro c-find-decl-prefix-search ()
   ;; Macro used inside `c-find-decl-spots'.  It ought to be a defun,
   ;; but it contains lots of free variables that refer to things
@@ -5221,6 +5424,9 @@ c-find-decl-prefix-search
 	       cfd-re-match nil)
        (setq cfd-match-pos cfd-prop-match
 	     cfd-prop-match nil))
+     (let ((stack (c-brace-stack-at cfd-match-pos)))
+       (setq cfd-top-level (or (null stack)
+			       (<= (cadr stack) 1))))
 
      (goto-char cfd-match-pos)
 
@@ -5319,7 +5525,11 @@ c-find-decl-spots
 	;; comments.
 	(cfd-token-pos 0)
 	;; The end position of the last entered macro.
-	(cfd-macro-end 0))
+	(cfd-macro-end 0)
+	;; Whether the last position returned from `c-find-decl-prefix-search'
+	;; is at the top-level (including directly in a class or namespace,
+	;; etc.).
+	cfd-top-level)
 
     ;; Initialize by finding a syntactically relevant start position
     ;; before the point, and do the first `c-decl-prefix-or-start-re'
@@ -5627,7 +5837,7 @@ c-find-decl-spots
 		   nil))))		; end of when condition
 
 	(c-debug-put-decl-spot-faces cfd-match-pos (point))
-	(if (funcall cfd-fun cfd-match-pos (/= cfd-macro-end 0))
+	(if (funcall cfd-fun cfd-match-pos (/= cfd-macro-end 0) cfd-top-level)
 	    (setq cfd-prop-match nil))
 
 	(when (/= cfd-macro-end 0)
@@ -7368,15 +7578,6 @@ c-forward-annotation
 	    t))
      (progn (goto-char pos) nil))))
 
-(defmacro c-pull-open-brace (ps)
-  ;; Pull the next open brace from PS (which has the form of paren-state),
-  ;; skipping over any brace pairs.  Returns NIL when PS is exhausted.
-  `(progn
-     (while (consp (car ,ps))
-       (setq ,ps (cdr ,ps)))
-     (prog1 (car ,ps)
-       (setq ,ps (cdr ,ps)))))
-
 (defun c-back-over-compound-identifier ()
   ;; Point is putatively just after a "compound identifier", i.e. something
   ;; looking (in C++) like this "FQN::of::base::Class".  Move to the start of
@@ -7552,10 +7753,12 @@ c-forward-declarator
   ;; Assuming point is at the start of a declarator, move forward over it,
   ;; leaving point at the next token after it (e.g. a ) or a ; or a ,).
   ;;
-  ;; Return a list (ID-START ID-END BRACKETS-AFTER-ID GOT-INIT), where ID-START and
-  ;; ID-END are the bounds of the declarator's identifier, and
-  ;; BRACKETS-AFTER-ID is non-nil if a [...] pair is present after the id.
-  ;; GOT-INIT is non-nil when the declarator is followed by "=" or "(".
+  ;; Return a list (ID-START ID-END BRACKETS-AFTER-ID GOT-INIT DECORATED),
+  ;; where ID-START and ID-END are the bounds of the declarator's identifier,
+  ;; and BRACKETS-AFTER-ID is non-nil if a [...] pair is present after the id.
+  ;; GOT-INIT is non-nil when the declarator is followed by "=" or "(",
+  ;; DECORATED it non-nil when the identifier is embellished by an operator,
+  ;; like "*x", or "(*x)".
   ;;
   ;; If ACCEPT-ANON is non-nil, move forward over any "anonymous declarator",
   ;; i.e. something like the (*) in int (*), such as might be found in a
@@ -7574,7 +7777,7 @@ c-forward-declarator
   ;; array/struct initialization) or "=" or terminating delimiter
   ;; (e.g. "," or ";" or "}").
   (let ((here (point))
-	id-start id-end brackets-after-id paren-depth)
+	id-start id-end brackets-after-id paren-depth decorated)
     (or limit (setq limit (point-max)))
     (if	(and
 	 (< (point) limit)
@@ -7614,6 +7817,8 @@ c-forward-declarator
 				 (setq got-identifier t)
 				 nil))
 			   t))
+		    (if (looking-at c-type-decl-operator-prefix-key)
+			(setq decorated t))
 		    (if (eq (char-after) ?\()
 			(progn
 			  (setq paren-depth (1+ paren-depth))
@@ -7668,7 +7873,7 @@ c-forward-declarator
 	     (setq brackets-after-id t))
 	   (backward-char)
 	   found))
-	(list id-start id-end brackets-after-id (match-beginning 1))
+	(list id-start id-end brackets-after-id (match-beginning 1) decorated)
 
       (goto-char here)
       nil)))
@@ -7744,6 +7949,9 @@ c-forward-decl-or-cast-1
   ;;           inside a function declaration arglist).
   ;; '<>       In an angle bracket arglist.
   ;; 'arglist  Some other type of arglist.
+  ;; 'top      Some other context and point is at the top-level (either
+  ;;           outside any braces or directly inside a class or namespace,
+  ;;           etc.)
   ;; nil       Some other context or unknown context.  Includes
   ;;           within the parens of an if, for, ... construct.
   ;; 'not-decl This value is never supplied to this function.  It
@@ -8152,7 +8360,7 @@ c-forward-decl-or-cast-1
 			      maybe-typeless
 			      backup-maybe-typeless
 			      (when c-recognize-typeless-decls
-				(and (not context)
+				(and (memq context '(nil top))
 				     ;; Deal with C++11's "copy-initialization"
 				     ;; where we have <type>(<constant>), by
 				     ;; contrasting with a typeless
@@ -8215,7 +8423,7 @@ c-forward-decl-or-cast-1
 
 	 (setq at-decl-end
 	       (looking-at (cond ((eq context '<>) "[,>]")
-				 (context "[,)]")
+				 ((not (memq context '(nil top))) "[,)]")
 				 (t "[,;]"))))
 
 	 ;; Now we've collected info about various characteristics of
@@ -8344,7 +8552,7 @@ c-forward-decl-or-cast-1
 
 	   (if (and got-parens
 		    (not got-prefix)
-		    (not context)
+		    (memq context '(nil top))
 		    (not (eq at-type t))
 		    (or backup-at-type
 			maybe-typeless
@@ -8394,6 +8602,18 @@ c-forward-decl-or-cast-1
 	       ;; instantiation expression).
 	       (throw 'at-decl-or-cast nil))))
 
+	 ;; CASE 9.5
+	 (when (and (not context)	; i.e. not at top level.
+		    (c-major-mode-is 'c++-mode)
+		    (eq at-decl-or-cast 'ids)
+		    after-paren-pos)
+	   ;; We've got something like "foo bar (...)" in C++ which isn't at
+	   ;; the top level.  This is probably a uniform initialization of bar
+	   ;; to the contents of the parens.  In this case the declarator ends
+	   ;; at the open paren.
+	   (goto-char (1- after-paren-pos))
+	   (throw 'at-decl-or-cast t))
+
 	 ;; CASE 10
 	 (when at-decl-or-cast
 	   ;; By now we've located the type in the declaration that we know
@@ -8402,7 +8622,7 @@ c-forward-decl-or-cast-1
 
 	 ;; CASE 11
 	 (when (and got-identifier
-		    (not context)
+		    (memq context '(nil top))
 		    (looking-at c-after-suffixed-type-decl-key)
 		    (if (and got-parens
 			     (not got-prefix)
@@ -8498,7 +8718,7 @@ c-forward-decl-or-cast-1
 	       (when (and got-prefix-before-parens
 			  at-type
 			  (or at-decl-end (looking-at "=[^=]"))
-			  (not context)
+			  (memq context '(nil top))
 			  (or (not got-suffix)
 			      at-decl-start))
 		 ;; Got something like "foo * bar;".  Since we're not inside
@@ -8524,7 +8744,7 @@ c-forward-decl-or-cast-1
 		 (throw 'at-decl-or-cast t)))
 
 	   ;; CASE 18
-	   (when (and context
+	   (when (and (not (memq context '(nil top)))
 		      (or got-prefix
 			  (and (eq context 'decl)
 			       (not c-recognize-paren-inits)
diff --git a/lisp/progmodes/cc-fonts.el b/lisp/progmodes/cc-fonts.el
index 735108f..26a002a 100644
--- a/lisp/progmodes/cc-fonts.el
+++ b/lisp/progmodes/cc-fonts.el
@@ -899,7 +899,8 @@ c-font-lock-complex-decl-prepare
 		  (c-get-char-property (1- (point)) 'c-type)))))
     (when (memq prop '(c-decl-id-start c-decl-type-start))
       (c-forward-syntactic-ws limit)
-      (c-font-lock-declarators limit t (eq prop 'c-decl-type-start))))
+      (c-font-lock-declarators limit t (eq prop 'c-decl-type-start)
+			       (c-bs-at-toplevel-p (point)))))
 
   (setq c-font-lock-context ;; (c-guess-font-lock-context)
 	(save-excursion
@@ -991,7 +992,7 @@ c-font-lock-<>-arglists
 	  (goto-char pos)))))
   nil)
 
-(defun c-font-lock-declarators (limit list types)
+(defun c-font-lock-declarators (limit list types not-top)
   ;; Assuming the point is at the start of a declarator in a declaration,
   ;; fontify the identifier it declares.  (If TYPES is set, it does this via
   ;; the macro `c-fontify-types-and-refs'.)
@@ -1001,7 +1002,9 @@ c-font-lock-declarators
   ;; additionally, mark the commas with c-type property 'c-decl-id-start or
   ;; 'c-decl-type-start (according to TYPES).  Stop at LIMIT.
   ;;
-  ;; If TYPES is non-nil, fontify all identifiers as types.
+  ;; If TYPES is non-nil, fontify all identifiers as types.  If NOT-TOP is
+  ;; non-nil, we are not at the top-level ("top-level" includes being directly
+  ;; inside a class or namespace, etc.).
   ;;
   ;; Nil is always returned.  The function leaves point at the delimiter after
   ;; the last declarator it processes.
@@ -1025,6 +1028,14 @@ c-font-lock-declarators
       (setq next-pos (point)
 	    id-start (car decl-res)
 	    id-face (if (and (eq (char-after) ?\()
+			     (or (not (c-major-mode-is 'c++-mode))
+				 (not not-top)
+				 (car (cddr (cddr decl-res))) ; Id is in
+							      ; parens, etc.
+				 (save-excursion
+				   (forward-char)
+				   (c-forward-syntactic-ws)
+				   (looking-at "[*&]")))
 			     (not (car (cddr decl-res))) ; brackets-after-id
 			     (or (not (c-major-mode-is 'c++-mode))
 				 (save-excursion
@@ -1199,7 +1210,7 @@ c-font-lock-declarations
        c-decl-start-re
        (eval c-maybe-decl-faces)
 
-       (lambda (match-pos inside-macro)
+       (lambda (match-pos inside-macro &optional toplev)
 	 ;; Note to maintainers: don't use `limit' inside this lambda form;
 	 ;; c-find-decl-spots sometimes narrows to less than `limit'.
 	 (setq start-pos (point))
@@ -1223,7 +1234,7 @@ c-font-lock-declarations
 	    (let ((type (and (> match-pos (point-min))
 			     (c-get-char-property (1- match-pos) 'c-type))))
 	      (cond ((not (memq (char-before match-pos) '(?\( ?, ?\[ ?< ?{)))
-		     (setq context nil
+		     (setq context (and toplev 'top)
 			   c-restricted-<>-arglists nil))
 		    ;; A control flow expression or a decltype
 		    ((and (eq (char-before match-pos) ?\()
@@ -1273,7 +1284,7 @@ c-font-lock-declarations
 					  'c-not-decl))
 		    ;; We're inside an "ordinary" open brace.
 		    ((eq (char-before match-pos) ?{)
-		     (setq context nil
+		     (setq context (and toplev 'top)
 			   c-restricted-<>-arglists nil))
 		    ;; Inside an angle bracket arglist.
 		    ((or (eq type 'c-<>-arg-sep)
@@ -1411,7 +1422,7 @@ c-font-lock-declarations
 		  (goto-char (car decl-or-cast))
 
 		  (let ((decl-list
-			 (if context
+			 (if (not (memq context '(nil top)))
 			     ;; Should normally not fontify a list of
 			     ;; declarators inside an arglist, but the first
 			     ;; argument in the ';' separated list of a "for"
@@ -1437,7 +1448,8 @@ c-font-lock-declarations
 						 'c-decl-id-start)))))
 
 		    (c-font-lock-declarators
-		     (min limit (point-max)) decl-list (cadr decl-or-cast)))
+		     (min limit (point-max)) decl-list
+		     (cadr decl-or-cast) (not toplev)))
 
 		  ;; A declaration has been successfully identified, so do all the
 		  ;; fontification of types and refs that've been recorded.
@@ -1498,7 +1510,7 @@ c-font-lock-enum-tail
       (c-put-char-property (1- (point)) 'c-type 'c-decl-id-start)
 
       (c-forward-syntactic-ws)
-      (c-font-lock-declarators limit t nil)))
+      (c-font-lock-declarators limit t nil t)))
   nil)
 
 (defun c-font-lock-cut-off-declarators (limit)
@@ -1512,6 +1524,7 @@ c-font-lock-cut-off-declarators
   ;; fontification".
   (let ((decl-search-lim (c-determine-limit 1000))
 	paren-state bod-res is-typedef encl-pos
+	(here (point))
 	c-recognize-knr-p)		; Strictly speaking, bogus, but it
 					; speeds up lisp.h tremendously.
     (save-excursion
@@ -1539,7 +1552,8 @@ c-font-lock-cut-off-declarators
 		  (c-forward-syntactic-ws))
 		;; At a real declaration?
 		(if (memq (c-forward-type t) '(t known found decltype))
-		    (c-font-lock-declarators limit t is-typedef)))))))
+		    (c-font-lock-declarators
+		     limit t is-typedef (not (c-bs-at-toplevel-p here)))))))))
     nil))
 
 (defun c-font-lock-enclosing-decls (limit)
@@ -1573,7 +1587,8 @@ c-font-lock-enclosing-decls
 	    (goto-char ps-elt)
 	    (when (c-safe (c-forward-sexp))
 	      (c-forward-syntactic-ws)
-	      (c-font-lock-declarators limit t in-typedef))))))))
+	      (c-font-lock-declarators limit t in-typedef
+				       (not (c-bs-at-toplevel-p (point)))))))))))
 
 (defun c-font-lock-raw-strings (limit)
   ;; Fontify C++ raw strings.
@@ -1741,7 +1756,7 @@ c-font-lock-c++-lambda-captures
       (eval . (list ,(c-make-font-lock-search-function
 		      'c-known-type-key
 		      '(1 'font-lock-type-face t)
-		      '((c-font-lock-declarators limit t nil)
+		      '((c-font-lock-declarators limit t nil nil)
 			(save-match-data
 			  (goto-char (match-end 1))
 			  (c-forward-syntactic-ws))
@@ -1763,7 +1778,7 @@ c-font-lock-c++-lambda-captures
 				 "\\)"))
 		 `(,type-match
 		   'font-lock-type-face t)
-		 `((c-font-lock-declarators limit t nil)
+		 `((c-font-lock-declarators limit t nil nil)
 		   (save-match-data
 		     (goto-char (match-end ,type-match))
 		     (c-forward-syntactic-ws))
@@ -1775,7 +1790,7 @@ c-font-lock-c++-lambda-captures
 	       (concat "\\<\\("
 		       (regexp-opt (c-lang-const c-typeless-decl-kwds))
 		       "\\)\\>")
-	       '((c-font-lock-declarators limit t nil)
+	       '((c-font-lock-declarators limit t nil nil)
 		 (save-match-data
 		   (goto-char (match-end 1))
 		   (c-forward-syntactic-ws))
@@ -2014,7 +2029,7 @@ c-font-lock-labels
 		;; before the '{' of the enum list, to avoid searching too far.
 		"[^][{};/#=]*"
 		"{")
-	       '((c-font-lock-declarators limit t nil)
+	       '((c-font-lock-declarators limit t nil t)
 		 (save-match-data
 		   (goto-char (match-end 0))
 		   (c-put-char-property (1- (point)) 'c-type
@@ -2413,7 +2428,7 @@ c-font-lock-objc-methods
      limit
      "[-+]"
      nil
-     (lambda (match-pos inside-macro)
+     (lambda (match-pos inside-macro &optional top-level)
        (forward-char)
        (c-font-lock-objc-method))))
   nil)
diff --git a/lisp/progmodes/cc-langs.el b/lisp/progmodes/cc-langs.el
index ae6e6a3..3c32848 100644
--- a/lisp/progmodes/cc-langs.el
+++ b/lisp/progmodes/cc-langs.el
@@ -479,10 +479,12 @@ c-populate-syntax-table
 	c-before-change-check-<>-operators
 	c-depropertize-CPP
 	c-before-after-change-digit-quote
-	c-invalidate-macro-cache)
+	c-invalidate-macro-cache
+	c-truncate-bs-cache)
   (c objc) '(c-extend-region-for-CPP
 	     c-depropertize-CPP
-	     c-invalidate-macro-cache)
+	     c-invalidate-macro-cache
+	     c-truncate-bs-cache)
   ;; java 'c-before-change-check-<>-operators
   awk 'c-awk-record-region-clear-NL)
 (c-lang-defvar c-get-state-before-change-functions
@@ -2588,6 +2590,41 @@ 'c-opt-op-identitier-prefix
 (c-lang-defvar c-opt-inexpr-brace-list-key
   (c-lang-const c-opt-inexpr-brace-list-key))
 
+(c-lang-defconst c-flat-decl-block-kwds
+  ;; Keywords that can introduce another declaration level, i.e. where a
+  ;; following "{" isn't a function block or brace list.  Note that, for
+  ;; historical reasons, `c-decl-block-key' is NOT constructed from this lang
+  ;; const.
+  t (c--delete-duplicates
+     (append (c-lang-const c-class-decl-kwds)
+	     (c-lang-const c-other-block-decl-kwds)
+	     (c-lang-const c-inexpr-class-kwds))
+     :test 'string-equal))
+
+(c-lang-defconst c-brace-stack-thing-key
+  ;; Regexp matching any keyword or operator relevant to the brace stack (see
+  ;; `c-update-brace-stack' in cc-engine.el).
+  t (c-make-keywords-re 'appendable
+      (append
+       (c-lang-const c-flat-decl-block-kwds)
+       (if (c-lang-const c-recognize-<>-arglists)
+	   '("{" "}" ";" "," ")" ":" "<")
+	 '("{" "}" ";" "," ")" ":")))))
+(c-lang-defvar c-brace-stack-thing-key (c-lang-const c-brace-stack-thing-key))
+
+(c-lang-defconst c-brace-stack-no-semi-key
+  ;; Regexp matching any keyword or operator relevant to the brace stack when
+  ;; a semicolon is not relevant (see `c-update-brace-stack' in
+  ;; cc-engine.el).
+  t (c-make-keywords-re 'appendable
+      (append
+       (c-lang-const c-flat-decl-block-kwds)
+       (if (c-lang-const c-recognize-<>-arglists)
+	   '("{" "}" "<")
+	 '("{" "}")))))
+(c-lang-defvar c-brace-stack-no-semi-key
+  (c-lang-const c-brace-stack-no-semi-key))
+
 (c-lang-defconst c-decl-block-key
   ;; Regexp matching keywords in any construct that contain another
   ;; declaration level, i.e. that isn't followed by a function block
@@ -3031,6 +3068,28 @@ 'c-opt-op-identitier-prefix
 (c-lang-defvar c-type-decl-prefix-key (c-lang-const c-type-decl-prefix-key)
   'dont-doc)
 
+(c-lang-defconst c-type-decl-operator-prefix-key
+  "Regexp matching any declarator operator which isn't a keyword
+that might precede the identifier in a declaration, e.g. the
+\"*\" in \"char *argv\".  The end of the first submatch is taken
+as the end of the operator.  Identifier syntax is in effect when
+this is matched \(see `c-identifier-syntax-table')."
+  t ;; Default to a regexp that never matches.
+    "\\<\\>"
+  ;; Check that there's no "=" afterwards to avoid matching tokens
+  ;; like "*=".
+  (c objc) (concat "\\(\\*\\)"
+		   "\\([^=]\\|$\\)")
+  c++  (concat "\\("
+	       "\\.\\.\\."
+	       "\\|"
+	       "\\*"
+	       "\\)"
+	       "\\([^=]\\|$\\)")
+  pike "\\(\\*\\)\\([^=]\\|$\\)")
+(c-lang-defvar c-type-decl-operator-prefix-key
+  (c-lang-const c-type-decl-operator-prefix-key))
+
 (c-lang-defconst c-type-decl-suffix-key
   "Regexp matching the declarator operators that might follow after the
 identifier in a declaration, e.g. the \"[\" in \"char argv[]\".  This
diff --git a/lisp/progmodes/cc-mode.el b/lisp/progmodes/cc-mode.el
index f2c6256..5b0679a 100644
--- a/lisp/progmodes/cc-mode.el
+++ b/lisp/progmodes/cc-mode.el
@@ -557,6 +557,8 @@ c-basic-common-init
 
   ;; Initialize the cache of brace pairs, and opening braces/brackets/parens.
   (c-state-cache-init)
+  ;; Initialize the "brace stack" cache.
+  (c-init-bs-cache)
 
   (when (or c-recognize-<>-arglists
 	    (c-major-mode-is 'awk-mode)



-- 
Alan Mackenzie (Nuremberg, Germany).

Re: CC-mode highlight change between 24.5 and 25

[Oleh Krehel]

Hi Alan,

> OK, time for a fresh patch.  We're getting there, slowly.
>
> This patch should fix the problems with cctest.cc, as described in my
> last post.  It also fixes a problem in the "top-level" counting where
> the comma in "class foo : bar, baz { ...." caused the code to think that
> the "...." weren't at top level.
>
> Again, please try this out on your real C++ code, and let me know how
> well it fares.

Thanks for the update. Everything looks good. All functions are
highlighted as functions, and non-functions aren't. I'll keep an eye out
for outliers, but it seems very solid now.

Oleh

Re: CC-mode highlight change between 24.5 and 25

[Alan Mackenzie]

Hello, Oleh.

On Fri, Sep 09, 2016 at 12:01:58PM +0200, Oleh Krehel wrote:
> Hi Alan,

> > OK, time for a fresh patch.  We're getting there, slowly.
> >
> > This patch should fix the problems with cctest.cc, as described in my
> > last post.  It also fixes a problem in the "top-level" counting where
> > the comma in "class foo : bar, baz { ...." caused the code to think that
> > the "...." weren't at top level.
> >
> > Again, please try this out on your real C++ code, and let me know how
> > well it fares.

> Thanks for the update. Everything looks good. All functions are
> highlighted as functions, and non-functions aren't. I'll keep an eye out
> for outliers, but it seems very solid now.

I've now committed that patch, both to CC Mode at SourceForge and to the
master branch at savannah.

Let's hope it doesn't need too much adjustment in the future.

> Oleh

-- 
Alan Mackenzie (Nuremberg, Germany).