Re: Font-lock decides function call is function declaration in C+ +

Re: Font-lock decides function call is function declaration in C+ +

[Chong Yidong]

Does this bug still exist with latest CVS?  If so, please provide a
simple test case (I couldn't find one in your previous messages).

> Hi, this bug remains[*] and a quick look at the C++ file I am editing shows
> that most of my methods have ended up with this sort of misfontification
> somewhere.  Note that misfontification does not occur during initial
> fontification, only after editing (though not in a way I can replicate).  I
> guess sometimes it thinks function calls are part of a declaration, though it
> doesn't necessarily happen when the preceding statement is a declaration.  In
> one occurrence in my C++ file, the misfontification has occurred immediately
> after a for loop.
>
> Again, I'm asking for help.  I never got a response last time.  What do I need
> to look at to see what is going on?
>
> [*] following my revision 1.18 date: 2006-11-15 16:31:03 fix in cc-fonts.el of
> another unrelated issue in CVS Emacs, Emacs CVS fontifies the below with
> font-lock-variable-name-face, rather than font-lock-function-name-face, which
> may/may not give a pointer to where the problem lies.
>
>     _____________________________________________
>     From:   Marshall, Simon 
>     Sent:   07 September 2006 17:41
>     To:     'bug-cc-mode@gnu.org'
>     Cc:     'emacs-pretest-bug@gnu.org'
>     Subject:        Font-lock decides function call is function declaration in
>     C++
>
>     Intermittently, but for ages, Emacs CVS suddenly decides that C++ lines of
>     code of the form:
>
>             foo(bar);
>
>     Or:
>
>             foo() = bar;
>
>     Are function declarations, and puts font-lock-function-name-face on (in
>     these examples) "foo".  It seems to happen when I am editing the line in
>     question, or maybe just on a line near it.  (I think the last time it
>     appeared when I did M-^ on the line following, but I could be wrong.  There
>     is nothing wrong with the code preceding the misfontification.) 
>
>     Editing the line or lines above it do not make the fontification
>     permanently go away.  For example, if I comment out all preceding lines,
>     the fontification is removed.  However, when I remove the comments, it
>     comes back.  The only way I've found to get rid of it is to turn
>     font-lock-mode off and on. 
>
>     It's driving me mad because I can't work out what, in terms of the code
>     preceding it, is causing it to fontify in this way.  In one case, the
>     function call was the first statement in the method, and I noticed that the
>     text also had the "c-in-sws" property (as well as the "face" property). 
>     Debugging through cc-engine.el doesn't look fun and I don't know if the
>     presence of this property is a red herring.
>
>     I can't reproduce it, so I'm really asking for help.  Any ideas?  What can
>     I do to work out what is going wrong?
>
>     Simon.
>
>
>
> This email message is intended for the named recipient only. It may be
> privileged and/or confidential. If you are not the named recipient of this
> email please notify us immediately and do not copy it or use it for any
> purpose, nor disclose its contents to any other person.   
>
>  
>
> Misys Banking Systems is a trading name of Misys International Banking Systems
> Limited which is registered in England and Wales under company registration
> number 00971479 and with its registered office address at Burleigh House,
> Chapel Oak, Salford Priors, Evesham WR11 8SP.
>
>  
>
> THIS E-MAIL DOES NOT CONSTITUTE THE COMMENCEMENT OF LEGAL RELATIONS BETWEEN YOU
> AND MISYS INTERNATIONAL BANKING SYSTEMS LIMITED. PLEASE REFER TO THE EXECUTED
> CONTRACT BETWEEN YOU AND THE RELEVANT MEMBER OF THE MISYS GROUP FOR THE
> IDENTITY OF THE CONTRACTING PARTY WITH WHICH YOU ARE DEALING.
>
> _______________________________________________
> emacs-pretest-bug mailing list
> emacs-pretest-bug@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-pretest-bug

Re: Font-lock decides function call is function declaration in C+ +

[Chong Yidong]

Does this bug still exist with latest CVS?  If so, please provide a
simple test case (I couldn't find one in your previous messages).

> Hi, this bug remains[*] and a quick look at the C++ file I am editing shows
> that most of my methods have ended up with this sort of misfontification
> somewhere.  Note that misfontification does not occur during initial
> fontification, only after editing (though not in a way I can replicate).  I
> guess sometimes it thinks function calls are part of a declaration, though it
> doesn't necessarily happen when the preceding statement is a declaration.  In
> one occurrence in my C++ file, the misfontification has occurred immediately
> after a for loop.
>
> Again, I'm asking for help.  I never got a response last time.  What do I need
> to look at to see what is going on?
>
> [*] following my revision 1.18 date: 2006-11-15 16:31:03 fix in cc-fonts.el of
> another unrelated issue in CVS Emacs, Emacs CVS fontifies the below with
> font-lock-variable-name-face, rather than font-lock-function-name-face, which
> may/may not give a pointer to where the problem lies.
>
>     _____________________________________________
>     From:   Marshall, Simon 
>     Sent:   07 September 2006 17:41
>     To:     'bug-cc-mode@gnu.org'
>     Cc:     'emacs-pretest-bug@gnu.org'
>     Subject:        Font-lock decides function call is function declaration in
>     C++
>
>     Intermittently, but for ages, Emacs CVS suddenly decides that C++ lines of
>     code of the form:
>
>             foo(bar);
>
>     Or:
>
>             foo() = bar;
>
>     Are function declarations, and puts font-lock-function-name-face on (in
>     these examples) "foo".  It seems to happen when I am editing the line in
>     question, or maybe just on a line near it.  (I think the last time it
>     appeared when I did M-^ on the line following, but I could be wrong.  There
>     is nothing wrong with the code preceding the misfontification.) 
>
>     Editing the line or lines above it do not make the fontification
>     permanently go away.  For example, if I comment out all preceding lines,
>     the fontification is removed.  However, when I remove the comments, it
>     comes back.  The only way I've found to get rid of it is to turn
>     font-lock-mode off and on. 
>
>     It's driving me mad because I can't work out what, in terms of the code
>     preceding it, is causing it to fontify in this way.  In one case, the
>     function call was the first statement in the method, and I noticed that the
>     text also had the "c-in-sws" property (as well as the "face" property). 
>     Debugging through cc-engine.el doesn't look fun and I don't know if the
>     presence of this property is a red herring.
>
>     I can't reproduce it, so I'm really asking for help.  Any ideas?  What can
>     I do to work out what is going wrong?
>
>     Simon.
>
>
>
> This email message is intended for the named recipient only. It may be
> privileged and/or confidential. If you are not the named recipient of this
> email please notify us immediately and do not copy it or use it for any
> purpose, nor disclose its contents to any other person.   
>
>  
>
> Misys Banking Systems is a trading name of Misys International Banking Systems
> Limited which is registered in England and Wales under company registration
> number 00971479 and with its registered office address at Burleigh House,
> Chapel Oak, Salford Priors, Evesham WR11 8SP.
>
>  
>
> THIS E-MAIL DOES NOT CONSTITUTE THE COMMENCEMENT OF LEGAL RELATIONS BETWEEN YOU
> AND MISYS INTERNATIONAL BANKING SYSTEMS LIMITED. PLEASE REFER TO THE EXECUTED
> CONTRACT BETWEEN YOU AND THE RELEVANT MEMBER OF THE MISYS GROUP FOR THE
> IDENTITY OF THE CONTRACTING PARTY WITH WHICH YOU ARE DEALING.
>
> _______________________________________________
> emacs-pretest-bug mailing list
> emacs-pretest-bug@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-pretest-bug

RE: Font-lock decides function call is function declaration in C+ +

[Marshall, Simon]

> Does this bug still exist with latest CVS?  If so, please provide a
> simple test case (I couldn't find one in your previous messages).

Yes, it does, with CVS Emacs as of 05/02/2007.  Originally, I couldn't
reproduce it on demand.  That's why, originally, I was asking for help to
track it down.

But, messing around just now, I've finally managed to do it.  I can't say
that these are the only way of reproducing it, as I've seen this
misfontification in a variety of situations that do not look obviously like
these.

1.  The goal is to write the code snippet:

int main() {
  foo();
  bar();
}

emacs -Q foo.cpp
int SPC main() SPC { RET } RET C-p C-o bar();

OK so far.  Now to insert the "foo();" line:

C-a C-o foo

At this point, "foo" is fontified as a type, and "bar" as a variable.  OK.
Now:

()

The fontification of "foo" and "bar" disappears.  OK.  Now complete the
snippet:

;

Now "foo" is fontified as a variable.  This is wrong.

2.  Here's a variation.  This time, the goal is to write the code snippet:

int main() {
  foo(fubar);
  bar();
}

emacs -Q bar.cpp
int SPC main() SPC { RET } RET C-p C-o bar();

OK so far.  Now to insert the "foo(fubar);" line:

C-a C-o foo(fubar

At this point, "bar" is fontified as a type.  Not sure why, but still.  Now:

);

Now "bar" is fontified as a variable.  This is wrong.

What is worse is that in both cases I cannot get rid of the misfontification
without turning Font Lock mode off and on again.

I hope this help to fix The Most Annoying Fontification Bug Ever.  Simon.


This email message is intended for the named recipient only. It may be privileged and/or confidential. If you are not the named recipient of this email please notify us immediately and do not copy it or use it for any purpose, nor disclose its contents to any other person.       Misys Banking Systems is a trading name of Misys International Banking Systems Limited which is registered in England and Wales under company registration number 00971479 and with its registered office address at Burleigh House, Chapel Oak, Salford Priors, Evesham WR11 8SP.    THIS E-MAIL DOES NOT CONSTITUTE THE COMMENCEMENT OF LEGAL RELATIONS BETWEEN YOU AND MISYS INTERNATIONAL BANKING SYSTEMS LIMITED. PLEASE REFER TO THE EXECUTED CONTRACT BETWEEN YOU AND THE RELEVANT MEMBER OF THE MISYS GROUP FOR THE IDENTITY OF THE CONTRACTING PARTY WITH WHICH YOU ARE DEALING. 

Re: Font-lock decides function call is function declaration in C+ +

[Chong Yidong]

"Marshall, Simon" <simon.marshall@misys.com> writes:

> emacs -Q foo.cpp
> int SPC main() SPC { RET } RET C-p C-o bar();
>
> OK so far.  Now to insert the "foo();" line:
>
> C-a C-o foo
>
> At this point, "foo" is fontified as a type, and "bar" as a variable.  OK.
> Now:
>
> ()
>
> The fontification of "foo" and "bar" disappears.  OK.  Now complete the
> snippet:
>
> ;
>
> Now "foo" is fontified as a variable.  This is wrong.

I can reproduce this now.  Note that it only happens in c++ mode, not
in plain c-mode.

Hopefully, someone familiar with the cc-mode code can debug it.

Re: Font-lock decides function call is function declaration in C+ +

[Alan Mackenzie]

Hi, Simon!

On Mon, Feb 05, 2007 at 04:46:32PM -0000, Marshall, Simon wrote:
> > Does this bug still exist with latest CVS?  If so, please provide a
> > simple test case (I couldn't find one in your previous messages).

> Yes, it does, with CVS Emacs as of 05/02/2007.  Originally, I couldn't
> reproduce it on demand.  That's why, originally, I was asking for help to
> track it down.

> But, messing around just now, I've finally managed to do it.  I can't say
> that these are the only way of reproducing it, as I've seen this
> misfontification in a variety of situations that do not look obviously like
> these.

> 1.  The goal is to write the code snippet:

> int main() {
>   foo();
>   bar();
> }

> emacs -Q foo.cpp
> int SPC main() SPC { RET } RET C-p C-o bar();

> OK so far.  Now to insert the "foo();" line:

> C-a C-o foo

> At this point, "foo" is fontified as a type, and "bar" as a variable.  OK.
> Now:

> ()

> The fontification of "foo" and "bar" disappears.  OK.  Now complete the
> snippet:

> ;

> Now "foo" is fontified as a variable.  This is wrong.

It is indeed wrong.  

Further observations:
(i) The bug doesn't happen in C Mode.
The bug happens:
(ii) in Emacs 5.21.3/CC Mode 5.31.4
(iii) With jit-lock disabled (thankfully ;-)
(iv) in Emacs 5.21.3/CC Mode 5.30.3

So it seems its entirely a CC Mode bug.  I'm going to try and track it
down, though I suspect it could be quite tricky to find.  Thanks for
giving that recipe for producing the bug.

> 2.  Here's a variation.  [ snipped ]

> What is worse is that in both cases I cannot get rid of the
> misfontification without turning Font Lock mode off and on again.

M-o M-o fixes the fontification for me.

> I hope this help to fix The Most Annoying Fontification Bug Ever.
> Simon.

I think there're also other candidates for that description.  ;-(

-- 
Alan Mackenzie (Ittersbach, Germany).

RE: Font-lock decides function call is function declaration in C+ +

[Marshall, Simon]

> So it seems its entirely a CC Mode bug.  I'm going to try and 
> track it down, though I suspect it could be quite tricky to 
> find.  Thanks for giving that recipe for producing the bug.

Thanks.

> > I hope this help to fix The Most Annoying Fontification Bug Ever.
> 
> I think there're also other candidates for that description.  ;-(

Don't poke the monster.

Uh-huh, he's woken.


This email message is intended for the named recipient only. It may be privileged and/or confidential. If you are not the named recipient of this email please notify us immediately and do not copy it or use it for any purpose, nor disclose its contents to any other person.       Misys Banking Systems is a trading name of Misys International Banking Systems Limited which is registered in England and Wales under company registration number 00971479 and with its registered office address at Burleigh House, Chapel Oak, Salford Priors, Evesham WR11 8SP.    THIS E-MAIL DOES NOT CONSTITUTE THE COMMENCEMENT OF LEGAL RELATIONS BETWEEN YOU AND MISYS INTERNATIONAL BANKING SYSTEMS LIMITED. PLEASE REFER TO THE EXECUTED CONTRACT BETWEEN YOU AND THE RELEVANT MEMBER OF THE MISYS GROUP FOR THE IDENTITY OF THE CONTRACTING PARTY WITH WHICH YOU ARE DEALING. 

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Re: Font-lock decides function call is function declaration in C+ +

[Chong Yidong]

Alan Mackenzie <acm@muc.de> writes:

> Further observations:
> (i) The bug doesn't happen in C Mode.
> The bug happens:
> (ii) in Emacs 5.21.3/CC Mode 5.31.4
> (iii) With jit-lock disabled (thankfully ;-)
> (iv) in Emacs 5.21.3/CC Mode 5.30.3
>
> So it seems its entirely a CC Mode bug.  I'm going to try and track it
> down, though I suspect it could be quite tricky to find.  Thanks for
> giving that recipe for producing the bug.

I think the problem is that when the buffer is in the state

  foo
  bar ();

foo looks like a type, so it is inserted in the cache variable
c-found-types.  When you add the "()":

  foo()
  bar();

this guess is no longer valid, but foo is not removed from
c-found-types.  Therefore, when you insert the final ";",

  foo();
  bar();

the call to foo(); is highlighted as though it were a constructor.
Note that if you do M-: (c-clear-found-types) prior to inserting the
final ";", the misfontification does not occur.

I don't know what the Right Fix is, however.

Re: Font-lock decides function call is function declaration in C+ +

[Alan Mackenzie]

Hi, Chong!

On Fri, Feb 09, 2007 at 04:25:21PM -0500, Chong Yidong wrote:
> Alan Mackenzie <acm@muc.de> writes:

> I think the problem is that when the buffer is in the state

>   foo
>   bar ();

> foo looks like a type, so it is inserted in the cache variable
> c-found-types.

Exactly!  Thanks for that.

> When you add the "()":

>   foo()
>   bar();

> this guess is no longer valid, but foo is not removed from
> c-found-types.  Therefore, when you insert the final ";",

>   foo();
>   bar();

> the call to foo(); is highlighted as though it were a constructor.
> Note that if you do M-: (c-clear-found-types) prior to inserting the
> final ";", the misfontification does not occur.

> I don't know what the Right Fix is, however.

Once a variable has been inserted into c-found-types, it will stay there
almost for ever; it will stay there until re-fontification is done from
BOB (I'm not sure whether or not that also means (point-min) on a
narrowed buffer).  This permanence seems to be the fundamental problem.

To Simon: Sorry about my saying that "M-o M-o will clear the spurious
font-lock-type-face".  It only does that when point is within 16 lines of
BOB.  :-(

I think a solution might be to remove "foo" from c-found-types whenever
text is inserted/deleted in the vicinity of "foo\n bar ();" which
syntactically destroys its status as a type identifier.  I'll need to
think a lot more about this.

-- 
Alan.

Re: Font-lock decides function call is function declaration in C+ +

[Stefan Monnier]

> I think a solution might be to remove "foo" from c-found-types whenever
> text is inserted/deleted in the vicinity of "foo\n bar ();" which
> syntactically destroys its status as a type identifier.  I'll need to
> think a lot more about this.

I'd be interested to know what solution you end up using.  I've bumped into
a similar problem in sml-mode where I tried to look at the infix precedence
declarations in the file while font-locking to adjust the indentation
algorithm.

The best solution I found so far is to add a special text property covering
the "infix" declaration and whose value is the symbol whose precedence and
associativity is being set, and then in font-lock-unfontify-region I begin
by looking for this special property and removing the corresponding symbols
from the precedence table.  But this failed when the whole line was deleted
since the unfontify gets called after the text (and its text-property) is
removed (i.e. too late), so I ended up using before-change-functions
instead, which seemed too costly compared to the importance of the feature,
so it's still in the "experimental" state.

Maybe a better approach is to use overlays rather than text-properties, so
they don't completely disappear if the text is removed.


        Stefan

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Re: Font-lock decides function call is function declaration in C+ +

[Chong Yidong]

Alan Mackenzie <acm@muc.de> writes:

> Once a variable has been inserted into c-found-types, it will stay there
> almost for ever; it will stay there until re-fontification is done from
> BOB (I'm not sure whether or not that also means (point-min) on a
> narrowed buffer).  This permanence seems to be the fundamental problem.
>
> I think a solution might be to remove "foo" from c-found-types whenever
> text is inserted/deleted in the vicinity of "foo\n bar ();" which
> syntactically destroys its status as a type identifier.  I'll need to
> think a lot more about this.

For the Emacs 22 release, if no simple fix is forthcoming, could we
simply get font-lock to avoid higlighting constructor functions?

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Re: Font-lock decides function call is function declaration in C+ +

[Stefan Monnier]

>> Once a variable has been inserted into c-found-types, it will stay there
>> almost for ever; it will stay there until re-fontification is done from
>> BOB (I'm not sure whether or not that also means (point-min) on a
>> narrowed buffer).  This permanence seems to be the fundamental problem.
>> 
>> I think a solution might be to remove "foo" from c-found-types whenever
>> text is inserted/deleted in the vicinity of "foo\n bar ();" which
>> syntactically destroys its status as a type identifier.  I'll need to
>> think a lot more about this.

> For the Emacs 22 release, if no simple fix is forthcoming, could we
> simply get font-lock to avoid higlighting constructor functions?

Or to not try to auto-learn types,


        Stefan

RE: Font-lock decides function call is function declaration in C+ +

[Marshall, Simon]

> >> Once a variable has been inserted into c-found-types, it will stay
there
> >> almost for ever; it will stay there until re-fontification is done from
> >> BOB (I'm not sure whether or not that also means (point-min) on a
> >> narrowed buffer).  This permanence seems to be the fundamental problem.
> >> 
> >> I think a solution might be to remove "foo" from c-found-types whenever
> >> text is inserted/deleted in the vicinity of "foo\n bar ();" which
> >> syntactically destroys its status as a type identifier.  I'll need to
> >> think a lot more about this.
> 
> > For the Emacs 22 release, if no simple fix is forthcoming, could we
> > simply get font-lock to avoid higlighting constructor functions?
> 
> Or to not try to auto-learn types,

Yes, unfortunately, I think that if you try to learn types on-the-fly then
you will always be vulnerable to this sort of problem.  The issue is that
cc-mode needs to know as soon as a change invalidates it as a candidate type
(ie, deletion of some/all of the text "foo" or interruption of the
whitespace between "foo" and its candidate identifier "bar").  I can think
of a few ways you could attempt to do it, but they are a bit intensive and
far from simple.

Perhaps cc-mode should only try to learn types when fontifying the whole
buffer?

Simon.


This email message is intended for the named recipient only. It may be privileged and/or confidential. If you are not the named recipient of this email please notify us immediately and do not copy it or use it for any purpose, nor disclose its contents to any other person.       Misys Banking Systems is a trading name of Misys International Banking Systems Limited which is registered in England and Wales under company registration number 00971479 and with its registered office address at Burleigh House, Chapel Oak, Salford Priors, Evesham WR11 8SP.    THIS E-MAIL DOES NOT CONSTITUTE THE COMMENCEMENT OF LEGAL RELATIONS BETWEEN YOU AND MISYS INTERNATIONAL BANKING SYSTEMS LIMITED. PLEASE REFER TO THE EXECUTED CONTRACT BETWEEN YOU AND THE RELEVANT MEMBER OF THE MISYS GROUP FOR THE IDENTITY OF THE CONTRACTING PARTY WITH WHICH YOU ARE DEALING. 

RE: Font-lock decides function call is function declaration in C+ +

[Stuart D. Herring]

> Yes, unfortunately, I think that if you try to learn types on-the-fly then
> you will always be vulnerable to this sort of problem.  The issue is that
> cc-mode needs to know as soon as a change invalidates it as a candidate
> type
> (ie, deletion of some/all of the text "foo" or interruption of the
> whitespace between "foo" and its candidate identifier "bar").  I can think
> of a few ways you could attempt to do it, but they are a bit intensive and
> far from simple.

It's worse than that.  Inserting a new letter that changes "foo", or
transposing two characters 3k back in the buffer that cause the whole
region to become a comment, or deleting "bar", or exchanging "foo" and
"bar", or merely exchanging 'f' and 'o', or adding #define in front of
"foo", or... any of which could be done as one change by an appropriate
Lisp helper function.

Davis

-- 
This product is sold by volume, not by mass.  If it appears too dense or
too sparse, it is because mass-energy conversion has occurred during
shipping.

Re: Font-lock decides function call is function declaration in C+ +

[Stefan Monnier]

>> Yes, unfortunately, I think that if you try to learn types on-the-fly then
>> you will always be vulnerable to this sort of problem.  The issue is that
>> cc-mode needs to know as soon as a change invalidates it as a candidate
>> type
>> (ie, deletion of some/all of the text "foo" or interruption of the
>> whitespace between "foo" and its candidate identifier "bar").  I can think
>> of a few ways you could attempt to do it, but they are a bit intensive and
>> far from simple.

> It's worse than that.  Inserting a new letter that changes "foo", or
> transposing two characters 3k back in the buffer that cause the whole
> region to become a comment, or deleting "bar", or exchanging "foo" and
> "bar", or merely exchanging 'f' and 'o', or adding #define in front of
> "foo", or... any of which could be done as one change by an appropriate
> Lisp helper function.

Yes, of course, the case of commenting code needs to be handled as well, but
re-using the font-lock machinery makes it "somewhat painless".


        Stefan

Re: Font-lock decides function call is function declaration in C+ +

[Alan Mackenzie]

Evening, Stuart and Simon!

On Mon, Feb 12, 2007 at 07:53:53AM -0800, Stuart D. Herring wrote:
> > Yes, unfortunately, I think that if you try to learn types
> > on-the-fly then you will always be vulnerable to this sort of
> > problem.  The issue is that cc-mode needs to know as soon as a
> > change invalidates it as a candidate type (ie, deletion of some/all
> > of the text "foo" or interruption of the whitespace between "foo"
> > and its candidate identifier "bar").  I can think of a few ways you
> > could attempt to do it, but they are a bit intensive and far from
> > simple.

This is true, but such a way has already been built into CC Mode (by
Martin Stjernholm).  It just[*] needs a little tweaking.

[*] To the cynical - hold off a few days before commenting on this
adverb, please!

> It's worse than that.  Inserting a new letter that changes "foo", ...

is taken care of, in the commonest case.  When you type the "l" after
"foo", this replaces "foo" with "fool" in the cache c-found-types.

> ... or transposing two characters 3k back in the buffer that cause the
> whole region to become a comment, ....

is a problem throughout lots of modes in Emacs.  c-found-types doesn't
exacerbate it much (or at all?) for CC Mode.

> ... or deleting "bar", or exchanging "foo" and "bar", or merely
> exchanging 'f' and 'o', or adding #define in front of "foo", or... any
> of which could be done as one change by an appropriate Lisp helper
> function.

I think most of these can be made to work quite soon, if they don't
already.  However, you're more than welcome to test these scenarios out
and post the results here.  ;-)

> Davis

-- 
Alan Mackenzie (Ittersbach, Germany).

Re: Font-lock decides function call is function declaration in C+ +

[Alan Mackenzie]

Hi, Chong and Stefan!

On Sun, Feb 11, 2007 at 06:18:34PM -0500, Chong Yidong wrote:
> Alan Mackenzie <acm@muc.de> writes:

> > Once a variable has been inserted into c-found-types, it will stay
> > there almost for ever; it will stay there until re-fontification is
> > done from BOB (I'm not sure whether or not that also means
> > (point-min) on a narrowed buffer).  This permanence seems to be the
> > fundamental problem.

> > I think a solution might be to remove "foo" from c-found-types
> > whenever text is inserted/deleted in the vicinity of "foo\n bar ();"
> > which syntactically destroys its status as a type identifier.  I'll
> > need to think a lot more about this.

> For the Emacs 22 release, if no simple fix is forthcoming, could we
> simply get font-lock to avoid higlighting constructor functions?

Give me a few more days, please!

int main () {
   foo(               <=================
   bar();
}

I think I can now see how to do this removal of "foo" from c-found-types:
After experimenting with C++ Mode, I conjecture that each "critical" use
of "foo" is marked with one of the values (c-decl-id-start
c-decl-type-start) for the property c-type.  (Hint: it's not at the
buffer position you might expect.)

If this proves true, or almost true, it will be easy enough to amend CC
Mode's before/after-change function to remove these identifiers from
c-found-types.

Give me a few more days to try this out, please!

-- 
Alan Mackenzie (Ittersbach, Germany).

Re: Font-lock decides function call is function declaration in C+ +

[Stuart D. Herring]

>> It's worse than that.  Inserting a new letter that changes "foo", ...
>
> is taken care of, in the commonest case.  When you type the "l" after
> "foo", this replaces "foo" with "fool" in the cache c-found-types.

Perhaps the cache should note how many times something is used?  Then the
relatively common case of

socket s;
socket_factory sf/*=...*/;
s.initializeFrom(sf);

would not remove `socket' from the cache, but typo-corrections can still
change it.

Davis

-- 
This product is sold by volume, not by mass.  If it appears too dense or
too sparse, it is because mass-energy conversion has occurred during
shipping.

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Re: Font-lock decides function call is function declaration in C+ +

[Alan Mackenzie]

Hi, Stuart!

On Mon, Feb 12, 2007 at 10:06:22AM -0800, Stuart D. Herring wrote:
> >> It's worse than that.  Inserting a new letter that changes "foo", ...

> > is taken care of, in the commonest case.  When you type the "l" after
> > "foo", this replaces "foo" with "fool" in the cache c-found-types.

> Perhaps the cache should note how many times something is used?  Then the
> relatively common case of

> socket s;
> socket_factory sf/*=...*/;
> s.initializeFrom(sf);

> would not remove `socket' from the cache, but typo-corrections can still
> change it.

NO!!!  It isn't that sort of cache.  It's purely there to speed things
up, since without it, the various after-change functions would have to
analyse the source extensively for types for every character you type.

It's important that the cache contains most of the type identifiers.  It
doesn't much matter if one or two are missing.  What DOES matter is when
the cache has identifiers which aren't types (even if they once were).
This is what's causing the bug here.

I've got a clearer idea of where and how to remove ex-types than I did
last night.

> Davis

-- 
Alan.

Re: Font-lock decides function call is function declaration in C+ + - embryonic solution.

[Alan Mackenzie]

Hi, Simon, Chong, Steffan!

On Mon, Feb 05, 2007 at 04:46:32PM -0000, Marshall, Simon wrote:

[ .... ]

> 1.  The goal is to write the code snippet:

> int main() {
>   foo();
>   bar();
> }

> emacs -Q foo.cpp
> int SPC main() SPC { RET } RET C-p C-o bar();

> OK so far.  Now to insert the "foo();" line:

> C-a C-o foo

> At this point, "foo" is fontified as a type, and "bar" as a variable.  OK.
> Now:

> ()

> The fontification of "foo" and "bar" disappears.  OK.  Now complete the
> snippet:

> ;

> Now "foo" is fontified as a variable.  This is wrong.

I've got an embryonic solution for the problem.  The basic idea is to
create a before-change function which looks for certain `c-type'
properties in the vicinity of the change - these indicate that "foo" is a
member of `c-found-types'.  An after-change function can then remove
"foo" from this cache.

To get an idea of what's going on, do
   M-: (c-list-found-types)
or
   M-: c-maybe-stale-found-type
.  The latter is the data structure passed between the
{before,after}-change-functions.

Don't get worried by the size/complexity of this new code.  The worst
thing it's going to do is wrongly take "foo" out of this cache - this
will slow Emacs, but won't cause it to crash.

THIS ISN'T PRODUCTION QUALITY CODE, or anywher near it, so please don't
"debug" it or "tidy it up" for me!  I'm posting it mainly to give
credibility to the notion that I'm making headway with this problem.  In
particular, it only solves Simon's first bug recipe.  It doesn't yet
solve the second one (which will probably be quite easy to fix), and it
doesn't yet deal with template types in `c-found-types', or with
comments, strings, macros, narrowed regions, .....

With that said, here is the embryonic patch to cc-engine.el and
cc-mode.el.



*** cc-engine.220207.el	2007-02-03 00:17:53.000000000 +0000
--- cc-engine.el	2007-02-23 00:09:24.096985896 +0000
***************
*** 2491,2514 ****
    ;; Move to the beginning of the current token.  Do not move if not
    ;; in the middle of one.  BACK-LIMIT may be used to bound the
    ;; backward search; if given it's assumed to be at the boundary
!   ;; between two tokens.
    ;;
    ;; This function might do hidden buffer changes.
-   (if (looking-at "\\w\\|\\s_")
-       (skip-syntax-backward "w_" back-limit)
      (let ((start (point)))
!       (when (< (skip-syntax-backward ".()" back-limit) 0)
! 	(while (let ((pos (or (and (looking-at c-nonsymbol-token-regexp)
! 				   (match-end 0))
! 			      ;; `c-nonsymbol-token-regexp' should always match
! 			      ;; since we've skipped backward over punctuator
! 			      ;; or paren syntax, but consume one char in case
! 			      ;; it doesn't so that we don't leave point before
! 			      ;; some earlier incorrect token.
! 			      (1+ (point)))))
! 		 (if (<= pos start)
! 		     (goto-char pos))
! 		 (< pos start)))))))
  
  (defun c-end-of-current-token (&optional back-limit)
    ;; Move to the end of the current token.  Do not move if not in the
--- 2491,2515 ----
    ;; Move to the beginning of the current token.  Do not move if not
    ;; in the middle of one.  BACK-LIMIT may be used to bound the
    ;; backward search; if given it's assumed to be at the boundary
!   ;; between two tokens.  Return non-nil if the point is move, nil
!   ;; otherwise.
    ;;
    ;; This function might do hidden buffer changes.
      (let ((start (point)))
!       (if (looking-at "\\w\\|\\s_")
! 	  (skip-syntax-backward "w_" back-limit)
! 	(when (< (skip-syntax-backward ".()" back-limit) 0)
! 	  (while (let ((pos (or (and (looking-at c-nonsymbol-token-regexp)
! 				     (match-end 0))
! 				;; `c-nonsymbol-token-regexp' should always match
! 				;; since we've skipped backward over punctuator
! 				;; or paren syntax, but consume one char in case
! 				;; it doesn't so that we don't leave point before
! 				;; some earlier incorrect token.
! 				(1+ (point)))))
! 		   (if (<= pos start)
! 		       (goto-char pos))))))
!       (< (point) start)))
  
  (defun c-end-of-current-token (&optional back-limit)
    ;; Move to the end of the current token.  Do not move if not in the
***************
*** 3957,3962 ****
--- 3958,3966 ----
  ;; file, and we only use this as a last resort in ambiguous cases (see
  ;; `c-forward-decl-or-cast-1').
  ;;
+ ;; Not every type need be in this cache.  However, things which have
+ ;; ceased to be types must be removed from it.
+ ;;
  ;; Template types in C++ are added here too but with the template
  ;; arglist replaced with "<>" in references or "<" for the one in the
  ;; primary type.  E.g. the type "Foo<A,B>::Bar<C>" is stored as
***************
*** 3990,3995 ****
--- 3994,4003 ----
        (unintern (substring type 0 -1) c-found-types)
        (intern type c-found-types))))
  
+ (defsubst c-unfind-type (name)
+   ;; Remove the "NAME" from c-found-types, if present.
+   (unintern name c-found-types))
+ 
  (defsubst c-check-type (from to)
    ;; Return non-nil if the given region contains a type in
    ;; `c-found-types'.
***************
*** 4008,4013 ****
--- 4016,4038 ----
  	      c-found-types)
      (sort type-list 'string-lessp)))
  
+ (defun c-clean-found-types (beg end old-len)
+   ;; An after change function which, in conjunction with the info in
+   ;; c-maybe-stale-found-type (set in c-before-change), removes a type
+   ;; from `c-found-types', should this type have become stale.  For
+   ;; example, this happens to "foo" when "foo \n bar();" becomes
+   ;; "foo(); \n bar();".  Such stale types, if not removed, foul up
+   ;; the fontification.
+   (if c-maybe-stale-found-type ; e.g. (c-decl-id-start "foo" 97 107 " (* ooka) " "o")
+       (cond
+       ;; Some cases which don't disrupt the string: don't needlessly
+       ;; remove "foo"
+        (nil)				; code this up.  FIXME!!!
+        ((eq (car c-maybe-stale-found-type) 'c-decl-id-start)
+ 	(c-unfind-type (cadr c-maybe-stale-found-type))))))
+ 
+ 
+ 
  \f
  ;; Handling of small scale constructs like types and names.
  
*** cc-mode.220207.el	2007-01-01 21:18:31.000000000 +0000
--- cc-mode.el	2007-02-23 00:34:17.569943496 +0000
***************
*** 412,419 ****
  ;; temporary changes in some font lock support modes, causing extra
  ;; unnecessary work and font lock glitches due to interactions between
  ;; various text properties.
  
! (defun c-after-change (beg end len)
    ;; Function put on `after-change-functions' to adjust various caches
    ;; etc.  Prefer speed to finesse here, since there will be an order
    ;; of magnitude more calls to this function than any of the
--- 412,525 ----
  ;; temporary changes in some font lock support modes, causing extra
  ;; unnecessary work and font lock glitches due to interactions between
  ;; various text properties.
+ ;; 
+ ;; (2007-02-12): The macro `combine-after-change-calls' ISN'T used any
+ ;; more.
+ 
+ ;; c-maybe-stale-found-type records a place near the region being
+ ;; changed where an element of `found-types' might become stale.  It 
+ ;; is set in c-before-change and is either nil, or has the form:
+ ;;
+ ;;   (97 107 c-decl-id-start " (* ooka) " "o"), where
+ ;;   
+ ;; o - 97 107 is the region potentially containing the stale type -
+ ;;   this is delimited by a non-nil c-type text property at 96 and
+ ;;   either another one or a ";", "{", or "}" at 107.
+ ;; 
+ ;; o - `c-decl-id-start' is the c-type text property value at buffer
+ ;;   pos 96.
+ ;; 
+ ;; o - " (* ooka) " is the (before change) buffer portion containing
+ ;;   the suspect type (here "ooka").
+ ;;
+ ;; o - "o" is the buffer contents which is about to be deleted.  This
+ ;;   would be the empty string for an insertion.
+ 
+ (defvar c-maybe-stale-found-type nil)
+ (make-variable-buffer-local 'c-maybe-stale-found-type)
+ (defun c-before-change (beg end)
+   ;; Function to be put on `before-change-function'.  Currently
+   ;; (2007-02) it is used only to remove stale entries from the
+   ;; `c-found-types' cache, and to record entries which a
+   ;; `c-after-change' function might confirm as stale.
+   ;; 
+   ;; Note that this function must be FAST rather than accurate.  Note
+   ;; also that it only has any effect when font locking is enabled.
+   ;; We exploit this by checking for font-lock-*-face instead of doing
+   ;; rigourous syntactic analysis.
+ 
+   ;; If either change boundary is wholly inside an identifier, delete
+   ;; it/them from the cache.  Don't worry about being inside a string
+   ;; or a comment - "wrongly" removing a symbol from `c-found-types'
+   ;; isn't critical.
+   (setq c-maybe-stale-found-type nil)
+   (save-excursion
+     ;; Are we inserting/deleting stuff in the middle of an identifier?
+     (cond
+      ((let (tok-beg tok-end)
+ 	(not (equal
+ 	      (mapcar
+ 	       (lambda (pos)
+ 		 (goto-char pos)
+ 		 (setq tok-beg (and (c-beginning-of-current-token) (point)))
+ 		 (goto-char pos)
+ 		 (setq tok-end (and (c-end-of-current-token) (point)))
+ 		 (if (and tok-beg tok-end)
+ 		     (c-unfind-type (buffer-substring-no-properties tok-beg tok-end))))
+ 	       `(,beg ,end))
+ 	      '(nil nil)))))
+ 
+     ;; Are we (potentially) disrupting the syntactic context which
+     ;; makes a type a type?  E.g. by inserting stuff after "foo" in
+     ;; "foo bar;", or before "foo" in "typedef foo *bar;"?
+     ;;
+     ;; We search for appropriate c-type properties "near" the change.
+     ;; First, find an appropriate boundary for this property search.
+      ((let (lim
+ 	   type type-pos
+ 	   marked-id term-pos
+ 	   (end1
+ 	    (if (eq (get-text-property end 'face) 'font-lock-comment-face)
+ 		(previous-single-property-change end 'face)
+ 	      end)))
+        (when (>= end1 beg) ; Don't hassle about changes entirely in comments.
+ 	 (skip-chars-backward "^;{}") ; FIXME!!!  loop for comment, maybe
+ 	 (setq lim (max (point-min) (1- (point))))
+ 	 (when (and (> end1 1)
+ 		    (setq type-pos
+ 			  (if (get-text-property (1- end1) 'c-type)
+ 			      end1
+ 			    (previous-single-property-change end1 'c-type nil lim))))
+ 	   (setq type (get-text-property (max (1- type-pos) lim) 'c-type))
+ 	   (cond
+ 	    ((memq type '(c-decl-id-start c-decl-type-start))
+ 	     ;; Get the identifier, if any, that the property is on.
+ 	     (goto-char (1- type-pos))
+ 	     (setq marked-id
+ 		   (when (looking-at "\\(\\sw\\|\\s_\\)")
+ 		     (c-beginning-of-current-token)
+ 		     (buffer-substring-no-properties (point) type-pos)))
+ 
+ 	     (goto-char end1)
+ 	     (skip-chars-forward "^;{}") ; FIXME!!!  loop for comment, maybe
+ 	     (setq lim (point))
+ 	     (setq term-pos
+ 		   (or (next-single-property-change end 'c-type nil lim) lim))
+ 	     (setq c-maybe-stale-found-type
+ 		   (list type marked-id
+ 			 type-pos term-pos
+ 			 (buffer-substring-no-properties type-pos term-pos)
+ 			 (buffer-substring-no-properties beg end))))
+ 
+ 	    ;; 	   ((eq type 'c-decl-type-start)
+ 	    ;; 	    (################
+ 
+ 	    (type (message "Unhandled c-type at %s" type-pos)))
+ 
+ 	   )))))))
+   
  
! (defun c-after-change (beg end old-len)
    ;; Function put on `after-change-functions' to adjust various caches
    ;; etc.  Prefer speed to finesse here, since there will be an order
    ;; of magnitude more calls to this function than any of the
***************
*** 441,446 ****
--- 547,553 ----
  	  (when (> beg end)
  	    (setq beg end)))
  
+ 	(c-clean-found-types beg end old-len) ; maybe we don't need all of these.
  	(c-invalidate-sws-region-after beg end)
  	(c-invalidate-state-cache beg)
  	(c-invalidate-find-decl-cache beg)
***************
*** 577,582 ****
--- 684,691 ----
  
    ;; Install the functions that ensure that various internal caches
    ;; don't become invalid due to buffer changes.
+   (make-local-hook 'before-change-functions)
+   (add-hook 'before-change-functions 'c-before-change nil t)
    (make-local-hook 'after-change-functions)
    (add-hook 'after-change-functions 'c-after-change nil t))



-- 
Alan Mackenzie (Ittersbach, Germany).

Re: Font-lock decides function call is function declaration in C+ + - embryonic solution.

[Chong Yidong]

Hi Alan,

Judging by the size of the patch you sent, and the time it's taking to
write it, this seems to be a lot of work.  Once we include the time
necessary for testing such a big change, it will probably
significantly delay the Emacs 22 release.

In the interest of release Emacs 22 sometime in 2007, I'd like to
suggest postphoning this fix to Emacs 23, and simply turning off
variable-highlighting for C++ constructors.

What do you think?

Re: Font-lock decides function call is function declaration in C+ + - embryonic solution.

[Richard Stallman]

    In the interest of release Emacs 22 sometime in 2007, I'd like to
    suggest postphoning this fix to Emacs 23, and simply turning off
    variable-highlighting for C++ constructors.

I agree.

Re: Font-lock decides function call is function declaration in C+ + - embryonic solution.

[Alan Mackenzie]

Hi, Chong and Richard!

On Thu, Mar 01, 2007 at 12:19:57PM -0500, Chong Yidong wrote:
> Hi Alan,

> Judging by the size of the patch you sent, and the time it's taking to
> write it, this seems to be a lot of work. 

It's really not a big patch.  It's lots of lines of lisp, yet it's mainly
straight through, top to bottom, without complicated loops, and the
functions it calls are almost entirely base Emacs functions.

I confess I got bogged down trying to write a general fix to the problem.
I've given that up for now.  Sorry.

> Once we include the time necessary for testing such a big change, it
> will probably significantly delay the Emacs 22 release.
 
> In the interest of release Emacs 22 sometime in 2007, I'd like to
> suggest postphoning this fix to Emacs 23, and simply turning off
> variable-highlighting for C++ constructors.
 
> What do you think?
 
That turning off the highlighting would take longer to implement and
test.  [BTW, I don't think it's connected with C++ constructors, unless
I've missed something - it's about removing type names from a cache when
those names cease to be types.]

I suggest instead that I tidy up that embryonic patch and commit it.  I
think I'll be able to do this today or tomorrow.  It will not fully
resolve the bug, but it will help.

#########################################################################

Simon's bug report also gave this "variation" of the bug:

int main() {
  foo(fubar);
  bar();
}

, where "bar" got fontified as a variable when the fragment is typed in
a certain way.  This is actually a distinct bug, which I think will be
easier to fix.

-- 
Alan Mackenzie (Ittersbach, Germany).

Re: Font-lock decides function call is function declaration in C+ + - Patch

[Alan Mackenzie]

Hi, Chong and Simon

On Thu, Mar 01, 2007 at 12:19:57PM -0500, Chong Yidong wrote:
> Hi Alan,

> Judging by the size of the patch you sent, and the time it's taking to
> write it, this seems to be a lot of work.  Once we include the time
> necessary for testing such a big change, it will probably
> significantly delay the Emacs 22 release.

int main () {
    foo         <===== type "();", then erase it.
    bar ();
}

I've just committed a patch to cc-mode.el and cc-engine.el which fixes
Simon's (first) bug.  [The "variant" he mentioned is a distinct bug.]

It's a fairly scrappy ad-hoc patch - it fixes a particular simple bug
scenario which will surely occur frequently.  Its idea is to remove
tokens from the cache `c-found-types' whenever these tokens are broken
up (e.g. by typing stuff in the middle of them, or removing space which
separated them from other tokens) or when their syntactic identity as
types ceases is disrupted.

I think this cache mechanism should to be thought through and either
abolished or amended to work consistently.

-- 
Alan Mackenzie (Ittersbach Germany).

Re: Font-lock decides function call is function declaration in C+ + - Patch

[Chong Yidong]

Alan Mackenzie <acm@muc.de> writes:

> It's a fairly scrappy ad-hoc patch - it fixes a particular simple bug
> scenario which will surely occur frequently.  Its idea is to remove
> tokens from the cache `c-found-types' whenever these tokens are broken
> up (e.g. by typing stuff in the middle of them, or removing space which
> separated them from other tokens) or when their syntactic identity as
> types ceases is disrupted.
>
> I think this cache mechanism should to be thought through and either
> abolished or amended to work consistently.

Thanks.

Re: Font-lock decides function call is function declaration in C+ + - Patch

[Richard Stallman]

    It's a fairly scrappy ad-hoc patch - it fixes a particular simple bug
    scenario which will surely occur frequently.

What I worry about is, how likely is it to cause new bugs in other
cases?

It would be safer just to turn off this cache feature.
How much would users mind that?

Re: Font-lock decides function call is function declaration in C+ + - Patch

[Alan Mackenzie]

Hi, Richard!

On Fri, Mar 09, 2007 at 04:25:35PM -0500, Richard Stallman wrote:
>     It's a fairly scrappy ad-hoc patch - it fixes a particular simple bug
>     scenario which will surely occur frequently.

> What I worry about is, how likely is it to cause new bugs in other
> cases?

I was worrying about that all the time I was formulating the bug and
writing the patch.  The answer is, IMBO[*], not very likely.  Here's my
analysis:

(i) The patch doesn't change the buffer contents (apart from, possibly,
text properties on some characters).  It intended effect is purely to
remove type-identifiers from the cache `c-found-types'.  Removing a type
from this cache when it should stay there wouldn't be that bad.  Failing
to remove a stale type would be no worse than the code prior to the fix.

(ii) Bugs like going off the end of a buffer, getting lisp errors with
empty buffers, and such like, are possible.

(iii) Bugs like forgetting to widen are possible.

[*] "In my biassed opion"

> It would be safer just to turn off this cache feature.

That would throw out some of the fontification, and would slow things
down.  I don't know by how much; I would guess quite a lot.

> How much would users mind that?

I would guess some (like Simon ;-) would mind a lot.  Others (like me)
would hardly notice.  I think, in general, it wouldn't be good to take
this cache out, now.  It's been in place since CC Mode 5.30 (summer
2003).

How about letting Simon test it on his C++ code?

-- 
Alan Mackenzie (Ittersbach, Germany).

Re: Font-lock decides function call is function declaration in C+ + - Patch

[Richard Stallman]

I guess we are stuck with this fix.