bug#5570: automatic cache validation

bug#5570: automatic cache validation

[Paul Pogonyshev]

After March the 1st changes indentation in C++ and related modes is
better, but it still breaks often enough --- to the tune of once every
3--5 minutes or so.  Both when editing C++ and Java code.

As I understand, the issue is that some cached information becomes
wrong after certain (undetermined) buffer changes.  Maybe it is better
to write an automated cache validation code rather than search for a
certain way to reproduce stuff?

E.g. just build information from scratch and assert that cached
information is the same.  Ideally, such checks should be performed
after every change, so that you'd be able to find which change exactly
trigerred a bug.  (Granted, this would make debugging version slow as
hell, but with this rate of error occurence it's not like you'd need
to run it for days.)

Without automated error detection tools cc-mode is bound to have
similar issues after every second non-trivial change.

Paul

bug#5570: automatic cache validation

[Chong Yidong]

Paul Pogonyshev <pogonyshev@gmx.net> writes:

> After March the 1st changes indentation in C++ and related modes is
> better, but it still breaks often enough --- to the tune of once every
> 3--5 minutes or so.  Both when editing C++ and Java code.

This is troubling.

If the CC-mode changes are still not quite stabilized, I would like to
consider postphoning them to post-23.2.  Alan, what's your opinion?
Could we revert to the version in the branch prior to 2010-02-04 (which
IIUC is the change that introduced these problems), or do you consider
those changes too important for the release?

bug#5570: automatic cache validation

[Alan Mackenzie]

Hi, Paul,

On Thu, Mar 18, 2010 at 10:05:11PM +0200, Paul Pogonyshev wrote:
> After March the 1st changes indentation in C++ and related modes is
> better, but it still breaks often enough --- to the tune of once every
> 3--5 minutes or so.  Both when editing C++ and Java code.

How about plain C?  Is the breakage connected with templates/generics, as
far as you can see?

What is the breakage like?  Is it a one-off failure which repairs itself
after a few more edits, or is it more like a creeping death?  Does
reinitializing the mode (M-x c++-mode) repair the indentation?  Does
saving the file and reloading it fix things?

Does
    M-x (c-state-cache-init)
fix it?

> As I understand, the issue is that some cached information becomes
> wrong after certain (undetermined) buffer changes.

This is very likely.

> Maybe it is better to write an automated cache validation code rather
> than search for a certain way to reproduce stuff?

For which cache, though?  Hence the questions above.  

> E.g. just build information from scratch and assert that cached
> information is the same.  Ideally, such checks should be performed
> after every change, so that you'd be able to find which change exactly
> trigerred a bug.  (Granted, this would make debugging version slow as
> hell, but with this rate of error occurence it's not like you'd need
> to run it for days.)

I should be able to get this done by, say, Monday for the "state-cache".
Again, does M-x (c-state-cache-init) clear the fault?

> Without automated error detection tools cc-mode is bound to have
> similar issues after every second non-trivial change.

I strongly believe it's a very small finite number of distinct bugs
causing the problems.  Anything more you can tell me about how it breaks
would be of the utmost help.

> Paul

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#5570: automatic cache validation

[Jan Djärv]

Alan Mackenzie skrev 2010-03-19 22.56:
> Hi, Paul,
>
> On Thu, Mar 18, 2010 at 10:05:11PM +0200, Paul Pogonyshev wrote:
>> After March the 1st changes indentation in C++ and related modes is
>> better, but it still breaks often enough --- to the tune of once every
>> 3--5 minutes or so.  Both when editing C++ and Java code.
>
> How about plain C?  Is the breakage connected with templates/generics, as
> far as you can see?

FWIW, I see breakage in plain C also.

>
> What is the breakage like?  Is it a one-off failure which repairs itself
> after a few more edits, or is it more like a creeping death?  Does
> reinitializing the mode (M-x c++-mode) repair the indentation?  Does
> saving the file and reloading it fix things?

I now more or less automatically do M-x revert buffer when this happens.

	Jan D.

bug#5570: automatic cache validation

[Alan Mackenzie]

Hi, Jan,

On Fri, Mar 19, 2010 at 11:45:40PM +0100, Jan Djärv wrote:


> Alan Mackenzie skrev 2010-03-19 22.56:
> >Hi, Paul,

> >On Thu, Mar 18, 2010 at 10:05:11PM +0200, Paul Pogonyshev wrote:
> >>After March the 1st changes indentation in C++ and related modes is
> >>better, but it still breaks often enough --- to the tune of once every
> >>3--5 minutes or so.  Both when editing C++ and Java code.

> >How about plain C?  Is the breakage connected with templates/generics, as
> >far as you can see?

> FWIW, I see breakage in plain C also.

That's worth a great deal to know.  It suggests that the changes of
2010-02-04 aren't responsble for this bug.

> >What is the breakage like?  Is it a one-off failure which repairs itself
> >after a few more edits, or is it more like a creeping death?  Does
> >reinitializing the mode (M-x c++-mode) repair the indentation?  Does
> >saving the file and reloading it fix things?

> I now more or less automatically do M-x revert buffer when this happens.

When _what_ happens?  I take it, then, that revert-buffer fixes the bad
state, and you save the file first before reverting it.  Any chance you
could answer the above questions, together with the further one:

Does
    M-x (c-state-cache-init)
fix it?
    
These answers would be of the utmost help.  Thanks!

> 	Jan D.

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#5570: automatic cache validation

[Jan D.]

Alan Mackenzie wrote:
> Hi, Jan,
> 
> On Fri, Mar 19, 2010 at 11:45:40PM +0100, Jan Djärv wrote:
> 
> 
>> Alan Mackenzie skrev 2010-03-19 22.56:
>>> Hi, Paul,
> 
>>> On Thu, Mar 18, 2010 at 10:05:11PM +0200, Paul Pogonyshev wrote:
>>>> After March the 1st changes indentation in C++ and related modes is
>>>> better, but it still breaks often enough --- to the tune of once every
>>>> 3--5 minutes or so.  Both when editing C++ and Java code.
> 
>>> How about plain C?  Is the breakage connected with templates/generics, as
>>> far as you can see?
> 
>> FWIW, I see breakage in plain C also.
> 
> That's worth a great deal to know.  It suggests that the changes of
> 2010-02-04 aren't responsble for this bug.
> 
>>> What is the breakage like?  Is it a one-off failure which repairs itself
>>> after a few more edits, or is it more like a creeping death?  Does
>>> reinitializing the mode (M-x c++-mode) repair the indentation?  Does
>>> saving the file and reloading it fix things?
> 
>> I now more or less automatically do M-x revert buffer when this happens.
> 
> When _what_ happens? 

When C-mode thinks everything is topmost-intro.

> I take it, then, that revert-buffer fixes the bad
> state, and you save the file first before reverting it.  Any chance you
> could answer the above questions, together with the further one:
> 
> Does
>     M-x (c-state-cache-init)
> fix it?
>     

No, it does not.

	Jan D.

bug#5570: automatic cache validation

[Alan Mackenzie]

Hi, Jan

On Sat, Mar 20, 2010 at 03:49:07PM +0100, Jan D. wrote:
> Alan Mackenzie wrote:
> >On Fri, Mar 19, 2010 at 11:45:40PM +0100, Jan Djärv wrote:

> >>>How about plain C?  Is the breakage connected with
> >>>templates/generics, as far as you can see?

> >>FWIW, I see breakage in plain C also.

> >That's worth a great deal to know.  It suggests that the changes of
> >2010-02-04 aren't responsble for this bug.

> >>I now more or less automatically do M-x revert buffer when this happens.

> >When _what_ happens? 

> When C-mode thinks everything is topmost-intro.

OK.  I get the impression this is VERY annoying, but that it happens very
frequently rather than just now and then.  Good!

> >I take it, then, that revert-buffer fixes the bad
> >state, and you save the file first before reverting it.  Any chance you
> >could answer the above questions, together with the further one:

> >Does
> >    M-x (c-state-cache-init)
> >fix it?


> No, it does not.

I'm assuming your Emacs is pretty much the newest bzr version.

Would you please choose a C file, as small as will work, and start an
Emacs session with a dribble file (see page "Recording Input" in the
Elisp manual).  Edit the C file until indentation goes wrong.  Then send
me the dribble file, the (original) C file, and your CC Mode
configuration (generated by C-c C-b in the C file).

Apologies for this horrible bug, and thanks for you help in nailing it
down!

> 	Jan D.

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#5570: automatic cache validation

[Jan Djärv]

Alan Mackenzie skrev 2010-03-21 12.16:

>
> I'm assuming your Emacs is pretty much the newest bzr version.

Pretty much.

>
> Would you please choose a C file, as small as will work, and start an
> Emacs session with a dribble file (see page "Recording Input" in the
> Elisp manual).  Edit the C file until indentation goes wrong.

It isn't that easy to reproduce, it just sort of happens.
I have no idea if it will happen on a small file.

> Then send
> me the dribble file, the (original) C file, and your CC Mode
> configuration (generated by C-c C-b in the C file).
>

I'll try.

	Jan D.

bug#5570: automatic cache validation

[Jan Djärv]

Alan Mackenzie skrev:
> Hi, Jan
> 
> 
>>> Does
>>>    M-x (c-state-cache-init)
>>> fix it?
> 
> 
>> No, it does not.
> 

I just got the problem again, while editing gtkutil.c in Emacs.  There
M-: (c-state-cache-init) did indeed fix the problem.

So it sometimes help.  I guess this only adds to the confusion...

	Jan D.

bug#5570: automatic cache validation

[Paul Pogonyshev]

Jan Djärv wrote:
> Alan Mackenzie skrev:
> >>> Does
> >>>    M-x (c-state-cache-init)
> >>> fix it?
> > 
> > 
> >> No, it does not.
> > 
> 
> I just got the problem again, while editing gtkutil.c in Emacs.  There
> M-: (c-state-cache-init) did indeed fix the problem.
> 
> So it sometimes help.  I guess this only adds to the confusion...

I guess that's because there are several similar bugs.  At the rate
fixing goes (or rather doesn't go) I think I'll compile and use an old
Emacs without indentation bugs again.

Paul


P.S.  By the way: maybe this change did fix the original bug, but it
didn't improve overall indentation problems in CC modes noticeably.

2010-03-24  Alan Mackenzie  <acm@muc.de>

	* progmodes/cc-engine.el (c-remove-stale-state-cache):
	Fix off-by-one error.  Fixes bug #5747.

bug#5570: automatic cache validation

[Chong Yidong]

> >>> Does
> >>>    M-x (c-state-cache-init)
> >>> fix it?
> >
> >> No, it does not.
> >
>
> I just got the problem again, while editing gtkutil.c in Emacs.  There
> M-: (c-state-cache-init) did indeed fix the problem.
>
> So it sometimes help.  I guess this only adds to the confusion...

You sent this message on March 22.  That was the date Alan checked this
into the branch:

   2010-03-22  Alan Mackenzie  <acm@muc.de>

      * progmodes/cc-engine.el (c-remove-stale-state-cache):
      Fix off-by-one error.  Fixes bug #5747.

The patch was merged to the trunk a couple days later.  I don't know
whether you were running with it when you made the above observation.

So, are you still observing the problem after the patch?

bug#5570: automatic cache validation

[Paul Pogonyshev]

Chong Yidong wrote:
> > >>> Does
> > >>>    M-x (c-state-cache-init)
> > >>> fix it?
> > >
> > >> No, it does not.
> > >
> >
> > I just got the problem again, while editing gtkutil.c in Emacs.  There
> > M-: (c-state-cache-init) did indeed fix the problem.
> >
> > So it sometimes help.  I guess this only adds to the confusion...
> 
> You sent this message on March 22.  That was the date Alan checked this
> into the branch:
> 
>    2010-03-22  Alan Mackenzie  <acm@muc.de>
> 
>       * progmodes/cc-engine.el (c-remove-stale-state-cache):
>       Fix off-by-one error.  Fixes bug #5747.
> 
> The patch was merged to the trunk a couple days later.  I don't know
> whether you were running with it when you made the above observation.
> 
> So, are you still observing the problem after the patch?

Not directed to me, but I can answer as well.

Yes, I see the problem, though I'm not sure if it is "the" problem,
since there might be many similar ones.

* At least in some cases M-x (c-state-cache-init) doesn't help, but
  C-x RET r (buffer reloading) does;

* Occasionally even buffer reloading doesn't help, so I have to kill
  the buffer and open the file again;

* Sometimes I get wrong indentation which miraculously fixes itself
  after I hit TAB next time;

* Sometimes when I type in a C string (like "bla-bla-bla"), first
  character after the opening quote gets eaten because of a Lisp
  error.  Will report as separate bug when it happens if I don't
  forget.

And yes, I double-verified that I do have the fix in the version I'm
running now.

Paul

bug#5570: automatic cache validation

[Jan Djärv]

Chong Yidong skrev 2010-04-04 01.55:
>>>>> Does
>>>>>     M-x (c-state-cache-init)
>>>>> fix it?
>>>
>>>> No, it does not.
>>>
>>
>> I just got the problem again, while editing gtkutil.c in Emacs.  There
>> M-: (c-state-cache-init) did indeed fix the problem.
>>
>> So it sometimes help.  I guess this only adds to the confusion...
>
> You sent this message on March 22.  That was the date Alan checked this
> into the branch:
>
>     2010-03-22  Alan Mackenzie<acm@muc.de>
>
>        * progmodes/cc-engine.el (c-remove-stale-state-cache):
>        Fix off-by-one error.  Fixes bug #5747.
>
> The patch was merged to the trunk a couple days later.  I don't know
> whether you were running with it when you made the above observation

I was running without it.
.
>
> So, are you still observing the problem after the patch?

Yes, but it seems to happen less frequently.  Also, sometimes just another TAB 
at BOL fixes it.

	Jan D.

bug#5570: automatic cache validation

[Chong Yidong]

Hi Alan,

I found a small recipe for reproducing a (the?) cache bug on the trunk:

1. emacs -Q

2. C-x C-f foo.c RET

3. Type the following (from view-lossage):

  i n t SPC m a i n SPC ( ) <return>
  { <return> f o o ; <return> } M-< <return>
  <up> # i n c l u d e SPC < s t d i o . h > <return>
  <down> <down> <down> <tab>

At this point, the line "foo;" becomes indented to the zeroth column.

Does this help?

bug#5570: automatic cache validation

[Alan Mackenzie]

Hi, Yidong and Paul!

On Wed, Apr 28, 2010 at 02:55:34PM -0400, Chong Yidong wrote:
> Hi Alan,

> I found a small recipe for reproducing a (the?) cache bug on the trunk:

> 1. emacs -Q

> 2. C-x C-f foo.c RET

> 3. Type the following (from view-lossage):

>   i n t SPC m a i n SPC ( ) <return>
>   { <return> f o o ; <return> } M-< <return>
>   <up> # i n c l u d e SPC < s t d i o . h > <return>
>   <down> <down> <down> <tab>

> At this point, the line "foo;" becomes indented to the zeroth column.

> Does this help?

Does it help?  That was an absolute dream of a bug report.  :-)

The problem was another off-by-one error.  What triggered the bug is
typing <return> at the end of the #include line.  The thing marking the
end of the macro (a category text property) got pushed onto the next line
instead of being properly removed.  This then fouled up the
"c-state-cache" mechanism.

It looks like the situation reported by Paul (in his mail of 2010-02-23,
Message-Id: <201002231528.09081.pogonyshev@gmx.net>) is likely caused by
the same bug.

I've committed a fix to the Bazaar HEAD, which of course is a fix for the
last bug in this area.  ;-)

Paul, would you please update your copy of the Emacs HEAD from savannah,
and see if any indentation bugs still manifest themselves.  Thanks!

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#5570: automatic cache validation

[Paul Pogonyshev]

Alan Mackenzie wrote:
> Paul, would you please update your copy of the Emacs HEAD from savannah,
> and see if any indentation bugs still manifest themselves.  Thanks!

Yes they do.  I still see all kinds of indentation bugs, none seem to
be gone:

* Sometimes everything is topmost-intro; this may or may not be gone
  after buffer reload with C-x RET r, suggesting at least two sources
  for the bug.

* Indentation sometimes spuriosly changes when hitting TAB (e.g. TAB
  -> to 0th column, TAB again -> to 8th column, TAB third time -> to
  0th once more; no intervening keystrokes).

* Sometimes character after opening " gets eaten see #5851.

I did 'bzr pull', then 'make maintainer-clean && ./configure && make'
and then 'sudo make install'.

Paul