bug#10047: Looking for help debugging an undo issue

bug#10047: Looking for help debugging an undo issue

[Phil Sainty]

Hi all,

I'm a bit stumped, and am not even sure how to approach this.

I'm writing a major mode for entering timesheet data for use with a
custom application, and I wanted to keep the columns automatically
aligned within a block of entries for a given day, so my mode uses
`post-command-hook' to (most of the time) call `align-current', which
works more or less as I want* except that I'm encountering situations
in testing where the undo system forgets some of the history and just
reports "No further undo information". This is happening when I delete
a few lines of data, and then try to re-do it all. It gets part way
through and then fails to re-do the remainder. This is something of a
show-stopper.

I'm seeing the same issue with both Emacs 23.3 and 24.0.91.1 on both
Windows and Ubuntu.

I'm really not sure how to go about debugging the problem. I was
hoping that maybe someone can provide some insight into how my use of
align-current and post-command-hook might affect the undo system, or
otherwise suggest some smart ways of narrowing down what the problem
is? I'm really not familiar with the inner workings of the mechanisms
in question. All I can think of offhand is that (despite the test data
being only ~200 bytes) I'm managing to exceed the maximum amount of
undo data permitted, and it's silently losing the data for the older
changes??

I've verified that the issue occurs when running emacs with the -Q
option, and loading only the relevant files, although I haven't yet
boiled the code down to a minimal test case. I imagine that's the next
step, if no one has any immediate suggestions, but unfortunately the
bug seems inconsistent in when it occurs, so I've been having some
trouble even trying to do that much.


thanks,

-Phil



(*) I'm finding it trickier than I had hoped to configure the align
rules for this purpose, but that's not relevant here other than to say
that the docstring for the `align-rules-list' variable should detail
the requirement for and use of the (end reverse) arguments for a
custom "regexp" function, because that's not obvious without searching
for example usages and/or reading through the somewhat complex code
in the align library. (The problem I'm having occurs regardless of
whether I'm using a custom function or a plain regexp, however.)

bug#10047: Looking for help debugging an undo issue

[Stefan Monnier]

> I'm a bit stumped, and am not even sure how to approach this.

> I'm writing a major mode for entering timesheet data for use with a
> custom application, and I wanted to keep the columns automatically
> aligned within a block of entries for a given day, so my mode uses
> `post-command-hook' to (most of the time) call `align-current', which
> works more or less as I want* except that I'm encountering situations
> in testing where the undo system forgets some of the history and just
> reports "No further undo information". This is happening when I delete
> a few lines of data, and then try to re-do it all. It gets part way
> through and then fails to re-do the remainder. This is something of a
> show-stopper.

> I'm seeing the same issue with both Emacs 23.3 and 24.0.91.1 on both
> Windows and Ubuntu.

> I'm really not sure how to go about debugging the problem. I was
> hoping that maybe someone can provide some insight into how my use of
> align-current and post-command-hook might affect the undo system, or
> otherwise suggest some smart ways of narrowing down what the problem
> is? I'm really not familiar with the inner workings of the mechanisms
> in question. All I can think of offhand is that (despite the test data
> being only ~200 bytes) I'm managing to exceed the maximum amount of
> undo data permitted, and it's silently losing the data for the older
> changes??

Your problem doesn't ring a bell, sorry.  The best options I can think
of are:
- log the value of buffer-undo-list at each step in some auxiliary
  debugging buffer, for post-mortem analysis.
- try to come up with a reproducible test case.
  

        Stefan

bug#10047: Looking for help debugging an undo issue

[Phil Sainty]

> Your problem doesn't ring a bell, sorry.  The best options I can think
> of are:
> - log the value of buffer-undo-list at each step in some auxiliary
>    debugging buffer, for post-mortem analysis.
> - try to come up with a reproducible test case.

As it turned out, most of my code & context was irrelevant.
You should be able to reproduce the bug like this:

emacs -Q
M-: (add-hook 'post-command-hook 'align-current) RET
<backspace> (until the scratch buffer message is gone)
C-/ (or otherwise undo, until it fails with the message
"No further undo information")

It will probably occur after deleting far fewer characters
than that, but the point of failure seems variable. However,
in a half-dozen test runs just now, I've yet to have it
get as far as restoring the first half of the first line
before failing.


-Phil

bug#10047: Looking for help debugging an undo issue

[Stefan Monnier]

> As it turned out, most of my code & context was irrelevant.
> You should be able to reproduce the bug like this:

> emacs -Q
> M-: (add-hook 'post-command-hook 'align-current) RET
> <backspace> (until the scratch buffer message is gone)
> C-/ (or otherwise undo, until it fails with the message
> "No further undo information")

OK, here's my analysis:
1- align-current uses several (call it M) temporary markers.
2- they all get added to the undo-list whenever you DEL (because their
   position is modified).
3- so after N uses of DEL, you have M * N^2 entries.  In my tests, M is
   10, so after deleting 50 chars, you have 10 * 2500 = 25000
   "marker-update" entries in your undo-log.
4- at the beginning of GC, the log is considered too long (because of
   those spurious entries) and gets truncated.
5- at the end of GC, those spurious entries are removed from the log
   because those markers were temporary and only the undo-log still had
   references to them.

The problem can be fixed in 2 ways, both of which are desired:
- change align.el so it doesn't leave so many temporary markers around
  (it could use fewer markers, and it could also unset them explicitly
  at the end).
- swap 4 and 5 (actually it was swapped in Emacs-22, so that would
  revert to the behavior of Emacs-21).

I've installed a patch which reduces align.el's use of markers a little
bit.  It seems sufficient to get your test case to work, but I can't
guarantee it will solve the problem in your more general case.
It'd take a bit more analysis of align.el's code to figure out how to
fix it further.


        Stefan

bug#10047: Looking for help debugging an undo issue

[Phil Sainty]

Thank you Stefan, that's very much appreciated. (Will I find that
code in the new 24.0.92 pretest, or do I need to build from version
control?)

> 4- at the beginning of GC, the log is considered too long (because of
>     those spurious entries) and gets truncated.
> 5- at the end of GC, those spurious entries are removed from the log
>     because those markers were temporary and only the undo-log still had
>     references to them.
[...]
> - swap 4 and 5 (actually it was swapped in Emacs-22, so that would
>    revert to the behavior of Emacs-21).

I see what you mean, and yet one presumes that change was made for
a reason. Is there any indication of why that change was made?


On a related note to all of this, I found what seems like a good
question, enquiring about how to determine safe/sensible values
for increasing the undo limits, but it never got an answer. It
would be nice to see that answered:

http://lists.gnu.org/archive/html/help-gnu-emacs/2011-02/msg00112.html

bug#10047: Looking for help debugging an undo issue

[Stefan Monnier]

> Thank you Stefan, that's very much appreciated. (Will I find that
> code in the new 24.0.92 pretest, or do I need to build from version
> control?)

Yes, it should be in 24.0.92.

>> - swap 4 and 5 (actually it was swapped in Emacs-22, so that would
>> revert to the behavior of Emacs-21).
> I see what you mean, and yet one presumes that change was made for
> a reason. Is there any indication of why that change was made?

Yes, it's because the truncation now does a few more things, one of
which can resulting in prompting the user, and that can't be done where
truncation used to take place (because it's a spot where the heap is in
a special inconsistent state).
So to swap things back, we'll need to split the "old plain truncation"
which can be done late from the "new prompting truncation" which will
have to stay where it is.

> On a related note to all of this, I found what seems like a good
> question, enquiring about how to determine safe/sensible values
> for increasing the undo limits, but it never got an answer.  It
> would be nice to see that answered:
> http://lists.gnu.org/archive/html/help-gnu-emacs/2011-02/msg00112.html

There is no good answer: increasing those values will increase your
memory use, which may be perfectly OK or not depending on
your circumstance.  The current default values have nothing magical to
them, they're just choices which seemed reasonable at some point.
We could probably increase them nowadays since memory resources have
increased.


        Stefan

bug#10047: Looking for help debugging an undo issue

[Chong Yidong]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

> I've installed a patch which reduces align.el's use of markers a
> little bit.  It seems sufficient to get your test case to work, but I
> can't guarantee it will solve the problem in your more general case.
> It'd take a bit more analysis of align.el's code to figure out how to
> fix it further.

I fixed align-region to delete the temporary markers it uses.  This
seems to clear up the remaining problems.