bug#24094: 25.1.50; revert-buffer error in CC mode

bug#24094: 25.1.50; revert-buffer error in CC mode

[Richard Copley]

When editing C++ files, if I change visited files outside emacs (for
example, by doing "svn revert -R ."), then visit one of the changed
files and accept the offer to revert the buffer, in some cases there
is an error (see below) and the buffer contents are corrupted (chunks
are missing because the revert operation was interrupted).

I haven't been able to reduce this to a recipe and I don't know if
the issue is present in the emacs-25 branch and/or in "emacs -Q".

Here is an example backtrace (control characters replaced):

Debugger entered--Lisp error: (error "Invalid search bound (wrong side
of point)")
  re-search-forward("[0-9a-fA-F]'[0-9a-fA-F]" 175 t)
  c-before-after-change-digit-quote(65 65 1625)
  #[(fn) "^H    \n^K#\207" [fn beg end old-len]
4](c-before-after-change-digit-quote)
  mapc(#[(fn) "^H    \n^K#\207" [fn beg end old-len] 4]
(c-depropertize-new-text c-extend-font-lock-region-for-macros
c-before-after-change-digit-quote c-after-change-re-mark-raw-strings
c-neutralize-syntax-in-and-mark-CPP c-restore-<>-properties
c-change-expand-fl-region))
  c-after-change(65 65 1625)
  insert-file-contents("g:/projects/polymorph/working3/src/settings.cpp"
t nil nil t)
  revert-buffer-insert-file-contents--default-function("g:/projects/polymorph/working3/src/settings.cpp"
nil)
  revert-buffer--default(t t)
  revert-buffer(t t)
  find-file-noselect("g:/projects/polymorph/working3/src/settings.cpp")
  compilation-find-file(#<marker at 1397 in *grep*> "settings.cpp" nil)
  apply(compilation-find-file #<marker at 1397 in *grep*>
"settings.cpp" nil nil)
  compilation-next-error-function(1 nil)
  next-error(nil)
  funcall-interactively(next-error nil)
  call-interactively(next-error nil nil)
  command-execute(next-error)


In GNU Emacs 25.1.50.1 (x86_64-w64-mingw32)
 of 2016-07-25 built on MACHINE
Repository revision: 6dc6b0079ed3632ed9082bc79d8cb6fc96d33f43
Windowing system distributor 'Microsoft Corp.', version 10.0.10586
Recent messages:
Undo!
Saving file g:/projects/polymorph/working3/src/model.cpp...
Wrote g:/projects/polymorph/working3/src/model.cpp
Reverted 'model.cpp'
Undo!
Saving file g:/projects/polymorph/working3/src/model.cpp...
Wrote g:/projects/polymorph/working3/src/model.cpp
Reverted 'model.cpp'
Undo!
Entering debugger...

Configured using:
 'configure --prefix /C/emacs/emacs-20160725-215227 --with-modules
 --without-imagemagick --disable-dependency-tracking
 --enable-locallisppath=%emacs_dir%/../site-lisp CFLAGS=-O3
 CPPFLAGS=-D_WIN32_WINNT=_WIN32_WINNT_WIN7'

Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND DBUS NOTIFY ACL GNUTLS LIBXML2 ZLIB
TOOLKIT_SCROLL_BARS MODULES

Important settings:
  value of $LANG: ENG
  locale-coding-system: cp1252

Major mode: Debugger

Minor modes in effect:
  diff-auto-refine-mode: t
  shell-dirtrack-mode: t
  global-hi-lock-mode: t
  hi-lock-mode: t
  show-paren-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  line-number-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug sendmail debug cus-start cus-load
log-edit message subr-x puny format-spec rfc822 mml mml-sec epa derived
epg gnus-util rmail rmail-loaddefs mm-decode mm-bodies mm-encode
mail-parse rfc2231 rfc2047 rfc2045 mm-util ietf-drums mail-prsvr
mailabbrev mail-utils gmm-utils mailheader pcvs-util add-log smerge-mode
hippie-exp cc-mode cc-fonts cc-guess cc-menus cc-cmds cc-styles cc-align
cc-engine cc-vars cc-defs cl hl-line ffap vc-cvs vc-rcs dired
dired-loaddefs view diff-mode misearch multi-isearch jka-compr shell
pcomplete vc-svn perl-mode ediff-merg ediff-wind ediff-diff ediff-mult
ediff-help ediff-init ediff-util ediff hi-lock grep compile comint
ansi-color ring paren server pascal opascal finder-inf tex-site info
package epg-config url-handlers url-parse auth-source cl-seq eieio
eieio-core cl-macs eieio-loaddefs password-cache url-vars seq byte-opt
gv bytecomp byte-compile cl-extra help-mode easymenu cconv cl-loaddefs
pcase cl-lib advice time-date mule-util tooltip eldoc electric uniquify
ediff-hook vc-hooks lisp-float-type mwheel dos-w32 ls-lisp disp-table
term/w32-win w32-win w32-vars term/common-win tool-bar dnd fontset image
regexp-opt fringe tabulated-list newcomment elisp-mode lisp-mode
prog-mode register page menu-bar rfn-eshadow timer select scroll-bar
mouse jit-lock font-lock syntax facemenu font-core term/tty-colors frame
cl-generic cham georgian utf-8-lang misc-lang vietnamese tibetan thai
tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek romanian
slovak czech european ethiopic indian cyrillic chinese charscript
case-table epa-hook jka-cmpr-hook help simple abbrev obarray minibuffer
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote w32notify dbusbind w32
multi-tty make-network-process emacs)

Memory information:
((conses 16 309928 44864)
 (symbols 56 34963 0)
 (miscs 48 368 1555)
 (strings 32 62494 6202)
 (string-bytes 1 2004295)
 (vectors 16 28361)
 (vector-slots 8 650573 30264)
 (floats 8 270 254)
 (intervals 56 12859 249)
 (buffers 976 83))

bug#24094: Acknowledgement (25.1.50; revert-buffer error in CC mode)

[Richard Copley]

forcemerge 24094 24074
thanks

Apologies for the noise. This is the same as bug #24074.

bug#24094: 25.1.50; revert-buffer error in CC mode

[Alan Mackenzie]

Hello, Richard.

In article <mailman.2189.1469713866.26859.bug-gnu-emacs@gnu.org> you wrote:
> When editing C++ files, if I change visited files outside emacs (for
> example, by doing "svn revert -R ."), then visit one of the changed
> files and accept the offer to revert the buffer, in some cases there
> is an error (see below) and the buffer contents are corrupted (chunks
> are missing because the revert operation was interrupted).

This looks like the same bug as bug #24074, but you've managed to capture
a backtrace, for which many thanks.

Could you be a bit more descriptive about the "chunks" that are missing,
please?  Are we talking about lots of isolated 2-character chunks, or
just one or two larger chunks, or what?  Are the chunks at the end of a
buffer, or in the "middle" of it?

> I haven't been able to reduce this to a recipe and I don't know if
> the issue is present in the emacs-25 branch and/or in "emacs -Q".

Almost certainly, the bug isn't in the emacs-25 branch, because the
function c-before-after-change-digit-quote isn't in that branch.

> Here is an example backtrace (control characters replaced):

> Debugger entered--Lisp error: (error "Invalid search bound (wrong side
> of point)")
>   re-search-forward("[0-9a-fA-F]'[0-9a-fA-F]" 175 t)
>   c-before-after-change-digit-quote(65 65 1625)
>   #[(fn) "^H    \n^K#\207" [fn beg end old-len]
> 4](c-before-after-change-digit-quote)
>   mapc(#[(fn) "^H    \n^K#\207" [fn beg end old-len] 4]
> (c-depropertize-new-text c-extend-font-lock-region-for-macros
> c-before-after-change-digit-quote c-after-change-re-mark-raw-strings
> c-neutralize-syntax-in-and-mark-CPP c-restore-<>-properties
> c-change-expand-fl-region))
>   c-after-change(65 65 1625)
>   insert-file-contents("g:/projects/polymorph/working3/src/settings.cpp"
> t nil nil t)
>   revert-buffer-insert-file-contents--default-function("g:/projects/polymorph/working3/src/settings.cpp"
> nil)
>   revert-buffer--default(t t)
>   revert-buffer(t t)
>   find-file-noselect("g:/projects/polymorph/working3/src/settings.cpp")
>   compilation-find-file(#<marker at 1397 in *grep*> "settings.cpp" nil)
>   apply(compilation-find-file #<marker at 1397 in *grep*>
> "settings.cpp" nil nil)
>   compilation-next-error-function(1 nil)
>   next-error(nil)
>   funcall-interactively(next-error nil)
>   call-interactively(next-error nil nil)
>   command-execute(next-error)


> In GNU Emacs 25.1.50.1 (x86_64-w64-mingw32)
>  of 2016-07-25 built on MACHINE
> Repository revision: 6dc6b0079ed3632ed9082bc79d8cb6fc96d33f43
> Windowing system distributor 'Microsoft Corp.', version 10.0.10586
> Recent messages:
> Undo!
> Saving file g:/projects/polymorph/working3/src/model.cpp...
> Wrote g:/projects/polymorph/working3/src/model.cpp
> Reverted 'model.cpp'
> Undo!
> Saving file g:/projects/polymorph/working3/src/model.cpp...
> Wrote g:/projects/polymorph/working3/src/model.cpp
> Reverted 'model.cpp'
> Undo!
> Entering debugger...

[ .... ]

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#24094: 25.1.50; revert-buffer error in CC mode

[Richard Copley]

On 29 July 2016 at 18:59, Alan Mackenzie <acm@muc.de> wrote:
> Hello, Richard.

Hi!

> In article <mailman.2189.1469713866.26859.bug-gnu-emacs@gnu.org> you wrote:
>> When editing C++ files, if I change visited files outside emacs (for
>> example, by doing "svn revert -R ."), then visit one of the changed
>> files and accept the offer to revert the buffer, in some cases there
>> is an error (see below) and the buffer contents are corrupted (chunks
>> are missing because the revert operation was interrupted).
>
> This looks like the same bug as bug #24074, but you've managed to capture
> a backtrace, for which many thanks.

Yes, I think it is likely the same bug (if I'd noticed #24074 sooner,
I would have sent the backtrace there). Happy to help.

> Could you be a bit more descriptive about the "chunks" that are missing,
> please?  Are we talking about lots of isolated 2-character chunks, or
> just one or two larger chunks, or what?  Are the chunks at the end of a
> buffer, or in the "middle" of it?

It's hard to describe precisely (especially as I don't have a
corrupted buffer here and now), but being guided by your question,
we're talking about one or two larger chunks and not at the end of the
buffer but in the "middle".

My impression FWIW is that it is *as if* Emacs has done
"diff-buffer-with-file" and is attempting to apply the resulting patch
to the buffer (perhaps with the laudable intention of saving space in
the undo buffer), and has failed after a deletion and before an
insertion. But that is uninformed speculation.

>> I haven't been able to reduce this to a recipe and I don't know if
>> the issue is present in the emacs-25 branch and/or in "emacs -Q".
>
> Almost certainly, the bug isn't in the emacs-25 branch, because the
> function c-before-after-change-digit-quote isn't in that branch.

That's useful to know.
Thanks Alan.

bug#24074: bug#24094: 25.1.50; revert-buffer error in CC mode

[Óscar Fuentes]

Alan Mackenzie <acm@muc.de> writes:

> Could you be a bit more descriptive about the "chunks" that are missing,
> please?  Are we talking about lots of isolated 2-character chunks, or
> just one or two larger chunks, or what?  Are the chunks at the end of a
> buffer, or in the "middle" of it?

It just happened again here. The missing chunk is everything below the
first 9 lines (the file has ~400 lines). Those preserved lines are
simply #include's. The final preserved line was truncated to

#include <

The original was

#include <string.h>

Prior the revert, the point was much below that 9nth line.

The reported failure is not always the same. In this case was:

c-determine-+ve-limit: Args out of range: #<buffer rawmem.cpp>, -7246, -6746


Or course, now that I'm trying to cause the error for obtaining an stack
trace, it doesn't happen :-( As mentioned on my bug report, it seems
that the problem is triggered when the point falls on certain places
on the reverted file's contents, but that's just my guess.

bug#24074: bug#24094: 25.1.50; revert-buffer error in CC mode

[Richard Copley]

On 29 July 2016 at 19:29, Óscar Fuentes <ofv@wanadoo.es> wrote:
> Alan Mackenzie <acm@muc.de> writes:
>
>> Could you be a bit more descriptive about the "chunks" that are missing,
>> please?  Are we talking about lots of isolated 2-character chunks, or
>> just one or two larger chunks, or what?  Are the chunks at the end of a
>> buffer, or in the "middle" of it?
>
> It just happened again here. The missing chunk is everything below the
> first 9 lines (the file has ~400 lines). Those preserved lines are
> simply #include's. The final preserved line was truncated to
>
> #include <
>
> The original was
>
> #include <string.h>
>
> Prior the revert, the point was much below that 9nth line.
>
> The reported failure is not always the same. In this case was:
>
> c-determine-+ve-limit: Args out of range: #<buffer rawmem.cpp>, -7246, -6746
>
>
> Or course, now that I'm trying to cause the error for obtaining an stack
> trace, it doesn't happen :-( As mentioned on my bug report, it seems
> that the problem is triggered when the point falls on certain places
> on the reverted file's contents, but that's just my guess.

Here is a recipe.

Prepare a file "test0.cpp" as follows: (<<END)
int main () {
  int a = 0;
  int b = 1;
  int c = 2;
  int d = 3;
}
END

In a shell: cp test0.cpp test.cpp
In Emacs: visit test.cpp, transpose "line b" and "line c", save the
buffer, and put point between the transposed lines (i.e., at the
beginning of "line b").
In the shell: cp test0.cpp test.cpp
In Emacs: revisit test.cpp (C-x f M-n RET).

I hope that helps.

bug#24074: bug#24094: 25.1.50; revert-buffer error in CC mode

[Óscar Fuentes]

Richard Copley <rcopley@gmail.com> writes:

> It's hard to describe precisely (especially as I don't have a
> corrupted buffer here and now), but being guided by your question,
> we're talking about one or two larger chunks and not at the end of the
> buffer but in the "middle".

Yes, sometimes the missing chunk(s) are in the middle of the file. AFAIR
they are always quite large. I can't confirm the case of more than one
chunk, though.

Appreciating the damage is complicated by the fact that it not evident
what's missing at first sight: in the case I describe on my previous
email, when the error happened and swtiched to the affected buffer it
only displayed one line (the first one); then, pressed cursor-down and
the other eight lines appeared from nowhere.

> My impression FWIW is that it is *as if* Emacs has done
> "diff-buffer-with-file" and is attempting to apply the resulting patch
> to the buffer (perhaps with the laudable intention of saving space in
> the undo buffer), and has failed after a deletion and before an
> insertion. But that is uninformed speculation.

I suspected some undo-related problem when I experienced the problem
some weeks ago. Then, disabled undo before the revert on my Magit code:

-            (revert-buffer t t)
+            (progn
+              (setq buffer-undo-list t)
+              (revert-buffer t t)

to no avail. It looks more like a cache issue to me now: c-mode is using
some stale info computed before the revert.

bug#24094: 25.1.50; revert-buffer error in CC mode

[Óscar Fuentes]

Richard Copley <rcopley@gmail.com> writes:

> Here is a recipe.
>
> Prepare a file "test0.cpp" as follows: (<<END)
> int main () {
>   int a = 0;
>   int b = 1;
>   int c = 2;
>   int d = 3;
> }
> END
>
> In a shell: cp test0.cpp test.cpp
> In Emacs: visit test.cpp, transpose "line b" and "line c", save the
> buffer, and put point between the transposed lines (i.e., at the
> beginning of "line b").
> In the shell: cp test0.cpp test.cpp
> In Emacs: revisit test.cpp (C-x f M-n RET).

Great! I can reproduce here with `emacs -Q'. It is important to note
that the `transpose' step above means `M-x transpose-lines' (or its
corresponding shorcut). Transposing with C-k <down> C-y invalidates the
recipe.

bug#24074: bug#24094: 25.1.50; revert-buffer error in CC mode

[Alan Mackenzie]

Hello, Richard and Óscar.

I've cracked it!

On Fri, Jul 29, 2016 at 07:41:39PM +0100, Richard Copley wrote:

> Here is a recipe.

> Prepare a file "test0.cpp" as follows: (<<END)
> int main () {
>   int a = 0;
>   int b = 1;
>   int c = 2;
>   int d = 3;
> }
> END

> In a shell: cp test0.cpp test.cpp
> In Emacs: visit test.cpp, transpose "line b" and "line c", save the
> buffer, and put point between the transposed lines (i.e., at the
> beginning of "line b").
> In the shell: cp test0.cpp test.cpp
> In Emacs: revisit test.cpp (C-x C-f M-n RET).

> I hope that helps.

Very much indeed.  What is happening in that sequence is that the C-x C-f
calls the hook after-change-functions without having first called
before-change-functions.  This screws up CC Mode.

The function doing this, insert-file-contents, is called as follows:
(insert-file-contents "test.cpp" t nil nil t)
                                 | |   |   |
                                 | |   |   replace
                                 | |   end
                                 | beg
                                 visit

The section of Finsert_file_contents which calls before-change-functions
(through prepare_to_modify_buffer) looks like this:

  if (NILP (visit) && total > 0)
    {
      if (!NILP (BVAR (current_buffer, file_truename))
          /* Make binding buffer-file-name to nil effective.  */
          && !NILP (BVAR (current_buffer, filename))
          && SAVE_MODIFF >= MODIFF)
        we_locked_file = true;
      prepare_to_modify_buffer (PT, PT, NULL);  <======================
    }

The brace block will not be executed since `visit' is t.

The section of code which calls after-change-functions looks like this:

  if (inserted > 0 && total > 0
      && (NILP (visit) || !NILP (replace)))
    {
      signal_after_change (PT, 0, inserted);     <=====================
      update_compositions (PT, PT, CHECK_BORDER);
    }

The brace block here _will_ get called, since `replace' is non-nil.
There are thus two different, conflicting, conditions governing whether
to call the change hooks.  At a guess, the `if' condition around the
after-change-functions call was modified at some stage, without the same
change being made to the condition around the before-change-functions
call.

I'll look into this further over the weekend.

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#24094: 25.1.50; revert-buffer error in CC mode

[Richard Copley]

On 29 July 2016 at 22:18, Alan Mackenzie <acm@muc.de> wrote:
> Hello, Richard and Óscar.
>
> I've cracked it!

In record time! Great stuff, and thanks for the explanation.

bug#24074: bug#24094: 25.1.50; revert-buffer error in CC mode

[Óscar Fuentes]

Alan Mackenzie <acm@muc.de> writes:

> Hello, Richard and Óscar.
>
> I've cracked it!

[snip]

As expected :-)

> I'll look into this further over the weekend.

Thank you Alan.

bug#24094: 25.1.50; revert-buffer error in CC mode

[Alan Mackenzie]

Bug fixed in savannah master branch.

-- 
Alan Mackenzie (Nuremberg, Germany).