bug#16433: 24.3.50; find_newline screws up in Rmail buffers

bug#16433: 24.3.50; find_newline screws up in Rmail buffers

[Richard Stallman]

mail-fetch-field gave the wrong result for the X-RMAIL-ATTRIBUTES
field, perhaps because it happened to be last.  It included the
newline after that field in the return value, which is wrong.
Setting cache-long-scans to nil in the Rmail buffer made the
bug go away, so I think the problem is in the newline cache.

I think it would be better to fix the underlying bug, rather than turn
off the cache in Rmail mode.  I suspect that the cache does not handle
Rmail's changes in narrowing.  However, I'm not prepared to debug it
all the way myself.




In GNU Emacs 24.3.50.18 (mips64el-unknown-linux-gnu, GTK+ Version 2.20.1)
 of 2013-12-14 on chiefs-gnewsense
Bzr revision: 115526 tzz@lifelogs.com-20131214180409-n2o5017gxsjvz8as
System Description:	gNewSense GNU/Linux 3.0 (parkes)

Configured using:
 `configure 'CFLAGS=-g -O0''

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix
  default enable-multibyte-characters: t

Major mode: Change Log

Minor modes in effect:
  bug-reference-mode: t
  diff-auto-refine-mode: t
  shell-dirtrack-mode: t
  gpm-mouse-mode: t
  tooltip-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t
  abbrev-mode: t

Recent input:
i n g SPC g ESC DEL ESC DEL ESC DEL m a k e SPC c ESC 
DEL ESC DEL e l i m i n a t e SPC t h e SPC p r o b 
l e m SPC o f SPC h o w SPC t o SPC o r g a n i z e 
RET s c DEL o c i e t y . C-o SPC SPC A d v a n c e 
d SPC t e c h n o l o g y SPC m i g h t SPC ESC DEL 
c a n SPC b e SPC g o o d ; SPC A I SPC m i g h t SPC 
b e SPC g o o d RET d e p e n d n t DEL DEL i n g SPC 
o n SPC h o w SPC i t SPC ESC DEL ESC DEL ESC DEL d 
e DEL DEL o n SPC d e t a i l s . SPC SPC B u t SPC 
t h e s e SPC d o n ' t SPC e l i m i n a t e SPC t 
h a t SPC p r o b l e m . ESC a ESC f ESC d ESC d SPC 
t h a t ESC d ESC d C-e DEL SPC r e m a i n s . C-c 
C-c C-d x C-x b RET C-x b C h TAB RET DEL RET C-x k 
RET C-x b C h TAB s TAB RET C-a C-p C-p C-@ C-u C-n 
ESC w ESC x r e p o r t SPC e m a c s SPC b u g RE
T

Recent messages:
Auto save file for draft message exists; consider M-x mail-recover
Auto-saving...done
Auto-saving...done
Auto-saving...done
Sending...
Wrote /home/rms/outgoing/out-30
Sending...done
Expunging deleted messages...done
Mark set
Source file `/home/rms/emacs-bzr/trunk/lisp/mail/emacsbug.el' newer than byte-compiled file

Load-path shadows:
/home/rms/emacs-bzr/trunk/lisp/leim/quail/lao hides /home/rms/emacs-bzr/trunk/lisp/language/lao
/home/rms/emacs-bzr/trunk/lisp/leim/quail/georgian hides /home/rms/emacs-bzr/trunk/lisp/language/georgian
/home/rms/emacs-bzr/trunk/lisp/leim/quail/thai hides /home/rms/emacs-bzr/trunk/lisp/language/thai
/home/rms/emacs-bzr/trunk/lisp/leim/quail/ethiopic hides /home/rms/emacs-bzr/trunk/lisp/language/ethiopic
/home/rms/emacs-bzr/trunk/lisp/leim/quail/japanese hides /home/rms/emacs-bzr/trunk/lisp/language/japanese
/home/rms/emacs-bzr/trunk/lisp/leim/quail/cyrillic hides /home/rms/emacs-bzr/trunk/lisp/language/cyrillic
/home/rms/emacs-bzr/trunk/lisp/leim/quail/indian hides /home/rms/emacs-bzr/trunk/lisp/language/indian
/home/rms/emacs-bzr/trunk/lisp/leim/quail/hebrew hides /home/rms/emacs-bzr/trunk/lisp/language/hebrew
/home/rms/emacs-bzr/trunk/lisp/leim/quail/greek hides /home/rms/emacs-bzr/trunk/lisp/language/greek
/home/rms/emacs-bzr/trunk/lisp/leim/quail/czech hides /home/rms/emacs-bzr/trunk/lisp/language/czech
/home/rms/emacs-bzr/trunk/lisp/leim/quail/slovak hides /home/rms/emacs-bzr/trunk/lisp/language/slovak
/home/rms/emacs-bzr/trunk/lisp/leim/quail/tibetan hides /home/rms/emacs-bzr/trunk/lisp/language/tibetan

Features:
(shadow emacsbug bug-reference cc-langs cl cc-mode cc-fonts cc-guess
cc-menus cc-cmds cc-styles cc-align cc-engine cc-vars cc-defs
smerge-mode whitespace log-view find-func iso-transl mule-util
cal-move cal-menu calendar cal-loaddefs nroff-mode goto-addr view
disp-table enriched conf-mode ffap thingatpt url-parse auth-source
eieio byte-opt bytecomp byte-compile cconv eieio-core gnus-util
password-cache url-vars noutline outline vc-bzr dcl-mode tempo tabify
imenu man jka-compr debug compare-w ispell epa-mail quail rmailsum
diff-mode log-edit easy-mmode pcvs-util add-log vc vc-dispatcher
dired-aux help-mode misearch multi-isearch wid-edit cl-loaddefs cl-lib
rmailout epa derived epg epg-config shell pcomplete grep compile
comint ansi-color dabbrev etags ring parse-time vc-cvs sgml-mode
mailalias qp rmailmm message sendmail format-spec rfc822 mml easymenu
mml-sec mm-decode mm-bodies mm-encode mailabbrev gmm-utils mailheader
mail-parse rfc2231 dired t-mouse package rmailedit rmail rfc2047
rfc2045 ietf-drums mm-util mail-prsvr mail-utils time-date paren
cus-start cus-load advice help-fns tooltip electric uniquify
ediff-hook vc-hooks lisp-float-type mwheel x-win x-dnd tool-bar dnd
fontset image regexp-opt fringe tabulated-list newcomment lisp-mode
prog-mode register page menu-bar rfn-eshadow timer select scroll-bar
mouse jit-lock font-lock syntax facemenu font-core frame cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese case-table epa-hook jka-cmpr-hook help simple abbrev
minibuffer nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote make-network-process
dbusbind gfilenotify dynamic-setting system-font-setting
font-render-setting move-toolbar gtk x-toolkit x multi-tty emacs)
[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]


-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

bug#16433: 24.3.50; find_newline screws up in Rmail buffers

[Eli Zaretskii]

> Date: Mon, 13 Jan 2014 14:37:11 -0500
> From: Richard Stallman <rms@gnu.org>
> 
> 
> mail-fetch-field gave the wrong result for the X-RMAIL-ATTRIBUTES
> field, perhaps because it happened to be last.  It included the
> newline after that field in the return value, which is wrong.
> Setting cache-long-scans to nil in the Rmail buffer made the
> bug go away, so I think the problem is in the newline cache.

I agree.

> I think it would be better to fix the underlying bug, rather than turn
> off the cache in Rmail mode.

Again, I agree.

> I suspect that the cache does not handle Rmail's changes in
> narrowing.

No, I don't think so.  I suspect it has something to do with
buffer-swap-text.

Is it possible to have a reproducible recipe for this problem?

bug#16433: 24.3.50; find_newline screws up in Rmail buffers

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    Is it possible to have a reproducible recipe for this problem?

Sorry, no, I don't have a recipe.  It was reproducible for the same
message, at that time, but I don't know a way to make it fail again.

Perhaps buffer-swap-text should simply invalidate the cache.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

bug#16433: 24.3.50; find_newline screws up in Rmail buffers

[Eli Zaretskii]

> Date: Tue, 14 Jan 2014 12:06:01 -0500
> From: Richard Stallman <rms@gnu.org>
> CC: 16433@debbugs.gnu.org
> 
> Sorry, no, I don't have a recipe.  It was reproducible for the same
> message, at that time, but I don't know a way to make it fail again.

To reproduce it at that time, do you happen to remember what sequence
of commands was needed?

> Perhaps buffer-swap-text should simply invalidate the cache.

Could be, but just by looking at the code, it seems like
buffer-swap-text does TRT, i.e. it swaps also the pointers to the
cache.  So maybe my guess was wrong, and some other factor is at work
here.

If and when it happens again, please try to look for editing
operations that change buffer text (insert or delete characters), but
do not invalidate the cache for the region where characters were
inserted or deleted.  There should be a single call to
invalidate_region_cache for each insertion and each deletion.  This
includes insertion/deletion performed indirectly, like when decoding
the message text when it is displayed.

Thanks.

bug#16433: 24.3.50; find_newline screws up in Rmail buffers

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    > Sorry, no, I don't have a recipe.  It was reproducible for the same
    > message, at that time, but I don't know a way to make it fail again.

    To reproduce it at that time, do you happen to remember what sequence
    of commands was needed?

(mail-fetch-field "X-RMAIL-ATTRIBUTES") reproduced the bug, when it
was happening.

    If and when it happens again, please try to look for editing
    operations that change buffer text (insert or delete characters), but
    do not invalidate the cache for the region where characters were
    inserted or deleted.

I had not done any manual editing on the RMAIL buffer.
It was operated on by Rmail commands.  If you see what editing
functions they call, you will see what I did to the RMAIL buffer.

It's possible I decrypted it with rmail-epa-decrypt.

    > Perhaps buffer-swap-text should simply invalidate the cache.

    Could be, but just by looking at the code, it seems like
    buffer-swap-text does TRT, i.e. it swaps also the pointers to the
    cache.  So maybe my guess was wrong, and some other factor is at work
    here.

If it isn't obvious what is wrong, we need to fix this bug somehow.  I
am thinking of turning off the newline cache in Rmail mode.
-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

bug#16433: 24.3.50; find_newline screws up in Rmail buffers

[Eli Zaretskii]

> Date: Wed, 15 Jan 2014 07:29:05 -0500
> From: Richard Stallman <rms@gnu.org>
> CC: 16433@debbugs.gnu.org
> 
>     To reproduce it at that time, do you happen to remember what sequence
>     of commands was needed?
> 
> (mail-fetch-field "X-RMAIL-ATTRIBUTES") reproduced the bug, when it
> was happening.

Thanks.

>     If and when it happens again, please try to look for editing
>     operations that change buffer text (insert or delete characters), but
>     do not invalidate the cache for the region where characters were
>     inserted or deleted.
> 
> I had not done any manual editing on the RMAIL buffer.
> It was operated on by Rmail commands.  If you see what editing
> functions they call, you will see what I did to the RMAIL buffer.

I meant editing operations that Rmail invokes behind the scenes, like
when it displays a message in rmail-view-buffer.

> It's possible I decrypted it with rmail-epa-decrypt.

Thanks, I'll take a look at that.

> If it isn't obvious what is wrong, we need to fix this bug somehow.  I
> am thinking of turning off the newline cache in Rmail mode.

That'd sweep the problem under the carpet, so I don't recommend that,
not yet anyway.

bug#16433: Test case for newline cache corruption

[Ken Olum]

[-- Attachment #1: Type: text/plain, Size: 823 bytes --]

Here is a procedure which I can use to reproduce the newline cache
corruption.

1.  Detach attached files test.mbox, test.in, and test.emacs.  

2.  Put test.emacs in place of your .emacs file.

3.  Run emacs.  I used a compilation from trunk revision 117139.

Because I had to try this many, many times, test.emacs puts testing
commands on keys F3 and F4.  So

3.  Push F3 to run rmail on test.mbox

4.  Push F4 to get new mail from test.in

5.  Push "p" to go to the previous message.  You should get a warning
about cache corruption.

The details of the "new mail" in test.in don't seem to matter much, but
it is very sensitive to precisely what is in the existing message in
test.mbox.  I removed everything that I could to simplify the test case.

I hope this is helpful.

                                        Ken


[-- Attachment #2: test.mbox --]
[-- Type: application/octet-stream, Size: 2042 bytes --]

From kdo@cosmos.phy.tufts.edu Wed May 21 15:19:31 2014
Envelope-to: kdo@cosmos.phy.tufts.edu
Delivery-date: Wed, 21 May 2014 15:19:31 -0400
From: kdo@cosmos.phy.tufts.edu
To: kdo@cosmos.phy.tufts.edu
Subject: test
Date: Wed, 21 May 2014 19:18:25 +0000
Message-ID: <A08616A381DDDA4F907CE430F65C67FE01C5118AA7@RMSMBX01.rms-law.com>
X-RMAIL-ATTRIBUTES: --------

<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Exchange Server">
<!-- converted from rtf -->
<style><!-- .EmailQuote { margin-left: 1pt; padding-left: 4pt; border-left: #800000 2px solid; } --></style>
</head>
<body>
<font face="Times New Roman" size="3"><span style="font-size:12pt;"><a name="OLE_LINK1"></a>
<div><font face="Calibri" size="2"><span style="font-size:11pt;">Send us your photographs and articles, reflections, reminiscences.&nbsp; Sail a multihull and write about it!&nbsp; </span></font></div>
<div><font face="Calibri" size="2"><span style="font-size:11pt;">--Andy Houlding</span></font></div>
<div><font face="Calibri" size="2"><span style="font-size:11pt;">&nbsp;</span></font></div>
<div><font face="Calibri" size="2"><span style="font-size:11pt;"> </span></font></div>
<div><font face="Calibri" size="2"><span style="font-size:11pt;">&nbsp;</span></font></div>
<div><font face="Calibri" size="2"><span style="font-size:11pt;">&nbsp;</span></font></div>
</span></font>
</body>
</html>

From kdo@cosmos.phy.tufts.edu Thu May 22 10:43:50 2014
Return-path: <kdo@cosmos.phy.tufts.edu>
Envelope-to: kdo@cosmos.phy.tufts.edu
Delivery-date: Thu, 22 May 2014 10:43:50 -0400
Received: from kdo by cosmos.phy.tufts.edu ([local]:local)
	with local id 1WnUDm-0007g8-1A - Using Exim-4.80.1 (MandrivaLinux) MTA 
	(return-path <kdo@cosmos.phy.tufts.edu>); Thu, 22 May 2014 10:43:50 -0400
Date: Thu, 22 May 2014 10:43:50 -0400
To: kdo@cosmos.phy.tufts.edu
Subject: test
Message-Id: <E1WnUDm-0007g8-1A@cosmos.phy.tufts.edu>
From: Ken Olum <kdo@cosmos.phy.tufts.edu>
X-RMAIL-ATTRIBUTES: ------U-



[-- Attachment #3: test.in --]
[-- Type: application/octet-stream, Size: 565 bytes --]

From kdo@cosmos.phy.tufts.edu Thu May 22 10:43:50 2014
Return-path: <kdo@cosmos.phy.tufts.edu>
Envelope-to: kdo@cosmos.phy.tufts.edu
Delivery-date: Thu, 22 May 2014 10:43:50 -0400
Received: from kdo by cosmos.phy.tufts.edu ([local]:local)
	with local id 1WnUDm-0007g8-1A - Using Exim-4.80.1 (MandrivaLinux) MTA 
	(return-path <kdo@cosmos.phy.tufts.edu>); Thu, 22 May 2014 10:43:50 -0400
Date: Thu, 22 May 2014 10:43:50 -0400
To: kdo@cosmos.phy.tufts.edu
Subject: test
Message-Id: <E1WnUDm-0007g8-1A@cosmos.phy.tufts.edu>
From: Ken Olum <kdo@cosmos.phy.tufts.edu>



[-- Attachment #4: test.emacs --]
[-- Type: application/octet-stream, Size: 693 bytes --]

(require 'rmail)

(defun maybe-check-newline-cache ()
  (if (eq major-mode 'rmail-mode)
      (progn
	(check-newline-cache)
	(if rmail-view-buffer
	    (with-current-buffer rmail-view-buffer
	      (check-newline-cache))))))

(defun check-newline-cache ()
  (let ((newlines (newline-cache-check)))
    (if (and newlines
	     (not (equal (aref newlines 0) (aref newlines 1))))
	(message "Newline cache corrupted in %s" (current-buffer)))))

(add-hook 'post-command-hook 'maybe-check-newline-cache)

(defun do-test ()
  (interactive)
  (rmail "test.mbox"))

(global-set-key [f3] 'do-test)

(defun do-test-1 ()
  (interactive)
  (rmail-get-new-mail "test.in"))

(global-set-key [f4] 'do-test-1)

bug#16433: Test case for newline cache corruption

[Eli Zaretskii]

> From: Ken Olum <kdo@cosmos.phy.tufts.edu>
> Date: Thu, 22 May 2014 16:38:27 -0400
> 
> Here is a procedure which I can use to reproduce the newline cache
> corruption.
> 
> 1.  Detach attached files test.mbox, test.in, and test.emacs.  
> 
> 2.  Put test.emacs in place of your .emacs file.
> 
> 3.  Run emacs.  I used a compilation from trunk revision 117139.
> 
> Because I had to try this many, many times, test.emacs puts testing
> commands on keys F3 and F4.  So
> 
> 3.  Push F3 to run rmail on test.mbox
> 
> 4.  Push F4 to get new mail from test.in
> 
> 5.  Push "p" to go to the previous message.  You should get a warning
> about cache corruption.
> 
> The details of the "new mail" in test.in don't seem to matter much, but
> it is very sensitive to precisely what is in the existing message in
> test.mbox.  I removed everything that I could to simplify the test case.
> 
> I hope this is helpful.

Thanks a lot, I will look into this as soon as I have time.

bug#16433: Test case for newline cache corruption

[Eli Zaretskii]

> From: Ken Olum <kdo@cosmos.phy.tufts.edu>
> Date: Thu, 22 May 2014 16:38:27 -0400
> 
> 1.  Detach attached files test.mbox, test.in, and test.emacs.  
> 
> 2.  Put test.emacs in place of your .emacs file.
> 
> 3.  Run emacs.  I used a compilation from trunk revision 117139.
> 
> Because I had to try this many, many times, test.emacs puts testing
> commands on keys F3 and F4.  So
> 
> 3.  Push F3 to run rmail on test.mbox
> 
> 4.  Push F4 to get new mail from test.in
> 
> 5.  Push "p" to go to the previous message.  You should get a warning
> about cache corruption.

I'm sorry, but I cannot reproduce the problem with this recipe.  I
tried both the current trunk and the emacs-24 branch, and neither
produced the message about cache corruption.

What are your locale settings?  This could somehow conceal the bug,
although I doubt that, since the mbox file you sent doesn't include
any non-ASCII text in any shape.

Also, is your build optimized or not?  More generally, what are the
values of system-configuration and system-configuration-options in the
build where you reproduce this?  If your build options are different
from mine, especially the optimization switches, perhaps that is part
of the issue somehow.

If the locale stuff and build options don't help, I think the only way
to make progress is to ask you debug this on your system with guidance
from me and others here.  I have written some debugging code that
could help in this matter, which I can post here.  How proficient are
you with using GDB on the C level?

Thanks.

bug#16433: Test case for newline cache corruption

[Ken Olum]

   From: Eli Zaretskii <eliz@gnu.org>
   Date: Sat, 24 May 2014 11:24:02 +0300

   What are your locale settings?

LC_ALL=C

   Also, is your build optimized or not?  More generally, what are the
   values of system-configuration and system-configuration-options in the
   build where you reproduce this?  If your build options are different
   from mine, especially the optimization switches, perhaps that is part
   of the issue somehow.

All I did was to download the trunk distribution and say "make".

system-configuration is "x86_64-unknown-linux-gnu"
system-configuration-options is ""

I'm running Ubuntu 12.04 LTS.

To avoid any possible dependence on window system issues, I tried
"emacs -nw" and got the same failure.

I'm happy to try to debug the C code if needed.  I can use GDB.

                                        Ken

bug#16433: Test case for newline cache corruption

[Eli Zaretskii]

> From: Ken Olum <kdo@cosmos.phy.tufts.edu>
> Cc: 16433@debbugs.gnu.org
> Date: Mon, 26 May 2014 11:29:22 -0400
> 
>    Also, is your build optimized or not?  More generally, what are the
>    values of system-configuration and system-configuration-options in the
>    build where you reproduce this?  If your build options are different
>    from mine, especially the optimization switches, perhaps that is part
>    of the issue somehow.
> 
> All I did was to download the trunk distribution and say "make".

This means the build is optimized.  This might get in the way when we
will need to use GDB, but for now it's OK.

Below please find a simple patch that adds instrumentation to the
cache operations.  The trace goes to the *Messages* buffer.  Please
apply the patch, then run your recipe, and show the trace for it.  (It
is best to mark the place in *Messages* where you actually start and
end the recipe with some prominent string, because the newline cache
is consulted _a_lot_.)

Thanks.

=== modified file 'src/region-cache.c'
--- a/src/region-cache.c	2014-05-03 08:24:51 +0000
+++ b/src/region-cache.c	2014-05-26 19:26:53 +0000
@@ -459,6 +459,15 @@
 invalidate_region_cache (struct buffer *buf, struct region_cache *c,
 			 ptrdiff_t head, ptrdiff_t tail)
 {
+  char buff[1024];
+
+  if (c == buf->newline_cache)
+    {
+      sprintf (buff, "invalidate_region_cache (%s, %d, %d, %d)",
+	       SSDATA (BVAR (buf, name)),
+	       BUF_BEG (buf) + head, BUF_Z (buf) - tail, BUF_Z (buf));
+      message_dolog (buff, strlen (buff), false, true);
+    }
   /* Let chead = c->beg_unchanged, and
          ctail = c->end_unchanged.
      If z-tail < beg+chead by a large amount, or
@@ -474,8 +483,14 @@
        > PRESERVE_THRESHOLD)
       || ((BUF_BEG (buf) + head) - (BUF_Z (buf) - c->end_unchanged)
           > PRESERVE_THRESHOLD))
-    revalidate_region_cache (buf, c);
-
+    {
+      revalidate_region_cache (buf, c);
+      if (c == buf->newline_cache)
+	message_dolog (" (revalidated)\n", sizeof (" (revalidated)\n") - 1,
+		       false, true);
+    }
+  else if (c == buf->newline_cache)
+    message_dolog ("\n", 1, false, true);
 
   if (head < c->beg_unchanged)
     c->beg_unchanged = head;
@@ -666,6 +681,15 @@
 know_region_cache (struct buffer *buf, struct region_cache *c,
 		   ptrdiff_t start, ptrdiff_t end)
 {
+  char buff[1024];
+
+  if (c == buf->newline_cache)
+    {
+      sprintf (buff, "know_region_cache (%s, %d, %d, %d)\n",
+	       SSDATA (BVAR (buf, name)), start, end, BUF_Z (buf));
+      message_dolog (buff, strlen (buff), false, true);
+    }
+
   revalidate_region_cache (buf, c);
 
   set_cache_region (c, start, end, 1);

bug#16433: Test case for newline cache corruption

[Ken Olum]

Here's the log of your debugging output and other messages for my test
case.

                                        Ken
----------------------------------------------------------------------
Counting messages...
know_region_cache (test.mbox, 351, 359, 1449)

Counting messages...done
know_region_cache ( *message-viewer test.mbox*, 7, 31, 1202)
know_region_cache ( *message-viewer test.mbox*, 70, 74, 1202)
know_region_cache (test.mbox, 1, 7, 1202)
know_region_cache (test.mbox, 32, 60, 1202)
know_region_cache (test.mbox, 61, 70, 1202)
know_region_cache (test.mbox, 75, 112, 1202)
know_region_cache (test.mbox, 114, 120, 1202)
know_region_cache (test.mbox, 121, 127, 1202)
know_region_cache (test.mbox, 128, 198, 1202)
know_region_cache (test.mbox, 199, 258, 1202)
know_region_cache (test.mbox, 259, 286, 1202)
know_region_cache (test.mbox, 287, 395, 1202)
know_region_cache (test.mbox, 396, 403, 1202)
know_region_cache (test.mbox, 404, 410, 1202)
know_region_cache (test.mbox, 411, 503, 1202)
know_region_cache (test.mbox, 504, 705, 1202)
know_region_cache (test.mbox, 706, 806, 1202)
know_region_cache (test.mbox, 807, 898, 1202)
know_region_cache (test.mbox, 899, 985, 1202)
know_region_cache (test.mbox, 986, 1077, 1202)
know_region_cache (test.mbox, 1078, 1169, 1202)
know_region_cache (test.mbox, 1170, 1184, 1202)
know_region_cache (test.mbox, 1185, 1192, 1202)
know_region_cache (test.mbox, 1193, 1200, 1202)
know_region_cache ( *message-viewer test.mbox*, 1, 55, 1449)
know_region_cache ( *message-viewer test.mbox*, 56, 93, 1449)
know_region_cache ( *message-viewer test.mbox*, 94, 140, 1449)
know_region_cache ( *message-viewer test.mbox*, 141, 171, 1449)
know_region_cache ( *message-viewer test.mbox*, 172, 200, 1449)
know_region_cache ( *message-viewer test.mbox*, 201, 214, 1449)
know_region_cache ( *message-viewer test.mbox*, 215, 252, 1449)
know_region_cache ( *message-viewer test.mbox*, 253, 330, 1449)
know_region_cache ( *message-viewer test.mbox*, 331, 351, 1449)
know_region_cache ( *message-viewer test.mbox*, 361, 367, 1449)
know_region_cache ( *message-viewer test.mbox*, 368, 374, 1449)
know_region_cache ( *message-viewer test.mbox*, 375, 445, 1449)
know_region_cache ( *message-viewer test.mbox*, 446, 505, 1449)
know_region_cache ( *message-viewer test.mbox*, 506, 533, 1449)
know_region_cache ( *message-viewer test.mbox*, 534, 642, 1449)
know_region_cache ( *message-viewer test.mbox*, 643, 650, 1449)
know_region_cache ( *message-viewer test.mbox*, 651, 657, 1449)
know_region_cache ( *message-viewer test.mbox*, 658, 750, 1449)
know_region_cache ( *message-viewer test.mbox*, 751, 952, 1449)
know_region_cache ( *message-viewer test.mbox*, 953, 1053, 1449)
know_region_cache ( *message-viewer test.mbox*, 1054, 1145, 1449)
know_region_cache ( *message-viewer test.mbox*, 1146, 1232, 1449)
know_region_cache ( *message-viewer test.mbox*, 1233, 1324, 1449)
know_region_cache ( *message-viewer test.mbox*, 1325, 1416, 1449)
know_region_cache ( *message-viewer test.mbox*, 1417, 1431, 1449)
know_region_cache ( *message-viewer test.mbox*, 1432, 1439, 1449)
know_region_cache ( *message-viewer test.mbox*, 1440, 1447, 1449)

Getting mail from /old/home/kdo/test.in...
invalidate_region_cache (test.mbox, 361, 361, 1449) (revalidated)
invalidate_region_cache (test.mbox, 1449, 2014, 2014) (revalidated)

invalidate_region_cache (test.mbox, 2012, 2012, 2014)
invalidate_region_cache (test.mbox, 2030, 2030, 2032) (revalidated)
know_region_cache (test.mbox, 2012, 2030, 2034)
invalidate_region_cache (test.mbox, 2032, 2032, 2034) (revalidated)
know_region_cache (test.mbox, 2030, 2032, 2042)
invalidate_region_cache (test.mbox, 2040, 2040, 2042) (revalidated)
know_region_cache (test.mbox, 2032, 2040, 2043)

Counting new messages...done (1)
Saving file /home/kdo/test.mbox...
Wrote /home/kdo/test.mbox [2 times]
1 new message read
invalidate_region_cache (test.mbox, 2038, 2039, 2043) (revalidated)
invalidate_region_cache (test.mbox, 2038, 2038, 2042) (revalidated)
know_region_cache (test.mbox, 2038, 2039, 2043)
invalidate_region_cache ( *message-viewer test.mbox*, 1, 1202, 1202)
invalidate_region_cache ( *message-viewer test.mbox*, 1, 1, 1)
invalidate_region_cache ( *message-viewer test.mbox*, 1, 1, 1)
invalidate_region_cache ( *message-viewer test.mbox*, 1, 1, 2)
invalidate_region_cache ( *message-viewer test.mbox*, 1, 1, 3)
invalidate_region_cache ( *message-viewer test.mbox*, 39, 39, 41)
invalidate_region_cache ( *message-viewer test.mbox*, 68, 68, 70)
invalidate_region_cache ( *message-viewer test.mbox*, 82, 82, 84)
know_region_cache ( *message-viewer test.mbox*, 77, 81, 126)
know_region_cache ( *message-viewer test.mbox*, 88, 123, 126)
invalidate_region_cache ( *message-viewer test.mbox*, 1, 126, 126)
invalidate_region_cache ( *message-viewer test.mbox*, 1, 1, 1)
invalidate_region_cache ( *message-viewer test.mbox*, 1, 1, 1)
invalidate_region_cache ( *message-viewer test.mbox*, 1, 1, 2)
invalidate_region_cache ( *message-viewer test.mbox*, 1, 1, 3)
invalidate_region_cache ( *message-viewer test.mbox*, 39, 39, 41)
invalidate_region_cache ( *message-viewer test.mbox*, 68, 68, 70)
invalidate_region_cache ( *message-viewer test.mbox*, 82, 82, 84)
know_region_cache ( *message-viewer test.mbox*, 77, 81, 126)
know_region_cache ( *message-viewer test.mbox*, 88, 123, 126)
know_region_cache (test.mbox, 1, 38, 126)
know_region_cache (test.mbox, 39, 67, 126)
know_region_cache (test.mbox, 68, 77, 126)
know_region_cache (test.mbox, 82, 88, 126)
know_region_cache ( *message-viewer test.mbox*, 1449, 1503, 2043)
know_region_cache ( *message-viewer test.mbox*, 1504, 1543, 2043)
know_region_cache ( *message-viewer test.mbox*, 1544, 1581, 2043)
know_region_cache ( *message-viewer test.mbox*, 1582, 1628, 2043)
know_region_cache ( *message-viewer test.mbox*, 1629, 1687, 2043)
know_region_cache ( *message-viewer test.mbox*, 1688, 1760, 2043)
know_region_cache ( *message-viewer test.mbox*, 1761, 1835, 2043)
know_region_cache ( *message-viewer test.mbox*, 1836, 1873, 2043)
know_region_cache ( *message-viewer test.mbox*, 1874, 1902, 2043)
know_region_cache ( *message-viewer test.mbox*, 1903, 1916, 2043)
know_region_cache ( *message-viewer test.mbox*, 1917, 1969, 2043)
know_region_cache ( *message-viewer test.mbox*, 1970, 2011, 2043)
invalidate_region_cache ( *message-viewer test.mbox*, 1, 126, 126)
invalidate_region_cache ( *message-viewer test.mbox*, 1, 1, 1)
invalidate_region_cache ( *message-viewer test.mbox*, 1, 1, 1)
invalidate_region_cache ( *message-viewer test.mbox*, 1, 1, 1089)
invalidate_region_cache ( *message-viewer test.mbox*, 1, 1, 1090)
invalidate_region_cache ( *message-viewer test.mbox*, 32, 32, 1121)
invalidate_region_cache ( *message-viewer test.mbox*, 61, 61, 1150)
invalidate_region_cache ( *message-viewer test.mbox*, 75, 75, 1164)
know_region_cache ( *message-viewer test.mbox*, 7, 31, 1202)
know_region_cache ( *message-viewer test.mbox*, 70, 74, 1202)
know_region_cache (test.mbox, 1, 7, 1202)
know_region_cache (test.mbox, 32, 60, 1202)
know_region_cache (test.mbox, 61, 70, 1202)
know_region_cache (test.mbox, 75, 112, 1202)
know_region_cache (test.mbox, 114, 120, 1202)
know_region_cache (test.mbox, 121, 127, 1202)
know_region_cache (test.mbox, 128, 198, 1202)
know_region_cache (test.mbox, 199, 258, 1202)
know_region_cache (test.mbox, 259, 286, 1202)
know_region_cache (test.mbox, 287, 395, 1202)
know_region_cache (test.mbox, 396, 403, 1202)
know_region_cache (test.mbox, 404, 410, 1202)
know_region_cache (test.mbox, 411, 503, 1202)
know_region_cache (test.mbox, 504, 705, 1202)
know_region_cache (test.mbox, 706, 806, 1202)
know_region_cache (test.mbox, 807, 898, 1202)
know_region_cache (test.mbox, 899, 985, 1202)
know_region_cache (test.mbox, 986, 1077, 1202)
know_region_cache (test.mbox, 1078, 1169, 1202)
know_region_cache (test.mbox, 1170, 1184, 1202)
know_region_cache (test.mbox, 1185, 1192, 1202)
know_region_cache (test.mbox, 1193, 1200, 1202)
know_region_cache ( *message-viewer test.mbox*, 361, 367, 2043)
know_region_cache ( *message-viewer test.mbox*, 368, 374, 2043)
know_region_cache ( *message-viewer test.mbox*, 375, 445, 2043)
know_region_cache ( *message-viewer test.mbox*, 446, 505, 2043)
know_region_cache ( *message-viewer test.mbox*, 506, 533, 2043)
know_region_cache ( *message-viewer test.mbox*, 534, 642, 2043)
know_region_cache ( *message-viewer test.mbox*, 643, 650, 2043)
know_region_cache ( *message-viewer test.mbox*, 651, 657, 2043)
know_region_cache ( *message-viewer test.mbox*, 658, 750, 2043)
know_region_cache ( *message-viewer test.mbox*, 751, 926, 2043)
know_region_cache ( *message-viewer test.mbox*, 932, 933, 2043)
know_region_cache ( *message-viewer test.mbox*, 939, 940, 2043)
know_region_cache ( *message-viewer test.mbox*, 1010, 1011, 2043)
know_region_cache ( *message-viewer test.mbox*, 1070, 1071, 2043)
know_region_cache ( *message-viewer test.mbox*, 1098, 1099, 2043)
know_region_cache ( *message-viewer test.mbox*, 1207, 1208, 2043)
know_region_cache ( *message-viewer test.mbox*, 1215, 1216, 2043)
know_region_cache ( *message-viewer test.mbox*, 1222, 1223, 2043)
know_region_cache ( *message-viewer test.mbox*, 1315, 1316, 2043)

Newline cache corrupted in  *message-viewer test.mbox*
invalidate_region_cache ( *Minibuf-1*, 1, 1, 1)
know_region_cache ( *Minibuf-1*, 1, 40, 40)
invalidate_region_cache ( *Minibuf-1*, 40, 40, 40)
invalidate_region_cache ( *Minibuf-1*, 41, 41, 41)
invalidate_region_cache ( *Minibuf-1*, 42, 42, 42)
invalidate_region_cache ( *Minibuf-1*, 43, 43, 43)
invalidate_region_cache ( *Minibuf-1*, 44, 44, 44)
invalidate_region_cache ( *Minibuf-1*, 1, 50, 50)
----------------------------------------------------------------------

bug#16433: Test case for newline cache corruption

[Ken Olum]

Here is some additional output which perhaps will be helpful.  I put
checks at the beginning and end of rmail-swap-buffers, and also
immediately before and after the call to buffer-swap-text.  I had these
checks print a message when the cache is OK, and print the character
positions on the cache is corrupted, giving the following.  Newlines are
absent from the cache starting with the one at position 952.

                                        Ken

----------------------------------------------------------------------
Counting messages...
know_region_cache (test.mbox, 351, 359, 1449)

Counting messages...done
know_region_cache ( *message-viewer test.mbox*, 7, 31, 1202)
know_region_cache ( *message-viewer test.mbox*, 70, 74, 1202)
know_region_cache (test.mbox, 1, 55, 1449)
know_region_cache (test.mbox, 56, 93, 1449)
know_region_cache (test.mbox, 94, 140, 1449)
know_region_cache (test.mbox, 141, 171, 1449)
know_region_cache (test.mbox, 172, 200, 1449)
know_region_cache (test.mbox, 201, 214, 1449)
know_region_cache (test.mbox, 215, 252, 1449)
know_region_cache (test.mbox, 253, 330, 1449)
know_region_cache (test.mbox, 331, 351, 1449)
know_region_cache (test.mbox, 361, 367, 1449)
know_region_cache (test.mbox, 368, 374, 1449)
know_region_cache (test.mbox, 375, 445, 1449)
know_region_cache (test.mbox, 446, 505, 1449)
know_region_cache (test.mbox, 506, 533, 1449)
know_region_cache (test.mbox, 534, 642, 1449)
know_region_cache (test.mbox, 643, 650, 1449)
know_region_cache (test.mbox, 651, 657, 1449)
know_region_cache (test.mbox, 658, 750, 1449)
know_region_cache (test.mbox, 751, 952, 1449)
know_region_cache (test.mbox, 953, 1053, 1449)
know_region_cache (test.mbox, 1054, 1145, 1449)
know_region_cache (test.mbox, 1146, 1232, 1449)
know_region_cache (test.mbox, 1233, 1324, 1449)
know_region_cache (test.mbox, 1325, 1416, 1449)
know_region_cache (test.mbox, 1417, 1431, 1449)
know_region_cache (test.mbox, 1432, 1439, 1449)
know_region_cache (test.mbox, 1440, 1447, 1449)

Cache OK in test.mbox at start of rmail-swap-buffers
know_region_cache ( *message-viewer test.mbox*, 1, 7, 1202)
know_region_cache ( *message-viewer test.mbox*, 32, 60, 1202)
know_region_cache ( *message-viewer test.mbox*, 61, 70, 1202)
know_region_cache ( *message-viewer test.mbox*, 75, 112, 1202)
know_region_cache ( *message-viewer test.mbox*, 114, 120, 1202)
know_region_cache ( *message-viewer test.mbox*, 121, 127, 1202)
know_region_cache ( *message-viewer test.mbox*, 128, 198, 1202)
know_region_cache ( *message-viewer test.mbox*, 199, 258, 1202)
know_region_cache ( *message-viewer test.mbox*, 259, 286, 1202)
know_region_cache ( *message-viewer test.mbox*, 287, 395, 1202)
know_region_cache ( *message-viewer test.mbox*, 396, 403, 1202)
know_region_cache ( *message-viewer test.mbox*, 404, 410, 1202)
know_region_cache ( *message-viewer test.mbox*, 411, 503, 1202)
know_region_cache ( *message-viewer test.mbox*, 504, 705, 1202)
know_region_cache ( *message-viewer test.mbox*, 706, 806, 1202)
know_region_cache ( *message-viewer test.mbox*, 807, 898, 1202)
know_region_cache ( *message-viewer test.mbox*, 899, 985, 1202)
know_region_cache ( *message-viewer test.mbox*, 986, 1077, 1202)
know_region_cache ( *message-viewer test.mbox*, 1078, 1169, 1202)
know_region_cache ( *message-viewer test.mbox*, 1170, 1184, 1202)
know_region_cache ( *message-viewer test.mbox*, 1185, 1192, 1202)
know_region_cache ( *message-viewer test.mbox*, 1193, 1200, 1202)

Cache OK in  *message-viewer test.mbox* at start of rmail-swap-buffers
Cache OK in test.mbox at right before swapping
Cache OK in  *message-viewer test.mbox* at right before swapping
Cache OK in test.mbox at right after swapping
Cache OK in  *message-viewer test.mbox* at right after swapping
Cache OK in test.mbox at end of rmail-swap-buffers
Cache OK in  *message-viewer test.mbox* at end of rmail-swap-buffers
Cache OK in test.mbox at start of rmail-swap-buffers
Cache OK in  *message-viewer test.mbox* at start of rmail-swap-buffers
Cache OK in test.mbox at right before swapping
Cache OK in  *message-viewer test.mbox* at right before swapping
Cache OK in test.mbox at right after swapping
Cache OK in  *message-viewer test.mbox* at right after swapping
Cache OK in test.mbox at end of rmail-swap-buffers
Cache OK in  *message-viewer test.mbox* at end of rmail-swap-buffers
Getting mail from /old/home/kdo/test.in...
invalidate_region_cache (test.mbox, 361, 361, 1449) (revalidated)
invalidate_region_cache (test.mbox, 1449, 2014, 2014) (revalidated)

invalidate_region_cache (test.mbox, 2012, 2012, 2014)
invalidate_region_cache (test.mbox, 2030, 2030, 2032) (revalidated)
know_region_cache (test.mbox, 2012, 2030, 2034)
invalidate_region_cache (test.mbox, 2032, 2032, 2034) (revalidated)
know_region_cache (test.mbox, 2030, 2032, 2042)
invalidate_region_cache (test.mbox, 2040, 2040, 2042) (revalidated)
know_region_cache (test.mbox, 2032, 2040, 2043)

Counting new messages...done (1)
Saving file /home/kdo/test.mbox...
Wrote /home/kdo/test.mbox [2 times]
1 new message read
invalidate_region_cache (test.mbox, 2038, 2039, 2043) (revalidated)
invalidate_region_cache (test.mbox, 2038, 2038, 2042) (revalidated)
know_region_cache (test.mbox, 2038, 2039, 2043)
invalidate_region_cache ( *message-viewer test.mbox*, 1, 1202, 1202)
invalidate_region_cache ( *message-viewer test.mbox*, 1, 1, 1)
invalidate_region_cache ( *message-viewer test.mbox*, 1, 1, 1)
invalidate_region_cache ( *message-viewer test.mbox*, 1, 1, 2)
invalidate_region_cache ( *message-viewer test.mbox*, 1, 1, 3)
invalidate_region_cache ( *message-viewer test.mbox*, 39, 39, 41)
invalidate_region_cache ( *message-viewer test.mbox*, 68, 68, 70)
invalidate_region_cache ( *message-viewer test.mbox*, 82, 82, 84)
know_region_cache ( *message-viewer test.mbox*, 77, 81, 126)
know_region_cache ( *message-viewer test.mbox*, 88, 123, 126)
know_region_cache (test.mbox, 1449, 1503, 2043)
know_region_cache (test.mbox, 1504, 1543, 2043)
know_region_cache (test.mbox, 1544, 1581, 2043)
know_region_cache (test.mbox, 1582, 1628, 2043)
know_region_cache (test.mbox, 1629, 1687, 2043)
know_region_cache (test.mbox, 1688, 1760, 2043)
know_region_cache (test.mbox, 1761, 1835, 2043)
know_region_cache (test.mbox, 1836, 1873, 2043)
know_region_cache (test.mbox, 1874, 1902, 2043)
know_region_cache (test.mbox, 1903, 1916, 2043)
know_region_cache (test.mbox, 1917, 1969, 2043)
know_region_cache (test.mbox, 1970, 2011, 2043)

Cache OK in test.mbox at start of rmail-swap-buffers
know_region_cache ( *message-viewer test.mbox*, 1, 38, 126)
know_region_cache ( *message-viewer test.mbox*, 39, 67, 126)
know_region_cache ( *message-viewer test.mbox*, 68, 77, 126)
know_region_cache ( *message-viewer test.mbox*, 82, 88, 126)

Cache OK in  *message-viewer test.mbox* at start of rmail-swap-buffers
Cache OK in test.mbox at right before swapping
Cache OK in  *message-viewer test.mbox* at right before swapping
Cache OK in test.mbox at right after swapping
Cache OK in  *message-viewer test.mbox* at right after swapping
Cache OK in test.mbox at end of rmail-swap-buffers
Cache OK in  *message-viewer test.mbox* at end of rmail-swap-buffers
Cache OK in test.mbox at start of rmail-swap-buffers
Cache OK in  *message-viewer test.mbox* at start of rmail-swap-buffers
Cache OK in test.mbox at right before swapping
Cache OK in  *message-viewer test.mbox* at right before swapping
Cache OK in test.mbox at right after swapping
Cache OK in  *message-viewer test.mbox* at right after swapping
Cache OK in test.mbox at end of rmail-swap-buffers
Cache OK in  *message-viewer test.mbox* at end of rmail-swap-buffers
invalidate_region_cache ( *message-viewer test.mbox*, 1, 126, 126)
invalidate_region_cache ( *message-viewer test.mbox*, 1, 1, 1)
invalidate_region_cache ( *message-viewer test.mbox*, 1, 1, 1)
invalidate_region_cache ( *message-viewer test.mbox*, 1, 1, 2)
invalidate_region_cache ( *message-viewer test.mbox*, 1, 1, 3)
invalidate_region_cache ( *message-viewer test.mbox*, 39, 39, 41)
invalidate_region_cache ( *message-viewer test.mbox*, 68, 68, 70)
invalidate_region_cache ( *message-viewer test.mbox*, 82, 82, 84)
know_region_cache ( *message-viewer test.mbox*, 77, 81, 126)
know_region_cache ( *message-viewer test.mbox*, 88, 123, 126)

Cache OK in test.mbox at start of rmail-swap-buffers
know_region_cache ( *message-viewer test.mbox*, 1, 38, 126)
know_region_cache ( *message-viewer test.mbox*, 39, 67, 126)
know_region_cache ( *message-viewer test.mbox*, 68, 77, 126)
know_region_cache ( *message-viewer test.mbox*, 82, 88, 126)

Cache OK in  *message-viewer test.mbox* at start of rmail-swap-buffers
Cache OK in test.mbox at right before swapping
Cache OK in  *message-viewer test.mbox* at right before swapping
Cache OK in test.mbox at right after swapping
Cache OK in  *message-viewer test.mbox* at right after swapping
Cache OK in test.mbox at end of rmail-swap-buffers
Cache OK in  *message-viewer test.mbox* at end of rmail-swap-buffers
Cache OK in test.mbox at start of rmail-swap-buffers
Cache OK in  *message-viewer test.mbox* at start of rmail-swap-buffers
Cache OK in test.mbox at right before swapping
Cache OK in  *message-viewer test.mbox* at right before swapping
Cache OK in test.mbox at right after swapping
Cache OK in  *message-viewer test.mbox* at right after swapping
Cache OK in test.mbox at end of rmail-swap-buffers
Cache OK in  *message-viewer test.mbox* at end of rmail-swap-buffers
invalidate_region_cache ( *message-viewer test.mbox*, 1, 126, 126)
invalidate_region_cache ( *message-viewer test.mbox*, 1, 1, 1)
invalidate_region_cache ( *message-viewer test.mbox*, 1, 1, 1)
invalidate_region_cache ( *message-viewer test.mbox*, 1, 1, 1089)
invalidate_region_cache ( *message-viewer test.mbox*, 1, 1, 1090)
invalidate_region_cache ( *message-viewer test.mbox*, 32, 32, 1121)
invalidate_region_cache ( *message-viewer test.mbox*, 61, 61, 1150)
invalidate_region_cache ( *message-viewer test.mbox*, 75, 75, 1164)
know_region_cache ( *message-viewer test.mbox*, 7, 31, 1202)
know_region_cache ( *message-viewer test.mbox*, 70, 74, 1202)
know_region_cache (test.mbox, 361, 367, 2043)
know_region_cache (test.mbox, 368, 374, 2043)
know_region_cache (test.mbox, 375, 445, 2043)
know_region_cache (test.mbox, 446, 505, 2043)
know_region_cache (test.mbox, 506, 533, 2043)
know_region_cache (test.mbox, 534, 642, 2043)
know_region_cache (test.mbox, 643, 650, 2043)
know_region_cache (test.mbox, 651, 657, 2043)
know_region_cache (test.mbox, 658, 750, 2043)
know_region_cache (test.mbox, 751, 926, 2043)
know_region_cache (test.mbox, 932, 933, 2043)
know_region_cache (test.mbox, 939, 940, 2043)
know_region_cache (test.mbox, 1010, 1011, 2043)
know_region_cache (test.mbox, 1070, 1071, 2043)
know_region_cache (test.mbox, 1098, 1099, 2043)
know_region_cache (test.mbox, 1207, 1208, 2043)
know_region_cache (test.mbox, 1215, 1216, 2043)
know_region_cache (test.mbox, 1222, 1223, 2043)
know_region_cache (test.mbox, 1315, 1316, 2043)

Newline cache corrupted in test.mbox at start of rmail-swap-buffers
Cache says [55 93 140 171 200 214 252 330 359 360 367 374 445 505 533 642 650 657 750]
Truth is [55 93 140 171 200 214 252 330 359 360 367 374 445 505 533 642 650 657 750 952 1053 1145 1232 1324 1416 1431 1439 1447 1448]

bug#16433: Test case for newline cache corruption

[Eli Zaretskii]

> From: Ken Olum <kdo@cosmos.phy.tufts.edu>
> Cc: 16433@debbugs.gnu.org
> Date: Tue, 27 May 2014 12:20:52 -0400
> 
> Here is some additional output which perhaps will be helpful.  I put
> checks at the beginning and end of rmail-swap-buffers, and also
> immediately before and after the call to buffer-swap-text.  I had these
> checks print a message when the cache is OK, and print the character
> positions on the cache is corrupted, giving the following.  Newlines are
> absent from the cache starting with the one at position 952.

Thanks, this is valuable data.  I will look into this soon.

bug#16433: Test case for newline cache corruption

[Eli Zaretskii]

> From: Ken Olum <kdo@cosmos.phy.tufts.edu>
> Cc: 16433@debbugs.gnu.org
> Date: Tue, 27 May 2014 12:20:52 -0400
> 
> Here is some additional output which perhaps will be helpful.

Thanks.  I know why I couldn't reproduce what you see: you've sent me
the wrong test.mbox file.  The one you are using has only 1 message in
it, whereas the one you sent me had 2 messages, probably because you
have already pressed F4 once to read a new message.

If I manually remove the 2nd message from test.mbox, I see the same
behavior as you reported.  Well, almost the same: for some reason the
traces from region-cache.c are very similar, but not exactly
identical.

I deduced that the initial mbox was not the one you used from the
traces you sent: they clearly indicated that your initial mbox was
smaller than mine (the last number in know_region_cache trace is the
buffer's point-max).

In any case, I seem to have now a reproducible recipe, and can work on
it locally.  Thanks so much for your help.

bug#16433: Test case for newline cache corruption

[Eli Zaretskii]

> Date: Sat, 31 May 2014 13:13:16 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 16433@debbugs.gnu.org
> 
> In any case, I seem to have now a reproducible recipe, and can work on
> it locally.  Thanks so much for your help.

I think I might have nailed the villain.  It was a very old bug.  The
fix is in revision 117181 on the emacs-24 branch; I reproduce it below
for your convenience.  Please try running with it for a while, and if
the problems in Rmail don't come back, I will close the bug.

Thanks again for your help.

--- src/fileio.c	2014-05-11 02:30:57 +0000
+++ src/fileio.c	2014-05-31 18:23:53 +0000
@@ -4087,7 +4087,7 @@ by calling `format-decode', which see.  
 	  && SAVE_MODIFF >= MODIFF)
 	we_locked_file = 1;
 #endif /* CLASH_DETECTION */
-      prepare_to_modify_buffer (GPT, GPT, NULL);
+      prepare_to_modify_buffer (PT, PT, NULL);
     }
 
   move_gap_both (PT, PT_BYTE);

bug#16433: Test case for newline cache corruption

[Ken Olum]

Thanks, Eli.  Sorry about the bad test case. I think I said "let me
check this one more time before sending it in" and then didn't delete
the additional message.

I couldn't apply the patch automatically, because I didn't have any
CLASH_DETECTION switch in my fileio.c, so patch couldn't find the
matching lines. But I applied it by hand and it seems to work.   I will
let you know in a few days whether the problem is completely gone.

How did this get fixed in the emacs-24 branch but not propagated to
trunk?

                                 Ken

bug#16433: Test case for newline cache corruption

[Eli Zaretskii]

> From: Ken Olum <kdo@cosmos.phy.tufts.edu>
> Cc: 16433@debbugs.gnu.org
> Date: Sun, 01 Jun 2014 18:30:00 -0400
> 
> I couldn't apply the patch automatically, because I didn't have any
> CLASH_DETECTION switch in my fileio.c, so patch couldn't find the
> matching lines.

I guess you tried to apply to the trunk, whereas I put the change in
the emacs-24 branch (as it is clearly a regression wrt the previous
release).

> How did this get fixed in the emacs-24 branch but not propagated to
> trunk?

It got merged to the trunk a few hours ago, as part of revision
117220.  The release branch is merged to the trunk from time to time.

bug#16433: Test case for newline cache corruption

[Ken Olum]

As far as I can tell, the fix is good.  I've been running with it all
this week and I have not seen any cache corruption.  As far as I'm
concerned the bug could be closed.

Sorry about my confusion earlier about the different branches.

                                        Ken

bug#16433: Test case for newline cache corruption

[Eli Zaretskii]

> From: Ken Olum <kdo@cosmos.phy.tufts.edu>
> Cc: 16433@debbugs.gnu.org
> Date: Thu, 05 Jun 2014 12:57:46 -0400
> 
> As far as I can tell, the fix is good.  I've been running with it all
> this week and I have not seen any cache corruption.  As far as I'm
> concerned the bug could be closed.

Thanks a lot, closing.

> Sorry about my confusion earlier about the different branches.

No sweat, anybody could make a mistake.

bug#16433: Test case for newline cache corruption

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Hooray for this fix, and kudos to Ken for discovering a test case.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.