bug#3769: 23.0.95; unrmail and b2m may create invalid mbox files (patch)

bug#3769: 23.0.95; unrmail and b2m may create invalid mbox files (patch)

[Jay Berkenbilt]

[-- Attachment #1: Type: text/plain, Size: 3960 bytes --]


This is a very serious problem with an easy fix.  When converting a
BABYL file to mbox format, both unrmail and b2m create invalid mbox
files if the message does not end with a blank line.  The vast majority
of my BABYL files contain messages that do not end with a blank line.
Virtually all of these have been written by gnus with
gnus-summary-save-article saving in rmail format
(gnus-summary-save-in-rmail).  When this happens, there is no newline
before the "^From " in the new mbox file, which makes the resulting file
unusable.

I'm attaching two files: a patch that fixes both unrmail.el and b2m.c,
and a sample RMAIL file that illustrates the problem.  If you load the
attached RMAIL file with C-x C-f or with M-x rmail, rmail will show only
1 message when there are in fact three messages.  Older versions of
emacs properly show this to have three messages.  Likewise, if you run
this through b2m, and then use M-x rmail on the result, you will see
only one message.  You can also just look at the resulting mbox file in
either case and see that there are no blank lines before "^From ".

I've removed the Received headers and other private information from the
RMAIL file.

In GNU Emacs 23.0.95.1 (i686-pc-linux-gnu, GTK+ Version 2.10.4)
 of 2009-06-23 on motoko.argon.local
Windowing system distributor `The X.Org Foundation', version 11.0.70101000
configured using `configure  '--prefix=/opt/tps/packages/linux.ix86.rhel5/emacs-23.0.95-1''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=none
  locale-coding-system: utf-8-unix
  default-enable-multibyte-characters: t

Major mode: Text

Minor modes in effect:
  which-function-mode: t
  tooltip-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  global-auto-composition-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t

Recent input:
s M-< C-g C-z C-r y e s <return> M-< C-M-s ^ F r o 
m SPC C-s C-s C-s C-s C-s C-s C-s C-s C-s C-s C-s C-s 
C-s C-s C-s C-s C-s C-s C-g C-x C-c C-f C-g C-x C-f 
/ t m p / p <return> C-x b <return> C-x b b <tab> <return> 
C-n C-n C-k C-k TAB C-n C-a C-k C-k C-k <C-backspace> 
C-n C-k C-k TAB C-n C-a C-k C-k C-x C-s C-x b p <return> 
C-z C-r y e s <return> C-v C-x b u <tab> <return> C-p 
C-p C-p C-p C-p C-p C-p C-p C-p C-p C-p C-p C-p M-f 
M-f C-d C-d C-n C-e M-b M-d 3 C-n C-a C-k C-k C-k C-k 
C-k C-k TAB C-n TAB C-n TAB C-p C-e <backspace> C-n 
TAB C-x C-s C-n C-n C-n C-n C-n C-e C-x C-e C-x b X 
<tab> C-g C-x C-f / t m p / X M <S-tab> <return> C-x 
k <return> C-p C-a C-p C-p C-p C-p C-p C-p C-p C-p 
C-p ; ; C-n C-a ; ; C-n C-a ; ; C-n C-a ; ; n C-a ; 
<backspace> <C-backspace> <C-backspace> <C-backspace> 
; ; C-n C-a ; ; C-x C-s C-n C-n C-n C-n C-n C-n C-e 
C-x C-e C-x k X <tab> C-g C-x C-f M-p <return> C-x 
k <return> <C-backspace> <C-backspace> <C-backspace> 
<C-backspace> <C-backspace> <C-backspace> <C-backspace> 
<C-backspace> <C-backspace> <C-backspace> <C-backspace> 
C-x C-s C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-e 
C-x C-e C-x C-f M-p <return> C-x k <return> C-x C-f 
/ t m p / p <return> y e s <return> M-< M-> M-< C-v 
C-x m C-g C-x k <return> y e s <return> M-x g n u s 
<M-backspace> r e p o r t SPC b SPC <return>

Recent messages:
Wrote /tmp/rmail23860-sQ
Writing messages to /tmp/rmail23860L3W...done
Replacing BABYL format with mbox format...
Marking buffer unmodified to avoid rewriting Babyl file as mbox file
Counting messages...done
Replacing BABYL format with mbox format...done
Counting messages...done
Marking buffer unmodified to avoid rewriting Babyl file as mbox file
Mark set [3 times]
Starting new Ispell process [default] ...
Quit

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: unrmail-b2m.patch --]
[-- Type: text/x-patch, Size: 1803 bytes --]

--- lib-src/b2m.c.qdist	2006-04-04 00:13:01.000000000 -0400
+++ lib-src/b2m.c	2009-07-06 16:42:11.000000000 -0400
@@ -94,7 +94,7 @@
      int argc;
      char **argv;
 {
-  logical labels_saved, printing, header;
+  logical labels_saved, printing, header, first, last_was_blank_line;
   time_t ltoday;
   struct tm *tm;
   char *labels, *p, *today;
@@ -139,7 +139,8 @@
       exit (EXIT_SUCCESS);
     }
 
-  labels_saved = printing = header = FALSE;
+  labels_saved = printing = header = last_was_blank_line = FALSE;
+  first = TRUE;
   ltoday = time (0);
   /* Convert to a string, checking for out-of-range time stamps.
      Don't use 'ctime', as that might dump core if the hardware clock
@@ -166,6 +167,13 @@
 
       if (data.buffer[0] == '\037')
 	{
+	  if (first)
+	    first = FALSE;
+	  else
+	    {
+	      if (! last_was_blank_line)
+		puts("");
+	    }
 	  if (data.buffer[1] == '\0')
 	    continue;
 	  else if (data.buffer[1] == '\f')
@@ -195,7 +203,13 @@
 	}
 
       if (printing)
-	puts (data.buffer);
+	{
+	  puts (data.buffer);
+	  if (data.buffer[0] == '\0')
+	    last_was_blank_line = TRUE;
+	  else
+	    last_was_blank_line = FALSE;
+	}
     }
 
   return EXIT_SUCCESS;
--- lisp/mail/unrmail.el.qdist	2009-04-10 17:02:58.000000000 -0400
+++ lisp/mail/unrmail.el	2009-07-06 16:43:20.000000000 -0400
@@ -232,6 +232,12 @@
 	      (while (search-forward "\nFrom " nil t)
 		(forward-char -5)
 		(insert ?>)))
+	    ;; Make sure the message ends with two newlines
+	    (goto-char (point-max))
+	    (insert "\n")
+	    (goto-char (- (point-max) 3))
+	    (if (looking-at "\n\n\n")
+		(delete-region (point) (1+ (point))))
 	    ;; Write it to the output file, suitably encoded.
 	    (let ((coding-system-for-write coding))
 	      (write-region (point-min) (point-max) to-file t

[-- Attachment #3: RMAIL --]
[-- Type: application/octet-stream, Size: 3593 bytes --]

bug#3769: 23.0.95; unrmail and b2m may create invalid mbox files (patch)

[Chong Yidong]

> This is a very serious problem with an easy fix.  When converting a
> BABYL file to mbox format, both unrmail and b2m create invalid mbox
> files if the message does not end with a blank line.  I'm attaching
> two files: a patch that fixes both unrmail.el and b2m.c, and a sample
> RMAIL file that illustrates the problem.

Thanks for spotting this bug.  I've checked in (slightly modified
versions of) your patches into CVS, and they will be included in 23.1.