From mboxrd@z Thu Jan  1 00:00:00 1970
Path: news.gmane.org!not-for-mail
From: Jay Berkenbilt <ejb@ql.org>
Newsgroups: gmane.emacs.bugs,gmane.emacs.pretest.bugs
Subject: bug#3769: 23.0.95;
	unrmail and b2m may create invalid mbox files (patch)
Date: Mon, 06 Jul 2009 16:59:33 -0400
Message-ID: <20090706165933.2386033337.qww314159@motoko.argon.local>
Reply-To: Jay Berkenbilt <ejb@ql.org>, 3769@emacsbugs.donarmstrong.com
NNTP-Posting-Host: lo.gmane.org
Mime-Version: 1.0
Content-Type: multipart/mixed; boundary="=-=-="
X-Trace: ger.gmane.org 1246915070 13135 80.91.229.12 (6 Jul 2009 21:17:50 GMT)
X-Complaints-To: usenet@ger.gmane.org
NNTP-Posting-Date: Mon, 6 Jul 2009 21:17:50 +0000 (UTC)
To: emacs-pretest-bug@gnu.org
Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Mon Jul 06 23:17:43 2009
Return-path: <bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org>
Envelope-to: geb-bug-gnu-emacs@m.gmane.org
Original-Received: from lists.gnu.org ([199.232.76.165])
	by lo.gmane.org with esmtp (Exim 4.50)
	id 1MNvZD-0004GQ-Ou
	for geb-bug-gnu-emacs@m.gmane.org; Mon, 06 Jul 2009 23:17:40 +0200
Original-Received: from localhost ([127.0.0.1]:36823 helo=lists.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.43)
	id 1MNvZD-0007sc-0r
	for geb-bug-gnu-emacs@m.gmane.org; Mon, 06 Jul 2009 17:17:39 -0400
Original-Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43)
	id 1MNvZ8-0007sO-Nr
	for bug-gnu-emacs@gnu.org; Mon, 06 Jul 2009 17:17:34 -0400
Original-Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43)
	id 1MNvZ4-0007oI-Ks
	for bug-gnu-emacs@gnu.org; Mon, 06 Jul 2009 17:17:34 -0400
Original-Received: from [199.232.76.173] (port=38496 helo=monty-python.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.43) id 1MNvZ4-0007o2-CF
	for bug-gnu-emacs@gnu.org; Mon, 06 Jul 2009 17:17:30 -0400
Original-Received: from rzlab.ucr.edu ([138.23.92.77]:49469)
	by monty-python.gnu.org with esmtps
	(TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60)
	(envelope-from <debbugs@rzlab.ucr.edu>) id 1MNvZ3-00082Y-Qj
	for bug-gnu-emacs@gnu.org; Mon, 06 Jul 2009 17:17:30 -0400
Original-Received: from rzlab.ucr.edu (rzlab.ucr.edu [127.0.0.1])
	by rzlab.ucr.edu (8.14.3/8.14.3/Debian-5) with ESMTP id n66LHQD9020676; 
	Mon, 6 Jul 2009 14:17:26 -0700
Original-Received: (from debbugs@localhost)
	by rzlab.ucr.edu (8.14.3/8.14.3/Submit) id n66L55jw017741;
	Mon, 6 Jul 2009 14:05:05 -0700
X-Loop: owner@emacsbugs.donarmstrong.com
Resent-From: Jay Berkenbilt <ejb@ql.org>
Resent-To: bug-submit-list@donarmstrong.com
Resent-CC: Emacs Bugs <bug-gnu-emacs@gnu.org>
Resent-Date: Mon, 06 Jul 2009 21:05:05 +0000
Resent-Message-ID: <handler.3769.B.124691398916454@emacsbugs.donarmstrong.com>
Resent-Sender: owner@emacsbugs.donarmstrong.com
X-Emacs-PR-Message: report 3769
X-Emacs-PR-Package: emacs
X-Emacs-PR-Keywords: 
Original-Received: via spool by submit@emacsbugs.donarmstrong.com id=B.124691398916454
	(code B ref -1); Mon, 06 Jul 2009 21:05:05 +0000
Original-Received: (at submit) by emacsbugs.donarmstrong.com; 6 Jul 2009 20:59:49 +0000
X-Spam-Bayes: score:0.5 Bayes not run. spammytokens:Tokens not available.
	hammytokens:Tokens not available.
Original-Received: from fencepost.gnu.org (fencepost.gnu.org [140.186.70.10])
	by rzlab.ucr.edu (8.14.3/8.14.3/Debian-5) with ESMTP id n66Kxgfb016440
	for <submit@emacsbugs.donarmstrong.com>; Mon, 6 Jul 2009 13:59:44 -0700
Original-Received: from mx10.gnu.org ([199.232.76.166]:34127)
	by fencepost.gnu.org with esmtp (Exim 4.67)
	(envelope-from <primary@qbilt.org>) id 1MNvHq-00009Q-4a
	for emacs-pretest-bug@gnu.org; Mon, 06 Jul 2009 16:59:42 -0400
Original-Received: from Debian-exim by monty-python.gnu.org with spam-scanned (Exim
	4.60) (envelope-from <primary@qbilt.org>) id 1MNvHm-00056T-OT
	for emacs-pretest-bug@gnu.org; Mon, 06 Jul 2009 16:59:41 -0400
Original-Received: from hermes.mail.tigertech.net ([64.62.209.72]:58802)
	by monty-python.gnu.org with esmtp (Exim 4.60)
	(envelope-from <primary@qbilt.org>) id 1MNvHm-00055u-2Q
	for emacs-pretest-bug@gnu.org; Mon, 06 Jul 2009 16:59:38 -0400
Original-Received: from localhost (localhost [127.0.0.1])
	by hermes.tigertech.net (Postfix) with ESMTP id AC98C430463;
	Mon,  6 Jul 2009 13:59:35 -0700 (PDT)
X-Virus-Scanned: Debian amavisd-new at hermes.tigertech.net
Original-Received: from motoko.argon.local (unknown [72.165.80.2])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by hermes.tigertech.net (Postfix) with ESMTP id D6D3143045A;
	Mon,  6 Jul 2009 13:59:34 -0700 (PDT)
X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6 (newer, 2)
X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6 (newer, 2)
Resent-Date: Mon, 06 Jul 2009 17:17:34 -0400
X-BeenThere: bug-gnu-emacs@gnu.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: "Bug reports for GNU Emacs,
	the Swiss army knife of text editors" <bug-gnu-emacs.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/listinfo/bug-gnu-emacs>,
	<mailto:bug-gnu-emacs-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/pipermail/bug-gnu-emacs>
List-Post: <mailto:bug-gnu-emacs@gnu.org>
List-Help: <mailto:bug-gnu-emacs-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/bug-gnu-emacs>,
	<mailto:bug-gnu-emacs-request@gnu.org?subject=subscribe>
Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org
Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org
Xref: news.gmane.org gmane.emacs.bugs:29208 gmane.emacs.pretest.bugs:24732
Archived-At: <http://permalink.gmane.org/gmane.emacs.bugs/29208>

--=-=-=


This is a very serious problem with an easy fix.  When converting a
BABYL file to mbox format, both unrmail and b2m create invalid mbox
files if the message does not end with a blank line.  The vast majority
of my BABYL files contain messages that do not end with a blank line.
Virtually all of these have been written by gnus with
gnus-summary-save-article saving in rmail format
(gnus-summary-save-in-rmail).  When this happens, there is no newline
before the "^From " in the new mbox file, which makes the resulting file
unusable.

I'm attaching two files: a patch that fixes both unrmail.el and b2m.c,
and a sample RMAIL file that illustrates the problem.  If you load the
attached RMAIL file with C-x C-f or with M-x rmail, rmail will show only
1 message when there are in fact three messages.  Older versions of
emacs properly show this to have three messages.  Likewise, if you run
this through b2m, and then use M-x rmail on the result, you will see
only one message.  You can also just look at the resulting mbox file in
either case and see that there are no blank lines before "^From ".

I've removed the Received headers and other private information from the
RMAIL file.

In GNU Emacs 23.0.95.1 (i686-pc-linux-gnu, GTK+ Version 2.10.4)
 of 2009-06-23 on motoko.argon.local
Windowing system distributor `The X.Org Foundation', version 11.0.70101000
configured using `configure  '--prefix=/opt/tps/packages/linux.ix86.rhel5/emacs-23.0.95-1''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=none
  locale-coding-system: utf-8-unix
  default-enable-multibyte-characters: t

Major mode: Text

Minor modes in effect:
  which-function-mode: t
  tooltip-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  global-auto-composition-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t

Recent input:
s M-< C-g C-z C-r y e s <return> M-< C-M-s ^ F r o 
m SPC C-s C-s C-s C-s C-s C-s C-s C-s C-s C-s C-s C-s 
C-s C-s C-s C-s C-s C-s C-g C-x C-c C-f C-g C-x C-f 
/ t m p / p <return> C-x b <return> C-x b b <tab> <return> 
C-n C-n C-k C-k TAB C-n C-a C-k C-k C-k <C-backspace> 
C-n C-k C-k TAB C-n C-a C-k C-k C-x C-s C-x b p <return> 
C-z C-r y e s <return> C-v C-x b u <tab> <return> C-p 
C-p C-p C-p C-p C-p C-p C-p C-p C-p C-p C-p C-p M-f 
M-f C-d C-d C-n C-e M-b M-d 3 C-n C-a C-k C-k C-k C-k 
C-k C-k TAB C-n TAB C-n TAB C-p C-e <backspace> C-n 
TAB C-x C-s C-n C-n C-n C-n C-n C-e C-x C-e C-x b X 
<tab> C-g C-x C-f / t m p / X M <S-tab> <return> C-x 
k <return> C-p C-a C-p C-p C-p C-p C-p C-p C-p C-p 
C-p ; ; C-n C-a ; ; C-n C-a ; ; C-n C-a ; ; n C-a ; 
<backspace> <C-backspace> <C-backspace> <C-backspace> 
; ; C-n C-a ; ; C-x C-s C-n C-n C-n C-n C-n C-n C-e 
C-x C-e C-x k X <tab> C-g C-x C-f M-p <return> C-x 
k <return> <C-backspace> <C-backspace> <C-backspace> 
<C-backspace> <C-backspace> <C-backspace> <C-backspace> 
<C-backspace> <C-backspace> <C-backspace> <C-backspace> 
C-x C-s C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-e 
C-x C-e C-x C-f M-p <return> C-x k <return> C-x C-f 
/ t m p / p <return> y e s <return> M-< M-> M-< C-v 
C-x m C-g C-x k <return> y e s <return> M-x g n u s 
<M-backspace> r e p o r t SPC b SPC <return>

Recent messages:
Wrote /tmp/rmail23860-sQ
Writing messages to /tmp/rmail23860L3W...done
Replacing BABYL format with mbox format...
Marking buffer unmodified to avoid rewriting Babyl file as mbox file
Counting messages...done
Replacing BABYL format with mbox format...done
Counting messages...done
Marking buffer unmodified to avoid rewriting Babyl file as mbox file
Mark set [3 times]
Starting new Ispell process [default] ...
Quit

--=-=-=
Content-Type: text/x-patch
Content-Disposition: inline; filename=unrmail-b2m.patch

--- lib-src/b2m.c.qdist	2006-04-04 00:13:01.000000000 -0400
+++ lib-src/b2m.c	2009-07-06 16:42:11.000000000 -0400
@@ -94,7 +94,7 @@
      int argc;
      char **argv;
 {
-  logical labels_saved, printing, header;
+  logical labels_saved, printing, header, first, last_was_blank_line;
   time_t ltoday;
   struct tm *tm;
   char *labels, *p, *today;
@@ -139,7 +139,8 @@
       exit (EXIT_SUCCESS);
     }
 
-  labels_saved = printing = header = FALSE;
+  labels_saved = printing = header = last_was_blank_line = FALSE;
+  first = TRUE;
   ltoday = time (0);
   /* Convert to a string, checking for out-of-range time stamps.
      Don't use 'ctime', as that might dump core if the hardware clock
@@ -166,6 +167,13 @@
 
       if (data.buffer[0] == '\037')
 	{
+	  if (first)
+	    first = FALSE;
+	  else
+	    {
+	      if (! last_was_blank_line)
+		puts("");
+	    }
 	  if (data.buffer[1] == '\0')
 	    continue;
 	  else if (data.buffer[1] == '\f')
@@ -195,7 +203,13 @@
 	}
 
       if (printing)
-	puts (data.buffer);
+	{
+	  puts (data.buffer);
+	  if (data.buffer[0] == '\0')
+	    last_was_blank_line = TRUE;
+	  else
+	    last_was_blank_line = FALSE;
+	}
     }
 
   return EXIT_SUCCESS;
--- lisp/mail/unrmail.el.qdist	2009-04-10 17:02:58.000000000 -0400
+++ lisp/mail/unrmail.el	2009-07-06 16:43:20.000000000 -0400
@@ -232,6 +232,12 @@
 	      (while (search-forward "\nFrom " nil t)
 		(forward-char -5)
 		(insert ?>)))
+	    ;; Make sure the message ends with two newlines
+	    (goto-char (point-max))
+	    (insert "\n")
+	    (goto-char (- (point-max) 3))
+	    (if (looking-at "\n\n\n")
+		(delete-region (point) (1+ (point))))
 	    ;; Write it to the output file, suitably encoded.
 	    (let ((coding-system-for-write coding))
 	      (write-region (point-min) (point-max) to-file t

--=-=-=
Content-Type: application/octet-stream
Content-Disposition: attachment; filename=RMAIL

BABYL OPTIONS: -*- rmail -*-
Version: 5
Labels:
Note:   This is the header of an rmail file.
Note:   If you are seeing it in rmail,
Note:    it means the file has no messages in it.

1,,
X-Coding-System: undecided-unix
Mail-from: From grabmail Tue Jul  6 19:47:28 GMT 2009
Return-Path: <zzz.org@tigertech.net>
X-Original-To: primary@zzz.org
Delivered-To: primary@zzz.org
X-TigerTech-Content-Filter: Clean
X-Original-To: zzz.org@mom.tigertech.net
Delivered-To: zzz.org@mom.tigertech.net
X-Virus-Scanned: Debian amavisd-new at hermes.tigertech.net
From: Jay Berkenbilt <ejb@zzz.org>
To: ejb@zzz.org
Subject: message 1
Message-ID: <20090706154703.0613297635.qww314159@motoko.argon.local>
Date: Mon, 06 Jul 2009 15:47:03 -0400
X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.0.2

*** EOOH ***
X-Original-To: primary@zzz.org
X-TigerTech-Content-Filter: Clean
X-Virus-Scanned: Debian amavisd-new at hgblob.tigertech.net
X-Original-To: zzz.org@mom.tigertech.net
X-TigerTech-Content-Filter: Clean
X-Virus-Scanned: Debian amavisd-new at hermes.tigertech.net
From: Jay Berkenbilt <ejb@zzz.org>
To: ejb@zzz.org
Subject: message 1
Date: Mon, 06 Jul 2009 15:47:03 -0400
X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.0.2

message 1

1,,
X-Coding-System: undecided-unix
Mail-from: From grabmail Tue Jul  6 19:47:29 GMT 2009
Return-Path: <zzz.org@tigertech.net>
X-Original-To: primary@zzz.org
Delivered-To: primary@zzz.org
X-TigerTech-Content-Filter: Clean
X-Virus-Scanned: Debian amavisd-new at hgblob.tigertech.net
X-Original-To: zzz.org@mom.tigertech.net
Delivered-To: zzz.org@mom.tigertech.net
X-TigerTech-Content-Filter: Clean
X-Virus-Scanned: Debian amavisd-new at hermes.tigertech.net
From: Jay Berkenbilt <ejb@zzz.org>
To: ejb@zzz.org
Subject: message 2
Message-ID: <20090706154710.0613259785.qww314159@motoko.argon.local>
Date: Mon, 06 Jul 2009 15:47:11 -0400
X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.0.2

*** EOOH ***
X-Original-To: primary@zzz.org
X-TigerTech-Content-Filter: Clean
X-Virus-Scanned: Debian amavisd-new at hgblob.tigertech.net
X-Original-To: zzz.org@mom.tigertech.net
X-TigerTech-Content-Filter: Clean
X-Virus-Scanned: Debian amavisd-new at hermes.tigertech.net
From: Jay Berkenbilt <ejb@zzz.org>
To: ejb@zzz.org
Subject: message 2
Date: Mon, 06 Jul 2009 15:47:11 -0400
X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.0.2

message 2

1,,
X-Coding-System: undecided-unix
Mail-from: From grabmail Tue Jul  6 19:47:29 GMT 2009
Return-Path: <zzz.org@tigertech.net>
X-Original-To: primary@zzz.org
Delivered-To: primary@zzz.org
X-TigerTech-Content-Filter: Clean
X-Virus-Scanned: Debian amavisd-new at hgblob.tigertech.net
X-Original-To: zzz.org@mom.tigertech.net
Delivered-To: zzz.org@mom.tigertech.net
X-TigerTech-Content-Filter: Clean
X-Virus-Scanned: Debian amavisd-new at hermes.tigertech.net
From: Jay Berkenbilt <ejb@zzz.org>
To: ejb@zzz.org
Subject: message 3
Message-ID: <20090706154719.0613238423.qww314159@motoko.argon.local>
Date: Mon, 06 Jul 2009 15:47:20 -0400
X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.0.2

*** EOOH ***
X-Original-To: primary@zzz.org
X-TigerTech-Content-Filter: Clean
X-Virus-Scanned: Debian amavisd-new at hgblob.tigertech.net
X-Original-To: zzz.org@mom.tigertech.net
X-TigerTech-Content-Filter: Clean
X-Virus-Scanned: Debian amavisd-new at hermes.tigertech.net
From: Jay Berkenbilt <ejb@zzz.org>
To: ejb@zzz.org
Subject: message 3
Date: Mon, 06 Jul 2009 15:47:20 -0400
X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.0.2

message 3

--=-=-=--