From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: Jay Berkenbilt Newsgroups: gmane.emacs.bugs,gmane.emacs.pretest.bugs Subject: bug#3769: 23.0.95; unrmail and b2m may create invalid mbox files (patch) Date: Mon, 06 Jul 2009 16:59:33 -0400 Message-ID: <20090706165933.2386033337.qww314159@motoko.argon.local> Reply-To: Jay Berkenbilt , 3769@emacsbugs.donarmstrong.com NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Trace: ger.gmane.org 1246915070 13135 80.91.229.12 (6 Jul 2009 21:17:50 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Mon, 6 Jul 2009 21:17:50 +0000 (UTC) To: emacs-pretest-bug@gnu.org Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Mon Jul 06 23:17:43 2009 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane.org Original-Received: from lists.gnu.org ([199.232.76.165]) by lo.gmane.org with esmtp (Exim 4.50) id 1MNvZD-0004GQ-Ou for geb-bug-gnu-emacs@m.gmane.org; Mon, 06 Jul 2009 23:17:40 +0200 Original-Received: from localhost ([127.0.0.1]:36823 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1MNvZD-0007sc-0r for geb-bug-gnu-emacs@m.gmane.org; Mon, 06 Jul 2009 17:17:39 -0400 Original-Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1MNvZ8-0007sO-Nr for bug-gnu-emacs@gnu.org; Mon, 06 Jul 2009 17:17:34 -0400 Original-Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1MNvZ4-0007oI-Ks for bug-gnu-emacs@gnu.org; Mon, 06 Jul 2009 17:17:34 -0400 Original-Received: from [199.232.76.173] (port=38496 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1MNvZ4-0007o2-CF for bug-gnu-emacs@gnu.org; Mon, 06 Jul 2009 17:17:30 -0400 Original-Received: from rzlab.ucr.edu ([138.23.92.77]:49469) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1MNvZ3-00082Y-Qj for bug-gnu-emacs@gnu.org; Mon, 06 Jul 2009 17:17:30 -0400 Original-Received: from rzlab.ucr.edu (rzlab.ucr.edu [127.0.0.1]) by rzlab.ucr.edu (8.14.3/8.14.3/Debian-5) with ESMTP id n66LHQD9020676; Mon, 6 Jul 2009 14:17:26 -0700 Original-Received: (from debbugs@localhost) by rzlab.ucr.edu (8.14.3/8.14.3/Submit) id n66L55jw017741; Mon, 6 Jul 2009 14:05:05 -0700 X-Loop: owner@emacsbugs.donarmstrong.com Resent-From: Jay Berkenbilt Resent-To: bug-submit-list@donarmstrong.com Resent-CC: Emacs Bugs Resent-Date: Mon, 06 Jul 2009 21:05:05 +0000 Resent-Message-ID: Resent-Sender: owner@emacsbugs.donarmstrong.com X-Emacs-PR-Message: report 3769 X-Emacs-PR-Package: emacs X-Emacs-PR-Keywords: Original-Received: via spool by submit@emacsbugs.donarmstrong.com id=B.124691398916454 (code B ref -1); Mon, 06 Jul 2009 21:05:05 +0000 Original-Received: (at submit) by emacsbugs.donarmstrong.com; 6 Jul 2009 20:59:49 +0000 X-Spam-Bayes: score:0.5 Bayes not run. spammytokens:Tokens not available. hammytokens:Tokens not available. Original-Received: from fencepost.gnu.org (fencepost.gnu.org [140.186.70.10]) by rzlab.ucr.edu (8.14.3/8.14.3/Debian-5) with ESMTP id n66Kxgfb016440 for ; Mon, 6 Jul 2009 13:59:44 -0700 Original-Received: from mx10.gnu.org ([199.232.76.166]:34127) by fencepost.gnu.org with esmtp (Exim 4.67) (envelope-from ) id 1MNvHq-00009Q-4a for emacs-pretest-bug@gnu.org; Mon, 06 Jul 2009 16:59:42 -0400 Original-Received: from Debian-exim by monty-python.gnu.org with spam-scanned (Exim 4.60) (envelope-from ) id 1MNvHm-00056T-OT for emacs-pretest-bug@gnu.org; Mon, 06 Jul 2009 16:59:41 -0400 Original-Received: from hermes.mail.tigertech.net ([64.62.209.72]:58802) by monty-python.gnu.org with esmtp (Exim 4.60) (envelope-from ) id 1MNvHm-00055u-2Q for emacs-pretest-bug@gnu.org; Mon, 06 Jul 2009 16:59:38 -0400 Original-Received: from localhost (localhost [127.0.0.1]) by hermes.tigertech.net (Postfix) with ESMTP id AC98C430463; Mon, 6 Jul 2009 13:59:35 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at hermes.tigertech.net Original-Received: from motoko.argon.local (unknown [72.165.80.2]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by hermes.tigertech.net (Postfix) with ESMTP id D6D3143045A; Mon, 6 Jul 2009 13:59:34 -0700 (PDT) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6 (newer, 2) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6 (newer, 2) Resent-Date: Mon, 06 Jul 2009 17:17:34 -0400 X-BeenThere: bug-gnu-emacs@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Xref: news.gmane.org gmane.emacs.bugs:29208 gmane.emacs.pretest.bugs:24732 Archived-At: --=-=-= This is a very serious problem with an easy fix. When converting a BABYL file to mbox format, both unrmail and b2m create invalid mbox files if the message does not end with a blank line. The vast majority of my BABYL files contain messages that do not end with a blank line. Virtually all of these have been written by gnus with gnus-summary-save-article saving in rmail format (gnus-summary-save-in-rmail). When this happens, there is no newline before the "^From " in the new mbox file, which makes the resulting file unusable. I'm attaching two files: a patch that fixes both unrmail.el and b2m.c, and a sample RMAIL file that illustrates the problem. If you load the attached RMAIL file with C-x C-f or with M-x rmail, rmail will show only 1 message when there are in fact three messages. Older versions of emacs properly show this to have three messages. Likewise, if you run this through b2m, and then use M-x rmail on the result, you will see only one message. You can also just look at the resulting mbox file in either case and see that there are no blank lines before "^From ". I've removed the Received headers and other private information from the RMAIL file. In GNU Emacs 23.0.95.1 (i686-pc-linux-gnu, GTK+ Version 2.10.4) of 2009-06-23 on motoko.argon.local Windowing system distributor `The X.Org Foundation', version 11.0.70101000 configured using `configure '--prefix=/opt/tps/packages/linux.ix86.rhel5/emacs-23.0.95-1'' Important settings: value of $LC_ALL: nil value of $LC_COLLATE: nil value of $LC_CTYPE: nil value of $LC_MESSAGES: nil value of $LC_MONETARY: nil value of $LC_NUMERIC: nil value of $LC_TIME: nil value of $LANG: en_US.UTF-8 value of $XMODIFIERS: @im=none locale-coding-system: utf-8-unix default-enable-multibyte-characters: t Major mode: Text Minor modes in effect: which-function-mode: t tooltip-mode: t mouse-wheel-mode: t file-name-shadow-mode: t global-font-lock-mode: t font-lock-mode: t blink-cursor-mode: t global-auto-composition-mode: t auto-composition-mode: t auto-encryption-mode: t auto-compression-mode: t column-number-mode: t line-number-mode: t Recent input: s M-< C-g C-z C-r y e s M-< C-M-s ^ F r o m SPC C-s C-s C-s C-s C-s C-s C-s C-s C-s C-s C-s C-s C-s C-s C-s C-s C-s C-s C-g C-x C-c C-f C-g C-x C-f / t m p / p C-x b C-x b b C-n C-n C-k C-k TAB C-n C-a C-k C-k C-k C-n C-k C-k TAB C-n C-a C-k C-k C-x C-s C-x b p C-z C-r y e s C-v C-x b u C-p C-p C-p C-p C-p C-p C-p C-p C-p C-p C-p C-p C-p M-f M-f C-d C-d C-n C-e M-b M-d 3 C-n C-a C-k C-k C-k C-k C-k C-k TAB C-n TAB C-n TAB C-p C-e C-n TAB C-x C-s C-n C-n C-n C-n C-n C-e C-x C-e C-x b X C-g C-x C-f / t m p / X M C-x k C-p C-a C-p C-p C-p C-p C-p C-p C-p C-p C-p ; ; C-n C-a ; ; C-n C-a ; ; C-n C-a ; ; n C-a ; ; ; C-n C-a ; ; C-x C-s C-n C-n C-n C-n C-n C-n C-e C-x C-e C-x k X C-g C-x C-f M-p C-x k C-x C-s C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-e C-x C-e C-x C-f M-p C-x k C-x C-f / t m p / p y e s M-< M-> M-< C-v C-x m C-g C-x k y e s M-x g n u s r e p o r t SPC b SPC Recent messages: Wrote /tmp/rmail23860-sQ Writing messages to /tmp/rmail23860L3W...done Replacing BABYL format with mbox format... Marking buffer unmodified to avoid rewriting Babyl file as mbox file Counting messages...done Replacing BABYL format with mbox format...done Counting messages...done Marking buffer unmodified to avoid rewriting Babyl file as mbox file Mark set [3 times] Starting new Ispell process [default] ... Quit --=-=-= Content-Type: text/x-patch Content-Disposition: inline; filename=unrmail-b2m.patch --- lib-src/b2m.c.qdist 2006-04-04 00:13:01.000000000 -0400 +++ lib-src/b2m.c 2009-07-06 16:42:11.000000000 -0400 @@ -94,7 +94,7 @@ int argc; char **argv; { - logical labels_saved, printing, header; + logical labels_saved, printing, header, first, last_was_blank_line; time_t ltoday; struct tm *tm; char *labels, *p, *today; @@ -139,7 +139,8 @@ exit (EXIT_SUCCESS); } - labels_saved = printing = header = FALSE; + labels_saved = printing = header = last_was_blank_line = FALSE; + first = TRUE; ltoday = time (0); /* Convert to a string, checking for out-of-range time stamps. Don't use 'ctime', as that might dump core if the hardware clock @@ -166,6 +167,13 @@ if (data.buffer[0] == '\037') { + if (first) + first = FALSE; + else + { + if (! last_was_blank_line) + puts(""); + } if (data.buffer[1] == '\0') continue; else if (data.buffer[1] == '\f') @@ -195,7 +203,13 @@ } if (printing) - puts (data.buffer); + { + puts (data.buffer); + if (data.buffer[0] == '\0') + last_was_blank_line = TRUE; + else + last_was_blank_line = FALSE; + } } return EXIT_SUCCESS; --- lisp/mail/unrmail.el.qdist 2009-04-10 17:02:58.000000000 -0400 +++ lisp/mail/unrmail.el 2009-07-06 16:43:20.000000000 -0400 @@ -232,6 +232,12 @@ (while (search-forward "\nFrom " nil t) (forward-char -5) (insert ?>))) + ;; Make sure the message ends with two newlines + (goto-char (point-max)) + (insert "\n") + (goto-char (- (point-max) 3)) + (if (looking-at "\n\n\n") + (delete-region (point) (1+ (point)))) ;; Write it to the output file, suitably encoded. (let ((coding-system-for-write coding)) (write-region (point-min) (point-max) to-file t --=-=-= Content-Type: application/octet-stream Content-Disposition: attachment; filename=RMAIL BABYL OPTIONS: -*- rmail -*- Version: 5 Labels: Note: This is the header of an rmail file. Note: If you are seeing it in rmail, Note: it means the file has no messages in it.  1,, X-Coding-System: undecided-unix Mail-from: From grabmail Tue Jul 6 19:47:28 GMT 2009 Return-Path: X-Original-To: primary@zzz.org Delivered-To: primary@zzz.org X-TigerTech-Content-Filter: Clean X-Original-To: zzz.org@mom.tigertech.net Delivered-To: zzz.org@mom.tigertech.net X-Virus-Scanned: Debian amavisd-new at hermes.tigertech.net From: Jay Berkenbilt To: ejb@zzz.org Subject: message 1 Message-ID: <20090706154703.0613297635.qww314159@motoko.argon.local> Date: Mon, 06 Jul 2009 15:47:03 -0400 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.0.2 *** EOOH *** X-Original-To: primary@zzz.org X-TigerTech-Content-Filter: Clean X-Virus-Scanned: Debian amavisd-new at hgblob.tigertech.net X-Original-To: zzz.org@mom.tigertech.net X-TigerTech-Content-Filter: Clean X-Virus-Scanned: Debian amavisd-new at hermes.tigertech.net From: Jay Berkenbilt To: ejb@zzz.org Subject: message 1 Date: Mon, 06 Jul 2009 15:47:03 -0400 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.0.2 message 1  1,, X-Coding-System: undecided-unix Mail-from: From grabmail Tue Jul 6 19:47:29 GMT 2009 Return-Path: X-Original-To: primary@zzz.org Delivered-To: primary@zzz.org X-TigerTech-Content-Filter: Clean X-Virus-Scanned: Debian amavisd-new at hgblob.tigertech.net X-Original-To: zzz.org@mom.tigertech.net Delivered-To: zzz.org@mom.tigertech.net X-TigerTech-Content-Filter: Clean X-Virus-Scanned: Debian amavisd-new at hermes.tigertech.net From: Jay Berkenbilt To: ejb@zzz.org Subject: message 2 Message-ID: <20090706154710.0613259785.qww314159@motoko.argon.local> Date: Mon, 06 Jul 2009 15:47:11 -0400 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.0.2 *** EOOH *** X-Original-To: primary@zzz.org X-TigerTech-Content-Filter: Clean X-Virus-Scanned: Debian amavisd-new at hgblob.tigertech.net X-Original-To: zzz.org@mom.tigertech.net X-TigerTech-Content-Filter: Clean X-Virus-Scanned: Debian amavisd-new at hermes.tigertech.net From: Jay Berkenbilt To: ejb@zzz.org Subject: message 2 Date: Mon, 06 Jul 2009 15:47:11 -0400 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.0.2 message 2  1,, X-Coding-System: undecided-unix Mail-from: From grabmail Tue Jul 6 19:47:29 GMT 2009 Return-Path: X-Original-To: primary@zzz.org Delivered-To: primary@zzz.org X-TigerTech-Content-Filter: Clean X-Virus-Scanned: Debian amavisd-new at hgblob.tigertech.net X-Original-To: zzz.org@mom.tigertech.net Delivered-To: zzz.org@mom.tigertech.net X-TigerTech-Content-Filter: Clean X-Virus-Scanned: Debian amavisd-new at hermes.tigertech.net From: Jay Berkenbilt To: ejb@zzz.org Subject: message 3 Message-ID: <20090706154719.0613238423.qww314159@motoko.argon.local> Date: Mon, 06 Jul 2009 15:47:20 -0400 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.0.2 *** EOOH *** X-Original-To: primary@zzz.org X-TigerTech-Content-Filter: Clean X-Virus-Scanned: Debian amavisd-new at hgblob.tigertech.net X-Original-To: zzz.org@mom.tigertech.net X-TigerTech-Content-Filter: Clean X-Virus-Scanned: Debian amavisd-new at hermes.tigertech.net From: Jay Berkenbilt To: ejb@zzz.org Subject: message 3 Date: Mon, 06 Jul 2009 15:47:20 -0400 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.0.2 message 3  --=-=-=--