documentation of Rmail 23 mailbox format?

documentation of Rmail 23 mailbox format?

[Mark Lillibridge]

    I've found at least one bug (get messages destroys all white space
at the end of the last message already in the mailbox) in Rmail 23 that
requires me to understand the correct Rmail 23 mailbox format in order
to fix the bug.  Is there any documentation of this?

    Unless told otherwise, I'm going to assume that Rmail 23 is supposed
be using the mboxo/mboxrd mailbox format (see
http://homepage.ntlworld.com./jonathan.deboynepollard/FGA/mail-mbox-formats.html).
In particular, this means that:

    Each message is preceded by a From_ line and followed by a blank line. A
    From_ line is a line that begins with the five characters 'F', 'r', 'o',
    'm', and ' '.

    Rmail 23 does not comply correctly with this.  For example, it fails
to add a blank line after the last message.  

- Mark

Re: documentation of Rmail 23 mailbox format?

[Chong Yidong]

Mark Lillibridge <mark.lillibridge@hp.com> writes:

> I've found at least one bug (get messages destroys all white space at
> the end of the last message already in the mailbox) in Rmail 23

I think that's coming from this code in rmail-get-new-mail:

      ;; In case of brain damage caused by require-final-newline.
      (goto-char (point-max))
      (skip-chars-backward " \t\n")
      (delete-region (point) (point-max))

Could you try removing these lines and seeing if the problem persists?
The require-final-newline variable is set to nil in rmail mode anyway so
this should not be necessary.

>     Unless told otherwise, I'm going to assume that Rmail 23 is supposed
> be using the mboxo/mboxrd mailbox format (see
> http://homepage.ntlworld.com./jonathan.deboynepollard/FGA/mail-mbox-formats.html).

IIRC, it is supposed to use the mboxrd format.  For instance, the
unquoting of From lines for message display is handled by this code in
rmail-show-message-1:

      ;; Unquote quoted From lines
      (while (re-search-forward "^>+From " nil t)
        (beginning-of-line)
        (delete-char 1)
        (forward-line))

> In particular, this means that:
>
>     Each message is preceded by a From_ line and followed by a blank
>     line. A From_ line is a line that begins with the five characters
>     'F', 'r', 'o', 'm', and ' '.
>
> Rmail 23 does not comply correctly with this.  For example, it fails
> to add a blank line after the last message.

I think that's the purpose of this code from rmail-insert-inbox-text:

      <after inserting the contents of the inbox>
      ;; Determine if a pair of newline message separators need
      ;; to be added to the new collection of messages.  This is
      ;; the case for all new message collections added to a
      ;; non-empty mail file.
      (unless (zerop size)
        (save-restriction
          (let ((start (point-min)))
            (widen)
            (unless (eq start (point-min))
              (goto-char start)
              (insert "\n\n")
              (setq size (+ 2 size))))))
      (goto-char (point-max))
      (or (= (preceding-char) ?\n)
          (zerop size)
          (insert ?\n))

Maybe it's not doing the right thing in some circumstances?

Re: documentation of Rmail 23 mailbox format?

[Mark Lillibridge]

Chong Yidong wrote:
>  Mark Lillibridge <mark.lillibridge@hp.com> writes:
>  
>  > I've found at least one bug (get messages destroys all white space at
>  > the end of the last message already in the mailbox) in Rmail 23
>  
>  I think that's coming from this code in rmail-get-new-mail:
>  
>        ;; In case of brain damage caused by require-final-newline.
>        (goto-char (point-max))
>        (skip-chars-backward " \t\n")
>        (delete-region (point) (point-max))

    Good eye.  That is indeed the primary cause of the bug.  They need
to be replaced with code that fixes up the mbox damage caused by this
bug; e.g., add newline's until the file ends with at least one blank
line.

    There's another bug further downstream revealed by that fix,
however.  In rmail.el:2022 in the rmail-insert-inbox-text function
is:

	    ;; Determine if a pair of newline message separators need
	    ;; to be added to the new collection of messages.  This is
	    ;; the case for all new message collections added to a
	    ;; non-empty mail file.
	    (unless (zerop size)
	      (save-restriction
		(let ((start (point-min)))
		  (widen)
		  (unless (eq start (point-min))
		    (goto-char start)
		    (insert "\n\n")
		    (setq size (+ 2 size))))))

This code is compensating for the previous broken code and needs to be
deleted.  I'm working on a patch and will include it with my official
bug report, hopefully later today.



>  >     Unless told otherwise, I'm going to assume that Rmail 23 is supposed
>  > be using the mboxo/mboxrd mailbox format (see
>  > http://homepage.ntlworld.com./jonathan.deboynepollard/FGA/mail-mbox-formats.html).
>  
>  IIRC, it is supposed to use the mboxrd format.  For instance, the
>  unquoting of From lines for message display is handled by this code in
>  rmail-show-message-1:
>  
>        ;; Unquote quoted From lines
>        (while (re-search-forward "^>+From " nil t)
>          (beginning-of-line)
>          (delete-char 1)
>          (forward-line))

    Ok.  Can we document this somewhere?  I'm happy to write up some
text describing mboxrd if desired.  Rmail edit assumes mboxo by the way
so the codebase is inconsistent.



>  > In particular, this means that:
>  >
>  >     Each message is preceded by a From_ line and followed by a blank
>  >     line. A From_ line is a line that begins with the five characters
>  >     'F', 'r', 'o', 'm', and ' '.
>  >
>  > Rmail 23 does not comply correctly with this.  For example, it fails
>  > to add a blank line after the last message.
>  
>  I think that's the purpose of this code from rmail-insert-inbox-text:
>  
>        <after inserting the contents of the inbox>
>        ;; Determine if a pair of newline message separators need
>        ;; to be added to the new collection of messages.  This is
>        ;; the case for all new message collections added to a
>        ;; non-empty mail file.
>        (unless (zerop size)
>          (save-restriction
>            (let ((start (point-min)))
>              (widen)
>              (unless (eq start (point-min))
>                (goto-char start)
>                (insert "\n\n")
>                (setq size (+ 2 size))))))
>        (goto-char (point-max))
>        (or (= (preceding-char) ?\n)
>            (zerop size)
>            (insert ?\n))
>  
>  Maybe it's not doing the right thing in some circumstances?

    More like doing the wrong thing in all circumstances when this bug
is fixed.  It mostly compensates for the bug when there is new mail but
as it doesn't run when there is no new mail, it fails to always
compensate.

- Mark