bug#65071: 29.1; RMAIL changed on disk leads to rmail-error-bad-format

bug#65071: 29.1; RMAIL changed on disk leads to rmail-error-bad-format

[emacs-bug]

In Emacs 29.1, starting with no ~/RMAIL file and empty primary mail
spool file ('$' indicates command line terminal prompt outside Emacs):

  $ mail me@foo ... # send a message to myself

  emacs -Q

  M-x RMAIL # retrieves/reads message

  $ cat /dev/null > ~/RMAIL # clear (or otherwise modify) RMAIL file

  $ mail me@foo ... # send another message to myself

  M-x RMAIL

  "File RMAIL changed on disk. Reread from disk? (yes or no) " yes

    "rmail-error-bad-format: Message is not a valid RFC 822 (or later)
    message" and:

      mode changes from RMAIL to fundamental

      ~/RMAIL file on disk looks good (proper RFC 822 file)

      ~/RMAIL file as visited in buffer looks bad, can be seen (e.g.,
        upon immediate save) to be missing top "From " and other header
        lines ("Return-Path:", "Received:", etc.)

In GNU Emacs 29.1 (build 1, aarch64-apple-darwin21.6.0, NS
 appkit-2113.60 Version 12.6.6 (Build 21G646)) of 2023-07-30 built on
 armbob.lan
Windowing system distributor 'Apple', version 10.3.2113
System Description:  macOS 12.6.8

Configured using:
 'configure --with-ns '--enable-locallisppath=/Library/Application
 Support/Emacs/${version}/site-lisp:/Library/Application
 Support/Emacs/site-lisp' --with-modules 'CFLAGS=-DFD_SETSIZE=10000
 -DDARWIN_UNLIMITED_SELECT' --with-x-toolkit=no'

Configured features:
ACL GLIB GMP GNUTLS JPEG JSON LIBXML2 MODULES NOTIFY KQUEUE NS PDUMPER
PNG RSVG THREADS TIFF TOOLKIT_SCROLL_BARS TREE_SITTER ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

bug#65071: 29.1; RMAIL changed on disk leads to rmail-error-bad-format

[Eli Zaretskii]

> From: <emacs-bug@rulnick.com>
> Date: Fri, 04 Aug 2023 14:18:52 -0400
> 
> In Emacs 29.1, starting with no ~/RMAIL file and empty primary mail
> spool file ('$' indicates command line terminal prompt outside Emacs):
> 
>   $ mail me@foo ... # send a message to myself
> 
>   emacs -Q
> 
>   M-x RMAIL # retrieves/reads message
> 
>   $ cat /dev/null > ~/RMAIL # clear (or otherwise modify) RMAIL file
> 
>   $ mail me@foo ... # send another message to myself
> 
>   M-x RMAIL
> 
>   "File RMAIL changed on disk. Reread from disk? (yes or no) " yes
> 
>     "rmail-error-bad-format: Message is not a valid RFC 822 (or later)
>     message" and:
> 
>       mode changes from RMAIL to fundamental
> 
>       ~/RMAIL file on disk looks good (proper RFC 822 file)
> 
>       ~/RMAIL file as visited in buffer looks bad, can be seen (e.g.,
>         upon immediate save) to be missing top "From " and other header
>         lines ("Return-Path:", "Received:", etc.)

Does using "M-x revert-buffer RET" avoid the problem?

In any case, this is not how you are supposed to read new email into
an existing Rmail buffer.  You are supposed to use the 'g' command.

bug#65071: 29.1; RMAIL changed on disk leads to rmail-error-bad-format

[emacs-bug]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: <emacs-bug@rulnick.com>
>> Date: Fri, 04 Aug 2023 14:18:52 -0400
>> 
>> In Emacs 29.1, starting with no ~/RMAIL file and empty primary mail
>> spool file ('$' indicates command line terminal prompt outside Emacs):
>> 
>>   $ mail me@foo ... # send a message to myself
>> 
>>   emacs -Q
>> 
>>   M-x RMAIL # retrieves/reads message
>> 
>>   $ cat /dev/null > ~/RMAIL # clear (or otherwise modify) RMAIL file
>> 
>>   $ mail me@foo ... # send another message to myself
>> 
>>   M-x RMAIL
>> 
>>   "File RMAIL changed on disk. Reread from disk? (yes or no) " yes
>> 
>>     "rmail-error-bad-format: Message is not a valid RFC 822 (or later)
>>     message" and:
>> 
>>       mode changes from RMAIL to fundamental
>> 
>>       ~/RMAIL file on disk looks good (proper RFC 822 file)
>> 
>>       ~/RMAIL file as visited in buffer looks bad, can be seen (e.g.,
>>         upon immediate save) to be missing top "From " and other header
>>         lines ("Return-Path:", "Received:", etc.)
>
> Does using "M-x revert-buffer RET" avoid the problem?
>
> In any case, this is not how you are supposed to read new email into
> an existing Rmail buffer.  You are supposed to use the 'g' command.

1. M-x revert-buffer RET does not avoid or solve the problem completely;
   the contents from disk appear to be restored as expected but the
   buffer remains in Fundamental mode. However, following this with
   RMAIL-mode RET does appear to at least be a fix.

2. Yes, I should have said 'g' rather than 'M-x rmail RET' in the second
   instance. The bug report applies in this case as well.

bug#65071: 29.1; RMAIL changed on disk leads to rmail-error-bad-format

[Eli Zaretskii]

> From: emacs-bug@rulnick.com
> Cc: 65071@debbugs.gnu.org
> Date: Sat, 05 Aug 2023 06:31:10 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: <emacs-bug@rulnick.com>
> >> Date: Fri, 04 Aug 2023 14:18:52 -0400
> >> 
> >> In Emacs 29.1, starting with no ~/RMAIL file and empty primary mail
> >> spool file ('$' indicates command line terminal prompt outside Emacs):
> >> 
> >>   $ mail me@foo ... # send a message to myself
> >> 
> >>   emacs -Q
> >> 
> >>   M-x RMAIL # retrieves/reads message
> >> 
> >>   $ cat /dev/null > ~/RMAIL # clear (or otherwise modify) RMAIL file
> >> 
> >>   $ mail me@foo ... # send another message to myself
> >> 
> >>   M-x RMAIL
> >> 
> >>   "File RMAIL changed on disk. Reread from disk? (yes or no) " yes
> >> 
> >>     "rmail-error-bad-format: Message is not a valid RFC 822 (or later)
> >>     message" and:
> >> 
> >>       mode changes from RMAIL to fundamental
> >> 
> >>       ~/RMAIL file on disk looks good (proper RFC 822 file)
> >> 
> >>       ~/RMAIL file as visited in buffer looks bad, can be seen (e.g.,
> >>         upon immediate save) to be missing top "From " and other header
> >>         lines ("Return-Path:", "Received:", etc.)
> >
> > Does using "M-x revert-buffer RET" avoid the problem?
> >
> > In any case, this is not how you are supposed to read new email into
> > an existing Rmail buffer.  You are supposed to use the 'g' command.
> 
> 1. M-x revert-buffer RET does not avoid or solve the problem completely;
>    the contents from disk appear to be restored as expected but the
>    buffer remains in Fundamental mode. However, following this with
>    RMAIL-mode RET does appear to at least be a fix.
> 
> 2. Yes, I should have said 'g' rather than 'M-x rmail RET' in the second
>    instance. The bug report applies in this case as well.

Then I'm sorry, I don't understand the recipe, because I cannot
reproduce this on my system.  After

   $ cat /dev/null > ~/RMAIL

the ~/RMAIL file is empty, right?  And the Emacs session that used
that file was terminated, i.e. Emacs doesn't run when you clear the
~/RMAIL file?

Then you do "M-x RMAIL"?   But it should be "M-x rmail", lower-case,
right?  Or is the command something else, like

  C-u M-x rmail RET ~/RMAIL RET

?

And how do you modify the recipe to use the 'g' command? can you show
the recipe with 'g' instead of a new "M-x rmail".

bug#65071: 29.1; RMAIL changed on disk leads to rmail-error-bad-format

[emacs-bug]

I hope the receipe below is clearer. In Emacs 29.1, starting with no
~/RMAIL file and empty primary mail spool file (please note that '$'
indicates Bash prompt in Terminal independent of Emacs):

  $ mail me@foo ... # send a message to myself

  $ emacs -Q

  (in Emacs) M-x rmail RET # retrieves/reads message

  $ cat /dev/null > ~/RMAIL # clear RMAIL file

  $ mail me@foo ... # send another message to myself

  (in Emacs) g

    "File RMAIL changed on disk. Reread from disk? (yes or no) " yes RET

    "rmail-error-bad-format: Message is not a valid RFC 822 (or later)
    message"; Emacs mode changes from RMAIL to Fundamental

At this point the ~/RMAIL file on disk looks good (proper RFC 822 file),
but the ~/RMAIL file as visited in buffer looks bad, as can be seen
(e.g., upon immediate save) to be missing top "From " and other header
lines ("Return-Path:", "Received:", etc.). I have many years experience
with Emacs and RMAIL and this behavior is new as of 29.1 or another very
recent version of Emacs.

bug#65071: 29.1; RMAIL changed on disk leads to rmail-error-bad-format

[Eli Zaretskii]

> From: emacs-bug@rulnick.com
> Cc: 65071@debbugs.gnu.org
> Date: Sat, 05 Aug 2023 14:07:40 -0400
> 
> I hope the receipe below is clearer. In Emacs 29.1, starting with no
> ~/RMAIL file and empty primary mail spool file (please note that '$'
> indicates Bash prompt in Terminal independent of Emacs):
> 
>   $ mail me@foo ... # send a message to myself
> 
>   $ emacs -Q
> 
>   (in Emacs) M-x rmail RET # retrieves/reads message
> 
>   $ cat /dev/null > ~/RMAIL # clear RMAIL file
> 
>   $ mail me@foo ... # send another message to myself
> 
>   (in Emacs) g
> 
>     "File RMAIL changed on disk. Reread from disk? (yes or no) " yes RET
> 
>     "rmail-error-bad-format: Message is not a valid RFC 822 (or later)
>     message"; Emacs mode changes from RMAIL to Fundamental
> 
> At this point the ~/RMAIL file on disk looks good (proper RFC 822 file),
> but the ~/RMAIL file as visited in buffer looks bad, as can be seen
> (e.g., upon immediate save) to be missing top "From " and other header
> lines ("Return-Path:", "Received:", etc.). I have many years experience
> with Emacs and RMAIL and this behavior is new as of 29.1 or another very
> recent version of Emacs.

Thanks, this should be fixed now on the emacs-29 branch.  The reason
for the bug was a change in revert-buffer we made during Emacs 29
development.

bug#65071: 29.1; RMAIL changed on disk leads to rmail-error-bad-format

[Eli Zaretskii]

> Cc: 65071@debbugs.gnu.org
> Date: Sun, 06 Aug 2023 09:36:45 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> >     "File RMAIL changed on disk. Reread from disk? (yes or no) " yes RET
> > 
> >     "rmail-error-bad-format: Message is not a valid RFC 822 (or later)
> >     message"; Emacs mode changes from RMAIL to Fundamental
> > 
> > At this point the ~/RMAIL file on disk looks good (proper RFC 822 file),
> > but the ~/RMAIL file as visited in buffer looks bad, as can be seen
> > (e.g., upon immediate save) to be missing top "From " and other header
> > lines ("Return-Path:", "Received:", etc.). I have many years experience
> > with Emacs and RMAIL and this behavior is new as of 29.1 or another very
> > recent version of Emacs.
> 
> Thanks, this should be fixed now on the emacs-29 branch.  The reason
> for the bug was a change in revert-buffer we made during Emacs 29
> development.

No comments within a week, so I assume the bug is indeed fixed, and
I'm closing it.