Unexpected fill-paragraph behaviour in message-mode

Unexpected fill-paragraph behaviour in message-mode

[Ryan Kavanagh]

Hi,

I am encountering unexpected behaviour when trying to fill paragraphs in
message-mode. In particular, fill-paragraph joins paragraphs (despite
separating new lines) or inserts unwanted tabs. How can I make
fill-paragraph behave as in other text modes?

Below is a small working example for emacs 28.2 that works when loaded
with `emacs -Q`.

Thanks,
Ryan

---------------------------------
From:
To:
Cc:
Bcc:
Subject: Weird fill-paragraph behaviour
Reply-To:

Emacs has weird wrapping behaviour in message mode. To reproduce,
first (M-x message-mode).

Filling this long line (M-q anywhere on it) joins it with the following indented text, which it should not because there is an empty line marking the end of this paragraph:

    Lorem ipsum dolor sit amet, consectetur adipiscing elit.
    Sed quis iaculis erat, at placerat ligula.

Moreover, if I have a long line in a paragraph following indented
text, then attempting to fill it (M-q) causes emacs to insert indents.
Indeed, place your cursor anywhere on the following line, and then hit M-q
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Mauris a massa pulvinar, lobortis massa at, viverra massa. Suspendisse fringilla malesuada erat. Curabitur eu dui lorem. Aenean tincidunt, leo eget tincidunt tincidunt, urna leo mattis eros, in vestibulum mi purus in nibh. Fusce sem elit, vulputate ut tristique a, ornare vitae mi. Quisque sed turpis in metus facilisis mollis et sed augue. Nam vel efficitur erat, id maximus eros. Vestibulum interdum nisl id suscipit sagittis.

Re: Unexpected fill-paragraph behaviour in message-mode

[Eli Zaretskii]

> From: Ryan Kavanagh <rak@rak.ac>
> Date: Sun, 12 Feb 2023 20:11:32 -0500
> 
> 
> I am encountering unexpected behaviour when trying to fill paragraphs in
> message-mode. In particular, fill-paragraph joins paragraphs (despite
> separating new lines) or inserts unwanted tabs. How can I make
> fill-paragraph behave as in other text modes?

This problem doesn't happen if the buffer has the expected separator
line saying "--text follows this line--".  Why are you trying to use
message-mode in a buffer that is not formatted as an email message?
Without that separator, message-mode doesn't know where the headers
end and the message body begins.

Re: Unexpected fill-paragraph behaviour in message-mode

[Ryan Kavanagh]

On Mon, Feb 13, 2023 at 03:35:08PM +0200, Eli Zaretskii wrote:
> This problem doesn't happen if the buffer has the expected separator
> line saying "--text follows this line--".  Why are you trying to use
> message-mode in a buffer that is not formatted as an email message?

The line was missing because I use (neo)mutt as my email client and I
decided to try setting my editor to "emacsclient -c", but mutt does not
insert said separator. Is there an alternate mode I should be using when
compose emails with non-emacs MUAs?

> Without that separator, message-mode doesn't know where the headers
> end and the message body begins.

Quickly skimming message.el, it looks like message-mode (only?) supports
mail and news. In those cases, RFC 5322 §2.1 [0] and RFC 5536 §2 [1]
give the following separator:

    The body is simply a sequence of characters that follows the header
    section and is separated from the header section by an empty line
    (i.e., a line with nothing preceding the CRLF).

Looking at the source in message.el, it usually finds the separation
between header and body using:

    (re-search-forward
           (concat "^" (regexp-quote mail-header-separator) "$")

It would be nice to be able to use emacs as an editor with non-emacs
MUAs, so I wonder if I shouldn't file a bug report asking message.el to
fallback on the RFC (and set mail-header-separator to "") in the cases
where "--text follows this line--" is missing.

Best wishes,
Ryan

[0] https://www.rfc-editor.org/rfc/rfc5322#section-2.1
[1] https://www.rfc-editor.org/rfc/rfc5536#section-2

-- 
|)|/  Ryan Kavanagh  | 4E46 9519 ED67 7734 268F
|\|\  https://rak.ac | BD95 8F7B F8FC 4A11 C97A

Re: Unexpected fill-paragraph behaviour in message-mode

[Teemu Likonen]

[-- Attachment #1: Type: text/plain, Size: 1634 bytes --]

* 2023-02-13 09:41:13-0500, Ryan Kavanagh wrote:

> On Mon, Feb 13, 2023 at 03:35:08PM +0200, Eli Zaretskii wrote:
>> This problem doesn't happen if the buffer has the expected separator
>> line saying "--text follows this line--". Why are you trying to use
>> message-mode in a buffer that is not formatted as an email message?
>
> The line was missing because I use (neo)mutt as my email client and I
> decided to try setting my editor to "emacsclient -c", but mutt does
> not insert said separator. Is there an alternate mode I should be
> using when compose emails with non-emacs MUAs?

> It would be nice to be able to use emacs as an editor with non-emacs
> MUAs, so I wonder if I shouldn't file a bug report asking message.el to
> fallback on the RFC (and set mail-header-separator to "") in the cases
> where "--text follows this line--" is missing.

I use rarely Mutt but for those cases I have defined a derived major
mode from message-mode. See the code below. I set mail-header-separator
variable locally to empty string, and it should answer your basic
question.


    (add-to-list 'auto-mode-alist '("/var/tmp/mutt-.*" . mutt-message-mode))

    (defvar mutt-message-mode-map
      (let ((map (make-sparse-keymap)))
        map))

    (define-derived-mode mutt-message-mode message-mode "Mutt"
      "Major mode for editing mail."
      (set-keymap-parent mutt-message-mode-map nil)
      (abbrev-mode 0)
      (mml-mode 0)
      (setq-local mail-header-separator ""))

-- 
/// Teemu Likonen - .-.. https://www.iki.fi/tlikonen/
// OpenPGP: 6965F03973F0D4CA22B9410F0F2CAE0E07608462

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 251 bytes --]

Re: Unexpected fill-paragraph behaviour in message-mode

[Eli Zaretskii]

> Date: Mon, 13 Feb 2023 09:41:13 -0500
> From: Ryan Kavanagh <rak@rak.ac>
> 
> On Mon, Feb 13, 2023 at 03:35:08PM +0200, Eli Zaretskii wrote:
> > This problem doesn't happen if the buffer has the expected separator
> > line saying "--text follows this line--".  Why are you trying to use
> > message-mode in a buffer that is not formatted as an email message?
> 
> The line was missing because I use (neo)mutt as my email client and I
> decided to try setting my editor to "emacsclient -c", but mutt does not
> insert said separator. Is there an alternate mode I should be using when
> compose emails with non-emacs MUAs?

Yes, use Emacs as $EDITOR (assuming those MUAs support that).

Alternatively, use text-mode instead of message-mode.

> > Without that separator, message-mode doesn't know where the headers
> > end and the message body begins.
> 
> Quickly skimming message.el, it looks like message-mode (only?) supports
> mail and news. In those cases, RFC 5322 §2.1 [0] and RFC 5536 §2 [1]
> give the following separator:
> 
>     The body is simply a sequence of characters that follows the header
>     section and is separated from the header section by an empty line
>     (i.e., a line with nothing preceding the CRLF).

Wrong place to look for the relevant stuff: those RFCs describes how
email messages look _in_transit_, i.e. when they are sent over the
wire.  Here we are talking about the message in the edit buffer.
Emacs removes the separator line before sending the message (actually,
it sends the parts above and below the separator separately), so we
are in full compliance with those RFCs.

> It would be nice to be able to use emacs as an editor with non-emacs
> MUAs, so I wonder if I shouldn't file a bug report asking message.el to
> fallback on the RFC (and set mail-header-separator to "") in the cases
> where "--text follows this line--" is missing.

You could try that, but my response will be that message-mode is not
meant for this job.  It is a mode for composing email that is intended
to be sent by Emacs MUA, not for editing arbitrary text.

Re: Unexpected fill-paragraph behaviour in message-mode

[Jean Louis]

* Ryan Kavanagh <rak@rak.ac> [2023-02-13 17:42]:
> Is there an alternate mode I should be using when compose emails
> with non-emacs MUAs?

I use all time best e-mail client for Emacs, also known as Mutt. I
know it is external and non-Emacs. 

My setting is peculiar:

macro compose 0 ":set editor=mutt-record-email.lisp<enter><edit-headers>:set editor=emacsclient<enter>"

as I like to record e-mail externally, for statistics and other
macros, but normal setting would be:

set editor=emacsclient

> Looking at the source in message.el, it usually finds the separation
> between header and body using:
> 
>     (re-search-forward
>            (concat "^" (regexp-quote mail-header-separator) "$")
> 
> It would be nice to be able to use emacs as an editor with non-emacs
> MUAs

The above is not much relevant to your quest.

Any external MUA can use Emacs either directly by "emacs" command or
"emacsclient". 

> so I wonder if I shouldn't file a bug report asking message.el to
> fallback on the RFC (and set mail-header-separator to "") in the cases
> where "--text follows this line--" is missing.

I may say how I do it:

M-x mail to compose e-mails. I have more easier way to do it, I can
compose directly to all people by single click, where my identity and
recipient, including Cc, Bcc is all setup correctly. Mail is first
saved in file, and later sorted in Maildirs, e-mail is dispatched.

To read e-mails, I use Mutt 

The Mutt E-Mail Client:
http://mutt.org/

and I am so much more efficient than with any Emacs client, but any
MUA can be used, including Thunderbird, it has external editor option
(or internal).

-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/

Re: Unexpected fill-paragraph behaviour in message-mode

[Ryan Kavanagh]

On Mon, Feb 13, 2023 at 05:20:42PM +0200, Teemu Likonen wrote:
> I use rarely Mutt but for those cases I have defined a derived major
> mode from message-mode. See the code below.

Thank you for this code snippet. It is very helpful and I will use some
variant of it.

On Mon, Feb 13, 2023 at 05:21:44PM +0200, Eli Zaretskii wrote:
> You could try that, but my response will be that message-mode is not
> meant for this job.  It is a mode for composing email that is intended
> to be sent by Emacs MUA, not for editing arbitrary text.

Thanks for this clarification. Having used message-mode to compose
emails in the past, I mistakenly assumed it could be used to compose
emails for non-emacs MUAs.

Best wishes,
Ryan

-- 
|)|/  Ryan Kavanagh  | 4E46 9519 ED67 7734 268F
|\|\  https://rak.ac | BD95 8F7B F8FC 4A11 C97A

Re: Unexpected fill-paragraph behaviour in message-mode

[Jean Louis]

* Ryan Kavanagh <rak@rak.ac> [2023-02-14 15:56]:
> Thanks for this clarification. Having used message-mode to compose
> emails in the past, I mistakenly assumed it could be used to compose
> emails for non-emacs MUAs.

It can be used. I now use message-mode to write this e-mail. But I do not use all features.

- I cannot attach or sign messages this way, I will do that in Mutt

- I cannot use all features of message-mode

- I can use mostly editing feature.

-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/

Re: Unexpected fill-paragraph behaviour in message-mode

[Emanuel Berg]

Jean Louis wrote:

>> Thanks for this clarification. Having used message-mode to
>> compose emails in the past, I mistakenly assumed it could
>> be used to compose emails for non-emacs MUAs.
>
> It can be used. I now use message-mode to write this e-mail.

Getting closer to the real powerhouse, Emacs Gnus, eh, "as we
speak", are we, Jean? :)

> But I do not use all features.

Healthy, as message.el is 9010 lines of Elisp :)

> I cannot attach or sign messages this way, I will do that in
> Mutt

Okay, but what exactly do you mean by "attach or sign
messages"? I wouldn't rule out it can be done our automated
without manual intervention in Mutt-land ...

Saying that with no disrespect to Mutt or Mutt users, be their
intervention manual or automatic ...

Please choose, manual or automatic transmission?

  https://www.youtube.com/watch?v=zF8cCHig6ns

-- 
underground experts united
https://dataswamp.org/~incal

Re: Unexpected fill-paragraph behaviour in message-mode

[Stefan Monnier via Users list for the GNU Emacs text editor]

> ---------------------------------
> From:
> To:
> Cc:
> Bcc:
> Subject: Weird fill-paragraph behaviour
> Reply-To:
>
> Emacs has weird wrapping behaviour in message mode. To reproduce,
> first (M-x message-mode).
>
> Filling this long line (M-q anywhere on it) joins it with the following
> indented text, which it should not because there is an empty line marking
> the end of this paragraph:

It looks like `message-mode` thinks you're still in the header part of
the message because of the lack of the separator that `message-mode`
uses (i.e. "--text follows this line--")

> Quickly skimming message.el, it looks like message-mode (only?) supports
> mail and news. In those cases, RFC 5322 §2.1 [0] and RFC 5536 §2 [1]
> give the following separator:

These RFCs document the format of messages when they leave the MUA, but
the format within the MUA (e.g. within a `message-mode` buffer) if up to
the MUA, of course.

> (re-search-forward
>        (concat "^" (regexp-quote mail-header-separator) "$")

For a more graceful degradation, `message-mode` should treat an empty
line as the header separator when the "--text follows this line--"
thingy is missing, indeed.  [ I wouldn't be surprised if `message-mode`
ends up generating incorrectly formatted messages if you include an
empty line the header.  ]

Patch welcome :-)


        Stefan

Re: Unexpected fill-paragraph behaviour in message-mode

[Stefan Monnier via Users list for the GNU Emacs text editor]

>> Thanks for this clarification. Having used message-mode to compose
>> emails in the past, I mistakenly assumed it could be used to compose
>> emails for non-emacs MUAs.
>
> It can be used.  I now use message-mode to write this e-mail.
> But I do not use all features.
>
> - I cannot attach or sign messages this way, I will do that in Mutt
>
> - I cannot use all features of message-mode
>
> - I can use mostly editing feature.

Very good points.  Maybe we should provide a specific function that sets
up message-mode for use as $EDITOR so that you can use
`mml-attache-file`, sign, etc...


        Stefan