exim pipe transport, notmuch insert and mbox-style messages

exim pipe transport, notmuch insert and mbox-style messages

[Edward Betts]

My mail arrives on a Debian machine running exim. It was being filtered
by procmail then piped into 'notmuch insert'. This was generating the
single-message mbox warning:

> Warning: ...  is an mbox containing a single message,
> likely caused by misconfigured mail delivery.  Support for single-message
> mboxes is deprecated and may be removed in the future.

I thought it was a problem with procmail delivering mbox style messages to
pipes, so I switched to maildrop. Same problem. This warning is produced
because there is an mbox-style From_ line before the first header line. Here
is an example:

> From MAILER-DAEMON Fri Jul  8 12:08:34 2011

The man page for maildrop says this style of message isn't supported, but it
doesn't detect and warn about them. It will pipe them into commands without
removing the From_ line.

Reading the exim documentation for pipe transports I found the message_prefix
option, which is by default is set like this:

> message_prefix = \
>   From ${if def:return_path{$return_path}{MAILER-DAEMON}}\
>   ${tod_bsdinbox}\n

This means that exim turns all messages into mbox style messages before
delivering them to a pipe. The justification given in the documentation is
"This is required by the commonly used /usr/bin/vacation program."

The fix is to unset the exim message_prefix option for delivery to maildrop.
Here is an example of the Debian exim maildrop_pipe transport with a blank
message_prefix to stop the From_ line being added before the message is piped
to maildrop.

> maildrop_pipe:
>   debug_print = "T: maildrop_pipe for $local_part@$domain"
>   driver = pipe
>   path = "/bin:/usr/bin:/usr/local/bin"
>   command = "/usr/bin/maildrop"
>   message_prefix =
>   message_suffix =
>   return_path_add
>   delivery_date_add
>   envelope_to_add

The same fix could be applied for procmail pipe or a pipe from exim to
'notmuch insert'.

I submitted a Debian bug for exim4-config with my change as a patch. The
maintainer has excepted my patch and uploaded a release of exim4-config with
the fix to Debian experimental. 

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=769396

I wonder if 'notmuch insert' could be modified to detect and drop the From_
line before writing the message to disk and index it. It could do that
silently or with a warning.

-- 
Edward.

Re: exim pipe transport, notmuch insert and mbox-style messages

[David Bremner]

Edward Betts <edward@4angle.com> writes:


> I wonder if 'notmuch insert' could be modified to detect and drop the From_
> line before writing the message to disk and index it. It could do that
> silently or with a warning.

I don't know about the larger question(s), but I'd suggest just escaping
it to something like X-Envelope-From: . There may even be some
semi-standard header to use for this.

d

Re: exim pipe transport, notmuch insert and mbox-style messages

[Andrei POPESCU]

[-- Attachment #1: Type: text/plain, Size: 1582 bytes --]

On Sb, 22 nov 14, 10:44:45, David Bremner wrote:
> Edward Betts <edward@4angle.com> writes:
> 
> 
> > I wonder if 'notmuch insert' could be modified to detect and drop the From_
> > line before writing the message to disk and index it. It could do that
> > silently or with a warning.
> 
> I don't know about the larger question(s), but I'd suggest just escaping
> it to something like X-Envelope-From: . There may even be some
> semi-standard header to use for this.

I had the same problem with postfix + maildrop, as local(8) 
unconditionally adds the 'From ' header when delivering to an external 
command and upstream did not consider my suggestion to make this 
configurable. The work-arounds I came up with:

1. use 'reformail -f0' as suggested in maildrop(1)

reformail(1):

    -f0
    Any initial blank lines are removed. If the first non-blank line is a 
    "From_" line, it gets converted to a "Return-Path:" header, and any
    existing "Return-Path:" header gets removed. If the message does not 
    start with a "From_" line, the message remains unchanged.

2. adjust the 'maildrop' destination in postfix' master.cf to remove the 
'F' flag.

pipe(8):

    F   Prepend a "From sender time_stamp" envelope header to the 
        message content.  This is expected by, for example, UUCP
        software.

3. Deliver from postfix to a Maildir, bypassing any filtering.

Kind regards,
Andrei
-- 
If you can't explain it simply, you don't understand it well enough.
(Albert Einstein)
http://nuvreauspam.ro/gpg-transition.txt

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: exim pipe transport, notmuch insert and mbox-style messages

[Tomi Ollila]

On Sat, Nov 22 2014, Edward Betts wrote:

> My mail arrives on a Debian machine running exim. It was being filtered
> by procmail then piped into 'notmuch insert'. This was generating the
> single-message mbox warning:
>
>> Warning: ...  is an mbox containing a single message,
>> likely caused by misconfigured mail delivery.  Support for single-message
>> mboxes is deprecated and may be removed in the future.
>

I personally would like to see this feature undeprecated, as I see no
reason accepting such messages...

Tomi


> I wonder if 'notmuch insert' could be modified to detect and drop the From_
> line before writing the message to disk and index it. It could do that
> silently or with a warning.
>
> -- 
> Edward.

Re: exim pipe transport, notmuch insert and mbox-style messages

[Tomi Ollila]

On Sun, Nov 23 2014, Tomi Ollila wrote:

> On Sat, Nov 22 2014, Edward Betts wrote:
>
>> My mail arrives on a Debian machine running exim. It was being filtered
>> by procmail then piped into 'notmuch insert'. This was generating the
>> single-message mbox warning:
>>
>>> Warning: ...  is an mbox containing a single message,
>>> likely caused by misconfigured mail delivery.  Support for single-message
>>> mboxes is deprecated and may be removed in the future.
>>
>
> I personally would like to see this feature undeprecated, as I see no
> reason accepting such messages...

+ /// *not* \\\

:)

>
> Tomi

Tomi

>
>
>> I wonder if 'notmuch insert' could be modified to detect and drop the From_
>> line before writing the message to disk and index it. It could do that
>> silently or with a warning.
>>
>> -- 
>> Edward.

Re: exim pipe transport, notmuch insert and mbox-style messages

[Edward Betts]

David Bremner <david@tethera.net> wrote:
> Edward Betts <edward@4angle.com> writes:
> > I wonder if 'notmuch insert' could be modified to detect and drop the From_
> > line before writing the message to disk and index it. It could do that
> > silently or with a warning.
> 
> I don't know about the larger question(s), but I'd suggest just escaping
> it to something like X-Envelope-From: . There may even be some
> semi-standard header to use for this.

The main piece of information in the From_ line is the return path address, a
properly configured MTA will add a Return-path header.

RFC 2821 says the Return-path header is added to a message "when the
delivery SMTP server makes the final delivery".

I don't think the From_ line needs to be captured into an X-Envelope-From
header.  We could make 'notmuch insert' add the 'Return-path' header to
messages if it is missing, but the From_ line is present. This is probably too
much complexity.

-- 
Edward.

Re: exim pipe transport, notmuch insert and mbox-style messages

[David Bremner]

Edward Betts <edward@4angle.com> writes:

> I don't think the From_ line needs to be captured into an X-Envelope-From
> header.  We could make 'notmuch insert' add the 'Return-path' header to
> messages if it is missing, but the From_ line is present. This is probably too
> much complexity.

I think we're going to solve this problem on the notmuch indexing side
with

        id:1416741312-26332-1-git-send-email-jani@nikula.org

This will let us stick to the principle of notmuch not modifying
mail.

d