bounce/forward not working due to CR at line end

bounce/forward not working due to CR at line end

[Kai Harries]

[-- Attachment #1: Type: text/plain, Size: 419 bytes --]

Hello,

I am new to notmuch so excuse if the fault is on my end. For me
resending and forwarding of emails are not working if
message-forward-show-mml is enabled. The problems are CR at the line
endings. With the attached two patches I got it working for me. Maybe
someone more knowledgeable can see from it what the right solution would
be. 

My configuration:
notmuch 0.25.1
Emacs 25.3.1
on GNU/Linux (NixOS 17.09)



[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: patch of mime-to-mml --]
[-- Type: text/x-patch, Size: 581 bytes --]

--- a/mml.el	2017-11-09 21:11:26.542291084 +0100
+++ b/mml.el	2017-11-09 21:12:18.838306315 +0100
@@ -929,10 +929,11 @@
 
 (defun mime-to-mml (&optional handles)
   "Translate the current buffer (which should be a message) into MML.
 If HANDLES is non-nil, use it instead reparsing the buffer."
   ;; First decode the head.
+  (save-excursion (icalendar--clean-up-line-endings))
   (save-restriction
     (message-narrow-to-head)
     (let ((rfc2047-quote-decoded-words-containing-tspecials t))
       (mail-decode-encoded-word-region (point-min) (point-max))))
   (unless handles

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: patch of notmuch-show-view-raw-message --]
[-- Type: text/x-patch, Size: 926 bytes --]

--- a/notmuch-show.el	2017-11-09 21:17:05.791414308 +0100
+++ b/notmuch-show.el	2017-11-09 21:17:57.958437388 +0100
@@ -2011,20 +2011,21 @@
 (defun notmuch-show-view-raw-message ()
   "View the original source of the current message."
   (interactive)
   (let* ((id (notmuch-show-get-message-id))
 	 (buf (get-buffer-create (concat "*notmuch-raw-" id "*")))
 	 (inhibit-read-only t))
     (switch-to-buffer buf)
     (erase-buffer)
     (let ((coding-system-for-read 'no-conversion))
       (call-process notmuch-command nil t nil "show" "--format=raw" id))
+    (save-excursion (icalendar--clean-up-line-endings))
     (goto-char (point-min))
     (set-buffer-modified-p nil)
     (setq buffer-read-only t)
     (view-buffer buf 'kill-buffer-if-not-modified)))
 
 (defun notmuch-show-resume-message ()
   "Resume EDITING the current draft message."
   (interactive)
   (notmuch-draft-resume (notmuch-show-get-message-id)))
 

[-- Attachment #4: Type: text/plain, Size: 27 bytes --]


Best regards

Kai Harries

Re: bounce/forward not working due to CR at line end

[David Bremner]

Kai Harries <kai.harries@gmail.com> writes:

> Hello,
>
> I am new to notmuch so excuse if the fault is on my end. For me
> resending and forwarding of emails are not working if
> message-forward-show-mml is enabled. The problems are CR at the line
> endings. With the attached two patches I got it working for me. Maybe
> someone more knowledgeable can see from it what the right solution would
> be. 

During off-list discussion, Kai mentioned id:87a7zsm5ol.fsf@gmail.com as
a message that he cannot bounce (and also not forward?). For me this
forwards fine, but does not bounce because it has a corrupted Cc header,
and my MTA rejects it.  That doesn't sound related to the original
problem report (nothing about line endings).

I guess we still need a good test case [1], but my best guess is that
something about Kai's settings. Kai, if you have the source unpacked
(even if not built), try

      $ ./devel/try-emacs-mua -Q

and see if the problem is duplicated there.



[1]: I tried both the copy of that message I already have and the copy
     Kai sent me, after changing it's message-id

Re: bounce/forward not working due to CR at line end

[Kai Harries]

David Bremner <david@tethera.net> writes:

> During off-list discussion, Kai mentioned id:87a7zsm5ol.fsf@gmail.com as
> a message that he cannot bounce (and also not forward?). For me this
> forwards fine, but does not bounce because it has a corrupted Cc header,
> and my MTA rejects it.  That doesn't sound related to the original
> problem report (nothing about line endings).

Can you please evaluate the following on your system:

  (call-process "notmuch" nil t nil "show" "--format=raw" "id:87a7zsm5ol.fsf@gmail.com")

If I do this on my system, then the text that is inserted into the
buffer has ^M (CR) at the line endings. Is this expected?

In contrast, if I evaluate the following:

  (require 'subr-x)
  (find-file (string-trim
              (shell-command-to-string "notmuch search --output=files id:87a7zsm5ol.fsf@gmail.com")))

then a buffer with the message is opened but with clean line endings (no ^M).

> I guess we still need a good test case [1], but my best guess is that
> something about Kai's settings. Kai, if you have the source unpacked
> (even if not built), try
>
>       $ ./devel/try-emacs-mua -Q
>
> and see if the problem is duplicated there.

Yes, bounce is also not working in this mode. I only evaluated 

  (setq message-send-mail-function (lambda () t))
  (notmuch-hello)

from the startup page (I have not loaded my ~/.emacs). But still the
bounce fails with:

  Search failed: "

  "

as message in the minibuffer.

> [1]: I tried both the copy of that message I already have and the copy
>      Kai sent me, after changing it's message-id

Re: bounce/forward not working due to CR at line end

[David Bremner]

Kai Harries <kai.harries@gmail.com> writes:

> David Bremner <david@tethera.net> writes:
>
>> During off-list discussion, Kai mentioned id:87a7zsm5ol.fsf@gmail.com as
>> a message that he cannot bounce (and also not forward?). For me this
>> forwards fine, but does not bounce because it has a corrupted Cc header,
>> and my MTA rejects it.  That doesn't sound related to the original
>> problem report (nothing about line endings).
>
> Can you please evaluate the following on your system:
>
>   (call-process "notmuch" nil t nil "show" "--format=raw" "id:87a7zsm5ol.fsf@gmail.com")
>
> If I do this on my system, then the text that is inserted into the
> buffer has ^M (CR) at the line endings. Is this expected?

No ^M line endings for me. Can you check the file on disk? Maybe "od -a"
lacking a better idea.

Re: bounce/forward not working due to CR at line end

[Tomi Ollila]

On Wed, Nov 15 2017, David Bremner wrote:

> Kai Harries <kai.harries@gmail.com> writes:
>
>> David Bremner <david@tethera.net> writes:
>>
>>> During off-list discussion, Kai mentioned id:87a7zsm5ol.fsf@gmail.com as
>>> a message that he cannot bounce (and also not forward?). For me this
>>> forwards fine, but does not bounce because it has a corrupted Cc header,
>>> and my MTA rejects it.  That doesn't sound related to the original
>>> problem report (nothing about line endings).
>>
>> Can you please evaluate the following on your system:
>>
>>   (call-process "notmuch" nil t nil "show" "--format=raw" "id:87a7zsm5ol.fsf@gmail.com")
>>
>> If I do this on my system, then the text that is inserted into the
>> buffer has ^M (CR) at the line endings. Is this expected?
>
> No ^M line endings for me. Can you check the file on disk? Maybe "od -a"
> lacking a better idea.

$ notmuch search --output=files id:87a7zsm5ol.fsf@gmail.com

outputs path to one file in my system. the mail content is base64-encoded
and there is no cr characters in my system 
(I used base64 -d 'edited-mail ' | tr '\r' '$' | less to look)

As Kai sent the mail, his local copy may be encoded differently...

Tomi

(Hoo, I just wondered whether there is base64-decode-region in emacs --
there is a function exactly that name... had to M-x read-only-mode (where
did M-x toggle-read-only disappear) in notmuch-raw-id:... buffer before
trying that (i.e. in emacs mua, in show buffer, first pressed 'V' to see
raw message contents.)

PS2: it is possible this mail doesn't reach Kai, as gmail is somewhat
picky from where it accepts email... we may get this fixed any century
now, though.

Re: bounce/forward not working due to CR at line end

[Kai Harries]

David Bremner <david@tethera.net> writes:

> Kai Harries <kai.harries@gmail.com> writes:
>
>> David Bremner <david@tethera.net> writes:
>>
>>> During off-list discussion, Kai mentioned id:87a7zsm5ol.fsf@gmail.com as
>>> a message that he cannot bounce (and also not forward?). For me this
>>> forwards fine, but does not bounce because it has a corrupted Cc header,
>>> and my MTA rejects it.  That doesn't sound related to the original
>>> problem report (nothing about line endings).
>>
>> Can you please evaluate the following on your system:
>>
>>   (call-process "notmuch" nil t nil "show" "--format=raw" "id:87a7zsm5ol.fsf@gmail.com")
>>
>> If I do this on my system, then the text that is inserted into the
>> buffer has ^M (CR) at the line endings. Is this expected?
>
> No ^M line endings for me. Can you check the file on disk? Maybe "od -a"
> lacking a better idea.

`od -a` does also show that the lines are ending with 'cr' and 'nl'
this is correct according to RFC 2822 [1].

According to this [2] Emacs normally converts line endings to only 'nl'
(newline) when opening a DOS file. This works on my system, if I open the
file from disk no carriage-return (^M) is shown.

Not working is inserting the output of `notmuch show` into a buffer. If
I do this, then the carriage-return (^M) is shown.

David, I assume your system is also a Linux, does

  notmuch show --format=raw id:1472041345236.7014@de.bosch.com | od -a

also show 'cr' and 'nl' as line ending?

If No, then my `notmuch show` behaves different.

If Yes, then my Emacs behaves different when inserting text with DOS
line endings into a buffer.

[1] https://tools.ietf.org/html/rfc2822#section-2.2
[2] https://www.gnu.org/software/emacs/manual/html_mono/emacs.html#Coding-Systems

Re: bounce/forward not working due to CR at line end

[David Bremner]

Kai Harries <kai.harries@gmail.com> writes:

> David Bremner <david@tethera.net> writes:
>
>> Kai Harries <kai.harries@gmail.com> writes:
>>
>>> David Bremner <david@tethera.net> writes:
>>>
>>>> During off-list discussion, Kai mentioned id:87a7zsm5ol.fsf@gmail.com as
>>>> a message that he cannot bounce (and also not forward?). For me this
>>>> forwards fine, but does not bounce because it has a corrupted Cc header,
>>>> and my MTA rejects it.  That doesn't sound related to the original
>>>> problem report (nothing about line endings).
>>>
>>> Can you please evaluate the following on your system:
>>>
>>>   (call-process "notmuch" nil t nil "show" "--format=raw" "id:87a7zsm5ol.fsf@gmail.com")
>>>
>>> If I do this on my system, then the text that is inserted into the
>>> buffer has ^M (CR) at the line endings. Is this expected?
>>
>> No ^M line endings for me. Can you check the file on disk? Maybe "od -a"
>> lacking a better idea.
>
> `od -a` does also show that the lines are ending with 'cr' and 'nl'
> this is correct according to RFC 2822 [1].

That's not so clear. RFC2822 is explicitely not about disk storage; see
the Scope paragraph of RFC2822 or RFC522. I think most peoples unix mail
delivery agents use line feeds, otherwise more people would be having
problems resending mail.  See e.g. point 2 in

     https://www.exim.org/exim-html-current/doc/html/spec_html/ch-message_processing.html

Be that as it may, apparently your system is using CRLF.

>
> According to this [2] Emacs normally converts line endings to only 'nl'
> (newline) when opening a DOS file. This works on my system, if I open the
> file from disk no carriage-return (^M) is shown.
>
> Not working is inserting the output of `notmuch show` into a buffer. If
> I do this, then the carriage-return (^M) is shown.

Yes, notmuch-show-resend-message uses notmuch-show-view-raw-message,
which intentionally turns off any conversion

>
> David, I assume your system is also a Linux, does
>
>   notmuch show --format=raw id:1472041345236.7014@de.bosch.com | od -a
>
> also show 'cr' and 'nl' as line ending?

No, I don't have carriage returns in my actual files on disk, and
consequently not in the output of notmuch show --format=raw.

I suspect notmuch is mostly fine with CRLF line endings, due to mainly
passing everything through gmime. But in the case of notmuch show
--format=raw, this is not the case.

I did confirm that adding CRLF line endings to an existing mail file
causes the symptoms you originally described.

I'm still not sure the best place to fix this; I'm pretty sure upstream
emacs won't accept your change (or equivalent) to mime-to-mml.

d

Re: bounce/forward not working due to CR at line end

[Kai Harries]

David Bremner <david@tethera.net> writes:

> Kai Harries <kai.harries@gmail.com> writes:
>
>> David Bremner <david@tethera.net> writes:
>>
>>> Kai Harries <kai.harries@gmail.com> writes:
>>>
>>>> David Bremner <david@tethera.net> writes:
>>>>
>>>>> During off-list discussion, Kai mentioned id:87a7zsm5ol.fsf@gmail.com as
>>>>> a message that he cannot bounce (and also not forward?). For me this
>>>>> forwards fine, but does not bounce because it has a corrupted Cc header,
>>>>> and my MTA rejects it.  That doesn't sound related to the original
>>>>> problem report (nothing about line endings).
>>>>
>>>> Can you please evaluate the following on your system:
>>>>
>>>>   (call-process "notmuch" nil t nil "show" "--format=raw" "id:87a7zsm5ol.fsf@gmail.com")
>>>>
>>>> If I do this on my system, then the text that is inserted into the
>>>> buffer has ^M (CR) at the line endings. Is this expected?
>>>
>>> No ^M line endings for me. Can you check the file on disk? Maybe "od -a"
>>> lacking a better idea.
>>
>> `od -a` does also show that the lines are ending with 'cr' and 'nl'
>> this is correct according to RFC 2822 [1].
>
> That's not so clear. RFC2822 is explicitely not about disk storage; see
> the Scope paragraph of RFC2822 or RFC522. I think most peoples unix mail
> delivery agents use line feeds, otherwise more people would be having
> problems resending mail.  See e.g. point 2 in
>
>      https://www.exim.org/exim-html-current/doc/html/spec_html/ch-message_processing.html
>
> Be that as it may, apparently your system is using CRLF.

I am using gmailier [1] to sync my maildir. For testing purpose I just
installed offlineimap and in contrast to gmailiier the mails on
disk have no CRLF. 

>> According to this [2] Emacs normally converts line endings to only 'nl'
>> (newline) when opening a DOS file. This works on my system, if I open the
>> file from disk no carriage-return (^M) is shown.
>>
>> Not working is inserting the output of `notmuch show` into a buffer. If
>> I do this, then the carriage-return (^M) is shown.
>
> Yes, notmuch-show-resend-message uses notmuch-show-view-raw-message,
> which intentionally turns off any conversion
>
>>
>> David, I assume your system is also a Linux, does
>>
>>   notmuch show --format=raw id:1472041345236.7014@de.bosch.com | od -a
>>
>> also show 'cr' and 'nl' as line ending?
>
> No, I don't have carriage returns in my actual files on disk, and
> consequently not in the output of notmuch show --format=raw.
>
> I suspect notmuch is mostly fine with CRLF line endings, due to mainly
> passing everything through gmime. But in the case of notmuch show
> --format=raw, this is not the case.
>
> I did confirm that adding CRLF line endings to an existing mail file
> causes the symptoms you originally described.
>
> I'm still not sure the best place to fix this; I'm pretty sure upstream
> emacs won't accept your change (or equivalent) to mime-to-mml.
>
> d

I am thinking about filing a ticket against gmailier because storing the
emails with CRLF on a *nix is at least surprising.

[1] https://github.com/gauteh/gmailieer

Re: bounce/forward not working due to CR at line end

[David Bremner]

Kai Harries <kai.harries@gmail.com> writes:

> David Bremner <david@tethera.net> writes:
>>
>> Be that as it may, apparently your system is using CRLF.
>
> I am using gmailier [1] to sync my maildir. For testing purpose I just
> installed offlineimap and in contrast to gmailiier the mails on
> disk have no CRLF. 
>
> I am thinking about filing a ticket against gmailier because storing the
> emails with CRLF on a *nix is at least surprising.

I think that's a good idea. I'd like to understand whether there's a
good reason to support CRLF line endings in maildirs, or if it's just an
oversight.  I talked a bit to the neomutt maintainer on IRC, and such
line endings cause problems for some operations in neomutt as well.

d

Re: bounce/forward not working due to CR at line end

[Gaute Hope]

Kai Harries writes on november 17, 2017 19:16:
> I am thinking about filing a ticket against gmailier because storing the
> emails with CRLF on a *nix is at least surprising.

Hi, 

this was actually fixed in commit 71ce2b2 where line-endings are
converted to \n if that is the OS line seperator. I am not modifying the
file in any other way, but I think the DOS bomb should hopefully be set
correctly already if on Linux [0].

  https://github.com/gauteh/gmailieer/blob/master/lieer/local.py#L326

GMail API supplies the content of the message base64 encoded (the whole 
message not just any MIME parts) with CRLF line endings.

Anyway, I realized I never made a release with that commit (I just did) 
so unless you were running 'master' you would have been missing out on 
that. Note that messages that were already downloaded before this change will not be converted by gmailieer, but you can do that youself. All gmailieer cares about is the file names.

[0] https://en.wikipedia.org/wiki/Byte_order_mark#UTF-8

- gaute

Re: bounce/forward not working due to CR at line end

[Kai Harries]

Gaute Hope <eg@gaute.vetsj.com> writes:

> Kai Harries writes on november 17, 2017 19:16:
>> I am thinking about filing a ticket against gmailier because storing the
>> emails with CRLF on a *nix is at least surprising.
>
> Hi, 
>
> this was actually fixed in commit 71ce2b2 where line-endings are
> converted to \n if that is the OS line seperator. I am not modifying the
> file in any other way, but I think the DOS bomb should hopefully be set
> correctly already if on Linux [0].
>
>   https://github.com/gauteh/gmailieer/blob/master/lieer/local.py#L326
>
> GMail API supplies the content of the message base64 encoded (the whole 
> message not just any MIME parts) with CRLF line endings.
>
> Anyway, I realized I never made a release with that commit (I just did) 
> so unless you were running 'master' you would have been missing out on 
> that. Note that messages that were already downloaded before this change will not be converted by gmailieer, but you can do that youself. All gmailieer cares about is the file names.
>
> [0] https://en.wikipedia.org/wiki/Byte_order_mark#UTF-8
>
> - gaute

Hi,

many thanks. My issue is indeed solved with the new release.

Kai