(emacs) Parsing problems replying to encrypted html

(emacs) Parsing problems replying to encrypted html

[Matthew Lear]

Hi,
I've noticed a problem when replying to an encrypted email when I have
html set as preferred over plain text. Looks like notmuch fails to parse
the body and the original body text in notmuch-reply just looks like this:

> [ text/html ]
> !!! Bodypart insert error: End of file during parsing !!!

A few things which might help understand what could be happening here...

notmuch from git with head @ 07b6220
gmime 2.6.20-r1 (gentoo)

(setq mm-text-html-renderer 'shr)
(setq shr-inhibit-images nil)
(setq mime-view-text/html-previewer 'shr)
(setq mm-inline-text-html-with-images t)

(set-locale-environment "en_GB.UTF-8")
(set-language-environment 'utf-8)
(set-selection-coding-system 'utf-8)
(prefer-coding-system 'utf-8)
(set-default-coding-systems 'utf-8)
(set-terminal-coding-system 'utf-8)
(set-keyboard-coding-system 'utf-8)
(setq default-buffer-file-coding-system 'utf-8)

(setq notmuch-multipart/alternative-discouraged '("text/plain"
"multipart/related"))

The message I'm replying to looks like:

User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101
 Thunderbird/38.6.0
MIME-Version: 1.0
In-Reply-To: <removed>
Content-Type: multipart/encrypted;
 protocol="application/pgp-encrypted";
 boundary="et6k1RmUSFFnhWFnwewqvineB8rlw3boE"

This is an OpenPGP/MIME encrypted message (RFC 4880 and 3156)
--et6k1RmUSFFnhWFnwewqvineB8rlw3boE
Content-Type: application/pgp-encrypted
Content-Description: PGP/MIME version identification

Version: 1

--et6k1RmUSFFnhWFnwewqvineB8rlw3boE
Content-Type: application/octet-stream; name="encrypted.asc"
Content-Description: OpenPGP encrypted message
Content-Disposition: inline; filename="encrypted.asc"

-----BEGIN PGP MESSAGE-----
Version: GnuPG v2

hQEMA41hzuYEoCb4AQgApMvj7nt6Goj1kLDUheofRWwORI/CNFgExIIAAz8ivJxH
...
...
6nVgRBQIJp+59xsmfC2axrASSAh11/ZdVl3T60LvlrsoKLkBHNAC4W+H/cyxxDk0
BnJC8+QJbgbyRBANUV4d5PCUbKcuDRW/TVQhtZY=
=rKjq
-----END PGP MESSAGE-----

--et6k1RmUSFFnhWFnwewqvineB8rlw3boE--


Notmuch shows this as:

[ multipart/encrypted ]
[ Decryption successful ]
[ Good signature by:  xxxx ]
[ application/pgp-encrypted ]
Version: 1
[ multipart/mixed ]
[ multipart/alternative ]
[ text/plain (hidden) ]
[ multipart/related ]
[ text/html (hidden) ]
[ image.jpg: image/jpeg (hidden) ]

Things of interest:

1/ I have the same encrypted email in plain text only and replying to it
works fine with notmuch.

2/ I can view the html from the email in a browser, and it looks fine.

3/ If I switch emacs to use w3m instead of shr, I don't see the 'Bodypart
insert error' fro notmuch-reply but there is no original message included
in the reply composition either.

I'm not quite sure if this is a notmuch issue as such, or attributed to my
emacs config, or something else...

I don't know exactly what is supposed to happen when replying to html
emails, but if I reply to an html email which is not encrypted, there is
no '> [ text/html ]' in the reply composition window - it's just text from
the original message that I'm replying to and nothing more.

Since the encrypted email I'm replying has a text part, can notmuch use
this and quote it when replying (obviously ensuring that the entire reply
will be encrypted).

Any thoughts?

Thanks a lot.
--  Matt

Re: (emacs) Parsing problems replying to encrypted html

[David Edmondson]

There is probably a bug here.

On Tue, Mar 01 2016, Matthew Lear wrote:
> Notmuch shows this as:
>
> [ multipart/encrypted ]
> [ Decryption successful ]
> [ Good signature by:  xxxx ]
> [ application/pgp-encrypted ]
> Version: 1
> [ multipart/mixed ]
> [ multipart/alternative ]
> [ text/plain (hidden) ]
> [ multipart/related ]
> [ text/html (hidden) ]
> [ image.jpg: image/jpeg (hidden) ]

You're saying here that you don't get shown the text/html part?

> (setq notmuch-multipart/alternative-discouraged '("text/plain"
> "multipart/related"))

I'm curious to understand why you discourage multipart/related. That's
almost always used to wrap a text/html part with an image/jpg that the
text/html part refers to by cid:.

Are things different if you discourage only text/plain?

Re: (emacs) Parsing problems replying to encrypted html

[Matthew Lear]

> There is probably a bug here.
>
> On Tue, Mar 01 2016, Matthew Lear wrote:
>> Notmuch shows this as:
>>
>> [ multipart/encrypted ]
>> [ Decryption successful ]
>> [ Good signature by:  xxxx ]
>> [ application/pgp-encrypted ]
>> Version: 1
>> [ multipart/mixed ]
>> [ multipart/alternative ]
>> [ text/plain (hidden) ]
>> [ multipart/related ]
>> [ text/html (hidden) ]
>> [ image.jpg: image/jpeg (hidden) ]
>
> You're saying here that you don't get shown the text/html part?

Apologies. No, I do get shown the text/html part. I just collapsed all the
parts here for illustration purposes.

>> (setq notmuch-multipart/alternative-discouraged '("text/plain"
>> "multipart/related"))
>
> I'm curious to understand why you discourage multipart/related. That's
> almost always used to wrap a text/html part with an image/jpg that the
> text/html part refers to by cid:.

Fair point. I think I had this discouraged as a hangover from having to
deal with malformatted PGP encrypted emails. I don't receive emails from
people which have been encrypted with that sw any more so I should
probably remove it. Even with multipart/related discouraged though, I
don't have any problem with embedded images / cid etc. All displayed ok.

> Are things different if you discourage only text/plain?

Nope. No change.

Re: (emacs) Parsing problems replying to encrypted html

[Matthew Lear]

On 01/03/16 13:59, Matthew Lear wrote:
>> There is probably a bug here.
>>
>> On Tue, Mar 01 2016, Matthew Lear wrote:
>>> Notmuch shows this as:
>>>
>>> [ multipart/encrypted ]
>>> [ Decryption successful ]
>>> [ Good signature by:  xxxx ]
>>> [ application/pgp-encrypted ]
>>> Version: 1
>>> [ multipart/mixed ]
>>> [ multipart/alternative ]
>>> [ text/plain (hidden) ]
>>> [ multipart/related ]
>>> [ text/html (hidden) ]
>>> [ image.jpg: image/jpeg (hidden) ]
>>
>> You're saying here that you don't get shown the text/html part?
> 
> Apologies. No, I do get shown the text/html part. I just collapsed all the
> parts here for illustration purposes.
> 
>>> (setq notmuch-multipart/alternative-discouraged '("text/plain"
>>> "multipart/related"))
>>
>> I'm curious to understand why you discourage multipart/related. That's
>> almost always used to wrap a text/html part with an image/jpg that the
>> text/html part refers to by cid:.
> 
> Fair point. I think I had this discouraged as a hangover from having to
> deal with malformatted PGP encrypted emails. I don't receive emails from
> people which have been encrypted with that sw any more so I should
> probably remove it. Even with multipart/related discouraged though, I
> don't have any problem with embedded images / cid etc. All displayed ok.
> 
>> Are things different if you discourage only text/plain?
> 
> Nope. No change.

This is easy to reproduce on linux with thunderbird (I'm using v38.5.0).
Edit -> Account Settings -> Composition & Addressing and check 'Compose
messages in HTML format' and send yourself a pgp encrypted email using
PGP/MIME, open it in notmuch-emacs and reply to it. You'll see the
'Bodypart insert error'.

If you need any more info let me know...

Re: (emacs) Parsing problems replying to encrypted html

[David Bremner]

[-- Attachment #1: Type: text/plain, Size: 576 bytes --]

Matthew Lear <matt@bubblegen.co.uk> writes:
>
> This is easy to reproduce on linux with thunderbird (I'm using v38.5.0).
> Edit -> Account Settings -> Composition & Addressing and check 'Compose
> messages in HTML format' and send yourself a pgp encrypted email using
> PGP/MIME, open it in notmuch-emacs and reply to it. You'll see the
> 'Bodypart insert error'.
>
> If you need any more info let me know...

Please encrypt a message to the attached gpg key (from the notmuch test
suite), and send it to the list as an attachement if you can replicate
the bug.

Cheers,

d



[-- Attachment #2: gnupg-secret-key.asc --]
[-- Type: text/plain, Size: 1957 bytes --]

-----BEGIN PGP PRIVATE KEY BLOCK-----
Version: GnuPG v1.4.10 (GNU/Linux)

lQHYBE1Mm18BBADlMsMlUeO6usp/XuulgimqlCSphHcYZvH6+Sy7u7W4TpJzid7e
jEOCrk3UZi2XMPW9+snDMhV9e28HeRz61zAO9G/gedn4N+mKOyTaELEmj9SP2IG2
ZTvdUvn30vWIHyfRIww3qEiSzNULKn6zTDfcg6BIY6ZDQ6GFSfH5EioxuQARAQAB
AAP8CM2/sS9JZWLHZHJrmsU6fygxlaarlxmyhxwLG9WZ+qUJ+xDQqWZkhssrMigP
7ZQehwLwZ7mvbvfOy/qwTPJMZjQMMuTGEzclwBTOTttSxEDS+kgYmZ05CBjIgXbo
8+k+L347l+kVRBFsi1cqOkCr+VZQwhOnbeNb8uJsUx27aAECAPD7jsBP73LRgoXQ
x650D2fzjjuomGVsIxSAPjkDRYmtorsRftaEy7DkvX3Ihu5WN6YRRjJavoL+f8ar
4escR40CAPN7NOFOGmiFZYzQcfJYQI2m7YDk4B51JxORFvLrvGT+LJnVwhtFsdGS
QnMyO4eNpKH0qeEkT5Zqha2oyAc0Yd0B/3f962YCmYlbDAvWjcbMvhV7G4DbazVp
2TNR0BhhEMiOlHuwmTO59s2iukuE5ifaVbwrj+NgpipTsaffKnhALlGjV7Q7Tm90
bXVjaCBUZXN0IFN1aXRlIDx0ZXN0X3N1aXRlQG5vdG11Y2htYWlsLm9yZz4gKElO
U0VDVVJFISmIuAQTAQIAIgUCTUybXwIbAwYLCQgHAwIGFQgCCQoLBBYCAwECHgEC
F4AACgkQbZJhLZTkY4GJFAP9E0mOw+RUGdmqbxSbd2rm0/inUSYOC0Pvt/D05pUY
xzXDAMZwsy1DWhfS7bSgdD3YTM/22b/LJ2FmbLUF1cU6cNslmdPdfHDZ5+C4qpa1
uW11y7djlBFAwxc3NBypT6Bmh/iIixrx413cw8CEU0lSZbSXUvbxZ7Rg4JYm2K6f
Y7SdAdgETUybXwEEAM74QJJWzPavquSF0IkKDFjEvI44WC1HGNsJF3JVuKv9G00P
RaHavNNcHEG8MorbfaWk7pipaEJ3+zbPKgp2vRCSJnLL6z813JIQqXJTZzu1ip63
s4icfOfXkxFJ5AaFd/pVdi+wjmEwvv+YMtJT9DyXANo6b2eQu+0bMtP4Xuv/ABEB
AAEAA/wJArUJw450070K6eoXeg22wT0iq/O0aCExSzoI5Kmywytj6KnnAmp9TftL
WVgrkQntVjrhzPsYoB40JEMrGKd7QL/6LPTNWq3eFW38PSpCiG83T0rtmKCKqHB1
Uo0B78AHfYYX7MUOEuCq2AhKTAdZukesoCpmVxcEFtjDEbOB8QIA3cvXrPJN/J2S
W61mdMT7KlaXZZD8Phs/TY2ZLAiMKUAP1dVYNDvRSDjZLvQrqKQjEAN5jM81cWAV
pvOIavLhOwIA7uMVIiaQ3vIy10C7ltiLT6YuJL/O6XDnXY/PDuXOatQahd/gmI0q
dGQLSaHIxYILPZPaW6t0orx+dduQ0ES0DQIA21nEKX0MZpYOY1eIt6OlKemsjL2a
UTdFhq/OgwVv+QRVHNdYQXmKpKDeW30lN/+BI3zyDTZjtehwKMMxNTu4AJu/iJ8E
GAECAAkFAk1Mm18CGwwACgkQbZJhLZTkY4H8kgQA4vHsTt8dlJdWJAu2SKZGOPRs
bIPu5XtRXe3RYbW5H7PqbAnrKIzlIKpkPNTwLL4wVXaF+R/aHa8ZKX3paohrPL74
qpbffwtHXyVEwyWlw3m9mgti0de1dy1YvVasCe/UQ8Frc6uNmOwtlQE20k4R4cLI
SWXT1JrwPoKh9xe++90=
=rvTR
-----END PGP PRIVATE KEY BLOCK-----

Re: (emacs) Parsing problems replying to encrypted html

[Matthew Lear]

[-- Attachment #1: Type: text/plain, Size: 683 bytes --]

On Mon, Mar 07 2016 at 12:20:37 pm GMT, David Bremner <david@tethera.net> wrote:
> Matthew Lear <matt@bubblegen.co.uk> writes:
>>
>> This is easy to reproduce on linux with thunderbird (I'm using v38.5.0).
>> Edit -> Account Settings -> Composition & Addressing and check 'Compose
>> messages in HTML format' and send yourself a pgp encrypted email using
>> PGP/MIME, open it in notmuch-emacs and reply to it. You'll see the
>> 'Bodypart insert error'.
>>
>> If you need any more info let me know...
>
> Please encrypt a message to the attached gpg key (from the notmuch test
> suite), and send it to the list as an attachement if you can replicate
> the bug.

Done. Attached here.


[-- Attachment #2: 1457383253_0.15104.sunrise,U=4513,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S --]
[-- Type: application/octet-stream, Size: 3964 bytes --]

Return-path: <matt@bubblegen.co.uk>
Envelope-to: matt@bubblegen.co.uk
Delivery-date: Mon, 07 Mar 2016 20:42:08 +0000
Received: from [212.159.9.108] (helo=avasin10.plus.net)
	  by inmx06.plus.net with esmtp (PlusNet MXCore v2.00) id 1ad1yi-0000XS-S1 
	  for matt@bubblegen.co.uk; Mon, 07 Mar 2016 20:42:08 +0000
Received: from avasout05.plus.net ([212.159.9.108])
	by avasin10.plus.net with Plusnet Cloudmark Gateway
	id T8d51s0012KrXh8018i8ZY; Mon, 07 Mar 2016 20:42:08 +0000
X-CM-Score: 0.00
X-CNFS-Analysis: v=2.1 cv=bfSWK77B c=1 sm=1 tr=0
 a=fS+5Zzvk6gANdtf7AAd0ug==:117 a=G4bc5lkgapKKm1P+Twxy3Q==:17
 a=L9H7d07YOLsA:10 a=9cW_t1CCXrUA:10 a=s5jvgZ67dGcA:10 a=7OsogOcEt9IA:10
 a=MPswtDWvrTtJOjTsUVEA:9 a=8YYFeU3zVMQA:10 a=M07ypZ-3nfwh3oN1yQkA:9
 a=4tzlBVgnB5GbwcgJivkA:9
Received: from mail.bubblegen.co.uk ([80.229.236.194])
	by avasout05 with smtp
	id T8i71s0024CLJ54018i8cK; Mon, 07 Mar 2016 20:42:08 +0000
X-CM-Score: 0.00
X-CNFS-Analysis: v=2.1 cv=Iat6Ijea c=1 sm=1 tr=0
 a=G4bc5lkgapKKm1P+Twxy3Q==:117 a=G4bc5lkgapKKm1P+Twxy3Q==:17
 a=L9H7d07YOLsA:10 a=9cW_t1CCXrUA:10 a=s5jvgZ67dGcA:10
 a=MPswtDWvrTtJOjTsUVEA:9 a=8YYFeU3zVMQA:10 a=M07ypZ-3nfwh3oN1yQkA:9
 a=4tzlBVgnB5GbwcgJivkA:9
To: test_suite@notmuchmail.org
Cc: Matthew Lear <matt@bubblegen.co.uk>
From: Matthew Lear <matt@bubblegen.co.uk>
Message-ID: <56DDE706.6060702@bubblegen.co.uk>
Date: Mon, 7 Mar 2016 20:39:34 +0000
MIME-Version: 1.0
Content-Type: multipart/encrypted;
 protocol="application/pgp-encrypted";
 boundary="avecXhbocAmvwbUWUdvP8vC7JPQwnvFMj"
X-PN-Virus-Filtered: by PlusNet MXCore (v5.00)
X-PN-Spam-Filtered: by PlusNet MXCore (v5.00)
Subject: pgp test - html msg

This is an OpenPGP/MIME encrypted message (RFC 4880 and 3156)
--avecXhbocAmvwbUWUdvP8vC7JPQwnvFMj
Content-Type: application/pgp-encrypted
Content-Description: PGP/MIME version identification

Version: 1

--avecXhbocAmvwbUWUdvP8vC7JPQwnvFMj
Content-Type: application/octet-stream; name="encrypted.asc"
Content-Description: OpenPGP encrypted message
Content-Disposition: inline; filename="encrypted.asc"

-----BEGIN PGP MESSAGE-----
Version: GnuPG v2

hIwDxE023q1UqxYBBADKLiLfANG4Zb+yjk14HbWPfIckHs3eE2WA2daXxA5v3y83
uNQ/6COdUxuG7K6Z1Z01S1OwbvQUua6g6nUIwdRSTADfTVcqSBuss6C4GumQ+6N2
EPID1CJ+oDxRKcUdHKJ9Ynad5F1XYuJvL+/PZ4sSi3hkpw8zr6kPNO73C/87LYUB
DANDVI1DELT3qwEIAKrX9/KIgHDeS0WQckLlaHwq+FZRF7vTavK1eMW68nQLuZI/
60x99nF0iqpSArrgDrc9F3ySHKhb57/RLv0G7u1KEU0Eqco/MyUs8wCcuUHlkVBm
pzCij2Gg8vM9eGTEOgkqGRz9es/s1qLy4XS0P9Hc1wUY0fCVOsERRgAKneHb2pzl
5Cizn0iIO3XezObLp0+lqDdr3NJUOFfcz/jRpklspOt6yFJikC3mYTYcE1JPfnj+
0ojyL2fTWFPmJi5ZLHZ76AnTu7icMw+0iGu+KtbThG8v50kNYdG+DsQvwy/kVr4j
4qye5qBHK8EiutoNwHMItbPt3yawMDvmTwq3ruHS6QFS+VNcNx15jCX1hN7jpgRm
0mWEvYIhWonsq28tHvZgewbjHGGngYYJ6nFyU4pkESJUrk4vXiwyCow5GVAgiRjG
dzoM0uknVIWIruYJbMNtWPU/n3gxcchm7k+s1ehye4FkWNlja3wS+lc89psRpySn
JsZ/DTwK8Uug/SyGmXNSTqAAjK3eg+UYAE6TadRGYXBnV95SFqDIBhw7ObKrfCm+
Pf8I6vz7sCSS9C2Yrl8LajW6TbycXTbiFn/5ILwGQbusaOTsswe/Xltc93V0JMX3
n0l//lEYdEyzRarEO4EMCb0HLmLTKyQwkkbVEqbERRw2m1IBqVmNnoPp3MriNwKA
6D/7jzGhknjjnECIshFRTTTtfl/0+B6LQHlGBwJJpDUnBHRCu4NlVENoQnASYLeT
WeMe37v4/SbsluXJQgYH+AL/J5P1YYPKouVFnJGOC0j6M0vh8hnTdw9bSTLjzXU1
iqm2ZFPZWJ49RmG2+/BDtGpFJE33bAoRxEsriljTlXjfpGShDivNCpc1dzsdxof0
70tX9dR+0t3EiFsWOQxpvbX3MzHcTH220bMQ8jkxKlF4BCc8lghj/ZpK7uMbGTAM
6F+afsQU+HQQZF8sQ3Ny4OIJ8c12RHBHv5t4ZwYTD7JY9QcMgWAMseqaN864pDUL
+931yP63xpSOZ87T7RcXwM1bpWuPfgFFju6uNrFUkVq4wEE3pPeoV+++tnZEotzh
N4dx45H7+d4rc24pArh6P1cvXMeoI2pa9o4XapWOM3MJtL3eApDxTECDCw+532ym
QaME7ZtdKbATShZkzEXoLfBIPBCYA/RopDKDcv4/bdA0XhSFz0/fWDUTefR3braa
73UWRxlRDqJsuTfusbR+bj7R/1gcEMxADAXFwowjBF8DLf0XvVHXqXTu2LIBTsl2
wJEy+fEQ1YBTuZ0w1Wl85nyo6QRsRPSnYpD7SS12j2zevYTbTQYoMCi4TMBNJ9Hh
ZbsCWmj80ffru8fI4+vIfyOzxhZGy2Ly3gjdnq65vx2byQLzfU1ykOh8WpJ3KFL7
wM7ZUDU61tx1NXISi/+rpAGNAcbu/CxH/6gwL5dT5LzISXXn+nN9I+BR+mm1wznc
45hufJnCEL+TesWKDzWlUSuRlQ/T12gxLhwH9aP5+7yldyAs1o6nJxer4sxd0kCr
ItH6NPGOa1HbtDcbsuxcp0JBB8P73xqzQNPWyRtB
=3FaG
-----END PGP MESSAGE-----

--avecXhbocAmvwbUWUdvP8vC7JPQwnvFMj--

Re: (emacs) Parsing problems replying to encrypted html

[David Bremner]

Matthew Lear <matt@bubblegen.co.uk> writes:

>>
>> Please encrypt a message to the attached gpg key (from the notmuch test
>> suite), and send it to the list as an attachement if you can replicate
>> the bug.
>
> Done. Attached here.
>
> [ 1457383253_0.15104.sunrise,U=4513,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S: application/octet-stream ]

Thanks for the test case. I can duplicate a (the?) bug as
follows. Replying to that message from notmuch-show mode works, but
replying from notmuch-search mode yields

> !!! Bodypart insert error: Wrong type argument: number-or-marker-p, nil !!!
> [ application/pgp-encrypted ]
> Version: 1
> [ encrypted.asc: application/octet-stream (as text/plain) ]
> !!! Bodypart insert error: Internal error: No :content from ("show" "--format=sexp" "--include-html" "--part=3" "id:56DDE706.6060702@bubblegen.co.uk") !!!

Followed by the ciphertext.
> -----BEGIN PGP MESSAGE-----

If anyone wants to play with this, you just need to import the notmuch
test key into your keyring, something like

gpg --import path/to/notmuch-source/test/gnupg-secret-key.asc

What's odd is that the notmuch command in the error message works on the
command line.

Re: (emacs) Parsing problems replying to encrypted html

[Matthew Lear]

> Thanks for the test case. I can duplicate a (the?) bug as
> follows. Replying to that message from notmuch-show mode works, but
> replying from notmuch-search mode yields
>
>> !!! Bodypart insert error: Wrong type argument: number-or-marker-p, nil
>> !!!
>> [ application/pgp-encrypted ]
>> Version: 1
>> [ encrypted.asc: application/octet-stream (as text/plain) ]
>> !!! Bodypart insert error: Internal error: No :content from ("show"
>> "--format=sexp" "--include-html" "--part=3"
>> "id:56DDE706.6060702@bubblegen.co.uk") !!!
>
> Followed by the ciphertext.
>> -----BEGIN PGP MESSAGE-----
>
> If anyone wants to play with this, you just need to import the notmuch
> test key into your keyring, something like
>
> gpg --import path/to/notmuch-source/test/gnupg-secret-key.asc
>
> What's odd is that the notmuch command in the error message works on the
> command line.
>

Hmm. In emacs, if I try to reply to the message from notmuch-search, I get
no error but there is no quoted text from the original message in the
reply body and all I see is the 'on <date>, <person> wrote:' citation.
Replying from notmuch-show gives me the error I mentioned previously.

Re: (emacs) Parsing problems replying to encrypted html

[David Bremner]

Matthew Lear <matt@bubblegen.co.uk> writes:
>
> Hmm. In emacs, if I try to reply to the message from notmuch-search, I get
> no error but there is no quoted text from the original message in the
> reply body and all I see is the 'on <date>, <person> wrote:' citation.
> Replying from notmuch-show gives me the error I mentioned previously.

I suppose the main difference is the setting of
notmuch-multipart/alternative-discouraged. If I copy your setting I also
get an error replying in notmuch-show mode.

Re: (emacs) Parsing problems replying to encrypted html

[Tomi Ollila]

On Tue, Mar 08 2016, David Bremner wrote:

> [ text/plain ]
> Matthew Lear <matt@bubblegen.co.uk> writes:
>
>>>
>>> Please encrypt a message to the attached gpg key (from the notmuch test
>>> suite), and send it to the list as an attachement if you can replicate
>>> the bug.
>>
>> Done. Attached here.
>>
>> [ 1457383253_0.15104.sunrise,U=4513,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S: application/octet-stream ]
>
> Thanks for the test case. I can duplicate a (the?) bug as
> follows. Replying to that message from notmuch-show mode works, but
> replying from notmuch-search mode yields
>
>> !!! Bodypart insert error: Wrong type argument: number-or-marker-p, nil !!!
>> [ application/pgp-encrypted ]
>> Version: 1
>> [ encrypted.asc: application/octet-stream (as text/plain) ]
>> !!! Bodypart insert error: Internal error: No :content from ("show" "--format=sexp" "--include-html" "--part=3" "id:56DDE706.6060702@bubblegen.co.uk") !!!
>
> Followed by the ciphertext.
>> -----BEGIN PGP MESSAGE-----
>
> If anyone wants to play with this, you just need to import the notmuch
> test key into your keyring, something like
>
> gpg --import path/to/notmuch-source/test/gnupg-secret-key.asc
>
> What's odd is that the notmuch command in the error message works on the
> command line.

I got these to my log:

when replying from search:

2016-03-08 (Tue) 14:16:01: reply --format=sexp --format-version=1 --reply-to=sender thread:0000000000004a6f
2016-03-08 (Tue) 14:16:01: show --format=raw --part=2 id:56DDE706.6060702@bubblegen.co.uk
2016-03-08 (Tue) 14:16:01: show --format=sexp --include-html --part=3 id:56DDE706.6060702@bubblegen.co.uk
2016-03-08 (Tue) 14:16:01: show --format=raw --part=3 id:56DDE706.6060702@bubblegen.co.uk

when replying from show:

2016-03-08 (Tue) 14:16:37: reply --format=sexp --format-version=1 --decrypt --reply-to=sender id:56DDE706.6060702@bubblegen.co.uk
2016-03-08 (Tue) 14:16:37: show --format=raw --part=2 id:56DDE706.6060702@bubblegen.co.uk

there's no --decrypt when from search ... and interestingly there are other
differences, too...

Tomi

Re: (emacs) Parsing problems replying to encrypted html

[Matthew Lear]

>> Hmm. In emacs, if I try to reply to the message from notmuch-search, I
>> get
>> no error but there is no quoted text from the original message in the
>> reply body and all I see is the 'on <date>, <person> wrote:' citation.
>> Replying from notmuch-show gives me the error I mentioned previously.
>
> I suppose the main difference is the setting of
> notmuch-multipart/alternative-discouraged. If I copy your setting I also
> get an error replying in notmuch-show mode.
>

I know this was mentioned earlier but just to avoid any doubt, my
discourage setting is currently:
(setq notmuch-multipart/alternative-discouraged '("text/plain"))

I removed the "multipart/related". (I agree it's probably not wise to
discourage it).

Re: (emacs) Parsing problems replying to encrypted html

[Tomi Ollila]

On Tue, Mar 08 2016, Tomi Ollila wrote:

> [ text/plain ]
> On Tue, Mar 08 2016, David Bremner wrote:
>
>> [ text/plain ]
>> Matthew Lear <matt@bubblegen.co.uk> writes:
>>
>>
>> Thanks for the test case. I can duplicate a (the?) bug as
>> follows. Replying to that message from notmuch-show mode works, but
>> replying from notmuch-search mode yields
>>
>>> !!! Bodypart insert error: Wrong type argument: number-or-marker-p, nil !!!
>>> [ application/pgp-encrypted ]
>>> Version: 1
>>> [ encrypted.asc: application/octet-stream (as text/plain) ]
>>> !!! Bodypart insert error: Internal error: No :content from ("show" "--format=sexp" "--include-html" "--part=3" "id:56DDE706.6060702@bubblegen.co.uk") !!!
>>
>> Followed by the ciphertext.
>>> -----BEGIN PGP MESSAGE-----
>>
>> If anyone wants to play with this, you just need to import the notmuch
>> test key into your keyring, something like
>>
>> gpg --import path/to/notmuch-source/test/gnupg-secret-key.asc
>>
>> What's odd is that the notmuch command in the error message works on the
>> command line.
>
> I got these to my log:
>
> when replying from search:
>
> 2016-03-08 (Tue) 14:16:01: reply --format=sexp --format-version=1 --reply-to=sender thread:0000000000004a6f
> 2016-03-08 (Tue) 14:16:01: show --format=raw --part=2 id:56DDE706.6060702@bubblegen.co.uk
> 2016-03-08 (Tue) 14:16:01: show --format=sexp --include-html --part=3 id:56DDE706.6060702@bubblegen.co.uk

The above line returns (same as in latest !!! bodypart insert error !!!)

(:id 3 :content-type "application/octet-stream" :filename "encrypted.asc" :content-length 1875)

-- there is no content to be included from that line :/ (and wat is this
   "encrypted.asc" anyway ?)

> when replying from show:
>
> 2016-03-08 (Tue) 14:16:37: reply --format=sexp --format-version=1 --decrypt --reply-to=sender id:56DDE706.6060702@bubblegen.co.uk

the --decrypt here provided the content (from part=3) in this case.


Tomi

Re: (emacs) Parsing problems replying to encrypted html

[Matthew Lear]

> On Tue, Mar 08 2016, Tomi Ollila wrote:
>
>> [ text/plain ]
>> On Tue, Mar 08 2016, David Bremner wrote:
>>
>>> [ text/plain ]
>>> Matthew Lear <matt@bubblegen.co.uk> writes:
>>>
>>>
>>> Thanks for the test case. I can duplicate a (the?) bug as
>>> follows. Replying to that message from notmuch-show mode works, but
>>> replying from notmuch-search mode yields
>>>
>>>> !!! Bodypart insert error: Wrong type argument: number-or-marker-p,
>>>> nil !!!
>>>> [ application/pgp-encrypted ]
>>>> Version: 1
>>>> [ encrypted.asc: application/octet-stream (as text/plain) ]
>>>> !!! Bodypart insert error: Internal error: No :content from ("show"
>>>> "--format=sexp" "--include-html" "--part=3"
>>>> "id:56DDE706.6060702@bubblegen.co.uk") !!!
>>>
>>> Followed by the ciphertext.
>>>> -----BEGIN PGP MESSAGE-----
>>>
>>> If anyone wants to play with this, you just need to import the notmuch
>>> test key into your keyring, something like
>>>
>>> gpg --import path/to/notmuch-source/test/gnupg-secret-key.asc
>>>
>>> What's odd is that the notmuch command in the error message works on
>>> the
>>> command line.
>>
>> I got these to my log:
>>
>> when replying from search:
>>
>> 2016-03-08 (Tue) 14:16:01: reply --format=sexp --format-version=1
>> --reply-to=sender thread:0000000000004a6f
>> 2016-03-08 (Tue) 14:16:01: show --format=raw --part=2
>> id:56DDE706.6060702@bubblegen.co.uk
>> 2016-03-08 (Tue) 14:16:01: show --format=sexp --include-html --part=3
>> id:56DDE706.6060702@bubblegen.co.uk
>
> The above line returns (same as in latest !!! bodypart insert error !!!)
>
> (:id 3 :content-type "application/octet-stream" :filename "encrypted.asc"
> :content-length 1875)
>
> -- there is no content to be included from that line :/ (and wat is this
>    "encrypted.asc" anyway ?)

encrypted.asc is what Enigmail uses when sending using PGP/MIME. I think
it wraps up the message body in this file (armor encrypted) and marks it
with disposition inline. Enigmail also sends PGP/MIME version
identification information in another MIME part
(application/pgp-encrypted) - so you have two MIME parts per PGP/MIME
encrypted message.

You can view all headers in the test message if you're in notmuch-show by
using V. You'll see something like:

Content-Type: multipart/encrypted;
 protocol="application/pgp-encrypted";
 boundary="xxx"

This is an OpenPGP/MIME encrypted message (RFC 4880 and 3156)
--xxx
Content-Type: application/pgp-encrypted
Content-Description: PGP/MIME version identification

Version: 1

--xxx
Content-Type: application/octet-stream; name="encrypted.asc"
Content-Description: OpenPGP encrypted message
Content-Disposition: inline; filename="encrypted.asc"

-----BEGIN PGP MESSAGE-----
Version: GnuPG v2
...
-----END PGP MESSAGE-----

--xxx--

Re: (emacs) Parsing problems replying to encrypted html

[David Bremner]

Tomi Ollila <tomi.ollila@iki.fi> writes:

> 2016-03-08 (Tue) 14:16:01: reply --format=sexp --format-version=1 --reply-to=sender thread:0000000000004a6f
> 2016-03-08 (Tue) 14:16:01: show --format=raw --part=2 id:56DDE706.6060702@bubblegen.co.uk
> 2016-03-08 (Tue) 14:16:01: show --format=sexp --include-html --part=3 id:56DDE706.6060702@bubblegen.co.uk
> 2016-03-08 (Tue) 14:16:01: show --format=raw --part=3 id:56DDE706.6060702@bubblegen.co.uk
>
> when replying from show:
>
> 2016-03-08 (Tue) 14:16:37: reply --format=sexp --format-version=1 --decrypt --reply-to=sender id:56DDE706.6060702@bubblegen.co.uk
> 2016-03-08 (Tue) 14:16:37: show --format=raw --part=2 id:56DDE706.6060702@bubblegen.co.uk
>
> there's no --decrypt when from search ... and interestingly there are other
> differences, too...

That's useful log info, thanks Tomi. With dme's series
id:1457457179-4707-1-git-send-email-dme@dme.org I get the ciphertext
when replying from search-mode, which suggests there are at least two
bugs here.

d

Re: (emacs) Parsing problems replying to encrypted html

[David Edmondson]

On Thu, Mar 10 2016, David Bremner wrote:
> Tomi Ollila <tomi.ollila@iki.fi> writes:
>
>> 2016-03-08 (Tue) 14:16:01: reply --format=sexp --format-version=1 --reply-to=sender thread:0000000000004a6f
>> 2016-03-08 (Tue) 14:16:01: show --format=raw --part=2 id:56DDE706.6060702@bubblegen.co.uk
>> 2016-03-08 (Tue) 14:16:01: show --format=sexp --include-html --part=3 id:56DDE706.6060702@bubblegen.co.uk
>> 2016-03-08 (Tue) 14:16:01: show --format=raw --part=3 id:56DDE706.6060702@bubblegen.co.uk
>>
>> when replying from show:
>>
>> 2016-03-08 (Tue) 14:16:37: reply --format=sexp --format-version=1 --decrypt --reply-to=sender id:56DDE706.6060702@bubblegen.co.uk
>> 2016-03-08 (Tue) 14:16:37: show --format=raw --part=2 id:56DDE706.6060702@bubblegen.co.uk
>>
>> there's no --decrypt when from search ... and interestingly there are other
>> differences, too...
>
> That's useful log info, thanks Tomi. With dme's series
> id:1457457179-4707-1-git-send-email-dme@dme.org I get the ciphertext
> when replying from search-mode, which suggests there are at least two
> bugs here.

That's related to "notmuch reply --decrypt thread:something" I believe,
which didn't seem to decrypt when I did a quick test (but I wasn't
concentrating on that aspect of things).

Re: (emacs) Parsing problems replying to encrypted html

[David Bremner]

Tomi Ollila <tomi.ollila@iki.fi> writes:

> 2016-03-08 (Tue) 14:16:01: reply --format=sexp --format-version=1 --reply-to=sender thread:0000000000004a6f
> 2016-03-08 (Tue) 14:16:01: show --format=raw --part=2 id:56DDE706.6060702@bubblegen.co.uk
> 2016-03-08 (Tue) 14:16:01: show --format=sexp --include-html --part=3 id:56DDE706.6060702@bubblegen.co.uk
> 2016-03-08 (Tue) 14:16:01: show --format=raw --part=3 id:56DDE706.6060702@bubblegen.co.uk
>

I think the problem is in the following let form, which uses
notmuch-show-process-crypto. That isn't set in search mode. I'm unclear
on what the right solution is here; we could check
notmuch-crypto-process-mime, but I'm not sure how this would interact
with the handling of prefix arguments (for temporary toggling).


(defun notmuch-mua-reply (query-string &optional sender reply-all)
  (let ((args '("reply" "--format=sexp" "--format-version=1"))
	(process-crypto notmuch-show-process-crypto)
	reply
	original)

Re: (emacs) Parsing problems replying to encrypted html

[David Edmondson]

On Sat, Mar 12 2016, David Bremner wrote:
> Tomi Ollila <tomi.ollila@iki.fi> writes:
>
>> 2016-03-08 (Tue) 14:16:01: reply --format=sexp --format-version=1 --reply-to=sender thread:0000000000004a6f
>> 2016-03-08 (Tue) 14:16:01: show --format=raw --part=2 id:56DDE706.6060702@bubblegen.co.uk
>> 2016-03-08 (Tue) 14:16:01: show --format=sexp --include-html --part=3 id:56DDE706.6060702@bubblegen.co.uk
>> 2016-03-08 (Tue) 14:16:01: show --format=raw --part=3 id:56DDE706.6060702@bubblegen.co.uk
>>
>
> I think the problem is in the following let form, which uses
> notmuch-show-process-crypto. That isn't set in search mode. I'm unclear
> on what the right solution is here; we could check
> notmuch-crypto-process-mime, but I'm not sure how this would interact
> with the handling of prefix arguments (for temporary toggling).

Is "reply" from search mode a common pattern of use? Currently it seems
generally un-useful (the thread has to contain just a single message -
not just a single matching message).

> (defun notmuch-mua-reply (query-string &optional sender reply-all)
>   (let ((args '("reply" "--format=sexp" "--format-version=1"))
> 	(process-crypto notmuch-show-process-crypto)
> 	reply
> 	original)

Re: (emacs) Parsing problems replying to encrypted html

[Jani Nikula]

On Sat, 12 Mar 2016, David Edmondson <dme@dme.org> wrote:
> Is "reply" from search mode a common pattern of use? Currently it seems
> generally un-useful (the thread has to contain just a single message -
> not just a single matching message).

I wish we could make it useful again. It was one of the neat features of
notmuch to be able to reply to multiple messages at once. Either all
messages in a thread from search view, or all expanded messages in a
thread from show view. We threw this out to make it easier to improve
the normal reply code.

As to whether it was a common pattern or not, I don't recall a single
complaint after the feature was dropped.

BR,
Jani.

Re: (emacs) Parsing problems replying to encrypted html

[David Edmondson]

On Sun, Mar 13 2016, Jani Nikula wrote:
> On Sat, 12 Mar 2016, David Edmondson <dme@dme.org> wrote:
>> Is "reply" from search mode a common pattern of use? Currently it seems
>> generally un-useful (the thread has to contain just a single message -
>> not just a single matching message).
>
> I wish we could make it useful again. It was one of the neat features of
> notmuch to be able to reply to multiple messages at once. Either all
> messages in a thread from search view, or all expanded messages in a
> thread from show view. We threw this out to make it easier to improve
> the normal reply code.

I had no idea that it previously worked as you describe. I will fix it.

Re: (emacs) Parsing problems replying to encrypted html

[Matthew Lear]

On Sat, Mar 12 2016 at  2:37:44 pm GMT, David Edmondson <dme@dme.org> wrote:
> On Sat, Mar 12 2016, David Bremner wrote:
>> Tomi Ollila <tomi.ollila@iki.fi> writes:
>>
>>> 2016-03-08 (Tue) 14:16:01: reply --format=sexp --format-version=1 --reply-to=sender thread:0000000000004a6f
>>> 2016-03-08 (Tue) 14:16:01: show --format=raw --part=2 id:56DDE706.6060702@bubblegen.co.uk
>>> 2016-03-08 (Tue) 14:16:01: show --format=sexp --include-html --part=3 id:56DDE706.6060702@bubblegen.co.uk
>>> 2016-03-08 (Tue) 14:16:01: show --format=raw --part=3 id:56DDE706.6060702@bubblegen.co.uk
>>>
>>
>> I think the problem is in the following let form, which uses
>> notmuch-show-process-crypto. That isn't set in search mode. I'm unclear
>> on what the right solution is here; we could check
>> notmuch-crypto-process-mime, but I'm not sure how this would interact
>> with the handling of prefix arguments (for temporary toggling).
>
> Is "reply" from search mode a common pattern of use? Currently it seems
> generally un-useful (the thread has to contain just a single message -
> not just a single matching message).
>
>> (defun notmuch-mua-reply (query-string &optional sender reply-all)
>>   (let ((args '("reply" "--format=sexp" "--format-version=1"))
>> 	(process-crypto notmuch-show-process-crypto)
>> 	reply
>> 	original)

Hi. I just wondered if anyone had managed to make any progress on this?

Re: (emacs) Parsing problems replying to encrypted html

[David Bremner]

Matthew Lear <matt@bubblegen.co.uk> writes:

> On Sat, Mar 12 2016 at  2:37:44 pm GMT, David Edmondson <dme@dme.org> wrote:
>> On Sat, Mar 12 2016, David Bremner wrote:
>>> Tomi Ollila <tomi.ollila@iki.fi> writes:
>>>
>>>> 2016-03-08 (Tue) 14:16:01: reply --format=sexp --format-version=1 --reply-to=sender thread:0000000000004a6f
>>>> 2016-03-08 (Tue) 14:16:01: show --format=raw --part=2 id:56DDE706.6060702@bubblegen.co.uk
>>>> 2016-03-08 (Tue) 14:16:01: show --format=sexp --include-html --part=3 id:56DDE706.6060702@bubblegen.co.uk
>>>> 2016-03-08 (Tue) 14:16:01: show --format=raw --part=3 id:56DDE706.6060702@bubblegen.co.uk
>>>>
>>>
>>> I think the problem is in the following let form, which uses
>>> notmuch-show-process-crypto. That isn't set in search mode. I'm unclear
>>> on what the right solution is here; we could check
>>> notmuch-crypto-process-mime, but I'm not sure how this would interact
>>> with the handling of prefix arguments (for temporary toggling).
>>
>> Is "reply" from search mode a common pattern of use? Currently it seems
>> generally un-useful (the thread has to contain just a single message -
>> not just a single matching message).
>>
>>> (defun notmuch-mua-reply (query-string &optional sender reply-all)
>>>   (let ((args '("reply" "--format=sexp" "--format-version=1"))
>>> 	(process-crypto notmuch-show-process-crypto)
>>> 	reply
>>> 	original)
>
> Hi. I just wondered if anyone had managed to make any progress on this?

Hi Matthew;

I'm not sure if you noticed, but there are patches in master now (and
will be in a release pretty soon) that handle the part of the problem
related to encrypted html, but nothing so far for the problem(s) of
replying from search view.

d

Re: (emacs) Parsing problems replying to encrypted html

[Matthew Lear]

> Matthew Lear <matt@bubblegen.co.uk> writes:
>
>> On Sat, Mar 12 2016 at  2:37:44 pm GMT, David Edmondson <dme@dme.org>
>> wrote:
>>> On Sat, Mar 12 2016, David Bremner wrote:
>>>> Tomi Ollila <tomi.ollila@iki.fi> writes:
>>>>
>>>>> 2016-03-08 (Tue) 14:16:01: reply --format=sexp --format-version=1
>>>>> --reply-to=sender thread:0000000000004a6f
>>>>> 2016-03-08 (Tue) 14:16:01: show --format=raw --part=2
>>>>> id:56DDE706.6060702@bubblegen.co.uk
>>>>> 2016-03-08 (Tue) 14:16:01: show --format=sexp --include-html --part=3
>>>>> id:56DDE706.6060702@bubblegen.co.uk
>>>>> 2016-03-08 (Tue) 14:16:01: show --format=raw --part=3
>>>>> id:56DDE706.6060702@bubblegen.co.uk
>>>>>
>>>>
>>>> I think the problem is in the following let form, which uses
>>>> notmuch-show-process-crypto. That isn't set in search mode. I'm
>>>> unclear
>>>> on what the right solution is here; we could check
>>>> notmuch-crypto-process-mime, but I'm not sure how this would interact
>>>> with the handling of prefix arguments (for temporary toggling).
>>>
>>> Is "reply" from search mode a common pattern of use? Currently it seems
>>> generally un-useful (the thread has to contain just a single message -
>>> not just a single matching message).
>>>
>>>> (defun notmuch-mua-reply (query-string &optional sender reply-all)
>>>>   (let ((args '("reply" "--format=sexp" "--format-version=1"))
>>>> 	(process-crypto notmuch-show-process-crypto)
>>>> 	reply
>>>> 	original)
>>
>> Hi. I just wondered if anyone had managed to make any progress on this?
>
> Hi Matthew;
>
> I'm not sure if you noticed, but there are patches in master now (and
> will be in a release pretty soon) that handle the part of the problem
> related to encrypted html, but nothing so far for the problem(s) of
> replying from search view.

Thanks David. I've just pulled and rebuilt and can see a difference. When
replying from show view, I see no quoted text from the original message,
but when I reply from search view I see the inline PGP ciphertext quoted
in the reply. I can understand the latter but not the former - this still
looks like a bug, especially since if I reply from show view to a plain
text encrypted message, I do see the original message quoted in the reply.
--  Matt

Re: (emacs) Parsing problems replying to encrypted html

[David Bremner]

Matthew Lear <matt@bubblegen.co.uk> writes:

> Thanks David. I've just pulled and rebuilt and can see a difference. When
> replying from show view, I see no quoted text from the original message,
> but when I reply from search view I see the inline PGP ciphertext quoted
> in the reply. I can understand the latter but not the former - this still
> looks like a bug, especially since if I reply from show view to a plain
> text encrypted message, I do see the original message quoted in the reply.
> --  Matt

Hmm. I can't duplicate the problem replying from show mode with the test
message you provided.  Is the appropriate key in your gpg keyring?

d

Re: (emacs) Parsing problems replying to encrypted html

[Matthew Lear]

> Matthew Lear <matt@bubblegen.co.uk> writes:
>
>> Thanks David. I've just pulled and rebuilt and can see a difference.
>> When
>> replying from show view, I see no quoted text from the original message,
>> but when I reply from search view I see the inline PGP ciphertext quoted
>> in the reply. I can understand the latter but not the former - this
>> still
>> looks like a bug, especially since if I reply from show view to a plain
>> text encrypted message, I do see the original message quoted in the
>> reply.
>> --  Matt
>
> Hmm. I can't duplicate the problem replying from show mode with the test
> message you provided.  Is the appropriate key in your gpg keyring?

Yes it is. I get the gpg gui prompt when opening the encrypted email and
can decrypt it prior to show. As I suspected earlier in the thread, I
wondered if this was related to some emacs config settings/options that I
have. I can't immediately think what they would be though, and why they
would trigger this particular issue for html only. Strange...

Re: (emacs) Parsing problems replying to encrypted html

[Matthew Lear]

>> Hmm. I can't duplicate the problem replying from show mode with the test
>> message you provided.  Is the appropriate key in your gpg keyring?
>
> Yes it is. I get the gpg gui prompt when opening the encrypted email and
> can decrypt it prior to show. As I suspected earlier in the thread, I
> wondered if this was related to some emacs config settings/options that I
> have. I can't immediately think what they would be though, and why they
> would trigger this particular issue for html only. Strange...

Do you have the following in your .emacs ?

(setq notmuch-multipart/alternative-discouraged '("text/plain"))

I think that's the work/not-work switch. I'm not sure if it makes sense
or not in terms of seeing the behaviour I do. I discard plain text
because I get a ton of html email at work with various embedded images
etc and I want to be able to read that (which works fine). But I can see
that to quote a reply from an encrypted message, you need to quote the
original text which will be plain text.
What do you think?
--  Matt

Re: (emacs) Parsing problems replying to encrypted html

[David Bremner]

Matthew Lear <matt@bubblegen.co.uk> writes:

>>> Hmm. I can't duplicate the problem replying from show mode with the test
>>> message you provided.  Is the appropriate key in your gpg keyring?
>>
>> Yes it is. I get the gpg gui prompt when opening the encrypted email and
>> can decrypt it prior to show. As I suspected earlier in the thread, I
>> wondered if this was related to some emacs config settings/options that I
>> have. I can't immediately think what they would be though, and why they
>> would trigger this particular issue for html only. Strange...
>
> Do you have the following in your .emacs ?
>
> (setq notmuch-multipart/alternative-discouraged '("text/plain"))
>
> I think that's the work/not-work switch. I'm not sure if it makes sense
> or not in terms of seeing the behaviour I do.

OK, with that setting I can duplicate the problem replying to encrypted
HTML mail.

It's also true that notmuch-reply doesn't include the content in the
html version of the multipart/alternative. Unlike notmuch show, it
doesn't have the ability to send parts later, so I guess that's a
problem there.

The following patch seems to fix it for me. What I haven't worked out is
if it will break other things. Perhaps David E or Mark (in copy) can
comment on whether this is a sensible change, given that we now give the
user control of showing the part on the emacs side.

diff --git a/notmuch-reply.c b/notmuch-reply.c
index 3c6d685..cd88d79 100644
--- a/notmuch-reply.c
+++ b/notmuch-reply.c
@@ -690,7 +690,7 @@ notmuch_reply_format_sprinter(void *ctx,
 
     /* Start the original */
     sp->map_key (sp, "original");
-    format_part_sprinter (ctx, sp, node, TRUE, TRUE, FALSE);
+    format_part_sprinter (ctx, sp, node, TRUE, TRUE, TRUE);
 
     /* End */
     sp->end (sp);

Re: (emacs) Parsing problems replying to encrypted html

[David Edmondson]

On Sat, Apr 02 2016, David Bremner <david@tethera.net> wrote:
> Matthew Lear <matt@bubblegen.co.uk> writes:
>
>>>> Hmm. I can't duplicate the problem replying from show mode with the test
>>>> message you provided.  Is the appropriate key in your gpg keyring?
>>>
>>> Yes it is. I get the gpg gui prompt when opening the encrypted email and
>>> can decrypt it prior to show. As I suspected earlier in the thread, I
>>> wondered if this was related to some emacs config settings/options that I
>>> have. I can't immediately think what they would be though, and why they
>>> would trigger this particular issue for html only. Strange...
>>
>> Do you have the following in your .emacs ?
>>
>> (setq notmuch-multipart/alternative-discouraged '("text/plain"))
>>
>> I think that's the work/not-work switch. I'm not sure if it makes sense
>> or not in terms of seeing the behaviour I do.
>
> OK, with that setting I can duplicate the problem replying to encrypted
> HTML mail.
>
> It's also true that notmuch-reply doesn't include the content in the
> html version of the multipart/alternative. Unlike notmuch show, it
> doesn't have the ability to send parts later, so I guess that's a
> problem there.
>
> The following patch seems to fix it for me. What I haven't worked out is
> if it will break other things. Perhaps David E or Mark (in copy) can
> comment on whether this is a sensible change, given that we now give the
> user control of showing the part on the emacs side.

As an alternative, could you try the patch in
id:1459708823-1852-1-git-send-email-dme@dme.org? I think that this is
really a failure on the emacs side rather than in notmuch proper.

> diff --git a/notmuch-reply.c b/notmuch-reply.c
> index 3c6d685..cd88d79 100644
> --- a/notmuch-reply.c
> +++ b/notmuch-reply.c
> @@ -690,7 +690,7 @@ notmuch_reply_format_sprinter(void *ctx,
>  
>      /* Start the original */
>      sp->map_key (sp, "original");
> -    format_part_sprinter (ctx, sp, node, TRUE, TRUE, FALSE);
> +    format_part_sprinter (ctx, sp, node, TRUE, TRUE, TRUE);
>  
>      /* End */
>      sp->end (sp);

Re: (emacs) Parsing problems replying to encrypted html

[Matthew Lear]

[-- Attachment #1: Type: text/plain, Size: 1822 bytes --]

Hi David,

> OK, with that setting I can duplicate the problem replying to encrypted
> HTML mail.
>
> It's also true that notmuch-reply doesn't include the content in the
> html version of the multipart/alternative. Unlike notmuch show, it
> doesn't have the ability to send parts later, so I guess that's a
> problem there.
>
> The following patch seems to fix it for me. What I haven't worked out is
> if it will break other things. Perhaps David E or Mark (in copy) can
> comment on whether this is a sensible change, given that we now give the
> user control of showing the part on the emacs side.
>
> diff --git a/notmuch-reply.c b/notmuch-reply.c
> index 3c6d685..cd88d79 100644
> --- a/notmuch-reply.c
> +++ b/notmuch-reply.c
> @@ -690,7 +690,7 @@ notmuch_reply_format_sprinter(void *ctx,
>
>      /* Start the original */
>      sp->map_key (sp, "original");
> -    format_part_sprinter (ctx, sp, node, TRUE, TRUE, FALSE);
> +    format_part_sprinter (ctx, sp, node, TRUE, TRUE, TRUE);
>
>      /* End */
>      sp->end (sp);

I could have sworn that I replied to this thread months ago but can't find
a record it in the archives... I think I also missed David E's reply to
yours mentioning the patch from id:1459708823-1852-1-git-send-email-dme at
dme.org, so I'm following up...

With David E's patch, the decryption problem I reported is resolved.
However, I notice that this change hasn't yet been committed / pushed. Is
there a particular reason for this?

I'm currently maintaining a version of this patch compatible with the
current master which makes this change to notmuch-mua.el
notmuch-mua-reply() by adding (notmuch-show-process-crypto process-crypto)
before doing the show. I've attached that patch here.

I just wondered if we were going to see this pushed to master any time soon?

Cheers,
--  Matt

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: fix-encrypted-html.patch --]
[-- Type: text/x-patch; name="fix-encrypted-html.patch", Size: 942 bytes --]

diff --git a/emacs/notmuch-mua.el b/emacs/notmuch-mua.el
index 0445975..0dcdb06 100644
--- a/emacs/notmuch-mua.el
+++ b/emacs/notmuch-mua.el
@@ -250,7 +250,11 @@ mutiple parts get a header."
 		       ;; Don't omit long parts.
 		       (notmuch-show-max-text-part-size 0)
 		       ;; Insert headers for parts as appropriate for replying.
-		       (notmuch-show-insert-header-p-function notmuch-mua-reply-insert-header-p-function)
+                       (notmuch-show-insert-header-p-function notmuch-mua-reply-insert-header-p-function)
+                       ;; Ensure that any encrypted parts are
+                       ;; decrypted during the generation of the reply
+                       ;; text.
+                       (notmuch-show-process-crypto process-crypto)
 		       ;; Don't indent multipart sub-parts.
 		       (notmuch-show-indent-multipart nil))
 		    (notmuch-show-insert-body original (plist-get original :body) 0)

Re: (emacs) Parsing problems replying to encrypted html

[David Bremner]

Matthew Lear <matt@bubblegen.co.uk> writes:

> I could have sworn that I replied to this thread months ago but can't find
> a record it in the archives... I think I also missed David E's reply to
> yours mentioning the patch from id:1459708823-1852-1-git-send-email-dme at
> dme.org, so I'm following up...
>
> With David E's patch, the decryption problem I reported is resolved.
> However, I notice that this change hasn't yet been committed / pushed. Is
> there a particular reason for this?

I guess lack of feedback as to whether the patch actually fixes the problem.

> I'm currently maintaining a version of this patch compatible with the
> current master which makes this change to notmuch-mua.el
> notmuch-mua-reply() by adding (notmuch-show-process-crypto process-crypto)
> before doing the show. I've attached that patch here.
>
> I just wondered if we were going to see this pushed to master any time soon?

Perhaps you or dme could send an updated version of the patch to the
list, as git-format-patch output (i.e. with a commit message and other
metadata).

d