Possible bug: incorrect processing of multipart, base64-encoded subject lines.

Possible bug: incorrect processing of multipart, base64-encoded subject lines.

[Thibault Polge]

Hi all,

Notmuch doesn't parse correctly some Subject lines consisting of
multiple base64-encoded snippets.  As an example, this is the raw
subject line of an e-mail from Amazon:

Subject: =?UTF-8?B?TGl2cmFpc29uIHByw6l2dWUgcG91ciBhdWpvdXJk4oCZaHU=?=
 =?UTF-8?B?aTogUkVTVFJBUCBTYWRkbGUgQmFnIFNhY2NvY2hlLi4u?=

Notmuch displays the subject as only the decoded contents of the first
fragment (or first line), "Livraison prévue pour aujourd’hu", fully
ignoring the second base64 fragment, "i: RESTRAP Saddle Bag
Saccoche...".  The complete subject line should thus read:

> Livraison prévue pour aujourd’hui: RESTRAP Saddle Bag Saccoche...

Regards,
Thibault\r

[PATCH] test: add regression test for Subject with newline.

[David Bremner]

This tests the issue reported by Thibault in id:87wn9w4xus.fsf@thb.lt
---

I could not duplicate the problem here. Maybe it depends on the version of gmime?
I have 3.2.9 here.

 test/T050-new.sh                         |  4 ++++
 test/corpora/indexing/subject-newline:2, | 10 ++++++++++
 2 files changed, 14 insertions(+)
 create mode 100644 test/corpora/indexing/subject-newline:2,

diff --git a/test/T050-new.sh b/test/T050-new.sh
index cb67889c..0ac9fd7c 100755
--- a/test/T050-new.sh
+++ b/test/T050-new.sh
@@ -463,4 +463,8 @@ notmuch search id:20200930101213.2m2pt3jrspvcrxfx@localhost.localdomain > EXPECT
 notmuch search id:20200930101213.2m2pt3jrspvcrxfx@localhost.localdomain and ersatz > OUTPUT
 test_expect_equal_file_nonempty EXPECTED OUTPUT
 
+test_begin_subtest "base64 subject with newline"
+output=$(notmuch show id:subject-with-newline@shopping.com | grep ^Subject:)
+test_expect_equal "${output}" \
+		  "Subject: Livraison prévue pour aujourd’hui: RESTRAP Saddle Bag Saccoche..."
 test_done
diff --git a/test/corpora/indexing/subject-newline:2, b/test/corpora/indexing/subject-newline:2,
new file mode 100644
index 00000000..bb2dde55
--- /dev/null
+++ b/test/corpora/indexing/subject-newline:2,
@@ -0,0 +1,10 @@
+From: "Thibault" <thibault@example.com>
+To: notmuch@notmuchmail.org
+Date: Wed, 18 Nov 2009 02:08:10 -0800
+Subject: =?UTF-8?B?TGl2cmFpc29uIHByw6l2dWUgcG91ciBhdWpvdXJk4oCZaHU=?=
+ =?UTF-8?B?aTogUkVTVFJBUCBTYWRkbGUgQmFnIFNhY2NvY2hlLi4u?=
+Message-ID: <subject-with-newline@shopping.com>
+
+The subject should be
+
+> Livraison prévue pour aujourd’hui: RESTRAP Saddle Bag Saccoche...
-- 
2.35.2
\r

Re: [PATCH] test: add regression test for Subject with newline.

[David Bremner]

David Bremner <david@tethera.net> writes:

> This tests the issue reported by Thibault in id:87wn9w4xus.fsf@thb.lt
> ---
>
> I could not duplicate the problem here. Maybe it depends on the version of gmime?
> I have 3.2.9 here.

Now that I have gmime 3.2.13 I can confirm your bug (my previously
posted test fails).  It seems the purely a gmime issue (the following
reproducer is thanks to Jakub Wilk; on debian it needs the package
gir1.2-gmime-3.0).

#!/usr/bin/python3

import pathlib
import tempfile

import gi
gi.require_version('GMime', '3.0')
from gi.repository import GMime
GMime.init()

msg = b'''\
Subject: =?UTF-8?B?SGVsbG8=?= =?UTF-8?B?IHdvcmxk?=

.
'''

with tempfile.NamedTemporaryFile() as tmpfile:
    tmpfile.write(msg)
    tmpfile.flush()
    fp = GMime.StreamFile.open(tmpfile.name, 'r')
    parser = GMime.Parser.new_with_stream(fp)
    msg = parser.construct_message()
    subject = msg.subject
    expected = 'Hello world'
    assert subject == expected, f'{subject!r} != {expected!r}'

Re: [PATCH] test: add regression test for Subject with newline.

[David Bremner]

David Bremner <david@tethera.net> writes:

> David Bremner <david@tethera.net> writes:
>
>> This tests the issue reported by Thibault in id:87wn9w4xus.fsf@thb.lt
>> ---
>>
>> I could not duplicate the problem here. Maybe it depends on the version of gmime?
>> I have 3.2.9 here.
>
> Now that I have gmime 3.2.13 I can confirm your bug (my previously
> posted test fails).  It seems the purely a gmime issue (the following
> reproducer is thanks to Jakub Wilk; on debian it needs the package
> gir1.2-gmime-3.0).

I think I won't go through the hassle of marking this broken for a
narrow range of gmime versions. Dropping the patch from the review queue
for now.

Re: Possible bug: incorrect processing of multipart, base64-encoded subject lines.

[Jakub Wilk]

* Thibault Polge <thibault@thb.lt>, 2022-09-21 16:15:
>Subject: =?UTF-8?B?TGl2cmFpc29uIHByw6l2dWUgcG91ciBhdWpvdXJk4oCZaHU=?=
> =?UTF-8?B?aTogUkVTVFJBUCBTYWRkbGUgQmFnIFNhY2NvY2hlLi4u?=
>
>Notmuch displays the subject as only the decoded contents of the first 
>fragment

I believe this is now fixed in GMime upstream:
https://github.com/jstedfast/gmime/commit/1a33a55baafc73b0

-- 
Jakub Wilk