From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by arlo.cworth.org (Postfix) with ESMTP id 6488E6DE1082 for ; Tue, 28 May 2019 15:55:10 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at cworth.org X-Spam-Flag: NO X-Spam-Score: -0.178 X-Spam-Level: X-Spam-Status: No, score=-0.178 tagged_above=-999 required=5 tests=[AWL=0.023, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001] autolearn=disabled Received: from arlo.cworth.org ([127.0.0.1]) by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4E_PUdLYZrvN for ; Tue, 28 May 2019 15:55:08 -0700 (PDT) Received: from che.mayfirst.org (che.mayfirst.org [162.247.75.118]) by arlo.cworth.org (Postfix) with ESMTPS id 203676DE106E for ; Tue, 28 May 2019 15:55:07 -0700 (PDT) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/simple; d=fifthhorseman.net; i=@fifthhorseman.net; q=dns/txt; s=2019; t=1559084104; h=from : to : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=ReYkjK2TuL41GRiUmOErkdsCuYZqrkqxVB8PpCMgFbA=; b=cyB5EYESWeqVmahE0MiA+cMwo7hsRSd6RxZ/hLECF2pie0fW6oeuKGpU i96JNSHjXPWN2wOHQQtETSVwvGJYDQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=fifthhorseman.net; i=@fifthhorseman.net; q=dns/txt; s=2019rsa; t=1559084103; h=from : to : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=ReYkjK2TuL41GRiUmOErkdsCuYZqrkqxVB8PpCMgFbA=; b=dFJMUcJJIkrpcs3SnFc3zvfqjvH5OntvGpUcfX4ewSWyLhKM5ENz7alF vwmMY/gZ1eHxgot1OP0ZRFq4+09pC2mC1lX+V+4NnZmaYO2phPKxIoCLZ7 wbFQeegCE6GJnWSe5E4SK9rcRtuDgdieQgmN93CXe2k41GzhUoc2M5j0I9 Qz8PWpGz3kqG/n0XeMAsDg/CHLDo+TrFeQmliPkeRkJXvcmeRDrxOMAzB2 uX9lLDYOwnW3ZB7ju70F67EU0ZObRO+W1coVMsCoXWWtGlt7Jgv45VPVhl gi8RXf05o6Kb3SRUHqocxlYTDOUCDdJ+OMPO0w4lvZ77G7v2VLZIUQ== Received: from fifthhorseman.net (unknown [38.109.115.130]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by che.mayfirst.org (Postfix) with ESMTPSA id A3B37F9A3 for ; Tue, 28 May 2019 18:55:03 -0400 (EDT) Received: by fifthhorseman.net (Postfix, from userid 1000) id DD0F320B6B; Tue, 28 May 2019 18:54:59 -0400 (EDT) From: Daniel Kahn Gillmor To: Notmuch Mail Subject: [PATCH 3/4] index: repair "Mixed Up" messages before indexing. Date: Tue, 28 May 2019 18:54:51 -0400 Message-Id: <20190528225452.17550-4-dkg@fifthhorseman.net> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190528225452.17550-1-dkg@fifthhorseman.net> References: <20190528225452.17550-1-dkg@fifthhorseman.net> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 May 2019 22:55:10 -0000 When encountering a message that has been mangled in the "mixed up" way by an intermediate MTA, notmuch should instead repair it and index the repaired form. When it does this, it also associates the index.repaired=mixedup property with the message. If a problem is found with this repair process, or an improved repair process is proposed later, this should make it easy for people to reindex the relevant message. The property will also hopefully make it easier to diagnose this particular problem in the future. Signed-off-by: Daniel Kahn Gillmor --- doc/man7/notmuch-properties.rst | 13 +++++++++++++ lib/index.cc | 22 +++++++++++++++++----- test/T351-pgpmime-mangling.sh | 2 -- 3 files changed, 30 insertions(+), 7 deletions(-) diff --git a/doc/man7/notmuch-properties.rst b/doc/man7/notmuch-properties.rst index 802e6763..31de576e 100644 --- a/doc/man7/notmuch-properties.rst +++ b/doc/man7/notmuch-properties.rst @@ -109,6 +109,19 @@ of its normal activity. example, an AES-128 key might be stashed in a notmuch property as: ``session-key=7:14B16AF65536C28AF209828DFE34C9E0``. +**index.repaired** + + Some mail transport agents mangle messages in transit in ways that + are both detectable and reversible. If notmuch encounters such a + mangling during indexing, it will try to index the repaired form + of the message (while still leaving the message on disk + untouched). If successful, it will use the ``index.repaired`` + property to note the kind of mangling that was repaired. + Currently, only one form of repairable mangling is detected and + repaired, which is denoted with ``index.repaired=mixedup``. See + https://tools.ietf.org/html/draft-dkg-openpgp-pgpmime-message-mangling + for more information. + SEE ALSO ======== diff --git a/lib/index.cc b/lib/index.cc index 1fd9e67e..44a42deb 100644 --- a/lib/index.cc +++ b/lib/index.cc @@ -385,11 +385,20 @@ _index_mime_part (notmuch_message_t *message, GMimeContentType *content_type; char *body; const char *charset; + GMimeObject *repaired_part = NULL; if (! part) { _notmuch_database_log (notmuch_message_get_database (message), "Warning: Not indexing empty mime part.\n"); - return; + goto DONE; + } + + repaired_part = _notmuch_repair_mixed_up_mangled (part); + if (repaired_part) { + /* This was likely "Mixed Up" in transit! We will instead use + * the more likely-to-be-correct variant. */ + notmuch_message_add_property (message, "index.repaired", "mixedup"); + part = repaired_part; } _index_content_type (message, part); @@ -441,7 +450,7 @@ _index_mime_part (notmuch_message_t *message, notmuch_status_to_string (status)); _index_mime_part (message, indexopts, child, msg_crypto); } - return; + goto DONE; } if (GMIME_IS_MESSAGE_PART (part)) { @@ -451,14 +460,14 @@ _index_mime_part (notmuch_message_t *message, _index_mime_part (message, indexopts, g_mime_message_get_mime_part (mime_message), msg_crypto); - return; + goto DONE; } if (! (GMIME_IS_PART (part))) { _notmuch_database_log (notmuch_message_get_database (message), "Warning: Not indexing unknown mime part: %s.\n", g_type_name (G_OBJECT_TYPE (part))); - return; + goto DONE; } disposition = g_mime_object_get_content_disposition (part); @@ -473,7 +482,7 @@ _index_mime_part (notmuch_message_t *message, /* XXX: Would be nice to call out to something here to parse * the attachment into text and then index that. */ - return; + goto DONE; } byte_array = g_byte_array_new (); @@ -519,6 +528,9 @@ _index_mime_part (notmuch_message_t *message, free (body); } + DONE: + if (repaired_part) + g_object_unref (repaired_part); } /* descend (if desired) into the cleartext part of an encrypted MIME diff --git a/test/T351-pgpmime-mangling.sh b/test/T351-pgpmime-mangling.sh index 17f94a31..a20066e4 100755 --- a/test/T351-pgpmime-mangling.sh +++ b/test/T351-pgpmime-mangling.sh @@ -13,7 +13,6 @@ test_json_nodes <<<"$output" \ 'body:[0][0][0]["body"][0]["content"][1]["content"]="The password is \"abcd1234!\", please do not tell anyone.\n"' test_begin_subtest "repaired 'Mixed-up' messages can be found with index.repaired=mixedup" -test_subtest_known_broken output=$(notmuch search --output=messages property:index.repaired=mixedup) test_expect_equal "$output" id:mixed-up@mangling.notmuchmail.org @@ -21,7 +20,6 @@ test_begin_subtest "reindex 'Mixed-Up' mangled PGP/MIME message" test_expect_success 'notmuch reindex --decrypt=true id:mixed-up@mangling.notmuchmail.org' test_begin_subtest "search cleartext of 'Mixed-Up' mangled PGP/MIME message" -test_subtest_known_broken output=$(notmuch search --output=messages body:password) test_expect_equal "$output" id:mixed-up@mangling.notmuchmail.org -- 2.20.1