[PATCH] lib: consider all instances of Delivered-To header

[PATCH] lib: consider all instances of Delivered-To header

[Hannu Hartikainen]

When using notmuch-reply and guessing the From: address from
Delivered-To headers, I had the wrong address chosen today. This was
because the messages from the notmuch list contain these headers in this
order:

Delivered-To: hannu.hartikainen@gmail.com
...
Delivered-To: hannu@hrtk.in

In my .notmuch-config I have the following configuration:

primary_email=hannu@hrtk.in
other_email=hannu.hartikainen@gmail.com;...

Before this change, notmuch-reply would guess From: @gmail.com because
that is the first Delivered-To header present. After the change, the
primary address is chosen as I would expect.
---
 lib/message-file.c | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/lib/message-file.c b/lib/message-file.c
index 647ccf3a..7e8ea09c 100644
--- a/lib/message-file.c
+++ b/lib/message-file.c
@@ -291,11 +291,15 @@ _notmuch_message_file_get_header (notmuch_message_file_t *message,
     if (value)
 	return value;
 
-    if (strcasecmp (header, "received") == 0) {
+    if (strcasecmp (header, "received") == 0 ||
+            strcasecmp (header, "delivered-to") == 0) {
 	/*
-	 * The Received: header is special. We concatenate all
-	 * instances of the header as we use this when analyzing the
-	 * path the mail has taken from sender to recipient.
+	 * The Received: header is special. We concatenate all instances of the
+	 * header as we use this when analyzing the path the mail has taken
+	 * from sender to recipient.
+	 *
+	 * Similarly, multiple instances of Delivered-To may be present. We
+	 * concatenate them so the one with highest priority may be picked.
 	 */
 	decoded = _notmuch_message_file_get_combined_header (message, header);
     } else {
-- 
2.32.0

Re: [PATCH] lib: consider all instances of Delivered-To header

[Michael J Gruber]

Hannu Hartikainen venit, vidit, dixit 2021-06-23 12:29:06:
> When using notmuch-reply and guessing the From: address from
> Delivered-To headers, I had the wrong address chosen today. This was
> because the messages from the notmuch list contain these headers in this
> order:
> 
> Delivered-To: hannu.hartikainen@gmail.com
> ...
> Delivered-To: hannu@hrtk.in
> 
> In my .notmuch-config I have the following configuration:
> 
> primary_email=hannu@hrtk.in
> other_email=hannu.hartikainen@gmail.com;...
> 
> Before this change, notmuch-reply would guess From: @gmail.com because
> that is the first Delivered-To header present. After the change, the
> primary address is chosen as I would expect.
> ---
>  lib/message-file.c | 12 ++++++++----
>  1 file changed, 8 insertions(+), 4 deletions(-)
> 
> diff --git a/lib/message-file.c b/lib/message-file.c
> index 647ccf3a..7e8ea09c 100644
> --- a/lib/message-file.c
> +++ b/lib/message-file.c
> @@ -291,11 +291,15 @@ _notmuch_message_file_get_header (notmuch_message_file_t *message,
>      if (value)
>         return value;
>  
> -    if (strcasecmp (header, "received") == 0) {
> +    if (strcasecmp (header, "received") == 0 ||
> +            strcasecmp (header, "delivered-to") == 0) {
>         /*
> -        * The Received: header is special. We concatenate all
> -        * instances of the header as we use this when analyzing the
> -        * path the mail has taken from sender to recipient.
> +        * The Received: header is special. We concatenate all instances of the
> +        * header as we use this when analyzing the path the mail has taken
> +        * from sender to recipient.
> +        *
> +        * Similarly, multiple instances of Delivered-To may be present. We
> +        * concatenate them so the one with highest priority may be picked.
>          */
>         decoded = _notmuch_message_file_get_combined_header (message, header);
>      } else {
> -- 
> 2.32.0

Without looking at the surrounding code I'm wondering:

Is an address from a received header (still) preferred over one from a
delivered-to, or does the order of headers in the mail envelope play a
role?

Michael

Re: [PATCH] lib: consider all instances of Delivered-To header

[Hannu Hartikainen]

On Wed, 23 Jun 2021 12:58:33 +0200, Michael J Gruber <git@grubix.eu> wrote:
> Is an address from a received header (still) preferred over one from a
> delivered-to, or does the order of headers in the mail envelope play a
> role?

In my understanding the From: address is looked up in the following
steps (each step may have their own internal priorizing logic):

1. To, Cc, Bcc, Reply-To, From
2. Envelope-To
3. X-Original-To
4. Delivered-To
5. Received (for)
6. Received (by)
7. configured primary address

Obviously the patch doesn't touch this logic; it only affects the
handling of multiple Delivered-To headers.

Hannu

Re: [PATCH] lib: consider all instances of Delivered-To header

[David Bremner]

Hannu Hartikainen <hannu@hrtk.in> writes:

> When using notmuch-reply and guessing the From: address from
> Delivered-To headers, I had the wrong address chosen today. This was
> because the messages from the notmuch list contain these headers in this
> order:
>
> Delivered-To: hannu.hartikainen@gmail.com
> ...
> Delivered-To: hannu@hrtk.in
>
> In my .notmuch-config I have the following configuration:
>
> primary_email=hannu@hrtk.in
> other_email=hannu.hartikainen@gmail.com;...
>
> Before this change, notmuch-reply would guess From: @gmail.com because
> that is the first Delivered-To header present. After the change, the
> primary address is chosen as I would expect.
> ---

Delivered-to seems to be a dark corner of (non)-standardization, but it
does seem that multiple deliver-to headers are relatively common and
more or less make sense:

     https://datatracker.ietf.org/doc/draft-crocker-email-deliveredto/

>  lib/message-file.c | 12 ++++++++----
>  1 file changed, 8 insertions(+), 4 deletions(-)
>
> diff --git a/lib/message-file.c b/lib/message-file.c
> index 647ccf3a..7e8ea09c 100644
> --- a/lib/message-file.c
> +++ b/lib/message-file.c
> @@ -291,11 +291,15 @@ _notmuch_message_file_get_header (notmuch_message_file_t *message,
>      if (value)
>  	return value;
>  
> -    if (strcasecmp (header, "received") == 0) {
> +    if (strcasecmp (header, "received") == 0 ||
> +            strcasecmp (header, "delivered-to") == 0) {

uncrustify suggests the following indentation change

-            strcasecmp (header, "delivered-to") == 0) {
+       strcasecmp (header, "delivered-to") == 0) {


>  	/*
> -	 * The Received: header is special. We concatenate all
> -	 * instances of the header as we use this when analyzing the
> -	 * path the mail has taken from sender to recipient.
> +	 * The Received: header is special. We concatenate all instances of the
> +	 * header as we use this when analyzing the path the mail has taken
> +	 * from sender to recipient.
> +	 *
> +	 * Similarly, multiple instances of Delivered-To may be present. We
> +	 * concatenate them so the one with highest priority may be picked.
>  	 */

Highest priority seems a bit vague here. Do you mean most recent?

It seems like the commentary in notmuch-reply should also be updated
since it claims received header is special in being concatenated

I would like to see a new test here, especially since the fix is far
from the effect.

The following change is enough to trigger the problem you had

diff --git a/test/T220-reply.sh b/test/T220-reply.sh
index b6d8f42a..0d245497 100755
--- a/test/T220-reply.sh
+++ b/test/T220-reply.sh
@@ -232,7 +232,8 @@ add_message '[from]="Sender <sender@example.com>"' \
            '[subject]="From guessing"' \
            '[date]="Tue, 05 Jan 2010 15:43:56 -0000"' \
            '[body]="From guessing"' \
-           '[header]="Delivered-To: test_suite_other@notmuchmail.org"'
+           '[header]="Delivered-To: test_suite_other2@notmuchmail.org
+Delivered-To: test_suite_other@notmuchmail.org"'

OTOH, it probably makes sense to test the case of multiple Delivered-to
headers seperately.

The idiomatic (for notmuch) thing to do for a bug fix is first to add a
test with "test_subtest_known_broken", then to remove that line in the
commit you fix the bug.

d

Re: [PATCH] lib: consider all instances of Delivered-To header

[Hannu Hartikainen]

Thanks for the review! Please consider the patch obsolete, I'll submit a
v2 later with the comments addressed.

On Sat, 26 Jun 2021 14:36:23 -0300, David Bremner <david@tethera.net> wrote:
> Hannu Hartikainen <hannu@hrtk.in> writes:
> > +	 * Similarly, multiple instances of Delivered-To may be present. We
> > +	 * concatenate them so the one with highest priority may be picked.
> >  	 */
> 
> Highest priority seems a bit vague here. Do you mean most recent?

I mean that the address configured as `primary_email` is chosen over
those configured as `other_email` if both are present. Basically, let
`user_address_in_string` in notmuch-reply.c do its thing. AFAICT the
addresses in `other_email` are checked in sequence so the first matching
one is chosen, ie. they also have a priority. Not sure if that is
intended and documented or if it could change later, so I didn't want to
go into specifics in the comment.

> The idiomatic (for notmuch) thing to do for a bug fix is first to add a
> test with "test_subtest_known_broken", then to remove that line in the
> commit you fix the bug.

So it's best to have a commit with only the (broken) test first as
opposed to adding the fix and the test in the same commit? Ok, I'll do
that.

Hannu

[PATCH v2 1/2] reply: add test for multiple Delivered-To headers

[Hannu Hartikainen]

Add a known broken subtest for guessing From: correctly when there are
multiple Delivered-To: headers. The address configured as primary_email
should get picked.
---
 test/T220-reply.sh | 23 ++++++++++++++++++++++-
 1 file changed, 22 insertions(+), 1 deletion(-)

diff --git a/test/T220-reply.sh b/test/T220-reply.sh
index b6d8f42a..9f711a04 100755
--- a/test/T220-reply.sh
+++ b/test/T220-reply.sh
@@ -245,6 +245,27 @@ On Tue, 05 Jan 2010 15:43:56 -0000, Sender <sender@example.com> wrote:
 > From guessing
 OK"
 
+test_begin_subtest "From guessing: multiple Delivered-To"
+test_subtest_known_broken
+add_message '[from]="Sender <sender@example.com>"' \
+	    '[to]="Recipient <recipient@example.com>"' \
+	    '[subject]="From guessing"' \
+	    '[date]="Tue, 05 Jan 2010 15:43:56 -0000"' \
+	    '[body]="From guessing"' \
+	    '[header]="Delivered-To: test_suite_other@notmuchmail.org
+Delivered-To: test_suite@notmuchmail.org"'
+
+output=$(notmuch reply id:${gen_msg_id} 2>&1 && echo OK)
+test_expect_equal "$output" "From: Notmuch Test Suite <test_suite@notmuchmail.org>
+Subject: Re: From guessing
+To: Sender <sender@example.com>, Recipient <recipient@example.com>
+In-Reply-To: <${gen_msg_id}>
+References: <${gen_msg_id}>
+
+On Tue, 05 Jan 2010 15:43:56 -0000, Sender <sender@example.com> wrote:
+> From guessing
+OK"
+
 test_begin_subtest "Reply with RFC 2047-encoded headers"
 add_message '[subject]="=?iso-8859-1?q?=e0=df=e7?="' \
 	    '[from]="=?utf-8?q?=e2=98=83?= <snowman@example.com>"' \
@@ -281,7 +302,7 @@ test_expect_equal_json "$output" '
         "crypto": {},
         "date_relative": "2010-01-05",
         "excluded": false,
-        "filename": ["'${MAIL_DIR}'/msg-014"],
+        "filename": ["'${MAIL_DIR}'/msg-015"],
         "headers": {
             "Date": "Tue, 05 Jan 2010 15:43:56 +0000",
             "From": "\u2603 <snowman@example.com>",
-- 
2.32.0

[PATCH v2 2/2] lib: consider all instances of Delivered-To header

[Hannu Hartikainen]

When using notmuch-reply and guessing the From: address from
Delivered-To headers, I had the wrong address chosen today. This was
because the messages from the notmuch list contain these headers in this
order:

Delivered-To: hannu.hartikainen@gmail.com
...
Delivered-To: hannu@hrtk.in

In my .notmuch-config I have the following configuration:

primary_email=hannu@hrtk.in
other_email=hannu.hartikainen@gmail.com;...

Before this change, notmuch-reply would guess From: @gmail.com because
that is the first Delivered-To header present. After the change, the
primary address is chosen as I would expect.
---
 lib/message-file.c | 13 +++++++++----
 notmuch-reply.c    |  7 +++++--
 test/T220-reply.sh |  1 -
 3 files changed, 14 insertions(+), 7 deletions(-)

diff --git a/lib/message-file.c b/lib/message-file.c
index 647ccf3a..68f646a4 100644
--- a/lib/message-file.c
+++ b/lib/message-file.c
@@ -291,11 +291,16 @@ _notmuch_message_file_get_header (notmuch_message_file_t *message,
     if (value)
 	return value;
 
-    if (strcasecmp (header, "received") == 0) {
+    if (strcasecmp (header, "received") == 0 ||
+	strcasecmp (header, "delivered-to") == 0) {
 	/*
-	 * The Received: header is special. We concatenate all
-	 * instances of the header as we use this when analyzing the
-	 * path the mail has taken from sender to recipient.
+	 * The Received: header is special. We concatenate all instances of the
+	 * header as we use this when analyzing the path the mail has taken
+	 * from sender to recipient.
+	 *
+	 * Similarly, multiple instances of Delivered-To may be present. We
+	 * concatenate them so the one with highest priority may be picked (eg.
+	 * primary_email before other_email).
 	 */
 	decoded = _notmuch_message_file_get_combined_header (message, header);
     } else {
diff --git a/notmuch-reply.c b/notmuch-reply.c
index 08140799..ebb621e0 100644
--- a/notmuch-reply.c
+++ b/notmuch-reply.c
@@ -464,8 +464,8 @@ guess_from_in_received_by (notmuch_database_t *notmuch, const char *received)
  * (last Received: header added) and try to extract from them
  * indications to which email address this message was delivered.
  *
- * The Received: header is special in our get_header function and is
- * always concatenated.
+ * The Received: header is among special ones in our get_header function
+ * and is always concatenated.
  *
  * Return the address that was found, if any, and NULL otherwise.
  */
@@ -499,6 +499,9 @@ guess_from_in_received_headers (notmuch_message_t *message)
  * headers: Envelope-To, X-Original-To, and Delivered-To (searched in
  * that order).
  *
+ * The Delivered-To: header is among special ones in our get_header
+ * function and is always concatenated.
+ *
  * Return the address that was found, if any, and NULL otherwise.
  */
 static const char *
diff --git a/test/T220-reply.sh b/test/T220-reply.sh
index 9f711a04..2db36fef 100755
--- a/test/T220-reply.sh
+++ b/test/T220-reply.sh
@@ -246,7 +246,6 @@ On Tue, 05 Jan 2010 15:43:56 -0000, Sender <sender@example.com> wrote:
 OK"
 
 test_begin_subtest "From guessing: multiple Delivered-To"
-test_subtest_known_broken
 add_message '[from]="Sender <sender@example.com>"' \
 	    '[to]="Recipient <recipient@example.com>"' \
 	    '[subject]="From guessing"' \
-- 
2.32.0

Re: [PATCH v2 1/2] reply: add test for multiple Delivered-To headers

[David Bremner]

Hannu Hartikainen <hannu@hrtk.in> writes:

> Add a known broken subtest for guessing From: correctly when there are
> multiple Delivered-To: headers. The address configured as primary_email
> should get picked.

series applied to master.

d