From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by olra.theworths.org (Postfix) with ESMTP id 00C21431FB6 for ; Tue, 26 Feb 2013 01:19:23 -0800 (PST) X-Virus-Scanned: Debian amavisd-new at olra.theworths.org X-Spam-Flag: NO X-Spam-Score: -0.7 X-Spam-Level: X-Spam-Status: No, score=-0.7 tagged_above=-999 required=5 tests=[RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled Received: from olra.theworths.org ([127.0.0.1]) by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7lMT0nEj4WZN for ; Tue, 26 Feb 2013 01:19:20 -0800 (PST) Received: from mail-we0-f175.google.com (mail-we0-f175.google.com [74.125.82.175]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by olra.theworths.org (Postfix) with ESMTPS id 52D14431FAF for ; Tue, 26 Feb 2013 01:19:20 -0800 (PST) Received: by mail-we0-f175.google.com with SMTP id x8so3317447wey.6 for ; Tue, 26 Feb 2013 01:19:17 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:from:to:subject:in-reply-to:references:user-agent:date :message-id:mime-version:content-type:x-gm-message-state; bh=fumzV0OKnmDWKk9etUJm2gd9dTO/IAToWlwm722R27Q=; b=l550UJYCKGunAujYO8aZVnSFVCGKoAEjkAGFiOijNfeyfSOUpnnTw1BDAIpcHNH55o uqWmLbVztjgJ/wlHaOBcNzGoaElHovcR/O9zZC4sjOCNJujVFcKOuPBJopRTiTibrl6F vz1LG+9R55TSmuQlQOhEg29SiYFdORNG1PMBKKY99zq0WdDdOAES2RJToAkyE3TPTy7o 8VOtLmtQNPGYCjBTSPIZXGDJ+HL5Nd4nT+IKAh/mJNoMa6wsurNMeE1HyQPU4ApCMXu4 /om6T1ohN8cWsBN7Mco+xSshgobofMjDectLbXsTYbaAfMCbO3YUNG7uAel/FhH7avnG AMPw== X-Received: by 10.180.92.39 with SMTP id cj7mr17802562wib.19.1361870357808; Tue, 26 Feb 2013 01:19:17 -0800 (PST) Received: from localhost ([2001:4b98:dc0:43:216:3eff:fe1b:25f3]) by mx.google.com with ESMTPS id bj9sm19843333wib.4.2013.02.26.01.19.15 (version=TLSv1.1 cipher=RC4-SHA bits=128/128); Tue, 26 Feb 2013 01:19:16 -0800 (PST) From: Jani Nikula To: Aaron Ecay , notmuch@notmuchmail.org Subject: Re: [RFC] [PATCH] lib/database.cc: change how the parent of a message is calculated In-Reply-To: <1361836225-17279-1-git-send-email-aaronecay@gmail.com> References: <1361836225-17279-1-git-send-email-aaronecay@gmail.com> User-Agent: Notmuch/0.14+259~gdee88db (http://notmuchmail.org) Emacs/23.2.1 (x86_64-pc-linux-gnu) Date: Tue, 26 Feb 2013 10:19:06 +0100 Message-ID: <878v6bjuut.fsf@nikula.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Gm-Message-State: ALoCoQlx3ALbGyQhHxtVa2iEPQQ0nC850x2OiuzgcWKxG64wFqbLyFq+Uzo+nSwk1lSSdYs/S+Xe X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Feb 2013 09:19:23 -0000 On Tue, 26 Feb 2013, Aaron Ecay wrote: > Presently, the code which finds the parent of a message as it is being > added to the database assumes that the first Message-ID-like substring > of the In-Reply-To header is the parent Message ID. Some mail clients, > however, put stuff other than the Message-ID of the parent in the > In-Reply-To header, such as the email address of the sender of the > parent. This can fool notmuch. Hi Aaron, please provide references to a few messages like this. If available on the notmuch list an id: reference would be best, but otherwise some archive that allows viewing full message headers or downloading the full message would be best. Thanks, Jani. > > The updated algorithm prefers the last Message ID in the References > header. The References header lists messages oldest-first, so the last > Message ID is the parent (RFC2822, p. 24). The References header is > also less likely to be in a non-standard > syntax (http://cr.yp.to/immhf/thread.html, > http://www.jwz.org/doc/threading.html). In case the References header > is not to be found, fall back to the old behavior. > --- > > I especially notice this problem on public mailing lists, where > certain people's messages always cause an "out-dent" of the threading, > instead of being nested under whichever message they are replies to. > > Technically, putting non-Message-ID crud in the In-Reply-To field is a > violation of RFC2822, but it appears that in practice the References > header is respected more often than the In-Reply-To one. > > lib/database.cc | 30 ++++++++++++++++++++++-------- > 1 file changed, 22 insertions(+), 8 deletions(-) > > diff --git a/lib/database.cc b/lib/database.cc > index 91d4329..cbf33ae 100644 > --- a/lib/database.cc > +++ b/lib/database.cc > @@ -501,8 +501,10 @@ _parse_message_id (void *ctx, const char *message_id, const char **next) > * 'message_id' in the result (to avoid mass confusion when a single > * message references itself cyclically---and yes, mail messages are > * not infrequent in the wild that do this---don't ask me why). > + * > + * Return the last reference parsed. > */ > -static void > +static char * > parse_references (void *ctx, > const char *message_id, > GHashTable *hash, > @@ -511,7 +513,7 @@ parse_references (void *ctx, > char *ref; > > if (refs == NULL || *refs == '\0') > - return; > + return NULL; > > while (*refs) { > ref = _parse_message_id (ctx, refs, &refs); > @@ -519,6 +521,8 @@ parse_references (void *ctx, > if (ref && strcmp (ref, message_id)) > g_hash_table_insert (hash, ref, NULL); > } > + > + return ref; > } > > notmuch_status_t > @@ -1365,7 +1369,7 @@ _notmuch_database_generate_doc_id (notmuch_database_t *notmuch) > notmuch->last_doc_id++; > > if (notmuch->last_doc_id == 0) > - INTERNAL_ERROR ("Xapian document IDs are exhausted.\n"); > + INTERNAL_ERROR ("Xapian document IDs are exhausted.\n"); > > return notmuch->last_doc_id; > } > @@ -1509,7 +1513,7 @@ _notmuch_database_link_message_to_parents (notmuch_database_t *notmuch, > const char **thread_id) > { > GHashTable *parents = NULL; > - const char *refs, *in_reply_to, *in_reply_to_message_id; > + const char *refs, *in_reply_to, *in_reply_to_message_id, *last_ref_message_id; > GList *l, *keys = NULL; > notmuch_status_t ret = NOTMUCH_STATUS_SUCCESS; > > @@ -1517,21 +1521,31 @@ _notmuch_database_link_message_to_parents (notmuch_database_t *notmuch, > _my_talloc_free_for_g_hash, NULL); > > refs = notmuch_message_file_get_header (message_file, "references"); > - parse_references (message, notmuch_message_get_message_id (message), > - parents, refs); > + last_ref_message_id = parse_references (message, > + notmuch_message_get_message_id (message), > + parents, refs); > > in_reply_to = notmuch_message_file_get_header (message_file, "in-reply-to"); > parse_references (message, notmuch_message_get_message_id (message), > parents, in_reply_to); > > - /* Carefully avoid adding any self-referential in-reply-to term. */ > in_reply_to_message_id = _parse_message_id (message, in_reply_to, NULL); > + /* If the parent message ID from the Reply-To and References > + * headers are different, use the References one. This is because > + * the Reply-To header is more likely to be in an non-standard > + * format. */ > + if (in_reply_to_message_id && > + last_ref_message_id && > + strcmp (last_ref_message_id, in_reply_to_message_id)) { > + in_reply_to_message_id = last_ref_message_id; > + } > + /* Carefully avoid adding any self-referential in-reply-to term. */ > if (in_reply_to_message_id && > strcmp (in_reply_to_message_id, > notmuch_message_get_message_id (message))) > { > _notmuch_message_add_term (message, "replyto", > - _parse_message_id (message, in_reply_to, NULL)); > + in_reply_to_message_id); > } > > keys = g_hash_table_get_keys (parents); > -- > 1.8.1.4 > _______________________________________________ > notmuch mailing list > notmuch@notmuchmail.org > http://notmuchmail.org/mailman/listinfo/notmuch