From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id SP73B7KlmmB22QAAgWs5BA (envelope-from ) for ; Tue, 11 May 2021 17:41:38 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id 2L2EA7KlmmDFYAAAbx9fmQ (envelope-from ) for ; Tue, 11 May 2021 15:41:38 +0000 Received: from mail.notmuchmail.org (nmbug.tethera.net [IPv6:2607:5300:201:3100::1657]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id CDC3AE79E for ; Tue, 11 May 2021 17:41:37 +0200 (CEST) Received: from nmbug.tethera.net (localhost [127.0.0.1]) by mail.notmuchmail.org (Postfix) with ESMTP id 1A6372717B; Tue, 11 May 2021 11:41:33 -0400 (EDT) Received: from mailproxy01.manitu.net (mailproxy01.manitu.net [217.11.48.65]) by mail.notmuchmail.org (Postfix) with ESMTPS id ECCF527176 for ; Tue, 11 May 2021 11:41:29 -0400 (EDT) Received: from localhost (200116b86017d900ae2f9e0a625a9927.dip.versatel-1u1.de [IPv6:2001:16b8:6017:d900:ae2f:9e0a:625a:9927]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: michael@grubix.eu) by mailproxy01.manitu.net (Postfix) with ESMTPSA id C67BF12617EB; Tue, 11 May 2021 17:41:26 +0200 (CEST) MIME-Version: 1.0 In-Reply-To: <9a30dccb8d00c3de861abb3057331949@condition-alpha.com> References: <162020111427.14194.6147957596067303106.git@grubix.eu> <87lf8tl5xp.fsf@tethera.net> <162048871025.4412.16478596244616302794.git@grubix.eu> <17c04a657363f96fac1c477f238b96d6@condition-alpha.com> <9a30dccb8d00c3de861abb3057331949@condition-alpha.com> Subject: Re: Possible threading issues in nm 0.32 From: Michael J Gruber To: Alexander Adolf , David Bremner , notmuch@notmuchmail.org Message-ID: <162074768557.189534.10699410442728726615.git@grubix.eu> Date: Tue, 11 May 2021 17:41:25 +0200 User-Agent: alot/0.9.1 Message-ID-Hash: 4U4WZ5NOQLQHWTQ7XS6577EETCRZ5QZB X-Message-ID-Hash: 4U4WZ5NOQLQHWTQ7XS6577EETCRZ5QZB X-MailFrom: michael@grubix.eu X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-notmuch.notmuchmail.org-0; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; suspicious-header X-Mailman-Version: 3.2.1 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Help: List-Post: List-Subscribe: List-Unsubscribe: Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1620747697; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=Aq44zcOwzz2b23Kj4QRqVRfc/O/ItNBl1B1TP5PmJxE=; b=qTvNXYCMLj1/3vVkUXNZ1OiLF4RmsWpvpmp+snYGlWomsXBepKaiAzmz8ncUdGIFnV4pWF aQV8AMW8BdDKHMgCAMmUYhTVktX/rkX1RdZ9UyhHBt+Ct85b9StCw5B7nOgKSyWdTilN7G j83/Ulo5CJoOXDoWW0Q70KXUwwbBShLw1BzQ2xcOkJFOUJx3ri8ilc6JBjh0MVBTxTynxR WKfGJvhWfQpAvaLfLJIbwvz4ThwBvzKeZU4uiRnNQY+rh2I6ZNWgFOYXA1deJ4+cwCy5rz 6VHQ5yVyW+2e3V1AX22HakbyT1Pb0zPBJQ6uLp/3RZ4NTNQCoSe48GSLnJe9cQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1620747697; a=rsa-sha256; cv=none; b=T3vFXm7eTyGtYWXtI8362zCTEp3ku20xprQrYW8v1WkHFzVhuac2WU57YZNINpGTgEGniC BsptRtoYHF+qTzF0OZga9fGYbJwBYed1M0y5Jm1aQoXh2h7NO9/p5iHhiTAAdmtCYdIcfs bPN9KWyDM8Q0qHiWyg7ToPk5lmnxKHRo/rmGU7OCBh8A1fcFhN8hS5iLWdbkHTjJlOgmFt E2sHw7Q5+R6MCNbB16NMG40z2MbndE9psxEoB8RnW2E8zB4AuvElpv1OL8c6OOhqUauelW LrpVgupvizlOz4oBZtx/rTERe3/iNJ7CHOn816wBC8z1PBa6uw1wzep1zFnMsA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 2607:5300:201:3100::1657 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org X-Migadu-Spam-Score: 0.66 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 2607:5300:201:3100::1657 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org X-Migadu-Queue-Id: CDC3AE79E X-Spam-Score: 0.66 X-Migadu-Scanner: scn0.migadu.com X-TUID: sZyrLWA7MgaO Alexander Adolf venit, vidit, dixit 2021-05-11 16:32:22: > Alexander Adolf writes: > > > Michael J Gruber writes: > > > >> [...] > >> So it seems: > >> > >> - The mis-threading happens during `notmuch new`, not with `notmuch > >> reindex`. > >> - In this new case (and if I remember correctly also in the others), > >> it's always a new message getting worngly put into an existing thread, > >> and if I'm not mistaken, the existing thread was tagged as trash > >> before in all cases. > >> [...] > > > > I can confirm both observations. > > [...] > > p.s.: Just got the weird threading with `notmuch reindex`, too. Oh my gosh... This is getting interesting. I'm delving (literally) into the xapian db now. "Put into an existing thread" (what I had wiritten) was not correct in terms of thread IDs. What's happening is the following: I have an existing message A which is tagged as trash. A is the only message in thread 0000000000021144. A new message B is put in the db by "notmuch new". Notmuch correctly creates a new thread 0000000000021148 (the next avalaible id) and puts B in this new thread. G0000000000021148 is the only thread term in the db for the document belonging to message B. So far so good, but: The document for message A has three thread terms now: G0000000000021144 G0000000000021148 G0000000000021149 Note that neither A nor B have any in-reply-to or references header. AFAIK multiple thread terms on a single message document are a complete no-go and indicate a problem, especially when an unrelated existing message's entry is touched. notmuch search --exclude=false thread:0000000000021148 lists both A and B now, of course. The third one, G0000000000021149, is completely weird. It leads to yet another message document with multiple thread entries. Looking at a few of the most recent messages, my suspicion is: - A new message with in-reply-to/references get's a single (existing) thread term correctly. - A new message without in-reply-to/references get's the correct new thread term; in addition, this get's assigned to some random existing message by *adding* it to the list of terms, thereby making that message part of multiple threads. I have not checked systematically yet whether it (the multi-G-terms) indeed affects Ktrash ones only. Michael