From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp11.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id MIneEi6GjWIjDAAAbAwnHQ (envelope-from ) for ; Wed, 25 May 2022 03:28:14 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp11.migadu.com with LMTPS id +FDbEi6GjWKoIAAA9RJhRA (envelope-from ) for ; Wed, 25 May 2022 03:28:14 +0200 Received: from mail.notmuchmail.org (yantan.tethera.net [IPv6:2a01:4f9:c011:7a79::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id BA67D9592 for ; Wed, 25 May 2022 03:28:13 +0200 (CEST) Received: from yantan.tethera.net (localhost [127.0.0.1]) by mail.notmuchmail.org (Postfix) with ESMTP id 488BC5F70D; Wed, 25 May 2022 01:28:10 +0000 (UTC) Received: from wnew3-smtp.messagingengine.com (wnew3-smtp.messagingengine.com [64.147.123.17]) by mail.notmuchmail.org (Postfix) with ESMTPS id 35D245F6DA for ; Wed, 25 May 2022 01:28:07 +0000 (UTC) Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailnew.west.internal (Postfix) with ESMTP id A35C72B05853; Tue, 24 May 2022 21:28:03 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Tue, 24 May 2022 21:28:03 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=eliza.sh; h=cc :content-type:date:date:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to; s=fm1; t=1653442083; x=1653445683; bh=gMSLpqoxsJ WaiQYD2HgYMj0peH3N7Ay3RMbQPV0YQtE=; b=IpMci9vb37BUNf85CMANOjiAJO mTwU2XR/ktUGn/1wnS9zaAI6MEA0nIZZ0+FUu2V+0l/KE9dX0cw25oPXrRhDHBuN e55ibpC64l0qDenHUNacMIhnoAqxSOUw6O1FW/UC08zxqPIXbI57S2sWA/daVVUO 8wFljiLkZT2cqsha8nus2uqUgwpDNJ+swGxBVpOUGm/SOf7mAkJtvOPIIGDPJJ+J +Ba8QUTuvUVWTSH6nneGALbXwUxlQKcuP5qFz1GI5axemlNffGVTe8BZ6fC3kfTp vm+Lj+lmMY6K2j6uLB2HgHAsXnlbvw3u/3HSZ+8UxRivoyxd2y1ayT33Di5w== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:date:feedback-id :feedback-id:from:from:in-reply-to:in-reply-to:message-id :mime-version:references:reply-to:sender:subject:subject:to:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm1; t=1653442083; x=1653445683; bh=gMSLpqoxsJWaiQYD2HgYMj0peH3N 7Ay3RMbQPV0YQtE=; b=QEPgLMmuHHppd9IPwUfGgQbo2lGe/6k/MZj7D7y/bBbG bOTjpgP2oF5VILUl/eCPQMcbJEG3ZCpj7Tuu6HAl2jzQhXcdhtQF5nSxFsU1TEFO wt1emDkg33QZcX/sgZc3kAx65UQ1xPfKekryaKndtynMierqgYJ44qZXwV9h1HAK oJ+njNVMF+5IXagwGwnX/X9YHF5hTpCn2Jj5LRsreZvzE5y0l3KinVspBUxTGpCh eZYON+aCOjtw3a277FxX7s3/vvN3z4e0vz/bH7lDwb5/7DBcykJAeCiLzHtwUhsx RHBAX2DLMUlZntt2IQomXl2xfyxBRVKmhcpWwRMjIg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvfedrjeeggdegiecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvffujghffffkgggtsehttdertddttddtnecuhfhrohhmpefglhhiiigrucgg vghlrghsqhhuvgiiuceovghlihiirgesvghlihiirgdrshhhqeenucggtffrrghtthgvrh hnpefhkeejffegvefgieelveekkefftdetvddvudduudffjeffheejtdefleeitdettden ucffohhmrghinhepghhithhhuhgsrdgtohhmpdhnohhtmhhutghhmhgrihhlrdhorhhgne cuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepvghlihii rgesvghlihiirgdrshhh X-ME-Proxy: Feedback-ID: i87c446d8:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 24 May 2022 21:28:02 -0400 (EDT) From: Eliza Velasquez To: David Bremner , notmuch@notmuchmail.org Subject: Re: [BUG] notmuch_message_tags_to_maildir_flags does not always invalidate filenames_list In-Reply-To: <87leur85gr.fsf@tethera.net> References: <877d6g1t7e.fsf@eliza.sh> <87leur85gr.fsf@tethera.net> Date: Tue, 24 May 2022 18:28:01 -0700 Message-ID: <87ee0iqu3i.fsf@eliza.sh> MIME-Version: 1.0 Message-ID-Hash: 4UAVDZGBM53ZUFI3BST7LPES53I7LRZ4 X-Message-ID-Hash: 4UAVDZGBM53ZUFI3BST7LPES53I7LRZ4 X-MailFrom: eliza@eliza.sh X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-notmuch.notmuchmail.org-0; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.3 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Migadu-Flow: FLOW_IN X-Migadu-To: larch@yhetil.org X-Migadu-Country: DE ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1653442093; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-owner:list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=aW5ko6ewW6JDo/6HnJ929XQz2Xh3FiOoxaVGJVgIFcg=; b=fKfJq14MqhZCT/qedzhRtfM0iMLcZFKunuOStQ1E9qkQkGNUMuHPv8j22uBGGSXHDVehoS Y43EPKpuLQrTP5IlH4CLVTxK0EGV4awLbrDV7HwS2aWDT11xWFraJ9zznohw62qb2AEvcJ gLmcLAFGuAsx7M4f04mG3cjTPy6vu5GS/aCr4L0//bm21+l2fjYlRjfGE+Bh1kjuIeX9Ws T17XwCICfuQNUVDV+8wjT62x8pcScPBwEzvP73vL2qRmoqycF4UUWZ0OFzXqomgcuA4/U4 rYt7jCBROhry4FGaYFuoOlnGTFV59TSFIX6JJOQsbmKz6SylHfZEJw8+ZRCkEA== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1653442093; a=rsa-sha256; cv=none; b=CFJ6QyvEUD1LRBKQazPPkMkCxjeLll9/1u4U7WOdHCYrg+N4zep9a18s/R18m7C+Y5ZCws 6Nz87TEnCVoMw3cDjcYABNn9Xytp3JqD+9U+GBCVt1AjAn6U3tj9ryizH+ru/2Jk3BSHk9 7epMjmx+5bzZnilS2L73AVSYb7JlnINGndVIxLX5jHSh71ugZYaBpU6tATL99BO62zL1a7 cuJhY5YBof88PyicskLC1/JniNyNgGzQNqWtMdCCuP8yB+3HuEL2VeFwPCv0nuhvai9rRs WedVWeok6a1I3e9xiIVSlNAULkBfcuwEaenUIx9bRa1UK81rZntMFlpospSDdg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("body hash did not verify") header.d=eliza.sh header.s=fm1 header.b=IpMci9vb; dkim=fail ("body hash did not verify") header.d=messagingengine.com header.s=fm1 header.b=QEPgLMmu; dmarc=none; spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 2a01:4f9:c011:7a79::1 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org X-Migadu-Spam-Score: 1.38 Authentication-Results: aspmx1.migadu.com; dkim=fail ("body hash did not verify") header.d=eliza.sh header.s=fm1 header.b=IpMci9vb; dkim=fail ("body hash did not verify") header.d=messagingengine.com header.s=fm1 header.b=QEPgLMmu; dmarc=none; spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 2a01:4f9:c011:7a79::1 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org X-Migadu-Queue-Id: BA67D9592 X-Spam-Score: 1.38 X-Migadu-Scanner: scn0.migadu.com X-TUID: ihPylOmYOU7Y On Tue, May 24 2022 at 09:45 -03, David Bremner wrote: > Eliza Velasquez writes: > >> Hello notmuch, >> >> As the subject says, it seems like the filename cache for a message is >> not reliably invalidated when notmuch_message_tags_to_maildir_flags is >> called, causing the following series of function calls to behave >> unexpectedly. >> >> - notmuch_database_index_file >> - notmuch_message_get_filenames >> - notmuch_message_tags_to_maildir_flags >> - notmuch_message_get_filenames (stale list returned) >> >> According to the investigator [1], this happens about 10% of the time, >> but it's possible that this is just because there is some quirk in the >> client-side code causing it to trigger only sometimes. >> >> [1] https://github.com/elizagamedev/mujmap/pull/10#issuecomment-1131587169 >> > > I have been looking at this code, and making some small improvements [1], > but without a C reproducer I don't really know if it helps the > submitter's issue. > > d > > [1]: https://nmbug.notmuchmail.org/nmweb/show/20220523233901.3506880-1-david@tethera.net Thank you for looking into this and putting out such a quick patch, I really appreciate it. It turns out the core issue that we were encountering actually seems like it's due to cache propagation issues across message instances, according to the investigator (Rob). To quote: > I think its a fairly standard cache write propagation issue, with the > outstanding message objects as the caches. Whether or not its a bug in > notmuch itself I think is entirely down to whether holding and > modifying multiple message objects for the "same" record (with > duplicates) is supposed to work or not. I'm in two minds about it; on > the one hand, multiple message objects is clearly a valid use case > (queries!), on the other, playing with the mail files themselves is a > bit of a niche feature. I'll think about it a bit more and maybe take > it up to the notmuch list myself. He wrote a more detailed writeup of it as well [1], which includes a minimal reproducible test case written in C. After reviewing this, I'm also not sure whether or not this should be considered a notmuch bug, but it was surprising behavior. [1] https://gist.github.com/robn/a56420062127238f57541eb47bc962f5