From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp12.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id qLPODACmh2LcEAEAbAwnHQ (envelope-from ) for ; Fri, 20 May 2022 16:30:24 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp12.migadu.com with LMTPS id aHtqDACmh2JnKQAAauVa8A (envelope-from ) for ; Fri, 20 May 2022 16:30:24 +0200 Received: from mail.notmuchmail.org (yantan.tethera.net [IPv6:2a01:4f9:c011:7a79::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id B23993A917 for ; Fri, 20 May 2022 16:30:23 +0200 (CEST) Received: from yantan.tethera.net (localhost [127.0.0.1]) by mail.notmuchmail.org (Postfix) with ESMTP id E76655F775; Fri, 20 May 2022 14:30:20 +0000 (UTC) Received: from fethera.tethera.net (fethera.tethera.net [IPv6:2607:5300:60:c5::1]) by mail.notmuchmail.org (Postfix) with ESMTP id A65875F6DA for ; Fri, 20 May 2022 14:30:18 +0000 (UTC) Received: by fethera.tethera.net (Postfix, from userid 1001) id 033485FBD7; Fri, 20 May 2022 10:30:17 -0400 (EDT) Received: (nullmailer pid 1952738 invoked by uid 1000); Fri, 20 May 2022 14:30:16 -0000 From: David Bremner To: Eliza Velasquez , notmuch@notmuchmail.org Subject: Re: [BUG] notmuch_message_tags_to_maildir_flags does not always invalidate filenames_list In-Reply-To: <877d6g1t7e.fsf@eliza.sh> References: <877d6g1t7e.fsf@eliza.sh> Date: Fri, 20 May 2022 11:30:16 -0300 Message-ID: <87pmk89szr.fsf@tethera.net> MIME-Version: 1.0 Message-ID-Hash: XKMBN5PORNGTIJ6G2RR3IQ53SX3FUWN2 X-Message-ID-Hash: XKMBN5PORNGTIJ6G2RR3IQ53SX3FUWN2 X-MailFrom: david@tethera.net X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-notmuch.notmuchmail.org-0; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.3 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Migadu-Flow: FLOW_IN X-Migadu-To: larch@yhetil.org X-Migadu-Country: DE ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1653057023; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-owner:list-unsubscribe:list-subscribe:list-post; bh=vCj1r2vH/LDVfXqqvGGRvvb2NbVEShKBRwiZgVrNU9E=; b=XV91dd+FqmUuCelKlKLl50z+TtQg76pm0jxbxH9G9aTxHyVlYwn6R/gGJhS/qZmkxs/t2b FjUjRDJ9vLaYHFII/WODVs/tRaDSNV8NrsgMlBPqxopaeEZ4wMenGnILacKkHrJCn0BX/v dz3pdjMBVPGqvdErW1b1ao9XJubYhPf+3q4IwhykFPDQ0dAD8H4F+rZleKdnXoInLQhRJG +nB5a0Sf90tYZqWotlgOL/Z120vMWEAaQRFRQehPCqGHKbJa1plCM6xo7zeI3nygjO9T1x aIPAEmTX9XGSTfcVL3zxJctDZ+kpzv0/xj+sb61zYXQxEoNRCrQZYHipHql16g== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1653057023; a=rsa-sha256; cv=none; b=k+F9J/VS1I0T94r5fIWnbpsRXMhOePxYO4oQUcwGkpkk/dP398NX2s+M1iG2y3SDpfqs8Y OnRXUGC/6Z+3cpD0hveo0lDFkgb5MgzRaGD4Rj7k6N4uYm5LuuZNpQ/b45TGDVGOd6DSJU lZHGinrf+SBj3FaCO/mP/dU7n2Y/7tl/P4UJ2qpVq4AO+3kpVD/HD3SwpPYWSxHx2dU+8L ope98CvuudySfKJ+xZu4eTC5Nz3uWNIwtGrusD+Tfn01Aw4QflkkLr6g01ki74NSmWD62I LOvTraUixVhJqQjMllwdWZbVb3eLEzR5ZPEo795ca+C1YgzkJeeIwPKh5lsGCw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 2a01:4f9:c011:7a79::1 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org X-Migadu-Spam-Score: -1.97 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 2a01:4f9:c011:7a79::1 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org X-Migadu-Queue-Id: B23993A917 X-Spam-Score: -1.97 X-Migadu-Scanner: scn1.migadu.com X-TUID: oOmfm+VENlhV Eliza Velasquez writes: > Hello notmuch, > > As the subject says, it seems like the filename cache for a message is > not reliably invalidated when notmuch_message_tags_to_maildir_flags is > called, causing the following series of function calls to behave > unexpectedly. > > - notmuch_database_index_file > - notmuch_message_get_filenames > - notmuch_message_tags_to_maildir_flags > - notmuch_message_get_filenames (stale list returned) > > According to the investigator [1], this happens about 10% of the time, > but it's possible that this is just because there is some quirk in the > client-side code causing it to trigger only sometimes. that cache is supposed to be invalidated by calls to _notmuch_message_add_filename and _notmuch_message_remove_filename. There is one unchecked call to _notmuch_message_add_term that looks a bit suspect. The obvious thing that can go wrong is filename getting too long. Here is a patch to maybe return better diagnostics in that case diff --git a/lib/message.cc b/lib/message.cc index 63b216b6..bebc5f6a 100644 --- a/lib/message.cc +++ b/lib/message.cc @@ -169,6 +169,7 @@ _notmuch_message_create_for_document (const void *talloc_owner, message->doc = doc; message->termpos = 0; + message->modified = FALSE; return message; } @@ -937,6 +938,7 @@ _notmuch_message_add_filename (notmuch_message_t *message, { const char *relative, *directory; notmuch_status_t status; + notmuch_private_status_t private_status; void *local = talloc_new (message); char *direntry; @@ -960,7 +962,11 @@ _notmuch_message_add_filename (notmuch_message_t *message, /* New file-direntry allows navigating to this message with * notmuch_directory_get_child_files() . */ - _notmuch_message_add_term (message, "file-direntry", direntry); + private_status = _notmuch_message_add_term (message, "file-direntry", direntry); + status = COERCE_STATUS (private_status, + "Unexpected error from _notmuch_message_add_term"); + if (status) + return status; _notmuch_message_add_folder_terms (message, directory); _notmuch_message_add_path_terms (message, directory); The other issue that might cause intermittent failures is uncaught exceptions. We cleaned a bunch of those up a few years ago, but it looks like we missed a few spots. Off hand I'm not sure what would be causing such exceptions, but _notmuch_message_add_term and _notmuch_message_remove_term should probably have the Xapian operations wrapped in try / catch blocks. It might be worth running the failing test under gdb to see if you can spot exceptions.