From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id YEmwN45cqGDPAQAAgWs5BA (envelope-from ) for ; Sat, 22 May 2021 03:21:18 +0200 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id OGZqM45cqGDbBgAA1q6Kng (envelope-from ) for ; Sat, 22 May 2021 01:21:18 +0000 Received: from mail.notmuchmail.org (nmbug.tethera.net [144.217.243.247]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 6185924AEC for ; Sat, 22 May 2021 03:21:18 +0200 (CEST) Received: from nmbug.tethera.net (localhost [127.0.0.1]) by mail.notmuchmail.org (Postfix) with ESMTP id 7E49229042; Fri, 21 May 2021 21:20:58 -0400 (EDT) Received: from fethera.tethera.net (fethera.tethera.net [IPv6:2607:5300:60:c5::1]) by mail.notmuchmail.org (Postfix) with ESMTP id 5D3CA29000 for ; Fri, 21 May 2021 21:20:52 -0400 (EDT) Received: by fethera.tethera.net (Postfix, from userid 1001) id 0E7885FD27; Fri, 21 May 2021 21:20:52 -0400 (EDT) Received: (nullmailer pid 1412733 invoked by uid 1000); Sat, 22 May 2021 01:20:48 -0000 From: David Bremner To: notmuch@notmuchmail.org Cc: David Bremner Subject: [PATCH 4/5] lib: autocommit after some number of completed transactions Date: Fri, 21 May 2021 22:20:39 -0300 Message-Id: <20210522012040.1412467-5-david@tethera.net> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210522012040.1412467-1-david@tethera.net> References: <20210522012040.1412467-1-david@tethera.net> MIME-Version: 1.0 Message-ID-Hash: BLIVZOE6VCGKLNVNK4XXXXTZVBN2Q66O X-Message-ID-Hash: BLIVZOE6VCGKLNVNK4XXXXTZVBN2Q66O X-MailFrom: bremner@tethera.net X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-notmuch.notmuchmail.org-0; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; suspicious-header X-Mailman-Version: 3.2.1 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Help: List-Post: List-Subscribe: List-Unsubscribe: Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1621646478; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=7e+pqngWTF2fbUsHYjoKbRBW2igXH/LkllDIgk1+jKQ=; b=mVWl10SmZ9pCCjitVnkeFjJXLnl96NsLqg3P3EzubYi6LFbB1shJKOXVvlVMIGyXbNwTCw UmuwDt7n5l20BtNr6buVrsdFK/zuhHPJpsi7DIV601aI5WNH3wDt9camjhVp/IOtRdoF+f QpCS6oSbwdmG10QHDMMZDlOJSZu4bWEnDpueyQtMDnSxlc2soa7Uv9m9h29V/sGGpaJr5+ isvjxWbrD+V4JrkijHl0F1mXUcC8OVOhfP9ydI00i6KQEmY7J3M4b9XW/885mR/2t1BNSH 0imXfJ0lXF42YJr7EbcwyRn6ReIJWZbAcm7MkeH3YrP2blvXk9RfR9B1LFWn6g== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1621646478; a=rsa-sha256; cv=none; b=djsvdfByzCeEt37WL0pweP5qcs3r9a2OFwd9T4MMZOOuPTfQ2W3UjHzNNrOCAXax8WOQzy Q+xmj8v2bVRoOrjugmLdI2qj1e+P/0My09FCBqKfk7DIAziyxk9SqAQFoUl5+nsYmxKwkn vJz/SXlXcLjIed0Y0qMHzO730F+HZwzLY7/VRVTamafxc0pg/o11l4/703TD22tmG19zID fHJH4ihTMnSymVo8hT1M9a3Bo7VSWmRGo/w9NHZL1Xxlx2K4PboEXzTElv9OBH9lp0dNTN Y/qGMXt2vOn9pKTxK8jp2DLWqZUl3zHG4KiLOTWUbq1CXG+QZAgXYHaYhqognQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 144.217.243.247 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org X-Migadu-Spam-Score: -1.07 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 144.217.243.247 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org X-Migadu-Queue-Id: 6185924AEC X-Spam-Score: -1.07 X-Migadu-Scanner: scn0.migadu.com X-TUID: P42ZB9uhFmec This change addresses two known issues with large sets of changes to the database. The first is that as reported by Steven Allen [1], notmuch commits are not "flushed" when they complete, which means that if there is an open transaction when the database closes (or e.g. the program crashes) then all changes since the last commit will be discarded (nothing is irrecoverably lost for "notmuch new", as the indexing process just restarts next time it is run). This does not really "fix" the issue reported in [1]; that seems rather difficult given how transactions work in Xapian. On the other hand, with the default settings, this should mean one only loses less than a minutes worth of work. The second issue is the occasionally reported "storm" of disk writes when notmuch finishes. I don't yet have a test for this, but I think committing as we go should reduce the amount of work when finalizing the database. [1]: id:20151025210215.GA3754@stebalien.com --- lib/database-private.h | 5 +++++ lib/database.cc | 18 +++++++++++++----- lib/open.cc | 12 ++++++++++++ test/T385-transactions.sh | 1 - 4 files changed, 30 insertions(+), 6 deletions(-) diff --git a/lib/database-private.h b/lib/database-private.h index 1a73dacc..9706c17e 100644 --- a/lib/database-private.h +++ b/lib/database-private.h @@ -212,6 +212,11 @@ struct _notmuch_database { char thread_id_str[17]; uint64_t last_thread_id; + /* How many transactions have successfully completed since we last committed */ + int transaction_count; + /* when to commit and reset the counter */ + int transaction_threshold; + /* error reporting; this value persists only until the * next library call. May be NULL */ char *status_string; diff --git a/lib/database.cc b/lib/database.cc index db7feca2..dafea4ce 100644 --- a/lib/database.cc +++ b/lib/database.cc @@ -1125,13 +1125,21 @@ notmuch_database_end_atomic (notmuch_database_t *notmuch) db = notmuch->writable_xapian_db; try { db->commit_transaction (); - - /* This is a hack for testing. Xapian never flushes on a - * non-flushed commit, even if the flush threshold is 1. - * However, we rely on flushing to test atomicity. */ + notmuch->transaction_count++; + + /* Xapian never flushes on a non-flushed commit, even if the + * flush threshold is 1. However, we rely on flushing to test + * atomicity. On the other hand, we can't straight replace + * XAPIAN_FLUSH_THRESHOLD with our autocommit counter, because + * the former also applies outside notmuch atomic + * commits. Hence the follow complicated test */ const char *thresh = getenv ("XAPIAN_FLUSH_THRESHOLD"); - if (thresh && atoi (thresh) == 1) + if ((notmuch->transaction_threshold > 0 && + notmuch->transaction_count >= notmuch->transaction_threshold) || + (thresh && atoi (thresh) == 1)) { db->commit (); + notmuch->transaction_count = 0; + } } catch (const Xapian::Error &error) { _notmuch_database_log (notmuch, "A Xapian exception occurred committing transaction: %s.\n", error.get_msg ().c_str ()); diff --git a/lib/open.cc b/lib/open.cc index 1ca69665..706a23e1 100644 --- a/lib/open.cc +++ b/lib/open.cc @@ -256,6 +256,8 @@ _alloc_notmuch () notmuch->writable_xapian_db = NULL; notmuch->config_path = NULL; notmuch->atomic_nesting = 0; + notmuch->transaction_count = 0; + notmuch->transaction_threshold = 0; notmuch->view = 1; return notmuch; } @@ -365,6 +367,8 @@ _finish_open (notmuch_database_t *notmuch, notmuch_status_t status = NOTMUCH_STATUS_SUCCESS; char *incompat_features; char *message = NULL; + const char *autocommit_str; + char *autocommit_end; unsigned int version; const char *database_path = notmuch_database_get_path (notmuch); @@ -461,6 +465,14 @@ _finish_open (notmuch_database_t *notmuch, if (status) goto DONE; + autocommit_str = notmuch_config_get (notmuch, NOTMUCH_CONFIG_AUTOCOMMIT); + if (unlikely (!autocommit_str)) { + INTERNAL_ERROR ("missing configuration for autocommit"); + } + notmuch->transaction_threshold = strtoul (autocommit_str, &autocommit_end, 10); + if (*autocommit_end != '\0') + INTERNAL_ERROR ("Malformed database database.autocommit value: %s", autocommit_str); + status = _notmuch_database_setup_standard_query_fields (notmuch); if (status) goto DONE; diff --git a/test/T385-transactions.sh b/test/T385-transactions.sh index ebfec2ed..d8bb502d 100755 --- a/test/T385-transactions.sh +++ b/test/T385-transactions.sh @@ -26,7 +26,6 @@ EOF test_expect_equal_file EXPECTED OUTPUT test_begin_subtest "Some changes saved with open transaction" -test_subtest_known_broken notmuch config set database.autocommit 1000 rm -r ${MAIL_DIR}/.notmuch notmuch_with_shim no-close new -- 2.30.2