From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by olra.theworths.org (Postfix) with ESMTP id CED26431FB6 for ; Mon, 23 Jun 2014 13:12:46 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at olra.theworths.org X-Spam-Flag: NO X-Spam-Score: 0.201 X-Spam-Level: X-Spam-Status: No, score=0.201 tagged_above=-999 required=5 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled Received: from olra.theworths.org ([127.0.0.1]) by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id KZhO+mMxiubE for ; Mon, 23 Jun 2014 13:12:42 -0700 (PDT) Received: from mail-we0-f176.google.com (mail-we0-f176.google.com [74.125.82.176]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by olra.theworths.org (Postfix) with ESMTPS id 79CFC431FAF for ; Mon, 23 Jun 2014 13:12:42 -0700 (PDT) Received: by mail-we0-f176.google.com with SMTP id u56so7781609wes.35 for ; Mon, 23 Jun 2014 13:12:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=x7pbQ1dkhjnOGE2vFTIL3+Rp69xxQzGhYctHkRhegB4=; b=dZp6dnlVd0N3vVrRUEfTMk4E1J1dYRB2ZZUL5AaMD3AaA6YP++ax8AVaAtk4gpOaDS bOIFcPaCIn+EmrBYT+KVkBviBMd72J9+6XST84I31Etpyo5dgybNJvFDx1y6cb6HcnsY hn5FV/8gQe2fMJoy49AE1rSC2p+kP8NVtLc/aBsuUqBlXeHSP8wdMpzjt7Ksgu1x8DCk 1h2ts4y9W/94Ty7RvmSmNGPFOOKZBjCs+zT6He8EiKbU6xQWG++j1cUgerN7zAwjpIA3 WXf0TR/+op1oKIjfsIJvzhcPfV7u1RIaxCwe3WF38y4ibHN6j8BJyshXRRHxfWl+Z+F3 DZRQ== X-Received: by 10.194.90.106 with SMTP id bv10mr31064118wjb.20.1403554359229; Mon, 23 Jun 2014 13:12:39 -0700 (PDT) Received: from localhost (5751dfa2.skybroadband.com. [87.81.223.162]) by mx.google.com with ESMTPSA id fb15sm36378270wid.23.2014.06.23.13.12.38 for (version=TLSv1.2 cipher=RC4-SHA bits=128/128); Mon, 23 Jun 2014 13:12:38 -0700 (PDT) From: Mark Walters To: notmuch@notmuchmail.org Subject: [PATCH] dump: make dump take Xapian write lock Date: Mon, 23 Jun 2014 21:12:29 +0100 Message-Id: <1403554349-8888-1-git-send-email-markwalters1009@gmail.com> X-Mailer: git-send-email 1.7.10.4 In-Reply-To: <87zjhh67e7.fsf@qmul.ac.uk> References: <87zjhh67e7.fsf@qmul.ac.uk> X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Jun 2014 20:12:47 -0000 Dump currently only takes the read lock. Xapian can cope with some changes while maintaining a read snapshot but with more changes it fails. Currently notmuch just gives a xapian error. To avoid this we take the write lock when dumping. This prevents other notmuch processes from modifying the xapian database preventing this error. Discussion with Olly on irc indicates that this is currently the best solution: in xapian trunk there may be better possibilities using snapshots but they need to make it to a release and propogate out to users before we can switch approach. Finally, this breaks one use case: pipelines of the form notmuch dump | ... | notmuch restore According to Olly this is already very fragile: it will only work on small databases. One of the tests relies on this behaviour so fix that to store the dump rather than use a pipe. --- I haven't tested this much: all tests pass (with the one fix) and it is simple. Best wishes Mark notmuch-dump.c | 2 +- test/T150-tagging.sh | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/notmuch-dump.c b/notmuch-dump.c index 887a208..9c6ad7f 100644 --- a/notmuch-dump.c +++ b/notmuch-dump.c @@ -212,7 +212,7 @@ notmuch_dump_command (notmuch_config_t *config, int argc, char *argv[]) int ret; if (notmuch_database_open (notmuch_config_get_database_path (config), - NOTMUCH_DATABASE_MODE_READ_ONLY, ¬much)) + NOTMUCH_DATABASE_MODE_READ_WRITE, ¬much)) return EXIT_FAILURE; char *output_file_name = NULL; diff --git a/test/T150-tagging.sh b/test/T150-tagging.sh index dc118f3..45471ac 100755 --- a/test/T150-tagging.sh +++ b/test/T150-tagging.sh @@ -247,8 +247,8 @@ ${TEST_DIRECTORY}/random-corpus --config-path=${NOTMUCH_CONFIG} \ notmuch dump --format=batch-tag | sed 's/^.* -- /+common_tag -- /' | \ sort > EXPECTED -notmuch dump --format=batch-tag | sed 's/^.* -- / -- /' | \ - notmuch restore --format=batch-tag +notmuch dump --format=batch-tag | sed 's/^.* -- / -- /' > INTERMEDIATE_STEP +notmuch restore --format=batch-tag < INTERMEDIATE_STEP notmuch tag --batch < EXPECTED -- 1.7.10.4