From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by olra.theworths.org (Postfix) with ESMTP id 7D774431FAE for ; Mon, 25 Aug 2014 10:27:08 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at olra.theworths.org X-Spam-Flag: NO X-Spam-Score: -2.3 X-Spam-Level: X-Spam-Status: No, score=-2.3 tagged_above=-999 required=5 tests=[RCVD_IN_DNSWL_MED=-2.3] autolearn=disabled Received: from olra.theworths.org ([127.0.0.1]) by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ny5HVBsOmgFa for ; Mon, 25 Aug 2014 10:27:02 -0700 (PDT) Received: from dmz-mailsec-scanner-6.mit.edu (dmz-mailsec-scanner-6.mit.edu [18.7.68.35]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by olra.theworths.org (Postfix) with ESMTPS id 0013C431FD0 for ; Mon, 25 Aug 2014 10:26:25 -0700 (PDT) X-AuditID: 12074423-f799d6d00000337c-c4-53fb71c11795 Received: from mailhub-auth-1.mit.edu ( [18.9.21.35]) (using TLS with cipher AES256-SHA (256/256 bits)) (Client did not present a certificate) by dmz-mailsec-scanner-6.mit.edu (Symantec Messaging Gateway) with SMTP id 6C.A0.13180.1C17BF35; Mon, 25 Aug 2014 13:26:25 -0400 (EDT) Received: from outgoing.mit.edu (outgoing-auth-1.mit.edu [18.9.28.11]) by mailhub-auth-1.mit.edu (8.13.8/8.9.2) with ESMTP id s7PHQEWx007794; Mon, 25 Aug 2014 13:26:14 -0400 Received: from drake.dyndns.org (31-35-14.wireless.csail.mit.edu [128.31.35.14]) (authenticated bits=0) (User authenticated as amdragon@ATHENA.MIT.EDU) by outgoing.mit.edu (8.13.8/8.12.4) with ESMTP id s7PHQC45029613 (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=NOT); Mon, 25 Aug 2014 13:26:13 -0400 Received: from amthrax by drake.dyndns.org with local (Exim 4.77) (envelope-from ) id 1XLy1z-0003jV-W5; Mon, 25 Aug 2014 13:26:11 -0400 From: Austin Clements To: notmuch@notmuchmail.org Subject: [PATCH v4 05/11] lib: Simplify upgrade code using a transaction Date: Mon, 25 Aug 2014 13:26:03 -0400 Message-Id: <1408987569-14146-6-git-send-email-amdragon@mit.edu> X-Mailer: git-send-email 2.0.0 In-Reply-To: <1408987569-14146-1-git-send-email-amdragon@mit.edu> References: <1408987569-14146-1-git-send-email-amdragon@mit.edu> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrKIsWRmVeSWpSXmKPExsUixCmqrHuw8Hewwa1Ofosbrd2MFk3TnS2u 35zJ7MDscev+a3aPZ6tuMXtsOfSeOYA5issmJTUnsyy1SN8ugStjbV9JwT3ZihN3LjM2MC4X 72Lk5JAQMJHYfvEvE4QtJnHh3nq2LkYuDiGB2UwSx4++ZoJwNjJK3O7cApU5xiTxe9FuRpAW IYG5jBLr/jqA2GwCGhK/by0GGyUiIC2x8+5sVhCbWcBLYtmFfjBbWMBDYuahFrBeFgFViV1b usHivAIOErtmrWCEOENOouHGJzYQm1PAUWL6wtlMELscJL5fPs88gZF/ASPDKkbZlNwq3dzE zJzi1GTd4uTEvLzUIl0zvdzMEr3UlNJNjKDQYndR3sH456DSIUYBDkYlHt4b8b+DhVgTy4or cw8xSnIwKYnyTssHCvEl5adUZiQWZ8QXleakFh9ilOBgVhLhbQbJ8aYkVlalFuXDpKQ5WJTE ed9aWwULCaQnlqRmp6YWpBbBZGU4OJQkeHMLgBoFi1LTUyvSMnNKENJMHJwgw3mAhseD1PAW FyTmFmemQ+RPMSpKifOuBdkqAJLIKM2D64XF/itGcaBXhHmPgrTzANMGXPcroMFMQINNe36C DC5JREhJNTAekyiWtjzKP5l12Yq1SxaLnRY4F36geNWWt3M9H4tvnRe7a0fS2nW9TVUlq14n /pDgXJ47Lzqb5WNjDkdm+PZZOw+yLbzCt3PyTr0Hq6Q+26x4Z9HAUnFf5IT9rvdeE65sNFm4 onq/l+fRRVdfecgc31Z6esrFrRN6eRInrfkms/q+0AHV+dO+nVFiKc5INNRiLipOBADOjV+J 2AIAAA== X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Aug 2014 17:27:09 -0000 Previously, the upgrade was organized as two passes -- an upgrade pass, and a separate cleanup pass -- so the database was always in a valid state. This change substantially simplifies this code by performing the upgrade in a transaction and combining both passes in to one. This 1) eliminates a lot of duplicate code between the passes, 2) speeds up the upgrade process, 3) makes progress reporting more accurate, 4) eliminates the potential for stale data if the upgrade is interrupted during the cleanup pass, and 5) makes it easier to reason about the safety of the upgrade code. --- lib/database.cc | 67 ++++++--------------------------------------------------- 1 file changed, 7 insertions(+), 60 deletions(-) diff --git a/lib/database.cc b/lib/database.cc index 2b566f7..1e46fc8 100644 --- a/lib/database.cc +++ b/lib/database.cc @@ -1239,6 +1239,9 @@ notmuch_database_upgrade (notmuch_database_t *notmuch, timer_is_active = TRUE; } + /* Perform the upgrade in a transaction. */ + db->begin_transaction (true); + /* Set the target features so we write out changes in the desired * format. */ notmuch->features |= NOTMUCH_FEATURES_CURRENT; @@ -1270,6 +1273,7 @@ notmuch_database_upgrade (notmuch_database_t *notmuch, filename = _notmuch_message_talloc_copy_data (message); if (filename && *filename != '\0') { _notmuch_message_add_filename (message, filename); + _notmuch_message_clear_data (message); _notmuch_message_sync (message); } talloc_free (filename); @@ -1317,6 +1321,8 @@ notmuch_database_upgrade (notmuch_database_t *notmuch, NOTMUCH_FIND_CREATE, &status); notmuch_directory_set_mtime (directory, mtime); notmuch_directory_destroy (directory); + + db->delete_document (*p); } } } @@ -1357,67 +1363,8 @@ notmuch_database_upgrade (notmuch_database_t *notmuch, db->set_metadata ("features", _print_features (local, notmuch->features)); db->set_metadata ("version", STRINGIFY (NOTMUCH_DATABASE_VERSION)); - db->flush (); - - /* Now that the upgrade is complete we can remove the old data - * and documents that are no longer needed. */ - if (version < 1) { - notmuch_query_t *query = notmuch_query_create (notmuch, ""); - notmuch_messages_t *messages; - notmuch_message_t *message; - char *filename; - - for (messages = notmuch_query_search_messages (query); - notmuch_messages_valid (messages); - notmuch_messages_move_to_next (messages)) - { - if (do_progress_notify) { - progress_notify (closure, (double) count / total); - do_progress_notify = 0; - } - - message = notmuch_messages_get (messages); - - filename = _notmuch_message_talloc_copy_data (message); - if (filename && *filename != '\0') { - _notmuch_message_clear_data (message); - _notmuch_message_sync (message); - } - talloc_free (filename); - - notmuch_message_destroy (message); - } - notmuch_query_destroy (query); - } - - if (version < 1) { - Xapian::TermIterator t, t_end; - - t_end = notmuch->xapian_db->allterms_end ("XTIMESTAMP"); - - for (t = notmuch->xapian_db->allterms_begin ("XTIMESTAMP"); - t != t_end; - t++) - { - Xapian::PostingIterator p, p_end; - std::string term = *t; - - p_end = notmuch->xapian_db->postlist_end (term); - - for (p = notmuch->xapian_db->postlist_begin (term); - p != p_end; - p++) - { - if (do_progress_notify) { - progress_notify (closure, (double) count / total); - do_progress_notify = 0; - } - - db->delete_document (*p); - } - } - } + db->commit_transaction (); if (timer_is_active) { /* Now stop the timer. */ -- 2.0.0