From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id OAPqFvMv8l4sUwAA0tVLHw (envelope-from ) for ; Tue, 23 Jun 2020 16:38:11 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id kLy+EvMv8l4tHAAAB5/wlQ (envelope-from ) for ; Tue, 23 Jun 2020 16:38:11 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id D09AE940A21 for ; Tue, 23 Jun 2020 16:38:10 +0000 (UTC) Received: from localhost ([::1]:48216 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jnlw0-00033S-Ko for larch@yhetil.org; Tue, 23 Jun 2020 12:38:08 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:43884) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jnlvu-00033I-SU for guix-patches@gnu.org; Tue, 23 Jun 2020 12:38:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:53713) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jnlvu-0000BG-Ii for guix-patches@gnu.org; Tue, 23 Jun 2020 12:38:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1jnlvu-00022t-Dy for guix-patches@gnu.org; Tue, 23 Jun 2020 12:38:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#42023] [PATCH] database: register-items: reduce transaction scope. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 23 Jun 2020 16:38:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 42023 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 42023@debbugs.gnu.org X-Debbugs-Original-To: guix-patches@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.15929302237784 (code B ref -1); Tue, 23 Jun 2020 16:38:02 +0000 Received: (at submit) by debbugs.gnu.org; 23 Jun 2020 16:37:03 +0000 Received: from localhost ([127.0.0.1]:37026 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jnlux-00021U-9H for submit@debbugs.gnu.org; Tue, 23 Jun 2020 12:37:03 -0400 Received: from lists.gnu.org ([209.51.188.17]:45702) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jnluv-000215-Ef for submit@debbugs.gnu.org; Tue, 23 Jun 2020 12:37:01 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:43310) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jnluv-00029C-6Y for guix-patches@gnu.org; Tue, 23 Jun 2020 12:37:01 -0400 Received: from mira.cbaines.net ([2a01:7e00:e000:2f8:fd4d:b5c7:13fb:3d27]:44423) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jnlup-0008GS-0E for guix-patches@gnu.org; Tue, 23 Jun 2020 12:37:00 -0400 Received: from localhost (unknown [46.237.174.39]) by mira.cbaines.net (Postfix) with ESMTPSA id D82EA27BBE1 for ; Tue, 23 Jun 2020 17:36:51 +0100 (BST) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 5d911ac2 for ; Tue, 23 Jun 2020 16:36:49 +0000 (UTC) From: Christopher Baines Date: Tue, 23 Jun 2020 17:36:49 +0100 Message-Id: <20200623163649.32444-1-mail@cbaines.net> X-Mailer: git-send-email 2.26.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Received-SPF: pass client-ip=2a01:7e00:e000:2f8:fd4d:b5c7:13fb:3d27; envelope-from=mail@cbaines.net; helo=mira.cbaines.net X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/23 12:36:52 X-ACL-Warn: Detected OS = ??? X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, UNPARSEABLE_RELAY=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-Spam-Score: -1.3 (-) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-Spam-Score: -2.3 (--) X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Scanner: scn0 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Spam-Score: 3.99 X-TUID: 0y+R5pClLAV0 It was made transactional in a4678c6ba18d8dbd79d931f80426eebf61be7ebe, with the reasoning to prevent broken intermediate states from being visible. I think this means something like an entry being in ValidPaths, but the Refs not being inserted. Using a transaction for this makes sense, but I think using one single transaction for the whole register-items call is unnecessary to avoid broken states from being visible, and could block other writes to the store database while register-items is running. Because the deduplication and resetting timestamps happens within the transaction as well, even though these things don't involve the database, writes to the database will still be blocked while this is happening. To reduce the potential for register-items to block other writers to the database for extended periods, this commit moves the transaction to just wrap the call to sqlite-register. This is the one place where writes occur, so that should prevent the broken intermediate states issue above. The one difference this will make is some of the registered items will be visible to other connections while others may be still being added. I think this is OK, as it's equivalent to just registering different items. * guix/store/database.scm (register-items): Reduce transaction scope. --- guix/store/database.scm | 37 +++++++++++++++++++------------------ 1 file changed, 19 insertions(+), 18 deletions(-) diff --git a/guix/store/database.scm b/guix/store/database.scm index a38e4d7e52..767335bc0f 100644 --- a/guix/store/database.scm +++ b/guix/store/database.scm @@ -441,24 +441,25 @@ in the database; #f means \"now\". Write a progress report to LOG-PORT." (when reset-timestamps? (reset-timestamps real-file-name)) (let-values (((hash nar-size) (nar-sha256 real-file-name))) - (sqlite-register db #:path to-register - #:references (store-info-references item) - #:deriver (store-info-deriver item) - #:hash (string-append "sha256:" - (bytevector->base16-string hash)) - #:nar-size nar-size - #:time registration-time) + (call-with-retrying-transaction db + (lambda () + (sqlite-register db #:path to-register + #:references (store-info-references item) + #:deriver (store-info-deriver item) + #:hash (string-append + "sha256:" + (bytevector->base16-string hash)) + #:nar-size nar-size + #:time registration-time))) (when deduplicate? (deduplicate real-file-name hash #:store store-dir))))) - (call-with-retrying-transaction db - (lambda () - (let* ((prefix (format #f "registering ~a items" (length items))) - (progress (progress-reporter/bar (length items) - prefix log-port))) - (call-with-progress-reporter progress - (lambda (report) - (for-each (lambda (item) - (register db item) - (report)) - items))))))) + (let* ((prefix (format #f "registering ~a items" (length items))) + (progress (progress-reporter/bar (length items) + prefix log-port))) + (call-with-progress-reporter progress + (lambda (report) + (for-each (lambda (item) + (register db item) + (report)) + items))))) -- 2.26.2