From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id 2AdBLy9/8l6LGQAA0tVLHw (envelope-from ) for ; Tue, 23 Jun 2020 22:16:15 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id kBYaKy9/8l7dCgAA1q6Kng (envelope-from ) for ; Tue, 23 Jun 2020 22:16:15 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 2B0F0940667 for ; Tue, 23 Jun 2020 22:16:15 +0000 (UTC) Received: from localhost ([::1]:44808 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jnrDA-00077C-6p for larch@yhetil.org; Tue, 23 Jun 2020 18:16:12 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:36904) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jnrD0-00075h-Tt for guix-patches@gnu.org; Tue, 23 Jun 2020 18:16:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:54009) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jnrD0-000731-LH for guix-patches@gnu.org; Tue, 23 Jun 2020 18:16:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1jnrD0-000857-Gk for guix-patches@gnu.org; Tue, 23 Jun 2020 18:16:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#42023] [PATCH] database: register-items: reduce transaction scope. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 23 Jun 2020 22:16:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 42023 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Christopher Baines Cc: 42023@debbugs.gnu.org, Caleb Ristvedt Received: via spool by 42023-submit@debbugs.gnu.org id=B42023.159295054231038 (code B ref 42023); Tue, 23 Jun 2020 22:16:02 +0000 Received: (at 42023) by debbugs.gnu.org; 23 Jun 2020 22:15:42 +0000 Received: from localhost ([127.0.0.1]:37322 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jnrCg-00084Y-4p for submit@debbugs.gnu.org; Tue, 23 Jun 2020 18:15:42 -0400 Received: from eggs.gnu.org ([209.51.188.92]:50624) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jnrCd-00084H-Cd for 42023@debbugs.gnu.org; Tue, 23 Jun 2020 18:15:41 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:41073) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jnrCX-0006nb-Hq; Tue, 23 Jun 2020 18:15:33 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=58638 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1jnrCW-0005Qv-F3; Tue, 23 Jun 2020 18:15:33 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <20200623163649.32444-1-mail@cbaines.net> Date: Wed, 24 Jun 2020 00:15:30 +0200 In-Reply-To: <20200623163649.32444-1-mail@cbaines.net> (Christopher Baines's message of "Tue, 23 Jun 2020 17:36:49 +0100") Message-ID: <87366lxwcd.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-Spam-Score: -3.3 (---) X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Scanner: scn0 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Spam-Score: -1.01 X-TUID: q+REO5t4ubTC Hi, (+Cc: reepca) Christopher Baines skribis: > It was made transactional in a4678c6ba18d8dbd79d931f80426eebf61be7ebe, wi= th > the reasoning to prevent broken intermediate states from being visible. I > think this means something like an entry being in ValidPaths, but the Ref= s not > being inserted. > > Using a transaction for this makes sense, but I think using one single > transaction for the whole register-items call is unnecessary to avoid bro= ken > states from being visible, and could block other writes to the store data= base > while register-items is running. Because the deduplication and resetting > timestamps happens within the transaction as well, even though these thin= gs > don't involve the database, writes to the database will still be blocked = while > this is happening. > > To reduce the potential for register-items to block other writers to the > database for extended periods, this commit moves the transaction to just = wrap > the call to sqlite-register. This is the one place where writes occur, so= that > should prevent the broken intermediate states issue above. The one differ= ence > this will make is some of the registered items will be visible to other > connections while others may be still being added. I think this is OK, as= it's > equivalent to just registering different items. > > * guix/store/database.scm (register-items): Reduce transaction scope. [...] > + (call-with-retrying-transaction db > + (lambda () ^^ Too much indentation (maybe we miss a rule in .dir-locals.el?). > + (sqlite-register db #:path to-register > + #:references (store-info-references item) > + #:deriver (store-info-deriver item) > + #:hash (string-append > + "sha256:" > + (bytevector->base16-string hash)) > + #:nar-size nar-size > + #:time registration-time))) I think it would be good to have a 2-line summary of the rationale right above =E2=80=98call-with-retrying-transaction=E2=80=99. Two questions: 1. Can another process come and fiddle with TO-REGISTER while we=E2=80=99= re still in =E2=80=98reset-timestamps=E2=80=99? Or can GC happen while w= e=E2=80=99re in =E2=80=98reset-timestamps=E2=80=99 and delete TO-REGISTER and remove i= t from the database? I think none of these scenarios can happen, as long as we=E2=80=99ve taken = the .lock file for TO-REGISTER before, like =E2=80=98finalize-store-file=E2=80= =99 does. 2. After the transaction, TO-REGISTER is considered valid. But are the effects of the on-going deduplication observable, due to non-atomicity of some operation? I think the =E2=80=98replace-with-link=E2=80=99 dance is atomic, so we shou= ld be fine. Thoughts? Ludo=E2=80=99.