From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id eLEnNuMwX1++HQAA0tVLHw (envelope-from ) for ; Mon, 14 Sep 2020 08:59:15 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id iOacMOMwX1+5TQAA1q6Kng (envelope-from ) for ; Mon, 14 Sep 2020 08:59:15 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 26BCD940390 for ; Mon, 14 Sep 2020 08:59:14 +0000 (UTC) Received: from localhost ([::1]:49600 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kHkKO-0001tG-OP for larch@yhetil.org; Mon, 14 Sep 2020 04:59:12 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:43542) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kHkKE-0001q7-C1 for guix-patches@gnu.org; Mon, 14 Sep 2020 04:59:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:40943) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kHkKE-0006PE-22 for guix-patches@gnu.org; Mon, 14 Sep 2020 04:59:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1kHkKD-0006K6-W3 for guix-patches@gnu.org; Mon, 14 Sep 2020 04:59:01 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#42023] [PATCH] database: register-items: reduce transaction scope. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 14 Sep 2020 08:59:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 42023 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Caleb Ristvedt Cc: 42023@debbugs.gnu.org, Christopher Baines Received: via spool by 42023-submit@debbugs.gnu.org id=B42023.160007393024271 (code B ref 42023); Mon, 14 Sep 2020 08:59:01 +0000 Received: (at 42023) by debbugs.gnu.org; 14 Sep 2020 08:58:50 +0000 Received: from localhost ([127.0.0.1]:52486 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kHkK2-0006JO-Af for submit@debbugs.gnu.org; Mon, 14 Sep 2020 04:58:50 -0400 Received: from eggs.gnu.org ([209.51.188.92]:36650) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kHkJy-0006J6-Hy for 42023@debbugs.gnu.org; Mon, 14 Sep 2020 04:58:48 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:48170) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kHkJt-0006Nr-2A; Mon, 14 Sep 2020 04:58:41 -0400 Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=44160 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1kHkJs-0001bu-25; Mon, 14 Sep 2020 04:58:40 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <20200623163649.32444-1-mail@cbaines.net> <87366lxwcd.fsf@gnu.org> <87v9jg4aiz.fsf@cune.org> <87eeq3vbat.fsf@gnu.org> <87tuxu2sz6.fsf@gnu.org> <87bljka234.fsf@cune.org> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 29 Fructidor an 228 de la =?UTF-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Mon, 14 Sep 2020 10:58:30 +0200 In-Reply-To: <87bljka234.fsf@cune.org> (Caleb Ristvedt's message of "Sat, 08 Aug 2020 23:13:35 -0500") Message-ID: <877dswpweh.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-Spam-Score: -3.3 (---) X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Scanner: scn0 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Spam-Score: -1.01 X-TUID: w6y0zebNMc7d Hi Caleb, And apologies for the delay. I think we=E2=80=99ve drifted from the original patch and that=E2=80=99s be= come a tricky 7-patch series, which partly explains the delay=E2=80=94not that I=E2=80=99= m looking for an excuse. ;-) I decided to go ahead and apply some of these on your behalf. Comments below. Caleb Ristvedt skribis: > From 4c8f0cc50e2a1a33d9ce2f8e58cc426872676a7f Mon Sep 17 00:00:00 2001 > From: Caleb Ristvedt > Date: Wed, 24 Jun 2020 01:00:40 -0500 > Subject: [PATCH 1/6] .dir-locals.el: fix call-with-{retrying-}transaction > indenting. > > * .dir-locals.el (call-with-transaction, call-with-retrying-transaction): > change scheme-indent-function property from 2 to 1. Applied. > From 9717568f922e0921e5fdc320cbe6689768d29a29 Mon Sep 17 00:00:00 2001 > From: Caleb Ristvedt > Date: Sat, 8 Aug 2020 10:05:22 -0500 > Subject: [PATCH 2/6] deduplication: pass store directory to replace-with-= link. > > This causes with-writable-file to take into consideration the actual store > being used, as passed to 'deduplicate', rather than > whatever (%store-directory) may return. > > * guix/store/deduplication.scm (replace-with-link): new keyword argument > 'store'. Pass to with-writable-file. > (with-writable-file, call-with-writable-file): new store argument. > (deduplicate): pass store to replace-with-link. Applied. > From b992a3aaac7e3b30222e0bf1df09093f18e25e6a Mon Sep 17 00:00:00 2001 > From: Caleb Ristvedt > Date: Sat, 8 Aug 2020 11:25:57 -0500 > Subject: [PATCH 3/6] deduplication: retry on ENOENT. > > It's possible for the garbage collector to remove the "canonical" link af= ter > it's been detected as existing by 'deduplicate'. This would cause an ENO= ENT > error when replace-with-link attempts to create the temporary link. This > changes it so that it will properly handle that by retrying. Would that ENOENT cause an error, or just a missed deduplication opportunit= y? > * guix/store/deduplication.scm (replace-with-link): renamed to > canonicalize-with-link, now also handles the case where the target link > doesn't exist yet, and retries on ENOENT. > (deduplicate): modified to use canonicalize-with-link. There=E2=80=99s an issue with this patch. I gave it a spin (offloading a f= ew builds) and it got stuck in a infinite loop: --8<---------------cut here---------------start------------->8--- stat("/gnu/store/.links/0k63r0n3681r2gqd00blq4z5xd7cw1knv0x049p99f0pq31brhr= k", 0x7ffe43898cd0) =3D -1 ENOENT (Dosiero a=C5=AD dosierujo ne ekzistas) link("/gnu/store/83jy739bn644w3pnwgb5kwjig0kzs92f-libwps-0.4.12/lib/libwps-= 0.4.so", "/gnu/store/.links/0k63r0n3681r2gqd00blq4z5xd7cw1knv0x049p99f0pq31= brhrk") =3D -1 EEXIST (Dosiero jam ekzistas) stat("/gnu/store/.links/0k63r0n3681r2gqd00blq4z5xd7cw1knv0x049p99f0pq31brhr= k", 0x7ffe43898cd0) =3D -1 ENOENT (Dosiero a=C5=AD dosierujo ne ekzistas) link("/gnu/store/83jy739bn644w3pnwgb5kwjig0kzs92f-libwps-0.4.12/lib/libwps-= 0.4.so", "/gnu/store/.links/0k63r0n3681r2gqd00blq4z5xd7cw1knv0x049p99f0pq31= brhrk") =3D -1 EEXIST (Dosiero jam ekzistas) stat("/gnu/store/.links/0k63r0n3681r2gqd00blq4z5xd7cw1knv0x049p99f0pq31brhr= k", 0x7ffe43898cd0) =3D -1 ENOENT (Dosiero a=C5=AD dosierujo ne ekzistas) link("/gnu/store/83jy739bn644w3pnwgb5kwjig0kzs92f-libwps-0.4.12/lib/libwps-= 0.4.so", "/gnu/store/.links/0k63r0n3681r2gqd00blq4z5xd7cw1knv0x049p99f0pq31= brhrk") =3D -1 EEXIST (Dosiero jam ekzistas) --8<---------------cut here---------------end--------------->8--- I think we should work on reducing the complexity of that code (e.g., there are several layers of system-error handling). I=E2=80=99ve omitted it and I propose discussing it in a separate issue if = we need to. > From 6b7d011680c77642f24396be0eb0015c20413d1a Mon Sep 17 00:00:00 2001 > From: Caleb Ristvedt > Date: Sat, 8 Aug 2020 08:31:38 -0500 > Subject: [PATCH 4/6] nar: ensure finalization target is a temp root during > registration. > > Note that this is currently unnecessary, as finalize-store-file is only u= sed > from the offload hook, and the daemon process that spawned the offload ho= ok > will have already registered the derivation outputs as temp roots prior to > attempting to offload (see haveDerivation() in nix/libstore/build.cc). B= ut > it's necessary to ensure that the register-items invocation works properly > when finalize-store-file is used in other contexts. > > * guix/nar.scm (finalize-store-file): make target a temp root during exte= nt of > register-items invocation. [...] > + ;; TODO: don't use an RPC for this, add it to *this process's*= temp > + ;; roots file. > + (with-store store > + (add-temp-root store target) I agree that this is the right thing but as you note, it=E2=80=99s currently unnecessary in the context of =E2=80=98guix offload=E2=80=99, and I=E2=80= =99d rather avoid opening more connections to the daemon from =E2=80=98guix offload=E2=80=99 = and this increases load, pressure on the store database, etc. > From 55dd48e88d641bbc17b4d9484d6ee84acfb29766 Mon Sep 17 00:00:00 2001 > From: Caleb Ristvedt > Date: Wed, 8 Jul 2020 11:33:23 -0500 > Subject: [PATCH 5/6] database: document extra registration requirements. > > It's necessary that store items be locked and protected from garbage > collection while they are being registered. This documents that. > > * guix/store/database.scm (register-path, register-items): document GC > protection and locking requirements. Applied. > From 30afb453ce4eb161bb87645a0e6314e6af82a61f Mon Sep 17 00:00:00 2001 > From: Christopher Baines > Date: Tue, 23 Jun 2020 17:36:49 +0100 > Subject: [PATCH 6/6] database: register-items: reduce transaction scope. > > It was made transactional in a4678c6ba18d8dbd79d931f80426eebf61be7ebe, wi= th > the reasoning to prevent broken intermediate states from being visible. I > think this means something like an entry being in ValidPaths, but the Ref= s not > being inserted. > > Using a transaction for this makes sense, but I think using one single > transaction for the whole register-items call is unnecessary to avoid bro= ken > states from being visible, and could block other writes to the store data= base > while register-items is running. Because the deduplication and resetting > timestamps happens within the transaction as well, even though these thin= gs > don't involve the database, writes to the database will still be blocked = while > this is happening. > > To reduce the potential for register-items to block other writers to the > database for extended periods, this commit moves the transaction to just = wrap > the call to sqlite-register. This is the one place where writes occur, so= that > should prevent the broken intermediate states issue above. The one differ= ence > this will make is some of the registered items will be visible to other > connections while others may be still being added. I think this is OK, as= it's > equivalent to just registering different items. > > * guix/store/database.scm (register-items): Reduce transaction scope. Applied. Thanks Caleb & Chris! Ludo=E2=80=99.