From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id TUzENUWos17lLwAA0tVLHw (envelope-from ) for ; Thu, 07 May 2020 06:18:45 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id CC4nOVGos14jPwAAB5/wlQ (envelope-from ) for ; Thu, 07 May 2020 06:18:57 +0000 Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:470:142::17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 8AAFF940053 for ; Thu, 7 May 2020 06:18:55 +0000 (UTC) Received: from localhost ([::1]:52004 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jWZrz-0006kw-R0 for larch@yhetil.org; Thu, 07 May 2020 02:18:55 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:47700) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jWXao-0004im-Fa for guix-patches@gnu.org; Wed, 06 May 2020 23:53:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:58398) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jWXao-0008Rw-6n for guix-patches@gnu.org; Wed, 06 May 2020 23:53:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1jWXao-0008Bn-3M for guix-patches@gnu.org; Wed, 06 May 2020 23:53:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#41119] [PATCH] fix some issues with (guix nar) Resent-From: Caleb Ristvedt Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 07 May 2020 03:53:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 41119 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 41119@debbugs.gnu.org X-Debbugs-Original-To: guix-patches@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.158882354131415 (code B ref -1); Thu, 07 May 2020 03:53:01 +0000 Received: (at submit) by debbugs.gnu.org; 7 May 2020 03:52:21 +0000 Received: from localhost ([127.0.0.1]:41711 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jWXa9-0008Ad-5Y for submit@debbugs.gnu.org; Wed, 06 May 2020 23:52:21 -0400 Received: from lists.gnu.org ([209.51.188.17]:55696) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jWXa7-0008AV-7L for submit@debbugs.gnu.org; Wed, 06 May 2020 23:52:19 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:47658) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jWXa6-0004cH-VA for guix-patches@gnu.org; Wed, 06 May 2020 23:52:18 -0400 Received: from mail-qt1-x832.google.com ([2607:f8b0:4864:20::832]:42914) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jWXa4-0008Op-2l for guix-patches@gnu.org; Wed, 06 May 2020 23:52:18 -0400 Received: by mail-qt1-x832.google.com with SMTP id x12so3595237qts.9 for ; Wed, 06 May 2020 20:52:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cune-org.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:user-agent:mime-version; bh=3AdYdekXyXTLaGOuE3HlEv7/cYF5p0Z3FNAaPWpyx+I=; b=g2S725CbT6UQmum8dUX47341NwCUfN4SblB2dUaakvYUnbfZICtOuGeExBsiuwlu4X PZ6IbtoqL4MNnFptq7PUKA/aIINsjs8dTodP+37HVLgH719M+jyVSsoeWMGvFFxXYh+p /CXrRV43u4lq8fZ9sdieSEAveefhprCHA6z/1N6mLUoDippSxhvN/VmVnKPEecqUvkkL OsIrrdMPiMccNJ4e8/Er+NJr0Wk9gRGxMoM4uBvzF/Lp2Mf6LwyIT7dIdYZjCcGcUzyv dHME0jMSdjNVffRJ53dnV7J4bvzXYyMSVqcu6GLmOESObTK3A0Xg4aVhDaq6fBOXZtyK vhIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:user-agent :mime-version; bh=3AdYdekXyXTLaGOuE3HlEv7/cYF5p0Z3FNAaPWpyx+I=; b=EUynK0ra4Nh5bXzvqSKeljjNVaQq86qYmfSnNoGrpZ0ZVsG6+LRiwS8UoNyyL2R+yH EzD+9BLIdnjg0tDCeXLNGf1HH9Bsflwu/IvD0b+pZo2vz/PJ0xiEUHgcwhP4ZmY5srxf CQNCyjZ0DbtJ9juRWRIB53xbl7HEpo3ZbCoBj3CXarlTQsKEmsnei6P5HvDonvSdwjLP mJzEMzKKhFrpHmHz2P3fy/V6FFbOQKaP2zE9at24KMQlFDDwNKrmPy+XvbDqyQakReQB F9s9QvH2ZQMG59jWGmP7pkcd4SxlXFwmdNgzRg5vxbrfiy+AhRlPkLIX6YogpgOTOiAE d0XA== X-Gm-Message-State: AGi0Pub9lJA8cy5UBW126Fe/dTj7vZ2Ia2LkMQBn3bBTEQYJey+2LxPY ydyPk2yFSUwk2bSo4gI6qVkSPfIICwc= X-Google-Smtp-Source: APiQypJAkpo1EuAbtxVjc4RgMj4ODqn47XHz5fyqQCBTeaw5mZbvcxS6HARgmkEjF+76+l+S87vHWg== X-Received: by 2002:ac8:7309:: with SMTP id x9mr12207068qto.68.1588823534218; Wed, 06 May 2020 20:52:14 -0700 (PDT) Received: from GuixPotato ([208.89.170.24]) by smtp.gmail.com with ESMTPSA id g7sm1627117qth.9.2020.05.06.20.52.12 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 06 May 2020 20:52:13 -0700 (PDT) From: Caleb Ristvedt Date: Wed, 06 May 2020 22:52:11 -0500 Message-ID: <87h7wsqu50.fsf@cune.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/signed; boundary="==-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" Received-SPF: pass client-ip=2607:f8b0:4864:20::832; envelope-from=caleb.ristvedt@cune.org; helo=mail-qt1-x832.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-Spam-Score: -1.4 (-) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-Spam-Score: -2.4 (--) X-Mailman-Approved-At: Thu, 07 May 2020 02:18:49 -0400 X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Scanner: scn0 X-Spam-Score: -2.11 Authentication-Results: aspmx1.migadu.com; dkim=fail (rsa verify failed) header.d=cune-org.20150623.gappssmtp.com header.s=20150623 header.b=g2S725Cb; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 2001:470:142::17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Scan-Result: default: False [-2.11 / 13.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; GENERIC_REPUTATION(0.00)[-0.49754715272427]; DWL_DNSWL_BLOCKED(0.00)[2001:470:142::17:from]; R_SPF_ALLOW(-0.20)[+ip6:2001:470:142::/48:c]; IP_REPUTATION_HAM(0.00)[asn: 22989(0.11), country: US(-0.00), ip: 2001:470:142::17(-0.50)]; HAS_ATTACHMENT(0.00)[]; TO_DN_NONE(0.00)[]; R_DKIM_REJECT(1.00)[cune-org.20150623.gappssmtp.com:s=20150623]; MX_GOOD(-0.50)[cached: eggs.gnu.org]; DKIM_TRACE(0.00)[cune-org.20150623.gappssmtp.com:-]; MAILLIST(-0.20)[mailman]; SIGNED_PGP(-2.00)[]; FORGED_RECIPIENTS_MAILLIST(0.00)[]; RCVD_IN_DNSWL_FAIL(0.00)[2001:470:142::17:server fail]; MIME_TRACE(0.00)[0:+,1:+,2:+,3:+,4:+,5:~]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:22989, ipnet:2001:470:142::/48, country:US]; FROM_NEQ_ENVFROM(0.00)[caleb.ristvedt@cune.org,guix-patches-bounces@gnu.org]; MID_RHS_MATCH_FROM(0.00)[]; TAGGED_FROM(0.00)[larch=yhetil.org]; ARC_NA(0.00)[]; URIBL_BLOCKED(0.00)[cune.org:email]; FROM_HAS_DN(0.00)[]; MIME_GOOD(-0.20)[multipart/signed,multipart/mixed,text/plain,text/x-patch]; DMARC_NA(0.00)[cune.org]; HAS_LIST_UNSUB(-0.01)[]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_SEVEN(0.00)[11]; FORGED_SENDER_MAILLIST(0.00)[] X-TUID: m1jAKGSifaFB --==-=-= Content-Type: multipart/mixed; boundary="=-=-=" --=-=-= Content-Type: text/plain I noticed two issues while looking at (guix nar): 1. The proper store-lock-handling protocol isn't used in FINALIZE-STORE-FILE. Lock acquisition needs to check for a deletion token, retrying if it exists, and lock release needs to delete the lock file and write the deletion token. 2. WITH-TEMPORARY-STORE-FILE opens a new daemon connection every time it retries with a new filename, and only closes any of them after the body has completed. So if we retry 20 times, we get 20 concurrent daemon connections. This also prevents the call to LOOP from being a tail call. The attached patches resolve these issues. There are of course going to be more places we need to (properly) acquire and release store locks as guile-daemon code gets merged, but for now this should work as a bandaid fix. - reepca --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0001-nar-finalize-store-file-follows-proper-store-lock-pr.patch Content-Transfer-Encoding: quoted-printable Content-Description: FINALIZE-STORE-FILE fix From=20b2c66b443bd42e05820cfb3920c96f1894820587 Mon Sep 17 00:00:00 2001 From: Caleb Ristvedt Date: Wed, 6 May 2020 11:48:21 -0500 Subject: [PATCH 1/2] nar: 'finalize-store-file' follows proper store lock protocol. * guix/nar.scm (finalize-store-file): check for deletion token when acquiri= ng lock, write deletion token and delete lock file before releasing lock. =2D-- guix/nar.scm | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/guix/nar.scm b/guix/nar.scm index 29636aa0f8..f91af72879 100644 =2D-- a/guix/nar.scm +++ b/guix/nar.scm @@ -82,10 +82,19 @@ REFERENCES and DERIVER. When LOCK? is true, acquire exclusive locks on TA= RGET before attempting to register it; otherwise, assume TARGET's locks are alr= eady held." + ;; TODO: make this reusable + (define (acquire-lock filename) + (let ((port (lock-file filename))) + (if (zero? (stat:size (stat port))) + port + (begin + (close port) + (acquire-lock filename))))) + (with-database %default-database-file db (unless (path-id db target) (let ((lock (and lock? =2D (lock-file (string-append target ".lock"))))) + (acquire-lock (string-append target ".lock"))))) =20 (unless (path-id db target) ;; If FILE already exists, delete it (it's invalid anyway.) @@ -102,6 +111,9 @@ held." #:deriver deriver)) =20 (when lock? + (delete-file (string-append target ".lock")) + (display "d" lock) + (force-output lock) (unlock-file lock)))))) =20 (define (temporary-store-file) =2D-=20 2.26.2 --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0002-nar-with-temporary-store-file-uses-a-single-connecti.patch Content-Transfer-Encoding: quoted-printable Content-Description: WITH-TEMPORARY-STORE-FILE fix From=2043ee61b405b01038b3e7c84aba64521ab8a62236 Mon Sep 17 00:00:00 2001 From: Caleb Ristvedt Date: Wed, 6 May 2020 11:52:16 -0500 Subject: [PATCH 2/2] nar: 'with-temporary-store-file' uses a single connect= ion Previously the 'with-store' form was entered every time a different tempora= ry file was tried. This caused there to be as many simultaneous open connecti= ons as there were attempts, and prevented the (loop ...) call from being a tail call. This change fixes that. * guix/nar.scm (with-temporary-store-file): open connection once prior to entering the loop. =2D-- guix/nar.scm | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/guix/nar.scm b/guix/nar.scm index f91af72879..404cef8b97 100644 =2D-- a/guix/nar.scm +++ b/guix/nar.scm @@ -126,8 +126,8 @@ held." (define-syntax-rule (with-temporary-store-file name body ...) "Evaluate BODY with NAME bound to the file name of a temporary store item protected from GC." =2D (let loop ((name (temporary-store-file))) =2D (with-store store + (with-store store + (let loop ((name (temporary-store-file))) ;; Add NAME to the current process' roots. (Opening this connection= to ;; the daemon allows us to reuse its code that deals with the ;; per-process roots file.) =2D-=20 2.26.2 --=-=-=-- --==-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEdNapMPRLm4SepVYGwWaqSV9/GJwFAl6zhesACgkQwWaqSV9/ GJx4+Af6AyCZByhnQmIT9akppywz1Mut+YKV7IWkKBLifnbuqbaQl3faBbo4io6l 1++rq1FypEKUlepgQLwUGWfEdR21WPPxwL+LopllcqTklZO45WB7PsOr1wIQjvW1 /mxbbLIU9de37gzNl4caKO1Ijlra2fKmWzFqbSpy5h17dg2Q+1LFf0epLOwyOq7E m1fpvcaPp8IOj2X/Bb25XDLCopkJB5NYdJYoT8yAsXNQd3ORmTw4GnS9NoYTYVR8 7w6fdlAhDP6xjVKyJPwPos/u7T7loskn8wxBi85TFZG1RcAha1ib2TuAi83xEzam Y/pM8cn76UL7SWUuJ9i3IwJ58Rj75Q== =FFN3 -----END PGP SIGNATURE----- --==-=-=--