From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp12.migadu.com ([2001:41d0:403:4789::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms9.migadu.com with LMTPS id CGjcNXsVDmUmagAAauVa8A:P1 (envelope-from ) for ; Sat, 23 Sep 2023 00:30:20 +0200 Received: from aspmx1.migadu.com ([2001:41d0:403:4789::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp12.migadu.com with LMTPS id CGjcNXsVDmUmagAAauVa8A (envelope-from ) for ; Sat, 23 Sep 2023 00:30:19 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 8A16949D6A for ; Sat, 23 Sep 2023 00:30:19 +0200 (CEST) Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gnu.org header.s=fencepost-gnu-org header.b=j8iM+swS; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gnu.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1695421819; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=WG9PJCFhQ2r3BrsfN+53baekgsye0ORiyoUrEAdMA+w=; b=D1+P12JLQpg8vYYsVkqdjWUh14mxM5dGMg/LemvWYEo56+dta5ylupWZSRbRfkuY0md784 uwizNa5DScw4VPjGFe3X5NtNjV76JWs2nMJmBQ0xkPnC8PRRkI4n2P4K4JhKgL4UAqEJNP q7tBaUnA6QBwic3zc/wjJQaI0hcsQWm6cfStvU8mgBre69lD5OdAN6Y8qbZ6EWUqHucC+n LsONQdWGLkI/u4D1LEEPFqVF0oGqAItnWV+CaGUqW8AkV747LyAQB/sgDnG8EgNMFV2n8U 57TAAs90CkcBcgO+AbEP0nFtPXQCqYAf5jU+PzHfvvspC0hwymdyC/iWdBJmqA== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1695421819; a=rsa-sha256; cv=none; b=XrDxBhcYENBfrzgECNxFkAhjhrsXrOQMS5oKrQwxf2rPOpLj8TJdN6wIej2vu/+OCig/WH 2aiHfPXmKpDep4rWXq9h5MyhdJLGO52fSRH5DxnEcywQP7HgGSj6nLPCweiBj/S9RZ2CM3 7qFzscfeGkZd718V5SDrMHL49AORNMk4PdXix8uB9cTTuT/N9Go5BsDvrmpJ66OauXmT97 20VTL52RY4JfJqXxIG23SJgU9NtuveUa3kODH3dJQN2RZcCWYrvTeCNQTY/iPV86p95yhV Uhh1EPaNDqk6bynk9MahU2y1Q0XgDuZTQnYAdej2ibFqbJlCBpvGl9X+t8d2YQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gnu.org header.s=fencepost-gnu-org header.b=j8iM+swS; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gnu.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qjoes-0001zr-RK; Fri, 22 Sep 2023 18:29:58 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qjoeq-0001z1-6g for guix-patches@gnu.org; Fri, 22 Sep 2023 18:29:56 -0400 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qjoep-0005zJ-Ul; Fri, 22 Sep 2023 18:29:55 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1qjoex-0006o6-Fx; Fri, 22 Sep 2023 18:30:03 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#65866] [PATCH v2 4/8] daemon: Add =?UTF-8?Q?=E2=80=9Cgit-download=E2=80=9D?= built-in builder. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix@cbaines.net, dev@jpoiret.xyz, ludo@gnu.org, othacehe@gnu.org, rekado@elephly.net, zimon.toutoune@gmail.com, me@tobias.gr, guix-patches@gnu.org Resent-Date: Fri, 22 Sep 2023 22:30:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 65866 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 65866@debbugs.gnu.org Cc: Ludovic =?UTF-8?Q?Court=C3=A8s?= , Christopher Baines , Josselin Poiret , Ludovic =?UTF-8?Q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice X-Debbugs-Original-Xcc: Christopher Baines , Josselin Poiret , Ludovic =?UTF-8?Q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice Received: via spool by 65866-submit@debbugs.gnu.org id=B65866.169542176926068 (code B ref 65866); Fri, 22 Sep 2023 22:30:03 +0000 Received: (at 65866) by debbugs.gnu.org; 22 Sep 2023 22:29:29 +0000 Received: from localhost ([127.0.0.1]:37431 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qjoeM-0006mE-2k for submit@debbugs.gnu.org; Fri, 22 Sep 2023 18:29:29 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:45556) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qjodq-0006jN-RA for 65866@debbugs.gnu.org; Fri, 22 Sep 2023 18:28:56 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qjodZ-0005m8-Ma; Fri, 22 Sep 2023 18:28:37 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:References:In-Reply-To:Date:Subject:To: From; bh=WG9PJCFhQ2r3BrsfN+53baekgsye0ORiyoUrEAdMA+w=; b=j8iM+swSnP/Om4P+Yl31 tJVYR1aobXf3/nxJi/IkIT3qbS0lc7fNu0RFQyPbzxe4RC6vm8xDzqSPpQtnNHuMoONVxBH5ejV9W qXOMgd3brUXEBzByQTBEts5PDQ8T0bhM8KE6wSuY10aqUEWEvTz9jpSvcIej7tfBGzAamn+bsGtoB fG34tT0tkZmtiDj/kbAU726hPLBoL2Mp00/oKOXOLeElE9duyQPOj4AEiY+8sFkitKlMm8ESV4TUX s0BkB7KfDEDdWkO/PxAetTxVb7fuc7NA64s2BqjZ+tkDhAMOxTnQQsBYYNAxi7E+5mmjzkEgjganu 80IW4MtQIFbPpQ==; From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Date: Sat, 23 Sep 2023 00:28:00 +0200 Message-ID: <72b8f3ee7ef875273240f5dcbc8e9b1a5bd75515.1695421391.git.ludo@gnu.org> X-Mailer: git-send-email 2.41.0 In-Reply-To: References: <87o7hwas61.fsf@gnu.org> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: guix-patches-bounces+larch=yhetil.org@gnu.org X-Migadu-Country: US X-Migadu-Flow: FLOW_IN X-Spam-Score: -1.48 X-Migadu-Spam-Score: -1.48 X-Migadu-Scanner: mx1.migadu.com X-Migadu-Queue-Id: 8A16949D6A X-TUID: N4yaUuNbKfK9 From: Ludovic Courtès The new builder makes it possible to break cycles that occurs when the fixed-output derivation for the source of a dependency of ‘git’ would itself depend on ‘git’. * guix/scripts/perform-download.scm (perform-git-download): New procedure. (perform-download): Move fixed-output derivation check to… (guix-perform-download): … here. Invoke ‘perform-download’ or ‘perform-git-download’ depending on what ‘derivation-builder’ returns. * nix/libstore/builtins.cc (builtins): Add “git-download”. * tests/derivations.scm ("built-in-builders"): Update. ("'git-download' built-in builder") ("'git-download' built-in builder, invalid hash") ("'git-download' built-in builder, invalid commit") ("'git-download' built-in builder, not found"): New tests. --- guix/scripts/perform-download.scm | 49 ++++++++++++--- nix/libstore/builtins.cc | 5 +- tests/derivations.scm | 100 +++++++++++++++++++++++++++++- 3 files changed, 142 insertions(+), 12 deletions(-) diff --git a/guix/scripts/perform-download.scm b/guix/scripts/perform-download.scm index 3b29a3c81d..bb1e51aa30 100644 --- a/guix/scripts/perform-download.scm +++ b/guix/scripts/perform-download.scm @@ -1,5 +1,5 @@ ;;; GNU Guix --- Functional package management for GNU -;;; Copyright © 2016, 2017, 2018, 2020 Ludovic Courtès +;;; Copyright © 2016-2018, 2020, 2023 Ludovic Courtès ;;; ;;; This file is part of GNU Guix. ;;; @@ -21,7 +21,8 @@ (define-module (guix scripts perform-download) #:use-module (guix scripts) #:use-module (guix derivations) #:use-module ((guix store) #:select (derivation-path? store-path?)) - #:use-module (guix build download) + #:autoload (guix build download) (url-fetch) + #:autoload (guix build git) (git-fetch-with-fallback) #:use-module (ice-9 match) #:export (guix-perform-download)) @@ -61,10 +62,6 @@ (define* (perform-download drv output (drv-output (assoc-ref (derivation-outputs drv) "out")) (algo (derivation-output-hash-algo drv-output)) (hash (derivation-output-hash drv-output))) - (unless (and algo hash) - (leave (G_ "~a is not a fixed-output derivation~%") - (derivation-file-name drv))) - ;; We're invoked by the daemon, which gives us write access to OUTPUT. (when (url-fetch url output #:print-build-trace? print-build-trace? @@ -89,6 +86,30 @@ (define* (perform-download drv output (when (and executable (string=? executable "1")) (chmod output #o755)))))) +(define* (perform-git-download drv output + #:key print-build-trace?) + "Perform the download described by DRV, a fixed-output derivation, to +OUTPUT. + +Note: OUTPUT may differ from the 'out' value of DRV, notably for 'bmCheck' or +'bmRepair' builds." + (derivation-let drv ((url "url") + (commit "commit") + (recursive? "recursive?")) + (unless url + (leave (G_ "~a: missing Git URL~%") (derivation-file-name drv))) + (unless commit + (leave (G_ "~a: missing Git commit~%") (derivation-file-name drv))) + + (let* ((url (call-with-input-string url read)) + (recursive? (and recursive? + (call-with-input-string recursive? read))) + (drv-output (assoc-ref (derivation-outputs drv) "out")) + (algo (derivation-output-hash-algo drv-output)) + (hash (derivation-output-hash drv-output))) + (git-fetch-with-fallback url commit output + #:recursive? recursive?)))) + (define (assert-low-privileges) (when (zero? (getuid)) (leave (G_ "refusing to run with elevated privileges (UID ~a)~%") @@ -117,8 +138,20 @@ (define-command (guix-perform-download . args) (match args (((? derivation-path? drv) (? store-path? output)) (assert-low-privileges) - (let ((drv (read-derivation-from-file drv))) - (perform-download drv output #:print-build-trace? print-build-trace?))) + (let* ((drv (read-derivation-from-file drv)) + (download (match (derivation-builder drv) + ("builtin:download" perform-download) + ("builtin:git-download" perform-git-download) + (unknown (leave (G_ "~a: unknown builtin builder") + unknown)))) + (drv-output (assoc-ref (derivation-outputs drv) "out")) + (algo (derivation-output-hash-algo drv-output)) + (hash (derivation-output-hash drv-output))) + (unless (and hash algo) + (leave (G_ "~a is not a fixed-output derivation~%") + (derivation-file-name drv))) + + (download drv output #:print-build-trace? print-build-trace?))) (("--version") (show-version-and-exit)) (x diff --git a/nix/libstore/builtins.cc b/nix/libstore/builtins.cc index 4111ac4760..6bf467354a 100644 --- a/nix/libstore/builtins.cc +++ b/nix/libstore/builtins.cc @@ -1,5 +1,5 @@ /* GNU Guix --- Functional package management for GNU - Copyright (C) 2016, 2017, 2018, 2019 Ludovic Courtès + Copyright (C) 2016-2019, 2023 Ludovic Courtès This file is part of GNU Guix. @@ -58,7 +58,8 @@ static void builtinDownload(const Derivation &drv, static const std::map builtins = { - { "download", builtinDownload } + { "download", builtinDownload }, + { "git-download", builtinDownload } }; derivationBuilder lookupBuiltinBuilder(const std::string & name) diff --git a/tests/derivations.scm b/tests/derivations.scm index 66c777cfe7..e1312bd46b 100644 --- a/tests/derivations.scm +++ b/tests/derivations.scm @@ -24,10 +24,15 @@ (define-module (test-derivations) #:use-module (guix utils) #:use-module ((gcrypt hash) #:prefix gcrypt:) #:use-module (guix base32) + #:use-module ((guix git) #:select (with-repository)) #:use-module (guix tests) + #:use-module (guix tests git) #:use-module (guix tests http) #:use-module ((guix packages) #:select (package-derivation base32)) - #:use-module ((guix build utils) #:select (executable-file?)) + #:use-module ((guix build utils) #:select (executable-file? which)) + #:use-module ((guix hash) #:select (file-hash*)) + #:use-module ((git oid) #:select (oid->string)) + #:use-module ((git reference) #:select (reference-name->oid)) #:use-module (gnu packages bootstrap) #:use-module ((gnu packages guile) #:select (guile-1.8)) #:use-module (srfi srfi-1) @@ -195,7 +200,7 @@ (define* (directory-contents dir #:optional (slurp get-bytevector-all)) (stat:ino (lstat file2)))))))) (test-equal "built-in-builders" - '("download") + '("download" "git-download") (built-in-builders %store)) (test-assert "unknown built-in builder" @@ -290,6 +295,97 @@ (define* (directory-contents dir #:optional (slurp get-bytevector-all)) get-string-all) text)))))) +;; 'with-temporary-git-repository' relies on the 'git' command. +(unless (which (git-command)) (test-skip 1)) +(test-equal "'git-download' built-in builder" + `(("/a.txt" . "AAA") + ("/b.scm" . "#t")) + (let ((nonce (random-text))) + (with-temporary-git-repository directory + `((add "a.txt" "AAA") + (add "b.scm" "#t") + (commit ,nonce)) + (let* ((commit (with-repository directory repository + (oid->string + (reference-name->oid repository "HEAD")))) + (drv (derivation %store "git-download" + "builtin:git-download" '() + #:env-vars + `(("url" + . ,(object->string + (string-append "file://" directory))) + ("commit" . ,commit)) + #:hash-algo 'sha256 + #:hash (file-hash* directory + #:algorithm + (gcrypt:hash-algorithm + gcrypt:sha256) + #:recursive? #t) + #:recursive? #t))) + (build-derivations %store (list drv)) + (directory-contents (derivation->output-path drv) get-string-all))))) + +(unless (which (git-command)) (test-skip 1)) +(test-assert "'git-download' built-in builder, invalid hash" + (with-temporary-git-repository directory + `((add "a.txt" "AAA") + (add "b.scm" "#t") + (commit "Commit!")) + (let* ((commit (with-repository directory repository + (oid->string + (reference-name->oid repository "HEAD")))) + (drv (derivation %store "git-download" + "builtin:git-download" '() + #:env-vars + `(("url" + . ,(object->string + (string-append "file://" directory))) + ("commit" . ,commit)) + #:hash-algo 'sha256 + #:hash (gcrypt:sha256 #vu8()) + #:recursive? #t))) + (guard (c ((store-protocol-error? c) + (string-contains (store-protocol-error-message c) "failed"))) + (build-derivations %store (list drv)) + #f)))) + +(unless (which (git-command)) (test-skip 1)) +(test-assert "'git-download' built-in builder, invalid commit" + (with-temporary-git-repository directory + `((add "a.txt" "AAA") + (add "b.scm" "#t") + (commit "Commit!")) + (let* ((drv (derivation %store "git-download" + "builtin:git-download" '() + #:env-vars + `(("url" + . ,(object->string + (string-append "file://" directory))) + ("commit" + . "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa")) + #:hash-algo 'sha256 + #:hash (gcrypt:sha256 #vu8()) + #:recursive? #t))) + (guard (c ((store-protocol-error? c) + (string-contains (store-protocol-error-message c) "failed"))) + (build-derivations %store (list drv)) + #f)))) + +(test-assert "'git-download' built-in builder, not found" + (let* ((drv (derivation %store "git-download" + "builtin:git-download" '() + #:env-vars + `(("url" . "file:///does-not-exist.git") + ("commit" + . "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa")) + #:hash-algo 'sha256 + #:hash (gcrypt:sha256 #vu8()) + #:recursive? #t))) + (guard (c ((store-protocol-error? c) + (string-contains (store-protocol-error-message c) "failed"))) + (build-derivations %store (list drv)) + #f))) + (test-equal "derivation-name" "foo-0.0" (let ((drv (derivation %store "foo-0.0" %bash '()))) -- 2.41.0