From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp10.migadu.com ([2001:41d0:403:4789::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms9.migadu.com with LMTPS id aE5hLp4j/2SezQAAG6o9tA:P1 (envelope-from ) for ; Mon, 11 Sep 2023 16:26:38 +0200 Received: from aspmx1.migadu.com ([2001:41d0:403:4789::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp10.migadu.com with LMTPS id aE5hLp4j/2SezQAAG6o9tA (envelope-from ) for ; Mon, 11 Sep 2023 16:26:38 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 76167472E2 for ; Mon, 11 Sep 2023 16:26:38 +0200 (CEST) Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gnu.org header.s=fencepost-gnu-org header.b=SmO+MULr; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gnu.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1694442398; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=99DJrUUViq7vxNIOv5rl+/tDT4n8RnWxwe8vYvOdTYY=; b=s6QwciuhHBseqBzQ+VAzo8KT8rHL9vjSPkDbhwggJqEzzTXXRGPrZXgGej8J/33hAq607s cgkor4RFF1RXFQ0HzT9fhmhbil7Mu0ZyL9Ea/jcF++jBoYp07ZJpjB1G+zIZxidicgtwbq lmfH52va+DVKozrTGZOCW05z7rTNKPQ5li4pbb/7Q4auQKzo21MDPHSF+hxs5KZjQz9gcn TBkzM2JO6ARTCJ8XCT/QUm5U/BXsxbrStgdxTuJfMK+/MO094UZxaqee0E5AmgO/IB8AqM 3dUocxV4F1gO4KsxSD/WEEHD8AFpgY27RDaVLfBmkScZEM3j7MKdjiMuk3XDYg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gnu.org header.s=fencepost-gnu-org header.b=SmO+MULr; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gnu.org ARC-Seal: i=1; s=key1; d=yhetil.org; t=1694442398; a=rsa-sha256; cv=none; b=Xq8Pkg9J44+QUWVgRK2ZCcW2bbR/GTKetNcPeVwTH2Jqc/h5ATdi/v7LbxPEo0j5BTpIMb Aln6cTZipTmtI14cne84YHOy3ZTexKiszcdKAeROlDtKYgSlJCuKKYPybweZIt78pMxhL+ TxhlugAM3cRZ5ZQNdPdGQYigA1wNQc+LIuU0lWw1nZ7d5HMm8AbbNWCZWjgqiZ2KscRpB8 tloojGmJ5MxJYodnXfydkTskYJQfqRr88d4ocksdBoPNxumQBYPiEMm36kgTKF/XSxjALD KYgWRkbsnpKy+HT/OPGMAIYZRDFmmFywDY9LWIknHwLk18gMjV4l8RcKW3Y1wA== Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qfhrX-0003qI-7U; Mon, 11 Sep 2023 10:26:03 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qfhrT-0003oS-Fa for guix-patches@gnu.org; Mon, 11 Sep 2023 10:25:59 -0400 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qfhrT-00035S-6W; Mon, 11 Sep 2023 10:25:59 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1qfhrX-0006hO-7Q; Mon, 11 Sep 2023 10:26:03 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#65866] [PATCH 4/8] daemon: Add =?UTF-8?Q?=E2=80=9Cgit-download=E2=80=9D?= built-in builder. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix@cbaines.net, dev@jpoiret.xyz, ludo@gnu.org, othacehe@gnu.org, rekado@elephly.net, zimon.toutoune@gmail.com, me@tobias.gr, guix-patches@gnu.org Resent-Date: Mon, 11 Sep 2023 14:26:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 65866 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 65866@debbugs.gnu.org Cc: Ludovic =?UTF-8?Q?Court=C3=A8s?= , Christopher Baines , Josselin Poiret , Ludovic =?UTF-8?Q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice X-Debbugs-Original-Xcc: Christopher Baines , Josselin Poiret , Ludovic =?UTF-8?Q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice Received: via spool by 65866-submit@debbugs.gnu.org id=B65866.169444235925704 (code B ref 65866); Mon, 11 Sep 2023 14:26:03 +0000 Received: (at 65866) by debbugs.gnu.org; 11 Sep 2023 14:25:59 +0000 Received: from localhost ([127.0.0.1]:54276 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qfhrS-0006gP-MT for submit@debbugs.gnu.org; Mon, 11 Sep 2023 10:25:59 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:55002) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qfhrM-0006fF-QW for 65866@debbugs.gnu.org; Mon, 11 Sep 2023 10:25:53 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qfhrC-0002zo-HX; Mon, 11 Sep 2023 10:25:42 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:References:In-Reply-To:Date:Subject:To: From; bh=99DJrUUViq7vxNIOv5rl+/tDT4n8RnWxwe8vYvOdTYY=; b=SmO+MULr6zuErUGBxYWQ 4NFu1jaKbhmVJdtar2NLvbY3+gAaBY6E1lgfxnSz7ZaS9jI03sNI+vQ+8cfJawun82NxH/LGDHItc YSgcwzszb9CW8hZS7qZ+gPW64CZgGPU9L5s6tvGURPcRW1BQL5ZJ9eCfUjvX/QaAOTycGYHJuKXem 5z1L4n7m7SmlnyGgq9qWGJQayds/9l5zu7eLujHraSbQz00RgDndmLSNBCUHsIInht13zTHXQaVSk /2u9qDe+bX3nEdaBbg6ozJs2ixYzX0OqL6dZQv2micbTZXqldF/kXyUONWPhX4DzgiJqt7WbQn7XB d2cbcJNN8HokxQ==; From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Date: Mon, 11 Sep 2023 16:25:22 +0200 Message-ID: <3c42634cb47dd7eaa81a198bc2d097ca74a973ed.1694441831.git.ludo@gnu.org> X-Mailer: git-send-email 2.41.0 In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: guix-patches-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Scanner: mx1.migadu.com X-Migadu-Spam-Score: -1.42 X-Spam-Score: -1.42 X-Migadu-Queue-Id: 76167472E2 X-TUID: t05sCAkVO8Eo From: Ludovic Courtès The new builder makes it possible to break cycles that occurs when the fixed-output derivation for the source of a dependency of ‘git’ would itself depend on ‘git’. * guix/scripts/perform-download.scm (perform-git-download): New procedure. (perform-download): Move fixed-output derivation check to… (guix-perform-download): … here. Invoke ‘perform-download’ or ‘perform-git-download’ depending on what ‘derivation-builder’ returns. * nix/libstore/builtins.cc (builtins): Add “git-download”. * tests/derivations.scm ("built-in-builders"): Update. ("'git-download' built-in builder") ("'git-download' built-in builder, invalid hash") ("'git-download' built-in builder, invalid commit") ("'git-download' built-in builder, not found"): New tests. --- guix/scripts/perform-download.scm | 52 +++++++++++++--- nix/libstore/builtins.cc | 5 +- tests/derivations.scm | 100 +++++++++++++++++++++++++++++- 3 files changed, 145 insertions(+), 12 deletions(-) diff --git a/guix/scripts/perform-download.scm b/guix/scripts/perform-download.scm index c8f044e82e..a287e97528 100644 --- a/guix/scripts/perform-download.scm +++ b/guix/scripts/perform-download.scm @@ -1,5 +1,5 @@ ;;; GNU Guix --- Functional package management for GNU -;;; Copyright © 2016, 2017, 2018, 2020 Ludovic Courtès +;;; Copyright © 2016-2018, 2020, 2023 Ludovic Courtès ;;; ;;; This file is part of GNU Guix. ;;; @@ -21,7 +21,8 @@ (define-module (guix scripts perform-download) #:use-module (guix scripts) #:use-module (guix derivations) #:use-module ((guix store) #:select (derivation-path? store-path?)) - #:use-module (guix build download) + #:autoload (guix build download) (url-fetch) + #:autoload (guix build git) (git-fetch-with-fallback) #:use-module (ice-9 match) #:export (guix-perform-download)) @@ -64,10 +65,6 @@ (define* (perform-download drv #:optional output (drv-output (assoc-ref (derivation-outputs drv) "out")) (algo (derivation-output-hash-algo drv-output)) (hash (derivation-output-hash drv-output))) - (unless (and algo hash) - (leave (G_ "~a is not a fixed-output derivation~%") - (derivation-file-name drv))) - ;; We're invoked by the daemon, which gives us write access to OUTPUT. (when (url-fetch url output #:print-build-trace? print-build-trace? @@ -92,6 +89,33 @@ (define* (perform-download drv #:optional output (when (and executable (string=? executable "1")) (chmod output #o755)))))) +(define* (perform-git-download drv #:optional output + #:key print-build-trace?) + "Perform the download described by DRV, a fixed-output derivation, to +OUTPUT. + +Note: Unless OUTPUT is #f, we don't read the value of 'out' in DRV since the +actual output is different from that when we're doing a 'bmCheck' or +'bmRepair' build." + (derivation-let drv ((output* "out") + (url "url") + (commit "commit") + (recursive? "recursive?")) + (unless url + (leave (G_ "~a: missing Git URL~%") (derivation-file-name drv))) + (unless commit + (leave (G_ "~a: missing Git commit~%") (derivation-file-name drv))) + + (let* ((output (or output output*)) + (url (call-with-input-string url read)) + (recursive? (and recursive? + (call-with-input-string recursive? read))) + (drv-output (assoc-ref (derivation-outputs drv) "out")) + (algo (derivation-output-hash-algo drv-output)) + (hash (derivation-output-hash drv-output))) + (git-fetch-with-fallback url commit output + #:recursive? recursive?)))) + (define (assert-low-privileges) (when (zero? (getuid)) (leave (G_ "refusing to run with elevated privileges (UID ~a)~%") @@ -120,8 +144,20 @@ (define-command (guix-perform-download . args) (match args (((? derivation-path? drv) (? store-path? output)) (assert-low-privileges) - (let ((drv (read-derivation-from-file drv))) - (perform-download drv output #:print-build-trace? print-build-trace?))) + (let* ((drv (read-derivation-from-file drv)) + (download (match (derivation-builder drv) + ("builtin:download" perform-download) + ("builtin:git-download" perform-git-download) + (unknown (leave (G_ "~a: unknown builtin builder") + unknown)))) + (drv-output (assoc-ref (derivation-outputs drv) "out")) + (algo (derivation-output-hash-algo drv-output)) + (hash (derivation-output-hash drv-output))) + (unless (and hash algo) + (leave (G_ "~a is not a fixed-output derivation~%") + (derivation-file-name drv))) + + (download drv output #:print-build-trace? print-build-trace?))) (("--version") (show-version-and-exit)) (x diff --git a/nix/libstore/builtins.cc b/nix/libstore/builtins.cc index 4111ac4760..6bf467354a 100644 --- a/nix/libstore/builtins.cc +++ b/nix/libstore/builtins.cc @@ -1,5 +1,5 @@ /* GNU Guix --- Functional package management for GNU - Copyright (C) 2016, 2017, 2018, 2019 Ludovic Courtès + Copyright (C) 2016-2019, 2023 Ludovic Courtès This file is part of GNU Guix. @@ -58,7 +58,8 @@ static void builtinDownload(const Derivation &drv, static const std::map builtins = { - { "download", builtinDownload } + { "download", builtinDownload }, + { "git-download", builtinDownload } }; derivationBuilder lookupBuiltinBuilder(const std::string & name) diff --git a/tests/derivations.scm b/tests/derivations.scm index 66c777cfe7..e1312bd46b 100644 --- a/tests/derivations.scm +++ b/tests/derivations.scm @@ -24,10 +24,15 @@ (define-module (test-derivations) #:use-module (guix utils) #:use-module ((gcrypt hash) #:prefix gcrypt:) #:use-module (guix base32) + #:use-module ((guix git) #:select (with-repository)) #:use-module (guix tests) + #:use-module (guix tests git) #:use-module (guix tests http) #:use-module ((guix packages) #:select (package-derivation base32)) - #:use-module ((guix build utils) #:select (executable-file?)) + #:use-module ((guix build utils) #:select (executable-file? which)) + #:use-module ((guix hash) #:select (file-hash*)) + #:use-module ((git oid) #:select (oid->string)) + #:use-module ((git reference) #:select (reference-name->oid)) #:use-module (gnu packages bootstrap) #:use-module ((gnu packages guile) #:select (guile-1.8)) #:use-module (srfi srfi-1) @@ -195,7 +200,7 @@ (define* (directory-contents dir #:optional (slurp get-bytevector-all)) (stat:ino (lstat file2)))))))) (test-equal "built-in-builders" - '("download") + '("download" "git-download") (built-in-builders %store)) (test-assert "unknown built-in builder" @@ -290,6 +295,97 @@ (define* (directory-contents dir #:optional (slurp get-bytevector-all)) get-string-all) text)))))) +;; 'with-temporary-git-repository' relies on the 'git' command. +(unless (which (git-command)) (test-skip 1)) +(test-equal "'git-download' built-in builder" + `(("/a.txt" . "AAA") + ("/b.scm" . "#t")) + (let ((nonce (random-text))) + (with-temporary-git-repository directory + `((add "a.txt" "AAA") + (add "b.scm" "#t") + (commit ,nonce)) + (let* ((commit (with-repository directory repository + (oid->string + (reference-name->oid repository "HEAD")))) + (drv (derivation %store "git-download" + "builtin:git-download" '() + #:env-vars + `(("url" + . ,(object->string + (string-append "file://" directory))) + ("commit" . ,commit)) + #:hash-algo 'sha256 + #:hash (file-hash* directory + #:algorithm + (gcrypt:hash-algorithm + gcrypt:sha256) + #:recursive? #t) + #:recursive? #t))) + (build-derivations %store (list drv)) + (directory-contents (derivation->output-path drv) get-string-all))))) + +(unless (which (git-command)) (test-skip 1)) +(test-assert "'git-download' built-in builder, invalid hash" + (with-temporary-git-repository directory + `((add "a.txt" "AAA") + (add "b.scm" "#t") + (commit "Commit!")) + (let* ((commit (with-repository directory repository + (oid->string + (reference-name->oid repository "HEAD")))) + (drv (derivation %store "git-download" + "builtin:git-download" '() + #:env-vars + `(("url" + . ,(object->string + (string-append "file://" directory))) + ("commit" . ,commit)) + #:hash-algo 'sha256 + #:hash (gcrypt:sha256 #vu8()) + #:recursive? #t))) + (guard (c ((store-protocol-error? c) + (string-contains (store-protocol-error-message c) "failed"))) + (build-derivations %store (list drv)) + #f)))) + +(unless (which (git-command)) (test-skip 1)) +(test-assert "'git-download' built-in builder, invalid commit" + (with-temporary-git-repository directory + `((add "a.txt" "AAA") + (add "b.scm" "#t") + (commit "Commit!")) + (let* ((drv (derivation %store "git-download" + "builtin:git-download" '() + #:env-vars + `(("url" + . ,(object->string + (string-append "file://" directory))) + ("commit" + . "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa")) + #:hash-algo 'sha256 + #:hash (gcrypt:sha256 #vu8()) + #:recursive? #t))) + (guard (c ((store-protocol-error? c) + (string-contains (store-protocol-error-message c) "failed"))) + (build-derivations %store (list drv)) + #f)))) + +(test-assert "'git-download' built-in builder, not found" + (let* ((drv (derivation %store "git-download" + "builtin:git-download" '() + #:env-vars + `(("url" . "file:///does-not-exist.git") + ("commit" + . "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa")) + #:hash-algo 'sha256 + #:hash (gcrypt:sha256 #vu8()) + #:recursive? #t))) + (guard (c ((store-protocol-error? c) + (string-contains (store-protocol-error-message c) "failed"))) + (build-derivations %store (list drv)) + #f))) + (test-equal "derivation-name" "foo-0.0" (let ((drv (derivation %store "foo-0.0" %bash '()))) -- 2.41.0