From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp11.migadu.com ([2001:41d0:306:2d92::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms9.migadu.com with LMTPS id kM56CP00C2Wt/QAA9RJhRA:P1 (envelope-from ) for ; Wed, 20 Sep 2023 20:07:57 +0200 Received: from aspmx1.migadu.com ([2001:41d0:306:2d92::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp11.migadu.com with LMTPS id kM56CP00C2Wt/QAA9RJhRA (envelope-from ) for ; Wed, 20 Sep 2023 20:07:57 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id CD6F53CEA8 for ; Wed, 20 Sep 2023 20:07:56 +0200 (CEST) Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20230601 header.b=JklLIp3z; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org"; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none) ARC-Seal: i=1; s=key1; d=yhetil.org; t=1695233276; a=rsa-sha256; cv=none; b=MiHJaiy9O88xxCuAAK1tYBQFdSOyYLxK8yQ2LiwD/YeuF4s3uTLU2QGWirBBeK2rKw+3pN YROBhnMIcL3hS3UQ09uCRC7iZPWOv+H4Fgryh0itKpBjE53WsNOaTrZR2XVklhv6KMHWEa 2/uKf+TCixfKPQbpvhsns2N1YO5w43iI9Me3qzDC8GVA8q4Wsxrm/6rQ2Y3LJGyXM6SwFj b3tUmNNoFUy4AwHdZ0LB7gnCgQKR3F7CBKnK2vOEGqlTWvf+J2biDlSyeHDAT9Mytt+UrH 0BzVgT5/NPFqMnXrPbGi/X3sQNKe5/sXR8X6r6EOmpY9f+NVBASHQ+5Cdp6tDA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1695233276; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=O2KeWgLd1w7IvPAAi7O9f8xdI1sZ7M0ORJqXKJHEbv4=; b=PefPK2lgm4sOs5GSh1GVPqGpq5niOxk/plAATHNUVArjMd9/WNxDb14k1OLxUzUh7RVggO YdRmtg0vgHKS5WUvQvfY46wrwtZKxFA12gzaDqhH8AXxNMhk6TaNisP0DTt8KLs1xWCmFb 1aaUG3Dd7M/mHn5hRjV6KFQ9mZeZP70XjGTiAGdB/TQT8dH7cQj39iXNyPLoMf8FQQHSNZ APYrGF/LUQBFTqJsqJPkcZ3Y9z+VIDYZHAIvtIyR7PL9SybtBQ2wuAvZRGGV+ZpulLruRz cEYLD3nrOFG6QLJ3QRRsujwJVUxfkUn/eaRTYCjtqSEC5m/svFQo2FVjWHFV0g== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20230601 header.b=JklLIp3z; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org"; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qj1Lj-0007n2-UV; Wed, 20 Sep 2023 13:50:55 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qj1Li-0007mt-GZ for guix-patches@gnu.org; Wed, 20 Sep 2023 13:50:54 -0400 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qj1Lh-0004EV-0v for guix-patches@gnu.org; Wed, 20 Sep 2023 13:50:53 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1qj1Lq-0006LS-83 for guix-patches@gnu.org; Wed, 20 Sep 2023 13:51:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#65866] [PATCH 0/8] Add built-in builder for Git checkouts Resent-From: Maxim Cournoyer Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Wed, 20 Sep 2023 17:51:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 65866 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: Josselin Poiret , Simon Tournier , Mathieu Othacehe , Tobias Geerinckx-Rice , Ricardo Wurmus , 65866@debbugs.gnu.org, Christopher Baines Received: via spool by 65866-submit@debbugs.gnu.org id=B65866.169523223924354 (code B ref 65866); Wed, 20 Sep 2023 17:51:02 +0000 Received: (at 65866) by debbugs.gnu.org; 20 Sep 2023 17:50:39 +0000 Received: from localhost ([127.0.0.1]:60322 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qj1LO-0006Kh-TJ for submit@debbugs.gnu.org; Wed, 20 Sep 2023 13:50:38 -0400 Received: from mail-qk1-x731.google.com ([2607:f8b0:4864:20::731]:49464) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qj1LG-0006KI-C6 for 65866@debbugs.gnu.org; Wed, 20 Sep 2023 13:50:34 -0400 Received: by mail-qk1-x731.google.com with SMTP id af79cd13be357-770ef903e80so714185a.3 for <65866@debbugs.gnu.org>; Wed, 20 Sep 2023 10:50:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1695232211; x=1695837011; darn=debbugs.gnu.org; h=content-transfer-encoding:mime-version:user-agent:message-id :in-reply-to:date:references:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=O2KeWgLd1w7IvPAAi7O9f8xdI1sZ7M0ORJqXKJHEbv4=; b=JklLIp3zzTAYFCYIy9pmzr35KOtKLfDCyBjIX3CQQeyqmxbE19suWK0/q4hjX1sIYK smIySxKH+sbDtmuLJ0zA3ciJqY/FqWhr1Wg/weyF783Pim93uKgb4UvcFx2XfH6Vdg3R vDzey6hNV8tVz0+MBNWm++Z6tyyeu9kBdMhvvirWyygLPkv7LnFuA2JOsi2fDE/lTOq2 rQLutahSidOWiurIahOJ2svoU3BwDsRlXKBn51JAK805LcJyNKoeH8dpoLS2tRYHC81w kI8izp1vQ8XVih5lzylgi+fA/od2cdKm1iW/FYUF0tH7BsrFJgJDaT7+PYuBLuUCB3wT 47tQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1695232211; x=1695837011; h=content-transfer-encoding:mime-version:user-agent:message-id :in-reply-to:date:references:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=O2KeWgLd1w7IvPAAi7O9f8xdI1sZ7M0ORJqXKJHEbv4=; b=lSuI+T+xilEnSa6PcTlvrPHNN7ynSqjS/ijU6mlGl73HTX1pNMA78ILLI0LckA3J3S HVCCMMLKZenAD+zX1QBsCHKOZCaN4ApLXdO6aS+RavNZ7MKarYqBekH7nZQNIXDTJICQ i3d8PGb0pW6SyZEg++m5qOSTxA6DKaLVmki8TtYinjzbV3j6JMeXhi0KzvDjfIkL3SJP mjxwCQro+eFhZ/feH8ZoVMgKe8x7Qwni7GBK7WM/5zJMSvqRv5OMqUd2owpJDt+qC1XC cAoduBiAAV/oijy0pPoxqf16kBcceMbFn7UMeOxBkriuT4mQ5yh2nCaGoZB0BIF5urof sb4w== X-Gm-Message-State: AOJu0YwZgfc0LppcHLuViITRmLYuG+vqHmGr1Xe+WqmYy6AeokrJzayE T/QBx7rf/Sb3Cgb8gGsYFF4= X-Google-Smtp-Source: AGHT+IHQEEYj43akXkemqMKQqmS2n9HVW5Ag6GFiV86fxYBKpQLFfougbH2eS5vw4phjCY/BoT/mxA== X-Received: by 2002:a05:620a:384f:b0:770:f1e1:202a with SMTP id po15-20020a05620a384f00b00770f1e1202amr2697056qkn.66.1695232211181; Wed, 20 Sep 2023 10:50:11 -0700 (PDT) Received: from hurd (dsl-10-134-200.b2b2c.ca. [72.10.134.200]) by smtp.gmail.com with ESMTPSA id hz11-20020a05622a678b00b0040fcf8c0aaasm4674753qtb.54.2023.09.20.10.50.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Sep 2023 10:50:10 -0700 (PDT) From: Maxim Cournoyer References: <2cd5b127be6d64e640e569f262cef3bbb89f58a6.1694441831.git.ludo@gnu.org> Date: Wed, 20 Sep 2023 13:50:09 -0400 In-Reply-To: <2cd5b127be6d64e640e569f262cef3bbb89f58a6.1694441831.git.ludo@gnu.org> ("Ludovic =?UTF-8?Q?Court=C3=A8s?="'s message of "Mon, 11 Sep 2023 16:25:25 +0200") Message-ID: <87ttro4tv2.fsf_-_@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: guix-patches-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Spam-Score: -0.60 X-Spam-Score: -0.60 X-Migadu-Queue-Id: CD6F53CEA8 X-Migadu-Scanner: mx2.migadu.com X-TUID: 5kpHWJ5kyqA+ Hello! Ludovic Court=C3=A8s writes: > Fixes . > > Longer-term this will remove Git from the derivation graph when its sole > use is to perform a checkout for a fixed-output derivation, thereby > breaking dependency cycles that can arise in these situations. > > * guix/git-download.scm (git-fetch): Rename to=E2=80=A6 > (git-fetch/in-band): =E2=80=A6 this. Deal with GIT or GUILE being #f. Nitpick, but I find this usage of dynamic default argument on top of default arguments inelegant; see my comments below for an alternative. > (git-fetch/built-in, built-in-builders*, git-fetch): New procedures. > * tests/builders.scm ("git-fetch, file URI"): New test. > --- > guix/git-download.scm | 68 +++++++++++++++++++++++++++++++++++++------ > tests/builders.scm | 29 +++++++++++++++++- > 2 files changed, 87 insertions(+), 10 deletions(-) > > diff --git a/guix/git-download.scm b/guix/git-download.scm > index f1f19397c6..505dff0a89 100644 > --- a/guix/git-download.scm > +++ b/guix/git-download.scm > @@ -27,6 +27,7 @@ (define-module (guix git-download) > #:use-module (guix records) > #:use-module (guix packages) > #:use-module (guix modules) > + #:use-module ((guix derivations) #:select (raw-derivation)) > #:autoload (guix build-system gnu) (standard-packages) > #:autoload (guix download) (%download-fallback-test) > #:autoload (git bindings) (libgit2-init!) > @@ -78,15 +79,19 @@ (define (git-package) > (let ((distro (resolve-interface '(gnu packages version-control)))) > (module-ref distro 'git-minimal))) >=20=20 > -(define* (git-fetch ref hash-algo hash > - #:optional name > - #:key (system (%current-system)) (guile (default-gui= le)) > - (git (git-package))) > - "Return a fixed-output derivation that fetches REF, a > -object. The output is expected to have recursive hash HASH of type > -HASH-ALGO (a symbol). Use NAME as the file name, or a generic name if #= f." > +(define* (git-fetch/in-band ref hash-algo hash > + #:optional name > + #:key (system (%current-system)) > + (guile (default-guile)) > + (git (git-package))) > + "Return a fixed-output derivation that performs a Git checkout of REF,= using > +GIT and GUILE (thus, said derivation depends on GIT and GUILE). > + > +This method is deprecated in favor of the \"builtin:git-download\" build= er. > +It will be removed when versions of guix-daemon implementing > +\"builtin:git-download\" will be sufficiently widespread." > (define inputs > - `(("git" ,git) > + `(("git" ,(or git (git-package))) Instead of using 'or' here to ensure git has a value, the default values should have been copied to the new definition of git-fetch. >=20=20 > ;; When doing 'git clone --recursive', we need sed, grep, etc. to = be > ;; available so that 'git submodule' works. > @@ -154,7 +159,8 @@ (define* (git-fetch ref hash-algo hash > #:recursive? recursive? > #:git-command "git"))))) >=20=20 > - (mlet %store-monad ((guile (package->derivation guile system))) > + (mlet %store-monad ((guile (package->derivation (or guile (default-gui= le)) > + system))) > (gexp->derivation (or name "git-checkout") build >=20=20 > ;; Use environment variables and a fixed script na= me so > @@ -181,6 +187,50 @@ (define* (git-fetch ref hash-algo hash > #:recursive? #t > #:guile-for-build guile))) >=20=20 > +(define* (git-fetch/built-in ref hash-algo hash > + #:optional name > + #:key (system (%current-system))) > + "Return a fixed-output derivation without any dependency that performs= a Git > +checkout of REF, using the \"builtin:git-download\" derivation builder." > + (raw-derivation (or name "git-checkout") "builtin:git-download" '() > + #:system system > + #:hash-algo hash-algo > + #:hash hash > + #:recursive? #t > + #:env-vars > + `(("url" . ,(object->string > + (match (%download-fallback-test) > + ('content-addressed-mirrors > + "https://example.org/does-not-exist") > + (_ > + (git-reference-url ref))))) > + ("commit" . ,(git-reference-commit ref)) > + ("recursive?" . ,(object->string > + (git-reference-recursive? ref)))) > + #:leaked-env-vars '("http_proxy" "https_proxy" > + "LC_ALL" "LC_MESSAGES" "LANG" > + "COLUMNS") > + #:local-build? #t)) > + > +(define built-in-builders* > + (store-lift built-in-builders)) > + > +(define* (git-fetch ref hash-algo hash > + #:optional name > + #:key (system (%current-system)) > + guile git) As mentioned above, I'd have kept the default values for guile and git here. > + "Return a fixed-output derivation that fetches REF, a > +object. The output is expected to have recursive hash HASH of type > +HASH-ALGO (a symbol). Use NAME as the file name, or a generic name if #= f." > + (mlet %store-monad ((builtins (built-in-builders*))) > + (if (member "git-download" builtins) > + (git-fetch/built-in ref hash-algo hash name > + #:system system) > + (git-fetch/in-band ref hash-algo hash name > + #:system system > + #:guile guile > + #:git git)))) > + > (define (git-version version revision commit) > "Return the version string for packages using git-download." > ;; git-version is almost exclusively executed while modules are being = loaded. > diff --git a/tests/builders.scm b/tests/builders.scm > index 0b5577c7a3..619caa5f31 100644 > --- a/tests/builders.scm > +++ b/tests/builders.scm > @@ -1,5 +1,5 @@ > ;;; GNU Guix --- Functional package management for GNU > -;;; Copyright =C2=A9 2012, 2013, 2014, 2015, 2018, 2019, 2021 Ludovic Co= urt=C3=A8s > +;;; Copyright =C2=A9 2012-2015, 2018-2019, 2021, 2023 Ludovic Court=C3= =A8s > ;;; Copyright =C2=A9 2021 Lars-Dominik Braun > ;;; > ;;; This file is part of GNU Guix. > @@ -20,6 +20,7 @@ >=20=20 > (define-module (tests builders) > #:use-module (guix download) > + #:use-module (guix git-download) > #:use-module (guix build-system) > #:use-module (guix build-system gnu) > #:use-module (guix build gnu-build-system) > @@ -31,9 +32,12 @@ (define-module (tests builders) > #:use-module (guix base32) > #:use-module (guix derivations) > #:use-module (gcrypt hash) > + #:use-module ((guix hash) #:select (file-hash*)) > #:use-module (guix tests) > + #:use-module (guix tests git) > #:use-module (guix packages) > #:use-module (gnu packages bootstrap) > + #:use-module ((ice-9 ftw) #:select (scandir)) > #:use-module (ice-9 match) > #:use-module (ice-9 textual-ports) > #:use-module (srfi srfi-1) > @@ -84,6 +88,29 @@ (define url-fetch* > (and (file-exists? out) > (valid-path? %store out)))) >=20=20 > +(test-equal "git-fetch, file URI" > + '("." ".." "a.txt" "b.scm") > + (let ((nonce (random-text))) > + (with-temporary-git-repository directory > + `((add "a.txt" ,nonce) > + (add "b.scm" "#t") > + (commit "Commit.") > + (tag "v1.0.0" "The tag.")) > + (run-with-store %store > + (mlet* %store-monad ((hash > + -> (file-hash* directory > + #:algorithm (hash-algorithm= sha256) > + #:recursive? #t)) > + (drv (git-fetch > + (git-reference > + (url (string-append "file://" direct= ory)) > + (commit "v1.0.0")) > + 'sha256 hash > + "git-fetch-test"))) > + (mbegin %store-monad > + (built-derivations (list drv)) > + (return (scandir (derivation->output-path drv))))))))) > + > (test-assert "gnu-build-system" > (build-system? gnu-build-system)) Pretty neat test! LGTM. You can add a 'Reviewed-by:' git trailer in Magit easily with 'C-u C-c C-r' :-) --=20 Thanks, Maxim