From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id FZ0XL/7Eb19LIgAA0tVLHw (envelope-from ) for ; Sat, 26 Sep 2020 22:47:26 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id qFt8Kv7Eb1+FZgAAB5/wlQ (envelope-from ) for ; Sat, 26 Sep 2020 22:47:26 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 3FD4A9404D5 for ; Sat, 26 Sep 2020 22:47:26 +0000 (UTC) Received: from localhost ([::1]:40858 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kMIyT-000410-2w for larch@yhetil.org; Sat, 26 Sep 2020 18:47:25 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39290) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kMIy6-00040s-KB for guix-patches@gnu.org; Sat, 26 Sep 2020 18:47:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:36466) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kMIy6-0006zN-Ag for guix-patches@gnu.org; Sat, 26 Sep 2020 18:47:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1kMIy6-000571-5U for guix-patches@gnu.org; Sat, 26 Sep 2020 18:47:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#43193] [PATCH] guix: Add --with-dependency-source option Resent-From: Jesse Gibbons Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 26 Sep 2020 22:47:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 43193 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: 43193@debbugs.gnu.org Received: via spool by 43193-submit@debbugs.gnu.org id=B43193.160116038919608 (code B ref 43193); Sat, 26 Sep 2020 22:47:02 +0000 Received: (at 43193) by debbugs.gnu.org; 26 Sep 2020 22:46:29 +0000 Received: from localhost ([127.0.0.1]:48012 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kMIxZ-00056B-3M for submit@debbugs.gnu.org; Sat, 26 Sep 2020 18:46:29 -0400 Received: from mail-pl1-f180.google.com ([209.85.214.180]:43456) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kMIxV-00055u-N8 for 43193@debbugs.gnu.org; Sat, 26 Sep 2020 18:46:27 -0400 Received: by mail-pl1-f180.google.com with SMTP id e4so1303170pln.10 for <43193@debbugs.gnu.org>; Sat, 26 Sep 2020 15:46:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language; bh=TfrrivFkJiF8MVWcZgjo+/2NOFiqjlWPvzjft7B3xtA=; b=DyuV8MMQeBgoJwOxU1SZOMIR9ifqKyZwlNE21V2yH5VowmOtJa51owm/CmEp2E7LVS N87thVy6GRmzBaDM6IQGm9tjhw6U7Od++viJbdww+UErPY8pq7eyNfZuWsk0X3aKEr+o ZpC504lI9M0Se+C/Tvvw77BpwdZizhhDtBspNoo0cY1qxtqti965Qh5fYNkBANfdAtZW ysTf71QNbGLHnqxh0cfPflWnDmo97Z1cNscIDRjS5xsGK/V11GJKG9sK0R50On6pK52v 2tYLhRDFAxi9XlfePrX0RjbHYqpP6rfN0l4QIXNJ4IyZLEKUeCbTHwMm8sBdsy8eHqHR Fi6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language; bh=TfrrivFkJiF8MVWcZgjo+/2NOFiqjlWPvzjft7B3xtA=; b=CLT0vSYBWySUkEw307aiepoDFd/IlGKkFCAuvBRJA5yK2Hk9peOuGKHjfP/WiXVf6u tBo9m91xmgzP9n5VCy4wQbv33HTcAM/0XzaGX/woQFVaJElALY3xLeMmNPGTOwMN9rIc c5GtieRa8Ww0ZJf/93DUb8Gsll+IXWrUueBBugwbbKnYMx+UjNdJQJ4InFXyGBJBwaot VYPL0++OblCo4r8JTxalcvWBDOT4E+T3LcRPt0GeGaW9sOUpriq9R8YzAwjcc9s3y2Uk DSnnwcdRl3HhWafZ8r3LUaXQrH/la2nX7R4NeLRk/xhb8wQdNp9PsZGFvccOe2akKcju MP/Q== X-Gm-Message-State: AOAM531toDAwNcMYGq7tiLiBVnNR/JbKpbGtfQkqGFcQIYLXK9wGA4vJ Ym6HAjtxZC+/ohKdN6iugotewzsgL6Y= X-Google-Smtp-Source: ABdhPJxYeT9DmWwA2Fg3CyWTCD0LkFG94iJAT38ACIYv0/vs6IJ69TS0pIyVVvSuMckX+bqKeXAfgw== X-Received: by 2002:a17:902:8e88:b029:d2:42fe:394b with SMTP id bg8-20020a1709028e88b02900d242fe394bmr5202316plb.41.1601160379257; Sat, 26 Sep 2020 15:46:19 -0700 (PDT) Received: from [192.168.1.25] ([38.141.58.134]) by smtp.gmail.com with ESMTPSA id r188sm2593094pfc.67.2020.09.26.15.46.17 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sat, 26 Sep 2020 15:46:18 -0700 (PDT) References: <9c3a00ba-19bf-a8e9-8803-d150e91f1b6e@gmail.com> <87y2lige5l.fsf@gnu.org> <7f52c72f-c280-b585-d1ad-ca012f804910@gmail.com> <87wo10s4j6.fsf@gnu.org> <4c890f8c-539c-1c9c-f558-7ba01cffa480@gmail.com> <87y2ld7s5b.fsf@gnu.org> From: Jesse Gibbons Message-ID: <9c60e5f2-e9fa-55c9-863f-cc3bccc2cf15@gmail.com> Date: Sat, 26 Sep 2020 16:46:16 -0600 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Icedove/68.12.0 MIME-Version: 1.0 In-Reply-To: <87y2ld7s5b.fsf@gnu.org> Content-Type: multipart/mixed; boundary="------------B5D816F1D9A948810AFD3908" Content-Language: en-US X-Spam-Score: 0.0 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-Spam-Score: -1.0 (-) X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Scanner: scn0 Authentication-Results: aspmx1.migadu.com; dkim=fail (rsa verify failed) header.d=gmail.com header.s=20161025 header.b=DyuV8MMQ; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Spam-Score: 0.09 X-TUID: jtnCrLr2slYB This is a multi-part message in MIME format. --------------B5D816F1D9A948810AFD3908 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Attached are the patches that make the --with-source option recursive, add documentation, add a test, adjust a test, and update the news. As expected, the changes I made are incompatible with the test "options->transformation, with-source, no matches". Since options->transformation, with-source does a recursive replacement, it returns a clone of the package in question. I have tried changing the comparison to eq?, eqv? and equal?, each returning false, so I settled on a (limited) comparison of the packages' attributes. On 9/13/20 6:55 AM, Ludovic Courtès wrote: > Hi, > > Jesse Gibbons skribis: > >> On 9/11/20 9:43 AM, Ludovic Courtès wrote: > [...] > >>> So maybe drop the second clause for non-recursive replacement, and drop >>> ‘transform-package-source’ as well. >> I included a fallback to transform-package-source because the >> following happens: >> >> $ ./pre-inst-env guix build --with-source=$(guix build --source hello) hello >> guix build: error: invalid source replacement specification: >> "/gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz" >> >> This does not fail when I fall back to the non-recursive logic. >> >> I can drop transform-package-source, but I will need to do some more >> hacking to figure out how the package name and version are parsed from >> the file name as described in the manual, and move it to the logic in >> transform-package-inputs/source. > Yes, that’d be nice. Namely, if you do: > > guix build hello --source=hello-1.2.3.tar.gz > > it should work just as now (from the source file name, we assume that > the source applies to package “hello”). > > Conversely, doing: > > guix build hello --source=xyz-hello-1.2.3.tar.gz > > would have no effect. It would not even emit a warning, unlike now. > >> I'm not going to have as much free time starting next week, so I might >> not be able to do that for a while, but I will try to get it done >> ASAP. > Sure, let’s stay in touch, I think we’re almost done! > > Thank you, > Ludo’. --------------B5D816F1D9A948810AFD3908 Content-Type: text/x-patch; charset=UTF-8; name="0001-guix-Make-with-source-option-recursive.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0001-guix-Make-with-source-option-recursive.patch" >From 2d2f6c97ad1deeb2fc8a214d992c7894a7c5e293 Mon Sep 17 00:00:00 2001 From: Jesse Gibbons Date: Thu, 3 Sep 2020 17:45:08 -0600 Subject: [PATCH 1/2] guix: Make --with-source option recursive * guix/scripts/build.scm: (transform-package-inputs/source): new function (evaluate-source-replacement-specs): new function (%transformations): change with-source to use evaluate-source-replacement-specs * doc/guix.texi (Package Transformation Options): document it. * tests/scripts-build.scm: (options->transformation, with-source, no matches): adjust to new expectations. (options->transformation, with-source, recursive): new test. --- doc/guix.texi | 4 +-- guix/scripts/build.scm | 61 ++++++++++++++++++++++++++++++++++++++--- tests/scripts-build.scm | 25 +++++++++++++++-- 3 files changed, 81 insertions(+), 9 deletions(-) diff --git a/doc/guix.texi b/doc/guix.texi index 82241b010a..3470ccc99c 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -9142,8 +9142,8 @@ without having to type in the definitions of package variants @itemx --with-source=@var{package}=@var{source} @itemx --with-source=@var{package}@@@var{version}=@var{source} Use @var{source} as the source of @var{package}, and @var{version} as -its version number. -@var{source} must be a file name or a URL, as for @command{guix +its version number. This replacement is applied recursively on all +dependencies. @var{source} must be a file name or a URL, as for @command{guix download} (@pxref{Invoking guix download}). When @var{package} is omitted, diff --git a/guix/scripts/build.scm b/guix/scripts/build.scm index 38e0516c95..a899f18a61 100644 --- a/guix/scripts/build.scm +++ b/guix/scripts/build.scm @@ -201,9 +201,9 @@ matching URIs given in SOURCES." (#f ;; Determine the package name and version from URI. (call-with-values - (lambda () - (hyphen-package-name->name+version - (tarball-base-name (basename uri)))) + (lambda () + (hyphen-package-name->name+version + (tarball-base-name (basename uri)))) (lambda (name version) (list name version uri)))) (index @@ -280,6 +280,26 @@ current 'gnutls' package, after which version 3.5.4 is grafted onto them." (rewrite obj) obj)))) +(define (transform-package-inputs/source replacement-specs) + "Return a procedure that, when passed a package, replaces its direct +dependencies according to REPLACEMENT-SPECS. REPLACEMENT-SPECS is a list of +strings like \"guile=/path/to/source\" or +\"guile=https://www.example.com/guile-source.tar.gz\" meaning that, any +dependency on a package called \"guile\" must be replaced with a dependency on a +\"guile\" built with the source at the specified location. SPECS may also +simply be a file location, in which case the package name and version are parsed +from the file name." + (lambda (store obj) + (let* ((replacements (evaluate-source-replacement-specs replacement-specs + (lambda* (old file #:optional version) + (package-with-source store old file version)))) + (rewrite (package-input-rewriting/spec replacements)) + (rewrite* (lambda (obj) + (rewrite obj)))) + (if (package? obj) + (rewrite* obj) + obj)))) + (define %not-equal (char-set-complement (char-set #\=))) @@ -314,6 +334,39 @@ syntax, or if a package it refers to could not be found." (leave (G_ "invalid replacement specification: ~s~%") spec)))) specs)) +(define (evaluate-source-replacement-specs specs proc) + "Parse SPECS, a list of strings like \"guile=/path/to/source\", and return a +list of package pairs, where (PROC PACKAGE URL) returns the replacement package. +Raise an error if an element of SPECS uses invalid syntax, or if a package it +refers to could not be found." + (define* (replacement file #:optional version) + (lambda (old) + (proc old file version))) + (map (lambda (spec) + (match (string-tokenize spec %not-equal) + ((package-spec file) + (let* ((spec-list (call-with-values + (lambda () + (package-specification->name+version+output package-spec)) + list)) + (name (list-ref spec-list 0)) + (version (list-ref spec-list 1))) + (cons name (replacement file version)))) + ((file) + (let* ((package-spec + (call-with-values + (lambda () + (hyphen-package-name->name+version + (tarball-base-name (basename file)))) + (lambda (name version) + (cons name version)))) + (name (car package-spec)) + (version (cdr package-spec))) + (cons name (replacement file version)))) + (_ + (leave (G_ "invalid source replacement specification: ~s~%") spec)))) + specs)) + (define (transform-package-source-branch replacement-specs) "Return a procedure that, when passed a package, replaces its direct dependencies according to REPLACEMENT-SPECS. REPLACEMENT-SPECS is a list of @@ -398,7 +451,7 @@ a checkout of the Git repository at the given URL." ;; key used in the option alist, and the cdr is the transformation ;; procedure; it is called with two arguments: the store, and a list of ;; things to build. - `((with-source . ,transform-package-source) + `((with-source . ,transform-package-inputs/source) (with-input . ,transform-package-inputs) (with-graft . ,transform-package-inputs/graft) (with-branch . ,transform-package-source-branch) diff --git a/tests/scripts-build.scm b/tests/scripts-build.scm index 32876e956a..40d7d03637 100644 --- a/tests/scripts-build.scm +++ b/tests/scripts-build.scm @@ -94,9 +94,9 @@ (let* ((port (open-output-string)) (new (parameterize ((guix-warning-port port)) (t store p)))) - (and (eq? new p) - (string-contains (get-output-string port) - "had no effect")))))) + (and (eq? (package-version new) (package-version p)) + (eq? (package-name new) (package-name p)) + (eq? (package-source new) (package-source p))))))) (test-assert "options->transformation, with-source, PKG=URI" (let* ((p (dummy-package "foo")) @@ -127,6 +127,25 @@ (add-to-store store (basename s) #t "sha256" s))))))) +(test-assert "options->transformation, with-source, recursive" + (let* ((q (dummy-package "foo")) + (p (dummy-package "guix.scm" + (inputs `(("foo" ,q))))) + (s (search-path %load-path "guix.scm")) + (f (string-append "foo@42.0=" s)) + (t (options->transformation `((with-source . ,f))))) + (with-store store + (let ((new (t store p))) + (and (not (eq? new p)) + (match (package-inputs new) + ((("foo" dep1)) + (and + (string=? (package-name dep1) "foo") + (string=? (package-version dep1) "42.0") + (string=? (package-source dep1) + (add-to-store store (basename s) #t + "sha256" s)))))))))) + (test-assert "options->transformation, with-input" (let* ((p (dummy-package "guix.scm" (inputs `(("foo" ,(specification->package "coreutils")) -- 2.28.0 --------------B5D816F1D9A948810AFD3908 Content-Type: text/x-patch; charset=UTF-8; name="0002-news-Add-entry-for-with-source.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-news-Add-entry-for-with-source.patch" >From 738fdb35af1449e245ce2e48c266c82f798d89af Mon Sep 17 00:00:00 2001 From: Jesse Gibbons Date: Sat, 26 Sep 2020 16:29:25 -0600 Subject: [PATCH 2/2] news: Add entry for "--with-source" * etc/news,scm: Add entry. --- etc/news.scm | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/etc/news.scm b/etc/news.scm index 1ef238ca2d..fcb195c679 100644 --- a/etc/news.scm +++ b/etc/news.scm @@ -13,6 +13,14 @@ (channel-news (version 0) + (entry (commit "2d2f6c97ad1deeb2fc8a214d992c7894a7c5e293") + (title (en "@option{--with-source} now recursive")) + (body (en "The @option{--with-source} common option now uses the +specified source for all matching dependencies of any packages guix is directed +to work with. This option is useful for all package maintainers, developers, +and, in general, all users who want guix to facilitate their rights to modify +their software and share their changes."))) + (entry (commit "a98712785e0b042a290420fd74e5a4a5da4fc68f") (title (en "New @command{guix git authenticate} command") (de "Neuer Befehl @command{guix git authenticate}") -- 2.28.0 --------------B5D816F1D9A948810AFD3908--