* Unexpected --export-manifest with simple transformations @ 2021-02-10 1:36 zimoun 2021-02-10 23:01 ` Ludovic Courtès 0 siblings, 1 reply; 5+ messages in thread From: zimoun @ 2021-02-10 1:36 UTC (permalink / raw) To: Guix Devel, Ludovic Courtès Hi, If the transformations are in the manifest.scm file, then they are lost. For example, consider: --8<---------------cut here---------------start------------->8--- $ guix package \ -p /tmp/profile-cli \ -i python python-numpy \ hello --with-c-toolchain=hello=gcc-toolchain@10 guix package: warning: transformation 'with-c-toolchain' had no effect on python@3.8.2 guix package: warning: transformation 'with-c-toolchain' had no effect on python-numpy@1.17.3 [...] $ guix package -p /tmp/profile-cli --export-manifest > /tmp/manifest.scm $ cat /tmp/manifest.scm ;; This "manifest" file can be passed to 'guix package -m' to reproduce ;; the content of your profile. This is "symbolic": it only specifies ;; package names. To reproduce the exact same profile, you also need to ;; capture the channels being used, as returned by "guix describe". ;; See the "Replicating Guix" section in the manual. (use-modules (guix transformations)) (define transform1 (options->transformation '((with-c-toolchain . "hello=gcc-toolchain@10")))) (packages->manifest (list (transform1 (specification->package "hello")) (specification->package "python-numpy") (specification->package "python"))) --8<---------------cut here---------------end--------------->8--- So far, so good. Then let use this manifest to create a profile. $ guix package -p /tmp/profile-exported -m /tmp/manifest.scm and the issue is that ’/tmp/profile-exported/manifest’ does not contain the transformation. Other said: --8<---------------cut here---------------start------------->8--- $ guix package -p /tmp/profile-exported --export-manifest ;; This "manifest" file can be passed to 'guix package -m' to reproduce ;; the content of your profile. This is "symbolic": it only specifies ;; package names. To reproduce the exact same profile, you also need to ;; capture the channels being used, as returned by "guix describe". ;; See the "Replicating Guix" section in the manual. (specifications->manifest (list "hello" "python-numpy" "python")) --8<---------------cut here---------------end--------------->8--- Note that obviously: --8<---------------cut here---------------start------------->8--- $ ls -l /tmp/profile-{cli,exported}/bin/ | grep hello lrwxrwxrwx 3 root root 64 Jan 1 1970 hello -> /gnu/store/qi7pqqsxhbwmy75hl43j7l0aw1xr7r42-hello-2.10/bin/hello lrwxrwxrwx 3 root root 64 Jan 1 1970 hello -> /gnu/store/qi7pqqsxhbwmy75hl43j7l0aw1xr7r42-hello-2.10/bin/hello --8<---------------cut here---------------end--------------->8--- Well, is it a bug or a feature? ;-) From my understanding, it makes sense to save the transformations from a manifest. But it is not clear what should be the good solution for that because it is not easy; again from my understanding. The obvious answer is: do not loose the manifest.scm file! :-) Sure, I am working in the context where Alice produced a Docker image with “guix pack -f docker --save-provenance -m manifest.scm” where manifest.scm uses transformations. Then since Alice is post-doc and goes doing other stuff elsewhere, the source probably vanishes. Bob wants to rebuild the Docker image. It is possible for simple case as I already illustrated it. :-) To be concrete, I am trying to write down a non-trivial example using a custom package and a transformation for a blog post and/or Cookbook recipe. Cheers, simon ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Unexpected --export-manifest with simple transformations 2021-02-10 1:36 Unexpected --export-manifest with simple transformations zimoun @ 2021-02-10 23:01 ` Ludovic Courtès 2021-02-11 5:33 ` zimoun 0 siblings, 1 reply; 5+ messages in thread From: Ludovic Courtès @ 2021-02-10 23:01 UTC (permalink / raw) To: zimoun; +Cc: Guix Devel Hi, zimoun <zimon.toutoune@gmail.com> skribis: > So far, so good. Then let use this manifest to create a profile. > > $ guix package -p /tmp/profile-exported -m /tmp/manifest.scm > > and the issue is that ’/tmp/profile-exported/manifest’ does not contain > the transformation. Other said: > > $ guix package -p /tmp/profile-exported --export-manifest > ;; This "manifest" file can be passed to 'guix package -m' to reproduce > ;; the content of your profile. This is "symbolic": it only specifies > ;; package names. To reproduce the exact same profile, you also need to > ;; capture the channels being used, as returned by "guix describe". > ;; See the "Replicating Guix" section in the manual. > > (specifications->manifest > (list "hello" "python-numpy" "python")) That’s because when using ‘-m’, transformations are not recorded. ‘--export-manifest’ can be thought of as a disassembler; in this case, it lacks debug info to perform better. HTH, Ludo’. ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Unexpected --export-manifest with simple transformations 2021-02-10 23:01 ` Ludovic Courtès @ 2021-02-11 5:33 ` zimoun 2021-02-22 22:41 ` Ludovic Courtès 0 siblings, 1 reply; 5+ messages in thread From: zimoun @ 2021-02-11 5:33 UTC (permalink / raw) To: Ludovic Courtès; +Cc: Guix Devel Hi Ludo, On Thu, 11 Feb 2021 at 00:01, Ludovic Courtès <ludo@gnu.org> wrote: > That’s because when using ‘-m’, transformations are not recorded. Yes. The question is: is it a conscientious choice or a missing feature? It makes sense to save the transformations from a manifest, IMHO. But it is not clear what should be the good solution for that because it is not easy for the general case; from my understanding. For instance, one could easily imagine this (restricted) typical manifest with transformation, --8<---------------cut here---------------start------------->8--- (use-modules (guix transformations)) (define transform1 (options->transformation '((with-c-toolchain . "hello=gcc-toolchain@10")))) (packages->manifest (list (transform1 (specification->package "hello")) (specification->package "python-numpy") (specification->package "python"))) --8<---------------cut here---------------end--------------->8--- should record the transformation in <profile>/manifest. Motivated by 1. consistency with the «equivalent» command-line and 2. the use-case “guix pack -f docker --save-provenance -m’. Other said, the option ’-m + --export-manifest’ should be a fixed-point, IMHO. For example, ’package->manifest-entry*’ is "Like 'package->manifest-entry', but attach PACKAGE provenance meta-data to the resulting manifest entry." (I have not tried.) Therefore, we could have something similar with ’options->transform’, i.e., attach somehow meta-data. Well, because the feature is missing, the story about #2 is incomplete. And I would like to have a self-reproducible Docker image produced by ‘guix pack’. This “missing feature“, is it a conscientious choice or an use-case not thought yet? WDYT? Cheers, simon ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Unexpected --export-manifest with simple transformations 2021-02-11 5:33 ` zimoun @ 2021-02-22 22:41 ` Ludovic Courtès 2021-02-25 14:48 ` zimoun 0 siblings, 1 reply; 5+ messages in thread From: Ludovic Courtès @ 2021-02-22 22:41 UTC (permalink / raw) To: zimoun; +Cc: Guix Devel [-- Attachment #1: Type: text/plain, Size: 1312 bytes --] Hi, zimoun <zimon.toutoune@gmail.com> skribis: > Hi Ludo, > > On Thu, 11 Feb 2021 at 00:01, Ludovic Courtès <ludo@gnu.org> wrote: > >> That’s because when using ‘-m’, transformations are not recorded. > > Yes. The question is: is it a conscientious choice or a missing > feature? [...] > For example, ’package->manifest-entry*’ is "Like > 'package->manifest-entry', but attach PACKAGE provenance meta-data to > the resulting manifest entry." (I have not tried.) Therefore, we could > have something similar with ’options->transform’, i.e., attach somehow > meta-data. > > Well, because the feature is missing, the story about #2 is incomplete. > And I would like to have a self-reproducible Docker image produced by > ‘guix pack’. > > This “missing feature“, is it a conscientious choice or an use-case not > thought yet? I looked into it more closely. It wasn’t really conscious. As you point out, we can have the ‘transformations’ property of manifest entries created automatically by default; the patch below does that. That way, the ‘transformations’ property is saved by default whether you use a manifest or the imperative model, and ‘--export-manifest’ and ‘-m’ are dual. How does that sound? Thanks, Ludo’. [-- Attachment #2: Type: text/x-patch, Size: 4989 bytes --] diff --git a/guix/profiles.scm b/guix/profiles.scm index ea8bc6e593..f942ae38d8 100644 --- a/guix/profiles.scm +++ b/guix/profiles.scm @@ -362,9 +362,14 @@ file name." #t lst))) +(define (default-properties package) + (match (assq-ref (package-properties package) 'transformations) + (#f '()) + (transformations `((transformations . ,transformations))))) + (define* (package->manifest-entry package #:optional (output "out") #:key (parent (delay #f)) - (properties '())) + (properties (default-properties package))) "Return a manifest entry for the OUTPUT of package PACKAGE." ;; For each dependency, keep a promise pointing to its "parent" entry. (letrec* ((deps (map (match-lambda diff --git a/guix/scripts/pack.scm b/guix/scripts/pack.scm index 8ecdcb823f..b653138f2c 100644 --- a/guix/scripts/pack.scm +++ b/guix/scripts/pack.scm @@ -1,5 +1,5 @@ ;;; GNU Guix --- Functional package management for GNU -;;; Copyright © 2015, 2017, 2018, 2019, 2020 Ludovic Courtès <ludo@gnu.org> +;;; Copyright © 2015, 2017, 2018, 2019, 2020, 2021 Ludovic Courtès <ludo@gnu.org> ;;; Copyright © 2017, 2018 Ricardo Wurmus <rekado@elephly.net> ;;; Copyright © 2018 Konrad Hinsen <konrad.hinsen@fastmail.net> ;;; Copyright © 2018 Chris Marusich <cmmarusich@gmail.com> @@ -1170,24 +1170,19 @@ Create a bundle of PACKAGE.\n")) manifest)) identity)) - (define (with-transformations manifest) - (map-manifest-entries manifest-entry-with-transformations - manifest)) - (with-provenance - (with-transformations - (cond - ((and (not (null? manifests)) (not (null? packages))) - (leave (G_ "both a manifest and a package list were given~%"))) - ((not (null? manifests)) - (concatenate-manifests - (map (lambda (file) - (let ((user-module (make-user-module - '((guix profiles) (gnu))))) - (load* file user-module))) - manifests))) - (else - (packages->manifest packages))))))) + (cond + ((and (not (null? manifests)) (not (null? packages))) + (leave (G_ "both a manifest and a package list were given~%"))) + ((not (null? manifests)) + (concatenate-manifests + (map (lambda (file) + (let ((user-module (make-user-module + '((guix profiles) (gnu))))) + (load* file user-module))) + manifests))) + (else + (packages->manifest packages)))))) (with-error-handling (with-store store diff --git a/guix/scripts/package.scm b/guix/scripts/package.scm index 8234a1703d..fc5bf8137b 100644 --- a/guix/scripts/package.scm +++ b/guix/scripts/package.scm @@ -235,14 +235,12 @@ non-zero relevance score." (case (version-compare candidate-version version) ((>) (manifest-transaction-install-entry - (manifest-entry-with-transformations - (package->manifest-entry* pkg output)) + (package->manifest-entry* pkg output) transaction)) ((<) transaction) ((=) - (let* ((new (manifest-entry-with-transformations - (package->manifest-entry* pkg output)))) + (let* ((new (package->manifest-entry* pkg output))) ;; Here we want to determine whether the NEW actually ;; differs from ENTRY, but we need to intercept ;; 'build-things' calls because they would prevent us from diff --git a/tests/transformations.scm b/tests/transformations.scm index 7877029486..902bd45a6a 100644 --- a/tests/transformations.scm +++ b/tests/transformations.scm @@ -20,6 +20,9 @@ #:use-module (guix tests) #:use-module (guix store) #:use-module ((guix gexp) #:select (lower-object)) + #:use-module ((guix profiles) + #:select (package->manifest-entry + manifest-entry-properties)) #:use-module (guix derivations) #:use-module (guix packages) #:use-module (guix git-download) @@ -413,6 +416,13 @@ `((with-latest . "foo"))))) (package-version (t p))))) +(test-equal "options->transformation + package->manifest-entry" + '((transformations . ((without-tests . "foo")))) + (let* ((p (dummy-package "foo")) + (t (options->transformation '((without-tests . "foo")))) + (e (package->manifest-entry (t p)))) + (manifest-entry-properties e))) + (test-end) ;;; Local Variables: ^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: Unexpected --export-manifest with simple transformations 2021-02-22 22:41 ` Ludovic Courtès @ 2021-02-25 14:48 ` zimoun 0 siblings, 0 replies; 5+ messages in thread From: zimoun @ 2021-02-25 14:48 UTC (permalink / raw) To: Ludovic Courtès; +Cc: Guix Devel Hi Ludo, On Mon, 22 Feb 2021 at 23:41, Ludovic Courtès <ludo@gnu.org> wrote: > As you point out, we can have the ‘transformations’ property of manifest > entries created automatically by default; the patch below does that. > That way, the ‘transformations’ property is saved by default whether you > use a manifest or the imperative model, and ‘--export-manifest’ and ‘-m’ > are dual. Thanks. > How does that sound? Sounds good. :-) The previous example works like a charm. I have not looked at the details yet. I will continue my story about reproduce "guix pack -f docker" to see if something is missing. Cheers, simon ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2021-02-25 14:48 UTC | newest] Thread overview: 5+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2021-02-10 1:36 Unexpected --export-manifest with simple transformations zimoun 2021-02-10 23:01 ` Ludovic Courtès 2021-02-11 5:33 ` zimoun 2021-02-22 22:41 ` Ludovic Courtès 2021-02-25 14:48 ` zimoun
Code repositories for project(s) associated with this public inbox https://git.savannah.gnu.org/cgit/guix.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).