Unexpected --export-manifest with simple transformations

Unexpected --export-manifest with simple transformations

[zimoun]

Hi,

If the transformations are in the manifest.scm file, then they are
lost.  For example, consider:

--8<---------------cut here---------------start------------->8---
$ guix package \
       -p /tmp/profile-cli \
       -i python python-numpy \
       hello --with-c-toolchain=hello=gcc-toolchain@10
guix package: warning: transformation 'with-c-toolchain' had no effect on python@3.8.2
guix package: warning: transformation 'with-c-toolchain' had no effect on python-numpy@1.17.3

[...]

$ guix package -p /tmp/profile-cli --export-manifest > /tmp/manifest.scm

$ cat /tmp/manifest.scm
;; This "manifest" file can be passed to 'guix package -m' to reproduce
;; the content of your profile.  This is "symbolic": it only specifies
;; package names.  To reproduce the exact same profile, you also need to
;; capture the channels being used, as returned by "guix describe".
;; See the "Replicating Guix" section in the manual.

(use-modules (guix transformations))

(define transform1
  (options->transformation
    '((with-c-toolchain . "hello=gcc-toolchain@10"))))

(packages->manifest
  (list (transform1 (specification->package "hello"))
        (specification->package "python-numpy")
        (specification->package "python")))
--8<---------------cut here---------------end--------------->8---


So far, so good.  Then let use this manifest to create a profile.

   $ guix package -p /tmp/profile-exported -m /tmp/manifest.scm

and the issue is that ’/tmp/profile-exported/manifest’ does not contain
the transformation.  Other said:

--8<---------------cut here---------------start------------->8---
$ guix package -p /tmp/profile-exported --export-manifest
;; This "manifest" file can be passed to 'guix package -m' to reproduce
;; the content of your profile.  This is "symbolic": it only specifies
;; package names.  To reproduce the exact same profile, you also need to
;; capture the channels being used, as returned by "guix describe".
;; See the "Replicating Guix" section in the manual.

(specifications->manifest
  (list "hello" "python-numpy" "python"))
--8<---------------cut here---------------end--------------->8---


Note that obviously:

--8<---------------cut here---------------start------------->8---
$ ls -l /tmp/profile-{cli,exported}/bin/ | grep hello
lrwxrwxrwx 3 root root 64 Jan  1  1970 hello -> /gnu/store/qi7pqqsxhbwmy75hl43j7l0aw1xr7r42-hello-2.10/bin/hello
lrwxrwxrwx 3 root root 64 Jan  1  1970 hello -> /gnu/store/qi7pqqsxhbwmy75hl43j7l0aw1xr7r42-hello-2.10/bin/hello
--8<---------------cut here---------------end--------------->8---


Well, is it a bug or a feature? ;-)

From my understanding, it makes sense to save the transformations from a
manifest.  But it is not clear what should be the good solution for
that because it is not easy; again from my understanding.


The obvious answer is: do not loose the manifest.scm file! :-) Sure, I
am working in the context where Alice produced a Docker image with “guix
pack -f docker --save-provenance -m manifest.scm” where manifest.scm
uses transformations.  Then since Alice is post-doc and goes doing other
stuff elsewhere, the source probably vanishes.  Bob wants to rebuild the
Docker image.  It is possible for simple case as I already illustrated
it. :-)

To be concrete, I am trying to write down a non-trivial example using a
custom package and a transformation for a blog post and/or Cookbook
recipe.


Cheers,
simon

Re: Unexpected --export-manifest with simple transformations

[Ludovic Courtès]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

> So far, so good.  Then let use this manifest to create a profile.
>
>    $ guix package -p /tmp/profile-exported -m /tmp/manifest.scm
>
> and the issue is that ’/tmp/profile-exported/manifest’ does not contain
> the transformation.  Other said:
>
> $ guix package -p /tmp/profile-exported --export-manifest
> ;; This "manifest" file can be passed to 'guix package -m' to reproduce
> ;; the content of your profile.  This is "symbolic": it only specifies
> ;; package names.  To reproduce the exact same profile, you also need to
> ;; capture the channels being used, as returned by "guix describe".
> ;; See the "Replicating Guix" section in the manual.
>
> (specifications->manifest
>   (list "hello" "python-numpy" "python"))

That’s because when using ‘-m’, transformations are not recorded.

‘--export-manifest’ can be thought of as a disassembler; in this case,
it lacks debug info to perform better.

HTH,
Ludo’.

Re: Unexpected --export-manifest with simple transformations

[zimoun]

Hi Ludo,

On Thu, 11 Feb 2021 at 00:01, Ludovic Courtès <ludo@gnu.org> wrote:

> That’s because when using ‘-m’, transformations are not recorded.

Yes.  The question is: is it a conscientious choice or a missing
feature?


It makes sense to save the transformations from a manifest, IMHO.  But
it is not clear what should be the good solution for that because it is
not easy for the general case; from my understanding.

For instance, one could easily imagine this (restricted) typical
manifest with transformation,

--8<---------------cut here---------------start------------->8---
(use-modules (guix transformations))

(define transform1
  (options->transformation
    '((with-c-toolchain . "hello=gcc-toolchain@10"))))

(packages->manifest
  (list (transform1 (specification->package "hello"))
        (specification->package "python-numpy")
        (specification->package "python")))
--8<---------------cut here---------------end--------------->8---

should record the transformation in <profile>/manifest.  Motivated by
1. consistency with the «equivalent» command-line and 2. the use-case
“guix pack -f docker --save-provenance -m’.

Other said, the option ’-m + --export-manifest’ should be a fixed-point,
IMHO. 

For example, ’package->manifest-entry*’ is "Like
'package->manifest-entry', but attach PACKAGE provenance meta-data to
the resulting manifest entry."  (I have not tried.)  Therefore, we could
have something similar with ’options->transform’, i.e., attach somehow
meta-data.

Well, because the feature is missing, the story about #2 is incomplete.
And I would like to have a self-reproducible Docker image produced by
‘guix pack’.

This “missing feature“, is it a conscientious choice or an use-case not
thought yet?


WDYT?


Cheers,
simon

Re: Unexpected --export-manifest with simple transformations

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 1312 bytes --]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

> Hi Ludo,
>
> On Thu, 11 Feb 2021 at 00:01, Ludovic Courtès <ludo@gnu.org> wrote:
>
>> That’s because when using ‘-m’, transformations are not recorded.
>
> Yes.  The question is: is it a conscientious choice or a missing
> feature?

[...]

> For example, ’package->manifest-entry*’ is "Like
> 'package->manifest-entry', but attach PACKAGE provenance meta-data to
> the resulting manifest entry."  (I have not tried.)  Therefore, we could
> have something similar with ’options->transform’, i.e., attach somehow
> meta-data.
>
> Well, because the feature is missing, the story about #2 is incomplete.
> And I would like to have a self-reproducible Docker image produced by
> ‘guix pack’.
>
> This “missing feature“, is it a conscientious choice or an use-case not
> thought yet?

I looked into it more closely.  It wasn’t really conscious.

As you point out, we can have the ‘transformations’ property of manifest
entries created automatically by default; the patch below does that.
That way, the ‘transformations’ property is saved by default whether you
use a manifest or the imperative model, and ‘--export-manifest’ and ‘-m’
are dual.

How does that sound?

Thanks,
Ludo’.


[-- Attachment #2: Type: text/x-patch, Size: 4989 bytes --]

diff --git a/guix/profiles.scm b/guix/profiles.scm
index ea8bc6e593..f942ae38d8 100644
--- a/guix/profiles.scm
+++ b/guix/profiles.scm
@@ -362,9 +362,14 @@ file name."
            #t
            lst)))
 
+(define (default-properties package)
+  (match (assq-ref (package-properties package) 'transformations)
+    (#f '())
+    (transformations `((transformations . ,transformations)))))
+
 (define* (package->manifest-entry package #:optional (output "out")
                                   #:key (parent (delay #f))
-                                  (properties '()))
+                                  (properties (default-properties package)))
   "Return a manifest entry for the OUTPUT of package PACKAGE."
   ;; For each dependency, keep a promise pointing to its "parent" entry.
   (letrec* ((deps  (map (match-lambda
diff --git a/guix/scripts/pack.scm b/guix/scripts/pack.scm
index 8ecdcb823f..b653138f2c 100644
--- a/guix/scripts/pack.scm
+++ b/guix/scripts/pack.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2015, 2017, 2018, 2019, 2020 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2015, 2017, 2018, 2019, 2020, 2021 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2017, 2018 Ricardo Wurmus <rekado@elephly.net>
 ;;; Copyright © 2018 Konrad Hinsen <konrad.hinsen@fastmail.net>
 ;;; Copyright © 2018 Chris Marusich <cmmarusich@gmail.com>
@@ -1170,24 +1170,19 @@ Create a bundle of PACKAGE.\n"))
                manifest))
             identity))
 
-      (define (with-transformations manifest)
-        (map-manifest-entries manifest-entry-with-transformations
-                              manifest))
-
       (with-provenance
-       (with-transformations
-        (cond
-         ((and (not (null? manifests)) (not (null? packages)))
-          (leave (G_ "both a manifest and a package list were given~%")))
-         ((not (null? manifests))
-          (concatenate-manifests
-           (map (lambda (file)
-                  (let ((user-module (make-user-module
-                                      '((guix profiles) (gnu)))))
-                    (load* file user-module)))
-                manifests)))
-         (else
-          (packages->manifest packages)))))))
+       (cond
+        ((and (not (null? manifests)) (not (null? packages)))
+         (leave (G_ "both a manifest and a package list were given~%")))
+        ((not (null? manifests))
+         (concatenate-manifests
+          (map (lambda (file)
+                 (let ((user-module (make-user-module
+                                     '((guix profiles) (gnu)))))
+                   (load* file user-module)))
+               manifests)))
+        (else
+         (packages->manifest packages))))))
 
   (with-error-handling
     (with-store store
diff --git a/guix/scripts/package.scm b/guix/scripts/package.scm
index 8234a1703d..fc5bf8137b 100644
--- a/guix/scripts/package.scm
+++ b/guix/scripts/package.scm
@@ -235,14 +235,12 @@ non-zero relevance score."
                (case (version-compare candidate-version version)
                  ((>)
                   (manifest-transaction-install-entry
-                   (manifest-entry-with-transformations
-                    (package->manifest-entry* pkg output))
+                   (package->manifest-entry* pkg output)
                    transaction))
                  ((<)
                   transaction)
                  ((=)
-                  (let* ((new (manifest-entry-with-transformations
-                               (package->manifest-entry* pkg output))))
+                  (let* ((new (package->manifest-entry* pkg output)))
                     ;; Here we want to determine whether the NEW actually
                     ;; differs from ENTRY, but we need to intercept
                     ;; 'build-things' calls because they would prevent us from
diff --git a/tests/transformations.scm b/tests/transformations.scm
index 7877029486..902bd45a6a 100644
--- a/tests/transformations.scm
+++ b/tests/transformations.scm
@@ -20,6 +20,9 @@
   #:use-module (guix tests)
   #:use-module (guix store)
   #:use-module ((guix gexp) #:select (lower-object))
+  #:use-module ((guix profiles)
+                #:select (package->manifest-entry
+                          manifest-entry-properties))
   #:use-module (guix derivations)
   #:use-module (guix packages)
   #:use-module (guix git-download)
@@ -413,6 +416,13 @@
                    `((with-latest . "foo")))))
           (package-version (t p)))))
 
+(test-equal "options->transformation + package->manifest-entry"
+  '((transformations . ((without-tests . "foo"))))
+  (let* ((p (dummy-package "foo"))
+         (t (options->transformation '((without-tests . "foo"))))
+         (e (package->manifest-entry (t p))))
+    (manifest-entry-properties e)))
+
 (test-end)
 
 ;;; Local Variables:

Re: Unexpected --export-manifest with simple transformations

[zimoun]

Hi Ludo,

On Mon, 22 Feb 2021 at 23:41, Ludovic Courtès <ludo@gnu.org> wrote:

> As you point out, we can have the ‘transformations’ property of manifest
> entries created automatically by default; the patch below does that.
> That way, the ‘transformations’ property is saved by default whether you
> use a manifest or the imperative model, and ‘--export-manifest’ and ‘-m’
> are dual.

Thanks.

> How does that sound?

Sounds good. :-)
The previous example works like a charm.   I have not looked at the details yet.

I will continue my story about reproduce "guix pack -f docker" to see
if something is missing.


Cheers,
simon