bug#74015: 'guix shell --export-manifest' fails for some transformations

bug#74015: 'guix shell --export-manifest' fails for some transformations

[Simon Tournier]

Hi,

I notice this:

--8<---------------cut here---------------start------------->8---
$ cat manifest.scm
(use-modules (guix transformations) (guix packages)
             (guix download) (guix build-system r))

(define r-knitr
  (specification->package "r-knitr"))

(define r-knitr-other
  (package
    (inherit r-knitr)
    (version "1.47")
    (source (origin
              (method url-fetch)
              (uri (cran-uri "knitr" version))
              (sha256
               (base32
                "1kx6g8ddbigwx3bmq771gwi3din3fxjn59l801904kjaz6dq9pgs"))))))

(define r-ggplot2
  (package
    (inherit
     ((package-input-rewriting `((,r-knitr . ,r-knitr-other)))
      (specification->package "r-ggplot2")))
    (name "my-ggplot")))

(define transform
  (options->transformation
    '((with-c-toolchain . "r-minimal=gcc-toolchain@12")
      (with-c-toolchain . "r=gcc-toolchain@12"))))

(packages->manifest
 (append
  (map specification->package
       (list "python" "python-numpy"))
  (map transform
       (list (specification->package "r")
             r-ggplot2))))

$ guix shell -m manifest.scm --export-manifest
guix shell: warning: transformation 'with-c-toolchain' had no effect on my-ggplot@3.5.1.
;; What follows is a "manifest" equivalent to the command line you gave.
;; You can store it in a file that you may then pass to any 'guix' command
;; that accepts a '--manifest' (or '-m') option.

(use-modules (guix transformations))

(define transform1
  (options->transformation
    '((with-c-toolchain . "r-minimal=gcc-toolchain@12")
      (with-c-toolchain . "r=gcc-toolchain@12"))))

(packages->manifest
  (list (specification->package "python")
        (specification->package "python-numpy")
        (transform1 (specification->package "r"))
        (transform1 (specification->package "my-ggplot"))))
--8<---------------cut here---------------end--------------->8---

Well, this is incorrect.  While I understand the approximation when
dealing with profiles built over time in the imperative way, here I am
missing why the transformation of ’r-ggplot2’ is skipped.  And even the
export manifest is broken.

It’s skipped because:

--8<---------------cut here---------------start------------->8---
$ guix shell -m manifest.scm
$ grep ggplot $GUIX_ENVIRONMENT/manifest
     ("my-ggplot"
      "/gnu/store/x0jbmvg0nbl7lyk8zd938rq2c7c9qkn4-my-ggplot-3.5.1"
--8<---------------cut here---------------end--------------->8---

Please note that if the symbol ’r-ggplot2’ is not named “my-ggplot” but
kept as ’r-ggplot2“:

        (define r-ggplot2
          ((package-input-rewriting `((,r-knitr . ,r-knitr-other)))
              (specification->package "r-ggplot2")))

then the bug is maybe worse because the exported manifest “works” but
without the transformation… which can be misleading and hard to detect.

Well, I understand it’s hard to capture this kind of transformation.
Still, it appears to me as an half-baked feature. :-)

That’s annoying in some context as “guix pack --save-provenance” [1].
Even, it defeats the idea of a self-contained reproducible binary
container. ;-)

Maybe, aside the profile ’manifest’ file, we could store all the
manifests provided by the command line.  Something as:

    /gnu/store/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-profile/manifest
    /gnu/store/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-profile/manifest.orig1
    /gnu/store/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-profile/manifest.orig2
    …

where manifest.orig1 and manifest.orig2 are the copy of files ’foo.scm’
and ’bar.scm’ from:

    -m foo.scm -m bar.scm

This way, the option --export-manifest could be improved.

WDYT?

Cheers,
simon

1: https://hpc.guix.info/blog/2021/10/when-docker-images-become-fixed-point

bug#74015: 'guix shell --export-manifest' fails for some transformations

[Ludovic Courtès]

Hi,

Simon Tournier <zimon.toutoune@gmail.com> skribis:

> $ guix shell -m manifest.scm --export-manifest

‘--export-manifest’ is meant to “translate” a command line, which it can
do faithfully; there’s no way it could possibly “recreate” what
‘manifest.scm’, which may contain arbitrary Scheme code (and someone who
already has a ‘manifest.scm’ file probably doesn’t need
‘--export-manifest’).

It’s necessarily a lossy process.  Some of the transformations
‘manifest.scm’ uses may be recorded as metadata in manifest entries, but
others, such as uses of ‘package-input-rewriting’ here, inevitably get
lost.

I would close it as not-a-bug; WDYT?

Ludo’.

bug#74015: 'guix shell --export-manifest' fails for some transformations

[Simon Tournier]

Hi,

On Sun, 10 Nov 2024 at 12:38, Ludovic Courtès <ludo@gnu.org> wrote:

>> $ guix shell -m manifest.scm --export-manifest
>
> ‘--export-manifest’ is meant to “translate” a command line, which it can
> do faithfully; there’s no way it could possibly “recreate” what
> ‘manifest.scm’, which may contain arbitrary Scheme code (and someone who
> already has a ‘manifest.scm’ file probably doesn’t need
> ‘--export-manifest’).

Yes, one might need. :-)

As I explained: « That’s annoying in some context as “guix pack
--save-provenance” [1].  Even, it defeats the idea of a self-contained
reproducible binary container. »

For sure, I understand it’s hard to capture this kind of transformation.
Still, an improvement appears to me possible (see below).

1: https://hpc.guix.info/blog/2021/10/when-docker-images-become-fixed-point


> It’s necessarily a lossy process.

This is where I disagree. :-) I mean, yes I agree that building
profile/manifest is somehow a lossy process because some Scheme is
potentially evaluated on the road.  However, I am proposing: Aside the
profile ’manifest’ file (lossy process), we could store all the
manifests provided by the command line.  Something as:

    /gnu/store/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-profile/manifest
    /gnu/store/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-profile/manifest.orig1
    /gnu/store/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-profile/manifest.orig2
    …

where manifest.orig1 and manifest.orig2 are the copy of files ’foo.scm’
and ’bar.scm’ from:

    -m foo.scm -m bar.scm

This way, the option --export-manifest could be improved, eventually.

In other word, I think that saving as-is the manifests costs almost
nothing and it paves the way to more robust self-contained binary packs.

Well, maybe this extra work could be only done when there is an option
’--save-provenance’ applied.

> I would close it as not-a-bug; WDYT?

I still think it’s a feature that it’s possible to improve.

Cheers,
simon

bug#74015: 'guix shell --export-manifest' fails for some transformations

[Suhail Singh]

Simon Tournier <zimon.toutoune@gmail.com> writes:

> However, I am proposing: Aside the profile ’manifest’ file (lossy
> process), we could store all the manifests provided by the command
> line.  Something as:
>
>     /gnu/store/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-profile/manifest
>     /gnu/store/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-profile/manifest.orig1
>     /gnu/store/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-profile/manifest.orig2
>     …
>
> where manifest.orig1 and manifest.orig2 are the copy of files ’foo.scm’
> and ’bar.scm’ from:
>
>     -m foo.scm -m bar.scm
>
> ...
>
> Well, maybe this extra work could be only done when there is an option
> ’--save-provenance’ applied.

For --save-provenance, is your proposal that in addition to saving the
additional manifests, the commandline that was used would also be saved?

-- 
Suhail

bug#74015: 'guix shell --export-manifest' fails for some transformations

[Simon Tournier]

Hi,

On Tue, 12 Nov 2024 at 10:33, Suhail Singh <suhailsingh247@gmail.com> wrote:
> Simon Tournier <zimon.toutoune@gmail.com> writes:
>
>> However, I am proposing: Aside the profile ’manifest’ file (lossy
>> process), we could store all the manifests provided by the command
>> line.  Something as:
>>
>>     /gnu/store/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-profile/manifest
>>     /gnu/store/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-profile/manifest.orig1
>>     /gnu/store/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-profile/manifest.orig2
>>     …
>>
>> where manifest.orig1 and manifest.orig2 are the copy of files ’foo.scm’
>> and ’bar.scm’ from:
>>
>>     -m foo.scm -m bar.scm
>>
>> ...
>>
>> Well, maybe this extra work could be only done when there is an option
>> ’--save-provenance’ applied.
>
> For --save-provenance, is your proposal that in addition to saving the
> additional manifests, the commandline that was used would also be saved?

We could save the command line too using another file. :-)

Cheers,
simon

bug#74015: 'guix shell --export-manifest' fails for some transformations

[Suhail Singh]

Simon Tournier <zimon.toutoune@gmail.com> writes:

>> For --save-provenance, is your proposal that in addition to saving the
>> additional manifests, the commandline that was used would also be saved?
>
> We could save the command line too using another file. :-)

That would be helpful :)

Given an image generated with --save-provenance enabled, making it
easier for it to be reconstructed by another would certainly be useful.

-- 
Suhail

bug#74015: 'guix shell --export-manifest' fails for some transformations

[Ludovic Courtès]

Hi,

Simon Tournier <zimon.toutoune@gmail.com> skribis:

> On Sun, 10 Nov 2024 at 12:38, Ludovic Courtès <ludo@gnu.org> wrote:
>
>>> $ guix shell -m manifest.scm --export-manifest
>>
>> ‘--export-manifest’ is meant to “translate” a command line, which it can
>> do faithfully; there’s no way it could possibly “recreate” what
>> ‘manifest.scm’, which may contain arbitrary Scheme code (and someone who
>> already has a ‘manifest.scm’ file probably doesn’t need
>> ‘--export-manifest’).
>
> Yes, one might need. :-)
>
> As I explained: « That’s annoying in some context as “guix pack
> --save-provenance” [1].  Even, it defeats the idea of a self-contained
> reproducible binary container. »

I see, but if you have the source ‘manifest.scm’, best practice is to
publish it.

>> It’s necessarily a lossy process.
>
> This is where I disagree. :-)

You are free to disagree but it’s a fact.

> I mean, yes I agree that building profile/manifest is somehow a lossy
> process because some Scheme is potentially evaluated on the road.
> However, I am proposing: Aside the profile ’manifest’ file (lossy
> process), we could store all the manifests provided by the command
> line.  Something as:
>
>     /gnu/store/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-profile/manifest
>     /gnu/store/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-profile/manifest.orig1
>     /gnu/store/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-profile/manifest.orig2
>     …
>
> where manifest.orig1 and manifest.orig2 are the copy of files ’foo.scm’
> and ’bar.scm’ from:
>
>     -m foo.scm -m bar.scm

Hmm yes, maybe ‘guix pack -m manifest.scm --save-provenance’ could
arrange to put ‘manifest.scm’ inside the pack, similar to how ‘guix
system reconfigure’ inserts ‘configuration.scm’ inside the system.

I’m all for it!

Thanks,
Ludo’.