Re: bug#45919: [PATCH 0/8] Exporting a manifest and channels from a profile

Re: bug#45919: [PATCH 0/8] Exporting a manifest and channels from a profile

[Ludovic Courtès]

Hi there!

Ludovic Courtès <ludo@gnu.org> skribis:

> Here’s a simple but oft-requested feature (I remember discussing
> with Pierre and Simon a year ago at the Guix Days about ways to
> implement it—time flies!): these patches provide ‘guix package
> --export-manifest’ and ‘--export-channels’.  These options spit
> a manifest and a channel spec, respectively, with the goal of
> helping users who wish to migrate to the declarative model.

I’m rather happy with this patch set but since this is something we’ve
discussed several times in the past, I think it’d be great if those
interested could chime in and comment:

  https://issues.guix.gnu.org/45919

I’ll leave a few more days and then… push!

TIA,
Ludo’.

Re: [bug#45919] [PATCH 0/8] Exporting a manifest and channels from a profile

[zimoun]

Hi Ludo,

On Wed, 27 Jan 2021 at 14:14, Ludovic Courtès <ludo@gnu.org> wrote:
> Ludovic Courtès <ludo@gnu.org> skribis:
>
>> Here’s a simple but oft-requested feature (I remember discussing
>> with Pierre and Simon a year ago at the Guix Days about ways to
>> implement it—time flies!): these patches provide ‘guix package
>> --export-manifest’ and ‘--export-channels’.  These options spit
>> a manifest and a channel spec, respectively, with the goal of
>> helping users who wish to migrate to the declarative model.
>
> I’m rather happy with this patch set but since this is something we’ve
> discussed several times in the past, I think it’d be great if those
> interested could chime in and comment:
>
>   https://issues.guix.gnu.org/45919
>
> I’ll leave a few more days and then… push!

You are looking at me? :-)

Well, the feature is nice!  At first, I thought that it could be a bit
smarter than using only one commit.  But as you said, the aim is for
transitioning.  Maybe a future improvement should to list somewhere in
the comments which commit provides which set of packages.  It could
help… or not. :-)

Using Docker, it works.  Let describe how in case people are interested.


On machine A, I have:

--8<---------------cut here---------------start------------->8---
$ guix describe -f channels
(list (channel
        (name 'guix)
        (url "https://git.savannah.gnu.org/git/guix.git")
        (commit
          "cb68ae668af2ade4b0777d82f227e5462768e9e5")
        (introduction
          (make-channel-introduction
            "9edb3f66fd807b096b48283debdcddccfea34bad"
            (openpgp-fingerprint
              "BBB0 2DDF 2CEA F6A8 0D1D  E643 A2A0 6DF2 A33A 54FA")))))

$ cat ~/.config/guix/manifests/python.scm
(specifications->manifest
 (append
  '("python"
    )
  (map
   (lambda (pkg)
     (string-append "python-" pkg))
   '("ipython"
     "numpy"
     "matplotlib"
     "scipy"
     "biopython"
  ))))
--8<---------------cut here---------------end--------------->8---

And I generate a Docker pack with:

--8<---------------cut here---------------start------------->8---
$ guix pack -f docker --save-provenance -m ~/.config/guix/manifests/python.scm
/gnu/store/wxymmnxdvdvf08ifsfy39xjaxilhrigk-docker-pack.tar.gz
--8<---------------cut here---------------end--------------->8---


Then on machine B, after fetching this tarball, I run:

--8<---------------cut here---------------start------------->8---
$ docker load < /tmp/img/wxymmnxdvdvf08ifsfy39xjaxilhrigk-docker-pack.tar.gz
$ docker images
REPOSITORY                           TAG                 IMAGE ID            CREATED             SIZE
python-python-ipython-python-numpy   latest              49ddfedf1e27        51 years ago        1.45GB
--8<---------------cut here---------------end--------------->8---

And it works as expected:

--8<---------------cut here---------------start------------->8---
$ docker run -ti python-python-ipython-python-numpy:latest python3
Python 3.8.2 (default, Jan  1 1970, 00:00:01)
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import this
The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!
>>> import numpy as np
>>> A = np.array([[1,0,1],[0,1,0],[0,0,1]])
>>> _, s, _ = np.linalg.svd(A); s; abs(s[0] - 1./s[2])
array([1.61803399, 1.        , 0.61803399])
0.0
>>>
--8<---------------cut here---------------end--------------->8---

Neat!

So far, so good.  Well, let extract the ’manifest’ from this Docker
blob.

--8<---------------cut here---------------start------------->8---
$ docker export -o /tmp/img/re-pack.tar $(docker ps -a --format "{{.ID}}" | head -n1)
$ tar -xf /tmp/img/re-pack.tar $(tar -tf /tmp/img/re-pack.tar | grep 'profile/manifest')
$ cat gnu/store/7frdchgf5sqw8b83azsml3lw0h52gfbk-profile/manifest | grep -E "(\(\"python|cb68ae)" | head -n5
    (("python"
              "cb68ae668af2ade4b0777d82f227e5462768e9e5")
     ("python-ipython"
        (("python-backcall"
         ("python-pyzmq"
--8<---------------cut here---------------end--------------->8---

Now, a trick to get the channels and specifications:

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix package -p /tmp/img/gnu/store/7frdchgf5sqw8b83azsml3lw0h52gfbk-profile --export-channels
;; This channel file can be passed to 'guix pull -C' or to
;; 'guix time-machine -C' to obtain the Guix revision that was
;; used to populate this profile.

(list
     (channel
       (name 'guix)
       (url "https://git.savannah.gnu.org/git/guix.git")
       (commit
         "cb68ae668af2ade4b0777d82f227e5462768e9e5")
       (introduction
         (make-channel-introduction
           "9edb3f66fd807b096b48283debdcddccfea34bad"
           (openpgp-fingerprint
             "BBB0 2DDF 2CEA F6A8 0D1D  E643 A2A0 6DF2 A33A 54FA"))))
)

$ ./pre-inst-env guix package -p /tmp/img/gnu/store/7frdchgf5sqw8b83azsml3lw0h52gfbk-profile --export-manifest
$ ./pre-inst-env guix package -p /tmp/img/gnu/store/7frdchgf5sqw8b83azsml3lw0h52gfbk-profile --export-manifest
;; This "manifest" file can be passed to 'guix package -m' to reproduce
;; the content of your profile.  This is "symbolic": it only specifies
;; package names.  To reproduce the exact same profile, you also need to
;; capture the channels being used, as returned by "guix describe".
;; See the "Replicating Guix" section in the manual.

(specifications->manifest
  (list "python"
        "python-ipython"
        "python-numpy"
        "python-matplotlib"
        "python-scipy"
        "python-biopython"))
--8<---------------cut here---------------end--------------->8---

Awesome!


The unexpected is this channels and manifests files do not reproduce the
same Docker pack tarball:

--8<---------------cut here---------------start------------->8---
$ guix describe
Generation 99   Jan 05 2021 16:56:39    (current)
  guix-past 829923f
    repository URL: https://gitlab.inria.fr/guix-hpc/guix-past
    branch: master
    commit: 829923f01f894f1e687735627025ada26230832f
  guix-bimsb a8b539d
    repository URL: https://github.com/BIMSBbioinfo/guix-bimsb
    branch: master
    commit: a8b539d61a359060c35f3cb34c7edd1d9d14241d
  bimsb-nonfree 4084e63
    repository URL: https://github.com/BIMSBbioinfo/guix-bimsb-nonfree.git
    branch: master
    commit: 4084e63c9c0d662780870aded9f5a6ca1b063780
  guix-science cf87b05
    repository URL: https://github.com/guix-science/guix-science.git
    branch: master
    commit: cf87b0501c4a38b96edf41025a27bf1cb91f521a
  guix 957f0c4
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: 957f0c40327ce00f53db22737e3775ce616ac258

$ guix time-machine -C /tmp/img/channels.scm -- pack -f docker --save-provenance -m /tmp/img/manifest.scm
Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...
/gnu/store/xzk604g8gysv4azn7sf9nylr6iah97gl-docker-pack.tar.gz
--8<---------------cut here---------------end--------------->8---

To compare with
/gnu/store/wxymmnxdvdvf08ifsfy39xjaxilhrigk-docker-pack.tar.gz.

On a third machine, I get:
/gnu/store/wxymmnxdvdvf08ifsfy39xjaxilhrigk-docker-pack.tar.gz

Well, that’s another story and I have not inspected yet the
derivations and what could be wrong on the machine B.


Cheers,
simon

Re: [bug#45919] [PATCH 0/8] Exporting a manifest and channels from a profile

[Ludovic Courtès]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

> On Wed, 27 Jan 2021 at 14:14, Ludovic Courtès <ludo@gnu.org> wrote:
>> Ludovic Courtès <ludo@gnu.org> skribis:
>>
>>> Here’s a simple but oft-requested feature (I remember discussing
>>> with Pierre and Simon a year ago at the Guix Days about ways to
>>> implement it—time flies!): these patches provide ‘guix package
>>> --export-manifest’ and ‘--export-channels’.  These options spit
>>> a manifest and a channel spec, respectively, with the goal of
>>> helping users who wish to migrate to the declarative model.
>>
>> I’m rather happy with this patch set but since this is something we’ve
>> discussed several times in the past, I think it’d be great if those
>> interested could chime in and comment:
>>
>>   https://issues.guix.gnu.org/45919
>>
>> I’ll leave a few more days and then… push!
>
> You are looking at me? :-)

Yes, but not just.  :-)

> Well, the feature is nice!  At first, I thought that it could be a bit
> smarter than using only one commit.  But as you said, the aim is for
> transitioning.  Maybe a future improvement should to list somewhere in
> the comments which commit provides which set of packages.  It could
> help… or not. :-)
>
> Using Docker, it works.  Let describe how in case people are interested.

[...]

> Awesome!

Woow, thanks for explaining the experiment, it’s a nice one!  Would be
worth turning into a blog post, maybe for hpc.guix.info?

> The unexpected is this channels and manifests files do not reproduce the
> same Docker pack tarball:
>
> $ guix describe
> Generation 99   Jan 05 2021 16:56:39    (current)
>   guix-past 829923f
>     repository URL: https://gitlab.inria.fr/guix-hpc/guix-past
>     branch: master
>     commit: 829923f01f894f1e687735627025ada26230832f
>   guix-bimsb a8b539d
>     repository URL: https://github.com/BIMSBbioinfo/guix-bimsb
>     branch: master
>     commit: a8b539d61a359060c35f3cb34c7edd1d9d14241d
>   bimsb-nonfree 4084e63
>     repository URL: https://github.com/BIMSBbioinfo/guix-bimsb-nonfree.git
>     branch: master
>     commit: 4084e63c9c0d662780870aded9f5a6ca1b063780
>   guix-science cf87b05
>     repository URL: https://github.com/guix-science/guix-science.git
>     branch: master
>     commit: cf87b0501c4a38b96edf41025a27bf1cb91f521a
>   guix 957f0c4
>     repository URL: https://git.savannah.gnu.org/git/guix.git
>     branch: master
>     commit: 957f0c40327ce00f53db22737e3775ce616ac258
>
> $ guix time-machine -C /tmp/img/channels.scm -- pack -f docker --save-provenance -m /tmp/img/manifest.scm
> Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...
> /gnu/store/xzk604g8gysv4azn7sf9nylr6iah97gl-docker-pack.tar.gz
>
> To compare with
> /gnu/store/wxymmnxdvdvf08ifsfy39xjaxilhrigk-docker-pack.tar.gz.
>
> On a third machine, I get:
> /gnu/store/wxymmnxdvdvf08ifsfy39xjaxilhrigk-docker-pack.tar.gz
>
> Well, that’s another story and I have not inspected yet the
> derivations and what could be wrong on the machine B.

You’d have to check the differences.  It may be that provenance data
differs, for example because the second attempt includes data about
channels that are actually unused.  (That’s the whole problem of
provenance data: it’s not a one-to-one mapping and it’s not a bijection
either.)

Thanks for trying it out!

Ludo’.

Re: bug#45919: [PATCH 0/8] Exporting a manifest and channels from a profile

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 932 bytes --]

Hi Ludo!

Ludovic Courtès <ludo@gnu.org> writes:

> Ludovic Courtès <ludo@gnu.org> skribis:
>
>> Here’s a simple but oft-requested feature (I remember discussing
>> with Pierre and Simon a year ago at the Guix Days about ways to
>> implement it—time flies!): these patches provide ‘guix package
>> --export-manifest’ and ‘--export-channels’.  These options spit
>> a manifest and a channel spec, respectively, with the goal of
>> helping users who wish to migrate to the declarative model.
>
> I’m rather happy with this patch set but since this is something we’ve
> discussed several times in the past, I think it’d be great if those
> interested could chime in and comment:
>
>   https://issues.guix.gnu.org/45919
>
> I’ll leave a few more days and then… push!

Thanks a lot for working on this, I'll try to give it a shot tomorrow! :)

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 511 bytes --]

Re: [bug#45919] [PATCH 0/8] Exporting a manifest and channels from a profile

[Bengt Richter]

Hi zimoun,

Thank you for your tutorial-style and resource-reference-footnoted posts!

I don't know about others, but I save many on my hard disk as reference examples.

I hope this provides positive feedback for your efforts.
Thank you.

On +2021-01-27 22:02:53 +0100, zimoun wrote:
> Hi Ludo,
> 
> On Wed, 27 Jan 2021 at 14:14, Ludovic Courtès <ludo@gnu.org> wrote:
> > Ludovic Courtès <ludo@gnu.org> skribis:
> >
> >> Here’s a simple but oft-requested feature (I remember discussing
> >> with Pierre and Simon a year ago at the Guix Days about ways to
> >> implement it—time flies!): these patches provide ‘guix package
> >> --export-manifest’ and ‘--export-channels’.  These options spit
> >> a manifest and a channel spec, respectively, with the goal of
> >> helping users who wish to migrate to the declarative model.
> >
> > I’m rather happy with this patch set but since this is something we’ve
> > discussed several times in the past, I think it’d be great if those
> > interested could chime in and comment:
> >
> >   https://issues.guix.gnu.org/45919
> >
> > I’ll leave a few more days and then… push!
> 
> You are looking at me? :-)
> 
> Well, the feature is nice!  At first, I thought that it could be a bit
> smarter than using only one commit.  But as you said, the aim is for
> transitioning.  Maybe a future improvement should to list somewhere in
> the comments which commit provides which set of packages.  It could
> help… or not. :-)
> 
> Using Docker, it works.  Let describe how in case people are interested.
> 
> 
> On machine A, I have:
> 
> --8<---------------cut here---------------start------------->8---
> $ guix describe -f channels
> (list (channel
>         (name 'guix)
>         (url "https://git.savannah.gnu.org/git/guix.git")
>         (commit
>           "cb68ae668af2ade4b0777d82f227e5462768e9e5")
>         (introduction
>           (make-channel-introduction
>             "9edb3f66fd807b096b48283debdcddccfea34bad"
>             (openpgp-fingerprint
>               "BBB0 2DDF 2CEA F6A8 0D1D  E643 A2A0 6DF2 A33A 54FA")))))
> 
> $ cat ~/.config/guix/manifests/python.scm
> (specifications->manifest
>  (append
>   '("python"
>     )
>   (map
>    (lambda (pkg)
>      (string-append "python-" pkg))
>    '("ipython"
>      "numpy"
>      "matplotlib"
>      "scipy"
>      "biopython"
>   ))))
> --8<---------------cut here---------------end--------------->8---
> 
> And I generate a Docker pack with:
> 
> --8<---------------cut here---------------start------------->8---
> $ guix pack -f docker --save-provenance -m ~/.config/guix/manifests/python.scm
> /gnu/store/wxymmnxdvdvf08ifsfy39xjaxilhrigk-docker-pack.tar.gz
> --8<---------------cut here---------------end--------------->8---
> 
> 
> Then on machine B, after fetching this tarball, I run:
> 
> --8<---------------cut here---------------start------------->8---
> $ docker load < /tmp/img/wxymmnxdvdvf08ifsfy39xjaxilhrigk-docker-pack.tar.gz
> $ docker images
> REPOSITORY                           TAG                 IMAGE ID            CREATED             SIZE
> python-python-ipython-python-numpy   latest              49ddfedf1e27        51 years ago        1.45GB
> --8<---------------cut here---------------end--------------->8---
> 
> And it works as expected:
> 
> --8<---------------cut here---------------start------------->8---
> $ docker run -ti python-python-ipython-python-numpy:latest python3
> Python 3.8.2 (default, Jan  1 1970, 00:00:01)
> [GCC 7.5.0] on linux
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import this
> The Zen of Python, by Tim Peters
> 
> Beautiful is better than ugly.
> Explicit is better than implicit.
> Simple is better than complex.
> Complex is better than complicated.
> Flat is better than nested.
> Sparse is better than dense.
> Readability counts.
> Special cases aren't special enough to break the rules.
> Although practicality beats purity.
> Errors should never pass silently.
> Unless explicitly silenced.
> In the face of ambiguity, refuse the temptation to guess.
> There should be one-- and preferably only one --obvious way to do it.
> Although that way may not be obvious at first unless you're Dutch.
> Now is better than never.
> Although never is often better than *right* now.
> If the implementation is hard to explain, it's a bad idea.
> If the implementation is easy to explain, it may be a good idea.
> Namespaces are one honking great idea -- let's do more of those!
> >>> import numpy as np
> >>> A = np.array([[1,0,1],[0,1,0],[0,0,1]])
> >>> _, s, _ = np.linalg.svd(A); s; abs(s[0] - 1./s[2])
> array([1.61803399, 1.        , 0.61803399])
> 0.0
> >>>
> --8<---------------cut here---------------end--------------->8---
> 
> Neat!
> 
> So far, so good.  Well, let extract the ’manifest’ from this Docker
> blob.
> 
> --8<---------------cut here---------------start------------->8---
> $ docker export -o /tmp/img/re-pack.tar $(docker ps -a --format "{{.ID}}" | head -n1)
> $ tar -xf /tmp/img/re-pack.tar $(tar -tf /tmp/img/re-pack.tar | grep 'profile/manifest')
> $ cat gnu/store/7frdchgf5sqw8b83azsml3lw0h52gfbk-profile/manifest | grep -E "(\(\"python|cb68ae)" | head -n5
>     (("python"
>               "cb68ae668af2ade4b0777d82f227e5462768e9e5")
>      ("python-ipython"
>         (("python-backcall"
>          ("python-pyzmq"
> --8<---------------cut here---------------end--------------->8---
> 
> Now, a trick to get the channels and specifications:
> 
> --8<---------------cut here---------------start------------->8---
> $ ./pre-inst-env guix package -p /tmp/img/gnu/store/7frdchgf5sqw8b83azsml3lw0h52gfbk-profile --export-channels
> ;; This channel file can be passed to 'guix pull -C' or to
> ;; 'guix time-machine -C' to obtain the Guix revision that was
> ;; used to populate this profile.
> 
> (list
>      (channel
>        (name 'guix)
>        (url "https://git.savannah.gnu.org/git/guix.git")
>        (commit
>          "cb68ae668af2ade4b0777d82f227e5462768e9e5")
>        (introduction
>          (make-channel-introduction
>            "9edb3f66fd807b096b48283debdcddccfea34bad"
>            (openpgp-fingerprint
>              "BBB0 2DDF 2CEA F6A8 0D1D  E643 A2A0 6DF2 A33A 54FA"))))
> )
> 
> $ ./pre-inst-env guix package -p /tmp/img/gnu/store/7frdchgf5sqw8b83azsml3lw0h52gfbk-profile --export-manifest
> $ ./pre-inst-env guix package -p /tmp/img/gnu/store/7frdchgf5sqw8b83azsml3lw0h52gfbk-profile --export-manifest
> ;; This "manifest" file can be passed to 'guix package -m' to reproduce
> ;; the content of your profile.  This is "symbolic": it only specifies
> ;; package names.  To reproduce the exact same profile, you also need to
> ;; capture the channels being used, as returned by "guix describe".
> ;; See the "Replicating Guix" section in the manual.
> 
> (specifications->manifest
>   (list "python"
>         "python-ipython"
>         "python-numpy"
>         "python-matplotlib"
>         "python-scipy"
>         "python-biopython"))
> --8<---------------cut here---------------end--------------->8---
> 
> Awesome!
> 
> 
> The unexpected is this channels and manifests files do not reproduce the
> same Docker pack tarball:
> 
> --8<---------------cut here---------------start------------->8---
> $ guix describe
> Generation 99   Jan 05 2021 16:56:39    (current)
>   guix-past 829923f
>     repository URL: https://gitlab.inria.fr/guix-hpc/guix-past
>     branch: master
>     commit: 829923f01f894f1e687735627025ada26230832f
>   guix-bimsb a8b539d
>     repository URL: https://github.com/BIMSBbioinfo/guix-bimsb
>     branch: master
>     commit: a8b539d61a359060c35f3cb34c7edd1d9d14241d
>   bimsb-nonfree 4084e63
>     repository URL: https://github.com/BIMSBbioinfo/guix-bimsb-nonfree.git
>     branch: master
>     commit: 4084e63c9c0d662780870aded9f5a6ca1b063780
>   guix-science cf87b05
>     repository URL: https://github.com/guix-science/guix-science.git
>     branch: master
>     commit: cf87b0501c4a38b96edf41025a27bf1cb91f521a
>   guix 957f0c4
>     repository URL: https://git.savannah.gnu.org/git/guix.git
>     branch: master
>     commit: 957f0c40327ce00f53db22737e3775ce616ac258
> 
> $ guix time-machine -C /tmp/img/channels.scm -- pack -f docker --save-provenance -m /tmp/img/manifest.scm
> Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...
> /gnu/store/xzk604g8gysv4azn7sf9nylr6iah97gl-docker-pack.tar.gz
> --8<---------------cut here---------------end--------------->8---
> 
> To compare with
> /gnu/store/wxymmnxdvdvf08ifsfy39xjaxilhrigk-docker-pack.tar.gz.
> 
> On a third machine, I get:
> /gnu/store/wxymmnxdvdvf08ifsfy39xjaxilhrigk-docker-pack.tar.gz
> 
> Well, that’s another story and I have not inspected yet the
> derivations and what could be wrong on the machine B.
> 
> 
> Cheers,
> simon
> 

KUTGW ;-)

-- 
Regards,
Bengt Richter

[bug#45919] [PATCH 0/8] Exporting a manifest and channels from a profile

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1371 bytes --]

Hi again!

I just had a cursory look, and my main concern is that maybe exposing
commands is one more thing to learn for the user, and one more step to
run to get a simple file.

Instead, what about doing what system profiles already do with

  /run/current-system/channels.scm
  /run/current-system/configuration.scm

and store the result of the proposed commands directly in
$PROFILE/channels.scm and $PROFILE/manifest-spec.scm, on each profile
upgrade?

We could add a warning comment at the top, explaining the limitations of
these files (that it does not represent the profile faithfully).

Thoughts?

Last, just a nit: what about listing the packages corresponding to the
commented commits next to them?  For instance

--8<---------------cut here---------------start------------->8---
(list
 ;; Note: these other commits were also used to install some of the packages in this profile:
 ;;   "458cb25b9e7e7c954f468023abea2bebb5d8c75b" monolith obs calibre
 ;;   "4969b51d175497bfcc354c91803e9d70542b7113" 0ad augustus
 ;;   "3d85c3ec652feb22824f355538b51e6955ded361" nyxt
 ;; ...
 ;;   "b76b1d3fb65fec98b96a2b4cfa984316dd956a29" tectonic
 (channel
  (name 'guix)
  ...
--8<---------------cut here---------------end--------------->8---

Thanks for working on this, Ludo!

Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 511 bytes --]

Re: [bug#45919] [PATCH 0/8] Exporting a manifest and channels from a profile

[zimoun]

Hi Pierre and Ludo,

On Fri, 29 Jan 2021 at 18:57, Pierre Neidhardt <mail@ambrevar.xyz> wrote:
> Hi again!
>
> I just had a cursory look, and my main concern is that maybe exposing
> commands is one more thing to learn for the user, and one more step to
> run to get a simple file.
>
> Instead, what about doing what system profiles already do with
>
>   /run/current-system/channels.scm
>   /run/current-system/configuration.scm
>
> and store the result of the proposed commands directly in
> $PROFILE/channels.scm and $PROFILE/manifest-spec.scm, on each profile
> upgrade?

If I understand correctly, Pierre, you are proposing 2 things: 1) only
one command and 2) export by default in $PROFILE.  Right?

Yeah, sounds a good idea to me.  Even if #2 seems a bit of re-work for
the patch set. :-)

> Last, just a nit: what about listing the packages corresponding to the
> commented commits next to them?  For instance
>
> --8<---------------cut here---------------start------------->8---
> (list
>  ;; Note: these other commits were also used to install some of the packages in this profile:
>  ;;   "458cb25b9e7e7c954f468023abea2bebb5d8c75b" monolith obs calibre
>  ;;   "4969b51d175497bfcc354c91803e9d70542b7113" 0ad augustus
>  ;;   "3d85c3ec652feb22824f355538b51e6955ded361" nyxt
>  ;; ...
>  ;;   "b76b1d3fb65fec98b96a2b4cfa984316dd956a29" tectonic
>  (channel
>   (name 'guix)
>   ...
> --8<---------------cut here---------------end--------------->8---

I agree.  I did the same note. :-)


Cheers,
simon

Re: [bug#45919] [PATCH 0/8] Exporting a manifest and channels from a profile

[zimoun]

Hi Bengt,

On Thu, 28 Jan 2021 at 18:57, Bengt Richter <bokr@bokr.com> wrote:

> Thank you for your tutorial-style and resource-reference-footnoted
> posts!

Thanks!  I am happy if it helps.  «Déformation professionnelle»¹ :-)

Cheers,
simon


¹: French pun with «professional training» and «stretched profession».

Re: [bug#45919] [PATCH 0/8] Exporting a manifest and channels from a profile

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1221 bytes --]

zimoun <zimon.toutoune@gmail.com> writes:

>> I just had a cursory look, and my main concern is that maybe exposing
>> commands is one more thing to learn for the user, and one more step to
>> run to get a simple file.
>>
>> Instead, what about doing what system profiles already do with
>>
>>   /run/current-system/channels.scm
>>   /run/current-system/configuration.scm
>>
>> and store the result of the proposed commands directly in
>> $PROFILE/channels.scm and $PROFILE/manifest-spec.scm, on each profile
>> upgrade?
>
> If I understand correctly, Pierre, you are proposing 2 things: 1) only
> one command and 2) export by default in $PROFILE.  Right?
>
> Yeah, sounds a good idea to me.  Even if #2 seems a bit of re-work for
> the patch set. :-)

Sorry, misunderstanding :)
I'm proposing:

1) Zero command :p

2) On every profile modification (install, upgrade, removal), store the
channel description and the manifest specification (as produced by this
patch commands) to $GUIX_PROFILE/channels.scm and
$GUIX_PROFILE/manifest-spec.scm, respectively.

($GUIX_PROFILE being the target profile, of course.)

Does that make more sense?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 511 bytes --]

Re: bug#45919: [PATCH 0/8] Exporting a manifest and channels from a profile

[Ludovic Courtès]

Hi Pierre,

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> I just had a cursory look, and my main concern is that maybe exposing
> commands is one more thing to learn for the user, and one more step to
> run to get a simple file.
>
> Instead, what about doing what system profiles already do with
>
>   /run/current-system/channels.scm
>   /run/current-system/configuration.scm
>
> and store the result of the proposed commands directly in
> $PROFILE/channels.scm and $PROFILE/manifest-spec.scm, on each profile
> upgrade?
>
> We could add a warning comment at the top, explaining the limitations of
> these files (that it does not represent the profile faithfully).
>
> Thoughts?

Here are practical issues that come to mind:

  • It would only work for newer profiles, created after the feature has
    been implemented (maybe that’s okay).

  • The generated files might use APIs that, in the meantime, got
    deprecated or changed somehow.  This is in contrast with
    ‘--export-profile’, which interprets ‘manifest’ (a versioned file
    format) and produces code that can use the API du jour.

  • One would still have to learn about these two files, and pick the
    right “manifest” file.

  • For users of ‘-m my-manifest.scm’, we would need to store
    ‘my-manifest.scm’ as is instead of generating an approximation
    thereof.

We already discussed these issues at length earlier.  :-)  Again, I agree
that following what we do with /run/current-system would be nice, but
the situation is different due to imperative operations.

Hmm, needs more thought…

> Last, just a nit: what about listing the packages corresponding to the
> commented commits next to them?  For instance
>
> (list
>  ;; Note: these other commits were also used to install some of the packages in this profile:
>  ;;   "458cb25b9e7e7c954f468023abea2bebb5d8c75b" monolith obs calibre
>  ;;   "4969b51d175497bfcc354c91803e9d70542b7113" 0ad augustus
>  ;;   "3d85c3ec652feb22824f355538b51e6955ded361" nyxt
>  ;; ...
>  ;;   "b76b1d3fb65fec98b96a2b4cfa984316dd956a29" tectonic
>  (channel
>   (name 'guix)
>   ...

Would be nice.

Thanks for your feedback!

Ludo’.

[bug#45919] [PATCH 0/8] Exporting a manifest and channels from a profile

[zimoun]

Hi,

On Fri, 29 Jan 2021 at 22:51, Pierre Neidhardt <mail@ambrevar.xyz> wrote:

> Sorry, misunderstanding :)

Have a drink together is missing because these times we have
communication issues. :-)

> I'm proposing:
>
> 1) Zero command :p
>
> 2) On every profile modification (install, upgrade, removal), store the
> channel description and the manifest specification (as produced by this
> patch commands) to $GUIX_PROFILE/channels.scm and
> $GUIX_PROFILE/manifest-spec.scm, respectively.
>
> ($GUIX_PROFILE being the target profile, of course.)
>
> Does that make more sense?

Yes, but it needs a lot of re-work and change, IMHO.  And we ends with 3
files: $GUIX_PROFILE/manifest capturing the states of $GUIX_PROFILE and
the 2 others deducing an approximation of these states.  Well, IMHO,
automatically produce this approximation at each «action» on the profile
adds complexity and I am not convinced it is worth.

Cheers,
simon

Re: bug#45919: [PATCH 0/8] Exporting a manifest and channels from a profile

[Ludovic Courtès]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

> Yes, but it needs a lot of re-work and change, IMHO.  And we ends with 3
> files: $GUIX_PROFILE/manifest capturing the states of $GUIX_PROFILE and
> the 2 others deducing an approximation of these states.  Well, IMHO,
> automatically produce this approximation at each «action» on the profile
> adds complexity and I am not convinced it is worth.

Yeah.

Actually, part of the code would be shared anyway, so we could always go
with ‘--export-manifest’ first and think about adding the extra files
later.  (Though I’m still unsure about these extra files, TBH.)

Ludo’.

Re: bug#45919: [PATCH 0/8] Exporting a manifest and channels from a profile

[Ryan Prior]

[-- Attachment #1: Type: text/plain, Size: 1479 bytes --]

On January 30, 2021, "Ludovic Courtès" <ludo@gnu.org> wrote:
> Actually, part of the code would be shared anyway, so we could always
> go
> with ‘--export-manifest’ first and think about adding the extra files
> later. (Though I’m still unsure about these extra files, TBH.)

I do like the extra files. It feels cozy to me, like you have your
little profile directory and it has a manifest and a list of channels in
it, that makes it feel more complete. Like having the family pictures on
the wall.

The question of what happens if you don't update a profile for a long
time and how Guix handles those files moves on is a good one. I think
those files should have an indication in there of what version of Guix
was used to produce them, such that you can time-machine back to ensure
you're compatible.

I don't think there's much drawback to having both the auto-generated
files and a command that generates them. That seems more discoverable -
you might happen across the files when you poke into a profile, or you
might notice the command while reading the docs or the help output.

Glad to see this capability land any which way, this is something that
comes up often!

While we're considering putting a manifest in the profile, is this a
good time to also bring up the idea of renaming the "manifest" file? It
confuses more people all the time. I'd be inclined to rename it
"profile-metadata" or "lockfile". 

[-- Attachment #2: Type: text/html, Size: 3805 bytes --]

Re: bug#45919: [PATCH 0/8] Exporting a manifest and channels from a profile

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 2071 bytes --]

Hi!

Ludovic Courtès <ludo@gnu.org> writes:

> Here are practical issues that come to mind:
>
>   • It would only work for newer profiles, created after the feature has
>     been implemented (maybe that’s okay).

Indeed, I think it's OK! :)

>   • The generated files might use APIs that, in the meantime, got
>     deprecated or changed somehow.  This is in contrast with
>     ‘--export-profile’, which interprets ‘manifest’ (a versioned file
>     format) and produces code that can use the API du jour.

/run/current-system/configuration.scm suffers from the same problem.
But with the manifest we could do better, we could include a version
number one way or another.
Besides, since it comes together with channels.scm, we know which Guix
was used, so we always have access to the Guix with the right API to
install the manifest.

All in all, I think this is not a problem.

>   • One would still have to learn about these two files, and pick the
>     right “manifest” file.

I think it would be easier than a command. See below.

>   • For users of ‘-m my-manifest.scm’, we would need to store
>     ‘my-manifest.scm’ as is instead of generating an approximation
>     thereof.

Which seems easy to do, isn't it?


Another use-case which I find useful and comes close to this feature is
that of channel/manifest versioning, in the sense of keeping these files
under version control for instance in a Git repository.  This can be
useful to keep the history of everything, even deleted generations, or
even in case of hardware failure.

To that end, it'd be nice if we could export these files automatically
to a designated location.

Example: I update ~/my-profile and it automatically produces / overwrite
~/repos/guix-profile-metadata.git/my-profile/channels.scm and
~/repos/guix-profile-metadata.git/my-profile/manifest.scm.

This way I can commit these 2 files in my guix-profile-metadata.git
repository.

Food for thoughts :)

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 511 bytes --]

Re: bug#45919: [PATCH 0/8] Exporting a manifest and channels from a profile

[Ludovic Courtès]

Hi,

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:

[...]

>>   • The generated files might use APIs that, in the meantime, got
>>     deprecated or changed somehow.  This is in contrast with
>>     ‘--export-profile’, which interprets ‘manifest’ (a versioned file
>>     format) and produces code that can use the API du jour.
>
> /run/current-system/configuration.scm suffers from the same problem.

Not really because it’s precisely the file that you gave to build the
system.  So you know you can run:

  cd /run/current-system
  guix time-machine -C channels.scm -- system reconfigure configuration.scm

and it’ll work (modulo the documented caveats).

> But with the manifest we could do better, we could include a version
> number one way or another.
> Besides, since it comes together with channels.scm, we know which Guix
> was used, so we always have access to the Guix with the right API to
> install the manifest.

Right.

>>   • One would still have to learn about these two files, and pick the
>>     right “manifest” file.
>
> I think it would be easier than a command. See below.
>
>>   • For users of ‘-m my-manifest.scm’, we would need to store
>>     ‘my-manifest.scm’ as is instead of generating an approximation
>>     thereof.
>
> Which seems easy to do, isn't it?

I take it that you’re volunteering?  :-)

Nothing’s difficult, but in this case we’d need to pass the original
manifest down to ‘profile-generation’.  Requires some redesign.

> Another use-case which I find useful and comes close to this feature is
> that of channel/manifest versioning, in the sense of keeping these files
> under version control for instance in a Git repository.  This can be
> useful to keep the history of everything, even deleted generations, or
> even in case of hardware failure.
>
> To that end, it'd be nice if we could export these files automatically
> to a designated location.
>
> Example: I update ~/my-profile and it automatically produces / overwrite
> ~/repos/guix-profile-metadata.git/my-profile/channels.scm and
> ~/repos/guix-profile-metadata.git/my-profile/manifest.scm.
>
> This way I can commit these 2 files in my guix-profile-metadata.git
> repository.

I guess you could do that either with ‘cp
~/.guix-profile/{channels,manifest.scm …’ or with ‘guix package
--export-manifest … > …’.

But again, that’s not really the goal here.  The goal is to help users
willing to migrate from the “imperative” mode to the declarative mode.
Once you’re using a manifest, probably you’ll want to put that under
version control, but that’s already the case.

Thanks,
Ludo’.

Re: bug#45919: [PATCH 0/8] Exporting a manifest and channels from a profile

[Ludovic Courtès]

Hi,

Ryan Prior <ryanprior@hey.com> skribis:

> I don't think there's much drawback to having both the auto-generated
> files and a command that generates them. That seems more discoverable -
> you might happen across the files when you poke into a profile, or you
> might notice the command while reading the docs or the help output.

Note that the ‘manifest’ file starts with a comment.  This patch set
updates this comment so it mentions ‘--export-channels’.

> Glad to see this capability land any which way, this is something that
> comes up often!

Heh, good.  :-)

> While we're considering putting a manifest in the profile, is this a
> good time to also bring up the idea of renaming the "manifest" file? It
> confuses more people all the time. I'd be inclined to rename it
> "profile-metadata" or "lockfile". 

It’s the kind of thing that can’t really be changed, or at least not
without a looong transition period, because older ‘guix package’
commands wouldn’t recognize profiles that lack a ‘manifest’ file.

Thanks for your feedback,
Ludo’.

Re: bug#45919: [PATCH 0/8] Exporting a manifest and channels from a profile

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1760 bytes --]

Ludovic Courtès <ludo@gnu.org> writes:

> Hi,
>
> Pierre Neidhardt <mail@ambrevar.xyz> skribis:
>
>> Ludovic Courtès <ludo@gnu.org> writes:
>
> [...]
>
>>>   • The generated files might use APIs that, in the meantime, got
>>>     deprecated or changed somehow.  This is in contrast with
>>>     ‘--export-profile’, which interprets ‘manifest’ (a versioned file
>>>     format) and produces code that can use the API du jour.
>>
>> /run/current-system/configuration.scm suffers from the same problem.
>
> Not really because it’s precisely the file that you gave to build the
> system.  So you know you can run:
>
>   cd /run/current-system
>   guix time-machine -C channels.scm -- system reconfigure configuration.scm
>
> and it’ll work (modulo the documented caveats).

Ha, actually no, it won't always work! :)
Configuration.scm is only the file passed to `guix system reconfigure`,
but is this file inherits from other user Guile modules, then the
configuration.scm won't be sufficient to regenerate the system.

You could argue that it's a user problem, but still, there is an
"impurity" here! :p

> But again, that’s not really the goal here.  The goal is to help users
> willing to migrate from the “imperative” mode to the declarative mode.

Sure, but my thinking is that we could hit two birds we one stone here.
If we only think one half of the problem, we might end up with too many
commands or an interface that's not flexible enough.

> Once you’re using a manifest, probably you’ll want to put that under
> version control, but that’s already the case.

The channels.scm files are missing though.

My two cents! :)

Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 511 bytes --]

bug#45919: [PATCH 0/8] Exporting a manifest and channels from a profile

[Ludovic Courtès]

Hi!

So I went ahead and pushed this patch series as
15078567c17851ef0f2b017119f305e0d5e8a140.

We can always improve from here, and hopefully getting actual user
feedback will help us see the pros and cons of this option.

Thanks!

Ludo’.

Unreproducible “guix pack -f docker” because config.scm-builder

[zimoun]

Hi Ludo,

On Thu, 28 Jan 2021 at 16:54, Ludovic Courtès <ludo@gnu.org> wrote:

>> $ guix time-machine -C /tmp/img/channels.scm -- pack -f docker --save-provenance -m /tmp/img/manifest.scm
>> Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...
>> /gnu/store/xzk604g8gysv4azn7sf9nylr6iah97gl-docker-pack.tar.gz
>>
>> To compare with
>> /gnu/store/wxymmnxdvdvf08ifsfy39xjaxilhrigk-docker-pack.tar.gz.
>>
>> On a third machine, I get:
>> /gnu/store/wxymmnxdvdvf08ifsfy39xjaxilhrigk-docker-pack.tar.gz
>>
>> Well, that’s another story and I have not inspected yet the
>> derivations and what could be wrong on the machine B.
>
> You’d have to check the differences.  It may be that provenance data
> differs, for example because the second attempt includes data about
> channels that are actually unused.  (That’s the whole problem of
> provenance data: it’s not a one-to-one mapping and it’s not a bijection
> either.)

After inspecting the derivations, the issue is from the file
’config.scm-builder’ which differs by:

    (define-public %sysconfdir "/usr/local/etc")

vs

    (define-public %sysconfdir "/etc")


What did I do wrong?  From where does this difference come?  How can I
fix it?



Below, the different commands to spot out the issue.

Cheers,
simon


Machine A

--8<---------------cut here---------------start------------->8---
$ guix describe
Generation 101  Jan 29 2021 16:22:06    (current)
  guix b9a54aa
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: b9a54aad0ba282ac78931b67e679bd0132419364
$ guix describe -f channels > /tmp/channels.scm
$ guix pack -f docker hello
/gnu/store/9vhl75vx60l56992hgy5818ndic608p5-docker-pack.tar.gz

$ guix time-machine -C /tmp/channels.scm -- pack -f docker hello
/gnu/store/9vhl75vx60l56992hgy5818ndic608p5-docker-pack.tar.gz
$ guix gc --derivers /gnu/store/9vhl75vx60l56992hgy5818ndic608p5-docker-pack.tar.gz                                        
/gnu/store/ih94c9ny68dfalrym9m1vz4wa40rpgvs-docker-pack.tar.gz.drv                                                                      
--8<---------------cut here---------------end--------------->8---

07fwgiz56f7dk760qpplnfaxribglqc7-config.scm-builder:

    (define-public %sysconfdir "/usr/local/etc")
    

Machine B (and C)

--8<---------------cut here---------------start------------->8---
$ guix describe
Génération 7    28 janv. 2021 01:51:17  (actuelle)
   guix 0f20b3f
     URL du dépôt : https://git.savannah.gnu.org/git/guix.git
     branche: master
     commit : 0f20b3fa2050ba6e442e340a204516b9375cd231
$ cat /tmp/channels.scm
(list (channel
        (name 'guix)
        (url "https://git.savannah.gnu.org/git/guix.git")
        (commit
          "b9a54aad0ba282ac78931b67e679bd0132419364")
        (introduction
          (make-channel-introduction
            "9edb3f66fd807b096b48283debdcddccfea34bad"
            (openpgp-fingerprint
              "BBB0 2DDF 2CEA F6A8 0D1D  E643 A2A0 6DF2 A33A 54FA")))))
$ guix time-machine -C /tmp/channels.scm -- pack -f docker hello
Mise à jour du canal « guix » depuis le dépôt Git «https://git.savannah.gnu.org/git/guix.git »...
/gnu/store/nkvlqwzvxdlhzlc7vhfcngxc19x2ay2f-docker-pack.tar.gz

$ guix pull --commit=b9a54aad0ba282ac78931b67e679bd0132419364
$ guix describe
Génération 8    01 févr. 2021 17:00:18  (actuelle)
   guix b9a54aa
     URL du dépôt : https://git.savannah.gnu.org/git/guix.git
     commit : b9a54aad0ba282ac78931b67e679bd0132419364
$ guix pack -f docker hello
/gnu/store/nkvlqwzvxdlhzlc7vhfcngxc19x2ay2f-docker-pack.tar.gz
$ guix gc --derivers /gnu/store/nkvlqwzvxdlhzlc7vhfcngxc19x2ay2f-docker-pack.tar.gz           
/gnu/store/323k33sfx869d0nkh69ary8sj6xiy4s4-docker-pack.tar.gz.drv
--8<---------------cut here---------------end--------------->8---

00cy802583sb82kcpzfd1sz1xwff9hln-config.scm-builder:

    (define-public %sysconfdir "/etc")           

Re: Unreproducible “guix pack -f docker” because config.scm-builder

[zimoun]

Hi,

In case someone reads and is interested by the fix.

On Mon, 1 Feb 2021 at 23:42, zimoun <zimon.toutoune@gmail.com> wrote:

> After inspecting the derivations, the issue is from the file
> ’config.scm-builder’ which differs by:
>
>     (define-public %sysconfdir "/usr/local/etc")
>
> vs
>
>     (define-public %sysconfdir "/etc")

Ludo explained the issue on IRC [1].  Well, I should have done
"./pre-inst-env guix pull" with a misconfiguration from source.

Therefore, to fix I did:

  ./configure --localstatedir=/var/ --sysconfigdir=/etc
  make
  ./pre-env-inst guix pull

and everything works fine. :-)


The tarballs are not the same:

A$ md5sum /gnu/store/nkvlqwzvxdlhzlc7vhfcngxc19x2ay2f-docker-pack.tar.gz
b5fe393d7966cbc3cd0be6e51d3aedc3
/gnu/store/nkvlqwzvxdlhzlc7vhfcngxc19x2ay2f-docker-pack.tar.gz

B$ md5sum /gnu/store/nkvlqwzvxdlhzlc7vhfcngxc19x2ay2f-docker-pack.tar.gz
e47b9a38b7162f7fb093b97e19dbc1ca
/gnu/store/nkvlqwzvxdlhzlc7vhfcngxc19x2ay2f-docker-pack.tar.gz

but it is another story, I guess.  Investigating...

--8<---------------cut here---------------start------------->8---
$ diff -r --no-dereference /tmp/docker-{pfiuh02,meary}
Les fichiers binaires
/tmp/docker-pfiuh02/4ca83868d5e98cb06179a2a7372afe029c10d43bdc9fbfcc5771b89da74889b8/layer.tar
et /tmp/docker-meary/4ca83868d5e98cb06179a2a7372afe029c10d43bdc9fbfcc5771b89da74889b8/layer.tar
sont différents
diff -r --no-dereference /tmp/docker-pfiuh02/config.json
/tmp/docker-meary/config.json
1c1
< {"architecture":"amd64","comment":"Generated by GNU
Guix","created":"1970-01-01T00:00:01Z","config":{"env":["PATH=/gnu/store/251bsvdbnggpjl4iv259sjdn6x4d1ly1-profile/bin"]},"container_config":null,"os":"linux","rootfs":{"type":"layers","diff_ids":["sha256:ac307c1c2da51ff6bbcd2d227843f2526028ef215068b36f16c7d0ecc62af397"]}}
\ Pas de fin de ligne à la fin du fichier
---
> {"architecture":"amd64","comment":"Generated by GNU Guix","created":"1970-01-01T00:00:01Z","config":{"env":["PATH=/gnu/store/251bsvdbnggpjl4iv259sjdn6x4d1ly1-profile/bin"]},"container_config":null,"os":"linux","rootfs":{"type":"layers","diff_ids":["sha256:95f38c548ea3ac95ec62b5ac59ef5099d03cf1dd4d2c9f8851ac21e8a7f0ee92"]}}
\ Pas de fin de ligne à la fin du fichier
--8<---------------cut here---------------end--------------->8---


All the best,
simon

1: <http://logs.guix.gnu.org/guix/2021-02-02.log#122945>

Re: Unreproducible “guix pack -f docker” because config.scm-builder

[Ludovic Courtès]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

> After inspecting the derivations, the issue is from the file
> ’config.scm-builder’ which differs by:
>
>     (define-public %sysconfdir "/usr/local/etc")
>
> vs
>
>     (define-public %sysconfdir "/etc")
>
>
> What did I do wrong?  From where does this difference come?  How can I
> fix it?

As discussed on IRC, ‘guix pull’ preserves the settings of the host
Guix.  So if your initial ‘guix’ has %sysconfdir set to /usr/local/etc,
‘guix pull’ will preserve that.

It turns out that, as is always the case with GNU Standards compliant
configure script, the default value for --prefix is /usr/local, and the
default for --sysconfdir is $prefix/etc.

Mostly likely, what happened is that at some point you built Guix from
source using the default prefix and sysconfdir, and then you ran ‘guix
pull’ from that Guix.

You did find other differences eventually though, right?

HTH,
Ludo’.

Re: Unreproducible “guix pack -f docker” because config.scm-builder

[zimoun]

Hi,

On Tue, 02 Feb 2021 at 19:12, Ludovic Courtès <ludo@gnu.org> wrote:

> It turns out that, as is always the case with GNU Standards compliant
> configure script, the default value for --prefix is /usr/local, and the
> default for --sysconfdir is $prefix/etc.

As discussed on IRC, it is not mentioned in the manual.  What the manual
describes is:

  ./bootstrap
  ./configure --localstatedir=/var/
  make

therefore, if one runs:

  ./pre-inst-env guix pull

then the sysconfdir is set to /usr/local/etc because it is the default.
And so it leads to subtle differences really hard to guess.  I think it
is worth to add one sentence or footnote at the end of the section
«Running Guix Before It Is Installed», right after:

        Note that ./pre-inst-env guix pull does not upgrade the local
        source tree; it simply updates the ~/.config/guix/current
        symlink (see Invoking guix pull). Run git pull instead if you
        want to upgrade your local source tree.

Something like: «Note that ’guix pull’ preserves the settings of the host
Guix, for instance ’sysconfdir’, and by default the GNU standards set
’prefix’ to ’/usr/local/’ and ’sysconfdir’ to ’$prefix/etc’, whereas
regular Guix uses ’--sysconfdir=/etc/’.»

WDYT?


> You did find other differences eventually though, right?

The produced tarballs have the same Guix hash, i.e., all the same
inputs, but not the same outputs, compare with commit b9a54aa:


A-machine$ md5sum /gnu/store/nkvlqwzvxdlhzlc7vhfcngxc19x2ay2f-docker-pack.tar.gz
b5fe393d7966cbc3cd0be6e51d3aedc3 /gnu/store/nkvlqwzvxdlhzlc7vhfcngxc19x2ay2f-docker-pack.tar.gz

B-machine$ md5sum /gnu/store/nkvlqwzvxdlhzlc7vhfcngxc19x2ay2f-docker-pack.tar.gz
e47b9a38b7162f7fb093b97e19dbc1ca /gnu/store/nkvlqwzvxdlhzlc7vhfcngxc19x2ay2f-docker-pack.tar.gz

C-machine$ md5sum /gnu/store/nkvlqwzvxdlhzlc7vhfcngxc19x2ay2f-docker-pack.tar.gz
b5fe393d7966cbc3cd0be6e51d3aedc3  /gnu/store/nkvlqwzvxdlhzlc7vhfcngxc19x2ay2f-docker-pack.tar.gz


And diffoscope returns these differences, something about links, IIUC:

--8<---------------cut here---------------start------------->8---
--- /tmp/docker-meary/4ca83868d5e98cb06179a2a7372afe029c10d43bdc9fbfcc5771b89da74889b8/layer.tar
+++ /tmp/docker-pfiuh02/4ca83868d5e98cb06179a2a7372afe029c10d43bdc9fbfcc5771b89da74889b8/layer.tar
├── file list
│ @@ -823,17 +823,17 @@

[...]

│ --r-x… 29960 gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/libexec/getconf/POSIX_V6_LP64_OFF64

[...]

│ +hr-x…     0 gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/libexec/getconf/POSIX_V6_LP64_OFF64

[...]

├── gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/libexec/getconf/POSIX_V6_LP64_OFF64
│ @@ -1,1873 +0,0 @@
│ -00000000: 7f45 4c46 0201 0100 0000 0000 0000 0000  .ELF............
│ -00000010: 0200 3e00 0100 0000 3015 4000 0000 0000  ..>.....0.@.....
│ -00000020: 4000 0000 0000 0000 486d 0000 0000 0000  @.......Hm......
│ -00000030: 0000 0000 4000 3800 0b00 4000 1f00 1e00  ....@.8...@.....
│ -00000040: 0600 0000 0400 0000 4000 0000 0000 0000  ........@.......
│ -00000050: 4000 4000 0000 0000 4000 4000 0000 0000  @.@.....@.@.....
│ -00000060: 6802 0000 0000 0000 6802 0000 0000 0000  h.......h.......

[...]
--8<---------------cut here---------------end--------------->8---

(Slightly edited to ease the reading, and raw at
<https://paste.debian.net/1183728/>.)


On machines A and C, empty file because hard link.  But on machine B,
the files are there.  Tiny files I guess, the size difference is:
23104 vs 23136.

Cheers,
simon

Re: Unreproducible “guix pack -f docker” because config.scm-builder

[Ludovic Courtès]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

> then the sysconfdir is set to /usr/local/etc because it is the default.
> And so it leads to subtle differences really hard to guess.  I think it
> is worth to add one sentence or footnote at the end of the section
> «Running Guix Before It Is Installed», right after:
>
>         Note that ./pre-inst-env guix pull does not upgrade the local
>         source tree; it simply updates the ~/.config/guix/current
>         symlink (see Invoking guix pull). Run git pull instead if you
>         want to upgrade your local source tree.
>
> Something like: «Note that ’guix pull’ preserves the settings of the host
> Guix, for instance ’sysconfdir’, and by default the GNU standards set
> ’prefix’ to ’/usr/local/’ and ’sysconfdir’ to ’$prefix/etc’, whereas
> regular Guix uses ’--sysconfdir=/etc/’.»
>
> WDYT?

As often, I have mixed feelings: we would end up +/- duplicating the
Standards in the manual, possibly even without citing the primary source
(I could be a Wikipedian :-)).  So to me that’s not great.

OTOH, as you write, letting people stumble upon this kind of issue is
not an option, either.

So I guess you can propose a patch and let someone else review it.  :-)

>> You did find other differences eventually though, right?
>
> The produced tarballs have the same Guix hash, i.e., all the same
> inputs, but not the same outputs, compare with commit b9a54aa:

[...]

> --- /tmp/docker-meary/4ca83868d5e98cb06179a2a7372afe029c10d43bdc9fbfcc5771b89da74889b8/layer.tar
> +++ /tmp/docker-pfiuh02/4ca83868d5e98cb06179a2a7372afe029c10d43bdc9fbfcc5771b89da74889b8/layer.tar
> ├── file list
> │ @@ -823,17 +823,17 @@
>
> [...]
>
> │ --r-x… 29960 gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/libexec/getconf/POSIX_V6_LP64_OFF64
>
> [...]
>
> │ +hr-x…     0 gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/libexec/getconf/POSIX_V6_LP64_OFF64

Looks like tar made this file a hard link in one case and not in the
other.  This is weird because we don’t ask it to create hard links
(there’s even a comment in (guix scripts pack)).

Is this docker image the result of the same derivation?  Could you try
building that derivation on different machines?  (You can copy the .drv
around with ‘guix copy’.)

Could you also show the output of:

  stat /gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/libexec/getconf/POSIX_V6_LP64_OFF64

on the two machines you used?

I wonder if it could be that tar nowadays decides to preserve hard links
by default and one of your machine had this file hard-linked while the
other one didn’t.

Ludo’.

Re: Unreproducible “guix pack -f docker” because config.scm-builder

[zimoun]

Hi Ludo,

On Fri, 05 Feb 2021 at 11:09, Ludovic Courtès <ludo@gnu.org> wrote:

> So I guess you can propose a patch and let someone else review it.
> :-)

I will. :-)


> Looks like tar made this file a hard link in one case and not in the
> other.  This is weird because we don’t ask it to create hard links
> (there’s even a comment in (guix scripts pack)).

I have 2 machines running Debian and one running Ubuntu.  The 2 Debian
produces the same things.  Ubuntu not.


> Is this docker image the result of the same derivation?  Could you try
> building that derivation on different machines?  (You can copy the .drv
> around with ‘guix copy’.)

It was built with different derivations.  I mean the scenario where
Alice wants to rebuild what Bob did.

Now, let use the same derivation.  Some details in case I am doing
something wrong:

--8<---------------cut here---------------start------------->8---
$ echo Ubuntu

$ guix gc -D $(guix pack -f docker hello)
finding garbage collector roots...
[0 MiB] deleting
'/gnu/store/nkvlqwzvxdlhzlc7vhfcngxc19x2ay2f-docker-pack.tar.gz'
deleting `/gnu/store/trash'
deleting unused links...
  C-c C-c
 
$ guix gc -D $(guix pack -f docker hello -d)
finding garbage collector roots...
[0 MiB] deleting
'/gnu/store/323k33sfx869d0nkh69ary8sj6xiy4s4-docker-pack.tar.gz.drv'
deleting `/gnu/store/trash'
deleting unused links...
  C-c C-c
--8<---------------cut here---------------end--------------->8---

--8<---------------cut here---------------start------------->8---
$ echo Debian

$ guix copy $(guix pack -f docker hello -d) --to=meary
guix copy: sending 1 store item (0 MiB) to '193.48.40.110'...
/gnu/store/323k33sfx869d0nkh69ary8sj6xiy4s4-docker-pack.tar.gz.drv
--8<---------------cut here---------------end--------------->8---

--8<---------------cut here---------------start------------->8---
$ echo Ubuntu

$ guix build /gnu/store/323k33sfx869d0nkh69ary8sj6xiy4s4-docker-pack.tar.gz.drv
substitute: mise à jour des substituts depuis « https://ci.guix.gnu.org»... 100.0 %
La dérivation suivante sera compilée :
/gnu/store/323k33sfx869d0nkh69ary8sj6xiy4s4-docker-pack.tar.gz.drv                                                                   
construction de
/gnu/store/323k33sfx869d0nkh69ary8sj6xiy4s4-docker-pack.tar.gz.drv...
tar: Removing leading `/' from member names
tar: Removing leading `/' from hard link targets
construction de
/gnu/store/323k33sfx869d0nkh69ary8sj6xiy4s4-docker-pack.tar.gz.drv réussie
/gnu/store/nkvlqwzvxdlhzlc7vhfcngxc19x2ay2f-docker-pack.tar.gz  
--8<---------------cut here---------------end--------------->8---

So tar is appearing here… Hum?!  However, if I redo the same steps, it
does not.  Well, I do not like that…

Building on Ubuntu using the derivation from Debian gives the same image
as building on Ubuntu using the derivation from Ubuntu.

I thought the tools were captured by the commit: same commit, same
tools.  Well, I miss something…

> Could you also show the output of:
>
>   stat /gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/libexec/getconf/POSIX_V6_LP64_OFF64
>
> on the two machines you used?

First on Debian and second on Ubuntu

--8<---------------cut here---------------start------------->8---
$ stat /gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/libexec/getconf/POSIX_V6_LP64_OFF64
  File: /gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/libexec/getconf/POSIX_V6_LP64_OFF64
  Size: 29960           Blocks: 64         IO Block: 4096   regular file
Device: 801h/2049d      Inode: 8129616     Links: 5
Access: (0555/-r-xr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2021-02-05 17:37:15.728728952 +0100
Modify: 1970-01-01 01:00:01.000000000 +0100
Change: 2020-06-17 12:40:06.389935679 +0200
Birth: -
--8<---------------cut here---------------end--------------->8---

--8<---------------cut here---------------start------------->8---
$ stat /gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/libexec/getconf/POSIX_V6_LP64_OFF64
Fichier : /gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/libexec/getconf/POSIX_V6_LP64_OFF64
Taille : 29960       Blocs : 64         Blocs d'E/S : 4096   fichier
Périphérique : 814h/2068d       Inœud : 148900093   Liens : 1
Accès : (0555/-r-xr-xr-x)  UID : (    0/    root)   GID : (    0/ root)
Accès : 2021-02-05 17:46:05.537903382 +0100
Modif. : 1970-01-01 01:00:01.000000000 +0100
Changt : 2020-12-04 23:16:33.155711694 +0100
Créé : -
--8<---------------cut here---------------end--------------->8---
  


> I wonder if it could be that tar nowadays decides to preserve hard links
> by default and one of your machine had this file hard-linked while the
> other one didn’t.

Wow, I thought that the same tar was used the one provided by Guix and
not by the host.  For sure the default tar are not the same on both
machine (v1.30 for Debian and v1.29 for Ubuntu).

What is the md5sum checksum of ’guix pack -f docker hello’ on Guix
System for commit b9a54aa?

Thanks for the help,
simon

Re: Unreproducible “guix pack -f docker” because config.scm-builder

[Ludovic Courtès]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

> First on Debian and second on Ubuntu
>
> $ stat /gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/libexec/getconf/POSIX_V6_LP64_OFF64
>   File: /gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/libexec/getconf/POSIX_V6_LP64_OFF64
>   Size: 29960           Blocks: 64         IO Block: 4096   regular file
> Device: 801h/2049d      Inode: 8129616     Links: 5
> Access: (0555/-r-xr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
> Access: 2021-02-05 17:37:15.728728952 +0100
> Modify: 1970-01-01 01:00:01.000000000 +0100
> Change: 2020-06-17 12:40:06.389935679 +0200
> Birth: -
>
> $ stat /gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/libexec/getconf/POSIX_V6_LP64_OFF64
> Fichier : /gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/libexec/getconf/POSIX_V6_LP64_OFF64
> Taille : 29960       Blocs : 64         Blocs d'E/S : 4096   fichier
> Périphérique : 814h/2068d       Inœud : 148900093   Liens : 1
> Accès : (0555/-r-xr-xr-x)  UID : (    0/    root)   GID : (    0/ root)
> Accès : 2021-02-05 17:46:05.537903382 +0100
> Modif. : 1970-01-01 01:00:01.000000000 +0100
> Changt : 2020-12-04 23:16:33.155711694 +0100
> Créé : -

See?  One has just 1 link (did you disable deduplication on that one?),
there other has 5 links.

>> I wonder if it could be that tar nowadays decides to preserve hard links
>> by default and one of your machine had this file hard-linked while the
>> other one didn’t.
>
> Wow, I thought that the same tar was used the one provided by Guix and
> not by the host.  For sure the default tar are not the same on both
> machine (v1.30 for Debian and v1.29 for Ubuntu).
>
> What is the md5sum checksum of ’guix pack -f docker hello’ on Guix
> System for commit b9a54aa?

--8<---------------cut here---------------start------------->8---
$ md5sum $(guix time-machine --commit=b9a54aa -- pack -f docker hello)
b5fe393d7966cbc3cd0be6e51d3aedc3  /gnu/store/nkvlqwzvxdlhzlc7vhfcngxc19x2ay2f-docker-pack.tar.gz
$ sha1sum $(guix time-machine --commit=b9a54aa -- pack -f docker hello)
a4dff93de6c0bc65eb325ebfeea398ebea944a85  /gnu/store/nkvlqwzvxdlhzlc7vhfcngxc19x2ay2f-docker-pack.tar.gz
$ guix time-machine --commit=b9a54aa -- pack -f docker hello -d
/gnu/store/323k33sfx869d0nkh69ary8sj6xiy4s4-docker-pack.tar.gz.drv
$ (mkdir /tmp/docker; tar xf $(guix time-machine --commit=b9a54aa -- pack -f docker hello ); tar tvf */layer.tar | grep ^h | head)
hr-xr-xr-x root/root         0 1970-01-01 01:00 gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/libexec/getconf/POSIX_V6_LP64_OFF64 kunligo al gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/bin/getconf
hr-xr-xr-x root/root         0 1970-01-01 01:00 gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/libexec/getconf/POSIX_V7_LP64_OFF64 kunligo al gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/bin/getconf
hr-xr-xr-x root/root         0 1970-01-01 01:00 gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/libexec/getconf/XBS5_LP64_OFF64 kunligo al gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/bin/getconf
hr--r--r-- root/root         0 1970-01-01 01:00 gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib/lib/gcc/x86_64-unknown-linux-gnu/7.5.0/crtendS.o kunligo al gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib/lib/gcc/x86_64-unknown-linux-gnu/7.5.0/crtend.o
hr--r--r-- root/root         0 1970-01-01 01:00 gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib/lib/gcc/x86_64-unknown-linux-gnu/7.5.0/install-tools/gsyslimits.h kunligo al gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib/lib/gcc/x86_64-unknown-linux-gnu/7.5.0/include-fixed/syslimits.h
hr--r--r-- root/root         0 1970-01-01 01:00 gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib/lib/gcc/x86_64-unknown-linux-gnu/7.5.0/install-tools/include/README kunligo al gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib/lib/gcc/x86_64-unknown-linux-gnu/7.5.0/include-fixed/README
hr--r--r-- root/root         0 1970-01-01 01:00 gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib/lib/gcc/x86_64-unknown-linux-gnu/7.5.0/install-tools/include/limits.h kunligo al gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib/lib/gcc/x86_64-unknown-linux-gnu/7.5.0/include-fixed/limits.h
hr--r--r-- root/root         0 1970-01-01 01:00 gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib/lib/gcc/x86_64-unknown-linux-gnu/7.5.0/plugin/include/gsyslimits.h kunligo al gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib/lib/gcc/x86_64-unknown-linux-gnu/7.5.0/include-fixed/syslimits.h
hrwxrwxrwx root/root         0 1970-01-01 01:00 gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib/lib/gcc/x86_64-unknown-linux-gnu/7.5.0/plugin/libcc1plugin.so.0 kunligo al gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib/lib/gcc/x86_64-unknown-linux-gnu/7.5.0/plugin/libcc1plugin.so
hrwxrwxrwx root/root         0 1970-01-01 01:00 gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib/lib/gcc/x86_64-unknown-linux-gnu/7.5.0/plugin/libcp1plugin.so.0 kunligo al gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib/lib/gcc/x86_64-unknown-linux-gnu/7.5.0/plugin/libcp1plugin.so
--8<---------------cut here---------------end--------------->8---

Most of these ‘h’ lines for hard links; ‘libcc1plugin.so’ seems to be an
exception (it’s a symlink).

Having looked at the tar manual (info "(tar) hard links"), it seems
clear that this is the default behavior and that we have to pass
‘--hard-dereference’ to avoid it.

Thanks,
Ludo’.

[PATCH] Fix unreproducible “guix pack -f docker” (hard link)

[zimoun]

[-- Attachment #1: Type: text/plain, Size: 467 bytes --]

Hi Ludo,

On Sat, 06 Feb 2021 at 22:46, Ludovic Courtès <ludo@gnu.org> wrote:

> Having looked at the tar manual (info "(tar) hard links"), it seems
> clear that this is the default behavior and that we have to pass
> ‘--hard-dereference’ to avoid it.

Attached a patch that fixes the issue, I guess.  Well, I am not sure to
understand all correctly, yet. :-)

Now the 2 images have the same checksum on my 2 machines.

WDYT?


Cheers,
simon


[-- Attachment #2: docker.patch --]
[-- Type: text/x-diff, Size: 1225 bytes --]

From 70e113ecccc831d2a111f7c8598157d41fe0de86 Mon Sep 17 00:00:00 2001
From: zimoun <zimon.toutoune@gmail.com>
Date: Mon, 8 Feb 2021 19:37:25 +0100
Subject: [PATCH] guix: docker: Fix undeterministic hard link.

* guix/docker.scm (%tar-determinism-options): Add '--hard-dereference'.
---
 guix/docker.scm | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/guix/docker.scm b/guix/docker.scm
index 97ac6d982b..39e85b7fc2 100644
--- a/guix/docker.scm
+++ b/guix/docker.scm
@@ -2,6 +2,7 @@
 ;;; Copyright © 2017 Ricardo Wurmus <rekado@elephly.net>
 ;;; Copyright © 2017, 2018, 2019 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2018 Chris Marusich <cmmarusich@gmail.com>
+;;; Copyright © 2021 Simon Tournier <zimon.toutoune@gmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -112,7 +113,7 @@ Return a version of TAG that follows these rules."
 
 (define %tar-determinism-options
   ;; GNU tar options to produce archives deterministically.
-  '("--sort=name" "--mtime=@1"
+  '("--sort=name" "--mtime=@1" "--hard-dereference"
     "--owner=root:0" "--group=root:0"))
 
 (define directive-file

base-commit: 4cd8bab7d4b345c5538b8d039d84a957df7a235f
-- 
2.29.2

Re: Unreproducible “guix pack -f docker” because config.scm-builder

[Ludovic Courtès]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

> On Sat, 06 Feb 2021 at 22:46, Ludovic Courtès <ludo@gnu.org> wrote:
>
>> See?  One has just 1 link (did you disable deduplication on that one?),
>> there other has 5 links.
>
> Yes, I see but I do not understand why.  I have not changed, well, only
> the number of cores and jobs:
>
> ExecStart=/var/guix/profiles/per-user/root/current-guix/bin/guix-daemon --build-users-group=guixbuild -c 128 -M 99
> Environment='GUIX_LOCPATH=/var/guix/profiles/per-user/root/guix-profile/lib/locale' LC_ALL=en_US.utf8
>
>
> And my 2 Debian systems give the same result, one is the default and the
> other is:
>
> ExecStart=/var/guix/profiles/per-user/root/current-guix/bin/guix-daemon --build-users-group=guixbuild --gc-keep-outputs=yes --gc-keep-derivations=yes
>
>
>
> There is another configuration file that I could miss?

No.

I’m not sure exactly why it happened, but it’s clear that (guix docker)
would let the nlink value influence tar.

>> Having looked at the tar manual (info "(tar) hard links"), it seems
>> clear that this is the default behavior and that we have to pass
>> ‘--hard-dereference’ to avoid it.
>
> Do you mean in (guix docker) here:
>
>           (apply invoke "tar" "-cf" "../layer.tar"
>                  `(,@transformation-options
>                    ,@%tar-determinism-options
>                    ,@paths
>                    ,@(scandir "."
>                               (lambda (file)
>                                 (not (member file '("." ".."))))))))
>
> if I read correctly?
>
> And the option «-h, --dereference: Follow symlinks; archive and dump the
> files they point to.» isn’t enough?

It’s really ‘--hard-dereference’.

Pushed as 18a4882e3029a084d2f0c63d9d0148682a854546, thank you!

Ludo’.

Re: Unreproducible “guix pack -f docker” because config.scm-builder

[zimoun]

Hi,

On Tue, 9 Feb 2021 at 09:35, Ludovic Courtès <ludo@gnu.org> wrote:

> Pushed as 18a4882e3029a084d2f0c63d9d0148682a854546, thank you!

Thanks!  Your comment is better that the none of mime. ;-)

I have not tried yours but I confirm with mine which is the same, the
issue is fixed. :-)

Cheers,
simon