Unreproducible «When Docker images become fixed-point»?

Unreproducible «When Docker images become fixed-point»?

[zimoun]

Hi,

Preparing a session about Guix, I was checking this blog post:

    https://hpc.guix.info/blog/2021/10/when-docker-images-become-fixed-point/

explaining how to rebuild a Docker pack using a Docker pack produced by
Guix itself.  It is from Oct. 2021.


The pack built by the post is:

/gnu/store/6rga6pz60di21mn37y5v3lvrwxfvzcz9-python-python-numpy-docker-pack.tar.gz

comparing with today, using 9d795fb and the time-machine:

--8<---------------cut here---------------start------------->8---
$ guix time-machine -C /tmp/channels.scm \
       -- pack -f docker --save-provenance -m /tmp/manifest.scm

/gnu/store/ryqvz83s11qb96d9i6ywv08vcshg6k17-python-python-numpy-docker-pack.tar.gz
--8<---------------cut here---------------end--------------->8---

Note that ’docker images’ said ea2d5e62b2d2 when now it says
e0a073dfa1ec. 

The channels.scm and manifest.scm files are the exact same – or the bug
is between my keyboard and my chair. ;-)


Even, following the post, let compare:

--8<---------------cut here---------------start------------->8---
$ tree gnu
gnu
└── store
    └── vdf5c49kzsmdm70134fdgy418ifxd7kh-profile
        └── manifest

2 directories, 1 file
--8<---------------cut here---------------end--------------->8---

and the post provides ia1sxr3qf3w9dj7y48rwvwyx289vpfgi-profile.

--8<---------------cut here---------------start------------->8---
$ guix package -p gnu/store/vdf5c49kzsmdm70134fdgy418ifxd7kh-profile/ --export-channels
;; This channel file can be passed to 'guix pull -C' or to
;; 'guix time-machine -C' to obtain the Guix revision that was
;; used to populate this profile.

(list
     (channel
       (name 'guix)
       (url "https://git.savannah.gnu.org/git/guix.git")
       (branch #f)
       (commit
         "fb32a38db1d3a6d9bc970e14df5be95e59a8ab02")
       (introduction
         (make-channel-introduction
           "9edb3f66fd807b096b48283debdcddccfea34bad"
           (openpgp-fingerprint
             "BBB0 2DDF 2CEA F6A8 0D1D  E643 A2A0 6DF2 A33A 54FA"))))
)

$ guix package -p gnu/store/vdf5c49kzsmdm70134fdgy418ifxd7kh-profile/ --export-manifest
;; This "manifest" file can be passed to 'guix package -m' to reproduce
;; the content of your profile.  This is "symbolic": it only specifies
;; package names.  To reproduce the exact same profile, you also need to
;; capture the channels being used, as returned by "guix describe".
;; See the "Replicating Guix" section in the manual.

(specifications->manifest
  (list "python@3.8" "python-numpy@1.17"))
--8<---------------cut here---------------end--------------->8---


What could be wrong?  The package ’python’ does not build reproductibly
but that should not change the hash of the profile nor the pack, it
should only change the hash of the ’docker images’.

What do I miss?


Cheers,
simon

Re: Unreproducible «When Docker images become fixed-point»?

[Ludovic Courtès]

Hello,

zimoun <zimon.toutoune@gmail.com> skribis:

> The pack built by the post is:
>
> /gnu/store/6rga6pz60di21mn37y5v3lvrwxfvzcz9-python-python-numpy-docker-pack.tar.gz
>
> comparing with today, using 9d795fb and the time-machine:
>
> $ guix time-machine -C /tmp/channels.scm \
>        -- pack -f docker --save-provenance -m /tmp/manifest.scm
>
> /gnu/store/ryqvz83s11qb96d9i6ywv08vcshg6k17-python-python-numpy-docker-pack.tar.gz

I have this:

--8<---------------cut here---------------start------------->8---
$ guix time-machine --commit=fb32a38db1d3a6d9bc970e14df5be95e59a8ab02 -- pack -f docker --save-provenance python python-numpy -d
/gnu/store/7kxwhszfvqxcryfid7n9fk4mamaxrwky-python-python-numpy-docker-pack.tar.gz.drv
$ guix time-machine --commit=fb32a38db1d3a6d9bc970e14df5be95e59a8ab02 -- pack -f docker --save-provenance python python-numpy 
/gnu/store/ryqvz83s11qb96d9i6ywv08vcshg6k17-python-python-numpy-docker-pack.tar.gz
$ guix hash $(guix time-machine --commit=fb32a38db1d3a6d9bc970e14df5be95e59a8ab02 -- pack -f docker --save-provenance python python-numpy )
1zn7kx2nj5ly8kcdl6lw0l8v1428ldg88j5zs7wyjxl27qz1kdrx
$ guix describe
Generation 221  Jul 03 2022 23:52:07    (current)
  guix e069de4
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: e069de452a2c923868f5137421b4b6349c38d754
--8<---------------cut here---------------end--------------->8---

Do you still have the original tarball mentioned in the post?

A possible reason why we’re building a different derivation than back
then is provenance info: as explained under ‘--save-provenance’ in the
manual, provenance info is not “canonical” and we could end up including
different provenance info.  I don’t have any clear scenario in mind but
that sounds plausible.

Thanks,
Ludo’.

Re: Unreproducible «When Docker images become fixed-point»?

[zimoun]

Hi,

On Mon, 04 Jul 2022 at 15:37, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:

> Do you still have the original tarball mentioned in the post?

Sadly no.


> A possible reason why we’re building a different derivation than back
> then is provenance info: as explained under ‘--save-provenance’ in the
> manual, provenance info is not “canonical” and we could end up including
> different provenance info.  I don’t have any clear scenario in mind but
> that sounds plausible.

I do not understand why provenance is not deterministic.  I mean I
understand that two provenances can build the same pack, but I miss why

        guix time-machine -C channels.scm \
             -- pack -f docker --save-provenance -m manifest.scm

is not building the same pack for the exact same channels.scm and
manifest.scm files.  Why the resulting provenance info should be
different?

Maybe I overlook a point.  From my understanding, in this case of “guix
machine -C channels.scm”, the provenance is only determined by the file
channels.scm and the provenance thus is unique.

Maybe there is bug in how the provenance is managed; but I do not think
it comes from this part.  Instead, I vaguely think the bug is from
elsewhere – dependent on filesystem or unsorted list or other creative
ideas. :-)

Sadly, we are lacking information for more investigations.  For
instance, it would have been useful to have the checksum of the pack and
so compare.  Or be able to compare the derivations.

Well, let save some data, replay this scenario 6 months later and
investigate. :-)  Keep you in touch.


Cheers,
simon

Re: Unreproducible «When Docker images become fixed-point»?

[Ludovic Courtès]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

>> A possible reason why we’re building a different derivation than back
>> then is provenance info: as explained under ‘--save-provenance’ in the
>> manual, provenance info is not “canonical” and we could end up including
>> different provenance info.  I don’t have any clear scenario in mind but
>> that sounds plausible.
>
> I do not understand why provenance is not deterministic.  I mean I
> understand that two provenances can build the same pack, but I miss why
>
>         guix time-machine -C channels.scm \
>              -- pack -f docker --save-provenance -m manifest.scm
>
> is not building the same pack for the exact same channels.scm and
> manifest.scm files.  Why the resulting provenance info should be
> different?

I don’t know either, but that’s the only plausible scenario I can think
of.  We would need the original .drv or the original pack to compare.

> Maybe there is bug in how the provenance is managed; but I do not think
> it comes from this part.  Instead, I vaguely think the bug is from
> elsewhere – dependent on filesystem or unsorted list or other creative
> ideas. :-)

No, no: the store file names differ.  That means we’re building
different derivations in the first place.

> Well, let save some data, replay this scenario 6 months later and
> investigate. :-)  Keep you in touch.

Yeah.

Another approach is to take the derivation returned by

  guix time-machine --commit=fb32a38db1d3a6d9bc970e14df5be95e59a8ab02 -- \
    pack -f docker --save-provenance python python-numpy -d

and to look for things that might vary between invocations or call
sites: provenance data, imported modules, (guix config) details,
whatever.

Thinking about it, (guix config) is one possible source of discrepancy:
it captures sysconfdir and localstatedir, so you’ll get a different
result if you have different settings.  The defaults are:

  (define-public %localstatedir "/var")
  (define-public %sysconfdir "/etc")

Could it be that you had something different back then?

Ludo’.

Re: Unreproducible «When Docker images become fixed-point»?

[zimoun]

Hi,

On Tue, 05 Jul 2022 at 09:44, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:

>   (define-public %localstatedir "/var")

This one is for sure the same.

>   (define-public %sysconfdir "/etc")

I do not remember.  However, I am aware of such potential issue since I
sent this patch [1] because I already hit non-reproducible pack. ;-)

So, I guess I did the correct thing using the default.  But I cannot
bet, who knows. :-)



1: <http://issues.guix.gnu.org/issue/47401>


Cheers,
simon