* bug#49801: Guix time machine provenance/manifest reproducibility issue?
@ 2021-08-01 0:21 Denis 'GNUtoo' Carikli
2021-08-17 12:11 ` zimoun
0 siblings, 1 reply; 6+ messages in thread
From: Denis 'GNUtoo' Carikli @ 2021-08-01 0:21 UTC (permalink / raw)
To: 49801
[-- Attachment #1: Type: text/plain, Size: 8512 bytes --]
Hi,
I've been trying to reproduce a tarball
(sz1lkq3ryr5iv6amy6f3d2pziks27g28-tarball-pack.tar.xz) that I generated
with guix pack on guix master the 28 January 2021.
To build it, in January, I used the following commands:
> guix pull
> guix pack \
> --compression=xz \
> --save-provenance \
> -RR \
> --symlink=/usr/local/bin/repo=bin/repo \
> --symlink=/usr/local/bin/repo-env.sh=etc/profile \
> git-repo le-certs nss-certs git python-certifi
That tarball is publicly available in the Replicant ftp server[1].
The extracted provenance file (named manifest) has the following
content:
> ;; This file was automatically generated and is for internal use only.
> ;; It cannot be passed to the '--manifest' option.
>
> (manifest
> (version 3)
> (packages
> (("git-repo"
> "2.4.1"
> "out"
> "/gnu/store/d4frkcdq15a7gyfjdggwg44ryi46fa2d-git-repo-2.4.1R"
> (propagated-inputs ())
> (search-paths ())
> (properties
> (provenance
> (repository
> (version 0)
> (url "https://git.savannah.gnu.org/git/guix.git")
> (branch "master")
> (commit
> "f9bd4621dd92a9415276706b476b9bd2973411fa")
> (introduction
> (channel-introduction
> (version 0)
> (commit
> "9edb3f66fd807b096b48283debdcddccfea34bad")
> (signer
> "BBB0 2DDF 2CEA F6A8 0D1D E643 A2A0 6DF2 A33A
> 54FA"))))))) ("le-certs"
> "0"
> "out"
> "/gnu/store/x004p4hnyy0ickg2f5msvrpszhy9hzpl-le-certs-0R"
> (propagated-inputs ())
> (search-paths ())
> (properties
> (provenance
> (repository
> (version 0)
> (url "https://git.savannah.gnu.org/git/guix.git")
> (branch "master")
> (commit
> "f9bd4621dd92a9415276706b476b9bd2973411fa")
> (introduction
> (channel-introduction
> (version 0)
> (commit
> "9edb3f66fd807b096b48283debdcddccfea34bad")
> (signer
> "BBB0 2DDF 2CEA F6A8 0D1D E643 A2A0 6DF2 A33A
> 54FA"))))))) ("nss-certs"
> "3.57"
> "out"
> "/gnu/store/shc8qpw1y2k7q668rx4gl6aff0wp1n6v-nss-certs-3.57R"
> (propagated-inputs ())
> (search-paths ())
> (properties
> (provenance
> (repository
> (version 0)
> (url "https://git.savannah.gnu.org/git/guix.git")
> (branch "master")
> (commit
> "f9bd4621dd92a9415276706b476b9bd2973411fa")
> (introduction
> (channel-introduction
> (version 0)
> (commit
> "9edb3f66fd807b096b48283debdcddccfea34bad")
> (signer
> "BBB0 2DDF 2CEA F6A8 0D1D E643 A2A0 6DF2 A33A
> 54FA"))))))) ("git"
> "2.30.0"
> "out"
> "/gnu/store/378nlw54nxy991jcilnnbrxasnfvv9wl-git-2.30.0R"
> (propagated-inputs ())
> (search-paths
> (("GIT_SSL_CAINFO"
> ("etc/ssl/certs/ca-certificates.crt")
> #f
> regular
> #f)
> ("GIT_EXEC_PATH"
> ("libexec/git-core")
> #f
> directory
> #f)))
> (properties
> (provenance
> (repository
> (version 0)
> (url "https://git.savannah.gnu.org/git/guix.git")
> (branch "master")
> (commit
> "f9bd4621dd92a9415276706b476b9bd2973411fa")
> (introduction
> (channel-introduction
> (version 0)
> (commit
> "9edb3f66fd807b096b48283debdcddccfea34bad")
> (signer
> "BBB0 2DDF 2CEA F6A8 0D1D E643 A2A0 6DF2 A33A
> 54FA"))))))) ("python-certifi"
> "2020.11.8"
> "out"
> "/gnu/store/hmp6ab9kw1z3hjns9h1fm3afsq4g6j7x-python-certifi-2020.11.8R"
> (propagated-inputs ())
> (search-paths ())
> (properties
> (provenance
> (repository
> (version 0)
> (url "https://git.savannah.gnu.org/git/guix.git")
> (branch "master")
> (commit
> "f9bd4621dd92a9415276706b476b9bd2973411fa")
> (introduction
> (channel-introduction
> (version 0)
> (commit
> "9edb3f66fd807b096b48283debdcddccfea34bad")
> (signer
> "BBB0 2DDF 2CEA F6A8 0D1D E643 A2A0 6DF2 A33A
> 54FA"))))))))))
So I tried to reproduce it with the following command:
> guix time-machine \
> --commit=f9bd4621dd92a9415276706b476b9bd2973411fa -- \
> pack \
> --compression=xz \
> --save-provenance \
> -RR \
> --symlink=/usr/local/bin/repo=bin/repo \
> --symlink=/usr/local/bin/repo-env.sh=etc/profile \
> git-repo le-certs nss-certs git python-certifi
But the new tarball filename was different.
vivien in #guix helped me a lot by trying to build that tarball too and
me and viven have the same filename with guix-time-machine:
bfxvk59q0m034iyq5zkk841zkisayyjl-tarball-pack.tar.xz
We then managed to get to the root cause of the difference.
All the binaries were the sames. All the differences comes from the
fact that the provenance file (named 'manifest') is different.
That difference then produces a different profile name and also affects
/usr/bin as that references the profile.
Diffing the two provenance files gives that:
> +++
> bfxvk59q0m034iyq5zkk841zkisayyjl-tarball-pack/gnu/store/216jiimdyw7zyx8s9b3fz67aw69ydkvw-profile/manifest
> 1970-01-01 01:00:01.000000000 +0100 @@ -15,9 +15,10 @@ (repository
> (version 0)
> (url "https://git.savannah.gnu.org/git/guix.git")
> - (branch "master")
> + (branch #f)
> (commit
> "f9bd4621dd92a9415276706b476b9bd2973411fa")
> + (name guix)
> (introduction
> (channel-introduction
> (version 0)
> @@ -36,9 +37,10 @@
> (repository
> (version 0)
> (url "https://git.savannah.gnu.org/git/guix.git")
> - (branch "master")
> + (branch #f)
> (commit
> "f9bd4621dd92a9415276706b476b9bd2973411fa")
> + (name guix)
> (introduction
> (channel-introduction
> (version 0)
> @@ -57,9 +59,10 @@
> (repository
> (version 0)
> (url "https://git.savannah.gnu.org/git/guix.git")
> - (branch "master")
> + (branch #f)
> (commit
> "f9bd4621dd92a9415276706b476b9bd2973411fa")
> + (name guix)
> (introduction
> (channel-introduction
> (version 0)
> @@ -88,9 +91,10 @@
> (repository
> (version 0)
> (url "https://git.savannah.gnu.org/git/guix.git")
> - (branch "master")
> + (branch #f)
> (commit
> "f9bd4621dd92a9415276706b476b9bd2973411fa")
> + (name guix)
> (introduction
> (channel-introduction
> (version 0)
> @@ -109,9 +113,10 @@
> (repository
> (version 0)
> (url "https://git.savannah.gnu.org/git/guix.git")
> - (branch "master")
> + (branch #f)
> (commit
> "f9bd4621dd92a9415276706b476b9bd2973411fa")
> + (name guix)
> (introduction
> (channel-introduction
I've tried to add --branch=master to guix time-machine and used guix
gc -D to remove the older tarball as it didn't rebuild it even with
--rounds=2, and at the end I still got the exact same
bfxvk59q0m034iyq5zkk841zkisayyjl-tarball-pack.tar.xz tarball (I've
compared both with cmp).
Am I doing something wrong, or is there an issue that needs to be fixed
somehow?
References:
-----------
[1]https://ftp.osuosl.org/pub/replicant/build-tools/repo/28-01-2021/
Denis.
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread* bug#49801: Guix time machine provenance/manifest reproducibility issue?
2021-08-01 0:21 bug#49801: Guix time machine provenance/manifest reproducibility issue? Denis 'GNUtoo' Carikli
@ 2021-08-17 12:11 ` zimoun
2021-09-01 22:27 ` Denis 'GNUtoo' Carikli
0 siblings, 1 reply; 6+ messages in thread
From: zimoun @ 2021-08-17 12:11 UTC (permalink / raw)
To: Denis 'GNUtoo' Carikli, 49801
Hi,
Thanks for the report.
On Sun, 01 Aug 2021 at 02:21, Denis 'GNUtoo' Carikli <GNUtoo@cyberdimension.org> wrote:
> Diffing the two provenance files gives that:
>> +++
>> bfxvk59q0m034iyq5zkk841zkisayyjl-tarball-pack/gnu/store/216jiimdyw7zyx8s9b3fz67aw69ydkvw-profile/manifest
>> 1970-01-01 01:00:01.000000000 +0100 @@ -15,9 +15,10 @@ (repository
>> (version 0)
>> (url "https://git.savannah.gnu.org/git/guix.git")
>> - (branch "master")
>> + (branch #f)
>> (commit
>> "f9bd4621dd92a9415276706b476b9bd2973411fa")
>> + (name guix)
>> (introduction
>> (channel-introduction
>> (version 0)
Well, I think it comes from ’channel-list’ in the ’time-machine’.
Specifically, it reads in guix/scripts/pull.scm:
--8<---------------cut here---------------start------------->8---
(channel (inherit guix)
(url url) (commit commit) (branch #f)))
--8<---------------cut here---------------end--------------->8---
other said, the name of the branch is “lost”. Hum, I do not know if
this is done on purpose or not. Maybe this change
--8<---------------cut here---------------start------------->8---
(cons (match ref
(('commit . commit)
(channel (inherit guix)
(url url) (commit commit))
(('branch . branch)
(channel (inherit guix)
(url url) (commit #f) (branch branch)))
(#f
(channel (inherit guix) (url url))))
(remove guix-channel? channels))
--8<---------------cut here---------------end--------------->8---
is enough. But, I do not know what would happens for:
guix pull --commit=<hash>
where <hash> is not a commit from the branch master.
All the best,
simon
^ permalink raw reply [flat|nested] 6+ messages in thread* bug#49801: Guix time machine provenance/manifest reproducibility issue?
2021-08-17 12:11 ` zimoun
@ 2021-09-01 22:27 ` Denis 'GNUtoo' Carikli
2021-09-02 8:10 ` zimoun
0 siblings, 1 reply; 6+ messages in thread
From: Denis 'GNUtoo' Carikli @ 2021-09-01 22:27 UTC (permalink / raw)
To: zimoun; +Cc: 49801
[-- Attachment #1: Type: text/plain, Size: 2941 bytes --]
Hi again.
With and without this patch:
> diff --git a/guix/scripts/pull.scm b/guix/scripts/pull.scm
> index fb8ce50fa7..af1cf77f07 100644
> --- a/guix/scripts/pull.scm
> +++ b/guix/scripts/pull.scm
> @@ -739,7 +739,7 @@ Use '~/.config/guix/channels.scm' instead."))
> (cons (match ref
> (('commit . commit)
> (channel (inherit guix)
> - (url url) (commit commit) (branch
> #f)))
> + (url url) (commit commit)))
> (('branch . branch)
> (channel (inherit guix)
> (url url) (commit #f) (branch
> branch)))
on top of 95c29d2746943733cbe8df7013854d45bb0df413 ("gnu: electron-cash:
Update to 4.2.5." which is today's master HEAD), I get the same diff
with and without time-machine.
I made and used this Makefile to build two hello tarball in both cases:
> COMMIT ?= 95c29d2746943733cbe8df7013854d45bb0df413
>
> all: \
> hello-guix-$(COMMIT).tar.xz \
> hello-time-machine-$(COMMIT).tar.xz \
>
> hello-guix-$(COMMIT).tar.xz:
> install -m 644 \
> `../pre-inst-env \
> guix pack \
> --compression=xz --save-provenance hello` \
> $@
>
> hello-time-machine-$(COMMIT).tar.xz:
> install -m 644 \
> `../pre-inst-env guix time-machine \
> --branch=master \
> --commit=$(COMMIT) \
> -- \
> pack --compression=xz --save-provenance hello` \
> $@
And once the file named manifest is extracted from both tarballs I get
this diff (with and without your slightly modified patch):
> --- ./hello-guix-95c29d2746943733cbe8df7013854d45bb0df413/gnu/store/lw9x5aimyqcq5iazj786fv7q5l3h0syk-profile/manifest 1970-01-01 01:00:01.000000000 +0100
> +++ ./hello-time-machine-95c29d2746943733cbe8df7013854d45bb0df413/gnu/store/30pf6ppiqpjsjaaiw35kc5lp6dcixpf1-profile/manifest 1970-01-01 01:00:01.000000000 +0100
> @@ -12,4 +12,19 @@
> "/gnu/store/a462kby1q51ndvxdv3b6p0rsixxrgx1h-hello-2.10"
> (propagated-inputs ())
> (search-paths ())
> - (properties)))))
> + (properties
> + (provenance
> + (repository
> + (version 0)
> + (url "https://git.savannah.gnu.org/git/guix.git")
> + (branch #f)
> + (commit
> + "95c29d2746943733cbe8df7013854d45bb0df413")
> + (name guix)
> + (introduction
> + (channel-introduction
> + (version 0)
> + (commit
> + "9edb3f66fd807b096b48283debdcddccfea34bad")
> + (signer
> + "BBB0 2DDF 2CEA F6A8 0D1D E643 A2A0 6DF2 A33A 54FA"))))))))))
PS: In the diff at the top there is a slight difference with the patch
that you suggested: I only removed (branch #f) so I end up with one
more parenthesis at the end.
Denis.
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread* bug#49801: Guix time machine provenance/manifest reproducibility issue?
2021-09-01 22:27 ` Denis 'GNUtoo' Carikli
@ 2021-09-02 8:10 ` zimoun
2021-09-02 14:12 ` Denis 'GNUtoo' Carikli
0 siblings, 1 reply; 6+ messages in thread
From: zimoun @ 2021-09-02 8:10 UTC (permalink / raw)
To: Denis 'GNUtoo' Carikli; +Cc: 49801
Hi Denis,
Thanks for the investigation and the attempt.
Well, I miss if it works or not...
On Thu, 2 Sept 2021 at 00:27, Denis 'GNUtoo' Carikli
<GNUtoo@cyberdimension.org> wrote:
> With and without this patch:
> > diff --git a/guix/scripts/pull.scm b/guix/scripts/pull.scm
> > index fb8ce50fa7..af1cf77f07 100644
> > --- a/guix/scripts/pull.scm
> > +++ b/guix/scripts/pull.scm
> > @@ -739,7 +739,7 @@ Use '~/.config/guix/channels.scm' instead."))
> > (cons (match ref
> > (('commit . commit)
> > (channel (inherit guix)
> > - (url url) (commit commit) (branch
> > #f)))
> > + (url url) (commit commit)))
> > (('branch . branch)
> > (channel (inherit guix)
> > (url url) (commit #f) (branch
> > branch)))
>
> on top of 95c29d2746943733cbe8df7013854d45bb0df413 ("gnu: electron-cash:
> Update to 4.2.5." which is today's master HEAD), I get the same diff
> with and without time-machine.
...here I understand the patch fixes the issue...
> I made and used this Makefile to build two hello tarball in both cases:
> > COMMIT ?= 95c29d2746943733cbe8df7013854d45bb0df413
> >
> > all: \
> > hello-guix-$(COMMIT).tar.xz \
> > hello-time-machine-$(COMMIT).tar.xz \
> >
> > hello-guix-$(COMMIT).tar.xz:
> > install -m 644 \
> > `../pre-inst-env \
> > guix pack \
> > --compression=xz --save-provenance hello` \
> > $@
> >
> > hello-time-machine-$(COMMIT).tar.xz:
> > install -m 644 \
> > `../pre-inst-env guix time-machine \
> > --branch=master \
> > --commit=$(COMMIT) \
> > -- \
> > pack --compression=xz --save-provenance hello` \
> > $@
>
> And once the file named manifest is extracted from both tarballs I get
> this diff (with and without your slightly modified patch):
> > --- ./hello-guix-95c29d2746943733cbe8df7013854d45bb0df413/gnu/store/lw9x5aimyqcq5iazj786fv7q5l3h0syk-profile/manifest 1970-01-01 01:00:01.000000000 +0100
> > +++ ./hello-time-machine-95c29d2746943733cbe8df7013854d45bb0df413/gnu/store/30pf6ppiqpjsjaaiw35kc5lp6dcixpf1-profile/manifest 1970-01-01 01:00:01.000000000 +0100
> > @@ -12,4 +12,19 @@
> > "/gnu/store/a462kby1q51ndvxdv3b6p0rsixxrgx1h-hello-2.10"
> > (propagated-inputs ())
> > (search-paths ())
> > - (properties)))))
> > + (properties
> > + (provenance
> > + (repository
> > + (version 0)
> > + (url "https://git.savannah.gnu.org/git/guix.git")
> > + (branch #f)
> > + (commit
> > + "95c29d2746943733cbe8df7013854d45bb0df413")
> > + (name guix)
> > + (introduction
> > + (channel-introduction
> > + (version 0)
> > + (commit
> > + "9edb3f66fd807b096b48283debdcddccfea34bad")
> > + (signer
> > + "BBB0 2DDF 2CEA F6A8 0D1D E643 A2A0 6DF2 A33A 54FA"))))))))))
...but then here I see it does not fix it.
However, because you run "./pre-inst-env guix pack --save-provenance",
it seems expected that the 'properties' is empty. From my
understanding, '(find guix-channels? channels)' does not return the
'guix' channel because it is the current Git checkout. It is not the
case with "guix time-machine" because it creates an inferior using the
'guix' channel.
Moreover, if you want to try the patch, you need to run:
./pre-inst-env guix pull -p /tmp/new
./tmp/new/bin/guix describe # return commit 12345
./tmp/new/bin/guix pack --save-provenance
./tmp/new/bin/guix time-machine --commit=12345 -- pack --save-provenance
and be careful with the '--localstatedir' and '--sysconfdir' variables
at './configure' time.
Well, from my point of view, the Guix way would be:
guix describe -f channels > channels.scm
guix pack --save-provenance
then later or elsewehere
guix time-machine -C channels.scm -- pack --save-provenance
Although, it will not fix the bug you are exposing. :-)
WDYT?
Last, I have not carefully checked and maybe I am wrong, the both
options "--commit=1234 --branch=master" are exclusive I guess; i.e.,
the argument 'master' passed to '--branch' is not used in this case,
IIUC.
Cheers,
simon
^ permalink raw reply [flat|nested] 6+ messages in thread* bug#49801: Guix time machine provenance/manifest reproducibility issue?
2021-09-02 8:10 ` zimoun
@ 2021-09-02 14:12 ` Denis 'GNUtoo' Carikli
2021-09-02 19:30 ` zimoun
0 siblings, 1 reply; 6+ messages in thread
From: Denis 'GNUtoo' Carikli @ 2021-09-02 14:12 UTC (permalink / raw)
To: zimoun; +Cc: 49801
[-- Attachment #1: Type: text/plain, Size: 838 bytes --]
On Thu, 2 Sep 2021 10:10:22 +0200
zimoun <zimon.toutoune@gmail.com> wrote:
> Hi Denis,
>
> Thanks for the investigation and the attempt.
>
> Well, I miss if it works or not...
It doesn't work.
The issue was that if you build a tarball with guix pack, without guix
time-machine, you can't reproduce it with guix-time-machine.
Between the two tarballs, everything is the same but the provenance
file.
So here the idea is to make sure that the provenance file is the same
between tarballs made with and without guix time-machine.
Here I get a diff between tarballs made with and without guix
time-machine, with or without your patch, so the patch doesn't fix it
yet for guix master of yesterday.
Between when I reported the bug and the test I did yesterday, the HEAD
of guix master changed though.
Denis.
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* bug#49801: Guix time machine provenance/manifest reproducibility issue?
2021-09-02 14:12 ` Denis 'GNUtoo' Carikli
@ 2021-09-02 19:30 ` zimoun
0 siblings, 0 replies; 6+ messages in thread
From: zimoun @ 2021-09-02 19:30 UTC (permalink / raw)
To: Denis 'GNUtoo' Carikli; +Cc: 49801
Hi Denis,
On Thu, 02 Sep 2021 at 16:12, Denis 'GNUtoo' Carikli <GNUtoo@cyberdimension.org> wrote:
> Between the two tarballs, everything is the same but the provenance
> file.
>
> So here the idea is to make sure that the provenance file is the same
> between tarballs made with and without guix time-machine.
Yes, for sure. IMHO, the Guix way would be:
guix describe -f channels > channels.scm
guix pack --save-provenance
then later or elsewehere
guix time-machine -C channels.scm -- pack --save-provenance
It is a workaround of the bug your reported. ;-)
Does it work?
> Here I get a diff between tarballs made with and without guix
> time-machine, with or without your patch, so the patch doesn't fix it
> yet for guix master of yesterday.
I think your tests about the patch are not correct. As I wrote, this is
what you should try, IIUC:
--8<---------------cut here---------------start------------->8---
Moreover, if you want to try the patch, you need to run:
./pre-inst-env guix pull -p /tmp/new
./tmp/new/bin/guix describe # return commit 12345
./tmp/new/bin/guix pack --save-provenance
./tmp/new/bin/guix time-machine --commit=12345 -- pack --save-provenance
and be careful with the '--localstatedir' and '--sysconfdir' variables
at './configure' time.
--8<---------------cut here---------------end--------------->8---
All the best,
simon
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2021-09-02 19:45 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-08-01 0:21 bug#49801: Guix time machine provenance/manifest reproducibility issue? Denis 'GNUtoo' Carikli
2021-08-17 12:11 ` zimoun
2021-09-01 22:27 ` Denis 'GNUtoo' Carikli
2021-09-02 8:10 ` zimoun
2021-09-02 14:12 ` Denis 'GNUtoo' Carikli
2021-09-02 19:30 ` zimoun
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).