Reproducibility of "core" packages in GNU Guix

Reproducibility of "core" packages in GNU Guix

[Vagrant Cascadian]

[-- Attachment #1: Type: text/plain, Size: 5235 bytes --]

On 2022-04-27, Vagrant Cascadian wrote:
> Lately, I've been trying to get a handle on the status of the really
> core packages in Debian
...
> I'd also be really curious to hear about the status of similar package
> sets in other distros!

With my metaphorical guix hoodie[1] on...

$ guix describe

Generation 73   May 02 2022 05:21:25    (current)
  guix 9dafaf1
    repository URL: /home/vagrant/src/guix
    branch: master
    commit: 9dafaf163574edca5cb4eac0f8dc3edbb0ef0a75

$ guix challenge --diff=none $(cat guix-base-set)

/gnu/store/8gmqvwf0ccqfyimficcnhxvrykwx6y8g-linux-libre-5.17.5 contents differ:
  no local build for '/gnu/store/8gmqvwf0ccqfyimficcnhxvrykwx6y8g-linux-libre-5.17.5'
  https://ci.guix.gnu.org/nar/zstd/8gmqvwf0ccqfyimficcnhxvrykwx6y8g-linux-libre-5.17.5: 19rg55v51wliy9v30sm82f38rxm1lqjpfqs6r63ikb3vklnj0pnw
  https://bordeaux.guix.gnu.org/nar/lzip/8gmqvwf0ccqfyimficcnhxvrykwx6y8g-linux-libre-5.17.5: 14fax6g9sx7qj64z73hrh8ydlbv6kxzhd1hbyqz7v0ra51bprv1k
/gnu/store/7qz2jlghm4gc87jww5j24c5mcip0whzy-keyutils-1.6.3 contents differ:
  no local build for '/gnu/store/7qz2jlghm4gc87jww5j24c5mcip0whzy-keyutils-1.6.3'
  https://ci.guix.gnu.org/nar/lzip/7qz2jlghm4gc87jww5j24c5mcip0whzy-keyutils-1.6.3: 1sag2bq9kbp5np3fpakyi4xg96kxq5xwbb7ib4hamx2bqh6vscr9
  https://bordeaux.guix.gnu.org/nar/lzip/7qz2jlghm4gc87jww5j24c5mcip0whzy-keyutils-1.6.3: 07ln4fqgvg0ag2d881xhgdw2h3m1lqzs6xlac8p7rz2rgx0wx1yr
/gnu/store/ajw8nnrnd6hr183skwqdgc8c7mazg97h-isl-0.23 contents differ:
  no local build for '/gnu/store/ajw8nnrnd6hr183skwqdgc8c7mazg97h-isl-0.23'
  https://ci.guix.gnu.org/nar/lzip/ajw8nnrnd6hr183skwqdgc8c7mazg97h-isl-0.23: 03a180af1my7lmsnig01qhrirxa2fp7j052jw9kv5ff4i6ya7fh4
  https://bordeaux.guix.gnu.org/nar/lzip/ajw8nnrnd6hr183skwqdgc8c7mazg97h-isl-0.23: 1j24gc6ysa9d3z4hq6lsxvdik94ddb7nj93krv7cs5lmbmjwmqw7
/gnu/store/45b6181w68a3lprx9m6riwgyinw3y145-guix-1.3.0-25.c1719a0 contents differ:
  no local build for '/gnu/store/45b6181w68a3lprx9m6riwgyinw3y145-guix-1.3.0-25.c1719a0'
  https://ci.guix.gnu.org/nar/lzip/45b6181w68a3lprx9m6riwgyinw3y145-guix-1.3.0-25.c1719a0: 0p7lhfxcx7bfjfwlyrp6h5j9fcyzswyj2wkbnhcd3fgxm5swdi6c
  https://bordeaux.guix.gnu.org/nar/lzip/45b6181w68a3lprx9m6riwgyinw3y145-guix-1.3.0-25.c1719a0: 0yfpcsmvbnzw0vpjrjwwrjih4ss3yvk7cy4k6ibdpsn7dcx9kw2c
/gnu/store/1jgcbdzx2ss6xv59w55g3kr3x4935dfb-guile-3.0.8 contents differ:
  no local build for '/gnu/store/1jgcbdzx2ss6xv59w55g3kr3x4935dfb-guile-3.0.8'
  https://ci.guix.gnu.org/nar/lzip/1jgcbdzx2ss6xv59w55g3kr3x4935dfb-guile-3.0.8: 0vppx6fk1a7gvk9ccz9ma992w1h5bhfk535acddrnkhyrk92z5ln
  https://bordeaux.guix.gnu.org/nar/lzip/1jgcbdzx2ss6xv59w55g3kr3x4935dfb-guile-3.0.8: 05w5i5zq1k1avqx2gqxnqynn5lmdizis9babk34dkmnazb3h77kb

47 store items were analyzed:
  - 42 (89.4%) were identical
  - 5 (10.6%) differed
  - 0 (0.0%) were inconclusive


I love that Guix really has batteries included when it comes to
reproducible builds verification! :)

At first, I thought I would have to build all this stuff locally, but
then I realized guix actually has two independent build farms, so guix
challenge can compare the results between them! For more data points,
one could build them all locally!


The fact that the guix and guile packages do not build reproducibly is a
little disappointing as they're both so central to guix itself; I
suspect parallelism triggers those reproducibility issues(from
experience with Debian), though that may just reveal other issue in
guile itself.


The linux-libre package *ought* to be reproducible; I hope it is
something easy to fix there...

$ guix challenge --diff=diffoscope linux-libre

/gnu/store/8gmqvwf0ccqfyimficcnhxvrykwx6y8g-linux-libre-5.17.5 contents differ:
  no local build for '/gnu/store/8gmqvwf0ccqfyimficcnhxvrykwx6y8g-linux-libre-5.17.5'
  https://ci.guix.gnu.org/nar/zstd/8gmqvwf0ccqfyimficcnhxvrykwx6y8g-linux-libre-5.17.5: 19rg55v51wliy9v30sm82f38rxm1lqjpfqs6r63ikb3vklnj0pnw
  https://bordeaux.guix.gnu.org/nar/lzip/8gmqvwf0ccqfyimficcnhxvrykwx6y8g-linux-libre-5.17.5: 14fax6g9sx7qj64z73hrh8ydlbv6kxzhd1hbyqz7v0ra51bprv1k
 ...
 0%  ETA:  4 days, 2:03:47

Ok... well, I guess I won't wait for the results...


A better "core" package set for GNU Guix could surely be created. I came
up with this list of packages by taking the essential, required and
build-essential package sets from Debian, tweaking the package names
appropriately, dropping debian-specific stuff, and adding guile and
guix to create "guix-base-set":

acl
attr
audit
bash
binutils
bzip2
coreutils
diffutils
e2fsprogs
elogind
findutils
gawk
gcc
glibc
gmp
grep
guile
guix
gzip
isl
keyutils
libcap
libcap-ng
libnsl
libselinux
libsigsegv
libtirpc
libxcrypt
linux-pam
linux-libre
mpfr
ncurses
openssl
patch
pcre
pcre2
perl
readline
rpcsvc-proto
sed
shadow
tar
tzdata
util-linux
xz
zlib
zstd


> I would also like to see if there is anything in Debian or other
> distros that still needs to be pushed upstream, so we can all benefit!

Will dig into some of these issues and see how Debian and Guix are
building them to see if there are any patches to share and push
upstream.


[1] Actually wearing my Aspiration Tech hoodie at the moment, but the
    Guix hoodie is around here somewhere...

live well,
  vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Reproducibility of "core" packages in GNU Guix

[zimoun]

Hi Vagrant,

Cool to see these reports.


On Mon, 02 May 2022 at 06:11, Vagrant Cascadian <vagrant@reproducible-builds.org> wrote:

> $ guix challenge --diff=none $(cat guix-base-set)
>
> /gnu/store/8gmqvwf0ccqfyimficcnhxvrykwx6y8g-linux-libre-5.17.5 contents differ:
>   no local build for '/gnu/store/8gmqvwf0ccqfyimficcnhxvrykwx6y8g-linux-libre-5.17.5'
>   https://ci.guix.gnu.org/nar/zstd/8gmqvwf0ccqfyimficcnhxvrykwx6y8g-linux-libre-5.17.5: 19rg55v51wliy9v30sm82f38rxm1lqjpfqs6r63ikb3vklnj0pnw
>   https://bordeaux.guix.gnu.org/nar/lzip/8gmqvwf0ccqfyimficcnhxvrykwx6y8g-linux-libre-5.17.5: 14fax6g9sx7qj64z73hrh8ydlbv6kxzhd1hbyqz7v0ra51bprv1k
> /gnu/store/7qz2jlghm4gc87jww5j24c5mcip0whzy-keyutils-1.6.3 contents differ:
>   no local build for '/gnu/store/7qz2jlghm4gc87jww5j24c5mcip0whzy-keyutils-1.6.3'
>   https://ci.guix.gnu.org/nar/lzip/7qz2jlghm4gc87jww5j24c5mcip0whzy-keyutils-1.6.3: 1sag2bq9kbp5np3fpakyi4xg96kxq5xwbb7ib4hamx2bqh6vscr9
>   https://bordeaux.guix.gnu.org/nar/lzip/7qz2jlghm4gc87jww5j24c5mcip0whzy-keyutils-1.6.3: 07ln4fqgvg0ag2d881xhgdw2h3m1lqzs6xlac8p7rz2rgx0wx1yr
> /gnu/store/ajw8nnrnd6hr183skwqdgc8c7mazg97h-isl-0.23 contents differ:
>   no local build for '/gnu/store/ajw8nnrnd6hr183skwqdgc8c7mazg97h-isl-0.23'
>   https://ci.guix.gnu.org/nar/lzip/ajw8nnrnd6hr183skwqdgc8c7mazg97h-isl-0.23: 03a180af1my7lmsnig01qhrirxa2fp7j052jw9kv5ff4i6ya7fh4
>   https://bordeaux.guix.gnu.org/nar/lzip/ajw8nnrnd6hr183skwqdgc8c7mazg97h-isl-0.23: 1j24gc6ysa9d3z4hq6lsxvdik94ddb7nj93krv7cs5lmbmjwmqw7
> /gnu/store/45b6181w68a3lprx9m6riwgyinw3y145-guix-1.3.0-25.c1719a0 contents differ:
>   no local build for '/gnu/store/45b6181w68a3lprx9m6riwgyinw3y145-guix-1.3.0-25.c1719a0'
>   https://ci.guix.gnu.org/nar/lzip/45b6181w68a3lprx9m6riwgyinw3y145-guix-1.3.0-25.c1719a0: 0p7lhfxcx7bfjfwlyrp6h5j9fcyzswyj2wkbnhcd3fgxm5swdi6c
>   https://bordeaux.guix.gnu.org/nar/lzip/45b6181w68a3lprx9m6riwgyinw3y145-guix-1.3.0-25.c1719a0: 0yfpcsmvbnzw0vpjrjwwrjih4ss3yvk7cy4k6ibdpsn7dcx9kw2c
> /gnu/store/1jgcbdzx2ss6xv59w55g3kr3x4935dfb-guile-3.0.8 contents differ:
>   no local build for '/gnu/store/1jgcbdzx2ss6xv59w55g3kr3x4935dfb-guile-3.0.8'
>   https://ci.guix.gnu.org/nar/lzip/1jgcbdzx2ss6xv59w55g3kr3x4935dfb-guile-3.0.8: 0vppx6fk1a7gvk9ccz9ma992w1h5bhfk535acddrnkhyrk92z5ln
>   https://bordeaux.guix.gnu.org/nar/lzip/1jgcbdzx2ss6xv59w55g3kr3x4935dfb-guile-3.0.8: 05w5i5zq1k1avqx2gqxnqynn5lmdizis9babk34dkmnazb3h77kb
>
> 47 store items were analyzed:
>   - 42 (89.4%) were identical
>   - 5 (10.6%) differed
>   - 0 (0.0%) were inconclusive

[...]

> The fact that the guix and guile packages do not build reproducibly is a
> little disappointing as they're both so central to guix itself; I
> suspect parallelism triggers those reproducibility issues(from
> experience with Debian), though that may just reveal other issue in
> guile itself.

About Guix, probably bug#44835 [1] for one, I guess.  And note this old
Guile bug#20272 [2] for two, which implies unreproducible Guix.

1: <http://issues.guix.gnu.org/issue/44835>
2: <https://issues.guix.gnu.org/issue/20272>


Cheers,
simon

Re: Reproducibility of "core" packages in GNU Guix

[Vagrant Cascadian]

[-- Attachment #1: Type: text/plain, Size: 1047 bytes --]

On 2022-05-02, zimoun wrote:
> On Mon, 02 May 2022 at 06:11, Vagrant Cascadian <vagrant@reproducible-builds.org> wrote:
>> $ guix challenge --diff=none $(cat guix-base-set)
...
>> The fact that the guix and guile packages do not build reproducibly is a
>> little disappointing as they're both so central to guix itself; I
>> suspect parallelism triggers those reproducibility issues(from
>> experience with Debian), though that may just reveal other issue in
>> guile itself.
>
> About Guix, probably bug#44835 [1] for one, I guess.  And note this old
> Guile bug#20272 [2] for two, which implies unreproducible Guix.
>
> 1: <http://issues.guix.gnu.org/issue/44835>

This one is regarding build paths, and since guix normalizes the build
path in the build environment, it should not affect builds in guix...

> 2: <https://issues.guix.gnu.org/issue/20272>

But this one definitely touches on the parallelism issue!

Thanks for the links! I remembered commenting on them... just didn't
find the links with a quick search...



live well,
  vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Reproducibility of "core" packages in GNU Guix

[Vagrant Cascadian]

[-- Attachment #1: Type: text/plain, Size: 1645 bytes --]

On 2022-05-02, Vagrant Cascadian wrote:
> $ guix challenge --diff=none $(cat guix-base-set)
>
> /gnu/store/8gmqvwf0ccqfyimficcnhxvrykwx6y8g-linux-libre-5.17.5 contents differ:

Proving more difficult than I'd hoped for, smallish diffs in the .ko
files and in the bzImage and System.map, but nothing obvious leaping out
at me. The corresponding files are reproducible in Debian bookworm...

Working on this lead me to notice a bug in diffoscope at least:

  https://salsa.debian.org/reproducible-builds/diffoscope/-/issues/305


> /gnu/store/7qz2jlghm4gc87jww5j24c5mcip0whzy-keyutils-1.6.3 contents differ:

Patch:

  https://issues.guix.gnu.org/55758

There was already a patch in debian to set the date using an environment
variable. Might be worth working up a patch to support SOURCE_DATE_EPOCH
and push it upstream... or nudging upstream to drop the timestamp
entirely. :)


> /gnu/store/ajw8nnrnd6hr183skwqdgc8c7mazg97h-isl-0.23 contents differ:

Patch:

  https://issues.guix.gnu.org/55757

Disabling parallel building in guix fixes it for me consistently,
although Debian's "isl" package is reproducible but... builds with
parallelism. (well, not reproducible on i386, but who's really
counting?)

What about other distros? Do you do anything to make "isl" reproducible?


> /gnu/store/45b6181w68a3lprx9m6riwgyinw3y145-guix-1.3.0-25.c1719a0 contents differ:
> /gnu/store/1jgcbdzx2ss6xv59w55g3kr3x4935dfb-guile-3.0.8 contents differ:

Both of these were not *just* due to parallelism as I'd
hoped... inscrutible guile...


So, 2 out of the 5 remaining packages have plausible fixes (out of 47
total)... not too bad!


live well,
  vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]