bug#41669: Cross-compiled powerpc64-linux bootstrap-tarballs not reproducible

[Chris Marusich <cmmarusich@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 4040 bytes --]

Hi all,

Chris Marusich <cmmarusich@gmail.com> writes:

> The derivation that produced the differing output was:
>
> /gnu/store/pygln3lr6qbxcps3kmn3w4bc0d0nlpd3-gcc-stripped-tarball-5.5.0.drv
>
> On my x86_64-linux system, twice I tried running "guix build --check" on
> this derivation, but each time it surprisingly reported no difference.

This derivation corresponds to %gcc-bootstrap-tarball from (gnu packages
make-bootstrap), which just creates a tarball of the output of
%gcc-stripped.  Therefore, it's not too surprising that it's
reproducible.  Similarly, %gcc-stripped just strips some store
references from the output of %gcc-static.  The %gcc-static package is
more interesting: it's where we actually build the statically linked
bootstrap GCC (which is then stripped and packed into a tarball as
mentioned above), so I thought that it might be where the
non-determinism is coming from.  However, I haven't yet pinpointed the
problem.

If you examine the derivations and their inputs, you'll find that they
depend upon each other in the following order:

guix build --target=powerpc64-linux-gnu -d -e '(@ (gnu packages make-bootstrap) %gcc-bootstrap-tarball)'
/gnu/store/pygln3lr6qbxcps3kmn3w4bc0d0nlpd3-gcc-stripped-tarball-5.5.0.drv

guix build --target=powerpc64-linux-gnu -d -e '(@@ (gnu packages make-bootstrap) %gcc-stripped)'
/gnu/store/kcv3ja1rfr93hw6ly51878zjhdwpgv7z-gcc-stripped-5.5.0.drv

guix build --target=powerpc64-linux-gnu -d -e '(@@ (gnu packages make-bootstrap) %gcc-static)'
/gnu/store/m9hfwppla8lph0vxa15lfkp81s2bbjjs-gcc-static-5.5.0.drv

In other words, gcc-static-5.5.0.drv is an input of
gcc-stripped-5.5.0.drv, which is an input of
gcc-stripped-tarball-5.5.0.drv.  Above, I've included example guix
commands you can use to obtain each derivation.  Using "guix build
--check", I confirmed that all three of these derivations build
reproducibly on my machine.

I hoped to find more information by invoking "guix build --check" on
every input of gcc-static-5.5.0.drv.  When I tried that, what I found
was that all of its inputs build reproducibly except the following two:

/gnu/store/x32cnfkd50fnxs10xp1jdn24h7ai2gxr-guile-3.0.2.drv
/gnu/store/g9fpkg2qa27mka1znqsvx8vxqyabsj2y-gcc-7.5.0.drv

I haven't investigated guile-3.0.2.drv.  However, gcc-7.5.0.drv felt
more suspicious to me, and it is actually the derivation that builds
gcc-final from (gnu packages commencement), which you can see via:

guix build -d -e '(@@ (gnu packages commencement) gcc-final)'

Using "guix gc", I deleted the outputs of
gcc-stripped-tarball-5.5.0.drv, gcc-stripped-5.5.0.drv,
gcc-static-5.5.0.drv, and gcc-7.5.0.drv.  I then tried building these
four derivations again (without substitutes, the same as before).
Before doing this, I stored the SHA-512 hashes of their output files,
and after the build succeeded, I compared the hashes of the new files
with the previous values.  I found that these derivations' newly rebuilt
outputs were identical to their original values, except for
gcc-7.5.0.drv, which produced some different files.  Most significantly,
this means that gcc-stripped-tarball-5.5.0.drv produced the exact same
tarball for me as it did the first time I built it, even though some of
its inputs are not themselves reproducible.

At present, it seems possible that within the context of a single
machine, gcc-stripped-tarball-5.5.0.drv builds reproducibly, but on a
different machine, it may (reproducibly) build a different output.  I'm
a bit paranoid about making mistakes, so I'll perform another full GC
and then try yet again to build gcc-stripped-tarball-5.5.0.drv in order
to verify whether it truly produces the same output when all (or nearly
all) of its inputs are rebuilt from scratch.

Some people have also shared their differing copies of the binaries on
the email list.  It could be productive to compare the contents with
diffoscope, although I suspect the diff might be too large to be useful.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]