bug#41669: Cross-compiled powerpc64-linux bootstrap-tarballs not reproducible

[Chris Marusich <cmmarusich@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 6011 bytes --]

Hi all,

It's great to see forward motion again!

Efraim Flashner <efraim@flashner.co.il> writes:

> As far as powerpc64 vs powerpc64le, I'll let those with the hardware
> have more of a say, they'll be the ones using it. As far as the
> bootstrap binaries go, I don't remember having this much pushback with
> my binaries for aarch64 (just a request to rebuild with guile-2.0.14
> since it was reproducible), and I'm not sure how much Janneke had with
> the Hurd binaries but I don't think it was this much. The ultimate goal
> anyway is to replace them with artisanally crafted mes binaries, and I
> understand we want to have them as reproducible as possible, but I don't
> think it's fair to keep this architecture out when we've let other ones
> in with similar reproducible problems.

Ludovic Courtès <ludo@gnu.org> writes:

> I didn’t follow the whole discussion nor did I try to investigate
> myself, but thanks a lot for going to great lengths trying to identify
> the issue; this is an impressive amount of work, and I can only share
> your disappointment.
>
> Given this effort, I agree that it may be best at this point to move on
> and start with these non-reproducible binaries.  At least, the problem
> is now documented.

I'm glad you agree.  For powerpc64 (big-endian), should we just make the
changes for bootstrapping on the master branch, or should we use a
separate branch?

If we use a separate branch, we could use the name "wip-ppc64", since
there is already a "wip-ppc64le" for powerpc64le.  Is it expected that
commits on these "wip" branches will never be modified (e.g., via
rebasing)?

Leo Le Bouter <lle-bout@zaclys.net> writes:

>> Do you have a preference big-endian vs little endian?
>
> I'd like both but little endian has the widest eco-system support
> especially w.r.t. to Linux drivers. Many Linux drivers have endianness
> bugs (lack of endian-safe serialization for DMA..), it's such a plague
> that sticking to little endian is just better right now. One common
> example being mpt3sas and amdgpu drivers required in some
> configurations of the Talos II system.

Ludovic Courtès <ludo@gnu.org> writes:

>> At this point, it might even make more sense to try bootstrapping for
>> powerpc64le instead of powerpc64, since the rest of the world seems to
>> be gravitating toward the little-endian variant on POWER9 hardware, and
>> thus various programs out there are more likely to be better tested on
>> powerpc64le than powerpc64.
>
> Yes, my understanding is that other people, in particular Tobias Platen
> and dftxbs3e, were looking at powerpc64le, so perhaps it’s a good idea
> to concentrate on that one?

I agree.  It's probably better to focus on little-endian.  However, it
isn't clear yet which will be ultimately harder for us to bootstrap, so
I also think it's fine to work on both in parallel and see how it goes.

Ludovic Courtès <ludo@gnu.org> writes:

> Anyhow, please let me know if/when bootstrap binaries should be uploaded
> to ftp.gnu.org (with a signed message).  When updating bootstrap.scm to
> refer to them, please include the commit ID used to build them in the
> commit message.

Over the last few days, Leo and I coordinated to try cross-compiling the
powerpc64 bootstrap tarballs one more time, using two Guix System VMs.
We did this because we thought that maybe if we took care to keep the
Guix Systems "the same" (e.g., same kernel), it would produce identical
results.  Instead, the result was the same as before: all bootstrap
tarballs except for gcc-static were identical, and gcc-static differed
in ways similar to what has already been described earlier in this bug
report.  In fact, with the exception of gcc-static, the bootstrap
tarballs were identical to the tarballs multiple people built 6 months
ago in June.  This means that (1) with the exception of gcc-static, the
bootstrap tarballs build reproducibly even across systems and after
quite a bit of change has taken place on the master branch, and (2) even
when built from source on two separate, fresh, practically identical
Guix System machines, without using substitutes, the gcc-static binary
still differs.

Now that we have decided to use these powerpc64 bootstrap tarballs, what
are the next steps for uploading them to the GNU FTP server?  I've never
done that before, and I don't think I have access.  For now I've put a
signed copy of the powerpc64-linux (big endian) bootstrap tarballs, with
a SHA-512 hash, here:

https://media.marusich.info/guix-ppc64-bootstrap/bootstrap-tarballs-from-guix-1ced8379c7.tar.xz
https://media.marusich.info/guix-ppc64-bootstrap/bootstrap-tarballs-from-guix-1ced8379c7.tar.xz.asc
https://media.marusich.info/guix-ppc64-bootstrap/bootstrap-tarballs-from-guix-1ced8379c7.tar.xz.sha512sum

For the record, these bootstrap tarballs were built via the following
steps:

- Use
  https://ftp.gnu.org/gnu/guix/guix-system-install-1.2.0.x86_64-linux.iso.xz
  to install Guix System 1.2.0 on an x86_64-linux machine.
- Run: guix pull --no-substitutes --commit=1ced8379c7641788fa607b19b7a66d18f045362b
- Run: guix build --no-substitutes --target=powerpc64-linux-gnu bootstrap-tarballs
- I didn't run "guix system reconfigure" after installing Guix System;
  theoretically it shouldn't matter, but for the purpose of our
  experiment, I just left the system in its default configuration in
  order to ensure that the kernel etc. would be the same on both VMs.

Once we get these binaries into the GNU FTP server, I'll get started on
updating gnu/packages/bootstrap.scm and other files necessary to begin
bootstrapping powerpc64-linux.  I'll mainly be adapting the work that
Leo already did, and following the lead of others like Efraim and his
work on the wip-ppc branch.  I will probably have questions as I go, so
I'll ask on guix-devel.

Please let me know if you'd like this done on a special branch.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]