bug#41669: Cross-compiled powerpc64-linux bootstrap-tarballs not reproducible

[Chris Marusich <cmmarusich@gmail.com>]

[-- Attachment #1.1: Type: text/plain, Size: 8566 bytes --]

Hi Ludo,

Ludovic Courtès <ludo@gnu.org> writes:

> You can update (gnu packages bootstrap) accordingly.

Thank you for uploading the little-endian bootstrap binaries!  I've
downloaded them, and I confirm that they are identical to the ones I
built.  Do you also plan to upload the big-endian bootstrap binaries?  I
think we were hoping to try both system types.

I'm afraid I've hit a snag using what you've uploaded, though.  It looks
like we'll need to extract bash, mkdir, tar, and xz from
static-binaries-0-powerpc64le-linux-gnu.tar.xz and place a copy of each
in the following locations:

- https://alpha.gnu.org/gnu/guix/bootstrap/powerpc64le-linux/20210106/bash
- https://alpha.gnu.org/gnu/guix/bootstrap/powerpc64le-linux/20210106/mkdir
- https://alpha.gnu.org/gnu/guix/bootstrap/powerpc64le-linux/20210106/tar
- https://alpha.gnu.org/gnu/guix/bootstrap/powerpc64le-linux/20210106/xz

Could you do that?  The reason why it's necessary is described below.

I've started making changes locally to (gnu packages bootstrap).  So
far, with the attached patch, I'm able to do the following things on my
POWER9 machine running ppc64le Debian (unstable).

I can successfully build Guix from source using Debian packages (and a
manually-built copy of guile-avahi and guile-gnutls).  With my patch, I
still have to supply the "--with-courage" configure option.  Although
"make" succeeded, "make check" failed on the following tests:

- tests/build-utils.scm
- test/challenge.scm
- tests/containers.scm
- tests/debug-link.scm

The failures fell into two categories:

- Some tests couldn't download the bootstrap bash.
- In tests/containers.scm, call-with-container evaluated to #f when it
  wasn't supposed to.

Nevertheless, I created the necessary build users and started the
guix-daemon via pre-inst-env.  I then tried building a simple package:

  ./pre-inst-env guix build -e '(@@ (gnu packages bootstrap) %bootstrap-coreutils&co)'

The result was promising, but it quickly failed like some of the tests
did - it couldn't download the bash bootstrap binary:

--8<---------------cut here---------------start------------->8---
marusich@suzaku:~/repos/guix$ ./pre-inst-env guix build -e '(@@ (gnu packages bootstrap) %bootstrap-coreutils&co)'
substitute: guix substitute: warning: ACL for archive imports seems to be uninitialized, substitutes may be unavailable
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
The following derivations will be built:
   /gnu/store/y9868ab6a4wjcvwzj4ln5fzk1y4y6zpz-bootstrap-binaries-0.drv
   /gnu/store/2nwml4l272qzq034hwf5icv9vxi813ja-xz.drv
   /gnu/store/c1v9lmsh0awbhpai72mzw4qv02rrbpw6-module-import-compiled.drv
   /gnu/store/n5hf44bybvqmsybjvnn61pkpmcdcrlbd-guile-bootstrap-2.0.drv
   /gnu/store/ix4mpvzxfi1hrmvdmmpgqhm9x1cdh347-bash.drv
   /gnu/store/jxh9xn77flxarwzcjga485pgrkjknrgb-tar.drv
   /gnu/store/yd1mib8s1f38qwdn61zj16ijx8p0ryzm-guile-static-stripped-2.0.14-powerpc64le-linux-gnu.tar.xz.drv
   /gnu/store/z66wc9z4qvffn60q4jdx7in6rxpswhx3-mkdir.drv
   /gnu/store/m4qv668s851v2ndzns3xwzg5rga9fhff-static-binaries-0-powerpc64le-linux-gnu.tar.xz.drv
building /gnu/store/m4qv668s851v2ndzns3xwzg5rga9fhff-static-binaries-0-powerpc64le-linux-gnu.tar.xz.drv...

Starting download of /gnu/store/2phdifnfw6i989rqbav04zakxx7qb165-static-binaries-0-powerpc64le-linux-gnu.tar.xz
From https://ftp.gnu.org/gnu/guix/bootstrap/powerpc64le-linux/20210106/static-binaries-0-powerpc64le-linux-gnu.tar.xz...
download failed "https://ftp.gnu.org/gnu/guix/bootstrap/powerpc64le-linux/20210106/static-binaries-0-powerpc64le-linux-gnu.tar.xz" 404 "Not Found"

Starting download of /gnu/store/2phdifnfw6i989rqbav04zakxx7qb165-static-binaries-0-powerpc64le-linux-gnu.tar.xz
From https://alpha.gnu.org/gnu/guix/bootstrap/powerpc64le-linux/20210106/static-binaries-0-powerpc64le-linux-gnu.tar.xz...
downloading from https://alpha.gnu.org/gnu/guix/bootstrap/powerpc64le-linux/20210106/static-binaries-0-powerpc64le-linux-gnu.tar.xz ...
 static-binaries-0-powerpc64le-linux-gnu.tar.xz  4.4MiB 2.8MiB/s 00:02 [##################] 100.0%
successfully built /gnu/store/m4qv668s851v2ndzns3xwzg5rga9fhff-static-binaries-0-powerpc64le-linux-gnu.tar.xz.drv
building /gnu/store/ix4mpvzxfi1hrmvdmmpgqhm9x1cdh347-bash.drv...

Starting download of /gnu/store/0kaj6l1ccw0qd0289hii7qhr828s71sv-bash
From https://git.savannah.gnu.org/cgit/guix.git/plain/gnu/packages/bootstrap/powerpc64le-linux/20210106/bash...
download failed "https://git.savannah.gnu.org/cgit/guix.git/plain/gnu/packages/bootstrap/powerpc64le-linux/20210106/bash" 404 "Not found"

Starting download of /gnu/store/0kaj6l1ccw0qd0289hii7qhr828s71sv-bash
From http://lilypond.org/janneke/guix/powerpc64le-linux/20210106/bash...
download failed "http://lilypond.org/janneke/guix/powerpc64le-linux/20210106/bash" 404 "Not Found"

Starting download of /gnu/store/0kaj6l1ccw0qd0289hii7qhr828s71sv-bash
From https://ci.guix.gnu.org/file/bash/sha256/1kiw7n6mkdy2x9in97646nb7aiayxr090ws1hbrlazah3fjqi6nj...
download failed "https://ci.guix.gnu.org/file/bash/sha256/1kiw7n6mkdy2x9in97646nb7aiayxr090ws1hbrlazah3fjqi6nj" 404 "Not Found"

Starting download of /gnu/store/0kaj6l1ccw0qd0289hii7qhr828s71sv-bash
From https://tarballs.nixos.org/sha256/1kiw7n6mkdy2x9in97646nb7aiayxr090ws1hbrlazah3fjqi6nj...
download failed "https://tarballs.nixos.org/sha256/1kiw7n6mkdy2x9in97646nb7aiayxr090ws1hbrlazah3fjqi6nj" 404 "Not Found"

Starting download of /gnu/store/0kaj6l1ccw0qd0289hii7qhr828s71sv-bash
From https://archive.softwareheritage.org/api/1/content/sha256:d29a88a51b507d45f38241739040ee5e45759635c49c6463eac2b7598d3d3cce/raw/...
download failed "https://archive.softwareheritage.org/api/1/content/sha256:d29a88a51b507d45f38241739040ee5e45759635c49c6463eac2b7598d3d3cce/raw/" 404 "Not Found"
failed to download "/gnu/store/0kaj6l1ccw0qd0289hii7qhr828s71sv-bash" from ("https://git.savannah.gnu.org/cgit/guix.git/plain/gnu/packages/bootstrap/powerpc64le-linux/20210106/bash" "http://lilypond.org/janneke/guix/powerpc64le-linux/20210106/bash")
builder for `/gnu/store/ix4mpvzxfi1hrmvdmmpgqhm9x1cdh347-bash.drv' failed to produce output path `/gnu/store/0kaj6l1ccw0qd0289hii7qhr828s71sv-bash'
build of /gnu/store/ix4mpvzxfi1hrmvdmmpgqhm9x1cdh347-bash.drv failed
View build log at '/var/log/guix/drvs/ix/4mpvzxfi1hrmvdmmpgqhm9x1cdh347-bash.drv.gz'.
cannot build derivation `/gnu/store/n5hf44bybvqmsybjvnn61pkpmcdcrlbd-guile-bootstrap-2.0.drv': 1 dependencies couldn't be built
building /gnu/store/jxh9xn77flxarwzcjga485pgrkjknrgb-tar.drv...
cannot build derivation `/gnu/store/y9868ab6a4wjcvwzj4ln5fzk1y4y6zpz-bootstrap-binaries-0.drv': 1 dependencies couldn't be built
guix build: error: build of `/gnu/store/y9868ab6a4wjcvwzj4ln5fzk1y4y6zpz-bootstrap-binaries-0.drv' failed
--8<---------------cut here---------------end--------------->8---

To resolve this issue, it seems that we must also extract the bootstrap
bash, mkdir, tar, and xz from
static-binaries-0-powerpc64le-linux-gnu.tar.xz and place a copy of each
in the following locations:

- https://alpha.gnu.org/gnu/guix/bootstrap/powerpc64le-linux/20210106/bash
- https://alpha.gnu.org/gnu/guix/bootstrap/powerpc64le-linux/20210106/mkdir
- https://alpha.gnu.org/gnu/guix/bootstrap/powerpc64le-linux/20210106/tar
- https://alpha.gnu.org/gnu/guix/bootstrap/powerpc64le-linux/20210106/xz

After you've uploaded these files, I'll also need to adjust my patch so
it adds alpha.gnu.org to %bootstrap-executable-base-urls in
gnu/packages/bootstrap.scm.  That's how Guix finds these raw, unpackaged
binaries.

By the way, I've noticed that the other architectures don't seem to have
"raw" binaries at all on alpha.gnu.org.  Maybe you already knew this,
but it seems that the "raw" binaries are actually downloaded from this
specific URL (see the bootstrap-executable-file-name procedure):

https://git.savannah.gnu.org/cgit/guix.git/plain/gnu/packages/bootstrap/?id=44f07d1dc6806e97c4e9ee3e6be883cc59dc666e

Is that intended?  I was surprised to discover that we store these four
"raw" binaries in a totally separate place.  That seems like it would
make it easy for someone to accidentally forget to update the "raw"
binaries when they update an architecture's bootstrap tarballs.

In any case, for powerpc64le-linux, the right thing to do is probably
just to add copies of these "raw" binaries like I mentioned above.
Could you do that when you have the time?

Thank you,

-- 
Chris

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.2: 0001-gnu-bootstrap-Add-support-for-powerpc64le-linux.patch --]
[-- Type: text/x-patch, Size: 8637 bytes --]

From eb41c77618538f1f6c2994c9245d38c377ac12ad Mon Sep 17 00:00:00 2001
From: Chris Marusich <cmmarusich@gmail.com>
Date: Sun, 10 Jan 2021 16:58:57 -0800
Subject: [PATCH] gnu: bootstrap: Add support for powerpc64le-linux.

The bootstrap tarballs used by these bootstrap packages were created via the
following steps:

- Create a new x86_64 VM using QEMU.
- Use
  https://ftp.gnu.org/gnu/guix/guix-system-install-1.2.0.x86_64-linux.iso.xz
  to install Guix System 1.2.0 in the VM.
- Run: guix pull --no-substitutes --commit=662e7e28d576ada91fc9dec7d27c100666114f03
- Run: guix build --no-substitutes --target=powerpc64le-linux-gnu bootstrap-tarballs

With the exception of gcc-static, all bootstrap binaries contained in these
tarballs can be built reproducibly.  Unfortunately, gcc-static is not always
reproducible when everything is built from source on separate machines.
Despite months of investigation, we still do not know why that is, so we have
decided to move forward with these binaries to unblock further bootstrapping
work.  For details, see <https://bugs.gnu.org/41669>, which contains a
discussion of both the big-endian and little-endian cases.

* gnu/packages/bootstrap.scm (%bootstrap-executables)
(bootstrap-executable-file-name, bootstrap-guile-url-path)
(bootstrap-guile-hash, %bootstrap-coreutils&co, %bootstrap-binutils)
(%bootstrap-glibc, %bootstrap-gcc): Add entries for powerpc64le-linux.
---
 gnu/packages/bootstrap.scm | 36 +++++++++++++++++++++++++++++++++++-
 1 file changed, 35 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/bootstrap.scm b/gnu/packages/bootstrap.scm
index c39e60ec8b..d9ba9f06c6 100644
--- a/gnu/packages/bootstrap.scm
+++ b/gnu/packages/bootstrap.scm
@@ -122,7 +122,16 @@
      ("tar"
       ,(base32 "06gmqdjq3rl8lr47b9fyx4ifnm5x56ymc8lyryp1ax1j2s4y5jb4"))
      ("xz"
-      ,(base32 "09j1d69qr0hhhx4k4ih8wp00dfc9y4rp01hfg3vc15yxd0jxabh5")))))
+      ,(base32 "09j1d69qr0hhhx4k4ih8wp00dfc9y4rp01hfg3vc15yxd0jxabh5")))
+    ("powerpc64le-linux"
+     ("bash"
+      ,(base32 "1kiw7n6mkdy2x9in97646nb7aiayxr090ws1hbrlazah3fjqi6nj"))
+     ("mkdir"
+      ,(base32 "04dpvi231zcl40ig048vqqnyvmnkw1byrm1q1qqvs1f0g16yhrrk"))
+     ("tar"
+      ,(base32 "150c8948cz8r208g6qgn2dn4f4zs5kpgbpbg6bwag6yw42rapw2l"))
+     ("xz"
+      ,(base32 "0v5738idy9pqzcbrjdpxi5c6qs5m78zrpsydmrpx5cfcfzbkxzjh")))))
 
 (define %bootstrap-executable-base-urls
   ;; This is where the bootstrap executables come from.
@@ -132,6 +141,7 @@
 (define (bootstrap-executable-file-name system program)
   "Return the FILE-NAME part of url where PROGRAM can be found for SYSTEM."
   (match system
+    ("powerpc64le-linux" (string-append system "/20210106/" program))
     ("i586-gnu" (string-append system "/20200326/" program))
     (_ (string-append system "/" program
                       "?id=44f07d1dc6806e97c4e9ee3e6be883cc59dc666e"))))
@@ -332,6 +342,8 @@ or false to signal an error."
                     "/20150101/guile-2.0.11.tar.xz")
                    ("i586-gnu"
                     "/20200326/guile-static-stripped-2.0.14-i586-pc-gnu.tar.xz")
+                   ("powerpc64le-linux"
+                    "/20210106/guile-static-stripped-2.0.14-powerpc64le-linux-gnu.tar.xz")
                    (_
                     "/20131110/guile-2.0.9.tar.xz"))))
 
@@ -346,6 +358,8 @@ or false to signal an error."
      (base32 "0fzp93lvi0hn54acc0fpvhc7bvl0yc853k62l958cihk03q80ilr"))
     ("armhf-linux"
      (base32 "1mi3brl7l58aww34rawhvja84xc7l1b4hmwdmc36fp9q9mfx0lg5"))
+    ("powerpc64le-linux"
+     (base32 "1rnyfz5q38jyvxddj617443bnnzql4vw0mxzqpj8wz48wx4bhbq0"))
     ("aarch64-linux"
      (base32 "1giy2aprjmn5fp9c4s9r125fljw4wv6ixy5739i5bffw4jgr0f9r"))
     ("i586-gnu"
@@ -479,6 +493,8 @@ $out/bin/guile --version~%"
                                              "/20150101/static-binaries.tar.xz")
                                             ("aarch64-linux"
                                              "/20170217/static-binaries.tar.xz")
+                                            ("powerpc64le-linux"
+                                             "/20210106/static-binaries-0-powerpc64le-linux-gnu.tar.xz")
                                             ("i586-gnu"
                                              "/20200326/static-binaries-0-i586-pc-gnu.tar.xz")
                                             (_
@@ -498,6 +514,9 @@ $out/bin/guile --version~%"
                               ("aarch64-linux"
                                (base32
                                 "18dfiq6c6xhsdpbidigw6480wh0vdgsxqq3xindq4lpdgqlccpfh"))
+                              ("powerpc64le-linux"
+                               (base32
+                                "0afs2j9z2d1hjq42myz4iwjh0aqgzf59inifw87x6b6p1z9wv92v"))
                               ("i586-gnu"
                                (base32
                                 "17kllqnf3fg79gzy9ansgi801c46yh9c23h4d923plvb0nfm1cfn"))
@@ -547,6 +566,8 @@ $out/bin/guile --version~%"
                                              "/20150101/binutils-2.25.tar.xz")
                                             ("aarch64-linux"
                                              "/20170217/binutils-2.27.tar.xz")
+                                            ("powerpc64le-linux"
+                                             "/20210106/binutils-static-stripped-2.34-powerpc64le-linux-gnu.tar.xz")
                                             ("i586-gnu"
                                              "/20200326/binutils-static-stripped-2.34-i586-pc-gnu.tar.xz")
                                             (_
@@ -566,6 +587,9 @@ $out/bin/guile --version~%"
                               ("aarch64-linux"
                                (base32
                                 "111s7ilfiby033rczc71797xrmaa3qlv179wdvsaq132pd51xv3n"))
+                              ("powerpc64le-linux"
+                               (base32
+                                "1klxy945c61134mzhqzz2gbk8w0n8jq7arwkrvz78d22ff2q0cwz"))
                               ("i586-gnu"
                                (base32
                                 "11kykv1kmqc5wln57rs4klaqa13hm952smkc57qcsyss21kfjprs"))
@@ -622,6 +646,8 @@ $out/bin/guile --version~%"
                                        "/20150101/glibc-2.20.tar.xz")
                                       ("aarch64-linux"
                                        "/20170217/glibc-2.25.tar.xz")
+                                      ("powerpc64le-linux"
+                                       "/20210106/gcc-stripped-5.5.0-powerpc64le-linux-gnu.tar.xz")
                                       ("i586-gnu"
                                        "/20200326/glibc-stripped-2.31-i586-pc-gnu.tar.xz")
                                       (_
@@ -641,6 +667,9 @@ $out/bin/guile --version~%"
                         ("aarch64-linux"
                          (base32
                           "07nx3x8598i2924rjnlrncg6rm61c9bmcczbbcpbx0fb742nvv5c"))
+                        ("powerpc64le-linux"
+                         (base32
+                          "151kjsai25vz2s667bgzpisx8f281fpl3n9pxz2yrp9jlnadz3m1"))
                         ("i586-gnu"
                          (base32
                           "14ddm10lpbas8bankmn5bcrlqvz1v5dnn1qjzxb19r57vd2w5952"))
@@ -713,6 +742,8 @@ exec ~a/bin/.gcc-wrapped -B~a/lib \
                                         "/20150101/gcc-4.8.4.tar.xz")
                                        ("aarch64-linux"
                                         "/20170217/gcc-5.4.0.tar.xz")
+                                       ("powerpc64le-linux"
+                                        "/20210106/gcc-stripped-5.5.0-powerpc64le-linux-gnu.tar.xz")
                                        ("i586-gnu"
                                         "/20200326/gcc-stripped-5.5.0-i586-pc-gnu.tar.xz")
                                        (_
@@ -732,6 +763,9 @@ exec ~a/bin/.gcc-wrapped -B~a/lib \
                          ("aarch64-linux"
                           (base32
                            "1ar3vdzyqbfm0z36kmvazvfswxhcihlacl2dzdjgiq25cqnq9ih1"))
+                         ("powerpc64le-linux"
+                          (base32
+                           "151kjsai25vz2s667bgzpisx8f281fpl3n9pxz2yrp9jlnadz3m1"))
                          ("i586-gnu"
                           (base32
                            "1j2zc58wzil71a34h7c70sd68dmqvcscrw3rmn2whq79vd70zvv5"))
-- 
2.26.2


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]