bug#36747: Official MesCC bootstrap binaries differ from my locally built ones

[Jan Nieuwenhuizen <janneke@gnu.org>]

[-- Attachment #1: Type: text/plain, Size: 3181 bytes --]

Mark H Weaver writes:

> I'm working to independently verify the new bootstrap binaries.  Toward
> that end, I locally built the new bootstrap tarballs by running the
> following command from a git checkout at commit
> ef809e3ac036eccc5f9c9edd8fb661d14ae15f2f.
>
>   ./pre-inst-env guix build bootstrap-tarballs --system=i686-linux
>
> This produces a mescc-tools-static-0.5.2-0.bb062b0-i686-linux.tar.xz
> tarball that differs from the official one.  See below for the precise
> differences, but what they boil down to is that the official tarball
> contains several references to:
>
>   /gnu/store/xjr9rv1p4vgg9sclwzsai17gqi1c1d9j-glibc-2.28
>
> Whereas in my locally built tarball, these references are instead to:
>
>   /gnu/store/ggj9164ahhn1kmdcyl3sm27qsxk46h01-glibc-2.28

Aww, that's not good :(

> First of all, I hope we can agree that our bootstrap binaries shouldn't
> include any references to the store.

I think that is our policy and it is probably safest not to do so; at
least in code.  A piece of documentation can include a string
/gnu/store/.... as long at it is static and not really a reference.

Store references in bootstrap binaries or scripts cannot be used or
depended on: if our bootstrap build is clean those references will not
exist when the bootstrap binaries run.

In any case, if store references are present in bootstrap binaries, then
they should be reproducible.  Removing store references is one way to
make them reproducible but I'm not sure if that's the best way?

Making a reference invalid helps making sure that it isn't used.  But if
it cannot be used anyway, removing it could hide the fact that it may
differs accros builds, as we are finding out right now.  Thoughts?

> However, it also raises the curious question of how I ended up with
> references to a different glibc than the one referenced by the official
> tarball.  Any idea what happened here?

Yes, that's something that I would like to know...

Anyway, I have built a new set of bootstrap binaries for mescc-tools and
mes using attched patches that remove store references (also on my
https://gitlab.com/janneke/guix core-updates).

I have just verified that tcc-boot0 builds by using this local
modification

--8<---------------cut here---------------start------------->8---
15:05:56 janneke@dundal:~/src/guix/core-updates [env]
$ git diff
diff --git a/gnu/packages/bootstrap.scm b/gnu/packages/bootstrap.scm
index 73fc7d3e5e..bca90bec94 100644
--- a/gnu/packages/bootstrap.scm
+++ b/gnu/packages/bootstrap.scm
@@ -288,7 +288,8 @@ or false to signal an error."
 
 (define %bootstrap-base-urls
   ;; This is where the initial binaries come from.
-  '("https://alpha.gnu.org/gnu/guix/bootstrap"
+  '("http://lilypond.org/janneke/guix"
+    "https://alpha.gnu.org/gnu/guix/bootstrap"
     "http://alpha.gnu.org/gnu/guix/bootstrap"
     "ftp://alpha.gnu.org/gnu/guix/bootstrap"
     "http://www.fdn.fr/~lcourtes/software/guix/packages"
--8<---------------cut here---------------end--------------->8---

and running
 
    ./pre-inst-env guix build --system=i686-linux -e '(@@ (gnu packages commencement) tcc-boot0)'

Thanks again for looking into this!

Greetings,
janneke


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-bootstrap-mescc-tools-static-stripped-Remove-store-r.patch --]
[-- Type: text/x-patch, Size: 2658 bytes --]

From 11c12e30bbdb1a10fd56dc9d5c14548e49b63c12 Mon Sep 17 00:00:00 2001
From: Jan Nieuwenhuizen <janneke@gnu.org>
Date: Sun, 21 Jul 2019 11:40:54 +0200
Subject: [PATCH 1/4] bootstrap: mescc-tools-static-stripped: Remove store
 references.

* gnu/packages/make-bootstrap.scm (%mescc-tools-static-stripped): Rename from
%mescc-tools-static; update users.  Remove store references.
---
 gnu/packages/make-bootstrap.scm | 27 +++++++++++++++++++--------
 1 file changed, 19 insertions(+), 8 deletions(-)

diff --git a/gnu/packages/make-bootstrap.scm b/gnu/packages/make-bootstrap.scm
index 2163b646f6..8cc4eef430 100644
--- a/gnu/packages/make-bootstrap.scm
+++ b/gnu/packages/make-bootstrap.scm
@@ -585,16 +585,27 @@ for `sh' in $PATH, and without nscd, and with static NSS modules."
            #t))))
     (inputs `(("gcc" ,%gcc-static)))))
 
-(define %mescc-tools-static
-  ;; A statically linked MesCC Tools for bootstrap.
+(define %mescc-tools-static-stripped
+  ;; A statically linked Mescc Tools with store references removed, for
+  ;; bootstrap.
   (package
     (inherit mescc-tools)
-    (name "mescc-tools-static")
+    (name "mescc-tools-static-stripped")
     (arguments
-     `(#:system "i686-linux"
-       ,@(substitute-keyword-arguments (package-arguments mescc-tools)
-           ((#:make-flags flags)
-            `(cons "CC=gcc -static" ,flags)))))))
+     `(#:make-flags (list (string-append "PREFIX=" (assoc-ref %outputs "out"))
+                          "CC=gcc -static")
+       #:test-target "test"
+       #:phases (modify-phases %standard-phases
+                  (delete 'configure)
+                  (add-after 'install 'strip-store-references
+                    (lambda _
+                      (let* ((out (assoc-ref %outputs "out"))
+                             (bin (string-append out "/bin")))
+                        (for-each (lambda (file)
+                                 (let ((target (string-append bin "/" file)))
+                                   (format #t "strippingg `~a'...~%" target)
+                                   (remove-store-references target)))
+                               '( "M1" "blood-elf" "hex2"))))))))))
 
 (define-public %mes-minimal-stripped
   ;; A minimal Mes without documentation dependencies, for bootstrap.
@@ -791,7 +802,7 @@ for `sh' in $PATH, and without nscd, and with static NSS modules."
 
 (define %mescc-tools-bootstrap-tarball
   ;; A tarball with MesCC binary seed.
-  (tarball-package %mescc-tools-static))
+  (tarball-package %mescc-tools-static-stripped))
 
 (define %mes-bootstrap-tarball
   ;; A tarball with Mes ASCII Seed and binary Mes C Library.
-- 
2.21.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-bootstrap-mes-minimal-stripped-Remove-store-referenc.patch --]
[-- Type: text/x-patch, Size: 2040 bytes --]

From 2e13b6fff94027158b3de5d3a1469dc9f89b0c39 Mon Sep 17 00:00:00 2001
From: Jan Nieuwenhuizen <janneke@gnu.org>
Date: Sun, 21 Jul 2019 14:27:07 +0200
Subject: [PATCH 2/4] bootstrap: mes-minimal-stripped: Remove store references.

* gnu/packages/make-bootstrap.scm (%mes-minimal-stripped): Remove store
references.
---
 gnu/packages/make-bootstrap.scm | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)

diff --git a/gnu/packages/make-bootstrap.scm b/gnu/packages/make-bootstrap.scm
index 8cc4eef430..af89ca8c3c 100644
--- a/gnu/packages/make-bootstrap.scm
+++ b/gnu/packages/make-bootstrap.scm
@@ -621,6 +621,7 @@ for `sh' in $PATH, and without nscd, and with static NSS modules."
          #:configure-flags '("--mes")
          #:phases
          (modify-phases %standard-phases
+           (delete 'patch-shebangs)
            (add-after 'install 'strip-install
              (lambda _
                (let* ((out (assoc-ref %outputs "out"))
@@ -628,10 +629,16 @@ for `sh' in $PATH, and without nscd, and with static NSS modules."
                  (delete-file-recursively (string-append out "/lib/guile"))
                  (delete-file-recursively (string-append share "/guile"))
                  (delete-file-recursively (string-append share "/mes/scaffold"))
-                 (for-each
-                  delete-file
-                  (find-files (string-append share  "/mes/lib")
-                              "\\.(h|c)")))))))))))
+
+                 (for-each delete-file
+                           (find-files
+                            (string-append share "/mes/lib") "\\.(h|c)"))
+
+                 (for-each (lambda (dir)
+                             (for-each remove-store-references
+                                       (find-files (string-append out "/" dir)
+                                                   ".*")))
+                           '("bin" "share/mes")))))))))))
 
 (define %guile-static
   ;; A statically-linked Guile that is relocatable--i.e., it can search
-- 
2.21.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: 0003-bootstrap-bootstrap-mescc-tools-Update.patch --]
[-- Type: text/x-patch, Size: 1704 bytes --]

From 172ee1045689cd42d2de837ee560d32c43730cc3 Mon Sep 17 00:00:00 2001
From: Jan Nieuwenhuizen <janneke@gnu.org>
Date: Sun, 21 Jul 2019 14:38:52 +0200
Subject: [PATCH 3/4] bootstrap: bootstrap-mescc-tools: Update.

    Built with
        2e13b6fff94027158b3de5d3a1469dc9f89b0c39 bootstrap: mes-minimal-stripped: Remove store references.

    * gnu/packages/bootstrap.scm (%bootstrap-mescc-tools): Update.
---
 gnu/packages/bootstrap.scm | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/gnu/packages/bootstrap.scm b/gnu/packages/bootstrap.scm
index 8dbe52435e..a8a5c93e01 100644
--- a/gnu/packages/bootstrap.scm
+++ b/gnu/packages/bootstrap.scm
@@ -703,7 +703,7 @@ exec ~a/bin/.gcc-wrapped -B~a/lib \
   ;; %MESCC-TOOLS-BOOTSTRAP-TARBALL.
   (package
     (name "bootstrap-mescc-tools")
-    (version "0.5.2")
+    (version "1")
     (source #f)
     (build-system trivial-build-system)
     (arguments
@@ -735,12 +735,12 @@ exec ~a/bin/.gcc-wrapped -B~a/lib \
             (method url-fetch)
             (uri (map
                   (cute string-append <>
-                        "/i686-linux/20181020/"
-                        "mescc-tools-static-0.5.2-0.bb062b0-i686-linux.tar.xz")
+                        "/i686-linux/20190721/"
+                        "mescc-tools-static-stripped-0.5.2-0.bb062b0-i686-linux.tar.xz")
                   %bootstrap-base-urls))
             (sha256
              (base32
-              "11lniw0vg61kmyhvnwkmcnkci9ym6hbmiksiqggd0hkipbq7hvlz")))))))
+              "0h7sbc5im74gxjx3xybqavq7hacbg7gcl4sznc1mpw67yja49f75")))))))
     (synopsis "Bootstrap binaries of MesCC Tools")
     (description synopsis)
     (home-page #f)
-- 
2.21.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #5: 0004-bootstrap-bootstrap-mes-Update.patch --]
[-- Type: text/x-patch, Size: 1591 bytes --]

From e316146f23dce4156fc8c5b3b28cd5e180487cda Mon Sep 17 00:00:00 2001
From: Jan Nieuwenhuizen <janneke@gnu.org>
Date: Sun, 21 Jul 2019 14:42:35 +0200
Subject: [PATCH 4/4] bootstrap: bootstrap-mes: Update.

    Built with
        2e13b6fff94027158b3de5d3a1469dc9f89b0c39 bootstrap: mes-minimal-stripped: Remove store references.

* gnu/packages/bootstrap.scm (%bootstrap-mes): Update.
---
 gnu/packages/bootstrap.scm | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/gnu/packages/bootstrap.scm b/gnu/packages/bootstrap.scm
index a8a5c93e01..73fc7d3e5e 100644
--- a/gnu/packages/bootstrap.scm
+++ b/gnu/packages/bootstrap.scm
@@ -752,7 +752,7 @@ exec ~a/bin/.gcc-wrapped -B~a/lib \
   ;; %MES-BOOTSTRAP-TARBALL.
   (package
     (name "bootstrap-mes")
-    (version "0")
+    (version "1")
     (source #f)
     (build-system trivial-build-system)
     (arguments
@@ -784,12 +784,12 @@ exec ~a/bin/.gcc-wrapped -B~a/lib \
             (method url-fetch)
             (uri (map
                   (cute string-append <>
-                        "/i686-linux/20181020/"
+                        "/i686-linux/20190721/"
                         "mes-minimal-stripped-0.19-i686-linux.tar.xz")
                   %bootstrap-base-urls))
             (sha256
              (base32
-              "0k7kkl68a6xaadv47ij0nr9jm5ca1ffj38n7f2lg80y72wdkwr9h")))))))
+              "0c8j9p84vx4w8fz6bkxf57wsarq3ng1ka09h18lv3lfxxvnpiak6")))))))
     (supported-systems '("i686-linux" "x86_64-linux"))
     (synopsis "Bootstrap binaries of Mes")
     (description synopsis)
-- 
2.21.0


[-- Attachment #6: Type: text/plain, Size: 152 bytes --]


-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com