Preparing the reduced bootstrap tarballs

Preparing the reduced bootstrap tarballs

[Ludovic Courtès]

Hello!

(This is a followup to <https://bugs.gnu.org/33038>.)

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

> Did those, pushed to core-updates-next as
>
>     d0bb7ed61ed9e356c53de1a8e9bd6c2ec030ffb6
>     doc: Update Preparing to Use the Bootstrap Binaries.

[...]

> I think that I'm "done" here, there's the `put new bootstrap binaries on
> ftp.gnu.org/pub/guix/...', are you taking care of that?

I (finally!) run “guix build bootstrap-tarballs” on ‘core-updates-next’,
but that was a bit silly of me since that built the x86_64-linux
tarballs—i.e., not the “reduced seed.”

So I was about to re-run it with “-s i686-linux”, but I noticed the
following issue in (gnu packages make-bootstrap):

--8<---------------cut here---------------start------------->8---
(define %bootstrap-tarballs
  ;; A single derivation containing all the bootstrap tarballs, for
  ;; convenience.
  (package

[...]

    (inputs `(("guile-tarball" ,%guile-bootstrap-tarball)
              ,@(match (%current-system)
                  ("i686-linux" `(("mescc-tools-seed" ,(@ (gnu packages bootstrap) %mescc-tools-seed))
                                  ("mes-seed" ,(@ (gnu packages bootstrap) %mes-seed))
                                  ("srfi-43" ,(@ (gnu packages bootstrap) %srfi-43))
                                  ("tinycc-seed" ,(@ (gnu packages bootstrap) %tinycc-seed))))
--8<---------------cut here---------------end--------------->8---

This does not actually build the bootstrap tarballs; instead it returns
the pre-built seeds (also the ‘%tinycc-seed’ variable doesn’t exist.)

What we would need here is something to build the things listed in
‘%bootstrap-inputs’, namely:
‘linux-libre-headers-stripped-4.14.26-i686-linux.tar.xz’ (easy :-)),
‘mescc-tools-seed-XYZ.tar.gz’, and
‘mes-stripped-0.18-0.08f04f5-i686-linux.tar.xz’ (do we really need an
x86_64 version of this Mes?).

Does that make sense?

Sorry for the delay, and apologies if I overlooked something!

Ludo’.

Re: Preparing the reduced bootstrap tarballs

[Jan Nieuwenhuizen]

Ludovic Courtès writes:

Hi!

> (This is a followup to <https://bugs.gnu.org/33038>.)

> I (finally!) run “guix build bootstrap-tarballs” on ‘core-updates-next’,
> but that was a bit silly of me since that built the x86_64-linux
> tarballs—i.e., not the “reduced seed.”
>
> So I was about to re-run it with “-s i686-linux”, but I noticed the
> following issue in (gnu packages make-bootstrap):
>
> (define %bootstrap-tarballs
>   ;; A single derivation containing all the bootstrap tarballs, for
>   ;; convenience.
>   (package

Ah right.  I saw that several times but did not use it.  I think because
initially it was of no use.  It would be nice if this built everything
we need, I agree :-)

> [...]
>
>     (inputs `(("guile-tarball" ,%guile-bootstrap-tarball)
>               ,@(match (%current-system)
>                   ("i686-linux" `(("mescc-tools-seed" ,(@ (gnu packages bootstrap) %mescc-tools-seed))
>                                   ("mes-seed" ,(@ (gnu packages bootstrap) %mes-seed))
>                                   ("srfi-43" ,(@ (gnu packages bootstrap) %srfi-43))
>                                   ("tinycc-seed" ,(@ (gnu packages bootstrap) %tinycc-seed))))
>
> This does not actually build the bootstrap tarballs; instead it returns
> the pre-built seeds (also the ‘%tinycc-seed’ variable doesn’t exist.)

Indeed.  "mes-seed" and "tinycc-seed" are remnants of the past; the only
things we need are

> What we would need here is something to build the things listed in
> ‘%bootstrap-inputs’, namely:
> ‘linux-libre-headers-stripped-4.14.26-i686-linux.tar.xz’ (easy :-)),
> ‘mescc-tools-seed-XYZ.tar.gz’, and
> ‘mes-stripped-0.18-0.08f04f5-i686-linux.tar.xz’

So if you like, please make that change.  There is only one little
thing: I have no (scripted) recipe to create mescc-tools-seed-XYZ.  But
wait: I have a great excuse for that...I was too lazy or too sloppy.

The thing is, I used to build mescc-tools-seed, mes-seed and tinycc-seed
manually from the mes+mescc+tinycc source trees.  Jeremiah Orians is
working to remove any need for mescc-tools-seed (esp. the forward
dependency on Mes) but I don't think we're there yet.

Anyway, I think we/I will have to put some work into scripting
mescc-tools-seed or otherwise changing the mescc-tools-boot build.
WDYT?

> (do we really need an x86_64 version of this Mes?).

No, I don't think so.  I added it esp. to get a preview and enable
future development of pure x86_64 bootstrap; but dependency-wise we
should be able to drop it!

> Does that make sense?

Yes!

> Sorry for the delay, and apologies if I overlooked something!

Thank you for looking into and pointing out!
janneke

Re: Preparing the reduced bootstrap tarballs

[Jeremiah]

> Indeed.  "mes-seed" and "tinycc-seed" are remnants of the past; the only
> things we need are
>
>> What we would need here is something to build the things listed in
>> ‘%bootstrap-inputs’, namely:
>> ‘linux-libre-headers-stripped-4.14.26-i686-linux.tar.xz’ (easy :-)),
>> ‘mescc-tools-seed-XYZ.tar.gz’, and
>> ‘mes-stripped-0.18-0.08f04f5-i686-linux.tar.xz’

That is correct (With mescc-tools-seed adding steps and removing
binaries over time)

> So if you like, please make that change.  There is only one little
> thing: I have no (scripted) recipe to create mescc-tools-seed-XYZ.  But
> wait: I have a great excuse for that...I was too lazy or too sloppy.

I do, in mescc-tools-seed; the script bootstrap.sh when run with the
option "sin" will build the mescc-tools-seed binaries using mescc-tools.
The .M1 files are always generated by cc_x86.s using the C source files.


> The thing is, I used to build mescc-tools-seed, mes-seed and tinycc-seed
> manually from the mes+mescc+tinycc source trees.
I've been building them from the stage0 bootstrap tree (which as you can
see is rather trivial:
https://github.com/oriansj/talk-notes/blob/master/Current%20bootstrap%20map.dot
)


> Jeremiah Orians is
> working to remove any need for mescc-tools-seed (esp. the forward
> dependency on Mes) but I don't think we're there yet.
We have eliminated the forward dependency on Mes for the creation of the
mescc-tools-seed already


> Anyway, I think we/I will have to put some work into scripting
> mescc-tools-seed or otherwise changing the mescc-tools-boot build.
Already done in bash and kaem but not in guix yet (Should be trivial)

> WDYT?
I think we will end up having several versions of mescc-tools-seed; as
each architecture guix supports will end up needing a variant if we plan
on keeping them small. (I also have no idea how to make a multi-arch fat
elf binary)

I am also curious if there is any demand for the stripped versions of
mescc-tools-seed as those binaries are nearly half the size.

>> (do we really need an x86_64 version of this Mes?).
> No, I don't think so.  I added it esp. to get a preview and enable
> future development of pure x86_64 bootstrap; but dependency-wise we
> should be able to drop it!
also, AMD64 does support i386 binaries without issue

- Jeremiah

Re: Preparing the reduced bootstrap tarballs

[Ludovic Courtès]

Hello!

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

> Ludovic Courtès writes:
>
> Hi!
>
>> (This is a followup to <https://bugs.gnu.org/33038>.)
>
>> I (finally!) run “guix build bootstrap-tarballs” on ‘core-updates-next’,
>> but that was a bit silly of me since that built the x86_64-linux
>> tarballs—i.e., not the “reduced seed.”
>>
>> So I was about to re-run it with “-s i686-linux”, but I noticed the
>> following issue in (gnu packages make-bootstrap):
>>
>> (define %bootstrap-tarballs
>>   ;; A single derivation containing all the bootstrap tarballs, for
>>   ;; convenience.
>>   (package
>
> Ah right.  I saw that several times but did not use it.  I think because
> initially it was of no use.  It would be nice if this built everything
> we need, I agree :-)

Heheh.  :-)  It’s what the “Building the Bootstrap Binaries” section
describes.

> Indeed.  "mes-seed" and "tinycc-seed" are remnants of the past; the only
> things we need are

OK.

>> What we would need here is something to build the things listed in
>> ‘%bootstrap-inputs’, namely:
>> ‘linux-libre-headers-stripped-4.14.26-i686-linux.tar.xz’ (easy :-)),
>> ‘mescc-tools-seed-XYZ.tar.gz’, and
>> ‘mes-stripped-0.18-0.08f04f5-i686-linux.tar.xz’
>
> So if you like, please make that change.  There is only one little
> thing: I have no (scripted) recipe to create mescc-tools-seed-XYZ.  But
> wait: I have a great excuse for that...I was too lazy or too sloppy.
>
> The thing is, I used to build mescc-tools-seed, mes-seed and tinycc-seed
> manually from the mes+mescc+tinycc source trees.  Jeremiah Orians is
> working to remove any need for mescc-tools-seed (esp. the forward
> dependency on Mes) but I don't think we're there yet.
>
> Anyway, I think we/I will have to put some work into scripting
> mescc-tools-seed or otherwise changing the mescc-tools-boot build.
> WDYT?

I’m confused: how did you build the seeds that (gnu packages bootstrap)
refers to in ‘core-updates-next’?

The goal is for the seeds to be built through Guix so we have a
transparent and documented way to reproduce/verify them.

I could propose a patch to do that, though from what you’re saying
generating ‘mescc-tools-seed’ is not something readily doable?

>> (do we really need an x86_64 version of this Mes?).
>
> No, I don't think so.  I added it esp. to get a preview and enable
> future development of pure x86_64 bootstrap; but dependency-wise we
> should be able to drop it!

Neat.

Thanks for the explanations!

Ludo’.

Re: Preparing the reduced bootstrap tarballs

[Ludovic Courtès]

Jeremiah@pdp10.guru skribis:

>> So if you like, please make that change.  There is only one little
>> thing: I have no (scripted) recipe to create mescc-tools-seed-XYZ.  But
>> wait: I have a great excuse for that...I was too lazy or too sloppy.
>
> I do, in mescc-tools-seed; the script bootstrap.sh when run with the
> option "sin" will build the mescc-tools-seed binaries using mescc-tools.
> The .M1 files are always generated by cc_x86.s using the C source files.

I saw this script but it’s not entirely clear to me how to package the
whole thing.  We don’t have a “stage0” package for instance in Guix, do
we?

>> WDYT?
> I think we will end up having several versions of mescc-tools-seed; as
> each architecture guix supports will end up needing a variant if we plan
> on keeping them small. (I also have no idea how to make a multi-arch fat
> elf binary)

For now let’s focus on x86_64/i686.  :-)

IMO we should change the seeds as rarely as possible because they are
managed “out-of-band” and verifying them is difficult (you need to fetch
the right Guix commit, run “guix build bootstrap-tarballs”, and compare
the result—assuming this is all bit-reproducible.)

The one we’re using today in Guix date back to 2013.

Thanks,
Ludo’.

Re: Preparing the reduced bootstrap tarballs

[Jan Nieuwenhuizen]

[-- Attachment #1: Type: text/plain, Size: 1701 bytes --]

Ludovic Courtès writes:

> Jeremiah@pdp10.guru skribis:
>
>>> So if you like, please make that change.  There is only one little
>>> thing: I have no (scripted) recipe to create mescc-tools-seed-XYZ.  But
>>> wait: I have a great excuse for that...I was too lazy or too sloppy.
>>
>> I do, in mescc-tools-seed; the script bootstrap.sh when run with the
>> option "sin" will build the mescc-tools-seed binaries using mescc-tools.
>> The .M1 files are always generated by cc_x86.s using the C source files.
>
> I saw this script but it’s not entirely clear to me how to package the
> whole thing.  We don’t have a “stage0” package for instance in Guix, do
> we?

I had a look but it seems we need either binary seeds for stage0, or
binary seeds for mescc-tools and I'd rather build them within Guix; so
that's what I did now.

>>> WDYT?
>> I think we will end up having several versions of mescc-tools-seed; as
>> each architecture guix supports will end up needing a variant if we plan
>> on keeping them small. (I also have no idea how to make a multi-arch fat
>> elf binary)
>
> For now let’s focus on x86_64/i686.  :-)
>
> IMO we should change the seeds as rarely as possible because they are
> managed “out-of-band” and verifying them is difficult (you need to fetch
> the right Guix commit, run “guix build bootstrap-tarballs”, and compare
> the result—assuming this is all bit-reproducible.)
>
> The one we’re using today in Guix date back to 2013.

Okay, I am proposing the attached patches, also available on my 
core-updates-next at http://gitlab.com/janneke/guix

Still I hope these won't last for five more years ;-)
janneke.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-bootstrap-Add-mescc-tools-static-mescc-tools-static-.patch --]
[-- Type: text/x-patch, Size: 1884 bytes --]

From 5702107a31f52a615c516084b7a82d9f5e2967e0 Mon Sep 17 00:00:00 2001
From: Jan Nieuwenhuizen <janneke@gnu.org>
Date: Fri, 16 Nov 2018 19:29:55 +0100
Subject: [PATCH 1/3] bootstrap: Add mescc-tools-static,
 mescc-tools-static-tarball.

* gnu/packages/make-bootstrap.scm (%mescc-tools-static,
%mescc-tools-bootstrap-tarball):  New variable.
---
 gnu/packages/make-bootstrap.scm | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/gnu/packages/make-bootstrap.scm b/gnu/packages/make-bootstrap.scm
index 7bfba3c14..36a5f6904 100644
--- a/gnu/packages/make-bootstrap.scm
+++ b/gnu/packages/make-bootstrap.scm
@@ -48,6 +48,7 @@
             %glibc-bootstrap-tarball
             %gcc-bootstrap-tarball
             %guile-bootstrap-tarball
+            %mescc-tools-bootstrap-tarball
             %mes-bootstrap-tarball
             %bootstrap-tarballs
 
@@ -535,6 +536,16 @@ for `sh' in $PATH, and without nscd, and with static NSS modules."
            #t))))
     (inputs `(("gcc" ,%gcc-static)))))
 
+(define %mescc-tools-static
+  ;; A statically linked MesCC Tools for bootstrap.
+  (package
+    (inherit mescc-tools)
+    (name "mescc-tools-static")
+    (arguments
+     (substitute-keyword-arguments (package-arguments mescc-tools)
+       ((#:make-flags flags)
+        `(cons "CC=gcc -static" ,flags))))))
+
 (define %mes-stripped
   ;; The subset of Mes files needed for bootstrap.
   (package
@@ -731,6 +742,10 @@ for `sh' in $PATH, and without nscd, and with static NSS modules."
   ;; A tarball with the statically-linked, relocatable Guile.
   (tarball-package %guile-static-stripped))
 
+(define %mescc-tools-bootstrap-tarball
+  ;; A tarball with MesCC binary seed.
+  (tarball-package %mescc-tools-static))
+
 (define %mes-bootstrap-tarball
   ;; A tarball with Mes ASCII Seed and binary Mes C Library.
   (tarball-package %mes-stripped))
-- 
2.19.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-bootstrap-Add-bootstrap-mescc-tools.patch --]
[-- Type: text/x-patch, Size: 3163 bytes --]

From cc3782c5f9d2c606a265be0dfdca44d79ec25622 Mon Sep 17 00:00:00 2001
From: Jan Nieuwenhuizen <janneke@gnu.org>
Date: Fri, 16 Nov 2018 21:30:00 +0100
Subject: [PATCH 2/3] bootstrap: Add %bootstrap-mescc-tools.

Built with
    5702107a31f52a615c516084b7a82d9f5e2967e0 bootstrap: Add mescc-tools-static, mescc-tools-static-tarball.

* gnu/packages/bootstrap.scm (%bootstrap-mescc-tools): New variable.
---
 gnu/packages/bootstrap.scm | 49 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 49 insertions(+)

diff --git a/gnu/packages/bootstrap.scm b/gnu/packages/bootstrap.scm
index 7e6200c57..1075325dc 100644
--- a/gnu/packages/bootstrap.scm
+++ b/gnu/packages/bootstrap.scm
@@ -49,6 +49,7 @@
             %bootstrap-gcc
             %bootstrap-glibc
             %bootstrap-inputs
+            %bootstrap-mescc-tools
             %bootstrap-mes
             %mescc-tools-seed
             %srfi-43))
@@ -615,6 +616,54 @@ exec ~a/bin/.gcc-wrapped -B~a/lib \
     (home-page #f)
     (license gpl3+)))
 
+(define %bootstrap-mescc-tools
+  ;; The initial MesCC tools.  Uses binaries from a tarball typically built by
+  ;; %MESCC-TOOLS-BOOTSTRAP-TARBALL.
+  (package
+    (name "bootstrap-mescc-tools")
+    (version "0.5.2")
+    (source #f)
+    (build-system trivial-build-system)
+    (arguments
+     `(#:guile ,%bootstrap-guile
+       #:modules ((guix build utils))
+       #:builder
+       (begin
+         (use-modules (guix build utils)
+                      (ice-9 popen))
+         (let ((out     (assoc-ref %outputs "out"))
+               (tar     (assoc-ref %build-inputs "tar"))
+               (xz      (assoc-ref %build-inputs "xz"))
+               (tarball (assoc-ref %build-inputs "tarball")))
+
+           (mkdir out)
+           (copy-file tarball "binaries.tar.xz")
+           (invoke xz "-d" "binaries.tar.xz")
+           (let ((builddir (getcwd))
+                 (bindir   (string-append out "/bin")))
+             (with-directory-excursion out
+               (invoke tar "xvf"
+                       (string-append builddir "/binaries.tar"))))))))
+    (inputs
+     `(("tar" ,(search-bootstrap-binary "tar" (%current-system)))
+       ("xz"  ,(search-bootstrap-binary "xz" (%current-system)))
+       ("tarball" ,(bootstrap-origin
+                    (origin
+                      (method url-fetch)
+                      (uri (string-append
+                            "http://lilypond.org/janneke/mes/"
+                            (match (%current-system)
+                              ((or "i686-linux" "x86_64-linux")
+                               "mescc-tools-static-0.5.2-0.bb062b0-i686-linux.tar.xz"))))
+                      (sha256
+                       (match (%current-system)
+                         ((or "i686-linux" "x86_64-linux")
+                          (base32 "1qcqz5lfl2blbfvh5ccg08pcxfhqrmlrsh6cyaxvm0hxsr1rvlm1")))))))))
+    (synopsis "Bootstrap binaries of MesCC Tools")
+    (description synopsis)
+    (home-page #f)
+    (license gpl3+)))
+
 (define %bootstrap-mes
   ;; The initial Mes.  Uses binaries from a tarball typically built by
   ;; %MES-BOOTSTRAP-TARBALL.
-- 
2.19.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: 0003-bootstrap-Replace-mescc-tools-seed-with-bootstrap-me.patch --]
[-- Type: text/x-patch, Size: 6662 bytes --]

From daa9ff9463917687aabed8330eacb304a7e4deae Mon Sep 17 00:00:00 2001
From: Jan Nieuwenhuizen <janneke@gnu.org>
Date: Fri, 16 Nov 2018 21:30:28 +0100
Subject: [PATCH 3/3] bootstrap: Replace %mescc-tools-seed with
 %bootstrap-mescc-tools.

* gnu/packages/bootstrap.scm (%mescc-tools-seed): Remove.
(%bootstrap-inputs): Replace %mescc-tools-seed with %bootstrap-mescc-tools.
* gnu/packages/commencement.scm (mes-boot): Likewise.
(tcc-boot0): Likewise.
(mescc-tools-boot): Remove.
---
 gnu/packages/bootstrap.scm    | 14 +------
 gnu/packages/commencement.scm | 79 ++---------------------------------
 2 files changed, 4 insertions(+), 89 deletions(-)

diff --git a/gnu/packages/bootstrap.scm b/gnu/packages/bootstrap.scm
index 1075325dc..2eac08cb4 100644
--- a/gnu/packages/bootstrap.scm
+++ b/gnu/packages/bootstrap.scm
@@ -51,7 +51,6 @@
             %bootstrap-inputs
             %bootstrap-mescc-tools
             %bootstrap-mes
-            %mescc-tools-seed
             %srfi-43))
 
 ;;; Commentary:
@@ -716,17 +715,6 @@ exec ~a/bin/.gcc-wrapped -B~a/lib \
     (home-page #f)
     (license gpl3+)))
 
-(define %mescc-tools-seed ; todo: add tarballs to alpha.gnu.org/pub/mes/bootstrap/
-  (let ((commit "dc4e20e74924a5c80a2b7a77b4d7b927234fa71c"))
-    (origin
-      (method url-fetch)
-      (uri (string-append "https://gitlab.com/janneke/mescc-tools-seed"
-                          "/-/archive/" commit
-                          "/mescc-tools-seed-" commit ".tar.gz"))
-      (sha256
-       (base32
-        "1lj7df73vxanmffmiwkhcn83r7yd9n8568nkki06bqq5zg526nyz")))))
-
 (define %srfi-43
   (origin
     (method url-fetch)
@@ -742,7 +730,7 @@ exec ~a/bin/.gcc-wrapped -B~a/lib \
   `(,@(match (%current-system)
         ((or "i686-linux" "x86_64-linux")
          `(("linux-libre-headers" ,%bootstrap-linux-libre-headers)
-           ("mescc-tools-seed" ,%mescc-tools-seed)
+           ("bootstrap-mescc-tools" ,%bootstrap-mescc-tools)
            ("mes" ,%bootstrap-mes)
            ("srfi-43" ,%srfi-43 )))
         (_
diff --git a/gnu/packages/commencement.scm b/gnu/packages/commencement.scm
index 4def178a8..43373b591 100644
--- a/gnu/packages/commencement.scm
+++ b/gnu/packages/commencement.scm
@@ -106,79 +106,6 @@
       (native-inputs '())
       (propagated-inputs '()))))
 
-(define mescc-tools-boot
-  (package-with-bootstrap-guile
-   (package
-     (inherit mescc-tools)
-     (name "mescc-tools-boot")
-     (version "0.5.2")
-     (source (origin
-               (method url-fetch)
-               (uri (string-append
-                     "http://git.savannah.nongnu.org/cgit/mescc-tools.git/snapshot/"
-                     name "-Release_" version
-                     ".tar.gz"))
-               (patches (search-patches "mescc-tools-boot.patch"))
-               (file-name (string-append "mescc-tools" "-" version ".tar.gz"))
-               (sha256
-                (base32
-                 "01x7bhmgwyf6mc2g1hcvibhps98nllacqm4f0j5l51b1mbi18pc2"))))
-     (inputs '())
-     (propagated-inputs '())
-     (native-inputs
-      `(("mescc-tools-seed" ,%mescc-tools-seed)
-        ("mes-source" ,(package-source mes-boot0))
-
-        ("bootstrap-mes" ,%bootstrap-mes)
-        ("coreutils" ,%bootstrap-coreutils&co)))
-     (build-system gnu-build-system)
-     (arguments
-      `(#:implicit-inputs? #f
-        #:guile ,%bootstrap-guile
-        #:strip-binaries? #f   ; binutil's strip b0rkes MesCC/M1/hex2 binaries
-        #:phases
-        (modify-phases %standard-phases
-          (add-after 'unpack 'unpack-seeds
-            (lambda* (#:key outputs #:allow-other-keys)
-              (let* ((coreutils (assoc-ref %build-inputs "coreutils"))
-                     (mescc-tools-seed (assoc-ref %build-inputs "mescc-tools-seed"))
-                     (mes-source (assoc-ref %build-inputs "mes-source"))
-                     (out (assoc-ref %outputs "out")))
-                (with-directory-excursion ".."
-                  (and
-                   (mkdir-p "mescc-tools-seed")
-                   (invoke "tar" "--strip=1" "-C" "mescc-tools-seed"
-                           "-xvf" mescc-tools-seed)
-                   (mkdir-p "mes-source")
-                   (invoke "tar" "--strip=1" "-C" "mes-source"
-                           "-xvf" mes-source)
-                  #t)))))
-          (replace 'configure
-            (lambda* (#:key outputs #:allow-other-keys)
-              (let ((coreutils (assoc-ref %build-inputs "coreutils"))
-                    (bootstrap-mes (assoc-ref %build-inputs "bootstrap-mes"))
-                    (out (assoc-ref %outputs "out")))
-                (setenv "PATH" (string-append coreutils "/bin"
-                                              ":" "../mescc-tools-seed"))
-                (format (current-error-port) "PATH=~s\n" (getenv "PATH"))
-                (setenv "PREFIX" out)
-                (setenv "MES_PREFIX" "../mes-source")
-                (setenv "MESCC_TOOLS_SEED" "../mescc-tools-seed")
-                (setenv "MES_SEED" (string-append bootstrap-mes "/lib"))
-                #t)))
-          (replace 'build
-            (lambda _
-              (invoke "sh" "build.sh")))
-          (replace 'check
-            (lambda _
-              ;; bootstrap build.sh lacks exec_enable, get_machine, and
-              ;; kaem_machine
-              ;; (invoke "sh" "check.sh")
-              #t))
-          (replace 'install
-            (lambda _
-              (invoke "sh" "install.sh")))))))))
-
 (define nyacc-boot
   (let ((version "0.86.0")
         (revision "0")
@@ -211,7 +138,7 @@
      (inputs '())
      (propagated-inputs '())
      (native-inputs
-      `(("mescc-tools" ,mescc-tools-boot)
+      `(("mescc-tools" ,%bootstrap-mescc-tools)
         ("nyacc-source" ,(package-source nyacc-boot))
 
         ("coreutils" , %bootstrap-coreutils&co)
@@ -284,7 +211,7 @@
                (setenv "MES_ARENA" "100000000")
                (setenv "DIFF" "sh scripts/diff.scm")
                ;; fail fast tests
-               ;; (invoke "sh" "-x" "build-aux/test.sh" "scaffold/tests/t")
+               (invoke "sh" "-x" "build-aux/test.sh" "scaffold/tests/t")
                ;; (invoke "sh" "-x" "build-aux/test.sh" "scaffold/tests/63-struct-cell")
                (invoke "sh" "check.sh"))))
           (replace 'install
@@ -332,7 +259,7 @@
        (propagated-inputs '())
        (native-inputs
         `(("mes" ,mes-boot)
-          ("mescc-tools" ,mescc-tools-boot)
+          ("mescc-tools" ,%bootstrap-mescc-tools)
           ("nyacc-source" ,(package-source nyacc-boot))
 
           ("coreutils" , %bootstrap-coreutils&co)
-- 
2.19.1


[-- Attachment #5: Type: text/plain, Size: 152 bytes --]


-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: Preparing the reduced bootstrap tarballs

[Jan Nieuwenhuizen]

Ludovic Courtès writes:

>> Ah right.  I saw that several times but did not use it.  I think because
>> initially it was of no use.  It would be nice if this built everything
>> we need, I agree :-)
>
> Heheh.  :-)  It’s what the “Building the Bootstrap Binaries” section
> describes.

Right...and I still forgot to add my new packages there...sigh.

>> Anyway, I think we/I will have to put some work into scripting
>> mescc-tools-seed or otherwise changing the mescc-tools-boot build.
>> WDYT?
>
> I’m confused: how did you build the seeds that (gnu packages bootstrap)
> refers to in ‘core-updates-next’?

I built them "by hand."  Mes used to have a `make seed' target.  I
dropped that.

> The goal is for the seeds to be built through Guix so we have a
> transparent and documented way to reproduce/verify them.

Sure.

> I could propose a patch to do that, though from what you’re saying
> generating ‘mescc-tools-seed’ is not something readily doable?

No, of course that's trivial.  I just sent patches to do that.  However,
building ascii seeds is not so trivial.  You need either M2-Planet or
Mes to produce those.  And then: ascii seeds is not what we want,
mescc-tools should be bootstrappable without any seeds, ascii or binary.

I don't see how to write a Guix packages to build mescc-tools without
any dependencies (apart from stage0 maybe).  It seems we still need a
mescc-tools-seed package anyway, which doesn't help?

janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: Preparing the reduced bootstrap tarballs

[Mark H Weaver]

ludo@gnu.org (Ludovic Courtès) writes:

> Jeremiah@pdp10.guru skribis:
>
>>> So if you like, please make that change.  There is only one little
>>> thing: I have no (scripted) recipe to create mescc-tools-seed-XYZ.  But
>>> wait: I have a great excuse for that...I was too lazy or too sloppy.
>>
>> I do, in mescc-tools-seed; the script bootstrap.sh when run with the
>> option "sin" will build the mescc-tools-seed binaries using mescc-tools.
>> The .M1 files are always generated by cc_x86.s using the C source files.
>
> I saw this script but it’s not entirely clear to me how to package the
> whole thing.  We don’t have a “stage0” package for instance in Guix, do
> we?
>
>>> WDYT?
>> I think we will end up having several versions of mescc-tools-seed; as
>> each architecture guix supports will end up needing a variant if we plan
>> on keeping them small. (I also have no idea how to make a multi-arch fat
>> elf binary)
>
> For now let’s focus on x86_64/i686.  :-)
>
> IMO we should change the seeds as rarely as possible because they are
> managed “out-of-band” and verifying them is difficult (you need to fetch
> the right Guix commit, run “guix build bootstrap-tarballs”, and compare
> the result—assuming this is all bit-reproducible.)
>
> The one we’re using today in Guix date back to 2013.

I think it's important that the new bootstrap-tarballs be
bit-reproducible, such that they can be independently verified by anyone
who wishes to do so.

In particular, *I* would like to independently verify them, on my own
laptops where I have avoided using binary substitutes for a long time,
and which I keep with me at all times.

My hope until now is that when we generated our existing bootstrap
binaries in 2013, Guix was too marginal a project to attract the
attention of hackers who might wish to compromise our bootstrap.  In
2018, as Guix has become more popular, we might well be considered a
worthy target of such efforts.

      Mark

Re: Preparing the reduced bootstrap tarballs

[Ludovic Courtès]

Hello,

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

>> IMO we should change the seeds as rarely as possible because they are
>> managed “out-of-band” and verifying them is difficult (you need to fetch
>> the right Guix commit, run “guix build bootstrap-tarballs”, and compare
>> the result—assuming this is all bit-reproducible.)
>>
>> The one we’re using today in Guix date back to 2013.
>
> Okay, I am proposing the attached patches, also available on my 
> core-updates-next at http://gitlab.com/janneke/guix
>
> Still I hope these won't last for five more years ;-)

Well I hope these will last for at least a couple of years, or at least
this is what we should aim for.

Replacing bootstrap binaries is not anecdotal as I wrote because these
are “binary blobs” that need a special procedure to verify them; that
they are smaller blobs is of course a crucial improvement, but it
doesn’t change the fact that one may want to verify them.

> From 5702107a31f52a615c516084b7a82d9f5e2967e0 Mon Sep 17 00:00:00 2001
> From: Jan Nieuwenhuizen <janneke@gnu.org>
> Date: Fri, 16 Nov 2018 19:29:55 +0100
> Subject: [PATCH 1/3] bootstrap: Add mescc-tools-static,
>  mescc-tools-static-tarball.
>
> * gnu/packages/make-bootstrap.scm (%mescc-tools-static,
> %mescc-tools-bootstrap-tarball):  New variable.

Neat; I’m glad it was as simple as this.  :-)

As discussed yesterday on IRC, please also make sure make-bootstrap.scm
does not refer to (gnu packages bootstrap) at all since that would be
like running in circles.

As discussed just now on IRC, we probably also need #:system
"i686-linux" in the appropriate places such that “guix build
bootstrap-tarballs” gives the same result on i686 and on x86_64.

> From cc3782c5f9d2c606a265be0dfdca44d79ec25622 Mon Sep 17 00:00:00 2001
> From: Jan Nieuwenhuizen <janneke@gnu.org>
> Date: Fri, 16 Nov 2018 21:30:00 +0100
> Subject: [PATCH 2/3] bootstrap: Add %bootstrap-mescc-tools.
>
> Built with
>     5702107a31f52a615c516084b7a82d9f5e2967e0 bootstrap: Add mescc-tools-static, mescc-tools-static-tarball.
>
> * gnu/packages/bootstrap.scm (%bootstrap-mescc-tools): New variable.

OK.

If you want, before pushing this patch, you can first push the one
above.  From there I (and possibly others!) will run “guix build
bootstrap-tarballs” and make sure we get the same result as you did.

> From daa9ff9463917687aabed8330eacb304a7e4deae Mon Sep 17 00:00:00 2001
> From: Jan Nieuwenhuizen <janneke@gnu.org>
> Date: Fri, 16 Nov 2018 21:30:28 +0100
> Subject: [PATCH 3/3] bootstrap: Replace %mescc-tools-seed with
>  %bootstrap-mescc-tools.
>
> * gnu/packages/bootstrap.scm (%mescc-tools-seed): Remove.
> (%bootstrap-inputs): Replace %mescc-tools-seed with %bootstrap-mescc-tools.
> * gnu/packages/commencement.scm (mes-boot): Likewise.

Also: (mescc-tools-boot): Remove.

> (tcc-boot0): Likewise.
> (mescc-tools-boot): Remove.

I suppose this is not directly related to the other changes, but anyway,
LGTM.

Thanks for the quick reply!

Ludo’.

Re: Preparing the reduced bootstrap tarballs

[Jeremiah]

> I think it's important that the new bootstrap-tarballs be
> bit-reproducible, such that they can be independently verified by anyone
> who wishes to do so.

Every Piece below M2-Planet has always been bit-reproducible.
In fact, each piece is designed in a way that you could by hand predict
what the resulting binary must be after any change.

and once I finally complete stage0; you would also have the blueprints
for making the virtual machine in hardware, hand toggle in the bits for
the hex0-monitor and have absolute proof that no trusting trust or Nexus
Intruder Class attacks have occurred in the creation of the binaries.

Every issue anyone is willing to bring, I will publicly address until
all bootstrap roots (even on arbitrary hardware) lead to the proof that
these binaries are perfectly reproducible and that they only behave in
the manner explicitly specified by the standards to which they conform.


> In particular, *I* would like to independently verify them, on my own
> laptops where I have avoided using binary substitutes for a long time,
> and which I keep with me at all times.
Already done; here are the steps currently for bootstrapping the
mescc-tools-seed and M2-Planet seed.M1:
git clone 'https://git.savannah.nongnu.org/git/stage0.git'
cd stage0
make test
cd ..
git clone 'https://git.savannah.nongnu.org/git/mescc-tools.git'
cd mescc-tools
make test
cd ..
git clone 'https://github.com/oriansj/mescc-tools-seed.git'
cd mescc-tools-seed
./bootstrap.sh sin

To generate the M2-Planet seed.M1 you need to either export
mescc-tools-seed's blood-elf, M1 and hex2 or mescc-tools (via copying
into your path or doing make install) then the steps to generate are as
follows:
git clone 'https://github.com/oriansj/M2-Planet.git'
cd M2-Planet
./bootstrap.sh refresh

Now you are done


> My hope until now is that when we generated our existing bootstrap
> binaries in 2013, Guix was too marginal a project to attract the
> attention of hackers who might wish to compromise our bootstrap.  In
> 2018, as Guix has become more popular, we might well be considered a
> worthy target of such efforts.

I like to go with the assumption that every binary is already
compromised; but by going back to the basics we can find and rip out
every single hook until we are finally secure.

I don't trust any hardware I can't or didn't make myself.
And the only root of trust we have is the ability to work as a
community, giving every member the ability to independently check our
assumptions and point out our mistakes. We will have false starts and
failures of imagination but we by working together will make us all a
dream that is too hard to achieve alone but easy now that we have each
other helping us all strive to a brighter future.

-Jeremiah

Re: Preparing the reduced bootstrap tarballs

[Mark H Weaver]

Hi Jeremiah,

Jeremiah@pdp10.guru writes:

>> I think it's important that the new bootstrap-tarballs be
>> bit-reproducible, such that they can be independently verified by anyone
>> who wishes to do so.
>
> Every Piece below M2-Planet has always been bit-reproducible.
> In fact, each piece is designed in a way that you could by hand predict
> what the resulting binary must be after any change.

Sure, and I'm glad to hear this.

However, my impression (correct me if I'm wrong) is that we are not yet
able to bootstrap Guix exclusively from M2-Planet.  For example, unless
I'm mistaken, we still need Guile in our bootstrap, and I'm guessing
that we are not yet able to build Guile exclusively from M2-Planet.
Is that right?

>> My hope until now is that when we generated our existing bootstrap
>> binaries in 2013, Guix was too marginal a project to attract the
>> attention of hackers who might wish to compromise our bootstrap.  In
>> 2018, as Guix has become more popular, we might well be considered a
>> worthy target of such efforts.
>
> I like to go with the assumption that every binary is already
> compromised;

I agree.  I didn't mean to imply that I consider our 2013 bootstrap
binaries trustworthy.  I very much look forward to the day when I
needn't blindly trust any precompiled binaries at all.

My only point is that if we cannot yet avoid blindly trusting
precompiled binaries, I have higher confidence in our 2013 binaries than
in binaries we would produce today, because (1) we are more likely to be
a target today because Guix has become far more popular, (2) I expect
that intelligence agencies have far more advanced tools today than they
did in 2013, and (3) I expect that governmental policies have become far
more favorable to permitting such attacks against projects such as ours.

> I don't trust any hardware I can't or didn't make myself.
> And the only root of trust we have is the ability to work as a
> community, giving every member the ability to independently check our
> assumptions and point out our mistakes. We will have false starts and
> failures of imagination but we by working together will make us all a
> dream that is too hard to achieve alone but easy now that we have each
> other helping us all strive to a brighter future.

I very much appreciate your work on this, which is quite important.

     Regards,
       Mark

Re: Preparing the reduced bootstrap tarballs

[Jan Nieuwenhuizen]

Ludovic Courtès writes:

Hi!

>> Still I hope these won't last for five more years ;-)
>
> Well I hope these will last for at least a couple of years, or at least
> this is what we should aim for.

Yes, we should aim for that while creating them.  I'm possibly a bit too
eager to replace the reduced binary seed bootstrap by a scheme only
bootstrap and reduce all of our seeds even further.

> Replacing bootstrap binaries is not anecdotal as I wrote because these
> are “binary blobs” that need a special procedure to verify them; that
> they are smaller blobs is of course a crucial improvement, but it
> doesn’t change the fact that one may want to verify them.

Fully agree.

> As discussed yesterday on IRC, please also make sure make-bootstrap.scm
> does not refer to (gnu packages bootstrap) at all since that would be
> like running in circles.

Thank you.  Right, that's fixed now.

> As discussed just now on IRC, we probably also need #:system
> "i686-linux" in the appropriate places such that “guix build
> bootstrap-tarballs” gives the same result on i686 and on x86_64.

I'm looking into that right now.

>> From cc3782c5f9d2c606a265be0dfdca44d79ec25622 Mon Sep 17 00:00:00 2001
>> From: Jan Nieuwenhuizen <janneke@gnu.org>
>> Date: Fri, 16 Nov 2018 21:30:00 +0100
>> Subject: [PATCH 2/3] bootstrap: Add %bootstrap-mescc-tools.
>>
>> Built with
>>     5702107a31f52a615c516084b7a82d9f5e2967e0 bootstrap: Add mescc-tools-static, mescc-tools-static-tarball.
>>
>> * gnu/packages/bootstrap.scm (%bootstrap-mescc-tools): New variable.
>
> OK.
>
> If you want, before pushing this patch, you can first push the one
> above.  From there I (and possibly others!) will run “guix build
> bootstrap-tarballs” and make sure we get the same result as you did.

Great, done.  I missed

    5b01b6034aeab32a5011c5757f18bd9772d3497d python: Honor '--cores=...' in tests.

happening yesterday; so pushed as

    fae3c0d953a7b0b552b4ce654ed9b5a51186f491
    bootstrap: Add mescc-tools-static, mescc-tools-static-tarball.

Note you'll also need something like these three extra commits

--8<---------------cut here---------------start------------->8---
2d8f7e2ac bootstrap: Update %bootstrap-tarballs.
57fbf077a bootstrap: Replace %mescc-tools-seed with %bootstrap-mescc-tools.
706574ee4 bootstrap: Add %bootstrap-mescc-tools.
--8<---------------cut here---------------end--------------->8---

from core-updates-next @ https://gitlab.com/janneke/guix to be able to
build `guix --system=i686-linux bootstrap-tarballs'.  Until then, the
recipe is

--8<---------------cut here---------------start------------->8---
./pre-inst-env guix build --system=i686-linux mes-stripped-tarball mescc-tools-static-tarball
--8<---------------cut here---------------end--------------->8---

Rebuilding that right now, let's compare what we get.

Thanks for all your input!
janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: Preparing the reduced bootstrap tarballs

[Jan Nieuwenhuizen]

Jan Nieuwenhuizen writes:

>> If you want, before pushing this patch, you can first push the one
>> above.  From there I (and possibly others!) will run “guix build
>> bootstrap-tarballs” and make sure we get the same result as you did.
>
> Great, done.  I missed
>
>     5b01b6034aeab32a5011c5757f18bd9772d3497d python: Honor '--cores=...' in tests.
>
> happening yesterday; so pushed as
>
>     fae3c0d953a7b0b552b4ce654ed9b5a51186f491
>     bootstrap: Add mescc-tools-static, mescc-tools-static-tarball.
>
> Note you'll also need something like these three extra commits
>
> 2d8f7e2ac bootstrap: Update %bootstrap-tarballs.
> 57fbf077a bootstrap: Replace %mescc-tools-seed with %bootstrap-mescc-tools.
> 706574ee4 bootstrap: Add %bootstrap-mescc-tools.
>
>
> from core-updates-next @ https://gitlab.com/janneke/guix to be able to
> build `guix --system=i686-linux bootstrap-tarballs'.  Until then, the
> recipe is
>
> ./pre-inst-env guix build --system=i686-linux mes-stripped-tarball mescc-tools-static-tarball
>
> Rebuilding that right now, let's compare what we get.

Just to give you a heads up: left to build this last night to find that
the python patch does not build for me.  I reverted the python patch on
my personal branch and start a new build.

Not sure what to do about this, YMMV but this is what I get

--8<---------------cut here---------------start------------->8---
building /gnu/store/k3h2iyv4b36a3d5ji7azgi8bcd8czyrz-python-minimal-3.7.0.drv...
[..]
-l and -j don't go together!
make: *** [Makefile:878: test] Error 2

Test suite failed, dumping logs.
Backtrace:
           4 (primitive-load "/gnu/store/9lz3rpgyrm1y97g6k9qg0f7a1h6?")
In ice-9/eval.scm:
   191:35  3 (_ _)
In srfi/srfi-1.scm:
   863:16  2 (every1 #<procedure 80fce20 at /gnu/store/4nhwwas62jab?> ?)
In /gnu/store/4nhwwas62jabgi8lmhimpg4v3fk2ynk1-module-import/guix/build/gnu-build-system.scm:
   799:28  1 (_ _)
    369:6  0 (check #:target _ #:make-flags _ #:tests? _ # _ # _ # _)

/gnu/store/4nhwwas62jabgi8lmhimpg4v3fk2ynk1-module-import/guix/build/gnu-build-system.scm:369:6: In procedure check:
Throw to key `srfi-34' with args `(#<condition &invoke-error [program: "make" arguments: ("test" "-j" "16" "EXTRATESTOPTS=-j16") exit-status: 2 term-signal: #f stop-signal: #f] 822ea20>)'.
builder for `/gnu/store/llvsk2pszk1cc7qwf6azi9877nhks59f-python2-2.7.15.drv' failed with exit code 1
build of /gnu/store/llvsk2pszk1cc7qwf6azi9877nhks59f-python2-2.7.15.drv failed
--8<---------------cut here---------------end--------------->8---

janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: Preparing the reduced bootstrap tarballs

[Jeremiah]

> However, my impression (correct me if I'm wrong) is that we are not yet
> able to bootstrap Guix exclusively from M2-Planet.
That is correct as the step of bootstrapping MesCC from M2-Planet is not
yet complete.
However once that is done, we can leverage Mes.c and gash to complete
the bootstrap of guix from that trusted reproducible source in a
reproducible fashion.

> For example, unless
> I'm mistaken, we still need Guile in our bootstrap, and I'm guessing
> that we are not yet able to build Guile exclusively from M2-Planet.
> Is that right?
We don't need it, so much as it is people wishing to avoid tedious work.
We already can bootstrap kaem without any shells or interpreters and it
can be used to run shell scripts that can perform the rest of the
bootstrap of a lisp or a proper shell.

I think because that work is less of a technical challenge that it has
been skipped.


> My only point is that if we cannot yet avoid blindly trusting
> precompiled binaries,

Depends on how restricted of an environment you ware willing to work in

> I have higher confidence in our 2013 binaries than
> in binaries we would produce today, because (1) we are more likely to be
> a target today because Guix has become far more popular, (2) I expect
> that intelligence agencies have far more advanced tools today than they
> did in 2013, and (3) I expect that governmental policies have become far
> more favorable to permitting such attacks against projects such as ours.

1) Granted
2) Not exactly; simply because the most advanced attack tool ever
invented was the Nexus Intruder Program in 1958. (Hardware that subverts
software that later subverts hardware designs and more software
[firmware, microcode, etc]). The tools might get more expensive but the
actual quality of attack tools depends on the teams and the market's
demand for pumping out vulnerable products and bugs. (Like the recent
Hard drive firmware attack which leveraged the vendor's cost cutting
process to hijack the drives and then lock out future attempts at
recovery.
3) Actually Government agencies are depending more and more on "Open
source tools" (Their words not mine) as software budgets have gotten
tighter and third party vendors integrate them more and more into their
commercial offerings purchased by Goverment agencies. Putting a backdoor
in the software most Government agencies depend upon, invites
vulnerabilities in our own Intelligence Agencies infrastructure and
increase the probablity that Spies will be identified before their
flight to their target country leaves the ground. To do such would not
only be suicidal for those Intelligence Agencies but also ensure
Cyberwarfare against the Countries they work for that much more
effective.

Now that isn't to say they consider that an extranality and doom us all
but nothing stays hidden when we can read the source and can DDC our
entire bootstrap across arbitrary hardware/operating system
combinations.

-Jeremiah

Re: Preparing the reduced bootstrap tarballs

[Mark H Weaver]

Hi,

Jeremiah@pdp10.guru writes:

>> However, my impression (correct me if I'm wrong) is that we are not yet
>> able to bootstrap Guix exclusively from M2-Planet.
> That is correct as the step of bootstrapping MesCC from M2-Planet is not
> yet complete.
> However once that is done, we can leverage Mes.c and gash to complete
> the bootstrap of guix from that trusted reproducible source in a
> reproducible fashion.
>
>> For example, unless
>> I'm mistaken, we still need Guile in our bootstrap, and I'm guessing
>> that we are not yet able to build Guile exclusively from M2-Planet.
>> Is that right?
> We don't need it, so much as it is people wishing to avoid tedious work.
> We already can bootstrap kaem without any shells or interpreters and it
> can be used to run shell scripts that can perform the rest of the
> bootstrap of a lisp or a proper shell.
>
> I think because that work is less of a technical challenge that it has
> been skipped.

I get the feeling that we are talking about two different things here.
I'm not saying that Guile is needed to bootstrap a GNU system from
source by hand.

When I say "we still need Guile in our bootstrap", I'm referring
specifically to the Guix "bootstrap binaries".  Guix uses Guile to
execute its automated build scripts.

Anyway, I've answered my own question by looking at:

  https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/make-bootstrap.scm?h=core-updates-next&id=d9ed0a46301e259c3082fcc67678e72ceae19a2d#n778

which indicates to me that although gcc, binutils, and glibc have been
eliminated from the Guix bootstrap binaries, they still include
statically-linked versions of Guile, Coreutils, Bzip2, XZ, Gawk, Tar,
and Grep.  These are the remaining precompiled binaries that I'm worried
about.

>> I have higher confidence in our 2013 binaries than
>> in binaries we would produce today, because (1) we are more likely to be
>> a target today because Guix has become far more popular, (2) I expect
>> that intelligence agencies have far more advanced tools today than they
>> did in 2013, and (3) I expect that governmental policies have become far
>> more favorable to permitting such attacks against projects such as ours.
>
> 1) Granted
> 2) Not exactly; simply because the most advanced attack tool ever
> invented was the Nexus Intruder Program in 1958. (Hardware that subverts
> software that later subverts hardware designs and more software
> [firmware, microcode, etc]). The tools might get more expensive but the
> actual quality of attack tools depends on the teams and the market's
> demand for pumping out vulnerable products and bugs. (Like the recent
> Hard drive firmware attack which leveraged the vendor's cost cutting
> process to hijack the drives and then lock out future attempts at
> recovery.

Interesting.  I didn't know about the Nexus Intruder Program, but even
so, I don't think this invalidates my point (2).

> 3) Actually Government agencies are depending more and more on "Open
> source tools" (Their words not mine) as software budgets have gotten
> tighter and third party vendors integrate them more and more into their
> commercial offerings purchased by Goverment agencies. Putting a backdoor
> in the software most Government agencies depend upon, invites
> vulnerabilities in our own Intelligence Agencies infrastructure and
> increase the probablity that Spies will be identified before their
> flight to their target country leaves the ground. To do such would not
> only be suicidal for those Intelligence Agencies but also ensure
> Cyberwarfare against the Countries they work for that much more
> effective.
>
> Now that isn't to say they consider that an extranality and doom us all

Right.  It is by now well accepted fact that the NSA has deliberately
weakened cryptographic standards, for example, and your argument above
would seem to argue against the truth of that fact as well.

> but nothing stays hidden when we can read the source and can DDC our
> entire bootstrap across arbitrary hardware/operating system
> combinations.

Here I disagree.  DDC is a crucially important step, but even after we
have DDC, a large number of security flaws will continue to stay hidden
in our code.  We have far too much code to audit, and too much of it
written in C, which makes it quite difficult to write secure code.
Common classes of security bugs in C such as signed integer overflows
and other undefined behavior can be very hard to find, and I have little
confidence that all of these bugs would be found in any practical audit.

Moreover, our current development practices ensure that new bugs of this
kind will be continuously reintroduced into our software, even as we
discover and fix many of the existing bugs.

So, your work is a very important part, but not a complete solution to
the problem of building a computer system that we can truly consider
trustworthy.

     Regards,
       Mark

Re: Preparing the reduced bootstrap tarballs

[Jan Nieuwenhuizen]

Mark H Weaver writes:

> Anyway, I've answered my own question by looking at:
>
>   https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/make-bootstrap.scm?h=core-updates-next&id=d9ed0a46301e259c3082fcc67678e72ceae19a2d#n778
>
> which indicates to me that although gcc, binutils, and glibc have been
> eliminated from the Guix bootstrap binaries, they still include
> statically-linked versions of Guile, Coreutils, Bzip2, XZ, Gawk, Tar,
> and Grep.  These are the remaining precompiled binaries that I'm worried
> about.

As far as I'm concerned our next step after the Reduced Binary Seed
bootstrap--that is yet to be released from core-updates-next---is the
Scheme-only bootstrap.

The plan is to have Guile and Gash initially replace all coreutils&co
and build them from source early in the bootstrap phase, right after
having built tcc and GNU make.  We have managed to build GNU make
without any dependency on any of the tool you mention (except for
Guile).

We're not there yet, gash cannot build mes or tcc.  Also, the latest mes
0.18 release cannot support buillding bash yet.  Lots of fun hacking
todo!

A next step could be to replace bootstrap-guile with mes, or create a
real small version of bootstrappable Guile.

Greetings,
janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: Preparing the reduced bootstrap tarballs

[Giovanni Biscuolo]

[-- Attachment #1: Type: text/plain, Size: 1514 bytes --]

Hi Jeremiah,

Jeremiah@pdp10.guru writes:

[...]

> and once I finally complete stage0; you would also have the blueprints
> for making the virtual machine in hardware,

so, if I don't get it wrong, every skilled engineer will be able to
build an "almost analogic" (zero bit of software preloaded) computing
machine ad use stage0/mes [1] as the "metre" [2] to calibrate all other
computing machines (thanks to reproducible builds)?

> hand toggle in the bits for the hex0-monitor

the first bit of code have to be "manually" introduced in the machine,
right?
for the lazyer like me, what about a punched card? :-) 

> and have absolute proof that no trusting trust or Nexus
> Intruder Class attacks have occurred in the creation of the binaries.

I didn't know about Nexus Intruder attacks: could you please give me
some links to the relevant bibliography?

> Every issue anyone is willing to bring, I will publicly address until
> all bootstrap roots (even on arbitrary hardware) lead to the proof that
> these binaries are perfectly reproducible and that they only behave in
> the manner explicitly specified by the standards to which they
> conform.

so, having the scientific proof that binary conforms to source, there
will be noo need to trust (the untrastable)

thank you!

Ciao
Giovanni

[2] I still have not fully understood the relationship between stage0
and mes
[1] https://en.m.wikipedia.org/wiki/Metre

-- 
Giovanni Biscuolo

Xelera IT Infrastructures

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Preparing the reduced bootstrap tarballs

[jeremiah]

> so, if I don't get it wrong, every skilled engineer will be able to
> build an "almost analogic" (zero bit of software preloaded) computing
> machine ad use stage0/mes [1] as the "metre" [2] to calibrate all other
> computing machines (thanks to reproducible builds)?

well, I haven't thought of it in those terms but yes I guess that is one
of the properties of the plan.

> the first bit of code have to be "manually" introduced in the machine,
> right?
Correct, otherwise you'll have to deal with firmware/bios as a trust
vector to be concerned about.

> for the lazyer like me, what about a punched card? :-)
If someone is willing to figure out how to read a deck of punched cards
without software, I'd be interested in learning more.

> I didn't know about Nexus Intruder attacks: could you please give me
> some links to the relevant bibliography?
I'll see if I can dig those up for you.

> so, having the scientific proof that binary conforms to source, there
> will be noo need to trust (the untrastable)
Well, that is what someone else could do with it but not a direct goal
of the work.

-Jeremiah

Re: Preparing the reduced bootstrap tarballs

[Ricardo Wurmus]

jeremiah@pdp10.guru writes:

>> for the lazyer like me, what about a punched card? :-)
> If someone is willing to figure out how to read a deck of punched cards
> without software, I'd be interested in learning more.

Now that’s a project for a free hardware design that I’d happily work
on :)

Reading and transmitting of the bitstream can be done in hardware alone
(shift register, MUX, photoresistors, …), though I’d need to know what
kind of interface there is on the receiving side.  Just flash memory /
TTL shift register?

--
Ricardo

Re: Preparing the reduced bootstrap tarballs

[Timothy Sample]

Hi Jan,

Jan Nieuwenhuizen <janneke@gnu.org> writes:

> The plan is to have Guile and Gash initially replace all coreutils&co
> and build them from source early in the bootstrap phase, right after
> having built tcc and GNU make.  We have managed to build GNU make
> without any dependency on any of the tool you mention (except for
> Guile).
>
> We're not there yet, gash cannot build mes or tcc.  Also, the latest mes
> 0.18 release cannot support buillding bash yet.  Lots of fun hacking
> todo!

I wanted to let you know that I’ve been doing more work on the little
Shell backend that we were talking about earlier.  It’s kind of like the
worse-is-better version of Gash: it is certainly not as nice, but the
development pace is much faster.  I did take a look at just working on
Gash directly, but it looked like it was going to be a lot slower.

So, keeping in mind that software estimates are very unreliable, I would
say that I might have a workable interpreter in the next week or two.
There are three big features missing: globbing, asynchronous commands,
and arithmetic substitutions.  I am almost finished globbing,
asynchronous commands should be pretty easy, and I plan to leave
arithmetic substitutions on the road-map for as long as possible (it’s a
little bit boring).

After I get those first two features working, I am going to start
testing running build scripts for Bash.  I will contact you then in case
you have any new advice on which scripts are important or anything else.

> Greetings,
> janneke


-- Tim

Re: Preparing the reduced bootstrap tarballs

[Jeremiah]

> Now that’s a project for a free hardware design that I’d happily work
> on :)
Sweet

> Reading and transmitting of the bitstream can be done in hardware alone
> (shift register, MUX, photoresistors, …), though I’d need to know what
> kind of interface there is on the receiving side.  Just flash memory /
> TTL shift register?
Well ultimately that would need to be a memory bus that is connected to
their TTL DRAM (64KB chips) or TTL SRAM (so we wouldn't have to worry
about memory refresh or other sort of issues like that)
and setting the memory address block starting from Zero until end of
deck.
The Memory is Big Endian Bit, Byte and Word Addressed.

The number address pins needs to be atleast 9 (otherwise the bootstrap
binaries would exceed the address space the reader could write to) and
not to exceed 64 (As I don't plan on implementing a 128bit ALU with
Floating point and Vector instructions needed to comply with the 128bit
version of the spec). I am planning on having the memory data bus to be
8bits wide as all instructions and datatypes are multiples of 8bits.

I can provide a clock signal for when the next value is to be loaded or
a line for you to directly drive the memory chips.

Or we could go the uart route and simply leverage a classic serial
connector.


> I?ve thought about this a bit.  Would it be possible to go even more
> mechanical, and implement the SPI protocol in switches toggled by holes
> cut in some kind of insulating tape?
Well that mechanical aspect would be what you would expect in punched
cards or paper tape (I was planning on doing a paper tape reader to
allow easier manual inspection of outputs, inputs and binaries)

> You could have a couple of steel
> balls (bearings?) balanced on top of each other, connected to a circuit
> using brushes or springs.  Then, you place the tape in between the
> bearings.  When there?s a hole in the insulating tape, they make
> contact, and when there is no hole, no contact.
Yep, classical through-hole reader technology. Not as reliable as
optical sensors but much more low tech and no room for anyone to insert
anything funny with the sensors.


> Then, you have a row of
> holes for the SPI clock, and a row of holes for the SPI data.  All you
> would have to do is hook up the right voltage to it, and you could write
> to a flash ROM by pulling manually punched tape through it.  (You could
> use a crank or motor if the timing needs to be more precise.)
Well you could have a hole aligned with the values being read (say a 9th
bit) which would trigger a clock pulse to indicate reading of the next byte.


> Then, you could take an x200 and setup the bootrom to be whatever you
> wanted with no software whatsoever.

Not possible, there is a 700+KB microcode blob in that CPU, without
which the CPU fails to power on for more than 10 clock cycles.

> If you adjusted stage0 to set up a
> ?cache-as-RAM? environment (like they do in coreboot), you could do
> quite a bit of manoeuvring before having to initialize RAM.
Well the 4MB of space (or 6MB if you have Penryn chip) is more than
enough to bootstrap cc_x86 and be able to compile a self-hosting C compiler


> That is,
> you could probably set up a tiny Scheme environment and do more hardware
> initialization from there.

Well the hardware initialization isn't a concern as that is currently
well documented in libreboot's code. The hard part is those computers
are only going to be working for a few more years before tin whiskers
kill the whole lot of them (the transistion to lead-free solder occurred
during that time) and then we will have a working solution to hardware
that is no longer available, no longer works and we can't build more
that will exactly match those specs without considerable effort.

> Obviously all of this would be easier on a simpler computer, but then
> presumably you have to move from a simpler computer to a more
> complicated computer at some point (depending on your needs).
I assume a 64bit processor with 16+GB of memory, a SATA interface and a
network interface should be sufficiently complicated to bootstrap
everything we will need to conern ourselves with in the next couple
years. (We can always enhance a design of our own making)

The Low end will be a 16bit processor with only 64KB of RAM and will
probably require ugly coding to fit some of the programs in that address size.

> The nice
> thing about booting the x200 as a simple computer is that you can just
> expand its boundaries until it is full-featured.
One needs to be precise by what one means by full-featured. Would you
consider Minix or Linux 1.0 a full-featured kernel? Would the ability to
build GCC and guile be considered full-featured enough? Because I can
build productive environments down to a hex0-monitor in a few hundred
bytes of total memory.

> Sorry for derailing a bit, I just wanted to share some idle thoughts.
I actively encourage alternative perspectives and I love being told
how I can do this better. I look forward to more ideas and suggestions
from you on this subject later ^_^

Just an open reminder our #bootstrappable channel is always looking for
people intersted in these sorts of topics and we love hearing about what
you have created in this regard.

-Jeremiah

Preparing the reduced bootstrap tarballs, take 2

[Jan Nieuwenhuizen]

[-- Attachment #1: Type: text/plain, Size: 1168 bytes --]

Hello!

Meanwhile, I have updated core-updates-next to include

    d9ed0a463 bootstrap: Update %bootstrap-tarballs.
    4fd4efc81 bootstrap: Replace %mescc-tools-seed with %bootstrap-mescc-tools.
    562c37c7b bootstrap: Update %bootstrap-mes.
    b00a95be2 bootstrap: Add %bootstrap-mescc-tools.

With these, I can build

   ./pre-inst-env guix build hello
   ./pre-inst-env guix build --system=i686-linux hello
   ./pre-inst-env guix build --system=i686-linux bootstrap-tarballs

If these reproduce, we're getting real close.  However...

On my core-updates-next at gitlab I also have (see attached)

    5c2209ad4 bootstrap: Force i686-linux for bootstrap-tarballs.

which tries to support

   ./pre-inst-env guix build bootstrap-tarballs

which fails when attempting to build gcc-cross-sans-libc-i686-unknown-linux-gnu-5.5.0

    /tmp/guix-build-gcc-cross-sans-libc-i686-unknown-linux-gnu-5.5.0.drv-0/build/./gcc/cc1: error while loading shared libraries: libisl.so.15: cannot open shared object file: No such file or directory

I do not understand the dependency chain of why exactly this is being
rebuilt and what %current-system is, or why it fails.

janneke


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-bootstrap-Force-i686-linux-for-bootstrap-tarballs.patch --]
[-- Type: text/x-patch, Size: 2406 bytes --]

From 5c2209ad4edd198f600dbf1fef58d434cd3bd92a Mon Sep 17 00:00:00 2001
From: Jan Nieuwenhuizen <janneke@gnu.org>
Date: Tue, 20 Nov 2018 08:12:09 +0100
Subject: [PATCH] bootstrap: Force i686-linux for bootstrap-tarballs.

* gnu/packages/make-bootstrap.scm (%mescc-tools-static): Force i686-linux for
bootstrap-tarballs.
(%mes-minimal): Likewise.
* gnu/packages/commencement.scm (mes-boot): Support strict i686-linux build on
x86_64.
---
 gnu/packages/commencement.scm   |  2 ++
 gnu/packages/make-bootstrap.scm | 10 ++++++----
 2 files changed, 8 insertions(+), 4 deletions(-)

diff --git a/gnu/packages/commencement.scm b/gnu/packages/commencement.scm
index ed8a429c2..3d3f8e665 100644
--- a/gnu/packages/commencement.scm
+++ b/gnu/packages/commencement.scm
@@ -174,6 +174,8 @@
                 (symlink (string-append "../nyacc-source/module") "nyacc")
                 (setenv "GUILE_LOAD_PATH" "nyacc")
                 (setenv "GUILE_TOOLS" "true") ; no tools in bootstrap-guile
+                (substitute* "configure.sh"
+                  (("^arch=.*") "arch=i686\n"))
                 (invoke "bash" "configure.sh"
                         (string-append "--prefix=" out))
                 (setenv "MES" "src/mes")
diff --git a/gnu/packages/make-bootstrap.scm b/gnu/packages/make-bootstrap.scm
index fa3914cde..d75289b45 100644
--- a/gnu/packages/make-bootstrap.scm
+++ b/gnu/packages/make-bootstrap.scm
@@ -542,9 +542,10 @@ for `sh' in $PATH, and without nscd, and with static NSS modules."
     (inherit mescc-tools)
     (name "mescc-tools-static")
     (arguments
-     (substitute-keyword-arguments (package-arguments mescc-tools)
-       ((#:make-flags flags)
-        `(cons "CC=gcc -static" ,flags))))))
+     `(#:system "i686-linux"
+       ,@(substitute-keyword-arguments (package-arguments mescc-tools)
+           ((#:make-flags flags)
+            `(cons "CC=gcc -static" ,flags)))))))
 
 (define %mes-minimal
   ;; A minimal Mes without documentation dependencies, for bootstrap.
@@ -562,7 +563,8 @@ for `sh' in $PATH, and without nscd, and with static NSS modules."
                  ("i686-linux-gcc" ,(cross-gcc triplet)))
                '())))
       (arguments
-       `(#:strip-binaries? #f
+       `(#:system "i686-linux"
+         #:strip-binaries? #f
          #:phases
          (modify-phases %standard-phases
            (add-before 'configure 'optional-dot
-- 
2.19.1


[-- Attachment #3: Type: text/plain, Size: 152 bytes --]


-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: Preparing the reduced bootstrap tarballs

[Jan Nieuwenhuizen]

Timothy Sample writes:

Hello Timothy!

>> The plan is to have Guile and Gash initially replace all coreutils&co
>> and build them from source early in the bootstrap phase, right after
>> having built tcc and GNU make.  We have managed to build GNU make
>> without any dependency on any of the tool you mention (except for
>> Guile).
>>
>> We're not there yet, gash cannot build mes or tcc.  Also, the latest mes
>> 0.18 release cannot support buillding bash yet.  Lots of fun hacking
>> todo!
>
> I wanted to let you know that I’ve been doing more work on the little
> Shell backend that we were talking about earlier.  It’s kind of like the
> worse-is-better version of Gash: it is certainly not as nice, but the
> development pace is much faster.  I did take a look at just working on
> Gash directly, but it looked like it was going to be a lot slower.

Oh, that's great news!  We have been making good progress on Gash.  The
parser has been completely revamped.  The next target is to add a
transformation that should result in a eval'able shelly Guile script,
ideally something that you'd want to write by hand.

To achieve that will take some time.  It would be nice if we could work
together shaping this shelly script language .

If Gash and Geesh would produce (after transformation) the same shelly
script language, we can also share the backend implementation, WDYT?

> So, keeping in mind that software estimates are very unreliable, I would
> say that I might have a workable interpreter in the next week or two.
> There are three big features missing: globbing, asynchronous commands,
> and arithmetic substitutions.  I am almost finished globbing,
> asynchronous commands should be pretty easy, and I plan to leave
> arithmetic substitutions on the road-map for as long as possible (it’s a
> little bit boring).

Hehe.  I had a look and you did make lots of progress!

> After I get those first two features working, I am going to start
> testing running build scripts for Bash.  I will contact you then in case
> you have any new advice on which scripts are important or anything else.

Great,
Keep up the good work!

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Preparing the reduced bootstrap tarballs, take 3

[Jan Nieuwenhuizen]

[-- Attachment #1: Type: text/plain, Size: 3383 bytes --]

Jan Nieuwenhuizen writes:

Hello again!

The last problem left for me before we can verify the new bootstrap
tarballs and move them to ftp.gnu.org:guix was for

   ./pre-inst-env guix bootstrap-tarballs

to succeed.  We needed to force #:system "i686-linux" for
mescc-tools-static and mes-minimal.  This triggered a build of
gcc-cross-sans-libc-i686-unknown-linux-gnu, that did not succeed.

I found that the mes-minimal recipe used #:system "i686-linux" and on
x86_64-linux still depended on a i686-linux cross compiler.  With that
removed, I now have for you the tarballs as I produced them.

I have updated core-updates-next with this fix, a documentation update
and to also include Marius' mes bootstrap cleanups from his wip-gcc7
branch.

So, finally for core-updates-next 4ae7dc7b9af64794081b1913740b97acd89c91bc
here are my lucky numbers

--8<---------------cut here---------------start------------->8---
08:21:04 janneke@dundal:~/src/guix-boot 
$ for i in $(./pre-inst-env guix build --system=i686-linux bootstrap-tarballs)/*; do echo $i;guix hash $i; done
/gnu/store/i8rrdivr5ab56kqwbasigp41wqpj1pyp-bootstrap-tarballs-0/guile-static-stripped-2.2.4-i686-linux.tar.xz
17p28xv0y1q729idvf2k1dpcfd469akxm8lnf7gd5a7hmnrb28l5
/gnu/store/i8rrdivr5ab56kqwbasigp41wqpj1pyp-bootstrap-tarballs-0/linux-libre-headers-stripped-4.14.67-i686-linux.tar.xz
0sm2z9x4wk45bh6qfs94p0w1d6hsy6dqx9sw38qsqbvxwa1qzk8s
/gnu/store/i8rrdivr5ab56kqwbasigp41wqpj1pyp-bootstrap-tarballs-0/mescc-tools-static-0.5.2-0.bb062b0-i686-linux.tar.xz
11lniw0vg61kmyhvnwkmcnkci9ym6hbmiksiqggd0hkipbq7hvlz
/gnu/store/i8rrdivr5ab56kqwbasigp41wqpj1pyp-bootstrap-tarballs-0/mes-minimal-stripped-0.18-0.08f04f5-i686-linux.tar.xz
0qwpby91hp6afmg5ibdrrk3fw85zxdazfk7rhrdsihsfzqwmfhfx
/gnu/store/i8rrdivr5ab56kqwbasigp41wqpj1pyp-bootstrap-tarballs-0/static-binaries-0-i686-linux.tar.xz
1rs8qh0n6zxmhs0h1b02sydi4vn09qngf3spfpfynkm2n0bxs368
--8<---------------cut here---------------end--------------->8---

--8<---------------cut here---------------start------------->8---
08:21:10 janneke@dundal:~/src/guix-boot 
$ for i in $(./pre-inst-env guix build bootstrap-tarballs)/*; do echo $i;guix hash $i; done
/gnu/store/n8lrszv7qf0rwqhibm57d8jykj253bw2-bootstrap-tarballs-0/guile-static-stripped-2.2.4-x86_64-linux.tar.xz
1285hlbfahp414hy3jjdghxjfgi94aj7y3fw0pz7z27421b7z2f9
/gnu/store/n8lrszv7qf0rwqhibm57d8jykj253bw2-bootstrap-tarballs-0/linux-libre-headers-stripped-4.14.67-x86_64-linux.tar.xz
0sm2z9x4wk45bh6qfs94p0w1d6hsy6dqx9sw38qsqbvxwa1qzk8s
/gnu/store/n8lrszv7qf0rwqhibm57d8jykj253bw2-bootstrap-tarballs-0/mescc-tools-static-0.5.2-0.bb062b0-x86_64-linux.tar.xz
11lniw0vg61kmyhvnwkmcnkci9ym6hbmiksiqggd0hkipbq7hvlz
/gnu/store/n8lrszv7qf0rwqhibm57d8jykj253bw2-bootstrap-tarballs-0/mes-minimal-stripped-0.18-0.08f04f5-x86_64-linux.tar.xz
0qwpby91hp6afmg5ibdrrk3fw85zxdazfk7rhrdsihsfzqwmfhfx
/gnu/store/n8lrszv7qf0rwqhibm57d8jykj253bw2-bootstrap-tarballs-0/static-binaries-0-x86_64-linux.tar.xz
0bqyjdbas67wgfwv4rcmr5a5b31l8kk8z6gsidxay7liih5rp5hn
--8<---------------cut here---------------end--------------->8---

We may want to change the bootstrap downloads for x86_64 in
bootstrap.scm, mescc-tools-static and mes-minimal-stripped and
linux-libre-headers-stripped to use the x86_64-linux variants (currently
we use the i686-linux tarballs); but as you can see their content is
identical.

Happy verifying ;-)

janneke


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-bootstrap-Force-i686-linux-for-bootstrap-tarballs.patch --]
[-- Type: text/x-patch, Size: 2982 bytes --]

From da91723cc5bba433f59ac40ee669d97e4cbc0e45 Mon Sep 17 00:00:00 2001
From: Jan Nieuwenhuizen <janneke@gnu.org>
Date: Tue, 20 Nov 2018 08:12:09 +0100
Subject: [PATCH] bootstrap: Force i686-linux for bootstrap-tarballs.

* gnu/packages/make-bootstrap.scm (%mescc-tools-static): Force i686-linux for
bootstrap-tarballs.
(%mes-minimal): Likewise.  Remove i686-linux cross-compiler dependency.
* gnu/packages/commencement.scm (mes-boot): Support strict i686-linux build on
x86_64.
---
 gnu/packages/commencement.scm   |  2 ++
 gnu/packages/make-bootstrap.scm | 19 +++++++------------
 2 files changed, 9 insertions(+), 12 deletions(-)

diff --git a/gnu/packages/commencement.scm b/gnu/packages/commencement.scm
index ed8a429c23..3d3f8e6650 100644
--- a/gnu/packages/commencement.scm
+++ b/gnu/packages/commencement.scm
@@ -174,6 +174,8 @@
                 (symlink (string-append "../nyacc-source/module") "nyacc")
                 (setenv "GUILE_LOAD_PATH" "nyacc")
                 (setenv "GUILE_TOOLS" "true") ; no tools in bootstrap-guile
+                (substitute* "configure.sh"
+                  (("^arch=.*") "arch=i686\n"))
                 (invoke "bash" "configure.sh"
                         (string-append "--prefix=" out))
                 (setenv "MES" "src/mes")
diff --git a/gnu/packages/make-bootstrap.scm b/gnu/packages/make-bootstrap.scm
index fa3914cde7..761420af67 100644
--- a/gnu/packages/make-bootstrap.scm
+++ b/gnu/packages/make-bootstrap.scm
@@ -542,9 +542,10 @@ for `sh' in $PATH, and without nscd, and with static NSS modules."
     (inherit mescc-tools)
     (name "mescc-tools-static")
     (arguments
-     (substitute-keyword-arguments (package-arguments mescc-tools)
-       ((#:make-flags flags)
-        `(cons "CC=gcc -static" ,flags))))))
+     `(#:system "i686-linux"
+       ,@(substitute-keyword-arguments (package-arguments mescc-tools)
+           ((#:make-flags flags)
+            `(cons "CC=gcc -static" ,flags)))))))
 
 (define %mes-minimal
   ;; A minimal Mes without documentation dependencies, for bootstrap.
@@ -553,16 +554,10 @@ for `sh' in $PATH, and without nscd, and with static NSS modules."
       (inherit mes)
       (name "mes-minimal")
       (native-inputs
-       `(("guile" ,guile-2.2)
-         ,@(if (not (string-prefix? "i686-linux" (or (%current-target-system)
-                                                     (%current-system))))
-               ;; Use cross-compiler rather than #:system "i686-linux" to get
-               ;; MesCC 64 bit .go files installed ready for use with Guile.
-               `(("i686-linux-binutils" ,(cross-binutils triplet))
-                 ("i686-linux-gcc" ,(cross-gcc triplet)))
-               '())))
+       `(("guile" ,guile-2.2)))
       (arguments
-       `(#:strip-binaries? #f
+       `(#:system "i686-linux"
+         #:strip-binaries? #f
          #:phases
          (modify-phases %standard-phases
            (add-before 'configure 'optional-dot
-- 
2.19.1


[-- Attachment #3: Type: text/plain, Size: 152 bytes --]


-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: Preparing the reduced bootstrap tarballs

[Timothy Sample]

Hi Jan,

Jan Nieuwenhuizen <janneke@gnu.org> writes:

> Timothy Sample writes:
>
>> I wanted to let you know that I’ve been doing more work on the little
>> Shell backend that we were talking about earlier.  It’s kind of like the
>> worse-is-better version of Gash: it is certainly not as nice, but the
>> development pace is much faster.  I did take a look at just working on
>> Gash directly, but it looked like it was going to be a lot slower.
>
> Oh, that's great news!  We have been making good progress on Gash.  The
> parser has been completely revamped.  The next target is to add a
> transformation that should result in a eval'able shelly Guile script,
> ideally something that you'd want to write by hand.

That’s what I like best about Gash.  My “shelly” code is a mess and no
one should write it by hand!  :)

> To achieve that will take some time.  It would be nice if we could work
> together shaping this shelly script language .

I agree, but I am hesitant to worry about this now.  The more I work on
Geesh the more I realize that building a shell is a big project.  I’ve
consistently been moving from worrying about making things work nicely
to just making them work.  I hope that once the basic features are in
place it will be a lot more fun to improve things gradually.  Does that
make sense?

> If Gash and Geesh would produce (after transformation) the same shelly
> script language, we can also share the backend implementation, WDYT?

I’m a little bit confused about this.  After we exchanged notes last
time, I thought we were going to share front-ends.  Now it seems you’ve
doubled-down on Gash’s PEG parser and want to share back-ends.  Why the
change in plans?  It makes more sense to me to share front-ends because
the parser in Geesh is quite complete.  Also, you already wrote some
code to transform its output into a Gash script.

>> So, keeping in mind that software estimates are very unreliable, I would
>> say that I might have a workable interpreter in the next week or two.
>> There are three big features missing: globbing, asynchronous commands,
>> and arithmetic substitutions.  I am almost finished globbing,
>> asynchronous commands should be pretty easy, and I plan to leave
>> arithmetic substitutions on the road-map for as long as possible (it’s a
>> little bit boring).
>
> Hehe.  I had a look and you did make lots of progress!

So have you!  It’s pretty funny that we both picked up working on this
at about the same time.  In hindsight, I should have contacted you
before diving in again in order to minimize duplicated work.  On the
other hand, it seems you’re guilty of the same thing!  Oh well.  ;)

I‘m not too worried about it, and I’m having fun.  I hope you are too.

>> After I get those first two features working, I am going to start
>> testing running build scripts for Bash.  I will contact you then in case
>> you have any new advice on which scripts are important or anything else.
>
> Great,
> Keep up the good work!

I skipped asynchronous commands and implemented a bunch of built-ins
instead.  Now Geesh will run most of Bash’s configure script.  It runs
until it tries to write “config.sub” which it writes incorrectly due to
an unset variable (ironically, it seems to be the “$SHELL” variable).
That means it chews through over 17,000 lines of configure script!  It
does this very s-l-o-w-l-y, but it’s exciting all the same.


-- Tim

Re: Preparing the reduced bootstrap tarballs, take 3

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 4425 bytes --]

Hello janneke,

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

> I have updated core-updates-next with this fix, a documentation update
> and to also include Marius' mes bootstrap cleanups from his wip-gcc7
> branch.
>
> So, finally for core-updates-next 4ae7dc7b9af64794081b1913740b97acd89c91bc
> here are my lucky numbers
>
> 08:21:04 janneke@dundal:~/src/guix-boot 
> $ for i in $(./pre-inst-env guix build --system=i686-linux bootstrap-tarballs)/*; do echo $i;guix hash $i; done
> /gnu/store/i8rrdivr5ab56kqwbasigp41wqpj1pyp-bootstrap-tarballs-0/guile-static-stripped-2.2.4-i686-linux.tar.xz
> 17p28xv0y1q729idvf2k1dpcfd469akxm8lnf7gd5a7hmnrb28l5
> /gnu/store/i8rrdivr5ab56kqwbasigp41wqpj1pyp-bootstrap-tarballs-0/linux-libre-headers-stripped-4.14.67-i686-linux.tar.xz
> 0sm2z9x4wk45bh6qfs94p0w1d6hsy6dqx9sw38qsqbvxwa1qzk8s
> /gnu/store/i8rrdivr5ab56kqwbasigp41wqpj1pyp-bootstrap-tarballs-0/mescc-tools-static-0.5.2-0.bb062b0-i686-linux.tar.xz
> 11lniw0vg61kmyhvnwkmcnkci9ym6hbmiksiqggd0hkipbq7hvlz
> /gnu/store/i8rrdivr5ab56kqwbasigp41wqpj1pyp-bootstrap-tarballs-0/mes-minimal-stripped-0.18-0.08f04f5-i686-linux.tar.xz
> 0qwpby91hp6afmg5ibdrrk3fw85zxdazfk7rhrdsihsfzqwmfhfx
> /gnu/store/i8rrdivr5ab56kqwbasigp41wqpj1pyp-bootstrap-tarballs-0/static-binaries-0-i686-linux.tar.xz
> 1rs8qh0n6zxmhs0h1b02sydi4vn09qngf3spfpfynkm2n0bxs368
>
> 08:21:10 janneke@dundal:~/src/guix-boot 
> $ for i in $(./pre-inst-env guix build bootstrap-tarballs)/*; do echo $i;guix hash $i; done
> /gnu/store/n8lrszv7qf0rwqhibm57d8jykj253bw2-bootstrap-tarballs-0/guile-static-stripped-2.2.4-x86_64-linux.tar.xz
> 1285hlbfahp414hy3jjdghxjfgi94aj7y3fw0pz7z27421b7z2f9
> /gnu/store/n8lrszv7qf0rwqhibm57d8jykj253bw2-bootstrap-tarballs-0/linux-libre-headers-stripped-4.14.67-x86_64-linux.tar.xz
> 0sm2z9x4wk45bh6qfs94p0w1d6hsy6dqx9sw38qsqbvxwa1qzk8s
> /gnu/store/n8lrszv7qf0rwqhibm57d8jykj253bw2-bootstrap-tarballs-0/mescc-tools-static-0.5.2-0.bb062b0-x86_64-linux.tar.xz
> 11lniw0vg61kmyhvnwkmcnkci9ym6hbmiksiqggd0hkipbq7hvlz
> /gnu/store/n8lrszv7qf0rwqhibm57d8jykj253bw2-bootstrap-tarballs-0/mes-minimal-stripped-0.18-0.08f04f5-x86_64-linux.tar.xz
> 0qwpby91hp6afmg5ibdrrk3fw85zxdazfk7rhrdsihsfzqwmfhfx
> /gnu/store/n8lrszv7qf0rwqhibm57d8jykj253bw2-bootstrap-tarballs-0/static-binaries-0-x86_64-linux.tar.xz
> 0bqyjdbas67wgfwv4rcmr5a5b31l8kk8z6gsidxay7liih5rp5hn
>
> We may want to change the bootstrap downloads for x86_64 in
> bootstrap.scm, mescc-tools-static and mes-minimal-stripped and
> linux-libre-headers-stripped to use the x86_64-linux variants (currently
> we use the i686-linux tarballs); but as you can see their content is
> identical.
>
> Happy verifying ;-)

So, on x86_64-linux, here’s what I got:

--8<---------------cut here---------------start------------->8---
ludo@ribbon ~/src/guix/+core-updates-next$ ./pre-inst-env guix build bootstrap-tarballs
/gnu/store/n8lrszv7qf0rwqhibm57d8jykj253bw2-bootstrap-tarballs-0
ludo@ribbon ~/src/guix/+core-updates-next$ (cd /gnu/store/n8lrszv7qf0rwqhibm57d8jykj253bw2-bootstrap-tarballs-0; for i in * ; do echo $i && guix hash $i ; done )
guile-static-stripped-2.2.4-x86_64-linux.tar.xz
06ygv0b3dpvy93jji5yl4hdz17w4hiqm64m7ypf6g2vi4nfn9xl9
linux-libre-headers-stripped-4.14.67-x86_64-linux.tar.xz
0sm2z9x4wk45bh6qfs94p0w1d6hsy6dqx9sw38qsqbvxwa1qzk8s
mescc-tools-static-0.5.2-0.bb062b0-x86_64-linux.tar.xz
11lniw0vg61kmyhvnwkmcnkci9ym6hbmiksiqggd0hkipbq7hvlz
mes-minimal-stripped-0.18-0.08f04f5-x86_64-linux.tar.xz
0qwpby91hp6afmg5ibdrrk3fw85zxdazfk7rhrdsihsfzqwmfhfx
static-binaries-0-x86_64-linux.tar.xz
0bqyjdbas67wgfwv4rcmr5a5b31l8kk8z6gsidxay7liih5rp5hn
ludo@ribbon ~/src/guix/+core-updates-next$ git describe
v0.15.0-3121-g4ae7dc7b9a
--8<---------------cut here---------------end--------------->8---

So only ‘guile-static-stripped’ differs from yours.  This may be due to
non-determinism in bindings introduced by psyntax (something we
addressed with <https://bugs.gnu.org/20272>, but maybe there are
remnants around.)  I’ve uploaded mine here:

  <http://web.fdn.fr/~lcourtes/tmp/guile-static-stripped-2.2.4-x86_64-linux.tar.xz>

Could you share yours so we can compare?

Anyway, I’ll run the “-s i686-linux” build because right now I only have
the 64-bit ‘guile-static-stripped’, which is not what we want, silly me…
Should we force #:system for this one as well?

Thanks,
Ludo’.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Preparing the reduced bootstrap tarballs, take 3

[Jan Nieuwenhuizen]

Ludovic Courtès writes:

Hello Ludo',

> Jan Nieuwenhuizen <janneke@gnu.org> skribis:
>
>> I have updated core-updates-next with this fix, a documentation update
>> and to also include Marius' mes bootstrap cleanups from his wip-gcc7
>> branch.
>>
>> $ for i in $(./pre-inst-env guix build bootstrap-tarballs)/*; do echo $i;guix hash $i; done
>> /gnu/store/n8lrszv7qf0rwqhibm57d8jykj253bw2-bootstrap-tarballs-0/guile-static-stripped-2.2.4-x86_64-linux.tar.xz
>> 1285hlbfahp414hy3jjdghxjfgi94aj7y3fw0pz7z27421b7z2f9
>> /gnu/store/n8lrszv7qf0rwqhibm57d8jykj253bw2-bootstrap-tarballs-0/linux-libre-headers-stripped-4.14.67-x86_64-linux.tar.xz

> So, on x86_64-linux, here’s what I got:
>
> ludo@ribbon ~/src/guix/+core-updates-next$ ./pre-inst-env guix build bootstrap-tarballs
> /gnu/store/n8lrszv7qf0rwqhibm57d8jykj253bw2-bootstrap-tarballs-0
> ludo@ribbon ~/src/guix/+core-updates-next$ (cd /gnu/store/n8lrszv7qf0rwqhibm57d8jykj253bw2-bootstrap-tarballs-0; for i in * ; do echo $i && guix hash $i ; done )
> guile-static-stripped-2.2.4-x86_64-linux.tar.xz
> 06ygv0b3dpvy93jji5yl4hdz17w4hiqm64m7ypf6g2vi4nfn9xl9

> So only ‘guile-static-stripped’ differs from yours.

Ah, beautiful!

> This may be due to
> non-determinism in bindings introduced by psyntax (something we
> addressed with <https://bugs.gnu.org/20272>, but maybe there are
> remnants around.)  I’ve uploaded mine here:
>
>   <http://web.fdn.fr/~lcourtes/tmp/guile-static-stripped-2.2.4-x86_64-linux.tar.xz>
>
> Could you share yours so we can compare?

Mine is here

    http://lilypond.org/janneke/mes/x86_64-linux/guile-static-stripped-2.2.4-x86_64-linux.tar.xz

...so it looks like we're walking a pretty big round , for --rounds=2 ;-)


> Anyway, I’ll run the “-s i686-linux” build because right now I only have
> the 64-bit ‘guile-static-stripped’, which is not what we want, silly me…

Oh!  Eh, is that right?  I wasn't aware...

> Should we force #:system for this one as well?

We could but then should make sure that any go files in x86_64-linux
bootstrap binaries (mes, and in time gash) are i686-linux too.

janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: Preparing the reduced bootstrap tarballs, take 3

[Jan Nieuwenhuizen]

Jan Nieuwenhuizen writes:

>>   <http://web.fdn.fr/~lcourtes/tmp/guile-static-stripped-2.2.4-x86_64-linux.tar.xz>
>>
>> Could you share yours so we can compare?
>
> Mine is here
>
>     http://lilypond.org/janneke/mes/x86_64-linux/guile-static-stripped-2.2.4-x86_64-linux.tar.xz
>
> ...so it looks like we're walking a pretty big round , for --rounds=2 ;-)

Diffoscope output is huge, here is the file diff list (manually edited)
--8<---------------cut here---------------start------------->8---
18:40:36 janneke@dundal:~/src/ludo [env]
$ PYTHONIOENCODING=utf-8 diffoscope ~/guile-static-stripped-2.2.4-x86_64-linux.tar.xz guile-static-stripped-2.2.4-x86_64-linux.tar.xz |grep .*.go
│ ├── ./lib/guile/2.2/ccache/ice-9/suspendable-ports.go
│ ├── ./lib/guile/2.2/ccache/ice-9/vlist.go
│ ├── ./lib/guile/2.2/ccache/language/cps/effects-analysis.go
│ ├── ./lib/guile/2.2/ccache/language/cps/intmap.go
│ ├── ./lib/guile/2.2/ccache/language/cps/intset.go
│ ├── ./lib/guile/2.2/ccache/language/cps/slot-allocation.go
│ ├── ./lib/guile/2.2/ccache/language/cps/types.go
│ ├── ./lib/guile/2.2/ccache/language/cps/utils.go
│ ├── ./lib/guile/2.2/ccache/srfi/srfi-18.go
│ ├── ./lib/guile/2.2/ccache/srfi/srfi-19.go
│ ├── ./lib/guile/2.2/ccache/srfi/srfi-27.go
--8<---------------cut here---------------end--------------->8---

Not sure if that's useful...hoping the specific names may ring a bell.

Greetings,
janneke

Re: Preparing the reduced bootstrap tarballs, take 3

[Ludovic Courtès]

Hello,

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

> Jan Nieuwenhuizen writes:
>
>>>   <http://web.fdn.fr/~lcourtes/tmp/guile-static-stripped-2.2.4-x86_64-linux.tar.xz>
>>>
>>> Could you share yours so we can compare?
>>
>> Mine is here
>>
>>     http://lilypond.org/janneke/mes/x86_64-linux/guile-static-stripped-2.2.4-x86_64-linux.tar.xz
>>
>> ...so it looks like we're walking a pretty big round , for --rounds=2 ;-)
>
> Diffoscope output is huge, here is the file diff list (manually edited)

Here’s an excerpt of the differences in the .go files:

--8<---------------cut here---------------start------------->8---
$ diffoscope --exclude-directory-metadata janneke ludo
--- janneke
+++ ludo
├── lib
│ ├── guile
│ │ ├── 2.2
│ │ │ ├── ccache
│ │ │ │ ├── language
│ │ │ │ │ ├── tree-il
│ │ │ │ │ │ ├── effects.go
│ │ │ │ │ │ │ ├── /gnu/store/02iklp4swqs0ipxhg5x9b2shmj6b30h1-binutils-2.31.1/bin/readelf --wide --decompress --hex-dump=.rodata {}
│ │ │ │ │ │ │ │ @@ -473,38 +473,38 @@
│ │ │ │ │ │ │ │    0x00016d78 6c2d3137 63356138 34616166 34633833 l-17c5a84aaf4c83
│ │ │ │ │ │ │ │    0x00016d88 30612d31 38620000 27000000 00000000 0a-18b..'.......
│ │ │ │ │ │ │ │    0x00016d98 09000000 00000000 70726f63 2d6e616d ........proc-nam
│ │ │ │ │ │ │ │    0x00016da8 65000000 00000000 27000000 00000000 e.......'.......
│ │ │ │ │ │ │ │    0x00016db8 04000000 00000000 61726773 00000000 ........args....
│ │ │ │ │ │ │ │    0x00016dc8 27000000 00000000 16000000 00000000 '...............
│ │ │ │ │ │ │ │    0x00016dd8 6c2d3638 30623737 35666233 37613436 l-680b775fb37a46
│ │ │ │ │ │ │ │ -  0x00016de8 332d3136 62630000 27000000 00000000 3-16bc..'.......
│ │ │ │ │ │ │ │ +  0x00016de8 332d3166 65300000 27000000 00000000 3-1fe0..'.......
│ │ │ │ │ │ │ │    0x00016df8 16000000 00000000 6c2d3638 30623737 ........l-680b77
│ │ │ │ │ │ │ │ -  0x00016e08 35666233 37613436 332d3136 62640000 5fb37a463-16bd..
│ │ │ │ │ │ │ │ +  0x00016e08 35666233 37613436 332d3166 65310000 5fb37a463-1fe1..
│ │ │ │ │ │ │ │    0x00016e18 27000000 00000000 07000000 00000000 '...............
│ │ │ │ │ │ │ │    0x00016e28 666f726d 616c7300 27000000 00000000 formals.'.......
│ │ │ │ │ │ │ │    0x00016e38 16000000 00000000 6c2d3638 30623737 ........l-680b77
│ │ │ │ │ │ │ │ -  0x00016e48 35666233 37613436 332d3136 62330000 5fb37a463-16b3..
│ │ │ │ │ │ │ │ +  0x00016e48 35666233 37613436 332d3166 64370000 5fb37a463-1fd7..
│ │ │ │ │ │ │ │    0x00016e58 27000000 00000000 16000000 00000000 '...............
│ │ │ │ │ │ │ │    0x00016e68 6c2d3638 30623737 35666233 37613436 l-680b775fb37a46
│ │ │ │ │ │ │ │ -  0x00016e78 332d3136 62340000 27000000 00000000 3-16b4..'.......
│ │ │ │ │ │ │ │ +  0x00016e78 332d3166 64380000 27000000 00000000 3-1fd8..'.......
│ │ │ │ │ │ │ │    0x00016e88 16000000 00000000 6c2d3638 30623737 ........l-680b77
│ │ │ │ │ │ │ │ -  0x00016e98 35666233 37613436 332d3136 62350000 5fb37a463-16b5..
│ │ │ │ │ │ │ │ +  0x00016e98 35666233 37613436 332d3166 64390000 5fb37a463-1fd9..
--8<---------------cut here---------------end--------------->8---

It clearly looks like the psyntax non-deterministic gensym bug described
at <https://bugs.gnu.org/20272>.  I guess we’ll have to give it another
go.

Ludo’.

Re: Preparing the reduced bootstrap tarballs, take 3

[Ludovic Courtès]

Hello,

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

> Jan Nieuwenhuizen writes:
>
>>>   <http://web.fdn.fr/~lcourtes/tmp/guile-static-stripped-2.2.4-x86_64-linux.tar.xz>
>>>
>>> Could you share yours so we can compare?
>>
>> Mine is here
>>
>>     http://lilypond.org/janneke/mes/x86_64-linux/guile-static-stripped-2.2.4-x86_64-linux.tar.xz
>>
>> ...so it looks like we're walking a pretty big round , for --rounds=2 ;-)
>
> Diffoscope output is huge, here is the file diff list (manually edited)

Here’s an excerpt of the differences in the .go files:

--8<---------------cut here---------------start------------->8---
$ diffoscope --exclude-directory-metadata janneke ludo
--- janneke
+++ ludo
├── lib
│ ├── guile
│ │ ├── 2.2
│ │ │ ├── ccache
│ │ │ │ ├── language
│ │ │ │ │ ├── tree-il
│ │ │ │ │ │ ├── effects.go
│ │ │ │ │ │ │ ├── /gnu/store/02iklp4swqs0ipxhg5x9b2shmj6b30h1-binutils-2.31.1/bin/readelf --wide --decompress --hex-dump=.rodata {}
│ │ │ │ │ │ │ │ @@ -473,38 +473,38 @@
│ │ │ │ │ │ │ │    0x00016d78 6c2d3137 63356138 34616166 34633833 l-17c5a84aaf4c83
│ │ │ │ │ │ │ │    0x00016d88 30612d31 38620000 27000000 00000000 0a-18b..'.......
│ │ │ │ │ │ │ │    0x00016d98 09000000 00000000 70726f63 2d6e616d ........proc-nam
│ │ │ │ │ │ │ │    0x00016da8 65000000 00000000 27000000 00000000 e.......'.......
│ │ │ │ │ │ │ │    0x00016db8 04000000 00000000 61726773 00000000 ........args....
│ │ │ │ │ │ │ │    0x00016dc8 27000000 00000000 16000000 00000000 '...............
│ │ │ │ │ │ │ │    0x00016dd8 6c2d3638 30623737 35666233 37613436 l-680b775fb37a46
│ │ │ │ │ │ │ │ -  0x00016de8 332d3136 62630000 27000000 00000000 3-16bc..'.......
│ │ │ │ │ │ │ │ +  0x00016de8 332d3166 65300000 27000000 00000000 3-1fe0..'.......
│ │ │ │ │ │ │ │    0x00016df8 16000000 00000000 6c2d3638 30623737 ........l-680b77
│ │ │ │ │ │ │ │ -  0x00016e08 35666233 37613436 332d3136 62640000 5fb37a463-16bd..
│ │ │ │ │ │ │ │ +  0x00016e08 35666233 37613436 332d3166 65310000 5fb37a463-1fe1..
│ │ │ │ │ │ │ │    0x00016e18 27000000 00000000 07000000 00000000 '...............
│ │ │ │ │ │ │ │    0x00016e28 666f726d 616c7300 27000000 00000000 formals.'.......
│ │ │ │ │ │ │ │    0x00016e38 16000000 00000000 6c2d3638 30623737 ........l-680b77
│ │ │ │ │ │ │ │ -  0x00016e48 35666233 37613436 332d3136 62330000 5fb37a463-16b3..
│ │ │ │ │ │ │ │ +  0x00016e48 35666233 37613436 332d3166 64370000 5fb37a463-1fd7..
│ │ │ │ │ │ │ │    0x00016e58 27000000 00000000 16000000 00000000 '...............
│ │ │ │ │ │ │ │    0x00016e68 6c2d3638 30623737 35666233 37613436 l-680b775fb37a46
│ │ │ │ │ │ │ │ -  0x00016e78 332d3136 62340000 27000000 00000000 3-16b4..'.......
│ │ │ │ │ │ │ │ +  0x00016e78 332d3166 64380000 27000000 00000000 3-1fd8..'.......
│ │ │ │ │ │ │ │    0x00016e88 16000000 00000000 6c2d3638 30623737 ........l-680b77
│ │ │ │ │ │ │ │ -  0x00016e98 35666233 37613436 332d3136 62350000 5fb37a463-16b5..
│ │ │ │ │ │ │ │ +  0x00016e98 35666233 37613436 332d3166 64390000 5fb37a463-1fd9..
--8<---------------cut here---------------end--------------->8---

It clearly looks like the psyntax non-deterministic gensym bug described
at <https://bugs.gnu.org/20272>.  I guess we’ll have to give it another
go.

Ludo’.

Re: Preparing the reduced bootstrap tarballs, take 3

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 1794 bytes --]

Hello!

ludo@gnu.org (Ludovic Courtès) skribis:

> So, on x86_64-linux, here’s what I got:
>
> ludo@ribbon ~/src/guix/+core-updates-next$ ./pre-inst-env guix build bootstrap-tarballs
> /gnu/store/n8lrszv7qf0rwqhibm57d8jykj253bw2-bootstrap-tarballs-0
> ludo@ribbon ~/src/guix/+core-updates-next$ (cd /gnu/store/n8lrszv7qf0rwqhibm57d8jykj253bw2-bootstrap-tarballs-0; for i in * ; do echo $i && guix hash $i ; done )
> guile-static-stripped-2.2.4-x86_64-linux.tar.xz
> 06ygv0b3dpvy93jji5yl4hdz17w4hiqm64m7ypf6g2vi4nfn9xl9
> linux-libre-headers-stripped-4.14.67-x86_64-linux.tar.xz
> 0sm2z9x4wk45bh6qfs94p0w1d6hsy6dqx9sw38qsqbvxwa1qzk8s
> mescc-tools-static-0.5.2-0.bb062b0-x86_64-linux.tar.xz
> 11lniw0vg61kmyhvnwkmcnkci9ym6hbmiksiqggd0hkipbq7hvlz
> mes-minimal-stripped-0.18-0.08f04f5-x86_64-linux.tar.xz
> 0qwpby91hp6afmg5ibdrrk3fw85zxdazfk7rhrdsihsfzqwmfhfx
> static-binaries-0-x86_64-linux.tar.xz
> 0bqyjdbas67wgfwv4rcmr5a5b31l8kk8z6gsidxay7liih5rp5hn
> ludo@ribbon ~/src/guix/+core-updates-next$ git describe
> v0.15.0-3121-g4ae7dc7b9a

I’ve just uploaded these to
<https://alpha.gnu.org/gnu/guix/bootstrap/i686-linux/20181020>:

  linux-libre-headers-stripped-4.14.67-i686-linux.tar.xz
  linux-libre-headers-stripped-4.14.67-i686-linux.tar.xz.sig
  mescc-tools-static-0.5.2-0.bb062b0-i686-linux.tar.xz
  mescc-tools-static-0.5.2-0.bb062b0-i686-linux.tar.xz.sig
  mes-minimal-stripped-0.18-0.08f04f5-i686-linux.tar.xz
  mes-minimal-stripped-0.18-0.08f04f5-i686-linux.tar.xz.sig

Could you adjust bootstrap.scm to refer to this URL?  Currently I see
bootstrap.scm refers to a different version of
linux-libre-headers-stripped so it should be the only one whose hash
needs to be changed.

Thanks, and let me know if we need anything else!

Ludo’.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Preparing the reduced bootstrap tarballs, take 3

[Jan Nieuwenhuizen]

[-- Attachment #1: Type: text/plain, Size: 930 bytes --]

Ludovic Courtès writes:

Hi!

> I’ve just uploaded these to
> <https://alpha.gnu.org/gnu/guix/bootstrap/i686-linux/20181020>:
>
>   linux-libre-headers-stripped-4.14.67-i686-linux.tar.xz
>   linux-libre-headers-stripped-4.14.67-i686-linux.tar.xz.sig
>   mescc-tools-static-0.5.2-0.bb062b0-i686-linux.tar.xz
>   mescc-tools-static-0.5.2-0.bb062b0-i686-linux.tar.xz.sig
>   mes-minimal-stripped-0.18-0.08f04f5-i686-linux.tar.xz
>   mes-minimal-stripped-0.18-0.08f04f5-i686-linux.tar.xz.sig

Great!

> Could you adjust bootstrap.scm to refer to this URL?  Currently I see
> bootstrap.scm refers to a different version of
> linux-libre-headers-stripped so it should be the only one whose hash
> needs to be changed.

I don't see that...and the hash matches.

> Thanks, and let me know if we need anything else!

See attached patch.  Push to my gitlab core-updates-next, not to
savannah yet.

janneke


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-bootstrap-Switch-to-official-bootstrap-urls.patch --]
[-- Type: text/x-patch, Size: 3524 bytes --]

From 3cc2f27208de71f9dd0b37593db974b7c1a305f3 Mon Sep 17 00:00:00 2001
From: Jan Nieuwenhuizen <janneke@gnu.org>
Date: Sun, 9 Dec 2018 19:05:45 +0100
Subject: [PATCH] bootstrap: Switch to official bootstrap urls.

* gnu/packages/bootstrap.scm (%bootstrap-linux-libre-headers): Switch to
official bootstrap urls.
(%bootstrap-mescc-tools): Likewise.
(%bootstrap-mes): Likewise.
---
 gnu/packages/bootstrap.scm | 41 +++++++++++++-------------------------
 1 file changed, 14 insertions(+), 27 deletions(-)

diff --git a/gnu/packages/bootstrap.scm b/gnu/packages/bootstrap.scm
index a835cbc206..eac729f785 100644
--- a/gnu/packages/bootstrap.scm
+++ b/gnu/packages/bootstrap.scm
@@ -405,17 +405,10 @@ $out/bin/guile --version~%"
    (lambda (system)
      (origin
        (method url-fetch)
-       (uri (match system
-              ("i686-linux"
-               (string-append
-                "http://lilypond.org/janneke/guix/20181124/"
-                "linux-libre-headers-stripped-4.14.67-i686-linux.tar.xz"))
-              ("x86_64-linux"
-               (string-append
-                "http://lilypond.org/janneke/guix/20181124/"
-                "linux-libre-headers-stripped-4.14.67-x86_64-linux.tar.xz"))
-              (_ (error "linux-libre-headers-bootstrap: system not supported:"
-                        (%current-system)))))
+       (uri (map (cute string-append <>
+                       "/i686-linux/20181020/"
+                       "linux-libre-headers-stripped-4.14.67-i686-linux.tar.xz")
+                 %bootstrap-base-urls))
        (sha256
         (base32
          "0sm2z9x4wk45bh6qfs94p0w1d6hsy6dqx9sw38qsqbvxwa1qzk8s"))))
@@ -656,15 +649,11 @@ exec ~a/bin/.gcc-wrapped -B~a/lib \
         ,(bootstrap-origin
           (origin
             (method url-fetch)
-            (uri (string-append
-                  "http://lilypond.org/janneke/guix/20181124/"
-                  (match (%current-system)
-                    ("i686-linux"
-                     "mescc-tools-static-0.5.2-0.bb062b0-i686-linux.tar.xz")
-                    ("x86_64-linux"
-                     "mescc-tools-static-0.5.2-0.bb062b0-x86_64-linux.tar.xz")
-                    (_ (error "bootstrap-mescc-tools: system not supported:"
-                              (%current-system))))))
+            (uri (map
+                  (cute string-append <>
+                        "/i686-linux/20181020/"
+                        "mescc-tools-static-0.5.2-0.bb062b0-i686-linux.tar.xz")
+                  %bootstrap-base-urls))
             (sha256
              (base32
               "11lniw0vg61kmyhvnwkmcnkci9ym6hbmiksiqggd0hkipbq7hvlz")))))))
@@ -708,13 +697,11 @@ exec ~a/bin/.gcc-wrapped -B~a/lib \
         ,(bootstrap-origin
           (origin
             (method url-fetch)
-            (uri (string-append
-                  "http://lilypond.org/janneke/guix/20181124/"
-                  (match (%current-system)
-                    ("i686-linux"
-                     "mes-minimal-stripped-0.18-0.08f04f5-i686-linux.tar.xz")
-                    ("x86_64-linux"
-                     "mes-minimal-stripped-0.18-0.08f04f5-x86_64-linux.tar.xz"))))
+            (uri (map
+                  (cute string-append <>
+                        "/i686-linux/20181020/"
+                        "mes-minimal-stripped-0.18-0.08f04f5-i686-linux.tar.xz")
+                 %bootstrap-base-urls))
             (sha256
              (match (%current-system)
                ((or "i686-linux" "x86_64-linux")
-- 
2.19.2

Re: Preparing the reduced bootstrap tarballs, take 3

[Ludovic Courtès]

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

> Ludovic Courtès writes:
>
> Hi!
>
>> I’ve just uploaded these to
>> <https://alpha.gnu.org/gnu/guix/bootstrap/i686-linux/20181020>:
>>
>>   linux-libre-headers-stripped-4.14.67-i686-linux.tar.xz
>>   linux-libre-headers-stripped-4.14.67-i686-linux.tar.xz.sig
>>   mescc-tools-static-0.5.2-0.bb062b0-i686-linux.tar.xz
>>   mescc-tools-static-0.5.2-0.bb062b0-i686-linux.tar.xz.sig
>>   mes-minimal-stripped-0.18-0.08f04f5-i686-linux.tar.xz
>>   mes-minimal-stripped-0.18-0.08f04f5-i686-linux.tar.xz.sig
>
> Great!
>
>> Could you adjust bootstrap.scm to refer to this URL?  Currently I see
>> bootstrap.scm refers to a different version of
>> linux-libre-headers-stripped so it should be the only one whose hash
>> needs to be changed.
>
> I don't see that...and the hash matches.

We’ve discussed it in person, and now I think we’re all set!  :-)

Ludo’.

Re: Preparing the reduced bootstrap tarballs, take 3

[Jan Nieuwenhuizen]

Ludovic Courtès writes:

> Jan Nieuwenhuizen <janneke@gnu.org> skribis:
>
>> Ludovic Courtès writes:
>>
>> Hi!
>>
>>> I’ve just uploaded these to
>>> <https://alpha.gnu.org/gnu/guix/bootstrap/i686-linux/20181020>:
>>>
>>>   linux-libre-headers-stripped-4.14.67-i686-linux.tar.xz
>>>   linux-libre-headers-stripped-4.14.67-i686-linux.tar.xz.sig
>>>   mescc-tools-static-0.5.2-0.bb062b0-i686-linux.tar.xz
>>>   mescc-tools-static-0.5.2-0.bb062b0-i686-linux.tar.xz.sig
>>>   mes-minimal-stripped-0.18-0.08f04f5-i686-linux.tar.xz
>>>   mes-minimal-stripped-0.18-0.08f04f5-i686-linux.tar.xz.sig
>>
>> Great!
>>
>>> Could you adjust bootstrap.scm to refer to this URL?  Currently I see
>>> bootstrap.scm refers to a different version of
>>> linux-libre-headers-stripped so it should be the only one whose hash
>>> needs to be changed.
>>
>> I don't see that...and the hash matches.
>
> We’ve discussed it in person, and now I think we’re all set!  :-)

Yes, that helped!  Pushed to core-updates as

da3c6a7f19ef1243af725f63c16c8fd92fde33b4
bootstrap: Switch to official bootstrap urls.

janneke

Re: Preparing the reduced bootstrap tarballs, take 3

[Mark H Weaver]

Ludovic Courtès <ludo@gnu.org> writes:

> Jan Nieuwenhuizen <janneke@gnu.org> skribis:
>
>> Ludovic Courtès writes:
>>
>> Hi!
>>
>>> I’ve just uploaded these to
>>> <https://alpha.gnu.org/gnu/guix/bootstrap/i686-linux/20181020>:
>>>
>>>   linux-libre-headers-stripped-4.14.67-i686-linux.tar.xz
>>>   linux-libre-headers-stripped-4.14.67-i686-linux.tar.xz.sig
>>>   mescc-tools-static-0.5.2-0.bb062b0-i686-linux.tar.xz
>>>   mescc-tools-static-0.5.2-0.bb062b0-i686-linux.tar.xz.sig
>>>   mes-minimal-stripped-0.18-0.08f04f5-i686-linux.tar.xz
>>>   mes-minimal-stripped-0.18-0.08f04f5-i686-linux.tar.xz.sig
>>
>> Great!
>>
>>> Could you adjust bootstrap.scm to refer to this URL?  Currently I see
>>> bootstrap.scm refers to a different version of
>>> linux-libre-headers-stripped so it should be the only one whose hash
>>> needs to be changed.
>>
>> I don't see that...and the hash matches.
>
> We’ve discussed it in person, and now I think we’re all set!  :-)

Does our documentation include instructions on how to reproducibly build
these new bootstrap binaries, to independently verify them?

     Thanks,
       Mark

Re: Preparing the reduced bootstrap tarballs, take 3

[Ludovic Courtès]

Hi Mark,

Mark H Weaver <mhw@netris.org> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Jan Nieuwenhuizen <janneke@gnu.org> skribis:
>>
>>> Ludovic Courtès writes:
>>>
>>> Hi!
>>>
>>>> I’ve just uploaded these to
>>>> <https://alpha.gnu.org/gnu/guix/bootstrap/i686-linux/20181020>:
>>>>
>>>>   linux-libre-headers-stripped-4.14.67-i686-linux.tar.xz
>>>>   linux-libre-headers-stripped-4.14.67-i686-linux.tar.xz.sig
>>>>   mescc-tools-static-0.5.2-0.bb062b0-i686-linux.tar.xz
>>>>   mescc-tools-static-0.5.2-0.bb062b0-i686-linux.tar.xz.sig
>>>>   mes-minimal-stripped-0.18-0.08f04f5-i686-linux.tar.xz
>>>>   mes-minimal-stripped-0.18-0.08f04f5-i686-linux.tar.xz.sig
>>>
>>> Great!
>>>
>>>> Could you adjust bootstrap.scm to refer to this URL?  Currently I see
>>>> bootstrap.scm refers to a different version of
>>>> linux-libre-headers-stripped so it should be the only one whose hash
>>>> needs to be changed.
>>>
>>> I don't see that...and the hash matches.
>>
>> We’ve discussed it in person, and now I think we’re all set!  :-)
>
> Does our documentation include instructions on how to reproducibly build
> these new bootstrap binaries, to independently verify them?

The build procedure remains unchanged (info "(guix) Bootstrapping"):

  guix build bootstrap-tarballs

To verify them, you can do what I described in this thread, which is to
use the same commit as Janneke and myself used, run “guix build
bootstrap-tarballs”, and compare the three tarballs listed above.

Ludo’.

Re: Preparing the reduced bootstrap tarballs, take 3

[Jan Nieuwenhuizen]

Ludovic Courtès writes:

>> Does our documentation include instructions on how to reproducibly build
>> these new bootstrap binaries, to independently verify them?
>
> The build procedure remains unchanged (info "(guix) Bootstrapping"):
>
>   guix build bootstrap-tarballs
>
> To verify them, you can do what I described in this thread, which is to
> use the same commit as Janneke and myself used, run “guix build
> bootstrap-tarballs”, and compare the three tarballs listed above.

Note however that while you're very welcome to do so now, Ricardo and
are working on at least a new mes-minimal-stripped-tarball that I
expect any day now.

janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: Preparing the reduced bootstrap tarballs, take 3

[Mark H Weaver]

Ludovic Courtès <ludo@gnu.org> writes:

> Mark H Weaver <mhw@netris.org> skribis:
>
>> Ludovic Courtès <ludo@gnu.org> writes:
>>
>>> Jan Nieuwenhuizen <janneke@gnu.org> skribis:
>>>
>>>> Ludovic Courtès writes:
>>>>
>>>> Hi!
>>>>
>>>>> I’ve just uploaded these to
>>>>> <https://alpha.gnu.org/gnu/guix/bootstrap/i686-linux/20181020>:
>>>>>
>>>>>   linux-libre-headers-stripped-4.14.67-i686-linux.tar.xz
>>>>>   linux-libre-headers-stripped-4.14.67-i686-linux.tar.xz.sig
>>>>>   mescc-tools-static-0.5.2-0.bb062b0-i686-linux.tar.xz
>>>>>   mescc-tools-static-0.5.2-0.bb062b0-i686-linux.tar.xz.sig
>>>>>   mes-minimal-stripped-0.18-0.08f04f5-i686-linux.tar.xz
>>>>>   mes-minimal-stripped-0.18-0.08f04f5-i686-linux.tar.xz.sig
>>>>
>>>> Great!
>>>>
>>>>> Could you adjust bootstrap.scm to refer to this URL?  Currently I see
>>>>> bootstrap.scm refers to a different version of
>>>>> linux-libre-headers-stripped so it should be the only one whose hash
>>>>> needs to be changed.
>>>>
>>>> I don't see that...and the hash matches.
>>>
>>> We’ve discussed it in person, and now I think we’re all set!  :-)
>>
>> Does our documentation include instructions on how to reproducibly build
>> these new bootstrap binaries, to independently verify them?
>
> The build procedure remains unchanged (info "(guix) Bootstrapping"):
>
>   guix build bootstrap-tarballs
>
> To verify them, you can do what I described in this thread, which is to
> use the same commit as Janneke and myself used, run “guix build
> bootstrap-tarballs”, and compare the three tarballs listed above.

What commit did you use?  It would be good to document this, so that
others can independently verify our bootstrap binaries, now or in the
future.

     Thanks,
       Mark

Re: Preparing the reduced bootstrap tarballs, take 3

[Ludovic Courtès]

Mark H Weaver <mhw@netris.org> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Mark H Weaver <mhw@netris.org> skribis:
>>
>>> Ludovic Courtès <ludo@gnu.org> writes:
>>>
>>>> Jan Nieuwenhuizen <janneke@gnu.org> skribis:
>>>>
>>>>> Ludovic Courtès writes:
>>>>>
>>>>> Hi!
>>>>>
>>>>>> I’ve just uploaded these to
>>>>>> <https://alpha.gnu.org/gnu/guix/bootstrap/i686-linux/20181020>:
>>>>>>
>>>>>>   linux-libre-headers-stripped-4.14.67-i686-linux.tar.xz
>>>>>>   linux-libre-headers-stripped-4.14.67-i686-linux.tar.xz.sig
>>>>>>   mescc-tools-static-0.5.2-0.bb062b0-i686-linux.tar.xz
>>>>>>   mescc-tools-static-0.5.2-0.bb062b0-i686-linux.tar.xz.sig
>>>>>>   mes-minimal-stripped-0.18-0.08f04f5-i686-linux.tar.xz
>>>>>>   mes-minimal-stripped-0.18-0.08f04f5-i686-linux.tar.xz.sig
>>>>>
>>>>> Great!
>>>>>
>>>>>> Could you adjust bootstrap.scm to refer to this URL?  Currently I see
>>>>>> bootstrap.scm refers to a different version of
>>>>>> linux-libre-headers-stripped so it should be the only one whose hash
>>>>>> needs to be changed.
>>>>>
>>>>> I don't see that...and the hash matches.
>>>>
>>>> We’ve discussed it in person, and now I think we’re all set!  :-)
>>>
>>> Does our documentation include instructions on how to reproducibly build
>>> these new bootstrap binaries, to independently verify them?
>>
>> The build procedure remains unchanged (info "(guix) Bootstrapping"):
>>
>>   guix build bootstrap-tarballs
>>
>> To verify them, you can do what I described in this thread, which is to
>> use the same commit as Janneke and myself used, run “guix build
>> bootstrap-tarballs”, and compare the three tarballs listed above.
>
> What commit did you use?  It would be good to document this, so that
> others can independently verify our bootstrap binaries, now or in the
> future.

The commit appears in the commit log and in this very thread, no?

Ludo’.

Re: Preparing the reduced bootstrap tarballs, take 3

[Mark H Weaver]

Hi Ludovic,

Ludovic Courtès <ludo@gnu.org> writes:

> Mark H Weaver <mhw@netris.org> skribis:
>
>> What commit did you use?  It would be good to document this, so that
>> others can independently verify our bootstrap binaries, now or in the
>> future.
>
> The commit appears in the commit log and in this very thread, no?

Yes, thanks.  I'm glad to see it in the commit log.  I thought it might
also be worthy of documentation in our source tree, possibly in HACKING,
for the sake of those who wish to independently verify our bootstrap
binaries in the future.  However, it's not something that I feel
strongly about.

     Thanks,
       Mark