Preparing the reduced bootstrap tarballs

Preparing the reduced bootstrap tarballs

[Ludovic Courtès]

Hello!

(This is a followup to <https://bugs.gnu.org/33038>.)

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

> Did those, pushed to core-updates-next as
>
>     d0bb7ed61ed9e356c53de1a8e9bd6c2ec030ffb6
>     doc: Update Preparing to Use the Bootstrap Binaries.

[...]

> I think that I'm "done" here, there's the `put new bootstrap binaries on
> ftp.gnu.org/pub/guix/...', are you taking care of that?

I (finally!) run “guix build bootstrap-tarballs” on ‘core-updates-next’,
but that was a bit silly of me since that built the x86_64-linux
tarballs—i.e., not the “reduced seed.”

So I was about to re-run it with “-s i686-linux”, but I noticed the
following issue in (gnu packages make-bootstrap):

--8<---------------cut here---------------start------------->8---
(define %bootstrap-tarballs
  ;; A single derivation containing all the bootstrap tarballs, for
  ;; convenience.
  (package

[...]

    (inputs `(("guile-tarball" ,%guile-bootstrap-tarball)
              ,@(match (%current-system)
                  ("i686-linux" `(("mescc-tools-seed" ,(@ (gnu packages bootstrap) %mescc-tools-seed))
                                  ("mes-seed" ,(@ (gnu packages bootstrap) %mes-seed))
                                  ("srfi-43" ,(@ (gnu packages bootstrap) %srfi-43))
                                  ("tinycc-seed" ,(@ (gnu packages bootstrap) %tinycc-seed))))
--8<---------------cut here---------------end--------------->8---

This does not actually build the bootstrap tarballs; instead it returns
the pre-built seeds (also the ‘%tinycc-seed’ variable doesn’t exist.)

What we would need here is something to build the things listed in
‘%bootstrap-inputs’, namely:
‘linux-libre-headers-stripped-4.14.26-i686-linux.tar.xz’ (easy :-)),
‘mescc-tools-seed-XYZ.tar.gz’, and
‘mes-stripped-0.18-0.08f04f5-i686-linux.tar.xz’ (do we really need an
x86_64 version of this Mes?).

Does that make sense?

Sorry for the delay, and apologies if I overlooked something!

Ludo’.

Re: Preparing the reduced bootstrap tarballs

[Jan Nieuwenhuizen]

Ludovic Courtès writes:

Hi!

> (This is a followup to <https://bugs.gnu.org/33038>.)

> I (finally!) run “guix build bootstrap-tarballs” on ‘core-updates-next’,
> but that was a bit silly of me since that built the x86_64-linux
> tarballs—i.e., not the “reduced seed.”
>
> So I was about to re-run it with “-s i686-linux”, but I noticed the
> following issue in (gnu packages make-bootstrap):
>
> (define %bootstrap-tarballs
>   ;; A single derivation containing all the bootstrap tarballs, for
>   ;; convenience.
>   (package

Ah right.  I saw that several times but did not use it.  I think because
initially it was of no use.  It would be nice if this built everything
we need, I agree :-)

> [...]
>
>     (inputs `(("guile-tarball" ,%guile-bootstrap-tarball)
>               ,@(match (%current-system)
>                   ("i686-linux" `(("mescc-tools-seed" ,(@ (gnu packages bootstrap) %mescc-tools-seed))
>                                   ("mes-seed" ,(@ (gnu packages bootstrap) %mes-seed))
>                                   ("srfi-43" ,(@ (gnu packages bootstrap) %srfi-43))
>                                   ("tinycc-seed" ,(@ (gnu packages bootstrap) %tinycc-seed))))
>
> This does not actually build the bootstrap tarballs; instead it returns
> the pre-built seeds (also the ‘%tinycc-seed’ variable doesn’t exist.)

Indeed.  "mes-seed" and "tinycc-seed" are remnants of the past; the only
things we need are

> What we would need here is something to build the things listed in
> ‘%bootstrap-inputs’, namely:
> ‘linux-libre-headers-stripped-4.14.26-i686-linux.tar.xz’ (easy :-)),
> ‘mescc-tools-seed-XYZ.tar.gz’, and
> ‘mes-stripped-0.18-0.08f04f5-i686-linux.tar.xz’

So if you like, please make that change.  There is only one little
thing: I have no (scripted) recipe to create mescc-tools-seed-XYZ.  But
wait: I have a great excuse for that...I was too lazy or too sloppy.

The thing is, I used to build mescc-tools-seed, mes-seed and tinycc-seed
manually from the mes+mescc+tinycc source trees.  Jeremiah Orians is
working to remove any need for mescc-tools-seed (esp. the forward
dependency on Mes) but I don't think we're there yet.

Anyway, I think we/I will have to put some work into scripting
mescc-tools-seed or otherwise changing the mescc-tools-boot build.
WDYT?

> (do we really need an x86_64 version of this Mes?).

No, I don't think so.  I added it esp. to get a preview and enable
future development of pure x86_64 bootstrap; but dependency-wise we
should be able to drop it!

> Does that make sense?

Yes!

> Sorry for the delay, and apologies if I overlooked something!

Thank you for looking into and pointing out!
janneke

Re: Preparing the reduced bootstrap tarballs

[Jeremiah]

> Indeed.  "mes-seed" and "tinycc-seed" are remnants of the past; the only
> things we need are
>
>> What we would need here is something to build the things listed in
>> ‘%bootstrap-inputs’, namely:
>> ‘linux-libre-headers-stripped-4.14.26-i686-linux.tar.xz’ (easy :-)),
>> ‘mescc-tools-seed-XYZ.tar.gz’, and
>> ‘mes-stripped-0.18-0.08f04f5-i686-linux.tar.xz’

That is correct (With mescc-tools-seed adding steps and removing
binaries over time)

> So if you like, please make that change.  There is only one little
> thing: I have no (scripted) recipe to create mescc-tools-seed-XYZ.  But
> wait: I have a great excuse for that...I was too lazy or too sloppy.

I do, in mescc-tools-seed; the script bootstrap.sh when run with the
option "sin" will build the mescc-tools-seed binaries using mescc-tools.
The .M1 files are always generated by cc_x86.s using the C source files.


> The thing is, I used to build mescc-tools-seed, mes-seed and tinycc-seed
> manually from the mes+mescc+tinycc source trees.
I've been building them from the stage0 bootstrap tree (which as you can
see is rather trivial:
https://github.com/oriansj/talk-notes/blob/master/Current%20bootstrap%20map.dot
)


> Jeremiah Orians is
> working to remove any need for mescc-tools-seed (esp. the forward
> dependency on Mes) but I don't think we're there yet.
We have eliminated the forward dependency on Mes for the creation of the
mescc-tools-seed already


> Anyway, I think we/I will have to put some work into scripting
> mescc-tools-seed or otherwise changing the mescc-tools-boot build.
Already done in bash and kaem but not in guix yet (Should be trivial)

> WDYT?
I think we will end up having several versions of mescc-tools-seed; as
each architecture guix supports will end up needing a variant if we plan
on keeping them small. (I also have no idea how to make a multi-arch fat
elf binary)

I am also curious if there is any demand for the stripped versions of
mescc-tools-seed as those binaries are nearly half the size.

>> (do we really need an x86_64 version of this Mes?).
> No, I don't think so.  I added it esp. to get a preview and enable
> future development of pure x86_64 bootstrap; but dependency-wise we
> should be able to drop it!
also, AMD64 does support i386 binaries without issue

- Jeremiah

Re: Preparing the reduced bootstrap tarballs

[Ludovic Courtès]

Hello!

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

> Ludovic Courtès writes:
>
> Hi!
>
>> (This is a followup to <https://bugs.gnu.org/33038>.)
>
>> I (finally!) run “guix build bootstrap-tarballs” on ‘core-updates-next’,
>> but that was a bit silly of me since that built the x86_64-linux
>> tarballs—i.e., not the “reduced seed.”
>>
>> So I was about to re-run it with “-s i686-linux”, but I noticed the
>> following issue in (gnu packages make-bootstrap):
>>
>> (define %bootstrap-tarballs
>>   ;; A single derivation containing all the bootstrap tarballs, for
>>   ;; convenience.
>>   (package
>
> Ah right.  I saw that several times but did not use it.  I think because
> initially it was of no use.  It would be nice if this built everything
> we need, I agree :-)

Heheh.  :-)  It’s what the “Building the Bootstrap Binaries” section
describes.

> Indeed.  "mes-seed" and "tinycc-seed" are remnants of the past; the only
> things we need are

OK.

>> What we would need here is something to build the things listed in
>> ‘%bootstrap-inputs’, namely:
>> ‘linux-libre-headers-stripped-4.14.26-i686-linux.tar.xz’ (easy :-)),
>> ‘mescc-tools-seed-XYZ.tar.gz’, and
>> ‘mes-stripped-0.18-0.08f04f5-i686-linux.tar.xz’
>
> So if you like, please make that change.  There is only one little
> thing: I have no (scripted) recipe to create mescc-tools-seed-XYZ.  But
> wait: I have a great excuse for that...I was too lazy or too sloppy.
>
> The thing is, I used to build mescc-tools-seed, mes-seed and tinycc-seed
> manually from the mes+mescc+tinycc source trees.  Jeremiah Orians is
> working to remove any need for mescc-tools-seed (esp. the forward
> dependency on Mes) but I don't think we're there yet.
>
> Anyway, I think we/I will have to put some work into scripting
> mescc-tools-seed or otherwise changing the mescc-tools-boot build.
> WDYT?

I’m confused: how did you build the seeds that (gnu packages bootstrap)
refers to in ‘core-updates-next’?

The goal is for the seeds to be built through Guix so we have a
transparent and documented way to reproduce/verify them.

I could propose a patch to do that, though from what you’re saying
generating ‘mescc-tools-seed’ is not something readily doable?

>> (do we really need an x86_64 version of this Mes?).
>
> No, I don't think so.  I added it esp. to get a preview and enable
> future development of pure x86_64 bootstrap; but dependency-wise we
> should be able to drop it!

Neat.

Thanks for the explanations!

Ludo’.

Re: Preparing the reduced bootstrap tarballs

[Ludovic Courtès]

Jeremiah@pdp10.guru skribis:

>> So if you like, please make that change.  There is only one little
>> thing: I have no (scripted) recipe to create mescc-tools-seed-XYZ.  But
>> wait: I have a great excuse for that...I was too lazy or too sloppy.
>
> I do, in mescc-tools-seed; the script bootstrap.sh when run with the
> option "sin" will build the mescc-tools-seed binaries using mescc-tools.
> The .M1 files are always generated by cc_x86.s using the C source files.

I saw this script but it’s not entirely clear to me how to package the
whole thing.  We don’t have a “stage0” package for instance in Guix, do
we?

>> WDYT?
> I think we will end up having several versions of mescc-tools-seed; as
> each architecture guix supports will end up needing a variant if we plan
> on keeping them small. (I also have no idea how to make a multi-arch fat
> elf binary)

For now let’s focus on x86_64/i686.  :-)

IMO we should change the seeds as rarely as possible because they are
managed “out-of-band” and verifying them is difficult (you need to fetch
the right Guix commit, run “guix build bootstrap-tarballs”, and compare
the result—assuming this is all bit-reproducible.)

The one we’re using today in Guix date back to 2013.

Thanks,
Ludo’.

Re: Preparing the reduced bootstrap tarballs

[Jan Nieuwenhuizen]

[-- Attachment #1: Type: text/plain, Size: 1701 bytes --]

Ludovic Courtès writes:

> Jeremiah@pdp10.guru skribis:
>
>>> So if you like, please make that change.  There is only one little
>>> thing: I have no (scripted) recipe to create mescc-tools-seed-XYZ.  But
>>> wait: I have a great excuse for that...I was too lazy or too sloppy.
>>
>> I do, in mescc-tools-seed; the script bootstrap.sh when run with the
>> option "sin" will build the mescc-tools-seed binaries using mescc-tools.
>> The .M1 files are always generated by cc_x86.s using the C source files.
>
> I saw this script but it’s not entirely clear to me how to package the
> whole thing.  We don’t have a “stage0” package for instance in Guix, do
> we?

I had a look but it seems we need either binary seeds for stage0, or
binary seeds for mescc-tools and I'd rather build them within Guix; so
that's what I did now.

>>> WDYT?
>> I think we will end up having several versions of mescc-tools-seed; as
>> each architecture guix supports will end up needing a variant if we plan
>> on keeping them small. (I also have no idea how to make a multi-arch fat
>> elf binary)
>
> For now let’s focus on x86_64/i686.  :-)
>
> IMO we should change the seeds as rarely as possible because they are
> managed “out-of-band” and verifying them is difficult (you need to fetch
> the right Guix commit, run “guix build bootstrap-tarballs”, and compare
> the result—assuming this is all bit-reproducible.)
>
> The one we’re using today in Guix date back to 2013.

Okay, I am proposing the attached patches, also available on my 
core-updates-next at http://gitlab.com/janneke/guix

Still I hope these won't last for five more years ;-)
janneke.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-bootstrap-Add-mescc-tools-static-mescc-tools-static-.patch --]
[-- Type: text/x-patch, Size: 1884 bytes --]

From 5702107a31f52a615c516084b7a82d9f5e2967e0 Mon Sep 17 00:00:00 2001
From: Jan Nieuwenhuizen <janneke@gnu.org>
Date: Fri, 16 Nov 2018 19:29:55 +0100
Subject: [PATCH 1/3] bootstrap: Add mescc-tools-static,
 mescc-tools-static-tarball.

* gnu/packages/make-bootstrap.scm (%mescc-tools-static,
%mescc-tools-bootstrap-tarball):  New variable.
---
 gnu/packages/make-bootstrap.scm | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/gnu/packages/make-bootstrap.scm b/gnu/packages/make-bootstrap.scm
index 7bfba3c14..36a5f6904 100644
--- a/gnu/packages/make-bootstrap.scm
+++ b/gnu/packages/make-bootstrap.scm
@@ -48,6 +48,7 @@
             %glibc-bootstrap-tarball
             %gcc-bootstrap-tarball
             %guile-bootstrap-tarball
+            %mescc-tools-bootstrap-tarball
             %mes-bootstrap-tarball
             %bootstrap-tarballs
 
@@ -535,6 +536,16 @@ for `sh' in $PATH, and without nscd, and with static NSS modules."
            #t))))
     (inputs `(("gcc" ,%gcc-static)))))
 
+(define %mescc-tools-static
+  ;; A statically linked MesCC Tools for bootstrap.
+  (package
+    (inherit mescc-tools)
+    (name "mescc-tools-static")
+    (arguments
+     (substitute-keyword-arguments (package-arguments mescc-tools)
+       ((#:make-flags flags)
+        `(cons "CC=gcc -static" ,flags))))))
+
 (define %mes-stripped
   ;; The subset of Mes files needed for bootstrap.
   (package
@@ -731,6 +742,10 @@ for `sh' in $PATH, and without nscd, and with static NSS modules."
   ;; A tarball with the statically-linked, relocatable Guile.
   (tarball-package %guile-static-stripped))
 
+(define %mescc-tools-bootstrap-tarball
+  ;; A tarball with MesCC binary seed.
+  (tarball-package %mescc-tools-static))
+
 (define %mes-bootstrap-tarball
   ;; A tarball with Mes ASCII Seed and binary Mes C Library.
   (tarball-package %mes-stripped))
-- 
2.19.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-bootstrap-Add-bootstrap-mescc-tools.patch --]
[-- Type: text/x-patch, Size: 3163 bytes --]

From cc3782c5f9d2c606a265be0dfdca44d79ec25622 Mon Sep 17 00:00:00 2001
From: Jan Nieuwenhuizen <janneke@gnu.org>
Date: Fri, 16 Nov 2018 21:30:00 +0100
Subject: [PATCH 2/3] bootstrap: Add %bootstrap-mescc-tools.

Built with
    5702107a31f52a615c516084b7a82d9f5e2967e0 bootstrap: Add mescc-tools-static, mescc-tools-static-tarball.

* gnu/packages/bootstrap.scm (%bootstrap-mescc-tools): New variable.
---
 gnu/packages/bootstrap.scm | 49 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 49 insertions(+)

diff --git a/gnu/packages/bootstrap.scm b/gnu/packages/bootstrap.scm
index 7e6200c57..1075325dc 100644
--- a/gnu/packages/bootstrap.scm
+++ b/gnu/packages/bootstrap.scm
@@ -49,6 +49,7 @@
             %bootstrap-gcc
             %bootstrap-glibc
             %bootstrap-inputs
+            %bootstrap-mescc-tools
             %bootstrap-mes
             %mescc-tools-seed
             %srfi-43))
@@ -615,6 +616,54 @@ exec ~a/bin/.gcc-wrapped -B~a/lib \
     (home-page #f)
     (license gpl3+)))
 
+(define %bootstrap-mescc-tools
+  ;; The initial MesCC tools.  Uses binaries from a tarball typically built by
+  ;; %MESCC-TOOLS-BOOTSTRAP-TARBALL.
+  (package
+    (name "bootstrap-mescc-tools")
+    (version "0.5.2")
+    (source #f)
+    (build-system trivial-build-system)
+    (arguments
+     `(#:guile ,%bootstrap-guile
+       #:modules ((guix build utils))
+       #:builder
+       (begin
+         (use-modules (guix build utils)
+                      (ice-9 popen))
+         (let ((out     (assoc-ref %outputs "out"))
+               (tar     (assoc-ref %build-inputs "tar"))
+               (xz      (assoc-ref %build-inputs "xz"))
+               (tarball (assoc-ref %build-inputs "tarball")))
+
+           (mkdir out)
+           (copy-file tarball "binaries.tar.xz")
+           (invoke xz "-d" "binaries.tar.xz")
+           (let ((builddir (getcwd))
+                 (bindir   (string-append out "/bin")))
+             (with-directory-excursion out
+               (invoke tar "xvf"
+                       (string-append builddir "/binaries.tar"))))))))
+    (inputs
+     `(("tar" ,(search-bootstrap-binary "tar" (%current-system)))
+       ("xz"  ,(search-bootstrap-binary "xz" (%current-system)))
+       ("tarball" ,(bootstrap-origin
+                    (origin
+                      (method url-fetch)
+                      (uri (string-append
+                            "http://lilypond.org/janneke/mes/"
+                            (match (%current-system)
+                              ((or "i686-linux" "x86_64-linux")
+                               "mescc-tools-static-0.5.2-0.bb062b0-i686-linux.tar.xz"))))
+                      (sha256
+                       (match (%current-system)
+                         ((or "i686-linux" "x86_64-linux")
+                          (base32 "1qcqz5lfl2blbfvh5ccg08pcxfhqrmlrsh6cyaxvm0hxsr1rvlm1")))))))))
+    (synopsis "Bootstrap binaries of MesCC Tools")
+    (description synopsis)
+    (home-page #f)
+    (license gpl3+)))
+
 (define %bootstrap-mes
   ;; The initial Mes.  Uses binaries from a tarball typically built by
   ;; %MES-BOOTSTRAP-TARBALL.
-- 
2.19.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: 0003-bootstrap-Replace-mescc-tools-seed-with-bootstrap-me.patch --]
[-- Type: text/x-patch, Size: 6662 bytes --]

From daa9ff9463917687aabed8330eacb304a7e4deae Mon Sep 17 00:00:00 2001
From: Jan Nieuwenhuizen <janneke@gnu.org>
Date: Fri, 16 Nov 2018 21:30:28 +0100
Subject: [PATCH 3/3] bootstrap: Replace %mescc-tools-seed with
 %bootstrap-mescc-tools.

* gnu/packages/bootstrap.scm (%mescc-tools-seed): Remove.
(%bootstrap-inputs): Replace %mescc-tools-seed with %bootstrap-mescc-tools.
* gnu/packages/commencement.scm (mes-boot): Likewise.
(tcc-boot0): Likewise.
(mescc-tools-boot): Remove.
---
 gnu/packages/bootstrap.scm    | 14 +------
 gnu/packages/commencement.scm | 79 ++---------------------------------
 2 files changed, 4 insertions(+), 89 deletions(-)

diff --git a/gnu/packages/bootstrap.scm b/gnu/packages/bootstrap.scm
index 1075325dc..2eac08cb4 100644
--- a/gnu/packages/bootstrap.scm
+++ b/gnu/packages/bootstrap.scm
@@ -51,7 +51,6 @@
             %bootstrap-inputs
             %bootstrap-mescc-tools
             %bootstrap-mes
-            %mescc-tools-seed
             %srfi-43))
 
 ;;; Commentary:
@@ -716,17 +715,6 @@ exec ~a/bin/.gcc-wrapped -B~a/lib \
     (home-page #f)
     (license gpl3+)))
 
-(define %mescc-tools-seed ; todo: add tarballs to alpha.gnu.org/pub/mes/bootstrap/
-  (let ((commit "dc4e20e74924a5c80a2b7a77b4d7b927234fa71c"))
-    (origin
-      (method url-fetch)
-      (uri (string-append "https://gitlab.com/janneke/mescc-tools-seed"
-                          "/-/archive/" commit
-                          "/mescc-tools-seed-" commit ".tar.gz"))
-      (sha256
-       (base32
-        "1lj7df73vxanmffmiwkhcn83r7yd9n8568nkki06bqq5zg526nyz")))))
-
 (define %srfi-43
   (origin
     (method url-fetch)
@@ -742,7 +730,7 @@ exec ~a/bin/.gcc-wrapped -B~a/lib \
   `(,@(match (%current-system)
         ((or "i686-linux" "x86_64-linux")
          `(("linux-libre-headers" ,%bootstrap-linux-libre-headers)
-           ("mescc-tools-seed" ,%mescc-tools-seed)
+           ("bootstrap-mescc-tools" ,%bootstrap-mescc-tools)
            ("mes" ,%bootstrap-mes)
            ("srfi-43" ,%srfi-43 )))
         (_
diff --git a/gnu/packages/commencement.scm b/gnu/packages/commencement.scm
index 4def178a8..43373b591 100644
--- a/gnu/packages/commencement.scm
+++ b/gnu/packages/commencement.scm
@@ -106,79 +106,6 @@
       (native-inputs '())
       (propagated-inputs '()))))
 
-(define mescc-tools-boot
-  (package-with-bootstrap-guile
-   (package
-     (inherit mescc-tools)
-     (name "mescc-tools-boot")
-     (version "0.5.2")
-     (source (origin
-               (method url-fetch)
-               (uri (string-append
-                     "http://git.savannah.nongnu.org/cgit/mescc-tools.git/snapshot/"
-                     name "-Release_" version
-                     ".tar.gz"))
-               (patches (search-patches "mescc-tools-boot.patch"))
-               (file-name (string-append "mescc-tools" "-" version ".tar.gz"))
-               (sha256
-                (base32
-                 "01x7bhmgwyf6mc2g1hcvibhps98nllacqm4f0j5l51b1mbi18pc2"))))
-     (inputs '())
-     (propagated-inputs '())
-     (native-inputs
-      `(("mescc-tools-seed" ,%mescc-tools-seed)
-        ("mes-source" ,(package-source mes-boot0))
-
-        ("bootstrap-mes" ,%bootstrap-mes)
-        ("coreutils" ,%bootstrap-coreutils&co)))
-     (build-system gnu-build-system)
-     (arguments
-      `(#:implicit-inputs? #f
-        #:guile ,%bootstrap-guile
-        #:strip-binaries? #f   ; binutil's strip b0rkes MesCC/M1/hex2 binaries
-        #:phases
-        (modify-phases %standard-phases
-          (add-after 'unpack 'unpack-seeds
-            (lambda* (#:key outputs #:allow-other-keys)
-              (let* ((coreutils (assoc-ref %build-inputs "coreutils"))
-                     (mescc-tools-seed (assoc-ref %build-inputs "mescc-tools-seed"))
-                     (mes-source (assoc-ref %build-inputs "mes-source"))
-                     (out (assoc-ref %outputs "out")))
-                (with-directory-excursion ".."
-                  (and
-                   (mkdir-p "mescc-tools-seed")
-                   (invoke "tar" "--strip=1" "-C" "mescc-tools-seed"
-                           "-xvf" mescc-tools-seed)
-                   (mkdir-p "mes-source")
-                   (invoke "tar" "--strip=1" "-C" "mes-source"
-                           "-xvf" mes-source)
-                  #t)))))
-          (replace 'configure
-            (lambda* (#:key outputs #:allow-other-keys)
-              (let ((coreutils (assoc-ref %build-inputs "coreutils"))
-                    (bootstrap-mes (assoc-ref %build-inputs "bootstrap-mes"))
-                    (out (assoc-ref %outputs "out")))
-                (setenv "PATH" (string-append coreutils "/bin"
-                                              ":" "../mescc-tools-seed"))
-                (format (current-error-port) "PATH=~s\n" (getenv "PATH"))
-                (setenv "PREFIX" out)
-                (setenv "MES_PREFIX" "../mes-source")
-                (setenv "MESCC_TOOLS_SEED" "../mescc-tools-seed")
-                (setenv "MES_SEED" (string-append bootstrap-mes "/lib"))
-                #t)))
-          (replace 'build
-            (lambda _
-              (invoke "sh" "build.sh")))
-          (replace 'check
-            (lambda _
-              ;; bootstrap build.sh lacks exec_enable, get_machine, and
-              ;; kaem_machine
-              ;; (invoke "sh" "check.sh")
-              #t))
-          (replace 'install
-            (lambda _
-              (invoke "sh" "install.sh")))))))))
-
 (define nyacc-boot
   (let ((version "0.86.0")
         (revision "0")
@@ -211,7 +138,7 @@
      (inputs '())
      (propagated-inputs '())
      (native-inputs
-      `(("mescc-tools" ,mescc-tools-boot)
+      `(("mescc-tools" ,%bootstrap-mescc-tools)
         ("nyacc-source" ,(package-source nyacc-boot))
 
         ("coreutils" , %bootstrap-coreutils&co)
@@ -284,7 +211,7 @@
                (setenv "MES_ARENA" "100000000")
                (setenv "DIFF" "sh scripts/diff.scm")
                ;; fail fast tests
-               ;; (invoke "sh" "-x" "build-aux/test.sh" "scaffold/tests/t")
+               (invoke "sh" "-x" "build-aux/test.sh" "scaffold/tests/t")
                ;; (invoke "sh" "-x" "build-aux/test.sh" "scaffold/tests/63-struct-cell")
                (invoke "sh" "check.sh"))))
           (replace 'install
@@ -332,7 +259,7 @@
        (propagated-inputs '())
        (native-inputs
         `(("mes" ,mes-boot)
-          ("mescc-tools" ,mescc-tools-boot)
+          ("mescc-tools" ,%bootstrap-mescc-tools)
           ("nyacc-source" ,(package-source nyacc-boot))
 
           ("coreutils" , %bootstrap-coreutils&co)
-- 
2.19.1


[-- Attachment #5: Type: text/plain, Size: 152 bytes --]


-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: Preparing the reduced bootstrap tarballs

[Jan Nieuwenhuizen]

Ludovic Courtès writes:

>> Ah right.  I saw that several times but did not use it.  I think because
>> initially it was of no use.  It would be nice if this built everything
>> we need, I agree :-)
>
> Heheh.  :-)  It’s what the “Building the Bootstrap Binaries” section
> describes.

Right...and I still forgot to add my new packages there...sigh.

>> Anyway, I think we/I will have to put some work into scripting
>> mescc-tools-seed or otherwise changing the mescc-tools-boot build.
>> WDYT?
>
> I’m confused: how did you build the seeds that (gnu packages bootstrap)
> refers to in ‘core-updates-next’?

I built them "by hand."  Mes used to have a `make seed' target.  I
dropped that.

> The goal is for the seeds to be built through Guix so we have a
> transparent and documented way to reproduce/verify them.

Sure.

> I could propose a patch to do that, though from what you’re saying
> generating ‘mescc-tools-seed’ is not something readily doable?

No, of course that's trivial.  I just sent patches to do that.  However,
building ascii seeds is not so trivial.  You need either M2-Planet or
Mes to produce those.  And then: ascii seeds is not what we want,
mescc-tools should be bootstrappable without any seeds, ascii or binary.

I don't see how to write a Guix packages to build mescc-tools without
any dependencies (apart from stage0 maybe).  It seems we still need a
mescc-tools-seed package anyway, which doesn't help?

janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: Preparing the reduced bootstrap tarballs

[Mark H Weaver]

ludo@gnu.org (Ludovic Courtès) writes:

> Jeremiah@pdp10.guru skribis:
>
>>> So if you like, please make that change.  There is only one little
>>> thing: I have no (scripted) recipe to create mescc-tools-seed-XYZ.  But
>>> wait: I have a great excuse for that...I was too lazy or too sloppy.
>>
>> I do, in mescc-tools-seed; the script bootstrap.sh when run with the
>> option "sin" will build the mescc-tools-seed binaries using mescc-tools.
>> The .M1 files are always generated by cc_x86.s using the C source files.
>
> I saw this script but it’s not entirely clear to me how to package the
> whole thing.  We don’t have a “stage0” package for instance in Guix, do
> we?
>
>>> WDYT?
>> I think we will end up having several versions of mescc-tools-seed; as
>> each architecture guix supports will end up needing a variant if we plan
>> on keeping them small. (I also have no idea how to make a multi-arch fat
>> elf binary)
>
> For now let’s focus on x86_64/i686.  :-)
>
> IMO we should change the seeds as rarely as possible because they are
> managed “out-of-band” and verifying them is difficult (you need to fetch
> the right Guix commit, run “guix build bootstrap-tarballs”, and compare
> the result—assuming this is all bit-reproducible.)
>
> The one we’re using today in Guix date back to 2013.

I think it's important that the new bootstrap-tarballs be
bit-reproducible, such that they can be independently verified by anyone
who wishes to do so.

In particular, *I* would like to independently verify them, on my own
laptops where I have avoided using binary substitutes for a long time,
and which I keep with me at all times.

My hope until now is that when we generated our existing bootstrap
binaries in 2013, Guix was too marginal a project to attract the
attention of hackers who might wish to compromise our bootstrap.  In
2018, as Guix has become more popular, we might well be considered a
worthy target of such efforts.

      Mark

Re: Preparing the reduced bootstrap tarballs

[Ludovic Courtès]

Hello,

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

>> IMO we should change the seeds as rarely as possible because they are
>> managed “out-of-band” and verifying them is difficult (you need to fetch
>> the right Guix commit, run “guix build bootstrap-tarballs”, and compare
>> the result—assuming this is all bit-reproducible.)
>>
>> The one we’re using today in Guix date back to 2013.
>
> Okay, I am proposing the attached patches, also available on my 
> core-updates-next at http://gitlab.com/janneke/guix
>
> Still I hope these won't last for five more years ;-)

Well I hope these will last for at least a couple of years, or at least
this is what we should aim for.

Replacing bootstrap binaries is not anecdotal as I wrote because these
are “binary blobs” that need a special procedure to verify them; that
they are smaller blobs is of course a crucial improvement, but it
doesn’t change the fact that one may want to verify them.

> From 5702107a31f52a615c516084b7a82d9f5e2967e0 Mon Sep 17 00:00:00 2001
> From: Jan Nieuwenhuizen <janneke@gnu.org>
> Date: Fri, 16 Nov 2018 19:29:55 +0100
> Subject: [PATCH 1/3] bootstrap: Add mescc-tools-static,
>  mescc-tools-static-tarball.
>
> * gnu/packages/make-bootstrap.scm (%mescc-tools-static,
> %mescc-tools-bootstrap-tarball):  New variable.

Neat; I’m glad it was as simple as this.  :-)

As discussed yesterday on IRC, please also make sure make-bootstrap.scm
does not refer to (gnu packages bootstrap) at all since that would be
like running in circles.

As discussed just now on IRC, we probably also need #:system
"i686-linux" in the appropriate places such that “guix build
bootstrap-tarballs” gives the same result on i686 and on x86_64.

> From cc3782c5f9d2c606a265be0dfdca44d79ec25622 Mon Sep 17 00:00:00 2001
> From: Jan Nieuwenhuizen <janneke@gnu.org>
> Date: Fri, 16 Nov 2018 21:30:00 +0100
> Subject: [PATCH 2/3] bootstrap: Add %bootstrap-mescc-tools.
>
> Built with
>     5702107a31f52a615c516084b7a82d9f5e2967e0 bootstrap: Add mescc-tools-static, mescc-tools-static-tarball.
>
> * gnu/packages/bootstrap.scm (%bootstrap-mescc-tools): New variable.

OK.

If you want, before pushing this patch, you can first push the one
above.  From there I (and possibly others!) will run “guix build
bootstrap-tarballs” and make sure we get the same result as you did.

> From daa9ff9463917687aabed8330eacb304a7e4deae Mon Sep 17 00:00:00 2001
> From: Jan Nieuwenhuizen <janneke@gnu.org>
> Date: Fri, 16 Nov 2018 21:30:28 +0100
> Subject: [PATCH 3/3] bootstrap: Replace %mescc-tools-seed with
>  %bootstrap-mescc-tools.
>
> * gnu/packages/bootstrap.scm (%mescc-tools-seed): Remove.
> (%bootstrap-inputs): Replace %mescc-tools-seed with %bootstrap-mescc-tools.
> * gnu/packages/commencement.scm (mes-boot): Likewise.

Also: (mescc-tools-boot): Remove.

> (tcc-boot0): Likewise.
> (mescc-tools-boot): Remove.

I suppose this is not directly related to the other changes, but anyway,
LGTM.

Thanks for the quick reply!

Ludo’.

Re: Preparing the reduced bootstrap tarballs

[Jeremiah]

> I think it's important that the new bootstrap-tarballs be
> bit-reproducible, such that they can be independently verified by anyone
> who wishes to do so.

Every Piece below M2-Planet has always been bit-reproducible.
In fact, each piece is designed in a way that you could by hand predict
what the resulting binary must be after any change.

and once I finally complete stage0; you would also have the blueprints
for making the virtual machine in hardware, hand toggle in the bits for
the hex0-monitor and have absolute proof that no trusting trust or Nexus
Intruder Class attacks have occurred in the creation of the binaries.

Every issue anyone is willing to bring, I will publicly address until
all bootstrap roots (even on arbitrary hardware) lead to the proof that
these binaries are perfectly reproducible and that they only behave in
the manner explicitly specified by the standards to which they conform.


> In particular, *I* would like to independently verify them, on my own
> laptops where I have avoided using binary substitutes for a long time,
> and which I keep with me at all times.
Already done; here are the steps currently for bootstrapping the
mescc-tools-seed and M2-Planet seed.M1:
git clone 'https://git.savannah.nongnu.org/git/stage0.git'
cd stage0
make test
cd ..
git clone 'https://git.savannah.nongnu.org/git/mescc-tools.git'
cd mescc-tools
make test
cd ..
git clone 'https://github.com/oriansj/mescc-tools-seed.git'
cd mescc-tools-seed
./bootstrap.sh sin

To generate the M2-Planet seed.M1 you need to either export
mescc-tools-seed's blood-elf, M1 and hex2 or mescc-tools (via copying
into your path or doing make install) then the steps to generate are as
follows:
git clone 'https://github.com/oriansj/M2-Planet.git'
cd M2-Planet
./bootstrap.sh refresh

Now you are done


> My hope until now is that when we generated our existing bootstrap
> binaries in 2013, Guix was too marginal a project to attract the
> attention of hackers who might wish to compromise our bootstrap.  In
> 2018, as Guix has become more popular, we might well be considered a
> worthy target of such efforts.

I like to go with the assumption that every binary is already
compromised; but by going back to the basics we can find and rip out
every single hook until we are finally secure.

I don't trust any hardware I can't or didn't make myself.
And the only root of trust we have is the ability to work as a
community, giving every member the ability to independently check our
assumptions and point out our mistakes. We will have false starts and
failures of imagination but we by working together will make us all a
dream that is too hard to achieve alone but easy now that we have each
other helping us all strive to a brighter future.

-Jeremiah

Re: Preparing the reduced bootstrap tarballs

[Mark H Weaver]

Hi Jeremiah,

Jeremiah@pdp10.guru writes:

>> I think it's important that the new bootstrap-tarballs be
>> bit-reproducible, such that they can be independently verified by anyone
>> who wishes to do so.
>
> Every Piece below M2-Planet has always been bit-reproducible.
> In fact, each piece is designed in a way that you could by hand predict
> what the resulting binary must be after any change.

Sure, and I'm glad to hear this.

However, my impression (correct me if I'm wrong) is that we are not yet
able to bootstrap Guix exclusively from M2-Planet.  For example, unless
I'm mistaken, we still need Guile in our bootstrap, and I'm guessing
that we are not yet able to build Guile exclusively from M2-Planet.
Is that right?

>> My hope until now is that when we generated our existing bootstrap
>> binaries in 2013, Guix was too marginal a project to attract the
>> attention of hackers who might wish to compromise our bootstrap.  In
>> 2018, as Guix has become more popular, we might well be considered a
>> worthy target of such efforts.
>
> I like to go with the assumption that every binary is already
> compromised;

I agree.  I didn't mean to imply that I consider our 2013 bootstrap
binaries trustworthy.  I very much look forward to the day when I
needn't blindly trust any precompiled binaries at all.

My only point is that if we cannot yet avoid blindly trusting
precompiled binaries, I have higher confidence in our 2013 binaries than
in binaries we would produce today, because (1) we are more likely to be
a target today because Guix has become far more popular, (2) I expect
that intelligence agencies have far more advanced tools today than they
did in 2013, and (3) I expect that governmental policies have become far
more favorable to permitting such attacks against projects such as ours.

> I don't trust any hardware I can't or didn't make myself.
> And the only root of trust we have is the ability to work as a
> community, giving every member the ability to independently check our
> assumptions and point out our mistakes. We will have false starts and
> failures of imagination but we by working together will make us all a
> dream that is too hard to achieve alone but easy now that we have each
> other helping us all strive to a brighter future.

I very much appreciate your work on this, which is quite important.

     Regards,
       Mark

Re: Preparing the reduced bootstrap tarballs

[Jan Nieuwenhuizen]

Ludovic Courtès writes:

Hi!

>> Still I hope these won't last for five more years ;-)
>
> Well I hope these will last for at least a couple of years, or at least
> this is what we should aim for.

Yes, we should aim for that while creating them.  I'm possibly a bit too
eager to replace the reduced binary seed bootstrap by a scheme only
bootstrap and reduce all of our seeds even further.

> Replacing bootstrap binaries is not anecdotal as I wrote because these
> are “binary blobs” that need a special procedure to verify them; that
> they are smaller blobs is of course a crucial improvement, but it
> doesn’t change the fact that one may want to verify them.

Fully agree.

> As discussed yesterday on IRC, please also make sure make-bootstrap.scm
> does not refer to (gnu packages bootstrap) at all since that would be
> like running in circles.

Thank you.  Right, that's fixed now.

> As discussed just now on IRC, we probably also need #:system
> "i686-linux" in the appropriate places such that “guix build
> bootstrap-tarballs” gives the same result on i686 and on x86_64.

I'm looking into that right now.

>> From cc3782c5f9d2c606a265be0dfdca44d79ec25622 Mon Sep 17 00:00:00 2001
>> From: Jan Nieuwenhuizen <janneke@gnu.org>
>> Date: Fri, 16 Nov 2018 21:30:00 +0100
>> Subject: [PATCH 2/3] bootstrap: Add %bootstrap-mescc-tools.
>>
>> Built with
>>     5702107a31f52a615c516084b7a82d9f5e2967e0 bootstrap: Add mescc-tools-static, mescc-tools-static-tarball.
>>
>> * gnu/packages/bootstrap.scm (%bootstrap-mescc-tools): New variable.
>
> OK.
>
> If you want, before pushing this patch, you can first push the one
> above.  From there I (and possibly others!) will run “guix build
> bootstrap-tarballs” and make sure we get the same result as you did.

Great, done.  I missed

    5b01b6034aeab32a5011c5757f18bd9772d3497d python: Honor '--cores=...' in tests.

happening yesterday; so pushed as

    fae3c0d953a7b0b552b4ce654ed9b5a51186f491
    bootstrap: Add mescc-tools-static, mescc-tools-static-tarball.

Note you'll also need something like these three extra commits

--8<---------------cut here---------------start------------->8---
2d8f7e2ac bootstrap: Update %bootstrap-tarballs.
57fbf077a bootstrap: Replace %mescc-tools-seed with %bootstrap-mescc-tools.
706574ee4 bootstrap: Add %bootstrap-mescc-tools.
--8<---------------cut here---------------end--------------->8---

from core-updates-next @ https://gitlab.com/janneke/guix to be able to
build `guix --system=i686-linux bootstrap-tarballs'.  Until then, the
recipe is

--8<---------------cut here---------------start------------->8---
./pre-inst-env guix build --system=i686-linux mes-stripped-tarball mescc-tools-static-tarball
--8<---------------cut here---------------end--------------->8---

Rebuilding that right now, let's compare what we get.

Thanks for all your input!
janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: Preparing the reduced bootstrap tarballs

[Jan Nieuwenhuizen]

Jan Nieuwenhuizen writes:

>> If you want, before pushing this patch, you can first push the one
>> above.  From there I (and possibly others!) will run “guix build
>> bootstrap-tarballs” and make sure we get the same result as you did.
>
> Great, done.  I missed
>
>     5b01b6034aeab32a5011c5757f18bd9772d3497d python: Honor '--cores=...' in tests.
>
> happening yesterday; so pushed as
>
>     fae3c0d953a7b0b552b4ce654ed9b5a51186f491
>     bootstrap: Add mescc-tools-static, mescc-tools-static-tarball.
>
> Note you'll also need something like these three extra commits
>
> 2d8f7e2ac bootstrap: Update %bootstrap-tarballs.
> 57fbf077a bootstrap: Replace %mescc-tools-seed with %bootstrap-mescc-tools.
> 706574ee4 bootstrap: Add %bootstrap-mescc-tools.
>
>
> from core-updates-next @ https://gitlab.com/janneke/guix to be able to
> build `guix --system=i686-linux bootstrap-tarballs'.  Until then, the
> recipe is
>
> ./pre-inst-env guix build --system=i686-linux mes-stripped-tarball mescc-tools-static-tarball
>
> Rebuilding that right now, let's compare what we get.

Just to give you a heads up: left to build this last night to find that
the python patch does not build for me.  I reverted the python patch on
my personal branch and start a new build.

Not sure what to do about this, YMMV but this is what I get

--8<---------------cut here---------------start------------->8---
building /gnu/store/k3h2iyv4b36a3d5ji7azgi8bcd8czyrz-python-minimal-3.7.0.drv...
[..]
-l and -j don't go together!
make: *** [Makefile:878: test] Error 2

Test suite failed, dumping logs.
Backtrace:
           4 (primitive-load "/gnu/store/9lz3rpgyrm1y97g6k9qg0f7a1h6?")
In ice-9/eval.scm:
   191:35  3 (_ _)
In srfi/srfi-1.scm:
   863:16  2 (every1 #<procedure 80fce20 at /gnu/store/4nhwwas62jab?> ?)
In /gnu/store/4nhwwas62jabgi8lmhimpg4v3fk2ynk1-module-import/guix/build/gnu-build-system.scm:
   799:28  1 (_ _)
    369:6  0 (check #:target _ #:make-flags _ #:tests? _ # _ # _ # _)

/gnu/store/4nhwwas62jabgi8lmhimpg4v3fk2ynk1-module-import/guix/build/gnu-build-system.scm:369:6: In procedure check:
Throw to key `srfi-34' with args `(#<condition &invoke-error [program: "make" arguments: ("test" "-j" "16" "EXTRATESTOPTS=-j16") exit-status: 2 term-signal: #f stop-signal: #f] 822ea20>)'.
builder for `/gnu/store/llvsk2pszk1cc7qwf6azi9877nhks59f-python2-2.7.15.drv' failed with exit code 1
build of /gnu/store/llvsk2pszk1cc7qwf6azi9877nhks59f-python2-2.7.15.drv failed
--8<---------------cut here---------------end--------------->8---

janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: Preparing the reduced bootstrap tarballs

[Jeremiah]

> However, my impression (correct me if I'm wrong) is that we are not yet
> able to bootstrap Guix exclusively from M2-Planet.
That is correct as the step of bootstrapping MesCC from M2-Planet is not
yet complete.
However once that is done, we can leverage Mes.c and gash to complete
the bootstrap of guix from that trusted reproducible source in a
reproducible fashion.

> For example, unless
> I'm mistaken, we still need Guile in our bootstrap, and I'm guessing
> that we are not yet able to build Guile exclusively from M2-Planet.
> Is that right?
We don't need it, so much as it is people wishing to avoid tedious work.
We already can bootstrap kaem without any shells or interpreters and it
can be used to run shell scripts that can perform the rest of the
bootstrap of a lisp or a proper shell.

I think because that work is less of a technical challenge that it has
been skipped.


> My only point is that if we cannot yet avoid blindly trusting
> precompiled binaries,

Depends on how restricted of an environment you ware willing to work in

> I have higher confidence in our 2013 binaries than
> in binaries we would produce today, because (1) we are more likely to be
> a target today because Guix has become far more popular, (2) I expect
> that intelligence agencies have far more advanced tools today than they
> did in 2013, and (3) I expect that governmental policies have become far
> more favorable to permitting such attacks against projects such as ours.

1) Granted
2) Not exactly; simply because the most advanced attack tool ever
invented was the Nexus Intruder Program in 1958. (Hardware that subverts
software that later subverts hardware designs and more software
[firmware, microcode, etc]). The tools might get more expensive but the
actual quality of attack tools depends on the teams and the market's
demand for pumping out vulnerable products and bugs. (Like the recent
Hard drive firmware attack which leveraged the vendor's cost cutting
process to hijack the drives and then lock out future attempts at
recovery.
3) Actually Government agencies are depending more and more on "Open
source tools" (Their words not mine) as software budgets have gotten
tighter and third party vendors integrate them more and more into their
commercial offerings purchased by Goverment agencies. Putting a backdoor
in the software most Government agencies depend upon, invites
vulnerabilities in our own Intelligence Agencies infrastructure and
increase the probablity that Spies will be identified before their
flight to their target country leaves the ground. To do such would not
only be suicidal for those Intelligence Agencies but also ensure
Cyberwarfare against the Countries they work for that much more
effective.

Now that isn't to say they consider that an extranality and doom us all
but nothing stays hidden when we can read the source and can DDC our
entire bootstrap across arbitrary hardware/operating system
combinations.

-Jeremiah

Re: Preparing the reduced bootstrap tarballs

[Mark H Weaver]

Hi,

Jeremiah@pdp10.guru writes:

>> However, my impression (correct me if I'm wrong) is that we are not yet
>> able to bootstrap Guix exclusively from M2-Planet.
> That is correct as the step of bootstrapping MesCC from M2-Planet is not
> yet complete.
> However once that is done, we can leverage Mes.c and gash to complete
> the bootstrap of guix from that trusted reproducible source in a
> reproducible fashion.
>
>> For example, unless
>> I'm mistaken, we still need Guile in our bootstrap, and I'm guessing
>> that we are not yet able to build Guile exclusively from M2-Planet.
>> Is that right?
> We don't need it, so much as it is people wishing to avoid tedious work.
> We already can bootstrap kaem without any shells or interpreters and it
> can be used to run shell scripts that can perform the rest of the
> bootstrap of a lisp or a proper shell.
>
> I think because that work is less of a technical challenge that it has
> been skipped.

I get the feeling that we are talking about two different things here.
I'm not saying that Guile is needed to bootstrap a GNU system from
source by hand.

When I say "we still need Guile in our bootstrap", I'm referring
specifically to the Guix "bootstrap binaries".  Guix uses Guile to
execute its automated build scripts.

Anyway, I've answered my own question by looking at:

  https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/make-bootstrap.scm?h=core-updates-next&id=d9ed0a46301e259c3082fcc67678e72ceae19a2d#n778

which indicates to me that although gcc, binutils, and glibc have been
eliminated from the Guix bootstrap binaries, they still include
statically-linked versions of Guile, Coreutils, Bzip2, XZ, Gawk, Tar,
and Grep.  These are the remaining precompiled binaries that I'm worried
about.

>> I have higher confidence in our 2013 binaries than
>> in binaries we would produce today, because (1) we are more likely to be
>> a target today because Guix has become far more popular, (2) I expect
>> that intelligence agencies have far more advanced tools today than they
>> did in 2013, and (3) I expect that governmental policies have become far
>> more favorable to permitting such attacks against projects such as ours.
>
> 1) Granted
> 2) Not exactly; simply because the most advanced attack tool ever
> invented was the Nexus Intruder Program in 1958. (Hardware that subverts
> software that later subverts hardware designs and more software
> [firmware, microcode, etc]). The tools might get more expensive but the
> actual quality of attack tools depends on the teams and the market's
> demand for pumping out vulnerable products and bugs. (Like the recent
> Hard drive firmware attack which leveraged the vendor's cost cutting
> process to hijack the drives and then lock out future attempts at
> recovery.

Interesting.  I didn't know about the Nexus Intruder Program, but even
so, I don't think this invalidates my point (2).

> 3) Actually Government agencies are depending more and more on "Open
> source tools" (Their words not mine) as software budgets have gotten
> tighter and third party vendors integrate them more and more into their
> commercial offerings purchased by Goverment agencies. Putting a backdoor
> in the software most Government agencies depend upon, invites
> vulnerabilities in our own Intelligence Agencies infrastructure and
> increase the probablity that Spies will be identified before their
> flight to their target country leaves the ground. To do such would not
> only be suicidal for those Intelligence Agencies but also ensure
> Cyberwarfare against the Countries they work for that much more
> effective.
>
> Now that isn't to say they consider that an extranality and doom us all

Right.  It is by now well accepted fact that the NSA has deliberately
weakened cryptographic standards, for example, and your argument above
would seem to argue against the truth of that fact as well.

> but nothing stays hidden when we can read the source and can DDC our
> entire bootstrap across arbitrary hardware/operating system
> combinations.

Here I disagree.  DDC is a crucially important step, but even after we
have DDC, a large number of security flaws will continue to stay hidden
in our code.  We have far too much code to audit, and too much of it
written in C, which makes it quite difficult to write secure code.
Common classes of security bugs in C such as signed integer overflows
and other undefined behavior can be very hard to find, and I have little
confidence that all of these bugs would be found in any practical audit.

Moreover, our current development practices ensure that new bugs of this
kind will be continuously reintroduced into our software, even as we
discover and fix many of the existing bugs.

So, your work is a very important part, but not a complete solution to
the problem of building a computer system that we can truly consider
trustworthy.

     Regards,
       Mark

Re: Preparing the reduced bootstrap tarballs

[Jan Nieuwenhuizen]

Mark H Weaver writes:

> Anyway, I've answered my own question by looking at:
>
>   https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/make-bootstrap.scm?h=core-updates-next&id=d9ed0a46301e259c3082fcc67678e72ceae19a2d#n778
>
> which indicates to me that although gcc, binutils, and glibc have been
> eliminated from the Guix bootstrap binaries, they still include
> statically-linked versions of Guile, Coreutils, Bzip2, XZ, Gawk, Tar,
> and Grep.  These are the remaining precompiled binaries that I'm worried
> about.

As far as I'm concerned our next step after the Reduced Binary Seed
bootstrap--that is yet to be released from core-updates-next---is the
Scheme-only bootstrap.

The plan is to have Guile and Gash initially replace all coreutils&co
and build them from source early in the bootstrap phase, right after
having built tcc and GNU make.  We have managed to build GNU make
without any dependency on any of the tool you mention (except for
Guile).

We're not there yet, gash cannot build mes or tcc.  Also, the latest mes
0.18 release cannot support buillding bash yet.  Lots of fun hacking
todo!

A next step could be to replace bootstrap-guile with mes, or create a
real small version of bootstrappable Guile.

Greetings,
janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: Preparing the reduced bootstrap tarballs

[Giovanni Biscuolo]

[-- Attachment #1: Type: text/plain, Size: 1514 bytes --]

Hi Jeremiah,

Jeremiah@pdp10.guru writes:

[...]

> and once I finally complete stage0; you would also have the blueprints
> for making the virtual machine in hardware,

so, if I don't get it wrong, every skilled engineer will be able to
build an "almost analogic" (zero bit of software preloaded) computing
machine ad use stage0/mes [1] as the "metre" [2] to calibrate all other
computing machines (thanks to reproducible builds)?

> hand toggle in the bits for the hex0-monitor

the first bit of code have to be "manually" introduced in the machine,
right?
for the lazyer like me, what about a punched card? :-) 

> and have absolute proof that no trusting trust or Nexus
> Intruder Class attacks have occurred in the creation of the binaries.

I didn't know about Nexus Intruder attacks: could you please give me
some links to the relevant bibliography?

> Every issue anyone is willing to bring, I will publicly address until
> all bootstrap roots (even on arbitrary hardware) lead to the proof that
> these binaries are perfectly reproducible and that they only behave in
> the manner explicitly specified by the standards to which they
> conform.

so, having the scientific proof that binary conforms to source, there
will be noo need to trust (the untrastable)

thank you!

Ciao
Giovanni

[2] I still have not fully understood the relationship between stage0
and mes
[1] https://en.m.wikipedia.org/wiki/Metre

-- 
Giovanni Biscuolo

Xelera IT Infrastructures

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Preparing the reduced bootstrap tarballs

[jeremiah]

> so, if I don't get it wrong, every skilled engineer will be able to
> build an "almost analogic" (zero bit of software preloaded) computing
> machine ad use stage0/mes [1] as the "metre" [2] to calibrate all other
> computing machines (thanks to reproducible builds)?

well, I haven't thought of it in those terms but yes I guess that is one
of the properties of the plan.

> the first bit of code have to be "manually" introduced in the machine,
> right?
Correct, otherwise you'll have to deal with firmware/bios as a trust
vector to be concerned about.

> for the lazyer like me, what about a punched card? :-)
If someone is willing to figure out how to read a deck of punched cards
without software, I'd be interested in learning more.

> I didn't know about Nexus Intruder attacks: could you please give me
> some links to the relevant bibliography?
I'll see if I can dig those up for you.

> so, having the scientific proof that binary conforms to source, there
> will be noo need to trust (the untrastable)
Well, that is what someone else could do with it but not a direct goal
of the work.

-Jeremiah

Re: Preparing the reduced bootstrap tarballs

[Ricardo Wurmus]

jeremiah@pdp10.guru writes:

>> for the lazyer like me, what about a punched card? :-)
> If someone is willing to figure out how to read a deck of punched cards
> without software, I'd be interested in learning more.

Now that’s a project for a free hardware design that I’d happily work
on :)

Reading and transmitting of the bitstream can be done in hardware alone
(shift register, MUX, photoresistors, …), though I’d need to know what
kind of interface there is on the receiving side.  Just flash memory /
TTL shift register?

--
Ricardo

Re: Preparing the reduced bootstrap tarballs

[Timothy Sample]

Hi Jan,

Jan Nieuwenhuizen <janneke@gnu.org> writes:

> The plan is to have Guile and Gash initially replace all coreutils&co
> and build them from source early in the bootstrap phase, right after
> having built tcc and GNU make.  We have managed to build GNU make
> without any dependency on any of the tool you mention (except for
> Guile).
>
> We're not there yet, gash cannot build mes or tcc.  Also, the latest mes
> 0.18 release cannot support buillding bash yet.  Lots of fun hacking
> todo!

I wanted to let you know that I’ve been doing more work on the little
Shell backend that we were talking about earlier.  It’s kind of like the
worse-is-better version of Gash: it is certainly not as nice, but the
development pace is much faster.  I did take a look at just working on
Gash directly, but it looked like it was going to be a lot slower.

So, keeping in mind that software estimates are very unreliable, I would
say that I might have a workable interpreter in the next week or two.
There are three big features missing: globbing, asynchronous commands,
and arithmetic substitutions.  I am almost finished globbing,
asynchronous commands should be pretty easy, and I plan to leave
arithmetic substitutions on the road-map for as long as possible (it’s a
little bit boring).

After I get those first two features working, I am going to start
testing running build scripts for Bash.  I will contact you then in case
you have any new advice on which scripts are important or anything else.

> Greetings,
> janneke


-- Tim

Re: Preparing the reduced bootstrap tarballs

[Jeremiah]

> Now that’s a project for a free hardware design that I’d happily work
> on :)
Sweet

> Reading and transmitting of the bitstream can be done in hardware alone
> (shift register, MUX, photoresistors, …), though I’d need to know what
> kind of interface there is on the receiving side.  Just flash memory /
> TTL shift register?
Well ultimately that would need to be a memory bus that is connected to
their TTL DRAM (64KB chips) or TTL SRAM (so we wouldn't have to worry
about memory refresh or other sort of issues like that)
and setting the memory address block starting from Zero until end of
deck.
The Memory is Big Endian Bit, Byte and Word Addressed.

The number address pins needs to be atleast 9 (otherwise the bootstrap
binaries would exceed the address space the reader could write to) and
not to exceed 64 (As I don't plan on implementing a 128bit ALU with
Floating point and Vector instructions needed to comply with the 128bit
version of the spec). I am planning on having the memory data bus to be
8bits wide as all instructions and datatypes are multiples of 8bits.

I can provide a clock signal for when the next value is to be loaded or
a line for you to directly drive the memory chips.

Or we could go the uart route and simply leverage a classic serial
connector.


> I?ve thought about this a bit.  Would it be possible to go even more
> mechanical, and implement the SPI protocol in switches toggled by holes
> cut in some kind of insulating tape?
Well that mechanical aspect would be what you would expect in punched
cards or paper tape (I was planning on doing a paper tape reader to
allow easier manual inspection of outputs, inputs and binaries)

> You could have a couple of steel
> balls (bearings?) balanced on top of each other, connected to a circuit
> using brushes or springs.  Then, you place the tape in between the
> bearings.  When there?s a hole in the insulating tape, they make
> contact, and when there is no hole, no contact.
Yep, classical through-hole reader technology. Not as reliable as
optical sensors but much more low tech and no room for anyone to insert
anything funny with the sensors.


> Then, you have a row of
> holes for the SPI clock, and a row of holes for the SPI data.  All you
> would have to do is hook up the right voltage to it, and you could write
> to a flash ROM by pulling manually punched tape through it.  (You could
> use a crank or motor if the timing needs to be more precise.)
Well you could have a hole aligned with the values being read (say a 9th
bit) which would trigger a clock pulse to indicate reading of the next byte.


> Then, you could take an x200 and setup the bootrom to be whatever you
> wanted with no software whatsoever.

Not possible, there is a 700+KB microcode blob in that CPU, without
which the CPU fails to power on for more than 10 clock cycles.

> If you adjusted stage0 to set up a
> ?cache-as-RAM? environment (like they do in coreboot), you could do
> quite a bit of manoeuvring before having to initialize RAM.
Well the 4MB of space (or 6MB if you have Penryn chip) is more than
enough to bootstrap cc_x86 and be able to compile a self-hosting C compiler


> That is,
> you could probably set up a tiny Scheme environment and do more hardware
> initialization from there.

Well the hardware initialization isn't a concern as that is currently
well documented in libreboot's code. The hard part is those computers
are only going to be working for a few more years before tin whiskers
kill the whole lot of them (the transistion to lead-free solder occurred
during that time) and then we will have a working solution to hardware
that is no longer available, no longer works and we can't build more
that will exactly match those specs without considerable effort.

> Obviously all of this would be easier on a simpler computer, but then
> presumably you have to move from a simpler computer to a more
> complicated computer at some point (depending on your needs).
I assume a 64bit processor with 16+GB of memory, a SATA interface and a
network interface should be sufficiently complicated to bootstrap
everything we will need to conern ourselves with in the next couple
years. (We can always enhance a design of our own making)

The Low end will be a 16bit processor with only 64KB of RAM and will
probably require ugly coding to fit some of the programs in that address size.

> The nice
> thing about booting the x200 as a simple computer is that you can just
> expand its boundaries until it is full-featured.
One needs to be precise by what one means by full-featured. Would you
consider Minix or Linux 1.0 a full-featured kernel? Would the ability to
build GCC and guile be considered full-featured enough? Because I can
build productive environments down to a hex0-monitor in a few hundred
bytes of total memory.

> Sorry for derailing a bit, I just wanted to share some idle thoughts.
I actively encourage alternative perspectives and I love being told
how I can do this better. I look forward to more ideas and suggestions
from you on this subject later ^_^

Just an open reminder our #bootstrappable channel is always looking for
people intersted in these sorts of topics and we love hearing about what
you have created in this regard.

-Jeremiah

Re: Preparing the reduced bootstrap tarballs

[Jan Nieuwenhuizen]

Timothy Sample writes:

Hello Timothy!

>> The plan is to have Guile and Gash initially replace all coreutils&co
>> and build them from source early in the bootstrap phase, right after
>> having built tcc and GNU make.  We have managed to build GNU make
>> without any dependency on any of the tool you mention (except for
>> Guile).
>>
>> We're not there yet, gash cannot build mes or tcc.  Also, the latest mes
>> 0.18 release cannot support buillding bash yet.  Lots of fun hacking
>> todo!
>
> I wanted to let you know that I’ve been doing more work on the little
> Shell backend that we were talking about earlier.  It’s kind of like the
> worse-is-better version of Gash: it is certainly not as nice, but the
> development pace is much faster.  I did take a look at just working on
> Gash directly, but it looked like it was going to be a lot slower.

Oh, that's great news!  We have been making good progress on Gash.  The
parser has been completely revamped.  The next target is to add a
transformation that should result in a eval'able shelly Guile script,
ideally something that you'd want to write by hand.

To achieve that will take some time.  It would be nice if we could work
together shaping this shelly script language .

If Gash and Geesh would produce (after transformation) the same shelly
script language, we can also share the backend implementation, WDYT?

> So, keeping in mind that software estimates are very unreliable, I would
> say that I might have a workable interpreter in the next week or two.
> There are three big features missing: globbing, asynchronous commands,
> and arithmetic substitutions.  I am almost finished globbing,
> asynchronous commands should be pretty easy, and I plan to leave
> arithmetic substitutions on the road-map for as long as possible (it’s a
> little bit boring).

Hehe.  I had a look and you did make lots of progress!

> After I get those first two features working, I am going to start
> testing running build scripts for Bash.  I will contact you then in case
> you have any new advice on which scripts are important or anything else.

Great,
Keep up the good work!

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: Preparing the reduced bootstrap tarballs

[Timothy Sample]

Hi Jan,

Jan Nieuwenhuizen <janneke@gnu.org> writes:

> Timothy Sample writes:
>
>> I wanted to let you know that I’ve been doing more work on the little
>> Shell backend that we were talking about earlier.  It’s kind of like the
>> worse-is-better version of Gash: it is certainly not as nice, but the
>> development pace is much faster.  I did take a look at just working on
>> Gash directly, but it looked like it was going to be a lot slower.
>
> Oh, that's great news!  We have been making good progress on Gash.  The
> parser has been completely revamped.  The next target is to add a
> transformation that should result in a eval'able shelly Guile script,
> ideally something that you'd want to write by hand.

That’s what I like best about Gash.  My “shelly” code is a mess and no
one should write it by hand!  :)

> To achieve that will take some time.  It would be nice if we could work
> together shaping this shelly script language .

I agree, but I am hesitant to worry about this now.  The more I work on
Geesh the more I realize that building a shell is a big project.  I’ve
consistently been moving from worrying about making things work nicely
to just making them work.  I hope that once the basic features are in
place it will be a lot more fun to improve things gradually.  Does that
make sense?

> If Gash and Geesh would produce (after transformation) the same shelly
> script language, we can also share the backend implementation, WDYT?

I’m a little bit confused about this.  After we exchanged notes last
time, I thought we were going to share front-ends.  Now it seems you’ve
doubled-down on Gash’s PEG parser and want to share back-ends.  Why the
change in plans?  It makes more sense to me to share front-ends because
the parser in Geesh is quite complete.  Also, you already wrote some
code to transform its output into a Gash script.

>> So, keeping in mind that software estimates are very unreliable, I would
>> say that I might have a workable interpreter in the next week or two.
>> There are three big features missing: globbing, asynchronous commands,
>> and arithmetic substitutions.  I am almost finished globbing,
>> asynchronous commands should be pretty easy, and I plan to leave
>> arithmetic substitutions on the road-map for as long as possible (it’s a
>> little bit boring).
>
> Hehe.  I had a look and you did make lots of progress!

So have you!  It’s pretty funny that we both picked up working on this
at about the same time.  In hindsight, I should have contacted you
before diving in again in order to minimize duplicated work.  On the
other hand, it seems you’re guilty of the same thing!  Oh well.  ;)

I‘m not too worried about it, and I’m having fun.  I hope you are too.

>> After I get those first two features working, I am going to start
>> testing running build scripts for Bash.  I will contact you then in case
>> you have any new advice on which scripts are important or anything else.
>
> Great,
> Keep up the good work!

I skipped asynchronous commands and implemented a bunch of built-ins
instead.  Now Geesh will run most of Bash’s configure script.  It runs
until it tries to write “config.sub” which it writes incorrectly due to
an unset variable (ironically, it seems to be the “$SHELL” variable).
That means it chews through over 17,000 lines of configure script!  It
does this very s-l-o-w-l-y, but it’s exciting all the same.


-- Tim