bootstrap: i686-linux now builds without binutils, gcc seeds

bootstrap: i686-linux now builds without binutils, gcc seeds

[Jan Nieuwenhuizen]

We have been working on the wip-bootstrap branch with the aim to remove
bootstrap binary seeds.  At the moment this is for i686-linux only.

The first step was to build binutils-mesboot-2.20.1a,
glibc-mesboot-2.2.5 and gcc-mesboot-4.7.4 without the use of any binary
binutils, glibc or gcc.

The second step was to remove binutils, glibc and gcc from the
%bootstrap-binaries for i686-linux and replace them with the *-mesboot
binaries.  That only got us to build make-boot0, other bootstrap tools
seem (!) to need a newer glibc.

Currently, the i686-linux bootstrap binaries are ~230MB in size
(unpacked).  Removing binutils, glibc and gcc reduces this to ~130MB.

The end goal is what we call a `full source bootstrap'; to only have a
~500 byte binary seed: the hex0 monitor.  Getting there will take some
time.  Meanwhile we settle for a `reduced binary seed' bootstrap.

Sadly, bootstrapping a glibc newer than 2.2.5 proved to be non-trivial.
As a temporary workaround, we chose to increase the bootstrap debt again
and inject the old %bootstrap-glibc after building all *-mesboot
binaries.

Now we have a prototype bootstrap that needs

        ("i686-linux" `(("linux-libre-headers" ,%bootstrap-linux-libre-headers)
                        ("mescc-tools-seed" ,%mescc-tools-seed)
                        ("mes-seed" ,%mes-seed)
                        ("srfi-43" ,%srfi-43 )
                        ("tinycc-seed" ,%tinycc-seed)))
    ("coreutils&co" ,%bootstrap-coreutils&co)
    ("bash" ,%bootstrap-coreutils&co)))

plus, to build the rest of Guix

                        ("libc" ,%bootstrap-glibc)

To try it, do something like

     git clone git://git.savannah.nongnu.org/guix
     cd guix
     git checkout -b wip-bootstrap
     guix environment guix
     ./bootstrap
     ./configure
     make

and then, if you're feeling courageous and lucky

     ./pre-inst-env guix build --system=i686-linux hello

or otherwise, you can do as I did

     ./pre-inst-env guix build --system=i686-linux gcc-mesboot
     ./pre-inst-env guix build --system=i686-linux -e '(@@ (gnu packages commencement) make-boot0)'
     ./pre-inst-env guix build --system=i686-linux -e '(@@ (gnu packages commencement) bintutils-cross-boot0)'
     ./pre-inst-env guix build --system=i686-linux -e '(@@ (gnu packages commencement) gcc-cross-boot0)'
     ./pre-inst-env guix build --system=i686-linux -e '(@@ (gnu packages commencement) gcc-final)'
     ./pre-inst-env guix build --system=i686-linux -e '(@@ (gnu packages commencement) guile-final)'
     ...

We need to determine the next steps and we could use some help.  A rough
list of things to be done

    * cleanup the wip-bootstrap branch, merge it
    * replace %bootstrap-glibc with a bootstrapped glibc again+)
    * release mes 0.18
    * replace coreutils&co by bootstrapped packages
    * other architectures: x86_64, arm*, ...
    * replace mes-seed by a full source bootstrapped mes.M2
    * replace Bash with Gash and/or Geesh*)

Wea; think that with "just a little bit" of extra effort we can have Guix
x86 be the first GNU/Linux that was bootstrapped without the use of
binary C compiler seed.

Greetings,
janneke


+) After bootstrapping a static glibc-2.2.5, I tried building a shared
   2.2.5, 2.3, 2.3.4, 2.3.6, 2.4, 2.5 and 2.6.  When all failed, I
   invested a couple of days to patch 2.3.6...but that started to look
   like a gargantuan undertaking.  Iow, I'm stuck here.

*) Gash and Geesh are exciting new projects that have the mostly
   overlapping goal to create a posix/bash compatible shell in Guile.
   Although it may not be the highest priority from a full source
   bootstrap point of view, having a shell in Guile is very exciting and
   would help Guix a lot.  We need to evaluate gash and geesh and
   formulate a strategy.

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: bootstrap: i686-linux now builds without binutils, gcc seeds

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 4473 bytes --]

Jan Nieuwenhuizen <janneke@gnu.org> ezt írta (időpont: 2018. aug. 31., P
18:32):

> We have been working on the wip-bootstrap branch with the aim to remove
> bootstrap binary seeds.  At the moment this is for i686-linux only.
>
> The first step was to build binutils-mesboot-2.20.1a,
> glibc-mesboot-2.2.5 and gcc-mesboot-4.7.4 without the use of any binary
> binutils, glibc or gcc.
>
> The second step was to remove binutils, glibc and gcc from the
> %bootstrap-binaries for i686-linux and replace them with the *-mesboot
> binaries.  That only got us to build make-boot0, other bootstrap tools
> seem (!) to need a newer glibc.
>
> Currently, the i686-linux bootstrap binaries are ~230MB in size
> (unpacked).  Removing binutils, glibc and gcc reduces this to ~130MB.
>
> The end goal is what we call a `full source bootstrap'; to only have a
> ~500 byte binary seed: the hex0 monitor.  Getting there will take some
> time.  Meanwhile we settle for a `reduced binary seed' bootstrap.
>
> Sadly, bootstrapping a glibc newer than 2.2.5 proved to be non-trivial.
> As a temporary workaround, we chose to increase the bootstrap debt again
> and inject the old %bootstrap-glibc after building all *-mesboot
> binaries.
>
> Now we have a prototype bootstrap that needs
>
>         ("i686-linux" `(("linux-libre-headers"
> ,%bootstrap-linux-libre-headers)
>                         ("mescc-tools-seed" ,%mescc-tools-seed)
>                         ("mes-seed" ,%mes-seed)
>                         ("srfi-43" ,%srfi-43 )
>                         ("tinycc-seed" ,%tinycc-seed)))
>     ("coreutils&co" ,%bootstrap-coreutils&co)
>     ("bash" ,%bootstrap-coreutils&co)))
>
> plus, to build the rest of Guix
>
>                         ("libc" ,%bootstrap-glibc)
>
> To try it, do something like
>
>      git clone git://git.savannah.nongnu.org/guix
>      cd guix
>      git checkout -b wip-bootstrap
>      guix environment guix
>      ./bootstrap
>      ./configure
>      make
>
> and then, if you're feeling courageous and lucky
>
>      ./pre-inst-env guix build --system=i686-linux hello
>
> or otherwise, you can do as I did
>
>      ./pre-inst-env guix build --system=i686-linux gcc-mesboot
>      ./pre-inst-env guix build --system=i686-linux -e '(@@ (gnu packages
> commencement) make-boot0)'
>      ./pre-inst-env guix build --system=i686-linux -e '(@@ (gnu packages
> commencement) bintutils-cross-boot0)'
>      ./pre-inst-env guix build --system=i686-linux -e '(@@ (gnu packages
> commencement) gcc-cross-boot0)'
>      ./pre-inst-env guix build --system=i686-linux -e '(@@ (gnu packages
> commencement) gcc-final)'
>      ./pre-inst-env guix build --system=i686-linux -e '(@@ (gnu packages
> commencement) guile-final)'
>      ...
>
> We need to determine the next steps and we could use some help.  A rough
> list of things to be done
>
>     * cleanup the wip-bootstrap branch, merge it
>     * replace %bootstrap-glibc with a bootstrapped glibc again+)
>     * release mes 0.18
>     * replace coreutils&co by bootstrapped packages
>     * other architectures: x86_64, arm*, ...
>     * replace mes-seed by a full source bootstrapped mes.M2
>     * replace Bash with Gash and/or Geesh*)
>
> Wea; think that with "just a little bit" of extra effort we can have Guix
> x86 be the first GNU/Linux that was bootstrapped without the use of
> binary C compiler seed.
>
> Greetings,
> janneke
>
>
> +) After bootstrapping a static glibc-2.2.5, I tried building a shared
>    2.2.5, 2.3, 2.3.4, 2.3.6, 2.4, 2.5 and 2.6.  When all failed, I
>    invested a couple of days to patch 2.3.6...but that started to look
>    like a gargantuan undertaking.  Iow, I'm stuck here.
>
Hello Jan,
I'm currently on vacation, but I would be interested to have a look into
this issue when I get home. Could you send me some details?

>
> *) Gash and Geesh are exciting new projects that have the mostly
>    overlapping goal to create a posix/bash compatible shell in Guile.
>    Although it may not be the highest priority from a full source
>    bootstrap point of view, having a shell in Guile is very exciting and
>    would help Guix a lot.  We need to evaluate gash and geesh and
>    formulate a strategy.
>
> --
> Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
> Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
>
>

[-- Attachment #2: Type: text/html, Size: 5865 bytes --]

Re: bootstrap: i686-linux now builds without binutils, gcc seeds

[Jan Nieuwenhuizen]

Gábor Boskovits writes:

>     +) After bootstrapping a static glibc-2.2.5, I tried building a shared
>        2.2.5, 2.3, 2.3.4, 2.3.6, 2.4, 2.5 and 2.6.  When all failed, I
>        invested a couple of days to patch 2.3.6...but that started to look
>        like a gargantuan undertaking.  Iow, I'm stuck here.
>
> Hello Jan, I'm currently on vacation, but I would be interested to
> have a look into this issue when I get home. Could you send me some
> details?

Thank you!  First, enjoy your vacation!

I'll share some details here, but things are changing pretty fast; it
appears I'm unstuck.

Building glibc-2.3.6, I found many files to complain about missing
size_t, wchar_t, NULL, offsetof and some other stray missing types or
macros.

While it's usual to make some patches like a missing include, this
didn't feel OK.

Yesterday I got the advice onl #glibc to try build 2.16 or 2.17, so I
gave that a try.  Same troubles...So i got down to inspecting the
include chain for `stddef.h' that failed to supply size_t...I found an
empty stddef.h from linux-libre-headers.  There was no stddef.h from the
new libc OR from the old libc included.  Although I did set -I flags to
the old libc, and for many files it worked.

That got me supsecting that I used glibc's --with-headers wrongly.  I
had it point to the linux-libre-headers.

The (hopefully) major step forward now is that I added two packages:
mesboot-headers (early) and glibc-mesboot-headers (later), that combine
their libc header and the linux-libre-headers in one package; and point
--with-headers to that combination include.

With this change, all the missing types and declarations turned in to
duplicate defines :-) So I have been removing those again from the glibc
patches.  Currently I'm finding that glibc-2.16.0 needs some more
linux/*.h headers that were stripped out of the bootstrap libc.

                "a.out.h                ; for 2.2.5"
                "atalk.h"               ; for 2.2.5
                "errno.h"
                "falloc.h"
                "if_ether.h"            ; for 2.2.5
                "ioctl.h"
                "kernel.h"
                "limits.h"
                "netlink.h"             ; for 2.16.0
                "param.h"
                "prctl.h"               ; for 2.16.0
                "posix_types.h"
                "rtnetlink.h"           ; for 2.16.0
                "socket.h"
                "stddef.h"
                "swab.h"                ; for 2.2.5
                "sysctl.h"
                "sysinfo.h"             ; for 2.2.5
                "types.h"
                "version.h"             ; for 2.2.5

I'll keep you all informed here, thanks.

Greetings,
janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: bootstrap: i686-linux now builds without binutils, gcc seeds

[Ludovic Courtès]

Hello Janneke!

It’s taken me a while but I successfully built /gnu/store/58q6748dydqxrhrw8xn2gsf60djl6gvv-hello-2.10
for i686 from commit 5d8b7131c23d2285dd3546c59022dd5953508943 of
‘wip-bootstrap’.  :-)

--8<---------------cut here---------------start------------->8---
$ guix hash -r /gnu/store/58q6748dydqxrhrw8xn2gsf60djl6gvv-hello-2.10
005zi21nvhmygjam6vsvkgmw9awsmyacz6a65naxcblrpbra1g94
--8<---------------cut here---------------end--------------->8---

The “interesting commits” are these:

  5d8b7131c * gnu: mes: Oops, use mirror://gnu download url.
  1ff6d26ac * bootstrap: Replace GNU toolchain seeds with Mes for i686-linux.
  ca8895bdb * bootstrap: Add Mes bootstrap.
  be65bf958 * gnu: Add Mes bootstrap seeds.
  45856f886 * gnu: Add linux-libre-headers-bootstrap-tarball.
  cd9830fcd * guix: copy-linux-headers: Prepare for Mes bootstrap.
  6265040e7 * guix: package-from-tarball: Allow PROGRAM-TO-TEST to be #f.
  8d9c16ed3 * gnu: texinfo-boot0: Remove bootstrap leaks.
  f9a76a79c * gnu: bison-boot0: Remove bootstrap leaks.
  12394cc83 * gnu: m4-boot0: New variable.
  6891da335 * gnu: perl-boot0: Remove bootstrap leaks.
  f933b162c * gnu: file-boot0: Remove bootstrap leaks.
  9c0912a20 * gnu: findutils-boot0: Remove bootstrap leaks.
  53616d896 * gnu: diffutils-boot0: Remove bootstrap leaks.
  7503cee76 * bootstrap: %bootstrap-inputs+toolchain: Prepare for Mes bootstrap.
  80bd4a995 * bootstrap: %bootstrap-inputs: Prepare for Mes bootstrap.

I looked at the result and overall it LGTM!  So I think the next step is
to rebase the branch on ‘core-updates’ (or merge it) and rename it
‘core-updates-next’.  WDYT?  Ricardo?

Previously we discussed that “-s i686-linux” on x86_64 would lead to a
different graph compared to a native i686-linux run.  Is it still the
case?  It looks like 80bd4a995 does the right thing in that respect.

Some comments on things that I think could be improved, in no particular
order:

  • It would be good to update the “Bootstrapping” section of the
    manual—no urgency though.  It would be useful both to Guix
    developers and to outsiders interested in bootstrapping
    (Bootstrappable, Reproducible Builds, maybe projects like GNU/Linux
    From Scratch, etc.)

  • I think all the “mesboot” & co. packages in commencement.scam can be
    made private–i.e., changing ‘define-public’ to ‘define’.

  • There’s a couple of tests (for example in tests/debug-link.scm) that
    rely on %gcc-bootstrap, on the assumption that building it is
    cheap.  We should double-check that these are still okay on i686.

  • IWBN to add comments for gcc-mesboot1-wrapper,
    glibc-headers-mesboot, %bootstrap-inputs+toolchain, just a line or
    two giving some context.

  • I saw a couple of hanging parens in commencement.scm, they want to
    be next to their friends.  ;-)

  • Minor style issues:
    `(,@(substitute-keyword-arguments …)) → (substitute-keyword-arguments …)
    (zero? (system* …)) → (invoke …)

  • Could you add a couple of lines of explanation at the top of the new
    gnu/packages/patches/*.patch files, as we do for other patches?
    Some of them could also be simplified; for instance
    ‘glibc-boot-2.2.5.patch’ contains the diff of what looks like a
    leftover file.

  • For ‘binutils-mesboot0’: package/inherit → package (inherit …)
    since replacement for ‘binutils’ certainly won’t apply to
    ‘binutils-mesboot0’.

  • For commit 80bd4a995 for instance, instead of “Prepare for Mes
    bootstrap”, the message could be “Wrap input lists into thunks.” or
    something like that to clarify what the actual change is.

I noticed that:

  ./pre-inst-env guix graph -e '(begin (use-modules (guix utils)) (%current-system "i686-linux")(@@ (gnu packages commencement) gnu-make-boot0))' | dot -Tps > t.ps

shows lots of repetitions, which defeats eq? memoization, but the
problem is not new; I’ll try to look into fixing this.


As we were discussing on IRC, we’ll have to figure out what the next
step should be, once this branch has become ‘core-updates-next’.  One
interesting bit is to remove the mes.M1 seed, as you wrote.  Another one
might be to do x86_64 bootstrapping (using i686 binaries at the
beginning of the graph, until we reach gcc@4.9, at which point we should
be able to build x86_64 binaries; though maybe I’m just too naive here
:-)).  Another low-hanging fruit (I think!) is using Gash instead of
Bash, though that’s completely orthogonal.

Thoughts?

Thank you for the amazing work, and again apologies for taking so long
to get back to you!

Ludo’.

Re: bootstrap: i686-linux now builds without binutils, gcc seeds

[Pjotr Prins]

On Sun, Sep 16, 2018 at 09:24:20PM +0200, Ludovic Courtès wrote:
> Hello Janneke!
> 
> It’s taken me a while but I successfully built /gnu/store/58q6748dydqxrhrw8xn2gsf60djl6gvv-hello-2.10
> for i686 from commit 5d8b7131c23d2285dd3546c59022dd5953508943 of
> ‘wip-bootstrap’.  :-)

I knew you would be fast about it. Kudos Ludo!!

Pj.

Re: bootstrap: i686-linux now builds without binutils, gcc seeds

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

> Hello Janneke!
>
> It’s taken me a while but I successfully built /gnu/store/58q6748dydqxrhrw8xn2gsf60djl6gvv-hello-2.10
> for i686 from commit 5d8b7131c23d2285dd3546c59022dd5953508943 of
> ‘wip-bootstrap’.  :-)
>
> --8<---------------cut here---------------start------------->8---
> $ guix hash -r /gnu/store/58q6748dydqxrhrw8xn2gsf60djl6gvv-hello-2.10
> 005zi21nvhmygjam6vsvkgmw9awsmyacz6a65naxcblrpbra1g94
> --8<---------------cut here---------------end--------------->8---
>
> The “interesting commits” are these:
>
>   5d8b7131c * gnu: mes: Oops, use mirror://gnu download url.
>   1ff6d26ac * bootstrap: Replace GNU toolchain seeds with Mes for i686-linux.
>   ca8895bdb * bootstrap: Add Mes bootstrap.
>   be65bf958 * gnu: Add Mes bootstrap seeds.
>   45856f886 * gnu: Add linux-libre-headers-bootstrap-tarball.
>   cd9830fcd * guix: copy-linux-headers: Prepare for Mes bootstrap.
>   6265040e7 * guix: package-from-tarball: Allow PROGRAM-TO-TEST to be #f.
>   8d9c16ed3 * gnu: texinfo-boot0: Remove bootstrap leaks.
>   f9a76a79c * gnu: bison-boot0: Remove bootstrap leaks.
>   12394cc83 * gnu: m4-boot0: New variable.
>   6891da335 * gnu: perl-boot0: Remove bootstrap leaks.
>   f933b162c * gnu: file-boot0: Remove bootstrap leaks.
>   9c0912a20 * gnu: findutils-boot0: Remove bootstrap leaks.
>   53616d896 * gnu: diffutils-boot0: Remove bootstrap leaks.
>   7503cee76 * bootstrap: %bootstrap-inputs+toolchain: Prepare for Mes bootstrap.
>   80bd4a995 * bootstrap: %bootstrap-inputs: Prepare for Mes bootstrap.
>
> I looked at the result and overall it LGTM!  So I think the next step is
> to rebase the branch on ‘core-updates’ (or merge it) and rename it
> ‘core-updates-next’.  WDYT?  Ricardo?

Using Janneke’s impressive work as the beginning of core-updates-next
sounds good to me.

I’m sorry I haven’t yet been able to review wip-bootstrap myself.

--
Ricardo

Re: bootstrap: i686-linux now builds without binutils, gcc seeds

[Jan Nieuwenhuizen]

Ricardo Wurmus writes:

> Using Janneke’s impressive work as the beginning of core-updates-next
> sounds good to me.

Thanks!

> I’m sorry I haven’t yet been able to review wip-bootstrap myself.

As you probably saw, Ludo' is already keeping me busy and quite some
things will need some more work.  Thanks for chiming in!

janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: bootstrap: i686-linux now builds without binutils, gcc seeds

[Jan Nieuwenhuizen]

Ludovic Courtès writes:

> It’s taken me a while but I successfully built /gnu/store/58q6748dydqxrhrw8xn2gsf60djl6gvv-hello-2.10
> for i686 from commit 5d8b7131c23d2285dd3546c59022dd5953508943 of
> ‘wip-bootstrap’.  :-)

That's great!

> $ guix hash -r /gnu/store/58q6748dydqxrhrw8xn2gsf60djl6gvv-hello-2.10
> 005zi21nvhmygjam6vsvkgmw9awsmyacz6a65naxcblrpbra1g94

And...you found the lucky numbers ;-)

> Previously we discussed that “-s i686-linux” on x86_64 would lead to a
> different graph compared to a native i686-linux run.  Is it still the
> case?  It looks like 80bd4a995 does the right thing in that respect.

Almost...that is to say for all packages up to gnu-make-boot0 and
all packages mentioned below: `Remove bootstrap leaks.'

As discussed on IRC, I created bug
    https://debbugs.gnu.org/cgi/bugreport.cgi?bug=32749

> The “interesting commits” are these:
>
>   5d8b7131c * gnu: mes: Oops, use mirror://gnu download url.
>   1ff6d26ac * bootstrap: Replace GNU toolchain seeds with Mes for i686-linux.
>   ca8895bdb * bootstrap: Add Mes bootstrap.
>   be65bf958 * gnu: Add Mes bootstrap seeds.
>   45856f886 * gnu: Add linux-libre-headers-bootstrap-tarball.
>   cd9830fcd * guix: copy-linux-headers: Prepare for Mes bootstrap.
>   6265040e7 * guix: package-from-tarball: Allow PROGRAM-TO-TEST to be #f.
>   8d9c16ed3 * gnu: texinfo-boot0: Remove bootstrap leaks.
>   f9a76a79c * gnu: bison-boot0: Remove bootstrap leaks.
>   12394cc83 * gnu: m4-boot0: New variable.
>   6891da335 * gnu: perl-boot0: Remove bootstrap leaks.
>   f933b162c * gnu: file-boot0: Remove bootstrap leaks.
>   9c0912a20 * gnu: findutils-boot0: Remove bootstrap leaks.
>   53616d896 * gnu: diffutils-boot0: Remove bootstrap leaks.
>   7503cee76 * bootstrap: %bootstrap-inputs+toolchain: Prepare for Mes bootstrap.
>   80bd4a995 * bootstrap: %bootstrap-inputs: Prepare for Mes bootstrap.

So we'll probably want to look into that bug and either remove all the
`Remove bootstrap leaks' patches, or rewrite the *-boot0 packages from
getext-boot0 onward (gcc-boot0-wrapped ld-wrapper-boot3 glibc-final
binutils-final zlib-final gcc-final bash-final guile-final
glibc-utf8-locales-final gnu-make-final coreutils-final grep-final
sed-final).

> I looked at the result and overall it LGTM!  So I think the next step is
> to rebase the branch on ‘core-updates’ (or merge it) and rename it
> ‘core-updates-next’.  WDYT?  Ricardo?

Amazing news!  I think it's best to "wait" until the `bootstrap leak'
thing is resolved; wip-bootstrap once more?

> Some comments on things that I think could be improved, in no particular
> order:
>
>   • It would be good to update the “Bootstrapping” section of the
>     manual—no urgency though.  It would be useful both to Guix
>     developers and to outsiders interested in bootstrapping
>     (Bootstrappable, Reproducible Builds, maybe projects like GNU/Linux
>     From Scratch, etc.)

I have added this text to wip-bootstrap, image not shown inline here, WDYT?

--8<---------------cut here---------------start------------->8---
diff --git a/doc/guix.texi b/doc/guix.texi
index 19a497c74..5a2cc8155 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -210,6 +210,7 @@ GNU Distribution
 * Package Modules::             Packages from the programmer's viewpoint.
 * Packaging Guidelines::        Growing the distribution.
 * Bootstrapping::               GNU/Linux built from scratch.
+* Reduced Binary Seed Bootstrap::  A Bootstrap worthy of GNU.
 * Porting::                     Targeting another platform or kernel.
 
 System Installation
@@ -8657,6 +8658,7 @@ For information on porting to other architectures or kernels,
 * Package Modules::             Packages from the programmer's viewpoint.
 * Packaging Guidelines::        Growing the distribution.
 * Bootstrapping::               GNU/Linux built from scratch.
+* Reduced Binary Seed Bootstrap::  A Bootstrap worthy of GNU.
 * Porting::                     Targeting another platform or kernel.
 @end menu
 
@@ -23447,6 +23449,9 @@ Binutils, libc, and the other packages mentioned above---the
 These bootstrap binaries are ``taken for granted'', though we can also
 re-create them if needed (more on that later).
 
+For @code{i686-linux} the Guix bootstrap process is more elaborate,
+@pxref{Reduced Binary Seed Bootstrap}.
+
 @unnumberedsubsec Preparing to Use the Bootstrap Binaries
 
 @c As of Emacs 24.3, Info-mode displays the image, but since it's a
@@ -23600,6 +23605,70 @@ bootstrap GCC with a sequence of assemblers, interpreters, and compilers
 of increasing complexity, which could be built from source starting from
 a simple and auditable assembler.  Your help is welcome!
 
+@node Reduced Binary Seed Bootstrap
+@section The Reduced Binary Seed Bootstrap
+
+Guix---like other GNU/Linux distributions---is traditionally bootstrapped from
+from a set of bootstrap binaries: Bourne shell, command-line tools provided by
+GNU Coreutils, Awk, Findutils, `sed', and `grep' and Guile, GCC, Binutils, and
+the GNU C Library (@pxref{Bootstrapping}).  Usually, these bootstrap-binaries
+are ``taken for granted.''
+
+What does this mean, really?  By taking these binaries for granted, trusting
+Guix depends on the trusting these binaries to be correct and clean.  Therein
+lies a problem: the current combined size of these bootstrap-binaries is about
+250MB (@pxref{Bootstrappable Builds,,, mes, Mes Reference Manual}).  Auditing
+or even inspecting these is next to impossible.
+
+For @code{i686-linux}, Guix now features a ``Reduced Binary Seed'' bootstrap
+@footnote{We would like to say: ``Full Source Bootstrap'' and while we are
+working towards that it would be a hyperbole to use that term for what we do
+now.}.
+
+The Reduced Binary Seed bootstrap removes the most critical tools---from a
+trust perspective---from the bootstrap binaries: GCC, Binutils and the GNU C
+Library are replaced by: @code{mescc-tools-seed} (a tiny assembler and linker)
+@code{mes-seed} (a small Scheme Interpreter and a C compiler writen in Scheme)
+and @code{tinycc-seed} (the Mes C Library, built for TinyCC).  Using these new
+binary seeds and a new set of
+@c
+package descriptions@footnote{@c
+mescc-tools-boot,
+nyacc-boot,
+mes-boot,
+tcc-boot0,
+tcc-boot,
+make-mesboot0,
+diffutils-mesboot,
+binutils-mesboot0,
+gcc-core-mesboot,
+mesboot-headers,
+glibc-mesboot0,
+gcc-mesboot0,
+binutils-mesboot,
+make-mesboot,
+gcc-mesboot1,
+gcc-mesboot1-wrapper,
+glibc-headers-mesboot,
+glibc-mesboot,
+gcc-mesboot,
+gcc-mesboot-wrapper.
+}
+@c
+the ``missing'' Binutils, and the GNU C Library are built, from source.  From
+here on the more traditional bootstrap process resumes.  This approach has
+reduced the bootstrap binaries in size to about 130MB.  Work is ongoing to
+reduce this further.  If you are interested, join us on @code{#boottrappable}
+on the Freenode IRC network.
+
+@c ./pre-inst-env guix graph --type=bag -e '(begin (use-modules (guix packages)) (%current-system "i686-linux") (@@ (gnu packages commencement) gcc-mesboot))' > doc/images/gcc-mesboot-bag-graph.dot
+@c dot -T png doc/images/gcc-mesboot-bag-graph.dot > doc/images/gcc-mesboot-bag-graph.png
+
+Below is the generated dependency graph to for @code{gcc-mesboot} that builds
+the rest of GuixSD.
+
+@image{images/gcc-mesboot-bag-graph,6in,,Dependency graph of the gcc-mesboot}
+
 
 @node Porting
 @section Porting to a New Platform
--8<---------------cut here---------------end--------------->8---

>   • I think all the “mesboot” & co. packages in commencement.scam can be
>     made private–i.e., changing ‘define-public’ to ‘define’.

I agree; I kept gcc-mesboot (the final gcc) public, is that OK?

>   • There’s a couple of tests (for example in tests/debug-link.scm) that
>     rely on %gcc-bootstrap, on the assumption that building it is
>     cheap.  We should double-check that these are still okay on i686.
>
>   • IWBN to add comments for gcc-mesboot1-wrapper,
>     glibc-headers-mesboot, %bootstrap-inputs+toolchain, just a line or
>     two giving some context.

Done.  Adding the comment for gcc-mesboot1-wrapper, I wonder if this
package is currently necessary.  I "played" (ahum) a lot with building
gcc-mesboot1 (the first 4.7.4) already using --enable-shared.  At the
moment it's built using --disable-shared and gcc-mesboot (4.9.4) is the
first gcc built using --enable shared.

I probably kept gcc-mesboot1-wrapper even if it was no longer needed,
because adding/removing it is also cumbersome.

I'm testing right now to see if we can remove gcc-mesboot1-wrapper (and
will do so if it's not currently needed); that information will help
getting the comment more informational if it's still needed.

>   • I saw a couple of hanging parens in commencement.scm, they want to
>     be next to their friends.  ;-)

Oh, how sad :-(  Fixed!

>   • Minor style issues:
>     `(,@(substitute-keyword-arguments …)) → (substitute-keyword-arguments …)

Sure, removed.

>     (zero? (system* …)) → (invoke …)

Done, ah...I still overlooked some of these (wip-bootstrap began before
`invoke' I think).  Still have some (zero? (system " ....")).

>   • Could you add a couple of lines of explanation at the top of the new
>     gnu/packages/patches/*.patch files, as we do for other patches?
>     Some of them could also be simplified; for instance
>     ‘glibc-boot-2.2.5.patch’ contains the diff of what looks like a
>     leftover file.

I'm looking into this.  I tried to "document" the problems that I
encountered by capturing the build errors in a file called `adiff'.

That was an ugly but convenient hack, I agree this should removed and
put into words.

>   • For ‘binutils-mesboot0’: package/inherit → package (inherit …)
>     since replacement for ‘binutils’ certainly won’t apply to
>     ‘binutils-mesboot0’.

OK.

>   • For commit 80bd4a995 for instance, instead of “Prepare for Mes
>     bootstrap”, the message could be “Wrap input lists into thunks.” or
>     something like that to clarify what the actual change is.

Nice.  Put that in a TODO for the next squash of wip-bootstrap.

> I noticed that:
>
>   ./pre-inst-env guix graph -e '(begin (use-modules (guix utils)) (%current-system "i686-linux")(@@ (gnu packages commencement) gnu-make-boot0))' | dot -Tps > t.ps
>
> shows lots of repetitions, which defeats eq? memoization, but the
> problem is not new; I’ll try to look into fixing this.

Great, thanks.  Yes, I wondered about that!

> As we were discussing on IRC, we’ll have to figure out what the next
> step should be, once this branch has become ‘core-updates-next’.  One
> interesting bit is to remove the mes.M1 seed, as you wrote.

This is certainly something that Jeremiah Orians will be working on,
even if not directly by helping with the mes.M2 rewrite, certainly
indirectly by creating M2-Planet v2.0.  This has a timeframe of about
half a year.

> Another one might be to do x86_64 bootstrapping (using i686 binaries
> at the beginning of the graph, until we reach gcc@4.9, at which point
> we should be able to build x86_64 binaries; though maybe I’m just too
> naive here :-)).

Yes, I think Gábor wanted to look into this, he had the same kind of
remark on IRC, Gábor?

I'm also working on a wip-x86_64 branch for Mes; this has a timeframe of
at least a couple of months (well, unless our resources magically
multiply :-).  Even when Mes is 64 bits, we would need to revise the
whole bootstrap chain (think: patch tcc, replace gcc-2.95 with
gcc-3.x/4.x?).

> Another low-hanging fruit (I think!) is using Gash instead of Bash,
> though that’s completely orthogonal.

Yes!  I'd love to get this discussion going too, and start to do some
planning.  I integrated most of bournish into Gash, and then also
Timothy's Geesh came on my radar.  Interesting puzzle this...Timothy?

> Thank you for the amazing work, and again apologies for taking so long
> to get back to you!

Thanks!  To be fair, although wip-bootstrap has been "mostly working"
for quite some time now, I only got wip-bootstrap in a somewhat
reviewable state and fixed the integration with Guix since a week.

janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: bootstrap: i686-linux now builds without binutils, gcc seeds

[Ricardo Wurmus]

Hi janneke!

> I have added this text to wip-bootstrap, image not shown inline here,
> WDYT?
[…]
> +@node Reduced Binary Seed Bootstrap
> +@section The Reduced Binary Seed Bootstrap
> +
> +Guix---like other GNU/Linux distributions---is traditionally bootstrapped from
> +from a set of bootstrap binaries: Bourne shell, command-line tools provided by

    ^— this is an extra “from”.

> +GNU Coreutils, Awk, Findutils, `sed', and `grep' and Guile, GCC, Binutils, and
> +the GNU C Library (@pxref{Bootstrapping}).  Usually, these bootstrap-binaries

“bootstrap binaries”

> +are ``taken for granted.''
> +
> +What does this mean, really?  By taking these binaries for granted, trusting
> +Guix depends on the trusting these binaries to be correct and clean.  Therein

How about this instead of the second sentence:

“Taking these binaries for granted means that we consider them to be a
correct and trustworthy ‘seed’ for building the complete system.”

> +lies a problem: the current combined size of these bootstrap-binaries is about

“bootstrap binaries”

> +250MB (@pxref{Bootstrappable Builds,,, mes, Mes Reference Manual}).  Auditing
> +or even inspecting these is next to impossible.
> +
> +For @code{i686-linux}, Guix now features a ``Reduced Binary Seed'' bootstrap
> +@footnote{We would like to say: ``Full Source Bootstrap'' and while we are
> +working towards that it would be a hyperbole to use that term for what we do
> +now.}.

“would be hyperbole” (no “a”)

> +The Reduced Binary Seed bootstrap removes the most critical tools---from a
> +trust perspective---from the bootstrap binaries: GCC, Binutils and the GNU C
> +Library are replaced by: @code{mescc-tools-seed} (a tiny assembler and linker)
> +@code{mes-seed} (a small Scheme Interpreter and a C compiler writen in Scheme)
> +and @code{tinycc-seed} (the Mes C Library, built for TinyCC).  Using these new
> +binary seeds and a new set of
> +@c
> +package descriptions@footnote{@c

I’d remove “descriptions”.

> +mescc-tools-boot,
> +nyacc-boot,
> +mes-boot,
> +tcc-boot0,
> +tcc-boot,
> +make-mesboot0,
> +diffutils-mesboot,
> +binutils-mesboot0,
> +gcc-core-mesboot,
> +mesboot-headers,
> +glibc-mesboot0,
> +gcc-mesboot0,
> +binutils-mesboot,
> +make-mesboot,
> +gcc-mesboot1,
> +gcc-mesboot1-wrapper,
> +glibc-headers-mesboot,
> +glibc-mesboot,
> +gcc-mesboot,

+ “and”

> +gcc-mesboot-wrapper.
> +}
> +@c
> +the ``missing'' Binutils, and the GNU C Library are built, from source.  From

“, from source” –> “ from source”.

> +here on the more traditional bootstrap process resumes.  This approach has
> +reduced the bootstrap binaries in size to about 130MB.  Work is ongoing to
> +reduce this further.  If you are interested, join us on @code{#boottrappable}

typo: #bootstrappable

> +on the Freenode IRC network.
> +
> +@c ./pre-inst-env guix graph --type=bag -e '(begin (use-modules (guix packages)) (%current-system "i686-linux") (@@ (gnu packages commencement) gcc-mesboot))' > doc/images/gcc-mesboot-bag-graph.dot
> +@c dot -T png doc/images/gcc-mesboot-bag-graph.dot > doc/images/gcc-mesboot-bag-graph.png
> +
> +Below is the generated dependency graph to for @code{gcc-mesboot} that builds
> +the rest of GuixSD.

“to for” –> “for”

How about this:

  Below is the generated dependency graph for @code{gcc-mesboot}, the
  bootstrap compiler used to build the rest of GuixSD.

>>   • I think all the “mesboot” & co. packages in commencement.scam can be
>>     made private–i.e., changing ‘define-public’ to ‘define’.
>
> I agree; I kept gcc-mesboot (the final gcc) public, is that OK?

If it isn’t referenced by other modules and would not be installed by
users I think it doesn’t need to be public, but I think either way is
fine.

Thanks!

--
Ricardo

Re: bootstrap: i686-linux now builds without binutils, gcc seeds

[Vincent Legoll]

Hello,

nice work here !

On Mon, Sep 17, 2018 at 9:48 PM Jan Nieuwenhuizen <janneke@gnu.org> wrote:

> --8<---------------cut here---------------start------------->8---
> +Guix---like other GNU/Linux distributions---is traditionally bootstrapped from
> +from a set of bootstrap binaries: Bourne shell, command-line tools provided by

double "from-from"

-- 
Vincent Legoll

Re: bootstrap: i686-linux now builds without binutils, gcc seeds

[Jan Nieuwenhuizen]

Ricardo Wurmus writes:

> Hi janneke!

Hi Ricardo!

>> +Guix---like other GNU/Linux distributions---is traditionally bootstrapped from
>> +from a set of bootstrap binaries: Bourne shell, command-line tools provided by
>
>     ^— this is an extra “from”.

thanks for pointing out all these mistakes and esp. for your
suggestions.  I used them all and am working towards a fresh
wip-bootstrap, now adding many small WIP commits that will be squashed
later.

> If it isn’t referenced by other modules and would not be installed by
> users I think it doesn’t need to be public, but I think either way is
> fine.

Ok, removed public here too.

janneke.

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: bootstrap: i686-linux now builds without binutils, gcc seeds

[Jan Nieuwenhuizen]

Vincent Legoll writes:

> nice work here !

Thanks!

>> +Guix---like other GNU/Linux distributions---is traditionally bootstrapped from
>> +from a set of bootstrap binaries: Bourne shell, command-line tools provided by
>
> double "from-from"

Thanks, fixed.
janneke

Re: bootstrap: i686-linux now builds without binutils, gcc seeds

[Jan Nieuwenhuizen]

Jan Nieuwenhuizen writes:

Hi Ludo'!

Replying to your initial review to notice the things that I initially
postponed and now have been done and the things that are still todo.

>> Previously we discussed that “-s i686-linux” on x86_64 would lead to a
>> different graph compared to a native i686-linux run.  Is it still the
>> case?  It looks like 80bd4a995 does the right thing in that respect.

I just push a fresh wip-bootstrap 6d975c901, and all differences between
native x86 and --system=i686-linux are now gone!

As discussed in bug-32749 I'm now using thunks for packages that use
package-with-explicit-inputs.  So, I reverted my previous `...leak'
commit rewrites.

Also, I changed more inputs to chunks (some ld-wrapper*) and I slightly
rewrote static-bash-for-glibc; just to make sure to look at
%current-system at run time, rather than load time.

>> I looked at the result and overall it LGTM!  So I think the next step is
>> to rebase the branch on ‘core-updates’ (or merge it) and rename it
>> ‘core-updates-next’.  WDYT?  Ricardo?

This new wip-bootstrap is rebased on core-updates, I think it's ready for a
rename to core-updates-next but wanted you all to have a look first.

>> Some comments on things that I think could be improved, in no particular
>> order:

>>   • There’s a couple of tests (for example in tests/debug-link.scm) that
>>     rely on %gcc-bootstrap, on the assumption that building it is
>>     cheap.  We should double-check that these are still okay on i686.

I haven't addressed this.

>>   • Could you add a couple of lines of explanation at the top of the new
>>     gnu/packages/patches/*.patch files, as we do for other patches?
>>     Some of them could also be simplified; for instance
>>     ‘glibc-boot-2.2.5.patch’ contains the diff of what looks like a
>>     leftover file.

I went through all patches, cleaned them up ad added comments.

Thanks!
janneke

Re: bootstrap: i686-linux now builds without binutils, gcc seeds

[Jan Nieuwenhuizen]

Jan Nieuwenhuizen writes:

> I just push a fresh wip-bootstrap 6d975c901, and all differences between
> native x86 and --system=i686-linux are now gone!

Just to give you a heads up: four `Oops' fixup-commits later I decided
to squash them again and create a fresh wip-bootstrap: d7feea557.

I think it could be ready for a rename to core-updates-next.

$ guix hash -r /gnu/store/y7wy6yr4j19f7d5xlqdln094nivy4cx3-hello-2.10
0ab0nazn6v0ybkig9xn58xjk4rqcyanp4wy4rywf32nyl8bgbly5

Thanks!
janneke

Re: bootstrap: i686-linux now builds without binutils, gcc seeds

[Jan Nieuwenhuizen]

Jan Nieuwenhuizen writes:

> Just to give you a heads up: four `Oops' fixup-commits later I decided
> to squash them again and create a fresh wip-bootstrap: d7feea557.
>
> I think it could be ready for a rename to core-updates-next.
>
> $ guix hash -r /gnu/store/y7wy6yr4j19f7d5xlqdln094nivy4cx3-hello-2.10
> 0ab0nazn6v0ybkig9xn58xjk4rqcyanp4wy4rywf32nyl8bgbly5

I finally realised what it really means that the GuixSD bootstrap has
`binutils-cross-boot0' (I really think that should be reflected in the
SCM name too?) and `gcc-cross-boot0': bolting the x86_64-linux bootstrap
onto the i686-linux Mes bootstrap proved to be "almost" trivial (very
low hanging fruit I think whas wat Ludovic said).

So, I took the freedom to add another commit onto wip-bootstrap that
also replaces the x86_64-linux bootstrap.  I realise that this commit
may have its own problems; so when moving to core-updates-next we may
want to be a bit careful before taking it in.

Greetings,
janneke

Re: bootstrap: i686-linux now builds without binutils, gcc seeds

[Ludovic Courtès]

Hello!

(And sorry for the delay; I reckon my mail scheduling algorithm leaves a
lot to be desired…)

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

> Ludovic Courtès writes:

[...]

>> $ guix hash -r /gnu/store/58q6748dydqxrhrw8xn2gsf60djl6gvv-hello-2.10
>> 005zi21nvhmygjam6vsvkgmw9awsmyacz6a65naxcblrpbra1g94
>
> And...you found the lucky numbers ;-)

It’s a “proof of work”, and one that’s nicer than SHA1 collisions.  :-)

> +@node Reduced Binary Seed Bootstrap
> +@section The Reduced Binary Seed Bootstrap

I really like it!  I was wondering whether to make it a subsection of
“Bootstrapping”.  Thoughts?

Regardless we should update the “Preparing to Use the Bootstrap
Binaries” section (and its image) as well as “Reducing the Set of
Bootstrap Binaries” (at least adding a cross-reference to the new
section.)

(Again I’m not suggesting to do it right away, this can come later!)

Thank you!

Ludo’.

Re: bootstrap: i686-linux now builds without binutils, gcc seeds

[Ludovic Courtès]

Hello!

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

> Jan Nieuwenhuizen writes:
>
>> Just to give you a heads up: four `Oops' fixup-commits later I decided
>> to squash them again and create a fresh wip-bootstrap: d7feea557.
>>
>> I think it could be ready for a rename to core-updates-next.
>>
>> $ guix hash -r /gnu/store/y7wy6yr4j19f7d5xlqdln094nivy4cx3-hello-2.10
>> 0ab0nazn6v0ybkig9xn58xjk4rqcyanp4wy4rywf32nyl8bgbly5
>
> I finally realised what it really means that the GuixSD bootstrap has
> `binutils-cross-boot0' (I really think that should be reflected in the
> SCM name too?) and `gcc-cross-boot0': bolting the x86_64-linux bootstrap
> onto the i686-linux Mes bootstrap proved to be "almost" trivial (very
> low hanging fruit I think whas wat Ludovic said).

Awesome, really great.

Hopefully ARMv7 and AArch64 can be addressed similarly in one go when we
get there.

> So, I took the freedom to add another commit onto wip-bootstrap that
> also replaces the x86_64-linux bootstrap.  I realise that this commit
> may have its own problems; so when moving to core-updates-next we may
> want to be a bit careful before taking it in.

I think as long as we can build Hello or similar things and “make check”
doesn’t show new regressions, we should be fine.

I’ve looked at the latest set of commits that ends with
78b8c9af45e7e261b22b198886daffcf12f22a56 and it LGTM!  So I think you
can go ahead and push as ‘core-updates-next’.

Something to keep in mind regarding “bootstrap: Add Mes bootstrap
seeds.”: once the branch is in place, we should regenerate the seeds
from the branch, upload them to ftp.gnu.org, adjust bootstrap.scm and
record in the commit log which commit we used to generated those seeds
(hoping they are bit-reproducible.)

Anyway, thumbs up for making what’s to my knowledge the distro with the
smallest set of binary seeds!

Thank you!

Ludo’.

Re: bootstrap: i686-linux now builds without binutils, gcc seeds

[Jan Nieuwenhuizen]

Ludovic Courtès writes:

Hi Ludo',

> (And sorry for the delay; I reckon my mail scheduling algorithm leaves a
> lot to be desired…)

AFAICS, esp. what's coming your way you schedule way better than I do ;)

>> +@node Reduced Binary Seed Bootstrap
>> +@section The Reduced Binary Seed Bootstrap
>
> I really like it!  I was wondering whether to make it a subsection of
> “Bootstrapping”.  Thoughts?

Yes, let's do that.  The reason I put it in a different section
altogether was mainly because it was only for i686-linux at the time.
Also, I thought it would be/grow bigger -- let's address that whenever
that happens.

> Regardless we should update the “Preparing to Use the Bootstrap
> Binaries” section (and its image) as well as “Reducing the Set of
> Bootstrap Binaries” (at least adding a cross-reference to the new
> section.)

Ah, yes and this may need some thought/work.  I can imagine having
"Reducing the Set of Bootstrap Binaries" first that mentions the problem
and wish for reducing the set and auditing, naturally flow into the
new bootstrap that takes steps works in that direction.  But we probably
need to mention the Reduced bootstrap earlier...hmm.

> (Again I’m not suggesting to do it right away, this can come later!)

OK, I'm keeping a TODO list.

janneke

Re: bootstrap: i686-linux now builds without binutils, gcc seeds

[Jan Nieuwenhuizen]

Ludovic Courtès writes:

Hi!

>> I finally realised what it really means that the GuixSD bootstrap has
>> `binutils-cross-boot0' (I really think that should be reflected in the
>> SCM name too?) and `gcc-cross-boot0': bolting the x86_64-linux bootstrap
>> onto the i686-linux Mes bootstrap proved to be "almost" trivial (very
>> low hanging fruit I think whas wat Ludovic said).
>
> Awesome, really great.
>
> Hopefully ARMv7 and AArch64 can be addressed similarly in one go when we
> get there.

Hmm, wondering how you see that...running the initial the Arm* bootstrap
on i686-linux and making the jump in *-cross-boot0?

I /am/ working on a native x86_64-linux Mes and one of the Arms could be
next...but you see a shortcut (at least temporary)?

>> So, I took the freedom to add another commit onto wip-bootstrap that
>> also replaces the x86_64-linux bootstrap.  I realise that this commit
>> may have its own problems; so when moving to core-updates-next we may
>> want to be a bit careful before taking it in.
>
> I think as long as we can build Hello or similar things and “make check”
> doesn’t show new regressions, we should be fine.

Hmm, make check showed some silly errors, I needed this patch

--8<---------------cut here---------------start------------->8---
test: Oops, syntax error and filter %bootstrap-inputs.

This is a fixup of

    86811e6faf bootstrap: %bootstrap-inputs: Wrap input lists into thunks.

* tests/builders.scm (%bootstrap-inputs): Oops, put parentheses in correct
place.
(%bootstrap-search-paths): Likewise.  Also, filter on package?.

1 file changed, 7 insertions(+), 5 deletions(-)
tests/builders.scm | 12 +++++++-----

modified   tests/builders.scm
@@ -28,7 +28,8 @@
   #:use-module (gcrypt hash)
   #:use-module (guix tests)
   #:use-module ((guix packages)
-                #:select (package-derivation package-native-search-paths))
+                #:select (package?
+                          package-derivation package-native-search-paths))
   #:use-module (gnu packages bootstrap)
   #:use-module (ice-9 match)
   #:use-module (srfi srfi-1)
@@ -47,14 +48,15 @@
        (map (match-lambda
              ((name package)
               (list name (package-derivation %store package))))
-            (@@ (gnu packages commencement) (%boot0-inputs)))))
+            ((@@ (gnu packages commencement) %boot0-inputs)))))
 
 (define %bootstrap-search-paths
   ;; Search path specifications that go with %BOOTSTRAP-INPUTS.
   (append-map (match-lambda
-               ((name package _ ...)
-                (package-native-search-paths package)))
-              (@@ (gnu packages commencement) (%boot0-inputs))))
+                ((name package _ ...)
+                 (package-native-search-paths package)))
+              (filter package?
+                      ((@@ (gnu packages commencement) %boot0-inputs)))))
 
 (define url-fetch*
   (store-lower url-fetch))
--8<---------------cut here---------------end--------------->8---

> I’ve looked at the latest set of commits that ends with
> 78b8c9af45e7e261b22b198886daffcf12f22a56 and it LGTM!  So I think you
> can go ahead and push as ‘core-updates-next’.

I folded the above patch into 86811e6faf for core-updates-next.  So,
wip-bootstrap and my local core-updates-next now differ in hash, but are
identical in git diff...

Hmm, I thought make check was OK now and almost pushed, but there are
still problems with make check on wip-bootstrap.  So I didn't dare to go
core-updates-next yet, if there's a good reason to do that anyway,
please do so or let me know.

I'll be looking into this later tonight, I could do with some help, just
in case someone is feeling bored ;)

> Something to keep in mind regarding “bootstrap: Add Mes bootstrap
> seeds.”: once the branch is in place, we should regenerate the seeds
> from the branch, upload them to ftp.gnu.org, adjust bootstrap.scm and
> record in the commit log which commit we used to generated those seeds
> (hoping they are bit-reproducible.)

Of course I agree, added to TODO.  I fear we need something more than
hope to have them bit-reproducible.  I haven't really looked into this
so we may get lucky but I also haven't patched binutils-2.20 or
gcc-2.95.3, gcc-4.7.4 wrt reproducibility.

> Anyway, thumbs up for making what’s to my knowledge the distro with the
> smallest set of binary seeds!

Yay!  I think the way forward to reduce them further that I like best is
to assume Guile and start building a Gash/Geesh and the other utilities
from there.

It would be nice to one day replace bootstrap-guile with bootstrap-mes
(or bootstrap Guile directly...) just dreaming here.

Thanks for the support!
janneke

Re: bootstrap: i686-linux now builds without binutils, gcc seeds

[Jan Nieuwenhuizen]

Jan Nieuwenhuizen writes:

> Hmm, make check showed some silly errors, I needed this patch

And I needed another one to get tests/builders.scm to pass.  And
although it passes it builds most of mes bootstrap in ./test-tmp, which
takes quite a bit of time.

Moreover, quite some other problems

============================================================================
Testsuite summary for GNU Guix 0.15.0.1569-63288
============================================================================
# TOTAL: 791
# PASS:  771
# SKIP:  10
# XFAIL: 0
# FAIL:  10
# XPASS: 0
# ERROR: 0

plain core-updates only lists one failure (FAIL: tests/debug-link.scm).
So, I have a core-updates-next on my gitlab and a wip-bootstrap with two
added patches on our savannah.

I will continue looking into the other 10 problems, but am already stuck
for half an our on what might be wrong with the first...

FAIL: tests/packages.scm
FAIL: tests/gexp.scm
FAIL: tests/union.scm
FAIL: tests/debug-link.scm
FAIL: tests/graph.scm
FAIL: tests/guix-package.sh
FAIL: tests/guix-environment.sh

janneke

branched core-updates-next [WAS: Re: bootstrap: i686-linux now builds without binutils, gcc seeds]

[Jan Nieuwenhuizen]

Jan Nieuwenhuizen writes:

> Jan Nieuwenhuizen writes:
>
>> Hmm, make check showed some silly errors, I needed this patch
>
> And I needed another one to get tests/builders.scm to pass.  And
> although it passes it builds most of Mes bootstrap in ./test-tmp, which
> takes quite a bit of time.
>
> Moreover, quite some other problems

OK, I fixed all 10 tests that the Mes bootstrap broke; make check now
shows no regressions so I created core-updates-next
0db65c168fd6dec57a357735fe130c80feba5460.

(There is one failure that's also present in core-updates, I will help
to look at that too.)

Special attention for two tests:

  tests/builders.scm: "gnu-build-system"
  tests/union.scm: "union-build"

These test now pass but they build most of Mes bootstrap in ./test-tmp,
which takes quite a bit of time with the new Mes bootstrap.

I suspect this is what Ludovic remarked that some tests may assume that
`gcc' can be built rather fast.  Thoughts?

TODO:
  * regenerate Mes bootstrap seeds, put them on ftp.gnu.org
  * slow tests: "gnu-build-system", "union-build"
  * doc: make "Reduced Binary Seed Bootstrap" subsection in Bootstrapping
  * doc: update “Preparing to Use the Bootstrap Binaries”
  * doc: update “Reducing the Set of Bootstrap Binaries”

janneke

Re: bootstrap: i686-linux now builds without binutils, gcc seeds

[Joshua Branson]

Jan Nieuwenhuizen <janneke@gnu.org> writes:

> Ludovic Courtès writes:
>
> Hi!
>
> Yay!  I think the way forward to reduce them further that I like best is
> to assume Guile and start building a Gash/Geesh and the other utilities
> from there.
>

Can you link to Gash/Geesh?  I can't find it via google.

>
> Thanks for the support!
> janneke

Re: bootstrap: i686-linux now builds without binutils, gcc seeds

[Jonathan Brielmaier]

On 9/23/18 6:09 PM, Joshua Branson wrote:
> Jan Nieuwenhuizen <janneke@gnu.org> writes:
> 
>> Ludovic Courtès writes:
>>
>> Hi!
>>
>> Yay!  I think the way forward to reduce them further that I like best is
>> to assume Guile and start building a Gash/Geesh and the other utilities
>> from there.
>>
> 
> Can you link to Gash/Geesh?  I can't find it via google.

https://gitlab.com/janneke/gash