Bootstrap binaries

Bootstrap binaries

[Ludovic Courtès]

Jookia <166291@gmail.com> skribis:

> On Wed, Feb 24, 2016 at 11:16:51AM +0200, Efraim Flashner wrote:
>> What about taking it a step further and having a multi-level bootstrap
>> process like when we have the core-updates? If we bootstrap away enough times
>> would we end up with the bootstrap binaries we have now?
>
> From what I understand the bootstrap binaries aren't reproducible yet.

Depends on what kind of reproducibility we’re talking about.  It’s
simple to build bootstrap binaries:

  https://www.gnu.org/software/guix/manual/html_node/Bootstrapping.html#Building-the-Bootstrap-Binaries

I think they are all bit-for-bit reproducible (that is, you can use
--rounds=3 and everything is fine), except for Guile due to
<http://bugs.gnu.org/20272>.)

However, if you build them today, you’ll obviously get something
different from the bootstrap binaries we currently use, which were from
Guile 2.0.9, libc 2.19, GCC 4.7.2, some old Coreutils, etc.

HTH,
Ludo’.

Re: Bootstrap binaries

[Christopher Allan Webber]

Ludovic Courtès writes:

> Jookia <166291@gmail.com> skribis:
>
>> On Wed, Feb 24, 2016 at 11:16:51AM +0200, Efraim Flashner wrote:
>>> What about taking it a step further and having a multi-level bootstrap
>>> process like when we have the core-updates? If we bootstrap away enough times
>>> would we end up with the bootstrap binaries we have now?
>>
>> From what I understand the bootstrap binaries aren't reproducible yet.
>
> Depends on what kind of reproducibility we’re talking about.  It’s
> simple to build bootstrap binaries:
>
>   https://www.gnu.org/software/guix/manual/html_node/Bootstrapping.html#Building-the-Bootstrap-Binaries
>
> I think they are all bit-for-bit reproducible (that is, you can use
> --rounds=3 and everything is fine), except for Guile due to
> <http://bugs.gnu.org/20272>.)
>
> However, if you build them today, you’ll obviously get something
> different from the bootstrap binaries we currently use, which were from
> Guile 2.0.9, libc 2.19, GCC 4.7.2, some old Coreutils, etc.
>
> HTH,
> Ludo’.

It seems like a good idea, once that bug in Guile is fixed, to move over
to a new set of bootstrap binaries... even if this involves some
difficulty for Guix users today.  It would certainly be a good thing to
do before we hit 1.0, whenever that is.

Re: Bootstrap binaries

[Ludovic Courtès]

Christopher Allan Webber <cwebber@dustycloud.org> skribis:

> Ludovic Courtès writes:
>
>> Jookia <166291@gmail.com> skribis:
>>
>>> On Wed, Feb 24, 2016 at 11:16:51AM +0200, Efraim Flashner wrote:
>>>> What about taking it a step further and having a multi-level bootstrap
>>>> process like when we have the core-updates? If we bootstrap away enough times
>>>> would we end up with the bootstrap binaries we have now?
>>>
>>> From what I understand the bootstrap binaries aren't reproducible yet.
>>
>> Depends on what kind of reproducibility we’re talking about.  It’s
>> simple to build bootstrap binaries:
>>
>>   https://www.gnu.org/software/guix/manual/html_node/Bootstrapping.html#Building-the-Bootstrap-Binaries
>>
>> I think they are all bit-for-bit reproducible (that is, you can use
>> --rounds=3 and everything is fine), except for Guile due to
>> <http://bugs.gnu.org/20272>.)
>>
>> However, if you build them today, you’ll obviously get something
>> different from the bootstrap binaries we currently use, which were from
>> Guile 2.0.9, libc 2.19, GCC 4.7.2, some old Coreutils, etc.
>>
>> HTH,
>> Ludo’.
>
> It seems like a good idea, once that bug in Guile is fixed, to move over
> to a new set of bootstrap binaries... even if this involves some
> difficulty for Guix users today.  It would certainly be a good thing to
> do before we hit 1.0, whenever that is.

I prefer to change those binaries as rarely as possible.  Intuitively
(and unscientifically), it gives more confidence to keep using the same
old binaries wrt. Ken Thompson attacks.

Ludo’.

Re: Bootstrap binaries

[Jookia]

On Sat, Feb 27, 2016 at 12:19:04AM +0100, Ludovic Courtès wrote:
> I prefer to change those binaries as rarely as possible.  Intuitively
> (and unscientifically), it gives more confidence to keep using the same
> old binaries wrt. Ken Thompson attacks.

I'm not sure about that, if we could establish the binaries could be
reproducibly built using the current bootstrap binaries it sounds like it could
be fine. Having reproducible bootstrap binaries seems like something incredibly
useful especially for packagers that for whatever reason want to verify that the
binaries can be built with Guix before signing them.

> Ludo’.

Jookia.

Re: Bootstrap binaries

[Ludovic Courtès]

Jookia <166291@gmail.com> skribis:

> On Sat, Feb 27, 2016 at 12:19:04AM +0100, Ludovic Courtès wrote:
>> I prefer to change those binaries as rarely as possible.  Intuitively
>> (and unscientifically), it gives more confidence to keep using the same
>> old binaries wrt. Ken Thompson attacks.
>
> I'm not sure about that, if we could establish the binaries could be
> reproducibly built using the current bootstrap binaries it sounds like it could
> be fine. Having reproducible bootstrap binaries seems like something incredibly
> useful especially for packagers that for whatever reason want to verify that the
> binaries can be built with Guix before signing them.

We would have to update them every time we change GCC, Guile, Coreutils,
etc. or one of their dependencies, which sounds impractical or even
infeasible to me.

Ludo’.

Re: Bootstrap binaries

[Jookia]

On Sun, Feb 28, 2016 at 04:08:00PM +0100, Ludovic Courtès wrote:
> We would have to update them every time we change GCC, Guile, Coreutils,
> etc. or one of their dependencies, which sounds impractical or even
> infeasible to me.

Perhaps there's some miscommunication here- the problem right now is that the
binaries aren't reproducible at all. Having them switched to reproducible builds
once and then tagging the Guix commit used would probably be enough to fix the
problem. I don't think we need to rebuild them every time.

> Ludo’.

Jookia.

Re: Bootstrap binaries

[Christopher Allan Webber]

Jookia writes:

> On Sun, Feb 28, 2016 at 04:08:00PM +0100, Ludovic Courtès wrote:
>> We would have to update them every time we change GCC, Guile, Coreutils,
>> etc. or one of their dependencies, which sounds impractical or even
>> infeasible to me.
>
> Perhaps there's some miscommunication here- the problem right now is that the
> binaries aren't reproducible at all. Having them switched to reproducible builds
> once and then tagging the Guix commit used would probably be enough to fix the
> problem. I don't think we need to rebuild them every time.

Yes, this is what I was originally suggesting!

Re: Bootstrap binaries

[Ludovic Courtès]

Jookia <166291@gmail.com> skribis:

> On Sun, Feb 28, 2016 at 04:08:00PM +0100, Ludovic Courtès wrote:
>> We would have to update them every time we change GCC, Guile, Coreutils,
>> etc. or one of their dependencies, which sounds impractical or even
>> infeasible to me.
>
> Perhaps there's some miscommunication here- the problem right now is that the
> binaries aren't reproducible at all. Having them switched to reproducible builds
> once and then tagging the Guix commit used would probably be enough to fix the
> problem. I don't think we need to rebuild them every time.

Ah sure, I agree!  Though to be on the safe side, we’d need to do
diverse double compilation (DDC) of these binaries, à la Wheeler.

Ludo’.

Bootstrap binaries

[Stefan]

Hi!

I'm playing around with the bootstrapping of Guix.  I figured out that at the very beginning executables for bash, mkdir, xz, tar, get downloaded into the store, which is done by some Guile, I think it is (default-guile).  Then a guile-*.tar.xz file is downloaded and extracted into the store by a crafted shell script using the former bash as shebang and the other executables for creation of the directory in the store and extraction.  This way we get the %bootstrap-guile and this is all in (gnu packages bootstrap).

Then the story continues in (gnu packages commencement), which uses %bootstrap-guile to build bootar, gash-boot, gash-utils-boot and stage0-posix.  The surprising part is that stage0-posix contains a kaem-optional-seed per architecture, which is a minimal shell implementation as another binary executable!

So up to this point these seven binary blobs are needed: (default-guile), bash, mkdir, xz, tar, %bootstrap-guile, kaem-optional-seed.

And up to this point three shell implementations are in use: bash, gash, kaem-optional-seed.

While playing around with all this I found a possibility to avoid five of these binary blobs, but it comes at the cost of using (default-guile) twice on the build-side.

• Build bootar-dirty with (default-guile), make use of #:allowed-references (list (default-guile) "out").
• Build %bootstrap-guile with (default-guile) and bootar-dirty, make use of #:allowed-references '().
• Build another bootar with %bootstrap-guile, make use of #:allowed-references (list %bootstrap-guile "out").

 From here on only %bootstrap-guile and bootar will be used.

• Build gash-boot.
• Build gash-utils-boot.
• Build STAGE0-POSIX-bootstrap, but deleting kaem-optional-seed and using gash-boot instead.

So at the cost of using (default-guile) twice on the build-side, the only remaining binary blobs are (default-guile) and guile-*.tar.xz.  The only shell implementation in use is gash.

Is this a possible and welcome alternative?

What are the implications of using (default-guile) on the build side?  Is this a bad idea?  Won't it (or some other Guile) be used anyway in future, if the guix-daemon is rewritten in Guile?

Is there a better Guile than (default-guile) to use?  Which is the Guile used by Guix itself?

Finally, I'm not clear about the advantage of trusting %bootstrap-guile alongside (default-guile).  Why should it be better to trust both than only one of them?  Well, the hash of %bootstrap-guile is checked.  But it is checked by (default-guile).


Bye

Stefan