Bootstrap binaries

Bootstrap binaries

[Stefan]

Hi!

I'm playing around with the bootstrapping of Guix.  I figured out that at the very beginning executables for bash, mkdir, xz, tar, get downloaded into the store, which is done by some Guile, I think it is (default-guile).  Then a guile-*.tar.xz file is downloaded and extracted into the store by a crafted shell script using the former bash as shebang and the other executables for creation of the directory in the store and extraction.  This way we get the %bootstrap-guile and this is all in (gnu packages bootstrap).

Then the story continues in (gnu packages commencement), which uses %bootstrap-guile to build bootar, gash-boot, gash-utils-boot and stage0-posix.  The surprising part is that stage0-posix contains a kaem-optional-seed per architecture, which is a minimal shell implementation as another binary executable!

So up to this point these seven binary blobs are needed: (default-guile), bash, mkdir, xz, tar, %bootstrap-guile, kaem-optional-seed.

And up to this point three shell implementations are in use: bash, gash, kaem-optional-seed.

While playing around with all this I found a possibility to avoid five of these binary blobs, but it comes at the cost of using (default-guile) twice on the build-side.

• Build bootar-dirty with (default-guile), make use of #:allowed-references (list (default-guile) "out").
• Build %bootstrap-guile with (default-guile) and bootar-dirty, make use of #:allowed-references '().
• Build another bootar with %bootstrap-guile, make use of #:allowed-references (list %bootstrap-guile "out").

 From here on only %bootstrap-guile and bootar will be used.

• Build gash-boot.
• Build gash-utils-boot.
• Build STAGE0-POSIX-bootstrap, but deleting kaem-optional-seed and using gash-boot instead.

So at the cost of using (default-guile) twice on the build-side, the only remaining binary blobs are (default-guile) and guile-*.tar.xz.  The only shell implementation in use is gash.

Is this a possible and welcome alternative?

What are the implications of using (default-guile) on the build side?  Is this a bad idea?  Won't it (or some other Guile) be used anyway in future, if the guix-daemon is rewritten in Guile?

Is there a better Guile than (default-guile) to use?  Which is the Guile used by Guix itself?

Finally, I'm not clear about the advantage of trusting %bootstrap-guile alongside (default-guile).  Why should it be better to trust both than only one of them?  Well, the hash of %bootstrap-guile is checked.  But it is checked by (default-guile).


Bye

Stefan

Re: GSoC ideas

[Tomáš Čech]

[-- Attachment #1: Type: text/plain, Size: 865 bytes --]

On Sat, Feb 06, 2016 at 10:54:34AM -0800, Christopher Allan Webber wrote:
>Maybe we should have a place to collect ideas?
>
>Anyway, here would be my top items for GSoC:
>
> - An installer wizard (written in Guile of course!)
> - g-expressions which generate packages of the Guix package manager for
>   .deb and .rpm based distros, and work to get those upstream in Debian
>   and Fedora

What would be purpose of that? Do you plan to use some other store path so it
won't collide with Guix maintained installation on the same machine?

OTOH it may be interesting to generate only (in my case) RPM metadata
so native system's package manager can track Guix generated files and
use pre/post install scripts for interactions with Guix (to
install/remove/protect them). That would glue Guix to system even
better. Just wild idea when thinking about your idea...

S_W

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: GSoC ideas

[Christopher Allan Webber]

Tomáš Čech writes:

> On Sat, Feb 06, 2016 at 10:54:34AM -0800, Christopher Allan Webber wrote:
>>Maybe we should have a place to collect ideas?
>>
>>Anyway, here would be my top items for GSoC:
>>
>> - An installer wizard (written in Guile of course!)
>> - g-expressions which generate packages of the Guix package manager for
>>   .deb and .rpm based distros, and work to get those upstream in Debian
>>   and Fedora
>
> What would be purpose of that? Do you plan to use some other store path so it
> won't collide with Guix maintained installation on the same machine?
>
> OTOH it may be interesting to generate only (in my case) RPM metadata
> so native system's package manager can track Guix generated files and
> use pre/post install scripts for interactions with Guix (to
> install/remove/protect them). That would glue Guix to system even
> better. Just wild idea when thinking about your idea...
>
> S_W

Sorry, I was unclear.  What I'm talking about is Guix *as a package
manager* to be able to be available on foreigh distros so I can
encourage friends to do:

  apt-get install guix

... and then do user-space hacking with Guix, and use "guix environment"
for hacking on my projects...  I think we'd increase our userbase
dramatically by getting Guix as a user-space pacakge manager into
various distros.

The gexp part was because of the gexp that (iirc) generates the "tarball
install" method of things as inspiration... could we build something
similar that builds a .deb or .rpm of Guix as an install method?

 - Chris

Re: GSoC ideas

[Thompson, David]

On Mon, Feb 8, 2016 at 12:24 PM, Christopher Allan Webber
<cwebber@dustycloud.org> wrote:
> What I'm talking about is Guix *as a package
> manager* to be able to be available on foreigh distros so I can
> encourage friends to do:
>
>   apt-get install guix
>
> ... and then do user-space hacking with Guix, and use "guix environment"
> for hacking on my projects...  I think we'd increase our userbase
> dramatically by getting Guix as a user-space pacakge manager into
> various distros.

Such packages would be convenient for users of these distros, but
getting them into Debian and Fedora proper is unlikely because Guix
does not conform to the FHS.

- Dave

Re: GSoC ideas

[Christopher Allan Webber]

Thompson, David writes:

> On Mon, Feb 8, 2016 at 12:24 PM, Christopher Allan Webber
> <cwebber@dustycloud.org> wrote:
>> What I'm talking about is Guix *as a package
>> manager* to be able to be available on foreigh distros so I can
>> encourage friends to do:
>>
>>   apt-get install guix
>>
>> ... and then do user-space hacking with Guix, and use "guix environment"
>> for hacking on my projects...  I think we'd increase our userbase
>> dramatically by getting Guix as a user-space pacakge manager into
>> various distros.
>
> Such packages would be convenient for users of these distros, but
> getting them into Debian and Fedora proper is unlikely because Guix
> does not conform to the FHS.
>
> - Dave

So, I talked about this with Stefano Zacchiroli while at FOSDEM.  He
suggested that we could do /opt/g/ or something like that and we *could*
get Guix in as a package manager to Debian.  Sure, it wouldn't be able
to use the Hydra packages, but it's probably okayish.

(We *might* be able to get Guix in even using the Hydra packages if the
Guix daemon could have a wrapper which bind-mounts it to /gnu/ or
something, when it starts up?  That might be a bit over the top, but
maybe would allow us to have both.  I'm not so sure about this strategy
though, I've just been bouncing it around in my head.)

We could also build apt / rpm packages which we just provide ourselves,
on the Guix website, or by providing some PPA'ish thing.  I would have
preferred this myself when I started running Debian on Guix... and in
this route we could keep everything as it already is.  (But people will
have to take an extra step to "get to" those nice packages.)

 - Chris

Re: GSoC ideas

[Pjotr Prins]

On Mon, Feb 08, 2016 at 11:47:31AM -0800, Christopher Allan Webber wrote:
> So, I talked about this with Stefano Zacchiroli while at FOSDEM.  He
> suggested that we could do /opt/g/ or something like that and we *could*
> get Guix in as a package manager to Debian.  Sure, it wouldn't be able
> to use the Hydra packages, but it's probably okayish.

Unless users get binaries they won't be happy.

> We could also build apt / rpm packages which we just provide ourselves,
> on the Guix website, or by providing some PPA'ish thing.  I would have
> preferred this myself when I started running Debian on Guix... and in
> this route we could keep everything as it already is.  (But people will
> have to take an extra step to "get to" those nice packages.)

That looks like the way forward to me. Such a package can setup and
start the daemon - which is enough. No need to get the blessing from
the distributions themselves (will take time, but it will come - there
really is no difference with allowing foreign packages to work anyway).

Pj.

Re: GSoC ideas

[Diane Trout]

> That looks like the way forward to me. Such a package can setup and
> start the daemon - which is enough. No need to get the blessing from
> the distributions themselves (will take time, but it will come -
> there
> really is no difference with allowing foreign packages to work
> anyway).


I wrote a basic Debian recipe to build guix, create the build users,
and install the systemd config file.

https://github.com/detrout/debian-guix

Currently I've only split the guix package into the emacs components
and everything else. I'd thought about splitting the daemon out into
its own package, but I wasn't sure what the daemon depended on. 

The daemon is still using the default /gnu/store path, and the user
needs to manually run guix authorize if they want to use hydra
binaries. The package is currently based on the stable 0.9.0 release,
and I'm not sure how security updates make it into a guix store if you
without updating the scheme packaging source tree.

It might be nice to prompt the user if they wanted to authorize hydra
on install but that's not implemented.

Currently its unlikely to go into Debian because Debian policy requires
everything to be built from source, and currently the Guix build
process downloads some bootstrap binaries.

However with the current packaging "guix environment --pure bash --
bash" does give me a clean guix environment, and the guix info docs get
installed when Debian emacs can see them.

Diane

Guix on Debian (was: GSoC ideas)

[Christopher Allan Webber]

Diane Trout writes:

>> That looks like the way forward to me. Such a package can setup and
>> start the daemon - which is enough. No need to get the blessing from
>> the distributions themselves (will take time, but it will come -
>> there
>> really is no difference with allowing foreign packages to work
>> anyway).
>
>
> I wrote a basic Debian recipe to build guix, create the build users,
> and install the systemd config file.
>
> https://github.com/detrout/debian-guix
>
> Currently I've only split the guix package into the emacs components
> and everything else. I'd thought about splitting the daemon out into
> its own package, but I wasn't sure what the daemon depended on.
>
> The daemon is still using the default /gnu/store path, and the user
> needs to manually run guix authorize if they want to use hydra
> binaries. The package is currently based on the stable 0.9.0 release,
> and I'm not sure how security updates make it into a guix store if you
> without updating the scheme packaging source tree.
>
> It might be nice to prompt the user if they wanted to authorize hydra
> on install but that's not implemented.
>
> Currently its unlikely to go into Debian because Debian policy requires
> everything to be built from source, and currently the Guix build
> process downloads some bootstrap binaries.
>
> However with the current packaging "guix environment --pure bash --
> bash" does give me a clean guix environment, and the guix info docs get
> installed when Debian emacs can see them.
>
> Diane

Great work Diane!

Are those bootstrapping binaries really necessary for getting Guix
going?  I guess for some reason I thought if you did the whole
configure/make/etc dance it wouldn't be but maybe I'm wrong.

Maybe this is a good step towards getting a Guix .deb we self-host on
the Guix website?

 - Chris

Re: Guix on Debian (was: GSoC ideas)

[Leo Famulari]

On Tue, Feb 23, 2016 at 03:52:30PM -0800, Christopher Allan Webber wrote:
> Diane Trout writes:
> 
> >> That looks like the way forward to me. Such a package can setup and
> >> start the daemon - which is enough. No need to get the blessing from
> >> the distributions themselves (will take time, but it will come -
> >> there
> >> really is no difference with allowing foreign packages to work
> >> anyway).
> >
> >
> > I wrote a basic Debian recipe to build guix, create the build users,
> > and install the systemd config file.
> >
> > https://github.com/detrout/debian-guix
> >
> > Currently I've only split the guix package into the emacs components
> > and everything else. I'd thought about splitting the daemon out into
> > its own package, but I wasn't sure what the daemon depended on.
> >
> > The daemon is still using the default /gnu/store path, and the user
> > needs to manually run guix authorize if they want to use hydra
> > binaries. The package is currently based on the stable 0.9.0 release,
> > and I'm not sure how security updates make it into a guix store if you
> > without updating the scheme packaging source tree.
> >
> > It might be nice to prompt the user if they wanted to authorize hydra
> > on install but that's not implemented.
> >
> > Currently its unlikely to go into Debian because Debian policy requires
> > everything to be built from source, and currently the Guix build
> > process downloads some bootstrap binaries.
> >
> > However with the current packaging "guix environment --pure bash --
> > bash" does give me a clean guix environment, and the guix info docs get
> > installed when Debian emacs can see them.
> >
> > Diane
> 
> Great work Diane!
> 
> Are those bootstrapping binaries really necessary for getting Guix
> going?  I guess for some reason I thought if you did the whole
> configure/make/etc dance it wouldn't be but maybe I'm wrong.

My understanding is that if you alter the bootstrap binaries, the entire
dependency graph will change, forcing a rebuild of everything. And of
course, the altered binaries may present different interfaces, breaking
things as well.

Re: Guix on Debian (was: GSoC ideas)

[Ricardo Wurmus]

Leo Famulari <leo@famulari.name> writes:

>> Are those bootstrapping binaries really necessary for getting Guix
>> going?  I guess for some reason I thought if you did the whole
>> configure/make/etc dance it wouldn't be but maybe I'm wrong.
>
> My understanding is that if you alter the bootstrap binaries, the entire
> dependency graph will change, forcing a rebuild of everything. And of
> course, the altered binaries may present different interfaces, breaking
> things as well.

This is correct.  Back then we ran into trouble with our Guix
installation at work when someone modified permission bits on the
bootstrap binaries, causing a rebuild of everything.  It took us a while
to find out the cause and revert the change.

Some of the bootstrap binaries can be reproduced from source (if you
make sure to follow the Guix recipes), but others (like Guile IIRC)
don’t have reproducible build systems, so reproducing the exact same
binaries without using Guix is going to be very challenging.

I don’t know if it is possible and if it would make sense to cheat,
i.e. just lie to Guix about the hashes of the bootstrap binaries.

~~ Ricardo

Re: Guix on Debian (was: GSoC ideas)

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1303 bytes --]

On Wed, 24 Feb 2016 10:03:34 +0100
Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> wrote:

> Leo Famulari <leo@famulari.name> writes:
> 
>  [...]  
>  [...]  
> 
> This is correct.  Back then we ran into trouble with our Guix
> installation at work when someone modified permission bits on the
> bootstrap binaries, causing a rebuild of everything.  It took us a while
> to find out the cause and revert the change.
> 
> Some of the bootstrap binaries can be reproduced from source (if you
> make sure to follow the Guix recipes), but others (like Guile IIRC)
> don’t have reproducible build systems, so reproducing the exact same
> binaries without using Guix is going to be very challenging.
> 
> I don’t know if it is possible and if it would make sense to cheat,
> i.e. just lie to Guix about the hashes of the bootstrap binaries.
> 
> ~~ Ricardo
> 

What about taking it a step further and having a multi-level bootstrap
process like when we have the core-updates? If we bootstrap away enough times
would we end up with the bootstrap binaries we have now?

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Guix on Debian (was: GSoC ideas)

[Jookia]

On Wed, Feb 24, 2016 at 11:16:51AM +0200, Efraim Flashner wrote:
> What about taking it a step further and having a multi-level bootstrap
> process like when we have the core-updates? If we bootstrap away enough times
> would we end up with the bootstrap binaries we have now?

From what I understand the bootstrap binaries aren't reproducible yet. If they
were, it would solve this issue as we could build them again.

Bootstrap binaries

[Ludovic Courtès]

Jookia <166291@gmail.com> skribis:

> On Wed, Feb 24, 2016 at 11:16:51AM +0200, Efraim Flashner wrote:
>> What about taking it a step further and having a multi-level bootstrap
>> process like when we have the core-updates? If we bootstrap away enough times
>> would we end up with the bootstrap binaries we have now?
>
> From what I understand the bootstrap binaries aren't reproducible yet.

Depends on what kind of reproducibility we’re talking about.  It’s
simple to build bootstrap binaries:

  https://www.gnu.org/software/guix/manual/html_node/Bootstrapping.html#Building-the-Bootstrap-Binaries

I think they are all bit-for-bit reproducible (that is, you can use
--rounds=3 and everything is fine), except for Guile due to
<http://bugs.gnu.org/20272>.)

However, if you build them today, you’ll obviously get something
different from the bootstrap binaries we currently use, which were from
Guile 2.0.9, libc 2.19, GCC 4.7.2, some old Coreutils, etc.

HTH,
Ludo’.

Re: Bootstrap binaries

[Christopher Allan Webber]

Ludovic Courtès writes:

> Jookia <166291@gmail.com> skribis:
>
>> On Wed, Feb 24, 2016 at 11:16:51AM +0200, Efraim Flashner wrote:
>>> What about taking it a step further and having a multi-level bootstrap
>>> process like when we have the core-updates? If we bootstrap away enough times
>>> would we end up with the bootstrap binaries we have now?
>>
>> From what I understand the bootstrap binaries aren't reproducible yet.
>
> Depends on what kind of reproducibility we’re talking about.  It’s
> simple to build bootstrap binaries:
>
>   https://www.gnu.org/software/guix/manual/html_node/Bootstrapping.html#Building-the-Bootstrap-Binaries
>
> I think they are all bit-for-bit reproducible (that is, you can use
> --rounds=3 and everything is fine), except for Guile due to
> <http://bugs.gnu.org/20272>.)
>
> However, if you build them today, you’ll obviously get something
> different from the bootstrap binaries we currently use, which were from
> Guile 2.0.9, libc 2.19, GCC 4.7.2, some old Coreutils, etc.
>
> HTH,
> Ludo’.

It seems like a good idea, once that bug in Guile is fixed, to move over
to a new set of bootstrap binaries... even if this involves some
difficulty for Guix users today.  It would certainly be a good thing to
do before we hit 1.0, whenever that is.

Re: Bootstrap binaries

[Ludovic Courtès]

Christopher Allan Webber <cwebber@dustycloud.org> skribis:

> Ludovic Courtès writes:
>
>> Jookia <166291@gmail.com> skribis:
>>
>>> On Wed, Feb 24, 2016 at 11:16:51AM +0200, Efraim Flashner wrote:
>>>> What about taking it a step further and having a multi-level bootstrap
>>>> process like when we have the core-updates? If we bootstrap away enough times
>>>> would we end up with the bootstrap binaries we have now?
>>>
>>> From what I understand the bootstrap binaries aren't reproducible yet.
>>
>> Depends on what kind of reproducibility we’re talking about.  It’s
>> simple to build bootstrap binaries:
>>
>>   https://www.gnu.org/software/guix/manual/html_node/Bootstrapping.html#Building-the-Bootstrap-Binaries
>>
>> I think they are all bit-for-bit reproducible (that is, you can use
>> --rounds=3 and everything is fine), except for Guile due to
>> <http://bugs.gnu.org/20272>.)
>>
>> However, if you build them today, you’ll obviously get something
>> different from the bootstrap binaries we currently use, which were from
>> Guile 2.0.9, libc 2.19, GCC 4.7.2, some old Coreutils, etc.
>>
>> HTH,
>> Ludo’.
>
> It seems like a good idea, once that bug in Guile is fixed, to move over
> to a new set of bootstrap binaries... even if this involves some
> difficulty for Guix users today.  It would certainly be a good thing to
> do before we hit 1.0, whenever that is.

I prefer to change those binaries as rarely as possible.  Intuitively
(and unscientifically), it gives more confidence to keep using the same
old binaries wrt. Ken Thompson attacks.

Ludo’.

Re: Bootstrap binaries

[Jookia]

On Sat, Feb 27, 2016 at 12:19:04AM +0100, Ludovic Courtès wrote:
> I prefer to change those binaries as rarely as possible.  Intuitively
> (and unscientifically), it gives more confidence to keep using the same
> old binaries wrt. Ken Thompson attacks.

I'm not sure about that, if we could establish the binaries could be
reproducibly built using the current bootstrap binaries it sounds like it could
be fine. Having reproducible bootstrap binaries seems like something incredibly
useful especially for packagers that for whatever reason want to verify that the
binaries can be built with Guix before signing them.

> Ludo’.

Jookia.

Re: Bootstrap binaries

[Ludovic Courtès]

Jookia <166291@gmail.com> skribis:

> On Sat, Feb 27, 2016 at 12:19:04AM +0100, Ludovic Courtès wrote:
>> I prefer to change those binaries as rarely as possible.  Intuitively
>> (and unscientifically), it gives more confidence to keep using the same
>> old binaries wrt. Ken Thompson attacks.
>
> I'm not sure about that, if we could establish the binaries could be
> reproducibly built using the current bootstrap binaries it sounds like it could
> be fine. Having reproducible bootstrap binaries seems like something incredibly
> useful especially for packagers that for whatever reason want to verify that the
> binaries can be built with Guix before signing them.

We would have to update them every time we change GCC, Guile, Coreutils,
etc. or one of their dependencies, which sounds impractical or even
infeasible to me.

Ludo’.

Re: Bootstrap binaries

[Jookia]

On Sun, Feb 28, 2016 at 04:08:00PM +0100, Ludovic Courtès wrote:
> We would have to update them every time we change GCC, Guile, Coreutils,
> etc. or one of their dependencies, which sounds impractical or even
> infeasible to me.

Perhaps there's some miscommunication here- the problem right now is that the
binaries aren't reproducible at all. Having them switched to reproducible builds
once and then tagging the Guix commit used would probably be enough to fix the
problem. I don't think we need to rebuild them every time.

> Ludo’.

Jookia.

Re: Bootstrap binaries

[Christopher Allan Webber]

Jookia writes:

> On Sun, Feb 28, 2016 at 04:08:00PM +0100, Ludovic Courtès wrote:
>> We would have to update them every time we change GCC, Guile, Coreutils,
>> etc. or one of their dependencies, which sounds impractical or even
>> infeasible to me.
>
> Perhaps there's some miscommunication here- the problem right now is that the
> binaries aren't reproducible at all. Having them switched to reproducible builds
> once and then tagging the Guix commit used would probably be enough to fix the
> problem. I don't think we need to rebuild them every time.

Yes, this is what I was originally suggesting!

Re: Bootstrap binaries

[Ludovic Courtès]

Jookia <166291@gmail.com> skribis:

> On Sun, Feb 28, 2016 at 04:08:00PM +0100, Ludovic Courtès wrote:
>> We would have to update them every time we change GCC, Guile, Coreutils,
>> etc. or one of their dependencies, which sounds impractical or even
>> infeasible to me.
>
> Perhaps there's some miscommunication here- the problem right now is that the
> binaries aren't reproducible at all. Having them switched to reproducible builds
> once and then tagging the Guix commit used would probably be enough to fix the
> problem. I don't think we need to rebuild them every time.

Ah sure, I agree!  Though to be on the safe side, we’d need to do
diverse double compilation (DDC) of these binaries, à la Wheeler.

Ludo’.