hydra.gnu.org now provides substitutes for ‘guix pull’

hydra.gnu.org now provides substitutes for ‘guix pull’

[Ludovic Courtès]

Hello Guix,

Yesterday Clément and I discussed on IRC what it would take for Hydra to
build build-aux/hydra/guix-modular.scm (i.e., the derivations that
correspond to ‘guix pull’.)  Clément rightly suggested that it was a
matter of removing ‘--fresh-auto-compiled’ from hydra-eval-guile-jobs¹,
which I did, and it works:

  https://hydra.gnu.org/jobset/guix/modular/

The job runs infrequently so far, but we can try to increase the
frequency and see if poor hydra.gnu.org keeps up.

Ludo’.

¹ https://github.com/NixOS/hydra/blob/master/src/script/hydra-eval-guile-jobs.in

Re: hydra.gnu.org now provides substitutes for ‘guix pull’

[swedebugia]

On June 22, 2018 9:32:58 AM GMT+02:00, ludo@gnu.org wrote: 

> Hello Guix,
> 
> Yesterday Clément and I discussed on IRC what it would take for Hydra to
> build build-aux/hydra/guix-modular.scm (i.e., the derivations that
> correspond to 'guix pull'.)  Clément rightly suggested that it was a
> matter of removing '--fresh-auto-compiled' from hydra-eval-guile-jobs¹,
> which I did, and it works:
> 
> https://hydra.gnu.org/jobset/guix/modular/

Fantastic!
This really drastically cut the time to pull for me. Now the only thing
that takes a toll is to compute the guix-derivation. Can we
skip/workaround that too? Maybe we can deduce this from the build hydra
already did?

I'm guessing that the computation is needed to map a commit to a
derivation that then maps to a build hydra has a substitute for. Hydra
according to my understanding already did all this. Maybe we can
communicate this to guix pull somehow?

Example 1 
I choose a commit that I know hydra has build e.g. 8edebd3
-> guix pull sees that this commit is already build and computed by
hydra.
-> guix pull fetches the resulting derivation and substitute and build
only the locale profile stuff.

Example 2
I simply run guix pull without knowing what hydra has done
-> guix pull checks the latest commit to master
-> checks if hydra has built it: no
-> informs the user and gives them an option to choose a commit a few
days back that hydra has built instead (saves time, power and is
environmentally friendly :) )
-> based on the users choice moves forward.

What do you think?

Re: hydra.gnu.org now provides substitutes for ‘guix pull’

[Ludovic Courtès]

Hi,

swedebugia@riseup.net skribis:

> On June 22, 2018 9:32:58 AM GMT+02:00, ludo@gnu.org wrote: 
>
>> Hello Guix,
>> 
>> Yesterday Clément and I discussed on IRC what it would take for Hydra to
>> build build-aux/hydra/guix-modular.scm (i.e., the derivations that
>> correspond to 'guix pull'.)  Clément rightly suggested that it was a
>> matter of removing '--fresh-auto-compiled' from hydra-eval-guile-jobs¹,
>> which I did, and it works:
>> 
>> https://hydra.gnu.org/jobset/guix/modular/
>
> Fantastic!
> This really drastically cut the time to pull for me. Now the only thing
> that takes a toll is to compute the guix-derivation. Can we
> skip/workaround that too? Maybe we can deduce this from the build hydra
> already did?

No, it cannot be deduced.  Essentially this slow part is about finding
out what needs to be built from scratch, IOW, bootstrapping.  See
<https://debbugs.gnu.org/cgi/bugreport.cgi?bug=27284#133> for more info.

We can probably optimize it, but I’m not sure how.

Ludo’.

Re: hydra.gnu.org now provides substitutes for ‘guix pull’

[Pjotr Prins]

On Fri, Jun 22, 2018 at 09:32:58AM +0200, Ludovic Courtès wrote:
>   https://hydra.gnu.org/jobset/guix/modular/

Looks like this is no longer working.

A 'guix pull' takes quite a long time. Any prospects of getting
substitutes? Not a priority, but it would be nice.

Pj.

Re: hydra.gnu.org now provides substitutes for ‘guix pull’

[Ludovic Courtès]

Pjotr Prins <pjotr.public12@thebird.nl> skribis:

> On Fri, Jun 22, 2018 at 09:32:58AM +0200, Ludovic Courtès wrote:
>>   https://hydra.gnu.org/jobset/guix/modular/
>
> Looks like this is no longer working.
>
> A 'guix pull' takes quite a long time. Any prospects of getting
> substitutes? Not a priority, but it would be nice.

You do get substitutes if you use berlin.guixsd.org.  The problem is
that the hydra.gnu.org setup broke after some changes in the ‘guix pull’
machinery.  I didn’t investigate much on the grounds that I’d like us to
finally switch to berlin.guixsd.org and Cuirass for the main build farm.
We’re getting there!

Thanks,
Ludo’.

Re: hydra.gnu.org now provides substitutes for ‘guix pull’

[Pjotr Prins]

The title is somewhat in-apt right now ;). I went through pain
today.

Even so, specifying the commit with 'guix pull' is incredibly useful
because it allows us to use our own substitutes and deploy the same
software across machines.

Now 'guix pull' itself is rather slow. Would it be possible to use
Guix pack or archive on an existing ~/.config/guix/current graph? I am
looking to quickly deploy guix itself as binary blobs too. 

Any idea of what could work? Maybe I should just do a 

  ~/.config/guix/current/bin/guix package -i guix -p ~/opt/guix

and copy that profile. It should contain the exact same package tree,
right?

Pj.

Re: hydra.gnu.org now provides substitutes for ‘guix pull’

[Ludovic Courtès]

Hello,

Pjotr Prins <pjotr.public12@thebird.nl> skribis:

> The title is somewhat in-apt right now ;). I went through pain
> today.

Like I wrote earlier today and before that, I think we should now focus
on berlin.guixsd.org and Cuirass (rather than hydra.gnu.org and Hydra.)

You can get substitutes for ‘guix pull’ from berlin, and indeed, I
highly encourage doing so because ‘guix pull’ without substitutes can
take way too much time (when all the substitutes are available, it’s
1.5–2mn on my laptop.)

> Now 'guix pull' itself is rather slow. Would it be possible to use
> Guix pack or archive on an existing ~/.config/guix/current graph?

Substitutes *are* such an archive.

HTH!

Ludo’.

Re: hydra.gnu.org now provides substitutes for ‘guix pull’

[Pjotr Prins]

On Tue, Nov 06, 2018 at 02:30:08PM +0100, Ludovic Courtès wrote:
> > Now 'guix pull' itself is rather slow. Would it be possible to use
> > Guix pack or archive on an existing ~/.config/guix/current graph?
> 
> Substitutes *are* such an archive.

Hmmm. How do I create one? Installing on my Guix publish server (build
host) I never get a substitute for guix pull substituting from that.

For normal builds I get substitutes fine.

I raised an issue on debbugs to that effect.

Pj.

Re: hydra.gnu.org now provides substitutes for ‘guix pull’

[Pjotr Prins]

On Tue, Nov 06, 2018 at 04:55:45PM +0100, Pjotr Prins wrote:
> On Tue, Nov 06, 2018 at 02:30:08PM +0100, Ludovic Courtès wrote:
> > > Now 'guix pull' itself is rather slow. Would it be possible to use
> > > Guix pack or archive on an existing ~/.config/guix/current graph?
> > 
> > Substitutes *are* such an archive.
> 
> Hmmm. How do I create one? Installing on my Guix publish server (build
> host) I never get a substitute for guix pull substituting from that.
> 
> For normal builds I get substitutes fine.
> 
> I raised an issue on debbugs to that effect.

Another thing I find a little strange. When I install a package or an
environment it completes fine. When I add then --no-substitutes flag
and it always builds stuff like openssl and libgcrypt, pango and such
from source.

I.e., it creates a build system (I suppose). If the install is
complete it should not have to do that. 

Even if I do 

  guix environment -C guix

(which should include a build system, right?) and after completion
run the same with --no-substitutes it starts to build right away.

Any thoughts on that?

Pj.

Unexpected building of dependencies

[Ricardo Wurmus]

Pjotr Prins <pjotr.public12@thebird.nl> writes:

> Another thing I find a little strange. When I install a package or an
> environment it completes fine. When I add then --no-substitutes flag
> and it always builds stuff like openssl and libgcrypt, pango and such
> from source.

Could you provide a list of commands to reproduce this exactly?

-- 
Ricardo

Re: hydra.gnu.org now provides substitutes for ‘guix pull’

[Mark H Weaver]

Hi Pjotr,

Pjotr Prins <pjotr.public12@thebird.nl> writes:

> Another thing I find a little strange. When I install a package or an
> environment it completes fine. When I add then --no-substitutes flag
> and it always builds stuff like openssl and libgcrypt, pango and such
> from source.
>
> I.e., it creates a build system (I suppose). If the install is
> complete it should not have to do that. 
>
> Even if I do 
>
>   guix environment -C guix
>
> (which should include a build system, right?) and after completion
> run the same with --no-substitutes it starts to build right away.
>
> Any thoughts on that?

My first guess is that it has to do with grafting.  If I'm not mistaken,
in order to generate a graft derivation (.drv file), we need to know
which store items are referenced in the ungrafted outputs.  When
substitutes are enabled, Guix can get the lists of references from the
substitute servers.  When --no-substitutes is passed, and the ungrafted
outputs are not in the local store, they must be locally built in order
to determine the set of references.

Or at least that's my guess.  Ludovic could answer more definitively, as
he wrote the relevant code.

       Mark

Re: hydra.gnu.org now provides substitutes for ‘guix pull’

[Ludovic Courtès]

Pjotr Prins <pjotr.public12@thebird.nl> skribis:

> On Tue, Nov 06, 2018 at 02:30:08PM +0100, Ludovic Courtès wrote:
>> > Now 'guix pull' itself is rather slow. Would it be possible to use
>> > Guix pack or archive on an existing ~/.config/guix/current graph?
>> 
>> Substitutes *are* such an archive.
>
> Hmmm. How do I create one?

Suppose you have pulled commit X on a machine and you run ‘guix publish’
on that machine.

If you run ‘guix pull --commit=X’ on another machine that takes
substitutes from the first machine, you should definitely get
substitutes.

HTH!

Ludo’.