all messages for Guix-related lists mirrored at yhetil.org
 help / color / mirror / code / Atom feed
* hydra.gnu.org now provides substitutes for ‘guix pull’
@ 2018-06-22  7:32 Ludovic Courtès
  2018-06-26 11:16 ` swedebugia
  2018-10-15  7:41 ` Pjotr Prins
  0 siblings, 2 replies; 12+ messages in thread
From: Ludovic Courtès @ 2018-06-22  7:32 UTC (permalink / raw)
  To: guix-devel; +Cc: guix-sysadmin, clement

Hello Guix,

Yesterday Clément and I discussed on IRC what it would take for Hydra to
build build-aux/hydra/guix-modular.scm (i.e., the derivations that
correspond to ‘guix pull’.)  Clément rightly suggested that it was a
matter of removing ‘--fresh-auto-compiled’ from hydra-eval-guile-jobs¹,
which I did, and it works:

  https://hydra.gnu.org/jobset/guix/modular/

The job runs infrequently so far, but we can try to increase the
frequency and see if poor hydra.gnu.org keeps up.

Ludo’.

¹ https://github.com/NixOS/hydra/blob/master/src/script/hydra-eval-guile-jobs.in

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: hydra.gnu.org now provides substitutes for ‘guix pull’
  2018-06-22  7:32 hydra.gnu.org now provides substitutes for ‘guix pull’ Ludovic Courtès
@ 2018-06-26 11:16 ` swedebugia
  2018-06-27 20:23   ` Ludovic Courtès
  2018-10-15  7:41 ` Pjotr Prins
  1 sibling, 1 reply; 12+ messages in thread
From: swedebugia @ 2018-06-26 11:16 UTC (permalink / raw)
  To: guix-devel, ludo; +Cc: guix-sysadmin, clement

On June 22, 2018 9:32:58 AM GMT+02:00, ludo@gnu.org wrote: 

> Hello Guix,
> 
> Yesterday Clément and I discussed on IRC what it would take for Hydra to
> build build-aux/hydra/guix-modular.scm (i.e., the derivations that
> correspond to 'guix pull'.)  Clément rightly suggested that it was a
> matter of removing '--fresh-auto-compiled' from hydra-eval-guile-jobs¹,
> which I did, and it works:
> 
> https://hydra.gnu.org/jobset/guix/modular/

Fantastic!
This really drastically cut the time to pull for me. Now the only thing
that takes a toll is to compute the guix-derivation. Can we
skip/workaround that too? Maybe we can deduce this from the build hydra
already did?

I'm guessing that the computation is needed to map a commit to a
derivation that then maps to a build hydra has a substitute for. Hydra
according to my understanding already did all this. Maybe we can
communicate this to guix pull somehow?

Example 1 
I choose a commit that I know hydra has build e.g. 8edebd3
-> guix pull sees that this commit is already build and computed by
hydra.
-> guix pull fetches the resulting derivation and substitute and build
only the locale profile stuff.

Example 2
I simply run guix pull without knowing what hydra has done
-> guix pull checks the latest commit to master
-> checks if hydra has built it: no
-> informs the user and gives them an option to choose a commit a few
days back that hydra has built instead (saves time, power and is
environmentally friendly :) )
-> based on the users choice moves forward.

What do you think?

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: hydra.gnu.org now provides substitutes for ‘guix pull’
  2018-06-26 11:16 ` swedebugia
@ 2018-06-27 20:23   ` Ludovic Courtès
  0 siblings, 0 replies; 12+ messages in thread
From: Ludovic Courtès @ 2018-06-27 20:23 UTC (permalink / raw)
  To: swedebugia; +Cc: guix-devel, guix-sysadmin, clement

Hi,

swedebugia@riseup.net skribis:

> On June 22, 2018 9:32:58 AM GMT+02:00, ludo@gnu.org wrote: 
>
>> Hello Guix,
>> 
>> Yesterday Clément and I discussed on IRC what it would take for Hydra to
>> build build-aux/hydra/guix-modular.scm (i.e., the derivations that
>> correspond to 'guix pull'.)  Clément rightly suggested that it was a
>> matter of removing '--fresh-auto-compiled' from hydra-eval-guile-jobs¹,
>> which I did, and it works:
>> 
>> https://hydra.gnu.org/jobset/guix/modular/
>
> Fantastic!
> This really drastically cut the time to pull for me. Now the only thing
> that takes a toll is to compute the guix-derivation. Can we
> skip/workaround that too? Maybe we can deduce this from the build hydra
> already did?

No, it cannot be deduced.  Essentially this slow part is about finding
out what needs to be built from scratch, IOW, bootstrapping.  See
<https://debbugs.gnu.org/cgi/bugreport.cgi?bug=27284#133> for more info.

We can probably optimize it, but I’m not sure how.

Ludo’.

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: hydra.gnu.org now provides substitutes for ‘guix pull’
  2018-06-22  7:32 hydra.gnu.org now provides substitutes for ‘guix pull’ Ludovic Courtès
  2018-06-26 11:16 ` swedebugia
@ 2018-10-15  7:41 ` Pjotr Prins
  2018-10-15 10:16   ` Ludovic Courtès
  1 sibling, 1 reply; 12+ messages in thread
From: Pjotr Prins @ 2018-10-15  7:41 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: guix-devel

On Fri, Jun 22, 2018 at 09:32:58AM +0200, Ludovic Courtès wrote:
>   https://hydra.gnu.org/jobset/guix/modular/

Looks like this is no longer working.

A 'guix pull' takes quite a long time. Any prospects of getting
substitutes? Not a priority, but it would be nice.

Pj.

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: hydra.gnu.org now provides substitutes for ‘guix pull’
  2018-10-15  7:41 ` Pjotr Prins
@ 2018-10-15 10:16   ` Ludovic Courtès
  2018-11-03 14:19     ` Pjotr Prins
  0 siblings, 1 reply; 12+ messages in thread
From: Ludovic Courtès @ 2018-10-15 10:16 UTC (permalink / raw)
  To: Pjotr Prins; +Cc: guix-devel

Pjotr Prins <pjotr.public12@thebird.nl> skribis:

> On Fri, Jun 22, 2018 at 09:32:58AM +0200, Ludovic Courtès wrote:
>>   https://hydra.gnu.org/jobset/guix/modular/
>
> Looks like this is no longer working.
>
> A 'guix pull' takes quite a long time. Any prospects of getting
> substitutes? Not a priority, but it would be nice.

You do get substitutes if you use berlin.guixsd.org.  The problem is
that the hydra.gnu.org setup broke after some changes in the ‘guix pull’
machinery.  I didn’t investigate much on the grounds that I’d like us to
finally switch to berlin.guixsd.org and Cuirass for the main build farm.
We’re getting there!

Thanks,
Ludo’.

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: hydra.gnu.org now provides substitutes for ‘guix pull’
  2018-10-15 10:16   ` Ludovic Courtès
@ 2018-11-03 14:19     ` Pjotr Prins
  2018-11-06 13:30       ` Ludovic Courtès
  0 siblings, 1 reply; 12+ messages in thread
From: Pjotr Prins @ 2018-11-03 14:19 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: guix-devel

The title is somewhat in-apt right now ;). I went through pain
today.

Even so, specifying the commit with 'guix pull' is incredibly useful
because it allows us to use our own substitutes and deploy the same
software across machines.

Now 'guix pull' itself is rather slow. Would it be possible to use
Guix pack or archive on an existing ~/.config/guix/current graph? I am
looking to quickly deploy guix itself as binary blobs too. 

Any idea of what could work? Maybe I should just do a 

  ~/.config/guix/current/bin/guix package -i guix -p ~/opt/guix

and copy that profile. It should contain the exact same package tree,
right?

Pj.

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: hydra.gnu.org now provides substitutes for ‘guix pull’
  2018-11-03 14:19     ` Pjotr Prins
@ 2018-11-06 13:30       ` Ludovic Courtès
  2018-11-06 15:55         ` Pjotr Prins
  0 siblings, 1 reply; 12+ messages in thread
From: Ludovic Courtès @ 2018-11-06 13:30 UTC (permalink / raw)
  To: Pjotr Prins; +Cc: guix-devel

Hello,

Pjotr Prins <pjotr.public12@thebird.nl> skribis:

> The title is somewhat in-apt right now ;). I went through pain
> today.

Like I wrote earlier today and before that, I think we should now focus
on berlin.guixsd.org and Cuirass (rather than hydra.gnu.org and Hydra.)

You can get substitutes for ‘guix pull’ from berlin, and indeed, I
highly encourage doing so because ‘guix pull’ without substitutes can
take way too much time (when all the substitutes are available, it’s
1.5–2mn on my laptop.)

> Now 'guix pull' itself is rather slow. Would it be possible to use
> Guix pack or archive on an existing ~/.config/guix/current graph?

Substitutes *are* such an archive.

HTH!

Ludo’.

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: hydra.gnu.org now provides substitutes for ‘guix pull’
  2018-11-06 13:30       ` Ludovic Courtès
@ 2018-11-06 15:55         ` Pjotr Prins
  2018-11-07  8:19           ` Pjotr Prins
  2018-11-07 22:03           ` Ludovic Courtès
  0 siblings, 2 replies; 12+ messages in thread
From: Pjotr Prins @ 2018-11-06 15:55 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: guix-devel

On Tue, Nov 06, 2018 at 02:30:08PM +0100, Ludovic Courtès wrote:
> > Now 'guix pull' itself is rather slow. Would it be possible to use
> > Guix pack or archive on an existing ~/.config/guix/current graph?
> 
> Substitutes *are* such an archive.

Hmmm. How do I create one? Installing on my Guix publish server (build
host) I never get a substitute for guix pull substituting from that.

For normal builds I get substitutes fine.

I raised an issue on debbugs to that effect.

Pj.

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: hydra.gnu.org now provides substitutes for ‘guix pull’
  2018-11-06 15:55         ` Pjotr Prins
@ 2018-11-07  8:19           ` Pjotr Prins
  2018-11-07 11:39             ` Unexpected building of dependencies Ricardo Wurmus
  2018-11-08  0:03             ` hydra.gnu.org now provides substitutes for ‘guix pull’ Mark H Weaver
  2018-11-07 22:03           ` Ludovic Courtès
  1 sibling, 2 replies; 12+ messages in thread
From: Pjotr Prins @ 2018-11-07  8:19 UTC (permalink / raw)
  To: Pjotr Prins; +Cc: guix-devel

On Tue, Nov 06, 2018 at 04:55:45PM +0100, Pjotr Prins wrote:
> On Tue, Nov 06, 2018 at 02:30:08PM +0100, Ludovic Courtès wrote:
> > > Now 'guix pull' itself is rather slow. Would it be possible to use
> > > Guix pack or archive on an existing ~/.config/guix/current graph?
> > 
> > Substitutes *are* such an archive.
> 
> Hmmm. How do I create one? Installing on my Guix publish server (build
> host) I never get a substitute for guix pull substituting from that.
> 
> For normal builds I get substitutes fine.
> 
> I raised an issue on debbugs to that effect.

Another thing I find a little strange. When I install a package or an
environment it completes fine. When I add then --no-substitutes flag
and it always builds stuff like openssl and libgcrypt, pango and such
from source.

I.e., it creates a build system (I suppose). If the install is
complete it should not have to do that. 

Even if I do 

  guix environment -C guix

(which should include a build system, right?) and after completion
run the same with --no-substitutes it starts to build right away.

Any thoughts on that?

Pj.

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Unexpected building of dependencies
  2018-11-07  8:19           ` Pjotr Prins
@ 2018-11-07 11:39             ` Ricardo Wurmus
  2018-11-08  0:03             ` hydra.gnu.org now provides substitutes for ‘guix pull’ Mark H Weaver
  1 sibling, 0 replies; 12+ messages in thread
From: Ricardo Wurmus @ 2018-11-07 11:39 UTC (permalink / raw)
  To: Pjotr Prins; +Cc: guix-devel


Pjotr Prins <pjotr.public12@thebird.nl> writes:

> Another thing I find a little strange. When I install a package or an
> environment it completes fine. When I add then --no-substitutes flag
> and it always builds stuff like openssl and libgcrypt, pango and such
> from source.

Could you provide a list of commands to reproduce this exactly?

-- 
Ricardo

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: hydra.gnu.org now provides substitutes for ‘guix pull’
  2018-11-06 15:55         ` Pjotr Prins
  2018-11-07  8:19           ` Pjotr Prins
@ 2018-11-07 22:03           ` Ludovic Courtès
  1 sibling, 0 replies; 12+ messages in thread
From: Ludovic Courtès @ 2018-11-07 22:03 UTC (permalink / raw)
  To: Pjotr Prins; +Cc: guix-devel

Pjotr Prins <pjotr.public12@thebird.nl> skribis:

> On Tue, Nov 06, 2018 at 02:30:08PM +0100, Ludovic Courtès wrote:
>> > Now 'guix pull' itself is rather slow. Would it be possible to use
>> > Guix pack or archive on an existing ~/.config/guix/current graph?
>> 
>> Substitutes *are* such an archive.
>
> Hmmm. How do I create one?

Suppose you have pulled commit X on a machine and you run ‘guix publish’
on that machine.

If you run ‘guix pull --commit=X’ on another machine that takes
substitutes from the first machine, you should definitely get
substitutes.

HTH!

Ludo’.

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: hydra.gnu.org now provides substitutes for ‘guix pull’
  2018-11-07  8:19           ` Pjotr Prins
  2018-11-07 11:39             ` Unexpected building of dependencies Ricardo Wurmus
@ 2018-11-08  0:03             ` Mark H Weaver
  1 sibling, 0 replies; 12+ messages in thread
From: Mark H Weaver @ 2018-11-08  0:03 UTC (permalink / raw)
  To: Pjotr Prins; +Cc: guix-devel

Hi Pjotr,

Pjotr Prins <pjotr.public12@thebird.nl> writes:

> Another thing I find a little strange. When I install a package or an
> environment it completes fine. When I add then --no-substitutes flag
> and it always builds stuff like openssl and libgcrypt, pango and such
> from source.
>
> I.e., it creates a build system (I suppose). If the install is
> complete it should not have to do that. 
>
> Even if I do 
>
>   guix environment -C guix
>
> (which should include a build system, right?) and after completion
> run the same with --no-substitutes it starts to build right away.
>
> Any thoughts on that?

My first guess is that it has to do with grafting.  If I'm not mistaken,
in order to generate a graft derivation (.drv file), we need to know
which store items are referenced in the ungrafted outputs.  When
substitutes are enabled, Guix can get the lists of references from the
substitute servers.  When --no-substitutes is passed, and the ungrafted
outputs are not in the local store, they must be locally built in order
to determine the set of references.

Or at least that's my guess.  Ludovic could answer more definitively, as
he wrote the relevant code.

       Mark

^ permalink raw reply	[flat|nested] 12+ messages in thread

end of thread, other threads:[~2018-11-08  0:04 UTC | newest]

Thread overview: 12+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2018-06-22  7:32 hydra.gnu.org now provides substitutes for ‘guix pull’ Ludovic Courtès
2018-06-26 11:16 ` swedebugia
2018-06-27 20:23   ` Ludovic Courtès
2018-10-15  7:41 ` Pjotr Prins
2018-10-15 10:16   ` Ludovic Courtès
2018-11-03 14:19     ` Pjotr Prins
2018-11-06 13:30       ` Ludovic Courtès
2018-11-06 15:55         ` Pjotr Prins
2018-11-07  8:19           ` Pjotr Prins
2018-11-07 11:39             ` Unexpected building of dependencies Ricardo Wurmus
2018-11-08  0:03             ` hydra.gnu.org now provides substitutes for ‘guix pull’ Mark H Weaver
2018-11-07 22:03           ` Ludovic Courtès

Code repositories for project(s) associated with this external index

	https://git.savannah.gnu.org/cgit/guix.git

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.