Leveraging a GuixSD Host's store in vm-images

Leveraging a GuixSD Host's store in vm-images

[George myglc2 Clemmer]

What is the best way to leverage a GuixSD host's store in vm-images?

I am building Guix from Git on a GuixSD host and running vm-images on
the GuixSD host and "foreign" distros in house.

Should I use 'guix publish' on the host? Helpful hints appreciated ;-)

TIA - George

Re: Leveraging a GuixSD Host's store in vm-images

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 1420 bytes --]

George myglc2 Clemmer <myglc2@gmail.com> writes:

> What is the best way to leverage a GuixSD host's store in vm-images?
>
> I am building Guix from Git on a GuixSD host and running vm-images on
> the GuixSD host and "foreign" distros in house.
>
> Should I use 'guix publish' on the host? Helpful hints appreciated ;-)

Hmmm.  I suspect that "guix publish," "guix archive," and "guix copy"
are probably the best options at the moment.  How are you making sure
that each Guix installation is using the same (or almost the same)
version of Guix?  That will be important for maximizing shareability.

Another long shot might be to try offloading, but that's intended more
to distribute the work of running the derivations, rather than
minimizing the amount of builds that need to happen.  If host A needs to
build derivation X, host B has not built X, host C has built X, and A
arbitrarily decides to offload to B, then B will waste time building X
when A could have just gotten it directly from C.  For a small number of
nodes, I suppose it's conceivable that it might help on average...but it
might just as well eat up all your CPU and network bandwidth, too.  :-)

If only substitutes published via "guix publish" were discoverable via
mDNS!  Then it would be easier to share within the same network, without
going through the tedium of configuring all the hosts individually...

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Leveraging a GuixSD Host's store in vm-images

[George myglc2 Clemmer]

Hi Chris,

Thank you for the comments ...

On 04/27/2018 at 03:56 Chris Marusich writes:

> George myglc2 Clemmer <myglc2@gmail.com> writes:
>
>> What is the best way to leverage a GuixSD host's store in vm-images?
>>
>> I am building Guix from Git on a GuixSD host and running vm-images on
>> the GuixSD host and "foreign" distros in house.
>>
>> Should I use 'guix publish' on the host? Helpful hints appreciated ;-)
>
> Hmmm.  I suspect that "guix publish," "guix archive," and "guix copy"
> are probably the best options at the moment.  How are you making sure
> that each Guix installation is using the same (or almost the same)
> version of Guix?  That will be important for maximizing shareability.

Agreed. The scheme outlined below attempts to do this.

> Another long shot might be to try offloading, but that's intended more
> to distribute the work of running the derivations, rather than
> minimizing the amount of builds that need to happen.  If host A needs to
> build derivation X, host B has not built X, host C has built X, and A
> arbitrarily decides to offload to B, then B will waste time building X
> when A could have just gotten it directly from C.  For a small number of
> nodes, I suppose it's conceivable that it might help on average...but it
> might just as well eat up all your CPU and network bandwidth, too.
> :-)

That does sound problematic.

> If only substitutes published via "guix publish" were discoverable via
> mDNS!  Then it would be easier to share within the same network, without
> going through the tedium of configuring all the hosts individually...

That would be cool indeed. In the meantime, how about ...

On the HOST:

-- set up a guix git repo with a "local master" that points to the most
   recent 'guix' package update.

-- Configure the host and VMs using the local master.

-- 'guix publish' the HOST store

-- resist the temptation to do 'git pull' or 'guix gc' ;-)

-- only when necessary "leap frog" the local master to a newer guix
   package update.

On the VMs:

-- only use 'guix pull --url=HOST’

-- 'guix copy --to=HOST' build results

-- set HOST as the primary substitute server

WDYT?

TIA - George

Re: Leveraging a GuixSD Host's store in vm-images

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 759 bytes --]

George myglc2 Clemmer <myglc2@gmail.com> writes:

> On the HOST:
>
> -- set up a guix git repo with a "local master" that points to the most
>    recent 'guix' package update.
>
> -- Configure the host and VMs using the local master.
>
> -- 'guix publish' the HOST store
>
> -- resist the temptation to do 'git pull' or 'guix gc' ;-)
>
> -- only when necessary "leap frog" the local master to a newer guix
>    package update.
>
> On the VMs:
>
> -- only use 'guix pull --url=HOST’
>
> -- 'guix copy --to=HOST' build results
>
> -- set HOST as the primary substitute server
>
> WDYT?

I think it's probably about the best you can do without setting up your
own build farm!  Which might be another good option.  ;-)

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]