Questions regarding offloading ( unprivileged setup , parallel builds )

Questions regarding offloading ( unprivileged setup , parallel builds )

[YOANN P]

[-- Attachment #1: Type: text/plain, Size: 1468 bytes --]

Hi Guix team,


I didn't find any documentation describing a way to use an alternative file for the offload settings instead of the default /etc/guix/machines.scm.


Did i missed something ? Is there other files that are put under /etc ?


It could be problematic for users who use custom store/state path in unprivileged environnements but want to use the offload isn't it ?


Another question regarding the offload, how are used the machines describe inside  /etc/guix/machines.scm ?

If the installation of a package require to build multiple packages, does the builds are all done on the first machine choose for the build or distribute over all machines describe on /etc/guix/machines.scm ?

The description of "parallel-builds" parameter and the fact than missing dependencies already present on the host who request the offload seem to be transferred through ssh let me think that only one machine at time is used for the offload even if all the machines present inside machines.scm share the same store.

Not sure about this too, there is no ssh transfert if the dependencies (same packages / store path ) are already on the offload store isn't it ?

Implementing something similar to the project "guix-relocate" from Pjotr inside the core project to be able to use native Guix packages with custom path when we install them would be a real huge improvement to prevent the use of proot/namespace/offloading :)

Thanks,

Best regards


[-- Attachment #2: Type: text/html, Size: 2594 bytes --]

Re: Questions regarding offloading ( unprivileged setup , parallel builds )

[Ludovic Courtès]

Hello,

YOANN P <yoann_mac_donald@hotmail.com> skribis:

> I didn't find any documentation describing a way to use an alternative file for the offload settings instead of the default /etc/guix/machines.scm.
>
>
> Did i missed something ?

No, this is correct.

> Is there other files that are put under /etc ?

/etc/guix also contains things related to substitutes.

> It could be problematic for users who use custom store/state path in unprivileged environnements but want to use the offload isn't it ?

Indeed, though if you’re customizing things anyway, you could run:

  ./configure --prefix=/my/prefix --sysconfdir=/my/etc

> Another question regarding the offload, how are used the machines describe inside  /etc/guix/machines.scm ?
>
> If the installation of a package require to build multiple packages, does the builds are all done on the first machine choose for the build or distribute over all machines describe on /etc/guix/machines.scm ?

When there are several matching machines, ‘guix offload’ currently picks
one at random; if that machine is already under high load, it goes on
and picks another one.  See:

  https://git.savannah.gnu.org/cgit/guix.git/tree/guix/scripts/offload.scm#n442

> The description of "parallel-builds" parameter and the fact than missing dependencies already present on the host who request the offload seem to be transferred through ssh let me think that only one machine at time is used for the offload even if all the machines present inside machines.scm share the same store.

Several machines can be used at the same time, no worries.  :-)

> Not sure about this too, there is no ssh transfert if the dependencies (same packages / store path ) are already on the offload store isn't it ?

Right, only missing items are transferred.

> Implementing something similar to the project "guix-relocate" from Pjotr inside the core project to be able to use native Guix packages with custom path when we install them would be a real huge improvement to prevent the use of proot/namespace/offloading :)

I think offloading serves a completely different purpose: it’s a way to
distribute builds to several machines.

But yeah, we discussed relocation at FOSDEM again and it’s a useful
thing in its own right, even though there are limitations.  :-)

Ludo’.

RE: Questions regarding offloading ( unprivileged setup , parallel builds )

[YOANN P]

[-- Attachment #1: Type: text/plain, Size: 1432 bytes --]

Hi,

Thanks you for yours answers :-)

>
> Indeed, though if you’re customizing things anyway, you could run:
>
>   ./configure --prefix=/my/prefix --sysconfdir=/my/etc
>

Thanks, I missed that part sorry

>
> Several machines can be used at the same time, no worries.  :-)
>

It is really nice if each dependencies could be build among different machines at the same time :-)


Anothers questions regarding the way the offload work:

- does the machines.scm is read at the start of the daemon or read each time the hook is called ? (just to be sure because the sources let me think it is read each time and is what i want)

- if a machine disappear during a build, does that build is retry on another machine ? is there a retry parameter ?

- Is there any project to had a parametrable post/pre offload-hook ? (Could be used to start preemptive cloud instances before a build and fill the "machines.scm" and shutdown instances at the end)

- No problem to use multiple daemon with the same store ? (I'm not sure of it because I never seen this kind of implementation on web, could be very useful if it not yet possible)

We could imagine a S3 storage for the store / state directories and when we need compute resources to build packages, only start preemptible instances who mount the same store through S3QL.

This way we only pay the storage for state / store directories

Best regards,
Yoann

[-- Attachment #2: Type: text/html, Size: 2621 bytes --]

Re: Questions regarding offloading ( unprivileged setup , parallel builds )

[Pjotr Prins]

On Mon, Feb 05, 2018 at 04:13:31PM +0000, YOANN P wrote:
>    We could imagine a S3 storage for the store / state directories and
>    when we need compute resources to build packages, only start
>    preemptible instances who mount the same store through S3QL.
>    This way we only pay the storage for state / store directories
>    Best regards,
>    Yoann

Are you on HPC? - we have a HPC working group, please join.

One thing we discussed is content addressable binaries - that way we can
fetch them from anywhere, even without signing!

Pj.
-- 

Re: Questions regarding offloading ( unprivileged setup , parallel builds )

[Ludovic Courtès]

Hi,

YOANN P <yoann_mac_donald@hotmail.com> skribis:

> Anothers questions regarding the way the offload work:
>
> - does the machines.scm is read at the start of the daemon or read
> each time the hook is called ? (just to be sure because the sources
> let me think it is read each time and is what i want)

It’s reach each time the hook is called.  You can add a ‘display’ call
in there if you want to see.

> - if a machine disappear during a build, does that build is retry on
> another machine ? is there a retry parameter ?

No.  In that case, I think the daemon returns a transient error in this
case, and the build can be restarted eventually, but we don’t do that
automatically currently.

> - Is there any project to had a parametrable post/pre offload-hook ?
> (Could be used to start preemptive cloud instances before a build and
> fill the "machines.scm" and shutdown instances at the end)

Currently no, though I guess you could do some of that in machines.scm.
More specifically you could have a “machine server” that does all the
heavy lifting, and have machines.scm simply make an RPC to that server
along the lines of “gimme a bunch of machines plz.”

> - No problem to use multiple daemon with the same store ? (I'm not
> sure of it because I never seen this kind of implementation on web,
> could be very useful if it not yet possible)

It kinda works but it’s not recommended.

HTH,
Ludo’.