Questions and Notes regarding Offloading and Substitutes

Questions and Notes regarding Offloading and Substitutes

[Carlos Sosa]

Hi,

for the past week I've been playing around with guix offloading, archive
and substitutes. I can tell you that authentication of an offloading
node is fairly straight forward just like the manual says.

What is a little bit confusing is how `guix build` and offloading works.
To be honest, in the begining, offloading was not that straight forward
since the manual lacks to point out that `guix offloading` is using
`lsh`. So there are certain steps that need to be addressed like,
creating a 'lsh' key and then exporting it to an openssh public key.
After that you might have a problem with the cyphers available in
OpenSSH and lsh, which are disabled by default in OpenSSH. After setting
all of that, builds offloading works flawlessly, I was really happy I
haven't encountered any problem.

Now what I don't get is why offloading mostly works for builds and not
for guix package installation. I might be confusing what builds are for,
but shouldn't an already finished build be used by `guix package` when
installing a package. For example, if I do 'guix build emacs' that will
build emacs and its dependencies, and I can call later `guix package -i
emacs`, right? When I do `guix package -i emacs` with the daemon with
'--max-jobs=0' instead of offloading the compilation and built to any of
the machines in 'machines.scm', it seems to offload some bits but still
compile some components in the machine. I think the latter is fine, but
I consider it's best if they did everything. In my case, I own two small
netbooks that I would love to offload all of the process of building and
compiling to a bigger node capable of doing that, but still keeping a
store and profile in those machines.

I'm having a couple of problems with substitutes, I can't seem to be
able to send all of the needed substitutes, for let us say the emacs
package, to the small netbooks from the big node at home. They do grab
some derivations and other elements, but seem to give me a Nix failure
from time to time. Are substitutes what I'm ultimately looking for,
having a build node build everything and send it as a substitute to the
smaller nodes?

Please let me know what I'm doing wrong. :)

- Carlos

Re: Questions and Notes regarding Offloading and Substitutes

[Ludovic Courtès]

Hi!

Sorry for the delay, that’s what you get for writing long messages.  ;-)

Carlos Sosa <gnusosa@gnusosa.net> skribis:

> What is a little bit confusing is how `guix build` and offloading works.
> To be honest, in the begining, offloading was not that straight forward
> since the manual lacks to point out that `guix offloading` is using
> `lsh`. So there are certain steps that need to be addressed like,
> creating a 'lsh' key and then exporting it to an openssh public key.
> After that you might have a problem with the cyphers available in
> OpenSSH and lsh, which are disabled by default in OpenSSH. After setting
> all of that, builds offloading works flawlessly, I was really happy I
> haven't encountered any problem.

I must say that the fact that lsh was not mentioned was originally
intentional, on the grounds that the implementation would “soon” switch
to Guile-SSH instead (which is compatible with OpenSSH, not lsh.)  But
that hasn’t happened yet, although there’s a branch, wip-guile-ssh.

> Now what I don't get is why offloading mostly works for builds and not
> for guix package installation. I might be confusing what builds are for,
> but shouldn't an already finished build be used by `guix package` when
> installing a package. For example, if I do 'guix build emacs' that will
> build emacs and its dependencies, and I can call later `guix package -i
> emacs`, right? When I do `guix package -i emacs` with the daemon with
> '--max-jobs=0' instead of offloading the compilation and built to any of
> the machines in 'machines.scm', it seems to offload some bits but still
> compile some components in the machine. I think the latter is fine, but
> I consider it's best if they did everything. In my case, I own two small
> netbooks that I would love to offload all of the process of building and
> compiling to a bigger node capable of doing that, but still keeping a
> store and profile in those machines.

Normally everything gets offloaded provided there are available slots on
the target machines (as per the ‘parallel-builds’ field.)

Now, some derivations explicitly ask not to be offloaded (search for
#:local-build? in the code.)  This is typically derivations that do very
little work and produce small files, where the overhead of offloading
would outweigh the gain.

For example, the derivation that builds a profile (when running ‘guix
package -i’) is not subject to offloading.  This may be what you were
observing no?

> I'm having a couple of problems with substitutes, I can't seem to be
> able to send all of the needed substitutes, for let us say the emacs
> package, to the small netbooks from the big node at home. They do grab
> some derivations and other elements, but seem to give me a Nix failure
> from time to time. Are substitutes what I'm ultimately looking for,
> having a build node build everything and send it as a substitute to the
> smaller nodes?

What do you mean by “send all of the needed substitutes”?  Note that
offloading and substitutes are two different mechanisms, even though
they seem similar.

HTH!

Ludo’.

Re: Questions and Notes regarding Offloading and Substitutes

[Carlos Sosa]

ludo@gnu.org (Ludovic Courtès) writes:

>
> I must say that the fact that lsh was not mentioned was originally
> intentional, on the grounds that the implementation would “soon” switch
> to Guile-SSH instead (which is compatible with OpenSSH, not lsh.)  But
> that hasn’t happened yet, although there’s a branch, wip-guile-ssh.

  Oh I did see that branch in Savannah. Do you think I can try out
  `wip-guile-ssh`? Is it in a working condition? Just like with these
  topic I will give you my feedback :)

> Normally everything gets offloaded provided there are available slots on
> the target machines (as per the ‘parallel-builds’ field.)
>
> Now, some derivations explicitly ask not to be offloaded (search for
> #:local-build? in the code.)  This is typically derivations that do very
> little work and produce small files, where the overhead of offloading
> would outweigh the gain.

  That makes sense, I didn't know about #:local-build, I actually don't
  mind small components I tried it out yesterday, and seems fine.

> What do you mean by “send all of the needed substitutes”?  Note that
> offloading and substitutes are two different mechanisms, even though
> they seem similar.

  I figured out what I was doing wrong. Don't mind that. What I wanted
  seems to be solved with a proper setup of subtitutes. My confusion
  with builds and subtitutes was resolved in the IRC channel.

  Btw, no worries about the late reply I understand.

  -- Carlos

Re: Questions and Notes regarding Offloading and Substitutes

[Ludovic Courtès]

Carlos Sosa <gnusosa@gnusosa.net> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>
>>
>> I must say that the fact that lsh was not mentioned was originally
>> intentional, on the grounds that the implementation would “soon” switch
>> to Guile-SSH instead (which is compatible with OpenSSH, not lsh.)  But
>> that hasn’t happened yet, although there’s a branch, wip-guile-ssh.
>
>   Oh I did see that branch in Savannah. Do you think I can try out
>   `wip-guile-ssh`? Is it in a working condition? Just like with these
>   topic I will give you my feedback :)

First you would need to rebase it on top of master.  It probably won’t
work out of the box, but I would certainly appreciate feedback and
patches!  :-)

I believe it requires the latest Guile-SSH.  Previous versions lacked
important features IIRC.

Thanks,
Ludo’.