bug#20381: Interacting with a remote daemon

bug#20381: Interacting with a remote daemon

[Ludovic Courtès]

In <http://elephly.net/posts/2015-04-17-gnu-guix.html>, Ricardo notes:

  At some point I think it would make sense to enhance Guix such that
  RPCs can be made over SSH, so that explicit logging on to a management
  machine is no longer necessary.

We should see exactly how much support is needed.

Is ‘socat’ enough, as suggested in
<https://lists.gnu.org/archive/html/guix-devel/2015-02/msg00680.html>?
That means there would be no authentication, but maybe that’s acceptable
in a cluster that is isolated on its own LAN anyway.

Option #2 would be to augment (guix store) and guix-daemon so that there
is built-in support for TCP sockets, which would be slightly more
convenient.

Option #3 would be to make (guix store) SSH-capable, with Guile-SSH.
But it sounds like SSH would incur relatively high latency when
connecting to the daemon.

There remains the problem that /var/guix/profiles/per-user/$USER would
have to be writable to this to be useful.

Thoughts?

Ludo’.

bug#20381: Interacting with a remote daemon

[Ricardo Wurmus]

I just tried the socat idea[1] with some success.

On the guix-builder host where guix-daemon is running and the NFS share
holding ‘/gnu’ (with $localstatedir set to ‘/gnu/var’) is mounted as
read-write I executed this:

    /root/.guix-profile/bin/socat TCP4-LISTEN:9999 UNIX:/gnu/var/guix/daemon-socket/socket

On a cluster node where /gnu is mounted read-only I ran this:

    socat UNIX-LISTEN:/home/rwurmus/foo TCP4:guix-builder:9999 &
    export GUIX_DAEMON_SOCKET=$HOME/foo

At this point I could use

    guix build hello
    guix environment hello

which is really great!

To make the “guix” command available on cluster nodes I just installed
it into my default user profile as ‘~/.guix-profile/bin/guix’.  The
problem with this is that profile commands don’t work as the regular
“guix” package as installed with $localstatedir set to ‘/var’.  This can
be fixed, of course, (e.g. by creating a slightly different “guix”
package with the appropriate configure flags set) but it’s still a minor
annoyance.  It would be great if $localstatedir could be overridden at
runtime or if it could default to whatever the daemon uses.

This would probably work fine if I limited the socket forwarding to just
the cluster nodes, because only there user ids are guaranteed to be
correct (not on workstations).  On workstations that are not centrally
managed this will not work, as the user ids could be arbitrary and it
would thus allow anyone to change anyone else’s profile by creating a
local account with the appropriate uid.

I prefer the socat approach over just running “guix” remotely through an
SSH connection, because with socat the “guix” command can actually be
used to spawn a new local shell with “guix environment”, which is very
useful.  I don’t think this would work if “guix” were just run
remotely.  (Please correct me if I’m wrong about this.)

~~ Ricardo

bug#20381: Interacting with a remote daemon

[Ludovic Courtès]

Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> skribis:

> I just tried the socat idea[1] with some success.
>
> On the guix-builder host where guix-daemon is running and the NFS share
> holding ‘/gnu’ (with $localstatedir set to ‘/gnu/var’) is mounted as
> read-write I executed this:
>
>     /root/.guix-profile/bin/socat TCP4-LISTEN:9999 UNIX:/gnu/var/guix/daemon-socket/socket
>
> On a cluster node where /gnu is mounted read-only I ran this:
>
>     socat UNIX-LISTEN:/home/rwurmus/foo TCP4:guix-builder:9999 &
>     export GUIX_DAEMON_SOCKET=$HOME/foo
>
> At this point I could use
>
>     guix build hello
>     guix environment hello
>
> which is really great!

Excellent, thanks for testing!

> To make the “guix” command available on cluster nodes I just installed
> it into my default user profile as ‘~/.guix-profile/bin/guix’.  The
> problem with this is that profile commands don’t work as the regular
> “guix” package as installed with $localstatedir set to ‘/var’.  This can
> be fixed, of course, (e.g. by creating a slightly different “guix”
> package with the appropriate configure flags set) but it’s still a minor
> annoyance.

What about installing Guix in /gnu/bin (say) and sharing it over NFS?

I would avoid installing Guix in a profile, because if things go wrong,
you may find yourself unable to do anything.  In practice, you can
always roll-back by hand (it’s simply a matter of switching the
profiles/per-user/$USER symlink), but still.

> It would be great if $localstatedir could be overridden at runtime or
> if it could default to whatever the daemon uses.

Actually it can be overridden via the intentionally-undocumented
NIX_STATE_DIR environment variable (see (guix config).)

> This would probably work fine if I limited the socket forwarding to just
> the cluster nodes, because only there user ids are guaranteed to be
> correct (not on workstations).  On workstations that are not centrally
> managed this will not work, as the user ids could be arbitrary and it
> would thus allow anyone to change anyone else’s profile by creating a
> local account with the appropriate uid.

The only problem would be with ‘guix package’, which you haven’t
mentioned yet.  :-)  For ‘guix package’ to work,
/gnu/var/guix/profiles/per-user must be shared read-write (over NFS)
with correct UID mapping.

> I prefer the socat approach over just running “guix” remotely through an
> SSH connection, because with socat the “guix” command can actually be
> used to spawn a new local shell with “guix environment”, which is very
> useful.  I don’t think this would work if “guix” were just run
> remotely.  (Please correct me if I’m wrong about this.)

Indeed, that would only allow you to spawn a shell on the machine where
the ‘guix’ command is executed (which, in your case, is not a compute
node AIUI.)

I think we should have a “Cluster Setup” section in the manual to
explain all this.  Would you like to give it a try?

Ludo’.

bug#20381: Interacting with a remote daemon

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

> What about installing Guix in /gnu/bin (say) and sharing it over NFS?
>
> I would avoid installing Guix in a profile, because if things go wrong,
> you may find yourself unable to do anything.  In practice, you can
> always roll-back by hand (it’s simply a matter of switching the
> profiles/per-user/$USER symlink), but still.
>
>> It would be great if $localstatedir could be overridden at runtime or
>> if it could default to whatever the daemon uses.
>
> Actually it can be overridden via the intentionally-undocumented
> NIX_STATE_DIR environment variable (see (guix config).)

Oh, nice.  Installing Guix somewhere into /gnu is actually a pretty good
idea.  I’ll try that and play with NIX_STATE_DIR as well.

>> This would probably work fine if I limited the socket forwarding to just
>> the cluster nodes, because only there user ids are guaranteed to be
>> correct (not on workstations).  On workstations that are not centrally
>> managed this will not work, as the user ids could be arbitrary and it
>> would thus allow anyone to change anyone else’s profile by creating a
>> local account with the appropriate uid.
>
> The only problem would be with ‘guix package’, which you haven’t
> mentioned yet.  :-)  For ‘guix package’ to work,
> /gnu/var/guix/profiles/per-user must be shared read-write (over NFS)
> with correct UID mapping.

Correct.  I haven’t tried ‘guix package’ at all because I just assumed
it would work.

> I think we should have a “Cluster Setup” section in the manual to
> explain all this.  Would you like to give it a try?

Sounds like a good idea.  I can give it a try but I’ll be on vacation
for a while and can only get around to writing in a couple of weeks.
But I think I’m a good candidate for drafting this section, given that
I’ve got a cluster to play with :)

Thanks for your helpful recommendations!

~~ Ricardo

bug#20381: Interacting with a remote daemon

[Ludovic Courtès]

Hello!

ludo@gnu.org (Ludovic Courtès) skribis:

> In <http://elephly.net/posts/2015-04-17-gnu-guix.html>, Ricardo notes:
>
>   At some point I think it would make sense to enhance Guix such that
>   RPCs can be made over SSH, so that explicit logging on to a management
>   machine is no longer necessary.
>
> We should see exactly how much support is needed.
>
> Is ‘socat’ enough, as suggested in
> <https://lists.gnu.org/archive/html/guix-devel/2015-02/msg00680.html>?
> That means there would be no authentication, but maybe that’s acceptable
> in a cluster that is isolated on its own LAN anyway.
>
> Option #2 would be to augment (guix store) and guix-daemon so that there
> is built-in support for TCP sockets, which would be slightly more
> convenient.
>
> Option #3 would be to make (guix store) SSH-capable, with Guile-SSH.
> But it sounds like SSH would incur relatively high latency when
> connecting to the daemon.

I think this is fixed by this series of commits:

  285f63e80 * store: Support 'ssh://' URIs in 'GUIX_DAEMON_SOCKET'.
  e53783372 * ssh: Decompose 'connect-to-remote-daemon'.
  615c5298f * ssh: Move 'open-ssh-session' to (guix ssh).
  ba97e454b * offload: Avoid using '_' as a 'match' pattern.
  3dff90ce3 * store: Add support for remote connections via 'guix://' URIs.
  3b5cd17a9 * store: Add 'system-error-to-connection-error' macro.
  1397b422e * store: 'GUIX_DAEMON_SOCKET' can now be a URI.

with the caveats given here:

  https://lists.gnu.org/archive/html/guix-devel/2017-04/msg00487.html

Closing!

Ludo’.