Non-privileged daemons and offloading

Non-privileged daemons and offloading

[Ben Woodcroft]

[-- Attachment #1: Type: text/plain, Size: 994 bytes --]

Hi there,

I've recently gotten access to a supercomputer where I don't have sudo, 
and I'm trying to hatch a plan to run guix packaged programs. 
Unfortunately, I don't have anything substantial like a container-based 
build environment to add here, only a potential workaround idea.

I could run the daemon normally as a regular user, but I'm afraid of the 
issues that arise due to inability to chroot as described in the manual. 
Instead, I'm considering offloading the builds to a separate machine, 
where a guix-daemon runs as sudo and so can run builds in the chroot. 
IIUC, this gets around the issue of impurities in builds while not 
requiring sudo.

One hole the plan I can see is that at least according to the manual, 
offloads are defined in "/etc/guix/machines.scm" which as a regular user 
I cannot modify. Is there any other way to specify offload machines e.g. 
via an argument to 'guix-daemon'? Is there any better ways to go about 
this whole thing in general?

Thanks,
ben

[-- Attachment #2: Type: text/html, Size: 1285 bytes --]

Re: Non-privileged daemons and offloading

[Ludovic Courtès]

Hello!

What you describe here is a hot topic and definitely a commonly
requested feature.  The difficulty here is that we’re hitting
limitations of the kernel, which requires root privileges to set up a
chroot and so on.

The way around it is Linux’ unprivileged “user namespaces”, as used by
‘guix environment --container’: they allow users to set up isolated
environments similar to what guix-daemon does, but without being root.
Unfortunately, this feature is disabled on some distros out of security
concerns (user namespaces are young and have a relatively bad track
record.)

You can check whether a system supports it like this:

  if [ -f /proc/self/ns/user ]
  then
      if [ -f /proc/sys/kernel/unprivileged_userns_clone ]
      then
          if [ `cat /proc/sys/kernel/unprivileged_userns_clone` -ne 0 ]
          then
              echo "unprivileged user namespaces supported"
          fi
      else
          echo "unprivileged user namespaces supported"
      fi
  fi

Regardless, it remains our best hope to support unprivileged daemons.

It would be nice to get stats on typical HPC systems.

Roel has been looking into these issues recently, so perhaps he has some
ideas.  The Nix daemon recently switch to user namespaces:

  https://github.com/NixOS/nix/commit/c68e5913c71badc89ff346d1c6948517ba720c93

We could backport this.  However, running builds with UID 0 is
potentially disruptive: some packages are sensitive to this and behave
differently under UID 0 (I remember Coreutils’ test suite does.)  Also,
this patch switches to user namespaces, but not specifically
_unprivileged_ user namespaces.

Using offloading as you suggest doesn’t help: you would still need a
daemon with access to /gnu/store.

(Thinking out loud.)

There’s a fun hack mind that could kinda work provided you use only
substitutes, where you wouldn’t even need a daemon:

  1. Compute the derivation of the package you want; normally that
     requires a daemon to which we make ‘add-to-store’ RPCs, but we
     should be able to fake them altogether;

  2. Use (guix scripts substitute) to download a substitute for that
     package, and unpack it under ~/.local, say;

  3. Use ‘call-with-container’ (thus, unprivileged user namespace) to
     put yourself in an environment where /gnu/store/foo inside is a
     bind-mount to ~/.local/gnu/store/foo outside.

There would remain the problem of profiles and grafts, which are normal
derivations.

When you think about it, it amounts to reimplementing (part of) the
daemon functionality as a library, which is probably the way to go.
That is, we could implement ‘add-to-store’ and ‘build-derivations’ such
that they would operate locally under ~/.local.  As a first milestone,
‘build-derivations’ could fail unless there’s a substitute available.

Food for thought!

It would probably help if one of us could work on it full time at some
point…

Thanks,
Ludo’.

Re: Non-privileged daemons and offloading

[Roel Janssen]

Hello Ben,

It seems like we are facing a similar problem.  A proper solution takes
a lot more work and a lot more time I believe.  I am also currently
working on a more complete guide to do this, but here I tried to get the
essentials written down.

As far as software deployment goes, I have done the following to get it
on the restricted environment (in my case a cluster, on your case, a
super computer):

1. Get /gnu/store, or bootstrap your own store with a custom prefix
(I've done the latter) on a VM or a machine that has super user
privileges (let's call this the "build host").

For a custom prefix, you need to build guix from source with:

  ./configure --with-store-dir=/hpc/custom/guix-store \
              --localstatedir=/hpc/custom/guix-state/guix

You should change the environment variables: NIX_STATE_DIR and
NIX_STORE_DIR, before running the daemon, and before running the guix
command as a user.  In my case, I used:

  export NIX_STATE_DIR=/hpc/custom/guix-state/guix
  export NIX_STORE_DIR=/hpc/custom/guix-store
  guix-daemon --cores=4 --max-jobs=4 --no-substitutes --build-users-group=guixbuild


2. Build the packages you want to deploy on the HPC on the build host.

  export NIX_STATE_DIR=/hpc/custom/guix-state/guix
  export NIX_STORE_DIR=/hpc/custom/guix-store
  guix package -i <anything-you-need>


3. Copy the store and profiles.  This is a bit more tricky.  In my case,
hardlinks would not work because of the properties of our storage
system.  I used the following to copy the store and the profile (and
update it later on):

  rsync -lrt --delete --exclude=.links /hpc/custom/guix-store user@restricted-machine:/hpc/custom
  rsync -lrt --delete --exclude=.links /hpc/custom/guix-state user@restricted-machine:/hpc/custom

I excluded the .links directory to save space (you could copy them as
normal files instead of hardlinks, and the size of your store will
double).  Without this directory, you cannot efficiently do package
management, so don't remove it on the build host.

I didn't use the offloading mechanism on Guix.  I avoid using the
guix-daemon entirely, and reduce the deployment problem to an rsyncable
thing.

From here on, you can run programs as usual by adding
/hpc/custom/guix-state/guix/profiles/per-user/<username>/guix-profile/bin
to your path (and the other relevant environment variables).

If you install guix in your store, you can run guix-daemon on the
restricted machine and get 'guix package --search-paths', 'guix graph
...' and even 'guix gc' to work.  I haven't tested the other commands
yet.

I hope this helps.

Groet,
Roel


Ben Woodcroft writes:

> Hi there,
>
> I've recently gotten access to a supercomputer where I don't have sudo, 
> and I'm trying to hatch a plan to run guix packaged programs. 
> Unfortunately, I don't have anything substantial like a container-based 
> build environment to add here, only a potential workaround idea.
>
> I could run the daemon normally as a regular user, but I'm afraid of the 
> issues that arise due to inability to chroot as described in the manual. 
> Instead, I'm considering offloading the builds to a separate machine, 
> where a guix-daemon runs as sudo and so can run builds in the chroot. 
> IIUC, this gets around the issue of impurities in builds while not 
> requiring sudo.
>
> One hole the plan I can see is that at least according to the manual, 
> offloads are defined in "/etc/guix/machines.scm" which as a regular user 
> I cannot modify. Is there any other way to specify offload machines e.g. 
> via an argument to 'guix-daemon'? Is there any better ways to go about 
> this whole thing in general?
>
> Thanks,
> ben

Re: Non-privileged daemons and offloading

[Ludovic Courtès]

ludo@gnu.org (Ludovic Courtès) skribis:

> Regardless, it remains our best hope to support unprivileged daemons.

Also, I did not explicitly mention it, but I think this unprivileged
user namespace thing should just be one part of the strategy.

In parallel, it’s worth discussing with cluster sysadmins and see if
they can have guix-daemon running on the cluster.  There are good
reasons for them to do that compared to letting each user do their own
thing, and one of them is improved resource usage.

Ricardo outlined the setup he came up with on a cluster here:

  http://elephly.net/posts/2015-04-17-gnu-guix.html

and we have a bunch of arguments in store ;-):

  https://hal.inria.fr/hal-01161771/en

In the end, for a sysadmin, it’s a cost/benefit tradeoff.  In some
situations, providing Guix may mean much less work for cluster admins.

Ludo’.

Re: Non-privileged daemons and offloading

[Thompson, David]

On Mon, Jun 20, 2016 at 4:05 AM, Ludovic Courtès <ludo@gnu.org> wrote:
> The Nix daemon recently switch to user namespaces:
>
>   https://github.com/NixOS/nix/commit/c68e5913c71badc89ff346d1c6948517ba720c93
>
> We could backport this.  However, running builds with UID 0 is
> potentially disruptive: some packages are sensitive to this and behave
> differently under UID 0 (I remember Coreutils’ test suite does.)  Also,
> this patch switches to user namespaces, but not specifically
> _unprivileged_ user namespaces.

It should be possible to create a new user within the container (say,
UID 1000) and map it to a "guixbuild" user on the host system,
avoiding this problem.

- Dave

Re: Non-privileged daemons and offloading

[Ben Woodcroft]

Hi,

Thanks all for the informative responses.

On 20/06/16 18:05, Ludovic Courtès wrote:
> Hello!
>
> What you describe here is a hot topic and definitely a commonly
> requested feature.  The difficulty here is that we’re hitting
> limitations of the kernel, which requires root privileges to set up a
> chroot and so on.
>
> The way around it is Linux’ unprivileged “user namespaces”, as used by
> ‘guix environment --container’: they allow users to set up isolated
> environments similar to what guix-daemon does, but without being root.
> Unfortunately, this feature is disabled on some distros out of security
> concerns (user namespaces are young and have a relatively bad track
> record.)
>
> You can check whether a system supports it like this:
>
>    if [ -f /proc/self/ns/user ]
>    then
>        if [ -f /proc/sys/kernel/unprivileged_userns_clone ]
>        then
>            if [ `cat /proc/sys/kernel/unprivileged_userns_clone` -ne 0 ]
>            then
>                echo "unprivileged user namespaces supported"
>            fi
>        else
>            echo "unprivileged user namespaces supported"
>        fi
>    fi

I'm afraid I didn't have much luck with this:
$ uname -a
Linux euramoo3.qld.nectar.org.au 2.6.32-642.3.1.el6.x86_64 #1 SMP Tue 
Jul 12 18:30:56 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

They are running CentOS 6.6 because they use Rocks:
http://www.rocksclusters.org/

> Regardless, it remains our best hope to support unprivileged daemons.
>
> It would be nice to get stats on typical HPC systems.

Hardware stats can be seen here, if that is what you mean?
https://rcc.uq.edu.au/euramoo

One difficulty is that there is 3 head nodes, and the daemon can only be 
run on one of them. For my personal situation this isn't a big 
difficulty since we can just designate one these as the "guix" node and 
always login to that one.

> Roel has been looking into these issues recently, so perhaps he has some
> ideas.  The Nix daemon recently switch to user namespaces:
>
>    https://github.com/NixOS/nix/commit/c68e5913c71badc89ff346d1c6948517ba720c93
>
> We could backport this.  However, running builds with UID 0 is
> potentially disruptive: some packages are sensitive to this and behave
> differently under UID 0 (I remember Coreutils’ test suite does.)  Also,
> this patch switches to user namespaces, but not specifically
> _unprivileged_ user namespaces.
>
> Using offloading as you suggest doesn’t help: you would still need a
> daemon with access to /gnu/store.

I'm working with a non-standard store location, so that I do have 
access. This means compiling everything from scratch, but then I find 
watching "guix build --max-jobs" very beautiful. Building also doesn't 
take too long after the first few packages are built, particularly in 
comparison to downloading and building them manually.

However, given Roel's tried and tested method I don't think I'll pursue 
this approach.

> (Thinking out loud.)
>
> There’s a fun hack mind that could kinda work provided you use only
> substitutes, where you wouldn’t even need a daemon:
>
>    1. Compute the derivation of the package you want; normally that
>       requires a daemon to which we make ‘add-to-store’ RPCs, but we
>       should be able to fake them altogether;
>
>    2. Use (guix scripts substitute) to download a substitute for that
>       package, and unpack it under ~/.local, say;
>
>    3. Use ‘call-with-container’ (thus, unprivileged user namespace) to
>       put yourself in an environment where /gnu/store/foo inside is a
>       bind-mount to ~/.local/gnu/store/foo outside.
>
> There would remain the problem of profiles and grafts, which are normal
> derivations.
>
> When you think about it, it amounts to reimplementing (part of) the
> daemon functionality as a library, which is probably the way to go.
> That is, we could implement ‘add-to-store’ and ‘build-derivations’ such
> that they would operate locally under ~/.local.  As a first milestone,
> ‘build-derivations’ could fail unless there’s a substitute available.
>
> Food for thought!

Indeed, clever.


On 20/06/16 22:23, Ludovic Courtès wrote:
> ludo@gnu.org (Ludovic Courtès) skribis:
>
>> Regardless, it remains our best hope to support unprivileged daemons.
> Also, I did not explicitly mention it, but I think this unprivileged
> user namespace thing should just be one part of the strategy.

I agree supporting non-privileged, non-container setups would be good. 
OTOH, I cannot think of a way to do this that doesn't require a separate 
build machine or a way that supports "guix environment".

> In parallel, it’s worth discussing with cluster sysadmins and see if
> they can have guix-daemon running on the cluster.  There are good
> reasons for them to do that compared to letting each user do their own
> thing, and one of them is improved resource usage.Ricardo outlined the setup he came up with on a cluster here:
>
>    http://elephly.net/posts/2015-04-17-gnu-guix.html
>
> and we have a bunch of arguments in store ;-):
>
>    https://hal.inria.fr/hal-01161771/en
>
> In the end, for a sysadmin, it’s a cost/benefit tradeoff.  In some
> situations, providing Guix may mean much less work for cluster admins.

I haven't had much luck with this so far, though there is some interest. 
I'm hoping to convince the sysadmins that Guix is mature enough software 
by running my own setup for a while. The difference between the number 
of bioinformatics packages available in Guix versus the number of 
available modules in the HPC reflects very well on us here - that would 
be a primary advantage. On a separate cluster where I do have root, 
we've also had good success in adapting profiles to sit behind 
modulefiles so that users are unaware that behind the scenes some 
packages have been transitioned to being built by Guix, especially as we 
upgrade the OS.


On 20/06/16 19:06, Roel Janssen wrote:
> Hello Ben,
>
> It seems like we are facing a similar problem.  A proper solution takes
> a lot more work and a lot more time I believe.  I am also currently
> working on a more complete guide to do this, but here I tried to get the
> essentials written down.
>
> As far as software deployment goes, I have done the following to get it
> on the restricted environment (in my case a cluster, on your case, a
> super computer):

Actually I misspoke, it is a cluster. I followed your instructions and 
it is working well, thanks! I would be happy to contribute to a guide on 
doing this if that is of use.

> 1. Get /gnu/store, or bootstrap your own store with a custom prefix
> (I've done the latter) on a VM or a machine that has super user
> privileges (let's call this the "build host").
>
> For a custom prefix, you need to build guix from source with:
>
>    ./configure --with-store-dir=/hpc/custom/guix-store \
>                --localstatedir=/hpc/custom/guix-state/guix
>
> You should change the environment variables: NIX_STATE_DIR and
> NIX_STORE_DIR, before running the daemon, and before running the guix
> command as a user.  In my case, I used:
>
>    export NIX_STATE_DIR=/hpc/custom/guix-state/guix
>    export NIX_STORE_DIR=/hpc/custom/guix-store
>    guix-daemon --cores=4 --max-jobs=4 --no-substitutes --build-users-group=guixbuild

I didn't find exporting those environment variables necessary, except 
when running guix and guix-daemon on the restricted machine.

> 2. Build the packages you want to deploy on the HPC on the build host.
>
>    export NIX_STATE_DIR=/hpc/custom/guix-state/guix
>    export NIX_STORE_DIR=/hpc/custom/guix-store
>    guix package -i <anything-you-need>
>
>
> 3. Copy the store and profiles.  This is a bit more tricky.  In my case,
> hardlinks would not work because of the properties of our storage
> system.  I used the following to copy the store and the profile (and
> update it later on):
>
>    rsync -lrt --delete --exclude=.links /hpc/custom/guix-store user@restricted-machine:/hpc/custom
>    rsync -lrt --delete --exclude=.links /hpc/custom/guix-state user@restricted-machine:/hpc/custom

I simplified this by putting all the things to rsync in a single folder 
so that a single call was needed, and I found "-z" helped too.

> I excluded the .links directory to save space (you could copy them as
> normal files instead of hardlinks, and the size of your store will
> double).  Without this directory, you cannot efficiently do package
> management, so don't remove it on the build host.
>
> I didn't use the offloading mechanism on Guix.  I avoid using the
> guix-daemon entirely, and reduce the deployment problem to an rsyncable
> thing.
>
>  From here on, you can run programs as usual by adding
> /hpc/custom/guix-state/guix/profiles/per-user/<username>/guix-profile/bin
> to your path (and the other relevant environment variables).
>
> If you install guix in your store, you can run guix-daemon on the
> restricted machine and get 'guix package --search-paths', 'guix graph
> ...' and even 'guix gc' to work.  I haven't tested the other commands
> yet.

I take it by 'guix gc' you mean 'guix gc --references' since you don't 
want to modify the store, correct? I only tried 'guix package 
--search-paths', which is probably the most important for users.

Thanks,
ben