Re: Non-privileged daemons and offloading

[Roel Janssen <roel@gnu.org>]

Hello Ben,

It seems like we are facing a similar problem.  A proper solution takes
a lot more work and a lot more time I believe.  I am also currently
working on a more complete guide to do this, but here I tried to get the
essentials written down.

As far as software deployment goes, I have done the following to get it
on the restricted environment (in my case a cluster, on your case, a
super computer):

1. Get /gnu/store, or bootstrap your own store with a custom prefix
(I've done the latter) on a VM or a machine that has super user
privileges (let's call this the "build host").

For a custom prefix, you need to build guix from source with:

  ./configure --with-store-dir=/hpc/custom/guix-store \
              --localstatedir=/hpc/custom/guix-state/guix

You should change the environment variables: NIX_STATE_DIR and
NIX_STORE_DIR, before running the daemon, and before running the guix
command as a user.  In my case, I used:

  export NIX_STATE_DIR=/hpc/custom/guix-state/guix
  export NIX_STORE_DIR=/hpc/custom/guix-store
  guix-daemon --cores=4 --max-jobs=4 --no-substitutes --build-users-group=guixbuild


2. Build the packages you want to deploy on the HPC on the build host.

  export NIX_STATE_DIR=/hpc/custom/guix-state/guix
  export NIX_STORE_DIR=/hpc/custom/guix-store
  guix package -i <anything-you-need>


3. Copy the store and profiles.  This is a bit more tricky.  In my case,
hardlinks would not work because of the properties of our storage
system.  I used the following to copy the store and the profile (and
update it later on):

  rsync -lrt --delete --exclude=.links /hpc/custom/guix-store user@restricted-machine:/hpc/custom
  rsync -lrt --delete --exclude=.links /hpc/custom/guix-state user@restricted-machine:/hpc/custom

I excluded the .links directory to save space (you could copy them as
normal files instead of hardlinks, and the size of your store will
double).  Without this directory, you cannot efficiently do package
management, so don't remove it on the build host.

I didn't use the offloading mechanism on Guix.  I avoid using the
guix-daemon entirely, and reduce the deployment problem to an rsyncable
thing.

From here on, you can run programs as usual by adding
/hpc/custom/guix-state/guix/profiles/per-user/<username>/guix-profile/bin
to your path (and the other relevant environment variables).

If you install guix in your store, you can run guix-daemon on the
restricted machine and get 'guix package --search-paths', 'guix graph
...' and even 'guix gc' to work.  I haven't tested the other commands
yet.

I hope this helps.

Groet,
Roel


Ben Woodcroft writes:

> Hi there,
>
> I've recently gotten access to a supercomputer where I don't have sudo, 
> and I'm trying to hatch a plan to run guix packaged programs. 
> Unfortunately, I don't have anything substantial like a container-based 
> build environment to add here, only a potential workaround idea.
>
> I could run the daemon normally as a regular user, but I'm afraid of the 
> issues that arise due to inability to chroot as described in the manual. 
> Instead, I'm considering offloading the builds to a separate machine, 
> where a guix-daemon runs as sudo and so can run builds in the chroot. 
> IIUC, this gets around the issue of impurities in builds while not 
> requiring sudo.
>
> One hole the plan I can see is that at least according to the manual, 
> offloads are defined in "/etc/guix/machines.scm" which as a regular user 
> I cannot modify. Is there any other way to specify offload machines e.g. 
> via an argument to 'guix-daemon'? Is there any better ways to go about 
> this whole thing in general?
>
> Thanks,
> ben