From mboxrd@z Thu Jan 1 00:00:00 1970 From: Ben Woodcroft Subject: Non-privileged daemons and offloading Date: Sun, 19 Jun 2016 23:07:38 +1000 Message-ID: <5766991A.1020505@uq.edu.au> Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="------------010008070403070106070700" Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:48229) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bEcSF-0006VG-JI for help-guix@gnu.org; Sun, 19 Jun 2016 09:08:00 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bEcSA-0006Ry-GA for help-guix@gnu.org; Sun, 19 Jun 2016 09:07:58 -0400 Received: from mailhub2.soe.uq.edu.au ([130.102.132.209]:55347 helo=newmailhub.uq.edu.au) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bEcS9-0006Pp-Sk for help-guix@gnu.org; Sun, 19 Jun 2016 09:07:54 -0400 Received: from smtp1.soe.uq.edu.au (smtp1.soe.uq.edu.au [10.138.113.40]) by newmailhub.uq.edu.au (8.14.5/8.14.5) with ESMTP id u5JD7gPS028502 for ; Sun, 19 Jun 2016 23:07:43 +1000 Received: from [192.168.1.105] (static.customers.nuskope.com.au [103.25.181.216] (may be forged)) (authenticated bits=0) by smtp1.soe.uq.edu.au (8.14.5/8.14.5) with ESMTP id u5JD7ejw001838 (version=TLSv1.2 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT) for ; Sun, 19 Jun 2016 23:07:42 +1000 List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: help-guix-bounces+gcggh-help-guix=m.gmane.org@gnu.org Sender: "Help-Guix" To: help-guix This is a multi-part message in MIME format. --------------010008070403070106070700 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Hi there, I've recently gotten access to a supercomputer where I don't have sudo, and I'm trying to hatch a plan to run guix packaged programs. Unfortunately, I don't have anything substantial like a container-based build environment to add here, only a potential workaround idea. I could run the daemon normally as a regular user, but I'm afraid of the issues that arise due to inability to chroot as described in the manual. Instead, I'm considering offloading the builds to a separate machine, where a guix-daemon runs as sudo and so can run builds in the chroot. IIUC, this gets around the issue of impurities in builds while not requiring sudo. One hole the plan I can see is that at least according to the manual, offloads are defined in "/etc/guix/machines.scm" which as a regular user I cannot modify. Is there any other way to specify offload machines e.g. via an argument to 'guix-daemon'? Is there any better ways to go about this whole thing in general? Thanks, ben --------------010008070403070106070700 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 7bit Hi there,

I've recently gotten access to a supercomputer where I don't have sudo, and I'm trying to hatch a plan to run guix packaged programs. Unfortunately, I don't have anything substantial like a container-based build environment to add here, only a potential workaround idea.

I could run the daemon normally as a regular user, but I'm afraid of the issues that arise due to inability to chroot as described in the manual. Instead, I'm considering offloading the builds to a separate machine, where a guix-daemon runs as sudo and so can run builds in the chroot. IIUC, this gets around the issue of impurities in builds while not requiring sudo.

One hole the plan I can see is that at least according to the manual, offloads are defined in "/etc/guix/machines.scm" which as a regular user I cannot modify. Is there any other way to specify offload machines e.g. via an argument to 'guix-daemon'? Is there any better ways to go about this whole thing in general?

Thanks,
ben
--------------010008070403070106070700-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) Subject: Re: Non-privileged daemons and offloading Date: Mon, 20 Jun 2016 10:05:49 +0200 Message-ID: <87y460709u.fsf@gnu.org> References: <5766991A.1020505@uq.edu.au> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:38990) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bEuDY-0007kJ-38 for help-guix@gnu.org; Mon, 20 Jun 2016 04:06:01 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bEuDS-0007bd-Rd for help-guix@gnu.org; Mon, 20 Jun 2016 04:05:58 -0400 In-Reply-To: <5766991A.1020505@uq.edu.au> (Ben Woodcroft's message of "Sun, 19 Jun 2016 23:07:38 +1000") List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: help-guix-bounces+gcggh-help-guix=m.gmane.org@gnu.org Sender: "Help-Guix" To: Ben Woodcroft Cc: help-guix Hello! What you describe here is a hot topic and definitely a commonly requested feature. The difficulty here is that we=E2=80=99re hitting limitations of the kernel, which requires root privileges to set up a chroot and so on. The way around it is Linux=E2=80=99 unprivileged =E2=80=9Cuser namespaces= =E2=80=9D, as used by =E2=80=98guix environment --container=E2=80=99: they allow users to set up = isolated environments similar to what guix-daemon does, but without being root. Unfortunately, this feature is disabled on some distros out of security concerns (user namespaces are young and have a relatively bad track record.) You can check whether a system supports it like this: if [ -f /proc/self/ns/user ] then if [ -f /proc/sys/kernel/unprivileged_userns_clone ] then if [ `cat /proc/sys/kernel/unprivileged_userns_clone` -ne 0 ] then echo "unprivileged user namespaces supported" fi else echo "unprivileged user namespaces supported" fi fi Regardless, it remains our best hope to support unprivileged daemons. It would be nice to get stats on typical HPC systems. Roel has been looking into these issues recently, so perhaps he has some ideas. The Nix daemon recently switch to user namespaces: https://github.com/NixOS/nix/commit/c68e5913c71badc89ff346d1c6948517ba720= c93 We could backport this. However, running builds with UID 0 is potentially disruptive: some packages are sensitive to this and behave differently under UID 0 (I remember Coreutils=E2=80=99 test suite does.) A= lso, this patch switches to user namespaces, but not specifically _unprivileged_ user namespaces. Using offloading as you suggest doesn=E2=80=99t help: you would still need a daemon with access to /gnu/store. (Thinking out loud.) There=E2=80=99s a fun hack mind that could kinda work provided you use only substitutes, where you wouldn=E2=80=99t even need a daemon: 1. Compute the derivation of the package you want; normally that requires a daemon to which we make =E2=80=98add-to-store=E2=80=99 RPCs= , but we should be able to fake them altogether; 2. Use (guix scripts substitute) to download a substitute for that package, and unpack it under ~/.local, say; 3. Use =E2=80=98call-with-container=E2=80=99 (thus, unprivileged user nam= espace) to put yourself in an environment where /gnu/store/foo inside is a bind-mount to ~/.local/gnu/store/foo outside. There would remain the problem of profiles and grafts, which are normal derivations. When you think about it, it amounts to reimplementing (part of) the daemon functionality as a library, which is probably the way to go. That is, we could implement =E2=80=98add-to-store=E2=80=99 and =E2=80=98bui= ld-derivations=E2=80=99 such that they would operate locally under ~/.local. As a first milestone, =E2=80=98build-derivations=E2=80=99 could fail unless there=E2=80=99s a sub= stitute available. Food for thought! It would probably help if one of us could work on it full time at some point=E2=80=A6 Thanks, Ludo=E2=80=99. From mboxrd@z Thu Jan 1 00:00:00 1970 From: Roel Janssen Subject: Re: Non-privileged daemons and offloading Date: Mon, 20 Jun 2016 11:06:20 +0200 Message-ID: <87mvmgfcvn.fsf@gnu.org> References: <5766991A.1020505@uq.edu.au> Mime-Version: 1.0 Content-Type: text/plain Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:53196) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bEv9t-0006XR-5A for help-guix@gnu.org; Mon, 20 Jun 2016 05:06:18 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bEv9o-0002kS-UU for help-guix@gnu.org; Mon, 20 Jun 2016 05:06:16 -0400 In-reply-to: <5766991A.1020505@uq.edu.au> List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: help-guix-bounces+gcggh-help-guix=m.gmane.org@gnu.org Sender: "Help-Guix" To: Ben Woodcroft Cc: help-guix Hello Ben, It seems like we are facing a similar problem. A proper solution takes a lot more work and a lot more time I believe. I am also currently working on a more complete guide to do this, but here I tried to get the essentials written down. As far as software deployment goes, I have done the following to get it on the restricted environment (in my case a cluster, on your case, a super computer): 1. Get /gnu/store, or bootstrap your own store with a custom prefix (I've done the latter) on a VM or a machine that has super user privileges (let's call this the "build host"). For a custom prefix, you need to build guix from source with: ./configure --with-store-dir=/hpc/custom/guix-store \ --localstatedir=/hpc/custom/guix-state/guix You should change the environment variables: NIX_STATE_DIR and NIX_STORE_DIR, before running the daemon, and before running the guix command as a user. In my case, I used: export NIX_STATE_DIR=/hpc/custom/guix-state/guix export NIX_STORE_DIR=/hpc/custom/guix-store guix-daemon --cores=4 --max-jobs=4 --no-substitutes --build-users-group=guixbuild 2. Build the packages you want to deploy on the HPC on the build host. export NIX_STATE_DIR=/hpc/custom/guix-state/guix export NIX_STORE_DIR=/hpc/custom/guix-store guix package -i 3. Copy the store and profiles. This is a bit more tricky. In my case, hardlinks would not work because of the properties of our storage system. I used the following to copy the store and the profile (and update it later on): rsync -lrt --delete --exclude=.links /hpc/custom/guix-store user@restricted-machine:/hpc/custom rsync -lrt --delete --exclude=.links /hpc/custom/guix-state user@restricted-machine:/hpc/custom I excluded the .links directory to save space (you could copy them as normal files instead of hardlinks, and the size of your store will double). Without this directory, you cannot efficiently do package management, so don't remove it on the build host. I didn't use the offloading mechanism on Guix. I avoid using the guix-daemon entirely, and reduce the deployment problem to an rsyncable thing. >From here on, you can run programs as usual by adding /hpc/custom/guix-state/guix/profiles/per-user//guix-profile/bin to your path (and the other relevant environment variables). If you install guix in your store, you can run guix-daemon on the restricted machine and get 'guix package --search-paths', 'guix graph ...' and even 'guix gc' to work. I haven't tested the other commands yet. I hope this helps. Groet, Roel Ben Woodcroft writes: > Hi there, > > I've recently gotten access to a supercomputer where I don't have sudo, > and I'm trying to hatch a plan to run guix packaged programs. > Unfortunately, I don't have anything substantial like a container-based > build environment to add here, only a potential workaround idea. > > I could run the daemon normally as a regular user, but I'm afraid of the > issues that arise due to inability to chroot as described in the manual. > Instead, I'm considering offloading the builds to a separate machine, > where a guix-daemon runs as sudo and so can run builds in the chroot. > IIUC, this gets around the issue of impurities in builds while not > requiring sudo. > > One hole the plan I can see is that at least according to the manual, > offloads are defined in "/etc/guix/machines.scm" which as a regular user > I cannot modify. Is there any other way to specify offload machines e.g. > via an argument to 'guix-daemon'? Is there any better ways to go about > this whole thing in general? > > Thanks, > ben From mboxrd@z Thu Jan 1 00:00:00 1970 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) Subject: Re: Non-privileged daemons and offloading Date: Mon, 20 Jun 2016 14:23:05 +0200 Message-ID: <87wplk3v86.fsf@gnu.org> References: <5766991A.1020505@uq.edu.au> <87y460709u.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:45888) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bEyEY-00079y-8o for help-guix@gnu.org; Mon, 20 Jun 2016 08:23:19 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bEyET-00015h-6q for help-guix@gnu.org; Mon, 20 Jun 2016 08:23:17 -0400 In-Reply-To: <87y460709u.fsf@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\= \=\?utf-8\?Q\?s\?\= message of "Mon, 20 Jun 2016 10:05:49 +0200") List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: help-guix-bounces+gcggh-help-guix=m.gmane.org@gnu.org Sender: "Help-Guix" To: Ben Woodcroft Cc: help-guix ludo@gnu.org (Ludovic Court=C3=A8s) skribis: > Regardless, it remains our best hope to support unprivileged daemons. Also, I did not explicitly mention it, but I think this unprivileged user namespace thing should just be one part of the strategy. In parallel, it=E2=80=99s worth discussing with cluster sysadmins and see if they can have guix-daemon running on the cluster. There are good reasons for them to do that compared to letting each user do their own thing, and one of them is improved resource usage. Ricardo outlined the setup he came up with on a cluster here: http://elephly.net/posts/2015-04-17-gnu-guix.html and we have a bunch of arguments in store ;-): https://hal.inria.fr/hal-01161771/en In the end, for a sysadmin, it=E2=80=99s a cost/benefit tradeoff. In some situations, providing Guix may mean much less work for cluster admins. Ludo=E2=80=99. From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Thompson, David" Subject: Re: Non-privileged daemons and offloading Date: Mon, 20 Jun 2016 08:44:53 -0400 Message-ID: References: <5766991A.1020505@uq.edu.au> <87y460709u.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:50017) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bEyZc-00084R-L5 for help-guix@gnu.org; Mon, 20 Jun 2016 08:45:05 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bEyZS-00058b-B0 for help-guix@gnu.org; Mon, 20 Jun 2016 08:45:03 -0400 Received: from mail-vk0-x22b.google.com ([2607:f8b0:400c:c05::22b]:36722) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bEyZS-00057j-6e for help-guix@gnu.org; Mon, 20 Jun 2016 08:44:54 -0400 Received: by mail-vk0-x22b.google.com with SMTP id u64so193645904vkf.3 for ; Mon, 20 Jun 2016 05:44:53 -0700 (PDT) In-Reply-To: <87y460709u.fsf@gnu.org> List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: help-guix-bounces+gcggh-help-guix=m.gmane.org@gnu.org Sender: "Help-Guix" To: =?UTF-8?Q?Ludovic_Court=C3=A8s?= Cc: help-guix On Mon, Jun 20, 2016 at 4:05 AM, Ludovic Court=C3=A8s wrote: > The Nix daemon recently switch to user namespaces: > > https://github.com/NixOS/nix/commit/c68e5913c71badc89ff346d1c6948517ba7= 20c93 > > We could backport this. However, running builds with UID 0 is > potentially disruptive: some packages are sensitive to this and behave > differently under UID 0 (I remember Coreutils=E2=80=99 test suite does.) = Also, > this patch switches to user namespaces, but not specifically > _unprivileged_ user namespaces. It should be possible to create a new user within the container (say, UID 1000) and map it to a "guixbuild" user on the host system, avoiding this problem. - Dave From mboxrd@z Thu Jan 1 00:00:00 1970 From: Ben Woodcroft Subject: Re: Non-privileged daemons and offloading Date: Sat, 30 Jul 2016 14:11:39 +1000 Message-ID: <579C28FB.3010306@uq.edu.au> References: <5766991A.1020505@uq.edu.au> <87mvmgfcvn.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:60263) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bTLd3-0000kr-4W for help-guix@gnu.org; Sat, 30 Jul 2016 00:12:03 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bTLd0-0006Sv-2M for help-guix@gnu.org; Sat, 30 Jul 2016 00:11:59 -0400 In-Reply-To: <87mvmgfcvn.fsf@gnu.org> List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: help-guix-bounces+gcggh-help-guix=m.gmane.org@gnu.org Sender: "Help-Guix" To: Roel Janssen Cc: Mark H Weaver , help-guix Hi, Thanks all for the informative responses. On 20/06/16 18:05, Ludovic Court=E8s wrote: > Hello! > > What you describe here is a hot topic and definitely a commonly > requested feature. The difficulty here is that we=92re hitting > limitations of the kernel, which requires root privileges to set up a > chroot and so on. > > The way around it is Linux=92 unprivileged =93user namespaces=94, as us= ed by > =91guix environment --container=92: they allow users to set up isolated > environments similar to what guix-daemon does, but without being root. > Unfortunately, this feature is disabled on some distros out of security > concerns (user namespaces are young and have a relatively bad track > record.) > > You can check whether a system supports it like this: > > if [ -f /proc/self/ns/user ] > then > if [ -f /proc/sys/kernel/unprivileged_userns_clone ] > then > if [ `cat /proc/sys/kernel/unprivileged_userns_clone` -ne 0 = ] > then > echo "unprivileged user namespaces supported" > fi > else > echo "unprivileged user namespaces supported" > fi > fi I'm afraid I didn't have much luck with this: $ uname -a Linux euramoo3.qld.nectar.org.au 2.6.32-642.3.1.el6.x86_64 #1 SMP Tue=20 Jul 12 18:30:56 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux They are running CentOS 6.6 because they use Rocks: http://www.rocksclusters.org/ > Regardless, it remains our best hope to support unprivileged daemons. > > It would be nice to get stats on typical HPC systems. Hardware stats can be seen here, if that is what you mean? https://rcc.uq.edu.au/euramoo One difficulty is that there is 3 head nodes, and the daemon can only be=20 run on one of them. For my personal situation this isn't a big=20 difficulty since we can just designate one these as the "guix" node and=20 always login to that one. > Roel has been looking into these issues recently, so perhaps he has som= e > ideas. The Nix daemon recently switch to user namespaces: > > https://github.com/NixOS/nix/commit/c68e5913c71badc89ff346d1c6948517= ba720c93 > > We could backport this. However, running builds with UID 0 is > potentially disruptive: some packages are sensitive to this and behave > differently under UID 0 (I remember Coreutils=92 test suite does.) Als= o, > this patch switches to user namespaces, but not specifically > _unprivileged_ user namespaces. > > Using offloading as you suggest doesn=92t help: you would still need a > daemon with access to /gnu/store. I'm working with a non-standard store location, so that I do have=20 access. This means compiling everything from scratch, but then I find=20 watching "guix build --max-jobs" very beautiful. Building also doesn't=20 take too long after the first few packages are built, particularly in=20 comparison to downloading and building them manually. However, given Roel's tried and tested method I don't think I'll pursue=20 this approach. > (Thinking out loud.) > > There=92s a fun hack mind that could kinda work provided you use only > substitutes, where you wouldn=92t even need a daemon: > > 1. Compute the derivation of the package you want; normally that > requires a daemon to which we make =91add-to-store=92 RPCs, but w= e > should be able to fake them altogether; > > 2. Use (guix scripts substitute) to download a substitute for that > package, and unpack it under ~/.local, say; > > 3. Use =91call-with-container=92 (thus, unprivileged user namespace)= to > put yourself in an environment where /gnu/store/foo inside is a > bind-mount to ~/.local/gnu/store/foo outside. > > There would remain the problem of profiles and grafts, which are normal > derivations. > > When you think about it, it amounts to reimplementing (part of) the > daemon functionality as a library, which is probably the way to go. > That is, we could implement =91add-to-store=92 and =91build-derivations= =92 such > that they would operate locally under ~/.local. As a first milestone, > =91build-derivations=92 could fail unless there=92s a substitute availa= ble. > > Food for thought! Indeed, clever. On 20/06/16 22:23, Ludovic Court=E8s wrote: > ludo@gnu.org (Ludovic Court=E8s) skribis: > >> Regardless, it remains our best hope to support unprivileged daemons. > Also, I did not explicitly mention it, but I think this unprivileged > user namespace thing should just be one part of the strategy. I agree supporting non-privileged, non-container setups would be good.=20 OTOH, I cannot think of a way to do this that doesn't require a separate=20 build machine or a way that supports "guix environment". > In parallel, it=92s worth discussing with cluster sysadmins and see if > they can have guix-daemon running on the cluster. There are good > reasons for them to do that compared to letting each user do their own > thing, and one of them is improved resource usage.Ricardo outlined the = setup he came up with on a cluster here: > > http://elephly.net/posts/2015-04-17-gnu-guix.html > > and we have a bunch of arguments in store ;-): > > https://hal.inria.fr/hal-01161771/en > > In the end, for a sysadmin, it=92s a cost/benefit tradeoff. In some > situations, providing Guix may mean much less work for cluster admins. I haven't had much luck with this so far, though there is some interest.=20 I'm hoping to convince the sysadmins that Guix is mature enough software=20 by running my own setup for a while. The difference between the number=20 of bioinformatics packages available in Guix versus the number of=20 available modules in the HPC reflects very well on us here - that would=20 be a primary advantage. On a separate cluster where I do have root,=20 we've also had good success in adapting profiles to sit behind=20 modulefiles so that users are unaware that behind the scenes some=20 packages have been transitioned to being built by Guix, especially as we=20 upgrade the OS. On 20/06/16 19:06, Roel Janssen wrote: > Hello Ben, > > It seems like we are facing a similar problem. A proper solution takes > a lot more work and a lot more time I believe. I am also currently > working on a more complete guide to do this, but here I tried to get th= e > essentials written down. > > As far as software deployment goes, I have done the following to get it > on the restricted environment (in my case a cluster, on your case, a > super computer): Actually I misspoke, it is a cluster. I followed your instructions and=20 it is working well, thanks! I would be happy to contribute to a guide on=20 doing this if that is of use. > 1. Get /gnu/store, or bootstrap your own store with a custom prefix > (I've done the latter) on a VM or a machine that has super user > privileges (let's call this the "build host"). > > For a custom prefix, you need to build guix from source with: > > ./configure --with-store-dir=3D/hpc/custom/guix-store \ > --localstatedir=3D/hpc/custom/guix-state/guix > > You should change the environment variables: NIX_STATE_DIR and > NIX_STORE_DIR, before running the daemon, and before running the guix > command as a user. In my case, I used: > > export NIX_STATE_DIR=3D/hpc/custom/guix-state/guix > export NIX_STORE_DIR=3D/hpc/custom/guix-store > guix-daemon --cores=3D4 --max-jobs=3D4 --no-substitutes --build-user= s-group=3Dguixbuild I didn't find exporting those environment variables necessary, except=20 when running guix and guix-daemon on the restricted machine. > 2. Build the packages you want to deploy on the HPC on the build host. > > export NIX_STATE_DIR=3D/hpc/custom/guix-state/guix > export NIX_STORE_DIR=3D/hpc/custom/guix-store > guix package -i > > > 3. Copy the store and profiles. This is a bit more tricky. In my case= , > hardlinks would not work because of the properties of our storage > system. I used the following to copy the store and the profile (and > update it later on): > > rsync -lrt --delete --exclude=3D.links /hpc/custom/guix-store user@r= estricted-machine:/hpc/custom > rsync -lrt --delete --exclude=3D.links /hpc/custom/guix-state user@r= estricted-machine:/hpc/custom I simplified this by putting all the things to rsync in a single folder=20 so that a single call was needed, and I found "-z" helped too. > I excluded the .links directory to save space (you could copy them as > normal files instead of hardlinks, and the size of your store will > double). Without this directory, you cannot efficiently do package > management, so don't remove it on the build host. > > I didn't use the offloading mechanism on Guix. I avoid using the > guix-daemon entirely, and reduce the deployment problem to an rsyncable > thing. > > From here on, you can run programs as usual by adding > /hpc/custom/guix-state/guix/profiles/per-user//guix-profile/b= in > to your path (and the other relevant environment variables). > > If you install guix in your store, you can run guix-daemon on the > restricted machine and get 'guix package --search-paths', 'guix graph > ...' and even 'guix gc' to work. I haven't tested the other commands > yet. I take it by 'guix gc' you mean 'guix gc --references' since you don't=20 want to modify the store, correct? I only tried 'guix package=20 --search-paths', which is probably the most important for users. Thanks, ben