From mboxrd@z Thu Jan 1 00:00:00 1970 From: Ben Woodcroft Subject: Re: Non-privileged daemons and offloading Date: Sat, 30 Jul 2016 14:11:39 +1000 Message-ID: <579C28FB.3010306@uq.edu.au> References: <5766991A.1020505@uq.edu.au> <87mvmgfcvn.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:60263) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bTLd3-0000kr-4W for help-guix@gnu.org; Sat, 30 Jul 2016 00:12:03 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bTLd0-0006Sv-2M for help-guix@gnu.org; Sat, 30 Jul 2016 00:11:59 -0400 In-Reply-To: <87mvmgfcvn.fsf@gnu.org> List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: help-guix-bounces+gcggh-help-guix=m.gmane.org@gnu.org Sender: "Help-Guix" To: Roel Janssen Cc: Mark H Weaver , help-guix Hi, Thanks all for the informative responses. On 20/06/16 18:05, Ludovic Court=E8s wrote: > Hello! > > What you describe here is a hot topic and definitely a commonly > requested feature. The difficulty here is that we=92re hitting > limitations of the kernel, which requires root privileges to set up a > chroot and so on. > > The way around it is Linux=92 unprivileged =93user namespaces=94, as us= ed by > =91guix environment --container=92: they allow users to set up isolated > environments similar to what guix-daemon does, but without being root. > Unfortunately, this feature is disabled on some distros out of security > concerns (user namespaces are young and have a relatively bad track > record.) > > You can check whether a system supports it like this: > > if [ -f /proc/self/ns/user ] > then > if [ -f /proc/sys/kernel/unprivileged_userns_clone ] > then > if [ `cat /proc/sys/kernel/unprivileged_userns_clone` -ne 0 = ] > then > echo "unprivileged user namespaces supported" > fi > else > echo "unprivileged user namespaces supported" > fi > fi I'm afraid I didn't have much luck with this: $ uname -a Linux euramoo3.qld.nectar.org.au 2.6.32-642.3.1.el6.x86_64 #1 SMP Tue=20 Jul 12 18:30:56 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux They are running CentOS 6.6 because they use Rocks: http://www.rocksclusters.org/ > Regardless, it remains our best hope to support unprivileged daemons. > > It would be nice to get stats on typical HPC systems. Hardware stats can be seen here, if that is what you mean? https://rcc.uq.edu.au/euramoo One difficulty is that there is 3 head nodes, and the daemon can only be=20 run on one of them. For my personal situation this isn't a big=20 difficulty since we can just designate one these as the "guix" node and=20 always login to that one. > Roel has been looking into these issues recently, so perhaps he has som= e > ideas. The Nix daemon recently switch to user namespaces: > > https://github.com/NixOS/nix/commit/c68e5913c71badc89ff346d1c6948517= ba720c93 > > We could backport this. However, running builds with UID 0 is > potentially disruptive: some packages are sensitive to this and behave > differently under UID 0 (I remember Coreutils=92 test suite does.) Als= o, > this patch switches to user namespaces, but not specifically > _unprivileged_ user namespaces. > > Using offloading as you suggest doesn=92t help: you would still need a > daemon with access to /gnu/store. I'm working with a non-standard store location, so that I do have=20 access. This means compiling everything from scratch, but then I find=20 watching "guix build --max-jobs" very beautiful. Building also doesn't=20 take too long after the first few packages are built, particularly in=20 comparison to downloading and building them manually. However, given Roel's tried and tested method I don't think I'll pursue=20 this approach. > (Thinking out loud.) > > There=92s a fun hack mind that could kinda work provided you use only > substitutes, where you wouldn=92t even need a daemon: > > 1. Compute the derivation of the package you want; normally that > requires a daemon to which we make =91add-to-store=92 RPCs, but w= e > should be able to fake them altogether; > > 2. Use (guix scripts substitute) to download a substitute for that > package, and unpack it under ~/.local, say; > > 3. Use =91call-with-container=92 (thus, unprivileged user namespace)= to > put yourself in an environment where /gnu/store/foo inside is a > bind-mount to ~/.local/gnu/store/foo outside. > > There would remain the problem of profiles and grafts, which are normal > derivations. > > When you think about it, it amounts to reimplementing (part of) the > daemon functionality as a library, which is probably the way to go. > That is, we could implement =91add-to-store=92 and =91build-derivations= =92 such > that they would operate locally under ~/.local. As a first milestone, > =91build-derivations=92 could fail unless there=92s a substitute availa= ble. > > Food for thought! Indeed, clever. On 20/06/16 22:23, Ludovic Court=E8s wrote: > ludo@gnu.org (Ludovic Court=E8s) skribis: > >> Regardless, it remains our best hope to support unprivileged daemons. > Also, I did not explicitly mention it, but I think this unprivileged > user namespace thing should just be one part of the strategy. I agree supporting non-privileged, non-container setups would be good.=20 OTOH, I cannot think of a way to do this that doesn't require a separate=20 build machine or a way that supports "guix environment". > In parallel, it=92s worth discussing with cluster sysadmins and see if > they can have guix-daemon running on the cluster. There are good > reasons for them to do that compared to letting each user do their own > thing, and one of them is improved resource usage.Ricardo outlined the = setup he came up with on a cluster here: > > http://elephly.net/posts/2015-04-17-gnu-guix.html > > and we have a bunch of arguments in store ;-): > > https://hal.inria.fr/hal-01161771/en > > In the end, for a sysadmin, it=92s a cost/benefit tradeoff. In some > situations, providing Guix may mean much less work for cluster admins. I haven't had much luck with this so far, though there is some interest.=20 I'm hoping to convince the sysadmins that Guix is mature enough software=20 by running my own setup for a while. The difference between the number=20 of bioinformatics packages available in Guix versus the number of=20 available modules in the HPC reflects very well on us here - that would=20 be a primary advantage. On a separate cluster where I do have root,=20 we've also had good success in adapting profiles to sit behind=20 modulefiles so that users are unaware that behind the scenes some=20 packages have been transitioned to being built by Guix, especially as we=20 upgrade the OS. On 20/06/16 19:06, Roel Janssen wrote: > Hello Ben, > > It seems like we are facing a similar problem. A proper solution takes > a lot more work and a lot more time I believe. I am also currently > working on a more complete guide to do this, but here I tried to get th= e > essentials written down. > > As far as software deployment goes, I have done the following to get it > on the restricted environment (in my case a cluster, on your case, a > super computer): Actually I misspoke, it is a cluster. I followed your instructions and=20 it is working well, thanks! I would be happy to contribute to a guide on=20 doing this if that is of use. > 1. Get /gnu/store, or bootstrap your own store with a custom prefix > (I've done the latter) on a VM or a machine that has super user > privileges (let's call this the "build host"). > > For a custom prefix, you need to build guix from source with: > > ./configure --with-store-dir=3D/hpc/custom/guix-store \ > --localstatedir=3D/hpc/custom/guix-state/guix > > You should change the environment variables: NIX_STATE_DIR and > NIX_STORE_DIR, before running the daemon, and before running the guix > command as a user. In my case, I used: > > export NIX_STATE_DIR=3D/hpc/custom/guix-state/guix > export NIX_STORE_DIR=3D/hpc/custom/guix-store > guix-daemon --cores=3D4 --max-jobs=3D4 --no-substitutes --build-user= s-group=3Dguixbuild I didn't find exporting those environment variables necessary, except=20 when running guix and guix-daemon on the restricted machine. > 2. Build the packages you want to deploy on the HPC on the build host. > > export NIX_STATE_DIR=3D/hpc/custom/guix-state/guix > export NIX_STORE_DIR=3D/hpc/custom/guix-store > guix package -i > > > 3. Copy the store and profiles. This is a bit more tricky. In my case= , > hardlinks would not work because of the properties of our storage > system. I used the following to copy the store and the profile (and > update it later on): > > rsync -lrt --delete --exclude=3D.links /hpc/custom/guix-store user@r= estricted-machine:/hpc/custom > rsync -lrt --delete --exclude=3D.links /hpc/custom/guix-state user@r= estricted-machine:/hpc/custom I simplified this by putting all the things to rsync in a single folder=20 so that a single call was needed, and I found "-z" helped too. > I excluded the .links directory to save space (you could copy them as > normal files instead of hardlinks, and the size of your store will > double). Without this directory, you cannot efficiently do package > management, so don't remove it on the build host. > > I didn't use the offloading mechanism on Guix. I avoid using the > guix-daemon entirely, and reduce the deployment problem to an rsyncable > thing. > > From here on, you can run programs as usual by adding > /hpc/custom/guix-state/guix/profiles/per-user//guix-profile/b= in > to your path (and the other relevant environment variables). > > If you install guix in your store, you can run guix-daemon on the > restricted machine and get 'guix package --search-paths', 'guix graph > ...' and even 'guix gc' to work. I haven't tested the other commands > yet. I take it by 'guix gc' you mean 'guix gc --references' since you don't=20 want to modify the store, correct? I only tried 'guix package=20 --search-paths', which is probably the most important for users. Thanks, ben