Unprivileged /gnu/store with PRoot

Unprivileged /gnu/store with PRoot

[Ludovic Courtès]

Hello Guix!

In hostile environments (read: machines that lack Guix and where you’re
not root, such as HPC clusters), it can be hard to manage software with
Guix.

We can use ‘guix pack’ to build a bundle on one machine and ship it to
the target machine.  But then, on the target machine, we need to be able
to “map” the pack’s root directory to the root directory of the
processes we run.

When user namespaces are enabled, this can be achieved with ‘unshare’
and ‘chroot’ as shown at
<https://www.gnu.org/software/guix/news/creating-bundles-with-guix-pack.html>.
However user namespaces are often disabled by distros for lack of
confidence in their security properties¹.

One way to work around the problem is to use PRoot, a ptrace(2)-based
tool to virtualize the file system².  With the ‘proot-static’ package I
just pushed, one can run, say, hwloc, on such a hostile machine by
sending locally-created packs as well as ‘proot’:

  scp $(guix build proot-static)/bin/proot hostile:
  scp $(guix pack hwloc -S /bin=bin) hostile:hwloc.tgz

and then on the hostile machine:

  mkdir ~/.local
  cd ~/.local
  tar xf ~/hwloc.tgz
  cd
  ./proot -b .local:/ /bin/lstopo

where “proot -b .local:/” essentially “bind-mounts” ~/.local to /.

Pretty cool no?  :-)

PRoot adds overhead since it has to intercept every syscall.  However,
for a mostly computational process, it should not be much of a problem.

Ludo’.

¹ See for instance
  <http://rhelblog.redhat.com/2015/07/07/whats-next-for-containers-user-namespaces/>
  and the recent AF_PACKET vulnerability
  <https://googleprojectzero.blogspot.com/2017/05/exploiting-linux-kernel-via-packet.html>.

² https://github.com/proot-me/PRoot

Re: Unprivileged /gnu/store with PRoot

[Alex Sassmannshausen]

Ludovic Courtès writes:

> Hello Guix!
>
> In hostile environments (read: machines that lack Guix and where you’re
> [...]
>
> Pretty cool no?  :-)

… Very cool! I once again stand gob-smacked :-D

Alex

Re: Unprivileged /gnu/store with PRoot

[Hartmut Goebel]

Am 12.05.2017 um 17:53 schrieb Ludovic Courtès:
> Pretty cool no?  :-)
Indeed. :-)

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Unprivileged /gnu/store with PRoot - or relocate

[Pjotr Prins]

I explored this and actually built Nix/Guix on proot in the past for the
same reasons

  https://github.com/pjotrp/nix-no-root

Mind, it *is* slow. That is why I came up with relocatable Guix which
works much better. No performance loss. Just an extra installation
step. Relocatable Guix works great, but it requires effort to make it
user friendly. We can automate creating relocatable packages with guix
pack. All it needs is an additional installer that rewrites the paths.
The other limitation is that the installer prefix can be no longer
than ~50 characters because it gobbles up the Guix store path + hash.

I have written all the code for that. The installer is fast.

Anyone interested in that type of solution? We can make it work, even
as automated builds. I think it would be great for HPC and can even be
part of non-root packagers, such as brew and conda.

Pj.

On Fri, May 12, 2017 at 05:53:21PM +0200, Ludovic Court??s wrote:
> Hello Guix!
> 
> In hostile environments (read: machines that lack Guix and where you???re
> not root, such as HPC clusters), it can be hard to manage software with
> Guix.
> 
> We can use ???guix pack??? to build a bundle on one machine and ship it to
> the target machine.  But then, on the target machine, we need to be able
> to ???map??? the pack???s root directory to the root directory of the
> processes we run.
> 
> When user namespaces are enabled, this can be achieved with ???unshare???
> and ???chroot??? as shown at
> <https://www.gnu.org/software/guix/news/creating-bundles-with-guix-pack.html>.
> However user namespaces are often disabled by distros for lack of
> confidence in their security properties??.
> 
> One way to work around the problem is to use PRoot, a ptrace(2)-based
> tool to virtualize the file system??.  With the ???proot-static??? package I
> just pushed, one can run, say, hwloc, on such a hostile machine by
> sending locally-created packs as well as ???proot???:
> 
>   scp $(guix build proot-static)/bin/proot hostile:
>   scp $(guix pack hwloc -S /bin=bin) hostile:hwloc.tgz
> 
> and then on the hostile machine:
> 
>   mkdir ~/.local
>   cd ~/.local
>   tar xf ~/hwloc.tgz
>   cd
>   ./proot -b .local:/ /bin/lstopo
> 
> where ???proot -b .local:/??? essentially ???bind-mounts??? ~/.local to /.
> 
> Pretty cool no?  :-)
> 
> PRoot adds overhead since it has to intercept every syscall.  However,
> for a mostly computational process, it should not be much of a problem.
> 
> Ludo???.
> 
> ?? See for instance
>   <http://rhelblog.redhat.com/2015/07/07/whats-next-for-containers-user-namespaces/>
>   and the recent AF_PACKET vulnerability
>   <https://googleprojectzero.blogspot.com/2017/05/exploiting-linux-kernel-via-packet.html>.
> 
> ?? https://github.com/proot-me/PRoot
> 

-- 

Re: Unprivileged /gnu/store with PRoot

[Maxim Cournoyer]

[-- Attachment #1: Type: text/plain, Size: 1396 bytes --]

Hi Ludovic!

ludovic.courtes@inria.fr (Ludovic Courtès) writes:

> Hello Guix!
>
> In hostile environments (read: machines that lack Guix and where you’re
> not root, such as HPC clusters), it can be hard to manage software with
> Guix.
>

[...]

>
> One way to work around the problem is to use PRoot, a ptrace(2)-based
> tool to virtualize the file system².  With the ‘proot-static’ package I
> just pushed, one can run, say, hwloc, on such a hostile machine by
> sending locally-created packs as well as ‘proot’:
>
>   scp $(guix build proot-static)/bin/proot hostile:
>   scp $(guix pack hwloc -S /bin=bin) hostile:hwloc.tgz
>
> and then on the hostile machine:
>
>   mkdir ~/.local
>   cd ~/.local
>   tar xf ~/hwloc.tgz
>   cd
>   ./proot -b .local:/ /bin/lstopo
>
> where “proot -b .local:/” essentially “bind-mounts” ~/.local to /.
>
> Pretty cool no?  :-)

Pretty clever hack! :) I'm almost looking forward my next "hostile"
machine encounter... eh!

>
> PRoot adds overhead since it has to intercept every syscall.  However,
> for a mostly computational process, it should not be much of a problem.
>

Which essentially means when the proot'd software is first loaded from disk?
Am I right to think that the performance would be the same after the
prooted program is all mapped in memory?

Thanks for sharing!

Maxim

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Unprivileged /gnu/store with PRoot

[Ludovic Courtès]

Hi!

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> Which essentially means when the proot'd software is first loaded from disk?
> Am I right to think that the performance would be the same after the
> prooted program is all mapped in memory?

No, every syscall is going to take much longer.  So if you try, say, to
run a build process under PRoot, that’s probably going to be much
slower.  OTOH, if you run a program that spends most of its time doing
computations, there’s probably not a big difference.

Ludo’.

Re: Unprivileged /gnu/store with PRoot - or relocate

[Ludovic Courtès]

Hello!

Pjotr Prins <pjotr.public12@thebird.nl> skribis:

> I explored this and actually built Nix/Guix on proot in the past for the
> same reasons
>
>   https://github.com/pjotrp/nix-no-root

Right, thanks for the reminder!

Back then the target was mostly to run a full Nix/Guix under PRoot,
right?  I suppose running guix-daemon & co. under PRoot would be
terribly slow, especially when building stuff.

> Mind, it *is* slow. That is why I came up with relocatable Guix which
> works much better. No performance loss. Just an extra installation
> step. Relocatable Guix works great, but it requires effort to make it
> user friendly. We can automate creating relocatable packages with guix
> pack. All it needs is an additional installer that rewrites the paths.
> The other limitation is that the installer prefix can be no longer
> than ~50 characters because it gobbles up the Guix store path + hash.
>
> I have written all the code for that. The installer is fast.
>
> Anyone interested in that type of solution? We can make it work, even
> as automated builds. I think it would be great for HPC and can even be
> part of non-root packagers, such as brew and conda.

Definitely.  As discussed before, my personal preference for the
technical solutions to this problem is, in this order:

  1. user namespaces (unfortunately rarely available, at least on HPC);

  2. PRoot, provided the performance is okay for the target workload;

  3. relocation as you presented it at FOSDEM.

When #1 is available that’s good.  In other cases, one has to choose
between #2 and #3 depending on performance and other tradeoffs.

Each of these approaches has its pros and cons.  We all know that the
one true solution to file system virtualization is GNU/Hurd… but we’re
not there yet!  :-)

Ludo’.

Re: Unprivileged /gnu/store with PRoot

[Ricardo Wurmus]

Ludovic Courtès <ludovic.courtes@inria.fr> writes:

> One way to work around the problem is to use PRoot, a ptrace(2)-based
> tool to virtualize the file system².  With the ‘proot-static’ package I
> just pushed, one can run, say, hwloc, on such a hostile machine by
> sending locally-created packs as well as ‘proot’:
>
>   scp $(guix build proot-static)/bin/proot hostile:
>   scp $(guix pack hwloc -S /bin=bin) hostile:hwloc.tgz
>
> and then on the hostile machine:
>
>   mkdir ~/.local
>   cd ~/.local
>   tar xf ~/hwloc.tgz
>   cd
>   ./proot -b .local:/ /bin/lstopo
>
> where “proot -b .local:/” essentially “bind-mounts” ~/.local to /.
>
> Pretty cool no?  :-)

That *is* very cool indeed!

> PRoot adds overhead since it has to intercept every syscall.  However,
> for a mostly computational process, it should not be much of a problem.

Can this be simplified?  We really only need that user-space bind-mount
feature, nothing else.  PRoot does more than that IIUC.  Can we preload
a little library that provides “open” and implementations of other file
access procedures, and simply (I know…) changes any access of “/gnu”
with the new target directory?

I suspect this is doomed to fail because of the way shebangs work and
because we cannot interpose internal glibc calls, though.  Ultimately,
ptrace really is what we want on Linux and sadly it’s just slow.

More power to the Hurd on HPC!

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: Unprivileged /gnu/store with PRoot

[Ludovic Courtès]

Hello!

Ricardo Wurmus <rekado@elephly.net> skribis:

> Ludovic Courtès <ludovic.courtes@inria.fr> writes:
>
>> One way to work around the problem is to use PRoot, a ptrace(2)-based
>> tool to virtualize the file system².  With the ‘proot-static’ package I
>> just pushed, one can run, say, hwloc, on such a hostile machine by
>> sending locally-created packs as well as ‘proot’:
>>
>>   scp $(guix build proot-static)/bin/proot hostile:
>>   scp $(guix pack hwloc -S /bin=bin) hostile:hwloc.tgz
>>
>> and then on the hostile machine:
>>
>>   mkdir ~/.local
>>   cd ~/.local
>>   tar xf ~/hwloc.tgz
>>   cd
>>   ./proot -b .local:/ /bin/lstopo
>>
>> where “proot -b .local:/” essentially “bind-mounts” ~/.local to /.
>>
>> Pretty cool no?  :-)
>
> That *is* very cool indeed!
>
>> PRoot adds overhead since it has to intercept every syscall.  However,
>> for a mostly computational process, it should not be much of a problem.
>
> Can this be simplified?  We really only need that user-space bind-mount
> feature, nothing else.  PRoot does more than that IIUC.  Can we preload
> a little library that provides “open” and implementations of other file
> access procedures, and simply (I know…) changes any access of “/gnu”
> with the new target directory?
>
> I suspect this is doomed to fail because of the way shebangs work and
> because we cannot interpose internal glibc calls, though.  Ultimately,
> ptrace really is what we want on Linux and sadly it’s just slow.

Exactly.  Many have written LD_PRELOADable libraries to perform file
system virtualization but it’s tedious (there are so many VFS-related
functions to interpose!) and fragile, and an application can always do a
direct syscall and bypass the thing.

So PRoot is not the preferred option, but it’s one we can resort to when
there’s no other option.  I have yet to measure its performance impact
when running some of the linear algebra software that people use/develop
here.

> More power to the Hurd on HPC!

:-)

Ludo’.

Re: Unprivileged /gnu/store with PRoot

[Pjotr Prins]

On Fri, May 12, 2017 at 05:53:21PM +0200, Ludovic Court??s wrote:
> One way to work around the problem is to use PRoot, a ptrace(2)-based
> tool to virtualize the file system??.  With the 'proot-static' package I
> just pushed, one can run, say, hwloc, on such a hostile machine by
> sending locally-created packs as well as 'proot':
> 
>   scp $(guix build proot-static)/bin/proot hostile:
>   scp $(guix pack hwloc -S /bin=bin) hostile:hwloc.tgz
> 
> and then on the hostile machine:
> 
>   mkdir ~/.local
>   cd ~/.local
>   tar xf ~/hwloc.tgz
>   cd
>   ./proot -b .local:/ /bin/lstopo
> 
> where 'proot -b .local:/' essentially 'bind-mounts' ~/.local to /.

I took it a bit further and can run guix-daemon in proot:

  https://gitlab.com/pjotrp/guix-notes/blob/master/GUIX-NO-ROOT.org

this means you can install binary packages inside proot and run them. That is
pretty cool already :).

I can also create a build system for building Guix from source. So we
are bootstrapping with our own standard environment.

Now, what I really want to do, from this, is to build packages that get
installed in a non-/gnu/store. E.g., ~/opt/store instead. It should be
possible with the instructions Roel wrote down in 

  https://github.com/UMCUGenetics/guix-additions/blob/master/doc/Deployment.md

but I am failing because it appears the guix-daemon is still linked
against the old store. After

  ./configure --localstatedir=$HOME/opt/var --with-store-dir=$HOME/opt/store --prefix=$HOME/opt/local
  make clean
  make guix-daemon
  make install

  ldd ~/opt/local/bin/guix-daemon
        linux-vdso.so.1 (0x00007ffcc4ffb000)
        libsqlite3.so.0 => not found
        libgcrypt.so.20 => not found
        libstdc++.so.6 => /gnu/store/b3z4d4zjibqix6pn58q6b1rgfhrarcaq-gcc-7.1.0-lib/lib/libstdc++.so.6 (0x00007f3c2ecb9000)
        libm.so.6 => /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libm.so.6 (0x00007f3c2e9a7000)
        libgcc_s.so.1 => /gnu/store/b3z4d4zjibqix6pn58q6b1rgfhrarcaq-gcc-7.1.0-lib/lib/libgcc_s.so.1 (0x00007f3c2e790000)
        libc.so.6 => /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libc.so.6 (0x00007f3c2e3f1000)
        /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/ld-linux-x86-64.so.2 (0x00007f3c2f039000)

Anything I am missing? 

Maybe it is logical but I want to link against $HOME/opt/store/... so I can run guix-daemon outside
proot. 

Note it still runs in proot:

   proot -0 -b /proc -b /dev -b /etc -r . -b etc_guix/acl:/etc/guix/acl guix/guix-daemon --disable-chroot
      warning: daemon is running as root, so using `--build-users-group' is highly recommended
      accepted connection from pid 15774, user user

only now the guix client balks with

   error: failed to run download program '/home/user/opt/local/libexec/guix/download': No such file or directory

That file actually exists in the proot environment

   ls -l /home/user/opt/local/libexec/guix/download
   -rwxr-xr-x 1 user 502 281 Jul 16 08:11 /home/user/opt/local/libexec/guix/download

So close to a solution...

Pj.

Re: Unprivileged /gnu/store with PRoot

[Pjotr Prins]

On Sun, Jul 16, 2017 at 02:35:14PM +0200, Pjotr Prins wrote:
>   ldd ~/opt/local/bin/guix-daemon
>         linux-vdso.so.1 (0x00007ffcc4ffb000)
>         libsqlite3.so.0 => not found
>         libgcrypt.so.20 => not found
>         libstdc++.so.6 => /gnu/store/b3z4d4zjibqix6pn58q6b1rgfhrarcaq-gcc-7.1.0-lib/lib/libstdc++.so.6 (0x00007f3c2ecb9000)
>         libm.so.6 => /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libm.so.6 (0x00007f3c2e9a7000)
>         libgcc_s.so.1 => /gnu/store/b3z4d4zjibqix6pn58q6b1rgfhrarcaq-gcc-7.1.0-lib/lib/libgcc_s.so.1 (0x00007f3c2e790000)
>         libc.so.6 => /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libc.so.6 (0x00007f3c2e3f1000)
>         /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/ld-linux-x86-64.so.2 (0x00007f3c2f039000)
> 
> Anything I am missing? 

Actually, running guix-daemon in proot is not a problem - who cares
that the build system is slow ;)

> Note it still runs in proot:
> 
>    proot -0 -b /proc -b /dev -b /etc -r . -b etc_guix/acl:/etc/guix/acl guix/guix-daemon --disable-chroot
>       warning: daemon is running as root, so using `--build-users-group' is highly recommended
>       accepted connection from pid 15774, user user
> 
> only now the guix client balks with
> 
>    error: failed to run download program '/home/user/opt/local/libexec/guix/download': No such file or directory
> 
> That file actually exists in the proot environment
> 
>    ls -l /home/user/opt/local/libexec/guix/download
>    -rwxr-xr-x 1 user 502 281 Jul 16 08:11 /home/user/opt/local/libexec/guix/download

If we can solve this we are set. Any idea why nix execv is not
running this script? I added some info to it and is simply not found.
It can't be a proot thing because I have been invoking builds etc.
already. I have a feeling I have seen this before.

If you read my document you'll see I have been running guix in proot
with success. I only want to use a different store now.

Pj.

Re: Unprivileged /gnu/store with PRoot

[Pjotr Prins]

On Sun, Jul 16, 2017 at 02:51:00PM +0200, Pjotr Prins wrote:
> >    error: failed to run download program '/home/user/opt/local/libexec/guix/download': No such file or directory
> > 
> > That file actually exists in the proot environment
> > 
> >    ls -l /home/user/opt/local/libexec/guix/download
> >    -rwxr-xr-x 1 user 502 281 Jul 16 08:11 /home/user/opt/local/libexec/guix/download
> 
> If you read my document you'll see I have been running guix in proot
> with success. I only want to use a different store now.

Fixed. The problem is that /bin/bash would not load from the download
script (even though it exists and /bin/bash starts a shell. Oh wait, I
did not bind /lib in proot - ayeee). Replacing that with a store bash
path it started downloading in proot into a new store path.

Hopefully a last hurdle. Cross fingers!

Pj.
--