Installing guix packages without root permissions (in HPC environments)

Installing guix packages without root permissions (in HPC environments)

[Pjotr Prins]

Or relocating Guix binary packages onto a different prefix w.o. admin
rights (on a non-guix host)

I have been working on the problem of using Guix packages without
administrator rights. In some HPC environments and on most super
computers, at this point, it is next to impossible to circumvent this
issue. Wanting to deploy existing binary Guix packages on a different
prefix (until now) we have the route of building Guix on that prefix,
say $HOME/gnu, but the problem is that you have to do a full Guix
rebuild for every different prefix.

Discussing this with Eelco Dolstra a few years back at FOSDEM we came
up with the idea of simply overwriting the Nix or Guix path inside
binaries. In elf binaries the Guix path is zero-terminated. As long as
the file was shorter we could simply overwrite the path and
zero-terminate. Guix paths are easily recognized - essentially a
finger print of /gnu/store + hash value uniquely identifies the path
that needs to be patched. The new path looks like $PREFIX/package/ +
hash values.

Interestingly, I found that CONDA does the same thing for relocating
binary software. So, in my first attempt I took this strategy and it
mostly worked. Using Eelco's patchelf tool it is even possible to make
the new path longer than the old one.  Also script files can be
edited, so they simply get patched with new paths. This works for
bash, ruby and Perl which do not byte compile to a different file
format.

In case of Python and JVM byte-code files, however, paths are not
zero-terminated. In fact, even in elf files there are instances of
non-zero terminated paths. To support relocating these an installer
would need to understand the format and the jump instructions
involved.

But, I thought the easy way is to patch a path with something the has
the exact same size(!). This has the advantage that it will always
work. Trying this second strategy I wrote a new tool which replaces
the old path with a new one that takes the prefix and truncates the
rest of the path so a prefix /usr/local/bin/hello overwrites

Found @512:     /gnu/store/qv7bk62c22ms9i11dhfl71hnivyc82k2-glibc-2.22/...
Replace with    /usr/local/bin/hello/glibc-2.22-qv7bk62c22ms9i11dhfl71/...

You can see we swap the hash position and start 'eating' the path from
the end.  This should work across almost all files, unless the path is
scrambled in some way. The downside of the fixed strategy is that a
prefix can not grow beyond the size of the one in the store. Also
every store path may look a bit different between installs.

You can try it out by downloading and unpacking

    http://biogems.info/contrib/genenetwork/guix-build-hello-2.10-x86_64.tgz

Run the contained installer with

    ./install.sh prefix

Use --help, -v and -d for more output.  And run the installed tool
with

    env LC_ALL=fr_FR prefix/profile-hfmsjsvx1p68wbx0fli/bin/hello
    Bonjour, le monde!

French (if the locale resolves ;).  On non-guix machines I have
already deployed some complex packages this way, including Ruby,
sambamba and the ldc compiler (which includes LLVM).

To create such an installable and relocatable tarball start from a
package with its dependencies, such as found in a Guix archive or
created with

    guix environment --container --ad-hoc mypackage tar gzip -- tar cvzf mypackage.tgz /gnu/store

Unpack that tar ball and copy the installer files that sit in the
hello example. That is all.

The source code for the guix-relocator is at
https://github.com/pjotrp/guix-relocate/blob/master/src/main.d and the
supporting scripts are at https://github.com/pjotrp/gnu-install-bin (I
may have to change that name).

To fix the prefix restriction there are two routes, one is building
Guix itself on a large(r) path, which is the easy route, or write a
relocate patcher that can handle the non-zero terminated paths. I
think the latter is feasible too. Even at this point the fixed length
strategy is useful for most environments - the prefix can be some 40
characters long.

What does this all mean? In short, if the prefix is not too long we
can run practically all Guix binary software on a non-guix host with
normal user permissions. Something we need for HPC and supercomputing
and is the subject of my FOSDEM talk on the HPC track:

  https://fosdem.org/2017/schedule/event/hpc_deployment_guix/

There is also scope for creating a one-step installer. A (trusted)
server could distribute binary software with its dependencies to any
Linux machine. I am sure software developers are interested in that.

Re: Installing guix packages without root permissions (in HPC environments)

[Hartmut Goebel]

[-- Attachment #1: Type: text/plain, Size: 584 bytes --]

Am 17.01.2017 um 10:15 schrieb Pjotr Prins:
> But, I thought the easy way is to patch a path with something the has
> the exact same size(!). This has the advantage that it will always
> work. Trying this second strategy I wrote a new tool which replaces
> the old path with a new one that takes the prefix and truncates the
> rest of the path so a prefix /usr/local/bin/hello overwrites

Pretty cool idea!


-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |


[-- Attachment #2: 0xBF773B65.asc --]
[-- Type: application/pgp-keys, Size: 14855 bytes --]

Re: Installing guix packages without root permissions (in HPC environments)

[Ludovic Courtès]

Hi Pjotr!

Pjotr Prins <pjotr.public12@thebird.nl> skribis:

> But, I thought the easy way is to patch a path with something the has
> the exact same size(!). This has the advantage that it will always
> work. Trying this second strategy I wrote a new tool which replaces
> the old path with a new one that takes the prefix and truncates the
> rest of the path so a prefix /usr/local/bin/hello overwrites
>
> Found @512:     /gnu/store/qv7bk62c22ms9i11dhfl71hnivyc82k2-glibc-2.22/...
> Replace with    /usr/local/bin/hello/glibc-2.22-qv7bk62c22ms9i11dhfl71/...
>
> You can see we swap the hash position and start 'eating' the path from
> the end.

Ah ha, clever and fun!  :-)

> This should work across almost all files, unless the path is scrambled
> in some way. The downside of the fixed strategy is that a prefix can
> not grow beyond the size of the one in the store. Also every store
> path may look a bit different between installs.

The latter is the price to pay, I guess.

> You can try it out by downloading and unpacking
>
>     http://biogems.info/contrib/genenetwork/guix-build-hello-2.10-x86_64.tgz
>
> Run the contained installer with
>
>     ./install.sh prefix
>
> Use --help, -v and -d for more output.  And run the installed tool
> with
>
>     env LC_ALL=fr_FR prefix/profile-hfmsjsvx1p68wbx0fli/bin/hello
>     Bonjour, le monde!
>
> French (if the locale resolves ;).  On non-guix machines I have
> already deployed some complex packages this way, including Ruby,
> sambamba and the ldc compiler (which includes LLVM).

Woow, good to know that it works well in practice.

> To create such an installable and relocatable tarball start from a
> package with its dependencies, such as found in a Guix archive or
> created with
>
>     guix environment --container --ad-hoc mypackage tar gzip -- tar cvzf mypackage.tgz /gnu/store
>
> Unpack that tar ball and copy the installer files that sit in the
> hello example. That is all.
>
> The source code for the guix-relocator is at
> https://github.com/pjotrp/guix-relocate/blob/master/src/main.d and the
> supporting scripts are at https://github.com/pjotrp/gnu-install-bin (I
> may have to change that name).
>
> To fix the prefix restriction there are two routes, one is building
> Guix itself on a large(r) path, which is the easy route, or write a
> relocate patcher that can handle the non-zero terminated paths. I
> think the latter is feasible too. Even at this point the fixed length
> strategy is useful for most environments - the prefix can be some 40
> characters long.

I think the fixed-length restriction is not that bad.  The two
workarounds above look hacky and super tricky, respectively.  ;-)

Also, if we look at the big picture of non-root usage, this solution
addresses the most hostile environments: no user namespaces, no
container-spawning facility, no root guix-daemon, etc.  Granted, these
hostile environments are still commonplace in HPC, so that’s good.  :-)

> What does this all mean? In short, if the prefix is not too long we
> can run practically all Guix binary software on a non-guix host with
> normal user permissions. Something we need for HPC and supercomputing
> and is the subject of my FOSDEM talk on the HPC track:
>
>   https://fosdem.org/2017/schedule/event/hpc_deployment_guix/
>
> There is also scope for creating a one-step installer. A (trusted)
> server could distribute binary software with its dependencies to any
> Linux machine. I am sure software developers are interested in that.

Awesome!

Ludo’.

Re: Installing guix packages without root permissions (in HPC environments)

[Pjotr Prins]

On Wed, Jan 18, 2017 at 10:42:09PM +0100, Ludovic Courtès wrote:
> > To fix the prefix restriction there are two routes, one is building
> > Guix itself on a large(r) path, which is the easy route, or write a
> > relocate patcher that can handle the non-zero terminated paths. I
> > think the latter is feasible too. Even at this point the fixed length
> > strategy is useful for most environments - the prefix can be some 40
> > characters long.
> 
> I think the fixed-length restriction is not that bad.  The two
> workarounds above look hacky and super tricky, respectively.  ;-)

Yes. Also I like the fact that we use standard binaries this way that
have been well tested on Guix. If anything goes wrong the problem can
only be the relocation. That makes things tractable.

> Also, if we look at the big picture of non-root usage, this solution
> addresses the most hostile environments: no user namespaces, no
> container-spawning facility, no root guix-daemon, etc.  Granted, these
> hostile environments are still commonplace in HPC, so that’s good.  :-)

Yes, if this works it will be rather good and create awareness for
Guix. Another use-case may be firefox plugins - or other software that
wants to install binaries.  Maybe they'll wake up to guix too. From a
developers point of view Guix is awesome because it creates
reproducible environments that we can develop against. What more do we
need ;). We are now using it for a bug hunt on sambamba which only
segfaults on particular HPC setups. The guix relocatable installer is
going to help.

I'll take the installer to the level that we can do one-step installs
and provide a few packages for download to reach out to certain
communities (dlang and elixir come to mind).

Pj.
-- 

Re: Installing guix packages without root permissions (in HPC environments)

[Ludovic Courtès]

Hi!

Pjotr Prins <pjotr.public12@thebird.nl> skribis:

> On Wed, Jan 18, 2017 at 10:42:09PM +0100, Ludovic Courtès wrote:

[...]

>> Also, if we look at the big picture of non-root usage, this solution
>> addresses the most hostile environments: no user namespaces, no
>> container-spawning facility, no root guix-daemon, etc.  Granted, these
>> hostile environments are still commonplace in HPC, so that’s good.  :-)
>
> Yes, if this works it will be rather good and create awareness for
> Guix. Another use-case may be firefox plugins - or other software that
> wants to install binaries.  Maybe they'll wake up to guix too. From a
> developers point of view Guix is awesome because it creates
> reproducible environments that we can develop against. What more do we
> need ;). We are now using it for a bug hunt on sambamba which only
> segfaults on particular HPC setups. The guix relocatable installer is
> going to help.

Indeed.

> I'll take the installer to the level that we can do one-step installs
> and provide a few packages for download to reach out to certain
> communities (dlang and elixir come to mind).

For these “guix archive -f docker” may also be handy (and safer)?

Cheers,
Ludo’.

Re: Installing guix packages without root permissions (in HPC environments)

[Pjotr Prins]

On Thu, Jan 19, 2017 at 12:55:30PM +0100, Ludovic Courtès wrote:
> For these “guix archive -f docker” may also be handy (and safer)?

I am known to dislike docker.

Pj.

Re: Installing guix packages without root permissions (in HPC environments)

[Pjotr Prins]

Relocatable guix in action: https://github.com/lomereiter/sambamba/issues/219