bug#62253: Fakechroot execution engine doesn’t outlive ‘exec’ calls

bug#62253: Fakechroot execution engine doesn’t outlive ‘exec’ calls

[Ludovic Courtès]

For ‘guix pack -RR’ packs, unlike the “userns” and “proot” execution
engines, the “fakechroot” execution engine doesn’t survive ‘exec’ calls:

--8<---------------cut here---------------start------------->8---
$ mkdir -p /tmp/fakechroot-test && cd /tmp/fakechroot-test/ && tar xf $(guix pack -RR openmpi intel-mpi-benchmarks bash-minimal -S /bin=bin)
$ unshare -m -U -r -f sh -c 'mount -t tmpfs none /gnu; GUIX_EXECUTION_ENGINE=fakechroot /tmp/fakechroot-test/bin/bash'
bash-5.1# echo /gnu/store/*coreutils*/bin/ls
/gnu/store/d251rfgc9nm2clzffzhgiipdvfvzkvwi-coreutils-8.32/bin/ls /gnu/store/vqdsrvs9jbn0ix2a58s99jwkh74124y5-coreutils-minimal-8.32/bin/ls
bash-5.1# test -f /gnu/store/*coreutils-8*/bin/ls
bash-5.1# echo $?
0
bash-5.1# /gnu/store/*coreutils-8*/bin/ls
bash: /gnu/store/d251rfgc9nm2clzffzhgiipdvfvzkvwi-coreutils-8.32/bin/ls: No such file or directory
--8<---------------cut here---------------end--------------->8---

This is because the ELF interpreter of the unwrapped ‘ls’ binary remains
/gnu/store/…-glibc-2.33/lib/ld-linux-x86-64-so.2 and no LD_PRELOAD
interposition can address that.

In this case, adding ‘coreutils’ to the profile (on the ‘guix pack’
command line) would give us wrapped binaries, and the problem is solved.
But in other cases, it’s not that simple.  For instance, libmpi.so from
Open MPI tries to exec one its programs, using its absolute file name:

--8<---------------cut here---------------start------------->8---
$ unshare -m -U -r -f sh -c 'mount -t tmpfs none /gnu; GUIX_EXECUTION_ENGINE=fakechroot /tmp/fakechroot-test/bin/IMB-MPI1'
--------------------------------------------------------------------------
The singleton application was not able to find the executable "orted" in
your PATH or in the directory where Open MPI/OpenRTE was initially installed,
and therefore cannot continue.

For reference, we tried the following command:

  /gnu/store/c7g9qalmbz4a94hwzk1v1cbq7n5m8plq-openmpi-4.1.4/bin/orted

and got the error No such file or directory.

[…]
--8<---------------cut here---------------end--------------->8---

I can think of several ways to address that:

  1. Change the exec* wrappers in libfakechroot such that, on ENOENT,
     they try a direct ld.so invocation to run program, like
     ‘run-in-namespace.c’ does.

     Problem is that for this to work correctly, it would need to
     compute the ‘--library-path’ argument at run time, by computing the
     equivalent of (map dirname (file-needed/recursive program)).
     Impractical at best.

  2. Wrap/graft every package in the closure (as opposed to generating
     wrappers for just those packages that appear in the profile, which
     is what ‘guix pack’ currently does).

     The downside is that the “userns” and “proot” execution engines
     don’t need something this heavyweight: they just need a leaf
     package to be wrapped.

  3. Ignore the problem.  After all, we’re talking about a corner case
     of the “fakechroot” engine, which is a niche within a niche.

Food for thought…

Ludo’.

bug#62253: Fakechroot execution engine doesn’t outlive ‘exec’ calls

[Josselin Poiret via Bug reports for GNU Guix]

[-- Attachment #1: Type: text/plain, Size: 1428 bytes --]

Hi Ludo,

Ludovic Courtès <ludovic.courtes@inria.fr> writes:

> I can think of several ways to address that:
>
>   1. Change the exec* wrappers in libfakechroot such that, on ENOENT,
>      they try a direct ld.so invocation to run program, like
>      ‘run-in-namespace.c’ does.
>
>      Problem is that for this to work correctly, it would need to
>      compute the ‘--library-path’ argument at run time, by computing the
>      equivalent of (map dirname (file-needed/recursive program)).
>      Impractical at best.
>
>   2. Wrap/graft every package in the closure (as opposed to generating
>      wrappers for just those packages that appear in the profile, which
>      is what ‘guix pack’ currently does).
>
>      The downside is that the “userns” and “proot” execution engines
>      don’t need something this heavyweight: they just need a leaf
>      package to be wrapped.
>
>   3. Ignore the problem.  After all, we’re talking about a corner case
>      of the “fakechroot” engine, which is a niche within a niche.
>
> Food for thought…

I would like to be proven wrong, but I don't think anyone has run into
this, and there are other possible engines (that do require more
privileges, sure). It seems quite non-trivial to fix, so this can
probably go on the back-burner until someone actually complains (please
do so).

Best,
-- 
Josselin Poiret

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 682 bytes --]

bug#62253: Fakechroot execution engine doesn’t outlive ‘exec’ calls

[Ludovic Courtès]

Hi,

Josselin Poiret <dev@jpoiret.xyz> skribis:

> I would like to be proven wrong, but I don't think anyone has run into
> this, and there are other possible engines (that do require more
> privileges, sure). It seems quite non-trivial to fix, so this can
> probably go on the back-burner until someone actually complains (please
> do so).

The Open MPI example I gave earlier is one that I’m interested in for
work (running packs on HPC clusters that have neither unprivileged user
namespaces nor Singularity¹).  It might be that we can work around the
problem though, we’ll see…

Ludo’.

¹ https://hpc.guix.info/blog/2020/05/faster-relocatable-packs-with-fakechroot/

bug#62253: Fakechroot execution engine doesn’t outlive ‘exec’ calls

[Ludovic Courtès]

Ludovic Courtès <ludovic.courtes@inria.fr> skribis:

> In this case, adding ‘coreutils’ to the profile (on the ‘guix pack’
> command line) would give us wrapped binaries, and the problem is solved.
> But in other cases, it’s not that simple.  For instance, libmpi.so from
> Open MPI tries to exec one its programs, using its absolute file name:
>
> $ unshare -m -U -r -f sh -c 'mount -t tmpfs none /gnu; GUIX_EXECUTION_ENGINE=fakechroot /tmp/fakechroot-test/bin/IMB-MPI1'
> --------------------------------------------------------------------------
> The singleton application was not able to find the executable "orted" in
> your PATH or in the directory where Open MPI/OpenRTE was initially installed,
> and therefore cannot continue.
>
> For reference, we tried the following command:
>
>   /gnu/store/c7g9qalmbz4a94hwzk1v1cbq7n5m8plq-openmpi-4.1.4/bin/orted
>
> and got the error No such file or directory.

In the case of Open MPI as shown above, there’s actually an easy fix:
telling Open MPI where to look for ‘orted’.

Assuming you created a pack with:

  guix pack -RR openmpi intel-mpi-benchmarks bash-minimal -S /bin=bin

You can run code extracted from the tarball like this:

--8<---------------cut here---------------start------------->8---
export GUIX_EXECUTION_ENGINE=performance
salloc -N2 ./bin/mpirun -np 2 --launch-agent ./bin/orted \
  --map-by node -x GUIX_EXECUTION_ENGINE=performance -- \
  ./bin/IMB-MPI1 PingPong
--8<---------------cut here---------------end--------------->8---

The ‘--launch-agent’ trick allows us to work around the fact that exec’d
code escapes the fakechroot environment.

I’m closing this bug as “wontfix”.

Ludo’.