Re: [WIP Patch] Adding an FHS container to guix shell

[John Kehayias <john.kehayias@protonmail.com>]

Hi Ludo’,

------- Original Message -------
On Monday, July 18th, 2022 at 7:40 AM, Ludovic Courtès wrote:

> Hello!
>
> John Kehayias skribis:
>
> > 2. Typically binaries will expect the ld loader to use
> > /etc/ld.so.cache for finding libraries.
>
> Not necessarily. /etc/ld.so.cache is an optimization, but it’s entirely
> possible to opt out: if that file is missing, things will work fine, but
> libc will ‘stat’ more to find files. That would make glibc-for-fhs
> unnecessary.
>

I was thinking mostly of binaries, though I've also encountered code that wanted to read the ldcache (e.g. parsing ldconfig -p directly), for some reason. Is there any difficulty with having already some /etc/ld.so.cache from some package in the container profile? Perhaps that leads some programs to rely on the cache and then not continue searching for libraries? I'm not sure how that works, but only know that I've found the ldcache necessary (at least without modifying source) for things I would need the FHS container for.

My overall goal with this FHS option is to mimic both the standard and the typical expectations of distros that follow it (global ld cache, certain directories in PATH, etc). In my testing I didn't get very far without the modified glibc, though perhaps there are other ways around it. (There's patchelf, though hard to scale that I think.)

For example, lots of language build sytems/package managers will bring in some binaries, whether it is just to install (rustup does this) or because something like a web automation framework downloads browser binaries to use. My goal here was to make this all "just work," but I can understand wanting to keep it minimal.

I'll proceed with the glibc-for-fhs and ldcache for now, but open to discussing and hearing what everyone thinks. Overall, I find it necessary to enable the use-cases I'm thinking of.

> > + ;; Set up an FHS container.
> > + (when fhs-container?
> > + ;; Set up the expected bin and library directories as symlinks to
> > + ;; the profile lib directory. Note that this is assuming a 64bit
> > + ;; architecture.
> > + (let ((lib-dir (string-append profile "/lib")))
> > + (symlink lib-dir "/lib64")
> > + (symlink lib-dir "/lib")
> > + (mkdir-p "/usr")
> > + (symlink lib-dir "/usr/lib"))
>
> Instead of adding code here, maybe you could do in a more declarative
> fashion, like:
>
> (define fhs-mappings
> (list (file-system-mapping
> (source (string-append profile "/bin")) (target "/bin"))
> …))
>
> and append that to the ‘mappings’ variable there. It’s not necessarily
> more compact, but maybe marginally easier to read?
>

Thanks, that's all a good idea for tidying up! I did so, further breaking down the FHS directories to ones we can map directly like this, others that need directory contents symlinked (because the container already has e.g. /bin not empty), and then creating the optional FHS directories to link to these. For example, /lib is mapped to profile/lib and /usr/lib is linked to /lib. Again, not strictly necessary, but I was trying to follow the structure of other distros here. And this way there is still "one" place where everything from the profile ends up, /lib, /bin, etc.

> > + ;; Define an entry script to start the container: generate
> > + ;; ld.so.cache, supplement $PATH, and include command.
>
> I’d leave ld.so.cache generation out.
>

As discussed above, I'm finding this rather necessary and makes for a smooth FHS-container experience. I'm open to other options, maybe making this an additional option or hiding it away, though I think the design here fits the "pretend to be like another distro and just work" that I was going for.

> > + (call-with-output-file "/tmp/fhs.sh"
> > + (lambda (port)
> > + (display "ldconfig -X -f /tmp/ld.so.conf" port)
> > + (newline port)
> > + (display "export PATH=/bin:/usr/bin:/sbin:/usr/sbin:$PATH" port)
>
> I think the default value of PATH in our libc is the FHS one:
>
> --8<---------------cut here---------------start------------->8---
>
> $ env -i $(type -P strace) -e execve -f $(type -P guile) -c '(system* "whatever")'
> execve("/gnu/store/lpcjxka7hx3ypv4nz47g08k4m2syqwlj-profile/bin/guile", ["/gnu/store/lpcjxka7hx3ypv4nz47g0"..., "-c", "(system* \"whatever\")"], 0x7ffede27ad38 /* 0 vars /) = 0
> /home/ludo/.guix-home/profile/bin/strace: Process 9727 attached
> /home/ludo/.guix-home/profile/bin/strace: Process 9728 attached
> /home/ludo/.guix-home/profile/bin/strace: Process 9729 attached
> /home/ludo/.guix-home/profile/bin/strace: Process 9730 attached
> /home/ludo/.guix-home/profile/bin/strace: Process 9731 attached
> /home/ludo/.guix-home/profile/bin/strace: Process 9732 attached
> [pid 9732] execve("/bin/whatever", ["whatever"], 0x7ffed6d967c8 / 0 vars /) = -1 ENOENT (No such file or directory)
> [pid 9732] execve("/usr/bin/whatever", ["whatever"], 0x7ffed6d967c8 / 0 vars */) = -1 ENOENT (No such file or directory)
> In execvp of whatever: No such file or directory
> --8<---------------cut here---------------end--------------->8---
>
>
> So you could leave it undefined, but ‘load-profile’ in
> ‘launch-environment’ will define it.
>

I'm not sure what you were showing here/I'm a bit confused. If I do just 'guix shell -C' for example, I can't just do 'sh' or 'exec sh' in this environment although /bin/sh exists (as a link to /gnu/store/...bash-.../bin/sh). While your example does indeed find sh (if I include the needed inputs to run the example), I'm not sure how that relates to running things directly. But maybe you just mean in terms of libc directly. Sorry, I don't know quite enough in this area.

In the container PATH only has profile/bin. Additionally, we frequently see hardcoded path assumptions like /bin or /usr/bin, or expectations of PATH being set with these. Adding, e.g. coreutils and which to the shell, which will happily find 'ls' but not 'sh', as which just searches PATH of course.

This could be worked around by the user, but again, I was going for "just works" as much as possible. Or at least as other distros "just work". (Side note: not that I think this is better! All this FHS stuff gives me even better appreciation for the design of Guix, despite the initial complications one sees as a new user.)

> Instead of the wrapper script, maybe you could extend
> ‘launch-environment’ so the caller can make it override certain
> variables? I would find it a bit clearer.
>

Would this still be better if we do keep the ldcache being generated? I didn't see a quick way to just add in some setup commands to the container, though that could be a nice extension. Admittedly, my script here was the simplest hack I saw that worked, so I'm not married to it by any means. Again, the user could also generate the ldcache, but was trying to make it all automatic.

> Thanks,
> Ludo’.

Thanks for the constructive comments! It definitely helped me clean up the implementation, which I'll be sending shortly to the patches list. I'll followup on this thread as well to point people to a patch they can try.

For my testing I've been able to do work with web automations (uses binaries of browsers), tested that AppImages can work, built a Rust project, etc. A useful tool, even if it is one I hope I don't have to reach for too often.

John