Re: [WIP Patch] Adding an FHS container to guix shell

[John Kehayias <john.kehayias@protonmail.com>]

Hi Maxim,

(with copious snips of the long original email)

------- Original Message -------
On Friday, July 15th, 2022 at 9:43 AM, Maxim Cournoyer wrote:

>
>
> Hi John,
>
> John Kehayias john.kehayias@protonmail.com writes:
>
> > Hi Guixers,
> >
> > Apologies for the long email, so let me start with the punchline:
> > attached is a diff which adds an '--fhs-container' (or -F) option to
> > guix shell/environment to set up an FHS-like container. This includes
> > the usual /lib directory and a glibc which loads (a generated in the
> > container) /etc/ld.so.cache. This should allow running most things
> > that expect a more "typical" Linux environment. Give it a try!
>
> Neat hack!

Thanks!

> > diff --git a/gnu/packages/base.scm b/gnu/packages/base.scm
> > index 4bdc3e7792..1b4c99d3e9 100644
> > --- a/gnu/packages/base.scm
> > +++ b/gnu/packages/base.scm
> > @@ -928,6 +928,20 @@ (define-public glibc
> > (license lgpl2.0+)
> > (home-page "https://www.gnu.org/software/libc/")))
> >
> > +;; Define glibc-for-fhs (with a name that allows grafts for glibc), a variation
> > +;; of glibc which uses the default ld.so.cache, useful in FHS containers.
> > +;; Note: should this be hidden?
>
> ^ This should definitely be hidden since it's for internal consumption
> only. Also, if grafting ended up not being needed, the variable should
> eb renamed 'glibc-for-fhs', an mentions of grafting in the comments
> removed.
>

I'll respond to the grafting comment more below, but for now I just wanted to have the options available in this "preview" patch. I'm also wondering if it is worthwhile to leave the grafting option open for a user (so they can still explicitly do a --with-graft), though I guess we can change in the future if a bug report comes up. And for selfish reasons I wanted the grafting code somewhere so I wouldn't forget how to add that option in the container code behind the scenes :)

> > + ;; Set up an FHS container.
>
> I feel like FHS should be spelled in full somewhere (in the doc would be
> nice).
>

Agreed, along with explicit documentation as you noted in your followup email. I would want to link to the spec in the manual for reference as well.

> > + (when fhs-container?
> > + ;; Set up the expected bin and library directories as symlinks to
> > + ;; the profile lib directory. Note that this is assuming a 64bit
> > + ;; architecture.
> > + (let ((lib-dir (string-append profile "/lib")))
> > + (symlink lib-dir "/lib64")
>
> I think /lib64 is not needed (see below).
>

Right. Everything was all linked so I had just picked one. As you detail below, this should be cleaned up to be the generic standard specified for FHS.

> > + (symlink lib-dir "/lib")
> > + (mkdir-p "/usr")
> > + (symlink lib-dir "/usr/lib"))
> > + ;; Note: can't symlink full /bin in the container due to the sh
> > + ;; symlink.
>
> Can you explain more exactly why symlinks prevent this to work? I think
> I've had issues like this before (e.g., #46782), but couldn't really
> pinpoint the exact nature of the limitation.
>

Oh, I didn't mean anything complicated, just that you can't link /bin since a symlink /bin/sh was already created for containers. Instead, we could mkdir /bin and then add the sh symlink along with everything from profile/bin with a little loop. I just didn't think to do it at the time for my first pass at this.

I think for the final version it would be good to do this, as well as looping through any profile/lib/ subdirectories to add to ld.so.conf. I know at least our libnss3 is in a subdirectory for some reason, which needs to be added to where ldconfig looks or else things that need libnss3 won't find it.

> > + (symlink (string-append profile "/bin") "/usr/bin")
> > + (symlink (string-append profile "/sbin") "/sbin")
> > + (symlink (string-append profile "/sbin") "/usr/sbin")
>
> Other symbolic links I think we should include:
>
> * /include -> /usr/include (section 4.3 of the FHS 3.0 spec)
> * /libexec -> /usr/libexec (optional, section 4.7)
> * /share -> /usr/share (section 4.11)
>

Thanks, good idea. Surprisingly (or not?) the /lib and ldcache seem to be the big one. I guess env variables (search-paths) take care of most everything else, but we should set it all up.

> > + ;; Provide a frequently expected 'cc' symlink to gcc, though this
> > + ;; could also be done by the user in the container, e.g. in
> > + ;; $HOME/.local/bin and adding that to $PATH. Note: we do this
> > + ;; in /bin since that already has the sh symlink and can't write
> > + ;; to the other bin directories that are already symlinks themselves.
> > + (symlink (string-append profile "/bin/gcc") "/bin/cc")
> > + ;; TODO: python may also be expected to symlink to python3.
>
> No need for a TODO, I think users wanting that could simply use
> 'python-wrapper', which exists for this purpose, no?
>

TIL. Yes, perfect.

Is there any use to have a similar one for cc/gcc? Would that be useful outside of this container? I know we frequently need to specify $CC in build patching, I just haven't done much building outside of packaging to know otherwise.

> > + ;; Guix's ldconfig doesn't seem to search in FHS default
> > + ;; locations, so provide a minimal ld.so.conf.
> > + ;; TODO: this may need more, e.g. libnss3 is in /lib/nss
> > + (call-with-output-file "/tmp/ld.so.conf"
> > + (lambda (port)
> > + (display "/lib64" port)
> > + (newline port)))
>
> Shouldn't this be '/lib' rather than '/lib64'? Per sections 3.9 and
> 3.10 of the FHS 3.0 [0], '/lib' is essential, while '/lib<qual>' is
> optional.
>
> [0] https://refspecs.linuxfoundation.org/FHS_3.0/fhs-3.0.pdf
>

Yes, thanks. I should make this all more generic per the standard.

> > + (launch-environment (if fhs-container?
> > + ;; Use the FHS start script.
> > + ;; FIXME: probably the default command should
> > + ;; be different as it spawns a different shell?
> > + '("/bin/sh" "/tmp/fhs.sh")
>
> I'm not sure what could be improved here (why the FIXME?); could you
> explain what you had on mind in more details?
>

The shell that is launched looks more like a plain 'sh' rather than what you normally get in an empty guix shell --container. So what I currently have is a call to /bin/sh /tmp/fhs.sh where the last line of the fhs.sh script is the command specified by the user, or else /bin/sh itself. This results in a different prompt at least. I may have gotten some warning after exiting the container as well (didn't find /bin/sh? I don't have it in front of me). I'll take a look, but open to how best to spawn the shell from this fhs.sh script as the fallback when no command is given.

> > + ;; For an FHS-container, add a glibc that uses
> > + ;; /etc/ld.so.cache.
> > + (if fhs-container?
> > + (alist-cons 'package '(ad-hoc-package "gcfhs")
> > + opts)
> > + ;; Alternatively, could graft all packages with
> > + ;; this glibc, though that seems unnecessary.
> > + ;; (alist-cons 'with-graft "glibc=gcfhs" opts)
> > + opts)))
>
> It's interesting that it's unnecessary. If we're confident it really
> is, then I'd remove this dead code and comment.
>

It is interesting, and at first I was having problems with linkers finding libgcc. It wasn't appearing in the ld cache, and I thought it was because the non-FHS libc was referenced in some gcc package (which then wasn't in the container). But I think that was a red herring and may have been something else I was doing while testing, since I haven't hit that problem since. And I don't think that should have caused a problem, as long as whatever libc is referenced is available, libgcc should have appeared in the cache. But I'm not sure now what was happening, so I had tried grafting, came back to trying without grafts and it still worked...

In any event, some testing would be helpful. Just including the FHS libc is very simple and quick rather than grafting. I think the main point is just that software/code tends to rely on the ld cache to find libraries, so the ld loader needs to do this for "default" behavior. After finding the libraries, everything else goes fine and is then linked properly in the binary. (Now I wonder if you build a binary in the FHS container if it will just run in regular Guix since everything should be linked to explicit store locations in the end...?)

>
> That's surprisingly little code, which is nice! Thanks for looking into
> it :-).
>

Thanks for the helpful comments! And credit to work that is out there that came before. Admittedly it was much easier this time (just one late evening) than when I first tried all of this for some binaries. The end result is really just needing the ld cache (generation/loading) and some symlinks, not bad at all.

> I've yet to try it, but I will shortly.
>
> Thank you,
>
> Maxim

Most welcome! And yes, some testing would be great to find what assumptions are here or should be here to make this useable without much extra fussing for each piece of software.

John