[WIP Patch] Adding an FHS container to guix shell

[John Kehayias <john.kehayias@protonmail.com>]

[-- Attachment #1: Type: text/plain, Size: 6638 bytes --]

Hi Guixers,

Apologies for the long email, so let me start with the punchline: attached is a diff which adds an '--fhs-container' (or -F) option to guix shell/environment to set up an FHS-like container. This includes the usual /lib directory and a glibc which loads (a generated in the container) /etc/ld.so.cache. This should allow running most things that expect a more "typical" Linux environment. Give it a try!

First, I wanted to ask how people feel about such a feature. Obviously, one use is to run pre-built binaries (isolated!), but this is also handy for setting up development environments when not able (or wanting) to with Guix packages only. For example, using the rustup [0] scripts, pretty much anything JS, or just following typical Readme instructions to try out something new before packaging. I won't debate the details here other than to say this topic comes up with Guix and I think it is yet another useful tool for guix shell and containers.

Nix, which I know almost nothing about, does have some FHS container/environment options as well.

Next, some technical points about implementation, which I hope will be informed by the first question and what we want from this tool. There are two main things needed for the FHS-container:

1. Set up directories like /lib. This is easy enough and can be done currently, like in roptat's response here [1] by building the profile first to know where to link to. Note that it is easier to do it within the environment code since we have access to the profile even if it is being built for the first time. There are some wrinkles with linking something like /bin since we currently add a link for sh; see the comments in my diff.

Right now I did not handle a multi-arch setup, though that shouldn't be too difficult. This would probably require an option to build either all or specified packages for an additional arch, like 32bit in a 64bit system, and make the libraries available (/lib32 or something). Though may run into a union-build bug [2]?

2. Typically binaries will expect the ld loader to use /etc/ld.so.cache for finding libraries. So, I made a glibc package that removes our dl cache patch to restore this behavior. It seems enough to add this as a default package to the container, though I commented out an option to automatically graft everything with this glibc. Both worked for me, but grafting didn't seem necessary.

The second step is to generate the ld cache, which is done with a simple startup script in the container, after creating a temporary ld.so.conf (our ldconfig doesn't use the usual default directories?). I'm sure I found the hackiest way to do this, but it works :) Again, this could be possible without modifying guix containers, but this is easier. (For example, you can see work done by myself and others at a certain non-free channel to do exactly this.)

Some questions going forward, besides overall cleanup and tweaking of the code (which I provided comments in for some details, please see there). It might be nice to be able to extend containers more easily with setup scripts, though again this can be done by making some Guile scripts to wrap a container and making a package around that (e.g. from the non-free channel). What kind of extensions would be useful to expose? I think I saw some talk on IRC recently about how to manage env variables when using guix shell. Perhaps an extended manifest format for shell?

Relatedly and more generally, perhaps it would be good to have somewhere (cookbook?) some recipes of useful guix shell container incantations. Sharing what you need from the host for graphical programs can be a little tricky to figure out. We have the --network option, maybe others would be useful? Or some base package lists for development: just like we have our various -build-system's, a package list with typical library sets might be a nice convenience.

What about other uses for this container, like providing an isolated environment to build and run software we can't do fully with bootstrap and sources (like JS)? Could this become some stop-gap to allow people to work with these ecosystems in a more controlled way within Guix? Or an alternative build environment? Not entirely sure what this could mean, just thinking out loud.

Okay, let me end by bringing it back to what we can currently do with this code I whipped up (and many thanks to [1] and efforts in a non-free channel for doing the work that I drew upon).

I don't know any Rust, so I figured trying out what I read on the internet about "just use rustup" and follow a readme is a good test case. So I did that and compiled something that looked hefty, a graphical widget system [3]. Here's the command I used and everything just worked: the rustup script ran and downloaded the rust tools, the cargo build command worked to build everything. I couldn't run the actual widgets as I am within a pure shell for my guix checkout (but have done this with similar code and it fully worked once sharing the right host env variables/directories). I used a directory as my container home to keep everything.

./pre-inst-env guix shell --network --fhs-container bash coreutils curl grep nss-certs gcc:lib gcc-toolchain pkg-config glib cairo atk pango@1.48.10 gdk-pixbuf gtk+ --share=$HOME/temp/fhs-home=$HOME

On IRC, apteryx mentioned wanting to try buildroot [4] as a use-case. This worked for me after setting TERM=xterm in the container (I think I run the shell not quite correctly right now, or because it is already with in a guix shell --pure). Unfortunately make failed with some broken URLs, but so far didn't need any tweaks.

./pre-inst-env guix shell --network --fhs-container bash coreutils grep make nss-certs diffutils findutils ncurses which sed unzip gzip bzip2 wget perl cpio bc patch rsync tar file python gawk util-linux-with-udev gcc:lib pkg-config gcc-toolchain --share=$HOME/temp/fhs-home=$HOME

So, things should work as long as you provide the necessary packages. There may be some tweaks needed with some symlinks or env variables, but I think the basics are there.

Happy to hear everyone's thoughts! And sorry for the long email, but the code is pretty short and straightforward.

John

PS: Is there an easy way to use the modified environment scripts outside of a guix checkout shell?

[0] https://rustup.rs/

[1] https://unix.stackexchange.com/questions/600311/how-to-run-a-dynamically-compiled-32-bit-x86-binary-on-a-amd64-bit-guix-system

[2] https://issues.guix.gnu.org/53406

[3] https://github.com/elkowar/eww

[4] https://git.buildroot.net/buildroot/

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: fhs-container.diff --]
[-- Type: text/x-patch; name=fhs-container.diff, Size: 10408 bytes --]

diff --git a/gnu/packages/base.scm b/gnu/packages/base.scm
index 4bdc3e7792..1b4c99d3e9 100644
--- a/gnu/packages/base.scm
+++ b/gnu/packages/base.scm
@@ -928,6 +928,20 @@ (define-public glibc
    (license lgpl2.0+)
    (home-page "https://www.gnu.org/software/libc/")))
 
+;; Define glibc-for-fhs (with a name that allows grafts for glibc), a variation
+;; of glibc which uses the default ld.so.cache, useful in FHS containers.
+;; Note: should this be hidden?
+(define-public gcfhs
+  (package
+    (inherit glibc)
+    (name "gcfhs")
+    (source (origin (inherit (package-source glibc))
+                    ;; Remove Guix's patch to read ld.so.cache from /gnu/store
+                    ;; directories, re-enabling the default /etc/ld.so.cache
+                    ;; behavior.
+                    (patches (delete (car (search-patches "glibc-dl-cache.patch"))
+                                     (origin-patches (package-source glibc))))))))
+
 ;; Below are old libc versions, which we use mostly to build locale data in
 ;; the old format (which the new libc cannot cope with.)
 (define-public glibc-2.32
diff --git a/guix/scripts/environment.scm b/guix/scripts/environment.scm
index 3216235937..425649b843 100644
--- a/guix/scripts/environment.scm
+++ b/guix/scripts/environment.scm
@@ -2,6 +2,7 @@
 ;;; Copyright © 2014, 2015, 2018 David Thompson <davet@gnu.org>
 ;;; Copyright © 2015-2022 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2018 Mike Gerwitz <mtg@gnu.org>
+;;; Copyright © 2021 John Kehayias <john.kehayias@protonmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -45,6 +46,7 @@ (define-module (guix scripts environment)
   #:autoload   (guix build syscalls) (set-network-interface-up openpty login-tty)
   #:use-module (gnu system file-systems)
   #:autoload   (gnu packages) (specification->package+output)
+  #:autoload   (gnu packages base) (gcfhs)
   #:autoload   (gnu packages bash) (bash)
   #:autoload   (gnu packages bootstrap) (bootstrap-executable %bootstrap-guile)
   #:use-module (ice-9 match)
@@ -101,6 +103,8 @@ (define (show-environment-options-help)
   (display (G_ "
   -C, --container        run command within an isolated container"))
   (display (G_ "
+  -F, --fhs-container    run command within an isolated FHS container"))
+  (display (G_ "
   -N, --network          allow containers to access the network"))
   (display (G_ "
   -P, --link-profile     link environment profile to ~/.guix-profile within
@@ -229,6 +233,10 @@ (define %options
          (option '(#\C "container") #f #f
                  (lambda (opt name arg result)
                    (alist-cons 'container? #t result)))
+         (option '(#\F "fhs-container") #f #f
+                 (lambda (opt name arg result)
+                   (alist-cons 'fhs-container? #t
+                               (alist-cons 'container? #t result))))
          (option '(#\N "network") #f #f
                  (lambda (opt name arg result)
                    (alist-cons 'network? #t result)))
@@ -606,9 +614,10 @@ (define* (launch-environment/fork command profile manifest
            ((_ . status)
             (validate-exit-status profile command status))))))
 
-(define* (launch-environment/container #:key command bash user user-mappings
-                                       profile manifest link-profile? network?
-                                       map-cwd? (white-list '()))
+(define* (launch-environment/container #:key command bash fhs-container? user
+                                       user-mappings profile manifest
+                                       link-profile? network? map-cwd?
+                                       (white-list '()))
   "Run COMMAND within a container that features the software in PROFILE.
 Environment variables are set according to the search paths of MANIFEST.
 The global shell is BASH, a file name for a GNU Bash binary in the
@@ -709,6 +718,49 @@ (define* (launch-environment/container #:key command bash user user-mappings
             (mkdir-p home-dir)
             (setenv "HOME" home-dir)
 
+            ;; Set up an FHS container.
+            (when fhs-container?
+              ;; Set up the expected bin and library directories as symlinks to
+              ;; the profile lib directory.  Note that this is assuming a 64bit
+              ;; architecture.
+              (let ((lib-dir (string-append profile "/lib")))
+                (symlink lib-dir "/lib64")
+                (symlink lib-dir "/lib")
+                (mkdir-p "/usr")
+                (symlink lib-dir "/usr/lib"))
+              ;; Note: can't symlink full /bin in the container due to the sh
+              ;; symlink.
+              (symlink (string-append profile "/bin") "/usr/bin")
+              (symlink (string-append profile "/sbin") "/sbin")
+              (symlink (string-append profile "/sbin") "/usr/sbin")
+
+              ;; Provide a frequently expected 'cc' symlink to gcc, though this
+              ;; could also be done by the user in the container, e.g. in
+              ;; $HOME/.local/bin and adding that to $PATH.  Note: we do this
+              ;; in /bin since that already has the sh symlink and can't write
+              ;; to the other bin directories that are already symlinks themselves.
+              (symlink (string-append profile "/bin/gcc") "/bin/cc")
+              ;; TODO: python may also be expected to symlink to python3.
+
+              ;; Guix's ldconfig doesn't seem to search in FHS default
+              ;; locations, so provide a minimal ld.so.conf.
+              ;; TODO: this may need more, e.g. libnss3 is in /lib/nss
+              (call-with-output-file "/tmp/ld.so.conf"
+                (lambda (port)
+                  (display "/lib64" port)
+                  (newline port)))
+
+              ;; Define an entry script to start the container: generate
+              ;; ld.so.cache, supplement $PATH, and include command.
+              (call-with-output-file "/tmp/fhs.sh"
+                (lambda (port)
+                  (display "ldconfig -X -f /tmp/ld.so.conf" port)
+                  (newline port)
+                  (display "export PATH=/bin:/usr/bin:/sbin:/usr/sbin:$PATH" port)
+                  (newline port)
+                  (display (car command) port)
+                  (newline port))))
+
             ;; If requested, link $GUIX_ENVIRONMENT to $HOME/.guix-profile;
             ;; this allows programs expecting that path to continue working as
             ;; expected within a container.
@@ -746,7 +798,12 @@ (define* (launch-environment/container #:key command bash user user-mappings
             (primitive-exit/status
              ;; A container's environment is already purified, so no need to
              ;; request it be purified again.
-             (launch-environment command
+             (launch-environment (if fhs-container?
+                                     ;; Use the FHS start script.
+                                     ;; FIXME: probably the default command should
+                                     ;; be different as it spawns a different shell?
+                                     '("/bin/sh" "/tmp/fhs.sh")
+                                     command)
                                  (if link-profile?
                                      (string-append home-dir "/.guix-profile")
                                      profile)
@@ -874,15 +931,16 @@ (define (guix-environment* opts)
   "Run the 'guix environment' command on OPTS, an alist resulting for
 command-line option processing with 'parse-command-line'."
   (with-error-handling
-    (let* ((pure?      (assoc-ref opts 'pure))
-           (container? (assoc-ref opts 'container?))
-           (link-prof? (assoc-ref opts 'link-profile?))
-           (network?   (assoc-ref opts 'network?))
-           (no-cwd?    (assoc-ref opts 'no-cwd?))
-           (user       (assoc-ref opts 'user))
-           (bootstrap? (assoc-ref opts 'bootstrap?))
-           (system     (assoc-ref opts 'system))
-           (profile    (assoc-ref opts 'profile))
+    (let* ((pure?          (assoc-ref opts 'pure))
+           (container?     (assoc-ref opts 'container?))
+           (fhs-container? (assoc-ref opts 'fhs-container?))
+           (link-prof?     (assoc-ref opts 'link-profile?))
+           (network?       (assoc-ref opts 'network?))
+           (no-cwd?        (assoc-ref opts 'no-cwd?))
+           (user           (assoc-ref opts 'user))
+           (bootstrap?     (assoc-ref opts 'bootstrap?))
+           (system         (assoc-ref opts 'system))
+           (profile        (assoc-ref opts 'profile))
            (command    (or (assoc-ref opts 'exec)
                            ;; Spawn a shell if the user didn't specify
                            ;; anything in particular.
@@ -927,7 +985,16 @@ (define (guix-environment* opts)
       (with-store/maybe store
         (with-status-verbosity (assoc-ref opts 'verbosity)
           (define manifest-from-opts
-            (options/resolve-packages store opts))
+            (options/resolve-packages store
+                                      ;; For an FHS-container, add a glibc that uses
+                                      ;; /etc/ld.so.cache.
+                                      (if fhs-container?
+                                          (alist-cons 'package '(ad-hoc-package "gcfhs")
+                                                      opts)
+                                          ;; Alternatively, could graft all packages with
+                                          ;; this glibc, though that seems unnecessary.
+                                          ;; (alist-cons 'with-graft "glibc=gcfhs" opts)
+                                          opts)))
 
           (define manifest
             (if profile
@@ -994,6 +1061,7 @@ (define (guix-environment* opts)
                                               "/bin/sh"))))
                       (launch-environment/container #:command command
                                                     #:bash bash-binary
+                                                    #:fhs-container? fhs-container?
                                                     #:user user
                                                     #:user-mappings mappings
                                                     #:profile profile