[WIP Patch] Adding an FHS container to guix shell

[WIP Patch] Adding an FHS container to guix shell

[John Kehayias]

[-- Attachment #1: Type: text/plain, Size: 6638 bytes --]

Hi Guixers,

Apologies for the long email, so let me start with the punchline: attached is a diff which adds an '--fhs-container' (or -F) option to guix shell/environment to set up an FHS-like container. This includes the usual /lib directory and a glibc which loads (a generated in the container) /etc/ld.so.cache. This should allow running most things that expect a more "typical" Linux environment. Give it a try!

First, I wanted to ask how people feel about such a feature. Obviously, one use is to run pre-built binaries (isolated!), but this is also handy for setting up development environments when not able (or wanting) to with Guix packages only. For example, using the rustup [0] scripts, pretty much anything JS, or just following typical Readme instructions to try out something new before packaging. I won't debate the details here other than to say this topic comes up with Guix and I think it is yet another useful tool for guix shell and containers.

Nix, which I know almost nothing about, does have some FHS container/environment options as well.

Next, some technical points about implementation, which I hope will be informed by the first question and what we want from this tool. There are two main things needed for the FHS-container:

1. Set up directories like /lib. This is easy enough and can be done currently, like in roptat's response here [1] by building the profile first to know where to link to. Note that it is easier to do it within the environment code since we have access to the profile even if it is being built for the first time. There are some wrinkles with linking something like /bin since we currently add a link for sh; see the comments in my diff.

Right now I did not handle a multi-arch setup, though that shouldn't be too difficult. This would probably require an option to build either all or specified packages for an additional arch, like 32bit in a 64bit system, and make the libraries available (/lib32 or something). Though may run into a union-build bug [2]?

2. Typically binaries will expect the ld loader to use /etc/ld.so.cache for finding libraries. So, I made a glibc package that removes our dl cache patch to restore this behavior. It seems enough to add this as a default package to the container, though I commented out an option to automatically graft everything with this glibc. Both worked for me, but grafting didn't seem necessary.

The second step is to generate the ld cache, which is done with a simple startup script in the container, after creating a temporary ld.so.conf (our ldconfig doesn't use the usual default directories?). I'm sure I found the hackiest way to do this, but it works :) Again, this could be possible without modifying guix containers, but this is easier. (For example, you can see work done by myself and others at a certain non-free channel to do exactly this.)

Some questions going forward, besides overall cleanup and tweaking of the code (which I provided comments in for some details, please see there). It might be nice to be able to extend containers more easily with setup scripts, though again this can be done by making some Guile scripts to wrap a container and making a package around that (e.g. from the non-free channel). What kind of extensions would be useful to expose? I think I saw some talk on IRC recently about how to manage env variables when using guix shell. Perhaps an extended manifest format for shell?

Relatedly and more generally, perhaps it would be good to have somewhere (cookbook?) some recipes of useful guix shell container incantations. Sharing what you need from the host for graphical programs can be a little tricky to figure out. We have the --network option, maybe others would be useful? Or some base package lists for development: just like we have our various -build-system's, a package list with typical library sets might be a nice convenience.

What about other uses for this container, like providing an isolated environment to build and run software we can't do fully with bootstrap and sources (like JS)? Could this become some stop-gap to allow people to work with these ecosystems in a more controlled way within Guix? Or an alternative build environment? Not entirely sure what this could mean, just thinking out loud.

Okay, let me end by bringing it back to what we can currently do with this code I whipped up (and many thanks to [1] and efforts in a non-free channel for doing the work that I drew upon).

I don't know any Rust, so I figured trying out what I read on the internet about "just use rustup" and follow a readme is a good test case. So I did that and compiled something that looked hefty, a graphical widget system [3]. Here's the command I used and everything just worked: the rustup script ran and downloaded the rust tools, the cargo build command worked to build everything. I couldn't run the actual widgets as I am within a pure shell for my guix checkout (but have done this with similar code and it fully worked once sharing the right host env variables/directories). I used a directory as my container home to keep everything.

./pre-inst-env guix shell --network --fhs-container bash coreutils curl grep nss-certs gcc:lib gcc-toolchain pkg-config glib cairo atk pango@1.48.10 gdk-pixbuf gtk+ --share=$HOME/temp/fhs-home=$HOME

On IRC, apteryx mentioned wanting to try buildroot [4] as a use-case. This worked for me after setting TERM=xterm in the container (I think I run the shell not quite correctly right now, or because it is already with in a guix shell --pure). Unfortunately make failed with some broken URLs, but so far didn't need any tweaks.

./pre-inst-env guix shell --network --fhs-container bash coreutils grep make nss-certs diffutils findutils ncurses which sed unzip gzip bzip2 wget perl cpio bc patch rsync tar file python gawk util-linux-with-udev gcc:lib pkg-config gcc-toolchain --share=$HOME/temp/fhs-home=$HOME

So, things should work as long as you provide the necessary packages. There may be some tweaks needed with some symlinks or env variables, but I think the basics are there.

Happy to hear everyone's thoughts! And sorry for the long email, but the code is pretty short and straightforward.

John

PS: Is there an easy way to use the modified environment scripts outside of a guix checkout shell?

[0] https://rustup.rs/

[1] https://unix.stackexchange.com/questions/600311/how-to-run-a-dynamically-compiled-32-bit-x86-binary-on-a-amd64-bit-guix-system

[2] https://issues.guix.gnu.org/53406

[3] https://github.com/elkowar/eww

[4] https://git.buildroot.net/buildroot/

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: fhs-container.diff --]
[-- Type: text/x-patch; name=fhs-container.diff, Size: 10408 bytes --]

diff --git a/gnu/packages/base.scm b/gnu/packages/base.scm
index 4bdc3e7792..1b4c99d3e9 100644
--- a/gnu/packages/base.scm
+++ b/gnu/packages/base.scm
@@ -928,6 +928,20 @@ (define-public glibc
    (license lgpl2.0+)
    (home-page "https://www.gnu.org/software/libc/")))
 
+;; Define glibc-for-fhs (with a name that allows grafts for glibc), a variation
+;; of glibc which uses the default ld.so.cache, useful in FHS containers.
+;; Note: should this be hidden?
+(define-public gcfhs
+  (package
+    (inherit glibc)
+    (name "gcfhs")
+    (source (origin (inherit (package-source glibc))
+                    ;; Remove Guix's patch to read ld.so.cache from /gnu/store
+                    ;; directories, re-enabling the default /etc/ld.so.cache
+                    ;; behavior.
+                    (patches (delete (car (search-patches "glibc-dl-cache.patch"))
+                                     (origin-patches (package-source glibc))))))))
+
 ;; Below are old libc versions, which we use mostly to build locale data in
 ;; the old format (which the new libc cannot cope with.)
 (define-public glibc-2.32
diff --git a/guix/scripts/environment.scm b/guix/scripts/environment.scm
index 3216235937..425649b843 100644
--- a/guix/scripts/environment.scm
+++ b/guix/scripts/environment.scm
@@ -2,6 +2,7 @@
 ;;; Copyright © 2014, 2015, 2018 David Thompson <davet@gnu.org>
 ;;; Copyright © 2015-2022 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2018 Mike Gerwitz <mtg@gnu.org>
+;;; Copyright © 2021 John Kehayias <john.kehayias@protonmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -45,6 +46,7 @@ (define-module (guix scripts environment)
   #:autoload   (guix build syscalls) (set-network-interface-up openpty login-tty)
   #:use-module (gnu system file-systems)
   #:autoload   (gnu packages) (specification->package+output)
+  #:autoload   (gnu packages base) (gcfhs)
   #:autoload   (gnu packages bash) (bash)
   #:autoload   (gnu packages bootstrap) (bootstrap-executable %bootstrap-guile)
   #:use-module (ice-9 match)
@@ -101,6 +103,8 @@ (define (show-environment-options-help)
   (display (G_ "
   -C, --container        run command within an isolated container"))
   (display (G_ "
+  -F, --fhs-container    run command within an isolated FHS container"))
+  (display (G_ "
   -N, --network          allow containers to access the network"))
   (display (G_ "
   -P, --link-profile     link environment profile to ~/.guix-profile within
@@ -229,6 +233,10 @@ (define %options
          (option '(#\C "container") #f #f
                  (lambda (opt name arg result)
                    (alist-cons 'container? #t result)))
+         (option '(#\F "fhs-container") #f #f
+                 (lambda (opt name arg result)
+                   (alist-cons 'fhs-container? #t
+                               (alist-cons 'container? #t result))))
          (option '(#\N "network") #f #f
                  (lambda (opt name arg result)
                    (alist-cons 'network? #t result)))
@@ -606,9 +614,10 @@ (define* (launch-environment/fork command profile manifest
            ((_ . status)
             (validate-exit-status profile command status))))))
 
-(define* (launch-environment/container #:key command bash user user-mappings
-                                       profile manifest link-profile? network?
-                                       map-cwd? (white-list '()))
+(define* (launch-environment/container #:key command bash fhs-container? user
+                                       user-mappings profile manifest
+                                       link-profile? network? map-cwd?
+                                       (white-list '()))
   "Run COMMAND within a container that features the software in PROFILE.
 Environment variables are set according to the search paths of MANIFEST.
 The global shell is BASH, a file name for a GNU Bash binary in the
@@ -709,6 +718,49 @@ (define* (launch-environment/container #:key command bash user user-mappings
             (mkdir-p home-dir)
             (setenv "HOME" home-dir)
 
+            ;; Set up an FHS container.
+            (when fhs-container?
+              ;; Set up the expected bin and library directories as symlinks to
+              ;; the profile lib directory.  Note that this is assuming a 64bit
+              ;; architecture.
+              (let ((lib-dir (string-append profile "/lib")))
+                (symlink lib-dir "/lib64")
+                (symlink lib-dir "/lib")
+                (mkdir-p "/usr")
+                (symlink lib-dir "/usr/lib"))
+              ;; Note: can't symlink full /bin in the container due to the sh
+              ;; symlink.
+              (symlink (string-append profile "/bin") "/usr/bin")
+              (symlink (string-append profile "/sbin") "/sbin")
+              (symlink (string-append profile "/sbin") "/usr/sbin")
+
+              ;; Provide a frequently expected 'cc' symlink to gcc, though this
+              ;; could also be done by the user in the container, e.g. in
+              ;; $HOME/.local/bin and adding that to $PATH.  Note: we do this
+              ;; in /bin since that already has the sh symlink and can't write
+              ;; to the other bin directories that are already symlinks themselves.
+              (symlink (string-append profile "/bin/gcc") "/bin/cc")
+              ;; TODO: python may also be expected to symlink to python3.
+
+              ;; Guix's ldconfig doesn't seem to search in FHS default
+              ;; locations, so provide a minimal ld.so.conf.
+              ;; TODO: this may need more, e.g. libnss3 is in /lib/nss
+              (call-with-output-file "/tmp/ld.so.conf"
+                (lambda (port)
+                  (display "/lib64" port)
+                  (newline port)))
+
+              ;; Define an entry script to start the container: generate
+              ;; ld.so.cache, supplement $PATH, and include command.
+              (call-with-output-file "/tmp/fhs.sh"
+                (lambda (port)
+                  (display "ldconfig -X -f /tmp/ld.so.conf" port)
+                  (newline port)
+                  (display "export PATH=/bin:/usr/bin:/sbin:/usr/sbin:$PATH" port)
+                  (newline port)
+                  (display (car command) port)
+                  (newline port))))
+
             ;; If requested, link $GUIX_ENVIRONMENT to $HOME/.guix-profile;
             ;; this allows programs expecting that path to continue working as
             ;; expected within a container.
@@ -746,7 +798,12 @@ (define* (launch-environment/container #:key command bash user user-mappings
             (primitive-exit/status
              ;; A container's environment is already purified, so no need to
              ;; request it be purified again.
-             (launch-environment command
+             (launch-environment (if fhs-container?
+                                     ;; Use the FHS start script.
+                                     ;; FIXME: probably the default command should
+                                     ;; be different as it spawns a different shell?
+                                     '("/bin/sh" "/tmp/fhs.sh")
+                                     command)
                                  (if link-profile?
                                      (string-append home-dir "/.guix-profile")
                                      profile)
@@ -874,15 +931,16 @@ (define (guix-environment* opts)
   "Run the 'guix environment' command on OPTS, an alist resulting for
 command-line option processing with 'parse-command-line'."
   (with-error-handling
-    (let* ((pure?      (assoc-ref opts 'pure))
-           (container? (assoc-ref opts 'container?))
-           (link-prof? (assoc-ref opts 'link-profile?))
-           (network?   (assoc-ref opts 'network?))
-           (no-cwd?    (assoc-ref opts 'no-cwd?))
-           (user       (assoc-ref opts 'user))
-           (bootstrap? (assoc-ref opts 'bootstrap?))
-           (system     (assoc-ref opts 'system))
-           (profile    (assoc-ref opts 'profile))
+    (let* ((pure?          (assoc-ref opts 'pure))
+           (container?     (assoc-ref opts 'container?))
+           (fhs-container? (assoc-ref opts 'fhs-container?))
+           (link-prof?     (assoc-ref opts 'link-profile?))
+           (network?       (assoc-ref opts 'network?))
+           (no-cwd?        (assoc-ref opts 'no-cwd?))
+           (user           (assoc-ref opts 'user))
+           (bootstrap?     (assoc-ref opts 'bootstrap?))
+           (system         (assoc-ref opts 'system))
+           (profile        (assoc-ref opts 'profile))
            (command    (or (assoc-ref opts 'exec)
                            ;; Spawn a shell if the user didn't specify
                            ;; anything in particular.
@@ -927,7 +985,16 @@ (define (guix-environment* opts)
       (with-store/maybe store
         (with-status-verbosity (assoc-ref opts 'verbosity)
           (define manifest-from-opts
-            (options/resolve-packages store opts))
+            (options/resolve-packages store
+                                      ;; For an FHS-container, add a glibc that uses
+                                      ;; /etc/ld.so.cache.
+                                      (if fhs-container?
+                                          (alist-cons 'package '(ad-hoc-package "gcfhs")
+                                                      opts)
+                                          ;; Alternatively, could graft all packages with
+                                          ;; this glibc, though that seems unnecessary.
+                                          ;; (alist-cons 'with-graft "glibc=gcfhs" opts)
+                                          opts)))
 
           (define manifest
             (if profile
@@ -994,6 +1061,7 @@ (define (guix-environment* opts)
                                               "/bin/sh"))))
                       (launch-environment/container #:command command
                                                     #:bash bash-binary
+                                                    #:fhs-container? fhs-container?
                                                     #:user user
                                                     #:user-mappings mappings
                                                     #:profile profile

Re: [WIP Patch] Adding an FHS container to guix shell

[Vagrant Cascadian]

[-- Attachment #1: Type: text/plain, Size: 1559 bytes --]

On 2022-07-12, John Kehayias wrote:
> Apologies for the long email, so let me start with the punchline:
> attached is a diff which adds an '--fhs-container' (or -F) option to
> guix shell/environment to set up an FHS-like container. This includes
> the usual /lib directory and a glibc which loads (a generated in the
> container) /etc/ld.so.cache. This should allow running most things
> that expect a more "typical" Linux environment. Give it a try!

Nice!


> 1. Set up directories like /lib. This is easy enough and can be done
> currently, like in roptat's response here [1] by building the profile
> first to know where to link to. Note that it is easier to do it within
> the environment code since we have access to the profile even if it is
> being built for the first time. There are some wrinkles with linking
> something like /bin since we currently add a link for sh; see the
> comments in my diff.
>
> Right now I did not handle a multi-arch setup, though that shouldn't
> be too difficult. This would probably require an option to build
> either all or specified packages for an additional arch, like 32bit in
> a 64bit system, and make the libraries available (/lib32 or
> something). Though may run into a union-build bug [2]?

This might be splitting hairs, but that sounds like a bi-arch setup
e.g. "/lib and /lib32" vs. a multi-arch setup "/lib,
/lib/aarch64-linux-gnu/ and /lib/i386-linux-gnu"

  https://wiki.debian.org/Multiarch

Not sure how many extra hoops you'd need to jump through to make either
work well.


live well,
  vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: [WIP Patch] Adding an FHS container to guix shell

[John Kehayias]

Hi Vagrant

------- Original Message -------
On Tuesday, July 12th, 2022 at 1:34 PM, Vagrant Cascadian wrote:

>
>
> On 2022-07-12, John Kehayias wrote:
>
> > Right now I did not handle a multi-arch setup, though that shouldn't
> > be too difficult. This would probably require an option to build
> > either all or specified packages for an additional arch, like 32bit in
> > a 64bit system, and make the libraries available (/lib32 or
> > something). Though may run into a union-build bug [2]?
>
>
> This might be splitting hairs, but that sounds like a bi-arch setup
> e.g. "/lib and /lib32" vs. a multi-arch setup "/lib,
> /lib/aarch64-linux-gnu/ and /lib/i386-linux-gnu"
>
> https://wiki.debian.org/Multiarch
>
> Not sure how many extra hoops you'd need to jump through to make either
> work well.
>

Thanks, was not aware of the different terminology. The use-case I'm thinking of mostly is for when one needs 32bit libraries in a 64bit x86 system, often for older programs (or maybe Wine).

In any event, I think for my first goals I'll leave that out and we can think about extensions for such a setup later. I don't think it would be very difficult, at least to build some libraries for a different system and link in a separate lib directory.

John

Re: [WIP Patch] Adding an FHS container to guix shell

[Dominic Martinez]

[-- Attachment #1: Type: text/plain, Size: 1750 bytes --]


John Kehayias <john.kehayias@protonmail.com> writes:

> First, I wanted to ask how people feel about such a feature. Obviously, one use
> is to run pre-built binaries (isolated!), but this is also handy for setting up
> development environments when not able (or wanting) to with Guix packages only.
> For example, using the rustup [0] scripts, pretty much anything JS, or just
> following typical Readme instructions to try out something new before packaging.
> I won't debate the details here other than to say this topic comes up with Guix
> and I think it is yet another useful tool for guix shell and containers.

Absolutely! I usually have to resort to Docker containers when building
something that doesn't support GuixSD, so being able to work with them
through Guix would be amazing.

> What about other uses for this container, like providing an isolated environment
> to build and run software we can't do fully with bootstrap and sources (like
> JS)? Could this become some stop-gap to allow people to work with these
> ecosystems in a more controlled way within Guix? Or an alternative build
> environment? Not entirely sure what this could mean, just thinking out loud.

I think an interesting idea would be to allow packages to transparently
run in the FHS container (i.e. a shim that turns 'x' into 'guix shell
--fhs-container x -- x'). That way software incompatible with GuixSD in
a way too difficult to patch could be still be packaged and used
transparently, albeit with a significant performance cost.

Even if packages in Guix proper don't use it, it could be useful for
third-party channels or end-users to whip up packages.

Thanks so much for this; I've been thinking about getting around to this
feature for quite a while.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: [WIP Patch] Adding an FHS container to guix shell

[Blake Shaw]

[-- Attachment #1: Type: text/plain, Size: 2041 bytes --]

hi John!

I think it sounds like a swell addition to Guix, something that I would
definitely reach for time to time.

good stuff.

ez,
b

On Wed, Jul 13, 2022 at 2:26 AM Dominic Martinez <dom@dominicm.dev> wrote:

>
> John Kehayias <john.kehayias@protonmail.com> writes:
>
> > First, I wanted to ask how people feel about such a feature. Obviously,
> one use
> > is to run pre-built binaries (isolated!), but this is also handy for
> setting up
> > development environments when not able (or wanting) to with Guix
> packages only.
> > For example, using the rustup [0] scripts, pretty much anything JS, or
> just
> > following typical Readme instructions to try out something new before
> packaging.
> > I won't debate the details here other than to say this topic comes up
> with Guix
> > and I think it is yet another useful tool for guix shell and containers.
>
> Absolutely! I usually have to resort to Docker containers when building
> something that doesn't support GuixSD, so being able to work with them
> through Guix would be amazing.
>
> > What about other uses for this container, like providing an isolated
> environment
> > to build and run software we can't do fully with bootstrap and sources
> (like
> > JS)? Could this become some stop-gap to allow people to work with these
> > ecosystems in a more controlled way within Guix? Or an alternative build
> > environment? Not entirely sure what this could mean, just thinking out
> loud.
>
> I think an interesting idea would be to allow packages to transparently
> run in the FHS container (i.e. a shim that turns 'x' into 'guix shell
> --fhs-container x -- x'). That way software incompatible with GuixSD in
> a way too difficult to patch could be still be packaged and used
> transparently, albeit with a significant performance cost.
>
> Even if packages in Guix proper don't use it, it could be useful for
> third-party channels or end-users to whip up packages.
>
> Thanks so much for this; I've been thinking about getting around to this
> feature for quite a while.
>

[-- Attachment #2: Type: text/html, Size: 2641 bytes --]

Re: [WIP Patch] Adding an FHS container to guix shell

[John Kehayias]

Hi Dominic,

------- Original Message -------
On Tuesday, July 12th, 2022 at 10:11 PM, Dominic Martinez wrote:

>
>
>
> John Kehayias john.kehayias@protonmail.com writes:
>
> > What about other uses for this container, like providing an isolated environment
> > to build and run software we can't do fully with bootstrap and sources (like
> > JS)? Could this become some stop-gap to allow people to work with these
> > ecosystems in a more controlled way within Guix? Or an alternative build
> > environment? Not entirely sure what this could mean, just thinking out loud.
>
>
> I think an interesting idea would be to allow packages to transparently
> run in the FHS container (i.e. a shim that turns 'x' into 'guix shell
> --fhs-container x -- x'). That way software incompatible with GuixSD in
> a way too difficult to patch could be still be packaged and used
> transparently, albeit with a significant performance cost.
>
> Even if packages in Guix proper don't use it, it could be useful for
> third-party channels or end-users to whip up packages.
>

Yes, this is something I was thinking of, as a partial stop-gap for when you need to use software that can't be fully packaged the way we want for Guix. Not sure what the details would look like exactly (internally), but could be a useful tool to have.

> Thanks so much for this; I've been thinking about getting around to this
> feature for quite a while.

You are welcome and thanks for the kind words! I've also been thinking about this for a while, and luckily a lot of this was laid out in what others have done (and myself for non-free packaging).

I'm already using a variation of this to do some development work that otherwise wouldn't be possible without maybe a VM or using another machine.

John

Re: [WIP Patch] Adding an FHS container to guix shell

[zimoun]

Hi

On Tue, 12 Jul 2022 at 15:59, John Kehayias <john.kehayias@protonmail.com> wrote:

> Apologies for the long email, so let me start with the punchline:
> attached is a diff which adds an '--fhs-container' (or -F) option to
> guix shell/environment to set up an FHS-like container. This includes
> the usual /lib directory and a glibc which loads (a generated in the
> container) /etc/ld.so.cache. This should allow running most things
> that expect a more "typical" Linux environment. Give it a try! 

Cool!  Well, I am not sure if this could help with the long time ago
proposed “guix run” [1].


1: <https://yhetil.org/guix/87zi51r3cg.fsf@gnu.org>


Cheers,
simon

Re: [WIP Patch] Adding an FHS container to guix shell

[John Kehayias]

Hi simon,


------- Original Message -------
On Thursday, July 14th, 2022 at 6:01 AM, zimoun wrote:

>
>
> Hi
>
> On Tue, 12 Jul 2022 at 15:59, John Kehayias wrote:
>
> > Apologies for the long email, so let me start with the punchline:
> > attached is a diff which adds an '--fhs-container' (or -F) option to
> > guix shell/environment to set up an FHS-like container. This includes
> > the usual /lib directory and a glibc which loads (a generated in the
> > container) /etc/ld.so.cache. This should allow running most things
> > that expect a more "typical" Linux environment. Give it a try!
>
>
> Cool! Well, I am not sure if this could help with the long time ago
> proposed “guix run” [1].
>
>
> 1: https://yhetil.org/guix/87zi51r3cg.fsf@gnu.org
>
>

Oh, I do remember seeing that mentioned once. From my quick read sounds similar to some of what we have for guix shell, with some thoughts on how to set up the environment better. Something to look into, thanks!

John

Re: [WIP Patch] Adding an FHS container to guix shell

[Liliana Marie Prikler]

Hi John,

Am Dienstag, dem 12.07.2022 um 15:59 +0000 schrieb John Kehayias:
> Hi Guixers,
> 
> Apologies for the long email, so let me start with the punchline:
> attached is a diff which adds an '--fhs-container' (or -F) option to
> guix shell/environment to set up an FHS-like container. This includes
> the usual /lib directory and a glibc which loads (a generated in the
> container) /etc/ld.so.cache. This should allow running most things
> that expect a more "typical" Linux environment. Give it a try!
> 
> First, I wanted to ask how people feel about such a feature.
> Obviously, one use is to run pre-built binaries (isolated!), but this
> is also handy for setting up development environments when not able
> (or wanting) to with Guix packages only. For example, using the
> rustup [0] scripts, pretty much anything JS, or just following
> typical Readme instructions to try out something new before
> packaging. I won't debate the details here other than to say this
> topic comes up with Guix and I think it is yet another useful tool
> for guix shell and containers.
> 
> Nix, which I know almost nothing about, does have some FHS
> container/environment options as well.
> 
> Next, some technical points about implementation, which I hope will
> be informed by the first question and what we want from this tool.
> There are two main things needed for the FHS-container:
> 
> 1. Set up directories like /lib. This is easy enough and can be done
> currently, like in roptat's response here [1] by building the profile
> first to know where to link to. Note that it is easier to do it
> within the environment code since we have access to the profile even
> if it is being built for the first time. There are some wrinkles with
> linking something like /bin since we currently add a link for sh; see
> the comments in my diff.
> 
> Right now I did not handle a multi-arch setup, though that shouldn't
> be too difficult. This would probably require an option to build
> either all or specified packages for an additional arch, like 32bit
> in a 64bit system, and make the libraries available (/lib32 or
> something). Though may run into a union-build bug [2]?
> 
> 2. Typically binaries will expect the ld loader to use
> /etc/ld.so.cache for finding libraries. So, I made a glibc package
> that removes our dl cache patch to restore this behavior. It seems
> enough to add this as a default package to the container, though I
> commented out an option to automatically graft everything with this
> glibc. Both worked for me, but grafting didn't seem necessary.
Would it make sense to keep our libc, but also symlink the cache to its
FHS place?  Or do we need to patch our libc so that this cache is tried
if a Guix-specific one isn't found?

> The second step is to generate the ld cache, which is done with a
> simple startup script in the container, after creating a temporary
> ld.so.conf (our ldconfig doesn't use the usual default directories?).
> I'm sure I found the hackiest way to do this, but it works :) Again,
> this could be possible without modifying guix containers, but this is
> easier. (For example, you can see work done by myself and others at a
> certain non-free channel to do exactly this.)
> 
> Some questions going forward, besides overall cleanup and tweaking of
> the code (which I provided comments in for some details, please see
> there). It might be nice to be able to extend containers more easily
> with setup scripts, though again this can be done by making some
> Guile scripts to wrap a container and making a package around that
> (e.g. from the non-free channel). What kind of extensions would be
> useful to expose? I think I saw some talk on IRC recently about how
> to manage env variables when using guix shell. Perhaps an extended
> manifest format for shell?
> 
> Relatedly and more generally, perhaps it would be good to have
> somewhere (cookbook?) some recipes of useful guix shell container
> incantations. Sharing what you need from the host for graphical
> programs can be a little tricky to figure out. We have the --network
> option, maybe others would be useful? Or some base package lists for
> development: just like we have our various -build-system's, a package
> list with typical library sets might be a nice convenience.
> 
> What about other uses for this container, like providing an isolated
> environment to build and run software we can't do fully with
> bootstrap and sources (like JS)? Could this become some stop-gap to
> allow people to work with these ecosystems in a more controlled way
> within Guix? Or an alternative build environment? Not entirely sure
> what this could mean, just thinking out loud.
I don't think we should rely on FHS containers in the build system. 
It's nice to have such a feature as a user, who really needs
$ELECTRON_APP and doesn't care about the fact that it's a packaging
nightmare, but on a system level, we ought to provide abstractions that
are meaningful and helpful to *everyone*, and FHS containers increase
both the complexity and the failure potential.  See also antioxidant-
build-system as an example for making Rust packaging sane (we'll likely
need a similar effort for JS).

> [...]

> +;;; Copyright © 2021 John Kehayias <john.kehayias@protonmail.com>
Is this the right year?

> +  -F, --fhs-container    run command within an isolated FHS
> container"))
Rather than adding "--fhs-container" as an alternative to --container,
I'd like something like "--emulate-fhs", which as the "--networking"
option is to be *combined* with -C.  This makes the interface more
symmetric.

> +                (symlink lib-dir "/lib64")
> +                (symlink lib-dir "/lib")
You'll probably want to split 32-bit libraries (if any) and 64-bit ones
here.  Also, bear in mind that certain architectures are 32-bit only.

Cheers

Re: [WIP Patch] Adding an FHS container to guix shell

[John Kehayias]

Hi Liliana,

On Thursday, July 14th, 2022 at 10:59 AM, Liliana Marie Prikler wrote:
>
>
> Hi John,
>
> Am Dienstag, dem 12.07.2022 um 15:59 +0000 schrieb John Kehayias:
>
(snip)
> > 2. Typically binaries will expect the ld loader to use
> > /etc/ld.so.cache for finding libraries. So, I made a glibc package
> > that removes our dl cache patch to restore this behavior. It seems
> > enough to add this as a default package to the container, though I
> > commented out an option to automatically graft everything with this
> > glibc. Both worked for me, but grafting didn't seem necessary.
>
> Would it make sense to keep our libc, but also symlink the cache to its
> FHS place? Or do we need to patch our libc so that this cache is tried
> if a Guix-specific one isn't found?
>

I think it may be possible, but I wasn't able to do this exactly in my efforts for a certain non-free package. So I don't remember the details exactly, but roughly I think it is that our libc will look based on $ORIGIN into a /gnu/store/.... location for an ld cache. I tried to make this happen with some symlinking in the container, but it didn't work for me (maybe just due to the complexity of the binary I was trying this with?). The general issue might just be making this adaptable if it could work. But admittedly I was doing a lot of guessing and learning, so maybe I just wasn't approaching this correctly. And then removing one patch was just too easy :)

As for patching libc so that it falls back to something like /etc/ld.so.cache, I would worry about undesired behavior on a foreign distro. Though we could have it look at some other location that wouldn't conflict (some special directory for Guix?). Still, this sounds like it would add complexity and difficult to diagnose behavior. Seems cleaner to me to distinguish the libc in the FHS container explicitly.

Still, I'm open to hear how this could work, just saying I wasn't able to figure it out using this approach before.

WDYT?

(snip)
> > What about other uses for this container, like providing an isolated
> > environment to build and run software we can't do fully with
> > bootstrap and sources (like JS)? Could this become some stop-gap to
> > allow people to work with these ecosystems in a more controlled way
> > within Guix? Or an alternative build environment? Not entirely sure
> > what this could mean, just thinking out loud.
>
> I don't think we should rely on FHS containers in the build system.
> It's nice to have such a feature as a user, who really needs
> $ELECTRON_APP and doesn't care about the fact that it's a packaging
> nightmare, but on a system level, we ought to provide abstractions that
> are meaningful and helpful to everyone, and FHS containers increase
> both the complexity and the failure potential. See also antioxidant-
> build-system as an example for making Rust packaging sane (we'll likely
> need a similar effort for JS).
>

Sure, I agree the main goal here is definitely for the end-user rather than building. Perhaps there are some applications in some building, but yeah, didn't have anything in mind.

> > [...]
>
> > +;;; Copyright © 2021 John Kehayias john.kehayias@protonmail.com
>
> Is this the right year?
>

[checks calendar] it is not. In fact, we're closer to 2023 than 2021 at this point.

Thanks for spotting!

> > + -F, --fhs-container run command within an isolated FHS
> > container"))
>
> Rather than adding "--fhs-container" as an alternative to --container,
> I'd like something like "--emulate-fhs", which as the "--networking"
> option is to be combined with -C. This makes the interface more
> symmetric.
>

Ah, this is a good idea. Then we would enforce that --container is also included when using the --emulate-fhs option. And the little code I figured out to hijack setting the container option without the user specifying it could be dropped to make it all more explicit from the command interface as well. I'll do that.

> > + (symlink lib-dir "/lib64")
> > + (symlink lib-dir "/lib")
>
> You'll probably want to split 32-bit libraries (if any) and 64-bit ones
> here. Also, bear in mind that certain architectures are 32-bit only.
>

Yes, thanks, I was sort of following what Arch does as my test case. But as pointed out by Maxim as well, we should keep a generic following of the standard. I would like some testing to know if some less standard links are widely used though, standards being one thing and what distros do in practice another. I'll also leave multiple arch setups for a followup, keeping this first attempt generic.

Thanks for the helpful comments!

John

Re: [WIP Patch] Adding an FHS container to guix shell

[Maxim Cournoyer]

Hi John,

John Kehayias <john.kehayias@protonmail.com> writes:

> Hi Guixers,
>
> Apologies for the long email, so let me start with the punchline:
> attached is a diff which adds an '--fhs-container' (or -F) option to
> guix shell/environment to set up an FHS-like container. This includes
> the usual /lib directory and a glibc which loads (a generated in the
> container) /etc/ld.so.cache. This should allow running most things
> that expect a more "typical" Linux environment. Give it a try!

Neat hack!

[...]

> diff --git a/gnu/packages/base.scm b/gnu/packages/base.scm
> index 4bdc3e7792..1b4c99d3e9 100644
> --- a/gnu/packages/base.scm
> +++ b/gnu/packages/base.scm
> @@ -928,6 +928,20 @@ (define-public glibc
>     (license lgpl2.0+)
>     (home-page "https://www.gnu.org/software/libc/")))
>
> +;; Define glibc-for-fhs (with a name that allows grafts for glibc), a variation
> +;; of glibc which uses the default ld.so.cache, useful in FHS containers.
> +;; Note: should this be hidden?

^ This should definitely be hidden since it's for internal consumption
only.  Also, if grafting ended up not being needed, the variable should
eb renamed 'glibc-for-fhs', an mentions of grafting in the comments
removed.

> +(define-public gcfhs
> +  (package
> +    (inherit glibc)
> +    (name "gcfhs")
> +    (source (origin (inherit (package-source glibc))
> +                    ;; Remove Guix's patch to read ld.so.cache from /gnu/store
> +                    ;; directories, re-enabling the default /etc/ld.so.cache
> +                    ;; behavior.
> +                    (patches (delete (car (search-patches "glibc-dl-cache.patch"))
> +                                     (origin-patches (package-source glibc))))))))
> +
>  ;; Below are old libc versions, which we use mostly to build locale data in
>  ;; the old format (which the new libc cannot cope with.)
>  (define-public glibc-2.32
> diff --git a/guix/scripts/environment.scm b/guix/scripts/environment.scm
> index 3216235937..425649b843 100644
> --- a/guix/scripts/environment.scm
> +++ b/guix/scripts/environment.scm
> @@ -2,6 +2,7 @@
>  ;;; Copyright © 2014, 2015, 2018 David Thompson <davet@gnu.org>
>  ;;; Copyright © 2015-2022 Ludovic Courtès <ludo@gnu.org>
>  ;;; Copyright © 2018 Mike Gerwitz <mtg@gnu.org>
> +;;; Copyright © 2021 John Kehayias <john.kehayias@protonmail.com>
>  ;;;
>  ;;; This file is part of GNU Guix.
>  ;;;
> @@ -45,6 +46,7 @@ (define-module (guix scripts environment)
>    #:autoload   (guix build syscalls) (set-network-interface-up openpty login-tty)
>    #:use-module (gnu system file-systems)
>    #:autoload   (gnu packages) (specification->package+output)
> +  #:autoload   (gnu packages base) (gcfhs)
>    #:autoload   (gnu packages bash) (bash)
>    #:autoload   (gnu packages bootstrap) (bootstrap-executable %bootstrap-guile)
>    #:use-module (ice-9 match)
> @@ -101,6 +103,8 @@ (define (show-environment-options-help)
>    (display (G_ "
>    -C, --container        run command within an isolated container"))
>    (display (G_ "
> +  -F, --fhs-container    run command within an isolated FHS container"))
> +  (display (G_ "
>    -N, --network          allow containers to access the network"))
>    (display (G_ "
>    -P, --link-profile     link environment profile to ~/.guix-profile within
> @@ -229,6 +233,10 @@ (define %options
>           (option '(#\C "container") #f #f
>                   (lambda (opt name arg result)
>                     (alist-cons 'container? #t result)))
> +         (option '(#\F "fhs-container") #f #f
> +                 (lambda (opt name arg result)
> +                   (alist-cons 'fhs-container? #t
> +                               (alist-cons 'container? #t result))))
>           (option '(#\N "network") #f #f
>                   (lambda (opt name arg result)
>                     (alist-cons 'network? #t result)))
> @@ -606,9 +614,10 @@ (define* (launch-environment/fork command profile manifest
>             ((_ . status)
>              (validate-exit-status profile command status))))))
>
> -(define* (launch-environment/container #:key command bash user user-mappings
> -                                       profile manifest link-profile? network?
> -                                       map-cwd? (white-list '()))
> +(define* (launch-environment/container #:key command bash fhs-container? user
> +                                       user-mappings profile manifest
> +                                       link-profile? network? map-cwd?
> +                                       (white-list '()))
>    "Run COMMAND within a container that features the software in PROFILE.
>  Environment variables are set according to the search paths of MANIFEST.
>  The global shell is BASH, a file name for a GNU Bash binary in the
> @@ -709,6 +718,49 @@ (define* (launch-environment/container #:key command bash user user-mappings
>              (mkdir-p home-dir)
>              (setenv "HOME" home-dir)
>
> +            ;; Set up an FHS container.

I feel like FHS should be spelled in full somewhere (in the doc would be
nice).

> +            (when fhs-container?
> +              ;; Set up the expected bin and library directories as symlinks to
> +              ;; the profile lib directory.  Note that this is assuming a 64bit
> +              ;; architecture.
> +              (let ((lib-dir (string-append profile "/lib")))
> +                (symlink lib-dir "/lib64")

I think /lib64 is not needed (see below).

> +                (symlink lib-dir "/lib")
> +                (mkdir-p "/usr")
> +                (symlink lib-dir "/usr/lib"))
> +              ;; Note: can't symlink full /bin in the container due to the sh
> +              ;; symlink.

Can you explain more exactly why symlinks prevent this to work?  I think
I've had issues like this before (e.g., #46782), but couldn't really
pinpoint the exact nature of the limitation.

> +              (symlink (string-append profile "/bin") "/usr/bin")
> +              (symlink (string-append profile "/sbin") "/sbin")
> +              (symlink (string-append profile "/sbin") "/usr/sbin")

Other symbolic links I think we should include:

* /include -> /usr/include (section 4.3 of the FHS 3.0 spec)
* /libexec -> /usr/libexec (optional, section 4.7)
* /share -> /usr/share (section 4.11)

> +              ;; Provide a frequently expected 'cc' symlink to gcc, though this
> +              ;; could also be done by the user in the container, e.g. in
> +              ;; $HOME/.local/bin and adding that to $PATH.  Note: we do this
> +              ;; in /bin since that already has the sh symlink and can't write
> +              ;; to the other bin directories that are already symlinks themselves.
> +              (symlink (string-append profile "/bin/gcc") "/bin/cc")
> +              ;; TODO: python may also be expected to symlink to python3.

No need for a TODO, I think users wanting that could simply use
'python-wrapper', which exists for this purpose, no?

> +              ;; Guix's ldconfig doesn't seem to search in FHS default
> +              ;; locations, so provide a minimal ld.so.conf.
> +              ;; TODO: this may need more, e.g. libnss3 is in /lib/nss
> +              (call-with-output-file "/tmp/ld.so.conf"
> +                (lambda (port)
> +                  (display "/lib64" port)
> +                  (newline port)))

Shouldn't this be '/lib' rather than '/lib64'?  Per sections 3.9 and
3.10 of the FHS 3.0 [0], '/lib' is essential, while '/lib<qual>' is
optional.

[0] https://refspecs.linuxfoundation.org/FHS_3.0/fhs-3.0.pdf

> +              ;; Define an entry script to start the container: generate
> +              ;; ld.so.cache, supplement $PATH, and include command.
> +              (call-with-output-file "/tmp/fhs.sh"
> +                (lambda (port)
> +                  (display "ldconfig -X -f /tmp/ld.so.conf" port)
> +                  (newline port)
> +                  (display "export PATH=/bin:/usr/bin:/sbin:/usr/sbin:$PATH" port)
> +                  (newline port)
> +                  (display (car command) port)
> +                  (newline port))))
> +
>              ;; If requested, link $GUIX_ENVIRONMENT to $HOME/.guix-profile;
>              ;; this allows programs expecting that path to continue working as
>              ;; expected within a container.
> @@ -746,7 +798,12 @@ (define* (launch-environment/container #:key command bash user user-mappings
>              (primitive-exit/status
>               ;; A container's environment is already purified, so no need to
>               ;; request it be purified again.
> -             (launch-environment command
> +             (launch-environment (if fhs-container?
> +                                     ;; Use the FHS start script.
> +                                     ;; FIXME: probably the default command should
> +                                     ;; be different as it spawns a different shell?
> +                                     '("/bin/sh" "/tmp/fhs.sh")

I'm not sure what could be improved here (why the FIXME?); could you
explain what you had on mind in more details?

> +                                     command)
>                                   (if link-profile?
>                                       (string-append home-dir "/.guix-profile")
>                                       profile)
> @@ -874,15 +931,16 @@ (define (guix-environment* opts)
>    "Run the 'guix environment' command on OPTS, an alist resulting for
>  command-line option processing with 'parse-command-line'."
>    (with-error-handling
> -    (let* ((pure?      (assoc-ref opts 'pure))
> -           (container? (assoc-ref opts 'container?))
> -           (link-prof? (assoc-ref opts 'link-profile?))
> -           (network?   (assoc-ref opts 'network?))
> -           (no-cwd?    (assoc-ref opts 'no-cwd?))
> -           (user       (assoc-ref opts 'user))
> -           (bootstrap? (assoc-ref opts 'bootstrap?))
> -           (system     (assoc-ref opts 'system))
> -           (profile    (assoc-ref opts 'profile))
> +    (let* ((pure?          (assoc-ref opts 'pure))
> +           (container?     (assoc-ref opts 'container?))
> +           (fhs-container? (assoc-ref opts 'fhs-container?))
> +           (link-prof?     (assoc-ref opts 'link-profile?))
> +           (network?       (assoc-ref opts 'network?))
> +           (no-cwd?        (assoc-ref opts 'no-cwd?))
> +           (user           (assoc-ref opts 'user))
> +           (bootstrap?     (assoc-ref opts 'bootstrap?))
> +           (system         (assoc-ref opts 'system))
> +           (profile        (assoc-ref opts 'profile))
>             (command    (or (assoc-ref opts 'exec)
>                             ;; Spawn a shell if the user didn't specify
>                             ;; anything in particular.
> @@ -927,7 +985,16 @@ (define (guix-environment* opts)
>        (with-store/maybe store
>          (with-status-verbosity (assoc-ref opts 'verbosity)
>            (define manifest-from-opts
> -            (options/resolve-packages store opts))
> +            (options/resolve-packages store
> +                                      ;; For an FHS-container, add a glibc that uses
> +                                      ;; /etc/ld.so.cache.
> +                                      (if fhs-container?
> +                                          (alist-cons 'package '(ad-hoc-package "gcfhs")
> +                                                      opts)
> +                                          ;; Alternatively, could graft all packages with
> +                                          ;; this glibc, though that seems unnecessary.
> +                                          ;; (alist-cons 'with-graft "glibc=gcfhs" opts)
> +                                          opts)))

It's interesting that it's unnecessary.  If we're confident it really
is, then I'd remove this dead code and comment.

>
>            (define manifest
>              (if profile
> @@ -994,6 +1061,7 @@ (define (guix-environment* opts)
>                                                "/bin/sh"))))
>                        (launch-environment/container #:command command
>                                                      #:bash bash-binary
> +                                                    #:fhs-container? fhs-container?
>                                                      #:user user
>                                                      #:user-mappings mappings
>                                                      #:profile profile

That's surprisingly little code, which is nice!  Thanks for looking into
it :-).

I've yet to try it, but I will shortly.

Thank you,

Maxim

Re: [WIP Patch] Adding an FHS container to guix shell

[John Kehayias]

Hi Maxim,

(with copious snips of the long original email)

------- Original Message -------
On Friday, July 15th, 2022 at 9:43 AM, Maxim Cournoyer wrote:

>
>
> Hi John,
>
> John Kehayias john.kehayias@protonmail.com writes:
>
> > Hi Guixers,
> >
> > Apologies for the long email, so let me start with the punchline:
> > attached is a diff which adds an '--fhs-container' (or -F) option to
> > guix shell/environment to set up an FHS-like container. This includes
> > the usual /lib directory and a glibc which loads (a generated in the
> > container) /etc/ld.so.cache. This should allow running most things
> > that expect a more "typical" Linux environment. Give it a try!
>
> Neat hack!

Thanks!

> > diff --git a/gnu/packages/base.scm b/gnu/packages/base.scm
> > index 4bdc3e7792..1b4c99d3e9 100644
> > --- a/gnu/packages/base.scm
> > +++ b/gnu/packages/base.scm
> > @@ -928,6 +928,20 @@ (define-public glibc
> > (license lgpl2.0+)
> > (home-page "https://www.gnu.org/software/libc/")))
> >
> > +;; Define glibc-for-fhs (with a name that allows grafts for glibc), a variation
> > +;; of glibc which uses the default ld.so.cache, useful in FHS containers.
> > +;; Note: should this be hidden?
>
> ^ This should definitely be hidden since it's for internal consumption
> only. Also, if grafting ended up not being needed, the variable should
> eb renamed 'glibc-for-fhs', an mentions of grafting in the comments
> removed.
>

I'll respond to the grafting comment more below, but for now I just wanted to have the options available in this "preview" patch. I'm also wondering if it is worthwhile to leave the grafting option open for a user (so they can still explicitly do a --with-graft), though I guess we can change in the future if a bug report comes up. And for selfish reasons I wanted the grafting code somewhere so I wouldn't forget how to add that option in the container code behind the scenes :)

> > + ;; Set up an FHS container.
>
> I feel like FHS should be spelled in full somewhere (in the doc would be
> nice).
>

Agreed, along with explicit documentation as you noted in your followup email. I would want to link to the spec in the manual for reference as well.

> > + (when fhs-container?
> > + ;; Set up the expected bin and library directories as symlinks to
> > + ;; the profile lib directory. Note that this is assuming a 64bit
> > + ;; architecture.
> > + (let ((lib-dir (string-append profile "/lib")))
> > + (symlink lib-dir "/lib64")
>
> I think /lib64 is not needed (see below).
>

Right. Everything was all linked so I had just picked one. As you detail below, this should be cleaned up to be the generic standard specified for FHS.

> > + (symlink lib-dir "/lib")
> > + (mkdir-p "/usr")
> > + (symlink lib-dir "/usr/lib"))
> > + ;; Note: can't symlink full /bin in the container due to the sh
> > + ;; symlink.
>
> Can you explain more exactly why symlinks prevent this to work? I think
> I've had issues like this before (e.g., #46782), but couldn't really
> pinpoint the exact nature of the limitation.
>

Oh, I didn't mean anything complicated, just that you can't link /bin since a symlink /bin/sh was already created for containers. Instead, we could mkdir /bin and then add the sh symlink along with everything from profile/bin with a little loop. I just didn't think to do it at the time for my first pass at this.

I think for the final version it would be good to do this, as well as looping through any profile/lib/ subdirectories to add to ld.so.conf. I know at least our libnss3 is in a subdirectory for some reason, which needs to be added to where ldconfig looks or else things that need libnss3 won't find it.

> > + (symlink (string-append profile "/bin") "/usr/bin")
> > + (symlink (string-append profile "/sbin") "/sbin")
> > + (symlink (string-append profile "/sbin") "/usr/sbin")
>
> Other symbolic links I think we should include:
>
> * /include -> /usr/include (section 4.3 of the FHS 3.0 spec)
> * /libexec -> /usr/libexec (optional, section 4.7)
> * /share -> /usr/share (section 4.11)
>

Thanks, good idea. Surprisingly (or not?) the /lib and ldcache seem to be the big one. I guess env variables (search-paths) take care of most everything else, but we should set it all up.

> > + ;; Provide a frequently expected 'cc' symlink to gcc, though this
> > + ;; could also be done by the user in the container, e.g. in
> > + ;; $HOME/.local/bin and adding that to $PATH. Note: we do this
> > + ;; in /bin since that already has the sh symlink and can't write
> > + ;; to the other bin directories that are already symlinks themselves.
> > + (symlink (string-append profile "/bin/gcc") "/bin/cc")
> > + ;; TODO: python may also be expected to symlink to python3.
>
> No need for a TODO, I think users wanting that could simply use
> 'python-wrapper', which exists for this purpose, no?
>

TIL. Yes, perfect.

Is there any use to have a similar one for cc/gcc? Would that be useful outside of this container? I know we frequently need to specify $CC in build patching, I just haven't done much building outside of packaging to know otherwise.

> > + ;; Guix's ldconfig doesn't seem to search in FHS default
> > + ;; locations, so provide a minimal ld.so.conf.
> > + ;; TODO: this may need more, e.g. libnss3 is in /lib/nss
> > + (call-with-output-file "/tmp/ld.so.conf"
> > + (lambda (port)
> > + (display "/lib64" port)
> > + (newline port)))
>
> Shouldn't this be '/lib' rather than '/lib64'? Per sections 3.9 and
> 3.10 of the FHS 3.0 [0], '/lib' is essential, while '/lib<qual>' is
> optional.
>
> [0] https://refspecs.linuxfoundation.org/FHS_3.0/fhs-3.0.pdf
>

Yes, thanks. I should make this all more generic per the standard.

> > + (launch-environment (if fhs-container?
> > + ;; Use the FHS start script.
> > + ;; FIXME: probably the default command should
> > + ;; be different as it spawns a different shell?
> > + '("/bin/sh" "/tmp/fhs.sh")
>
> I'm not sure what could be improved here (why the FIXME?); could you
> explain what you had on mind in more details?
>

The shell that is launched looks more like a plain 'sh' rather than what you normally get in an empty guix shell --container. So what I currently have is a call to /bin/sh /tmp/fhs.sh where the last line of the fhs.sh script is the command specified by the user, or else /bin/sh itself. This results in a different prompt at least. I may have gotten some warning after exiting the container as well (didn't find /bin/sh? I don't have it in front of me). I'll take a look, but open to how best to spawn the shell from this fhs.sh script as the fallback when no command is given.

> > + ;; For an FHS-container, add a glibc that uses
> > + ;; /etc/ld.so.cache.
> > + (if fhs-container?
> > + (alist-cons 'package '(ad-hoc-package "gcfhs")
> > + opts)
> > + ;; Alternatively, could graft all packages with
> > + ;; this glibc, though that seems unnecessary.
> > + ;; (alist-cons 'with-graft "glibc=gcfhs" opts)
> > + opts)))
>
> It's interesting that it's unnecessary. If we're confident it really
> is, then I'd remove this dead code and comment.
>

It is interesting, and at first I was having problems with linkers finding libgcc. It wasn't appearing in the ld cache, and I thought it was because the non-FHS libc was referenced in some gcc package (which then wasn't in the container). But I think that was a red herring and may have been something else I was doing while testing, since I haven't hit that problem since. And I don't think that should have caused a problem, as long as whatever libc is referenced is available, libgcc should have appeared in the cache. But I'm not sure now what was happening, so I had tried grafting, came back to trying without grafts and it still worked...

In any event, some testing would be helpful. Just including the FHS libc is very simple and quick rather than grafting. I think the main point is just that software/code tends to rely on the ld cache to find libraries, so the ld loader needs to do this for "default" behavior. After finding the libraries, everything else goes fine and is then linked properly in the binary. (Now I wonder if you build a binary in the FHS container if it will just run in regular Guix since everything should be linked to explicit store locations in the end...?)

>
> That's surprisingly little code, which is nice! Thanks for looking into
> it :-).
>

Thanks for the helpful comments! And credit to work that is out there that came before. Admittedly it was much easier this time (just one late evening) than when I first tried all of this for some binaries. The end result is really just needing the ld cache (generation/loading) and some symlinks, not bad at all.

> I've yet to try it, but I will shortly.
>
> Thank you,
>
> Maxim

Most welcome! And yes, some testing would be great to find what assumptions are here or should be here to make this useable without much extra fussing for each piece of software.

John

Re: [WIP Patch] Adding an FHS container to guix shell

[Maxim Cournoyer]

Hi,

I forgot to add, this would of course need to be documented in our
'Invoking guix shell' section of the manual.

Thanks,

Maxim

Re: [WIP Patch] Adding an FHS container to guix shell

[Ludovic Courtès]

Hello!

John Kehayias <john.kehayias@protonmail.com> skribis:

> 2. Typically binaries will expect the ld loader to use
> /etc/ld.so.cache for finding libraries.

Not necessarily.  /etc/ld.so.cache is an optimization, but it’s entirely
possible to opt out: if that file is missing, things will work fine, but
libc will ‘stat’ more to find files.  That would make glibc-for-fhs
unnecessary.

> +(define* (launch-environment/container #:key command bash fhs-container? user
> +                                       user-mappings profile manifest
> +                                       link-profile? network? map-cwd?
> +                                       (white-list '()))
>    "Run COMMAND within a container that features the software in PROFILE.
>  Environment variables are set according to the search paths of MANIFEST.
>  The global shell is BASH, a file name for a GNU Bash binary in the
> @@ -709,6 +718,49 @@ (define* (launch-environment/container #:key command bash user user-mappings
>              (mkdir-p home-dir)
>              (setenv "HOME" home-dir)
>  
> +            ;; Set up an FHS container.
> +            (when fhs-container?
> +              ;; Set up the expected bin and library directories as symlinks to
> +              ;; the profile lib directory.  Note that this is assuming a 64bit
> +              ;; architecture.
> +              (let ((lib-dir (string-append profile "/lib")))
> +                (symlink lib-dir "/lib64")
> +                (symlink lib-dir "/lib")
> +                (mkdir-p "/usr")
> +                (symlink lib-dir "/usr/lib"))

Instead of adding code here, maybe you could do in a more declarative
fashion, like:

  (define fhs-mappings
    (list (file-system-mapping
            (source (string-append profile "/bin")) (target "/bin"))
          …))

and append that to the ‘mappings’ variable there.  It’s not necessarily
more compact, but maybe marginally easier to read?

> +              ;; Define an entry script to start the container: generate
> +              ;; ld.so.cache, supplement $PATH, and include command.

I’d leave ld.so.cache generation out.

> +              (call-with-output-file "/tmp/fhs.sh"
> +                (lambda (port)
> +                  (display "ldconfig -X -f /tmp/ld.so.conf" port)
> +                  (newline port)
> +                  (display "export PATH=/bin:/usr/bin:/sbin:/usr/sbin:$PATH" port)

I think the default value of PATH in our libc is the FHS one:

--8<---------------cut here---------------start------------->8---
$ env -i $(type -P strace) -e execve -f $(type -P guile) -c '(system* "whatever")'
execve("/gnu/store/lpcjxka7hx3ypv4nz47g08k4m2syqwlj-profile/bin/guile", ["/gnu/store/lpcjxka7hx3ypv4nz47g0"..., "-c", "(system* \"whatever\")"], 0x7ffede27ad38 /* 0 vars */) = 0
/home/ludo/.guix-home/profile/bin/strace: Process 9727 attached
/home/ludo/.guix-home/profile/bin/strace: Process 9728 attached
/home/ludo/.guix-home/profile/bin/strace: Process 9729 attached
/home/ludo/.guix-home/profile/bin/strace: Process 9730 attached
/home/ludo/.guix-home/profile/bin/strace: Process 9731 attached
/home/ludo/.guix-home/profile/bin/strace: Process 9732 attached
[pid  9732] execve("/bin/whatever", ["whatever"], 0x7ffed6d967c8 /* 0 vars */) = -1 ENOENT (No such file or directory)
[pid  9732] execve("/usr/bin/whatever", ["whatever"], 0x7ffed6d967c8 /* 0 vars */) = -1 ENOENT (No such file or directory)
In execvp of whatever: No such file or directory
--8<---------------cut here---------------end--------------->8---

So you could leave it undefined, but ‘load-profile’ in
‘launch-environment’ will define it.

Instead of the wrapper script, maybe you could extend
‘launch-environment’ so the caller can make it override certain
variables?  I would find it a bit clearer.

Thanks,
Ludo’.

Re: [WIP Patch] Adding an FHS container to guix shell

[John Kehayias]

Hi Ludo’,

------- Original Message -------
On Monday, July 18th, 2022 at 7:40 AM, Ludovic Courtès wrote:

> Hello!
>
> John Kehayias skribis:
>
> > 2. Typically binaries will expect the ld loader to use
> > /etc/ld.so.cache for finding libraries.
>
> Not necessarily. /etc/ld.so.cache is an optimization, but it’s entirely
> possible to opt out: if that file is missing, things will work fine, but
> libc will ‘stat’ more to find files. That would make glibc-for-fhs
> unnecessary.
>

I was thinking mostly of binaries, though I've also encountered code that wanted to read the ldcache (e.g. parsing ldconfig -p directly), for some reason. Is there any difficulty with having already some /etc/ld.so.cache from some package in the container profile? Perhaps that leads some programs to rely on the cache and then not continue searching for libraries? I'm not sure how that works, but only know that I've found the ldcache necessary (at least without modifying source) for things I would need the FHS container for.

My overall goal with this FHS option is to mimic both the standard and the typical expectations of distros that follow it (global ld cache, certain directories in PATH, etc). In my testing I didn't get very far without the modified glibc, though perhaps there are other ways around it. (There's patchelf, though hard to scale that I think.)

For example, lots of language build sytems/package managers will bring in some binaries, whether it is just to install (rustup does this) or because something like a web automation framework downloads browser binaries to use. My goal here was to make this all "just work," but I can understand wanting to keep it minimal.

I'll proceed with the glibc-for-fhs and ldcache for now, but open to discussing and hearing what everyone thinks. Overall, I find it necessary to enable the use-cases I'm thinking of.

> > + ;; Set up an FHS container.
> > + (when fhs-container?
> > + ;; Set up the expected bin and library directories as symlinks to
> > + ;; the profile lib directory. Note that this is assuming a 64bit
> > + ;; architecture.
> > + (let ((lib-dir (string-append profile "/lib")))
> > + (symlink lib-dir "/lib64")
> > + (symlink lib-dir "/lib")
> > + (mkdir-p "/usr")
> > + (symlink lib-dir "/usr/lib"))
>
> Instead of adding code here, maybe you could do in a more declarative
> fashion, like:
>
> (define fhs-mappings
> (list (file-system-mapping
> (source (string-append profile "/bin")) (target "/bin"))
> …))
>
> and append that to the ‘mappings’ variable there. It’s not necessarily
> more compact, but maybe marginally easier to read?
>

Thanks, that's all a good idea for tidying up! I did so, further breaking down the FHS directories to ones we can map directly like this, others that need directory contents symlinked (because the container already has e.g. /bin not empty), and then creating the optional FHS directories to link to these. For example, /lib is mapped to profile/lib and /usr/lib is linked to /lib. Again, not strictly necessary, but I was trying to follow the structure of other distros here. And this way there is still "one" place where everything from the profile ends up, /lib, /bin, etc.

> > + ;; Define an entry script to start the container: generate
> > + ;; ld.so.cache, supplement $PATH, and include command.
>
> I’d leave ld.so.cache generation out.
>

As discussed above, I'm finding this rather necessary and makes for a smooth FHS-container experience. I'm open to other options, maybe making this an additional option or hiding it away, though I think the design here fits the "pretend to be like another distro and just work" that I was going for.

> > + (call-with-output-file "/tmp/fhs.sh"
> > + (lambda (port)
> > + (display "ldconfig -X -f /tmp/ld.so.conf" port)
> > + (newline port)
> > + (display "export PATH=/bin:/usr/bin:/sbin:/usr/sbin:$PATH" port)
>
> I think the default value of PATH in our libc is the FHS one:
>
> --8<---------------cut here---------------start------------->8---
>
> $ env -i $(type -P strace) -e execve -f $(type -P guile) -c '(system* "whatever")'
> execve("/gnu/store/lpcjxka7hx3ypv4nz47g08k4m2syqwlj-profile/bin/guile", ["/gnu/store/lpcjxka7hx3ypv4nz47g0"..., "-c", "(system* \"whatever\")"], 0x7ffede27ad38 /* 0 vars /) = 0
> /home/ludo/.guix-home/profile/bin/strace: Process 9727 attached
> /home/ludo/.guix-home/profile/bin/strace: Process 9728 attached
> /home/ludo/.guix-home/profile/bin/strace: Process 9729 attached
> /home/ludo/.guix-home/profile/bin/strace: Process 9730 attached
> /home/ludo/.guix-home/profile/bin/strace: Process 9731 attached
> /home/ludo/.guix-home/profile/bin/strace: Process 9732 attached
> [pid 9732] execve("/bin/whatever", ["whatever"], 0x7ffed6d967c8 / 0 vars /) = -1 ENOENT (No such file or directory)
> [pid 9732] execve("/usr/bin/whatever", ["whatever"], 0x7ffed6d967c8 / 0 vars */) = -1 ENOENT (No such file or directory)
> In execvp of whatever: No such file or directory
> --8<---------------cut here---------------end--------------->8---
>
>
> So you could leave it undefined, but ‘load-profile’ in
> ‘launch-environment’ will define it.
>

I'm not sure what you were showing here/I'm a bit confused. If I do just 'guix shell -C' for example, I can't just do 'sh' or 'exec sh' in this environment although /bin/sh exists (as a link to /gnu/store/...bash-.../bin/sh). While your example does indeed find sh (if I include the needed inputs to run the example), I'm not sure how that relates to running things directly. But maybe you just mean in terms of libc directly. Sorry, I don't know quite enough in this area.

In the container PATH only has profile/bin. Additionally, we frequently see hardcoded path assumptions like /bin or /usr/bin, or expectations of PATH being set with these. Adding, e.g. coreutils and which to the shell, which will happily find 'ls' but not 'sh', as which just searches PATH of course.

This could be worked around by the user, but again, I was going for "just works" as much as possible. Or at least as other distros "just work". (Side note: not that I think this is better! All this FHS stuff gives me even better appreciation for the design of Guix, despite the initial complications one sees as a new user.)

> Instead of the wrapper script, maybe you could extend
> ‘launch-environment’ so the caller can make it override certain
> variables? I would find it a bit clearer.
>

Would this still be better if we do keep the ldcache being generated? I didn't see a quick way to just add in some setup commands to the container, though that could be a nice extension. Admittedly, my script here was the simplest hack I saw that worked, so I'm not married to it by any means. Again, the user could also generate the ldcache, but was trying to make it all automatic.

> Thanks,
> Ludo’.

Thanks for the constructive comments! It definitely helped me clean up the implementation, which I'll be sending shortly to the patches list. I'll followup on this thread as well to point people to a patch they can try.

For my testing I've been able to do work with web automations (uses binaries of browsers), tested that AppImages can work, built a Rust project, etc. A useful tool, even if it is one I hope I don't have to reach for too often.

John

Re: [WIP Patch] Adding an FHS container to guix shell

[Maxime Devos]

[-- Attachment #1.1.1: Type: text/plain, Size: 267 bytes --]

On 20-07-2022 23:22, John Kehayias wrote:

> I was thinking mostly of binaries, though I've also encountered code that wanted to read the ldcache (e.g. parsing ldconfig -p directly), for some reason.

Do you have an example in the wild?

Greetings,
Maxime.


[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 929 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

Re: [WIP Patch] Adding an FHS container to guix shell

[John Kehayias]

Hi Maxime,

------- Original Message -------
On Wednesday, July 20th, 2022 at 7:49 PM, Maxime Devos wrote:

>
> On 20-07-2022 23:22, John Kehayias wrote:
>
> > I was thinking mostly of binaries, though I've also encountered code that wanted to read the ldcache (e.g. parsing ldconfig -p directly), for some reason.
>
>
> Do you have an example in the wild?
>

I've seen it in a couple of python packages. Not the one I was thinking of, but pillow we substitute* to not invoke ldconfig -p: https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/python-xyz.scm?id=49e1396cf5130a80bdb6f09c62f7d0fa6f7c73aa#n7092

(Lutris does this also, but we do not package that)

debbugs irritation Was: [WIP Patch] Adding an FHS container to guix shell

[Csepp]

I was looking for this patch on issues.guix.gnu.org and could not figure
out why it's not there and I'd just like to point out that having like 6
mailing lists with at least two separate indexes is... not great :) I
appreciate everyone's work on Mumi and Debbugs but...  maybe this is an
avoidable problem?

Re: debbugs irritation Was: [WIP Patch] Adding an FHS container to guix shell

[John Kehayias]

Hi,

------- Original Message -------
On Wednesday, July 20th, 2022 at 8:20 PM, Csepp wrote:


>
> I was looking for this patch on issues.guix.gnu.org and could not figure
> out why it's not there and I'd just like to point out that having like 6
> mailing lists with at least two separate indexes is... not great :) I
> appreciate everyone's work on Mumi and Debbugs but... maybe this is an
> avoidable problem?

Err...I was optimistic, as always, in my timing for submitting the complete patch. So only the draft .diff is in the guix-devel list, nothing on debbugs yet. Sorry about that! I was going to (and now will) email the original discussion the the link to the submitted patch series.

Though I also like Mumi, the search can be a bit limiting sometimes when searching for something like "go". Which made me just realize in such cases maybe I should put in the path of the files I'm looking for a patch of. But anyway, I appreciate that search can be difficult with certain words.

And now, please see https://issues.guix.gnu.org/56677 :)

Re: debbugs irritation Was: [WIP Patch] Adding an FHS container to guix shell

[zimoun]

Hi,

On Thu, 21 Jul 2022 at 02:20, Csepp <raingloom@riseup.net> wrote:
> I was looking for this patch on issues.guix.gnu.org and could not figure
> out why it's not there and I'd just like to point out that having like 6
> mailing lists with at least two separate indexes is... not great :) I
> appreciate everyone's work on Mumi and Debbugs but...  maybe this is an
> avoidable problem?

Which are these 6 mailing lists?  What do you mean by 2 separate
indexes?

For the good or not, the GNU project has only one Debbugs instance.
Mumi is just reading this Debbugs instance database to display the
information nicely.  Somehow, Mumi is just another frontend to the GNU
Debbugs instance.  Nothing conceptually different from emacs-debbugs.


Cheers,
simon

Re: debbugs irritation Was: [WIP Patch] Adding an FHS container to guix shell

[Csepp]

zimoun <zimon.toutoune@gmail.com> writes:

> Hi,
>
> On Thu, 21 Jul 2022 at 02:20, Csepp <raingloom@riseup.net> wrote:
>> I was looking for this patch on issues.guix.gnu.org and could not figure
>> out why it's not there and I'd just like to point out that having like 6
>> mailing lists with at least two separate indexes is... not great :) I
>> appreciate everyone's work on Mumi and Debbugs but...  maybe this is an
>> avoidable problem?
>
> Which are these 6 mailing lists?  What do you mean by 2 separate
> indexes?
>
> For the good or not, the GNU project has only one Debbugs instance.
> Mumi is just reading this Debbugs instance database to display the
> information nicely.  Somehow, Mumi is just another frontend to the GNU
> Debbugs instance.  Nothing conceptually different from emacs-debbugs.
>
>
> Cheers,
> simon

Mumi and Debbugs have different search interfaces and seem to use
different ordering.
6 was a slight exaggeration, but development communication happens at
least on:
* patches
* devel
* bugs
And then there is also help, which often overlaps with bugs.
There are also a few others for CI, translation, etc.

IMHO these papercuts add up.  Browsing and cross-referencing issues and
patches is way harder than it is with other forges.
Not saying we need to switch, maybe it's easier to just add the missing
functionality.  Or maybe it doesn't matter to anyone else.

Re: debbugs irritation Was: [WIP Patch] Adding an FHS container to guix shell

[zimoun]

Hi,

On Thu, 21 Jul 2022 at 22:22, Csepp <raingloom@riseup.net> wrote:

> Mumi and Debbugs have different search interfaces and seem to use
> different ordering.

Hum, I am confused because from my understanding, there is one Debbugs
instance – which is quickly said some Perl scripts managing mailing
lists and thus implementing a bug tracker database.  This database is
then manipulated via SOAP-interface.

This Debbugs instance is out of the control of the Guix project, AFAIK.

On the top of this instance, various front-ends are implemented:

 + historical one running at: https://debbugs.gnu.org 
 + Mumi running at: https://issues.guix.gnu.org/
 + emacs-debbugs running locally

And even Debian folks provide many others, as python3-debianbts or
reportbug coming with the CLI tool bts, mailscripts, etc.

https://git.spwhitton.name/mailscripts/tree/

 
> IMHO these papercuts add up.  Browsing and cross-referencing issues and
> patches is way harder than it is with other forges.

Well, harder depends on the point of view. :-)

Indeed, you need more than just type bug#12345 to cross-link.  Using a
descent mailreader, it appears to me easy to have helper.  For instance,
using Emacs, I have a custom function [1] “M-x my/guix-issues“ which
adds to the kill-ring an URL.  Recently, Ricardo added Message-ID to
Mumi which helps too; using emacs-notmuch, just "ci” for stashing and
then pasting elsewhere.

1: <https://gitlab.com/zimoun/my-conf/-/blob/master/.config/emacs/lisp/funs.el#L271-300>


About browsing, Mumi needs more love. :-)


> Not saying we need to switch, maybe it's easier to just add the missing
> functionality.  Or maybe it doesn't matter to anyone else.

Well, the GNU instance of Debbugs has many flaws.  But the project will
not switch from it, IMHO.  That’s why Mumi as front-end tries to improve
the situation by adding the missing functionalities.


Cheers,
simon

Re: debbugs irritation Was: [WIP Patch] Adding an FHS container to guix shell

[jbranso]

August 18, 2022 11:01 AM, "zimoun" <zimon.toutoune@gmail.com> wrote:

> Hi,
> 
> On Thu, 21 Jul 2022 at 22:22, Csepp <raingloom@riseup.net> wrote:
> 
>> Mumi and Debbugs have different search interfaces and seem to use
>> different ordering.
> 
> Hum, I am confused because from my understanding, there is one Debbugs
> instance – which is quickly said some Perl scripts managing mailing
> lists and thus implementing a bug tracker database. This database is
> then manipulated via SOAP-interface.
> 
> This Debbugs instance is out of the control of the Guix project, AFAIK.
> 
> On the top of this instance, various front-ends are implemented:
> 
> + historical one running at: https://debbugs.gnu.org
> + Mumi running at: https://issues.guix.gnu.org
> + emacs-debbugs running locally
> 
> And even Debian folks provide many others, as python3-debianbts or
> reportbug coming with the CLI tool bts, mailscripts, etc.
> 
> https://git.spwhitton.name/mailscripts/tree
> 
>> IMHO these papercuts add up. Browsing and cross-referencing issues and
>> patches is way harder than it is with other forges.
> 
> Well, harder depends on the point of view. :-)
> 
> Indeed, you need more than just type bug#12345 to cross-link. Using a
> descent mailreader, it appears to me easy to have helper. For instance,
> using Emacs, I have a custom function [1] “M-x my/guix-issues“ which
> adds to the kill-ring an URL. Recently, Ricardo added Message-ID to
> Mumi which helps too; using emacs-notmuch, just "ci” for stashing and
> then pasting elsewhere.
> 
> 1: <https://gitlab.com/zimoun/my-conf/-/blob/master/.config/emacs/lisp/funs.el#L271-300>
> 
> About browsing, Mumi needs more love. :-)
> 
>> Not saying we need to switch, maybe it's easier to just add the missing
>> functionality. Or maybe it doesn't matter to anyone else.
> 
> Well, the GNU instance of Debbugs has many flaws. But the project will
> not switch from it, IMHO. That’s why Mumi as front-end tries to improve
> the situation by adding the missing functionalities.

I am actually working on a patch to emacs-debbugs that will provide:

debbugs-gnu-my-open-bugs   

https://issues.guix.gnu.org/56987

(I'm also fairly slow at submitting patches, so please be patient).  :)

Will show you the bugs, which you submitted that are still open.

At least the emacs guys seem fairly interested in improving emacs-debbugs.

:)

> 
> Cheers,
> simon

Re: [WIP Patch] Adding an FHS container to guix shell

[John Kehayias]

Thanks everyone for the useful comments and suggestions! I took most of them into account in polishing the patch for submission and more formal review (with the exception of keeping in glibc-for-fhs and the ld cache; to be discussed). I feel the patch looks much better now thanks to you all.

Please see https://issues.guix.gnu.org/56677 for the two patches to test and review.

I've been using some form of the FHS container in my daily work and for testing and have found it useful, much as I wish to do these things in Guix proper. While for some this is a long way off probably (electron and JS), for others I hope this is just a useful stop-gap until software or systems are packaged for Guix.

Thanks again!
John