From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp11.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id mB0NEH+X0WKtCAEAbAwnHQ (envelope-from ) for ; Fri, 15 Jul 2022 18:36:15 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp11.migadu.com with LMTPS id YGTgD3+X0WJ93QAA9RJhRA (envelope-from ) for ; Fri, 15 Jul 2022 18:36:15 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id E14B2153BB for ; Fri, 15 Jul 2022 18:36:14 +0200 (CEST) Received: from localhost ([::1]:46548 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1oCOIY-0004RU-1k for larch@yhetil.org; Fri, 15 Jul 2022 12:36:14 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:37354) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oCOIF-0004QV-LU for guix-devel@gnu.org; Fri, 15 Jul 2022 12:35:56 -0400 Received: from mail-4316.protonmail.ch ([185.70.43.16]:54607) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oCOIC-00011r-FD for guix-devel@gnu.org; Fri, 15 Jul 2022 12:35:55 -0400 Date: Fri, 15 Jul 2022 16:35:46 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=protonmail.com; s=protonmail3; t=1657902949; x=1658162149; bh=bb6HQQ/7DHVzq4RSeP+kf+Ebs4vKlkYs2GPgkcrIc8U=; h=Date:To:From:Cc:Reply-To:Subject:Message-ID:In-Reply-To: References:Feedback-ID:From:To:Cc:Date:Subject:Reply-To: Feedback-ID:Message-ID; b=U7kyeeqRVUUZHaIt6RgU42zzbas11vveRdrPcC6ErA028YX9mqUnYIWEK7aM06DzX eWpopgtZfD9aF/D8TwK658A7RSwIWMsgzNPv6f6a5ERRuZsoFu6NBoU6Z3zt2r9FV/ YjOiMA/SmxIO6RnTm9ni0Y7vxlwQCwiRAtpqeEazI9b7EMMFSqJyvAqLkrQpjcgYIN sAor6iyTaZoC/xjE0Br+mN8/QpoKwCkU7R2vKYo0LNZERpcBH09b9iUkKZ4La6Rh/i p/+3IPrVTjBLdNPXE2CvLBIFrtTcyLPWQWFZAssbz33HkNEBlamgm/5fuJXwZ0Ym6O 05Hw9tm/t1eEg== To: Maxim Cournoyer From: John Kehayias Cc: "guix-devel@gnu.org" Subject: Re: [WIP Patch] Adding an FHS container to guix shell Message-ID: In-Reply-To: <871qumwkxv.fsf@gmail.com> References: <871qumwkxv.fsf@gmail.com> Feedback-ID: 7805494:user:proton MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=185.70.43.16; envelope-from=john.kehayias@protonmail.com; helo=mail-4316.protonmail.ch X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: John Kehayias Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-To: larch@yhetil.org X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1657902975; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=bb6HQQ/7DHVzq4RSeP+kf+Ebs4vKlkYs2GPgkcrIc8U=; b=mB0R3ja+04r/ZhHYh+6ZmtzISNIZ+6vVa2bp/uz+66vRsIAc9kjFec+KFAOKFcL5NVqzzu pXvdRSR5a2ngvaJQj4pbMXxbQld5l6ARXDW+pIffVhS8B4Rchdo/RFBS/v80vvcf4jfjqh 6MP1fJvJtKwcmSrlO7Yq3dtsQoDxgZXpJ67AWO6vN8EnTcpZT8SQG7mwwdiyLkJ9NDwh4u 6rXj7GnPHYcpxMeVW2Otusj96HbDdcr4Ow0Ab8uOMzGIrxT75Y+CfeEHFFhpsyZW1f/GfW F+5MoYd1lmctcBNC0C8Xm5GUJN01DAqvt4MVTi8aSXwMpeLJkL2Q4aszhErpYg== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1657902975; a=rsa-sha256; cv=none; b=Y2FW5xFYZpIGWgAf7jRgJQZZI67KXUrx6NnHzMKcPQ/+BeA819Ex6BB1qFSjF4vQQjzOIf zdDRG5ImCLgfiKNNWJ4ERJ4t9meUO3mzx6e1RW5z50nXBGiHnAM8QqYO+DyvWj20aMLHyl QxlZeLlDS+aATzioT7K4JB9Ap8O/MwEwKjrhB20E80QMW6Sq64D/e1Dg7dCh/oEs9BphmT f4C6JjWIy7EnTg0793AghMQMDm87Bj4MKVTYkdq/IVnv8CGnKjXV6MWC01CpT2ERCOMrrg 7PKLd919c645TMYWkV7fb8MZbRLZhKYhV+9kpH5vJiDNGMP3q+gCsp9GUX9LKg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=protonmail.com header.s=protonmail3 header.b=U7kyeeqR; dmarc=pass (policy=quarantine) header.from=protonmail.com; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -8.94 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=protonmail.com header.s=protonmail3 header.b=U7kyeeqR; dmarc=pass (policy=quarantine) header.from=protonmail.com; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: E14B2153BB X-Spam-Score: -8.94 X-Migadu-Scanner: scn0.migadu.com X-TUID: 6B34TveUIKCR Hi Maxim, (with copious snips of the long original email) ------- Original Message ------- On Friday, July 15th, 2022 at 9:43 AM, Maxim Cournoyer wrote: > > > Hi John, > > John Kehayias john.kehayias@protonmail.com writes: > > > Hi Guixers, > > > > Apologies for the long email, so let me start with the punchline: > > attached is a diff which adds an '--fhs-container' (or -F) option to > > guix shell/environment to set up an FHS-like container. This includes > > the usual /lib directory and a glibc which loads (a generated in the > > container) /etc/ld.so.cache. This should allow running most things > > that expect a more "typical" Linux environment. Give it a try! > > Neat hack! Thanks! > > diff --git a/gnu/packages/base.scm b/gnu/packages/base.scm > > index 4bdc3e7792..1b4c99d3e9 100644 > > --- a/gnu/packages/base.scm > > +++ b/gnu/packages/base.scm > > @@ -928,6 +928,20 @@ (define-public glibc > > (license lgpl2.0+) > > (home-page "https://www.gnu.org/software/libc/"))) > > > > +;; Define glibc-for-fhs (with a name that allows grafts for glibc), a = variation > > +;; of glibc which uses the default ld.so.cache, useful in FHS containe= rs. > > +;; Note: should this be hidden? > > ^ This should definitely be hidden since it's for internal consumption > only. Also, if grafting ended up not being needed, the variable should > eb renamed 'glibc-for-fhs', an mentions of grafting in the comments > removed. > I'll respond to the grafting comment more below, but for now I just wanted = to have the options available in this "preview" patch. I'm also wondering i= f it is worthwhile to leave the grafting option open for a user (so they ca= n still explicitly do a --with-graft), though I guess we can change in the = future if a bug report comes up. And for selfish reasons I wanted the graft= ing code somewhere so I wouldn't forget how to add that option in the conta= iner code behind the scenes :) > > + ;; Set up an FHS container. > > I feel like FHS should be spelled in full somewhere (in the doc would be > nice). > Agreed, along with explicit documentation as you noted in your followup ema= il. I would want to link to the spec in the manual for reference as well. > > + (when fhs-container? > > + ;; Set up the expected bin and library directories as symlinks to > > + ;; the profile lib directory. Note that this is assuming a 64bit > > + ;; architecture. > > + (let ((lib-dir (string-append profile "/lib"))) > > + (symlink lib-dir "/lib64") > > I think /lib64 is not needed (see below). > Right. Everything was all linked so I had just picked one. As you detail be= low, this should be cleaned up to be the generic standard specified for FHS= . > > + (symlink lib-dir "/lib") > > + (mkdir-p "/usr") > > + (symlink lib-dir "/usr/lib")) > > + ;; Note: can't symlink full /bin in the container due to the sh > > + ;; symlink. > > Can you explain more exactly why symlinks prevent this to work? I think > I've had issues like this before (e.g., #46782), but couldn't really > pinpoint the exact nature of the limitation. > Oh, I didn't mean anything complicated, just that you can't link /bin since= a symlink /bin/sh was already created for containers. Instead, we could mk= dir /bin and then add the sh symlink along with everything from profile/bin= with a little loop. I just didn't think to do it at the time for my first = pass at this. I think for the final version it would be good to do this, as well as loopi= ng through any profile/lib/ subdirectories to add to ld.so.conf. I know at = least our libnss3 is in a subdirectory for some reason, which needs to be a= dded to where ldconfig looks or else things that need libnss3 won't find it= . > > + (symlink (string-append profile "/bin") "/usr/bin") > > + (symlink (string-append profile "/sbin") "/sbin") > > + (symlink (string-append profile "/sbin") "/usr/sbin") > > Other symbolic links I think we should include: > > * /include -> /usr/include (section 4.3 of the FHS 3.0 spec) > * /libexec -> /usr/libexec (optional, section 4.7) > * /share -> /usr/share (section 4.11) > Thanks, good idea. Surprisingly (or not?) the /lib and ldcache seem to be t= he big one. I guess env variables (search-paths) take care of most everythi= ng else, but we should set it all up. > > + ;; Provide a frequently expected 'cc' symlink to gcc, though this > > + ;; could also be done by the user in the container, e.g. in > > + ;; $HOME/.local/bin and adding that to $PATH. Note: we do this > > + ;; in /bin since that already has the sh symlink and can't write > > + ;; to the other bin directories that are already symlinks themselves. > > + (symlink (string-append profile "/bin/gcc") "/bin/cc") > > + ;; TODO: python may also be expected to symlink to python3. > > No need for a TODO, I think users wanting that could simply use > 'python-wrapper', which exists for this purpose, no? > TIL. Yes, perfect. Is there any use to have a similar one for cc/gcc? Would that be useful out= side of this container? I know we frequently need to specify $CC in build p= atching, I just haven't done much building outside of packaging to know oth= erwise. > > + ;; Guix's ldconfig doesn't seem to search in FHS default > > + ;; locations, so provide a minimal ld.so.conf. > > + ;; TODO: this may need more, e.g. libnss3 is in /lib/nss > > + (call-with-output-file "/tmp/ld.so.conf" > > + (lambda (port) > > + (display "/lib64" port) > > + (newline port))) > > Shouldn't this be '/lib' rather than '/lib64'? Per sections 3.9 and > 3.10 of the FHS 3.0 [0], '/lib' is essential, while '/lib' is > optional. > > [0] https://refspecs.linuxfoundation.org/FHS_3.0/fhs-3.0.pdf > Yes, thanks. I should make this all more generic per the standard. > > + (launch-environment (if fhs-container? > > + ;; Use the FHS start script. > > + ;; FIXME: probably the default command should > > + ;; be different as it spawns a different shell? > > + '("/bin/sh" "/tmp/fhs.sh") > > I'm not sure what could be improved here (why the FIXME?); could you > explain what you had on mind in more details? > The shell that is launched looks more like a plain 'sh' rather than what yo= u normally get in an empty guix shell --container. So what I currently have= is a call to /bin/sh /tmp/fhs.sh where the last line of the fhs.sh script = is the command specified by the user, or else /bin/sh itself. This results = in a different prompt at least. I may have gotten some warning after exitin= g the container as well (didn't find /bin/sh? I don't have it in front of m= e). I'll take a look, but open to how best to spawn the shell from this fhs= .sh script as the fallback when no command is given. > > + ;; For an FHS-container, add a glibc that uses > > + ;; /etc/ld.so.cache. > > + (if fhs-container? > > + (alist-cons 'package '(ad-hoc-package "gcfhs") > > + opts) > > + ;; Alternatively, could graft all packages with > > + ;; this glibc, though that seems unnecessary. > > + ;; (alist-cons 'with-graft "glibc=3Dgcfhs" opts) > > + opts))) > > It's interesting that it's unnecessary. If we're confident it really > is, then I'd remove this dead code and comment. > It is interesting, and at first I was having problems with linkers finding = libgcc. It wasn't appearing in the ld cache, and I thought it was because t= he non-FHS libc was referenced in some gcc package (which then wasn't in th= e container). But I think that was a red herring and may have been somethin= g else I was doing while testing, since I haven't hit that problem since. A= nd I don't think that should have caused a problem, as long as whatever lib= c is referenced is available, libgcc should have appeared in the cache. But= I'm not sure now what was happening, so I had tried grafting, came back to= trying without grafts and it still worked... In any event, some testing would be helpful. Just including the FHS libc is= very simple and quick rather than grafting. I think the main point is just= that software/code tends to rely on the ld cache to find libraries, so the= ld loader needs to do this for "default" behavior. After finding the libra= ries, everything else goes fine and is then linked properly in the binary. = (Now I wonder if you build a binary in the FHS container if it will just ru= n in regular Guix since everything should be linked to explicit store locat= ions in the end...?) > > That's surprisingly little code, which is nice! Thanks for looking into > it :-). > Thanks for the helpful comments! And credit to work that is out there that = came before. Admittedly it was much easier this time (just one late evening= ) than when I first tried all of this for some binaries. The end result is = really just needing the ld cache (generation/loading) and some symlinks, no= t bad at all. > I've yet to try it, but I will shortly. > > Thank you, > > Maxim Most welcome! And yes, some testing would be great to find what assumptions= are here or should be here to make this useable without much extra fussing= for each piece of software. John