From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <guix-devel-bounces+larch=yhetil.org@gnu.org>
Received: from mp11.migadu.com ([2001:41d0:2:bcc0::])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
	by ms5.migadu.com with LMTPS
	id 0HQAOF9ZMWMUMgAAbAwnHQ
	(envelope-from <guix-devel-bounces+larch=yhetil.org@gnu.org>)
	for <larch@yhetil.org>; Mon, 26 Sep 2022 09:48:47 +0200
Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
	by mp11.migadu.com with LMTPS
	id MKYKOF9ZMWMpQgAA9RJhRA
	(envelope-from <guix-devel-bounces+larch=yhetil.org@gnu.org>)
	for <larch@yhetil.org>; Mon, 26 Sep 2022 09:48:47 +0200
Received: from lists.gnu.org (lists.gnu.org [209.51.188.17])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by aspmx1.migadu.com (Postfix) with ESMTPS id 674C5158F6
	for <larch@yhetil.org>; Mon, 26 Sep 2022 09:48:47 +0200 (CEST)
Received: from localhost ([::1]:44128 helo=lists1p.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.90_1)
	(envelope-from <guix-devel-bounces+larch=yhetil.org@gnu.org>)
	id 1ociCQ-0004yy-PL
	for larch@yhetil.org; Mon, 26 Sep 2022 03:06:42 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:47980)
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <philip@philipmcgrath.com>)
 id 1ociAn-0004ym-TK
 for guix-devel@gnu.org; Mon, 26 Sep 2022 03:05:02 -0400
Received: from out3-smtp.messagingengine.com ([66.111.4.27]:42389)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <philip@philipmcgrath.com>)
 id 1ociAk-000365-Qj
 for guix-devel@gnu.org; Mon, 26 Sep 2022 03:05:01 -0400
Received: from compute4.internal (compute4.nyi.internal [10.202.2.44])
 by mailout.nyi.internal (Postfix) with ESMTP id ADE885C0114;
 Mon, 26 Sep 2022 03:04:56 -0400 (EDT)
Received: from mailfrontend1 ([10.202.2.162])
 by compute4.internal (MEProxy); Mon, 26 Sep 2022 03:04:56 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 philipmcgrath.com; h=cc:cc:content-transfer-encoding
 :content-type:date:date:from:from:in-reply-to:in-reply-to
 :message-id:mime-version:references:reply-to:sender:subject
 :subject:to:to; s=fm2; t=1664175896; x=1664262296; bh=zOduf3RLuv
 ghW8bYZ/YlODafej0MrGgKkSJ+zcnXUxM=; b=F8eyniRtKcmSbt204/0wjvZIKP
 N+MlcvFu+B98JyhlHFPWJZ/Vyoh9WLuT0C7KaD3fP3rTl5ULr9ilh+qTvHpLmlxc
 aMJyJvQtKJP7fGrez6enX5oqCtaMPsBR0aH5pD3Pl8HPLjUUBZW1PTvhXbZ1eA5L
 VKKgs1ZZ2bOy50YKJS5FerJprETYAP3ivXDMY+3crxzeuZcmNkE+dlZfXSRo0880
 l649duZa242MKsV1FME1HNz1Lu/u0H3g7VeQ2FMP1SZ69N6ItQf367P2jUCbOOlg
 WdJC6vjMQ3egnYnLJQxiNWmH3V44GQuh356fJI+hyG2hX/+mWnVfG6b7yNdQ==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:cc:content-transfer-encoding
 :content-type:date:date:feedback-id:feedback-id:from:from
 :in-reply-to:in-reply-to:message-id:mime-version:references
 :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy
 :x-me-sender:x-me-sender:x-sasl-enc; s=fm2; t=1664175896; x=
 1664262296; bh=zOduf3RLuvghW8bYZ/YlODafej0MrGgKkSJ+zcnXUxM=; b=z
 aLGFMPDk4W8U66q04+117ohjJtEkoba/XdT2AkkUHr+xXGV85mWLOdKP2ck8rTHu
 Xn2AvW4bzF1t4LmY5xunuDbQogeTEkqBcf3muSnsjFR656NAQbvbgWpR//nepzfS
 +ZHYzvyQ0b8vPJoDAmqnBiYlj20YGgNO/eUkX3eFRKu5XMVPo0f/I+OD3lc2pkmm
 rYhDA8ti7sOfjBskdL+FEdLuwmhKvBqmKHTn27Qj2FL9mMQIj37NtcnvGA26ozob
 rB5JCQ5IGbU1r/UyILKLs8RRKT5DrnAj1btH34AmtH3Dt7eL1prDdTICOkO9197W
 gFhhcFX97ByLVyUsCJT+g==
X-ME-Sender: <xms:GE8xY-IUZyBYTI0wQjBeRkOymrz1PRC0FdLS2oVxtuxZ944lCAQ7qw>
 <xme:GE8xY2IM9eyUukdLSlj4NfWRH9yBZrBCtyBajnyPpNF2XOhQ1csM50uZpVZ5UuvIv
 MqqlGcvO7aBS8X8LNw>
X-ME-Received: <xmr:GE8xY-upLmsy303eyyvywEsNtBkOY9WtRX_tSNmaVbD9hX3aAWbyHOJQb0ma2WD9v0arwDxogL0vN-J7JDQZGSD4Ac3e0ijVbbnkAg>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvfedrfeeguddgudeggecutefuodetggdotefrod
 ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh
 necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd
 enucfjughrpefkffggfgfhuffvvehfjggtgfesthekredttdefjeenucfhrhhomheprfhh
 ihhlihhpucfotgfirhgrthhhuceophhhihhlihhpsehphhhilhhiphhmtghgrhgrthhhrd
 gtohhmqeenucggtffrrghtthgvrhhnpeffkeehledtledufefhudeltdejleektdduvdff
 udetheeiieegueeuueduieffvdenucffohhmrghinhepohhpvghnghhrohhuphdrohhrgh
 enucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehphhhi
 lhhiphesphhhihhlihhpmhgtghhrrghthhdrtghomh
X-ME-Proxy: <xmx:GE8xYzbnDkPzDd7OPtgdKnZykfRyHFhJPUZ0hKLaUZhFZ44CaCed3Q>
 <xmx:GE8xY1a6HidnXaFPV4JliqH-iSPuPH-8zSZxaPSGRdeG6XEaEBq78g>
 <xmx:GE8xY_A2rR9r0FLL9ZVaXTEe7bwsJsA3RVtruc0vP5XdvkxIZMojuQ>
 <xmx:GE8xY6N2K-o4rwbbLV6t1Nd3ACD0PqBDc_pNRtUg5cM2fugfFT4Fgg>
Feedback-ID: i2b1146f3:Fastmail
Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon,
 26 Sep 2022 03:04:55 -0400 (EDT)
Message-ID: <c77defd0-2db3-6f07-883c-c7fd808d5a56@philipmcgrath.com>
Date: Mon, 26 Sep 2022 03:04:54 -0400
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.3.0
From: Philip McGrath <philip@philipmcgrath.com>
Subject: Re: What 'sh' should 'system' use?
To: Maxime Devos <maximedevos@telenet.be>, guix <guix-devel@gnu.org>
Cc: Liliana Marie Prikler <liliana.prikler@gmail.com>,
 Liliana Marie Prikler <liliana.prikler@ist.tugraz.at>
References: <2284386.8hzESeGDPO@bastet>
 <84acf321-e5b8-a138-0fc4-14e4e849f774@telenet.be>
Content-Language: en-US
In-Reply-To: <84acf321-e5b8-a138-0fc4-14e4e849f774@telenet.be>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Received-SPF: pass client-ip=66.111.4.27;
 envelope-from=philip@philipmcgrath.com; helo=out3-smtp.messagingengine.com
X-Spam_score_int: -65
X-Spam_score: -6.6
X-Spam_bar: ------
X-Spam_report: (-6.6 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, NICE_REPLY_A=-3.766,
 RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_PASS=-0.001,
 SPF_PASS=-0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-BeenThere: guix-devel@gnu.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: "Development of GNU Guix and the GNU System distribution."
 <guix-devel.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/guix-devel>,
 <mailto:guix-devel-request@gnu.org?subject=unsubscribe>
List-Archive: <https://lists.gnu.org/archive/html/guix-devel>
List-Post: <mailto:guix-devel@gnu.org>
List-Help: <mailto:guix-devel-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/guix-devel>,
 <mailto:guix-devel-request@gnu.org?subject=subscribe>
Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org
Sender: "Guix-devel" <guix-devel-bounces+larch=yhetil.org@gnu.org>
X-Migadu-Flow: FLOW_IN
X-Migadu-To: larch@yhetil.org
X-Migadu-Country: US
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org;
	s=key1; t=1664178527;
	h=from:from:sender:sender:reply-to:subject:subject:date:date:
	 message-id:message-id:to:to:cc:cc:mime-version:mime-version:
	 content-type:content-type:
	 content-transfer-encoding:content-transfer-encoding:
	 in-reply-to:in-reply-to:references:references:list-id:list-help:
	 list-unsubscribe:list-subscribe:list-post:dkim-signature;
	bh=zOduf3RLuvghW8bYZ/YlODafej0MrGgKkSJ+zcnXUxM=;
	b=ZuMpMozVd+AyMEbtAUKrCNmh9MkPL/5kvr7PkmGaLTyqC4VGfCleSQkv03ytygR6wgMcx3
	s/NttWVEcx6wjTUYqra5LhfOpGGyg9qrikKX+suC8TOrkPz+3Ssny4dw/zxouJMr9CQIdB
	34HxPYkovbi/REqKemXR65U1wvgGydgk/cAbK10tXr+m6F1tz3j3UN3YynKOrr2+OehMqe
	+O0i01h7YjvIghw8si7jvodBHgx8zOP9PZHr3gnCb6ejz5QNzGD/Vsp5eXIX/myEm4GXYj
	irXr+jjITjxiRphJ/7QJe3Rem607Lon/E/MiGdNzEUzgUp3fiidi2/BDRQ7MyQ==
ARC-Seal: i=1; s=key1; d=yhetil.org; t=1664178527; a=rsa-sha256; cv=none;
	b=MfzuD9lYQ/7gMCZd5lvbI1FFqmstIXz+sPTuskgMXoNGpyAlzorVedFVMsWL/0W94nMY3o
	OGlqK/Drvq1yK9ktZa9UQ/6+uqj2oi+vykT8fY1eQAmg4NCnIstsiO229t7IJ5vi+Pw8AI
	MW3Y0mq33yeTzSgUOAzAXj3nB1+e1cS6h3LaW0e4cn2F7OwcKqFl187E1Bnvttv4f3wr/o
	hPshtcS+UbqElFEt2GWmaqAr9o0k+OIVeVADVqBjYzG22c0CFVOxzKMNxTQgrIokDWEZ28
	Pu2Rt9x2l7QackYtB6eXGsYVYfeHxmmD2wmeGDVvLptdOr8rjpQ3xtj0WHNJYA==
ARC-Authentication-Results: i=1;
	aspmx1.migadu.com;
	dkim=fail ("headers rsa verify failed") header.d=philipmcgrath.com header.s=fm2 header.b=F8eyniRt;
	dkim=fail ("headers rsa verify failed") header.d=messagingengine.com header.s=fm2 header.b="z aLGFMP";
	dmarc=none;
	spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"
X-Migadu-Spam-Score: 4.06
Authentication-Results: aspmx1.migadu.com;
	dkim=fail ("headers rsa verify failed") header.d=philipmcgrath.com header.s=fm2 header.b=F8eyniRt;
	dkim=fail ("headers rsa verify failed") header.d=messagingengine.com header.s=fm2 header.b="z aLGFMP";
	dmarc=none;
	spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"
X-Migadu-Queue-Id: 674C5158F6
X-Spam-Score: 4.06
X-Migadu-Scanner: scn0.migadu.com
X-TUID: 5MXqxP+HE2Up

Hi,


On 9/19/22 08:55, Maxime Devos wrote:
> (4) Stop using 'system' in applications -- instead use whatever the 
> language's equivalent of Guile's system*, execl ... or Guix' 
> 'invoke'. Why?  Because 'system'-like functions requires quoting the
>  command line arguments whereas in 'system*'-like functions you could
>  just pass a list of command line arguments, and it's easy to get the
>  quoting wrong, especially if some of the arguments are generated 
> dynamically.
> 
> As a bonus, this could often remove a dependency on 
> bash{-minimal,-static,}.
> 

I definitely advocate 'system*'-like functions in general. Still,
'system'-like functions exist: I'm advocating that Guix should should
have a consistent answer for how such functions should behave.

(Very occasionally, a program really does want to invoke the shell, such
as when shell expansion is part of an existing API.)

 From a different perspective, this is part of why I've recently been
thinking we should find 'sh' dynamically: most programs/environments
don't, and shouldn't, need bash{-minimal,-static}, so it seems wrong to
make it a mandatory dependency of libc.

On 9/19/22 08:55, Maxime Devos wrote:
> 
> On 19-09-2022 02:13, Philip McGrath wrote:
>> 1) If we want to continue to hard-code a specific shell into 
>> Glibc,
> 
> We do, for reproducibility -- otherwise, the behaviour of the 
> 'system' function depends on whatever is the current /bin/sh, and 
> sometimes /bin/sh is updated (and on some foreign systems it might 
> not even be the bash we are used to).
> 
> [...]
> 
>> 
>> 2) If we want to make 'sh' a weak/dynamic reference, I think we 
>> should strongly consider arranging to make it available at 
>> '/bin/sh' when present. I expect this option would require less 
>> patching of other packages*by far*  than any other approach.
> 
> See (1) (reproducibility) -- also, you would need to modify the 
> daemon for that, so there are compatibility concerns, and then we're
>  stuck with the /bin/sh special case forever (unless breaking 
> compatibility would later be considered acceptable).
> 

I don't think there's a reproducibility problem. Guix already can create
reproducible containers with "/bin/sh" (e.g. 'guix shell coreutils
--container -- ls /bin') and without "/bin/sh" (as in package build
environments).

I haven't investigated whether adding the ability to create "/bin/sh" in
build containers would require modifying the daemon or just sending the
daemon different instructions. However, AIUI, Nix *always* creates
"/bin/sh" in build containers, which makes me further expect that any
change needed to the daemon would be small.

To be clear, I'm not proposing that we always create "/bin/sh" in build
containers. At a low level, I'm suggesting that we add the ability to
create "/bin/sh" when desired. I can imagine one possibility for a
high-level interface would be to create "/bin/sh" by default when an
input provides "bin/sh", and it might turn out that we end up wanting
"/bin/sh" in most or all build containers in practice, but I see those
as secondary questions.

>> and recommendations for how packages should use it: '_PATH_BSHELL' 
>> is the best mechanism I've heard of so far, though I wish it were 
>> standardized, and the fact that it can't be portably assumed to be
>> a string constant could be surprising.
> 
> I consider _not_ using it, and using (4) instead, to be best. If not
>  suitable (for example, because a shell is needed to run an actual 
> shell script), then a plain "sh" looked up in the $PATH (like other 
> binaries) and substitute*-ed by Guix should suffice.
> 

As I said above, I agree that 'system*' should be preferred over
'system' when possible.

There are a few dimensions here that I want to try to pick apart.

When you say:

> a plain "sh" looked up in the $PATH (like other binaries) and 
> substitute*-ed by Guix should suffice

there are a few different things that might mean.

I think you're probably referring to the status quo, where "sh" is
looked up in the 'inputs' or a G-expression equivalent and an absolute
reference to that particular "sh" is embedded into the package being
built. (But, when cross-compiling, that "sh" would not be in the $PATH
in the build environment.)

There's a different question about $PATH vs. _CS_PATH that I'll address
later.

I see at least two reasons to prefer finding "sh" dynamically at run-time.

First, we have other POSIX-like shells packaged in Guix, such as dash,
zsh, and gash. Currently, to create an environment where one of these
shells is used to run 'system'-like functions (e.g. because dash is
supposed to be faster than bash), you would have to recompile everything
that depends on glibc. (Maybe you could craft a very ugly graft.)

Second, sometimes people may want to create environments, images, etc.
without an "sh" available. In some sense this is a special case of using
an alternate shell, but the consequences of the status quo are
especially notable. Currently, practically any environment or image Guix
creates will include bash-static, because glibc refers to it.

For an especially ironic example, consider this note from `info
"(guix)Invoking guix pack"`:


> Note: Singularity _requires_ you to provide ‘/bin/sh’ in the image.
> For that reason, ‘guix pack -f squashfs’ always implies ‘-S
> /bin=bin’.  Thus, your ‘guix pack’ invocation must always start with
> something like:
> 
>     guix pack -f squashfs bash ...
> 
> If you forget the ‘bash’ (or similar) package, ‘singularity run’ and
> ‘singularity exec’ will fail with an unhelpful “no such file or
> directory” message.

Running `guix pack -f squashfs hello` warns you about the lack of a
shell, and indeed the resulting image doesn't contain "/bin/sh" ... but
it does contain
"/gnu/store/720rj90bch716isd8z7lcwrnvz28ap4y-bash-static-5.1.8/bin/sh"!

Furthermore, if you run `guix pack -f squashfs hello bash-static`, the
resulting image contains both
"/gnu/store/720rj90bch716isd8z7lcwrnvz28ap4y-bash-static-5.1.8/bin/sh"
and "/gnu/store/4f304c7dp68hkcp1zi1i07zm8nfvvyp7-bash-static-5.1.8/bin/sh".

> 
>> 
>> 3) If we want a dynamic 'sh' not located at '/bin/sh', I think we 
>> should implement a function similar to '__bionic_get_shell_path()'
>>  and use it for '_PATH_BSHELL', 'system', etc. That begs the 
>> question of how the function should find 'sh', and I don't have an
>>  answer for that.
> 
> How about $PATH?
> 

This is a subtle point, and it depends in some ways on what you are
trying to use the 'sh' for. From the "Rationale" in the POSIX spec for
'confstr' 
<https://pubs.opengroup.org/onlinepubs/9699919799/functions/confstr.html>:

> The original need for this function was to provide a way of finding
> the configuration-defined default value for the environment variable
> PATH. Since PATH can be modified by the user to include directories
> that could contain utilities replacing the standard utilities in the
> Shell and Utilities volume of POSIX.1-2017, applications need a way
> to determine the system-supplied PATH environment variable value that
> contains the correct search path for the standard utilities.

I don't have a strong view about the merits of using PATH or not in 
general, and, again 'confstr' with '_CS_PATH' doesn't currently give a 
useful result on Guix.

For 'sh' specifically, though, there's some set of programs that look at 
$SHELL or /etc/passwd or other mechanisms for a highly-configurable 
choice of shell: those aren't relevant here. This question concerns a 
different set of programs that are looking for a reliable plain-vanilla 
'sh': this may be configured at the level of the environment (OS, 
container, chroot, etc.)---including configuring it not to exist---but 
it's a less fine-grained sort of configuration, and there's a stronger 
expectation that it will be a POSIX-like 'sh' (not fish or 
/usr/sbin/nologin).

It seems POSIX would like 'sh' to be found using '_CS_PATH', but I don't 
know of any programs that actually do that, and it doesn't work on Guix.

Programs in practice seem to look at "/bin/sh", and environments 
configuring it by choosing what (possibly nothing) to put at "/bin/sh" 
from the perspective of programs in that environment.

>> I think we should document the decision (for example, why 
>> 'bash-static' vs. 'bash- minimal'?)
> 
> Because cycles -- bash-minimal is linked to a (shared) glibc, which 
> is a separate package from bash-minimal, so glibc cannot use 
> bash-minimal, it uses bash-static instead which is linked to a 
> (static) glibc (which might use a bootstrap bash (not 100% sure), but
> it's statically linked, so no reference to the bootstrap bash remains
> IIUC).
> 
> Also, why?  This is an implementation detail.  Who would the target 
> audience be for this documentation?
> 

I don't mean "document the decision" to necessarily imply something 
elaborate or formal, but I think the next person packaging a language 
with a function like 'system' in its standard library shouldn't have to 
reevaluate these questions from scratch. Also, if we decided the right 
thing were to advocate for upstreams to do something differently for the 
sake of portability (e.g. trying to get people to use _CS_PATH---which 
I'm not suggesting), it would help to have a rationale to point to.

Specifically with respect to bash-minimal vs. bash-static, I'm still not 
clear on when I should use which. The package descriptions are 
identical, and I haven't found a clear (to me, at least) explanation in 
the source code comments. For example, if bash-static is needed to avoid 
a cycle as you say, what is the benefit of also having bash-minimal?

-Philip