How to add /bin/sh to the build environment

How to add /bin/sh to the build environment

[Foo Chuan Wei]

I am trying to package the Standard ML of New Jersey (SML/NJ) compiler
in Guix. My current understanding is that its installation process
relies on the existence of /bin/sh in a fundamental way. Is there a way
to add /bin/sh to the build environment?

In the build phase of the package definition, I tried something like
this:

    (mkdir-p "/bin")
    (symlink (which "sh") "/bin/sh")

However, `mkdir-p` always fails with:

    ice-9/boot-9.scm:1669:16: In procedure raise-exception:
    In procedure mkdir: Permission denied

Re: How to add /bin/sh to the build environment

[Xinglu Chen]

[-- Attachment #1: Type: text/plain, Size: 767 bytes --]

Hi,

On Mon, Nov 08 2021, Foo Chuan Wei wrote:

> I am trying to package the Standard ML of New Jersey (SML/NJ) compiler
> in Guix. My current understanding is that its installation process
> relies on the existence of /bin/sh in a fundamental way. Is there a way
> to add /bin/sh to the build environment?
>
> In the build phase of the package definition, I tried something like
> this:
>
>     (mkdir-p "/bin")
>     (symlink (which "sh") "/bin/sh")

I would use ‘substitute*’ from (guix build utils) to patch the
references to /bin/sh

  (lambda* (#:key inputs #:allow-other-keys)
    (let ((sh (assoc-ref inputs "bash")))
      (substitute* "some-file"
       (("\"/bin/sh\"") (string-append "\"" sh "/bin/sh\"")))))

Hope that helps!

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 861 bytes --]

Re: How to add /bin/sh to the build environment

[Foo Chuan Wei]

On 2021-11-08 17:54 +0100, Xinglu Chen wrote:
> I would use ‘substitute*’ from (guix build utils) to patch the
> references to /bin/sh
> 
>   (lambda* (#:key inputs #:allow-other-keys)
>     (let ((sh (assoc-ref inputs "bash")))
>       (substitute* "some-file"
>        (("\"/bin/sh\"") (string-append "\"" sh "/bin/sh\"")))))
> 
> Hope that helps!

Thank you for the suggestion, but this will not work for packaging
SML/NJ. The build process depends on the existence of /bin/sh in a
fundamental way:

* Part of the build process relies on programs written in Standard ML.
* The part written in Standard ML uses a function from the SML Basis
  Library: `OS.Process.system`.
* `OS.Process.system` will always use /bin/sh. This is hardcoded. I
 cannot change it. If I change it, the behavior of `OS.Process.system`
 would probably be wrong in the final product (the sml compiler).

I'm not even sure if I can use `substitute*`, since the build process
relies on binary "bootfiles" where /bin/sh is hardcoded.

Note that I've managed to work around the problem.
Before building:
* Create a symlink /tmp/sh that points to `(which "sh")`.
* Edit the binary bootfiles so that they use /tmp/sh instead of /bin/sh.

After building:
* Edit the resulting binaries so that they use /bin/sh instead of /tmp/sh.

Is there really no way to create /bin/sh in the build environment, so
that binary patching can be avoided?

Re: How to add /bin/sh to the build environment

[Leo Famulari]

As you found, it's not possible to write to the root directory of the
build environment as the build user.

I don't remember exactly, but I believe this was done as part of the fix
for CVE-2021-27851 [0].

Of course it can be annoying that /bin/sh cannot be added to the build
environment, but specifying a particular shell is an important part of
how Guix achieves the functional packaging model.

Relying on /bin/sh while building *could* be done properly with some
care, but since its use could indicate that the package will use /bin/sh
at run-time, we choose to omit it from the build environment, although
it does exist on Guix System and most systems that Guix supports.

On Mon, Nov 08, 2021 at 08:19:27PM +0000, Foo Chuan Wei wrote:
> * `OS.Process.system` will always use /bin/sh. This is hardcoded. I
>  cannot change it. If I change it, the behavior of `OS.Process.system`
>  would probably be wrong in the final product (the sml compiler).
> 
> I'm not even sure if I can use `substitute*`, since the build process
> relies on binary "bootfiles" where /bin/sh is hardcoded.

It will be necessary to fix this issue in the SML/NJ package definition
(I think you've done this in a followup patch), because Guix does not
accept packages that rely on pre-built binaries — everything must be
built from source code. There are some exceptions but I don't think that
SML/NJ will qualify due to its obscurity compared to other language
implementations. If SML/NJ cannot be built without the binary bootfiles,
it probably won't be eligible for inclusion in GNU Guix.

[0] The CVE ID will appear on the blog post shortly... it took several
weeks to get the ID assigned and I forgot to add it to the blog post
until now.
https://guix.gnu.org/en/blog/2021/risk-of-local-privilege-escalation-via-guix-daemon/

Re: How to add /bin/sh to the build environment

[Philip McGrath]

Hi,

On 11/8/21 15:19, Foo Chuan Wei wrote:
> * Part of the build process relies on programs written in Standard ML.
> * The part written in Standard ML uses a function from the SML Basis
>    Library: `OS.Process.system`.
> * `OS.Process.system` will always use /bin/sh. This is hardcoded. I
>   cannot change it. If I change it, the behavior of `OS.Process.system`
>   would probably be wrong in the final product (the sml compiler).

A similar issue came up for Racket, which has a family of functions like 
`process` and `system` that execute commands using `sh`.[0]

Maybe this kind of function is common enough in programming language 
implementations that it should be documented in the Guix manual's 
packaging guidelines? For example, even libc has a `system` function 
specified to use "/bin/sh" (I haven't looked at how Guix handles it).

Part of the issue with Guix is that, even if you somehow arranged for
"/bin/sh" to exist in the build environment, Guix users will likely also 
want to run the SML/NJ compiler and compiled SML/NJ programs in 
environments where "/bin/sh" doesn't exist. For Racket, thanks to some 
discussion with Jack Hill,[1] I added a patch that uses a Guix-specific 
C preprocessor macro to intercept attempts to execute "/bin/sh" in 
librktio (Racket's cross-platform IO library) and substitute the store 
path to Guix's `bash-minimal`. The current patch is at [2].

(A subtlety I'd forgotten is that, in addition to functions like 
`system` that are specified to use `sh`, Racket's build process was also 
calling functions like `system*`, which are more like `execve`, with 
"/bin/sh" as an argument. Since that approach is recommended for 
portability and robustness, the patch intercepts those attempts, too.)

The approach was especially defensible for Racket because the Racket 
Reference merely specifies that the functions will use some `sh`, not 
that it will be any particular `sh` or will be located at any particular 
path. From a quick look at the SML Basis Library reference[3], it 
specifies that `OS.Process.system` uses "the operating system's default 
shell", though it does go on to say that, "on Unix systems" (but 
apparently not Mac OS?), the default shell is '/bin/sh'". Since Guix 
doesn't really have a "default shell" in general, I think using the an 
explicit package input might make sense for SML/NJ's 
`OS.Process.system`, too.

[0]: 
https://docs.racket-lang.org/reference/subprocess.html#%28def._%28%28lib._racket%2Fsystem..rkt%29._process%29%29
[1]: https://issues.guix.gnu.org/47180
[2]: 
https://git.savannah.gnu.org/gitweb/?p=guix.git;a=blob;f=gnu/packages/patches/racket-minimal-sh-via-rktio.patch;h=6bc2ee8331f046959fd493f8e1604e682ddc4a0d;hb=HEAD
[3]: 
https://smlfamily.github.io/Basis/os-process.html#SIG:OS_PROCESS.system:VAL

Re: How to add /bin/sh to the build environment

[Philip McGrath]

(I wrote my earlier reply before I saw Leo's.)

On 11/8/21 17:23, Leo Famulari wrote:
> It will be necessary to fix this issue in the SML/NJ package definition
> (I think you've done this in a followup patch)

Is this <https://issues.guix.gnu.org/38606>, or somewhere else?

> Guix does not
> accept packages that rely on pre-built binaries — everything must be
> built from source code. There are some exceptions but I don't think that
> SML/NJ will qualify due to its obscurity compared to other language
> implementations. If SML/NJ cannot be built without the binary bootfiles,
> it probably won't be eligible for inclusion in GNU Guix.

Mostly I wanted to follow up to say that, from my perspective, SML/NJ is 
a very noteworthy and influential language implementation. Perhaps the 
most familiar example for Guix folks would be that Guile's fibers are 
based on Concurrent ML, which is an SML/NJ extension: 
https://wingolog.org/archives/2017/06/29/a-new-concurrent-ml

Personally I've only read ML, not written it, but from a Debian bug I 
found through searching online 
(<https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=220046>), it appears 
that there are no known Standard ML implementations that avoid this 
bootstrapping problem. SML/NJ, in addition to being one of the most 
important implementations in its own right, is apparently capable of 
bootstrapping at least one other implementation, MLton. (Unlike some on 
that thread, I don't consider the 5x slowdown for the bootstrap build to 
be prohibitive: bootstrapping Chez Scheme via Racket has a 10x slowdown.)

It is very valuable that Guix maintains such high standards for 
bootstrapping from source. However, Guix does still contain some 
compilers that have bootstrapping limitations. SML/NJ is no worse that 
Chez Scheme in this respect, and comparably important. (One difference 
is that there is a path to backporting the bootstrap simulation from 
Racket's fork to the upstream variant, which would remove the 
problem---but I don't that wasn't true when Chez Scheme was first added 
to Guix.) The status quo is that no Standard ML language implementation 
at all seems to be in Guix. I think including SML/NJ would be better 
than waiting for someone to write a hypothetical Standard ML compiler in 
a host language with a better bootstrapping story.

-Philip

Re: How to add /bin/sh to the build environment

[Foo Chuan Wei]

On 2021-11-08 20:36 -0500, Philip McGrath wrote:
> On 11/8/21 17:23, Leo Famulari wrote:
> > It will be necessary to fix this issue in the SML/NJ package definition
> > (I think you've done this in a followup patch)
>
> Is this <https://issues.guix.gnu.org/38606>, or somewhere else?

On 2021-11-07, I submitted a patch that can be found at
https://issues.guix.gnu.org/38606, but that patch doesn't fix the
/bin/sh issue. It merely avoids the issue by disabling the building of
SML/NJ libraries. `OS.Process.system` is used during the building of
SML/NJ libraries.

I eventually figured out how to build the whole thing (including
libraries), but it involves using `sed` to patch strings in some
binaries. I will submit a non-WIP patch, hopefully within this week.

By the way, should I submit the new patch to
https://issues.guix.gnu.org/38606 (38606@debbugs.gnu.org), or should I
start a new thread by submitting it to guix-patches@gnu.org?

Re: How to add /bin/sh to the build environment

[Leo Famulari]

On Mon, Nov 08, 2021 at 08:36:08PM -0500, Philip McGrath wrote:
> Mostly I wanted to follow up to say that, from my perspective, SML/NJ is a
> very noteworthy and influential language implementation. Perhaps the most
> familiar example for Guix folks would be that Guile's fibers are based on
> Concurrent ML, which is an SML/NJ extension:
> https://wingolog.org/archives/2017/06/29/a-new-concurrent-ml

Thanks for sharing your knowledge. I stand corrected!

> Personally I've only read ML, not written it, but from a Debian bug I found
> through searching online
> (<https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=220046>), it appears
> that there are no known Standard ML implementations that avoid this
> bootstrapping problem. SML/NJ, in addition to being one of the most
> important implementations in its own right, is apparently capable of
> bootstrapping at least one other implementation, MLton. (Unlike some on that
> thread, I don't consider the 5x slowdown for the bootstrap build to be
> prohibitive: bootstrapping Chez Scheme via Racket has a 10x slowdown.)
> 
> It is very valuable that Guix maintains such high standards for
> bootstrapping from source. However, Guix does still contain some compilers
> that have bootstrapping limitations. SML/NJ is no worse that Chez Scheme in
> this respect, and comparably important. (One difference is that there is a
> path to backporting the bootstrap simulation from Racket's fork to the
> upstream variant, which would remove the problem---but I don't that wasn't
> true when Chez Scheme was first added to Guix.) The status quo is that no
> Standard ML language implementation at all seems to be in Guix. I think
> including SML/NJ would be better than waiting for someone to write a
> hypothetical Standard ML compiler in a host language with a better
> bootstrapping story.

I see. Maybe it's worth attempting to patch the binary to use /tmp/sh
while bootstrapping, in that case. Or maybe there *is* some way to add
/bin/sh to the build environment for just this package, without
having to make the change for all packages, which would be undesirable.