thesis: guixsd should provide /usr/bin/env

thesis: guixsd should provide /usr/bin/env

[Andy Wingo]

Hi!

I think GuixSD should follow NixOS's lead and provide /usr/bin/env.  The
reason is to support scripts that can run either on GuixSD or on some
other system.  For example instead of:

  #!/bin/bash

we can do

  #!/usr/bin/env bash

So we just need /usr/bin/env in the root and nothing else.

Alternately, I am not sure if this would work but we could make a form
of "guix environment" which populates a profile that is mounted at /usr
in a container.  That would allow many more non-Guix tools to run.

WDYT?

Andy

Re: thesis: guixsd should provide /usr/bin/env

[Ricardo Wurmus]

Andy Wingo <wingo@igalia.com> writes:

> I think GuixSD should follow NixOS's lead and provide /usr/bin/env.  The
> reason is to support scripts that can run either on GuixSD or on some
> other system.  For example instead of:
>
>   #!/bin/bash
>
> we can do
>
>   #!/usr/bin/env bash
>
> So we just need /usr/bin/env in the root and nothing else.
>
> Alternately, I am not sure if this would work but we could make a form
> of "guix environment" which populates a profile that is mounted at /usr
> in a container.  That would allow many more non-Guix tools to run.
>
> WDYT?

I just did this on GuixSD:

  sudo -E guix package -p /usr -i bash coreutils

Now I have /usr/bin/env and /usr/bin/bash.  I don’t think I needed it
before, though.  “guix environment” is probably not the appropriate tool
for this.

I think it’s better to do this explicitly with profiles than to treat
/usr/bin/env somehow specially.  Providing /usr/bin/env doesn’t
necessarily help — one would still have to have an interpreter installed
in an active profile.

~~ Ricardo

Re: thesis: guixsd should provide /usr/bin/env

[Ludovic Courtès]

Andy Wingo <wingo@igalia.com> skribis:

> I think GuixSD should follow NixOS's lead and provide /usr/bin/env.  The
> reason is to support scripts that can run either on GuixSD or on some
> other system.  For example instead of:
>
>   #!/bin/bash
>
> we can do
>
>   #!/usr/bin/env bash
>
> So we just need /usr/bin/env in the root and nothing else.

In ~8 years of NixOS and then GuixSD, I’ve lived without /usr/bin/env.
I use #!/bin/sh in scripts I write, which is known to be Bash on GuixSD.
But I don’t write so many throwaway scripts, and when I do, it’s either
Guile or Bash.

On IRC, Mark pointed out that, if /usr/bin/env were available by
default, then scripts or generated scripts may more often end up relying
on /usr/bin/env rather that precisely specify what interpreter they
need, which would partly defeat the point of all this.

My current inclination would be to not provide /usr/bin/env by default,
and instead let users add it if they want to, either using the
sledgehammer Ricardo suggests ;-), or simply with:

  ln -s /run/current-system/profile/bin/env /usr/bin/env

We could document it, and/or even add a switch in ‘operating-system’ to
do that.

How does that sound?

> Alternately, I am not sure if this would work but we could make a form
> of "guix environment" which populates a profile that is mounted at /usr
> in a container.  That would allow many more non-Guix tools to run.

Technically ‘guix environment --container’ could create /usr, just like
it creates /bin/sh.  Not sure if it’s a good idea, though.

Thanks,
Ludo’.

Re: thesis: guixsd should provide /usr/bin/env

[Andy Wingo]

On Thu 19 Nov 2015 16:07, ludo@gnu.org (Ludovic Courtès) writes:

> My current inclination would be to not provide /usr/bin/env by default,
> and instead let users add it if they want to, either using the
> sledgehammer Ricardo suggests ;-), or simply with:
>
>   ln -s /run/current-system/profile/bin/env /usr/bin/env
>
> We could document it, and/or even add a switch in ‘operating-system’ to
> do that.
>
> How does that sound?

Fine with me.  My use case is when working on Chromium, which has a
bunch of scripts that are part of its development environment.  Most of
them start with /usr/bin/env, just a couple don't and I think those are
bugs.  In practice though that's something of a hack and what I really
should go for is the --container thing...

>> Alternately, I am not sure if this would work but we could make a form
>> of "guix environment" which populates a profile that is mounted at /usr
>> in a container.  That would allow many more non-Guix tools to run.
>
> Technically ‘guix environment --container’ could create /usr, just like
> it creates /bin/sh.  Not sure if it’s a good idea, though.

I think it is definitely interesting.  The reason being, you might hack
on something or have to deploy something and it's not part of Guix --
you don't want to rewrite the shebang lines for files in git that aren't
build products.  Being able to make a just-FHS-enough environment inside
a container sounds to me like a useful tool to have for shimming Guix
and the outside world, while also benefitting from Guix's reproducible
environments, rollbacks, isolation, and so on.

Andy

Re: thesis: guixsd should provide /usr/bin/env

[Ludovic Courtès]

Andy Wingo <wingo@igalia.com> skribis:

> On Thu 19 Nov 2015 16:07, ludo@gnu.org (Ludovic Courtès) writes:

[...]

>>> Alternately, I am not sure if this would work but we could make a form
>>> of "guix environment" which populates a profile that is mounted at /usr
>>> in a container.  That would allow many more non-Guix tools to run.
>>
>> Technically ‘guix environment --container’ could create /usr, just like
>> it creates /bin/sh.  Not sure if it’s a good idea, though.
>
> I think it is definitely interesting.  The reason being, you might hack
> on something or have to deploy something and it's not part of Guix --
> you don't want to rewrite the shebang lines for files in git that aren't
> build products.  Being able to make a just-FHS-enough environment inside
> a container sounds to me like a useful tool to have for shimming Guix
> and the outside world, while also benefitting from Guix's reproducible
> environments, rollbacks, isolation, and so on.

Yeah, makes sense.

One can already do:

  guix environment --container --ad-hoc coreutils \
    --expose=$(which env)=/usr/bin/env

Should we add a --fhs or --/usr/bin/env option to simplify this?

Thanks,
Ludo’.

Re: thesis: guixsd should provide /usr/bin/env

[Andy Wingo]

On Fri 20 Nov 2015 14:00, ludo@gnu.org (Ludovic Courtès) writes:

> Should we add a --fhs or --/usr/bin/env option to simplify this?

Oh my goodness this would be really nice!  Exposing the profile in /usr
would be quite interesting.

Andy

Re: thesis: guixsd should provide /usr/bin/env

[宋文武]

Someone (sorry, I forget who) mentioned in #guix that we can use
binfmt_misc to handle the shebang.

I tried coding it in C, and it seems to working fine.
Here it is: <https://github.com/iyzsong/duang>.

Try it with care:
--8<---------------cut here---------------start------------->8---
$ sudo mount -t binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc
$ git clone https://github.com/iyzsong/duang.git
$ cd duang
$ make
$ sudo make install
$ sudo make register
--8<---------------cut here---------------end--------------->8---

This will install the handler into `/libexec/@', and register it in
`/proc/sys/fs/binfmt_misc/duang'.  With it activated, intepreter in
the shebang doesn't need to be absolute path, and can handle multiple
arguments, eg: `#!emacs -Q --script'.  And when the interpreter in
absolute path is not exist, it will fallback to be searched in the PATH,
so `#!/usr/bin/perl -w' equals to `#!perl -w'.  On GuixSD, this means
if we remove the `/bin/sh' symlink, the sh-scripts will fallbacked to
use `sh' in PATH, so we can remove /bin.

I dunno if it's a good idea though :-)

Re: thesis: guixsd should provide /usr/bin/env

[Jookia]

On Wed, Feb 17, 2016 at 03:31:42PM +0800, 宋文武 wrote:
> Someone (sorry, I forget who) mentioned in #guix that we can use
> binfmt_misc to handle the shebang.

I might have been me since I thought this would be a neat idea.

> I tried coding it in C, and it seems to working fine.
> Here it is: <https://github.com/iyzsong/duang>.
> 
> Try it with care:
> --8<---------------cut here---------------start------------->8---
> $ sudo mount -t binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc
> $ git clone https://github.com/iyzsong/duang.git
> $ cd duang
> $ make
> $ sudo make install
> $ sudo make register
> --8<---------------cut here---------------end--------------->8---
> 
> This will install the handler into `/libexec/@', and register it in
> `/proc/sys/fs/binfmt_misc/duang'.  With it activated, intepreter in
> the shebang doesn't need to be absolute path, and can handle multiple
> arguments, eg: `#!emacs -Q --script'.  And when the interpreter in
> absolute path is not exist, it will fallback to be searched in the PATH,
> so `#!/usr/bin/perl -w' equals to `#!perl -w'.  On GuixSD, this means
> if we remove the `/bin/sh' symlink, the sh-scripts will fallbacked to
> use `sh' in PATH, so we can remove /bin.
> 
> I dunno if it's a good idea though :-)

Does this handle /usr/bin/env though? ;)

That said, I think it's the wrong approach since it's automated, invisible and
hides the issues of /usr/bin/env. Instead we need to do two things:

Write some scripts to ease migration and documentation on how to use Autotools
or other tools to fix shebangs. This is portable and will work on all systems.
Basically, find all shebangs, replace them with a macro that expands to the
binary, move the file from 'name.sh' to 'name.in.sh' and then use './configure'
or 'make' to build the 'name.sh' scripts with fixed shebangs.

Write a script that patches shebangs in a directory tree and has support for
unpatching them. This is useful for developing or running unpatched third party
trees where you want to change the source code and submit patches without
actually fixing the problem.

Both of these solutions can help destroy /usr/bin/env upstream and downstream.

Jookia.

Re: thesis: guixsd should provide /usr/bin/env

[Ludovic Courtès]

Jookia <166291@gmail.com> skribis:

> That said, I think it's the wrong approach since it's automated, invisible and
> hides the issues of /usr/bin/env. Instead we need to do two things:
>
> Write some scripts to ease migration and documentation on how to use Autotools
> or other tools to fix shebangs. This is portable and will work on all systems.
> Basically, find all shebangs, replace them with a macro that expands to the
> binary, move the file from 'name.sh' to 'name.in.sh' and then use './configure'
> or 'make' to build the 'name.sh' scripts with fixed shebangs.

I think projects already do that (or at least, They Should Be Doing
That.)  That is, either any Bourne shell is fine, and they use
#!/bin/sh, or they require Bash specifically, and they use AC_PATH_PROG
and #!@BASH@ (same for Python, Perl, etc.)

Now, reality is that there’s probably a lot of typically small packages
that use hard-coded shebangs.

> Write a script that patches shebangs in a directory tree and has support for
> unpatching them. This is useful for developing or running unpatched third party
> trees where you want to change the source code and submit patches without
> actually fixing the problem.

I think patching is perfect in our build environment (the
‘patch-shebangs’ and related phases), but probably inconvenient when
working on another project where it may force you to carry a patch
forever.  ISTR that Andy mentioned working on WebKit or Chromium, which
has hard-coded shebangs, and where patching things is annoying at best.

The ‘binfmt_misc’ hack (or other hacks to that effect) can be helpful in
those cases.

Ludo’.

Re: thesis: guixsd should provide /usr/bin/env

[Ludovic Courtès]

iyzsong@member.fsf.org (宋文武) skribis:

> Someone (sorry, I forget who) mentioned in #guix that we can use
> binfmt_misc to handle the shebang.
>
> I tried coding it in C, and it seems to working fine.
> Here it is: <https://github.com/iyzsong/duang>.
>
> Try it with care:
> $ sudo mount -t binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc
> $ git clone https://github.com/iyzsong/duang.git
> $ cd duang
> $ make
> $ sudo make install
> $ sudo make register
>
> This will install the handler into `/libexec/@', and register it in
> `/proc/sys/fs/binfmt_misc/duang'.  With it activated, intepreter in
> the shebang doesn't need to be absolute path, and can handle multiple
> arguments, eg: `#!emacs -Q --script'.  And when the interpreter in
> absolute path is not exist, it will fallback to be searched in the PATH,
> so `#!/usr/bin/perl -w' equals to `#!perl -w'.  On GuixSD, this means
> if we remove the `/bin/sh' symlink, the sh-scripts will fallbacked to
> use `sh' in PATH, so we can remove /bin.
>
> I dunno if it's a good idea though :-)

Nice hack!  :-)

The downside is that it requires scripts to use this feature, and
requires users to have Duang installed and registered.

I think what would be nice to handle the use case that Andy raised is a
binfmt_misc handler that simply handles /usr/bin/env (and possibly
/bin/sh), and to add an ‘operating-system’ knob to enable it.  That way,
third-party scripts would work automagically.

Thanks,
Ludo’.

Re: thesis: guixsd should provide /usr/bin/env

[Christopher Allan Webber]

Ludovic Courtès writes:

> Andy Wingo <wingo@igalia.com> skribis:
>
>> On Thu 19 Nov 2015 16:07, ludo@gnu.org (Ludovic Courtès) writes:
>
> [...]
>
>>>> Alternately, I am not sure if this would work but we could make a form
>>>> of "guix environment" which populates a profile that is mounted at /usr
>>>> in a container.  That would allow many more non-Guix tools to run.
>>>
>>> Technically ‘guix environment --container’ could create /usr, just like
>>> it creates /bin/sh.  Not sure if it’s a good idea, though.
>>
>> I think it is definitely interesting.  The reason being, you might hack
>> on something or have to deploy something and it's not part of Guix --
>> you don't want to rewrite the shebang lines for files in git that aren't
>> build products.  Being able to make a just-FHS-enough environment inside
>> a container sounds to me like a useful tool to have for shimming Guix
>> and the outside world, while also benefitting from Guix's reproducible
>> environments, rollbacks, isolation, and so on.
>
> Yeah, makes sense.
>
> One can already do:
>
>   guix environment --container --ad-hoc coreutils \
>     --expose=$(which env)=/usr/bin/env
>
> Should we add a --fhs or --/usr/bin/env option to simplify this?
>
> Thanks,
> Ludo’.

Would this possibly screw us up on Hurd, where container support I don't
think exists?

Re: thesis: guixsd should provide /usr/bin/env

[Andy Wingo]

Hi :)

> Ludovic Courtès writes:
>
>> Should we add a --fhs or --/usr/bin/env option to simplify this?

I think /usr/bin/env is probably going to be insufficient -- lazy people
like me are always going to be asking for more things :)  If we can
manage to expose a profile at /usr, that would be better, I think.

On Mon 29 Feb 2016 19:11, Christopher Allan Webber <cwebber@dustycloud.org> writes:

> Would this possibly screw us up on Hurd, where container support I don't
> think exists?

In general, I think we can have the Hurd inform but not gate Guix
features.

In this case, exposing a custom view of the file system to a sub-process
is something that the Hurd should be able to do anyway :)

Andy

Re: thesis: guixsd should provide /usr/bin/env

[Jookia]

On Tue, Mar 01, 2016 at 09:09:45AM +0100, Andy Wingo wrote:
> Hi :)
> 
> > Ludovic Courtès writes:
> >
> >> Should we add a --fhs or --/usr/bin/env option to simplify this?
> 
> I think /usr/bin/env is probably going to be insufficient -- lazy people
> like me are always going to be asking for more things :)  If we can
> manage to expose a profile at /usr, that would be better, I think.
> 
> On Mon 29 Feb 2016 19:11, Christopher Allan Webber <cwebber@dustycloud.org> writes:
> 
> > Would this possibly screw us up on Hurd, where container support I don't
> > think exists?
> 
> In general, I think we can have the Hurd inform but not gate Guix
> features.
> 
> In this case, exposing a custom view of the file system to a sub-process
> is something that the Hurd should be able to do anyway :)

We can somewhat do this with containers already, perhaps it'd be best to have
hacks that run in containers?

> Andy

Re: thesis: guixsd should provide /usr/bin/env

[Ludovic Courtès]

Christopher Allan Webber <cwebber@dustycloud.org> skribis:

> Ludovic Courtès writes:

[...]

>> One can already do:
>>
>>   guix environment --container --ad-hoc coreutils \
>>     --expose=$(which env)=/usr/bin/env
>>
>> Should we add a --fhs or --/usr/bin/env option to simplify this?
>>
>> Thanks,
>> Ludo’.
>
> Would this possibly screw us up on Hurd, where container support I don't
> think exists?

‘call-with-container’ currently doesn’t work on GNU/Hurd.  So whatever
we do, there’s no regression.  :-)

But I agree with Andy: I think it’s OK to add optional features that
require the Linux kernel.

Besides, ‘call-with-container’ can definitely be ported to GNU/Hurd,
only using different APIs.  After all, the Hurd had “containers”
(lightweight virtualization) right from the beginning.

Ludo’.