Why does glibc provide bash?

Why does glibc provide bash?

[Andy Wingo]

Hi,

Let's start bash with no RC file in a null environment:

    $ env -i ~/.guix-profile/bin/bash --norc

Check to make sure the env is empty:

    bash-4.3$ env
    PWD=/home/wingo
    SHLVL=1
    _=/usr/bin/env

Now look at this little sample file:

    bash-4.3$ .guix-profile/bin/cat /tmp/foo.bash
    _foo_complete ()
    {
        false
    }

    complete -F _foo_complete foo

OK now we run bash on that file:

    bash-4.3$ .guix-profile/bin/bash --norc /tmp/foo.bash
    /tmp/foo.bash: line 6: complete: command not found

What could be going on?  Well here's a lollerskates thing, back in the
main environment:

    $ ls -l `which bash`
    lrwxrwxrwx 11 root guixbuild 63 Jan  1  1970 /home/wingo/.guix-profile/bin/bash -> /gnu/store/5995q4p9ayvicd8qxjmn8zrwis4y7a8c-glibc-2.21/bin/bash

It's coming from glibc!!!  Zomg.  Why is this?  From what I can tell
it's not in the propagated inputs of glibc, so this shouldn't be
happening.

Andy

Re: Why does glibc provide bash?

[Mark H Weaver]

Hi Andy, it's good to see you here :)

Andy Wingo <wingo@igalia.com> writes:

>     bash-4.3$ .guix-profile/bin/bash --norc /tmp/foo.bash
>     /tmp/foo.bash: line 6: complete: command not found
>
> What could be going on?  Well here's a lollerskates thing, back in the
> main environment:
>
>     $ ls -l `which bash`
>     lrwxrwxrwx 11 root guixbuild 63 Jan  1  1970 /home/wingo/.guix-profile/bin/bash -> /gnu/store/5995q4p9ayvicd8qxjmn8zrwis4y7a8c-glibc-2.21/bin/bash
>
> It's coming from glibc!!!  Zomg.  Why is this?  From what I can tell
> it's not in the propagated inputs of glibc, so this shouldn't be
> happening.

Indeed, we should have dealt with this issue long ago, but it seems to
have fallen through the cracks.  The following email, including the
quotations, summarizes the problems and possible solutions:

  https://lists.gnu.org/archive/html/guix-devel/2014-03/msg00349.html

Ludovic pointed out that glibc's static bash should be in $libexecdir
not $bindir.  That's an easy fix, but I spoke against it because, IMO,
it sweeps the real problem under the rug where it might be forgotten.

The real problem is that there is a circular dependency between bash and
glibc, but guix-daemon does not allow circular dependencies between
different store items, a limitation inherited from nix-daemon.  This
means that we either have to avoid the circularity by introducing
multiple copies of glibc and/or bash (currently we duplicate both), or
else we must put glibc and bash into the same store item.

Obviously, putting glibc and bash into the same store item feels wrong
on multiple levels.  First of all, users might reasonably want to
install one in a profile but not both.  One solution would be to have a
'glibc+bash' package behind the scenes, and then separate 'glibc' and
'bash' packages that have only symlinks into the 'glibc+bash' package.
It could even be just two symlinks, where /gnu/store/…-bash-VERSION and
/gnu/store/…-glibc-VERSION are each symlinks into subdirectories of
/gnu/store/…-glibc+bash.

Another problem is that it's rather gross to try to build these two
packages together within a single package definition.  I guess that
first glibc would be built and installed, configured to look for a
yet-to-be-built bash that will eventually be installed in glibc's
libexecdir.  Then 'bash' would be built against that 'glibc' and
installed into the place where 'glibc' will look for it.  It's workable
but would be a bit gross to implement.

A better way might be to initially build the two packages separately,
with 'glibc' initially configured to use the bootstrap bash, and then to
have a separate 'glibc+bash' package that simply copies the contents of
the two packages together into a new package, while mutating the
reference(s) to bash in 'glibc' to use the new bash instead of the
bootstrap one, thus creating the cycle for the first time.  Care should
be taken to ensure that the new package does not refer to either the
bootstrap bash or the original separate glibc or bash packages.

Finally, I should raise the possibility of eventually eliminating this
prohibition on circular dependencies between different store items.
However, I'm not confident that I'm aware of all the implications of
doing this, e.g. how it might effect the desirable properties of
Nix/Guix and what parts of the code would need adjustment.  I'd like to
investigate this some day, but not today.

What do you think?

      Mark

Re: Why does glibc provide bash?

[Andy Wingo]

Heya Mark, good to see you here too :)

On Tue 11 Aug 2015 06:23, Mark H Weaver <mhw@netris.org> writes:

> Andy Wingo <wingo@igalia.com> writes:
>
>>     $ ls -l `which bash`
>>     lrwxrwxrwx 11 root guixbuild 63 Jan  1  1970 /home/wingo/.guix-profile/bin/bash -> /gnu/store/5995q4p9ayvicd8qxjmn8zrwis4y7a8c-glibc-2.21/bin/bash
>
> Indeed, we should have dealt with this issue long ago, but it seems to
> have fallen through the cracks.  The following email, including the
> quotations, summarizes the problems and possible solutions:
>
>   https://lists.gnu.org/archive/html/guix-devel/2014-03/msg00349.html

I see that glibc depends on static-bash as an input, but why would
installing glibc cause me to have a static bash in my profile?

Is it because the glibc package installs bash to its $bindir, or because
the input is being propagated correctly, or because the input is being
propagated incorrectly?

I read through the linked mail a few times but if the answer is there, I
missed it.

The circular dep issue sounds pretty exciting but it seems like there
might be another bug here.  Dunno.  I wonder how Nix deals with this.

Andy

Re: Why does glibc provide bash?

[Ludovic Courtès]

Andy Wingo <wingo@igalia.com> skribis:

> I see that glibc depends on static-bash as an input, but why would
> installing glibc cause me to have a static bash in my profile?

That’s because the static bash ends up being copied in glibc’s BINDIR.

> The circular dep issue sounds pretty exciting but it seems like there
> might be another bug here.  Dunno.  I wonder how Nix deals with this.

Nix has /bin/sh in its default build environment, so it doesn’t have
this problem but has a bunch of other issues.  ;-)

  https://lists.gnu.org/archive/html/bug-guix/2013-01/msg00041.html

(Now, I need to think more about the options Mark suggests.)

Ludo’.

Re: Why does glibc provide bash?

[Ludovic Courtès]

I’ve pushed an attempt for it in wip-core-updates, specifically this
commit:

  http://git.savannah.gnu.org/cgit/guix.git/commit/?h=wip-core-updates&id=e0a33ce07b56111329575b0ccc90c9c314fbd221

The trick is simply to not copy the ‘bash’ binary to libc (not sure why
I didn’t do it this way from the start.)

Mark: WDYT?  I think it solves the problem at hand and it’s a reasonably
simple solution.

(The branch is called ‘wip-’ because the glibc upgrade happens to cause
troubles: since it has new locale category elements, the locale data is
incompatible with that older libcs expect, which means the bootstrap
binaries fail with an assertion failure when trying to load the new
locale data, like:

  xz: loadlocale.c:130: _nl_intern_locale_data: Assertion `cnt < (sizeof (_nl_value_type_LC_COLLATE) / sizeof (_nl_value_type_LC_COLLATE[0]))' failed.

I’m looking for a fix a will otherwise postpone the upgrade.)

Ludo’.

Re: Why does glibc provide bash?

[Ludovic Courtès]

ludo@gnu.org (Ludovic Courtès) skribis:

> I’ve pushed an attempt for it in wip-core-updates, specifically this
> commit:
>
>   http://git.savannah.gnu.org/cgit/guix.git/commit/?h=wip-core-updates&id=e0a33ce07b56111329575b0ccc90c9c314fbd221
>
> The trick is simply to not copy the ‘bash’ binary to libc (not sure why
> I didn’t do it this way from the start.)
>
> Mark: WDYT?  I think it solves the problem at hand and it’s a reasonably
> simple solution.

I’ve pushed that solution in ‘core-updates’.

> (The branch is called ‘wip-’ because the glibc upgrade happens to cause
> troubles: since it has new locale category elements, the locale data is
> incompatible with that older libcs expect, which means the bootstrap
> binaries fail with an assertion failure when trying to load the new
> locale data, like:
>
>   xz: loadlocale.c:130: _nl_intern_locale_data: Assertion `cnt < (sizeof (_nl_value_type_LC_COLLATE) / sizeof (_nl_value_type_LC_COLLATE[0]))' failed.
>
> I’m looking for a fix a will otherwise postpone the upgrade.)

Still looking into it...

Ludo’.

libc upgrade vs. incompatible locales

[Ludovic Courtès]

ludo@gnu.org (Ludovic Courtès) skribis:

> (The branch is called ‘wip-’ because the glibc upgrade happens to cause
> troubles: since it has new locale category elements, the locale data is
> incompatible with that older libcs expect, which means the bootstrap
> binaries fail with an assertion failure when trying to load the new
> locale data, like:
>
>   xz: loadlocale.c:130: _nl_intern_locale_data: Assertion `cnt < (sizeof (_nl_value_type_LC_COLLATE) / sizeof (_nl_value_type_LC_COLLATE[0]))' failed.

I thought spelling out the details of why this is annoying might help
find a solution, so here we go.

The binary format for locales is dependent on the libc version.  Over
the last few releases, it turned out to be compatible, but that of 2.22
differs from that of 2.21 (a new element was added to locale categories,
according to ChangeLog.)

During bootstrapping, at some point we build ‘guile-final’ against the
latest libc (2.22.)  In gnu-build-system.scm we heavily use
‘regexp-exec’ (via ‘substitute*’), which calls C code, and thus uses
‘scm_to_locale_string’.

If we run in the “C” locale, we can only pass to ‘regexp-exec’ purely
ASCII strings.  However, it turns out that, occasionally, strings read
from files (in ‘patch-shebangs’ etc.) are not ASCII, but rather UTF-8
(see commit 87c8b92.)  Thus, calls to ‘regexp-exec’ with these strings
lead to a “failed to convert to locale encoding” error.

So ‘guile-final’ needs to run in a UTF-8 locale (the bootstrap Guile
doesn’t have that problem thanks to the hacky
‘guile-default-utf8.patch’.)

However, it we set LOCPATH to point to the libc 2.22 locales, we satisfy
‘guile-final’, but we break all the bootstrap binaries, which were built
with an older libc; specifically, these binaries terminate with the
assertion failure above.  (If you’re still reading, I thank you for your
support.)

So we have some sort of an “interesting” checking-and-egg problem.

We could side-step the issue by using the pure-Scheme SRFI-105 instead
of ‘regexp-exec’.  That may work to some extent, but we cannot get rid
of ‘substitute*’ entirely overnight, so it’s not clear whether this
would be enough.

Apart from that, I can only think of dirty hacks.

What do people think?

Ludo’.

Re: libc upgrade vs. incompatible locales

[Andy Wingo]

On Sun 30 Aug 2015 21:46, ludo@gnu.org (Ludovic Courtès) writes:

> The binary format for locales is dependent on the libc version.  Over
> the last few releases, it turned out to be compatible, but that of 2.22
> differs from that of 2.21 (a new element was added to locale categories,
> according to ChangeLog.)

Does this amount to a binary-incompatible change to libc?  I guess not
if you make sure that if you had a statically linked binary, that you
set LOCPATH appropriately....

What if we built bootstrap binaries to statically link their LOCPATH ?
Is that even possible?

Andy

Re: libc upgrade vs. incompatible locales

[Ludovic Courtès]

Andy Wingo <wingo@igalia.com> skribis:

> On Sun 30 Aug 2015 21:46, ludo@gnu.org (Ludovic Courtès) writes:
>
>> The binary format for locales is dependent on the libc version.  Over
>> the last few releases, it turned out to be compatible, but that of 2.22
>> differs from that of 2.21 (a new element was added to locale categories,
>> according to ChangeLog.)
>
> Does this amount to a binary-incompatible change to libc?  I guess not
> if you make sure that if you had a statically linked binary, that you
> set LOCPATH appropriately....
>
> What if we built bootstrap binaries to statically link their LOCPATH ?
> Is that even possible?

I don’t think locale data can be embedded in binaries.  Also, it’s a
good strategy to avoid rebuilding the bootstrap binaries as much as
possible, as it intuitively suggests that a Thompson attack is unlikely.

Ludo’.

Re: libc upgrade vs. incompatible locales

[Andy Wingo]

On Mon 31 Aug 2015 13:49, ludo@gnu.org (Ludovic Courtès) writes:

> Andy Wingo <wingo@igalia.com> skribis:
>
>> On Sun 30 Aug 2015 21:46, ludo@gnu.org (Ludovic Courtès) writes:
>>
>>> The binary format for locales is dependent on the libc version.  Over
>>> the last few releases, it turned out to be compatible, but that of 2.22
>>> differs from that of 2.21 (a new element was added to locale categories,
>>> according to ChangeLog.)
>>
>> Does this amount to a binary-incompatible change to libc?  I guess not
>> if you make sure that if you had a statically linked binary, that you
>> set LOCPATH appropriately....
>>
>> What if we built bootstrap binaries to statically link their LOCPATH ?
>> Is that even possible?
>
> I don’t think locale data can be embedded in binaries.  Also, it’s a
> good strategy to avoid rebuilding the bootstrap binaries as much as
> possible, as it intuitively suggests that a Thompson attack is unlikely.

Sorry, I meant to say: why not prevent LOCPATH from being overridden on
bootstrap binaries?  Right now they are effectively dynamically linked
to their locale data.  A nice fix would be to statically link them to
their locale data.

A

Re: libc upgrade vs. incompatible locales

[Ludovic Courtès]

I ended up with a reasonable option (commit 28cbc58): in stage 5
(‘%boot5-inputs’), most of the base packages get rebuilt against the new
libc, so they can load the new locale data, but they are rebuilt using
the bootstrap Guile.

In that stage, we also create wrappers for ‘%bootstrap-coreutils&co’
(Coreutils, tar, gzip, xz, etc.), which unset ‘LOCPATH’.  This is faster
than rebuilding them and it’s sufficient since these programs typically
don’t run sub-processes (if they did, then those processed would have an
unset ‘LOCPATH’, which would lead to inconsistencies.)

Comments welcome!

Ludo’.