Boostrap tar cannot exec xz

Boostrap tar cannot exec xz

[Carlos Sánchez de La Lama]

Hi Ludo,

still trying too bootstrap linux-powerpc. I find the boostrap binary
"tar" fails to execute xz (which is un turn required to build make on
continue the bootstrap process), because it can not find bash.

The problem was introduced in your commit

74d637397b44a90d2dbc2ac100b38591dd4f81eb

which upgraded tar to 1.29. That commit also added the substitution of
"/bin/sh" to "<bash-store-path>/bin/sh" in src/system.c when compiling
tar, which is fine in general, but gets broken when %static-binaries
does a remove-store-reference for all the static binaries.

Is that substitution really needed?

BR

Carlos

Re: Boostrap tar cannot exec xz

[Ludovic Courtès]

Hi!

csanchezdll@gmail.com (Carlos Sánchez de La Lama) skribis:

> still trying too bootstrap linux-powerpc.

This is exciting, BTW!

> I find the boostrap binary "tar" fails to execute xz (which is un turn
> required to build make on continue the bootstrap process), because it
> can not find bash.
>
> The problem was introduced in your commit
>
> 74d637397b44a90d2dbc2ac100b38591dd4f81eb
>
> which upgraded tar to 1.29. That commit also added the substitution of
> "/bin/sh" to "<bash-store-path>/bin/sh" in src/system.c when compiling
> tar, which is fine in general, but gets broken when %static-binaries
> does a remove-store-reference for all the static binaries.
>
> Is that substitution really needed?

That substitution is needed, because otherwise ‘tar’ fails to spawn
external programs when in a build environment (the chroot lacks
/bin/sh), which causes its own test suite to fail.

I understand that this is not OK for the binaries produced by
make-bootstrap.scm.  However, /bin/sh is not OK either, AFAICS.

In ‘raw-build’ in bootstrap.scm, we run:

  xz -dc thing.tar.xz | tar xv

to work around this problem.  Isn’t it enough?

Thanks,
Ludo’.

Re: Boostrap tar cannot exec xz

[Carlos Sánchez de La Lama]

Hi!

>> still trying too bootstrap linux-powerpc.
>
> This is exciting, BTW!

Yes, my plan is to make a full GuixSD install at some point. But
step by step, first thing is bootstrapping on top of my debian.

>> That commit also added the substitution of "/bin/sh" to
>> "<bash-store-path>/bin/sh" in src/system.c when compiling tar, which
>> is fine in general, but gets broken when %static-binaries does a
>> remove-store-reference for all the static binaries.
>
> That substitution is needed, because otherwise ‘tar’ fails to spawn
> external programs when in a build environment (the chroot lacks
> /bin/sh), which causes its own test suite to fail.

Noticed. I tried removing that substitution extra phase and tar
testsuite failed.

> In ‘raw-build’ in bootstrap.scm, we run:
>
>   xz -dc thing.tar.xz | tar xv

If I have understood correctly the code, raw-build is only used for
%bootstrap-guile. However, the problem arises when building make (I do
not have access to that machine righg now, so I am unsure of the exact
package). i guess one option would be to create bootstrap versions of
all packages on which tar depends (most probably already exist) and use
xz | tar on them, until the requirements for a new non-bootstrap tar
build can be met.

Another option that just csme to my mind is replacing all the nuked
store referenced in the static binaries with references to the unpacked
store path after extraction. As the nuked references can be easily
identified by the "eeeeee..." fake path, this should be doable.

Which solution do you think makes more sense?

BR

Carlos

Re: Boostrap tar cannot exec xz

[Ludovic Courtès]

Hi!

csanchezdll@gmail.com (Carlos Sánchez de La Lama) skribis:

>> In ‘raw-build’ in bootstrap.scm, we run:
>>
>>   xz -dc thing.tar.xz | tar xv
>
> If I have understood correctly the code, raw-build is only used for
> %bootstrap-guile. However, the problem arises when building make (I do
> not have access to that machine righg now, so I am unsure of the exact
> package). i guess one option would be to create bootstrap versions of
> all packages on which tar depends (most probably already exist) and use
> xz | tar on them, until the requirements for a new non-bootstrap tar
> build can be met.

The question is how ‘tar’ in the ‘bootstrap-binaries’ we use on other
arches differs from the one you just built.

--8<---------------cut here---------------start------------->8---
$ guix build bootstrap-binaries --fallback
/gnu/store/zh914iy23h97fwskh12vl3py1frkhrqn-bootstrap-binaries-0
ludo@pluto ~/src/guix$ /gnu/store/zh914iy23h97fwskh12vl3py1frkhrqn-bootstrap-binaries-0/bin/tar --version
tar (GNU tar) 1.27
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by John Gilmore and Jay Fenlason.
$ strings /gnu/store/zh914iy23h97fwskh12vl3py1frkhrqn-bootstrap-binaries-0/bin/tar |grep bin/sh
/bin/sh
$ touch foo.tar.xz
$ strace -e execve -f /gnu/store/zh914iy23h97fwskh12vl3py1frkhrqn-bootstrap-binaries-0/bin/tar xf foo.tar.xz
execve("/gnu/store/zh914iy23h97fwskh12vl3py1frkhrqn-bootstrap-binaries-0/bin/tar", ["/gnu/store/zh914iy23h97fwskh12vl"..., "xf", "foo.tar.xz"], [/* 63 vars */]) = 0
/gnu/store/zh914iy23h97fwskh12vl3py1frkhrqn-bootstrap-binaries-0/bin/tar: This does not look like a tar archive
Process 28766 attached
[pid 28766] execve("/home/ludo/.guix-profile/bin/xz", ["xz", "-d"], [/* 63 vars */]) = 0
xz: (stdin): File format not recognized
[pid 28766] +++ exited with 1 +++
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=28766, si_status=1, si_utime=0, si_stime=0} ---
/gnu/store/zh914iy23h97fwskh12vl3py1frkhrqn-bootstrap-binaries-0/bin/tar: Child returned status 1
/gnu/store/zh914iy23h97fwskh12vl3py1frkhrqn-bootstrap-binaries-0/bin/tar: Error is not recoverable: exiting now
+++ exited with 2 +++
--8<---------------cut here---------------end--------------->8---

So apparently, tar 1.27 did not execute xz via /bin/sh, but current tar
does not either:

--8<---------------cut here---------------start------------->8---
$ strace -e execve -f tar xf ~/src/guix/foo.tar.gz 
execve("/run/current-system/profile/bin/tar", ["tar", "xf", "/home/ludo/src/guix/foo.tar.gz"], [/* 63 vars */]) = 0
Process 29347 attached
[pid 29347] execve("/home/ludo/soft/bin/gzip", ["gzip", "-d"], [/* 63 vars */]) = -1 ENOENT (No such file or directory)
[pid 29347] execve("/home/ludo/.opam/system/bin/gzip", ["gzip", "-d"], [/* 63 vars */]) = -1 ENOENT (No such file or directory)
[pid 29347] execve("/home/ludo/.guix-profile/bin/gzip", ["gzip", "-d"], [/* 63 vars */]) = -1 ENOENT (No such file or directory)
[pid 29347] execve("/home/ludo/.guix-profile/sbin/gzip", ["gzip", "-d"], [/* 63 vars */]) = -1 ENOENT (No such file or directory)
[pid 29347] execve("/run/setuid-programs/gzip", ["gzip", "-d"], [/* 63 vars */]) = -1 ENOENT (No such file or directory)
[pid 29347] execve("/run/current-system/profile/bin/gzip", ["gzip", "-d"], [/* 63 vars */]) = 0
[pid 29347] +++ exited with 0 +++
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=29347, si_status=0, si_utime=0, si_stime=0} ---
+++ exited with 0 +++
$ tar --version
tar (GNU tar) 1.29
Copyright (C) 2015 Free Software Foundation, Inc.
Permeso GPLv3+: GNU GPL versio 3 aŭ posta <http://gnu.org/licenses/gpl.html>.
Tio ĉi estas libera programaro: vi estas libera por ŝanĝi kaj redisdoni ĝin.
Ekzistas NENIU GARANTIO, laŭ plej amplekse permesate de la leĝoj.

Verkita de John Gilmore kaj Jay Fenlason.
--8<---------------cut here---------------end--------------->8---

Could you check exactly what’s going on and send more details?

> Another option that just csme to my mind is replacing all the nuked
> store referenced in the static binaries with references to the unpacked
> store path after extraction. As the nuked references can be easily
> identified by the "eeeeee..." fake path, this should be doable.

That’s not doable because we don’t know the hash in advance.

Thanks,
Ludo’.

Re: Boostrap tar cannot exec xz

[Carlos Sánchez de La Lama]

Hi Ludo,

> So apparently, tar 1.27 did not execute xz via /bin/sh, but current tar
> does not either:
>

It does not when extracting, but seems to do when compressing (which is
what causes my bootstrapping problem). Your experiment yields similar
results here:

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix build bootstrap-binaries
/gnu/store/2g5cgcv8fsmaps528jn6np241mkyyidm-bootstrap-binaries-0
$ /gnu/store/2g5cgcv8fsmaps528jn6np241mkyyidm-bootstrap-binaries-0/bin/tar --version
tar (GNU tar) 1.29
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by John Gilmore and Jay Fenlason.
$ strace -e execve -f /gnu/store/2g5cgcv8fsmaps528jn6np241mkyyidm-bootstrap-binaries-0/bin/tar xf ~/foo.tar.xz 
execve("/gnu/store/2g5cgcv8fsmaps528jn6np241mkyyidm-bootstrap-binaries-0/bin/tar", ["/gnu/store/2g5cgcv8fsmaps528jn6n"..., "xf", "/home/csanchez/foo.tar.xz"], [/* 25 vars */]) = 0
/gnu/store/2g5cgcv8fsmaps528jn6np241mkyyidm-bootstrap-binaries-0/bin/tar: This does not look like a tar archive
Process 1608 attached
[pid  1608] execve("/home/csanchez/bin/xz", ["xz", "-d"], [/* 25 vars */]) = -1 ENOENT (No such file or directory)
[pid  1608] execve("/home/csanchez/bin/xz", ["xz", "-d"], [/* 25 vars */]) = -1 ENOENT (No such file or directory)
[pid  1608] execve("/home/csanchez/bin/xz", ["xz", "-d"], [/* 25 vars */]) = -1 ENOENT (No such file or directory)
[pid  1608] execve("/bin/xz", ["xz", "-d"], [/* 25 vars */]) = -1 ENOENT (No such file or directory)
[pid  1608] execve("/sbin/xz", ["xz", "-d"], [/* 25 vars */]) = -1 ENOENT (No such file or directory)
[pid  1608] execve("/usr/bin/xz", ["xz", "-d"], [/* 25 vars */]) = 0
xz: (stdin): File format not recognized
[pid  1608] +++ exited with 1 +++
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1608, si_uid=1000, si_status=1, si_utime=0, si_stime=0} ---
/gnu/store/2g5cgcv8fsmaps528jn6np241mkyyidm-bootstrap-binaries-0/bin/tar: Child returned status 1
/gnu/store/2g5cgcv8fsmaps528jn6np241mkyyidm-bootstrap-binaries-0/bin/tar: Error is not recoverable: exiting now
+++ exited with 2 +++
--8<---------------cut here---------------end--------------->8---

However, when *compressing*, it needs /bin/sh.

--8<---------------cut here---------------start------------->8---
$ touch ~/foo
$ strace -f -e execve /gnu/store/2g5cgcv8fsmaps528jn6np241mkyyidm-bootstrap-binaries-0/bin/tar cvfa ~/foo.tar.xz ~/foo
execve("/gnu/store/2g5cgcv8fsmaps528jn6np241mkyyidm-bootstrap-binaries-0/bin/tar", ["/gnu/store/2g5cgcv8fsmaps528jn6n"..., "cvfa", "/home/csanchez/foo.tar.xz", "/home/csanchez/foo"], [/* 25 vars */]) = 0
/gnu/store/2g5cgcv8fsmaps528jn6np241mkyyidm-bootstrap-binaries-0/bin/tar: Removing leading `/' from member names
/home/csanchez/foo
Process 1615 attached
[pid  1615] execve("/gnu/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-bash-4.3.42/bin/sh", ["/gnu/store/eeeeeeeeeeeeeeeeeeeee"..., "-c", "xz"], [/* 25 vars */]) = -1 ENOENT (No such file or directory)
tar (child): xz: Cannot exec: No such file or directory
tar (child): Error is not recoverable: exiting now
[pid  1615] +++ exited with 2 +++
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1615, si_uid=1000, si_status=2, si_utime=0, si_stime=0} ---
/gnu/store/2g5cgcv8fsmaps528jn6np241mkyyidm-bootstrap-binaries-0/bin/tar: Child returned status 2
/gnu/store/2g5cgcv8fsmaps528jn6np241mkyyidm-bootstrap-binaries-0/bin/tar: Error is not recoverable: exiting now
+++ exited with 2 +++
--8<---------------cut here---------------end--------------->8---

This is what makes <hash>-make-4.2.tar.xz-builder to fail, stopping the
bootstrap.

>> Another option that just came to my mind is replacing all the nuked
>> store referenced in the static binaries with references to the unpacked
>> store path after extraction. As the nuked references can be easily
>> identified by the "eeeeee..." fake path, this should be doable.
>
> That’s not doable because we don’t know the hash in advance.

When creating the tarball, we don't know the hash it will have when
extracted, so the references are nuked. But when creating the package
from the tarball, in the bootstraping machine (my linux-powerpc in this
case), couldn't we "un-nuke" the references, making them point to the
store path we just created?

Thanks!

Carlos

Boostrap tar cannot exec /bin/sh

[Danny Milosavljevic]

Hi,

sorry to butt in but why in the world does tar require a shell?

It *shouldn't* exec sh at all.

Tar changelog mentions:

2016-04-14  Sergey Poznyakoff  <gray@gnu.org.ua>

        Fix argument handling when running external commands.
        * src/system.c (xexec): Use sh -c to run the command.  This fixed
        bug introduced by 7b5e80396 (tar 1.27)

xexec is:

static _Noreturn void
xexec (const char *cmd)
{
  char *argv[4];

  argv[0] = (char *) "/bin/sh";
  argv[1] = (char *) "-c";
  argv[2] = (char *) cmd;
  argv[3] = NULL;

  execv ("/bin/sh", argv);
  exec_fatal (cmd);
}

That is not safe. What if a semicolon is in cmd (or hundreds of other things)?

configure then tries to find the the command line to use and then uses set_use_compress_program_option to remember it as *one string*. 

A simple (and ugly) workaround for you would be:

static _Noreturn void
xexec (const char *cmd)
{
  char *argv[4];

  if (strchr(cmd, ' ') == NULL) {
    argv[0] = cmd;
    argv[1] = NULL;
  } else {
    argv[0] = (char *) "/bin/sh";
    argv[1] = (char *) "-c";
    argv[2] = (char *) cmd;
    argv[3] = NULL;
  }

  execvp (argv[0], argv);
  exec_fatal (cmd);
}

Better would be to pass an array in the first place :P

Even better would be to use tar for creating archives and xz for compression (in your case, in "make").

I've checked what guix environment tar will configure:

tar_cv_compressor_bzip2=bzip2
tar_cv_compressor_compress=compress
tar_cv_compressor_gzip=gzip
tar_cv_compressor_lzip=lzip
tar_cv_compressor_lzma=lzma
tar_cv_compressor_lzop=lzop
tar_cv_compressor_xz=xz

Re: Boostrap tar cannot exec /bin/sh

[Carlos Sánchez de La Lama]

Hi Danny,

> sorry to butt in but why in the world does tar require a shell?

Agreed, but that's something that should be fixed upstream in tar.

> A simple (and ugly) workaround for you would be:
>
> static _Noreturn void
> xexec (const char *cmd)
> {
>   char *argv[4];
>
>   if (strchr(cmd, ' ') == NULL) {
>     argv[0] = cmd;
>     argv[1] = NULL;
>   } else {
>     argv[0] = (char *) "/bin/sh";
>     argv[1] = (char *) "-c";
>     argv[2] = (char *) cmd;
>     argv[3] = NULL;
>   }
>
>   execvp (argv[0], argv);
>   exec_fatal (cmd);
> }

Looks like that will fix it, true, thougth it might be a little bit too
agressive patching on guix side. I would be more confortable finding a
way to use current tar (until, hopefully, this gets solved upstream).

> Even better would be to use tar for creating archives and xz for
> compression (in your case, in "make").

Yup, if "de-nuking" the references is not a possibility (I am still
considering it), then I wil try changing the build system so instead of
"tar cvfa file.xz", "tar cv | xz" is used.

Thanks!

Carlos

Re: Boostrap tar cannot exec xz

[Ludovic Courtès]

csanchezdll@gmail.com (Carlos Sánchez de La Lama) skribis:

> Hi Ludo,
>
>> So apparently, tar 1.27 did not execute xz via /bin/sh, but current tar
>> does not either:
>>
>
> It does not when extracting, but seems to do when compressing (which is
> what causes my bootstrapping problem). Your experiment yields similar
> results here:

[...]

> However, when *compressing*, it needs /bin/sh.
>
> $ touch ~/foo
> $ strace -f -e execve /gnu/store/2g5cgcv8fsmaps528jn6np241mkyyidm-bootstrap-binaries-0/bin/tar cvfa ~/foo.tar.xz ~/foo
> execve("/gnu/store/2g5cgcv8fsmaps528jn6np241mkyyidm-bootstrap-binaries-0/bin/tar", ["/gnu/store/2g5cgcv8fsmaps528jn6n"..., "cvfa", "/home/csanchez/foo.tar.xz", "/home/csanchez/foo"], [/* 25 vars */]) = 0
> /gnu/store/2g5cgcv8fsmaps528jn6np241mkyyidm-bootstrap-binaries-0/bin/tar: Removing leading `/' from member names
> /home/csanchez/foo
> Process 1615 attached
> [pid  1615] execve("/gnu/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-bash-4.3.42/bin/sh", ["/gnu/store/eeeeeeeeeeeeeeeeeeeee"..., "-c", "xz"], [/* 25 vars */]) = -1 ENOENT (No such file or directory)
> tar (child): xz: Cannot exec: No such file or directory
> tar (child): Error is not recoverable: exiting now
> [pid  1615] +++ exited with 2 +++
> --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1615, si_uid=1000, si_status=2, si_utime=0, si_stime=0} ---
> /gnu/store/2g5cgcv8fsmaps528jn6np241mkyyidm-bootstrap-binaries-0/bin/tar: Child returned status 2
> /gnu/store/2g5cgcv8fsmaps528jn6np241mkyyidm-bootstrap-binaries-0/bin/tar: Error is not recoverable: exiting now
> +++ exited with 2 +++
>
> This is what makes <hash>-make-4.2.tar.xz-builder to fail, stopping the
> bootstrap.

Ooh, and the Tar commit Danny mentions is a post-1.27 change, which is
why our other bootstrap binaries don’t have this problem, is that right?

If so, you have two options: using Tar 1.27 in ‘static-binaries’, or
using a Tar variant with a patch along the lines of what Danny suggests.
Do whichever is easier.

>>> Another option that just came to my mind is replacing all the nuked
>>> store referenced in the static binaries with references to the unpacked
>>> store path after extraction. As the nuked references can be easily
>>> identified by the "eeeeee..." fake path, this should be doable.
>>
>> That’s not doable because we don’t know the hash in advance.
>
> When creating the tarball, we don't know the hash it will have when
> extracted, so the references are nuked. But when creating the package
> from the tarball, in the bootstraping machine (my linux-powerpc in this
> case), couldn't we "un-nuke" the references, making them point to the
> store path we just created?

Forget about it.  :-)

Ludo’.