bug#17122: More problems with --no-substitutes

bug#17122: More problems with --no-substitutes

[Mark H Weaver]

This just happened to me on core-updates, on my YeeLoong:

--8<---------------cut here---------------start------------->8---
mhw:~/guix-core-updates$ ./pre-inst-env guix build -S expect lua zip pth bazaar ocaml
substitute-binary: Backtrace:
substitute-binary: In ice-9/boot-9.scm:
substitute-binary:  157: 0 [catch #t #<catch-closure 107fb4f0> ...]
substitute-binary: 
substitute-binary: ice-9/boot-9.scm:157:17: In procedure catch:
substitute-binary: ice-9/boot-9.scm:157:17: In procedure system-async-mark: thread has already exited
  C-c C-c
--8<---------------cut here---------------end--------------->8---

No doubt, the "system-async-mark: thread has already exited" is a
problem, but that's not what bothers me.

What disturbs me the most is that 'substitute-binary' is being called at
all.  I'm 100% certain that I passed '--no-substitutes' to guix-daemon.
I use a script to start guix-daemon with the options I prefer, to avoid
mistakes.  I also just checked with 'ps', and indeed '--no-substitutes'
is there on the command line.

It's very important to me to trust that guix-daemon will not accept
binaries from the internet, even if there's a man-in-the-middle that
pretends to be hydra.gnu.org with mips64el binaries for me.

I'm surprised and concerned that we seem to be having so much trouble
making '--no-substitutes' work reliably.  How hard can it be?

Until we get this straightened out, what's the most reliable way for me
to hack the code to ensure that substitutes cannot work, ever?

     Mark

bug#17122: More problems with --no-substitutes

[Sree Harsha Totakura]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 03/27/2014 05:12 PM, Mark H Weaver wrote:
> Until we get this straightened out, what's the most reliable way for me to 
> hack the code to ensure that substitutes cannot work, ever?

As a hack, you can try removing the code after the first line from
nix/scripts/substitute-binary.in and just have the script exit a return value
of 1.  (you'll have to run config.status and make install after that).

This script substitute-binary when installed, is called by the daemon to fetch
the substitutes.  If you just return 1 without doing anything in this script,
I am guessing the daemon would fall back to build locally.

Sree
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Icedove - http://www.enigmail.net/

iQEcBAEBAgAGBQJTNFb1AAoJECthXLMALpxGi1kIALlZqaI3a0/j/cUuxoqGBHzs
qIZEsTa9xTpkXR2K7DUl8IHjbHUDNlTcj+s8tnuREHUIeBcy+pxECLp5664Zxby0
/OqnVeEad+CgFS5kT3EfpvH5uVJ1kfP/OxYFgyp3hdIxyvh6pKc80M7wVjGiMyLt
zjFzkkVNrMijp9jnIBUbeCcBsTnK7FIEZfKL1Jo6hMdQV/5HEE2KLQiWMp42pDCP
HEd2y5umybZ3x2uDxmTADwaYC/ZUidKLvAwccV5P2PYGLn2ajpq2OO9xLHr5H9Bh
4wexsWWgXd8+ynAm3belv6FBdEMcyRA8oLvlWUF7aLZfoV5jfo6lAg5LBvnfBSU=
=kU5j
-----END PGP SIGNATURE-----

bug#17122: More problems with --no-substitutes

[Mark H Weaver]

I should have pasted the output of the 'ps' command I did immediately
after this error.  Here it is:

--8<---------------cut here---------------start------------->8---
mhw:~/guix-core-updates$ ./pre-inst-env guix build -S expect lua zip pth bazaar ocaml
substitute-binary: Backtrace:
substitute-binary: In ice-9/boot-9.scm:
substitute-binary:  157: 0 [catch #t #<catch-closure 107fb4f0> ...]
substitute-binary: 
substitute-binary: ice-9/boot-9.scm:157:17: In procedure catch:
substitute-binary: ice-9/boot-9.scm:157:17: In procedure system-async-mark: thread has already exited
  C-c C-c
mhw:~/guix-core-updates$ ps auxww | grep guix-daemon
mhw       1275  3.0  0.2   5248  2688 pts/0    S+   11:40   0:00 grep guix-daemon
root     13020  0.0  0.2   7552  2496 tty6     S+   Mar22   0:00 guix-daemon --no-substitutes --build-users-group=guix-builder
root     14429  0.1  0.9  15808 10304 ?        Ss   Mar26   1:05 guix-daemon 14425            --build-users-group=guix-builder
--8<---------------cut here---------------end--------------->8---

Notice that there's another 'guix-daemon' process here (pid 14429) which
I didn't start, and which has "--build-users-group=guix-builder" but is
missing "--no-substitutes".

Could it be that there's some code in 'guix-daemon' that launches
another 'guix-daemon' subprocess, and which takes care to propagate some
of the options but not all of them?

      Mark

bug#17122: More problems with --no-substitutes

[Ludovic Courtès]

Mark H Weaver <mhw@netris.org> skribis:

> What disturbs me the most is that 'substitute-binary' is being called at
> all.  I'm 100% certain that I passed '--no-substitutes' to guix-daemon.
> I use a script to start guix-daemon with the options I prefer, to avoid
> mistakes.  I also just checked with 'ps', and indeed '--no-substitutes'
> is there on the command line.

Can you check with current master?  (See in particular commits 968e84a
and c9e2b0b.)  Does tests/guix-daemon.sh pass?

Make sure to use the “right” daemon code, but updating the nix-upstream
module and running nix/sync-with-upstream.

> I'm surprised and concerned that we seem to be having so much trouble
> making '--no-substitutes' work reliably.  How hard can it be?

The issue is that guix-daemon.cc glues into Nix’s code, and Nix changed
the way it handles substituter settings in the last update.

Specifically, in Nix commit dcaea042, the Settings::update method is
made to re-read $NIX_SUBSTITUTERS:
<https://github.com/NixOS/nix/commit/dcaea042fc895667bf6f529471ff9f449629774c>;
then in Guix commit 89faa5c I adjusted guix-daemon.cc accordingly, but
inadvertently removed the ‘if’ branch that clears the substituter list.

Commit c9e2b0b augments tests/guix-daemon.sh to test guix-daemon
--no-substitutes.

Ludo’.

bug#17122: More problems with --no-substitutes

[Ludovic Courtès]

Mark H Weaver <mhw@netris.org> skribis:

> mhw:~/guix-core-updates$ ps auxww | grep guix-daemon
> mhw       1275  3.0  0.2   5248  2688 pts/0    S+   11:40   0:00 grep guix-daemon
> root     13020  0.0  0.2   7552  2496 tty6     S+   Mar22   0:00 guix-daemon --no-substitutes --build-users-group=guix-builder
> root     14429  0.1  0.9  15808 10304 ?        Ss   Mar26   1:05 guix-daemon 14425            --build-users-group=guix-builder
>
> Notice that there's another 'guix-daemon' process here (pid 14429) which
> I didn't start, and which has "--build-users-group=guix-builder" but is
> missing "--no-substitutes".

Process 14429 is a child of the real guix-daemon, as spawned upon
‘accept’ success; see nix-daemon.cc:878.

The fact that “--no-substitutes” is replaced with “14425” is a result
from the modification of argvSaved[1] in subsequent lines, and doesn’t
change anything regarding substitute handling (which is handled by the
parent process anyway.)

Ludo’.

bug#17122: More problems with --no-substitutes

[Mark H Weaver]

ludo@gnu.org (Ludovic Courtès) writes:

> Mark H Weaver <mhw@netris.org> skribis:
>
>> What disturbs me the most is that 'substitute-binary' is being called at
>> all.  I'm 100% certain that I passed '--no-substitutes' to guix-daemon.
>> I use a script to start guix-daemon with the options I prefer, to avoid
>> mistakes.  I also just checked with 'ps', and indeed '--no-substitutes'
>> is there on the command line.
>
> Can you check with current master?  (See in particular commits 968e84a
> and c9e2b0b.)  Does tests/guix-daemon.sh pass?

Yes, it does.

I also hacked 'guix-substitute-binary' to unconditionally raise an error
as soon as it is called (a local patch I intend to keep indefinitely).

Before your recent commits, 'guix-substitute-binary' was always being
called by 'guix build' (unless --no-substitutes was passed to it), but
that seems to be fixed now.  Thanks.

>> I'm surprised and concerned that we seem to be having so much trouble
>> making '--no-substitutes' work reliably.  How hard can it be?
>
> The issue is that guix-daemon.cc glues into Nix’s code, and Nix changed
> the way it handles substituter settings in the last update.

Ah, okay.  I wish this wasn't so fragile, but the new test case you
added helps, as does my hack to raise an error if the substituter is
called, which will immediately alert me to any similar problems in the
future.

> Specifically, in Nix commit dcaea042, the Settings::update method is
> made to re-read $NIX_SUBSTITUTERS:
> <https://github.com/NixOS/nix/commit/dcaea042fc895667bf6f529471ff9f449629774c>;
> then in Guix commit 89faa5c I adjusted guix-daemon.cc accordingly, but
> inadvertently removed the ‘if’ branch that clears the substituter list.
>
> Commit c9e2b0b augments tests/guix-daemon.sh to test guix-daemon
> --no-substitutes.

Thanks very much!  I'm closing this bug now.

     Mark

bug#17122: More problems with --no-substitutes

[Ludovic Courtès]

Mark H Weaver <mhw@netris.org> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>
>> Mark H Weaver <mhw@netris.org> skribis:
>>
>>> What disturbs me the most is that 'substitute-binary' is being called at
>>> all.  I'm 100% certain that I passed '--no-substitutes' to guix-daemon.
>>> I use a script to start guix-daemon with the options I prefer, to avoid
>>> mistakes.  I also just checked with 'ps', and indeed '--no-substitutes'
>>> is there on the command line.
>>
>> Can you check with current master?  (See in particular commits 968e84a
>> and c9e2b0b.)  Does tests/guix-daemon.sh pass?
>
> Yes, it does.
>
> I also hacked 'guix-substitute-binary' to unconditionally raise an error
> as soon as it is called (a local patch I intend to keep indefinitely).

Hopefully we’ll catch any regression in the future before it bites.

Ludo’.