Binary substituter

Binary substituter

[Ludovic Courtès]

Hello,

Commit f65cf81 adds the beginning of a “binary substituter”.

The substituter (in Nix parlance) is the mechanism by which a pre-built
binary can be substituted to a locally-built binary.  The idea is that,
when you’re about to build, say, /nix/store/xxx-emacs-24.3.1, the daemon
ask substituters whether they have substitutes for that; in turn,
substituters typically ask some server whether it has that path and
download it.  Thus, it allows transparent binary deployment, building
from source when no other option is available (or the user insists).

This commit adds the ‘guix substitute-binary’ command, which is intended
for use by the daemon, not by normal users.  So far, the daemon can
invoke it to know which substitutes are available on hydra.gnu.org.  It
cannot actually download them yet, but that should be done Real Soon, so
stay tuned.  :-)

Ludo’.

Re: Binary substituter

[Ludovic Courtès]

Hi!

ludo@gnu.org (Ludovic Courtès) skribis:

> This commit adds the ‘guix substitute-binary’ command, which is intended
> for use by the daemon, not by normal users.  So far, the daemon can
> invoke it to know which substitutes are available on hydra.gnu.org.  It
> cannot actually download them yet, but that should be done Real Soon, so
> stay tuned.  :-)

Commit fe0cff14f6c5facee4192529f5c7b7a972f185ca implements the long
awaited (at least by me) feature.  :-)

That is, when you run ‘guix build foo’ or ‘guix package -i foo’ the
daemon invokes ‘guix substitute-binary’ and asks it to fetch a pre-built
binary from hydra.gnu.org, unless you pass ‘--no-substitutes’.

I’m now using it to try to build stuff in the ‘core-updates’ branch
without having to spend CPU time on it.  Please report any problems you
may find!

Ludo’.

Re: Binary substituter

[Ludovic Courtès]

Starting from eba783b, the substituter maintains a local cache (by
default under /nix/var/nix/substitute-binary/cache) of lookups of
binaries.

So if you look for a given /nix/store file name several times in a row,
only the first time does it actually connect to hydra.gnu.org to check
if it’s there.

(Nix’s substituter implements the same strategy, though slightly
differently.)

Ludo’.

Re: Binary substituter

[Ludovic Courtès]

And with commit dd36b51, ‘guix build’ & co. report what will be
downloaded in addition to what will be built.

Please test!

Ludo’.

Re: Binary substituter

[Andreas Enge]

Am Mittwoch, 17. April 2013 schrieb Ludovic Courtès:
> And with commit dd36b51, ‘guix build’ & co. report what will be
> downloaded in addition to what will be built.
> 
> Please test!

Do you have an example of a package that should be downloaded? I tried 
hello, and it is still build locally.

Andreas

Re: Binary substituter

[Nikita Karetnikov]

[-- Attachment #1: Type: text/plain, Size: 55 bytes --]

> Please test!

It should only work for x86_64, right?

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: Binary substituter

[Ludovic Courtès]

ludo@gnu.org (Ludovic Courtès) skribis:

> ludo@gnu.org (Ludovic Courtès) skribis:
>
>> This commit adds the ‘guix substitute-binary’ command, which is intended
>> for use by the daemon, not by normal users.  So far, the daemon can
>> invoke it to know which substitutes are available on hydra.gnu.org.  It
>> cannot actually download them yet, but that should be done Real Soon, so
>> stay tuned.  :-)
>
> Commit fe0cff14f6c5facee4192529f5c7b7a972f185ca implements the long
> awaited (at least by me) feature.  :-)

And guess what: until commit ea0ee75 (right now), the substituter
mechanism was actually disabled in the installed daemon.

Now it should really be enabled.  :-)

Ludo’.

Re: Binary substituter

[Ludovic Courtès]

Nikita Karetnikov <nikita@karetnikov.org> skribis:

>> Please test!
>
> It should only work for x86_64, right?

Currently yes.

Ludo’.

Re: Binary substituter

[Ludovic Courtès]

Andreas Enge <andreas@enge.fr> skribis:

> Am Mittwoch, 17. April 2013 schrieb Ludovic Courtès:
>> And with commit dd36b51, ‘guix build’ & co. report what will be
>> downloaded in addition to what will be built.
>> 
>> Please test!
>
> Do you have an example of a package that should be downloaded? I tried 
> hello, and it is still build locally.

Basically 90% of the packages from either branch.  See
<http://hydra.gnu.org:3000/eval/692?full=1> for master, for instance.

Ludo’.

Re: Binary substituter

[Andreas Enge]

Am Donnerstag, 18. April 2013 schrieb Ludovic Courtès:
> Basically 90% of the packages from either branch.  See
> <http://hydra.gnu.org:3000/eval/692?full=1> for master, for instance.

It definitely does not work for me.
   guix build attr
in master builds
   /nix/store/mwrk9wcwwzji8xqh5pr7a5ksicyb70f3-attr-2.4.46
which is available in hydra, see
   http://hydra.gnu.org:3000/build/6991

Do I need to do anything special to activate binary packages?

Andreas

Re: Binary substituter

[Andreas Enge]

Am Donnerstag, 18. April 2013 schrieb Ludovic Courtès:
> And guess what: until commit ea0ee75 (right now), the substituter
> mechanism was actually disabled in the installed daemon.
> 
> Now it should really be enabled.  :-)

PS: I am using the later commit 6858f9d13217b14eeeacede9c42a279468242891 
without success.

Re: Binary substituter

[Ludovic Courtès]

Andreas Enge <andreas@enge.fr> skribis:

> Am Donnerstag, 18. April 2013 schrieb Ludovic Courtès:
>> Basically 90% of the packages from either branch.  See
>> <http://hydra.gnu.org:3000/eval/692?full=1> for master, for instance.
>
> It definitely does not work for me.
>    guix build attr

What’s the output of that command?

Are you sure you run the daemon from yesterday?

What’s in /nix/var/nix/substitute-binary/cache?  (Adjust according to
your $localstatedir.)

> in master builds
>    /nix/store/mwrk9wcwwzji8xqh5pr7a5ksicyb70f3-attr-2.4.46
> which is available in hydra, see
>    http://hydra.gnu.org:3000/build/6991
>
> Do I need to do anything special to activate binary packages?

No.

Thanks for testing!

Ludo’.

Re: Binary substituter

[Nikita Karetnikov]

[-- Attachment #1: Type: text/plain, Size: 1313 bytes --]

>> It should only work for x86_64, right?

> Currently yes.

I suppose it shouldn't fail like this:

# ./pre-inst-env guix package -i gnutls
accepted connection from pid 23127, uid 0
The following package will be installed:
   gnutls-3.1.9.1	/nix/store/7k65yjwz4xs6vfyqggi77lbacj5hwcz7-gnutls-3.1.9.1

guix package: error: build failed: substituter `/home/guix-test/nix/scripts/substitute-binary' failed: Backtrace:
In ice-9/boot-9.scm:
 157: 9 [catch #t #<catch-closure 9258ef0> ...]
In unknown file:
   ?: 8 [apply-smob/1 #<catch-closure 9258ef0>]
In ice-9/boot-9.scm:
  63: 7 [call-with-prompt prompt0 ...]
In ice-9/eval.scm:
 432: 6 [eval # #]
In ice-9/boot-9.scm:
2320: 5 [save-module-excursion #<procedure 91a3e80 at ice-9/boot-9.scm:3961:3 ()>]
3966: 4 [#<procedure 91a3e80 at ice-9/boot-9.scm:3961:3 ()>]
In unknown file:
   ?: 3 [load-compiled/vm "/root/.cache/guile/ccache/2.0-LE-4-2.0/home/guix-test/scripts/guix.go"]
In guix/ui.scm:
 403: 2 [guix-main "/home/guix-test/scripts/guix" "substitute-binary" "--query"]
In guix/scripts/substitute-binary.scm:
 422: 1 [guix-substitute-binary "--query"]
In ice-9/threads.scm:
  99: 0 [loop (#)]

ice-9/threads.scm:99:22: In procedure loop:
ice-9/threads.scm:99:22: In procedure string-append: Wrong type (expecting string): #f

'--no-substitutes' works fine.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: Binary substituter

[Nikita Karetnikov]

[-- Attachment #1: Type: text/plain, Size: 110 bytes --]

> I suppose it shouldn't fail like this: [...]

It happened on 'master'.  Though, it works on 'core-updates'.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: Binary substituter

[Ludovic Courtès]

Nikita Karetnikov <nikita@karetnikov.org> skribis:

> I suppose it shouldn't fail like this:

Indeed.  :-)

> # ./pre-inst-env guix package -i gnutls
> accepted connection from pid 23127, uid 0
> The following package will be installed:
>    gnutls-3.1.9.1	/nix/store/7k65yjwz4xs6vfyqggi77lbacj5hwcz7-gnutls-3.1.9.1
>
> guix package: error: build failed: substituter `/home/guix-test/nix/scripts/substitute-binary' failed: Backtrace:
> In ice-9/boot-9.scm:
>  157: 9 [catch #t #<catch-closure 9258ef0> ...]
> In unknown file:
>    ?: 8 [apply-smob/1 #<catch-closure 9258ef0>]
> In ice-9/boot-9.scm:
>   63: 7 [call-with-prompt prompt0 ...]
> In ice-9/eval.scm:
>  432: 6 [eval # #]
> In ice-9/boot-9.scm:
> 2320: 5 [save-module-excursion #<procedure 91a3e80 at ice-9/boot-9.scm:3961:3 ()>]
> 3966: 4 [#<procedure 91a3e80 at ice-9/boot-9.scm:3961:3 ()>]
> In unknown file:
>    ?: 3 [load-compiled/vm "/root/.cache/guile/ccache/2.0-LE-4-2.0/home/guix-test/scripts/guix.go"]
> In guix/ui.scm:
>  403: 2 [guix-main "/home/guix-test/scripts/guix" "substitute-binary" "--query"]
> In guix/scripts/substitute-binary.scm:
>  422: 1 [guix-substitute-binary "--query"]
> In ice-9/threads.scm:
>   99: 0 [loop (#)]
>
> ice-9/threads.scm:99:22: In procedure loop:
> ice-9/threads.scm:99:22: In procedure string-append: Wrong type (expecting string): #f

Unfortunately, the backtrace isn’t helpful because of threads, and I
can’t reproduce it.

Could you replace ‘par-map’ with ‘map’ and try again?

You can also add ‘pk’ calls in ‘lookup-narinfo’ and similar to get more
details.

TIA,
Ludo’.

Re: Binary substituter

[Ludovic Courtès]

Having discussed it today, the problem seems to relate to the use of
Guile 2.0.5: the web client would not work as expected, and thus the
substituter would erroneously think that the queried store paths are
missing on hydra.gnu.org, and cache that.

I believe that commit 1c9e7d6 partly solves that.  That is, if you
‘rm -rf /nix/var/nix/substitute-binary/’, and then run
‘guix build hello -n’, it should report things that it would download.

However, if you type ‘guix build hello’, it will barf in ‘filtered-port’
when trying to actually download the thing.

If you run Guile 2.0.5, can you confirm that my predictions are correct?
:-)

(Again, there’s no problem with Guile > 2.0.5.)

Ludo’.

Re: Binary substituter

[Andreas Enge]

Am Donnerstag, 25. April 2013 schrieb Ludovic Courtès:
> I believe that commit 1c9e7d6 partly solves that.  That is, if you
> ‘rm -rf /nix/var/nix/substitute-binary/’, and then run
> ‘guix build hello -n’, it should report things that it would download.

Yes, it tells it will download hello. And the cache directory is filled 
with data containing the urls of packages to download.

> However, if you type ‘guix build hello’, it will barf in ‘filtered-port’
> when trying to actually download the thing.

Maybe. Here is the output:

the following file will be downloaded:
   /nix/store/w2121wnp8xv3ycjsgj3ymhb147mrgpc9-hello-2.8
@ substituter-started /nix/store/w2121wnp8xv3ycjsgj3ymhb147mrgpc9-hello-2.8 
/usr/local/guix-git/libexec/guix/substitute-binary
guix substitute-binary: warning: using Guile 2.0.5-deb+1-3, which does not 
support HTTP () encoding
guix substitute-binary: error: download failed; use a newer Guile
@ substituter-failed /nix/store/w2121wnp8xv3ycjsgj3ymhb147mrgpc9-hello-2.8 
256 fetching path `/nix/store/w2121wnp8xv3ycjsgj3ymhb147mrgpc9-hello-2.8' 
failed with exit code 1
guix build: error: build failed: some substitutes for the outputs of 
derivation `/nix/store/7b51j955338q2kxj3ssvly0i2pw20z7q-hello-2.8.drv' 
failed; try `--fallback'

Adding "--fallback" leads to an error message about an unrecognised option.

Andreas

Re: Binary substituter

[Ludovic Courtès]

Andreas Enge <andreas@enge.fr> skribis:

> Am Donnerstag, 25. April 2013 schrieb Ludovic Courtès:
>> I believe that commit 1c9e7d6 partly solves that.  That is, if you
>> ‘rm -rf /nix/var/nix/substitute-binary/’, and then run
>> ‘guix build hello -n’, it should report things that it would download.
>
> Yes, it tells it will download hello. And the cache directory is filled 
> with data containing the urls of packages to download.

Good.

>> However, if you type ‘guix build hello’, it will barf in ‘filtered-port’
>> when trying to actually download the thing.
>
> Maybe. Here is the output:
>
> the following file will be downloaded:
>    /nix/store/w2121wnp8xv3ycjsgj3ymhb147mrgpc9-hello-2.8
> @ substituter-started /nix/store/w2121wnp8xv3ycjsgj3ymhb147mrgpc9-hello-2.8 
> /usr/local/guix-git/libexec/guix/substitute-binary
> guix substitute-binary: warning: using Guile 2.0.5-deb+1-3, which does not 
> support HTTP () encoding
> guix substitute-binary: error: download failed; use a newer Guile

Oh, it’s not failing where I was expecting.

Hmm, I see no easy workaround for this one.  Mark?

> @ substituter-failed /nix/store/w2121wnp8xv3ycjsgj3ymhb147mrgpc9-hello-2.8 
> 256 fetching path `/nix/store/w2121wnp8xv3ycjsgj3ymhb147mrgpc9-hello-2.8' 
> failed with exit code 1
> guix build: error: build failed: some substitutes for the outputs of 
> derivation `/nix/store/7b51j955338q2kxj3ssvly0i2pw20z7q-hello-2.8.drv' 
> failed; try `--fallback'
>
> Adding "--fallback" leads to an error message about an unrecognised option.

Yes, it’s actually a ‘nix-store’ option.  We should add that option.

Thanks for testing!

Ludo’.