Re: Proposal: prefetch tarballs in a batch

[Nikita Karetnikov <nikita@karetnikov.org>]

[-- Attachment #1.1: Type: text/plain, Size: 4507 bytes --]

I’m attaching the version incorporating your suggestions, which was
tested with the empty store.  There are a bunch of problems:

1. It doesn’t seem to prefetch all the needed dependencies.  ‘guix build
   hello’ (without network access) fails after prefetching the said
   package.

2. The substituter fails from time to time.  Note that eight tests fail
   on the machine I used: ‘builders.scm’, ‘utils.scm’, ‘packages.scm’,
   ‘store.scm’, ‘monads.scm’, ‘gexp.scm’, ‘guix-package.sh’,
   ‘guix-register.sh’.  Perhaps we ought to fix the mentioned failures
   first.  Which log files would you like to see?

   $ ./pre-inst-env guix prefetch -n icecat
   substitute-binary: guile: hashtab.c:137: vacuum_weak_hash_table: Assertion `removed <= len' failed.
   Backtrace:
   In ice-9/boot-9.scm:
    157: 15 [catch #t #<catch-closure 9c5bef0> ...]
   In unknown file:
      ?: 14 [apply-smob/1 #<catch-closure 9c5bef0>]
   In ice-9/boot-9.scm:
     63: 13 [call-with-prompt prompt0 ...]
   In ice-9/eval.scm:
    432: 12 [eval # #]
   In ice-9/boot-9.scm:
   2320: 11 [save-module-excursion #<procedure 9b8ae60 at ice-9/boot-9.scm:3961:3 ()>]
   3966: 10 [#<procedure 9b8ae60 at ice-9/boot-9.scm:3961:3 ()>]
   1645: 9 [%start-stack load-stack ...]
   1650: 8 [#<procedure 9c5c960 ()>]
   In unknown file:
      ?: 7 [primitive-load "/home/nikita/guix/guix-savannah/scripts/guix"]
   In guix/ui.scm:
    630: 6 [run-guix-command prefetch "-n" "icecat"]
   In ice-9/eval.scm:
    432: 5 [eval # #]
   In guix/ui.scm:
    265: 4 [show-what-to-build #<build-daemon 256.14 a369660> (# # # ...) ...]
   In guix/utils.scm:
    667: 3 [loop (# # # # ...) () (# # # # ...)]
   In guix/ui.scm:
    267: 2 [#<procedure c6fc288 at guix/ui.scm:265:24 (drv build download)> # # ()]
   In guix/derivations.scm:
    175: 1 [derivation-prerequisites-to-build # # # ...]
   In guix/store.scm:
    695: 0 [substitutable-paths # #]

   guix/store.scm:695:2: In procedure substitutable-paths:
   guix/store.scm:695:2: Throw to key `srfi-34' with args `(#<condition &nix-protocol-error [message: "substituter `substitute-binary' died unexpectedly" status: 1] be0d570>)'.

   I’ve also seen this one.  In case it matters, that was before running
   ‘chgrp 1001 /gnu/store; chmod 1775 /gnu/store’.

   $ ./pre-inst-env guix prefetch -n gnunet
   Backtrace:
   In ice-9/boot-9.scm:
    157: 17 [catch #t #<catch-closure 9319ef0> ...]
   In unknown file:
      ?: 16 [apply-smob/1 #<catch-closure 9319ef0>]
   In ice-9/boot-9.scm:
     63: 15 [call-with-prompt prompt0 ...]
   In ice-9/eval.scm:
    432: 14 [eval # #]
   In ice-9/boot-9.scm:
   2320: 13 [save-module-excursion #<procedure 9289e60 at ice-9/boot-9.scm:3961:3 ()>]
   3966: 12 [#<procedure 9289e60 at ice-9/boot-9.scm:3961:3 ()>]
   1645: 11 [%start-stack load-stack ...]
   1650: 10 [#<procedure 931a960 ()>]
   In unknown file:
      ?: 9 [primitive-load "/home/nikita/guix/guix-savannah/scripts/guix"]
   In guix/ui.scm:
    630: 8 [run-guix-command prefetch "-n" "gnunet"]
   In ice-9/eval.scm:
    432: 7 [eval # #]
   In guix/ui.scm:
    265: 6 [show-what-to-build #<build-daemon 256.14 9a29660> (# # # ...) ...]
   In guix/utils.scm:
    667: 5 [loop () () ...]
   In guix/ui.scm:
    267: 4 [#<procedure aab63c0 at guix/ui.scm:265:24 (drv build download)> # () ()]
   In guix/derivations.scm:
    175: 3 [derivation-prerequisites-to-build # # # ...]
   In guix/store.scm:
    695: 2 [substitutable-paths # #]
    392: 1 [process-stderr #<build-daemon 256.14 9a29660> #f]
   In guix/serialization.scm:
     51: 0 [read-int #<input-output: socket 11>]

   guix/serialization.scm:51:4: In procedure read-int:
   guix/serialization.scm:51:4: In procedure bv-u32-ref: Wrong type argument in position 1 (expecting bytevector): #<eof>

3. When using the substituter, the command takes much more time.  Do
   we even need it in this case?  I seem to recall that the GNUnet
   tarball was served by Hydra, but I forgot the details.

   $ time ./pre-inst-env guix prefetch -n icecat
   The following derivations would be built:
      […]
   101987 operations

   real	4m7.477s
   user	0m19.249s
   sys	0m4.848s

   $ time ./pre-inst-env guix prefetch -n --no-substitutes icecat
   The following derivations would be built:
      […]
   101675 operations

   real	0m22.933s
   user	0m14.745s
   sys	0m3.168s


[-- Attachment #1.2: prefetch.patch.8 --]
[-- Type: text/x-diff, Size: 5975 bytes --]

diff --git a/guix/scripts/build.scm b/guix/scripts/build.scm
index 35b10a0..8937d76 100644
--- a/guix/scripts/build.scm
+++ b/guix/scripts/build.scm
@@ -37,6 +37,7 @@
   #:export (%standard-build-options
             set-build-options-from-command-line
             show-build-options-help
+            specification->package
 
             guix-build))
 
diff --git a/guix/scripts/prefetch.scm b/guix/scripts/prefetch.scm
new file mode 100644
index 0000000..dbcef0f
--- /dev/null
+++ b/guix/scripts/prefetch.scm
@@ -0,0 +1,141 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2014 Nikita Karetnikov <nikita@karetnikov.org>
+;;; Copyright © 2014 Ludovic Courtès <ludo@gnu.org>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix scripts prefetch)
+  #:use-module (guix derivations)
+  #:use-module (guix packages)
+  #:use-module (guix store)
+  #:use-module (guix ui)
+  #:use-module (guix utils)
+  #:use-module (guix scripts build)
+  #:use-module (ice-9 match)
+  #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-11)
+  #:use-module (srfi srfi-26)
+  #:use-module (srfi srfi-37)
+  #:export (guix-prefetch))
+
+;;; Commentary:
+;;;
+;;; This program is used to download and add to the store all inputs that are
+;;; needed to build the specified packages.
+;;;
+;;; Code.
+
+(define (derivation-input->derivation input)
+  (call-with-input-file (derivation-input-path input)
+    read-derivation))
+
+(define* (derivations-to-prefetch store package #:key (use-substitutes? #t))
+  "Return the list of fixed-output derivations for PACKAGE."
+  (filter fixed-output-derivation?
+          (map derivation-input->derivation
+               (derivation-prerequisites-to-build
+                store
+                (package-derivation store package)
+                #:use-substitutes? use-substitutes?))))
+
+\f
+;;;
+;;; Command-line options.
+;;;
+
+(define %default-options
+  '((substitutes? . #t)))
+
+(define (show-help)
+  (display (_ "Usage: guix prefetch [OPTION]... PACKAGES...
+Download and add to the store all inputs that are needed to build
+PACKAGES.\n"))
+  (display (_ "
+  -n, --dry-run          do not build the derivations"))
+  (display (_ "
+      --no-substitutes   build instead of resorting to pre-built substitutes"))
+  (display (_ "
+  -h, --help             display this help and exit"))
+  (display (_ "
+  -V, --version          display version information and exit"))
+  (newline)
+  (show-bug-report-information))
+
+(define %options
+  ;; Specification of the command-line options.
+  (list (option '("no-substitutes") #f #f
+                (lambda (opt name arg result . rest)
+                  (apply values
+                         (alist-cons 'substitutes? #f
+                                     (alist-delete 'substitutes? result))
+                         rest)))
+        (option '(#\n "dry-run") #f #f
+                (lambda (opt name arg result)
+                  (alist-cons 'dry-run? #t result)))
+
+        (option '(#\h "help") #f #f
+                (lambda args
+                  (show-help)
+                  (exit 0)))
+        (option '(#\V "version") #f #f
+                (lambda args
+                  (show-version-and-exit "guix prefetch")))))
+
+\f
+;;;
+;;; Entry point.
+;;;
+
+(define (guix-prefetch . args)
+  (define (parse-options)
+    ;; Return the alist of option values.
+    (args-fold* args %options
+                (lambda (opt name arg result)
+                  (leave (_ "~A: unrecognized option~%") name))
+                (lambda (arg result)
+                  (alist-cons 'argument arg result))
+                %default-options))
+
+  (let* ((opts         (parse-options))
+         (dry-run?     (assoc-ref opts 'dry-run?))
+         (substitutes? (assoc-ref opts 'substitutes?))
+         (store        (open-connection))
+         (drvs
+          (append-map (lambda (package)
+                        (derivations-to-prefetch
+                         store
+                         package
+                         #:use-substitutes? substitutes?))
+
+                      (delete-duplicates
+                       (filter-map (match-lambda
+                                    (('argument . value)
+                                     (identity  ; discard the second value
+                                      ;; Check that all VALUEs in the list are
+                                      ;; valid packages before calling
+                                      ;; 'derivations-to-prefetch'.  If VALUE
+                                      ;; is not a valid package,
+                                      ;; 'specification->package' will raise
+                                      ;; an error.
+                                      (specification->package value)))
+                                    (_ #f))
+                                   (reverse opts))))))
+    (show-what-to-build store
+                        drvs
+                        #:dry-run? dry-run?
+                        #:use-substitutes? substitutes?)
+    (unless dry-run?
+      (build-derivations store drvs))))

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]