all messages for Guix-related lists mirrored at yhetil.org
 help / color / mirror / code / Atom feed
From: Nikita Karetnikov <nikita@karetnikov.org>
To: "Ludovic Courtès" <ludo@gnu.org>
Cc: guix-devel@gnu.org
Subject: Re: Proposal: prefetch tarballs in a batch
Date: Tue, 06 May 2014 16:37:21 +0400	[thread overview]
Message-ID: <87r447t75q.fsf@karetnikov.org> (raw)
In-Reply-To: <874n1haa47.fsf@gnu.org> ("Ludovic Courtès"'s message of "Fri, 25 Apr 2014 14:04:24 +0200")


[-- Attachment #1.1: Type: text/plain, Size: 4507 bytes --]

I’m attaching the version incorporating your suggestions, which was
tested with the empty store.  There are a bunch of problems:

1. It doesn’t seem to prefetch all the needed dependencies.  ‘guix build
   hello’ (without network access) fails after prefetching the said
   package.

2. The substituter fails from time to time.  Note that eight tests fail
   on the machine I used: ‘builders.scm’, ‘utils.scm’, ‘packages.scm’,
   ‘store.scm’, ‘monads.scm’, ‘gexp.scm’, ‘guix-package.sh’,
   ‘guix-register.sh’.  Perhaps we ought to fix the mentioned failures
   first.  Which log files would you like to see?

   $ ./pre-inst-env guix prefetch -n icecat
   substitute-binary: guile: hashtab.c:137: vacuum_weak_hash_table: Assertion `removed <= len' failed.
   Backtrace:
   In ice-9/boot-9.scm:
    157: 15 [catch #t #<catch-closure 9c5bef0> ...]
   In unknown file:
      ?: 14 [apply-smob/1 #<catch-closure 9c5bef0>]
   In ice-9/boot-9.scm:
     63: 13 [call-with-prompt prompt0 ...]
   In ice-9/eval.scm:
    432: 12 [eval # #]
   In ice-9/boot-9.scm:
   2320: 11 [save-module-excursion #<procedure 9b8ae60 at ice-9/boot-9.scm:3961:3 ()>]
   3966: 10 [#<procedure 9b8ae60 at ice-9/boot-9.scm:3961:3 ()>]
   1645: 9 [%start-stack load-stack ...]
   1650: 8 [#<procedure 9c5c960 ()>]
   In unknown file:
      ?: 7 [primitive-load "/home/nikita/guix/guix-savannah/scripts/guix"]
   In guix/ui.scm:
    630: 6 [run-guix-command prefetch "-n" "icecat"]
   In ice-9/eval.scm:
    432: 5 [eval # #]
   In guix/ui.scm:
    265: 4 [show-what-to-build #<build-daemon 256.14 a369660> (# # # ...) ...]
   In guix/utils.scm:
    667: 3 [loop (# # # # ...) () (# # # # ...)]
   In guix/ui.scm:
    267: 2 [#<procedure c6fc288 at guix/ui.scm:265:24 (drv build download)> # # ()]
   In guix/derivations.scm:
    175: 1 [derivation-prerequisites-to-build # # # ...]
   In guix/store.scm:
    695: 0 [substitutable-paths # #]

   guix/store.scm:695:2: In procedure substitutable-paths:
   guix/store.scm:695:2: Throw to key `srfi-34' with args `(#<condition &nix-protocol-error [message: "substituter `substitute-binary' died unexpectedly" status: 1] be0d570>)'.

   I’ve also seen this one.  In case it matters, that was before running
   ‘chgrp 1001 /gnu/store; chmod 1775 /gnu/store’.

   $ ./pre-inst-env guix prefetch -n gnunet
   Backtrace:
   In ice-9/boot-9.scm:
    157: 17 [catch #t #<catch-closure 9319ef0> ...]
   In unknown file:
      ?: 16 [apply-smob/1 #<catch-closure 9319ef0>]
   In ice-9/boot-9.scm:
     63: 15 [call-with-prompt prompt0 ...]
   In ice-9/eval.scm:
    432: 14 [eval # #]
   In ice-9/boot-9.scm:
   2320: 13 [save-module-excursion #<procedure 9289e60 at ice-9/boot-9.scm:3961:3 ()>]
   3966: 12 [#<procedure 9289e60 at ice-9/boot-9.scm:3961:3 ()>]
   1645: 11 [%start-stack load-stack ...]
   1650: 10 [#<procedure 931a960 ()>]
   In unknown file:
      ?: 9 [primitive-load "/home/nikita/guix/guix-savannah/scripts/guix"]
   In guix/ui.scm:
    630: 8 [run-guix-command prefetch "-n" "gnunet"]
   In ice-9/eval.scm:
    432: 7 [eval # #]
   In guix/ui.scm:
    265: 6 [show-what-to-build #<build-daemon 256.14 9a29660> (# # # ...) ...]
   In guix/utils.scm:
    667: 5 [loop () () ...]
   In guix/ui.scm:
    267: 4 [#<procedure aab63c0 at guix/ui.scm:265:24 (drv build download)> # () ()]
   In guix/derivations.scm:
    175: 3 [derivation-prerequisites-to-build # # # ...]
   In guix/store.scm:
    695: 2 [substitutable-paths # #]
    392: 1 [process-stderr #<build-daemon 256.14 9a29660> #f]
   In guix/serialization.scm:
     51: 0 [read-int #<input-output: socket 11>]

   guix/serialization.scm:51:4: In procedure read-int:
   guix/serialization.scm:51:4: In procedure bv-u32-ref: Wrong type argument in position 1 (expecting bytevector): #<eof>

3. When using the substituter, the command takes much more time.  Do
   we even need it in this case?  I seem to recall that the GNUnet
   tarball was served by Hydra, but I forgot the details.

   $ time ./pre-inst-env guix prefetch -n icecat
   The following derivations would be built:
      […]
   101987 operations

   real	4m7.477s
   user	0m19.249s
   sys	0m4.848s

   $ time ./pre-inst-env guix prefetch -n --no-substitutes icecat
   The following derivations would be built:
      […]
   101675 operations

   real	0m22.933s
   user	0m14.745s
   sys	0m3.168s


[-- Attachment #1.2: prefetch.patch.8 --]
[-- Type: text/x-diff, Size: 5975 bytes --]

diff --git a/guix/scripts/build.scm b/guix/scripts/build.scm
index 35b10a0..8937d76 100644
--- a/guix/scripts/build.scm
+++ b/guix/scripts/build.scm
@@ -37,6 +37,7 @@
   #:export (%standard-build-options
             set-build-options-from-command-line
             show-build-options-help
+            specification->package
 
             guix-build))
 
diff --git a/guix/scripts/prefetch.scm b/guix/scripts/prefetch.scm
new file mode 100644
index 0000000..dbcef0f
--- /dev/null
+++ b/guix/scripts/prefetch.scm
@@ -0,0 +1,141 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2014 Nikita Karetnikov <nikita@karetnikov.org>
+;;; Copyright © 2014 Ludovic Courtès <ludo@gnu.org>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix scripts prefetch)
+  #:use-module (guix derivations)
+  #:use-module (guix packages)
+  #:use-module (guix store)
+  #:use-module (guix ui)
+  #:use-module (guix utils)
+  #:use-module (guix scripts build)
+  #:use-module (ice-9 match)
+  #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-11)
+  #:use-module (srfi srfi-26)
+  #:use-module (srfi srfi-37)
+  #:export (guix-prefetch))
+
+;;; Commentary:
+;;;
+;;; This program is used to download and add to the store all inputs that are
+;;; needed to build the specified packages.
+;;;
+;;; Code.
+
+(define (derivation-input->derivation input)
+  (call-with-input-file (derivation-input-path input)
+    read-derivation))
+
+(define* (derivations-to-prefetch store package #:key (use-substitutes? #t))
+  "Return the list of fixed-output derivations for PACKAGE."
+  (filter fixed-output-derivation?
+          (map derivation-input->derivation
+               (derivation-prerequisites-to-build
+                store
+                (package-derivation store package)
+                #:use-substitutes? use-substitutes?))))
+
+\f
+;;;
+;;; Command-line options.
+;;;
+
+(define %default-options
+  '((substitutes? . #t)))
+
+(define (show-help)
+  (display (_ "Usage: guix prefetch [OPTION]... PACKAGES...
+Download and add to the store all inputs that are needed to build
+PACKAGES.\n"))
+  (display (_ "
+  -n, --dry-run          do not build the derivations"))
+  (display (_ "
+      --no-substitutes   build instead of resorting to pre-built substitutes"))
+  (display (_ "
+  -h, --help             display this help and exit"))
+  (display (_ "
+  -V, --version          display version information and exit"))
+  (newline)
+  (show-bug-report-information))
+
+(define %options
+  ;; Specification of the command-line options.
+  (list (option '("no-substitutes") #f #f
+                (lambda (opt name arg result . rest)
+                  (apply values
+                         (alist-cons 'substitutes? #f
+                                     (alist-delete 'substitutes? result))
+                         rest)))
+        (option '(#\n "dry-run") #f #f
+                (lambda (opt name arg result)
+                  (alist-cons 'dry-run? #t result)))
+
+        (option '(#\h "help") #f #f
+                (lambda args
+                  (show-help)
+                  (exit 0)))
+        (option '(#\V "version") #f #f
+                (lambda args
+                  (show-version-and-exit "guix prefetch")))))
+
+\f
+;;;
+;;; Entry point.
+;;;
+
+(define (guix-prefetch . args)
+  (define (parse-options)
+    ;; Return the alist of option values.
+    (args-fold* args %options
+                (lambda (opt name arg result)
+                  (leave (_ "~A: unrecognized option~%") name))
+                (lambda (arg result)
+                  (alist-cons 'argument arg result))
+                %default-options))
+
+  (let* ((opts         (parse-options))
+         (dry-run?     (assoc-ref opts 'dry-run?))
+         (substitutes? (assoc-ref opts 'substitutes?))
+         (store        (open-connection))
+         (drvs
+          (append-map (lambda (package)
+                        (derivations-to-prefetch
+                         store
+                         package
+                         #:use-substitutes? substitutes?))
+
+                      (delete-duplicates
+                       (filter-map (match-lambda
+                                    (('argument . value)
+                                     (identity  ; discard the second value
+                                      ;; Check that all VALUEs in the list are
+                                      ;; valid packages before calling
+                                      ;; 'derivations-to-prefetch'.  If VALUE
+                                      ;; is not a valid package,
+                                      ;; 'specification->package' will raise
+                                      ;; an error.
+                                      (specification->package value)))
+                                    (_ #f))
+                                   (reverse opts))))))
+    (show-what-to-build store
+                        drvs
+                        #:dry-run? dry-run?
+                        #:use-substitutes? substitutes?)
+    (unless dry-run?
+      (build-derivations store drvs))))

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

  reply	other threads:[~2014-05-06 12:43 UTC|newest]

Thread overview: 22+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2014-03-27 23:25 Proposal: prefetch tarballs in a batch Nikita Karetnikov
2014-03-28 13:23 ` Ludovic Courtès
2014-04-01 23:23   ` Nikita Karetnikov
2014-04-02 12:57     ` Ludovic Courtès
2014-04-02 22:37       ` Nikita Karetnikov
2014-04-03  7:24         ` Ludovic Courtès
2014-04-09 22:29           ` Nikita Karetnikov
2014-04-10  8:21             ` Ludovic Courtès
2014-04-12 20:19               ` Nikita Karetnikov
2014-04-12 22:44                 ` Ludovic Courtès
2014-04-24 21:20                   ` Nikita Karetnikov
2014-04-25 11:51                     ` Ludovic Courtès
2014-04-25 12:04                     ` Ludovic Courtès
2014-05-06 12:37                       ` Nikita Karetnikov [this message]
2014-05-06 20:02                         ` Ludovic Courtès
2014-05-07 23:14                           ` Nikita Karetnikov
2014-05-08 16:35                             ` Ludovic Courtès
2014-05-11 14:52                               ` Nikita Karetnikov
2014-05-11 19:17                                 ` Ludovic Courtès
2014-09-02 21:06                                   ` Ludovic Courtès
2014-09-02 22:05                                     ` Nikita Karetnikov
2014-09-02 21:04                               ` Ludovic Courtès

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=87r447t75q.fsf@karetnikov.org \
    --to=nikita@karetnikov.org \
    --cc=guix-devel@gnu.org \
    --cc=ludo@gnu.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this external index

	https://git.savannah.gnu.org/cgit/guix.git

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.