From mboxrd@z Thu Jan 1 00:00:00 1970 From: Nikita Karetnikov Subject: Re: Proposal: prefetch tarballs in a batch Date: Tue, 06 May 2014 16:37:21 +0400 Message-ID: <87r447t75q.fsf@karetnikov.org> References: <87ha6jkyv8.fsf@karetnikov.org> <877g7epico.fsf@gnu.org> <87wqf8prau.fsf@karetnikov.org> <87ha6blwii.fsf@gnu.org> <87d2gznysk.fsf@karetnikov.org> <87vbuqvpsc.fsf@gnu.org> <87ha62dtmh.fsf@karetnikov.org> <87vbuhbnn8.fsf@gnu.org> <87ioqe2tdg.fsf@karetnikov.org> <87bnw6yxql.fsf@gnu.org> <871twmo25m.fsf@karetnikov.org> <874n1haa47.fsf@gnu.org> Mime-Version: 1.0 Content-Type: multipart/signed; boundary="==-=-="; micalg=pgp-sha1; protocol="application/pgp-signature" Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:33423) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1WheiP-0008Kn-WD for guix-devel@gnu.org; Tue, 06 May 2014 08:43:23 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1WheiN-0005m9-PR for guix-devel@gnu.org; Tue, 06 May 2014 08:43:21 -0400 In-Reply-To: <874n1haa47.fsf@gnu.org> ("Ludovic =?utf-8?Q?Court=C3=A8s=22'?= =?utf-8?Q?s?= message of "Fri, 25 Apr 2014 14:04:24 +0200") List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org To: Ludovic =?utf-8?Q?Court=C3=A8s?= Cc: guix-devel@gnu.org --==-=-= Content-Type: multipart/mixed; boundary="=-=-=" --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable I=E2=80=99m attaching the version incorporating your suggestions, which was tested with the empty store. There are a bunch of problems: 1. It doesn=E2=80=99t seem to prefetch all the needed dependencies. =E2=80= =98guix build hello=E2=80=99 (without network access) fails after prefetching the said package. 2. The substituter fails from time to time. Note that eight tests fail on the machine I used: =E2=80=98builders.scm=E2=80=99, =E2=80=98utils.sc= m=E2=80=99, =E2=80=98packages.scm=E2=80=99, =E2=80=98store.scm=E2=80=99, =E2=80=98monads.scm=E2=80=99, =E2=80=98gexp= .scm=E2=80=99, =E2=80=98guix-package.sh=E2=80=99, =E2=80=98guix-register.sh=E2=80=99. Perhaps we ought to fix the mention= ed failures first. Which log files would you like to see? $ ./pre-inst-env guix prefetch -n icecat substitute-binary: guile: hashtab.c:137: vacuum_weak_hash_table: Asserti= on `removed <=3D len' failed. Backtrace: In ice-9/boot-9.scm: 157: 15 [catch #t # ...] In unknown file: ?: 14 [apply-smob/1 #] In ice-9/boot-9.scm: 63: 13 [call-with-prompt prompt0 ...] In ice-9/eval.scm: 432: 12 [eval # #] In ice-9/boot-9.scm: 2320: 11 [save-module-excursion #] 3966: 10 [#] 1645: 9 [%start-stack load-stack ...] 1650: 8 [#] In unknown file: ?: 7 [primitive-load "/home/nikita/guix/guix-savannah/scripts/guix"] In guix/ui.scm: 630: 6 [run-guix-command prefetch "-n" "icecat"] In ice-9/eval.scm: 432: 5 [eval # #] In guix/ui.scm: 265: 4 [show-what-to-build # (# # # ...) .= ..] In guix/utils.scm: 667: 3 [loop (# # # # ...) () (# # # # ...)] In guix/ui.scm: 267: 2 [#= # # ()] In guix/derivations.scm: 175: 1 [derivation-prerequisites-to-build # # # ...] In guix/store.scm: 695: 0 [substitutable-paths # #] guix/store.scm:695:2: In procedure substitutable-paths: guix/store.scm:695:2: Throw to key `srfi-34' with args `(#)'. I=E2=80=99ve also seen this one. In case it matters, that was before ru= nning =E2=80=98chgrp 1001 /gnu/store; chmod 1775 /gnu/store=E2=80=99. $ ./pre-inst-env guix prefetch -n gnunet Backtrace: In ice-9/boot-9.scm: 157: 17 [catch #t # ...] In unknown file: ?: 16 [apply-smob/1 #] In ice-9/boot-9.scm: 63: 15 [call-with-prompt prompt0 ...] In ice-9/eval.scm: 432: 14 [eval # #] In ice-9/boot-9.scm: 2320: 13 [save-module-excursion #] 3966: 12 [#] 1645: 11 [%start-stack load-stack ...] 1650: 10 [#] In unknown file: ?: 9 [primitive-load "/home/nikita/guix/guix-savannah/scripts/guix"] In guix/ui.scm: 630: 8 [run-guix-command prefetch "-n" "gnunet"] In ice-9/eval.scm: 432: 7 [eval # #] In guix/ui.scm: 265: 6 [show-what-to-build # (# # # ...) .= ..] In guix/utils.scm: 667: 5 [loop () () ...] In guix/ui.scm: 267: 4 [#= # () ()] In guix/derivations.scm: 175: 3 [derivation-prerequisites-to-build # # # ...] In guix/store.scm: 695: 2 [substitutable-paths # #] 392: 1 [process-stderr # #f] In guix/serialization.scm: 51: 0 [read-int #] guix/serialization.scm:51:4: In procedure read-int: guix/serialization.scm:51:4: In procedure bv-u32-ref: Wrong type argumen= t in position 1 (expecting bytevector): # 3. When using the substituter, the command takes much more time. Do we even need it in this case? I seem to recall that the GNUnet tarball was served by Hydra, but I forgot the details. $ time ./pre-inst-env guix prefetch -n icecat The following derivations would be built: [=E2=80=A6] 101987 operations real 4m7.477s user 0m19.249s sys 0m4.848s $ time ./pre-inst-env guix prefetch -n --no-substitutes icecat The following derivations would be built: [=E2=80=A6] 101675 operations real 0m22.933s user 0m14.745s sys 0m3.168s --=-=-= Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename=prefetch.patch.8 Content-Transfer-Encoding: quoted-printable diff --git a/guix/scripts/build.scm b/guix/scripts/build.scm index 35b10a0..8937d76 100644 =2D-- a/guix/scripts/build.scm +++ b/guix/scripts/build.scm @@ -37,6 +37,7 @@ #:export (%standard-build-options set-build-options-from-command-line show-build-options-help + specification->package =20 guix-build)) =20 diff --git a/guix/scripts/prefetch.scm b/guix/scripts/prefetch.scm new file mode 100644 index 0000000..dbcef0f =2D-- /dev/null +++ b/guix/scripts/prefetch.scm @@ -0,0 +1,141 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright =C2=A9 2014 Nikita Karetnikov +;;; Copyright =C2=A9 2014 Ludovic Court=C3=A8s +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +(define-module (guix scripts prefetch) + #:use-module (guix derivations) + #:use-module (guix packages) + #:use-module (guix store) + #:use-module (guix ui) + #:use-module (guix utils) + #:use-module (guix scripts build) + #:use-module (ice-9 match) + #:use-module (srfi srfi-1) + #:use-module (srfi srfi-11) + #:use-module (srfi srfi-26) + #:use-module (srfi srfi-37) + #:export (guix-prefetch)) + +;;; Commentary: +;;; +;;; This program is used to download and add to the store all inputs that = are +;;; needed to build the specified packages. +;;; +;;; Code. + +(define (derivation-input->derivation input) + (call-with-input-file (derivation-input-path input) + read-derivation)) + +(define* (derivations-to-prefetch store package #:key (use-substitutes? #t= )) + "Return the list of fixed-output derivations for PACKAGE." + (filter fixed-output-derivation? + (map derivation-input->derivation + (derivation-prerequisites-to-build + store + (package-derivation store package) + #:use-substitutes? use-substitutes?)))) + + +;;; +;;; Command-line options. +;;; + +(define %default-options + '((substitutes? . #t))) + +(define (show-help) + (display (_ "Usage: guix prefetch [OPTION]... PACKAGES... +Download and add to the store all inputs that are needed to build +PACKAGES.\n")) + (display (_ " + -n, --dry-run do not build the derivations")) + (display (_ " + --no-substitutes build instead of resorting to pre-built substitut= es")) + (display (_ " + -h, --help display this help and exit")) + (display (_ " + -V, --version display version information and exit")) + (newline) + (show-bug-report-information)) + +(define %options + ;; Specification of the command-line options. + (list (option '("no-substitutes") #f #f + (lambda (opt name arg result . rest) + (apply values + (alist-cons 'substitutes? #f + (alist-delete 'substitutes? result)) + rest))) + (option '(#\n "dry-run") #f #f + (lambda (opt name arg result) + (alist-cons 'dry-run? #t result))) + + (option '(#\h "help") #f #f + (lambda args + (show-help) + (exit 0))) + (option '(#\V "version") #f #f + (lambda args + (show-version-and-exit "guix prefetch"))))) + + +;;; +;;; Entry point. +;;; + +(define (guix-prefetch . args) + (define (parse-options) + ;; Return the alist of option values. + (args-fold* args %options + (lambda (opt name arg result) + (leave (_ "~A: unrecognized option~%") name)) + (lambda (arg result) + (alist-cons 'argument arg result)) + %default-options)) + + (let* ((opts (parse-options)) + (dry-run? (assoc-ref opts 'dry-run?)) + (substitutes? (assoc-ref opts 'substitutes?)) + (store (open-connection)) + (drvs + (append-map (lambda (package) + (derivations-to-prefetch + store + package + #:use-substitutes? substitutes?)) + + (delete-duplicates + (filter-map (match-lambda + (('argument . value) + (identity ; discard the second value + ;; Check that all VALUEs in the list= are + ;; valid packages before calling + ;; 'derivations-to-prefetch'. If VA= LUE + ;; is not a valid package, + ;; 'specification->package' will rai= se + ;; an error. + (specification->package value))) + (_ #f)) + (reverse opts)))))) + (show-what-to-build store + drvs + #:dry-run? dry-run? + #:use-substitutes? substitutes?) + (unless dry-run? + (build-derivations store drvs)))) --=-=-=-- --==-=-= Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) iQIcBAEBAgAGBQJTaNeEAAoJEM+IQzI9IQ38ztMP/2lNuefpJwmSIokXyOVUbRFL ZIYRgeJCo7ihkJ5QLFFGGBUBe425Bg9BIV09IFou6LgmqgMuDbIRI8y1ish34inR qL2aqm/Pum5K3GkN30G/9KKpJB7kB2GHrfC4ZCvJFKhDQm6WYJSsavzwlVgdF3W9 yw3CG5uaw1PMFdtP8gobazaraDcQ0e93i6hawefae4zj0me1qhnh0BZ5khFdXmdj sdbIvcMN5hTC6+vCZ811EHkLDmWjkOpj+4GI7I9MGtFkHl+OeOXTa+H6UGCj1j2B FX9FyfdeD6p6klJVp3B+qPLmoUeX7drtgJeUn114w49sJr6Za9TD8mZWewZ4lzRt 2KM72Jn3DGax777nu26v0vrnaaM/zSaIZEnmHzhBacmYPHoeR4/c2u18bb3/l31T aT18yOjZF49a8sgLwkRV2Q4YLoRhjzkonO5c0YyUMNs0mhO9NFc7e4PabDSh8xCA So3RKORrMrIc5mjtlQSVmaik1MZv7IO6iQrnps7l+AZnAg468m6pzPGCTFJagp6C K7qgEw6cSbQRd2F4LK49fTdX10TN//Fe/0ufT2xdrzpEKcP9/z7wJ+fQrOzmnJXn 74tnDbkfAxrFsbYYMKjq91xNQxGPelJddU4kPSbel/Q11/kLXtxPjBjM/3ExCnsz FxFXGZ+zwsqtd2VO1uu9 =eMky -----END PGP SIGNATURE----- --==-=-=--