From mboxrd@z Thu Jan 1 00:00:00 1970 From: Nikita Karetnikov Subject: Re: Proposal: prefetch tarballs in a batch Date: Fri, 25 Apr 2014 01:20:21 +0400 Message-ID: <871twmo25m.fsf@karetnikov.org> References: <87ha6jkyv8.fsf@karetnikov.org> <877g7epico.fsf@gnu.org> <87wqf8prau.fsf@karetnikov.org> <87ha6blwii.fsf@gnu.org> <87d2gznysk.fsf@karetnikov.org> <87vbuqvpsc.fsf@gnu.org> <87ha62dtmh.fsf@karetnikov.org> <87vbuhbnn8.fsf@gnu.org> <87ioqe2tdg.fsf@karetnikov.org> <87bnw6yxql.fsf@gnu.org> Mime-Version: 1.0 Content-Type: multipart/signed; boundary="==-=-="; micalg=pgp-sha1; protocol="application/pgp-signature" Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:37873) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1WdUUV-0001GZ-De for guix-devel@gnu.org; Thu, 24 Apr 2014 20:59:56 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1WdR9k-0002rs-S9 for guix-devel@gnu.org; Thu, 24 Apr 2014 17:26:10 -0400 In-Reply-To: <87bnw6yxql.fsf@gnu.org> ("Ludovic =?utf-8?Q?Court=C3=A8s=22'?= =?utf-8?Q?s?= message of "Sun, 13 Apr 2014 00:44:02 +0200") List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org To: Ludovic =?utf-8?Q?Court=C3=A8s?= Cc: guix-devel@gnu.org --==-=-= Content-Type: multipart/mixed; boundary="=-=-=" --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable I=E2=80=99m attaching the patch. Several issues must be addressed before i= t can be pushed: 1. =E2=80=98guix prefetch=E2=80=99 tries to configure and build some packag= es instead of just downloading them. I tried the following on a machine with the empty store: a. Ran =E2=80=98guix prefetch hello=E2=80=99. Some packages were downlo= aded and built (I gathered those are needed to download the other packages, so I didn=E2=80=99t write down the names. If I recall correctly, Gui= le was in that list.) b. Invoked =E2=80=98guix build hello=E2=80=99 without network access. T= he command succeeded. No questions here. c. Called =E2=80=98guix prefetch emacs=E2=80=99. These packages were co= nfigured and built: pkg-config, libunistring, ncurses, libatomic_ops, patch, tar, xz, gzip, zlib, which, gc, perl, m4, readline, libffi, texinfo, libtool, gmp, libtasn1, nettle, guile, gnutls. Are these essential, or is there an error in =E2=80=98guix prefetch= =E2=80=99? According to the build logs, the install prefix of gmp is =E2=80=98/gnu/store/hpaz49xz76p4qmpjc9r33hni1w781y66-gmp-5.1.3=E2=80= =99, which doesn=E2=80=99t appear in the lists provided by =E2=80=98guix prefetc= h hello=E2=80=99 and =E2=80=98guix prefetch emacs=E2=80=99. I=E2=80=99m not sure why it= =E2=80=99s the case. 2. I think it would be nice to output the location of a tarball after running =E2=80=98guix prefetch=E2=80=99, but I=E2=80=99m not sure how to= get the relevant derivation. For =E2=80=98hello=E2=80=99, it=E2=80=99s neither (package-source-derivation store (package-source hello)) nor (package-derivation store hello) because the derivation listed in the initial output of =E2=80=98guix pre= fetch hello=E2=80=99 differs. Of course, it=E2=80=99s possible to filter the = list produced by (build-derivations store (map (lambda (drv) ;; (format #t " ~a~%" (derivation-file-name drv)) (format #t " ~a~%" drv) drv) (derivations-to-prefetch store (package-derivation store package))))) But it=E2=80=99s error-prone, so I=E2=80=99d rather avoid that. 3. I=E2=80=99m planning to add =E2=80=98--dry-run=E2=80=99, which will simp= ly print the list of needed derivations. What options would you like to see implemented? --=-=-= Content-Type: text/plain; charset=utf-8 Content-Disposition: attachment; filename=prefetch.patch.4 Content-Transfer-Encoding: quoted-printable diff --git a/guix/scripts/prefetch.scm b/guix/scripts/prefetch.scm new file mode 100644 index 0000000..16346bb =2D-- /dev/null +++ b/guix/scripts/prefetch.scm @@ -0,0 +1,149 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright =C2=A9 2014 Nikita Karetnikov +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +(define-module (guix scripts prefetch) + #:use-module (guix derivations) + #:use-module (guix packages) + #:use-module (guix store) + #:use-module (guix ui) + #:use-module (guix utils) + #:use-module (srfi srfi-1) + #:use-module (srfi srfi-11) + #:use-module (srfi srfi-26) + #:use-module (srfi srfi-37) + #:use-module (ice-9 match) + #:export (guix-prefetch)) + +;;; Commentary: +;;; +;;; This program is used to download and add to the store all inputs that = are +;;; needed to build the specified packages. +;;; +;;; Code: + +(define (fold-values f acc seen lst) + (if (null-list? lst) + (values acc seen) + (let-values (((acc* seen*) + (f acc seen (first lst)))) + (fold-values f + acc* + seen* + (cdr lst))))) + +(define (derivations-to-prefetch store drv) + "Return the list of fixed-output derivations that DRV depends on, direct= ly +or indirectly." + (define (unique-derivations acc seen lst) + ;; Return two values: the list of unique fixed-output derivations and = the + ;; list of seen derivations. + (fold-values (lambda (acc seen drv-input) + (let ((drv* (call-with-input-file (derivation-input-pat= h drv-input) + read-derivation))) + (cond ((fixed-output-derivation? drv*) + (values (lset-adjoin equal? acc drv*) + seen)) + ((member drv* seen) + (values acc seen)) + (else + (unique-derivations acc + (cons drv* seen) + (derivation-inputs drv*)))= ))) + acc + seen + lst)) + + (identity ; discard the second value + (unique-derivations '() '() (derivation-inputs drv)))) + + +;;; +;;; Command-line options. +;;; + +(define %default-options + '()) + +(define (show-help) + (display (_ "Usage: guix prefetch [OPTION]... PACKAGES... +Download and add to the store all inputs that are needed to build +PACKAGES.\n")) + (display (_ " + -h, --help display this help and exit")) + (display (_ " + -V, --version display version information and exit")) + (newline) + (show-bug-report-information)) + +(define %options + ;; Specification of the command-line options. + (list (option '(#\h "help") #f #f + (lambda args + (show-help) + (exit 0))) + (option '(#\V "version") #f #f + (lambda args + (show-version-and-exit "guix prefetch"))))) + + +;;; +;;; Entry point. +;;; + +;; XXX: remove me. +(define specification->package+output + (@@ (guix scripts package) specification->package+output)) + +(define (guix-prefetch . args) + (define (parse-options) + ;; Return the alist of option values. + (args-fold* args %options + (lambda (opt name arg result) + (leave (_ "~A: unrecognized option~%") name)) + (lambda (arg result) + (alist-cons 'argument arg result)) + %default-options)) + + (let ((opts (parse-options)) + (store (open-connection))) + (map (lambda (package) + (format #t "Prefetching the derivations for '~a':~%" + (package-name package)) + + (build-derivations + store + (map (lambda (drv) + ;; (format #t " ~a~%" (derivation-file-name drv)) + (format #t " ~a~%" drv) + drv) + (derivations-to-prefetch + store + (package-derivation store package))))) + + (delete-duplicates + (filter-map (match-lambda + (('argument . value) + (identity ; discard the second value + ;; Check that all VALUEs in the list are valid + ;; packages before calling 'derivations-to-prefet= ch'. + ;; If VALUE is not a valid package, + ;; 'specification->package+output' will raise an + ;; error. + (specification->package+output value))) + (_ #f)) + (reverse opts)))))) --=-=-=-- --==-=-= Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) iQIcBAEBAgAGBQJTWYAbAAoJEM+IQzI9IQ38nu0P/2FitHp9P7462j1qRzQNBl93 ByT3WEXwYxhmJi//kJX86lv4k5jQNIQFK/PqenIu8SGwpkTjCbVr+3Lpx7RoXGR3 kVyWAc6zdD7SN+8I8bC2mQHd3YeBfgfFp007i0t3ysKPLz8yoU6ijODJZJayTp/1 /e7In0gVwIQDWyGxcoyMpHgdJ/DLTpuyP48aZpTj71YqmM6umd7cdBsUnokMeFXJ 0lyt9IOoFYOjrbXWQjDg/Xf23GLmYQG9MpSS34/fGLZBC71EtRj4WRED9YT4f9cM qTAmEF/K3HKUh5mrhTZPXw8qoJ+YTxpHPhGsSyXN/nfU8ygxxw5uJ6jD2i4n18tI /lTTAWcuKW1a4+7nebXHtlXcEVj7V3FTms3NWsmUI/H6ncyMbOeKfOjAGWJ837PC RY3d9OX3Qy7UoanF/NCrCR+FSd6gAlGW8DHawQMs8+XKkycUA5xcEhdHL5Wu54s2 yb6UYcwA5q5MZGqJu4QIfSMCf5Qqp/Pu0CKnuXLG+wdCWfKS8qp5uQlIMjQJ/RtA irtSk1plLSfYdnVteZQVzl/xJShzPk0buCgBlJWPl6UL9ypa/V7wjb198UNNW038 eBZRh+m724w3NWi9t5c401sluBY70+TMU+g2g7gBLsov96tqTa+MmDnpuoMmIgsq q0bH6xWzaZHntZCObVd6 =sSoc -----END PGP SIGNATURE----- --==-=-=--