From mboxrd@z Thu Jan 1 00:00:00 1970 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) Subject: Re: Proposal: prefetch tarballs in a batch Date: Fri, 25 Apr 2014 14:04:24 +0200 Message-ID: <874n1haa47.fsf@gnu.org> References: <87ha6jkyv8.fsf@karetnikov.org> <877g7epico.fsf@gnu.org> <87wqf8prau.fsf@karetnikov.org> <87ha6blwii.fsf@gnu.org> <87d2gznysk.fsf@karetnikov.org> <87vbuqvpsc.fsf@gnu.org> <87ha62dtmh.fsf@karetnikov.org> <87vbuhbnn8.fsf@gnu.org> <87ioqe2tdg.fsf@karetnikov.org> <87bnw6yxql.fsf@gnu.org> <871twmo25m.fsf@karetnikov.org> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:53838) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Wderr-0006yl-TN for guix-devel@gnu.org; Fri, 25 Apr 2014 08:04:41 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Wderi-0005qq-Rd for guix-devel@gnu.org; Fri, 25 Apr 2014 08:04:35 -0400 Received: from hera.aquilenet.fr ([2a01:474::1]:43700) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Wderi-0005qN-Ek for guix-devel@gnu.org; Fri, 25 Apr 2014 08:04:26 -0400 In-Reply-To: <871twmo25m.fsf@karetnikov.org> (Nikita Karetnikov's message of "Fri, 25 Apr 2014 01:20:21 +0400") List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org To: Nikita Karetnikov Cc: guix-devel@gnu.org I forgot to comment on the actual patch... Nikita Karetnikov skribis: > +(define (fold-values f acc seen lst) Use =E2=80=98fold2=E2=80=99 from (guix utils) instead. > +(define (derivations-to-prefetch store drv) > + "Return the list of fixed-output derivations that DRV depends on, dire= ctly > +or indirectly." > + (define (unique-derivations acc seen lst) > + ;; Return two values: the list of unique fixed-output derivations an= d the > + ;; list of seen derivations. > + (fold-values (lambda (acc seen drv-input) > + (let ((drv* (call-with-input-file (derivation-input-p= ath drv-input) > + read-derivation))) > + (cond ((fixed-output-derivation? drv*) > + (values (lset-adjoin equal? acc drv*) > + seen)) > + ((member drv* seen) > + (values acc seen)) > + (else > + (unique-derivations acc > + (cons drv* seen) > + (derivation-inputs drv*)= ))))) > + acc > + seen > + lst)) > + > + (identity ; discard the second value > + (unique-derivations '() '() (derivation-inputs drv)))) Can=E2=80=99t it be simplified along these lines: --8<---------------cut here---------------start------------->8--- scheme@(guile-user)> (define (derivation-input->derivation input)=20 (call-with-input-file (derivation-input-path input) read-derivation)) scheme@(guile-user)> (filter fixed-output-derivation? (map derivation-input->derivation (derivation-prerequisites $6))) $10 =3D (# /gnu/store/zgh2nvvxkwvmijchf4gyrqb4cq11znvd-li= nux-libre-3.3.8-gnu.tar.xz 3cbc2d0> #8--- > +;; XXX: remove me. > +(define specification->package+output > + (@@ (guix scripts package) specification->package+output)) I think =E2=80=98specification->package=E2=80=99 from (guix scripts build) = should be used instead (it can be exported from there), because the output part of the specification isn=E2=80=99t needed here: it would make no sense to type guix prefetch glibc:debug because the source of glibc:debug is the same as that of glibc. > + (let ((opts (parse-options)) > + (store (open-connection))) > + (map (lambda (package) > + (format #t "Prefetching the derivations for '~a':~%" > + (package-name package)) > + > + (build-derivations > + store > + (map (lambda (drv) > + ;; (format #t " ~a~%" (derivation-file-name drv)) > + (format #t " ~a~%" drv) > + drv) > + (derivations-to-prefetch > + store > + (package-derivation store package))))) > + > + (delete-duplicates > + (filter-map (match-lambda > + (('argument . value) > + (identity ; discard the second value > + ;; Check that all VALUEs in the list are valid > + ;; packages before calling 'derivations-to-pref= etch'. > + ;; If VALUE is not a valid package, > + ;; 'specification->package+output' will raise an > + ;; error. > + (specification->package+output value))) > + (_ #f)) > + (reverse opts)))))) It should be a single =E2=80=98build-derivations=E2=80=99 call, to allow for parallelism. Thanks! Ludo=E2=80=99.