Re: Proposal: prefetch tarballs in a batch

[ludo@gnu.org (Ludovic Courtès)]

I forgot to comment on the actual patch...

Nikita Karetnikov <nikita@karetnikov.org> skribis:

> +(define (fold-values f acc seen lst)

Use ‘fold2’ from (guix utils) instead.

> +(define (derivations-to-prefetch store drv)
> +  "Return the list of fixed-output derivations that DRV depends on, directly
> +or indirectly."
> +  (define (unique-derivations acc seen lst)
> +    ;; Return two values: the list of unique fixed-output derivations and the
> +    ;; list of seen derivations.
> +    (fold-values (lambda (acc seen drv-input)
> +                   (let ((drv* (call-with-input-file (derivation-input-path drv-input)
> +                                 read-derivation)))
> +                     (cond ((fixed-output-derivation? drv*)
> +                            (values (lset-adjoin equal? acc drv*)
> +                                    seen))
> +                           ((member drv* seen)
> +                            (values acc seen))
> +                           (else
> +                            (unique-derivations acc
> +                                                (cons drv* seen)
> +                                                (derivation-inputs drv*))))))
> +                 acc
> +                 seen
> +                 lst))
> +
> +  (identity  ; discard the second value
> +   (unique-derivations '() '() (derivation-inputs drv))))

Can’t it be simplified along these lines:

--8<---------------cut here---------------start------------->8---
scheme@(guile-user)> (define (derivation-input->derivation input) 
		       (call-with-input-file (derivation-input-path input)
			 read-derivation))
scheme@(guile-user)> (filter fixed-output-derivation?
			     (map derivation-input->derivation (derivation-prerequisites $6)))
$10 = (#<derivation /gnu/store/slmkaqybzszjxbl78ymp1dn9c79m9kmy-linux-libre-3.3.8-gnu.tar.xz.drv => /gnu/store/zgh2nvvxkwvmijchf4gyrqb4cq11znvd-linux-libre-3.3.8-gnu.tar.xz 3cbc2d0> #<derivation /gnu/store/c41c2g3g54ipwf5ja3wip5wc...
--8<---------------cut here---------------end--------------->8---

> +;; XXX: remove me.
> +(define specification->package+output
> +  (@@ (guix scripts package) specification->package+output))

I think ‘specification->package’ from (guix scripts build) should be
used instead (it can be exported from there), because the output part of
the specification isn’t needed here: it would make no sense to type

  guix prefetch glibc:debug

because the source of glibc:debug is the same as that of glibc.

> +  (let ((opts  (parse-options))
> +        (store (open-connection)))
> +    (map (lambda (package)
> +           (format #t "Prefetching the derivations for '~a':~%"
> +                   (package-name package))
> +
> +           (build-derivations
> +            store
> +            (map (lambda (drv)
> +                   ;; (format #t "   ~a~%" (derivation-file-name drv))
> +                   (format #t "   ~a~%" drv)
> +                   drv)
> +                 (derivations-to-prefetch
> +                  store
> +                  (package-derivation store package)))))
> +
> +         (delete-duplicates
> +          (filter-map (match-lambda
> +                       (('argument . value)
> +                        (identity  ; discard the second value
> +                         ;; Check that all VALUEs in the list are valid
> +                         ;; packages before calling 'derivations-to-prefetch'.
> +                         ;; If VALUE is not a valid package,
> +                         ;; 'specification->package+output' will raise an
> +                         ;; error.
> +                         (specification->package+output value)))
> +                       (_ #f))
> +                      (reverse opts))))))

It should be a single ‘build-derivations’ call, to allow for
parallelism.

Thanks!

Ludo’.