From mboxrd@z Thu Jan 1 00:00:00 1970 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) Subject: Re: Proposal: prefetch tarballs in a batch Date: Tue, 06 May 2014 22:02:24 +0200 Message-ID: <87r4461xrj.fsf@gnu.org> References: <87ha6jkyv8.fsf@karetnikov.org> <877g7epico.fsf@gnu.org> <87wqf8prau.fsf@karetnikov.org> <87ha6blwii.fsf@gnu.org> <87d2gznysk.fsf@karetnikov.org> <87vbuqvpsc.fsf@gnu.org> <87ha62dtmh.fsf@karetnikov.org> <87vbuhbnn8.fsf@gnu.org> <87ioqe2tdg.fsf@karetnikov.org> <87bnw6yxql.fsf@gnu.org> <871twmo25m.fsf@karetnikov.org> <874n1haa47.fsf@gnu.org> <87r447t75q.fsf@karetnikov.org> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:53394) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1WhlZQ-0003nH-9r for guix-devel@gnu.org; Tue, 06 May 2014 16:02:37 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1WhlZL-0004oQ-0v for guix-devel@gnu.org; Tue, 06 May 2014 16:02:32 -0400 Received: from hera.aquilenet.fr ([2a01:474::1]:36930) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1WhlZK-0004oA-Jh for guix-devel@gnu.org; Tue, 06 May 2014 16:02:26 -0400 In-Reply-To: <87r447t75q.fsf@karetnikov.org> (Nikita Karetnikov's message of "Tue, 06 May 2014 16:37:21 +0400") List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org To: Nikita Karetnikov Cc: guix-devel@gnu.org Nikita Karetnikov skribis: > 1. It doesn=E2=80=99t seem to prefetch all the needed dependencies. =E2= =80=98guix build > hello=E2=80=99 (without network access) fails after prefetching the sa= id > package. Fails how? > 2. The substituter fails from time to time. Note that eight tests fail > on the machine I used: =E2=80=98builders.scm=E2=80=99, =E2=80=98utils.= scm=E2=80=99, =E2=80=98packages.scm=E2=80=99, > =E2=80=98store.scm=E2=80=99, =E2=80=98monads.scm=E2=80=99, =E2=80=98ge= xp.scm=E2=80=99, =E2=80=98guix-package.sh=E2=80=99, > =E2=80=98guix-register.sh=E2=80=99. Perhaps we ought to fix the menti= oned failures > first. Which log files would you like to see? The SRFI-64 log files of the failing tests, plus test-suite.log. Test failures must not remain uncorrected! ;-) When there are so many failures, it=E2=80=99s likely that there=E2=80=99s a= setup issue, like socket names are too long. > $ ./pre-inst-env guix prefetch -n icecat > substitute-binary: guile: hashtab.c:137: vacuum_weak_hash_table: Asser= tion `removed <=3D len' failed. What Guile and libgc version is this, and what platform? > I=E2=80=99ve also seen this one. In case it matters, that was before = running > =E2=80=98chgrp 1001 /gnu/store; chmod 1775 /gnu/store=E2=80=99. > > $ ./pre-inst-env guix prefetch -n gnunet > Backtrace: > In ice-9/boot-9.scm: > 157: 17 [catch #t # ...] [...] > In guix/derivations.scm: > 175: 3 [derivation-prerequisites-to-build # # # ...] > In guix/store.scm: > 695: 2 [substitutable-paths # #] > 392: 1 [process-stderr # #f] > In guix/serialization.scm: > 51: 0 [read-int #] > > guix/serialization.scm:51:4: In procedure read-int: > guix/serialization.scm:51:4: In procedure bv-u32-ref: Wrong type argum= ent in position 1 (expecting bytevector): # That seems similar no? Does the installation seem sane, basically? Do =E2=80=98guix build=E2=80= =99, =E2=80=98guix package=E2=80=99 etc. work somehow, or not even? > 3. When using the substituter, the command takes much more time. Do > we even need it in this case? I seem to recall that the GNUnet > tarball was served by Hydra, but I forgot the details. There=E2=80=99s a local cache of substituter hits/failures in /var/guix/substitute-binary (or similar.) When that cache is empty or outdated, a lot of HTTP queries are made to hydra.gnu.org, which can take a bit of time. When using a recent Guile, these queries are made in parallel. On my machine, it goes like this: --8<---------------cut here---------------start------------->8--- $ sudo rm -rf /var/guix/substitute-binary/cache/ $ time guix build emacs -n real 0m30.551s user 0m2.519s sys 0m0.154s $ time guix build emacs -n real 0m3.381s user 0m2.039s sys 0m0.123s --8<---------------cut here---------------end--------------->8--- The cost is high when starting from an empty cache, but afterwards it=E2=80= =99s small. > +(define (show-help) Add call to =E2=80=98show-build-options-help=E2=80=99 from (guix scripts bu= ild). > +(define %options > + ;; Specification of the command-line options. > + (list (option '("no-substitutes") #f #f > + (lambda (opt name arg result . rest) > + (apply values > + (alist-cons 'substitutes? #f > + (alist-delete 'substitutes? result)) > + rest))) Remove this option, and concatenate with =E2=80=98%standard-build-options= =E2=80=99. > + (filter-map (match-lambda > + (('argument . value) > + (identity ; discard the second val= ue The extra value gets truncated here, so it=E2=80=99s not strictly needed. Thanks! Ludo=E2=80=99.