From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:470:142:3::10]:40158) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j4SAn-0000TY-Hs for guix-patches@gnu.org; Wed, 19 Feb 2020 11:26:06 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1j4SAk-0007rn-VT for guix-patches@gnu.org; Wed, 19 Feb 2020 11:26:05 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:37345) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1j4SAk-0007rb-JP for guix-patches@gnu.org; Wed, 19 Feb 2020 11:26:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1j4SAk-0005cH-Gl for guix-patches@gnu.org; Wed, 19 Feb 2020 11:26:02 -0500 Subject: [bug#39530] [PATCH] guix: Support partial download Resent-Message-ID: Date: Wed, 19 Feb 2020 11:25:05 -0500 In-Reply-To: <874kvmmulr.fsf@gnu.org> References: <20200209202358.17bb4a39@tachikoma.lepiller.eu> <874kvmmulr.fsf@gnu.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable From: Julien Lepiller Message-ID: <8F45CBA3-9DCE-4BDD-A23E-543F035F87DC@lepiller.eu> List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+kyle=kyleam.com@gnu.org Sender: "Guix-patches" To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: 39530@debbugs.gnu.org Le 19 f=C3=A9vrier 2020 11:04:32 GMT-05:00, "Ludovic Court=C3=A8s" a =C3=A9crit : >Hi, > >Julien Lepiller skribis: > >> First, I make sure that the guix daemon will not remove previously >> failed attempts when trying to build something again, when that is a >> fixed-output derivation=2E Then, I add a Range HTTP header when >> performing an HTTP fetch; this ensures that we only query for the >part >> we don't already have, and append it to the target file=2E >> >> If a partial download fails, the same mirror/url is tried again, but >> the partial file is removed first, ensuring we do a complete fetch >this >> time around=2E If that failed too, we try with the following url=2E If = we >> only perform a complete fetch, we proceed as usual=2E The next url will >> be a partial fetch if there is already something locally=2E > >Nice! > >> However, with that daemon there was a lot of new builds required to >run >> guix environment guix as my user (and nothing was substituted, which >> is weird), whereas with the system's daemon, there was nothing to >> build=2E Maybe there's something fishy in that patch=2E=2E=2E > >Hmm, that sounds really weird=2E Could you clarify what you did? Actually I'm not sure how to test the new daemon=2E I ran the guix daemon = from a checkout and probably forgot to set sysconfdir to /etc, so it was mi= ssing authorized keys, hence no substitutes=2E > >>>>From 332793b7f29ea68ac9a1af22e3d1c4745200da7e Mon Sep 17 00:00:00 >2001 >> From: Julien Lepiller >> Date: Sun, 9 Feb 2020 19:47:27 +0100 >> Subject: [PATCH] guix: download: Add partial download support=2E > >Nitpick: you can remove =E2=80=9Cguix:=E2=80=9D from the subject=2E > >> * nix/libstore/build=2Ecc (tryToBuild): Do not remove invalid >fixed-output >> derivations=2E >> * guix/build/download=2Escm (http-fetch): Add a range argument=2E >> (url-fetch): Performa partial download if a file already exists=2E > >[=2E=2E=2E] > >> -(define* (http-fetch uri #:key timeout (verify-certificate? #t)) >> +(define* (http-fetch uri #:key timeout (verify-certificate? #t) >range) >> "Return an input port containing the data at URI, and the expected >number of >> bytes available or #f=2E When TIMEOUT is true, bail out if the >connection could >> not be established in less than TIMEOUT seconds=2E When >VERIFY-CERTIFICATE? is >> -true, verify HTTPS certificates; otherwise simply ignore them=2E" >> +true, verify HTTPS certificates; otherwise simply ignore them=2E When >RANGE is >> +a number, it is the number of bytes we want to skip from the data at >URI; >> +otherwise the full document is requested=2E" > >I=E2=80=99d suggest to rename #:range to #:offset because it denotes the = start >offset=2E > >What response do we get if the server doesn=E2=80=99t support =E2=80=9CRa= nge=E2=80=9D? > >Can servers silently ignore =E2=80=9CRange=E2=80=9D? > >> + (if (file-exists? file) >> + (http-fetch uri >> + #:verify-certificate? >verify-certificate? >> + #:timeout timeout >> + #:range (stat:size (stat file))) >> + (http-fetch uri >> + #:verify-certificate? >verify-certificate? >> + #:timeout timeout)))) > >I=E2=80=99d remove the =E2=80=98if=E2=80=99: > > (http-fetch =E2=80=A6 > #:offset (and=3D> (stat file #f) stat:size)) > >> --- a/nix/libstore/build=2Ecc >> +++ b/nix/libstore/build=2Ecc >> @@ -1320,6 +1320,7 @@ void DerivationGoal::tryToBuild() >> Path path =3D i->second=2Epath; >> if (worker=2Estore=2EisValidPath(path)) continue; >> if (!pathExists(path)) continue; >> + if (fixedOutput) continue; > >Please add a comment above explaining why fixed outputs are not >deleted=2E > >Also please: not tabs=2E :-) > >Thanks! > >Ludo=E2=80=99=2E Thanks!