From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:470:142:3::10]:36285) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j4RqR-0006Ze-QZ for guix-patches@gnu.org; Wed, 19 Feb 2020 11:05:08 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1j4RqQ-0006VD-Jx for guix-patches@gnu.org; Wed, 19 Feb 2020 11:05:03 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:37331) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1j4RqQ-0006V6-Gg for guix-patches@gnu.org; Wed, 19 Feb 2020 11:05:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1j4RqQ-00057N-C6 for guix-patches@gnu.org; Wed, 19 Feb 2020 11:05:02 -0500 Subject: [bug#39530] [PATCH] guix: Support partial download Resent-Message-ID: From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <20200209202358.17bb4a39@tachikoma.lepiller.eu> Date: Wed, 19 Feb 2020 17:04:32 +0100 In-Reply-To: <20200209202358.17bb4a39@tachikoma.lepiller.eu> (Julien Lepiller's message of "Sun, 9 Feb 2020 20:23:58 +0100") Message-ID: <874kvmmulr.fsf@gnu.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+kyle=kyleam.com@gnu.org Sender: "Guix-patches" To: Julien Lepiller Cc: 39530@debbugs.gnu.org Hi, Julien Lepiller skribis: > First, I make sure that the guix daemon will not remove previously > failed attempts when trying to build something again, when that is a > fixed-output derivation. Then, I add a Range HTTP header when > performing an HTTP fetch; this ensures that we only query for the part > we don't already have, and append it to the target file. > > If a partial download fails, the same mirror/url is tried again, but > the partial file is removed first, ensuring we do a complete fetch this > time around. If that failed too, we try with the following url. If we > only perform a complete fetch, we proceed as usual. The next url will > be a partial fetch if there is already something locally. Nice! > However, with that daemon there was a lot of new builds required to run > guix environment guix as my user (and nothing was substituted, which > is weird), whereas with the system's daemon, there was nothing to > build. Maybe there's something fishy in that patch... Hmm, that sounds really weird. Could you clarify what you did? >>>From 332793b7f29ea68ac9a1af22e3d1c4745200da7e Mon Sep 17 00:00:00 2001 > From: Julien Lepiller > Date: Sun, 9 Feb 2020 19:47:27 +0100 > Subject: [PATCH] guix: download: Add partial download support. Nitpick: you can remove =E2=80=9Cguix:=E2=80=9D from the subject. > * nix/libstore/build.cc (tryToBuild): Do not remove invalid fixed-output > derivations. > * guix/build/download.scm (http-fetch): Add a range argument. > (url-fetch): Performa partial download if a file already exists. [...] > -(define* (http-fetch uri #:key timeout (verify-certificate? #t)) > +(define* (http-fetch uri #:key timeout (verify-certificate? #t) range) > "Return an input port containing the data at URI, and the expected num= ber of > bytes available or #f. When TIMEOUT is true, bail out if the connection= could > not be established in less than TIMEOUT seconds. When VERIFY-CERTIFICAT= E? is > -true, verify HTTPS certificates; otherwise simply ignore them." > +true, verify HTTPS certificates; otherwise simply ignore them. When RAN= GE is > +a number, it is the number of bytes we want to skip from the data at URI; > +otherwise the full document is requested." I=E2=80=99d suggest to rename #:range to #:offset because it denotes the st= art offset. What response do we get if the server doesn=E2=80=99t support =E2=80=9CRang= e=E2=80=9D? Can servers silently ignore =E2=80=9CRange=E2=80=9D? > + (if (file-exists? file) > + (http-fetch uri > + #:verify-certificate? verify-certifi= cate? > + #:timeout timeout > + #:range (stat:size (stat file))) > + (http-fetch uri > + #:verify-certificate? verify-certifi= cate? > + #:timeout timeout)))) I=E2=80=99d remove the =E2=80=98if=E2=80=99: (http-fetch =E2=80=A6 #:offset (and=3D> (stat file #f) stat:size)) > --- a/nix/libstore/build.cc > +++ b/nix/libstore/build.cc > @@ -1320,6 +1320,7 @@ void DerivationGoal::tryToBuild() > Path path =3D i->second.path; > if (worker.store.isValidPath(path)) continue; > if (!pathExists(path)) continue; > + if (fixedOutput) continue; Please add a comment above explaining why fixed outputs are not deleted. Also please: not tabs. :-) Thanks! Ludo=E2=80=99.