From mboxrd@z Thu Jan 1 00:00:00 1970 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= Subject: Re: Parallel downloads Date: Sun, 17 Nov 2019 18:52:13 +0100 Message-ID: <87sgmmmlv6.fsf@gnu.org> References: <87pnid3qdh.fsf@ambrevar.xyz> <87k18kyjlm.fsf@nckx> <87wock97ye.fsf@ambrevar.xyz> <87k18h3tii.fsf@gnu.org> <87muddm110.fsf@ambrevar.xyz> <87eeyl10jq.fsf@gnu.org> <875zjxq978.fsf@ambrevar.xyz> <87h83d7xsn.fsf@gnu.org> <87pnhvda3z.fsf@netris.org> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:470:142:3::10]:42517) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iWOie-0002GJ-Iv for guix-devel@gnu.org; Sun, 17 Nov 2019 12:52:17 -0500 In-Reply-To: <87pnhvda3z.fsf@netris.org> (Mark H. Weaver's message of "Wed, 13 Nov 2019 11:16:53 -0500") List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: Mark H Weaver Cc: guix-devel@gnu.org Hi Mark, Mark H Weaver skribis: > For these reasons, I'm inclined to think that parallel downloads is the > wrong approach. If a single download process is not making efficient > use of the available bandwidth, I'd be more inclined to look carefully > at why it's failing to do so. For example, I'm not sure if this is the > case (and don't have time to look right now), but if the current code > waits until a NAR has finished downloading before asking for the next > one, that's an issue that could be fixed by use of HTTP pipelining, > without multiplying the memory usage. I agree. There=E2=80=99s HTTP pipelining for narinfos but not for nars. W= orse, before fetching a nar, we do a GET /nix-cache-info, and in fact we spawn a new =E2=80=98guix substitute=E2=80=99 process for each download (for =E2= =80=9Chistorical reasons=E2=80=9D). So there=E2=80=99s room for optimization there! Ludo=E2=80=99.