From mboxrd@z Thu Jan 1 00:00:00 1970 From: Mark H Weaver Subject: Re: Parallel downloads Date: Wed, 13 Nov 2019 11:16:53 -0500 Message-ID: <87pnhvda3z.fsf@netris.org> References: <87pnid3qdh.fsf@ambrevar.xyz> <87k18kyjlm.fsf@nckx> <87wock97ye.fsf@ambrevar.xyz> <87k18h3tii.fsf@gnu.org> <87muddm110.fsf@ambrevar.xyz> <87eeyl10jq.fsf@gnu.org> <875zjxq978.fsf@ambrevar.xyz> <87h83d7xsn.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:470:142:3::10]:41686) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iUvLT-0001iy-IA for guix-devel@gnu.org; Wed, 13 Nov 2019 11:18:16 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iUvLS-0005KX-0h for guix-devel@gnu.org; Wed, 13 Nov 2019 11:18:15 -0500 In-Reply-To: <87h83d7xsn.fsf@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\= \=\?utf-8\?Q\?s\?\= message of "Sat, 09 Nov 2019 18:40:56 +0100") List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: Ludovic =?utf-8?Q?Court=C3=A8s?= Cc: guix-devel@gnu.org Hi, Ludovic Court=C3=A8s writes: > Pierre Neidhardt skribis: > >> Ludo, what do you think of Tobias suggestion and have an extra knob to >> specifically configure the number of download jobs? > > Like I wrote, it=E2=80=99s not that simple (we=E2=80=99d first need the d= aemon to > distinguish substitution jobs from other jobs, but note that there are > also =E2=80=9Cdownloads=E2=80=9D that are actually derivation builds), an= d it=E2=80=99s not > clear to me that it=E2=80=99s overall beneficial anyway: it=E2=80=99s not= supposed to be > faster to download 10 things in parallel from ci.guix.gnu.org, than to > download them sequentially. I'll also note that parallel downloads would increase the memory usage on the server. If users, on average, configured 4 parallel downloads, I guess that would have the effect of multiplying the server memory usage by about 4. It might also create an incentive for users to configure more parallel downloads in order to grab a larger ratio of the server's available resources. For these reasons, I'm inclined to think that parallel downloads is the wrong approach. If a single download process is not making efficient use of the available bandwidth, I'd be more inclined to look carefully at why it's failing to do so. For example, I'm not sure if this is the case (and don't have time to look right now), but if the current code waits until a NAR has finished downloading before asking for the next one, that's an issue that could be fixed by use of HTTP pipelining, without multiplying the memory usage. What do you think? Mark