From mboxrd@z Thu Jan 1 00:00:00 1970 From: Leo Famulari Subject: Re: Parallel downloads Date: Wed, 13 Nov 2019 13:25:55 -0500 Message-ID: <20191113182555.GA3059@jasmine.lan> References: <87pnid3qdh.fsf@ambrevar.xyz> <87k18kyjlm.fsf@nckx> <87wock97ye.fsf@ambrevar.xyz> <87k18h3tii.fsf@gnu.org> <87muddm110.fsf@ambrevar.xyz> <87eeyl10jq.fsf@gnu.org> <875zjxq978.fsf@ambrevar.xyz> <87h83d7xsn.fsf@gnu.org> <87pnhvda3z.fsf@netris.org> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Return-path: Received: from eggs.gnu.org ([2001:470:142:3::10]:35685) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iUxL6-0008FM-RT for guix-devel@gnu.org; Wed, 13 Nov 2019 13:26:01 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iUxL5-0002QG-Ni for guix-devel@gnu.org; Wed, 13 Nov 2019 13:26:00 -0500 Content-Disposition: inline In-Reply-To: <87pnhvda3z.fsf@netris.org> List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: Mark H Weaver Cc: guix-devel@gnu.org On Wed, Nov 13, 2019 at 11:16:53AM -0500, Mark H Weaver wrote: > For these reasons, I'm inclined to think that parallel downloads is the > wrong approach. If a single download process is not making efficient > use of the available bandwidth, I'd be more inclined to look carefully > at why it's failing to do so. For example, I'm not sure if this is the > case (and don't have time to look right now), but if the current code > waits until a NAR has finished downloading before asking for the next > one, that's an issue that could be fixed by use of HTTP pipelining, > without multiplying the memory usage. > > What do you think? I agree that parallel downloads is a kludge to work around the issue of slow set-up and tear-down of our download code. Pipelining would help a lot, and we could also profile the relevant Guile code to see if there are any easy speedups. This issue was actually discussed a year ago: https://lists.gnu.org/archive/html/guix-devel/2018-11/msg00148.html I'll quote Ludo's suggestion from then: > I’d be in favor of a solution where ‘guix substitute’ is kept alive > across substitutions (like what happens with ‘guix substitute --query’), > which would allow it to keep connections alive and thus save the TLS > handshake and a few extra round trips per download.