From mboxrd@z Thu Jan 1 00:00:00 1970 From: Pierre Neidhardt Subject: Re: Parallel downloads Date: Wed, 13 Nov 2019 20:34:45 +0100 Message-ID: <87pnhvfu3u.fsf@ambrevar.xyz> References: <87pnid3qdh.fsf@ambrevar.xyz> <87k18kyjlm.fsf@nckx> <87wock97ye.fsf@ambrevar.xyz> <87k18h3tii.fsf@gnu.org> <87muddm110.fsf@ambrevar.xyz> <87eeyl10jq.fsf@gnu.org> <875zjxq978.fsf@ambrevar.xyz> <87h83d7xsn.fsf@gnu.org> <87pnhvda3z.fsf@netris.org> Mime-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" Return-path: Received: from eggs.gnu.org ([2001:470:142:3::10]:59646) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iUyPj-0005kN-39 for guix-devel@gnu.org; Wed, 13 Nov 2019 14:34:52 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iUyPh-000890-6X for guix-devel@gnu.org; Wed, 13 Nov 2019 14:34:50 -0500 In-Reply-To: <87pnhvda3z.fsf@netris.org> List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: Mark H Weaver , Ludovic =?utf-8?Q?Court=C3=A8s?= Cc: guix-devel@gnu.org --=-=-= Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Mark H Weaver writes: > For these reasons, I'm inclined to think that parallel downloads is the > wrong approach. If a single download process is not making efficient > use of the available bandwidth, I'd be more inclined to look carefully > at why it's failing to do so. For example, I'm not sure if this is the > case (and don't have time to look right now), but if the current code > waits until a NAR has finished downloading before asking for the next > one, that's an issue that could be fixed by use of HTTP pipelining, > without multiplying the memory usage. I think so too. More generally, the Guix daemon jobs can be categorized in 2: downloads and builds. These 2 categories demands almost complementary hardware resources for the local machine. With that in mind, we could have 2 pipelines: one for the builds and one for the downloads. This, I think, is general enough that it could be used by default and improve performance for everyone, regardless of your Internet bandwidth or CPU power. Cheers! =2D-=20 Pierre Neidhardt https://ambrevar.xyz/ --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEUPM+LlsMPZAEJKvom9z0l6S7zH8FAl3MWtUACgkQm9z0l6S7 zH8nSggAnXMi2Ws0xDBiIdYxN/IS6+nIff2d2RCE5iNR6+VLNwn3f8szVKi6g8dc ZzuGVXUHiX1pZW1DQtEn7nVJesX0D3cQ1DCztV7Uj/5jFWE2m5L65EUgnZEtyNCf SWKJ8dUhdcBhBbOf5pBzxXXY1xFuqmDnpsN8RQvw6NkRE2by9Su/+1aRY+rllKpv EG1DcYiDRwhJ6l84CyaCEC8c8zYF+H7BL0Ud1TH78c4f6jkSxouaaPM/P7zkX5gJ bIjEobzFbx0QzQVJQ8D+oPygbabk1KIFzuek2uodGerJi6HbIQRKyN28mavNIND2 xfSXg/ftCeRdPJMEDLVjfoSJ9DBEig== =l3p2 -----END PGP SIGNATURE----- --=-=-=--