From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id wHpDNiS0oGFAAQEAgWs5BA (envelope-from ) for ; Fri, 26 Nov 2021 11:17:08 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id OH0EMiS0oGHafgAA1q6Kng (envelope-from ) for ; Fri, 26 Nov 2021 10:17:08 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id A11D137020 for ; Fri, 26 Nov 2021 11:17:08 +0100 (CET) Received: from localhost ([::1]:51428 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mqYHz-0000ht-Rm for larch@yhetil.org; Fri, 26 Nov 2021 05:17:07 -0500 Received: from eggs.gnu.org ([209.51.188.92]:38804) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mqYHu-0000hV-TG for guix-patches@gnu.org; Fri, 26 Nov 2021 05:17:02 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:47198) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mqYHu-00034c-Kt for guix-patches@gnu.org; Fri, 26 Nov 2021 05:17:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1mqYHu-0002Qu-Ev for guix-patches@gnu.org; Fri, 26 Nov 2021 05:17:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#39728] [PATCH v2] Allow parallel downloads and builds Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 26 Nov 2021 10:17:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 39728 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Julien Lepiller Cc: 39728@debbugs.gnu.org, Pierre Neidhardt , zimoun Received: via spool by 39728-submit@debbugs.gnu.org id=B39728.16379217879304 (code B ref 39728); Fri, 26 Nov 2021 10:17:02 +0000 Received: (at 39728) by debbugs.gnu.org; 26 Nov 2021 10:16:27 +0000 Received: from localhost ([127.0.0.1]:58744 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mqYHK-0002Q0-Oc for submit@debbugs.gnu.org; Fri, 26 Nov 2021 05:16:26 -0500 Received: from eggs.gnu.org ([209.51.188.92]:37922) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mqYHJ-0002Pn-DO for 39728@debbugs.gnu.org; Fri, 26 Nov 2021 05:16:25 -0500 Received: from [2001:470:142:3::e] (port=53614 helo=fencepost.gnu.org) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mqYHD-0002wE-DA; Fri, 26 Nov 2021 05:16:19 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To: From; bh=Bqga2G5FYambx/++7B/qMahni2nVvsRC+QFQxJdLNz0=; b=sb0Ro9FfSxCvX7reu6Vp bRSpU9u/qHeBk90voKC0DEZ6zMsqKyXtRT8Rh3aPyVpGV9uN66cCmpd6rBnZJsTULB/LjanEEKqye izHEKvg1EqxyBGDCHCGVbZRKfOzc+djRAF54qWcUsl0HqqKaavK0d6+K5eRFxlAbOMicWbSoS8AlN 9FMp2gPNgaV7dZCzsl9O+uylce015wAprr0AtMs0v1qb/3LaN5jlNH91/lM0G9Pvq0gwariWr9+JG pnwHnFUxEW+uThicTXVgAcaKub4cQm7Qp26wmh6XABNRhqYkyIj2mLrCCdZknuJ78E+vtOUjMNeHq TAsXp3NS5ObI8w==; Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=49512 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mqYHD-0002j6-6v; Fri, 26 Nov 2021 05:16:19 -0500 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <20200221235307.535fb453@tachikoma.lepiller.eu> <87h7zfel26.fsf@gnu.org> <62A3FE42-52DD-4E51-9AD1-242B86B6A391@lepiller.eu> <87zhd58wjz.fsf@ambrevar.xyz> <20211121234941.32724b68@tachikoma.lepiller.eu> <87ilwge6re.fsf@gnu.org> <20211125140145.1c00decc@tachikoma.lepiller.eu> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 6 Frimaire an 230 de la =?UTF-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Fri, 26 Nov 2021 11:16:16 +0100 In-Reply-To: <20211125140145.1c00decc@tachikoma.lepiller.eu> (Julien Lepiller's message of "Thu, 25 Nov 2021 14:01:45 +0100") Message-ID: <87r1b39q8f.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1637921828; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=Bqga2G5FYambx/++7B/qMahni2nVvsRC+QFQxJdLNz0=; b=nMfsu1s9mHM9eZ7pW4ZT87S0ZcBPetSMzB+tfUylXOdUHefR55rls4H55e90+Fb5dO5djs Jh50cRxorSs+DzAp7Xh7uiiT9Hv7U/bW6QFk3lXti4yRgIgNVLlSzn4MzIcctBxTbIS8XF ahqmGdA3do+lwh94NaTy8/u1vltQV2j0o9zUw3c9vwUQMU9T2+2YLUYKSzTqm9PIPYJ0p4 bzjSmpPK+AN3oBwnfkAsLO6CnW9XVqm8U458zWQ78xaTjYxAN9vT+icF1OWVoCq+DJHb+k JM4NZFOYTezNVlRfPgdxy62yeP1jD/8m8mmNRqsGwzUPko8mSnt/a9zwStG58w== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1637921828; a=rsa-sha256; cv=none; b=j5Va1V1/EG2gfF/wRUSyuwQSDaMSjZNUjVJqm1gatiBm0XYYaO25I7DtiKa0GtE74aDNvK 0JMiejUn+0XkAXvOHxVXqfKmH4jjb4pzzf7SI7F8r5yCrc6iLMYtTAS3GO17xbu8O3OrUL sfRyL+0jk40Vpqa/UKheoXcCLmEjjIYn6boGvnpSJKEdzZ6CfI7jLXcCP0o+ZtUqXJoVgP Ho5Q7NcRhQGxaCEx8qCGx9CqN8r205Aocy9nuOwgYsnxXCJWn3bzqIRUiwVNrpgjSSrjBY wdhFEim2fEVQ7auxTz/hdZZOx8kBTucOh2kE5V5RmFHNmYCM1c42KR4m4e533w== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gnu.org header.s=fencepost-gnu-org header.b=sb0Ro9Ff; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Spam-Score: -2.10 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gnu.org header.s=fencepost-gnu-org header.b=sb0Ro9Ff; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: A11D137020 X-Spam-Score: -2.10 X-Migadu-Scanner: scn1.migadu.com X-TUID: ncQ8wRIIf5bo Hi! Julien Lepiller skribis: > I would still say yes, because the output from berlin is often much > less than my throughput. With multiple downloads in parallel it at > least feels quicker, probably because I can download at full speed. It would be nice to measure that because like I wrote, I think we=E2=80=99re pretty much network-bound these days, at least with zstd and uncompressed downloads. > In any case, I see often a build start while downloads are in progress, > so I think it's still a win if you can get a few derivations built > while waiting for a big download to finish at the same time :) True! Overlapping downloads and builds sounds like a good idea. > At some point we might want to prioritize builds/downloads that help > unlock as much builds as possible early, so we don't have builds > waiting for downloads. Right now the daemon starts with substitutes and builds afterwards. BTW, we=E2=80=99re assuming downloads =3D substitutes in this whole discuss= ion, but we could/should take fixed-output derivations into account too. I=E2=80=99ll take a closer look later on=E2=80=A6 Thanks, Ludo=E2=80=99.