From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id eNioFXO6yF+6VgAA0tVLHw (envelope-from ) for ; Thu, 03 Dec 2020 10:14:11 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id EEWPEXO6yF8NZQAA1q6Kng (envelope-from ) for ; Thu, 03 Dec 2020 10:14:11 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 18256940AFD for ; Thu, 3 Dec 2020 10:14:10 +0000 (UTC) Received: from localhost ([::1]:49822 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kklcn-0002tU-So for larch@yhetil.org; Thu, 03 Dec 2020 05:14:09 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:36000) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kklch-0002tG-3Q for guix-patches@gnu.org; Thu, 03 Dec 2020 05:14:03 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:55140) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kklcg-0004Kr-Qt for guix-patches@gnu.org; Thu, 03 Dec 2020 05:14:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1kklcg-0008WQ-Lc for guix-patches@gnu.org; Thu, 03 Dec 2020 05:14:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#45018] [PATCH 0/6] Process and connection reuse for substitutions Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 03 Dec 2020 10:14:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 45018 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 45018@debbugs.gnu.org Cc: Ludovic =?UTF-8?Q?Court=C3=A8s?= X-Debbugs-Original-To: guix-patches@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.160699040732706 (code B ref -1); Thu, 03 Dec 2020 10:14:02 +0000 Received: (at submit) by debbugs.gnu.org; 3 Dec 2020 10:13:27 +0000 Received: from localhost ([127.0.0.1]:38453 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kklc7-0008VS-8e for submit@debbugs.gnu.org; Thu, 03 Dec 2020 05:13:27 -0500 Received: from lists.gnu.org ([209.51.188.17]:43068) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kklc5-0008VK-Gd for submit@debbugs.gnu.org; Thu, 03 Dec 2020 05:13:26 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:35890) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kklc5-0002UA-AV for guix-patches@gnu.org; Thu, 03 Dec 2020 05:13:25 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:40594) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kklc4-00047o-DL; Thu, 03 Dec 2020 05:13:24 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=49148 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1kklc3-0007MC-SA; Thu, 03 Dec 2020 05:13:24 -0500 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Date: Thu, 3 Dec 2020 11:13:14 +0100 Message-Id: <20201203101314.10842-1-ludo@gnu.org> X-Mailer: git-send-email 2.29.2 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN X-Migadu-Spam-Score: -1.78 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: 18256940AFD X-Spam-Score: -1.78 X-Migadu-Scanner: ns3122888.ip-94-23-21.eu X-TUID: uT73p0mhTx3d Hi Guix! The attached patches optimize substitute downloads by: (1) keeping a single ‘guix substitute’ process for all the substitutes (instead of respawning a new one each time a store item is substituted), and (2) reusing connections to substitute servers in between substitutions. (Note that ‘guix publish’ does not keep connections alive, but on ci.guix.gnu.org we run nginx in front of ‘guix publish’, and nginx keeps them alive.) Anecdotally, the effect is a 10% improvement on the wall-clock time of ‘guix build --sources=transitive vim’ on my laptop, from 29s to 26s. Of course it cannot be much better since the rest of the time is spent actually retrieving bits over the network. Overall the impact depends on a number of factors. My laptop has an SSD and I have fiber-to-the-home (FFTH) with low latency: forking and initiating a TLS connection to ci.guix.gnu.org are both quite cheap, so I probably don’t benefit that much. The impact may be more acute on a low-end device. In the ‘guix build --sources=transitive vim’ case, there are a few large tarballs and several small ones. The gain is in the lack of a pause time in between small tarballs. Also this case is quite advantageous: there’s no decompression and no unpacking happening. Anyway, I think it’s a welcome improvement, especially when downloading lots of stuff such as during the initial system installation. Plus, there are more deletions than insertions. :-) Thoughts? Ludo’. Ludovic Courtès (6): daemon: 'Agent' constructor takes a list of environment variables. daemon: Use 'Agent' to spawn 'guix substitute --query'. daemon: Factorize substituter agent spawning. daemon: Run 'guix substitute --substitute' as an agent. substitute: Cache and reuse connections while substituting. daemon: Raise an error if substituter doesn't send the expected hash. guix/http-client.scm | 12 +- guix/progress.scm | 8 +- guix/scripts/substitute.scm | 215 +++++++++++++++++++++++------------- nix/libstore/build.cc | 166 ++++++++++++++-------------- nix/libstore/local-store.cc | 170 +++++++--------------------- nix/libstore/local-store.hh | 25 +---- nix/libutil/util.cc | 6 +- nix/libutil/util.hh | 7 +- tests/substitute.scm | 98 +++++++++------- 9 files changed, 350 insertions(+), 357 deletions(-) -- 2.29.2