From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id cLqBIbM202CrIQAAgWs5BA (envelope-from ) for ; Wed, 23 Jun 2021 15:27:15 +0200 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id GA80HbM202BBUgAA1q6Kng (envelope-from ) for ; Wed, 23 Jun 2021 13:27:15 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id F07811BC5D for ; Wed, 23 Jun 2021 15:27:14 +0200 (CEST) Received: from localhost ([::1]:40968 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lw2uO-0001Si-VU for larch@yhetil.org; Wed, 23 Jun 2021 09:27:12 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:33700) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lw2uF-0001SZ-2F for bug-guix@gnu.org; Wed, 23 Jun 2021 09:27:03 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:57290) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lw2uE-00030J-Qu for bug-guix@gnu.org; Wed, 23 Jun 2021 09:27:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lw2uE-0007vd-Fw for bug-guix@gnu.org; Wed, 23 Jun 2021 09:27:02 -0400 X-Loop: help-debbugs@gnu.org Subject: bug#49174: Poor 'guix substitute' performance when receiving Zstd-compressed substitutes Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Wed, 23 Jun 2021 13:27:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 49174 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Maxim Cournoyer Received: via spool by 49174-submit@debbugs.gnu.org id=B49174.162445481330462 (code B ref 49174); Wed, 23 Jun 2021 13:27:02 +0000 Received: (at 49174) by debbugs.gnu.org; 23 Jun 2021 13:26:53 +0000 Received: from localhost ([127.0.0.1]:40603 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lw2u5-0007vG-7a for submit@debbugs.gnu.org; Wed, 23 Jun 2021 09:26:53 -0400 Received: from eggs.gnu.org ([209.51.188.92]:35924) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lw2u3-0007v3-H1 for 49174@debbugs.gnu.org; Wed, 23 Jun 2021 09:26:51 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:37170) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lw2ty-0002oP-6m; Wed, 23 Jun 2021 09:26:46 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=33526 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lw2tx-0004kj-Uq; Wed, 23 Jun 2021 09:26:46 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <87fsx93f99.fsf@gmail.com> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 5 Messidor an 229 de la =?UTF-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Wed, 23 Jun 2021 15:26:43 +0200 In-Reply-To: <87fsx93f99.fsf@gmail.com> (Maxim Cournoyer's message of "Tue, 22 Jun 2021 14:50:42 -0400") Message-ID: <87eecsyang.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: 49174@debbugs.gnu.org Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: "bug-Guix" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1624454835; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=h8RqLA7MqQIRJhfUjkYr0st9IQ94aq0yNAJwK/3f2b8=; b=fuP1FrDpwFAg4O3SzY0ONT7Udxfiy+2141Cb7rkAgw3uCWDDieupMnorJ+jm3ZNZOqwdQS mtvYl6/us6bgQpizQwFUdX6UCk5pHcCe7/uDPltNobZbSAsWNPI5080q9uvJhxkYZdTkSz AKLRa1LbBC9ZvQ/0vtGPvGvB8ZFSYnnt6wWItwgMz3g/nAQpix/UWf6tX1FgjLx83sOs8o EuWtkoSj83xm3WBOATnK3DuFh+B0RdmJCJtDOPP2G2HL4WU7ZVrowHRmC0rf8C1594YtHO brTwlbXOai3fDHV1cYJ6HwTsGlqv1+o+ZPz0iPIkXGz3t8v/4Y/bDsSLKmJgPg== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1624454835; a=rsa-sha256; cv=none; b=BLD5OPr8eosrOskVja62AMBJNyMSjjpUgrK9aAZMABvr85GRJrbcvoyU8j+KswXomm/mZj ML7sHBENnnQTZ4RAJW325mqur3FzIowXGCqZv0A08koJlmo4khYP/obfMbhfzUIVneJcUe Y3kRacJcPFkRGAlNP9Tt3ZpW5CN57PQRbUxoNfOhlL+sJcExm737dGRCcIDq2cW0PtjNmz Yjo9vasu23xP9r7QR0/HXLwDWjAkpkvVKBQy+awci2XbY0x2DCW81XLp4yJ3TAGxyCS0lr RjMTqKamntGZHvDyP8T5V6ddMxLsxSjSQ0/asyjX7en4qXQfoou1gmkUXALTqQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; spf=pass (aspmx1.migadu.com: domain of bug-guix-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=bug-guix-bounces@gnu.org X-Migadu-Spam-Score: -2.93 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of bug-guix-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=bug-guix-bounces@gnu.org X-Migadu-Queue-Id: F07811BC5D X-Spam-Score: -2.93 X-Migadu-Scanner: scn1.migadu.com X-TUID: NJ/EshhmRM0R Hi, Maxim Cournoyer skribis: > It's something I've been observing for a while, but substitutes are very > IO intensive (as can be seen in iotop, the substitute process is waiting > on IO > 99% of the time) and is much slower than expected (3 minutes to > transfer 100 MiB uncompressed over a 50 mbps downstream link): > > TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND > 13934 be/4 root 1033.09 K/s 1485.06 K/s 0.00 % 93.36 % guile \ /gnu/= store/vphx2839xv0qj9xwcwrb95592lzrrnx7-guix-1.3.0-3.50dfbbf/bin/guix substi= tute --substitute > > > The publisher (remote machine) is has its guix-daemon configured via: > > (service guix-publish-service-type > (guix-publish-configuration > (advertise? #t) > (compression '(("zstd" 3))) > (host "0.0.0.0"))) ;listen on all interfaces Note that in this case nars are built and compressed on the fly on the server side, which puts an upper bound on the bandwidth you can achieve. I showed earlier how I profiled these things: https://guix.gnu.org/en/blog/2021/getting-bytes-to-disk-more-quickly/ If the client is I/O-bound, that=E2=80=99s good: it means we can=E2=80=99t = do any better (unless we skip unpacking as demonstrated by distri). If you can provide detailed profiles of either the server side or the client side (but in that case, make sure the server is caching things), that=E2=80=99d be great! Otherwise I=E2=80=99m afraid this is not actionable. :-) Thanks, Ludo=E2=80=99.