From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp12.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id +KADGPZ8P2MfYwEAbAwnHQ (envelope-from ) for ; Fri, 07 Oct 2022 03:12:22 +0200 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp12.migadu.com with LMTPS id uOfyF/Z8P2OfMAAAauVa8A (envelope-from ) for ; Fri, 07 Oct 2022 03:12:22 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 142469601 for ; Fri, 7 Oct 2022 03:12:22 +0200 (CEST) Received: from localhost ([::1]:55732 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ogbuW-0006Gi-Mw for larch@yhetil.org; Thu, 06 Oct 2022 21:12:20 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:51156) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ogbuI-0006GN-I0 for bug-guix@gnu.org; Thu, 06 Oct 2022 21:12:06 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:34819) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1ogbuE-0002Y6-Ar for bug-guix@gnu.org; Thu, 06 Oct 2022 21:12:06 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1ogbuD-0003S5-RH for bug-guix@gnu.org; Thu, 06 Oct 2022 21:12:01 -0400 Subject: bug#49174: Poor 'guix substitute' performance when receiving Zstd-compressed substitutes Resent-From: Maxim Cournoyer Original-Sender: "Debbugs-submit" Resent-To: bug-guix@gnu.org Resent-Date: Fri, 07 Oct 2022 01:12:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: cc-closed 49174 X-GNU-PR-Package: guix X-GNU-PR-Keywords: moreinfo To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: 49174-done@debbugs.gnu.org Mail-Followup-To: 49174@debbugs.gnu.org, maxim.cournoyer@gmail.com, maxim.cournoyer@gmail.com Received: via spool by 49174-done@debbugs.gnu.org id=D49174.166510509913223 (code D ref 49174); Fri, 07 Oct 2022 01:12:01 +0000 Received: (at 49174-done) by debbugs.gnu.org; 7 Oct 2022 01:11:39 +0000 Received: from localhost ([127.0.0.1]:33895 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ogbtr-0003RD-C8 for submit@debbugs.gnu.org; Thu, 06 Oct 2022 21:11:39 -0400 Received: from mail-qt1-f177.google.com ([209.85.160.177]:37887) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ogbtm-0003Qq-Cr for 49174-done@debbugs.gnu.org; Thu, 06 Oct 2022 21:11:38 -0400 Received: by mail-qt1-f177.google.com with SMTP id l27so2064636qtv.4 for <49174-done@debbugs.gnu.org>; Thu, 06 Oct 2022 18:11:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:user-agent:message-id :in-reply-to:date:references:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=IM4x3rx5PZeqJqnoO06acx5ag93Yd/qVaNtE7ojViP0=; b=EgnGCSwK+QgNEIvPgkpI2wEWBkoics0APDd6YaGP8l3H5wSgzZZFbkjkOCjqeGDY75 0riHOnbRO30zIZA5G2YB4Hm1Bix72kNZW62UNql31cCDUsTBWAGaSpKP1CrjQlJfMsSt FqsHmPEwr2WBNmZmL4nidReT5Q8x/1sx2OACWMKslCeoFIcZXN4Sy8IjEm+DYZbyTNO2 U25+MwckjXYwMevjf7t0Q0Aq7ETIX13BCgUXSPuxJwTX8sZQBsS86Oqi9IzR1jUpxQKb pBOFA/LsFFvG3DwzRGivY62DRUAl1bLxJOwhfgfjWVfdoKmlxnMa5Rhxqv2Ba96kDc23 gqGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:user-agent:message-id :in-reply-to:date:references:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=IM4x3rx5PZeqJqnoO06acx5ag93Yd/qVaNtE7ojViP0=; b=lvVyyDCjsA30XMmqqi9NyInnxYQXce3IjTXkMzNF6Y3U5FE/I7WTAOoUtax71mSdeh xF1JbjoYpmfGotu0mdbAKGEUdTA0QM7pyg7v6sSZunL9kbE8MPdImk7zWVz1OFcmp4mx hXVZdWX5jxIy/0AyVTd90WAQP6eennxY/DQa7IePViNpPrZBeyf+c6rcp9Yi2yZURzMI pglV3alWe8Ybg+OfXAAQNYIt7pkr/2Wsz8qme0ZP8zqbMo1gZYxLuSBlMAC55Xr0nm8E nW1h+5h7RWTLf5ckvkTbv9Q1IUK9if7KYDMPPxHKG8L4pa2ETI3LtJZc7SawtaCmADJL Cgmw== X-Gm-Message-State: ACrzQf1dqWUdC5CTki5jfvtzhMIH9IGpENqdVGHHgXLJRHxEbv+GRNRT L8U9mO11If1vRHzzJJv9zchD2NavUhA= X-Google-Smtp-Source: AMsMyM7OPWYVX2r8spH6+E6SU4tp1fuMn2gQETSvU8ZZhN1Uxdx9NOqBVolhGUZG0BGYvcOHpHgZDQ== X-Received: by 2002:ac8:5745:0:b0:35c:ca39:cc9f with SMTP id 5-20020ac85745000000b0035cca39cc9fmr2588524qtx.670.1665105087574; Thu, 06 Oct 2022 18:11:27 -0700 (PDT) Received: from hurd ([2607:fad8:4:3::1005]) by smtp.gmail.com with ESMTPSA id o19-20020a05620a2a1300b006cddf59a600sm718759qkp.34.2022.10.06.18.11.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 06 Oct 2022 18:11:27 -0700 (PDT) From: Maxim Cournoyer References: <87fsx93f99.fsf@gmail.com> <87eecsyang.fsf@gnu.org> Date: Thu, 06 Oct 2022 21:11:24 -0400 In-Reply-To: <87eecsyang.fsf@gnu.org> ("Ludovic =?UTF-8?Q?Court=C3=A8s?="'s message of "Wed, 23 Jun 2021 15:26:43 +0200") Message-ID: <87ilkw7803.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: "bug-Guix" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1665105142; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-to: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=IM4x3rx5PZeqJqnoO06acx5ag93Yd/qVaNtE7ojViP0=; b=nV5P97E37ZPTupWvqOpnoybDbE1cbc8NxBse7tKkyl4esAmDqkCT0rLB726JOXMVEzM8eN v26rYTCf5tkvInIsbREOixVXkR+v8nPz1CjKQQPIt7gWGbLA4ccsA3TAvL+MwZTMOiCgWE CmB99UxQ+h9BZdyrtRDx2jDExF95uxHAbj6mX+Ni8dmo2+d19nU2tQ2Z+Um+AIrPTRUTJe YeRFYQyZraSJa93dXuIbYoTBmr3gsyfMuOeaNbvu4pVuztY6AY3JUBw9JtphS1MYmhoCzc ojaQUW/xqwjfZt4h19GqOf33glrfuM4IKqH7FCgCRAbXFIjI+N5alT9wgKICjw== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1665105142; a=rsa-sha256; cv=none; b=frURtcsz0tCFK9UKxvlpFWHdNWIMFgzjdOiSSTe7kR7PEVQhNO45pe1jvec6VLEa0Te89A kTT1G/bMy31msgdOYW0VRLm7Mj4oI11I/DHqSZf2se6m1bIOZVRjja1QRsJEASTmo8oPw2 0lId6LYS9If32Ej8PEpTbu+7leGd3Z+ytxugSYhI+vBRnyce5GnamxBahA3a7SsLABDKOx IAKWzunLekrvz16dFw42hFGNEGOx/n2ZcbSbgHDI/+1h2RuHB8/4RFMbbDNYKAUGqHoUj9 2nbc5H/K4VoEw8EZ1bp7fWu7qjJwIYWIaX0K+0gD2beYZ9CloAMXsBhfOOsJWw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20210112 header.b=EgnGCSwK; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: 7.03 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20210112 header.b=EgnGCSwK; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 142469601 X-Spam-Score: 7.03 X-Migadu-Scanner: scn0.migadu.com X-TUID: An0GNZypcbDc Hi, Ludovic Court=C3=A8s writes: > Hi, > > Maxim Cournoyer skribis: > >> It's something I've been observing for a while, but substitutes are very >> IO intensive (as can be seen in iotop, the substitute process is waiting >> on IO > 99% of the time) and is much slower than expected (3 minutes to >> transfer 100 MiB uncompressed over a 50 mbps downstream link): >> >> TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND >> 13934 be/4 root 1033.09 K/s 1485.06 K/s 0.00 % 93.36 % guile \ /gnu= /store/vphx2839xv0qj9xwcwrb95592lzrrnx7-guix-1.3.0-3.50dfbbf/bin/guix subst= itute --substitute >> >> >> The publisher (remote machine) is has its guix-daemon configured via: >> >> (service guix-publish-service-type >> (guix-publish-configuration >> (advertise? #t) >> (compression '(("zstd" 3))) >> (host "0.0.0.0"))) ;listen on all interfaces > > Note that in this case nars are built and compressed on the fly on the > server side, which puts an upper bound on the bandwidth you can achieve. > > I showed earlier how I profiled these things: > > https://guix.gnu.org/en/blog/2021/getting-bytes-to-disk-more-quickly/ > > If the client is I/O-bound, that=E2=80=99s good: it means we can=E2=80=99= t do any better > (unless we skip unpacking as demonstrated by distri). > > If you can provide detailed profiles of either the server side or the > client side (but in that case, make sure the server is caching things), > that=E2=80=99d be great! > > Otherwise I=E2=80=99m afraid this is not actionable. :-) Since moving from HDDs to SSDs, I haven't seen this problem, so I suspect the poor IO of the HDDs was really the culprit rather than something to do with guile-zstd (and we had also benchmarked the late some when I experimented with using zstd-compressed man pages). Closing! --=20 Thanks, Maxim