From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id YEzdJSYx0mDKQQEAgWs5BA (envelope-from ) for ; Tue, 22 Jun 2021 20:51:18 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id KHKPISYx0mCPYAAAB5/wlQ (envelope-from ) for ; Tue, 22 Jun 2021 18:51:18 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id DC56424168 for ; Tue, 22 Jun 2021 20:51:17 +0200 (CEST) Received: from localhost ([::1]:43536 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lvlUS-0003d6-Uk for larch@yhetil.org; Tue, 22 Jun 2021 14:51:16 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:44192) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lvlUE-0003b0-Gc for bug-guix@gnu.org; Tue, 22 Jun 2021 14:51:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:56569) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lvlUE-0005zP-7j for bug-guix@gnu.org; Tue, 22 Jun 2021 14:51:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lvlUE-0007i6-8K for bug-guix@gnu.org; Tue, 22 Jun 2021 14:51:02 -0400 X-Loop: help-debbugs@gnu.org Subject: bug#49174: Poor 'guix substitute' performance when receiving Zstd-compressed substitutes Resent-From: Maxim Cournoyer Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Tue, 22 Jun 2021 18:51:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 49174 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: 49174@debbugs.gnu.org X-Debbugs-Original-To: bug-guix Received: via spool by submit@debbugs.gnu.org id=B.162438784929618 (code B ref -1); Tue, 22 Jun 2021 18:51:02 +0000 Received: (at submit) by debbugs.gnu.org; 22 Jun 2021 18:50:49 +0000 Received: from localhost ([127.0.0.1]:39882 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lvlU1-0007he-9E for submit@debbugs.gnu.org; Tue, 22 Jun 2021 14:50:49 -0400 Received: from lists.gnu.org ([209.51.188.17]:58646) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lvlTz-0007hW-Nr for submit@debbugs.gnu.org; Tue, 22 Jun 2021 14:50:48 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:44130) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lvlTz-00037z-DF for bug-guix@gnu.org; Tue, 22 Jun 2021 14:50:47 -0400 Received: from mail-qt1-x831.google.com ([2607:f8b0:4864:20::831]:39775) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lvlTx-0005oB-Oq for bug-guix@gnu.org; Tue, 22 Jun 2021 14:50:47 -0400 Received: by mail-qt1-x831.google.com with SMTP id 17so213411qtt.6 for ; Tue, 22 Jun 2021 11:50:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:mime-version; bh=o2RRDeRl+CBINwa9K6J/YA6WHkrgnD5KAggvM6kcAW4=; b=WtAIf7c5u/MG0Z+xUnrNjjPDHQyzD9k7yKj0Qv+dJ8roq9B9hoJWDgDJLJ12a2s3Pp zCXrawuUjjnBBn8uv/wj/c/s0ApBlGHAAmZGOrICw1MiDTBJLdN98SnH9+FMida9kS5E dhCO45FlqF9/bELJTTIRFO00jeYVlMQBSmlmvSO+YVZ2Q0Khk6xqbirQwa4ou1hA3Mse 848HYBECanGWsr6xytLGffBa1TjyRxFZ2iTbRnglrVMw9pXKJeWa3H61fwd28dXcCUKh 92PtpTrnzgLyRTwuRJsV8INVezBp40RdmyUQyn1iFg3afMWGJKHo+DQsQCUM79TCS7i0 cN7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:mime-version; bh=o2RRDeRl+CBINwa9K6J/YA6WHkrgnD5KAggvM6kcAW4=; b=D2WN5N/+3fWTgz2+f5JpQhale15TAv/4auInTcG73hmUh0ro2VQRrmXTdSF43ErlUw ESTB5VBJjxD8naota6I4okDcx+Dmpe5DCMkGYVYAerrHBS4BK0YvgkXIstY3yRlAAZi0 GikvDY8jhFLXdzhyaN5kVBQhG5nZhqZDFaaNkjMEjbK4Sfat4qKF7Qb7QHK31ER4r+Vp 5baWaGGaT/0iavpeKHciRVlYhYFhriHYBbPOSlfszyd644M27hq6/ENVYzy5hYKTKEKC +sVdV4ufB/9+rGzzY0aP44G1eA9NZ+4OCHObCG1mZX1Dvyv3oKAo0MvVv+0W5iQ6f8V5 LAbQ== X-Gm-Message-State: AOAM530AF/rILdL4TntbjRh2irHkfc7zMMCVcL0Pm+qwJtalWVzuYq1L 2p2acMD7GagtkIwA67vLfay0xSqkfIRgiA== X-Google-Smtp-Source: ABdhPJyIz3N//jlnEYm5InkAZaT0dbds848gMsrFbijbTb9GNrwc2p7mF0nwWO2+qQ+GHX4BnGWonA== X-Received: by 2002:ac8:4682:: with SMTP id g2mr221010qto.274.1624387843674; Tue, 22 Jun 2021 11:50:43 -0700 (PDT) Received: from hurd (dsl-205-236-230-118.b2b2c.ca. [205.236.230.118]) by smtp.gmail.com with ESMTPSA id p6sm10151984qkp.80.2021.06.22.11.50.43 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Jun 2021 11:50:43 -0700 (PDT) From: Maxim Cournoyer Date: Tue, 22 Jun 2021 14:50:42 -0400 Message-ID: <87fsx93f99.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain Received-SPF: pass client-ip=2607:f8b0:4864:20::831; envelope-from=maxim.cournoyer@gmail.com; helo=mail-qt1-x831.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: "bug-Guix" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1624387878; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:list-id:list-help:list-unsubscribe:list-subscribe: list-post:dkim-signature; bh=o2RRDeRl+CBINwa9K6J/YA6WHkrgnD5KAggvM6kcAW4=; b=Nf79DpDoq+nQJWs9Cq6U57NQGALlVAZWR7lF2It8bNKukYAR0bGxZMazABIammfofFP7MZ ZJkuDsjvkakY1loiJRKWRUp/TYx6dDDmTQF8zb+0TGDPv81jI4rA0FD5zOXtCFJGd6Yjw0 ZwVF6I7MnnRYYqIINPkvng4812P0eSKLsCP4hE2MF/1qC1z6NDf+Tp6C7ejtEyGdEzRnLQ tE9aS76IizHhFkBimeJZnoU6KmYIN3pqQw9RQwrKSgSM0UAcwqu/nROcUH+aSnP1MLYy5H M8pA0mLueBicl9xy3HzTT0m7kgjOxgowDkLOgoWAzrMIZ1Q7RnDGo9BYjhbIpA== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1624387878; a=rsa-sha256; cv=none; b=RTcoR3N6zOXxo3owxeBNRBIS6wNynj33OitFWaWdBCNL2Ysq1uBuAUJtD/LSGEIe/CZOZ+ XbiwDofhkJcZwF8o+NhV1v2YjATUE5fe+sECdzwaT4lLkVY1vvERHXXEk6ied+789Xikan qrvU/lXKydDmLRePZkWd2j968Vy8b2JVosONwvndIkDCA4/TOihGhm4R4OmcTMEcM0eFKr MbtGQWvAUrEt5yl7xarIudf+lS2+KpmvS1bDu4NxN+kZf+mfkkphrMluaqzeuwn3IB9xIb 0p4H0mPWAWizynkdHCNAIB0+lESTwuscqqxZuqSZeol2MGbSU9C/RS8Ur6BS7Q== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20161025 header.b=WtAIf7c5; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of bug-guix-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=bug-guix-bounces@gnu.org X-Migadu-Spam-Score: -1.32 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20161025 header.b=WtAIf7c5; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of bug-guix-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=bug-guix-bounces@gnu.org X-Migadu-Queue-Id: DC56424168 X-Spam-Score: -1.32 X-Migadu-Scanner: scn0.migadu.com X-TUID: vgj6woqcpbWe Hello, It's something I've been observing for a while, but substitutes are very IO intensive (as can be seen in iotop, the substitute process is waiting on IO > 99% of the time) and is much slower than expected (3 minutes to transfer 100 MiB uncompressed over a 50 mbps downstream link): --8<---------------cut here---------------start------------->8--- TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND 13934 be/4 root 1033.09 K/s 1485.06 K/s 0.00 % 93.36 % guile \ /gnu/store/vphx2839xv0qj9xwcwrb95592lzrrnx7-guix-1.3.0-3.50dfbbf/bin/guix substitute --substitute --8<---------------cut here---------------end--------------->8--- The publisher (remote machine) is has its guix-daemon configured via: --8<---------------cut here---------------start------------->8--- (service guix-publish-service-type (guix-publish-configuration (advertise? #t) (compression '(("zstd" 3))) (host "0.0.0.0"))) ;listen on all interfaces --8<---------------cut here---------------end--------------->8--- CPU is idling during on both sides during the transfer (all the work appears to be in the IO on the receiving end). The above example was observed downloading /gnu/store/4fcwwlv9bzfrraxiz41b4vcv131930fx-libstdc++-doc-9.4.0, but I do not think it is substitute-specific. Maxim