From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1.migadu.com ([2001:41d0:1008:1e59::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms8.migadu.com with LMTPS id 0MN5M4iBnmVMjQAAkFu2QA (envelope-from ) for ; Wed, 10 Jan 2024 12:37:44 +0100 Received: from aspmx1.migadu.com ([2001:41d0:403:58f0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1.migadu.com with LMTPS id eIjGLoiBnmXplAAA62LTzQ (envelope-from ) for ; Wed, 10 Jan 2024 12:37:44 +0100 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gnu.org header.s=fencepost-gnu-org header.b="TXygX/kr"; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gnu.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1704886664; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=za+4Zrpi/3Z9JKaQs1UrpffdBMbkvXPmd94TymkT25Y=; b=IXbTE0MlsZX//r8qE/nYbK4jwwFCK9asiOmZHA3HKzQ+KHK9YCE/7JSCJYpQYjsrLoDUFj iP7SIiiuvOAv2a/IEXhODx2fNQug+Z13rvfO69gEVDOHZOv/Lp0EkyaDwr021/r9njmh7/ bWEK58alN8D/wLtTelEYS1lsjS3U0KzLgUN0gF8R1Z8mt4Cr8JLjB6mJu+YuU1niIcOM35 GjZI/r1UrydBqm2oyHfL6FOa2BD7PM86RKgQuqaZq8m4GD4Il53U2jCPfiYO/IJtDjnKM3 pLGtp2kegrrclysB+FWzNKo9As6jzaUPVsyvafWfOQmGeB/6PwgLnHVjQu9bJg== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1704886664; a=rsa-sha256; cv=none; b=YSyfIqAJRo5iOdaa4nqCSxYcC6NVPhj3VAHYnRl1jyOcXtYZRGTV9GGqNou/IkVZGxjjCM 9W9QcQJY4AI1Da+icyufi8K/ilUKoIawedvgPEXS7DknaPhp50KB372Jf7sNOs1Cy79RTO kRYL2Jvpu83n+DsejTqFrv9foQ8DWRulnH1IoZWVLIbbmJ6e2WPuQVvL6tM4113hO9qgUK Wq8N1kqxCEoIlKJbCmcQg8aaI+P3IJ27TqzesqUDEpUG2o/kzTzSdie4i0+OFEfSVvDWg/ beeqaxob8biFGGTKNwW65qY4LBTqL8BE6HEugPUUDQCm58HAls+woEJlUJqmpg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gnu.org header.s=fencepost-gnu-org header.b="TXygX/kr"; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gnu.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 47A5A3C929 for ; Wed, 10 Jan 2024 12:37:44 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rNWtK-0001b4-08; Wed, 10 Jan 2024 06:37:02 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rNWtH-0001al-LU; Wed, 10 Jan 2024 06:36:59 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rNWtH-0000JW-B1; Wed, 10 Jan 2024 06:36:59 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To: From; bh=za+4Zrpi/3Z9JKaQs1UrpffdBMbkvXPmd94TymkT25Y=; b=TXygX/kr+qBqnQxwaKaY uGnm2vOG/KHNK65BytG7KmdkwEOGsmAIJlrjzvfcHqnp4mS2ZhyIOr+1wesPM93ib4er2tiBNOctp a9qwiHnRtF+wpogoyofTV5PDhiRBlVHtV4JYqze9rYN/bejfG+HjJ35i2bmuDmp+ErTcsx99Wim6h ZsYqNEFsFDEA2AKxkkRvwTz8a74wjI3HpDf1Wdu4VfzcqSGJ+3OAkDrNyvGbXyg+YgYtFg/47UlLU gTutJZyaaPuHhwCeFL9KrnXmeXvsvYV6j/9ijDvsZMVaorGlnNnIHrdaJLVXfP8bUjPUkc+C5ogo4 9GyGNi7XvM1Efw==; From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Maxim Cournoyer Cc: guix-devel , guix-sysadmin Subject: Re: Proposition to streamline our NAR collection to just zstd-compressed ones In-Reply-To: <87y1cxuddm.fsf@gmail.com> (Maxim Cournoyer's message of "Tue, 09 Jan 2024 21:32:37 -0500") References: <87y1cxuddm.fsf@gmail.com> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: Primidi 21 =?utf-8?Q?Niv=C3=B4se?= an 232 de la =?utf-8?Q?R=C3=A9volution=2C?= jour de la Pierre =?utf-8?B?w6AgcGzDonRyZQ==?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Wed, 10 Jan 2024 12:36:51 +0100 Message-ID: <8734v5v2r0.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: guix-devel-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Spam-Score: -11.02 X-Spam-Score: -11.02 X-Migadu-Queue-Id: 47A5A3C929 X-Migadu-Scanner: mx11.migadu.com X-TUID: ZOFbn+0GLRbk Hello, Maxim Cournoyer skribis: > It's been on my head for quite a bit of time (about 2 years, according > to [0]), to streamline our offering of cached nars. Letting go of gzip > 2 years ago, along a more aggressive garbage collection policy allowed > us to reduce our storage needs by at least 6.5 TiB. I'm proposing to do > the same with our lzip compressed nars, to let go of an additional 3.9 > TiB: Those space savings would be welcome. > The above suggests that zstd compressed nars are about 5% larger than > the lzip ones, which is not big enough to justify carrying both, in my > opinion. In exchange for a little bit more bandwidth, users would have > the nars decompressed much faster with less CPU overhead locally. The difference is slightly higher, with lzip being 8% smaller, for a big package like ungoogled-chromium or icecat: --8<---------------cut here---------------start------------->8--- $ wget -qO- https://ci.guix.gnu.org/7n95j1zlnwzc44azjs7nj8givnzdfs87.narinf= o|grep -B1 ^FileSize Compression: lzip FileSize: 85783483 -- Compression: zstd FileSize: 92796393 $ wget -qO- https://ci.guix.gnu.org/prpjnnnhay0alanmkgjh66vfwjlb98kq.narinf= o|grep -B1 ^FileSize Compression: lzip FileSize: 295991 -- Compression: zstd FileSize: 323456 --8<---------------cut here---------------end--------------->8--- But yeah, even though adaptive compression selection on the client is a minor improvement, whether it warrants the extra space is debatable. > What do you think? Should we go ahead and effect the following simple > change for the Berlin build farm? > > modified hydra/modules/sysadmin/services.scm > @@ -683,7 +683,7 @@ to a selected directory.") > ;; > ;; for the compression ratio/decompression speed > ;; tradeoffs. > - (compression '(("lzip" 9) ("zstd" 19))) > + (compression '(("zstd" 19))) No objection from me, but=E2=80=A6 =E2=80=A6 an important consideration: zstd support was added in 1.3.0, rele= ased in May 2021. >From experience we know that users on foreign distros rarely, if ever, upgrade the daemon (on top of that, upgrading the daemon is non-trivial to someone who initially installed the Debian package, from what I=E2=80=99= ve seen, because one needs to fiddle with the .service file to adjust file names and the likes), and we can be sure that many are still running an old daemon. We spent a lot of time on user support after gzip substitutes had been removed (=E2=80=98guix substitute=E2=80=99 would just = crash) and we must avoid that. (guix store) emits a warning when connecting to an =E2=80=9Cold=E2=80=9D da= emon, but only for daemons older than 2018. We could emit a warning based on whether or not =E2=80=9Cbuiltin:git-download=E2=80=9D is available, but may= be that=E2=80=99s too early? In addition to the warning, we should communicate in advance and make sure our instructions on how to upgrade the daemon are accurate and clear. Thoughts? Ludo=E2=80=99.