From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id KC5UCvy8AGCLFQAA0tVLHw (envelope-from ) for ; Thu, 14 Jan 2021 21:51:56 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id WKYtBvy8AGD1WAAAB5/wlQ (envelope-from ) for ; Thu, 14 Jan 2021 21:51:56 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 7F54D9402AF for ; Thu, 14 Jan 2021 21:51:55 +0000 (UTC) Received: from localhost ([::1]:54536 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l0AX4-0004pb-A6 for larch@yhetil.org; Thu, 14 Jan 2021 16:51:54 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:48478) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l0AWu-0004n8-I3 for guix-devel@gnu.org; Thu, 14 Jan 2021 16:51:44 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:60711) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l0AWt-000373-2e; Thu, 14 Jan 2021 16:51:43 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=53120 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1l0AWs-0005Ib-Ih; Thu, 14 Jan 2021 16:51:42 -0500 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Guillaume Le Vaillant Subject: Re: When substitute download + decompression is CPU-bound References: <87im94qbby.fsf@gnu.org> <94405d66-b13c-e6e6-e8d5-df23b93e5d97@web.de> <87im92voqw.fsf@dismail.de> <87ft3d2fge.fsf@yamatai> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 25 =?utf-8?Q?Niv=C3=B4se?= an 229 de la =?utf-8?Q?R?= =?utf-8?Q?=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Thu, 14 Jan 2021 22:51:40 +0100 In-Reply-To: <87ft3d2fge.fsf@yamatai> (Guillaume Le Vaillant's message of "Thu, 07 Jan 2021 11:45:21 +0100") Message-ID: <87bldr191v.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: guix-devel@gnu.org Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-Spam-Score: -2.86 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Queue-Id: 7F54D9402AF X-Spam-Score: -2.86 X-Migadu-Scanner: scn1.migadu.com X-TUID: AKm3IDizsn8h Hi Guillaume, Guillaume Le Vaillant skribis: > I compared gzip, lzip and zstd when compressing a 580 MB pack (therefore > containing "subsitutes" for several packages) with different compression > levels. Maybe the results can be of some use to someone. It=E2=80=99s insightful, thanks a lot! One takeaway for me is that zstd decompression remains an order of magnitude faster than the others, regardless of the compression level. Another one is that at level 10 and higher zstd achieves compression ratios that are more in the ballpark of lzip. If we are to change the compression methods used at ci.guix.gnu.org, we could use zstd >=3D 10. We could also drop gzip, but there are probably pre-1.1 daemons out there that understand nothing but gzip=C2=B9, so perhaps that=E2=80=99ll ha= ve to wait. Now, compressing substitutes three times may be somewhat unreasonable. Thoughts? Ludo=E2=80=99. =C2=B9 https://guix.gnu.org/en/blog/2020/gnu-guix-1.1.0-released/