From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id UAnhM+Sf2F8FEQAA0tVLHw (envelope-from ) for ; Tue, 15 Dec 2020 11:37:08 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id KDbCL+Sf2F+GHwAAB5/wlQ (envelope-from ) for ; Tue, 15 Dec 2020 11:37:08 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 70DFC940148 for ; Tue, 15 Dec 2020 11:37:08 +0000 (UTC) Received: from localhost ([::1]:34660 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kp8df-0006kh-BV for larch@yhetil.org; Tue, 15 Dec 2020 06:37:07 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:57424) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kp8dQ-0006jV-7D for guix-devel@gnu.org; Tue, 15 Dec 2020 06:36:52 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:49579) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kp8dO-0005VI-Om; Tue, 15 Dec 2020 06:36:51 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=44642 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1kp8dH-0005U0-SS; Tue, 15 Dec 2020 06:36:44 -0500 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: =?utf-8?Q?Nicol=C3=B2?= Balzarotti Subject: Re: When substitute download + decompression is CPU-bound References: <87im94qbby.fsf@gnu.org> <87sg88ngd1.fsf@guixSD.i-did-not-set--mail-host-address--so-tickle-me> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 25 Frimaire an 229 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Tue, 15 Dec 2020 12:36:42 +0100 In-Reply-To: <87sg88ngd1.fsf@guixSD.i-did-not-set--mail-host-address--so-tickle-me> (=?utf-8?Q?=22Nicol=C3=B2?= Balzarotti"'s message of "Mon, 14 Dec 2020 23:59:54 +0100") Message-ID: <87im93mhbp.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: guix-devel Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-Spam-Score: -2.81 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Queue-Id: 70DFC940148 X-Spam-Score: -2.81 X-Migadu-Scanner: scn0.migadu.com X-TUID: 36Fd/EU8ubO7 Hi, Nicol=C3=B2 Balzarotti skribis: > I guess this benchmark follows the distri talk, doesn't it? :) Yes, that and my own quest for optimization opportunities. :-) > File size with zstd vs zstd -9 vs current lzip: > - 71M uc.nar.lz > - 87M uc.nar.zst-9 > - 97M uc.nar.zst-default > >> Where to go from here? Several options: > >> 1. Since ci.guix.gnu.org still provides both gzip and lzip archives, >> =E2=80=98guix substitute=E2=80=99 could automatically pick one or t= he other >> depending on the CPU and bandwidth. Perhaps a simple trick would >> be to check the user/wall-clock time ratio and switch to gzip for >> subsequent downloads if that ratio is close to one. How well would >> that work? > > I'm not sure using heuristics (i.e., guessing what should work better, > like in 1.) is the way to go, as temporary slowdowns to the network/cpu > will during the first download would affect the decision. I suppose we could time each substitute download and adjust the choice continually. It might be better to provide a command-line flag to choose between optimizing for bandwidth usage (users with limited Internet access may prefer that) or for speed. >> 2. Use Zstd like all the cool kids since it seems to have a much >> higher decompression speed: . >> 630=C2=A0MB/s on ungoogled-chromium on my laptop. Woow. > > I know this means more work to do, but it seems to be the best > alternative. However, if we go that way, will we keep lzip substitutes? > The 20% difference in size between lzip/zstd would mean a lot with slow > (mobile) network connections. A lot in what sense? In terms of bandwidth usage, right? In terms of speed, zstd would probably reduce the time-to-disk as soon as you have ~15=C2=A0MB/s peak bandwidth or more. Anyway, we=E2=80=99re not there yet, but I suppose if we get zstd support, = we could configure berlin to keep lzip and zstd (rather than lzip and gzip as is currently the case). Ludo=E2=80=99.