From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id EGOyNKHxE2AjYwAA0tVLHw (envelope-from ) for ; Fri, 29 Jan 2021 11:29:37 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id ePZ+MKHxE2AmKgAA1q6Kng (envelope-from ) for ; Fri, 29 Jan 2021 11:29:37 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 15EC7940111 for ; Fri, 29 Jan 2021 11:29:37 +0000 (UTC) Received: from localhost ([::1]:41634 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l5Ry2-0001ar-FD for larch@yhetil.org; Fri, 29 Jan 2021 06:29:36 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:35686) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l5Rsi-000395-KU for guix-devel@gnu.org; Fri, 29 Jan 2021 06:24:04 -0500 Received: from mout02.posteo.de ([185.67.36.66]:52695) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l5Rsf-0008AV-FG for guix-devel@gnu.org; Fri, 29 Jan 2021 06:24:03 -0500 Received: from submission (posteo.de [89.146.220.130]) by mout02.posteo.de (Postfix) with ESMTPS id 52C00240104 for ; Fri, 29 Jan 2021 12:23:57 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1611919437; bh=ZNBaCFQ9wTUxzcWY4PZL3/Pe+SZAu0Lf/ReqkSBoggs=; h=From:To:Cc:Subject:Date:From; b=ZXKEcp/f4ToCIkePbuA2enWGugWnG3X++NuDMzLxw3gpS/5ukrbODnTx0HtkRrVEu U1YxWTNy84AQCCmNrhZnbQTPc/0gRre8cBCyBOsDXbmktahDV+oHilZcYfZgaA+iVT 4EubdMnddIYXca/GueczU0pwg3n674z0sDmOF1htfw9Cotyxs40InfPznKB0EqYO9s I5ENRnPXgzyd/efAKAzJv0XWPZcwWuBouJ6U0Z2zEqGGZbg5V/Bjas/hPINlqDt0/T rR9/WF6Swd7+mTCDWeleSravFyi++WstLgXyngaRnlsBuak0+y1ZsDA3M21zXVWJDA lu5QYe2CvgXbw== Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4DRw2g0lZ4z9rxn; Fri, 29 Jan 2021 12:23:54 +0100 (CET) References: <87im94qbby.fsf@gnu.org> <94405d66-b13c-e6e6-e8d5-df23b93e5d97@web.de> <87im92voqw.fsf@dismail.de> <87ft3d2fge.fsf@yamatai> <87bldr191v.fsf@gnu.org> <87h7ni62nt.fsf@ambrevar.xyz> <87bld9j651.fsf@gnu.org> <87eei4rs8x.fsf@ambrevar.xyz> User-agent: mu4e 1.4.14; emacs 27.1 From: Guillaume Le Vaillant To: Pierre Neidhardt Subject: Re: When substitute download + decompression is CPU-bound In-reply-to: <87eei4rs8x.fsf@ambrevar.xyz> Date: Fri, 29 Jan 2021 12:23:52 +0100 Message-ID: <874kj0x9yv.fsf@yamatai> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Received-SPF: pass client-ip=185.67.36.66; envelope-from=glv@posteo.net; helo=mout02.posteo.de X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: guix-devel@gnu.org Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-Spam-Score: -2.85 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=posteo.net header.s=2017 header.b="ZXKEcp/f"; dmarc=fail reason="SPF not aligned (strict)" header.from=posteo.net (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Queue-Id: 15EC7940111 X-Spam-Score: -2.85 X-Migadu-Scanner: scn0.migadu.com X-TUID: 9UoI6cBY5IKO --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Pierre Neidhardt skribis: > Hi Ludo! > > Ludovic Court=C3=A8s writes: > >> I suppose a possible agenda would be: >> >> 1. Start providing zstd susbstitutes anytime. However, most clients >> will keep choosing lzip because it usually compresses better. >> >> 2. After the next release, stop providing lzip substitutes and provide >> only gzip + zstd-19. >> >> This option has the advantage that it wouldn=E2=80=99t break any install= ation. > > But why would we keep gzip since it offers no benefits compared to zstd? > It feels like continuing to carry a (huge) burden forever... > > Besides, dropping Lzip seems like a step backward in my opinion. Users > with lower bandwidth (or simply further away from Berlin) will be > impacted a lot. > > I would opt for dropping gzip instead, only to keep zstd-19 and lzip-9 > (possibly plzip-9 if we update the bindings). > >> It=E2=80=99s not as nice as the ability to choose a download strategy, a= s we >> discussed earlier, but implementing that download strategy sounds >> tricky. > > If the user can choose their favourite substitute compression, I believe > it's usually enough since they are the best judge of their bandwidth / > hardware requirements. > > Wouldn't this simple enough? Here are a few numbers for the installation time in seconds (download time + decompression time) when fetching 580 MB of substitutes for download speeds between 0.5 MB/s and 20 MB/s. | Download speed | gzip -9 | lzip -9 | zstd -19 | |----------------+---------+---------+----------| | 0.5 | 287 | 151 | 181 | | 1.0 | 144 | 78 | 91 | | 1.5 | 97 | 54 | 61 | | 2.0 | 73 | 42 | 46 | | 2.5 | 59 | 35 | 37 | | 3.0 | 49 | 30 | 31 | | 3.5 | 42 | 27 | 26 | | 4.0 | 37 | 24 | 23 | | 4.5 | 33 | 22 | 21 | | 5.0 | 30 | 21 | 19 | | 5.5 | 28 | 19 | 17 | | 6.0 | 25 | 18 | 16 | | 6.5 | 24 | 17 | 14 | | 7.0 | 22 | 17 | 14 | | 7.5 | 21 | 16 | 13 | | 8.0 | 20 | 15 | 12 | | 8.5 | 18 | 15 | 11 | | 9.0 | 18 | 14 | 11 | | 9.5 | 17 | 14 | 10 | | 10.0 | 16 | 13 | 10 | | 11.0 | 15 | 13 | 9 | | 12.0 | 14 | 12 | 8 | | 13.0 | 13 | 12 | 8 | | 14.0 | 12 | 11 | 7 | | 15.0 | 11 | 11 | 7 | | 16.0 | 11 | 11 | 6 | | 17.0 | 10 | 10 | 6 | | 18.0 | 10 | 10 | 6 | | 19.0 | 9 | 10 | 5 | | 20.0 | 9 | 10 | 5 | When the download speed is lower than 3.5 MB/s, Lzip is better, and above that speed Zstd is better. As Gzip is never the best choice, it would make sense to drop it, even if we have to wait a little until everyone has updated their Guix daemon to a version with at least Lzip support. I think there are many people (like me) with a download speed slower than 3 MB/s, so like Pierre I would prefer keeping "lzip -9" and "zstd -19". --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iIUEAREKAC0WIQTLxZxm7Ce5cXlAaz5r6CCK3yH+PwUCYBPwSA8cZ2x2QHBvc3Rl by5uZXQACgkQa+ggit8h/j+VRAD+JaWE9/UGHJaKz4DD+VYeXa40KsrQxG30A1WK 6L3MGCkA/Rrqqr4/JNh2ZdH5uw3sm4hcNgQY5t2n+GTfgP5xixdA =wVGV -----END PGP SIGNATURE----- --=-=-=--