From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id SKTVJD/7EmDTGgAA0tVLHw (envelope-from ) for ; Thu, 28 Jan 2021 17:58:23 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id SN2lID/7EmAgVwAA1q6Kng (envelope-from ) for ; Thu, 28 Jan 2021 17:58:23 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 3E3E09402A7 for ; Thu, 28 Jan 2021 17:58:23 +0000 (UTC) Received: from localhost ([::1]:58864 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l5BYj-0004d3-Sr for larch@yhetil.org; Thu, 28 Jan 2021 12:58:21 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:39672) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l5BUE-00005S-Q3 for guix-devel@gnu.org; Thu, 28 Jan 2021 12:53:42 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:48562) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l5BUE-0003oA-28; Thu, 28 Jan 2021 12:53:42 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=58666 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1l5BUD-0004M1-MW; Thu, 28 Jan 2021 12:53:41 -0500 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: =?utf-8?Q?Nicol=C3=B2?= Balzarotti Subject: Are gzip-compressed substitutes still used? References: <87im94qbby.fsf@gnu.org> <94405d66-b13c-e6e6-e8d5-df23b93e5d97@web.de> <87im92voqw.fsf@dismail.de> <87ft3d2fge.fsf@yamatai> <87bldr191v.fsf@gnu.org> <87v9bzmat1.fsf@guixSD.i-did-not-set--mail-host-address--so-tickle-me> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 9 =?utf-8?Q?Pluvi=C3=B4se?= an 229 de la =?utf-8?Q?R?= =?utf-8?Q?=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Thu, 28 Jan 2021 18:53:40 +0100 In-Reply-To: <87v9bzmat1.fsf@guixSD.i-did-not-set--mail-host-address--so-tickle-me> (=?utf-8?Q?=22Nicol=C3=B2?= Balzarotti"'s message of "Thu, 14 Jan 2021 23:08:10 +0100") Message-ID: <87im7hj6cb.fsf_-_@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: guix-devel@gnu.org Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-Spam-Score: -1.85 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Queue-Id: 3E3E09402A7 X-Spam-Score: -1.85 X-Migadu-Scanner: scn0.migadu.com X-TUID: nWVOvHBoxyaD Hi Nicol=C3=B2, Nicol=C3=B2 Balzarotti skribis: > Ludovic Court=C3=A8s writes: > >> We could also drop gzip, but there are probably pre-1.1 daemons out >> there that understand nothing but gzip=C2=B9, so perhaps that=E2=80=99ll= have to >> wait. Now, compressing substitutes three times may be somewhat >> unreasonable. >> >> Thoughts? >> > Is there a request log where we can check whether this is true? I finally got around to checking this. I picked a relatively popular substitute for which the lzip-compressed variant is smaller than the gzip-compressed variant, and thus modern =E2=80=98guix substitute=E2=80=99 chooses lzip over gzip: --8<---------------cut here---------------start------------->8--- $ wget -q -O - https://ci.guix.gnu.org/7rpj4dmn9g64zqp8vkc0byx93glix2pm.nar= info | head -7 StorePath: /gnu/store/7rpj4dmn9g64zqp8vkc0byx93glix2pm-gtk+-3.24.23 URL: nar/gzip/7rpj4dmn9g64zqp8vkc0byx93glix2pm-gtk%2B-3.24.23 Compression: gzip FileSize: 13982949 URL: nar/lzip/7rpj4dmn9g64zqp8vkc0byx93glix2pm-gtk%2B-3.24.23 Compression: lzip FileSize: 7223862 --8<---------------cut here---------------end--------------->8--- On berlin, I looked at the HTTPS nginx logs and did this: --8<---------------cut here---------------start------------->8--- ludo@berlin /var/log/nginx$ tail -10000000 < https.access.log > /tmp/sample= .log ludo@berlin /var/log/nginx$ date Thu 28 Jan 2021 06:18:01 PM CET ludo@berlin /var/log/nginx$ grep /7rpj4dmn9g64zqp8vkc0byx93glix2pm-gtk < /t= mp/sample.log |wc -l 1304 ludo@berlin /var/log/nginx$ grep /gzip/7rpj4dmn9g64zqp8vkc0byx93glix2pm-gtk= < /tmp/sample.log |wc -l 17 ludo@berlin /var/log/nginx$ grep /lzip/7rpj4dmn9g64zqp8vkc0byx93glix2pm-gtk= < /tmp/sample.log |wc -l 1287 --8<---------------cut here---------------end--------------->8--- The 10M-request sample covers requests from Jan. 10th to now. Over that period, 99% of the GTK+ downloads were made as lzip. We see similar results with less popular packages and with core packages: --8<---------------cut here---------------start------------->8--- ludo@berlin /var/log/nginx$ grep /01xi3sig314wgwa1j9sxk37vl816mj74-r-minima= l < /tmp/sample.log | wc -l 85 ludo@berlin /var/log/nginx$ grep /gzip/01xi3sig314wgwa1j9sxk37vl816mj74-r-m= inimal < /tmp/sample.log | wc -l 1 ludo@berlin /var/log/nginx$ grep /lzip/01xi3sig314wgwa1j9sxk37vl816mj74-r-m= inimal < /tmp/sample.log | wc -l 84 ludo@berlin /var/log/nginx$ grep /0m0vd873jp61lcm4xa3ljdgx381qa782-guile-3.= 0.2 < /tmp/sample.log |wc -l 1601 ludo@berlin /var/log/nginx$ grep /gzip/0m0vd873jp61lcm4xa3ljdgx381qa782-gui= le-3.0.2 < /tmp/sample.log |wc -l 8 ludo@berlin /var/log/nginx$ grep /lzip/0m0vd873jp61lcm4xa3ljdgx381qa782-gui= le-3.0.2 < /tmp/sample.log |wc -l 1593 --8<---------------cut here---------------end--------------->8--- >From that, we could deduce that about 1% of our users who take substitutes from ci.guix are still using a pre-1.1.0 daemon without support for lzip compression. I find it surprisingly low: 1.1.0 was released =E2=80=9Conly=E2=80=9D 9 mon= ths ago, which is not a lot for someone used to the long release cycles of =E2=80=9Cstable=E2=80=9D distros. It might be underestimated: users running an old daemon probably update less often and may thus be underrepresented in the substitute logs. As for whether it=E2=80=99s OK to drop gzip substitutes altogether: I=E2=80= =99m not confident about knowingly breaking 1% or more of the deployed Guixes, but it=E2=80=99s all about tradeoffs. Ludo=E2=80=99.