From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:c151::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id GP+OMbw6UmDzWwAA0tVLHw (envelope-from ) for ; Wed, 17 Mar 2021 17:22:04 +0000 Received: from aspmx2.migadu.com ([2001:41d0:2:c151::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id qBN2Lbw6UmDUHgAAbx9fmQ (envelope-from ) for ; Wed, 17 Mar 2021 17:22:04 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx2.migadu.com (Postfix) with ESMTPS id 80DD91FE43 for ; Wed, 17 Mar 2021 18:22:03 +0100 (CET) Received: from localhost ([::1]:39574 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lMZru-0008F8-An for larch@yhetil.org; Wed, 17 Mar 2021 13:22:02 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:45484) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lMZiM-0008FK-5w for guix-devel@gnu.org; Wed, 17 Mar 2021 13:12:10 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:39888) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lMZiL-00053f-U8 for guix-devel@gnu.org; Wed, 17 Mar 2021 13:12:09 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=48608 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1lMZiK-0008Uh-72 for guix-devel@gnu.org; Wed, 17 Mar 2021 13:12:09 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: guix-devel@gnu.org Subject: Re: Are gzip-compressed substitutes still used? References: <87im94qbby.fsf@gnu.org> <94405d66-b13c-e6e6-e8d5-df23b93e5d97@web.de> <87im92voqw.fsf@dismail.de> <87ft3d2fge.fsf@yamatai> <87bldr191v.fsf@gnu.org> <87v9bzmat1.fsf@guixSD.i-did-not-set--mail-host-address--so-tickle-me> <87im7hj6cb.fsf_-_@gnu.org> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 27 =?utf-8?Q?Vent=C3=B4se?= an 229 de la =?utf-8?Q?R?= =?utf-8?Q?=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Wed, 17 Mar 2021 18:12:05 +0100 In-Reply-To: <87im7hj6cb.fsf_-_@gnu.org> ("Ludovic =?utf-8?Q?Court=C3=A8s?= =?utf-8?Q?=22's?= message of "Thu, 28 Jan 2021 18:53:40 +0100") Message-ID: <87im5p3dsq.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1616001724; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=5xSDy0YWU4OIg908YG8ctIcLHpxaw4Wjw+tOg/ZFNIM=; b=X5FfNe9NmBZw1DjXm1PGI9P/mqngpfzuR7sbWYN0Gl7/V5XOogqJF3iNqfdsyGM3oaswCo Qe4Y4K8yG1nM4/UI57lkBkLVAyE3EvjoDhNfngmYIXCSyGnCvb78FvKzJETimQDB3kSUWB tuuMmzbMdvkpaM9WOreCamvErCM/3+HSEToh+t1m6ZaibrjG6zYsYDQav8Xds2tqt3Leb4 O3TNoyGjCgX1SbkrIqhMqBBVtNoTF0wAv86WMVF1dXdTlUvlJaebQFo5UJOPsKPI4tCBDm wekhEBRxqyAbwtksrxDkOQWWoUg9Hobu2FDALsjWubdkTBONyNjhrUEMOQyAyg== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1616001724; a=rsa-sha256; cv=none; b=LUGRnppZCxNoniQa1JHsDCJQuKUvbgzuy8xfDe1diXAG1W3DKtiFd5+ypRy9w8CLaoHUy3 kzzdQYVnUiO+0768gCeSY+9efSPt6dMignqKwwIVLjefJX+F7wym8DgRxeTZ082L3ckhvx C9VOnF07LAoKOJrjW8KMOBMwVJZqMjYJ+P09DgEgY4G12VkGMr7OHfiwL1/A88w6BFkFmS jl/vnH1vW+zCWimbZn+NHF438po6bL13PhOw9tfouhBITwysVakCSJsPUnjCsPwQeRP2CO 9ftSPiygSh/enM+/RU1A2LMNqHr7/6OI2BZ/urO2JPVhoaiJr6CG0bke96uRyg== ARC-Authentication-Results: i=1; aspmx2.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx2.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Spam-Score: 0.70 Authentication-Results: aspmx2.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx2.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Queue-Id: 80DD91FE43 X-Spam-Score: 0.70 X-Migadu-Scanner: scn0.migadu.com X-TUID: oo9dQHxyvFq6 --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi, Ludovic Court=C3=A8s skribis: > From that, we could deduce that about 1% of our users who take > substitutes from ci.guix are still using a pre-1.1.0 daemon without > support for lzip compression. > > I find it surprisingly low: 1.1.0 was released =E2=80=9Conly=E2=80=9D 9 m= onths ago, > which is not a lot for someone used to the long release cycles of > =E2=80=9Cstable=E2=80=9D distros. (See for the initial message.) Here=E2=80=99s an update, 1.5 month later. This time I=E2=80=99m looking a= t nginx logs covering Feb 8th to Mar 17th and using a laxer regexp than in the message above, here are the gzip/lzip download ratio for several packages: --8<---------------cut here---------------start------------->8--- ludo@berlin ~$ ./nar-download-stats.sh /tmp/sample3.log = gtk%2B-3: gzip/lzip ratio: 37/3= 255 1% glib-2: gzip/lzip ratio: 97/8629 1% coreutils-8: gzip/lzip ratio: 81/2306 3% python-3: gzip/lzip ratio: 120/7177 1% r-minimal-[34]: gzip/lzip ratio: 8/302 2% openmpi-4: gzip/lzip ratio: 19/236 8% hwloc-2: gzip/lzip ratio: 10/43 23% gfortran-7: gzip/lzip ratio: 6/225 2% --8<---------------cut here---------------end--------------->8--- (Script attached.) The hwloc/openmpi outlier is intriguing. Is it one HPC web site running an old daemon, or several of them? Looking more closely, it=E2=80=99s 22 of them on 8 different networks (looking at the first three digits of the IP address): --8<---------------cut here---------------start------------->8--- ludo@berlin ~$ grep -E '/gzip/[[:alnum:]]{32}-(hwloc-2|openmpi-4)\.[[:digit= :]]+\.[[:digit:]]+ ' < /tmp/sample3.log | cut -f1 -d- | sort -u | wc -l 22 ludo@berlin ~$ grep -E '/gzip/[[:alnum:]]{32}-(hwloc-2|openmpi-4)\.[[:digit= :]]+\.[[:digit:]]+ ' < /tmp/sample3.log | cut -f1 -d- | cut -f 1-3 -d. | so= rt -u | wc -l 8 --8<---------------cut here---------------end--------------->8--- Conclusion? It still sounds like we can=E2=80=99t reasonably remove gzip support just yet. I=E2=80=99d still like to start providing zstd-compressed substitutes thoug= h. So I think what we can do is: =E2=80=A2 start providing zstd substitutes on berlin right now so that wh= en 1.2.1 comes out, at least some substitutes are available as zstd; =E2=80=A2 when 1.2.1 is announced, announce that gzip substitutes may be removed in the future and invite users to upgrade; =E2=80=A2 revisit this issue with an eye on dropping gzip within 6=E2=80= =9318 months. Thoughts? Ludo=E2=80=99. --=-=-= Content-Type: text/plain Content-Disposition: inline; filename=nar-download-stats.sh Content-Description: the script #!/bin/sh if [ ! "$#" = 1 ] then echo "Usage: $1 NGINX-LOG-FILE" exit 1 fi set -e sample="$1" items="gtk%2B-3 glib-2 coreutils-8 python-3 r-minimal-[34] openmpi-4 hwloc-2 gfortran-7" for i in $items do # Tweak the regexp so we don't catch ".drv" substitutes as these # usually compress better with gzip. lzip="$(grep -E "/lzip/[[:alnum:]]{32}-$i\\.[[:digit:]]+(\\.[[:digit:]]+)? " < "$sample" | wc -l)" gzip="$(grep -E "/gzip/[[:alnum:]]{32}-$i\\.[[:digit:]]+(\\.[[:digit:]]+)? " < "$sample" | wc -l)" echo "$i: gzip/lzip ratio: $gzip/$lzip $(($gzip * 100 / $lzip))%" done --=-=-=--