From mboxrd@z Thu Jan 1 00:00:00 1970 From: ludo@gnu.org (Ludovic =?UTF-8?Q?Court=C3=A8s?=) Subject: bug#26201: Bandwidth when retrieving substitutes Date: Mon, 27 Mar 2017 13:20:42 +0200 Message-ID: <8760ivm0dx.fsf_-_@gnu.org> References: <20170320184449.5ac06051@khaalida> <144e9ba8-af93-fb18-d2b9-f198ae7c11e9@tobias.gr> <20170320195247.05f72fc9@khaalida> <8e7e07d1-563f-666f-2c32-2a772757c86f@tobias.gr> <8760j2wpfy.fsf@gnu.org> <9889a4b5-c300-cd03-1095-1115428067fb@tobias.gr> <87r31pyms2.fsf_-_@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:53778) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1csShu-00037w-9z for bug-guix@gnu.org; Mon, 27 Mar 2017 07:21:11 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1csShq-0003a4-40 for bug-guix@gnu.org; Mon, 27 Mar 2017 07:21:06 -0400 Received: from debbugs.gnu.org ([208.118.235.43]:48673) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1csShp-0003Zc-VK for bug-guix@gnu.org; Mon, 27 Mar 2017 07:21:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1csShp-0003zR-M1 for bug-guix@gnu.org; Mon, 27 Mar 2017 07:21:01 -0400 Sender: "Debbugs-submit" Resent-Message-ID: In-Reply-To: <87r31pyms2.fsf_-_@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\?\= \=\?utf-8\?Q\?\=22's\?\= message of "Wed, 22 Mar 2017 23:22:37 +0100") List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+gcggb-bug-guix=m.gmane.org@gnu.org Sender: "bug-Guix" To: Tobias Geerinckx-Rice Cc: 26201@debbugs.gnu.org, guix-sysadmin@gnu.org Hi there! ludo@gnu.org (Ludovic Court=C3=A8s) skribis: > =E2=80=98guix publish=E2=80=99 should be faster and less resource-hungry = than Hydra. It > uses in-process gzip for nar compression instead of bzip2 (I chose level > 7, which seems to provide compression ratios close to what bzip2 > provides with its default compression level, while being 3 times > faster). Unlike Hydra it never forks so for instance, 404 responses for > .narinfo URLs should be quicker. Hopefully, that will improve the > worst-case (cache miss) throughput. Another interesting data point on the client side this time: --8<---------------cut here---------------start------------->8--- $ wget -O- https://mirror.hydra.gnu.org/nar/v6rq6j9wdx8ixsks05dxhxr26jgmr6z= 3-mysql-5.7.17 |bunzip2 >/dev/null --2017-03-27 13:12:50-- https://mirror.hydra.gnu.org/nar/v6rq6j9wdx8ixsks0= 5dxhxr26jgmr6z3-mysql-5.7.17 Resolving mirror.hydra.gnu.org (mirror.hydra.gnu.org)... 131.159.14.26, 200= 1:4ca0:2001:10:225:90ff:fedb:c720 Connecting to mirror.hydra.gnu.org (mirror.hydra.gnu.org)|131.159.14.26|:44= 3... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [application/x-nix-archive] Saving to: =E2=80=98STDOUT=E2=80=99 - [ <=3D> = ] 53.01M 9.29MB/s in 5.5s=20=20=20=20 2017-03-27 13:12:55 (9.57 MB/s) - written to stdout [55582050] $ wget -O- https://mirror.hydra.gnu.org/guix/nar/gzip/v6rq6j9wdx8ixsks05dxh= xr26jgmr6z3-mysql-5.7.17 |gunzip >/dev/null --2017-03-27 13:13:00-- https://mirror.hydra.gnu.org/guix/nar/gzip/v6rq6j9= wdx8ixsks05dxhxr26jgmr6z3-mysql-5.7.17 Resolving mirror.hydra.gnu.org (mirror.hydra.gnu.org)... 131.159.14.26, 200= 1:4ca0:2001:10:225:90ff:fedb:c720 Connecting to mirror.hydra.gnu.org (mirror.hydra.gnu.org)|131.159.14.26|:44= 3... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [application/x-nix-archive] Saving to: =E2=80=98STDOUT=E2=80=99 - [ <=3D> = ] 59.19M 40.8MB/s in 1.4s=20=20=20=20 2017-03-27 13:13:02 (40.8 MB/s) - written to stdout [62068901] $ wget -O- https://mirror.hydra.gnu.org/guix/nar/gzip/v6rq6j9wdx8ixsks05dxh= xr26jgmr6z3-mysql-5.7.17 >/dev/null --2017-03-27 13:15:58-- https://mirror.hydra.gnu.org/guix/nar/gzip/v6rq6j9= wdx8ixsks05dxhxr26jgmr6z3-mysql-5.7.17 Resolving mirror.hydra.gnu.org (mirror.hydra.gnu.org)... 131.159.14.26, 200= 1:4ca0:2001:10:225:90ff:fedb:c720 Connecting to mirror.hydra.gnu.org (mirror.hydra.gnu.org)|131.159.14.26|:44= 3... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [application/x-nix-archive] Saving to: =E2=80=98STDOUT=E2=80=99 - [ <=3D> = ] 59.19M 42.5MB/s in 1.4s=20=20=20=20 2017-03-27 13:16:00 (42.5 MB/s) - written to stdout [62068901] --8<---------------cut here---------------end--------------->8--- 40=C2=A0MB/s vs. 10=C2=A0MB/s! (Both items were cached on mirror.hydra.gnu= .org.) IOW, bunzip2 was the bottleneck when retrieving substitutes (and that=E2=80= =99s on an i7.) With =E2=80=98perf timechart=E2=80=99 we see that bunzip2 is in= deed busy all the time right from the start. Ludo=E2=80=99.