From: ludo@gnu.org (Ludovic Courtès)
To: Tobias Geerinckx-Rice <me@tobias.gr>
Cc: 26201@debbugs.gnu.org, guix-sysadmin@gnu.org
Subject: bug#26201: Bandwidth when retrieving substitutes
Date: Mon, 27 Mar 2017 13:20:42 +0200 [thread overview]
Message-ID: <8760ivm0dx.fsf_-_@gnu.org> (raw)
In-Reply-To: <87r31pyms2.fsf_-_@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\?\= \=\?utf-8\?Q\?\=22's\?\= message of "Wed, 22 Mar 2017 23:22:37 +0100")
Hi there!
ludo@gnu.org (Ludovic Courtès) skribis:
> ‘guix publish’ should be faster and less resource-hungry than Hydra. It
> uses in-process gzip for nar compression instead of bzip2 (I chose level
> 7, which seems to provide compression ratios close to what bzip2
> provides with its default compression level, while being 3 times
> faster). Unlike Hydra it never forks so for instance, 404 responses for
> .narinfo URLs should be quicker. Hopefully, that will improve the
> worst-case (cache miss) throughput.
Another interesting data point on the client side this time:
--8<---------------cut here---------------start------------->8---
$ wget -O- https://mirror.hydra.gnu.org/nar/v6rq6j9wdx8ixsks05dxhxr26jgmr6z3-mysql-5.7.17 |bunzip2 >/dev/null
--2017-03-27 13:12:50-- https://mirror.hydra.gnu.org/nar/v6rq6j9wdx8ixsks05dxhxr26jgmr6z3-mysql-5.7.17
Resolving mirror.hydra.gnu.org (mirror.hydra.gnu.org)... 131.159.14.26, 2001:4ca0:2001:10:225:90ff:fedb:c720
Connecting to mirror.hydra.gnu.org (mirror.hydra.gnu.org)|131.159.14.26|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/x-nix-archive]
Saving to: ‘STDOUT’
- [ <=> ] 53.01M 9.29MB/s in 5.5s
2017-03-27 13:12:55 (9.57 MB/s) - written to stdout [55582050]
$ wget -O- https://mirror.hydra.gnu.org/guix/nar/gzip/v6rq6j9wdx8ixsks05dxhxr26jgmr6z3-mysql-5.7.17 |gunzip >/dev/null
--2017-03-27 13:13:00-- https://mirror.hydra.gnu.org/guix/nar/gzip/v6rq6j9wdx8ixsks05dxhxr26jgmr6z3-mysql-5.7.17
Resolving mirror.hydra.gnu.org (mirror.hydra.gnu.org)... 131.159.14.26, 2001:4ca0:2001:10:225:90ff:fedb:c720
Connecting to mirror.hydra.gnu.org (mirror.hydra.gnu.org)|131.159.14.26|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/x-nix-archive]
Saving to: ‘STDOUT’
- [ <=> ] 59.19M 40.8MB/s in 1.4s
2017-03-27 13:13:02 (40.8 MB/s) - written to stdout [62068901]
$ wget -O- https://mirror.hydra.gnu.org/guix/nar/gzip/v6rq6j9wdx8ixsks05dxhxr26jgmr6z3-mysql-5.7.17 >/dev/null
--2017-03-27 13:15:58-- https://mirror.hydra.gnu.org/guix/nar/gzip/v6rq6j9wdx8ixsks05dxhxr26jgmr6z3-mysql-5.7.17
Resolving mirror.hydra.gnu.org (mirror.hydra.gnu.org)... 131.159.14.26, 2001:4ca0:2001:10:225:90ff:fedb:c720
Connecting to mirror.hydra.gnu.org (mirror.hydra.gnu.org)|131.159.14.26|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/x-nix-archive]
Saving to: ‘STDOUT’
- [ <=> ] 59.19M 42.5MB/s in 1.4s
2017-03-27 13:16:00 (42.5 MB/s) - written to stdout [62068901]
--8<---------------cut here---------------end--------------->8---
40 MB/s vs. 10 MB/s! (Both items were cached on mirror.hydra.gnu.org.)
IOW, bunzip2 was the bottleneck when retrieving substitutes (and that’s
on an i7.) With ‘perf timechart’ we see that bunzip2 is indeed busy
all the time right from the start.
Ludo’.
prev parent reply other threads:[~2017-03-27 11:21 UTC|newest]
Thread overview: 37+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-03-21 1:44 bug#26201: No notification of cache misses when downloading substitutes dian_cecht
2017-03-21 2:46 ` Tobias Geerinckx-Rice
2017-03-21 2:52 ` dian_cecht
2017-03-21 3:57 ` Tobias Geerinckx-Rice
2017-03-21 4:48 ` dian_cecht
2017-03-21 6:21 ` Tobias Geerinckx-Rice
2017-03-21 6:49 ` dian_cecht
2017-03-21 14:55 ` Tobias Geerinckx-Rice
2017-03-21 15:32 ` dian_cecht
2017-03-21 16:07 ` Tobias Geerinckx-Rice
2017-03-24 2:15 ` Maxim Cournoyer
2017-03-21 12:59 ` Florian Pelz
2017-03-21 15:35 ` dian_cecht
2017-03-21 16:02 ` Tobias Geerinckx-Rice
2017-03-21 16:13 ` dian_cecht
2017-03-21 16:33 ` Tobias Geerinckx-Rice
2017-03-21 16:55 ` dian_cecht
2017-03-21 17:18 ` Tobias Geerinckx-Rice
2017-03-21 16:43 ` Ludovic Courtès
2017-03-21 17:08 ` Tobias Geerinckx-Rice
2017-03-22 22:06 ` Ludovic Courtès
2017-03-23 19:25 ` bug#26201: hydra.gnu.org uses ‘guix publish’ for nars and narinfos Tobias Geerinckx-Rice
2017-03-22 22:22 ` Ludovic Courtès
2017-03-23 10:29 ` Ricardo Wurmus
2017-03-23 18:36 ` Mark H Weaver
2017-03-23 18:52 ` Tobias Geerinckx-Rice
2017-03-24 8:12 ` Mark H Weaver
2017-03-24 9:25 ` Ludovic Courtès
2017-04-17 21:36 ` Ludovic Courtès
2017-04-18 21:27 ` Ludovic Courtès
2017-04-19 14:24 ` bug#26201: Heads-up: hydra.gnu.org uses ‘guix publish --cache’ Ludovic Courtès
2017-03-26 17:35 ` bug#26201: hydra.gnu.org uses ‘guix publish’ for nars and narinfos Tobias Geerinckx-Rice
2017-03-27 18:47 ` Tobias Geerinckx-Rice
2017-03-28 14:47 ` Ludovic Courtès
2017-05-03 8:11 ` Mark H Weaver
2017-05-03 9:25 ` Ludovic Courtès
2017-03-27 11:20 ` Ludovic Courtès [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=8760ivm0dx.fsf_-_@gnu.org \
--to=ludo@gnu.org \
--cc=26201@debbugs.gnu.org \
--cc=guix-sysadmin@gnu.org \
--cc=me@tobias.gr \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this external index
https://git.savannah.gnu.org/cgit/guix.git
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.