From: Guillaume Le Vaillant <glv@posteo.net>
To: Pierre Neidhardt <mail@ambrevar.xyz>
Cc: guix-devel@gnu.org
Subject: Re: When substitute download + decompression is CPU-bound
Date: Fri, 29 Jan 2021 12:23:52 +0100 [thread overview]
Message-ID: <874kj0x9yv.fsf@yamatai> (raw)
In-Reply-To: <87eei4rs8x.fsf@ambrevar.xyz>
[-- Attachment #1: Type: text/plain, Size: 3571 bytes --]
Pierre Neidhardt <mail@ambrevar.xyz> skribis:
> Hi Ludo!
>
> Ludovic Courtès <ludo@gnu.org> writes:
>
>> I suppose a possible agenda would be:
>>
>> 1. Start providing zstd susbstitutes anytime. However, most clients
>> will keep choosing lzip because it usually compresses better.
>>
>> 2. After the next release, stop providing lzip substitutes and provide
>> only gzip + zstd-19.
>>
>> This option has the advantage that it wouldn’t break any installation.
>
> But why would we keep gzip since it offers no benefits compared to zstd?
> It feels like continuing to carry a (huge) burden forever...
>
> Besides, dropping Lzip seems like a step backward in my opinion. Users
> with lower bandwidth (or simply further away from Berlin) will be
> impacted a lot.
>
> I would opt for dropping gzip instead, only to keep zstd-19 and lzip-9
> (possibly plzip-9 if we update the bindings).
>
>> It’s not as nice as the ability to choose a download strategy, as we
>> discussed earlier, but implementing that download strategy sounds
>> tricky.
>
> If the user can choose their favourite substitute compression, I believe
> it's usually enough since they are the best judge of their bandwidth /
> hardware requirements.
>
> Wouldn't this simple enough?
Here are a few numbers for the installation time in seconds (download
time + decompression time) when fetching 580 MB of substitutes for
download speeds between 0.5 MB/s and 20 MB/s.
| Download speed | gzip -9 | lzip -9 | zstd -19 |
|----------------+---------+---------+----------|
| 0.5 | 287 | 151 | 181 |
| 1.0 | 144 | 78 | 91 |
| 1.5 | 97 | 54 | 61 |
| 2.0 | 73 | 42 | 46 |
| 2.5 | 59 | 35 | 37 |
| 3.0 | 49 | 30 | 31 |
| 3.5 | 42 | 27 | 26 |
| 4.0 | 37 | 24 | 23 |
| 4.5 | 33 | 22 | 21 |
| 5.0 | 30 | 21 | 19 |
| 5.5 | 28 | 19 | 17 |
| 6.0 | 25 | 18 | 16 |
| 6.5 | 24 | 17 | 14 |
| 7.0 | 22 | 17 | 14 |
| 7.5 | 21 | 16 | 13 |
| 8.0 | 20 | 15 | 12 |
| 8.5 | 18 | 15 | 11 |
| 9.0 | 18 | 14 | 11 |
| 9.5 | 17 | 14 | 10 |
| 10.0 | 16 | 13 | 10 |
| 11.0 | 15 | 13 | 9 |
| 12.0 | 14 | 12 | 8 |
| 13.0 | 13 | 12 | 8 |
| 14.0 | 12 | 11 | 7 |
| 15.0 | 11 | 11 | 7 |
| 16.0 | 11 | 11 | 6 |
| 17.0 | 10 | 10 | 6 |
| 18.0 | 10 | 10 | 6 |
| 19.0 | 9 | 10 | 5 |
| 20.0 | 9 | 10 | 5 |
When the download speed is lower than 3.5 MB/s, Lzip is better, and
above that speed Zstd is better.
As Gzip is never the best choice, it would make sense to drop it, even
if we have to wait a little until everyone has updated their Guix daemon
to a version with at least Lzip support.
I think there are many people (like me) with a download speed slower
than 3 MB/s, so like Pierre I would prefer keeping "lzip -9" and
"zstd -19".
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 247 bytes --]
next prev parent reply other threads:[~2021-01-29 11:29 UTC|newest]
Thread overview: 43+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-12-14 22:20 When substitute download + decompression is CPU-bound Ludovic Courtès
2020-12-14 22:29 ` Julien Lepiller
2020-12-14 22:59 ` Nicolò Balzarotti
2020-12-15 7:52 ` Pierre Neidhardt
2020-12-15 9:45 ` Nicolò Balzarotti
2020-12-15 9:54 ` Pierre Neidhardt
2020-12-15 10:03 ` Nicolò Balzarotti
2020-12-15 10:13 ` Pierre Neidhardt
2020-12-15 10:14 ` Pierre Neidhardt
2020-12-15 11:42 ` Ludovic Courtès
2020-12-15 12:31 ` Pierre Neidhardt
2020-12-18 14:59 ` Ludovic Courtès
2020-12-18 15:33 ` Pierre Neidhardt
2020-12-15 11:36 ` Ludovic Courtès
2020-12-15 11:45 ` Nicolò Balzarotti
2020-12-15 10:40 ` Jonathan Brielmaier
2020-12-15 19:43 ` Joshua Branson
2021-01-07 10:45 ` Guillaume Le Vaillant
2021-01-07 11:00 ` Pierre Neidhardt
2021-01-07 11:33 ` Guillaume Le Vaillant
2021-01-14 21:51 ` Ludovic Courtès
2021-01-14 22:08 ` Nicolò Balzarotti
2021-01-28 17:53 ` Are gzip-compressed substitutes still used? Ludovic Courtès
2021-03-17 17:12 ` Ludovic Courtès
2021-03-17 17:33 ` Léo Le Bouter
2021-03-17 18:08 ` Vagrant Cascadian
2021-03-18 0:03 ` zimoun
2021-03-18 16:00 ` Vagrant Cascadian
2021-03-18 18:53 ` Leo Famulari
2021-03-20 11:23 ` Ludovic Courtès
2021-03-17 18:06 ` zimoun
2021-03-17 18:20 ` Jonathan Brielmaier
2021-03-18 17:25 ` Pierre Neidhardt
2021-01-15 8:10 ` When substitute download + decompression is CPU-bound Pierre Neidhardt
2021-01-28 17:58 ` Ludovic Courtès
2021-01-29 9:45 ` Pierre Neidhardt
2021-01-29 11:23 ` Guillaume Le Vaillant [this message]
2021-01-29 11:55 ` Nicolò Balzarotti
2021-01-29 12:13 ` Pierre Neidhardt
2021-01-29 13:06 ` Guillaume Le Vaillant
2021-01-29 14:55 ` Nicolò Balzarotti
2021-02-01 22:18 ` Ludovic Courtès
2021-01-29 13:33 ` zimoun
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=874kj0x9yv.fsf@yamatai \
--to=glv@posteo.net \
--cc=guix-devel@gnu.org \
--cc=mail@ambrevar.xyz \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this external index
https://git.savannah.gnu.org/cgit/guix.git
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.