unofficial mirror of guix-devel@gnu.org 
 help / color / mirror / code / Atom feed
From: "Ludovic Courtès" <ludo@gnu.org>
To: "Nicolò Balzarotti" <anothersms@gmail.com>
Cc: guix-devel <guix-devel@gnu.org>
Subject: Re: When substitute download + decompression is CPU-bound
Date: Tue, 15 Dec 2020 12:36:42 +0100	[thread overview]
Message-ID: <87im93mhbp.fsf@gnu.org> (raw)
In-Reply-To: <87sg88ngd1.fsf@guixSD.i-did-not-set--mail-host-address--so-tickle-me> ("Nicolò Balzarotti"'s message of "Mon, 14 Dec 2020 23:59:54 +0100")

Hi,

Nicolò Balzarotti <anothersms@gmail.com> skribis:

> I guess this benchmark follows the distri talk, doesn't it? :)

Yes, that and my own quest for optimization opportunities.  :-)

> File size with zstd vs zstd -9 vs current lzip:
> - 71M uc.nar.lz
> - 87M uc.nar.zst-9
> - 97M uc.nar.zst-default
>
>> Where to go from here?  Several options:
>
>>   1. Since ci.guix.gnu.org still provides both gzip and lzip archives,
>>      ‘guix substitute’ could automatically pick one or the other
>>      depending on the CPU and bandwidth.  Perhaps a simple trick would
>>      be to check the user/wall-clock time ratio and switch to gzip for
>>      subsequent downloads if that ratio is close to one.  How well would
>>      that work?
>
> I'm not sure using heuristics (i.e., guessing what should work better,
> like in 1.) is the way to go, as temporary slowdowns to the network/cpu
> will during the first download would affect the decision.

I suppose we could time each substitute download and adjust the choice
continually.

It might be better to provide a command-line flag to choose between
optimizing for bandwidth usage (users with limited Internet access may
prefer that) or for speed.

>>   2. Use Zstd like all the cool kids since it seems to have a much
>>      higher decompression speed: <https://facebook.github.io/zstd/>.
>>      630 MB/s on ungoogled-chromium on my laptop.  Woow.
>
> I know this means more work to do, but it seems to be the best
> alternative.  However, if we go that way, will we keep lzip substitutes?
> The 20% difference in size between lzip/zstd would mean a lot with slow
> (mobile) network connections.

A lot in what sense?  In terms of bandwidth usage, right?

In terms of speed, zstd would probably reduce the time-to-disk as soon
as you have ~15 MB/s peak bandwidth or more.

Anyway, we’re not there yet, but I suppose if we get zstd support, we
could configure berlin to keep lzip and zstd (rather than lzip and gzip
as is currently the case).

Ludo’.


  parent reply	other threads:[~2020-12-15 11:37 UTC|newest]

Thread overview: 43+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-12-14 22:20 When substitute download + decompression is CPU-bound Ludovic Courtès
2020-12-14 22:29 ` Julien Lepiller
2020-12-14 22:59 ` Nicolò Balzarotti
2020-12-15  7:52   ` Pierre Neidhardt
2020-12-15  9:45     ` Nicolò Balzarotti
2020-12-15  9:54       ` Pierre Neidhardt
2020-12-15 10:03         ` Nicolò Balzarotti
2020-12-15 10:13           ` Pierre Neidhardt
2020-12-15 10:14             ` Pierre Neidhardt
2020-12-15 11:42     ` Ludovic Courtès
2020-12-15 12:31       ` Pierre Neidhardt
2020-12-18 14:59         ` Ludovic Courtès
2020-12-18 15:33           ` Pierre Neidhardt
2020-12-15 11:36   ` Ludovic Courtès [this message]
2020-12-15 11:45     ` Nicolò Balzarotti
2020-12-15 10:40 ` Jonathan Brielmaier
2020-12-15 19:43   ` Joshua Branson
2021-01-07 10:45     ` Guillaume Le Vaillant
2021-01-07 11:00       ` Pierre Neidhardt
2021-01-07 11:33         ` Guillaume Le Vaillant
2021-01-14 21:51       ` Ludovic Courtès
2021-01-14 22:08         ` Nicolò Balzarotti
2021-01-28 17:53           ` Are gzip-compressed substitutes still used? Ludovic Courtès
2021-03-17 17:12             ` Ludovic Courtès
2021-03-17 17:33               ` Léo Le Bouter
2021-03-17 18:08                 ` Vagrant Cascadian
2021-03-18  0:03                   ` zimoun
2021-03-18 16:00                     ` Vagrant Cascadian
2021-03-18 18:53                       ` Leo Famulari
2021-03-20 11:23                   ` Ludovic Courtès
2021-03-17 18:06               ` zimoun
2021-03-17 18:20               ` Jonathan Brielmaier
2021-03-18 17:25               ` Pierre Neidhardt
2021-01-15  8:10         ` When substitute download + decompression is CPU-bound Pierre Neidhardt
2021-01-28 17:58           ` Ludovic Courtès
2021-01-29  9:45             ` Pierre Neidhardt
2021-01-29 11:23               ` Guillaume Le Vaillant
2021-01-29 11:55                 ` Nicolò Balzarotti
2021-01-29 12:13                   ` Pierre Neidhardt
2021-01-29 13:06                     ` Guillaume Le Vaillant
2021-01-29 14:55                     ` Nicolò Balzarotti
2021-02-01 22:18                 ` Ludovic Courtès
2021-01-29 13:33             ` zimoun

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://guix.gnu.org/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=87im93mhbp.fsf@gnu.org \
    --to=ludo@gnu.org \
    --cc=anothersms@gmail.com \
    --cc=guix-devel@gnu.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/guix.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).