bug#49174: Poor 'guix substitute' performance when receiving Zstd-compressed substitutes

["Ludovic Courtès" <ludo@gnu.org>]

Hi,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> It's something I've been observing for a while, but substitutes are very
> IO intensive (as can be seen in iotop, the substitute process is waiting
> on IO > 99% of the time) and is much slower than expected (3 minutes to
> transfer 100 MiB uncompressed over a 50 mbps downstream link):
>
> TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
> 13934 be/4 root     1033.09 K/s 1485.06 K/s  0.00 % 93.36 % guile \ /gnu/store/vphx2839xv0qj9xwcwrb95592lzrrnx7-guix-1.3.0-3.50dfbbf/bin/guix substitute --substitute
>
>
> The publisher (remote machine) is has its guix-daemon configured via:
>
>       (service guix-publish-service-type
>                (guix-publish-configuration
>                 (advertise? #t)
>                 (compression '(("zstd" 3)))
>                 (host "0.0.0.0")))	;listen on all interfaces

Note that in this case nars are built and compressed on the fly on the
server side, which puts an upper bound on the bandwidth you can achieve.

I showed earlier how I profiled these things:

  https://guix.gnu.org/en/blog/2021/getting-bytes-to-disk-more-quickly/

If the client is I/O-bound, that’s good: it means we can’t do any better
(unless we skip unpacking as demonstrated by distri).

If you can provide detailed profiles of either the server side or the
client side (but in that case, make sure the server is caching things),
that’d be great!

Otherwise I’m afraid this is not actionable.  :-)

Thanks,
Ludo’.