bug#49174: Poor 'guix substitute' performance when receiving Zstd-compressed substitutes

[Maxim Cournoyer <maxim.cournoyer@gmail.com>]

Hi,

Ludovic Courtès <ludo@gnu.org> writes:

> Hi,
>
> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>
>> It's something I've been observing for a while, but substitutes are very
>> IO intensive (as can be seen in iotop, the substitute process is waiting
>> on IO > 99% of the time) and is much slower than expected (3 minutes to
>> transfer 100 MiB uncompressed over a 50 mbps downstream link):
>>
>> TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
>> 13934 be/4 root     1033.09 K/s 1485.06 K/s  0.00 % 93.36 % guile \ /gnu/store/vphx2839xv0qj9xwcwrb95592lzrrnx7-guix-1.3.0-3.50dfbbf/bin/guix substitute --substitute
>>
>>
>> The publisher (remote machine) is has its guix-daemon configured via:
>>
>>       (service guix-publish-service-type
>>                (guix-publish-configuration
>>                 (advertise? #t)
>>                 (compression '(("zstd" 3)))
>>                 (host "0.0.0.0")))	;listen on all interfaces
>
> Note that in this case nars are built and compressed on the fly on the
> server side, which puts an upper bound on the bandwidth you can achieve.
>
> I showed earlier how I profiled these things:
>
>   https://guix.gnu.org/en/blog/2021/getting-bytes-to-disk-more-quickly/
>
> If the client is I/O-bound, that’s good: it means we can’t do any better
> (unless we skip unpacking as demonstrated by distri).
>
> If you can provide detailed profiles of either the server side or the
> client side (but in that case, make sure the server is caching things),
> that’d be great!
>
> Otherwise I’m afraid this is not actionable.  :-)

Since moving from HDDs to SSDs, I haven't seen this problem, so I
suspect the poor IO of the HDDs was really the culprit rather than
something to do with guile-zstd (and we had also benchmarked the late
some when I experimented with using zstd-compressed man pages).

Closing!

-- 
Thanks,
Maxim