Re: Parallel downloads

[Mark H Weaver <mhw@netris.org>]

Hi,

Ludovic Courtès <ludo@gnu.org> writes:

> Pierre Neidhardt <mail@ambrevar.xyz> skribis:
>
>> Ludo, what do you think of Tobias suggestion and have an extra knob to
>> specifically configure the number of download jobs?
>
> Like I wrote, it’s not that simple (we’d first need the daemon to
> distinguish substitution jobs from other jobs, but note that there are
> also “downloads” that are actually derivation builds), and it’s not
> clear to me that it’s overall beneficial anyway: it’s not supposed to be
> faster to download 10 things in parallel from ci.guix.gnu.org, than to
> download them sequentially.

I'll also note that parallel downloads would increase the memory usage
on the server.  If users, on average, configured 4 parallel downloads, I
guess that would have the effect of multiplying the server memory usage
by about 4.  It might also create an incentive for users to configure
more parallel downloads in order to grab a larger ratio of the server's
available resources.

For these reasons, I'm inclined to think that parallel downloads is the
wrong approach.  If a single download process is not making efficient
use of the available bandwidth, I'd be more inclined to look carefully
at why it's failing to do so.  For example, I'm not sure if this is the
case (and don't have time to look right now), but if the current code
waits until a NAR has finished downloading before asking for the next
one, that's an issue that could be fixed by use of HTTP pipelining,
without multiplying the memory usage.

What do you think?

      Mark