Substituting in parallel to save time?

Substituting in parallel to save time?

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 902 bytes --]

Substitution honors the --max-jobs argument to the guix-daemon, which
means that, by default, substitutes are downloaded in sequence, one at a
time.

I noticed that in many cases, it takes longer to set up the substitution
than to actually download the substitute on my 30 Mbps downlink.

The effect is that it might take a few minutes to to download a few
dozen small substitutes, which may only total 5 megabytes of data, even
though my connection to the substitute server is fast enough that it
should only take a few seconds.

So, it benefits me to increase max-jobs for substitute downloads.
However, I rarely want to actually build packages in parallel; one
package build may fully load my system.

Currently, is it possible to distinguish between substitute downloads
and builds? Maybe it would be an overall performance win to download
substitutes in parallel, in tune with parallel-job-count.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Substituting in parallel to save time?

[Ludovic Courtès]

Hi!

Leo Famulari <leo@famulari.name> skribis:

> I noticed that in many cases, it takes longer to set up the substitution
> than to actually download the substitute on my 30 Mbps downlink.

Yeah.

> The effect is that it might take a few minutes to to download a few
> dozen small substitutes, which may only total 5 megabytes of data, even
> though my connection to the substitute server is fast enough that it
> should only take a few seconds.
>
> So, it benefits me to increase max-jobs for substitute downloads.
> However, I rarely want to actually build packages in parallel; one
> package build may fully load my system.
>
> Currently, is it possible to distinguish between substitute downloads
> and builds? Maybe it would be an overall performance win to download
> substitutes in parallel, in tune with parallel-job-count.

No, they cannot be distinguished.

I’d be in favor of a solution where ‘guix substitute’ is kept alive
across substitutions (like what happens with ‘guix substitute --query’),
which would allow it to keep connections alive and thus save the TLS
handshake and a few extra round trips per download.

Thoughts?

Ludo’.