Parallel downloads

Parallel downloads

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 269 bytes --]

Hi,

Is there any plan to support parallel downloads?  Currently downloads are a
bottleneck for `guix install / upgrade`, parallel downloads could reduce
the operation duration by an order of magnitude.

Thoughts?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Parallel downloads

[Tobias Geerinckx-Rice]

[-- Attachment #1: Type: text/plain, Size: 868 bytes --]

Hullo Pierre!

Pierre Neidhardt 写道：
> Is there any plan to support parallel downloads?

Guix already downloads sources and substitutes in parallel with 
other builds/downloads through the --max-jobs option.

You could add a separate knob for downloads that defaults to 
--max-jobs.  Or even (* max-jobs cores).  No plans for that AFAIK. 
I don't think it's a trivial tweak.

I'm interested in the numbers behind this claim:

> Currently downloads are a bottleneck for `guix install / 
> upgrade`,
> parallel downloads could reduce the operation duration by an 
> order
> of magnitude.

…because on a substitute-only workload, my default --max-jobs=4 
connections give me 4 MiB/s versus 1.5 MiB/s on a single job.

That's not even a linear increase, let alone an order of magnitude 
(base 2 doesn't count :-).

Kind regards,

T G-R

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Parallel downloads

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 421 bytes --]

Hi Tobias,

I'm not sure I understand: if I run

--8<---------------cut here---------------start------------->8---
guix build -M 4 vlc
--8<---------------cut here---------------end--------------->8---

download progress bars are updated one at a time.  Or am I mistaken?

Also is there a way to change the default value for --max-jobs from 1 to
something else?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Parallel downloads

[zimoun]

Hi Pierre,

Maybe you would like an concurrent output in this spirit:
https://joeyh.name/code/concurrent-output/
Right?

However, I am not sure that parallel downloads drastically change the
performance (bottleneck) because first often it is not linear and
second it strongly depends on the available bandwidth.

All the best,
simon

Re: Parallel downloads

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 151 bytes --]

For sure it depends on the bandwidth.  It would mostly be beneficial to
users with a high bandwidth.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Parallel downloads

[Ludovic Courtès]

Hi Pierre,

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> I'm not sure I understand: if I run
>
> guix build -M 4 vlc
>
> download progress bars are updated one at a time.  Or am I mistaken?

With -M4 you could potentially have 4 substitutions (downloads) going on
in parallel.

Tobias is right: the daemon works in terms of “jobs”, where a job can be
either a build or a substitution (a download).  ‘--max-jobs’ controls
that.

Ludo’.

Re: Parallel downloads

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 167 bytes --]

A few questions:

- How does --mac-jobs work with regard to progress bars?

- Can we configure the default value?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Parallel downloads

[Ludovic Courtès]

Hi,

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> A few questions:
>
> - How does --mac-jobs work with regard to progress bars?

When there are several jobs running at the same time, it doesn’t display
any progress bar.  (In theory, it could do a multi-line display but that
wouldn’t work with all terminals, it’s tricky, and overall not all that
useful.)

> - Can we configure the default value?

Yup, just pass ‘--max-jobs=N’ to the daemon.

HTH!

Ludo’.

Re: Parallel downloads

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 680 bytes --]

Ludovic Courtès <ludo@gnu.org> writes:

>> - Can we configure the default value?
>
> Yup, just pass ‘--max-jobs=N’ to the daemon.

So I suppose you mean to use the `extra-options` field, e.g.

--8<---------------cut here---------------start------------->8---
(guix-service-type config =>
                   (guix-configuration
                    (inherit config)
                    (extra-options '("--max-jobs=4"))))
--8<---------------cut here---------------end--------------->8---

Ludo, what do you think of Tobias suggestion and have an extra knob to
specifically configure the number of download jobs?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Parallel downloads

[Ludovic Courtès]

Hi Pierre!

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>>> - Can we configure the default value?
>>
>> Yup, just pass ‘--max-jobs=N’ to the daemon.
>
> So I suppose you mean to use the `extra-options` field, e.g.
>
> (guix-service-type config =>
>                    (guix-configuration
>                     (inherit config)
>                     (extra-options '("--max-jobs=4"))))

Yes.

> Ludo, what do you think of Tobias suggestion and have an extra knob to
> specifically configure the number of download jobs?

Like I wrote, it’s not that simple (we’d first need the daemon to
distinguish substitution jobs from other jobs, but note that there are
also “downloads” that are actually derivation builds), and it’s not
clear to me that it’s overall beneficial anyway: it’s not supposed to be
faster to download 10 things in parallel from ci.guix.gnu.org, than to
download them sequentially.

If it _is_ faster, then we need to investigate why that is the case.
For example, I’m aware that for some reason, nginx gives us low
bandwidth when downloading a nar that’s not already in its cache.  Its
probably an nginx misconfiguration issue but I couldn’t find out.  (This
has been discussed before on one of the mailing lists, I think.)

Does that make sense?  :-)

Thanks,
Ludo’.

Re: Parallel downloads

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 623 bytes --]

Ludovic Courtès <ludo@gnu.org> writes:

> it’s not supposed to be
> faster to download 10 things in parallel from ci.guix.gnu.org, than to
> download them sequentially.

I was thinking a little bit ahead in the future, e.g. when we have more
build farms (if it ever happens), or with Torrent / IPFS support.  Then
I believe it would be highly beneficial: a sufficiently fast  connection
could effectively parallelize enough downloads to fill the bandwidth.

Also what about the CDNs?  Is it currently possible to download from
multiple CDNs at the same time?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Parallel downloads

[Ludovic Courtès]

Hello,

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> it’s not supposed to be
>> faster to download 10 things in parallel from ci.guix.gnu.org, than to
>> download them sequentially.
>
> I was thinking a little bit ahead in the future, e.g. when we have more
> build farms (if it ever happens), or with Torrent / IPFS support.  Then
> I believe it would be highly beneficial: a sufficiently fast  connection
> could effectively parallelize enough downloads to fill the bandwidth.

I think we’ll have to revisit this issue at that point; it’s hard to
discuss things in the abstract.  :-)

> Also what about the CDNs?  Is it currently possible to download from
> multiple CDNs at the same time?

We’d have to ask Chris Marusich about that, but I think CDNs should
typically use all the available bandwidth so I suspect there’s little to
be gained by having several connections in parallel.

Ludo’.

Re: Parallel downloads

[John Soo]

Hi everyone,

I’ve been watching this from afar and one thing and while I have to agree with this:

> .. I suspect there’s little to
> be gained by having several connections in parallel.

I do have to say that more fine grained concurrency would really help speed up builds without substitutes.

 Especially on old hardware, some builds can actually exhaust all resources. That means you can’t really get a speed up by bumping the max jobs. 

What would help is doing downloads of sources and substitutes asynchronously while a non-substitute job is taking place. If some network activity could be backgrounded, I think you might find build times decrease a lot. 

Thanks!

John

Re: Parallel downloads

[zimoun]

Hi,
It is not related with parallel download but on old machines "guix
build --no-substitutes" can eat a lot of resources; for example if the
package has a lot of dependencies. I would like to be able to list
which dependencies I want to build and which I want to substitute.
Maybe it is already possible.

Currently, I am doing:

  guix build `guix show PKG | recsel -R dependencies`
  guix build PKG --no-substitutes


What do you think?


All the best,
simon

Re: Parallel downloads

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 936 bytes --]

On Tue, Nov 12, 2019 at 05:48:16PM +0100, zimoun wrote:
> Hi,
> It is not related with parallel download but on old machines "guix
> build --no-substitutes" can eat a lot of resources; for example if the
> package has a lot of dependencies. I would like to be able to list
> which dependencies I want to build and which I want to substitute.
> Maybe it is already possible.
> 
> Currently, I am doing:
> 
>   guix build `guix show PKG | recsel -R dependencies`
>   guix build PKG --no-substitutes
> 
> 
> What do you think?
> 

Not a true solution of 'guix build foo bar --no-substitutes baz' but for
your example you can do it in one line with:
'guix environment PKG -- guix build PKG --no-subsitutes'

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Parallel downloads

[zimoun]

Hi Efraim,

On Wed, 13 Nov 2019 at 08:44, Efraim Flashner <efraim@flashner.co.il> wrote:

> >   guix build `guix show PKG | recsel -R dependencies`
> >   guix build PKG --no-substitutes

> 'guix environment PKG -- guix build PKG --no-subsitutes'

Thank you.
It is a better solution, indeed! :-)

All the best,
simon

Re: Parallel downloads

[Leo Famulari]

On Sat, Nov 09, 2019 at 06:40:56PM +0100, Ludovic Courtès wrote:
> Like I wrote, it’s not that simple (we’d first need the daemon to
> distinguish substitution jobs from other jobs, but note that there are
> also “downloads” that are actually derivation builds), and it’s not
> clear to me that it’s overall beneficial anyway: it’s not supposed to be
> faster to download 10 things in parallel from ci.guix.gnu.org, than to
> download them sequentially.

Parallel downloading is not faster in terms of overall transfer rate
from ci.guix.gnu.org.

However, installing things with Guix involves downloading a lot of very
small files like derivations, and Guix spends a lot of time initiating
these downloads.

For example, I can download things at 100 megabits, but when Guix needs
to sequentially download 50 10-kilobyte files, it may take an entire
minute.

So there is a huge speedup with parallel downloading.

Re: Parallel downloads

[Ludovic Courtès]

Hi,

Leo Famulari <leo@famulari.name> skribis:

> On Sat, Nov 09, 2019 at 06:40:56PM +0100, Ludovic Courtès wrote:
>> Like I wrote, it’s not that simple (we’d first need the daemon to
>> distinguish substitution jobs from other jobs, but note that there are
>> also “downloads” that are actually derivation builds), and it’s not
>> clear to me that it’s overall beneficial anyway: it’s not supposed to be
>> faster to download 10 things in parallel from ci.guix.gnu.org, than to
>> download them sequentially.
>
> Parallel downloading is not faster in terms of overall transfer rate
> from ci.guix.gnu.org.
>
> However, installing things with Guix involves downloading a lot of very
> small files like derivations, and Guix spends a lot of time initiating
> these downloads.

Good point.

Note that .drv files are never downloaded, but nevertheless it’s true
that there are often small files like the “module-import-compiled”
things.  This happens when building a system, but not so much when
building a package, though.

> For example, I can download things at 100 megabits, but when Guix needs
> to sequentially download 50 10-kilobyte files, it may take an entire
> minute.
>
> So there is a huge speedup with parallel downloading.

One thing I’d like to get rid of is the initial HTTP GET for
/nix-cache-info which is completely useless now.  That could help a bit.

Thanks,
Ludo’.

Re: Parallel downloads

[Mark H Weaver]

Hi,

Ludovic Courtès <ludo@gnu.org> writes:

> Pierre Neidhardt <mail@ambrevar.xyz> skribis:
>
>> Ludo, what do you think of Tobias suggestion and have an extra knob to
>> specifically configure the number of download jobs?
>
> Like I wrote, it’s not that simple (we’d first need the daemon to
> distinguish substitution jobs from other jobs, but note that there are
> also “downloads” that are actually derivation builds), and it’s not
> clear to me that it’s overall beneficial anyway: it’s not supposed to be
> faster to download 10 things in parallel from ci.guix.gnu.org, than to
> download them sequentially.

I'll also note that parallel downloads would increase the memory usage
on the server.  If users, on average, configured 4 parallel downloads, I
guess that would have the effect of multiplying the server memory usage
by about 4.  It might also create an incentive for users to configure
more parallel downloads in order to grab a larger ratio of the server's
available resources.

For these reasons, I'm inclined to think that parallel downloads is the
wrong approach.  If a single download process is not making efficient
use of the available bandwidth, I'd be more inclined to look carefully
at why it's failing to do so.  For example, I'm not sure if this is the
case (and don't have time to look right now), but if the current code
waits until a NAR has finished downloading before asking for the next
one, that's an issue that could be fixed by use of HTTP pipelining,
without multiplying the memory usage.

What do you think?

      Mark

Re: Parallel downloads

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 142 bytes --]

Maybe the solution to this would be to only allow parallel downloads
from separate servers.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Parallel downloads

[Leo Famulari]

On Wed, Nov 13, 2019 at 11:16:53AM -0500, Mark H Weaver wrote:
> For these reasons, I'm inclined to think that parallel downloads is the
> wrong approach.  If a single download process is not making efficient
> use of the available bandwidth, I'd be more inclined to look carefully
> at why it's failing to do so.  For example, I'm not sure if this is the
> case (and don't have time to look right now), but if the current code
> waits until a NAR has finished downloading before asking for the next
> one, that's an issue that could be fixed by use of HTTP pipelining,
> without multiplying the memory usage.
> 
> What do you think?

I agree that parallel downloads is a kludge to work around the issue of
slow set-up and tear-down of our download code. Pipelining would help a
lot, and we could also profile the relevant Guile code to see if there
are any easy speedups.

This issue was actually discussed a year ago:

https://lists.gnu.org/archive/html/guix-devel/2018-11/msg00148.html

I'll quote Ludo's suggestion from then:

> I’d be in favor of a solution where ‘guix substitute’ is kept alive
> across substitutions (like what happens with ‘guix substitute --query’),
> which would allow it to keep connections alive and thus save the TLS
> handshake and a few extra round trips per download.

Re: Parallel downloads

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1110 bytes --]

Mark H Weaver <mhw@netris.org> writes:

> For these reasons, I'm inclined to think that parallel downloads is the
> wrong approach.  If a single download process is not making efficient
> use of the available bandwidth, I'd be more inclined to look carefully
> at why it's failing to do so.  For example, I'm not sure if this is the
> case (and don't have time to look right now), but if the current code
> waits until a NAR has finished downloading before asking for the next
> one, that's an issue that could be fixed by use of HTTP pipelining,
> without multiplying the memory usage.

I think so too.

More generally, the Guix daemon jobs can be categorized in 2: downloads
and builds.  These 2 categories demands almost complementary hardware
resources for the local machine.

With that in mind, we could have 2 pipelines: one for the builds and one
for the downloads.  This, I think, is general enough that it could be
used by default and improve performance for everyone, regardless of your
Internet bandwidth or CPU power.

Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Parallel downloads

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 479 bytes --]

Update: I've been using --max-jobs=2 by default for about 2 weeks now,
and it feels like a much smoother experience overall: faster downloads, faster
builds.

This is obviously a very dumb "optimization" but at least it serves to
underline that Guix could still do much better by parallelizing
downloads and build in a smart way.

I couple of ideas were mentioned in this thread: Anyone interested in
working on them? :)

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Parallel downloads

[Brett Gilio]

Pierre Neidhardt <mail@ambrevar.xyz> writes:

> Update: I've been using --max-jobs=2 by default for about 2 weeks now,
> and it feels like a much smoother experience overall: faster downloads, faster
> builds.
>
> This is obviously a very dumb "optimization" but at least it serves to
> underline that Guix could still do much better by parallelizing
> downloads and build in a smart way.

Agree.

>
> I couple of ideas were mentioned in this thread: Anyone interested in
> working on them? :)

When I get some time, I would be more than happy to help with this.

-- 
Brett M. Gilio
Homepage -- https://scm.pw/
GNU Guix -- https://guix.gnu.org/

Re: Parallel downloads

[Ludovic Courtès]

Hi Mark,

Mark H Weaver <mhw@netris.org> skribis:

> For these reasons, I'm inclined to think that parallel downloads is the
> wrong approach.  If a single download process is not making efficient
> use of the available bandwidth, I'd be more inclined to look carefully
> at why it's failing to do so.  For example, I'm not sure if this is the
> case (and don't have time to look right now), but if the current code
> waits until a NAR has finished downloading before asking for the next
> one, that's an issue that could be fixed by use of HTTP pipelining,
> without multiplying the memory usage.

I agree.  There’s HTTP pipelining for narinfos but not for nars.  Worse,
before fetching a nar, we do a GET /nix-cache-info, and in fact we spawn
a new ‘guix substitute’ process for each download (for “historical
reasons”).

So there’s room for optimization there!

Ludo’.

Re: Parallel downloads

[Bengt Richter]

On +2019-11-06 16:34:17 +0100, Ludovic Courtès wrote:
> Hi,
> 
> Pierre Neidhardt <mail@ambrevar.xyz> skribis:
> 
> > A few questions:
> >
> > - How does --mac-jobs work with regard to progress bars?
> 
> When there are several jobs running at the same time, it doesn’t display
> any progress bar.  (In theory, it could do a multi-line display but that
> wouldn’t work with all terminals, it’s tricky, and overall not all that
> useful.)
>

I suppose you could do a single line with cpu numbers like 222232444
but you'd need to compute how long the full line should be and divide by
the number of cpus allocated and make the line buffer writes thread-safe.
... and use base>10 digits potentially ;-)

> > - Can we configure the default value?
> 
> Yup, just pass ‘--max-jobs=N’ to the daemon.
> 
> HTH!
> 
> Ludo’.
> 
--
Regards,
Bengt Richter

Re: Parallel downloads

[Joshua Branson]

Pierre Neidhardt <mail@ambrevar.xyz> writes:

> Hi,
>
> Is there any plan to support parallel downloads?  Currently downloads are a
> bottleneck for `guix install / upgrade`, parallel downloads could reduce
> the operation duration by an order of magnitude.

On a related note, would getting guix to download substitutes via IPFS
mean that one could update one's system faster? 

Thanks,

Joshua

>
> Thoughts?

--
Joshua Branson
Sent from Emacs and Gnus

Re: Parallel downloads

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 298 bytes --]

Joshua Branson <jbranso@dismail.de> writes:

> On a related note, would getting guix to download substitutes via IPFS
> mean that one could update one's system faster? 

As of November 2019, no, IPFS is still too slow.  In the future, maybe :)

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Parallel downloads

[Ludovic Courtès]

Hi,

Joshua Branson <jbranso@dismail.de> skribis:

> Pierre Neidhardt <mail@ambrevar.xyz> writes:
>
>> Hi,
>>
>> Is there any plan to support parallel downloads?  Currently downloads are a
>> bottleneck for `guix install / upgrade`, parallel downloads could reduce
>> the operation duration by an order of magnitude.
>
> On a related note, would getting guix to download substitutes via IPFS
> mean that one could update one's system faster? 

Potentially!  For that I encourage people to pick up the work on
<https://issues.guix.gnu.org/issue/33899> and to start experimenting
with substitutes on IPFS.

Thanks,
Ludo’.