Using HTTP pipelining for substitutes

Using HTTP pipelining for substitutes

[Ludovic Courtès]

Hello!

The new ‘wip-http-pipelining’ branch changes ‘guix substitute-binary’ so
that, instead of using the current brute force approach—a bunch of
threads performing HTTP GETs[*] in parallel to hydra.gnu.org–, it sends all
HTTP GET requests in a row and then reads responses (“HTTP pipeline”.)

I suspect it performs more or less the same as the brute force approach,
but it uses much less resources on the client (memory, file descriptors,
etc.), and likewise on the server (doesn’t have to handle a large number
of parallel connections.)

It should also be more robust, and will make it easy to have progress
report when updating the list of substitutes.

I’ll test it some more and make sure it works with Guile 2.0.5.

Comments welcome!

Ludo’.

[*] The GETs are to check whether hydra.gnu.org has a substitute for a
    given store item.

Re: Using HTTP pipelining for substitutes

[Ludovic Courtès]

ludo@gnu.org (Ludovic Courtès) skribis:

> The new ‘wip-http-pipelining’ branch changes ‘guix substitute-binary’ so
> that, instead of using the current brute force approach—a bunch of
> threads performing HTTP GETs[*] in parallel to hydra.gnu.org–, it sends all
> HTTP GET requests in a row and then reads responses (“HTTP pipeline”.)
>
> I suspect it performs more or less the same as the brute force approach,

Actually that intuition was wrong, because it takes some time for Hydra
to process .narinfo GET requests.

Namely: it runs at 3.5 reqs/s. when signing narinfos (our case), and
37.0 reqs/s. when not signing them.

I’ll see if we can adjust Hydra to do some caching.

Ludo’.