bug#26201: hydra.gnu.org uses ‘guix publish’ for nars and narinfos

[ludo@gnu.org (Ludovic Courtès)]

Hello,

ludo@gnu.org (Ludovic Courtès) skribis:

> Other solutions I’ve thought about:
>
>   1. Produce narinfos and nars periodically rather than on-demand and
>      serve them as static files.
>
>      pros: better HTTP latency and bandwidth
>      pros: allows us to add a Content-Length for nars
>      cons: doesn’t reduce load on hydra.gnu.org
>      cons: introduces arbitrary delays in delivering nars
>      cons: difficult/expensive to know what new store items are available
>
>   2. Produce a narinfo and corresponding nar the first time they are
>      requested.  So, the first time we receive “GET foo.narinfo”, return
>      404 and spawn a thread to compute foo.narinfo and foo.nar.  Return
>      200 only when both are ready.
>
>      The precomputed nar{,info}s would be kept in a cache and we could
>      make sure a narinfo and its nar have the same lifetime, which
>      addresses one of the problems we have.
>
>      pros: better HTTP latency and bandwidth
>      pros: allows us to add a Content-Length for nars
>      pros: helps keep narinfo/nar lifetime in sync
>      cons: doesn’t reduce load on hydra.gnu.org
>      cons: exposes inconsistency between the store contents and the HTTP
>            response (you may get 404 even if the thing is actually in
>            store), but maybe that’s not a problem

The ‘wip-publish-baking’ implements #2 as a new option to ‘guix
publish’.  It gives some control on the upper bound on CPU usage since
we can specify how many worker threads are used.

I’ll finish it soon so we can experiment with it.

Thanks,
Ludo’.