Re: Substitute timeouts

[Mathieu Othacehe <othacehe@gnu.org>]

Hey Ludo,

Thanks for taking the time to read my wall of text :D.

> Yeah, it’s a double-edged sword.  If this is a problem on the main ‘guix
> publish’ server, we can lower the bypass threshold, which is currently
> 50 MiB:
>
>   https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/modules/sysadmin/services.scm#n450
>
> WDYT?

That would maybe help, but on the other hand, I would prefer to find a
more definitive solution :).

> First, in terms of UI, you’d have a command sitting there and doing
> nothing, which can be off-putting.  Second, clients have no idea how
> long they’re going to wait; it could be that the nar is going to be
> baked within seconds, or it could take 20mn if the baking queue is
> already crowded or if the user is asking for a big store item like
> libreoffice.  Third, in many cases, building locally is likely to be
> faster than waiting for substitutes to be available (the majority of
> packages build very quickly, though the few most popular leaf packages
> take a long time to build).

It would be interesting to monitor the status of the baking
workers. Could it really take 20 minutes to bake a substitute from your
experience?

Personally, I have always found this baking 404 and bypass cache a bit
misleading. When substituting libreoffice, I would much rather wait a
few minutes than trying to build it while there's an almost ready
substitute. I get that this is a personal choice and maybe it should be
an optional behaviour.

>> It will also allow the Cuirass build farm to use directly the main guix
>> publish server, simplifying the current CI setup.
>
> The only reason why Cuirass runs its own publish server is to avoid
> overloading the main one?

No, the main reason is that with the use of a publish cache, the Cuirass
workers would probably hit 404 errors while the substitutes are being
baked. Using a publish server without cache was a way to work around it.

The motivation of the 202 waiting patch was to solve both problems at
once. Maybe I should explore the narinfo dedicated thread solution as a
short term solution, while starting to think about a more long term
solution based on Fiber/Nginx.

A Cuirass dedicated solution could also be to declare a build successful
only when a nar is available and stop using a non-caching publish
server.

Thanks,

Mathieu