bug#43564: cuirass: Contention while registering new builds.

[Mathieu Othacehe <othacehe@gnu.org>]

Hello,

> As each evaluation has to register around 50k builds, there might be
> some sort of "writing" contention on the database, explaining the long
> build registration time.

Turns out, once an evaluation is over, new builds are registered. This
registration tries to insert a new build for each derivation returned by
the evaluation phase. If the new build does not add a new output, the
insertion query is then rollbacked. This means that there are at least
as many insertion queries as new derivations.

SQlite allows at most one writer at a time, and even though we are using WAL
mode, performing a lot of insertions will reduce the reading performances.
When multiple evaluations are performed in parallel, the large number of
concurrent insertion queries also causes contention.

Having 10 pending evaluations means that we are probably trying to
insert around 500.000 records concurrently. This is something that
SQLite does not seem to be designed for.

To avoid those issues, we need to check first in the "Outputs" table
which derivations are already registered. This means that most of the
insertion queries will be replaced by reading queries, that are much
less expensive and more suitable for Cuirass concurrent implementation.

In the best case of one pending evaluation, the registration duration is
reduced from 1800 seconds to 320 seconds. I think that the gain is way
larger when there are multiple pending evaluations.

Pushed a fix as 461e07e14e1c8013343c0a2cb26c0e022e10d5e4.

Thanks,

Mathieu