Limiting resource usage on ci.guix.gnu.org

Limiting resource usage on ci.guix.gnu.org

[Ludovic Courtès]

Hello Guix,

Lately ci.guix has regularly been bubbling around 100,000 pending
builds, which cannot be absorbed in a timely fashion:

  https://ci.guix.gnu.org/metrics

Currently there are several jobsets producing many builds:
‘mesa-updates’, ‘meson-parallel-tests’, ‘rust-team’, ‘python-team’.
That’s fine, it’s the whole point of the platform, but I think we should
follow some rules to keep things under control:

  1. Until it’s “your turn” to merge according to the requests to merge
     listed at <https://qa.guix.gnu.org/>, please restrict builds to
     x86_64-linux, unless portability is a significant aspect of the
     branch.  This is necessary due to limited resources for aarch64
     and, to a lesser extent, powerpc64le.

  2. When rebasing on top of ‘master’, please cancel pending builds from
     previous evaluations that have become irrelevant.

If you don’t have a certificate for ci.guix or SSH access to berlin to
perform these tasks, you can ask on IRC or on guix-sysadmin@gnu.org.

Thanks,
Ludo’.

Re: Limiting resource usage on ci.guix.gnu.org

[Greg Hogan]

On Wed, Oct 23, 2024 at 12:14 PM Ludovic Courtès <ludo@gnu.org> wrote:
>
> Hello Guix,

[...]

>   2. When rebasing on top of ‘master’, please cancel pending builds from
>      previous evaluations that have become irrelevant.

How is this issue limited to rebasing to 'master'? For example, if the
rust team pushes a patch increasing the base-rust (causing a rebuild
of all rust packages) would not the previous (when pending) evaluation
need to be cancelled? Why do we not cancel old evaluations
automatically? Any unchanged derivations would be built under the new
evaluation.

Re: Limiting resource usage on ci.guix.gnu.org

[Ludovic Courtès]

Greg Hogan <code@greghogan.com> skribis:

> On Wed, Oct 23, 2024 at 12:14 PM Ludovic Courtès <ludo@gnu.org> wrote:
>>
>> Hello Guix,
>
> [...]
>
>>   2. When rebasing on top of ‘master’, please cancel pending builds from
>>      previous evaluations that have become irrelevant.
>
> How is this issue limited to rebasing to 'master'? For example, if the
> rust team pushes a patch increasing the base-rust (causing a rebuild
> of all rust packages) would not the previous (when pending) evaluation
> need to be cancelled? Why do we not cancel old evaluations
> automatically? Any unchanged derivations would be built under the new
> evaluation.

You’re right, it’s a more general issue.  The idea is: if you know
builds from past evaluations are no longer relevant, please cancel them.

And yes, it would be nice if we could specify, for some jobsets, that
every new evaluation cancels pending builds form the previous one
(unless they’re the same).

Ludo’.