bug#32539: Cuirass: a commit that updates doc will produce 86 tests

bug#32539: Cuirass: a commit that updates doc will produce 86 tests

[Clément Lassieur]

Evaluation 338[1] is triggered by commit 1d8d69c[2], which only updates
the documentation.  This evaluation produces 86 useless tests:
test.mcron.i686-linux, test.opensmtpd.i686-linux, etc.  I believe those
tests are triggered at each evaluation (almost), and it's a considerable
waste of resources.

[1]: https://berlin.guixsd.org/eval/338
[2]: https://git.savannah.gnu.org/cgit/guix.git/commit/?id=1d8d69c863ae88d5fc20e52e49a22f68c531c1db

bug#32539: Cuirass: a commit that updates doc will produce 86 tests

[Ludovic Courtès]

Hi Clément,

Clément Lassieur <clement@lassieur.org> skribis:

> Evaluation 338[1] is triggered by commit 1d8d69c[2], which only updates
> the documentation.  This evaluation produces 86 useless tests:
> test.mcron.i686-linux, test.opensmtpd.i686-linux, etc.  I believe those
> tests are triggered at each evaluation (almost), and it's a considerable
> waste of resources.

I claim that these tests are useful and that having them under CI is a
good thing!

AFAICS the result of test.basic.*, test.mcron.*, etc. should be
unchanged when doc/*.texi are modified; if that’s not the case, we have
a bug. :-)   So there’s no waste of resources in this case.

However the tests in (gnu tests install) do entail a full build of Guix
from the checkout, among other things, at every commit, so they are
expensive.  They are useful though, so I’d rather keep it this way if
our build farm can keep up.

Thoughts?

Ludo’.

bug#32539: Cuirass: a commit that updates doc will produce 86 tests

[Clément Lassieur]

Heya!

Ludovic Courtès <ludo@gnu.org> writes:

> Hi Clément,
>
> Clément Lassieur <clement@lassieur.org> skribis:
>
>> Evaluation 338[1] is triggered by commit 1d8d69c[2], which only updates
>> the documentation.  This evaluation produces 86 useless tests:
>> test.mcron.i686-linux, test.opensmtpd.i686-linux, etc.  I believe those
>> tests are triggered at each evaluation (almost), and it's a considerable
>> waste of resources.
>
> I claim that these tests are useful and that having them under CI is a
> good thing!
>
> AFAICS the result of test.basic.*, test.mcron.*, etc. should be
> unchanged when doc/*.texi are modified; if that’s not the case, we have
> a bug. :-)   So there’s no waste of resources in this case.

My point is that the derivations files are different at each commit,
despite the changes being only about documentation.  It seems to me that
documentation is not an input of the tests, therefore the inputs didn't
change and the test derivations shouldn't change.

> However the tests in (gnu tests install) do entail a full build of Guix
> from the checkout, among other things, at every commit, so they are
> expensive.  They are useful though, so I’d rather keep it this way if
> our build farm can keep up.
>
> Thoughts?
>
> Ludo’.

bug#32539: Cuirass: a commit that updates doc will produce 86 tests

[Mathieu Othacehe]

Hello,

The tests are now evaluated periodically (every 24 hours) which largely
mitigates this issue. Closing this one.

Mathieu