bug#38232: Cuirass: Some builds fail although their log file ends with 'build-succeeded'

bug#38232: Cuirass: Some builds fail although their log file ends with 'build-succeeded'

[Clément Lassieur]

See https://ci.guix.info/build/1934383/details.

bug#38232: Cuirass: Some builds fail although their log file ends with 'build-succeeded'

[Mathieu Othacehe]

Hello,

It seems to be also an issue for evaluations that look successful but
are reported as failed, see here:

https://ci.guix.gnu.org/eval/8641/log/raw -> mark as failed
https://ci.guix.gnu.org/eval/8725/log/raw -> mark as failed

Mathieu

bug#38232: Cuirass: Some builds fail although their log file ends with 'build-succeeded'

[Mathieu Othacehe]

> It seems to be also an issue for evaluations that look successful but
> are reported as failed, see here:

Looking at those evaluation logs, I think there is yet another issue. On
core-updates:

* Evaluation 8747 fails because it cannot build curl
(https://ci.guix.gnu.org/eval/8748/log/raw).

* Next evaluation 8762, considers curl build as previously failed:
--8<---------------cut here---------------start------------->8---
Computing Guix derivation for 'x86_64-linux'...  builder for `/gnu/store/1510sg9h2ivq4841gqm9dhmq5l50a52b-curl-7.66.0' failed previously (cached)
@ build-failed /gnu/store/pqfnxfk5xx0ngc32wlq87vhchh80mnkw-curl-7.66.0.drv - cached--8<---------------cut here---------------end--------------->8---
and fails immediately.

The curl build problem (at 8747 evaluation) seems related to some failed
test case (1242) that I cannot reproduce locally.

Will all future evaluations re-use this cache and also fail immediately?
If yes, how can we unblock the situation?

Mathieu

bug#38232: Cuirass: Some builds fail although their log file ends with 'build-succeeded'

[Ludovic Courtès]

Hi Clément,

Clément Lassieur <clement@lassieur.org> skribis:

> See https://ci.guix.info/build/1934383/details.

Perhaps the build failed (transient failure or something) but was
eventually restarted either as a dependency of some other build or
manually on berlin.  In that case Cuirass may be unaware that the build
eventually succeeded, hence the discrepancy.

Thanks,
Ludo’.

bug#38232: Cuirass: Some builds fail although their log file ends with 'build-succeeded'

[Clément Lassieur]

Hi Ludovic,

> Perhaps the build failed (transient failure or something) but was
> eventually restarted either as a dependency of some other build or
> manually on berlin.  In that case Cuirass may be unaware that the build
> eventually succeeded, hence the discrepancy.

Thank you for the explanation.  Would it make sense to copy the log file
to some other place when the build is done?  The web interface would
fetch it there, and we would be sure it matches the associated Cuirass
build.

Clément

bug#38232: Cuirass: Some builds fail although their log file ends with 'build-succeeded'

[Ludovic Courtès]

Hello,

Clément Lassieur <clement@lassieur.org> skribis:

>> Perhaps the build failed (transient failure or something) but was
>> eventually restarted either as a dependency of some other build or
>> manually on berlin.  In that case Cuirass may be unaware that the build
>> eventually succeeded, hence the discrepancy.
>
> Thank you for the explanation.  Would it make sense to copy the log file
> to some other place when the build is done?  The web interface would
> fetch it there, and we would be sure it matches the associated Cuirass
> build.

Build logs are kept by guix-daemon under /var/log/guix/drvs.  They are
indexed by derivation, meaning that there can be only one log per
derivation.

This is a limitation in cases where builds are retried.  Initially
Mathieu Lirzin thought about keeping logs elsewhere so we could
distinguish between several attempts to build a derivation.  However,
that’s kind of redundant with what the daemon does, and not frequently
useful.

Now, it is indeed useful in some cases, so we could do something like
you describe.

Thanks,
Ludo’.

bug#38232: Cuirass: Some builds fail although their log file ends with 'build-succeeded'

[Mathieu Othacehe]

Hello,

This is no longer an issue on Cuirass master.

Thanks,

Mathieu