Why is this build considered a failure by hydra?

Why is this build considered a failure by hydra?

[Mark H Weaver]

Looking at http://hydra.gnu.org/jobset/gnu/master , it seems that a
large percentage of our builds have started failing.  I was curious so I
took a look.  The first failure I looked at was 'file', since I recently
upgraded it:

  http://hydra.gnu.org/build/46637

Looking at the tail of the log, it looks like a successful build.
However, hydra reports that it failed with exit code 100.

What's going on here?

     Mark

Re: Why is this build considered a failure by hydra?

[Ludovic Courtès]

Mark H Weaver <mhw@netris.org> skribis:

> Looking at http://hydra.gnu.org/jobset/gnu/master , it seems that a
> large percentage of our builds have started failing.  I was curious so I
> took a look.  The first failure I looked at was 'file', since I recently
> upgraded it:
>
>   http://hydra.gnu.org/build/46637

I’ve just restarted it, and now it looks as succeeding.

> Looking at the tail of the log, it looks like a successful build.
> However, hydra reports that it failed with exit code 100.
>
> What's going on here?

Offloading failed in some way.  hydra.gnunet.org was down for some time,
so perhaps we hit that just at this moment.  Or some networking issue
occurred just as we were retrieving the build outputs to hydra.gnu.org
(although these are normally distinguished from permanent build
failures, and marked as aborted.)  Or there’s a bug in the offload hook.

This is clearly an annoyance.  We must keep monitoring Hydra to see if
there’s any clue of an actual bug or setup issue.

Ludo’.

Re: Why is this build considered a failure by hydra?

[Andreas Enge]

On Thu, Apr 03, 2014 at 05:14:42PM +0200, Ludovic Courtès wrote:
> I’ve just restarted it, and now it looks as succeeding.

This is something I have occasionally observed before, even without offload-
ing: A build log shows that everything worked well, and the build fails.
Restarting the build solved the problem.

> Offloading failed in some way.  hydra.gnunet.org was down for some time,
> so perhaps we hit that just at this moment.  Or some networking issue
> occurred just as we were retrieving the build outputs to hydra.gnu.org
> (although these are normally distinguished from permanent build
> failures, and marked as aborted.)  Or there’s a bug in the offload hook.

So this may be true, but the problem may also be unrelated to offloading.

Andreas

Re: Why is this build considered a failure by hydra?

[Ludovic Courtès]

ludo@gnu.org (Ludovic Courtès) skribis:

> Offloading failed in some way.  hydra.gnunet.org was down for some time,
> so perhaps we hit that just at this moment.  Or some networking issue
> occurred just as we were retrieving the build outputs to hydra.gnu.org
> (although these are normally distinguished from permanent build
> failures, and marked as aborted.)  Or there’s a bug in the offload hook.

A couple of days ago I fixed a possible problem: on build slaves, the GC
could run at the wrong moment and wipe the derivation about to be built,
leading to a guix build “No such file or directory” error (see commit
4b00f34.)

Ludo’.

Re: Why is this build considered a failure by hydra?

[Mark H Weaver]

ludo@gnu.org (Ludovic Courtès) writes:

> ludo@gnu.org (Ludovic Courtès) skribis:
>
>> Offloading failed in some way.  hydra.gnunet.org was down for some time,
>> so perhaps we hit that just at this moment.  Or some networking issue
>> occurred just as we were retrieving the build outputs to hydra.gnu.org
>> (although these are normally distinguished from permanent build
>> failures, and marked as aborted.)  Or there’s a bug in the offload hook.
>
> A couple of days ago I fixed a possible problem: on build slaves, the GC
> could run at the wrong moment and wipe the derivation about to be built,
> leading to a guix build “No such file or directory” error (see commit
> 4b00f34.)

This problem is still happening.  Here's the most successful evaluation
we've had in a while:

  http://hydra.gnu.org/eval/66919

Here are some builds from that evaluation that failed because the
derivation had been deleted:

  http://hydra.gnu.org/build/48451 (asciidoc i686)
  http://hydra.gnu.org/build/48070 (boost i686)
  http://hydra.gnu.org/build/48516 (glib x86_64)
  http://hydra.gnu.org/build/48385 (lapack x86_64)
  http://hydra.gnu.org/build/48494 (python-setuptools x86_64)
  http://hydra.gnu.org/build/48018 (gpgme i686)
  http://hydra.gnu.org/build/48336 (libxdamage i686)
  http://hydra.gnu.org/build/48459 (libxkbfile i686)
  http://hydra.gnu.org/build/48402 (libxmu i686)
  http://hydra.gnu.org/build/47928 (mailutils i686)
  http://hydra.gnu.org/build/48511 (samba i686)
  http://hydra.gnu.org/build/47968 (shadow i686)

All of these are from the last day or two.

     Mark

Re: Why is this build considered a failure by hydra?

[Ludovic Courtès]

Mark H Weaver <mhw@netris.org> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>
>> ludo@gnu.org (Ludovic Courtès) skribis:
>>
>>> Offloading failed in some way.  hydra.gnunet.org was down for some time,
>>> so perhaps we hit that just at this moment.  Or some networking issue
>>> occurred just as we were retrieving the build outputs to hydra.gnu.org
>>> (although these are normally distinguished from permanent build
>>> failures, and marked as aborted.)  Or there’s a bug in the offload hook.
>>
>> A couple of days ago I fixed a possible problem: on build slaves, the GC
>> could run at the wrong moment and wipe the derivation about to be built,
>> leading to a guix build “No such file or directory” error (see commit
>> 4b00f34.)
>
> This problem is still happening.

[...]

> All of these are from the last day or two.

I fixed it when I sent the email, i.e., ~2.5 hours ago.

There’s a lot of rebuild going on now (due to OpenSSL), so I think we’ll
soon have a better idea of whether the problem still exists.

Ludo’.