all messages for Guix-related lists mirrored at yhetil.org
 help / color / mirror / code / Atom feed
* Why is this build considered a failure by hydra?
@ 2014-04-03  5:57 Mark H Weaver
  2014-04-03 15:14 ` Ludovic Courtès
  0 siblings, 1 reply; 6+ messages in thread
From: Mark H Weaver @ 2014-04-03  5:57 UTC (permalink / raw)
  To: guix-devel

Looking at http://hydra.gnu.org/jobset/gnu/master , it seems that a
large percentage of our builds have started failing.  I was curious so I
took a look.  The first failure I looked at was 'file', since I recently
upgraded it:

  http://hydra.gnu.org/build/46637

Looking at the tail of the log, it looks like a successful build.
However, hydra reports that it failed with exit code 100.

What's going on here?

     Mark

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Why is this build considered a failure by hydra?
  2014-04-03  5:57 Why is this build considered a failure by hydra? Mark H Weaver
@ 2014-04-03 15:14 ` Ludovic Courtès
  2014-04-03 18:02   ` Andreas Enge
  2014-04-05 11:01   ` Ludovic Courtès
  0 siblings, 2 replies; 6+ messages in thread
From: Ludovic Courtès @ 2014-04-03 15:14 UTC (permalink / raw)
  To: Mark H Weaver; +Cc: guix-devel

Mark H Weaver <mhw@netris.org> skribis:

> Looking at http://hydra.gnu.org/jobset/gnu/master , it seems that a
> large percentage of our builds have started failing.  I was curious so I
> took a look.  The first failure I looked at was 'file', since I recently
> upgraded it:
>
>   http://hydra.gnu.org/build/46637

I’ve just restarted it, and now it looks as succeeding.

> Looking at the tail of the log, it looks like a successful build.
> However, hydra reports that it failed with exit code 100.
>
> What's going on here?

Offloading failed in some way.  hydra.gnunet.org was down for some time,
so perhaps we hit that just at this moment.  Or some networking issue
occurred just as we were retrieving the build outputs to hydra.gnu.org
(although these are normally distinguished from permanent build
failures, and marked as aborted.)  Or there’s a bug in the offload hook.

This is clearly an annoyance.  We must keep monitoring Hydra to see if
there’s any clue of an actual bug or setup issue.

Ludo’.

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Why is this build considered a failure by hydra?
  2014-04-03 15:14 ` Ludovic Courtès
@ 2014-04-03 18:02   ` Andreas Enge
  2014-04-05 11:01   ` Ludovic Courtès
  1 sibling, 0 replies; 6+ messages in thread
From: Andreas Enge @ 2014-04-03 18:02 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: guix-devel

On Thu, Apr 03, 2014 at 05:14:42PM +0200, Ludovic Courtès wrote:
> I’ve just restarted it, and now it looks as succeeding.

This is something I have occasionally observed before, even without offload-
ing: A build log shows that everything worked well, and the build fails.
Restarting the build solved the problem.

> Offloading failed in some way.  hydra.gnunet.org was down for some time,
> so perhaps we hit that just at this moment.  Or some networking issue
> occurred just as we were retrieving the build outputs to hydra.gnu.org
> (although these are normally distinguished from permanent build
> failures, and marked as aborted.)  Or there’s a bug in the offload hook.

So this may be true, but the problem may also be unrelated to offloading.

Andreas

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Why is this build considered a failure by hydra?
  2014-04-03 15:14 ` Ludovic Courtès
  2014-04-03 18:02   ` Andreas Enge
@ 2014-04-05 11:01   ` Ludovic Courtès
  2014-04-08  3:29     ` Mark H Weaver
  1 sibling, 1 reply; 6+ messages in thread
From: Ludovic Courtès @ 2014-04-05 11:01 UTC (permalink / raw)
  To: Mark H Weaver; +Cc: guix-devel

ludo@gnu.org (Ludovic Courtès) skribis:

> Offloading failed in some way.  hydra.gnunet.org was down for some time,
> so perhaps we hit that just at this moment.  Or some networking issue
> occurred just as we were retrieving the build outputs to hydra.gnu.org
> (although these are normally distinguished from permanent build
> failures, and marked as aborted.)  Or there’s a bug in the offload hook.

A couple of days ago I fixed a possible problem: on build slaves, the GC
could run at the wrong moment and wipe the derivation about to be built,
leading to a guix build “No such file or directory” error (see commit
4b00f34.)

Ludo’.

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Why is this build considered a failure by hydra?
  2014-04-05 11:01   ` Ludovic Courtès
@ 2014-04-08  3:29     ` Mark H Weaver
  2014-04-08 10:16       ` Ludovic Courtès
  0 siblings, 1 reply; 6+ messages in thread
From: Mark H Weaver @ 2014-04-08  3:29 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: guix-devel

ludo@gnu.org (Ludovic Courtès) writes:

> ludo@gnu.org (Ludovic Courtès) skribis:
>
>> Offloading failed in some way.  hydra.gnunet.org was down for some time,
>> so perhaps we hit that just at this moment.  Or some networking issue
>> occurred just as we were retrieving the build outputs to hydra.gnu.org
>> (although these are normally distinguished from permanent build
>> failures, and marked as aborted.)  Or there’s a bug in the offload hook.
>
> A couple of days ago I fixed a possible problem: on build slaves, the GC
> could run at the wrong moment and wipe the derivation about to be built,
> leading to a guix build “No such file or directory” error (see commit
> 4b00f34.)

This problem is still happening.  Here's the most successful evaluation
we've had in a while:

  http://hydra.gnu.org/eval/66919

Here are some builds from that evaluation that failed because the
derivation had been deleted:

  http://hydra.gnu.org/build/48451 (asciidoc i686)
  http://hydra.gnu.org/build/48070 (boost i686)
  http://hydra.gnu.org/build/48516 (glib x86_64)
  http://hydra.gnu.org/build/48385 (lapack x86_64)
  http://hydra.gnu.org/build/48494 (python-setuptools x86_64)
  http://hydra.gnu.org/build/48018 (gpgme i686)
  http://hydra.gnu.org/build/48336 (libxdamage i686)
  http://hydra.gnu.org/build/48459 (libxkbfile i686)
  http://hydra.gnu.org/build/48402 (libxmu i686)
  http://hydra.gnu.org/build/47928 (mailutils i686)
  http://hydra.gnu.org/build/48511 (samba i686)
  http://hydra.gnu.org/build/47968 (shadow i686)

All of these are from the last day or two.

     Mark

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Why is this build considered a failure by hydra?
  2014-04-08  3:29     ` Mark H Weaver
@ 2014-04-08 10:16       ` Ludovic Courtès
  0 siblings, 0 replies; 6+ messages in thread
From: Ludovic Courtès @ 2014-04-08 10:16 UTC (permalink / raw)
  To: Mark H Weaver; +Cc: guix-devel

Mark H Weaver <mhw@netris.org> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>
>> ludo@gnu.org (Ludovic Courtès) skribis:
>>
>>> Offloading failed in some way.  hydra.gnunet.org was down for some time,
>>> so perhaps we hit that just at this moment.  Or some networking issue
>>> occurred just as we were retrieving the build outputs to hydra.gnu.org
>>> (although these are normally distinguished from permanent build
>>> failures, and marked as aborted.)  Or there’s a bug in the offload hook.
>>
>> A couple of days ago I fixed a possible problem: on build slaves, the GC
>> could run at the wrong moment and wipe the derivation about to be built,
>> leading to a guix build “No such file or directory” error (see commit
>> 4b00f34.)
>
> This problem is still happening.

[...]

> All of these are from the last day or two.

I fixed it when I sent the email, i.e., ~2.5 hours ago.

There’s a lot of rebuild going on now (due to OpenSSL), so I think we’ll
soon have a better idea of whether the problem still exists.

Ludo’.

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2014-04-08 10:16 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-04-03  5:57 Why is this build considered a failure by hydra? Mark H Weaver
2014-04-03 15:14 ` Ludovic Courtès
2014-04-03 18:02   ` Andreas Enge
2014-04-05 11:01   ` Ludovic Courtès
2014-04-08  3:29     ` Mark H Weaver
2014-04-08 10:16       ` Ludovic Courtès

Code repositories for project(s) associated with this external index

	https://git.savannah.gnu.org/cgit/guix.git

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.