Hydra status

Hydra status

[Ludovic Courtès]

You may have noticed that hydra.gnu.org has been, hmm, unhealthy
lately.  There are several bugs in the offload hook that were fixed
recently, notably fixing errors such as:

  guix build: error: lstat: No such file or directory: ...foo.drv

You may have also seen:

  derivation was garbage-collected prior to build

This one should now be fixed, on the Hydra side:

  https://github.com/NixOS/hydra/commit/cc4645659896113cb8677110a1ed768405709296

So it’s not optimal yes, but making progress.

Thanks,
Ludo’.

Re: Hydra status

[Andreas Enge]

There are still quite a few of the following kind:
   guix build: error: open-file: No such file or directory: "/gnu/store/gpcxqii07ivdk6ny8svhayw7hc98gh8i-xnee-3.18.drv"
See, for instance,
   http://hydra.gnu.org/build/49330
I restarted most of them, but kept this one for reference.

And there are others that, when one tries to restart them, reply with
   500 Internal Server Error
   This build cannot be restarted.
For instance,
   http://hydra.gnu.org/build/47968
What can we do there?

Andreas

Re: Hydra status

[Andreas Enge]

On Tue, Apr 08, 2014 at 10:34:16PM +0200, Ludovic Courtès wrote:
> You may have noticed that hydra.gnu.org has been, hmm, unhealthy
> lately.  There are several bugs in the offload hook that were fixed
> recently, notably fixing errors such as:
>   guix build: error: lstat: No such file or directory: ...foo.drv

As far as I can tell, this one stil persists, see
   http://hydra.gnu.org/build/49425/nixlog/3

Andreas

Re: Hydra status

[Andreas Enge]

And the most mysterious of them all:
   http://hydra.gnu.org/build/49569/nixlog/3

/gnu/store/rf5dkfwss2r402d52kkjbijlv2qjyi0s-texlive-2013-data
/gnu/store/fda07hqbif99wzrab9x3gzqkvddxsx52-texlive-2013

Nothing more!

Andreas

Re: Hydra status

[Ludovic Courtès]

Andreas Enge <andreas@enge.fr> skribis:

> There are still quite a few of the following kind:
>    guix build: error: open-file: No such file or directory: "/gnu/store/gpcxqii07ivdk6ny8svhayw7hc98gh8i-xnee-3.18.drv"
> See, for instance,
>    http://hydra.gnu.org/build/49330
> I restarted most of them, but kept this one for reference.

Hmm, given its date, this one could be problematic.  However, there’s
still the possibility that this is a cached failure (guix-daemon runs
with --cache-failures on hydra.gnu.org.)

> And there are others that, when one tries to restart them, reply with
>    500 Internal Server Error
>    This build cannot be restarted.
> For instance,
>    http://hydra.gnu.org/build/47968
> What can we do there?

I installed the offload fixes (specifically 4b00f34 plus a related setup
issue on chapters.gnu.org) on Apr. 8th.  This one dates from before;
presumably its .drv has been GC’d, which is why it cannot be restarted.
So no worries on this one.

Ludo’.

Re: Hydra status

[Ludovic Courtès]

Andreas Enge <andreas@enge.fr> skribis:

> And the most mysterious of them all:
>    http://hydra.gnu.org/build/49569/nixlog/3
>
> /gnu/store/rf5dkfwss2r402d52kkjbijlv2qjyi0s-texlive-2013-data
> /gnu/store/fda07hqbif99wzrab9x3gzqkvddxsx52-texlive-2013

Going to http://hydra.gnu.org/build/49569/ shows that the texlive builds
were actually aborted.  The “Nix error output” box shows this:

--8<---------------cut here---------------start------------->8---
offloading '/gnu/store/j0b550qfqaffffc1w550v0zq2nlb0sqc-texlive-2013.drv' to 'hydra.gnunet.org'...
@ build-remote /gnu/store/j0b550qfqaffffc1w550v0zq2nlb0sqc-texlive-2013.drv hydra.gnunet.org
retrieving 2 files from 'hydra.gnunet.org'...
Backtrace:
In ice-9/boot-9.scm:
2401: 19 [save-module-excursion #<procedure 1acc980 at ice-9/boot-9.scm:4045:3 ()>]
4050: 18 [#<procedure 1acc980 at ice-9/boot-9.scm:4045:3 ()>]
1724: 17 [%start-stack load-stack ...]
1729: 16 [#<procedure 1ad85d0 ()>]
In unknown file:
   ?: 15 [primitive-load "/usr/local/bin/guix"]
In guix/ui.scm:
 597: 14 [run-guix-command offload "x86_64-linux" "3600" "1" "72000"]
In guix/scripts/offload.scm:
 686: 13 [guix-offload "x86_64-linux" "3600" "1" "72000"]
In ice-9/boot-9.scm:
 157: 12 [catch srfi-34 ...]
In guix/scripts/offload.scm:
 413: 11 [transfer-and-offload # # # ...]
 488: 10 [retrieve-files # #]
In ice-9/boot-9.scm:
 157: 9 [catch srfi-34 ...]
In guix/scripts/offload.scm:
 497: 8 [#<procedure 262b660 at guix/scripts/offload.scm:488:11 ()>]
In guix/utils.scm:
 240: 7 [call-with-decompressed-port xz #<closed: file 0> ...]
In guix/nar.scm:
 426: 6 [loop 1 ()]
 256: 5 [restore "/gnu/store/guix-8uVCz3"]
 256: 4 [restore "/gnu/store/guix-8uVCz3/texmf-dist"]
 256: 3 [restore "/gnu/store/guix-8uVCz3/texmf-dist/doc"]
 256: 2 [restore "/gnu/store/guix-8uVCz3/texmf-dist/doc/generic"]
 248: 1 [restore "/gnu/store/guix-8uVCz3/texmf-dist/doc/generic/babel-swedish"]
In unknown file:
   ?: 0 [mkdir "/gnu/store/guix-8uVCz3/texmf-dist/doc/generic/babel-swedish" ...]

ERROR: In procedure mkdir:
ERROR: In procedure mkdir: No such file or directory
lsh: io.c: write failed (errno = 32): Broken pipe
lsh: channel.c: stdout: I/O error, Broken pipe
lsh: Ignoring data on channel which is closing
builder for `/gnu/store/j0b550qfqaffffc1w550v0zq2nlb0sqc-texlive-2013.drv' failed with exit code 1
@ hook-failed /gnu/store/j0b550qfqaffffc1w550v0zq2nlb0sqc-texlive-2013.drv - 256 builder for `/gnu/store/j0b550qfqaffffc1w550v0zq2nlb0sqc-texlive-2013.drv' failed with exit code 1
error: build of `/gnu/store/j0b550qfqaffffc1w550v0zq2nlb0sqc-texlive-2013.drv' failed
--8<---------------cut here---------------end--------------->8---

So the problem occurred while retrieving and restoring the build result
(the log shows just these two lines because apparently, the result was
already available on the build machine hydra.gnunet.org–which is no
surprise given that /gnu is 1.5T on that machine, vs. 400G on
hydra.gnu.org.)

The ‘mkdir’ error here suggests that the temporary place were we are
restoring, /gnu/store/guix-8uVCz3, has been removed right in the middle
of the process by the GC.

I believe this is fixed by 9132b9b.

Thanks!

Ludo’.

Re: Hydra status

[Andreas Enge]

On Wed, Apr 09, 2014 at 10:35:40PM +0200, Ludovic Courtès wrote:
> Andreas Enge <andreas@enge.fr> skribis:
> > And there are others that, when one tries to restart them, reply with
> >    500 Internal Server Error
> >    This build cannot be restarted.
> I installed the offload fixes (specifically 4b00f34 plus a related setup
> issue on chapters.gnu.org) on Apr. 8th.  This one dates from before;
> presumably its .drv has been GC’d, which is why it cannot be restarted.
> So no worries on this one.

Okay for the future. But can we force hydra to recompile the package now
instead of only after the next change in the package or one of its inputs?
Particularly annoying is gnu-pw-mgr:
   http://hydra.gnu.org/build/45105/nixlog/3/tail-reload
From the error message, it looks as if the new download location of 8bae41a
has not been taken into account.

Andreas

Re: Hydra status

[Andreas Enge]

On Wed, Apr 09, 2014 at 11:08:32PM +0200, Ludovic Courtès wrote:
> Going to http://hydra.gnu.org/build/49569/ shows that the texlive builds
> were actually aborted.  The “Nix error output” box shows this:

Now texlive is marked as failed, with the following message at
   http://hydra.gnu.org/build/49569/nixlog/5

guix build: error: open-file: No such file or directory: "/gnu/store/j0b550qfqaffffc1w550v0zq2nlb0sqc-texlive-2013.drv"

Andreas

Re: Hydra status

[Andreas Enge]

On Thu, Apr 10, 2014 at 11:45:05AM +0200, Andreas Enge wrote:
> Now texlive is marked as failed, with the following message at
> guix build: error: open-file: No such file or directory: "/gnu/store/j0b550qfqaffffc1w550v0zq2nlb0sqc-texlive-2013.drv"

And even fresher:
   http://hydra.gnu.org/build/49864
after commit 1f3838a.

Andreas

Re: Hydra status

[Ludovic Courtès]

Andreas Enge <andreas@enge.fr> skribis:

> On Thu, Apr 10, 2014 at 11:45:05AM +0200, Andreas Enge wrote:
>> Now texlive is marked as failed, with the following message at
>> guix build: error: open-file: No such file or directory: "/gnu/store/j0b550qfqaffffc1w550v0zq2nlb0sqc-texlive-2013.drv"
>
> And even fresher:
>    http://hydra.gnu.org/build/49864
> after commit 1f3838a.

Thanks.

I haven’t forgotten about the issue, but my investigations haven’t yet
been fruitful.

Ludo’.

Re: Hydra status

[Ludovic Courtès]

ludo@gnu.org (Ludovic Courtès) skribis:

> Going to http://hydra.gnu.org/build/49569/ shows that the texlive builds
> were actually aborted.  The “Nix error output” box shows this:
>
> offloading '/gnu/store/j0b550qfqaffffc1w550v0zq2nlb0sqc-texlive-2013.drv' to 'hydra.gnunet.org'...
> @ build-remote /gnu/store/j0b550qfqaffffc1w550v0zq2nlb0sqc-texlive-2013.drv hydra.gnunet.org
> retrieving 2 files from 'hydra.gnunet.org'...
> Backtrace:
> In ice-9/boot-9.scm:
> 2401: 19 [save-module-excursion #<procedure 1acc980 at ice-9/boot-9.scm:4045:3 ()>]
> 4050: 18 [#<procedure 1acc980 at ice-9/boot-9.scm:4045:3 ()>]
> 1724: 17 [%start-stack load-stack ...]
> 1729: 16 [#<procedure 1ad85d0 ()>]
> In unknown file:
>    ?: 15 [primitive-load "/usr/local/bin/guix"]
> In guix/ui.scm:
>  597: 14 [run-guix-command offload "x86_64-linux" "3600" "1" "72000"]
> In guix/scripts/offload.scm:
>  686: 13 [guix-offload "x86_64-linux" "3600" "1" "72000"]
> In ice-9/boot-9.scm:
>  157: 12 [catch srfi-34 ...]
> In guix/scripts/offload.scm:
>  413: 11 [transfer-and-offload # # # ...]
>  488: 10 [retrieve-files # #]
> In ice-9/boot-9.scm:
>  157: 9 [catch srfi-34 ...]
> In guix/scripts/offload.scm:
>  497: 8 [#<procedure 262b660 at guix/scripts/offload.scm:488:11 ()>]
> In guix/utils.scm:
>  240: 7 [call-with-decompressed-port xz #<closed: file 0> ...]
> In guix/nar.scm:
>  426: 6 [loop 1 ()]
>  256: 5 [restore "/gnu/store/guix-8uVCz3"]
>  256: 4 [restore "/gnu/store/guix-8uVCz3/texmf-dist"]
>  256: 3 [restore "/gnu/store/guix-8uVCz3/texmf-dist/doc"]
>  256: 2 [restore "/gnu/store/guix-8uVCz3/texmf-dist/doc/generic"]
>  248: 1 [restore "/gnu/store/guix-8uVCz3/texmf-dist/doc/generic/babel-swedish"]
> In unknown file:
>    ?: 0 [mkdir "/gnu/store/guix-8uVCz3/texmf-dist/doc/generic/babel-swedish" ...]
>
> ERROR: In procedure mkdir:
> ERROR: In procedure mkdir: No such file or directory
> lsh: io.c: write failed (errno = 32): Broken pipe
> lsh: channel.c: stdout: I/O error, Broken pipe
> lsh: Ignoring data on channel which is closing
> builder for `/gnu/store/j0b550qfqaffffc1w550v0zq2nlb0sqc-texlive-2013.drv' failed with exit code 1
> @ hook-failed /gnu/store/j0b550qfqaffffc1w550v0zq2nlb0sqc-texlive-2013.drv - 256 builder for `/gnu/store/j0b550qfqaffffc1w550v0zq2nlb0sqc-texlive-2013.drv' failed with exit code 1
> error: build of `/gnu/store/j0b550qfqaffffc1w550v0zq2nlb0sqc-texlive-2013.drv' failed
>
> So the problem occurred while retrieving and restoring the build result
> (the log shows just these two lines because apparently, the result was
> already available on the build machine hydra.gnunet.org–which is no
> surprise given that /gnu is 1.5T on that machine, vs. 400G on
> hydra.gnu.org.)
>
> The ‘mkdir’ error here suggests that the temporary place were we are
> restoring, /gnu/store/guix-8uVCz3, has been removed right in the middle
> of the process by the GC.
>
> I believe this is fixed by 9132b9b.

Actually no (I’ve seen other occurrences of that in the meantime.)

9132b9b had a thinko, which is (hopefully!) fixed by 6071b55.  I’ve just
installed it on hydra.gnu.org.

Ludo’.

Re: Hydra status

[Ludovic Courtès]

Andreas Enge <andreas@enge.fr> skribis:

> On Wed, Apr 09, 2014 at 11:08:32PM +0200, Ludovic Courtès wrote:
>> Going to http://hydra.gnu.org/build/49569/ shows that the texlive builds
>> were actually aborted.  The “Nix error output” box shows this:
>
> Now texlive is marked as failed, with the following message at
>    http://hydra.gnu.org/build/49569/nixlog/5
>
> guix build: error: open-file: No such file or directory: "/gnu/store/j0b550qfqaffffc1w550v0zq2nlb0sqc-texlive-2013.drv"

I did some testing and I think 30ce801 solves this problem.

The issue was actually not a GC issue, but a synchronization issue.
Since I introduced ‘call-with-compressed-output-port’ & co., the
‘send-files’ and ‘retrieve-files’ procedures in offload.scm no longer
explicitly waited for the lsh process to complete.

Thus, the ‘transfer-and-offload’ procedure could call the ‘offload’
procedure (which invokes the remote ‘guix build’) before ‘send-files’
was done uploading the .drv file, hence the error.

I’ve just installed the change on hydra.gnu.org, so let’s see how things
work now.

Thanks,
Ludo’.

Re: Hydra status

[Ludovic Courtès]

ludo@gnu.org (Ludovic Courtès) skribis:

> ludo@gnu.org (Ludovic Courtès) skribis:
>
>> Going to http://hydra.gnu.org/build/49569/ shows that the texlive builds
>> were actually aborted.  The “Nix error output” box shows this:
>>
>> offloading '/gnu/store/j0b550qfqaffffc1w550v0zq2nlb0sqc-texlive-2013.drv' to 'hydra.gnunet.org'...
>> @ build-remote /gnu/store/j0b550qfqaffffc1w550v0zq2nlb0sqc-texlive-2013.drv hydra.gnunet.org
>> retrieving 2 files from 'hydra.gnunet.org'...
>> Backtrace:
>> In ice-9/boot-9.scm:
>> 2401: 19 [save-module-excursion #<procedure 1acc980 at ice-9/boot-9.scm:4045:3 ()>]
>> 4050: 18 [#<procedure 1acc980 at ice-9/boot-9.scm:4045:3 ()>]
>> 1724: 17 [%start-stack load-stack ...]
>> 1729: 16 [#<procedure 1ad85d0 ()>]
>> In unknown file:
>>    ?: 15 [primitive-load "/usr/local/bin/guix"]
>> In guix/ui.scm:
>>  597: 14 [run-guix-command offload "x86_64-linux" "3600" "1" "72000"]
>> In guix/scripts/offload.scm:
>>  686: 13 [guix-offload "x86_64-linux" "3600" "1" "72000"]
>> In ice-9/boot-9.scm:
>>  157: 12 [catch srfi-34 ...]
>> In guix/scripts/offload.scm:
>>  413: 11 [transfer-and-offload # # # ...]
>>  488: 10 [retrieve-files # #]
>> In ice-9/boot-9.scm:
>>  157: 9 [catch srfi-34 ...]
>> In guix/scripts/offload.scm:
>>  497: 8 [#<procedure 262b660 at guix/scripts/offload.scm:488:11 ()>]
>> In guix/utils.scm:
>>  240: 7 [call-with-decompressed-port xz #<closed: file 0> ...]
>> In guix/nar.scm:
>>  426: 6 [loop 1 ()]
>>  256: 5 [restore "/gnu/store/guix-8uVCz3"]
>>  256: 4 [restore "/gnu/store/guix-8uVCz3/texmf-dist"]
>>  256: 3 [restore "/gnu/store/guix-8uVCz3/texmf-dist/doc"]
>>  256: 2 [restore "/gnu/store/guix-8uVCz3/texmf-dist/doc/generic"]
>>  248: 1 [restore "/gnu/store/guix-8uVCz3/texmf-dist/doc/generic/babel-swedish"]
>> In unknown file:
>>    ?: 0 [mkdir "/gnu/store/guix-8uVCz3/texmf-dist/doc/generic/babel-swedish" ...]
>>
>> ERROR: In procedure mkdir:
>> ERROR: In procedure mkdir: No such file or directory
>> lsh: io.c: write failed (errno = 32): Broken pipe
>> lsh: channel.c: stdout: I/O error, Broken pipe
>> lsh: Ignoring data on channel which is closing
>> builder for `/gnu/store/j0b550qfqaffffc1w550v0zq2nlb0sqc-texlive-2013.drv' failed with exit code 1
>> @ hook-failed /gnu/store/j0b550qfqaffffc1w550v0zq2nlb0sqc-texlive-2013.drv - 256 builder for `/gnu/store/j0b550qfqaffffc1w550v0zq2nlb0sqc-texlive-2013.drv' failed with exit code 1
>> error: build of `/gnu/store/j0b550qfqaffffc1w550v0zq2nlb0sqc-texlive-2013.drv' failed
>>
>> So the problem occurred while retrieving and restoring the build result
>> (the log shows just these two lines because apparently, the result was
>> already available on the build machine hydra.gnunet.org–which is no
>> surprise given that /gnu is 1.5T on that machine, vs. 400G on
>> hydra.gnu.org.)
>>
>> The ‘mkdir’ error here suggests that the temporary place were we are
>> restoring, /gnu/store/guix-8uVCz3, has been removed right in the middle
>> of the process by the GC.
>>
>> I believe this is fixed by 9132b9b.
>
> Actually no (I’ve seen other occurrences of that in the meantime.)
>
> 9132b9b had a thinko, which is (hopefully!) fixed by 6071b55.  I’ve just
> installed it on hydra.gnu.org.

There was another problem with that “fix”, namely that the daemon
ignores normal GC roots pointing to “abnormal” store files like
/gnu/store/guix-8uVCz3.

This led to the (now abandoned) proposal at
<https://lists.gnu.org/archive/html/guix-devel/2014-04/msg00198.html>.
I finally came up with a simpler solution in commit 50db7d8: a
connection to the daemon is opened for the sake of holding a per-process
GC root to that temporary store file.

Ludo’.