GNUnet build failure on mips64el

GNUnet build failure on mips64el

[Mark H Weaver]

[-- Attachment #1: Type: text/plain, Size: 2308 bytes --]

Hi Sree,

I had said on IRC that I successfully built GNUnet on mips64el, but that
was due to my faulty memory.  In fact, I've not yet successfully passed
GNUnet's test suite.

Here's what happened with my most recent attempt.  Interestingly,
although the 'testbed_api_topology' test passed, much later in the test
suite I see warnings about timeouts, with testbed-api-topology mentioned
in the warnings, and then there's an hour of silence before guix-daemon
cancels the build.

--8<---------------cut here---------------start------------->8---
make  check-TESTS
make[3]: Entering directory '/tmp/nix-build-gnunet-0.10.0.drv-1/gnunet-0.10.0/src/revocation'
Creating an ego revoc_test
Testing key 4UVTFR895A3HUCG5PSDFM4AJEV8NNG4MVD5V5QDN73DA4P8QU61G
Key was valid
Revoking key 4UVTFR895A3HUCG5PSDFM4AJEV8NNG4MVD5V5QDN73DA4P8QU61G
Revocation certificate not ready, calculating proof of work
.. - @  12% (estimate)
Key for ego `revoc_test' has been successfully revoked
Testing revoked key 4UVTFR895A3HUCG5PSDFM4AJEV8NNG4MVD5V5QDN73DA4P8QU61G
Key was revoked
PASS: test_local_revocation.py
Feb 21 04:28:41-858886 testbed-api-topology-26165 WARNING Error while establishing a link: 0x4: Timeout during TRANSPORT_try_connect() at peer 6ULB -- Retrying
Feb 21 04:28:42-222351 testbed-api-topology-26165 WARNING Error while establishing a link: 0x5: Timeout during TRANSPORT_try_connect() at peer 6DER -- Retrying
Feb 21 04:28:47-254943 testbed-api-topology-26165 WARNING Error while establishing a link: 0x6: Timeout during TRANSPORT_try_connect() at peer 6ULB -- Retrying
Feb 21 04:28:47-274780 testbed-api-topology-26165 WARNING Error while establishing a link: 0x7: Timeout during TRANSPORT_try_connect() at peer 6DER -- Retrying
building of `/nix/store/vj5s3nd8mvbf8fjd9m6aikjcb5qnnhim-gnunet-0.10.0.drv' timed out after 3600 seconds of silence
@ build-failed /nix/store/vj5s3nd8mvbf8fjd9m6aikjcb5qnnhim-gnunet-0.10.0.drv - timeout
note: keeping build directory `/tmp/nix-build-gnunet-0.10.0.drv-1'
killing process 2543
guix build: error: build failed: build of `/nix/store/vj5s3nd8mvbf8fjd9m6aikjcb5qnnhim-gnunet-0.10.0.drv' failed
--8<---------------cut here---------------end--------------->8---

Any idea what's going on here?  In case it's helpful, I've also attached
the entire build log.

      Mark



[-- Attachment #2: Build log --]
[-- Type: application/octet-stream, Size: 37484 bytes --]

Re: GNUnet build failure on mips64el

[Andreas Enge]

Hello,

there are also (apparently unrelated) test failures on x86_64:
   http://hydra.gnu.org/build/39813

...
FAIL: test_namestore_api_lookup_private
...
Feb 20 08:58:58-161120 test-namestore-api-28190 ERROR Test to too long to store records, cannot run test!
FAIL: test_namestore_api_lookup_shadow_filter
...
2 of 19 tests failed


And there are some suspicious warnings in the middle of the compilation:
  CC     gnunet-gns-proxy.o
In file included from /nix/store/kczsfj4890k5xfqlimn2l37slfqyyjk9-gnurl-7.35.0/include/curl/curl.h:2283:0,
                 from gnunet-gns-proxy.c:32:
In function 'check_ssl_certificate',
    inlined from 'curl_check_hdr' at gnunet-gns-proxy.c:993:39:
/nix/store/kczsfj4890k5xfqlimn2l37slfqyyjk9-gnurl-7.35.0/include/curl/typecheck-gcc.h:126:42: warning: call to '_curl_easy_getinfo_err_curl_slist' declared with attribute warning: curl_easy_getinfo expects a pointer to struct curl_slist * for this info [enabled by default]
         _curl_easy_getinfo_err_curl_slist();                                  \
                                          ^
gnunet-gns-proxy.c:826:7: note: in expansion of macro 'curl_easy_getinfo'
       curl_easy_getinfo (s5r->curl,
       ^
In file included from ../../src/include/gnunet_crypto_lib.h:56:0,
                 from ../../src/include/gnunet_util_lib.h:39,
                 from gnunet-gns-proxy.c:41:
gnunet-gns-proxy.c: In function 'curl_task_download':
/nix/store/kczsfj4890k5xfqlimn2l37slfqyyjk9-gnurl-7.35.0/include/curl/typecheck-gcc.h:117:38: warning: call to '_curl_easy_getinfo_err_string' declared with attribute warning: curl_easy_getinfo expects a pointer to char * for this info [enabled by default]
         _curl_easy_getinfo_err_string();                                      \
                                      ^
../../src/include/gnunet_common.h:581:41: note: in definition of macro 'GNUNET_break'
 #define GNUNET_break(cond)  do { if (! (cond)) { GNUNET_log(GNUNET_ERROR_TYPE_ERROR, _("Assertion failed at %s:%d.\n"), __FILE__, __LINE__); } } while(0)
                                         ^
gnunet-gns-proxy.c:1338:7: note: in expansion of macro 'curl_easy_getinfo'
       curl_easy_getinfo (msg->easy_handle,
       ^
  CCLD   gnunet-gns-proxy
make[4]: Leaving directory '/tmp/nix-build-gnunet-0.10.0.drv-0/gnunet-0.10.0/src/gns'

Andreas

Re: GNUnet build failure on mips64el

[Sree Harsha Totakura]

On 02/21/2014 07:22 AM, Mark H Weaver wrote:
> Any idea what's going on here?  In case it's helpful, I've also attached
> the entire build log.

Can you go into the build directory
'/tmp/nix-build-gnunet-0.10.0.drv-1/gnunet-0.10.0/src/revocation' and
run 'make check' and see if you get the same output?

Sree

Re: GNUnet build failure on mips64el

[Sree Harsha Totakura]

On 02/21/2014 12:27 PM, Andreas Enge wrote:
> there are also (apparently unrelated) test failures on x86_64:
>    http://hydra.gnu.org/build/39813

We were discussing this yesterday on the IRC.  It looks like the tests
are timing out due to hydra being slow.  I tested this on my machine and
2 other X86_64s and they did well.

> And there are some suspicious warnings in the middle of the compilation:
>   CC     gnunet-gns-proxy.o

These warning are harmless ATM.  Anyway, I shall file a bug report if
they still exist upstream.

Sree

Re: GNUnet build failure on mips64el

[Ludovic Courtès]

Sree Harsha Totakura <sreeharsha@totakura.in> skribis:

> On 02/21/2014 12:27 PM, Andreas Enge wrote:
>> there are also (apparently unrelated) test failures on x86_64:
>>    http://hydra.gnu.org/build/39813
>
> We were discussing this yesterday on the IRC.  It looks like the tests
> are timing out due to hydra being slow.  I tested this on my machine and
> 2 other X86_64s and they did well.

Indeed, it built fine on my x86_64 machine.

On the GNUnet side, it would be great if the tests were less timing sensitive.

Ludo’.

Re: GNUnet build failure on mips64el

[Andreas Enge]

On Fri, Feb 21, 2014 at 04:31:17PM +0100, Ludovic Courtès wrote:
> > We were discussing this yesterday on the IRC.  It looks like the tests
> > are timing out due to hydra being slow.  I tested this on my machine and
> > 2 other X86_64s and they did well.
> Indeed, it built fine on my x86_64 machine.

The same holds for Qt - its build times out after 2 hours, when it is simply
not finished. Could we raise the timeout on hydra?

Another thing: We have "four cores" (I suppose actually two hyperthreaded
ones) on hydra. On my machine, "top" shows four times 100%. On hydra, I get
the impression that only one core is working. Is this normal (due to outside
load, maybe), or do we need to tweak something?

Andreas

Re: GNUnet build failure on mips64el

[Ludovic Courtès]

Andreas Enge <andreas@enge.fr> skribis:

> On Fri, Feb 21, 2014 at 04:31:17PM +0100, Ludovic Courtès wrote:
>> > We were discussing this yesterday on the IRC.  It looks like the tests
>> > are timing out due to hydra being slow.  I tested this on my machine and
>> > 2 other X86_64s and they did well.
>> Indeed, it built fine on my x86_64 machine.
>
> The same holds for Qt - its build times out after 2 hours, when it is simply
> not finished. Could we raise the timeout on hydra?

IIUC the code, there’s not absolute timeout by default, only a
timeout-on-silence.  Which one do we hit here?

> Another thing: We have "four cores" (I suppose actually two hyperthreaded
> ones) on hydra. On my machine, "top" shows four times 100%. On hydra, I get
> the impression that only one core is working. Is this normal (due to outside
> load, maybe), or do we need to tweak something?

It’s supposed to use all the cores.  What makes you think only one core
is used?

Thanks,
Ludo’.

Re: GNUnet build failure on mips64el

[Andreas Enge]

On Fri, Feb 21, 2014 at 06:49:43PM +0100, Ludovic Courtès wrote:
> > The same holds for Qt - its build times out after 2 hours, when it is simply
> > not finished. Could we raise the timeout on hydra?
> IIUC the code, there’s not absolute timeout by default, only a
> timeout-on-silence.  Which one do we hit here?

Things seem to stop after exactly two hours, and the last line of the log
looks as if this happens in the middle of compilation.

> > Another thing: We have "four cores" (I suppose actually two hyperthreaded
> > ones) on hydra. On my machine, "top" shows four times 100%. On hydra, I get
> > the impression that only one core is working. Is this normal (due to outside
> > load, maybe), or do we need to tweak something?
> It’s supposed to use all the cores.  What makes you think only one core
> is used?

"top" does not reach the 4 times 100%, but is usually rather below 100%.

Andreas

Re: GNUnet build failure on mips64el

[Mark H Weaver]

Sree Harsha Totakura <sreeharsha@totakura.in> writes:

> On 02/21/2014 07:22 AM, Mark H Weaver wrote:
>> Any idea what's going on here?  In case it's helpful, I've also attached
>> the entire build log.
>
> Can you go into the build directory
> '/tmp/nix-build-gnunet-0.10.0.drv-1/gnunet-0.10.0/src/revocation' and
> run 'make check' and see if you get the same output?

That failed immediately.  The reason is that apparently GNUnet's test
suite only works if GNUnet has already been installed.  In this case,
since the build failed, the guix-daemon deleted the /nix/store/*
directory where it had been installed :-(

I think it's going to hard to reproduce the build environment outside of
guix-daemon.  I think it would be better to try a modified recipe that
only runs a subset of the tests.

      Mark

Re: GNUnet build failure on mips64el

[Sree Harsha Totakura]

On 02/21/2014 10:46 PM, Mark H Weaver wrote:
> That failed immediately.  The reason is that apparently GNUnet's test
> suite only works if GNUnet has already been installed.  In this case,
> since the build failed, the guix-daemon deleted the /nix/store/*
> directory where it had been installed :-(

True.  I forgot about that.

I suspect it is the timing issue here.  Can you perhaps compile gnunet
manually with its dependencies from Guix and then run the tests in
revocation?

Sree

Re: GNUnet build failure on mips64el

[Sree Harsha Totakura]

On 02/21/2014 04:31 PM, Ludovic Courtès wrote:
> On the GNUnet side, it would be great if the tests were less timing sensitive.

We faced these problems earlier when we added Sheevaplug and RasberryPi
into our buildbots.  Unfortunately, since we are having a network
application, the timeouts are necessary.

We can try increasing the timeouts but that makes the test runs long if
a testcase were to fail genuinely.  Guess we'll have to come up with
some automated measurement for timeouts depending on the host
configuration.  I just don't know how; any ideas?

Sree

Re: GNUnet build failure on mips64el

[Mark H Weaver]

Sree Harsha Totakura <sreeharsha@totakura.in> writes:

> On 02/21/2014 10:46 PM, Mark H Weaver wrote:
>> That failed immediately.  The reason is that apparently GNUnet's test
>> suite only works if GNUnet has already been installed.  In this case,
>> since the build failed, the guix-daemon deleted the /nix/store/*
>> directory where it had been installed :-(
>
> True.  I forgot about that.
>
> I suspect it is the timing issue here.  Can you perhaps compile gnunet
> manually with its dependencies from Guix and then run the tests in
> revocation?

I tried this, and it fails with similar output:

--8<---------------cut here---------------start------------->8---
make  check-TESTS
make[1]: Entering directory '/home/mhw/build/gnunet-0.10.0/src/revocation'
Creating an ego revoc_test
Testing key U58NUTO81QEJ51AFHK2EIPQ6QVKNML0A2KNIAJRG9FTDS1483G0G
Key was valid
Revoking key U58NUTO81QEJ51AFHK2EIPQ6QVKNML0A2KNIAJRG9FTDS1483G0G
Revocation certificate not ready, calculating proof of work
.. - @  12% (estimate)
Key for ego `revoc_test' has been successfully revoked
Testing revoked key U58NUTO81QEJ51AFHK2EIPQ6QVKNML0A2KNIAJRG9FTDS1483G0G
Key was revoked
PASS: test_local_revocation.py
Feb 22 07:17:50-660583 testbed-api-topology-16547 WARNING Error while establishing a link: 0x4: Timeout while connecting to CORE of peer with id: 0 -- Retrying
Feb 22 07:17:51-098510 testbed-api-topology-16547 WARNING Error while establishing a link: 0x5: Timeout while connecting to CORE of peer with id: 1 -- Retrying
Feb 22 07:17:56-127624 testbed-api-topology-16547 WARNING Error while establishing a link: 0x7: Timeout while acquiring HELLO of peer 6ULB -- Retrying
Feb 22 07:17:56-134033 testbed-api-topology-16547 WARNING Error while establishing a link: 0x6: Timeout while acquiring HELLO of peer 6DER -- Retrying
Feb 22 07:18:01-142761 testbed-api-topology-16547 WARNING Error while establishing a link: 0x9: Timeout while acquiring HELLO of peer 6DER -- Retrying
Feb 22 07:18:01-143920 testbed-api-topology-16547 WARNING Error while establishing a link: 0x8: Timeout while acquiring HELLO of peer 6ULB -- Retrying
--8<---------------cut here---------------end--------------->8---

After printing the text above, it gets stuck in
'src/revocation/.libs/test_revocation', with the two
'gnunet-service-nse' subprocesses both running continuously and all
other gnunet processes sleeping.

     Mark

Re: GNUnet build failure on mips64el

[Sree Harsha Totakura]

On 02/22/2014 08:44 AM, Mark H Weaver wrote:
> PASS: test_local_revocation.py
> Feb 22 07:17:50-660583 testbed-api-topology-16547 WARNING Error while establishing a link: 0x4: Timeout while connecting to CORE of peer with id: 0 -- Retrying
> Feb 22 07:17:51-098510 testbed-api-topology-16547 WARNING Error while establishing a link: 0x5: Timeout while connecting to CORE of peer with id: 1 -- Retrying
> Feb 22 07:17:56-127624 testbed-api-topology-16547 WARNING Error while establishing a link: 0x7: Timeout while acquiring HELLO of peer 6ULB -- Retrying
> Feb 22 07:17:56-134033 testbed-api-topology-16547 WARNING Error while establishing a link: 0x6: Timeout while acquiring HELLO of peer 6DER -- Retrying
> Feb 22 07:18:01-142761 testbed-api-topology-16547 WARNING Error while establishing a link: 0x9: Timeout while acquiring HELLO of peer 6DER -- Retrying
> Feb 22 07:18:01-143920 testbed-api-topology-16547 WARNING Error while establishing a link: 0x8: Timeout while acquiring HELLO of peer 6ULB -- Retrying
> --8<---------------cut here---------------end--------------->8---

These warnings are likely caused due to the system getting overloaded.
The testcase has a test ramp-up part which starts 2 peers and then
connects them.  This part is failing as peers are unable to connect.
> 
> After printing the text above, it gets stuck in
> 'src/revocation/.libs/test_revocation', with the two
> 'gnunet-service-nse' subprocesses both running continuously and all
> other gnunet processes sleeping.

This is likely to cause the load on the system.  'gnunet-service-nse'
tries to find a proof-of-work to participate in the size estimation
application of GNUnet.  This proof-of-work involves find a value whose
hash matches its first n digits; this is required to avoid Sybil attacks
and can be computationally expensive and can be taxing on the loongson
you have.

I am not sure if the NSE service is needed for this testcase.  If not it
should be easy to fix.

In any case, the testcase should not run for more than ~1 minute.  I'd
like to debug this.  Would it be possible to provide me access to this
machine?

Sree

Re: GNUnet build failure on mips64el

[Christian Grothoff]

[-- Attachment #1: Type: text/plain, Size: 622 bytes --]

On 02/22/2014 09:46 AM, Sree Harsha Totakura wrote:
> This is likely to cause the load on the system.  'gnunet-service-nse'
> tries to find a proof-of-work to participate in the size estimation
> application of GNUnet.  This proof-of-work involves find a value whose
> hash matches its first n digits; this is required to avoid Sybil attacks
> and can be computationally expensive and can be taxing on the loongson
> you have.
> 
> I am not sure if the NSE service is needed for this testcase.  If not it
> should be easy to fix.

I've disabled the POW calculation in SVN 32456, so the CPU load issue
should be fixed now.

[-- Attachment #2: 0x48426C7E.asc --]
[-- Type: application/pgp-keys, Size: 7098 bytes --]

Re: GNUnet build failure on mips64el

[Andreas Enge]

On Fri, Feb 21, 2014 at 07:20:07PM +0100, Andreas Enge wrote:
> On Fri, Feb 21, 2014 at 06:49:43PM +0100, Ludovic Courtès wrote:
> > IIUC the code, there’s not absolute timeout by default, only a
> > timeout-on-silence.  Which one do we hit here?
> Things seem to stop after exactly two hours, and the last line of the log
> looks as if this happens in the middle of compilation.

Yet another one:
  http://hydra.gnu.org/build/39518

Timed out after: 2h 0m 0s

Last lines of output:
c++ -o nsGeolocation.o -c -I../../../dist/stl_wrappers -I../../../dist/system_wrappers -include ../../../config/gcc_hidden.h -DMOZ_GLUE_IN_PROGRAM -DMOZILLA_INTERNAL_API -D_IMPL_NS_COM -DEXPORT_XPT_API -DEXPORT_XPTC_API -D_IMPL_NS_GFX -D_IMPL_NS_WIDGET -DIMPL_XREAPI -DIMPL_NS_NET -DIMPL_THEBES  -DSTATIC_EXPORTABLE_JS_API -DNO_NSPR_10_SUPPORT -DOS_POSIX=1 -DOS_LINUX=1  -D_IMPL_NS_LAYOUT -I../../../dom/base -I../../../dom/ipc -I../../../content/base/src -I../../../content/events/src  -I../../../ipc/chromium/src -I../../../ipc/glue -I../../../ipc/ipdl/_ipdlheaders  -I../../../dom/src/geolocation -I. -I../../../dist/include  -I/tmp/nix-build-icecat-24.0.drv-0/icecat-24.0/dist/include/nspr -I/tmp/nix-build-icecat-24.0.drv-0/icecat-24.0/dist/include/nss      -fPIC  -Wall -Wpointer-arith -Woverloaded-virtual -Werror=return-type -Wtype-limits -Wempty-body -Wsign-compare -Wno-invalid-offsetof -Wcast-align -fno-exceptions -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -std=gnu++0x -pthread -pipe  -DNDEBUG -DTRIMMED -pipe -O3 -fomit-frame-pointer    -DMOZILLA_CLIENT -include ../../../mozilla-config.h -MD -MP -MF .deps/nsGeolocation.o.pp  /tmp/nix-build-icecat-24.0.drv-0/icecat-24.0/dom/src/geolocation/nsGeolocation.cpp


I am quite convinced there is a forced timeout after 2 hours of compilation.

Andreas

Build timeout on Hydra

[Ludovic Courtès]

Andreas Enge <andreas@enge.fr> skribis:

> On Fri, Feb 21, 2014 at 07:20:07PM +0100, Andreas Enge wrote:
>> On Fri, Feb 21, 2014 at 06:49:43PM +0100, Ludovic Courtès wrote:
>> > IIUC the code, there’s not absolute timeout by default, only a
>> > timeout-on-silence.  Which one do we hit here?
>> Things seem to stop after exactly two hours, and the last line of the log
>> looks as if this happens in the middle of compilation.
>
> Yet another one:
>   http://hydra.gnu.org/build/39518
>
> Timed out after: 2h 0m 0s

Right and the page above reads “Timed out” (not prominently enough, though.)

[...]

> I am quite convinced there is a forced timeout after 2 hours of compilation.

Indeed.  I investigated and found out that ‘hydra-eval-guile-jobs’, the
Guile program used by Hydra to evaluate jobs written in Guile (instead
of Nix), would set a default timeout of 2h...

I’m not sure exactly why we never hit that problem before.

Anyway, I fixed it in Hydra, and added a workaround in our Hydra
recipes:

  https://github.com/NixOS/hydra/commit/61448ca2bd078683e45184f1c27fe0979f7438a8
  http://git.savannah.gnu.org/cgit/guix.git/commit/?id=65f7c35d02175806f676b8e130236dd3e6c8ec60

I’ll merge the latter to core-updates in a moment.

So thanks for the heads-up!

Ludo’.