bug#46780: java-snappy: Test failure on ci.guix.gnu.org

bug#46780: java-snappy: Test failure on ci.guix.gnu.org

[Björn Höfling]

[-- Attachment #1: Type: text/plain, Size: 1870 bytes --]

java-snappy fails as a dependency of several Java evaluations, for
example:

http://ci.guix.gnu.org/eval/120281?status=failed
http://ci.guix.gnu.org/build/359649/log/raw

These failures exist only on ci.guix.gnu.org, but locally it build
perfectly, even with --rounds=100!

Why? Because our buildmachine has too much memory :-) [And I can
remember, we once had that for another package, I just can't remember
again which one].

What should we do about it? Just disable the test, or something more
complicated?

Analysis:

To analyze, I let it once fail with the -K switch to have the directory

/tmp/guix-build-java-snappy-1.1.7.drv-0/

available. Enter the environment with

$ guix environment java-snappy
$ cd /tmp/guix-build-java-snappy-1.1.7.drv-0/
$ source ./environment-variables
$ cd source
$ ant check
--> everything is fine.  

Then increase the memory used by junit: Edit the build.xml file, add a
huge "maxmemory" property like this :

<junit maxmemory="30G" printsummary="true" ...

Either have enough RAM or add some extra swap space, then run again:

$ ant check

It fails. The junit logs can be found in

src/test/test-reports/TEST-org.xerial.snappy.pool.CachingBufferPoolTest.txt

Testsuite: org.xerial.snappy.pool.CachingBufferPoolTest
Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 7.363 sec

Testcase: testDirectByteBuffers took 0.002 sec
Testcase: testSoftReferences took 7.353 sec
        FAILED
count: 2048
junit.framework.AssertionFailedError: count: 2048
        at org.xerial.snappy.pool.CachingBufferPoolTest.testSoftReferences(Unknown
Source)

Testcase: testArrays took 0 sec
Testcase: testAdjustSize took 0 sec

If you look at the test case source code, it tries to allocate about
20GB and hopes that this fails. It just does not fail on huge machines...

Björn

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

bug#46780: java-snappy: Test failure on ci.guix.gnu.org

[zimoun]

Hi,

On Thu, 25 Feb 2021 at 21:52, Björn Höfling <bjoern.hoefling@bjoernhoefling.de> wrote:

> Then increase the memory used by junit: Edit the build.xml file, add a
> huge "maxmemory" property like this :
>
> <junit maxmemory="30G" printsummary="true" ...

[...]

> If you look at the test case source code, it tries to allocate about
> 20GB and hopes that this fails. It just does not fail on huge machines...

What about adding a phase just before the test?  For example in this
build.xml file fix the max memory to 20G if it is greater.


All the best,
simon

bug#46780: java-snappy: Test failure on ci.guix.gnu.org

[Björn Höfling]

[-- Attachment #1: Type: text/plain, Size: 1247 bytes --]

On Fri, 26 Feb 2021 23:08:42 +0100
zimoun <zimon.toutoune@gmail.com> wrote:

> Hi,
> 
> On Thu, 25 Feb 2021 at 21:52, Björn Höfling
> <bjoern.hoefling@bjoernhoefling.de> wrote:
> 
> > Then increase the memory used by junit: Edit the build.xml file,
> > add a huge "maxmemory" property like this :
> >
> > <junit maxmemory="30G" printsummary="true" ...  
> 
> [...]
> 
> > If you look at the test case source code, it tries to allocate about
> > 20GB and hopes that this fails. It just does not fail on huge
> > machines...  
> 
> What about adding a phase just before the test?  For example in this
> build.xml file fix the max memory to 20G if it is greater.

The build.xml is auto-generated by our ant-build-system. So we could
add some parameter into it. But I found this too much of an overkill,
and working on the ant-build-system would mean this goes into staging
(we have too many Java package).

Thus, at first I thought the idea with the additional phase wouldn't
fit here. But at the second glimpse: The build.xml file definitely
already exists before running the tests (even before build). So we
could add a phase that substitutestars and adds the maxmemory parameter
in the build.xml.

Björn




[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

bug#46780: java-snappy: Test failure on ci.guix.gnu.org

[Björn Höfling]

[-- Attachment #1: Type: text/plain, Size: 1025 bytes --]

On Sat, 27 Feb 2021 14:58:47 +0100
Björn Höfling <bjoern.hoefling@bjoernhoefling.de> wrote:

> On Fri, 26 Feb 2021 23:08:42 +0100
> zimoun <zimon.toutoune@gmail.com> wrote:
> 
> > What about adding a phase just before the test?  For example in this
> > build.xml file fix the max memory to 20G if it is greater.  

Thanks for the idea.

I first tested it with maxmemory="1M" and this fails with a OOM-error
on my pc, thus I knew this setting is really honored.

Now I set it to 2G and sent a patch under commit:

23dcf4339d1dc102b2c509a151734f4caff793bd

And it works now on Berlin:

https://ci.guix.gnu.org/build/361691/details

Closing.

Björn

PS: While looking at the Cuirass logs, I remembered the other
memory-problem we head, it was with java-kafka-clients:

https://ci.guix.gnu.org/build/361612/details
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=40554

At first sight, we could try the same trick here. I will investigate
this in the next week, if not someone else is quicker.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

bug#46780: java-snappy: Test failure on ci.guix.gnu.org

[zimoun]

Hi,

On Sun, 28 Feb 2021 at 00:29, Björn Höfling <bjoern.hoefling@bjoernhoefling.de> wrote:

> I first tested it with maxmemory="1M" and this fails with a OOM-error
> on my pc, thus I knew this setting is really honored.
>
> Now I set it to 2G and sent a patch under commit:
>
> 23dcf4339d1dc102b2c509a151734f4caff793bd
>
> And it works now on Berlin:
>
> https://ci.guix.gnu.org/build/361691/details
>
> Closing.

Cool if it worked! :-)


Cheers,
simon