bug#35521: Mariadb test suite failures on x86_64-linux

[Marius Bakke <mbakke@fastmail.com>]

[-- Attachment #1: Type: text/plain, Size: 1917 bytes --]

Mark H Weaver <mhw@netris.org> writes:

> Hi,
>
> Marius Bakke <mbakke@fastmail.com> writes:
>
>> Chris Marusich <cmmarusich@gmail.com> writes:
>>
>>> Hi,
>>>
>>> I've been encountering this failure off and on for a few weeks now, and
>>> I'd like to help fix it.  In short, it seems like non-deterministic test
>>> failures, to me.  I think we should gather data and report the issue
>>> upstream, and maybe disable the offending tests in the meantime.
>>
>> I agree.  I notice many of these failing tests are for the TokuDB
>> backend, which I doubt anyone is using in Guix anyway.
>>
>> Here is a patch that disables all tests mentioned in this report.  I
>> would like to push it to core-updates.  Are there others?
>
> I'm concerned by how frequently and casually we simply disable failing
> tests.  What is the utility of running test suites at all, if this is
> how we respond?

I had no idea this issue was so widespread until I noticed Berlins
builders hit it more often than not.  I have not been able to reproduce
these failures on my machines.  So it was kind of a panic reaction,
being the person responsible for running these tests and all.

Looking further into the changes between 10.1.37 and 10.1.38, I notice
the 'tokudb.*' tests were enabled:

https://github.com/MariaDB/server/commit/4c490d6df63695dc97b2c808e59954e6877d3a51

Watching the build on Berlin in real time, I also see that the test
output grind nearly to a halt while running those.
'tokudb.hotindex-insert-2' took 2700439 milliseconds, or 45 minutes, if
I'm reading the test output correctly.

The default test case timeout is 40 minutes (as specified in the Guix
package), but I'm using 80 for this build (60 was insufficient).

I suspect the problem is that the 'tokudb.*' tests put a lot of strain
on the file system, which causes these other tests to fail.  It's
interesting that disabling parallel build was insufficient though.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]