Running the full MariaDB test suite

Running the full MariaDB test suite

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 1182 bytes --]

Hello!

So far our MariaDB package has only run some very lightweight tests.
10.1.34 flipped a switch that runs the entire "MTR" test suite.

After patching /bin/ls and /bin/sh, and fixing a couple of failures that
showed up after about 4 hours each, the most recent attempt took eight
hours during which my SSD was completely thrashed.  And then it failed a
test case related to the "disks" plugin.

The switch was flipped back shortly after the release:

https://github.com/MariaDB/server/commit/0a9d78f51d74be7708f2efd940311bf7b33108e9

Since 10.1.35 won't run these tests anymore, we could skip them for this
release and don't care about it.

Alternatively, we can replace the 'check' phase with something that
invokes "mtr" with sensible arguments, and passing our own list of tests
to skip.  That is what Debian does:

https://salsa.debian.org/mariadb-team/mariadb-10.1/blob/stretch/debian/rules#L96

Thoughts?

I'm leaning towards the latter approach, since I've already spent
considerable time tracking down related failures, and it already
identified a potential problem in the "disks" plugin.  But it means
building MariaDB will take many hours even on powerful machines.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Running the full MariaDB test suite

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 1549 bytes --]

On Tue, Jul 17, 2018 at 10:04:42PM +0200, Marius Bakke wrote:
> Hello!
> 
> So far our MariaDB package has only run some very lightweight tests.
> 10.1.34 flipped a switch that runs the entire "MTR" test suite.
> 
> After patching /bin/ls and /bin/sh, and fixing a couple of failures that
> showed up after about 4 hours each, the most recent attempt took eight
> hours during which my SSD was completely thrashed.  And then it failed a
> test case related to the "disks" plugin.
> 
> The switch was flipped back shortly after the release:
> 
> https://github.com/MariaDB/server/commit/0a9d78f51d74be7708f2efd940311bf7b33108e9
> 
> Since 10.1.35 won't run these tests anymore, we could skip them for this
> release and don't care about it.
> 
> Alternatively, we can replace the 'check' phase with something that
> invokes "mtr" with sensible arguments, and passing our own list of tests
> to skip.  That is what Debian does:
> 
> https://salsa.debian.org/mariadb-team/mariadb-10.1/blob/stretch/debian/rules#L96
> 
> Thoughts?
> 
> I'm leaning towards the latter approach, since I've already spent
> considerable time tracking down related failures, and it already
> identified a potential problem in the "disks" plugin.  But it means
> building MariaDB will take many hours even on powerful machines.

No strong opinion but the latter approach sounds good to me. There are
already some packages with very long timeouts for the armhf (and
formerly MIPS) build machines, so there is precedent for very long
builds.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Running the full MariaDB test suite

[Ludovic Courtès]

Leo Famulari <leo@famulari.name> skribis:

> On Tue, Jul 17, 2018 at 10:04:42PM +0200, Marius Bakke wrote:
>> Hello!
>> 
>> So far our MariaDB package has only run some very lightweight tests.
>> 10.1.34 flipped a switch that runs the entire "MTR" test suite.
>> 
>> After patching /bin/ls and /bin/sh, and fixing a couple of failures that
>> showed up after about 4 hours each, the most recent attempt took eight
>> hours during which my SSD was completely thrashed.  And then it failed a
>> test case related to the "disks" plugin.
>> 
>> The switch was flipped back shortly after the release:
>> 
>> https://github.com/MariaDB/server/commit/0a9d78f51d74be7708f2efd940311bf7b33108e9
>> 
>> Since 10.1.35 won't run these tests anymore, we could skip them for this
>> release and don't care about it.
>> 
>> Alternatively, we can replace the 'check' phase with something that
>> invokes "mtr" with sensible arguments, and passing our own list of tests
>> to skip.  That is what Debian does:
>> 
>> https://salsa.debian.org/mariadb-team/mariadb-10.1/blob/stretch/debian/rules#L96
>> 
>> Thoughts?
>> 
>> I'm leaning towards the latter approach, since I've already spent
>> considerable time tracking down related failures, and it already
>> identified a potential problem in the "disks" plugin.  But it means
>> building MariaDB will take many hours even on powerful machines.
>
> No strong opinion but the latter approach sounds good to me.

Same here.

Great that you spent energy tracking these issues!

Ludo’.