Test failure in Ubuntu 22.04 and 22.10 (new test)

Test failure in Ubuntu 22.04 and 22.10 (new test)

[Gianfranco Costamagna]

Hello, the test is now failing in Ubuntu 22.04 (last LTS), when run with parallel > 1

(version 0.37)

e.g. here you can see a build fail

/<<PKGBUILDDIR>>/test/test-lib.sh: line 904: ./test19: No such file or directory
make[2]: *** read jobs pipe: Bad file descriptor.  Stop.
make[2]: *** Waiting for unfinished jobs....
lto-wrapper: fatal error: make returned 2 exit status


https://launchpadlibrarian.net/627359226/buildlog_ubuntu-kinetic-amd64.notmuch_0.37-1ubuntu3_BUILDING.txt.gz

Running dh_auto_test with --no-parallel flag works as workaround, and using last gmame from Debian doesn't fix the issue
(you can see in the above build log the version that is used Get:2 http://ppa.launchpadcontent.net/costamagnagianfranco/locutusofborg-ppa/ubuntu kinetic/main amd64 libgmime-3.0-0 amd64 3.2.13+dfsg-2 [177 kB] )


Looks like all the tests using test_private_C function are failing, but I can't figure out where its the race-condition.

Thanks

Gianfranco\r

Re: Test failure in Ubuntu 22.04 and 22.10 (new test)

[David Bremner]

Gianfranco Costamagna <locutusofborg@debian.org> writes:

> Hello, the test is now failing in Ubuntu 22.04 (last LTS), when run with parallel > 1
>
> (version 0.37)
>
> e.g. here you can see a build fail
>
> /<<PKGBUILDDIR>>/test/test-lib.sh: line 904: ./test19: No such file or directory
> make[2]: *** read jobs pipe: Bad file descriptor.  Stop.
> make[2]: *** Waiting for unfinished jobs....
> lto-wrapper: fatal error: make returned 2 exit status
>
>
> https://launchpadlibrarian.net/627359226/buildlog_ubuntu-kinetic-amd64.notmuch_0.37-1ubuntu3_BUILDING.txt.gz
>
> Running dh_auto_test with --no-parallel flag works as workaround, and using last gmame from Debian doesn't fix the issue
> (you can see in the above build log the version that is used Get:2 http://ppa.launchpadcontent.net/costamagnagianfranco/locutusofborg-ppa/ubuntu kinetic/main amd64 libgmime-3.0-0 amd64 3.2.13+dfsg-2 [177 kB] )
>
>
> Looks like all the tests using test_private_C function are failing, but I can't figure out where its the race-condition.

It's especially puzzling as make -j$(whatever) should not enable running
the test suite in parallel, only the presence of (GNU|moreutils)
parallel. So it's like something is cleaning up after the test suite too
early.\r

Re: Test failure in Ubuntu 22.04 and 22.10 (new test)

[David Bremner]

David Bremner <david@tethera.net> writes:

> Gianfranco Costamagna <locutusofborg@debian.org> writes:
>
>> Hello, the test is now failing in Ubuntu 22.04 (last LTS), when run with parallel > 1
>>
>> (version 0.37)
>>
>> e.g. here you can see a build fail
>>
>> /<<PKGBUILDDIR>>/test/test-lib.sh: line 904: ./test19: No such file or directory
>> make[2]: *** read jobs pipe: Bad file descriptor.  Stop.
>> make[2]: *** Waiting for unfinished jobs....
>> lto-wrapper: fatal error: make returned 2 exit status
>>
>>
>> https://launchpadlibrarian.net/627359226/buildlog_ubuntu-kinetic-amd64.notmuch_0.37-1ubuntu3_BUILDING.txt.gz
>>
>> Running dh_auto_test with --no-parallel flag works as workaround, and using last gmame from Debian doesn't fix the issue
>> (you can see in the above build log the version that is used Get:2 http://ppa.launchpadcontent.net/costamagnagianfranco/locutusofborg-ppa/ubuntu kinetic/main amd64 libgmime-3.0-0 amd64 3.2.13+dfsg-2 [177 kB] )
>>
>>
>> Looks like all the tests using test_private_C function are failing, but I can't figure out where its the race-condition.
>
> It's especially puzzling as make -j$(whatever) should not enable running
> the test suite in parallel, only the presence of (GNU|moreutils)
> parallel. So it's like something is cleaning up after the test suite too
> early.

A second look at the log I see lto-wrapper failing, presumably in trying
to the test binaries.  I can duplicate the problem by adding -flto to
CFLAGS and CXXFLAGS. It _looks_ like lto-wrapper is invoking make
internally? 

Anyway, if you have an lto expert (or you are one), it would be
interesting to get feedback on what those error messages mean, and why
they are only triggered under make -j.


d\r

Re: Test failure in Ubuntu 22.04 and 22.10 (new test)

[David Bremner]

Michael J Gruber <michaeljgruber+grubix+git@gmail.com> writes:

>
> Yes, lto-wrapper calls make.
>
> Are we compiling test functions on the fly during the test? In that
> case we need to make sure that each test depends on the build
> products, or else the test helper compilation and its users might run
> in parallel ...

Yes, we compile C code on the fly during the run of the tests. I'm not
really clear on what race condition you are anticipating, as neither the
compilation nor the other parts of the test are directly run by make.
Execution is sequential within each T*.sh file. Unless gcc is returning
before it has finished compilation (which I think we'd all agree would
be gcc bug), I don't see how a race can arise there. One thing I can
imagine happening is gcc's recursive invocation of make somehow fails
under make -j, possibly something to do with violated assumptions about
the jobserver and/or environment variables.

d

Re: Test failure in Ubuntu 22.04 and 22.10 (new test)

[Michael J Gruber]

Am Do., 6. Okt. 2022 um 18:34 Uhr schrieb David Bremner <david@tethera.net>:
>
> Michael J Gruber <michaeljgruber+grubix+git@gmail.com> writes:
>
> >
> > Yes, lto-wrapper calls make.
> >
> > Are we compiling test functions on the fly during the test? In that
> > case we need to make sure that each test depends on the build
> > products, or else the test helper compilation and its users might run
> > in parallel ...
>
> Yes, we compile C code on the fly during the run of the tests. I'm not
> really clear on what race condition you are anticipating, as neither the
> compilation nor the other parts of the test are directly run by make.
> Execution is sequential within each T*.sh file. Unless gcc is returning
> before it has finished compilation (which I think we'd all agree would
> be gcc bug), I don't see how a race can arise there. One thing I can
> imagine happening is gcc's recursive invocation of make somehow fails
> under make -j, possibly something to do with violated assumptions about
> the jobserver and/or environment variables.

What I mean is:
make calls T*.sh
T*.sh calls gcc
gcc calls make (for lto)

Could it be that within a parallel make session, that gcc-make-call
gets delegated to the master make jobserver and thus gcc returns too
early? Wild speculation, I admit.
I haven't checked the code, but having those testhelpers as
prerequisites of the test scripts may help in that case.

Michael

Re: Test failure in Ubuntu 22.04 and 22.10 (new test)

[David Bremner]

Michael J Gruber <michaeljgruber+grubix+git@gmail.com> writes:

> Could it be that within a parallel make session, that gcc-make-call
> gets delegated to the master make jobserver and thus gcc returns too
> early? Wild speculation, I admit.

That's the kind of thing I would consider a bug in gcc.

> I haven't checked the code, but having those testhelpers as
> prerequisites of the test scripts may help in that case.

The test scripts are not make targets, so I don't really know how we
would specify prerequisites.

Re: Test failure in Ubuntu 22.04 and 22.10 (new test)

[Tomi Ollila]

On Thu, Oct 06 2022, Michael J. Gruber wrote:

> Am Do., 6. Okt. 2022 um 18:34 Uhr schrieb David Bremner <david@tethera.net>:
>>
>> Michael J Gruber <michaeljgruber+grubix+git@gmail.com> writes:
>>
>> >
>> > Yes, lto-wrapper calls make.
>> >
>> > Are we compiling test functions on the fly during the test? In that
>> > case we need to make sure that each test depends on the build
>> > products, or else the test helper compilation and its users might run
>> > in parallel ...
>>
>> Yes, we compile C code on the fly during the run of the tests. I'm not
>> really clear on what race condition you are anticipating, as neither the
>> compilation nor the other parts of the test are directly run by make.
>> Execution is sequential within each T*.sh file. Unless gcc is returning
>> before it has finished compilation (which I think we'd all agree would
>> be gcc bug), I don't see how a race can arise there. One thing I can
>> imagine happening is gcc's recursive invocation of make somehow fails
>> under make -j, possibly something to do with violated assumptions about
>> the jobserver and/or environment variables.
>
> What I mean is:
> make calls T*.sh
> T*.sh calls gcc
> gcc calls make (for lto)
>
> Could it be that within a parallel make session, that gcc-make-call
> gets delegated to the master make jobserver and thus gcc returns too
> early? Wild speculation, I admit.

Like David said, that would be bug in gcc... (nasty one I'd admit, how
can one expect that the world around has set make to run its jobs
parallely (if that is the case))

anyway, one could try unset MAKEFLAGS in ... test-lib.sh and see
if that helps (perhaps also MFLAGS))

$ printf %s\\n all: $'\tenv' | make -j -f /dev/stdin | sort | grep FLA
MAKEFLAGS= -j4 --jobserver-auth=4,5
MFLAGS=-j4 --jobserver-auth=4,5

> I haven't checked the code, but having those testhelpers as
> prerequisites of the test scripts may help in that case.
>
> Michael

Tomi