Re: shortening the git test suite

[ludo@gnu.org (Ludovic Courtès)]

Hello,

Mark H Weaver <mhw@netris.org> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:

[...]

>> Currently if test fails, the whole derivation fails, and you can’t
>> install your package.  If tests were run separately, this would no
>> longer hold: you could get your package regardless of whether tests
>> fail.
>
> Indeed, and I agree that we would need to address this.
>
>> How would you address this?  I guess that calls for a new build
>> model, no?
>
> I'm not sure what you mean by "build model", whether you are talking
> about the daemon interface or something else, but I think the changes
> could be confined to the Guix user interface.  A field could be added to
> <package>, somewhat similar to 'replacement', but pointing to a package
> object which runs tests, or perhaps a list of package objects.  The guix
> client could simply add the test packages to the list of derivations to
> build.  This could be inhibited via a "--no-tests" guix build option.
>
> In typical cases where "make check" expects to be run from a fully built
> source directory, the main package would typically produce a tarball of
> the built source directory as an additional output.  The test package
> would simply unpack this tarball and run "make check" in it.
>
> Regarding the build order: ideally, we would run the tests for package
> FOO immediately after building FOO, or at least before building anything
> that depends on FOO, to avoid wasted work in case the tests fail.  I'm
> not sure how best to accomplish this.
>
> What do you think?

I’m afraid doing everything at the level of package objects rather than
derivations would lead to something clunky (grafts already suffer from
that.)

Namely, Guix clients would have to explicitly do a second
‘build-derivations’ RPC once they’re done building packages.  If they
don’t do that, then tests are not run and you get to install a possibly
broken package.

Besides, there are lots of details to get right in a proposal like
yours.  In particular, saving the build tree and for later reuse may
prove to be more challenging than it seems.

IOW, this looks like a radical change with lots of potential pitfalls,
one I’d rather not make before 1.0.  I agree that long test suites can
be a problem, but for now I think we should focus on package-specific
solutions, to the extent possible, such as what we discussed for Git, as
well as improvements to our build farm infrastructure.

Thoughts?

Ludo’.