Re: shortening the git test suite

[Mark H Weaver <mhw@netris.org>]

Hi Ricardo,

Ricardo Wurmus <rekado@elephly.net> writes:

> Chris Marusich <cmmarusich@gmail.com> writes:
>
>> Ricardo Wurmus <rekado@elephly.net> writes:
>>
>>> Hi Guix,
>>>
>>> git takes a very long time to build, because it has an extensive test
>>> suite.  Most of the time is spent in running the SVN interoperability
>>> tests, though, which are not really all that interesting for most uses
>>> of git.
>>>
>>> The Makefile says this:
>>>
>>>   # Define NO_SVN_TESTS if you want to skip time-consuming SVN interoperability
>>>   # tests.  These tests take up a significant amount of the total test time
>>>   # but are not needed unless you plan to talk to SVN repos.
>>>
>>> What do you think about disabling the SVN tests in the git package?
>>
>> This sounds similar to the discussion we had earlier about treating
>> tests as a special case:
>>
>> https://lists.gnu.org/archive/html/guix-devel/2018-04/msg00071.html
>>
>> I felt that the conclusion of that thread was basically that if someone
>> is concerned about the build time, then they ought to be able to use
>> substitutes to speed things up, and we should continue to run as many
>> tests as possible in order to discover problems sooner.
>
> What I’m worried about is availability of substitutes.  When we keep
> changing packages on the “master” branch that have very expensive test
> suites then we accept that people won’t have substitutes for a while.
> The duration of that while depends on how quickly the build of this
> package is started by our continuous integration software, and how long
> it takes to complete the build.
>
> Granted, disabling parts of the test suite in an attempt to shorten it
> is really a technical fix to a social problem.
>
> Personally, I think that the SVN tests are non-essential (after all,
> we’re building Git here, we keep running the individual test suites of
> Git and Subversion, and git-svn interop seems like a thing that only
> upstream need to worry about), which is why I made this proposal.

Why do you say that only upstream needs to worry about it?  I would
think that you could say the same thing about almost any test suite, but
there's always the possibility that our particular combination of input
versions, or the unusual aspects of Guix, might break something that
upstream doesn't know about.

I would think that git-svn interop is something that any user of the
git/svn integration needs to worry about.

If we feel that very few of our users care about git-svn interop,
another option would be to add a lighter variant of 'git' that does not
include SVN support.  It would probably be a good idea to have a
'git-minimal' package anyway, for use by our 'git-fetch' origin method.
Naturally, only the 'git' package variant that includes SVN support
would need to run the SVN tests.

Also, looking ahead, I think it would be great if we could eventually
move to a model where the tests of some packages are split off into
separate derivations.  Similarly, we could work toward splitting off
documentation generation to separate derivation for selected packages.
The most important advantage to this approach is that it would allow
inputs needed only for tests or docs to be omitted from the inputs of
the main package.  I expect that this will in many cases be needed to
prevent circular dependencies, and it could also greatly reduce the
amount of rebuilding needed after updating certain packages.

What do you think?

       Mark