Re: shortening the git test suite

[Chris Marusich <cmmarusich@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 4684 bytes --]

Hi Ricardo, Mark, and Ludo,

Ricardo Wurmus <rekado@elephly.net> writes:

> What do you think of pushing some package updates only to feature
> branches that follow a certain naming convention (e.g. “_update-foo” for
> updating the “foo” package), which causes Cuirass to build them and
> merge the branch into “master” (semi-)automatically when the build is
> successful?

If the goal is to ensure that the substitutes for a newly pulled Guix
are available when "guix pull" runs, then wouldn't channels accomplish
that goal?  I realize we don't have channels yet, though.

I think your proposal would be a reasonable improvement.  In the case of
a branch like "_update-foo", Cuirass would probably need to build not
only the package "foo" but also any packages that depend on it, right?
Since we're talking about updates to the master branch, this shouldn't
cause too many rebuilds, so it shouldn't add much load to the build
farm.

If we're going to allow Cuirass to merge an update branch into master
automatically, it might also make sense to allow Cuirass to
automatically rebase the update branch onto master before merging it (to
ensure the merge is a fast-forward merge), and delete the update branch
after merging it (to prevent clutter in commands like "git branch -av").
Alternatively, we could leave those steps to a human and just have
Cuirass send a summary email or make the results available via its web
interface.  I think that with the right automated tests to protect
against silly bugs (like accidentally deleting the master branch), it
would probably be safer to have Cuirass automatically merge and clean up
after itself, rather than rely on humans to do it.  Humans are prone to
make mistakes, especially when performing tedious tasks.

Mark H Weaver <mhw@netris.org> writes:

>> Personally, I think that the SVN tests are non-essential (after all,
>> we’re building Git here, we keep running the individual test suites of
>> Git and Subversion, and git-svn interop seems like a thing that only
>> upstream need to worry about), which is why I made this proposal.
>
> Why do you say that only upstream needs to worry about it?  I would
> think that you could say the same thing about almost any test suite, but
> there's always the possibility that our particular combination of input
> versions, or the unusual aspects of Guix, might break something that
> upstream doesn't know about.
>
> I would think that git-svn interop is something that any user of the
> git/svn integration needs to worry about.

I agree: just because I don't use a feature doesn't mean that we
shouldn't test that feature if we can.  We ought to be able to run all
the tests.  I think Ricardo's proposal would help increase the
availability of substitutes, which would make long-running tests more
tolerable.

ludo@gnu.org (Ludovic Courtès) writes:

> Mark H Weaver <mhw@netris.org> skribis:
>
>> ludo@gnu.org (Ludovic Courtès) writes:
>
> [...]
>
>>> Currently if test fails, the whole derivation fails, and you can’t
>>> install your package.  If tests were run separately, this would no
>>> longer hold: you could get your package regardless of whether tests
>>> fail.
>>
>> Indeed, and I agree that we would need to address this.
>>
>>> How would you address this?  I guess that calls for a new build
>>> model, no?
>>
>> I'm not sure what you mean by "build model" [...]
>
> [...]
>
> IOW, this looks like a radical change with lots of potential pitfalls,
> one I’d rather not make before 1.0.  I agree that long test suites can
> be a problem, but for now I think we should focus on package-specific
> solutions, to the extent possible, such as what we discussed for Git, as
> well as improvements to our build farm infrastructure.
>
> Thoughts?

I agree that changes like those seem especially tricky to get right.

While I'm here, I'll suggest another far-fetched idea.  Perhaps we could
rethink the relationship between packages, phases (of build systems),
and derivations.  Currently, I believe one package corresponds to one
derivation, and the build phases are just procedures executed during
that derivation.  Perhaps we could change this so that a package
corresponds to multiple derivations: one derivation per phase.  If that
were possible, then a package that runs all phases plus the test phase
would be able to easily re-use the results of a package that runs the
same phases but not the test phase.  However, I'm not sure if this
really makes sense, and in any case, the details would probably be
difficult to design and implement.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]