shortening the git test suite

shortening the git test suite

[Ricardo Wurmus]

Hi Guix,

git takes a very long time to build, because it has an extensive test
suite.  Most of the time is spent in running the SVN interoperability
tests, though, which are not really all that interesting for most uses
of git.

The Makefile says this:

  # Define NO_SVN_TESTS if you want to skip time-consuming SVN interoperability
  # tests.  These tests take up a significant amount of the total test time
  # but are not needed unless you plan to talk to SVN repos.

What do you think about disabling the SVN tests in the git package?

--
Ricardo

Re: shortening the git test suite

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 745 bytes --]

Ricardo Wurmus <rekado@elephly.net> ezt írta (időpont: 2018. júl. 1., V,
1:20):

> Hi Guix,
>
> git takes a very long time to build, because it has an extensive test
> suite.  Most of the time is spent in running the SVN interoperability
> tests, though, which are not really all that interesting for most uses
> of git.
>
> The Makefile says this:
>
>   # Define NO_SVN_TESTS if you want to skip time-consuming SVN
> interoperability
>   # tests.  These tests take up a significant amount of the total test time
>   # but are not needed unless you plan to talk to SVN repos.
>
> What do you think about disabling the SVN tests in the git package?
>
>
I think it is ok to disable SVN tests.


> --
> Ricardo
>
>
>

[-- Attachment #2: Type: text/html, Size: 1169 bytes --]

Re: shortening the git test suite

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 962 bytes --]

Ricardo Wurmus <rekado@elephly.net> writes:

> Hi Guix,
>
> git takes a very long time to build, because it has an extensive test
> suite.  Most of the time is spent in running the SVN interoperability
> tests, though, which are not really all that interesting for most uses
> of git.
>
> The Makefile says this:
>
>   # Define NO_SVN_TESTS if you want to skip time-consuming SVN interoperability
>   # tests.  These tests take up a significant amount of the total test time
>   # but are not needed unless you plan to talk to SVN repos.
>
> What do you think about disabling the SVN tests in the git package?

I'm wary of this because it also tests our SVN package, and makes sure
they work together.  But I'm not completely opposed to it either.  How
much time does it save?

When debugging the git build, I usually set #:parallel-tests? #t.  So
far I haven't come across a failure related to that.  I wonder if
<https://bugs.gnu.org/29512> is still relevant.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: shortening the git test suite

[Mark H Weaver]

Ricardo Wurmus <rekado@elephly.net> writes:

> git takes a very long time to build, because it has an extensive test
> suite.  Most of the time is spent in running the SVN interoperability
> tests, though, which are not really all that interesting for most uses
> of git.
>
> The Makefile says this:
>
>   # Define NO_SVN_TESTS if you want to skip time-consuming SVN interoperability
>   # tests.  These tests take up a significant amount of the total test time
>   # but are not needed unless you plan to talk to SVN repos.
>
> What do you think about disabling the SVN tests in the git package?

My preference would be to keep the SVN tests enabled, but it's not
something I feel very strongly about.

      Thanks,
        Mark

Re: shortening the git test suite

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 1003 bytes --]

Mark H Weaver <mhw@netris.org> ezt írta (időpont: 2018. júl. 2., H, 6:58):

> Ricardo Wurmus <rekado@elephly.net> writes:
>
> > git takes a very long time to build, because it has an extensive test
> > suite.  Most of the time is spent in running the SVN interoperability
> > tests, though, which are not really all that interesting for most uses
> > of git.
> >
> > The Makefile says this:
> >
> >   # Define NO_SVN_TESTS if you want to skip time-consuming SVN
> interoperability
> >   # tests.  These tests take up a significant amount of the total test
> time
> >   # but are not needed unless you plan to talk to SVN repos.
> >
> > What do you think about disabling the SVN tests in the git package?
>
> My preference would be to keep the SVN tests enabled, but it's not
> something I feel very strongly about.
>
>       Thanks,
>         Mark
>
>
Do we have a way to make running this SVN test suite optional? For example
to run it only before release, or on demand?

[-- Attachment #2: Type: text/html, Size: 1428 bytes --]

Re: shortening the git test suite

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 1520 bytes --]

Ricardo Wurmus <rekado@elephly.net> writes:

> Hi Guix,
>
> git takes a very long time to build, because it has an extensive test
> suite.  Most of the time is spent in running the SVN interoperability
> tests, though, which are not really all that interesting for most uses
> of git.
>
> The Makefile says this:
>
>   # Define NO_SVN_TESTS if you want to skip time-consuming SVN interoperability
>   # tests.  These tests take up a significant amount of the total test time
>   # but are not needed unless you plan to talk to SVN repos.
>
> What do you think about disabling the SVN tests in the git package?

This sounds similar to the discussion we had earlier about treating
tests as a special case:

https://lists.gnu.org/archive/html/guix-devel/2018-04/msg00071.html

I felt that the conclusion of that thread was basically that if someone
is concerned about the build time, then they ought to be able to use
substitutes to speed things up, and we should continue to run as many
tests as possible in order to discover problems sooner.

I'm in favor of disabling tests if they fail for reasons other than bugs
in the program and there isn't an easy fix (e.g., they require root
privileges, or they fail due to randomness in the tests), but disabling
them simply because they take a "long" time doesn't seem like a great
reason, to me.  I'm sympathetic, since I don't like waiting around,
either, but substitutes should be sufficient for those who need faster
builds.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: shortening the git test suite

[Ricardo Wurmus]

Chris Marusich <cmmarusich@gmail.com> writes:

> Ricardo Wurmus <rekado@elephly.net> writes:
>
>> Hi Guix,
>>
>> git takes a very long time to build, because it has an extensive test
>> suite.  Most of the time is spent in running the SVN interoperability
>> tests, though, which are not really all that interesting for most uses
>> of git.
>>
>> The Makefile says this:
>>
>>   # Define NO_SVN_TESTS if you want to skip time-consuming SVN interoperability
>>   # tests.  These tests take up a significant amount of the total test time
>>   # but are not needed unless you plan to talk to SVN repos.
>>
>> What do you think about disabling the SVN tests in the git package?
>
> This sounds similar to the discussion we had earlier about treating
> tests as a special case:
>
> https://lists.gnu.org/archive/html/guix-devel/2018-04/msg00071.html
>
> I felt that the conclusion of that thread was basically that if someone
> is concerned about the build time, then they ought to be able to use
> substitutes to speed things up, and we should continue to run as many
> tests as possible in order to discover problems sooner.

What I’m worried about is availability of substitutes.  When we keep
changing packages on the “master” branch that have very expensive test
suites then we accept that people won’t have substitutes for a while.
The duration of that while depends on how quickly the build of this
package is started by our continuous integration software, and how long
it takes to complete the build.

Granted, disabling parts of the test suite in an attempt to shorten it
is really a technical fix to a social problem.

Personally, I think that the SVN tests are non-essential (after all,
we’re building Git here, we keep running the individual test suites of
Git and Subversion, and git-svn interop seems like a thing that only
upstream need to worry about), which is why I made this proposal.  But
as it seems that the people who responded to this message rather lean in
the other direction, let’s try to address the social problem instead:

How can we change our workflow to make sure that for packages with long
build/test times we can provide substitutes more quickly?  Currently,
our policy for pushing changes to “staging” and “core-updates” is based
on package counts.  I’m not suggesting that changes to Git be pushed to
“staging”.

What do you think of pushing some package updates only to feature
branches that follow a certain naming convention (e.g. “_update-foo” for
updating the “foo” package), which causes Cuirass to build them and
merge the branch into “master” (semi-)automatically when the build is
successful?

(Obviously, any kind of automation has to be thought out carefully, but
I’m sure we would be able to find a solution.)

What do you think?

--
Ricardo

Re: shortening the git test suite

[Ricardo Wurmus]

Gábor Boskovits <boskovits@gmail.com> writes:

> Mark H Weaver <mhw@netris.org> ezt írta (időpont: 2018. júl. 2., H, 6:58):
>
>> Ricardo Wurmus <rekado@elephly.net> writes:
>>
>> > git takes a very long time to build, because it has an extensive test
>> > suite.  Most of the time is spent in running the SVN interoperability
>> > tests, though, which are not really all that interesting for most uses
>> > of git.
>> >
>> > The Makefile says this:
>> >
>> >   # Define NO_SVN_TESTS if you want to skip time-consuming SVN
>> interoperability
>> >   # tests.  These tests take up a significant amount of the total test
>> time
>> >   # but are not needed unless you plan to talk to SVN repos.
>> >
>> > What do you think about disabling the SVN tests in the git package?
>>
>> My preference would be to keep the SVN tests enabled, but it's not
>> something I feel very strongly about.
>>
>>       Thanks,
>>         Mark
>>
>>
> Do we have a way to make running this SVN test suite optional? For example
> to run it only before release, or on demand?

No, and I would think that a mechanism to execute them optionally would
necessarily be a hack that undermines some of the guarantees that Guix
tries hard to provide.

--
Ricardo

Re: shortening the git test suite

[Ricardo Wurmus]

Marius Bakke <mbakke@fastmail.com> writes:

> When debugging the git build, I usually set #:parallel-tests? #t.  So
> far I haven't come across a failure related to that.  I wonder if
> <https://bugs.gnu.org/29512> is still relevant.

That’s a good question.  I don’t know.

-- 
Ricardo

Re: shortening the git test suite

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 3310 bytes --]

Ricardo Wurmus <rekado@elephly.net> ezt írta (időpont: 2018. júl. 5., Cs
11:22):

>
> Chris Marusich <cmmarusich@gmail.com> writes:
>
> > Ricardo Wurmus <rekado@elephly.net> writes:
> >
> >> Hi Guix,
> >>
> >> git takes a very long time to build, because it has an extensive test
> >> suite.  Most of the time is spent in running the SVN interoperability
> >> tests, though, which are not really all that interesting for most uses
> >> of git.
> >>
> >> The Makefile says this:
> >>
> >>   # Define NO_SVN_TESTS if you want to skip time-consuming SVN
> interoperability
> >>   # tests.  These tests take up a significant amount of the total test
> time
> >>   # but are not needed unless you plan to talk to SVN repos.
> >>
> >> What do you think about disabling the SVN tests in the git package?
> >
> > This sounds similar to the discussion we had earlier about treating
> > tests as a special case:
> >
> > https://lists.gnu.org/archive/html/guix-devel/2018-04/msg00071.html
> >
> > I felt that the conclusion of that thread was basically that if someone
> > is concerned about the build time, then they ought to be able to use
> > substitutes to speed things up, and we should continue to run as many
> > tests as possible in order to discover problems sooner.
>
> What I’m worried about is availability of substitutes.  When we keep
> changing packages on the “master” branch that have very expensive test
> suites then we accept that people won’t have substitutes for a while.
> The duration of that while depends on how quickly the build of this
> package is started by our continuous integration software, and how long
> it takes to complete the build.
>
> Granted, disabling parts of the test suite in an attempt to shorten it
> is really a technical fix to a social problem.
>
> Personally, I think that the SVN tests are non-essential (after all,
> we’re building Git here, we keep running the individual test suites of
> Git and Subversion, and git-svn interop seems like a thing that only
> upstream need to worry about), which is why I made this proposal.  But
> as it seems that the people who responded to this message rather lean in
> the other direction, let’s try to address the social problem instead:
>
> How can we change our workflow to make sure that for packages with long
> build/test times we can provide substitutes more quickly?  Currently,
> our policy for pushing changes to “staging” and “core-updates” is based
> on package counts.  I’m not suggesting that changes to Git be pushed to
> “staging”.
>
> What do you think of pushing some package updates only to feature
> branches that follow a certain naming convention (e.g. “_update-foo” for
> updating the “foo” package), which causes Cuirass to build them and
> merge the branch into “master” (semi-)automatically when the build is
> successful?
>
>
>
> I think this is a very reasonable approach. If this will be the way to go,
then we also need a way to select packages to which this applies, and find
a way to communicate this.

> (Obviously, any kind of automation has to be thought out carefully, but
> I’m sure we would be able to find a solution.)
>
> What do you think?
>
> --
> Ricardo
>
>
>

[-- Attachment #2: Type: text/html, Size: 4304 bytes --]

Re: shortening the git test suite

[Mark H Weaver]

Hi Ricardo,

Ricardo Wurmus <rekado@elephly.net> writes:

> Chris Marusich <cmmarusich@gmail.com> writes:
>
>> Ricardo Wurmus <rekado@elephly.net> writes:
>>
>>> Hi Guix,
>>>
>>> git takes a very long time to build, because it has an extensive test
>>> suite.  Most of the time is spent in running the SVN interoperability
>>> tests, though, which are not really all that interesting for most uses
>>> of git.
>>>
>>> The Makefile says this:
>>>
>>>   # Define NO_SVN_TESTS if you want to skip time-consuming SVN interoperability
>>>   # tests.  These tests take up a significant amount of the total test time
>>>   # but are not needed unless you plan to talk to SVN repos.
>>>
>>> What do you think about disabling the SVN tests in the git package?
>>
>> This sounds similar to the discussion we had earlier about treating
>> tests as a special case:
>>
>> https://lists.gnu.org/archive/html/guix-devel/2018-04/msg00071.html
>>
>> I felt that the conclusion of that thread was basically that if someone
>> is concerned about the build time, then they ought to be able to use
>> substitutes to speed things up, and we should continue to run as many
>> tests as possible in order to discover problems sooner.
>
> What I’m worried about is availability of substitutes.  When we keep
> changing packages on the “master” branch that have very expensive test
> suites then we accept that people won’t have substitutes for a while.
> The duration of that while depends on how quickly the build of this
> package is started by our continuous integration software, and how long
> it takes to complete the build.
>
> Granted, disabling parts of the test suite in an attempt to shorten it
> is really a technical fix to a social problem.
>
> Personally, I think that the SVN tests are non-essential (after all,
> we’re building Git here, we keep running the individual test suites of
> Git and Subversion, and git-svn interop seems like a thing that only
> upstream need to worry about), which is why I made this proposal.

Why do you say that only upstream needs to worry about it?  I would
think that you could say the same thing about almost any test suite, but
there's always the possibility that our particular combination of input
versions, or the unusual aspects of Guix, might break something that
upstream doesn't know about.

I would think that git-svn interop is something that any user of the
git/svn integration needs to worry about.

If we feel that very few of our users care about git-svn interop,
another option would be to add a lighter variant of 'git' that does not
include SVN support.  It would probably be a good idea to have a
'git-minimal' package anyway, for use by our 'git-fetch' origin method.
Naturally, only the 'git' package variant that includes SVN support
would need to run the SVN tests.

Also, looking ahead, I think it would be great if we could eventually
move to a model where the tests of some packages are split off into
separate derivations.  Similarly, we could work toward splitting off
documentation generation to separate derivation for selected packages.
The most important advantage to this approach is that it would allow
inputs needed only for tests or docs to be omitted from the inputs of
the main package.  I expect that this will in many cases be needed to
prevent circular dependencies, and it could also greatly reduce the
amount of rebuilding needed after updating certain packages.

What do you think?

       Mark

Re: shortening the git test suite

[Ludovic Courtès]

Hello,

Mark H Weaver <mhw@netris.org> skribis:

> Why do you say that only upstream needs to worry about it?  I would
> think that you could say the same thing about almost any test suite, but
> there's always the possibility that our particular combination of input
> versions, or the unusual aspects of Guix, might break something that
> upstream doesn't know about.
>
> I would think that git-svn interop is something that any user of the
> git/svn integration needs to worry about.

I agree.

> If we feel that very few of our users care about git-svn interop,
> another option would be to add a lighter variant of 'git' that does not
> include SVN support.  It would probably be a good idea to have a
> 'git-minimal' package anyway, for use by our 'git-fetch' origin method.
> Naturally, only the 'git' package variant that includes SVN support
> would need to run the SVN tests.

Yeah, I think we could to do something like this in this case.

Instead of ‘git-minimal’, we could have ‘git’ without SVN support, and
have a separate ‘git-svn’ package.  We can probably arrange for that
‘git-svn’ package to only provide the relevant libexec/git-core bits
(git-svn is just a Perl script anyway.)

Thoughts?

> Also, looking ahead, I think it would be great if we could eventually
> move to a model where the tests of some packages are split off into
> separate derivations.  Similarly, we could work toward splitting off
> documentation generation to separate derivation for selected packages.
> The most important advantage to this approach is that it would allow
> inputs needed only for tests or docs to be omitted from the inputs of
> the main package.  I expect that this will in many cases be needed to
> prevent circular dependencies, and it could also greatly reduce the
> amount of rebuilding needed after updating certain packages.

Currently if test fails, the whole derivation fails, and you can’t
install your package.  If tests were run separately, this would no
longer hold: you could get your package regardless of whether tests
fail.  How would you address this?  I guess that calls for a new build
model, no?

Thanks,
Ludo’.

Automatic branch merging upon build completion

[Ludovic Courtès]

Hi Ricardo,

Ricardo Wurmus <rekado@elephly.net> skribis:

> What do you think of pushing some package updates only to feature
> branches that follow a certain naming convention (e.g. “_update-foo” for
> updating the “foo” package), which causes Cuirass to build them and
> merge the branch into “master” (semi-)automatically when the build is
> successful?

FWIW I’m all for it, at least for feature updates—security updates
should still go directly to master.  That’s a bit of work, though.

I wonder if we could reuse similar features available in GitLab-style
tools.

Ludo’.

Re: shortening the git test suite

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 2893 bytes --]

Ludovic Courtès <ludo@gnu.org> ezt írta (időpont: 2018. júl. 6., P, 23:05):

> Hello,
>
> Mark H Weaver <mhw@netris.org> skribis:
>
> > Why do you say that only upstream needs to worry about it?  I would
> > think that you could say the same thing about almost any test suite, but
> > there's always the possibility that our particular combination of input
> > versions, or the unusual aspects of Guix, might break something that
> > upstream doesn't know about.
> >
> > I would think that git-svn interop is something that any user of the
> > git/svn integration needs to worry about.
>
> I agree.
>
> > If we feel that very few of our users care about git-svn interop,
> > another option would be to add a lighter variant of 'git' that does not
> > include SVN support.  It would probably be a good idea to have a
> > 'git-minimal' package anyway, for use by our 'git-fetch' origin method.
> > Naturally, only the 'git' package variant that includes SVN support
> > would need to run the SVN tests.
>
> Yeah, I think we could to do something like this in this case.
>
> Instead of ‘git-minimal’, we  could have ‘git’ without SVN support, and
> have a separate ‘git-svn’ package.  We can probably arrange for that
> ‘git-svn’ package to only provide the relevant libexec/git-core bits
> (git-svn is just a Perl script anyway.)
>
> Thoughts?
>
> > Also, looking ahead, I think it would be great if we could eventually
> > move to a model where the tests of some packages are split off into
> > separate derivations.  Similarly, we could work toward splitting off
> > documentation generation to separate derivation for selected packages.
> > The most important advantage to this approach is that it would allow
> > inputs needed only for tests or docs to be omitted from the inputs of
> > the main package.  I expect that this will in many cases be needed to
> > prevent circular dependencies, and it could also greatly reduce the
> > amount of rebuilding needed after updating certain packages.
>
> Currently if test fails, the whole derivation fails, and you can’t
> install your package.  If tests were run separately, this would no
> longer hold: you could get your package regardless of whether tests
> fail.  How would you address this?  I guess that calls for a new build
> model, no?
>
>
One example where this can be seen in action is the dependency of
java-jarjar on java-asm-bootstrap. java-asm-bootstrap is java-asm without
the tests, it is exactly
there to break a dependency cycle. java-asm is a full rebuild using tests.
java-asm-bootstrap is defined, while java-asm is define-public-ed (here a
full rebuild is involved). I'm not sure that a full rebuild can be worked
around in all cases, I guess that sometimes test capabilities are detected
at configure time. WDYT?

Thanks,
> Ludo’.
>
>

[-- Attachment #2: Type: text/html, Size: 3565 bytes --]

Re: shortening the git test suite

[Mark H Weaver]

Hi Ludovic,

ludo@gnu.org (Ludovic Courtès) writes:

> Mark H Weaver <mhw@netris.org> skribis:
>
>> If we feel that very few of our users care about git-svn interop,
>> another option would be to add a lighter variant of 'git' that does not
>> include SVN support.  It would probably be a good idea to have a
>> 'git-minimal' package anyway, for use by our 'git-fetch' origin method.
>> Naturally, only the 'git' package variant that includes SVN support
>> would need to run the SVN tests.
>
> Yeah, I think we could to do something like this in this case.
>
> Instead of ‘git-minimal’, we could have ‘git’ without SVN support, and
> have a separate ‘git-svn’ package.  We can probably arrange for that
> ‘git-svn’ package to only provide the relevant libexec/git-core bits
> (git-svn is just a Perl script anyway.)
>
> Thoughts?

Sure, that sounds good to me.

Whether it makes to have a 'git-minimal' package depends on how many
other optional dependencies could reasonably be removed from it, besides
subversion.  If not much else could be removed, then I suppose there
would be no need for a 'git-minimal' variant.

>> Also, looking ahead, I think it would be great if we could eventually
>> move to a model where the tests of some packages are split off into
>> separate derivations.  Similarly, we could work toward splitting off
>> documentation generation to separate derivation for selected packages.
>> The most important advantage to this approach is that it would allow
>> inputs needed only for tests or docs to be omitted from the inputs of
>> the main package.  I expect that this will in many cases be needed to
>> prevent circular dependencies, and it could also greatly reduce the
>> amount of rebuilding needed after updating certain packages.
>
> Currently if test fails, the whole derivation fails, and you can’t
> install your package.  If tests were run separately, this would no
> longer hold: you could get your package regardless of whether tests
> fail.

Indeed, and I agree that we would need to address this.

> How would you address this?  I guess that calls for a new build
> model, no?

I'm not sure what you mean by "build model", whether you are talking
about the daemon interface or something else, but I think the changes
could be confined to the Guix user interface.  A field could be added to
<package>, somewhat similar to 'replacement', but pointing to a package
object which runs tests, or perhaps a list of package objects.  The guix
client could simply add the test packages to the list of derivations to
build.  This could be inhibited via a "--no-tests" guix build option.

In typical cases where "make check" expects to be run from a fully built
source directory, the main package would typically produce a tarball of
the built source directory as an additional output.  The test package
would simply unpack this tarball and run "make check" in it.

Regarding the build order: ideally, we would run the tests for package
FOO immediately after building FOO, or at least before building anything
that depends on FOO, to avoid wasted work in case the tests fail.  I'm
not sure how best to accomplish this.

What do you think?

      Mark

Re: shortening the git test suite

[Ricardo Wurmus]

Mark H Weaver <mhw@netris.org> writes:

>>> Also, looking ahead, I think it would be great if we could eventually
>>> move to a model where the tests of some packages are split off into
>>> separate derivations.  Similarly, we could work toward splitting off
>>> documentation generation to separate derivation for selected packages.
>>> The most important advantage to this approach is that it would allow
>>> inputs needed only for tests or docs to be omitted from the inputs of
>>> the main package.  I expect that this will in many cases be needed to
>>> prevent circular dependencies, and it could also greatly reduce the
>>> amount of rebuilding needed after updating certain packages.
>>
>> Currently if test fails, the whole derivation fails, and you can’t
>> install your package.  If tests were run separately, this would no
>> longer hold: you could get your package regardless of whether tests
>> fail.
>
> Indeed, and I agree that we would need to address this.
>
>> How would you address this?  I guess that calls for a new build
>> model, no?
>
> I'm not sure what you mean by "build model", whether you are talking
> about the daemon interface or something else, but I think the changes
> could be confined to the Guix user interface.  A field could be added to
> <package>, somewhat similar to 'replacement', but pointing to a package
> object which runs tests, or perhaps a list of package objects.  The guix
> client could simply add the test packages to the list of derivations to
> build.  This could be inhibited via a "--no-tests" guix build option.

A problem that would need solving is that tests often depend on the
build directory, which is different from what is installed (and ends up
in the store).  The build directory is lost.

This is not true for all packages, but I’ve encountered enough for which
this would not work.

A change to the “build model” might be to allow for build directories to
be provided as inputs to a derivation.  (If this were to be implemented
developers could use that as a way to debug failed builds by keeping the
build directory, make changes, and continuing the build using the
modified build directory, but this feature would have to be heavily
guarded from being abused as a means to get stateful builds into the
store.)

--
Ricardo

Re: shortening the git test suite

[Mike Gerwitz]

[-- Attachment #1: Type: text/plain, Size: 715 bytes --]

On Fri, Jul 06, 2018 at 23:05:04 +0200, Ludovic Courtès wrote:
> Instead of ‘git-minimal’, we could have ‘git’ without SVN support, and
> have a separate ‘git-svn’ package.  We can probably arrange for that
> ‘git-svn’ package to only provide the relevant libexec/git-core bits
> (git-svn is just a Perl script anyway.)
>
> Thoughts?

I'm a git-svn user, and that sounds fine with me.

I agree that's much better to provide a separate package with the SVN
tests rather than simply disable SVN tests in a full-featured Git.

-- 
Mike Gerwitz
Free Software Hacker+Activist | GNU Maintainer & Volunteer
GPG: D6E9 B930 028A 6C38 F43B  2388 FEF6 3574 5E6F 6D05
https://mikegerwitz.com

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

Re: shortening the git test suite

[Mark H Weaver]

Hi Ricardo,

Ricardo Wurmus <rekado@elephly.net> writes:

> Mark H Weaver <mhw@netris.org> writes:
>
>>>> Also, looking ahead, I think it would be great if we could eventually
>>>> move to a model where the tests of some packages are split off into
>>>> separate derivations.  Similarly, we could work toward splitting off
>>>> documentation generation to separate derivation for selected packages.
>>>> The most important advantage to this approach is that it would allow
>>>> inputs needed only for tests or docs to be omitted from the inputs of
>>>> the main package.  I expect that this will in many cases be needed to
>>>> prevent circular dependencies, and it could also greatly reduce the
>>>> amount of rebuilding needed after updating certain packages.
>>>
>>> Currently if test fails, the whole derivation fails, and you can’t
>>> install your package.  If tests were run separately, this would no
>>> longer hold: you could get your package regardless of whether tests
>>> fail.
>>
>> Indeed, and I agree that we would need to address this.
>>
>>> How would you address this?  I guess that calls for a new build
>>> model, no?
>>
>> I'm not sure what you mean by "build model", whether you are talking
>> about the daemon interface or something else, but I think the changes
>> could be confined to the Guix user interface.  A field could be added to
>> <package>, somewhat similar to 'replacement', but pointing to a package
>> object which runs tests, or perhaps a list of package objects.  The guix
>> client could simply add the test packages to the list of derivations to
>> build.  This could be inhibited via a "--no-tests" guix build option.
>
> A problem that would need solving is that tests often depend on the
> build directory, which is different from what is installed (and ends up
> in the store).  The build directory is lost.

I suggested a solution to this problem in my paragraph immediately
following the one you quoted above.  Did you see it?  Here it is again:

  In typical cases where "make check" expects to be run from a fully built
  source directory, the main package would typically produce a tarball of
  the built source directory as an additional output.  The test package
  would simply unpack this tarball and run "make check" in it.

       Mark

Re: shortening the git test suite

[Ricardo Wurmus]

Hi Mark,

>>> I'm not sure what you mean by "build model", whether you are talking
>>> about the daemon interface or something else, but I think the changes
>>> could be confined to the Guix user interface.  A field could be added to
>>> <package>, somewhat similar to 'replacement', but pointing to a package
>>> object which runs tests, or perhaps a list of package objects.  The guix
>>> client could simply add the test packages to the list of derivations to
>>> build.  This could be inhibited via a "--no-tests" guix build option.
>>
>> A problem that would need solving is that tests often depend on the
>> build directory, which is different from what is installed (and ends up
>> in the store).  The build directory is lost.
>
> I suggested a solution to this problem in my paragraph immediately
> following the one you quoted above.  Did you see it?  Here it is again:
>
>   In typical cases where "make check" expects to be run from a fully built
>   source directory, the main package would typically produce a tarball of
>   the built source directory as an additional output.  The test package
>   would simply unpack this tarball and run "make check" in it.

Yes, I read it, but I misunderstood your suggestion regarding the
tarball.  This build directory output which would be used to continue
the build would likely not be reproducible as it contains files that are
not meant to be installed, such as logs with timestamps.  Is this okay?

--
Ricardo

Re: shortening the git test suite

[Ludovic Courtès]

Hello,

Mark H Weaver <mhw@netris.org> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:

[...]

>> Currently if test fails, the whole derivation fails, and you can’t
>> install your package.  If tests were run separately, this would no
>> longer hold: you could get your package regardless of whether tests
>> fail.
>
> Indeed, and I agree that we would need to address this.
>
>> How would you address this?  I guess that calls for a new build
>> model, no?
>
> I'm not sure what you mean by "build model", whether you are talking
> about the daemon interface or something else, but I think the changes
> could be confined to the Guix user interface.  A field could be added to
> <package>, somewhat similar to 'replacement', but pointing to a package
> object which runs tests, or perhaps a list of package objects.  The guix
> client could simply add the test packages to the list of derivations to
> build.  This could be inhibited via a "--no-tests" guix build option.
>
> In typical cases where "make check" expects to be run from a fully built
> source directory, the main package would typically produce a tarball of
> the built source directory as an additional output.  The test package
> would simply unpack this tarball and run "make check" in it.
>
> Regarding the build order: ideally, we would run the tests for package
> FOO immediately after building FOO, or at least before building anything
> that depends on FOO, to avoid wasted work in case the tests fail.  I'm
> not sure how best to accomplish this.
>
> What do you think?

I’m afraid doing everything at the level of package objects rather than
derivations would lead to something clunky (grafts already suffer from
that.)

Namely, Guix clients would have to explicitly do a second
‘build-derivations’ RPC once they’re done building packages.  If they
don’t do that, then tests are not run and you get to install a possibly
broken package.

Besides, there are lots of details to get right in a proposal like
yours.  In particular, saving the build tree and for later reuse may
prove to be more challenging than it seems.

IOW, this looks like a radical change with lots of potential pitfalls,
one I’d rather not make before 1.0.  I agree that long test suites can
be a problem, but for now I think we should focus on package-specific
solutions, to the extent possible, such as what we discussed for Git, as
well as improvements to our build farm infrastructure.

Thoughts?

Ludo’.

Re: shortening the git test suite

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 4684 bytes --]

Hi Ricardo, Mark, and Ludo,

Ricardo Wurmus <rekado@elephly.net> writes:

> What do you think of pushing some package updates only to feature
> branches that follow a certain naming convention (e.g. “_update-foo” for
> updating the “foo” package), which causes Cuirass to build them and
> merge the branch into “master” (semi-)automatically when the build is
> successful?

If the goal is to ensure that the substitutes for a newly pulled Guix
are available when "guix pull" runs, then wouldn't channels accomplish
that goal?  I realize we don't have channels yet, though.

I think your proposal would be a reasonable improvement.  In the case of
a branch like "_update-foo", Cuirass would probably need to build not
only the package "foo" but also any packages that depend on it, right?
Since we're talking about updates to the master branch, this shouldn't
cause too many rebuilds, so it shouldn't add much load to the build
farm.

If we're going to allow Cuirass to merge an update branch into master
automatically, it might also make sense to allow Cuirass to
automatically rebase the update branch onto master before merging it (to
ensure the merge is a fast-forward merge), and delete the update branch
after merging it (to prevent clutter in commands like "git branch -av").
Alternatively, we could leave those steps to a human and just have
Cuirass send a summary email or make the results available via its web
interface.  I think that with the right automated tests to protect
against silly bugs (like accidentally deleting the master branch), it
would probably be safer to have Cuirass automatically merge and clean up
after itself, rather than rely on humans to do it.  Humans are prone to
make mistakes, especially when performing tedious tasks.

Mark H Weaver <mhw@netris.org> writes:

>> Personally, I think that the SVN tests are non-essential (after all,
>> we’re building Git here, we keep running the individual test suites of
>> Git and Subversion, and git-svn interop seems like a thing that only
>> upstream need to worry about), which is why I made this proposal.
>
> Why do you say that only upstream needs to worry about it?  I would
> think that you could say the same thing about almost any test suite, but
> there's always the possibility that our particular combination of input
> versions, or the unusual aspects of Guix, might break something that
> upstream doesn't know about.
>
> I would think that git-svn interop is something that any user of the
> git/svn integration needs to worry about.

I agree: just because I don't use a feature doesn't mean that we
shouldn't test that feature if we can.  We ought to be able to run all
the tests.  I think Ricardo's proposal would help increase the
availability of substitutes, which would make long-running tests more
tolerable.

ludo@gnu.org (Ludovic Courtès) writes:

> Mark H Weaver <mhw@netris.org> skribis:
>
>> ludo@gnu.org (Ludovic Courtès) writes:
>
> [...]
>
>>> Currently if test fails, the whole derivation fails, and you can’t
>>> install your package.  If tests were run separately, this would no
>>> longer hold: you could get your package regardless of whether tests
>>> fail.
>>
>> Indeed, and I agree that we would need to address this.
>>
>>> How would you address this?  I guess that calls for a new build
>>> model, no?
>>
>> I'm not sure what you mean by "build model" [...]
>
> [...]
>
> IOW, this looks like a radical change with lots of potential pitfalls,
> one I’d rather not make before 1.0.  I agree that long test suites can
> be a problem, but for now I think we should focus on package-specific
> solutions, to the extent possible, such as what we discussed for Git, as
> well as improvements to our build farm infrastructure.
>
> Thoughts?

I agree that changes like those seem especially tricky to get right.

While I'm here, I'll suggest another far-fetched idea.  Perhaps we could
rethink the relationship between packages, phases (of build systems),
and derivations.  Currently, I believe one package corresponds to one
derivation, and the build phases are just procedures executed during
that derivation.  Perhaps we could change this so that a package
corresponds to multiple derivations: one derivation per phase.  If that
were possible, then a package that runs all phases plus the test phase
would be able to easily re-use the results of a package that runs the
same phases but not the test phase.  However, I'm not sure if this
really makes sense, and in any case, the details would probably be
difficult to design and implement.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]