* shortening the git test suite @ 2018-06-30 19:34 Ricardo Wurmus 2018-06-30 23:52 ` Gábor Boskovits ` (3 more replies) 0 siblings, 4 replies; 21+ messages in thread From: Ricardo Wurmus @ 2018-06-30 19:34 UTC (permalink / raw) To: guix-devel Hi Guix, git takes a very long time to build, because it has an extensive test suite. Most of the time is spent in running the SVN interoperability tests, though, which are not really all that interesting for most uses of git. The Makefile says this: # Define NO_SVN_TESTS if you want to skip time-consuming SVN interoperability # tests. These tests take up a significant amount of the total test time # but are not needed unless you plan to talk to SVN repos. What do you think about disabling the SVN tests in the git package? -- Ricardo ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: shortening the git test suite 2018-06-30 19:34 shortening the git test suite Ricardo Wurmus @ 2018-06-30 23:52 ` Gábor Boskovits 2018-07-01 13:54 ` Marius Bakke ` (2 subsequent siblings) 3 siblings, 0 replies; 21+ messages in thread From: Gábor Boskovits @ 2018-06-30 23:52 UTC (permalink / raw) To: Ricardo Wurmus; +Cc: Guix-devel [-- Attachment #1: Type: text/plain, Size: 745 bytes --] Ricardo Wurmus <rekado@elephly.net> ezt írta (időpont: 2018. júl. 1., V, 1:20): > Hi Guix, > > git takes a very long time to build, because it has an extensive test > suite. Most of the time is spent in running the SVN interoperability > tests, though, which are not really all that interesting for most uses > of git. > > The Makefile says this: > > # Define NO_SVN_TESTS if you want to skip time-consuming SVN > interoperability > # tests. These tests take up a significant amount of the total test time > # but are not needed unless you plan to talk to SVN repos. > > What do you think about disabling the SVN tests in the git package? > > I think it is ok to disable SVN tests. > -- > Ricardo > > > [-- Attachment #2: Type: text/html, Size: 1169 bytes --] ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: shortening the git test suite 2018-06-30 19:34 shortening the git test suite Ricardo Wurmus 2018-06-30 23:52 ` Gábor Boskovits @ 2018-07-01 13:54 ` Marius Bakke 2018-07-05 9:28 ` Ricardo Wurmus 2018-07-02 4:56 ` Mark H Weaver 2018-07-05 8:44 ` Chris Marusich 3 siblings, 1 reply; 21+ messages in thread From: Marius Bakke @ 2018-07-01 13:54 UTC (permalink / raw) To: Ricardo Wurmus, guix-devel [-- Attachment #1: Type: text/plain, Size: 962 bytes --] Ricardo Wurmus <rekado@elephly.net> writes: > Hi Guix, > > git takes a very long time to build, because it has an extensive test > suite. Most of the time is spent in running the SVN interoperability > tests, though, which are not really all that interesting for most uses > of git. > > The Makefile says this: > > # Define NO_SVN_TESTS if you want to skip time-consuming SVN interoperability > # tests. These tests take up a significant amount of the total test time > # but are not needed unless you plan to talk to SVN repos. > > What do you think about disabling the SVN tests in the git package? I'm wary of this because it also tests our SVN package, and makes sure they work together. But I'm not completely opposed to it either. How much time does it save? When debugging the git build, I usually set #:parallel-tests? #t. So far I haven't come across a failure related to that. I wonder if <https://bugs.gnu.org/29512> is still relevant. [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 487 bytes --] ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: shortening the git test suite 2018-07-01 13:54 ` Marius Bakke @ 2018-07-05 9:28 ` Ricardo Wurmus 0 siblings, 0 replies; 21+ messages in thread From: Ricardo Wurmus @ 2018-07-05 9:28 UTC (permalink / raw) To: Marius Bakke; +Cc: guix-devel Marius Bakke <mbakke@fastmail.com> writes: > When debugging the git build, I usually set #:parallel-tests? #t. So > far I haven't come across a failure related to that. I wonder if > <https://bugs.gnu.org/29512> is still relevant. That’s a good question. I don’t know. -- Ricardo ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: shortening the git test suite 2018-06-30 19:34 shortening the git test suite Ricardo Wurmus 2018-06-30 23:52 ` Gábor Boskovits 2018-07-01 13:54 ` Marius Bakke @ 2018-07-02 4:56 ` Mark H Weaver 2018-07-02 8:39 ` Gábor Boskovits 2018-07-05 8:44 ` Chris Marusich 3 siblings, 1 reply; 21+ messages in thread From: Mark H Weaver @ 2018-07-02 4:56 UTC (permalink / raw) To: Ricardo Wurmus; +Cc: guix-devel Ricardo Wurmus <rekado@elephly.net> writes: > git takes a very long time to build, because it has an extensive test > suite. Most of the time is spent in running the SVN interoperability > tests, though, which are not really all that interesting for most uses > of git. > > The Makefile says this: > > # Define NO_SVN_TESTS if you want to skip time-consuming SVN interoperability > # tests. These tests take up a significant amount of the total test time > # but are not needed unless you plan to talk to SVN repos. > > What do you think about disabling the SVN tests in the git package? My preference would be to keep the SVN tests enabled, but it's not something I feel very strongly about. Thanks, Mark ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: shortening the git test suite 2018-07-02 4:56 ` Mark H Weaver @ 2018-07-02 8:39 ` Gábor Boskovits 2018-07-05 9:28 ` Ricardo Wurmus 0 siblings, 1 reply; 21+ messages in thread From: Gábor Boskovits @ 2018-07-02 8:39 UTC (permalink / raw) To: Mark H Weaver; +Cc: Guix-devel [-- Attachment #1: Type: text/plain, Size: 1003 bytes --] Mark H Weaver <mhw@netris.org> ezt írta (időpont: 2018. júl. 2., H, 6:58): > Ricardo Wurmus <rekado@elephly.net> writes: > > > git takes a very long time to build, because it has an extensive test > > suite. Most of the time is spent in running the SVN interoperability > > tests, though, which are not really all that interesting for most uses > > of git. > > > > The Makefile says this: > > > > # Define NO_SVN_TESTS if you want to skip time-consuming SVN > interoperability > > # tests. These tests take up a significant amount of the total test > time > > # but are not needed unless you plan to talk to SVN repos. > > > > What do you think about disabling the SVN tests in the git package? > > My preference would be to keep the SVN tests enabled, but it's not > something I feel very strongly about. > > Thanks, > Mark > > Do we have a way to make running this SVN test suite optional? For example to run it only before release, or on demand? [-- Attachment #2: Type: text/html, Size: 1428 bytes --] ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: shortening the git test suite 2018-07-02 8:39 ` Gábor Boskovits @ 2018-07-05 9:28 ` Ricardo Wurmus 0 siblings, 0 replies; 21+ messages in thread From: Ricardo Wurmus @ 2018-07-05 9:28 UTC (permalink / raw) To: Gábor Boskovits; +Cc: Guix-devel Gábor Boskovits <boskovits@gmail.com> writes: > Mark H Weaver <mhw@netris.org> ezt írta (időpont: 2018. júl. 2., H, 6:58): > >> Ricardo Wurmus <rekado@elephly.net> writes: >> >> > git takes a very long time to build, because it has an extensive test >> > suite. Most of the time is spent in running the SVN interoperability >> > tests, though, which are not really all that interesting for most uses >> > of git. >> > >> > The Makefile says this: >> > >> > # Define NO_SVN_TESTS if you want to skip time-consuming SVN >> interoperability >> > # tests. These tests take up a significant amount of the total test >> time >> > # but are not needed unless you plan to talk to SVN repos. >> > >> > What do you think about disabling the SVN tests in the git package? >> >> My preference would be to keep the SVN tests enabled, but it's not >> something I feel very strongly about. >> >> Thanks, >> Mark >> >> > Do we have a way to make running this SVN test suite optional? For example > to run it only before release, or on demand? No, and I would think that a mechanism to execute them optionally would necessarily be a hack that undermines some of the guarantees that Guix tries hard to provide. -- Ricardo ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: shortening the git test suite 2018-06-30 19:34 shortening the git test suite Ricardo Wurmus ` (2 preceding siblings ...) 2018-07-02 4:56 ` Mark H Weaver @ 2018-07-05 8:44 ` Chris Marusich 2018-07-05 9:21 ` Ricardo Wurmus 3 siblings, 1 reply; 21+ messages in thread From: Chris Marusich @ 2018-07-05 8:44 UTC (permalink / raw) To: Ricardo Wurmus; +Cc: guix-devel [-- Attachment #1: Type: text/plain, Size: 1520 bytes --] Ricardo Wurmus <rekado@elephly.net> writes: > Hi Guix, > > git takes a very long time to build, because it has an extensive test > suite. Most of the time is spent in running the SVN interoperability > tests, though, which are not really all that interesting for most uses > of git. > > The Makefile says this: > > # Define NO_SVN_TESTS if you want to skip time-consuming SVN interoperability > # tests. These tests take up a significant amount of the total test time > # but are not needed unless you plan to talk to SVN repos. > > What do you think about disabling the SVN tests in the git package? This sounds similar to the discussion we had earlier about treating tests as a special case: https://lists.gnu.org/archive/html/guix-devel/2018-04/msg00071.html I felt that the conclusion of that thread was basically that if someone is concerned about the build time, then they ought to be able to use substitutes to speed things up, and we should continue to run as many tests as possible in order to discover problems sooner. I'm in favor of disabling tests if they fail for reasons other than bugs in the program and there isn't an easy fix (e.g., they require root privileges, or they fail due to randomness in the tests), but disabling them simply because they take a "long" time doesn't seem like a great reason, to me. I'm sympathetic, since I don't like waiting around, either, but substitutes should be sufficient for those who need faster builds. -- Chris [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 832 bytes --] ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: shortening the git test suite 2018-07-05 8:44 ` Chris Marusich @ 2018-07-05 9:21 ` Ricardo Wurmus 2018-07-06 9:46 ` Gábor Boskovits ` (2 more replies) 0 siblings, 3 replies; 21+ messages in thread From: Ricardo Wurmus @ 2018-07-05 9:21 UTC (permalink / raw) To: Chris Marusich; +Cc: guix-devel Chris Marusich <cmmarusich@gmail.com> writes: > Ricardo Wurmus <rekado@elephly.net> writes: > >> Hi Guix, >> >> git takes a very long time to build, because it has an extensive test >> suite. Most of the time is spent in running the SVN interoperability >> tests, though, which are not really all that interesting for most uses >> of git. >> >> The Makefile says this: >> >> # Define NO_SVN_TESTS if you want to skip time-consuming SVN interoperability >> # tests. These tests take up a significant amount of the total test time >> # but are not needed unless you plan to talk to SVN repos. >> >> What do you think about disabling the SVN tests in the git package? > > This sounds similar to the discussion we had earlier about treating > tests as a special case: > > https://lists.gnu.org/archive/html/guix-devel/2018-04/msg00071.html > > I felt that the conclusion of that thread was basically that if someone > is concerned about the build time, then they ought to be able to use > substitutes to speed things up, and we should continue to run as many > tests as possible in order to discover problems sooner. What I’m worried about is availability of substitutes. When we keep changing packages on the “master” branch that have very expensive test suites then we accept that people won’t have substitutes for a while. The duration of that while depends on how quickly the build of this package is started by our continuous integration software, and how long it takes to complete the build. Granted, disabling parts of the test suite in an attempt to shorten it is really a technical fix to a social problem. Personally, I think that the SVN tests are non-essential (after all, we’re building Git here, we keep running the individual test suites of Git and Subversion, and git-svn interop seems like a thing that only upstream need to worry about), which is why I made this proposal. But as it seems that the people who responded to this message rather lean in the other direction, let’s try to address the social problem instead: How can we change our workflow to make sure that for packages with long build/test times we can provide substitutes more quickly? Currently, our policy for pushing changes to “staging” and “core-updates” is based on package counts. I’m not suggesting that changes to Git be pushed to “staging”. What do you think of pushing some package updates only to feature branches that follow a certain naming convention (e.g. “_update-foo” for updating the “foo” package), which causes Cuirass to build them and merge the branch into “master” (semi-)automatically when the build is successful? (Obviously, any kind of automation has to be thought out carefully, but I’m sure we would be able to find a solution.) What do you think? -- Ricardo ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: shortening the git test suite 2018-07-05 9:21 ` Ricardo Wurmus @ 2018-07-06 9:46 ` Gábor Boskovits 2018-07-06 18:19 ` Mark H Weaver 2018-07-06 21:09 ` Automatic branch merging upon build completion Ludovic Courtès 2 siblings, 0 replies; 21+ messages in thread From: Gábor Boskovits @ 2018-07-06 9:46 UTC (permalink / raw) To: Ricardo Wurmus; +Cc: Guix-devel [-- Attachment #1: Type: text/plain, Size: 3310 bytes --] Ricardo Wurmus <rekado@elephly.net> ezt írta (időpont: 2018. júl. 5., Cs 11:22): > > Chris Marusich <cmmarusich@gmail.com> writes: > > > Ricardo Wurmus <rekado@elephly.net> writes: > > > >> Hi Guix, > >> > >> git takes a very long time to build, because it has an extensive test > >> suite. Most of the time is spent in running the SVN interoperability > >> tests, though, which are not really all that interesting for most uses > >> of git. > >> > >> The Makefile says this: > >> > >> # Define NO_SVN_TESTS if you want to skip time-consuming SVN > interoperability > >> # tests. These tests take up a significant amount of the total test > time > >> # but are not needed unless you plan to talk to SVN repos. > >> > >> What do you think about disabling the SVN tests in the git package? > > > > This sounds similar to the discussion we had earlier about treating > > tests as a special case: > > > > https://lists.gnu.org/archive/html/guix-devel/2018-04/msg00071.html > > > > I felt that the conclusion of that thread was basically that if someone > > is concerned about the build time, then they ought to be able to use > > substitutes to speed things up, and we should continue to run as many > > tests as possible in order to discover problems sooner. > > What I’m worried about is availability of substitutes. When we keep > changing packages on the “master” branch that have very expensive test > suites then we accept that people won’t have substitutes for a while. > The duration of that while depends on how quickly the build of this > package is started by our continuous integration software, and how long > it takes to complete the build. > > Granted, disabling parts of the test suite in an attempt to shorten it > is really a technical fix to a social problem. > > Personally, I think that the SVN tests are non-essential (after all, > we’re building Git here, we keep running the individual test suites of > Git and Subversion, and git-svn interop seems like a thing that only > upstream need to worry about), which is why I made this proposal. But > as it seems that the people who responded to this message rather lean in > the other direction, let’s try to address the social problem instead: > > How can we change our workflow to make sure that for packages with long > build/test times we can provide substitutes more quickly? Currently, > our policy for pushing changes to “staging” and “core-updates” is based > on package counts. I’m not suggesting that changes to Git be pushed to > “staging”. > > What do you think of pushing some package updates only to feature > branches that follow a certain naming convention (e.g. “_update-foo” for > updating the “foo” package), which causes Cuirass to build them and > merge the branch into “master” (semi-)automatically when the build is > successful? > > > > I think this is a very reasonable approach. If this will be the way to go, then we also need a way to select packages to which this applies, and find a way to communicate this. > (Obviously, any kind of automation has to be thought out carefully, but > I’m sure we would be able to find a solution.) > > What do you think? > > -- > Ricardo > > > [-- Attachment #2: Type: text/html, Size: 4304 bytes --] ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: shortening the git test suite 2018-07-05 9:21 ` Ricardo Wurmus 2018-07-06 9:46 ` Gábor Boskovits @ 2018-07-06 18:19 ` Mark H Weaver 2018-07-06 21:05 ` Ludovic Courtès 2018-07-06 21:09 ` Automatic branch merging upon build completion Ludovic Courtès 2 siblings, 1 reply; 21+ messages in thread From: Mark H Weaver @ 2018-07-06 18:19 UTC (permalink / raw) To: Ricardo Wurmus; +Cc: guix-devel Hi Ricardo, Ricardo Wurmus <rekado@elephly.net> writes: > Chris Marusich <cmmarusich@gmail.com> writes: > >> Ricardo Wurmus <rekado@elephly.net> writes: >> >>> Hi Guix, >>> >>> git takes a very long time to build, because it has an extensive test >>> suite. Most of the time is spent in running the SVN interoperability >>> tests, though, which are not really all that interesting for most uses >>> of git. >>> >>> The Makefile says this: >>> >>> # Define NO_SVN_TESTS if you want to skip time-consuming SVN interoperability >>> # tests. These tests take up a significant amount of the total test time >>> # but are not needed unless you plan to talk to SVN repos. >>> >>> What do you think about disabling the SVN tests in the git package? >> >> This sounds similar to the discussion we had earlier about treating >> tests as a special case: >> >> https://lists.gnu.org/archive/html/guix-devel/2018-04/msg00071.html >> >> I felt that the conclusion of that thread was basically that if someone >> is concerned about the build time, then they ought to be able to use >> substitutes to speed things up, and we should continue to run as many >> tests as possible in order to discover problems sooner. > > What I’m worried about is availability of substitutes. When we keep > changing packages on the “master” branch that have very expensive test > suites then we accept that people won’t have substitutes for a while. > The duration of that while depends on how quickly the build of this > package is started by our continuous integration software, and how long > it takes to complete the build. > > Granted, disabling parts of the test suite in an attempt to shorten it > is really a technical fix to a social problem. > > Personally, I think that the SVN tests are non-essential (after all, > we’re building Git here, we keep running the individual test suites of > Git and Subversion, and git-svn interop seems like a thing that only > upstream need to worry about), which is why I made this proposal. Why do you say that only upstream needs to worry about it? I would think that you could say the same thing about almost any test suite, but there's always the possibility that our particular combination of input versions, or the unusual aspects of Guix, might break something that upstream doesn't know about. I would think that git-svn interop is something that any user of the git/svn integration needs to worry about. If we feel that very few of our users care about git-svn interop, another option would be to add a lighter variant of 'git' that does not include SVN support. It would probably be a good idea to have a 'git-minimal' package anyway, for use by our 'git-fetch' origin method. Naturally, only the 'git' package variant that includes SVN support would need to run the SVN tests. Also, looking ahead, I think it would be great if we could eventually move to a model where the tests of some packages are split off into separate derivations. Similarly, we could work toward splitting off documentation generation to separate derivation for selected packages. The most important advantage to this approach is that it would allow inputs needed only for tests or docs to be omitted from the inputs of the main package. I expect that this will in many cases be needed to prevent circular dependencies, and it could also greatly reduce the amount of rebuilding needed after updating certain packages. What do you think? Mark ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: shortening the git test suite 2018-07-06 18:19 ` Mark H Weaver @ 2018-07-06 21:05 ` Ludovic Courtès 2018-07-07 8:07 ` Gábor Boskovits ` (2 more replies) 0 siblings, 3 replies; 21+ messages in thread From: Ludovic Courtès @ 2018-07-06 21:05 UTC (permalink / raw) To: Mark H Weaver; +Cc: guix-devel Hello, Mark H Weaver <mhw@netris.org> skribis: > Why do you say that only upstream needs to worry about it? I would > think that you could say the same thing about almost any test suite, but > there's always the possibility that our particular combination of input > versions, or the unusual aspects of Guix, might break something that > upstream doesn't know about. > > I would think that git-svn interop is something that any user of the > git/svn integration needs to worry about. I agree. > If we feel that very few of our users care about git-svn interop, > another option would be to add a lighter variant of 'git' that does not > include SVN support. It would probably be a good idea to have a > 'git-minimal' package anyway, for use by our 'git-fetch' origin method. > Naturally, only the 'git' package variant that includes SVN support > would need to run the SVN tests. Yeah, I think we could to do something like this in this case. Instead of ‘git-minimal’, we could have ‘git’ without SVN support, and have a separate ‘git-svn’ package. We can probably arrange for that ‘git-svn’ package to only provide the relevant libexec/git-core bits (git-svn is just a Perl script anyway.) Thoughts? > Also, looking ahead, I think it would be great if we could eventually > move to a model where the tests of some packages are split off into > separate derivations. Similarly, we could work toward splitting off > documentation generation to separate derivation for selected packages. > The most important advantage to this approach is that it would allow > inputs needed only for tests or docs to be omitted from the inputs of > the main package. I expect that this will in many cases be needed to > prevent circular dependencies, and it could also greatly reduce the > amount of rebuilding needed after updating certain packages. Currently if test fails, the whole derivation fails, and you can’t install your package. If tests were run separately, this would no longer hold: you could get your package regardless of whether tests fail. How would you address this? I guess that calls for a new build model, no? Thanks, Ludo’. ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: shortening the git test suite 2018-07-06 21:05 ` Ludovic Courtès @ 2018-07-07 8:07 ` Gábor Boskovits 2018-07-07 16:44 ` Mark H Weaver 2018-07-08 1:29 ` Mike Gerwitz 2 siblings, 0 replies; 21+ messages in thread From: Gábor Boskovits @ 2018-07-07 8:07 UTC (permalink / raw) To: Ludovic Courtès; +Cc: Guix-devel [-- Attachment #1: Type: text/plain, Size: 2893 bytes --] Ludovic Courtès <ludo@gnu.org> ezt írta (időpont: 2018. júl. 6., P, 23:05): > Hello, > > Mark H Weaver <mhw@netris.org> skribis: > > > Why do you say that only upstream needs to worry about it? I would > > think that you could say the same thing about almost any test suite, but > > there's always the possibility that our particular combination of input > > versions, or the unusual aspects of Guix, might break something that > > upstream doesn't know about. > > > > I would think that git-svn interop is something that any user of the > > git/svn integration needs to worry about. > > I agree. > > > If we feel that very few of our users care about git-svn interop, > > another option would be to add a lighter variant of 'git' that does not > > include SVN support. It would probably be a good idea to have a > > 'git-minimal' package anyway, for use by our 'git-fetch' origin method. > > Naturally, only the 'git' package variant that includes SVN support > > would need to run the SVN tests. > > Yeah, I think we could to do something like this in this case. > > Instead of ‘git-minimal’, we could have ‘git’ without SVN support, and > have a separate ‘git-svn’ package. We can probably arrange for that > ‘git-svn’ package to only provide the relevant libexec/git-core bits > (git-svn is just a Perl script anyway.) > > Thoughts? > > > Also, looking ahead, I think it would be great if we could eventually > > move to a model where the tests of some packages are split off into > > separate derivations. Similarly, we could work toward splitting off > > documentation generation to separate derivation for selected packages. > > The most important advantage to this approach is that it would allow > > inputs needed only for tests or docs to be omitted from the inputs of > > the main package. I expect that this will in many cases be needed to > > prevent circular dependencies, and it could also greatly reduce the > > amount of rebuilding needed after updating certain packages. > > Currently if test fails, the whole derivation fails, and you can’t > install your package. If tests were run separately, this would no > longer hold: you could get your package regardless of whether tests > fail. How would you address this? I guess that calls for a new build > model, no? > > One example where this can be seen in action is the dependency of java-jarjar on java-asm-bootstrap. java-asm-bootstrap is java-asm without the tests, it is exactly there to break a dependency cycle. java-asm is a full rebuild using tests. java-asm-bootstrap is defined, while java-asm is define-public-ed (here a full rebuild is involved). I'm not sure that a full rebuild can be worked around in all cases, I guess that sometimes test capabilities are detected at configure time. WDYT? Thanks, > Ludo’. > > [-- Attachment #2: Type: text/html, Size: 3565 bytes --] ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: shortening the git test suite 2018-07-06 21:05 ` Ludovic Courtès 2018-07-07 8:07 ` Gábor Boskovits @ 2018-07-07 16:44 ` Mark H Weaver 2018-07-07 21:37 ` Ricardo Wurmus ` (2 more replies) 2018-07-08 1:29 ` Mike Gerwitz 2 siblings, 3 replies; 21+ messages in thread From: Mark H Weaver @ 2018-07-07 16:44 UTC (permalink / raw) To: Ludovic Courtès; +Cc: guix-devel Hi Ludovic, ludo@gnu.org (Ludovic Courtès) writes: > Mark H Weaver <mhw@netris.org> skribis: > >> If we feel that very few of our users care about git-svn interop, >> another option would be to add a lighter variant of 'git' that does not >> include SVN support. It would probably be a good idea to have a >> 'git-minimal' package anyway, for use by our 'git-fetch' origin method. >> Naturally, only the 'git' package variant that includes SVN support >> would need to run the SVN tests. > > Yeah, I think we could to do something like this in this case. > > Instead of ‘git-minimal’, we could have ‘git’ without SVN support, and > have a separate ‘git-svn’ package. We can probably arrange for that > ‘git-svn’ package to only provide the relevant libexec/git-core bits > (git-svn is just a Perl script anyway.) > > Thoughts? Sure, that sounds good to me. Whether it makes to have a 'git-minimal' package depends on how many other optional dependencies could reasonably be removed from it, besides subversion. If not much else could be removed, then I suppose there would be no need for a 'git-minimal' variant. >> Also, looking ahead, I think it would be great if we could eventually >> move to a model where the tests of some packages are split off into >> separate derivations. Similarly, we could work toward splitting off >> documentation generation to separate derivation for selected packages. >> The most important advantage to this approach is that it would allow >> inputs needed only for tests or docs to be omitted from the inputs of >> the main package. I expect that this will in many cases be needed to >> prevent circular dependencies, and it could also greatly reduce the >> amount of rebuilding needed after updating certain packages. > > Currently if test fails, the whole derivation fails, and you can’t > install your package. If tests were run separately, this would no > longer hold: you could get your package regardless of whether tests > fail. Indeed, and I agree that we would need to address this. > How would you address this? I guess that calls for a new build > model, no? I'm not sure what you mean by "build model", whether you are talking about the daemon interface or something else, but I think the changes could be confined to the Guix user interface. A field could be added to <package>, somewhat similar to 'replacement', but pointing to a package object which runs tests, or perhaps a list of package objects. The guix client could simply add the test packages to the list of derivations to build. This could be inhibited via a "--no-tests" guix build option. In typical cases where "make check" expects to be run from a fully built source directory, the main package would typically produce a tarball of the built source directory as an additional output. The test package would simply unpack this tarball and run "make check" in it. Regarding the build order: ideally, we would run the tests for package FOO immediately after building FOO, or at least before building anything that depends on FOO, to avoid wasted work in case the tests fail. I'm not sure how best to accomplish this. What do you think? Mark ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: shortening the git test suite 2018-07-07 16:44 ` Mark H Weaver @ 2018-07-07 21:37 ` Ricardo Wurmus 2018-07-08 22:41 ` Mark H Weaver 2018-07-11 12:46 ` Ludovic Courtès 2018-07-12 9:27 ` Chris Marusich 2 siblings, 1 reply; 21+ messages in thread From: Ricardo Wurmus @ 2018-07-07 21:37 UTC (permalink / raw) To: Mark H Weaver; +Cc: guix-devel Mark H Weaver <mhw@netris.org> writes: >>> Also, looking ahead, I think it would be great if we could eventually >>> move to a model where the tests of some packages are split off into >>> separate derivations. Similarly, we could work toward splitting off >>> documentation generation to separate derivation for selected packages. >>> The most important advantage to this approach is that it would allow >>> inputs needed only for tests or docs to be omitted from the inputs of >>> the main package. I expect that this will in many cases be needed to >>> prevent circular dependencies, and it could also greatly reduce the >>> amount of rebuilding needed after updating certain packages. >> >> Currently if test fails, the whole derivation fails, and you can’t >> install your package. If tests were run separately, this would no >> longer hold: you could get your package regardless of whether tests >> fail. > > Indeed, and I agree that we would need to address this. > >> How would you address this? I guess that calls for a new build >> model, no? > > I'm not sure what you mean by "build model", whether you are talking > about the daemon interface or something else, but I think the changes > could be confined to the Guix user interface. A field could be added to > <package>, somewhat similar to 'replacement', but pointing to a package > object which runs tests, or perhaps a list of package objects. The guix > client could simply add the test packages to the list of derivations to > build. This could be inhibited via a "--no-tests" guix build option. A problem that would need solving is that tests often depend on the build directory, which is different from what is installed (and ends up in the store). The build directory is lost. This is not true for all packages, but I’ve encountered enough for which this would not work. A change to the “build model” might be to allow for build directories to be provided as inputs to a derivation. (If this were to be implemented developers could use that as a way to debug failed builds by keeping the build directory, make changes, and continuing the build using the modified build directory, but this feature would have to be heavily guarded from being abused as a means to get stateful builds into the store.) -- Ricardo ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: shortening the git test suite 2018-07-07 21:37 ` Ricardo Wurmus @ 2018-07-08 22:41 ` Mark H Weaver 2018-07-09 7:47 ` Ricardo Wurmus 0 siblings, 1 reply; 21+ messages in thread From: Mark H Weaver @ 2018-07-08 22:41 UTC (permalink / raw) To: Ricardo Wurmus; +Cc: guix-devel Hi Ricardo, Ricardo Wurmus <rekado@elephly.net> writes: > Mark H Weaver <mhw@netris.org> writes: > >>>> Also, looking ahead, I think it would be great if we could eventually >>>> move to a model where the tests of some packages are split off into >>>> separate derivations. Similarly, we could work toward splitting off >>>> documentation generation to separate derivation for selected packages. >>>> The most important advantage to this approach is that it would allow >>>> inputs needed only for tests or docs to be omitted from the inputs of >>>> the main package. I expect that this will in many cases be needed to >>>> prevent circular dependencies, and it could also greatly reduce the >>>> amount of rebuilding needed after updating certain packages. >>> >>> Currently if test fails, the whole derivation fails, and you can’t >>> install your package. If tests were run separately, this would no >>> longer hold: you could get your package regardless of whether tests >>> fail. >> >> Indeed, and I agree that we would need to address this. >> >>> How would you address this? I guess that calls for a new build >>> model, no? >> >> I'm not sure what you mean by "build model", whether you are talking >> about the daemon interface or something else, but I think the changes >> could be confined to the Guix user interface. A field could be added to >> <package>, somewhat similar to 'replacement', but pointing to a package >> object which runs tests, or perhaps a list of package objects. The guix >> client could simply add the test packages to the list of derivations to >> build. This could be inhibited via a "--no-tests" guix build option. > > A problem that would need solving is that tests often depend on the > build directory, which is different from what is installed (and ends up > in the store). The build directory is lost. I suggested a solution to this problem in my paragraph immediately following the one you quoted above. Did you see it? Here it is again: In typical cases where "make check" expects to be run from a fully built source directory, the main package would typically produce a tarball of the built source directory as an additional output. The test package would simply unpack this tarball and run "make check" in it. Mark ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: shortening the git test suite 2018-07-08 22:41 ` Mark H Weaver @ 2018-07-09 7:47 ` Ricardo Wurmus 0 siblings, 0 replies; 21+ messages in thread From: Ricardo Wurmus @ 2018-07-09 7:47 UTC (permalink / raw) To: Mark H Weaver; +Cc: guix-devel Hi Mark, >>> I'm not sure what you mean by "build model", whether you are talking >>> about the daemon interface or something else, but I think the changes >>> could be confined to the Guix user interface. A field could be added to >>> <package>, somewhat similar to 'replacement', but pointing to a package >>> object which runs tests, or perhaps a list of package objects. The guix >>> client could simply add the test packages to the list of derivations to >>> build. This could be inhibited via a "--no-tests" guix build option. >> >> A problem that would need solving is that tests often depend on the >> build directory, which is different from what is installed (and ends up >> in the store). The build directory is lost. > > I suggested a solution to this problem in my paragraph immediately > following the one you quoted above. Did you see it? Here it is again: > > In typical cases where "make check" expects to be run from a fully built > source directory, the main package would typically produce a tarball of > the built source directory as an additional output. The test package > would simply unpack this tarball and run "make check" in it. Yes, I read it, but I misunderstood your suggestion regarding the tarball. This build directory output which would be used to continue the build would likely not be reproducible as it contains files that are not meant to be installed, such as logs with timestamps. Is this okay? -- Ricardo ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: shortening the git test suite 2018-07-07 16:44 ` Mark H Weaver 2018-07-07 21:37 ` Ricardo Wurmus @ 2018-07-11 12:46 ` Ludovic Courtès 2018-07-12 9:27 ` Chris Marusich 2 siblings, 0 replies; 21+ messages in thread From: Ludovic Courtès @ 2018-07-11 12:46 UTC (permalink / raw) To: Mark H Weaver; +Cc: guix-devel Hello, Mark H Weaver <mhw@netris.org> skribis: > ludo@gnu.org (Ludovic Courtès) writes: [...] >> Currently if test fails, the whole derivation fails, and you can’t >> install your package. If tests were run separately, this would no >> longer hold: you could get your package regardless of whether tests >> fail. > > Indeed, and I agree that we would need to address this. > >> How would you address this? I guess that calls for a new build >> model, no? > > I'm not sure what you mean by "build model", whether you are talking > about the daemon interface or something else, but I think the changes > could be confined to the Guix user interface. A field could be added to > <package>, somewhat similar to 'replacement', but pointing to a package > object which runs tests, or perhaps a list of package objects. The guix > client could simply add the test packages to the list of derivations to > build. This could be inhibited via a "--no-tests" guix build option. > > In typical cases where "make check" expects to be run from a fully built > source directory, the main package would typically produce a tarball of > the built source directory as an additional output. The test package > would simply unpack this tarball and run "make check" in it. > > Regarding the build order: ideally, we would run the tests for package > FOO immediately after building FOO, or at least before building anything > that depends on FOO, to avoid wasted work in case the tests fail. I'm > not sure how best to accomplish this. > > What do you think? I’m afraid doing everything at the level of package objects rather than derivations would lead to something clunky (grafts already suffer from that.) Namely, Guix clients would have to explicitly do a second ‘build-derivations’ RPC once they’re done building packages. If they don’t do that, then tests are not run and you get to install a possibly broken package. Besides, there are lots of details to get right in a proposal like yours. In particular, saving the build tree and for later reuse may prove to be more challenging than it seems. IOW, this looks like a radical change with lots of potential pitfalls, one I’d rather not make before 1.0. I agree that long test suites can be a problem, but for now I think we should focus on package-specific solutions, to the extent possible, such as what we discussed for Git, as well as improvements to our build farm infrastructure. Thoughts? Ludo’. ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: shortening the git test suite 2018-07-07 16:44 ` Mark H Weaver 2018-07-07 21:37 ` Ricardo Wurmus 2018-07-11 12:46 ` Ludovic Courtès @ 2018-07-12 9:27 ` Chris Marusich 2 siblings, 0 replies; 21+ messages in thread From: Chris Marusich @ 2018-07-12 9:27 UTC (permalink / raw) To: Ricardo Wurmus, Mark H Weaver, Ludovic Courtès; +Cc: guix-devel [-- Attachment #1: Type: text/plain, Size: 4684 bytes --] Hi Ricardo, Mark, and Ludo, Ricardo Wurmus <rekado@elephly.net> writes: > What do you think of pushing some package updates only to feature > branches that follow a certain naming convention (e.g. “_update-foo” for > updating the “foo” package), which causes Cuirass to build them and > merge the branch into “master” (semi-)automatically when the build is > successful? If the goal is to ensure that the substitutes for a newly pulled Guix are available when "guix pull" runs, then wouldn't channels accomplish that goal? I realize we don't have channels yet, though. I think your proposal would be a reasonable improvement. In the case of a branch like "_update-foo", Cuirass would probably need to build not only the package "foo" but also any packages that depend on it, right? Since we're talking about updates to the master branch, this shouldn't cause too many rebuilds, so it shouldn't add much load to the build farm. If we're going to allow Cuirass to merge an update branch into master automatically, it might also make sense to allow Cuirass to automatically rebase the update branch onto master before merging it (to ensure the merge is a fast-forward merge), and delete the update branch after merging it (to prevent clutter in commands like "git branch -av"). Alternatively, we could leave those steps to a human and just have Cuirass send a summary email or make the results available via its web interface. I think that with the right automated tests to protect against silly bugs (like accidentally deleting the master branch), it would probably be safer to have Cuirass automatically merge and clean up after itself, rather than rely on humans to do it. Humans are prone to make mistakes, especially when performing tedious tasks. Mark H Weaver <mhw@netris.org> writes: >> Personally, I think that the SVN tests are non-essential (after all, >> we’re building Git here, we keep running the individual test suites of >> Git and Subversion, and git-svn interop seems like a thing that only >> upstream need to worry about), which is why I made this proposal. > > Why do you say that only upstream needs to worry about it? I would > think that you could say the same thing about almost any test suite, but > there's always the possibility that our particular combination of input > versions, or the unusual aspects of Guix, might break something that > upstream doesn't know about. > > I would think that git-svn interop is something that any user of the > git/svn integration needs to worry about. I agree: just because I don't use a feature doesn't mean that we shouldn't test that feature if we can. We ought to be able to run all the tests. I think Ricardo's proposal would help increase the availability of substitutes, which would make long-running tests more tolerable. ludo@gnu.org (Ludovic Courtès) writes: > Mark H Weaver <mhw@netris.org> skribis: > >> ludo@gnu.org (Ludovic Courtès) writes: > > [...] > >>> Currently if test fails, the whole derivation fails, and you can’t >>> install your package. If tests were run separately, this would no >>> longer hold: you could get your package regardless of whether tests >>> fail. >> >> Indeed, and I agree that we would need to address this. >> >>> How would you address this? I guess that calls for a new build >>> model, no? >> >> I'm not sure what you mean by "build model" [...] > > [...] > > IOW, this looks like a radical change with lots of potential pitfalls, > one I’d rather not make before 1.0. I agree that long test suites can > be a problem, but for now I think we should focus on package-specific > solutions, to the extent possible, such as what we discussed for Git, as > well as improvements to our build farm infrastructure. > > Thoughts? I agree that changes like those seem especially tricky to get right. While I'm here, I'll suggest another far-fetched idea. Perhaps we could rethink the relationship between packages, phases (of build systems), and derivations. Currently, I believe one package corresponds to one derivation, and the build phases are just procedures executed during that derivation. Perhaps we could change this so that a package corresponds to multiple derivations: one derivation per phase. If that were possible, then a package that runs all phases plus the test phase would be able to easily re-use the results of a package that runs the same phases but not the test phase. However, I'm not sure if this really makes sense, and in any case, the details would probably be difficult to design and implement. -- Chris [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 832 bytes --] ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: shortening the git test suite 2018-07-06 21:05 ` Ludovic Courtès 2018-07-07 8:07 ` Gábor Boskovits 2018-07-07 16:44 ` Mark H Weaver @ 2018-07-08 1:29 ` Mike Gerwitz 2 siblings, 0 replies; 21+ messages in thread From: Mike Gerwitz @ 2018-07-08 1:29 UTC (permalink / raw) To: Ludovic Courtès, Mark H. Weaver; +Cc: guix-devel [-- Attachment #1: Type: text/plain, Size: 715 bytes --] On Fri, Jul 06, 2018 at 23:05:04 +0200, Ludovic Courtès wrote: > Instead of ‘git-minimal’, we could have ‘git’ without SVN support, and > have a separate ‘git-svn’ package. We can probably arrange for that > ‘git-svn’ package to only provide the relevant libexec/git-core bits > (git-svn is just a Perl script anyway.) > > Thoughts? I'm a git-svn user, and that sounds fine with me. I agree that's much better to provide a separate package with the SVN tests rather than simply disable SVN tests in a full-featured Git. -- Mike Gerwitz Free Software Hacker+Activist | GNU Maintainer & Volunteer GPG: D6E9 B930 028A 6C38 F43B 2388 FEF6 3574 5E6F 6D05 https://mikegerwitz.com [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 818 bytes --] ^ permalink raw reply [flat|nested] 21+ messages in thread
* Automatic branch merging upon build completion 2018-07-05 9:21 ` Ricardo Wurmus 2018-07-06 9:46 ` Gábor Boskovits 2018-07-06 18:19 ` Mark H Weaver @ 2018-07-06 21:09 ` Ludovic Courtès 2 siblings, 0 replies; 21+ messages in thread From: Ludovic Courtès @ 2018-07-06 21:09 UTC (permalink / raw) To: Ricardo Wurmus; +Cc: guix-devel Hi Ricardo, Ricardo Wurmus <rekado@elephly.net> skribis: > What do you think of pushing some package updates only to feature > branches that follow a certain naming convention (e.g. “_update-foo” for > updating the “foo” package), which causes Cuirass to build them and > merge the branch into “master” (semi-)automatically when the build is > successful? FWIW I’m all for it, at least for feature updates—security updates should still go directly to master. That’s a bit of work, though. I wonder if we could reuse similar features available in GitLab-style tools. Ludo’. ^ permalink raw reply [flat|nested] 21+ messages in thread
end of thread, other threads:[~2018-07-12 9:27 UTC | newest] Thread overview: 21+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2018-06-30 19:34 shortening the git test suite Ricardo Wurmus 2018-06-30 23:52 ` Gábor Boskovits 2018-07-01 13:54 ` Marius Bakke 2018-07-05 9:28 ` Ricardo Wurmus 2018-07-02 4:56 ` Mark H Weaver 2018-07-02 8:39 ` Gábor Boskovits 2018-07-05 9:28 ` Ricardo Wurmus 2018-07-05 8:44 ` Chris Marusich 2018-07-05 9:21 ` Ricardo Wurmus 2018-07-06 9:46 ` Gábor Boskovits 2018-07-06 18:19 ` Mark H Weaver 2018-07-06 21:05 ` Ludovic Courtès 2018-07-07 8:07 ` Gábor Boskovits 2018-07-07 16:44 ` Mark H Weaver 2018-07-07 21:37 ` Ricardo Wurmus 2018-07-08 22:41 ` Mark H Weaver 2018-07-09 7:47 ` Ricardo Wurmus 2018-07-11 12:46 ` Ludovic Courtès 2018-07-12 9:27 ` Chris Marusich 2018-07-08 1:29 ` Mike Gerwitz 2018-07-06 21:09 ` Automatic branch merging upon build completion Ludovic Courtès
Code repositories for project(s) associated with this public inbox https://git.savannah.gnu.org/cgit/guix.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).