* Re: Caching test results separately?
2023-03-13 22:21 ` Josselin Poiret
@ 2023-03-13 22:44 ` Ryan Prior
2023-03-13 23:10 ` Towards generalized testing (Was: Caching test results separately?) Felix Lechner via Development of GNU Guix and the GNU System distribution.
` (2 subsequent siblings)
3 siblings, 0 replies; 6+ messages in thread
From: Ryan Prior @ 2023-03-13 22:44 UTC (permalink / raw)
To: Josselin Poiret; +Cc: Dr. Arne Babenhauserheide, guix-devel
------- Original Message -------
On Monday, March 13th, 2023 at 10:21 PM, Josselin Poiret <dev@jpoiret.xyz> wrote:
> But I would really like for tests to move out of build phases
I've mentioned this previously in IRC as well. Fundamentally, it strikes me as wrong that a change which only affects tests, leaving the installed package the same byte for byte, should require a full package re-build.
An improvement would be to treat tests, and their pass/fail status, are metadata separate from package builds. In Eelco's thesis, testing is described as a major motivator for nix: but in his construction, nix's role is to produce package builds, and then you run tests on the outputs to decide whether to promote a build into production or not. We should use Guix to create testing, staging and production deployments, not to gatekeep and prevent packages with failing tests from building.
Many packages I've created or upgraded had some or all tests disabled because they require a network stack, eg a client library for postgres requires a postgres server. This should be easy to model using Guile and Guix. I would love to contribute to development in this direction.
Ryan
^ permalink raw reply [flat|nested] 6+ messages in thread* Towards generalized testing (Was: Caching test results separately?)
2023-03-13 22:21 ` Josselin Poiret
2023-03-13 22:44 ` Ryan Prior
@ 2023-03-13 23:10 ` Felix Lechner via Development of GNU Guix and the GNU System distribution.
2023-03-14 11:03 ` Caching test results separately? Simon Tournier
2023-03-16 14:08 ` Ludovic Courtès
3 siblings, 0 replies; 6+ messages in thread
From: Felix Lechner via Development of GNU Guix and the GNU System distribution. @ 2023-03-13 23:10 UTC (permalink / raw)
To: Josselin Poiret; +Cc: Dr. Arne Babenhauserheide, Guix Devel
Hi Josselin,
On Mon, Mar 13, 2023 at 3:21 PM Josselin Poiret <dev@jpoiret.xyz> wrote:
>
> I would really like for tests to move out of build phases
>
> That would require a huge change to Guix though
Fortunately, there is a great precedent for that work in Debian. It is
called autopkgtest. [1]
One package I maintained there, called Lintian, had a comprehensive
test suite that mimicked large parts of Debian including building
packages. When I moved the tests out of the build process, it
shortened the build time from six hours to seven minutes.
That was on Gitlab with external runners provided by a donor.
The key to understanding Debian's autopkgtest process is that, while
the test suites are distributed together with the sources, they
actually test the *installed* versions of the software (which are
loaded separately). That crucial distinction catches important
user-facing errors that were previously undetectable, such as
installation in the wrong path.
We already have many good starting points in Guix. Some truly great
work has been done. [2]
There are also two SRFIs [3][4] although I am personally biased toward
TAP (the "Test-Anything Protocol") [5] and already used that in one of
my Guile modules. [6]
Either way, such testing efforts will make our distribution even
better, especially as we support additional architectures. I would be
eager to cooperate if there is an extra seat at the table.
Kind regards,
Felix
[1] https://people.debian.org/~eriberto/README.package-tests.html
[2] https://luis-felipe.gitlab.io/guile-proba/manual/
[3] https://srfi.schemers.org/srfi-64/srfi-64.html
[4] https://srfi.schemers.org/srfi-78/srfi-78.html
[5] https://codeberg.org/ft/guile-tap
[6] https://codeberg.org/lechner/guile-naptcha/src/branch/history/scm/test/naptcha/pseudo-random.scm
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Caching test results separately?
2023-03-13 22:21 ` Josselin Poiret
2023-03-13 22:44 ` Ryan Prior
2023-03-13 23:10 ` Towards generalized testing (Was: Caching test results separately?) Felix Lechner via Development of GNU Guix and the GNU System distribution.
@ 2023-03-14 11:03 ` Simon Tournier
2023-03-16 14:08 ` Ludovic Courtès
3 siblings, 0 replies; 6+ messages in thread
From: Simon Tournier @ 2023-03-14 11:03 UTC (permalink / raw)
To: Josselin Poiret, Dr. Arne Babenhauserheide, guix-devel
Hi,
On Mon, 13 Mar 2023 at 23:21, Josselin Poiret <dev@jpoiret.xyz> wrote:
>> (⇒ keep the test result (boolean) longer than the build result)
[...]
> As it stands it's really not possible, as
>
> 1) testing is part of the build process itself and
> 2) we can't look-up any stateful info like this from the building
> process (of course!)
Indeed, the builder derivation is not the same and so the item is not
the same.
--8<---------------cut here---------------start------------->8---
$ guix build hello --no-grafts
/gnu/store/s5pd3rnzymliafb4la5sca63j86xs0y0-hello-2.12.1
$ guix build hello --no-grafts --without-tests=hello
/gnu/store/0h3d2z53xx2idy6pnqa8k781hlf40zmx-hello-2.12.1
--8<---------------cut here---------------end--------------->8---
Compare,
/gnu/store/s5y2w04jiydki5wb0zb9189x88v5288s-hello-2.12.1-builder
/gnu/store/d5asvb7mla28mf93hjrk6fffnx14n8b1-hello-2.12.1-builder
(quote
())
#:out-of-source? #f #:tests? #t
(quote
())
#:out-of-source? #f #:tests? #f
Last, note the self reference,
--8<---------------cut here---------------start------------->8---
$ diff -r --no-dereference \
$(guix build hello --no-grafts) \
$(guix build hello --no-grafts --without-tests=hello)
Binary files
/gnu/store/s5pd3rnzymliafb4la5sca63j86xs0y0-hello-2.12.1/bin/hello and
/gnu/store/0h3d2z53xx2idy6pnqa8k781hlf40zmx-hello-2.12.1/bin/hello
differ
$ grep s5pd3rnzymliafb4la5sca63j86xs0y0 $(guix build hello --no-grafts)/bin/hello
grep: /gnu/store/s5pd3rnzymliafb4la5sca63j86xs0y0-hello-2.12.1/bin/hello: binary file matches
$ grep 0h3d2z53xx2idy6pnqa8k781hlf40zmx $(guix build hello --no-grafts --without-tests=hello)/bin/hello
grep: /gnu/store/0h3d2z53xx2idy6pnqa8k781hlf40zmx-hello-2.12.1/bin/hello: binary file matches
--8<---------------cut here---------------end--------------->8---
> But I would really like for tests to move out of build phases, the
> advantages would be twofold: less build time for a lot of packages, and
> better environment management for tests (we could rely on better
> interaction with other packages, more complicated mocking, use linux
> namespaces to their fullest extent, etc.). That would require a huge
> change to Guix though, so it's more of a dream than anything concrete.
I agree:
+ it would simplify many things if we were able to run or re-run
specific phases – Nix does that I guess.
+ it would be huge complex change. :-)
Well, similarly of the ’without-tests’ transformation, maybe we could
imagine another transformation for just run the test suite. I mean, it
would not solve Arne’s issue but the converse: sometimes a package
requiring many compilation is failing because of one test and debugging
that is for some cases painful.
Cheers,
simon
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: Caching test results separately?
2023-03-13 22:21 ` Josselin Poiret
` (2 preceding siblings ...)
2023-03-14 11:03 ` Caching test results separately? Simon Tournier
@ 2023-03-16 14:08 ` Ludovic Courtès
3 siblings, 0 replies; 6+ messages in thread
From: Ludovic Courtès @ 2023-03-16 14:08 UTC (permalink / raw)
To: Josselin Poiret; +Cc: Dr. Arne Babenhauserheide, guix-devel
Hello,
Josselin Poiret <dev@jpoiret.xyz> skribis:
> As it stands it's really not possible, as
>
> 1) testing is part of the build process itself and
> 2) we can't look-up any stateful info like this from the building
> process (of course!)
>
> But I would really like for tests to move out of build phases, the
> advantages would be twofold: less build time for a lot of packages, and
> better environment management for tests (we could rely on better
> interaction with other packages, more complicated mocking, use linux
> namespaces to their fullest extent, etc.). That would require a huge
> change to Guix though, so it's more of a dream than anything concrete.
I don’t think that’s really possible in practice.
But… there’s another idea I’ve been willing to work on for some time:
recognizing build results that are identical modulo references, such
that we can replace full rebuilds by mere reference rewriting in some
cases.
Examples of derivations that lead the same result modulo references:
building a package with/without tests, building a package with various
test suite adjustments, building a package with different versions of
Bash or whatever interpreter is used by its build scripts, etc.
I’m kinda stuck by the fact I don’t want to add more C++ code though,
which means we first need to get that daemon in Scheme…
Ludo’.
^ permalink raw reply [flat|nested] 6+ messages in thread