Caching test results separately?

Caching test results separately?

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 588 bytes --]

Hi,

A large part of the build time is being consumed by tests … could we
separate those phases so a package whose tests succeeded once does not
have to be rebuild just because its package got garbage collected?

(⇒ keep the test result (boolean) longer than the build result)

Keeping just a boolean that says whether the tests for a given hash
succeeded could make that pretty cheap and it could avoid half the build
time for packages that got garbage collected before.

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein,
ohne es zu merken.
draketo.de

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: Caching test results separately?

[Josselin Poiret]

[-- Attachment #1: Type: text/plain, Size: 1211 bytes --]

Hi,

"Dr. Arne Babenhauserheide" <arne_bab@web.de> writes:

> Hi,
>
> A large part of the build time is being consumed by tests … could we
> separate those phases so a package whose tests succeeded once does not
> have to be rebuild just because its package got garbage collected?
>
> (⇒ keep the test result (boolean) longer than the build result)
>
> Keeping just a boolean that says whether the tests for a given hash
> succeeded could make that pretty cheap and it could avoid half the build
> time for packages that got garbage collected before.

As it stands it's really not possible, as

1) testing is part of the build process itself and
2) we can't look-up any stateful info like this from the building
process (of course!)

But I would really like for tests to move out of build phases, the
advantages would be twofold: less build time for a lot of packages, and
better environment management for tests (we could rely on better
interaction with other packages, more complicated mocking, use linux
namespaces to their fullest extent, etc.).  That would require a huge
change to Guix though, so it's more of a dream than anything concrete.

Best,
-- 
Josselin Poiret

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 682 bytes --]

Re: Caching test results separately?

[Ryan Prior]

------- Original Message -------
On Monday, March 13th, 2023 at 10:21 PM, Josselin Poiret <dev@jpoiret.xyz> wrote:

> But I would really like for tests to move out of build phases

I've mentioned this previously in IRC as well. Fundamentally, it strikes me as wrong that a change which only affects tests, leaving the installed package the same byte for byte, should require a full package re-build.

An improvement would be to treat tests, and their pass/fail status, are metadata separate from package builds. In Eelco's thesis, testing is described as a major motivator for nix: but in his construction, nix's role is to produce package builds, and then you run tests on the outputs to decide whether to promote a build into production or not. We should use Guix to create testing, staging and production deployments, not to gatekeep and prevent packages with failing tests from building.

Many packages I've created or upgraded had some or all tests disabled because they require a network stack, eg a client library for postgres requires a postgres server. This should be easy to model using Guile and Guix. I would love to contribute to development in this direction.

Ryan

Towards generalized testing (Was: Caching test results separately?)

[Felix Lechner via Development of GNU Guix and the GNU System distribution.]

Hi Josselin,

On Mon, Mar 13, 2023 at 3:21 PM Josselin Poiret <dev@jpoiret.xyz> wrote:
>
> I would really like for tests to move out of build phases
>
> That would require a huge change to Guix though

Fortunately, there is a great precedent for that work in Debian. It is
called autopkgtest. [1]

One package I maintained there, called Lintian, had a comprehensive
test suite that mimicked large parts of Debian including building
packages. When I moved the tests out of the build process, it
shortened the build time from six hours to seven minutes.

That was on Gitlab with external runners provided by a donor.

The key to understanding Debian's autopkgtest process is that, while
the test suites are distributed together with the sources, they
actually test the *installed* versions of the software (which are
loaded separately). That crucial distinction catches important
user-facing errors that were previously undetectable, such as
installation in the wrong path.

We already have many good starting points in Guix. Some truly great
work has been done. [2]

There are also two SRFIs [3][4] although I am personally biased toward
TAP (the "Test-Anything Protocol") [5] and already used that in one of
my Guile modules. [6]

Either way, such testing efforts will make our distribution even
better, especially as we support additional architectures. I would be
eager to cooperate if there is an extra seat at the table.

Kind regards,
Felix

[1] https://people.debian.org/~eriberto/README.package-tests.html
[2] https://luis-felipe.gitlab.io/guile-proba/manual/
[3] https://srfi.schemers.org/srfi-64/srfi-64.html
[4] https://srfi.schemers.org/srfi-78/srfi-78.html
[5] https://codeberg.org/ft/guile-tap
[6] https://codeberg.org/lechner/guile-naptcha/src/branch/history/scm/test/naptcha/pseudo-random.scm

Re: Caching test results separately?

[Simon Tournier]

Hi,

On Mon, 13 Mar 2023 at 23:21, Josselin Poiret <dev@jpoiret.xyz> wrote:

>> (⇒ keep the test result (boolean) longer than the build result)

[...]

> As it stands it's really not possible, as
>
> 1) testing is part of the build process itself and
> 2) we can't look-up any stateful info like this from the building
> process (of course!)

Indeed, the builder derivation is not the same and so the item is not
the same.

--8<---------------cut here---------------start------------->8---
$ guix build hello --no-grafts
/gnu/store/s5pd3rnzymliafb4la5sca63j86xs0y0-hello-2.12.1

$ guix build hello --no-grafts --without-tests=hello
/gnu/store/0h3d2z53xx2idy6pnqa8k781hlf40zmx-hello-2.12.1
--8<---------------cut here---------------end--------------->8---

Compare,
    /gnu/store/s5y2w04jiydki5wb0zb9189x88v5288s-hello-2.12.1-builder
    /gnu/store/d5asvb7mla28mf93hjrk6fffnx14n8b1-hello-2.12.1-builder
    
               (quote
                ())
               #:out-of-source? #f #:tests? #t 

               (quote
                ())
               #:out-of-source? #f #:tests? #f 



Last, note the self reference,

--8<---------------cut here---------------start------------->8---
$ diff -r --no-dereference             \
      $(guix build hello --no-grafts)  \
      $(guix build hello --no-grafts --without-tests=hello)
Binary files
/gnu/store/s5pd3rnzymliafb4la5sca63j86xs0y0-hello-2.12.1/bin/hello and
/gnu/store/0h3d2z53xx2idy6pnqa8k781hlf40zmx-hello-2.12.1/bin/hello
differ

$ grep s5pd3rnzymliafb4la5sca63j86xs0y0 $(guix build hello --no-grafts)/bin/hello
grep: /gnu/store/s5pd3rnzymliafb4la5sca63j86xs0y0-hello-2.12.1/bin/hello: binary file matches

$ grep 0h3d2z53xx2idy6pnqa8k781hlf40zmx $(guix build hello --no-grafts --without-tests=hello)/bin/hello
grep: /gnu/store/0h3d2z53xx2idy6pnqa8k781hlf40zmx-hello-2.12.1/bin/hello: binary file matches
--8<---------------cut here---------------end--------------->8---


> But I would really like for tests to move out of build phases, the
> advantages would be twofold: less build time for a lot of packages, and
> better environment management for tests (we could rely on better
> interaction with other packages, more complicated mocking, use linux
> namespaces to their fullest extent, etc.).  That would require a huge
> change to Guix though, so it's more of a dream than anything concrete.

I agree:

 + it would simplify many things if we were able to run or re-run
   specific phases – Nix does that I guess.
 + it would be huge complex change. :-)

Well, similarly of the ’without-tests’ transformation, maybe we could
imagine another transformation for just run the test suite.  I mean, it
would not solve Arne’s issue but the converse: sometimes a package
requiring many compilation is failing because of one test and debugging
that is for some cases painful.


Cheers,
simon

Re: Caching test results separately?

[Ludovic Courtès]

Hello,

Josselin Poiret <dev@jpoiret.xyz> skribis:

> As it stands it's really not possible, as
>
> 1) testing is part of the build process itself and
> 2) we can't look-up any stateful info like this from the building
> process (of course!)
>
> But I would really like for tests to move out of build phases, the
> advantages would be twofold: less build time for a lot of packages, and
> better environment management for tests (we could rely on better
> interaction with other packages, more complicated mocking, use linux
> namespaces to their fullest extent, etc.).  That would require a huge
> change to Guix though, so it's more of a dream than anything concrete.

I don’t think that’s really possible in practice.

But… there’s another idea I’ve been willing to work on for some time:
recognizing build results that are identical modulo references, such
that we can replace full rebuilds by mere reference rewriting in some
cases.

Examples of derivations that lead the same result modulo references:
building a package with/without tests, building a package with various
test suite adjustments, building a package with different versions of
Bash or whatever interpreter is used by its build scripts, etc.

I’m kinda stuck by the fact I don’t want to add more C++ code though,
which means we first need to get that daemon in Scheme…

Ludo’.