Towards generalized testing (Was: Caching test results separately?)

[Felix Lechner via "Development of GNU Guix and the GNU System distribution." <guix-devel@gnu.org>]

Hi Josselin,

On Mon, Mar 13, 2023 at 3:21 PM Josselin Poiret <dev@jpoiret.xyz> wrote:
>
> I would really like for tests to move out of build phases
>
> That would require a huge change to Guix though

Fortunately, there is a great precedent for that work in Debian. It is
called autopkgtest. [1]

One package I maintained there, called Lintian, had a comprehensive
test suite that mimicked large parts of Debian including building
packages. When I moved the tests out of the build process, it
shortened the build time from six hours to seven minutes.

That was on Gitlab with external runners provided by a donor.

The key to understanding Debian's autopkgtest process is that, while
the test suites are distributed together with the sources, they
actually test the *installed* versions of the software (which are
loaded separately). That crucial distinction catches important
user-facing errors that were previously undetectable, such as
installation in the wrong path.

We already have many good starting points in Guix. Some truly great
work has been done. [2]

There are also two SRFIs [3][4] although I am personally biased toward
TAP (the "Test-Anything Protocol") [5] and already used that in one of
my Guile modules. [6]

Either way, such testing efforts will make our distribution even
better, especially as we support additional architectures. I would be
eager to cooperate if there is an extra seat at the table.

Kind regards,
Felix

[1] https://people.debian.org/~eriberto/README.package-tests.html
[2] https://luis-felipe.gitlab.io/guile-proba/manual/
[3] https://srfi.schemers.org/srfi-64/srfi-64.html
[4] https://srfi.schemers.org/srfi-78/srfi-78.html
[5] https://codeberg.org/ft/guile-tap
[6] https://codeberg.org/lechner/guile-naptcha/src/branch/history/scm/test/naptcha/pseudo-random.scm