Re: [RFC]: Skipping rust crate tests by default

[Felix Lechner via "Development of GNU Guix and the GNU System distribution." <guix-devel@gnu.org>]

Hi Ephraim,

On Thu, Oct 05 2023, Efraim Flashner wrote:

> I propose changing the cargo-build-system to have '#:tests? #f' by
> default

At first sight, it appears improper to turn off tests because they
fail. Please allow me to remind everyone that build-time tests cover
only a small proportion of problems actually encountered by users.

Most packaging errors, like improper permissions or the installation of
components in a wrong path, usually go undetected.

One of Debian's solutions to that realization is autopkgtest [1] which
allows maintainers to specify a test bed that tests the *installed*
versions and not the versions in the build trees.

A common strategy for Debian maintainers is to convert the build-time
tests to an autopkgtest suite. That way, folks get the benefit of both.

Unfortunately, the setup of test beds is complicated in Debian, as the
installation of the packages being tested has to take place in
containers. In Guix, package installations are decoupled from the
running system. Guix would make that process a lot easier, faster and
more reliable!

In summary, I believe that #:test? should be turned off for all build
systems. Guix should instead test installed versions like Debian's
autopkgtest.

It would be an extra burden on contributors because such a
'test-installed phase would require more attention. It may be
worthwhile, however, because than packages could be built without
testing them---as Ephraim would like to do here.

In addition, pre-built substitutes could be tested by consumers on their
own systems. The substitutes could even be tested before they become
part of any Guix profile.

For Debian's QA tool Lintian, which I maintained for several years, the
speed-up in the development process was remarkable. As a Perl script,
the build toook seven minutes, while the build-time tests took seven
hours.

The builds were initiated with each commit in Salsa (in an online runner
sponsored by a large company).

Lintian was extreme because thousands of tests replicated much of what
happens daily in Debian. The extreme duration of build-time the tests
also took up further resources in downstream distributions. The tests
ran for each backport and for each derivative, such as Ubuntu.

For Guix, which relies on frequent rebuilds, the speed benefit could be
remarkable. Substitutes could become available for testing in perhaps
half the time.

That being said, old habits die hard. The attachment to build-time tests
is formidable. The people who maintained Lintian after me enabled them
again.

Kind regards
Felix

[1] https://salsa.debian.org/ci-team/autopkgtest/-/blob/master/doc/README.package-tests.rst