[bug#58365] [PATCH 0/6] Support #:tests? in guile-build-system

[Maxime Devos <maximedevos@telenet.be>]

[-- Attachment #1.1.1: Type: text/plain, Size: 4905 bytes --]



On 18-10-2022 14:36, Ludovic Courtès wrote:
> Hello,
> 
> Maxime Devos <maximedevos@telenet.be> skribis:
> 
>> A copy is made of test-driver.scm to avoid potentially surprising rebuilds
>> when changes are made.
>>
>> * gnu/packages/guile.scm (guile-test-driver): New variable.
>> * gnu/packages/aux-files/test-driver.scm: New file.
>> * Makefile.am (AUX_FILES): Register it.
>> * build-aux/test-driver.scm: Add a note.
> 
> I very much like the idea of supporting #:tests? in
> ‘guile-build-system’, but I’m skeptical about this approach.
> 
> The reason I’m skeptical is because ‘test-driver.scm’ is written as an
> Automake test driver (it follows the “protocol” defined by Automake)

It is perfectly usable outside Automake, as this patch series 
demonstrates -- it does not depend on any Automake details.

Also, the alternative would be to do "guile -l the-test-suite.scm" 
directly (without a test runner) -- while sometimes this works, 
sometimes the SRFI-64 implementation really cares that there is actually 
some test runner (I got a failure for some unpackaged Scheme library 
when running it without test-driver.scm), and additionally IMO the 
output of "guile -l the-test-suite.scm" isn't very nice, the 
test-driver.scm has much more nice output and is more consistent with 
other packages' test output.

(A third alternative would be to ignore the tests altogether, but 
running tests is good and I have some patches for adding support for the 
tests.)

> for tests that use SRFI-64—

That's the standard Scheme API for test suites in Scheme -- widely 
applicable.  (There's also SRFI-78, but that's currently unsupported by 
Guile, so it's currently not relevant for guile-build-system.)

> that’s a specific kind of tool, and one approach
> to writing test suites in Guile.

It's the only approach to my knowledge, aside from 'let's just put a 
bunch of (when wrong? (error ...)) in a script and do 'guile -l 
the-script.scm' or reimplement our own thing, but that's not a nice 
approach, I would rather make it possible to use the more-standard and 
nicer (even though it's somewhat stateful) SRFI-64.

Also, I don't see what's wrong with 'specific kind of tools' -- should 
we remove 'nml' then because it's super-specific to openttd?  And even 
if there exists another tool that would fit here, why would that matter? 
I mean, the 'gcc' approach to compiling C wasn't rejected in Guix even 
though it's just 'one' approach to compiling C -- for example, 'clang' 
is another approach.

If someone has another approach to running SRFI-64 test suites in 
guile-build-system, they can propose it and it can be discussed which 
one would be better for guile-build-system.

Additionally, a nitpick -- test-driver.scm is an approach to running 
(SRFI-64) test suites, not for writing them, for writing test suites, 
SRFI-64 is an approach.

> I guess all I’m saying is that I doubt this is widely applicable, which
> the diff seems to confirm:
> 
>>   4 files changed, 326 insertions(+), 2 deletions(-)

I don't how you get from the diff that it isn't widely applicable -- it 
is applicable to every package that has a SRFI-64 test suite and uses 
guile-build-system.  I just didn't check every guile-build-system thing 
for whether it has a SRFI-64 test suite.  And once more Scheme libraries 
are packaged (especially Scheme libraries not written with Guile in 
mind, where gnu-build-system is likely inapplicable), presumably we 
would get more packages where this patch series can run tests.

Additionally, have you considered that one possible reason for lack of 
SRFI-64 test suites in guile-build-system packages, is that previously 
there was no good method to run those test suites, so (for someone using 
Guix and Guile) writing them would be pointless?

> Perhaps at this stage there really isn’t much we can factorize, after
> all.  It would be nice to fix this,

I don't see any problem to fix or anything to factorize.  What is this 
'this' you are referring to?

> but that’s probably work to be done
> upstream—for example, by adding a “guild test” command.

Going by "git log build-aux/test-driver.scm", Guix is the upstream, not 
Guile, though I suppose it could eventually be moved to Guile if desired 
by both.

> We can even
> beta-test such a command as an external project before including it in
> Guile proper. >
> WDYT?

I think it already has seen plenty of testing -- in Guix, in 
scheme-gnunet, I think I've also seen it used in the GnuTLS bindings, 
likely by most other Guile programs that use gnu-build-system and have a 
test suite as well.

I mean, it existed since 2015 according to "git log", that's 7 years of 
testing, seems plenty to me, and it also has been used by various projects.

Greetings,
Maxime.

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 929 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]