Replacing Guile test-suite with SRFI-64?

Replacing Guile test-suite with SRFI-64?

[Bruno Victal]

Hi,

Would a rewrite of the Guile tests using SRFI-64 be a welcome idea?
IMO the advantages of doing so are:

* Less “specifics” that need to be learned solely for working with
Guile. It's easier for people already using SRFI-64 to “bring in” their
experience.

* Standardized/portable. Straightforward to integrate external modules
or SRFI sample implementations into Guile.

* Pre-existing documentation.


Opinions?

-- 
Thanks,
Bruno.

Re: Replacing Guile test-suite with SRFI-64?

[Mike Gran]

> Hi,

> Would a rewrite of the Guile tests using SRFI-64 be a welcome idea?
> IMO the advantages of doing so are:

...

>Opinions?

IMHO, I think Guile should not depend too heavily on Guile for testing itself.
It is bad bootstrapping practice.

And personally, I find that errors in srfi-64 tests are more difficult to interpret
that errors in the guile test suite framework. The logging is worse.


- Mike

Re: Replacing Guile test-suite with SRFI-64?

[Maxime Devos]

[-- Attachment #1.1.1: Type: text/plain, Size: 1482 bytes --]



Op 23-09-2023 om 17:13 schreef Mike Gran:
>> Hi,
> 
>> Would a rewrite of the Guile tests using SRFI-64 be a welcome idea?
>> IMO the advantages of doing so are:
> 
> ...
> 
>> Opinions?
> 
> IMHO, I think Guile should not depend too heavily on Guile for testing itself.
> It is bad bootstrapping practice.

Whether you use Guile's custom test utilities or SRFI-64, in both cases 
you are equally depending on Guile to test itself.

A benefit of SRFI-64 is that the SRFI-64 implementation has tests 
whereas (IIRC) Guile's custom thing doesn't.

Another benefit is that the SRFI-64 implementation is also tested in 
other Scheme implementations, which partially mitigates the ‘tester 
testing itself’ issue (it turns out that it has bugs, see patch by 
Taylan Kammer (*)).

Also, there are not bootstrapping issues, because Guile is built before 
the tests are run (‘make check’ implies ‘make’).

> And personally, I find that errors in srfi-64 tests are more difficult to interpret
> that errors in the guile test suite framework. The logging is worse.

There is a not-yet-applied patch (*) that changes the SRFI 
implementation to be better, but I don't know if it improves error 
messages and logging.

If there is anything concrete you consider worse about error reporting 
in SRFI-64, perhaps the test runner could simply be tweaked to fix that.

(*) https://lists.nongnu.org/archive/html/guile-devel/2021-05/msg00007.html

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 929 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

Re: Replacing Guile test-suite with SRFI-64?

[Mike Gran]

> A benefit of SRFI-64 is that the SRFI-64 implementation has tests 
> whereas (IIRC) Guile's custom thing doesn't.

OK two separate issues.
1. Mike G's gripes about SRFI-64.
2. Updating over 65,000 lines of test code

I don't want to be put in the position of having to make a critique of
SRFI-64, and, once my critique is dissected, make it
appear that that is justification for updating the test suite.

I wouldn't be the decision maker, anyway.

It is 2 MB of test code that has served Guile well so far.
I think the argument for srfi-64 is not self-evident. That
srfi-64 is familiar to some and has a test suite does not prove
that the current test suite is difficult for newcomers to understand
and is buggy.

Respectfully,
-Mike

Re: Replacing Guile test-suite with SRFI-64?

[Maxime Devos]

[-- Attachment #1.1.1: Type: text/plain, Size: 2708 bytes --]

Op 25-09-2023 om 01:32 schreef Mike Gran:
>> A benefit of SRFI-64 is that the SRFI-64 implementation has tests
>> whereas (IIRC) Guile's custom thing doesn't.
> 
> OK two separate issues.
> 1. Mike G's gripes about SRFI-64.
> 2. Updating over 65,000 lines of test code

These aren't separate issues -- if SRFI-64 is worse than what Guile has 
currently and SRFI-64 can't be improved to make it better, then that's a 
good reason

> I don't want to be put in the position of having to make a critique of
> SRFI-64, and, once my critique is dissected, make it
> appear that that is justification for updating the test suite.
> 
> I wouldn't be the decision maker, anyway.
This last sentence is irrelevant.

> It is 2 MB of test code that has served Guile well so far.

Nobody claimed that these 2MB hasn't served Guile well.  Instead, the 
proposal is about serving Guile better.  That things might be going well 
currently doesn't mean we can't do better.

Also, I dispute that it has served Guile well.  It is difficult to 
select which tests to run (it is easier to delete all the other tests 
than to find/remember the documentation stating how to do this), whereas 
with typical SRFI-64 test runners you can just set 
TESTS=some-scheme-file.scm.  Also test-suite/README is rather obsolete.

A more general problem is that Guile's test library is undocumented.

> I think the argument for srfi-64 is not self-evident. 

Nobody claimed it is self-evident -- instead, Bruno and I gave some 
explanation on how SRFI-64 can be better and didn't say ‘SRFI-64 better 
because duh SRFI-64, self-evident, no?’. (And conversely, you pointed 
out some problems with the current SRFI-64.)

> That
> srfi-64 is familiar to some and has a test suite does not prove
> that the current test suite is difficult for newcomers to understand
> and is buggy.

This message is weird -- you already made a (short) critique about 
SRFI-64, and nobody claimed ‘has test suite -> current test suite is 
difficult to understand / buggy’.  To be clear, this critique is the 
following message.

> IMHO, I think Guile should not depend too heavily on Guile for testing itself.
> It is bad bootstrapping practice.
> 
> And personally, I find that errors in srfi-64 tests are more difficult to interpret
> that errors in the guile test suite framework. The logging is worse.

And if it turns out to be justification for updating the test suite, 
then how would that be a bad thing?  After all, it would be justified.

(Also, which test suite are you talking about -- the test suite of 
SRFI-64, or Guile's test suite in its entirety?)

Best regards,
Maxime Devos.

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 929 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

Re: Replacing Guile test-suite with SRFI-64?

[Taylan Kammer]

Re. issues with the standard SRFI 64 implementation, let me point out
that I've written an alternative implementation with cleaner code and
behavior of the default test runner; it's part of this (currently
dormant) project:

https://github.com/TaylanUB/scheme-srfis

(I've not yet had time to migrate away from GitHub. Sorry.)

If I remember correctly, it also runs slightly faster.

To use with Guile, just add the root dir of the repo to the load path
and import (srfi srfi-64) as usual; it should override the default one
if the Guile load path is set up correctly.

To make sure you're not overriding other SRFIs offered by Guile,
remove all but the following files:

srfi/64/*
srfi/64.sld
srfi/srfi-64.sld

Or copy just those into a different directory and add that to your
Guile load path.  You could also rename 'srfi-64.sld' so as not to
override the default SRFI 64, and I think it should still work
under the changed name so long as srfi/64/* aren't renamed.

-- 
Taylan

Re: Replacing Guile test-suite with SRFI-64?

[Bruno Victal]

Hi Mike,

On 2023-09-23 16:13, Mike Gran wrote:
> And personally, I find that errors in srfi-64 tests are more difficult to interpret
> that errors in the guile test suite framework. The logging is worse.

If I'm not mistaken the logging behavior can be customized by writing
your own SRFI-64 test runner. (as an example Guix does this in [1])


[1]: <https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/build/marionette.scm?id=b2887da40c6cb9d541a9df67aa726694f64d8943#n453>

-- 
Furthermore, I consider that nonfree software must be eradicated.

Cheers,
Bruno.

Re: Replacing Guile test-suite with SRFI-64?

[Taylan Kammer]

On 26.09.2023 10:17, Taylan Kammer wrote:
> 
> https://github.com/TaylanUB/scheme-srfis
> 
> (I've not yet had time to migrate away from GitHub. Sorry.)
> 

Fixed:

https://codeberg.org/taylan/scheme-srfis

-- 
Taylan

Re: Replacing Guile test-suite with SRFI-64?

[Maxim Cournoyer]

Hi,

Bruno Victal <mirai@makinata.eu> writes:

> Hi,
>
> Would a rewrite of the Guile tests using SRFI-64 be a welcome idea?
> IMO the advantages of doing so are:
>
> * Less “specifics” that need to be learned solely for working with
> Guile. It's easier for people already using SRFI-64 to “bring in” their
> experience.
>
> * Standardized/portable. Straightforward to integrate external modules
> or SRFI sample implementations into Guile.
>
> * Pre-existing documentation.

I do not have extensive experience with the current test runner used by
Guile itself, but it sounds like a good idea to standardize to use
SRFI-64, as more people already know that library, and many SRFIs
published,
e.g. <https://github.com/scheme-requests-for-implementation/srfi-209/tree/master>,
come with a SRFI-64 test suite, so it'd be less boilerplate to integrate
them.

Also, I like the idea of dogfooding *the* testing library of
Scheme/Guile.  If it fails short of the current custom testing framework
used by Guile, we should improve it.  This effort will help testing any
Guile/Scheme authored project out there, so is worth it in my opinion.

-- 
Thanks,
Maxim