bug#72383: srfi-64: test-runner-reset clobbers the run list

bug#72383: srfi-64: test-runner-reset clobbers the run list

[Tomas Volf]

Hello,

I think I found a bug in (srfi srfi-64) module shipped with GNU Guile.

Test runner is specified as:

> A test-runner is an object that runs a test-suite, and manages the state. The
> test group path, and the sets skip and expected-fail specifiers are part of
> the test-runner. A test-runner will also typically accumulate statistics about
> executed tests.

Based on this description, run list is *not* supposed to be part of the test
runner state, therefore test-runner-reset should not touch it.  However that
does not seem to be the case:

    (use-modules (srfi srfi-64))
    (test-apply (test-match-name "t-a")
                (λ ()
                  (test-begin "x")
                  (test-assert "t-a" (pk 't-a))
                  (test-assert "t-b" (pk 't-b))

                  (test-runner-reset (test-runner-current))

                  (test-begin "x")
                  (test-assert "t-a" (pk 't-a))
                  (test-assert "t-b" (pk 't-b))))

Prints:

    %%%% Starting test x  (Writing full log to "x.log")

    ;;; (t-a)
    %%%% Starting test x  (Writing full log to "x.log")

    ;;; (t-a)

    ;;; (t-b)

Notice that in the second part, t-b is executed despite not supposed to be on
the run list.

Have a nice day
Tomas Volf

bug#72383: srfi-64: test-runner-reset clobbers the run list

[Taylan Kammer]

On 30.07.2024 21:51, Tomas Volf wrote:
> Hello,
>
> I think I found a bug in (srfi srfi-64) module shipped with GNU Guile.
>
> Test runner is specified as:
>
>> A test-runner is an object that runs a test-suite, and manages the state. The
>> test group path, and the sets skip and expected-fail specifiers are part of
>> the test-runner. A test-runner will also typically accumulate statistics about
>> executed tests.
> Based on this description, run list is *not* supposed to be part of the test
> runner state, therefore test-runner-reset should not touch it.  However that
> does not seem to be the case:
>
>     (use-modules (srfi srfi-64))
>     (test-apply (test-match-name "t-a")
>                 (λ ()
>                   (test-begin "x")
>                   (test-assert "t-a" (pk 't-a))
>                   (test-assert "t-b" (pk 't-b))
>
>                   (test-runner-reset (test-runner-current))
>
>                   (test-begin "x")
>                   (test-assert "t-a" (pk 't-a))
>                   (test-assert "t-b" (pk 't-b))))
>
> Prints:
>
>     %%%% Starting test x  (Writing full log to "x.log")
>
>     ;;; (t-a)
>     %%%% Starting test x  (Writing full log to "x.log")
>
>     ;;; (t-a)
>
>     ;;; (t-b)
>
> Notice that in the second part, t-b is executed despite not supposed to be on
> the run list.
>
> Have a nice day
> Tomas Volf
>
>
I agree. The run-list is an implementation detail that only exists for test-apply, isn't used anywhere else, isn't exposed to the user, and is never mentioned in the spec. It shouldn't be possible for the user to ever affect it directly.

I've fixed this in my implementation of SRFI-64 as follows:

    https://codeberg.org/taylan/scheme-srfis/commit/ec07393f809edd249748a884641fb2e63292da79

- Taylan