bug#72369: srfi-64: test-end fails to signal an error with null runner

bug#72369: srfi-64: test-end fails to signal an error with null runner

[Tomas Volf]

Hello,

I think I found a bug in (srfi srfi-64) module shipped with GNU Guile.

The specification says the following about the test-end:

> An error is reported if the suite-name does not match the current test group
> name.

Thus the following should signal an error:

    (use-modules (srfi srfi-64))
    (let ((r (test-runner-null)))
      (test-runner-current r)
      (test-begin "x")
      (test-end   "y"))

However it does not.

Have a nice day
Tomas Volf

bug#72369: srfi-64: test-end fails to signal an error with null runner

[Taylan Kammer]

On 30.07.2024 21:51, Tomas Volf wrote:
> Hello,
>
> I think I found a bug in (srfi srfi-64) module shipped with GNU Guile.
>
> The specification says the following about the test-end:
>
>> An error is reported if the suite-name does not match the current test group
>> name.
> Thus the following should signal an error:
>
>     (use-modules (srfi srfi-64))
>     (let ((r (test-runner-null)))
>       (test-runner-current r)
>       (test-begin "x")
>       (test-end   "y"))
>
> However it does not.
>
> Have a nice day
> Tomas Volf

This would be easy to change, but the on-bad-end-name handler would be kind of useless if test-end was hardcoded to always raise an error. I think the intended meaning of the spec is that the default/simple test runner reports an error in this case (by implementing the on-bad-end-name handler) but not test-end itself.

One could argue that "reporting" an error is not the same thing as signaling/raising one. We could make test-end always print something to stderr, but not actually raise an error, so it technically fulfills the spec's promise that it "reports" an error, but the usefulness of this is unclear to me.

Opinions welcome.

- Taylan