bug#72384: srfi-64: test-end should not clear fail list

bug#72384: srfi-64: test-end should not clear fail list

[Tomas Volf]

Hello,

I think I found a bug in (srfi srfi-64) module shipped with GNU Guile.

Reading the specification for test-expect-fail I do not see a mandate to clear
expect-fail list on test-end.  test-skip does have such provision, but it is
lacking in the test-expect-fail.  Therefore I think current behavior is wrong:

    (use-modules (srfi srfi-64))

    (test-begin "x")

    (test-begin "group1")
    (test-expect-fail "test-a")
    (test-assert "test-a" #t)
    (pk (test-result-kind))
    (test-end "group1")
    (test-assert "test-a" #t)
    (pk (test-result-kind))

    (test-end)

Leading to:

    ;;; (xpass)

    ;;; (pass)

Have a nice day
Tomas Volf

bug#72384: srfi-64: test-end should not clear fail list

[Taylan Kammer]

[-- Attachment #1: Type: text/plain, Size: 2040 bytes --]

On 30.07.2024 21:52, Tomas Volf wrote:
> Hello,
>
> I think I found a bug in (srfi srfi-64) module shipped with GNU Guile.
>
> Reading the specification for test-expect-fail I do not see a mandate to clear
> expect-fail list on test-end.  test-skip does have such provision, but it is
> lacking in the test-expect-fail.  Therefore I think current behavior is wrong:
>
>     (use-modules (srfi srfi-64))
>
>     (test-begin "x")
>
>     (test-begin "group1")
>     (test-expect-fail "test-a")
>     (test-assert "test-a" #t)
>     (pk (test-result-kind))
>     (test-end "group1")
>     (test-assert "test-a" #t)
>     (pk (test-result-kind))
>
>     (test-end)
>
> Leading to:
>
>     ;;; (xpass)
>
>     ;;; (pass)
>
> Have a nice day
> Tomas Volf
>
>
I'm inclined to see this as an error/omission in the spec itself. It makes sense for test-end to clear the expected-fail list, just like it clears the skip list. If it didn't, it might cause one to accidentally mark tests as "expected failure" that weren't meant to be marked as such. Consider the following:

(test-group "group1"
  (test-expect-fail "test-a")
  (test-assert "test-a" #t))

(test-group "group2"
  (test-assert "test-a" #t))

Since test-group is equivalent to a pair of test-begin/test-end calls, this code would be very "deceptive" if the implicit test-end didn't clear the expected-fail list. After all, the two groups look completely disjoint, and one wouldn't expect any state from the former to implicitly bleed into the latter. Also, if I'm not mistaken, there's not even a way to clear the expected-fail list explicitly.

I actually have some "real-world" code that uses repetitive names in a test suite within different groups, so this isn't just a theoretical issue either:

    https://codeberg.org/taylan/scheme-bytestructures/src/branch/master/run-tests.body.scm

Notice how often the names "ref" and "set" are used.

So, I think the behavior of the reference implementation is correct/desirable here. My implementation of SRFI-64 does the same.

- Taylan

[-- Attachment #2: Type: text/html, Size: 2625 bytes --]