bug#24647: SRFI 64 − Test marked as passed when it should have been marked as fail

bug#24647: SRFI 64 − Test marked as passed when it should have been marked as fail

[Amirouche Boubekki]

The code for SRFI-64 does implement error catching, but that it might 
return #f if an error happens causing a passed test when it should have 
been marked as FAIL

(use-modules (srfi srfi-64))

(test-begin "repository")

(test-equal
     #f
   (throw 'key))

(test-end)

bug#24647: SRFI 64 − Test marked as passed when it should have been marked as fail

[Ludovic Courtès]

Hi,

Amirouche Boubekki <amirouche@hypermove.net> skribis:

> The code for SRFI-64 does implement error catching, but that it might
> return #f if an error happens causing a passed test when it should
> have been marked as FAIL
>
> (use-modules (srfi srfi-64))
>
> (test-begin "repository")
>
> (test-equal
>     #f
>   (throw 'key))
>
> (test-end)

It may look surprising, but I think it’s intended (although the SRFI
document does not explicitly say so.)  First, the generate log file
reads:

--8<---------------cut here---------------start------------->8---
Test begin:
  test-name: "foo"
  source-file: "/home/ludo/src/guile/t.scm"
  source-line: 5
  source-form: (test-equal "foo" #f (throw (quote key)))
Test end:
  result-kind: pass
  actual-value: #f
  actual-error: (key)
  expected-value: #f
--8<---------------cut here---------------end--------------->8---

… which clearly shows that the exception was caught, but that the test
was considered a success because the actual value of #f matches the
expected value.  Second, SRFI-64 uses ‘%test-evaluate-with-catch’, whose
name makes the intent quite clear.

So I think you should be using ‘test-assert’ in this case, like this:

  (test-assert "foo"
    (throw 'key))

This is correctly flagged as a failure.

Thanks,
Ludo’.