issues unit testing with srfi-64

issues unit testing with srfi-64

[Aleix Conchillo Flaqué]

Hi!

happy holidays and almost happy new year to everyone!

I've added unit tests to guile-json and I have found a couple of
issues. This is the branch:

https://github.com/aconchillo/guile-json/tree/switch-to-alist-and-vectors

First one (which might not be a real issue) is that (test-end) doesn't
exit with an exit status different than 0 if a test fails. And this
will cause automake to believe the tests passed when they actually did
not. I have solved this with the following which works for my use
case, but I'm not sure if that's the right way:
   ...
   (test-end "test-builder")

   (exit (if (eqv? 0 (test-runner-fail-count (test-runner-current))) 0 1))

(I could have returned the fail-count directly, but since there are
only 255 exit values I thought it could wrap around.)

The other issue is that the beginning of the log files are corrupted:

%%%% Starting test test-builder  (Writing full log to "test-builder.log")
# of expected passes      19
r.scm"
  source-line: 33
  source-form: (test-equal "1234" (scm->json-string 1234))

This should be something like:

Test begin:
  source-file: "tests/test-builder.scm"
  source-line: 33
  source-form: (test-equal "1234" (scm->json-string 1234))
Test end:
  result-kind: pass
  actual-value: "1234"
  expected-value: "1234"

I don't know how to fix this one yet.

Has anyone else found these issues before?

Thanks in advance!

Aleix

Re: issues unit testing with srfi-64

[Christopher Lam]

Hi
In gnucash I've resorted to writing my own srfi-64 test-runner. See the
entry point at (run-test-proper) at
https://github.com/Gnucash/gnucash/blob/maint/gnucash/report/standard-reports/test/test-transaction.scm

Its definition is at
https://github.com/Gnucash/gnucash/blob/maint/libgnucash/engine/test/srfi64-extras.scm
and takes care of both returning #f if any test fails, and also doing an
IMHO better job at logging tests.

On Sun., 30 Dec. 2018, 08:05 Aleix Conchillo Flaqué <aconchillo@gmail.com
wrote:

> Hi!
>
> happy holidays and almost happy new year to everyone!
>
> I've added unit tests to guile-json and I have found a couple of
> issues. This is the branch:
>
> https://github.com/aconchillo/guile-json/tree/switch-to-alist-and-vectors
>
> First one (which might not be a real issue) is that (test-end) doesn't
> exit with an exit status different than 0 if a test fails. And this
> will cause automake to believe the tests passed when they actually did
> not. I have solved this with the following which works for my use
> case, but I'm not sure if that's the right way:
>    ...
>    (test-end "test-builder")
>
>    (exit (if (eqv? 0 (test-runner-fail-count (test-runner-current))) 0 1))
>
> (I could have returned the fail-count directly, but since there are
> only 255 exit values I thought it could wrap around.)
>
> The other issue is that the beginning of the log files are corrupted:
>
> %%%% Starting test test-builder  (Writing full log to "test-builder.log")
> # of expected passes      19
> r.scm"
>   source-line: 33
>   source-form: (test-equal "1234" (scm->json-string 1234))
>
> This should be something like:
>
> Test begin:
>   source-file: "tests/test-builder.scm"
>   source-line: 33
>   source-form: (test-equal "1234" (scm->json-string 1234))
> Test end:
>   result-kind: pass
>   actual-value: "1234"
>   expected-value: "1234"
>
> I don't know how to fix this one yet.
>
> Has anyone else found these issues before?
>
> Thanks in advance!
>
> Aleix
>
>

Re: issues unit testing with srfi-64

[Aleix Conchillo Flaqué]

That worked great and fixed both of my issues. Thanks!

On Sat, Dec 29, 2018 at 6:17 PM Christopher Lam
<christopher.lck@gmail.com> wrote:
>
> Hi
> In gnucash I've resorted to writing my own srfi-64 test-runner. See the entry point at (run-test-proper) at https://github.com/Gnucash/gnucash/blob/maint/gnucash/report/standard-reports/test/test-transaction.scm
>
> Its definition is at https://github.com/Gnucash/gnucash/blob/maint/libgnucash/engine/test/srfi64-extras.scm and takes care of both returning #f if any test fails, and also doing an IMHO better job at logging tests.
>
> On Sun., 30 Dec. 2018, 08:05 Aleix Conchillo Flaqué <aconchillo@gmail.com wrote:
>>
>> Hi!
>>
>> happy holidays and almost happy new year to everyone!
>>
>> I've added unit tests to guile-json and I have found a couple of
>> issues. This is the branch:
>>
>> https://github.com/aconchillo/guile-json/tree/switch-to-alist-and-vectors
>>
>> First one (which might not be a real issue) is that (test-end) doesn't
>> exit with an exit status different than 0 if a test fails. And this
>> will cause automake to believe the tests passed when they actually did
>> not. I have solved this with the following which works for my use
>> case, but I'm not sure if that's the right way:
>>    ...
>>    (test-end "test-builder")
>>
>>    (exit (if (eqv? 0 (test-runner-fail-count (test-runner-current))) 0 1))
>>
>> (I could have returned the fail-count directly, but since there are
>> only 255 exit values I thought it could wrap around.)
>>
>> The other issue is that the beginning of the log files are corrupted:
>>
>> %%%% Starting test test-builder  (Writing full log to "test-builder.log")
>> # of expected passes      19
>> r.scm"
>>   source-line: 33
>>   source-form: (test-equal "1234" (scm->json-string 1234))
>>
>> This should be something like:
>>
>> Test begin:
>>   source-file: "tests/test-builder.scm"
>>   source-line: 33
>>   source-form: (test-equal "1234" (scm->json-string 1234))
>> Test end:
>>   result-kind: pass
>>   actual-value: "1234"
>>   expected-value: "1234"
>>
>> I don't know how to fix this one yet.
>>
>> Has anyone else found these issues before?
>>
>> Thanks in advance!
>>
>> Aleix
>>

Re: issues unit testing with srfi-64

[Taylan Kammer]

FWIW, I have a SRFI-64 implementation whose code-base and default test
runner is somewhat cleaner than the upstream one:

    https://github.com/TaylanUB/scheme-srfis/tree/master/srfi/64

(All actual code is in the *.body.scm files.  Others are R7RS library
wrappers.)

It sounds like using a custom test runner was the right thing in your
case though, so the default test runner of the implementation is
probably unimportant.

- Taylan