test-equal: actual returned value is #f when tested expression raises execption

test-equal: actual returned value is #f when tested expression raises execption

[Jérémy Korwin-Zmijowski]

Dear Schemers,

As a Guile user, I rely mostly on srfi-64 to write tests.

Recently, a Guile fellow pointed out a strange behavior from one of my
code :

Say I write a test suite :

   ;; char-sets-test.scm

   (use-modules (srfi srfi-64)
                (char-sets))

   (test-begin "harness-char-sets")

   (test-equal "empty password is not valid"
     #f
     (password-valid? ""))

   (test-end "harness-char-sets")

Running `guile -L . char-sets-test.scm` in the file location will
produce the following output :

   $ guile -L . char-sets-test.scm 
   ;;; note: auto-compilation is enabled, set GUILE_AUTO_COMPILE=0
   ;;;       or pass the --no-auto-compile argument to disable.
   ;;; compiling /tmp/char-sets-test.scm
   ;;; WARNING: compilation of /tmp/char-sets-test.scm failed:
   ;;; no code for module (char-sets)
   Backtrace:
              9 (primitive-load "/tmp/char-sets-test.scm")
   In ice-9/eval.scm:
      721:20  8 (primitive-eval (use-modules (srfi srfi-64) (char-
   sets)))
   In ice-9/psyntax.scm:
     1241:36  7 (expand-top-sequence ((use-modules (srfi srfi-64) (#)))
   …)
     1233:19  6 (parse _ (("placeholder" placeholder)) ((top) #(# # …))
   …)
      285:10  5 (parse _ (("placeholder" placeholder)) (()) _ c&e
   (eval) …)
   In ice-9/boot-9.scm:
     3898:20  4 (process-use-modules _)
      222:29  3 (map1 (((srfi srfi-64)) ((char-sets))))
      222:17  2 (map1 (((char-sets))))
     3899:31  1 (_ ((char-sets)))
      3300:6  0 (resolve-interface (char-sets) #:select _ #:hide _ # _
   # …)

   ice-9/boot-9.scm:3300:6: In procedure resolve-interface:
   no code for module (char-sets)

All good so far. Then I create the `(char-sets)` module in a file next
to the test file :

   ;; char-sets.scm

   (define-module (char-sets))

And now, I rerun the tests :

$ guile -L . char-sets-test.scm 
;;; note: auto-compilation is enabled, set GUILE_AUTO_COMPILE=0
;;;       or pass the --no-auto-compile argument to disable.
;;; compiling /tmp/char-sets-test.scm
;;; compiling ./char-sets.scm
;;; compiled /home/jeko/.cache/guile/ccache/3.0-LE-8-4.4/tmp/char-
sets.scm.go
;;; char-sets-test.scm:10:2: warning: possibly unbound variable
`password-valid?'
;;; compiled /home/jeko/.cache/guile/ccache/3.0-LE-8-4.4/tmp/char-sets-
test.scm.go
%%%% Starting test harness-char-sets  (Writing full log to "harness-
char-sets.log")
# of expected passes      1

Here, the result of the test feel weird to me. As the tested procedure
is not defined I was expecting the test to fail.

Is there a way to get a failing test in such situation ? I fear to miss
things like those and so build non working softwares.

Cheers,

Jérémy

Re: test-equal: actual returned value is #f when tested expression raises execption

[divoplade]

Hello,

This is what I understand from guile. I am no expert, so maybe it is
not 100% accurate!

Le samedi 20 février 2021 à 12:03 +0100, Jérémy Korwin-Zmijowski a
écrit :
> 
> Running `guile -L . char-sets-test.scm` in the file location will
> produce the following output :
> 
>    $ guile -L . char-sets-test.scm 
>    ;;; note: auto-compilation is enabled, set GUILE_AUTO_COMPILE=0
>    ;;;       or pass the --no-auto-compile argument to disable.
>    ;;; compiling /tmp/char-sets-test.scm
>    ;;; WARNING: compilation of /tmp/char-sets-test.scm failed:
>    ;;; no code for module (char-sets)

From now on, char-sets-test.scm has been compiled, but guile did not
find password-valid?, so it assumed that this function will be
available at run time when needed. Maybe some crazy macro expansion,
who knows?
> 
> And now, I rerun the tests :
> 
> $ guile -L . char-sets-test.scm 
> ;;; note: auto-compilation is enabled, set GUILE_AUTO_COMPILE=0
> ;;;       or pass the --no-auto-compile argument to disable.
> ;;; compiling /tmp/char-sets-test.scm
> ;;; compiling ./char-sets.scm
> ;;; compiled /home/jeko/.cache/guile/ccache/3.0-LE-8-4.4/tmp/char-
> sets.scm.go
> ;;; char-sets-test.scm:10:2: warning: possibly unbound variable
> `password-valid?'
> ;;; compiled /home/jeko/.cache/guile/ccache/3.0-LE-8-4.4/tmp/char-
> sets-
> test.scm.go
> %%%% Starting test harness-char-sets  (Writing full log to "harness-
> char-sets.log")
> # of expected passes      1

Guile did not re-compile the test, because the code did not change, but
remembered from last time that password-valid? was potentially missing,
so it issues the warning again when loading the bytecode. However,
since now there's a char-sets.scm, it can use the module and find the
function, and everything works correctly.

Everything works because guile does not inline code from a module to
another. So the warning is just a warning. You don't need to worry.

Re: test-equal: actual returned value is #f when tested expression raises execption

[Jérémy Korwin-Zmijowski]

Hi Divoplade,

Thank you for your explanation. I might not have put enough emphase on my issue.

When you said:

"However,
since now there's a char-sets.scm, it can use the module and find the
function, and everything works correctly."

My concern here is that I did not defined the procedure in the module. It's empty.

Jérémy

Le 20 février 2021 12:26:46 GMT+01:00, divoplade <d@divoplade.fr> a écrit :
>Hello,
>
>This is what I understand from guile. I am no expert, so maybe it is
>not 100% accurate!
>
>Le samedi 20 février 2021 à 12:03 +0100, Jérémy Korwin-Zmijowski a
>écrit :
>> 
>> Running `guile -L . char-sets-test.scm` in the file location will
>> produce the following output :
>> 
>>    $ guile -L . char-sets-test.scm 
>>    ;;; note: auto-compilation is enabled, set GUILE_AUTO_COMPILE=0
>>    ;;;       or pass the --no-auto-compile argument to disable.
>>    ;;; compiling /tmp/char-sets-test.scm
>>    ;;; WARNING: compilation of /tmp/char-sets-test.scm failed:
>>    ;;; no code for module (char-sets)
>
>From now on, char-sets-test.scm has been compiled, but guile did not
>find password-valid?, so it assumed that this function will be
>available at run time when needed. Maybe some crazy macro expansion,
>who knows?
>> 
>> And now, I rerun the tests :
>> 
>> $ guile -L . char-sets-test.scm 
>> ;;; note: auto-compilation is enabled, set GUILE_AUTO_COMPILE=0
>> ;;;       or pass the --no-auto-compile argument to disable.
>> ;;; compiling /tmp/char-sets-test.scm
>> ;;; compiling ./char-sets.scm
>> ;;; compiled /home/jeko/.cache/guile/ccache/3.0-LE-8-4.4/tmp/char-
>> sets.scm.go
>> ;;; char-sets-test.scm:10:2: warning: possibly unbound variable
>> `password-valid?'
>> ;;; compiled /home/jeko/.cache/guile/ccache/3.0-LE-8-4.4/tmp/char-
>> sets-
>> test.scm.go
>> %%%% Starting test harness-char-sets  (Writing full log to "harness-
>> char-sets.log")
>> # of expected passes      1
>
>Guile did not re-compile the test, because the code did not change, but
>remembered from last time that password-valid? was potentially missing,
>so it issues the warning again when loading the bytecode. However,
>since now there's a char-sets.scm, it can use the module and find the
>function, and everything works correctly.
>
>Everything works because guile does not inline code from a module to
>another. So the warning is just a warning. You don't need to worry.
>

-- 
Envoyé de mon appareil Android avec Courriel K-9 Mail. Veuillez excuser ma brièveté.

Re: test-equal: actual returned value is #f when tested expression raises execption

[Taylan Kammer]

On 20.02.2021 13:09, Jérémy Korwin-Zmijowski wrote:
> My concern here is that I did not defined the procedure in the module. 
> It's empty.

I can reproduce this on Guile 2.2 (don't have access to 3.0 right now) 
and in my opinion it's a bug.

I've also checked what my own SRFI-64 implementation does in this case, 
and it reports the test as failed.

In my not-so-humble opinion, the reference implementation of SRFI-64, 
which is what Guile ships with, is really terrible.  The code is 
unreadable and unsurprisingly you end up having bugs.  It also doesn't 
fully conform to its own specification.

My implementation is written as an R7RS module, which should work out of 
the box with Guile 3.0 I suppose, since it supports R7RS.  With earlier 
versions of Guile, you can make it work like this:

   (import (srfi srfi-1)
           (srfi srfi-9)
           (srfi srfi-11)
           (srfi srfi-35)
           (rnrs exceptions (6)))

   (load "path/to/scheme-srfis/srfi/64/test-runner.body.scm")
   (load "path/to/scheme-srfis/srfi/64/test-runner-simple.body.scm")
   (load "path/to/scheme-srfis/srfi/64/source-info.body.scm")
   (load "path/to/scheme-srfis/srfi/64/execution.body.scm")

The last line will produce a warning about possibly wrong number of 
arguments to 'eval', but you can safely ignore that.

After this, use the regular SRFI-64 forms from the specification.

Here's the scheme-srfis repo that contains the implementation:

   https://github.com/TaylanUB/scheme-srfis

In the README you can see a couple notes about the SRFI-64 of this 
project.  It offers two extensions that aren't in the standard.

- Taylan