unit tests best practice

unit tests best practice

[Catonano]

Wat's the best practice to instrument a Guile based project for running
unit tests ?

guile-git has this fragment in the Makefile.am file

SCM_LOG_DRIVER =                                \
  $(top_builddir)/pre-inst-env                  \
  $(GUILE) --no-auto-compile -e main            \
      $(top_srcdir)/build-aux/test-driver.scm

as you can see, what Automake produces calls a "main" procedure in a
"test-driver.scm" script in the "build-aux" folder

The effect is that you can call

make check

in the root folder and have the tests run

Instead, in the Makefile.am file produced by guile-hall wired projects you
ave this fragment

AM_TESTS_ENVIRONMENT = abs_top_srcdir=\"$(abs_top_srcdir)\"
SCM_LOG_COMPILER = $(top_builddir)/test-env $(GUILE)
AM_SCM_LOG_FLAGS = --no-auto-compile -L \"$(top_srcdir)\"

so there's a SCM_LOG_COMPILER instead of a SCM_LOG_DRIVER

What is this LOG_COMPILER doing, exactly ?

This is what the test-env file contains

#!/bin/sh

\"@abs_top_builddir@/pre-inst-env\" \"$@\"

exit $?

what is this ?

What does this do ?

Alex admits he goes full cargo culting when dealing with the Autotools, in
fact there's this comment line at the beginning of the section producing
the Autotools related files in guile-hall

;;;; Full on cargo cult!

So, if there's any kind soul here willing to clarify/illustrate what is
going on we could make the effort to condensate such knowledge in
guile-hall

Improving the experience in dealing with Guile based project would be a
great progress

One last thing

test-driver uses the API provided by srfi-64 to actually run the tests and
display the results

Does Guile provide a standard version of such script ?
Should Guile provide one ?

Re: unit tests best practice

[Mike Gran]

On Thu, Jan 03, 2019 at 05:53:33PM +0100, Catonano wrote:
> Wat's the best practice to instrument a Guile based project for running
> unit tests ?
> 
> guile-git has this fragment in the Makefile.am file
> 
> SCM_LOG_DRIVER =                                \
>   $(top_builddir)/pre-inst-env                  \
>   $(GUILE) --no-auto-compile -e main            \
>       $(top_srcdir)/build-aux/test-driver.scm
> 
> as you can see, what Automake produces calls a "main" procedure in a
> "test-driver.scm" script in the "build-aux" folder
> 
> The effect is that you can call
> 
> make check
> 
> in the root folder and have the tests run
> 
> Instead, in the Makefile.am file produced by guile-hall wired projects you
> ave this fragment
> 
> AM_TESTS_ENVIRONMENT = abs_top_srcdir=\"$(abs_top_srcdir)\"
> SCM_LOG_COMPILER = $(top_builddir)/test-env $(GUILE)
> AM_SCM_LOG_FLAGS = --no-auto-compile -L \"$(top_srcdir)\"
> 
> so there's a SCM_LOG_COMPILER instead of a SCM_LOG_DRIVER
> 
> What is this LOG_COMPILER doing, exactly ?

OK, this is a muddle, but, it goes like this.

LOG_DRIVER and LOG_COMPILER come from Automake.  If you use them,
Automake will generate "make check" rules for all the test scripts in
TESTS.

When you use LOG_COMPILER, Automake considers a test to pass if it
returns an exit status of 0, perhaps by calling '(exit 0)' in Guile.
Exit code 1 = failure. 77 = skip.  99 = fatal eror.  So your TESTS
scripts as run by the command line in LOG_COMPILER need to return
proper exit codes.  Automake will also automatically gather anything
your tests print into .log files and .trs (test result files).

If you use LOG_DRIVER, Automake expects your scripts to manually
create its .log and .trs files.  You use LOG_DRIVER if the simple
exit-code-based pass/fail criteria isn't good enough.  Look
in the Automake docs under 'Custom Test Drivers' for info.

> 
> This is what the test-env file contains
> 
> #!/bin/sh
> 
> \"@abs_top_builddir@/pre-inst-env\" \"$@\"
> 
> exit $?
> 
> what is this ?
> 
> What does this do ?

Typically these pre-inst-env scripts set up the GUILE_LOAD_PATH for
when you are running the tests in the development tree before
installing them.  (There is no standard pre-inst-env script; there are
a few different versions.)  GUILE_LOAD_PATH works better with absolute
paths.  It may also set compiled paths, environment variables, or
LD_LIBRARY_PATH.

> 
> Alex admits he goes full cargo culting when dealing with the Autotools, in
> fact there's this comment line at the beginning of the section producing
> the Autotools related files in guile-hall
> 
> ;;;; Full on cargo cult!
> 
> So, if there's any kind soul here willing to clarify/illustrate what is
> going on we could make the effort to condensate such knowledge in
> guile-hall
> 
> Improving the experience in dealing with Guile based project would be a
> great progress
> 
> One last thing
> 
> test-driver uses the API provided by srfi-64 to actually run the tests and
> display the results
> 
> Does Guile provide a standard version of such script ?
> Should Guile provide one ?

I should note that srfi-64 doesn't work out of the box with
LOG_COMPILER because it doesn't return the proper exit status on
failure, either 0 = pass, 1 = fail, 77 on skip, 99 on hard error.  It
isn't that complicated to add that functionality in, and that would be
a great project for someone to do.

For the one project were I tried to make a full test suite
(guile-ncurses), I went with lots of tiny scripts and used the
LOG_COMPILER exit code strategy.  For that I make a small test library
here that converts #t and #f from a procedure into exit codes.

http://git.savannah.gnu.org/cgit/guile-ncurses.git/tree/test/automake-test-lib.scm

Regards,

Mike Gran

Re: unit tests best practice

[Catonano]

Il giorno gio 3 gen 2019 alle ore 19:45 Mike Gran <spk121@yahoo.com> ha
scritto:

> On Thu, Jan 03, 2019 at 05:53:33PM +0100, Catonano wrote:
> > Wat's the best practice to instrument a Guile based project for running
> > unit tests ?
> >
> > guile-git has this fragment in the Makefile.am file
> >
> > SCM_LOG_DRIVER =                                \
> >   $(top_builddir)/pre-inst-env                  \
> >   $(GUILE) --no-auto-compile -e main            \
> >       $(top_srcdir)/build-aux/test-driver.scm
> >
> > as you can see, what Automake produces calls a "main" procedure in a
> > "test-driver.scm" script in the "build-aux" folder
> >
> > The effect is that you can call
> >
> > make check
> >
> > in the root folder and have the tests run
> >
> > Instead, in the Makefile.am file produced by guile-hall wired projects
> you
> > ave this fragment
> >
> > AM_TESTS_ENVIRONMENT = abs_top_srcdir=\"$(abs_top_srcdir)\"
> > SCM_LOG_COMPILER = $(top_builddir)/test-env $(GUILE)
> > AM_SCM_LOG_FLAGS = --no-auto-compile -L \"$(top_srcdir)\"
> >
> > so there's a SCM_LOG_COMPILER instead of a SCM_LOG_DRIVER
> >
> > What is this LOG_COMPILER doing, exactly ?
>
> OK, this is a muddle, but, it goes like this.
>
>
Mike,

thank you for taking the time to clarify

I think the less painful step at this stage would be to make guile-hall
arrange the tests in the same way that guile-git does

Right now it's the other way around but I don't think Alex would complain,
he openly declares his cluelessness in the matter

In this way, guile-hall could provide a working solution for new projects
without too much work in wrapping SRFFIs in order to provide compliance
with Automake

The wrapping could me made in a further step

I opened an issue on the guile-hall repository and I'll propose my solution
there

Alex are you reading us ?
What do you think about this ?

Re: unit tests best practice

[Amirouche Boubekki]

Insightful post for those interested in autotool-fu.

By the way, I have a demo repo with how to setup coverage, if you are
interested ?

Le jeu. 3 janv. 2019 17:54, Catonano <catonano@gmail.com> a écrit :

> Wat's the best practice to instrument a Guile based project for running
> unit tests ?
>
> guile-git has this fragment in the Makefile.am file
>
> SCM_LOG_DRIVER =                                \
>   $(top_builddir)/pre-inst-env                  \
>   $(GUILE) --no-auto-compile -e main            \
>       $(top_srcdir)/build-aux/test-driver.scm
>
> as you can see, what Automake produces calls a "main" procedure in a
> "test-driver.scm" script in the "build-aux" folder
>
> The effect is that you can call
>
> make check
>
> in the root folder and have the tests run
>
> Instead, in the Makefile.am file produced by guile-hall wired projects you
> ave this fragment
>
> AM_TESTS_ENVIRONMENT = abs_top_srcdir=\"$(abs_top_srcdir)\"
> SCM_LOG_COMPILER = $(top_builddir)/test-env $(GUILE)
> AM_SCM_LOG_FLAGS = --no-auto-compile -L \"$(top_srcdir)\"
>
> so there's a SCM_LOG_COMPILER instead of a SCM_LOG_DRIVER
>
> What is this LOG_COMPILER doing, exactly ?
>
> This is what the test-env file contains
>
> #!/bin/sh
>
> \"@abs_top_builddir@/pre-inst-env\" \"$@\"
>
> exit $?
>
> what is this ?
>
> What does this do ?
>
> Alex admits he goes full cargo culting when dealing with the Autotools, in
> fact there's this comment line at the beginning of the section producing
> the Autotools related files in guile-hall
>
> ;;;; Full on cargo cult!
>
> So, if there's any kind soul here willing to clarify/illustrate what is
> going on we could make the effort to condensate such knowledge in
> guile-hall
>
> Improving the experience in dealing with Guile based project would be a
> great progress
>
> One last thing
>
> test-driver uses the API provided by srfi-64 to actually run the tests and
> display the results
>
> Does Guile provide a standard version of such script ?
> Should Guile provide one ?
>

Re: unit tests best practice

[Catonano]

Il giorno sab 5 gen 2019 alle ore 17:49 Amirouche Boubekki <
amirouche.boubekki@gmail.com> ha scritto:

> Insightful post for those interested in autotool-fu.
>
> By the way, I have a demo repo with how to setup coverage, if you are
> interested ?
>

yes, tanks

Re: unit tests best practice

[Amirouche Boubekki]

Le sam. 5 janv. 2019 à 18:12, Catonano <catonano@gmail.com> a écrit :

>
>
> Il giorno sab 5 gen 2019 alle ore 17:49 Amirouche Boubekki <
> amirouche.boubekki@gmail.com> ha scritto:
>
>> Insightful post for those interested in autotool-fu.
>>
>> By the way, I have a demo repo with how to setup coverage, if you are
>> interested ?
>>
>
> yes, tanks
>
>
Here is it https://github.com/a-guile-mind/coverage

Best regards!

Re: unit tests best practice

[Alex Sassmannshausen]

Hi Cato and Mike,

Thanks for the discussion — this is already shedding a ton of light :-)

Catonano writes:

> Il giorno gio 3 gen 2019 alle ore 19:45 Mike Gran <spk121@yahoo.com> ha
> scritto:
>
>> On Thu, Jan 03, 2019 at 05:53:33PM +0100, Catonano wrote:
>> > Wat's the best practice to instrument a Guile based project for running
>> > unit tests ?
>> >
>> > guile-git has this fragment in the Makefile.am file
>> >
>> > SCM_LOG_DRIVER =                                \
>> >   $(top_builddir)/pre-inst-env                  \
>> >   $(GUILE) --no-auto-compile -e main            \
>> >       $(top_srcdir)/build-aux/test-driver.scm
>> >
>> > as you can see, what Automake produces calls a "main" procedure in a
>> > "test-driver.scm" script in the "build-aux" folder
>> >
>> > The effect is that you can call
>> >
>> > make check
>> >
>> > in the root folder and have the tests run
>> >
>> > Instead, in the Makefile.am file produced by guile-hall wired projects
>> you
>> > ave this fragment
>> >
>> > AM_TESTS_ENVIRONMENT = abs_top_srcdir=\"$(abs_top_srcdir)\"
>> > SCM_LOG_COMPILER = $(top_builddir)/test-env $(GUILE)
>> > AM_SCM_LOG_FLAGS = --no-auto-compile -L \"$(top_srcdir)\"
>> >
>> > so there's a SCM_LOG_COMPILER instead of a SCM_LOG_DRIVER
>> >
>> > What is this LOG_COMPILER doing, exactly ?
>>
>> OK, this is a muddle, but, it goes like this.
>>
>>
> Mike,
>
> thank you for taking the time to clarify

Indeed, this is very helpful.

> I think the less painful step at this stage would be to make guile-hall
> arrange the tests in the same way that guile-git does

So, currently, guile-hall does as Mike did: it uses LOG_COMPILER and
expects the test scripts to provide an exit code.

A simple but probably bad way to achieve that is to add:
(exit (= (test-runner-fail-count (test-runner-current)) 0))

to the bottom of srfi-64 test files.  The reason why it might be bad is
that it does not generate nice exit codes, but rather simply non-zero if
any test fails.

I would be happy to have guile-hall support LOG_DRIVER instead/as well.

The best way to achieve that would be to have a template driver script,
and appropriate changes to the Makefile.am generator.

We could then default to TEST_DRIVER, but provide a command line flag to
support LOG_COMPILER.

> Right now it's the other way around but I don't think Alex would complain,
> he openly declares his cluelessness in the matter

Quite.

> In this way, guile-hall could provide a working solution for new projects
> without too much work in wrapping SRFFIs in order to provide compliance
> with Automake

Agreed, TEST_DRIVER would probably be closer to what people would expect.

> The wrapping could me made in a further step

Also agreed.

Alex

Re: unit tests best practice

[Catonano]

Il giorno dom 6 gen 2019 alle ore 16:27 Alex Sassmannshausen <
alex.sassmannshausen@gmail.com> ha scritto:

> Hi Cato and Mike,
>
> Thanks for the discussion — this is already shedding a ton of light :-)
>
> Catonano writes:
>
> > Il giorno gio 3 gen 2019 alle ore 19:45 Mike Gran <spk121@yahoo.com> ha
> > scritto:
> >
> >> On Thu, Jan 03, 2019 at 05:53:33PM +0100, Catonano wrote:
> >> > Wat's the best practice to instrument a Guile based project for
> running
> >> > unit tests ?
> >> >
> >> > guile-git has this fragment in the Makefile.am file
> >> >
> >> > SCM_LOG_DRIVER =                                \
> >> >   $(top_builddir)/pre-inst-env                  \
> >> >   $(GUILE) --no-auto-compile -e main            \
> >> >       $(top_srcdir)/build-aux/test-driver.scm
> >> >
> >> > as you can see, what Automake produces calls a "main" procedure in a
> >> > "test-driver.scm" script in the "build-aux" folder
> >> >
> >> > The effect is that you can call
> >> >
> >> > make check
> >> >
> >> > in the root folder and have the tests run
> >> >
> >> > Instead, in the Makefile.am file produced by guile-hall wired projects
> >> you
> >> > ave this fragment
> >> >
> >> > AM_TESTS_ENVIRONMENT = abs_top_srcdir=\"$(abs_top_srcdir)\"
> >> > SCM_LOG_COMPILER = $(top_builddir)/test-env $(GUILE)
> >> > AM_SCM_LOG_FLAGS = --no-auto-compile -L \"$(top_srcdir)\"
> >> >
> >> > so there's a SCM_LOG_COMPILER instead of a SCM_LOG_DRIVER
> >> >
> >> > What is this LOG_COMPILER doing, exactly ?
> >>
> >> OK, this is a muddle, but, it goes like this.
> >>
> >>
> > Mike,
> >
> > thank you for taking the time to clarify
>
> Indeed, this is very helpful.
>
> > I think the less painful step at this stage would be to make guile-hall
> > arrange the tests in the same way that guile-git does
>
> So, currently, guile-hall does as Mike did: it uses LOG_COMPILER and
> expects the test scripts to provide an exit code.
>
> A simple but probably bad way to achieve that is to add:
> (exit (= (test-runner-fail-count (test-runner-current)) 0))
>
> to the bottom of srfi-64 test files.  The reason why it might be bad is
> that it does not generate nice exit codes, but rather simply non-zero if
> any test fails.
>
> I would be happy to have guile-hall support LOG_DRIVER instead/as well.
>
> The best way to achieve that would be to have a template driver script,
> and appropriate changes to the Makefile.am generator.
>
> We could then default to TEST_DRIVER, but provide a command line flag to
> support LOG_COMPILER.
>
> > Right now it's the other way around but I don't think Alex would
> complain,
> > he openly declares his cluelessness in the matter
>
> Quite.
>

Sorry.

No offense meant.

As for me I do feel clueless and like crawling in the mud, in addressing
these issues


> In this way, guile-hall could provide a working solution for new projects
> > without too much work in wrapping SRFFIs in order to provide compliance
> > with Automake
>
> Agreed, TEST_DRIVER would probably be closer to what people would expect.
>
> > The wrapping could me made in a further step
>
> Also agreed.
>
> Alex
>

I'm glad that you agree.

I provided an example of the "experience" that both solutions offer using a
daydream project of mine as a starting point

I present the thing here
https://gitlab.com/a-sassmannshausen/guile-hall/issues/1#note_130446033

As Mike explained, with the guile-hall provided solution, the Automake
machinery expects return codes and the Guile side doesn't provide them

So running the tests with that infrastructure does succeed but returns a
result that in my view is wrong
Unless I'm doing some mistake here.

I could try to send a patch to let guile-hall use the guil-git method in
the next few days