* unit tests best practice @ 2019-01-03 16:53 Catonano 2019-01-03 18:44 ` Mike Gran 2019-01-05 16:49 ` Amirouche Boubekki 0 siblings, 2 replies; 8+ messages in thread From: Catonano @ 2019-01-03 16:53 UTC (permalink / raw) To: Guile User Wat's the best practice to instrument a Guile based project for running unit tests ? guile-git has this fragment in the Makefile.am file SCM_LOG_DRIVER = \ $(top_builddir)/pre-inst-env \ $(GUILE) --no-auto-compile -e main \ $(top_srcdir)/build-aux/test-driver.scm as you can see, what Automake produces calls a "main" procedure in a "test-driver.scm" script in the "build-aux" folder The effect is that you can call make check in the root folder and have the tests run Instead, in the Makefile.am file produced by guile-hall wired projects you ave this fragment AM_TESTS_ENVIRONMENT = abs_top_srcdir=\"$(abs_top_srcdir)\" SCM_LOG_COMPILER = $(top_builddir)/test-env $(GUILE) AM_SCM_LOG_FLAGS = --no-auto-compile -L \"$(top_srcdir)\" so there's a SCM_LOG_COMPILER instead of a SCM_LOG_DRIVER What is this LOG_COMPILER doing, exactly ? This is what the test-env file contains #!/bin/sh \"@abs_top_builddir@/pre-inst-env\" \"$@\" exit $? what is this ? What does this do ? Alex admits he goes full cargo culting when dealing with the Autotools, in fact there's this comment line at the beginning of the section producing the Autotools related files in guile-hall ;;;; Full on cargo cult! So, if there's any kind soul here willing to clarify/illustrate what is going on we could make the effort to condensate such knowledge in guile-hall Improving the experience in dealing with Guile based project would be a great progress One last thing test-driver uses the API provided by srfi-64 to actually run the tests and display the results Does Guile provide a standard version of such script ? Should Guile provide one ? ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: unit tests best practice 2019-01-03 16:53 unit tests best practice Catonano @ 2019-01-03 18:44 ` Mike Gran 2019-01-05 13:28 ` Catonano 2019-01-05 16:49 ` Amirouche Boubekki 1 sibling, 1 reply; 8+ messages in thread From: Mike Gran @ 2019-01-03 18:44 UTC (permalink / raw) To: Catonano; +Cc: Guile User On Thu, Jan 03, 2019 at 05:53:33PM +0100, Catonano wrote: > Wat's the best practice to instrument a Guile based project for running > unit tests ? > > guile-git has this fragment in the Makefile.am file > > SCM_LOG_DRIVER = \ > $(top_builddir)/pre-inst-env \ > $(GUILE) --no-auto-compile -e main \ > $(top_srcdir)/build-aux/test-driver.scm > > as you can see, what Automake produces calls a "main" procedure in a > "test-driver.scm" script in the "build-aux" folder > > The effect is that you can call > > make check > > in the root folder and have the tests run > > Instead, in the Makefile.am file produced by guile-hall wired projects you > ave this fragment > > AM_TESTS_ENVIRONMENT = abs_top_srcdir=\"$(abs_top_srcdir)\" > SCM_LOG_COMPILER = $(top_builddir)/test-env $(GUILE) > AM_SCM_LOG_FLAGS = --no-auto-compile -L \"$(top_srcdir)\" > > so there's a SCM_LOG_COMPILER instead of a SCM_LOG_DRIVER > > What is this LOG_COMPILER doing, exactly ? OK, this is a muddle, but, it goes like this. LOG_DRIVER and LOG_COMPILER come from Automake. If you use them, Automake will generate "make check" rules for all the test scripts in TESTS. When you use LOG_COMPILER, Automake considers a test to pass if it returns an exit status of 0, perhaps by calling '(exit 0)' in Guile. Exit code 1 = failure. 77 = skip. 99 = fatal eror. So your TESTS scripts as run by the command line in LOG_COMPILER need to return proper exit codes. Automake will also automatically gather anything your tests print into .log files and .trs (test result files). If you use LOG_DRIVER, Automake expects your scripts to manually create its .log and .trs files. You use LOG_DRIVER if the simple exit-code-based pass/fail criteria isn't good enough. Look in the Automake docs under 'Custom Test Drivers' for info. > > This is what the test-env file contains > > #!/bin/sh > > \"@abs_top_builddir@/pre-inst-env\" \"$@\" > > exit $? > > what is this ? > > What does this do ? Typically these pre-inst-env scripts set up the GUILE_LOAD_PATH for when you are running the tests in the development tree before installing them. (There is no standard pre-inst-env script; there are a few different versions.) GUILE_LOAD_PATH works better with absolute paths. It may also set compiled paths, environment variables, or LD_LIBRARY_PATH. > > Alex admits he goes full cargo culting when dealing with the Autotools, in > fact there's this comment line at the beginning of the section producing > the Autotools related files in guile-hall > > ;;;; Full on cargo cult! > > So, if there's any kind soul here willing to clarify/illustrate what is > going on we could make the effort to condensate such knowledge in > guile-hall > > Improving the experience in dealing with Guile based project would be a > great progress > > One last thing > > test-driver uses the API provided by srfi-64 to actually run the tests and > display the results > > Does Guile provide a standard version of such script ? > Should Guile provide one ? I should note that srfi-64 doesn't work out of the box with LOG_COMPILER because it doesn't return the proper exit status on failure, either 0 = pass, 1 = fail, 77 on skip, 99 on hard error. It isn't that complicated to add that functionality in, and that would be a great project for someone to do. For the one project were I tried to make a full test suite (guile-ncurses), I went with lots of tiny scripts and used the LOG_COMPILER exit code strategy. For that I make a small test library here that converts #t and #f from a procedure into exit codes. http://git.savannah.gnu.org/cgit/guile-ncurses.git/tree/test/automake-test-lib.scm Regards, Mike Gran ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: unit tests best practice 2019-01-03 18:44 ` Mike Gran @ 2019-01-05 13:28 ` Catonano 2019-01-06 15:27 ` Alex Sassmannshausen 0 siblings, 1 reply; 8+ messages in thread From: Catonano @ 2019-01-05 13:28 UTC (permalink / raw) To: Mike Gran; +Cc: Guile User Il giorno gio 3 gen 2019 alle ore 19:45 Mike Gran <spk121@yahoo.com> ha scritto: > On Thu, Jan 03, 2019 at 05:53:33PM +0100, Catonano wrote: > > Wat's the best practice to instrument a Guile based project for running > > unit tests ? > > > > guile-git has this fragment in the Makefile.am file > > > > SCM_LOG_DRIVER = \ > > $(top_builddir)/pre-inst-env \ > > $(GUILE) --no-auto-compile -e main \ > > $(top_srcdir)/build-aux/test-driver.scm > > > > as you can see, what Automake produces calls a "main" procedure in a > > "test-driver.scm" script in the "build-aux" folder > > > > The effect is that you can call > > > > make check > > > > in the root folder and have the tests run > > > > Instead, in the Makefile.am file produced by guile-hall wired projects > you > > ave this fragment > > > > AM_TESTS_ENVIRONMENT = abs_top_srcdir=\"$(abs_top_srcdir)\" > > SCM_LOG_COMPILER = $(top_builddir)/test-env $(GUILE) > > AM_SCM_LOG_FLAGS = --no-auto-compile -L \"$(top_srcdir)\" > > > > so there's a SCM_LOG_COMPILER instead of a SCM_LOG_DRIVER > > > > What is this LOG_COMPILER doing, exactly ? > > OK, this is a muddle, but, it goes like this. > > Mike, thank you for taking the time to clarify I think the less painful step at this stage would be to make guile-hall arrange the tests in the same way that guile-git does Right now it's the other way around but I don't think Alex would complain, he openly declares his cluelessness in the matter In this way, guile-hall could provide a working solution for new projects without too much work in wrapping SRFFIs in order to provide compliance with Automake The wrapping could me made in a further step I opened an issue on the guile-hall repository and I'll propose my solution there Alex are you reading us ? What do you think about this ? ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: unit tests best practice 2019-01-05 13:28 ` Catonano @ 2019-01-06 15:27 ` Alex Sassmannshausen 2019-01-10 10:41 ` Catonano 0 siblings, 1 reply; 8+ messages in thread From: Alex Sassmannshausen @ 2019-01-06 15:27 UTC (permalink / raw) To: Catonano; +Cc: Guile User Hi Cato and Mike, Thanks for the discussion — this is already shedding a ton of light :-) Catonano writes: > Il giorno gio 3 gen 2019 alle ore 19:45 Mike Gran <spk121@yahoo.com> ha > scritto: > >> On Thu, Jan 03, 2019 at 05:53:33PM +0100, Catonano wrote: >> > Wat's the best practice to instrument a Guile based project for running >> > unit tests ? >> > >> > guile-git has this fragment in the Makefile.am file >> > >> > SCM_LOG_DRIVER = \ >> > $(top_builddir)/pre-inst-env \ >> > $(GUILE) --no-auto-compile -e main \ >> > $(top_srcdir)/build-aux/test-driver.scm >> > >> > as you can see, what Automake produces calls a "main" procedure in a >> > "test-driver.scm" script in the "build-aux" folder >> > >> > The effect is that you can call >> > >> > make check >> > >> > in the root folder and have the tests run >> > >> > Instead, in the Makefile.am file produced by guile-hall wired projects >> you >> > ave this fragment >> > >> > AM_TESTS_ENVIRONMENT = abs_top_srcdir=\"$(abs_top_srcdir)\" >> > SCM_LOG_COMPILER = $(top_builddir)/test-env $(GUILE) >> > AM_SCM_LOG_FLAGS = --no-auto-compile -L \"$(top_srcdir)\" >> > >> > so there's a SCM_LOG_COMPILER instead of a SCM_LOG_DRIVER >> > >> > What is this LOG_COMPILER doing, exactly ? >> >> OK, this is a muddle, but, it goes like this. >> >> > Mike, > > thank you for taking the time to clarify Indeed, this is very helpful. > I think the less painful step at this stage would be to make guile-hall > arrange the tests in the same way that guile-git does So, currently, guile-hall does as Mike did: it uses LOG_COMPILER and expects the test scripts to provide an exit code. A simple but probably bad way to achieve that is to add: (exit (= (test-runner-fail-count (test-runner-current)) 0)) to the bottom of srfi-64 test files. The reason why it might be bad is that it does not generate nice exit codes, but rather simply non-zero if any test fails. I would be happy to have guile-hall support LOG_DRIVER instead/as well. The best way to achieve that would be to have a template driver script, and appropriate changes to the Makefile.am generator. We could then default to TEST_DRIVER, but provide a command line flag to support LOG_COMPILER. > Right now it's the other way around but I don't think Alex would complain, > he openly declares his cluelessness in the matter Quite. > In this way, guile-hall could provide a working solution for new projects > without too much work in wrapping SRFFIs in order to provide compliance > with Automake Agreed, TEST_DRIVER would probably be closer to what people would expect. > The wrapping could me made in a further step Also agreed. Alex ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: unit tests best practice 2019-01-06 15:27 ` Alex Sassmannshausen @ 2019-01-10 10:41 ` Catonano 0 siblings, 0 replies; 8+ messages in thread From: Catonano @ 2019-01-10 10:41 UTC (permalink / raw) To: Alex Sassmannshausen; +Cc: Guile User Il giorno dom 6 gen 2019 alle ore 16:27 Alex Sassmannshausen < alex.sassmannshausen@gmail.com> ha scritto: > Hi Cato and Mike, > > Thanks for the discussion — this is already shedding a ton of light :-) > > Catonano writes: > > > Il giorno gio 3 gen 2019 alle ore 19:45 Mike Gran <spk121@yahoo.com> ha > > scritto: > > > >> On Thu, Jan 03, 2019 at 05:53:33PM +0100, Catonano wrote: > >> > Wat's the best practice to instrument a Guile based project for > running > >> > unit tests ? > >> > > >> > guile-git has this fragment in the Makefile.am file > >> > > >> > SCM_LOG_DRIVER = \ > >> > $(top_builddir)/pre-inst-env \ > >> > $(GUILE) --no-auto-compile -e main \ > >> > $(top_srcdir)/build-aux/test-driver.scm > >> > > >> > as you can see, what Automake produces calls a "main" procedure in a > >> > "test-driver.scm" script in the "build-aux" folder > >> > > >> > The effect is that you can call > >> > > >> > make check > >> > > >> > in the root folder and have the tests run > >> > > >> > Instead, in the Makefile.am file produced by guile-hall wired projects > >> you > >> > ave this fragment > >> > > >> > AM_TESTS_ENVIRONMENT = abs_top_srcdir=\"$(abs_top_srcdir)\" > >> > SCM_LOG_COMPILER = $(top_builddir)/test-env $(GUILE) > >> > AM_SCM_LOG_FLAGS = --no-auto-compile -L \"$(top_srcdir)\" > >> > > >> > so there's a SCM_LOG_COMPILER instead of a SCM_LOG_DRIVER > >> > > >> > What is this LOG_COMPILER doing, exactly ? > >> > >> OK, this is a muddle, but, it goes like this. > >> > >> > > Mike, > > > > thank you for taking the time to clarify > > Indeed, this is very helpful. > > > I think the less painful step at this stage would be to make guile-hall > > arrange the tests in the same way that guile-git does > > So, currently, guile-hall does as Mike did: it uses LOG_COMPILER and > expects the test scripts to provide an exit code. > > A simple but probably bad way to achieve that is to add: > (exit (= (test-runner-fail-count (test-runner-current)) 0)) > > to the bottom of srfi-64 test files. The reason why it might be bad is > that it does not generate nice exit codes, but rather simply non-zero if > any test fails. > > I would be happy to have guile-hall support LOG_DRIVER instead/as well. > > The best way to achieve that would be to have a template driver script, > and appropriate changes to the Makefile.am generator. > > We could then default to TEST_DRIVER, but provide a command line flag to > support LOG_COMPILER. > > > Right now it's the other way around but I don't think Alex would > complain, > > he openly declares his cluelessness in the matter > > Quite. > Sorry. No offense meant. As for me I do feel clueless and like crawling in the mud, in addressing these issues > In this way, guile-hall could provide a working solution for new projects > > without too much work in wrapping SRFFIs in order to provide compliance > > with Automake > > Agreed, TEST_DRIVER would probably be closer to what people would expect. > > > The wrapping could me made in a further step > > Also agreed. > > Alex > I'm glad that you agree. I provided an example of the "experience" that both solutions offer using a daydream project of mine as a starting point I present the thing here https://gitlab.com/a-sassmannshausen/guile-hall/issues/1#note_130446033 As Mike explained, with the guile-hall provided solution, the Automake machinery expects return codes and the Guile side doesn't provide them So running the tests with that infrastructure does succeed but returns a result that in my view is wrong Unless I'm doing some mistake here. I could try to send a patch to let guile-hall use the guil-git method in the next few days ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: unit tests best practice 2019-01-03 16:53 unit tests best practice Catonano 2019-01-03 18:44 ` Mike Gran @ 2019-01-05 16:49 ` Amirouche Boubekki 2019-01-05 17:11 ` Catonano 1 sibling, 1 reply; 8+ messages in thread From: Amirouche Boubekki @ 2019-01-05 16:49 UTC (permalink / raw) To: Catonano; +Cc: Guile User Insightful post for those interested in autotool-fu. By the way, I have a demo repo with how to setup coverage, if you are interested ? Le jeu. 3 janv. 2019 17:54, Catonano <catonano@gmail.com> a écrit : > Wat's the best practice to instrument a Guile based project for running > unit tests ? > > guile-git has this fragment in the Makefile.am file > > SCM_LOG_DRIVER = \ > $(top_builddir)/pre-inst-env \ > $(GUILE) --no-auto-compile -e main \ > $(top_srcdir)/build-aux/test-driver.scm > > as you can see, what Automake produces calls a "main" procedure in a > "test-driver.scm" script in the "build-aux" folder > > The effect is that you can call > > make check > > in the root folder and have the tests run > > Instead, in the Makefile.am file produced by guile-hall wired projects you > ave this fragment > > AM_TESTS_ENVIRONMENT = abs_top_srcdir=\"$(abs_top_srcdir)\" > SCM_LOG_COMPILER = $(top_builddir)/test-env $(GUILE) > AM_SCM_LOG_FLAGS = --no-auto-compile -L \"$(top_srcdir)\" > > so there's a SCM_LOG_COMPILER instead of a SCM_LOG_DRIVER > > What is this LOG_COMPILER doing, exactly ? > > This is what the test-env file contains > > #!/bin/sh > > \"@abs_top_builddir@/pre-inst-env\" \"$@\" > > exit $? > > what is this ? > > What does this do ? > > Alex admits he goes full cargo culting when dealing with the Autotools, in > fact there's this comment line at the beginning of the section producing > the Autotools related files in guile-hall > > ;;;; Full on cargo cult! > > So, if there's any kind soul here willing to clarify/illustrate what is > going on we could make the effort to condensate such knowledge in > guile-hall > > Improving the experience in dealing with Guile based project would be a > great progress > > One last thing > > test-driver uses the API provided by srfi-64 to actually run the tests and > display the results > > Does Guile provide a standard version of such script ? > Should Guile provide one ? > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: unit tests best practice 2019-01-05 16:49 ` Amirouche Boubekki @ 2019-01-05 17:11 ` Catonano 2019-01-06 13:41 ` Amirouche Boubekki 0 siblings, 1 reply; 8+ messages in thread From: Catonano @ 2019-01-05 17:11 UTC (permalink / raw) To: Amirouche Boubekki; +Cc: Guile User Il giorno sab 5 gen 2019 alle ore 17:49 Amirouche Boubekki < amirouche.boubekki@gmail.com> ha scritto: > Insightful post for those interested in autotool-fu. > > By the way, I have a demo repo with how to setup coverage, if you are > interested ? > yes, tanks ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: unit tests best practice 2019-01-05 17:11 ` Catonano @ 2019-01-06 13:41 ` Amirouche Boubekki 0 siblings, 0 replies; 8+ messages in thread From: Amirouche Boubekki @ 2019-01-06 13:41 UTC (permalink / raw) To: Catonano; +Cc: Guile User Le sam. 5 janv. 2019 à 18:12, Catonano <catonano@gmail.com> a écrit : > > > Il giorno sab 5 gen 2019 alle ore 17:49 Amirouche Boubekki < > amirouche.boubekki@gmail.com> ha scritto: > >> Insightful post for those interested in autotool-fu. >> >> By the way, I have a demo repo with how to setup coverage, if you are >> interested ? >> > > yes, tanks > > Here is it https://github.com/a-guile-mind/coverage Best regards! ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2019-01-10 10:41 UTC | newest] Thread overview: 8+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2019-01-03 16:53 unit tests best practice Catonano 2019-01-03 18:44 ` Mike Gran 2019-01-05 13:28 ` Catonano 2019-01-06 15:27 ` Alex Sassmannshausen 2019-01-10 10:41 ` Catonano 2019-01-05 16:49 ` Amirouche Boubekki 2019-01-05 17:11 ` Catonano 2019-01-06 13:41 ` Amirouche Boubekki
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).