From: Maxim Cournoyer <maxim.cournoyer@gmail.com>
To: guile-devel@gnu.org
Cc: Maxim Cournoyer <maxim.cournoyer@gmail.com>
Subject: [PATCH v2] doc: Document SRFI 64.
Date: Fri, 17 Nov 2023 23:42:32 -0500 [thread overview]
Message-ID: <20231118044244.32462-1-maxim.cournoyer@gmail.com> (raw)
In-Reply-To: <20231118035109.29673-1-maxim.cournoyer@gmail.com>
This is an import of the 'Abstract', 'Rationale', and 'Specification'
sections from the upstream specification text, with some manual
adjustment.
* doc/ref/srfi-modules.texi (SRFI 64): New subsection.
---
Changes in v2:
- Fix the category of many definitions
doc/ref/srfi-modules.texi | 829 +++++++++++++++++++++++++++++++++++++-
1 file changed, 822 insertions(+), 7 deletions(-)
diff --git a/doc/ref/srfi-modules.texi b/doc/ref/srfi-modules.texi
index 0cdf56923..eba0a6e23 100644
--- a/doc/ref/srfi-modules.texi
+++ b/doc/ref/srfi-modules.texi
@@ -1,6 +1,6 @@
@c -*-texinfo-*-
@c This is part of the GNU Guile Reference Manual.
-@c Copyright (C) 1996, 1997, 2000-2004, 2006, 2007-2014, 2017, 2018, 2019, 2020
+@c Copyright (C) 1996-1997, 2000-2004, 2006-2014, 2017-2020, 2023
@c Free Software Foundation, Inc.
@c See the file guile.texi for copying conditions.
@@ -55,7 +55,7 @@ get the relevant SRFI documents from the SRFI home page
* SRFI-60:: Integers as bits.
* SRFI-61:: A more general `cond' clause
* SRFI-62:: S-expression comments.
-* SRFI-64:: A Scheme API for test suites.
+* SRFI 64:: A Scheme API for test suites.
* SRFI-67:: Compare procedures
* SRFI-69:: Basic hash tables.
* SRFI-71:: Extended let-syntax for multiple values.
@@ -5289,12 +5289,827 @@ needed to get SRFI-61 itself. Extended @code{cond} is documented in
Starting from version 2.0, Guile's @code{read} supports SRFI-62/R7RS
S-expression comments by default.
-@node SRFI-64
-@subsection SRFI-64 - A Scheme API for test suites.
-@cindex SRFI-64
+@c This SRFI 64 documentation was "snarfed" from upstream specification
+@c HTML document using the 'snarfi' script.
+@node SRFI 64
+@subsection SRFI 64: A Scheme API for test suites
+@cindex SRFI 64
-See @uref{http://srfi.schemers.org/srfi-64/srfi-64.html, the
-specification of SRFI-64}.
+@menu
+* SRFI 64 Abstract::
+* SRFI 64 Rationale::
+* SRFI 64 Writing basic test suites::
+* SRFI 64 Conditonal test-suites and other advanced features::
+* SRFI 64 Test-runner::
+* SRFI 64 Test results::
+* SRFI 64 Writing a new test-runner::
+@end menu
+
+@node SRFI 64 Abstract
+@subsubsection SRFI 64 Abstract
+
+This defines an API for writing @dfn{test suites}, to make it easy to
+portably test Scheme APIs, libraries, applications, and implementations.
+A test suite is a collection of @dfn{test cases} that execute in the
+context of a @dfn{test-runner}. This specification also supports
+writing new test-runners, to allow customization of reporting and
+processing the result of running test suites.
+
+@node SRFI 64 Rationale
+@subsubsection SRFI 64 Rationale
+
+The Scheme community needs a standard for writing test suites. Every
+SRFI or other library should come with a test suite. Such a test suite
+must be portable, without requiring any non-standard features, such as
+modules. The test suite implementation or "runner" need not be
+portable, but it is desirable that it be possible to write a portable
+basic implementation.
+
+There are other testing frameworks written in Scheme, including
+@url{http://schematics.sourceforge.net/schemeunit.html, SchemeUnit}.
+However SchemeUnit is not portable. It is also a bit on the verbose
+side. It would be useful to have a bridge between this framework and
+SchemeUnit so SchemeUnit tests could run under this framework and vice
+versa. There exists also at least one Scheme wrapper providing a Scheme
+interface to the ``standard'' @url{http://www.junit.org/, JUnit} API for
+Java. It would be useful to have a bridge so that tests written using
+this framework can run under a JUnit runner. Neither of these features
+are part of this specification.
+
+This API makes use of implicit dynamic state, including an implicit
+``test runner''. This makes the API convenient and terse to use, but it
+may be a little less elegant and ``compositional'' than using explicit
+test objects, such as JUnit-style frameworks. It is not claimed to
+follow either object-oriented or functional design principles, but I
+hope it is useful and convenient to use and extend.
+
+This proposal allows converting a Scheme source file to a
+test suite by just adding a few macros. You don't have to
+write the entire file in a new form, thus you don't have to
+re-indent it.
+
+All names defined by the API start with the prefix @samp{test-}. All
+function-like forms are defined as syntax. They may be implemented as
+functions or macros or built-ins. The reason for specifying them as
+syntax is to allow specific tests to be skipped without evaluating
+sub-expressions, or for implementations to add features such as printing
+line numbers or catching exceptions.
+
+@node SRFI 64 Writing basic test suites
+@subsubsection SRFI 64 Writing basic test suites
+
+Let's start with a simple example. This is a complete self-contained
+test-suite.
+
+@lisp
+;; Initialize and give a name to a simple testsuite.
+(test-begin "vec-test")
+(define v (make-vector 5 99))
+;; Require that an expression evaluate to true.
+(test-assert (vector? v))
+;; Test that an expression is eqv? to some other expression.
+(test-eqv 99 (vector-ref v 2))
+(vector-set! v 2 7)
+(test-eqv 7 (vector-ref v 2))
+;; Finish the testsuite, and report results.
+(test-end "vec-test")
+@end lisp
+
+This testsuite could be saved in its own source file. Nothing else is
+needed: We do not require any top-level forms, so it is easy to wrap an
+existing program or test to this form, without adding indentation. It
+is also easy to add new tests, without having to name individual tests
+(though that is optional).
+
+Test cases are executed in the context of a @dfn{test runner}, which is
+a object that accumulates and reports test results. This specification
+defines how to create and use custom test runners, but implementations
+should also provide a default test runner. It is suggested (but not
+required) that loading the above file in a top-level environment will
+cause the tests to be executed using an implementation-specified default
+test runner, and @code{test-end} will cause a summary to be displayed in
+an implementation-specified manner.
+
+@subsubheading Simple test-cases
+
+Primitive test cases test that a given condition is true. They may have
+a name. The core test case form is @code{test-assert}:
+
+@deffn {Scheme Syntax} test-assert [test-name] expression
+
+This evaluates the @var{expression}. The test passes if the result is
+true; if the result is false, a test failure is reported. The test also
+fails if an exception is raised, assuming the implementation has a way
+to catch exceptions. How the failure is reported depends on the test
+runner environment. The @var{test-name} is a string that names the test
+case. (Though the @var{test-name} is a string literal in the examples,
+it is an expression. It is evaluated only once.) It is used when
+reporting errors, and also when skipping tests, as described below. It
+is an error to invoke @code{test-assert}if there is no current test
+runner.
+@end deffn
+
+The following forms may be more convenient than using @code{test-assert}
+directly:
+
+@deffn {Scheme Syntax} test-eqv [test-name] expected test-expr
+
+This is equivalent to:
+
+@lisp
+(test-assert [@var{test-name}] (eqv? @var{expected} @var{test-expr}))
+@end lisp
+
+@end deffn
+
+Similarly @code{test-equal} and @code{test-eq} are shorthand for
+@code{test-assert} combined with @code{equal?} or @code{eq?},
+respectively:
+
+@deffn {Scheme Syntax} test-equal [test-name] expected test-expr
+@deffnx {Scheme Syntax} test-eq [test-name] expected test-expr
+
+Here is a simple example:
+
+@lisp
+(define (mean x y) (/ (+ x y) 2.0))
+(test-eqv 4 (mean 3 5))
+@end lisp
+@end deffn
+
+For testing approximate equality of inexact reals
+we can use @code{test-approximate}:
+
+@deffn {Scheme Syntax} test-approximate [test-name] expected test-expr error
+
+This is equivalent to (except that each argument is only evaluated
+once):
+
+@lisp
+(test-assert [test-name]
+ (and (>= test-expr (- expected error))
+ (<= test-expr (+ expected error))))
+@end lisp
+@end deffn
+
+@subsubheading Tests for catching errors
+
+We need a way to specify that evaluation @emph{should} fail. This
+verifies that errors are detected when required.
+
+@deffn {Scheme Syntax} test-error [[test-name] error-type] test-expr
+
+Evaluating @var{test-expr} is expected to signal an error. The kind of
+error is indicated by @var{error-type}.
+
+If the @var{error-type} is left out, or it is @code{#t}, it means "some
+kind of unspecified error should be signaled". For example:
+
+@lisp
+(test-error #t (vector-ref '#(1 2) 9))
+@end lisp
+
+This specification leaves it implementation-defined (or for a future
+specification) what form @var{test-error} may take, though all
+implementations must allow @code{#t}. Some implementations may support
+@url{https://srfi.schemers.org/srfi-35/srfi-35.html, SRFI-35's
+conditions}, but these are only standardized for
+@url{https://srfi.schemers.org/srfi-36/srfi-36.html, SRFI-36's I/O
+conditions}, which are seldom useful in test suites. An implementation
+may also allow implementation-specific ``exception types''. For example
+Java-based implementations may allow the names of Java exception
+classes:
+
+@lisp
+;; Kawa-specific example
+(test-error <java.lang.IndexOutOfBoundsException> (vector-ref '#(1 2) 9))
+@end lisp
+
+An implementation that cannot catch exceptions should skip
+@code{test-error} forms.
+@end deffn
+
+@subsubheading Testing syntax
+
+Testing syntax is tricky, especially if we want to check that invalid
+syntax is causing an error. The following utility function can help:
+
+@deffn {Scheme Procedure} test-read-eval-string string
+
+This function parses @var{string} (using @code{read}) and evaluates the
+result. The result of evaluation is returned from
+@code{test-read-eval-string}. An error is signalled if there are unread
+characters after the @code{read} is done. For example:
+@code{(test-read-eval-string "(+ 3 4)")} @i{evaluates to} @code{7}.
+@code{(test-read-eval-string "(+ 3 4")} @i{signals an error}.
+@code{(test-read-eval-string "(+ 3 4) ")} @i{signals an error}, because
+there is extra ``junk'' (@i{i.e.} a space) after the list is read.
+
+The @code{test-read-eval-string} used in tests:
+
+@lisp
+(test-equal 7 (test-read-eval-string "(+ 3 4)"))
+(test-error (test-read-eval-string "(+ 3"))
+(test-equal #\newline (test-read-eval-string "#\\newline"))
+(test-error (test-read-eval-string "#\\newlin"))
+;; Skip the next 2 tests unless srfi-62 is available.
+(test-skip (cond-expand (srfi-62 0) (else 2)))
+(test-equal 5 (test-read-eval-string "(+ 1 #;(* 2 3) 4)"))
+(test-equal '(x z) (test-read-string "(list 'x #;'y 'z)"))
+@end lisp
+@end deffn
+
+@subsubheading Test groups and paths
+
+A @dfn{test group} is a named sequence of forms containing testcases,
+expressions, and definitions. Entering a group sets the @dfn{test group
+name}; leaving a group restores the previous group name. These are
+dynamic (run-time) operations, and a group has no other effect or
+identity. Test groups are informal groupings: they are neither Scheme
+values, nor are they syntactic forms.
+@c (More formal <q>test suite</q> values are introduced below.)
+A test group may contain nested inner test groups.
+The @dfn{test group path} is a list of the currently-active
+(entered) test group names, oldest (outermost) first.
+
+@deffn {Scheme Syntax} test-begin suite-name [count]
+
+A @code{test-begin} enters a new test group. The @var{suite-name}
+becomes the current test group name, and is added to the end of the test
+group path. Portable test suites should use a string literal for
+@var{suite-name}; the effect of expressions or other kinds of literals
+is unspecified.
+
+@emph{Rationale:} In some ways using symbols would be preferable.
+However, we want human-readable names, and standard Scheme does not
+provide a way to include spaces or mixed-case text in literal symbols.
+
+The optional @var{count} must match the number of test-cases executed by
+this group. (Nested test groups count as a single test case for this
+count.) This extra test may be useful to catch cases where a test
+doesn't get executed because of some unexpected error.
+
+Additionally, if there is no currently executing test runner,
+one is installed in an implementation-defined manner.
+@end deffn
+
+@deffn {Scheme Syntax} test-end [suite-name]
+
+A @code{test-end} leaves the current test group.
+An error is reported if the @var{suite-name} does not
+match the current test group name.
+@c If it does match an earlier name in the test group path, intervening
+@c groups are left.
+
+Additionally, if the matching @code{test-begin}installed a new
+test-runner, then the @code{test-end} will uninstall it, after reporting
+the accumulated test results in an implementation-defined manner.
+@end deffn
+
+@deffn {Scheme Syntax} test-group suite-name decl-or-expr @dots{}
+
+Equivalent to:
+
+@lisp
+(if (not (test-to-skip% (var suite-name)))
+ (dynamic-wind
+ (lambda () (test-begin (var suite-name)))
+ (lambda () (var decl-or-expr) ...)
+ (lambda () (test-end (var suite-name)))))
+@end lisp
+
+This is usually equivalent to executing the @var{decl-or-expr}s
+within the named test group. However, the entire group is skipped
+if it matched an active @code{test-skip} (see later).
+Also, the @code{test-end} is executed in case of an exception.
+@end deffn
+
+@subsubheading Handling set-up and cleanup
+
+@deffn {Scheme Syntax} test-group-with-cleanup suite-name decl-or-expr @dots{} cleanup-form
+
+Execute each of the @var{decl-or-expr} forms in order (as in a
+@var{<body>}), and then execute the @var{cleanup-form}. The latter
+should be executed even if one of a @var{decl-or-expr} forms raises an
+exception (assuming the implementation has a way to catch exceptions).
+
+For example:
+
+@lisp
+(let ((f (open-output-file "log")))
+ (test-group-with-cleanup "test-file"
+ (do-a-bunch-of-tests f)
+ (close-output-port f)))
+@end lisp
+@end deffn
+
+@node SRFI 64 Conditonal test-suites and other advanced features
+@subsubsection SRFI 64 Conditonal test-suites and other advanced features
+
+The following describes features for controlling which tests to execute,
+or specifying that some tests are @emph{expected} to fail.
+
+@subsubheading Test specifiers
+
+Sometimes we want to only run certain tests, or we know that certain
+tests are expected to fail. A @dfn{test specifier} is one-argument
+function that takes a test-runner and returns a boolean. The specifier
+may be run before a test is performed, and the result may control
+whether the test is executed. For convenience, a specifier may also be
+a non-procedure value, which is coerced to a specifier procedure, as
+described below for @var{count} and @var{name}.
+
+A simple example is:
+
+@lisp
+(if (var some-condition) (test-skip 2)) ;; skip next 2 tests
+@end lisp
+
+@deffn {Scheme Procedure} test-match-name name
+
+The resulting specifier matches if the current test name (as returned by
+@code{test-runner-test-name}) is @code{equal?} to @var{name}.
+@end deffn
+
+@deffn {Scheme Syntax} test-match-nth n [count]
+
+This evaluates to a @emph{stateful} predicate: A counter keeps track of
+how many times it has been called. The predicate matches the @var{n}'th
+time it is called (where @code{1} is the first time), and the next
+@samp{(- @var{count} 1)} times, where @var{count} defaults to @code{1}.
+@end deffn
+
+@deffn {Scheme Syntax} test-match-any specifier @dots{}
+
+The resulting specifier matches if any @var{specifier} matches. Each
+@var{specifier} is applied, in order, so side-effects from a later
+@var{specifier} happen even if an earlier @var{specifier} is true.
+@end deffn
+
+@deffn {Scheme Syntax} test-match-all specifier @dots{}
+
+The resulting specifier matches if each @var{specifier} matches. Each
+@var{specifier} is applied, in order, so side-effects from a later
+@var{specifier} happen even if an earlier @var{specifier} is false.
+@end deffn
+
+@var{count} @i{(i.e. an integer)}
+Convenience short-hand for: @samp{(test-match-nth 1 @var{count})}.
+
+@var{name} @i{(i.e. a string)}
+Convenience short-hand for @samp{(test-match-name @var{name})}.
+
+@subsubheading Skipping selected tests
+
+In some cases you may want to skip a test.
+
+@deffn {Scheme Syntax} test-skip specifier
+
+Evaluating @code{test-skip} adds the resulting @var{specifier} to the
+set of currently active skip-specifiers. Before each test (or
+@code{test-group}) the set of active skip-specifiers are applied to the
+active test-runner. If any specifier matches, then the test is skipped.
+
+For convenience, if the @var{specifier} is a string that is syntactic
+sugar for @code{(test-match-name @var{specifier})}. For example:
+
+@lisp
+(test-skip "test-b")
+(test-assert "test-a") ;; executed
+(test-assert "test-b") ;; skipped
+@end lisp
+
+Any skip specifiers introduced by a @code{test-skip} are removed by a
+following non-nested @code{test-end}.
+
+@lisp
+(test-begin "group1")
+(test-skip "test-a")
+(test-assert "test-a") ;; skipped
+(test-end "group1") ;; Undoes the prior test-skip
+(test-assert "test-a") ;; executed
+@end lisp
+@end deffn
+
+@subsubheading Expected failures
+
+Sometimes you know a test case will fail, but you don't have time to or
+can't fix it. Maybe a certain feature only works on certain platforms.
+However, you want the test-case to be there to remind you to fix it.
+You want to note that such tests are expected to fail.
+
+@deffn {Scheme Syntax} test-expect-fail specifier
+
+Matching tests (where matching is defined as in @code{test-skip})
+are expected to fail. This only affects test reporting,
+not test execution. For example:
+
+@lisp
+(test-expect-fail 2)
+(test-eqv ...) ;; expected to fail
+(test-eqv ...) ;; expected to fail
+(test-eqv ...) ;; expected to pass
+@end lisp
+@end deffn
+
+@node SRFI 64 Test-runner
+@subsubsection SRFI 64 Test-runner
+
+A @dfn{test-runner} is an object that runs a test-suite, and manages the
+state. The test group path, and the sets skip and expected-fail
+specifiers are part of the test-runner. A test-runner will also
+typically accumulate statistics about executed tests,
+
+@deffn {Scheme Procedure} test-runner? value
+
+True if and only if @var{value} is a test-runner object.
+@end deffn
+
+@deffn {Scheme Parameter} test-runner-current
+@deffnx {Scheme Parameter} test-runner-current runner
+
+Get or set the current test-runner.
+@end deffn
+
+@deffn {Scheme Procedure} test-runner-get
+
+Same as @code{(test-runner-current)}, but throws an exception if there
+is no current test-runner.
+@end deffn
+
+@deffn {Scheme Procedure} test-runner-simple
+
+Creates a new simple test-runner, that prints errors and a summary on
+the standard output port.
+@end deffn
+
+@deffn {Scheme Procedure} test-runner-null
+
+Creates a new test-runner, that does nothing with the test results.
+This is mainly meant for extending when writing a custom runner.
+@end deffn
+
+@deffn {Scheme Procedure} test-runner-create
+
+Create a new test-runner. Equivalent to @samp{((test-runner-factory))}.
+@end deffn
+
+@deffn {Scheme Parameter} test-runner-factory
+@deffnx {Scheme Parameter} test-runner-factory factory
+
+Get or set the current test-runner factory. A factory is a
+zero-argument function that creates a new test-runner. The default
+value is @code{test-runner-simple}.
+@end deffn
+
+@subsubheading Running specific tests with a specified runner
+
+@deffn {Scheme Procedure} test-apply [runner] specifier @dots{} procedure
+
+Calls @var{procedure} with no arguments using the specified @var{runner}
+as the current test-runner. If @var{runner} is omitted, then
+@code{(test-runner-current)} is used. (If there is no current runner,
+one is created as in @code{test-begin}.) If one or more
+@var{specifier}s are listed then only tests matching the
+@var{specifier}s are executed. A @var{specifier} has the same form as
+one used for @code{test-skip}. A test is executed if it matches any of
+the @var{specifier}s in the @code{test-apply} @emph{and} does not match
+any active @code{test-skip} specifiers.
+@end deffn
+
+@deffn {Scheme Syntax} test-with-runner runner decl-or-expr @dots{}
+
+Executes each @var{decl-or-expr} in order in a context where the current
+test-runner is @var{runner}.
+@end deffn
+
+@node SRFI 64 Test results
+@subsubsection SRFI 64 Test results
+
+Running a test sets various status properties in the current test-runner.
+This can be examined by a custom test-runner,
+or (more rarely) in a test-suite.
+
+@subsubheading Result kind
+
+Running a test may yield one of the following
+status symbols:
+
+@table @asis
+@item @code{'pass}
+The test passed, as expected.
+
+@item @code{'fail}
+The test failed (and was not expected to).
+
+@item @code{'xfail}
+The test failed and was expected to.
+
+@item @code{'xpass}
+The test passed, but was expected to fail.
+
+@item @code{'skip}
+The test was skipped.
+@end table
+
+@deffn {Scheme Procedure} test-result-kind [runner]
+
+Returns one of the above result codes from the most recent tests.
+Returns @code{#f} if no tests have been run yet. If we've started on a
+new test, but don't have a result yet, then the result kind is
+@code{'xfail} if the test is expected to fail, @code{'skip} if the test
+is supposed to be skipped, or @code{#f} otherwise.
+@end deffn
+
+@deffn {Scheme Procedure} test-passed? [runner]
+
+True if the value of @samp{(test-result-kind [@var{runner}])} is one of
+@code{'pass} or @code{'xpass}. This is a convenient shorthand that
+might be useful in a test suite to only run certain tests if the
+previous test passed.
+@end deffn
+
+@subsubheading Test result properties
+
+A test runner also maintains a set of more detailed
+``result@tie{}properties'' associated with the current or most recent
+test. (I.e. the properties of the most recent test are available as
+long as a new test hasn't started.) Each property has a name (a symbol)
+and a value (any value). Some properties are standard or set by the
+implementation; implementations can add more.
+
+@deffn {Scheme Procedure} test-result-ref runner pname [default]
+
+Returns the property value associated with the @var{pname} property name
+(a symbol). If there is no value associated with @var{pname} return
+@var{default}, or @code{#f} if @var{default} isn't specified.
+@end deffn
+
+@deffn {Scheme Syntax} test-result-set! runner pname value
+
+Sets the property value associated with the @var{pname} property name to
+@var{value}. Usually implementation code should call this function, but
+it may be useful for a custom test-runner to add extra properties.
+@end deffn
+
+@deffn {Scheme Procedure} test-result-remove runner pname
+
+Remove the property with the name @var{pname}.
+@end deffn
+
+@deffn {Scheme Procedure} test-result-clear runner
+
+Remove all result properties. The implementation automatically calls
+@code{test-result-clear} at the start of a @code{test-assert} and
+similar procedures.
+@end deffn
+
+@deffn {Scheme Procedure} test-result-alist runner
+
+Returns an association list of the current result properties. It is
+unspecified if the result shares state with the test-runner. The result
+should not be modified; on the other hand, the result may be implicitly
+modified by future @code{test-result-set!} or @code{test-result-remove}
+calls. However, a @code{test-result-clear} does not modify the returned
+alist. Thus you can ``archive'' result objects from previous runs.
+@end deffn
+
+@subsubheading Standard result properties
+
+The set of available result properties is implementation-specific.
+However, it is suggested that the following might be provided:
+@table @asis
+
+@item @code{'result-kind}
+The result kind, as defined previously.
+This is the only mandatory result property.
+@code{(test-result-kind @var{runner})} is equivalent to:
+@code{(test-result-ref @var{runner} 'result-kind)}
+
+@item @code{'source-file}
+@itemx @code{'source-line}
+If known, the location of test statements (such as @code{test-assert})
+in test suite source code.
+
+@item @code{'source-form}
+The source form, if meaningful and known.
+
+@item @code{'expected-value}
+The expected non-error result, if meaningful and known.
+
+@item @code{'expected-error}
+The @var{error-type}specified in a @code{test-error}, if it meaningful and known.
+
+@item @code{'actual-value}
+The actual non-error result value, if meaningful and known.
+
+@item @code{'actual-error}
+The error value, if an error was signalled and it is known.
+The actual error value is implementation-defined.
+@end table
+
+@node SRFI 64 Writing a new test-runner
+@subsubsection SRFI 64 Writing a new test-runner
+
+This section specifies how to write a test-runner. It can be ignored if
+you just want to write test-cases.
+
+@subsubheading Call-back functions
+
+These call-back functions are ``methods'' (in the object-oriented sense)
+of a test-runner. A method @code{test-runner-on-@var{event}} is called
+by the implementation when @var{event} happens.
+
+To define (set) the callback function for @var{event} use the following
+expression. (This is normally done when initializing a test-runner.)
+
+@code{(test-runner-on-@var{event}! @var{runner} @var{event-function})}
+
+An @var{event-function} takes a test-runner argument, and possibly other
+arguments, depending on the @var{event}.
+
+To extract (get) the callback function for @var{event} do this:
+@code{(test-runner-on-@var{event} @var{runner})}
+
+To extract call the callback function for @var{event} use the following
+expression. (This is normally done by the implementation core.)
+@samp{((test-runner-on-@var{event} @var{runner}) @var{runner}
+@var{other-args} @dots{})}.
+
+The following call-back hooks are available.
+
+@deffn {Scheme Procedure} test-runner-on-test-begin runner
+@deffnx {Scheme Procedure} test-runner-on-test-begin! runner on-test-begin-function
+@deffnx {Scheme Procedure} on-test-begin-function runner
+
+The @var{on-test-begin-function} is called at the start of an
+individual testcase, before the test expression (and expected value) are
+evaluated.
+
+@end deffn
+
+@deffn {Scheme Procedure} test-runner-on-test-end runner
+@deffnx {Scheme Procedure} test-runner-on-test-end! runner on-test-end-function
+@deffnx {Scheme Procedure} on-test-end-function runner
+
+The @var{on-test-end-function} is called at the end of an
+individual testcase, when the result of the test is available.
+@end deffn
+
+@deffn {Scheme Procedure} test-runner-on-group-begin runner
+@deffnx {Scheme Procedure} test-runner-on-group-begin! runner on-group-begin-function
+@deffnx {Scheme Procedure} on-group-begin-function runner suite-name count
+
+The @var{on-group-begin-function} is called by a @code{test-begin},
+including at the start of a @code{test-group}. The @var{suite-name} is
+a Scheme string, and @var{count} is an integer or @code{#f}.
+@end deffn
+
+@deffn {Scheme Procedure} test-runner-on-group-end runner
+@deffnx {Scheme Procedure} test-runner-on-group-end! runner on-group-end-function
+@deffnx {Scheme Procedure} on-group-end-function runner
+
+The @var{on-group-end-function} is called by a @code{test-end},
+including at the end of a @code{test-group}.
+@end deffn
+
+@deffn {Scheme Procedure} test-runner-on-bad-count runner
+@deffnx {Scheme Procedure} test-runner-on-bad-count! runner on-bad-count-function
+@deffnx {Scheme Procedure} on-bad-count-function runner actual-count expected-count
+
+Called from @code{test-end} (before the @var{on-group-end-function} is
+called) if an @var{expected-count} was specified by the matching
+@code{test-begin} and the @var{expected-count} does not match the
+@var{actual-count} of tests actually executed or skipped.
+@end deffn
+
+@deffn {Scheme Procedure} test-runner-on-bad-end-name runner
+@deffnx {Scheme Procedure} test-runner-on-bad-end-name! runner on-bad-end-name-function
+@deffnx {Scheme Procedure} on-bad-end-name-function runner begin-name end-name
+
+Called from @code{test-end} (before the @var{on-group-end-function} is
+called) if a @var{suite-name} was specified, and it did not that the
+name in the matching @code{test-begin}.
+@end deffn
+
+@deffn {Scheme Procedure} test-runner-on-final runner
+@deffnx {Scheme Procedure} test-runner-on-final! runner on-final-function
+@deffnx {Scheme Procedure} on-final-function runner
+
+The @var{on-final-function} takes one parameter (a test-runner) and
+typically displays a summary (count) of the tests. The
+@var{on-final-function} is called after called the
+@var{on-group-end-function} correspondiong to the outermost
+@code{test-end}. The default value is @code{test-on-final-simple} which
+writes to the standard output port the number of tests of the various
+kinds.
+@end deffn
+
+The default test-runner returned by @code{test-runner-simple} uses the
+following call-back functions:
+
+@deffn {Scheme Procedure} test-on-test-begin-simple runner
+@deffnx {Scheme Procedure} test-on-test-end-simple runner
+@deffnx {Scheme Procedure} test-on-group-begin-simple runner suite-name count
+@deffnx {Scheme Procedure} test-on-group-end-simple runner
+@deffnx {Scheme Procedure} test-on-bad-count-simple runner actual-count expected-count
+@deffnx {Scheme Procedure} test-on-bad-end-name-simple runner begin-name end-name
+
+You can call those if you want to write your own test-runner.
+@end deffn
+
+@subsubheading Test-runner components
+
+The following functions are for accessing the other components of a
+test-runner. They would normally only be used to write a new
+test-runner or a match-predicate.
+
+@deffn {Scheme Procedure} test-runner-pass-count runner
+
+Returns the number of tests that passed, and were expected to pass.
+@end deffn
+
+@deffn {Scheme Procedure} test-runner-fail-count runner
+
+Returns the number of tests that failed, but were expected to pass.
+@end deffn
+
+@deffn {Scheme Procedure} test-runner-xpass-count runner
+
+Returns the number of tests that passed, but were expected to fail.
+@end deffn
+
+@deffn {Scheme Procedure} test-runner-xfail-count runner
+
+Returns the number of tests that failed, and were expected to pass.
+@end deffn
+
+@deffn {Scheme Procedure} test-runner-skip-count runner
+
+Returns the number of tests or test groups that were skipped.
+@end deffn
+
+@deffn {Scheme Procedure} test-runner-test-name runner
+
+Returns the name of the current test or test group, as a string. During
+execution of @code{test-begin} this is the name of the test group;
+during the execution of an actual test, this is the name of the
+test-case. If no name was specified, the name is the empty string.
+@end deffn
+
+@deffn {Scheme Procedure} test-runner-group-path runner
+
+A list of names of groups we're nested in, with the outermost group
+first.
+@end deffn
+
+@deffn {Scheme Procedure} test-runner-group-stack runner
+
+A list of names of groups we're nested in, with the outermost group
+last. (This is more efficient than @code{test-runner-group-path}, since
+it doesn't require any copying.)
+@end deffn
+
+@deffn {Scheme Procedure} test-runner-aux-value runner
+@deffnx {Scheme Procedure} test-runner-aux-value! runner on-test
+
+Get or set the @code{aux-value} field of a test-runner. This field is
+not used by this API or the @code{test-runner-simple} test-runner, but
+may be used by custom test-runners to store extra state.
+@end deffn
+
+@deffn {Scheme Procedure} test-runner-reset runner
+
+Resets the state of the @var{runner} to its initial state.
+@end deffn
+
+@subsubheading Example
+
+This is an example of a simple custom test-runner.
+Loading this program before running a test-suite will install
+it as the default test runner.
+
+@lisp
+(define (my-simple-runner filename)
+ (let ((runner (test-runner-null))
+ (port (open-output-file filename))
+ (num-passed 0)
+ (num-failed 0))
+ (test-runner-on-test-end! runner
+ (lambda (runner)
+ (case (test-result-kind runner)
+ ((pass xpass) (set! num-passed (+ num-passed 1)))
+ ((fail xfail) (set! num-failed (+ num-failed 1)))
+ (else #t))))
+ (test-runner-on-final! runner
+ (lambda (runner)
+ (format port "Passing tests: ~d.~%Failing tests: ~d.~%"
+ num-passed num-failed)
+ (close-output-port port)))
+ runner))
+(test-runner-factory
+ (lambda () (my-simple-runner "/tmp/my-test.log")))
+@end lisp
@node SRFI-67
@subsection SRFI-67 - Compare procedures
base-commit: 75cd95060fb1ea7586f0e4b9081694c6d61f1d3b
--
2.41.0
next prev parent reply other threads:[~2023-11-18 4:42 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-11-18 3:50 [PATCH] doc: Document SRFI 64 Maxim Cournoyer
2023-11-18 4:42 ` Maxim Cournoyer [this message]
2023-11-18 8:45 ` [PATCH v2] " Filip Łajszczak
2023-11-21 23:02 ` Maxime Devos
2023-11-22 20:42 ` Maxime Devos
2023-11-26 7:38 ` [semi-OT] Snarfing [was: [PATCH v2] doc: Document SRFI 64.] tomas
2023-11-23 3:55 ` [PATCH v3] doc: Document SRFI 64 Maxim Cournoyer
2023-11-23 4:00 ` [PATCH v2] " Maxim Cournoyer
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://www.gnu.org/software/guile/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20231118044244.32462-1-maxim.cournoyer@gmail.com \
--to=maxim.cournoyer@gmail.com \
--cc=guile-devel@gnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).