From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Maxim Cournoyer Newsgroups: gmane.lisp.guile.devel Subject: [PATCH] doc: Document SRFI 64. Date: Fri, 17 Nov 2023 22:50:29 -0500 Message-ID: <20231118035109.29673-1-maxim.cournoyer@gmail.com> Mime-Version: 1.0 Content-Transfer-Encoding: 8bit Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="16779"; mail-complaints-to="usenet@ciao.gmane.io" Cc: Maxim Cournoyer To: guile-devel@gnu.org Original-X-From: guile-devel-bounces+guile-devel=m.gmane-mx.org@gnu.org Sat Nov 18 04:51:58 2023 Return-path: Envelope-to: guile-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1r4CNB-00047H-6t for guile-devel@m.gmane-mx.org; Sat, 18 Nov 2023 04:51:57 +0100 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1r4CMc-0004su-Nu; Fri, 17 Nov 2023 22:51:22 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1r4CMb-0004sd-B6 for guile-devel@gnu.org; Fri, 17 Nov 2023 22:51:21 -0500 Original-Received: from mail-qt1-x82e.google.com ([2607:f8b0:4864:20::82e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1r4CMX-0006zk-EO for guile-devel@gnu.org; Fri, 17 Nov 2023 22:51:21 -0500 Original-Received: by mail-qt1-x82e.google.com with SMTP id d75a77b69052e-41cc56255e3so15481861cf.3 for ; Fri, 17 Nov 2023 19:51:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1700279475; x=1700884275; darn=gnu.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=y5LvDBKTHXzCJTqGSSTTClhWLQlJ1JqoYi2U6kRQTb8=; b=YcsXthmfncb+iE2Q88d7NmvCpD1ZfMgP+EY1/cljcULsA087wre0mo9SWRlqVt7nZP kKlprfDXvVE2546XnHq7L1wW156Mk2Q+rx2KY4aqIUR46tPeJzOWayXiKSdOFfGYE3ut 1gaXTqS1w+DWwes+BUK/wuEWr2O+twzFxnfNGtcicc2NHw2zV8RGWvv1W3MFpHlC3TgI zgXiG7DcsxS8x0cdnZKnJ29etKMHxRPUQh59aWRoELRSz7WbulQs/oH4eludKl+whQMF WJ1JxOF3C2z8WIKjv4dhh7Ni254+fliMDWDYqQYuLjAp/0H4Ts8A3iaitRJ/cRb3ORpK mBHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700279475; x=1700884275; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=y5LvDBKTHXzCJTqGSSTTClhWLQlJ1JqoYi2U6kRQTb8=; b=WPrUoFHYzcLpw5AlmskC0tkOOAk/9SH0ANouKynYl217Oo1Sqz5FOhV2p3eJEnsruK c2Eo45BnR2Wr5GGOfC3UnyTTD0naKnf/No+0HOoPlKy2n/CwL5TIWuDI7cbOSpJKzOe5 qqWyAMobi8lZehFs5NRXnd3yUW/7CvCV6A1JDRBX9U5aooeasGEX12FAdy+f9hoy2Evx cvlFQ+Sfud4JCI/9Y8IfldmTH1ljXFQszRjyQNEUqxXkIsoBaPoeUO/yFDQC6xGl8r7K Q3+PJgAO1Ctg5KwKyW7EwyBvSQ1FmgXO+Y+Ydv2JZ4SyURPzs6lcsnTw3UJvR0xFBH+b eIqQ== X-Gm-Message-State: AOJu0YzqTg/c5dYXEXcf97qHVrm2R/56NKUP1ZB4SpxdgQQPtyWAqlqe HQ/Ue/a/WsGw22NysF1zhpXnoK0lUy0= X-Google-Smtp-Source: AGHT+IE5BdnB6muXhe2l/wNm+gyl1MpN8j1o3SLGbvUakUAvaQPQe7tjGtPXpe+PAbhsErr0Wk38qw== X-Received: by 2002:ac8:5b95:0:b0:418:16e0:f19e with SMTP id a21-20020ac85b95000000b0041816e0f19emr1940500qta.14.1700279474812; Fri, 17 Nov 2023 19:51:14 -0800 (PST) Original-Received: from localhost.localdomain (dsl-154-55.b2b2c.ca. [66.158.154.55]) by smtp.gmail.com with ESMTPSA id z8-20020ac86b88000000b00421a0b66bd2sm1048812qts.4.2023.11.17.19.51.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 17 Nov 2023 19:51:14 -0800 (PST) X-Mailer: git-send-email 2.41.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::82e; envelope-from=maxim.cournoyer@gmail.com; helo=mail-qt1-x82e.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guile-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Developers list for Guile, the GNU extensibility library" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guile-devel-bounces+guile-devel=m.gmane-mx.org@gnu.org Original-Sender: guile-devel-bounces+guile-devel=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.lisp.guile.devel:22083 Archived-At: This is an import of the 'Abstract', 'Rationale', and 'Specification' sections from the upstream specification text, with some manual adjustments. * doc/ref/srfi-modules.texi (SRFI 64): New subsection. --- doc/ref/srfi-modules.texi | 840 +++++++++++++++++++++++++++++++++++++- 1 file changed, 833 insertions(+), 7 deletions(-) diff --git a/doc/ref/srfi-modules.texi b/doc/ref/srfi-modules.texi index 0cdf56923..ce1c0222a 100644 --- a/doc/ref/srfi-modules.texi +++ b/doc/ref/srfi-modules.texi @@ -1,6 +1,6 @@ @c -*-texinfo-*- @c This is part of the GNU Guile Reference Manual. -@c Copyright (C) 1996, 1997, 2000-2004, 2006, 2007-2014, 2017, 2018, 2019, 2020 +@c Copyright (C) 1996-1997, 2000-2004, 2006-2014, 2017-2020, 2023 @c Free Software Foundation, Inc. @c See the file guile.texi for copying conditions. @@ -55,7 +55,7 @@ get the relevant SRFI documents from the SRFI home page * SRFI-60:: Integers as bits. * SRFI-61:: A more general `cond' clause * SRFI-62:: S-expression comments. -* SRFI-64:: A Scheme API for test suites. +* SRFI 64:: A Scheme API for test suites. * SRFI-67:: Compare procedures * SRFI-69:: Basic hash tables. * SRFI-71:: Extended let-syntax for multiple values. @@ -5289,12 +5289,838 @@ needed to get SRFI-61 itself. Extended @code{cond} is documented in Starting from version 2.0, Guile's @code{read} supports SRFI-62/R7RS S-expression comments by default. -@node SRFI-64 -@subsection SRFI-64 - A Scheme API for test suites. -@cindex SRFI-64 +@c This SRFI 64 documentation was "snarfed" from upstream specification +@c HTML document using the 'snarfi' script. +@node SRFI 64 +@subsection SRFI 64: A Scheme API for test suites +@cindex SRFI 64 -See @uref{http://srfi.schemers.org/srfi-64/srfi-64.html, the -specification of SRFI-64}. +@menu +* SRFI 64 Abstract:: +* SRFI 64 Rationale:: +* SRFI 64 Writing basic test suites:: +* SRFI 64 Conditonal test-suites and other advanced features:: +* SRFI 64 Test-runner:: +* SRFI 64 Test results:: +* SRFI 64 Writing a new test-runner:: +@end menu + +@node SRFI 64 Abstract +@subsubsection SRFI 64 Abstract + +This defines an API for writing @dfn{test suites}, to make it easy to +portably test Scheme APIs, libraries, applications, and implementations. +A test suite is a collection of @dfn{test cases} that execute in the +context of a @dfn{test-runner}. This specification also supports +writing new test-runners, to allow customization of reporting and +processing the result of running test suites. + +@node SRFI 64 Rationale +@subsubsection SRFI 64 Rationale + +The Scheme community needs a standard for writing test suites. Every +SRFI or other library should come with a test suite. Such a test suite +must be portable, without requiring any non-standard features, such as +modules. The test suite implementation or "runner" need not be +portable, but it is desirable that it be possible to write a portable +basic implementation. + +There are other testing frameworks written in Scheme, including +@url{http://schematics.sourceforge.net/schemeunit.html, SchemeUnit}. +However SchemeUnit is not portable. It is also a bit on the verbose +side. It would be useful to have a bridge between this framework and +SchemeUnit so SchemeUnit tests could run under this framework and vice +versa. There exists also at least one Scheme wrapper providing a Scheme +interface to the ``standard'' @url{http://www.junit.org/, JUnit} API for +Java. It would be useful to have a bridge so that tests written using +this framework can run under a JUnit runner. Neither of these features +are part of this specification. + +This API makes use of implicit dynamic state, including an implicit +``test runner''. This makes the API convenient and terse to use, but it +may be a little less elegant and ``compositional'' than using explicit +test objects, such as JUnit-style frameworks. It is not claimed to +follow either object-oriented or functional design principles, but I +hope it is useful and convenient to use and extend. + +This proposal allows converting a Scheme source file to a +test suite by just adding a few macros. You don't have to +write the entire file in a new form, thus you don't have to +re-indent it. + +All names defined by the API start with the prefix @samp{test-}. All +function-like forms are defined as syntax. They may be implemented as +functions or macros or built-ins. The reason for specifying them as +syntax is to allow specific tests to be skipped without evaluating +sub-expressions, or for implementations to add features such as printing +line numbers or catching exceptions. + +@node SRFI 64 Writing basic test suites +@subsubsection SRFI 64 Writing basic test suites + +Let's start with a simple example. This is a complete self-contained +test-suite. + +@lisp +;; Initialize and give a name to a simple testsuite. +(test-begin "vec-test") +(define v (make-vector 5 99)) +;; Require that an expression evaluate to true. +(test-assert (vector? v)) +;; Test that an expression is eqv? to some other expression. +(test-eqv 99 (vector-ref v 2)) +(vector-set! v 2 7) +(test-eqv 7 (vector-ref v 2)) +;; Finish the testsuite, and report results. +(test-end "vec-test") +@end lisp + +This testsuite could be saved in its own source file. Nothing else is +needed: We do not require any top-level forms, so it is easy to wrap an +existing program or test to this form, without adding indentation. It +is also easy to add new tests, without having to name individual tests +(though that is optional). + +Test cases are executed in the context of a @dfn{test runner}, which is +a object that accumulates and reports test results. This specification +defines how to create and use custom test runners, but implementations +should also provide a default test runner. It is suggested (but not +required) that loading the above file in a top-level environment will +cause the tests to be executed using an implementation-specified default +test runner, and @code{test-end} will cause a summary to be displayed in +an implementation-specified manner. + +@subsubheading Simple test-cases + +Primitive test cases test that a given condition is true. They may have +a name. The core test case form is @code{test-assert}: + +@deffn {Scheme Procedure} test-assert [test-name] expression + +This evaluates the @var{expression}. The test passes if the result is +true; if the result is false, a test failure is reported. The test also +fails if an exception is raised, assuming the implementation has a way +to catch exceptions. How the failure is reported depends on the test +runner environment. The @var{test-name} is a string that names the test +case. (Though the @var{test-name} is a string literal in the examples, +it is an expression. It is evaluated only once.) It is used when +reporting errors, and also when skipping tests, as described below. It +is an error to invoke @code{test-assert}if there is no current test +runner. +@end deffn + +The following forms may be more convenient than using @code{test-assert} +directly: + +@deffn {Scheme Procedure} test-eqv [test-name] expected test-expr + +This is equivalent to: + +@lisp +(test-assert [@var{test-name}] (eqv? @var{expected} @var{test-expr})) +@end lisp + +@end deffn + +Similarly @code{test-equal} and @code{test-eq} are shorthand for +@code{test-assert} combined with @code{equal?} or @code{eq?}, +respectively: + +@deffn {Scheme Procedure} test-equal [test-name] expected test-expr +@deffnx {Scheme Procedure} test-eq [test-name] expected test-expr + +Here is a simple example: + +@lisp +(define (mean x y) (/ (+ x y) 2.0)) +(test-eqv 4 (mean 3 5)) +@end lisp +@end deffn + +For testing approximate equality of inexact reals +we can use @code{test-approximate}: + +@deffn {Scheme Procedure} test-approximate [test-name] expected test-expr error + +This is equivalent to (except that each argument is only evaluated +once): + +@lisp +(test-assert [test-name] + (and (>= test-expr (- expected error)) + (<= test-expr (+ expected error)))) +@end lisp +@end deffn + +@subsubheading Tests for catching errors + +We need a way to specify that evaluation @emph{should} fail. This +verifies that errors are detected when required. + +@deffn {Scheme Procedure} test-error [[test-name] error-type] test-expr + +Evaluating @var{test-expr} is expected to signal an error. The kind of +error is indicated by @var{error-type}. + +If the @var{error-type} is left out, or it is @code{#t}, it means "some +kind of unspecified error should be signaled". For example: + +@lisp +(test-error #t (vector-ref '#(1 2) 9)) +@end lisp + +This specification leaves it implementation-defined (or for a future +specification) what form @var{test-error} may take, though all +implementations must allow @code{#t}. Some implementations may support +@url{https://srfi.schemers.org/srfi-35/srfi-35.html, SRFI-35's +conditions}, but these are only standardized for +@url{https://srfi.schemers.org/srfi-36/srfi-36.html, SRFI-36's I/O +conditions}, which are seldom useful in test suites. An implementation +may also allow implementation-specific ``exception types''. For example +Java-based implementations may allow the names of Java exception +classes: + +@lisp +;; Kawa-specific example +(test-error (vector-ref '#(1 2) 9)) +@end lisp + +An implementation that cannot catch exceptions should skip +@code{test-error} forms. +@end deffn + +@subsubheading Testing syntax + +Testing syntax is tricky, especially if we want to check that invalid +syntax is causing an error. The following utility function can help: + +@deffn {Scheme Procedure} test-read-eval-string string + +This function parses @var{string} (using @code{read}) and evaluates the +result. The result of evaluation is returned from +@code{test-read-eval-string}. An error is signalled if there are unread +characters after the @code{read} is done. For example: +@code{(test-read-eval-string "(+ 3 4)")} @i{evaluates to} @code{7}. +@code{(test-read-eval-string "(+ 3 4")} @i{signals an error}. +@code{(test-read-eval-string "(+ 3 4) ")} @i{signals an error}, because +there is extra ``junk'' (@i{i.e.} a space) after the list is read. + +The @code{test-read-eval-string} used in tests: + +@lisp +(test-equal 7 (test-read-eval-string "(+ 3 4)")) +(test-error (test-read-eval-string "(+ 3")) +(test-equal #\newline (test-read-eval-string "#\\newline")) +(test-error (test-read-eval-string "#\\newlin")) +;; Skip the next 2 tests unless srfi-62 is available. +(test-skip (cond-expand (srfi-62 0) (else 2))) +(test-equal 5 (test-read-eval-string "(+ 1 #;(* 2 3) 4)")) +(test-equal '(x z) (test-read-string "(list 'x #;'y 'z)")) +@end lisp +@end deffn + +@subsubheading Test groups and paths + +A @dfn{test group} is a named sequence of forms containing testcases, +expressions, and definitions. Entering a group sets the @dfn{test group +name}; leaving a group restores the previous group name. These are +dynamic (run-time) operations, and a group has no other effect or +identity. Test groups are informal groupings: they are neither Scheme +values, nor are they syntactic forms. +@c (More formal test suite values are introduced below.) +A test group may contain nested inner test groups. +The @dfn{test group path} is a list of the currently-active +(entered) test group names, oldest (outermost) first. + +@deffn {Scheme Procedure} test-begin suite-name [count] + +A @code{test-begin} enters a new test group. The @var{suite-name} +becomes the current test group name, and is added to the end of the test +group path. Portable test suites should use a string literal for +@var{suite-name}; the effect of expressions or other kinds of literals +is unspecified. + +@emph{Rationale:} In some ways using symbols would be preferable. +However, we want human-readable names, and standard Scheme does not +provide a way to include spaces or mixed-case text in literal symbols. + +The optional @var{count} must match the number of test-cases executed by +this group. (Nested test groups count as a single test case for this +count.) This extra test may be useful to catch cases where a test +doesn't get executed because of some unexpected error. + +Additionally, if there is no currently executing test runner, +one is installed in an implementation-defined manner. +@end deffn + +@deffn {Scheme Procedure} test-end [suite-name] + +A @code{test-end} leaves the current test group. +An error is reported if the @var{suite-name} does not +match the current test group name. +@c If it does match an earlier name in the test group path, intervening +@c groups are left. + +Additionally, if the matching @code{test-begin}installed a new +test-runner, then the @code{test-end} will uninstall it, after reporting +the accumulated test results in an implementation-defined manner. +@end deffn + +@deffn {Scheme Procedure} test-group suite-name decl-or-expr @dots{} + +Equivalent to: + +@lisp +(if (not (test-to-skip% (var suite-name))) + (dynamic-wind + (lambda () (test-begin (var suite-name))) + (lambda () (var decl-or-expr) ...) + (lambda () (test-end (var suite-name))))) +@end lisp + +This is usually equivalent to executing the @var{decl-or-expr}s +within the named test group. However, the entire group is skipped +if it matched an active @code{test-skip} (see later). +Also, the @code{test-end} is executed in case of an exception. +@end deffn + +@subsubheading Handling set-up and cleanup + +@deffn {Scheme Procedure} test-group-with-cleanup suite-name decl-or-expr @dots{} cleanup-form + +Execute each of the @var{decl-or-expr} forms in order (as in a +@var{}), and then execute the @var{cleanup-form}. The latter +should be executed even if one of a @var{decl-or-expr} forms raises an +exception (assuming the implementation has a way to catch exceptions). + +For example: + +@lisp +(let ((f (open-output-file "log"))) + (test-group-with-cleanup "test-file" + (do-a-bunch-of-tests f) + (close-output-port f))) +@end lisp +@end deffn + +@node SRFI 64 Conditonal test-suites and other advanced features +@subsubsection SRFI 64 Conditonal test-suites and other advanced features + +The following describes features for controlling which tests to execute, +or specifying that some tests are @emph{expected} to fail. + +@subsubheading Test specifiers + +Sometimes we want to only run certain tests, or we know that certain +tests are expected to fail. A @dfn{test specifier} is one-argument +function that takes a test-runner and returns a boolean. The specifier +may be run before a test is performed, and the result may control +whether the test is executed. For convenience, a specifier may also be +a non-procedure value, which is coerced to a specifier procedure, as +described below for @var{count} and @var{name}. + +A simple example is: + +@lisp +(if (var some-condition) (test-skip 2)) ;; skip next 2 tests +@end lisp + +@deffn {Scheme Procedure} test-match-name name + +The resulting specifier matches if the current test name (as returned by +@code{test-runner-test-name}) is @code{equal?} to @var{name}. +@end deffn + +@deffn {Scheme Procedure} test-match-nth n [count] + +This evaluates to a @emph{stateful} predicate: A counter keeps track of +how many times it has been called. The predicate matches the @var{n}'th +time it is called (where @code{1} is the first time), and the next +@samp{(- @var{count} 1)} times, where @var{count} defaults to @code{1}. +@end deffn + +@deffn {Scheme Procedure} test-match-any specifier @dots{} + +The resulting specifier matches if any @var{specifier} matches. Each +@var{specifier} is applied, in order, so side-effects from a later +@var{specifier} happen even if an earlier @var{specifier} is true. +@end deffn + +@deffn {Scheme Procedure} test-match-all specifier @dots{} + +The resulting specifier matches if each @var{specifier} matches. Each +@var{specifier} is applied, in order, so side-effects from a later +@var{specifier} happen even if an earlier @var{specifier} is false. +@end deffn + +@var{count} @i{(i.e. an integer)} +Convenience short-hand for: @samp{(test-match-nth 1 @var{count})}. + +@var{name} @i{(i.e. a string)} +Convenience short-hand for @samp{(test-match-name @var{name})}. + +@subsubheading Skipping selected tests + +In some cases you may want to skip a test. + +@deffn {Scheme Procedure} test-skip specifier + +Evaluating @code{test-skip} adds the resulting @var{specifier} to the +set of currently active skip-specifiers. Before each test (or +@code{test-group}) the set of active skip-specifiers are applied to the +active test-runner. If any specifier matches, then the test is skipped. + +For convenience, if the @var{specifier} is a string that is syntactic +sugar for @code{(test-match-name @var{specifier})}. For example: + +@lisp +(test-skip "test-b") +(test-assert "test-a") ;; executed +(test-assert "test-b") ;; skipped +@end lisp + +Any skip specifiers introduced by a @code{test-skip} are removed by a +following non-nested @code{test-end}. + +@lisp +(test-begin "group1") +(test-skip "test-a") +(test-assert "test-a") ;; skipped +(test-end "group1") ;; Undoes the prior test-skip +(test-assert "test-a") ;; executed +@end lisp +@end deffn + +@subsubheading Expected failures + +Sometimes you know a test case will fail, but you don't have time to or +can't fix it. Maybe a certain feature only works on certain platforms. +However, you want the test-case to be there to remind you to fix it. +You want to note that such tests are expected to fail. + +@deffn {Scheme Procedure} test-expect-fail specifier + +Matching tests (where matching is defined as in @code{test-skip}) +are expected to fail. This only affects test reporting, +not test execution. For example: + +@lisp +(test-expect-fail 2) +(test-eqv ...) ;; expected to fail +(test-eqv ...) ;; expected to fail +(test-eqv ...) ;; expected to pass +@end lisp +@end deffn + +@node SRFI 64 Test-runner +@subsubsection SRFI 64 Test-runner + +A @dfn{test-runner} is an object that runs a test-suite, and manages the +state. The test group path, and the sets skip and expected-fail +specifiers are part of the test-runner. A test-runner will also +typically accumulate statistics about executed tests, + +@deffn {Scheme Procedure} test-runner? value + +True if and only if @var{value} is a test-runner object. +@end deffn + +@deffn {Scheme Procedure} test-runner-current +@deffnx {Scheme Procedure} test-runner-current runner + +Get or set the current test-runner. If an implementation supports +parameter objects (as in +@url{https://srfi.schemers.org/srfi-39/srfi-39.html, SRFI-39}), then +@code{test-runner-current} can be a parameter object. Alternatively, +@code{test-runner-current} may be implemented as a macro or function +that uses a fluid or thread-local variable, or a plain global variable. +@end deffn + +@deffn {Scheme Procedure} test-runner-get + +Same as @code{(test-runner-current)}, but throws an exception if there +is no current test-runner. +@end deffn + +@deffn {Scheme Procedure} test-runner-simple + +Creates a new simple test-runner, that prints errors and a summary on +the standard output port. +@end deffn + +@deffn {Scheme Procedure} test-runner-null + +Creates a new test-runner, that does nothing with the test results. +This is mainly meant for extending when writing a custom runner. +@end deffn + +Implementations @emph{may} provide other test-runners, perhaps a +@code{(test-runner-gui)}. + +@deffn {Scheme Procedure} test-runner-create + +Create a new test-runner. Equivalent to @samp{((test-runner-factory))}. +@end deffn + +@deffn {Scheme Procedure} test-runner-factory +@deffnx {Scheme Procedure} test-runner-factory factory + +Get or set the current test-runner factory. A factory is a +zero-argument function that creates a new test-runner. The default +value is @code{test-runner-simple}, but implementations may provide a +way to override the default. As with @code{test-runner-current}, this +may be a parameter object, or use a per-thread, fluid, or global +variable. +@end deffn + +@subsubheading Running specific tests with a specified runner + +@deffn {Scheme Procedure} test-apply [runner] specifier @dots{} procedure + +Calls @var{procedure} with no arguments using the specified @var{runner} +as the current test-runner. If @var{runner} is omitted, then +@code{(test-runner-current)} is used. (If there is no current runner, +one is created as in @code{test-begin}.) If one or more +@var{specifier}s are listed then only tests matching the +@var{specifier}s are executed. A @var{specifier} has the same form as +one used for @code{test-skip}. A test is executed if it matches any of +the @var{specifier}s in the @code{test-apply} @emph{and} does not match +any active @code{test-skip} specifiers. +@end deffn + +@deffn {Scheme Procedure} test-with-runner runner decl-or-expr @dots{} + +Executes each @var{decl-or-expr} in order in a context where the current +test-runner is @var{runner}. +@end deffn + +@node SRFI 64 Test results +@subsubsection SRFI 64 Test results + +Running a test sets various status properties in the current test-runner. +This can be examined by a custom test-runner, +or (more rarely) in a test-suite. + +@subsubheading Result kind + +Running a test may yield one of the following +status symbols: + +@table @asis +@item @code{'pass} +The test passed, as expected. + +@item @code{'fail} +The test failed (and was not expected to). + +@item @code{'xfail} +The test failed and was expected to. + +@item @code{'xpass} +The test passed, but was expected to fail. + +@item @code{'skip} +The test was skipped. +@end table + +@deffn {Scheme Procedure} test-result-kind [runner] + +Returns one of the above result codes from the most recent tests. +Returns @code{#f} if no tests have been run yet. If we've started on a +new test, but don't have a result yet, then the result kind is +@code{'xfail} if the test is expected to fail, @code{'skip} if the test +is supposed to be skipped, or @code{#f} otherwise. +@end deffn + +@deffn {Scheme Procedure} test-passed? [runner] + +True if the value of @samp{(test-result-kind [@var{runner}])} is one of +@code{'pass} or @code{'xpass}. This is a convenient shorthand that +might be useful in a test suite to only run certain tests if the +previous test passed. +@end deffn + +@subsubheading Test result properties + +A test runner also maintains a set of more detailed +``result@tie{}properties'' associated with the current or most recent +test. (I.e. the properties of the most recent test are available as +long as a new test hasn't started.) Each property has a name (a symbol) +and a value (any value). Some properties are standard or set by the +implementation; implementations can add more. + +@deffn {Scheme Procedure} test-result-ref runner pname [default] + +Returns the property value associated with the @var{pname} property name +(a symbol). If there is no value associated with @var{pname} return +@var{default}, or @code{#f} if @var{default} isn't specified. +@end deffn + +@deffn {Scheme Procedure} test-result-set! runner pname value + +Sets the property value associated with the @var{pname} property name to +@var{value}. Usually implementation code should call this function, but +it may be useful for a custom test-runner to add extra properties. +@end deffn + +@deffn {Scheme Procedure} test-result-remove runner pname + +Remove the property with the name @var{pname}. +@end deffn + +@deffn {Scheme Procedure} test-result-clear runner + +Remove all result properties. The implementation automatically calls +@code{test-result-clear} at the start of a @code{test-assert} and +similar procedures. +@end deffn + +@deffn {Scheme Procedure} test-result-alist runner + +Returns an association list of the current result properties. It is +unspecified if the result shares state with the test-runner. The result +should not be modified; on the other hand, the result may be implicitly +modified by future @code{test-result-set!} or @code{test-result-remove} +calls. However, a @code{test-result-clear} does not modify the returned +alist. Thus you can ``archive'' result objects from previous runs. +@end deffn + +@subsubheading Standard result properties + +The set of available result properties is implementation-specific. +However, it is suggested that the following might be provided: +@table @asis + +@item @code{'result-kind} +The result kind, as defined previously. +This is the only mandatory result property. +@code{(test-result-kind @var{runner})} is equivalent to: +@code{(test-result-ref @var{runner} 'result-kind)} + +@item @code{'source-file} +@itemx @code{'source-line} +If known, the location of test statements (such as @code{test-assert}) +in test suite source code. + +@item @code{'source-form} +The source form, if meaningful and known. + +@item @code{'expected-value} +The expected non-error result, if meaningful and known. + +@item @code{'expected-error} +The @var{error-type}specified in a @code{test-error}, if it meaningful and known. + +@item @code{'actual-value} +The actual non-error result value, if meaningful and known. + +@item @code{'actual-error} +The error value, if an error was signalled and it is known. +The actual error value is implementation-defined. +@end table + +@node SRFI 64 Writing a new test-runner +@subsubsection SRFI 64 Writing a new test-runner + +This section specifies how to write a test-runner. It can be ignored if +you just want to write test-cases. + +@subsubheading Call-back functions + +These call-back functions are ``methods'' (in the object-oriented sense) +of a test-runner. A method @code{test-runner-on-@var{event}} is called +by the implementation when @var{event} happens. + +To define (set) the callback function for @var{event} use the following +expression. (This is normally done when initializing a test-runner.) + +@code{(test-runner-on-@var{event}! @var{runner} @var{event-function})} + +An @var{event-function} takes a test-runner argument, and possibly other +arguments, depending on the @var{event}. + +To extract (get) the callback function for @var{event} do this: +@code{(test-runner-on-@var{event} @var{runner})} + +To extract call the callback function for @var{event} use the following +expression. (This is normally done by the implementation core.) +@samp{((test-runner-on-@var{event} @var{runner}) @var{runner} +@var{other-args} @dots{})}. + +The following call-back hooks are available. + +@deffn {Scheme Procedure} test-runner-on-test-begin runner +@deffnx {Scheme Procedure} test-runner-on-test-begin! runner on-test-begin-function +@deffnx {Scheme Procedure} on-test-begin-function runner + +The @var{on-test-begin-function} is called at the start of an +individual testcase, before the test expression (and expected value) are +evaluated. + +@end deffn + +@deffn {Scheme Procedure} test-runner-on-test-end runner +@deffnx {Scheme Procedure} test-runner-on-test-end! runner on-test-end-function +@deffnx {Scheme Procedure} on-test-end-function runner + +The @var{on-test-end-function} is called at the end of an +individual testcase, when the result of the test is available. +@end deffn + +@deffn {Scheme Procedure} test-runner-on-group-begin runner +@deffnx {Scheme Procedure} test-runner-on-group-begin! runner on-group-begin-function +@deffnx {Scheme Procedure} on-group-begin-function runner suite-name count + +The @var{on-group-begin-function} is called by a @code{test-begin}, +including at the start of a @code{test-group}. The @var{suite-name} is +a Scheme string, and @var{count} is an integer or @code{#f}. +@end deffn + +@deffn {Scheme Procedure} test-runner-on-group-end runner +@deffnx {Scheme Procedure} test-runner-on-group-end! runner on-group-end-function +@deffnx {Scheme Procedure} on-group-end-function runner + +The @var{on-group-end-function} is called by a @code{test-end}, +including at the end of a @code{test-group}. +@end deffn + +@deffn {Scheme Procedure} test-runner-on-bad-count runner +@deffnx {Scheme Procedure} test-runner-on-bad-count! runner on-bad-count-function +@deffnx {Scheme Procedure} on-bad-count-function runner actual-count expected-count + +Called from @code{test-end} (before the @var{on-group-end-function} is +called) if an @var{expected-count} was specified by the matching +@code{test-begin} and the @var{expected-count} does not match the +@var{actual-count} of tests actually executed or skipped. +@end deffn + +@deffn {Scheme Procedure} test-runner-on-bad-end-name runner +@deffnx {Scheme Procedure} test-runner-on-bad-end-name! runner on-bad-end-name-function +@deffnx {Scheme Procedure} on-bad-end-name-function runner begin-name end-name + +Called from @code{test-end} (before the @var{on-group-end-function} is +called) if a @var{suite-name} was specified, and it did not that the +name in the matching @code{test-begin}. +@end deffn + +@deffn {Scheme Procedure} test-runner-on-final runner +@deffnx {Scheme Procedure} test-runner-on-final! runner on-final-function +@deffnx {Scheme Procedure} on-final-function runner + +The @var{on-final-function} takes one parameter (a test-runner) and +typically displays a summary (count) of the tests. The +@var{on-final-function} is called after called the +@var{on-group-end-function} correspondiong to the outermost +@code{test-end}. The default value is @code{test-on-final-simple} which +writes to the standard output port the number of tests of the various +kinds. +@end deffn + +The default test-runner returned by @code{test-runner-simple} uses the +following call-back functions: + +@deffn {Scheme Procedure} test-on-test-begin-simple runner +@deffnx {Scheme Procedure} test-on-test-end-simple runner +@deffnx {Scheme Procedure} test-on-group-begin-simple runner suite-name count +@deffnx {Scheme Procedure} test-on-group-end-simple runner +@deffnx {Scheme Procedure} test-on-bad-count-simple runner actual-count expected-count +@deffnx {Scheme Procedure} test-on-bad-end-name-simple runner begin-name end-name + +You can call those if you want to write your own test-runner. +@end deffn + +@subsubheading Test-runner components + +The following functions are for accessing the other components of a +test-runner. They would normally only be used to write a new +test-runner or a match-predicate. + +@deffn {Scheme Procedure} test-runner-pass-count runner + +Returns the number of tests that passed, and were expected to pass. +@end deffn + +@deffn {Scheme Procedure} test-runner-fail-count runner + +Returns the number of tests that failed, but were expected to pass. +@end deffn + +@deffn {Scheme Procedure} test-runner-xpass-count runner + +Returns the number of tests that passed, but were expected to fail. +@end deffn + +@deffn {Scheme Procedure} test-runner-xfail-count runner + +Returns the number of tests that failed, and were expected to pass. +@end deffn + +@deffn {Scheme Procedure} test-runner-skip-count runner + +Returns the number of tests or test groups that were skipped. +@end deffn + +@deffn {Scheme Procedure} test-runner-test-name runner + +Returns the name of the current test or test group, as a string. During +execution of @code{test-begin} this is the name of the test group; +during the execution of an actual test, this is the name of the +test-case. If no name was specified, the name is the empty string. +@end deffn + +@deffn {Scheme Procedure} test-runner-group-path runner + +A list of names of groups we're nested in, with the outermost group +first. +@end deffn + +@deffn {Scheme Procedure} test-runner-group-stack runner + +A list of names of groups we're nested in, with the outermost group +last. (This is more efficient than @code{test-runner-group-path}, since +it doesn't require any copying.) +@end deffn + +@deffn {Scheme Procedure} test-runner-aux-value runner +@deffnx {Scheme Procedure} test-runner-aux-value! runner on-test + +Get or set the @code{aux-value} field of a test-runner. This field is +not used by this API or the @code{test-runner-simple} test-runner, but +may be used by custom test-runners to store extra state. +@end deffn + +@deffn {Scheme Procedure} test-runner-reset runner + +Resets the state of the @var{runner} to its initial state. +@end deffn + +@subsubheading Example + +This is an example of a simple custom test-runner. +Loading this program before running a test-suite will install +it as the default test runner. + +@lisp +(define (my-simple-runner filename) + (let ((runner (test-runner-null)) + (port (open-output-file filename)) + (num-passed 0) + (num-failed 0)) + (test-runner-on-test-end! runner + (lambda (runner) + (case (test-result-kind runner) + ((pass xpass) (set! num-passed (+ num-passed 1))) + ((fail xfail) (set! num-failed (+ num-failed 1))) + (else #t)))) + (test-runner-on-final! runner + (lambda (runner) + (format port "Passing tests: ~d.~%Failing tests: ~d.~%" + num-passed num-failed) + (close-output-port port))) + runner)) +(test-runner-factory + (lambda () (my-simple-runner "/tmp/my-test.log"))) +@end lisp @node SRFI-67 @subsection SRFI-67 - Compare procedures base-commit: 75cd95060fb1ea7586f0e4b9081694c6d61f1d3b -- 2.41.0