From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Maxim Cournoyer Newsgroups: gmane.lisp.guile.devel Subject: [PATCH v3] doc: Document SRFI 64. Date: Wed, 22 Nov 2023 22:55:52 -0500 Message-ID: <20231123035624.10330-1-maxim.cournoyer@gmail.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="10601"; mail-complaints-to="usenet@ciao.gmane.io" Cc: =?UTF-8?q?Filip=20=C5=81ajszczak?= , Maxime Devos , Maxim Cournoyer To: guile-devel@gnu.org Original-X-From: guile-devel-bounces+guile-devel=m.gmane-mx.org@gnu.org Thu Nov 23 04:57:23 2023 Return-path: Envelope-to: guile-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1r60qA-0002Y4-Vn for guile-devel@m.gmane-mx.org; Thu, 23 Nov 2023 04:57:23 +0100 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1r60pj-0003cy-1H; Wed, 22 Nov 2023 22:56:55 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1r60pb-0003cX-3U for guile-devel@gnu.org; Wed, 22 Nov 2023 22:56:47 -0500 Original-Received: from mail-qk1-x72b.google.com ([2607:f8b0:4864:20::72b]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1r60pW-0006Ts-GR for guile-devel@gnu.org; Wed, 22 Nov 2023 22:56:46 -0500 Original-Received: by mail-qk1-x72b.google.com with SMTP id af79cd13be357-7789a4c01easo25541285a.0 for ; Wed, 22 Nov 2023 19:56:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1700711801; x=1701316601; darn=gnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=SZMj0TntgHSqbQ9Jjc5hjELOREaRsTPD+N7JpIZz4Iw=; b=HKigyM252bsw2w+16gV08q97Pwh58GouK17TEpsaF+vWNR77yimpNOiOC0rQ0sxwL3 PKvajtUUijQOtdHE3nDcDsrYP9lg8R/DYPeCo1XUZYdQnrbhH8NGb5X1dShCacq7Yxrh y/gChxIMJgZXGkT94gVN3EcjhwTZ+2yausp2oSXqqa/s8wyVIetmcE2/eEmhj1LPDnYU bONxnZMn5TmF8lG0L3Zi2SSg6S1igw6/mtLXmsf3VHSY31Awf+NGg3vvhUk0f+UClUk1 d1WwVekSByMPU4oTN3ulzmQC5GPOdOSXZaNq+J1YjFbQablUoNbry4h32Pg98Z7TPRn0 hvxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700711801; x=1701316601; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SZMj0TntgHSqbQ9Jjc5hjELOREaRsTPD+N7JpIZz4Iw=; b=AiYdUYgwz9NJeuH4G3vUWMyBCiPqpksGf765zggEUarhNC2InhTE0t2kx48VFPWmNb 37aMbMA1M87YlsF9mYypJzJ3fcbfScTEZoBwDJN7vn2JGMRPFMbDlmQ1Sg6hbMYkgBbn lsE4vNAsuCmSKm4rf8M3u7FomO1oxUbJPQBGZ00o1oKnJ9OWwtOzhROWCvgV4YzSgLQN Ci9aEQl1Xe3ViurbOLL9/CuD/pCYoT5kfa2JJZeAstQHOiAwCIdn4Ciao9lwy9hrkyX0 chwctkRkDJORRMwsF/pgPQPPmq4hdlhbEOpx0P80fjxM1mYvPxNBeEpHc4thxSqHwNbx t/yA== X-Gm-Message-State: AOJu0YzXW1gNtrZqZXKmJGrUvamIXuvJecN6SXDzg8V+tU6hac32cutJ ROvdzWYJIcESYXNe/iv7+svlRJVtV/g= X-Google-Smtp-Source: AGHT+IEBSdh1om5I3fJS91Q5TuNL1FiEc9r29vme1230Fw008ewZ6Ce5qtyCnPsATWTefG5TbA/RHQ== X-Received: by 2002:a05:6214:c8b:b0:671:43fc:c275 with SMTP id r11-20020a0562140c8b00b0067143fcc275mr5391999qvr.48.1700711799924; Wed, 22 Nov 2023 19:56:39 -0800 (PST) Original-Received: from localhost.localdomain (dsl-205-233-124-102.b2b2c.ca. [205.233.124.102]) by smtp.gmail.com with ESMTPSA id j2-20020a0cae82000000b00679f7657dbdsm161450qvd.99.2023.11.22.19.56.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Nov 2023 19:56:39 -0800 (PST) X-Mailer: git-send-email 2.41.0 In-Reply-To: Received-SPF: pass client-ip=2607:f8b0:4864:20::72b; envelope-from=maxim.cournoyer@gmail.com; helo=mail-qk1-x72b.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guile-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Developers list for Guile, the GNU extensibility library" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guile-devel-bounces+guile-devel=m.gmane-mx.org@gnu.org Original-Sender: guile-devel-bounces+guile-devel=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.lisp.guile.devel:22118 Archived-At: This is an import of the 'Abstract', 'Rationale', and 'Specification' sections from the upstream specification text, with some manual adjustment. * doc/ref/srfi-modules.texi (SRFI 64): New subsection. --- Changes in v3: - Add copyright / license information - Replace SchemeUnit mentions with RackUnit Changes in v2: - Fix the category of many definitions doc/ref/guile.texi | 25 +- doc/ref/srfi-modules.texi | 830 +++++++++++++++++++++++++++++++++++++- 2 files changed, 847 insertions(+), 8 deletions(-) diff --git a/doc/ref/guile.texi b/doc/ref/guile.texi index 8414c3e2d..307b0f342 100644 --- a/doc/ref/guile.texi +++ b/doc/ref/guile.texi @@ -23,8 +23,31 @@ any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled ``GNU Free Documentation License.'' -@end copying +Additionally, the documentation of the SRFI 64 module is adapted from +its specification text, which is made available under the following +Expat license: + +Permission is hereby granted, free of charge, to any person obtaining a +copy of this software and associated documentation files (the +"Software"), to deal in the Software without restriction, including +without limitation the rights to use, copy, modify, merge, publish, +distribute, sublicense, and/or sell copies of the Software, and to +permit persons to whom the Software is furnished to do so, subject to +the following conditions: + +The above copyright notice and this permission notice shall be included +in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS +OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE +LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION +OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION +WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +@end copying @c Notes @c diff --git a/doc/ref/srfi-modules.texi b/doc/ref/srfi-modules.texi index 0cdf56923..d76b6e81a 100644 --- a/doc/ref/srfi-modules.texi +++ b/doc/ref/srfi-modules.texi @@ -1,7 +1,8 @@ @c -*-texinfo-*- @c This is part of the GNU Guile Reference Manual. -@c Copyright (C) 1996, 1997, 2000-2004, 2006, 2007-2014, 2017, 2018, 2019, 2020 +@c Copyright (C) 1996-1997, 2000-2004, 2006-2014, 2017-2020, 2023 @c Free Software Foundation, Inc. +@c Copyright (C) 2005-2006 Per Bothner @c See the file guile.texi for copying conditions. @node SRFI Support @@ -55,7 +56,7 @@ get the relevant SRFI documents from the SRFI home page * SRFI-60:: Integers as bits. * SRFI-61:: A more general `cond' clause * SRFI-62:: S-expression comments. -* SRFI-64:: A Scheme API for test suites. +* SRFI 64:: A Scheme API for test suites. * SRFI-67:: Compare procedures * SRFI-69:: Basic hash tables. * SRFI-71:: Extended let-syntax for multiple values. @@ -5289,12 +5290,827 @@ needed to get SRFI-61 itself. Extended @code{cond} is documented in Starting from version 2.0, Guile's @code{read} supports SRFI-62/R7RS S-expression comments by default. -@node SRFI-64 -@subsection SRFI-64 - A Scheme API for test suites. -@cindex SRFI-64 +@c This SRFI 64 documentation was "snarfed" from upstream specification +@c HTML document using the 'snarfi' script. +@node SRFI 64 +@subsection SRFI 64: A Scheme API for test suites +@cindex SRFI 64 -See @uref{http://srfi.schemers.org/srfi-64/srfi-64.html, the -specification of SRFI-64}. +@menu +* SRFI 64 Abstract:: +* SRFI 64 Rationale:: +* SRFI 64 Writing basic test suites:: +* SRFI 64 Conditonal test-suites and other advanced features:: +* SRFI 64 Test-runner:: +* SRFI 64 Test results:: +* SRFI 64 Writing a new test-runner:: +@end menu + +@node SRFI 64 Abstract +@subsubsection SRFI 64 Abstract + +This defines an API for writing @dfn{test suites}, to make it easy to +portably test Scheme APIs, libraries, applications, and implementations. +A test suite is a collection of @dfn{test cases} that execute in the +context of a @dfn{test-runner}. This specification also supports +writing new test-runners, to allow customization of reporting and +processing the result of running test suites. + +@node SRFI 64 Rationale +@subsubsection SRFI 64 Rationale + +The Scheme community needs a standard for writing test suites. Every +SRFI or other library should come with a test suite. Such a test suite +must be portable, without requiring any non-standard features, such as +modules. The test suite implementation or "runner" need not be +portable, but it is desirable that it be possible to write a portable +basic implementation. + +There are other testing frameworks written in Scheme, including +@url{https://docs.racket-lang.org/rackunit/, RackUnit}. However +RackUnit is not portable. It is also a bit on the verbose side. It +would be useful to have a bridge between this framework and RackUnit so +RackUnit tests could run under this framework and vice versa. There +exists also at least one Scheme wrapper providing a Scheme interface to +the ``standard'' @url{https://www.junit.org/, JUnit} API for Java. It +would be useful to have a bridge so that tests written using this +framework can run under a JUnit runner. Neither of these features are +part of this specification. + +This API makes use of implicit dynamic state, including an implicit +``test runner''. This makes the API convenient and terse to use, but it +may be a little less elegant and ``compositional'' than using explicit +test objects, such as JUnit-style frameworks. It is not claimed to +follow either object-oriented or functional design principles, but I +hope it is useful and convenient to use and extend. + +This proposal allows converting a Scheme source file to a +test suite by just adding a few macros. You don't have to +write the entire file in a new form, thus you don't have to +re-indent it. + +All names defined by the API start with the prefix @samp{test-}. All +function-like forms are defined as syntax. They may be implemented as +functions or macros or built-ins. The reason for specifying them as +syntax is to allow specific tests to be skipped without evaluating +sub-expressions, or for implementations to add features such as printing +line numbers or catching exceptions. + +@node SRFI 64 Writing basic test suites +@subsubsection SRFI 64 Writing basic test suites + +Let's start with a simple example. This is a complete self-contained +test-suite. + +@lisp +;; Initialize and give a name to a simple testsuite. +(test-begin "vec-test") +(define v (make-vector 5 99)) +;; Require that an expression evaluate to true. +(test-assert (vector? v)) +;; Test that an expression is eqv? to some other expression. +(test-eqv 99 (vector-ref v 2)) +(vector-set! v 2 7) +(test-eqv 7 (vector-ref v 2)) +;; Finish the testsuite, and report results. +(test-end "vec-test") +@end lisp + +This testsuite could be saved in its own source file. Nothing else is +needed: We do not require any top-level forms, so it is easy to wrap an +existing program or test to this form, without adding indentation. It +is also easy to add new tests, without having to name individual tests +(though that is optional). + +Test cases are executed in the context of a @dfn{test runner}, which is +a object that accumulates and reports test results. This specification +defines how to create and use custom test runners, but implementations +should also provide a default test runner. It is suggested (but not +required) that loading the above file in a top-level environment will +cause the tests to be executed using an implementation-specified default +test runner, and @code{test-end} will cause a summary to be displayed in +an implementation-specified manner. + +@subsubheading Simple test-cases + +Primitive test cases test that a given condition is true. They may have +a name. The core test case form is @code{test-assert}: + +@deffn {Scheme Syntax} test-assert [test-name] expression + +This evaluates the @var{expression}. The test passes if the result is +true; if the result is false, a test failure is reported. The test also +fails if an exception is raised, assuming the implementation has a way +to catch exceptions. How the failure is reported depends on the test +runner environment. The @var{test-name} is a string that names the test +case. (Though the @var{test-name} is a string literal in the examples, +it is an expression. It is evaluated only once.) It is used when +reporting errors, and also when skipping tests, as described below. It +is an error to invoke @code{test-assert}if there is no current test +runner. +@end deffn + +The following forms may be more convenient than using @code{test-assert} +directly: + +@deffn {Scheme Syntax} test-eqv [test-name] expected test-expr + +This is equivalent to: + +@lisp +(test-assert [@var{test-name}] (eqv? @var{expected} @var{test-expr})) +@end lisp + +@end deffn + +Similarly @code{test-equal} and @code{test-eq} are shorthand for +@code{test-assert} combined with @code{equal?} or @code{eq?}, +respectively: + +@deffn {Scheme Syntax} test-equal [test-name] expected test-expr +@deffnx {Scheme Syntax} test-eq [test-name] expected test-expr + +Here is a simple example: + +@lisp +(define (mean x y) (/ (+ x y) 2.0)) +(test-eqv 4 (mean 3 5)) +@end lisp +@end deffn + +For testing approximate equality of inexact reals +we can use @code{test-approximate}: + +@deffn {Scheme Syntax} test-approximate [test-name] expected test-expr error + +This is equivalent to (except that each argument is only evaluated +once): + +@lisp +(test-assert [test-name] + (and (>= test-expr (- expected error)) + (<= test-expr (+ expected error)))) +@end lisp +@end deffn + +@subsubheading Tests for catching errors + +We need a way to specify that evaluation @emph{should} fail. This +verifies that errors are detected when required. + +@deffn {Scheme Syntax} test-error [[test-name] error-type] test-expr + +Evaluating @var{test-expr} is expected to signal an error. The kind of +error is indicated by @var{error-type}. + +If the @var{error-type} is left out, or it is @code{#t}, it means "some +kind of unspecified error should be signaled". For example: + +@lisp +(test-error #t (vector-ref '#(1 2) 9)) +@end lisp + +This specification leaves it implementation-defined (or for a future +specification) what form @var{test-error} may take, though all +implementations must allow @code{#t}. Some implementations may support +@url{https://srfi.schemers.org/srfi-35/srfi-35.html, SRFI-35's +conditions}, but these are only standardized for +@url{https://srfi.schemers.org/srfi-36/srfi-36.html, SRFI-36's I/O +conditions}, which are seldom useful in test suites. An implementation +may also allow implementation-specific ``exception types''. For example +Java-based implementations may allow the names of Java exception +classes: + +@lisp +;; Kawa-specific example +(test-error (vector-ref '#(1 2) 9)) +@end lisp + +An implementation that cannot catch exceptions should skip +@code{test-error} forms. +@end deffn + +@subsubheading Testing syntax + +Testing syntax is tricky, especially if we want to check that invalid +syntax is causing an error. The following utility function can help: + +@deffn {Scheme Procedure} test-read-eval-string string + +This function parses @var{string} (using @code{read}) and evaluates the +result. The result of evaluation is returned from +@code{test-read-eval-string}. An error is signalled if there are unread +characters after the @code{read} is done. For example: +@code{(test-read-eval-string "(+ 3 4)")} @i{evaluates to} @code{7}. +@code{(test-read-eval-string "(+ 3 4")} @i{signals an error}. +@code{(test-read-eval-string "(+ 3 4) ")} @i{signals an error}, because +there is extra ``junk'' (@i{i.e.} a space) after the list is read. + +The @code{test-read-eval-string} used in tests: + +@lisp +(test-equal 7 (test-read-eval-string "(+ 3 4)")) +(test-error (test-read-eval-string "(+ 3")) +(test-equal #\newline (test-read-eval-string "#\\newline")) +(test-error (test-read-eval-string "#\\newlin")) +;; Skip the next 2 tests unless srfi-62 is available. +(test-skip (cond-expand (srfi-62 0) (else 2))) +(test-equal 5 (test-read-eval-string "(+ 1 #;(* 2 3) 4)")) +(test-equal '(x z) (test-read-string "(list 'x #;'y 'z)")) +@end lisp +@end deffn + +@subsubheading Test groups and paths + +A @dfn{test group} is a named sequence of forms containing testcases, +expressions, and definitions. Entering a group sets the @dfn{test group +name}; leaving a group restores the previous group name. These are +dynamic (run-time) operations, and a group has no other effect or +identity. Test groups are informal groupings: they are neither Scheme +values, nor are they syntactic forms. +@c (More formal test suite values are introduced below.) +A test group may contain nested inner test groups. +The @dfn{test group path} is a list of the currently-active +(entered) test group names, oldest (outermost) first. + +@deffn {Scheme Syntax} test-begin suite-name [count] + +A @code{test-begin} enters a new test group. The @var{suite-name} +becomes the current test group name, and is added to the end of the test +group path. Portable test suites should use a string literal for +@var{suite-name}; the effect of expressions or other kinds of literals +is unspecified. + +@emph{Rationale:} In some ways using symbols would be preferable. +However, we want human-readable names, and standard Scheme does not +provide a way to include spaces or mixed-case text in literal symbols. + +The optional @var{count} must match the number of test-cases executed by +this group. (Nested test groups count as a single test case for this +count.) This extra test may be useful to catch cases where a test +doesn't get executed because of some unexpected error. + +Additionally, if there is no currently executing test runner, +one is installed in an implementation-defined manner. +@end deffn + +@deffn {Scheme Syntax} test-end [suite-name] + +A @code{test-end} leaves the current test group. +An error is reported if the @var{suite-name} does not +match the current test group name. +@c If it does match an earlier name in the test group path, intervening +@c groups are left. + +Additionally, if the matching @code{test-begin}installed a new +test-runner, then the @code{test-end} will uninstall it, after reporting +the accumulated test results in an implementation-defined manner. +@end deffn + +@deffn {Scheme Syntax} test-group suite-name decl-or-expr @dots{} + +Equivalent to: + +@lisp +(if (not (test-to-skip% (var suite-name))) + (dynamic-wind + (lambda () (test-begin (var suite-name))) + (lambda () (var decl-or-expr) ...) + (lambda () (test-end (var suite-name))))) +@end lisp + +This is usually equivalent to executing the @var{decl-or-expr}s +within the named test group. However, the entire group is skipped +if it matched an active @code{test-skip} (see later). +Also, the @code{test-end} is executed in case of an exception. +@end deffn + +@subsubheading Handling set-up and cleanup + +@deffn {Scheme Syntax} test-group-with-cleanup suite-name decl-or-expr @dots{} cleanup-form + +Execute each of the @var{decl-or-expr} forms in order (as in a +@var{}), and then execute the @var{cleanup-form}. The latter +should be executed even if one of a @var{decl-or-expr} forms raises an +exception (assuming the implementation has a way to catch exceptions). + +For example: + +@lisp +(let ((f (open-output-file "log"))) + (test-group-with-cleanup "test-file" + (do-a-bunch-of-tests f) + (close-output-port f))) +@end lisp +@end deffn + +@node SRFI 64 Conditonal test-suites and other advanced features +@subsubsection SRFI 64 Conditonal test-suites and other advanced features + +The following describes features for controlling which tests to execute, +or specifying that some tests are @emph{expected} to fail. + +@subsubheading Test specifiers + +Sometimes we want to only run certain tests, or we know that certain +tests are expected to fail. A @dfn{test specifier} is one-argument +function that takes a test-runner and returns a boolean. The specifier +may be run before a test is performed, and the result may control +whether the test is executed. For convenience, a specifier may also be +a non-procedure value, which is coerced to a specifier procedure, as +described below for @var{count} and @var{name}. + +A simple example is: + +@lisp +(if (var some-condition) (test-skip 2)) ;; skip next 2 tests +@end lisp + +@deffn {Scheme Procedure} test-match-name name + +The resulting specifier matches if the current test name (as returned by +@code{test-runner-test-name}) is @code{equal?} to @var{name}. +@end deffn + +@deffn {Scheme Syntax} test-match-nth n [count] + +This evaluates to a @emph{stateful} predicate: A counter keeps track of +how many times it has been called. The predicate matches the @var{n}'th +time it is called (where @code{1} is the first time), and the next +@samp{(- @var{count} 1)} times, where @var{count} defaults to @code{1}. +@end deffn + +@deffn {Scheme Syntax} test-match-any specifier @dots{} + +The resulting specifier matches if any @var{specifier} matches. Each +@var{specifier} is applied, in order, so side-effects from a later +@var{specifier} happen even if an earlier @var{specifier} is true. +@end deffn + +@deffn {Scheme Syntax} test-match-all specifier @dots{} + +The resulting specifier matches if each @var{specifier} matches. Each +@var{specifier} is applied, in order, so side-effects from a later +@var{specifier} happen even if an earlier @var{specifier} is false. +@end deffn + +@var{count} @i{(i.e. an integer)} +Convenience short-hand for: @samp{(test-match-nth 1 @var{count})}. + +@var{name} @i{(i.e. a string)} +Convenience short-hand for @samp{(test-match-name @var{name})}. + +@subsubheading Skipping selected tests + +In some cases you may want to skip a test. + +@deffn {Scheme Syntax} test-skip specifier + +Evaluating @code{test-skip} adds the resulting @var{specifier} to the +set of currently active skip-specifiers. Before each test (or +@code{test-group}) the set of active skip-specifiers are applied to the +active test-runner. If any specifier matches, then the test is skipped. + +For convenience, if the @var{specifier} is a string that is syntactic +sugar for @code{(test-match-name @var{specifier})}. For example: + +@lisp +(test-skip "test-b") +(test-assert "test-a") ;; executed +(test-assert "test-b") ;; skipped +@end lisp + +Any skip specifiers introduced by a @code{test-skip} are removed by a +following non-nested @code{test-end}. + +@lisp +(test-begin "group1") +(test-skip "test-a") +(test-assert "test-a") ;; skipped +(test-end "group1") ;; Undoes the prior test-skip +(test-assert "test-a") ;; executed +@end lisp +@end deffn + +@subsubheading Expected failures + +Sometimes you know a test case will fail, but you don't have time to or +can't fix it. Maybe a certain feature only works on certain platforms. +However, you want the test-case to be there to remind you to fix it. +You want to note that such tests are expected to fail. + +@deffn {Scheme Syntax} test-expect-fail specifier + +Matching tests (where matching is defined as in @code{test-skip}) +are expected to fail. This only affects test reporting, +not test execution. For example: + +@lisp +(test-expect-fail 2) +(test-eqv ...) ;; expected to fail +(test-eqv ...) ;; expected to fail +(test-eqv ...) ;; expected to pass +@end lisp +@end deffn + +@node SRFI 64 Test-runner +@subsubsection SRFI 64 Test-runner + +A @dfn{test-runner} is an object that runs a test-suite, and manages the +state. The test group path, and the sets skip and expected-fail +specifiers are part of the test-runner. A test-runner will also +typically accumulate statistics about executed tests, + +@deffn {Scheme Procedure} test-runner? value + +True if and only if @var{value} is a test-runner object. +@end deffn + +@deffn {Scheme Parameter} test-runner-current +@deffnx {Scheme Parameter} test-runner-current runner + +Get or set the current test-runner. +@end deffn + +@deffn {Scheme Procedure} test-runner-get + +Same as @code{(test-runner-current)}, but throws an exception if there +is no current test-runner. +@end deffn + +@deffn {Scheme Procedure} test-runner-simple + +Creates a new simple test-runner, that prints errors and a summary on +the standard output port. +@end deffn + +@deffn {Scheme Procedure} test-runner-null + +Creates a new test-runner, that does nothing with the test results. +This is mainly meant for extending when writing a custom runner. +@end deffn + +@deffn {Scheme Procedure} test-runner-create + +Create a new test-runner. Equivalent to @samp{((test-runner-factory))}. +@end deffn + +@deffn {Scheme Parameter} test-runner-factory +@deffnx {Scheme Parameter} test-runner-factory factory + +Get or set the current test-runner factory. A factory is a +zero-argument function that creates a new test-runner. The default +value is @code{test-runner-simple}. +@end deffn + +@subsubheading Running specific tests with a specified runner + +@deffn {Scheme Procedure} test-apply [runner] specifier @dots{} procedure + +Calls @var{procedure} with no arguments using the specified @var{runner} +as the current test-runner. If @var{runner} is omitted, then +@code{(test-runner-current)} is used. (If there is no current runner, +one is created as in @code{test-begin}.) If one or more +@var{specifier}s are listed then only tests matching the +@var{specifier}s are executed. A @var{specifier} has the same form as +one used for @code{test-skip}. A test is executed if it matches any of +the @var{specifier}s in the @code{test-apply} @emph{and} does not match +any active @code{test-skip} specifiers. +@end deffn + +@deffn {Scheme Syntax} test-with-runner runner decl-or-expr @dots{} + +Executes each @var{decl-or-expr} in order in a context where the current +test-runner is @var{runner}. +@end deffn + +@node SRFI 64 Test results +@subsubsection SRFI 64 Test results + +Running a test sets various status properties in the current test-runner. +This can be examined by a custom test-runner, +or (more rarely) in a test-suite. + +@subsubheading Result kind + +Running a test may yield one of the following +status symbols: + +@table @asis +@item @code{'pass} +The test passed, as expected. + +@item @code{'fail} +The test failed (and was not expected to). + +@item @code{'xfail} +The test failed and was expected to. + +@item @code{'xpass} +The test passed, but was expected to fail. + +@item @code{'skip} +The test was skipped. +@end table + +@deffn {Scheme Procedure} test-result-kind [runner] + +Returns one of the above result codes from the most recent tests. +Returns @code{#f} if no tests have been run yet. If we've started on a +new test, but don't have a result yet, then the result kind is +@code{'xfail} if the test is expected to fail, @code{'skip} if the test +is supposed to be skipped, or @code{#f} otherwise. +@end deffn + +@deffn {Scheme Procedure} test-passed? [runner] + +True if the value of @samp{(test-result-kind [@var{runner}])} is one of +@code{'pass} or @code{'xpass}. This is a convenient shorthand that +might be useful in a test suite to only run certain tests if the +previous test passed. +@end deffn + +@subsubheading Test result properties + +A test runner also maintains a set of more detailed +``result@tie{}properties'' associated with the current or most recent +test. (I.e. the properties of the most recent test are available as +long as a new test hasn't started.) Each property has a name (a symbol) +and a value (any value). Some properties are standard or set by the +implementation; implementations can add more. + +@deffn {Scheme Procedure} test-result-ref runner pname [default] + +Returns the property value associated with the @var{pname} property name +(a symbol). If there is no value associated with @var{pname} return +@var{default}, or @code{#f} if @var{default} isn't specified. +@end deffn + +@deffn {Scheme Syntax} test-result-set! runner pname value + +Sets the property value associated with the @var{pname} property name to +@var{value}. Usually implementation code should call this function, but +it may be useful for a custom test-runner to add extra properties. +@end deffn + +@deffn {Scheme Procedure} test-result-remove runner pname + +Remove the property with the name @var{pname}. +@end deffn + +@deffn {Scheme Procedure} test-result-clear runner + +Remove all result properties. The implementation automatically calls +@code{test-result-clear} at the start of a @code{test-assert} and +similar procedures. +@end deffn + +@deffn {Scheme Procedure} test-result-alist runner + +Returns an association list of the current result properties. It is +unspecified if the result shares state with the test-runner. The result +should not be modified; on the other hand, the result may be implicitly +modified by future @code{test-result-set!} or @code{test-result-remove} +calls. However, a @code{test-result-clear} does not modify the returned +alist. Thus you can ``archive'' result objects from previous runs. +@end deffn + +@subsubheading Standard result properties + +The set of available result properties is implementation-specific. +However, it is suggested that the following might be provided: +@table @asis + +@item @code{'result-kind} +The result kind, as defined previously. +This is the only mandatory result property. +@code{(test-result-kind @var{runner})} is equivalent to: +@code{(test-result-ref @var{runner} 'result-kind)} + +@item @code{'source-file} +@itemx @code{'source-line} +If known, the location of test statements (such as @code{test-assert}) +in test suite source code. + +@item @code{'source-form} +The source form, if meaningful and known. + +@item @code{'expected-value} +The expected non-error result, if meaningful and known. + +@item @code{'expected-error} +The @var{error-type}specified in a @code{test-error}, if it meaningful and known. + +@item @code{'actual-value} +The actual non-error result value, if meaningful and known. + +@item @code{'actual-error} +The error value, if an error was signalled and it is known. +The actual error value is implementation-defined. +@end table + +@node SRFI 64 Writing a new test-runner +@subsubsection SRFI 64 Writing a new test-runner + +This section specifies how to write a test-runner. It can be ignored if +you just want to write test-cases. + +@subsubheading Call-back functions + +These call-back functions are ``methods'' (in the object-oriented sense) +of a test-runner. A method @code{test-runner-on-@var{event}} is called +by the implementation when @var{event} happens. + +To define (set) the callback function for @var{event} use the following +expression. (This is normally done when initializing a test-runner.) + +@code{(test-runner-on-@var{event}! @var{runner} @var{event-function})} + +An @var{event-function} takes a test-runner argument, and possibly other +arguments, depending on the @var{event}. + +To extract (get) the callback function for @var{event} do this: +@code{(test-runner-on-@var{event} @var{runner})} + +To extract call the callback function for @var{event} use the following +expression. (This is normally done by the implementation core.) +@samp{((test-runner-on-@var{event} @var{runner}) @var{runner} +@var{other-args} @dots{})}. + +The following call-back hooks are available. + +@deffn {Scheme Procedure} test-runner-on-test-begin runner +@deffnx {Scheme Procedure} test-runner-on-test-begin! runner on-test-begin-function +@deffnx {Scheme Procedure} on-test-begin-function runner + +The @var{on-test-begin-function} is called at the start of an +individual testcase, before the test expression (and expected value) are +evaluated. + +@end deffn + +@deffn {Scheme Procedure} test-runner-on-test-end runner +@deffnx {Scheme Procedure} test-runner-on-test-end! runner on-test-end-function +@deffnx {Scheme Procedure} on-test-end-function runner + +The @var{on-test-end-function} is called at the end of an +individual testcase, when the result of the test is available. +@end deffn + +@deffn {Scheme Procedure} test-runner-on-group-begin runner +@deffnx {Scheme Procedure} test-runner-on-group-begin! runner on-group-begin-function +@deffnx {Scheme Procedure} on-group-begin-function runner suite-name count + +The @var{on-group-begin-function} is called by a @code{test-begin}, +including at the start of a @code{test-group}. The @var{suite-name} is +a Scheme string, and @var{count} is an integer or @code{#f}. +@end deffn + +@deffn {Scheme Procedure} test-runner-on-group-end runner +@deffnx {Scheme Procedure} test-runner-on-group-end! runner on-group-end-function +@deffnx {Scheme Procedure} on-group-end-function runner + +The @var{on-group-end-function} is called by a @code{test-end}, +including at the end of a @code{test-group}. +@end deffn + +@deffn {Scheme Procedure} test-runner-on-bad-count runner +@deffnx {Scheme Procedure} test-runner-on-bad-count! runner on-bad-count-function +@deffnx {Scheme Procedure} on-bad-count-function runner actual-count expected-count + +Called from @code{test-end} (before the @var{on-group-end-function} is +called) if an @var{expected-count} was specified by the matching +@code{test-begin} and the @var{expected-count} does not match the +@var{actual-count} of tests actually executed or skipped. +@end deffn + +@deffn {Scheme Procedure} test-runner-on-bad-end-name runner +@deffnx {Scheme Procedure} test-runner-on-bad-end-name! runner on-bad-end-name-function +@deffnx {Scheme Procedure} on-bad-end-name-function runner begin-name end-name + +Called from @code{test-end} (before the @var{on-group-end-function} is +called) if a @var{suite-name} was specified, and it did not that the +name in the matching @code{test-begin}. +@end deffn + +@deffn {Scheme Procedure} test-runner-on-final runner +@deffnx {Scheme Procedure} test-runner-on-final! runner on-final-function +@deffnx {Scheme Procedure} on-final-function runner + +The @var{on-final-function} takes one parameter (a test-runner) and +typically displays a summary (count) of the tests. The +@var{on-final-function} is called after called the +@var{on-group-end-function} correspondiong to the outermost +@code{test-end}. The default value is @code{test-on-final-simple} which +writes to the standard output port the number of tests of the various +kinds. +@end deffn + +The default test-runner returned by @code{test-runner-simple} uses the +following call-back functions: + +@deffn {Scheme Procedure} test-on-test-begin-simple runner +@deffnx {Scheme Procedure} test-on-test-end-simple runner +@deffnx {Scheme Procedure} test-on-group-begin-simple runner suite-name count +@deffnx {Scheme Procedure} test-on-group-end-simple runner +@deffnx {Scheme Procedure} test-on-bad-count-simple runner actual-count expected-count +@deffnx {Scheme Procedure} test-on-bad-end-name-simple runner begin-name end-name + +You can call those if you want to write your own test-runner. +@end deffn + +@subsubheading Test-runner components + +The following functions are for accessing the other components of a +test-runner. They would normally only be used to write a new +test-runner or a match-predicate. + +@deffn {Scheme Procedure} test-runner-pass-count runner + +Returns the number of tests that passed, and were expected to pass. +@end deffn + +@deffn {Scheme Procedure} test-runner-fail-count runner + +Returns the number of tests that failed, but were expected to pass. +@end deffn + +@deffn {Scheme Procedure} test-runner-xpass-count runner + +Returns the number of tests that passed, but were expected to fail. +@end deffn + +@deffn {Scheme Procedure} test-runner-xfail-count runner + +Returns the number of tests that failed, and were expected to pass. +@end deffn + +@deffn {Scheme Procedure} test-runner-skip-count runner + +Returns the number of tests or test groups that were skipped. +@end deffn + +@deffn {Scheme Procedure} test-runner-test-name runner + +Returns the name of the current test or test group, as a string. During +execution of @code{test-begin} this is the name of the test group; +during the execution of an actual test, this is the name of the +test-case. If no name was specified, the name is the empty string. +@end deffn + +@deffn {Scheme Procedure} test-runner-group-path runner + +A list of names of groups we're nested in, with the outermost group +first. +@end deffn + +@deffn {Scheme Procedure} test-runner-group-stack runner + +A list of names of groups we're nested in, with the outermost group +last. (This is more efficient than @code{test-runner-group-path}, since +it doesn't require any copying.) +@end deffn + +@deffn {Scheme Procedure} test-runner-aux-value runner +@deffnx {Scheme Procedure} test-runner-aux-value! runner on-test + +Get or set the @code{aux-value} field of a test-runner. This field is +not used by this API or the @code{test-runner-simple} test-runner, but +may be used by custom test-runners to store extra state. +@end deffn + +@deffn {Scheme Procedure} test-runner-reset runner + +Resets the state of the @var{runner} to its initial state. +@end deffn + +@subsubheading Example + +This is an example of a simple custom test-runner. +Loading this program before running a test-suite will install +it as the default test runner. + +@lisp +(define (my-simple-runner filename) + (let ((runner (test-runner-null)) + (port (open-output-file filename)) + (num-passed 0) + (num-failed 0)) + (test-runner-on-test-end! runner + (lambda (runner) + (case (test-result-kind runner) + ((pass xpass) (set! num-passed (+ num-passed 1))) + ((fail xfail) (set! num-failed (+ num-failed 1))) + (else #t)))) + (test-runner-on-final! runner + (lambda (runner) + (format port "Passing tests: ~d.~%Failing tests: ~d.~%" + num-passed num-failed) + (close-output-port port))) + runner)) +(test-runner-factory + (lambda () (my-simple-runner "/tmp/my-test.log"))) +@end lisp @node SRFI-67 @subsection SRFI-67 - Compare procedures base-commit: d579848cb5d65440af5afd9c8968628665554c22 -- 2.41.0