From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id 4UwPDzYqBWBQXwAA0tVLHw (envelope-from ) for ; Mon, 18 Jan 2021 06:27:02 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id gBuvCjYqBWCqVAAAB5/wlQ (envelope-from ) for ; Mon, 18 Jan 2021 06:27:02 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id E501294042F for ; Mon, 18 Jan 2021 06:27:01 +0000 (UTC) Received: from localhost ([::1]:52814 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l1O0C-0006oD-Tn for larch@yhetil.org; Mon, 18 Jan 2021 01:27:00 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:59656) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l1NzI-0006mo-A8 for guix-patches@gnu.org; Mon, 18 Jan 2021 01:26:04 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:34614) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1l1NzI-0001Cu-3I for guix-patches@gnu.org; Mon, 18 Jan 2021 01:26:04 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1l1NzI-0003tk-0g for guix-patches@gnu.org; Mon, 18 Jan 2021 01:26:04 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#45948] [PATCH 4/5] build: test-driver.scm: Provide the ability to filter on test case names. Resent-From: Maxim Cournoyer Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 18 Jan 2021 06:26:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45948 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 45948@debbugs.gnu.org Received: via spool by 45948-submit@debbugs.gnu.org id=B45948.161095113414909 (code B ref 45948); Mon, 18 Jan 2021 06:26:03 +0000 Received: (at 45948) by debbugs.gnu.org; 18 Jan 2021 06:25:34 +0000 Received: from localhost ([127.0.0.1]:46155 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l1Nyn-0003sI-MT for submit@debbugs.gnu.org; Mon, 18 Jan 2021 01:25:34 -0500 Received: from mail-qv1-f50.google.com ([209.85.219.50]:39113) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l1Nyk-0003rS-MY for 45948@debbugs.gnu.org; Mon, 18 Jan 2021 01:25:31 -0500 Received: by mail-qv1-f50.google.com with SMTP id s6so7036781qvn.6 for <45948@debbugs.gnu.org>; Sun, 17 Jan 2021 22:25:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=YPxoJE0otRQnTRVl637Mr60xbpEyNm3wV1l7DNEQ+O8=; b=W1xIkCWWmhPS7flrDWDCFGiYMHzHg/vfffyAht/YNylO7dEVhwNOlJMTGPvO7nYi3I qabiVhMnYtish1xg1GIAeh/StRuuRXnw+mH4s6LajNxahvLLNi+pXeC4/TNGKuBktuel rmVhfI9HZSHaOakw6J+OuikcBjm1hwMdWy4fzTwNSWZxkDsldtZjr3iajs2txAliewRq fNvPsPOaqMLorx1v0jN6gez99ztteYxlNbEMGeE6L53vcPwo8eJwhF+wIfYLbC2JFr4W g7MUBd/OaV6pF4Bw5qhlrMUwazoVVluDV1vlpPlzqDCZXiWTuiZXJVT34tfBatBV1tJL /jKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=YPxoJE0otRQnTRVl637Mr60xbpEyNm3wV1l7DNEQ+O8=; b=X0q62cFYLob9dl1ftuIBVzOB9GDppILhKZ7jj4PBdipHtmRDEt4Je4kWBvAvTcYYzG 1P3sKqEIELWzlT6+84Kf4E7VitVAfJFk7BbpthBzKrs9ZSJi/WOtDIrNkw9jrgBrwkXH 1GI8Y5q861HTj2cTVJFNlabipcmd3TFdmRKUiae7boeb36B5FyMXTshmdVW7JgTcKQxG AS/0h7rWepEzLmZnHwQbM44/xihCnrTqTXA6o2NGafjnpprGjMaz1NpuT7YTElnNQ1pT gpBEBMVDqjEbsrl2h7t+LcuLozCCK2MsEZio72wdCnETp6y5+bpIdFggI9A4frKJFCSB OCzQ== X-Gm-Message-State: AOAM530tKx4CqiT4vSfZVGIm98rqdRYrxyRA+4r8/seaWu0XUynECbbi VJuL0qW0Sfw8fOdwS811NE4Gc+bAmhg= X-Google-Smtp-Source: ABdhPJxIRNAuHwqyw1Dqqq3JmyquT6aPHkew8aQ4KPUv9lYcSpqMoHsW/QoCRaM1VOWJ7WyLVGUDCA== X-Received: by 2002:a05:6214:b2f:: with SMTP id w15mr23182347qvj.8.1610951124943; Sun, 17 Jan 2021 22:25:24 -0800 (PST) Received: from localhost.localdomain (dsl-149-228.b2b2c.ca. [66.158.149.228]) by smtp.gmail.com with ESMTPSA id k187sm10266726qkc.74.2021.01.17.22.25.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 17 Jan 2021 22:25:23 -0800 (PST) From: Maxim Cournoyer Date: Mon, 18 Jan 2021 01:25:00 -0500 Message-Id: <20210118062501.27022-5-maxim.cournoyer@gmail.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210118062501.27022-1-maxim.cournoyer@gmail.com> References: <20210118062501.27022-1-maxim.cournoyer@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN X-Migadu-Spam-Score: 2.74 Authentication-Results: aspmx1.migadu.com; dkim=fail ("body hash did not verify") header.d=gmail.com header.s=20161025 header.b=W1xIkCWW; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: E501294042F X-Spam-Score: 2.74 X-Migadu-Scanner: scn1.migadu.com X-TUID: fS9W2WmqROMl * build-aux/test-driver.scm (show-help): Add help text for the new --select and --exclude options. (%options): Add the new select and exclude options. (test-runner-gnu): Pass them to the test runner. Update doc. (test-match-name*, test-match-name*/negated, %test-match-all): New variables. (main): Compute the test specifier based on the values of the new options and apply it to the current test runner when running the test file. * doc/guix.texi (Running the Test Suite): Document the new options. --- build-aux/test-driver.scm | 66 ++++++++++++++++++++++++++++++++------- doc/guix.texi | 12 +++++++ 2 files changed, 66 insertions(+), 12 deletions(-) diff --git a/build-aux/test-driver.scm b/build-aux/test-driver.scm index 767a9fb25d..3ad6b0c93f 100644 --- a/build-aux/test-driver.scm +++ b/build-aux/test-driver.scm @@ -27,6 +27,8 @@ (use-modules (ice-9 getopt-long) (ice-9 pretty-print) + (ice-9 regex) + (srfi srfi-1) (srfi srfi-26) (srfi srfi-64)) @@ -34,14 +36,19 @@ (display "Usage: test-driver --test-name=NAME --log-file=PATH --trs-file=PATH [--expect-failure={yes|no}] [--color-tests={yes|no}] + [--select=REGEXP] [--exclude=REGEXP] [--enable-hard-errors={yes|no}] [--brief={yes|no}}] [--] TEST-SCRIPT [TEST-SCRIPT-ARGUMENTS] -The '--test-name', '--log-file' and '--trs-file' options are mandatory.\n")) +The '--test-name', '--log-file' and '--trs-file' options are mandatory. +The '--select' and '--exclude' options allow selecting or excluding individual +test cases via a regexp, respectively.\n")) (define %options '((test-name (value #t) (required? #t)) (log-file (value #t)) (trs-file (value #t)) + (select (value #t)) + (exclude (value #t)) (color-tests (value #t)) (expect-failure (value #t)) ;XXX: not implemented yet (enable-hard-errors (value #t)) ;not implemented in SRFI-64 @@ -76,14 +83,22 @@ The '--test-name', '--log-file' and '--trs-file' options are mandatory.\n")) "") ;no color result))) + +;;; +;;; SRFI 64 custom test runner. +;;; + (define* (test-runner-gnu test-name #:key color? brief? (out-port (current-output-port)) - (trs-port (%make-void-port "w"))) + (trs-port (%make-void-port "w")) + select exclude) "Return an custom SRFI-64 test runner. TEST-NAME is a string specifying the file name of the current the test. COLOR? specifies whether to use colors, and BRIEF?, well, you know. OUT-PORT and TRS-PORT must be output ports. OUT-PORT defaults to the current output port, while TRS-PORT defaults to a -void port, which means no TRS output is logged." +void port, which means no TRS output is logged. SELECT and EXCLUDE may take a +regular expression to select or exclude individual test cases based on their +names." (define (test-on-test-begin-gnu runner) ;; Procedure called at the start of an individual test case, before the @@ -148,6 +163,24 @@ void port, which means no TRS output is logged." (test-runner-on-bad-end-name! runner test-on-bad-end-name-simple) runner)) + +;;; +;;; SRFI 64 test specifiers. +;;; +(define (test-match-name* regexp) + "Return a test specifier that matches a test name against REGEXP." + (lambda (runner) + (string-match regexp (test-runner-test-name runner)))) + +(define (test-match-name*/negated regexp) + "Return a negated test specifier version of test-match-name*." + (lambda (runner) + (not (string-match regexp (test-runner-test-name runner))))) + +;;; XXX: test-match-all is a syntax, which isn't convenient to use with a list +;;; of test specifiers computed at run time. +(define %test-match-all (@@ (srfi srfi-64) %test-match-all)) + ;;; ;;; Entry point. @@ -158,15 +191,22 @@ void port, which means no TRS output is logged." (option (cut option-ref opts <> <>))) (cond ((option 'help #f) (show-help)) - ((option 'version #f) (format #t "test-driver.scm ~A" script-version)) + ((option 'version #f) (format #t "test-driver.scm ~A" 'script-version)) (else - (let ((log (and=> (option 'log-file #f) (cut open-file <> "w0"))) - (trs (and=> (option 'trs-file #f) (cut open-file <> "wl"))) - (out (duplicate-port (current-output-port) "wl")) - (test-name (option 'test-name #f)) - (color-tests (if (assoc 'color-tests opts) - (option->boolean opts 'color-tests) - #t))) + (let* ((log (and=> (option 'log-file #f) (cut open-file <> "w0"))) + (trs (and=> (option 'trs-file #f) (cut open-file <> "wl"))) + (out (duplicate-port (current-output-port) "wl")) + (test-name (option 'test-name #f)) + (select (option 'select #f)) + (exclude (option 'exclude #f)) + (test-specifiers (filter-map + identity + (list (and=> select test-match-name*) + (and=> exclude test-match-name*/negated)))) + (test-specifier (apply %test-match-all test-specifiers)) + (color-tests (if (assoc 'color-tests opts) + (option->boolean opts 'color-tests) + #t))) (when log (redirect-port log (current-output-port)) (redirect-port log (current-warning-port)) @@ -176,7 +216,9 @@ void port, which means no TRS output is logged." #:color? color-tests #:brief? (option->boolean opts 'brief) #:out-port out #:trs-port trs) - (load-from-path test-name)) + (test-apply test-specifier + (lambda _ + (load-from-path test-name)))) (and=> log close-port) (and=> trs close-port) (close-port out)))) diff --git a/doc/guix.texi b/doc/guix.texi index 194bb3a314..d59d2806cb 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -915,6 +915,18 @@ the @code{SCM_LOG_DRIVER_FLAGS} makefile variable as in this example: make check TESTS="tests/base64.scm" SCM_LOG_DRIVER_FLAGS="--brief=no" @end example +The underlying SRFI 64 custom Automake test driver used for the 'check' +test suite (located at @file{build-aux/test-driver.scm}) also allows +selecting which test cases to run at a finer level, via its +@option{--select} and @option{--exclude} options. Here's an example, to +run all the test cases from the @file{tests/packages.scm} test file +whose names start with ``transaction-upgrade-entry'': + +@example +export SCM_LOG_DRIVER_FLAGS="--select=^transaction-upgrade-entry" +make check TESTS="tests/packages.scm" +@end example + Upon failure, please email @email{bug-guix@@gnu.org} and attach the @file{test-suite.log} file. Please specify the Guix version being used as well as version numbers of the dependencies (@pxref{Requirements}) in -- 2.29.2