From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id CqyuBD4qBWBRXwAA0tVLHw (envelope-from ) for ; Mon, 18 Jan 2021 06:27:10 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id sGaKOz0qBWA1VgAAB5/wlQ (envelope-from ) for ; Mon, 18 Jan 2021 06:27:09 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 1287E94042F for ; Mon, 18 Jan 2021 06:27:09 +0000 (UTC) Received: from localhost ([::1]:53642 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l1O0K-0007BL-1H for larch@yhetil.org; Mon, 18 Jan 2021 01:27:08 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:59660) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l1NzI-0006nL-Q9 for guix-patches@gnu.org; Mon, 18 Jan 2021 01:26:04 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:34616) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1l1NzI-0001D3-IK for guix-patches@gnu.org; Mon, 18 Jan 2021 01:26:04 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1l1NzI-0003ts-EW for guix-patches@gnu.org; Mon, 18 Jan 2021 01:26:04 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#45948] [PATCH 5/5] build: test-driver.scm: Add a new '--errors-only' option. Resent-From: Maxim Cournoyer Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 18 Jan 2021 06:26:04 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45948 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 45948@debbugs.gnu.org Received: via spool by 45948-submit@debbugs.gnu.org id=B45948.161095113814920 (code B ref 45948); Mon, 18 Jan 2021 06:26:04 +0000 Received: (at 45948) by debbugs.gnu.org; 18 Jan 2021 06:25:38 +0000 Received: from localhost ([127.0.0.1]:46157 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l1Nys-0003sZ-6f for submit@debbugs.gnu.org; Mon, 18 Jan 2021 01:25:38 -0500 Received: from mail-qt1-f180.google.com ([209.85.160.180]:46492) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l1Nyl-0003rU-Qb for 45948@debbugs.gnu.org; Mon, 18 Jan 2021 01:25:32 -0500 Received: by mail-qt1-f180.google.com with SMTP id h19so10557405qtq.13 for <45948@debbugs.gnu.org>; Sun, 17 Jan 2021 22:25:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=dwh4qyXoSLILAiAytmhHLl4/osgRShSj7MzucfUFLug=; b=A6YjNY87nRt7JcWI3+h9+5Geh2aleRymUYlutfnzhDJE9g2bsaT355beurlkC2FBqa rft88rQcSohRVR8FOZ9cReljm1KUEZK1lXGGZEJ2raM2dFhABAwGQ74zAIgHZffC6ebN KV2+fI0ACs3lpXlxUNOWbuMLU4CRUFnAYJH9tLGgkpGEHOgeJFb79xAKvo3Z2HLoJVpP 5d5+mnniGO8ghzV5cMIopf9whQTlxVVYvyZD05FY+p8K+LKMdNcAYysB5sSZ6oBJOfEW V81IrXwvJxig7piEiRQrQ81OHKBlL3sxKAVGOUFEHBbwo3c2TV53v5sniDhoS3kbyDby r6mg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=dwh4qyXoSLILAiAytmhHLl4/osgRShSj7MzucfUFLug=; b=QihlMiHSlt/+mHeQ23cfXrU4UxQnDbsFED+1aIl5A+lzgiBDcgiJZwh6vP/k6beb33 DuhgnibJIyNNVJg72Gf8/NM00ibWx38mTXGOTyjbMxZvL8uu9c1ox95UcpEiBm9QVJoD GYRqldsdYnkeoQaeY/f5ZLOFmgP9z0T1MO7XkICSq9tbiqbzbZv8jjG48uhAQBcuHlPV Cs+zq9xVLLTGxz1SqVJZes9uHzspZWWiqgdlz0zcd59O5tGEy57QWxzYI5LKlH1IlGVJ tw3xlAW5wZjnvVYUrjuoptD4h8Rwmnh0EUGP7TsSlgc3Zx2l3hILPaT9XChKSq7g0ls0 jvLw== X-Gm-Message-State: AOAM530eSyRVRYZYxDRkg41xcHNmY3CbksmuXN61fFTubIBmwCze9+YS umNR7/sJ6UgyYNX6U/6Mfvk4Kf0tFVI= X-Google-Smtp-Source: ABdhPJw/ERNTEtYOqStqVWIkZZArI9GnxNkVstgu9+PydRdt0BkFet8mJIOToJ3RAohaUvmGBOSh7g== X-Received: by 2002:ac8:7153:: with SMTP id h19mr3371443qtp.118.1610951126025; Sun, 17 Jan 2021 22:25:26 -0800 (PST) Received: from localhost.localdomain (dsl-149-228.b2b2c.ca. [66.158.149.228]) by smtp.gmail.com with ESMTPSA id k187sm10266726qkc.74.2021.01.17.22.25.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 17 Jan 2021 22:25:25 -0800 (PST) From: Maxim Cournoyer Date: Mon, 18 Jan 2021 01:25:01 -0500 Message-Id: <20210118062501.27022-6-maxim.cournoyer@gmail.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210118062501.27022-1-maxim.cournoyer@gmail.com> References: <20210118062501.27022-1-maxim.cournoyer@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN X-Migadu-Spam-Score: 2.74 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20161025 header.b=A6YjNY87; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: 1287E94042F X-Spam-Score: 2.74 X-Migadu-Scanner: scn0.migadu.com X-TUID: H9q3qrBOCiOM * build-aux/test-driver.scm (show-help): Add the help text for the new '--errors-only' option. (%options): Add the errors-only option. (test-runner-gnu): Add the errors-only? parameter and update doc. Move the logging of the test data after the test has completed, so a choice can be made whether to keep it or discard it based on the value of the test result. (main): Pass the errors-only? option to the driver. * doc/guix.texi (Running the Test Suite): Document the new option. --- build-aux/test-driver.scm | 73 ++++++++++++++++++++++----------------- doc/guix.texi | 12 +++++++ 2 files changed, 54 insertions(+), 31 deletions(-) diff --git a/build-aux/test-driver.scm b/build-aux/test-driver.scm index 3ad6b0c93f..3f593abc7b 100644 --- a/build-aux/test-driver.scm +++ b/build-aux/test-driver.scm @@ -36,12 +36,15 @@ (display "Usage: test-driver --test-name=NAME --log-file=PATH --trs-file=PATH [--expect-failure={yes|no}] [--color-tests={yes|no}] - [--select=REGEXP] [--exclude=REGEXP] + [--select=REGEXP] [--exclude=REGEXP] [--errors-only={yes|no}] [--enable-hard-errors={yes|no}] [--brief={yes|no}}] [--] TEST-SCRIPT [TEST-SCRIPT-ARGUMENTS] The '--test-name', '--log-file' and '--trs-file' options are mandatory. The '--select' and '--exclude' options allow selecting or excluding individual -test cases via a regexp, respectively.\n")) +test cases via a regexp, respectively. The '--errors-only' option can be set +to \"yes\" to limit the logged test case metadata to only those test cases +that failed. When set to \"yes\", the '--brief' option disables printing the +individual test case result to the console.\n")) (define %options '((test-name (value #t) (required? #t)) @@ -49,6 +52,7 @@ test cases via a regexp, respectively.\n")) (trs-file (value #t)) (select (value #t)) (exclude (value #t)) + (errors-only (value #t)) (color-tests (value #t)) (expect-failure (value #t)) ;XXX: not implemented yet (enable-hard-errors (value #t)) ;not implemented in SRFI-64 @@ -88,27 +92,26 @@ test cases via a regexp, respectively.\n")) ;;; SRFI 64 custom test runner. ;;; -(define* (test-runner-gnu test-name #:key color? brief? +(define* (test-runner-gnu test-name #:key color? brief? errors-only? (out-port (current-output-port)) (trs-port (%make-void-port "w")) select exclude) "Return an custom SRFI-64 test runner. TEST-NAME is a string specifying the -file name of the current the test. COLOR? specifies whether to use colors, -and BRIEF?, well, you know. OUT-PORT and TRS-PORT must be output ports. -OUT-PORT defaults to the current output port, while TRS-PORT defaults to a -void port, which means no TRS output is logged. SELECT and EXCLUDE may take a -regular expression to select or exclude individual test cases based on their -names." - - (define (test-on-test-begin-gnu runner) - ;; Procedure called at the start of an individual test case, before the - ;; test expression (and expected value) are evaluated. - (let ((result (cute assq-ref (test-result-alist runner) <>))) - (format #t "test-name: ~A~%" (result 'test-name)) - (format #t "location: ~A~%" - (string-append (result 'source-file) ":" - (number->string (result 'source-line)))) - (test-display "source" (result 'source-form) #:pretty? #t))) +file name of the current the test. COLOR? specifies whether to use colors. +When BRIEF? is true, the individual test cases results are masked and only the +summary is shown. ERRORS-ONLY? reduces the amount of test case metadata +logged to only that of the failed test cases. OUT-PORT and TRS-PORT must be +output ports. OUT-PORT defaults to the current output port, while TRS-PORT +defaults to a void port, which means no TRS output is logged. SELECT and +EXCLUDE may take a regular expression to select or exclude individual test +cases based on their names." + + (define (test-skipped? runner) + (eq? 'skip (test-result-kind runner))) + + (define (test-failed? runner) + (not (or (test-passed? runner) + (test-skipped? runner)))) (define (test-on-test-end-gnu runner) ;; Procedure called at the end of an individual test case, when the result @@ -116,21 +119,29 @@ names." (let* ((results (test-result-alist runner)) (result? (cut assq <> results)) (result (cut assq-ref results <>))) - (unless brief? + (unless (or brief? (and errors-only? (test-skipped? runner))) ;; Display the result of each test case on the console. (format out-port "~A: ~A - ~A~%" (result->string (test-result-kind runner) #:colorize? color?) test-name (test-runner-test-name runner))) - (when (result? 'expected-value) - (test-display "expected-value" (result 'expected-value))) - (when (result? 'expected-error) - (test-display "expected-error" (result 'expected-error) #:pretty? #t)) - (when (result? 'actual-value) - (test-display "actual-value" (result 'actual-value))) - (when (result? 'actual-error) - (test-display "actual-error" (result 'actual-error) #:pretty? #t)) - (format #t "result: ~a~%" (result->string (result 'result-kind))) - (newline) + + (unless (and errors-only? (not (test-failed? runner))) + (format #t "test-name: ~A~%" (result 'test-name)) + (format #t "location: ~A~%" + (string-append (result 'source-file) ":" + (number->string (result 'source-line)))) + (test-display "source" (result 'source-form) #:pretty? #t) + (when (result? 'expected-value) + (test-display "expected-value" (result 'expected-value))) + (when (result? 'expected-error) + (test-display "expected-error" (result 'expected-error) #:pretty? #t)) + (when (result? 'actual-value) + (test-display "actual-value" (result 'actual-value))) + (when (result? 'actual-error) + (test-display "actual-error" (result 'actual-error) #:pretty? #t)) + (format #t "result: ~a~%" (result->string (result 'result-kind))) + (newline)) + (format trs-port ":test-result: ~A ~A~%" (result->string (test-result-kind runner)) (test-runner-test-name runner)))) @@ -157,7 +168,6 @@ names." #f)) (let ((runner (test-runner-null))) - (test-runner-on-test-begin! runner test-on-test-begin-gnu) (test-runner-on-test-end! runner test-on-test-end-gnu) (test-runner-on-group-end! runner test-on-group-end-gnu) (test-runner-on-bad-end-name! runner test-on-bad-end-name-simple) @@ -215,6 +225,7 @@ names." (test-runner-gnu test-name #:color? color-tests #:brief? (option->boolean opts 'brief) + #:errors-only? (option->boolean opts 'errors-only) #:out-port out #:trs-port trs) (test-apply test-specifier (lambda _ diff --git a/doc/guix.texi b/doc/guix.texi index d59d2806cb..ac20a541f5 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -927,6 +927,18 @@ export SCM_LOG_DRIVER_FLAGS="--select=^transaction-upgrade-entry" make check TESTS="tests/packages.scm" @end example +Those wishing to inspect the results of failed tests directly from the +command line can add the @option{--errors-only=yes} option to the +@code{SCM_LOG_DRIVER_FLAGS} makefile variable and set the @code{VERBOSE} +Automake makefile variable, as in: + +@example +make check SCM_LOG_DRIVER_FLAGS="--brief=no --errors-only=yes" VERBOSE=1 +@end example + +@xref{Parallel Test Harness,,,automake,GNU Automake} for more +information about the Automake Parallel Test Harness. + Upon failure, please email @email{bug-guix@@gnu.org} and attach the @file{test-suite.log} file. Please specify the Guix version being used as well as version numbers of the dependencies (@pxref{Requirements}) in -- 2.29.2