From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1.migadu.com ([2001:41d0:303:e16b::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms13.migadu.com with LMTPS id YBpjGEyXlWb7uQAA62LTzQ:P1 (envelope-from ) for ; Mon, 15 Jul 2024 21:40:28 +0000 Received: from aspmx1.migadu.com ([2001:41d0:303:e16b::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1.migadu.com with LMTPS id YBpjGEyXlWb7uQAA62LTzQ (envelope-from ) for ; Mon, 15 Jul 2024 23:40:28 +0200 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=wolfsden.cz header.s=mail header.b=x2mR4W3F; dkim=fail ("headers rsa verify failed") header.d=wolfsden.cz header.s=mail header.b=Wql17Civ; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org"; dmarc=fail reason="SPF not aligned (relaxed)" header.from=wolfsden.cz (policy=none) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1721079628; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:list-id:list-help:list-unsubscribe:list-subscribe: list-post:dkim-signature; bh=Kya6EopJmYpK53p92dL9oiK6BkEglBvC//MXunotRcw=; b=Lnd9ZaSUDP9vWB4luIv3EB93EXvUNHyOiFzfzleEIi8pUAaRETjxgUphEoekqwzSB35y/K vQwYeOUot69TVDU9J5OLOJCB+d5Mtpin9PAvwJypxm06ux4m2Lbnv/R6TINirJ+iXUd98t CsdyrkJS3FTI0cnxIyzYqtioLqEjGUzhrKD4c2+21qXlwRyFJKZzZgMFZXRO1H5GqpYQOn 5f17iVHGv3xNOhk2mdekHec5NJKC8mQR/FHMDkI9xl6CUk1B5AwqalgEmsc08CAxMoUQqB mPgS+BysNiWurE0TtEaG3RcebpfuYFvDJfl2VIR8VMap8mcxB0KbaH4gxdCa/Q== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=wolfsden.cz header.s=mail header.b=x2mR4W3F; dkim=fail ("headers rsa verify failed") header.d=wolfsden.cz header.s=mail header.b=Wql17Civ; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org"; dmarc=fail reason="SPF not aligned (relaxed)" header.from=wolfsden.cz (policy=none) ARC-Seal: i=1; s=key1; d=yhetil.org; t=1721079628; a=rsa-sha256; cv=none; b=roP9Yr9YoyPBwf5v/6g1xenx4t64FpCY/YI8FRXcmog0lL6c93r6VcaWcJCJQnHHk7v49G BX3FIUdz2hv3H6gYaEXr/vsTGZ1M7OIx4Oh9RCiSyr3fZ4NP6Dt0G3brNuI+IVKnv4W7Ke zzchytOBPz7rGyHN6lhHBMHvmw4ae4g0alJ6UBBvqkLumzi6p935ak9gA6FUTw6d0P8nBf rrL+s3dlMl3fxu73fBEXqkb9f1TC9BGV8YNOwBvS7ogltX/Bxrz7dn/joIiOdxKXNfCbK6 xQVV76OeQd1stZMysukpOXobM0iFo4LCsPX/bkP+D01qf0ZqcFOPw0dBkmaI2g== Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id B958264A0F for ; Mon, 15 Jul 2024 23:40:27 +0200 (CEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sTTQQ-0003YF-6e; Mon, 15 Jul 2024 17:40:02 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sTTQO-0003TL-Ih for bug-guix@gnu.org; Mon, 15 Jul 2024 17:40:00 -0400 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1sTTQO-0000Km-AJ for bug-guix@gnu.org; Mon, 15 Jul 2024 17:40:00 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1sTTQQ-0003K7-6p for bug-guix@gnu.org; Mon, 15 Jul 2024 17:40:02 -0400 X-Loop: help-debbugs@gnu.org Subject: bug#72130: test-driver.scm produces .trs file with undefined behavior Resent-From: Tomas Volf <~@wolfsden.cz> Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Mon, 15 Jul 2024 21:40:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 72130 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: 72130@debbugs.gnu.org Cc: Mathieu Lirzin , Maxim Cournoyer X-Debbugs-Original-To: bug-guix@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.172107959512752 (code B ref -1); Mon, 15 Jul 2024 21:40:02 +0000 Received: (at submit) by debbugs.gnu.org; 15 Jul 2024 21:39:55 +0000 Received: from localhost ([127.0.0.1]:60507 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sTTQH-0003JY-Ml for submit@debbugs.gnu.org; Mon, 15 Jul 2024 17:39:54 -0400 Received: from lists.gnu.org ([209.51.188.17]:41752) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <~@wolfsden.cz>) id 1sTTQF-0003JR-Sp for submit@debbugs.gnu.org; Mon, 15 Jul 2024 17:39:53 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from <~@wolfsden.cz>) id 1sTTQD-0003NM-Cc for bug-guix@gnu.org; Mon, 15 Jul 2024 17:39:49 -0400 Received: from wolfsden.cz ([37.205.8.62]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from <~@wolfsden.cz>) id 1sTTQA-0000K2-2g; Mon, 15 Jul 2024 17:39:49 -0400 Received: by wolfsden.cz (Postfix, from userid 104) id 3D9C824BFC9; Mon, 15 Jul 2024 21:39:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=wolfsden.cz; s=mail; t=1721079580; bh=FoeZDW6q3R2Gko+xntJieIcp6cDqEHv8ru8pqkzzq2g=; h=Date:From:To:Cc:Subject; b=x2mR4W3Fs9MYhsuE146an3Yx/NINwTRIxos9XYtosodrQIdLuvDFQ12cV4HSb96AY dShTTh0VoUENaSohTHFwZXpugsHRsZC8NWNbh9UCb4BMzycXQN1aLwhkqbQzsUcPxE 6EWhvsNcow1kWz47dSmQtfCXtybmUcpBBm/8O4t/ovNOpl1Kfe96I03R592re9NsJp t6+oWXae9CtkxbUH0QbNogo25gieJnqFqXXgZ6ms3mDT1D8ycDdjgi43fxiDLSDmnv +2rAncndYJILTp8lt822UmPd2V5RDXXFDLjYGx4odAvy0hiKRueWtQ1/duBvL0gtdw 2Uf/HU9ttdg9lckrUZGy5NSngoCk9cvmQ31OPDmbNL5ubHOJ9xkyan3j3VlRI29Lp+ JU48RfcJOoZDXfV4IJKZCWOLUUpg5zd3KZ6+JZPyuk9H57T4mrNc0Gx+6YWaZGY/tc SytQUaYRRzUk5J3KRPyOEgqf5lQ208AlT8Eo1n551F6hn4NBSnZ6jHtQlw5OQo5lOZ VTmdo6yFN7ojFiNRJ1JeXqjrKqiupqNWQ4+kBKKuvGdIqZC8cU6tRwdF8hZ+Lgz8Hp Vkon3Mf4mdjW7AvfRcUr81bzP64UHZDsBdNERKiyHKLD1uZ53+97yFNreVMb7hie+j jDvLWHXUE9sPNINeI7cJiLGk= Received: from localhost (unknown [146.70.134.139]) by wolfsden.cz (Postfix) with ESMTPSA id 3DAC124BBD2; Mon, 15 Jul 2024 21:39:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=wolfsden.cz; s=mail; t=1721079579; bh=FoeZDW6q3R2Gko+xntJieIcp6cDqEHv8ru8pqkzzq2g=; h=Date:From:To:Cc:Subject; b=Wql17CivJlwfWTHKmJvCfI7xr3KrTpTIJYNg0SPYq+rx70UU/PLJqHUAcWhl7p+8J lFRRhyR2QkxPMs5Dww4c9c3CUdoSlDuCawlfrszBA5/W45fW81DrL9d/6PpS85Hh7k rpjxKal1C2E/MWGSQMf4YCuv7Yopv+a1TqgnmLXXpMhidZ6eA1RUOAzqvG2LpJ/vvS Chf5Q6axBGntUYuCfY4qar+AwOtSRW65FIPOF/jSnBZlqmTksm1+dzHHrnvisxa+nn plXHvtOlfHMpKhbYQIWBZuYduwU4h5DagRoblADAa/qZ4X3LTDltfu9Afp9MmRiFmT b9Zn18yh5NmN6x/fAO9tqnwN/VLdzTf6pa5GeVHM9KN94BqM9gkrGP6gHfGjiAe0oV mnvId0mNN9KuNYVMJKcotNVEXqVLdFUWwlsk7VKGsKYeku2t8L+tZ9ZvcvKTSw8VEC tXEfRUA+YFMyeHJySIyCGRLiuQv9y9dWeZq55Hh3AyJhQjrXuwmSC//4hVrr6Nsrfu saeBXgWdKTRn6+3kHW9qBqiOgaKLYMOTwp7zDB5XKNJ6KrWDy6ZRJJRvzes2V4jbtC I+DQnRTjPuELGEjBa0IHy/9+JxOpg5GWKiINRTR0yEvvv/9D5K4cS6vxFmFFaRGiej Sn82a9amB30mEmu9+eT5Xdx8= Date: Mon, 15 Jul 2024 23:39:38 +0200 From: Tomas Volf <~@wolfsden.cz> Message-ID: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="D7V7UDJ/WFLnWPqb" Content-Disposition: inline Received-SPF: pass client-ip=37.205.8.62; envelope-from=~@wolfsden.cz; helo=wolfsden.cz X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: bug-guix-bounces+larch=yhetil.org@gnu.org X-Migadu-Country: US X-Migadu-Flow: FLOW_IN X-Migadu-Queue-Id: B958264A0F X-Migadu-Scanner: mx13.migadu.com X-Migadu-Spam-Score: -7.10 X-Spam-Score: -7.10 X-TUID: rTzo4gaiOGEv --D7V7UDJ/WFLnWPqb Content-Type: multipart/mixed; boundary="jhXV0P+yS+fKLGww" Content-Disposition: inline --jhXV0P+yS+fKLGww Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Hello, currently there are two issues with build-aux/test-driver.scm. 1. :test-global-result: is mistyped as :global-test-result:. 2. Summarizing metadata are outputted on end of each test-group, meaning they will be present multiple times, if there is more than one test group. First is somewhat fine, since, per the manual page, unknown keys are (for now) ignored. However second is explicitly stated to be undefined behavior, so it should be fixed. I am CC-ing both authors listed in the test-driver.scm, since I have no idea where the upstream is (it seems to be just copied around between projects?). You can find fixes for the first (0001) and second (0002) issues attached to this email. I am also including two additional patches (0003 refining :test-global-result: for XPASS as XXX mentioned, 0005 improving support for test group since it was somewhat lacking) which I consider useful and I have applied them to my local copy of test-driver.scm file. Feel free to apply them, or ignore them, as you see fit. Have a nice day, Tomas Volf -- There are only two hard things in Computer Science: cache invalidation, naming things and off-by-one errors. --jhXV0P+yS+fKLGww Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: attachment; filename="0001-build-test-driver.scm-Fix-test-global-result-.trs-me.patch" Content-Transfer-Encoding: quoted-printable =46rom ff4a7a69e7f35ddf80287b230ac1b377e54b43e9 Mon Sep 17 00:00:00 2001 =46rom: Tomas Volf <~@wolfsden.cz> Date: Sun, 14 Jul 2024 01:08:49 +0200 Subject: [PATCH 1/5] build: test-driver.scm: Fix :test-global-result: .trs metadata. According to the manual page, the correct metadata key is :test-global-resu= lt: instead of the current :global-test-result:. * build-aux/test-driver.scm (test-on-group-end-gnu): Fix :test-global-resul= t: metadata output. --- build-aux/test-driver.scm | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/build-aux/test-driver.scm b/build-aux/test-driver.scm index 9a0dcde..471333b 100755 --- a/build-aux/test-driver.scm +++ b/build-aux/test-driver.scm @@ -3,7 +3,7 @@ exec guile --no-auto-compile -e main -s "$0" "$@" !# ;;;; test-driver.scm - Guile test driver for Automake testsuite harness =20 -(define script-version "2023-12-08.16") ;UTC +(define script-version "2024-07-14.10") ;UTC =20 ;;; Copyright =A9 2015, 2016 Mathieu Lirzin ;;; Copyright =A9 2021 Maxim Cournoyer @@ -178,7 +178,7 @@ cases based on their names." (skip (or (positive? (test-runner-skip-count runner)) (positive? (test-runner-xfail-count runner))))) ;; XXX: The global results need some refinements for XPASS. - (format trs-port ":global-test-result: ~A~%" + (format trs-port ":test-global-result: ~A~%" (if fail "FAIL" (if skip "SKIP" "PASS"))) (format trs-port ":recheck: ~A~%" (if fail "yes" "no")) --=20 2.45.2 --jhXV0P+yS+fKLGww Content-Type: text/plain; charset=utf-8 Content-Disposition: attachment; filename="0002-build-test-driver.scm-Output-singleton-metadata-just.patch" Content-Transfer-Encoding: quoted-printable =46rom 8357a98e7c0c8690be7fe2903a310a3b93b2ab8f Mon Sep 17 00:00:00 2001 =46rom: Tomas Volf <~@wolfsden.cz> Date: Sun, 14 Jul 2024 13:00:14 +0200 Subject: [PATCH 2/5] build: test-driver.scm: Output singleton metadata just once. Current implementation printed metadata supposed to be present just once per =2Etrs file on the end of each test group. According to the automake's man= ual that is undefined behavior. This commit fixes it by printing that metadata just once, after all tests did run. Since there is no built-in hook that could be used for that (test-runner-on-final runs on *each* outermost test-end), I introduced new `finalize' procedure that need to be called by the user. Possibly not = the most elegant solution, but since we are the only user, it works fine and produces actually valid .trs file. That also means there is no longer any use for test-runner-on-test-end!. * build-aux/test-driver.scm (test-runner-gnu): Define new procedure `finali= ze' and return it together with the runner. Do not call test-runner-on-group-end!. (main): Call the `finalize' after all tests are done. --- build-aux/test-driver.scm | 59 +++++++++++++++++++++------------------ 1 file changed, 32 insertions(+), 27 deletions(-) diff --git a/build-aux/test-driver.scm b/build-aux/test-driver.scm index 471333b..71bca73 100755 --- a/build-aux/test-driver.scm +++ b/build-aux/test-driver.scm @@ -3,10 +3,11 @@ exec guile --no-auto-compile -e main -s "$0" "$@" !# ;;;; test-driver.scm - Guile test driver for Automake testsuite harness =20 -(define script-version "2024-07-14.10") ;UTC +(define script-version "2024-07-14.11") ;UTC =20 ;;; Copyright =C2=A9 2015, 2016 Mathieu Lirzin ;;; Copyright =C2=A9 2021 Maxim Cournoyer +;;; Copyright =C2=A9 2024 Tomas Volf <~@wolfsden.cz> ;;; ;;; This program is free software; you can redistribute it and/or modify it ;;; under the terms of the GNU General Public License as published by @@ -35,7 +36,8 @@ exec guile --no-auto-compile -e main -s "$0" "$@" (srfi srfi-1) (srfi srfi-19) (srfi srfi-26) - (srfi srfi-64)) + (srfi srfi-64) + (srfi srfi-71)) =20 (define (show-help) (display "Usage: @@ -105,15 +107,18 @@ case is shown.\n")) (out-port (current-output-port)) (trs-port (%make-void-port "w")) select exclude) - "Return an custom SRFI-64 test runner. TEST-NAME is a string specifying= the -file name of the current the test. COLOR? specifies whether to use colors. -When BRIEF? is true, the individual test cases results are masked and only= the -summary is shown. ERRORS-ONLY? reduces the amount of test case metadata -logged to only that of the failed test cases. OUT-PORT and TRS-PORT must = be -output ports. OUT-PORT defaults to the current output port, while TRS-PORT -defaults to a void port, which means no TRS output is logged. SELECT and -EXCLUDE may take a regular expression to select or exclude individual test -cases based on their names." + "Return an custom SRFI-64 test runner and a finalize procedure. TEST-NA= ME +is a string specifying the file name of the current the test. COLOR? +specifies whether to use colors. When BRIEF? is true, the individual test +cases results are masked and only the summary is shown. ERRORS-ONLY? redu= ces +the amount of test case metadata logged to only that of the failed test ca= ses. +OUT-PORT and TRS-PORT must be output ports. OUT-PORT defaults to the curr= ent +output port, while TRS-PORT defaults to a void port, which means no TRS ou= tput +is logged. SELECT and EXCLUDE may take a regular expression to select or +exclude individual test cases based on their names. + +After all your tests are finished, you need to call the finalize procedure= to +do the final reporting." =20 (define test-cases-start-time (make-hash-table)) =20 @@ -171,8 +176,8 @@ cases based on their names." (result->string (test-result-kind runner)) (test-runner-test-name runner) time-elapsed-seconds))) =20 - (define (test-on-group-end-gnu runner) - ;; Procedure called by a 'test-end', including at the end of a test-gr= oup. + (define (finalize runner) + "Procedure to call after all tests finish to do the final reporting." (let ((fail (or (positive? (test-runner-fail-count runner)) (positive? (test-runner-xpass-count runner)))) (skip (or (positive? (test-runner-skip-count runner)) @@ -189,15 +194,14 @@ cases based on their names." (format out-port "~A: ~A~%" (result->string (if fail 'fail (if skip 'skip 'pass)) #:colorize? color?) - test-name)) - #f)) + test-name)))) =20 (let ((runner (test-runner-null))) (test-runner-on-test-begin! runner test-on-test-begin-gnu) (test-runner-on-test-end! runner test-on-test-end-gnu) - (test-runner-on-group-end! runner test-on-group-end-gnu) (test-runner-on-bad-end-name! runner test-on-bad-end-name-simple) - runner)) + (values runner + (=CE=BB () (finalize runner))))) =20 =0C ;;; @@ -257,17 +261,18 @@ cases based on their names." (redirect-port log (current-output-port)) (redirect-port log (current-warning-port)) (redirect-port log (current-error-port))) - (test-with-runner - (test-runner-gnu test-name - #:color? color-tests - #:brief? (option->boolean opts 'brief) - #:errors-only? (option->boolean opts 'errors-= only) - #:show-duration? (option->boolean - opts 'show-duration) - #:out-port out #:trs-port trs) - (test-apply test-specifier + (let ((runner finalize (test-runner-gnu + test-name + #:color? color-tests + #:brief? (option->boolean opts 'brief) + #:errors-only? (option->boolean opts 'erro= rs-only) + #:show-duration? (option->boolean + opts 'show-duration) + #:out-port out #:trs-port trs))) + (test-apply runner test-specifier (lambda _ - (load-from-path test-name)))) + (load-from-path test-name))) + (finalize)) (and=3D> log close-port) (and=3D> trs close-port) (close-port out)))) --=20 2.45.2 --jhXV0P+yS+fKLGww Content-Type: text/plain; charset=utf-8 Content-Disposition: attachment; filename="0003-build-test-driver.scm-Refine-the-global-test-result.patch" Content-Transfer-Encoding: quoted-printable =46rom 250cf5af8324bc09591f178c868f409574692de0 Mon Sep 17 00:00:00 2001 =46rom: Tomas Volf <~@wolfsden.cz> Date: Sun, 14 Jul 2024 22:45:16 +0200 Subject: [PATCH 3/5] build: test-driver.scm: Refine the global test result. MIME-Version: 1.0 Content-Type: text/plain; charset=3DUTF-8 Content-Transfer-Encoding: 8bit The :test-global-result: .trs metadata contained just either FAIL, SKIP or PASS with a comment that further refinements are required for XPASS. The description of :test-global-result: is in the manual is as follows: This is used to declare the "global result" of the script. Currently, the value of this field is needed only to be reported (more or less verbatim) in the generated global log file =E2=80=98$(TEST_SUITE_LOG)=E2=80=99, so it=E2=80=99s quite free-form. = For example, a test script which runs 10 test cases, 6 of which pass and 4 of which are skipped, could reasonably have a =E2=80=98PASS/SKIP=E2=80=99 value for= this field, while a test script which runs 19 successful tests and one failed test could have an =E2=80=98ALMOST PASSED=E2=80=99 value. As we can see, the examples as `PASS/SKIP' and `ALMOST PASSED', so there is= no need to stick to strict model. Hence this commit changes the resulting val= ue to be comma-separated list of PASS, FAIL, XPASS, XFAIL and SKIP. The respective elements are present only when the count of tests with such a result is positive. In practice, that should usually produce lines such as :test-global-result: PASS,FAIL or :test-global-result: PASS * build-aux/test-driver.scm (test-runner-gnu)[finalize]: Refine the output = of :test-global-result:. --- build-aux/test-driver.scm | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-) diff --git a/build-aux/test-driver.scm b/build-aux/test-driver.scm index 71bca73..9d412e1 100755 --- a/build-aux/test-driver.scm +++ b/build-aux/test-driver.scm @@ -3,7 +3,7 @@ exec guile --no-auto-compile -e main -s "$0" "$@" !# ;;;; test-driver.scm - Guile test driver for Automake testsuite harness =20 -(define script-version "2024-07-14.11") ;UTC +(define script-version "2024-07-14.20") ;UTC =20 ;;; Copyright =C2=A9 2015, 2016 Mathieu Lirzin ;;; Copyright =C2=A9 2021 Maxim Cournoyer @@ -182,9 +182,20 @@ do the final reporting." (positive? (test-runner-xpass-count runner)))) (skip (or (positive? (test-runner-skip-count runner)) (positive? (test-runner-xfail-count runner))))) - ;; XXX: The global results need some refinements for XPASS. - (format trs-port ":test-global-result: ~A~%" - (if fail "FAIL" (if skip "SKIP" "PASS"))) + (format trs-port ":test-global-result: ~{~A~^,~}~%" + (filter-map (=CE=BB (proc str) + (let ((n (proc runner))) + (if (positive? n) str #f))) + (list test-runner-pass-count + test-runner-fail-count + test-runner-xpass-count + test-runner-xfail-count + test-runner-skip-count) + (list "PASS" + "FAIL" + "XPASS" + "XFAIL" + "SKIP"))) (format trs-port ":recheck: ~A~%" (if fail "yes" "no")) (format trs-port ":copy-in-global-log: ~A~%" --=20 2.45.2 --jhXV0P+yS+fKLGww Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: attachment; filename="0005-build-test-driver.scm-Utilize-test-runner-group-path.patch" Content-Transfer-Encoding: quoted-printable =46rom a3277fbf435ca6d3f732ca690513b975e0d66f08 Mon Sep 17 00:00:00 2001 =46rom: Tomas Volf <~@wolfsden.cz> Date: Mon, 15 Jul 2024 22:53:02 +0200 Subject: [PATCH 5/5] build: test-driver.scm: Utilize test-runner-group-path. Test groups were not used in any meaningful way. The group path was not printed and it was not used in test selection mechanism. I think groups are useful, and it is nice to be able to, for example, run tests from a single group. This commit does two things. First, it changes the test reporting to inclu= de the value returned from test-runner-group-path, so you will know not only t= he test name, but the test group(s) as well. And second, it changes the test selection (and exclusion) process to match against the "full" test name, so group path + test name. Hence (test-begin "failing tests") (test-equal "this should fail" 1 2) (test-end) will, depending on the output location, produce following text. =2Etrs: :test-result: FAIL failing tests: this should fail [0.000s] :test-global-result: FAIL :recheck: yes :copy-in-global-log: yes =2Elog: test-name: failing tests: this should fail location: test.scm:140 source: + (test-equal "this should fail" 1 2) expected-value: 1 actual-value: 2 result: FAIL stdout: FAIL: test.scm - failing tests: this should fail [0.000s] * build-aux/test-driver.scm (current-test-full-name): New procedure. (test-runner-gnu): Use current-test-full-name instead of test-runner-test-name. (test-match-name*): Match against current-test-full-name. (test-match-name*/negated): Rewrite in terms of test-match-name*. --- build-aux/test-driver.scm | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-) diff --git a/build-aux/test-driver.scm b/build-aux/test-driver.scm index 9d412e1..8d39046 100755 --- a/build-aux/test-driver.scm +++ b/build-aux/test-driver.scm @@ -3,7 +3,7 @@ exec guile --no-auto-compile -e main -s "$0" "$@" !# ;;;; test-driver.scm - Guile test driver for Automake testsuite harness =20 -(define script-version "2024-07-14.20") ;UTC +(define script-version "2024-07-15.15") ;UTC =20 ;;; Copyright =A9 2015, 2016 Mathieu Lirzin ;;; Copyright =A9 2021 Maxim Cournoyer @@ -102,6 +102,12 @@ case is shown.\n")) ;;; SRFI 64 custom test runner. ;;; =20 +(define (current-test-full-name runner) + "Get full name (test group path + name) of current test." + (format #f "~{~a~^/~}: ~a" + (test-runner-group-path runner) + (test-runner-test-name runner))) + (define* (test-runner-gnu test-name #:key color? brief? errors-only? show-duration? (out-port (current-output-port)) @@ -125,7 +131,7 @@ do the final reporting." (define (test-on-test-begin-gnu runner) ;; Procedure called at the start of an individual test case, before the ;; test expression (and expected value) are evaluated. - (let ((test-case-name (test-runner-test-name runner)) + (let ((test-case-name (current-test-full-name runner)) (start-time (current-time time-monotonic))) (hash-set! test-cases-start-time test-case-name start-time))) =20 @@ -142,7 +148,7 @@ do the final reporting." (let* ((results (test-result-alist runner)) (result? (cut assq <> results)) (result (cut assq-ref results <>)) - (test-case-name (test-runner-test-name runner)) + (test-case-name (current-test-full-name runner)) (start (hash-ref test-cases-start-time test-case-name)) (end (current-time time-monotonic)) (time-elapsed (time-difference end start)) @@ -156,7 +162,7 @@ do the final reporting." (and show-duration? time-elapsed-seconds))) =20 (unless (and errors-only? (not (test-failed? runner))) - (format #t "test-name: ~A~%" (result 'test-name)) + (format #t "test-name: ~A~%" test-case-name) (format #t "location: ~A~%" (string-append (result 'source-file) ":" (number->string (result 'source-line)))) @@ -174,7 +180,7 @@ do the final reporting." =20 (format trs-port ":test-result: ~A ~A [~,3fs]~%" (result->string (test-result-kind runner)) - (test-runner-test-name runner) time-elapsed-seconds))) + test-case-name time-elapsed-seconds))) =20 (define (finalize runner) "Procedure to call after all tests finish to do the final reporting." @@ -221,12 +227,12 @@ do the final reporting." (define (test-match-name* regexp) "Return a test specifier that matches a test name against REGEXP." (lambda (runner) - (string-match regexp (test-runner-test-name runner)))) + (string-match regexp (current-test-full-name runner)))) =20 (define (test-match-name*/negated regexp) "Return a negated test specifier version of test-match-name*." - (lambda (runner) - (not (string-match regexp (test-runner-test-name runner))))) + (compose not + (test-match-name* regexp))) =20 ;;; XXX: test-match-all is a syntax, which isn't convenient to use with a = list ;;; of test specifiers computed at run time. Copy this SRFI 64 internal --=20 2.45.2 --jhXV0P+yS+fKLGww-- --D7V7UDJ/WFLnWPqb Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCgAdFiEEt4NJs4wUfTYpiGikL7/ufbZ/wakFAmaVlxoACgkQL7/ufbZ/ wan6dRAAhLhHKzlOxDVQn5DG3JVQcp/co0uLsOQoB/KNrIxL/wqeDwJJRLKKCl0V D6IMpJLFq7arst9zGLyPUkDbJXnHP4WqdLm+VHwSbJVe9JPm96hJkiTyiN/cYUp7 yThZlGHWe9LCXnmSNrbu2F4KxD9kraWGUh5K4lX1JN0m7NGQCYSjdGlXa6yQsAEr vuQSCJj1Cqqe9MtiTp1hQLuGQvhJghZ/Infy3t0OddZz6K3OId1Wffm+Qfa5Tr7/ IC1wN/bsX7HgFR3+eS6B461tHXC+rgn1CEnxeCWDvqRbH6sRVsdm134OnH7J8BK4 xhpfYy/bmtDXlM67aBUUeMii5vaTOKN4HrhoXdcagAolP7mDuiL7R3yTX4znNyzJ 1mB85ULlLuLWV4/Fgrs6FmnpJaedj4gigTZJiTpBAYwozGuNN8uOxHlfcO+ys9a8 /An7POSPjmfuVzKPfnkdvEkb/hrjS0S3W2/yxooubdjtzzqc6lsEAezDh1iEFw07 GziNpnmzxS7kFBfYekrrN26OnVc0oQdTFoCrWnwqgOdNMBJ7TK+hWzP88oU8ESOp JJgAno9vT4ITQP1fKgneoerPbjeDaUE2oJiJYycJUL9HkdNQDINtDELhdETiyIqW 6+brSGgWr85pMei60PHauOTyTbf3II9XfRL1WgutBVjDX/Y1xoI= =VcZs -----END PGP SIGNATURE----- --D7V7UDJ/WFLnWPqb--