From mboxrd@z Thu Jan 1 00:00:00 1970 From: Mathieu Lirzin Subject: Re: [PATCH] build: Add a scheme custom test driver using SRFI-64. Date: Sun, 01 Nov 2015 20:48:34 +0100 Message-ID: <87lhahcxgt.fsf@gnu.org> References: <87lhaiv30f.fsf@gnu.org> <87h9l54p8p.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:59183) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ZsycR-0007th-92 for guix-devel@gnu.org; Sun, 01 Nov 2015 14:48:48 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ZsycP-0005W9-Sf for guix-devel@gnu.org; Sun, 01 Nov 2015 14:48:47 -0500 In-Reply-To: <87h9l54p8p.fsf@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\= \=\?utf-8\?Q\?s\?\= message of "Sun, 01 Nov 2015 18:13:26 +0100") List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org To: Ludovic =?utf-8?Q?Court=C3=A8s?= Cc: guix-devel@gnu.org Hi, ludo@gnu.org (Ludovic Court=C3=A8s) writes: > Mathieu Lirzin skribis: > >> While reading the Automake manual I have discovered that it provides an >> API for custom test drivers, so I've tried to implement something for >> the scheme API for test suites (SRFI-64) which Guix is using. The >> result provides support for multiple test cases in a single file, and >> allow =E2=80=98make check=E2=80=99 display the results individually: >> >> PASS: tests/base32.scm - bytevector->base32-string >> PASS: tests/base32.scm - base32-string->bytevector >> PASS: tests/base32.scm - nix-base32-string->bytevector >> PASS: tests/base32.scm - &invalid-base32-character >> SKIP: tests/base32.scm - sha256 & bytevector->nix-base32-string >> PASS: tests/base64.scm - empty string > > Awesome! Are the =E2=80=9Cinner=E2=80=9D tests displayed by default? Or= is there some > environment variable to control that? Yes, each test case is displayed by default. > I=E2=80=99d prefer the default to display only file names, as is currentl= y the > case (it=E2=80=99s more concise.) IMO This would be reasonable to have an option for making the output more concise since there are 563 test cases in Guix. this is not part of the test driver API specification, but it does not seem too hard to add an additional option to the test driver script like what the TAP driver is doing: https://www.gnu.org/software/automake/manual/automake.html#Use-TAP-with-t= he-Automake-test-harness I think =E2=80=98--brief=E2=80=99 could be a good name for this option. De= fault verbosity would specified by adding =E2=80=9CAM_SCM_LOG_DRIVER=3D --brief= =3Dyes=E2=80=9D in Makefile.am, then the user can overide that with: make check -j4 TESTS=3Dtests/rocket-science.scm SCM_LOG_DRIVER=3D"--brief= =3Dno" >> Finally the =E2=80=98.log=E2=80=99 files are no longer fragmented between >> =E2=80=9C${top_srcdir}=E2=80=9D and =E2=80=9Ctests/=E2=80=9D. If someth= ing went wrong (FAIL, SKIP, >> XPASS) then the entire log file is copied in =E2=80=98test-suite.log=E2= =80=99 so no need >> to ask for an additional file when someone is reporting some =E2=80=98ma= ke >> check=E2=80=99 error. > > Nice. =E2=80=9CRunning the Test Suite=E2=80=9D needs to be updated accor= dingly. Thanks for reminding me this. ;) >> One caveat is that =E2=80=98tests/cpio.scm=E2=80=98 is now failing. > > Why is that? Does it relate to this change? I didn't try to debug the problem but it has appeared when I started messing with redirection of the output/error ports. Here is the failure output. --8<---------------cut here---------------start------------->8--- test-name: bit-identical to GNU cpio's output location: /home/mthl/src/gnu/guix/tests/cpio.scm:49 source: + (test-assert + "bit-identical to GNU cpio's output" + (call-with-temporary-output-file + (lambda (link _) + (delete-file link) + (symlink "chbouib" link) + (let ((files (cons* "/" + (canonicalize-path + (dirname (search-path %load-path "guix.scm"))) + link + (map (compose + canonicalize-path + (cut search-path %load-path <>)) + '("guix.scm" + "guix/build/syscalls.scm" + "guix/packages.scm"))))) + (call-with-temporary-output-file + (lambda (ref-file _) + (let ((pipe (open-pipe* + OPEN_WRITE + %cpio-program + "-o" + "-O" + ref-file + "-H" + "newc" + "--null"))) + (for-each + (lambda (file) (format pipe "~a\x00" file)) + files) + (and (zero? (close-pipe pipe)) + (call-with-temporary-output-file + (lambda (file port) + (write-cpio-archive files port) + (close-port port) + (or (file=3D? ref-file file) + (throw 'cpio-archives-differ + files + ref-file + file + (stat:size (stat ref-file)) + (stat:size (stat file)))))))))))))) value: #f result: FAIL --8<---------------cut here---------------end--------------->8--- >> Since this script is not intented to be exclusively used by Guix, I have >> used a generic copyright notice. I guess Guix is the best place to >> challenge and improve it, but IMO it will be better hosted elsewhere >> like in Gnulib. Opinions? > > I think we could start using it and testing it for a while in Guix, and > eventually submit it for inclusion in Gnulib once we are more confident. So you recommend to add =E2=80=9Dthis file is part of GNU Guix=E2=80=9D and= use =E2=80=9CGNU Guix=E2=80=9D instead of =E2=80=9Cthis program=E2=80=9D for now? >> From 977e0586a6689df204fd9b522f8508ec318180c0 Mon Sep 17 00:00:00 2001 >> From: Mathieu Lirzin >> Date: Mon, 26 Oct 2015 23:47:24 +0100 >> Subject: [PATCH] build: Add a scheme custom test driver using SRFI-64. >> >> This provides support for multiple scheme test cases in a unique file and >> fixes the fragmentation of '.log' files. >> >> * build-aux/test-driver.scm: New file. >> * Makefile.am (SCM_LOG_DRIVER): New variable. >> (SCM_LOG_COMPILER, AM_SCM_LOG_FLAGS): Delete variables. >> (AM_TESTS_ENVIRONMENT): Set GUILE_AUTO_COMPILE to 0. >> * tests/base32.scm, tests/build-utils.scm, tests/builders.scm, >> tests/challenge.scm, tests/containers.scm, tests/cpan.scm, >> tests/cpio.scm, tests/cran.scm, tests/derivations.scm, tests/elpa.scm, >> tests/file-systems.scm, tests/gem.scm, tests/gexp.scm, >> tests/graph.scm, tests/gremlin.scm, tests/hackage.scm, tests/hash.scm, >> tests/lint.scm, tests/monads.scm, tests/nar.scm, tests/packages.scm, >> tests/pk-crypto.scm, tests/pki.scm, tests/profiles.scm, >> tests/publish.scm, tests/pypi.scm, tests/records.scm, >> tests/scripts.scm, tests/services.scm, tests/sets.scm, tests/size.scm, >> tests/snix.scm, tests/store.scm, tests/substitute.scm, >> tests/syscalls.scm, tests/ui.scm, tests/union.scm, tests/utils.scm: >> Don't exit at the end of each test. > > AIUI each tests/foo.scm file is still run with its own Guile instance, > in ./test-env, right? (This is important. It also means tests can run > in parallel.) > > If this is OK, it LGTM. I think it is OK :) > However, I=E2=80=99m unsure if we should push it now, or after the releas= e. On > one hand, I=E2=80=99d rather avoid potentially disrupting changes like th= is > now. On the other hand, since it makes it easier (and different) to > report test failures, it=E2=80=99s nice. > > How confident are you? :-) IMHO we should wait after the next release in order to make this test driver more bullet proof. Thanks for your comments! -- Mathieu Lirzin