From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp12.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id CNlLD+VNbGIbFQAAbAwnHQ (envelope-from ) for ; Fri, 29 Apr 2022 22:43:17 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp12.migadu.com with LMTPS id YPaeD+VNbGISwQAAauVa8A (envelope-from ) for ; Fri, 29 Apr 2022 22:43:17 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 3EEC32E14B for ; Fri, 29 Apr 2022 22:43:16 +0200 (CEST) Received: from localhost ([::1]:39036 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nkXSN-0001JP-CG for larch@yhetil.org; Fri, 29 Apr 2022 16:43:15 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49200) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nkXSI-0001In-Pk for gwl-devel@gnu.org; Fri, 29 Apr 2022 16:43:10 -0400 Received: from sender4-of-o50.zoho.com ([136.143.188.50]:21043) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nkXSG-0002wt-RR for gwl-devel@gnu.org; Fri, 29 Apr 2022 16:43:10 -0400 ARC-Seal: i=1; a=rsa-sha256; t=1651264985; cv=none; d=zohomail.com; s=zohoarc; b=OTHffuZAh+cxI3cyMkEO08FjSGAydPaSeBfo/MwzanpaBID3PjVfrikAqhBvN03pV53x7ZllCHL5RK32+zhR5szdrRVVndvDS5eC/soV2fYyABvxkgUldrZbCxPL7TN48exsUZnf/XGMwC0oMKx2VR6/6CmfknK/81vMjjUmPGw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1651264985; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:To; bh=/PazKSdk43ZmRQF1HL5FRA5D8aDcJ3QXwBSijng7DaU=; b=IYIt3Wn41Gc4h/goAMtX9j/KN4VSiC9+n/Xoy38Oi8W+qDDjP+84r2MCMzFKmSOii3pe8stcF1txj0K1nTnkGNXCY/y0TiIk9lLmszhApfVT+Vv2Efb+QNF/hoYYmS0qmXosRef/0X1wHgTCBGwWlqb9nnZ7bT51xliVrMiuwbk= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=elephly.net; spf=pass smtp.mailfrom=rekado@elephly.net; dmarc=pass header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1651264985; s=zoho; d=elephly.net; i=rekado@elephly.net; h=References:From:From:To:To:Cc:Cc:Subject:Subject:Date:Date:In-reply-to:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-Id:Reply-To; bh=/PazKSdk43ZmRQF1HL5FRA5D8aDcJ3QXwBSijng7DaU=; b=Ks5uzbSUNblCh/nY85PdIUci+F+326jySdX2861psRe7QwwjAZ9YZwC49/WpWKpo tEf0tTMKNK5GWjjtlmxoT3k9LGTQZYk87DVk46P/ASuIONu90J3JVyAQfEgfLE+XD70 1Scq/g0dnB19MTK3wpJsL21G7paDNTpRMNkc9rvk= Received: from localhost (143-111-142-46.pool.kielnet.net [46.142.111.143]) by mx.zohomail.com with SMTPS id 165126498353498.16685540812955; Fri, 29 Apr 2022 13:43:03 -0700 (PDT) References: <20220429175500.17176-1-olivier.dion@polymtl.ca> User-agent: mu4e 1.6.10; emacs 28.0.50 From: Ricardo Wurmus To: Olivier Dion Subject: Re: [PATCH] tests/examples: Add running of workflow examples Date: Fri, 29 Apr 2022 22:32:08 +0200 In-reply-to: <20220429175500.17176-1-olivier.dion@polymtl.ca> X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Message-ID: <87sfpvzkmj.fsf@elephly.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-ZohoMailClient: External Received-SPF: pass client-ip=136.143.188.50; envelope-from=rekado@elephly.net; helo=sender4-of-o50.zoho.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: gwl-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: gwl-devel@gnu.org Errors-To: gwl-devel-bounces+larch=yhetil.org@gnu.org Sender: "gwl-devel" X-Migadu-Flow: FLOW_IN X-Migadu-To: larch@yhetil.org X-Migadu-Country: US ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1651264997; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=/PazKSdk43ZmRQF1HL5FRA5D8aDcJ3QXwBSijng7DaU=; b=glNqfIiZJF8odFejrlPdPXLxYNhRth3igdXIy/UYHtyCxUDxq0FmWUsFmHWizxvW+bm6QO hxFhKTu+CbqrEz3go/qWyLFVUEOBjYwsxZR8pGJCXNtB5vZ/NPoSKN/VCLCMWDwnZfchmP MT3I66Eq0Jb98a6LY83SeOAZnFzg4A3ZKmQ/zkpc3f/C5nzWuP+PIn9y4+YXasbL/P+OJO wEld2DJOjB0gAiuiw9M6y+S/+3t82pry1jGYOahXgRmcboFdUQVJefdPDUhfnjZ+J8by3F benhXu2J/p4B8FMM7X8ZaXvi9B268lGFxx3cEUpPS1lRNUXJ+Aa4T+7d1gn6lQ== ARC-Seal: i=2; s=key1; d=yhetil.org; t=1651264997; a=rsa-sha256; cv=pass; b=BUQNdcNiKYExdm+EXrL7eE5l0h2mEQxgNBhI98pMI+d2HMFpmcihGcClaq8lBr3S8t7lMa wuiZ73xk9qUxOFyVJQapGnd3Obico54SoflkS6GydVb6Qo8b7xnPPbm86ojfnPLP4But8U FHRJI06XOZqkltDbn/nEnhF3aoAvtaJgsbUU4fMqeD8VSz8/HOfm/SJWt1zrzeRBMVgEwP RoiC/3tzaZy3w7fLlGtP2uTNZrl8KH/OpKoeU8LSAl4/FQJt7SffhEOEIiXxWr7A9KtqOM PFdSWviRXpmVb9O8YO6ZAwa5lcydYRWnyZ1M5S2nLftNKOHcGZ3mqOPCvxFhZQ== ARC-Authentication-Results: i=2; aspmx1.migadu.com; dkim=pass header.d=elephly.net header.s=zoho header.b=Ks5uzbSU; arc=pass ("zohomail.com:s=zohoarc:i=1"); dmarc=none; spf=pass (aspmx1.migadu.com: domain of "gwl-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="gwl-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -6.50 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=elephly.net header.s=zoho header.b=Ks5uzbSU; arc=pass ("zohomail.com:s=zohoarc:i=1"); dmarc=none; spf=pass (aspmx1.migadu.com: domain of "gwl-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="gwl-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 3EEC32E14B X-Spam-Score: -6.50 X-Migadu-Scanner: scn0.migadu.com X-TUID: XbIquLbhgIvY Hi Olivier, thanks for the patch! > End to end testing of pre-defined scenarios are a good way to check for > regression. > > Here we introduce testing of some examples available in the documentation= . That > way, we're sure that new users should be able to run them without problem= s. > > Each scenario is a different test and is run in a different temporary > directory which get destroyed if the scenario succeeded. It=E2=80=99s a good idea to run the examples. However, this requires a ful= ly functional Guix installation (which we don=E2=80=99t have when building with Guix, for example), so we should not include them in =E2=80=9Cmake check=E2= =80=9D but add a separate target for these tests. An alternative may be to do what Guix does for system tests, but I=E2=80=99d be okay with just having a sepa= rate target that is run manually before releases or in CI. > +(define-syntax-rule (with-directory-excursion dir body body* ...) > + (let ((old-dir (getcwd))) > + (dynamic-wind > + (lambda () (chdir dir)) > + (lambda () body body* ...) > + (lambda () (chdir old-dir))))) > + > +(with-directory-excursion > + (string-append top-srcdir "/doc/examples") > + > + (define (process-success? status) > + (=3D 0 (or (status:exit-val status) > + (status:term-sig status)))) Use ZERO? here. > + (define scenarios > + (list "extended-example-workflow.w")) Should these better be discovered automatically via SCANDIR? > + (for-each (lambda (example) > + (test-assert example > + (let* ((tmp-dir (mkdtemp > + (format #f "gwl-example-~a.XXXXXX" exam= ple))) > + (abs-example (canonicalize-path example)) > + (success? > + (with-directory-excursion tmp-dir > + (process-success? > + (system > + (format #f "guix workflow run -fc ~a -l all" > + abs-example)))))) Please don=E2=80=99t use SYSTEM. How about (system* "guix" "workflow" "run" "--force" "--container" "--log-events=3Dall" (canonicalize-path example)) ? > + (if success? > + (system* "rm" "-rf" tmp-dir) Why shell out to RM when we have DELETE-FILE and its recursive friend in Guix? I=E2=80=99d also rather move clean-up work to a DYNAMIC-UNWIND handl= er. > + (format (error-output-port) > + "Example directory: ~a\n" tmp-dir)) Nitpick: ~% instead of \n. > + success?))) > + scenarios)) This FOR-EACH loop combines test definition with test running, which seems wrong to me. Maybe SRFI-64 is not the best fit for tests that only care about whether a shell command was run successfully. Perhaps we should do as Guix does and just have a shell script to run these tests. --=20 Ricardo