From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id D955Lp7nGGD3LAAA0tVLHw (envelope-from ) for ; Tue, 02 Feb 2021 05:48:14 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id oFYHKp7nGGDePAAAB5/wlQ (envelope-from ) for ; Tue, 02 Feb 2021 05:48:14 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 36A7E940483 for ; Tue, 2 Feb 2021 05:48:14 +0000 (UTC) Received: from localhost ([::1]:49772 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l6oXt-0005OM-0S for larch@yhetil.org; Tue, 02 Feb 2021 00:48:13 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:60190) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l6oXh-0005OD-UT for guix-patches@gnu.org; Tue, 02 Feb 2021 00:48:01 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:50562) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1l6oXh-00009j-NG for guix-patches@gnu.org; Tue, 02 Feb 2021 00:48:01 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1l6oXh-00012k-JK for guix-patches@gnu.org; Tue, 02 Feb 2021 00:48:01 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#45948] [PATCH 0/5] Improvements to the Automake SRFI 64 test driver. Resent-From: Maxim Cournoyer Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 02 Feb 2021 05:48:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45948 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Received: via spool by 45948-done@debbugs.gnu.org id=D45948.16122448423960 (code D ref 45948); Tue, 02 Feb 2021 05:48:01 +0000 Received: (at 45948-done) by debbugs.gnu.org; 2 Feb 2021 05:47:22 +0000 Received: from localhost ([127.0.0.1]:33875 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l6oX3-00011n-VA for submit@debbugs.gnu.org; Tue, 02 Feb 2021 00:47:22 -0500 Received: from mail-qk1-f176.google.com ([209.85.222.176]:43856) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l6oX0-00011a-VV for 45948-done@debbugs.gnu.org; Tue, 02 Feb 2021 00:47:21 -0500 Received: by mail-qk1-f176.google.com with SMTP id a12so18784266qkh.10 for <45948-done@debbugs.gnu.org>; Mon, 01 Feb 2021 21:47:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version:content-transfer-encoding; bh=7KtunbuFuJ2Pl5Wpy9MUW5C/wLK/+j6BuKaLxGUHvDg=; b=RhM4BXL3joFN4R0PyN23e4AQMAsAci39W4k/meOXrS2+rwjK8LIbQ5GEdEVlnu1T6m ba+L/NlNFDIjLGFh2nyhDSBVJODd0F5adKGw7JSOxCIfoEjApFmkfyOTdszThXhnp0s+ i3P61/r8HKX39W1EwgXyqo9WT0xq5lsUtdGkxnQjHuWs5G50YG3lWl+Wgjj1wBV07BQF fEgokneZjiS91we2qYUrJ+DHWCq2C2aRwCo43UcpfH6ZqlP3I/+AqjS1tkuR3sc06DqY OT7GXlYCMDPQlQjpFmbYsmfEspJPz6l16gMrQqrrhi8mhAy9y0zl33PBjP5uYyvTC1Lr BiiA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version:content-transfer-encoding; bh=7KtunbuFuJ2Pl5Wpy9MUW5C/wLK/+j6BuKaLxGUHvDg=; b=BGfiEcliY2nPEMVpkJ9pFBNtZjez0Yn+KglPNtG5DasaZYV+KPCdxUNHfdGnaO60yN mOGQgwOAYKVVVD6TFyaMwhysX2jNjCmbZ9M2e61s4GMfdA+4qEs452j7BPuz8qnRvKUS OB5TVp5Iwn0r2xAX3NPS8Yl20wzdh6JrZdYJ+e8GtUmn3sADb3/s9CorCZzU/1krFo84 MsQLbm9zn2PZ0VkSrjuf1dkCZu0DlRkxPHDUs0bPyJ1dX+t6UAf4c7jGpW1ngmLwnnQX Qmh3wgR7iMgIPyp0R5xcw5xlhIRa7W+Dxm75/wQp2n3Acq0bAYkeVAwMIGXfn4dcWLZD 4lPw== X-Gm-Message-State: AOAM532fO4/FKvZo1XikodR8A4b7LbIsPEDzmGXwMH1JmqpYIifiLzfJ VcRaLSpxv6P/1VijmV/AU/AzATsVz8Fj1Q== X-Google-Smtp-Source: ABdhPJx8PYsJ1fpZbhkpUHiWoNB/GiSN/K9lQYJX54bPFOjxR1h+9QOaEQY4eubdqVjDVOhcs+8qog== X-Received: by 2002:a37:6609:: with SMTP id a9mr19340012qkc.123.1612244833067; Mon, 01 Feb 2021 21:47:13 -0800 (PST) Received: from hurd (dsl-205-236-230-168.b2b2c.ca. [205.236.230.168]) by smtp.gmail.com with ESMTPSA id u133sm11381138qka.116.2021.02.01.21.47.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Feb 2021 21:47:12 -0800 (PST) From: Maxim Cournoyer References: <20210118062501.27022-1-maxim.cournoyer@gmail.com> <20210118062501.27022-5-maxim.cournoyer@gmail.com> <87eei2p0qu.fsf_-_@gnu.org> <87lfc8sb8w.fsf@gmail.com> <87czxjh1os.fsf@inria.fr> Date: Tue, 02 Feb 2021 00:47:11 -0500 In-Reply-To: <87czxjh1os.fsf@inria.fr> ("Ludovic =?UTF-8?Q?Court=C3=A8s?="'s message of "Mon, 01 Feb 2021 23:18:27 +0100") Message-ID: <87h7mvqaw0.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: 45948-done@debbugs.gnu.org Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN X-Migadu-Spam-Score: -1.26 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20161025 header.b=RhM4BXL3; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: 36A7E940483 X-Spam-Score: -1.26 X-Migadu-Scanner: scn1.migadu.com X-TUID: g/IXxfoWhADn Hi Ludo, Ludovic Court=C3=A8s writes: > Hello! > > Maxim Cournoyer skribis: > >> Ludovic Court=C3=A8s writes: >> >> [...] >> >>> I never felt the need for this since most individual files run quickly >>> enough (and those that don=E2=80=99t should be optimized=E2=80=A6), but= it can be >>> useful. >> >> What triggered it for me was trying to iterate using tests added to the >> tests/packages.scm test module: > > [...] > >> 1.6 s; better than 46 s! > > Quite a difference, indeed! > > Some tests, like =E2=80=98tests/store.scm=E2=80=99, invoke the GC (via = =E2=80=98delete-paths=E2=80=99 > calls in particular), and this is a bad idea because it takes ages. > What I meant by =E2=80=9Cshould be optimized=E2=80=9D is that these tests= should be > tweaked to avoid invoking the GC as much as possible. > > That=E2=80=99s not the case of =E2=80=98tests/packages.scm=E2=80=99 thoug= h. Which gives me an > idea: what would it take to modify the test driver so it can print the > time spent in each test? :-) > > That would probably prove helpful to optimize core Guix. Good idea! I've pushed 5e652e94a9 that does this! Here's a sample, showing the three slowest test cases of the tests/packages.scm test: $ export SCM_LOG_DRIVER_FLAGS=3D"--brief=3Dno --show-duration=3Dyes" $ make check TESTS=3Dtests/packages.scm --8<---------------cut here---------------start------------->8--- [...] PASS: tests/packages.scm - fold-available-packages with/without cache [17.1= 21s] PASS: tests/packages.scm - find-packages-by-name [0.000s] PASS: tests/packages.scm - find-packages-by-name with version [0.000s] PASS: tests/packages.scm - find-packages-by-name with cache [9.492s] PASS: tests/packages.scm - find-packages-by-name + version, with cache [7.4= 12s] [...] PASS: tests/packages.scm - find-package-locations with cache [7.484s] PASS: tests/packages.scm - specification->location [0.000s] --8<---------------cut here---------------end--------------->8--- Automake could be extended to support this extra data; it would be interesting to be able to set the number of slowest tests to display, such as DURATIONS=3D5, to show only the 5 slowest tests, for example. > In many cases, I also run the one test I=E2=80=99m interested in through = Geiser, > which gives an optimally fast feedback loop. I've done that too, although it's somewhat clunky to use: the test modules name not being mapped to an actual filename hierarchy trips the module loading mechanism of Guile/Geiser. I usually fix that such that (test-packages) -> (tests packages), then I can load the module normally. Except that's not what I want to do as it runs all the tests. So I must copy paste the module declaration to the REPL to get the symbols imported, then the other required definitions and the (test-begin... due to SRFI 64 stateful design. Not my idea of a sleek test feedback loop, coming from pytest and other modern test frameworks; still very powerful to be able to do everything at the REPL though! >> We can also check the time the suspected slow test took: >> >> $ time make check TESTS=3Dtests/packages.scm >> SCM_LOG_DRIVER_FLAGS=3D"--select=3D'fold-available-packages with/without >> cache'" >> [...] >> PASS: tests/packages.scm - fold-available-packages with/without cache > > Ah ha! Turns out a large part of the time was due to the O(n=C2=B2) beha= vior > of the various list operations (the list of packages is big enough!). > Fixed in 73744725dd0a65cddaa9251f104f17ca27756479. Woohoo! That looks clever! Too clever for me to understand in the 2 minutes I starred at it ;-). That test now runs almost 6x faster (~8 s on a fast machine). Well done! >>>> +The underlying SRFI 64 custom Automake test driver used for the 'chec= k' >>>> +test suite (located at @file{build-aux/test-driver.scm}) also allows >>> >>> Maybe shorten to =E2=80=9CThe underlying test driver (located at >>> @file{build-aux/test-driver.scm}) also allows=E2=80=9D. >> >> I see value in explicitly stating what it is, as it took me some effort >> to be able to answer that question when I started looking at it (the >> test driver). > > Agreed. It just seemed to me that we were mentioning three new > concepts/tools in passing: SRFI-64, Automake, and test drivers. Yeah, it's a mouthful, for better or worse. Thank you, Maxim