From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2.migadu.com ([2001:41d0:403:58f0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms13.migadu.com with LMTPS id 0GBAIFjGYGdPggAAe85BDQ:P1 (envelope-from ) for ; Tue, 17 Dec 2024 00:31:20 +0000 Received: from aspmx1.migadu.com ([2001:41d0:403:58f0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2.migadu.com with LMTPS id 0GBAIFjGYGdPggAAe85BDQ (envelope-from ) for ; Tue, 17 Dec 2024 01:31:20 +0100 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=debbugs.gnu.org header.s=debbugs-gnu-org header.b=h8fOnqrg; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20230601 header.b=KCYNcc6a; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org"; dmarc=fail reason="SPF not aligned (relaxed), DKIM not aligned (relaxed)" header.from=gmail.com (policy=none) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1734395479; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=407peAdfE0XPtE+DMmVaQOsUNP/m/C+2xsk5zauvlaU=; b=uk/jFtNqh1LzP9Pb7lwAk2hmXuhTLyH02N2gOzuWSB+TgwptXyFd3UZQEYt92R+zbG2LCX oWg9nIj9S3se9ATL0udHcpZNRDz5yK+nNMlCCc1RI71vmTVTeWaLiaVc1yTG3MN+eZc/WA lL7YxrY0C3bJ4JmVmMh0UWuMbSMEh7/AhS02GQ/gfh1HSoNeyXgNOqKv7l+i+ZCd1L/l6N rKAW3jim+6lU4ZPBooFBBSvACW4C4YIxybklMVEPcEC24KzexraSft2ZbylDp46Pnys8mC B4C0iNxcIpIfU3VYy+gEgfKkI/DdK+/FhnexzpkJm/s/MuKfnPb0QPNaXpytIw== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1734395479; a=rsa-sha256; cv=none; b=esb/N9v9R9NvNKZLBUMlVPil6pOvMW4sE7gii47QdyqQifv+eHD44plDrBEEC7j1yQAHDR QBkHzHNMNKF+B53/gd71NZgfx5JJTy474ZcSed4m+phtto0nGwoz/gvUoJtJ0yQ0GNwRX6 zrWzMktt2AF6SC8bsPnmgU0qBAkhD40M5Hm6dkH1JWWUVLy/i9+zyF+wTf6ytgXOD0dYEU FFpqSnVguu6UnQpzao67yH+puVehnSK1koWmiuVL4Bi8Kefj4mx74rPXghJ4UXeImlAkwH 4tWo5HllY2d389iV/SA/nLDuHAu1blDU9qckfQD0koshlt2HjJygdN0AQ07JaA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=debbugs.gnu.org header.s=debbugs-gnu-org header.b=h8fOnqrg; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20230601 header.b=KCYNcc6a; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org"; dmarc=fail reason="SPF not aligned (relaxed), DKIM not aligned (relaxed)" header.from=gmail.com (policy=none) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 3BA27814E7 for ; Tue, 17 Dec 2024 01:31:19 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tNLUP-0005w2-76; Mon, 16 Dec 2024 19:31:05 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tNLUN-0005vp-5t for guix-patches@gnu.org; Mon, 16 Dec 2024 19:31:03 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1tNLUM-0008Gy-TP for guix-patches@gnu.org; Mon, 16 Dec 2024 19:31:02 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debbugs.gnu.org; s=debbugs-gnu-org; h=MIME-Version:Date:References:In-Reply-To:From:To:Subject; bh=407peAdfE0XPtE+DMmVaQOsUNP/m/C+2xsk5zauvlaU=; b=h8fOnqrgT9P7GUJ56aLxZeiJOHqM/6PGuC66GLKx5iVIOMOtYyF8AL7eqCIJCx/x0Y0ASyzje8KxxVPKd5hFjM3EfzVa1enDTj0I0ZaY/yJWy24I++WdPC/0WaFbPNFfDgZI5Pb7JK+lReLTaHY9uBq0X+U504s8LC6GcchKD57uU8oj6cO1Wit6Zwp0+w/Ce72gRRmFYsj9oyZ8DL00o5mNgahr0HfMbnignyPlYU+ciYlrgEugxFz5UQkTMpZHeibsH4y/Bdtk8YqxhXgn3e/ofP0MtbpOyT6keBOZoGJPDxDImhhCZwOXjiYGEIrEUL4n1vnAA6EKUN0YVsYoUw==; Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1tNLUL-0003fV-NL for guix-patches@gnu.org; Mon, 16 Dec 2024 19:31:01 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#74394] [PATCH 0/2] Skip slow tests by default and run 'check' in Git pre-push hook. Resent-From: Maxim Cournoyer Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 17 Dec 2024 00:31:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 74394 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: 74394@debbugs.gnu.org Received: via spool by 74394-submit@debbugs.gnu.org id=B74394.173439541514008 (code B ref 74394); Tue, 17 Dec 2024 00:31:01 +0000 Received: (at 74394) by debbugs.gnu.org; 17 Dec 2024 00:30:15 +0000 Received: from localhost ([127.0.0.1]:56779 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1tNLTa-0003ds-Ke for submit@debbugs.gnu.org; Mon, 16 Dec 2024 19:30:15 -0500 Received: from mail-pj1-f43.google.com ([209.85.216.43]:52408) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1tNLTX-0003ck-Vi for 74394@debbugs.gnu.org; Mon, 16 Dec 2024 19:30:13 -0500 Received: by mail-pj1-f43.google.com with SMTP id 98e67ed59e1d1-2efded08c79so3237058a91.0 for <74394@debbugs.gnu.org>; Mon, 16 Dec 2024 16:30:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1734395350; x=1735000150; darn=debbugs.gnu.org; h=content-transfer-encoding:mime-version:user-agent:message-id:date :references:in-reply-to:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=407peAdfE0XPtE+DMmVaQOsUNP/m/C+2xsk5zauvlaU=; b=KCYNcc6aFOvWh1Iqg1UZdxHJpew6XIl1BXYsDvIm4jgqFcAPNFgld5G7/5YhlVeVv8 ur4VPb2DqG6ECeerrWfywjhtViq34HYPh4cbLqeEPTzujkyTq49EQGVVUhuF6yQUxpWB 3m6fWasH9nlA/35Z1nt4roZGv1crj6Q6/GF6cU++HZ0NTEHdgMS2R2rfhXmQJ+EY0Jwz 6WBqaCy2z2YZNa/RQyDVBYPWmt8S3AQDIEY470Umdpj+ds07oD85HEvboItZCdSDHfsa qSl6sYczcZ5D2zOd/kHRcef5rEpDAAUe8WPmRZoQNxTY5kuap3h+Cgz0iC3rXVBcaVhF ClKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734395350; x=1735000150; h=content-transfer-encoding:mime-version:user-agent:message-id:date :references:in-reply-to:subject:cc:to:from:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=407peAdfE0XPtE+DMmVaQOsUNP/m/C+2xsk5zauvlaU=; b=Zn72KeA+bz7I7wfml49fkdc6G54CqiNT/RG3g4CAMUm1VzSzBxtv54YDrzobyuT9Iu k2ciDUgJ5iMVw4rqltFdqWW8FeT4N4ESk5WpH4wZFW8jSbtfko2JTtYVADFe7E+cae4/ +1KqvAQGiKIJNRhQrbQG76PO2mENayRs38ssNGUS+KuW9Fi5uy5GJ6cW6EiAYJsD8ghi dQN4iOwRk3tbn6KUe2+vRQWgeQF4URTb0pW77r73AgWfOarNdMmN44ufKKiv19giOob/ bDlezDlVUIbCKbr2OBY8/k7Eo1k+A8TyGcj6qpnVZwaDTGmcCtC6j7MFnjDISLX/hqI1 IzCw== X-Gm-Message-State: AOJu0YxTraooxJQX0636rX/RWtFHTSVY7eM/T6TXh+TG0akUqhIpuzgt UbLvo/oOBNDTetgBAOdengghPY//yhUEwXaUs1dtC/zKg7CzE2Em92GeJhrP X-Gm-Gg: ASbGncu9knMVzeeX++SdozKBfm2s3MoCnw9Rm+SYmUbGc2KFaamB5aITOs0DuWtItiy PcpiCTWTD7sarJz38dwQgEyZIsf7WjXqQK/0JArxBdw1hNQXR81b0+G0I6kJ7Eu/9l/zwpWNlvH /a4sxYmGzUY4/Zc3z0tcfhhh+XalA+8+Ug73k4RvK41hVJFj1kTFkSBH5xGqFX7fPshzVVS6VKC mLlKdOTdJDrmKW5WQT6FSifbVhQeKCXuuLd91FGjpyaa036Dy8MWg== X-Google-Smtp-Source: AGHT+IH7pYWJ5W7wg7aNFShbQfjKs0b+OwoBPsXv0OKuSNB+n4dIs//38kwlIraxlOiKNHQ25sTXpw== X-Received: by 2002:a17:90a:e70d:b0:2ee:df57:b194 with SMTP id 98e67ed59e1d1-2f28fd6b922mr18604676a91.21.1734395350052; Mon, 16 Dec 2024 16:29:10 -0800 (PST) Received: from terra ([2405:6586:be0:0:c8ff:1707:9b9:af89]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2f142de475bsm8723189a91.31.2024.12.16.16.29.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Dec 2024 16:29:09 -0800 (PST) From: Maxim Cournoyer In-Reply-To: <87y1122w3k.fsf@gnu.org> ("Ludovic =?UTF-8?Q?Court=C3=A8s?="'s message of "Fri, 29 Nov 2024 11:05:51 +0100") References: <87y1122w3k.fsf@gnu.org> Date: Tue, 17 Dec 2024 09:28:59 +0900 Message-ID: <87zfkvrw38.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: guix-patches-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Spam-Score: -2.34 X-Spam-Score: -2.34 X-Migadu-Queue-Id: 3BA27814E7 X-Migadu-Scanner: mx11.migadu.com X-TUID: nbzPIf4RYh84 Hi Ludovic, Ludovic Court=C3=A8s writes: > Hi Maxim, > > Maxim Cournoyer skribis: > >> This is a simple change that should ensure test suite breakages are dete= cted >> as early as possible and avoid tests breaking changes to be pushed. Thi= s is >> made possible by skipping a few expensive tests suite, bringing down the= total >> test time to about 1 minute on a fast machine. >> >> We could call it a "distributed CI" approach ;-). > > I agree with the goal, of course, but not with the method: even without > expensive tests, =E2=80=9Cmake check=E2=80=9D is going to take maybe 5=E2= =80=9310 minutes, and > having that happen when you run =E2=80=9Cgit push=E2=80=9D can be a terri= ble development > experience (especially since the developer most likely either already > ran the test suite or part of it right before, or pushes package changes > that have infinitely small probability of breaking =E2=80=9Cmake check=E2= =80=9D). As I wrote, 'make check' with this change takes about 1 minute on my machine; I'd be curious to know how long it takes on other people machines; I suspect a bit more; if it's too slow, we can skip more, or find out ways to make the tests run faster. > Back to CI and not breaking things: I think that we should have a > workflow where the forge triggers those checks and puts a green light if > it passes, red light otherwise. (Basically what everybody else is > doing. :-)) > > To me this should be one of the goals for the project in 2025. That would be great. I wonder how far QA is from making this achievable. >> To run the complete test suite including the slow tests (as is the case = prior >> this change): >> >> make check WITH_SLOW_TESTS=3D1 > > This variable itself may still be useful though (I=E2=80=99d call it > =E2=80=98RUN_EXPENSIVE_TESTS=E2=80=99 or something like that=E2=80=94that= =E2=80=99s the name used in > Coreutils=E2=80=94, =E2=80=9Cexpensive=E2=80=9D being the key word). I w= ould also keep it on by > default. One of the tests that was unbearably long when I measured was the time-machine test. It took about 20 minutes to fetch the git repository with guile-git and run the test (which does extra work compared to the CLI like validating each object). I don't think we want this kind of experience by default (because that'd probably convince people that running the test suite often is not a reasonable thing to do). The other tests were more reasonable, with the longer ones in the 2-3 minutes range on my machine, IIRC. Perhaps we could have this 20 minute outlier skipped by default, maybe with a RUN_PROHIBITIVE_TESTS flag that would default to 0 (false). A long time ago I had read a blog post that argued that unit tests should be small and fast [0], and there's a lot of good about that. Fast tests usually translates in running the test suite more often and catching breakage earlier. As the author states, it also makes it possible to determine whether a bug lies in the core logic or in the integration of the many parts (unit tests vs integration tests), when unit tests are decoupled from the whole system (typically by mocking all external interfaces). [0] https://www.artima.com/weblogs/viewpost.jsp?thread=3D126923 --=20 Thanks, Maxim