From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Eli Zaretskii Newsgroups: gmane.emacs.devel Subject: Re: master 262d0c6: Mark some tests as expensive Date: Sat, 12 Sep 2020 13:38:01 +0300 Message-ID: <83a6xve0vq.fsf@gnu.org> References: <20200910182904.20559.25935@vcs0.savannah.gnu.org> <20200910182905.F0E4520A2E@vcs0.savannah.gnu.org> <877dt0is0p.fsf@gmx.de> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="6310"; mail-complaints-to="usenet@ciao.gmane.io" Cc: michael.albinus@gmx.de, stefan@marxist.se, emacs-devel@gnu.org To: Daniel =?iso-8859-1?Q?Mart=EDn?= Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Sat Sep 12 12:38:34 2020 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1kH2vS-0001YS-Ng for ged-emacs-devel@m.gmane-mx.org; Sat, 12 Sep 2020 12:38:34 +0200 Original-Received: from localhost ([::1]:46700 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kH2vR-0006Ng-Qj for ged-emacs-devel@m.gmane-mx.org; Sat, 12 Sep 2020 06:38:33 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:43012) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kH2uz-0005wy-Pb for emacs-devel@gnu.org; Sat, 12 Sep 2020 06:38:06 -0400 Original-Received: from fencepost.gnu.org ([2001:470:142:3::e]:33084) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kH2uy-0001JW-Op; Sat, 12 Sep 2020 06:38:04 -0400 Original-Received: from [176.228.60.248] (port=4180 helo=home-c4e4a596f7) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1kH2uw-0003HC-WA; Sat, 12 Sep 2020 06:38:04 -0400 In-Reply-To: (message from Daniel =?iso-8859-1?Q?Mart=EDn?= on Sat, 12 Sep 2020 12:25:17 +0200) X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:255293 Archived-At: > From: Daniel Martín > Cc: Michael Albinus , emacs-devel@gnu.org > Date: Sat, 12 Sep 2020 12:25:17 +0200 > > Stefan Kangas writes: > > > > The idea is to avoid that it will take a very long time to run the unit > > tests as the test suite grows. Arguably a unit test should never take > > longer than a second, and even that is in the slow end. If we have > > 10.000 unit tests taking a second each, just do the math of how long it > > will take to run (even with parallelization). We should not postpone > > working on this until we are in that situation, IMHO, because by then it > > will be a major pain. > > I agree. We should differentiate between unit tests (they run in under a > second) and end-to-end tests, which are slower but still valuable > because they cover whole scenarios that unit tests can't cover. Tramp tests need more time because they involve a remote system. Moreover, the time taken by each Tramp test depends on the speed of the connection, which you cannot know in advance. So I conclude that the "under one second" criterion is not smart enough to be applicable to Tramp tests, and maybe to some others as well. I also don't understand the more general issue with how long the test suite runs. While it runs, you can do something useful on a modern system (or just go for a coffee), so why does it matter? I've seen test suites that take a very long time to run (e.g., look at Texinfo or at Guile), and I never felt I was wasting my time by running those comprehensive test suites. I think we are exaggerating the importance of the time it takes to run the tests. > > I think the solution for important tests is to refactor the code to make > > them run faster, for example by avoiding I/O or to make timers trigger > > immediately. > > Yes, I think this is a good approach to follow in general. I don't, not in general. Artificially making such changes will run a risk of missing real problems, because the test runs in an environment different from a real one. This approach can be a good idea in some cases, but in general we should try to run each test as close to real-life conditions as possible.