From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Michael Albinus Newsgroups: gmane.emacs.devel Subject: Re: master 262d0c6: Mark some tests as expensive Date: Sat, 12 Sep 2020 12:52:26 +0200 Message-ID: <87pn6rgtcl.fsf@gmx.de> References: <20200910182904.20559.25935@vcs0.savannah.gnu.org> <20200910182905.F0E4520A2E@vcs0.savannah.gnu.org> <877dt0is0p.fsf@gmx.de> Mime-Version: 1.0 Content-Type: text/plain Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="35767"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (gnu/linux) Cc: emacs-devel@gnu.org To: Stefan Kangas Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Sat Sep 12 12:55:26 2020 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1kH3Bm-0009Du-Am for ged-emacs-devel@m.gmane-mx.org; Sat, 12 Sep 2020 12:55:26 +0200 Original-Received: from localhost ([::1]:50890 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kH3Bl-0000Xd-7c for ged-emacs-devel@m.gmane-mx.org; Sat, 12 Sep 2020 06:55:25 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:44778) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kH390-0007wS-D7 for emacs-devel@gnu.org; Sat, 12 Sep 2020 06:52:34 -0400 Original-Received: from mout.gmx.net ([212.227.17.22]:33621) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kH38y-0002kH-5Y for emacs-devel@gnu.org; Sat, 12 Sep 2020 06:52:34 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1599907948; bh=WmSoaggtRc+i0WNNQpe/zkJkahSe7KTnLN4tTurSKhY=; h=X-UI-Sender-Class:From:To:Cc:Subject:References:Date:In-Reply-To; b=AMlpNkQhzIUDGSiSGXO/O6hujzeIn3cacFEc4X9eMzVwEocGoZnZ09qgxdK9yF8k3 YHSUdmLFEc1C1BRS17bbH69y+nA5apBWLuXVB1BPrmIQGk9wZzDt2mrijjqXWargSk aO4S647KIqAUdI3R6LjQ6dcnbRigy0T/8ONQRroM= X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c Original-Received: from gandalf.gmx.de ([212.86.41.134]) by mail.gmx.com (mrgmx104 [212.227.17.168]) with ESMTPSA (Nemesis) id 1M8QS8-1kCfQi0Oye-004Rm0; Sat, 12 Sep 2020 12:52:28 +0200 In-Reply-To: (Stefan Kangas's message of "Fri, 11 Sep 2020 11:06:48 -0700") X-Provags-ID: V03:K1:CzCDp0itYQn6TQAKYPvLOz7l/Mb/gYXcHTKYja2VweVBYs6N7Za Fvth+0yZRLQ9Qt2icZ+Rz4uacjzV3o2y+Py3T7GYoqdiQ4UMu9sZiVFxH2fGP3TwjdQWmds za/DGlo/0FWWsyqhxJeR+Izfi6oNiHgNRVeqU80622HE4zCiqIHIo/pE8hutMWV+AWZHsfC IJMRO7DckG7JJbHWIAUEA== X-UI-Out-Filterresults: notjunk:1;V03:K0:52D6Zgsk090=:UeEjdql3Y5gXWr0MCp/9vs gv6lxJhzZRPlRZz625OTy+pufhIeYDufo/XKSMY8iT8sO0N/j+X2mgYnYq9nx8oeWujclCj0o NYTW5ZxUTbzBu3v0b3xcD7FdS8hLedV/ALNGMulu3x7m6SrANfO4QACUhVjZwblnGRFuMOUBB STHxmW1AjYLj5wCaQl6rW0a4e9U07OZursowwWfCnO1UYu3vJsxXSCFtjqPlJUYUbqj7YY1fA EK5ElrPbNdrDIz/OSwKxK/yKITUqf8Ykr1bY6GBCEDpadznCsIhwRlHC37eEbyc5vh3kRAFh+ LdINUSXAR6gvC14zLgPjBS1zc3hT1kAYQAE2/9BoVIjOhwrIP3vknn6Wb9pMso8nZdQ0WRCGP yvF5bNUz4Frr3YbgEfhe3rOVTD2xgu9LVwGYuKNGSnbD7eCsOFfnCXPvi8u9y6Mf27MZbJbNe o/3Bg+Uo4nW9iwtw+xeGgSHSGzW7A0bE64CfFYK+08kClQpCTvkRkmu9MTJETJ4bqECkm0Okl YCNmRm/O4T1ymHWTjmLEh11iPe16gdlhNcINXUoM5suNPLvQCVxETNKvAubY3mf/ysPy4Cvc+ 624lSNhNzd3oWSp9e3Lm+xV3LC3CCjm+75G9C+U8T/FXbMalrOoyLANbIaZzF4iwZiKNPUiDS v2FSWdJm9acVrKQYPDEOZ7NJAwXAtrxcFd7qyZemATuJQyQ6GW5tEeXisS3riswNWRjvR4+f3 HMmApuqXuTvammjfIOzyHVI4tAR8k21MQRDi3qQGq2i54HGDOrW+Z1lW9ViwUenRh60XSX+v Received-SPF: pass client-ip=212.227.17.22; envelope-from=michael.albinus@gmx.de; helo=mout.gmx.net X-detected-operating-system: by eggs.gnu.org: First seen = 2020/09/12 06:13:21 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] X-Spam_score_int: -25 X-Spam_score: -2.6 X-Spam_bar: -- X-Spam_report: (-2.6 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:255295 Archived-At: Stefan Kangas writes: > Hi Michael, Hi Stefan, >> Could you please give a reasoning? Declaring tests as expansive decreases >> heavily their application, giving us less chances to detect errors. > > These tests all took 5-60 seconds to run, most in the lower end > admittedly. With your change, you have tagged *all* autorevert tests as :expensive-test. In practice this means, they won't run by default ever, because people call "make check" only. Any error there, which is not related to GNU/Linux, will be hidden (hydra and emba run "make check-expensive", but only for GNU/Linux). I don't think this is your intention. > The idea is to avoid that it will take a very long time to run the unit > tests as the test suite grows. Arguably a unit test should never take > longer than a second, and even that is in the slow end. If we have > 10.000 unit tests taking a second each, just do the math of how long it > will take to run (even with parallelization). We should not postpone > working on this until we are in that situation, IMHO, because by then it > will be a major pain. There have been discussions about how long tests shall run. But there hasn't been ever a conclusion to limit them to 1 second. Please discuss this first, before making such changes. > Best regards, > Stefan Kangas Best regards, Michael.