From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Lars Ingebrigtsen Newsgroups: gmane.emacs.devel Subject: Re: master 262d0c6: Mark some tests as expensive Date: Sun, 13 Sep 2020 14:30:07 +0200 Message-ID: <87zh5tzwog.fsf@gnus.org> References: <20200910182904.20559.25935@vcs0.savannah.gnu.org> <20200910182905.F0E4520A2E@vcs0.savannah.gnu.org> <877dt0is0p.fsf@gmx.de> <83a6xve0vq.fsf@gnu.org> <87a6xv456t.fsf@gnus.org> <837dszdyqv.fsf@gnu.org> <877dsz2o03.fsf@gnus.org> <83363ndvql.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="24356"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (gnu/linux) Cc: stefan@marxist.se, emacs-devel@gnu.org, michael.albinus@gmx.de, mardani29@yahoo.es To: Eli Zaretskii Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Sun Sep 13 14:31:19 2020 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1kHRA6-0006Ee-UU for ged-emacs-devel@m.gmane-mx.org; Sun, 13 Sep 2020 14:31:18 +0200 Original-Received: from localhost ([::1]:59686 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kHRA6-0008O3-0b for ged-emacs-devel@m.gmane-mx.org; Sun, 13 Sep 2020 08:31:18 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:48404) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kHR9B-0007kd-D4 for emacs-devel@gnu.org; Sun, 13 Sep 2020 08:30:21 -0400 Original-Received: from quimby.gnus.org ([2a01:4f9:2b:f0f::2]:33852) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kHR99-0000gP-NZ; Sun, 13 Sep 2020 08:30:21 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnus.org; s=20200322; h=Content-Type:MIME-Version:Message-ID:In-Reply-To:Date: References:Subject:Cc:To:From:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=YYRQ0HwuYgyzh9vy7/XwLII8Uzu5wjYgoL+uv6UAtiM=; b=mJmhciloQXYL41cmZR8GtHlwCj 52QqnmMI+xo42B9Vm41dI+LkcDaLI6s4YEaa+y/ThwFwfHTD3Wke+LnmUnVzxJF1Y4I32wzNQ77Af XAN/Cbxl9UOekhwobZO/2yqTjJDvLHoArHuVodDfqrhHItMgmMkxhOLQupAnvIvsVkAY=; Original-Received: from cm-84.212.202.86.getinternet.no ([84.212.202.86] helo=xo) by quimby with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1kHR8y-0002qc-T5; Sun, 13 Sep 2020 14:30:11 +0200 X-Now-Playing: =?utf-8?B?UsOzaXPDrW4=?= Murphy's _Jacuzzi Rollercoaster_: =?utf-8?B?IlLDs2lzw61u?= Murphy - Can't Hang On" In-Reply-To: <83363ndvql.fsf@gnu.org> (Eli Zaretskii's message of "Sat, 12 Sep 2020 15:29:06 +0300") Received-SPF: pass client-ip=2a01:4f9:2b:f0f::2; envelope-from=larsi@gnus.org; helo=quimby.gnus.org X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:255467 Archived-At: Eli Zaretskii writes: >> Yes, if they take more than a second to run. > > So how do we make sure some change didn't introduce a bug in those > features? We either say "make check-expensive", or we live on in uncertainty, as with all those other functions we've marked as expensive. >> We've already made this decision with all the tests previously marked as >> expensive, so there's nothing new here. > > There's a large gap between what is currently marked as "expensive" > tests and having entire packages not tested at all. The latter sounds > too radical to me. E.g., auto-revert is an important feature, used by > many people. Not having it in regression testing sounds like a step > backward to me. It's still being tested -- just not as often. > I'm talking about a balance here. Losing the tests of complete > features just because we want tests to finish quickly sounds > sub-optimal to me. Can we make a smarter balance? Sure, there's a balance, and I think the current balance is a bit too skewed towards having too many slow tests in the "make check". I think that, ideally, "make check" should be so fast that people run it as their standard workflow before pushing a change, and we're not quite there. Having people do "make check" as a matter of routine, and running 97% of the tests is, in my opinion, better than people doing a "make check" seldom, but running 98% of the tests. Where the cutoff is a matter of balance, yes, but I think a test that takes a second is way too slow to be run in the routine case. -- (domestic pets only, the antidote for overdose, milk.) bloggy blog: http://lars.ingebrigtsen.no