From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Eli Zaretskii Newsgroups: gmane.emacs.devel Subject: Re: master 262d0c6: Mark some tests as expensive Date: Sat, 12 Sep 2020 17:14:01 +0300 Message-ID: <83tuw3ccba.fsf@gnu.org> References: <20200910182904.20559.25935@vcs0.savannah.gnu.org> <20200910182905.F0E4520A2E@vcs0.savannah.gnu.org> <877dt0is0p.fsf@gmx.de> <83a6xve0vq.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="26386"; mail-complaints-to="usenet@ciao.gmane.io" Cc: michael.albinus@gmx.de, stefan@marxist.se, emacs-devel@gnu.org To: Daniel =?iso-8859-1?Q?Mart=EDn?= Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Sat Sep 12 16:14:34 2020 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1kH6IT-0006kE-Ss for ged-emacs-devel@m.gmane-mx.org; Sat, 12 Sep 2020 16:14:33 +0200 Original-Received: from localhost ([::1]:34356 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kH6IS-0005Lg-U6 for ged-emacs-devel@m.gmane-mx.org; Sat, 12 Sep 2020 10:14:32 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:59134) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kH6Hz-0004uw-6V for emacs-devel@gnu.org; Sat, 12 Sep 2020 10:14:03 -0400 Original-Received: from fencepost.gnu.org ([2001:470:142:3::e]:35996) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kH6Hy-0002uw-ID; Sat, 12 Sep 2020 10:14:02 -0400 Original-Received: from [176.228.60.248] (port=1650 helo=home-c4e4a596f7) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1kH6Hx-00037T-QY; Sat, 12 Sep 2020 10:14:02 -0400 In-Reply-To: (message from Daniel =?iso-8859-1?Q?Mart=EDn?= on Sat, 12 Sep 2020 16:00:00 +0200) X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:255350 Archived-At: > From: Daniel Martín > Cc: stefan@marxist.se, michael.albinus@gmx.de, emacs-devel@gnu.org > Date: Sat, 12 Sep 2020 16:00:00 +0200 > > Eli Zaretskii writes: > > > > Tramp tests need more time because they involve a remote system. > > Moreover, the time taken by each Tramp test depends on the speed of > > the connection, which you cannot know in advance. > > If they depend on the connection speed, or if a host is online or > offline, they also introduce non-determinism that perhaps is not > apparent now, but can cause problems when the Tramp codebase (and its > number of tests) scale. For example, if a Tramp test has a 0.1% chance > of failure because of an unrelated network problem, then if the Tramp > test suite reaches a point where 10000 tests are run per day, people > would be investigating 10 test flakes per day. That's a good reason for > people to lose confidence in the Tramp test suite and ignore failures. I don't understand what you are arguing here. AFAIK, Michael already either marked "unstable" or rewrote any tests that are known to suffer from non-determinism. Slow speed doesn't necessarily add non-determinism, not if Emacs waits for the operation to complete. So I don't see how this is relevant to the issue at hand, which is the criteria for marking tests "expensive" and not running them by default. What am I missing?