From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: =?utf-8?Q?Daniel_Mart=C3=ADn?= Newsgroups: gmane.emacs.devel Subject: Re: master 262d0c6: Mark some tests as expensive Date: Sat, 12 Sep 2020 16:00:00 +0200 Message-ID: References: <20200910182904.20559.25935@vcs0.savannah.gnu.org> <20200910182905.F0E4520A2E@vcs0.savannah.gnu.org> <877dt0is0p.fsf@gmx.de> <83a6xve0vq.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="11070"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (darwin) Cc: stefan@marxist.se, michael.albinus@gmx.de, emacs-devel@gnu.org To: Eli Zaretskii Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Sat Sep 12 16:01:17 2020 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1kH65b-0002mB-M9 for ged-emacs-devel@m.gmane-mx.org; Sat, 12 Sep 2020 16:01:15 +0200 Original-Received: from localhost ([::1]:57366 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kH65a-0002Lk-Mg for ged-emacs-devel@m.gmane-mx.org; Sat, 12 Sep 2020 10:01:14 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:56646) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kH64k-0001dh-Cj for emacs-devel@gnu.org; Sat, 12 Sep 2020 10:00:22 -0400 Original-Received: from sonic313-20.consmr.mail.ir2.yahoo.com ([77.238.179.187]:33201) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kH64h-0001XC-Pi for emacs-devel@gnu.org; Sat, 12 Sep 2020 10:00:22 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.es; s=s2048; t=1599919216; bh=srDSmWCcBQl/cm+bYneHG77ThdukdFmf55kDkMEIfwU=; h=From:To:Cc:Subject:References:Date:In-Reply-To:From:Subject; b=gtnvVw/FVDDdpUhwTnJ05CcRDEbdl7VNPyWRKOYpqnFvvIBI3PEeYPvDac4ITc0Nm6lLIRGWsdEi5P9WKKwWZDdx34F21ftZfyn9TVE+viYoIrfsUZfIIiA/5sT/Ipb5Qf49xXEBiHOTy3ymbutLgLbiHf2MFB/ogs6IaP+Avoqg10vrAxs3byITe9wrS1mul9K4VMTgFHLkK4x7gwY89cgQ0mIzZFh8D66f2bP+mvlN/hsw2AaOmNfXjIkBPNTVOaJj+Q/bLM/WytkIFiIDsksQatjnRJs9DyYXzg3R9yLtRfRJrtk5BRLvOhN2O8oLYZNSjvvP/1dlD4yB7R1AKg== X-YMail-OSG: KjJMkLMVM1mtCka3WuM7GQZon3Puep4zSPapvGy50O41hMrtEBuX5IGyMfcAm4Z XrlJ58Ic05LAtDxQCHz0mr8oknuXbc6.YSd823T.FinUGuCZyHAiAFvdYvIMxPP77nJmoV9LPm1t 5RrC_vlm9DLop.EOyZ8E2Kp1cr8GqYmomi7qs6KPgjZox2cGG9QayZv8wXpnRlG62Zzg6WynTReS iVxym0F4cjf9AMg6XWxdjzrLfBWo.croXXWH.3HEunvz5rUmLK6qwhpj8624WNaXhCRm1xMSQc_6 p_a2RtzPLRHzgOaKGeRQAsgd_qfFtESwIn3kZK6uP.1IB7mYUZUqvKPka4pgyf4saxC7XDUG6SZj HgkXFl8DYm1skaU_6UL6na8RKE.BkQz4kMdDc2tiU0Uu1WcIcEloIjuyC50NGu_KCTmJNk9._OMf cJST.JA_5XIbxxAOAI7wvbbZR5lKOn7yJ_ULSTbjBEu.mnZsCrkZ..1XmcC54sAdQwiCuLg45dGZ NsihzbxLum_Ml86_UbOAReEO_YKEGYwG9aLgKq4a8B9pygHeQ0bL5_h6k1l2aVCBzUjm5DiN1ti7 2vjGAm10.5XPkAbjqcPoYncoHP6DoE.ITvA1oWw4VKZzecUn_qmJM53jPGbo9ajTWkFvsYWEkcnZ XWF1xFxATh7WCpPp9Z_ft8FMlpsGfDSE3VZPjee63jMXIxPt5wzMwtXtWd8Gj9K.NrJD3m4nnsoB Iy61OCjJYfXtqTwuHK.GrNSAPy0OngA6xvlHDpFk6vcTUV5OMuagzeoeF0H5GibjsfPYWIva3n6a 2ZMVfzaZ.v_l5x1rfEUI0_EyehIA479PwllbZpEqf_ Original-Received: from sonic.gate.mail.ne1.yahoo.com by sonic313.consmr.mail.ir2.yahoo.com with HTTP; Sat, 12 Sep 2020 14:00:16 +0000 Original-Received: by smtp422.mail.ir2.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID b291c9fde8e87c48806ce10986234921; Sat, 12 Sep 2020 14:00:12 +0000 (UTC) In-Reply-To: <83a6xve0vq.fsf@gnu.org> (Eli Zaretskii's message of "Sat, 12 Sep 2020 13:38:01 +0300") X-Mailer: WebService/1.1.16565 mail.backend.jedi.jws.acl:role.jedi.acl.token.atz.jws.hermes.yahoo Apache-HttpAsyncClient/4.1.4 (Java/11.0.7) Received-SPF: pass client-ip=77.238.179.187; envelope-from=mardani29@yahoo.es; helo=sonic313-20.consmr.mail.ir2.yahoo.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/09/12 10:00:16 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:255348 Archived-At: Eli Zaretskii writes: > > Tramp tests need more time because they involve a remote system. > Moreover, the time taken by each Tramp test depends on the speed of > the connection, which you cannot know in advance. If they depend on the connection speed, or if a host is online or offline, they also introduce non-determinism that perhaps is not apparent now, but can cause problems when the Tramp codebase (and its number of tests) scale. For example, if a Tramp test has a 0.1% chance of failure because of an unrelated network problem, then if the Tramp test suite reaches a point where 10000 tests are run per day, people would be investigating 10 test flakes per day. That's a good reason for people to lose confidence in the Tramp test suite and ignore failures. One could argue that the Tramp test suite will never reach such a high number of tests/contributions, and maybe that's fine, but the problem of scale exists, IMHO. > > I don't, not in general. Artificially making such changes will run a > risk of missing real problems, because the test runs in an environment > different from a real one. This approach can be a good idea in some > cases, but in general we should try to run each test as close to > real-life conditions as possible. I'd say "we should try to run *some* tests as close to real-life conditions as possible". By abstracting the environment in some tests, one could potentially test an infinite number of environments and error conditions, not only what the test happens to run on. We would still have a few end-to-end tests that check that the program as a whole works fine, of course. The trade-off is that writing that kind of hermetic tests takes more time, specially for packages like Tramp. Having said that, I think that covering Emacs functionality with an end-to-end test is much better than having no test at all. But I'd expect some of those end-to-end tests marked as ":expensive-test" to evolve into smaller, more hermetic ones, as the test suite matures.