From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: =?utf-8?Q?Daniel_Mart=C3=ADn?= Newsgroups: gmane.emacs.devel Subject: Re: master 262d0c6: Mark some tests as expensive Date: Sat, 12 Sep 2020 12:25:17 +0200 Message-ID: References: <20200910182904.20559.25935@vcs0.savannah.gnu.org> <20200910182905.F0E4520A2E@vcs0.savannah.gnu.org> <877dt0is0p.fsf@gmx.de> Mime-Version: 1.0 Content-Type: text/plain Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="36535"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (darwin) Cc: Michael Albinus , emacs-devel@gnu.org To: Stefan Kangas Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Sat Sep 12 12:26:27 2020 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1kH2jg-0009Le-Kw for ged-emacs-devel@m.gmane-mx.org; Sat, 12 Sep 2020 12:26:24 +0200 Original-Received: from localhost ([::1]:54450 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kH2jf-0005mE-Jo for ged-emacs-devel@m.gmane-mx.org; Sat, 12 Sep 2020 06:26:23 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:40090) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kH2ip-0004np-Pr for emacs-devel@gnu.org; Sat, 12 Sep 2020 06:25:31 -0400 Original-Received: from sonic314-19.consmr.mail.ir2.yahoo.com ([77.238.177.145]:38308) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kH2im-00081V-AR for emacs-devel@gnu.org; Sat, 12 Sep 2020 06:25:31 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.es; s=s2048; t=1599906323; bh=D0OXNYQa0FXquneq4FdhLd1evWyRUHvtonGJrt0MmGc=; h=From:To:Cc:Subject:References:Date:In-Reply-To:From:Subject; b=YPU/Kod4Omm9D0bGQPYnCSCIHvo9jcEl0LVfQk9QZSS4zUTwK9vFQz7P9TpS6IVuR66BlZG7PanZDAudOeznqha2r27F1H7YMclnFSzKmy84E4U+QNR5SdvMHneuL5YngAWzjN+KIx1ECjA8qqvdo7Lz4GYAtpgqgTkXcx6XVi+v61OT7kxbJnTwB/S0MSJiHnEdiooLRiEsNSC/rqFjQUmE9/xBgUUcf3lqCQxvNbrql79/AvHQHf1kxLKuPh0hJHr1UMIPm9oej5bb7PNM2A+5npngz6XM6YSxl3OIVvUIM0A6UekTu0l8hZozdGJN/aHyqgLAnTMTJazpni+AGw== X-YMail-OSG: 8tk4Bv0VM1kx4M_JeqagYVb1cdZ2dPiebkh3c.zvL50nxXhThZIGxyl0jj6C0xV XhyOaEh9kK.1eTeO_LqzbgHiROvPDHdi2HPFWZK8raKMQgli06BHdPv433PskX6LwesLkrKrBpyb 0Jk.ixQVhBpF9jrQkJEmt4t_k.1KY9Wxl5lHDRsIV9Xiemx9e6.N78sUVRUdU_2pJA3XGzR1FI_l 02MoXJbyZDA7X6PZgmQXG3_SHsRB3XW25_wUd9GW19YqO4yknCgqXcrnk2kJSplXgukp_YYwj0nR vZy83uGIvmF9M_vsDuDDDEp8OpsKmVRbFOFUm1Qw5JT82I7TPXxStX34zV1cwCLKYCRIw6C24VX4 822W5mGY_0ossJUY41W0waSq.tY_vl530uwxtopeH6PYYTViP08U9G4z.xBHE2ww03MWOEzui1Tg a9KnIwoBRe7niLYZtdrb093ss6Sa1SkOcyVAO35rDXSE53wH4m3.zsyfy6egQJYLOpehbrJMNEVH UfFWPgEjAFuG5CidgwomMVuslqb3fhXRetZiUCCo.THS1DGFaD7DHrNq4Wpj2OEk1At3RH2_tWih pUEDKMxI7SGwpwmOHBS58QB3R05Jmpsp5xXj6qaf9Wqnxj30BimD77RwnwawXyX6JZi.CPlhlYeN 9UavHxHFrxPvmZQJ3zbJO5BUzZ.wr2rlraIKnveuzKzwkefAos.9Fxah85lskZW2fHohpPzQOkKS sIw86B..mpbiz7zKyyTpPNlMXxqGr9h9a8jZNUuKvmtcARP8nM.ZpGod289ov0dzBxrK_tIWvmJd ylIdf0B1Adqi92m.treTRys8mgZ2cEMptn84EstU.X Original-Received: from sonic.gate.mail.ne1.yahoo.com by sonic314.consmr.mail.ir2.yahoo.com with HTTP; Sat, 12 Sep 2020 10:25:23 +0000 Original-Received: by smtp421.mail.ir2.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID 394869fc8a42254284662d32e2c0cf05; Sat, 12 Sep 2020 10:25:19 +0000 (UTC) In-Reply-To: (Stefan Kangas's message of "Fri, 11 Sep 2020 11:06:48 -0700") X-Mailer: WebService/1.1.16565 mail.backend.jedi.jws.acl:role.jedi.acl.token.atz.jws.hermes.yahoo Apache-HttpAsyncClient/4.1.4 (Java/11.0.7) Received-SPF: pass client-ip=77.238.177.145; envelope-from=mardani29@yahoo.es; helo=sonic314-19.consmr.mail.ir2.yahoo.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/09/12 06:25:23 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:255291 Archived-At: Stefan Kangas writes: > > The idea is to avoid that it will take a very long time to run the unit > tests as the test suite grows. Arguably a unit test should never take > longer than a second, and even that is in the slow end. If we have > 10.000 unit tests taking a second each, just do the math of how long it > will take to run (even with parallelization). We should not postpone > working on this until we are in that situation, IMHO, because by then it > will be a major pain. I agree. We should differentiate between unit tests (they run in under a second) and end-to-end tests, which are slower but still valuable because they cover whole scenarios that unit tests can't cover. To keep things scalable, we should strive to have plenty of fast unit tests and fewer end-to-end tests. See https://martinfowler.com/bliki/TestPyramid.html > > I think the solution for important tests is to refactor the code to make > them run faster, for example by avoiding I/O or to make timers trigger > immediately. Yes, I think this is a good approach to follow in general. Many of the test flakiness in the Emacs suite happen because many tests depend implicitly in things outside the module they are testing (like configuration, OS, etc.). Having more stable tests would help us detect real regressions more accurately. > > Best regards, > Stefan Kangas