From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: Eli Zaretskii Newsgroups: gmane.emacs.devel Subject: Re: mark expensive tests Date: Mon, 04 Jan 2016 17:47:56 +0200 Message-ID: <83si2d73df.fsf@gnu.org> References: <8337vab7nx.fsf@gnu.org> <0d7fkmdxj1.fsf@fencepost.gnu.org> <566D0BEB.4010707@cs.ucla.edu> <52wpsif21j.fsf@fencepost.gnu.org> <6tpoy9aorv.fsf@fencepost.gnu.org> <83fuz54sfk.fsf@gnu.org> <566F5371.7010506@cs.ucla.edu> <87wprqzb0y.fsf_-_@gmx.de> Reply-To: Eli Zaretskii NNTP-Posting-Host: plane.gmane.org X-Trace: ger.gmane.org 1451922495 18455 80.91.229.3 (4 Jan 2016 15:48:15 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Mon, 4 Jan 2016 15:48:15 +0000 (UTC) Cc: michael.albinus@gmx.de, emacs-devel@gnu.org To: John Wiegley Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Mon Jan 04 16:48:10 2016 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1aG7Mg-0004sZ-7m for ged-emacs-devel@m.gmane.org; Mon, 04 Jan 2016 16:48:10 +0100 Original-Received: from localhost ([::1]:45562 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aG7Mf-0001mH-JT for ged-emacs-devel@m.gmane.org; Mon, 04 Jan 2016 10:48:09 -0500 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:56893) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aG7MT-0001mA-0K for emacs-devel@gnu.org; Mon, 04 Jan 2016 10:47:57 -0500 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aG7MP-0006u6-0q for emacs-devel@gnu.org; Mon, 04 Jan 2016 10:47:56 -0500 Original-Received: from fencepost.gnu.org ([2001:4830:134:3::e]:35163) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aG7MO-0006u2-TZ; Mon, 04 Jan 2016 10:47:52 -0500 Original-Received: from 84.94.185.246.cable.012.net.il ([84.94.185.246]:3023 helo=HOME-C4E4A596F7) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_128_CBC_SHA1:128) (Exim 4.82) (envelope-from ) id 1aG7MO-0001xB-5O; Mon, 04 Jan 2016 10:47:52 -0500 In-reply-to: (message from John Wiegley on Sun, 03 Jan 2016 12:09:12 -0800) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Xref: news.gmane.org gmane.emacs.devel:197599 Archived-At: > From: John Wiegley > Date: Sun, 03 Jan 2016 12:09:12 -0800 > Cc: emacs-devel@gnu.org > > Yes, we will have three test intensity levels: > > 1. Sanity tests just make sure that the environment works, and the whole of > them should run in <30 seconds on typical hardware. Some shops call these > 'smoke tests', as they are just there to make sure that Emacs can > function in the most basic ways. > > 2. Regular tests do not incur intensive CPU or memory costs, and so can be > run on any hardware. These should finish on a scale of minutes, likely > <10 minutes at the most. > > 3. Extensive tests are given free reign, and may not even be able to > complete on systems with CPU or memory constraints. These should feel > free to take up to an hour, maybe even beyond. > > 4. Selective tests are never run automatically, but exist to stress test > some particular area of the system. These could take days, it doesn't > really matter what their requirements are. I'm fine with this gradation, but here's some reality check: . The current test suite takes 3.5 min for a full run, including compilation of all the *.el files, 2.25 min if the *.el files are already compiled . For comparison, the test suite of the latest version of Texinfo I just built takes 4.5 min on the same machine, even though a lot of tests are skipped (due to some required infrastructure not being installed) So it sounds like we have no candidates for #3 and #4, and the division between #1 and #2 is questionable, since 2 min is not such a long time to wait, IMO. > I'd especially like the file-notify tests to move to #3, since these are what > consistently bog down my testing environment. file-notify-tests takes about 25 sec here, so I'm unsure why it should be in #3. Maybe that's because of remote tests (which are skipped on my system), in which case perhaps the remote tests should be separated into a separate test file and run on demand only.