From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: John Wiegley Newsgroups: gmane.emacs.devel Subject: Re: mark expensive tests Date: Sun, 03 Jan 2016 12:09:12 -0800 Message-ID: References: <8337vab7nx.fsf@gnu.org> <0d7fkmdxj1.fsf@fencepost.gnu.org> <566D0BEB.4010707@cs.ucla.edu> <52wpsif21j.fsf@fencepost.gnu.org> <6tpoy9aorv.fsf@fencepost.gnu.org> <83fuz54sfk.fsf@gnu.org> <566F5371.7010506@cs.ucla.edu> <87wprqzb0y.fsf_-_@gmx.de> NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Trace: ger.gmane.org 1451851786 22587 80.91.229.3 (3 Jan 2016 20:09:46 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Sun, 3 Jan 2016 20:09:46 +0000 (UTC) Cc: emacs-devel@gnu.org To: Michael Albinus Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Sun Jan 03 21:09:39 2016 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1aFoyA-0008Sx-Uw for ged-emacs-devel@m.gmane.org; Sun, 03 Jan 2016 21:09:39 +0100 Original-Received: from localhost ([::1]:42783 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aFoyA-0001Xm-CK for ged-emacs-devel@m.gmane.org; Sun, 03 Jan 2016 15:09:38 -0500 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:55306) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aFoxw-0001XR-Gw for emacs-devel@gnu.org; Sun, 03 Jan 2016 15:09:25 -0500 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aFoxr-0002Ln-PM for emacs-devel@gnu.org; Sun, 03 Jan 2016 15:09:24 -0500 Original-Received: from mail-pf0-x230.google.com ([2607:f8b0:400e:c00::230]:34166) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aFoxr-0002Lj-Iw for emacs-devel@gnu.org; Sun, 03 Jan 2016 15:09:19 -0500 Original-Received: by mail-pf0-x230.google.com with SMTP id e65so138048381pfe.1 for ; Sun, 03 Jan 2016 12:09:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:to:cc:subject:in-reply-to:date:message-id:references :user-agent:mail-followup-to:mime-version:content-type; bh=XwihQMHtF+wHKIn9fhkfglib5IQRG0JOI9cvL6vuEi8=; b=hBbQ6sFbBlAsehOoF6c2C7SUzoEJtxIE5v8nMEhoec3IX+ZT26uCKdMUayJK4ZtZdQ j+DlroBg1EQGQ8ZcZMuC7NKa/iqGRgWATgZuhylOtoVNodaUjwpaYjyVcCn+BIMZLz4X Xymlf7YgTd4d0ztGCRql/G+AehueipxE2WMncYT5NBU59ELQHRWcqEh96yKEOfIT6ea1 G3AXOQy502HlSoLYnz3YB9D/SWPj1a4/Fvm1LcEjbrvCwxBukmHeGYzL8Sq0R94ZgheY gegCX4E/Hdb02Zlfr1ztCtJE0gFo7+KKmbrxHP//I/CesNxVdqmMDYfXf4krbn5rTDr/ azZg== X-Received: by 10.98.67.67 with SMTP id q64mr50994982pfa.133.1451851759035; Sun, 03 Jan 2016 12:09:19 -0800 (PST) Original-Received: from Vulcan.local (76-234-68-79.lightspeed.frokca.sbcglobal.net. [76.234.68.79]) by smtp.gmail.com with ESMTPSA id y85sm49122854pfi.52.2016.01.03.12.09.18 (version=TLS1 cipher=AES128-SHA bits=128/128); Sun, 03 Jan 2016 12:09:18 -0800 (PST) X-Google-Original-From: "John Wiegley" Original-Received: by Vulcan.local (Postfix, from userid 501) id A12C0120003B8; Sun, 3 Jan 2016 12:09:17 -0800 (PST) In-Reply-To: <87wprqzb0y.fsf_-_@gmx.de> (Michael Albinus's message of "Sun, 03 Jan 2016 21:02:53 +0100") User-Agent: Gnus/5.130014 (Ma Gnus v0.14) Emacs/24.5 (darwin) Mail-Followup-To: Michael Albinus , emacs-devel@gnu.org X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2607:f8b0:400e:c00::230 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Xref: news.gmane.org gmane.emacs.devel:197497 Archived-At: --=-=-= Content-Type: text/plain Content-Transfer-Encoding: quoted-printable >>>>> Michael Albinus writes: >> That sounds like a great idea. I usually define three too: sanity tests, >> regular tests, extensive tests. > Do we have an agreement how to handle this? There was no further reaction > for 3 weeks ... Yes, we will have three test intensity levels: 1. Sanity tests just make sure that the environment works, and the whole = of them should run in <30 seconds on typical hardware. Some shops call th= ese 'smoke tests', as they are just there to make sure that Emacs can function in the most basic ways. 2. Regular tests do not incur intensive CPU or memory costs, and so can be run on any hardware. These should finish on a scale of minutes, likely <10 minutes at the most. 3. Extensive tests are given free reign, and may not even be able to complete on systems with CPU or memory constraints. These should feel free to take up to an hour, maybe even beyond. 4. Selective tests are never run automatically, but exist to stress test some particular area of the system. These could take days, it doesn't really matter what their requirements are. Each test should be marked, and the test runner should take these into acco= unt and provide options for picking which grade of test (or lower) to be run. T= his way I can tell Nix to build and run #2 tests. 'make check' should run #2, 'make fullcheck' should run #3, and there should be individual check targets for the tests in #4. I'd especially like the file-notify tests to move to #3, since these are wh= at consistently bog down my testing environment. Michael, are these changes you would be interested in making? =2D-=20 John Wiegley GPG fingerprint =3D 4710 CF98 AF9B 327B B80F http://newartisans.com 60E1 46C4 BD1A 7AC1 4BA2 --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQGcBAEBCgAGBQJWiX/oAAoJEMFE2PTxn+Ywz0UMAM/eA1Nf4nGwWGa/9IbfIiLg obVsVousOC+Q6JweUFP/KXiIAdqk16D25dkjUQEbOgkFrr7p2AFUei5RAa4vaNFi X5zUv4sJyqEPjsR77uK0QS7RRP9ZscHAvxAqY8kQISDOiKMZABlwfqpuwP+zHSbN tL94v5XUHWd7kmAXdX9jU+C2DbRD+ksCahzRIa19a+Tpx+mglfDlxQq293+6Al5o gr3a2STiLbUp3IEv9jzVjZsCEE/Ma8XxZbcEHTh00BaEYiqaWgeS5eUf7hCjK9OU 0P7TiJWGJbVJ3XdLl+cJH5MaoXZA0XGfBN3SyenSMIkAMzdGWXjW3ZOOhPGMLZu8 Ds476lPlWV97Xq0PV7fiufSghEHuOP5lksb9z64+sJHmbaGsfHjRTd5sM/Vd4OAI HqjOnCyoTuePEP8SV3SR8fCtadSzuAsZYZERBDCzrSQdB+1EBimwyhqXyMuEV8f6 eHWS3ZQa4ns370QZxBBLOEBpWclQ/j4RZ+hBLTNLSw== =kKVQ -----END PGP SIGNATURE----- --=-=-=--