From mboxrd@z Thu Jan 1 00:00:00 1970 From: Pjotr Prins Subject: Re: Treating tests as special case Date: Fri, 6 Apr 2018 08:06:01 +0200 Message-ID: <20180406060601.GA4343@thebird.nl> References: <20180405052439.GA30291@thebird.nl> <87tvsp4cxh.fsf@netris.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:51351) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1f4KaD-00022N-Eb for guix-devel@gnu.org; Fri, 06 Apr 2018 02:10:46 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1f4KaA-00063A-9s for guix-devel@gnu.org; Fri, 06 Apr 2018 02:10:45 -0400 Received: from mail.thebird.nl ([95.154.246.10]:33817) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1f4KaA-000609-2I for guix-devel@gnu.org; Fri, 06 Apr 2018 02:10:42 -0400 Content-Disposition: inline In-Reply-To: <87tvsp4cxh.fsf@netris.org> List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: Mark H Weaver Cc: guix-devel@gnu.org On Thu, Apr 05, 2018 at 04:26:50PM -0400, Mark H Weaver wrote: > Tests on different hardware/kernel/kernel-config/file-system > combinations are quite useful for those who care about reliability of > their systems. I, for one, would like to keep running test suites on my > own systems. Sure. And it is a great example why to test scenarios. But why force it down everyone's throat? I don't want to test Scipy or ldc over and over again. Note that I can work around it, but we are forcing our methods here on others. If I do not like it, others won't. I am just looking at running test billion times uselessly around the planet. Does that not matter? We need to be green. Ludo is correct that provisioning binary substitutes is one solution. But not cheap. Can we guarantee keeping all substitutes? At least the ones with long running tests ;). I don't know how we remove substitutes now, but it would make sense to me to base that on download metrics and size. How about ranking downloads in the last 3 months times the time to build? And trim from the end. That may be interesting. Even so, with my idea of test substitutes you don't have to opt out of testing. And you would still have found that bug. Those who care can test all they please. Anyway, that is enough. I made my point and I am certain that we will change our ways at some point. The laborious solution is to remove all meaningless tests. And I am sure over 90% are pretty damn meaningless for our purposes. Like the glut in binaries, we will trim it down over time. One suggestion: let's also look at tests that are *not* about integration or hardware/kernel configuration and allow for running them optionally. Stupidly running all tests that people come up with is not a great idea. We just run what authors decide that should be run. Pj.