From mboxrd@z Thu Jan 1 00:00:00 1970 From: Pjotr Prins Subject: Re: Treating tests as special case Date: Thu, 5 Apr 2018 10:43:00 +0200 Message-ID: <20180405084300.GB31585@thebird.nl> References: <20180405052439.GA30291@thebird.nl> <20180405082115.60e604a6@alma-ubu> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:55110) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1f40YS-0007P0-Ls for guix-devel@gnu.org; Thu, 05 Apr 2018 04:47:37 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1f40YP-0003JR-HI for guix-devel@gnu.org; Thu, 05 Apr 2018 04:47:36 -0400 Received: from mail.thebird.nl ([95.154.246.10]:60352) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1f40YP-0003IH-AR for guix-devel@gnu.org; Thu, 05 Apr 2018 04:47:33 -0400 Content-Disposition: inline In-Reply-To: <20180405082115.60e604a6@alma-ubu> List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: =?iso-8859-1?Q?Bj=F6rn_H=F6fling?= Cc: guix-devel@gnu.org On Thu, Apr 05, 2018 at 08:21:15AM +0200, Bj=F6rn H=F6fling wrote: > great ideas! >=20 > Last night I did a=20 >=20 > guix pull && guix package -i git >=20 > We have substitutes, right? Yeah, but someone updated git, on my new > machine I didn't configure berlin.guixsd.org yet and hydra didn't have > any substitutes (build wasn't started yet?). >=20 > Building git was relatively fast, but all the tests took ages. And it > was just git. It should work. The git maintainers ran the tests. Marius > when he updated it in commit 5c151862c ran the tests. And that should > be enough of testing. Let's skip the tests. Not exactly what I am proposing ;). But, even so, I think we should have a switch for turning off tests. Let the builder decide what is good or bad. Too much nannying serves no one. > On the other hand, if I create a new package definition and forget to > run the tests. If upstream is too sloppy, did not run the tests and had > no continuous integration. Who will run the tests then? Hydra should always test before providing a hash that testing is done. > What if I build my package with different sources? >=20 > And you mentioned different environment conditions like machine and > kernel. We still have "only" 70-90% reproducibility. The complement > should have tests enabled. And the question "is my package > reproducible?" is not trivial to answer, and is not computable. Well, I believe that case is overrated and we prove that by actually providing binary substitutes without testing ;) > We saw tests that failed only in 2% of the runs and were fine in 98%. > If we would run those tests "just once", we couldn't figure out that > there is a problem (assuming the problem really is in the software, not > just the tests). >=20 > There could also be practible problems with that: If all write there > software nice and with autoconfigure and we just have a "make && make > test && make install" it's easy to skip the test. But for more > complicated things we have to find a way to tell the build-system how > to skip tests. Totally agree. At this point I patch the tree not to run tests. Pj.