From mboxrd@z Thu Jan 1 00:00:00 1970 From: Pjotr Prins Subject: Treating tests as special case Date: Thu, 5 Apr 2018 07:24:39 +0200 Message-ID: <20180405052439.GA30291@thebird.nl> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:43292) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1f3xSc-0001G7-Dr for guix-devel@gnu.org; Thu, 05 Apr 2018 01:29:23 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1f3xSY-0002gg-Fk for guix-devel@gnu.org; Thu, 05 Apr 2018 01:29:22 -0400 Received: from mail.thebird.nl ([95.154.246.10]:60218) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1f3xSY-0002e6-8K for guix-devel@gnu.org; Thu, 05 Apr 2018 01:29:18 -0400 Content-Disposition: inline List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: guix-devel@gnu.org Last night I was watching Rich Hickey's on Specs and deployment. It is a very interesting talk in many ways, recommended. He talks about tests at 1:02 into the talk: https://www.youtube.com/watch?v=oyLBGkS5ICk and he gave me a new insight which rang immediately true. He said: what is the point of running tests everywhere? If two people test the same thing, what is the added value of that? (I paraphrase) With Guix a reproducibly building package generates the same Hash on all dependencies. Running the same tests every time on that makes no sense. And this hooks in with my main peeve about building from source. The building takes long enough. Testing takes incredibly long with many packages (especially language related) and are usually single core (unlike the build). It is also bad for our carbon foot print. Assuming everyone uses Guix on the planet, is that where we want to end up? Burning down the house. Like we pull substitutes we could pull a list of hashes of test cases that are known to work (on Hydra or elsewhere). This is much lighter than storing substitutes, so when the binaries get removed we can still retain the test hashes and have fast builds. Also true for guix repo itself. I know there are two 'inputs' I am not accounting for: (1) hardware variants and (2) the Linux kernel. But, honestly, I do not think we are in the business of testing those. We can assume these work. If not, any issues will be found in other ways (typically a segfault ;). Our tests are generally meaningless when it comes to (1) and (2). And packages that build differently on different platforms, like openblas, we should opt out on. I think this would be a cool innovation (in more ways than one). Pj.