From mboxrd@z Thu Jan 1 00:00:00 1970 From: =?UTF-8?Q?G=C3=A1bor_Boskovits?= Subject: Re: Treating tests as special case Date: Thu, 5 Apr 2018 08:05:39 +0200 Message-ID: References: <20180405052439.GA30291@thebird.nl> Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="001a113eb8d49da3e7056913bad3" Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:50399) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1f3y1m-0005RT-E3 for guix-devel@gnu.org; Thu, 05 Apr 2018 02:05:43 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1f3y1l-0006AR-0s for guix-devel@gnu.org; Thu, 05 Apr 2018 02:05:42 -0400 Received: from mail-io0-x22a.google.com ([2607:f8b0:4001:c06::22a]:37806) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1f3y1k-0006AA-PT for guix-devel@gnu.org; Thu, 05 Apr 2018 02:05:40 -0400 Received: by mail-io0-x22a.google.com with SMTP id y128so29157758iod.4 for ; Wed, 04 Apr 2018 23:05:40 -0700 (PDT) In-Reply-To: <20180405052439.GA30291@thebird.nl> List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: Pjotr Prins Cc: Guix-devel --001a113eb8d49da3e7056913bad3 Content-Type: text/plain; charset="UTF-8" 2018-04-05 7:24 GMT+02:00 Pjotr Prins : > Last night I was watching Rich Hickey's on Specs and deployment. It is > a very interesting talk in many ways, recommended. He talks about > tests at 1:02 into the talk: > > https://www.youtube.com/watch?v=oyLBGkS5ICk > > and he gave me a new insight which rang immediately true. He said: > what is the point of running tests everywhere? If two people test the > same thing, what is the added value of that? (I paraphrase) Actually running tests test the behaviour of a software. Unfortunately reproducible build does not guarantee reproducible behaviour. Furthermore there are still cases, where the environment is not the same around these running software, like hardware or kernel configuration settings leaking into the environment. These can be spotted by running tests. Nondeterministic failures can also be spotted more easily. There are a lot of packages where pulling tests can be done, I guess, but probably not for all of them. WDYT? > > With Guix a reproducibly building package generates the same Hash on > all dependencies. Running the same tests every time on that makes no > sense. > > And this hooks in with my main peeve about building from source. The > building takes long enough. Testing takes incredibly long with many > packages (especially language related) and are usually single core > (unlike the build). It is also bad for our carbon foot print. Assuming > everyone uses Guix on the planet, is that where we want to end up? > > Burning down the house. > > Like we pull substitutes we could pull a list of hashes of test cases > that are known to work (on Hydra or elsewhere). This is much lighter > than storing substitutes, so when the binaries get removed we can > still retain the test hashes and have fast builds. Also true for guix > repo itself. > > I know there are two 'inputs' I am not accounting for: (1) hardware > variants and (2) the Linux kernel. But, honestly, I do not think we > are in the business of testing those. We can assume these work. If > not, any issues will be found in other ways (typically a segfault ;). > Our tests are generally meaningless when it comes to (1) and (2). And > packages that build differently on different platforms, like openblas, > we should opt out on. > > I think this would be a cool innovation (in more ways than one). > > Pj. > > --001a113eb8d49da3e7056913bad3 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
2018= -04-05 7:24 GMT+02:00 Pjotr Prins <pjotr.public12@thebird.nl&g= t;:
Last night I was watching Rich= Hickey's on Specs and deployment. It is
a very interesting talk in many ways, recommended. He talks about
tests at 1:02 into the talk:

=C2=A0 https://www.youtube.com/watch?v=3DoyLBGkS5I= Ck

and he gave me a new insight which rang immediately true. He said:
what is the point of running tests everywhere? If two people test the
same thing, what is the added value of that? (I paraphrase)

Actually running tests test the behaviour of a software. U= nfortunately
reproducible build does not guarantee reproducible b= ehaviour.
Furthermore there are still cases, where the environmen= t is
not the same around these running software, like hardware or=
kernel configuration settings leaking into the environment.
These can be spotted by running tests. Nondeterministic
fai= lures can also be spotted more easily. There are a lot of
package= s where pulling tests can be done, I guess, but probably not
for = all of them. WDYT?=C2=A0
=C2=A0
With Guix a reproducibly building package generates the same Hash on
all dependencies. Running the same tests every time on that makes no
sense.

And this hooks in with my main peeve about building from source. The
building takes long enough. Testing takes incredibly long with many
packages (especially language related) and are usually single core
(unlike the build). It is also bad for our carbon foot print. Assuming
everyone uses Guix on the planet, is that where we want to end up?

Burning down the house.

Like we pull substitutes we could pull a list of hashes of test cases
that are known to work (on Hydra or elsewhere). This is much lighter
than storing substitutes, so when the binaries get removed we can
still retain the test hashes and have fast builds. Also true for guix
repo itself.

I know there are two 'inputs' I am not accounting for: (1) hardware=
variants and (2) the Linux kernel. But, honestly, I do not think we
are in the business of testing those. We can assume these work. If
not, any issues will be found in other ways (typically a segfault ;).
Our tests are generally meaningless when it comes to (1) and (2). And
packages that build differently on different platforms, like openblas,
we should opt out on.

I think this would be a cool innovation (in more ways than one).

Pj.


--001a113eb8d49da3e7056913bad3--