From mboxrd@z Thu Jan 1 00:00:00 1970 From: Pjotr Prins Subject: Re: Treating tests as special case Date: Thu, 5 Apr 2018 10:39:15 +0200 Message-ID: <20180405083915.GA31585@thebird.nl> References: <20180405052439.GA30291@thebird.nl> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:53613) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1f40Uq-0005WU-J2 for guix-devel@gnu.org; Thu, 05 Apr 2018 04:43:53 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1f40Um-0001ET-Lb for guix-devel@gnu.org; Thu, 05 Apr 2018 04:43:52 -0400 Received: from mail.thebird.nl ([95.154.246.10]:60347) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1f40Um-0001E3-E2 for guix-devel@gnu.org; Thu, 05 Apr 2018 04:43:48 -0400 Content-Disposition: inline In-Reply-To: List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: =?iso-8859-1?Q?G=E1bor?= Boskovits Cc: Guix-devel On Thu, Apr 05, 2018 at 08:05:39AM +0200, G=E1bor Boskovits wrote: > Actually running tests test the behaviour of a software. Unfortunate= ly > reproducible build does not guarantee reproducible behaviour. > Furthermore there are still cases, where the environment is > not the same around these running software, like hardware or > kernel configuration settings leaking into the environment. > These can be spotted by running tests. Nondeterministic > failures can also be spotted more easily. There are a lot of > packages where pulling tests can be done, I guess, but probably not > for all of them. WDYT? Hi Gabor, If that were a real problem we should not be providing substitutes - same problem. With substitutes we also provide software with tests that have been run once (at least). We should not forbid people to run tests. But I don't think it should be the default once tests have been run in a configuation. Think of it as functional programming. In my opinion rerunning tests can be cached. My point is that we should not overestimate/overdo the idea of leakage. Save the planet. We have responsibility. Pj.