From mboxrd@z Thu Jan 1 00:00:00 1970 From: Mark H Weaver Subject: Re: Treating tests as special case Date: Thu, 05 Apr 2018 16:26:50 -0400 Message-ID: <87tvsp4cxh.fsf@netris.org> References: <20180405052439.GA30291@thebird.nl> Mime-Version: 1.0 Content-Type: text/plain Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:44502) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1f4BU7-0004q1-AK for guix-devel@gnu.org; Thu, 05 Apr 2018 16:27:52 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1f4BU2-00029w-Al for guix-devel@gnu.org; Thu, 05 Apr 2018 16:27:51 -0400 Received: from world.peace.net ([50.252.239.5]:45612) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1f4BU2-00029c-5P for guix-devel@gnu.org; Thu, 05 Apr 2018 16:27:46 -0400 In-Reply-To: <20180405052439.GA30291@thebird.nl> (Pjotr Prins's message of "Thu, 5 Apr 2018 07:24:39 +0200") List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: Pjotr Prins Cc: guix-devel@gnu.org Hi Pjotr, Pjotr Prins writes: > and he gave me a new insight which rang immediately true. He said: > what is the point of running tests everywhere? If two people test the > same thing, what is the added value of that? (I paraphrase) > > With Guix a reproducibly building package generates the same Hash on > all dependencies. Running the same tests every time on that makes no > sense. I appreciate your thoughts on this, but I respectfully disagree. > I know there are two 'inputs' I am not accounting for: (1) hardware > variants and (2) the Linux kernel. But, honestly, I do not think we > are in the business of testing those. We can assume these work. No, we can't. For example, I recently discovered that GNU Tar fails one of its tests on my GuixSD system based on Btrfs. It turned out to be a real bug in GNU Tar that could lead to data loss when creating an archive of recently written files, with --sparse enabled. I fixed it in commit 45413064c9db1712c845e5a1065aa81f66667abe on core-updates. I would not have discovered this bug if I had simply assumed that since GNU Tar passes its tests on ext4fs, it surely must also pass its tests on every other file system. > If not, any issues will be found in other ways (typically a segfault > ;). The GNU Tar bug on Btrfs would never produce a segfault. The only way the bug could be observed is by noticing that data was lost. I don't think that's a good way to discover a bug. I'd much rather discover the bug by a failing test suite. Tests on different hardware/kernel/kernel-config/file-system combinations are quite useful for those who care about reliability of their systems. I, for one, would like to keep running test suites on my own systems. Regards, Mark