From mboxrd@z Thu Jan 1 00:00:00 1970 From: Ricardo Wurmus Subject: Re: Treating tests as special case Date: Thu, 05 Apr 2018 16:10:04 +0200 Message-ID: <87bmexkamb.fsf@elephly.net> References: <20180405052439.GA30291@thebird.nl> <20180405082115.60e604a6@alma-ubu> <87fu4aj6xu.fsf@elephly.net> <20180405141947.6dfeb927@alma-ubu> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:49153) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1f4RrW-00059w-RJ for guix-devel@gnu.org; Fri, 06 Apr 2018 09:57:08 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1f4RrT-00028O-Ot for guix-devel@gnu.org; Fri, 06 Apr 2018 09:57:06 -0400 Received: from sender-of-o51.zoho.com ([135.84.80.216]:21093) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1f4RrT-00028D-Ek for guix-devel@gnu.org; Fri, 06 Apr 2018 09:57:03 -0400 In-reply-to: <20180405141947.6dfeb927@alma-ubu> List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: =?utf-8?Q?Bj=C3=B6rn_H=C3=B6fling?= Cc: guix-devel@gnu.org Hi Bj=C3=B6rn, > On Thu, 05 Apr 2018 12:14:53 +0200 > Ricardo Wurmus wrote: > >> Bj=C3=B6rn H=C3=B6fling writes: >> >> > And you mentioned different environment conditions like machine and >> > kernel. We still have "only" 70-90% reproducibility. >> >> Where does that number come from? In my tests for a non-trivial set >> of bioinfo pipelines I got to 97.7% reproducibility (or 95.2% if you >> include very minor problems) for 355 direct inputs. >> >> I rebuilt on three different machines. > > I have no own numbers but checked Ludivic's blog post from October 2017: > > https://www.gnu.org/software/guix/blog/2017/reproducible-builds-a-status-= update/ > > "We=E2=80=99re somewhere between 78% and 91%=E2=80=94not as good as Debia= n yet, [..]". Ah, I see. Back then we didn=E2=80=99t have a fix for Python bytecode, which affects a large number of packages in Guix but not on Debian (who simply don=E2=80=99= t distribute bytecode AFAIU). > So if your numbers are valid for the whole repository, that is good > news and would mean we are now better than Debian [1], and that would > be worth a new blog post. The analysis was only done for the =E2=80=9Cpigx=E2=80=9D package and its direct/propagated inputs. I=E2=80=99d like to investigate the sources of non-determinism for remainin= g packages and fix them one by one. For some we already know what=E2=80=99s = wrong (e.g. for Haskell packages the random order of packages in the database seems to be responsible), but for others we haven=E2=80=99t made an effort = to look closely enough. I=E2=80=99d also take the Debian numbers with a spoonful of salt (and then = take probiotics in an effort to undo some of the damage, see[1]), because they aren=E2=80=99t actually rebuilding all Debian packages. [1]: https://insights.mdc-berlin.de/en/2017/11/gut-bacteria-sensitive-salt/ -- Ricardo GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC https://elephly.net