From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id mCx/B4gK2F4lYwAA0tVLHw (envelope-from ) for ; Wed, 03 Jun 2020 20:39:36 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id ACl5A4gK2F6KUgAA1q6Kng (envelope-from ) for ; Wed, 03 Jun 2020 20:39:36 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 923969404CF for ; Wed, 3 Jun 2020 20:39:35 +0000 (UTC) Received: from localhost ([::1]:33276 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jgaAg-0003yF-KC for larch@yhetil.org; Wed, 03 Jun 2020 16:39:34 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:41672) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jgaA7-0003Sx-LA for guix-devel@gnu.org; Wed, 03 Jun 2020 16:38:59 -0400 Received: from mira.cbaines.net ([212.71.252.8]:59574) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jgaA6-0003RO-CK for guix-devel@gnu.org; Wed, 03 Jun 2020 16:38:59 -0400 Received: from localhost (unknown [46.237.173.126]) by mira.cbaines.net (Postfix) with ESMTPSA id 4CF2627BBE1 for ; Wed, 3 Jun 2020 21:38:56 +0100 (BST) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id c3754e02 for ; Wed, 3 Jun 2020 20:38:53 +0000 (UTC) User-agent: mu4e 1.2.0; emacs 26.3 From: Christopher Baines To: guix-devel@gnu.org Subject: Build reproducibility metrics Date: Wed, 03 Jun 2020 21:38:51 +0100 Message-ID: <87lfl351gk.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Received-SPF: pass client-ip=212.71.252.8; envelope-from=mail@cbaines.net; helo=mira.cbaines.net X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/03 16:38:56 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, UNPARSEABLE_RELAY=0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Scanner: scn0 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Spam-Score: -3.11 X-TUID: 1G1lozHqPLqj --=-=-= Content-Type: text/plain Hey, So there's the guix challenge command for looking at reproducibility issues, but I've been wanting to more continuously and automatically monitoring the reproducibility of packages. The Guix Data Service has had the ability to do this for a few months now, but there's not been enough data on substitutes. That's starting to change though. ci.guix.gnu.org has been doing a reasonable job of building packages, but the proportion of packages built by bayfront.guix.gnu.org hasn't always been very high. I'm hoping just having bayfront not build core-updates and staging will help with this. I've also been writing and trying to use the Guix Build Coordinator [1] to build Guix packages and provide substitutes. That has got to the point where it's not getting stuck every day at least, and there's more than 80% of packages are available. 1: https://git.cbaines.net/guix/build-coordinator/about/ Combining that with the substitute server operated by Tobias, which has a pretty awesome substitute availability of over 90% for recent revisions, not only is there data from 4 different substitute servers to use in the comparison, but the proportion of packages where there isn't sufficient data is pretty low, below 10%. I'm currently using the data.guix-patches.cbaines.net instance of the Guix Data Service, you can see the package substitute availability for the latest revision using this URL [1], and the package reproducibility at this URL [2]. 1: https://data.guix-patches.cbaines.net/repository/2/branch/master/latest-processed-revision/package-substitute-availability 2: https://data.guix-patches.cbaines.net/repository/2/branch/master/latest-processed-revision/package-reproducibility Some caution is needed when interpreting this data. It's most probably less up to date than what you'd get through running the guix weather or guix challenge commands, as it takes the Guix Data Service time to query the data, that querying process isn't very reliable at the moment either. Additionally, the "matching" percentage could easily go down if that output is built with a different hash in the future. While the number itself maybe isn't the most useful thing, I like that clicking through to the "Not matching" outputs will show a list of outputs which didn't build reproducibly, which is something that could help identify reproducibility issues to investigate and fix. I think things are coming together on the substitute server side. The goal I have in mind for this is for users of Guix to be able to have greater trust in the substitutes they use, through trusting substitutes only if it's been built reproducibly on multiple substitute servers. It would be great to see work start soon on how guix as a client to substitute servers might be enhanced to check for reproducibility when fetching substitutes. Thanks, Chris --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEEPonu50WOcg2XVOCyXiijOwuE9XcFAl7YCltfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcACgkQXiijOwuE 9XftTg/+MLj92d0oE22yOyfpPtdKZZAeQQn46GetCsABJkyFdGHHAWO4MH153+sd +vuFt3Flmxyzz2ejWhy6rMlxJgpLmXApkxRko820pczqiRgSHYj6h+s52FRF1beP cZKdfLMXE5XOmes8oZW73hUwdOSFOd1NtvVAkTOB+j3FvdeNhio1eMJQvMz+rrvs ohgIQaG4G7Q1/RnNMeWKkxv2Yswf3r4z1bpzBuekLAsD1IHpo1GZ8YLp8hzcq+cu WNO0+IhxJtDVLCQDs2egyOKEJyl4IMohSnfOsWho9uTTrkOor8pnvJ/WHfNjzhqU RzuB4I8IOp1xh1CFliiHypfTgVjTnDbGAGsLFE1B5UCblWqA5OwAD4H6CPHh9jUX 4e28/DzFfpEp0+VPPqxZZsCZQ7bakiUpCISftNVnkvJrNMxKtwHqwRgu0gcBEpyT YtZAqTwDloGl++BqdA5KSpK4DHgnBNqVzi393Qciu+2hmY6rJTiQBzV45oMxugSv FVqtf5MtVa+KHagBoUGD53DPg7n6axGBcnlg7cWGUBxe1UOaSz6W63g95SOz9141 txXKUsmhAgH1y0pZnOTCp7SyhwsngtFH1fhjmVOvJCw9X+TyftUiE0l876QxL9S4 hxm0ktey2qJIFTpAzr38yFWm5lPC9gQYNxVG94dUTTmdM6VoyhQ= =j8ZE -----END PGP SIGNATURE----- --=-=-=--