From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id UE9+CuAej2C3HwAAgWs5BA (envelope-from ) for ; Sun, 02 May 2021 23:51:28 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id EQ4eBuAej2BaBwAA1q6Kng (envelope-from ) for ; Sun, 02 May 2021 21:51:28 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id C1B99B688 for ; Sun, 2 May 2021 23:51:27 +0200 (CEST) Received: from localhost ([::1]:48914 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ldJzq-00055T-T3 for larch@yhetil.org; Sun, 02 May 2021 17:51:26 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:54672) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ldJzd-000526-GJ for guix-devel@gnu.org; Sun, 02 May 2021 17:51:14 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:40469) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ldJzc-00075B-UK; Sun, 02 May 2021 17:51:12 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=47670 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1ldJzc-0004rN-F3; Sun, 02 May 2021 17:51:12 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Christopher Baines Subject: Re: Bringing substitutes from the Guix Build Coordinator to users References: <878s4ye116.fsf@cbaines.net> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 13 =?utf-8?Q?Flor=C3=A9al?= an 229 de la =?utf-8?Q?R?= =?utf-8?Q?=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Sun, 02 May 2021 23:51:10 +0200 In-Reply-To: <878s4ye116.fsf@cbaines.net> (Christopher Baines's message of "Sat, 01 May 2021 19:56:05 +0100") Message-ID: <87eeeo24a9.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: guix-devel@gnu.org Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1619992287; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=DqDwKoaKjFgiudUUAG1IV51uj85oJQZg1FQtknWJksE=; b=ZC3l/+2EhUJ2AyL1Z+nQImZXoXgb98CoZYdYIxO14AAay91aP5gcZNeb/+u0ykegSKiEEA p7LonULbmIrL1fbuDoQnkej0PjRB4tNBmrujyVapp/GCWE0x7aiuMEy0UyZARvhXNVDH22 aRrbikbfn4ADywLVOFDBiTVLT3R7oGS7aSkLjy3qVFfUBUECwSd3s8qUEQdovzEq6Mt1Mp d2W8ADeTMgHq4/kLjAB5z64A34mfEH5ZpRTsLH0fd9sQZndzpBxVdEHJgDsaY5l4k2umC7 A4hGhovv8DECYtEAYp2Pq6nO6EAPW0gUWJKMT/ebK3cUYGXCj1VeccG0ojf18A== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1619992287; a=rsa-sha256; cv=none; b=gOYe1nz1MQ++BukGFAKTYp3GNinjfYjVTWuR/Pai4PDbY/VWd6GNcg2vFvYGzntHH4f2lU Ch5RsdnaddsoRCVqcqEUiScEIywSDCWS/CKQY8sXgpK+xNkMPyszGYmAjWgwtiX1VkbvHB 8bBss3HrDxSExfEAAth//W7ckYC8Yw6CBlqTwo/brkzSSqNEmte2jgq1MdlukFgbV2Zo0/ mGZLiQxOdV0yO2yd1m/9QLMbOXU3uOyOu0ngDviYkh1cU2xi7jl7LD7O+uBV7fWWif4Ztp J3hcsoMK/5X4lVUb63UvrpqXltwTdle2SDaeEUVaSPdTK0bERBMtTVJcG8WrPA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Spam-Score: -2.96 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Queue-Id: C1B99B688 X-Spam-Score: -2.96 X-Migadu-Scanner: scn0.migadu.com X-TUID: 9Bz4K1JVXs29 Hi Christopher, Thanks for your message! It=E2=80=99s great to see a summary of what=E2=80= =99s been cooking in the Coordinator and your vision around it. Christopher Baines skribis: > More specifically, while the architecture is similar to daemon > offloading, there are some practical advantages. Since the switch to > Cuirass remote workers for ci.guix.gnu.org, the advantages of building > things across multiple machines in a coordinated manor have also become > clear. Additionally, there are things that the Guix Build Coordinator > enables, like automated retries, with the option to target specific > agents, which can help with avoiding issues with non-reproducible > failures, as well as helping to avoid issues when mixing QEMU emulated > and native builds. These are nice examples of current QA blind spots that would be nice to address. [...] > This is a topic I haven't got directly involved in, until now. I'm just > a volunteer, in some ways the most involved I am is that I host an idle > ARM build machine, I don't have any particular connection in the default > approach to substitutes for users, or the hardware currently > involved. It would be great to have that OverDrive plugged in behind ci.guix because we=E2=80=99re still short on CPU power for ARM. (Not great we had forgotten about this one!) It could be reconfigured with a config similar to , with a different host name and IP. We can discuss the details in a separate thread and/or on guix-sysadmin, let me know! > However, I think some of the stuff I've been working on could be > helpful, as I say, I think the Guix Build Coordinator is a step > forward in terms of building things for substitutes, and that shows in > the substitute availability percentages. > > Is there still a path to bring some of these benefits to users, and if > so, what things need doing? My understanding is that, to build substitutes, we need more than the Coordinator; we also need something to perform evaluations, similar to the =E2=80=9Ctop half=E2=80=9D of Cuirass and the Data Service, right? How= would you envision setting it up? To me, the way forward is not necessarily substitutes; it could be QA, as you showed that it has helpful features. You already have access to bayfront and it=E2=80=99d be great to experiment there. Do you have ideas = on how to do make progress on that front? Besides, I feel that the Data Service has a huge role to play with a variety of applications, including QA, but also way beyond as you showed in the blog post=C2=B9. I think it=E2=80=99s time take advantage of it. := -) Specifically, as far as QA is concerned, I think we should place the Data Service at the center of the stage. It already does things that Cuirass cannot and will not do, notably: linting, challenging substitute servers to estimate reproducibility, and recording the history of all this. Couldn=E2=80=99t it gather these other QA metrics mentioned above, possibly from a Coordinator-powered bayfront? What I=E2=80=99d love to see is a set of =E2=80=9Cskins=E2=80=9D for the Da= ta Service: purpose-specific interfaces to the service. There=E2=80=99s already the reproducibility one, the one for package versions, etc. It would be great to have a qa.guix.gnu.org entry point with a simplified interface that would only show (say) reproducibility data, lint warnings, and so forth. (Then we can also think of other skins, like Guix Weekly News, a page dedicated to browsing package version history, a security-related page, etc. But now I feel like a spoiled kid asking for yet more presents. :-)) How does that sound? Thanks! Ludo=E2=80=99. =C2=B9 https://guix.gnu.org/en/blog/2020/introduction-to-the-guix-data-serv= ice-the-missing-blog-post/