From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id YOWvHgGrjWA68gAAgWs5BA (envelope-from ) for ; Sat, 01 May 2021 21:24:49 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id 2NxdGgGrjWAUagAAB5/wlQ (envelope-from ) for ; Sat, 01 May 2021 19:24:49 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 2332E1775E for ; Sat, 1 May 2021 21:24:49 +0200 (CEST) Received: from localhost ([::1]:39136 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lcvEO-0005jI-Ae for larch@yhetil.org; Sat, 01 May 2021 15:24:48 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:47150) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lcuml-0003Fr-0G for guix-devel@gnu.org; Sat, 01 May 2021 14:56:15 -0400 Received: from mira.cbaines.net ([212.71.252.8]:55306) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lcumg-0006Ed-Fw for guix-devel@gnu.org; Sat, 01 May 2021 14:56:14 -0400 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id 0F12427BC7C for ; Sat, 1 May 2021 19:56:08 +0100 (BST) Received: from capella (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id cdbeffb6 for ; Sat, 1 May 2021 18:56:07 +0000 (UTC) User-agent: mu4e 1.4.15; emacs 27.1 From: Christopher Baines To: guix-devel@gnu.org Subject: Bringing substitutes from the Guix Build Coordinator to users Date: Sat, 01 May 2021 19:56:05 +0100 Message-ID: <878s4ye116.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Received-SPF: pass client-ip=212.71.252.8; envelope-from=mail@cbaines.net; helo=mira.cbaines.net X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1619897089; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=4tRLWq3m6jaFglH+DhUJ8PW+3QBlkF33OtUu83K5rKQ=; b=cUzGLrBwyE5Gk+CfP0jbDS4a6NG7BT3rQGMvJe6feOm/rN0UkMKISxURD9Gm2hAfRZX81b X3cVosxuIRVxG93jXsCIPOTEaP0KMQDWIfexJim3BItS3HQCMOvenZJX7zlf9Gi1fR3+ue J8lfFYDjrJ0mLGo4E60aMlw7PKDPedU6+nHtU7LOy7KkB4kIzL3PzMifVB+qE34pJYV+qL Qc7Usi6w0TqhfgjVtqH2NfNUSZZotV2AMQgB6/YfBqytkcZARotPiT37mR5IyPgTH07Wq0 BdruI61nY1qIYGs4Ldk0ricVH7CRxdqDKDD1WQlb+9zqfg+A16R/wPWJAq+VTw== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1619897089; a=rsa-sha256; cv=none; b=OGu7gUjrLIJFjNwCPSEFzfQ9miIGCsZj/BQfe+NU0FVeYcQDUjnUmEU35U9TVOSyD3eyIN AJ7wu0F2ChJ9wrLUb0lUAJkOOMEic+ECDU8E85FSqCsizidfMR57SCAHam8PltPzjYKwv/ QVfWxOGww6BnFV9Ihz/EGRFuH+mJLXLPLZaz2C7SvTUt94EtJBVq+uk1fgJt76/lUsxoQn 8TOzJEbtcVsVgdI/Dtrw4y0IABBxkgXz8+L5xFHur2icATnTbh4jGtA6I33TDRkSYenxOk PFBLL8l7c0ofeFIt3TG2fstTjXdlv/bMaJHyemHGiDwmeyaS8ctqYieNvohKdg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Spam-Score: -4.06 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Queue-Id: 2332E1775E X-Spam-Score: -4.06 X-Migadu-Scanner: scn0.migadu.com X-TUID: p9awd0Z8Z2BK --=-=-= Content-Type: text/plain Hey, The Guix Build Coordinator has been around for a while, it's been available in Guix for more than 6 months now. The setup I've been using to test the Guix Build Coordinator for building things for substitutes (guix.cbaines.net) has been running since mid 2020. Anecdotally, the test setup I've been running (guix.cbaines.net) has for a while now, generally had higher percentage substitute availability compared to ci.guix.gnu.org, as reported by guix weather. I think it might also be building more things since cross derivations are built plus i586-gnu builds. I think there are some benefits for using the Guix Build Coordinator to build things for substitutes, and it would be good to work out how to get those benefits to users of Guix generally. More specifically, while the architecture is similar to daemon offloading, there are some practical advantages. Since the switch to Cuirass remote workers for ci.guix.gnu.org, the advantages of building things across multiple machines in a coordinated manor have also become clear. Additionally, there are things that the Guix Build Coordinator enables, like automated retries, with the option to target specific agents, which can help with avoiding issues with non-reproducible failures, as well as helping to avoid issues when mixing QEMU emulated and native builds. Going back to discussions in 2020, I think there was a path for starting to experiment with the Guix Build Coordinator, trying it on a few machines in the berlin build farm, but as I say, this was last year and before any work on implementing similar functionality in Cuirass as far as I'm aware. Since then, I've also got an instance of the Guix Build Coordinator running on bayfront with a couple of extra machines also involved for performing builds, but this is just starting off, and doesn't bring any benefit in terms of substitutes, at least not yet. This is a small part of the wider puzzle, when I was designing the Guix Build Coordinator, it was meant for much more than building things for substitutes, and those things are also happening. The Guix Build Coordinator has enabled things like building packages affected by patches, building fixed output derivations regularly, and will hopefully enable a whole load of quality assurance things. But my hope was also to try and tackle building things for substitutes, such that there's not more software to maintain, and also to inform future work on the guix-daemon itself, like whether it would be good for it to have an asynchronous API, and how that might work. This is a topic I haven't got directly involved in, until now. I'm just a volunteer, in some ways the most involved I am is that I host an idle ARM build machine, I don't have any particular connection in the default approach to substitutes for users, or the hardware currently involved. However, I think some of the stuff I've been working on could be helpful, as I say, I think the Guix Build Coordinator is a step forward in terms of building things for substitutes, and that shows in the substitute availability percentages. Is there still a path to bring some of these benefits to users, and if so, what things need doing? Thanks, Chris --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmCNpEVfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9XdcWA//RFWdslZDercndLhdRd54bza3rSwljWwn NR4E1lJfZJrtNQNFVbLNmveRDserI2TO2Q1P8kcTUjZKtsYr7A7sPzni6yJbt1lx L9blg82tzTKqgpx+/25Tme/WrBTgEWUfcOSFHcfz6ttOS/Xxbf9wgXI2NwPHjC6N vK2LyNMKghb586Vlzomt9QJ9J4XJdJKhz+VyXQJDnPnPW4bii5VuJTzKjDbqIWNg ybvl78f/yTxa2LyL1p+Y/jk62zAi9LcLQA2SSb7DxUh9BK2OlDO71yqOSGw48mE6 uHZJf4EFJ6RF1BHcFzi7iGl//2w+VwZkd1pfACKhPtQhpPcANf5PTT+mJC60n8gm TU1IpAKJe3wQezzKJdWbBqBjUzOjl8DMBrZ5qS6j2C62UKzi/yaKOStNXDGSLjsX WWVz8HnL98YyAEgYd6X7/N6sIsNgtS1fa60mqtBsMpzmiIleITU7jrQnwMCo+C6q R50QOxJ/u7nFRvYmEo5wG9dH9djGkxpOI3DilqEz/fFBXrBCypVBPjRGlHTX2Qq3 i9nOywkyWrcQppyYtoCsbqMqaWJeyqrlFHJgCfmvO/gu6bD5lGI12BDwrjlJxF2R xfdk1v71LPMjkbRzGOkhXtND2xjSfhL+z0HaqtpGwK7san/3WVw3q7/mZstVTSYd Og2E38o5+Ew= =gHoB -----END PGP SIGNATURE----- --=-=-=--