From mboxrd@z Thu Jan 1 00:00:00 1970 From: Christopher Baines Subject: Prototype tool for building derivations Date: Fri, 17 Apr 2020 21:22:18 +0100 Message-ID: <87h7xh6ex1.fsf@cbaines.net> Mime-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Return-path: Received: from eggs.gnu.org ([2001:470:142:3::10]:57713) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jPXVK-00076V-6f for guix-devel@gnu.org; Fri, 17 Apr 2020 16:22:28 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jPXVI-0002H1-Ra for guix-devel@gnu.org; Fri, 17 Apr 2020 16:22:25 -0400 Received: from mira.cbaines.net ([212.71.252.8]:56584) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1jPXVI-0002Fd-L1 for guix-devel@gnu.org; Fri, 17 Apr 2020 16:22:24 -0400 Received: from localhost (unknown [46.237.173.52]) by mira.cbaines.net (Postfix) with ESMTPSA id C5FFC27BBE1 for ; Fri, 17 Apr 2020 21:22:23 +0100 (BST) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 4e2be4b4 for ; Fri, 17 Apr 2020 20:22:21 +0000 (UTC) List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane-mx.org@gnu.org Sender: "Guix-devel" To: guix-devel@gnu.org --=-=-= Content-Type: text/plain Hey, Over the last couple of weeks, I've made some time to implement something I was thinking about for a while. In terms of getting to a point where Guix packages build reliably and reproducibly, I think more testing is what's going to help. By taking packages and building them more, on a wide variety of hardware and software configurations, we'll get data on what works, what doesn't, and where improvements and fixes can be made. It's very much a prototype, but I've pushed some code up here [1] now, the README.org file [2] contains usage instructions as well as a description of the architecture. 1: https://git.cbaines.net/guix/build-coordinator/ 2: https://git.cbaines.net/guix/build-coordinator/about/ So far, I've mostly done the boring stuff, but I'm excited about what this could support. Because the allocation/scheduling of builds is controlled, this offers the possibility of doing some builds before others. If you were using this for providing substitutes for example, it could be valuable to try and prioritise building things that are requested more often, or those that are more expensive (in time or space) to build. Often there are concurrency issues with builds, I want to add a way of specifying where builds should run. This would make it easy to test building the same derivation in different setups, then capture where it succeeds, fails, and how the output differs (if at all) across the different environments. I think it would be good to get point where there are many different individuals and groups providing independent sources of Guix packages, such that users can have a high level of confidence that the substitutes they're getting correspond to the source code. Getting there will be easier if substitute servers are easy to operate, and part of that I think comes down to how easy it is to see what's going on. With the current daemon implementation, I'm not sure how to get much data out (this could be possible, I haven't looked very closely). This approach however where the scheduling is done outside the daemon makes the information more accessible. I think some of the design decisions here are quite short sighted. I think it would be better if some of this functionality could be handled by the guix-daemon, especially things like providing information on builds that are going to happen, but haven't happened yet. Once there's a Guile implementation of the guix-daemon, hopefully some of this technical debt can be repaid. Just let me know if you have any questions or comments! Thanks, Chris --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEEPonu50WOcg2XVOCyXiijOwuE9XcFAl6aD/pfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcACgkQXiijOwuE 9XfZKA/9Gq6ZtnOG6YXs4plLq7ljWJn5Y9MDwfNlajU369x+OvutbNVczgYMAqt4 ke2Y62vnybZREXsoYyzqU3w5wRYKBT88LdtFsEgCNEX3K8NySa0ZkCQ/9hsfP6cM arOmmTXmYmamMEyAfgNVw1yRRBjaDGMp8wwbTuS+/Z6/aVjkKRrQU0Yo00gUyws2 QfNQXhRqljNERMzbzOEkV5TkVBTczAySMGIlQBjND6f9321FgvAuzOii56FX1mUi gT7yozZ+G8jg8b0LJED9+O4ptmX6sdwrP6DTV8dim0GgaQ3i7vQgj18GH8x0Bp3Q YDHduxpgLMb7Y1Ca22/zpaFKszKgyVSdBb36A0TV8BKH093Utmu2/ryBftO7osdw rPCgCfg7Nmde7ksLlEv+H4mC0qamksVBAwXYI03vhvoP3+cENiILGwJ2sIqEfoG1 ghzg8xp/jwdFqeVi9vFKvGlEz201sgeYhZi64MdmMMxoXZ8Kuxs1JuYSbjuwnQAy oUz+q12LGI7iBGb3jQXo48h52S108ET5aQsRODNUOz9iArXVSsX+4eLR1DFdeg7t waR0UnEHT+XeqvXWqB0l9BANFLpfFTdkqI6wrJyQQ+imuEe/2oVYBdh1/dqX6N43 w013rb07hYoxPYyvVJeokd6v5VbkU2JYTE+oiGC7mmnX852qjpg= =ILvr -----END PGP SIGNATURE----- --=-=-=--