From mboxrd@z Thu Jan 1 00:00:00 1970 From: Andreas Enge Subject: Re: [GSoC] Rewrite Hydra to be more integrated with Guix. Date: Tue, 22 Mar 2016 22:31:24 +0100 Message-ID: <20160322213124.GB2135@solar> References: <87bn6c944a.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:33586) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aiTtm-0004sg-7F for guix-devel@gnu.org; Tue, 22 Mar 2016 17:31:35 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aiTti-0000st-1Y for guix-devel@gnu.org; Tue, 22 Mar 2016 17:31:34 -0400 Received: from mailrelay1.public.one.com ([91.198.169.124]:38232) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aiTth-0000rz-Mp for guix-devel@gnu.org; Tue, 22 Mar 2016 17:31:29 -0400 Content-Disposition: inline In-Reply-To: <87bn6c944a.fsf@gnu.org> List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org To: Mathieu Lirzin Cc: guix-devel Hello, over some lunch I have been chatting with a famous Debian developer to under- stand a bit how Debian handles their build process. What is interesting with their approach is that the master manages a queue of packages to be built, and the slaves connect to fetch work when they are ready (for instance, when they have empty resources). With our offload mechanism, the master contacts all the build slaves to schedule the work. This is not totally different, but I wonder whether the Debian approach is not more flexible and more geared towards a distributed approach. (Of course, in both cases the master machine needs to keep a list of authorised build machines, so the degree of centra- lisation is ultimately the same.) In the end, I think it would be interesting to not use the offload approach, where the offloading machine sends all inputs to the build machine, which returns the result, so that more or less the complete set of binary packages flows along the tree of build machines to the root and the other way round. (As I recently understood during discussions with Ludovic, currently we have a tree of height one: hydra is the root and all build machines are the leaves; but the offloading mechanism could transparently work with trees of bigger height, where hydra offloads to a root of a smaller build cluster, which itself offloads to one of its subordinate build machines. And so on. The same thing would also work in the Debian setting, assuming that build requests come from the leaves and are forwarded towards the root, while a work package is handed down the other way.) Now it would be interesting if machines just got build jobs (or made requests) and looked for the necessary inputs inside a big magma/cloud/gnunet/ipfs/mirror network, compiled the package, and dropped the result into the same magma. Already in our current set-up, it would be useful if hydra need not send the build inputs, but build machines could just fetch them from an arbitrary nginx mirror, where they are cached. Andreas