From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:8:6d80::]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id kHDjB98YjGBRZgAAgWs5BA (envelope-from ) for ; Fri, 30 Apr 2021 16:49:03 +0200 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id qACQA98YjGBAWwAAB5/wlQ (envelope-from ) for ; Fri, 30 Apr 2021 14:49:03 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 787F0271D1 for ; Fri, 30 Apr 2021 16:49:02 +0200 (CEST) Received: from localhost ([::1]:49072 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lcURx-00080e-Je for larch@yhetil.org; Fri, 30 Apr 2021 10:49:01 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:35146) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lcURl-0007zi-Ei for guix-devel@gnu.org; Fri, 30 Apr 2021 10:48:49 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:48513) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lcURl-0008RN-6A; Fri, 30 Apr 2021 10:48:49 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=40346 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1lcURi-0000xY-6H; Fri, 30 Apr 2021 10:48:46 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Mathieu Othacehe Subject: Re: New blog post on the Guix Build Coordinator: Building derivations, how complicated can it be? References: <878s58gprg.fsf@cbaines.net> <8735vggi91.fsf@cbaines.net> <87czuir7m7.fsf@gnu.org> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 11 =?utf-8?Q?Flor=C3=A9al?= an 229 de la =?utf-8?Q?R?= =?utf-8?Q?=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Fri, 30 Apr 2021 16:48:44 +0200 In-Reply-To: <87czuir7m7.fsf@gnu.org> (Mathieu Othacehe's message of "Sun, 25 Apr 2021 12:17:20 +0200") Message-ID: <87czubde0j.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: guix-devel Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1619794142; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=RFbOKk52Ftsw+8B+CgiuuUpryfizmQ//S7cEdBESWPU=; b=IdEpe3c/ebZy9u7IHb7o1rY5XelH2f6UYDR5JgpNdEgzQEwdep6FDoAnnPNyGknpDJvmmE 1TWwXhVF6cK/HnktzfAPSI4EsYSiv7kxPFTKHD/JxU4t0/iVO+921uNqGaggYTS3i/S+VM L7rU11JQ+9zEV9W4bCpw9/gkjiLgKtImD2Qm+TgdRfLWZq0EaaEqFR7pFCVjMLKIflZKZZ DWJJucnWntiSrocRy8dZDHJoPFEXweTcenQZrjXdEN0AbZqAPviDun1+z+yU/FCmyksFv+ +XPj3S3QG0QQuMxXWl5O3kHqy4htHYmjPNUlPXvnJhcL0jmOB31V9AJCLdC1mw== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1619794142; a=rsa-sha256; cv=none; b=Sba4UqjezVkYYh5M6a2os/kd/vggDf/t8y/JxuPWSr0DUT31jgC49HE+2B2HuFcHwI59bT PXotVpBwAPO7KUbnGvTI/7d1rvZxSSDagbnYR+G+hnWg+ogiPL9htHHgoiicxNqaUXIkG2 0qg0bRCJdjHg+4SzaMk9ocDJDfi3aYXdYL2Q/pPGm2+qhxumNvOYwqHejcfsbBgCfX7Wf2 AY45nHvi75EWlVFyFszzAhOKUCehqhbIChVL4TRLkF4Qv7pNEJblp41K6n7DJk2IcmBzCD /U4UK1wNNqHUOAiROH9CXEUm8p4EvohXfRsjSRiG5JGGKlGjXpGEUThgIzNRPg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Spam-Score: -1.96 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Queue-Id: 787F0271D1 X-Spam-Score: -1.96 X-Migadu-Scanner: scn0.migadu.com X-TUID: NvFhen7hq1ms Hello! Mathieu Othacehe skribis: > We have already discussed the Cuirass vs Build coordinator situation in > the past. I haven't changed by mind on that subject. The Guix Build > Coordinator is more or less the equivalent of the Cuirass remote build > mechanism[2]. Yes. So to answer Vincent=E2=80=99s question with my own words and as an outsider: the Guix Build Coordinator focuses on distributing builds across several machines that may come and go dynamically. This is similar to cuirass-remote-worker. One difference I see is that cuirass-remote-worker uses ZMQ for communication while the Coordinator uses HTTP, which makes it easier to use in a more distributed and dynamic context (HTTP goes through all firewalls). Cuirass-remote-worker can discover the central server via Avahi discovery, which makes it easy to add new workers but is limited to LANs. On berlin, a WireGuard VPN was set up to address this (the non-x86 build nodes are far away from the rest of the build farm). Another one is that, because the Coordinator focuses on builds, it brings specific features, such as retrying failed builds. I like that the Coordinator is quite orthogonal; you can use it together with the Data Service, or you can use it as an alternative to the daemon=E2=80=99s offload mechanism. Conversely, Cuirass is more of an all-in-one solution. Depending on how you look at it, it can be a weakness, but it=E2=80=99s also a strength when= it comes to deploying a =E2=80=9Cbuild farm=E2=80=9D kind of service (I think = it=E2=80=99s a good fit for ci.guix). Its monitoring features and dashboards have become very nice and well adapted to someone willing to build packages from a bunch of channels. > Integrating the Build coordinator in the Berlin build farm would mean > hooking in to Cuirass as an alternative to the remote build mechanism. I > don't think it would be wise because: [...] > It really makes me sad that we have two pieces of software that are > stepping on each other toes. The Build coordinator has a nice concept > and represents a huge amount of work. However, integrating it to Berlin > is not an option for me. > > I think that the next challenges on the CI front are: > > * Increase the number of ARM machines in the build farm. > > * Work on substitutes mirrors. > > * Find a way to make Cuirass and the Data Service work together. > > and we should face those issues together, rather than having competing > software sharing the few build machines we own. Right, I think the fruitful way to frame it is not =E2=80=9Cwhich one is better=E2=80=9D, but rather how can we take the good things from each appro= ach. For example, I believe the Guix Data Service relied on the former Cuirass notification mechanism to learn about build status. That mechanism is gone, so =E2=80=9Cwe=E2=80=9D (i.e., you :-) Chris & Mathieu) = should make sure the Data Service can take advantage of the new notification mechanism, possibly adjusting it so there=E2=80=99s a good match. There are certainly other opportunities for cooperation in this area. The Data Service does things that Cuirass doesn=E2=80=99t do, such as linti= ng. It wouldn=E2=80=99t make sense IMO to extend Cuirass to do these things; in= stead we should find ways to aggregate and present all this information in the Data Service. It already does that to a large extent, but maybe we can think of tighter integration, such as providing QA-oriented pages instead of the more generic interface it currently provides. Thoughts? Ludo=E2=80=99.