From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:c151::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id cEZxGayAM2BKTgAA0tVLHw (envelope-from ) for ; Mon, 22 Feb 2021 10:00:12 +0000 Received: from aspmx2.migadu.com ([2001:41d0:2:c151::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id wHMlFayAM2AXOgAAbx9fmQ (envelope-from ) for ; Mon, 22 Feb 2021 10:00:12 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx2.migadu.com (Postfix) with ESMTPS id 6C1AF8634 for ; Mon, 22 Feb 2021 11:00:11 +0100 (CET) Received: from localhost ([::1]:53684 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lE80g-0001sQ-AB for larch@yhetil.org; Mon, 22 Feb 2021 05:00:10 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:44346) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lE80Y-0001sE-HF for guix-patches@gnu.org; Mon, 22 Feb 2021 05:00:02 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:44128) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lE80Y-0006Nq-8U for guix-patches@gnu.org; Mon, 22 Feb 2021 05:00:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lE80Y-0006N2-5H for guix-patches@gnu.org; Mon, 22 Feb 2021 05:00:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#46585] ci: Remove hydra support. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 22 Feb 2021 10:00:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 46585 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: To: Mathieu Othacehe Cc: 46585@debbugs.gnu.org Received: via spool by 46585-submit@debbugs.gnu.org id=B46585.161398797524426 (code B ref 46585); Mon, 22 Feb 2021 10:00:02 +0000 Received: (at 46585) by debbugs.gnu.org; 22 Feb 2021 09:59:35 +0000 Received: from localhost ([127.0.0.1]:55674 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lE7zt-0006Ld-67 for submit@debbugs.gnu.org; Mon, 22 Feb 2021 04:59:35 -0500 Received: from eggs.gnu.org ([209.51.188.92]:54410) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lE7zr-0006LQ-GJ for 46585@debbugs.gnu.org; Mon, 22 Feb 2021 04:59:19 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:51491) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lE7zm-00062K-88 for 46585@debbugs.gnu.org; Mon, 22 Feb 2021 04:59:14 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=42338 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1lE7zk-0005L1-HE; Mon, 22 Feb 2021 04:59:13 -0500 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <87h7mbgk6e.fsf@gnu.org> Date: Mon, 22 Feb 2021 10:59:11 +0100 In-Reply-To: <87h7mbgk6e.fsf@gnu.org> (Mathieu Othacehe's message of "Wed, 17 Feb 2021 09:42:01 +0100") Message-ID: <87pn0sctjk.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN X-Migadu-Spam-Score: -2.87 Authentication-Results: aspmx2.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx2.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: 6C1AF8634 X-Spam-Score: -2.87 X-Migadu-Scanner: scn1.migadu.com X-TUID: 194J9IOZZaSi Hi, Mathieu Othacehe skribis: > This removes hydra support to use Cuirass as the only continuous > integration system. Yay! That part could have been a separate commit, for clarity, but that=E2=80=99s okay. > It also simplifies the evaluation process. Here's how it's working > now: > > * Cuirass fetches new commits and calls its "evaluate" process. > > * The "evaluate" process calls the "cuirass-jobs" procedure in the > newly checkouted Guix "build-aux/cuirass/gnu-system.scm" file. > > * The "hydra-jobs" procedure in "build-aux/hydra/gnu-system.scm" file > starts the evaluation of "hydra-jobs" of (gnu ci) module in an > inferior using the new commit. > > * This procedure returns the list of all the package derivations at that > very commit under Hydra job format. > > * This list is converted to the Cuirass job format and written on the > stdout port. > > * The main Cuirass process reads the "evaluate" output using a pipe, and > registers the derivation that needs to be built in database. > > This is quite complex and it requires to pass around a huge list of > jobs, consuming a lot of memory. > > Here's the simplified method: > > * The first two steps are identical. > > * The "cuirass-jobs" procedure starts the evaluation of "cuirass-jobs" > of (gnu ci) module in an inferior using the new commit. This procedure > is passed a registration callback that directly registers the given > jobs in database. It doesn't return anything. > > As the "register" procedure is a part of Cuirass, the inversion on > control caused by the inferior is problematic. I had to proxy the > registration requests from the inferior to the evaluation process using > a named pipe. OK. Another option, to avoid the callback, would have been to write jobs to stdout as before, but to write them one by one instead of building the entire list in memory. Perhaps that would be slightly simpler than the callback + named pipe? > Other than that, the process seems now less obfuscated. Yeah, thumbs up. Do you know how this affects memory consumption? Thanks! Ludo=E2=80=99.