From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id 2GJTJtS7o2HFoAAAgWs5BA (envelope-from ) for ; Sun, 28 Nov 2021 18:26:44 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id iML8IdS7o2HVFwAA1q6Kng (envelope-from ) for ; Sun, 28 Nov 2021 17:26:44 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 45121E4EC for ; Sun, 28 Nov 2021 18:26:44 +0100 (CET) Received: from localhost ([::1]:42080 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mrNwp-0006hr-Dq for larch@yhetil.org; Sun, 28 Nov 2021 12:26:43 -0500 Received: from eggs.gnu.org ([209.51.188.92]:60892) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mrNwG-0006gt-L1 for guix-devel@gnu.org; Sun, 28 Nov 2021 12:26:08 -0500 Received: from [2001:470:142:3::e] (port=39288 helo=fencepost.gnu.org) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mrNwG-0005Ww-6A; Sun, 28 Nov 2021 12:26:08 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To: From; bh=p0KQI0fw0WOtjQTsHpJMcIgaFR1Qs910rTSKWlh57xo=; b=XFSv38Rr1DBl/AomNmS9 h2tR+LA2LzaxOnzqAlagdlEm+Q1eNRe/cJQ5wJqf/M47odpF9F7etyrv2tVfjveHXuclaQsWCroNg EdUYys6DMaG7gnKx2elGaWjbDrqTMT8JjHjwj6GNPBQRz2kGw8baPEy37ceNJNgcH5xA/c/Q0W8IA Ybirj40Nrr1mW85O1mpoxV6tzKfTY5IdaMVgq/y+fwdjHe64woBGftpkfCw7PysS5PlYUwo8Q4YxF +4kkAf61BD+bIN+QF6qPEi8objk8RPdFtsu2yooMXvfJtPU/qG9n5SCFlcsrG35ZxJgJVRTeUY6yT icE5e0z5DRn0dA==; Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201]:62858 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mrNwG-0005WI-0b; Sun, 28 Nov 2021 12:26:08 -0500 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Christopher Baines Subject: Re: Update on bordeaux.guix.gnu.org References: <87ee762at1.fsf@cbaines.net> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 8 Frimaire an 230 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Sun, 28 Nov 2021 18:26:05 +0100 In-Reply-To: <87ee762at1.fsf@cbaines.net> (Christopher Baines's message of "Wed, 24 Nov 2021 08:52:27 +0000") Message-ID: <8735ngb39u.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: guix-devel@gnu.org Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1638120404; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=p0KQI0fw0WOtjQTsHpJMcIgaFR1Qs910rTSKWlh57xo=; b=J6TgnoWfzmQM4MsXLvuGTTYCC+o2cPYDmdiksRzgvhD/ZxjhBMFYEfIYl0hIB6rzItroN3 bfHzkRNZ1zKZUrgOf/xs6Yhiy36HFj5XEv8ITNPfti3Xkuyx3EIm7FwWzkXtoq9wim/g5Q NRylWranw0wRwvZWWypYnE7/aauz+eLArFfjDNmjFGi9lxWpLj4E9ffuFabYzDJkOfmsul X6NrSc2HpF7lcLkkgJcK1czH0LKHlcctBRwMhHe+iPGyKsOhe20hVwyuufGBAe7B4r6AFi Xs5h/ebm44fiTN9yOqNQ5X/vTdXIHwK87InR0qyCJdPun/qdgl5WtMi+dAPW6Q== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1638120404; a=rsa-sha256; cv=none; b=DChX3rp3pdszh87jQsEDOOojvQIdI84X2fef2xW+UhJjXwJw26/tZXJ3Z6+9HiMq2C4I6/ m2sfstowPTyBv4jyXTn9M9AoK2mjk54fdWNwt6T9kNm96FeR5DNVnN+AIVpJARYVCYwYh+ XYWSQkHqbN53ODZK7/PhmXk3lDpD3ls9ZB9W4dgwM7FxW1b6rUUvzM+Tqnrv37URXsyxG6 9bUJGUJcSeUnxII9F9Cd5h2Bw3tNFYetEiJApoVIAoRBencOynSq/YdN3pkaqKC/9omjfs NuRbKDXkhDi594OSY6avB0YRz2/ZKvRJ3mkMUcznEncri8esOhq6V1cihs1ApA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gnu.org header.s=fencepost-gnu-org header.b=XFSv38Rr; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -5.31 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gnu.org header.s=fencepost-gnu-org header.b=XFSv38Rr; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 45121E4EC X-Spam-Score: -5.31 X-Migadu-Scanner: scn0.migadu.com X-TUID: 1+hr2IuZOZoZ Hello, Christopher Baines skribis: > I've been doing some performance tuning, submitting builds is now more > parallelised, a source of slowness when fetching builds has been > addressed, and one of the long queries involved in allocating builds has > been removed, which also improved handling of the WAL (Sqlite write > ahead log). > > There's also a few new features. Agents can be deactivated which means > they won't get any builds allocated. The coordinator now checks the > hashes of outputs which are submitted, a safeguard which I added because > the coordinator now also supports resuming the uploads of outputs. This > is particularly important when trying to upload large (> 1GiB) outputs > over slow connections. > > I also added a new x86_64 build machine. It's a 4 core Intel NUC that I > had sitting around, but I cleaned it up and got it building things. This > was particularly useful as I was able to use it to retry building > guile@3.0.7, which is extremely hard to build [2]. This was blocking > building the channel instance derivations for x86_64-linux. > > 2: https://data.guix.gnu.org/gnu/store/7k6s13bzbz5fd72ha1gx9rf6rrywhxzz-g= uile-3.0.7.drv Neat! (Though I wouldn=E2=80=99t say building Guile is =E2=80=9Cextremely = hard=E2=80=9D, especially on x86_64. :-)) The ability to keep retrying is much welcome. > On the related subject of data.guix.gnu.org (which is the source of > derivations for bordeaux.guix.gnu.org, as well as a recipient of build > information), there have been a couple of changes. There was some web > crawler activity that was slowing data.guix.gnu.org down significantly, > NGinx now has some rate limiting configuration to prevent crawlers > abusing the service. The other change is that substitutes for the latest > processed revision of master will be queried on a regular basis, so this > page [3] should be roughly up to date, including for ci.guix.gnu.org. > > 3: https://data.guix.gnu.org/repository/1/branch/master/latest-processed-= revision/package-substitute-availability That=E2=80=99s good news. That also means that things like should be more up-to-date, which is really cool! This can have a drastic impact in how we monitor and address reproducibility issues. > Now for some not so good things: > > Submitting builds wasn't working quite right for around a month, one of > the changes I made to speed things up led to some builds being > missed. This is now fixed, and all the missed builds have been > submitted, but this was more than 50,000 builds. This, along with all > the channel instance derivation builds that can now proceed mean that > there's a very large backlog of x86 and ARM builds which will probably > take at least another week to clear. While this backlog exists, > substitute availability for x86_64-linux will be lower than usual. At least it=E2=80=99s nice to have a clear picture of which builds are miss= ing, how much of a backlog we have, and what needs to be rebuilt. > Space is running out on bayfront, the machine that runs the coordinator, > stores all the nars and build logs, and serves the substitutes. I knew > this was probably going to be an issue, bayfront didn't have much space > to begin with, but I had hoped I'd be further forward in developing some > way to allow moving the nars around between multiple machines, to remove > the need to store all of them on bayfront. I have got a plan, there's > some ideas I mentioned back in February [4], but I haven't got around to > implementing anything yet. The disk space usage trend is pretty much > linear, so if things continue without any change, I think it will be > necessary to pause the agents within a month, to avoid filling up > bayfront entirely. Ah, bummer. I hope we can find a solution one way or another. Certainly we could replicate nars on another machine with more disk, possibly buying the necessary hardware with the project funds. Thanks for the update! Ludo=E2=80=99.