From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <guix-devel-bounces+larch=yhetil.org@gnu.org>
Received: from mp0 ([2001:41d0:2:4a6f::])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
	by ms0.migadu.com with LMTPS
	id 2GJTJtS7o2HFoAAAgWs5BA
	(envelope-from <guix-devel-bounces+larch=yhetil.org@gnu.org>)
	for <larch@yhetil.org>; Sun, 28 Nov 2021 18:26:44 +0100
Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
	by mp0 with LMTPS
	id iML8IdS7o2HVFwAA1q6Kng
	(envelope-from <guix-devel-bounces+larch=yhetil.org@gnu.org>)
	for <larch@yhetil.org>; Sun, 28 Nov 2021 17:26:44 +0000
Received: from lists.gnu.org (lists.gnu.org [209.51.188.17])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by aspmx1.migadu.com (Postfix) with ESMTPS id 45121E4EC
	for <larch@yhetil.org>; Sun, 28 Nov 2021 18:26:44 +0100 (CET)
Received: from localhost ([::1]:42080 helo=lists1p.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.90_1)
	(envelope-from <guix-devel-bounces+larch=yhetil.org@gnu.org>)
	id 1mrNwp-0006hr-Dq
	for larch@yhetil.org; Sun, 28 Nov 2021 12:26:43 -0500
Received: from eggs.gnu.org ([209.51.188.92]:60892)
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@gnu.org>) id 1mrNwG-0006gt-L1
 for guix-devel@gnu.org; Sun, 28 Nov 2021 12:26:08 -0500
Received: from [2001:470:142:3::e] (port=39288 helo=fencepost.gnu.org)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@gnu.org>)
 id 1mrNwG-0005Ww-6A; Sun, 28 Nov 2021 12:26:08 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To:
 From; bh=p0KQI0fw0WOtjQTsHpJMcIgaFR1Qs910rTSKWlh57xo=; b=XFSv38Rr1DBl/AomNmS9
 h2tR+LA2LzaxOnzqAlagdlEm+Q1eNRe/cJQ5wJqf/M47odpF9F7etyrv2tVfjveHXuclaQsWCroNg
 EdUYys6DMaG7gnKx2elGaWjbDrqTMT8JjHjwj6GNPBQRz2kGw8baPEy37ceNJNgcH5xA/c/Q0W8IA
 Ybirj40Nrr1mW85O1mpoxV6tzKfTY5IdaMVgq/y+fwdjHe64woBGftpkfCw7PysS5PlYUwo8Q4YxF
 +4kkAf61BD+bIN+QF6qPEi8objk8RPdFtsu2yooMXvfJtPU/qG9n5SCFlcsrG35ZxJgJVRTeUY6yT
 icE5e0z5DRn0dA==;
Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201]:62858
 helo=ribbon)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@gnu.org>)
 id 1mrNwG-0005WI-0b; Sun, 28 Nov 2021 12:26:08 -0500
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@gnu.org>
To: Christopher Baines <mail@cbaines.net>
Subject: Re: Update on bordeaux.guix.gnu.org
References: <87ee762at1.fsf@cbaines.net>
X-URL: http://www.fdn.fr/~lcourtes/
X-Revolutionary-Date: 8 Frimaire an 230 de la =?utf-8?Q?R=C3=A9volution?=
X-PGP-Key-ID: 0x090B11993D9AEBB5
X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc
X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4  0CFB 090B 1199 3D9A EBB5
X-OS: x86_64-pc-linux-gnu
Date: Sun, 28 Nov 2021 18:26:05 +0100
In-Reply-To: <87ee762at1.fsf@cbaines.net> (Christopher Baines's message of
 "Wed, 24 Nov 2021 08:52:27 +0000")
Message-ID: <8735ngb39u.fsf@gnu.org>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-BeenThere: guix-devel@gnu.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: "Development of GNU Guix and the GNU System distribution."
 <guix-devel.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/guix-devel>,
 <mailto:guix-devel-request@gnu.org?subject=unsubscribe>
List-Archive: <https://lists.gnu.org/archive/html/guix-devel>
List-Post: <mailto:guix-devel@gnu.org>
List-Help: <mailto:guix-devel-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/guix-devel>,
 <mailto:guix-devel-request@gnu.org?subject=subscribe>
Cc: guix-devel@gnu.org
Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org
Sender: "Guix-devel" <guix-devel-bounces+larch=yhetil.org@gnu.org>
X-Migadu-Flow: FLOW_IN
X-Migadu-Country: US
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org;
	s=key1; t=1638120404;
	h=from:from:sender:sender:reply-to:subject:subject:date:date:
	 message-id:message-id:to:to:cc:cc:mime-version:mime-version:
	 content-type:content-type:
	 content-transfer-encoding:content-transfer-encoding:
	 in-reply-to:in-reply-to:references:references:list-id:list-help:
	 list-unsubscribe:list-subscribe:list-post:dkim-signature;
	bh=p0KQI0fw0WOtjQTsHpJMcIgaFR1Qs910rTSKWlh57xo=;
	b=J6TgnoWfzmQM4MsXLvuGTTYCC+o2cPYDmdiksRzgvhD/ZxjhBMFYEfIYl0hIB6rzItroN3
	bfHzkRNZ1zKZUrgOf/xs6Yhiy36HFj5XEv8ITNPfti3Xkuyx3EIm7FwWzkXtoq9wim/g5Q
	NRylWranw0wRwvZWWypYnE7/aauz+eLArFfjDNmjFGi9lxWpLj4E9ffuFabYzDJkOfmsul
	X6NrSc2HpF7lcLkkgJcK1czH0LKHlcctBRwMhHe+iPGyKsOhe20hVwyuufGBAe7B4r6AFi
	Xs5h/ebm44fiTN9yOqNQ5X/vTdXIHwK87InR0qyCJdPun/qdgl5WtMi+dAPW6Q==
ARC-Seal: i=1; s=key1; d=yhetil.org; t=1638120404; a=rsa-sha256; cv=none;
	b=DChX3rp3pdszh87jQsEDOOojvQIdI84X2fef2xW+UhJjXwJw26/tZXJ3Z6+9HiMq2C4I6/
	m2sfstowPTyBv4jyXTn9M9AoK2mjk54fdWNwt6T9kNm96FeR5DNVnN+AIVpJARYVCYwYh+
	XYWSQkHqbN53ODZK7/PhmXk3lDpD3ls9ZB9W4dgwM7FxW1b6rUUvzM+Tqnrv37URXsyxG6
	9bUJGUJcSeUnxII9F9Cd5h2Bw3tNFYetEiJApoVIAoRBencOynSq/YdN3pkaqKC/9omjfs
	NuRbKDXkhDi594OSY6avB0YRz2/ZKvRJ3mkMUcznEncri8esOhq6V1cihs1ApA==
ARC-Authentication-Results: i=1;
	aspmx1.migadu.com;
	dkim=pass header.d=gnu.org header.s=fencepost-gnu-org header.b=XFSv38Rr;
	dmarc=pass (policy=none) header.from=gnu.org;
	spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"
X-Migadu-Spam-Score: -5.31
Authentication-Results: aspmx1.migadu.com;
	dkim=pass header.d=gnu.org header.s=fencepost-gnu-org header.b=XFSv38Rr;
	dmarc=pass (policy=none) header.from=gnu.org;
	spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"
X-Migadu-Queue-Id: 45121E4EC
X-Spam-Score: -5.31
X-Migadu-Scanner: scn0.migadu.com
X-TUID: 1+hr2IuZOZoZ

Hello,

Christopher Baines <mail@cbaines.net> skribis:

> I've been doing some performance tuning, submitting builds is now more
> parallelised, a source of slowness when fetching builds has been
> addressed, and one of the long queries involved in allocating builds has
> been removed, which also improved handling of the WAL (Sqlite write
> ahead log).
>
> There's also a few new features. Agents can be deactivated which means
> they won't get any builds allocated. The coordinator now checks the
> hashes of outputs which are submitted, a safeguard which I added because
> the coordinator now also supports resuming the uploads of outputs. This
> is particularly important when trying to upload large (> 1GiB) outputs
> over slow connections.
>
> I also added a new x86_64 build machine. It's a 4 core Intel NUC that I
> had sitting around, but I cleaned it up and got it building things. This
> was particularly useful as I was able to use it to retry building
> guile@3.0.7, which is extremely hard to build [2]. This was blocking
> building the channel instance derivations for x86_64-linux.
>
> 2: https://data.guix.gnu.org/gnu/store/7k6s13bzbz5fd72ha1gx9rf6rrywhxzz-g=
uile-3.0.7.drv

Neat!  (Though I wouldn=E2=80=99t say building Guile is =E2=80=9Cextremely =
hard=E2=80=9D,
especially on x86_64.  :-))  The ability to keep retrying is much
welcome.

> On the related subject of data.guix.gnu.org (which is the source of
> derivations for bordeaux.guix.gnu.org, as well as a recipient of build
> information), there have been a couple of changes. There was some web
> crawler activity that was slowing data.guix.gnu.org down significantly,
> NGinx now has some rate limiting configuration to prevent crawlers
> abusing the service. The other change is that substitutes for the latest
> processed revision of master will be queried on a regular basis, so this
> page [3] should be roughly up to date, including for ci.guix.gnu.org.
>
> 3: https://data.guix.gnu.org/repository/1/branch/master/latest-processed-=
revision/package-substitute-availability

That=E2=80=99s good news.  That also means that things like
<https://data.guix.gnu.org/repository/1/branch/master/latest-processed-revi=
sion/package-reproducibility>
should be more up-to-date, which is really cool!  This can have a
drastic impact in how we monitor and address reproducibility issues.

> Now for some not so good things:
>
> Submitting builds wasn't working quite right for around a month, one of
> the changes I made to speed things up led to some builds being
> missed. This is now fixed, and all the missed builds have been
> submitted, but this was more than 50,000 builds. This, along with all
> the channel instance derivation builds that can now proceed mean that
> there's a very large backlog of x86 and ARM builds which will probably
> take at least another week to clear. While this backlog exists,
> substitute availability for x86_64-linux will be lower than usual.

At least it=E2=80=99s nice to have a clear picture of which builds are miss=
ing,
how much of a backlog we have, and what needs to be rebuilt.

> Space is running out on bayfront, the machine that runs the coordinator,
> stores all the nars and build logs, and serves the substitutes. I knew
> this was probably going to be an issue, bayfront didn't have much space
> to begin with, but I had hoped I'd be further forward in developing some
> way to allow moving the nars around between multiple machines, to remove
> the need to store all of them on bayfront. I have got a plan, there's
> some ideas I mentioned back in February [4], but I haven't got around to
> implementing anything yet. The disk space usage trend is pretty much
> linear, so if things continue without any change, I think it will be
> necessary to pause the agents within a month, to avoid filling up
> bayfront entirely.

Ah, bummer.  I hope we can find a solution one way or another.
Certainly we could replicate nars on another machine with more disk,
possibly buying the necessary hardware with the project funds.

Thanks for the update!

Ludo=E2=80=99.