From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id WJF6GfX9nWHyegEAgWs5BA (envelope-from ) for ; Wed, 24 Nov 2021 09:55:17 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id mPIKFfX9nWEUfgAAbx9fmQ (envelope-from ) for ; Wed, 24 Nov 2021 08:55:17 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id C912437984 for ; Wed, 24 Nov 2021 09:55:16 +0100 (CET) Received: from localhost ([::1]:42524 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mpo3f-0000qi-Gj for larch@yhetil.org; Wed, 24 Nov 2021 03:55:15 -0500 Received: from eggs.gnu.org ([209.51.188.92]:42524) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mpo3D-0000qa-4O for guix-devel@gnu.org; Wed, 24 Nov 2021 03:54:47 -0500 Received: from mira.cbaines.net ([212.71.252.8]:56370) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mpo3A-0001AC-WF for guix-devel@gnu.org; Wed, 24 Nov 2021 03:54:46 -0500 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id CDF3627BBE9 for ; Wed, 24 Nov 2021 08:54:37 +0000 (GMT) Received: from capella (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id a892dfa8 for ; Wed, 24 Nov 2021 08:54:36 +0000 (UTC) User-agent: mu4e 1.6.6; emacs 27.2 From: Christopher Baines To: guix-devel@gnu.org Subject: Update on bordeaux.guix.gnu.org Date: Wed, 24 Nov 2021 08:52:27 +0000 Message-ID: <87ee762at1.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Received-SPF: pass client-ip=212.71.252.8; envelope-from=mail@cbaines.net; helo=mira.cbaines.net X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1637744116; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=Nsf5qFOEddORRA9QqTxE5Zivs4qeMVpp97ZKBmUostM=; b=son31tph21mx0fS4jLCMeYcDBCHLUWk7IKXJozPiv8ETUZllNAUnZ0Kii/U0nPV2N+5KUb DPewZyD4D/DzKLdThYSB+B8uIDMmjnH6bjlmYmBFmoG4Aw5NLwW/PMSfx7py4jeej4STMG 2pPIvn1SFXQqSvalPAzzVgaTuUDkFxoxtOzxpbvVfXStSfpffd2Rm7E/Hky5f63vjjEfV+ 1kDBUN/tFhjGzDmHp8gGwAF266YUc44ON7ZG8OQ/wHUulH6qZlGGW2c4Kvkwji9uhJzigg xRIKD263tgkImps1nKPn6ktfrWoviBB7eKFutXgX/Vs+l7napBTOs4NEFnbB1g== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1637744116; a=rsa-sha256; cv=none; b=NkB+7ooT8ISvb6dxOe3gNwGSWwZpm2vHtdd2Lyd9Xfzhj/vAvIfuIKMlyMzxhJ8F9pUNLm aADHbkecwbn2MNWdizMXVgWzOQ5izhCeswwxdccLzgegPg/1UXgy0/uJX+rmm74TpYdH6U ll2DgAzbfnIdDQKnGuE1Uuj/W+IcEUqcFAyahIhmUGrgGtWcVVpqlK4gBBO1yn10JP6EaN xMlQsoUOJ4Ph99JfeR6cg1WgXi+p2PErsxLowhaHubbKQulesBX6ImkIiosd3yOQQvkTOZ nAFn9xGtWX/HEDgy3nODVnxatXAyl7Sp/xaGOI3cTJly5TsOBZ9vOUJ2xATZlg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Spam-Score: -4.99 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Queue-Id: C912437984 X-Spam-Score: -4.99 X-Migadu-Scanner: scn1.migadu.com X-TUID: riGAsaScy4Zd --=-=-= Content-Type: text/plain Hey! It's been 3 months since I sent the last update [1]. This email was meant to go out on Friday, but it seems opensmtpd was broken on my machine, so it got stuck. 1: https://lists.gnu.org/archive/html/guix-devel/2021-08/msg00075.html First, some good things: I've been doing some performance tuning, submitting builds is now more parallelised, a source of slowness when fetching builds has been addressed, and one of the long queries involved in allocating builds has been removed, which also improved handling of the WAL (Sqlite write ahead log). There's also a few new features. Agents can be deactivated which means they won't get any builds allocated. The coordinator now checks the hashes of outputs which are submitted, a safeguard which I added because the coordinator now also supports resuming the uploads of outputs. This is particularly important when trying to upload large (> 1GiB) outputs over slow connections. I also added a new x86_64 build machine. It's a 4 core Intel NUC that I had sitting around, but I cleaned it up and got it building things. This was particularly useful as I was able to use it to retry building guile@3.0.7, which is extremely hard to build [2]. This was blocking building the channel instance derivations for x86_64-linux. 2: https://data.guix.gnu.org/gnu/store/7k6s13bzbz5fd72ha1gx9rf6rrywhxzz-guile-3.0.7.drv On the related subject of data.guix.gnu.org (which is the source of derivations for bordeaux.guix.gnu.org, as well as a recipient of build information), there have been a couple of changes. There was some web crawler activity that was slowing data.guix.gnu.org down significantly, NGinx now has some rate limiting configuration to prevent crawlers abusing the service. The other change is that substitutes for the latest processed revision of master will be queried on a regular basis, so this page [3] should be roughly up to date, including for ci.guix.gnu.org. 3: https://data.guix.gnu.org/repository/1/branch/master/latest-processed-revision/package-substitute-availability Now for some not so good things: Submitting builds wasn't working quite right for around a month, one of the changes I made to speed things up led to some builds being missed. This is now fixed, and all the missed builds have been submitted, but this was more than 50,000 builds. This, along with all the channel instance derivation builds that can now proceed mean that there's a very large backlog of x86 and ARM builds which will probably take at least another week to clear. While this backlog exists, substitute availability for x86_64-linux will be lower than usual. Space is running out on bayfront, the machine that runs the coordinator, stores all the nars and build logs, and serves the substitutes. I knew this was probably going to be an issue, bayfront didn't have much space to begin with, but I had hoped I'd be further forward in developing some way to allow moving the nars around between multiple machines, to remove the need to store all of them on bayfront. I have got a plan, there's some ideas I mentioned back in February [4], but I haven't got around to implementing anything yet. The disk space usage trend is pretty much linear, so if things continue without any change, I think it will be necessary to pause the agents within a month, to avoid filling up bayfront entirely. 4: https://lists.gnu.org/archive/html/guix-devel/2021-02/msg00104.html Thanks, Chris --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmGd/cpfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9XdO7w//cVw2VcQ2QXKT/rAa4uUkv5CCaDu4WtKS kDimmAagK2kv9DYn2jC8rQc59kOiMbK+RAFihlWDuDAWVNQ5fr8zMAHl5ZcfuSvQ h4aaYFmZSK7hnf9gsXotEhFoVCOfGE8xc2zcd9SoZytoo5JQzSooD5HJ/c8eCrF6 UKQkYHPtyCggb4M7DAu+YEnJhvgQDGI16X1MiM5NJYzDmTIgjNAHdPzEm+G5TX3f 0i9VyBkQKz4BLT0WOal0v8THhMeust3YKmFU3eqFPkanRhe5JEG97wsb02j9UIqB P6xQTZIINhn0LmIi6cMTjwTNSC3eEioMonzdVBcYmLd5ex+ZqUKbCnwScpyFU7RG t/ABu/l0lJkoT6udohr7ZI3qJueDZn64dKDifXCN0MMGWog2stGBJSxlzTI1o9Jt cDw3gu5bh7BpN/VZu4YBlg1gxb9jQZhU+ekS+3WobAkHRC+mn8L6mlSbMl8NsAAW 0xryO3J9Eb8Y3UOf3K0Vjhxj1ACoiPDR4luGVWiWR/UORNp7JNqiU/QKzD0M0Y70 lCeWkdOL3z6tSb0D8WB3UkMyyAsTLCJMCnMjpBdsAsNUZhLqPP0gWxgygMDa5fR3 2An0NrPeG/FJQBu0fYAK25f6ijo6AkNzJU/28eFBWeske8bzYkHVt5T7rXUm5dwv FAhy4qrL94E= =RGx3 -----END PGP SIGNATURE----- --=-=-=--