From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp10.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id EAznODOfh2K5GQAAbAwnHQ (envelope-from ) for ; Fri, 20 May 2022 16:01:24 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp10.migadu.com with LMTPS id KIKvNzOfh2IxWwAAG6o9tA (envelope-from ) for ; Fri, 20 May 2022 16:01:23 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 778D32B493 for ; Fri, 20 May 2022 16:01:23 +0200 (CEST) Received: from localhost ([::1]:33738 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ns3By-0000iZ-9N for larch@yhetil.org; Fri, 20 May 2022 10:01:22 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:43768) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ns34S-00083q-LB for guix-devel@gnu.org; Fri, 20 May 2022 09:53:38 -0400 Received: from mira.cbaines.net ([212.71.252.8]:36342) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ns34P-000878-4q for guix-devel@gnu.org; Fri, 20 May 2022 09:53:35 -0400 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:54d1:d5d4:280e:f699]) by mira.cbaines.net (Postfix) with ESMTPSA id 060C927BBE9 for ; Fri, 20 May 2022 14:53:31 +0100 (BST) Received: from felis (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id 0a07fc24 for ; Fri, 20 May 2022 13:53:30 +0000 (UTC) User-agent: mu4e 1.6.10; emacs 27.2 From: Christopher Baines To: guix-devel@gnu.org Subject: May update on bordeaux.guix.gnu.org Date: Fri, 20 May 2022 12:34:11 +0100 Message-ID: <87tu9kmht3.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Received-SPF: pass client-ip=212.71.252.8; envelope-from=mail@cbaines.net; helo=mira.cbaines.net X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-To: larch@yhetil.org X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1653055283; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=OvGyN8rKxMhftEdIJ3aegmrmiCKQF3R9NaW/sO1/oGk=; b=iccUlZjK1pUUl8q1FZQJxt0qwHuxcRlWuDpBbNMmh2J2UacLoQyqD5elcjdhOOiuYSeOUU q2Ky2lHFbjOK6F63W2Borjs5iLvAR8gkzGHk+XEv+Vgf61nhzUluJXGEcLqqdrOr2zubfN pKy6aD7bvfuKrPQUclFIZrAJJ9EYrdU/CF8VIxDmWG2QHzlFKh/kGPDph2KdK3z4oKcEKm wz9/9TanYzINJgAsXPfLpDIj7zyGuxCOAfz1Q8Hn3X5TspEmSvDiqLnJDVIZBd/cbVO8jv xkeDYvmYWpTHjQRUdOMgc4V4ThwuMg8bgnHczKo9Y7l5QFLJiei7u37QT5S9Ow== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1653055283; a=rsa-sha256; cv=none; b=k4PLRr0/VQ1uLBdyxl66uTDf9KSTPNqIMONoxhd+f3SX2nBO8a/B+iK290WphB5JY/ZB6T 3qL0EFwjfztL3dDqYPhMRrJK6WKkGI7t2wMjE1Z+5n4d4chn0yRftvoPt4tQXe45hLSwgV Op+dL+buDRRWvktFkFzPfvFdQbSa1NVJCMBzeM9KGG0a1pq9mT7Vqnlzf1ZeoIxtgZ0Syx tSfL7iQ5q4mJ96GgYkkskzqTHZafc1XMPKAs7TVNroHoRa2AmG6JXqrXbLGfWI32ckfb// XzrXVAcuVClWRq3sQQAZBHCmFOeRphlzwwzOofXpeWQ/ifU83H9vEdy7SpoBFA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -4.94 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 778D32B493 X-Spam-Score: -4.94 X-Migadu-Scanner: scn1.migadu.com X-TUID: xLHTwwpScZSm --=-=-= Content-Type: text/plain Hi! The last update was sent out in February [1], so this update covers roughly the last 3 months. 1: https://lists.gnu.org/archive/html/guix-devel/2022-02/msg00099.html ### Summary bordeaux.guix.gnu.org is one of the default sources of substitutes, the other one being ci.guix.gnu.org. I haven't had much time to spend on Guix, so the little time I've had has gone in to general maintenance. There's also been some problems that have come up in the last few months. Those problems have been overcome, and I'm looking forward to doing more with mirrors and continuing to work on building non-master branches and packages affected by patches. If you want the details, read on. ### Some problems, and addressing them data.guix.gnu.org has had some issues recently processing revisions, which delayed builds starting. This has been addressed for now by clearing out cached connections within the inferior process used to gather information about the revision being processed. As the memory required for this processing is probably going to increase in the future, this'll area probably require further work. In other Guix Data Service news, data.qa.guix.gnu.org now exists. This is just a hopefully stable name for a Guix Data Service instance that processes non-master branches and branches related to patches. There's also been some improvements which specifically benefit data.qa.guix.gnu.org, including fixing a locking issue that prevented build events from being processed and a memory management improvement to close the inferior processes when loading revisions before waiting on the big database lock. Another problem was a disk failure on lakeside.guix.gnu.org, the machine that should have been storing all the built nars and supporting serving requests to bordeaux.guix.gnu.org. There's now a new machine, bishan.guix.gnu.org which has more storage space, and hopefully working hard disks. It's the nar-herder that is meant to assist with managing all the nars, and generally I think this problem was well handled. Even though one machine involved had a hardware issue, builds were still happening and substitutes were still available. I was able to setup a new machine, have the nar-herder download all the nars, and then switch over to using the new machine. One nar [2] was lost because it was only stored on lakeside, and the file could not longer be read from the disk (it's now been built again, so it should be available). 2: https://bordeaux.guix.gnu.org/nar/lzip/phcqx40viymdxlfaa5fpbx43np8qhzpn-qtbase-6.1.1-debug Out of this, there have been a bunch of improvements made to the nar-herder: - Logging has improved, and Prometheus style metrics are now available - The nar-herder supports handling /nar requests. The nars are still served by nginx, but the nar-herder metrics are updated. - The nar-herder now supports removing nars, which came in useful when removing the irretrievable nar mentioned above. - The "and" component of the storage nar removal criteria now works, and this is now used on bordeaux.guix.gnu.org to ensure that nars are only removed from bayfront once they're stored on two machines (rather than just one as was the case previously) - Mirroring nars has also been parallelised in the case where there is no storage limit. This helped speed up bishan downloading all the nars. The other problem has been I/O performance and probably related issues on bayfront, the machine that runs the coordinator and serves substitutes. Bayfront also does a bunch more things, with only two hard drives in RAID 1, so in some ways it's a good test that the Guix Build Coordinator can work with less performant hardware. One particular issue was that Garbage Collection (GC) in /gnu/store on bayfront was taking days to run, and while it was running, the coordinator couldn't substitute derivations from data.guix.gnu.org, which happens as part of submitting new builds. This meant that builds were delayed. To address this, if derivations aren't in the local store, the coordinator now can read them directly from a substitute server (data.guix.gnu.org in this case). This skips out unnecessarily adding them to the store, and maybe waiting for garbage collection to finish to allow this. It also means that there will be less things in the store to garbage collect as well. ### Looking forward I mentioned last time [1] that nar capacity was something that needed work soon. The new bishan machine has ~6TB of free storage, which should be enough for a while, but hatysa that also stores all the nars only has 610GB of free space. One way or another, I'll try to add 6TB of more storage at some point soon. Also mentioned last time was mirrors in different geographical locations. I've now setup a mirror in the US [3] to enable testing of this, and I'll send out a specific email about this shortly. 3: https://bordeaux-us-east-mirror.cbaines.net/ Time permitting, I want to try and keep progressing the work to build non-master branches and packages affected by patches on bordeaux.guix.gnu.org. Currently, data.qa.guix.gnu.org is processing revisions too slowly, but once that's fixed, it should be possible to look at submitting builds. Let me know if you have any questions or comments! Thanks, Chris --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmKHnVhfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9XfdGA/+LlBrpKCbwKkfa3iPzOLYwPsQ3091A0qh cKsSCmLu3v8pmIdpJY3N59Muj9XTsPLATBcacBK2j1F1ZUEuEqkrxkou9i3uB26q aCwCDQ0ESnunGqvzg4boRJejXUs7+sefdF7NylKjxFJGbpUjW3oCKLRLW6dWufry vXDdyqrzYbx0H2xZ00xeWSNg23ITG5B4K0/VuLxjABgkzItZtT5tY3PCoXTx5AE7 G+fuVoNuOK2wpVSO/84soqn5RyWBAfWIFv7L4k5s7Z2tqGdEifOip7J5tpsBp9vh HDUmQi71THOuXv4BeSLXIc56fJ3Nik+4F/c/KlvUolcCBAOr3z2hFrZgqqxh7tiH XUPhD3Nt969p4VYUsXipe7iVHEQRHVELJhCHt8vpbLjKh5ojBHScFd5pI7jCmUBj 5QAx9eI12QWsF5Vk7Xgo+9kw/g4Xz6BHO16AjsP9qwF0yozKkiYGnFeFxwWkOLBg bbf+Cggy3g5nPxbEAuj4zRE6Dkt4cDNgy7k1EcxCnXH5g5gJBl+79D9JRhMMjrYd /7EiyC/hFeImqDD8LPQcd+uZ8Gr2/09LVIQPTPEDJZ7YpSYOjBndTirvYiDkXE+c JhzQlUxbujGFKgWcmE40HsBjmpJ5AowGIXvEU01NzPJ8BPJwffteOYOI0YnA5jJl 9bDGXZw+boI= =OrG8 -----END PGP SIGNATURE----- --=-=-=--