From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp10.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms9.migadu.com with LMTPS id WMXnNla6QmQysAAASxT56A (envelope-from ) for ; Fri, 21 Apr 2023 18:31:19 +0200 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp10.migadu.com with LMTPS id IFH0NVa6QmRPSwAAG6o9tA (envelope-from ) for ; Fri, 21 Apr 2023 18:31:18 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 836A52A673 for ; Fri, 21 Apr 2023 18:31:18 +0200 (CEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pptet-0006ND-Oo; Fri, 21 Apr 2023 12:30:51 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pnckp-0006o8-C7 for guix-devel@gnu.org; Sat, 15 Apr 2023 06:03:35 -0400 Received: from mail-wm1-x329.google.com ([2a00:1450:4864:20::329]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pnckn-0002Of-ED for guix-devel@gnu.org; Sat, 15 Apr 2023 06:03:35 -0400 Received: by mail-wm1-x329.google.com with SMTP id 5b1f17b1804b1-3f0a0c4e505so1818115e9.0 for ; Sat, 15 Apr 2023 03:03:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1681553011; x=1684145011; h=mime-version:message-id:date:subject:to:user-agent:from:from:to:cc :subject:date:message-id:reply-to; bh=dpSkLqPseLl76PwtaUPAw3+xZTTjkZcK4w4rtJH5IIo=; b=LCPUTshJpb76/ASZUKSPY7vBmuZjnVjxD+mPUdtw64Q/fizqXZsFjR3ignWpt4VntE pYBcwkc75RSmJy+3SJB/tpXaRq9nOj8aWdUEEFxCu9nT99LNTzX3+IDupXDgyCRiupkB YRPMfoFb5KmRZJwzjfDr5liUHvakrFmtKo3SnPSSyWoDQNUFGE4v5usUdFyPwKHgM9oF hrc+hHIerUwrURRNxdkBCt5nr6J6NvA65Ett9/MmasbWzyThB6as4LP+3APD+lCIrCMU ughbBpQxXeJQkFvdwcKW4sVMuqsu7Qq6B4EhUMQGrnIAJ3XFQ+BkzhGTNOKuoBOInQ2e X50A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681553011; x=1684145011; h=mime-version:message-id:date:subject:to:user-agent:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=dpSkLqPseLl76PwtaUPAw3+xZTTjkZcK4w4rtJH5IIo=; b=SsfSZ43qNieIh9E/KIwJsc+WypjfSAQB2vjYVC5X6cVpT/9vJy3lJHj7R7hXuWilM9 kSTF8ce58WInu6MLOEX54wbh9ZfBAkfUotEgoy1oYVLfHUu0Q09jOeQF7PyD19XbeoT+ N0qfDtIeCdo25o3SeDE9BMFSnDTj65HekppK0Xrr3jeGppOBdmROZi7jZmzQ1QpPyEjL OOMb9cazLt+IllR2xB3M2hMTCiLs/GWc9/1ylO58FBisnq2MqtS0a7inHMV6xnuhVdZk lKx016MDdLmJckmyFmLsOSczx7qtGAVPEIidC7H5zQbBBSiHEltjjF/lROBY5BV96tUd XOnA== X-Gm-Message-State: AAQBX9fhUAhagFuZM2GOzxajsjfBgRHnDnRrEynY3bf8fHjqndODn87R LbF0gsFlsJ3PZld/N1WA6qBsCWY6yJA= X-Google-Smtp-Source: AKy350b7BablHLyuFrHKJlLw5esjFPrg66/mehCfm81COaRSFWw3LmkQ/9qFZmjj3HK8l42Hk80/iQ== X-Received: by 2002:adf:ed85:0:b0:2f5:fbe5:bb1c with SMTP id c5-20020adfed85000000b002f5fbe5bb1cmr1244495wro.63.1681553011423; Sat, 15 Apr 2023 03:03:31 -0700 (PDT) Received: from felis ([2a02:8010:68c1:0:54d1:d5d4:280e:f699]) by smtp.gmail.com with ESMTPSA id t12-20020a5d460c000000b002c561805a4csm5439781wrq.45.2023.04.15.03.03.30 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 15 Apr 2023 03:03:30 -0700 (PDT) From: Christopher Baines X-Google-Original-From: Christopher Baines User-agent: mu4e 1.8.13; emacs 28.2 To: guix-devel@gnu.org Subject: April update on bordeaux.guix.gnu.org / guix-build-coordinator Date: Thu, 13 Apr 2023 19:32:54 +0100 Message-ID: <87v8hxv4wg.fsf@gmail.com> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Received-SPF: pass client-ip=2a00:1450:4864:20::329; envelope-from=cbaines8@gmail.com; helo=mail-wm1-x329.google.com X-Spam_score_int: -4 X-Spam_score: -0.5 X-Spam_bar: / X-Spam_report: (-0.5 / 5.0 requ) BAYES_00=-1.9, DATE_IN_PAST_24_48=1.34, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-Mailman-Approved-At: Fri, 21 Apr 2023 12:30:50 -0400 X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: guix-devel-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Seal: i=1; s=key1; d=yhetil.org; t=1682094678; a=rsa-sha256; cv=none; b=hBY+pxgDhwg/WAvIb80hFqIKXfapMBMOUAxMKisHV8KU2H5lf3/hwuAl8qg8Lf2TlWQhs0 S0FeHTCRv67umdQkpA56zBoQ7ayVS4lsbXU3gg6KR86FcuPHhNkBzatY6QXmzEcpTDiU1R 4e976SMLdQEPGB/MPHAXJQrcgWPSoTcozfz6fXS+KIC7kPUwHvh3Rqk2EHHv+ssVp3lJ4c z3f9WosZGNTAysy38SXwUcCgA7G8zeJKzqjKhK8lWft3ZXFde2VA/HZMes+ST1y//rxnW6 BakEZpm4HVbg1LAuX2BZNuVSfDsol1NfWInz5+p8nmkyzetTCZbD0tCzM0Vefg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=LCPUTshJ; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1682094678; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=dpSkLqPseLl76PwtaUPAw3+xZTTjkZcK4w4rtJH5IIo=; b=cZwbpBAzhTszRC2cY1In5ZYBqX8htqxMNJ/MgyAeW2reGL4qvqTRPrfXoyhyVxQ2sVvbyo RzkR/Pw7J14gn8vJndWHZre82EL+6WOVF+PzitnQ97JDqbupD5rNQzBjn5810iGpp0ZU+G 4yFN6KivpWa5D1kuvkrSWKOXHkifsZ1CdDMbAmLCphJqfRFMuEPfzNoZbbwjYq4+81dE7Y M0DwERDeaoTBzxWouZuCvS48Abho4hC+CnTE38mlFbO49dTRjRbo0PuKLT8M3vXmOaQe1O 57IgWXScx9hTANhFQIOBfkoguF7MPDh2568+N7fANqRovITMI8qIta9r4m0Cqw== X-Migadu-Scanner: scn1.migadu.com Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=LCPUTshJ; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -6.05 X-Spam-Score: -6.05 X-Migadu-Queue-Id: 836A52A673 X-TUID: MQmaGjEUFn7e --=-=-= Content-Type: text/plain Hey! I sent out the last update about a month ago in March [1]. 1: https://lists.gnu.org/archive/html/guix-devel/2023-03/msg00262.html ## Numbers bordeaux.guix.gnu.org currently provides ~2.4 million nars, which take up ~10TiB to store. ## Storage There are two machines with all the nars, hatysa and bishan. On the plus size, I recently installed a new hard drive in hatysa so it now has plenty of storage for more nars. However, space on bishan is running out, it now has less than 1TiB of free space and will probably run out of space within the next 4 to 8 weeks. ## Problems and bug fixes In the past, I've seen the coordinator memory usage unexpectedly spike and I haven't really understood why. This started happening again recently, and I managed to make some progress on tracking it down. It seems that the problem happens during calls to (backtrace). I was able to reproduce this with the error handling for hooks, but I haven't been able to reproduce it in a more standalone manor yet. I've opened a Guile bug here [2], and for now, I've been trying to work around this issue by removing backtrace calls from the coordinator. Obviously this isn't ideal, but I'd also like to avoid this problem. 2: https://issues.guix.gnu.org/62623 Another odd issue that I've been coming up against for a while is some port encoding issue, I've filed a bug about that too [3]. I've been working around this one by adding silent error handling to logging in critical areas, so that things keep working even if the logging raises an exception. 3: https://issues.guix.gnu.org/62590 I also investigated why there were problems substituting derivations on the childhurds. This was broken by some timeout related changes I made that don't work on the hurd, this is now fixed and the error reporting in that area is improved. I recently spotted a crash in the build coordinator when building anthy-9100h [4], that turned out to be due to a bug in Guile with handling invalid unicode when using suspendable ports. This is now fixed upstream [5] and the relevant Guile package in Guix has this patch included. 4: https://issues.guix.gnu.org/62240 5: https://issues.guix.gnu.org/62290 I think I made some progress on the write_wait_fd errors I've been seeing from the coordinator agents. Luckily Ludovic seems to have done most of the work, so I was able to send a patch for guile-gnutls [6]. 6: https://gitlab.com/gnutls/guile/-/merge_requests/10 ## Progress The Git repositories for the guix-build-coordinator and nar-herder are now on Savannah [7], which is great as it means other committers can now easily push to these repositories. 7: https://git.savannah.gnu.org/cgit/guix The big new feature I've been working on is support for listening for events from the coordinator. This is only possible recently with the support for streaming responses in the guile-fibers web server. While the build coordinator isn't intended as a web service, it does make use of http for talking to clients and agents. I've followed the standard for server sent events [8] for this new functionality. 8: https://html.spec.whatwg.org/multipage/server-sent-events.html The client interface for the coordinator isn't exposed since there's no authentication mechanism. However, I've also got a prototype for a web frontend [9] for the build farm up and running. This does expose the events stream, along with the state information that's needed to make sense of this. The result so far is this activity page [10], which shows information about the agents and the builds allocated to them. It's still a very rough prototype though, and there's more work needed to make it reliable and include more information to make it more useful. 9: https://git.cbaines.net/guix/bffe/ 10: https://bordeaux.guix.gnu.org/activity I've also made some improvements to the build coordinator in terms of cancelling builds, the combined post publish hook (to enable validating the availability of referenced nars), parallel hook processing in the coordinator and prioritising post build actions in the agents (which helps mitigate congested uploading). ## Next steps The bishan storage problem is growing ever closer, there's still a need to come up with a plan. One which ideally reduces the amount of hardware I'm personally renting. I still need to do a bit more work on validating nar reference availability when asking the nar-herder to import nars. Now that the bordeaux build farm is being used for QA, there will be some nars that no longer need to be kept (as they correspond to some derivation that didn't end up on the master branch). It would be good to start automatically removing these to free up space. As above, I think a good start has been made on making the build coordinator behind bordeaux more observable, but there's still lots of room for improvement with that. I also have some sysadmin things to do. The Overdrive (ARM) machine monokuma has some btrfs issue with it's drive, so I need to reinstall Guix on it to get it back working. I also have a RiscV board that I've had for ages, and should get connected up to the build farm to start building things. If you're interested in working on any of this, do let me know, and let me know if you have any comments or questions! Thanks, Chris --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmQ6dm9fFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9XeP0g//QXa/Jlb/FpvNOaypYjW/sXi8/COTMuE1 Hgs66REzkRcBGKrAGKxgmySxmZ34+TAplIqZh3G23pfv3LfPjVDr36/1yidqBgA/ Z6zy5ubdrDb1VCQsCDcOlDKA/u3QaEx0pTMzJv4h1Wt7bHZ+sE997wywPbLjY7WC 6el48MNC++OHaLT5+kVTs7aGn8Vh8QSzOmPVRANKDYjcYlFsyOaINnJDPFLLaA/R 2KjuU/j74deoMDF834ESk26KAp07g8Kf62e87dSE1dcDyfvIqEkeIgWBtI5J82gi OzUc5SJz6+++OfqWQdQoGE8wOH038PZIrPVitcTFhlRvwxQG0CnpDjX/mB8fg7dY iCSImYR3XmvSoLo1JanwovMkWE4W9XdwaVFv+wU9adO/MUJPjrXcaEkHU9K2IxyB JqMLpTsR3SFdCPl+hoUnJGKJGrTjbF8siCBkQ/rpMKSOGdCUtNCDESFv3g7cJWmw hBjAdtbcijzl9z89xpkM9H5vGiVO8Fd/sbX9uLrlrHZUIW8RuJMJw0pIwM1D98tT QmmTj5wPCHzoaK0TTQMHqr//ZheuZz6csOMsEfKF241faN0UkZAn76Nw67Xtmv3A K2dZ9sdoPLfhgsZ2nnimyQeghV+/qjcwC1ounrUzDHn9PvH8/hA1qalLFDGueOJF WhG/Qm0Wiqg= =6AfT -----END PGP SIGNATURE----- --=-=-=--