From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp11.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms9.migadu.com with LMTPS id EPo3LMGpW2TYuwAASxT56A (envelope-from ) for ; Wed, 10 May 2023 16:27:13 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp11.migadu.com with LMTPS id SM4hLMGpW2TEWQEA9RJhRA (envelope-from ) for ; Wed, 10 May 2023 16:27:13 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id DACE62B42F for ; Wed, 10 May 2023 16:27:12 +0200 (CEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pwkmD-0005h8-MR; Wed, 10 May 2023 10:26:45 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pwkm6-0005fP-DJ for guix-devel@gnu.org; Wed, 10 May 2023 10:26:41 -0400 Received: from mira.cbaines.net ([212.71.252.8]) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pwkm3-0001Xb-P5 for guix-devel@gnu.org; Wed, 10 May 2023 10:26:37 -0400 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:54d1:d5d4:280e:f699]) by mira.cbaines.net (Postfix) with ESMTPSA id 5E8E627BBEC; Wed, 10 May 2023 15:26:33 +0100 (BST) Received: from felis (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id 55245a62; Wed, 10 May 2023 14:26:32 +0000 (UTC) References: User-agent: mu4e 1.8.13; emacs 28.2 From: Christopher Baines To: Andreas Enge Cc: guix-devel@gnu.org Subject: Re: Tooling for branch workflows Date: Wed, 10 May 2023 15:17:19 +0100 In-reply-to: Message-ID: <87bkis1cvd.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Received-SPF: pass client-ip=212.71.252.8; envelope-from=mail@cbaines.net; helo=mira.cbaines.net X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: guix-devel-bounces+larch=yhetil.org@gnu.org X-Migadu-Country: US X-Migadu-Flow: FLOW_IN ARC-Seal: i=1; s=key1; d=yhetil.org; t=1683728833; a=rsa-sha256; cv=none; b=F+Z+TzF/G8WJwK/Ll0rJ7VM5dIYhl1qohU7Bx5dLM/uQqBDefSUwqIrsS5jakSRPX5sZ4n XCN7XvaOU++say54L0BFz+Dvf53lfC0qbtMN9gK+PSmJnPx9uUgSpYbykxrxdutknbirQf kIZc4LawtFRAKVyP9NLALKlalrPr1HDcjwhEi8HR06RWcUv5Xpd72kUOLT0qatvXpJzDae dcY/jTjLn/FXT6M2bsArzHuZFoC1y74dwe5apguDIUriBazz+3jRWDarsAuf+S5eyLI0p+ OSLeGFAfGIlAAxijCoSW0Qf3plhxD7ZV8jqVQg4NRNQtiH51xy64gXpPyIGDBA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1683728833; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=9FcMNh9v0jJxzTLs35XAYjL8IR1cuMqIcLGJBKNNRjo=; b=gfEBx0Lq8KCv17IJhovVY0/AMleios+LHiaxjuUFdX53gmeaNXZSABO03CBIumvSU9qltq Kq5aeI3j6f8rEb1KDKrAmqfPOui5AY0X27JhOFYFn6HlY8lgZc0sLD9m1gtpfGCF6g7R+S JCgVMUON/IG/VO9/aM0jZBpnSTfGvMEWcytEZz+knCwJAAeKCarBEwL2GesMzyugN9WZon ngmKWKohE599YkxA96DDzHTcwVlZBRrpHQ9eCtGeI07oJyVknIQrLNVrI9q36IFm6ff4C/ BiQfsI0lpZiE/045oQvgMoQhrF0zP48Hwzb51mTabHy7GOj+sjYEvk1xhMnSNA== X-Migadu-Spam-Score: -3.81 X-Spam-Score: -3.81 X-Migadu-Queue-Id: DACE62B42F X-Migadu-Scanner: scn0.migadu.com Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-TUID: wcNir9trYnY0 --=-=-= Content-Type: text/plain Andreas Enge writes: > the title says it all, I wish to share some conclusions from working on > the core-updates merge. Clearly our tooling could be improved for the task; > there was some flying by night without instruments, and in the end I > merged the branch without being really able to tell how it compared > to master... (You may also blame it partially on my lack of patience.) > Having feature branches may or may not make things a bit easier, but it > will definitely not solve the problems. > This mail is also of course a bit politically sensitive: It may look like > I am complaining about other people's work, who are volunteers and do what > they can, without offering to work on the code myself. So as a preamble, > let me express my gratitude to the few people who have been working > tirelessly on our tooling and contributing to our infrastructure, without > whom big code changes like we did on core-updates (and now on feature > branches) would simply be impossible; their work is vital to the project > and often not very visible. If I am critical, it is not to diminish their > work, but to discuss about a positive path forward; and I hope more people > will find the motivation to do infrastructure work, which I think will be > decisive for the success of Guix (together with policy and organisational > questions). I too would like to see improved tooling for managing changes. I've been working on the qa-frontpage for smaller changes, but I think this can be extended to large changes too. > We have two build farms, berlin and bordeaux (which is a good thing for > checking reproducibility and for redundancy, but maybe a bit of a problem > concerning hardware requirements for "exotic" architectures), running > two different CI projects, cuirass and the Guix build coordinator (gbc in > the following); both have a very low bus factor (1 to 2?), and it would be > nice to get more people onboard. For this, more documentation would be > helpful. Both have pros and cons, and are architectured quite differently, > so I do not know whether convergence is achievable. This is something I'd like to see too. I've been thinking about this already and want to spend more time working on at least making it easier for more people to get involved. Let me know if you have any specific requests! > I ended up relying mostly on cuirass for reasons I do not completely > remember any more. The dashboard with its green and red dots is a very > useful tool compared to lists of builds, which become unusable with over > 20000 packages. The bigger build power on bordeaux is helpful, and I found > the web interface of gbc a bit slow and down a bit too often. With this > experience, I just filed three wishlist bugs for cuirass: > - Topological sorting in cuirass > https://issues.guix.gnu.org/63412 > The lack of ordering the builds is a big problem wasting a lot of build > power; it is solved in gbc and, I think, the reason why the bordeaux > build farm fares better for aarch64 with fewer machines. > I would tag this as "important". > - Evaluation comparison on cuirass > https://issues.guix.gnu.org/63414 > Without being able to compare a branch to master, it is difficult to > decide whether one should merge. This is sort of solved in gbc, but so > far the bordeaux build farm has been used more for QA of single patches > (or a short list of patches featuring in a single issue) than for building > complete branches. > - Stop and restart builds in cuirass > https://issues.guix.gnu.org/63413 > Manual intervention is not easy in cuirass (I spent hours clicking on > "restart" or using the REST API with a shell script through wget, which > resulted in my IP being banned as a DoS suspect...); and to my knowledge, > there is no web interface for doing so in gbc. In both systems one can > probably tinker with the underlying databases, but this also does not > qualify as "easy". There's intentionally no concept of restarting builds in the build coordinator, however unlike Cuirass, it does support having multiple builds for the same derivation. So, to try to build something again you just submit another build for the derivation, which can be as simple as running this on bayfront: guix-build-coordinator build /gnu/store/foo.drv Having to have SSH access to bayfront to do this isn't ideal though, so we can expose some kind of authenticated interface to do this, maybe through the new bffe or the qa-frontpage. Thanks, Chris --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmRbqZZfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9XfUcQ//da+78t5NXgM3MQs5AEcEwLj9b0IEUleA B5yrVROD+c27vcvXZqLZhq8TxB5EzFzkklSbdrt3fu7uueUSm9cs2++za1TmtL0v hOg95p4NbWfxfTm3BzddaPRYp/wpJd6RhDEKStesa/53qjOeAlxSlpELC6KGoiG/ K80tfoj5DPG0Y19M2nabiHcgA8IhtkHgrwLKuYN/oX35cIrYR2tbrJ+bpsiVwm4C HNPT9CslBQ39pqbCLzGu7ujzQxfxinPyw6FMwL/ytR2Mql9Y4ELQlMjpb+CilTc8 KCR9Qopr4H3f8Ux5oS+Tf6rrsx4t5L4jdnbuLmJ/uwAx13v54Oiip0fXbMMFMG9x bcMh0aN2i43Hg+qs6YP94Q7XWVrjPIztB2qS3D6rVCOSA1X7SAUQR4zAZkb4WEBe 4p9Qieu9hHxsdJVqkhPD67zYefF5/m/BmUfKif4Amo7EBeIImzD1EBxjlGvf72Y5 65UW4SETDsO2/grslV1E5GjIrlaUws2PTF2308EOef31LLpX9zwdF4JyMxMYEDZm l9qj1MpD5Tj4t3dA8PGivlRQ3l0+yjeW2jjn60Vu26i6A9DoYvoHuMwEFYr2Vyhj ZKtCCPmxC6WG9DBVa1y6E0ayeHoqSkzWzVV3usWR30JNL3btrdlxn+qN+qvIQtC5 mjZoS9h0C1A= =DE11 -----END PGP SIGNATURE----- --=-=-=--