From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1.migadu.com ([2001:41d0:403:4876::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms13.migadu.com with LMTPS id qP4eCGEy32atSwAA62LTzQ:P1 (envelope-from ) for ; Mon, 09 Sep 2024 17:37:37 +0000 Received: from aspmx1.migadu.com ([2001:41d0:403:4876::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1.migadu.com with LMTPS id qP4eCGEy32atSwAA62LTzQ (envelope-from ) for ; Mon, 09 Sep 2024 19:37:37 +0200 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=PmGQUCu1; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1725903456; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=rULy3wxYou2Yb7RJQ+6xEaePAEweTQcMG8dii6gdfWU=; b=J5bF3Sl6exg68ycg4fU3Em7iXf7BGL9u18DbgrrwtwS72Pj2NrVDPOlaPD2La2gym6Ftb6 6RglKONlUuAFtDhw2P5iMOiFovFRyaGra0a4IwWRE5ys+q04gTN7s8jBbcnKa1WiqJlvGz Z0sS0rgTuwyuBPfbzmMQDLSm5yKR8JjEMRdaClS8DphTxsNtYmCjf5TcGFLOEE+56wUQXN OKgtunN9pFGjwQIOk8PfgDaHnCCSuMzxNUqcJVQyh6rr6UDtI1l0267IUn/kCS6PCAfXhx 0CsptZOpM4rgbjfstNR1vfrtqYaabWFMOPkKpel3BYFgU0057mX9d19fozrMWA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=PmGQUCu1; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=key1; d=yhetil.org; t=1725903456; a=rsa-sha256; cv=none; b=UUax3Mu1JYwE88nkSsjd12RgKwmTI9s6pl14g9p+tJWy4/2vVc3FwbIZcam082E3jmSNwO lDnbQH6L8JFTkiwznF/7NXJJEmFJZhTBpH4SJPCEKQZBuhRFY/hsmg1qUVdR03JAV7GX5P fS7hZpnP6OxxZl8t2Q9h/r5S7bRYCH+4qUDH5+xXb0bHEPCsjExPd1kk5AyuGI93rfXdE8 i7iZ18Oek+dwml8e7nzzXI0NwzBZiTtQZ5v1N0jWQzcVw89sJ39NNRoK4IGcxj5v4oIjIz 1mDx37wqfZ0MgqVV89/03KOJv6jOm+iEuIhWoJmOUA5loiw3uz1BpQuasWuPoA== Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id ADF28701EE for ; Mon, 09 Sep 2024 19:37:36 +0200 (CEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sniJz-00018l-67; Mon, 09 Sep 2024 13:37:03 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sniJv-0000vC-QH for guix-devel@gnu.org; Mon, 09 Sep 2024 13:36:59 -0400 Received: from mail-wr1-x435.google.com ([2a00:1450:4864:20::435]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1sniJt-0006xQ-UF; Mon, 09 Sep 2024 13:36:59 -0400 Received: by mail-wr1-x435.google.com with SMTP id ffacd0b85a97d-374ca7a10d4so2748704f8f.3; Mon, 09 Sep 2024 10:36:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1725903414; x=1726508214; darn=gnu.org; h=content-transfer-encoding:mime-version:message-id:date:references :in-reply-to:subject:cc:to:from:from:to:cc:subject:date:message-id :reply-to; bh=rULy3wxYou2Yb7RJQ+6xEaePAEweTQcMG8dii6gdfWU=; b=PmGQUCu1xGJ6dtBNIKowHxf4tUHrOAJGzeydWuHpWcBp9ccjETbQU8tkM97f3dZe2t sbdOUXI3bnIKbWnXkvNZEEHppIKKKX2RCVASw3AC7n2BOsXiiNwVO3amukpvpjts39zd cl+elDVjI71XCfWqtUdsNi3eyoMGhgyEqju9AH9fFPAFEsnPAbCFDysSweSDaFzCOoXM UFa+bftsbgDDHMnQyeJCJy63YQfgCsWHKm16VCQ2Voq9zylgFGo3gH/dXPI54cNvg3wQ CT2IyRWNWvHVyBeNsLpq+U7NtAAIDsBSrqkJDkRgPOWS08NEz4wXUJJumnfRR7ggPSCr Bqog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725903414; x=1726508214; h=content-transfer-encoding:mime-version:message-id:date:references :in-reply-to:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rULy3wxYou2Yb7RJQ+6xEaePAEweTQcMG8dii6gdfWU=; b=TWlmZ+SnRWsMyf8daooc5tGYzuRnnegsf2bgUZNtR+yrmqfIfeuMwELQ0nAh+Ixw0y G+sxR/+KY9u/ufBWOw/Rm+k4o4whUoOt8gO2GL41R1nhuFvgtkHAtO1Kz6m3+zQOYavU 7SY91E8Yy0yZL9hC6SSCv62UM6Aa2/t2FejYNUrRH632XCuvgk1Tw7YZqhb8MORIdOme +NYdAVCyqf+meIeYZVb367N8/yZ9QZx1QDLS09gIX+OEnMJL4LYdxztTEY9+NI89tuMM POvCkGaXN/gIkCbhX6ea9Xba0ywGrQZKip+4AAKA97bRTcU3zD2+Jm9/B8WihvkfPS53 Q5zA== X-Gm-Message-State: AOJu0YxTWpmcjewM5B9GUGRQ8AC129oHdkw3nRDOGDIvf5vAFoQl/Twl 04Ov9heviotX/tTlJF5YGk8189Gi2se0vVz323vcs1wUJ4mpdDmCuDVtiQ== X-Google-Smtp-Source: AGHT+IGEuHCKKHXtBethJUHR1UF1fu/eOy1G+TAnoYFlgdf6Jv8dN1wDLbhkNVeyDiQ4xvOn66MC6A== X-Received: by 2002:adf:f948:0:b0:367:9881:7d66 with SMTP id ffacd0b85a97d-37892703fe7mr4411672f8f.41.1725903413873; Mon, 09 Sep 2024 10:36:53 -0700 (PDT) Received: from lili (roam-nat-fw-prg-194-254-61-47.net.univ-paris-diderot.fr. [194.254.61.47]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-378956d375asm6543829f8f.66.2024.09.09.10.36.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Sep 2024 10:36:53 -0700 (PDT) From: Simon Tournier To: Ludovic =?utf-8?Q?Court=C3=A8s?= Cc: guix-devel Subject: Naming =?utf-8?Q?=E2=80=9Cbuild_train=E2=80=9D?= instead of =?utf-8?Q?=E2=80=9Cmerge_train=E2=80=9D=3F?= In-Reply-To: <87zfol170t.fsf@gnu.org> References: <87le0cj13e.fsf@inria.fr> <87v7zby3r6.fsf@gmail.com> <87zfol170t.fsf@gnu.org> Date: Mon, 09 Sep 2024 19:28:57 +0200 Message-ID: <878qw0sply.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=2a00:1450:4864:20::435; envelope-from=zimon.toutoune@gmail.com; helo=mail-wr1-x435.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: guix-devel-bounces+larch=yhetil.org@gnu.org X-Migadu-Country: US X-Migadu-Flow: FLOW_IN X-Spam-Score: -8.81 X-Migadu-Queue-Id: ADF28701EE X-Migadu-Scanner: mx10.migadu.com X-Migadu-Spam-Score: -8.81 X-TUID: VOilP4Dg5y19 Hi Ludo, all On Fri, 06 Sep 2024 at 11:11, Ludovic Court=C3=A8s wrote: > In the end, perhaps we=E2=80=99ll have to negotiate on a case-by-case bas= is. > The important thing to me is: independent testing as much as possible, > and well-defined responsibilities and scope for the people/teams > engaging in such changes. I agree. My main concern is about the potential (excessive) unnecessary wasted rebuilds. We need a policy that clarifies how to start large rebuilds, which implies cross the final line with a merge. Let take the example from this another message [1] as one case. On Fri, 06 Sep 2024 at 11:01, Ludovic Court=C3=A8s wrote: > =E2=80=A2 Attach to a =E2=80=9Cmerge train=E2=80=9D. For instance, ass= ume there=E2=80=99s a branch > changing =E2=80=98gsl=E2=80=99: this is totally unrelated to =E2=80= =98ffmpeg=E2=80=99 but it also > triggers a lot of rebuilds. We could tack that second branch on top > of the known-good =E2=80=98ffmpeg=E2=80=99 branch, and, once it=E2=80= =99s all good, merge > that =E2=80=9Ctrain=E2=80=9D into =E2=80=98master=E2=80=99. > > (To be clear, the term =E2=80=9Cmerge train=E2=80=9D originates from GitL= ab-CI and > similar CI tool, which use it as a merge scheduling strategy: > . GitLab-CI > can create merge trains automatically I believe, but in our case we=E2=80= =99d do > that manually, at least for now.) If we consider this specific case, the =E2=80=9Cmerge train=E2=80=9D workfl= ow means: a) the branch changing =E2=80=9Cgsl=E2=80=9D is built, so 1473 rebuilds. b) the branch changing =E2=80=9Cgsl=E2=80=9D and =E2=80=9Cffmpeg=E2=80=9D= is also built in parallel, so 521 rebuilds. The =E2=80=9Cmerge train=E2=80=9D workflow also reads: i) the branch changing =E2=80=9Cffmpeg=E2=80=9D is built, so 521 rebuilds. ii) the branch changing =E2=80=9Cgsl=E2=80=9D and =E2=80=9Cffmpeg=E2=80=9D= is also built in parallel, so 1473 rebuilds including all the 521 rebuilds of i). Therefore, for each scenario, 521 builds are =E2=80=9Cwasted=E2=80=9C, comp= ared to the optimal 1473 in this case. I do not think it=E2=80=99s what you have in mind when speaking about =E2= =80=9Cmerge train=E2=80=9D. Is it? Maybe it points that =E2=80=9Cmerge train=E2=80=9D as described by Gitlab i= s not what we need. The =E2=80=9Cmerge train=E2=80=9D workflow runs *in parallel* the bu= ilds of several branches and I am not convinced this is wanted for heavy branches as it might happens. Therefore in order to avoid some confusion, maybe we could avoid the name =E2=80=9Cmerge train=E2=80=9D and use another name for the strategy we= are discussing and we would like to implement. For instance =E2=80=9Crebase tr= ain=E2=80=9D or =E2=80=9Cbuild train=E2=80=9D. Or yet another name. :-) Let try to not waste resource. I think we could have a policy when a branch contains =E2=80=9Cheavy=E2=80=9D rebuilds. Because we cannot contin= uously rebuild the world. ;-) Somehow, the team of this =E2=80=9Cheavy=E2=80=9D rebuild branch says: =E2= =80=9Chey build train of branch X starts soon=E2=80=9D, meaning the team is ready and the state o= f the branch X looks good so it require to set up the CI, i.e., the team is going to trigger more than 1000 rebuilds. The message =E2=80=9Cbuild train= of branch X starts soon=E2=80=9D is sent to guix-devel and one week is let to = the other teams to rebase their own branch onto branch X, if they have something to build, are ready, etc. The team who asked the =E2=80=9Cbuild train=E2=80=9D is responsible for cro= ssing the final line (the merge to master) and also responsible to synchronize with the wagons, e.g., by sending some updates about how is going the world rebuild, when the final merge is planned, etc. If no one takes this train, no big deal because a next train is probably going to start soon. ;-) WDYT? Cheers, simon --8<---------------cut here---------------start------------->8--- $ guix refresh -l gsl | cut -d':' -f2 | tr ' ' '\n' | tail -n +2 | sort | uniq > gsl.deps $ for i in $(seq 2 7); do guix refresh -l ffmpeg@$i \ | cut -d':' -f2 | tr ' ' '\n' | tail -n +2 ;done | sort | uniq > ffmpeg.= deps $ wc -l gsl.deps ffmpeg.deps=20 1473 gsl.deps 521 ffmpeg.deps 1994 total $ for line in $(cat ffmpeg.deps); do grep -n ^$line gsl.deps ;done | wc -l 521 --8<---------------cut here---------------end--------------->8--- PS: Please note the number here are not true the number of rebuilds but the number of packages that triggers all the necessary rebuilds. 1: Re: =E2=80=98core-updates=E2=80=99 is gone; long live =E2=80=98core-pack= ages-team=E2=80=99! Ludovic Court=C3=A8s Fri, 06 Sep 2024 11:01:46 +0200 id:87h6at2m11.fsf@gnu.org https://lists.gnu.org/archive/html/guix-devel/2024-09 https://yhetil.org/guix/87h6at2m11.fsf@gnu.org