From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2.migadu.com ([2001:41d0:303:e16b::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms13.migadu.com with LMTPS id qFdUFeFZ2GbqTgEAe85BDQ:P1 (envelope-from ) for ; Wed, 04 Sep 2024 13:00:17 +0000 Received: from aspmx1.migadu.com ([2001:41d0:303:e16b::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2.migadu.com with LMTPS id qFdUFeFZ2GbqTgEAe85BDQ (envelope-from ) for ; Wed, 04 Sep 2024 15:00:17 +0200 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Jy0ZKyTl; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1725454817; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=e6ms1brXBVZGrogrW0qLvC1T51OX0VVdql7SzfmrmiU=; b=D6cUIQurDIt3/SWNqMzn+MwgLJfosNePWJlYZEWF5ntt1hTqi0MRmC/wf+OkwXP1oPPgp0 5oJ0RKrW5h/wFYFCqzGLDizoD38Z7DvINedy+mGSF7Gg4DcvRNgPb+NLK5qIfHop7GdOpe QbZlMQzX1eqHAKbmtdf2Z3UNpNlenSakQCd0l7OC/124sZfZRqTqUoPVSLG1y7pc1tRm8l iZ/YYza16xLl09QwGyTXg6lkdtaAILv5gJS7xyjyEWTyuvA4uVe8b1uDfDMiqkCkiwjLC0 oUnBtqh3PznIwI2SVKpaC11ELJrxwbFqbCWByrg5Ifhu766nbvRiLjB5hnvTNQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Jy0ZKyTl; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=key1; d=yhetil.org; t=1725454817; a=rsa-sha256; cv=none; b=lCyx76BGnf0puBVOiECm5fBkY/d2cZ3e19TlmKaAX29LgmAjhmUr0bj39eV9kWpzp1OsBA GAxSi5KZxHr2hHRZq26+uQqUV7OftoGlrnOpj4zj8vc0BscSB1mcFPoRb+OlotvlwlK5rk dGVPMmMGXHk+FKiSPLtsw3n6cubsVyMvUktFmPNGmXL519MQKlFH5LvfSs0SqIu/cejqlB b6iApdOrczT4mLjQg/+zacpIywAE9zeAoNHLT91zDiO1GutQ1yZMEKCH/nCZyoV4/JahEg o7wdEiXLUe8TOcD2iuyj55F5vhNvnY14GyhJhnESKp8zKZHi8ey9v8BR9s1mng== Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 22F38175DC for ; Wed, 4 Sep 2024 15:00:17 +0200 (CEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1slpbk-0002VI-05; Wed, 04 Sep 2024 08:59:36 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1slpbh-0002Ma-7L for guix-devel@gnu.org; Wed, 04 Sep 2024 08:59:33 -0400 Received: from mail-wr1-x42a.google.com ([2a00:1450:4864:20::42a]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1slpbf-0004v3-9y; Wed, 04 Sep 2024 08:59:32 -0400 Received: by mail-wr1-x42a.google.com with SMTP id ffacd0b85a97d-374c6187b6eso2207414f8f.0; Wed, 04 Sep 2024 05:59:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1725454768; x=1726059568; darn=gnu.org; h=content-transfer-encoding:mime-version:message-id:date:references :in-reply-to:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=e6ms1brXBVZGrogrW0qLvC1T51OX0VVdql7SzfmrmiU=; b=Jy0ZKyTlYwHfLEYGYkAu7rb/SOExt7xgjLDechcFZkArvTfhO7dYmKCfyXpxI1AP1j rafq9c6RjxCMk1h92hGDFsJXg3IoGzwswnw75F5vrIS7aPeZW8KLCkpv6MTM2da24aIj W5ewP3/VlcJakwpUaRggZDl4Ksc33sqRe/9mNnra20V1rOrWRkfKgH+/BSn9xFcm+CjI 16Z8+IZfyz7evMxyUny2aqBICGtIPnEJYb+4zwct2n41EFkyzKMukXPR8cXrRk+ZqZOB 3hvclPUZTxvUCBvsPveujREMI5FWYArDIjs0++StHIT96ORsKfkl/tX9uiFZ8wVpN9VM nXBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725454768; x=1726059568; h=content-transfer-encoding:mime-version:message-id:date:references :in-reply-to:subject:to:from:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=e6ms1brXBVZGrogrW0qLvC1T51OX0VVdql7SzfmrmiU=; b=JY1z457btJNrB30iP3uwbrSoRDnEic64+ieGIuPJqDEGG8w1HOkKyA6SGkrz5PrQsu v2GChQaJWFIH6UgZrtiA1uM/1Cw0H0/xHkhJNdKhVfsiTSketfYOjWHuXrzId/iumh9p Vudb7IKQ3AuP28JAEzAtuZRhFaJDDRZGv3NXthPD51PxL+H7PqgJOPw/1pGprT6TyZIW fuSCLfir3xkuVT343LXTWtzgqz95/xSeBhnWNELB8el3hkEdc7kO2NJ6WHpVtZIWSDfP j1Pw3DNcLrhDz2l0i+xHYBHwqduv6K15mpDHkpUzDnLC51l62+hXe0QfLTR0aY0365yo 1MdA== X-Forwarded-Encrypted: i=1; AJvYcCWj1IkPXfc8EUm0yQxYTZHInmcYBxPOYdHk6V9psK0hUf/3fi607Tnhf0UIDBlciK4Y/UzovGlENYwF@gnu.org X-Gm-Message-State: AOJu0Yw+rFe+jGpimAxp9pLRhbmaZ8lmI9U5BfjQ8dL+/JAKoM8Vmog2 At0x3pavOJJxhFDm3gjdPrDV4TxIqZfC2jMc1OH3X5RDnx7DSvettG8IKg== X-Google-Smtp-Source: AGHT+IE0CTZPFQ7l70qqupYs8v2lJXU+iv2MSgo0zbRvA8vZtBaf/1hFip+mTWnsFGMf3xAzKqbp3w== X-Received: by 2002:a5d:4cc6:0:b0:371:8c06:82ee with SMTP id ffacd0b85a97d-374cd0d4572mr6670155f8f.49.1725454768272; Wed, 04 Sep 2024 05:59:28 -0700 (PDT) Received: from lili ([2a01:e0a:59b:9120:b32f:9ceb:a68d:c5bf]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-42c819e356asm117334365e9.42.2024.09.04.05.59.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Sep 2024 05:59:27 -0700 (PDT) From: Simon Tournier To: Ludovic =?utf-8?Q?Court=C3=A8s?= , guix-devel Subject: Re: =?utf-8?Q?=E2=80=98core-updates=E2=80=99?= is gone; long live =?utf-8?Q?=E2=80=98core-packages-team=E2=80=99!?= In-Reply-To: <87le0cj13e.fsf@inria.fr> References: <87le0cj13e.fsf@inria.fr> Date: Wed, 04 Sep 2024 14:58:37 +0200 Message-ID: <87v7zby3r6.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=2a00:1450:4864:20::42a; envelope-from=zimon.toutoune@gmail.com; helo=mail-wr1-x42a.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: guix-devel-bounces+larch=yhetil.org@gnu.org X-Migadu-Country: US X-Migadu-Flow: FLOW_IN X-Migadu-Queue-Id: 22F38175DC X-Migadu-Scanner: mx13.migadu.com X-Migadu-Spam-Score: -9.80 X-Spam-Score: -9.80 X-TUID: JVPP9gkmdH4A Hi Ludo, all, On Sat, 31 Aug 2024 at 15:03, Ludovic Court=C3=A8s wrote: > To reduce world rebuilds, perhaps we=E2=80=99ll sometimes create =E2=80= =9Cmerge trains=E2=80=9D, > whereby we=E2=80=99ll merge, say, the branch upgrading CMake and that ung= rafting > ibus on top of =E2=80=98core-packages-team=E2=80=99, and then merge this = combination in > =E2=80=98master=E2=80=99. I do not see how the world rebuild would be reduced. Correct me if my understanding is misleading, the branch =E2=80=99core-upda= tes=E2=80=99 had been extended to more than core packages because the project had (has?) not enough CPU power for continuously building any change with a massive impact. Therefore, this =E2=80=99core-updates=E2=80=99 branch was built only every = X months in order to reduce the workload. As you perfectly know, all these grouped changes is a mess and it requires a lot of effort to stabilize. Hence, a very boring task=E2=80=A6 i.e., an arbitrary X. ;-) For sure, that=E2=80=99s a good idea to have a core-packages-team that focu= ses only on core packages =E2=80=93 the initial aim. :-) However, it does not address the issue with changes that have a massive impact. This new core-packages-team just becomes one of other branches that will also contain packages triggering world rebuilds. And the question is how do we manage that? > The key being: these branches will have been developed and > tested independently of one another by dedicated teams, and the merge > train will be a mere formality. In this picture of =E2=80=9Cmerge train=E2=80=9D, the CI workload and world= rebuilds will increase, no? Consider the two teams: science and this new core-packages. Then science takes care of openblas (6000+ dependent packages) and core-packages of grep (6000+ dependent packages). When science team wants to merge to master, they ask for a rebuild. When core-packages team wants to merge to master, they ask for a rebuild. Therefore, we have two world rebuilds. Or the both teams need to synchronize and thus the branches are indeed developed independently but not tested independently. Are they? Well, my understanding of =E2=80=9Cmerge train=E2=80=9D is an automatic syn= chronization workflow: science team does not ask for a rebuild but ask for a merge and core-packages team idem =E2=80=93 so they cannot evaluate their impact = and what needs to be fixed by their changes =E2=80=93 then both merges are the = two wagons of the train. Well, it=E2=80=99s not clear for me what CI builds: f= irst one wagon then the two wagons? Or first the two wagons and then what? Aside, the core-packages or science merges might have an impact to packages tracked by other teams. Other said, fixes unrelated to their scope. Somehow, it requires that other teams also follow =E2=80=93 it mean= s a kind of synchronization. That=E2=80=99s why core-updates became the big catch-all and complex mess. That=E2=80=99s just a poor man way to synchronize: first it minimizes the n= umber of world rebuilds because they are more or less manually triggered, and second it eases to have fixes unrelated to core changes. For sure, I agree that having more branches will help to reduce the complexity of merging core-updates-like changes. However, it=E2=80=99s not clear how it will work in practise. Especially, the number of world rebuilds will increase, IMHO. Well, all in all, I understand the =E2=80=9Cmerge train=E2=80=9D story with= =E2=80=9Cregular=E2=80=9D branches. But I do not see how it would work for different branches that trigger world rebuilds. Could you detail a bit your perspective with =E2=80=9Cmerge train=E2=80=9D?= Or the workflow for merging branches that contains world rebuild changes. I know about [1] but I am lacking imagination for applying it to our situation. Because, if we have 3 merge requests (A, B and C), the =E2=80=9Cmerge train=E2=80=9D workflow implies the pipeline is run 3 times: with A on top of master, with A+B (B on the top of A on the top of master), with A+B+C (C on the top of B on the top of A on the top of master). If A+B fails then B is removed from the train and the pipeline runs A+C. And you can also consider if A fails then B and B+C is built. The =E2=80=9Cmerge train=E2=80=9D workflow makes sens when the number of ch= anges in A, B and C is high and thus it helps to keep the merges under control. However, one implicit assumption is a low-cost pipeline, IMHO. Bah the question how to merge different branches containing world rebuilds without the big catch-all old core-updates branch is not addressed under the constraint of reducing as much as possible the number of world rebuilds, for what my opinion is worth. Cheers, simon 1: https://docs.gitlab.com/ee/ci/pipelines/merge_trains.html PS: My understanding of Gitlab Merge Train selling point is that several pipelines are run in parallel. How to waste power for increasing the throughput?