Naming “build train” instead of “merge train”?

[Simon Tournier <zimon.toutoune@gmail.com>]

Hi Ludo, all

On Fri, 06 Sep 2024 at 11:11, Ludovic Courtès <ludo@gnu.org> wrote:

> In the end, perhaps we’ll have to negotiate on a case-by-case basis.
> The important thing to me is: independent testing as much as possible,
> and well-defined responsibilities and scope for the people/teams
> engaging in such changes.

I agree.

My main concern is about the potential (excessive) unnecessary wasted
rebuilds.

We need a policy that clarifies how to start large rebuilds, which
implies cross the final line with a merge.


Let take the example from this another message [1] as one case.


On Fri, 06 Sep 2024 at 11:01, Ludovic Courtès <ludo@gnu.org> wrote:

>   • Attach to a “merge train”.  For instance, assume there’s a branch
>     changing ‘gsl’: this is totally unrelated to ‘ffmpeg’ but it also
>     triggers a lot of rebuilds.  We could tack that second branch on top
>     of the known-good ‘ffmpeg’ branch, and, once it’s all good, merge
>     that “train” into ‘master’.
>
> (To be clear, the term “merge train” originates from GitLab-CI and
> similar CI tool, which use it as a merge scheduling strategy:
> <https://docs.gitlab.com/ee/ci/pipelines/merge_trains.html>.  GitLab-CI
> can create merge trains automatically I believe, but in our case we’d do
> that manually, at least for now.)

If we consider this specific case, the “merge train” workflow means:

  a) the branch changing “gsl” is built, so 1473 rebuilds.

  b) the branch changing “gsl” and “ffmpeg” is also built in parallel, so
     521 rebuilds.

The “merge train” workflow also reads:

  i) the branch changing “ffmpeg” is built, so 521 rebuilds.

 ii) the branch changing “gsl” and “ffmpeg” is also built in parallel, so
     1473 rebuilds including all the 521 rebuilds of i).

Therefore, for each scenario, 521 builds are “wasted“, compared to the
optimal 1473 in this case.

I do not think it’s what you have in mind when speaking about “merge
train”.  Is it?

Maybe it points that “merge train” as described by Gitlab is not what we
need.  The “merge train” workflow runs *in parallel* the builds of
several branches and I am not convinced this is wanted for heavy
branches as it might happens.

Therefore in order to avoid some confusion, maybe we could avoid the
name “merge train” and use another name for the strategy we are
discussing and we would like to implement.  For instance “rebase train”
or “build train”.  Or yet another name. :-)

Let try to not waste resource.  I think we could have a policy when a
branch contains “heavy” rebuilds.  Because we cannot continuously
rebuild the world. ;-)

Somehow, the team of this “heavy” rebuild branch says: “hey build train
of branch X starts soon”, meaning the team is ready and the state of the
branch X looks good so it require to set up the CI, i.e., the team is
going to trigger more than 1000 rebuilds.  The message “build train of
branch X starts soon” is sent to guix-devel and one week is let to the
other teams to rebase their own branch onto branch X, if they have
something to build, are ready, etc.

The team who asked the “build train” is responsible for crossing the
final line (the merge to master) and also responsible to synchronize
with the wagons, e.g., by sending some updates about how is going the
world rebuild, when the final merge is planned, etc.

If no one takes this train, no big deal because a next train is probably
going to start soon. ;-)

WDYT?

Cheers,
simon

--8<---------------cut here---------------start------------->8---
$ guix refresh -l gsl
   | cut -d':' -f2 | tr ' ' '\n' | tail -n +2 | sort | uniq > gsl.deps
$ for i in $(seq 2 7); do guix refresh -l ffmpeg@$i \
   | cut -d':' -f2 | tr ' ' '\n' | tail -n +2 ;done | sort | uniq > ffmpeg.deps

$ wc -l gsl.deps ffmpeg.deps 
 1473 gsl.deps
  521 ffmpeg.deps
 1994 total

$ for line in $(cat ffmpeg.deps); do grep -n ^$line gsl.deps ;done | wc -l
521
--8<---------------cut here---------------end--------------->8---

PS: Please note the number here are not true the number of rebuilds but
    the number of packages that triggers all the necessary rebuilds.

1: Re: ‘core-updates’ is gone; long live ‘core-packages-team’!
Ludovic Courtès <ludo@gnu.org>
Fri, 06 Sep 2024 11:01:46 +0200
id:87h6at2m11.fsf@gnu.org
https://lists.gnu.org/archive/html/guix-devel/2024-09
https://yhetil.org/guix/87h6at2m11.fsf@gnu.org