Discussion notes on releases and branches

Discussion notes on releases and branches

[Andreas Enge]

[-- Attachment #1: Type: text/plain, Size: 126 bytes --]

Attached are the notes of our discussion at the Guix Days concerning
releases, branches, teams and related matters.

Andreas


[-- Attachment #2: Release.txt --]
[-- Type: text/plain, Size: 5024 bytes --]

Releases, branches, teams

Problem: It took us 1.5 years for the last release, and the coreupdates
branch has been open for at least as much.


RELEASES

Why do we make releases?
- How else would people install Guix?
- Quality assurance: Fix the installer before release.
- Regular points for translation.
- Communication and publicity.

- Releases are separate from branches; the release team should pick
  a commit and spin off a release branch from there.
- It is possible to have long term support release, mixed with
  more frequent intermediate releases; example from another project:
  "big" release every year, "smaller" releases every 4 months.
- Stable branches are a frequent user request!
- A release schedule for a release every 4 to 6 months sounds
  reasonable.
- Mathieu, Simon, Julien and Andreas volunteer to be members of a release
  team. The idea would be to change the team for each release, with an
  overlap of about half the people: everyone would have "two terms".
  Such a team could commit on a release date and coordinate contributors,
  keep track of the status, ping people fixing bugs etc. Work should
  start immediately after a release to spot problematic areas (poorly
  supported architectures...) and big goals for the next release.
  It could also coordinate with other teams on planned big changes.
  Communication is key! For instance send a status update once per week.
  At the same time, the team should lower their operational involvement
  to avoid burn-out.


BRANCHES

Suggestion:
- Spin off a stable branch from master with security fixes, maybe important
  backports; after 6 months, branch a new stable branch from master; this
  is almost like a release, but continued into some future.

Counter-suggestion:
- Create branches with a few patches or patchsets; in any case with
  a "semantic" description of the changes. The branches could be
  shortlived. Feature branches are one incarnation of the concept.
- The numerical criteria for staging and core-updates is outdated:
  Even non-core packages may create an enormous number of rebuilds.
- Negative point: There is a risk of churn, by not regrouping world-
  rebuilding changes - but two non related world rebuilding changes
  might be difficult to review.

Before creating new branches, we need to clarify how the old branches
are handled!

- Smaller branches could be taken care of by dedicated persons
  responsible for pushing them forward. For instance by teams.
- Some people already do this for a feature branch on their local
  machine for medium-sized updates (ocaml), or even on ci (haskell,
  kernel updates).
- Branch creators should fix a goal and tentative timeline.
- We need a mapping between branches and maintainers responsible
  for the merge. This could be a team leader, if such a role is created.
  A wiki could be used to keep track of the branches.
- There is discussion whether we need a core-updates branch.
  Core updates concern the toolchain, build phase changes, everything
  that has big ramifications all over the system. It would be important
  to not have several "parallel" branches with related (for instance,
  but not exclusively, core-update) changes, which in fact should come
  one after the other. Either they could be collected in one branch,
  or would require coordination of branch creation (inside a team, say).
- "Merge trains" of gitlab are mentioned, as a way of merging several
  branches at the same time.
- Grafts can also be handled in feature branches: The graft is applied
  to master; the graft is applied to a different branch, directly followed
  by an ungrafting commit and an update of the corresponding package;
  once the branch is built it is merged back to master.
- Minor drawback: If qa has treated a world-rebuilding patch, the
  substitutes will be available on bordeaux, but not on berlin; people
  who have installed a long time ago and not authorised bordeaux might
  be affected. If there are complaints, they can be handled on the
  mailing list.
- Moving fast poses problems for non-x86 architectures, but the build farm
  situation has improved for aarch64 and armhf sufficiently to keep up
  with master. Handling feature branches remains an unsolved problem.
- Currently there is a cap on qa, only patches with at most 300 dependents
  are treated. This cap could be increased. Or it could be weighed with
  the build times of the packages.


TEAMS

- Issues could be tagged more often with responsible teams, or with
  severity information (blocking bug or not).
- Each module should be covered by a team; otherwise it would be
  difficult to get important updates through a feature branches.


ARCHITECTURES

- We distinguish in the manual between available architectures that are
  "supported" or "unsupported". It would make sense to have a category
  in-between, with architectures that are supported in principle, but may
  be low on substitute availability.
- We could use the "supported-systems" fields in packages more liberally.

Moving forward with teams and feature branches (was: Discussion notes on releases and branches)

[Josselin Poiret]

[-- Attachment #1: Type: text/plain, Size: 4245 bytes --]

Hi Andreas and thanks for taking the notes during the discussion!

I think what came out of that discussion was very interesting and that
it would greatly help the scaling problems Guix is facing as well as
maintainer/contributor burnout, but that also means that we need to
create some actionnable plan out of it.  The window before the eventual
core-updates merge would be the best time for it, IMO!

For that plan to materialize, the best would be to agree on 1) what we
want the new workflow to be, and 2) how to get there. To start the
discussion, here's what I got out of the discussion (and is probably
opinionated):


The envisioned goal state would be to stop relying on
staging/core-updates to manage non-trivial changes to Guix, but rather
have specific branches that could be merged way faster, because they are
focused on specific features that people can feel responsible for,
instead of a hodgepodge of unrelated patches that no one wants to
debug. These feature branches would be managed by teams, with one person
from the team being responsible for the feature and it getting merged
into master. Those features could be ephemeral or long-running and each
team would be free to organize around them as they see fit, since e.g. a
Stack upgrade is very different from a Mesa upgrade, or a
gnu-build-system change.

With the improved CI/QA we have now, we shouldn't have to worry too much
since substitutes will be available right away (we should weigh what's
acceptable for people using --no-substitutes vs. keeping software more
up-to-date though, but my opinion is that in favor of the latter).

Teams should thus better document how they work and what they do, either
in the Guix documentation or a specific wiki/manual that would be
maintained by team members themselves. Having all information in one
place would help outsiders find their way around the inner workings of
the project. Maintaining roadmaps for each team, with who's working on
them and where that work can be found would also lure unsuspecting
bystanders into working on Guix features, as well as structure the
future of Guix; it shouldn't be too codified though, since we don't want
to further burden precious team members. One nice thing about
documenting all of this using a manual is that the changes go through
the guix-patches ML, so that they can be discussed and recorded for the
public to see.

Note that this change would also mean that contributors are no longer
responsible for gauging whether a patch belongs on master/staging/c-u,
but rather the reviewer/committer.  Also, as a nice side-effect of the
change, I can see people getting interested in joining teams because
they structure longer-term effort that's put into Guix, not just because
they've been maintaining python-foo-for-bar for 5 years against their
will.

Regarding releases: a release team would have to keep in touch with what
the other teams are doing, make some sort of periodic status report, and
set deadlines such as feature freezes before preparing a new release.


For a potential roadmap (doesn't have to be sequential):
1. Document this workflow in the manual, in a dedicated node, with a
   rationale as well.  One thing worth mentioning would be how to handle
   grafting/ungrafting now.  Also remove the staging/core-updates
   criterion.
2. Teams start organizing around the features they are currently working
   on, and document them accordingly. They can also draft how they work,
   what they do and their roadmap if they wish.
3. CI/QA gets examined and updated to support the new workflow. We test
   it out with a sample feature.
4. staging merge happens, and the branch gets deleted.
5. core-updates merge happens, and the branch gets deleted or
   repurposed (up to the core team).


I bet I forgot a bunch of things, but at least we get the ball rolling!

WDYT?

NB: Just noticed there is no tooling/infrastructure team, this would
probably be a good idea, to publicize the incredible work that is being
put into all of it (mumi/cuirass/qa front page/the data service), as
well as bring in some new souls and document how they all fit together.

Best,
-- 
Josselin Poiret

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 682 bytes --]

Re: Moving forward with teams and feature branches (was: Discussion notes on releases and branches)

[Andreas Enge]

Hello Josselin,

Am Sun, Feb 12, 2023 at 10:13:35PM +0100 schrieb Josselin Poiret:
> For a potential roadmap (doesn't have to be sequential):
> 1. Document this workflow in the manual, in a dedicated node, with a
>    rationale as well.  One thing worth mentioning would be how to handle
>    grafting/ungrafting now.  Also remove the staging/core-updates
>    criterion.
> 2. Teams start organizing around the features they are currently working
>    on, and document them accordingly. They can also draft how they work,
>    what they do and their roadmap if they wish.
> 3. CI/QA gets examined and updated to support the new workflow. We test
>    it out with a sample feature.
> 4. staging merge happens, and the branch gets deleted.
> 5. core-updates merge happens, and the branch gets deleted or
>    repurposed (up to the core team).

This (and the other points you wrote) sound good to me.
I would start with 4. and 5., and in parallel consider 1. to 3.

Andreas

Time for RFC? (was Re: Moving forward with teams and feature branches (was: Discussion notes on releases and branches))

[zimoun]

Hi,

On Sun, 12 Feb 2023 at 22:13, Josselin Poiret <dev@jpoiret.xyz> wrote:

> 1. Document this workflow in the manual, in a dedicated node, with a
>    rationale as well.  One thing worth mentioning would be how to handle
>    grafting/ungrafting now.  Also remove the staging/core-updates
>    criterion.

Maybe it is also time for “RFC” as discussed earlier.  It would fit this
item #1.

        Time for a request-for-comments process?
        id:87cznqb1sl.fsf@inria.fr
        Wed, 27 Oct 2021 23:22:50 +0200
        https://yhetil.org/guix/87cznqb1sl.fsf@inria.fr

Some examples of other projects.

https://github.com/NixOS/rfcs
https://peps.python.org/pep-0000/
https://github.com/rust-lang/rfcs

Cheers,
simon

Re: Moving forward with teams and feature branches (was: Discussion notes on releases and branches)

[Andreas Enge]

Am Sun, Feb 12, 2023 at 10:13:35PM +0100 schrieb Josselin Poiret:
> 4. staging merge happens, and the branch gets deleted.

I failed to compile my profile on staging, since rust-rav1e, a dependency
of ffmpeg, failed to build; see bug #61475.

Andreas

Re: Moving forward with teams and feature branches (was: Discussion notes on releases and branches)

[Leo Famulari]

On Mon, Feb 13, 2023 at 03:07:58PM +0100, Andreas Enge wrote:
> Am Sun, Feb 12, 2023 at 10:13:35PM +0100 schrieb Josselin Poiret:
> > 4. staging merge happens, and the branch gets deleted.
> 
> I failed to compile my profile on staging, since rust-rav1e, a dependency
> of ffmpeg, failed to build; see bug #61475.

If this can't be fixed quickly, my recommendation would be to remove the
dependency.

Rust-rav1e is an AV1 video encoder, but FFmpeg also has this
functionality via libaom, which is the reference implementation and
considered to be very high quality.

Release (was Re: Discussion notes on releases and branches)

[Simon Tournier]

Hi,

Thanks Andreas for the notes. :-)

On Thu, 09 Feb 2023 at 13:19, Andreas Enge <andreas@enge.fr> wrote:

> - A release schedule for a release every 4 to 6 months sounds
>   reasonable.
> - Mathieu, Simon, Julien and Andreas volunteer to be members of a release

Ludo was volunteer for helping, if I remember correctly. ;-)

First we should start more or less now for the next release. :-)

Second, it was not clear and is still not clear for me when the branch
is created, if a core-updates branch is merged before, etc.  Well, I do
not remember we discussed the workflow.  Do we?  If yes, what was the
consensus?


Cheers,
simon

Rust team branch (was Re: Discussion notes on releases and branches)

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 4448 bytes --]

On Thu, Feb 09, 2023 at 01:19:28PM +0100, Andreas Enge wrote:
> Attached are the notes of our discussion at the Guix Days concerning
> releases, branches, teams and related matters.
> 
> Andreas
> 

> BRANCHES
> 
> Suggestion:
> - Spin off a stable branch from master with security fixes, maybe important
>   backports; after 6 months, branch a new stable branch from master; this
>   is almost like a release, but continued into some future.
> 
> Counter-suggestion:
> - Create branches with a few patches or patchsets; in any case with
>   a "semantic" description of the changes. The branches could be
>   shortlived. Feature branches are one incarnation of the concept.
> - The numerical criteria for staging and core-updates is outdated:
>   Even non-core packages may create an enormous number of rebuilds.
> - Negative point: There is a risk of churn, by not regrouping world-
>   rebuilding changes - but two non related world rebuilding changes
>   might be difficult to review.
> 
> Before creating new branches, we need to clarify how the old branches
> are handled!
> 
> - Smaller branches could be taken care of by dedicated persons
>   responsible for pushing them forward. For instance by teams.
> - Some people already do this for a feature branch on their local
>   machine for medium-sized updates (ocaml), or even on ci (haskell,
>   kernel updates).
> - Branch creators should fix a goal and tentative timeline.
> - We need a mapping between branches and maintainers responsible
>   for the merge. This could be a team leader, if such a role is created.
>   A wiki could be used to keep track of the branches.
> - There is discussion whether we need a core-updates branch.
>   Core updates concern the toolchain, build phase changes, everything
>   that has big ramifications all over the system. It would be important
>   to not have several "parallel" branches with related (for instance,
>   but not exclusively, core-update) changes, which in fact should come
>   one after the other. Either they could be collected in one branch,
>   or would require coordination of branch creation (inside a team, say).
> - "Merge trains" of gitlab are mentioned, as a way of merging several
>   branches at the same time.
> - Grafts can also be handled in feature branches: The graft is applied
>   to master; the graft is applied to a different branch, directly followed
>   by an ungrafting commit and an update of the corresponding package;
>   once the branch is built it is merged back to master.
> - Minor drawback: If qa has treated a world-rebuilding patch, the
>   substitutes will be available on bordeaux, but not on berlin; people
>   who have installed a long time ago and not authorised bordeaux might
>   be affected. If there are complaints, they can be handled on the
>   mailing list.
> - Moving fast poses problems for non-x86 architectures, but the build farm
>   situation has improved for aarch64 and armhf sufficiently to keep up
>   with master. Handling feature branches remains an unsolved problem.
> - Currently there is a cap on qa, only patches with at most 300 dependents
>   are treated. This cap could be increased. Or it could be weighed with
>   the build times of the packages.
> 
> 
> TEAMS
> 
> - Issues could be tagged more often with responsible teams, or with
>   severity information (blocking bug or not).
> - Each module should be covered by a team; otherwise it would be
>   difficult to get important updates through a feature branches.

On behalf of the Rust Team™¹ we'd like to check our rust source tarballs
for any hidden binaries² and to do a mass upgrade of many of the crates.
Currently there are many crates queued up in the staging branch but we'd
like to pull them out and run a rust-team branch.

As a project we haven't setup anything for starting the team-based
branches and upgrades, and the Rust Team volunteers to go first.

Although the rust team would consider adding librsvg (and
python-cryptography) to our list of packages, we'd like to not touch it
this round, to keep it "small" (as far as rust goes) as a test.


¹ Help wanted
² https://issues.guix.gnu.org/61352


-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Rust team branch

[Andreas Enge]

Am Tue, Feb 14, 2023 at 12:14:04PM +0200 schrieb Efraim Flashner:
> On behalf of the Rust Team™¹ we'd like to check our rust source tarballs
> for any hidden binaries² and to do a mass upgrade of many of the crates.
> Currently there are many crates queued up in the staging branch but we'd
> like to pull them out and run a rust-team branch.

By "pull out" you mean revert them in staging and apply them on a separate
branch? That would also delay #61475 and maybe ease merging of the staging
branch.

Would it make sense to wait with the creation of the feature branch
until after the core-updates merge?

> As a project we haven't setup anything for starting the team-based
> branches and upgrades, and the Rust Team volunteers to go first.
> ¹ Help wanted

Splendid! I suppose that increasing the team size beyond 1 would be
a first important goal, call out to volunteers!

Andreas

Re: Rust team branch

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1430 bytes --]

On Tue, Feb 14, 2023 at 05:36:13PM +0100, Andreas Enge wrote:
> Am Tue, Feb 14, 2023 at 12:14:04PM +0200 schrieb Efraim Flashner:
> > On behalf of the Rust Team™¹ we'd like to check our rust source tarballs
> > for any hidden binaries² and to do a mass upgrade of many of the crates.
> > Currently there are many crates queued up in the staging branch but we'd
> > like to pull them out and run a rust-team branch.
> 
> By "pull out" you mean revert them in staging and apply them on a separate
> branch? That would also delay #61475 and maybe ease merging of the staging
> branch.

I was thinking more of cherry-picking them into a branch, not
necessarily reverting them on staging.

> Would it make sense to wait with the creation of the feature branch
> until after the core-updates merge?

Normally I'd say yes, but it might be a while until we can get
core-updates merged in.

> > As a project we haven't setup anything for starting the team-based
> > branches and upgrades, and the Rust Team volunteers to go first.
> > ¹ Help wanted
> 
> Splendid! I suppose that increasing the team size beyond 1 would be
> a first important goal, call out to volunteers!
> 
> Andreas
> 
> 

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Rust team branch

[Andreas Enge]

Am Tue, Feb 14, 2023 at 10:07:12PM +0200 schrieb Efraim Flashner:
> > By "pull out" you mean revert them in staging and apply them on a separate
> > branch? That would also delay #61475 and maybe ease merging of the staging
> > branch.
> I was thinking more of cherry-picking them into a branch, not
> necessarily reverting them on staging.

Okay. But it would be nice then to revert at least as much in staging
that the branch builds and can be merged.

Andreas

Re: Rust team branch (was Re: Discussion notes on releases and branches)

[Katherine Cox-Buday]

Efraim Flashner <efraim@flashner.co.il> writes:

> As a project we haven't setup anything for starting the team-based
> branches and upgrades, and the Rust Team volunteers to go first.

Leo has attempted[1] to setup a branch for Go, but we're waiting to hear from
sysadmins on why there's a build error.

[1] https://lists.gnu.org/archive/html/guix-devel/2023-01/msg00131.html

-- 
Katherine

Re: Rust team branch (was Re: Discussion notes on releases and branches)

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 834 bytes --]

On Tue, Feb 14, 2023 at 09:36:46AM -0700, Katherine Cox-Buday wrote:
> Efraim Flashner <efraim@flashner.co.il> writes:
> 
> > As a project we haven't setup anything for starting the team-based
> > branches and upgrades, and the Rust Team volunteers to go first.
> 
> Leo has attempted[1] to setup a branch for Go, but we're waiting to hear from
> sysadmins on why there's a build error.
> 
> [1] https://lists.gnu.org/archive/html/guix-devel/2023-01/msg00131.html

Second is also good :)

Also I don't think there's a lot of overlap between go and rust so we
can probably have both going at the same time.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Rust team branch (was Re: Discussion notes on releases and branches)

[Katherine Cox-Buday]

On 2/14/23 1:08 PM, Efraim Flashner wrote:

> Also I don't think there's a lot of overlap between go and rust so we
> can probably have both going at the same time.

Oh, yes, I didn't mean to imply that the two were related, or 
conflicted. I just wanted to point it out in case it was of any help.

Best of luck to the rust team!

--
Katherine

Re: Discussion notes on releases and branches

[Felix Lechner via Development of GNU Guix and the GNU System distribution.]

Hi Andreas,

On Thu, Feb 9, 2023 at 4:20 AM Andreas Enge <andreas@enge.fr> wrote:
>
> [excerpt from the attachment follows]
>
> - Smaller branches could be taken care of by dedicated persons
> [...]
> - Branch creators should fix a goal and tentative timeline.
> - We need a mapping between branches and maintainers responsible
>   for the merge.

Instead of dividing similar tasks among many people with different
priorities, I would appoint a "feature branch manager" (or perhaps,
chief pruner).

That person's job would be to keep an eye on the number of feature
branches on Savannah so they do not grow stale, or out of control.

Kind regards
Felix Lechner

Re: Discussion notes on releases and branches

[Leo Famulari]

On Fri, Mar 17, 2023 at 08:24:44AM -0700, Felix Lechner via Development of GNU Guix and the GNU System distribution. wrote:
> Instead of dividing similar tasks among many people with different
> priorities, I would appoint a "feature branch manager" (or perhaps,
> chief pruner).
> 
> That person's job would be to keep an eye on the number of feature
> branches on Savannah so they do not grow stale, or out of control.

There should be someone like this, although I suspect this particular
role could be informally performed by the people who are paying
attention to loads on the build farm at any given time. Generally, we
don't have success appointing people to jobs; rather, they volunteer or
even emerge non-explicitly.