Discussion notes on releases and branches

Discussion notes on releases and branches

[Andreas Enge]

[-- Attachment #1: Type: text/plain, Size: 126 bytes --]

Attached are the notes of our discussion at the Guix Days concerning
releases, branches, teams and related matters.

Andreas


[-- Attachment #2: Release.txt --]
[-- Type: text/plain, Size: 5024 bytes --]

Releases, branches, teams

Problem: It took us 1.5 years for the last release, and the coreupdates
branch has been open for at least as much.


RELEASES

Why do we make releases?
- How else would people install Guix?
- Quality assurance: Fix the installer before release.
- Regular points for translation.
- Communication and publicity.

- Releases are separate from branches; the release team should pick
  a commit and spin off a release branch from there.
- It is possible to have long term support release, mixed with
  more frequent intermediate releases; example from another project:
  "big" release every year, "smaller" releases every 4 months.
- Stable branches are a frequent user request!
- A release schedule for a release every 4 to 6 months sounds
  reasonable.
- Mathieu, Simon, Julien and Andreas volunteer to be members of a release
  team. The idea would be to change the team for each release, with an
  overlap of about half the people: everyone would have "two terms".
  Such a team could commit on a release date and coordinate contributors,
  keep track of the status, ping people fixing bugs etc. Work should
  start immediately after a release to spot problematic areas (poorly
  supported architectures...) and big goals for the next release.
  It could also coordinate with other teams on planned big changes.
  Communication is key! For instance send a status update once per week.
  At the same time, the team should lower their operational involvement
  to avoid burn-out.


BRANCHES

Suggestion:
- Spin off a stable branch from master with security fixes, maybe important
  backports; after 6 months, branch a new stable branch from master; this
  is almost like a release, but continued into some future.

Counter-suggestion:
- Create branches with a few patches or patchsets; in any case with
  a "semantic" description of the changes. The branches could be
  shortlived. Feature branches are one incarnation of the concept.
- The numerical criteria for staging and core-updates is outdated:
  Even non-core packages may create an enormous number of rebuilds.
- Negative point: There is a risk of churn, by not regrouping world-
  rebuilding changes - but two non related world rebuilding changes
  might be difficult to review.

Before creating new branches, we need to clarify how the old branches
are handled!

- Smaller branches could be taken care of by dedicated persons
  responsible for pushing them forward. For instance by teams.
- Some people already do this for a feature branch on their local
  machine for medium-sized updates (ocaml), or even on ci (haskell,
  kernel updates).
- Branch creators should fix a goal and tentative timeline.
- We need a mapping between branches and maintainers responsible
  for the merge. This could be a team leader, if such a role is created.
  A wiki could be used to keep track of the branches.
- There is discussion whether we need a core-updates branch.
  Core updates concern the toolchain, build phase changes, everything
  that has big ramifications all over the system. It would be important
  to not have several "parallel" branches with related (for instance,
  but not exclusively, core-update) changes, which in fact should come
  one after the other. Either they could be collected in one branch,
  or would require coordination of branch creation (inside a team, say).
- "Merge trains" of gitlab are mentioned, as a way of merging several
  branches at the same time.
- Grafts can also be handled in feature branches: The graft is applied
  to master; the graft is applied to a different branch, directly followed
  by an ungrafting commit and an update of the corresponding package;
  once the branch is built it is merged back to master.
- Minor drawback: If qa has treated a world-rebuilding patch, the
  substitutes will be available on bordeaux, but not on berlin; people
  who have installed a long time ago and not authorised bordeaux might
  be affected. If there are complaints, they can be handled on the
  mailing list.
- Moving fast poses problems for non-x86 architectures, but the build farm
  situation has improved for aarch64 and armhf sufficiently to keep up
  with master. Handling feature branches remains an unsolved problem.
- Currently there is a cap on qa, only patches with at most 300 dependents
  are treated. This cap could be increased. Or it could be weighed with
  the build times of the packages.


TEAMS

- Issues could be tagged more often with responsible teams, or with
  severity information (blocking bug or not).
- Each module should be covered by a team; otherwise it would be
  difficult to get important updates through a feature branches.


ARCHITECTURES

- We distinguish in the manual between available architectures that are
  "supported" or "unsupported". It would make sense to have a category
  in-between, with architectures that are supported in principle, but may
  be low on substitute availability.
- We could use the "supported-systems" fields in packages more liberally.

Moving forward with teams and feature branches (was: Discussion notes on releases and branches)

[Josselin Poiret]

[-- Attachment #1: Type: text/plain, Size: 4245 bytes --]

Hi Andreas and thanks for taking the notes during the discussion!

I think what came out of that discussion was very interesting and that
it would greatly help the scaling problems Guix is facing as well as
maintainer/contributor burnout, but that also means that we need to
create some actionnable plan out of it.  The window before the eventual
core-updates merge would be the best time for it, IMO!

For that plan to materialize, the best would be to agree on 1) what we
want the new workflow to be, and 2) how to get there. To start the
discussion, here's what I got out of the discussion (and is probably
opinionated):


The envisioned goal state would be to stop relying on
staging/core-updates to manage non-trivial changes to Guix, but rather
have specific branches that could be merged way faster, because they are
focused on specific features that people can feel responsible for,
instead of a hodgepodge of unrelated patches that no one wants to
debug. These feature branches would be managed by teams, with one person
from the team being responsible for the feature and it getting merged
into master. Those features could be ephemeral or long-running and each
team would be free to organize around them as they see fit, since e.g. a
Stack upgrade is very different from a Mesa upgrade, or a
gnu-build-system change.

With the improved CI/QA we have now, we shouldn't have to worry too much
since substitutes will be available right away (we should weigh what's
acceptable for people using --no-substitutes vs. keeping software more
up-to-date though, but my opinion is that in favor of the latter).

Teams should thus better document how they work and what they do, either
in the Guix documentation or a specific wiki/manual that would be
maintained by team members themselves. Having all information in one
place would help outsiders find their way around the inner workings of
the project. Maintaining roadmaps for each team, with who's working on
them and where that work can be found would also lure unsuspecting
bystanders into working on Guix features, as well as structure the
future of Guix; it shouldn't be too codified though, since we don't want
to further burden precious team members. One nice thing about
documenting all of this using a manual is that the changes go through
the guix-patches ML, so that they can be discussed and recorded for the
public to see.

Note that this change would also mean that contributors are no longer
responsible for gauging whether a patch belongs on master/staging/c-u,
but rather the reviewer/committer.  Also, as a nice side-effect of the
change, I can see people getting interested in joining teams because
they structure longer-term effort that's put into Guix, not just because
they've been maintaining python-foo-for-bar for 5 years against their
will.

Regarding releases: a release team would have to keep in touch with what
the other teams are doing, make some sort of periodic status report, and
set deadlines such as feature freezes before preparing a new release.


For a potential roadmap (doesn't have to be sequential):
1. Document this workflow in the manual, in a dedicated node, with a
   rationale as well.  One thing worth mentioning would be how to handle
   grafting/ungrafting now.  Also remove the staging/core-updates
   criterion.
2. Teams start organizing around the features they are currently working
   on, and document them accordingly. They can also draft how they work,
   what they do and their roadmap if they wish.
3. CI/QA gets examined and updated to support the new workflow. We test
   it out with a sample feature.
4. staging merge happens, and the branch gets deleted.
5. core-updates merge happens, and the branch gets deleted or
   repurposed (up to the core team).


I bet I forgot a bunch of things, but at least we get the ball rolling!

WDYT?

NB: Just noticed there is no tooling/infrastructure team, this would
probably be a good idea, to publicize the incredible work that is being
put into all of it (mumi/cuirass/qa front page/the data service), as
well as bring in some new souls and document how they all fit together.

Best,
-- 
Josselin Poiret

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 682 bytes --]

Re: Moving forward with teams and feature branches (was: Discussion notes on releases and branches)

[Andreas Enge]

Hello Josselin,

Am Sun, Feb 12, 2023 at 10:13:35PM +0100 schrieb Josselin Poiret:
> For a potential roadmap (doesn't have to be sequential):
> 1. Document this workflow in the manual, in a dedicated node, with a
>    rationale as well.  One thing worth mentioning would be how to handle
>    grafting/ungrafting now.  Also remove the staging/core-updates
>    criterion.
> 2. Teams start organizing around the features they are currently working
>    on, and document them accordingly. They can also draft how they work,
>    what they do and their roadmap if they wish.
> 3. CI/QA gets examined and updated to support the new workflow. We test
>    it out with a sample feature.
> 4. staging merge happens, and the branch gets deleted.
> 5. core-updates merge happens, and the branch gets deleted or
>    repurposed (up to the core team).

This (and the other points you wrote) sound good to me.
I would start with 4. and 5., and in parallel consider 1. to 3.

Andreas

Release (was Re: Discussion notes on releases and branches)

[Simon Tournier]

Hi,

Thanks Andreas for the notes. :-)

On Thu, 09 Feb 2023 at 13:19, Andreas Enge <andreas@enge.fr> wrote:

> - A release schedule for a release every 4 to 6 months sounds
>   reasonable.
> - Mathieu, Simon, Julien and Andreas volunteer to be members of a release

Ludo was volunteer for helping, if I remember correctly. ;-)

First we should start more or less now for the next release. :-)

Second, it was not clear and is still not clear for me when the branch
is created, if a core-updates branch is merged before, etc.  Well, I do
not remember we discussed the workflow.  Do we?  If yes, what was the
consensus?


Cheers,
simon

Time for RFC? (was Re: Moving forward with teams and feature branches (was: Discussion notes on releases and branches))

[zimoun]

Hi,

On Sun, 12 Feb 2023 at 22:13, Josselin Poiret <dev@jpoiret.xyz> wrote:

> 1. Document this workflow in the manual, in a dedicated node, with a
>    rationale as well.  One thing worth mentioning would be how to handle
>    grafting/ungrafting now.  Also remove the staging/core-updates
>    criterion.

Maybe it is also time for “RFC” as discussed earlier.  It would fit this
item #1.

        Time for a request-for-comments process?
        id:87cznqb1sl.fsf@inria.fr
        Wed, 27 Oct 2021 23:22:50 +0200
        https://yhetil.org/guix/87cznqb1sl.fsf@inria.fr

Some examples of other projects.

https://github.com/NixOS/rfcs
https://peps.python.org/pep-0000/
https://github.com/rust-lang/rfcs

Cheers,
simon

bug#61475: Staging branch (was: Moving forward with teams and feature branches)

[Andreas Enge]

Am Sun, Feb 12, 2023 at 10:13:35PM +0100 schrieb Josselin Poiret:
> 4. staging merge happens, and the branch gets deleted.

I tried to build staging for my profile on x86_64, but it failed with
a dependency of ffmpeg, rust-rav1e-0.5.1. There is a newer version 0.6.3,
but I did not simply update it, since the package looks particularly
complicated, containing a phase:
         (add-after 'configure 'force-rust-edition-2018
           (lambda* (#:key vendor-dir #:allow-other-keys)
             ;; Force all the dependencies to not be higher than edition 2018.
             (with-fluids ((%default-port-encoding #f))
               (substitute* (find-files vendor-dir "Cargo.toml")
                 (("edition = \\\"2021\\\"") "edition = \"2018\"")))))
and many other changes.

If someone who knows rust could have a look and make a suggestion, that
would be great.

Andreas

Re: Moving forward with teams and feature branches (was: Discussion notes on releases and branches)

[Andreas Enge]

Am Sun, Feb 12, 2023 at 10:13:35PM +0100 schrieb Josselin Poiret:
> 4. staging merge happens, and the branch gets deleted.

I failed to compile my profile on staging, since rust-rav1e, a dependency
of ffmpeg, failed to build; see bug #61475.

Andreas

Rust team branch (was Re: Discussion notes on releases and branches)

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 4448 bytes --]

On Thu, Feb 09, 2023 at 01:19:28PM +0100, Andreas Enge wrote:
> Attached are the notes of our discussion at the Guix Days concerning
> releases, branches, teams and related matters.
> 
> Andreas
> 

> BRANCHES
> 
> Suggestion:
> - Spin off a stable branch from master with security fixes, maybe important
>   backports; after 6 months, branch a new stable branch from master; this
>   is almost like a release, but continued into some future.
> 
> Counter-suggestion:
> - Create branches with a few patches or patchsets; in any case with
>   a "semantic" description of the changes. The branches could be
>   shortlived. Feature branches are one incarnation of the concept.
> - The numerical criteria for staging and core-updates is outdated:
>   Even non-core packages may create an enormous number of rebuilds.
> - Negative point: There is a risk of churn, by not regrouping world-
>   rebuilding changes - but two non related world rebuilding changes
>   might be difficult to review.
> 
> Before creating new branches, we need to clarify how the old branches
> are handled!
> 
> - Smaller branches could be taken care of by dedicated persons
>   responsible for pushing them forward. For instance by teams.
> - Some people already do this for a feature branch on their local
>   machine for medium-sized updates (ocaml), or even on ci (haskell,
>   kernel updates).
> - Branch creators should fix a goal and tentative timeline.
> - We need a mapping between branches and maintainers responsible
>   for the merge. This could be a team leader, if such a role is created.
>   A wiki could be used to keep track of the branches.
> - There is discussion whether we need a core-updates branch.
>   Core updates concern the toolchain, build phase changes, everything
>   that has big ramifications all over the system. It would be important
>   to not have several "parallel" branches with related (for instance,
>   but not exclusively, core-update) changes, which in fact should come
>   one after the other. Either they could be collected in one branch,
>   or would require coordination of branch creation (inside a team, say).
> - "Merge trains" of gitlab are mentioned, as a way of merging several
>   branches at the same time.
> - Grafts can also be handled in feature branches: The graft is applied
>   to master; the graft is applied to a different branch, directly followed
>   by an ungrafting commit and an update of the corresponding package;
>   once the branch is built it is merged back to master.
> - Minor drawback: If qa has treated a world-rebuilding patch, the
>   substitutes will be available on bordeaux, but not on berlin; people
>   who have installed a long time ago and not authorised bordeaux might
>   be affected. If there are complaints, they can be handled on the
>   mailing list.
> - Moving fast poses problems for non-x86 architectures, but the build farm
>   situation has improved for aarch64 and armhf sufficiently to keep up
>   with master. Handling feature branches remains an unsolved problem.
> - Currently there is a cap on qa, only patches with at most 300 dependents
>   are treated. This cap could be increased. Or it could be weighed with
>   the build times of the packages.
> 
> 
> TEAMS
> 
> - Issues could be tagged more often with responsible teams, or with
>   severity information (blocking bug or not).
> - Each module should be covered by a team; otherwise it would be
>   difficult to get important updates through a feature branches.

On behalf of the Rust Team™¹ we'd like to check our rust source tarballs
for any hidden binaries² and to do a mass upgrade of many of the crates.
Currently there are many crates queued up in the staging branch but we'd
like to pull them out and run a rust-team branch.

As a project we haven't setup anything for starting the team-based
branches and upgrades, and the Rust Team volunteers to go first.

Although the rust team would consider adding librsvg (and
python-cryptography) to our list of packages, we'd like to not touch it
this round, to keep it "small" (as far as rust goes) as a test.


¹ Help wanted
² https://issues.guix.gnu.org/61352


-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Rust team branch

[Andreas Enge]

Am Tue, Feb 14, 2023 at 12:14:04PM +0200 schrieb Efraim Flashner:
> On behalf of the Rust Team™¹ we'd like to check our rust source tarballs
> for any hidden binaries² and to do a mass upgrade of many of the crates.
> Currently there are many crates queued up in the staging branch but we'd
> like to pull them out and run a rust-team branch.

By "pull out" you mean revert them in staging and apply them on a separate
branch? That would also delay #61475 and maybe ease merging of the staging
branch.

Would it make sense to wait with the creation of the feature branch
until after the core-updates merge?

> As a project we haven't setup anything for starting the team-based
> branches and upgrades, and the Rust Team volunteers to go first.
> ¹ Help wanted

Splendid! I suppose that increasing the team size beyond 1 would be
a first important goal, call out to volunteers!

Andreas

Re: Rust team branch (was Re: Discussion notes on releases and branches)

[Katherine Cox-Buday]

Efraim Flashner <efraim@flashner.co.il> writes:

> As a project we haven't setup anything for starting the team-based
> branches and upgrades, and the Rust Team volunteers to go first.

Leo has attempted[1] to setup a branch for Go, but we're waiting to hear from
sysadmins on why there's a build error.

[1] https://lists.gnu.org/archive/html/guix-devel/2023-01/msg00131.html

-- 
Katherine

Re: Moving forward with teams and feature branches (was: Discussion notes on releases and branches)

[Leo Famulari]

On Mon, Feb 13, 2023 at 03:07:58PM +0100, Andreas Enge wrote:
> Am Sun, Feb 12, 2023 at 10:13:35PM +0100 schrieb Josselin Poiret:
> > 4. staging merge happens, and the branch gets deleted.
> 
> I failed to compile my profile on staging, since rust-rav1e, a dependency
> of ffmpeg, failed to build; see bug #61475.

If this can't be fixed quickly, my recommendation would be to remove the
dependency.

Rust-rav1e is an AV1 video encoder, but FFmpeg also has this
functionality via libaom, which is the reference implementation and
considered to be very high quality.

Re: Rust team branch

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1430 bytes --]

On Tue, Feb 14, 2023 at 05:36:13PM +0100, Andreas Enge wrote:
> Am Tue, Feb 14, 2023 at 12:14:04PM +0200 schrieb Efraim Flashner:
> > On behalf of the Rust Team™¹ we'd like to check our rust source tarballs
> > for any hidden binaries² and to do a mass upgrade of many of the crates.
> > Currently there are many crates queued up in the staging branch but we'd
> > like to pull them out and run a rust-team branch.
> 
> By "pull out" you mean revert them in staging and apply them on a separate
> branch? That would also delay #61475 and maybe ease merging of the staging
> branch.

I was thinking more of cherry-picking them into a branch, not
necessarily reverting them on staging.

> Would it make sense to wait with the creation of the feature branch
> until after the core-updates merge?

Normally I'd say yes, but it might be a while until we can get
core-updates merged in.

> > As a project we haven't setup anything for starting the team-based
> > branches and upgrades, and the Rust Team volunteers to go first.
> > ¹ Help wanted
> 
> Splendid! I suppose that increasing the team size beyond 1 would be
> a first important goal, call out to volunteers!
> 
> Andreas
> 
> 

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Rust team branch (was Re: Discussion notes on releases and branches)

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 834 bytes --]

On Tue, Feb 14, 2023 at 09:36:46AM -0700, Katherine Cox-Buday wrote:
> Efraim Flashner <efraim@flashner.co.il> writes:
> 
> > As a project we haven't setup anything for starting the team-based
> > branches and upgrades, and the Rust Team volunteers to go first.
> 
> Leo has attempted[1] to setup a branch for Go, but we're waiting to hear from
> sysadmins on why there's a build error.
> 
> [1] https://lists.gnu.org/archive/html/guix-devel/2023-01/msg00131.html

Second is also good :)

Also I don't think there's a lot of overlap between go and rust so we
can probably have both going at the same time.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Rust team branch (was Re: Discussion notes on releases and branches)

[Katherine Cox-Buday]

On 2/14/23 1:08 PM, Efraim Flashner wrote:

> Also I don't think there's a lot of overlap between go and rust so we
> can probably have both going at the same time.

Oh, yes, I didn't mean to imply that the two were related, or 
conflicted. I just wanted to point it out in case it was of any help.

Best of luck to the rust team!

--
Katherine

Re: Rust team branch

[Andreas Enge]

Am Tue, Feb 14, 2023 at 10:07:12PM +0200 schrieb Efraim Flashner:
> > By "pull out" you mean revert them in staging and apply them on a separate
> > branch? That would also delay #61475 and maybe ease merging of the staging
> > branch.
> I was thinking more of cherry-picking them into a branch, not
> necessarily reverting them on staging.

Okay. But it would be nice then to revert at least as much in staging
that the branch builds and can be merged.

Andreas

Re: Discussion notes on releases and branches

[Felix Lechner via Development of GNU Guix and the GNU System distribution.]

Hi Andreas,

On Thu, Feb 9, 2023 at 4:20 AM Andreas Enge <andreas@enge.fr> wrote:
>
> [excerpt from the attachment follows]
>
> - Smaller branches could be taken care of by dedicated persons
> [...]
> - Branch creators should fix a goal and tentative timeline.
> - We need a mapping between branches and maintainers responsible
>   for the merge.

Instead of dividing similar tasks among many people with different
priorities, I would appoint a "feature branch manager" (or perhaps,
chief pruner).

That person's job would be to keep an eye on the number of feature
branches on Savannah so they do not grow stale, or out of control.

Kind regards
Felix Lechner

Re: Discussion notes on releases and branches

[Leo Famulari]

On Fri, Mar 17, 2023 at 08:24:44AM -0700, Felix Lechner via Development of GNU Guix and the GNU System distribution. wrote:
> Instead of dividing similar tasks among many people with different
> priorities, I would appoint a "feature branch manager" (or perhaps,
> chief pruner).
> 
> That person's job would be to keep an eye on the number of feature
> branches on Savannah so they do not grow stale, or out of control.

There should be someone like this, although I suspect this particular
role could be informally performed by the people who are paying
attention to loads on the build farm at any given time. Generally, we
don't have success appointing people to jobs; rather, they volunteer or
even emerge non-explicitly.

bug#61475: Staging branch (was: Moving forward with teams and feature branches)

[Maxim Cournoyer]

Hi,

Andreas Enge <andreas@enge.fr> writes:

> Am Sun, Feb 12, 2023 at 10:13:35PM +0100 schrieb Josselin Poiret:
>> 4. staging merge happens, and the branch gets deleted.
>
> I tried to build staging for my profile on x86_64, but it failed with
> a dependency of ffmpeg, rust-rav1e-0.5.1. There is a newer version 0.6.3,
> but I did not simply update it, since the package looks particularly
> complicated, containing a phase:
>          (add-after 'configure 'force-rust-edition-2018
>            (lambda* (#:key vendor-dir #:allow-other-keys)
>              ;; Force all the dependencies to not be higher than edition 2018.
>              (with-fluids ((%default-port-encoding #f))
>                (substitute* (find-files vendor-dir "Cargo.toml")
>                  (("edition = \\\"2021\\\"") "edition = \"2018\"")))))
> and many other changes.
>
> If someone who knows rust could have a look and make a suggestion, that
> would be great.

Closing, both staging and the rust branch having been merged.

-- 
Thanks,
Maxim

On the quest for a new release model (was: Discussion notes on releases and branches)

[Cayetano Santos]

[-- Attachment #1: Type: text/plain, Size: 2154 bytes --]


Hi Guix,

Let me spin off last year thread on releases (flavored by an external to
the project point of view), with the goal to relaunch the debate on
releases.

Disclaimer: the rolling release model is great and more than enough for
me, and teams constitute an mportant step forward.

Disclaimer 2: guix-system is probably great, but using guix as a foreign
package manager is absolutely paramount, and would open the door to all
things guix, bringing lots of new users/contributors.

Now, any user in the world may install guix and upgrade to latest
version, sure. But.

First, for the current needs:

- distributions privilege and package guix releases (alpine, debian,
- arch, etc.), as guix itself does
- 1.4.0 dates back from 2 years ago
- in my experience, people tend to give guix a try, check obsolete sw,
  just give up, no matter the remaining advantages
- remote ci/cd images need to refer to a release for traceability
- apt-get install guix (1.4.0) && guix pull is a painful experience

Additionally, releases makes people talk about guix, give an overall
positive impression of the community, and are a good argument (and
publicity !) in favor of using it (unfortunate, but that’s the way it
goes). See emacs devel cycle, for example.

Now, for (a very naif) proposal (from a non contributor).

- simver[1] is the way to go
- devel as the branch for developments, master for releases and
  security/bug fixes
- major should follow core merges to devel
- minor should follow non-core teams merges
- patch fixes are backported to master

Yes, you guess [2] it.

Please, consider all the previous as an excuse to motivate the need for
releases, not as serious guidelines (I’m not the right person for that.)
For what I see around me, guix could easily be used by a larger audience
(foreign first, system then). Releases are a real requirement to that
goal.

Best,

[1] https://semver.org/
[2] https://nvie.com/posts/a-successful-git-branching-model/

--
Cayetano Santos
GnuPG Key:   https://meta.sr.ht/~csantosb.pgp
FingerPrint: CCB8 1842 F9D7 058E CD67 377A BF5C DF4D F6BF 6682

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 259 bytes --]

Re: On the quest for a new release model

[Ricardo Wurmus]

Hi,

Thanks for moving this discussion forward.  I do think we need much more
regular releases.

> - devel as the branch for developments, master for releases and
>   security/bug fixes

Changing the branching model is very difficult to do.  I think it is
better to ignore branches for now and focus on coming to an agreement
about more frequent releases, lest this discussion, too, ends up
reiterating "stable" branches and the finer points of release
maintenance.

> - major should follow core merges to devel
> - minor should follow non-core teams merges

I think this is a good idea to start with.  Releases are made a short
time after the core team branch is merged.  This would give us a new
release whenever e.g. the default GCC and glibc is bumped up.  We could
aim for a release two months after the merge to allow for minor fixes
after the merge.  I'm not sure if these merges should justify a new *major*
release, but I think it is good to have a new release then.

Not all team branch merges may justify a new release.  The r-team
branch, for example, usually contains just a couple hundred patch-level
package upgrades that are restricted to packages from CRAN and
Bioconductor.  It is only sometimes that the R version is increased or
the Bioconductor release version is changed --- only in those cases I
would consider it appropriate to bump up the Guix minor (or patch-level)
version number.

-- 
Ricardo

Re: On the quest for a new release model

[Suhail Singh]

Ricardo Wurmus <rekado@elephly.net> writes:

> Releases are made a short time after the core team branch is merged.
> This would give us a new release whenever e.g. the default GCC and
> glibc is bumped up.  We could aim for a release two months after the
> merge to allow for minor fixes after the merge.

I think agreeing on and following a release schedule is the crucial
part.  Both for how frequently core team branch is merged
(aspirationally) as well as how long (roughly) after the merge the
release is made.

I propose that we have a time-based release model rather than a
feature-based one (similar to the Linux kernel).  However, what's
perhaps even more important is to have a release model that we adhere
to.

-- 
Suhail

Re: On the quest for a new release model

[Greg Hogan]

On Fri, Dec 13, 2024 at 8:02 AM Suhail Singh <suhailsingh247@gmail.com> wrote:
>
> Ricardo Wurmus <rekado@elephly.net> writes:
>
> > Releases are made a short time after the core team branch is merged.
> > This would give us a new release whenever e.g. the default GCC and
> > glibc is bumped up.  We could aim for a release two months after the
> > merge to allow for minor fixes after the merge.
>
> I think agreeing on and following a release schedule is the crucial
> part.  Both for how frequently core team branch is merged
> (aspirationally) as well as how long (roughly) after the merge the
> release is made.
>
> I propose that we have a time-based release model rather than a
> feature-based one (similar to the Linux kernel).  However, what's
> perhaps even more important is to have a release model that we adhere
> to.

Having relatively recently asked [1] about a "1.5.0" release, I think
temver should be preferred over semver. What information has been
conveyed by Guix's version numbers beyond the symbolic "1.0"? Many
user-facing projects have adopted temporal versioning, including
NixOS.

We only need a release team and a documented release process. Releases
should be scheduled rather than depending on other teams. What benefit
is there to the Guix user when glibc or the default gcc are updated?
You're only a "guix pull" away from updated packages.

As I recall, one issue for past releases was having to freeze all
development on the master branch. With the new teams-branches model
the release-team branch is just another branch, moving to the queue
when ready to cut a new release.

Greg

[1] https://lists.gnu.org/archive/html/guix-devel/2024-09/msg00018.html

Re: On the quest for a new release model

[Suhail Singh]

Greg Hogan <code@greghogan.com> writes:

> We only need a release team and a documented release process. Releases
> should be scheduled rather than depending on other teams. What benefit
> is there to the Guix user when glibc or the default gcc are updated?
> You're only a "guix pull" away from updated packages.
>
> As I recall, one issue for past releases was having to freeze all
> development on the master branch. With the new teams-branches model
> the release-team branch is just another branch, moving to the queue
> when ready to cut a new release.

My sentiments precisely.  Thank you, Greg, for describing the situation
clearly.

If the goal is to increase the frequency of releases while maintaining
quality, the only consideration that the teams-branch ought to make is
to ensure that the commit in question (corresponding to the release)
builds correctly (i.e., may, in future, do some automated testing beyond
the test suites included in the individual packages) past some threshold
(i.e., on platforms of interest etc.).  Importantly, instead of making a
release soon after a major merge, such a team may decide to not make a
release too close to major changes.

However, to me, the release cadence is orthogonal to how we do the
versioning.  It's possible for a project to have time-based releases
while still using semver.  I propose that this thread be only about the
release process, and that the discusssion about the versioning happen in
a different thread.

-- 
Suhail

Re: On the quest for a new release model

[Simon Josefsson via Development of GNU Guix and the GNU System distribution.]

[-- Attachment #1: Type: text/plain, Size: 611 bytes --]

How much of the guix git repository content can you remove and still
have a working OS?  In other words, can we strip away almost all of the
packages and still have a minimal bootable system?  If we minimize that
set of really-core packages, maybe there can be one team that works on
this minimal bootable OS and produce stable images that are fast to
update.  If people want to add more packages, there is always the
current "fat" rolling Guix git repository that you can add on top of the
small base OS.  Just an idea that I've come back to while watching the
Guix git repository grow over the years.

/Simon

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

Re: On the quest for a new release model

[Suhail Singh]

Suhail Singh <suhailsingh247@gmail.com> writes:

> If the goal is to increase the frequency of releases while maintaining
> quality, the only consideration that the teams-branch ought to make is
                                           ^^^^^^^^^^^^
> to ensure that the commit in question (corresponding to the release)
> builds correctly (i.e., may, in future, do some automated testing beyond
> the test suites included in the individual packages) past some threshold
> (i.e., on platforms of interest etc.).  Importantly, instead of making a
> release soon after a major merge, such a team may decide to not make a
> release too close to major changes.

I meant release-team, above.

-- 
Suhail

Re: On the quest for a new release model

[Cayetano Santos]

[-- Attachment #1: Type: text/plain, Size: 1556 bytes --]


>ven. 13 déc. 2024 at 10:52, Suhail Singh <suhailsingh247@gmail.com> wrote:

> Greg Hogan <code@greghogan.com> writes:
>
>> We only need a release team and a documented release process. Releases
>> should be scheduled rather than depending on other teams. What benefit
>> is there to the Guix user when glibc or the default gcc are updated?
>> You're only a "guix pull" away from updated packages.
>>
>> As I recall, one issue for past releases was having to freeze all
>> development on the master branch. With the new teams-branches model
>> the release-team branch is just another branch, moving to the queue
>> when ready to cut a new release.
>
> My sentiments precisely.  Thank you, Greg, for describing the situation
> clearly.

Let me just play the dummies advocate for a moment, before leaving the
room to most experienced people: consider the army of regular potential
users not necessarily concerned about monads, variants and derivations
(or even substitutes).

They will be doing `whatever install guix`, and then they’ll happily
start using guix: with a bit of documentation, they’ll love it. This is
all they need to know. Don’t tell them to do a `python pull`, `firefox
pull` or `guix pull`. It comes to the same for them: they won’t
understand.

Once they get used to use guix daily, they’ll care about pulls, channels
or asking support for guix on the cluster.

--
Cayetano Santos
GnuPG Key:   https://meta.sr.ht/~csantosb.pgp
FingerPrint: CCB8 1842 F9D7 058E CD67 377A BF5C DF4D F6BF 6682

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 259 bytes --]

Re: On the quest for a new release model

[Suhail Singh]

Cayetano Santos <csantosb@inventati.org> writes:

> Let me just play the dummies advocate for a moment ...

Thank you.  I believe it is important to consider such scenarios (among
others).

> They will be doing `whatever install guix`, and then they’ll happily
> start using guix: with a bit of documentation, they’ll love it. This
> is all they need to know. Don’t tell them to do a `python pull`,
> `firefox pull` or `guix pull`. It comes to the same for them: they
> won’t understand.

Is my understanding of your scenario as summarized below correct, could
you please confirm?

There are some individuals who would be able to follow the instructions
to install Guix.  However, these individuals wouldn't understand the
instructions to perform a "guix pull".  Further, the number of such
individuals is greater than zero, and you are proposing that we consider
their needs when determining the release process for Guix.

Assuming my understanding above is correct, wouldn't you agree that
(even) for those individuals what's most important is that there is a
_stable_ and _not-very-outdated_ release available?  My (and I believe
Greg's) contention is that following a time-based release process
achieves these objectives more effectively than following a
feature-based release process.

-- 
Suhail

Re: On the quest for a new release model

[Suhail Singh]

Simon Josefsson via "Development of GNU Guix and the GNU System
distribution." <guix-devel@gnu.org> writes:

> can we strip away almost all of the packages and still have a minimal
> bootable system?  If we minimize that set of really-core packages,
> maybe there can be one team that works on this minimal bootable OS and
> produce stable images that are fast to update.  If people want to add
> more packages, there is always the current "fat" rolling Guix git
> repository that you can add on top of the small base OS.

I proposed something similar some weeks ago [1].  Specifically,
splitting Guix's monolithic repository into at least two different
channels (perhaps more).  These channels would be included in
%default-channels, however, the main Guix repository would only include
the packages from Ring-0 and Ring-1 (taking inspiration from OpenSUSE)
[2]:

#+begin_quote
  The core of Factory is divided into two rings (0-Bootstrap,
  1-MinimalX). Ring 0 contains packages that form the most basic,
  minimalist system that can compile itself. On top of that Ring 1 adds
  what's in the default installation of the two primary Desktops. All
  other packages are not part of a ring.
#+end_quote

[1]: <https://yhetil.org/guix/87plnm6cnj.fsf@gmail.com/>
[2]: <https://en.opensuse.org/openSUSE:Factory_development_model#Submit_requests_to_Ring_0_and_Ring_1>

-- 
Suhail

Re: On the quest for a new release model

[Tomas Volf]

[-- Attachment #1: Type: text/plain, Size: 1505 bytes --]

Suhail Singh <suhailsingh247@gmail.com> writes:

> Simon Josefsson via "Development of GNU Guix and the GNU System
> distribution." <guix-devel@gnu.org> writes:
>
>> can we strip away almost all of the packages and still have a minimal
>> bootable system?  If we minimize that set of really-core packages,
>> maybe there can be one team that works on this minimal bootable OS and
>> produce stable images that are fast to update.  If people want to add
>> more packages, there is always the current "fat" rolling Guix git
>> repository that you can add on top of the small base OS.
>
> I proposed something similar some weeks ago [1].  Specifically,
> splitting Guix's monolithic repository into at least two different
> channels (perhaps more).  These channels would be included in
> %default-channels, however, the main Guix repository would only include
> the packages from Ring-0 and Ring-1 (taking inspiration from OpenSUSE)
> [2]:

While I explicitly do not comment on this proposal, I would like to
point out that there is more work involved than just splitting it into
separate channels and adding those into %default-channels.

While multiple channels do work well with guix pull, there are annoying
limitations when used in other places.

Also you would need some way to specify what channel commits are able to
work together version-wise.

Tomas

-- 
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 853 bytes --]

Re: On the quest for a new release model

[Cayetano Santos]

[-- Attachment #1: Type: text/plain, Size: 2040 bytes --]


>ven. 13 déc. 2024 at 12:21, Suhail Singh <suhailsingh247@gmail.com> wrote:

> Is my understanding of your scenario as summarized below correct, could
> you please confirm?
>
> There are some individuals who would be able to follow the instructions
> to install Guix.  However, these individuals wouldn't understand the
> instructions to perform a "guix pull".  Further, the number of such
> individuals is greater than zero, and you are proposing that we consider
> their needs when determining the release process for Guix.

Correct. Except the number of such individuals is huge. Some are doing
research, and they don’t need any more complexity, that they see as an
extra overload : they’ll install guix, and start using it. That’s it.
The more it becomes complex, the less they’ll be prone to try.

> Assuming my understanding above is correct, wouldn't you agree that
> (even) for those individuals what's most important is that there is a
> _stable_ and _not-very-outdated_ release available?

Correct again. At least, on a first contact with guix: one release every
4 months, for example, means using on average a two months out to date
guix, which is fine.

Take julia, for example. The cycle is : install guix, check julia,
outdated, forget guix and use julia installer.

> My (and I believe Greg's) contention is that following a time-based
> release process achieves these objectives more effectively than
> following a feature-based release process.

I’m not against: higher instances decide. Anything, but a 2 years old release.

Once again: sure, anything can be done, but it takes time and skills.
And guix is already good and useful enough for most people to use it
smoothly on a daily basis, solving lots of real life problems. Once they
fall into the rabbit hole, they’ll pull substitutes from channels.

But we need releases, first.

--
Cayetano Santos
GnuPG Key:   https://meta.sr.ht/~csantosb.pgp
FingerPrint: CCB8 1842 F9D7 058E CD67 377A BF5C DF4D F6BF 6682

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 259 bytes --]

Re: On the quest for a new release model

[Ricardo Wurmus]

Suhail Singh <suhailsingh247@gmail.com> writes:

> Assuming my understanding above is correct, wouldn't you agree that
> (even) for those individuals what's most important is that there is a
> _stable_ and _not-very-outdated_ release available?  My (and I believe
> Greg's) contention is that following a time-based release process
> achieves these objectives more effectively than following a
> feature-based release process.

I agree that more frequent releases are necessary, but I don't see much
value in “automatic” time-triggered releases.  After all, that's what
"guix pull" already provides.  Our releases should mean something.

Also note that Guix itself is a library.  I don't think it would be a
good idea to inflate the number of releases.  Our installer script
already offers a way to install the latest untested version of the Guix
package manager.

-- 
Ricardo

Re: On the quest for a new release model

[Suhail Singh]

Tomas Volf <~@wolfsden.cz> writes:

> I would like to point out that there is more work involved than just
> splitting it into separate channels and adding those into
> %default-channels.

Could you please elaborate?  Or were these the two points you noted
below?

> While multiple channels do work well with guix pull, there are annoying
> limitations when used in other places.

Could you please share some references for this?

> Also you would need some way to specify what channel commits are able to
> work together version-wise.

Let's say we have two channels in the future: $guix-slim and
$guix-extras.  Wouldn't it be sufficient for $guix-extras to depend on
$guix-slim ?  If not, could you please elaborate?

-- 
Suhail

Re: On the quest for a new release model

[Suhail Singh]

Ricardo Wurmus <rekado@elephly.net> writes:

> Suhail Singh <suhailsingh247@gmail.com> writes:
>
>> Assuming my understanding above is correct, wouldn't you agree that
>> (even) for those individuals what's most important is that there is a
>> _stable_ and _not-very-outdated_ release available?  My (and I believe
>> Greg's) contention is that following a time-based release process
>> achieves these objectives more effectively than following a
>> feature-based release process.
>
> I agree that more frequent releases are necessary, but I don't see much
> value in “automatic” time-triggered releases.  After all, that's what
> "guix pull" already provides.  Our releases should mean something.

I was not proposing for "automatic" time-triggered releases.  I was
proposing that a periodic cadence start the _process_ of "vetting" the
release candidate.  Notably, the vetting process (particulars yet to be
decided) would be focusing on stability, test and build coverage etc.
I.e., some notions of quality.  This is something "guix pull" does _not_
provide.

I would describe the linux kernel as following a time-based release
process.  The meaningful distinction from a feature-based release
process is that the release process doesn't wait for any feature.

> Also note that Guix itself is a library.  I don't think it would be a
> good idea to inflate the number of releases.

If they are versioned according to an agreed upon versioning scheme, why
would the proliferation of versions _not_ be a good idea?

-- 
Suhail

Re: On the quest for a new release model

[Felix Lechner via Development of GNU Guix and the GNU System distribution.]

Hi Ricardo,

On Fri, Dec 13 2024, Ricardo Wurmus wrote:

> Our releases should mean something.

What do they mean, please?

Kind regards
Felix

P.S. No flames, please.  It's an innocent question.

Re: On the quest for a new release model

[John Kehayias via Development of GNU Guix and the GNU System distribution.]

Hi all,

On Fri, Dec 13, 2024 at 03:08 PM, Felix Lechner via \"Development of GNU Guix and the GNU System distribution.\" wrote:

> On Fri, Dec 13 2024, Ricardo Wurmus wrote:
>
>> Our releases should mean something.
>
> What do they mean, please?
>

This is actually related to the topic I wanted to bring up before we really get lost in the details of release building and so on.

I've always thought about Guix as a rolling distribution, where our "releases" are essentially installer "snapshots." In other words, the installer has maybe gotten improvements and certainly been thoroughly tested along with good substitute coverage and functionality across all (or as much as we can muster) of Guix. This tagged version is important to make sure there is binary substitute downloads (and maybe a corresponding manual online).

There is no expectation, nor really support, for staying on a "release." One is expected (and warned by guix even) to occasionally at least run "guix pull" if not upgrading, reconfiguring, etc. However, we do have powerful tools for staying at some particular commit, through time-machine, channel pinning, and so on.

I think what I wrote above is where we are currently at, though this perhaps is not documented explicitly as it should be. Many times the question comes up about a Guix "release" and missing that we are really a rolling distribution. Clearly there is room to improve here.

Is this something we agree on? I think being clear what Guix is (in terms of "releases"), and what we want it to be, is essential first.

That is my understanding above, and if that's what we want to continue with, we should say so clearly in the manual, finally resolve the confusing different manual version links, and be clear what a "release" is for Guix. I would move to revise our terminology to something like an "installer" version or "installation snapshot." (I don't see us having the resources to fully support releases as in a versioned distro, at this time.)

Just my 2 insignificant-currency-units, happy to hear this discussed further!

John

Re: On the quest for a new release model

[Ricardo Wurmus]

Simon Josefsson via "Development of GNU Guix and the GNU System distribution." <guix-devel@gnu.org> writes:

> How much of the guix git repository content can you remove and still
> have a working OS?  In other words, can we strip away almost all of the
> packages and still have a minimal bootable system?  If we minimize that
> set of really-core packages, [...]

This has been discussed in the past and it is far from trivial.
Consider that glibc needs Python at build time and the Linux kernel may
need Rust.  Consider also that it is easy to introduce a package from
the outer ring to the inner ring through transitive dependencies.

Yes, there are many leaf packages, but it seems to me that it's a fool's
errand to attempt to separate packages like this.  (An exception is
package collections like those in CRAN or Bioconductor.)  It's certainly
not work that should be done manually.

We probably don't want to rehash this discussion as part of a discussion
about releases.

-- 
Ricardo

Re: On the quest for a new release model

[Ricardo Wurmus]

Suhail Singh <suhailsingh247@gmail.com> writes:

>> Also you would need some way to specify what channel commits are able to
>> work together version-wise.
>
> Let's say we have two channels in the future: $guix-slim and
> $guix-extras.  Wouldn't it be sufficient for $guix-extras to depend on
> $guix-slim ?  If not, could you please elaborate?

You have multiplied the release problem by two.

guix-slim and guix-extras would both need to be versioned somehow, so
that whatever packages and Guix library infrastructure guix-extras needs
are in fact provided by the version of guix-slim.

I strongly advise not to go down this route.  See also my other email on
why it is not practical; feel free to search the archives for prior
discussions.

-- 
Ricardo

Re: On the quest for a new release model

[Tomas Volf]

[-- Attachment #1: Type: text/plain, Size: 2324 bytes --]

Suhail Singh <suhailsingh247@gmail.com> writes:

> Tomas Volf <~@wolfsden.cz> writes:
>
>> I would like to point out that there is more work involved than just
>> splitting it into separate channels and adding those into
>> %default-channels.
>
> Could you please elaborate?  Or were these the two points you noted
> below?

Yes, I have meant that the various issues would need to be fixed or
figured out.

>
>> While multiple channels do work well with guix pull, there are annoying
>> limitations when used in other places.
>
> Could you please share some references for this?

Definitely.  I personally have run into these two issues:

#74396
  When you install system-wide guix with multiple channels, it shadows
  channels from your guix-pulled guix (except for 'guix channel).  That
  was unpleasant surprise.

guix-for-channels does not cache
  `guix-for-channels' procedure is a way how to produce a guix package
  for a channel list.  However it does not cache the computation, so it
  is extremely slow when actually used.  I wrote a bit about that here:
  https://wolfsden.cz/blog/post/what-goes-into-guix-shaped-hole.html .

I am not sure if there are more, but my impression is that using
multiple channels (outside of guix-pull) is not well explored territory,
so I would not be surprised.

>
>> Also you would need some way to specify what channel commits are able to
>> work together version-wise.
>
> Let's say we have two channels in the future: $guix-slim and
> $guix-extras.  Wouldn't it be sufficient for $guix-extras to depend on
> $guix-slim ?  If not, could you please elaborate?

You say depend on guix-slim, but on *what* commit from guix-slim?  You
can pin channels.  What happens when you pin guix-slim, but pull
guix-extras?  I believe some human readable error should be given, not a
failure to build something due to, for example, not having the required
version of gcc now (for example because guix-extras increased the
required version).

Also what about substitutes?  Which combinations of commits will have
substitutes built?

I just feel there is a lot to figure out with this proposal.

Have a nice day,
Tomas

-- 
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 853 bytes --]

Re: On the quest for a new release model

[Suhail Singh]

Ricardo Wurmus <rekado@elephly.net> writes:

>> Let's say we have two channels in the future: $guix-slim and
>> $guix-extras.  Wouldn't it be sufficient for $guix-extras to depend on
>> $guix-slim ?  If not, could you please elaborate?
>
> You have multiplied the release problem by two.

If more releases pose a problem, then yes, we've increased the problem.
The set of challenges imposed by a release are not clear.  As well, it's
unclear whether the challenges are insurmountable, and if not what the
cost of overcoming them is.

However, in order to keep the discussion about releases focused, I
agree/propose that this tangent, about possibly splitting up the
monolithic Guix channel, be discussed in a separate thread (by those
interested).

-- 
Suhail

Re: On the quest for a new release model

[Suhail Singh]

Tomas Volf <~@wolfsden.cz> writes:

> I just feel there is a lot to figure out with this proposal.

Thank you for the detailed response.  And I agree that this isn't a
solved problem (yet).  I propose that further discussion on this tangent
happen on a separate thread.

Whether Guix is split into more than one channel doesn't obviate the
need to define and agree upon a release model (which is what this thread
is about): either the current state where installer snapshots come after
several months, or some alternate future state the community sees value
in.

-- 
Suhail