Reflections on the release process

Reflections on the release process

[Ludovic Courtès]

Hello Guix!

The release process is long and tedious.  I’m super happy we made that
release and did a lot of work collectively testing and fixing things,
but I thought I’d share my frustrations while it’s fresh in memory so we
can brainstorm and make it better for next time!


1. “make release” requires ~6 hours to build everything.  You have
to have offloading enabled and the connections have to be stable all
along.  In total it builds the ‘guix’ package six times (once for each
binary tarball, then once for each ISO image).

2. “make release” commits to update ‘guix’ and these commits are signed
because that’s our Git config.  That means gpg-agent must not forget
your passphrase during that time or the process will stop in the middle.
I work around that by temporarily increasing gpg-agent’s
‘default-cache-ttl’, only when I’m at home and near my laptop, but it’s
obviously not great.

3. ‘guix’ in the ISO image can install 1.1.0 precisely, but that means
that the commit that defines 1.1.0 has typically not been picked up by
ci.guix.gnu.org (or we’d have to push it explicitly).  So you have to
manually get berlin to build it afterwards or anyone installing Guix
System will build from source.

4. We lack a clear way to mark bugs as release-critical.  I’m really
happy Florian, Mathieu, and I have been able to work together and squash
bugs one by one (thank you!).  Still, it would have been better if we
could have tagged which is release-critical and which is not, to prevent
misunderstandings such as regarding the NVMe bug:
<https://issues.guix.gnu.org/issue/40590>.  Can Debbugs help?  The GCC
folks have a system that sends email with an update on the number of
release-critical bugs.  I’m sure we can learn from how others deal with
that.

5. With the installer, we have to test on actual hardware, and that
necessarily takes time.  Thumbs up to everyone who tried the RCs and
reported back!  Perhaps we need to make time for more RCs in the future.
Yet, we probably don’t want the frozen branch to live for too long, so
that should still be fast-paced.

6. Writing release notes, announcements, etc. takes a lot of time.



Part of the pain for 1–3 is due to the insane amount of time it takes to
build Guix entirely.  That’s mostly work to be done on Guile’s compiler,
which is a top priority for Guix.  Perhaps there are also things we can
do on the Guix side, such as arranging for ISO images to use a Guix
built with (guix self) rather than a ‘guix’ package to rebuild build
times.

For the other issues, I’m interested in any ideas you may have!

Until then, big thanks to everyone who helped out over the months and
especially over the last few days—now it’s party time!  :-)

Ludo’.

Re: Reflections on the release process

[zimoun]

Hi Ludo,

Thanks to everyone !


On Wed, 15 Apr 2020 at 22:18, Ludovic Courtès <ludo@gnu.org> wrote:

> 4. We lack a clear way to mark bugs as release-critical.  I’m really
> happy Florian, Mathieu, and I have been able to work together and squash
> bugs one by one (thank you!).  Still, it would have been better if we
> could have tagged which is release-critical and which is not, to prevent
> misunderstandings such as regarding the NVMe bug:
> <https://issues.guix.gnu.org/issue/40590>.  Can Debbugs help?  The GCC
> folks have a system that sends email with an update on the number of
> release-critical bugs.  I’m sure we can learn from how others deal with
> that.

The first easy step seems to tag the relevant bug as release-critical
or simply rc.
Currently, the tags nomal, security, serious, moreinfo, important,
unreproducible are used in Debbugs.

Then a second step could be to collect these release critical bugs and
display them on issues.guix.gnu.org (or data.guix.gnu.org) for example
remplacing the circle exclamation mark by a red triangle exclamation
mark (or any really visible symbol). And because the "recent
activities" is already sorted, it becomes easier to point the
remaining release critical bugs, the ones stuck, etc.


> For the other issues, I’m interested in any ideas you may have!

About the "frozen" window, does it make to schedule it in advance? For
example a couple of months before.
And for example, does it make sense to say: at least one release each
year on the March 14th (Pi day ;-) or approximately.
Because in general FOSDEM is at the beginning of February and it is a
big party where some of us come then back home refill of energy, we
could agree around this date (beginning of Feb.) on the frozen window
date (say end of February) and then release around middle of March.
From my point of view, using the Guix Days as catalyst for releasing
should help the process.


Cheers,
simon

Re: Reflections on the release process

[Ludovic Courtès]

Hi!

zimoun <zimon.toutoune@gmail.com> skribis:

> On Wed, 15 Apr 2020 at 22:18, Ludovic Courtès <ludo@gnu.org> wrote:
>
>> 4. We lack a clear way to mark bugs as release-critical.  I’m really
>> happy Florian, Mathieu, and I have been able to work together and squash
>> bugs one by one (thank you!).  Still, it would have been better if we
>> could have tagged which is release-critical and which is not, to prevent
>> misunderstandings such as regarding the NVMe bug:
>> <https://issues.guix.gnu.org/issue/40590>.  Can Debbugs help?  The GCC
>> folks have a system that sends email with an update on the number of
>> release-critical bugs.  I’m sure we can learn from how others deal with
>> that.
>
> The first easy step seems to tag the relevant bug as release-critical
> or simply rc.
> Currently, the tags nomal, security, serious, moreinfo, important,
> unreproducible are used in Debbugs.

Yes, but how do we do that?  I don’t think Debbugs supports a
“release-critical” tag, does it?  Or perhaps we could take advantage of
Mumi to add special tags or something?

> Then a second step could be to collect these release critical bugs and
> display them on issues.guix.gnu.org (or data.guix.gnu.org) for example
> remplacing the circle exclamation mark by a red triangle exclamation
> mark (or any really visible symbol). And because the "recent
> activities" is already sorted, it becomes easier to point the
> remaining release critical bugs, the ones stuck, etc.

Agreed.

>> For the other issues, I’m interested in any ideas you may have!
>
> About the "frozen" window, does it make to schedule it in advance? For
> example a couple of months before.
> And for example, does it make sense to say: at least one release each
> year on the March 14th (Pi day ;-) or approximately.
> Because in general FOSDEM is at the beginning of February and it is a
> big party where some of us come then back home refill of energy, we
> could agree around this date (beginning of Feb.) on the frozen window
> date (say end of February) and then release around middle of March.
> From my point of view, using the Guix Days as catalyst for releasing
> should help the process.

I think there’s no question we want more than one release per year.  For
that to happen, we should make sure the process is well documented, as
smooth as possible, largely automated, and not tied to a single person
(ahem…).

Once we have that, plus some planning such as marking RC-critical bugs,
it’ll be easier to make releases IMO.

Thanks,
Ludo’.

Re: Reflections on the release process

[zimoun]

On Wed, 22 Apr 2020 at 22:09, Ludovic Courtès <ludo@gnu.org> wrote:


> Yes, but how do we do that?  I don’t think Debbugs supports a
> “release-critical” tag, does it?  Or perhaps we could take advantage of
> Mumi to add special tags or something?

The GNU instance of Debbugs does not support it, I agree.
However, the Debian instance supports more tags than the GNU one, e.g.,

patch, wontfix, moreinfo, unreproducible, help, security, upstream,
pending, confirmed, ipv6, lfs, d-i, l10n, newcomer, a11y, ftbfs,
fixed-upstream, fixed, fixed-in-experimental, potato, woody, sarge,
etch, lenny, squeeze, wheezy, jessie, stretch, buster, bullseye,
bookworm, sid, experimental, sarge-ignore, etch-ignore, lenny-ignore,
squeeze-ignore, wheezy-ignore, jessie-ignore, stretch-ignore,
buster-ignore, bullseye-ignore, bookworm-ignore

https://www.debian.org/Bugs/

I have not dove into the Debbugs configuration so I do not know if it
is easy or not to extend the list of tags. However, because it is
managed by GNU sysadmin, it should be a long road. ;-)

And I do not know what is the "effect" of the 'block' command. Because
a "release vx.y" bug could be open. And the release critical issues
could be marked as "block release-vx.y". This bug "release vx.y" could
help to synchronise/coordinate and/or track progress and should help
for managing the schedule and deadlines.

I do not know. I am thinking loud.



> I think there’s no question we want more than one release per year.  For
> that to happen, we should make sure the process is well documented, as
> smooth as possible, largely automated, and not tied to a single person
> (ahem…).

Agree.


Cheers,
simon

Re: Reflections on the release process

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 3002 bytes --]

Hello Ludo,

Ludovic Courtès <ludo@gnu.org> ezt írta (időpont: 2020. ápr. 22., Sze
22:09):

> Hi!
>
> zimoun <zimon.toutoune@gmail.com> skribis:
>
> > On Wed, 15 Apr 2020 at 22:18, Ludovic Courtès <ludo@gnu.org> wrote:
> >
> >> 4. We lack a clear way to mark bugs as release-critical.  I’m really
> >> happy Florian, Mathieu, and I have been able to work together and squash
> >> bugs one by one (thank you!).  Still, it would have been better if we
> >> could have tagged which is release-critical and which is not, to prevent
> >> misunderstandings such as regarding the NVMe bug:
> >> <https://issues.guix.gnu.org/issue/40590>.  Can Debbugs help?  The GCC
> >> folks have a system that sends email with an update on the number of
> >> release-critical bugs.  I’m sure we can learn from how others deal with
> >> that.
> >
> > The first easy step seems to tag the relevant bug as release-critical
> > or simply rc.
> > Currently, the tags nomal, security, serious, moreinfo, important,
> > unreproducible are used in Debbugs.
>
> Yes, but how do we do that?  I don’t think Debbugs supports a
> “release-critical” tag, does it?  Or perhaps we could take advantage of
> Mumi to add special tags or something?
>

Actually the description of the severity level serious says its intended
meaning is making a bug release critical. I believe we could simply use
that.


> > Then a second step could be to collect these release critical bugs and
> > display them on issues.guix.gnu.org (or data.guix.gnu.org) for example
> > remplacing the circle exclamation mark by a red triangle exclamation
> > mark (or any really visible symbol). And because the "recent
> > activities" is already sorted, it becomes easier to point the
> > remaining release critical bugs, the ones stuck, etc.
>
> Agreed.
>
> >> For the other issues, I’m interested in any ideas you may have!
> >
> > About the "frozen" window, does it make to schedule it in advance? For
> > example a couple of months before.
> > And for example, does it make sense to say: at least one release each
> > year on the March 14th (Pi day ;-) or approximately.
> > Because in general FOSDEM is at the beginning of February and it is a
> > big party where some of us come then back home refill of energy, we
> > could agree around this date (beginning of Feb.) on the frozen window
> > date (say end of February) and then release around middle of March.
> > From my point of view, using the Guix Days as catalyst for releasing
> > should help the process.
>
> I think there’s no question we want more than one release per year.  For
> that to happen, we should make sure the process is well documented, as
> smooth as possible, largely automated, and not tied to a single person
> (ahem…).
>
> Once we have that, plus some planning such as marking RC-critical bugs,
> it’ll be easier to make releases IMO.
>
> Thanks,
> Ludo’.
>

Best regards,
g_bor

>
>

[-- Attachment #2: Type: text/html, Size: 4438 bytes --]