Re: Emacs release cadence

[Eli Zaretskii <eliz@gnu.org>]

> Date: Tue, 29 Aug 2023 02:52:56 +0300
> Cc: luangruo@yahoo.com, philipk@posteo.net, danny@dfreeman.email,
>  stefankangas@gmail.com, emacs-devel@gnu.org, manuel.uberti@inventati.org
> From: Dmitry Gutov <dmitry@gutov.dev>

I will respond to some of your points below, but I must say that this
style of discussing the aspects and frequency of our releases is not
useful, and TBH quite frustrating.  You present some up- and
down-sides, with which everyone will agree.  You say some truisms that
again are accepted by everyone.  But there are no suggestions that are
practical and based on our actual experience and practices, just
theoretical considerations, hypothetical examples, and numbers taken
out of thin air.  I don't see how can anything useful and practical
come out of such a discussion, sorry.

> Looking at the current release history, it took 9 months between the 
> release branch was initially cut and the release. The development time 
> on master preceding that was, depending on how to measure, either ~1 
> year or 1 year plus however the previous pre-release also took.
> 
> Anyway, suppose next time we start the release branch only after 6 
> months. Would you say it will still take 9 months to stabilize?

It depends on the number of significant changes that got installed.
OTOH, if we cut the release branch when master doesn't have enough new
features, why make a new major release?

IOW, you seem to be arguing for a flat, one-level release system, with
no major and minor releases.  This is IMO appropriate only for very
stable programs, where there are no major changes, only minor ones.
Emacs is not there, and probably never will be, because there's no
limit to new features it can have, unlike, say, Binutils (which indeed
rolls out a new version every 6 months, and its NEWS files typically
have just several dozen lines about each new version).

> > If we get feedback, but do not verify that the fixes indeed fixed the
> > problems, and didn't cause unintended problems elsewhere (something
> > that happens with alarming frequency around here), these frequent
> > releases become pointless.
> 
> We cannot really "verify" a fix, just get reasonably confident that it's 
> good. Any time interval we use will increase the confidence, but also at 
> the expense of velocity. And we won't reach 100% confidence still.

This is just word semantics and truisms.  You can assume we both
understand that, and that when I say "verify" I mean something
practical and tangible, and am well aware that it is impossible to
produce a bug-free software of this magnitude.

But Emacs is an interactive editor, which users use to edit their
precious files, in sessions that are supposed to stay up and running
for weeks on end, so it must be reasonably stable and avoid crashing
and losing edits.  Unlike a Web browser, for example.

> And velocity is not just about the next shiny thing, it's also about 
> getting fixes to a certain large category of users sooner.

Another truism, with which no one will argue.  How does it help to
keep this discussion useful and focused?

> >>> This guy just recently upgraded to Emacs 29, and is reporting a
> >>> significant issue only now, a month after Emacs 29.1 was released.
> >>
> >> As our (and others') experience shows, indeed, there is no way to ensure
> >> that all bugs are fixed, all regressions are ironed out, and nobody is
> >> ever unhappy.
> > 
> > Is this what you seriously think we are doing -- waiting for all the
> > bugs to be fixed?  It is not useful to discuss serious subjects such
> > as this one with such a non-serious attitude.
> 
> It's a simplification to make the explanation shorter.

Which again doesn't help.  What it does do is duck the serious and
important question that I raised, one that is very pertinent to this
discussion, if we want to have some useful outcome out of it.  Namely:
how do you collect feedback efficiently under these circumstances,
whether it can be sped up, and how does the typical schedule of
collecting feedback affect the time frame for the next release?

> >> Some people will wait longer before upgrading and ignore all pretests. I
> >> only know, again, two things we could possibly do:
> >>
> >> - Get releases out earlier (so the "one month since release" day comes
> >> faster).
> >> - Get a lot more users and/or a lot more feedback from the same users.
> >>
> >> The latter is a bet that even if, maybe, user C only uses Emacs
> >> releases, there will be users D and E with similar enough workflows that
> >> do test our snapshot builds and would report the same problems sooner.
> >>
> >> Some problems will remain unreported anyway. Some stay that way for decades.
> > 
> > Are you talking from experience with releasing Emacs and collecting
> > the feedback?
> 
> No exactly, but I've been around the bug tracker for a little while.
> 
> > Because I am, and I can tell that (a) we have no
> > control on how many users will return feedback,
> 
> But there are factors we could improve (mentioned previously).

Which ones?  Name them, please!  Above you just say "we could", and
IME we cannot.  So let's discuss how to "get a lot more users and/or a
lot more feedback from the same users" in practice, if you have
practical ideas to that effect.

> > and (b) it takes
> > several weeks(!) to get useful feedback with real problems flowing
> > after a release.  This basically invalidates the simplistic model you
> > describe above.
> 
> Suppose we have a release loop of 6 months (again, a contrived example). 

"A contrived example".  Why do you think it's useful to discuss
theoretical examples which are not at all connected or even related to
the real situation we have?

> Even then we'd have the first month to collect the initial feedback, and 
> then 5 months to act on it (either release a minor version for simple 
> fixes or important regressions, or put it into master and iterate).

I said it takes several weeks to get useful feedback, I didn't say it
takes just one month.  IME, 1 month is barely enough to get some
feedback for a minor release; for major releases it's far from being
enough.  We are still getting regression reports about Emacs 28.1,
more than a year after it was released!

> Further, a shorter release cycle would mean fewer features and less 
> changes in each release (which is both a pro and a con). But less 
> changes also implies fewer potential new bugs to deal with. It's not a 
> linear relation, but a reduction seems logical.

This is again a more-or-less obvious tradeoff, but without any
practical conclusion or suggestion.  How do you know we aren't already
at the optimal point in this particular tradeoff?  That we all want
more frequent releases doesn't mean they aren't already "as frequent
as reasonable, but no more frequent"?  We need specific, practical
suggestions and ideas, based on our actual experience and users, to
seriously consider significant changes in what we currently do.

Please also keep in mind that no one counts weeks or months when
deciding whether to cut the release branch or make a release.  The
main considerations are not the time passed, but the perceived
stability of the package and the absence of serious known bugs.  The
schedule is the result, not the control variable.

> >> People who stay silent about their problems will get what their pay for.
> > 
> > Problem is: most of them do.  People who want the latest ASAP simply
> > track the development branches, and we have their feedback more or
> > less in real time; more frequent releases will not improve the
> > feedback we get from them.
> 
> If we release the next version of Emacs in 1 year, rather than in 2 
> years, we'll get some faster feedback anyway from those slow users (even 
> if they take 1-2 months after the release to upgrade and send reports).

Yes.  And we will have a less "interesting" release.  Look at the
state of the master branch a year ago and try to think whether what we
had back then would justify a major release.

Also, how do you propose to get to the 1 year mark?  A major release
needs at least 2 - 3 pretests, with each pretest taking a month or so.
And we need some time before the first pretest to stabilize the
branch, complete and fix the missing/incomplete documentation,
etc. etc.  And before that we need some time to develop and install
new features on master, after the previous release is out.  So please
try to present a reasonable schedule that would allow us to make a
major release once a year, and if you still think it's doable, let's
see the schedule and talk about some of your assumptions.

This is what I call practical suggestions based on our experience.
Without using real, not imaginary, figures, procedures, and time
frames we need, all this discussion becomes a futile academic exercise
with no practical importance whatsoever.

And, btw, the last 3 major releases took 1.25 to 1.5 years, so already
quite close to the 1-year mark.  Not 2 years, as you say.

> There is definitely a cutoff point somewhere when more frequent releases 
> would hurt rather than help: perhaps if it got to a point where the vast 
> majority of our users chose not to upgrade for 2-3 release cycles anyway.
> 
> But I figure we're not there yet, since the question of shorter 
> development cycles gets brought up with some regularity.

How do you figure that?  The fact that this is being brought up
doesn't mean we are not at the optimum already.  The only way to have
a useful discussion about this is to talk specifics and use numbers
from our long experience.