Re: Emacs release cadence

[Dmitry Gutov <dmitry@gutov.dev>]

On 28/08/2023 15:02, Eli Zaretskii wrote:

>> What I meant are more frequent major releases, and some more patch
>> releases between them. And it looks like we're taking ~2 years between
>> major releases now.
> 
> Yes, but this started as a comparison with others, in which case the
> difference between major and minor is not very relevant, since
> Firefox, for example, doesn't have that distinction.

Firefox has been releasing patch versions for majorly breaking bugs or 
0-day security issues from time to time.

Our minor versions carry more changes, but the idea is that we would be 
releasing more frequently from the main branch (meaning, pushing out new 
developments). Because of course we could release a hundred minor 
versions in a month from an already-closed release branch, if we didn't 
change anything serious inside.

>> Anyway, specific intervals are not that important, it was just an
>> example: we could increase frequency, and nothing major would break.
> 
> IME, nope, we cannot, not by a lot, anyway.

Looking at the current release history, it took 9 months between the 
release branch was initially cut and the release. The development time 
on master preceding that was, depending on how to measure, either ~1 
year or 1 year plus however the previous pre-release also took.

Anyway, suppose next time we start the release branch only after 6 
months. Would you say it will still take 9 months to stabilize?

>>>> But we would get more and faster feedback for new features and
>>>> changes.
>>>
>>> Maybe, maybe not.  See below.
>>
>> I'm pretty sure what I said is self-obvious: when instead of waiting for
>> somebody to check out our test builds we cut a release, a lot of people
>> will get it fairly soon, some later, but on the whole we'll get a lot
>> more feedback, much faster.
> 
> If we get feedback, but do not verify that the fixes indeed fixed the
> problems, and didn't cause unintended problems elsewhere (something
> that happens with alarming frequency around here), these frequent
> releases become pointless.

We cannot really "verify" a fix, just get reasonably confident that it's 
good. Any time interval we use will increase the confidence, but also at 
the expense of velocity. And we won't reach 100% confidence still.

And velocity is not just about the next shiny thing, it's also about 
getting fixes to a certain large category of users sooner.

>>> Are you sure this will help?  Here's a typical case:
>>>
>>>     https://lists.gnu.org/archive/html/help-gnu-emacs/2023-08/msg00454.html
>>>
>>> This guy just recently upgraded to Emacs 29, and is reporting a
>>> significant issue only now, a month after Emacs 29.1 was released.
>>
>> As our (and others') experience shows, indeed, there is no way to ensure
>> that all bugs are fixed, all regressions are ironed out, and nobody is
>> ever unhappy.
> 
> Is this what you seriously think we are doing -- waiting for all the
> bugs to be fixed?  It is not useful to discuss serious subjects such
> as this one with such a non-serious attitude.

It's a simplification to make the explanation shorter.

>> Some people will wait longer before upgrading and ignore all pretests. I
>> only know, again, two things we could possibly do:
>>
>> - Get releases out earlier (so the "one month since release" day comes
>> faster).
>> - Get a lot more users and/or a lot more feedback from the same users.
>>
>> The latter is a bet that even if, maybe, user C only uses Emacs
>> releases, there will be users D and E with similar enough workflows that
>> do test our snapshot builds and would report the same problems sooner.
>>
>> Some problems will remain unreported anyway. Some stay that way for decades.
> 
> Are you talking from experience with releasing Emacs and collecting
> the feedback?

No exactly, but I've been around the bug tracker for a little while.

> Because I am, and I can tell that (a) we have no
> control on how many users will return feedback,

But there are factors we could improve (mentioned previously).

> and (b) it takes
> several weeks(!) to get useful feedback with real problems flowing
> after a release.  This basically invalidates the simplistic model you
> describe above.

Suppose we have a release loop of 6 months (again, a contrived example). 
Even then we'd have the first month to collect the initial feedback, and 
then 5 months to act on it (either release a minor version for simple 
fixes or important regressions, or put it into master and iterate).

Further, a shorter release cycle would mean fewer features and less 
changes in each release (which is both a pro and a con). But less 
changes also implies fewer potential new bugs to deal with. It's not a 
linear relation, but a reduction seems logical.

>>> I've seen similar things many times: people upgrade to a new Emacs
>>> version months, and sometimes years, after that version was released.
>>> Try to collect feedback for a release quickly given this upgrade
>>> schedule.
>>
>> People who stay silent about their problems will get what their pay for.
> 
> Problem is: most of them do.  People who want the latest ASAP simply
> track the development branches, and we have their feedback more or
> less in real time; more frequent releases will not improve the
> feedback we get from them.

If we release the next version of Emacs in 1 year, rather than in 2 
years, we'll get some faster feedback anyway from those slow users (even 
if they take 1-2 months after the release to upgrade and send reports).

With those who wait years, we won't get any improvement, of course, but 
they cannot be helped anyhow.

>> But another upside of a shorter release cycle: even when encountering
>> late, embarrassing regressions, we would be able to say "it'll be fixed
>> in the next point-release" and people will know that they won't have to
>> wait long.
> 
> Not useful if people upgrade slowly and lazily.  Once again, those who
> upgrade eagerly simply build from Git.

There is definitely a cutoff point somewhere when more frequent releases 
would hurt rather than help: perhaps if it got to a point where the vast 
majority of our users chose not to upgrade for 2-3 release cycles anyway.

But I figure we're not there yet, since the question of shorter 
development cycles gets brought up with some regularity.