Re: Move to a cadence release model?

[Eli Zaretskii <eliz@gnu.org>]

> Date: Tue, 10 Nov 2015 20:55:59 -0500
> From: John Yates <john@yates-sheets.org>
> 
> Re Eli's points: In my short time at Mathworks it is clear to me that the model is very different from what he imagines.  At Mathworks a culture of cadence-base release starts with an absolute awareness that no one knows what will be in the next release.

I sincerely doubt that.  Someone in the company's management must
know.  Moreover, someone must actively _control_ that.  You cannot run
a successful business any other way.

> Concretely that translates into _never_ making such commitments to customers.

I never said it should be known to customers.  It should, however, be
known internally.  Otherwise, Mathworks is just another chaotic firm
which we have nothing to learn from (and can actually teach them some ;-).

> I am told that what corresponds to "roadmap presentations" come down to describing areas that the company views as strategic and hence where it is investing development resources.

There you go: we don't have any such roadmap in Emacs.  And I actually
very much doubt if we can practically have one.  At least all past
attempts to start something like that failed spectacularly.  Countless
discussions about adding WYSIWYG word-processor like editing features,
developing an Emacs IDE, etc. -- these all are attempts to set just
the first milestone on that roadmap.  They all eat dust every single
time the issues are raised here.  Doesn't that say something about our
culture and preferences?

> The clear message is that shipping a high-quality product on a predictable schedule is more important than delaying the release or shipping something buggy.

Emacs doesn't "ship buggy", so that's not the issue here.

As for predictable release schedule, whether this is important should
be based on user surveys.  Did someone make such surveys for Emacs?  I
don't think so.  FWIW, it makes little sense to me to install a new
non-bugfix release of a major package such as Emacs without getting
significant new features, but I'm just one user, and most probably not
the typical one.

In any case, if we want to focus on frequent high-quality releases, we
should work on our QA.  That's the point Óscar was making: our QA is
abysmally inadequate.

> rapid availability of bug fix releases containing _nothing_ but high priority bug fixes.

That is relatively much easier, and we already tried that on the
Emacs-24 branch: the last bugfix release was out the door in about 10
days since the first pretest (which we declared an RC).  But this is
only possible on the release branch, and only if we allow absolutely
nothing on that branch except relatively safe bug fixes.

> I have been thinking about the difference between Emacs' development culture and that of project with a strong commitment to maintaining a cadence.  My conclusion is that it comes down to the effort put into keeping the master branch so healthy that a release branch could be cut at nearly any moment.  Inevitably this comes with a strong emphasis on code review, gatekeeping and usually continuous testing.  The Linux kernel is somewhat atypical in having its gatekeeping function  entrusted entirely to humans (Linus' lieutenants and their underlings).  (I am unfamiliar with Firefox.  I did read https://www.mozilla.org/en-US/about/governance/policies/commit/access-policy/ but failed to get a good sense of how things work there.)  All of the remaining projects (Chromium, LibreOffice and OpenStack) use gerrit to enforce a review processes (and to support continuous integration testing).  The important part of all of these review cultures is that work _must_ be presented in reviewable uni

So we essentially agree: switching to any cadence model requires deep
changes in how Emacs development works, which in turn requires a much
larger core development team and a very different organization of that
team.  E.g., a gatekeeper model won't work without a gatekeeper.  We
can try working towards such new models, if we believe we can achieve
them.  But until we get close to that goal, discussing such radically
different models will remain a pipe dream at best.

> Re current pattern of 6 month code freeze: This seems to be a manifestation of that fact that once a sufficient collection of new features have been accumulated we recognize in our heart of hearts that our code is not ready for release.  At a minimum it has not necessarily been built on all of the platforms we claim to support, various feature have received waivers (hence are incomplete) and lots of documentation remains to be written.  Moving more rapidly from cutting a release branch to asserting a release-ready tarball requires addressing those cultural patterns.

There is no 6-month code freeze.  There are long pretest periods for
major version releases.  Emacs 21.1, which included a complete rewrite
of the display engine and many other significant changes, took 11
months since the first pretest till the release.  Emacs 22.1 took 7.5
months, Emacs 23.1 5 months, Emacs 24.1 8.5 months.  A large portion
of that time is spent updating the documentation to match the changes,
especially documenting the new features, and then proof-reading the
manuals.  (I'm guessing at Mathworks you don't have developers writing
and proof-reading the documentation, you have special staff for that.
We don't have such luxury.)  Slashing those months-long pretest
periods means requesting that every user-visible change be accompanied
by the corresponding documentation changes -- are we prepared to do
that?  Do we have enough people in place to review and comment on
those documentation changes if and when they do come?  Don't forget
that some of us cannot write good English documentation and/or don't
command written technical English enough to produce something decent.
And I didn't even start talking about how many of us _want_ to deal
with documentation even if their English is fine.

The other reason for the long pretests is indeed the need to test the
upcoming release on a wide array of different platforms and system
configurations.  We used to have a group of pretesters who represented
the supported platforms, and who were instrumental in that process.
We no longer seem to have such a group, and the platforms we care
about changed anyway.  As result, our coverage of the possible
configurations is mostly random, and we can no longer be sure that a
long enough pretest period will shake out enough bugs.  Note that it
is not enough to just build on a platform, you must actually use Emacs
there to find bugs.  And since a typical user uses only a small
fraction of Emacs features, a single tester per platform/configuration
is not enough for thorough testing.  Some work in that area is needed,
e.g. to identify the features which might be platform-dependent
(example: file notifications), and then compile a list of platforms
that share the same implementation of a given feature.  Then we need a
few testers for each such group, and we need to inform them about each
pretest (there used to be a mailing list for that) and be sure to
collect their reports.

Lots of job opportunities there, and too few volunteers...