Periodical releases

Periodical releases

[Carsten Mattner]

Did the Emacs maintainers ever consider introducing a time based
release process?

I've been prompted to ask this because of the recent Org merge
discussion.

Maybe 2 or 4 releases as a minimum per year with less major
changes in each? That would have made the concerns in the
Org merge thread moot, wouldn't it?

It's even possible to have a 1 or 2 weeks merge window with
4 weeks of stabilization and no other merges.

To some on the list this may seem familiar from other projects
they contribute to.

Re: Periodical releases

[Stefan Monnier]

> Maybe 2 or 4 releases as a minimum per year with less major
> changes in each?

For Emacs-21, the main time constraint was to get the new redisplay
engine stable (with support for proportional fonts and such).
Can't remember any such large change for Emacs-22, off hand.
But for Emacs-23, there was a similar issue with the change to a utf-8
based internal representation and the new font handling.
For Emacs-24, it was the bidi support mostly.

These are fairly significant structural changes which are difficult to
perform piecemeal and tend to introduce significant breakage which takes
months if not years to test&debug (maybe partly for lack of a good
regression test suite, but also because of very complex semantics, most
of which is the result of accidental interferences between
"independent" features).


        Stefan

Re: Periodical releases

[Carsten Mattner]

On Sun, Jan 1, 2012 at 11:41 PM, Stefan Monnier
<monnier@iro.umontreal.ca> wrote:
>> Maybe 2 or 4 releases as a minimum per year with less major
>> changes in each?
>
> For Emacs-21, the main time constraint was to get the new redisplay
> engine stable (with support for proportional fonts and such).
> Can't remember any such large change for Emacs-22, off hand.
> But for Emacs-23, there was a similar issue with the change to a utf-8
> based internal representation and the new font handling.
> For Emacs-24, it was the bidi support mostly.
>
> These are fairly significant structural changes which are difficult to
> perform piecemeal and tend to introduce significant breakage which takes
> months if not years to test&debug (maybe partly for lack of a good
> regression test suite, but also because of very complex semantics, most
> of which is the result of accidental interferences between
> "independent" features).

The solution for that is to let it evolve in a branch for longer than
one release cycle while merging finished changes to the release
branch and deliver those features sooner.
That way finished features like say package support, built-in colour
theme support, cc-mode and other mode updates, etc., which are less
invasive, are delivered in a stable release faster.

Re: Periodical releases

[Eli Zaretskii]

> Date: Mon, 2 Jan 2012 11:40:20 +0100
> From: Carsten Mattner <carstenmattner@googlemail.com>
> Cc: Emacs developers <emacs-devel@gnu.org>
> 
> > These are fairly significant structural changes which are difficult to
> > perform piecemeal and tend to introduce significant breakage which takes
> > months if not years to test&debug (maybe partly for lack of a good
> > regression test suite, but also because of very complex semantics, most
> > of which is the result of accidental interferences between
> > "independent" features).
> 
> The solution for that is to let it evolve in a branch for longer than
> one release cycle

Been there, done that.  In Emacs, this generally means leave the code
to bit-rot into oblivion.  Examples:

  . The person who wrote the multi-tty code disappeared after merging
    the branch onto the trunk; if we would have waited longer with the
    merge, we would have no one who knew the code enough to merge it
    and take care of merge complications.

  . The bidi branch actually did bitrot, for at least 3 years, until
    yours truly decided it was now or never, and somehow managed to
    find time to do the job.  Knowing now how much effort it took, I
    can assure you that work would never have been done had Stefan and
    Chong not supported me all the way and urged me to merge early.  A
    year from now, I cannot even promise I will have enough time and
    health left to do anything comparable.

> That way finished features like say package support, built-in colour
> theme support, cc-mode and other mode updates, etc., which are less
> invasive, are delivered in a stable release faster.

That's a nice theory, but implementing it in practice needs a much
larger and probably different organization than Emacs development we
have now.  Unlike many other projects, Emacs is a hodgepodge of a
myriad of separate and almost independent subsystems, many of which
require very specific domain knowledge and target different audiences,
sometimes quite narrow ones.  Exposing significant changes to a wide
audience is perhaps the only practical way of testing those changes
efficiently; leaving them on a branch would mean features remain
largely untested (read: buggy) for many months if not years.

If we want to move in the direction of periodical releases, we will
have to come up with a plan that includes organizational and
procedural changes, and we will have to convince ourselves that such a
plan is doable in practice.  First step in the plan should be to bring
much more developers on board.

Re: Periodical releases

[Carsten Mattner]

On Mon, Jan 2, 2012 at 12:36 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Mon, 2 Jan 2012 11:40:20 +0100
>> From: Carsten Mattner <carstenmattner@googlemail.com>
>> Cc: Emacs developers <emacs-devel@gnu.org>
>>
>> > These are fairly significant structural changes which are difficult to
>> > perform piecemeal and tend to introduce significant breakage which takes
>> > months if not years to test&debug (maybe partly for lack of a good
>> > regression test suite, but also because of very complex semantics, most
>> > of which is the result of accidental interferences between
>> > "independent" features).
>>
>> The solution for that is to let it evolve in a branch for longer than
>> one release cycle
>
> Been there, done that.  In Emacs, this generally means leave the code
> to bit-rot into oblivion.  Examples:
>
>  . The person who wrote the multi-tty code disappeared after merging
>    the branch onto the trunk; if we would have waited longer with the
>    merge, we would have no one who knew the code enough to merge it
>    and take care of merge complications.
>
>  . The bidi branch actually did bitrot, for at least 3 years, until
>    yours truly decided it was now or never, and somehow managed to
>    find time to do the job.  Knowing now how much effort it took, I
>    can assure you that work would never have been done had Stefan and
>    Chong not supported me all the way and urged me to merge early.  A
>    year from now, I cannot even promise I will have enough time and
>    health left to do anything comparable.
>
>> That way finished features like say package support, built-in colour
>> theme support, cc-mode and other mode updates, etc., which are less
>> invasive, are delivered in a stable release faster.
>
> That's a nice theory, but implementing it in practice needs a much
> larger and probably different organization than Emacs development we
> have now.  Unlike many other projects, Emacs is a hodgepodge of a
> myriad of separate and almost independent subsystems, many of which
> require very specific domain knowledge and target different audiences,
> sometimes quite narrow ones.  Exposing significant changes to a wide
> audience is perhaps the only practical way of testing those changes
> efficiently; leaving them on a branch would mean features remain
> largely untested (read: buggy) for many months if not years.
>
> If we want to move in the direction of periodical releases, we will
> have to come up with a plan that includes organizational and
> procedural changes, and we will have to convince ourselves that such a
> plan is doable in practice.  First step in the plan should be to bring
> much more developers on board.

okay, this may not work for the current development workflow.

Another proposal:
Don't wait until "perfection" and release trunk more often
with bug releases if needed. Emacs trunk has never been
unstable for me. I'm even using the NS port and it's still
stable.
I would have less of a need to build emacs manually if
there were more release and therefore standard emacs
Linux distros was more up to date.

Re: Periodical releases

[Eli Zaretskii]

> Date: Mon, 2 Jan 2012 13:57:50 +0100
> From: Carsten Mattner <carstenmattner@googlemail.com>
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> Don't wait until "perfection" and release trunk more often
> with bug releases if needed. Emacs trunk has never been
> unstable for me. I'm even using the NS port and it's still
> stable.

That could work, but it still needs:

 . a much more disciplined commit process, probably coupled with 1
   more branch (3 instead of just 2)

 . more active patch review process (currently, lots of serious bugs
   are detected months if not years after being introduced)

 . several volunteers to review and update the manuals to keep them in
   sync with development

You see, even for few months worth of development, the amount of
changes in the Emacs repository is so large that it is impossible to
release a dependable and accurately documented version without much
more active "continuous integration" style maintenance.  You may
consider it stable enough for your workflows, but someone who
constantly uses a certain mode that became broken by a release will
feel otherwise.

Now, a simple calculation will show that if it takes about a year to
finish a pretest of a major release, doing a release every 2 months
will need at least a month of pretest-style fixups.  Unless this month
is invested in parallel with development, 2 months become 3 months,
which then need 1.5 months of fixups, and so on, ad nauseam.

And without even a test suite with good coverage, how do you do a
continuous integration?

IOW, even this limited goal needs preconditions to become practical.
Volunteers are welcome.

RE: Periodical releases

[Drew Adams]

> Don't wait until "perfection" and release trunk more often
> with bug releases if needed. 

No, no, no, please.  Just the opposite.
Bake Emacs _more_ fully before releasing it.

Get it right.  Document it well.  Mention all user-visible changes in NEWS.  Fix
outstanding bugs.

Richard had exactly the right approach to releasing Emacs, IMO.  He was attacked
by some because they felt the release cycle was too short.  I, for one,
appreciated his thoroughness and insistence on high quality.

> Emacs trunk has never been unstable for me.

Oh.  So please continue to use the trunk.

It's been quite unstable (not to mention incomplete) at various times for
others; believe me.  And that's been true forever.  It's normal.  To expect
anything else is naive, IMHO.

> I'm even using the NS port and it's still stable.

So please continue to use it.

> I would have less of a need to build emacs manually if
> there were more release and therefore standard emacs
> Linux distros was more up to date.

The problem you raise here is apparently one of the difficulty/nuisance of
building Emacs.  It is not about how often to publish releases.

Publishing non-release MS Windows binaries periodically has been _very_ helpful
(thank you again to those who have created and posted the builds).  At least on
that platform, that is a solution to the problem you raise.

If the same cannot be done for other platforms then the solution would be to
somehow simplify the difficulty/nuisance of building Emacs on those platforms.

But in no case should that difficulty/nuisance of building be an excuse for
releasing the product before it is fully baked.

Re: Periodical releases

[chad]

As far as I can tell, the primary motivator for more frequent releases of emacs itself is `so emacs doesn't drift so far from its sub-parts', like as Gnus, Org, CEDET, CC-mode (in the past), etc.  This is a laudable goal, but emacs doesn't seem to currently have the developer cycles necessary to handle this - releases are frequently held up by documentation changes and reviews even now, with the infrequent releases we have.

Hopefully, the included package system can cover this gap, once it's fully spread through the userbase; users will be able to easily opt-in to newer Gnus releases (for example) via elpa(s)/packages.  If we get this working reliably, we could even remove some packages from the emacs core, in favor of more frequent package-based releases.  While this model might have once seemed burdensome to users, several years of such systems (for example, in web browser extension updates) will have already trained them to deal with such things before emacs starts doing something similar.

That's my hope, anyway.

*Chad

Re: Periodical releases

[Carsten Mattner]

On Mon, Jan 2, 2012 at 8:54 PM, chad <yandros@gmail.com> wrote:
> As far as I can tell, the primary motivator for more frequent releases
> of emacs itself is `so emacs doesn't drift so far from its sub-parts', like
> as Gnus, Org, CEDET, CC-mode (in the past), etc.  This is a laudable
> goal, but emacs doesn't seem to currently have the developer cycles
> necessary to handle this - releases are frequently held up by documentation
> changes and reviews even now, with the infrequent releases we have.
>
> Hopefully, the included package system can cover this gap, once it's fully spread
> through the userbase; users will be able to easily opt-in to newer Gnus releases
> (for example) via elpa(s)/packages.  If we get this working reliably, we could even
> remove some packages from the emacs core, in favor of more frequent
> package-based releases.  While this model might have once seemed burdensome
> to users, several years of such systems (for example, in web browser extension
> updates) will have already trained them to deal with such things before emacs
> starts doing something similar.
>
> That's my hope, anyway.

I fully support removing non-essential modes from emacs, now that
packaging is supported officially in a consistent way.

Re: Periodical releases

[Eric Schulte]

>> I would have less of a need to build emacs manually if
>> there were more release and therefore standard emacs
>> Linux distros was more up to date.
>
> The problem you raise here is apparently one of the difficulty/nuisance of
> building Emacs.  It is not about how often to publish releases.
>
> Publishing non-release MS Windows binaries periodically has been _very_ helpful
> (thank you again to those who have created and posted the builds).  At least on
> that platform, that is a solution to the problem you raise.
>

It seems that the fix here is not to change the Emacs development and
release process, but rather to encourage package maintainers to provide
packages tracking the Emacs trunk.  The debian emacs-snapshot package
described as "The GNU Emacs editor (development snapshot)" returns the
following which is certainly many months (maybe years?) out of date.

$ emacs --version
GNU Emacs 23.2.1

Best,

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: Periodical releases

[Carsten Mattner]

On Mon, Jan 2, 2012 at 9:29 PM, Eric Schulte <eric.schulte@gmx.com> wrote:
>>> I would have less of a need to build emacs manually if
>>> there were more release and therefore standard emacs
>>> Linux distros was more up to date.
>>
>> The problem you raise here is apparently one of the difficulty/nuisance of
>> building Emacs.  It is not about how often to publish releases.
>>
>> Publishing non-release MS Windows binaries periodically has been _very_ helpful
>> (thank you again to those who have created and posted the builds).  At least on
>> that platform, that is a solution to the problem you raise.
>>
>
> It seems that the fix here is not to change the Emacs development and
> release process, but rather to encourage package maintainers to provide
> packages tracking the Emacs trunk.  The debian emacs-snapshot package
> described as "The GNU Emacs editor (development snapshot)" returns the
> following which is certainly many months (maybe years?) out of date.
>
> $ emacs --version
> GNU Emacs 23.2.1

This is one of the things which I always hoped Gentoo, Exherbo or rPath
would solve. Of those Gentoo and Exherbo seem to be the most likely
solution.

Fedora stable is a good way to get current software while not having
to rely on Debian sid.

Debian has the advantage of doko releasing gcc-snapshot regularly
as a nice way to check for future warnings in unreleased gcc versions.

Re: Periodical releases

[Lluís]

Drew Adams writes:
[...]
>> I would have less of a need to build emacs manually if
>> there were more release and therefore standard emacs
>> Linux distros was more up to date.

> The problem you raise here is apparently one of the difficulty/nuisance of
> building Emacs.  It is not about how often to publish releases.

> Publishing non-release MS Windows binaries periodically has been _very_ helpful
> (thank you again to those who have created and posted the builds).  At least on
> that platform, that is a solution to the problem you raise.

> If the same cannot be done for other platforms then the solution would be to
> somehow simplify the difficulty/nuisance of building Emacs on those platforms.

> But in no case should that difficulty/nuisance of building be an excuse for
> releasing the product before it is fully baked.

If building is the only problem, then you might like this (note: debian only):

   http://emacs.naquadah.org/


Lluis

-- 
 "And it's much the same thing with knowledge, for whenever you learn
 something new, the whole world becomes that much richer."
 -- The Princess of Pure Reason, as told by Norton Juster in The Phantom
 Tollbooth

Re: Periodical releases

[Carsten Mattner]

On Mon, Jan 2, 2012 at 8:05 PM, Drew Adams <drew.adams@oracle.com> wrote:
>> Don't wait until "perfection" and release trunk more often
>> with bug releases if needed.
>
> No, no, no, please.  Just the opposite.
> Bake Emacs _more_ fully before releasing it.
>
> Get it right.  Document it well.  Mention all user-visible changes in NEWS.  Fix
> outstanding bugs.
>
> Richard had exactly the right approach to releasing Emacs, IMO.  He was attacked
> by some because they felt the release cycle was too short.  I, for one,
> appreciated his thoroughness and insistence on high quality.
>
>> Emacs trunk has never been unstable for me.
>
> Oh.  So please continue to use the trunk.
>
> It's been quite unstable (not to mention incomplete) at various times for
> others; believe me.  And that's been true forever.  It's normal.  To expect
> anything else is naive, IMHO.
>
>> I'm even using the NS port and it's still stable.
>
> So please continue to use it.
>
>> I would have less of a need to build emacs manually if
>> there were more release and therefore standard emacs
>> Linux distros was more up to date.
>
> The problem you raise here is apparently one of the difficulty/nuisance of
> building Emacs.  It is not about how often to publish releases.
>
> Publishing non-release MS Windows binaries periodically has been _very_ helpful
> (thank you again to those who have created and posted the builds).  At least on
> that platform, that is a solution to the problem you raise.
>
> If the same cannot be done for other platforms then the solution would be to
> somehow simplify the difficulty/nuisance of building Emacs on those platforms.
>
> But in no case should that difficulty/nuisance of building be an excuse for
> releasing the product before it is fully baked.

I don't mind building my emacs.
It's a nuisance to have years old Emacs version in distributions.
To get a less buggy Emacs a better approach is releasing it
more often so that there are more testers who wouldn't even consider
building manually or install a prebuilt snapshot from trunk.

RE: Periodical releases

[Drew Adams]

> > But in no case should that difficulty/nuisance of building 
> > be an excuse for releasing the product before it is fully baked.
> 
> I don't mind building my emacs.

Great.  But you did cite "less of a need to build emacs manually" as an argument
why releases should be more frequent.

> It's a nuisance to have years old Emacs version in distributions.

Why?  What's the nuisance?  Anyway, you build Emacs yourself from trunk as often
as you like, and you don't mind doing so.  Where's the beef?

> To get a less buggy Emacs a better approach is releasing it
> more often so that there are more testers who wouldn't even consider
> building manually or install a prebuilt snapshot from trunk.

Bon sang mais c'est bien sur!  Ye olde cathedrale vs bazaar song...

There are already plenty of pre-release bugs reported, needing to be fixed.
Maybe when they are fixed you can argue that Emacs should be released to get
more testers.  For now, there seem to be enough people using pre-release builds
that give rise to bug reports.  More input is always welcome of course, but
Emacs users are not _only_ testers and fixers.

FWIW, where I work the default, prebuilt Emacs for developers on GNU/Linux is
Emacs 21.3.1!  And my impression is that few of them bother to build a more
recent version.  They are seemingly not clamouring for a new version, especially
since there are already 6 (!) releases available more recent than the one they
use.  (Some of them still use XEmacs, and apparently happily so.)

It's sometimes good to step back a bit and imagine that not all, or even most,
Emacs users are involved in, or even interested in, Emacs development.
Ponder...  Hard to fathom, I know.

Many, probably most, users use Emacs the way I use my car: just to get around
and do the things I need to do.  I don't work on the engine; I don't want to
understand the electrical system; and I upgrade only when the old horse just
can't make it around the block anymore even with a little coaxing.

Now I'm sure carmakers do not consider me their prime marketing target, and I'm
perhaps not a typical car owner.  But they really do need to sell new cars, to
stay in business.  They need new models each year - periodic releases.

We're not selling cars, and we're not even in business.  We're not in a race or
a competition for market share.  We cannot go out of business.  We improve Emacs
only because we want it to be better or we enjoy improving it or both -
intrinsic reward.  And there's no hurry in that.

Re: Periodical releases

[Carsten Mattner]

[-- Attachment #1: Type: text/plain, Size: 2883 bytes --]

On Monday, January 2, 2012, Drew Adams <drew.adams@oracle.com> wrote:
>> > But in no case should that difficulty/nuisance of building
>> > be an excuse for releasing the product before it is fully baked.
>>
>> I don't mind building my emacs.
>
> Great.  But you did cite "less of a need to build emacs manually" as an
argument
> why releases should be more frequent.

So that distros don't carry ancient emacs versions.

>> It's a nuisance to have years old Emacs version in distributions.
>
> Why?  What's the nuisance?  Anyway, you build Emacs yourself from trunk
as often
> as you like, and you don't mind doing so.  Where's the beef?

I'd like to use the packaged emacs if it wasn't too old (for me).

>> To get a less buggy Emacs a better approach is releasing it
>> more often so that there are more testers who wouldn't even consider
>> building manually or install a prebuilt snapshot from trunk.
>
> Bon sang mais c'est bien sur!  Ye olde cathedrale vs bazaar song...
>
> There are already plenty of pre-release bugs reported, needing to be
fixed.
> Maybe when they are fixed you can argue that Emacs should be released to
get
> more testers.  For now, there seem to be enough people using pre-release
builds
> that give rise to bug reports.  More input is always welcome of course,
but
> Emacs users are not _only_ testers and fixers.

Not much to argue here.

> FWIW, where I work the default, prebuilt Emacs for developers on
GNU/Linux is
> Emacs 21.3.1!  And my impression is that few of them bother to build a
more
> recent version.  They are seemingly not clamouring for a new version,
especially
> since there are already 6 (!) releases available more recent than the one
they
> use.  (Some of them still use XEmacs, and apparently happily so.)
>
> It's sometimes good to step back a bit and imagine that not all, or even
most,
> Emacs users are involved in, or even interested in, Emacs development.
> Ponder...  Hard to fathom, I know.
>
> Many, probably most, users use Emacs the way I use my car: just to get
around
> and do the things I need to do.  I don't work on the engine; I don't want
to
> understand the electrical system; and I upgrade only when the old horse
just
> can't make it around the block anymore even with a little coaxing.
>
> Now I'm sure carmakers do not consider me their prime marketing target,
and I'm
> perhaps not a typical car owner.  But they really do need to sell new
cars, to
> stay in business.  They need new models each year - periodic releases.
>
> We're not selling cars, and we're not even in business.  We're not in a
race or
> a competition for market share.  We cannot go out of business.  We
improve Emacs
> only because we want it to be better or we enjoy improving it or both -
> intrinsic reward.  And there's no hurry in that.

If we sold cars, looks would b more important than function. It's good this
ain't a car.

[-- Attachment #2: Type: text/html, Size: 3388 bytes --]

Re: Periodical releases

[Dave Abrahams]

on Mon Jan 02 2012, "Drew Adams" <drew.adams-AT-oracle.com> wrote:

>> Don't wait until "perfection" and release trunk more often
>> with bug releases if needed. 
>
> No, no, no, please.  Just the opposite.
> Bake Emacs _more_ fully before releasing it.
>
> Get it right.  Document it well.  Mention all user-visible changes in
> NEWS.  Fix outstanding bugs.
>
> Richard had exactly the right approach to releasing Emacs, IMO.  He
> was attacked by some because they felt the release cycle was too
> short.  I, for one, appreciated his thoroughness and insistence on
> high quality.

Is it not possible to have thorougness and high quality with more
frequent releases of smaller scope?  Maybe the problem (if there is one)
is that trunk is allowed to drift too far from a releasable state.  I'm
just saying.

-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com

Re: Periodical releases

[Chong Yidong]

Dave Abrahams <dave@boostpro.com> writes:

> Is it not possible to have thorougness and high quality with more
> frequent releases of smaller scope?  Maybe the problem (if there is
> one) is that trunk is allowed to drift too far from a releasable
> state.  I'm just saying.

Features like bidi can't really be added piecemeal, and this goes back
to Eli's point about bit-rotting feature trees.

Frankly, I'm not convinced there's any problem.  Emacs 23.3 was released
less than one year ago, and we are already well into the pretest for
24.1.  Doing considerably better would require resources we don't have.
Note that all those projects that are being held up as models for their
periodic release systems have people working on them full time!

Re: Periodical releases

[Dave Abrahams]

on Wed Jan 04 2012, Chong Yidong <cyd-AT-gnu.org> wrote:

> Dave Abrahams <dave@boostpro.com> writes:
>
>> Is it not possible to have thorougness and high quality with more
>> frequent releases of smaller scope?  Maybe the problem (if there is
>> one) is that trunk is allowed to drift too far from a releasable
>> state.  I'm just saying.
>
> Features like bidi can't really be added piecemeal, and this goes back
> to Eli's point about bit-rotting feature trees.
>
> Frankly, I'm not convinced there's any problem.  Emacs 23.3 was released
> less than one year ago, and we are already well into the pretest for
> 24.1.  Doing considerably better would require resources we don't have.
> Note that all those projects that are being held up as models for their
> periodic release systems have people working on them full time!

For what it's worth, Boost has no full-time people, and it releases four
times yearly.  It wasn't always that way, though: for a while there we
had a period of painfully slow multi-year releases.

-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com

Re: Periodical releases

[Óscar Fuentes]

Dave Abrahams <dave@boostpro.com> writes:

> For what it's worth, Boost has no full-time people, and it releases four
> times yearly.  It wasn't always that way, though: for a while there we
> had a period of painfully slow multi-year releases.

Boost is anything but similar to Emacs from the project manager POV.

After using the development sources of Emacs for years with great
pleasure (it is, possibly, the most stable piece of software after the
OS kernel itself on any of my machines, if we take into account its
complexity and rate of defects) one is tempted to ask for more frequent
releases, as it is on a release-ready state most of the time, mainly if
we accept the quality level of other popular software that is well below
Emacs'. But then one wonders if that semi-permanent release-ready state
is not caused by the long feature-freeze periods which stabilizing
effect pervades the subsequent years of development.

The fact that Emacs is used by its own developers helps a lot to its
stability, but going into a permanent feature-frenzy mode probably would
cause a steady deterioration on Emacs' quality.

As mentioned on a previous message, IMO offloading some of the big Lisp
packages from the Emacs distribution would benefit the developers and
the users.

Re: Periodical releases

[Eli Zaretskii]

> From: Dave Abrahams <dave@boostpro.com>
> Date: Wed, 04 Jan 2012 22:36:24 -0500
> Cc: emacs-devel@gnu.org
> 
> For what it's worth, Boost has no full-time people, and it releases four
> times yearly.

How tightly are the various libraries in Boost connected?  I.e., if
one of them is broken, how many others will become broken as result?

Core features in Emacs are likely to break many different and
unrelated features.  Emacs is a program, not a library.

The "People" page of the Boost site shows 48 developers.  How
intensely does each one of them work on Boost?  For that matter, how
many hours do you personally spend on Boost weekly, if I may ask?

I would be surprised if Emacs had more than 10 to 15 active
maintainers at any given time.  I don't know how many hours others
spend, but I can only devote about maybe 5 hours or less each week.

Re: Periodical releases

[Dave Abrahams]

on Thu Jan 05 2012, Eli Zaretskii <eliz-AT-gnu.org> wrote:

>> From: Dave Abrahams <dave@boostpro.com>
>> Date: Wed, 04 Jan 2012 22:36:24 -0500
>> Cc: emacs-devel@gnu.org
>> 
>> For what it's worth, Boost has no full-time people, and it releases four
>> times yearly.
>
> How tightly are the various libraries in Boost connected?  I.e., if
> one of them is broken, how many others will become broken as result?

Some are extremely loose and some have direct dependencies.  But one
thing we have had practically since the beginning is a fairly complete
set of regression tests.  Without that, I don't see how our pace of
releases would be possible.

> The "People" page of the Boost site shows 48 developers.  How
> intensely does each one of them work on Boost?  

I think Ohloh has the best information on this:
http://www.ohloh.net/p/boost

> For that matter, how many hours do you personally spend on Boost
> weekly, if I may ask?

I haven't had much time to do programming for Boost in the past few
years.  I'm not sure what the relevance is, though...

> I would be surprised if Emacs had more than 10 to 15 active
> maintainers at any given time.  I don't know how many hours others
> spend, but I can only devote about maybe 5 hours or less each week.

Arguably, the more people contributing code, the harder the job of
managing stable releases gets :-)

-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com

Re: Periodical releases

[Bastien]

Chong Yidong <cyd@gnu.org> writes:

> Frankly, I'm not convinced there's any problem.

I can see two problems:

1) merging Org into Emacs trunk is a bit painful.  I'm still looking for
   a good workflow -- given the fact that Org is using git and Emacs is
   using bzr.  Any concret suggestion here is welcome.  I didn't grok
   Stefan's last suggestion.

2) I will probably release Org 8.0 before the end of the feature freeze
   (say march or april).  Not having it in Emacs 24.1 is not such a big
   deal, agreed, but it creates a situation where people often install
   Org on top of the one that comes with Emacs, which doesn't feel
   optimal and leads to many questions, because of conflicting installs.

Both are separate, and I can try to take care of (1) myself.

As for (2), let me refine my suggestion: what about a main feature
freeze for core functionalities and another one for big modules like
Org, Gnus, etc.?  The feature freeze for those module could happen for 
a *limited* and *known* period of time -- say 2 months before the Emacs
release date.

> Emacs 23.3 was released
> less than one year ago, and we are already well into the pretest for
> 24.1.  Doing considerably better would require resources we don't have.
> Note that all those projects that are being held up as models for their
> periodic release systems have people working on them full time!

Yes.  It should also be noted that things are getting at a very good
pace since you and Stefan took over maintainership, I guess everyone
acknowledges it.

-- 
 Bastien

Re: Periodical releases

[Leo]

On 2012-01-05 17:31 +0800, Bastien wrote:
> 2) I will probably release Org 8.0 before the end of the feature freeze
>    (say march or april).  Not having it in Emacs 24.1 is not such a big
>    deal, agreed, but it creates a situation where people often install
>    Org on top of the one that comes with Emacs, which doesn't feel
>    optimal and leads to many questions, because of conflicting installs.
>
> Both are separate, and I can try to take care of (1) myself.

I think in future it may be desirable to make more use of the packaging
system now that we have one. It is like many of those GNU/Linux
distributions. The core developers/maintainers are concerned with the
core packages/functionality (it is better use of their time since they
usually have better insight of the project) and the rest can be taken
care by community members. For example the value of including org-mode
in the source tree is quite little. If it is distributed with the
packing system people can receive more timely bug fixes free of any
constraint of the release cycle.

Leo

Re: Periodical releases

[Carsten Mattner]

On Thu, Jan 5, 2012 at 11:11 AM, Leo <sdl.web@gmail.com> wrote:
> On 2012-01-05 17:31 +0800, Bastien wrote:
>> 2) I will probably release Org 8.0 before the end of the feature freeze
>>    (say march or april).  Not having it in Emacs 24.1 is not such a big
>>    deal, agreed, but it creates a situation where people often install
>>    Org on top of the one that comes with Emacs, which doesn't feel
>>    optimal and leads to many questions, because of conflicting installs.
>>
>> Both are separate, and I can try to take care of (1) myself.
>
> I think in future it may be desirable to make more use of the packaging
> system now that we have one. It is like many of those GNU/Linux
> distributions. The core developers/maintainers are concerned with the
> core packages/functionality (it is better use of their time since they
> usually have better insight of the project) and the rest can be taken
> care by community members. For example the value of including org-mode
> in the source tree is quite little. If it is distributed with the
> packing system people can receive more timely bug fixes free of any
> constraint of the release cycle.

I second that. Cedet, Gnus, Org, themes (hash verification issues?)...

Re: Periodical releases

[Jambunathan K]

> 1 For example the value of including org-mode in the source tree is
> quite little. If it is distributed with the packing system people can
> receive more timely bug fixes free of any constraint of the release
> cycle.

Org's daily builds are part of GNU ELPA for atleast around an year now.

> Leo

-- 

Re: Periodical releases

[Leo]

On 2012-01-05 20:56 +0800, Jambunathan K wrote:
>> 1 For example the value of including org-mode in the source tree is
>> quite little. If it is distributed with the packing system people can
>> receive more timely bug fixes free of any constraint of the release
>> cycle.
>
> Org's daily builds are part of GNU ELPA for atleast around an year now.

Point missed.

Leo

Re: Periodical releases

[Jambunathan K]

Leo <sdl.web@gmail.com> writes:

> On 2012-01-05 20:56 +0800, Jambunathan K wrote:
>>> 1 For example the value of including org-mode in the source tree is
>>> quite little. If it is distributed with the packing system people can
>>> receive more timely bug fixes free of any constraint of the release
>>> cycle.
>>
>> Org's daily builds are part of GNU ELPA for atleast around an year now.
>
> Point missed.

Too terse for me to understand - not that I care to understand.

> Leo
>
>
>

-- 

Re: Periodical releases

[Carsten Mattner]

On Thu, Jan 5, 2012 at 10:31 AM, Bastien <bzg@altern.org> wrote:
> 2) I will probably release Org 8.0 before the end of the feature freeze
>   (say march or april).  Not having it in Emacs 24.1 is not such a big
>   deal, agreed, but it creates a situation where people often install
>   Org on top of the one that comes with Emacs, which doesn't feel
>   optimal and leads to many questions, because of conflicting installs.

Out of curiosity. If I have 24.1 with Org 7.x and to use Org latest
I put that in ~/.emacs.d/<somewhere>, is there some potential
ambiguity/conflict/version-clash/mixup going on if I add that
local dir to the load path? Do I have to use some unknown-to-me
(prepend-to-load-path path)? I know that Vim has ~/.vim/* first in the
path list. What's it for Emacs?

Re: Periodical releases

[Eric Schulte]

Carsten Mattner <carstenmattner@googlemail.com> writes:

> On Thu, Jan 5, 2012 at 10:31 AM, Bastien <bzg@altern.org> wrote:
>> 2) I will probably release Org 8.0 before the end of the feature freeze
>>   (say march or april).  Not having it in Emacs 24.1 is not such a big
>>   deal, agreed, but it creates a situation where people often install
>>   Org on top of the one that comes with Emacs, which doesn't feel
>>   optimal and leads to many questions, because of conflicting installs.
>
> Out of curiosity. If I have 24.1 with Org 7.x and to use Org latest
> I put that in ~/.emacs.d/<somewhere>, is there some potential
> ambiguity/conflict/version-clash/mixup going on if I add that
> local dir to the load path? Do I have to use some unknown-to-me
> (prepend-to-load-path path)? I know that Vim has ~/.vim/* first in the
> path list. What's it for Emacs?

The `add-to-list' function will place any new directory at the front of
your load path.  In general adding the new version of Org-mode to your
load path, loading or requiring org, and then calling the `org-reload'
function should be sufficient.  For more information see [1].

However, when an older version of Org-mode (e.g., one shipped with
Emacs) is loaded first it defines all Org-mode variables.  Subsequent
versions of Org-mode will not update the values of these variables as a
defvar of an existing variable does not change its value.  This
frequently leads to buggy behavior.  This point would argue for
migration of Org-mode to ELPA in future releases.

Best -- Eric

Footnotes: 
[1]  http://orgmode.org/worg/org-faq.html#Keeping-current

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: Periodical releases

[Carsten Mattner]

On Thu, Jan 5, 2012 at 10:31 AM, Bastien <bzg@altern.org> wrote:
> Yes.  It should also be noted that things are getting at a very good
> pace since you and Stefan took over maintainership, I guess everyone
> acknowledges it.

Definitely, thanks to the new maintainers. You guys do a great job!

Re: Periodical releases

[Eli Zaretskii]

> From: Dave Abrahams <dave@boostpro.com>
> Date: Wed, 04 Jan 2012 21:34:25 -0500
> 
> Is it not possible to have thorougness and high quality with more
> frequent releases of smaller scope?

It is, with more resources than we currently have.

> Maybe the problem (if there is one) is that trunk is allowed to
> drift too far from a releasable state.

You cannot control that drift without a test suite with decent
coverage.  We don't have that yet.  We don't even have a concerted
effort to get there, AFAIK.  Volunteers are welcome, as always.

Re: Periodical releases

[Dave Abrahams]

on Thu Jan 05 2012, Eli Zaretskii <eliz-AT-gnu.org> wrote:

>> Maybe the problem (if there is one) is that trunk is allowed to
>> drift too far from a releasable state.
>
> You cannot control that drift without a test suite with decent
> coverage.  

Excellent point.

-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com

RE: Periodical releases

[Drew Adams]

> >> Don't wait until "perfection" and release trunk more often
> >> with bug releases if needed. 
> >
> > No, no, no, please.  Just the opposite.
> > Bake Emacs _more_ fully before releasing it.
> >
> > Get it right.  Document it well.  Mention all user-visible 
> > changes in NEWS.  Fix outstanding bugs.
> >
> > Richard had exactly the right approach to releasing Emacs, IMO.  He
> > was attacked by some because they felt the release cycle was too
> > short.  I, for one, appreciated his thoroughness and insistence on
> > high quality.
> 
> Is it not possible to have thorougness and high quality with more
> frequent releases of smaller scope?  Maybe the problem (if 
> there is one) is that trunk is allowed to drift too far from a
> releasable state.  I'm just saying.

Not when there are large changes made for significant features (new or old).

Consider, for Emacs 24, (1) the myriad user-visible effects of radically
changing the `display-buffer'/window behavior, and (2) the significant
development effort related to bidi (non-trivial, takes time to implement and
test).

Re: Periodical releases

[Stefan Monnier]

> The solution for that is to let it evolve in a branch for longer than
> one release cycle while merging finished changes to the release
> branch and deliver those features sooner.

I agree with Eli's explanation here (you can take the lexbind branch or
the concurrency branch as another example), but there's a more positive
way to look at it:

Take the "trunk" as the branch on which we do those long-term changes,
while the "emacs-23", or "emacs-24" branch is the stable branch from
which we make "regular" releases.

If you take this point of view as a starting point, you might come up
with suggestions for changes which we may be able to implement.


        Stefan

What's in a feature? (was: Periodical releases)

[Bastien]

Well, I am not in favor of a fixed calendar, a strict release
cycle -- or whatever looks like it, like strong deadlines.

While thinking again about it, I guess the problem lies in the
meaning of "feature" -- since this meaning can go from very little
things to core functionalities, there is no real test for each new
"feature" to know whether discarding it is a safe move or a silly
one.  And this leaves the door open for too many discussions.

So perhaps we could have three "feature freeze" steps: one for core
functionalities, and another one for features that depends on big
components like Gnus, Org, ERC, etc., and a last one for the rest
of the code.

Just a suggestion.

-- 
 Bastien

What's in a feature? (was: Periodical releases)

[Stephen J. Turnbull]

Bastien writes:

 > While thinking again about it, I guess the problem lies in the
 > meaning of "feature"

Bug:                        doesn't work as currently documented.
Undocumented functionality: undocumented but works anyway.
Feature:                    undocumented and doesn't work.

For completeness:
Miracle:                    working as currently documented.

All of the above must be viewed from the context of the *current*
content of the prospective release, *not* from the context of the
release after addition of the feature.

Re: What's in a feature?

[Bastien]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> Miracle:                    working as currently documented.

:)

My suggestion is just to distinguish between important miracles 
(like resurrection) and unintrusive ones (like giving bread to the
fishermen).

-- 
 Bastien

Re: What's in a feature?

[Juanma Barranquero]

On Wed, Jan 4, 2012 at 11:03, Bastien <bzg@altern.org> wrote:

> My suggestion is just to distinguish between important miracles
> (like resurrection) and unintrusive ones (like giving bread to the
> fishermen).

OTOH, resurrection is just fixing some damage, while giving bread to
the fishermen implies creation of matter out of nothing.

    Juanma