Re: What a modern collaboration toolkit looks like

["Eric S. Raymond" <esr@thyrsus.com>]

Alan Mackenzie <acm@muc.de>:
> Emacs is an atypical, very old, piece of free software.  Wesnoth seems to
> be about 5 years old.  (I haven't found the repository online.)  This has
> some bearing on the differences in development processes.

Indeed.  It means we have a lot of history and bad habits to unburden
ourselves of, and that's a problem Wesnoth does not have as
acutely. (It dates from 2003.)

> > a) That change can be referenced, through its Subversion revision ID.  
> >    (So, for example, "Hey boucman, your r22615 broke linger mode!")
> 
> Hopefully, people are considerate enough not to do this too often;
> continually having to look somewhere else to get context is not nice.

In fact that particular situation is rare.  My point was that Wesnoth's
communications tools help us correct such problems quickly.  What a 
fighter pilot would call our "OODA loop" is very fast.
 
> > b) That change can be backed out as a unit.
> 
> That's fine if nearly all changes are independent.  In Emacs, I think,
> this is not the case.

It's actually very useful even when changes are intertangled.  For one
thing, it makes isolating regressions via a bisection search practical.
While this is theoretically possible under CVS, there is just enough
friction to make it something people seldom or never actually do.

> Such easy backing out could lead to problems.

I find that the problems of easy backout are far outweighed by its benefits.

As a recent example: my lieutenant on one of my other projects (GPSD) committed
a change that broke our regression tests.  It was a day and a half and about
a dozen commits later when I noticed this.  It matters that I was *immediately*
able to back out both the bad commits with high confidence that I was not
leaving the code in an inconsistent state.  (And rap him on the knuckles,
but that was a social act and not a technical one.)
 
> > 2. My commit is also echoed to the IRC channel, where there are almost
> > never fewer than three or four devs hanging out as they hack, chatting
> > in real time.  Review tends to happen instantly.
> 
> What about fixes that are too big for instant review?  Emacs has lots of
> bugs (and often fixes ;-) that are barely tractable, so that the slower
> pace of email seems a better fit.

IRC functions as a kind of triage on such changes.  If we can't resolve
a problem in real time, we fall back to the mailing list.  This happens
much less often than I myself would have expected before I became used
to this style.

I'll admit, it did take me some adjustment to get used to.  The
Wesnoth devs are mostly kids who have grown up in a post-MTV world,
used to constant multitasking.  I just turned 50.  When I first
contemplated getting involved, about nine months ago now, I had to
wrestle with some fear that (a) I would be distracted and losing focus
all the time, and (b) worse, I might discover that I was simply too
old and lacked the mental agility to keep up.

The second, especially, would have been crushing. I had to screw up my
courage to try doing things their way lest I discover myself to
be inadequate.

In fact, both fears proved groundless.  Nowadays I routinely have an IRC
client open in five tabs to three different projects, and it all just
becomes part of my flow.  I think it matters a lot that IRC chat is 
silent -- I absolutely can't imagine imagine maintaining concentration
while multiplexing that many voice channels.

And far from finding I can't keep up, I've discovered that I like the
stimulation.  I grok how the kids feel about this, because mailing-list-only
projects have started to seem slow and boring to me, too.

> I've no experience of IIRC.  But doesn't this way of working mean
> continually dancing from one thing to another?  I don't do that very
> well. 

See above.  I myself was afraid I wouldn't be able to handle this.  But 
I adapted, and now I feel really liberated by having done so.  

>       The "hyperlinks" bit sounds like you have to look at 3 or 4 things
> simultaneously to be able to synthesise a context.  How does this working
> style work for difficult problems?

Same way it does for easy ones.  The difference is that with older, slower
workflows we had to absorb all those bits of context more or less serially
and buffer them in our heads for relatively long periods of time.  I don't 
have to do that as much now -- it's more about keeping track of where the
context is and quickly swapping in the parts I need at any given time.

> > Yet a third is that when we decide to do it, we can converge on a
> > releasable state with almost absurd ease.  Like, Ivanovic (our release
> > manager) will announce "Point release scheduled this coming Wednesday"
> > and everyone will pretty much flip into bug-stomping mode.  The tracker
> > bug list tends to shrink dramatically when this happens -- not only do
> > we get prepared for release but *we know we've done so*.
> 
> Eric, how well do you think this could work at all for Emacs?

I'm going to stop waving my hands and run some numbers.  According to
sloccount, the C core of Emacs is just about twice the size of
Wesnoth's C++.  Assuming the usual quadratic rise in bug density with
LOC, the equivalent release tempo would be three months.

But I think that's an overestimate.  Our devs are more capable, and a fair
amount of the Emacs core is either effectively dead (like, say, the VMS
support) or so stable that it never needs to be touched.  Then there's
the C vs. C++ difference.  

All in all, I think six weeks is a reasonable guess for where we'd end up.

"But..." I hear you say "you're ignoring the Lisp!".  Yes, I am -- and
that's because one thing rather clear from the Changelogs and NEWS
files is that the Lisp code is not where the long-term release
blockers tend to pop up.

> > More generally, development happens *fast*.  I don't have to wait weeks
> > to find out whether other devs think of an idea.  Commonly I know
> > within minutes.
> 
> On emacs-devel, this takes hours, not weeks.  At any given time, most
> Emacs developers are either asleep or doing other things (like earning
> their living).  Doesn't this quick-fire style end up excluding lots of
> hackers from participation?

It doesn't seem to work that way in practice.  Or, at least, I don't
hear hackers bitching about not being involved in decisions.  What the
core devs (including me) seem to do is keep one eye on the project IRC channel 
more or less constantly while doing those other things.
 
> > The Wesnoth devs are good but not exceptionally so, and we're weighed
> > down by a crappy implementation language (C++).
> 
> Yuck!!

Indeed.  I've had to become competent at it.  This has been painful.

> > Nevertheless our productivity, in terms of goals achieved per hour of
> > work, is quite high.  That's because our collaboration tools support
> > everything we do without imposing noticeable process friction.  This
> > makes us dramatically more effective as individuals and as a group than
> > we could possibly be without them.
> 
> Might it also be because your code base is still young, clean and
> flexible?

It's an appealing theory, but...no.  The Wesnoth code is not at all clean
internally, and the Wesnoth devs themselves often bitch about its rigidity.

There are two reasons for this.  One is C++; enough said. The other is
that the Wesnoth devs are, on average, somewhat less capable and a lot
less experienced than Emacs's have been -- and their most serious
weakness tends to be a very limited understanding of design for
maintainability.

The result, unfortunately, is that while the Wesnoth code is much younger
than of Emacs, it has already achieved a comparable cruft level :-(.

> > Lessons for the Emacs project?  Hell, yes.  But I'm not going to write
> > that rant until y'all have had a little time to absorb this description
> > of how things can be.
> 
> I'll look forward to that!  We'd all welcome a process for releasing
> every few months rather than twice a decade.

Yeah.  That would be good.  

My worst-case estimate, with modern tools, would be a three-to-four
month release cycle. 
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>