buildbots (was: eshell-defgroup. Do we really need this?)

[Ted Zlatanov <tzz@lifelogs.com>]

(subject changed, sorry I didn't do it sooner)

On Tue, 05 Aug 2008 17:20:00 +0900 "Stephen J. Turnbull" <stephen@xemacs.org> wrote: 

SJT> Ted Zlatanov writes:
>> I guess I come from a background of sysadmin, where things that can go
>> wrong will, so I'd rather not assume this.  I've had enough experience
>> with "this should never happen" happening at 3 AM.

SJT> That's just an argument for never doing any testing, since *any* test
SJT> could fail due to a flaky memory chip.

My point is simple: redundant testing reduces the chance of false
positives.  How is anticipating a system failure an argument for never
testing?  I can't follow your reasoning, sorry.

>> What I'm trying to help provide is a proactive mechanism.

SJT> Then look elsewhere than buildbot, which is just an attempt to speed
SJT> up the reaction.  A proactive solution would be to convince the Scons
SJT> people to join GNU.<wink>

I think you're mistaking the automated *process* with the tools that
implement it.  I want to provide the former, and don't care about the
latter (though buildbot seems easiest to set up).

SJT> That is, this thread was occasioned by breakage that happened to *one*
SJT> person, and we do not yet know what caused that problem.  Since the
SJT> details the OP has since given "shouldn't happen" the maintainers are
SJT> almost certainly going to table the matter and wait for more
SJT> evidence.  The buildbots won't help with that.

Buildbots provide independent verification of breakage.  We will have a
testing baseline, a date when things broke, and the knowledge that
things used to work before that date.  User reports can't provide all
three with the same degree of assurance (if at all).

>> My suggestion was to look for 5 or more broken build reports from
>> buildbots in the community.

SJT> I think you vastly overestimate the number of buildbots that will be
SJT> forthcoming.

It works for CPAN testers, why not for Emacs?  I can contribute 3
buildbots easily, and I'm sure others can do it too.  5 is a good number
but it can go down to 2 if needed.  The point is to avoid false positives.

>> I could go on, but the point is a broken build from a single system
>> can be caused by too many factors external to the build process.

SJT> Surely you can distinguish between the number of ways to go wrong and
SJT> the probability of going wrong?  The evidence I've seen in the Python
SJT> project is that nobody has ever complained in about 2 years of running
SJT> buildbots of spurious reports.  The vast majority of red bots are bugs
SJT> in the Python build or regressions.

You mentioned the situation with Python is different.  I think it's a
worthwhile experiment in the Emacs community.

Stefan or Chong, can you suggest a place to send the build failure
reports?

Ted