Re: "Pace is nothing without guile"

["Neil Jerram" <neiljerram@googlemail.com>]

2008/7/13 Greg Troxel <gdt@ir.bbn.com>:
> My immediate reaction is that test suites aren't good benchmarks because
> we will often want to add to test suites, while changing the benchmark
> invalidates previous data so we will not want to change the benchmark.

Yes, but...

> Now, if you mean to use the test suite as a collection of
> micro-benchmarks, so that we just have a rule that individual tests
> aren't modified without serious cause, but new ones can be added, then
> that makes senes.

Yes, this is what I meant.  And I also think there will be enough test
cases not changing, over the long term, that we won't have to worry in
practice about changing a few here and there.

I've taken a better look now at the existing benchmark-suite - which
doesn't contain very many benchmarks, but does provide a sensible
discussion of timings, and useful library functions.  I don't think it
will be very hard to somehow incorporate the test-suite into the set
of benchmarks.

Regards,
          Neil