Re: "Pace is nothing without guile"

[ludo@gnu.org (Ludovic Courtès)]

Hi,

"Neil Jerram" <neiljerram@googlemail.com> writes:

> So this email is about systematic performance data.  I was wondering
> what benchmarks we could run to get good coverage of all Guile's
> function, and suddenly thought "of course, the test suite!"

Like Greg, I'm a bit suspicious about using the test suite as a
collection of micro-benchmarks.  Usually, micro-benchmarks aim to assess
the cost of a specific operation, which must consequently be isolated to
avoid interference with unrelated computations.

Conversely, unit tests aim to verify that certain invariants hold,
regardless of "peripheral" computations required for that verification.
For instance, a few SRFI-14 tests use `every', a few SRFI-69 tests use
`lset=', various tests use `memq', etc.  (OTOH, looking at the test
suite, I'm not sure whether these tests are exceptions.)

Other than that, my feeling is that it may be harder to analyze timings
of tests that were not written as micro-benchmarks in the first place,
since one must first determine what the test actually measures.

> - anyone running the tests and uploading data, not just Guile core developers

Although quite in fashion these days (see CDash), I'm not sure we
absolutely need such a tool.  Having *some* benchmark suite to run seems
more important as a first step.  :-)

Thanks,
Ludo'.