"Pace is nothing without guile"

"Pace is nothing without guile"

[Neil Jerram]

... That's a comment from coverage of the current England v South
Africa cricket match
(http://uk.cricinfo.com/talk/content/current/multimedia/360921.html).

But is Guile nothing without pace?

Well obviously it isn't "nothing", but I think Guile is perceived,
among both Scheme implementations and free scripting languages, as
being a bit slow, and I think that a large part of the reason for this
is that we have no systematic benchmarking.

So this email is about systematic performance data.  I was wondering
what benchmarks we could run to get good coverage of all Guile's
function, and suddenly thought "of course, the test suite!"  The test
suite should, by definition, provide coverage of everything that we
care about.  Therefore I think that we should be able to start
collecting a lot of useful performance data by implementing a version
of "make check" that measures and stores off the time that each test
takes to run.

What I'd like input/advice on, is exactly how we store and collate
such data.  I think the system should ideally support

- arbitrary later analysis of the collected data

- correlation of the result for a specific test with the exact source
code of that test at the time it was run...

- ...and hence, being able to work out (later) that the results
changed because the content of the test changed

- anyone running the tests and uploading data, not just Guile core developers

- associating a set of results with the relevant information about the
machine that they were obtained on (CPUs, RAM) in such a way that the
information is trustable, but without invading the privacy of the
uploader.

So how do we do that?  Perhaps the test content identification could
be done by its Git (SHA-1) hash - together with the path of the repo
containing that version.  And I imagine that the form of the results
could be a file containing lines like:

("numbers.test" SHA1-HASH REPO-PATH DATE+TIME MACHINE-INFO MEASURED-DURATION)

That would allow sets of results to be concatenated for later
analysis.  But I'm not sure what the relevant MACHINE-INFO is and how
to represent that.

Any thoughts / comments / ideas?  Thanks for reading!

      Neil

Re: "Pace is nothing without guile"

[Greg Troxel]

My immediate reaction is that test suites aren't good benchmarks because
we will often want to add to test suites, while changing the benchmark
invalidates previous data so we will not want to change the benchmark.

Now, if you mean to use the test suite as a collection of
micro-benchmarks, so that we just have a rule that individual tests
aren't modified without serious cause, but new ones can be added, then
that makes senes.

Re: "Pace is nothing without guile"

[Neil Jerram]

2008/7/13 Greg Troxel <gdt@ir.bbn.com>:
> My immediate reaction is that test suites aren't good benchmarks because
> we will often want to add to test suites, while changing the benchmark
> invalidates previous data so we will not want to change the benchmark.

Yes, but...

> Now, if you mean to use the test suite as a collection of
> micro-benchmarks, so that we just have a rule that individual tests
> aren't modified without serious cause, but new ones can be added, then
> that makes senes.

Yes, this is what I meant.  And I also think there will be enough test
cases not changing, over the long term, that we won't have to worry in
practice about changing a few here and there.

I've taken a better look now at the existing benchmark-suite - which
doesn't contain very many benchmarks, but does provide a sensible
discussion of timings, and useful library functions.  I don't think it
will be very hard to somehow incorporate the test-suite into the set
of benchmarks.

Regards,
          Neil

Re: "Pace is nothing without guile"

[Ludovic Courtès]

Hi,

"Neil Jerram" <neiljerram@googlemail.com> writes:

> So this email is about systematic performance data.  I was wondering
> what benchmarks we could run to get good coverage of all Guile's
> function, and suddenly thought "of course, the test suite!"

Like Greg, I'm a bit suspicious about using the test suite as a
collection of micro-benchmarks.  Usually, micro-benchmarks aim to assess
the cost of a specific operation, which must consequently be isolated to
avoid interference with unrelated computations.

Conversely, unit tests aim to verify that certain invariants hold,
regardless of "peripheral" computations required for that verification.
For instance, a few SRFI-14 tests use `every', a few SRFI-69 tests use
`lset=', various tests use `memq', etc.  (OTOH, looking at the test
suite, I'm not sure whether these tests are exceptions.)

Other than that, my feeling is that it may be harder to analyze timings
of tests that were not written as micro-benchmarks in the first place,
since one must first determine what the test actually measures.

> - anyone running the tests and uploading data, not just Guile core developers

Although quite in fashion these days (see CDash), I'm not sure we
absolutely need such a tool.  Having *some* benchmark suite to run seems
more important as a first step.  :-)

Thanks,
Ludo'.