Re: An Emacs benchmarking suite

[phillip.lord@russet.org.uk (Phillip Lord)]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> So, something like time to create a big list (for CPU), and time to read
>>> a defined file (for IO). Then you could say "this file should parse in
>>> 1x IO base-line + 10x CPU base-line.
>>
>> You assume a linear scalability, but that is not necessarily so.
>> The ratio between performance indices of different codes could vary
>> depending on the build option and the underlying OS.
>
> Another issue is that performance measuring is notoriously difficult
> (even on an otherwise idle machine, let alone on some server that has
> other tasks running at the same time).  So you might be able to catch
> the "10x slower" case easily with a fairly high confidence that the
> problem is indeed that the code got slower, but if you want to catch the
> "20% slowdown" with any kind of confidence (without being drowned in
> false positives), you'll need either a very tight control on the test
> runs, or a good statistical analysis.


Indeed, it would be very broad brush. Or, alternatively, it would be
tied very tightly to a specific infrastructure.

Phil