Re: New "make benchmark" target

[Andrea Corallo <acorallo@gnu.org>]

Pip Cet <pipcet@protonmail.com> writes:

> In particular, as you (Andrea) correctly pointed out, it is sometimes
> appropriate to use an average run time (or, non-equivalently, an average
> speed) for reporting test results; the assumptions needed for this are
> very significant and need to be spelled out explicitly.  The vast
> majority of "make benchmark" uses which I think should happen cannot
> meet these stringent requirements.
>
> To put things simply, it is better to discard outliers (test runs which
> take significantly longer than the rest).  Averaging doesn't do that: it
> simply ruins your entire test run if there is a significant outlier.
> IOW, running the benchmarks with a large repetition count is very likely
> to result in useful data being discarded, and a useless result.

As mentioned, I disagree with having some logic put in place to
arbitrarily decide which value is worth to be considered and which value
should be discarded.  If a system is producing noisy measures this has
to be reported as error of the measure.  Those numbers are there for
some real reason and have to be accounted.