Benchmark tracking

Benchmark tracking

[Neil Jerram]

I've been running benchmarks for a while in my overnight builds, and
have put up a web page to show the results from that:

http://ossau.homelinux.net/~neil/

In summary, there's nothing to be concerned about (apart from a possible
slight slowdown in the last week or so).  The real point of this email
is just to mention this tracking - which I hope will help us notice if
we introduce significant performance gains or losses in the future.

If anyone has suggestions on how to present this information better or
differently, I'd be happy to hear them.

Regards,
        Neil

Re: Benchmark tracking

[Andy Wingo]

On Tue 27 Oct 2009 22:34, Neil Jerram <neil@ossau.uklinux.net> writes:

> I've been running benchmarks for a while in my overnight builds, and
> have put up a web page to show the results from that:
>
> http://ossau.homelinux.net/~neil/

Interesting!

> In summary, there's nothing to be concerned about (apart from a possible
> slight slowdown in the last week or so).

I think callee-parsed args did slow things down a bit. Would like to
figure out why, and at what exact commit.

Hm, would be good to bisect this more closely. Would it be possible to
map the X axis to commits?

Also, we should have more of the stock Gabriel benchmarks, like in the
Clinger or Feeley set.

>  The real point of this email is just to mention this tracking - which
> I hope will help us notice if we introduce significant performance
> gains or losses in the future.

Great!

Cheers,

Andy
-- 
http://wingolog.org/

Re: Benchmark tracking

[Ludovic Courtès]

Hello,

Neil Jerram <neil@ossau.uklinux.net> writes:

> I've been running benchmarks for a while in my overnight builds, and
> have put up a web page to show the results from that:
>
> http://ossau.homelinux.net/~neil/

Nice!

> If anyone has suggestions on how to present this information better or
> differently, I'd be happy to hear them.

As Andy suggested, commits on the X axis would be nice.  And perhaps
individual plots for each benchmark too.  And free lunch, and...  :-)

Thanks!

Ludo’.

Re: Benchmark tracking

[Neil Jerram]

Andy Wingo <wingo@pobox.com> writes:

> Hm, would be good to bisect this more closely. Would it be possible to
> map the X axis to commits?

I've added this, and updated the graphs to cover the last few days.

I'm planning to look at your other suggestions too - but I think next
will be Ludo's request for showing each benchmark separately.

     Neil

Re: Benchmark tracking

[Mike Gran]

> I've added this, and updated the graphs to cover the last few days.
> 
> I'm planning to look at your other suggestions too - but I think next
> will be Ludo's request for showing each benchmark separately.
> 
>      Neil

I did once play around with plotting the benchmarks.log files.
For your bemusement, I've attached a link to a script that I wrote.

http://git.savannah.gnu.org/cgit/guile.git/tree/benchmark-suite/benchplot?h=string_abstraction2

-Mike Gran

Re: Benchmark tracking

[Neil Jerram]

Mike Gran <spk121@yahoo.com> writes:

> I did once play around with plotting the benchmarks.log files.
> For your bemusement, I've attached a link to a script that I wrote.
>
> http://git.savannah.gnu.org/cgit/guile.git/tree/benchmark-suite/benchplot?h=string_abstraction2

Thanks, Mike.  I like the idea of generating the gnuplot command file
and then running it in a separate step; may switch my code to do that
too.  I also appreciate how your code shows how to do more cunning
gnuplot things, like xtics and rotate; and the use of ~{ and ~} in
format, which I hadn't taken on board before.

My code is odd, in comparison with yours, in that it doesn't just use
`read' to read the benchmark data.  That's because I've been using the
standard output from ./benchmark-guile, which begins with two
uncommented lines:

   Benchmarking /home/neil/SW/Guile/ovnight/meta/guile ...
   with GUILE_LOAD_PATH=/home/neil/SW/Guile/ovnight/benchmark-suite
   ;; running guile version 1.9.3
   ...

But I see now that I could use benchmark-guile.log and just `read'
instead...

In summary, much appreciated!

     Neil

Re: Benchmark tracking

[Neil Jerram]

ludo@gnu.org (Ludovic Courtès) writes:

> And perhaps individual plots for each benchmark too.

I've added those now.  Please let me know if any comments or suggestions
for improvement...

     Neil

Re: Benchmark tracking

[Neil Jerram]

FYI I've added some small improvements to my benchmark plots:

1. Skip plotting points for days when there were no results
2. Avoid plotting benchmark times that are too small for useful comparison
3. Scale results by number of iterations
4. Show grid in combined plot

In case it's not clear, (2) means not trusting benchmarks that run for
less than 0.05s - because the ratio between 2 such small times has
not enough precision to be useful.  (3) means that we can increase the
iteration count for such benchmarks, to increase precision, and that the
changes in iteration count will be correctly accounted for in the graph
over time.

  http://ossau.homelinux.net/~neil/

Regards,
        Neil

Re: Benchmark tracking

[Neil Jerram]

Andy Wingo <wingo@pobox.com> writes:

> Also, we should have more of the stock Gabriel benchmarks, like in the
> Clinger or Feeley set.

I found this page:
http://www.ccs.neu.edu/home/will/Twobit/benchmarksAbout.html

Is that what you meant?

Do you know anything about how this code is licensed (and hence if we
can add it to the Guile repository and/or distribution)?  I don't see
anything about this on Clinger's web site.

Thanks,
        Neil

Re: Benchmark tracking

[Neil Jerram]

Neil Jerram <neil@ossau.uklinux.net> writes:

> Andy Wingo <wingo@pobox.com> writes:
>
>> Also, we should have more of the stock Gabriel benchmarks, like in the
>> Clinger or Feeley set.
>
> I found this page:
> http://www.ccs.neu.edu/home/will/Twobit/benchmarksAbout.html
>
> Is that what you meant?
>
> Do you know anything about how this code is licensed (and hence if we
> can add it to the Guile repository and/or distribution)?

On second thoughts, I don't think we need to add them to the repository
or distribution.  Instead I can automate the process of downloading
them, storing them just long enough to run the benchmarks, and recording
the results.

      Neil

Re: Benchmark tracking

[Ludovic Courtès]

Hello,

Neil Jerram <neil@ossau.uklinux.net> writes:

> I found this page:
> http://www.ccs.neu.edu/home/will/Twobit/benchmarksAbout.html
>
> Is that what you meant?
>
> Do you know anything about how this code is licensed (and hence if we
> can add it to the Guile repository and/or distribution)?  I don't see
> anything about this on Clinger's web site.

Some of them are already under ‘gc-benchmarks’ in the repository; those
under ‘larceny’ are from
http://www.ccs.neu.edu/home/will/GC/sourcecode.html .

Thanks,
Ludo’.

Re: Benchmark tracking

[Ludovic Courtès]

Hi Neil,

Neil Jerram <neil@ossau.uklinux.net> writes:

>   http://ossau.homelinux.net/~neil/

This looks really good to me.  The only thing that’s missing is a
mapping from units on the X axis to commits.

Looking at http://ossau.homelinux.net/~neil/bm_master_i.html, it seems
that several things got slightly slower around x = 25.

Thanks,
Ludo’.

Re: Benchmark tracking

[Ludovic Courtès]

Neil Jerram <neil@ossau.uklinux.net> writes:

> Thanks, Mike.  I like the idea of generating the gnuplot command file
> and then running it in a separate step; may switch my code to do that
> too.  I also appreciate how your code shows how to do more cunning
> gnuplot things, like xtics and rotate; and the use of ~{ and ~} in
> format, which I hadn't taken on board before.

FWIW, GNU Plotutils seems to be a pleasant toolset, which might be worth
looking at (I’ve always found it somewhat painful to tweak Gnuplot’s
output...).  There’s also Andy’s
http://wingolog.org/software/guile-charting/ which seems pretty nice.

Thanks,
Ludo’.

Re: Benchmark tracking

[Neil Jerram]

ludo@gnu.org (Ludovic Courtès) writes:

> Hi Neil,
>
> Neil Jerram <neil@ossau.uklinux.net> writes:
>
>>   http://ossau.homelinux.net/~neil/
>
> This looks really good to me.  The only thing that’s missing is a
> mapping from units on the X axis to commits.

That's at http://ossau.homelinux.net/~neil/bm_master_x.html, and it
is linked from the main page above.

> Looking at http://ossau.homelinux.net/~neil/bm_master_i.html, it seems
> that several things got slightly slower around x = 25.

x = 25 => early morning of 2009-10-23, commit
04c68c039194f33d5bd7e8b1f21eba7c8bd6adbe.

I agree that many tests made a bit of a jump then, but remember that
that could be caused just by something unusual happening on my build
machine.  I'm not aware of anything unusual on that date... but it's
still possible.

Also, in many cases the loss seems to have been recovered again since
then.

And, looking at the commits on 2009-10-22:

	Compile Guile modules with `-Wunbound-variable'.

	Fix typos leading to unbound variable references.

	Adjust `unbound-variable' GOOPS heuristic for `goops... 

	Fix bytecode disassembler.

	SRFI-88: Call `read-set!' at compile time and run time.

None of these look like they could have caused a slow down.

I'm not sure we can rely very much on these graphs for day-to-day
changes, because I haven't eliminated all possible external factors, so
there is considerable random variation.  I'd say their value is more to
give us confidence that we don't slow Guile down over the medium term.

Regards,
      Neil

Re: Benchmark tracking

[Andy Wingo]

On Sun 08 Nov 2009 20:11, ludo@gnu.org (Ludovic Courtès) writes:

> Neil Jerram <neil@ossau.uklinux.net> writes:
>
>> Thanks, Mike.  I like the idea of generating the gnuplot command file
>> and then running it in a separate step; may switch my code to do that
>> too.  I also appreciate how your code shows how to do more cunning
>> gnuplot things, like xtics and rotate; and the use of ~{ and ~} in
>> format, which I hadn't taken on board before.
>
> FWIW, GNU Plotutils seems to be a pleasant toolset, which might be worth
> looking at (I’ve always found it somewhat painful to tweak Gnuplot’s
> output...).  There’s also Andy’s
> http://wingolog.org/software/guile-charting/ which seems pretty nice.

Guile-charting's output looks nice but doesn't include very much. It's
definitely a build-your-own thing ;)

A
-- 
http://wingolog.org/

Re: Benchmark tracking

[Ludovic Courtès]

Neil Jerram <neil@ossau.uklinux.net> writes:

> ludo@gnu.org (Ludovic Courtès) writes:

[...]

>> This looks really good to me.  The only thing that’s missing is a
>> mapping from units on the X axis to commits.
>
> That's at http://ossau.homelinux.net/~neil/bm_master_x.html, and it
> is linked from the main page above.

Oops, indeed, perfect!

> I'm not sure we can rely very much on these graphs for day-to-day
> changes, because I haven't eliminated all possible external factors, so
> there is considerable random variation.  I'd say their value is more to
> give us confidence that we don't slow Guile down over the medium term.

Yes, you may be right.

Thanks,
Ludo’.