a benchmark suite for Emacs

a benchmark suite for Emacs

[Gemini Lasswell]

Hello all,

I've just pushed to scratch/benchmarks my WIP benchmarking suite for
Emacs and Emacs packages.

It's a long ways from done.  Some major missing parts are:

- Better/more complete user interface
- Documentation
- Statistical analysis (report on noisiness of data)
- Tramp integration/run benchmarks remotely
- Use VC rather than calling git directly
- More benchmark tasks
- Tests

But, given:

- a local clone of a git repo
- a program/shell script that produces an executable given that
  git repo and a commit
- a directory in the git repo with Lisp files containing tasks to be
  benchmarked
- a range of commits (as for git-rev-list(1))
- a number of commits to select from the range

it can use the shell script to build older versions of Emacs from
commits selected from the commit range given, run the tasks in those
older Emacs, record timing data, errors, process output and messages,
and then produce a report in the form of an org-mode file ready for
org-babel-gnuplot.

The benchmark data is stored in a directory hierarchy organized so that
benchmark data created on separate machines can be easily git-merged
together.

There's no documentation yet, but if any of you have time and interest
in running benchmarks, let me know and I'll write up instructions.

For those of you who generously spend your time reviewing code, the new
benchmark directory, thread.el and erb-task.el are ready for your
attention, but erb.el is not.

I also welcome ideas for more benchmarks, or anyone who wants to
write some.

Re: a benchmark suite for Emacs

[Joshua Branson]

This is super cool!


Gemini Lasswell <gazally@runbox.com> writes:


> Hello all,
>
> I've just pushed to scratch/benchmarks my WIP benchmarking suite for
> Emacs and Emacs packages.
>
> It's a long ways from done.  Some major missing parts are:
>
> - Better/more complete user interface
> - Documentation
> - Statistical analysis (report on noisiness of data)
> - Tramp integration/run benchmarks remotely
> - Use VC rather than calling git directly
> - More benchmark tasks
> - Tests
>
> But, given:
>
> - a local clone of a git repo
> - a program/shell script that produces an executable given that
>   git repo and a commit
> - a directory in the git repo with Lisp files containing tasks to be
>   benchmarked
> - a range of commits (as for git-rev-list(1))
> - a number of commits to select from the range
>
> it can use the shell script to build older versions of Emacs from
> commits selected from the commit range given, run the tasks in those
> older Emacs, record timing data, errors, process output and messages,
> and then produce a report in the form of an org-mode file ready for
> org-babel-gnuplot.
>
> The benchmark data is stored in a directory hierarchy organized so that
> benchmark data created on separate machines can be easily git-merged
> together.
>
> There's no documentation yet, but if any of you have time and interest
> in running benchmarks, let me know and I'll write up instructions.
>
> For those of you who generously spend your time reviewing code, the new
> benchmark directory, thread.el and erb-task.el are ready for your
> attention, but erb.el is not.
>
> I also welcome ideas for more benchmarks, or anyone who wants to
> write some.

Re: a benchmark suite for Emacs

[Juri Linkov]

> - a local clone of a git repo
> - a program/shell script that produces an executable given that
>   git repo and a commit
> - a directory in the git repo with Lisp files containing tasks to be
>   benchmarked
> - a range of commits (as for git-rev-list(1))
> - a number of commits to select from the range
>
> it can use the shell script to build older versions of Emacs from
> commits selected from the commit range given, run the tasks in those
> older Emacs, record timing data, errors, process output and messages,
> and then produce a report in the form of an org-mode file ready for
> org-babel-gnuplot.

Is it like Diffbench for Ruby?  Something like that was sorely missed
for a long time in Emacs.

Re: a benchmark suite for Emacs

[Gemini Lasswell]

Juri Linkov <juri@linkov.net> writes:

>> - a local clone of a git repo
>> - a program/shell script that produces an executable given that
>>   git repo and a commit
>> - a directory in the git repo with Lisp files containing tasks to be
>>   benchmarked
>> - a range of commits (as for git-rev-list(1))
>> - a number of commits to select from the range
>>
>> it can use the shell script to build older versions of Emacs from
>> commits selected from the commit range given, run the tasks in those
>> older Emacs, record timing data, errors, process output and messages,
>> and then produce a report in the form of an org-mode file ready for
>> org-babel-gnuplot.
>
> Is it like Diffbench for Ruby?  Something like that was sorely missed
> for a long time in Emacs.

Like Diffbench, but bigger and better.  Diffbench looks like it does a
quick comparison of two states of the project.  That's something this
can do, but my vision for it is to build a database of benchmark results
for multiple machines and for commits going back as far as we can figure
out how to build and run them, so we can see how performance has changed
over time and how it varies on different systems.