Re: a benchmark suite for Emacs

[Joshua Branson <jbranso@fastmail.com>]

This is super cool!


Gemini Lasswell <gazally@runbox.com> writes:


> Hello all,
>
> I've just pushed to scratch/benchmarks my WIP benchmarking suite for
> Emacs and Emacs packages.
>
> It's a long ways from done.  Some major missing parts are:
>
> - Better/more complete user interface
> - Documentation
> - Statistical analysis (report on noisiness of data)
> - Tramp integration/run benchmarks remotely
> - Use VC rather than calling git directly
> - More benchmark tasks
> - Tests
>
> But, given:
>
> - a local clone of a git repo
> - a program/shell script that produces an executable given that
>   git repo and a commit
> - a directory in the git repo with Lisp files containing tasks to be
>   benchmarked
> - a range of commits (as for git-rev-list(1))
> - a number of commits to select from the range
>
> it can use the shell script to build older versions of Emacs from
> commits selected from the commit range given, run the tasks in those
> older Emacs, record timing data, errors, process output and messages,
> and then produce a report in the form of an org-mode file ready for
> org-babel-gnuplot.
>
> The benchmark data is stored in a directory hierarchy organized so that
> benchmark data created on separate machines can be easily git-merged
> together.
>
> There's no documentation yet, but if any of you have time and interest
> in running benchmarks, let me know and I'll write up instructions.
>
> For those of you who generously spend your time reviewing code, the new
> benchmark directory, thread.el and erb-task.el are ready for your
> attention, but erb.el is not.
>
> I also welcome ideas for more benchmarks, or anyone who wants to
> write some.