Re: An Emacs benchmarking suite (was: An idea, now that we have dynamic loading)

[Eli Zaretskii <eliz@gnu.org>]

> Cc: johnw@gnu.org, emacs-devel@gnu.org
> From: "Siraphob (Ben) Phipathananunth" <siraben@disroot.org>
> Date: Fri, 11 May 2018 22:22:47 +0700
> 
> Eli Zaretskii wrote:
> > John even asked for creation of such a suite of performance tests, but
> > AFAIK no one has picked the gauntlet till now.
> 
> Would it suffice to use Emacs Lisp to run these performance tests
> (i.e. using benchmark.el)?

I don't see why not.

> Even so, how would one account for external factors in the operating
> system?  Perhaps when the test is performed, the deltas between
> commits are given as percentages instead of CPU time.  Once we have
> that, it would be great to have tests run multiple times and/or on
> various devices to refine the data further.

Performance should indeed compare several versions n the same system.

> An area that would be interesting to look at is memory usage of
> functions over commits, but I don't know of a way of measuring memory
> usage in Emacs (especially over an extended period, to analyze things
> such as maximum memory usage).

Memory analysis is tricky on modern systems. but it isn't impossible.
One could start using the values reported by process-attributes.