Re: New "make benchmark" target

[Eli Zaretskii <eliz@gnu.org>]

> Date: Mon, 30 Dec 2024 21:34:55 +0000
> From: Pip Cet <pipcet@protonmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>, stefankangas@gmail.com, mattiase@acm.org, eggert@cs.ucla.edu, emacs-devel@gnu.org
> 
> > I'm open to patches to elisp-benchmarks (and to its hypothetical copy in
> > emacs-core).  My opinion that something can potentially be improved in
> 
> What's the best way to report the need for such improvements?

Since you've pushed that to a branch, I suggest to submit bug reports
about these issues, using "[scratch/elisp-benchmarks]" in the Subject
of the bug.

> > it (why not), but I personally ATM don't understand the need for ERT.
> 
> Let's focus on the basics right now: people know how to write ERT tests.
> We have hundreds of them.  Some of them could be benchmarks, and we want
> to make that as easy as possible.

We can later add more benchmarks using ERT.  There's no contradiction.

> It also allows a third class of tests: stress tests which we want to
> execute more often than once per test run, which identify occasional
> failures in code that needs to be executed very often to establish
> stability (think bug#75105: (cl-random 1.0e+INF) produces an incorrect
> result once every 8 million runs).  IIRC, right now ERT uses ad-hoc
> loops for such tests, but it'd be nicer to expose the repetition count
> in the framework (I'm not going to run the non-expensive testsuite on
> FreeDOS if that means waiting for a million iterations on an emulated
> machine).
> 
> (I also think we should introduce an ert-how structure that describes how
> a test is to be run: do we want to inhibit GC or allow it?  Run some
> warm-up test runs or not? What's the expected time, and when should we
> time out? We can't run the complete matrix for all tests, so we need
> some hints in the test, and the lack of a test declaration in
> elisp-benchmarks hurts us there).

These seem to be long-term goals of improving the benchmark suite.
They are fine by me, but I don't see why they should preclude
installing the benchmarks we have without first converting them to
ERT.  We can do that later, if we decide it's worth the effort.