Re: New "make benchmark" target

[Eli Zaretskii <eliz@gnu.org>]

> Date: Mon, 30 Dec 2024 15:49:30 +0000
> From: Pip Cet <pipcet@protonmail.com>
> Cc: acorallo@gnu.org, stefankangas@gmail.com, mattiase@acm.org, eggert@cs.ucla.edu, emacs-devel@gnu.org, joaotavora@gmail.com
> 
> >> https://lists.gnu.org/archive/html/emacs-devel/2024-12/msg00595.html
> >
> > Thanks, but AFAICT this just says that you intended to use/extend ERT
> > to run this benchmark suite, but doesn't explain why you think using
> > ERT would be an advantage worthy of keeping.
> 
> I think some advantages are stated in that email: the ERT tagging
> mechanism is more general, works, and can be extended (I describe one
> such extension).  All that isn't currently true for elisp-benchmarks.

Unlike the rest of the test suite, where we need a way to be able to
run individual tests, a benchmark suite is much more likely to run as
a whole, because benchmarking a single kind of jobs in Emacs is much
less useful than producing a benchmark of a representative sample of
jobs.  So I'm not sure this particular aspect is such a serious
problem, certainly if it makes the job of adding the suite much
harder.

> The other big difference is resource management, which elisp-benchmarks
> does via a global variable, reusing one test as data for another.  ERT
> has a somewhat better mechanism.

This is just an argument for cleaner, more elegant code, right?  If
so, I think we can live with the issue.  Having a ready-t-run
benchmark suite is so much more important that I'm prepared to make
compromises.

> > Why do you think it is wrong to do the (AFAIU) simple change that
> > Andrea proposed?
> 
> Because it's a de facto commitment to not doing it in ERT.

Probably, but that in itself is not a catastrophe, surely?

> It might be relevant that elisp-benchmarks hasn't seen very active
> development lately.  I think switching to ERT might help there, if only
> because of the mailing list traffic.

My vote is to get the job done the fastest way possible.  Tests are
not Emacs itself, they are means towards a certain end, and so can use
less clean code and designs, at least IMO.