Re: New "make benchmark" target

[Eli Zaretskii <eliz@gnu.org>]

> Date: Mon, 30 Dec 2024 17:25:44 +0000
> From: Pip Cet <pipcet@protonmail.com>
> Cc: acorallo@gnu.org, stefankangas@gmail.com, mattiase@acm.org, eggert@cs.ucla.edu, emacs-devel@gnu.org
> 
> "Eli Zaretskii" <eliz@gnu.org> writes:
> 
> Top-posted TL;DR: let's call Andrea's code "make elisp-benchmarks" and
> include it now?  That would preserve the Git history and importantly (to
> me) reserve the name for now.

Fine by me.

> And I don't usually run ERT tests individually, while I'm trying to get
> in the habit of running the (non-expensive) test suite before I push.

I do it all the time, when I install some change and want to make sure
the related tests still pass.

> My suggestion is a compromise: add "make elisp-benchmarks" now, using
> Andrea's code, then consider more complicated ERT-based approaches
> without being in any hurry to do so.  But, also, let's agree that the
> ERT-based approaches are "allowed" to reuse the elisp-benchmarks code
> without providing comparable results or a porting mechanism, and keep
> the "make benchmark" name reserved for a while.

Of course ERT-based approaches are allowed.  I only chimed into this
discussion because that approach bumped into some difficulties.

> My prediction is that it will turn out "make elisp-benchmarks" doesn't
> usually provide very useful results, and expansion of the test framework
> to produce useful results is easier reusing the ERT framework.

We'll see, and if you are right, we will work on improving the
benchmarks.

> My preference would be a top-level directory called "elisp-benchmarks",
> but ultimately that's a minor question, so just let me know the
> preferred destination.

I thought we wanted it under test/ ?

But I'm also okay with having a directory that is sibling to test/ if
there are no objections from Andrea and others.