Re: New "make benchmark" target

[Eli Zaretskii <eliz@gnu.org>]

> Date: Sat, 04 Jan 2025 16:34:24 +0000
> From: Pip Cet <pipcet@protonmail.com>
> Cc: acorallo@gnu.org, stefankangas@gmail.com, mattiase@acm.org, eggert@cs.ucla.edu, emacs-devel@gnu.org
> 
> "Eli Zaretskii" <eliz@gnu.org> writes:
> 
> > Since you've pushed that to a branch, I suggest to submit bug reports
> > about these issues, using "[scratch/elisp-benchmarks]" in the Subject
> > of the bug.
> 
> I've studied the issues a bit more.  This is a bit long, but in summary,
> I think improving elisp-benchmarks.el (the specific file, not the entire
> package, which I still intend to reuse) would take more time than
> starting from ERT

You mean, starting from scratch??  How can this be less work than
fixing whatever bugs you found in the benchmarks (assuming that we
want to fix all of them)?

> I'm reporting a small number of elisp-benchmarks "bugs" (I think the
> term is likely to be contentious; I use it because that's what the
> mailing list is called).  All of them are fixable.  Most of them are
> easily fixable by moving to ERT.

You mean, if we move to ERT, which by itself is a significant job,
then some or most of these bugs could be fixed as a side effect or
with much less work, is that it?  That could be so, but then the move
to ERT itself is not a small job, so we need to take that into
consideration when deciding whether to move to ERT right now.

> My conclusion is that elisp-benchmarks.el (again, the benchmarks are
> fine) isn't the right way forward.

Well, you though that to begin with, so forgive me if I say that I'd
like a second independent opinion in this case.

> I'm happy to change the scratch/elisp-benchmarks branch in the ways
> we've discussed, and it should be merged, but if someone decides to
> incrementally solve some of the issues, that, while not very harmful,
> would be an inefficient use of resources.

Let's hear what Andrea thinks about the issues you reported (let's
please discuss them on the bug tracker, not here), and let's take it
from there.

> Benchmarks need a test framework.

I don't necessarily agree.  A benchmark doesn't have to have a
"correct" or "expected" result, so a test framework is not necessarily
justified or needed.