Re: New "make benchmark" target

[Eli Zaretskii <eliz@gnu.org>]

> Date: Mon, 30 Dec 2024 15:00:29 +0000
> From: Pip Cet <pipcet@protonmail.com>
> Cc: acorallo@gnu.org, stefankangas@gmail.com, mattiase@acm.org, eggert@cs.ucla.edu, emacs-devel@gnu.org, joaotavora@gmail.com
> 
> "Eli Zaretskii" <eliz@gnu.org> writes:
> 
> >> Date: Mon, 30 Dec 2024 11:45:36 +0000
> >> From: Pip Cet <pipcet@protonmail.com>
> >> Cc: Stefan Kangas <stefankangas@gmail.com>, Eli Zaretskii
> >> <eliz@gnu.org>, Mattias Engdegård <mattiase@acm.org>, Paul Eggert
> >> <eggert@cs.ucla.edu>, emacs-devel@gnu.org, João Távora
> >> <joaotavora@gmail.com>
> >>
> >> "Andrea Corallo" <acorallo@gnu.org> writes:
> >>
> >> > I'd personally drop the requirement of using ERT as a framework for
> >> > benchmarks, I'd just move elisp-benchmarks code in emacs core and add
> >> > the target.
> >>
> >> Well, as is obvious from the quoted paragraph, I disagree.  I've stated
> >> why in the thread; if someone wants a summary, I can provide one
> >
> > Can you point to the message where you explained your rationale for
> > using ERT for this?  I've scanned the discussion, but couldn't find
> > such a message.
> 
> The best I can find is this:
> 
> https://lists.gnu.org/archive/html/emacs-devel/2024-12/msg00595.html

Thanks, but AFAICT this just says that you intended to use/extend ERT
to run this benchmark suite, but doesn't explain why you think using
ERT would be an advantage worthy of keeping.

> I can try to provide a more detailed/structured rationale if that's
> helpful.  (Is it, though?  Reusing someone's code in a way which reduces
> their user base and might cause them more work isn't something we should
> do lightly.)

I'm not sure I follow.  Andrea suggests to move elisp-benchmarks into
the repository, and add a target to the test/ Makefile to run it.
AFAIU he suggested that because it should be less work, not more.

Why do you think it is wrong to do the (AFAIU) simple change that
Andrea proposed?

The reason I'm asking is because I think we want this suite to be part
of our test, but don't necessarily want the addition of the benchmarks
to the test suite be a large job that complicates the benchmarks and
the test suite alike.