bug#75356: impossible to benchmark byte-compiled code in a native-comp build

[Eli Zaretskii <eliz@gnu.org>]

> Cc: pipcet@protonmail.com
> From: Andrea Corallo <acorallo@gnu.org>
> Date: Mon, 06 Jan 2025 04:24:44 -0500
> 
> Pip Cet via "Bug reports for GNU Emacs, the Swiss army knife of text
> editors" <bug-gnu-emacs@gnu.org> writes:
> 
> > elisp-benchmarks.el unconditionally forces use of native compilation
> > if run on an Emacs which is compiled with native-comp enabled.  This
> > is a minor issue, but it's still important to benchmark this
> > configuration once in a while.
> 
> Why is this important?  On a native compiled Emacs all code is supposed
> to run native compiled, not only the test, but runtime libraries as
> well, further more even Emacs primitives are different in order to
> accomodate native code execution.  My opinion is that results of such a
> mixed tests would make little no sense in general.

I think the idea is that we don't want to force someone who wants to
benchmark bytecode to build a separate Emacs configured without native
compilation.

It is okay to default to native code, but it would be nice to have a
knob to run the benchmarks without compiling them to native code.

> Anyway if you really want to run bytecode on a native compiled Emacs
> elisp-benchamerks let you do this with:
> 
> emacs -batch -l ./elisp-benchmarks.el --eval '(progn (setq elb-speed -1) (elisp-benchmarks-run))'

Would this refrain from loading *.eln files if they are already there?
Or does the benchmark suite remove all the *.eln files after it
finishes?  (Apologies for not taking a closer look at the code.)

> > In general, the compilation logic of elisp-benchmarks.el is fragile
> > and will lead to unreliabe test results,
> 
> Why do you think so?

I suggest to avoid such general remarks, and instead talk about
specific issues where you think the logic could easily break.