Re: New "make benchmark" target

[Pip Cet via "Emacs development discussions." <emacs-devel@gnu.org>]

"Stefan Kangas" <stefankangas@gmail.com> writes:

> Pip Cet <pipcet@protonmail.com> writes:

>> We also need to decide on the directory structure; right now, I've
>> created a lisp/emacs-lisp/benchmarks/ directory; I'd prefer
>> lisp/benchmarks (which would make it easier to exclude the benchmark
>> files from compilation), but I don't have a strong preference and others
>> should make that decision.  (I haven't included the
>> lisp/emacs-lisp/subdirs.el file, but if we decide to keep the benchmarks
>> in lisp/emacs-lisp/benchmarks/, we'll need to gitignore that, too).
>
> I don't have a strong opinion here, but maybe this stuff belongs under
> test/ even?

I'm still working on this, but it turns out it's harder than I thought
to turn the .el files for the benchmarks into something that's usable
both with ERT and with the existing elisp-benchmarks.el infrastructure.

For example, there's the use of elb-bench-directory to locate resource
files; ERT has its own function for that, but it turns out one of the
resources one benchmark uses is the source file for another benchmark.
Usually I'd just use letf around the benchmark call, but that may affect
performance too much for the benchmarks to be comparable between the ERT
and elisp-benchmarks invocations.

I just don't know whether I'd feel comfortable invoking the benchmarks
in such different ways and presenting the results in a way that would
make people compare them.

The rest of the issues are trivial: whitespace issues, two different
files calling Fprovide with the same feature, elb-scroll.el merged into
elb-smie.el rather than maintaining them as two separate files.  These
are very definitely not deficiencies in the current elisp-benchmarks
package, just different conventions.  However, that amounts to
significant changes to the benchmark .el files overall; rather than
copies of the elisp-benchmarks files, we now have modified versions and
would have to port any changes between the two different sets of files.

Ultimately, my current benchmark branch doesn't do what I set out to do,
which is to share the elisp-benchmarks suite between an unmodified
elisp-benchmarks and the new ERT framework, yielding comparable results.
Getting it to work isn't the main problem, comparability of results is.

So it is with some trepidation that I suggest that the best remaining
option may be to fork or "freeze"/archive elisp-benchmarks and move
development of benchmarks for current Emacs builds entirely to the ERT
framework.  Forking causes a lot of extra synchronization work.
Archiving the package means we will never add new benchmarks for
pre-make-benchmark Emacs builds.

I'm convinced a "make benchmark" target is worth it.  I also think that
we should use the ERT framework, because benchmarks and pass-or-fail
tests are quite similar.

Maybe I'm missing an obvious solution here?

Pip