Re: New "make benchmark" target

[Andrea Corallo <acorallo@gnu.org>]

Pip Cet <pipcet@protonmail.com> writes:

> "Stefan Kangas" <stefankangas@gmail.com> writes:
>
>> Pip Cet <pipcet@protonmail.com> writes:
>
>>> We also need to decide on the directory structure; right now, I've
>>> created a lisp/emacs-lisp/benchmarks/ directory; I'd prefer
>>> lisp/benchmarks (which would make it easier to exclude the benchmark
>>> files from compilation), but I don't have a strong preference and others
>>> should make that decision.  (I haven't included the
>>> lisp/emacs-lisp/subdirs.el file, but if we decide to keep the benchmarks
>>> in lisp/emacs-lisp/benchmarks/, we'll need to gitignore that, too).
>>
>> I don't have a strong opinion here, but maybe this stuff belongs under
>> test/ even?
>
> I'm still working on this, but it turns out it's harder than I thought
> to turn the .el files for the benchmarks into something that's usable
> both with ERT and with the existing elisp-benchmarks.el infrastructure.
>
> For example, there's the use of elb-bench-directory to locate resource
> files; ERT has its own function for that, but it turns out one of the
> resources one benchmark uses is the source file for another benchmark.
> Usually I'd just use letf around the benchmark call, but that may affect
> performance too much for the benchmarks to be comparable between the ERT
> and elisp-benchmarks invocations.
>
> I just don't know whether I'd feel comfortable invoking the benchmarks
> in such different ways and presenting the results in a way that would
> make people compare them.
>
> The rest of the issues are trivial: whitespace issues, two different
> files calling Fprovide with the same feature, elb-scroll.el merged into
> elb-smie.el rather than maintaining them as two separate files.  These
> are very definitely not deficiencies in the current elisp-benchmarks
> package, just different conventions.  However, that amounts to
> significant changes to the benchmark .el files overall; rather than
> copies of the elisp-benchmarks files, we now have modified versions and
> would have to port any changes between the two different sets of files.
>
> Ultimately, my current benchmark branch doesn't do what I set out to do,
> which is to share the elisp-benchmarks suite between an unmodified
> elisp-benchmarks and the new ERT framework, yielding comparable results.
> Getting it to work isn't the main problem, comparability of results is.
>
> So it is with some trepidation that I suggest that the best remaining
> option may be to fork or "freeze"/archive elisp-benchmarks and move
> development of benchmarks for current Emacs builds entirely to the ERT
> framework.  Forking causes a lot of extra synchronization work.
> Archiving the package means we will never add new benchmarks for
> pre-make-benchmark Emacs builds.
>
> I'm convinced a "make benchmark" target is worth it.  I also think that
> we should use the ERT framework, because benchmarks and pass-or-fail
> tests are quite similar.
>
> Maybe I'm missing an obvious solution here?

I'd personally drop the requirement of using ERT as a framework for
benchmarks, I'd just move elisp-benchmarks code in emacs core and add
the target.

My 2 cents.

  Andrea