unofficial mirror of emacs-devel@gnu.org 
 help / color / mirror / code / Atom feed
From: Pip Cet via "Emacs development discussions." <emacs-devel@gnu.org>
To: Stefan Kangas <stefankangas@gmail.com>
Cc: "Andrea Corallo" <acorallo@gnu.org>,
	"Eli Zaretskii" <eliz@gnu.org>,
	"Mattias Engdegård" <mattiase@acm.org>,
	"Paul Eggert" <eggert@cs.ucla.edu>,
	emacs-devel@gnu.org, "João Távora" <joaotavora@gmail.com>
Subject: Re: New "make benchmark" target
Date: Sun, 22 Dec 2024 16:04:13 +0000	[thread overview]
Message-ID: <87y107g0xc.fsf@protonmail.com> (raw)
In-Reply-To: <CADwFkmmRTp1+YEK2TUqc6WsTjk3jHr3NSjoxtbz2QF76v00tKw@mail.gmail.com>

"Stefan Kangas" <stefankangas@gmail.com> writes:

> Pip Cet <pipcet@protonmail.com> writes:

>> We also need to decide on the directory structure; right now, I've
>> created a lisp/emacs-lisp/benchmarks/ directory; I'd prefer
>> lisp/benchmarks (which would make it easier to exclude the benchmark
>> files from compilation), but I don't have a strong preference and others
>> should make that decision.  (I haven't included the
>> lisp/emacs-lisp/subdirs.el file, but if we decide to keep the benchmarks
>> in lisp/emacs-lisp/benchmarks/, we'll need to gitignore that, too).
>
> I don't have a strong opinion here, but maybe this stuff belongs under
> test/ even?

I'm still working on this, but it turns out it's harder than I thought
to turn the .el files for the benchmarks into something that's usable
both with ERT and with the existing elisp-benchmarks.el infrastructure.

For example, there's the use of elb-bench-directory to locate resource
files; ERT has its own function for that, but it turns out one of the
resources one benchmark uses is the source file for another benchmark.
Usually I'd just use letf around the benchmark call, but that may affect
performance too much for the benchmarks to be comparable between the ERT
and elisp-benchmarks invocations.

I just don't know whether I'd feel comfortable invoking the benchmarks
in such different ways and presenting the results in a way that would
make people compare them.

The rest of the issues are trivial: whitespace issues, two different
files calling Fprovide with the same feature, elb-scroll.el merged into
elb-smie.el rather than maintaining them as two separate files.  These
are very definitely not deficiencies in the current elisp-benchmarks
package, just different conventions.  However, that amounts to
significant changes to the benchmark .el files overall; rather than
copies of the elisp-benchmarks files, we now have modified versions and
would have to port any changes between the two different sets of files.

Ultimately, my current benchmark branch doesn't do what I set out to do,
which is to share the elisp-benchmarks suite between an unmodified
elisp-benchmarks and the new ERT framework, yielding comparable results.
Getting it to work isn't the main problem, comparability of results is.

So it is with some trepidation that I suggest that the best remaining
option may be to fork or "freeze"/archive elisp-benchmarks and move
development of benchmarks for current Emacs builds entirely to the ERT
framework.  Forking causes a lot of extra synchronization work.
Archiving the package means we will never add new benchmarks for
pre-make-benchmark Emacs builds.

I'm convinced a "make benchmark" target is worth it.  I also think that
we should use the ERT framework, because benchmarks and pass-or-fail
tests are quite similar.

Maybe I'm missing an obvious solution here?

Pip




  reply	other threads:[~2024-12-22 16:04 UTC|newest]

Thread overview: 29+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2024-12-11 22:37 Improving EQ Pip Cet via Emacs development discussions.
2024-12-12  6:36 ` Eli Zaretskii
2024-12-12  8:23   ` Andrea Corallo
2024-12-12  8:36   ` Pip Cet via Emacs development discussions.
2024-12-12  9:18     ` Eli Zaretskii
2024-12-12  9:35     ` Visuwesh
2024-12-12 10:40     ` Andrea Corallo
2024-12-12 17:46       ` Pip Cet via Emacs development discussions.
2024-12-12 19:09         ` Eli Zaretskii
2024-12-12 10:53     ` New "make benchmark" target Stefan Kangas
2024-12-12 10:59       ` Andrea Corallo
2024-12-12 16:53         ` Pip Cet via Emacs development discussions.
2024-12-13  0:49           ` Stefan Kangas
2024-12-13  7:37             ` Andrea Corallo
2024-12-14 12:00               ` Stefan Kangas
2024-12-14 14:06                 ` Stefan Monnier
2024-12-14 11:34             ` Pip Cet via Emacs development discussions.
2024-12-14 11:58               ` Stefan Kangas
2024-12-14 20:07                 ` Pip Cet via Emacs development discussions.
2024-12-14 20:20                   ` João Távora
2024-12-15  0:57                   ` Stefan Kangas
2024-12-22 16:04                     ` Pip Cet via Emacs development discussions. [this message]
2024-12-15  0:58                   ` Stefan Kangas
2024-12-12 10:42 ` Improving EQ Óscar Fuentes
2024-12-12 10:50   ` Andrea Corallo
2024-12-12 11:21     ` Óscar Fuentes
2024-12-13 12:24       ` Pip Cet via Emacs development discussions.
2024-12-12 17:05     ` Pip Cet via Emacs development discussions.
2024-12-12 18:10     ` John ff

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://www.gnu.org/software/emacs/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=87y107g0xc.fsf@protonmail.com \
    --to=emacs-devel@gnu.org \
    --cc=acorallo@gnu.org \
    --cc=eggert@cs.ucla.edu \
    --cc=eliz@gnu.org \
    --cc=joaotavora@gmail.com \
    --cc=mattiase@acm.org \
    --cc=pipcet@protonmail.com \
    --cc=stefankangas@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/emacs.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).