From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Pip Cet via "Emacs development discussions." Newsgroups: gmane.emacs.devel Subject: Re: New "make benchmark" target Date: Sun, 22 Dec 2024 16:04:13 +0000 Message-ID: <87y107g0xc.fsf@protonmail.com> References: <87h679kftn.fsf@protonmail.com> <87msh0j12c.fsf@protonmail.com> <87zfkyfqia.fsf@protonmail.com> <875xnmf2qp.fsf@protonmail.com> Reply-To: Pip Cet Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="1155"; mail-complaints-to="usenet@ciao.gmane.io" Cc: Andrea Corallo , Eli Zaretskii , =?utf-8?Q?Mattias_Engdeg=C3=A5rd?= , Paul Eggert , emacs-devel@gnu.org, =?utf-8?Q?Jo=C3=A3o_T=C3=A1vora?= To: Stefan Kangas Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Sun Dec 22 17:23:17 2024 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1tPOjd-00007K-M3 for ged-emacs-devel@m.gmane-mx.org; Sun, 22 Dec 2024 17:23:17 +0100 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tPOiw-0007uC-Pm; Sun, 22 Dec 2024 11:22:34 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tPORK-000621-34 for emacs-devel@gnu.org; Sun, 22 Dec 2024 11:04:22 -0500 Original-Received: from mail-40133.protonmail.ch ([185.70.40.133]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tPORG-0000PU-Pw for emacs-devel@gnu.org; Sun, 22 Dec 2024 11:04:21 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=protonmail.com; s=protonmail3; t=1734883456; x=1735142656; bh=5NElXNwDcannR7xeaF86oFAd6DHfld1fZfd6peVGwn8=; h=Date:To:From:Cc:Subject:Message-ID:In-Reply-To:References: Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID: Message-ID:BIMI-Selector:List-Unsubscribe:List-Unsubscribe-Post; b=pVD2YJ822borcjOSChFZNo4h3hG1IDOxVZxjOkBFOmNTVi4a1GtksJDvB1CuOrnw6 8RrgF6cKqF53Di3SI4WzQ86RUAUrByGFmDXacIrywCl2iS0PLysx/xw/I3WJHSaLS/ g/13Lm7aP6y58iQERsFfYqT+CfLcdp+ipF4MFjvAi0XE9uoRM0DliTA0ITx37cv7Gr fG3g9/jwIF5IvACbHSv4hzUgxy4l3yZRp6D3VOB+6WsERvYBDdIUVliP6r49FasjzM hhHG7El5EvagYj+N00phC4H64m83DSgWxteZl7aBPHH4rt029Hg07wGSgsSQrlODyw 8BXgbRc+Dr8dw== In-Reply-To: Feedback-ID: 112775352:user:proton X-Pm-Message-ID: c2704d30fd0b82e3fd14c391e170baed3e403540 Received-SPF: pass client-ip=185.70.40.133; envelope-from=pipcet@protonmail.com; helo=mail-40133.protonmail.ch X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-Mailman-Approved-At: Sun, 22 Dec 2024 11:22:31 -0500 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.devel:326860 Archived-At: "Stefan Kangas" writes: > Pip Cet writes: >> We also need to decide on the directory structure; right now, I've >> created a lisp/emacs-lisp/benchmarks/ directory; I'd prefer >> lisp/benchmarks (which would make it easier to exclude the benchmark >> files from compilation), but I don't have a strong preference and others >> should make that decision. (I haven't included the >> lisp/emacs-lisp/subdirs.el file, but if we decide to keep the benchmarks >> in lisp/emacs-lisp/benchmarks/, we'll need to gitignore that, too). > > I don't have a strong opinion here, but maybe this stuff belongs under > test/ even? I'm still working on this, but it turns out it's harder than I thought to turn the .el files for the benchmarks into something that's usable both with ERT and with the existing elisp-benchmarks.el infrastructure. For example, there's the use of elb-bench-directory to locate resource files; ERT has its own function for that, but it turns out one of the resources one benchmark uses is the source file for another benchmark. Usually I'd just use letf around the benchmark call, but that may affect performance too much for the benchmarks to be comparable between the ERT and elisp-benchmarks invocations. I just don't know whether I'd feel comfortable invoking the benchmarks in such different ways and presenting the results in a way that would make people compare them. The rest of the issues are trivial: whitespace issues, two different files calling Fprovide with the same feature, elb-scroll.el merged into elb-smie.el rather than maintaining them as two separate files. These are very definitely not deficiencies in the current elisp-benchmarks package, just different conventions. However, that amounts to significant changes to the benchmark .el files overall; rather than copies of the elisp-benchmarks files, we now have modified versions and would have to port any changes between the two different sets of files. Ultimately, my current benchmark branch doesn't do what I set out to do, which is to share the elisp-benchmarks suite between an unmodified elisp-benchmarks and the new ERT framework, yielding comparable results. Getting it to work isn't the main problem, comparability of results is. So it is with some trepidation that I suggest that the best remaining option may be to fork or "freeze"/archive elisp-benchmarks and move development of benchmarks for current Emacs builds entirely to the ERT framework. Forking causes a lot of extra synchronization work. Archiving the package means we will never add new benchmarks for pre-make-benchmark Emacs builds. I'm convinced a "make benchmark" target is worth it. I also think that we should use the ERT framework, because benchmarks and pass-or-fail tests are quite similar. Maybe I'm missing an obvious solution here? Pip