From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Eli Zaretskii Newsgroups: gmane.emacs.devel Subject: Re: New "make benchmark" target Date: Sat, 04 Jan 2025 20:33:01 +0200 Message-ID: <86r05ibfaa.fsf@gnu.org> References: <87h679kftn.fsf@protonmail.com> <87frm5z06l.fsf@protonmail.com> <86msgdnqmv.fsf@gnu.org> <87wmfhxjce.fsf@protonmail.com> <86jzbhnmzg.fsf@gnu.org> <87o70txew4.fsf@protonmail.com> <871pxorh30.fsf@protonmail.com> <86wmfgm3a5.fsf@gnu.org> <87pll2fsj7.fsf@protonmail.com> Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="15814"; mail-complaints-to="usenet@ciao.gmane.io" Cc: acorallo@gnu.org, stefankangas@gmail.com, mattiase@acm.org, eggert@cs.ucla.edu, emacs-devel@gnu.org To: Pip Cet Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Sat Jan 04 19:34:25 2025 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1tU8ye-0003yA-Vp for ged-emacs-devel@m.gmane-mx.org; Sat, 04 Jan 2025 19:34:25 +0100 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tU8y1-0006kJ-0V; Sat, 04 Jan 2025 13:33:45 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tU8xz-0006jo-HZ for emacs-devel@gnu.org; Sat, 04 Jan 2025 13:33:43 -0500 Original-Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tU8xx-0000pC-S0; Sat, 04 Jan 2025 13:33:42 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date: mime-version; bh=p4OTsdCw6a18tZFo/6ZSzqBhFgPIfCyoS8mBPXH5M1Q=; b=hVEeGYy4CPeU H9mRs7/QAbtnTGMFHNpeV3Xx8pcwhGBFI0+FwuWutSr/u28v79Mo0C0e1mqWUWqpJ90/lUi8kAb+j KWIPFEbBlEYszteOXVALoMiS8C9Twg1CTNic5GxLh6nMB+lq0vMh5rNdWLlfrs7V6gNhHrfa3Hw+e /xR/HSWvpcTAL37gWh3eUTR82r0m6lvZe7mQ1/VtNYw/0rt/LQ3g8tnqtIbLqmWBKdBQ/rioX87Do NPVWIen+bC3TEQvk1s9DzqbMfR0TvjhL0Qqcqde1Q1VE70X1LJ1MDR5xZXLm9SE6M1lgDomr1MRnB A2Nd4BhpIO5HfVf3/PxjAg==; In-Reply-To: <87pll2fsj7.fsf@protonmail.com> (message from Pip Cet on Sat, 04 Jan 2025 16:34:24 +0000) X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.devel:327677 Archived-At: > Date: Sat, 04 Jan 2025 16:34:24 +0000 > From: Pip Cet > Cc: acorallo@gnu.org, stefankangas@gmail.com, mattiase@acm.org, eggert@cs.ucla.edu, emacs-devel@gnu.org > > "Eli Zaretskii" writes: > > > Since you've pushed that to a branch, I suggest to submit bug reports > > about these issues, using "[scratch/elisp-benchmarks]" in the Subject > > of the bug. > > I've studied the issues a bit more. This is a bit long, but in summary, > I think improving elisp-benchmarks.el (the specific file, not the entire > package, which I still intend to reuse) would take more time than > starting from ERT You mean, starting from scratch?? How can this be less work than fixing whatever bugs you found in the benchmarks (assuming that we want to fix all of them)? > I'm reporting a small number of elisp-benchmarks "bugs" (I think the > term is likely to be contentious; I use it because that's what the > mailing list is called). All of them are fixable. Most of them are > easily fixable by moving to ERT. You mean, if we move to ERT, which by itself is a significant job, then some or most of these bugs could be fixed as a side effect or with much less work, is that it? That could be so, but then the move to ERT itself is not a small job, so we need to take that into consideration when deciding whether to move to ERT right now. > My conclusion is that elisp-benchmarks.el (again, the benchmarks are > fine) isn't the right way forward. Well, you though that to begin with, so forgive me if I say that I'd like a second independent opinion in this case. > I'm happy to change the scratch/elisp-benchmarks branch in the ways > we've discussed, and it should be merged, but if someone decides to > incrementally solve some of the issues, that, while not very harmful, > would be an inefficient use of resources. Let's hear what Andrea thinks about the issues you reported (let's please discuss them on the bug tracker, not here), and let's take it from there. > Benchmarks need a test framework. I don't necessarily agree. A benchmark doesn't have to have a "correct" or "expected" result, so a test framework is not necessarily justified or needed.