From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Pip Cet via "Emacs development discussions." Newsgroups: gmane.emacs.devel Subject: Re: New "make benchmark" target Date: Mon, 30 Dec 2024 17:25:44 +0000 Message-ID: <87o70txew4.fsf@protonmail.com> References: <87h679kftn.fsf@protonmail.com> <87y107g0xc.fsf@protonmail.com> <87frm51jkr.fsf@protonmail.com> <861pxpp88q.fsf@gnu.org> <87frm5z06l.fsf@protonmail.com> <86msgdnqmv.fsf@gnu.org> <87wmfhxjce.fsf@protonmail.com> <86jzbhnmzg.fsf@gnu.org> Reply-To: Pip Cet Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="28005"; mail-complaints-to="usenet@ciao.gmane.io" Cc: acorallo@gnu.org, stefankangas@gmail.com, mattiase@acm.org, eggert@cs.ucla.edu, emacs-devel@gnu.org To: Eli Zaretskii Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Mon Dec 30 18:34:36 2024 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1tSJf1-000788-Kr for ged-emacs-devel@m.gmane-mx.org; Mon, 30 Dec 2024 18:34:35 +0100 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tSJeD-0004pn-A2; Mon, 30 Dec 2024 12:33:49 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tSJWk-0003Sm-07 for emacs-devel@gnu.org; Mon, 30 Dec 2024 12:26:02 -0500 Original-Received: from mail-40134.protonmail.ch ([185.70.40.134]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tSJWc-0004yR-Sf; Mon, 30 Dec 2024 12:25:59 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=protonmail.com; s=protonmail3; t=1735579550; x=1735838750; bh=XoXAhqgevQtfgk/Jcjh+Db4d2ObaXUxHTPV4WxUBI98=; h=Date:To:From:Cc:Subject:Message-ID:In-Reply-To:References: Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID: Message-ID:BIMI-Selector:List-Unsubscribe:List-Unsubscribe-Post; b=yKT/V6zYKqGatoWjLToYLhV1Ch1IkMaUImOdk9diU6ww2eKxLUuARJCRKX8rRS5Hp P2w5I6Jw+UNrsDpwR7lm4oLQIiP+l4izq+hT5/qlmD2fmAXtl+SCAgkwd45hl8U+av vhp3O+FFlYcfQDfwXKM31FKQxeRrJKcYlIn/SOQ9t7JonYqq5dy37g9D2ED8AjdF8w MF2a4djU9erP1t+LDZDGkt0cylVXlO8TNNWUYMDR0qjEFAFBgN4gf6IqyznPsYmgjf yfeUQIS7Yd00XkAO2K2SlINsCfEBjGLxywOICkN51UhlWSly4yVfyQdytnVvzrVSla PVl3Hq9ikuDjw== In-Reply-To: <86jzbhnmzg.fsf@gnu.org> Feedback-ID: 112775352:user:proton X-Pm-Message-ID: c0aeeca3a2605a966c0a52f2db0773e20ed2f88b Received-SPF: pass client-ip=185.70.40.134; envelope-from=pipcet@protonmail.com; helo=mail-40134.protonmail.ch X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Mailman-Approved-At: Mon, 30 Dec 2024 12:33:38 -0500 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.devel:327453 Archived-At: "Eli Zaretskii" writes: Top-posted TL;DR: let's call Andrea's code "make elisp-benchmarks" and include it now? That would preserve the Git history and importantly (to me) reserve the name for now. >> Date: Mon, 30 Dec 2024 15:49:30 +0000 >> From: Pip Cet >> Cc: acorallo@gnu.org, stefankangas@gmail.com, mattiase@acm.org, eggert@c= s.ucla.edu, emacs-devel@gnu.org, joaotavora@gmail.com >> >> >> https://lists.gnu.org/archive/html/emacs-devel/2024-12/msg00595.html >> > >> > Thanks, but AFAICT this just says that you intended to use/extend ERT >> > to run this benchmark suite, but doesn't explain why you think using >> > ERT would be an advantage worthy of keeping. >> >> I think some advantages are stated in that email: the ERT tagging >> mechanism is more general, works, and can be extended (I describe one >> such extension). All that isn't currently true for elisp-benchmarks. > > Unlike the rest of the test suite, where we need a way to be able to > run individual tests, a benchmark suite is much more likely to run as > a whole, because benchmarking a single kind of jobs in Emacs is much > less useful than producing a benchmark of a representative sample of > jobs. So I'm not sure this particular aspect is such a serious Not my experience. Running the entire suite is much more likely not to produce usable data due to such issues as CPU thermal management (for example: the first few tests are run at full clock speed and heat up the system so much that thermal throttling is activated; the next few tests are run at a reduced rate while the fan is running; eventually we run out of amperes that we're allowed to drain the battery by and reduce clock speed even further; this results in reduced temperature, so the fan speed is reduced, which means we will eventually decide to try a higher clock speed again, which will work for a while only before repeating the cycle. The whole thing will appear regular enough we won't notice the data is bad, but it will be, until we rerun the test on the same system in a different room and get wildly different results). A single-second test run in a loop produces the occasional mid-stream result which is actually useful (and promptly lost to the averaging mechanism of elisp-benchmarks). Benchmarking is hard, and I wouldn't have provided this very verbose example if I hadn't seen "paradoxical" results that can only be explained by such mechanisms. We need to move away from average run times either way, and that requires code changes. And I don't usually run ERT tests individually, while I'm trying to get in the habit of running the (non-expensive) test suite before I push. > problem, certainly if it makes the job of adding the suite much > harder. I don't think time-to-Emacs is very different for the two approaches. The difference is the post-merge work. >> The other big difference is resource management, which elisp-benchmarks >> does via a global variable, reusing one test as data for another. ERT >> has a somewhat better mechanism. > > This is just an argument for cleaner, more elegant code, right? If > so, I think we can live with the issue. Having a ready-t-run > benchmark suite is so much more important that I'm prepared to make > compromises. > >> > Why do you think it is wrong to do the (AFAIU) simple change that >> > Andrea proposed? >> >> Because it's a de facto commitment to not doing it in ERT. > > Probably, but that in itself is not a catastrophe, surely? Hmm. I disagree, but I think the real catastrophe would be wasting the make target name, not the inclusion of another directory. My suggestion is a compromise: add "make elisp-benchmarks" now, using Andrea's code, then consider more complicated ERT-based approaches without being in any hurry to do so. But, also, let's agree that the ERT-based approaches are "allowed" to reuse the elisp-benchmarks code without providing comparable results or a porting mechanism, and keep the "make benchmark" name reserved for a while. My prediction is that it will turn out "make elisp-benchmarks" doesn't usually provide very useful results, and expansion of the test framework to produce useful results is easier reusing the ERT framework. >> It might be relevant that elisp-benchmarks hasn't seen very active >> development lately. I think switching to ERT might help there, if only >> because of the mailing list traffic. > > My vote is to get the job done the fastest way possible. Tests are Well, the ERT patch is ready "right now" (meaning it needs rebasing). The elisp-benchmarks code would require, at least, whitespace fixes ;-) > not Emacs itself, they are means towards a certain end, and so can use > less clean code and designs, at least IMO. To be perfectly honest, I was worried about the commit history because in the Good Old CVS Days, file renames were more of a problem than they are with git. With git, I would prefer one "this is elisp-benchmarks" commit even if the subsequent history modifies and moves those files. So no reason not to do that now? Thanks for your patience. I can prepare a git branch doing that. As for the git history rewriting magic code, I've heard from Joao and the other developer involved; he doesn't have a copyright assignment, but also thinks this is small enough to be paperwork-exempt, so we can (probably) include a script in admin/ to migrate elpa packages to core. My preference would be a top-level directory called "elisp-benchmarks", but ultimately that's a minor question, so just let me know the preferred destination. Pip