From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Eli Zaretskii Newsgroups: gmane.emacs.devel Subject: Re: New "make benchmark" target Date: Mon, 30 Dec 2024 18:40:03 +0200 Message-ID: <86jzbhnmzg.fsf@gnu.org> References: <87h679kftn.fsf@protonmail.com> <875xnmf2qp.fsf@protonmail.com> <87y107g0xc.fsf@protonmail.com> <87frm51jkr.fsf@protonmail.com> <861pxpp88q.fsf@gnu.org> <87frm5z06l.fsf@protonmail.com> <86msgdnqmv.fsf@gnu.org> <87wmfhxjce.fsf@protonmail.com> Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="30662"; mail-complaints-to="usenet@ciao.gmane.io" Cc: acorallo@gnu.org, stefankangas@gmail.com, mattiase@acm.org, eggert@cs.ucla.edu, emacs-devel@gnu.org To: Pip Cet Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Mon Dec 30 17:40:48 2024 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1tSIow-0007pI-Ra for ged-emacs-devel@m.gmane-mx.org; Mon, 30 Dec 2024 17:40:46 +0100 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tSIoM-0001ds-Np; Mon, 30 Dec 2024 11:40:10 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tSIoK-0001d5-0P for emacs-devel@gnu.org; Mon, 30 Dec 2024 11:40:08 -0500 Original-Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tSIoI-0000hm-8r; Mon, 30 Dec 2024 11:40:06 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date: mime-version; bh=5z9SmORNf0jmudBRdsiET1ZSYdOIYiOoSWEZRcqHIJQ=; b=Mkx4QBG7XK8M 16VnJ7N2EgDQGiXJ9h7QOElT0ETgwDcsJFC/EfCujbWbZjg99yjuX+PzsG4yCbYVOBWbYr3RaL4de JgfstQzdcAD4i3GX1rftOQReGuDwZeb5Z9KbYh/yHewQ4nPhPVccg/aUpOCBpeCvKio897WBCswwK yMkc/h6FXcL1jOHKs6Fndj3TYACAeSTcKPflaSB7J6mVvdvvYFEmvLd/D/E9a7lh7QsH2F4Am1nyx JebfAM091q2WXbpJY9S1qFrKpbE4Gergapo7swUa4sMqv14hulPeeeyV5gRs09EvLGc7Z1mH4xdiM o6B7PvyZK6YpXSWlUBAJSQ==; In-Reply-To: <87wmfhxjce.fsf@protonmail.com> (message from Pip Cet on Mon, 30 Dec 2024 15:49:30 +0000) X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.devel:327448 Archived-At: > Date: Mon, 30 Dec 2024 15:49:30 +0000 > From: Pip Cet > Cc: acorallo@gnu.org, stefankangas@gmail.com, mattiase@acm.org, eggert@cs.ucla.edu, emacs-devel@gnu.org, joaotavora@gmail.com > > >> https://lists.gnu.org/archive/html/emacs-devel/2024-12/msg00595.html > > > > Thanks, but AFAICT this just says that you intended to use/extend ERT > > to run this benchmark suite, but doesn't explain why you think using > > ERT would be an advantage worthy of keeping. > > I think some advantages are stated in that email: the ERT tagging > mechanism is more general, works, and can be extended (I describe one > such extension). All that isn't currently true for elisp-benchmarks. Unlike the rest of the test suite, where we need a way to be able to run individual tests, a benchmark suite is much more likely to run as a whole, because benchmarking a single kind of jobs in Emacs is much less useful than producing a benchmark of a representative sample of jobs. So I'm not sure this particular aspect is such a serious problem, certainly if it makes the job of adding the suite much harder. > The other big difference is resource management, which elisp-benchmarks > does via a global variable, reusing one test as data for another. ERT > has a somewhat better mechanism. This is just an argument for cleaner, more elegant code, right? If so, I think we can live with the issue. Having a ready-t-run benchmark suite is so much more important that I'm prepared to make compromises. > > Why do you think it is wrong to do the (AFAIU) simple change that > > Andrea proposed? > > Because it's a de facto commitment to not doing it in ERT. Probably, but that in itself is not a catastrophe, surely? > It might be relevant that elisp-benchmarks hasn't seen very active > development lately. I think switching to ERT might help there, if only > because of the mailing list traffic. My vote is to get the job done the fastest way possible. Tests are not Emacs itself, they are means towards a certain end, and so can use less clean code and designs, at least IMO.