From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Eli Zaretskii Newsgroups: gmane.emacs.devel Subject: Re: New "make benchmark" target Date: Tue, 31 Dec 2024 14:43:14 +0200 Message-ID: <86wmfgm3a5.fsf@gnu.org> References: <87h679kftn.fsf@protonmail.com> <87frm51jkr.fsf@protonmail.com> <861pxpp88q.fsf@gnu.org> <87frm5z06l.fsf@protonmail.com> <86msgdnqmv.fsf@gnu.org> <87wmfhxjce.fsf@protonmail.com> <86jzbhnmzg.fsf@gnu.org> <87o70txew4.fsf@protonmail.com> <871pxorh30.fsf@protonmail.com> Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="14146"; mail-complaints-to="usenet@ciao.gmane.io" Cc: acorallo@gnu.org, stefankangas@gmail.com, mattiase@acm.org, eggert@cs.ucla.edu, emacs-devel@gnu.org To: Pip Cet Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Tue Dec 31 13:44:09 2024 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1tSbbU-0003YA-P5 for ged-emacs-devel@m.gmane-mx.org; Tue, 31 Dec 2024 13:44:08 +0100 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tSbak-0005QO-7B; Tue, 31 Dec 2024 07:43:22 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tSbai-0005Q9-3s for emacs-devel@gnu.org; Tue, 31 Dec 2024 07:43:20 -0500 Original-Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tSbag-0001xl-PW; Tue, 31 Dec 2024 07:43:18 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date: mime-version; bh=7jn3F8whMd4jp/52mc7JzkbaOG5RfGP55BCq6wXkVmM=; b=mCeBZSslhM2f aUntwdYwjCc/4ZyZ9+lxAIzH/bLCXzUd+ApYyc/4qtsI7XIHEsiPEkqxhAlmlOHO0zRzTgeDtmhrP MfY6rKsmQcfDNVnclT0MpgavN8MdnrBlVeQ5glUrvrphtEJ3zX+7LfKZQcd2CVi2UNQXVR7I+9vGp ZnrnJDliKmD5szXLrWMPQad/O7LVRPWv7KZ4LVY+slFGmKj1AMJUsTfAx0lW/hKCM05eUrhilQBoj yntRYDniI9Js0bnpO/Qk+ern/ReZWHF+mDkrQCn3BHGK386bYbDG58Gd7JN9bbNp/uOFK4j7r5htm 1+OsmEM98czdUQ+t3yEdVA==; In-Reply-To: <871pxorh30.fsf@protonmail.com> (message from Pip Cet on Mon, 30 Dec 2024 21:34:55 +0000) X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.devel:327496 Archived-At: > Date: Mon, 30 Dec 2024 21:34:55 +0000 > From: Pip Cet > Cc: Eli Zaretskii , stefankangas@gmail.com, mattiase@acm.org, eggert@cs.ucla.edu, emacs-devel@gnu.org > > > I'm open to patches to elisp-benchmarks (and to its hypothetical copy in > > emacs-core). My opinion that something can potentially be improved in > > What's the best way to report the need for such improvements? Since you've pushed that to a branch, I suggest to submit bug reports about these issues, using "[scratch/elisp-benchmarks]" in the Subject of the bug. > > it (why not), but I personally ATM don't understand the need for ERT. > > Let's focus on the basics right now: people know how to write ERT tests. > We have hundreds of them. Some of them could be benchmarks, and we want > to make that as easy as possible. We can later add more benchmarks using ERT. There's no contradiction. > It also allows a third class of tests: stress tests which we want to > execute more often than once per test run, which identify occasional > failures in code that needs to be executed very often to establish > stability (think bug#75105: (cl-random 1.0e+INF) produces an incorrect > result once every 8 million runs). IIRC, right now ERT uses ad-hoc > loops for such tests, but it'd be nicer to expose the repetition count > in the framework (I'm not going to run the non-expensive testsuite on > FreeDOS if that means waiting for a million iterations on an emulated > machine). > > (I also think we should introduce an ert-how structure that describes how > a test is to be run: do we want to inhibit GC or allow it? Run some > warm-up test runs or not? What's the expected time, and when should we > time out? We can't run the complete matrix for all tests, so we need > some hints in the test, and the lack of a test declaration in > elisp-benchmarks hurts us there). These seem to be long-term goals of improving the benchmark suite. They are fine by me, but I don't see why they should preclude installing the benchmarks we have without first converting them to ERT. We can do that later, if we decide it's worth the effort.