From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Pip Cet via "Emacs development discussions." Newsgroups: gmane.emacs.devel Subject: Re: New "make benchmark" target Date: Tue, 31 Dec 2024 14:01:00 +0000 Message-ID: <87pll8nean.fsf@protonmail.com> References: <87h679kftn.fsf@protonmail.com> <87frm5z06l.fsf@protonmail.com> <86msgdnqmv.fsf@gnu.org> <87wmfhxjce.fsf@protonmail.com> <86jzbhnmzg.fsf@gnu.org> <87o70txew4.fsf@protonmail.com> <871pxorh30.fsf@protonmail.com> <86wmfgm3a5.fsf@gnu.org> Reply-To: Pip Cet Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="19706"; mail-complaints-to="usenet@ciao.gmane.io" Cc: acorallo@gnu.org, stefankangas@gmail.com, mattiase@acm.org, eggert@cs.ucla.edu, emacs-devel@gnu.org To: Eli Zaretskii Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Tue Dec 31 15:16:03 2024 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1tSd2R-0004xJ-3O for ged-emacs-devel@m.gmane-mx.org; Tue, 31 Dec 2024 15:16:03 +0100 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tSd1p-0005lq-76; Tue, 31 Dec 2024 09:15:25 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tSco2-0003iU-DU for emacs-devel@gnu.org; Tue, 31 Dec 2024 09:01:11 -0500 Original-Received: from mail-10630.protonmail.ch ([79.135.106.30]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tScnz-0006iY-A3 for emacs-devel@gnu.org; Tue, 31 Dec 2024 09:01:10 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=protonmail.com; s=protonmail3; t=1735653664; x=1735912864; bh=LiEOURgTBGErVjuVfG9wKzHaejEa3gYkdTe3trrbLwk=; h=Date:To:From:Cc:Subject:Message-ID:In-Reply-To:References: Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID: Message-ID:BIMI-Selector:List-Unsubscribe:List-Unsubscribe-Post; b=AfTus/X22lsuT9YFd6LRtwRAYrNOP28ic06Jy6btFSAEQGRH46rEmkSZJpnrsk/5I X0MhCFjlfxuylUkm2mfUIOmjrxb5CXbipnuhbKEvU/7JRfknDL+S4EuK/82t1JQpiZ VAuH6sLO181RLmxY6f9JZTQUeyciRce67OsR0whnB9hwgtixUIEC/m13D4UxiUxyq6 WxUzdRGZoNZZxWFjW4ZsmWjKUEqsCVzoJWBAw74qjhRmKEG4nv9Oj2qrnNgho4QVuS CS8TY7doQE4ISpQrr+zEvCLJjkCR4kmCMTW12hdgJGKGyVrel6mMLkzlJJEfR+ZmjD cvlyvbLRCBoNQ== In-Reply-To: <86wmfgm3a5.fsf@gnu.org> Feedback-ID: 112775352:user:proton X-Pm-Message-ID: 109c039f0c515b7f2fa955f7c5d9e9786a2dde28 Received-SPF: pass client-ip=79.135.106.30; envelope-from=pipcet@protonmail.com; helo=mail-10630.protonmail.ch X-Spam_score_int: -10 X-Spam_score: -1.1 X-Spam_bar: - X-Spam_report: (-1.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, FREEMAIL_REPLY=1, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-Mailman-Approved-At: Tue, 31 Dec 2024 09:15:19 -0500 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.devel:327507 Archived-At: "Eli Zaretskii" writes: >> Date: Mon, 30 Dec 2024 21:34:55 +0000 >> From: Pip Cet >> Cc: Eli Zaretskii , stefankangas@gmail.com, mattiase@acm.o= rg, eggert@cs.ucla.edu, emacs-devel@gnu.org >> >> > I'm open to patches to elisp-benchmarks (and to its hypothetical copy = in >> > emacs-core). My opinion that something can potentially be improved in >> >> What's the best way to report the need for such improvements? > > Since you've pushed that to a branch, I suggest to submit bug reports > about these issues, using "[scratch/elisp-benchmarks]" in the Subject > of the bug. Okay. >> > it (why not), but I personally ATM don't understand the need for ERT. >> >> Let's focus on the basics right now: people know how to write ERT tests. >> We have hundreds of them. Some of them could be benchmarks, and we want >> to make that as easy as possible. > > We can later add more benchmarks using ERT. There's no contradiction. I agree. There's definitely no "right now" need for ERT, I was explaining why it's the change I'll be investigating. >> It also allows a third class of tests: stress tests which we want to >> execute more often than once per test run, which identify occasional >> failures in code that needs to be executed very often to establish >> stability (think bug#75105: (cl-random 1.0e+INF) produces an incorrect >> result once every 8 million runs). IIRC, right now ERT uses ad-hoc >> loops for such tests, but it'd be nicer to expose the repetition count >> in the framework (I'm not going to run the non-expensive testsuite on >> FreeDOS if that means waiting for a million iterations on an emulated >> machine). >> >> (I also think we should introduce an ert-how structure that describes ho= w >> a test is to be run: do we want to inhibit GC or allow it? Run some >> warm-up test runs or not? What's the expected time, and when should we >> time out? We can't run the complete matrix for all tests, so we need >> some hints in the test, and the lack of a test declaration in >> elisp-benchmarks hurts us there). > > These seem to be long-term goals of improving the benchmark suite. > They are fine by me, but I don't see why they should preclude > installing the benchmarks we have without first converting them to > ERT. We can do that later, if we decide it's worth the effort. We seem to agree here: my intention, too, is to merge the elisp-benchmarks branch ASAP. Let's establish which changes are required on that branch, then do a synchronized rebase-merge to preserve history? Pip