From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Andrea Corallo Newsgroups: gmane.emacs.devel Subject: Re: New "make benchmark" target Date: Sun, 29 Dec 2024 05:47:29 -0500 Message-ID: References: <87h679kftn.fsf@protonmail.com> <87msh0j12c.fsf@protonmail.com> <87zfkyfqia.fsf@protonmail.com> <875xnmf2qp.fsf@protonmail.com> <87y107g0xc.fsf@protonmail.com> Mime-Version: 1.0 Content-Type: text/plain Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="5918"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Gnus/5.13 (Gnus v5.13) Cc: Stefan Kangas , Eli Zaretskii , Mattias =?utf-8?Q?Engdeg=C3=A5rd?= , Paul Eggert , emacs-devel@gnu.org, =?utf-8?B?Sm/Do28gVMOhdm9yYQ==?= To: Pip Cet Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Sun Dec 29 11:48:11 2024 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1tRqqB-0001N0-0z for ged-emacs-devel@m.gmane-mx.org; Sun, 29 Dec 2024 11:48:11 +0100 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tRqpa-00053x-A7; Sun, 29 Dec 2024 05:47:34 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tRqpY-00053m-1X for emacs-devel@gnu.org; Sun, 29 Dec 2024 05:47:32 -0500 Original-Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tRqpW-0005Zg-7Q; Sun, 29 Dec 2024 05:47:30 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To: From; bh=OjZAY3AVS0l9+rV1dvnYlUfuFI+MI5jSBKSwkzTytvw=; b=r515p4LZmx1Je/d2cID7 +9IEEeFfBFNMIXob8afe+V109UHEHV0FmxXoRGqag6Jr0+ozRU37Of1o3KGZe8rEZtapQ8nEOIOGp kfeGnwQC+CGT1DqvXvlrABjUWdJzFKhUwG5lUrV3vT7FOu+RRG27MkH08hyxeLRQ6pAlINJQh7soE mB6+Wq8FfUs5bzVDIMG5n9lFWcJMIexNUwYwaVi3/9ueOupqm2BTTTFPB2uDyY4id3h6ZT18RqtPn U/H75gxd+O490xnn0kOs9mU+/StUj8kvdGi1G2BkKh1A2p808kx1y30UEFif45JJSvHH4XpTJDh14 01gL1GgPZooQGw==; Original-Received: from acorallo by fencepost.gnu.org with local (Exim 4.90_1) (envelope-from ) id 1tRqpV-00063B-DX; Sun, 29 Dec 2024 05:47:29 -0500 In-Reply-To: <87y107g0xc.fsf@protonmail.com> (Pip Cet's message of "Sun, 22 Dec 2024 16:04:13 +0000") X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.devel:327330 Archived-At: Pip Cet writes: > "Stefan Kangas" writes: > >> Pip Cet writes: > >>> We also need to decide on the directory structure; right now, I've >>> created a lisp/emacs-lisp/benchmarks/ directory; I'd prefer >>> lisp/benchmarks (which would make it easier to exclude the benchmark >>> files from compilation), but I don't have a strong preference and others >>> should make that decision. (I haven't included the >>> lisp/emacs-lisp/subdirs.el file, but if we decide to keep the benchmarks >>> in lisp/emacs-lisp/benchmarks/, we'll need to gitignore that, too). >> >> I don't have a strong opinion here, but maybe this stuff belongs under >> test/ even? > > I'm still working on this, but it turns out it's harder than I thought > to turn the .el files for the benchmarks into something that's usable > both with ERT and with the existing elisp-benchmarks.el infrastructure. > > For example, there's the use of elb-bench-directory to locate resource > files; ERT has its own function for that, but it turns out one of the > resources one benchmark uses is the source file for another benchmark. > Usually I'd just use letf around the benchmark call, but that may affect > performance too much for the benchmarks to be comparable between the ERT > and elisp-benchmarks invocations. > > I just don't know whether I'd feel comfortable invoking the benchmarks > in such different ways and presenting the results in a way that would > make people compare them. > > The rest of the issues are trivial: whitespace issues, two different > files calling Fprovide with the same feature, elb-scroll.el merged into > elb-smie.el rather than maintaining them as two separate files. These > are very definitely not deficiencies in the current elisp-benchmarks > package, just different conventions. However, that amounts to > significant changes to the benchmark .el files overall; rather than > copies of the elisp-benchmarks files, we now have modified versions and > would have to port any changes between the two different sets of files. > > Ultimately, my current benchmark branch doesn't do what I set out to do, > which is to share the elisp-benchmarks suite between an unmodified > elisp-benchmarks and the new ERT framework, yielding comparable results. > Getting it to work isn't the main problem, comparability of results is. > > So it is with some trepidation that I suggest that the best remaining > option may be to fork or "freeze"/archive elisp-benchmarks and move > development of benchmarks for current Emacs builds entirely to the ERT > framework. Forking causes a lot of extra synchronization work. > Archiving the package means we will never add new benchmarks for > pre-make-benchmark Emacs builds. > > I'm convinced a "make benchmark" target is worth it. I also think that > we should use the ERT framework, because benchmarks and pass-or-fail > tests are quite similar. > > Maybe I'm missing an obvious solution here? I'd personally drop the requirement of using ERT as a framework for benchmarks, I'd just move elisp-benchmarks code in emacs core and add the target. My 2 cents. Andrea