From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Stefan Monnier via "Bug reports for GNU Emacs, the Swiss army knife of text editors" Newsgroups: gmane.emacs.bugs Subject: bug#75359: [scratch/elisp-benchmarks] test dependencies Date: Mon, 06 Jan 2025 09:35:40 -0500 Message-ID: References: <878qrqfse2.fsf@protonmail.com> Reply-To: Stefan Monnier Mime-Version: 1.0 Content-Type: text/plain Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="39506"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Gnus/5.13 (Gnus v5.13) Cc: pipcet@protonmail.com, 75359@debbugs.gnu.org To: Andrea Corallo Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Mon Jan 06 15:36:19 2025 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1tUoDK-000A8O-TJ for geb-bug-gnu-emacs@m.gmane-mx.org; Mon, 06 Jan 2025 15:36:19 +0100 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tUoD7-0002PE-JV; Mon, 06 Jan 2025 09:36:05 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tUoD5-0002OU-FB for bug-gnu-emacs@gnu.org; Mon, 06 Jan 2025 09:36:03 -0500 Original-Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1tUoD5-0004MU-0z for bug-gnu-emacs@gnu.org; Mon, 06 Jan 2025 09:36:03 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debbugs.gnu.org; s=debbugs-gnu-org; h=MIME-Version:Date:References:In-Reply-To:From:To:Subject; bh=qbWH8EgQyoa/NIlylSgYbUQdsRIexzG/+vQy5sjiEMk=; b=py1YWYvIpGXAR9SbEMkZagu1n85eNSqksLOkWmpqOs/2aLHL4enk4DkK/LGtQrdMMNC3KEfVCJ5lz7+VaKXinbpMj1xjqiBfIgTIOOslTt9lC8qcCMmhiVKtQ438yFXlBfi+HxNXXNZZfSHKi8d6Hafoa9D3tikjPo77SdyIRVUIrEFhHHCzaDZ0UuWPo7LeCkczBBQ35fuJ+HCdKOJXgRyuNLdGz5yUckebnLjJ4gHXRJFZ/JvQ7oYeelnAzouhW3y2vT+knQuy+lzGCJEXDI60pjuffSFce+By0THQSNrPn60kv4NG1w1mxnWlJ/KD0PRBWreKZgPI6vdcPZMxkg==; Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1tUoD4-0007kC-PO for bug-gnu-emacs@gnu.org; Mon, 06 Jan 2025 09:36:02 -0500 X-Loop: help-debbugs@gnu.org Resent-From: Stefan Monnier Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Mon, 06 Jan 2025 14:36:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 75359 X-GNU-PR-Package: emacs X-Debbugs-Original-Cc: Pip Cet , "Pip Cet via Bug reports for GNU Emacs, the Swiss army knife of text editors" , 75359@debbugs.gnu.org Original-Received: via spool by 75359-submit@debbugs.gnu.org id=B75359.173617415129724 (code B ref 75359); Mon, 06 Jan 2025 14:36:02 +0000 Original-Received: (at 75359) by debbugs.gnu.org; 6 Jan 2025 14:35:51 +0000 Original-Received: from localhost ([127.0.0.1]:37465 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1tUoCs-0007jM-Nr for submit@debbugs.gnu.org; Mon, 06 Jan 2025 09:35:51 -0500 Original-Received: from mailscanner.iro.umontreal.ca ([132.204.25.50]:40493) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1tUoCq-0007j5-5a for 75359@debbugs.gnu.org; Mon, 06 Jan 2025 09:35:49 -0500 Original-Received: from pmg3.iro.umontreal.ca (localhost [127.0.0.1]) by pmg3.iro.umontreal.ca (Proxmox) with ESMTP id 608E44410E5; Mon, 6 Jan 2025 09:35:42 -0500 (EST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=iro.umontreal.ca; s=mail; t=1736174141; bh=V7RdlGDlA+L9ZUI5JbZXx393Mu92bYN7VDRWGUljfB4=; h=From:To:Cc:Subject:In-Reply-To:References:Date:From; b=DRwZrS5ihhQiSDsI52DavIugFZwQPwZbGIVk1/RdYCf5mOD+fUEGS0P9qB5w5TLQ8 NVVuqvNCudQcgOUqnEQHa6guNmeNh95L14b4lgVroMDruf7LvetzhnIiL39GMsLUnv PGjQmF5oh4KyfojPG4Y5TTATeVzb/U9jrlAcaups14/nDCs0S+++51LN/z1VfPkNx0 GoVIxmRuKRngWQyaAH3vgUjTvhBMVE/DSEEeaeAG5Oo75nIRhT0X7FOMai0uAxTYZu EcInICmpzyu85r+6v3uJ5zCGNVK2NQv/okthPHbm859OvKcT9BPrFs8+VjK00vNbTN NhUdeqcdlg98A== Original-Received: from mail01.iro.umontreal.ca (unknown [172.31.2.1]) by pmg3.iro.umontreal.ca (Proxmox) with ESMTP id 2244D441090; Mon, 6 Jan 2025 09:35:41 -0500 (EST) Original-Received: from pastel (69-165-162-104.dsl.teksavvy.com [69.165.162.104]) by mail01.iro.umontreal.ca (Postfix) with ESMTPSA id E661312022C; Mon, 6 Jan 2025 09:35:40 -0500 (EST) In-Reply-To: (Andrea Corallo's message of "Mon, 06 Jan 2025 04:53:45 -0500") X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.bugs:298666 Archived-At: >> There are dependencies between emacs-lisp-benchmarks tests, meaning we >> can't always modify one test without affecting others. The ones I'm >> aware of are: >> >> - elb-bytecomp.el uses elb-smie.el as compilation input (in this >> context, elb-smie.el is a "resource", not source code). If we >> modify elb-smie.el, elb-bytecomp.el results will change and become >> uncomparable. > > Shouldn't we just copy of elb-smie.el to resources? If so, we could just as well use some other file. I used `elb-smie.el` because it was a large enough file and it was already part of `elisp-benchmarks`. >> - elb-scroll.el uses elb-smie.el as a prerequisite. If we modify >> elb-smie.el, elb-scroll.el results will change and become >> uncomparable. Furthermore, elb-scroll.el relies on elb-smie.el >> being compiled in the appropriate fashion by some external >> mechanism. (I think the best solution here is to merge elb-smie.el >> and elb-scroll.el into a single file declaring several tests, using >> the established ERT test framework) > > I'm not sure these are real/serious bugs, different versions of > elisp-benchmarks don't guarantee to give comparable results (like every > benchmark). Agreed. This said, merging `elb-smie.el` and `elb-scroll.el` would be fine. Can't remember why I didn't do that from the start. [ Not sure what ERT has to do with it, BTW. ] >> - bubble.el and bubble-no-cons.el both "provide" the same feature. >> I'm not sure whether this causes any acute problems, but it >> demonstrates that the inconsistent use of (provide) in the >> benchmarks is problematic. If we ever load tests using Frequire, we >> will load the wrong files. If we don't, provide shouldn't be >> necessary. > > AFAIU the provide is not used by elisp-benchmarks, anyway the typo is > now fixed. FWIW, these two files date back to the time where the benchmarks were defined by their file (i.e. one file per benchmark), whereas nowadays the benchmarks are defined by the `elb-*-entry` functions, so we should probably merge those two files into one. >> - xmenu.c is used as a resource; this is fine. But we must make sure >> we never copy the "current" xmenu.c to the benchmarks directory >> without clearly indicating that the benchmark results will become >> uncomparable. > Again, changes to the benchmark pertubating results should be tagged as > a new version, so will be fine to update xmenu.c if necessary as long as > we update elisp-benchmarks version. Also I can't think of any reason why we'd want to "update" that file. At least not until its use of the C language is considered too antiquated/deprecated. Stefan