From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Mattias =?UTF-8?Q?Engdeg=C3=A5rd?= Newsgroups: gmane.emacs.bugs Subject: bug#51941: Detect duplication of ERT tests Date: Thu, 18 Nov 2021 11:16:09 +0100 Message-ID: <21CD2DED-354E-4C46-9794-07172E3BD5F1@acm.org> Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.21\)) Content-Type: multipart/mixed; boundary="Apple-Mail=_BD303CC1-2ED4-488B-B6ED-5CA8DDCC5EBD" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="22990"; mail-complaints-to="usenet@ciao.gmane.io" To: 51941@debbugs.gnu.org Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Thu Nov 18 11:17:13 2021 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mneTg-0005ok-HW for geb-bug-gnu-emacs@m.gmane-mx.org; Thu, 18 Nov 2021 11:17:12 +0100 Original-Received: from localhost ([::1]:60446 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mneTe-0001Wk-Dx for geb-bug-gnu-emacs@m.gmane-mx.org; Thu, 18 Nov 2021 05:17:10 -0500 Original-Received: from eggs.gnu.org ([209.51.188.92]:58922) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mneTW-0001W0-Cj for bug-gnu-emacs@gnu.org; Thu, 18 Nov 2021 05:17:02 -0500 Original-Received: from debbugs.gnu.org ([209.51.188.43]:51555) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mneTW-0002it-4A for bug-gnu-emacs@gnu.org; Thu, 18 Nov 2021 05:17:02 -0500 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1mneTW-0004sH-0Y for bug-gnu-emacs@gnu.org; Thu, 18 Nov 2021 05:17:02 -0500 X-Loop: help-debbugs@gnu.org Resent-From: Mattias =?UTF-8?Q?Engdeg=C3=A5rd?= Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Thu, 18 Nov 2021 10:17:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 51941 X-GNU-PR-Package: emacs X-Debbugs-Original-To: bug-gnu-emacs@gnu.org Original-Received: via spool by submit@debbugs.gnu.org id=B.163723058418641 (code B ref -1); Thu, 18 Nov 2021 10:17:01 +0000 Original-Received: (at submit) by debbugs.gnu.org; 18 Nov 2021 10:16:24 +0000 Original-Received: from localhost ([127.0.0.1]:34859 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mneSt-0004qb-Uk for submit@debbugs.gnu.org; Thu, 18 Nov 2021 05:16:24 -0500 Original-Received: from lists.gnu.org ([209.51.188.17]:33848) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mneSp-0004qP-BO for submit@debbugs.gnu.org; Thu, 18 Nov 2021 05:16:22 -0500 Original-Received: from eggs.gnu.org ([209.51.188.92]:58764) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mneSp-0001SG-67 for bug-gnu-emacs@gnu.org; Thu, 18 Nov 2021 05:16:19 -0500 Original-Received: from mail1441c50.megamailservers.eu ([91.136.14.41]:36634 helo=mail264c50.megamailservers.eu) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mneSm-0002S6-DT for bug-gnu-emacs@gnu.org; Thu, 18 Nov 2021 05:16:18 -0500 X-Authenticated-User: mattiase@bredband.net DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=megamailservers.eu; s=maildub; t=1637230571; bh=kTmQ4RkVMKPTUVCsKGv6Rdqu8VUM1nano3H/WKJdE9I=; h=From:Subject:Date:To:From; b=h/5J4v8f/4warPBnZ21X8aAMkM8kAtN4ETcjbwyx2+BTnkFrniRn2n2QYjNup/zEh GYnT+boG5kQKgWAUHIelA3CSa9u6Dr7MAVnBPyI98L83svlnu+7Bub6UV+nbtqH84s FfkaTpW0ke21DOnXyUxhuZELvg6rZHFTBBVpcdsc= Feedback-ID: mattiase@acm.or Original-Received: from [192.168.0.4] (c188-150-171-71.bredband.tele2.se [188.150.171.71]) (authenticated bits=0) by mail264c50.megamailservers.eu (8.14.9/8.13.1) with ESMTP id 1AIAG9RR030467 for ; Thu, 18 Nov 2021 10:16:11 +0000 X-Mailer: Apple Mail (2.3445.104.21) X-CTCH-RefID: str=0001.0A742F26.619627EB.0052, ss=1, re=0.000, recu=0.000, reip=0.000, cl=1, cld=1, fgs=0 X-CTCH-VOD: Unknown X-CTCH-Spam: Unknown X-CTCH-Score: 0.000 X-CTCH-Flags: 0 X-CTCH-ScoreCust: 0.000 X-CSC: 0 X-CHA: v=2.4 cv=LMcF/La9 c=1 sm=1 tr=0 ts=619627eb a=SF+I6pRkHZhrawxbOkkvaA==:117 a=SF+I6pRkHZhrawxbOkkvaA==:17 a=M51BFTxLslgA:10 a=1CXTRZ9jVQddLJpEC4wA:9 a=CjuIK1q_8ugA:10 a=sz-ed9RZRR2bCHALpOIA:9 a=De_Ol2h6w80A:10 X-Origin-Country: SE Received-SPF: softfail client-ip=91.136.14.41; envelope-from=mattiase@acm.org; helo=mail264c50.megamailservers.eu X-Spam_score_int: -11 X-Spam_score: -1.2 X-Spam_bar: - X-Spam_report: (-1.2 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, SPF_HELO_NONE=0.001, SPF_SOFTFAIL=0.665 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.io gmane.emacs.bugs:220290 Archived-At: --Apple-Mail=_BD303CC1-2ED4-488B-B6ED-5CA8DDCC5EBD Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Duplicated `ert-deftest` forms (ie, two or more definitions with the = same test name) is a serious problem: the older test is silently = ignored, which has the consequence of the test covering less than = intended. A quick experiment in the Emacs tree immediately reveals 51 instances. = There's no telling how many lie waiting in external code. Having explored various options I've come to the conclusion that it = needs to be a hard error, at run-time, when running non-interactively = (see attached diff). Rationale: * We can't just warn at run time because tests tend to spam a lot when = run and users only care about whether the tests passed and won't even = look at the logs otherwise. * We can't just warn at compile time because many tests aren't compiled = at all, and where they are (as in the Emacs tree) nobody cares much = about the warnings, because they are "just tests". * We can't issue a complaint when running interactively because it's = natural to keep redefining tests during development. Since `ert-deftest` forms are often generated as a result of = macro-expansion, passive static textual linting won't do. The effect of this change will be a visible and easily remedied failure = in broken test code. I volunteer to fix the first-order errors in Emacs = (by renaming the clashes); domain specialists may need to help fixing = secondary failures (failures in previously ignored tests). --Apple-Mail=_BD303CC1-2ED4-488B-B6ED-5CA8DDCC5EBD Content-Disposition: attachment; filename=ert-deftest-redefine-error.diff Content-Type: application/octet-stream; x-unix-mode=0644; name="ert-deftest-redefine-error.diff" Content-Transfer-Encoding: 7bit diff --git a/lisp/emacs-lisp/ert.el b/lisp/emacs-lisp/ert.el index 36b4408dc8..92a1315f13 100644 --- a/lisp/emacs-lisp/ert.el +++ b/lisp/emacs-lisp/ert.el @@ -151,6 +151,10 @@ ert-set-test ;; Note that nil is still a valid value for the `name' slot in ;; ert-test objects. It designates an anonymous test. (error "Attempt to define a test named nil")) + (when (and noninteractive (get symbol 'ert--test)) + ;; If a test is redefined when running noninteractively, treat it + ;; as a hard error to make sure it is discovered. + (error "Test `%s' redefined" symbol)) (define-symbol-prop symbol 'ert--test definition) definition) --Apple-Mail=_BD303CC1-2ED4-488B-B6ED-5CA8DDCC5EBD--