From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Xiyue Deng Newsgroups: gmane.emacs.bugs Subject: bug#66782: 29.1; ERT tests report test redefined depending on loading sequence Date: Sat, 28 Oct 2023 04:50:37 -0700 Message-ID: <87cywzge42.fsf@debian-hx90.lan> References: <87msw393z8.fsf@debian-hx90.lan> <83y1fncl4k.fsf@gnu.org> <87o7gjgk65.fsf_-_@debian-hx90.lan> <83pm0zcc25.fsf@gnu.org> <87jzr7gfso.fsf@debian-hx90.lan> <83jzr7c74y.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="33603"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Gnus/5.13 (Gnus v5.13) Cc: mattiase@acm.org, 66782@debbugs.gnu.org To: Eli Zaretskii Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Sat Oct 28 13:51:39 2023 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1qwhqt-0008X1-2t for geb-bug-gnu-emacs@m.gmane-mx.org; Sat, 28 Oct 2023 13:51:39 +0200 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qwhql-00006V-HF; Sat, 28 Oct 2023 07:51:31 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qwhqk-00006A-Fi for bug-gnu-emacs@gnu.org; Sat, 28 Oct 2023 07:51:30 -0400 Original-Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qwhqk-0000l0-7e for bug-gnu-emacs@gnu.org; Sat, 28 Oct 2023 07:51:30 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1qwhrG-00060D-4t for bug-gnu-emacs@gnu.org; Sat, 28 Oct 2023 07:52:02 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Xiyue Deng Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Sat, 28 Oct 2023 11:52:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 66782 X-GNU-PR-Package: emacs Original-Received: via spool by 66782-submit@debbugs.gnu.org id=B66782.169849388023008 (code B ref 66782); Sat, 28 Oct 2023 11:52:02 +0000 Original-Received: (at 66782) by debbugs.gnu.org; 28 Oct 2023 11:51:20 +0000 Original-Received: from localhost ([127.0.0.1]:37886 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qwhqa-0005z2-Cv for submit@debbugs.gnu.org; Sat, 28 Oct 2023 07:51:20 -0400 Original-Received: from mail-pl1-x62e.google.com ([2607:f8b0:4864:20::62e]:58533) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qwhqW-0005yj-C4 for 66782@debbugs.gnu.org; Sat, 28 Oct 2023 07:51:18 -0400 Original-Received: by mail-pl1-x62e.google.com with SMTP id d9443c01a7336-1c8a1541232so26404255ad.0 for <66782@debbugs.gnu.org>; Sat, 28 Oct 2023 04:50:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1698493838; x=1699098638; darn=debbugs.gnu.org; h=mime-version:user-agent:message-id:date:references:in-reply-to :subject:cc:to:from:from:to:cc:subject:date:message-id:reply-to; bh=FtCNWrorbV0nlXRLFX2DOr3um9pjVvOqLM4rQXnhe+E=; b=aWAhysm8s/ccSPzkTOUW9tjGEMoQzHy1CDhcddfnypvhBqJsHhA61Tg1/txcfUmtiH Hlm2Ew+WmOasgOFUVIrghqqCyK7PiePfrum+VSBKWKi215hAbcwzGruV/6AD5WznD/CT Mo7pP46lxWuYvW0MqhH6iXqio+yEb//h3v0VPtiRC4QilrWgK3Y+E8TSZ3qUDFfpujr7 3ElKieUhbtXIxbRKTuvgznz3lkL6YpcEax4Dm0T5xBsBImHoTl6E9kimQ3x4fLfMr7bC q7Uq7uoLVSOBpo9thMwqgYcceq1ISl/vtwEKnDphsD0LopWYnw0hqaVvFUtkUL0yH8Fm H56w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698493838; x=1699098638; h=mime-version:user-agent:message-id:date:references:in-reply-to :subject:cc:to:from:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=FtCNWrorbV0nlXRLFX2DOr3um9pjVvOqLM4rQXnhe+E=; b=POPyLW9owOE9TgvepvROAoq4pmmizjG4N3FxkG5A/l3wFJ+zHY4DF+EH6nnHXrXHje K2iL22pyaHPRxOpkLiA5nt+b3dDCrOdfDGahZ8Oam0A38FWlbtJgvUhE2lFSlYx7QJRb e3vRc7zqI5BpN6JtRhEWBkUB+xN8GncVpsuIF8ygc/nQobp/P0y+97MBwxMePjAgGtvL ZvtpjD3p00JuFdl8cOFKaB+fTszkJ8ZsUHPi2Vm4xCetIQz7nP77xOAPakpfMk89tFU0 ggc8HBbGJplnHIQfOX5StBotEJwhAxmrrYZFjhEL4VPZdl7ayx8rt4Pps1dRJczSe9m1 RB5A== X-Gm-Message-State: AOJu0Yz7PVZYtmpmTU2wcTLtKufZ0woE7VRMbNVP7LGkozIHMkiYQRn+ 6TxojWbe/1agdyjfkP66r1odIbAZvdw0WA== X-Google-Smtp-Source: AGHT+IG+O2YkOJ0+HNUv4mlRWcxugfkE/caoDztsVTD/nuPVf4z4iOgJh4DMCA7mxJm/Muxfutc99A== X-Received: by 2002:a17:902:e881:b0:1ca:e6c6:92f6 with SMTP id w1-20020a170902e88100b001cae6c692f6mr6794487plg.9.1698493838387; Sat, 28 Oct 2023 04:50:38 -0700 (PDT) Original-Received: from debian-hx90 (2603-8000-a400-0cdc-2578-79b2-8435-c317.res6.spectrum.com. [2603:8000:a400:cdc:2578:79b2:8435:c317]) by smtp.gmail.com with ESMTPSA id v7-20020a1709029a0700b001b898595be7sm3088419plp.291.2023.10.28.04.50.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Oct 2023 04:50:38 -0700 (PDT) In-Reply-To: <83jzr7c74y.fsf@gnu.org> (Eli Zaretskii's message of "Sat, 28 Oct 2023 14:34:53 +0300") X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.bugs:273431 Archived-At: Eli Zaretskii writes: >> From: Xiyue Deng >> Cc: mattiase@acm.org, 66782@debbugs.gnu.org >> Date: Sat, 28 Oct 2023 04:14:15 -0700 >> >> Eli Zaretskii writes: >> >> > OK, but why do that in a single Emacs session? That is unclean for >> > much more important reasons: some left-overs from previous tests could >> > affect the following tests. IMO, each test should be run in a fresh >> > Emacs session, i.e. no command line should ever load more than a >> > single test file. >> >> I think one reason is that doing all test in one session gives you a >> nice summary of how many tests passed and which ones failed in the end. >> If you run just one test per session you'll have to inspect the results >> manually which could be much more time consuming for large test suites, >> and plus it may be slower. > > The usual way of doing this is to generate summary by some shell > script, which runs all the tests one by one. Emacs is not the only > program that can count ;-) > That also works of course, though personally I would prefer the good and reliable `ert-summarize-tests-batch-and-exit'[1] so that I don't have to reinvent the wheel ;-) [1] https://www.gnu.org/software/emacs/manual/html_node/ert/Running-Tests-in-Batch-Mode.html P.S. I also thought of another potential counter-example of preferring running one test file over running many in one batch: the tests within the same test file may also leave garbage for the rest of the tests, so to avoid that we would have to prefer running one test per session (do we have such a filter?), which may become even slower. -- Xiyue Deng