bug#66782: 29.1; ERT tests report test redefined depending on loading sequence

[Xiyue Deng <manphiz@gmail.com>]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Xiyue Deng <manphiz@gmail.com>
>> Cc: mattiase@acm.org,  66782@debbugs.gnu.org
>> Date: Sat, 28 Oct 2023 04:14:15 -0700
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > OK, but why do that in a single Emacs session?  That is unclean for
>> > much more important reasons: some left-overs from previous tests could
>> > affect the following tests.  IMO, each test should be run in a fresh
>> > Emacs session, i.e. no command line should ever load more than a
>> > single test file.
>> 
>> I think one reason is that doing all test in one session gives you a
>> nice summary of how many tests passed and which ones failed in the end.
>> If you run just one test per session you'll have to inspect the results
>> manually which could be much more time consuming for large test suites,
>> and plus it may be slower.
>
> The usual way of doing this is to generate summary by some shell
> script, which runs all the tests one by one.  Emacs is not the only
> program that can count ;-)
>

That also works of course, though personally I would prefer the good and
reliable `ert-summarize-tests-batch-and-exit'[1] so that I don't have to
reinvent the wheel ;-)

[1] https://www.gnu.org/software/emacs/manual/html_node/ert/Running-Tests-in-Batch-Mode.html

P.S. I also thought of another potential counter-example of preferring
running one test file over running many in one batch: the tests within
the same test file may also leave garbage for the rest of the tests, so
to avoid that we would have to prefer running one test per session (do
we have such a filter?), which may become even slower.

-- 
Xiyue Deng