bug#14781: coding system changes cause make check to hang in icalendar tests

[Handa Kenichi <handa@gnu.org>]

In article <dmr4f96lki.fsf@fencepost.gnu.org>, Glenn Morris
<rgm@gnu.org> writes:

> >  Actually, the culprit was test/automated/decoder-tests.el which
> > changes priorities of coding systems. I didn't know that all tests
> > under test/automated are executed in a single Emacs session, and
> > thus
> > they must be side-effect free.

> I think they should all be side-effect free (as much as possible)
> anyway. Someone might well want to run an individual test in an
> existing
> Emacs session, without that session getting messed up.

I generally agree that keeping a test code site-effect free
is a good thing, and it makes testing a test code easier.
But, sometimes achieving side-effect free is difficult, and
in such a case, I think we don't have to spend much effort
on it.  The main purpose of test is to find bugs.

> > Isn't it better to execute each test file in its own Emacs session?

> I don't know. Running them all in one is the only way to get the final
> number of tests run / passed / failed to be accurate.

Here's a rough idea to do that in test/automated/Makefile.in.

TEST_RESULTS = $(patsubst %.el, %.dat, $(wildcard *.el))

.SUFFIXES: .dat .el 

.el.dat:
        @$(emacs) -f ert-run-tests-batch-and-exit-with-report $< $@

check: $(TEST_RESULTS)
        @$(emacs) -f ert-summarize-test-report $(TEST_RESULTS)

Note that ert-run-tests-batch-and-exit-with-report and
ert-report-test-results are not yet implemented.

> Running them in
> separate Emacs's would allow them to run in parallel, which would be
> faster, but might be confusing?

The faster the better, but I think the speed is not that
important here.  The merit of the above method is that we
can run a single test file just by "make XXXX.dat".

---
Kenichi Handa
handa@gnu.org