Re: master 262d0c6: Mark some tests as expensive

[Eli Zaretskii <eliz@gnu.org>]

> From: Daniel Martín <mardani29@yahoo.es>
> Cc: Michael Albinus <michael.albinus@gmx.de>,  emacs-devel@gnu.org
> Date: Sat, 12 Sep 2020 12:25:17 +0200
> 
> Stefan Kangas <stefan@marxist.se> writes:
> >
> > The idea is to avoid that it will take a very long time to run the unit
> > tests as the test suite grows.  Arguably a unit test should never take
> > longer than a second, and even that is in the slow end.  If we have
> > 10.000 unit tests taking a second each, just do the math of how long it
> > will take to run (even with parallelization).  We should not postpone
> > working on this until we are in that situation, IMHO, because by then it
> > will be a major pain.
> 
> I agree. We should differentiate between unit tests (they run in under a
> second) and end-to-end tests, which are slower but still valuable
> because they cover whole scenarios that unit tests can't cover.

Tramp tests need more time because they involve a remote system.
Moreover, the time taken by each Tramp test depends on the speed of
the connection, which you cannot know in advance.

So I conclude that the "under one second" criterion is not smart
enough to be applicable to Tramp tests, and maybe to some others as
well.

I also don't understand the more general issue with how long the test
suite runs.  While it runs, you can do something useful on a modern
system (or just go for a coffee), so why does it matter?  I've seen
test suites that take a very long time to run (e.g., look at Texinfo
or at Guile), and I never felt I was wasting my time by running those
comprehensive test suites.

I think we are exaggerating the importance of the time it takes to run
the tests.

> > I think the solution for important tests is to refactor the code to make
> > them run faster, for example by avoiding I/O or to make timers trigger
> > immediately.
> 
> Yes, I think this is a good approach to follow in general.

I don't, not in general.  Artificially making such changes will run a
risk of missing real problems, because the test runs in an environment
different from a real one.  This approach can be a good idea in some
cases, but in general we should try to run each test as close to
real-life conditions as possible.