Re: master 262d0c6: Mark some tests as expensive

[Michael Albinus <michael.albinus@gmx.de>]

Daniel Martín <mardani29@yahoo.es> writes:

Hi Daniel,

> Eli Zaretskii <eliz@gnu.org> writes:
>>
>> Tramp tests need more time because they involve a remote system.
>> Moreover, the time taken by each Tramp test depends on the speed of
>> the connection, which you cannot know in advance.
>
> If they depend on the connection speed, or if a host is online or
> offline, they also introduce non-determinism that perhaps is not
> apparent now, but can cause problems when the Tramp codebase (and its
> number of tests) scale. For example, if a Tramp test has a 0.1% chance
> of failure because of an unrelated network problem, then if the Tramp
> test suite reaches a point where 10000 tests are run per day, people
> would be investigating 10 test flakes per day. That's a good reason for
> people to lose confidence in the Tramp test suite and ignore failures.

Tramp tests don't need a remote connection by default. They simulate a
connection by a "mock" method, which is in fact a local shell.

Real remote connections can be tested also, but this doesn't happen with
"make check". Read the Commentary section of tramp-tests.el for details.

> I'd say "we should try to run *some* tests as close to real-life
> conditions as possible". By abstracting the environment in some tests,
> one could potentially test an infinite number of environments and error
> conditions, not only what the test happens to run on. We would still
> have a few end-to-end tests that check that the program as a whole works
> fine, of course. The trade-off is that writing that kind of hermetic
> tests takes more time, specially for packages like Tramp.

For the records, I keep an ansible script which runs tramp-tests.el in
~75 different configurations, all of them using real remote hosts. I run
it at least prior releasing a new Tramp version; it takes up to 2 days
(the longest single run of tramp-tests.el takes about 10 hours). This
won't go into Emacs' "make check", of course.

Best regards, Michael.