Re: Run (some) tests more automagically?

[Pip Cet <pipcet@gmail.com>]

On Sun, Feb 21, 2021 at 1:27 PM Lars Ingebrigtsen <larsi@gnus.org> wrote:
> I wonder whether anybody's considered making "make" run some tests.
>
> It's a recurring issue that we make a change, and then we forget to run
> the test suite.  I mean, it's not a major problem, because it'll
> eventually get run by somebody, but I'd feel more confident in my own
> changes if I'd remember to run more tests more often.
>
> The reason we don't is because we don't want to wait for some minutes
> while running the entire suite.

That's certainly part of the reason for me. I find it important to
remember that (to me) make is a tool to save time, and that requires
compromise between remaking everything that could possibly have
changed (which would require recompiling all elcs (and elns) for every
change in the C sources, and also would require us to accurately model
all potential ELisp-to-ELisp dependencies, some of which are circular)
and just making things once. Doing the former is possible for projects
that have flat dependency trees, not for deep (and possibly cyclic)
dependency graphs.

> So here's today's not-thought-out-at-all idea: Since tests for
> lisp/foo.el live in test/lisp/foo-tests.el, could we add some Makefile
> magic to automatically run foo-tests.el if lisp/foo.el has been changed?

Yes, let's! Except we need to agree on what "has been changed" means.
My initial idea would be to create a time stamp file the first time
make is run in an emacs directory and consider only those source files
which are newer than the time stamp, only if they're recompiled. But
what should we do when we run "git pull"? Should the time stamp be
updated for all tests after every make, for successful tests only, or
only if all tests that were run were successful? Should the check run
after the rest of make has (possibly after a "build successful,
continuing automatically to run tests" message so the impatient can
interrupt it at that point), at the same time as the rest of make
(increasing time-to-test, IMHO) or asynchronously after make has
finished?

It's easy to get this wrong, and I half-expect I'll have to add
"nocheck" to all my Emacs makefiles if the feature lands.

> Running just those tests shouldn't take too much time, and would
> probably cover 93% of the relevant code changes for foo.el.

Speaking of code changes, one thing I'm worried about is that one
developer on, say, Windows does something, forgets to adjust the tests
so there's a spurious failure on GNU/Linux, and then the next
GNU/Linux developer to touch the file will see a test failure that has
nothing to do with their changes. I don't have a good solution for
that one.

> One disadvantage to doing this (if we find a way) is that "make
> foo-tests" is often very noisy, so we'd need a way to run tests silently
> and just report if anything actually fails unexpectedly, I guess.

I think that's an advantage: tests should be silent by default, rather
than producing all kinds of irrelevant "by the way, your headlight's
working" noise. If we can help enforce that by running them in make,
that would be great.

Pip