unofficial mirror of emacs-devel@gnu.org 
 help / color / mirror / code / Atom feed
* Run (some) tests more automagically?
@ 2021-02-21 13:25 Lars Ingebrigtsen
  2021-02-21 13:48 ` dick.r.chiang
                   ` (3 more replies)
  0 siblings, 4 replies; 12+ messages in thread
From: Lars Ingebrigtsen @ 2021-02-21 13:25 UTC (permalink / raw)
  To: emacs-devel

I wonder whether anybody's considered making "make" run some tests.

It's a recurring issue that we make a change, and then we forget to run
the test suite.  I mean, it's not a major problem, because it'll
eventually get run by somebody, but I'd feel more confident in my own
changes if I'd remember to run more tests more often.

The reason we don't is because we don't want to wait for some minutes
while running the entire suite.

So here's today's not-thought-out-at-all idea: Since tests for
lisp/foo.el live in test/lisp/foo-tests.el, could we add some Makefile
magic to automatically run foo-tests.el if lisp/foo.el has been changed?
Running just those tests shouldn't take too much time, and would
probably cover 93% of the relevant code changes for foo.el.

One disadvantage to doing this (if we find a way) is that "make
foo-tests" is often very noisy, so we'd need a way to run tests silently
and just report if anything actually fails unexpectedly, I guess.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no




^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Run (some) tests more automagically?
  2021-02-21 13:25 Run (some) tests more automagically? Lars Ingebrigtsen
@ 2021-02-21 13:48 ` dick.r.chiang
  2021-02-21 14:28 ` Pip Cet
                   ` (2 subsequent siblings)
  3 siblings, 0 replies; 12+ messages in thread
From: dick.r.chiang @ 2021-02-21 13:48 UTC (permalink / raw)
  Cc: emacs-devel

Triggering tests on baseline `make` is antithetical to the fifty-years of make
"common-law."  That contributors continue to submit patches without running
`make check` or `make check-maybe` is unavoidable, but that maintainers merge
those patches without running same is a little strange.

Of course, this all goes away with so-called "CICD" pipelines, the
FSF-sanctioned equivalent being something called "emba," the efficacy of which
remains dubious.



^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Run (some) tests more automagically?
  2021-02-21 13:25 Run (some) tests more automagically? Lars Ingebrigtsen
  2021-02-21 13:48 ` dick.r.chiang
@ 2021-02-21 14:28 ` Pip Cet
  2021-02-21 14:45   ` Philipp
  2021-02-21 16:05   ` Lars Ingebrigtsen
  2021-02-21 15:37 ` Eli Zaretskii
  2021-02-24 10:42 ` Phillip Lord
  3 siblings, 2 replies; 12+ messages in thread
From: Pip Cet @ 2021-02-21 14:28 UTC (permalink / raw)
  To: Lars Ingebrigtsen; +Cc: emacs-devel

On Sun, Feb 21, 2021 at 1:27 PM Lars Ingebrigtsen <larsi@gnus.org> wrote:
> I wonder whether anybody's considered making "make" run some tests.
>
> It's a recurring issue that we make a change, and then we forget to run
> the test suite.  I mean, it's not a major problem, because it'll
> eventually get run by somebody, but I'd feel more confident in my own
> changes if I'd remember to run more tests more often.
>
> The reason we don't is because we don't want to wait for some minutes
> while running the entire suite.

That's certainly part of the reason for me. I find it important to
remember that (to me) make is a tool to save time, and that requires
compromise between remaking everything that could possibly have
changed (which would require recompiling all elcs (and elns) for every
change in the C sources, and also would require us to accurately model
all potential ELisp-to-ELisp dependencies, some of which are circular)
and just making things once. Doing the former is possible for projects
that have flat dependency trees, not for deep (and possibly cyclic)
dependency graphs.

> So here's today's not-thought-out-at-all idea: Since tests for
> lisp/foo.el live in test/lisp/foo-tests.el, could we add some Makefile
> magic to automatically run foo-tests.el if lisp/foo.el has been changed?

Yes, let's! Except we need to agree on what "has been changed" means.
My initial idea would be to create a time stamp file the first time
make is run in an emacs directory and consider only those source files
which are newer than the time stamp, only if they're recompiled. But
what should we do when we run "git pull"? Should the time stamp be
updated for all tests after every make, for successful tests only, or
only if all tests that were run were successful? Should the check run
after the rest of make has (possibly after a "build successful,
continuing automatically to run tests" message so the impatient can
interrupt it at that point), at the same time as the rest of make
(increasing time-to-test, IMHO) or asynchronously after make has
finished?

It's easy to get this wrong, and I half-expect I'll have to add
"nocheck" to all my Emacs makefiles if the feature lands.

> Running just those tests shouldn't take too much time, and would
> probably cover 93% of the relevant code changes for foo.el.

Speaking of code changes, one thing I'm worried about is that one
developer on, say, Windows does something, forgets to adjust the tests
so there's a spurious failure on GNU/Linux, and then the next
GNU/Linux developer to touch the file will see a test failure that has
nothing to do with their changes. I don't have a good solution for
that one.

> One disadvantage to doing this (if we find a way) is that "make
> foo-tests" is often very noisy, so we'd need a way to run tests silently
> and just report if anything actually fails unexpectedly, I guess.

I think that's an advantage: tests should be silent by default, rather
than producing all kinds of irrelevant "by the way, your headlight's
working" noise. If we can help enforce that by running them in make,
that would be great.

Pip



^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Run (some) tests more automagically?
  2021-02-21 14:28 ` Pip Cet
@ 2021-02-21 14:45   ` Philipp
  2021-02-21 16:05   ` Lars Ingebrigtsen
  1 sibling, 0 replies; 12+ messages in thread
From: Philipp @ 2021-02-21 14:45 UTC (permalink / raw)
  To: Pip Cet; +Cc: Lars Ingebrigtsen, emacs-devel



> Am 21.02.2021 um 15:28 schrieb Pip Cet <pipcet@gmail.com>:
> 
> 
> Speaking of code changes, one thing I'm worried about is that one
> developer on, say, Windows does something, forgets to adjust the tests
> so there's a spurious failure on GNU/Linux, and then the next
> GNU/Linux developer to touch the file will see a test failure that has
> nothing to do with their changes. I don't have a good solution for
> that one.

The typical solution is to run CI pipelines for all supported platforms and not allow pushing commits that fail any of the pipelines.  In the context of Emacs, there are still some downsides to this: it would require running nonfree OSes, and right now the CI pipelines are still a bit slow.


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Run (some) tests more automagically?
  2021-02-21 13:25 Run (some) tests more automagically? Lars Ingebrigtsen
  2021-02-21 13:48 ` dick.r.chiang
  2021-02-21 14:28 ` Pip Cet
@ 2021-02-21 15:37 ` Eli Zaretskii
  2021-02-21 15:57   ` Lars Ingebrigtsen
  2021-02-24 10:42 ` Phillip Lord
  3 siblings, 1 reply; 12+ messages in thread
From: Eli Zaretskii @ 2021-02-21 15:37 UTC (permalink / raw)
  To: Lars Ingebrigtsen; +Cc: emacs-devel

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Sun, 21 Feb 2021 14:25:29 +0100
> 
> So here's today's not-thought-out-at-all idea: Since tests for
> lisp/foo.el live in test/lisp/foo-tests.el, could we add some Makefile
> magic to automatically run foo-tests.el if lisp/foo.el has been changed?

We need to run every test that uses any function in foo.el, not just
foo-tests.el.

This should be "maintainer-only" features, so that users who just
build their own Emacs won't be forced to run the test suite.



^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Run (some) tests more automagically?
  2021-02-21 15:37 ` Eli Zaretskii
@ 2021-02-21 15:57   ` Lars Ingebrigtsen
  0 siblings, 0 replies; 12+ messages in thread
From: Lars Ingebrigtsen @ 2021-02-21 15:57 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: emacs-devel

Eli Zaretskii <eliz@gnu.org> writes:

>> So here's today's not-thought-out-at-all idea: Since tests for
>> lisp/foo.el live in test/lisp/foo-tests.el, could we add some Makefile
>> magic to automatically run foo-tests.el if lisp/foo.el has been changed?
>
> We need to run every test that uses any function in foo.el, not just
> foo-tests.el.

Of course.  This would just be a 93% solution, like I think I said?  But
it might be worth it if it's so efficient that it becomes a part of our
normal work flow.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no



^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Run (some) tests more automagically?
  2021-02-21 14:28 ` Pip Cet
  2021-02-21 14:45   ` Philipp
@ 2021-02-21 16:05   ` Lars Ingebrigtsen
  1 sibling, 0 replies; 12+ messages in thread
From: Lars Ingebrigtsen @ 2021-02-21 16:05 UTC (permalink / raw)
  To: Pip Cet; +Cc: emacs-devel

Pip Cet <pipcet@gmail.com> writes:

> Yes, let's! Except we need to agree on what "has been changed" means.
> My initial idea would be to create a time stamp file the first time
> make is run in an emacs directory and consider only those source files
> which are newer than the time stamp, only if they're recompiled.

Hm, yes, a time stamp would probably be the way to go...

> But what should we do when we run "git pull"? Should the time stamp be
> updated for all tests after every make, for successful tests only, or
> only if all tests that were run were successful? Should the check run
> after the rest of make has (possibly after a "build successful,
> continuing automatically to run tests" message so the impatient can
> interrupt it at that point), at the same time as the rest of make
> (increasing time-to-test, IMHO) or asynchronously after make has
> finished?

These are all good questions.  :-)

Off the top of my head without thinking about it more than ten seconds:
Since this is a low-cost low-effort thing, I think the timestamp should
just be updated when it's run, and just a single timestamp, no matter
how much fails.

So if you do a git pull, and that updates foo.el and bar.el, then "make"
would (as now) compile foo.el and bar.el, and Emacs would then notice
that foo.elc and bar.elc are newer than the previous timestamp, and then
run the foo-tests.el and bar-tests.elc files, and then update the
timestamp.

I think that might be unnoticeable enough that developers wouldn't be
disabling this when working?

Of course, if we rely on this too much, then there's the temptation to
never actually run a full "make check", but we have the CI for that, so
perhaps that's not too big a worry.

> Speaking of code changes, one thing I'm worried about is that one
> developer on, say, Windows does something, forgets to adjust the tests
> so there's a spurious failure on GNU/Linux, and then the next
> GNU/Linux developer to touch the file will see a test failure that has
> nothing to do with their changes. I don't have a good solution for
> that one.

Me neither.  But that's also the case today.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no



^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Run (some) tests more automagically?
  2021-02-21 13:25 Run (some) tests more automagically? Lars Ingebrigtsen
                   ` (2 preceding siblings ...)
  2021-02-21 15:37 ` Eli Zaretskii
@ 2021-02-24 10:42 ` Phillip Lord
  2021-02-24 14:34   ` Lars Ingebrigtsen
  3 siblings, 1 reply; 12+ messages in thread
From: Phillip Lord @ 2021-02-24 10:42 UTC (permalink / raw)
  To: Lars Ingebrigtsen; +Cc: emacs-devel

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I wonder whether anybody's considered making "make" run some tests.
>
> It's a recurring issue that we make a change, and then we forget to run
> the test suite.  I mean, it's not a major problem, because it'll
> eventually get run by somebody, but I'd feel more confident in my own
> changes if I'd remember to run more tests more often.
>
> The reason we don't is because we don't want to wait for some minutes
> while running the entire suite.
>
> So here's today's not-thought-out-at-all idea: Since tests for
> lisp/foo.el live in test/lisp/foo-tests.el, could we add some Makefile
> magic to automatically run foo-tests.el if lisp/foo.el has been changed?
> Running just those tests shouldn't take too much time, and would
> probably cover 93% of the relevant code changes for foo.el.


It is supposed to do this. That was the point of moving all the tests to
a standard naming scheme in the first place.

make check-maybe

It uses out of date log files. I don't remember whether it will skip
previously failed tests the second time around which might be a problem.

Phil




^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Run (some) tests more automagically?
  2021-02-24 10:42 ` Phillip Lord
@ 2021-02-24 14:34   ` Lars Ingebrigtsen
  2021-02-26 11:44     ` Phillip Lord
  0 siblings, 1 reply; 12+ messages in thread
From: Lars Ingebrigtsen @ 2021-02-24 14:34 UTC (permalink / raw)
  To: Phillip Lord; +Cc: emacs-devel

Phillip Lord <phillip.lord@russet.org.uk> writes:

> It is supposed to do this. That was the point of moving all the tests to
> a standard naming scheme in the first place.
>
> make check-maybe

Oh!  I had no idea; thanks.

It is very noisy; though:

----
[hundreds of lines like this]:
make[3]: 'src/undo-tests.log' is up to date.
make[3]: 'src/xdisp-tests.log' is up to date.
make[3]: 'src/xfaces-tests.log' is up to date.
make[3]: 'src/xml-tests.log' is up to date.
make[3]: Leaving directory '/home/larsi/src/emacs/trunk/test'

SUMMARY OF TEST RESULTS
-----------------------
Files examined: 348
Ran 4738 tests, 4659 results as expected, 0 unexpected, 79 skipped
make[2]: Leaving directory '/home/larsi/src/emacs/trunk/test'
make[1]: Leaving directory '/home/larsi/src/emacs/trunk/test'
----

Perhaps sticking it in the admin/emake script and doing some filtering
would be nice...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no



^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Run (some) tests more automagically?
  2021-02-24 14:34   ` Lars Ingebrigtsen
@ 2021-02-26 11:44     ` Phillip Lord
  2021-02-26 11:50       ` Lars Ingebrigtsen
  0 siblings, 1 reply; 12+ messages in thread
From: Phillip Lord @ 2021-02-26 11:44 UTC (permalink / raw)
  To: Lars Ingebrigtsen; +Cc: emacs-devel

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Phillip Lord <phillip.lord@russet.org.uk> writes:
>
>> It is supposed to do this. That was the point of moving all the tests to
>> a standard naming scheme in the first place.
>>
>> make check-maybe
>
> Oh!  I had no idea; thanks.

It also seems not to be working. It picks up a missing log file and just
regenerates that, which is nice. But it doesn't seem to be picking up an
out-of-date .el dependency. I thought that it did. I will have a poke
when I can find time.



> It is very noisy; though:
>
> ----
> [hundreds of lines like this]:
> make[3]: 'src/undo-tests.log' is up to date.
> make[3]: 'src/xdisp-tests.log' is up to date.
> make[3]: 'src/xfaces-tests.log' is up to date.
> make[3]: 'src/xml-tests.log' is up to date.
> make[3]: Leaving directory '/home/larsi/src/emacs/trunk/test'
>
> SUMMARY OF TEST RESULTS
> -----------------------
> Files examined: 348
> Ran 4738 tests, 4659 results as expected, 0 unexpected, 79 skipped
> make[2]: Leaving directory '/home/larsi/src/emacs/trunk/test'
> make[1]: Leaving directory '/home/larsi/src/emacs/trunk/test'
> ----
>
> Perhaps sticking it in the admin/emake script and doing some filtering
> would be nice...

Having a target that was designed for a .git hook would seem sensible to
me. Something that we would normally expect to run in a second or two.

Phil



^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Run (some) tests more automagically?
  2021-02-26 11:44     ` Phillip Lord
@ 2021-02-26 11:50       ` Lars Ingebrigtsen
  2021-03-01 17:50         ` Phillip Lord
  0 siblings, 1 reply; 12+ messages in thread
From: Lars Ingebrigtsen @ 2021-02-26 11:50 UTC (permalink / raw)
  To: Phillip Lord; +Cc: emacs-devel

Phillip Lord <phillip.lord@russet.org.uk> writes:

> It also seems not to be working. It picks up a missing log file and just
> regenerates that, which is nice. But it doesn't seem to be picking up an
> out-of-date .el dependency. I thought that it did. I will have a poke
> when I can find time.

It seems to be working for me:

touch lisp/files.el
make check-maybe
[...]
make[3]: 'src/xfaces-tests.log' is up to date.
make[3]: 'src/xml-tests.log' is up to date.
  GEN      lisp/files-tests.log
make[3]: Leaving directory '/home/larsi/src/emacs/trunk/test'

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no



^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Run (some) tests more automagically?
  2021-02-26 11:50       ` Lars Ingebrigtsen
@ 2021-03-01 17:50         ` Phillip Lord
  0 siblings, 0 replies; 12+ messages in thread
From: Phillip Lord @ 2021-03-01 17:50 UTC (permalink / raw)
  To: Lars Ingebrigtsen; +Cc: emacs-devel

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Phillip Lord <phillip.lord@russet.org.uk> writes:
>
>> It also seems not to be working. It picks up a missing log file and just
>> regenerates that, which is nice. But it doesn't seem to be picking up an
>> out-of-date .el dependency. I thought that it did. I will have a poke
>> when I can find time.
>
> It seems to be working for me:
>
> touch lisp/files.el
> make check-maybe
> [...]
> make[3]: 'src/xfaces-tests.log' is up to date.
> make[3]: 'src/xml-tests.log' is up to date.
>   GEN      lisp/files-tests.log
> make[3]: Leaving directory '/home/larsi/src/emacs/trunk/test'


Hmmm. Yes. But

make -W lisp/files.el check-maybe

does not work. Probably something to do with the recursive call.

Phil



^ permalink raw reply	[flat|nested] 12+ messages in thread

end of thread, other threads:[~2021-03-01 17:50 UTC | newest]

Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-02-21 13:25 Run (some) tests more automagically? Lars Ingebrigtsen
2021-02-21 13:48 ` dick.r.chiang
2021-02-21 14:28 ` Pip Cet
2021-02-21 14:45   ` Philipp
2021-02-21 16:05   ` Lars Ingebrigtsen
2021-02-21 15:37 ` Eli Zaretskii
2021-02-21 15:57   ` Lars Ingebrigtsen
2021-02-24 10:42 ` Phillip Lord
2021-02-24 14:34   ` Lars Ingebrigtsen
2021-02-26 11:44     ` Phillip Lord
2021-02-26 11:50       ` Lars Ingebrigtsen
2021-03-01 17:50         ` Phillip Lord

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/emacs.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).