why does make check run all tests?

why does make check run all tests?

[Phillip Lord]

I've been looking through the Makefile for the test directory.

Currently, the check target says this.


## Re-run all the tests every time.
check:
	-@for f in *.log; do test ! -f $$f || mv $$f $$f~; done
	@${MAKE} check-maybe

I realised last night, that my recent changes have actually broken this
(check is now incremental).

This to me seems to be a good thing (although it was accidental). Why
should check run all tests everytime? Especially now that I have added
dependencies between tests and source (so, if you change abbrevs.el,
then abbrevs-test.el will get run).

Should make check just not build incrementally? If you want to run
everything unconditionally, then why not use

make clean check


Or am I missing something obvious? In which case, I will need to restore
the old behaviour for make check and leave make check-maybe as is.

Phil

Re: why does make check run all tests?

[Dmitry Gutov]

On 12/02/2015 02:12 PM, Phillip Lord wrote:

> Should make check just not build incrementally? If you want to run
> everything unconditionally, then why not use
>
> make clean check

That would fully recompile Emacs, rebuild the info files, and only then 
run the tests. That's too slow.

> Or am I missing something obvious? In which case, I will need to restore
> the old behaviour for make check and leave make check-maybe as is.

Yes, I think we should have two build targets for tests.

Re: why does make check run all tests?

[Michael Albinus]

phillip.lord@russet.org.uk (Phillip Lord) writes:

> Should make check just not build incrementally? If you want to run
> everything unconditionally, then why not use
>
> make clean check
>
> Or am I missing something obvious? In which case, I will need to restore
> the old behaviour for make check and leave make check-maybe as is.

"make check" is not intented to check, whether a *changed* file still
runs successfully. "make check" is intended to run a full regression
test over Emacs. Pls keep this functionality.

> Phil

Best regards, Michael.

Re: why does make check run all tests?

[Phillip Lord]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 12/02/2015 02:12 PM, Phillip Lord wrote:
>
>> Should make check just not build incrementally? If you want to run
>> everything unconditionally, then why not use
>>
>> make clean check
>
> That would fully recompile Emacs, rebuild the info files, and only then run
> the tests. That's too slow.

In the root directory yes. In the test directory, it doesn't.


>> Or am I missing something obvious? In which case, I will need to restore
>> the old behaviour for make check and leave make check-maybe as is.
>
> Yes, I think we should have two build targets for tests.

I guess that there would no problem adding "check-maybe" to the root
Makefile. At the moment, this is only runnable from tests.

Phil

Re: why does make check run all tests?

[Mathieu Lirzin]

Hi,

phillip.lord@russet.org.uk (Phillip Lord) writes:

> ## Re-run all the tests every time.
> check:
> 	-@for f in *.log; do test ! -f $$f || mv $$f $$f~; done
> 	@${MAKE} check-maybe
>
> I realised last night, that my recent changes have actually broken this
> (check is now incremental).
>
> This to me seems to be a good thing (although it was accidental). Why
> should check run all tests everytime? Especially now that I have added
> dependencies between tests and source (so, if you change abbrevs.el,
> then abbrevs-test.el will get run).

According to GCS, ‘make check’ is supposed to run the test suite.  When
someone wants to run a specific test he should be able (in Automake's
conventions) to override the TESTS variable with a make flag like this:

  make check TESTS="tests/.."

Since Emacs does not use Automake, it would be nice to adapt your changes
to enforce this behavior.

Thanks,

--
Mathieu Lirzin

Re: why does make check run all tests?

[Stephen Leake]

Michael Albinus <michael.albinus@gmx.de> writes:

> phillip.lord@russet.org.uk (Phillip Lord) writes:
>
>> Should make check just not build incrementally? If you want to run
>> everything unconditionally, then why not use
>>
>> make clean check
>>
>> Or am I missing something obvious? In which case, I will need to restore
>> the old behaviour for make check and leave make check-maybe as is.
>
> "make check" is not intented to check, whether a *changed* file still
> runs successfully. "make check" is intended to run a full regression
> test over Emacs. Pls keep this functionality.

Especially since the dependencies are not complete; elisp-mode-tests.el
only depends on elisp-mode.el, not on the transitive require closure.

-- 
-- Stephe

Re: why does make check run all tests?

[Phillip Lord]

Mathieu Lirzin <mthl@gnu.org> writes:

> According to GCS, ‘make check’ is supposed to run the test suite.  When
> someone wants to run a specific test he should be able (in Automake's
> conventions) to override the TESTS variable with a make flag like this:
>
>   make check TESTS="tests/.."
>
> Since Emacs does not use Automake, it would be nice to adapt your changes
> to enforce this behavior.

I fear that this would be rather clunky, since there are already about
90 test files. The test makefile already supports individual builds,
though, and so I haven't changed this (not deliberately, anyway, the
implementation has).

So

make abbrev-tests

or

make lisp/abbrev-tests

runs the abbrev-tests and dumps to standard out, while

make lisp/abbrev-tests.log

generates the log file.

Phil

Re: why does make check run all tests?

[Phillip Lord]

Stephen Leake <stephen_leake@stephe-leake.org> writes:

> Michael Albinus <michael.albinus@gmx.de> writes:
>
>> phillip.lord@russet.org.uk (Phillip Lord) writes:
>>
>>> Should make check just not build incrementally? If you want to run
>>> everything unconditionally, then why not use
>>>
>>> make clean check
>>>
>>> Or am I missing something obvious? In which case, I will need to restore
>>> the old behaviour for make check and leave make check-maybe as is.
>>
>> "make check" is not intented to check, whether a *changed* file still
>> runs successfully. "make check" is intended to run a full regression
>> test over Emacs. Pls keep this functionality.

I've restored this functionality now (which I accidentally removed).

make check-maybe

is incremental as before, due also depends on the files being tested.


> Especially since the dependencies are not complete; elisp-mode-tests.el
> only depends on elisp-mode.el, not on the transitive require closure.

Strictly, yes, this is true, but then I think this is true of other
parts of the Emacs build -- elisp-mode.elc depends only on
elisp-mode.el (in the makefile), although, IIUC, elisp-mode.elc also
depends on cl-generic and it's macros. In practice, the dependencies I
have added make a reasonable guess, I think.

Still, your point is well made. I think both incremental and
unconditional tests need to be there.

Phil