Re: running each test file independently in test/automated

[Barry OReilly <gundaetiapo@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 3922 bytes --]

>> Hi, I've found I want to run only one or a few of the tests at a
>> time rather than the whole suite.

> http://lists.gnu.org/archive/html/emacs-devel/2013-08/msg00339.html
>
> is supposed to allow this. I don't know if it is waiting on
> something.

Yes indeed, it provides a way to run individual tests:

  $ rm cl-lib.log ; make cl-lib.log
  Running tests in cl-lib.el... passed all 8 tests

Some comments on the patch follow.

> TEST_LOGS = $(patsubst %.el, %.log, $(wildcard $(test)/*.el))

Other recipes in the same Makefile determine the set of .el files a
different way: they include .el files in subdirectories except data/.
There aren't actually such .el files, but the moment someone adds one
the make code is inconsistent.

>     @test -d `dirname "$@"` || mkdir `dirname "$@"`

Why not: mkdir -p `dirname "$@"`

> parallel: $(TEST_LOGS)
>     @cd $(test); $(emacs) -f ert-summary-report $(TEST_LOGS)

Instead of creating the new "parallel" target, could we just have the
"check" target run the tests individually?

One argument against might be that a -j1 build would be longer. Here
are some benchmarks (2 CPU cores). 3 different invocations, 2 samples
each:

  $ rm *.log ; time make parallel
  real    1m21.869s
  real    1m21.918s
  $ rm *.log ; time make -j4 parallel
  real    1m2.816s
  real    1m4.667s
  $ time make check
  real    1m17.989s
  real    1m16.836s

(Note: the file-notify-tests alone take about 1min 1sec, which puts a
lower bound on 'time make -j4 parallel'.)

If however we keep the parallel target, it should be renamed. It seems
off to name a target "parallel" just because it is parallelizable. If
the user doesn't pass -j then the target name is technically
incorrect. "summary" would be a good name given what it does.

> (defun ert-run-tests-batch-and-exit-single ()
> [...]
>          ;; Load a byte-compiled one or TEST-FILE itself.
>          (if (file-newer-than-file-p compiled test-file)
>              (progn
>                (setq base (file-name-nondirectory compiled))
>                (load-file compiled))
>            (let ((buf (find-file-noselect test-file)))
>              (if (with-current-buffer buf
>                    (and (boundp 'no-byte-compile) no-byte-compile))
>                  (with-current-buffer buf
>                    (eval-buffer))
>                (if (byte-compile-file test-file t)
>                    (setq base (file-name-nondirectory compiled))
>                  (princ (format "%s failed to compile the file\n" prefix))
>                  (message "##REPORT:(compile-error \"%s\")##" base)
>                  (kill-emacs 0))))

Why shouldn't Make have compiled the test-file? Perhaps the log files
should depend on the .elc files instead of the .el files.

> (defun ert-run-tests-batch-and-exit-single ()
> [...]
>                  (message "##REPORT:(compile-error \"%s\")##" base)
> [...]
>            (message "##REPORT:(done %d %d)##" total expected)
> [...]
>       (message "##REPORT:(load-error \"%s\")##" base)

It seems the only reason to have ert-run-tests-batch-and-exit-single
is to insert these "##REPORT" tokens. But why can't ert-summary-report
parse:

  '^Ran \([0-9]*\) tests, \([0-9]*\) results as expected'

to get the same information? Then could you remove the
ert-run-tests-batch-and-exit-single function and invoke the existing
ert-run-tests-batch-and-exit?

> (defun ert-summary-report ()
> [...]
>     (when errors
>       (message "\n  Following test files have problems:")

When I ran the parallel target, I didn't get this message at all, even
though I have some test failures. eg from my file-notify-tests.log:

  1 unexpected results:
     FAILED  file-notify-test00-availability

My stdout was:

  Running tests in add-log-tests.el... passed all 4 tests
  [...]
  Running tests in vc-bzr.el... passed 0 tests out of 3
    ## Summary ##
    Ran 441 tests, 420 results as expected, 21 unexpected
  $

[-- Attachment #2: Type: text/html, Size: 4742 bytes --]