* running each test file independently in test/automated
[not found] <jwvbo6akdj2.fsf-monnier+emacs@gnu.org>
@ 2013-08-13 11:45 ` Kenichi Handa
2013-08-13 14:45 ` Stefan Monnier
0 siblings, 1 reply; 3+ messages in thread
From: Kenichi Handa @ 2013-08-13 11:45 UTC (permalink / raw)
To: Stefan Monnier; +Cc: emacs-devel
[-- Attachment #1: Type: text/plain, Size: 441 bytes --]
In article <jwvbo6akdj2.fsf-monnier+emacs@gnu.org>, Stefan Monnier <monnier@IRO.UMontreal.CA> writes:
> I'm not sure about requiring GNU Make to build Emacs out of a tarball,
> but for "side operations" it should not be a problem.
I tried to implement a code to run each test file in a
different emacs session (and in parallel) assuming that we
can use GNU Make there. May I commit the attached changes?
---
Kenichi Handa
handa@gnu.org
[-- Attachment #2: for-parallel-test.diff --]
[-- Type: text/x-diff, Size: 8014 bytes --]
=== modified file '.bzrignore'
--- .bzrignore 2013-06-22 02:41:14 +0000
+++ .bzrignore 2013-08-13 11:08:17 +0000
@@ -172,6 +172,8 @@
src/stamp-h.in
src/temacs
test/indent/*.new
+test/automated/*.log
+test/automated/failure-tests
+*
src/globals.h
src/gl-stamp
=== modified file 'ChangeLog'
--- ChangeLog 2013-07-13 00:01:43 +0000
+++ ChangeLog 2013-08-13 11:08:33 +0000
@@ -1,3 +1,8 @@
+2013-08-13 Kenichi Handa <handa@gnu.org>
+
+ * .bzrignore: Add test/automated/*.log and
+ test/automated/failure-tests.
+
2013-07-13 Paul Eggert <eggert@cs.ucla.edu>
* configure.ac: Simplify --with-file-notification handling.
=== modified file 'lisp/ChangeLog'
--- lisp/ChangeLog 2013-07-20 11:51:53 +0000
+++ lisp/ChangeLog 2013-08-13 09:18:54 +0000
@@ -1,3 +1,8 @@
+2013-08-13 Kenichi Handa <handa@gnu.org>
+
+ * emacs-lisp/ert.el (ert-run-tests-batch-and-exit-single)
+ (ert-summary-report): New functions.
+
2013-07-20 Kenichi Handa <handa@gnu.org>
* international/mule.el (coding-system-iso-2022-flags): Add
=== modified file 'lisp/emacs-lisp/ert.el'
--- lisp/emacs-lisp/ert.el 2013-07-11 16:13:38 +0000
+++ lisp/emacs-lisp/ert.el 2013-08-13 10:54:58 +0000
@@ -31,7 +31,9 @@
;; `defun' but defines a test, and `ert-run-tests-interactively',
;; which runs tests and offers an interactive interface for inspecting
;; results and debugging. There is also
-;; `ert-run-tests-batch-and-exit' for non-interactive use.
+;; `ert-run-tests-batch-and-exit',
+;; `ert-run-tests-batch-and-exit-single', and `ert-summary-report' for
+;; non-interactive use.
;;
;; The body of `ert-deftest' forms resembles a function body, but the
;; additional operators `should', `should-not' and `should-error' are
@@ -1409,6 +1411,93 @@
(backtrace))
(kill-emacs 2))))
+;;;###autoload
+(defun ert-run-tests-batch-and-exit-single ()
+ "Like `ert-run-tests-batch-and-exit', but run tests in a single file."
+ ;; It is intended that this function is called in batch mode with a
+ ;; test file as the argument and stderr redirected to a log file. A
+ ;; specially formated line is written at the end of that log file.
+ ;; That line is parsed by `ert-summary-report' later.
+ (let* ((test-file (pop command-line-args-left))
+ (compiled (concat (file-name-sans-extension test-file) ".elc"))
+ (base (file-name-nondirectory test-file))
+ ;; Some tests (e.g. flymake-tests.el) calls kill-buffer which
+ ;; may lead to process-kill-buffer-query-function requiring
+ ;; user iteraction. Bind this variable to nil avoid that
+ ;; problem.
+ (kill-buffer-query-functions nil)
+ (prefix (format "Running tests in %s..." base)))
+ (unwind-protect
+ (progn
+ ;; Load a byte-compiled one or TEST-FILE itself.
+ (if (file-newer-than-file-p compiled test-file)
+ (progn
+ (setq base (file-name-nondirectory compiled))
+ (load-file compiled))
+ (let ((buf (find-file-noselect test-file)))
+ (if (with-current-buffer buf
+ (and (boundp 'no-byte-compile) no-byte-compile))
+ (with-current-buffer buf
+ (eval-buffer))
+ (if (byte-compile-file test-file t)
+ (setq base (file-name-nondirectory compiled))
+ (princ (format "%s failed to compile the file\n" prefix))
+ (message "##REPORT:(compile-error \"%s\")##" base)
+ (kill-emacs 0)))))
+ ;; Run tests.
+ (let* ((stats (ert-run-tests-batch))
+ (total (ert-stats-total stats))
+ (expected (ert-stats-completed-expected stats)))
+ (if (= total expected)
+ (princ (format "%s passed all %d tests\n" prefix total))
+ (princ (format "%s passed %d tests out of %d\n"
+ prefix expected total)))
+ (message "##REPORT:(done %d %d)##" total expected)
+ (kill-emacs 0)))
+ (princ (format "%s failed to load the file\n" prefix))
+ (message "##REPORT:(load-error \"%s\")##" base)
+ (kill-emacs 0))))
+
+;;;###autoload
+(defun ert-summary-report ()
+ "Print a test report summarizing test log files."
+ ;; It is intended that the function is called after all log files
+ ;; corresponding to test files were generated by
+ ;; ert-run-tests-batch-and-exit-single.
+ (let ((result (cons 0 0))
+ errors)
+ (with-temp-buffer
+ (dolist (logfile command-line-args-left)
+ (erase-buffer)
+ (insert-file-contents logfile)
+ (goto-char (point-max))
+ (or (search-backward "##REPORT:" nil t)
+ (error "no report line: %s" logfile))
+ (goto-char (match-end 0))
+ (let ((form (read (current-buffer))))
+ (if (eq (car form) 'done)
+ (progn
+ (setcar result (+ (car result) (nth 1 form)))
+ (setcdr result (+ (cdr result) (nth 2 form))))
+ (push form errors)))))
+ (setq command-line-args-left nil)
+ (message " ## Summary ##")
+ (message " Ran %s tests, %d results as expected, %d unexpected"
+ (car result) (cdr result) (- (car result) (cdr result)))
+ (when errors
+ (message "\n Following test files have problems:")
+ (dolist (err errors)
+ (let* ((tag (car err))
+ (reason (cond ((eq tag 'emacs-error)
+ "emacs aborted while processing this file")
+ ((eq tag 'load-error)
+ "loading this file failed")
+ ((eq tag 'compile-error)
+ "compiling this file failed")
+ (t
+ "unknown error"))))
+ (message " %s: %s" (nth 1 err) reason)))))
+ (kill-emacs 0))
;;; Utility functions for load/unload actions.
=== modified file 'test/ChangeLog'
--- test/ChangeLog 2013-07-13 01:55:58 +0000
+++ test/ChangeLog 2013-08-13 11:05:23 +0000
@@ -1,3 +1,11 @@
+2013-08-13 Kenichi Handa <handa@gnu.org>
+
+ * automated/Makefile.in (.SUFFIXES): Add .log.
+ (TEST_LOGS, FAILURE_TESTS, FAILURE_LOGS): New variables.
+ (.el.log, parallel, failure-tests/compile-error.el)
+ (failure-tests/load-error.el, failure-tests/emacs-error.el)
+ (parallel-with-failure-test, clean): New targets.
+
2013-07-13 Fabián Ezequiel Gallina <fgallina@gnu.org>
* automated/python-tests.el (python-imenu-create-index-2)
=== modified file 'test/automated/Makefile.in'
--- test/automated/Makefile.in 2013-06-27 09:26:54 +0000
+++ test/automated/Makefile.in 2013-08-13 06:17:51 +0000
@@ -67,7 +67,7 @@
# subdirectories, to make sure require's and load's in the files being
# compiled find the right files.
-.SUFFIXES: .elc .el
+.SUFFIXES: .elc .el .log
# An old-fashioned suffix rule, which, according to the GNU Make manual,
# cannot have prerequisites.
@@ -136,6 +136,41 @@
cd $(test); rm -f *.elc */*.elc */*/*.elc */*/*/*.elc
$(MAKE) $(MFLAGS) compile EMACS=$(EMACS)
+TEST_LOGS = $(patsubst %.el, %.log, $(wildcard $(test)/*.el))
+
+.el.log:
+ @cd $(test); \
+ ($(emacs) -f ert-run-tests-batch-and-exit-single $< 2> $@ || \
+ (BASE="`basename $<`"; \
+ echo "Running tests in $$BASE... fatal error"; \
+ echo "##REPORT:(emacs-error \"$$BASE\")###" > $@))
+
+parallel: $(TEST_LOGS)
+ @cd $(test); $(emacs) -f ert-summary-report $(TEST_LOGS)
+
+FAILURE_TESTS = \
+ failure-tests/compile-error.el \
+ failure-tests/load-error.el \
+ failure-tests/emacs-error.el
+
+FAILURE_LOGS = $(patsubst %.el, %.log, $(FAILURE_TESTS))
+
+failure-tests/compile-error.el: $(lastword $(MAKEFILE_LIST))
+ @test -d `dirname "$@"` || mkdir `dirname "$@"`
+ @echo '(ert-deftest test-run-error (should kkk))' > $@
+failure-tests/load-error.el: $(lastword $(MAKEFILE_LIST))
+ @test -d `dirname "$@"` || mkdir `dirname "$@"`
+ @echo 'kkk' > $@
+failure-tests/emacs-error.el: $(lastword $(MAKEFILE_LIST))
+ @test -d `dirname "$@"` || mkdir `dirname "$@"`
+ @echo '(kill-emacs 1)' > $@
+
+parallel-with-failure-test: $(TEST_LOGS) $(FAILURE_LOGS)
+ @cd $(test); $(emacs) -f ert-summary-report $(TEST_LOGS) $(FAILURE_LOGS)
+
+clean:
+ cd $(test); rm -f $(TEST_LOGS) $(FAILURE_TESTS)
+
bootstrap-clean:
cd $(test); rm -f *.elc */*.elc */*/*.elc */*/*/*.elc
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: running each test file independently in test/automated
2013-08-13 11:45 ` running each test file independently in test/automated Kenichi Handa
@ 2013-08-13 14:45 ` Stefan Monnier
0 siblings, 0 replies; 3+ messages in thread
From: Stefan Monnier @ 2013-08-13 14:45 UTC (permalink / raw)
To: Kenichi Handa; +Cc: emacs-devel
> I tried to implement a code to run each test file in a
> different emacs session (and in parallel) assuming that we
> can use GNU Make there. May I commit the attached changes?
Sounds OK, but please rename the file to GNUmakefile.
Stefan
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: running each test file independently in test/automated
@ 2013-08-26 22:41 Barry OReilly
0 siblings, 0 replies; 3+ messages in thread
From: Barry OReilly @ 2013-08-26 22:41 UTC (permalink / raw)
To: handa, emacs-devel
[-- Attachment #1: Type: text/plain, Size: 3922 bytes --]
>> Hi, I've found I want to run only one or a few of the tests at a
>> time rather than the whole suite.
> http://lists.gnu.org/archive/html/emacs-devel/2013-08/msg00339.html
>
> is supposed to allow this. I don't know if it is waiting on
> something.
Yes indeed, it provides a way to run individual tests:
$ rm cl-lib.log ; make cl-lib.log
Running tests in cl-lib.el... passed all 8 tests
Some comments on the patch follow.
> TEST_LOGS = $(patsubst %.el, %.log, $(wildcard $(test)/*.el))
Other recipes in the same Makefile determine the set of .el files a
different way: they include .el files in subdirectories except data/.
There aren't actually such .el files, but the moment someone adds one
the make code is inconsistent.
> @test -d `dirname "$@"` || mkdir `dirname "$@"`
Why not: mkdir -p `dirname "$@"`
> parallel: $(TEST_LOGS)
> @cd $(test); $(emacs) -f ert-summary-report $(TEST_LOGS)
Instead of creating the new "parallel" target, could we just have the
"check" target run the tests individually?
One argument against might be that a -j1 build would be longer. Here
are some benchmarks (2 CPU cores). 3 different invocations, 2 samples
each:
$ rm *.log ; time make parallel
real 1m21.869s
real 1m21.918s
$ rm *.log ; time make -j4 parallel
real 1m2.816s
real 1m4.667s
$ time make check
real 1m17.989s
real 1m16.836s
(Note: the file-notify-tests alone take about 1min 1sec, which puts a
lower bound on 'time make -j4 parallel'.)
If however we keep the parallel target, it should be renamed. It seems
off to name a target "parallel" just because it is parallelizable. If
the user doesn't pass -j then the target name is technically
incorrect. "summary" would be a good name given what it does.
> (defun ert-run-tests-batch-and-exit-single ()
> [...]
> ;; Load a byte-compiled one or TEST-FILE itself.
> (if (file-newer-than-file-p compiled test-file)
> (progn
> (setq base (file-name-nondirectory compiled))
> (load-file compiled))
> (let ((buf (find-file-noselect test-file)))
> (if (with-current-buffer buf
> (and (boundp 'no-byte-compile) no-byte-compile))
> (with-current-buffer buf
> (eval-buffer))
> (if (byte-compile-file test-file t)
> (setq base (file-name-nondirectory compiled))
> (princ (format "%s failed to compile the file\n" prefix))
> (message "##REPORT:(compile-error \"%s\")##" base)
> (kill-emacs 0))))
Why shouldn't Make have compiled the test-file? Perhaps the log files
should depend on the .elc files instead of the .el files.
> (defun ert-run-tests-batch-and-exit-single ()
> [...]
> (message "##REPORT:(compile-error \"%s\")##" base)
> [...]
> (message "##REPORT:(done %d %d)##" total expected)
> [...]
> (message "##REPORT:(load-error \"%s\")##" base)
It seems the only reason to have ert-run-tests-batch-and-exit-single
is to insert these "##REPORT" tokens. But why can't ert-summary-report
parse:
'^Ran \([0-9]*\) tests, \([0-9]*\) results as expected'
to get the same information? Then could you remove the
ert-run-tests-batch-and-exit-single function and invoke the existing
ert-run-tests-batch-and-exit?
> (defun ert-summary-report ()
> [...]
> (when errors
> (message "\n Following test files have problems:")
When I ran the parallel target, I didn't get this message at all, even
though I have some test failures. eg from my file-notify-tests.log:
1 unexpected results:
FAILED file-notify-test00-availability
My stdout was:
Running tests in add-log-tests.el... passed all 4 tests
[...]
Running tests in vc-bzr.el... passed 0 tests out of 3
## Summary ##
Ran 441 tests, 420 results as expected, 21 unexpected
$
[-- Attachment #2: Type: text/html, Size: 4742 bytes --]
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2013-08-26 22:41 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <jwvbo6akdj2.fsf-monnier+emacs@gnu.org>
2013-08-13 11:45 ` running each test file independently in test/automated Kenichi Handa
2013-08-13 14:45 ` Stefan Monnier
2013-08-26 22:41 Barry OReilly
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/emacs.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).