From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: Kenichi Handa Newsgroups: gmane.emacs.devel Subject: running each test file independently in test/automated Date: Tue, 13 Aug 2013 20:45:32 +0900 Message-ID: <87fvudyftv.fsf@gnu.org> References: NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Trace: ger.gmane.org 1376394372 26598 80.91.229.3 (13 Aug 2013 11:46:12 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Tue, 13 Aug 2013 11:46:12 +0000 (UTC) Cc: emacs-devel@gnu.org To: Stefan Monnier Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Tue Aug 13 13:46:16 2013 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1V9D3H-00008n-S6 for ged-emacs-devel@m.gmane.org; Tue, 13 Aug 2013 13:46:16 +0200 Original-Received: from localhost ([::1]:39591 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1V9D3H-0002nt-I4 for ged-emacs-devel@m.gmane.org; Tue, 13 Aug 2013 07:46:15 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:36771) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1V9D39-0002nm-94 for emacs-devel@gnu.org; Tue, 13 Aug 2013 07:46:12 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1V9D35-0007nq-87 for emacs-devel@gnu.org; Tue, 13 Aug 2013 07:46:07 -0400 Original-Received: from fencepost.gnu.org ([2001:4830:134:3::e]:57890) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1V9D35-0007nm-48 for emacs-devel@gnu.org; Tue, 13 Aug 2013 07:46:03 -0400 Original-Received: from fl1-119-240-85-153.iba.mesh.ad.jp ([119.240.85.153]:64619 helo=shatin) by fencepost.gnu.org with esmtpsa (TLS1.0:DHE_RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1V9D33-0000Rs-8g; Tue, 13 Aug 2013 07:46:02 -0400 Original-Received: from handa by shatin with local (Exim 4.80) (envelope-from ) id 1V9D2a-0007th-Ue; Tue, 13 Aug 2013 20:45:33 +0900 In-Reply-To: (message from Stefan Monnier on Wed, 10 Jul 2013 04:49:25 -0400) X-detected-operating-system: by eggs.gnu.org: Error: Malformed IPv6 address (bad octet value). X-Received-From: 2001:4830:134:3::e X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Xref: news.gmane.org gmane.emacs.devel:162651 Archived-At: --=-=-= Content-Type: text/plain In article , Stefan Monnier writes: > I'm not sure about requiring GNU Make to build Emacs out of a tarball, > but for "side operations" it should not be a problem. I tried to implement a code to run each test file in a different emacs session (and in parallel) assuming that we can use GNU Make there. May I commit the attached changes? --- Kenichi Handa handa@gnu.org --=-=-= Content-Type: text/x-diff; charset=iso-8859-1 Content-Disposition: inline; filename=for-parallel-test.diff Content-Transfer-Encoding: quoted-printable =3D=3D=3D modified file '.bzrignore' --- .bzrignore 2013-06-22 02:41:14 +0000 +++ .bzrignore 2013-08-13 11:08:17 +0000 @@ -172,6 +172,8 @@ src/stamp-h.in src/temacs test/indent/*.new +test/automated/*.log +test/automated/failure-tests +* src/globals.h src/gl-stamp =3D=3D=3D modified file 'ChangeLog' --- ChangeLog 2013-07-13 00:01:43 +0000 +++ ChangeLog 2013-08-13 11:08:33 +0000 @@ -1,3 +1,8 @@ +2013-08-13 Kenichi Handa + + * .bzrignore: Add test/automated/*.log and + test/automated/failure-tests. + 2013-07-13 Paul Eggert =20 * configure.ac: Simplify --with-file-notification handling. =3D=3D=3D modified file 'lisp/ChangeLog' --- lisp/ChangeLog 2013-07-20 11:51:53 +0000 +++ lisp/ChangeLog 2013-08-13 09:18:54 +0000 @@ -1,3 +1,8 @@ +2013-08-13 Kenichi Handa + + * emacs-lisp/ert.el (ert-run-tests-batch-and-exit-single) + (ert-summary-report): New functions. + 2013-07-20 Kenichi Handa =20 * international/mule.el (coding-system-iso-2022-flags): Add =3D=3D=3D modified file 'lisp/emacs-lisp/ert.el' --- lisp/emacs-lisp/ert.el 2013-07-11 16:13:38 +0000 +++ lisp/emacs-lisp/ert.el 2013-08-13 10:54:58 +0000 @@ -31,7 +31,9 @@ ;; `defun' but defines a test, and `ert-run-tests-interactively', ;; which runs tests and offers an interactive interface for inspecting ;; results and debugging. There is also -;; `ert-run-tests-batch-and-exit' for non-interactive use. +;; `ert-run-tests-batch-and-exit', +;; `ert-run-tests-batch-and-exit-single', and `ert-summary-report' for +;; non-interactive use. ;; ;; The body of `ert-deftest' forms resembles a function body, but the ;; additional operators `should', `should-not' and `should-error' are @@ -1409,6 +1411,93 @@ (backtrace)) (kill-emacs 2)))) =20 +;;;###autoload +(defun ert-run-tests-batch-and-exit-single () + "Like `ert-run-tests-batch-and-exit', but run tests in a single file." + ;; It is intended that this function is called in batch mode with a + ;; test file as the argument and stderr redirected to a log file. A + ;; specially formated line is written at the end of that log file. + ;; That line is parsed by `ert-summary-report' later. + (let* ((test-file (pop command-line-args-left)) + (compiled (concat (file-name-sans-extension test-file) ".elc")) + (base (file-name-nondirectory test-file)) + ;; Some tests (e.g. flymake-tests.el) calls kill-buffer which + ;; may lead to process-kill-buffer-query-function requiring + ;; user iteraction. Bind this variable to nil avoid that + ;; problem. + (kill-buffer-query-functions nil) + (prefix (format "Running tests in %s..." base))) + (unwind-protect + (progn + ;; Load a byte-compiled one or TEST-FILE itself. + (if (file-newer-than-file-p compiled test-file) + (progn + (setq base (file-name-nondirectory compiled)) + (load-file compiled)) + (let ((buf (find-file-noselect test-file))) + (if (with-current-buffer buf + (and (boundp 'no-byte-compile) no-byte-compile)) + (with-current-buffer buf + (eval-buffer)) + (if (byte-compile-file test-file t) + (setq base (file-name-nondirectory compiled)) + (princ (format "%s failed to compile the file\n" prefix)) + (message "##REPORT:(compile-error \"%s\")##" base) + (kill-emacs 0))))) + ;; Run tests. + (let* ((stats (ert-run-tests-batch)) + (total (ert-stats-total stats)) + (expected (ert-stats-completed-expected stats))) + (if (=3D total expected) + (princ (format "%s passed all %d tests\n" prefix total)) + (princ (format "%s passed %d tests out of %d\n" + prefix expected total))) + (message "##REPORT:(done %d %d)##" total expected) + (kill-emacs 0))) + (princ (format "%s failed to load the file\n" prefix)) + (message "##REPORT:(load-error \"%s\")##" base) + (kill-emacs 0)))) + +;;;###autoload +(defun ert-summary-report () + "Print a test report summarizing test log files." + ;; It is intended that the function is called after all log files + ;; corresponding to test files were generated by + ;; ert-run-tests-batch-and-exit-single. + (let ((result (cons 0 0)) + errors) + (with-temp-buffer + (dolist (logfile command-line-args-left) + (erase-buffer) + (insert-file-contents logfile) + (goto-char (point-max)) + (or (search-backward "##REPORT:" nil t) + (error "no report line: %s" logfile)) + (goto-char (match-end 0)) + (let ((form (read (current-buffer)))) + (if (eq (car form) 'done) + (progn + (setcar result (+ (car result) (nth 1 form))) + (setcdr result (+ (cdr result) (nth 2 form)))) + (push form errors))))) + (setq command-line-args-left nil) + (message " ## Summary ##") + (message " Ran %s tests, %d results as expected, %d unexpected" + (car result) (cdr result) (- (car result) (cdr result))) + (when errors + (message "\n Following test files have problems:") + (dolist (err errors) + (let* ((tag (car err)) + (reason (cond ((eq tag 'emacs-error) + "emacs aborted while processing this file") + ((eq tag 'load-error) + "loading this file failed") + ((eq tag 'compile-error) + "compiling this file failed") + (t + "unknown error")))) + (message " %s: %s" (nth 1 err) reason))))) + (kill-emacs 0)) =20 ;;; Utility functions for load/unload actions. =20 =3D=3D=3D modified file 'test/ChangeLog' --- test/ChangeLog 2013-07-13 01:55:58 +0000 +++ test/ChangeLog 2013-08-13 11:05:23 +0000 @@ -1,3 +1,11 @@ +2013-08-13 Kenichi Handa + + * automated/Makefile.in (.SUFFIXES): Add .log. + (TEST_LOGS, FAILURE_TESTS, FAILURE_LOGS): New variables. + (.el.log, parallel, failure-tests/compile-error.el) + (failure-tests/load-error.el, failure-tests/emacs-error.el) + (parallel-with-failure-test, clean): New targets. + 2013-07-13 Fabi=E1n Ezequiel Gallina =20 * automated/python-tests.el (python-imenu-create-index-2) =3D=3D=3D modified file 'test/automated/Makefile.in' --- test/automated/Makefile.in 2013-06-27 09:26:54 +0000 +++ test/automated/Makefile.in 2013-08-13 06:17:51 +0000 @@ -67,7 +67,7 @@ # subdirectories, to make sure require's and load's in the files being # compiled find the right files. =20 -.SUFFIXES: .elc .el +.SUFFIXES: .elc .el .log =20 # An old-fashioned suffix rule, which, according to the GNU Make manual, # cannot have prerequisites. @@ -136,6 +136,41 @@ cd $(test); rm -f *.elc */*.elc */*/*.elc */*/*/*.elc $(MAKE) $(MFLAGS) compile EMACS=3D$(EMACS) =20 +TEST_LOGS =3D $(patsubst %.el, %.log, $(wildcard $(test)/*.el)) + +.el.log: + @cd $(test); \ + ($(emacs) -f ert-run-tests-batch-and-exit-single $< 2> $@ || \ + (BASE=3D"`basename $<`"; \ + echo "Running tests in $$BASE... fatal error"; \ + echo "##REPORT:(emacs-error \"$$BASE\")###" > $@)) + +parallel: $(TEST_LOGS) + @cd $(test); $(emacs) -f ert-summary-report $(TEST_LOGS) + +FAILURE_TESTS =3D \ + failure-tests/compile-error.el \ + failure-tests/load-error.el \ + failure-tests/emacs-error.el + +FAILURE_LOGS =3D $(patsubst %.el, %.log, $(FAILURE_TESTS)) + +failure-tests/compile-error.el: $(lastword $(MAKEFILE_LIST)) + @test -d `dirname "$@"` || mkdir `dirname "$@"` + @echo '(ert-deftest test-run-error (should kkk))' > $@ +failure-tests/load-error.el: $(lastword $(MAKEFILE_LIST)) + @test -d `dirname "$@"` || mkdir `dirname "$@"` + @echo 'kkk' > $@ +failure-tests/emacs-error.el: $(lastword $(MAKEFILE_LIST)) + @test -d `dirname "$@"` || mkdir `dirname "$@"` + @echo '(kill-emacs 1)' > $@ + +parallel-with-failure-test: $(TEST_LOGS) $(FAILURE_LOGS) + @cd $(test); $(emacs) -f ert-summary-report $(TEST_LOGS) $(FAILURE_LOGS) + +clean: + cd $(test); rm -f $(TEST_LOGS) $(FAILURE_TESTS) + bootstrap-clean: cd $(test); rm -f *.elc */*.elc */*/*.elc */*/*/*.elc =20 --=-=-=--