From: Barry OReilly <gundaetiapo@gmail.com>
To: handa@gnu.org, emacs-devel@gnu.org
Subject: Re: running each test file independently in test/automated
Date: Mon, 26 Aug 2013 18:41:28 -0400 [thread overview]
Message-ID: <CAFM41H1pJ6=JfLuJ7DjfAf5p8SYgjqY8grJob0AKM9NShY3KUQ@mail.gmail.com> (raw)
[-- Attachment #1: Type: text/plain, Size: 3922 bytes --]
>> Hi, I've found I want to run only one or a few of the tests at a
>> time rather than the whole suite.
> http://lists.gnu.org/archive/html/emacs-devel/2013-08/msg00339.html
>
> is supposed to allow this. I don't know if it is waiting on
> something.
Yes indeed, it provides a way to run individual tests:
$ rm cl-lib.log ; make cl-lib.log
Running tests in cl-lib.el... passed all 8 tests
Some comments on the patch follow.
> TEST_LOGS = $(patsubst %.el, %.log, $(wildcard $(test)/*.el))
Other recipes in the same Makefile determine the set of .el files a
different way: they include .el files in subdirectories except data/.
There aren't actually such .el files, but the moment someone adds one
the make code is inconsistent.
> @test -d `dirname "$@"` || mkdir `dirname "$@"`
Why not: mkdir -p `dirname "$@"`
> parallel: $(TEST_LOGS)
> @cd $(test); $(emacs) -f ert-summary-report $(TEST_LOGS)
Instead of creating the new "parallel" target, could we just have the
"check" target run the tests individually?
One argument against might be that a -j1 build would be longer. Here
are some benchmarks (2 CPU cores). 3 different invocations, 2 samples
each:
$ rm *.log ; time make parallel
real 1m21.869s
real 1m21.918s
$ rm *.log ; time make -j4 parallel
real 1m2.816s
real 1m4.667s
$ time make check
real 1m17.989s
real 1m16.836s
(Note: the file-notify-tests alone take about 1min 1sec, which puts a
lower bound on 'time make -j4 parallel'.)
If however we keep the parallel target, it should be renamed. It seems
off to name a target "parallel" just because it is parallelizable. If
the user doesn't pass -j then the target name is technically
incorrect. "summary" would be a good name given what it does.
> (defun ert-run-tests-batch-and-exit-single ()
> [...]
> ;; Load a byte-compiled one or TEST-FILE itself.
> (if (file-newer-than-file-p compiled test-file)
> (progn
> (setq base (file-name-nondirectory compiled))
> (load-file compiled))
> (let ((buf (find-file-noselect test-file)))
> (if (with-current-buffer buf
> (and (boundp 'no-byte-compile) no-byte-compile))
> (with-current-buffer buf
> (eval-buffer))
> (if (byte-compile-file test-file t)
> (setq base (file-name-nondirectory compiled))
> (princ (format "%s failed to compile the file\n" prefix))
> (message "##REPORT:(compile-error \"%s\")##" base)
> (kill-emacs 0))))
Why shouldn't Make have compiled the test-file? Perhaps the log files
should depend on the .elc files instead of the .el files.
> (defun ert-run-tests-batch-and-exit-single ()
> [...]
> (message "##REPORT:(compile-error \"%s\")##" base)
> [...]
> (message "##REPORT:(done %d %d)##" total expected)
> [...]
> (message "##REPORT:(load-error \"%s\")##" base)
It seems the only reason to have ert-run-tests-batch-and-exit-single
is to insert these "##REPORT" tokens. But why can't ert-summary-report
parse:
'^Ran \([0-9]*\) tests, \([0-9]*\) results as expected'
to get the same information? Then could you remove the
ert-run-tests-batch-and-exit-single function and invoke the existing
ert-run-tests-batch-and-exit?
> (defun ert-summary-report ()
> [...]
> (when errors
> (message "\n Following test files have problems:")
When I ran the parallel target, I didn't get this message at all, even
though I have some test failures. eg from my file-notify-tests.log:
1 unexpected results:
FAILED file-notify-test00-availability
My stdout was:
Running tests in add-log-tests.el... passed all 4 tests
[...]
Running tests in vc-bzr.el... passed 0 tests out of 3
## Summary ##
Ran 441 tests, 420 results as expected, 21 unexpected
$
[-- Attachment #2: Type: text/html, Size: 4742 bytes --]
next reply other threads:[~2013-08-26 22:41 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-08-26 22:41 Barry OReilly [this message]
-- strict thread matches above, loose matches on Subject: below --
2013-07-10 8:49 bug#14781: coding system changes cause make check to hang in icalendar tests Stefan Monnier
2013-08-13 11:45 ` running each test file independently in test/automated Kenichi Handa
2013-08-13 14:45 ` Stefan Monnier
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAFM41H1pJ6=JfLuJ7DjfAf5p8SYgjqY8grJob0AKM9NShY3KUQ@mail.gmail.com' \
--to=gundaetiapo@gmail.com \
--cc=emacs-devel@gnu.org \
--cc=handa@gnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this external index
https://git.savannah.gnu.org/cgit/emacs.git
https://git.savannah.gnu.org/cgit/emacs/org-mode.git
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.