* bug#9803: Add ERT option to skip test @ 2011-10-20 3:42 Glenn Morris 2013-07-04 18:40 ` Michael Albinus 2013-10-18 13:37 ` bug#9803: [PATCH] " Michael Albinus 0 siblings, 2 replies; 17+ messages in thread From: Glenn Morris @ 2011-10-20 3:42 UTC (permalink / raw) To: 9803 Package: emacs Version: 24.0.90 Severity: wishlist I think it would be nice if ert had the ability to skip tests. Eg, a :skip argument that works the same way as :expected-result. This would be useful eg when a test relies on external executable that might not be installed on the system running the tests. You can get the same result by using :expected-result, but :skip might be nicer in such cases. ^ permalink raw reply [flat|nested] 17+ messages in thread
* bug#9803: Add ERT option to skip test 2011-10-20 3:42 bug#9803: Add ERT option to skip test Glenn Morris @ 2013-07-04 18:40 ` Michael Albinus 2013-10-18 13:37 ` bug#9803: [PATCH] " Michael Albinus 1 sibling, 0 replies; 17+ messages in thread From: Michael Albinus @ 2013-07-04 18:40 UTC (permalink / raw) To: Glenn Morris; +Cc: 9803 Glenn Morris <rgm@gnu.org> writes: > I think it would be nice if ert had the ability to skip tests. > Eg, a :skip argument that works the same way as :expected-result. > This would be useful eg when a test relies on external executable that > might not be installed on the system running the tests. You can get the > same result by using :expected-result, but :skip might be nicer in such > cases. Skipping tests is already possible. Instead of writing (ert-deftest vc-bzr-test-bug9726 () "Test for http://debbugs.gnu.org/9726 ." :expected-result (if (executable-find vc-bzr-program) :passed :failed) (should (executable-find vc-bzr-program)) ... you could do (when (executable-find vc-bzr-program) (ert-deftest vc-bzr-test-bug9726 () "Test for http://debbugs.gnu.org/9726 ." ... The only drawback I see is that such skipped tests are not listed in the output of an ert run. Best regards, Michael. ^ permalink raw reply [flat|nested] 17+ messages in thread
* bug#9803: [PATCH] Add ERT option to skip test 2011-10-20 3:42 bug#9803: Add ERT option to skip test Glenn Morris 2013-07-04 18:40 ` Michael Albinus @ 2013-10-18 13:37 ` Michael Albinus 2013-10-19 1:02 ` Glenn Morris 1 sibling, 1 reply; 17+ messages in thread From: Michael Albinus @ 2013-10-18 13:37 UTC (permalink / raw) To: Glenn Morris; +Cc: 9803 [-- Attachment #1: Type: text/plain, Size: 744 bytes --] Glenn Morris <rgm@gnu.org> writes: Hi Glenn, > I think it would be nice if ert had the ability to skip tests. > Eg, a :skip argument that works the same way as :expected-result. > This would be useful eg when a test relies on external executable that > might not be installed on the system running the tests. You can get the > same result by using :expected-result, but :skip might be nicer in such > cases. I have written a new macro, which should do the job. It is called `skip-if' and works like `should' and companions. You pass a form as argument, and when it returns non-nil the test is skipped. Test summary is showing skipped tests. Could you, please, check the appended patch, whether it fits your needs? Best regards, Michael. [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: ert.el.patch --] [-- Type: text/x-diff, Size: 9284 bytes --] === modified file 'lisp/emacs-lisp/ert.el' *** lisp/emacs-lisp/ert.el 2013-09-17 07:39:54 +0000 --- lisp/emacs-lisp/ert.el 2013-10-18 13:07:57 +0000 *************** *** 174,181 **** BODY is evaluated as a `progn' when the test is run. It should signal a condition on failure or just return if the test passes. ! `should', `should-not' and `should-error' are useful for ! assertions in BODY. Use `ert' to run tests interactively. --- 174,181 ---- BODY is evaluated as a `progn' when the test is run. It should signal a condition on failure or just return if the test passes. ! `should', `should-not', `should-error' and `skip-if' are useful ! for assertions in BODY. Use `ert' to run tests interactively. *************** *** 237,242 **** --- 237,243 ---- (define-error 'ert-test-failed "Test failed") + (define-error 'ert-test-skipped "Test skipped") (defun ert-pass () "Terminate the current test and mark it passed. Does not return." *************** *** 247,252 **** --- 248,258 ---- DATA is displayed to the user and should state the reason of the failure." (signal 'ert-test-failed (list data))) + (defun ert-skip (data) + "Terminate the current test and mark it skipped. Does not return. + DATA is displayed to the user and should state the reason of the failure." + (signal 'ert-test-skipped (list data))) + ;;; The `should' macros. *************** *** 425,430 **** --- 431,446 ---- (list :fail-reason "did not signal an error"))))))))) + (cl-defmacro skip-if (form) + "Evaluate FORM. If it returns non-nil, skip the current test. + + Returns nil." + (declare (debug t)) + (ert--expand-should `(skip-if ,form) form + (lambda (inner-form form-description-form _value-var) + `(unless (not ,inner-form) + (ert-skip ,form-description-form))))) + ;;; Explanation of `should' failures. *************** *** 644,649 **** --- 660,666 ---- (infos (cl-assert nil))) (cl-defstruct (ert-test-quit (:include ert-test-result-with-condition))) (cl-defstruct (ert-test-failed (:include ert-test-result-with-condition))) + (cl-defstruct (ert-test-skipped (:include ert-test-result-with-condition))) (cl-defstruct (ert-test-aborted-with-non-local-exit (:include ert-test-result))) *************** *** 728,733 **** --- 745,751 ---- (let* ((condition (car more-debugger-args)) (type (cl-case (car condition) ((quit) 'quit) + ((ert-test-skipped) 'skipped) (otherwise 'failed))) (backtrace (ert--record-backtrace)) (infos (reverse ert--infos))) *************** *** 737,742 **** --- 755,764 ---- (make-ert-test-quit :condition condition :backtrace backtrace :infos infos)) + (skipped + (make-ert-test-skipped :condition condition + :backtrace backtrace + :infos infos)) (failed (make-ert-test-failed :condition condition :backtrace backtrace *************** *** 862,868 **** nil -- Never matches. t -- Always matches. ! :failed, :passed -- Matches corresponding results. \(and TYPES...\) -- Matches if all TYPES match. \(or TYPES...\) -- Matches if some TYPES match. \(not TYPE\) -- Matches if TYPE does not match. --- 884,890 ---- nil -- Never matches. t -- Always matches. ! :failed, :passed :skipped -- Matches corresponding results. \(and TYPES...\) -- Matches if all TYPES match. \(or TYPES...\) -- Matches if some TYPES match. \(not TYPE\) -- Matches if TYPE does not match. *************** *** 875,880 **** --- 897,903 ---- ((member t) t) ((member :failed) (ert-test-failed-p result)) ((member :passed) (ert-test-passed-p result)) + ((member :skipped) (ert-test-skipped-p result)) (cons (cl-destructuring-bind (operator &rest operands) result-type (cl-ecase operator *************** *** 899,905 **** (defun ert-test-result-expected-p (test result) "Return non-nil if TEST's expected result type matches RESULT." ! (ert-test-result-type-p result (ert-test-expected-result-type test))) (defun ert-select-tests (selector universe) "Return a list of tests that match SELECTOR. --- 922,930 ---- (defun ert-test-result-expected-p (test result) "Return non-nil if TEST's expected result type matches RESULT." ! (or ! (ert-test-result-type-p result :skipped) ! (ert-test-result-type-p result (ert-test-expected-result-type test)))) (defun ert-select-tests (selector universe) "Return a list of tests that match SELECTOR. *************** *** 1085,1090 **** --- 1110,1116 ---- (passed-unexpected 0) (failed-expected 0) (failed-unexpected 0) + (skipped 0) (start-time nil) (end-time nil) (aborted-p nil) *************** *** 1103,1112 **** (+ (ert--stats-passed-unexpected stats) (ert--stats-failed-unexpected stats))) (defun ert-stats-completed (stats) "Number of tests in STATS that have run so far." (+ (ert-stats-completed-expected stats) ! (ert-stats-completed-unexpected stats))) (defun ert-stats-total (stats) "Number of tests in STATS, regardless of whether they have run yet." --- 1129,1143 ---- (+ (ert--stats-passed-unexpected stats) (ert--stats-failed-unexpected stats))) + (defun ert-stats-skipped (stats) + "Number of tests in STATS that have skipped." + (ert--stats-skipped stats)) + (defun ert-stats-completed (stats) "Number of tests in STATS that have run so far." (+ (ert-stats-completed-expected stats) ! (ert-stats-completed-unexpected stats) ! (ert-stats-skipped stats))) (defun ert-stats-total (stats) "Number of tests in STATS, regardless of whether they have run yet." *************** *** 1138,1143 **** --- 1169,1176 ---- (cl-incf (ert--stats-passed-expected stats) d)) (ert-test-failed (cl-incf (ert--stats-failed-expected stats) d)) + (ert-test-skipped + (cl-incf (ert--stats-skipped stats) d)) (null) (ert-test-aborted-with-non-local-exit) (ert-test-quit)) *************** *** 1146,1151 **** --- 1179,1186 ---- (cl-incf (ert--stats-passed-unexpected stats) d)) (ert-test-failed (cl-incf (ert--stats-failed-unexpected stats) d)) + (ert-test-skipped + (cl-incf (ert--stats-skipped stats) d)) (null) (ert-test-aborted-with-non-local-exit) (ert-test-quit))))) *************** *** 1240,1245 **** --- 1275,1281 ---- (let ((s (cl-etypecase result (ert-test-passed ".P") (ert-test-failed "fF") + (ert-test-skipped "sS") (null "--") (ert-test-aborted-with-non-local-exit "aA") (ert-test-quit "qQ")))) *************** *** 1252,1257 **** --- 1288,1294 ---- (let ((s (cl-etypecase result (ert-test-passed '("passed" "PASSED")) (ert-test-failed '("failed" "FAILED")) + (ert-test-skipped '("skipped" "SKIPPED")) (null '("unknown" "UNKNOWN")) (ert-test-aborted-with-non-local-exit '("aborted" "ABORTED")) (ert-test-quit '("quit" "QUIT"))))) *************** *** 1562,1576 **** (ert--insert-human-readable-selector (ert--stats-selector stats)) (insert "\n") (insert ! (format (concat "Passed: %s\n" ! "Failed: %s\n" ! "Total: %s/%s\n\n") (ert--results-format-expected-unexpected (ert--stats-passed-expected stats) (ert--stats-passed-unexpected stats)) (ert--results-format-expected-unexpected (ert--stats-failed-expected stats) (ert--stats-failed-unexpected stats)) run-count (ert-stats-total stats))) (insert --- 1599,1615 ---- (ert--insert-human-readable-selector (ert--stats-selector stats)) (insert "\n") (insert ! (format (concat "Passed: %s\n" ! "Failed: %s\n" ! "Skipped: %s\n" ! "Total: %s/%s\n\n") (ert--results-format-expected-unexpected (ert--stats-passed-expected stats) (ert--stats-passed-unexpected stats)) (ert--results-format-expected-unexpected (ert--stats-failed-expected stats) (ert--stats-failed-unexpected stats)) + (ert-stats-skipped stats) run-count (ert-stats-total stats))) (insert ^ permalink raw reply [flat|nested] 17+ messages in thread
* bug#9803: [PATCH] Add ERT option to skip test 2013-10-18 13:37 ` bug#9803: [PATCH] " Michael Albinus @ 2013-10-19 1:02 ` Glenn Morris 2013-10-19 2:12 ` Stefan Monnier ` (2 more replies) 0 siblings, 3 replies; 17+ messages in thread From: Glenn Morris @ 2013-10-19 1:02 UTC (permalink / raw) To: Michael Albinus; +Cc: 9803 Michael Albinus wrote: >> I think it would be nice if ert had the ability to skip tests. >> Eg, a :skip argument that works the same way as :expected-result. [...] > I have written a new macro, which should do the job. It is called > `skip-if' and works like `should' and companions. You pass a form as > argument, and when it returns non-nil the test is skipped. Thank you, looks nice. Your method is not quite how I imagined it working, but maybe your way is better, I haven't thought about it much... With your approach, it seems like I have to specify the skip condition twice? Eg I have to write: (ert-deftest foo-test () "Test for foo." :expected-result (if (executable-find "foo") :passed :skipped) (skip-if (not (executable-find "foo"))) t ; in a real use case, some test using "foo" here ) rather than: (ert-deftest foo-test () "Test for foo." :skip (not (executable-find "foo")) t) I think `skip-if' should have an ert- prefix. (I know `should' doesn't, but I think it, err, should as well. But too late for that one now.) Also, ert-run-tests-batch-and-exit seems to need updating: Running 1 tests (2013-10-18 17:49:11-0700) skipped 1/1 foo-test Ran 1 tests, 0 results as expected (2013-10-18 17:49:11-0700) I don't think "0 results as expected" is appropriate. Eg automake uses a summary like this: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=11745 ^ permalink raw reply [flat|nested] 17+ messages in thread
* bug#9803: [PATCH] Add ERT option to skip test 2013-10-19 1:02 ` Glenn Morris @ 2013-10-19 2:12 ` Stefan Monnier 2013-10-19 6:44 ` Michael Albinus 2013-10-19 6:44 ` Michael Albinus 2013-10-20 14:09 ` Michael Albinus 2 siblings, 1 reply; 17+ messages in thread From: Stefan Monnier @ 2013-10-19 2:12 UTC (permalink / raw) To: Glenn Morris; +Cc: Michael Albinus, 9803 > I think `skip-if' should have an ert- prefix. (I know `should' doesn't, > but I think it, err, should as well. But too late for that one now.) Rather than add a prefix to `should' we could/should make it so it's only defined within an ert-deftest (like the `go' macro in `cl-tagbody'). Stefan ^ permalink raw reply [flat|nested] 17+ messages in thread
* bug#9803: [PATCH] Add ERT option to skip test 2013-10-19 2:12 ` Stefan Monnier @ 2013-10-19 6:44 ` Michael Albinus 0 siblings, 0 replies; 17+ messages in thread From: Michael Albinus @ 2013-10-19 6:44 UTC (permalink / raw) To: Stefan Monnier; +Cc: 9803 Stefan Monnier <monnier@iro.umontreal.ca> writes: >> I think `skip-if' should have an ert- prefix. (I know `should' doesn't, >> but I think it, err, should as well. But too late for that one now.) > > Rather than add a prefix to `should' we could/should make it so it's > only defined within an ert-deftest (like the `go' macro in `cl-tagbody'). Yep. Will do. > Stefan Best regards, Michael. ^ permalink raw reply [flat|nested] 17+ messages in thread
* bug#9803: [PATCH] Add ERT option to skip test 2013-10-19 1:02 ` Glenn Morris 2013-10-19 2:12 ` Stefan Monnier @ 2013-10-19 6:44 ` Michael Albinus 2013-10-20 2:02 ` Glenn Morris 2013-10-20 14:09 ` Michael Albinus 2 siblings, 1 reply; 17+ messages in thread From: Michael Albinus @ 2013-10-19 6:44 UTC (permalink / raw) To: Glenn Morris; +Cc: 9803 Glenn Morris <rgm@gnu.org> writes: > Thank you, looks nice. > Your method is not quite how I imagined it working, but maybe your way is > better, I haven't thought about it much... With your approach, it seems > like I have to specify the skip condition twice? Eg I have to write: > > (ert-deftest foo-test () > "Test for foo." > :expected-result (if (executable-find "foo") :passed :skipped) > (skip-if (not (executable-find "foo"))) > t ; in a real use case, some test using "foo" here > ) > > rather than: > > (ert-deftest foo-test () > "Test for foo." > :skip (not (executable-find "foo")) > t) No. The idea is that skip-if works whatever you have defined in :expected-result. It simply ignores :expected-result, when it finds a non-nil form. > Also, ert-run-tests-batch-and-exit seems to need updating: > > Running 1 tests (2013-10-18 17:49:11-0700) > skipped 1/1 foo-test > > Ran 1 tests, 0 results as expected (2013-10-18 17:49:11-0700) > > I don't think "0 results as expected" is appropriate. Will check. I've tested the interactive call so far. Best regards, Michael. ^ permalink raw reply [flat|nested] 17+ messages in thread
* bug#9803: [PATCH] Add ERT option to skip test 2013-10-19 6:44 ` Michael Albinus @ 2013-10-20 2:02 ` Glenn Morris 0 siblings, 0 replies; 17+ messages in thread From: Glenn Morris @ 2013-10-20 2:02 UTC (permalink / raw) To: Michael Albinus; +Cc: 9803 Michael Albinus wrote: > No. The idea is that skip-if works whatever you have defined in > :expected-result. It simply ignores :expected-result, when it finds a > non-nil form. For some reason I thought I had to specify :expected-result :skipped as well, but I can no longer reproduce whatever led me to think that. ^ permalink raw reply [flat|nested] 17+ messages in thread
* bug#9803: [PATCH] Add ERT option to skip test 2013-10-19 1:02 ` Glenn Morris 2013-10-19 2:12 ` Stefan Monnier 2013-10-19 6:44 ` Michael Albinus @ 2013-10-20 14:09 ` Michael Albinus 2013-10-21 15:08 ` Michael Albinus 2 siblings, 1 reply; 17+ messages in thread From: Michael Albinus @ 2013-10-20 14:09 UTC (permalink / raw) To: Glenn Morris; +Cc: 9803 [-- Attachment #1: Type: text/plain, Size: 845 bytes --] Glenn Morris <rgm@gnu.org> writes: Hi Glenn, > I think `skip-if' should have an ert- prefix. (I know `should' doesn't, > but I think it, err, should as well. But too late for that one now.) I've tried to change it as proposed by Stefan, but I'm too stupid to manage all this sophisticated cl-* stuff :-( > Also, ert-run-tests-batch-and-exit seems to need updating: > > Running 1 tests (2013-10-18 17:49:11-0700) > skipped 1/1 foo-test > > Ran 1 tests, 0 results as expected (2013-10-18 17:49:11-0700) > > I don't think "0 results as expected" is appropriate. > > Eg automake uses a summary like this: > http://debbugs.gnu.org/cgi/bugreport.cgi?bug=11745 Such a verbose summary is returned when you run the tests interactively. I've reworked the patch in order to give also valid statistics in batch mode. Best regards, Michael. [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: 123.patch --] [-- Type: text/x-patch, Size: 11887 bytes --] === modified file 'lisp/emacs-lisp/ert.el' *** lisp/emacs-lisp/ert.el 2013-09-17 07:39:54 +0000 --- lisp/emacs-lisp/ert.el 2013-10-20 13:55:50 +0000 *************** *** 174,181 **** BODY is evaluated as a `progn' when the test is run. It should signal a condition on failure or just return if the test passes. ! `should', `should-not' and `should-error' are useful for ! assertions in BODY. Use `ert' to run tests interactively. --- 174,181 ---- BODY is evaluated as a `progn' when the test is run. It should signal a condition on failure or just return if the test passes. ! `should', `should-not', `should-error' and `skip-if' are useful ! for assertions in BODY. Use `ert' to run tests interactively. *************** *** 237,242 **** --- 237,243 ---- (define-error 'ert-test-failed "Test failed") + (define-error 'ert-test-skipped "Test skipped") (defun ert-pass () "Terminate the current test and mark it passed. Does not return." *************** *** 247,252 **** --- 248,258 ---- DATA is displayed to the user and should state the reason of the failure." (signal 'ert-test-failed (list data))) + (defun ert-skip (data) + "Terminate the current test and mark it skipped. Does not return. + DATA is displayed to the user and should state the reason of the failure." + (signal 'ert-test-skipped (list data))) + ;;; The `should' macros. *************** *** 425,430 **** --- 431,446 ---- (list :fail-reason "did not signal an error"))))))))) + (cl-defmacro skip-if (form) + "Evaluate FORM. If it returns non-nil, skip the current test. + + Returns nil." + (declare (debug t)) + (ert--expand-should `(skip-if ,form) form + (lambda (inner-form form-description-form _value-var) + `(unless (not ,inner-form) + (ert-skip ,form-description-form))))) + ;;; Explanation of `should' failures. *************** *** 644,649 **** --- 660,666 ---- (infos (cl-assert nil))) (cl-defstruct (ert-test-quit (:include ert-test-result-with-condition))) (cl-defstruct (ert-test-failed (:include ert-test-result-with-condition))) + (cl-defstruct (ert-test-skipped (:include ert-test-result-with-condition))) (cl-defstruct (ert-test-aborted-with-non-local-exit (:include ert-test-result))) *************** *** 728,733 **** --- 745,751 ---- (let* ((condition (car more-debugger-args)) (type (cl-case (car condition) ((quit) 'quit) + ((ert-test-skipped) 'skipped) (otherwise 'failed))) (backtrace (ert--record-backtrace)) (infos (reverse ert--infos))) *************** *** 737,742 **** --- 755,764 ---- (make-ert-test-quit :condition condition :backtrace backtrace :infos infos)) + (skipped + (make-ert-test-skipped :condition condition + :backtrace backtrace + :infos infos)) (failed (make-ert-test-failed :condition condition :backtrace backtrace *************** *** 862,868 **** nil -- Never matches. t -- Always matches. ! :failed, :passed -- Matches corresponding results. \(and TYPES...\) -- Matches if all TYPES match. \(or TYPES...\) -- Matches if some TYPES match. \(not TYPE\) -- Matches if TYPE does not match. --- 884,890 ---- nil -- Never matches. t -- Always matches. ! :failed, :passed :skipped -- Matches corresponding results. \(and TYPES...\) -- Matches if all TYPES match. \(or TYPES...\) -- Matches if some TYPES match. \(not TYPE\) -- Matches if TYPE does not match. *************** *** 875,880 **** --- 897,903 ---- ((member t) t) ((member :failed) (ert-test-failed-p result)) ((member :passed) (ert-test-passed-p result)) + ((member :skipped) (ert-test-skipped-p result)) (cons (cl-destructuring-bind (operator &rest operands) result-type (cl-ecase operator *************** *** 899,905 **** (defun ert-test-result-expected-p (test result) "Return non-nil if TEST's expected result type matches RESULT." ! (ert-test-result-type-p result (ert-test-expected-result-type test))) (defun ert-select-tests (selector universe) "Return a list of tests that match SELECTOR. --- 922,930 ---- (defun ert-test-result-expected-p (test result) "Return non-nil if TEST's expected result type matches RESULT." ! (or ! (ert-test-result-type-p result :skipped) ! (ert-test-result-type-p result (ert-test-expected-result-type test)))) (defun ert-select-tests (selector universe) "Return a list of tests that match SELECTOR. *************** *** 1085,1090 **** --- 1110,1116 ---- (passed-unexpected 0) (failed-expected 0) (failed-unexpected 0) + (skipped 0) (start-time nil) (end-time nil) (aborted-p nil) *************** *** 1103,1112 **** (+ (ert--stats-passed-unexpected stats) (ert--stats-failed-unexpected stats))) (defun ert-stats-completed (stats) "Number of tests in STATS that have run so far." (+ (ert-stats-completed-expected stats) ! (ert-stats-completed-unexpected stats))) (defun ert-stats-total (stats) "Number of tests in STATS, regardless of whether they have run yet." --- 1129,1143 ---- (+ (ert--stats-passed-unexpected stats) (ert--stats-failed-unexpected stats))) + (defun ert-stats-skipped (stats) + "Number of tests in STATS that have skipped." + (ert--stats-skipped stats)) + (defun ert-stats-completed (stats) "Number of tests in STATS that have run so far." (+ (ert-stats-completed-expected stats) ! (ert-stats-completed-unexpected stats) ! (ert-stats-skipped stats))) (defun ert-stats-total (stats) "Number of tests in STATS, regardless of whether they have run yet." *************** *** 1138,1143 **** --- 1169,1176 ---- (cl-incf (ert--stats-passed-expected stats) d)) (ert-test-failed (cl-incf (ert--stats-failed-expected stats) d)) + (ert-test-skipped + (cl-incf (ert--stats-skipped stats) d)) (null) (ert-test-aborted-with-non-local-exit) (ert-test-quit)) *************** *** 1146,1151 **** --- 1179,1186 ---- (cl-incf (ert--stats-passed-unexpected stats) d)) (ert-test-failed (cl-incf (ert--stats-failed-unexpected stats) d)) + (ert-test-skipped + (cl-incf (ert--stats-skipped stats) d)) (null) (ert-test-aborted-with-non-local-exit) (ert-test-quit))))) *************** *** 1240,1245 **** --- 1275,1281 ---- (let ((s (cl-etypecase result (ert-test-passed ".P") (ert-test-failed "fF") + (ert-test-skipped "sS") (null "--") (ert-test-aborted-with-non-local-exit "aA") (ert-test-quit "qQ")))) *************** *** 1252,1257 **** --- 1288,1294 ---- (let ((s (cl-etypecase result (ert-test-passed '("passed" "PASSED")) (ert-test-failed '("failed" "FAILED")) + (ert-test-skipped '("skipped" "SKIPPED")) (null '("unknown" "UNKNOWN")) (ert-test-aborted-with-non-local-exit '("aborted" "ABORTED")) (ert-test-quit '("quit" "QUIT"))))) *************** *** 1317,1330 **** (ert--format-time-iso8601 (ert--stats-start-time stats))))) (run-ended (cl-destructuring-bind (stats abortedp) event-args ! (let ((unexpected (ert-stats-completed-unexpected stats)) (expected-failures (ert--stats-failed-expected stats))) ! (message "\n%sRan %s tests, %s results as expected%s (%s)%s\n" (if (not abortedp) "" "Aborted: ") (ert-stats-total stats) ! (ert-stats-completed-expected stats) (if (zerop unexpected) "" (format ", %s unexpected" unexpected)) --- 1354,1371 ---- (ert--format-time-iso8601 (ert--stats-start-time stats))))) (run-ended (cl-destructuring-bind (stats abortedp) event-args ! (let ((skipped (ert-stats-skipped stats)) ! (unexpected (ert-stats-completed-unexpected stats)) (expected-failures (ert--stats-failed-expected stats))) ! (message "\n%sRan %s tests, %s results as expected%s%s (%s)%s\n" (if (not abortedp) "" "Aborted: ") (ert-stats-total stats) ! (ert-stats-completed-expected stats) ! (if (zerop skipped) ! "" ! (format ", %s skipped" skipped)) (if (zerop unexpected) "" (format ", %s unexpected" unexpected)) *************** *** 1332,1337 **** --- 1373,1387 ---- (if (zerop expected-failures) "" (format "\n%s expected failures" expected-failures))) + (unless (zerop skipped) + (message "%s skipped results:" skipped) + (cl-loop for test across (ert--stats-tests stats) + for result = (ert-test-most-recent-result test) do + (when (ert-test-result-type-p result :skipped) + (message "%9s %S" + (ert-string-for-test-result result nil) + (ert-test-name test)))) + (message "%s" "")) (unless (zerop unexpected) (message "%s unexpected results:" unexpected) (cl-loop for test across (ert--stats-tests stats) *************** *** 1562,1576 **** (ert--insert-human-readable-selector (ert--stats-selector stats)) (insert "\n") (insert ! (format (concat "Passed: %s\n" ! "Failed: %s\n" ! "Total: %s/%s\n\n") (ert--results-format-expected-unexpected (ert--stats-passed-expected stats) (ert--stats-passed-unexpected stats)) (ert--results-format-expected-unexpected (ert--stats-failed-expected stats) (ert--stats-failed-unexpected stats)) run-count (ert-stats-total stats))) (insert --- 1612,1628 ---- (ert--insert-human-readable-selector (ert--stats-selector stats)) (insert "\n") (insert ! (format (concat "Passed: %s\n" ! "Failed: %s\n" ! "Skipped: %s\n" ! "Total: %s/%s\n\n") (ert--results-format-expected-unexpected (ert--stats-passed-expected stats) (ert--stats-passed-unexpected stats)) (ert--results-format-expected-unexpected (ert--stats-failed-expected stats) (ert--stats-failed-unexpected stats)) + (ert-stats-skipped stats) run-count (ert-stats-total stats))) (insert ^ permalink raw reply [flat|nested] 17+ messages in thread
* bug#9803: [PATCH] Add ERT option to skip test 2013-10-20 14:09 ` Michael Albinus @ 2013-10-21 15:08 ` Michael Albinus 2013-10-21 16:53 ` Stefan Monnier 0 siblings, 1 reply; 17+ messages in thread From: Michael Albinus @ 2013-10-21 15:08 UTC (permalink / raw) To: Glenn Morris; +Cc: 9803 [-- Attachment #1: Type: text/plain, Size: 719 bytes --] Michael Albinus <michael.albinus@gmx.de> writes: Hi Glenn, >> I think `skip-if' should have an ert- prefix. (I know `should' doesn't, >> but I think it, err, should as well. But too late for that one now.) > > I've tried to change it as proposed by Stefan, but I'm too stupid to > manage all this sophisticated cl-* stuff :-( Well, I've made a change to call (fset 'skip-if 'ert--skip-if) in ert--run-test-internal. After running the test, it is reverted by (unintern 'skip-if nil). Therefore, `skip-if' is visible only inside tests defined with `ert-deftest'. If this is acceptable, I could apply this change also for `should', `should-not' and `should-error'. Do you (and Stefan) agree? Best regards, Michael. [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: 123.patch --] [-- Type: text/x-diff, Size: 14279 bytes --] === modified file 'lisp/emacs-lisp/ert.el' *** lisp/emacs-lisp/ert.el 2013-09-17 07:39:54 +0000 --- lisp/emacs-lisp/ert.el 2013-10-21 15:04:51 +0000 *************** *** 34,47 **** ;; `ert-run-tests-batch-and-exit' for non-interactive use. ;; ;; The body of `ert-deftest' forms resembles a function body, but the ! ;; additional operators `should', `should-not' and `should-error' are ! ;; available. `should' is similar to cl's `assert', but signals a ! ;; different error when its condition is violated that is caught and ! ;; processed by ERT. In addition, it analyzes its argument form and ! ;; records information that helps debugging (`assert' tries to do ! ;; something similar when its second argument SHOW-ARGS is true, but ! ;; `should' is more sophisticated). For information on `should-not' ! ;; and `should-error', see their docstrings. ;; ;; See ERT's info manual as well as the docstrings for more details. ;; To compile the manual, run `makeinfo ert.texinfo' in the ERT --- 34,50 ---- ;; `ert-run-tests-batch-and-exit' for non-interactive use. ;; ;; The body of `ert-deftest' forms resembles a function body, but the ! ;; additional operators `should', `should-not', `should-error' and ! ;; `skip-if' are available. `should' is similar to cl's `assert', but ! ;; signals a different error when its condition is violated that is ! ;; caught and processed by ERT. In addition, it analyzes its argument ! ;; form and records information that helps debugging (`assert' tries ! ;; to do something similar when its second argument SHOW-ARGS is true, ! ;; but `should' is more sophisticated). For information on ! ;; `should-not' and `should-error', see their docstrings. `skip-if' ! ;; skips the test without checking the result, this is useful for ! ;; checking the test environmont (like availability of external ! ;; binaries, etc). ;; ;; See ERT's info manual as well as the docstrings for more details. ;; To compile the manual, run `makeinfo ert.texinfo' in the ERT *************** *** 174,181 **** BODY is evaluated as a `progn' when the test is run. It should signal a condition on failure or just return if the test passes. ! `should', `should-not' and `should-error' are useful for ! assertions in BODY. Use `ert' to run tests interactively. --- 177,184 ---- BODY is evaluated as a `progn' when the test is run. It should signal a condition on failure or just return if the test passes. ! `should', `should-not', `should-error' and `skip-if' are useful ! for assertions in BODY. Use `ert' to run tests interactively. *************** *** 237,242 **** --- 240,246 ---- (define-error 'ert-test-failed "Test failed") + (define-error 'ert-test-skipped "Test skipped") (defun ert-pass () "Terminate the current test and mark it passed. Does not return." *************** *** 247,252 **** --- 251,261 ---- DATA is displayed to the user and should state the reason of the failure." (signal 'ert-test-failed (list data))) + (defun ert-skip (data) + "Terminate the current test and mark it skipped. Does not return. + DATA is displayed to the user and should state the reason for skipping." + (signal 'ert-test-skipped (list data))) + ;;; The `should' macros. *************** *** 425,430 **** --- 434,449 ---- (list :fail-reason "did not signal an error"))))))))) + (cl-defmacro ert--skip-if (form) + "Evaluate FORM. If it returns non-nil, skip the current test. + + Returns nil." + (declare (debug t)) + (ert--expand-should `(skip-if ,form) form + (lambda (inner-form form-description-form _value-var) + `(unless (not ,inner-form) + (ert-skip ,form-description-form))))) + ;;; Explanation of `should' failures. *************** *** 644,649 **** --- 663,669 ---- (infos (cl-assert nil))) (cl-defstruct (ert-test-quit (:include ert-test-result-with-condition))) (cl-defstruct (ert-test-failed (:include ert-test-result-with-condition))) + (cl-defstruct (ert-test-skipped (:include ert-test-result-with-condition))) (cl-defstruct (ert-test-aborted-with-non-local-exit (:include ert-test-result))) *************** *** 728,733 **** --- 748,754 ---- (let* ((condition (car more-debugger-args)) (type (cl-case (car condition) ((quit) 'quit) + ((ert-test-skipped) 'skipped) (otherwise 'failed))) (backtrace (ert--record-backtrace)) (infos (reverse ert--infos))) *************** *** 737,742 **** --- 758,767 ---- (make-ert-test-quit :condition condition :backtrace backtrace :infos infos)) + (skipped + (make-ert-test-skipped :condition condition + :backtrace backtrace + :infos infos)) (failed (make-ert-test-failed :condition condition :backtrace backtrace *************** *** 774,781 **** ;; and consider it in `ert--run-test-debugger'? (debug-ignored-errors nil) (ert--infos '())) (funcall (ert-test-body (ert--test-execution-info-test ! test-execution-info)))))) (ert-pass)) (setf (ert--test-execution-info-result test-execution-info) (make-ert-test-passed)) --- 799,808 ---- ;; and consider it in `ert--run-test-debugger'? (debug-ignored-errors nil) (ert--infos '())) + (fset 'skip-if 'ert--skip-if) (funcall (ert-test-body (ert--test-execution-info-test ! test-execution-info))) ! (unintern 'skip-if nil)))) (ert-pass)) (setf (ert--test-execution-info-result test-execution-info) (make-ert-test-passed)) *************** *** 862,868 **** nil -- Never matches. t -- Always matches. ! :failed, :passed -- Matches corresponding results. \(and TYPES...\) -- Matches if all TYPES match. \(or TYPES...\) -- Matches if some TYPES match. \(not TYPE\) -- Matches if TYPE does not match. --- 889,895 ---- nil -- Never matches. t -- Always matches. ! :failed, :passed, :skipped -- Matches corresponding results. \(and TYPES...\) -- Matches if all TYPES match. \(or TYPES...\) -- Matches if some TYPES match. \(not TYPE\) -- Matches if TYPE does not match. *************** *** 875,880 **** --- 902,908 ---- ((member t) t) ((member :failed) (ert-test-failed-p result)) ((member :passed) (ert-test-passed-p result)) + ((member :skipped) (ert-test-skipped-p result)) (cons (cl-destructuring-bind (operator &rest operands) result-type (cl-ecase operator *************** *** 899,905 **** (defun ert-test-result-expected-p (test result) "Return non-nil if TEST's expected result type matches RESULT." ! (ert-test-result-type-p result (ert-test-expected-result-type test))) (defun ert-select-tests (selector universe) "Return a list of tests that match SELECTOR. --- 927,935 ---- (defun ert-test-result-expected-p (test result) "Return non-nil if TEST's expected result type matches RESULT." ! (or ! (ert-test-result-type-p result :skipped) ! (ert-test-result-type-p result (ert-test-expected-result-type test)))) (defun ert-select-tests (selector universe) "Return a list of tests that match SELECTOR. *************** *** 1085,1090 **** --- 1115,1121 ---- (passed-unexpected 0) (failed-expected 0) (failed-unexpected 0) + (skipped 0) (start-time nil) (end-time nil) (aborted-p nil) *************** *** 1103,1112 **** (+ (ert--stats-passed-unexpected stats) (ert--stats-failed-unexpected stats))) (defun ert-stats-completed (stats) "Number of tests in STATS that have run so far." (+ (ert-stats-completed-expected stats) ! (ert-stats-completed-unexpected stats))) (defun ert-stats-total (stats) "Number of tests in STATS, regardless of whether they have run yet." --- 1134,1148 ---- (+ (ert--stats-passed-unexpected stats) (ert--stats-failed-unexpected stats))) + (defun ert-stats-skipped (stats) + "Number of tests in STATS that have skipped." + (ert--stats-skipped stats)) + (defun ert-stats-completed (stats) "Number of tests in STATS that have run so far." (+ (ert-stats-completed-expected stats) ! (ert-stats-completed-unexpected stats) ! (ert-stats-skipped stats))) (defun ert-stats-total (stats) "Number of tests in STATS, regardless of whether they have run yet." *************** *** 1138,1143 **** --- 1174,1181 ---- (cl-incf (ert--stats-passed-expected stats) d)) (ert-test-failed (cl-incf (ert--stats-failed-expected stats) d)) + (ert-test-skipped + (cl-incf (ert--stats-skipped stats) d)) (null) (ert-test-aborted-with-non-local-exit) (ert-test-quit)) *************** *** 1146,1151 **** --- 1184,1191 ---- (cl-incf (ert--stats-passed-unexpected stats) d)) (ert-test-failed (cl-incf (ert--stats-failed-unexpected stats) d)) + (ert-test-skipped + (cl-incf (ert--stats-skipped stats) d)) (null) (ert-test-aborted-with-non-local-exit) (ert-test-quit))))) *************** *** 1240,1245 **** --- 1280,1286 ---- (let ((s (cl-etypecase result (ert-test-passed ".P") (ert-test-failed "fF") + (ert-test-skipped "sS") (null "--") (ert-test-aborted-with-non-local-exit "aA") (ert-test-quit "qQ")))) *************** *** 1252,1257 **** --- 1293,1299 ---- (let ((s (cl-etypecase result (ert-test-passed '("passed" "PASSED")) (ert-test-failed '("failed" "FAILED")) + (ert-test-skipped '("skipped" "SKIPPED")) (null '("unknown" "UNKNOWN")) (ert-test-aborted-with-non-local-exit '("aborted" "ABORTED")) (ert-test-quit '("quit" "QUIT"))))) *************** *** 1318,1325 **** (run-ended (cl-destructuring-bind (stats abortedp) event-args (let ((unexpected (ert-stats-completed-unexpected stats)) ! (expected-failures (ert--stats-failed-expected stats))) ! (message "\n%sRan %s tests, %s results as expected%s (%s)%s\n" (if (not abortedp) "" "Aborted: ") --- 1360,1368 ---- (run-ended (cl-destructuring-bind (stats abortedp) event-args (let ((unexpected (ert-stats-completed-unexpected stats)) ! (skipped (ert-stats-skipped stats)) ! (expected-failures (ert--stats-failed-expected stats))) ! (message "\n%sRan %s tests, %s results as expected%s%s (%s)%s\n" (if (not abortedp) "" "Aborted: ") *************** *** 1328,1333 **** --- 1371,1379 ---- (if (zerop unexpected) "" (format ", %s unexpected" unexpected)) + (if (zerop skipped) + "" + (format ", %s skipped" skipped)) (ert--format-time-iso8601 (ert--stats-end-time stats)) (if (zerop expected-failures) "" *************** *** 1340,1345 **** --- 1386,1400 ---- (message "%9s %S" (ert-string-for-test-result result nil) (ert-test-name test)))) + (message "%s" "")) + (unless (zerop skipped) + (message "%s skipped results:" skipped) + (cl-loop for test across (ert--stats-tests stats) + for result = (ert-test-most-recent-result test) do + (when (ert-test-result-type-p result :skipped) + (message "%9s %S" + (ert-string-for-test-result result nil) + (ert-test-name test)))) (message "%s" ""))))) (test-started ) *************** *** 1562,1576 **** (ert--insert-human-readable-selector (ert--stats-selector stats)) (insert "\n") (insert ! (format (concat "Passed: %s\n" ! "Failed: %s\n" ! "Total: %s/%s\n\n") (ert--results-format-expected-unexpected (ert--stats-passed-expected stats) (ert--stats-passed-unexpected stats)) (ert--results-format-expected-unexpected (ert--stats-failed-expected stats) (ert--stats-failed-unexpected stats)) run-count (ert-stats-total stats))) (insert --- 1617,1633 ---- (ert--insert-human-readable-selector (ert--stats-selector stats)) (insert "\n") (insert ! (format (concat "Passed: %s\n" ! "Failed: %s\n" ! "Skipped: %s\n" ! "Total: %s/%s\n\n") (ert--results-format-expected-unexpected (ert--stats-passed-expected stats) (ert--stats-passed-unexpected stats)) (ert--results-format-expected-unexpected (ert--stats-failed-expected stats) (ert--stats-failed-unexpected stats)) + (ert-stats-skipped stats) run-count (ert-stats-total stats))) (insert ^ permalink raw reply [flat|nested] 17+ messages in thread
* bug#9803: [PATCH] Add ERT option to skip test 2013-10-21 15:08 ` Michael Albinus @ 2013-10-21 16:53 ` Stefan Monnier 2013-10-21 17:02 ` Michael Albinus 0 siblings, 1 reply; 17+ messages in thread From: Stefan Monnier @ 2013-10-21 16:53 UTC (permalink / raw) To: Michael Albinus; +Cc: 9803 > Well, I've made a change to call (fset 'skip-if 'ert--skip-if) in > ert--run-test-internal. After running the test, it is reverted by > (unintern 'skip-if nil). Yuck! > Therefore, `skip-if' is visible only inside tests defined with > `ert-deftest'. Not "inside" but "during". I.e. it can still affect any code run during the tests. > If this is acceptable, I could apply this change also for > `should', `should-not' and `should-error'. No, better leave them as always-global. > I've tried to change it as proposed by Stefan, but I'm too stupid to > manage all this sophisticated cl-* stuff :-( What have you tried and how did it fail? Stefan ^ permalink raw reply [flat|nested] 17+ messages in thread
* bug#9803: [PATCH] Add ERT option to skip test 2013-10-21 16:53 ` Stefan Monnier @ 2013-10-21 17:02 ` Michael Albinus 2013-10-21 19:23 ` Stefan Monnier 0 siblings, 1 reply; 17+ messages in thread From: Michael Albinus @ 2013-10-21 17:02 UTC (permalink / raw) To: Stefan Monnier; +Cc: 9803 Stefan Monnier <monnier@iro.umontreal.ca> writes: >> I've tried to change it as proposed by Stefan, but I'm too stupid to >> manage all this sophisticated cl-* stuff :-( > > What have you tried and how did it fail? I've tried a kind of `letf'; this failed because it replaces only function calls of functions which exist already. Does not work, because `skip-if' shall not exist. Then I did traverse `remaining' in `ert--parse-keys-and-body', replacing all `skip-if' occurences in the car's by `ert--skip-if'. This did work somehow, but only if `skip-if' is called on the top-level of the body of an ert test. Parsing the whole structure might be possible, but I don't know a respective function/macro, and writing an own recursive parsing seems to expensive to me. > Stefan Best regards, Michael. ^ permalink raw reply [flat|nested] 17+ messages in thread
* bug#9803: [PATCH] Add ERT option to skip test 2013-10-21 17:02 ` Michael Albinus @ 2013-10-21 19:23 ` Stefan Monnier 2013-10-22 8:23 ` Michael Albinus 0 siblings, 1 reply; 17+ messages in thread From: Stefan Monnier @ 2013-10-21 19:23 UTC (permalink / raw) To: Michael Albinus; +Cc: 9803 > I've tried a kind of `letf'; this failed because it replaces only > function calls of functions which exist already. Does not work, because > `skip-if' shall not exist. You want to use either `cl-flet' or `cl-macrolet', depending on whether you want to define skip-if as a local function or a local macro. Stefan ^ permalink raw reply [flat|nested] 17+ messages in thread
* bug#9803: [PATCH] Add ERT option to skip test 2013-10-21 19:23 ` Stefan Monnier @ 2013-10-22 8:23 ` Michael Albinus 2013-10-23 12:21 ` Michael Albinus 0 siblings, 1 reply; 17+ messages in thread From: Michael Albinus @ 2013-10-22 8:23 UTC (permalink / raw) To: Stefan Monnier; +Cc: 9803 Stefan Monnier <monnier@iro.umontreal.ca> writes: > You want to use either `cl-flet' or `cl-macrolet', depending on whether > you want to define skip-if as a local function or a local macro. `cl-macrolet' does it, thanks! I will also handle `should', `should-not' and `should-error' this way. Meanwhile I'm thinking, we should introduce `skip-unless' instead of `skip-if'. This would allow to catch errors, like this: (skip-unless (ignore-errors (a-test-which-might-raise-a-signal))) We could even move `ignore-errors' into `skip-unless'. Opinions? > Stefan Best regards, Michael. ^ permalink raw reply [flat|nested] 17+ messages in thread
* bug#9803: [PATCH] Add ERT option to skip test 2013-10-22 8:23 ` Michael Albinus @ 2013-10-23 12:21 ` Michael Albinus 2013-10-24 7:46 ` Michael Albinus 0 siblings, 1 reply; 17+ messages in thread From: Michael Albinus @ 2013-10-23 12:21 UTC (permalink / raw) To: Stefan Monnier, 9803 Michael Albinus <michael.albinus@gmx.de> writes: > Stefan Monnier <monnier@iro.umontreal.ca> writes: > >> You want to use either `cl-flet' or `cl-macrolet', depending on whether >> you want to define skip-if as a local function or a local macro. > > `cl-macrolet' does it, thanks! I will also handle `should', `should-not' > and `should-error' this way. Well, it works only for tests which use `should', `should-not' and `should-error' in the body of `ert-deftest'. Unfortunately, there are several tests which use helper functions, calling `should' and friends. For those tests it doesn't work this way. Since this is not related to the `skip-unless' check, I will commit just a patch introducing this macro. Changing the scope of `should', `should-not' and `should-error' could be fixed later. >> Stefan Best regards, Michael. ^ permalink raw reply [flat|nested] 17+ messages in thread
* bug#9803: [PATCH] Add ERT option to skip test 2013-10-23 12:21 ` Michael Albinus @ 2013-10-24 7:46 ` Michael Albinus 2013-10-24 8:00 ` Glenn Morris 0 siblings, 1 reply; 17+ messages in thread From: Michael Albinus @ 2013-10-24 7:46 UTC (permalink / raw) To: 9803 Hi, I've committed the final patch to the trunk. Best regards, Michael. ^ permalink raw reply [flat|nested] 17+ messages in thread
* bug#9803: [PATCH] Add ERT option to skip test 2013-10-24 7:46 ` Michael Albinus @ 2013-10-24 8:00 ` Glenn Morris 0 siblings, 0 replies; 17+ messages in thread From: Glenn Morris @ 2013-10-24 8:00 UTC (permalink / raw) To: Michael Albinus; +Cc: 9803 Michael Albinus wrote: > I've committed the final patch to the trunk. Fantastic, thank you. ^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2013-10-24 8:00 UTC | newest] Thread overview: 17+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2011-10-20 3:42 bug#9803: Add ERT option to skip test Glenn Morris 2013-07-04 18:40 ` Michael Albinus 2013-10-18 13:37 ` bug#9803: [PATCH] " Michael Albinus 2013-10-19 1:02 ` Glenn Morris 2013-10-19 2:12 ` Stefan Monnier 2013-10-19 6:44 ` Michael Albinus 2013-10-19 6:44 ` Michael Albinus 2013-10-20 2:02 ` Glenn Morris 2013-10-20 14:09 ` Michael Albinus 2013-10-21 15:08 ` Michael Albinus 2013-10-21 16:53 ` Stefan Monnier 2013-10-21 17:02 ` Michael Albinus 2013-10-21 19:23 ` Stefan Monnier 2013-10-22 8:23 ` Michael Albinus 2013-10-23 12:21 ` Michael Albinus 2013-10-24 7:46 ` Michael Albinus 2013-10-24 8:00 ` Glenn Morris
Code repositories for project(s) associated with this public inbox https://git.savannah.gnu.org/cgit/emacs.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).