bug#47320: Improve failure reporting in test/lisp/electrict-tests.el

[Michael Albinus <michael.albinus@gmx.de>]

Alan Mackenzie <acm@muc.de> writes:

> Hello, Michael.

Hi Alan,

>> There is the explainer functionality in ert, see
>
>> (info "(ert)Defining Explanation Functions")
>
> That description is rather terse.  In fact it is incomplete - it does
> not say when the explanation function gets called, nor does it say what
> is done with any resulting explanation.

100% d'accord. The ert manual isn't as helpful as it should ...

> It looks like a explanation function needs to duplicate the test code of
> the test being explained - it seems the function has no access to the
> internal state of the test.

No, it has full access to the test via the ert machinery.

(ert-get-test SYMBOL) returns the test object which has been created via
(ert-deftest SYMBOL-NAME ...)

(ert-running-test) returns the test object of the test just running.

With that test object, you have different access functions, like
ert-test-name, ert-test-documentation, ert-test-most-recent-result; see
cl-defstruct ert-test. And there are also some other defstructs, which
give you convenience functions like ert-test-failed-p.

> I only wish to print the extra information when there is a test failure.
> Otherwise the information in electric-tests.log would be swamped by
> pointless voluminous doc strings.

The ert explainer writes only something for failed functions.

> Or am I missing something?

<shameless advertisement>I've spent hours and hours to understand these
ert functions. You might have a look on filenotify-tests.el or
tramp-tests.el, for example.</>

>> I recommend to use it, for better readability of the test code. Examples
>> are in our test files.
>
> Thanks!

If needed, I could help you in setting upt these explainers. But I must
confess, I have no idea about electrict-tests.el.

For the beginning, you might look on this:

--8<---------------cut here---------------start------------->8---
(defun test-function ()
  "The result."
  t)

(defun test-function-explainer ()
  "The explainer, a string."
  (format "%s: %s" (ert-test-name (ert-running-test)) (ert-test-documentation (ert-running-test))))

(ert-deftest first-test ()
  "Just the first test"
  (should (test-function)))

(ert-deftest second-test ()
  "Just the second test"
  (should-not (test-function)))

(put 'test-function 'ert-explainer
     'test-function-explainer)
--8<---------------cut here---------------end--------------->8---

If you run both tests, you get

--8<---------------cut here---------------start------------->8---
.F

. first-test
    Just the first test
    passed

F second-test
    Just the second test
    (ert-test-failed
     ((should-not
       (test-function))
      :form
      (test-function)
      :value t :explanation "second-test: Just the second test"))
--8<---------------cut here---------------end--------------->8---

Best regards, Michael.