bug#22471: ert batch should print compile parsable error messages

[Stefan Kangas <stefan@marxist.se>]

[-- Attachment #1: Type: text/plain, Size: 1795 bytes --]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> phillip.lord@russet.org.uk (Phillip Lord) writes:
>
>> Currently, when running in batch, ert prints out messages like so:
>>
>> Running 24 tests (2016-01-27 09:05:17+0000)
>>    passed   1/24  buffer-string=
>>    passed   2/24  buffer=
>> ...
>>    passed  22/24  sisyphus-test-with-find-file
>>    passed  23/24  to-string
>>    passed  24/24  with-temp-buffers
>>
>> Ran 24 tests, 23 results as expected, 1 unexpected (2016-01-27 09:05:18+0000)
>>
>> 1 unexpected results:
>>    FAILED  crash-out
>>
>> It would be nice to add some compilation-mode parsable data to this. So,
>> something like
>>
>>    passed   1/24  buffer-string=  (in ./test.sisyphus-test.el:22:)
>
> I think that would look rather cluttered for the "passed" lines, but it
> would indeed be helpful on the FAILED lines.

This would in my opinion be very useful and a great timesaver.  But yes,
for FAILED results only, much like how the compiler only gives us line
numbers if there is a problem.  (AFAIR, that's how the test frameworks
I've used do it.)

So I updated the patch from the feature/parsable-ert-output against
current master (patch attached), made sure that one of our tests failed,
and then tested it with "make check".

I got the following backtrace, but gave up after looking at it for a
while:

Debugger entered--Lisp error: (error "Don’t know where
‘fns-tests-string-bytes’ is defin...")
  signal(error ("Don’t know where ‘fns-tests-string-bytes’ is defin..."))
  error("Don't know where `%s' is defined" fns-tests-string-bytes)
  find-function-search-for-symbol(fns-tests-string-bytes ert-deftest nil)
  find-definition-noselect(fns-tests-string-bytes ert-deftest)
  ert-test-location([...])
[...]

[-- Attachment #2: 0001-Add-source-information-to-ERT-batch-mode.patch --]
[-- Type: text/x-diff, Size: 2569 bytes --]

From e6c185f280e2c330825beebad7d2393c21199564 Mon Sep 17 00:00:00 2001
From: Phillip Lord <phillip.lord@russet.org.uk>
Date: Wed, 27 Jan 2016 20:59:50 +0000
Subject: [PATCH] Add source information to ERT batch mode

* lisp/emacs-lisp/ert.el (ert-test-location): New function.
(ert-run-tests-batch): Add test location to failed tests.  (Bug#22471)
---
 lisp/emacs-lisp/ert.el | 24 ++++++++++++++++++++++--
 1 file changed, 22 insertions(+), 2 deletions(-)

diff --git a/lisp/emacs-lisp/ert.el b/lisp/emacs-lisp/ert.el
index 57655403c2..36f5062e83 100644
--- a/lisp/emacs-lisp/ert.el
+++ b/lisp/emacs-lisp/ert.el
@@ -1353,6 +1353,23 @@ ert--insert-infos
 (defvar ert-quiet nil
   "Non-nil makes ERT only print important information in batch mode.")
 
+(defun ert-test-location (test)
+  "Return a string description the source location of TEST."
+  (let* ((loc
+          (find-definition-noselect (ert-test-name test) 'ert-deftest))
+         (buffer
+          (car loc))
+         (point (cdr loc))
+         (file
+          (file-relative-name
+           (buffer-file-name buffer)))
+         (line (with-current-buffer buffer
+                 (line-number-at-pos point))))
+    (format "at %s line %s." file line)))
+
+(defvar ert-batch-backtrace-right-margin 70
+  "The maximum line length for printing backtraces in `ert-run-tests-batch'.")
+
 ;;;###autoload
 (defun ert-run-tests-batch (&optional selector)
   "Run the tests specified by SELECTOR, printing results to the terminal.
@@ -1471,14 +1488,17 @@ ert-run-tests-batch
             (let* ((max (prin1-to-string (length (ert--stats-tests stats))))
                    (format-string (concat "%9s  %"
                                           (prin1-to-string (length max))
-                                          "s/" max "  %S (%f sec)")))
+                                          "s/" max "  %S (%f sec)%s")))
               (message format-string
                        (ert-string-for-test-result result
                                                    (ert-test-result-expected-p
                                                     test result))
                        (1+ (ert--stats-test-pos stats test))
                        (ert-test-name test)
-                       (ert-test-result-duration result))))))))
+                       (ert-test-result-duration result)
+                       (if (ert-test-result-expected-p test result)
+                           ""
+                         (concat " " (ert-test-location test))))))))))
    nil))
 
 ;;;###autoload
-- 
2.30.2