bug#22471: ert batch should print compile parsable error messages

bug#22471: ert batch should print compile parsable error messages

[Phillip Lord]

Currently, when running in batch, ert prints out messages like so:


Running 24 tests (2016-01-27 09:05:17+0000)
   passed   1/24  buffer-string=
   passed   2/24  buffer=
...
   passed  22/24  sisyphus-test-with-find-file
   passed  23/24  to-string
   passed  24/24  with-temp-buffers

Ran 24 tests, 23 results as expected, 1 unexpected (2016-01-27 09:05:18+0000)

1 unexpected results:
   FAILED  crash-out

It would be nice to add some compilation-mode parsable data to this. So,
something like


   passed   1/24  buffer-string=  (in ./test.sisyphus-test.el:22:)

It would also be nice to add colourisation: red and green for passed and
failed. In the idea world, these could be added as ansi colour codes to
the batch output of ERT. They should work on the command line then also.

More of an RFE than a bug, but it is odd that compilation mode works
with lots and lots of test frameworks, but not Emacs'

Phil

bug#22471: ert batch should print compile parsable error messages

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Date: Wed, 27 Jan 2016 09:15:44 +0000
> 
> 
> Currently, when running in batch, ert prints out messages like so:
> 
> 
> Running 24 tests (2016-01-27 09:05:17+0000)
>    passed   1/24  buffer-string=
>    passed   2/24  buffer=
> ...
>    passed  22/24  sisyphus-test-with-find-file
>    passed  23/24  to-string
>    passed  24/24  with-temp-buffers
> 
> Ran 24 tests, 23 results as expected, 1 unexpected (2016-01-27 09:05:18+0000)
> 
> 1 unexpected results:
>    FAILED  crash-out
> 
> It would be nice to add some compilation-mode parsable data to this. So,
> something like
> 
> 
>    passed   1/24  buffer-string=  (in ./test.sisyphus-test.el:22:)

FWIW, I have never seen a test suite that displayed in
compilation-mode compatible format.  If we do that (and I'm
necessarily opposed to that), we'd be the odd one out, I think.

bug#22471: ert batch should print compile parsable error messages

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: phillip.lord@russet.org.uk (Phillip Lord)
>> Date: Wed, 27 Jan 2016 09:15:44 +0000
>> 
>> 
>> Currently, when running in batch, ert prints out messages like so:
>> 
>> 
>> Running 24 tests (2016-01-27 09:05:17+0000)
>>    passed   1/24  buffer-string=
>>    passed   2/24  buffer=
>> ...
>>    passed  22/24  sisyphus-test-with-find-file
>>    passed  23/24  to-string
>>    passed  24/24  with-temp-buffers
>> 
>> Ran 24 tests, 23 results as expected, 1 unexpected (2016-01-27 09:05:18+0000)
>> 
>> 1 unexpected results:
>>    FAILED  crash-out
>> 
>> It would be nice to add some compilation-mode parsable data to this. So,
>> something like
>> 
>> 
>>    passed   1/24  buffer-string=  (in ./test.sisyphus-test.el:22:)
>
> FWIW, I have never seen a test suite that displayed in
> compilation-mode compatible format.  If we do that (and I'm
> necessarily opposed to that), we'd be the odd one out, I think.

There's a couple in the compilation mode regexp -- cucumber, ruby and
perl test harnesses and so forth.

I put a quick attempt at this up on feature/parsable-ert-output.

   passed  22/24  sisyphus-test-with-find-file at test/sisyphus-test.el line 277.
   passed  23/24  to-string at test/sisyphus-test.el line 96.
   passed  24/24  with-temp-buffers at test/sisyphus-test.el line 232.

Works, but makes for rather a cluttered output.

Thoughts welcome.

Phil

bug#22471: ert batch should print compile parsable error messages

[Glenn Morris]

Why would you call the batch version of ert from within Emacs,
when M-x ert exists and does colour, hyperlinking to source, etc?

bug#22471: ert batch should print compile parsable error messages

[Phillip Lord]

Glenn Morris <rgm@gnu.org> writes:

> Why would you call the batch version of ert from within Emacs,
> when M-x ert exists and does colour, hyperlinking to source, etc?

M-x ert is nice, but has one main failure. It's not running in a clean,
controllable Emacs; it's running in my Emacs. I use M-x ert only when I
am debugging failing tests.

The rest of the time, I run tests in a clean, newly launched Emacs with
a sandboxed .emacs.d/elpa. As a secondary advantage, it can be easier to
test two or three Emacs versions at the same time.

I've been writing a library to help get around this problem, but
fundamentally, the only clean way to test code is not to run it in the
same Emacs as is launching it. At some point, I'd like to add the
ability to launch an emacs for every test.

Phil

bug#22471: ert batch should print compile parsable error messages

[Lars Ingebrigtsen]

phillip.lord@russet.org.uk (Phillip Lord) writes:

> Currently, when running in batch, ert prints out messages like so:
>
> Running 24 tests (2016-01-27 09:05:17+0000)
>    passed   1/24  buffer-string=
>    passed   2/24  buffer=
> ...
>    passed  22/24  sisyphus-test-with-find-file
>    passed  23/24  to-string
>    passed  24/24  with-temp-buffers
>
> Ran 24 tests, 23 results as expected, 1 unexpected (2016-01-27 09:05:18+0000)
>
> 1 unexpected results:
>    FAILED  crash-out
>
> It would be nice to add some compilation-mode parsable data to this. So,
> something like
>
>    passed   1/24  buffer-string=  (in ./test.sisyphus-test.el:22:)

I think that would look rather cluttered for the "passed" lines, but it
would indeed be helpful on the FAILED lines.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#22471: ert batch should print compile parsable error messages

[Philipp Stephani]

Am Do., 20. Aug. 2020 um 18:41 Uhr schrieb Lars Ingebrigtsen <larsi@gnus.org>:
>
> phillip.lord@russet.org.uk (Phillip Lord) writes:
>
> > Currently, when running in batch, ert prints out messages like so:
> >
> > Running 24 tests (2016-01-27 09:05:17+0000)
> >    passed   1/24  buffer-string=
> >    passed   2/24  buffer=
> > ...
> >    passed  22/24  sisyphus-test-with-find-file
> >    passed  23/24  to-string
> >    passed  24/24  with-temp-buffers
> >
> > Ran 24 tests, 23 results as expected, 1 unexpected (2016-01-27 09:05:18+0000)
> >
> > 1 unexpected results:
> >    FAILED  crash-out
> >
> > It would be nice to add some compilation-mode parsable data to this. So,
> > something like
> >
> >    passed   1/24  buffer-string=  (in ./test.sisyphus-test.el:22:)
>
> I think that would look rather cluttered for the "passed" lines, but it
> would indeed be helpful on the FAILED lines.
>

I'm doing something similar in my Bazel ruleset for Elisp
(https://github.com/phst/rules_elisp/blob/0b24aa1660af2f6c668899bdd78aaba383d7ac18/elisp/ert/runner.el#L446).
It would be nice to have an API for this as my code needs a few hacks
and subtleties.

bug#22471: ert batch should print compile parsable error messages

[Stefan Kangas]

[-- Attachment #1: Type: text/plain, Size: 1795 bytes --]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> phillip.lord@russet.org.uk (Phillip Lord) writes:
>
>> Currently, when running in batch, ert prints out messages like so:
>>
>> Running 24 tests (2016-01-27 09:05:17+0000)
>>    passed   1/24  buffer-string=
>>    passed   2/24  buffer=
>> ...
>>    passed  22/24  sisyphus-test-with-find-file
>>    passed  23/24  to-string
>>    passed  24/24  with-temp-buffers
>>
>> Ran 24 tests, 23 results as expected, 1 unexpected (2016-01-27 09:05:18+0000)
>>
>> 1 unexpected results:
>>    FAILED  crash-out
>>
>> It would be nice to add some compilation-mode parsable data to this. So,
>> something like
>>
>>    passed   1/24  buffer-string=  (in ./test.sisyphus-test.el:22:)
>
> I think that would look rather cluttered for the "passed" lines, but it
> would indeed be helpful on the FAILED lines.

This would in my opinion be very useful and a great timesaver.  But yes,
for FAILED results only, much like how the compiler only gives us line
numbers if there is a problem.  (AFAIR, that's how the test frameworks
I've used do it.)

So I updated the patch from the feature/parsable-ert-output against
current master (patch attached), made sure that one of our tests failed,
and then tested it with "make check".

I got the following backtrace, but gave up after looking at it for a
while:

Debugger entered--Lisp error: (error "Don’t know where
‘fns-tests-string-bytes’ is defin...")
  signal(error ("Don’t know where ‘fns-tests-string-bytes’ is defin..."))
  error("Don't know where `%s' is defined" fns-tests-string-bytes)
  find-function-search-for-symbol(fns-tests-string-bytes ert-deftest nil)
  find-definition-noselect(fns-tests-string-bytes ert-deftest)
  ert-test-location([...])
[...]

[-- Attachment #2: 0001-Add-source-information-to-ERT-batch-mode.patch --]
[-- Type: text/x-diff, Size: 2569 bytes --]

From e6c185f280e2c330825beebad7d2393c21199564 Mon Sep 17 00:00:00 2001
From: Phillip Lord <phillip.lord@russet.org.uk>
Date: Wed, 27 Jan 2016 20:59:50 +0000
Subject: [PATCH] Add source information to ERT batch mode

* lisp/emacs-lisp/ert.el (ert-test-location): New function.
(ert-run-tests-batch): Add test location to failed tests.  (Bug#22471)
---
 lisp/emacs-lisp/ert.el | 24 ++++++++++++++++++++++--
 1 file changed, 22 insertions(+), 2 deletions(-)

diff --git a/lisp/emacs-lisp/ert.el b/lisp/emacs-lisp/ert.el
index 57655403c2..36f5062e83 100644
--- a/lisp/emacs-lisp/ert.el
+++ b/lisp/emacs-lisp/ert.el
@@ -1353,6 +1353,23 @@ ert--insert-infos
 (defvar ert-quiet nil
   "Non-nil makes ERT only print important information in batch mode.")
 
+(defun ert-test-location (test)
+  "Return a string description the source location of TEST."
+  (let* ((loc
+          (find-definition-noselect (ert-test-name test) 'ert-deftest))
+         (buffer
+          (car loc))
+         (point (cdr loc))
+         (file
+          (file-relative-name
+           (buffer-file-name buffer)))
+         (line (with-current-buffer buffer
+                 (line-number-at-pos point))))
+    (format "at %s line %s." file line)))
+
+(defvar ert-batch-backtrace-right-margin 70
+  "The maximum line length for printing backtraces in `ert-run-tests-batch'.")
+
 ;;;###autoload
 (defun ert-run-tests-batch (&optional selector)
   "Run the tests specified by SELECTOR, printing results to the terminal.
@@ -1471,14 +1488,17 @@ ert-run-tests-batch
             (let* ((max (prin1-to-string (length (ert--stats-tests stats))))
                    (format-string (concat "%9s  %"
                                           (prin1-to-string (length max))
-                                          "s/" max "  %S (%f sec)")))
+                                          "s/" max "  %S (%f sec)%s")))
               (message format-string
                        (ert-string-for-test-result result
                                                    (ert-test-result-expected-p
                                                     test result))
                        (1+ (ert--stats-test-pos stats test))
                        (ert-test-name test)
-                       (ert-test-result-duration result))))))))
+                       (ert-test-result-duration result)
+                       (if (ert-test-result-expected-p test result)
+                           ""
+                         (concat " " (ert-test-location test))))))))))
    nil))
 
 ;;;###autoload
-- 
2.30.2

bug#22471: ert batch should print compile parsable error messages

[Lars Ingebrigtsen]

Stefan Kangas <stefan@marxist.se> writes:

> I got the following backtrace, but gave up after looking at it for a
> while:
>
> Debugger entered--Lisp error: (error "Don’t know where
> ‘fns-tests-string-bytes’ is defin...")
>   signal(error ("Don’t know where ‘fns-tests-string-bytes’ is defin..."))

Well -- ert-deftest doesn't record what file it's from, I think?
Looking at the definition, it just adds the test to the symbol-plist.
So that macro would also have to stash the file name somewhere.
load-history would be possible (but sounds unclean), so stashing it in
the symbol-plist would be better.

Then find-definition-noselect could be extended to use that for this
type.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#22471: ert batch should print compile parsable error messages

[Stefan Kangas]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Well -- ert-deftest doesn't record what file it's from, I think?
> Looking at the definition, it just adds the test to the symbol-plist.
> So that macro would also have to stash the file name somewhere.
> load-history would be possible (but sounds unclean), so stashing it in
> the symbol-plist would be better.
>
> Then find-definition-noselect could be extended to use that for this
> type.

Right, I'm just wondering what has changed as presumably the patch
worked when Phillip Lord first wrote it?

bug#22471: ert batch should print compile parsable error messages

[Lars Ingebrigtsen]

Stefan Kangas <stefan@marxist.se> writes:

> Right, I'm just wondering what has changed as presumably the patch
> worked when Phillip Lord first wrote it?

Oh, I interpreted the branch as a sketch towards a solution but I see
that he posted apparent output from it, too:

> I put a quick attempt at this up on feature/parsable-ert-output.
> 
>    passed 22/24 sisyphus-test-with-find-file at test/sisyphus-test.el
> line 277.
>    passed  23/24  to-string at test/sisyphus-test.el line 96.
>    passed  24/24  with-temp-buffers at test/sisyphus-test.el line 232.

So it worked at one point?  (I'm not able to compile that branch at
all -- it segfaults while bootstrapping.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#22471: ert batch should print compile parsable error messages

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

>> Right, I'm just wondering what has changed as presumably the patch
>> worked when Phillip Lord first wrote it?
>
> Oh, I interpreted the branch as a sketch towards a solution but I see
> that he posted apparent output from it, too:

I've now fixed up the patch and am pushing to Emacs 29.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no