bug#51037: [PATCH] Make `print-level` & `print-length` customizable in ERT batch tests

bug#51037: [PATCH] Make `print-level` & `print-length` customizable in ERT batch tests

[Michael]

[-- Attachment #1: Type: text/plain, Size: 556 bytes --]

Hello,

When running ERT tests in batch mode, the conservative values
chosen for `print-level` and `print-length` sometimes make it
difficult to see what exactly is wrong.  This patch introduces
two new variables (`ert-batch-print-level` &
`ert-batch-print-length`) that one can use to customize them;
e.g.

    emacs -batch -l ert -l my-tests.el \
          --eval "(let ((ert-batch-print-level 10) \
                        (ert-batch-print-length 120)) \
                    (ert-run-tests-batch-and-exit))"

Please let me know what should be changed.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Make-print-length-print-level-in-ert-run-tests-batch.patch --]
[-- Type: text/x-patch, Size: 12702 bytes --]

From da7eff94f23f4798df36be2c2bd072038ba01e2d Mon Sep 17 00:00:00 2001
From: Michael Herstine <sp1ff@pobox.com>
Date: Tue, 5 Oct 2021 06:53:02 -0700
Subject: [PATCH] Make print-length & print-level in ert-run-tests-batch
 configurable

This commit introduces two new ert variables (ert-batch-print-length
and ert-batch-print-level) that make these settings configurable.  It
also adds an optional message-fn parameter to ert-batch-test (in the
style of of ert-run-tests-interactively) to facilitate testing.

* lisp/emacs-lisp/ert.el (ert-batch-print-length, ert-batch-print-level,
ert-batch-test): Added the two variables, added optional message-fn
parameter, set print-level & print-length to these settings when
formatting test results.
* test/lisp/emacs-lisp/ert-tests.el (ert-test-run-tests-batch): new
tests
* doc/misc/ert.texi: document the new variables & their usage
---
 doc/misc/ert.texi                 | 16 ++++++++
 etc/NEWS                          |  5 +++
 lisp/emacs-lisp/ert.el            | 61 +++++++++++++++++++------------
 test/lisp/emacs-lisp/ert-tests.el | 34 +++++++++++++++++
 4 files changed, 93 insertions(+), 23 deletions(-)

diff --git a/doc/misc/ert.texi b/doc/misc/ert.texi
index fafdb8c4eb4..12f1df75fcc 100644
--- a/doc/misc/ert.texi
+++ b/doc/misc/ert.texi
@@ -323,6 +323,22 @@ Running Tests in Batch Mode
 emacs -batch -l ert -f ert-summarize-tests-batch-and-exit output.log
 @end example
 
+@vindex ert-batch-print-level
+@vindex ert-batch-print-length
+ERT attempts to limit the output size for failed tests by choosing
+conservative values for @code{print-level} & @code{print-length}
+when printing Lisp values.  This can in some cases make it difficult
+to see which portions of those values are incorrect.  Use
+@code{ert-batch-print-level} and @code{ert-batch-print-length}
+to customize that:
+
+@example
+emacs -batch -l ert -l my-tests.el \
+      --eval "(let ((ert-batch-print-level 10) \
+                    (ert-batch-print-length 120)) \
+                (ert-run-tests-batch-and-exit))"
+@end example
+
 @vindex ert-quiet
 By default, ERT in batch mode is quite verbose, printing a line with
 result after each test.  This gives you progress information: how many
diff --git a/etc/NEWS b/etc/NEWS
index 17c42ce104d..be102dbc501 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -128,6 +128,11 @@ of files visited via 'C-x C-f' and other commands.
 \f
 * Changes in Emacs 28.1
 
++++
+** New ERT batch variables 'ert-batch-print-length' & 'ert-batch-print-level'
+These variables will override 'print-length' & 'print-level' when
+printing Lisp values in ERT batch test results.
+
 ---
 ** Emacs now supports Unicode Standard version 14.0.
 
diff --git a/lisp/emacs-lisp/ert.el b/lisp/emacs-lisp/ert.el
index 92acfe7246f..2f0b5536d7e 100644
--- a/lisp/emacs-lisp/ert.el
+++ b/lisp/emacs-lisp/ert.el
@@ -76,6 +76,18 @@ ert-batch-backtrace-right-margin
 Use nil for no limit (caution: backtrace lines can be very long)."
   :type '(choice (const :tag "No truncation" nil) integer))
 
+(defvar ert-batch-print-length 10
+  "`print-length' setting used in `ert-run-tests-batch'.
+
+When formatting lists in test results, `print-length' will be
+temporarily set to this value.")
+
+(defvar ert-batch-print-level 5
+  "`print-level' setting used in `ert-run-tests-batch'.
+
+When formatting lists in test results, `print-level' will be
+temporarily set to this value.")
+
 (defface ert-test-result-expected '((((class color) (background light))
                                      :background "green1")
                                     (((class color) (background dark))
@@ -1338,7 +1350,7 @@ ert-quiet
   "Non-nil makes ERT only print important information in batch mode.")
 
 ;;;###autoload
-(defun ert-run-tests-batch (&optional selector)
+(defun ert-run-tests-batch (&optional selector message-fn)
   "Run the tests specified by SELECTOR, printing results to the terminal.
 
 SELECTOR works as described in `ert-select-tests', except if
@@ -1346,8 +1358,12 @@ ert-run-tests-batch
 run; this makes the command line \"emacs -batch -l my-tests.el -f
 ert-run-tests-batch-and-exit\" useful.
 
-Returns the stats object."
+Returns the stats object.
+
+MESSAGE-FN should normally be nil; it is used for automated
+self-tests and specify how to display messages."
   (unless selector (setq selector 't))
+  (unless message-fn (setq message-fn #'message))
   (ert-run-tests
    selector
    (lambda (event-type &rest event-args)
@@ -1355,7 +1371,7 @@ ert-run-tests-batch
        (run-started
         (unless ert-quiet
           (cl-destructuring-bind (stats) event-args
-            (message "Running %s tests (%s, selector `%S')"
+            (funcall message-fn "Running %s tests (%s, selector `%S')"
                      (length (ert--stats-tests stats))
                      (ert--format-time-iso8601 (ert--stats-start-time stats))
                      selector))))
@@ -1364,7 +1380,7 @@ ert-run-tests-batch
           (let ((unexpected (ert-stats-completed-unexpected stats))
                 (skipped (ert-stats-skipped stats))
 		(expected-failures (ert--stats-failed-expected stats)))
-            (message "\n%sRan %s tests, %s results as expected, %s unexpected%s (%s, %f sec)%s\n"
+            (funcall message-fn "\n%sRan %s tests, %s results as expected, %s unexpected%s (%s, %f sec)%s\n"
                      (if (not abortedp)
                          ""
                        "Aborted: ")
@@ -1383,44 +1399,43 @@ ert-run-tests-batch
                          ""
                        (format "\n%s expected failures" expected-failures)))
             (unless (zerop unexpected)
-              (message "%s unexpected results:" unexpected)
+              (funcall message-fn "%s unexpected results:" unexpected)
               (cl-loop for test across (ert--stats-tests stats)
                        for result = (ert-test-most-recent-result test) do
                        (when (not (ert-test-result-expected-p test result))
-                         (message "%9s  %S%s"
+                         (funcall message-fn "%9s  %S%s"
                                   (ert-string-for-test-result result nil)
                                   (ert-test-name test)
                                   (if (getenv "EMACS_TEST_VERBOSE")
                                       (ert-reason-for-test-result result)
                                     ""))))
-              (message "%s" ""))
+              (funcall message-fn "%s" ""))
             (unless (zerop skipped)
-              (message "%s skipped results:" skipped)
+              (funcall message-fn "%s skipped results:" skipped)
               (cl-loop for test across (ert--stats-tests stats)
                        for result = (ert-test-most-recent-result test) do
                        (when (ert-test-result-type-p result :skipped)
-                         (message "%9s  %S%s"
+                         (funcall message-fn "%9s  %S%s"
                                   (ert-string-for-test-result result nil)
                                   (ert-test-name test)
                                   (if (getenv "EMACS_TEST_VERBOSE")
                                       (ert-reason-for-test-result result)
                                     ""))))
-              (message "%s" "")))))
-       (test-started
-        )
+              (funcall message-fn "%s" "")))))
+       (test-started)
        (test-ended
         (cl-destructuring-bind (stats test result) event-args
           (unless (ert-test-result-expected-p test result)
             (cl-etypecase result
               (ert-test-passed
-               (message "Test %S passed unexpectedly" (ert-test-name test)))
+               (funcall message-fn "Test %S passed unexpectedly" (ert-test-name test)))
               (ert-test-result-with-condition
-               (message "Test %S backtrace:" (ert-test-name test))
+               (funcall message-fn "Test %S backtrace:" (ert-test-name test))
                (with-temp-buffer
                  (insert (backtrace-to-string
                           (ert-test-result-with-condition-backtrace result)))
                  (if (not ert-batch-backtrace-right-margin)
-                     (message "%s"
+                     (funcall message-fn "%s"
                               (buffer-substring-no-properties (point-min)
                                                               (point-max)))
                    (goto-char (point-min))
@@ -1430,33 +1445,33 @@ ert-run-tests-batch
                        (setq end (min end
                                       (+ start
                                          ert-batch-backtrace-right-margin)))
-                       (message "%s" (buffer-substring-no-properties
+                       (funcall message-fn "%s" (buffer-substring-no-properties
                                       start end)))
                      (forward-line 1))))
                (with-temp-buffer
                  (ert--insert-infos result)
                  (insert "    ")
                  (let ((print-escape-newlines t)
-                       (print-level 5)
-                       (print-length 10))
+                       (print-level ert-batch-print-level)
+                       (print-length ert-batch-print-length))
                    (ert--pp-with-indentation-and-newline
                     (ert-test-result-with-condition-condition result)))
                  (goto-char (1- (point-max)))
                  (cl-assert (looking-at "\n"))
                  (delete-char 1)
-                 (message "Test %S condition:" (ert-test-name test))
-                 (message "%s" (buffer-string))))
+                 (funcall message-fn "Test %S condition:" (ert-test-name test))
+                 (funcall message-fn "%s" (buffer-string))))
               (ert-test-aborted-with-non-local-exit
-               (message "Test %S aborted with non-local exit"
+               (funcall message-fn "Test %S aborted with non-local exit"
                         (ert-test-name test)))
               (ert-test-quit
-               (message "Quit during %S" (ert-test-name test)))))
+               (funcall message-fn "Quit during %S" (ert-test-name test)))))
           (unless ert-quiet
             (let* ((max (prin1-to-string (length (ert--stats-tests stats))))
                    (format-string (concat "%9s  %"
                                           (prin1-to-string (length max))
                                           "s/" max "  %S (%f sec)")))
-              (message format-string
+              (funcall message-fn format-string
                        (ert-string-for-test-result result
                                                    (ert-test-result-expected-p
                                                     test result))
diff --git a/test/lisp/emacs-lisp/ert-tests.el b/test/lisp/emacs-lisp/ert-tests.el
index 5c9696105e9..d4872e0bab7 100644
--- a/test/lisp/emacs-lisp/ert-tests.el
+++ b/test/lisp/emacs-lisp/ert-tests.el
@@ -551,6 +551,40 @@ ert-test-run-tests-interactively
             (when (get-buffer buffer-name)
               (kill-buffer buffer-name))))))))
 
+(ert-deftest ert-test-run-tests-batch ()
+  (let* ((complex-list '((:1 (:2 (:3 (:4 (:5 (:6 "abc"))))))))
+	 (long-list (make-list 121 1))
+	 (failing-test-1
+          (make-ert-test :name 'failing-test-1
+			 :body (lambda () (should (equal complex-list 1)))))
+	 (failing-test-2
+          (make-ert-test :name 'failing-test-2
+			 :body (lambda () (should (equal long-list 1))))))
+    (let ((ert-debug-on-error nil))
+      (let* ((messages nil)
+             (mock-message-fn
+              (lambda (format-string &rest args)
+                (push (apply #'format format-string args) messages))))
+        (save-window-excursion
+          (unwind-protect
+              (let ((case-fold-search nil)
+		    (ert-batch-print-level nil)
+		    (ert-batch-print-length nil))
+                (ert-run-tests-batch
+                 `(member ,failing-test-1 ,failing-test-2)
+                 mock-message-fn))))
+	(let ((long-text (mapconcat #'identity (make-list 121 "1") " "))
+	      (complex-text "(:6 \"abc\")")
+	      found-complex found-long)
+	  (cl-loop for msg in messages
+		   do
+		   (unless found-long
+		     (setq found-long (cl-search long-text msg :test 'equal)))
+		   (unless found-complex
+		     (setq found-complex (cl-search complex-text msg :test 'equal))))
+	  (should found-complex)
+	  (should found-long))))))
+
 (ert-deftest ert-test-special-operator-p ()
   (should (ert--special-operator-p 'if))
   (should-not (ert--special-operator-p 'car))
-- 
2.33.0


[-- Attachment #3: Type: text/plain, Size: 32 bytes --]


-- 
Michael <sp1ff@runbox.com>

bug#51037: [PATCH] Make `print-level` & `print-length` customizable in ERT batch tests

[Lars Ingebrigtsen]

Michael <sp1ff@runbox.com> writes:

> When running ERT tests in batch mode, the conservative values
> chosen for `print-level` and `print-length` sometimes make it
> difficult to see what exactly is wrong.  This patch introduces
> two new variables (`ert-batch-print-level` &
> `ert-batch-print-length`) that one can use to customize them;
> e.g.
>
>    emacs -batch -l ert -l my-tests.el \
>          --eval "(let ((ert-batch-print-level 10) \
>                        (ert-batch-print-length 120)) \
>                    (ert-run-tests-batch-and-exit))"

Sounds like a good idea.

> +MESSAGE-FN should normally be nil; it is used for automated
> +self-tests and specify how to display messages."

I don't understand this bit, though.  If you want to test how this
function outputs messages, you can just `cl-letf' like this:

(cl-letf (((symbol-function 'message) ...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#51037: [PATCH] Make `print-level` & `print-length` customizable in ERT batch tests

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Wed, 06 Oct 2021 11:30:03 +0200
> Cc: 51037@debbugs.gnu.org
> 
> Michael <sp1ff@runbox.com> writes:
> 
> > When running ERT tests in batch mode, the conservative values
> > chosen for `print-level` and `print-length` sometimes make it
> > difficult to see what exactly is wrong.  This patch introduces
> > two new variables (`ert-batch-print-level` &
> > `ert-batch-print-length`) that one can use to customize them;
> > e.g.
> >
> >    emacs -batch -l ert -l my-tests.el \
> >          --eval "(let ((ert-batch-print-level 10) \
> >                        (ert-batch-print-length 120)) \
> >                    (ert-run-tests-batch-and-exit))"
> 
> Sounds like a good idea.

Why do we limit print-length in batch mode?  In interactive session,
one can click or type RET on the ellipsis and get it expanded, but no
such joy in batch.  See

  https://lists.gnu.org/archive/html/emacs-devel/2021-10/msg00395.html

for how that makes debugging more difficult than it has to be.

So how about using huge values in batch?  Are there any downsides?

bug#51037: [PATCH] Make `print-level` & `print-length` customizable in ERT batch tests

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Why do we limit print-length in batch mode?  In interactive session,
> one can click or type RET on the ellipsis and get it expanded, but no
> such joy in batch.  See
>
>   https://lists.gnu.org/archive/html/emacs-devel/2021-10/msg00395.html
>
> for how that makes debugging more difficult than it has to be.
>
> So how about using huge values in batch?  Are there any downsides?

It'll make the error summaries completely unreadable, which isn't what
you want when doing a "make check" -- you just want to see that there's
a problem and where it is.  (Well, at least I do.)

But when making a specific test (i.e., "make subr-tests" etc) it would
indeed be nice to have untruncated backtraces.  I think adding these new
variables would allow us to do this.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#51037: [PATCH] Make `print-level` & `print-length` customizable in ERT batch tests

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: sp1ff@runbox.com,  51037@debbugs.gnu.org
> Date: Thu, 07 Oct 2021 09:59:01 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >   https://lists.gnu.org/archive/html/emacs-devel/2021-10/msg00395.html
> >
> > for how that makes debugging more difficult than it has to be.
> >
> > So how about using huge values in batch?  Are there any downsides?
> 
> It'll make the error summaries completely unreadable

Sorry, I don't think I understand: why would that make them
unreadable?  Text terminals wrap long lines.  Or what am I missing?
Can you perhaps show an example of such an unreadable error summary?

And even if this is something specific to ERT, why not use a huge
value in batch-mode by default, and bind the variables to smaller
values in ERT?

> But when making a specific test (i.e., "make subr-tests" etc) it would
> indeed be nice to have untruncated backtraces.  I think adding these new
> variables would allow us to do this.

OK, but I was thinking about a more general problem, not just about
ERT.  The backtraces displayed in batch-mode are truncated too early,
and that makes them useless many times.

bug#51037: [PATCH] Make `print-level` & `print-length` customizable in ERT batch tests

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Sorry, I don't think I understand: why would that make them
> unreadable?  Text terminals wrap long lines.  Or what am I missing?
> Can you perhaps show an example of such an unreadable error summary?

If you have very long structures, a simple backtrace can take many
screenfuls, which isn't what most people want or expect.

> OK, but I was thinking about a more general problem, not just about
> ERT.  The backtraces displayed in batch-mode are truncated too early,
> and that makes them useless many times.

That's true.  Programmers that need to debug these things should have an
easy way to get complete backtraces, but I don't think users should be
presented with them.  So perhaps we should just introduce a new switch,
like --complete-backtraces (or something) that would do the right thing
both in general and in ert.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#51037: [PATCH] Make `print-level` & `print-length` customizable in ERT batch tests

[Michael]

> If you have very long structures, a simple backtrace can take 
> many
> screenfuls, which isn't what most people want or expect.
>
>> OK, but I was thinking about a more general problem, not just 
>> about
>> ERT.  The backtraces displayed in batch-mode are truncated too 
>> early,
>> and that makes them useless many times.
>
> That's true.  Programmers that need to debug these things should 
> have an
> easy way to get complete backtraces, but I don't think users 
> should be
> presented with them.  So perhaps we should just introduce a new 
> switch,
> like --complete-backtraces (or something) that would do the 
> right thing
> both in general and in ert.

Sorry to chime-in late, but my patch does _not_ affect the
printing of backtraces, just results. That said, I do agree with
Eli about the truncated backtraces often being useless; what do
you think of another variable controlling
that... `ert-batch-full-backtraces` or something?

-- 
Michael <sp1ff@runbox.com>

bug#51037: [PATCH] Make `print-level` & `print-length` customizable in ERT batch tests

[Lars Ingebrigtsen]

Michael <sp1ff@runbox.com> writes:

> Sorry to chime-in late, but my patch does _not_ affect the
> printing of backtraces, just results. That said, I do agree with
> Eli about the truncated backtraces often being useless; what do
> you think of another variable controlling
> that... `ert-batch-full-backtraces` or something?

I'm not sure I quite follow you here -- the original patch would make
the backtraces printed by ert more complete, wouldn't they?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#51037: [PATCH] Make `print-level` & `print-length` customizable in ERT batch tests

[Michael]

Hey Lars,

>> Sorry to chime-in late, but my patch does _not_ affect the
>> printing of backtraces, just results. That said, I do agree 
>> with
>> Eli about the truncated backtraces often being useless; what do
>> you think of another variable controlling
>> that... `ert-batch-full-backtraces` or something?
>
> I'm not sure I quite follow you here -- the original patch would 
> make
> the backtraces printed by ert more complete, wouldn't they?

Only when comparing _results_. Say I have test code like:

    (let ((a (<some complex list>)
          (b (<another complex list>))))
      ...
      (should (equal a b))

My changes would case the error message from the failed `should`
invocation to print `a` & `b` more fully (i.e. with fewwer
ellipses).

The backtrace logic is separate, and un-touched by my patch as
submitted (tho of course I can change that).

-- 
Michael <sp1ff@runbox.com>

bug#51037: [PATCH] Make `print-level` & `print-length` customizable in ERT batch tests

[Lars Ingebrigtsen]

Michael <sp1ff@runbox.com> writes:

> Only when comparing _results_. Say I have test code like:
>
>    (let ((a (<some complex list>)
>          (b (<another complex list>))))
>      ...
>      (should (equal a b))
>
> My changes would case the error message from the failed `should`
> invocation to print `a` & `b` more fully (i.e. with fewwer
> ellipses).

Ah, I see.

> The backtrace logic is separate, and un-touched by my patch as
> submitted (tho of course I can change that).

Perhaps it would be better if the same variables affected both result
printing and backtraces in the code that ert is testing.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#51037: [PATCH] Make `print-level` & `print-length` customizable in ERT batch tests

[Michael]

+Gemini

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Michael <sp1ff@runbox.com> writes:
>
>> Only when comparing _results_. Say I have test code like:
>>
>>    (let ((a (<some complex list>)
>>          (b (<another complex list>))))
>>      ...
>>      (should (equal a b))
>>
>> My changes would case the error message from the failed 
>> `should`
>> invocation to print `a` & `b` more fully (i.e. with fewwer
>> ellipses).
>
> Ah, I see.
>
>> The backtrace logic is separate, and un-touched by my patch as
>> submitted (tho of course I can change that).
>
> Perhaps it would be better if the same variables affected both 
> result
> printing and backtraces in the code that ert is testing.

I've run down a bit of a rabbit hole on this.

The issue: stack traces are printed by the `backtrace`
package. backtrace.el was authored by Gemini (which is why I've
added him to this thread). backtrace.el doesn't directly work in
terms of `print-le{ngth,vel}`: it defines a custom variable
`backtrace-line-length` and then adjusts print-level &
print-length in let bindings in order to:

  1. try to respect the desired line length
  2. not trigger bug 31919 (Lisp Debugger doesn't work when at
  stack limit)

Bug 31919 <https://debbugs.gnu.org/cgi/bugreport.cgi?bug=31919>
occurs when the debugger is invoked in response to
`max-lisp-eval-depth` being exceeded. Since we're close the the
top of the stack already, a too-high setting of `print-level`
will cause backtraces to break.

Because of this arrangement, any attempt on the part of ert to
customize print-length & print-level comes to nothing--
`backtrace` will adjust them to satisfy conditions 1 & 2 above.

For myself, I was quite surprised to all learn this: I had to
spend a fair bit of time digging through the source to find out
what was happening to my settings for print-length &
print-level.

My personal incliniation is to remove the `backtrace-line-length`
variable entirely, and make the `debug` package responsible for
controlling print-level so as to avoid 31919. But that's me: is
there a compelling use-case for backtrace.el working in terms of
limiting line length rather than just using `print-le{ve,ength}`?

-- 
Michael <sp1ff@runbox.com>

bug#51037: [PATCH] Make `print-level` & `print-length` customizable in ERT batch tests

[Gemini Lasswell]

Michael writes:

> My personal incliniation is to remove the `backtrace-line-length`
> variable entirely, and make the `debug` package responsible for
> controlling print-level so as to avoid 31919. But that's me: is
> there a compelling use-case for backtrace.el working in terms of
> limiting line length rather than just using `print-le{ve,ength}`?

The motivation for `backtrace-line-length` was to find a compromise that
made backtraces more user-friendly in interactive debugging for both
small and large data structures.  Setting `print-level` and
`print-length` too small will create a lot of ellipses in small data
structures, making you have to expand those ellipses to see your data;
setting those variables a little bit bigger will unleash exponential
growth in line length for large data structures, causing delays of
seconds to minutes when rendering and navigating your backtrace.  The
heuristics in `cl-print-to-string-with-limit` try to do a reasonable job
with both small and large data structures, so the user can get on with
debugging, rather than first having to figure out values of the `print-`
variables which will make debugging possible.

Another goal of backtrace buffers is to make the data structures
represented there be completely navigable, which is why lines are not
simply truncated.

If you don't want ellipses in your backtraces, then bind
`backtrace-line-length` to nil or zero.  If you want to see how long a
backtrace line can get, try that out with magit or org-export.

bug#51037: [PATCH] Make `print-level` & `print-length` customizable in ERT batch tests

[Michael]

Gemini Lasswell <gazally@runbox.com> writes:

> Michael writes:
>
>> My personal incliniation is to remove the 
>> `backtrace-line-length`
>> variable entirely, and make the `debug` package responsible for
>> controlling print-level so as to avoid 31919. But that's me: is
>> there a compelling use-case for backtrace.el working in terms 
>> of
>> limiting line length rather than just using 
>> `print-le{ve,ength}`?
>
> The motivation for `backtrace-line-length` was to find a 
> compromise that
> made backtraces more user-friendly in interactive debugging for 
> both
> small and large data structures.  Setting `print-level` and
> `print-length` too small will create a lot of ellipses in small 
> data
> structures, making you have to expand those ellipses to see your 
> data;
> setting those variables a little bit bigger will unleash 
> exponential
> growth in line length for large data structures, causing delays 
> of
> seconds to minutes when rendering and navigating your backtrace. 
> The
> heuristics in `cl-print-to-string-with-limit` try to do a 
> reasonable job
> with both small and large data structures, so the user can get 
> on with
> debugging, rather than first having to figure out values of the 
> `print-`
> variables which will make debugging possible.
>
> Another goal of backtrace buffers is to make the data structures
> represented there be completely navigable, which is why lines 
> are not
> simply truncated.
>
> If you don't want ellipses in your backtraces, then bind
> `backtrace-line-length` to nil or zero.  If you want to see how 
> long a
> backtrace line can get, try that out with magit or org-export.

lol... I see. So you found `backtrace-line-length` to be more
ergonomic than print-level & print-length at controlling
verbosity (at least where backtraces are concerned)-- fair?

One thing: when I evaluate this:

    (let ((print-length nil)
          (print-level nil)
          (backtrace-line-length nil))
      (backtrace-to-string))

I receive the following error message:

    backtrace-print-frame: Wrong type argument: 
    number-or-marker-p, nil

I take it that is unexpected? It's fine if so, I'm happy to
debug, I just want to be sure I'm not running off down the garden
path.

-- 
Michael <sp1ff@runbox.com>

bug#51037: [PATCH] Make `print-level` & `print-length` customizable in ERT batch tests

[Lars Ingebrigtsen]

Michael <sp1ff@runbox.com> writes:

> One thing: when I evaluate this:
>
>    (let ((print-length nil)
>          (print-level nil)
>          (backtrace-line-length nil))
>      (backtrace-to-string))
>
> I receive the following error message:
>
>    backtrace-print-frame: Wrong type argument:     number-or-marker-p,
>   nil
>
> I take it that is unexpected? It's fine if so, I'm happy to
> debug, I just want to be sure I'm not running off down the garden
> path.

nil should be a valid value for that variable, so it sounds like a bug.
Oddly enough, I'm not able to get a backtrace here...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#51037: [PATCH] Make `print-level` & `print-length` customizable in ERT batch tests

[Lars Ingebrigtsen]

Michael <sp1ff@runbox.com> writes:

> The issue: stack traces are printed by the `backtrace`
> package. backtrace.el was authored by Gemini (which is why I've
> added him to this thread). backtrace.el doesn't directly work in
> terms of `print-le{ngth,vel}`: it defines a custom variable
> `backtrace-line-length` and then adjusts print-level &
> print-length in let bindings in order to:
>
>  1. try to respect the desired line length
>  2. not trigger bug 31919 (Lisp Debugger doesn't work when at
>  stack limit)

[...]

> For myself, I was quite surprised to all learn this: I had to
> spend a fair bit of time digging through the source to find out
> what was happening to my settings for print-length &
> print-level.

Yeah, I didn't remember `backtrace-line-length' either.  Perhaps it
should be linked to from the doc strings of print-length and
print-level... 

> My personal incliniation is to remove the `backtrace-line-length`
> variable entirely, and make the `debug` package responsible for
> controlling print-level so as to avoid 31919. But that's me: is
> there a compelling use-case for backtrace.el working in terms of
> limiting line length rather than just using `print-le{ve,ength}`?

Well...  if we're talking in an ert context, it could increase both the
backtrace-line-length and max-lisp-eval-depth variables when gathering
the backtrace, I think?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#51037: [PATCH] Make `print-level` & `print-length` customizable in ERT batch tests

[Michael]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Michael <sp1ff@runbox.com> writes:
>
>> The issue: stack traces are printed by the `backtrace`
>> package. backtrace.el was authored by Gemini (which is why I've
>> added him to this thread). backtrace.el doesn't directly work 
>> in
>> terms of `print-le{ngth,vel}`: it defines a custom variable
>> `backtrace-line-length` and then adjusts print-level &
>> print-length in let bindings in order to:
>>
>>  1. try to respect the desired line length
>>  2. not trigger bug 31919 (Lisp Debugger doesn't work when at
>>  stack limit)
>
> [...]
>
>> For myself, I was quite surprised to all learn this: I had to
>> spend a fair bit of time digging through the source to find out
>> what was happening to my settings for print-length &
>> print-level.
>
> Yeah, I didn't remember `backtrace-line-length' either.  Perhaps 
> it
> should be linked to from the doc strings of print-length and
> print-level... 

I was thinking to augment the docstring for
`backtrace-line-lenght' to note that print-level & print-length
will be overiden to honor it.

>> My personal incliniation is to remove the 
>> `backtrace-line-length`
>> variable entirely, and make the `debug` package responsible for
>> controlling print-level so as to avoid 31919. But that's me: is
>> there a compelling use-case for backtrace.el working in terms 
>> of
>> limiting line length rather than just using 
>> `print-le{ve,ength}`?
>
> Well...  if we're talking in an ert context, it could increase 
> both the
> backtrace-line-length and max-lisp-eval-depth variables when 
> gathering
> the backtrace, I think?

The problem is that I can't know a priori how big I have to set
them in order to honor print-level & print-length. I think I can
short-circuit this entirely by setting `backtrace-line-length' to
nil or zero before printing the backtrace from ert. eval-level
shouldn't enter into it I think; the concern was that if code
exceeded max-lisp-eval-depth, and the debugger was triggered,
*then* backtrace.el might have problems. Do you see a reason to
be concerned outside of that corner case?

-- 
Michael <sp1ff@runbox.com>

bug#51037: [PATCH] Make `print-level` & `print-length` customizable in ERT batch tests

[Lars Ingebrigtsen]

Michael <sp1ff@runbox.com> writes:

> I was thinking to augment the docstring for
> `backtrace-line-lenght' to note that print-level & print-length
> will be overiden to honor it.

Sounds good.

> The problem is that I can't know a priori how big I have to set
> them in order to honor print-level & print-length.

No, but I thought the issue was with the corner case where we're almost
up against the limit already?  To work around that, doubling the limit
is fine in this context.

> I think I can short-circuit this entirely by setting
> `backtrace-line-length' to nil or zero before printing the backtrace
> from ert. eval-level shouldn't enter into it I think; the concern was
> that if code exceeded max-lisp-eval-depth, and the debugger was
> triggered, *then* backtrace.el might have problems. Do you see a
> reason to be concerned outside of that corner case?

If that works without upping the limit, then that's fine.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#51037: [PATCH] Make `print-level` & `print-length` customizable in ERT batch tests

[Michael]

[-- Attachment #1: Type: text/plain, Size: 1702 bytes --]


Lars Ingebrigtsen <larsi@gnus.org> writes:

> Michael <sp1ff@runbox.com> writes:
>
>> I was thinking to augment the docstring for
>> `backtrace-line-lenght' to note that print-level & print-length
>> will be overiden to honor it.
>
> Sounds good.
>
>> The problem is that I can't know a priori how big I have to set
>> them in order to honor print-level & print-length.
>
> No, but I thought the issue was with the corner case where we're 
> almost
> up against the limit already?  To work around that, doubling the 
> limit
> is fine in this context.
>
>> I think I can short-circuit this entirely by setting
>> `backtrace-line-length' to nil or zero before printing the 
>> backtrace
>> from ert. eval-level shouldn't enter into it I think; the 
>> concern was
>> that if code exceeded max-lisp-eval-depth, and the debugger was
>> triggered, *then* backtrace.el might have problems. Do you see 
>> a
>> reason to be concerned outside of that corner case?
>
> If that works without upping the limit, then that's fine.

OK-- here's the revised patch.

Gemini was on to something-- even modest settings for print-level
& print-length can produce *extremely* long lines in backtraces,
with corresponding execution pauses. So I:

  - fixed the minor error in backtrace.el so that setting
    backtrace-length to nil has the desired effect (i.e. removing
    any line length limitation)
  - introduced a third variable ert-batch-backtrace-line-length,
    with the following permissible values:
    - nil: remove any limitation on backtrace line length (full
    backtraces)
    - t: just use the value of backtrace-line-length
    - positive int: use this value to limit line lengths in
    backtraces



[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Updated patch for ert-batch-print-length & -level --]
[-- Type: text/x-patch, Size: 22327 bytes --]

From e256f6ff60547bb08e17f99f7eb5e9f08607ea17 Mon Sep 17 00:00:00 2001
From: Michael Herstine <sp1ff@pobox.com>
Date: Tue, 5 Oct 2021 06:53:02 -0700
Subject: [PATCH] Make print-length & print-level in ert-run-tests-batch
 configurable

This commit introduces two new ert variables (ert-batch-print-length
and ert-batch-print-level) that make these settings configurable.  It
also adds an optional message-fn parameter to ert-batch-test (in the
style of of ert-run-tests-interactively) to facilitate testing.

Since even modest values of print-level & print-length can produce
extremely long lines in backtraces along with correspondingly long
processing times to format them, this commit also introduces the
variable ert-backtrace-line-length which, can optionally, override
backtrace-line-length during ert batch test stack traces.

* lisp/emacs-lisp/ert.el (ert-batch-print-length, ert-batch-print-level,
ert-batch-backtrace-line-length, ert-batch-test,
ert-run-tests-interactively): Added the three variables, bound them to
these settings when formatting batch test results including backtraces.
Removed the optional parameters output-buffer & message-fn from
ert-run-tests-interactively.
* test/lisp/emacs-lisp/ert-tests.el (ert-test-run-tests-interactively,
ert-test-run-tests-batch): use cl-letf to capture output, new tests
resp.
* test/lisp/ert-x-tests.el (ert-test-run-tests-interactively-2):
Changed to use cl-letf to capture output instead of using message-fn.
* lisp/emacs-lisp/backtrace.el (backtrace--line-length-or-nil,
backtrace--print-func-and-args): Fixed a bug when setting
backtrace-line-length to nil by adding a new function to check
for that case & having backtrace--print-func-and-args use it.
* doc/misc/ert.texi: document the new variables & their usage
---
 doc/misc/ert.texi                   | 25 +++++++++
 etc/NEWS                            |  8 +++
 lisp/emacs-lisp/backtrace.el        | 21 +++++---
 lisp/emacs-lisp/ert.el              | 82 +++++++++++++++++++----------
 test/lisp/emacs-lisp/ert-tests.el   | 81 ++++++++++++++++++++++++----
 test/lisp/emacs-lisp/ert-x-tests.el | 41 +++++++--------
 6 files changed, 193 insertions(+), 65 deletions(-)

diff --git a/doc/misc/ert.texi b/doc/misc/ert.texi
index 440c61add8e..522b09dd617 100644
--- a/doc/misc/ert.texi
+++ b/doc/misc/ert.texi
@@ -390,6 +390,31 @@ Running Tests in Batch Mode
 emacs -batch -l ert -f ert-summarize-tests-batch-and-exit output.log
 @end example
 
+@vindex ert-batch-print-level
+@vindex ert-batch-print-length
+ERT attempts to limit the output size for failed tests by choosing
+conservative values for @code{print-level} & @code{print-length}
+when printing Lisp values.  This can in some cases make it difficult
+to see which portions of those values are incorrect.  Use
+@code{ert-batch-print-level} and @code{ert-batch-print-length}
+to customize that:
+
+@example
+emacs -batch -l ert -l my-tests.el \
+      --eval "(let ((ert-batch-print-level 10) \
+                    (ert-batch-print-length 120)) \
+                (ert-run-tests-batch-and-exit))"
+@end example
+
+@vindex ert-batch-backtrace-line-length
+Even modest settings for @code{print-level} & @code{print-length} can
+produce extremely long lines in backtraces, however, with attendant
+pauses in execution progress. Set
+@code{ert-batch-backtrace-line-length} to t to use the value of
+@code{backtrace-line-length}, nil to stop any limitations on backtrace
+line lengths (i.e. to get full backtraces), or a positive integer to
+limit backtrace line length to that number.
+
 @vindex ert-quiet
 By default, ERT in batch mode is quite verbose, printing a line with
 result after each test.  This gives you progress information: how many
diff --git a/etc/NEWS b/etc/NEWS
index 3cad0995ac5..b315fa5be2b 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -46,6 +46,14 @@ time.
 \f
 * Changes in Emacs 29.1
 
++++
+** New ERT batch variables 'ert-batch-print-length' & 'ert-batch-print-level'
+These variables will override 'print-length' & 'print-level' when
+printing Lisp values in ERT batch test results.
+
+** Emacs now supports Unicode Standard version 14.0.
+>>>>>>> 129b2425c79 (Make print-length & print-level in ert-run-tests-batch configurable)
+
 ** Emoji
 
 +++
diff --git a/lisp/emacs-lisp/backtrace.el b/lisp/emacs-lisp/backtrace.el
index a5721aa3193..90201442f08 100644
--- a/lisp/emacs-lisp/backtrace.el
+++ b/lisp/emacs-lisp/backtrace.el
@@ -55,9 +55,9 @@ backtrace-fontify
 (defcustom backtrace-line-length 5000
   "Target length for lines in Backtrace buffers.
 Backtrace mode will attempt to abbreviate printing of backtrace
-frames to make them shorter than this, but success is not
-guaranteed.  If set to nil or zero, Backtrace mode will not
-abbreviate the forms it prints."
+frames by setting `print-level' & `print-length' to make them
+shorter than this, but success is not guaranteed.  If set to nil
+or zero, Backtrace mode will not abbreviate the forms it prints."
   :type 'integer
   :group 'backtrace
   :version "27.1")
@@ -751,6 +751,13 @@ backtrace--print-flags
     (insert (make-string (- backtrace--flags-width (- (point) beg)) ?\s))
     (put-text-property beg (point) 'backtrace-section 'func)))
 
+(defun backtrace--line-length-or-nil ()
+  "Return `backtrace-line-length' if valid, nil else."
+  ;; mirror the logic in `cl-print-to-string-with-limits'
+  (and (natnump backtrace-line-length)
+       (not (zerop backtrace-line-length))
+       backtrace-line-length))
+
 (defun backtrace--print-func-and-args (frame _view)
   "Print the function, arguments and buffer position of a backtrace FRAME.
 Format it according to VIEW."
@@ -769,11 +776,13 @@ backtrace--print-func-and-args
       (if (atom fun)
           (funcall backtrace-print-function fun)
         (insert
-         (backtrace--print-to-string fun (when args (/ backtrace-line-length 2)))))
+         (backtrace--print-to-string fun (when (and args (backtrace--line-length-or-nil)) (/ backtrace-line-length 2)))))
       (if args
           (insert (backtrace--print-to-string
-                   args (max (truncate (/ backtrace-line-length 5))
-                             (- backtrace-line-length (- (point) beg)))))
+                   args
+                   (if (backtrace--line-length-or-nil)
+                       (max (truncate (/ backtrace-line-length 5))
+                            (- backtrace-line-length (- (point) beg))))))
         ;; The backtrace-form property is so that backtrace-multi-line
         ;; will find it.  backtrace-multi-line doesn't do anything
         ;; useful with it, just being consistent.
diff --git a/lisp/emacs-lisp/ert.el b/lisp/emacs-lisp/ert.el
index 8ebc81fd418..9659c55cb64 100644
--- a/lisp/emacs-lisp/ert.el
+++ b/lisp/emacs-lisp/ert.el
@@ -77,6 +77,37 @@ ert-batch-backtrace-right-margin
 Use nil for no limit (caution: backtrace lines can be very long)."
   :type '(choice (const :tag "No truncation" nil) integer))
 
+(defvar ert-batch-print-length 10
+  "`print-length' setting used in `ert-run-tests-batch'.
+
+When formatting lists in test conditions, `print-length' will be
+temporarily set to this value.  See also
+`ert-batch-backtrace-line-length' for its effect on stack
+traces.")
+
+(defvar ert-batch-print-level 5
+  "`print-level' setting used in `ert-run-tests-batch'.
+
+When formatting lists in test conditions, `print-level' will be
+temporarily set to this value.  See also
+`ert-batch-backtrace-line-length' for its effect on stack
+traces.")
+
+(defvar ert-batch-backtrace-line-length t
+  "Target length for lines in ERT batch backtraces.
+
+Even modest settings for `print-length' & `print-level' can
+produce extremely long lines in backtraces & lengthy delays in
+forming them.  This variable governs the target maximum line
+length by manipulating these two variables while printing stack
+traces.  Setting this variable to t will re-use the value of
+`backtrace-line-length' while print stack traces in ERT batch
+mode.  A value of nil will short-circuit this mechanism; line
+lengths will be completely determined by `ert-batch-line-length'
+& `ert-batch-line-level'.  Any other value will be temporarily
+bound to `backtrace-line-length' when producing stack traces
+in batch mode.")
+
 (defface ert-test-result-expected '((((class color) (background light))
                                      :background "green1")
                                     (((class color) (background dark))
@@ -1402,8 +1433,7 @@ ert-run-tests-batch
                                       (ert-reason-for-test-result result)
                                     ""))))
               (message "%s" "")))))
-       (test-started
-        )
+       (test-started)
        (test-ended
         (cl-destructuring-bind (stats test result) event-args
           (unless (ert-test-result-expected-p test result)
@@ -1413,8 +1443,15 @@ ert-run-tests-batch
               (ert-test-result-with-condition
                (message "Test %S backtrace:" (ert-test-name test))
                (with-temp-buffer
-                 (insert (backtrace-to-string
-                          (ert-test-result-with-condition-backtrace result)))
+                 (let ((backtrace-line-length
+                        (cond
+                         ((eq ert-batch-backtrace-line-length t) backtrace-line-length)
+                         ((eq ert-batch-backtrace-line-length nil) nil)
+                         (t ert-batch-backtrace-line-length)))
+                       (print-level ert-batch-print-level)
+                       (print-length ert-batch-print-length))
+                   (insert (backtrace-to-string
+                            (ert-test-result-with-condition-backtrace result))))
                  (if (not ert-batch-backtrace-right-margin)
                      (message "%s"
                               (buffer-substring-no-properties (point-min)
@@ -1433,8 +1470,8 @@ ert-run-tests-batch
                  (ert--insert-infos result)
                  (insert "    ")
                  (let ((print-escape-newlines t)
-                       (print-level 5)
-                       (print-length 10))
+                       (print-level ert-batch-print-level)
+                       (print-length ert-batch-print-length))
                    (ert--pp-with-indentation-and-newline
                     (ert-test-result-with-condition-condition result)))
                  (goto-char (1- (point-max)))
@@ -1962,13 +1999,13 @@ ert--results-font-lock-function
   (ewoc-refresh ert--results-ewoc)
   (font-lock-default-function enabledp))
 
-(defun ert--setup-results-buffer (stats listener buffer-name)
+(defvar ert--output-buffer-name "*ert*")
+
+(defun ert--setup-results-buffer (stats listener)
   "Set up a test results buffer.
 
-STATS is the stats object; LISTENER is the results listener;
-BUFFER-NAME, if non-nil, is the buffer name to use."
-  (unless buffer-name (setq buffer-name "*ert*"))
-  (let ((buffer (get-buffer-create buffer-name)))
+STATS is the stats object; LISTENER is the results listener."
+  (let ((buffer (get-buffer-create ert--output-buffer-name)))
     (with-current-buffer buffer
       (let ((inhibit-read-only t))
         (buffer-disable-undo)
@@ -2000,18 +2037,11 @@ ert--setup-results-buffer
 (defvar ert--selector-history nil
   "List of recent test selectors read from terminal.")
 
-;; Should OUTPUT-BUFFER-NAME and MESSAGE-FN really be arguments here?
-;; They are needed only for our automated self-tests at the moment.
-;; Or should there be some other mechanism?
 ;;;###autoload
-(defun ert-run-tests-interactively (selector
-                                    &optional output-buffer-name message-fn)
+(defun ert-run-tests-interactively (selector)
   "Run the tests specified by SELECTOR and display the results in a buffer.
 
-SELECTOR works as described in `ert-select-tests'.
-OUTPUT-BUFFER-NAME and MESSAGE-FN should normally be nil; they
-are used for automated self-tests and specify which buffer to use
-and how to display message."
+SELECTOR works as described in `ert-select-tests'."
   (interactive
    (list (let ((default (if ert--selector-history
                             ;; Can't use `first' here as this form is
@@ -2024,23 +2054,17 @@ ert-run-tests-interactively
                              obarray #'ert-test-boundp nil nil
                              'ert--selector-history default nil)))
          nil))
-  (unless message-fn (setq message-fn 'message))
-  (let ((output-buffer-name output-buffer-name)
-        buffer
-        listener
-        (message-fn message-fn))
+  (let (buffer listener)
     (setq listener
           (lambda (event-type &rest event-args)
             (cl-ecase event-type
               (run-started
                (cl-destructuring-bind (stats) event-args
-                 (setq buffer (ert--setup-results-buffer stats
-                                                         listener
-                                                         output-buffer-name))
+                 (setq buffer (ert--setup-results-buffer stats listener))
                  (pop-to-buffer buffer)))
               (run-ended
                (cl-destructuring-bind (stats abortedp) event-args
-                 (funcall message-fn
+                 (message
                           "%sRan %s tests, %s results were as expected%s%s"
                           (if (not abortedp)
                               ""
diff --git a/test/lisp/emacs-lisp/ert-tests.el b/test/lisp/emacs-lisp/ert-tests.el
index 79576d24032..f783532aaff 100644
--- a/test/lisp/emacs-lisp/ert-tests.el
+++ b/test/lisp/emacs-lisp/ert-tests.el
@@ -39,10 +39,11 @@ ert-test-body-runs
 (defun ert-self-test ()
   "Run ERT's self-tests and make sure they actually ran."
   (let ((window-configuration (current-window-configuration)))
-    (let ((ert--test-body-was-run nil))
+    (let ((ert--test-body-was-run nil)
+          (ert--output-buffer-name " *ert self-tests*"))
       ;; The buffer name chosen here should not compete with the default
       ;; results buffer name for completion in `switch-to-buffer'.
-      (let ((stats (ert-run-tests-interactively "^ert-" " *ert self-tests*")))
+      (let ((stats (ert-run-tests-interactively "^ert-")))
         (cl-assert ert--test-body-was-run)
         (if (zerop (ert-stats-completed-unexpected stats))
             ;; Hide results window only when everything went well.
@@ -519,17 +520,17 @@ ert-test-run-tests-interactively
                                      :body (lambda () (ert-skip
                                                        "skip message")))))
     (let ((ert-debug-on-error nil))
-      (let* ((buffer-name (generate-new-buffer-name " *ert-test-run-tests*"))
-             (messages nil)
-             (mock-message-fn
-              (lambda (format-string &rest args)
-                (push (apply #'format format-string args) messages))))
+      (cl-letf* ((buffer-name (generate-new-buffer-name " *ert-test-run-tests*"))
+                 (ert--output-buffer-name buffer-name)
+                 (messages nil)
+                 ((symbol-function 'message)
+                  (lambda (format-string &rest args)
+                    (push (apply #'format format-string args) messages))))
         (save-window-excursion
           (unwind-protect
               (let ((case-fold-search nil))
                 (ert-run-tests-interactively
-                 `(member ,passing-test ,failing-test, skipped-test) buffer-name
-                 mock-message-fn)
+                 `(member ,passing-test ,failing-test, skipped-test))
                 (should (equal messages `(,(concat
                                             "Ran 3 tests, 1 results were "
                                             "as expected, 1 unexpected, "
@@ -551,6 +552,68 @@ ert-test-run-tests-interactively
             (when (get-buffer buffer-name)
               (kill-buffer buffer-name))))))))
 
+(ert-deftest ert-test-run-tests-batch ()
+  (let* ((complex-list '((:1 (:2 (:3 (:4 (:5 (:6 "abc"))))))))
+	 (long-list (make-list 11 1))
+	 (failing-test-1
+          (make-ert-test :name 'failing-test-1
+			 :body (lambda () (should (equal complex-list 1)))))
+	 (failing-test-2
+          (make-ert-test :name 'failing-test-2
+			 :body (lambda () (should (equal long-list 1))))))
+    (let ((ert-debug-on-error nil)
+          messages)
+      (cl-letf* (((symbol-function 'message)
+                  (lambda (format-string &rest args)
+                    (push (apply #'format format-string args) messages))))
+        (save-window-excursion
+          (unwind-protect
+              (let ((case-fold-search nil)
+                    (ert-batch-backtrace-right-margin nil)
+		    (ert-batch-print-level 10)
+		    (ert-batch-print-length 11))
+                (ert-run-tests-batch
+                 `(member ,failing-test-1 ,failing-test-2))))))
+      (let ((long-text "(different-types[ \t\n]+(1 1 1 1 1 1 1 1 1 1 1)[ \t\n]+1)))[ \t\n]*$")
+	    (complex-text "(different-types[ \t\n]+((:1[ \t\n]+(:2[ \t\n]+(:3[ \t\n]+(:4[ \t\n]+(:5[ \t\n]+(:6[ \t\n]+\"abc\")))))))[ \t\n]+1)))[ \t\n]*$")
+            found-long
+	    found-complex)
+	(cl-loop for msg in (reverse messages)
+		 do
+		 (unless found-long
+		   (setq found-long (string-match long-text msg)))
+		 (unless found-complex
+		   (setq found-complex (string-match complex-text msg))))
+	(should found-long)
+	(should found-complex)))))
+
+(ert-deftest ert-test-run-tests-batch-expensive ()
+  (let* ((complex-list '((:1 (:2 (:3 (:4 (:5 (:6 "abc"))))))))
+	 (failing-test-1
+          (make-ert-test :name 'failing-test-1
+			 :body (lambda () (should (equal complex-list 1))))))
+    (let ((ert-debug-on-error nil)
+          messages)
+      (cl-letf* (((symbol-function 'message)
+                  (lambda (format-string &rest args)
+                    (push (apply #'format format-string args) messages))))
+        (save-window-excursion
+          (unwind-protect
+              (let ((case-fold-search nil)
+                    (ert-batch-backtrace-right-margin nil)
+                    (ert-batch-backtrace-line-length nil)
+		    (ert-batch-print-level 6)
+		    (ert-batch-print-length 11))
+                (ert-run-tests-batch
+                 `(member ,failing-test-1))))))
+      (let ((frame "ert-fail(((should (equal complex-list 1)) :form (equal ((:1 (:2 (:3 (:4 (:5 (:6 \"abc\"))))))) 1) :value nil :explanation (different-types ((:1 (:2 (:3 (:4 (:5 (:6 \"abc\"))))))) 1)))")
+            found-frame)
+	(cl-loop for msg in (reverse messages)
+		 do
+		 (unless found-frame
+		   (setq found-frame (cl-search frame msg :test 'equal))))
+        (should found-frame)))))
+
 (ert-deftest ert-test-special-operator-p ()
   (should (ert--special-operator-p 'if))
   (should-not (ert--special-operator-p 'car))
diff --git a/test/lisp/emacs-lisp/ert-x-tests.el b/test/lisp/emacs-lisp/ert-x-tests.el
index 9baa9941586..1001525dee9 100644
--- a/test/lisp/emacs-lisp/ert-x-tests.el
+++ b/test/lisp/emacs-lisp/ert-x-tests.el
@@ -103,23 +103,24 @@ ert-propertized-string
 
 (ert-deftest ert-test-run-tests-interactively-2 ()
   :tags '(:causes-redisplay)
-  (let* ((passing-test (make-ert-test :name 'passing-test
-                                      :body (lambda () (ert-pass))))
-         (failing-test (make-ert-test :name 'failing-test
-                                      :body (lambda ()
-                                              (ert-info ((propertize "foo\nbar"
-                                                                     'a 'b))
-                                                (ert-fail
-                                                 "failure message")))))
-         (skipped-test (make-ert-test :name 'skipped-test
-                                      :body (lambda () (ert-skip
-							"skip message"))))
-         (ert-debug-on-error nil)
-         (buffer-name (generate-new-buffer-name "*ert-test-run-tests*"))
-         (messages nil)
-         (mock-message-fn
-          (lambda (format-string &rest args)
-            (push (apply #'format format-string args) messages))))
+  (cl-letf* ((passing-test (make-ert-test :name 'passing-test
+                                          :body (lambda () (ert-pass))))
+             (failing-test (make-ert-test :name 'failing-test
+                                          :body (lambda ()
+                                                  (ert-info ((propertize "foo\nbar"
+                                                                         'a 'b))
+                                                            (ert-fail
+                                                             "failure message")))))
+             (skipped-test (make-ert-test :name 'skipped-test
+                                          :body (lambda () (ert-skip
+							    "skip message"))))
+             (ert-debug-on-error nil)
+             (messages nil)
+             (buffer-name (generate-new-buffer-name "*ert-test-run-tests*"))
+             ((symbol-function 'message)
+              (lambda (format-string &rest args)
+                (push (apply #'format format-string args) messages)))
+             (ert--output-buffer-name buffer-name))
     (cl-flet ((expected-string (with-font-lock-p)
                 (ert-propertized-string
                  "Selector: (member <passing-test> <failing-test> "
@@ -152,14 +153,12 @@ ert-test-run-tests-interactively-2
                  "failing-test"
                  nil "\n    Info: " '(a b) "foo\n"
                  nil "          " '(a b) "bar"
-                 nil "\n    (ert-test-failed \"failure message\")\n\n\n"
-                 )))
+                 nil "\n    (ert-test-failed \"failure message\")\n\n\n")))
       (save-window-excursion
         (unwind-protect
             (let ((case-fold-search nil))
               (ert-run-tests-interactively
-               `(member ,passing-test ,failing-test ,skipped-test) buffer-name
-               mock-message-fn)
+               `(member ,passing-test ,failing-test ,skipped-test))
               (should (equal messages `(,(concat
                                           "Ran 3 tests, 1 results were "
                                           "as expected, 1 unexpected, "
-- 
2.33.1


[-- Attachment #3: Type: text/plain, Size: 31 bytes --]

-- 
Michael <sp1ff@runbox.com>

bug#51037: [PATCH] Make `print-level` & `print-length` customizable in ERT batch tests

[Eli Zaretskii]

> From: Michael <sp1ff@runbox.com>
> Date: Sat, 13 Nov 2021 18:55:08 -0800
> Cc: gazally@runbox.com, 51037@debbugs.gnu.org
> 
> OK-- here's the revised patch.

Thanks.

The patch includes some remnant of a merge conflict:

> ++++
> +** New ERT batch variables 'ert-batch-print-length' & 'ert-batch-print-level'
> +These variables will override 'print-length' & 'print-level' when
> +printing Lisp values in ERT batch test results.
> +
> +** Emacs now supports Unicode Standard version 14.0.
> +>>>>>>> 129b2425c79 (Make print-length & print-level in ert-run-tests-batch configurable)

Also, please try to keep the lines in the log message shorter than 65
characters.

bug#51037: [PATCH] Make `print-level` & `print-length` customizable in ERT batch tests

[Michael]

[-- Attachment #1: Type: text/plain, Size: 796 bytes --]


>> From: Michael <sp1ff@runbox.com>
>> Date: Sat, 13 Nov 2021 18:55:08 -0800
>> Cc: gazally@runbox.com, 51037@debbugs.gnu.org
>> 
>> OK-- here's the revised patch.
>
> Thanks.
>
> The patch includes some remnant of a merge conflict:
>
>> ++++
>> +** New ERT batch variables 'ert-batch-print-length' & 
>> 'ert-batch-print-level'
>> +These variables will override 'print-length' & 'print-level' 
>> when
>> +printing Lisp values in ERT batch test results.
>> +
>> +** Emacs now supports Unicode Standard version 14.0.
>> +>>>>>>> 129b2425c79 (Make print-length & print-level in 
>> ert-run-tests-batch configurable)
>
> Also, please try to keep the lines in the log message shorter 
> than 65
> characters.

That's embarassing. Both fixed, and commit message & NEWS entry
tidied up, altogether.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Make `print-level` & `print-length` configurable in ERT batch tests --]
[-- Type: text/x-patch, Size: 22489 bytes --]

From 71ba49d5c8822430025e2921d0d4376d9962ae8a Mon Sep 17 00:00:00 2001
From: Michael Herstine <sp1ff@pobox.com>
Date: Tue, 5 Oct 2021 06:53:02 -0700
Subject: [PATCH] Make results details in ert-run-tests-batch configurable.

This commit introduces three new ert variables:
ert-batch-print-length, ert-batch-print-level &
ert-batch-backtrace-length. The first two control print-length
and print-level, resp., when printing test results.

Since even modest values of print-level & print-length can
produce extremely long lines in backtraces along with
correspondingly long processing times to format them, this commit
also introduces the variable ert-backtrace-line-length which can,
optionally, override backtrace-line-length during ert batch test
stack traces.

Finally, it also removes the optional message-fn & output-buffer
arguments to ert-run-test-interactively (since they were only
used for testing purposes) and re-implements its tests in
terms of cl-letf.

* lisp/emacs-lisp/ert.el (ert-batch-print-length,
ert-batch-print-level,.ert-batch-backtrace-line-length,
ert-batch-test, ert-run-tests-interactively): Added the three
variables, bound them to these settings when formatting batch
test results including backtraces. Removed the optional
parameters output-buffer & message-fn from
ert-run-tests-interactively.
* test/lisp/emacs-lisp/ert-tests.el
(ert-test-run-tests-interactively, ert-test-run-tests-batch): use
cl-letf to capture output, new tests resp.
* test/lisp/ert-x-tests.el (ert-test-run-tests-interactively-2):
Changed to use cl-letf to capture output instead of using
message-fn.
* lisp/emacs-lisp/backtrace.el (backtrace--line-length-or-nil,
backtrace--print-func-and-args): Fixed a bug when setting
backtrace-line-length to nil by adding a new function to check
for that case & having backtrace--print-func-and-args use it.
* doc/misc/ert.texi: document the new variables & their usage
---
 doc/misc/ert.texi                   | 25 +++++++++
 etc/NEWS                            | 11 ++++
 lisp/emacs-lisp/backtrace.el        | 21 +++++---
 lisp/emacs-lisp/ert.el              | 82 +++++++++++++++++++----------
 test/lisp/emacs-lisp/ert-tests.el   | 81 ++++++++++++++++++++++++----
 test/lisp/emacs-lisp/ert-x-tests.el | 41 +++++++--------
 6 files changed, 196 insertions(+), 65 deletions(-)

diff --git a/doc/misc/ert.texi b/doc/misc/ert.texi
index 440c61add8e..522b09dd617 100644
--- a/doc/misc/ert.texi
+++ b/doc/misc/ert.texi
@@ -390,6 +390,31 @@ Running Tests in Batch Mode
 emacs -batch -l ert -f ert-summarize-tests-batch-and-exit output.log
 @end example
 
+@vindex ert-batch-print-level
+@vindex ert-batch-print-length
+ERT attempts to limit the output size for failed tests by choosing
+conservative values for @code{print-level} & @code{print-length}
+when printing Lisp values.  This can in some cases make it difficult
+to see which portions of those values are incorrect.  Use
+@code{ert-batch-print-level} and @code{ert-batch-print-length}
+to customize that:
+
+@example
+emacs -batch -l ert -l my-tests.el \
+      --eval "(let ((ert-batch-print-level 10) \
+                    (ert-batch-print-length 120)) \
+                (ert-run-tests-batch-and-exit))"
+@end example
+
+@vindex ert-batch-backtrace-line-length
+Even modest settings for @code{print-level} & @code{print-length} can
+produce extremely long lines in backtraces, however, with attendant
+pauses in execution progress. Set
+@code{ert-batch-backtrace-line-length} to t to use the value of
+@code{backtrace-line-length}, nil to stop any limitations on backtrace
+line lengths (i.e. to get full backtraces), or a positive integer to
+limit backtrace line length to that number.
+
 @vindex ert-quiet
 By default, ERT in batch mode is quite verbose, printing a line with
 result after each test.  This gives you progress information: how many
diff --git a/etc/NEWS b/etc/NEWS
index 3cad0995ac5..286f01f6864 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -46,6 +46,17 @@ time.
 \f
 * Changes in Emacs 29.1
 
++++
+** New ERT batch variables
+
+Add three variables 'ert-batch-print-length', 'ert-batch-print-level'
+& 'ert-batch-backtrace-line-length'.  The first two will override
+'print-length' & 'print-level' when printing Lisp values in ERT
+batch test results.  The third controls line length when
+printing stack traces.
+
+** Emacs now supports Unicode Standard version 14.0.
+
 ** Emoji
 
 +++
diff --git a/lisp/emacs-lisp/backtrace.el b/lisp/emacs-lisp/backtrace.el
index a5721aa3193..90201442f08 100644
--- a/lisp/emacs-lisp/backtrace.el
+++ b/lisp/emacs-lisp/backtrace.el
@@ -55,9 +55,9 @@ backtrace-fontify
 (defcustom backtrace-line-length 5000
   "Target length for lines in Backtrace buffers.
 Backtrace mode will attempt to abbreviate printing of backtrace
-frames to make them shorter than this, but success is not
-guaranteed.  If set to nil or zero, Backtrace mode will not
-abbreviate the forms it prints."
+frames by setting `print-level' & `print-length' to make them
+shorter than this, but success is not guaranteed.  If set to nil
+or zero, Backtrace mode will not abbreviate the forms it prints."
   :type 'integer
   :group 'backtrace
   :version "27.1")
@@ -751,6 +751,13 @@ backtrace--print-flags
     (insert (make-string (- backtrace--flags-width (- (point) beg)) ?\s))
     (put-text-property beg (point) 'backtrace-section 'func)))
 
+(defun backtrace--line-length-or-nil ()
+  "Return `backtrace-line-length' if valid, nil else."
+  ;; mirror the logic in `cl-print-to-string-with-limits'
+  (and (natnump backtrace-line-length)
+       (not (zerop backtrace-line-length))
+       backtrace-line-length))
+
 (defun backtrace--print-func-and-args (frame _view)
   "Print the function, arguments and buffer position of a backtrace FRAME.
 Format it according to VIEW."
@@ -769,11 +776,13 @@ backtrace--print-func-and-args
       (if (atom fun)
           (funcall backtrace-print-function fun)
         (insert
-         (backtrace--print-to-string fun (when args (/ backtrace-line-length 2)))))
+         (backtrace--print-to-string fun (when (and args (backtrace--line-length-or-nil)) (/ backtrace-line-length 2)))))
       (if args
           (insert (backtrace--print-to-string
-                   args (max (truncate (/ backtrace-line-length 5))
-                             (- backtrace-line-length (- (point) beg)))))
+                   args
+                   (if (backtrace--line-length-or-nil)
+                       (max (truncate (/ backtrace-line-length 5))
+                            (- backtrace-line-length (- (point) beg))))))
         ;; The backtrace-form property is so that backtrace-multi-line
         ;; will find it.  backtrace-multi-line doesn't do anything
         ;; useful with it, just being consistent.
diff --git a/lisp/emacs-lisp/ert.el b/lisp/emacs-lisp/ert.el
index 8ebc81fd418..9659c55cb64 100644
--- a/lisp/emacs-lisp/ert.el
+++ b/lisp/emacs-lisp/ert.el
@@ -77,6 +77,37 @@ ert-batch-backtrace-right-margin
 Use nil for no limit (caution: backtrace lines can be very long)."
   :type '(choice (const :tag "No truncation" nil) integer))
 
+(defvar ert-batch-print-length 10
+  "`print-length' setting used in `ert-run-tests-batch'.
+
+When formatting lists in test conditions, `print-length' will be
+temporarily set to this value.  See also
+`ert-batch-backtrace-line-length' for its effect on stack
+traces.")
+
+(defvar ert-batch-print-level 5
+  "`print-level' setting used in `ert-run-tests-batch'.
+
+When formatting lists in test conditions, `print-level' will be
+temporarily set to this value.  See also
+`ert-batch-backtrace-line-length' for its effect on stack
+traces.")
+
+(defvar ert-batch-backtrace-line-length t
+  "Target length for lines in ERT batch backtraces.
+
+Even modest settings for `print-length' & `print-level' can
+produce extremely long lines in backtraces & lengthy delays in
+forming them.  This variable governs the target maximum line
+length by manipulating these two variables while printing stack
+traces.  Setting this variable to t will re-use the value of
+`backtrace-line-length' while print stack traces in ERT batch
+mode.  A value of nil will short-circuit this mechanism; line
+lengths will be completely determined by `ert-batch-line-length'
+& `ert-batch-line-level'.  Any other value will be temporarily
+bound to `backtrace-line-length' when producing stack traces
+in batch mode.")
+
 (defface ert-test-result-expected '((((class color) (background light))
                                      :background "green1")
                                     (((class color) (background dark))
@@ -1402,8 +1433,7 @@ ert-run-tests-batch
                                       (ert-reason-for-test-result result)
                                     ""))))
               (message "%s" "")))))
-       (test-started
-        )
+       (test-started)
        (test-ended
         (cl-destructuring-bind (stats test result) event-args
           (unless (ert-test-result-expected-p test result)
@@ -1413,8 +1443,15 @@ ert-run-tests-batch
               (ert-test-result-with-condition
                (message "Test %S backtrace:" (ert-test-name test))
                (with-temp-buffer
-                 (insert (backtrace-to-string
-                          (ert-test-result-with-condition-backtrace result)))
+                 (let ((backtrace-line-length
+                        (cond
+                         ((eq ert-batch-backtrace-line-length t) backtrace-line-length)
+                         ((eq ert-batch-backtrace-line-length nil) nil)
+                         (t ert-batch-backtrace-line-length)))
+                       (print-level ert-batch-print-level)
+                       (print-length ert-batch-print-length))
+                   (insert (backtrace-to-string
+                            (ert-test-result-with-condition-backtrace result))))
                  (if (not ert-batch-backtrace-right-margin)
                      (message "%s"
                               (buffer-substring-no-properties (point-min)
@@ -1433,8 +1470,8 @@ ert-run-tests-batch
                  (ert--insert-infos result)
                  (insert "    ")
                  (let ((print-escape-newlines t)
-                       (print-level 5)
-                       (print-length 10))
+                       (print-level ert-batch-print-level)
+                       (print-length ert-batch-print-length))
                    (ert--pp-with-indentation-and-newline
                     (ert-test-result-with-condition-condition result)))
                  (goto-char (1- (point-max)))
@@ -1962,13 +1999,13 @@ ert--results-font-lock-function
   (ewoc-refresh ert--results-ewoc)
   (font-lock-default-function enabledp))
 
-(defun ert--setup-results-buffer (stats listener buffer-name)
+(defvar ert--output-buffer-name "*ert*")
+
+(defun ert--setup-results-buffer (stats listener)
   "Set up a test results buffer.
 
-STATS is the stats object; LISTENER is the results listener;
-BUFFER-NAME, if non-nil, is the buffer name to use."
-  (unless buffer-name (setq buffer-name "*ert*"))
-  (let ((buffer (get-buffer-create buffer-name)))
+STATS is the stats object; LISTENER is the results listener."
+  (let ((buffer (get-buffer-create ert--output-buffer-name)))
     (with-current-buffer buffer
       (let ((inhibit-read-only t))
         (buffer-disable-undo)
@@ -2000,18 +2037,11 @@ ert--setup-results-buffer
 (defvar ert--selector-history nil
   "List of recent test selectors read from terminal.")
 
-;; Should OUTPUT-BUFFER-NAME and MESSAGE-FN really be arguments here?
-;; They are needed only for our automated self-tests at the moment.
-;; Or should there be some other mechanism?
 ;;;###autoload
-(defun ert-run-tests-interactively (selector
-                                    &optional output-buffer-name message-fn)
+(defun ert-run-tests-interactively (selector)
   "Run the tests specified by SELECTOR and display the results in a buffer.
 
-SELECTOR works as described in `ert-select-tests'.
-OUTPUT-BUFFER-NAME and MESSAGE-FN should normally be nil; they
-are used for automated self-tests and specify which buffer to use
-and how to display message."
+SELECTOR works as described in `ert-select-tests'."
   (interactive
    (list (let ((default (if ert--selector-history
                             ;; Can't use `first' here as this form is
@@ -2024,23 +2054,17 @@ ert-run-tests-interactively
                              obarray #'ert-test-boundp nil nil
                              'ert--selector-history default nil)))
          nil))
-  (unless message-fn (setq message-fn 'message))
-  (let ((output-buffer-name output-buffer-name)
-        buffer
-        listener
-        (message-fn message-fn))
+  (let (buffer listener)
     (setq listener
           (lambda (event-type &rest event-args)
             (cl-ecase event-type
               (run-started
                (cl-destructuring-bind (stats) event-args
-                 (setq buffer (ert--setup-results-buffer stats
-                                                         listener
-                                                         output-buffer-name))
+                 (setq buffer (ert--setup-results-buffer stats listener))
                  (pop-to-buffer buffer)))
               (run-ended
                (cl-destructuring-bind (stats abortedp) event-args
-                 (funcall message-fn
+                 (message
                           "%sRan %s tests, %s results were as expected%s%s"
                           (if (not abortedp)
                               ""
diff --git a/test/lisp/emacs-lisp/ert-tests.el b/test/lisp/emacs-lisp/ert-tests.el
index 79576d24032..f783532aaff 100644
--- a/test/lisp/emacs-lisp/ert-tests.el
+++ b/test/lisp/emacs-lisp/ert-tests.el
@@ -39,10 +39,11 @@ ert-test-body-runs
 (defun ert-self-test ()
   "Run ERT's self-tests and make sure they actually ran."
   (let ((window-configuration (current-window-configuration)))
-    (let ((ert--test-body-was-run nil))
+    (let ((ert--test-body-was-run nil)
+          (ert--output-buffer-name " *ert self-tests*"))
       ;; The buffer name chosen here should not compete with the default
       ;; results buffer name for completion in `switch-to-buffer'.
-      (let ((stats (ert-run-tests-interactively "^ert-" " *ert self-tests*")))
+      (let ((stats (ert-run-tests-interactively "^ert-")))
         (cl-assert ert--test-body-was-run)
         (if (zerop (ert-stats-completed-unexpected stats))
             ;; Hide results window only when everything went well.
@@ -519,17 +520,17 @@ ert-test-run-tests-interactively
                                      :body (lambda () (ert-skip
                                                        "skip message")))))
     (let ((ert-debug-on-error nil))
-      (let* ((buffer-name (generate-new-buffer-name " *ert-test-run-tests*"))
-             (messages nil)
-             (mock-message-fn
-              (lambda (format-string &rest args)
-                (push (apply #'format format-string args) messages))))
+      (cl-letf* ((buffer-name (generate-new-buffer-name " *ert-test-run-tests*"))
+                 (ert--output-buffer-name buffer-name)
+                 (messages nil)
+                 ((symbol-function 'message)
+                  (lambda (format-string &rest args)
+                    (push (apply #'format format-string args) messages))))
         (save-window-excursion
           (unwind-protect
               (let ((case-fold-search nil))
                 (ert-run-tests-interactively
-                 `(member ,passing-test ,failing-test, skipped-test) buffer-name
-                 mock-message-fn)
+                 `(member ,passing-test ,failing-test, skipped-test))
                 (should (equal messages `(,(concat
                                             "Ran 3 tests, 1 results were "
                                             "as expected, 1 unexpected, "
@@ -551,6 +552,68 @@ ert-test-run-tests-interactively
             (when (get-buffer buffer-name)
               (kill-buffer buffer-name))))))))
 
+(ert-deftest ert-test-run-tests-batch ()
+  (let* ((complex-list '((:1 (:2 (:3 (:4 (:5 (:6 "abc"))))))))
+	 (long-list (make-list 11 1))
+	 (failing-test-1
+          (make-ert-test :name 'failing-test-1
+			 :body (lambda () (should (equal complex-list 1)))))
+	 (failing-test-2
+          (make-ert-test :name 'failing-test-2
+			 :body (lambda () (should (equal long-list 1))))))
+    (let ((ert-debug-on-error nil)
+          messages)
+      (cl-letf* (((symbol-function 'message)
+                  (lambda (format-string &rest args)
+                    (push (apply #'format format-string args) messages))))
+        (save-window-excursion
+          (unwind-protect
+              (let ((case-fold-search nil)
+                    (ert-batch-backtrace-right-margin nil)
+		    (ert-batch-print-level 10)
+		    (ert-batch-print-length 11))
+                (ert-run-tests-batch
+                 `(member ,failing-test-1 ,failing-test-2))))))
+      (let ((long-text "(different-types[ \t\n]+(1 1 1 1 1 1 1 1 1 1 1)[ \t\n]+1)))[ \t\n]*$")
+	    (complex-text "(different-types[ \t\n]+((:1[ \t\n]+(:2[ \t\n]+(:3[ \t\n]+(:4[ \t\n]+(:5[ \t\n]+(:6[ \t\n]+\"abc\")))))))[ \t\n]+1)))[ \t\n]*$")
+            found-long
+	    found-complex)
+	(cl-loop for msg in (reverse messages)
+		 do
+		 (unless found-long
+		   (setq found-long (string-match long-text msg)))
+		 (unless found-complex
+		   (setq found-complex (string-match complex-text msg))))
+	(should found-long)
+	(should found-complex)))))
+
+(ert-deftest ert-test-run-tests-batch-expensive ()
+  (let* ((complex-list '((:1 (:2 (:3 (:4 (:5 (:6 "abc"))))))))
+	 (failing-test-1
+          (make-ert-test :name 'failing-test-1
+			 :body (lambda () (should (equal complex-list 1))))))
+    (let ((ert-debug-on-error nil)
+          messages)
+      (cl-letf* (((symbol-function 'message)
+                  (lambda (format-string &rest args)
+                    (push (apply #'format format-string args) messages))))
+        (save-window-excursion
+          (unwind-protect
+              (let ((case-fold-search nil)
+                    (ert-batch-backtrace-right-margin nil)
+                    (ert-batch-backtrace-line-length nil)
+		    (ert-batch-print-level 6)
+		    (ert-batch-print-length 11))
+                (ert-run-tests-batch
+                 `(member ,failing-test-1))))))
+      (let ((frame "ert-fail(((should (equal complex-list 1)) :form (equal ((:1 (:2 (:3 (:4 (:5 (:6 \"abc\"))))))) 1) :value nil :explanation (different-types ((:1 (:2 (:3 (:4 (:5 (:6 \"abc\"))))))) 1)))")
+            found-frame)
+	(cl-loop for msg in (reverse messages)
+		 do
+		 (unless found-frame
+		   (setq found-frame (cl-search frame msg :test 'equal))))
+        (should found-frame)))))
+
 (ert-deftest ert-test-special-operator-p ()
   (should (ert--special-operator-p 'if))
   (should-not (ert--special-operator-p 'car))
diff --git a/test/lisp/emacs-lisp/ert-x-tests.el b/test/lisp/emacs-lisp/ert-x-tests.el
index 9baa9941586..1001525dee9 100644
--- a/test/lisp/emacs-lisp/ert-x-tests.el
+++ b/test/lisp/emacs-lisp/ert-x-tests.el
@@ -103,23 +103,24 @@ ert-propertized-string
 
 (ert-deftest ert-test-run-tests-interactively-2 ()
   :tags '(:causes-redisplay)
-  (let* ((passing-test (make-ert-test :name 'passing-test
-                                      :body (lambda () (ert-pass))))
-         (failing-test (make-ert-test :name 'failing-test
-                                      :body (lambda ()
-                                              (ert-info ((propertize "foo\nbar"
-                                                                     'a 'b))
-                                                (ert-fail
-                                                 "failure message")))))
-         (skipped-test (make-ert-test :name 'skipped-test
-                                      :body (lambda () (ert-skip
-							"skip message"))))
-         (ert-debug-on-error nil)
-         (buffer-name (generate-new-buffer-name "*ert-test-run-tests*"))
-         (messages nil)
-         (mock-message-fn
-          (lambda (format-string &rest args)
-            (push (apply #'format format-string args) messages))))
+  (cl-letf* ((passing-test (make-ert-test :name 'passing-test
+                                          :body (lambda () (ert-pass))))
+             (failing-test (make-ert-test :name 'failing-test
+                                          :body (lambda ()
+                                                  (ert-info ((propertize "foo\nbar"
+                                                                         'a 'b))
+                                                            (ert-fail
+                                                             "failure message")))))
+             (skipped-test (make-ert-test :name 'skipped-test
+                                          :body (lambda () (ert-skip
+							    "skip message"))))
+             (ert-debug-on-error nil)
+             (messages nil)
+             (buffer-name (generate-new-buffer-name "*ert-test-run-tests*"))
+             ((symbol-function 'message)
+              (lambda (format-string &rest args)
+                (push (apply #'format format-string args) messages)))
+             (ert--output-buffer-name buffer-name))
     (cl-flet ((expected-string (with-font-lock-p)
                 (ert-propertized-string
                  "Selector: (member <passing-test> <failing-test> "
@@ -152,14 +153,12 @@ ert-test-run-tests-interactively-2
                  "failing-test"
                  nil "\n    Info: " '(a b) "foo\n"
                  nil "          " '(a b) "bar"
-                 nil "\n    (ert-test-failed \"failure message\")\n\n\n"
-                 )))
+                 nil "\n    (ert-test-failed \"failure message\")\n\n\n")))
       (save-window-excursion
         (unwind-protect
             (let ((case-fold-search nil))
               (ert-run-tests-interactively
-               `(member ,passing-test ,failing-test ,skipped-test) buffer-name
-               mock-message-fn)
+               `(member ,passing-test ,failing-test ,skipped-test))
               (should (equal messages `(,(concat
                                           "Ran 3 tests, 1 results were "
                                           "as expected, 1 unexpected, "
-- 
2.33.1


[-- Attachment #3: Type: text/plain, Size: 32 bytes --]


-- 
Michael <sp1ff@runbox.com>

bug#51037: [PATCH] Make `print-level` & `print-length` customizable in ERT batch tests

[Eli Zaretskii]

> From: Michael <sp1ff@runbox.com>
> Cc: larsi@gnus.org, gazally@runbox.com, 51037@debbugs.gnu.org
> Date: Sun, 14 Nov 2021 07:42:27 -0800
> 
> This commit introduces three new ert variables:
> ert-batch-print-length, ert-batch-print-level &
> ert-batch-backtrace-length. The first two control print-length
                            ^^
Please leave 2 spaces between sentences, per our coding conventions
(here and elsewhere in your patch).

> +line lengths (i.e. to get full backtraces), or a positive integer to

Either "i.e.," (with a comma), or "i.e.@:".  Otherwise TeX will
typeset "i.e." as if it ends a sentence.

> +** New ERT batch variables
> +
> +Add three variables 'ert-batch-print-length', 'ert-batch-print-level'

In NEWS we use a different style, like this:

  Three new variables 'ert-batch-print-length', ...

IOW, "Add" is not appropriate here.

>  Backtrace mode will attempt to abbreviate printing of backtrace
> -frames to make them shorter than this, but success is not
> -guaranteed.  If set to nil or zero, Backtrace mode will not
> -abbreviate the forms it prints."
> +frames by setting `print-level' & `print-length' to make them

Please don't use "&" in doc strings; use "and" instead (here and
elsewhere in the patch).

> +shorter than this, but success is not guaranteed.  If set to nil
> +or zero, Backtrace mode will not abbreviate the forms it prints."
            ^^^^^^^^^
"backtrace", not capitalized/

> -(defun ert--setup-results-buffer (stats listener buffer-name)
> +(defvar ert--output-buffer-name "*ert*")
> +
> +(defun ert--setup-results-buffer (stats listener)
>    "Set up a test results buffer.
>  
> -STATS is the stats object; LISTENER is the results listener;
> -BUFFER-NAME, if non-nil, is the buffer name to use."
> -  (unless buffer-name (setq buffer-name "*ert*"))
> -  (let ((buffer (get-buffer-create buffer-name)))
> +STATS is the stats object; LISTENER is the results listener."
> +  (let ((buffer (get-buffer-create ert--output-buffer-name)))

Why this change in the signature of the function?

bug#51037: [PATCH] Make `print-level` & `print-length` customizable in ERT batch tests

[Michael]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Michael <sp1ff@runbox.com>
>> Cc: larsi@gnus.org, gazally@runbox.com, 51037@debbugs.gnu.org
>> Date: Sun, 14 Nov 2021 07:42:27 -0800
>> 
>> This commit introduces three new ert variables:
>> ert-batch-print-length, ert-batch-print-level &
>> ert-batch-backtrace-length. The first two control print-length
>                             ^^
> Please leave 2 spaces between sentences, per our coding 
> conventions
> (here and elsewhere in your patch).

Fixed.

>> +line lengths (i.e. to get full backtraces), or a positive 
>> integer to
>
> Either "i.e.," (with a comma), or "i.e.@:".  Otherwise TeX will
> typeset "i.e." as if it ends a sentence.

Fixed.

>> +** New ERT batch variables
>> +
>> +Add three variables 'ert-batch-print-length', 
>> 'ert-batch-print-level'
>
> In NEWS we use a different style, like this:
>
>   Three new variables 'ert-batch-print-length', ...
>
> IOW, "Add" is not appropriate here.

Kindly fixed already by Lars.

>>  Backtrace mode will attempt to abbreviate printing of 
>>  backtrace
>> -frames to make them shorter than this, but success is not
>> -guaranteed.  If set to nil or zero, Backtrace mode will not
>> -abbreviate the forms it prints."
>> +frames by setting `print-level' & `print-length' to make them
>
> Please don't use "&" in doc strings; use "and" instead (here and
> elsewhere in the patch).

Fixed.

>> +shorter than this, but success is not guaranteed.  If set to 
>> nil
>> +or zero, Backtrace mode will not abbreviate the forms it 
>> prints."
>             ^^^^^^^^^
> "backtrace", not capitalized/

Oh! Good catch -- fixed.

>> -(defun ert--setup-results-buffer (stats listener buffer-name)
>> +(defvar ert--output-buffer-name "*ert*")
>> +
>> +(defun ert--setup-results-buffer (stats listener)
>>    "Set up a test results buffer.
>>  
>> -STATS is the stats object; LISTENER is the results listener;
>> -BUFFER-NAME, if non-nil, is the buffer name to use."
>> -  (unless buffer-name (setq buffer-name "*ert*"))
>> -  (let ((buffer (get-buffer-create buffer-name)))
>> +STATS is the stats object; LISTENER is the results listener."
>> +  (let ((buffer (get-buffer-create ert--output-buffer-name)))
>
> Why this change in the signature of the function?

A consequence of a decision discussed elsewhere in this thread:
the only reason that parameter existed was to enable unit
testing, which I re-wrote to use `cl-letf`.

Background: prior to this patch, `ert-run-tests-interactively`
accepted two optional parameters: output-buffer-name and
message-fn. The were soley used for the purpose of unit testing,
and were even documented in commentary as such:

;; Should OUTPUT-BUFFER-NAME and MESSAGE-FN really be arguments 
   here?
;; They are needed only for our automated self-tests at the 
   moment.
;; Or should there be some other mechanism?

I carried over the idiom to ert-run-tests-batch when adding my
own unit tests. Lars pointed out that I could use `cl-letf` to
override the function definition of `message`, rather than using
a "test-only" parameter. I did that, then removed the test-only
parameters from `ert-run-tests-interactively` an re-wrote those
unit tests to also use `cl-letf`. At that point, there was no one
using the the `buffer-name` parameter to
`ert--setup-results-buffer`. Since it's an internal function, I
felt comfortable just removing the parameter.

-- 
Michael <sp1ff@runbox.com>

bug#51037: [PATCH] Make `print-level` & `print-length` customizable in ERT batch tests

[Lars Ingebrigtsen]

Michael <sp1ff@runbox.com> writes:

> OK-- here's the revised patch.

Thanks; applied to Emacs 29 (with some whitespace changes to make the
lines less wide).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#51037: [PATCH] Make `print-level` & `print-length` customizable in ERT batch tests

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Michael <sp1ff@runbox.com> writes:
>
>> OK-- here's the revised patch.
>
> Thanks; applied to Emacs 29 (with some whitespace changes to make the
> lines less wide).

I spoke to soon -- we don't seem to have copyright assignment papers on
file?  Is that in the process of happening?

So here's the patch again with the whitespace tweaks while we wait for
that.  🤨

diff --git a/doc/misc/ert.texi b/doc/misc/ert.texi
index 440c61add8..522b09dd61 100644
--- a/doc/misc/ert.texi
+++ b/doc/misc/ert.texi
@@ -390,6 +390,31 @@ Running Tests in Batch Mode
 emacs -batch -l ert -f ert-summarize-tests-batch-and-exit output.log
 @end example
 
+@vindex ert-batch-print-level
+@vindex ert-batch-print-length
+ERT attempts to limit the output size for failed tests by choosing
+conservative values for @code{print-level} & @code{print-length}
+when printing Lisp values.  This can in some cases make it difficult
+to see which portions of those values are incorrect.  Use
+@code{ert-batch-print-level} and @code{ert-batch-print-length}
+to customize that:
+
+@example
+emacs -batch -l ert -l my-tests.el \
+      --eval "(let ((ert-batch-print-level 10) \
+                    (ert-batch-print-length 120)) \
+                (ert-run-tests-batch-and-exit))"
+@end example
+
+@vindex ert-batch-backtrace-line-length
+Even modest settings for @code{print-level} & @code{print-length} can
+produce extremely long lines in backtraces, however, with attendant
+pauses in execution progress. Set
+@code{ert-batch-backtrace-line-length} to t to use the value of
+@code{backtrace-line-length}, nil to stop any limitations on backtrace
+line lengths (i.e. to get full backtraces), or a positive integer to
+limit backtrace line length to that number.
+
 @vindex ert-quiet
 By default, ERT in batch mode is quite verbose, printing a line with
 result after each test.  This gives you progress information: how many
diff --git a/etc/NEWS b/etc/NEWS
index 312fc18f4f..ccae7a052c 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -54,6 +54,13 @@ This is in addition to previously-supported ways of discovering 24-bit
 color support: either via the "RGB" or "setf24" capabilities, or if
 the 'COLORTERM' environment variable is set to the value "truecolor".
 
++++
+** New ERT variables 'ert-batch-print-length' and 'ert-batch-print-level'.
+These variables will override 'print-length' and 'print-level' when
+printing Lisp values in ERT batch test results.
+
+** Emacs now supports Unicode Standard version 14.0.
+
 ** Emoji
 
 +++
diff --git a/lisp/emacs-lisp/backtrace.el b/lisp/emacs-lisp/backtrace.el
index a5721aa319..90201442f0 100644
--- a/lisp/emacs-lisp/backtrace.el
+++ b/lisp/emacs-lisp/backtrace.el
@@ -55,9 +55,9 @@ backtrace-fontify
 (defcustom backtrace-line-length 5000
   "Target length for lines in Backtrace buffers.
 Backtrace mode will attempt to abbreviate printing of backtrace
-frames to make them shorter than this, but success is not
-guaranteed.  If set to nil or zero, Backtrace mode will not
-abbreviate the forms it prints."
+frames by setting `print-level' & `print-length' to make them
+shorter than this, but success is not guaranteed.  If set to nil
+or zero, Backtrace mode will not abbreviate the forms it prints."
   :type 'integer
   :group 'backtrace
   :version "27.1")
@@ -751,6 +751,13 @@ backtrace--print-flags
     (insert (make-string (- backtrace--flags-width (- (point) beg)) ?\s))
     (put-text-property beg (point) 'backtrace-section 'func)))
 
+(defun backtrace--line-length-or-nil ()
+  "Return `backtrace-line-length' if valid, nil else."
+  ;; mirror the logic in `cl-print-to-string-with-limits'
+  (and (natnump backtrace-line-length)
+       (not (zerop backtrace-line-length))
+       backtrace-line-length))
+
 (defun backtrace--print-func-and-args (frame _view)
   "Print the function, arguments and buffer position of a backtrace FRAME.
 Format it according to VIEW."
@@ -769,11 +776,13 @@ backtrace--print-func-and-args
       (if (atom fun)
           (funcall backtrace-print-function fun)
         (insert
-         (backtrace--print-to-string fun (when args (/ backtrace-line-length 2)))))
+         (backtrace--print-to-string fun (when (and args (backtrace--line-length-or-nil)) (/ backtrace-line-length 2)))))
       (if args
           (insert (backtrace--print-to-string
-                   args (max (truncate (/ backtrace-line-length 5))
-                             (- backtrace-line-length (- (point) beg)))))
+                   args
+                   (if (backtrace--line-length-or-nil)
+                       (max (truncate (/ backtrace-line-length 5))
+                            (- backtrace-line-length (- (point) beg))))))
         ;; The backtrace-form property is so that backtrace-multi-line
         ;; will find it.  backtrace-multi-line doesn't do anything
         ;; useful with it, just being consistent.
diff --git a/lisp/emacs-lisp/ert.el b/lisp/emacs-lisp/ert.el
index 8ebc81fd41..5396cbbd5d 100644
--- a/lisp/emacs-lisp/ert.el
+++ b/lisp/emacs-lisp/ert.el
@@ -77,6 +77,37 @@ ert-batch-backtrace-right-margin
 Use nil for no limit (caution: backtrace lines can be very long)."
   :type '(choice (const :tag "No truncation" nil) integer))
 
+(defvar ert-batch-print-length 10
+  "`print-length' setting used in `ert-run-tests-batch'.
+
+When formatting lists in test conditions, `print-length' will be
+temporarily set to this value.  See also
+`ert-batch-backtrace-line-length' for its effect on stack
+traces.")
+
+(defvar ert-batch-print-level 5
+  "`print-level' setting used in `ert-run-tests-batch'.
+
+When formatting lists in test conditions, `print-level' will be
+temporarily set to this value.  See also
+`ert-batch-backtrace-line-length' for its effect on stack
+traces.")
+
+(defvar ert-batch-backtrace-line-length t
+  "Target length for lines in ERT batch backtraces.
+
+Even modest settings for `print-length' & `print-level' can
+produce extremely long lines in backtraces & lengthy delays in
+forming them.  This variable governs the target maximum line
+length by manipulating these two variables while printing stack
+traces.  Setting this variable to t will re-use the value of
+`backtrace-line-length' while print stack traces in ERT batch
+mode.  A value of nil will short-circuit this mechanism; line
+lengths will be completely determined by `ert-batch-line-length'
+& `ert-batch-line-level'.  Any other value will be temporarily
+bound to `backtrace-line-length' when producing stack traces
+in batch mode.")
+
 (defface ert-test-result-expected '((((class color) (background light))
                                      :background "green1")
                                     (((class color) (background dark))
@@ -1402,8 +1433,7 @@ ert-run-tests-batch
                                       (ert-reason-for-test-result result)
                                     ""))))
               (message "%s" "")))))
-       (test-started
-        )
+       (test-started)
        (test-ended
         (cl-destructuring-bind (stats test result) event-args
           (unless (ert-test-result-expected-p test result)
@@ -1413,8 +1443,19 @@ ert-run-tests-batch
               (ert-test-result-with-condition
                (message "Test %S backtrace:" (ert-test-name test))
                (with-temp-buffer
-                 (insert (backtrace-to-string
-                          (ert-test-result-with-condition-backtrace result)))
+                 (let ((backtrace-line-length
+                        (cond
+                         ((eq ert-batch-backtrace-line-length t)
+                          backtrace-line-length)
+                         ((eq ert-batch-backtrace-line-length nil)
+                          nil)
+                         (t
+                          ert-batch-backtrace-line-length)))
+                       (print-level ert-batch-print-level)
+                       (print-length ert-batch-print-length))
+                   (insert (backtrace-to-string
+                            (ert-test-result-with-condition-backtrace
+                             result))))
                  (if (not ert-batch-backtrace-right-margin)
                      (message "%s"
                               (buffer-substring-no-properties (point-min)
@@ -1433,8 +1474,8 @@ ert-run-tests-batch
                  (ert--insert-infos result)
                  (insert "    ")
                  (let ((print-escape-newlines t)
-                       (print-level 5)
-                       (print-length 10))
+                       (print-level ert-batch-print-level)
+                       (print-length ert-batch-print-length))
                    (ert--pp-with-indentation-and-newline
                     (ert-test-result-with-condition-condition result)))
                  (goto-char (1- (point-max)))
@@ -1962,13 +2003,13 @@ ert--results-font-lock-function
   (ewoc-refresh ert--results-ewoc)
   (font-lock-default-function enabledp))
 
-(defun ert--setup-results-buffer (stats listener buffer-name)
+(defvar ert--output-buffer-name "*ert*")
+
+(defun ert--setup-results-buffer (stats listener)
   "Set up a test results buffer.
 
-STATS is the stats object; LISTENER is the results listener;
-BUFFER-NAME, if non-nil, is the buffer name to use."
-  (unless buffer-name (setq buffer-name "*ert*"))
-  (let ((buffer (get-buffer-create buffer-name)))
+STATS is the stats object; LISTENER is the results listener."
+  (let ((buffer (get-buffer-create ert--output-buffer-name)))
     (with-current-buffer buffer
       (let ((inhibit-read-only t))
         (buffer-disable-undo)
@@ -2000,18 +2041,11 @@ ert--setup-results-buffer
 (defvar ert--selector-history nil
   "List of recent test selectors read from terminal.")
 
-;; Should OUTPUT-BUFFER-NAME and MESSAGE-FN really be arguments here?
-;; They are needed only for our automated self-tests at the moment.
-;; Or should there be some other mechanism?
 ;;;###autoload
-(defun ert-run-tests-interactively (selector
-                                    &optional output-buffer-name message-fn)
+(defun ert-run-tests-interactively (selector)
   "Run the tests specified by SELECTOR and display the results in a buffer.
 
-SELECTOR works as described in `ert-select-tests'.
-OUTPUT-BUFFER-NAME and MESSAGE-FN should normally be nil; they
-are used for automated self-tests and specify which buffer to use
-and how to display message."
+SELECTOR works as described in `ert-select-tests'."
   (interactive
    (list (let ((default (if ert--selector-history
                             ;; Can't use `first' here as this form is
@@ -2024,23 +2058,17 @@ ert-run-tests-interactively
                              obarray #'ert-test-boundp nil nil
                              'ert--selector-history default nil)))
          nil))
-  (unless message-fn (setq message-fn 'message))
-  (let ((output-buffer-name output-buffer-name)
-        buffer
-        listener
-        (message-fn message-fn))
+  (let (buffer listener)
     (setq listener
           (lambda (event-type &rest event-args)
             (cl-ecase event-type
               (run-started
                (cl-destructuring-bind (stats) event-args
-                 (setq buffer (ert--setup-results-buffer stats
-                                                         listener
-                                                         output-buffer-name))
+                 (setq buffer (ert--setup-results-buffer stats listener))
                  (pop-to-buffer buffer)))
               (run-ended
                (cl-destructuring-bind (stats abortedp) event-args
-                 (funcall message-fn
+                 (message
                           "%sRan %s tests, %s results were as expected%s%s"
                           (if (not abortedp)
                               ""
@@ -2394,7 +2422,7 @@ ert-results-rerun-all-tests
   (interactive nil ert-results-mode)
   (cl-assert (eql major-mode 'ert-results-mode))
   (let ((selector (ert--stats-selector ert--results-stats)))
-    (ert-run-tests-interactively selector (buffer-name))))
+    (ert-run-tests-interactively selector)))
 
 (defun ert-results-rerun-test-at-point ()
   "Re-run the test at point.
diff --git a/test/lisp/emacs-lisp/ert-tests.el b/test/lisp/emacs-lisp/ert-tests.el
index 79576d2403..1a8c9bf4f0 100644
--- a/test/lisp/emacs-lisp/ert-tests.el
+++ b/test/lisp/emacs-lisp/ert-tests.el
@@ -39,10 +39,11 @@ ert-test-body-runs
 (defun ert-self-test ()
   "Run ERT's self-tests and make sure they actually ran."
   (let ((window-configuration (current-window-configuration)))
-    (let ((ert--test-body-was-run nil))
+    (let ((ert--test-body-was-run nil)
+          (ert--output-buffer-name " *ert self-tests*"))
       ;; The buffer name chosen here should not compete with the default
       ;; results buffer name for completion in `switch-to-buffer'.
-      (let ((stats (ert-run-tests-interactively "^ert-" " *ert self-tests*")))
+      (let ((stats (ert-run-tests-interactively "^ert-")))
         (cl-assert ert--test-body-was-run)
         (if (zerop (ert-stats-completed-unexpected stats))
             ;; Hide results window only when everything went well.
@@ -519,17 +520,18 @@ ert-test-run-tests-interactively
                                      :body (lambda () (ert-skip
                                                        "skip message")))))
     (let ((ert-debug-on-error nil))
-      (let* ((buffer-name (generate-new-buffer-name " *ert-test-run-tests*"))
-             (messages nil)
-             (mock-message-fn
-              (lambda (format-string &rest args)
-                (push (apply #'format format-string args) messages))))
+      (cl-letf* ((buffer-name (generate-new-buffer-name
+                               " *ert-test-run-tests*"))
+                 (ert--output-buffer-name buffer-name)
+                 (messages nil)
+                 ((symbol-function 'message)
+                  (lambda (format-string &rest args)
+                    (push (apply #'format format-string args) messages))))
         (save-window-excursion
           (unwind-protect
               (let ((case-fold-search nil))
                 (ert-run-tests-interactively
-                 `(member ,passing-test ,failing-test, skipped-test) buffer-name
-                 mock-message-fn)
+                 `(member ,passing-test ,failing-test, skipped-test))
                 (should (equal messages `(,(concat
                                             "Ran 3 tests, 1 results were "
                                             "as expected, 1 unexpected, "
@@ -551,6 +553,68 @@ ert-test-run-tests-interactively
             (when (get-buffer buffer-name)
               (kill-buffer buffer-name))))))))
 
+(ert-deftest ert-test-run-tests-batch ()
+  (let* ((complex-list '((:1 (:2 (:3 (:4 (:5 (:6 "abc"))))))))
+	 (long-list (make-list 11 1))
+	 (failing-test-1
+          (make-ert-test :name 'failing-test-1
+			 :body (lambda () (should (equal complex-list 1)))))
+	 (failing-test-2
+          (make-ert-test :name 'failing-test-2
+			 :body (lambda () (should (equal long-list 1))))))
+    (let ((ert-debug-on-error nil)
+          messages)
+      (cl-letf* (((symbol-function 'message)
+                  (lambda (format-string &rest args)
+                    (push (apply #'format format-string args) messages))))
+        (save-window-excursion
+          (unwind-protect
+              (let ((case-fold-search nil)
+                    (ert-batch-backtrace-right-margin nil)
+		    (ert-batch-print-level 10)
+		    (ert-batch-print-length 11))
+                (ert-run-tests-batch
+                 `(member ,failing-test-1 ,failing-test-2))))))
+      (let ((long-text "(different-types[ \t\n]+(1 1 1 1 1 1 1 1 1 1 1)[ \t\n]+1)))[ \t\n]*$")
+	    (complex-text "(different-types[ \t\n]+((:1[ \t\n]+(:2[ \t\n]+(:3[ \t\n]+(:4[ \t\n]+(:5[ \t\n]+(:6[ \t\n]+\"abc\")))))))[ \t\n]+1)))[ \t\n]*$")
+            found-long
+	    found-complex)
+	(cl-loop for msg in (reverse messages)
+		 do
+		 (unless found-long
+		   (setq found-long (string-match long-text msg)))
+		 (unless found-complex
+		   (setq found-complex (string-match complex-text msg))))
+	(should found-long)
+	(should found-complex)))))
+
+(ert-deftest ert-test-run-tests-batch-expensive ()
+  (let* ((complex-list '((:1 (:2 (:3 (:4 (:5 (:6 "abc"))))))))
+	 (failing-test-1
+          (make-ert-test :name 'failing-test-1
+			 :body (lambda () (should (equal complex-list 1))))))
+    (let ((ert-debug-on-error nil)
+          messages)
+      (cl-letf* (((symbol-function 'message)
+                  (lambda (format-string &rest args)
+                    (push (apply #'format format-string args) messages))))
+        (save-window-excursion
+          (unwind-protect
+              (let ((case-fold-search nil)
+                    (ert-batch-backtrace-right-margin nil)
+                    (ert-batch-backtrace-line-length nil)
+		    (ert-batch-print-level 6)
+		    (ert-batch-print-length 11))
+                (ert-run-tests-batch
+                 `(member ,failing-test-1))))))
+      (let ((frame "ert-fail(((should (equal complex-list 1)) :form (equal ((:1 (:2 (:3 (:4 (:5 (:6 \"abc\"))))))) 1) :value nil :explanation (different-types ((:1 (:2 (:3 (:4 (:5 (:6 \"abc\"))))))) 1)))")
+            found-frame)
+	(cl-loop for msg in (reverse messages)
+		 do
+		 (unless found-frame
+		   (setq found-frame (cl-search frame msg :test 'equal))))
+        (should found-frame)))))
+
 (ert-deftest ert-test-special-operator-p ()
   (should (ert--special-operator-p 'if))
   (should-not (ert--special-operator-p 'car))
diff --git a/test/lisp/emacs-lisp/ert-x-tests.el b/test/lisp/emacs-lisp/ert-x-tests.el
index 9baa994158..5813fc44bf 100644
--- a/test/lisp/emacs-lisp/ert-x-tests.el
+++ b/test/lisp/emacs-lisp/ert-x-tests.el
@@ -103,23 +103,26 @@ ert-propertized-string
 
 (ert-deftest ert-test-run-tests-interactively-2 ()
   :tags '(:causes-redisplay)
-  (let* ((passing-test (make-ert-test :name 'passing-test
-                                      :body (lambda () (ert-pass))))
-         (failing-test (make-ert-test :name 'failing-test
-                                      :body (lambda ()
-                                              (ert-info ((propertize "foo\nbar"
-                                                                     'a 'b))
-                                                (ert-fail
-                                                 "failure message")))))
-         (skipped-test (make-ert-test :name 'skipped-test
-                                      :body (lambda () (ert-skip
-							"skip message"))))
-         (ert-debug-on-error nil)
-         (buffer-name (generate-new-buffer-name "*ert-test-run-tests*"))
-         (messages nil)
-         (mock-message-fn
-          (lambda (format-string &rest args)
-            (push (apply #'format format-string args) messages))))
+  (cl-letf* ((passing-test (make-ert-test
+                            :name 'passing-test
+                            :body (lambda () (ert-pass))))
+             (failing-test (make-ert-test
+                            :name 'failing-test
+                            :body (lambda ()
+                                    (ert-info ((propertize "foo\nbar"
+                                                           'a 'b))
+                                              (ert-fail
+                                               "failure message")))))
+             (skipped-test (make-ert-test :name 'skipped-test
+                                          :body (lambda () (ert-skip
+							    "skip message"))))
+             (ert-debug-on-error nil)
+             (messages nil)
+             (buffer-name (generate-new-buffer-name "*ert-test-run-tests*"))
+             ((symbol-function 'message)
+              (lambda (format-string &rest args)
+                (push (apply #'format format-string args) messages)))
+             (ert--output-buffer-name buffer-name))
     (cl-flet ((expected-string (with-font-lock-p)
                 (ert-propertized-string
                  "Selector: (member <passing-test> <failing-test> "
@@ -152,14 +155,12 @@ ert-test-run-tests-interactively-2
                  "failing-test"
                  nil "\n    Info: " '(a b) "foo\n"
                  nil "          " '(a b) "bar"
-                 nil "\n    (ert-test-failed \"failure message\")\n\n\n"
-                 )))
+                 nil "\n    (ert-test-failed \"failure message\")\n\n\n")))
       (save-window-excursion
         (unwind-protect
             (let ((case-fold-search nil))
               (ert-run-tests-interactively
-               `(member ,passing-test ,failing-test ,skipped-test) buffer-name
-               mock-message-fn)
+               `(member ,passing-test ,failing-test ,skipped-test))
               (should (equal messages `(,(concat
                                           "Ran 3 tests, 1 results were "
                                           "as expected, 1 unexpected, "

bug#51037: [PATCH] Make `print-level` & `print-length` customizable in ERT batch tests

[Michael]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I spoke to soon -- we don't seem to have copyright assignment 
> papers on
> file?  Is that in the process of happening?

Right-- this is my first contribution. What's the first step on
getting that setup?

-- 
Michael <sp1ff@runbox.com>

bug#51037: [PATCH] Make `print-level` & `print-length` customizable in ERT batch tests

[Lars Ingebrigtsen]

Michael <sp1ff@runbox.com> writes:

> Right-- this is my first contribution. What's the first step on
> getting that setup?

Here's the form to get started:

Please email the following information to assign@gnu.org, and we
will send you the assignment form for your past and future changes.

Please use your full legal name (in ASCII characters) as the subject
line of the message.
----------------------------------------------------------------------
REQUEST: SEND FORM FOR PAST AND FUTURE CHANGES

[What is the name of the program or package you're contributing to?]
Emacs

[Did you copy any files or text written by someone else in these changes?
Even if that material is free software, we need to know about it.]

[Do you have an employer who might have a basis to claim to own
your changes?  Do you attend a school which might make such a claim?]

[For the copyright registration, what country are you a citizen of?]

[What year were you born?]

[Please write your email address here.]

[Please write your postal address here.]

[Which files have you changed so far, and which new files have you written
so far?]

bug#51037: [PATCH] Make `print-level` & `print-length` customizable in ERT batch tests

[Michael]

[-- Attachment #1: Type: text/plain, Size: 623 bytes --]


> Lars Ingebrigtsen <larsi@gnus.org> writes:
>
>> Michael <sp1ff@runbox.com> writes:
>>
>>> OK-- here's the revised patch.
>>
>> Thanks; applied to Emacs 29 (with some whitespace changes to 
>> make the
>> lines less wide).
>
> I spoke to soon -- we don't seem to have copyright assignment 
> papers on
> file?  Is that in the process of happening?
>
> So here's the patch again with the whitespace tweaks while we 
> wait for
> that.  🤨
>
> diff --git a/doc/misc/ert.texi b/doc/misc/ert.texi
[snip]

New patch attached. Incorporates (I hope) these changes, as well
as addresses Eli's issues.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: bug#51037 --]
[-- Type: text/x-patch, Size: 22860 bytes --]

From ccc3a72949f8298c2b8194d3b302d019f934af43 Mon Sep 17 00:00:00 2001
From: Michael Herstine <sp1ff@pobox.com>
Date: Tue, 5 Oct 2021 06:53:02 -0700
Subject: [PATCH] Make results details in ert-run-tests-batch configurable.

This commit introduces three new ert variables:
ert-batch-print-length, ert-batch-print-level &
ert-batch-backtrace-length.  The first two control print-length
and print-level, resp., when printing test results.

Since even modest values of print-level & print-length can
produce extremely long lines in backtraces along with
correspondingly long processing times to format them, this commit
also introduces the variable ert-backtrace-line-length which can,
optionally, override backtrace-line-length during ert batch test
stack traces.

Finally, it also removes the optional message-fn & output-buffer
arguments to ert-run-test-interactively (since they were only
used for testing purposes) and re-implements its tests in
terms of cl-letf.

* lisp/emacs-lisp/ert.el (ert-batch-print-length,
ert-batch-print-level,.ert-batch-backtrace-line-length,
ert-batch-test, ert-run-tests-interactively): Added the three
variables, bound them to these settings when formatting batch
test results including backtraces. Removed the optional
parameters output-buffer & message-fn from
ert-run-tests-interactively.
* test/lisp/emacs-lisp/ert-tests.el
(ert-test-run-tests-interactively, ert-test-run-tests-batch): use
cl-letf to capture output, new tests resp.
* test/lisp/ert-x-tests.el (ert-test-run-tests-interactively-2):
Changed to use cl-letf to capture output instead of using
message-fn.
* lisp/emacs-lisp/backtrace.el (backtrace--line-length-or-nil,
backtrace--print-func-and-args): Fixed a bug when setting
backtrace-line-length to nil by adding a new function to check
for that case & having backtrace--print-func-and-args use it.
* doc/misc/ert.texi: document the new variables & their usage
---
 doc/misc/ert.texi                   | 25 +++++++++
 etc/NEWS                            |  7 +++
 lisp/emacs-lisp/backtrace.el        | 21 +++++--
 lisp/emacs-lisp/ert.el              | 87 +++++++++++++++++++----------
 test/lisp/emacs-lisp/ert-tests.el   | 82 ++++++++++++++++++++++++---
 test/lisp/emacs-lisp/ert-x-tests.el | 44 ++++++++-------
 6 files changed, 200 insertions(+), 66 deletions(-)

diff --git a/doc/misc/ert.texi b/doc/misc/ert.texi
index 440c61add8e..2de3946db1e 100644
--- a/doc/misc/ert.texi
+++ b/doc/misc/ert.texi
@@ -390,6 +390,31 @@ Running Tests in Batch Mode
 emacs -batch -l ert -f ert-summarize-tests-batch-and-exit output.log
 @end example
 
+@vindex ert-batch-print-level
+@vindex ert-batch-print-length
+ERT attempts to limit the output size for failed tests by choosing
+conservative values for @code{print-level} & @code{print-length}
+when printing Lisp values.  This can in some cases make it difficult
+to see which portions of those values are incorrect.  Use
+@code{ert-batch-print-level} and @code{ert-batch-print-length}
+to customize that:
+
+@example
+emacs -batch -l ert -l my-tests.el \
+      --eval "(let ((ert-batch-print-level 10) \
+                    (ert-batch-print-length 120)) \
+                (ert-run-tests-batch-and-exit))"
+@end example
+
+@vindex ert-batch-backtrace-line-length
+Even modest settings for @code{print-level} & @code{print-length} can
+produce extremely long lines in backtraces, however, with attendant
+pauses in execution progress.  Set
+@code{ert-batch-backtrace-line-length} to t to use the value of
+@code{backtrace-line-length}, nil to stop any limitations on backtrace
+line lengths (i.e.@: to get full backtraces), or a positive integer to
+limit backtrace line length to that number.
+
 @vindex ert-quiet
 By default, ERT in batch mode is quite verbose, printing a line with
 result after each test.  This gives you progress information: how many
diff --git a/etc/NEWS b/etc/NEWS
index 3cad0995ac5..2fccea2a2eb 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -46,6 +46,13 @@ time.
 \f
 * Changes in Emacs 29.1
 
++++
+** New ERT variables 'ert-batch-print-length' and 'ert-batch-print-level'.
+These variables will override 'print-length' and 'print-level' when
+printing Lisp values in ERT batch test results.
+
+** Emacs now supports Unicode Standard version 14.0.
+
 ** Emoji
 
 +++
diff --git a/lisp/emacs-lisp/backtrace.el b/lisp/emacs-lisp/backtrace.el
index a5721aa3193..0e5f85d7b8b 100644
--- a/lisp/emacs-lisp/backtrace.el
+++ b/lisp/emacs-lisp/backtrace.el
@@ -55,9 +55,9 @@ backtrace-fontify
 (defcustom backtrace-line-length 5000
   "Target length for lines in Backtrace buffers.
 Backtrace mode will attempt to abbreviate printing of backtrace
-frames to make them shorter than this, but success is not
-guaranteed.  If set to nil or zero, Backtrace mode will not
-abbreviate the forms it prints."
+frames by setting `print-level' and `print-length' to make them
+shorter than this, but success is not guaranteed.  If set to nil
+or zero, backtrace mode will not abbreviate the forms it prints."
   :type 'integer
   :group 'backtrace
   :version "27.1")
@@ -751,6 +751,13 @@ backtrace--print-flags
     (insert (make-string (- backtrace--flags-width (- (point) beg)) ?\s))
     (put-text-property beg (point) 'backtrace-section 'func)))
 
+(defun backtrace--line-length-or-nil ()
+  "Return `backtrace-line-length' if valid, nil else."
+  ;; mirror the logic in `cl-print-to-string-with-limits'
+  (and (natnump backtrace-line-length)
+       (not (zerop backtrace-line-length))
+       backtrace-line-length))
+
 (defun backtrace--print-func-and-args (frame _view)
   "Print the function, arguments and buffer position of a backtrace FRAME.
 Format it according to VIEW."
@@ -769,11 +776,13 @@ backtrace--print-func-and-args
       (if (atom fun)
           (funcall backtrace-print-function fun)
         (insert
-         (backtrace--print-to-string fun (when args (/ backtrace-line-length 2)))))
+         (backtrace--print-to-string fun (when (and args (backtrace--line-length-or-nil)) (/ backtrace-line-length 2)))))
       (if args
           (insert (backtrace--print-to-string
-                   args (max (truncate (/ backtrace-line-length 5))
-                             (- backtrace-line-length (- (point) beg)))))
+                   args
+                   (if (backtrace--line-length-or-nil)
+                       (max (truncate (/ backtrace-line-length 5))
+                            (- backtrace-line-length (- (point) beg))))))
         ;; The backtrace-form property is so that backtrace-multi-line
         ;; will find it.  backtrace-multi-line doesn't do anything
         ;; useful with it, just being consistent.
diff --git a/lisp/emacs-lisp/ert.el b/lisp/emacs-lisp/ert.el
index 8ebc81fd418..36b4408dc8e 100644
--- a/lisp/emacs-lisp/ert.el
+++ b/lisp/emacs-lisp/ert.el
@@ -77,6 +77,37 @@ ert-batch-backtrace-right-margin
 Use nil for no limit (caution: backtrace lines can be very long)."
   :type '(choice (const :tag "No truncation" nil) integer))
 
+(defvar ert-batch-print-length 10
+  "`print-length' setting used in `ert-run-tests-batch'.
+
+When formatting lists in test conditions, `print-length' will be
+temporarily set to this value.  See also
+`ert-batch-backtrace-line-length' for its effect on stack
+traces.")
+
+(defvar ert-batch-print-level 5
+  "`print-level' setting used in `ert-run-tests-batch'.
+
+When formatting lists in test conditions, `print-level' will be
+temporarily set to this value.  See also
+`ert-batch-backtrace-line-length' for its effect on stack
+traces.")
+
+(defvar ert-batch-backtrace-line-length t
+  "Target length for lines in ERT batch backtraces.
+
+Even modest settings for `print-length' and `print-level' can
+produce extremely long lines in backtraces and lengthy delays in
+forming them.  This variable governs the target maximum line
+length by manipulating these two variables while printing stack
+traces.  Setting this variable to t will re-use the value of
+`backtrace-line-length' while print stack traces in ERT batch
+mode.  A value of nil will short-circuit this mechanism; line
+lengths will be completely determined by `ert-batch-line-length'
+and `ert-batch-line-level'.  Any other value will be temporarily
+bound to `backtrace-line-length' when producing stack traces
+in batch mode.")
+
 (defface ert-test-result-expected '((((class color) (background light))
                                      :background "green1")
                                     (((class color) (background dark))
@@ -1402,8 +1433,7 @@ ert-run-tests-batch
                                       (ert-reason-for-test-result result)
                                     ""))))
               (message "%s" "")))))
-       (test-started
-        )
+       (test-started)
        (test-ended
         (cl-destructuring-bind (stats test result) event-args
           (unless (ert-test-result-expected-p test result)
@@ -1413,8 +1443,18 @@ ert-run-tests-batch
               (ert-test-result-with-condition
                (message "Test %S backtrace:" (ert-test-name test))
                (with-temp-buffer
-                 (insert (backtrace-to-string
-                          (ert-test-result-with-condition-backtrace result)))
+                 (let ((backtrace-line-length
+                        (cond
+                         ((eq ert-batch-backtrace-line-length t)
+                          backtrace-line-length)
+                         ((eq ert-batch-backtrace-line-length nil)
+                          nil)
+                         (t
+                          ert-batch-backtrace-line-length)))
+                       (print-level ert-batch-print-level)
+                       (print-length ert-batch-print-length))
+                   (insert (backtrace-to-string
+                            (ert-test-result-with-condition-backtrace result))))
                  (if (not ert-batch-backtrace-right-margin)
                      (message "%s"
                               (buffer-substring-no-properties (point-min)
@@ -1433,8 +1473,8 @@ ert-run-tests-batch
                  (ert--insert-infos result)
                  (insert "    ")
                  (let ((print-escape-newlines t)
-                       (print-level 5)
-                       (print-length 10))
+                       (print-level ert-batch-print-level)
+                       (print-length ert-batch-print-length))
                    (ert--pp-with-indentation-and-newline
                     (ert-test-result-with-condition-condition result)))
                  (goto-char (1- (point-max)))
@@ -1962,13 +2002,13 @@ ert--results-font-lock-function
   (ewoc-refresh ert--results-ewoc)
   (font-lock-default-function enabledp))
 
-(defun ert--setup-results-buffer (stats listener buffer-name)
+(defvar ert--output-buffer-name "*ert*")
+
+(defun ert--setup-results-buffer (stats listener)
   "Set up a test results buffer.
 
-STATS is the stats object; LISTENER is the results listener;
-BUFFER-NAME, if non-nil, is the buffer name to use."
-  (unless buffer-name (setq buffer-name "*ert*"))
-  (let ((buffer (get-buffer-create buffer-name)))
+STATS is the stats object; LISTENER is the results listener."
+  (let ((buffer (get-buffer-create ert--output-buffer-name)))
     (with-current-buffer buffer
       (let ((inhibit-read-only t))
         (buffer-disable-undo)
@@ -2000,18 +2040,11 @@ ert--setup-results-buffer
 (defvar ert--selector-history nil
   "List of recent test selectors read from terminal.")
 
-;; Should OUTPUT-BUFFER-NAME and MESSAGE-FN really be arguments here?
-;; They are needed only for our automated self-tests at the moment.
-;; Or should there be some other mechanism?
 ;;;###autoload
-(defun ert-run-tests-interactively (selector
-                                    &optional output-buffer-name message-fn)
+(defun ert-run-tests-interactively (selector)
   "Run the tests specified by SELECTOR and display the results in a buffer.
 
-SELECTOR works as described in `ert-select-tests'.
-OUTPUT-BUFFER-NAME and MESSAGE-FN should normally be nil; they
-are used for automated self-tests and specify which buffer to use
-and how to display message."
+SELECTOR works as described in `ert-select-tests'."
   (interactive
    (list (let ((default (if ert--selector-history
                             ;; Can't use `first' here as this form is
@@ -2024,23 +2057,17 @@ ert-run-tests-interactively
                              obarray #'ert-test-boundp nil nil
                              'ert--selector-history default nil)))
          nil))
-  (unless message-fn (setq message-fn 'message))
-  (let ((output-buffer-name output-buffer-name)
-        buffer
-        listener
-        (message-fn message-fn))
+  (let (buffer listener)
     (setq listener
           (lambda (event-type &rest event-args)
             (cl-ecase event-type
               (run-started
                (cl-destructuring-bind (stats) event-args
-                 (setq buffer (ert--setup-results-buffer stats
-                                                         listener
-                                                         output-buffer-name))
+                 (setq buffer (ert--setup-results-buffer stats listener))
                  (pop-to-buffer buffer)))
               (run-ended
                (cl-destructuring-bind (stats abortedp) event-args
-                 (funcall message-fn
+                 (message
                           "%sRan %s tests, %s results were as expected%s%s"
                           (if (not abortedp)
                               ""
@@ -2394,7 +2421,7 @@ ert-results-rerun-all-tests
   (interactive nil ert-results-mode)
   (cl-assert (eql major-mode 'ert-results-mode))
   (let ((selector (ert--stats-selector ert--results-stats)))
-    (ert-run-tests-interactively selector (buffer-name))))
+    (ert-run-tests-interactively selector)))
 
 (defun ert-results-rerun-test-at-point ()
   "Re-run the test at point.
diff --git a/test/lisp/emacs-lisp/ert-tests.el b/test/lisp/emacs-lisp/ert-tests.el
index 79576d24032..1a8c9bf4f08 100644
--- a/test/lisp/emacs-lisp/ert-tests.el
+++ b/test/lisp/emacs-lisp/ert-tests.el
@@ -39,10 +39,11 @@ ert-test-body-runs
 (defun ert-self-test ()
   "Run ERT's self-tests and make sure they actually ran."
   (let ((window-configuration (current-window-configuration)))
-    (let ((ert--test-body-was-run nil))
+    (let ((ert--test-body-was-run nil)
+          (ert--output-buffer-name " *ert self-tests*"))
       ;; The buffer name chosen here should not compete with the default
       ;; results buffer name for completion in `switch-to-buffer'.
-      (let ((stats (ert-run-tests-interactively "^ert-" " *ert self-tests*")))
+      (let ((stats (ert-run-tests-interactively "^ert-")))
         (cl-assert ert--test-body-was-run)
         (if (zerop (ert-stats-completed-unexpected stats))
             ;; Hide results window only when everything went well.
@@ -519,17 +520,18 @@ ert-test-run-tests-interactively
                                      :body (lambda () (ert-skip
                                                        "skip message")))))
     (let ((ert-debug-on-error nil))
-      (let* ((buffer-name (generate-new-buffer-name " *ert-test-run-tests*"))
-             (messages nil)
-             (mock-message-fn
-              (lambda (format-string &rest args)
-                (push (apply #'format format-string args) messages))))
+      (cl-letf* ((buffer-name (generate-new-buffer-name
+                               " *ert-test-run-tests*"))
+                 (ert--output-buffer-name buffer-name)
+                 (messages nil)
+                 ((symbol-function 'message)
+                  (lambda (format-string &rest args)
+                    (push (apply #'format format-string args) messages))))
         (save-window-excursion
           (unwind-protect
               (let ((case-fold-search nil))
                 (ert-run-tests-interactively
-                 `(member ,passing-test ,failing-test, skipped-test) buffer-name
-                 mock-message-fn)
+                 `(member ,passing-test ,failing-test, skipped-test))
                 (should (equal messages `(,(concat
                                             "Ran 3 tests, 1 results were "
                                             "as expected, 1 unexpected, "
@@ -551,6 +553,68 @@ ert-test-run-tests-interactively
             (when (get-buffer buffer-name)
               (kill-buffer buffer-name))))))))
 
+(ert-deftest ert-test-run-tests-batch ()
+  (let* ((complex-list '((:1 (:2 (:3 (:4 (:5 (:6 "abc"))))))))
+	 (long-list (make-list 11 1))
+	 (failing-test-1
+          (make-ert-test :name 'failing-test-1
+			 :body (lambda () (should (equal complex-list 1)))))
+	 (failing-test-2
+          (make-ert-test :name 'failing-test-2
+			 :body (lambda () (should (equal long-list 1))))))
+    (let ((ert-debug-on-error nil)
+          messages)
+      (cl-letf* (((symbol-function 'message)
+                  (lambda (format-string &rest args)
+                    (push (apply #'format format-string args) messages))))
+        (save-window-excursion
+          (unwind-protect
+              (let ((case-fold-search nil)
+                    (ert-batch-backtrace-right-margin nil)
+		    (ert-batch-print-level 10)
+		    (ert-batch-print-length 11))
+                (ert-run-tests-batch
+                 `(member ,failing-test-1 ,failing-test-2))))))
+      (let ((long-text "(different-types[ \t\n]+(1 1 1 1 1 1 1 1 1 1 1)[ \t\n]+1)))[ \t\n]*$")
+	    (complex-text "(different-types[ \t\n]+((:1[ \t\n]+(:2[ \t\n]+(:3[ \t\n]+(:4[ \t\n]+(:5[ \t\n]+(:6[ \t\n]+\"abc\")))))))[ \t\n]+1)))[ \t\n]*$")
+            found-long
+	    found-complex)
+	(cl-loop for msg in (reverse messages)
+		 do
+		 (unless found-long
+		   (setq found-long (string-match long-text msg)))
+		 (unless found-complex
+		   (setq found-complex (string-match complex-text msg))))
+	(should found-long)
+	(should found-complex)))))
+
+(ert-deftest ert-test-run-tests-batch-expensive ()
+  (let* ((complex-list '((:1 (:2 (:3 (:4 (:5 (:6 "abc"))))))))
+	 (failing-test-1
+          (make-ert-test :name 'failing-test-1
+			 :body (lambda () (should (equal complex-list 1))))))
+    (let ((ert-debug-on-error nil)
+          messages)
+      (cl-letf* (((symbol-function 'message)
+                  (lambda (format-string &rest args)
+                    (push (apply #'format format-string args) messages))))
+        (save-window-excursion
+          (unwind-protect
+              (let ((case-fold-search nil)
+                    (ert-batch-backtrace-right-margin nil)
+                    (ert-batch-backtrace-line-length nil)
+		    (ert-batch-print-level 6)
+		    (ert-batch-print-length 11))
+                (ert-run-tests-batch
+                 `(member ,failing-test-1))))))
+      (let ((frame "ert-fail(((should (equal complex-list 1)) :form (equal ((:1 (:2 (:3 (:4 (:5 (:6 \"abc\"))))))) 1) :value nil :explanation (different-types ((:1 (:2 (:3 (:4 (:5 (:6 \"abc\"))))))) 1)))")
+            found-frame)
+	(cl-loop for msg in (reverse messages)
+		 do
+		 (unless found-frame
+		   (setq found-frame (cl-search frame msg :test 'equal))))
+        (should found-frame)))))
+
 (ert-deftest ert-test-special-operator-p ()
   (should (ert--special-operator-p 'if))
   (should-not (ert--special-operator-p 'car))
diff --git a/test/lisp/emacs-lisp/ert-x-tests.el b/test/lisp/emacs-lisp/ert-x-tests.el
index 9baa9941586..7106b7abc0c 100644
--- a/test/lisp/emacs-lisp/ert-x-tests.el
+++ b/test/lisp/emacs-lisp/ert-x-tests.el
@@ -103,23 +103,27 @@ ert-propertized-string
 
 (ert-deftest ert-test-run-tests-interactively-2 ()
   :tags '(:causes-redisplay)
-  (let* ((passing-test (make-ert-test :name 'passing-test
-                                      :body (lambda () (ert-pass))))
-         (failing-test (make-ert-test :name 'failing-test
-                                      :body (lambda ()
-                                              (ert-info ((propertize "foo\nbar"
-                                                                     'a 'b))
-                                                (ert-fail
-                                                 "failure message")))))
-         (skipped-test (make-ert-test :name 'skipped-test
-                                      :body (lambda () (ert-skip
-							"skip message"))))
-         (ert-debug-on-error nil)
-         (buffer-name (generate-new-buffer-name "*ert-test-run-tests*"))
-         (messages nil)
-         (mock-message-fn
-          (lambda (format-string &rest args)
-            (push (apply #'format format-string args) messages))))
+  (cl-letf* ((passing-test (make-ert-test
+                            :name 'passing-test
+                            :body (lambda () (ert-pass))))
+             (failing-test (make-ert-test
+                            :name 'failing-test
+                            :body (lambda ()
+                                    (ert-info ((propertize "foo\nbar"
+                                                           'a 'b))
+                                              (ert-fail
+                                               "failure message")))))
+             (skipped-test (make-ert-test
+                            :name 'skipped-test
+                            :body (lambda () (ert-skip
+					      "skip message"))))
+             (ert-debug-on-error nil)
+             (messages nil)
+             (buffer-name (generate-new-buffer-name "*ert-test-run-tests*"))
+             ((symbol-function 'message)
+              (lambda (format-string &rest args)
+                (push (apply #'format format-string args) messages)))
+             (ert--output-buffer-name buffer-name))
     (cl-flet ((expected-string (with-font-lock-p)
                 (ert-propertized-string
                  "Selector: (member <passing-test> <failing-test> "
@@ -152,14 +156,12 @@ ert-test-run-tests-interactively-2
                  "failing-test"
                  nil "\n    Info: " '(a b) "foo\n"
                  nil "          " '(a b) "bar"
-                 nil "\n    (ert-test-failed \"failure message\")\n\n\n"
-                 )))
+                 nil "\n    (ert-test-failed \"failure message\")\n\n\n")))
       (save-window-excursion
         (unwind-protect
             (let ((case-fold-search nil))
               (ert-run-tests-interactively
-               `(member ,passing-test ,failing-test ,skipped-test) buffer-name
-               mock-message-fn)
+               `(member ,passing-test ,failing-test ,skipped-test))
               (should (equal messages `(,(concat
                                           "Ran 3 tests, 1 results were "
                                           "as expected, 1 unexpected, "
-- 
2.33.1


[-- Attachment #3: Type: text/plain, Size: 32 bytes --]


-- 
Michael <sp1ff@runbox.com>

bug#51037: [PATCH] Make `print-level` & `print-length` customizable in ERT batch tests

[Lars Ingebrigtsen]

Michael <sp1ff@runbox.com> writes:

> New patch attached. Incorporates (I hope) these changes, as well
> as addresses Eli's issues.

Thanks; applied to Emacs 29 with some minor changes.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#51037: [PATCH] Make `print-level` & `print-length` customizable in ERT batch tests

[Filipp Gunbin]

On 16/11/2021 08:48 +0100, Lars Ingebrigtsen wrote:

> Michael <sp1ff@runbox.com> writes:
>
>> New patch attached. Incorporates (I hope) these changes, as well
>> as addresses Eli's issues.
>
> Thanks; applied to Emacs 29 with some minor changes.
>
> -- 
> (domestic pets only, the antidote for overdose, milk.)
>    bloggy blog: http://lars.ingebrigtsen.no

I've got this fixup patch for this, below.

Mainly this fixes `ert' interactive spec, where there was a leftover
second argument, nil.

However I'm not quite sure in this docstring fix:

-mode.  A value of nil will short-circuit this mechanism; line
-lengths will be completely determined by `ert-batch-line-length'
-and `ert-batch-line-level'.  Any other value will be temporarily

The docstring for backtrace-line-length says: "If set to nil or zero,
backtrace mode will not abbreviate the forms it prints."  So the above
sentence, which I removed, was indeed not true?  (a quick look over
its usages confirmed that to me, but I'd like someone else to check)

Thanks.


diff --git a/lisp/emacs-lisp/ert.el b/lisp/emacs-lisp/ert.el
index 36b4408dc8..7f9b7b22c7 100644
--- a/lisp/emacs-lisp/ert.el
+++ b/lisp/emacs-lisp/ert.el
@@ -101,12 +101,10 @@ ert-batch-backtrace-line-length
 forming them.  This variable governs the target maximum line
 length by manipulating these two variables while printing stack
 traces.  Setting this variable to t will re-use the value of
-`backtrace-line-length' while print stack traces in ERT batch
-mode.  A value of nil will short-circuit this mechanism; line
-lengths will be completely determined by `ert-batch-line-length'
-and `ert-batch-line-level'.  Any other value will be temporarily
-bound to `backtrace-line-length' when producing stack traces
-in batch mode.")
+`backtrace-line-length' while printing stack traces in ERT batch
+mode.  Any other value will be temporarily bound to
+`backtrace-line-length' when producing stack traces in batch
+mode.")
 
 (defface ert-test-result-expected '((((class color) (background light))
                                      :background "green1")
@@ -1447,8 +1445,6 @@ ert-run-tests-batch
                         (cond
                          ((eq ert-batch-backtrace-line-length t)
                           backtrace-line-length)
-                         ((eq ert-batch-backtrace-line-length nil)
-                          nil)
                          (t
                           ert-batch-backtrace-line-length)))
                        (print-level ert-batch-print-level)
@@ -2055,8 +2051,7 @@ ert-run-tests-interactively
            (read
             (completing-read (format-prompt "Run tests" default)
                              obarray #'ert-test-boundp nil nil
-                             'ert--selector-history default nil)))
-         nil))
+                             'ert--selector-history default nil)))))
   (let (buffer listener)
     (setq listener
           (lambda (event-type &rest event-args)

bug#51037: [PATCH] Make `print-level` & `print-length` customizable in ERT batch tests

[Lars Ingebrigtsen]

Filipp Gunbin <fgunbin@fastmail.fm> writes:

> The docstring for backtrace-line-length says: "If set to nil or zero,
> backtrace mode will not abbreviate the forms it prints."  So the above
> sentence, which I removed, was indeed not true?  (a quick look over
> its usages confirmed that to me, but I'd like someone else to check)

I think the intention is that a nil value of backtrace-line-length
shouldn't limit the lengths -- but I haven't tested it.  Doesn't it
work?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#51037: [PATCH] Make `print-level` & `print-length` customizable in ERT batch tests

[Filipp Gunbin]

On 18/11/2021 10:27 +0100, Lars Ingebrigtsen wrote:

> Filipp Gunbin <fgunbin@fastmail.fm> writes:
>
>> The docstring for backtrace-line-length says: "If set to nil or zero,
>> backtrace mode will not abbreviate the forms it prints."  So the above
>> sentence, which I removed, was indeed not true?  (a quick look over
>> its usages confirmed that to me, but I'd like someone else to check)
>
> I think the intention is that a nil value of backtrace-line-length
> shouldn't limit the lengths -- but I haven't tested it.  Doesn't it
> work?

ert-batch-print-length/level are 10 by default, and print-length/level
are let-bound to them during ert batch runs.  With these values, the
backtraces should be limited, if backtrace-line-length says so.

Then we go:

emacs --batch -l ert \
  --eval "
(setq ert-batch-backtrace-right-margin nil
      backtrace-line-length 50
      ert-batch-backtrace-line-length t)
" \
  --eval '(ert-deftest my-test () (error "test error"))' \
  -f ert-run-tests-batch-and-exit

The output is limited.

Try the same with ert-batch-backtrace-line-length nil in --eval, and the
output is not limited at all.  So none of (ert-batch-)print-level/length
get into play.

So I think I'll install my patch now..

Filipp

bug#51037: [PATCH] Make `print-level` & `print-length` customizable in ERT batch tests

[Michael]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Filipp Gunbin <fgunbin@fastmail.fm> writes:
>
>> The docstring for backtrace-line-length says: "If set to nil or 
>> zero,
>> backtrace mode will not abbreviate the forms it prints."  So 
>> the above
>> sentence, which I removed, was indeed not true?  (a quick look 
>> over
>> its usages confirmed that to me, but I'd like someone else to 
>> check)
>
> I think the intention is that a nil value of 
> backtrace-line-length
> shouldn't limit the lengths -- but I haven't tested it.  Doesn't 
> it
> work?

Yes, I'm a bit lost, Filipp... what is your concern? And I
thought I *had* tested that.

-- 
Michael <sp1ff@pobox.com>

bug#51037: [PATCH] Make `print-level` & `print-length` customizable in ERT batch tests

[Filipp Gunbin]

On 19/11/2021 07:24 -0800, Michael wrote:

> Lars Ingebrigtsen <larsi@gnus.org> writes:
>
>> Filipp Gunbin <fgunbin@fastmail.fm> writes:
>>
>>> The docstring for backtrace-line-length says: "If set to nil or
>>> zero,
>>> backtrace mode will not abbreviate the forms it prints."  So the
>>> above
>>> sentence, which I removed, was indeed not true?  (a quick look over
>>> its usages confirmed that to me, but I'd like someone else to
>>> check)
>>
>> I think the intention is that a nil value of backtrace-line-length
>> shouldn't limit the lengths -- but I haven't tested it.  Doesn't it
>> work?
>
> Yes, I'm a bit lost, Filipp... what is your concern? And I
> thought I *had* tested that.

That the docstring contained a sentence which is not true.  A minor
issue, of course.  I give examples here:
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=51037#117

(there was also fix of interactive spec of `ert-run-tests-interactively',
but that's obvious I think)

Filipp

bug#51037: [PATCH] Make `print-level` & `print-length` customizable in ERT batch tests

[Michael]

> On 19/11/2021 07:24 -0800, Michael wrote:
>
>> Lars Ingebrigtsen <larsi@gnus.org> writes:
>>
>>> Filipp Gunbin <fgunbin@fastmail.fm> writes:
>>>
>>>> The docstring for backtrace-line-length says: "If set to nil 
>>>> or
>>>> zero,
>>>> backtrace mode will not abbreviate the forms it prints."  So 
>>>> the
>>>> above
>>>> sentence, which I removed, was indeed not true?  (a quick 
>>>> look over
>>>> its usages confirmed that to me, but I'd like someone else to
>>>> check)
>>>
>>> I think the intention is that a nil value of 
>>> backtrace-line-length
>>> shouldn't limit the lengths -- but I haven't tested it. 
>>> Doesn't it
>>> work?
>>
>> Yes, I'm a bit lost, Filipp... what is your concern? And I
>> thought I *had* tested that.
>
> That the docstring contained a sentence which is not true.  A 
> minor
> issue, of course.  I give examples here:
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=51037#117
>
> (there was also fix of interactive spec of 
> `ert-run-tests-interactively',
> but that's obvious I think)
>
Hey Filipp,

Note that _is_ true, as far as I know, and was intended to be
true before my patch, and I verified that it became true with my
patch :) Can you provide a test case that demonstrates it to be
not true?

-- 
Michael <sp1ff@pobox.com>

bug#51037: [PATCH] Make `print-level` & `print-length` customizable in ERT batch tests

[Filipp Gunbin]

On 20/11/2021 08:49 -0800, Michael wrote:

>> On 19/11/2021 07:24 -0800, Michael wrote:
>>
>>> Lars Ingebrigtsen <larsi@gnus.org> writes:
>>>
>>>> Filipp Gunbin <fgunbin@fastmail.fm> writes:
>>>>
>>>>> The docstring for backtrace-line-length says: "If set to nil
>>>>> or
>>>>> zero,
>>>>> backtrace mode will not abbreviate the forms it prints."  So
>>>>> the
>>>>> above
>>>>> sentence, which I removed, was indeed not true?  (a quick
>>>>> look over
>>>>> its usages confirmed that to me, but I'd like someone else to
>>>>> check)
>>>>
>>>> I think the intention is that a nil value of
>>>> backtrace-line-length
>>>> shouldn't limit the lengths -- but I haven't tested it.
>>>> Doesn't it
>>>> work?
>>>
>>> Yes, I'm a bit lost, Filipp... what is your concern? And I
>>> thought I *had* tested that.
>>
>> That the docstring contained a sentence which is not true.  A
>> minor
>> issue, of course.  I give examples here:
>> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=51037#117
>>
>> (there was also fix of interactive spec of
>> `ert-run-tests-interactively',
>> but that's obvious I think)
>>
> Hey Filipp,
>
> Note that _is_ true, as far as I know, and was intended to be
> true before my patch, and I verified that it became true with my
> patch :) Can you provide a test case that demonstrates it to be
> not true?

Yes, here https://debbugs.gnu.org/cgi/bugreport.cgi?bug=51037#117, as I
wrote above..

bug#51037: [PATCH] Make `print-level` & `print-length` customizable in ERT batch tests

[Michael]

Filipp Gunbin <fgunbin@fastmail.fm> writes:

> On 20/11/2021 08:49 -0800, Michael wrote:
>
>>> On 19/11/2021 07:24 -0800, Michael wrote:
>>>
>>>> Lars Ingebrigtsen <larsi@gnus.org> writes:
>>>>
>>>>> Filipp Gunbin <fgunbin@fastmail.fm> writes:
>>>>>
>>>>>> The docstring for backtrace-line-length says: "If set to 
>>>>>> nil
>>>>>> or
>>>>>> zero,
>>>>>> backtrace mode will not abbreviate the forms it prints." 
>>>>>> So
>>>>>> the
>>>>>> above
>>>>>> sentence, which I removed, was indeed not true?  (a quick
>>>>>> look over
>>>>>> its usages confirmed that to me, but I'd like someone else 
>>>>>> to
>>>>>> check)
>>>>>
>>>>> I think the intention is that a nil value of
>>>>> backtrace-line-length
>>>>> shouldn't limit the lengths -- but I haven't tested it.
>>>>> Doesn't it
>>>>> work?
>>>>
>>>> Yes, I'm a bit lost, Filipp... what is your concern? And I
>>>> thought I *had* tested that.
>>>
>>> That the docstring contained a sentence which is not true.  A
>>> minor
>>> issue, of course.  I give examples here:
>>> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=51037#117
>>>
>>> (there was also fix of interactive spec of
>>> `ert-run-tests-interactively',
>>> but that's obvious I think)
>>>
>> Hey Filipp,
>>
>> Note that _is_ true, as far as I know, and was intended to be
>> true before my patch, and I verified that it became true with 
>> my
>> patch :) Can you provide a test case that demonstrates it to be
>> not true?
>
> Yes, here 
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=51037#117, as I
> wrote above..

Oh.... oh, my-- you are on to something here, Filipp. My
apologies for not seeing it sooner. The problem lies in:

    (defun cl-print-to-string-with-limit (print-function value 
    limit)
      ...
      (setq limit (and (natnump limit)
                       (not (zerop limit))
                       limit))
      ;; Since this is used by the debugger when stack space may 
      be
      ;; limited, if you increase print-level here, add more depth 
      in
      ;; call_debugger (bug#31919).
      (let* ((print-length (when limit (min limit 50)))
             (print-level (when limit (min 8 (truncate (log 
             limit)))))


backtrace uses this function to print each frame, and as you can
see, if called with `limit' set to nil, `print-le{vel,ength}`
will *not* keep their current values, they will *also* be set to
`nil`.

So, we could do either of:

    1. As Filipp suggests, just change the contract for
    `ert-batch-backtrace-line-length' to: if set to nil, you get
    full stack traces, period.

    2. change `cl-print-to-string-with-limit' to respect the
    existing values of `print-length' & `print-level' when
    `limit' is nil. Note that the only caller (in Emacs) of this
    function is the backtrace package.

I vote for option 2. What say you all?

-- 
Michael <sp1ff@pobox.com>

bug#51037: [PATCH] Make `print-level` & `print-length` customizable in ERT batch tests

[Filipp Gunbin]

On 22/11/2021 06:07 -0800, Michael wrote:

> So, we could do either of:
>
>    1. As Filipp suggests, just change the contract for
>    `ert-batch-backtrace-line-length' to: if set to nil, you get
>    full stack traces, period.
>
>    2. change `cl-print-to-string-with-limit' to respect the
>    existing values of `print-length' & `print-level' when
>    `limit' is nil. Note that the only caller (in Emacs) of this
>    function is the backtrace package.
>
> I vote for option 2. What say you all?

I think the right way is 1 (which I already installed) because then, as
the result of your patch, we get the ability to override
backtrace-line-length with ert-batch-backtrace-line-length, and nothing
more.

Changing the semantics of backtrace-line-length should be done (if done)
in a separate issue, IMO.

Personally, I like that LIMIT in cl-print-to-string-with-limit is "hard
override".

Filipp

bug#51037: [PATCH] Make `print-level` & `print-length` customizable in ERT batch tests

[Michael]

>> So, we could do either of:
>>
>>    1. As Filipp suggests, just change the contract for
>>    `ert-batch-backtrace-line-length' to: if set to nil, you get
>>    full stack traces, period.
>>
>>    2. change `cl-print-to-string-with-limit' to respect the
>>    existing values of `print-length' & `print-level' when
>>    `limit' is nil. Note that the only caller (in Emacs) of this
>>    function is the backtrace package.
>>
>> I vote for option 2. What say you all?
>
> I think the right way is 1 (which I already installed) because 
> then, as
> the result of your patch, we get the ability to override
> backtrace-line-length with ert-batch-backtrace-line-length, and 
> nothing
> more.
>
> Changing the semantics of backtrace-line-length should be done 
> (if done)
> in a separate issue, IMO.
>
> Personally, I like that LIMIT in cl-print-to-string-with-limit 
> is "hard
> override".

*shrugs* OK... so can we close this issue?

-- 
Michael <sp1ff@pobox.com>

bug#51037: [PATCH] Make `print-level` & `print-length` customizable in ERT batch tests

[Filipp Gunbin]

On 24/11/2021 08:29 -0800, Michael wrote:

>>> So, we could do either of:
>>>
>>>    1. As Filipp suggests, just change the contract for
>>>    `ert-batch-backtrace-line-length' to: if set to nil, you get
>>>    full stack traces, period.
>>>
>>>    2. change `cl-print-to-string-with-limit' to respect the
>>>    existing values of `print-length' & `print-level' when
>>>    `limit' is nil. Note that the only caller (in Emacs) of this
>>>    function is the backtrace package.
>>>
>>> I vote for option 2. What say you all?
>>
>> I think the right way is 1 (which I already installed) because
>> then, as
>> the result of your patch, we get the ability to override
>> backtrace-line-length with ert-batch-backtrace-line-length, and
>> nothing
>> more.
>>
>> Changing the semantics of backtrace-line-length should be done
>> (if done)
>> in a separate issue, IMO.
>>
>> Personally, I like that LIMIT in cl-print-to-string-with-limit
>> is "hard
>> override".
>
> *shrugs* OK... so can we close this issue?

But it's closed already?

bug#51037: [PATCH] Make `print-level` & `print-length` customizable in ERT batch tests

[Michael]

> On 24/11/2021 08:29 -0800, Michael wrote:
>
>>>> So, we could do either of:
>>>>
>>>>    1. As Filipp suggests, just change the contract for
>>>>    `ert-batch-backtrace-line-length' to: if set to nil, you 
>>>>    get
>>>>    full stack traces, period.
>>>>
>>>>    2. change `cl-print-to-string-with-limit' to respect the
>>>>    existing values of `print-length' & `print-level' when
>>>>    `limit' is nil. Note that the only caller (in Emacs) of 
>>>>    this
>>>>    function is the backtrace package.
>>>>
>>>> I vote for option 2. What say you all?
>>>
>>> I think the right way is 1 (which I already installed) because
>>> then, as
>>> the result of your patch, we get the ability to override
>>> backtrace-line-length with ert-batch-backtrace-line-length, 
>>> and
>>> nothing
>>> more.
>>>
>>> Changing the semantics of backtrace-line-length should be done
>>> (if done)
>>> in a separate issue, IMO.
>>>
>>> Personally, I like that LIMIT in cl-print-to-string-with-limit
>>> is "hard
>>> override".
>>
>> *shrugs* OK... so can we close this issue?
>
> But it's closed already?

So it is-- thanks.

-- 
Michael <sp1ff@pobox.com>

bug#51037: [PATCH] Make `print-level` & `print-length` customizable in ERT batch tests

[Andy Moreton]

On Thu 07 Oct 2021, Lars Ingebrigtsen wrote:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> Why do we limit print-length in batch mode?  In interactive session,
>> one can click or type RET on the ellipsis and get it expanded, but no
>> such joy in batch.  See
>>
>>   https://lists.gnu.org/archive/html/emacs-devel/2021-10/msg00395.html
>>
>> for how that makes debugging more difficult than it has to be.
>>
>> So how about using huge values in batch?  Are there any downsides?
>
> It'll make the error summaries completely unreadable, which isn't what
> you want when doing a "make check" -- you just want to see that there's
> a problem and where it is.  (Well, at least I do.)
>
> But when making a specific test (i.e., "make subr-tests" etc) it would
> indeed be nice to have untruncated backtraces.  I think adding these new
> variables would allow us to do this.

Isn't that what ert-batch-backtrace-right-margin is for ?

  (defcustom ert-batch-backtrace-right-margin 70
    "Maximum length of lines in ERT backtraces in batch mode.
  Use nil for no limit (caution: backtrace lines can be very long)."
    :type '(choice (const :tag "No truncation" nil) integer))

Also the TEST_BACKTRACE_LINE_LENGTH environment variable can be
used to set it (see test/Makefile.in).

    AndyM

bug#51037: [PATCH] Make `print-level` & `print-length` customizable in ERT batch tests

[Lars Ingebrigtsen]

Andy Moreton <andrewjmoreton@gmail.com> writes:

> Isn't that what ert-batch-backtrace-right-margin is for ?
>
>   (defcustom ert-batch-backtrace-right-margin 70
>     "Maximum length of lines in ERT backtraces in batch mode.
>   Use nil for no limit (caution: backtrace lines can be very long)."
>     :type '(choice (const :tag "No truncation" nil) integer))
>
> Also the TEST_BACKTRACE_LINE_LENGTH environment variable can be
> used to set it (see test/Makefile.in).

Right.  But ert will still shorten "internally" with print-length etc
before chopping the lines to 70 characters.  Here's with a longer value:

est test-ensure-list2 backtrace:
  signal(error ("(\"foo\" \"foo\" \"foo\" \"foo\" \"foo\" \"foo\" \"foo\" \"foo\" \"..."))
  error("%S" ("foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" ...))
  (closure (t) nil (error "%S" (make-list 100 "foo")))()

Note the "..." at the end of the foo list -- that's from print-length.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#51037: [PATCH] Make `print-level` & `print-length` customizable in ERT batch tests

[Andy Moreton]

On Fri 08 Oct 2021, Lars Ingebrigtsen wrote:

> Andy Moreton <andrewjmoreton@gmail.com> writes:
>
>> Isn't that what ert-batch-backtrace-right-margin is for ?
>>
>>   (defcustom ert-batch-backtrace-right-margin 70
>>     "Maximum length of lines in ERT backtraces in batch mode.
>>   Use nil for no limit (caution: backtrace lines can be very long)."
>>     :type '(choice (const :tag "No truncation" nil) integer))
>>
>> Also the TEST_BACKTRACE_LINE_LENGTH environment variable can be
>> used to set it (see test/Makefile.in).
>
> Right.  But ert will still shorten "internally" with print-length etc
> before chopping the lines to 70 characters.  Here's with a longer value:
>
> est test-ensure-list2 backtrace:
>   signal(error ("(\"foo\" \"foo\" \"foo\" \"foo\" \"foo\" \"foo\" \"foo\" \"foo\" \"..."))
>   error("%S" ("foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo"
> "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo"
> "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo"
> "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo" "foo"
> "foo" ...))
>   (closure (t) nil (error "%S" (make-list 100 "foo")))()
>
> Note the "..." at the end of the foo list -- that's from print-length.


Good point. The truncation would not be so bothersome if there were
separate console and logfile streams (with the truncation only happening
in the console stream, and the full details in the logfile stream).

Anything that can make failed tests easier to diagnose is a useful
improvement.

    AndyM

bug#51037: [PATCH] Make `print-level` & `print-length` customizable in ERT batch tests

[Michael]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Michael <sp1ff@runbox.com> writes:
>
>> When running ERT tests in batch mode, the conservative values
>> chosen for `print-level` and `print-length` sometimes make it
>> difficult to see what exactly is wrong.  This patch introduces
>> two new variables (`ert-batch-print-level` &
>> `ert-batch-print-length`) that one can use to customize them;
>> e.g.
>>
>>    emacs -batch -l ert -l my-tests.el \
>>          --eval "(let ((ert-batch-print-level 10) \
>>                        (ert-batch-print-length 120)) \
>>                    (ert-run-tests-batch-and-exit))"
>
> Sounds like a good idea.
>
>> +MESSAGE-FN should normally be nil; it is used for automated
>> +self-tests and specify how to display messages."
>
> I don't understand this bit, though.  If you want to test how 
> this
> function outputs messages, you can just `cl-letf' like this:
>
> (cl-letf (((symbol-function 'message) ...

Good point. I simply carried the strategy over from that used to
unit test `ert-run-tests-interactively`. TBH I was unaware of
`cl-letf`, and there *is* this comment above
`ert-run-tests-interactively`:

    ;; Should OUTPUT-BUFFER-NAME and MESSAGE-FN really be 
    arguments here?
    ;; They are needed only for our automated self-tests at the 
    moment.
    ;; Or should there be some other mechanism?

Perhaps I should change *both* to just use `cl-letf`?

-- 
Michael <sp1ff@runbox.com>

bug#51037: [PATCH] Make `print-level` & `print-length` customizable in ERT batch tests

[Lars Ingebrigtsen]

Michael <sp1ff@runbox.com> writes:

> Good point. I simply carried the strategy over from that used to
> unit test `ert-run-tests-interactively`. TBH I was unaware of
> `cl-letf`, and there *is* this comment above
> `ert-run-tests-interactively`:
>
>    ;; Should OUTPUT-BUFFER-NAME and MESSAGE-FN really be     arguments
>      here?
>    ;; They are needed only for our automated self-tests at the
>       moment.
>    ;; Or should there be some other mechanism?
>
> Perhaps I should change *both* to just use `cl-letf`?

Sounds good to me.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no