From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Michael Newsgroups: gmane.emacs.bugs Subject: bug#51037: [PATCH] Make `print-level` & `print-length` customizable in ERT batch tests Date: Tue, 05 Oct 2021 07:28:38 -0700 Message-ID: <86tuhvmtpl.fsf@runbox.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="8921"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: mu4e 1.4.15; emacs 28.0.50 To: 51037@debbugs.gnu.org Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Tue Oct 05 17:11:57 2021 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mXm6m-00022U-Lw for geb-bug-gnu-emacs@m.gmane-mx.org; Tue, 05 Oct 2021 17:11:57 +0200 Original-Received: from localhost ([::1]:40262 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mXm6k-0004Ur-Ml for geb-bug-gnu-emacs@m.gmane-mx.org; Tue, 05 Oct 2021 11:11:54 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:56936) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mXlmY-000872-T9 for bug-gnu-emacs@gnu.org; Tue, 05 Oct 2021 10:51:03 -0400 Original-Received: from debbugs.gnu.org ([209.51.188.43]:58301) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mXlmY-0001dP-Lw for bug-gnu-emacs@gnu.org; Tue, 05 Oct 2021 10:51:02 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1mXlmY-0005Fv-Gk for bug-gnu-emacs@gnu.org; Tue, 05 Oct 2021 10:51:02 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Michael Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Tue, 05 Oct 2021 14:51:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 51037 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: patch X-Debbugs-Original-To: bug-gnu-emacs@gnu.org Original-Received: via spool by submit@debbugs.gnu.org id=B.163344542620155 (code B ref -1); Tue, 05 Oct 2021 14:51:02 +0000 Original-Received: (at submit) by debbugs.gnu.org; 5 Oct 2021 14:50:26 +0000 Original-Received: from localhost ([127.0.0.1]:41614 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mXllx-0005F0-Da for submit@debbugs.gnu.org; Tue, 05 Oct 2021 10:50:26 -0400 Original-Received: from lists.gnu.org ([209.51.188.17]:47572) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mXlR7-0004f0-Mt for submit@debbugs.gnu.org; Tue, 05 Oct 2021 10:28:56 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:51586) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mXlR7-0007gD-1L for bug-gnu-emacs@gnu.org; Tue, 05 Oct 2021 10:28:53 -0400 Original-Received: from aibo.runbox.com ([91.220.196.211]:50246) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mXlR3-0007zc-Of for bug-gnu-emacs@gnu.org; Tue, 05 Oct 2021 10:28:52 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=runbox.com; s=selector2; h=Content-Type:MIME-Version:Message-ID:Date:Subject:To:From; bh=7/3eDTsyjMGNM4OpzE61Iqu+epn8QY/fZpb6fTNGDzM=; b=Qxtj9l4Da9JCvqSGUHCYa8k8I p2LoHOfPwzsIni1mYPcqNO5JePu3Q4FMbpdriVM4V93v0gyZZo7oQ9AEaefxR+TIwWJ8AN1F/Wy45 u+RrHvagZwRoNNxGdjXN6/bZ5BwBjKDo6Uk2/R0YiVXNocvd6g+CuYj9+/FG+N2NSte/OwFs7gvtT sJdtT3n4didukhYiebLe3nLSwJTjQF9EiFHYSYnQ8ICoTeVCgNKRamGSlKOOP9aCtS9OD1Yosbqja EwoBuHUa9FaXZf4nNUGm3pDW1j44+Kbz/G1to6uBBCCiPHoUdxCRybPn8fybzlTlcuvAV49aMp7Vx /R98KEUHQ==; Original-Received: from [10.9.9.73] (helo=submission02.runbox) by mailtransmit02.runbox with esmtp (Exim 4.86_2) (envelope-from ) id 1mXlQz-0003Cn-Lu for bug-gnu-emacs@gnu.org; Tue, 05 Oct 2021 16:28:45 +0200 Original-Received: by submission02.runbox with esmtpsa [Authenticated ID (942723)] (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) id 1mXlQw-0008Je-5k for bug-gnu-emacs@gnu.org; Tue, 05 Oct 2021 16:28:42 +0200 Received-SPF: pass client-ip=91.220.196.211; envelope-from=sp1ff@runbox.com; helo=aibo.runbox.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Mailman-Approved-At: Tue, 05 Oct 2021 10:50:24 -0400 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.io gmane.emacs.bugs:216461 Archived-At: --=-=-= Content-Type: text/plain; format=flowed Hello, When running ERT tests in batch mode, the conservative values chosen for `print-level` and `print-length` sometimes make it difficult to see what exactly is wrong. This patch introduces two new variables (`ert-batch-print-level` & `ert-batch-print-length`) that one can use to customize them; e.g. emacs -batch -l ert -l my-tests.el \ --eval "(let ((ert-batch-print-level 10) \ (ert-batch-print-length 120)) \ (ert-run-tests-batch-and-exit))" Please let me know what should be changed. --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0001-Make-print-length-print-level-in-ert-run-tests-batch.patch Content-Description: 0001-Make-print-length-print-level-in-ert-run-tests-batch.patch >From da7eff94f23f4798df36be2c2bd072038ba01e2d Mon Sep 17 00:00:00 2001 From: Michael Herstine Date: Tue, 5 Oct 2021 06:53:02 -0700 Subject: [PATCH] Make print-length & print-level in ert-run-tests-batch configurable This commit introduces two new ert variables (ert-batch-print-length and ert-batch-print-level) that make these settings configurable. It also adds an optional message-fn parameter to ert-batch-test (in the style of of ert-run-tests-interactively) to facilitate testing. * lisp/emacs-lisp/ert.el (ert-batch-print-length, ert-batch-print-level, ert-batch-test): Added the two variables, added optional message-fn parameter, set print-level & print-length to these settings when formatting test results. * test/lisp/emacs-lisp/ert-tests.el (ert-test-run-tests-batch): new tests * doc/misc/ert.texi: document the new variables & their usage --- doc/misc/ert.texi | 16 ++++++++ etc/NEWS | 5 +++ lisp/emacs-lisp/ert.el | 61 +++++++++++++++++++------------ test/lisp/emacs-lisp/ert-tests.el | 34 +++++++++++++++++ 4 files changed, 93 insertions(+), 23 deletions(-) diff --git a/doc/misc/ert.texi b/doc/misc/ert.texi index fafdb8c4eb4..12f1df75fcc 100644 --- a/doc/misc/ert.texi +++ b/doc/misc/ert.texi @@ -323,6 +323,22 @@ Running Tests in Batch Mode emacs -batch -l ert -f ert-summarize-tests-batch-and-exit output.log @end example +@vindex ert-batch-print-level +@vindex ert-batch-print-length +ERT attempts to limit the output size for failed tests by choosing +conservative values for @code{print-level} & @code{print-length} +when printing Lisp values. This can in some cases make it difficult +to see which portions of those values are incorrect. Use +@code{ert-batch-print-level} and @code{ert-batch-print-length} +to customize that: + +@example +emacs -batch -l ert -l my-tests.el \ + --eval "(let ((ert-batch-print-level 10) \ + (ert-batch-print-length 120)) \ + (ert-run-tests-batch-and-exit))" +@end example + @vindex ert-quiet By default, ERT in batch mode is quite verbose, printing a line with result after each test. This gives you progress information: how many diff --git a/etc/NEWS b/etc/NEWS index 17c42ce104d..be102dbc501 100644 --- a/etc/NEWS +++ b/etc/NEWS @@ -128,6 +128,11 @@ of files visited via 'C-x C-f' and other commands. * Changes in Emacs 28.1 ++++ +** New ERT batch variables 'ert-batch-print-length' & 'ert-batch-print-level' +These variables will override 'print-length' & 'print-level' when +printing Lisp values in ERT batch test results. + --- ** Emacs now supports Unicode Standard version 14.0. diff --git a/lisp/emacs-lisp/ert.el b/lisp/emacs-lisp/ert.el index 92acfe7246f..2f0b5536d7e 100644 --- a/lisp/emacs-lisp/ert.el +++ b/lisp/emacs-lisp/ert.el @@ -76,6 +76,18 @@ ert-batch-backtrace-right-margin Use nil for no limit (caution: backtrace lines can be very long)." :type '(choice (const :tag "No truncation" nil) integer)) +(defvar ert-batch-print-length 10 + "`print-length' setting used in `ert-run-tests-batch'. + +When formatting lists in test results, `print-length' will be +temporarily set to this value.") + +(defvar ert-batch-print-level 5 + "`print-level' setting used in `ert-run-tests-batch'. + +When formatting lists in test results, `print-level' will be +temporarily set to this value.") + (defface ert-test-result-expected '((((class color) (background light)) :background "green1") (((class color) (background dark)) @@ -1338,7 +1350,7 @@ ert-quiet "Non-nil makes ERT only print important information in batch mode.") ;;;###autoload -(defun ert-run-tests-batch (&optional selector) +(defun ert-run-tests-batch (&optional selector message-fn) "Run the tests specified by SELECTOR, printing results to the terminal. SELECTOR works as described in `ert-select-tests', except if @@ -1346,8 +1358,12 @@ ert-run-tests-batch run; this makes the command line \"emacs -batch -l my-tests.el -f ert-run-tests-batch-and-exit\" useful. -Returns the stats object." +Returns the stats object. + +MESSAGE-FN should normally be nil; it is used for automated +self-tests and specify how to display messages." (unless selector (setq selector 't)) + (unless message-fn (setq message-fn #'message)) (ert-run-tests selector (lambda (event-type &rest event-args) @@ -1355,7 +1371,7 @@ ert-run-tests-batch (run-started (unless ert-quiet (cl-destructuring-bind (stats) event-args - (message "Running %s tests (%s, selector `%S')" + (funcall message-fn "Running %s tests (%s, selector `%S')" (length (ert--stats-tests stats)) (ert--format-time-iso8601 (ert--stats-start-time stats)) selector)))) @@ -1364,7 +1380,7 @@ ert-run-tests-batch (let ((unexpected (ert-stats-completed-unexpected stats)) (skipped (ert-stats-skipped stats)) (expected-failures (ert--stats-failed-expected stats))) - (message "\n%sRan %s tests, %s results as expected, %s unexpected%s (%s, %f sec)%s\n" + (funcall message-fn "\n%sRan %s tests, %s results as expected, %s unexpected%s (%s, %f sec)%s\n" (if (not abortedp) "" "Aborted: ") @@ -1383,44 +1399,43 @@ ert-run-tests-batch "" (format "\n%s expected failures" expected-failures))) (unless (zerop unexpected) - (message "%s unexpected results:" unexpected) + (funcall message-fn "%s unexpected results:" unexpected) (cl-loop for test across (ert--stats-tests stats) for result = (ert-test-most-recent-result test) do (when (not (ert-test-result-expected-p test result)) - (message "%9s %S%s" + (funcall message-fn "%9s %S%s" (ert-string-for-test-result result nil) (ert-test-name test) (if (getenv "EMACS_TEST_VERBOSE") (ert-reason-for-test-result result) "")))) - (message "%s" "")) + (funcall message-fn "%s" "")) (unless (zerop skipped) - (message "%s skipped results:" skipped) + (funcall message-fn "%s skipped results:" skipped) (cl-loop for test across (ert--stats-tests stats) for result = (ert-test-most-recent-result test) do (when (ert-test-result-type-p result :skipped) - (message "%9s %S%s" + (funcall message-fn "%9s %S%s" (ert-string-for-test-result result nil) (ert-test-name test) (if (getenv "EMACS_TEST_VERBOSE") (ert-reason-for-test-result result) "")))) - (message "%s" ""))))) - (test-started - ) + (funcall message-fn "%s" ""))))) + (test-started) (test-ended (cl-destructuring-bind (stats test result) event-args (unless (ert-test-result-expected-p test result) (cl-etypecase result (ert-test-passed - (message "Test %S passed unexpectedly" (ert-test-name test))) + (funcall message-fn "Test %S passed unexpectedly" (ert-test-name test))) (ert-test-result-with-condition - (message "Test %S backtrace:" (ert-test-name test)) + (funcall message-fn "Test %S backtrace:" (ert-test-name test)) (with-temp-buffer (insert (backtrace-to-string (ert-test-result-with-condition-backtrace result))) (if (not ert-batch-backtrace-right-margin) - (message "%s" + (funcall message-fn "%s" (buffer-substring-no-properties (point-min) (point-max))) (goto-char (point-min)) @@ -1430,33 +1445,33 @@ ert-run-tests-batch (setq end (min end (+ start ert-batch-backtrace-right-margin))) - (message "%s" (buffer-substring-no-properties + (funcall message-fn "%s" (buffer-substring-no-properties start end))) (forward-line 1)))) (with-temp-buffer (ert--insert-infos result) (insert " ") (let ((print-escape-newlines t) - (print-level 5) - (print-length 10)) + (print-level ert-batch-print-level) + (print-length ert-batch-print-length)) (ert--pp-with-indentation-and-newline (ert-test-result-with-condition-condition result))) (goto-char (1- (point-max))) (cl-assert (looking-at "\n")) (delete-char 1) - (message "Test %S condition:" (ert-test-name test)) - (message "%s" (buffer-string)))) + (funcall message-fn "Test %S condition:" (ert-test-name test)) + (funcall message-fn "%s" (buffer-string)))) (ert-test-aborted-with-non-local-exit - (message "Test %S aborted with non-local exit" + (funcall message-fn "Test %S aborted with non-local exit" (ert-test-name test))) (ert-test-quit - (message "Quit during %S" (ert-test-name test))))) + (funcall message-fn "Quit during %S" (ert-test-name test))))) (unless ert-quiet (let* ((max (prin1-to-string (length (ert--stats-tests stats)))) (format-string (concat "%9s %" (prin1-to-string (length max)) "s/" max " %S (%f sec)"))) - (message format-string + (funcall message-fn format-string (ert-string-for-test-result result (ert-test-result-expected-p test result)) diff --git a/test/lisp/emacs-lisp/ert-tests.el b/test/lisp/emacs-lisp/ert-tests.el index 5c9696105e9..d4872e0bab7 100644 --- a/test/lisp/emacs-lisp/ert-tests.el +++ b/test/lisp/emacs-lisp/ert-tests.el @@ -551,6 +551,40 @@ ert-test-run-tests-interactively (when (get-buffer buffer-name) (kill-buffer buffer-name)))))))) +(ert-deftest ert-test-run-tests-batch () + (let* ((complex-list '((:1 (:2 (:3 (:4 (:5 (:6 "abc")))))))) + (long-list (make-list 121 1)) + (failing-test-1 + (make-ert-test :name 'failing-test-1 + :body (lambda () (should (equal complex-list 1))))) + (failing-test-2 + (make-ert-test :name 'failing-test-2 + :body (lambda () (should (equal long-list 1)))))) + (let ((ert-debug-on-error nil)) + (let* ((messages nil) + (mock-message-fn + (lambda (format-string &rest args) + (push (apply #'format format-string args) messages)))) + (save-window-excursion + (unwind-protect + (let ((case-fold-search nil) + (ert-batch-print-level nil) + (ert-batch-print-length nil)) + (ert-run-tests-batch + `(member ,failing-test-1 ,failing-test-2) + mock-message-fn)))) + (let ((long-text (mapconcat #'identity (make-list 121 "1") " ")) + (complex-text "(:6 \"abc\")") + found-complex found-long) + (cl-loop for msg in messages + do + (unless found-long + (setq found-long (cl-search long-text msg :test 'equal))) + (unless found-complex + (setq found-complex (cl-search complex-text msg :test 'equal)))) + (should found-complex) + (should found-long)))))) + (ert-deftest ert-test-special-operator-p () (should (ert--special-operator-p 'if)) (should-not (ert--special-operator-p 'car)) -- 2.33.0 --=-=-= Content-Type: text/plain; format=flowed -- Michael --=-=-=--