From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Mike Kupfer Newsgroups: gmane.emacs.bugs Subject: bug#50629: 28.0.50; hard to debug an uncaught error with ert Date: Thu, 16 Sep 2021 16:28:08 -0700 Message-ID: <15445.1631834888@alto> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="12127"; mail-complaints-to="usenet@ciao.gmane.io" To: 50629@debbugs.gnu.org Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Fri Sep 17 01:29:29 2021 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mR0oq-0002ze-UF for geb-bug-gnu-emacs@m.gmane-mx.org; Fri, 17 Sep 2021 01:29:29 +0200 Original-Received: from localhost ([::1]:43736 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mR0op-0007AI-Sl for geb-bug-gnu-emacs@m.gmane-mx.org; Thu, 16 Sep 2021 19:29:28 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:51232) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mR0oR-0007A1-5L for bug-gnu-emacs@gnu.org; Thu, 16 Sep 2021 19:29:03 -0400 Original-Received: from debbugs.gnu.org ([209.51.188.43]:46282) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mR0oQ-0004NN-Tz for bug-gnu-emacs@gnu.org; Thu, 16 Sep 2021 19:29:02 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1mR0oQ-0004Ht-Op for bug-gnu-emacs@gnu.org; Thu, 16 Sep 2021 19:29:02 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Mike Kupfer Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Thu, 16 Sep 2021 23:29:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 50629 X-GNU-PR-Package: emacs X-Debbugs-Original-To: bug-gnu-emacs@gnu.org Original-Received: via spool by submit@debbugs.gnu.org id=B.163183490316428 (code B ref -1); Thu, 16 Sep 2021 23:29:02 +0000 Original-Received: (at submit) by debbugs.gnu.org; 16 Sep 2021 23:28:23 +0000 Original-Received: from localhost ([127.0.0.1]:57828 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mR0nm-0004Gs-Kx for submit@debbugs.gnu.org; Thu, 16 Sep 2021 19:28:23 -0400 Original-Received: from lists.gnu.org ([209.51.188.17]:46800) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mR0ni-0004Gj-TQ for submit@debbugs.gnu.org; Thu, 16 Sep 2021 19:28:21 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:51046) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mR0ni-0006t2-OY for bug-gnu-emacs@gnu.org; Thu, 16 Sep 2021 19:28:18 -0400 Original-Received: from shell1.rawbw.com ([198.144.192.42]:13847) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mR0ng-0003oZ-I5 for bug-gnu-emacs@gnu.org; Thu, 16 Sep 2021 19:28:18 -0400 Original-Received: from alto (96-95-200-133-static.hfc.comcastbusiness.net [96.95.200.133]) (authenticated bits=0) by shell1.rawbw.com (8.15.1/8.15.1) with ESMTPSA id 18GNS8K8060534 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Thu, 16 Sep 2021 16:28:14 -0700 (PDT) (envelope-from mkupfer@alum.berkeley.edu) X-Authentication-Warning: shell1.rawbw.com: Host 96-95-200-133-static.hfc.comcastbusiness.net [96.95.200.133] claimed to be alto X-Mailer: MH-E 8.6+git; nmh 1.7.1; GNU Emacs 28.0.50 Received-SPF: none client-ip=198.144.192.42; envelope-from=mkupfer@alum.berkeley.edu; helo=shell1.rawbw.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.io gmane.emacs.bugs:214514 Archived-At: (This is similar to bug#49805, but I'm filing a new bug in case this issue needs a different resolution than #49805.) Consider the following stripped-down test case: $ emacs -Q -batch -l ert -eval '(ert-deftest f () (defun foo () (bar)) (def= un bar () (error "bar")) (should (equal 42 (foo))))' -f ert-run-tests-batch= -and-exit It produces -----8<-----8<----- Running 1 tests (2021-09-16 15:55:17-0700) Test f backtrace: signal(error ("bar")) apply(signal (error ("bar"))) (setq value-2 (apply fn-0 args-1)) (unwind-protect (setq value-2 (apply fn-0 args-1)) (setq form-descri (if (unwind-protect (setq value-2 (apply fn-0 args-1)) (setq form-de (let (form-description-4) (if (unwind-protect (setq value-2 (apply f (let ((value-2 'ert-form-evaluation-aborted-3)) (let (form-descripti (let* ((fn-0 (function equal)) (args-1 (condition-case err (let ((si (lambda nil (defalias 'foo (function (lambda nil (bar)))) (defalias=20 ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test ert-run-test(#s(ert-test :name f :documentation nil :body (lambda ni ert-run-or-rerun-test(#s(ert--stats :selector t :tests [#s(ert-test=20 ert-run-tests(t #f(compiled-function (event-type &rest event-args) # ert-run-tests-batch(nil) ert-run-tests-batch-and-exit() command-line-1(("-l" "ert" "-eval" "(ert-deftest f () (defun foo ()=20 command-line() normal-top-level() Test f condition: (error "bar") FAILED 1/1 f Ran 1 tests, 0 results as expected, 1 unexpected (2021-09-16 15:55:17-0700) 1 unexpected results: FAILED f ----->8----->8----- Knowing that error "bar" was raised is helpful, but in a more realistic test scenario, it can take some time to figure out where the error was raised. Setting debug-on-error to t doesn't help (does ert hijack the normal error handling mechanisms?). The only way I know of to track it down is to run the test multiple times, and use #'message at key points in the code to record execution progress. I ran into this while running some (not-yet-integrated) ert tests for MH-E. The message in the *ert* buffer looked like F mh-folder-completion-function-10-plus-slash-abs-folder Test =E2=80=98mh-folder-completion-function=E2=80=99 with =E2=80=98+/= abso-folder=E2=80=99. (args-out-of-range "folder/bar" 44 nil) The backtrace produced by ert was over 1MB long, and I was unable to use it to determine where the error was coming from. Here are two suggestions for how to make this better. 1. When ert detects an uncaught error, capture the backtrace at that time and include the backtrace in the results. 2. Provide a way for an interactive user to disable whatever ert is doing that lets it capture errors and drive on. Instead, just let the error be raised, and let the user use the Lisp debugger to investigate. In GNU Emacs 28.0.50 (build 2, x86_64-pc-linux-gnu, X toolkit, cairo versio= n 1.16.0, Xaw scroll bars) of 2021-09-02 built on deb10 Repository revision: 6885c62a104dcf52378860d8f951dbdce52b73c5 Repository branch: master Windowing system distributor 'The X.Org Foundation', version 11.0.12004000 System Description: Debian GNU/Linux 10 (buster) Configured using: 'configure --prefix=3D/usr/new' Configured features: CAIRO FREETYPE GIF GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG LIBSELINUX LIBXML2 MODULES NOTIFY INOTIFY PDUMPER PNG SECCOMP SOUND THREADS TIFF TOOLKIT_SCROLL_BARS X11 XDBE XIM XPM LUCID ZLIB Important settings: value of $LC_TIME: C value of $LANG: en_US.UTF-8 locale-coding-system: utf-8-unix Major mode: ERT-Results Minor modes in effect: shell-dirtrack-mode: t delete-selection-mode: t global-eldoc-mode: t mouse-wheel-mode: t file-name-shadow-mode: t global-font-lock-mode: t font-lock-mode: t auto-composition-mode: t auto-encryption-mode: t auto-compression-mode: t buffer-read-only: t indent-tabs-mode: t transient-mark-mode: t Load-path shadows: None found. Features: (shadow mh-mime mh-identity mh-tool-bar mh-letter mh-show goto-addr gnus-cite gnus-art mm-uu mml2015 gnus-sum shr kinsoku svg dom browse-url url url-proxy url-privacy url-expand url-methods url-history url-cookie url-domsuf url-util url-parse url-vars gnus-group gnus-undo gnus-start gnus-dbus dbus xml gnus-cloud nnimap nnmail mail-source utf7 netrc nnoo parse-time iso8601 gnus-spec gnus-int gnus-win gnus-range mh-xface mh-comp mh-scan mh-gnus mm-view mml-smime smime dig mailcap emacsbug eieio-opt speedbar ezimage dframe shortdoc apropos thingatpt help-fns radix-tree mule-util jka-compr info misearch multi-isearch ediff-vers ediff ediff-merg ediff-mult ediff-wind ediff-diff ediff-help ediff-init ediff-util smerge-mode diff whitespace cl-extra gnus nnheader wid-edit mdk-mail smtpmail sendmail message rmc puny rfc822 mml mml-sec epa derived epg rfc6068 epg-config gnus-util rmail rmail-loaddefs auth-source eieio eieio-core eieio-loaddefs password-cache json map text-property-search time-date subr-x mm-decode mm-bodies mm-encode mail-parse rfc2231 rfc2047 rfc2045 mm-util ietf-drums mail-prsvr mail-utils gmm-utils mailheader cl-macs mh-utils mh-e mh-compat mailabbrev mh-buffers mh-loaddefs ert pp ewoc debug backtrace help-mode find-func vc-git diff-mode bug-reference dired-aux dired-x seq gv dired dired-loaddefs server noutline outline easy-mmode cc-mode cc-fonts cc-guess cc-menus cc-cmds cc-styles cc-align cc-engine cc-vars cc-defs byte-opt bytecomp byte-compile cconv shell pcomplete comint ansi-color ring xcscope advice delsel cl-seq vc vc-dispatcher timeclock cl-loaddefs cl-lib mdk-hacks iso-transl tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type mwheel term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe tabulated-list replace newcomment text-mode elisp-mode lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu timer select scroll-bar mouse jit-lock font-lock syntax font-core term/tty-colors frame minibuffer cl-generic cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european ethiopic indian cyrillic chinese composite charscript charprop case-table epa-hook jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice button loaddefs faces cus-face macroexp files window text-properties overlay sha1 md5 base64 format env code-pages mule custom widget hashtable-print-readable backquote threads inotify dynamic-setting system-font-setting font-render-setting cairo x-toolkit x multi-tty make-network-process emacs) Memory information: ((conses 16 314657 10561) (symbols 48 22275 4) (strings 32 75491 1172) (string-bytes 1 2515695) (vectors 16 38901) (vector-slots 8 438662 8550) (floats 8 300 273) (intervals 56 27726 0) (buffers 992 25))