From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED!not-for-mail From: Alex Newsgroups: gmane.emacs.bugs Subject: bug#24402: should-error doesn't catch all errors Date: Thu, 13 Jul 2017 22:45:15 -0600 Message-ID: <878tjrlj8k.fsf@lylat> References: <3654D8E9-D3CB-402B-922F-B132C1871E9F@runbox.com> <596E65D2-E780-43A1-A75B-603B61B6F9F4@runbox.com> <87zickhoco.fsf_-_@lylat> <877eznda7v.fsf@lylat> <874lur0zki.fsf@calancha-pc> <87o9sywtbz.fsf@lylat> <87fue3f9p8.fsf@users.sourceforge.net> <87vamyl3j3.fsf@lylat> <87tw2het1b.fsf@users.sourceforge.net> <874luhbo4l.fsf@lylat> <87lgntdswo.fsf@users.sourceforge.net> <87k23d9gvt.fsf@lylat> <87d195dmr0.fsf@users.sourceforge.net> <87lgns7y7g.fsf@lylat> <877ezbew3d.fsf@users.sourceforge.net> <87d193ljdp.fsf@lylat> NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Trace: blaine.gmane.org 1500007584 32075 195.159.176.226 (14 Jul 2017 04:46:24 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Fri, 14 Jul 2017 04:46:24 +0000 (UTC) User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.0.50 (gnu/linux) Cc: 24402@debbugs.gnu.org To: npostavs@users.sourceforge.net Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Fri Jul 14 06:46:18 2017 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dVsUS-0007Zn-Oq for geb-bug-gnu-emacs@m.gmane.org; Fri, 14 Jul 2017 06:46:08 +0200 Original-Received: from localhost ([::1]:35401 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dVsUY-0002Pn-4E for geb-bug-gnu-emacs@m.gmane.org; Fri, 14 Jul 2017 00:46:14 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:57146) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dVsUR-0002Pf-Sa for bug-gnu-emacs@gnu.org; Fri, 14 Jul 2017 00:46:09 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dVsUM-0002Ks-W8 for bug-gnu-emacs@gnu.org; Fri, 14 Jul 2017 00:46:07 -0400 Original-Received: from debbugs.gnu.org ([208.118.235.43]:34927) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1dVsUM-0002KY-R8 for bug-gnu-emacs@gnu.org; Fri, 14 Jul 2017 00:46:02 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1dVsUM-00077P-Ih for bug-gnu-emacs@gnu.org; Fri, 14 Jul 2017 00:46:02 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Alex Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Fri, 14 Jul 2017 04:46:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 24402 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: confirmed patch Original-Received: via spool by 24402-submit@debbugs.gnu.org id=B24402.150000752627297 (code B ref 24402); Fri, 14 Jul 2017 04:46:02 +0000 Original-Received: (at 24402) by debbugs.gnu.org; 14 Jul 2017 04:45:26 +0000 Original-Received: from localhost ([127.0.0.1]:37604 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dVsTm-00076D-32 for submit@debbugs.gnu.org; Fri, 14 Jul 2017 00:45:26 -0400 Original-Received: from mail-it0-f41.google.com ([209.85.214.41]:33838) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dVsTk-00075x-GC for 24402@debbugs.gnu.org; Fri, 14 Jul 2017 00:45:24 -0400 Original-Received: by mail-it0-f41.google.com with SMTP id v202so6491364itb.1 for <24402@debbugs.gnu.org>; Thu, 13 Jul 2017 21:45:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=tbIIip2GRJxJZvwV+kXxwVnhbfJx2lB5DAXJ/VMpLMo=; b=PV4+Lz8EShgNJD5s21rsZwabzVFdklODiv9gfTr562b6L5e3vTX4Fi/JOHSw8bCFce fhUpNQqn8zFyzsWR7dVZlOsH5JQRPtM3vt+1iugApadJwoQAI/iI993Q0ueTq2iY8DdM 7zaCesnU45Ho5Z5x6cqgAHrg/OjUw2L2e1x7a06conel3Tjy7i3Bl/XRMEr6rxtURVsF ajxizHwD2XK8C+I8E1ESwSjlKg0+CGIWhG2M1FDMXmPkj6rHmPuWNVK9V/9K5OWNLxyP edW7pfv6JZSvBBSxMjwwAuOuMcfGMYwyz7xNVKUNTLoSJthuGayuoDEmxqtVeGSIK+ca f2uQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version; bh=tbIIip2GRJxJZvwV+kXxwVnhbfJx2lB5DAXJ/VMpLMo=; b=gz0ziBW4Se6WUg/cdxnIkPWKGO8Z40w9dNDwK9dxRt8BH034ZbnnF86vtguASoyDaU j/vH4hwMT1WlK7bYbg8hkOv2HgLBakhXx5iQh8gf/gRDDBIsSmf06/rvijqGhBlcLt8x 23+s2yKITn6VYZeMJFZt3eaMWTMJrJr1vctlCQJbcgzp1ANFC1GAv39muHy3fkRrbPJp 4h2BXyuYzsxa6l3wp8wQqX5mrisvdOHyvagsNHPTlZksX1v/g25SMOZa5I3SWSt/WxIv rENptEFVh+S7DyWr06ufSGDy0qVLB2n/LSXPpRE1lZcGdorzWFelfAocC88m7EBeiXC0 UZJg== X-Gm-Message-State: AIVw113elAAcrGA+0HOHbPDyWg3O0XMeAPDHgfniaFes936r/VFEQym6 zjhS8ApaYAz4L2WX X-Received: by 10.107.164.207 with SMTP id d76mr6436459ioj.158.1500007518746; Thu, 13 Jul 2017 21:45:18 -0700 (PDT) Original-Received: from lylat (S010664777d9cebe3.ss.shawcable.net. [70.64.85.59]) by smtp.gmail.com with ESMTPSA id n19sm4118459ioe.3.2017.07.13.21.45.17 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 13 Jul 2017 21:45:17 -0700 (PDT) In-Reply-To: <87d193ljdp.fsf@lylat> (Alex's message of "Thu, 13 Jul 2017 22:42:10 -0600") X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 208.118.235.43 X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.org gmane.emacs.bugs:134528 Archived-At: --=-=-= Content-Type: text/plain Alex writes: > I believe the below patch does that, though it has some issues. Here's the patch, sorry: --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=0001-Catch-argument-and-expansion-errors-in-ert.patch >From ba2703bce46e7caa6e86d7406a712bd92d06f84d Mon Sep 17 00:00:00 2001 From: Alexander Gramiak Date: Thu, 13 Jul 2017 14:54:35 -0600 Subject: [PATCH] Catch argument and expansion errors in ert This kludge catches errors caused by evaluating arguments in ert's should, should-not, and should-error macros; it also catches macro and inline function expansion errors inside of the above macros (Bug#24402). * lisp/emacs-lisp/ert.el: (ert--should-signal-hook): New function. (ert--expand-should-1): Catch expansion errors. * test/lisp/emacs-lisp/ert-tests.el (ert-test-should-error-argument) (ert-test-should-error-macroexpansion): Tests for argument and expansion errors. --- lisp/emacs-lisp/ert.el | 53 +++++++++++++++++++++++++++++++-------- test/lisp/emacs-lisp/ert-tests.el | 9 +++++++ 2 files changed, 52 insertions(+), 10 deletions(-) diff --git a/lisp/emacs-lisp/ert.el b/lisp/emacs-lisp/ert.el index eb2b2e3e11..353cbb8d57 100644 --- a/lisp/emacs-lisp/ert.el +++ b/lisp/emacs-lisp/ert.el @@ -266,6 +266,14 @@ DATA is displayed to the user and should state the reason for skipping." (when ert--should-execution-observer (funcall ert--should-execution-observer form-description))) +;; See Bug#24402 for why this exists +(defun ert--should-signal-hook (error-symbol data) + "Stupid hack to stop `condition-case' from catching ert signals. +It should only be stopped when ran from inside ert--run-test-internal." + (when (and (not (symbolp debugger)) ; only run on anonymous debugger + (memq error-symbol '(ert-test-failed ert-test-skipped))) + (funcall debugger 'error data))) + (defun ert--special-operator-p (thing) "Return non-nil if THING is a symbol naming a special operator." (and (symbolp thing) @@ -276,13 +284,27 @@ DATA is displayed to the user and should state the reason for skipping." (defun ert--expand-should-1 (whole form inner-expander) "Helper function for the `should' macro and its variants." (let ((form - (macroexpand form (append (bound-and-true-p - byte-compile-macro-environment) - (cond - ((boundp 'macroexpand-all-environment) - macroexpand-all-environment) - ((boundp 'cl-macro-environment) - cl-macro-environment)))))) + ;; catch all macro and inline function expansion errors + ;; FIXME: Code inside of tests should be evaluated like it is + ;; outside of tests, with the sole exception of error + ;; handling + (condition-case err + (funcall + ;; expand only when byte-compiling + (lambda (form) + (if (and (bound-and-true-p byte-compile-current-file) + (consp form)) + (byte-compile-inline-expand form) + form)) + (macroexpand-all form + (append (bound-and-true-p + byte-compile-macro-environment) + (cond + ((boundp 'macroexpand-all-environment) + macroexpand-all-environment) + ((boundp 'cl-macro-environment) + cl-macro-environment))))) + (error `(signal ',(car err) ',(cdr err)))))) (cond ((or (atom form) (ert--special-operator-p (car form))) (let ((value (cl-gensym "value-"))) @@ -298,13 +320,20 @@ DATA is displayed to the user and should state the reason for skipping." (cl-assert (or (symbolp fn-name) (and (consp fn-name) (eql (car fn-name) 'lambda) - (listp (cdr fn-name))))) + (listp (cdr fn-name))) + ;; for inline functions + (byte-code-function-p fn-name))) (let ((fn (cl-gensym "fn-")) (args (cl-gensym "args-")) (value (cl-gensym "value-")) (default-value (cl-gensym "ert-form-evaluation-aborted-"))) - `(let ((,fn (function ,fn-name)) - (,args (list ,@arg-forms))) + `(let* ((,fn (function ,fn-name)) + (,args (condition-case err + (let ((signal-hook-function #'ert--should-signal-hook)) + (list ,@arg-forms)) + (error (progn (setq ,fn #'signal) + (list (car err) + (cdr err))))))) (let ((,value ',default-value)) ,(funcall inner-expander `(setq ,value (apply ,fn ,args)) @@ -766,6 +795,10 @@ This mainly sets up debugger-related bindings." ;; too expensive, we can remove it. (with-temp-buffer (save-window-excursion + ;; FIXME: Use `signal-hook-function' instead of `debugger' to + ;; handle ert errors. Once that's done, remove + ;; `ert--should-signal-hook'. See Bug#24402 and Bug#11218 for + ;; details. (let ((debugger (lambda (&rest args) (ert--run-test-debugger test-execution-info args))) diff --git a/test/lisp/emacs-lisp/ert-tests.el b/test/lisp/emacs-lisp/ert-tests.el index 317838b250..b6a7eb68da 100644 --- a/test/lisp/emacs-lisp/ert-tests.el +++ b/test/lisp/emacs-lisp/ert-tests.el @@ -294,6 +294,15 @@ failed or if there was a problem." "the error signaled was a subtype of the expected type"))))) )) +(ert-deftest ert-test-should-error-argument () + "Errors due to evaluating arguments should not break tests." + (should-error (identity (/ 1 0)))) + +(ert-deftest ert-test-should-error-macroexpansion () + "Errors due to expanding macros should not break tests." + (cl-macrolet ((test () (error "Foo"))) + (should-error (test)))) + (ert-deftest ert-test-skip-unless () ;; Don't skip. (let ((test (make-ert-test :body (lambda () (skip-unless t))))) -- 2.13.2 --=-=-=--