From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: =?UTF-8?Q?Jo=C3=A3o_?= =?UTF-8?Q?T=C3=A1vora?= Newsgroups: gmane.emacs.bugs Subject: bug#47320: Improve failure reporting in test/lisp/electrict-tests.el Date: Tue, 23 Mar 2021 15:59:03 +0000 Message-ID: References: <87v99i5k0f.fsf@gmx.de> <877dlx3nb8.fsf@gmx.de> Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="000000000000e6e49705be364316" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="26426"; mail-complaints-to="usenet@ciao.gmane.io" Cc: Alan Mackenzie , 47320@debbugs.gnu.org To: Michael Albinus Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Tue Mar 23 17:45:24 2021 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lOk9j-0006lB-Ku for geb-bug-gnu-emacs@m.gmane-mx.org; Tue, 23 Mar 2021 17:45:23 +0100 Original-Received: from localhost ([::1]:34338 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lOk9i-0003IA-I8 for geb-bug-gnu-emacs@m.gmane-mx.org; Tue, 23 Mar 2021 12:45:22 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:41454) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lOjRq-00051r-Rt for bug-gnu-emacs@gnu.org; Tue, 23 Mar 2021 12:00:02 -0400 Original-Received: from debbugs.gnu.org ([209.51.188.43]:49498) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lOjRq-0007uY-FF for bug-gnu-emacs@gnu.org; Tue, 23 Mar 2021 12:00:02 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lOjRq-0002dV-EU for bug-gnu-emacs@gnu.org; Tue, 23 Mar 2021 12:00:02 -0400 X-Loop: help-debbugs@gnu.org Resent-From: =?UTF-8?Q?Jo=C3=A3o_?= =?UTF-8?Q?T=C3=A1vora?= Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Tue, 23 Mar 2021 16:00:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 47320 X-GNU-PR-Package: emacs Original-Received: via spool by 47320-submit@debbugs.gnu.org id=B47320.161651516910061 (code B ref 47320); Tue, 23 Mar 2021 16:00:02 +0000 Original-Received: (at 47320) by debbugs.gnu.org; 23 Mar 2021 15:59:29 +0000 Original-Received: from localhost ([127.0.0.1]:32811 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lOjRI-0002cC-LT for submit@debbugs.gnu.org; Tue, 23 Mar 2021 11:59:29 -0400 Original-Received: from mail-io1-f42.google.com ([209.85.166.42]:39623) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lOjRG-0002bw-6x for 47320@debbugs.gnu.org; Tue, 23 Mar 2021 11:59:27 -0400 Original-Received: by mail-io1-f42.google.com with SMTP id k25so6088009iob.6 for <47320@debbugs.gnu.org>; Tue, 23 Mar 2021 08:59:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=wmTFCmfHordcflPHB1qIHdje7gJdo3i2lPdD1qpOVuc=; b=NSqHpZcngoD/T97ydL7Q+VSZ4RvffiIB0JBY5tmjSS0vOtX0kcaAg5p+orX2Pd+9Xv yWuMFUgbiYYoUamq4Vq19JCrJKN+h82NxS2J1ZZ0ADXFpjFncRliEc5DlJ0N6kcPyKu6 qVyWX5M3KKwzwI3XbWRY+DTy7ApRO8SAOckc2hzRdUyaO9PPrqT0Zq8r+OUKvxDSx+TQ ATmNLZZ+CrYKXCsoM7t+t+0Pp6a8fchaNy/q2UHD/eu4VemR7QJoRO68CvNj4UbYhSQF gbF7BMIAAchtW2FK10YJYrWodPw7KTsNTgf0Vfpe0JRSyP0KnPKFNCa/zvnNS0WCEElC rXcg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=wmTFCmfHordcflPHB1qIHdje7gJdo3i2lPdD1qpOVuc=; b=AXegbdPnMS1z4xma3KD+4FxxznPoerdOkiL/2tbARyvLDvUjrMApwdQLcY0eW5CELq LXNc2Ie5RCz8s8ltsu1+E/POMjF1K7Yfcfajqo87gZQqXoDwGAYXltsM3RUErs/xVYk3 4xlnB6CLJ2kPNjAvkDlkx+VD+DPgFfnhpoi/9ttDLwqPzMblXoVH6/MtzhtduDBvIEk3 QZ0scYVWinZp+RtWVMJUhcgNacWl1oGm7gdTDWAahtKBgBckiIGXgOLVGijc0b7V67eM p2Jx+MVDzLb3SzENLtnRcMCB5esqwz8yXBlpey7SafuS4yuTf5qrpHk3v3qn/+bc76+J sxzQ== X-Gm-Message-State: AOAM532XzZ1FKsgBN4CIcf707bhofYdo9h0BhdDO0WXlqReNb8OgHV0t nxQ92piKxWg5QyGevyM5sc3ldrgPuWeHDucWPnE= X-Google-Smtp-Source: ABdhPJwyQQdQjLc7hDvDazFjlvTT1lwvx2gKG4JNYvY97gY816rCWyf2T9D8lz11tgLCGyX8pk5NTrmL8p+PaBXnzeA= X-Received: by 2002:a02:661c:: with SMTP id k28mr5228217jac.78.1616515160268; Tue, 23 Mar 2021 08:59:20 -0700 (PDT) In-Reply-To: <877dlx3nb8.fsf@gmx.de> X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.io gmane.emacs.bugs:202911 Archived-At: --000000000000e6e49705be364316 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable As the author of electric-tests.el, I'm fine with either approach. Michael's would seem more integrated, at a glance, not sure how hard it is to switch to it. I'd have to see that patch. Jo=C3=A3o On Tue, Mar 23, 2021, 15:24 Michael Albinus wrote: > Alan Mackenzie writes: > > > Hello, Michael. > > Hi Alan, > > >> There is the explainer functionality in ert, see > > > >> (info "(ert)Defining Explanation Functions") > > > > That description is rather terse. In fact it is incomplete - it does > > not say when the explanation function gets called, nor does it say what > > is done with any resulting explanation. > > 100% d'accord. The ert manual isn't as helpful as it should ... > > > It looks like a explanation function needs to duplicate the test code o= f > > the test being explained - it seems the function has no access to the > > internal state of the test. > > No, it has full access to the test via the ert machinery. > > (ert-get-test SYMBOL) returns the test object which has been created via > (ert-deftest SYMBOL-NAME ...) > > (ert-running-test) returns the test object of the test just running. > > With that test object, you have different access functions, like > ert-test-name, ert-test-documentation, ert-test-most-recent-result; see > cl-defstruct ert-test. And there are also some other defstructs, which > give you convenience functions like ert-test-failed-p. > > > I only wish to print the extra information when there is a test failure= . > > Otherwise the information in electric-tests.log would be swamped by > > pointless voluminous doc strings. > > The ert explainer writes only something for failed functions. > > > Or am I missing something? > > I've spent hours and hours to understand these > ert functions. You might have a look on filenotify-tests.el or > tramp-tests.el, for example. > > >> I recommend to use it, for better readability of the test code. Exampl= es > >> are in our test files. > > > > Thanks! > > If needed, I could help you in setting upt these explainers. But I must > confess, I have no idea about electrict-tests.el. > > For the beginning, you might look on this: > > --8<---------------cut here---------------start------------->8--- > (defun test-function () > "The result." > t) > > (defun test-function-explainer () > "The explainer, a string." > (format "%s: %s" (ert-test-name (ert-running-test)) > (ert-test-documentation (ert-running-test)))) > > (ert-deftest first-test () > "Just the first test" > (should (test-function))) > > (ert-deftest second-test () > "Just the second test" > (should-not (test-function))) > > (put 'test-function 'ert-explainer > 'test-function-explainer) > --8<---------------cut here---------------end--------------->8--- > > If you run both tests, you get > > --8<---------------cut here---------------start------------->8--- > .F > > . first-test > Just the first test > passed > > F second-test > Just the second test > (ert-test-failed > ((should-not > (test-function)) > :form > (test-function) > :value t :explanation "second-test: Just the second test")) > --8<---------------cut here---------------end--------------->8--- > > Best regards, Michael. > --000000000000e6e49705be364316 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
As the author of electric-tests.el, I'm fine wit= h either approach. Michael's would seem more integrated, at a glance, n= ot sure how hard it is to switch to it. I'd have to see that patch. Jo= =C3=A3o

On Tue, Mar 23, 2021, 15:24 Michael Albinus <michael.albinus@gmx.de> wrote:
Alan Mackenzie <acm@muc.de> writes:

> Hello, Michael.

Hi Alan,

>> There is the explainer functionality in ert, see
>
>> (info "(ert)Defining Explanation Functions")
>
> That description is rather terse.=C2=A0 In fact it is incomplete - it = does
> not say when the explanation function gets called, nor does it say wha= t
> is done with any resulting explanation.

100% d'accord. The ert manual isn't as helpful as it should ...

> It looks like a explanation function needs to duplicate the test code = of
> the test being explained - it seems the function has no access to the<= br> > internal state of the test.

No, it has full access to the test via the ert machinery.

(ert-get-test SYMBOL) returns the test object which has been created via (ert-deftest SYMBOL-NAME ...)

(ert-running-test) returns the test object of the test just running.

With that test object, you have different access functions, like
ert-test-name, ert-test-documentation, ert-test-most-recent-result; see
cl-defstruct ert-test. And there are also some other defstructs, which
give you convenience functions like ert-test-failed-p.

> I only wish to print the extra information when there is a test failur= e.
> Otherwise the information in electric-tests.log would be swamped by > pointless voluminous doc strings.

The ert explainer writes only something for failed functions.

> Or am I missing something?

<shameless advertisement>I've spent hours and hours to understand= these
ert functions. You might have a look on filenotify-tests.el or
tramp-tests.el, for example.</>

>> I recommend to use it, for better readability of the test code. Ex= amples
>> are in our test files.
>
> Thanks!

If needed, I could help you in setting upt these explainers. But I must
confess, I have no idea about electrict-tests.el.

For the beginning, you might look on this:

--8<---------------cut here---------------start------------->8---
(defun test-function ()
=C2=A0 "The result."
=C2=A0 t)

(defun test-function-explainer ()
=C2=A0 "The explainer, a string."
=C2=A0 (format "%s: %s" (ert-test-name (ert-running-test)) (ert-t= est-documentation (ert-running-test))))

(ert-deftest first-test ()
=C2=A0 "Just the first test"
=C2=A0 (should (test-function)))

(ert-deftest second-test ()
=C2=A0 "Just the second test"
=C2=A0 (should-not (test-function)))

(put 'test-function 'ert-explainer
=C2=A0 =C2=A0 =C2=A0'test-function-explainer)
--8<---------------cut here---------------end--------------->8---

If you run both tests, you get

--8<---------------cut here---------------start------------->8---
.F

. first-test
=C2=A0 =C2=A0 Just the first test
=C2=A0 =C2=A0 passed

F second-test
=C2=A0 =C2=A0 Just the second test
=C2=A0 =C2=A0 (ert-test-failed
=C2=A0 =C2=A0 =C2=A0((should-not
=C2=A0 =C2=A0 =C2=A0 =C2=A0(test-function))
=C2=A0 =C2=A0 =C2=A0 :form
=C2=A0 =C2=A0 =C2=A0 (test-function)
=C2=A0 =C2=A0 =C2=A0 :value t :explanation "second-test: Just the seco= nd test"))
--8<---------------cut here---------------end--------------->8---

Best regards, Michael.
--000000000000e6e49705be364316--