From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp10.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id sMBgARDL6mGoWwEAgWs5BA (envelope-from ) for ; Fri, 21 Jan 2022 16:02:40 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp10.migadu.com with LMTPS id GMGqNQ/L6mH5WgEAG6o9tA (envelope-from ) for ; Fri, 21 Jan 2022 16:02:39 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 0908936281 for ; Fri, 21 Jan 2022 16:02:39 +0100 (CET) Received: from localhost ([::1]:48122 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nAvQz-0003QZ-N6 for larch@yhetil.org; Fri, 21 Jan 2022 10:02:37 -0500 Received: from eggs.gnu.org ([209.51.188.92]:36760) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nAvPo-0003Oc-IV for emacs-orgmode@gnu.org; Fri, 21 Jan 2022 10:01:24 -0500 Received: from ciao.gmane.io ([116.202.254.214]:50930) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nAvPj-0004Aq-78 for emacs-orgmode@gnu.org; Fri, 21 Jan 2022 10:01:24 -0500 Received: from list by ciao.gmane.io with local (Exim 4.92) (envelope-from ) id 1nAvPg-000A1g-QY for emacs-orgmode@gnu.org; Fri, 21 Jan 2022 16:01:16 +0100 X-Injected-Via-Gmane: http://gmane.org/ To: emacs-orgmode@gnu.org From: Max Nikulin Subject: Re: [PATCH] make test: Make failure results more verbose Date: Fri, 21 Jan 2022 22:01:05 +0700 Message-ID: References: <87ee5q5lic.fsf@localhost> <87lezrr3i7.fsf@localhost> <87h79xz015.fsf@localhost> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="------------49539284F4E3A6C81E64DFC3" User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.14.0 In-Reply-To: <87h79xz015.fsf@localhost> Content-Language: en-US Received-SPF: pass client-ip=116.202.254.214; envelope-from=geo-emacs-orgmode@m.gmane-mx.org; helo=ciao.gmane.io X-Spam_score_int: 28 X-Spam_score: 2.8 X-Spam_bar: ++ X-Spam_report: (2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_ADSP_CUSTOM_MED=0.001, FORGED_GMAIL_RCVD=1, FORGED_MUA_MOZILLA=2.309, FREEMAIL_FORGED_FROMDOMAIN=0.249, FREEMAIL_FROM=0.001, HEADER_FROM_DIFFERENT_DOMAINS=0.249, NICE_REPLY_A=-0.001, NML_ADSP_CUSTOM_MED=0.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-orgmode@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+larch=yhetil.org@gnu.org Sender: "Emacs-orgmode" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1642777359; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=NwflQYC6EgZVBG1v2te1yC4RhYwAKPuS1H3BX2t+zfU=; b=WQhj9Yu6nYZUfOkHXN+bYRqYDcSTz2mCqTts274JjorOoyav19Egv/A6uW71cfEmQ9j8Ro GThGa0YysXQ/AeRaVc4Ia8UgyY3Rvi6wl92RGwZ8ZWfpk27WZp18NcKU09wjqwe7TrMP44 kUl/zsacW4dL3FT3FowMYOhsA4CPBDyb7gy/EaxU7uARKYWTgGBV83qM/FBN34dOPPE1Br ocbfIRqD4SmgmJ6rdJ266qD+6Yrz0vK07XB6vZpCKutaOcZobf60n8NoshjFjLZ7Ni8IFL yF+zMTvwdFZQggS1Vemrj1Ty/FlqA84kgLAEB1wxiEJcr3OoaZJ1JQZg5ID/GQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1642777359; a=rsa-sha256; cv=none; b=n63gDbJJBofrkNwqhGkrdq+5cislTTDQywJ7mUqPUoeyL/SZVYZKIeIEozqeGLQi5N1nqW zuvehFbXFbjebrZ45KsS9YVD0icvAfCG4gA7KVXgKiZE8I4bUPY3weAxZGWYxNNe5KAEYS BzZZzKZao5Pb5TvvBz/KqnY/JYpvqc9n4ehM2CYJLisBPb+7no0+ko9m13bncsSISIgJ4l U1kqxDu+c4gwhZLXIgg4pHYICxnR6p4AaBNHIuoh3Lepkf+n8bjUiLkKtwcmuBi2Ohpf0x /B08sEIGDS5ZMq6GXYGb06WPCAoFkFod6+2C/EGpPAb0rIKeK05Gk+2rPp8L0w== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -2.82 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 0908936281 X-Spam-Score: -2.82 X-Migadu-Scanner: scn0.migadu.com X-TUID: cBlWLbI2Ul13 This is a multi-part message in MIME format. --------------49539284F4E3A6C81E64DFC3 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit On 21/01/2022 20:33, Ihor Radchenko wrote: > Max Nikulin writes: > >> I am attaching a tentative patch for Org that should make >> EMACS_TEST_VERBOSE setting more transparent. > > Thanks! Your patch looks cleaner. In the meanwhile Lars fixed ERT in Emacs-29, so empty string is considered as false now, see https://debbugs.gnu.org/53313 Values like "no" should not be considered as false by Org makefiles, so I am attaching an updated version. > In addition, I am attaching a tentative hack to make ERT pretty-print > the failure reason as you wished. WDYT? Sorry, I was not clear enough. Summary should remain single-line. There are pretty-printed failure reason in the main part of the log. I was complained concerning the following brick > signal(ert-test-failed (((should (equal "aB " (org-test-with-parsed- > ert-fail(((should (equal "aB " (org-test-with-parsed-data "* Headlin > (if (unwind-protect (setq value-5061 (apply fn-5059 args-5060)) (set > (let (form-description-5063) (if (unwind-protect (setq value-5061 (a > (let ((value-5061 'ert-form-evaluation-aborted-5062)) (let (form-des > (let* ((fn-5059 #'equal) (args-5060 (condition-case err (let ((signa > (closure (t) nil (let* ((fn-5059 #'equal) (args-5060 (condition-case > ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test > ert-run-test(#s(ert-test :name test-org-export/get-previous-element > ert-run-or-rerun-test(#s(ert--stats :selector "test-org-export/get-p > ert-run-tests("test-org-export/get-previous-element" #f(compiled-fun > ert-run-tests-batch("test-org-export/get-previous-element") > ert-run-tests-batch-and-exit("test-org-export/get-previous-element") > (let ((org-id-track-globally t) (org-test-selector (if org-test-sele > org-test-run-batch-tests("test-org-export/get-previous-element") > eval((org-test-run-batch-tests org-test-select-re) t) > command-line-1(("-L" "/home/ubuntu/ert" "--eval" "(setq vc-handled-b > command-line() > normal-top-level() However it will be too long when pretty printed. --------------49539284F4E3A6C81E64DFC3 Content-Type: text/x-patch; charset=UTF-8; name="0001-make-test-Make-failure-summary-more-verbose.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0001-make-test-Make-failure-summary-more-verbose.patch" >From a53095fa1c6679376708dcc1a05a446c3967d914 Mon Sep 17 00:00:00 2001 From: Max Nikulin Date: Sat, 15 Jan 2022 22:54:30 +0700 Subject: [PATCH] make test: Make failure summary more verbose * mk/default.mk: By default enable verbose failure summary for Emacs-28 or newer. Set or unset EMACS_TEST_VERBOSE environment to control reporting of failure reasons in summary since in Emacs-28 ERT switches to verbose mode even by an empty string (fixed in Emacs-29). --- mk/default.mk | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/mk/default.mk b/mk/default.mk index c8a15bdd2..804089280 100644 --- a/mk/default.mk +++ b/mk/default.mk @@ -32,6 +32,20 @@ TMPDIR ?= /tmp testdir = $(TMPDIR)/tmp-orgtest # Configuration for testing +# Verbose ERT summary by default for Emacs-28 and above. +# To override: +# - Add to local.mk +# EMACS_TEST_VERBOSE = +# - Export EMACS_TEST_VERBOSE environment variable with empty value +# - Run tests as +# EMACS_TEST_VERBOSE= make test [OTHER_ARGUMENTS...] +# or as +# make test EMACS_TEST_VERBOSE= [OTHER_ARGUMENTS...] +EMACS_TEST_VERBOSE ?= yes +ifeq (,$(EMACS_TEST_VERBOSE)) +# Emacs-28 considers empty value as true, fixed in Emacs-29 +unexport EMACS_TEST_VERBOSE +endif # add options before standard load-path BTEST_PRE = # add options after standard load path -- 2.25.1 --------------49539284F4E3A6C81E64DFC3--