Test failures

Test failures

[Nick Dokos]

I just updated to Org-mode version 8.3beta (release_8.3beta-40-g9cf3c4)
and got three test failures:

,----
| 3 unexpected results:
|    FAILED  test-org/open-at-point-in-comment
|    FAILED  test-org/open-at-point-in-property
|    FAILED  test-org/open-at-point/info
`----

Apparently these tests assume that the org info file can be found, but
the `make clea'n that is done at the beginning of make test wipes it
out. That's probably because I'm working out of the cloned git tree, but
if possible, I would like `make test' to work in that case.  My
workaround for now is to add `make info' in targets.mk:

--8<---------------cut here---------------start------------->8---
index d390fdb..6d8b61a 100644
--- a/mk/targets.mk
+++ b/mk/targets.mk
@@ -100,6 +100,7 @@ vanilla:
 
 check test::	compile
 check test test-dirty::
+	make info
 	-$(MKDIR) $(testdir)
 	TMPDIR=$(testdir) $(BTEST)
 ifeq ($(TEST_NO_AUTOCLEAN),) # define this variable to leave $(testdir) around for inspection
--8<---------------cut here---------------end--------------->8---

Thanks!
-- 
Nick

Re: Test failures

[Achim Gratz]

Nick Dokos <ndokos <at> gmail.com> writes:
> Apparently these tests assume that the org info file can be found, but
> the `make clea'n that is done at the beginning of make test wipes it
> out. That's probably because I'm working out of the cloned git tree, but
> if possible, I would like `make test' to work in that case.  My
> workaround for now is to add `make info' in targets.mk:

Actually it needs just about _any_ Org info file, it's not looking
specifically for the one from the Git tree, but depending on how your Emacs
installation looks like it may not find one.  I really don't think this is a
good test anyway, but pulling in the info target (as a dependency, not as an
actual make invocation) is too heavy-handed.  The test should actually
provide the info file (and set up info path) specifically for testing,
patches welcome.


Regards,
Achim.

Re: Test failures

[Bastien]

Hi Nick,

Nick Dokos <ndokos@gmail.com> writes:

> Apparently these tests assume that the org info file can be found, but
> the `make clea'n that is done at the beginning of make test wipes it
> out. That's probably because I'm working out of the cloned git tree, but
> if possible, I would like `make test' to work in that case.  My
> workaround for now is to add `make info' in targets.mk:

I pushed another workaround, which is to test on Emacs info manual instead.

-- 
 Bastien

test failures

[Stephen Berman]

I just built current master (at commit 348686b3f3) and then ran make
check (for the first time in a long time), and this is the result:

SUMMARY OF TEST RESULTS
-----------------------
Files examined: 298
Ran 4216 tests, 4148 results as expected, 20 unexpected, 48 skipped
3 files contained unexpected results:
  lisp/progmodes/elisp-mode-tests.log
  lisp/help-fns-tests.log
  lisp/emacs-lisp/cl-generic-tests.log

Here are the tests that unexpectedly failed:

Ran 45 tests, 28 results as expected, 14 unexpected, 3 skipped (2020-08-21 21:40:33+0200, 2.127913 sec)

14 unexpected results:
   FAILED  xref-elisp-test-find-defs-defgeneric-co-located-default
   FAILED  xref-elisp-test-find-defs-defgeneric-el
   FAILED  xref-elisp-test-find-defs-defgeneric-implicit-generic
   FAILED  xref-elisp-test-find-defs-defgeneric-no-default
   FAILED  xref-elisp-test-find-defs-defgeneric-no-methods
   FAILED  xref-elisp-test-find-defs-defgeneric-separate-default
   FAILED  xref-elisp-test-find-defs-define-overload-co-located-default
   FAILED  xref-elisp-test-find-defs-define-overload-no-default
   FAILED  xref-elisp-test-find-defs-define-overload-no-methods
   FAILED  xref-elisp-test-find-defs-define-overload-separate-default
   FAILED  xref-elisp-test-find-defs-defun-defvar-el
   FAILED  xref-elisp-test-find-defs-defun-el
   FAILED  xref-elisp-test-find-defs-defvar-el
   FAILED  xref-elisp-test-find-defs-feature-el

Ran 19 tests, 14 results as expected, 5 unexpected (2020-08-21 21:39:58+0200, 0.689917 sec)

5 unexpected results:
   FAILED  help-fns-test-alias-to-defun
   FAILED  help-fns-test-bug23887
   FAILED  help-fns-test-lisp-defsubst
   FAILED  help-fns-test-lisp-defun
   FAILED  help-fns-test-lisp-macro

Ran 17 tests, 16 results as expected, 1 unexpected (2020-08-21 21:39:05+0200, 0.073067 sec)

1 unexpected results:
   FAILED  cl-generic-tests--method-files--finds-methods


Are these known failures, or did something go wrong with my build, or
would perhaps make bootstrap eliminate the failures?

Steve Berman

Re: test failures

[Lars Ingebrigtsen]

Stephen Berman <stephen.berman@gmx.net> writes:

> Are these known failures, or did something go wrong with my build, or
> would perhaps make bootstrap eliminate the failures?

I'm not getting any errors with the current trunk on Debian (with make
check).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: test failures

[Stephen Berman]

On Fri, 21 Aug 2020 22:36:40 +0200 Lars Ingebrigtsen <larsi@gnus.org> wrote:

> Stephen Berman <stephen.berman@gmx.net> writes:
>
>> Are these known failures, or did something go wrong with my build, or
>> would perhaps make bootstrap eliminate the failures?
>
> I'm not getting any errors with the current trunk on Debian (with make
> check).

Hm, maybe I should try bootstrapping...

Steve Berman

Re: test failures

[Eli Zaretskii]

> From: Stephen Berman <stephen.berman@gmx.net>
> Date: Fri, 21 Aug 2020 22:23:05 +0200
> 
> 3 files contained unexpected results:
>   lisp/progmodes/elisp-mode-tests.log
>   lisp/help-fns-tests.log
>   lisp/emacs-lisp/cl-generic-tests.log

These all pass for me with the current master.

Re: test failures

[Stephen Berman]

On Fri, 21 Aug 2020 22:36:40 +0200 Lars Ingebrigtsen <larsi@gnus.org> wrote:

> Stephen Berman <stephen.berman@gmx.net> writes:
>
>> Are these known failures, or did something go wrong with my build, or
>> would perhaps make bootstrap eliminate the failures?
>
> I'm not getting any errors with the current trunk on Debian (with make
> check).

On Sat, 22 Aug 2020 09:31:29 +0300 Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Stephen Berman <stephen.berman@gmx.net>
>> Date: Fri, 21 Aug 2020 22:23:05 +0200
>> 
>> 3 files contained unexpected results:
>>   lisp/progmodes/elisp-mode-tests.log
>>   lisp/help-fns-tests.log
>>   lisp/emacs-lisp/cl-generic-tests.log
>
> These all pass for me with the current master.

It seems that the failures arise because my Emacs sources are in a
directory on a different partition from my home directory, and the
source directory is symlinked from §HOME, and my Emacs build directory
is under $HOME.  When I do batch runs of the test three files in
question starting from my home directory (i.e. dereferencing the symlink
to the sources), I get the the failures I reported; but when I start the
batch runs from the real directory containing the test sources, the one
test in cl-generic-tests.el that failed now passes, and of the fourteen
xref tests in elisp-mode-tests.el that failed, now only five fail; yet
all five of the failing tests in help-fns-tests.el still fail.

However, concerning the latter, it seems that the tests expect
help-fns-function-description-header to return the quoted basename of
the file, while in my environment it returns the absolute filename.
Adjusting the regexp used in the tests fixes this issue (see the first
patch below) and then all tests pass regardless of where I start the
batch run.

Concerning cl-generic-tests.el, using file-truename as in the second
patch below prevents the one failure, also regardless of where I start
the batch run.

As for the failing xref tests in elisp-mode-tests.el, simply wrapping
the uses of expand-file-name in file-truename did not prevent the
errors.  I don't know cl-generic well enough to debug further without
assistence.

Steve Berman


diff --git a/test/lisp/help-fns-tests.el b/test/lisp/help-fns-tests.el
index da2b49e6b8..7782a41b9f 100644
--- a/test/lisp/help-fns-tests.el
+++ b/test/lisp/help-fns-tests.el
@@ -56,28 +56,28 @@ help-fns-test-interactive-built-in
     (should (string-match regexp result))))
 
 (ert-deftest help-fns-test-lisp-macro ()
-  (let ((regexp "a Lisp macro in .subr\\.el")
+  (let ((regexp "a Lisp macro in .+subr\\.el")
         (result (help-fns-tests--describe-function 'when)))
     (should (string-match regexp result))))
 
 (ert-deftest help-fns-test-lisp-defun ()
-  (let ((regexp "a compiled Lisp function in .subr\\.el")
+  (let ((regexp "a compiled Lisp function in .+subr\\.el")
         (result (help-fns-tests--describe-function 'last)))
     (should (string-match regexp result))))
 
 (ert-deftest help-fns-test-lisp-defsubst ()
-  (let ((regexp "a compiled Lisp function in .subr\\.el")
+  (let ((regexp "a compiled Lisp function in .+subr\\.el")
         (result (help-fns-tests--describe-function 'posn-window)))
     (should (string-match regexp result))))
 
 (ert-deftest help-fns-test-alias-to-defun ()
-  (let ((regexp "an alias for .set-file-modes. in .subr\\.el")
+  (let ((regexp "an alias for .set-file-modes. in .+subr\\.el")
         (result (help-fns-tests--describe-function 'chmod)))
     (should (string-match regexp result))))
 
 (ert-deftest help-fns-test-bug23887 ()
   "Test for https://debbugs.gnu.org/23887 ."
-  (let ((regexp "an alias for .re-search-forward. in .subr\\.el")
+  (let ((regexp "an alias for .re-search-forward. in .+subr\\.el")
         (result (help-fns-tests--describe-function 'search-forward-regexp)))
     (should (string-match regexp result))))
 

diff --git a/test/lisp/emacs-lisp/cl-generic-tests.el b/test/lisp/emacs-lisp/cl-generic-tests.el
index 5aa58782f3..9582907e51 100644
--- a/test/lisp/emacs-lisp/cl-generic-tests.el
+++ b/test/lisp/emacs-lisp/cl-generic-tests.el
@@ -240,7 +240,7 @@ cl-generic-tests--method-files--finds-methods
   (let ((retval (cl--generic-method-files 'cl-generic-tests--generic)))
     (should (equal (length retval) 2))
     (mapc (lambda (x)
-            (should (equal (car x) cl-generic-tests--this-file))
+            (should (equal (file-truename (car x)) cl-generic-tests--this-file))
             (should (equal (cadr x) 'cl-generic-tests--generic)))
           retval)
     (should-not (equal (nth 0 retval) (nth 1 retval)))))

Test Failures

[Vin Shelton]

[-- Attachment #1: Type: text/plain, Size: 592 bytes --]

Using the most recent sources and building in the source tree:

(
today=$(date +'%Y-%m-%d')
git clone --quiet git://git.savannah.gnu.org/emacs.git emacs-$today
cd ./emacs-$today
./autogen.sh >& /dev/null
./configure >& /dev/null
make bootstrap >& mk.out
make -k check |& tee check.out
)

I'm seeing three test failures:

3 unexpected results:
   FAILED  python-shell-buffer-substring-10
   FAILED  python-shell-buffer-substring-11
   FAILED  python-shell-buffer-substring-12
Please let me know if you need more details.

Regards,
  Vin Shelton

-- 
*If Cooperstown is calling, it's no fluke*

[-- Attachment #2: Type: text/html, Size: 1874 bytes --]

Re: Test Failures

[Lars Ingebrigtsen]

Vin Shelton <acs@alumni.princeton.edu> writes:

> 3 unexpected results:
>    FAILED  python-shell-buffer-substring-10
>    FAILED  python-shell-buffer-substring-11
>    FAILED  python-shell-buffer-substring-12

Should be fixed now.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Test Failures

[Vin Shelton]

[-- Attachment #1: Type: text/plain, Size: 839 bytes --]

I can confirm that the python-shell failures are gone.  Thanks, Larsi.

FYI, on an unrelated note, I'm seeing five failures when BLDDIR != SRCDIR:

   FAILED  xref-elisp-test-find-defs-defgeneric-el
   FAILED  xref-elisp-test-find-defs-defun-defvar-el
   FAILED  xref-elisp-test-find-defs-defun-el
   FAILED  xref-elisp-test-find-defs-defvar-el
   FAILED  xref-elisp-test-find-defs-feature-el

HTH,
  Vin

On Wed, Oct 28, 2020 at 4:20 AM Lars Ingebrigtsen <larsi@gnus.org> wrote:

> Vin Shelton <acs@alumni.princeton.edu> writes:
>
> > 3 unexpected results:
> >    FAILED  python-shell-buffer-substring-10
> >    FAILED  python-shell-buffer-substring-11
> >    FAILED  python-shell-buffer-substring-12
>
> Should be fixed now.
>
> --
> (domestic pets only, the antidote for overdose, milk.)
>    bloggy blog: http://lars.ingebrigtsen.no
>

[-- Attachment #2: Type: text/html, Size: 2091 bytes --]

Re: Test Failures

[Dmitry Gutov]

On 28.10.2020 15:59, Vin Shelton wrote:
> FYI, on an unrelated note, I'm seeing five failures when BLDDIR != SRCDIR:
> 
>     FAILED  xref-elisp-test-find-defs-defgeneric-el
>     FAILED  xref-elisp-test-find-defs-defun-defvar-el
>     FAILED  xref-elisp-test-find-defs-defun-el
>     FAILED  xref-elisp-test-find-defs-defvar-el
>     FAILED  xref-elisp-test-find-defs-feature-el

This looks like bug#43004.