bug#30700: 27.0.50; Broken tests on macOS

bug#30700: 27.0.50; Broken tests on macOS

[Philipp]

The following tests fail for me on macOS with a recent master build:

network-stream-tests:

Test echo-server-with-dns backtrace:
  make-network-process(:name "server" :server t :noquery t :family ipv
  make-server("p")
  (let* ((server (make-server (system-name))) (port (aref (process-con
  (closure (t) nil (let* ((server (make-server (system-name))) (port (
  ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test
  ert-run-test(#s(ert-test :name echo-server-with-dns :documentation n
  ert-run-or-rerun-test(#s(ert--stats :selector (not (or (tag :expensi
  ert-run-tests((not (or (tag :expensive-test) (tag :unstable))) #f(co
  ert-run-tests-batch((not (or (tag :expensive-test) (tag :unstable)))
  ert-run-tests-batch-and-exit((not (or (tag :expensive-test) (tag :un
  eval((ert-run-tests-batch-and-exit '(not (or (tag :expensive-test) (
  command-line-1(("-L" ":." "-l" "ert" "-l" "lisp/net/network-stream-t
  command-line()
  normal-top-level()
Test echo-server-with-dns condition:
    (error "p/0 nodename nor servname provided, or not known")

The macOS firewall prevents creating the server here.  Probably we
should just skip this test on macOS.


flymake-tests:

3 unexpected results:
   FAILED  included-c-header-files
   FAILED  warning-predicate-function-gcc
   FAILED  warning-predicate-rx-gcc

These tests fail because they seem to invoke GCC, and on macOS GCC is
actually Clang, so the output format is different from what's expected.
I'd suggest two fixes:

1. Instead of invoking GCC, save GCC's output once in a checked-in file.
   That makes the test more robust and reproducible because it no longer
   depends on the behavior of external programs.

2. Make Flymake work with the Clang output, i.e. remove the hard
   dependency on GCC.


In GNU Emacs 27.0.50 (build 57, x86_64-apple-darwin17.4.0, NS appkit-1561.20 Version 10.13.3 (Build 17D102))
 of 2018-03-02 built on p
Repository revision: e58f9a45d6a31be4ca379e10a866766e0993f49e
Windowing system distributor 'Apple', version 10.3.1561
System Description:  Mac OS X 10.13.3

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.

Configured using:
 'configure --with-modules --without-pop --with-mailutils
 --enable-gcc-warnings=yes --enable-checking
 --enable-check-lisp-object-type 'CFLAGS=-ggdb3 -O0''

Configured features:
NOTIFY ACL GNUTLS LIBXML2 ZLIB TOOLKIT_SCROLL_BARS NS MODULES THREADS
JSON

Important settings:
  value of $LANG: de_DE.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny seq byte-opt gv
bytecomp byte-compile cconv cl-loaddefs cl-lib dired dired-loaddefs
format-spec rfc822 mml easymenu mml-sec password-cache epa derived epg
epg-config gnus-util rmail rmail-loaddefs mm-decode mm-bodies mm-encode
mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail rfc2047
rfc2045 ietf-drums mm-util mail-prsvr mail-utils elec-pair time-date
tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel term/ns-win ns-win ucs-normalize mule-util term/common-win
tool-bar dnd fontset image regexp-opt fringe tabulated-list replace
newcomment text-mode elisp-mode lisp-mode prog-mode register page
menu-bar rfn-eshadow isearch timer select scroll-bar mouse jit-lock
font-lock syntax facemenu font-core term/tty-colors frame cl-generic
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray minibuffer
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote kqueue cocoa ns
multi-tty make-network-process emacs)

Memory information:
((conses 16 204785 6390)
 (symbols 48 20096 1)
 (miscs 40 56 156)
 (strings 32 28897 1948)
 (string-bytes 1 771870)
 (vectors 16 35249)
 (vector-slots 8 722256 13436)
 (floats 8 52 64)
 (intervals 56 211 0)
 (buffers 992 11))

bug#30700: 27.0.50; Broken tests on macOS

[Glenn Morris]

Philipp wrote:

> 1. Instead of invoking GCC, save GCC's output once in a checked-in file.
>    That makes the test more robust and reproducible because it no longer
>    depends on the behavior of external programs.

To me, part of the point of tests like these is to flag when changes
(or local variations in behaviour) in the external tool necessitate
changes in Emacs.

bug#30700: 27.0.50; Broken tests on macOS

[Alan Third]

On Sun, Mar 04, 2018 at 08:05:02PM +0100, Philipp wrote:
> 
> The following tests fail for me on macOS with a recent master build:
> 
> network-stream-tests:
> 
> Test echo-server-with-dns backtrace:
>   make-network-process(:name "server" :server t :noquery t :family ipv
>   make-server("p")
>   (let* ((server (make-server (system-name))) (port (aref (process-con
>   (closure (t) nil (let* ((server (make-server (system-name))) (port (
>   ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test
>   ert-run-test(#s(ert-test :name echo-server-with-dns :documentation n
>   ert-run-or-rerun-test(#s(ert--stats :selector (not (or (tag :expensi
>   ert-run-tests((not (or (tag :expensive-test) (tag :unstable))) #f(co
>   ert-run-tests-batch((not (or (tag :expensive-test) (tag :unstable)))
>   ert-run-tests-batch-and-exit((not (or (tag :expensive-test) (tag :un
>   eval((ert-run-tests-batch-and-exit '(not (or (tag :expensive-test) (
>   command-line-1(("-L" ":." "-l" "ert" "-l" "lisp/net/network-stream-t
>   command-line()
>   normal-top-level()
> Test echo-server-with-dns condition:
>     (error "p/0 nodename nor servname provided, or not known")
> 
> The macOS firewall prevents creating the server here.  Probably we
> should just skip this test on macOS.

Are you sure that’s what’s going on? The other server tests appear to
work. I would expect this one to fail on my set up because the DNS
entry for my computer only has an IPv6 address, but this test seems to
rely on IPv4.

> flymake-tests:
> 
> 3 unexpected results:
>    FAILED  included-c-header-files
>    FAILED  warning-predicate-function-gcc
>    FAILED  warning-predicate-rx-gcc
> 
> These tests fail because they seem to invoke GCC, and on macOS GCC is
> actually Clang, so the output format is different from what's expected.

They work in an interactive emacs session. I just manually ran through
the first one to see if there’s anything clearly different going on
but it seems right. I don’t understand why they fail from ‘make
check’.

-- 
Alan Third

bug#30700: 27.0.50; Broken tests on macOS

[Alan Third]

Alan Third <alan@idiocy.org> writes:

> On Sun, Mar 04, 2018 at 08:05:02PM +0100, Philipp wrote:
>> 
>> The following tests fail for me on macOS with a recent master build:
>> 
>> network-stream-tests:
>> 
>> Test echo-server-with-dns backtrace:
>>   make-network-process(:name "server" :server t :noquery t :family ipv
>>   make-server("p")
>>   (let* ((server (make-server (system-name))) (port (aref (process-con
>>   (closure (t) nil (let* ((server (make-server (system-name))) (port (
>>   ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test
>>   ert-run-test(#s(ert-test :name echo-server-with-dns :documentation n
>>   ert-run-or-rerun-test(#s(ert--stats :selector (not (or (tag :expensi
>>   ert-run-tests((not (or (tag :expensive-test) (tag :unstable))) #f(co
>>   ert-run-tests-batch((not (or (tag :expensive-test) (tag :unstable)))
>>   ert-run-tests-batch-and-exit((not (or (tag :expensive-test) (tag :un
>>   eval((ert-run-tests-batch-and-exit '(not (or (tag :expensive-test) (
>>   command-line-1(("-L" ":." "-l" "ert" "-l" "lisp/net/network-stream-t
>>   command-line()
>>   normal-top-level()
>> Test echo-server-with-dns condition:
>>     (error "p/0 nodename nor servname provided, or not known")
>> 
>> The macOS firewall prevents creating the server here.  Probably we
>> should just skip this test on macOS.
>
> Are you sure that’s what’s going on? The other server tests appear to
> work. I would expect this one to fail on my set up because the DNS
> entry for my computer only has an IPv6 address, but this test seems to
> rely on IPv4.

Looking further into this, if I change ":family" in make-server, either
by removing it, or changing it to 'ipv6, I get the following:

  (file-error "make client process failed" "Connection refused" :name
  "foo" :buffer #<buffer *foo*<19>> :host "breton.holly.idiocy.org"
  :service ...)

which implies the server is being created, but then the client can't
connect. Looking at netstat I can't see the port.

Turning off the macOS firewall makes no difference to any tests.
-- 
Alan Third

bug#30700: 27.0.50; Broken tests on macOS

[Philipp Stephani]

Am So., 6. Jan. 2019 um 23:40 Uhr schrieb Alan Third <alan@idiocy.org>:
>
> Alan Third <alan@idiocy.org> writes:
>
> > On Sun, Mar 04, 2018 at 08:05:02PM +0100, Philipp wrote:
> >>
> >> The following tests fail for me on macOS with a recent master build:
> >>
> >> network-stream-tests:
> >>
> >> Test echo-server-with-dns backtrace:
> >>   make-network-process(:name "server" :server t :noquery t :family ipv
> >>   make-server("p")
> >>   (let* ((server (make-server (system-name))) (port (aref (process-con
> >>   (closure (t) nil (let* ((server (make-server (system-name))) (port (
> >>   ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test
> >>   ert-run-test(#s(ert-test :name echo-server-with-dns :documentation n
> >>   ert-run-or-rerun-test(#s(ert--stats :selector (not (or (tag :expensi
> >>   ert-run-tests((not (or (tag :expensive-test) (tag :unstable))) #f(co
> >>   ert-run-tests-batch((not (or (tag :expensive-test) (tag :unstable)))
> >>   ert-run-tests-batch-and-exit((not (or (tag :expensive-test) (tag :un
> >>   eval((ert-run-tests-batch-and-exit '(not (or (tag :expensive-test) (
> >>   command-line-1(("-L" ":." "-l" "ert" "-l" "lisp/net/network-stream-t
> >>   command-line()
> >>   normal-top-level()
> >> Test echo-server-with-dns condition:
> >>     (error "p/0 nodename nor servname provided, or not known")
> >>
> >> The macOS firewall prevents creating the server here.  Probably we
> >> should just skip this test on macOS.
> >
> > Are you sure that’s what’s going on? The other server tests appear to
> > work. I would expect this one to fail on my set up because the DNS
> > entry for my computer only has an IPv6 address, but this test seems to
> > rely on IPv4.
>
> Looking further into this, if I change ":family" in make-server, either
> by removing it, or changing it to 'ipv6, I get the following:
>
>   (file-error "make client process failed" "Connection refused" :name
>   "foo" :buffer #<buffer *foo*<19>> :host "breton.holly.idiocy.org"
>   :service ...)
>
> which implies the server is being created, but then the client can't
> connect. Looking at netstat I can't see the port.
>
> Turning off the macOS firewall makes no difference to any tests.

The reason why I suspected the firewall is that I when running the
tests I get a firewall warning whether I want to allow gnutls-serv to
accept incoming requests. Does that warning not appear on your system?

bug#30700: 27.0.50; Broken tests on macOS

[Philipp Stephani]

Am So., 4. März 2018 um 20:06 Uhr schrieb Philipp <p.stephani2@gmail.com>:
>
>
> The following tests fail for me on macOS with a recent master build:

An update: I've run the testsuite again. The errors I got this time are:

Test tramp-test19-directory-files-and-attributes backtrace:
  signal(ert-test-failed (((should (tramp--test-file-attributes-equal-
  ert-fail(((should (tramp--test-file-attributes-equal-p (file-attribu
  #f(compiled-function () #<bytecode 0x1fe0c361f401>)()
  ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test
  ert-run-test(#s(ert-test :name tramp-test19-directory-files-and-attr
  ert-run-or-rerun-test(#s(ert--stats :selector (not ...) :tests [...
  ert-run-tests((not (or (tag :expensive-test) (tag :unstable))) #f(co
  ert-run-tests-batch((not (or (tag :expensive-test) (tag :unstable)))
  ert-run-tests-batch-and-exit((not (or (tag :expensive-test) (tag :un
  eval((ert-run-tests-batch-and-exit '(not (or (tag :expensive-test) (
  command-line-1(("-L" ":." "-l" "ert" "-l" "lisp/net/tramp-tests" "--
  command-line()
  normal-top-level()
Test tramp-test19-directory-files-and-attributes condition:
    (ert-test-failed
     ((should
       (tramp--test-file-attributes-equal-p
(file-attributes ...)
(cdr elt)))
      :form
      (tramp--test-file-attributes-equal-p
       (t 5 501 20
  (0 0 0 1000)
  (23735 29792)
  (23735 29792)
  160 "drwxr-xr-x" nil ...)
       (t 5 501 20
  (0 0 0 1000)
  (23735 29792)
  (23735 29792)
  160 "drwxr-xr-x" nil ...))
      :value nil))
   FAILED  25/46  tramp-test19-directory-files-and-attributes (1.931638 sec)


When using the explainer for equal, I get

(list-elt 10
(different-types 8887934387
(4547 . 7603)))

i.e. the inode numbers are different. I also don't understand why it
used the dotted format for the RHS; with big integers this should
never be necessary.


Test tramp-test22-file-times backtrace:
  signal(ert-test-failed (((should (equal (nth 5 (file-attributes tmp-
  ert-fail(((should (equal (nth 5 (file-attributes tmp-name1)) (second
  #f(compiled-function () #<bytecode 0x1fe0c361faad>)()
  ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test
  ert-run-test(#s(ert-test :name tramp-test22-file-times :documentatio
  ert-run-or-rerun-test(#s(ert--stats :selector (not ...) :tests [...
  ert-run-tests((not (or (tag :expensive-test) (tag :unstable))) #f(co
  ert-run-tests-batch((not (or (tag :expensive-test) (tag :unstable)))
  ert-run-tests-batch-and-exit((not (or (tag :expensive-test) (tag :un
  eval((ert-run-tests-batch-and-exit '(not (or (tag :expensive-test) (
  command-line-1(("-L" ":." "-l" "ert" "-l" "lisp/net/tramp-tests" "--
  command-line()
  normal-top-level()
Test tramp-test22-file-times condition:
    (ert-test-failed
     ((should
       (equal
(nth 5 ...)
(seconds-to-time 1)))
      :form
      (equal
       (0 1)
       (0 1 0 0))
      :value nil :explanation
      (proper-lists-of-different-length 2 4
(0 1)
(0 1 0 0)
first-mismatch-at 2)))
   FAILED  28/46  tramp-test22-file-times (2.090504 sec)

These are two different representations of the same time. Probably
something should normalize the return values to the canonical format.


Test echo-server-with-dns backtrace:
  make-network-process(:name "server" :server t :noquery t :family ipv
  make-server("p")
  #f(compiled-function () #<bytecode 0x1fe754a46039>)()
  ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test
  ert-run-test(#s(ert-test :name echo-server-with-dns :documentation n
  ert-run-or-rerun-test(#s(ert--stats :selector (not ...) :tests [...
  ert-run-tests((not (or (tag :expensive-test) (tag :unstable))) #f(co
  ert-run-tests-batch((not (or (tag :expensive-test) (tag :unstable)))
  ert-run-tests-batch-and-exit((not (or (tag :expensive-test) (tag :un
  eval((ert-run-tests-batch-and-exit '(not (or (tag :expensive-test) (
  command-line-1(("-L" ":." "-l" "ert" "-l" "lisp/net/network-stream-t
  command-line()
  normal-top-level()
Test echo-server-with-dns condition:
    (error "p/0 nodename nor servname provided, or not known")
   FAILED   5/24  echo-server-with-dns (0.003934 sec)

Same as before.


Test shadow-test08-shadow-todo backtrace:
  signal(ert-test-failed (((should (member (cons file (shadow-contract
  ert-fail(((should (member (cons file (shadow-contract-file-name (con
  #f(compiled-function () #<bytecode 0x1fef4521fecd>)()
  ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test
  ert-run-test(#s(ert-test :name shadow-test08-shadow-todo :documentat
  ert-run-or-rerun-test(#s(ert--stats :selector (not (or ... ...)) :te
  ert-run-tests((not (or (tag :expensive-test) (tag :unstable))) #f(co
  ert-run-tests-batch((not (or (tag :expensive-test) (tag :unstable)))
  ert-run-tests-batch-and-exit((not (or (tag :expensive-test) (tag :un
  eval((ert-run-tests-batch-and-exit '(not (or (tag :expensive-test) (
  command-line-1(("-L" ":." "-l" "ert" "-l" "lisp/shadowfile-tests" "-
  command-line()
  normal-top-level()
Test shadow-test08-shadow-todo condition:
    (ert-test-failed
     ((should
       (member
(cons file ...)
shadow-files-to-copy))
      :form
      (member
       ("/var/folders/ks/sxl3n3w51td0ry21j3tjs1wh0000gn/T/shadowfile-testsoznrLv"
. "/cluster2:/var/folders/ks/sxl3n3w51td0ry21j3tjs1wh0000gn/T/shadowfile-testsoznrLv")
       nil)
      :value nil))
   FAILED   9/10  shadow-test08-shadow-todo (0.170742 sec)


Test shadow-test09-shadow-copy-files backtrace:
  signal(ert-test-failed (((should (looking-at (regexp-quote "(setq sh
  ert-fail(((should (looking-at (regexp-quote "(setq shadow-files-to-c
  #f(compiled-function () #<bytecode 0x1fef45220eb5>)()
  ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test
  ert-run-test(#s(ert-test :name shadow-test09-shadow-copy-files :docu
  ert-run-or-rerun-test(#s(ert--stats :selector (not (or ... ...)) :te
  ert-run-tests((not (or (tag :expensive-test) (tag :unstable))) #f(co
  ert-run-tests-batch((not (or (tag :expensive-test) (tag :unstable)))
  ert-run-tests-batch-and-exit((not (or (tag :expensive-test) (tag :un
  eval((ert-run-tests-batch-and-exit '(not (or (tag :expensive-test) (
  command-line-1(("-L" ":." "-l" "ert" "-l" "lisp/shadowfile-tests" "-
  command-line()
  normal-top-level()
Test shadow-test09-shadow-copy-files condition:
    (ert-test-failed
     ((should
       (looking-at
(regexp-quote "(setq shadow-files-to-copy nil)")))
      :form
      (looking-at "(setq shadow-files-to-copy nil)")
      :value nil))
   FAILED  10/10  shadow-test09-shadow-copy-files (2.697340 sec)

I don't understand these yet.


That is, some tests are fixed, but there are also a few new failures.

bug#30700: 27.0.50; Broken tests on macOS

[Stefan Kangas]

Philipp Stephani <p.stephani2@gmail.com> writes:

>> The following tests fail for me on macOS with a recent master build:
>
> An update: I've run the testsuite again. The errors I got this time are:

>    FAILED  25/46  tramp-test19-directory-files-and-attributes (1.931638 sec)
>    FAILED  28/46  tramp-test22-file-times (2.090504 sec)

> These are two different representations of the same time. Probably
> something should normalize the return values to the canonical format.

These do not fail for me after the fixes applied for Bug#36940.

>    FAILED   5/24  echo-server-with-dns (0.003934 sec)
>
> Same as before.

I'm not seeing this failure.

>    FAILED   9/10  shadow-test08-shadow-todo (0.170742 sec)
>    FAILED  10/10  shadow-test09-shadow-copy-files (2.697340 sec)
>
> I don't understand these yet.

These no longer fails for me after the fixes applied for Bug#37202.

---

As a general suggestion, perhaps it's easier if we can track the
failures in one bug per package and platform, instead of having them all
per platform.

That way, I believe that they will be more visible to the relevant
maintainers in the bug tracker, and it will be easier to find out what
is still failing without having to sift through a mass of information
regarding separate issues.  (Compare the response Bug#37202 got with the
one you received here for one data point.)

If you agree with the above, could I suggest that we close this bug
report, since most issues are fixed, and that if you are still seeing
the echo-server-with-dns test failure, that you open a new one
specifically for that?

Best regards,
Stefan Kangas

bug#30700: 27.0.50; Broken tests on macOS

[Stefan Kangas]

Stefan Kangas <stefan@marxist.se> writes:

> If you agree with the above, could I suggest that we close this bug
> report, since most issues are fixed, and that if you are still seeing
> the echo-server-with-dns test failure, that you open a new one
> specifically for that?

No objections within 4 weeks, so I'm now closing this bug report.

If you are still seeing any failures for which there are not already
open bugs, please open individual bug reports for each issue.  Thanks
in advance.

Best regards,
Stefan Kangas