bug#62450: 29.0.60; Skip failing tests on Cygwin with native compilation

bug#62450: 29.0.60; Skip failing tests on Cygwin with native compilation

[Ken Brown]

[-- Attachment #1: Type: text/plain, Size: 515 bytes --]

When 'make check' is run on Cygwin and Emacs has been built with native
compilation, all the tests in comp-tests.el and benchmark-tests.el fail
with fork errors, for the reasons explained in etc/PROBLEMS.  I don't
see an easy way to fix this by rebasing, as is done elsewhere in the
code, and I don't think it's worth investing any effort into this just
for the sake of making some tests pass.  I propose that we just skip
these tests.

Patch attached (with a placeholder for the bug number).  OK for emacs-29?

Ken

[-- Attachment #2: 0001-Skip-failing-tests-on-Cygwin-with-native-compilation.patch --]
[-- Type: text/plain, Size: 1732 bytes --]

From 4b6913bb8c1043e732aa3b35d1d892f0b5d795c7 Mon Sep 17 00:00:00 2001
From: Ken Brown <kbrown@cornell.edu>
Date: Sat, 25 Mar 2023 16:37:51 -0400
Subject: [PATCH] Skip failing tests on Cygwin with native compilation
 (bug#624XX)

* test/Makefile.in (TEST_NATIVE_COMP): New variable, used to
determine whether to run tests tagged with :nativecomp.  Set it to
"no" on Cygwin and to $(HAVE_NATIVE_COMP) otherwise.

* test/lisp/emacs-lisp/benchmark-tests.el (benchmark-tests): Skip
on Cygwin with native-compilation.
---
 test/Makefile.in                        | 7 ++++++-
 test/lisp/emacs-lisp/benchmark-tests.el | 2 ++
 2 files changed, 8 insertions(+), 1 deletion(-)

diff --git a/test/Makefile.in b/test/Makefile.in
index fd21695f5bc9..0e4f3f31122a 100644
--- a/test/Makefile.in
+++ b/test/Makefile.in
@@ -124,7 +124,12 @@ .PHONY:
 
 all: check
 
-ifeq ($(HAVE_NATIVE_COMP),yes)
+SYSTEM_TYPE = @SYSTEM_TYPE@
+TEST_NATIVE_COMP = $(HAVE_NATIVE_COMP)
+ifeq ($(SYSTEM_TYPE),cygwin)
+TEST_NATIVE_COMP = no
+endif
+ifeq ($(TEST_NATIVE_COMP),yes)
 SELECTOR_DEFAULT = (not (or (tag :expensive-test) (tag :unstable)))
 SELECTOR_EXPENSIVE = (not (tag :unstable))
 SELECTOR_ALL = t
diff --git a/test/lisp/emacs-lisp/benchmark-tests.el b/test/lisp/emacs-lisp/benchmark-tests.el
index 31357f24a0de..9c57a2af4ac8 100644
--- a/test/lisp/emacs-lisp/benchmark-tests.el
+++ b/test/lisp/emacs-lisp/benchmark-tests.el
@@ -23,6 +23,8 @@
 (require 'ert)
 
 (ert-deftest benchmark-tests ()
+  (skip-unless (not (and (eq system-type 'cygwin)
+                         (featurep 'native-compile))))
   (let (str t-long t-short m)
     (should (consp (benchmark-run nil (setq m (1+ 0)))))
     (should (consp (benchmark-run 1 (setq m (1+ 0)))))
-- 
2.39.0

bug#62450: 29.0.60; Skip failing tests on Cygwin with native compilation

[Eli Zaretskii]

> Date: Sat, 25 Mar 2023 16:55:49 -0400
> From: Ken Brown <kbrown@cornell.edu>
> 
> When 'make check' is run on Cygwin and Emacs has been built with native
> compilation, all the tests in comp-tests.el and benchmark-tests.el fail
> with fork errors, for the reasons explained in etc/PROBLEMS.  I don't
> see an easy way to fix this by rebasing, as is done elsewhere in the
> code, and I don't think it's worth investing any effort into this just
> for the sake of making some tests pass.  I propose that we just skip
> these tests.
> 
> Patch attached (with a placeholder for the bug number).  OK for emacs-29?

OK, but please add in both places comments explaining the reason, and
in particular include there a pointer to etc/PROBLEMS.

(I do wonder whether this is a good idea in the long run.  Native
compilation is here to stay, so avoiding its testing for Cygwin might
not be the best solution.  The PROBLEMS entry seems to imply that
relatively simple measures, which the test suite could take, could
avoid the fork failures, so my recommendation would be to try to find
a better fix in the future, albeit not in Emacs 29.)

bug#62450: 29.0.60; Skip failing tests on Cygwin with native compilation

[Ken Brown]

On 3/26/2023 12:44 AM, Eli Zaretskii wrote:
> OK, but please add in both places comments explaining the reason, and
> in particular include there a pointer to etc/PROBLEMS.

Done.

> (I do wonder whether this is a good idea in the long run.  Native
> compilation is here to stay, so avoiding its testing for Cygwin might
> not be the best solution.  The PROBLEMS entry seems to imply that
> relatively simple measures, which the test suite could take, could
> avoid the fork failures, so my recommendation would be to try to find
> a better fix in the future, albeit not in Emacs 29.)

I agree, and I hope I can do it.  By the way, I've been experimenting 
with building the .eln files with ASLR enabled, via the following:

--- a/lisp/emacs-lisp/comp.el
+++ b/lisp/emacs-lisp/comp.el
@@ -186,8 +186,9 @@ native-comp-compiler-options
    :type '(repeat string)
    :version "28.1")

-(defcustom native-comp-driver-options (when (eq system-type 'darwin)
-                                        '("-Wl,-w"))
+(defcustom native-comp-driver-options
+  (cond ((eq system-type 'darwin) '("-Wl,-w"))
+        ((eq system-type 'cygwin) '("-Wl,-dynamicbase")))
    "Options passed verbatim to the native compiler's back-end driver.
  Note that not all options are meaningful; typically only the options
  affecting the assembler and linker are likely to be useful.

I've been building and running Emacs like this for a couple months and 
not doing any rebasing, and I haven't seen a single fork failure outside 
of the test suite.  So this might be the way to handle the problem in 
the long run.  But I still hope to figure out what's different about 
native compilation in the test suite.

Ken

bug#62450: 29.0.60; Skip failing tests on Cygwin with native compilation

[Eli Zaretskii]

> Date: Sun, 26 Mar 2023 10:15:17 -0400
> Cc: 62450-done@debbugs.gnu.org
> From: Ken Brown <kbrown@cornell.edu>
> 
> On 3/26/2023 12:44 AM, Eli Zaretskii wrote:
> 
> > (I do wonder whether this is a good idea in the long run.  Native
> > compilation is here to stay, so avoiding its testing for Cygwin might
> > not be the best solution.  The PROBLEMS entry seems to imply that
> > relatively simple measures, which the test suite could take, could
> > avoid the fork failures, so my recommendation would be to try to find
> > a better fix in the future, albeit not in Emacs 29.)
> 
> I agree, and I hope I can do it.

Great, thanks.

> By the way, I've been experimenting with building the .eln files
> with ASLR enabled, via the following:
> 
> --- a/lisp/emacs-lisp/comp.el
> +++ b/lisp/emacs-lisp/comp.el
> @@ -186,8 +186,9 @@ native-comp-compiler-options
>     :type '(repeat string)
>     :version "28.1")
> 
> -(defcustom native-comp-driver-options (when (eq system-type 'darwin)
> -                                        '("-Wl,-w"))
> +(defcustom native-comp-driver-options
> +  (cond ((eq system-type 'darwin) '("-Wl,-w"))
> +        ((eq system-type 'cygwin) '("-Wl,-dynamicbase")))
>     "Options passed verbatim to the native compiler's back-end driver.
>   Note that not all options are meaningful; typically only the options
>   affecting the assembler and linker are likely to be useful.
> 
> I've been building and running Emacs like this for a couple months and 
> not doing any rebasing, and I haven't seen a single fork failure outside 
> of the test suite.  So this might be the way to handle the problem in 
> the long run.

Feel free to install the above on master.

> But I still hope to figure out what's different about native
> compilation in the test suite.

You mean, why the fork fails?  I thought the PROBLEMS entry explains
that?

bug#62450: 29.0.60; Skip failing tests on Cygwin with native compilation

[Ken Brown]

On 3/26/2023 10:54 AM, Eli Zaretskii wrote:
>> Date: Sun, 26 Mar 2023 10:15:17 -0400
>> Cc: 62450-done@debbugs.gnu.org
>> From: Ken Brown <kbrown@cornell.edu>

>> I've been building and running Emacs like this for a couple months and
>> not doing any rebasing, and I haven't seen a single fork failure outside
>> of the test suite.  So this might be the way to handle the problem in
>> the long run.
> 
> Feel free to install the above on master.

Good idea.  That way maybe it will get some more testing.  AFAIK, I'm 
the only one who's tried this so far (at the suggestion of Corinna, the 
Cygwin maintainer).

>> But I still hope to figure out what's different about native
>> compilation in the test suite.
> 
> You mean, why the fork fails?  I thought the PROBLEMS entry explains
> that?

No, I mean why ASLR doesn't prevent the fork failure in the test suite, 
even though it prevents it while building Emacs and in my daily use of 
Emacs.  Or maybe a better question is whether my ASLR patch is actually 
taking effect for the compilations done in the test suite.  There has to 
be something different going on there, but I don't know what.

Ken