WIP pypy3.3 - tests still fail? Anyone see why?

WIP pypy3.3 - tests still fail? Anyone see why?

[Danny Milosavljevic]

Hi,

here's a work-in-progress package for pypy3.3. It builds - but the tests fail. A build takes about 6 GB of RAM and about 9 hours.

Maybe someone can spot how to fix one of the tests (usually they don't find libraries etc). (I'll work it out eventually but I thought I'd ask)

Also if guix build does fail executing some test - can I continue there somehow?

What do we do about setupterm? Disable the test? If hydra builds it there's probably no terminal available, right? The test it just testing readline's history manipulation so it's not like it's using the terminal...
I've already disabled test_curses.

(define-public pypy3.3
  (package
    (name "pypy3.3")
    (version "5.2.0")
    (source (origin
              (method url-fetch)
              (uri (string-append "https://bitbucket.org/pypy/pypy/downloads/"
                                  "pypy3.3-v" version "-alpha1-src.tar.bz2"))
              (sha256
               (base32
                "09hllsxds9ck22m92gy0ii9rwgqdp1d51c2bjrs15sl2ih42yk1l"))
              (patches (search-patches "pypy3.3-fix-compiler.patch"))
              (file-name (string-append "pypy3.3-v"
                                        version
                                        "-alpha1-src.tar.bz2"))))
    (build-system gnu-build-system)
    ;; TODO libgc for the untranslated tests
    (native-inputs
     ;; Note: python2-minimal doesn't work because ctypes.c_long is missing.
     `(("python-2" ,python-2)
       ("pkg-config" ,pkg-config)
       ("tar" ,tar)
       ("gzip" ,gzip)))
    (inputs
     `( ; for some reason this is not needed. ("bash" ,bash) ; actually /bin/sh
       ("libffi" ,libffi)
       ("zlib" ,zlib)
       ("ncurses" ,ncurses)
       ("openssl" ,openssl)
       ("expat" ,expat)
       ("bzip2" ,bzip2)
       ("sqlite" ,sqlite)
       ("gdbm" ,gdbm)
       ("tcl" ,tcl)
       ("tk" ,tk)
       ("xz" ,xz))) ; lzma
    (arguments
      `( ;#:modules ((srfi srfi-1)
          ;         (guix build utils))
        #:phases (modify-phases %standard-phases
         (delete 'configure)
         (replace 'build
            (lambda* (#:key inputs #:allow-other-keys)
              ;; distutils is used to verify extensions.
              ;; However, distutils build_ext uses a compiler that can only
              ;; be specified via CC environment variable.
              ;; Should have been used by customize_compiler() but wasn't.
              ;; Therefore we patch as well (in addition to setting it here).
              (setenv "CC" "gcc")
              ;; TODO set CPPFLAGS to something useful.
              ;; TODO set LDFLAGS rpath and other stuff
              (setenv "CCSHARED" "-fPIC")
               ;; TODO set LDFLAGS -L for tcl, tk, sqlite and rpath (mostly for the verification)
              (substitute* '("rpython/rtyper/lltypesystem/ll2ctypes.py")
                ;; Python ctypes.util.find_library documentation specifies
                ;; that it returns a file path, however all the actual
                ;; implementation cases return a SONAME.
                ;; Pypy expects it to return a file path.
                ;; Therefore, hotpatch ctypes.util to disable SONAME
                ;; determination.
                (("    import ctypes.util") "    import ctypes.util
    ctypes.util._get_soname = lambda x: x"))
              (substitute* '("lib-python/3/subprocess.py")
                (("\"/bin/sh\"") (string-append "\"" (which "sh") "\"")))
              (substitute* '("lib-python/3/distutils/log.py") ; FIXME remove
                (("__init__.self, threshold=WARN.") "__init__(self, threshold=INFO)")
                (("_global_log.threshold = level") "_x = level"))
              (substitute* '("lib_pypy/_tkinter/tklib_build.py")
                (("/usr/include/tcl") (string-append (assoc-ref inputs "tcl") "/include"))
                (("'tcl' + _ver, 'tk' + _ver") "'tcl8.6', 'tk8.6'") ; FIXME dyna version
)
              (substitute* '("lib_pypy/_sqlite3_build.py")
                (("libraries=.'sqlite3.$") (string-append "libraries= ['sqlite3'], "
                                                           "include_dirs='" (assoc-ref inputs "sqlite")  "/include', "
                                                           "library_dirs='" (assoc-ref inputs "sqlite") "/lib'")))
              ;; FIXME just use main Makefile
              (chdir "pypy/goal") ; FIXME error checking
              (and
                (zero? (system* "python" "../../rpython/bin/rpython"
                                "-Ojit" "targetpypystandalone")))))
         (replace 'check
            (lambda* (#:key inputs outputs #:allow-other-keys)
              (chdir "../..")
              (system* "pwd")
              (system* "ls") ; FIXME remove
              (display (getcwd))
              (setenv "LC_ALL" "en_US.utf8") ; make sure Python Unicode works
              (substitute* '("lib-python/3/test/test_shutil.py")
                ;; Make sure that the tests don't try to use uid / grp
                ;; (because we have no passwd entries in the container).
                (("UID_GID_SUPPORT = True") "UID_GID_SUPPORT = False")
                (("#!/bin/sh") (string-append "#!" (which "sh"))))
              (substitute* '("lib-python/3/test/test_posixpath.py")
                ;; Make sure that the tests don't try to use $HOME
                (("def test_expanduser") "
@unittest.skipIf(True, "Disabled on GNU Guix")
def test_expanduser"))
              (setenv "PYTHONPATH" (string-append (getcwd) ":"
                                                  (getcwd) "/lib_pypy:" ; for '_testcapi'
                                                  (getenv "PYTHONPATH")))
              (and
                ;; TODO: " ./pypy/test_all.py"
                ;;       "lib-python/2.7.0/test/test_complex.py"
                ;;       "--pypy=foobar"
                (zero? (system* "mkdir" "/tmp/pytest"))
                (zero? (system* "python" "-S", "testrunner/runner.py"
                                "--config=pypy/pytest-A.cfg" ; FIXME logfile is mandatory!
                                "--config=pypy/pytest-A.py"
                                "--logfile=/tmp/pytest/runner.log"
                                (string-append "--root=" (getcwd))
                                "--timeout=3600")) ; --config=~/machine-A_cfg.py
                (zero? (system* "python" "-S", "pypy/test_all.py"
                                "--pypy=pypy/goal/pypy-c" "--timeout=3600"
                                "lib-python"))
                (zero? (system* "python" "-S", "pypy/test_all.py"
                                "--pypy=pypy/goal/pypy-c"
                                "pypy/module/pypyjit/test"))
                (zero? (system* "pypy/goal/pypy-c" "pypy/test_all.py"
                                "pypy/module/pypyjit/test_pypy_c")))))
         (replace 'install
            (lambda* (#:key outputs #:allow-other-keys)
              (chdir "../tool/release")
              ; TODO package.py --archive-name foobar
              (zero? (system* "./package.py pypy-VER-PLATFORM")))))))
    (home-page "http://www.pypy.org/")
    (synopsis "Python 3.3 JIT")
    (description "This package provides a Python 3.3 just-in-time compiler.")
    (license (list license:expat
                   license:psfl
                   license:asl2.0
                   license:gpl2+
                   license:bsd-2
                   license:bsd-3
                   (license:non-copyleft
                    "http://www.unicode.org/copyright.html")))))

TODO:
        Make it find '_testcapi' which is in
./pypy3.3-v5.2.0-alpha1-src/lib_pypy/__pycache__/_pypy_testcapi.pypy3-52.pyc
./pypy3.3-v5.2.0-alpha1-src/lib_pypy/_testcapi.py
        Make it find sqlite (still).

FAIL lib-python/3/test/test_capi.py::unmodified
        can't find module '_testcapi'
FAIL lib-python/3/test/test_compileall.py::unmodified
        pyo vs. pyc
FAIL lib-python/3/test/test_ctypes.py::unmodified
        can't find module 'ctypes_test'
FAIL lib-python/3/test/test_decimal.py::unmodified
        MemoryError (MPD_Malloc_error) in test_context_subclassing, others.
        Invalid Operation in _eval_equation
FAIL lib-python/3/test/test_faulthandler.py::unmodified
        Traceback capturing
FAIL lib-python/3/test/test_imp.py::unmodified
        test_imp: test_load_dynamic_ImportError_path 'NoneType' object is not iterable
FAIL lib-python/3/test/test_keywordonlyarg.py::unmodified
        test.test_keywordonlyarg.KeywordOnlyArgTestCase
                Captured error message isn't nice
FAIL lib-python/3/test/test_numeric_tower.py::unmodified
        test_integers fails (equal hash: InvalidOperation)
        test_numeric_tower.ComparisonTest: MemoryError
FAIL lib-python/3/test/test_peepholer.py::unmodified
        test_binary_subscr_on_unicode IR
FAIL lib-python/3/test/test_posixpath.py::unmodified
        Fixed; skipping now
FAIL lib-python/3/test/test_readline.py::unmodified
        _minimal_curses.error: setupterm: could not find terminfo database
FAIL lib-python/3/test/test_runpy.py::unmodified
        Main recursion (overflow) test fails

Re: WIP pypy3.3 - tests still fail? Anyone see why?

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 17 bytes --]

and the patch...

[-- Attachment #2: pypy3.3-fix-compiler.patch --]
[-- Type: text/x-patch, Size: 1385 bytes --]

--- /tmp/sysconfig_pypy.py	2016-09-14 00:38:36.325334296 +0200
+++ pypy3.3-v5.2.0-alpha1-src/lib-python/3/distutils/sysconfig_pypy.py	2016-09-14 00:37:55.068908258 +0200
@@ -123,6 +123,19 @@
     optional C speedup components.
     """
     if compiler.compiler_type == "unix":
+        varnames = ["CC", "CXX", "LDSHARED", "CPP", "LDFLAGS", "CFLAGS", "CPPFLAGS", "AR", "ARFLAGS", "OPT", "CFLAGS", "CCSHARED", "SHLIB_SUFFIX"]
+        cfg = get_config_vars()
+        g = {}
+        for varname in varnames:
+            g[varname] = os.environ.get(varname) or cfg.get(varname)
+        cc = g["CC"] or "cc"
+        cpp = g["CPP"] if "CPP" in os.environ else ("{} -E".format(g["CC"]))
+        cc_cmd = "{} {}".format(g["CC"], g["CFLAGS"] or "")
+        ccshared = g["CCSHARED"] or ""
+        cxx = g["CXX"]
+        ldshared = "{} -shared {} {} {}".format(g["LDSHARED"] or g["CC"], os.environ.get("LDFLAGS", ""), os.environ.get("CFLAGS", ""), os.environ.get("CPPFLAGS", ""))
+        archiver = "{} {}".format(g["AR"], g["ARFLAGS"])
+        compiler.set_executables(preprocessor=cpp, compiler=cc_cmd, compiler_so=cc_cmd + " " + ccshared, compiler_cxx=cxx, linker_so=ldshared, linker_exe=cc, archiver=archiver)
         compiler.compiler_so.extend(['-O2', '-fPIC', '-Wimplicit'])
         compiler.shared_lib_extension = get_config_var('SO')
         if "CPPFLAGS" in os.environ:

Re: WIP pypy3.3 - tests still fail? Anyone see why?

[Danny Milosavljevic]

> Also if guix build does fail executing some test - can I continue there somehow?

Uhhh... I mean: if a test fails, can I make guix open a shell there (with all the right environment etc) so I can examine what's up, let me fix it and then restart the phase?

Re: WIP pypy3.3 - tests still fail? Anyone see why?

[Leo Famulari]

On Sat, Oct 08, 2016 at 07:27:08PM +0200, Danny Milosavljevic wrote:
> > Also if guix build does fail executing some test - can I continue there somehow?
> 
> Uhhh... I mean: if a test fails, can I make guix open a shell there (with all the right environment etc) so I can examine what's up, let me fix it and then restart the phase?

You can approximate it with `guix build --keep-failed`.

Then, you could enter the directory of the failed build and do `guix
environment --container`. I think that should be a very similar
environment that you can repeat the tests in.