all messages for Guix-related lists mirrored at yhetil.org
 help / color / mirror / code / Atom feed
* WIP pypy3.3 - tests still fail? Anyone see why?
@ 2016-10-07 10:12 Danny Milosavljevic
  2016-10-08  8:09 ` Danny Milosavljevic
  2016-10-08 17:27 ` Danny Milosavljevic
  0 siblings, 2 replies; 4+ messages in thread
From: Danny Milosavljevic @ 2016-10-07 10:12 UTC (permalink / raw)
  To: guix-devel

Hi,

here's a work-in-progress package for pypy3.3. It builds - but the tests fail. A build takes about 6 GB of RAM and about 9 hours.

Maybe someone can spot how to fix one of the tests (usually they don't find libraries etc). (I'll work it out eventually but I thought I'd ask)

Also if guix build does fail executing some test - can I continue there somehow?

What do we do about setupterm? Disable the test? If hydra builds it there's probably no terminal available, right? The test it just testing readline's history manipulation so it's not like it's using the terminal...
I've already disabled test_curses.

(define-public pypy3.3
  (package
    (name "pypy3.3")
    (version "5.2.0")
    (source (origin
              (method url-fetch)
              (uri (string-append "https://bitbucket.org/pypy/pypy/downloads/"
                                  "pypy3.3-v" version "-alpha1-src.tar.bz2"))
              (sha256
               (base32
                "09hllsxds9ck22m92gy0ii9rwgqdp1d51c2bjrs15sl2ih42yk1l"))
              (patches (search-patches "pypy3.3-fix-compiler.patch"))
              (file-name (string-append "pypy3.3-v"
                                        version
                                        "-alpha1-src.tar.bz2"))))
    (build-system gnu-build-system)
    ;; TODO libgc for the untranslated tests
    (native-inputs
     ;; Note: python2-minimal doesn't work because ctypes.c_long is missing.
     `(("python-2" ,python-2)
       ("pkg-config" ,pkg-config)
       ("tar" ,tar)
       ("gzip" ,gzip)))
    (inputs
     `( ; for some reason this is not needed. ("bash" ,bash) ; actually /bin/sh
       ("libffi" ,libffi)
       ("zlib" ,zlib)
       ("ncurses" ,ncurses)
       ("openssl" ,openssl)
       ("expat" ,expat)
       ("bzip2" ,bzip2)
       ("sqlite" ,sqlite)
       ("gdbm" ,gdbm)
       ("tcl" ,tcl)
       ("tk" ,tk)
       ("xz" ,xz))) ; lzma
    (arguments
      `( ;#:modules ((srfi srfi-1)
          ;         (guix build utils))
        #:phases (modify-phases %standard-phases
         (delete 'configure)
         (replace 'build
            (lambda* (#:key inputs #:allow-other-keys)
              ;; distutils is used to verify extensions.
              ;; However, distutils build_ext uses a compiler that can only
              ;; be specified via CC environment variable.
              ;; Should have been used by customize_compiler() but wasn't.
              ;; Therefore we patch as well (in addition to setting it here).
              (setenv "CC" "gcc")
              ;; TODO set CPPFLAGS to something useful.
              ;; TODO set LDFLAGS rpath and other stuff
              (setenv "CCSHARED" "-fPIC")
               ;; TODO set LDFLAGS -L for tcl, tk, sqlite and rpath (mostly for the verification)
              (substitute* '("rpython/rtyper/lltypesystem/ll2ctypes.py")
                ;; Python ctypes.util.find_library documentation specifies
                ;; that it returns a file path, however all the actual
                ;; implementation cases return a SONAME.
                ;; Pypy expects it to return a file path.
                ;; Therefore, hotpatch ctypes.util to disable SONAME
                ;; determination.
                (("    import ctypes.util") "    import ctypes.util
    ctypes.util._get_soname = lambda x: x"))
              (substitute* '("lib-python/3/subprocess.py")
                (("\"/bin/sh\"") (string-append "\"" (which "sh") "\"")))
              (substitute* '("lib-python/3/distutils/log.py") ; FIXME remove
                (("__init__.self, threshold=WARN.") "__init__(self, threshold=INFO)")
                (("_global_log.threshold = level") "_x = level"))
              (substitute* '("lib_pypy/_tkinter/tklib_build.py")
                (("/usr/include/tcl") (string-append (assoc-ref inputs "tcl") "/include"))
                (("'tcl' + _ver, 'tk' + _ver") "'tcl8.6', 'tk8.6'") ; FIXME dyna version
)
              (substitute* '("lib_pypy/_sqlite3_build.py")
                (("libraries=.'sqlite3.$") (string-append "libraries= ['sqlite3'], "
                                                           "include_dirs='" (assoc-ref inputs "sqlite")  "/include', "
                                                           "library_dirs='" (assoc-ref inputs "sqlite") "/lib'")))
              ;; FIXME just use main Makefile
              (chdir "pypy/goal") ; FIXME error checking
              (and
                (zero? (system* "python" "../../rpython/bin/rpython"
                                "-Ojit" "targetpypystandalone")))))
         (replace 'check
            (lambda* (#:key inputs outputs #:allow-other-keys)
              (chdir "../..")
              (system* "pwd")
              (system* "ls") ; FIXME remove
              (display (getcwd))
              (setenv "LC_ALL" "en_US.utf8") ; make sure Python Unicode works
              (substitute* '("lib-python/3/test/test_shutil.py")
                ;; Make sure that the tests don't try to use uid / grp
                ;; (because we have no passwd entries in the container).
                (("UID_GID_SUPPORT = True") "UID_GID_SUPPORT = False")
                (("#!/bin/sh") (string-append "#!" (which "sh"))))
              (substitute* '("lib-python/3/test/test_posixpath.py")
                ;; Make sure that the tests don't try to use $HOME
                (("def test_expanduser") "
@unittest.skipIf(True, "Disabled on GNU Guix")
def test_expanduser"))
              (setenv "PYTHONPATH" (string-append (getcwd) ":"
                                                  (getcwd) "/lib_pypy:" ; for '_testcapi'
                                                  (getenv "PYTHONPATH")))
              (and
                ;; TODO: " ./pypy/test_all.py"
                ;;       "lib-python/2.7.0/test/test_complex.py"
                ;;       "--pypy=foobar"
                (zero? (system* "mkdir" "/tmp/pytest"))
                (zero? (system* "python" "-S", "testrunner/runner.py"
                                "--config=pypy/pytest-A.cfg" ; FIXME logfile is mandatory!
                                "--config=pypy/pytest-A.py"
                                "--logfile=/tmp/pytest/runner.log"
                                (string-append "--root=" (getcwd))
                                "--timeout=3600")) ; --config=~/machine-A_cfg.py
                (zero? (system* "python" "-S", "pypy/test_all.py"
                                "--pypy=pypy/goal/pypy-c" "--timeout=3600"
                                "lib-python"))
                (zero? (system* "python" "-S", "pypy/test_all.py"
                                "--pypy=pypy/goal/pypy-c"
                                "pypy/module/pypyjit/test"))
                (zero? (system* "pypy/goal/pypy-c" "pypy/test_all.py"
                                "pypy/module/pypyjit/test_pypy_c")))))
         (replace 'install
            (lambda* (#:key outputs #:allow-other-keys)
              (chdir "../tool/release")
              ; TODO package.py --archive-name foobar
              (zero? (system* "./package.py pypy-VER-PLATFORM")))))))
    (home-page "http://www.pypy.org/")
    (synopsis "Python 3.3 JIT")
    (description "This package provides a Python 3.3 just-in-time compiler.")
    (license (list license:expat
                   license:psfl
                   license:asl2.0
                   license:gpl2+
                   license:bsd-2
                   license:bsd-3
                   (license:non-copyleft
                    "http://www.unicode.org/copyright.html")))))

TODO:
        Make it find '_testcapi' which is in
./pypy3.3-v5.2.0-alpha1-src/lib_pypy/__pycache__/_pypy_testcapi.pypy3-52.pyc
./pypy3.3-v5.2.0-alpha1-src/lib_pypy/_testcapi.py
        Make it find sqlite (still).

FAIL lib-python/3/test/test_capi.py::unmodified
        can't find module '_testcapi'
FAIL lib-python/3/test/test_compileall.py::unmodified
        pyo vs. pyc
FAIL lib-python/3/test/test_ctypes.py::unmodified
        can't find module 'ctypes_test'
FAIL lib-python/3/test/test_decimal.py::unmodified
        MemoryError (MPD_Malloc_error) in test_context_subclassing, others.
        Invalid Operation in _eval_equation
FAIL lib-python/3/test/test_faulthandler.py::unmodified
        Traceback capturing
FAIL lib-python/3/test/test_imp.py::unmodified
        test_imp: test_load_dynamic_ImportError_path 'NoneType' object is not iterable
FAIL lib-python/3/test/test_keywordonlyarg.py::unmodified
        test.test_keywordonlyarg.KeywordOnlyArgTestCase
                Captured error message isn't nice
FAIL lib-python/3/test/test_numeric_tower.py::unmodified
        test_integers fails (equal hash: InvalidOperation)
        test_numeric_tower.ComparisonTest: MemoryError
FAIL lib-python/3/test/test_peepholer.py::unmodified
        test_binary_subscr_on_unicode IR
FAIL lib-python/3/test/test_posixpath.py::unmodified
        Fixed; skipping now
FAIL lib-python/3/test/test_readline.py::unmodified
        _minimal_curses.error: setupterm: could not find terminfo database
FAIL lib-python/3/test/test_runpy.py::unmodified
        Main recursion (overflow) test fails

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: WIP pypy3.3 - tests still fail? Anyone see why?
  2016-10-07 10:12 WIP pypy3.3 - tests still fail? Anyone see why? Danny Milosavljevic
@ 2016-10-08  8:09 ` Danny Milosavljevic
  2016-10-08 17:27 ` Danny Milosavljevic
  1 sibling, 0 replies; 4+ messages in thread
From: Danny Milosavljevic @ 2016-10-08  8:09 UTC (permalink / raw)
  To: guix-devel

[-- Attachment #1: Type: text/plain, Size: 17 bytes --]

and the patch...

[-- Attachment #2: pypy3.3-fix-compiler.patch --]
[-- Type: text/x-patch, Size: 1385 bytes --]

--- /tmp/sysconfig_pypy.py	2016-09-14 00:38:36.325334296 +0200
+++ pypy3.3-v5.2.0-alpha1-src/lib-python/3/distutils/sysconfig_pypy.py	2016-09-14 00:37:55.068908258 +0200
@@ -123,6 +123,19 @@
     optional C speedup components.
     """
     if compiler.compiler_type == "unix":
+        varnames = ["CC", "CXX", "LDSHARED", "CPP", "LDFLAGS", "CFLAGS", "CPPFLAGS", "AR", "ARFLAGS", "OPT", "CFLAGS", "CCSHARED", "SHLIB_SUFFIX"]
+        cfg = get_config_vars()
+        g = {}
+        for varname in varnames:
+            g[varname] = os.environ.get(varname) or cfg.get(varname)
+        cc = g["CC"] or "cc"
+        cpp = g["CPP"] if "CPP" in os.environ else ("{} -E".format(g["CC"]))
+        cc_cmd = "{} {}".format(g["CC"], g["CFLAGS"] or "")
+        ccshared = g["CCSHARED"] or ""
+        cxx = g["CXX"]
+        ldshared = "{} -shared {} {} {}".format(g["LDSHARED"] or g["CC"], os.environ.get("LDFLAGS", ""), os.environ.get("CFLAGS", ""), os.environ.get("CPPFLAGS", ""))
+        archiver = "{} {}".format(g["AR"], g["ARFLAGS"])
+        compiler.set_executables(preprocessor=cpp, compiler=cc_cmd, compiler_so=cc_cmd + " " + ccshared, compiler_cxx=cxx, linker_so=ldshared, linker_exe=cc, archiver=archiver)
         compiler.compiler_so.extend(['-O2', '-fPIC', '-Wimplicit'])
         compiler.shared_lib_extension = get_config_var('SO')
         if "CPPFLAGS" in os.environ:

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: WIP pypy3.3 - tests still fail? Anyone see why?
  2016-10-07 10:12 WIP pypy3.3 - tests still fail? Anyone see why? Danny Milosavljevic
  2016-10-08  8:09 ` Danny Milosavljevic
@ 2016-10-08 17:27 ` Danny Milosavljevic
  2016-10-09  6:51   ` Leo Famulari
  1 sibling, 1 reply; 4+ messages in thread
From: Danny Milosavljevic @ 2016-10-08 17:27 UTC (permalink / raw)
  To: guix-devel

> Also if guix build does fail executing some test - can I continue there somehow?

Uhhh... I mean: if a test fails, can I make guix open a shell there (with all the right environment etc) so I can examine what's up, let me fix it and then restart the phase?

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: WIP pypy3.3 - tests still fail? Anyone see why?
  2016-10-08 17:27 ` Danny Milosavljevic
@ 2016-10-09  6:51   ` Leo Famulari
  0 siblings, 0 replies; 4+ messages in thread
From: Leo Famulari @ 2016-10-09  6:51 UTC (permalink / raw)
  To: Danny Milosavljevic; +Cc: guix-devel

On Sat, Oct 08, 2016 at 07:27:08PM +0200, Danny Milosavljevic wrote:
> > Also if guix build does fail executing some test - can I continue there somehow?
> 
> Uhhh... I mean: if a test fails, can I make guix open a shell there (with all the right environment etc) so I can examine what's up, let me fix it and then restart the phase?

You can approximate it with `guix build --keep-failed`.

Then, you could enter the directory of the failed build and do `guix
environment --container`. I think that should be a very similar
environment that you can repeat the tests in.

^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2016-10-09  6:51 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2016-10-07 10:12 WIP pypy3.3 - tests still fail? Anyone see why? Danny Milosavljevic
2016-10-08  8:09 ` Danny Milosavljevic
2016-10-08 17:27 ` Danny Milosavljevic
2016-10-09  6:51   ` Leo Famulari

Code repositories for project(s) associated with this external index

	https://git.savannah.gnu.org/cgit/guix.git

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.