unofficial mirror of guix-devel@gnu.org 
 help / color / mirror / code / Atom feed
* [Python] pypy3 integration
@ 2020-07-19  8:27 Lars-Dominik Braun
  2020-07-20 11:17 ` zimoun
  2020-07-20 19:42 ` Jakub Kądziołka
  0 siblings, 2 replies; 10+ messages in thread
From: Lars-Dominik Braun @ 2020-07-19  8:27 UTC (permalink / raw)
  To: guix-devel

[-- Attachment #1: Type: text/plain, Size: 1609 bytes --]

Hi,

I’ve been looking into integrating pypy3 (a JIT Python 3 implementation)
tighter into guix. Right now we have a package, but it’s not that
useful, because there are no pypy3-* libraries.

The WIP patch attached – I’ll split it up later if this gets to code
review – adds some essential Python libraries like pytest and numpy to
pypy3. One issue however is the way Python’s site-packages directory is
discovered.  Usually it’s in /lib/python<python_version>/site-packages.
I modified pypy3 to use /lib/pypy<python_version>/site-packages
(python_version!=pypy version), but I’m having a hard time figuring out
how to build this path inside guix/build/python-build-system.scm’s
site-packages, because the inputs have been lowered to store paths
already. Right now I’m simply scanning the input’s /lib subdirectories,
but I had to add python→python3 symlinks to our python package to make
it work. This is discouraged by PEP 394[1] and triggers a world rebuild.

I thought about multiple alternative options:

1) Introducing a symlink in /share/guix/well-known/python/site-packages
   (or similar) for all three Python implementations (Python 2, Python
   3, PyPy 3) and use that
2) Use fixed /lib/python/site-packages for all Python packages (no two
   implementations can be used in the same environment right now anyway,
   due to shared PYTHONPATH)
3) Somehow access package metadata (native search path, properties, …)
   and encode the path there. This would be preferred, but is it even
   possible?

Thoughs?

Thanks,
Lars

[1] https://legacy.python.org/dev/peps/pep-0394/


[-- Attachment #2: 0001-WIP-Proper-pypy3-support.patch --]
[-- Type: text/x-diff, Size: 41268 bytes --]

From 66e5568885199a734fda7946a6cbe32a09218dee Mon Sep 17 00:00:00 2001
From: Lars-Dominik Braun <lars@6xq.net>
Date: Fri, 10 Jul 2020 14:12:36 +0200
Subject: [PATCH] WIP: Proper pypy3 support

---
 gnu/packages/check.scm                        |  62 ++++++-
 .../patches/pypy3-7.3.1-install-paths.patch   |  19 ++
 .../patches/pypy3-7.3.1-ssl-paths.patch       |  41 +++++
 gnu/packages/python-science.scm               |   3 +
 gnu/packages/python-xyz.scm                   | 168 +++++++++++++++++-
 gnu/packages/python.scm                       | 103 +++++++----
 gnu/packages/xml.scm                          |  20 ++-
 guix/build-system/python.scm                  |  24 ++-
 guix/build/python-build-system.scm            |  18 +-
 9 files changed, 403 insertions(+), 55 deletions(-)
 create mode 100644 gnu/packages/patches/pypy3-7.3.1-install-paths.patch
 create mode 100644 gnu/packages/patches/pypy3-7.3.1-ssl-paths.patch

diff --git a/gnu/packages/check.scm b/gnu/packages/check.scm
index 3b946d7482..29ad6e144a 100644
--- a/gnu/packages/check.scm
+++ b/gnu/packages/check.scm
@@ -726,6 +726,9 @@ have been used.")
          ("python2-funcsigs" ,python2-funcsigs)
          ,@(package-propagated-inputs base))))))
 
+(define-public pypy3-mock
+  (package-with-pypy3 (strip-pypy3-variant python-mock)))
+
 (define-public python-nose
   (package
     (name "python-nose")
@@ -749,6 +752,9 @@ have been used.")
 (define-public python2-nose
   (package-with-python2 python-nose))
 
+(define-public pypy3-nose
+  (package-with-pypy3 python-nose))
+
 (define-public python-nose2
   (package
     (name "python-nose2")
@@ -778,6 +784,9 @@ interfaces and processes.")
 (define-public python2-nose2
   (package-with-python2 python-nose2))
 
+(define-public pypy3-nose2
+  (package-with-pypy3 python-nose2))
+
 (define-public python-unittest2
   (package
     (name "python-unittest2")
@@ -865,7 +874,8 @@ standard library.")
 and functions, detailed info on failing assert statements, modular fixtures,
 and many external plugins.")
     (license license:expat)
-    (properties `((python2-variant . ,(delay python2-pytest))))))
+    (properties `((python2-variant . ,(delay python2-pytest))
+                  (pypy3-variant . ,(delay pypy3-pytest))))))
 
 ;; Pytest 4.x are the last versions that support Python 2.
 (define-public python2-pytest
@@ -903,13 +913,17 @@ and many external plugins.")
        ("python-pytest" ,python2-pytest-bootstrap)
        ("python-setuptools-scm" ,python2-setuptools-scm)))))
 
+(define-public pypy3-pytest
+			   (package-with-pypy3 (strip-pypy3-variant python-pytest)))
+
 (define-public python-pytest-bootstrap
   (package
     (inherit (strip-python2-variant python-pytest))
     (name "python-pytest-bootstrap")
     (native-inputs `(("python-setuptools-scm" ,python-setuptools-scm)))
     (arguments `(#:tests? #f))
-    (properties `((python2-variant . ,(delay python2-pytest-bootstrap))))))
+    (properties `((python2-variant . ,(delay python2-pytest-bootstrap))
+				  (pypy3-variant . ,(delay pypy3-pytest-bootstrap))))))
 
 (define-public python2-pytest-bootstrap
   (hidden-package
@@ -933,6 +947,22 @@ and many external plugins.")
         ("python-py" ,python2-py)
         ("python-wcwidth" ,python2-wcwidth))))))
 
+(define-public pypy3-pytest-bootstrap
+  (let ((base (package-with-pypy3 (strip-pypy3-variant python-pytest-bootstrap))))
+    (package/inherit
+      base
+      (name "pypy3-pytest-bootstrap")
+    (native-inputs `(("pypy3-setuptools-scm" ,pypy3-setuptools-scm)))
+    (propagated-inputs
+     `(("python-atomicwrites" ,pypy3-atomicwrites)
+       ("python-attrs" ,pypy3-attrs-bootstrap)
+       ("python-more-itertools" ,pypy3-more-itertools)
+       ("python-packaging" ,pypy3-packaging-bootstrap)
+       ("python-pluggy" ,pypy3-pluggy-bootstrap)
+       ("python-py" ,pypy3-py)
+       ("python-six" ,pypy3-six-bootstrap)
+       ("python-wcwidth" ,pypy3-wcwidth))))))
+
 (define-public python-pytest-cov
   (package
     (name "python-pytest-cov")
@@ -1840,6 +1870,9 @@ seamlessly into your existing Python unit testing work flow.")
        `(("python2-enum34" ,python2-enum34)
          ,@(package-propagated-inputs hypothesis))))))
 
+(define-public pypy3-hypothesis
+  (package-with-pypy3 python-hypothesis))
+
 (define-public python-lit
   (package
     (name "python-lit")
@@ -2687,11 +2720,12 @@ grew out of the @dfn{Vc} project.")
        (modify-phases %standard-phases
          ;; The default test suite does not run these extra tests.
          (add-after 'check 'check-pytest-plugin
-           (lambda _
+           (lambda* (#:key tests? #:allow-other-keys)
+			 (if tests?
              (invoke
               "python" "-m" "pytest"
               "pyfakefs/pytest_tests/pytest_plugin_test.py")
-             #t)))))
+             #t))))))
     (native-inputs
      `(("python-pytest" ,python-pytest)))
     (build-system python-build-system)
@@ -2710,11 +2744,20 @@ the test.  The pyfakefs library provides a solution to problems like this by
 mocking file system interactions.  In other words, it arranges for the code
 under test to interact with a fake file system instead of the real file
 system.  The code under test requires no modification to work with pyfakefs.")
-    (license license:asl2.0)))
+    (license license:asl2.0)
+    (properties `((pypy3-variant . ,(delay pypy3-pyfakefs))))))
 
 (define-public python2-pyfakefs
   (package-with-python2 python-pyfakefs))
 
+(define-public pypy3-pyfakefs
+  (let ((base (package-with-pypy3 (strip-pypy3-variant python-pyfakefs))))
+    (package/inherit
+	  base
+      (arguments
+        `(#:tests? #f
+          ,@(package-arguments base))))))
+
 ;; This minimal variant is used to avoid a circular dependency between
 ;; python2-importlib-metadata, which requires pyfakefs for its tests, and
 ;; python2-pytest, which requires python2-importlib-metadata.
@@ -2728,6 +2771,15 @@ system.  The code under test requires no modification to work with pyfakefs.")
       `(#:python ,python-2
         #:tests? #f)))))
 
+(define-public pypy3-pyfakefs-bootstrap
+  (hidden-package
+   (package
+     (inherit pypy3-pyfakefs)
+     (name "pypy3-pyfakefs-bootstrap")
+     (native-inputs '())
+     (arguments
+      `(#:tests? #f ,@(package-arguments pypy3-pyfakefs))))))
+
 (define-public python-aiounittest
   (package
     (name "python-aiounittest")
diff --git a/gnu/packages/patches/pypy3-7.3.1-install-paths.patch b/gnu/packages/patches/pypy3-7.3.1-install-paths.patch
new file mode 100644
index 0000000000..bd987bd4bc
--- /dev/null
+++ b/gnu/packages/patches/pypy3-7.3.1-install-paths.patch
@@ -0,0 +1,19 @@
+Move site-packages to /lib/pypyX.X/site-packages, so it matches Python’s
+behavior.
+
+diff -Naur pypy3.6-v7.3.1-src.orig/lib-python/3/distutils/command/install.py pypy3.6-v7.3.1-src/lib-python/3/distutils/command/install.py
+--- pypy3.6-v7.3.1-src.orig/lib-python/3/distutils/command/install.py	1970-01-01 01:00:01.000000000 +0100
++++ pypy3.6-v7.3.1-src/lib-python/3/distutils/command/install.py	2020-07-11 10:18:34.357208155 +0200
+@@ -44,9 +44,9 @@
+         },
+     'nt': WINDOWS_SCHEME,
+     'pypy': {
+-        'purelib': '$base/site-packages',
+-        'platlib': '$base/site-packages',
+-        'headers': '$base/include/$dist_name',
++        'purelib': '$base/lib/pypy$py_version_short/site-packages',
++        'platlib': '$platbase/lib/pypy$py_version_short/site-packages',
++        'headers': '$base/include/pypy$py_version_short$abiflags/$dist_name',
+         'scripts': '$base/bin',
+         'data'   : '$base',
+         },
diff --git a/gnu/packages/patches/pypy3-7.3.1-ssl-paths.patch b/gnu/packages/patches/pypy3-7.3.1-ssl-paths.patch
new file mode 100644
index 0000000000..d21133b4ae
--- /dev/null
+++ b/gnu/packages/patches/pypy3-7.3.1-ssl-paths.patch
@@ -0,0 +1,41 @@
+Fix default certificate search path, still allowing the user to override it
+with environment variables.
+
+--- a/lib_pypy/_cffi_ssl/_stdssl/__init__.py
++++ b/lib_pypy/_cffi_ssl/_stdssl/__init__.py
+@@ -1679,20 +1679,9 @@ def get_default_verify_paths():
+      https://golang.org/src/crypto/x509/root_linux.go (for the files)
+     '''
+     certFiles = [
+-        "/etc/ssl/certs/ca-certificates.crt",                # Debian/Ubuntu/Gentoo etc.
+-        "/etc/pki/tls/certs/ca-bundle.crt",                  # Fedora/RHEL 6
+-        "/etc/ssl/ca-bundle.pem",                            # OpenSUSE
+-        "/etc/pki/tls/cacert.pem",                           # OpenELEC
+-        "/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem", # CentOS/RHEL 7
+-        "/etc/ssl/cert.pem",                                 # Alpine Linux
+     ]
+     certDirectories = [
+-        "/etc/ssl/certs",               # SLES10/SLES11
+-        "/system/etc/security/cacerts", # Android
+-        "/usr/local/share/certs",       # FreeBSD
+-        "/etc/pki/tls/certs",           # Fedora/RHEL
+-        "/etc/openssl/certs",           # NetBSD
+-        "/var/ssl/certs",               # AIX
++        "@GUIX_CERT_PATH@",
+     ]
+ 
+     # optimization: reuse the values from a local varaible
+@@ -1707,9 +1696,10 @@ def get_default_verify_paths():
+     ofile = _cstr_decode_fs(lib.X509_get_default_cert_file())
+     odir = _cstr_decode_fs(lib.X509_get_default_cert_dir())
+ 
+-    if os.path.exists(ofile) and os.path.exists(odir):
+-        get_default_verify_paths.retval = (ofile_env, ofile, odir_env, odir)
+-        return get_default_verify_paths.retval
++    if not os.path.exists(ofile):
++        ofile = None
++    if not os.path.exists(odir):
++        odir = None
+ 
+     # OpenSSL didn't supply the goods. Try some other options
+     for f in certFiles:
diff --git a/gnu/packages/python-science.scm b/gnu/packages/python-science.scm
index 3d6d4e909e..750f46fb67 100644
--- a/gnu/packages/python-science.scm
+++ b/gnu/packages/python-science.scm
@@ -355,6 +355,9 @@ doing practical, real world data analysis in Python.")
                        "if 'NULL byte' in msg or 'line contains NUL' in msg:"))
                     #t)))))))
 
+(define-public pypy3-pandas
+			   (package-with-pypy3 python-pandas))
+
 (define-public python-xarray
   (package
     (name "python-xarray")
diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm
index 7079bbda5d..8229db3af9 100644
--- a/gnu/packages/python-xyz.scm
+++ b/gnu/packages/python-xyz.scm
@@ -1644,6 +1644,9 @@ Python file, so it can be easily copied into your project.")
 (define-public python2-six-bootstrap
   (package-with-python2 python-six-bootstrap))
 
+(define-public pypy3-six-bootstrap
+  (package-with-pypy3 python-six-bootstrap))
+
 (define-public python-schedule
   (package
     (name "python-schedule")
@@ -2245,6 +2248,9 @@ code introspection, and logging.")
 (define-public python2-py
   (package-with-python2 python-py))
 
+(define-public pypy3-py
+  (package-with-pypy3 python-py))
+
 ;; Recent versions of python-fixtures and python-testrepository need
 ;; python-pbr for packaging, which itself needs these two packages for
 ;; testing.
@@ -2961,6 +2967,9 @@ e.g. filters, callbacks and errbacks can all be promises.")
     (properties `((python2-variant . ,(delay python2-virtualenv))))
     (license license:expat)))
 
+(define-public pypy3-virtualenv
+  (package-with-pypy3 python-virtualenv))
+
 (define-public python2-virtualenv
   (let ((base (package-with-python2 (strip-python2-variant python-virtualenv))))
     (package
@@ -4031,7 +4040,9 @@ provides additional functionality on the produced Mallard documents.")
            ;; some tests require access to "$HOME/.cython"
            (lambda _ (setenv "HOME" "/tmp") #t))
          (replace 'check
-           (lambda _
+           (lambda* (#:key tests? #:allow-other-keys)
+             (if tests?
+               (begin
              ;; Disable compiler optimizations to greatly reduce the running
              ;; time of the test suite.
              (setenv "CFLAGS" "-O0")
@@ -4043,14 +4054,16 @@ provides additional functionality on the produced Mallard documents.")
                      ;; <https://github.com/cython/cython/issues/2807>.
                      ,@(if (not (target-64bit?))
                            '("-x" "run.parallel")
-                           '())))))))
+                           '()))
+             #t)))))))
     (home-page "https://cython.org/")
     (synopsis "C extensions for Python")
     (description "Cython is an optimising static compiler for both the Python
 programming language and the extended Cython programming language.  It makes
 writing C extensions for Python as easy as Python itself.")
     (license license:asl2.0)
-    (properties `((python2-variant . ,(delay python2-cython))))))
+    (properties `((python2-variant . ,(delay python2-cython))
+				  (pypy3-variant . ,(delay pypy3-cython))))))
 
 (define-public python2-cython
   (let ((base (package-with-python2 (strip-python2-variant python-cython))))
@@ -4087,6 +4100,12 @@ writing C extensions for Python as easy as Python itself.")
                     ""))
                  #t)))))))))
 
+(define-public pypy3-cython
+  (let ((pkg (package-with-pypy3 (strip-pypy3-variant python-cython))))
+  (package (inherit pkg)
+		   ; Tests get stuck at some point
+	(arguments `(#:tests? #f ,@(package-arguments pkg))))))
+
 ;; The RPython toolchain currently does not support Python 3.
 (define-public python2-rpython
   (package
@@ -4176,6 +4195,7 @@ include_dirs = ~a/include
            (lambda* (#:key outputs inputs #:allow-other-keys)
              ;; Make installed package available for running the tests
              (add-installed-pythonpath inputs outputs)
+             (system "export > $NIX_BUILD_TOP/environment-variables2")
              ;; Make sure "f2py" etc is found.
              (setenv "PATH" (string-append (assoc-ref outputs "out") "/bin"
                                            ":" (getenv "PATH")))
@@ -4208,6 +4228,9 @@ capabilities.")
                  (base32
                   "0lg1cycxzi4rvvrd5zxinpdz0ni792fpx6xjd75z1923zcac8qrb")))))))
 
+(define-public pypy3-numpy
+  (package-with-pypy3 python-numpy))
+
 ;; NOTE: NumPy 1.8 is packaged only for Python 2 because it is of
 ;; interest only for legacy code going back to NumPy's predecessor
 ;; Numeric.
@@ -5198,7 +5221,16 @@ by pycodestyle.")
     (description "Distlib is a library which implements low-level functions that
 relate to packaging and distribution of Python software.  It is intended to be
 used as the basis for third-party packaging tools.")
-    (license license:psfl)))
+    (license license:psfl)
+    (properties `((pypy3-variant . ,(delay pypy3-distlib))))))
+
+(define-public pypy3-distlib
+  (let ((base (package-with-pypy3 (strip-pypy3-variant python-distlib))))
+	(package/inherit
+	  base
+	  (arguments
+		; tests failb because they try to rmtree a read-only store copy
+		`(#:tests? #f ,@(package-arguments base))))))
 
 (define-public python-distutils-extra
   (package
@@ -5943,6 +5975,9 @@ them as the version argument or in a SCM managed file.")
 (define-public python2-setuptools-scm
   (package-with-python2 python-setuptools-scm))
 
+(define-public pypy3-setuptools-scm
+  (package-with-pypy3 python-setuptools-scm))
+
 (define-public python-sexpdata
   (package
     (name "python-sexpdata")
@@ -5985,6 +6020,9 @@ all the newest features of the standard @code{pathlib} can be used also on
 older Python versions.")
     (license license:expat)))
 
+(define-public pypy3-pathlib2
+  (package-with-pypy3 python-pathlib2))
+
 (define-public python2-importlib-resources
   (package
     (name "python2-importlib-resources")
@@ -6018,6 +6056,39 @@ older Python versions.")
 for older versions of Python.")
     (license license:asl2.0)))
 
+(define-public pypy3-importlib-resources
+  (package
+    (name "pypy3-importlib-resources")
+    (version "1.0.2")
+    (source (origin
+              (method url-fetch)
+              (uri (pypi-uri "importlib_resources" version))
+              (sha256
+               (base32
+                "0y3hg12iby1qyaspnbisz4s4vxax7syikk3skznwqizqyv89y9yk"))))
+    (build-system python-build-system)
+    (arguments
+     `(#:python ,pypy3
+       #:phases (modify-phases %standard-phases
+                  ;; The build system tests for python-wheel, but it is
+                  ;; not required for Guix nor the test suite.  Just drop
+                  ;; it to make bootstrapping pytest easier.
+                  (add-after 'unpack 'drop-wheel-dependency
+                    (lambda _
+                      (substitute* "setup.cfg"
+                        (("^[[:blank:]]+wheel")
+                         ""))
+                      #t)))))
+    (propagated-inputs
+     `(("python-pathlib2" ,pypy3-pathlib2)
+       ("python-typing" ,pypy3-typing)))
+    (home-page "https://gitlab.com/python-devs/importlib_resources")
+    (synopsis "Backport of @code{importlib.resources} from Python 3.7")
+    (description
+     "This package provides an implementation of @code{importlib.resources}
+for older versions of Python.")
+    (license license:asl2.0)))
+
 ;; For importlib-metadata-bootstrap below.
 (define-public python2-importlib-resources-bootstrap
   (hidden-package
@@ -6028,6 +6099,15 @@ for older versions of Python.")
      `(("python-pathlib2-bootstrap" ,python2-pathlib2-bootstrap)
        ("python-typing" ,python2-typing))))))
 
+(define-public pypy3-importlib-resources-bootstrap
+  (hidden-package
+   (package/inherit
+    pypy3-importlib-resources
+    (name "pypy3-importlib-resources-bootstrap")
+    (propagated-inputs
+     `(("python-pathlib2-bootstrap" ,pypy3-pathlib2-bootstrap)
+       ("python-typing" ,pypy3-typing))))))
+
 (define-public python-importlib-metadata
   (package
     (name "python-importlib-metadata")
@@ -6055,7 +6135,7 @@ its top-level name.  This functionality intends to replace most uses of
 @code{pkg_resources} entry point API and metadata API.  Along with
 @code{importlib.resources} in Python 3.7 and newer, this can eliminate the
 need to use the older and less efficient @code{pkg_resources} package.")
-    (properties `((python2-variant . ,(delay python2-importlib-metadata))))
+    (properties `((python2-variant . ,(delay python2-importlib-metadata)) (pypy3-variant . ,(delay pypy3-importlib-metadata))))
     (license license:asl2.0)))
 
 (define-public python2-importlib-metadata
@@ -6075,6 +6155,18 @@ need to use the older and less efficient @code{pkg_resources} package.")
         ("python-pathlib2" ,python2-pathlib2)
         ,@(package-propagated-inputs base))))))
 
+(define-public pypy3-importlib-metadata
+	(let ((base (package-with-pypy3 (strip-pypy3-variant python-importlib-metadata))))
+	  (package/inherit
+		base
+		(name "pypy3-importlib-metadata")
+    (native-inputs
+     `(("python-setuptools-scm" ,pypy3-setuptools-scm)
+	   ;; break loop: pytest → pluggy → importlib-metadata
+       ("python-pyfakefs" ,pypy3-pyfakefs-bootstrap)
+       ("python-packaging" ,pypy3-packaging-bootstrap)
+	   ("python-importlib-resources" ,pypy3-importlib-resources))))))
+
 ;; This package is used by python2-pytest, and thus must not depend on it.
 (define-public python2-importlib-metadata-bootstrap
   (hidden-package
@@ -6091,6 +6183,20 @@ need to use the older and less efficient @code{pkg_resources} package.")
        ("python-contextlib2" ,python2-contextlib2-bootstrap)
        ("python-importlib-resources" ,python2-importlib-resources-bootstrap))))))
 
+(define-public pypy3-importlib-metadata-bootstrap
+  (let ((base pypy3-importlib-metadata))
+	(hidden-package
+	  (package
+		(inherit base)
+		(name "pypy3-importlib-metadata-bootstrap")
+		(arguments `(#:tests? #f ,@(package-arguments pypy3-importlib-metadata)))
+(native-inputs
+     `(("python-setuptools-scm" ,pypy3-setuptools-scm)
+       ("python-pyfakefs" ,pypy3-pyfakefs-bootstrap)
+       ("python-packaging" ,pypy3-packaging-bootstrap)
+	   ("python-importlib-resources" ,pypy3-importlib-resources-bootstrap)))
+		))))
+
 (define-public python-importmagic
   (package
     (name "python-importmagic")
@@ -9699,6 +9805,9 @@ for atomic file system operations.")
 (define-public python2-atomicwrites
   (package-with-python2 python-atomicwrites))
 
+(define-public pypy3-atomicwrites
+  (package-with-pypy3 python-atomicwrites))
+
 (define-public python-click-threading
   (package
     (name "python-click-threading")
@@ -10169,7 +10278,8 @@ library as well as on the command line.")
    (description "Pluggy is an extraction of the plugin manager as used by
 Pytest but stripped of Pytest specific details.")
    (home-page "https://pypi.org/project/pluggy/")
-   (properties `((python2-variant . ,(delay python2-pluggy))))
+   (properties `((python2-variant . ,(delay python2-pluggy))
+                 (pypy3-variant . ,(delay pypy3-pluggy))))
    (license license:expat)))
 
 (define-public python2-pluggy
@@ -10180,6 +10290,14 @@ Pytest but stripped of Pytest specific details.")
      (propagated-inputs
       `(("python-importlib-metadata" ,python2-importlib-metadata))))))
 
+(define-public pypy3-pluggy
+  (let ((base (package-with-pypy3 (strip-pypy3-variant
+                                     python-pluggy))))
+    (package/inherit
+     base
+     (propagated-inputs
+      `(("python-importlib-metadata" ,pypy3-importlib-metadata))))))
+
 ;; This package requires python2-importlib-metadata, but that package
 ;; ends up needing python2-pluggy via python2-pytest, so we need this
 ;; variant to solve the circular dependency.
@@ -10194,6 +10312,17 @@ Pytest but stripped of Pytest specific details.")
     (propagated-inputs
      `(("python-importlib-metadata" ,python2-importlib-metadata-bootstrap))))))
 
+(define-public pypy3-pluggy-bootstrap
+  (hidden-package
+   (package/inherit
+    pypy3-pluggy
+    (name "pypy3-pluggy-bootstrap")
+    (arguments
+     `(#:tests? #f
+       ,@(package-arguments pypy3-pluggy)))
+    (propagated-inputs
+     `(("python-importlib-metadata" ,pypy3-importlib-metadata-bootstrap))))))
+
 (define-public python-tox
   (package
    (name "python-tox")
@@ -10966,6 +11095,15 @@ ambiguities (forward vs. backward slashes, etc.).
       `(("python2-scandir" ,python2-scandir)
         ("python2-six" ,python2-six-bootstrap))))))
 
+(define-public pypy3-pathlib2-bootstrap
+  (hidden-package
+   (package
+     (inherit pypy3-pathlib2)
+     (name "pypy3-pathlib2-bootstrap")
+     (propagated-inputs
+      `(("python-scandir" ,pypy3-scandir)
+        ("python-six" ,pypy3-six-bootstrap))))))
+
 (define-public python-jellyfish
   (package
     (name "python-jellyfish")
@@ -11870,6 +12008,9 @@ specified in POSIX.1-2001 and POSIX.1-2008.")
 (define-public python2-wcwidth
   (package-with-python2 python-wcwidth))
 
+(define-public pypy3-wcwidth
+  (package-with-pypy3 python-wcwidth))
+
 (define-public python2-jsonrpclib
   (package
     (name "python2-jsonrpclib")
@@ -14026,6 +14167,9 @@ protocols.")
 (define-public python2-attrs-bootstrap
   (package-with-python2 python-attrs-bootstrap))
 
+(define-public pypy3-attrs-bootstrap
+  (package-with-pypy3 python-attrs-bootstrap))
+
 (define-public python2-cliapp
   (package
     (name "python2-cliapp")
@@ -14574,6 +14718,9 @@ This package is part of the Python standard library since version 3.5.")
 (define-public python2-scandir
   (package-with-python2 python-scandir))
 
+(define-public pypy3-scandir
+  (package-with-pypy3 python-scandir))
+
 (define-public python2-stemming
   (package
     (name "python2-stemming")
@@ -14735,6 +14882,9 @@ information.")
      `(#:tests? #f
        ,@(package-arguments python2-packaging))))))
 
+(define-public pypy3-packaging-bootstrap
+  (package-with-pypy3 python-packaging-bootstrap))
+
 (define-public python-relatorio
   (package
     (name "python-relatorio")
@@ -16525,6 +16675,9 @@ and other tools.")
 (define-public python2-typing
   (package-with-python2 python-typing))
 
+(define-public pypy3-typing
+  (package-with-pypy3 python-typing))
+
 (define-public python-typing-extensions
   (package
     (name "python-typing-extensions")
@@ -16766,6 +16919,9 @@ working with iterables.")
     (propagated-inputs
      `(("python2-six" ,python2-six-bootstrap)))))
 
+(define-public pypy3-more-itertools
+  (package-with-pypy3 python-more-itertools))
+
 (define-public python-latexcodec
   (package
     (name "python-latexcodec")
diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index e2b254bf6f..fe240eaf5e 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -426,6 +426,18 @@ data types.")
             (lambda _ (unsetenv "SOURCE_DATE_EPOCH") #t))
           (add-after 'check 'reset-SOURCE_DATE_EPOCH
             (lambda _ (setenv "SOURCE_DATE_EPOCH" "1") #t))
+          (add-after 'install 'install-symlinks
+             (lambda* (#:key outputs #:allow-other-keys)
+           (let ((bin (string-append (assoc-ref outputs "out") "/bin")))
+			 (for-each
+                  (lambda (old new)
+                    (let ((oldpath (string-append bin "/" old)))
+					  (when (file-exists? oldpath)
+							(symlink oldpath
+                             (string-append bin "/" new)))))
+                  `("python3" ,"pydoc3" ,"idle3" ,"pip3" ,"python3-config")
+                  `("python"  ,"pydoc"  ,"idle"  ,"pip" ,"python-config"))
+			 #t)))
            (replace 'rebuild-bytecode
              (lambda* (#:key outputs #:allow-other-keys)
                (let ((out (assoc-ref outputs "out")))
@@ -659,7 +671,9 @@ ease from the desktop to a microcontroller or embedded system.")
               (sha256
                (base32
                 "10zsk8jby8j6visk5mzikpb1cidvz27qq4pfpa26jv53klic6b0c"))
-              (patches (search-patches "pypy3-7.3.1-fix-tests.patch"))))
+              (patches (search-patches "pypy3-7.3.1-fix-tests.patch"
+                                       "pypy3-7.3.1-install-paths.patch"
+                                       "pypy3-7.3.1-ssl-paths.patch"))))
     (build-system gnu-build-system)
     (native-inputs
      `(("python-2" ,python-2)
@@ -667,13 +681,13 @@ ease from the desktop to a microcontroller or embedded system.")
        ("tar" ,tar)                     ; Required for package.py
        ("python2-pycparser" ,python2-pycparser)
        ("python2-hypothesis" ,python2-hypothesis)
-       ("nss-certs" ,nss-certs)         ; For ssl tests
        ("gzip" ,gzip)))
     (inputs
      `(("libffi" ,libffi)
        ("zlib" ,zlib)
        ("ncurses" ,ncurses)
        ("openssl" ,openssl)
+       ("nss-certs" ,nss-certs)         ; For ssl module
        ("expat" ,expat)
        ("bzip2" ,bzip2)
        ("sqlite" ,sqlite)
@@ -684,7 +698,7 @@ ease from the desktop to a microcontroller or embedded system.")
        ("bash-minimal" ,bash-minimal)   ; Used as /bin/sh
        ("xz" ,xz)))                     ; liblzma
     (arguments
-     `(#:tests? #f     ;FIXME: Disabled for now, there are many tests failing.
+     `(#:tests? #f ;FIXME: Disabled for now, there are many tests failing.
        #:modules ((ice-9 ftw) (ice-9 match)
                   (guix build utils) (guix build gnu-build-system))
        #:phases (modify-phases %standard-phases
@@ -713,6 +727,9 @@ ease from the desktop to a microcontroller or embedded system.")
                       (substitute* '("lib_pypy/_curses_build.py")
                         ;; Find curses
                         (("/usr/local") (assoc-ref inputs "ncurses")))
+                      (substitute* '("lib_pypy/_dbm.py")
+                        ;; Use gdbm compat library, so we don’t need to pull in bdb
+                        (("ctypes.util.find_library\\('db'\\)") (string-append "'" (assoc-ref inputs "gdbm") "/lib/libgdbm_compat.so'")))
                       (substitute* '("lib_pypy/_sqlite3_build.py")
                         ;; Always use search paths
                         (("sys\\.platform\\.startswith\\('freebsd'\\)") "True")
@@ -729,6 +746,14 @@ ease from the desktop to a microcontroller or embedded system.")
                       (substitute* '("lib-python/3/distutils/unixccompiler.py")
                         ;; gcc-toolchain does not provide symlink cc -> gcc
                         (("\"cc\"") "\"gcc\""))
+                      (substitute* '("lib_pypy/_cffi_ssl/_stdssl/__init__.py")
+                        ;; Add nss-certs to default certificate search path,
+                        ;; otherwise every packages has to specify nss-certs and
+                        ;; openssl as input to set the proper env variables.
+                        ;; Depends on -ssl-paths.patch.
+                        (("@GUIX_CERT_PATH@")
+                         (string-append (assoc-ref inputs "nss-certs")
+                                        "/etc/ssl/certs")))
                       #t))
                   (add-after
                       'unpack 'set-source-file-times-to-1980
@@ -748,7 +773,8 @@ ease from the desktop to a microcontroller or embedded system.")
                                 (string-append "--make-jobs="
                                                (number->string (parallel-job-count)))
                                 "-Ojit"
-                                "targetpypystandalone"))
+                                "targetpypystandalone"
+                                "--allworkingmodules"))
                       ;; Build c modules and package everything, so tests work.
                       (with-directory-excursion "pypy/tool/release"
                         (unsetenv "PYTHONPATH") ; Do not use the system’s python libs:
@@ -756,7 +782,9 @@ ease from the desktop to a microcontroller or embedded system.")
                                         ; attribute 'IntFlag'
                         (invoke "python2" "package.py"
                                 "--archive-name" "pypy-dist"
-                                "--builddir" (getcwd)))))
+                                "--builddir" (getcwd))
+                        ;; install pip and setuptools into the dist directory
+                        (invoke "pypy-dist/bin/pypy3" "-m" "ensurepip"))))
                   (replace 'check
                     (lambda* (#:key tests? #:allow-other-keys)
                       (if tests?
@@ -774,32 +802,45 @@ ease from the desktop to a microcontroller or embedded system.")
                       #t))
                   (replace 'install
                     (lambda* (#:key inputs outputs #:allow-other-keys)
-                      (with-directory-excursion "pypy/tool/release"
-                        ;; Delete test data.
-                        (for-each
-                         (lambda (x)
-                           (delete-file-recursively (string-append
-                                                     "pypy-dist/lib-python/3/" x)))
-                         '("tkinter/test"
-                           "test"
-                           "sqlite3/test"
-                           "lib2to3/tests"
-                           "idlelib/idle_test"
-                           "distutils/tests"
-                           "ctypes/test"
-                           "unittest/test"))
-                        ;; Patch shebang referencing python2
-                        (substitute* '("pypy-dist/lib-python/3/cgi.py"
-                                       "pypy-dist/lib-python/3/encodings/rot_13.py")
-                          (("#!.+/bin/python")
-                           (string-append "#!" (assoc-ref outputs "out") "/bin/pypy3")))
-                        (with-fluids ((%default-port-encoding "ISO-8859-1"))
-                          (substitute* '("pypy-dist/lib_pypy/_md5.py"
-                                         "pypy-dist/lib_pypy/_sha1.py")
-                            (("#!.+/bin/python")
-                             (string-append "#!" (assoc-ref outputs "out") "/bin/pypy3"))))
-                        (copy-recursively "pypy-dist" (assoc-ref outputs "out")))
-                      #t)))))
+                      (let* ((out (assoc-ref outputs "out"))
+                             (bin-pypy3 (string-append out "/bin/pypy3"))
+                             (shebang-match-python "#!.+/bin/python")
+                             (shebang-pypy3 (string-append "#!" bin-pypy3))
+                             (dist-dir "pypy/tool/release/pypy-dist"))
+                        (with-directory-excursion dist-dir
+                          ;; Delete test data.
+                          (for-each
+                           (lambda (x)
+                             (delete-file-recursively (string-append
+                                                       "lib-python/3/" x)))
+                           '("tkinter/test"
+                             "test"
+                             "sqlite3/test"
+                             "lib2to3/tests"
+                             "idlelib/idle_test"
+                             "distutils/tests"
+                             "ctypes/test"
+                             "unittest/test"
+                             "ensurepip" ; we already used to to install pip/setuptools
+                             ))
+                          ;; Patch shebang referencing python2
+                          (substitute* '("lib-python/3/cgi.py"
+                                         "lib-python/3/encodings/rot_13.py")
+                            ((shebang-match-python) shebang-pypy3))
+                          (with-fluids ((%default-port-encoding "ISO-8859-1"))
+                                       (substitute* '("lib_pypy/_md5.py"
+                                                      "lib_pypy/_sha1.py")
+                                         ((shebang-match-python) shebang-pypy3))))
+                        (copy-recursively dist-dir out)
+                        ;; Make sure pypy3 is callable as python/python3, so we
+                        ;; don’t have to patch every single package.
+                        (symlink bin-pypy3 (string-append out "/bin/python"))
+                        (symlink bin-pypy3 (string-append out "/bin/python3"))
+                        #t))))))
+    (native-search-paths
+     (list (search-path-specification
+            (variable "PYTHONPATH")
+            (files '("lib/pypy3.6/site-packages")))))
     (home-page "https://www.pypy.org/")
     (synopsis "Python implementation with just-in-time compilation")
     (description "PyPy is a faster, alternative implementation of the Python
diff --git a/gnu/packages/xml.scm b/gnu/packages/xml.scm
index 90181a1773..d2dad42a1c 100644
--- a/gnu/packages/xml.scm
+++ b/gnu/packages/xml.scm
@@ -2083,8 +2083,10 @@ because lxml.etree already has it's own implementation of XPath 1.0.")
     (arguments
      `(#:phases (modify-phases %standard-phases
                   (replace 'check
-                    (lambda _
-                      (invoke "make" "test"))))))
+                    (lambda* (#:key tests? #:allow-other-keys)
+                             (if tests?
+                               (begin
+                      (invoke "make" "test"))))))))
     (inputs
      `(("libxml2" ,libxml2)
        ("libxslt" ,libxslt)))
@@ -2093,11 +2095,20 @@ because lxml.etree already has it's own implementation of XPath 1.0.")
     (description
      "The lxml XML toolkit is a Pythonic binding for the C libraries
 libxml2 and libxslt.")
-    (license license:bsd-3))) ; and a few more, see LICENSES.txt
+    (license license:bsd-3) ; and a few more, see LICENSES.txt
+    (properties `((pypy3-variant . ,(delay pypy3-lxml))))))
 
 (define-public python2-lxml
   (package-with-python2 python-lxml))
 
+(define-public pypy3-lxml
+               (let ((pkg (package-with-pypy3 (strip-pypy3-variant python-lxml))))
+  (package (inherit pkg)
+		   (arguments `(
+                        ;; some tests hang for unknown reasons
+                        #:tests? #f ,@(package-arguments pkg)
+                             )))))
+
 (define-public python-xmlschema
   (package
     (name "python-xmlschema")
@@ -2140,6 +2151,9 @@ finding schema's elements and attributes; and can encode and decode
 XML data to JSON and other formats.")
     (license license:expat)))
 
+(define-public pypy3-xmlschema
+               (package-with-pypy3 python-xmlschema))
+
 (define-public python-xmltodict
   (package
     (name "python-xmltodict")
diff --git a/guix/build-system/python.scm b/guix/build-system/python.scm
index e39c06528e..59395ef19f 100644
--- a/guix/build-system/python.scm
+++ b/guix/build-system/python.scm
@@ -32,7 +32,9 @@
   #:use-module (srfi srfi-26)
   #:export (%python-build-system-modules
             package-with-python2
+            package-with-pypy3
             strip-python2-variant
+            strip-pypy3-variant
             default-python
             default-python2
             python-build
@@ -63,13 +65,19 @@ extension, such as '.tar.gz'."
   "Return the default Python package."
   ;; Lazily resolve the binding to avoid a circular dependency.
   (let ((python (resolve-interface '(gnu packages python))))
-    (module-ref python 'python-wrapper)))
+	;; Must *not* be python-wrapper, which lacks site-libraries directory
+    (module-ref python 'python)))
 
 (define (default-python2)
   "Return the default Python 2 package."
   (let ((python (resolve-interface '(gnu packages python))))
     (module-ref python 'python-2)))
 
+(define (default-pypy3)
+  "Return the default pypy3 package."
+  (let ((python (resolve-interface '(gnu packages python))))
+    (module-ref python 'pypy3)))
+
 (define* (package-with-explicit-python python old-prefix new-prefix
                                        #:key variant-property)
   "Return a procedure of one argument, P.  The procedure creates a package with
@@ -129,12 +137,26 @@ pre-defined variants."
                                 "python-" "python2-"
                                 #:variant-property 'python2-variant))
 
+(define package-with-pypy3
+  ;; Note: delay call to 'default-pypy3' until after the 'arguments' field
+  ;; of packages is accessed to avoid a circular dependency when evaluating
+  ;; the top-level of (gnu packages python).
+  (package-with-explicit-python (delay (default-pypy3))
+                                "python-" "pypy3-"
+                                #:variant-property 'pypy3-variant))
+
 (define (strip-python2-variant p)
   "Remove the 'python2-variant' property from P."
   (package
     (inherit p)
     (properties (alist-delete 'python2-variant (package-properties p)))))
 
+(define (strip-pypy3-variant p)
+  "Remove the 'pypy3-variant' property from P."
+  (package
+    (inherit p)
+    (properties (alist-delete 'pypy3-variant (package-properties p)))))
+
 (define* (lower name
                 #:key source inputs native-inputs outputs system target
                 (python (default-python))
diff --git a/guix/build/python-build-system.scm b/guix/build/python-build-system.scm
index 09bd8465c8..34e33fd4c1 100644
--- a/guix/build/python-build-system.scm
+++ b/guix/build/python-build-system.scm
@@ -28,6 +28,7 @@
   #:use-module (ice-9 match)
   #:use-module (ice-9 ftw)
   #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-11)
   #:use-module (srfi srfi-26)
   #:export (%standard-phases
             add-installed-pythonpath
@@ -158,9 +159,12 @@
   "Return the path of the current output's Python site-package."
   (let* ((out (assoc-ref outputs "out"))
          (python (assoc-ref inputs "python")))
-    (string-append out "/lib/python"
-                   (python-version python)
-                   "/site-packages/")))
+    (string-append
+      out "/"
+    (with-directory-excursion python
+      (first (filter
+               file-exists?
+               (map (lambda (x) (string-append "lib/" x "/site-packages")) (scandir "lib" (lambda (x) (not (member x '("." ".."))))))))))))
 
 (define (add-installed-pythonpath inputs outputs)
   "Prepend the Python site-package of OUTPUT to PYTHONPATH.  This is useful
@@ -202,9 +206,7 @@ when running checks after installing the package."
   (let* ((out  (assoc-ref outputs "out"))
          (python (assoc-ref inputs "python"))
          (var `("PYTHONPATH" prefix
-                ,(cons (string-append out "/lib/python"
-                                      (python-version python)
-                                      "/site-packages")
+                ,(cons (site-packages inputs outputs)
                        (search-path-as-string->list
                         (or (getenv "PYTHONPATH") ""))))))
     (for-each (lambda (dir)
@@ -222,9 +224,7 @@ installed with setuptools."
   ;; some good reason.
   (let* ((out (assoc-ref outputs "out"))
          (python (assoc-ref inputs "python"))
-         (site-packages (string-append out "/lib/python"
-                                       (python-version python)
-                                       "/site-packages"))
+         (site-packages (site-packages inputs outputs))
          (easy-install-pth (string-append site-packages "/easy-install.pth"))
          (new-pth (string-append site-packages "/" name ".pth")))
     (when (file-exists? easy-install-pth)
-- 
2.26.2


^ permalink raw reply related	[flat|nested] 10+ messages in thread

* Re: [Python] pypy3 integration
  2020-07-19  8:27 [Python] pypy3 integration Lars-Dominik Braun
@ 2020-07-20 11:17 ` zimoun
  2020-07-20 19:16   ` Lars-Dominik Braun
  2020-07-20 19:42 ` Jakub Kądziołka
  1 sibling, 1 reply; 10+ messages in thread
From: zimoun @ 2020-07-20 11:17 UTC (permalink / raw)
  To: Lars-Dominik Braun; +Cc: Guix Devel

Dear,

Thank you for looking at this.

Does it make sense to have something like "package-with-pypy" using
"package-with-explicit-python"?  Because it would be easy to rebuild
all the pure Python packages with PyPy instead of CPython.

All the best,
simon


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [Python] pypy3 integration
  2020-07-20 11:17 ` zimoun
@ 2020-07-20 19:16   ` Lars-Dominik Braun
  0 siblings, 0 replies; 10+ messages in thread
From: Lars-Dominik Braun @ 2020-07-20 19:16 UTC (permalink / raw)
  To: zimoun; +Cc: Guix Devel

Hi simon,

> Does it make sense to have something like "package-with-pypy" using
> "package-with-explicit-python"?  Because it would be easy to rebuild
> all the pure Python packages with PyPy instead of CPython.
that’s what I am proposing with my patch, however due to different
search paths (/lib/pypy3.6/site-packages vs.
/lib/python3.8/site-packages) the python build system needs some changes
too. And that’s where my problems begin.

Lars



^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [Python] pypy3 integration
  2020-07-19  8:27 [Python] pypy3 integration Lars-Dominik Braun
  2020-07-20 11:17 ` zimoun
@ 2020-07-20 19:42 ` Jakub Kądziołka
  2020-07-22  6:46   ` Lars-Dominik Braun
  2020-07-22 10:34   ` zimoun
  1 sibling, 2 replies; 10+ messages in thread
From: Jakub Kądziołka @ 2020-07-20 19:42 UTC (permalink / raw)
  To: Lars-Dominik Braun; +Cc: guix-devel

[-- Attachment #1: Type: text/plain, Size: 1147 bytes --]

On Sun, Jul 19, 2020 at 10:27:14AM +0200, Lars-Dominik Braun wrote:
> Hi,
> 
> I’ve been looking into integrating pypy3 (a JIT Python 3 implementation)
> tighter into guix. Right now we have a package, but it’s not that
> useful, because there are no pypy3-* libraries.

pypy3 works somewhat well for me already in this regard:

~$ genv --pure --ad-hoc pypy3 python-sympy python -- pypy3
Python 3.6.9 (?, Jan 01 1970, 00:00:00)
[PyPy 7.3.1 with GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>> import sympy
>>>> sympy.isprime(694201337)
True

The `python' package needs to be included in the environment so that
PYTHONPATH gets set, but apart from that, the normal python-prefixed
packages seem to work with pypy. At least, for a sample size of 1 ;)

I suppose it could be worth it to precompile the packages for use in
pypy too, though. I'm not sure what's the best way of solving this. Just
creating separate pypy3-* packages has the big drawback that if you use
python-* instead, you silently get worse loading performance.

Thoughts?

Regards,
Jakub Kądziołka

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [Python] pypy3 integration
  2020-07-20 19:42 ` Jakub Kądziołka
@ 2020-07-22  6:46   ` Lars-Dominik Braun
  2020-07-22 10:39     ` zimoun
  2020-07-27 10:48     ` Ludovic Courtès
  2020-07-22 10:34   ` zimoun
  1 sibling, 2 replies; 10+ messages in thread
From: Lars-Dominik Braun @ 2020-07-22  6:46 UTC (permalink / raw)
  To: Jakub Kądziołka; +Cc: guix-devel

Hi,

> pypy3 works somewhat well for me already in this regard:
indeed, you’re right.

This will probably break for some packages, because python provides
Python 3.8 whereas pypy3 provides Python 3.6.  (They’ve always lagged
behind and given that we’re going towards 3.10, well…) One example are
packages depending on importlib.resources, which only became available
with Python 3.7. Unfortunately this includes the widely-used pytest (or
rather: its dependency python-pluggy).

Also Python’s C ABI is not stable[1] and thus extensions compiled for 3.8
can fail in unpredictable ways with 3.6. And looking at python-numpy,
it seems they won’t even load.

So, does this justify creating pypy3-* packages?

Cheers,
Lars

[1] https://docs.python.org/3/c-api/stable.html



^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [Python] pypy3 integration
  2020-07-20 19:42 ` Jakub Kądziołka
  2020-07-22  6:46   ` Lars-Dominik Braun
@ 2020-07-22 10:34   ` zimoun
  1 sibling, 0 replies; 10+ messages in thread
From: zimoun @ 2020-07-22 10:34 UTC (permalink / raw)
  To: Jakub Kądziołka, Lars-Dominik Braun; +Cc: guix-devel

Dear,

On Mon, 20 Jul 2020 at 21:42, Jakub Kądziołka <kuba@kadziolka.net> wrote:

> ~$ genv --pure --ad-hoc pypy3 python-sympy python -- pypy3
> Python 3.6.9 (?, Jan 01 1970, 00:00:00)
> [PyPy 7.3.1 with GCC 7.5.0] on linux
> Type "help", "copyright", "credits" or "license" for more information.
>>>>> import sympy
>>>>> sympy.isprime(694201337)
> True
>

Does it work in all situations?
Other said, the package python-sympy is “compiled” with CPython and then
used with pypy.  Moreover, CPython and PyPy are not the same VM, so not
the same optimizations and therefore I am not sure that the performances
of your trick vs an hypothetical pypy-sympy would be the same.


All the best,
simon


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [Python] pypy3 integration
  2020-07-22  6:46   ` Lars-Dominik Braun
@ 2020-07-22 10:39     ` zimoun
  2020-07-27 10:48     ` Ludovic Courtès
  1 sibling, 0 replies; 10+ messages in thread
From: zimoun @ 2020-07-22 10:39 UTC (permalink / raw)
  To: Lars-Dominik Braun, Jakub Kądziołka; +Cc: guix-devel

Dear Lars,

On Wed, 22 Jul 2020 at 08:46, Lars-Dominik Braun <lars@6xq.net> wrote:

>> pypy3 works somewhat well for me already in this regard:
> indeed, you’re right.

Well, python-package and pypy-package would not be compiled with the
same VM.  So the performances (optimizations) are not necessary the
sames.

For example, it is the same situation with emacs-package “compiled” with
emacs and run with emacs-next.  Even if the bytecode is stable, there is
no guarantee.


> This will probably break for some packages, because python provides
> Python 3.8 whereas pypy3 provides Python 3.6.  (They’ve always lagged
> behind and given that we’re going towards 3.10, well…) One example are
> packages depending on importlib.resources, which only became available
> with Python 3.7. Unfortunately this includes the widely-used pytest (or
> rather: its dependency python-pluggy).
>
> Also Python’s C ABI is not stable[1] and thus extensions compiled for 3.8
> can fail in unpredictable ways with 3.6. And looking at python-numpy,
> it seems they won’t even load.
>
> So, does this justify creating pypy3-* packages?

From my point of view, yes.


All the best,
simon


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [Python] pypy3 integration
  2020-07-22  6:46   ` Lars-Dominik Braun
  2020-07-22 10:39     ` zimoun
@ 2020-07-27 10:48     ` Ludovic Courtès
  2020-07-27 19:15       ` zimoun
  1 sibling, 1 reply; 10+ messages in thread
From: Ludovic Courtès @ 2020-07-27 10:48 UTC (permalink / raw)
  To: Lars-Dominik Braun; +Cc: Jakub Kądziołka, guix-devel

Hello,

Lars-Dominik Braun <lars@6xq.net> skribis:

>> pypy3 works somewhat well for me already in this regard:
> indeed, you’re right.
>
> This will probably break for some packages, because python provides
> Python 3.8 whereas pypy3 provides Python 3.6.  (They’ve always lagged
> behind and given that we’re going towards 3.10, well…) One example are
> packages depending on importlib.resources, which only became available
> with Python 3.7. Unfortunately this includes the widely-used pytest (or
> rather: its dependency python-pluggy).
>
> Also Python’s C ABI is not stable[1] and thus extensions compiled for 3.8
> can fail in unpredictable ways with 3.6. And looking at python-numpy,
> it seems they won’t even load.

Also, what about .pyc files?  Does pypy create compatible .pyc files?

> So, does this justify creating pypy3-* packages?

It probably does.  But do we want to mirror all the ‘python-’ packages,
or just some of them?  It seems overkill to mirror all of them.

Perhaps we could have a package transformation option to turn a
‘python-build-system’ package into a pypy package?

Thanks,
Ludo’, who knows little about Python.


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [Python] pypy3 integration
  2020-07-27 10:48     ` Ludovic Courtès
@ 2020-07-27 19:15       ` zimoun
  2020-08-01  8:23         ` Lars-Dominik Braun
  0 siblings, 1 reply; 10+ messages in thread
From: zimoun @ 2020-07-27 19:15 UTC (permalink / raw)
  To: Ludovic Courtès
  Cc: Jakub Kądziołka, Guix Devel, Lars-Dominik Braun

Hi,

On Mon, 27 Jul 2020 at 12:48, Ludovic Courtès <ludo@gnu.org> wrote:
> Lars-Dominik Braun <lars@6xq.net> skribis:
>
> >> pypy3 works somewhat well for me already in this regard:
> > indeed, you’re right.
> >
> > This will probably break for some packages, because python provides
> > Python 3.8 whereas pypy3 provides Python 3.6.  (They’ve always lagged
> > behind and given that we’re going towards 3.10, well…) One example are
> > packages depending on importlib.resources, which only became available
> > with Python 3.7. Unfortunately this includes the widely-used pytest (or
> > rather: its dependency python-pluggy).
> >
> > Also Python’s C ABI is not stable[1] and thus extensions compiled for 3.8
> > can fail in unpredictable ways with 3.6. And looking at python-numpy,
> > it seems they won’t even load.
>
> Also, what about .pyc files?  Does pypy create compatible .pyc files?

What do you mean by "compatible .pyc files"?

Well, the .pyc generated by CPython should be compatible with the ones
generated by Pypy, both VM targeting say Python 3.6.
But there is no necessary compatibility between .pyc of Python 3.6 and
Python 3.8, at least for CPython as Lars wrote.

CPython being the reference implementation, Pypy is always late.


> > So, does this justify creating pypy3-* packages?
>
> It probably does.  But do we want to mirror all the ‘python-’ packages,
> or just some of them?  It seems overkill to mirror all of them.

With the current situation, somehow you have to.  But...

> Perhaps we could have a package transformation option to turn a
> ‘python-build-system’ package into a pypy package?

...Yes it could be nice to be able to change the "package builder" of
the build-system.  (Obviously, without any guarantee that the build
would be correct for all combinatorial :-))
The issue is the same for emacs vs emacs-next, GCC versions (without
saying gcc vs clang ;-)), OCaml 4.07 vs OCaml 4.09 etc..
We already discussed this kind of issue when discussing "package parameters".


All the best,
simon


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [Python] pypy3 integration
  2020-07-27 19:15       ` zimoun
@ 2020-08-01  8:23         ` Lars-Dominik Braun
  0 siblings, 0 replies; 10+ messages in thread
From: Lars-Dominik Braun @ 2020-08-01  8:23 UTC (permalink / raw)
  To: zimoun; +Cc: Jakub Kądziołka, Guix Devel

Hi,

> > Also, what about .pyc files?  Does pypy create compatible .pyc files?
> Well, the .pyc generated by CPython should be compatible with the ones
> generated by Pypy, both VM targeting say Python 3.6.
> But there is no necessary compatibility between .pyc of Python 3.6 and
> Python 3.8, at least for CPython as Lars wrote.
both C extensions and .pyc files are guarded by encoding implementation
and Python version in the file name. If we look at python-numpy for
example, we can see files like decorators.cpython-38.pyc and
common.cpython-38-x86_64-linux-gnu.so. If I build numpy with pypy the
identifier (cpython-38) changes to pypy36 and pypy36-pp73 respectively.
Thus pypy won’t load CPython’s files.

> > Perhaps we could have a package transformation option to turn a
> > ‘python-build-system’ package into a pypy package?
> ...Yes it could be nice to be able to change the "package builder" of
> the build-system.  (Obviously, without any guarantee that the build
> would be correct for all combinatorial :-))
> The issue is the same for emacs vs emacs-next, GCC versions (without
> saying gcc vs clang ;-)), OCaml 4.07 vs OCaml 4.09 etc..
> We already discussed this kind of issue when discussing "package parameters".
I think that would be really helpful. What’s the status of this work? Is
there a PoC I can extend to cover CPython vs. PyPy?

Cheers,
Lars



^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2020-08-01 21:25 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-07-19  8:27 [Python] pypy3 integration Lars-Dominik Braun
2020-07-20 11:17 ` zimoun
2020-07-20 19:16   ` Lars-Dominik Braun
2020-07-20 19:42 ` Jakub Kądziołka
2020-07-22  6:46   ` Lars-Dominik Braun
2020-07-22 10:39     ` zimoun
2020-07-27 10:48     ` Ludovic Courtès
2020-07-27 19:15       ` zimoun
2020-08-01  8:23         ` Lars-Dominik Braun
2020-07-22 10:34   ` zimoun

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/guix.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).