* [bug#44656] [PATCH] Upgrade pypy3
@ 2020-11-15 12:39 Lars-Dominik Braun
2020-11-15 19:33 ` Christopher Baines
2021-08-03 21:39 ` Maxim Cournoyer
0 siblings, 2 replies; 14+ messages in thread
From: Lars-Dominik Braun @ 2020-11-15 12:39 UTC (permalink / raw)
To: 44656
[-- Attachment #1: Type: text/plain, Size: 122 bytes --]
Hi,
the first patch in this series upgrades pypy3 to 7.3.2 and the second
adjusts several aspects of the package.
Lars
[-- Attachment #2: 0001-gnu-pypy3-Update-to-7.3.2.patch --]
[-- Type: text/x-diff, Size: 1307 bytes --]
From 0669431bc4ee103cf159f5f1d64684e6db8d6bb5 Mon Sep 17 00:00:00 2001
From: Lars-Dominik Braun <lars@6xq.net>
Date: Sat, 17 Oct 2020 13:46:21 +0200
Subject: [PATCH 1/2] gnu: pypy3: Update to 7.3.2.
* gnu/packages/python.scm (pypy3): Update to 7.3.2.
---
gnu/packages/python.scm | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index 43704bccae..8ef8ae2e1d 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -688,14 +688,14 @@ ease from the desktop to a microcontroller or embedded system.")
(define-public pypy3
(package
(name "pypy3")
- (version "7.3.1")
+ (version "7.3.2")
(source (origin
(method url-fetch)
- (uri (string-append "https://bitbucket.org/pypy/pypy/downloads/" ;
+ (uri (string-append "https://downloads.python.org/pypy/"
"pypy3.6-v" version "-src.tar.bz2"))
(sha256
(base32
- "10zsk8jby8j6visk5mzikpb1cidvz27qq4pfpa26jv53klic6b0c"))
+ "03f1fdw6yk2mypa9pbmgk26r8y1hhmw801l6g36zry9zsvz7aqgx"))
(patches (search-patches "pypy3-7.3.1-fix-tests.patch"))))
(build-system gnu-build-system)
(native-inputs
--
2.26.2
[-- Attachment #3: 0002-gnu-pypy3-Various-package-fixes.patch --]
[-- Type: text/x-diff, Size: 13949 bytes --]
From caf5e4bf9acd10f91de2e6a9c60185475144a1b9 Mon Sep 17 00:00:00 2001
From: Lars-Dominik Braun <lars@6xq.net>
Date: Sun, 15 Nov 2020 10:54:26 +0100
Subject: [PATCH 2/2] gnu: pypy3: Various package fixes.
* gnu/packages/python.scm (pypy3) [patches]: Add new patch.
[inputs]: Remove bash-minimal and add nss-certs.
[native-inputs]: Remove nss-certs here.
[arguments]: Use gdbm compat library, add 2to3 binary.
[native-search-paths]: Add search path.
* gnu/packages/patches/pypy3-7.3.1-ssl-paths.patch: New file.
* gnu/local.mk: Add it.
---
gnu/local.mk | 1 +
.../patches/pypy3-7.3.1-ssl-paths.patch | 41 +++++++
gnu/packages/python.scm | 109 ++++++++++++------
3 files changed, 116 insertions(+), 35 deletions(-)
create mode 100644 gnu/packages/patches/pypy3-7.3.1-ssl-paths.patch
diff --git a/gnu/local.mk b/gnu/local.mk
index 91a3295e75..b644391a11 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -1536,6 +1536,7 @@ dist_patch_DATA = \
%D%/packages/patches/python-unittest2-remove-argparse.patch \
%D%/packages/patches/python-waitress-fix-tests.patch \
%D%/packages/patches/pypy3-7.3.1-fix-tests.patch \
+ %D%/packages/patches/pypy3-7.3.1-ssl-paths.patch \
%D%/packages/patches/qemu-build-info-manual.patch \
%D%/packages/patches/qemu-glibc-2.27.patch \
%D%/packages/patches/qrcodegen-cpp-make-install.patch \
diff --git a/gnu/packages/patches/pypy3-7.3.1-ssl-paths.patch b/gnu/packages/patches/pypy3-7.3.1-ssl-paths.patch
new file mode 100644
index 0000000000..d21133b4ae
--- /dev/null
+++ b/gnu/packages/patches/pypy3-7.3.1-ssl-paths.patch
@@ -0,0 +1,41 @@
+Fix default certificate search path, still allowing the user to override it
+with environment variables.
+
+--- a/lib_pypy/_cffi_ssl/_stdssl/__init__.py
++++ b/lib_pypy/_cffi_ssl/_stdssl/__init__.py
+@@ -1679,20 +1679,9 @@ def get_default_verify_paths():
+ https://golang.org/src/crypto/x509/root_linux.go (for the files)
+ '''
+ certFiles = [
+- "/etc/ssl/certs/ca-certificates.crt", # Debian/Ubuntu/Gentoo etc.
+- "/etc/pki/tls/certs/ca-bundle.crt", # Fedora/RHEL 6
+- "/etc/ssl/ca-bundle.pem", # OpenSUSE
+- "/etc/pki/tls/cacert.pem", # OpenELEC
+- "/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem", # CentOS/RHEL 7
+- "/etc/ssl/cert.pem", # Alpine Linux
+ ]
+ certDirectories = [
+- "/etc/ssl/certs", # SLES10/SLES11
+- "/system/etc/security/cacerts", # Android
+- "/usr/local/share/certs", # FreeBSD
+- "/etc/pki/tls/certs", # Fedora/RHEL
+- "/etc/openssl/certs", # NetBSD
+- "/var/ssl/certs", # AIX
++ "@GUIX_CERT_PATH@",
+ ]
+
+ # optimization: reuse the values from a local varaible
+@@ -1707,9 +1696,10 @@ def get_default_verify_paths():
+ ofile = _cstr_decode_fs(lib.X509_get_default_cert_file())
+ odir = _cstr_decode_fs(lib.X509_get_default_cert_dir())
+
+- if os.path.exists(ofile) and os.path.exists(odir):
+- get_default_verify_paths.retval = (ofile_env, ofile, odir_env, odir)
+- return get_default_verify_paths.retval
++ if not os.path.exists(ofile):
++ ofile = None
++ if not os.path.exists(odir):
++ odir = None
+
+ # OpenSSL didn't supply the goods. Try some other options
+ for f in certFiles:
diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index 8ef8ae2e1d..c0bd3335e3 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -696,7 +696,8 @@ ease from the desktop to a microcontroller or embedded system.")
(sha256
(base32
"03f1fdw6yk2mypa9pbmgk26r8y1hhmw801l6g36zry9zsvz7aqgx"))
- (patches (search-patches "pypy3-7.3.1-fix-tests.patch"))))
+ (patches (search-patches "pypy3-7.3.1-fix-tests.patch"
+ "pypy3-7.3.1-ssl-paths.patch"))))
(build-system gnu-build-system)
(native-inputs
`(("python-2" ,python-2)
@@ -704,13 +705,13 @@ ease from the desktop to a microcontroller or embedded system.")
("tar" ,tar) ; Required for package.py
("python2-pycparser" ,python2-pycparser)
("python2-hypothesis" ,python2-hypothesis)
- ("nss-certs" ,nss-certs) ; For ssl tests
("gzip" ,gzip)))
(inputs
`(("libffi" ,libffi)
("zlib" ,zlib)
("ncurses" ,ncurses)
("openssl" ,openssl)
+ ("nss-certs" ,nss-certs) ; For ssl module
("expat" ,expat)
("bzip2" ,bzip2)
("sqlite" ,sqlite)
@@ -718,10 +719,9 @@ ease from the desktop to a microcontroller or embedded system.")
("tcl" ,tcl)
("tk" ,tk)
("glibc" ,glibc)
- ("bash-minimal" ,bash-minimal) ; Used as /bin/sh
("xz" ,xz))) ; liblzma
(arguments
- `(#:tests? #f ;FIXME: Disabled for now, there are many tests failing.
+ `(#:tests? #f ;FIXME: Disabled for now, there are many tests failing.
#:modules ((ice-9 ftw) (ice-9 match)
(guix build utils) (guix build gnu-build-system))
#:phases (modify-phases %standard-phases
@@ -750,6 +750,11 @@ ease from the desktop to a microcontroller or embedded system.")
(substitute* '("lib_pypy/_curses_build.py")
;; Find curses
(("/usr/local") (assoc-ref inputs "ncurses")))
+ (substitute* '("lib_pypy/_dbm.py")
+ ;; Use gdbm compat library, so we don’t need to pull in bdb
+ (("ctypes.util.find_library\\('db'\\)")
+ (string-append "'" (assoc-ref inputs "gdbm")
+ "/lib/libgdbm_compat.so'")))
(substitute* '("lib_pypy/_sqlite3_build.py")
;; Always use search paths
(("sys\\.platform\\.startswith\\('freebsd'\\)") "True")
@@ -761,11 +766,18 @@ ease from the desktop to a microcontroller or embedded system.")
"/lib/libsqlite3.so.0'")))
(substitute* '("lib-python/3/subprocess.py")
;; Fix shell path
- (("/bin/sh")
- (string-append (assoc-ref inputs "bash-minimal") "/bin/sh")))
+ (("/bin/sh") (which "sh")))
(substitute* '("lib-python/3/distutils/unixccompiler.py")
;; gcc-toolchain does not provide symlink cc -> gcc
(("\"cc\"") "\"gcc\""))
+ (substitute* '("lib_pypy/_cffi_ssl/_stdssl/__init__.py")
+ ;; Add nss-certs to default certificate search path,
+ ;; otherwise every packages has to specify nss-certs and
+ ;; openssl as input to set the proper env variables.
+ ;; Depends on -ssl-paths.patch.
+ (("@GUIX_CERT_PATH@")
+ (string-append (assoc-ref inputs "nss-certs")
+ "/etc/ssl/certs")))
#t))
(add-after
'unpack 'set-source-file-times-to-1980
@@ -785,7 +797,8 @@ ease from the desktop to a microcontroller or embedded system.")
(string-append "--make-jobs="
(number->string (parallel-job-count)))
"-Ojit"
- "targetpypystandalone"))
+ "targetpypystandalone"
+ "--allworkingmodules"))
;; Build c modules and package everything, so tests work.
(with-directory-excursion "pypy/tool/release"
(unsetenv "PYTHONPATH") ; Do not use the system’s python libs:
@@ -793,7 +806,12 @@ ease from the desktop to a microcontroller or embedded system.")
; attribute 'IntFlag'
(invoke "python2" "package.py"
"--archive-name" "pypy-dist"
- "--builddir" (getcwd)))))
+ "--builddir" (getcwd))
+ ;; install pip and setuptools into the dist directory.
+ ;; XXX: Breaks virtualenv, because it does not set +w
+ ;; on files copied from the store.
+ ;(invoke "pypy-dist/bin/pypy3" "-m" "ensurepip")
+ )))
(replace 'check
(lambda* (#:key tests? #:allow-other-keys)
(if tests?
@@ -811,32 +829,54 @@ ease from the desktop to a microcontroller or embedded system.")
#t))
(replace 'install
(lambda* (#:key inputs outputs #:allow-other-keys)
- (with-directory-excursion "pypy/tool/release"
- ;; Delete test data.
- (for-each
- (lambda (x)
- (delete-file-recursively (string-append
- "pypy-dist/lib-python/3/" x)))
- '("tkinter/test"
- "test"
- "sqlite3/test"
- "lib2to3/tests"
- "idlelib/idle_test"
- "distutils/tests"
- "ctypes/test"
- "unittest/test"))
- ;; Patch shebang referencing python2
- (substitute* '("pypy-dist/lib-python/3/cgi.py"
- "pypy-dist/lib-python/3/encodings/rot_13.py")
- (("#!.+/bin/python")
- (string-append "#!" (assoc-ref outputs "out") "/bin/pypy3")))
- (with-fluids ((%default-port-encoding "ISO-8859-1"))
- (substitute* '("pypy-dist/lib_pypy/_md5.py"
- "pypy-dist/lib_pypy/_sha1.py")
- (("#!.+/bin/python")
- (string-append "#!" (assoc-ref outputs "out") "/bin/pypy3"))))
- (copy-recursively "pypy-dist" (assoc-ref outputs "out")))
- #t)))))
+ (let* ((out (assoc-ref outputs "out"))
+ (bin-pypy3 (string-append out "/bin/pypy3"))
+ (shebang-match-python "#!.+/bin/python")
+ (shebang-pypy3 (string-append "#!" bin-pypy3))
+ (dist-dir "pypy/tool/release/pypy-dist"))
+ (with-directory-excursion dist-dir
+ ;; Delete test data.
+ (for-each
+ (lambda (x)
+ (delete-file-recursively (string-append
+ "lib-python/3/" x)))
+ '("tkinter/test"
+ "test"
+ "sqlite3/test"
+ "lib2to3/tests"
+ "idlelib/idle_test"
+ "distutils/tests"
+ "ctypes/test"
+ "unittest/test"))
+ ;; Patch shebang referencing python2
+ (substitute* '("lib-python/3/cgi.py"
+ "lib-python/3/encodings/rot_13.py")
+ ((shebang-match-python) shebang-pypy3))
+ (with-fluids ((%default-port-encoding "ISO-8859-1"))
+ (substitute* '("lib_pypy/_md5.py"
+ "lib_pypy/_sha1.py")
+ ((shebang-match-python) shebang-pypy3))))
+ (copy-recursively dist-dir out)
+ ;; Make sure pypy3 is callable as python/python3, so we
+ ;; don’t have to patch every single package.
+ (symlink bin-pypy3 (string-append out "/bin/python"))
+ (symlink bin-pypy3 (string-append out "/bin/python3"))
+ ;; 2to3 is missing from pypy3, create it.
+ (let ((2to3 (string-append out "/bin/2to3")))
+ (call-with-output-file 2to3
+ (lambda (port)
+ (format port "#!~a~%" (string-append out "/bin/pypy3"))
+ (format port "
+import sys
+from lib2to3.main import main
+
+sys.exit(main('lib2to3.fixes'))")))
+ (chmod 2to3 #o755))
+ #t))))))
+ (native-search-paths
+ (list (search-path-specification
+ (variable "PYTHONPATH")
+ (files '("lib/pypy3.6/site-packages")))))
(home-page "https://www.pypy.org/")
(synopsis "Python implementation with just-in-time compilation")
(description "PyPy is a faster, alternative implementation of the Python
@@ -849,4 +889,3 @@ Python code natively, including C extensions.")
license:bsd-3 ; lib_pypy/cffi/_pycparser/ply/
(license:non-copyleft
"http://www.unicode.org/copyright.html")))))
-
--
2.26.2
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [bug#44656] [PATCH] Upgrade pypy3
2020-11-15 12:39 [bug#44656] [PATCH] Upgrade pypy3 Lars-Dominik Braun
@ 2020-11-15 19:33 ` Christopher Baines
2020-11-15 19:55 ` Lars-Dominik Braun
2021-08-03 21:39 ` Maxim Cournoyer
1 sibling, 1 reply; 14+ messages in thread
From: Christopher Baines @ 2020-11-15 19:33 UTC (permalink / raw)
To: Lars-Dominik Braun; +Cc: 44656
[-- Attachment #1: Type: text/plain, Size: 792 bytes --]
Lars-Dominik Braun <lars@6xq.net> writes:
> the first patch in this series upgrades pypy3 to 7.3.2 and the second
> adjusts several aspects of the package.
Hi,
Given that the two patches really form just one change, the version
change with the necessary other changes to the package, would it be
possible to merge most or all of the two patches together?
Effectively, anything that's essential to change about the pypy3 package
when upgrading the version would ideally be in the same commit as the
version change. One reason for this is that it makes the change easier
to understand, as the actual changes are kept together in one commit.
If there are things that aren't essential to change when changing the
version, they're fine to have in subsequent commits/patches.
Thanks,
Chris
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]
^ permalink raw reply [flat|nested] 14+ messages in thread
* [bug#44656] [PATCH] Upgrade pypy3
2020-11-15 19:33 ` Christopher Baines
@ 2020-11-15 19:55 ` Lars-Dominik Braun
2020-11-16 17:26 ` Christopher Baines
0 siblings, 1 reply; 14+ messages in thread
From: Lars-Dominik Braun @ 2020-11-15 19:55 UTC (permalink / raw)
To: Christopher Baines; +Cc: 44656
Hi Chris,
> If there are things that aren't essential to change when changing the
> version, they're fine to have in subsequent commits/patches.
the patches do not depend on each other. The second patch merely
contains quality of life improvements that work with or without bumping
the version. But I though I just send them in one go.
Cheers,
Lars
^ permalink raw reply [flat|nested] 14+ messages in thread
* [bug#44656] [PATCH] Upgrade pypy3
2020-11-15 19:55 ` Lars-Dominik Braun
@ 2020-11-16 17:26 ` Christopher Baines
2020-11-17 9:38 ` Lars-Dominik Braun
0 siblings, 1 reply; 14+ messages in thread
From: Christopher Baines @ 2020-11-16 17:26 UTC (permalink / raw)
To: Lars-Dominik Braun; +Cc: 44656
[-- Attachment #1: Type: text/plain, Size: 3727 bytes --]
Lars-Dominik Braun <lars@6xq.net> writes:
>> If there are things that aren't essential to change when changing the
>> version, they're fine to have in subsequent commits/patches.
> the patches do not depend on each other. The second patch merely
> contains quality of life improvements that work with or without bumping
> the version. But I though I just send them in one go.
Fair enough, the version upgrade works fine for me.
Regarding the second patch:
diff --git a/gnu/packages/patches/pypy3-7.3.1-ssl-paths.patch b/gnu/packages/patches/pypy3-7.3.1-ssl-paths.patch
new file mode 100644
index 0000000000..d21133b4ae
--- /dev/null
+++ b/gnu/packages/patches/pypy3-7.3.1-ssl-paths.patch
@@ -0,0 +1,41 @@
+Fix default certificate search path, still allowing the user to override it
+with environment variables.
+
+--- a/lib_pypy/_cffi_ssl/_stdssl/__init__.py
++++ b/lib_pypy/_cffi_ssl/_stdssl/__init__.py
+@@ -1679,20 +1679,9 @@ def get_default_verify_paths():
+ https://golang.org/src/crypto/x509/root_linux.go (for the files)
+ '''
+ certFiles = [
+- "/etc/ssl/certs/ca-certificates.crt", # Debian/Ubuntu/Gentoo etc.
+- "/etc/pki/tls/certs/ca-bundle.crt", # Fedora/RHEL 6
+- "/etc/ssl/ca-bundle.pem", # OpenSUSE
+- "/etc/pki/tls/cacert.pem", # OpenELEC
+- "/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem", # CentOS/RHEL 7
+- "/etc/ssl/cert.pem", # Alpine Linux
+ ]
+ certDirectories = [
+- "/etc/ssl/certs", # SLES10/SLES11
+- "/system/etc/security/cacerts", # Android
+- "/usr/local/share/certs", # FreeBSD
+- "/etc/pki/tls/certs", # Fedora/RHEL
+- "/etc/openssl/certs", # NetBSD
+- "/var/ssl/certs", # AIX
++ "@GUIX_CERT_PATH@",
+ ]
I'm not sure about removing these bits. pypy3 from Guix may be used on
Debian or Fedora, and maybe using certificates from those systems would
be appropriate then?
+
+ # optimization: reuse the values from a local varaible
+@@ -1707,9 +1696,10 @@ def get_default_verify_paths():
+ ofile = _cstr_decode_fs(lib.X509_get_default_cert_file())
+ odir = _cstr_decode_fs(lib.X509_get_default_cert_dir())
+
+- if os.path.exists(ofile) and os.path.exists(odir):
+- get_default_verify_paths.retval = (ofile_env, ofile, odir_env, odir)
+- return get_default_verify_paths.retval
++ if not os.path.exists(ofile):
++ ofile = None
++ if not os.path.exists(odir):
++ odir = None
+
+ # OpenSSL didn't supply the goods. Try some other options
+ for f in certFiles:
@@ -793,7 +806,12 @@ ease from the desktop to a microcontroller or embedded system.")
; attribute 'IntFlag'
(invoke "python2" "package.py"
"--archive-name" "pypy-dist"
- "--builddir" (getcwd)))))
+ "--builddir" (getcwd))
+ ;; install pip and setuptools into the dist directory.
+ ;; XXX: Breaks virtualenv, because it does not set +w
+ ;; on files copied from the store.
+ ;(invoke "pypy-dist/bin/pypy3" "-m" "ensurepip")
+ )))
(replace 'check
(lambda* (#:key tests? #:allow-other-keys)
(if tests?
I'm not quite sure what this would do if it's commented back in.
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [bug#44656] [PATCH] Upgrade pypy3
2020-11-16 17:26 ` Christopher Baines
@ 2020-11-17 9:38 ` Lars-Dominik Braun
0 siblings, 0 replies; 14+ messages in thread
From: Lars-Dominik Braun @ 2020-11-17 9:38 UTC (permalink / raw)
To: Christopher Baines; +Cc: 44656
Hi Chris,
> I'm not sure about removing these bits. pypy3 from Guix may be used on
> Debian or Fedora, and maybe using certificates from those systems would
> be appropriate then?
I disagree, because then pypy might behave differently on Guix System
vs. foreign distros. A question up for discussion however is whether
it should depend on nss-certs at all.
> + ;; install pip and setuptools into the dist directory.
> + ;; XXX: Breaks virtualenv, because it does not set +w
> + ;; on files copied from the store.
> + ;(invoke "pypy-dist/bin/pypy3" "-m" "ensurepip")
> I'm not quite sure what this would do if it's commented back in.
It would install pip into the same output. Currently our python package
has pip built in and this change would change pypy3 to do the same.
Unfortunately it does not work well on foreign distros, thus I’m leaving
it commented out as future work.
Cheers,
Lars
^ permalink raw reply [flat|nested] 14+ messages in thread
* [bug#44656] [PATCH] Upgrade pypy3
2020-11-15 12:39 [bug#44656] [PATCH] Upgrade pypy3 Lars-Dominik Braun
2020-11-15 19:33 ` Christopher Baines
@ 2021-08-03 21:39 ` Maxim Cournoyer
2021-08-05 16:54 ` bug#44656: " Maxim Cournoyer
2021-08-06 7:30 ` [bug#44656] " Lars-Dominik Braun
1 sibling, 2 replies; 14+ messages in thread
From: Maxim Cournoyer @ 2021-08-03 21:39 UTC (permalink / raw)
To: Lars-Dominik Braun; +Cc: 44656
Hi Lars,
Lars-Dominik Braun <lars@6xq.net> writes:
> Hi,
>
> the first patch in this series upgrades pypy3 to 7.3.2 and the second
> adjusts several aspects of the package.
>
> Lars
>
>
>>From caf5e4bf9acd10f91de2e6a9c60185475144a1b9 Mon Sep 17 00:00:00 2001
> From: Lars-Dominik Braun <lars@6xq.net>
> Date: Sun, 15 Nov 2020 10:54:26 +0100
> Subject: [PATCH 2/2] gnu: pypy3: Various package fixes.
>
> * gnu/packages/python.scm (pypy3) [patches]: Add new patch.
> [inputs]: Remove bash-minimal and add nss-certs.
> [native-inputs]: Remove nss-certs here.
In Guix we leave the choice of the TLS certs to the users; meaning we
don't hard-code their location in packages and instead have the users
explicitly install them (and in the case of OpenSSL, set the necessary
environment variables), the same we do for icons and other 'choosable'
things; while convenient the above goes against this tradition.
> [arguments]: Use gdbm compat library, add 2to3 binary.
> [native-search-paths]: Add search path.
> * gnu/packages/patches/pypy3-7.3.1-ssl-paths.patch: New file.
> * gnu/local.mk: Add it.
> ---
> gnu/local.mk | 1 +
> .../patches/pypy3-7.3.1-ssl-paths.patch | 41 +++++++
> gnu/packages/python.scm | 109 ++++++++++++------
> 3 files changed, 116 insertions(+), 35 deletions(-)
> create mode 100644 gnu/packages/patches/pypy3-7.3.1-ssl-paths.patch
>
> diff --git a/gnu/local.mk b/gnu/local.mk
> index 91a3295e75..b644391a11 100644
> --- a/gnu/local.mk
> +++ b/gnu/local.mk
> @@ -1536,6 +1536,7 @@ dist_patch_DATA = \
> %D%/packages/patches/python-unittest2-remove-argparse.patch \
> %D%/packages/patches/python-waitress-fix-tests.patch \
> %D%/packages/patches/pypy3-7.3.1-fix-tests.patch \
> + %D%/packages/patches/pypy3-7.3.1-ssl-paths.patch \
> %D%/packages/patches/qemu-build-info-manual.patch \
> %D%/packages/patches/qemu-glibc-2.27.patch \
> %D%/packages/patches/qrcodegen-cpp-make-install.patch \
> diff --git a/gnu/packages/patches/pypy3-7.3.1-ssl-paths.patch b/gnu/packages/patches/pypy3-7.3.1-ssl-paths.patch
> new file mode 100644
> index 0000000000..d21133b4ae
> --- /dev/null
> +++ b/gnu/packages/patches/pypy3-7.3.1-ssl-paths.patch
> @@ -0,0 +1,41 @@
> +Fix default certificate search path, still allowing the user to override it
> +with environment variables.
> +
> +--- a/lib_pypy/_cffi_ssl/_stdssl/__init__.py
> ++++ b/lib_pypy/_cffi_ssl/_stdssl/__init__.py
> +@@ -1679,20 +1679,9 @@ def get_default_verify_paths():
> + https://golang.org/src/crypto/x509/root_linux.go (for the files)
> + '''
> + certFiles = [
> +- "/etc/ssl/certs/ca-certificates.crt", # Debian/Ubuntu/Gentoo etc.
> +- "/etc/pki/tls/certs/ca-bundle.crt", # Fedora/RHEL 6
> +- "/etc/ssl/ca-bundle.pem", # OpenSUSE
> +- "/etc/pki/tls/cacert.pem", # OpenELEC
> +- "/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem", # CentOS/RHEL 7
> +- "/etc/ssl/cert.pem", # Alpine Linux
> + ]
> + certDirectories = [
> +- "/etc/ssl/certs", # SLES10/SLES11
> +- "/system/etc/security/cacerts", # Android
> +- "/usr/local/share/certs", # FreeBSD
> +- "/etc/pki/tls/certs", # Fedora/RHEL
> +- "/etc/openssl/certs", # NetBSD
> +- "/var/ssl/certs", # AIX
> ++ "@GUIX_CERT_PATH@",
> + ]
> +
> + # optimization: reuse the values from a local varaible
> +@@ -1707,9 +1696,10 @@ def get_default_verify_paths():
> + ofile = _cstr_decode_fs(lib.X509_get_default_cert_file())
> + odir = _cstr_decode_fs(lib.X509_get_default_cert_dir())
> +
> +- if os.path.exists(ofile) and os.path.exists(odir):
> +- get_default_verify_paths.retval = (ofile_env, ofile, odir_env, odir)
> +- return get_default_verify_paths.retval
> ++ if not os.path.exists(ofile):
> ++ ofile = None
> ++ if not os.path.exists(odir):
> ++ odir = None
> +
> + # OpenSSL didn't supply the goods. Try some other options
> + for f in certFiles:
As mentioned above, the choice of certs should not be hard coded the in
the package definition. The correct behavior of honoring SSL_CERT_DIR
and SSL_CERT_FILE environment variables, and falling back to the system
provided location is sane, so the above patch is unwelcome, IMHO.
> diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
> index 8ef8ae2e1d..c0bd3335e3 100644
> --- a/gnu/packages/python.scm
> +++ b/gnu/packages/python.scm
> @@ -696,7 +696,8 @@ ease from the desktop to a microcontroller or embedded system.")
> (sha256
> (base32
> "03f1fdw6yk2mypa9pbmgk26r8y1hhmw801l6g36zry9zsvz7aqgx"))
> - (patches (search-patches "pypy3-7.3.1-fix-tests.patch"))))
> + (patches (search-patches "pypy3-7.3.1-fix-tests.patch"
> + "pypy3-7.3.1-ssl-paths.patch"))))
> (build-system gnu-build-system)
> (native-inputs
> `(("python-2" ,python-2)
> @@ -704,13 +705,13 @@ ease from the desktop to a microcontroller or embedded system.")
> ("tar" ,tar) ; Required for package.py
> ("python2-pycparser" ,python2-pycparser)
> ("python2-hypothesis" ,python2-hypothesis)
> - ("nss-certs" ,nss-certs) ; For ssl tests
> ("gzip" ,gzip)))
> (inputs
> `(("libffi" ,libffi)
> ("zlib" ,zlib)
> ("ncurses" ,ncurses)
> ("openssl" ,openssl)
> + ("nss-certs" ,nss-certs) ; For ssl module
This change is not needed without the TLS patching above.
> ("expat" ,expat)
> ("bzip2" ,bzip2)
> ("sqlite" ,sqlite)
> @@ -718,10 +719,9 @@ ease from the desktop to a microcontroller or embedded system.")
> ("tcl" ,tcl)
> ("tk" ,tk)
> ("glibc" ,glibc)
> - ("bash-minimal" ,bash-minimal) ; Used as /bin/sh
> ("xz" ,xz))) ; liblzma
> (arguments
> - `(#:tests? #f ;FIXME: Disabled for now, there are many tests failing.
> + `(#:tests? #f ;FIXME: Disabled for now, there are many tests failing.
> #:modules ((ice-9 ftw) (ice-9 match)
> (guix build utils) (guix build gnu-build-system))
> #:phases (modify-phases %standard-phases
> @@ -750,6 +750,11 @@ ease from the desktop to a microcontroller or embedded system.")
> (substitute* '("lib_pypy/_curses_build.py")
> ;; Find curses
> (("/usr/local") (assoc-ref inputs "ncurses")))
> + (substitute* '("lib_pypy/_dbm.py")
> + ;; Use gdbm compat library, so we don’t need to pull in bdb
> + (("ctypes.util.find_library\\('db'\\)")
> + (string-append "'" (assoc-ref inputs "gdbm")
> + "/lib/libgdbm_compat.so'")))
OK.
> (substitute* '("lib_pypy/_sqlite3_build.py")
> ;; Always use search paths
> (("sys\\.platform\\.startswith\\('freebsd'\\)") "True")
> @@ -761,11 +766,18 @@ ease from the desktop to a microcontroller or embedded system.")
> "/lib/libsqlite3.so.0'")))
> (substitute* '("lib-python/3/subprocess.py")
> ;; Fix shell path
> - (("/bin/sh")
> - (string-append (assoc-ref inputs "bash-minimal") "/bin/sh")))
> + (("/bin/sh") (which "sh")))
OK.
> (substitute* '("lib-python/3/distutils/unixccompiler.py")
> ;; gcc-toolchain does not provide symlink cc -> gcc
> (("\"cc\"") "\"gcc\""))
> + (substitute* '("lib_pypy/_cffi_ssl/_stdssl/__init__.py")
> + ;; Add nss-certs to default certificate search path,
> + ;; otherwise every packages has to specify nss-certs and
> + ;; openssl as input to set the proper env variables.
> + ;; Depends on -ssl-paths.patch.
> + (("@GUIX_CERT_PATH@")
> + (string-append (assoc-ref inputs "nss-certs")
> + "/etc/ssl/certs")))
Not every package; the required SSL environment variables it should be
set in the environment by the user or by via a native search path, and
the certs manually provided (installed) by the user.
> #t))
> (add-after
> 'unpack 'set-source-file-times-to-1980
> @@ -785,7 +797,8 @@ ease from the desktop to a microcontroller or embedded system.")
> (string-append "--make-jobs="
> (number->string (parallel-job-count)))
> "-Ojit"
> - "targetpypystandalone"))
> + "targetpypystandalone"
> + "--allworkingmodules"))
> ;; Build c modules and package everything, so tests work.
> (with-directory-excursion "pypy/tool/release"
> (unsetenv "PYTHONPATH") ; Do not use the system’s python libs:
> @@ -793,7 +806,12 @@ ease from the desktop to a microcontroller or embedded system.")
> ; attribute 'IntFlag'
> (invoke "python2" "package.py"
> "--archive-name" "pypy-dist"
> - "--builddir" (getcwd)))))
> + "--builddir" (getcwd))
> + ;; install pip and setuptools into the dist directory.
> + ;; XXX: Breaks virtualenv, because it does not set +w
> + ;; on files copied from the store.
> + ;(invoke "pypy-dist/bin/pypy3" "-m" "ensurepip")
> + )))
Since the above attempt to bundle pip failed and is new, I'd just leave
it out.
> (replace 'check
> (lambda* (#:key tests? #:allow-other-keys)
> (if tests?
> @@ -811,32 +829,54 @@ ease from the desktop to a microcontroller or embedded system.")
> #t))
> (replace 'install
> (lambda* (#:key inputs outputs #:allow-other-keys)
> - (with-directory-excursion "pypy/tool/release"
> - ;; Delete test data.
> - (for-each
> - (lambda (x)
> - (delete-file-recursively (string-append
> - "pypy-dist/lib-python/3/" x)))
> - '("tkinter/test"
> - "test"
> - "sqlite3/test"
> - "lib2to3/tests"
> - "idlelib/idle_test"
> - "distutils/tests"
> - "ctypes/test"
> - "unittest/test"))
> - ;; Patch shebang referencing python2
> - (substitute* '("pypy-dist/lib-python/3/cgi.py"
> - "pypy-dist/lib-python/3/encodings/rot_13.py")
> - (("#!.+/bin/python")
> - (string-append "#!" (assoc-ref outputs "out") "/bin/pypy3")))
> - (with-fluids ((%default-port-encoding "ISO-8859-1"))
> - (substitute* '("pypy-dist/lib_pypy/_md5.py"
> - "pypy-dist/lib_pypy/_sha1.py")
> - (("#!.+/bin/python")
> - (string-append "#!" (assoc-ref outputs "out") "/bin/pypy3"))))
> - (copy-recursively "pypy-dist" (assoc-ref outputs "out")))
> - #t)))))
> + (let* ((out (assoc-ref outputs "out"))
> + (bin-pypy3 (string-append out "/bin/pypy3"))
> + (shebang-match-python "#!.+/bin/python")
> + (shebang-pypy3 (string-append "#!" bin-pypy3))
> + (dist-dir "pypy/tool/release/pypy-dist"))
> + (with-directory-excursion dist-dir
> + ;; Delete test data.
> + (for-each
> + (lambda (x)
> + (delete-file-recursively (string-append
> + "lib-python/3/" x)))
> + '("tkinter/test"
> + "test"
> + "sqlite3/test"
> + "lib2to3/tests"
> + "idlelib/idle_test"
> + "distutils/tests"
> + "ctypes/test"
> + "unittest/test"))
> + ;; Patch shebang referencing python2
> + (substitute* '("lib-python/3/cgi.py"
> + "lib-python/3/encodings/rot_13.py")
> + ((shebang-match-python) shebang-pypy3))
> + (with-fluids ((%default-port-encoding "ISO-8859-1"))
> + (substitute* '("lib_pypy/_md5.py"
> + "lib_pypy/_sha1.py")
> + ((shebang-match-python) shebang-pypy3))))
> + (copy-recursively dist-dir out)
> + ;; Make sure pypy3 is callable as python/python3, so we
> + ;; don’t have to patch every single package.
> + (symlink bin-pypy3 (string-append out "/bin/python"))
> + (symlink bin-pypy3 (string-append out "/bin/python3"))
> + ;; 2to3 is missing from pypy3, create it.
> + (let ((2to3 (string-append out "/bin/2to3")))
> + (call-with-output-file 2to3
> + (lambda (port)
> + (format port "#!~a~%" (string-append out "/bin/pypy3"))
> + (format port "
> +import sys
> +from lib2to3.main import main
> +
> +sys.exit(main('lib2to3.fixes'))")))
> + (chmod 2to3 #o755))
> + #t))))))
I'm unconvinced about the above symlinks; it seems preferable for users
to be able to unambiguously run both pypy3 and python3 (cpython) in the
same profile without conflicts. The pypy3 package on Debian contains
the following, for example:
--8<---------------cut here---------------start------------->8---
$ apt-file show pypy3
pypy3: /usr/bin/pypy3
pypy3: /usr/bin/pypy3clean
pypy3: /usr/bin/pypy3compile
pypy3: /usr/lib/libpypy3-c.so
pypy3: /usr/lib/pypy3/bin/libpypy3-c.so
pypy3: /usr/lib/pypy3/bin/pypy3-c
pypy3: /usr/lib/pypy3/include/pypy_decl.h
pypy3: /usr/lib/pypy3/include/pypy_macros.h
pypy3: /usr/lib/pypy3/include/pypy_marshal_decl.h
pypy3: /usr/lib/pypy3/include/pypy_structmember_decl.h
pypy3: /usr/lib/pypy3/lib_pypy/_audioop_cffi.pypy36-pp73-x86_64-linux-gnu.so
pypy3: /usr/lib/pypy3/lib_pypy/_blake2/_blake2b_cffi.pypy36-pp73-x86_64-linux-gnu.so
pypy3: /usr/lib/pypy3/lib_pypy/_blake2/_blake2s_cffi.pypy36-pp73-x86_64-linux-gnu.so
pypy3: /usr/lib/pypy3/lib_pypy/_curses_cffi.pypy36-pp73-x86_64-linux-gnu.so
pypy3: /usr/lib/pypy3/lib_pypy/_decimal_cffi.pypy36-pp73-x86_64-linux-gnu.so
pypy3: /usr/lib/pypy3/lib_pypy/_gdbm_cffi.pypy36-pp73-x86_64-linux-gnu.so
pypy3: /usr/lib/pypy3/lib_pypy/_lzma_cffi.pypy36-pp73-x86_64-linux-gnu.so
pypy3: /usr/lib/pypy3/lib_pypy/_pwdgrp_cffi.pypy36-pp73-x86_64-linux-gnu.so
pypy3: /usr/lib/pypy3/lib_pypy/_pypy_openssl.pypy36-pp73-x86_64-linux-gnu.so
pypy3: /usr/lib/pypy3/lib_pypy/_resource_cffi.pypy36-pp73-x86_64-linux-gnu.so
pypy3: /usr/lib/pypy3/lib_pypy/_sha3/_sha3_cffi.pypy36-pp73-x86_64-linux-gnu.so
pypy3: /usr/lib/pypy3/lib_pypy/_sqlite3_cffi.pypy36-pp73-x86_64-linux-gnu.so
pypy3: /usr/lib/pypy3/lib_pypy/_syslog_cffi.pypy36-pp73-x86_64-linux-gnu.so
pypy3: /usr/share/doc/pypy3/TODO.Debian
pypy3: /usr/share/doc/pypy3/changelog.Debian.gz
pypy3: /usr/share/doc/pypy3/copyright
pypy3: /usr/share/lintian/overrides/pypy3
pypy3: /usr/share/man/man1/pypy3.1.gz
--8<---------------cut here---------------end--------------->8---
> + (native-search-paths
> + (list (search-path-specification
> + (variable "PYTHONPATH")
> + (files '("lib/pypy3.6/site-packages")))))
About this search path, how it is supposed to work? The version is
wrong (3.6) and the package doesn't include this directory, and we do
not have a pypy build system or other pypy packages that would make use
of it. Am I missing something?
Thank you!
Maxim
^ permalink raw reply [flat|nested] 14+ messages in thread
* bug#44656: [PATCH] Upgrade pypy3
2021-08-03 21:39 ` Maxim Cournoyer
@ 2021-08-05 16:54 ` Maxim Cournoyer
2021-08-06 7:30 ` [bug#44656] " Lars-Dominik Braun
1 sibling, 0 replies; 14+ messages in thread
From: Maxim Cournoyer @ 2021-08-05 16:54 UTC (permalink / raw)
To: Lars-Dominik Braun; +Cc: 44656-done
Hello again,
Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
> Hi Lars,
>
> Lars-Dominik Braun <lars@6xq.net> writes:
>
>> Hi,
>>
>> the first patch in this series upgrades pypy3 to 7.3.2 and the second
>> adjusts several aspects of the package.
>>
>> Lars
>>
>>
>>>>From caf5e4bf9acd10f91de2e6a9c60185475144a1b9 Mon Sep 17 00:00:00 2001
>> From: Lars-Dominik Braun <lars@6xq.net>
>> Date: Sun, 15 Nov 2020 10:54:26 +0100
>> Subject: [PATCH 2/2] gnu: pypy3: Various package fixes.
>>
>> * gnu/packages/python.scm (pypy3) [patches]: Add new patch.
>> [inputs]: Remove bash-minimal and add nss-certs.
>> [native-inputs]: Remove nss-certs here.
>
> In Guix we leave the choice of the TLS certs to the users; meaning we
> don't hard-code their location in packages and instead have the users
> explicitly install them (and in the case of OpenSSL, set the necessary
> environment variables), the same we do for icons and other 'choosable'
> things; while convenient the above goes against this tradition.
>
>> [arguments]: Use gdbm compat library, add 2to3 binary.
>> [native-search-paths]: Add search path.
>> * gnu/packages/patches/pypy3-7.3.1-ssl-paths.patch: New file.
>> * gnu/local.mk: Add it.
>> ---
>> gnu/local.mk | 1 +
>> .../patches/pypy3-7.3.1-ssl-paths.patch | 41 +++++++
>> gnu/packages/python.scm | 109 ++++++++++++------
>> 3 files changed, 116 insertions(+), 35 deletions(-)
>> create mode 100644 gnu/packages/patches/pypy3-7.3.1-ssl-paths.patch
>>
>> diff --git a/gnu/local.mk b/gnu/local.mk
>> index 91a3295e75..b644391a11 100644
>> --- a/gnu/local.mk
>> +++ b/gnu/local.mk
>> @@ -1536,6 +1536,7 @@ dist_patch_DATA = \
>> %D%/packages/patches/python-unittest2-remove-argparse.patch \
>> %D%/packages/patches/python-waitress-fix-tests.patch \
>> %D%/packages/patches/pypy3-7.3.1-fix-tests.patch \
>> + %D%/packages/patches/pypy3-7.3.1-ssl-paths.patch \
>> %D%/packages/patches/qemu-build-info-manual.patch \
>> %D%/packages/patches/qemu-glibc-2.27.patch \
>> %D%/packages/patches/qrcodegen-cpp-make-install.patch \
>> diff --git a/gnu/packages/patches/pypy3-7.3.1-ssl-paths.patch b/gnu/packages/patches/pypy3-7.3.1-ssl-paths.patch
>> new file mode 100644
>> index 0000000000..d21133b4ae
>> --- /dev/null
>> +++ b/gnu/packages/patches/pypy3-7.3.1-ssl-paths.patch
>> @@ -0,0 +1,41 @@
>> +Fix default certificate search path, still allowing the user to override it
>> +with environment variables.
>> +
>> +--- a/lib_pypy/_cffi_ssl/_stdssl/__init__.py
>> ++++ b/lib_pypy/_cffi_ssl/_stdssl/__init__.py
>> +@@ -1679,20 +1679,9 @@ def get_default_verify_paths():
>> + https://golang.org/src/crypto/x509/root_linux.go (for the files)
>> + '''
>> + certFiles = [
>> +- "/etc/ssl/certs/ca-certificates.crt", # Debian/Ubuntu/Gentoo etc.
>> +- "/etc/pki/tls/certs/ca-bundle.crt", # Fedora/RHEL 6
>> +- "/etc/ssl/ca-bundle.pem", # OpenSUSE
>> +- "/etc/pki/tls/cacert.pem", # OpenELEC
>> +- "/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem", # CentOS/RHEL 7
>> +- "/etc/ssl/cert.pem", # Alpine Linux
>> + ]
>> + certDirectories = [
>> +- "/etc/ssl/certs", # SLES10/SLES11
>> +- "/system/etc/security/cacerts", # Android
>> +- "/usr/local/share/certs", # FreeBSD
>> +- "/etc/pki/tls/certs", # Fedora/RHEL
>> +- "/etc/openssl/certs", # NetBSD
>> +- "/var/ssl/certs", # AIX
>> ++ "@GUIX_CERT_PATH@",
>> + ]
>> +
>> + # optimization: reuse the values from a local varaible
>> +@@ -1707,9 +1696,10 @@ def get_default_verify_paths():
>> + ofile = _cstr_decode_fs(lib.X509_get_default_cert_file())
>> + odir = _cstr_decode_fs(lib.X509_get_default_cert_dir())
>> +
>> +- if os.path.exists(ofile) and os.path.exists(odir):
>> +- get_default_verify_paths.retval = (ofile_env, ofile, odir_env, odir)
>> +- return get_default_verify_paths.retval
>> ++ if not os.path.exists(ofile):
>> ++ ofile = None
>> ++ if not os.path.exists(odir):
>> ++ odir = None
>> +
>> + # OpenSSL didn't supply the goods. Try some other options
>> + for f in certFiles:
>
> As mentioned above, the choice of certs should not be hard coded the in
> the package definition. The correct behavior of honoring SSL_CERT_DIR
> and SSL_CERT_FILE environment variables, and falling back to the system
> provided location is sane, so the above patch is unwelcome, IMHO.
>
>> diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
>> index 8ef8ae2e1d..c0bd3335e3 100644
>> --- a/gnu/packages/python.scm
>> +++ b/gnu/packages/python.scm
>> @@ -696,7 +696,8 @@ ease from the desktop to a microcontroller or embedded system.")
>> (sha256
>> (base32
>> "03f1fdw6yk2mypa9pbmgk26r8y1hhmw801l6g36zry9zsvz7aqgx"))
>> - (patches (search-patches "pypy3-7.3.1-fix-tests.patch"))))
>> + (patches (search-patches "pypy3-7.3.1-fix-tests.patch"
>> + "pypy3-7.3.1-ssl-paths.patch"))))
>> (build-system gnu-build-system)
>> (native-inputs
>> `(("python-2" ,python-2)
>> @@ -704,13 +705,13 @@ ease from the desktop to a microcontroller or embedded system.")
>> ("tar" ,tar) ; Required for package.py
>> ("python2-pycparser" ,python2-pycparser)
>> ("python2-hypothesis" ,python2-hypothesis)
>> - ("nss-certs" ,nss-certs) ; For ssl tests
>> ("gzip" ,gzip)))
>> (inputs
>> `(("libffi" ,libffi)
>> ("zlib" ,zlib)
>> ("ncurses" ,ncurses)
>> ("openssl" ,openssl)
>> + ("nss-certs" ,nss-certs) ; For ssl module
>
> This change is not needed without the TLS patching above.
>
>> ("expat" ,expat)
>> ("bzip2" ,bzip2)
>> ("sqlite" ,sqlite)
>> @@ -718,10 +719,9 @@ ease from the desktop to a microcontroller or embedded system.")
>> ("tcl" ,tcl)
>> ("tk" ,tk)
>> ("glibc" ,glibc)
>> - ("bash-minimal" ,bash-minimal) ; Used as /bin/sh
>> ("xz" ,xz))) ; liblzma
>> (arguments
>> - `(#:tests? #f ;FIXME: Disabled for now, there are many tests failing.
>> + `(#:tests? #f ;FIXME: Disabled for now, there are many tests failing.
>> #:modules ((ice-9 ftw) (ice-9 match)
>> (guix build utils) (guix build gnu-build-system))
>> #:phases (modify-phases %standard-phases
>> @@ -750,6 +750,11 @@ ease from the desktop to a microcontroller or embedded system.")
>> (substitute* '("lib_pypy/_curses_build.py")
>> ;; Find curses
>> (("/usr/local") (assoc-ref inputs "ncurses")))
>> + (substitute* '("lib_pypy/_dbm.py")
>> + ;; Use gdbm compat library, so we don’t need to pull in bdb
>> + (("ctypes.util.find_library\\('db'\\)")
>> + (string-append "'" (assoc-ref inputs "gdbm")
>> + "/lib/libgdbm_compat.so'")))
>
> OK.
>
>> (substitute* '("lib_pypy/_sqlite3_build.py")
>> ;; Always use search paths
>> (("sys\\.platform\\.startswith\\('freebsd'\\)") "True")
>> @@ -761,11 +766,18 @@ ease from the desktop to a microcontroller or embedded system.")
>> "/lib/libsqlite3.so.0'")))
>> (substitute* '("lib-python/3/subprocess.py")
>> ;; Fix shell path
>> - (("/bin/sh")
>> - (string-append (assoc-ref inputs "bash-minimal") "/bin/sh")))
>> + (("/bin/sh") (which "sh")))
>
> OK.
>
>> (substitute* '("lib-python/3/distutils/unixccompiler.py")
>> ;; gcc-toolchain does not provide symlink cc -> gcc
>> (("\"cc\"") "\"gcc\""))
>> + (substitute* '("lib_pypy/_cffi_ssl/_stdssl/__init__.py")
>> + ;; Add nss-certs to default certificate search path,
>> + ;; otherwise every packages has to specify nss-certs and
>> + ;; openssl as input to set the proper env variables.
>> + ;; Depends on -ssl-paths.patch.
>> + (("@GUIX_CERT_PATH@")
>> + (string-append (assoc-ref inputs "nss-certs")
>> + "/etc/ssl/certs")))
>
> Not every package; the required SSL environment variables it should be
> set in the environment by the user or by via a native search path, and
> the certs manually provided (installed) by the user.
>
>> #t))
>> (add-after
>> 'unpack 'set-source-file-times-to-1980
>> @@ -785,7 +797,8 @@ ease from the desktop to a microcontroller or embedded system.")
>> (string-append "--make-jobs="
>> (number->string (parallel-job-count)))
>> "-Ojit"
>> - "targetpypystandalone"))
>> + "targetpypystandalone"
>> + "--allworkingmodules"))
>> ;; Build c modules and package everything, so tests work.
>> (with-directory-excursion "pypy/tool/release"
>> (unsetenv "PYTHONPATH") ; Do not use the system’s python libs:
>> @@ -793,7 +806,12 @@ ease from the desktop to a microcontroller or embedded system.")
>> ; attribute 'IntFlag'
>> (invoke "python2" "package.py"
>> "--archive-name" "pypy-dist"
>> - "--builddir" (getcwd)))))
>> + "--builddir" (getcwd))
>> + ;; install pip and setuptools into the dist directory.
>> + ;; XXX: Breaks virtualenv, because it does not set +w
>> + ;; on files copied from the store.
>> + ;(invoke "pypy-dist/bin/pypy3" "-m" "ensurepip")
>> + )))
>
> Since the above attempt to bundle pip failed and is new, I'd just leave
> it out.
>
>> (replace 'check
>> (lambda* (#:key tests? #:allow-other-keys)
>> (if tests?
>> @@ -811,32 +829,54 @@ ease from the desktop to a microcontroller or embedded system.")
>> #t))
>> (replace 'install
>> (lambda* (#:key inputs outputs #:allow-other-keys)
>> - (with-directory-excursion "pypy/tool/release"
>> - ;; Delete test data.
>> - (for-each
>> - (lambda (x)
>> - (delete-file-recursively (string-append
>> - "pypy-dist/lib-python/3/" x)))
>> - '("tkinter/test"
>> - "test"
>> - "sqlite3/test"
>> - "lib2to3/tests"
>> - "idlelib/idle_test"
>> - "distutils/tests"
>> - "ctypes/test"
>> - "unittest/test"))
>> - ;; Patch shebang referencing python2
>> - (substitute* '("pypy-dist/lib-python/3/cgi.py"
>> - "pypy-dist/lib-python/3/encodings/rot_13.py")
>> - (("#!.+/bin/python")
>> - (string-append "#!" (assoc-ref outputs "out") "/bin/pypy3")))
>> - (with-fluids ((%default-port-encoding "ISO-8859-1"))
>> - (substitute* '("pypy-dist/lib_pypy/_md5.py"
>> - "pypy-dist/lib_pypy/_sha1.py")
>> - (("#!.+/bin/python")
>> - (string-append "#!" (assoc-ref outputs "out") "/bin/pypy3"))))
>> - (copy-recursively "pypy-dist" (assoc-ref outputs "out")))
>> - #t)))))
>> + (let* ((out (assoc-ref outputs "out"))
>> + (bin-pypy3 (string-append out "/bin/pypy3"))
>> + (shebang-match-python "#!.+/bin/python")
>> + (shebang-pypy3 (string-append "#!" bin-pypy3))
>> + (dist-dir "pypy/tool/release/pypy-dist"))
>> + (with-directory-excursion dist-dir
>> + ;; Delete test data.
>> + (for-each
>> + (lambda (x)
>> + (delete-file-recursively (string-append
>> + "lib-python/3/" x)))
>> + '("tkinter/test"
>> + "test"
>> + "sqlite3/test"
>> + "lib2to3/tests"
>> + "idlelib/idle_test"
>> + "distutils/tests"
>> + "ctypes/test"
>> + "unittest/test"))
>> + ;; Patch shebang referencing python2
>> + (substitute* '("lib-python/3/cgi.py"
>> + "lib-python/3/encodings/rot_13.py")
>> + ((shebang-match-python) shebang-pypy3))
>> + (with-fluids ((%default-port-encoding "ISO-8859-1"))
>> + (substitute* '("lib_pypy/_md5.py"
>> + "lib_pypy/_sha1.py")
>> + ((shebang-match-python) shebang-pypy3))))
>> + (copy-recursively dist-dir out)
>> + ;; Make sure pypy3 is callable as python/python3, so we
>> + ;; don’t have to patch every single package.
>> + (symlink bin-pypy3 (string-append out "/bin/python"))
>> + (symlink bin-pypy3 (string-append out "/bin/python3"))
>> + ;; 2to3 is missing from pypy3, create it.
>> + (let ((2to3 (string-append out "/bin/2to3")))
>> + (call-with-output-file 2to3
>> + (lambda (port)
>> + (format port "#!~a~%" (string-append out "/bin/pypy3"))
>> + (format port "
>> +import sys
>> +from lib2to3.main import main
>> +
>> +sys.exit(main('lib2to3.fixes'))")))
>> + (chmod 2to3 #o755))
>> + #t))))))
>
> I'm unconvinced about the above symlinks; it seems preferable for users
> to be able to unambiguously run both pypy3 and python3 (cpython) in the
> same profile without conflicts. The pypy3 package on Debian contains
> the following, for example:
>
> $ apt-file show pypy3
> pypy3: /usr/bin/pypy3
> pypy3: /usr/bin/pypy3clean
> pypy3: /usr/bin/pypy3compile
> pypy3: /usr/lib/libpypy3-c.so
> pypy3: /usr/lib/pypy3/bin/libpypy3-c.so
> pypy3: /usr/lib/pypy3/bin/pypy3-c
> pypy3: /usr/lib/pypy3/include/pypy_decl.h
> pypy3: /usr/lib/pypy3/include/pypy_macros.h
> pypy3: /usr/lib/pypy3/include/pypy_marshal_decl.h
> pypy3: /usr/lib/pypy3/include/pypy_structmember_decl.h
> pypy3: /usr/lib/pypy3/lib_pypy/_audioop_cffi.pypy36-pp73-x86_64-linux-gnu.so
> pypy3: /usr/lib/pypy3/lib_pypy/_blake2/_blake2b_cffi.pypy36-pp73-x86_64-linux-gnu.so
> pypy3: /usr/lib/pypy3/lib_pypy/_blake2/_blake2s_cffi.pypy36-pp73-x86_64-linux-gnu.so
> pypy3: /usr/lib/pypy3/lib_pypy/_curses_cffi.pypy36-pp73-x86_64-linux-gnu.so
> pypy3: /usr/lib/pypy3/lib_pypy/_decimal_cffi.pypy36-pp73-x86_64-linux-gnu.so
> pypy3: /usr/lib/pypy3/lib_pypy/_gdbm_cffi.pypy36-pp73-x86_64-linux-gnu.so
> pypy3: /usr/lib/pypy3/lib_pypy/_lzma_cffi.pypy36-pp73-x86_64-linux-gnu.so
> pypy3: /usr/lib/pypy3/lib_pypy/_pwdgrp_cffi.pypy36-pp73-x86_64-linux-gnu.so
> pypy3: /usr/lib/pypy3/lib_pypy/_pypy_openssl.pypy36-pp73-x86_64-linux-gnu.so
> pypy3: /usr/lib/pypy3/lib_pypy/_resource_cffi.pypy36-pp73-x86_64-linux-gnu.so
> pypy3: /usr/lib/pypy3/lib_pypy/_sha3/_sha3_cffi.pypy36-pp73-x86_64-linux-gnu.so
> pypy3: /usr/lib/pypy3/lib_pypy/_sqlite3_cffi.pypy36-pp73-x86_64-linux-gnu.so
> pypy3: /usr/lib/pypy3/lib_pypy/_syslog_cffi.pypy36-pp73-x86_64-linux-gnu.so
> pypy3: /usr/share/doc/pypy3/TODO.Debian
> pypy3: /usr/share/doc/pypy3/changelog.Debian.gz
> pypy3: /usr/share/doc/pypy3/copyright
> pypy3: /usr/share/lintian/overrides/pypy3
> pypy3: /usr/share/man/man1/pypy3.1.gz
>
>> + (native-search-paths
>> + (list (search-path-specification
>> + (variable "PYTHONPATH")
>> + (files '("lib/pypy3.6/site-packages")))))
>
> About this search path, how it is supposed to work? The version is
> wrong (3.6) and the package doesn't include this directory, and we do
> not have a pypy build system or other pypy packages that would make use
> of it. Am I missing something?
>
> Thank you!
>
> Maxim
I've upgraded pypy to version 7.3.5 and pushed the non-controversial
changes from your 'small fixes' commit above as
7e05fda6e8dd72b63009181072a282ad15c25c4f. We can fix the search path
after the next core-updates merge, where we'll be able to uniformize how
it works with Python (GUIX_PYTHONPATH).
Thanks,
Closing.
Maxim
^ permalink raw reply [flat|nested] 14+ messages in thread
* [bug#44656] [PATCH] Upgrade pypy3
2021-08-03 21:39 ` Maxim Cournoyer
2021-08-05 16:54 ` bug#44656: " Maxim Cournoyer
@ 2021-08-06 7:30 ` Lars-Dominik Braun
2021-08-06 14:22 ` Maxim Cournoyer
1 sibling, 1 reply; 14+ messages in thread
From: Lars-Dominik Braun @ 2021-08-06 7:30 UTC (permalink / raw)
To: Maxim Cournoyer; +Cc: 44656
Hi Maxim,
I realize I’m late, so only one comment:
> > + (native-search-paths
> > + (list (search-path-specification
> > + (variable "PYTHONPATH")
> > + (files '("lib/pypy3.6/site-packages")))))
> About this search path, how it is supposed to work? The version is
> wrong (3.6) and the package doesn't include this directory, and we do
> not have a pypy build system or other pypy packages that would make use
> of it. Am I missing something?
No, the the version is not wrong. pypy tracks CPython’s development
(features, standard library) and thus implements a certain “Python
version”, in this case 3.6, even though pypy’s own version says
something else. But, as you said, pypy in Guix is not really useful
anyway right now without any packages.
(That was the idea of the symlinks by the way, providing pypy as a
drop-in replacement for python-wrapper in python-build-system.)
Cheers,
Lars
^ permalink raw reply [flat|nested] 14+ messages in thread
* [bug#44656] [PATCH] Upgrade pypy3
2021-08-06 7:30 ` [bug#44656] " Lars-Dominik Braun
@ 2021-08-06 14:22 ` Maxim Cournoyer
2021-08-06 14:43 ` Lars-Dominik Braun
0 siblings, 1 reply; 14+ messages in thread
From: Maxim Cournoyer @ 2021-08-06 14:22 UTC (permalink / raw)
To: Lars-Dominik Braun; +Cc: 44656
Hello Lars-Dominik,
Lars-Dominik Braun <lars@6xq.net> writes:
> Hi Maxim,
>
> I realize I’m late, so only one comment:
>
>> > + (native-search-paths
>> > + (list (search-path-specification
>> > + (variable "PYTHONPATH")
>> > + (files '("lib/pypy3.6/site-packages")))))
>> About this search path, how it is supposed to work? The version is
>> wrong (3.6) and the package doesn't include this directory, and we do
>> not have a pypy build system or other pypy packages that would make use
>> of it. Am I missing something?
> No, the the version is not wrong. pypy tracks CPython’s development
> (features, standard library) and thus implements a certain “Python
> version”, in this case 3.6, even though pypy’s own version says
> something else. But, as you said, pypy in Guix is not really useful
> anyway right now without any packages.
>
> (That was the idea of the symlinks by the way, providing pypy as a
> drop-in replacement for python-wrapper in python-build-system.)
>
> Cheers,
> Lars
Thanks for taking the time to explain! It makes sense, but I'd need to
educate myself better about how Pypy can be used along our current
Python version library (as in, does it require its own .pyc files? Does
it dismiss CPython's produced ones? Can we not point it to consume our
existing Python libraries (it's different sys.path suggest differently),
etc.)
Maxim
^ permalink raw reply [flat|nested] 14+ messages in thread
* [bug#44656] [PATCH] Upgrade pypy3
2021-08-06 14:22 ` Maxim Cournoyer
@ 2021-08-06 14:43 ` Lars-Dominik Braun
2021-08-07 1:29 ` Maxim Cournoyer
0 siblings, 1 reply; 14+ messages in thread
From: Lars-Dominik Braun @ 2021-08-06 14:43 UTC (permalink / raw)
To: Maxim Cournoyer; +Cc: 44656
Hello Maxim,
that’s answered quickly:
> does it require its own .pyc files?
Yes, .pyc files are implementation-dependent, as their name suggests
(module.implementation-version.pyc).
> Does it dismiss CPython's produced ones?
It won’t look at them.
> Can we not point it to consume our existing Python libraries (it's
> different sys.path suggest differently), etc.)
Technically plain Python modules should “just work”, but C extensions
must be recompiled, so usually it’s safer to run setup.py using PyPy and
install into a PyPy-specific site-modules.
Cheers,
Lars
^ permalink raw reply [flat|nested] 14+ messages in thread
* [bug#44656] [PATCH] Upgrade pypy3
2021-08-06 14:43 ` Lars-Dominik Braun
@ 2021-08-07 1:29 ` Maxim Cournoyer
2021-08-07 6:59 ` Lars-Dominik Braun
0 siblings, 1 reply; 14+ messages in thread
From: Maxim Cournoyer @ 2021-08-07 1:29 UTC (permalink / raw)
To: Lars-Dominik Braun; +Cc: 44656
Hi Lars-Dominik,
Lars-Dominik Braun <lars@6xq.net> writes:
> Hello Maxim,
>
> that’s answered quickly:
>
>> does it require its own .pyc files?
> Yes, .pyc files are implementation-dependent, as their name suggests
> (module.implementation-version.pyc).
>
>> Does it dismiss CPython's produced ones?
> It won’t look at them.
>
>> Can we not point it to consume our existing Python libraries (it's
>> different sys.path suggest differently), etc.)
> Technically plain Python modules should “just work”, but C extensions
> must be recompiled, so usually it’s safer to run setup.py using PyPy and
> install into a PyPy-specific site-modules.
Thanks for the insights.
I guess one way it could work was if we had some transformation option
to rebuild the Python collection against Pypy (do we have something to
do this already?).
Thanks,
Maxim
^ permalink raw reply [flat|nested] 14+ messages in thread
* [bug#44656] [PATCH] Upgrade pypy3
2021-08-07 1:29 ` Maxim Cournoyer
@ 2021-08-07 6:59 ` Lars-Dominik Braun
2021-08-20 10:39 ` zimoun
0 siblings, 1 reply; 14+ messages in thread
From: Lars-Dominik Braun @ 2021-08-07 6:59 UTC (permalink / raw)
To: Maxim Cournoyer; +Cc: 44656
Hi Maxim,
> I guess one way it could work was if we had some transformation option
> to rebuild the Python collection against Pypy (do we have something to
> do this already?).
yes, there is package-with-explicit-python in (guix build-system
python), but you’ll need transform every single existing Python package
with it to make it usable with PyPy (just like Python 2). There’s also
--with-input=python=pypy3, but I think it will rewrite too much (i.e.
not just packages using python-build-system, but *every* package in the
graph using Python in any way).
Cheers,
Lars
^ permalink raw reply [flat|nested] 14+ messages in thread
* [bug#44656] [PATCH] Upgrade pypy3
2021-08-07 6:59 ` Lars-Dominik Braun
@ 2021-08-20 10:39 ` zimoun
2021-08-20 14:25 ` Lars-Dominik Braun
0 siblings, 1 reply; 14+ messages in thread
From: zimoun @ 2021-08-20 10:39 UTC (permalink / raw)
To: Lars-Dominik Braun, Maxim Cournoyer; +Cc: 44656
Hi,
On Sat, 07 Aug 2021 at 08:59, Lars-Dominik Braun <lars@6xq.net> wrote:
> Hi Maxim,
>
>> I guess one way it could work was if we had some transformation option
>> to rebuild the Python collection against Pypy (do we have something to
>> do this already?).
> yes, there is package-with-explicit-python in (guix build-system
> python), but you’ll need transform every single existing Python package
> with it to make it usable with PyPy (just like Python 2). There’s also
What do you mean? Using ’package-with-explicit-python’ on a package
will replace the Python used by the ’python-build-system’ for all the
dependencies. Modulo the issues what C extension, if one wants the
Python packages bytecompiled with PyPy, using a manifest containing:
--8<---------------cut here---------------start------------->8---
(define package-with-pypy
(package-with-explicit-python (delay pypy3
"python-" "pypy-")))
(packages->manifest
(map
(compose package-with-pypy specification->package)
(list "python-foo" "python-bar")))
--8<---------------cut here---------------end--------------->8---
would be the somehow thing I would do. Well, something in the spirit of
«Defining Package Vaiants». ;-)
<https://guix.gnu.org/manual/devel/en/guix.html#Defining-Package-Variants>
However…
> --with-input=python=pypy3, but I think it will rewrite too much (i.e.
> not just packages using python-build-system, but *every* package in the
> graph using Python in any way).
…similarly as replacing the Emacs VM [0], I guess some packages
will be broken. Well, for instance if some packages use
’python-minimal’ instead of ’python’, the replacement will not happen.
Last, is it an issue to rewrite too much?
Cheers,
simon
0: <http://issues.guix.gnu.org/41732#14>
^ permalink raw reply [flat|nested] 14+ messages in thread
* [bug#44656] [PATCH] Upgrade pypy3
2021-08-20 10:39 ` zimoun
@ 2021-08-20 14:25 ` Lars-Dominik Braun
0 siblings, 0 replies; 14+ messages in thread
From: Lars-Dominik Braun @ 2021-08-20 14:25 UTC (permalink / raw)
To: zimoun; +Cc: 44656, Maxim Cournoyer
Hi zimoun,
> What do you mean? Using ’package-with-explicit-python’ on a package
> will replace the Python used by the ’python-build-system’ for all the
> dependencies. Modulo the issues what C extension, if one wants the
> Python packages bytecompiled with PyPy, using a manifest containing:
true, that’s what I meant and why I mentioned Python 2 (we used to do it
for every single Python 2 package, so they are installable on their own).
> …similarly as replacing the Emacs VM [0], I guess some packages
> will be broken. Well, for instance if some packages use
> ’python-minimal’ instead of ’python’, the replacement will not happen.
> Last, is it an issue to rewrite too much?
Yes, rewriting too much is a problem, because Python is used alot you
end up rebuilding everything, starting from the bootstrap binaries. And
then it’s not even sure you’ll be able to build all the way up to the
package you actually want, because our Python and PyPy packages actually
implement different versions of the Python language/standard library
(3.8 vs 3.7).
Cheers,
Lars
^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2021-08-20 14:26 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-11-15 12:39 [bug#44656] [PATCH] Upgrade pypy3 Lars-Dominik Braun
2020-11-15 19:33 ` Christopher Baines
2020-11-15 19:55 ` Lars-Dominik Braun
2020-11-16 17:26 ` Christopher Baines
2020-11-17 9:38 ` Lars-Dominik Braun
2021-08-03 21:39 ` Maxim Cournoyer
2021-08-05 16:54 ` bug#44656: " Maxim Cournoyer
2021-08-06 7:30 ` [bug#44656] " Lars-Dominik Braun
2021-08-06 14:22 ` Maxim Cournoyer
2021-08-06 14:43 ` Lars-Dominik Braun
2021-08-07 1:29 ` Maxim Cournoyer
2021-08-07 6:59 ` Lars-Dominik Braun
2021-08-20 10:39 ` zimoun
2021-08-20 14:25 ` Lars-Dominik Braun
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).