[bug#46848] [PATCHES] [core-updates] PEP 517 python-build-system

[Lars-Dominik Braun <lars@6xq.net>]

[-- Attachment #1: Type: text/plain, Size: 82 bytes --]

Hi,

I rebased my changes on top of the current core-updates HEAD.

Cheers,
Lars


[-- Attachment #2: 0001-build-python-Handle-missing-setuptools-in-sanity-che.patch --]
[-- Type: text/x-diff, Size: 1103 bytes --]

From 9d9c417fba869913366a12c89b7309ecc402777d Mon Sep 17 00:00:00 2001
From: Lars-Dominik Braun <lars@6xq.net>
Date: Fri, 19 Feb 2021 17:22:35 +0100
Subject: [PATCH 01/14] build/python: Handle missing setuptools in
 sanity-check.py

Just skip testing if required dependencies (setuptools) are not
available.

* gnu/packages/aux-files/python/sanity-check.py: Handle ImportError.
---
 gnu/packages/aux-files/python/sanity-check.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/aux-files/python/sanity-check.py b/gnu/packages/aux-files/python/sanity-check.py
index 83b6d583ca..240155cecc 100644
--- a/gnu/packages/aux-files/python/sanity-check.py
+++ b/gnu/packages/aux-files/python/sanity-check.py
@@ -19,9 +19,13 @@
 
 from __future__ import print_function  # Python 2 support.
 import importlib
-import pkg_resources
 import sys
 import traceback
+try:
+    import pkg_resources
+except ImportError:
+    print('Warning: Skipping, because python-setuptools are not available.')
+    sys.exit(0)
 
 try:
     from importlib.machinery import PathFinder
-- 
2.26.3


[-- Attachment #3: 0002-gnu-python-pypa-build-Update-to-0.3.0.patch --]
[-- Type: text/x-diff, Size: 1187 bytes --]

From 5cdfe3f6de27be54eeaa5c507a62319c3783ff1a Mon Sep 17 00:00:00 2001
From: Lars-Dominik Braun <lars@6xq.net>
Date: Sun, 28 Feb 2021 12:50:42 +0100
Subject: [PATCH 02/14] gnu: python-pypa-build: Update to 0.3.0.

* gnu/packages/python-build.scm (python-pypa-build): Update to 0.3.0.
---
 gnu/packages/python-build.scm | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/gnu/packages/python-build.scm b/gnu/packages/python-build.scm
index 59ee91aa9c..25e2f1e60f 100644
--- a/gnu/packages/python-build.scm
+++ b/gnu/packages/python-build.scm
@@ -136,13 +136,13 @@ Language (TOML) configuration files.")
 (define-public python-pypa-build
   (package
     (name "python-pypa-build")
-    (version "0.1.0")
+    (version "0.3.0")
     (source (origin
               (method url-fetch)
               (uri (pypi-uri "build" version))
               (sha256
                (base32
-                "1d6m21lijwm04g50nwgsgj7x3vhblzw7jv05ah8psqgzk20bbch8"))))
+                "1pazq66c35whrqd5b0zcydjvy2rmghi8riljkd67q3bpiln5pf8f"))))
     (build-system python-build-system)
     (arguments
      `(#:tests? #f                      ;to tests in the PyPI release
-- 
2.26.3


[-- Attachment #4: 0003-gnu-python-wheel-Install-entrypoint-scripts.patch --]
[-- Type: text/x-diff, Size: 1651 bytes --]

From 9516a9e734009f7e38acb37a51f704de9e6198ef Mon Sep 17 00:00:00 2001
From: Lars-Dominik Braun <lars@6xq.net>
Date: Sun, 28 Feb 2021 13:02:15 +0100
Subject: [PATCH 03/14] gnu: python-wheel: Install entrypoint scripts

* gnu/packages/python-build.scm (pythont-wheel) [arguments]: Add phase
'patch-enable-entrypoints.
---
 gnu/packages/python-build.scm | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)

diff --git a/gnu/packages/python-build.scm b/gnu/packages/python-build.scm
index 25e2f1e60f..bbd273e5de 100644
--- a/gnu/packages/python-build.scm
+++ b/gnu/packages/python-build.scm
@@ -49,10 +49,17 @@
           "0ii6f34rvpjg3nmw4bc2h7fhdsy38y1h93hghncfs5akfrldmj8h"))))
     (build-system python-build-system)
     (arguments
-     ;; FIXME: The test suite runs "python setup.py bdist_wheel", which in turn
-     ;; fails to find the newly-built bdist_wheel library, even though it is
-     ;; available on PYTHONPATH.  What search path is consulted by setup.py?
-     '(#:tests? #f))
+     `(#:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'patch-enable-entrypoints
+           (lambda _
+             ;; python-wheel tells setuptools to not install entry point scripts
+             ;; by default. Stop doing that, so wheels built contain all data
+             ;; required.
+             (substitute* "wheel/bdist_wheel.py"
+               (("(install_scripts\\.no_ep = )True" all assignment)
+				(string-append assignment "False")))
+             #t)))))
     (home-page "https://bitbucket.org/pypa/wheel/")
     (synopsis "Format for built Python packages")
     (description
-- 
2.26.3


[-- Attachment #5: 0004-gnu-python-setuptools-Bootstrap-using-itself.patch --]
[-- Type: text/x-diff, Size: 1610 bytes --]

From 19349bf56a88feb5ebc0d24c4f3324e776b2f5ff Mon Sep 17 00:00:00 2001
From: Lars-Dominik Braun <lars@6xq.net>
Date: Sun, 28 Feb 2021 13:05:51 +0100
Subject: [PATCH 04/14] gnu: python-setuptools: Bootstrap using itself

* gnu/packages/python-xyz.scm (python-setuptools) [arguments]: Add phase
setting GUIX_PYTHONPATH to source directory.
---
 gnu/packages/python-xyz.scm | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm
index eebb44b9dc..8f472dea42 100644
--- a/gnu/packages/python-xyz.scm
+++ b/gnu/packages/python-xyz.scm
@@ -1390,7 +1390,18 @@ other machines, such as over the network.")
     ;; FIXME: Tests require pytest, which itself relies on setuptools.
     ;; One could bootstrap with an internal untested setuptools.
     (arguments
-     `(#:tests? #f))
+     `(#:tests? #f
+       #:python ,python-wrapper
+       #:phases (modify-phases %standard-phases
+                  ;; Use this setuptools’ sources to bootstrap themselves.
+                  (add-before 'build 'set-PYTHONPATH
+                    (lambda _
+                      (format #t "current working dir ~s~%" (getcwd))
+                      (setenv "GUIX_PYTHONPATH"
+                              (string-append ".:" (getenv "GUIX_PYTHONPATH")))
+                      #t)))))
+    ;; Not required when not building a wheel
+    ;(propagated-inputs `(("python-wheel" ,python-wheel)))
     (home-page "https://pypi.org/project/setuptools/")
     (synopsis
      "Library designed to facilitate packaging Python projects")
-- 
2.26.3


[-- Attachment #6: 0005-python-build-Switch-to-PEP-517-based-build.patch --]
[-- Type: text/x-diff, Size: 24126 bytes --]

From 0e023d986fe6a093d5f13923b84c54a674fd2278 Mon Sep 17 00:00:00 2001
From: Lars-Dominik Braun <lars@6xq.net>
Date: Sun, 28 Feb 2021 13:08:58 +0100
Subject: [PATCH 05/14] python-build: Switch to PEP 517-based build

* gnu/packages/python-commencement.scm: New file, containing
python-toolchain.
* gnu/local.mk: Add it.
* gnu/packages/python.scm (python): Disable installing bundled
pip/setuptools.
* guix/build/python-build-system.scm: Rewrite using python-pypa-build.
* guix/build-system/python.scm (default-python): Switch to
python-toolchain
(lower): Remove unused parameter.

XXX: rationale
---
 gnu/local.mk                         |   1 +
 gnu/packages/python-commencement.scm | 175 +++++++++++++++++++
 gnu/packages/python.scm              |   2 +-
 guix/build-system/python.scm         |   6 +-
 guix/build/python-build-system.scm   | 249 ++++++++++++++++++---------
 5 files changed, 343 insertions(+), 90 deletions(-)
 create mode 100644 gnu/packages/python-commencement.scm

diff --git a/gnu/local.mk b/gnu/local.mk
index 0e1e83b3d5..e1e41adff7 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -465,6 +465,7 @@ GNU_SYSTEM_MODULES =				\
   %D%/packages/python.scm			\
   %D%/packages/python-build.scm			\
   %D%/packages/python-check.scm			\
+  %D%/packages/python-commencement.scm		\
   %D%/packages/python-compression.scm		\
   %D%/packages/python-crypto.scm		\
   %D%/packages/python-science.scm		\
diff --git a/gnu/packages/python-commencement.scm b/gnu/packages/python-commencement.scm
new file mode 100644
index 0000000000..2ced3079bc
--- /dev/null
+++ b/gnu/packages/python-commencement.scm
@@ -0,0 +1,175 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2014 Andreas Enge <andreas@enge.fr>
+;;; Copyright © 2012 Nikita Karetnikov <nikita@karetnikov.org>
+;;; Copyright © 2014, 2015, 2017 Mark H Weaver <mhw@netris.org>
+;;; Copyright © 2017, 2018, 2019, 2021 Efraim Flashner <efraim@flashner.co.il>
+;;; Copyright © 2018 Tobias Geerinckx-Rice <me@tobias.gr>
+;;; Copyright © 2018, 2019, 2020 Jan (janneke) Nieuwenhuizen <janneke@gnu.org>
+;;; Copyright © 2019, 2020 Marius Bakke <mbakke@fastmail.com>
+;;; Copyright © 2020 Timothy Sample <samplet@ngyro.com>
+;;; Copyright © 2020 Guy Fleury Iteriteka <gfleury@disroot.org>
+;;; Copyright © 2021 Maxim Cournoyer <maxim.cournoyer@gmail.com>
+;;; Copyright © 2021 Lars-Dominik Braun <lars@6xq.net>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (gnu packages python-commencement)
+  #:use-module ((guix licenses) #:prefix license:)
+  #:use-module (guix download)
+  #:use-module (guix packages)
+  #:use-module (guix build-system trivial)
+  #:use-module (guix build-system python)
+  #:use-module (gnu packages)
+  #:use-module (gnu packages python)
+  #:use-module (gnu packages python-build)
+  #:use-module (gnu packages python-xyz)
+  #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-26))
+
+;; Python toolchain and all packages required to bootstrap it.
+
+(define-public python-toolchain
+  (package
+    (name "python-toolchain")
+    (version (package-version python))
+    (source #f)
+    (build-system trivial-build-system)
+    (arguments
+     '(#:modules ((guix build union))
+       #:builder (begin
+                   (use-modules (ice-9 match)
+                                (srfi srfi-1)
+                                (srfi srfi-26)
+                                (guix build union))
+
+                   (let ((out (assoc-ref %outputs "out")))
+                     (union-build out (filter-map (match-lambda
+                                                ((_ . directory) directory))
+                                              %build-inputs))
+                     #t))))
+    (inputs
+     `(("python" ,python-wrapper)
+       ("python-setuptools" ,python-setuptools)
+       ("python-pip" ,python-pip))) ; XXX Maybe virtualenv/venv too? It kind of
+                                    ; defeats the purpose of guix, but is used
+                                    ; alot in local development.
+    (native-search-paths
+     (package-native-search-paths python))
+    (search-paths
+     (package-search-paths python))
+    (license (package-license python)) ; XXX
+    (synopsis "Python toolchain")
+    (description
+     "Python toolchain including Python itself, setuptools and pip.  Use this
+package if you need a fully-fledged Python toolchain instead of just the
+interpreter.")
+    (home-page (package-home-page python))))
+
+;; Python 3 toolchain for python-build-system. We cannot use python-toolchain
+;; here, since we’d need to bootstrap python-pip somehow.
+(define-public python-toolchain-for-build
+  (package
+    (inherit python-toolchain)
+    (name "python-toolchain-for-build")
+    (inputs
+      `(("python" ,python-wrapper)
+        ("python-setuptools" ,python-setuptools)
+        ("python-pypa-build" ,python-pypa-build-from-setuptools)))))
+
+;; Python 3 toolchain to bootstrap python-pypa-build
+(define-public python-toolchain-only-setuptools
+  (package
+    (inherit python-toolchain)
+    (name "python-toolchain-only-setuptools")
+    (inputs
+      `(("python" ,python-wrapper)
+        ("python-setuptools" ,python-setuptools)))))
+
+(define-public python-pypa-build-from-setuptools
+  (package
+	(inherit python-pypa-build)
+    (name "python-pypa-build-from-setuptools")
+    (arguments
+     `(#:tests? #f
+       #:python ,python-toolchain-only-setuptools))
+    (propagated-inputs
+      `(("python-pep517" ,python-pep517-from-setuptools)
+        ("python-packaging" ,python-packaging-from-setuptools)))))
+
+(define-public python-pep517-from-setuptools
+  (package
+	(inherit python-pep517-bootstrap)
+    (name "python-pep517-from-setuptools")
+    (arguments
+     `(#:tests? #f
+       #:python ,python-toolchain-only-setuptools
+       #:phases (modify-phases %standard-phases
+         (add-after 'unpack 'patch
+           (lambda _
+             (substitute* "setup.py"
+               (("distutils\\.core") "setuptools"))
+             #t)))))
+    (propagated-inputs
+     `(("python-toml" ,python-toml-from-setuptools)
+       ("python-wheel" ,python-wheel-from-setuptools)))
+	;; Drop cyclic dependency.
+	(native-inputs '())))
+
+(define-public python-toml-from-setuptools
+  (package
+    (inherit python-toml)
+    (arguments
+     `(#:tests? #f
+       #:python ,python-toolchain-only-setuptools
+       ,@(package-arguments python-toml)))))
+
+(define-public python-packaging-from-setuptools
+  (package
+    (inherit python-packaging-bootstrap)
+    (name "python-packaging-from-setuptools")
+    (arguments
+     `(#:python ,python-toolchain-only-setuptools
+       ,@(package-arguments python-packaging-bootstrap)))
+    (propagated-inputs
+     `(("python-pyparsing" ,python-pyparsing-from-setuptools)
+       ("python-six" ,python-six-from-setuptools)))))
+
+(define-public python-pyparsing-from-setuptools
+  (package
+    (inherit python-pyparsing)
+    (name "python-pyparsing-from-setuptools")
+    (arguments
+     `(#:tests? #f
+       #:python ,python-toolchain-only-setuptools
+       ,@(package-arguments python-pyparsing)))))
+
+(define-public python-six-from-setuptools
+  (package
+    (inherit python-six-bootstrap)
+    (name "python-six-from-setuptools")
+    (arguments
+     `(#:python ,python-toolchain-only-setuptools
+       ,@(package-arguments python-six-bootstrap)))))
+
+(define-public python-wheel-from-setuptools
+  (package
+    (inherit python-wheel)
+    (name "python-wheel-from-setuptools")
+    (arguments
+     `(#:python ,python-toolchain-only-setuptools
+       ,@(package-arguments python-wheel)))))
+
diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index b2ec486d7a..38a1ebe49f 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -183,7 +183,7 @@
        (list "--enable-shared"          ;allow embedding
              "--with-system-expat"      ;for XML support
              "--with-system-ffi"        ;build ctypes
-             "--with-ensurepip=install" ;install pip and setuptools
+             "--with-ensurepip=no"      ;do not install pip and setuptools
              "--enable-unicode=ucs4"
 
              ;; Prevent the installed _sysconfigdata.py from retaining a reference
diff --git a/guix/build-system/python.scm b/guix/build-system/python.scm
index efade6f74b..4a23b42628 100644
--- a/guix/build-system/python.scm
+++ b/guix/build-system/python.scm
@@ -67,8 +67,8 @@ extension, such as '.tar.gz'."
 (define (default-python)
   "Return the default Python package."
   ;; Lazily resolve the binding to avoid a circular dependency.
-  (let ((python (resolve-interface '(gnu packages python))))
-    (module-ref python 'python-wrapper)))
+  (let ((python (resolve-interface '(gnu packages python-commencement))))
+    (module-ref python 'python-toolchain-for-build)))
 
 (define (default-python2)
   "Return the default Python 2 package."
@@ -172,9 +172,9 @@ pre-defined variants."
 (define* (python-build name inputs
                        #:key source
                        (tests? #t)
+                       (configure-flags ''())
                        (test-target "test")
                        (use-setuptools? #t)
-                       (configure-flags ''())
                        (phases '%standard-phases)
                        (outputs '("out"))
                        (search-paths '())
diff --git a/guix/build/python-build-system.scm b/guix/build/python-build-system.scm
index 5b1339d14c..fbc90d3655 100644
--- a/guix/build/python-build-system.scm
+++ b/guix/build/python-build-system.scm
@@ -34,6 +34,7 @@
   #:use-module (ice-9 format)
   #:use-module (srfi srfi-1)
   #:use-module (srfi srfi-26)
+  #:use-module (srfi srfi-35)
   #:export (%standard-phases
             add-installed-pythonpath
             site-packages
@@ -108,30 +109,17 @@
 ;; "--single-version-externally-managed" is set, thus the .egg-info directory
 ;; and the scripts defined in entry-points will always be created.
 
+;; Base error type.
+(define-condition-type &python-build-error &error
+  python-build-error?)
 
-(define setuptools-shim
-  ;; Run setup.py with "setuptools" being imported, which will patch
-  ;; "distutils". This is needed for packages using "distutils" instead of
-  ;; "setuptools" since the former does not understand the
-  ;; "--single-version-externally-managed" flag.
-  ;; Python code taken from pip 9.0.1 pip/utils/setuptools_build.py
-  (string-append
-   "import setuptools, tokenize;__file__='setup.py';"
-   "f=getattr(tokenize, 'open', open)(__file__);"
-   "code=f.read().replace('\\r\\n', '\\n');"
-   "f.close();"
-   "exec(compile(code, __file__, 'exec'))"))
-
-(define (call-setuppy command params use-setuptools?)
-  (if (file-exists? "setup.py")
-      (begin
-         (format #t "running \"python setup.py\" with command ~s and parameters ~s~%"
-                command params)
-         (if use-setuptools?
-             (apply invoke "python" "-c" setuptools-shim
-                    command params)
-             (apply invoke "python" "./setup.py" command params)))
-      (error "no setup.py found")))
+;; Raised when 'check cannot find a valid test system in the inputs.
+(define-condition-type &test-system-not-found &python-build-error
+  test-system-not-found?)
+
+;; Raised when multiple wheels are created by 'build.
+(define-condition-type &cannot-extract-multiple-wheels &python-build-error
+  cannot-extract-multiple-wheels?)
 
 (define* (sanity-check #:key tests? inputs outputs #:allow-other-keys)
   "Ensure packages depending on this package via setuptools work properly,
@@ -142,23 +130,51 @@ without errors."
     (with-directory-excursion "/tmp"
       (invoke "python" sanity-check.py (site-packages inputs outputs)))))
 
-(define* (build #:key use-setuptools? #:allow-other-keys)
+(define* (build #:key outputs #:allow-other-keys)
   "Build a given Python package."
-  (call-setuppy "build" '() use-setuptools?)
+
+  (define pyproject-build (which "pyproject-build"))
+
+  (define (build-pep517)
+    ;; XXX: should probably use a different path, outside of source directory,
+    ;; maybe secondary output “wheel”?
+    (mkdir-p "dist")
+    (invoke pyproject-build "--outdir" "dist" "--no-isolation" "--wheel" "."))
+
+      ;; XXX Would be nice, if we could use bdist_wheel here to remove extra
+      ;; code path in 'install, but that depends on python-wheel.
+  (define (build-setuptools)
+    (invoke "python" "setup.py" "build"))
+
+  (if pyproject-build
+    (build-pep517)
+    (build-setuptools))
   #t)
 
-(define* (check #:key tests? test-target use-setuptools? #:allow-other-keys)
+(define* (check #:key inputs outputs tests? #:allow-other-keys)
   "Run the test suite of a given Python package."
   (if tests?
-      ;; Running `setup.py test` creates an additional .egg-info directory in
-      ;; build/lib in some cases, e.g. if the source is in a sub-directory
-      ;; (given with `package_dir`). This will by copied to the output, too,
-      ;; so we need to remove.
-      (let ((before (find-files "build" "\\.egg-info$" #:directories? #t)))
-        (call-setuppy test-target '() use-setuptools?)
-        (let* ((after (find-files "build" "\\.egg-info$" #:directories? #t))
-               (inter (lset-difference string=? after before)))
-          (for-each delete-file-recursively inter)))
+    ;; Unfortunately with PEP 517 there is no common method to specify test
+    ;; systems. Guess test system based on inputs instead.
+    (let ((pytest (which "pytest"))
+            (have-setup-py (file-exists? "setup.py")))
+        ;; Prefer pytest
+        ;; XXX: support nose
+        (cond
+          (pytest
+            (begin
+              (format #t "using pytest~%")
+              (invoke pytest "-vv"))) ; XXX: support skipping tests based on name/extra arguments?
+          ;; But fall back to setup.py, which should work for most
+          ;; packages. XXX: would be nice not to depend on setup.py here? fails
+          ;; more often than not to find any tests at all. Maybe we can run
+          ;; `python -m unittest`?
+          (have-setup-py
+            (begin
+              (format #t "using setup.py~%")
+                (invoke "python" "setup.py" "test" "-v")))
+          ;; The developer should explicitly disable tests in this case.
+          (#t (raise (condition (&test-system-not-found))))))
       (format #t "test suite not run~%"))
   #t)
 
@@ -195,31 +211,109 @@ running checks after installing the package."
                                 "/bin:"
                                 (getenv "PATH"))))
 
-(define* (install #:key inputs outputs (configure-flags '()) use-setuptools?
-                  #:allow-other-keys)
-  "Install a given Python package."
-  (let* ((out (python-output outputs))
-         (python (assoc-ref inputs "python"))
-         (major-minor (map string->number
-                           (take (string-split (python-version python) #\.) 2)))
-         (<3.7? (match major-minor
-                   ((major minor)
-                    (or (< major 3) (and (= major 3) (< minor 7))))))
-         (params (append (list (string-append "--prefix=" out)
-                               "--no-compile")
-                         (if use-setuptools?
-                             ;; distutils does not accept these flags
-                             (list "--single-version-externally-managed"
-                                   "--root=/")
-                             '())
-                         configure-flags)))
-    (call-setuppy "install" params use-setuptools?)
-    ;; Rather than produce potentially non-reproducible .pyc files on Pythons
-    ;; older than 3.7, whose 'compileall' module lacks the
-    ;; '--invalidation-mode' option, do not generate any.
-    (unless <3.7?
-      (invoke "python" "-m" "compileall" "--invalidation-mode=unchecked-hash"
-              out))))
+(define* (install #:key inputs outputs (configure-flags '()) #:allow-other-keys)
+  "Install a wheel file according to PEP 427"
+  ;; See https://www.python.org/dev/peps/pep-0427/#installing-a-wheel-distribution-1-0-py32-none-any-whl
+  (let* ((site-dir (site-packages inputs outputs))
+         (out (assoc-ref outputs "out")))
+    (define (extract file)
+      "Extract wheel (ZIP file) into site-packages directory"
+      ;; Use Python’s zipfile to avoid extra dependency
+      (invoke "python" "-m" "zipfile" "-e" file site-dir))
+
+    (define python-hashbang
+      (string-append "#!" (assoc-ref inputs "python") "/bin/python"))
+
+    (define (move-data source destination)
+      (mkdir-p (dirname destination))
+      (rename-file source destination))
+
+    (define (move-script source destination)
+      "Move executable script file from .data/scripts to out/bin and replace
+temporary hashbang"
+	  (move-data source destination)
+      ;; ZIP does not save/restore permissions, make executable
+      ;; XXX: might not be a file, but directory with subdirectories
+      (chmod destination #o755)
+      (substitute* destination (("#!python") python-hashbang)))
+
+    ;; Python’s distutils.command.install defines this mapping from source to
+    ;; destination mapping.
+    (define install-schemes
+      `(("scripts" "bin" ,move-script)
+        ;; XXX: Why does Python not use share/ here?
+        ("data" "share" ,move-data)))
+
+    (define (expand-data-directory directory)
+      "Move files from all .data subdirectories to their respective
+destinations."
+      (for-each
+        (match-lambda ((source destination function)
+          (let ((source-path (string-append directory "/" source))
+                (destination-path (string-append out "/" destination)))
+            (when (file-exists? source-path)
+              (begin
+                ;; This assumes only files exist in the scripts/ directory.
+                (for-each
+                  (lambda (file)
+                    (apply
+                      function
+                      (list
+                        (string-append source-path "/" file)
+                        (string-append destination-path "/" file))))
+                  (scandir source-path (negate (cut member <> '("." "..")))))
+                (rmdir source-path))))))
+        install-schemes))
+    
+  (define pyproject-build (which "pyproject-build"))
+
+  (define (list-directories base predicate)
+    ;; Cannot use find-files here, because it’s recursive.
+    (scandir
+      base
+      (lambda (name)
+        (let ((stat (lstat (string-append base "/" name))))
+        (and
+          (not (member name '("." "..")))
+          (eq? (stat:type stat) 'directory)
+          (predicate name stat))))))
+
+  (define (install-pep517)
+    "Install a wheel generated by a PEP 517-compatible builder."
+    (let ((wheels (find-files "dist" "\\.whl$"))) ; XXX: do not recurse
+      (when (> (length wheels) 1) ; This code does not support multiple wheels
+                                  ; yet, because their outputs would have to be
+                                  ; merged properly.
+        (raise (condition (&cannot-extract-multiple-wheels))))
+      (for-each extract wheels))
+    (let ((datadirs (map
+					  (cut string-append site-dir "/" <>)
+					  (list-directories site-dir (file-name-predicate "\\.data$")))))
+      (for-each (lambda (directory)
+                  (expand-data-directory directory)
+                  (rmdir directory))
+                datadirs)))
+
+    (define (install-setuptools)
+      "Install using setuptools."
+      (let ((out (assoc-ref outputs "out")))
+        (invoke "python" "setup.py"
+				"install"
+				"--prefix" out
+				"--single-version-externally-managed"
+				"--root=/")))
+
+    (if pyproject-build
+      (install-pep517)
+      (install-setuptools))
+    #t))
+
+(define* (compile-bytecode #:key inputs outputs (configure-flags '()) #:allow-other-keys)
+  "Compile installed byte-code in site-packages."
+  (let ((site-dir (site-packages inputs outputs)))
+    (invoke "python" "-m" "compileall" site-dir)
+    ;; XXX: We could compile with -O and -OO too here, at the cost of more space.
+    #t))
 
 (define* (wrap #:key inputs outputs #:allow-other-keys)
   (define (list-of-files dir)
@@ -243,29 +337,12 @@ running checks after installing the package."
                             files)))
               bindirs)))
 
-(define* (rename-pth-file #:key name inputs outputs #:allow-other-keys)
-  "Rename easy-install.pth to NAME.pth to avoid conflicts between packages
-installed with setuptools."
-  ;; Even if the "easy-install.pth" is not longer created, we kept this phase.
-  ;; There still may be packages creating an "easy-install.pth" manually for
-  ;; some good reason.
-  (let* ((site-packages (site-packages inputs outputs))
-         (easy-install-pth (string-append site-packages "/easy-install.pth"))
-         (new-pth (string-append site-packages "/" name ".pth")))
-    (when (file-exists? easy-install-pth)
-      (rename-file easy-install-pth new-pth))))
-
-(define* (ensure-no-mtimes-pre-1980 #:rest _)
-  "Ensure that there are no mtimes before 1980-01-02 in the source tree."
-  ;; Rationale: patch-and-repack creates tarballs with timestamps at the POSIX
-  ;; epoch, 1970-01-01 UTC.  This causes problems with Python packages,
-  ;; because Python eggs are ZIP files, and the ZIP format does not support
-  ;; timestamps before 1980.
-  (let ((early-1980 315619200))  ; 1980-01-02 UTC
-    (ftw "." (lambda (file stat flag)
-               (unless (<= early-1980 (stat:mtime stat))
-                 (utime file early-1980 early-1980))
-               #t))))
+(define* (set-SOURCE-DATE-EPOCH #:rest _)
+  "Set the 'SOURCE_DATE_EPOCH' environment variable.  This is used by tools
+that incorporate timestamps as a way to tell them to use a fixed timestamp.
+See https://reproducible-builds.org/specs/source-date-epoch/."
+  (setenv "SOURCE_DATE_EPOCH" "315619200") ;; python-wheel respects this variable and sets pre-1980 times on files in zip files, which is unsupported
+  #t)
 
 (define* (enable-bytecode-determinism #:rest _)
   "Improve determinism of pyc files."
@@ -292,11 +369,11 @@ by Cython."
   ;; prefix directory.  The check phase is moved after the installation phase
   ;; to ease testing the built package.
   (modify-phases gnu:%standard-phases
-    (add-after 'unpack 'ensure-no-mtimes-pre-1980 ensure-no-mtimes-pre-1980)
-    (add-after 'ensure-no-mtimes-pre-1980 'enable-bytecode-determinism
+    (add-after 'unpack 'enable-bytecode-determinism
       enable-bytecode-determinism)
     (add-after 'enable-bytecode-determinism 'ensure-no-cythonized-files
       ensure-no-cythonized-files)
+    (replace 'set-SOURCE-DATE-EPOCH set-SOURCE-DATE-EPOCH)
     (delete 'bootstrap)
     (delete 'configure)                 ;not needed
     (replace 'build build)
@@ -308,7 +385,7 @@ by Cython."
     (add-after 'add-install-to-path 'wrap wrap)
     (add-after 'wrap 'check check)
     (add-after 'check 'sanity-check sanity-check)
-    (add-before 'strip 'rename-pth-file rename-pth-file)))
+    (add-before 'check 'compile-bytecode compile-bytecode)))
 
 (define* (python-build #:key inputs (phases %standard-phases)
                        #:allow-other-keys #:rest args)
-- 
2.26.3


[-- Attachment #7: 0006-gnu-Add-python-pytoml.patch --]
[-- Type: text/x-diff, Size: 1325 bytes --]

From 881a8e817a1fe21a39bcfdc3e5be2d2ac7760cb9 Mon Sep 17 00:00:00 2001
From: Lars-Dominik Braun <lars@6xq.net>
Date: Sun, 28 Feb 2021 13:17:10 +0100
Subject: [PATCH 06/14] gnu: Add python-pytoml.

* gnu/packages/python-build.scm (python-pytoml): Add new variable.
---
 gnu/packages/python-build.scm | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/gnu/packages/python-build.scm b/gnu/packages/python-build.scm
index bbd273e5de..238fc8b3a5 100644
--- a/gnu/packages/python-build.scm
+++ b/gnu/packages/python-build.scm
@@ -191,3 +191,20 @@ implementation developed for Poetry.  This project is intended to be
 a light weight, fully compliant, self-contained package allowing PEP 517
 compatible build front-ends to build Poetry managed projects.")
     (license license:expat)))
+
+(define-public python-pytoml
+  (package
+    (name "python-pytoml")
+    (version "0.1.21")
+    (source
+      (origin
+        (method url-fetch)
+        (uri (pypi-uri "pytoml" version))
+        (sha256
+          (base32
+            "1rv1byiw82k7mj6aprcrqi2vdabs801y97xhfnrz7kxds34ggv4f"))))
+    (build-system python-build-system)
+    (home-page "https://github.com/avakar/pytoml")
+    (synopsis "A parser for TOML-0.4.0")
+    (description "A parser for TOML-0.4.0")
+    (license license:expat)))
-- 
2.26.3


[-- Attachment #8: 0007-gnu-Add-python-flit-core.patch --]
[-- Type: text/x-diff, Size: 1643 bytes --]

From 545fa78c3a42f5cc22ccc9ea70e0ff1f9bcebe5c Mon Sep 17 00:00:00 2001
From: Lars-Dominik Braun <lars@6xq.net>
Date: Sun, 28 Feb 2021 13:18:17 +0100
Subject: [PATCH 07/14] gnu: Add python-flit-core.

* gnu/packages/python-build.scm (python-flit-core): New variable.
---
 gnu/packages/python-build.scm | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/gnu/packages/python-build.scm b/gnu/packages/python-build.scm
index 238fc8b3a5..8940e6490b 100644
--- a/gnu/packages/python-build.scm
+++ b/gnu/packages/python-build.scm
@@ -208,3 +208,30 @@ compatible build front-ends to build Poetry managed projects.")
     (synopsis "A parser for TOML-0.4.0")
     (description "A parser for TOML-0.4.0")
     (license license:expat)))
+
+(define-public python-flit-core
+  (package
+    (name "python-flit-core")
+    (version "3.0.0")
+    (source
+      (origin
+        (method url-fetch)
+        (uri (pypi-uri "flit_core" version))
+        (sha256
+          (base32
+            "0bbw84r33gwi0xyp7m8dzp2dzpjs4harj3l5wrbxkmp2awh0ard4"))))
+    (build-system python-build-system)
+    (arguments
+     `(;; No tests.
+       #:tests? #f))
+    (propagated-inputs
+      `(("python-pytoml" ,python-pytoml)))
+    (home-page "https://github.com/takluyver/flit")
+    (synopsis
+      "Simplified packaging of Python modules, distribution-building parts")
+    (description
+      "Flit is a simple way to put Python packages and modules on PyPI.  It
+tries to require less thought about packaging and help you avoid common
+mistakes.  Distribution-building parts of Flit.")
+    (license license:bsd-3)))
+
-- 
2.26.3


[-- Attachment #9: 0008-gnu-python-pep517-bootstrap-Build-using-flit-core.patch --]
[-- Type: text/x-diff, Size: 1596 bytes --]

From 4ffe19ccc9b6d077c811cfbdd715d10a79730e8f Mon Sep 17 00:00:00 2001
From: Lars-Dominik Braun <lars@6xq.net>
Date: Sun, 28 Feb 2021 13:20:48 +0100
Subject: [PATCH 08/14] gnu: python-pep517-bootstrap: Build using flit-core.

* gnu/packages/python-build.scm (python-pep517-bootstrap) [arguments]:
Relax dependency on flit-core version.
[native-inputs]: Add flit-core.
---
 gnu/packages/python-build.scm | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/python-build.scm b/gnu/packages/python-build.scm
index 8940e6490b..c53d49c287 100644
--- a/gnu/packages/python-build.scm
+++ b/gnu/packages/python-build.scm
@@ -128,10 +128,21 @@ Language (TOML) configuration files.")
           "0zqidxah03qpnp6zkg3zd1kmd5f79hhdsfmlc0cldaniy80qddxf"))))
      (build-system python-build-system)
      (arguments
-      `(#:tests? #f))                     ;to avoid circular dependencies
+     `(#:tests? #f ; To avoid circular dependencies.
+       #:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'relax-dependency
+           (lambda _
+             (substitute* "pyproject.toml"
+               (("flit_core >=2,<3")
+                "flit_core >=2,<4"))
+             #t)))))
      (propagated-inputs
       `(("python-toml" ,python-toml)
         ("python-wheel" ,python-wheel)))
+     (native-inputs
+     `(;; Build system.
+       ("python-flit-core" ,python-flit-core)))
      (home-page "https://github.com/pypa/pep517")
      (synopsis "Wrappers to build Python packages using PEP 517 hooks")
      (description
-- 
2.26.3


[-- Attachment #10: 0009-gnu-python-iniconfig-Add-missing-build-input.patch --]
[-- Type: text/x-diff, Size: 992 bytes --]

From f1eea04f204dc0fef64675ac2605f42535aa0815 Mon Sep 17 00:00:00 2001
From: Lars-Dominik Braun <lars@6xq.net>
Date: Sun, 28 Feb 2021 13:23:53 +0100
Subject: [PATCH 09/14] gnu: python-iniconfig: Add missing build input.

* gnu/packages/python-xyz.scm (python-iniconfig) [native-inputs] Add
setuptools-scm.
---
 gnu/packages/python-xyz.scm | 1 +
 1 file changed, 1 insertion(+)

diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm
index 8f472dea42..47466d31f0 100644
--- a/gnu/packages/python-xyz.scm
+++ b/gnu/packages/python-xyz.scm
@@ -17110,6 +17110,7 @@ in other versions.")
         (base32
          "0ckzngs3scaa1mcfmsi1w40a1l8cxxnncscrxzjjwjyisx8z0fmw"))))
     (build-system python-build-system)
+    (native-inputs `(("python-setuptools-scm" ,python-setuptools-scm)))
     (home-page "https://github.com/RonnyPfannschmidt/iniconfig")
     (synopsis "Simple INI-file parser")
     (description "The @code{iniconfig} package provides a small and simple
-- 
2.26.3


[-- Attachment #11: 0010-gnu-Add-python-u-msgpack.patch --]
[-- Type: text/x-diff, Size: 1705 bytes --]

From 271997517b2b03aec44e8b158c911412caccf7bf Mon Sep 17 00:00:00 2001
From: Lars-Dominik Braun <lars@6xq.net>
Date: Mon, 1 Mar 2021 14:16:07 +0100
Subject: [PATCH 10/14] gnu: Add python-u-msgpack.

* gnu/packages/python-xyz.scm (python-u-msgpack): New variable.

The redundant -python postfix from the original package name has been
removed.
---
 gnu/packages/python-xyz.scm | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm
index 47466d31f0..fdc9bd85b7 100644
--- a/gnu/packages/python-xyz.scm
+++ b/gnu/packages/python-xyz.scm
@@ -25575,3 +25575,27 @@ is the cythonized version of @code{fractions.Fraction}.")
      "@code{pathvalidate} is a Python library to sanitize/validate strings
 representing paths or filenames.")
     (license license:expat)))
+
+(define-public python-u-msgpack
+  (package
+    (name "python-u-msgpack")
+    (version "2.7.1")
+    (source
+      (origin
+        (method url-fetch)
+        (uri (pypi-uri "u-msgpack-python" version))
+        (sha256
+          (base32
+            "0lcmlr7gc4dydpxn6l5bdcq40c3ghf8mv1sjqyj72wdpr8rx9rxp"))))
+    (build-system python-build-system)
+    (home-page
+      "https://github.com/vsergeev/u-msgpack-python")
+    (synopsis
+      "Portable, lightweight MessagePack serializer and deserializer")
+    (description
+      "A portable, lightweight MessagePack serializer and deserializer written
+in pure Python.  u-msgpack-python is fully compliant with the latest MessagePack
+specification. In particular, it supports the new binary, UTF-8 string,
+application-defined ext, and timestamp types.")
+    (license license:expat)))
+
-- 
2.26.3


[-- Attachment #12: 0011-gnu-Add-python-pytest-expect.patch --]
[-- Type: text/x-diff, Size: 1835 bytes --]

From 77f5e2e445e01c3ce7b1c9484ca27ba86b9d5def Mon Sep 17 00:00:00 2001
From: Lars-Dominik Braun <lars@6xq.net>
Date: Mon, 1 Mar 2021 14:20:03 +0100
Subject: [PATCH 11/14] gnu: Add python-pytest-expect.

* gnu/packages/python-check.scm (python-pytest-expect): New variable.
---
 gnu/packages/python-check.scm | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)

diff --git a/gnu/packages/python-check.scm b/gnu/packages/python-check.scm
index ff29f5d7ce..80fb5f704a 100644
--- a/gnu/packages/python-check.scm
+++ b/gnu/packages/python-check.scm
@@ -1515,3 +1515,31 @@ allows one to create a set of tests using @emph{pairwise combinations} method,
 reducing a number of combinations of variables into a lesser set that covers
 most situations.")
     (license license:expat)))
+
+(define-public python-pytest-expect
+  (package
+    (name "python-pytest-expect")
+    (version "1.1.0")
+    (source
+      (origin
+        (method url-fetch)
+        (uri (pypi-uri "pytest-expect" version))
+        (sha256
+          (base32
+            "0iyq3zd1g5ffaz2wv6mskjfn84sfbyh0j209glcrh1s50hkldd1n"))))
+    (build-system python-build-system)
+    (arguments `(#:tests? #f)) ; no tests via pypi
+    (propagated-inputs
+      `(("python-pytest" ,python-pytest) ; package declares this dependency
+        ("python-u-msgpack" ,python-u-msgpack)))
+    (home-page
+      "https://github.com/gsnedders/pytest-expect")
+    (synopsis
+      "Py.test plugin to store test expectations and mark tests based on them")
+    (description
+      "A py.test plugin that stores test expectations by saving the set of
+failing tests, allowing them to be marked as xfail when running them in future.
+The tests expectations are stored such that they can be distributed alongside
+the tests.")
+    (license license:expat)))
+
-- 
2.26.3


[-- Attachment #13: 0012-gnu-python-html5lib-Fix-tests-with-pytest-6.patch --]
[-- Type: text/x-diff, Size: 1899 bytes --]

From 2e2eb3c05f285f3bfc4c990309584eb98503cd91 Mon Sep 17 00:00:00 2001
From: Lars-Dominik Braun <lars@6xq.net>
Date: Mon, 1 Mar 2021 14:23:07 +0100
Subject: [PATCH 12/14] gnu: python-html5lib: Fix tests with pytest 6.

* gnu/packages/python-web.scm (python-html5lib) [source]: Add upstream
patch.
[native-inputs]: Add test dependencies.
[arguments]: Remove unsupported #:test-target.
---
 gnu/packages/python-web.scm | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/python-web.scm b/gnu/packages/python-web.scm
index 8b31368424..8ab783bbb3 100644
--- a/gnu/packages/python-web.scm
+++ b/gnu/packages/python-web.scm
@@ -1079,7 +1079,20 @@ storage.")
         (uri (pypi-uri "html5lib" version))
         (sha256
           (base32
-            "0vqlhk0hgbsfkh7ybmby93xhlx8dq6pr5blf356ka3z2c41b9rdj"))))
+            "0vqlhk0hgbsfkh7ybmby93xhlx8dq6pr5blf356ka3z2c41b9rdj"))
+        (patches
+          (list
+            ;; Adds Pytest 6 support.
+            (origin
+              (method url-fetch)
+              (uri (string-append
+                     "https://github.com/html5lib/"
+                     "html5lib-python/commit/"
+                     "2c19b9899ab3a3e8bd0ca35e5d78544334204169.patch"))
+              (file-name "python-html5lib-support-pytest6.patch")
+              (sha256
+                (base32
+                  "0jg2ry0439q8n7j1mf4p2hdq54i704pq9scv4wwa2pp3cwvb6dvg")))))))
     (build-system python-build-system)
     (propagated-inputs
      `(("python-six" ,python-six)
@@ -1088,6 +1101,9 @@ storage.")
        ("python-chardet" ,python-chardet)))
     (arguments
      `(#:test-target "check"))
+    (native-inputs
+      `(("python-pytest" ,python-pytest)
+        ("python-pytest-expect" ,python-pytest-expect)))
     (home-page
       "https://github.com/html5lib/html5lib-python")
     (synopsis
-- 
2.26.3


[-- Attachment #14: 0013-gnu-python-libxml2-Fix-build.patch --]
[-- Type: text/x-diff, Size: 1145 bytes --]

From 312ee407e5d878da99098753ba24641c2c468451 Mon Sep 17 00:00:00 2001
From: Lars-Dominik Braun <lars@6xq.net>
Date: Mon, 1 Mar 2021 17:18:50 +0100
Subject: [PATCH 13/14] gnu: python-libxml2: Fix build.

* gnu/packages/xml.scm (python-libxml2) [#:phases]: Replace setuptools
and do not exit at the end of setup.py.
---
 gnu/packages/xml.scm | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/xml.scm b/gnu/packages/xml.scm
index 9accd08a7e..9d93d21e27 100644
--- a/gnu/packages/xml.scm
+++ b/gnu/packages/xml.scm
@@ -370,7 +370,10 @@ It uses libxml2 to access the XML files.")
                  (format #f "ROOT = r'~a'" libxml2))
                 ;; For 'iconv.h'.
                 (("/opt/include")
-                 (string-append glibc "/include"))))
+                 (string-append glibc "/include"))
+                (("distutils\\.core") "setuptools")
+                ;; python-pypa-build does not like it if setup.py exits.
+                (("sys\\.exit\\(0\\)") "")))
             #t)))))
     (inputs `(("libxml2" ,libxml2)))
     (synopsis "Python bindings for the libxml2 library")))
-- 
2.26.3


[-- Attachment #15: 0014-gnu-scons-Remove-obsolete-argument.patch --]
[-- Type: text/x-diff, Size: 1090 bytes --]

From ec72099a90b0fc7f6538f578ab3e28411391dacb Mon Sep 17 00:00:00 2001
From: Lars-Dominik Braun <lars@6xq.net>
Date: Mon, 1 Mar 2021 17:21:35 +0100
Subject: [PATCH 14/14] gnu: scons: Remove obsolete argument.

* gnu/packages/python-xyz.scm (scons) [arguments]: Remove #:use-setuptools?
argument.
---
 gnu/packages/python-xyz.scm | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm
index fdc9bd85b7..a61451f8ea 100644
--- a/gnu/packages/python-xyz.scm
+++ b/gnu/packages/python-xyz.scm
@@ -2641,8 +2641,7 @@ and is not compatible with JSON.")
                "1xy8jrwz87y589ihcld4hv7wn122sjbz914xn8h50ww77wbhk8hn"))))
     (build-system python-build-system)
     (arguments
-     `(#:use-setuptools? #f                ; still relies on distutils
-       #:tests? #f                         ; no 'python setup.py test' command
+     `(#:tests? #f                         ; no 'python setup.py test' command
        #:phases
        (modify-phases %standard-phases
          (add-before 'build 'bootstrap
-- 
2.26.3