[bug#45712] [PATCHES] Improve Python package quality

[Lars-Dominik Braun <lars@6xq.net>]

[-- Attachment #1: Type: text/plain, Size: 3439 bytes --]

Hi Hartmut,

> Please but this int a new line - makeing it easier to copy & paste.
> (Leading emptry lines doe nor effect "from __future__ import").
adopted Ricardo’s suggestion here.

> This looks very uncommon (and make my mind hooking on it). Please use
> this, which is more common and less mindbogling ;-)
Done.

> Any reason for converting the req into a string first? IC,
> "`requirements` must be a string". So I'd move the "str()" to the
> function call:
Yes, the string is used in the error message.

>     if group not in {'console_scripts', 'gui_scripts'}:
> Why not gui-scripts?
> If not adding gui-scripts, just use "if group != 'concolse_scrips':"
I wasn’t aware this exists. Added, thanks!

> Does it make sence to try loading the top-level modules *after* the the
> entry-point? Chances are very high that the entry-points implicitly test
> the top-level modules already.
> IMHO it would make more sense to first try the top-level modules, and
> even stop processing if this fails.
True, I reversed the order. Still, I think continuing with the entry
points might be worth, if it points out more (different) errors. For
small packages this might not be the case, but larger ones often only
re-export specific names in the top-level module, thus testing might
reveal more issues.

> Please add a short explanation why there can be unavailable top-level
> modules.
Done.

> While not having antoher idea, I'm not happy with this name. "loadable"
> is not easy to get. (See also below, where this term is used in commit message.)
> top-level-modules-are-importable?
I’m also not happy with the name, but can’t think of a better one right
now. Anyone?

> AFAIKS the packages only differ by some requirements. So I suggest
> using a function to return the packages. This makes it easier to spot
> the actull differences.
> […]
> (mkdir-p "src/dummy")
> (chdir "src")
Done.

> I would strip this down (version is not required AFAIK) and put it one
> line (her assuming you use (format) for creating the code within a function:
Not sure whether it’s optional or not, [1] does not say it is. No harm
in keeping it?

[1] https://setuptools.readthedocs.io/en/latest/references/keywords.html?highlight=version#keywords

> Arguments are not used?
Fixed.

> Any reason for relaxing this? Why not use python-pytest-6 as input?
Yes, our pytest@6 package reports its version as 0.0.0, so this patch is
required with either package. Or we figure out how to fix pytest@6
(works fine with PEP 517 compliant build system).

> Please rephrase into something like
> Do not validate loadability
Done.

> Dosn't this change fix the checks? So this comment would be obsoltes and
> "#:tests #t" can be removed.
Should’ve been #f, sorry, fixed.

> I suggest keeping the old way, as this is will automatically update to
> upstream changes.
Okay, I’ve added another phase that disables the test manually, because
it fails for me every time.

See attached patchset (v2) or git repository for updated patches.

I’ve also rebuilt all 2284 packages depending on python-build-system.
Currently 426 of them fail to build for any reason (not necessarily
caused by this patchset; I’ve seen some unrelated issues). Due to the
large number of packages failing I suggest moving forward with applying
this first batch and then fixing everything else incrementally, unless
some major package is broken (see attached list). WDYT?

Cheers,
Lars


[-- Attachment #2: 0001-build-system-python-Validate-installed-package.patch --]
[-- Type: text/x-diff, Size: 9127 bytes --]

From 69bd0e11b9a054837e1733858490f0aec3830eca Mon Sep 17 00:00:00 2001
From: Lars-Dominik Braun <lars@6xq.net>
Date: Sun, 3 Jan 2021 10:30:29 +0100
Subject: [PATCH v2 01/16] build-system/python: Validate installed package
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Adds a new phase validating usalibity of installed Python packages.

* guix/build/python-build-system.scm (validate-script): Add script.
(validate-loadable): New phase.
(%standard-phases): Use it.
* tests/builders.scm (make-python-dummy): Add test package generator.
(check-build-{success,failure}): Add build helper functions.
(python-dummy-*): Add test packages.
("python-build-system: …"): Add tests.
---
 guix/build/python-build-system.scm |  80 +++++++++++++++++++++
 tests/builders.scm                 | 108 ++++++++++++++++++++++++++++-
 2 files changed, 185 insertions(+), 3 deletions(-)

diff --git a/guix/build/python-build-system.scm b/guix/build/python-build-system.scm
index 09bd8465c8..3c29efea8b 100644
--- a/guix/build/python-build-system.scm
+++ b/guix/build/python-build-system.scm
@@ -148,6 +148,85 @@
       (format #t "test suite not run~%"))
   #t)
 
+(define validate-script
+  "\
+from __future__ import print_function # Python 2 support.
+import pkg_resources, sys, importlib, traceback
+try:
+    from importlib.machinery import PathFinder
+except ImportError:
+    PathFinder = None
+ret = 0
+# Only check site-packages installed by this package, but not dependencies
+# (which pkg_resources.working_set would include). Path supplied via argv.
+ws = pkg_resources.find_distributions(sys.argv[1])
+for dist in ws:
+    print('validating', repr(dist.project_name), dist.location)
+    try:
+        print('...checking requirements: ', end='')
+        req = str(dist.as_requirement())
+        # dist.activate() is not enough to actually check requirements, we have to
+        # .require() it.
+        pkg_resources.require(req)
+        print('OK')
+    except Exception as e:
+        print('ERROR:', req, e)
+        ret = 1
+        continue
+
+    # Try to load top level modules. This should not have any side-effects.
+    try:
+        metalines = dist.get_metadata_lines('top_level.txt')
+    except KeyError:
+		# distutils (i.e. #:use-setuptools? #f) will not install any metadata.
+        print('WARNING: cannot determine top-level modules')
+        continue
+    for name in metalines:
+        # Only available on Python 3.
+        if PathFinder and PathFinder.find_spec(name) is None:
+            # Ignore unavailable modules, often C modules, which were not
+            # installed at the top-level. Cannot use ModuleNotFoundError,
+            # because it is raised by failed imports too.
+            continue
+        try:
+            print('...trying to load module', name, end=': ')
+            importlib.import_module(name)
+            print('OK')
+        except Exception:
+            print('ERROR:')
+            traceback.print_exc(file=sys.stdout)
+            ret = 1
+            continue
+
+    # Try to load entry points of console scripts too, making sure they work. They
+    # should be removed if they don’t. Other groups may not be safe, as they can
+    # depend on optional packages.
+    for group, v in dist.get_entry_map().items():
+       if group not in {'console_scripts', 'gui_scripts'}:
+           continue
+       for name, ep in v.items():
+           try:
+               print('...trying to load endpoint', group, name, end=': ')
+               ep.load()
+               print('OK')
+           except Exception:
+               print('ERROR:')
+               traceback.print_exc(file=sys.stdout)
+               ret = 1
+               continue
+
+sys.exit(ret)")
+
+(define* (validate-loadable #:key tests? inputs outputs #:allow-other-keys)
+  "Ensure packages depending on this package via setuptools work properly,
+their advertised endpoints work and their top level modules are importable
+without errors."
+  (add-installed-pythonpath inputs outputs)
+  ;; Make sure the working directory is empty (i.e. no Python modules in it)
+  (with-directory-excursion "/tmp"
+    (invoke "python" "-c" validate-script (site-packages inputs outputs)))
+  #t)
+
 (define (python-version python)
   (let* ((version     (last (string-split python #\-)))
          (components  (string-split version #\.))
@@ -267,6 +346,7 @@ installed with setuptools."
     (replace 'install install)
     (add-after 'install 'check check)
     (add-after 'install 'wrap wrap)
+    (add-after 'check 'validate-loadable validate-loadable)
     (add-before 'strip 'rename-pth-file rename-pth-file)))
 
 (define* (python-build #:key inputs (phases %standard-phases)
diff --git a/tests/builders.scm b/tests/builders.scm
index fdcf38ded3..929d1a906e 100644
--- a/tests/builders.scm
+++ b/tests/builders.scm
@@ -21,15 +21,15 @@
   #:use-module (guix download)
   #:use-module (guix build-system)
   #:use-module (guix build-system gnu)
+  #:use-module (guix build-system python)
   #:use-module (guix store)
+  #:use-module (guix monads)
   #:use-module (guix utils)
   #:use-module (guix base32)
   #:use-module (guix derivations)
   #:use-module (gcrypt hash)
   #:use-module (guix tests)
-  #:use-module ((guix packages)
-                #:select (package?
-                          package-derivation package-native-search-paths))
+  #:use-module (guix packages)
   #:use-module (gnu packages bootstrap)
   #:use-module (ice-9 match)
   #:use-module (srfi srfi-1)
@@ -78,4 +78,106 @@
 (test-assert "gnu-build-system"
   (build-system? gnu-build-system))
 
+\f
+(define* (make-python-dummy name #:key (setup-py-extra "") (init-py "") (use-setuptools? #t))
+  (package
+    (name (string-append "python-dummy-" name))
+    (version "0.1")
+    (source #f) ; source is generated in 'unpack
+    (build-system python-build-system)
+    (arguments
+     `(#:tests? #f
+       #:use-setuptools? ,use-setuptools?
+       #:phases
+       (modify-phases %standard-phases
+         (replace 'unpack
+           (lambda _
+             (mkdir-p "src/dummy")
+             (chdir "src")
+             (with-output-to-file "dummy/__init__.py"
+               (lambda _
+                 (display ,init-py)))
+             (with-output-to-file "setup.py"
+               (lambda _
+                 (format #t "\
+~a
+setup(
+     name='dummy-~a',
+     version='0.1',
+     packages=['dummy'],
+     ~a
+     )"
+                    (if ,use-setuptools?
+                      "from setuptools import setup"
+                      "from distutils.core import setup")
+                    ,name ,setup-py-extra)))
+               #t)))))
+    (home-page #f)
+    (synopsis #f)
+    (description #f)
+    (license #f)))
+
+(define python-dummy-ok
+  (make-python-dummy "ok"))
+
+(define python2-dummy-ok
+  (package-with-python2 python-dummy-ok))
+
+;; distutil won’t install any metadata, so make sure our script does not fail
+;; on a otherwise fine package.
+(define python-dummy-no-setuptools
+  (make-python-dummy
+    "no-setuptools" #:use-setuptools? #f))
+
+(define python2-dummy-no-setuptools
+  (package-with-python2 python-dummy-no-setuptools))
+
+(define python-dummy-fail-requirements
+  (make-python-dummy "fail-requirements"
+    #:setup-py-extra "install_requires=['nonexistent'],"))
+
+(define python2-dummy-fail-requirements
+  (package-with-python2 python-dummy-fail-requirements))
+
+(define python-dummy-fail-import
+  (make-python-dummy "fail-import" #:init-py "import nonexistent"))
+
+(define python2-dummy-fail-import
+  (package-with-python2 python-dummy-fail-import))
+
+(define python-dummy-fail-console-script
+  (make-python-dummy "fail-console-script"
+    #:setup-py-extra (string-append "entry_points={'console_scripts': "
+                                    "['broken = dummy:nonexistent']},")))
+
+(define python2-dummy-fail-console-script
+  (package-with-python2 python-dummy-fail-console-script))
+
+(define (check-build-success store p)
+  (unless store (test-skip 1))
+  (test-assert (string-append "python-build-system: " (package-name p))
+    (let* ((drv (package-derivation store p)))
+      (build-derivations store (list drv)))))
+
+(define (check-build-failure store p)
+  (unless store (test-skip 1))
+  (test-assert (string-append "python-build-system: " (package-name p))
+    (not (false-if-exception (package-derivation store python-dummy-fail-requirements)))))
+
+(with-external-store store
+  (for-each (lambda (p) (check-build-success store p))
+            (list
+              python-dummy-ok
+              python-dummy-no-setuptools
+              python2-dummy-ok
+              python2-dummy-no-setuptools))
+  (for-each (lambda (p) (check-build-failure store p))
+            (list
+              python-dummy-fail-requirements
+              python-dummy-fail-import
+              python-dummy-fail-console-script
+              python2-dummy-fail-requirements
+              python2-dummy-fail-import
+              python2-dummy-fail-console-script)))
+
 (test-end "builders")
-- 
2.26.2


[-- Attachment #3: 0002-gnu-pytest-6-Add-missing-propagated-input.patch --]
[-- Type: text/x-diff, Size: 1375 bytes --]

From e99707fb4cea9db88721192e84044dbdf1b1ca9b Mon Sep 17 00:00:00 2001
From: Lars-Dominik Braun <lars@6xq.net>
Date: Thu, 7 Jan 2021 13:25:56 +0100
Subject: [PATCH v2 02/16] gnu: pytest@6: Add missing propagated-input.

* gnu/packages/check.scm (python-pytest-6) [native-inputs]: Remove
python-iniconfig.
[propagated-inputs]: Move it here.
---
 gnu/packages/check.scm | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/gnu/packages/check.scm b/gnu/packages/check.scm
index 1300f9e1a6..9d1e0b8173 100644
--- a/gnu/packages/check.scm
+++ b/gnu/packages/check.scm
@@ -964,13 +964,13 @@ and many external plugins.")
     (propagated-inputs
      (append (alist-delete "python-py"
                            (package-propagated-inputs python-pytest))
-             `(("python-py" ,python-py-next))))
+             `(("python-py" ,python-py-next)
+               ("python-iniconfig" ,python-iniconfig))))
     (native-inputs
      (append (alist-delete "python-pytest"
                            (package-native-inputs python-pytest))
              `(("python-pytest" ,python-pytest-6-bootstrap)
-               ("python-toml" ,python-toml)
-               ("python-iniconfig" ,python-iniconfig))))))
+               ("python-toml" ,python-toml))))))
 
 ;; Pytest 4.x are the last versions that support Python 2.
 (define-public python2-pytest
-- 
2.26.2


[-- Attachment #4: 0003-gnu-python-pytest-xdist-Add-missing-input-relax-pyte.patch --]
[-- Type: text/x-diff, Size: 2090 bytes --]

From 23077674975a0194f38ed70d0a912ab52c4af3fe Mon Sep 17 00:00:00 2001
From: Lars-Dominik Braun <lars@6xq.net>
Date: Thu, 7 Jan 2021 13:27:55 +0100
Subject: [PATCH v2 03/16] gnu: python-pytest-xdist: Add missing input, relax
 pytest requirement.

* gnu/packages/check.scm: (python-pytest-xdist)
[arguments]: Relax pytest version requirements.
[propagated-inputs]: Add python-pytest-forked.
---
 gnu/packages/check.scm | 25 ++++++++++++++++---------
 1 file changed, 16 insertions(+), 9 deletions(-)

diff --git a/gnu/packages/check.scm b/gnu/packages/check.scm
index 9d1e0b8173..2136b8fc4f 100644
--- a/gnu/packages/check.scm
+++ b/gnu/packages/check.scm
@@ -1216,20 +1216,27 @@ same arguments.")
            #t))))
     (build-system python-build-system)
     (arguments
-     '(#:tests? #f)) ;FIXME: Some tests are failing.
-       ;; #:phases
-       ;; (modify-phases %standard-phases
-       ;;   (delete 'check)
-       ;;   (add-after 'install 'check
-       ;;     (lambda* (#:key inputs outputs #:allow-other-keys)
-       ;;       (add-installed-pythonpath inputs outputs)
-       ;;       (zero? (system* "py.test" "-v")))))
+     '(#:tests? #f ; Lots of tests fail.
+       #:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'patch-setup-py
+           (lambda _
+             ;; Relax pytest requirement.
+             (substitute* "setup.py"
+               (("pytest>=6\\.0\\.0") "pytest"))
+             #t))
+         (replace 'check
+            (lambda* (#:key tests? inputs outputs #:allow-other-keys)
+              (when tests?
+                (add-installed-pythonpath inputs outputs)
+                (invoke "py.test" "-v")))))))
     (native-inputs
      `(("python-setuptools-scm" ,python-setuptools-scm)))
     (propagated-inputs
      `(("python-execnet" ,python-execnet)
        ("python-pytest" ,python-pytest)
-       ("python-py" ,python-py)))
+       ("python-py" ,python-py)
+       ("python-pytest-forked" ,python-pytest-forked)))
     (home-page
      "https://github.com/pytest-dev/pytest-xdist")
     (synopsis
-- 
2.26.2


[-- Attachment #5: 0004-gnu-python-fixtures-bootstrap-Do-not-validate-loadab.patch --]
[-- Type: text/x-diff, Size: 1111 bytes --]

From 478a266a4299a05835bbfc19099a1f0afc318d8a Mon Sep 17 00:00:00 2001
From: Lars-Dominik Braun <lars@6xq.net>
Date: Thu, 7 Jan 2021 13:29:59 +0100
Subject: [PATCH v2 04/16] gnu: python-fixtures-bootstrap: Do not validate
 loadability

* gnu/packages/check.scm (python-fixtures-bootstrap) [arguments]: Delete
'validate-loadable.
---
 gnu/packages/check.scm | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/check.scm b/gnu/packages/check.scm
index 2136b8fc4f..e4761eb66a 100644
--- a/gnu/packages/check.scm
+++ b/gnu/packages/check.scm
@@ -1541,7 +1541,12 @@ protocol.")))
                (base32
                 "1vxj29bzz3rd4pcy51d05wng9q9dh4jq6wx92yklsm7i6h1ddw7w"))))
     (build-system python-build-system)
-    (arguments `(#:tests? #f))
+    (arguments
+      `(#:tests? #f
+        #:phases
+         (modify-phases %standard-phases
+           ;; Package is not loadable on its own at this stage.
+           (delete 'validate-loadable))))
     (propagated-inputs
      `(("python-pbr-minimal" ,python-pbr-minimal)
        ("python-six" ,python-six)))
-- 
2.26.2


[-- Attachment #6: 0005-gnu-python-pytest-pep8-Fix-package.patch --]
[-- Type: text/x-diff, Size: 1518 bytes --]

From ac7d664c3748afb9059ed3d147c13a9033fff45b Mon Sep 17 00:00:00 2001
From: Lars-Dominik Braun <lars@6xq.net>
Date: Thu, 7 Jan 2021 13:35:11 +0100
Subject: [PATCH v2 05/16] gnu: python-pytest-pep8: Fix package.

* gnu/packages/check.scm (python-pytest-pep8) [arguments]: Remove
dependency on pytest-cache and add proper 'check phase.
---
 gnu/packages/check.scm | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/check.scm b/gnu/packages/check.scm
index e4761eb66a..c12cb98a6c 100644
--- a/gnu/packages/check.scm
+++ b/gnu/packages/check.scm
@@ -2043,7 +2043,19 @@ failures.")
                 "06032agzhw1i9d9qlhfblnl3dw5hcyxhagn7b120zhrszbjzfbh3"))))
     (build-system python-build-system)
     (arguments
-     `(#:tests? #f)) ; Fails with recent pytest and pep8. See upstream issues #8 and #12.
+     `(#:tests? #f ; Fails with recent pytest and pep8. See upstream issues #8 and #12.
+       #:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'fix-dependencies
+           (lambda _
+             (substitute* "setup.py"
+               (("'pytest-cache', ") "")) ; Included in recent pytest
+             #t))
+         (replace 'check
+            (lambda* (#:key tests? inputs outputs #:allow-other-keys)
+              (when tests?
+                (add-installed-pythonpath inputs outputs)
+                (invoke "pytest" "-v")))))))
     (native-inputs
      `(("python-pytest" ,python-pytest)))
     (propagated-inputs
-- 
2.26.2


[-- Attachment #7: 0006-gnu-python-pyfakefs-Disable-unreliable-test.patch --]
[-- Type: text/x-diff, Size: 1256 bytes --]

From e76db5088bafd7a3788a80a834d4983e0eed0e6a Mon Sep 17 00:00:00 2001
From: Lars-Dominik Braun <lars@6xq.net>
Date: Mon, 11 Jan 2021 14:44:02 +0100
Subject: [PATCH v2 06/16] gnu: python-pyfakefs: Disable unreliable test

* gnu/packages/check.scm (python-pyfakefs) [arguments]: Add new phase to
skip single test.
---
 gnu/packages/check.scm | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/gnu/packages/check.scm b/gnu/packages/check.scm
index c12cb98a6c..9ddf656324 100644
--- a/gnu/packages/check.scm
+++ b/gnu/packages/check.scm
@@ -2858,6 +2858,16 @@ grew out of the @dfn{Vc} project.")
     (arguments
      `(#:phases
        (modify-phases %standard-phases
+         (add-after 'unpack 'patch-testsuite
+           (lambda _
+             ;; Time difference is larger than expected.
+             (substitute* "pyfakefs/tests/fake_filesystem_unittest_test.py"
+               (("(\\s+)def test_copy_real_file" all indent)
+                (string-append
+                  indent
+                  "@unittest.skip('disabled by guix')\n"
+                  all)))
+             #t))
          ;; The default test suite does not run these extra tests.
          (add-after 'check 'check-pytest-plugin
            (lambda _
-- 
2.26.2


[-- Attachment #8: 0007-gnu-python-slugify-Add-missing-input.patch --]
[-- Type: text/x-diff, Size: 963 bytes --]

From 203cd356f44081a753b1f67ec14ab00ec1bd8520 Mon Sep 17 00:00:00 2001
From: Lars-Dominik Braun <lars@6xq.net>
Date: Thu, 7 Jan 2021 13:48:36 +0100
Subject: [PATCH v2 07/16] gnu: python-slugify: Add missing input.

* gnu/packages/python-web.scm (python-slugify) [propagated-inputs]: Add
python-text-unidecode.
---
 gnu/packages/python-web.scm | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/python-web.scm b/gnu/packages/python-web.scm
index 622f5fc6e2..e3cf25a687 100644
--- a/gnu/packages/python-web.scm
+++ b/gnu/packages/python-web.scm
@@ -4398,7 +4398,8 @@ Python.")
        (sha256
         (base32 "0w22fapghmzk3xdasc4dn7h8sl58l08d1h5zbf72dh80drv1g9b9"))))
     (propagated-inputs
-     `(("python-unidecode" ,python-unidecode)))
+     `(("python-unidecode" ,python-unidecode)
+       ("python-text-unidecode" ,python-text-unidecode)))
     (arguments
      `(#:phases
        (modify-phases %standard-phases
-- 
2.26.2


[-- Attachment #9: 0008-gnu-python-websockets-Fix-Python-package-name.patch --]
[-- Type: text/x-diff, Size: 1442 bytes --]

From fe3ec027ce3fe198a2cdfe3e1329094f2525fa42 Mon Sep 17 00:00:00 2001
From: Lars-Dominik Braun <lars@6xq.net>
Date: Thu, 7 Jan 2021 13:49:01 +0100
Subject: [PATCH v2 08/16] gnu: python-websockets: Fix Python package name.

* gnu/packages/python-web.scm (python-websockets) [arguments]: Add new
phase to fix package name.
---
 gnu/packages/python-web.scm | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/python-web.scm b/gnu/packages/python-web.scm
index e3cf25a687..e3318a18ce 100644
--- a/gnu/packages/python-web.scm
+++ b/gnu/packages/python-web.scm
@@ -5086,7 +5086,16 @@ Plus all the standard features of requests:
          (base32
           "03s3ml6sbki24aajllf8aily0xzrn929zxi84p50zkkbikdd4raw"))))
     (build-system python-build-system)
-    (arguments '(#:tests? #f))  ; Tests not included in release tarball.
+    (arguments
+     '(#:tests? #f  ; Tests not included in release tarball.
+       #:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'patch
+           (lambda* (#:key inputs #:allow-other-keys)
+             ;; Python package names use dot as separator.
+             (substitute* "setup.py"
+               (("websockets/extensions") "websockets.extensions"))
+             #t)))))
     (home-page "https://github.com/aaugustin/websockets")
     (synopsis
      "Python implementation of the WebSocket Protocol (RFC 6455 & 7692)")
-- 
2.26.2


[-- Attachment #10: 0009-gnu-python-black-Remove-blackd.patch --]
[-- Type: text/x-diff, Size: 1300 bytes --]

From d8199ab0353aa215d193ca511a63d4eea355b6c6 Mon Sep 17 00:00:00 2001
From: Lars-Dominik Braun <lars@6xq.net>
Date: Thu, 7 Jan 2021 13:50:18 +0100
Subject: [PATCH v2 09/16] gnu: python-black: Remove blackd.

* gnu/packages/python-xyz.scm (python-black) [arguments]: Add new phase
to prevent installation of blackd.
---
 gnu/packages/python-xyz.scm | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm
index 818a244378..a02960f1c6 100644
--- a/gnu/packages/python-xyz.scm
+++ b/gnu/packages/python-xyz.scm
@@ -4277,7 +4277,14 @@ matching of file paths.")
              (substitute* "tests/test_black.py"
                (("( *)def test_python38" match indent)
                 (string-append indent "@unittest.skip(\"guix\")\n" match)))
-             #t)))))
+             #t))
+         ;; Remove blackd, because it depends on python-aiohttp and
+         ;; python-aiohttp-cors.
+         (add-after 'unpack 'remove-entrypoint
+           (lambda _
+             (substitute* "setup.py"
+               (("\\s*\"blackd=blackd:patched_main \\[d\\]\",\n") "")
+                (("\"blackd\", ") "")))))))
     (propagated-inputs
      `(("python-click" ,python-click)
        ("python-attrs" ,python-attrs)
-- 
2.26.2


[-- Attachment #11: 0010-gnu-python-traitlets-Add-missing-input.patch --]
[-- Type: text/x-diff, Size: 1078 bytes --]

From b7ec4d0705bab46fa682a79de530336e2ff46775 Mon Sep 17 00:00:00 2001
From: Lars-Dominik Braun <lars@6xq.net>
Date: Thu, 7 Jan 2021 13:53:25 +0100
Subject: [PATCH v2 10/16] gnu: python-traitlets: Add missing input.

* gnu/packages/python-xyz.scm (python-traitlets) [propagated-inputs]:
Add python-six.
---
 gnu/packages/python-xyz.scm | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm
index a02960f1c6..4b7fff5750 100644
--- a/gnu/packages/python-xyz.scm
+++ b/gnu/packages/python-xyz.scm
@@ -6973,7 +6973,8 @@ cluster down and deletes the throwaway profile.")
          (replace 'check (lambda _ (invoke "pytest" "-vv" "traitlets"))))))
     (propagated-inputs
      `(("python-ipython-genutils" ,python-ipython-genutils)
-       ("python-decorator" ,python-decorator)))
+       ("python-decorator" ,python-decorator)
+       ("python-six" ,python-six)))
     (native-inputs
      `(("python-pytest" ,python-pytest)))
     (properties `((python2-variant . ,(delay python2-traitlets))))
-- 
2.26.2


[-- Attachment #12: 0011-gnu-python-idna-ssl-Add-missing-input.patch --]
[-- Type: text/x-diff, Size: 999 bytes --]

From e17eb838b9d77588200ac6db652e87bcd73e3f87 Mon Sep 17 00:00:00 2001
From: Lars-Dominik Braun <lars@6xq.net>
Date: Thu, 7 Jan 2021 14:12:56 +0100
Subject: [PATCH v2 11/16] gnu: python-idna-ssl: Add missing input.

* gnu/packages/python-xyz.scm (python-idna-ssl) [propagated-inputs]: Add
python-idna.
---
 gnu/packages/python-xyz.scm | 1 +
 1 file changed, 1 insertion(+)

diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm
index 4b7fff5750..58ad3b6e55 100644
--- a/gnu/packages/python-xyz.scm
+++ b/gnu/packages/python-xyz.scm
@@ -9517,6 +9517,7 @@ is binding LibSass.")
     (build-system python-build-system)
     (arguments
      `(#:tests? #f))          ;circular dependency with python-aiohttp
+    (propagated-inputs `(("python-idna" ,python-idna)))
     (home-page "https://github.com/aio-libs/idna-ssl")
     (synopsis "Patch @code{ssl.match_hostname} for Unicode(idna) domains support")
     (description "Patch @code{ssl.match_hostname} for Unicode(idna)
-- 
2.26.2


[-- Attachment #13: 0012-gnu-python-twisted-Remove-broken-console-scripts.patch --]
[-- Type: text/x-diff, Size: 1330 bytes --]

From 3372011b15cd28b203590981f5e1c561f977e31f Mon Sep 17 00:00:00 2001
From: Lars-Dominik Braun <lars@6xq.net>
Date: Thu, 7 Jan 2021 14:13:53 +0100
Subject: [PATCH v2 12/16] gnu: python-twisted: Remove broken console scripts.

* gnu/packages/python-xyz.scm (python-twisted) [arguments]: Patch
setup.py.
---
 gnu/packages/python-xyz.scm | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm
index 58ad3b6e55..d8b61cd1d2 100644
--- a/gnu/packages/python-xyz.scm
+++ b/gnu/packages/python-xyz.scm
@@ -12716,7 +12716,14 @@ format.")
                 "17d3hnxv9qndagzz63mdpyk99xj63p9gq586vjn0rxk8cl197nym"))))
     (build-system python-build-system)
     (arguments
-     '(#:tests? #f))                    ; FIXME: some tests fail
+     '(#:tests? #f                    ; FIXME: some tests fail
+       #:phases
+       (modify-phases %standard-phases
+         ;; Remove scripts, because they depend on [conch]
+         (add-after 'unpack 'remove-entrypoint
+           (lambda _
+             (substitute* "src/twisted/python/_setup.py"
+               (("\".+ = twisted\\.conch\\.scripts\\..+\",") "")))))))
     (propagated-inputs
      `(("python-zope-interface" ,python-zope-interface)
        ("python-pyhamcrest" ,python-pyhamcrest)
-- 
2.26.2


[-- Attachment #14: 0013-gnu-python-automat-Remove-broken-console-script.patch --]
[-- Type: text/x-diff, Size: 1332 bytes --]

From 7f644bb845249c32a2504fad0625dd75a6c01ce1 Mon Sep 17 00:00:00 2001
From: Lars-Dominik Braun <lars@6xq.net>
Date: Thu, 7 Jan 2021 14:15:12 +0100
Subject: [PATCH v2 13/16] gnu: python-automat: Remove broken console script.

* gnu/packages/python-xyz.scm (python-automat) [arguments]: Patch
setup.py.
---
 gnu/packages/python-xyz.scm | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm
index d8b61cd1d2..acd674878a 100644
--- a/gnu/packages/python-xyz.scm
+++ b/gnu/packages/python-xyz.scm
@@ -15151,7 +15151,15 @@ instead of servers and network commands.")
     ;; python-twisted depends on python-automat.  Twisted is optional, but the
     ;; tests fail if it is not available.  Also see
     ;; <https://github.com/glyph/automat/issues/71>.
-    (arguments '(#:tests? #f))
+    (arguments
+     `(#:tests? #f
+       #:phases
+       (modify-phases %standard-phases
+         ;; Remove script, because it depends on python-twisted.
+         (add-after 'unpack 'remove-entrypoint
+           (lambda _
+             (substitute* "setup.py"
+               (("\"automat-visualize = automat._visualize:tool\"") "")))))))
     (native-inputs
      `(("python-m2r" ,python-m2r)
        ("python-setuptools-scm" ,python-setuptools-scm)
-- 
2.26.2


[-- Attachment #15: 0014-gnu-python-packaging-bootstrap-Remove-dependency.patch --]
[-- Type: text/x-diff, Size: 1414 bytes --]

From ff4f539c9c4cd18e7f5c9bea4f87eec42ee81394 Mon Sep 17 00:00:00 2001
From: Lars-Dominik Braun <lars@6xq.net>
Date: Thu, 7 Jan 2021 14:16:05 +0100
Subject: [PATCH v2 14/16] gnu: python-packaging-bootstrap: Remove dependency.

* gnu/packages/python-xyz.scm (python-packaging-bootstrap) [arguments]:
Remove dependency from setup.py, which we do not provide for this
variant.
---
 gnu/packages/python-xyz.scm | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm
index acd674878a..e78016221f 100644
--- a/gnu/packages/python-xyz.scm
+++ b/gnu/packages/python-xyz.scm
@@ -16049,10 +16049,18 @@ information.")
    (package/inherit
     python-packaging
     (name "python-packaging-bootstrap")
+    (arguments
+      (substitute-keyword-arguments (package-arguments python-packaging)
+        ((#:phases phases)
+         `(modify-phases ,phases
+            (add-after 'unpack 'fix-dependencies
+              (lambda* (#:key tests? #:allow-other-keys)
+                (substitute* "setup.py" (("\"six\"") ""))
+                #t))))
+         ((#:tests? _ #f) #f)))
     (native-inputs '())
     (propagated-inputs
-     `(("python-pyparsing" ,python-pyparsing)))
-    (arguments '(#:tests? #f)))))
+     `(("python-pyparsing" ,python-pyparsing))))))
 
 (define-public python2-packaging-bootstrap
   (hidden-package
-- 
2.26.2


[-- Attachment #16: 0015-gnu-python-traceback2-Add-missing-dependency.patch --]
[-- Type: text/x-diff, Size: 999 bytes --]

From 65d2ab65d4dd73e1bdb0248cc41848c55859f98d Mon Sep 17 00:00:00 2001
From: Lars-Dominik Braun <lars@6xq.net>
Date: Thu, 7 Jan 2021 14:17:20 +0100
Subject: [PATCH v2 15/16] gnu: python-traceback2: Add missing dependency.

* gnu/packages/python-xyz.scm (python-traceback2) [propagated-inputs]:
Add python-six.
---
 gnu/packages/python-xyz.scm | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm
index e78016221f..1df9807626 100644
--- a/gnu/packages/python-xyz.scm
+++ b/gnu/packages/python-xyz.scm
@@ -17266,7 +17266,8 @@ lines are read from a single file.")
     (native-inputs
      `(("python-pbr" ,python-pbr-minimal)))
     (propagated-inputs
-      `(("python-linecache2" ,python-linecache2)))
+      `(("python-linecache2" ,python-linecache2)
+        ("python-six" ,python-six)))
     (home-page
       "https://github.com/testing-cabal/traceback2")
     (synopsis "Backports of the traceback module")
-- 
2.26.2


[-- Attachment #17: 0016-gnu-python2-packaging-bootstrap-Add-missing-dependen.patch --]
[-- Type: text/x-diff, Size: 984 bytes --]

From 55aab0eba972c6672be9d8de5e0ec9340b10c243 Mon Sep 17 00:00:00 2001
From: Lars-Dominik Braun <lars@6xq.net>
Date: Mon, 11 Jan 2021 15:19:01 +0100
Subject: [PATCH v2 16/16] gnu: python2-packaging-bootstrap: Add missing
 dependency

* gnu/packages/python-xyz.scm (python2-packaging-bootstrap)
[propagated-inputs]: Add python2-six-bootstrap.
---
 gnu/packages/python-xyz.scm | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm
index 1df9807626..434946acfd 100644
--- a/gnu/packages/python-xyz.scm
+++ b/gnu/packages/python-xyz.scm
@@ -16069,7 +16069,8 @@ information.")
     (name "python2-packaging-bootstrap")
     (native-inputs '())
     (propagated-inputs
-     `(("python-pyparsing" ,python2-pyparsing)))
+     `(("python-pyparsing" ,python2-pyparsing)
+       ("python-six" ,python2-six-bootstrap)))
     (arguments
      `(#:tests? #f
        ,@(package-arguments python2-packaging))))))
-- 
2.26.2


[-- Attachment #18: failing-packages.txt --]
[-- Type: text/plain, Size: 7445 bytes --]

awscli
b4
beets
beets-bandcamp
behave
bpython
buku
calibre
certbot
curseradio
dbxfs
debops
diffoscope
docker-compose
electron-cash
electrum
enjarify
fabric
fdroidserver
fenics
frescobaldi
gandi.cli
gaupol
gitless
gpodder
grit
grocsvs
instantmusic
jrnl
jupyter
keepkey-agent
khmer
ledger-agent
letsencrypt
linkchecker
mailman
miso
mps-youtube
nmoldyn
nototools
offlate
onionshare
openshot
pagekite
patchwork
pbtranscript-tofu
pepr
picard
poetry
poretools
postorius
pre-commit
presentty
protonvpn-cli
ptpython2
pulseaudio-dlna
pybitmessage
pyicoteo
python2-anaconda-client
python2-apache-libcloud
python2-argcomplete
python2-arrow
python2-autograd
python2-axolotl
python2-bcrypt
python2-behave-web-api
python2-betamax
python2-betamax-matchers
python2-bigfloat
python2-biom-format
python2-booleanoperations
python2-botocore
python2-cachecontrol
python2-cairocffi
python2-celery
python2-checkm-genome
python2-clf
python2-cloudpickle
python2-consul
python2-cryptography
python2-cvxopt
python2-cypari2
python2-discogs-client
python2-dns-lexicon
python2-drmaa
python2-dulwich
python2-ecpy
python2-elasticsearch
python2-email-validator
python2-empy
python2-entrypoints
python2-factory-boy
python2-faker
python2-fakeredis
python2-falcon
python2-falcon-cors
python2-fastlmm
python2-fido2
python2-flake8
python2-flake8-polyfill
python2-flask
python2-flask-babel
python2-flask-htmlmin
python2-flask-login
python2-flask-multistatic
python2-flask-wtf
python2-flex
python2-furl
python2-future
python2-git-review
python2-google-api-client
python2-graphql-core
python2-gridmap
python2-guzzle-sphinx-theme
python2-hdf4
python2-html5-parser
python2-htseq
python2-httpbin
python2-ipykernel
python2-ipyparallel
python2-ipython
python2-ipython-cluster-helper
python2-ipywidgets
python2-josepy
python2-jupyter-client
python2-jupyter-console
python2-kivy
python2-kombu
python2-ledgerblue
python2-libadalang
python2-lzstring
python2-mapnik
python2-matplotlib
python2-matplotlib-documentation
python2-mechanicalsoup
python2-mutagen
python2-mwclient
python2-nbconvert
python2-nbxmpp
python2-ndg-httpsclient
python2-nose2
python2-notebook
python2-notify2
python2-numpydoc
python2-numpy-documentation
python2-oauthlib
python2-openid-cla
python2-openid-teams
python2-openpyxl
python2-orator
python2-os-client-config
python2-os-testr
python2-pandas
python2-paramiko
python2-parsedatetime
python2-partd
python2-pathpy
python2-patsy
python2-pendulum
python2-pep517
python2-pgpdump
python2-pika
python2-plastid
python2-plotly
python2-py2neo
python2-pybedtools
python2-pyclipper
python2-pyfakefs
python2-pylibmc
python2-pymysql
python2-pynacl
python2-pynamecheap
python2-pyopenssl
python2-pysnptools
python2-pytest-xdist
python2-radon
python2-rauth
python2-renpy
python2-requests
python2-requests-file
python2-requests-mock
python2-requests-oauthlib
python2-requests-toolbelt
python2-responses
python2-roca-detect
python2-rpython
python2-rq
python2-rst.linker
python2-ruamel.yaml
python2-s3transfer
python2-scikit-learn
python2-scipy
python2-scripttest
python2-seaborn
python2-service-identity
python2-setuptools-scm-git-archive
python2-sh
python2-shapely
python2-sockjs-tornado
python2-sphinx-cloud-sptheme
python2-sphinxcontrib-programoutput
python2-sphinx-repoze-autointerface
python2-statsmodels
python2-stem
python2-stevedore
python2-tables
python2-terminado
python2-tldextract
python2-tornado
python2-translate-toolkit
python2-trollius-redis
python2-twine
python2-urlgrabber
python2-urllib3
python2-utils
python2-validators
python2-warpedlmm
python2-widgetsnbextension
python2-wsgiproxy2
python2-xenon
python2-yubikey-manager
python2-zope-component
python2-zope-configuration
python2-zope-exceptions
python2-zope-i18nmessageid
python2-zope-location
python2-zope-proxy
python2-zope-schema
python2-zope-security
python2-zope-testrunner
python-args
python-aws-xray-sdk
python-bbknn
python-behave-web-api
python-bigfloat
python-btrees
python-cachy
python-clint
python-codacy-coverage
python-cypari2
python-daemux
python-dendropy
python-django-auth-ldap
python-django-bulk-update
python-django-classy-tags
python-django-compressor
python-django-contact-form
python-django-debug-toolbar
python-django-debug-toolbar-alchemy
python-django-extensions
python-django-filter
python-django-haystack
python-django-jinja
python-django-logging-json
python-django-mailman3
python-django-netfields
python-django-redis
python-djangorestframework
python-django-rq
python-django-sekizai
python-django-statici18n
python-django-url-filter
python-docker
python-drmaa
python-duniterpy
python-eliot
python-enum34
python-eventlet
python-faiss
python-fakeredis
python-flasgger
python-flask-restplus
python-flask-session
python-flit
python-funcparserlib
python-graphene
python-gridmap
python-gyp
python-hacking
python-hdf4
python-html5-parser
python-hy
python-hyperkitty
python-internetarchive
python-ipykernel
python-ipyparallel
python-ipython-cluster-helper
python-ipython-documentation
python-ipywidgets
python-jaraco-packaging
python-jose
python-jsonpickle
python-jupyter-client
python-jupyter-console
python-jupyter-kernel-mgmt
python-jupyter-kernel-test
python-jupyter-protocol
python-keyring
python-keyrings.alt
python-keystoneclient
python-kivy
python-ldap
python-lfdfiles
python-libsass
python-logwrap
python-mailman-hyperkitty
python-mamba
python-matplotlib-documentation
python-matrix-synapse-ldap3
python-miniboa
python-moto
python-nautilus
python-nbval
python-ndg-httpsclient
python-notebook
python-numpy-documentation
python-onnx
python-ont-fast5-api
python-openid-cla
python-openid-teams
python-orator
python-os-client-config
python-oslo.config
python-oslo.context
python-oslo.i18n
python-oslo.log
python-oslo.serialization
python-oslosphinx
python-oslo.utils
python-os-testr
python-owslib
python-pafy
python-parallel
python-pari-jupyter
python-pathpy
python-pendulum
python-persistent
python-petsc4py
python-pifpaf
python-plotly
python-py3dns
python-pyacoustid
python-pyclipper
python-pydub
python-pykafka
python-pykka
python-pylibmc
python-pyramid
python-pytest-arraydiff
python-pytest-checkdocs
python-pytest-django
python-pytest-mpl
python-pytest-shutil
python-pytest-virtualenv
python-pytidylib
python-pyxb
python-qdarkstyle
python-qtconsole
python-qtpy
python-radon
python-scripttest
python-setools
python-setuptools-scm-git-archive
python-slepc4py
python-socksipychain
python-sphinx-copybutton
python-stestr
python-swiftclient
python-symengine
python-tempest-lib
python-tortoise-orm
python-translate-toolkit
python-translation-finder
python-trollius-redis
python-trytond-account
python-trytond-account-invoice
python-trytond-account-invoice-stock
python-trytond-account-product
python-trytond-analytic-account
python-trytond-company
python-trytond-country
python-trytond-currency
python-trytond-party
python-trytond-product
python-trytond-purchase
python-trytond-purchase-request
python-trytond-stock
python-trytond-stock-lot
python-trytond-stock-supply
python-txacme
python-typing
python-widgetsnbextension
python-xattr
python-xenon
python-zope-component
python-zope-configuration
python-zope-copy
python-zope-deferredimport
python-zope-exceptions
python-zope-i18nmessageid
python-zope-location
python-zope-proxy
python-zope-schema
python-zope-security
python-zope-testrunner
qutebrowser
rapid-photo-downloader
renpy
reprotest
ribodiff
rtv
silkaj
solaar
synapse
syncthing-gtk
tadbit
taxtastic
tbsp
trezor-agent
tuir
unknown-horizons
virtaal
xpra
you-get
youtube-dl
youtube-dl-gui