bug#33046: pypi importer doesn't print the correct source

bug#33046: pypi importer doesn't print the correct source

[Julien Lepiller]

Hi, I tried to use the importer to refresh python-twisted, and got this 
source:

(uri (pypi-uri "twisted" version))

while the correct one should be

(uri (pypi-uri "Twisted" version ".tar.bz2"))

bug#33046: pypi importer doesn't print the correct source

[Maxim Cournoyer]

[-- Attachment #1: Type: text/plain, Size: 4808 bytes --]

Hello!

Julien Lepiller <julien@lepiller.eu> writes:

> Hi, I tried to use the importer to refresh python-twisted, and got
> this source:
>
> (uri (pypi-uri "twisted" version))
>
> while the correct one should be
>
> (uri (pypi-uri "Twisted" version ".tar.bz2"))

I believe the attached patch fixes this :-). It's based on the patch set
attached to bug #24450 though, so that one should be reviewed/merged
first.

Here's the result, for a couple packages that were problematic in the
mixed case regards:

--8<---------------cut here---------------start------------->8---
./pre-inst-env guix import pypi Twisted

Starting download of /tmp/guix-file.BdpmRc
From https://files.pythonhosted.org/packages/5d/0e/a72d85a55761c2c3ff1cb968143a2fd5f360220779ed90e0fadf4106d4f2/Twisted-18.9.0.tar.bz2...
 …9.0.tar.bz2  2.9MiB                 1.5MiB/s 00:02 [##################] 100.0%
(package
  (name "python-twisted")
  (version "18.9.0")
  (source
    (origin
      (method url-fetch)
      (uri (pypi-uri "Twisted" version))
      (sha256
        (base32
          "15d3gmkrg8g27hyd6ihawv2y2dv5gnpyg67wy9npgbl4pz3f4jr9"))))
  (build-system python-build-system)
  (propagated-inputs
    `(("python-attrs" ,python-attrs)
      ("python-automat" ,python-automat)
      ("python-constantly" ,python-constantly)
      ("python-hyperlink" ,python-hyperlink)
      ("python-incremental" ,python-incremental)
      ("python-pyhamcrest" ,python-pyhamcrest)
      ("python-zope.interface" ,python-zope.interface)))
  (native-inputs
    `(("python-pydoctor" ,python-pydoctor)
      ("python-pyflakes" ,python-pyflakes)
      ("python-sphinx" ,python-sphinx)
      ("python-subunit" ,python-subunit)
      ("python-towncrier" ,python-towncrier)
      ("python-twisted-dev-tools"
       ,python-twisted-dev-tools)
      ("python-twistedchecker" ,python-twistedchecker)))
  (home-page "http://twistedmatrix.com/")
  (synopsis
    "An asynchronous networking framework written in Python")
  (description
    "An asynchronous networking framework written in Python")
  (license license:expat))
--8<---------------cut here---------------end--------------->8---


--8<---------------cut here---------------start------------->8---
./pre-inst-env guix import pypi uWSGI

Starting download of /tmp/guix-file.TNSMx7
From https://files.pythonhosted.org/packages/e7/1e/3dcca007f974fe4eb369bf1b8629d5e342bb3055e2001b2e5340aaefae7a/uwsgi-2.0.18.tar.gz...
 ….18.tar.gz  783KiB                  751KiB/s 00:01 [##################] 100.0%
guix import: warning: Cannot guess requirements from source archive: no requires.txt file found.
(package
  (name "python-uwsgi")
  (version "2.0.18")
  (source
    (origin
      (method url-fetch)
      (uri (pypi-uri "uwsgi" version))
      (sha256
        (base32
          "10zmk4npknigmbqcq1wmhd461dk93159px172112vyq0i19sqwj9"))))
  (build-system python-build-system)
  (home-page
    "https://uwsgi-docs.readthedocs.io/en/latest/")
  (synopsis "The uWSGI server")
  (description "The uWSGI server")
  (license #f))
--8<---------------cut here---------------end--------------->8---

--8<---------------cut here---------------start------------->8---
./pre-inst-env guix import pypi Deprecated

Starting download of /tmp/guix-file.cHmIDG
From https://files.pythonhosted.org/packages/2c/6a/f7c9d1db90d758a9dc21daaa5660a01b3ab22ca56533f6a62b24c0d113e6/Deprecated-1.2.5.tar.gz...
 …1.2.5.tar.gz  4.8MiB                1.2MiB/s 00:04 [##################] 100.0%

Starting download of /tmp/guix-file.Yj88Cr
From https://files.pythonhosted.org/packages/9f/7a/003fa432f1e45625626549726c2fbb7a29baa764e9d1fdb2323a5d779f8a/Deprecated-1.2.5-py2.py3-none-any.whl...
 …y2.py3-none-any.whl  8KiB           2.1MiB/s 00:00 [##################] 100.0%
(package
  (name "python-deprecated")
  (version "1.2.5")
  (source
    (origin
      (method url-fetch)
      (uri (pypi-uri "Deprecated" version))
      (sha256
        (base32
          "0mw5hv0bci5lzzi7ip62vm1ivmxhqj7zwc557p7iyjz3xsq3wa9g"))))
  (build-system python-build-system)
  (propagated-inputs
    `(("python-wrapt" ,python-wrapt)))
  (native-inputs
    `(("python-bumpversion" ,python-bumpversion)
      ("python-pytest" ,python-pytest)
      ("python-pytest-cov" ,python-pytest-cov)
      ("python-sphinx" ,python-sphinx)
      ("python-tox" ,python-tox)))
  (home-page
    "https://github.com/tantale/deprecated")
  (synopsis
    "Python @deprecated decorator to deprecate old python classes, functions or methods.")
  (description
    "Python @deprecated decorator to deprecate old python classes, functions or methods.")
  (license license:expat))
--8<---------------cut here---------------end--------------->8---


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-import-pypi-Preserve-package-name-case-when-forming-.patch --]
[-- Type: text/x-patch, Size: 3114 bytes --]

From 1290f9d1f0d594fdd4723d76b94116be25da9dd5 Mon Sep 17 00:00:00 2001
From: Maxim Cournoyer <maxim.cournoyer@gmail.com>
Date: Sat, 30 Mar 2019 20:27:35 -0400
Subject: [PATCH] import: pypi: Preserve package name case when forming
 pypi-uri.

Fixes issue: #33046.

* guix/build-system/python.scm (pypi-uri): Update the host URI to
"files.pythonhosted.org".
* guix/import/pypi.scm (make-pypi-sexp): Preserve the package name case when
the source URL calls for it.
---
 guix/build-system/python.scm |  2 +-
 guix/import/pypi.scm         | 23 ++++++++++++++---------
 2 files changed, 15 insertions(+), 10 deletions(-)

diff --git a/guix/build-system/python.scm b/guix/build-system/python.scm
index b753940bad..e39c06528e 100644
--- a/guix/build-system/python.scm
+++ b/guix/build-system/python.scm
@@ -50,7 +50,7 @@
   "Return a URI string for the Python package hosted on the Python Package
 Index (PyPI) corresponding to NAME and VERSION.  EXTENSION is the file name
 extension, such as '.tar.gz'."
-  (string-append "https://pypi.org/packages/source/"
+  (string-append "https://files.pythonhosted.org/packages/source/"
                  (string-take name 1) "/" name "/"
                  name "-" version extension))
 
diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm
index a2ce14b192..fecf95d0a7 100644
--- a/guix/import/pypi.scm
+++ b/guix/import/pypi.scm
@@ -371,15 +371,20 @@ VERSION, SOURCE-URL, HOME-PAGE, SYNOPSIS, DESCRIPTION, and LICENSE."
                 `(package
                    (name ,(python->package-name name))
                    (version ,version)
-                   (source (origin
-                             (method url-fetch)
-                             ;; Sometimes 'pypi-uri' doesn't quite work due to mixed
-                             ;; cases in NAME, for instance, as is the case with
-                             ;; "uwsgi".  In that case, fall back to a full URL.
-                             (uri (pypi-uri ,(string-downcase name) version))
-                             (sha256
-                              (base32
-                               ,(guix-hash-url temp)))))
+                   (source
+                    (origin
+                      (method url-fetch)
+                      ;; PyPI URL are case sensitive, but sometimes a project
+                      ;; named using mixed case has a URL using lower case, so
+                      ;; we must work around this inconsistency.  For actual
+                      ;; examples, compare the URLs of the "Deprecated" and
+                      ;; "uWSGI" PyPI packages.
+                      (uri ,(if (string-contains source-url name)
+                                `(pypi-uri ,name version)
+                                `(pypi-uri ,(string-downcase name) version)))
+                      (sha256
+                       (base32
+                        ,(guix-hash-url temp)))))
                    (build-system python-build-system)
                    ,@(maybe-inputs required-inputs 'propagated-inputs)
                    ,@(maybe-inputs test-inputs 'native-inputs)
-- 
2.20.1


[-- Attachment #3: Type: text/plain, Size: 7 bytes --]


Maxim

bug#33046: [PATCH v2] bug#33046: pypi importer doesn't print the correct source

[Maxim Cournoyer]

[-- Attachment #1.1: Type: text/plain, Size: 181 bytes --]

Hello,

I just noticed that by changing the PyPI URL, the PyPI updater coverage
had gone from 15.3% to 0%!  Luckily, the fix was trivial and is included
in my v2 patch (attached).


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.2: 0001-v2-import-pypi-Preserve-package-name-case-when-forming-.patch --]
[-- Type: text/x-patch, Size: 3791 bytes --]

From 3c9c4229c460b6fdf0da804dddcd3d2cdd3aec4b Mon Sep 17 00:00:00 2001
From: Maxim Cournoyer <maxim.cournoyer@gmail.com>
Date: Sat, 30 Mar 2019 20:27:35 -0400
Subject: [PATCH] import: pypi: Preserve package name case when forming
 pypi-uri.

Fixes issue: #33046.

* guix/build-system/python.scm (pypi-uri): Update the host URI to
"files.pythonhosted.org".
* guix/import/pypi.scm (make-pypi-sexp): Preserve the package name case when
the source URL calls for it.
(pypi-package?): Augment to include new PyPI URL.
---
 guix/build-system/python.scm |  2 +-
 guix/import/pypi.scm         | 26 ++++++++++++++++----------
 2 files changed, 17 insertions(+), 11 deletions(-)

diff --git a/guix/build-system/python.scm b/guix/build-system/python.scm
index b753940bad..e39c06528e 100644
--- a/guix/build-system/python.scm
+++ b/guix/build-system/python.scm
@@ -50,7 +50,7 @@
   "Return a URI string for the Python package hosted on the Python Package
 Index (PyPI) corresponding to NAME and VERSION.  EXTENSION is the file name
 extension, such as '.tar.gz'."
-  (string-append "https://pypi.org/packages/source/"
+  (string-append "https://files.pythonhosted.org/packages/source/"
                  (string-take name 1) "/" name "/"
                  name "-" version extension))
 
diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm
index a2ce14b192..a02b718d39 100644
--- a/guix/import/pypi.scm
+++ b/guix/import/pypi.scm
@@ -371,15 +371,20 @@ VERSION, SOURCE-URL, HOME-PAGE, SYNOPSIS, DESCRIPTION, and LICENSE."
                 `(package
                    (name ,(python->package-name name))
                    (version ,version)
-                   (source (origin
-                             (method url-fetch)
-                             ;; Sometimes 'pypi-uri' doesn't quite work due to mixed
-                             ;; cases in NAME, for instance, as is the case with
-                             ;; "uwsgi".  In that case, fall back to a full URL.
-                             (uri (pypi-uri ,(string-downcase name) version))
-                             (sha256
-                              (base32
-                               ,(guix-hash-url temp)))))
+                   (source
+                    (origin
+                      (method url-fetch)
+                      ;; PyPI URL are case sensitive, but sometimes a project
+                      ;; named using mixed case has a URL using lower case, so
+                      ;; we must work around this inconsistency.  For actual
+                      ;; examples, compare the URLs of the "Deprecated" and
+                      ;; "uWSGI" PyPI packages.
+                      (uri ,(if (string-contains source-url name)
+                                `(pypi-uri ,name version)
+                                `(pypi-uri ,(string-downcase name) version)))
+                      (sha256
+                       (base32
+                        ,(guix-hash-url temp)))))
                    (build-system python-build-system)
                    ,@(maybe-inputs required-inputs 'propagated-inputs)
                    ,@(maybe-inputs test-inputs 'native-inputs)
@@ -437,7 +442,8 @@ VERSION, SOURCE-URL, HOME-PAGE, SYNOPSIS, DESCRIPTION, and LICENSE."
   (define (pypi-url? url)
     (or (string-prefix? "https://pypi.org/" url)
         (string-prefix? "https://pypi.python.org/" url)
-        (string-prefix? "https://pypi.org/packages" url)))
+        (string-prefix? "https://pypi.org/packages" url)
+        (string-prefix? "https://files.pythonhosted.org" url)))
 
   (let ((source-url (and=> (package-source package) origin-uri))
         (fetch-method (and=> (package-source package) origin-method)))
-- 
2.20.1


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

bug#33046: Status: pypi importer doesn't print the correct source

[Maxim Cournoyer]

Fixed per commit 4b60ab8c006964d026dee8cf5f1260eba0b2bb81. Closing.

Thanks!

Maxim