[bug#50854] [PATCH 0/2] Make the PyPI importer honor the ‘upstream-name’ package property

[bug#50854] [PATCH 0/2] Make the PyPI importer honor the ‘upstream-name’ package property

[Xinglu Chen]

[-- Attachment #1: Type: text/plain, Size: 1345 bytes --]

If a PyPI package contains a “-” followed by one or more digits, e.g.,
“AV-98”, the PyPI importer/updater will think that the digits after the
“-” is the version of the package, and instead of trying to update
“AV-98” it will try to update “AV”.

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix refresh av-98
following redirection to `https://pypi.org/pypi/av/json'...
/home/yoctocell/src/guix/gnu/packages/web-browsers.scm:914:13: av-98 would be upgraded from 1.0.1 to 8.0.3
--8<---------------cut here---------------end--------------->8---

The first patch makes the PyPI importer honor the ‘upstream-name’
property; it will also generate a package with an ‘upstream-name’
property if the package name contains a “-” followed by one or more
digits.

The second patch sets the ‘upstream-name’ property for the ‘av-98’
package.

Xinglu Chen (2):
  import: pypi: Honor the 'upstream-name' package property.
  gnu: av-98: Set 'upstream-name' property.

 gnu/packages/web-browsers.scm |   2 +
 guix/import/pypi.scm          |  20 +++++--
 tests/pypi.scm                | 106 +++++++++++++++++++++++++++++++---
 3 files changed, 115 insertions(+), 13 deletions(-)


base-commit: 6ae4644984608b7eff7ab54d3a5787c661d85b2e
-- 
2.33.0




[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 861 bytes --]

[bug#50854] [PATCH 1/2] import: pypi: Honor the 'upstream-name' package property.

[Xinglu Chen]

Previously, when a PyPI package had a “-” followed by one or more digits in
its name, e.g., “AV-98”, the importer would interpret “98” as the version of
the package and thus mistake the “AV-98” package for the “av” package on PyPI.

  $ ./pre-inst-env guix refresh av-98
  following redirection to `https://pypi.org/pypi/av/json'...
  /home/yoctocell/src/guix/gnu/packages/web-browsers.scm:914:13: av-98 would be upgraded from 1.0.1 to 8.0.3

Setting the ‘upstream-name’ property to “AV-98” would solve the problem.

  $ ./pre-inst-env guix refresh av-98
  /home/yoctocell/src/guix/gnu/packages/web-browsers.scm:914:13: 1.0.1 is already the latest version of av-98

* guix/import/pypi.scm (guix-package->pypi-name): Honor ‘upstream-name’
property.
(make-pypi-sexp): Set ‘upstream-name’ property when appropriate.
* tests/pypi.scm (test-json): Rename to ...
(test-json-1): ... this.
(test-json-2): New variable
("guix-package->pypi-name, honor 'upstream-name'"): New test.
("pypi->guix-package, package name contains \"-\" followed by digits"):
Likewise.
---
 guix/import/pypi.scm |  20 +++++---
 tests/pypi.scm       | 106 ++++++++++++++++++++++++++++++++++++++++---
 2 files changed, 113 insertions(+), 13 deletions(-)

diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm
index 6731d50891..b7859c8341 100644
--- a/guix/import/pypi.scm
+++ b/guix/import/pypi.scm
@@ -9,6 +9,7 @@
 ;;; Copyright © 2020 Lars-Dominik Braun <ldb@leibniz-psychology.org>
 ;;; Copyright © 2020 Arun Isaac <arunisaac@systemreboot.net>
 ;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net>
+;;; Copyright © 2021 Xinglu Chen <public@yoctocell.xyz>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -163,12 +164,13 @@ (define (url->pypi-name url)
     (hyphen-package-name->name+version
      (basename (file-sans-extension url))))
 
-  (match (and=> (package-source package) origin-uri)
-    ((? string? url)
-     (url->pypi-name url))
-    ((lst ...)
-     (any url->pypi-name lst))
-    (#f #f)))
+  (or (assoc-ref (package-properties package) 'upstream-name)
+      (match (and=> (package-source package) origin-uri)
+        ((? string? url)
+         (url->pypi-name url))
+        ((lst ...)
+         (any url->pypi-name lst))
+        (#f #f))))
 
 (define (wheel-url->extracted-directory wheel-url)
   (match (string-split (basename wheel-url) #\-)
@@ -423,6 +425,11 @@ (define (make-pypi-sexp name version source-url wheel-url home-page synopsis
                         description license)
   "Return the `package' s-expression for a python package with the given NAME,
 VERSION, SOURCE-URL, HOME-PAGE, SYNOPSIS, DESCRIPTION, and LICENSE."
+  (define (maybe-upstream-name name)
+    (if (string-match ".*\\-[0-9]+" (pk name))
+        `((properties ,`'(("upstream-name" . ,name))))
+        '()))
+  
   (call-with-temporary-output-file
    (lambda (temp port)
      (and (url-fetch source-url temp)
@@ -461,6 +468,7 @@ (define (make-pypi-sexp name version source-url wheel-url home-page synopsis
                       (sha256
                        (base32
                         ,(guix-hash-url temp)))))
+                   ,@(maybe-upstream-name name)
                    (build-system python-build-system)
                    ,@(maybe-inputs required-inputs 'propagated-inputs)
                    ,@(maybe-inputs native-inputs 'native-inputs)
diff --git a/tests/pypi.scm b/tests/pypi.scm
index f421d6d9df..70f4298a90 100644
--- a/tests/pypi.scm
+++ b/tests/pypi.scm
@@ -2,6 +2,7 @@
 ;;; Copyright © 2014 David Thompson <davet@gnu.org>
 ;;; Copyright © 2016 Ricardo Wurmus <rekado@elephly.net>
 ;;; Copyright © 2019 Maxim Cournoyer <maxim.cournoyer@gmail.com>
+;;; Copyright © 2021 Xinglu Chen <public@yoctocell.xyz>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -29,7 +30,7 @@ (define-module (test-pypi)
   #:use-module (srfi srfi-64)
   #:use-module (ice-9 match))
 
-(define test-json
+(define test-json-1
   "{
   \"info\": {
     \"version\": \"1.0.0\",
@@ -57,6 +58,34 @@ (define test-json
   }
 }")
 
+(define test-json-2
+  "{
+  \"info\": {
+    \"version\": \"1.0.0\",
+    \"name\": \"foo-99\",
+    \"license\": \"GNU LGPL\",
+    \"summary\": \"summary\",
+    \"home_page\": \"http://example.com\",
+    \"classifiers\": [],
+    \"download_url\": \"\"
+  },
+  \"urls\": [],
+  \"releases\": {
+    \"1.0.0\": [
+      {
+        \"url\": \"https://example.com/foo-99-1.0.0.egg\",
+        \"packagetype\": \"bdist_egg\"
+      }, {
+        \"url\": \"https://example.com/foo-99-1.0.0.tar.gz\",
+        \"packagetype\": \"sdist\"
+      }, {
+        \"url\": \"https://example.com/foo-99-1.0.0-py2.py3-none-any.whl\",
+        \"packagetype\": \"bdist_wheel\"
+      }
+    ]
+  }
+}")
+
 (define test-source-hash
   "")
 
@@ -147,6 +176,13 @@ (define test-metadata-with-extras-jedi "\
                     (uri (list "https://bitheap.org/cram/cram-0.7.tar.gz"
                                (pypi-uri "cram" "0.7"))))))))
 
+(test-equal "guix-package->pypi-name, honor 'upstream-name'"
+  "bar-3"
+  (guix-package->pypi-name
+   (dummy-package "foo"
+                  (properties
+                   '((upstream-name . "bar-3"))))))
+
 (test-equal "specification->requirement-name"
   '("Fizzy" "PickyThing" "SomethingWithMarker" "requests" "pip")
   (map specification->requirement-name test-specifications))
@@ -198,8 +234,8 @@ (define test-metadata-with-extras-jedi "\
                  (lambda (url . rest)
                    (match url
                      ("https://pypi.org/pypi/foo/json"
-                      (values (open-input-string test-json)
-                              (string-length test-json)))
+                      (values (open-input-string test-json-1)
+                              (string-length test-json-1)))
                      ("https://example.com/foo-1.0.0-py2.py3-none-any.whl" #f)
                      (_ (error "Unexpected URL: " url)))))
                 (match (pypi->guix-package "foo")
@@ -264,8 +300,8 @@ (define test-metadata-with-extras-jedi "\
                (lambda (url . rest)
                  (match url
                    ("https://pypi.org/pypi/foo/json"
-                    (values (open-input-string test-json)
-                            (string-length test-json)))
+                    (values (open-input-string test-json-1)
+                            (string-length test-json-1)))
                    ("https://example.com/foo-1.0.0-py2.py3-none-any.whl" #f)
                    (_ (error "Unexpected URL: " url)))))
               ;; Not clearing the memoization cache here would mean returning the value
@@ -317,8 +353,8 @@ (define test-metadata-with-extras-jedi "\
                (lambda (url . rest)
                  (match url
                    ("https://pypi.org/pypi/foo/json"
-                    (values (open-input-string test-json)
-                            (string-length test-json)))
+                    (values (open-input-string test-json-1)
+                            (string-length test-json-1)))
                    ("https://example.com/foo-1.0.0-py2.py3-none-any.whl" #f)
                    (_ (error "Unexpected URL: " url)))))
               ;; Not clearing the memoization cache here would mean returning the value
@@ -345,4 +381,60 @@ (define test-metadata-with-extras-jedi "\
                 (x
                  (pk 'fail x #f))))))
 
+(test-assert "pypi->guix-package, package name contains \"-\" followed by digits"
+  ;; Replace network resources with sample data.
+  (mock ((guix import utils) url-fetch
+         (lambda (url file-name)
+           (match url
+             ("https://example.com/foo-99-1.0.0.tar.gz"
+              (begin
+                ;; Unusual requires.txt location should still be found.
+                (mkdir-p "foo-99-1.0.0/src/bizarre.egg-info")
+                (with-output-to-file "foo-99-1.0.0/src/bizarre.egg-info/requires.txt"
+                  (lambda ()
+                    (display test-requires.txt)))
+                (parameterize ((current-output-port (%make-void-port "rw+")))
+                  (system* "tar" "czvf" file-name "foo-99-1.0.0/"))
+                (delete-file-recursively "foo-99-1.0.0")
+                (set! test-source-hash
+                  (call-with-input-file file-name port-sha256))))
+             ("https://example.com/foo-99-1.0.0-py2.py3-none-any.whl" #f)
+             (_ (error "Unexpected URL: " url)))))
+        (mock ((guix http-client) http-fetch
+               (lambda (url . rest)
+                 (match url
+                   ("https://pypi.org/pypi/foo-99/json"
+                    (values (open-input-string test-json-2)
+                            (string-length test-json-2)))
+                   ("https://example.com/foo-99-1.0.0-py2.py3-none-any.whl" #f)
+                   (_ (error "Unexpected URL: " url)))))
+              (match (pypi->guix-package "foo-99")
+                (('package
+                   ('name "python-foo-99")
+                   ('version "1.0.0")
+                   ('source ('origin
+                              ('method 'url-fetch)
+                              ('uri ('pypi-uri "foo-99" 'version))
+                              ('sha256
+                               ('base32
+                                (? string? hash)))))
+                   ('properties ('quote (("upstream-name" . "foo-99"))))
+                   ('build-system 'python-build-system)
+                   ('propagated-inputs
+                    ('quasiquote
+                     (("python-bar" ('unquote 'python-bar))
+                      ("python-foo" ('unquote 'python-foo)))))
+                   ('native-inputs
+                    ('quasiquote
+                     (("python-pytest" ('unquote 'python-pytest)))))
+                   ('home-page "http://example.com")
+                   ('synopsis "summary")
+                   ('description "summary")
+                   ('license 'license:lgpl2.0))
+                 (string=? (bytevector->nix-base32-string
+                            test-source-hash)
+                           hash))
+                (x
+                 (pk 'fail x #f))))))
+
 (test-end "pypi")
-- 
2.33.0

[bug#50854] [PATCH 2/2] gnu: av-98: Set 'upstream-name' property.

[Xinglu Chen]

* gnu/packages/web-browsers.scm (av-98)[properties]: Set ‘upstream-name’.
---
 gnu/packages/web-browsers.scm | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/gnu/packages/web-browsers.scm b/gnu/packages/web-browsers.scm
index f0aefa3eb4..596d9f7f2b 100644
--- a/gnu/packages/web-browsers.scm
+++ b/gnu/packages/web-browsers.scm
@@ -912,6 +912,8 @@ (define-public av-98
   (package
     (name "av-98")
     (version "1.0.1")
+    (properties
+     '((upstream-name . "AV-98")))
     (source
      (origin
        (method url-fetch)
-- 
2.33.0

bug#50854: [PATCH 0/2] Make the PyPI importer honor the ‘upstream-name’ package property

[Ludovic Courtès]

Xinglu Chen <public@yoctocell.xyz> skribis:

> If a PyPI package contains a “-” followed by one or more digits, e.g.,
> “AV-98”, the PyPI importer/updater will think that the digits after the
> “-” is the version of the package, and instead of trying to update
> “AV-98” it will try to update “AV”.
>
> $ ./pre-inst-env guix refresh av-98
> following redirection to `https://pypi.org/pypi/av/json'...
> /home/yoctocell/src/guix/gnu/packages/web-browsers.scm:914:13: av-98 would be upgraded from 1.0.1 to 8.0.3
>
> The first patch makes the PyPI importer honor the ‘upstream-name’
> property; it will also generate a package with an ‘upstream-name’
> property if the package name contains a “-” followed by one or more
> digits.
>
> The second patch sets the ‘upstream-name’ property for the ‘av-98’
> package.

Nice.

> Xinglu Chen (2):
>   import: pypi: Honor the 'upstream-name' package property.
>   gnu: av-98: Set 'upstream-name' property.

Applied, thanks!

Ludo'.