[PATCH 0/5] Improve the pypi updater.

[PATCH 0/5] Improve the pypi updater.

[Cyril Roelandt]

A couple of minor improvements, and a huge one: we should now be able to
automatically compute the test requirements of our Python packages!

Cyril.
--

Cyril Roelandt (5):
  import: pypi: Move runtime dependencies to the propagated-inputs.
  import: pypi: Remove setuptools from the inputs.
  import: pypi: Compute test requirements when using wheels.
  import: pypi: Compute test requirements when reading requirements
    files.
  import: pypi: Add more tests.

 guix/import/pypi.scm | 118 ++++++++++++++++++++---------------
 tests/pypi.scm       | 170 +++++++++++++++++++++++++++++++++++++++++++++++++--
 2 files changed, 234 insertions(+), 54 deletions(-)

-- 
2.6.2

[PATCH 1/5] import: pypi: Move runtime dependencies to the propagated-inputs.

[Cyril Roelandt]

* guix/import/pypi.scm (maybe-inputs): Pass INPUT-TYPE as a parameter.
* guix/import/pypi.scm (make-pypi-sexp): Use this parameter.
* tests/pypi.scm ("pypi->guix-package", "pypi->guix-package, wheels"): Update
accordingly.
---
 guix/import/pypi.scm | 9 +++++----
 tests/pypi.scm       | 4 ++--
 2 files changed, 7 insertions(+), 6 deletions(-)

diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm
index efa6908..474974b 100644
--- a/guix/import/pypi.scm
+++ b/guix/import/pypi.scm
@@ -103,14 +103,14 @@ package on PyPI."
     ((name version _ ...)
      (string-append name "-" version ".dist-info"))))
 
-(define (maybe-inputs package-inputs)
-  "Given a list of PACKAGE-INPUTS, tries to generate the 'inputs' field of a
+(define (maybe-inputs input-type package-inputs)
+  "Given a list of PACKAGE-INPUTS, tries to generate the INPUT-TYPE field of a
 package definition."
   (match package-inputs
     (()
      '())
     ((package-inputs ...)
-     `((inputs (,'quasiquote ,package-inputs))))))
+     `((,input-type (,'quasiquote ,package-inputs))))))
 
 (define (guess-requirements source-url wheel-url tarball)
   "Given SOURCE-URL, WHEEL-URL and a TARBALL of the package, return a list of
@@ -266,7 +266,8 @@ VERSION, SOURCE-URL, HOME-PAGE, SYNOPSIS, DESCRIPTION, and LICENSE."
                         (base32
                          ,(guix-hash-url temp)))))
              (build-system python-build-system)
-             ,@(maybe-inputs (compute-inputs source-url wheel-url temp))
+             ,@(maybe-inputs 'propagated-inputs
+                             (compute-inputs source-url wheel-url temp))
              (home-page ,home-page)
              (synopsis ,synopsis)
              (description ,description)
diff --git a/tests/pypi.scm b/tests/pypi.scm
index 379c288..ef43d50 100644
--- a/tests/pypi.scm
+++ b/tests/pypi.scm
@@ -106,7 +106,7 @@ baz > 13.37")
                      ('base32
                       (? string? hash)))))
          ('build-system 'python-build-system)
-         ('inputs
+         ('propagated-inputs
           ('quasiquote
            (("python-bar" ('unquote 'python-bar))
             ("python-baz" ('unquote 'python-baz))
@@ -166,7 +166,7 @@ baz > 13.37")
                      ('base32
                       (? string? hash)))))
          ('build-system 'python-build-system)
-         ('inputs
+         ('propagated-inputs
           ('quasiquote
            (("python-bar" ('unquote 'python-bar))
             ("python-baz" ('unquote 'python-baz))
-- 
2.6.2

[PATCH 2/5] import: pypi: Remove setuptools from the inputs.

[Cyril Roelandt]

* guix/import/pypi.scm (compute-inputs): remove python-setuptools from the
inputs.
* tests/pypi.scm ("pypi->guix-package", "pypi->guix-package, wheels"): Update
accordingly.
---
 guix/import/pypi.scm | 7 +++----
 tests/pypi.scm       | 6 ++----
 2 files changed, 5 insertions(+), 8 deletions(-)

diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm
index 474974b..af9b108 100644
--- a/guix/import/pypi.scm
+++ b/guix/import/pypi.scm
@@ -232,10 +232,9 @@ name/variable pairs describing the required inputs of this package."
   (sort
     (map (lambda (input)
            (list input (list 'unquote (string->symbol input))))
-         (append '("python-setuptools")
-                 ;; Argparse has been part of Python since 2.7.
-                 (remove (cut string=? "python-argparse" <>)
-                         (guess-requirements source-url wheel-url tarball))))
+         ;; Argparse has been part of Python since 2.7.
+         (remove (cut string=? "python-argparse" <>)
+                 (guess-requirements source-url wheel-url tarball)))
     (lambda args
       (match args
         (((a _ ...) (b _ ...))
diff --git a/tests/pypi.scm b/tests/pypi.scm
index ef43d50..7f0ad23 100644
--- a/tests/pypi.scm
+++ b/tests/pypi.scm
@@ -109,8 +109,7 @@ baz > 13.37")
          ('propagated-inputs
           ('quasiquote
            (("python-bar" ('unquote 'python-bar))
-            ("python-baz" ('unquote 'python-baz))
-            ("python-setuptools" ('unquote 'python-setuptools)))))
+            ("python-baz" ('unquote 'python-baz)))))
          ('home-page "http://example.com")
          ('synopsis "summary")
          ('description "summary")
@@ -169,8 +168,7 @@ baz > 13.37")
          ('propagated-inputs
           ('quasiquote
            (("python-bar" ('unquote 'python-bar))
-            ("python-baz" ('unquote 'python-baz))
-            ("python-setuptools" ('unquote 'python-setuptools)))))
+            ("python-baz" ('unquote 'python-baz)))))
          ('home-page "http://example.com")
          ('synopsis "summary")
          ('description "summary")
-- 
2.6.2

[PATCH 3/5] import: pypi: Compute test requirements when using wheels.

[Cyril Roelandt]

* guix/import/pypi.scm (read-wheel-metadata): Read test requirements from the
metadata included in the wheel.
* guix/import/pypi.scm (guess-requirement-from-source): Return an empty list
for test requirements.
* guix/import/pypi.scm (compute-inputs): Return both the runtime requirements
and the test requirements.
* guix/import.pypi.scm (make-pypi-sexp): Compute native inputs.
* tests/pypi.scm ("pypi->guix-package, wheels"): Update accordingly.
---
 guix/import/pypi.scm | 96 +++++++++++++++++++++++++++++-----------------------
 tests/pypi.scm       | 12 +++++++
 2 files changed, 66 insertions(+), 42 deletions(-)

diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm
index af9b108..f43d7d5 100644
--- a/guix/import/pypi.scm
+++ b/guix/import/pypi.scm
@@ -175,13 +175,21 @@ cannot determine package dependencies"))
                  (lambda (port)
                    (let* ((metadata (json->scm port))
                           (run_requires (hash-ref metadata "run_requires"))
+                          (test_requires (hash-ref metadata "test_requires"))
                           (requirements (if run_requires
                                             (hash-ref (list-ref run_requires 0)
                                                        "requires")
-                                            '())))
-                     (map (lambda (r)
-                            (python->package-name (clean-requirement r)))
-                          requirements)))))
+                                            '()))
+                          (test-requirements (if test_requires
+                                                 (hash-ref (list-ref test_requires 0)
+                                                            "requires")
+                                                 '())))
+                     (list (map (lambda (r)
+                                  (python->package-name (clean-requirement r)))
+                                requirements)
+                           (map (lambda (r)
+                                  (python->package-name (clean-requirement r)))
+                                test-requirements))))))
              (lambda ()
                (delete-file json-file)
                (rmdir dirname))))))
@@ -208,15 +216,16 @@ cannot determine package dependencies"))
                 (dynamic-wind
                   (const #t)
                   (lambda ()
-                    (read-requirements req-file))
+                    (list (read-requirements req-file)
+                          '()))
                   (lambda ()
                     (delete-file req-file)
                     (rmdir dirname)))
                 (begin
                   (warning (_ "'tar xf' failed with exit code ~a\n")
                            exit-code)
-                  '())))
-          '())))
+                  (list '() '()))))
+          (list '() '()))))
 
   ;; First, try to compute the requirements using the wheel, since that is the
   ;; most reliable option. If a wheel is not provided for this package, try
@@ -228,17 +237,21 @@ cannot determine package dependencies"))
 
 (define (compute-inputs source-url wheel-url tarball)
   "Given the SOURCE-URL of an already downloaded TARBALL, return a list of
-name/variable pairs describing the required inputs of this package."
-  (sort
-    (map (lambda (input)
-           (list input (list 'unquote (string->symbol input))))
-         ;; Argparse has been part of Python since 2.7.
-         (remove (cut string=? "python-argparse" <>)
-                 (guess-requirements source-url wheel-url tarball)))
-    (lambda args
-      (match args
-        (((a _ ...) (b _ ...))
-         (string-ci<? a b))))))
+two lists: the runtime and test requirements of this package. Each element
+is a list of name/variable pairs describing the required inputs of this
+package."
+  (map (lambda (inputs)
+         (sort
+           (map (lambda (input)
+                  (list input (list 'unquote (string->symbol input))))
+                        ;; Argparse has been part of Python since 2.7.
+                (remove (cut string=? "python-argparse" <>)
+                        inputs))
+           (lambda args
+             (match args
+               (((a _ ...) (b _ ...))
+                (string-ci<? a b))))))
+       (guess-requirements source-url wheel-url tarball)))
 
 (define (make-pypi-sexp name version source-url wheel-url home-page synopsis
                         description license)
@@ -247,30 +260,29 @@ VERSION, SOURCE-URL, HOME-PAGE, SYNOPSIS, DESCRIPTION, and LICENSE."
   (call-with-temporary-output-file
    (lambda (temp port)
      (and (url-fetch source-url temp)
-          `(package
-             (name ,(python->package-name name))
-             (version ,version)
-             (source (origin
-                       (method url-fetch)
-
-                       ;; Sometimes 'pypi-uri' doesn't quite work due to mixed
-                       ;; cases in NAME, for instance, as is the case with
-                       ;; "uwsgi".  In that case, fall back to a full URL.
-                       (uri ,(if (equal? (pypi-uri name version) source-url)
-                                 `(pypi-uri ,name version)
-                                 `(string-append
-                                   ,@(factorize-uri source-url version))))
-
-                       (sha256
-                        (base32
-                         ,(guix-hash-url temp)))))
-             (build-system python-build-system)
-             ,@(maybe-inputs 'propagated-inputs
-                             (compute-inputs source-url wheel-url temp))
-             (home-page ,home-page)
-             (synopsis ,synopsis)
-             (description ,description)
-             (license ,(license->symbol license)))))))
+          (let ((inputs (compute-inputs source-url wheel-url temp)))
+            `(package
+               (name ,(python->package-name name))
+               (version ,version)
+               (source (origin
+                         (method url-fetch)
+                         ;; Sometimes 'pypi-uri' doesn't quite work due to mixed
+                         ;; cases in NAME, for instance, as is the case with
+                         ;; "uwsgi".  In that case, fall back to a full URL.
+                         (uri ,(if (equal? (pypi-uri name version) source-url)
+                                   `(pypi-uri ,name version)
+                                   `(string-append
+                                     ,@(factorize-uri source-url version))))
+                         (sha256
+                          (base32
+                           ,(guix-hash-url temp)))))
+               (build-system python-build-system)
+               ,@(maybe-inputs 'propagated-inputs (list-ref inputs 0))
+               ,@(maybe-inputs 'native-inputs (list-ref inputs 1))
+               (home-page ,home-page)
+               (synopsis ,synopsis)
+               (description ,description)
+               (license ,(license->symbol license))))))))
 
 (define (pypi->guix-package package-name)
   "Fetch the metadata for PACKAGE-NAME from pypi.python.org, and return the
diff --git a/tests/pypi.scm b/tests/pypi.scm
index 7f0ad23..cbf7066 100644
--- a/tests/pypi.scm
+++ b/tests/pypi.scm
@@ -68,6 +68,14 @@ baz > 13.37")
         \"baz (>13.37)\"
       ]
     }
+  ],
+  \"test_requires\": [
+    {
+      \"requires\": [
+        \"test-bar\",
+        \"test-baz (>42.42)\"
+      ]
+    }
   ]
 }")
 
@@ -169,6 +177,10 @@ baz > 13.37")
           ('quasiquote
            (("python-bar" ('unquote 'python-bar))
             ("python-baz" ('unquote 'python-baz)))))
+         ('native-inputs
+          ('quasiquote
+           (("python-test-bar" ('unquote 'python-test-bar))
+            ("python-test-baz" ('unquote 'python-test-baz)))))
          ('home-page "http://example.com")
          ('synopsis "summary")
          ('description "summary")
-- 
2.6.2

[PATCH 4/5] import: pypi: Compute test requirements when reading requirements files.

[Cyril Roelandt]

* guix/import/pypi.scm (guess-requirement-from-source): Read
test-requirements.txt as well as requirements.txt.
* tests/pypi.scm ("pypi->guix-package"): Update accordingly.
---
 guix/import/pypi.scm | 20 +++++++++++++++-----
 tests/pypi.scm       | 11 +++++++++++
 2 files changed, 26 insertions(+), 5 deletions(-)

diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm
index f43d7d5..917713d 100644
--- a/guix/import/pypi.scm
+++ b/guix/import/pypi.scm
@@ -210,16 +210,26 @@ cannot determine package dependencies"))
     (let ((dirname (tarball-directory source-url)))
       (if (string? dirname)
           (let* ((req-file (string-append dirname "/requirements.txt"))
-                 (exit-code (system* "tar" "xf" tarball req-file)))
+                 (test-req-file (string-append dirname "/test-requirements.txt"))
+                 (exit-code (system* "tar" "xf" tarball req-file))
+                 (test-exit-code (system* "tar" "xf" tarball test-req-file)))
             ;; TODO: support more formats.
-            (if (zero? exit-code)
+            (if (or (zero? exit-code)
+                    (zero? test-exit-code))
                 (dynamic-wind
                   (const #t)
                   (lambda ()
-                    (list (read-requirements req-file)
-                          '()))
+                    (list (if (zero? exit-code)
+                              (read-requirements req-file)
+                              '())
+                          (if (zero? test-exit-code)
+                              (read-requirements test-req-file)
+                              '())))
                   (lambda ()
-                    (delete-file req-file)
+                    (when (zero? exit-code)
+                          (delete-file req-file))
+                    (when (zero? test-exit-code)
+                          (delete-file test-req-file))
                     (rmdir dirname)))
                 (begin
                   (warning (_ "'tar xf' failed with exit code ~a\n")
diff --git a/tests/pypi.scm b/tests/pypi.scm
index cbf7066..7a38d6e 100644
--- a/tests/pypi.scm
+++ b/tests/pypi.scm
@@ -59,6 +59,10 @@
 bar
 baz > 13.37")
 
+(define test-test-requirements
+"test-bar
+test-baz > 42.42")
+
 (define test-metadata
   "{
   \"run_requires\": [
@@ -96,6 +100,9 @@ baz > 13.37")
                  (with-output-to-file "foo-1.0.0/requirements.txt"
                    (lambda ()
                      (display test-requirements)))
+                 (with-output-to-file "foo-1.0.0/test-requirements.txt"
+                   (lambda ()
+                     (display test-test-requirements)))
                  (system* "tar" "czvf" file-name "foo-1.0.0/")
                  (delete-file-recursively "foo-1.0.0")
                  (set! test-source-hash
@@ -118,6 +125,10 @@ baz > 13.37")
           ('quasiquote
            (("python-bar" ('unquote 'python-bar))
             ("python-baz" ('unquote 'python-baz)))))
+         ('native-inputs
+          ('quasiquote
+           (("python-test-bar" ('unquote 'python-test-bar))
+            ("python-test-baz" ('unquote 'python-test-baz)))))
          ('home-page "http://example.com")
          ('synopsis "summary")
          ('description "summary")
-- 
2.6.2

[PATCH 5/5] import: pypi: Add more tests.

[Cyril Roelandt]

* tests/pypi.scm ("pypi->guix-package, no requirements", "pypi->guix-package,
no test requirements", "pypi->guix-package, no runtime requirements"): New
tests.
---
 tests/pypi.scm | 137 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 137 insertions(+)

diff --git a/tests/pypi.scm b/tests/pypi.scm
index 7a38d6e..dd516c5 100644
--- a/tests/pypi.scm
+++ b/tests/pypi.scm
@@ -85,6 +85,143 @@ test-baz > 42.42")
 
 (test-begin "pypi")
 
+(test-assert "pypi->guix-package, no requirements"
+  ;; Replace network resources with sample data.
+  (mock ((guix import utils) url-fetch
+         (lambda (url file-name)
+           (match url
+             ("https://pypi.python.org/pypi/foo/json"
+              (with-output-to-file file-name
+                (lambda ()
+                  (display test-json))))
+             ("https://example.com/foo-1.0.0.tar.gz"
+               (begin
+                 (mkdir "foo-1.0.0")
+                 ;; Do not create any requirements files.
+                 (system* "tar" "czvf" file-name "foo-1.0.0/")
+                 (delete-file-recursively "foo-1.0.0")
+                 (set! test-source-hash
+                       (call-with-input-file file-name port-sha256))))
+             ("https://example.com/foo-1.0.0-py2.py3-none-any.whl" #f)
+             (_ (error "Unexpected URL: " url)))))
+    (match (pypi->guix-package "foo")
+      (('package
+         ('name "python-foo")
+         ('version "1.0.0")
+         ('source ('origin
+                    ('method 'url-fetch)
+                    ('uri (string-append "https://example.com/foo-"
+                                         version ".tar.gz"))
+                    ('sha256
+                     ('base32
+                      (? string? hash)))))
+         ('build-system 'python-build-system)
+         ('home-page "http://example.com")
+         ('synopsis "summary")
+         ('description "summary")
+         ('license 'lgpl2.0))
+       (string=? (bytevector->nix-base32-string
+                  test-source-hash)
+                 hash))
+      (x
+       (pk 'fail x #f)))))
+
+(test-assert "pypi->guix-package, no test requirements"
+  ;; Replace network resources with sample data.
+  (mock ((guix import utils) url-fetch
+         (lambda (url file-name)
+           (match url
+             ("https://pypi.python.org/pypi/foo/json"
+              (with-output-to-file file-name
+                (lambda ()
+                  (display test-json))))
+             ("https://example.com/foo-1.0.0.tar.gz"
+               (begin
+                 (mkdir "foo-1.0.0")
+                  ;; Create only the runtime requirements.
+                 (with-output-to-file "foo-1.0.0/requirements.txt"
+                   (lambda ()
+                     (display test-requirements)))
+                 (system* "tar" "czvf" file-name "foo-1.0.0/")
+                 (delete-file-recursively "foo-1.0.0")
+                 (set! test-source-hash
+                       (call-with-input-file file-name port-sha256))))
+             ("https://example.com/foo-1.0.0-py2.py3-none-any.whl" #f)
+             (_ (error "Unexpected URL: " url)))))
+    (match (pypi->guix-package "foo")
+      (('package
+         ('name "python-foo")
+         ('version "1.0.0")
+         ('source ('origin
+                    ('method 'url-fetch)
+                    ('uri (string-append "https://example.com/foo-"
+                                         version ".tar.gz"))
+                    ('sha256
+                     ('base32
+                      (? string? hash)))))
+         ('build-system 'python-build-system)
+         ('propagated-inputs
+          ('quasiquote
+           (("python-bar" ('unquote 'python-bar))
+            ("python-baz" ('unquote 'python-baz)))))
+         ('home-page "http://example.com")
+         ('synopsis "summary")
+         ('description "summary")
+         ('license 'lgpl2.0))
+       (string=? (bytevector->nix-base32-string
+                  test-source-hash)
+                 hash))
+      (x
+       (pk 'fail x #f)))))
+
+(test-assert "pypi->guix-package, no runtime requirements"
+  ;; Replace network resources with sample data.
+  (mock ((guix import utils) url-fetch
+         (lambda (url file-name)
+           (match url
+             ("https://pypi.python.org/pypi/foo/json"
+              (with-output-to-file file-name
+                (lambda ()
+                  (display test-json))))
+             ("https://example.com/foo-1.0.0.tar.gz"
+               (begin
+                 (mkdir "foo-1.0.0")
+                 ;; Create only the test requirements.
+                 (with-output-to-file "foo-1.0.0/test-requirements.txt"
+                   (lambda ()
+                     (display test-test-requirements)))
+                 (system* "tar" "czvf" file-name "foo-1.0.0/")
+                 (delete-file-recursively "foo-1.0.0")
+                 (set! test-source-hash
+                       (call-with-input-file file-name port-sha256))))
+             ("https://example.com/foo-1.0.0-py2.py3-none-any.whl" #f)
+             (_ (error "Unexpected URL: " url)))))
+    (match (pypi->guix-package "foo")
+      (('package
+         ('name "python-foo")
+         ('version "1.0.0")
+         ('source ('origin
+                    ('method 'url-fetch)
+                    ('uri (string-append "https://example.com/foo-"
+                                         version ".tar.gz"))
+                    ('sha256
+                     ('base32
+                      (? string? hash)))))
+         ('build-system 'python-build-system)
+         ('native-inputs
+          ('quasiquote
+           (("python-test-bar" ('unquote 'python-test-bar))
+            ("python-test-baz" ('unquote 'python-test-baz)))))
+         ('home-page "http://example.com")
+         ('synopsis "summary")
+         ('description "summary")
+         ('license 'lgpl2.0))
+       (string=? (bytevector->nix-base32-string
+                  test-source-hash)
+                 hash))
+      (x
+       (pk 'fail x #f)))))
+
 (test-assert "pypi->guix-package"
   ;; Replace network resources with sample data.
   (mock ((guix import utils) url-fetch
-- 
2.6.2

Re: [PATCH 0/5] Improve the pypi updater.

[Hartmut Goebel]

Am 16.07.2016 um 17:23 schrieb Cyril Roelandt:
> and a huge one: we should now be able to
> automatically compute the test requirements of our Python packages!
Could you please add some documentation to the manual, too? Thanks.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: [PATCH 0/5] Improve the pypi updater.

[Ludovic Courtès]

Hello!

Cyril Roelandt <tipecaml@gmail.com> skribis:

> A couple of minor improvements, and a huge one: we should now be able to
> automatically compute the test requirements of our Python packages!

Neat!  So the test dependencies go to ‘inputs’ instead of
‘propagated-inputs’, right?  That can definitely save time and avoid
errors.

Is there an example package that illustrates this?

> * guix/import/pypi.scm (maybe-inputs): Pass INPUT-TYPE as a parameter.
> * guix/import/pypi.scm (make-pypi-sexp): Use this parameter.

Nitpick: no need to repeat the file name.

> * tests/pypi.scm ("pypi->guix-package", "pypi->guix-package, wheels"): Update
> accordingly.

LGTM!

Thanks,
Ludo’.

Re: [PATCH 2/5] import: pypi: Remove setuptools from the inputs.

[Ludovic Courtès]

Cyril Roelandt <tipecaml@gmail.com> skribis:

> * guix/import/pypi.scm (compute-inputs): remove python-setuptools from the
> inputs.
> * tests/pypi.scm ("pypi->guix-package", "pypi->guix-package, wheels"): Update
> accordingly.

IIRC python-setuptools is needed for Python 2.x only, right?  I keep
forgetting what the deal is.

Ludo’.

Re: [PATCH 3/5] import: pypi: Compute test requirements when using wheels.

[Ludovic Courtès]

Cyril Roelandt <tipecaml@gmail.com> skribis:

> * guix/import/pypi.scm (read-wheel-metadata): Read test requirements from the
> metadata included in the wheel.
> * guix/import/pypi.scm (guess-requirement-from-source): Return an empty list
> for test requirements.
> * guix/import/pypi.scm (compute-inputs): Return both the runtime requirements
> and the test requirements.
> * guix/import.pypi.scm (make-pypi-sexp): Compute native inputs.
> * tests/pypi.scm ("pypi->guix-package, wheels"): Update accordingly.

[...]

>                            (requirements (if run_requires
>                                              (hash-ref (list-ref run_requires 0)
>                                                         "requires")
> -                                            '())))
> -                     (map (lambda (r)
> -                            (python->package-name (clean-requirement r)))
> -                          requirements)))))
> +                                            '()))
> +                          (test-requirements (if test_requires
> +                                                 (hash-ref (list-ref test_requires 0)
> +                                                            "requires")
> +                                                 '())))

Maybe for a subsequent patch: these two occurrences could be written as

  (match test_requires
    ((head _ ...)
     (hash-ref head "requires"))
    (_
     '()))

LGTM, thanks!

Ludo’.

Re: [PATCH 4/5] import: pypi: Compute test requirements when reading requirements files.

[Ludovic Courtès]

Cyril Roelandt <tipecaml@gmail.com> skribis:

> * guix/import/pypi.scm (guess-requirement-from-source): Read
> test-requirements.txt as well as requirements.txt.
> * tests/pypi.scm ("pypi->guix-package"): Update accordingly.

[...]

>            (let* ((req-file (string-append dirname "/requirements.txt"))
> -                 (exit-code (system* "tar" "xf" tarball req-file)))
> +                 (test-req-file (string-append dirname "/test-requirements.txt"))
> +                 (exit-code (system* "tar" "xf" tarball req-file))
> +                 (test-exit-code (system* "tar" "xf" tarball test-req-file)))
>              ;; TODO: support more formats.
> -            (if (zero? exit-code)
> +            (if (or (zero? exit-code)
> +                    (zero? test-exit-code))

This seems to suggest that this could be factorized somehow.  Maybe
unpack once and read the two files at once?

Ludo’.

Re: [PATCH 5/5] import: pypi: Add more tests.

[Ludovic Courtès]

Cyril Roelandt <tipecaml@gmail.com> skribis:

> * tests/pypi.scm ("pypi->guix-package, no requirements", "pypi->guix-package,
> no test requirements", "pypi->guix-package, no runtime requirements"): New
> tests.

Perfect.  :-)

Thank you!

Ludo'.

Re: [PATCH 2/5] import: pypi: Remove setuptools from the inputs.

[Leo Famulari]

On Fri, Jul 22, 2016 at 11:24:52PM +0200, Ludovic Courtès wrote:
> Cyril Roelandt <tipecaml@gmail.com> skribis:
> 
> > * guix/import/pypi.scm (compute-inputs): remove python-setuptools from the
> > inputs.
> > * tests/pypi.scm ("pypi->guix-package", "pypi->guix-package, wheels"): Update
> > accordingly.
> 
> IIRC python-setuptools is needed for Python 2.x only, right?  I keep
> forgetting what the deal is.

In my experience, it is more often required for Python 2.x, but
sometimes it is required by Python 3.x packages, and sometimes not for
Python 2.x packages.

I'd be happy to get a more systematic explanation, or to be shown wrong.
A simple rule would be great :)

Re: [PATCH 2/5] import: pypi: Remove setuptools from the inputs.

[Cyril Roelandt]

On 07/22/2016 11:43 PM, Leo Famulari wrote:
> In my experience, it is more often required for Python 2.x, but
> sometimes it is required by Python 3.x packages, and sometimes not for
> Python 2.x packages.
> 
> I'd be happy to get a more systematic explanation, or to be shown wrong.
> A simple rule would be great :)
> 

Yes, we usually do not need to include python-setuptools in the native
inputs of our Python 3 packages, we just add it to the Python 2 version.

Cyril.

Re: [PATCH 4/5] import: pypi: Compute test requirements when reading requirements files.

[Cyril Roelandt]

On 07/22/2016 11:30 PM, Ludovic Courtès wrote:
> This seems to suggest that this could be factorized somehow.  Maybe
> unpack once and read the two files at once?

The problem is that both files might not be there, and unzip will return
a non-zero exit code if any of them is missing, so it seems easier to
just run unzip twice. WDYT?

Cyril.

Re: [PATCH 0/5] Improve the pypi updater.

[Cyril Roelandt]

On 07/22/2016 11:22 PM, Ludovic Courtès wrote:
> Neat!  So the test dependencies go to ‘inputs’ instead of
> ‘propagated-inputs’, right?  That can definitely save time and avoid
> errors.
> 

Indeed, the runtime dependencies are propagated inputs, and the test
dependencies are native inputs. This is what we currently do when
writing packages "by hand".

> Is there an example package that illustrates this?

Anything from OpenStack :-D

Cyril.

Re: [PATCH 2/5] import: pypi: Remove setuptools from the inputs.

[Ludovic Courtès]

Cyril Roelandt <tipecaml@gmail.com> skribis:

> On 07/22/2016 11:43 PM, Leo Famulari wrote:
>> In my experience, it is more often required for Python 2.x, but
>> sometimes it is required by Python 3.x packages, and sometimes not for
>> Python 2.x packages.
>> 
>> I'd be happy to get a more systematic explanation, or to be shown wrong.
>> A simple rule would be great :)
>> 
>
> Yes, we usually do not need to include python-setuptools in the native
> inputs of our Python 3 packages, we just add it to the Python 2 version.

OK, thanks for explaining.  This patch looks good!

Ludo’.

Re: [PATCH 4/5] import: pypi: Compute test requirements when reading requirements files.

[Ludovic Courtès]

Cyril Roelandt <tipecaml@gmail.com> skribis:

> On 07/22/2016 11:30 PM, Ludovic Courtès wrote:
>> This seems to suggest that this could be factorized somehow.  Maybe
>> unpack once and read the two files at once?
>
> The problem is that both files might not be there, and unzip will return
> a non-zero exit code if any of them is missing, so it seems easier to
> just run unzip twice. WDYT?

OK, I see.  Then what about adding a procedure like:

  (define (file-from-zip-archive archive file)
    "Return the contents of FILE from ARCHIVE as a string, or #f if FILE
  could not be found in ARCHIVE or extraction failed."
    ;; … invoke unzip in temporary dir, check return code, then:
    (call-with-input-file (string-append tempdir "/" file)
      get-string-all))

That would move the bits about exit codes and all that out of the main
logic.

WDYT?

Thanks,
Ludo’.

Re: [PATCH 0/5] Improve the pypi updater.

[Ludovic Courtès]

Cyril Roelandt <tipecaml@gmail.com> skribis:

> On 07/22/2016 11:22 PM, Ludovic Courtès wrote:
>> Neat!  So the test dependencies go to ‘inputs’ instead of
>> ‘propagated-inputs’, right?  That can definitely save time and avoid
>> errors.
>> 
>
> Indeed, the runtime dependencies are propagated inputs, and the test
> dependencies are native inputs. This is what we currently do when
> writing packages "by hand".

OK.

>> Is there an example package that illustrates this?
>
> Anything from OpenStack :-D

Like ‘guix import pypi hacking’ or other packages found in
openstack.scm, right?

Thanks,
Ludo’.