all messages for Guix-related lists mirrored at yhetil.org
 help / color / mirror / code / Atom feed
* [PATCH 0/5] Improve the pypi updater.
@ 2016-07-16 15:23 Cyril Roelandt
  2016-07-16 15:23 ` [PATCH 1/5] import: pypi: Move runtime dependencies to the propagated-inputs Cyril Roelandt
                   ` (6 more replies)
  0 siblings, 7 replies; 19+ messages in thread
From: Cyril Roelandt @ 2016-07-16 15:23 UTC (permalink / raw)
  To: guix-devel

A couple of minor improvements, and a huge one: we should now be able to
automatically compute the test requirements of our Python packages!

Cyril.
--

Cyril Roelandt (5):
  import: pypi: Move runtime dependencies to the propagated-inputs.
  import: pypi: Remove setuptools from the inputs.
  import: pypi: Compute test requirements when using wheels.
  import: pypi: Compute test requirements when reading requirements
    files.
  import: pypi: Add more tests.

 guix/import/pypi.scm | 118 ++++++++++++++++++++---------------
 tests/pypi.scm       | 170 +++++++++++++++++++++++++++++++++++++++++++++++++--
 2 files changed, 234 insertions(+), 54 deletions(-)

-- 
2.6.2

^ permalink raw reply	[flat|nested] 19+ messages in thread

* [PATCH 1/5] import: pypi: Move runtime dependencies to the propagated-inputs.
  2016-07-16 15:23 [PATCH 0/5] Improve the pypi updater Cyril Roelandt
@ 2016-07-16 15:23 ` Cyril Roelandt
  2016-07-16 15:23 ` [PATCH 2/5] import: pypi: Remove setuptools from the inputs Cyril Roelandt
                   ` (5 subsequent siblings)
  6 siblings, 0 replies; 19+ messages in thread
From: Cyril Roelandt @ 2016-07-16 15:23 UTC (permalink / raw)
  To: guix-devel

* guix/import/pypi.scm (maybe-inputs): Pass INPUT-TYPE as a parameter.
* guix/import/pypi.scm (make-pypi-sexp): Use this parameter.
* tests/pypi.scm ("pypi->guix-package", "pypi->guix-package, wheels"): Update
accordingly.
---
 guix/import/pypi.scm | 9 +++++----
 tests/pypi.scm       | 4 ++--
 2 files changed, 7 insertions(+), 6 deletions(-)

diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm
index efa6908..474974b 100644
--- a/guix/import/pypi.scm
+++ b/guix/import/pypi.scm
@@ -103,14 +103,14 @@ package on PyPI."
     ((name version _ ...)
      (string-append name "-" version ".dist-info"))))
 
-(define (maybe-inputs package-inputs)
-  "Given a list of PACKAGE-INPUTS, tries to generate the 'inputs' field of a
+(define (maybe-inputs input-type package-inputs)
+  "Given a list of PACKAGE-INPUTS, tries to generate the INPUT-TYPE field of a
 package definition."
   (match package-inputs
     (()
      '())
     ((package-inputs ...)
-     `((inputs (,'quasiquote ,package-inputs))))))
+     `((,input-type (,'quasiquote ,package-inputs))))))
 
 (define (guess-requirements source-url wheel-url tarball)
   "Given SOURCE-URL, WHEEL-URL and a TARBALL of the package, return a list of
@@ -266,7 +266,8 @@ VERSION, SOURCE-URL, HOME-PAGE, SYNOPSIS, DESCRIPTION, and LICENSE."
                         (base32
                          ,(guix-hash-url temp)))))
              (build-system python-build-system)
-             ,@(maybe-inputs (compute-inputs source-url wheel-url temp))
+             ,@(maybe-inputs 'propagated-inputs
+                             (compute-inputs source-url wheel-url temp))
              (home-page ,home-page)
              (synopsis ,synopsis)
              (description ,description)
diff --git a/tests/pypi.scm b/tests/pypi.scm
index 379c288..ef43d50 100644
--- a/tests/pypi.scm
+++ b/tests/pypi.scm
@@ -106,7 +106,7 @@ baz > 13.37")
                      ('base32
                       (? string? hash)))))
          ('build-system 'python-build-system)
-         ('inputs
+         ('propagated-inputs
           ('quasiquote
            (("python-bar" ('unquote 'python-bar))
             ("python-baz" ('unquote 'python-baz))
@@ -166,7 +166,7 @@ baz > 13.37")
                      ('base32
                       (? string? hash)))))
          ('build-system 'python-build-system)
-         ('inputs
+         ('propagated-inputs
           ('quasiquote
            (("python-bar" ('unquote 'python-bar))
             ("python-baz" ('unquote 'python-baz))
-- 
2.6.2

^ permalink raw reply related	[flat|nested] 19+ messages in thread

* [PATCH 2/5] import: pypi: Remove setuptools from the inputs.
  2016-07-16 15:23 [PATCH 0/5] Improve the pypi updater Cyril Roelandt
  2016-07-16 15:23 ` [PATCH 1/5] import: pypi: Move runtime dependencies to the propagated-inputs Cyril Roelandt
@ 2016-07-16 15:23 ` Cyril Roelandt
  2016-07-22 21:24   ` Ludovic Courtès
  2016-07-16 15:23 ` [PATCH 3/5] import: pypi: Compute test requirements when using wheels Cyril Roelandt
                   ` (4 subsequent siblings)
  6 siblings, 1 reply; 19+ messages in thread
From: Cyril Roelandt @ 2016-07-16 15:23 UTC (permalink / raw)
  To: guix-devel

* guix/import/pypi.scm (compute-inputs): remove python-setuptools from the
inputs.
* tests/pypi.scm ("pypi->guix-package", "pypi->guix-package, wheels"): Update
accordingly.
---
 guix/import/pypi.scm | 7 +++----
 tests/pypi.scm       | 6 ++----
 2 files changed, 5 insertions(+), 8 deletions(-)

diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm
index 474974b..af9b108 100644
--- a/guix/import/pypi.scm
+++ b/guix/import/pypi.scm
@@ -232,10 +232,9 @@ name/variable pairs describing the required inputs of this package."
   (sort
     (map (lambda (input)
            (list input (list 'unquote (string->symbol input))))
-         (append '("python-setuptools")
-                 ;; Argparse has been part of Python since 2.7.
-                 (remove (cut string=? "python-argparse" <>)
-                         (guess-requirements source-url wheel-url tarball))))
+         ;; Argparse has been part of Python since 2.7.
+         (remove (cut string=? "python-argparse" <>)
+                 (guess-requirements source-url wheel-url tarball)))
     (lambda args
       (match args
         (((a _ ...) (b _ ...))
diff --git a/tests/pypi.scm b/tests/pypi.scm
index ef43d50..7f0ad23 100644
--- a/tests/pypi.scm
+++ b/tests/pypi.scm
@@ -109,8 +109,7 @@ baz > 13.37")
          ('propagated-inputs
           ('quasiquote
            (("python-bar" ('unquote 'python-bar))
-            ("python-baz" ('unquote 'python-baz))
-            ("python-setuptools" ('unquote 'python-setuptools)))))
+            ("python-baz" ('unquote 'python-baz)))))
          ('home-page "http://example.com")
          ('synopsis "summary")
          ('description "summary")
@@ -169,8 +168,7 @@ baz > 13.37")
          ('propagated-inputs
           ('quasiquote
            (("python-bar" ('unquote 'python-bar))
-            ("python-baz" ('unquote 'python-baz))
-            ("python-setuptools" ('unquote 'python-setuptools)))))
+            ("python-baz" ('unquote 'python-baz)))))
          ('home-page "http://example.com")
          ('synopsis "summary")
          ('description "summary")
-- 
2.6.2

^ permalink raw reply related	[flat|nested] 19+ messages in thread

* [PATCH 3/5] import: pypi: Compute test requirements when using wheels.
  2016-07-16 15:23 [PATCH 0/5] Improve the pypi updater Cyril Roelandt
  2016-07-16 15:23 ` [PATCH 1/5] import: pypi: Move runtime dependencies to the propagated-inputs Cyril Roelandt
  2016-07-16 15:23 ` [PATCH 2/5] import: pypi: Remove setuptools from the inputs Cyril Roelandt
@ 2016-07-16 15:23 ` Cyril Roelandt
  2016-07-22 21:28   ` Ludovic Courtès
  2016-07-16 15:23 ` [PATCH 4/5] import: pypi: Compute test requirements when reading requirements files Cyril Roelandt
                   ` (3 subsequent siblings)
  6 siblings, 1 reply; 19+ messages in thread
From: Cyril Roelandt @ 2016-07-16 15:23 UTC (permalink / raw)
  To: guix-devel

* guix/import/pypi.scm (read-wheel-metadata): Read test requirements from the
metadata included in the wheel.
* guix/import/pypi.scm (guess-requirement-from-source): Return an empty list
for test requirements.
* guix/import/pypi.scm (compute-inputs): Return both the runtime requirements
and the test requirements.
* guix/import.pypi.scm (make-pypi-sexp): Compute native inputs.
* tests/pypi.scm ("pypi->guix-package, wheels"): Update accordingly.
---
 guix/import/pypi.scm | 96 +++++++++++++++++++++++++++++-----------------------
 tests/pypi.scm       | 12 +++++++
 2 files changed, 66 insertions(+), 42 deletions(-)

diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm
index af9b108..f43d7d5 100644
--- a/guix/import/pypi.scm
+++ b/guix/import/pypi.scm
@@ -175,13 +175,21 @@ cannot determine package dependencies"))
                  (lambda (port)
                    (let* ((metadata (json->scm port))
                           (run_requires (hash-ref metadata "run_requires"))
+                          (test_requires (hash-ref metadata "test_requires"))
                           (requirements (if run_requires
                                             (hash-ref (list-ref run_requires 0)
                                                        "requires")
-                                            '())))
-                     (map (lambda (r)
-                            (python->package-name (clean-requirement r)))
-                          requirements)))))
+                                            '()))
+                          (test-requirements (if test_requires
+                                                 (hash-ref (list-ref test_requires 0)
+                                                            "requires")
+                                                 '())))
+                     (list (map (lambda (r)
+                                  (python->package-name (clean-requirement r)))
+                                requirements)
+                           (map (lambda (r)
+                                  (python->package-name (clean-requirement r)))
+                                test-requirements))))))
              (lambda ()
                (delete-file json-file)
                (rmdir dirname))))))
@@ -208,15 +216,16 @@ cannot determine package dependencies"))
                 (dynamic-wind
                   (const #t)
                   (lambda ()
-                    (read-requirements req-file))
+                    (list (read-requirements req-file)
+                          '()))
                   (lambda ()
                     (delete-file req-file)
                     (rmdir dirname)))
                 (begin
                   (warning (_ "'tar xf' failed with exit code ~a\n")
                            exit-code)
-                  '())))
-          '())))
+                  (list '() '()))))
+          (list '() '()))))
 
   ;; First, try to compute the requirements using the wheel, since that is the
   ;; most reliable option. If a wheel is not provided for this package, try
@@ -228,17 +237,21 @@ cannot determine package dependencies"))
 
 (define (compute-inputs source-url wheel-url tarball)
   "Given the SOURCE-URL of an already downloaded TARBALL, return a list of
-name/variable pairs describing the required inputs of this package."
-  (sort
-    (map (lambda (input)
-           (list input (list 'unquote (string->symbol input))))
-         ;; Argparse has been part of Python since 2.7.
-         (remove (cut string=? "python-argparse" <>)
-                 (guess-requirements source-url wheel-url tarball)))
-    (lambda args
-      (match args
-        (((a _ ...) (b _ ...))
-         (string-ci<? a b))))))
+two lists: the runtime and test requirements of this package. Each element
+is a list of name/variable pairs describing the required inputs of this
+package."
+  (map (lambda (inputs)
+         (sort
+           (map (lambda (input)
+                  (list input (list 'unquote (string->symbol input))))
+                        ;; Argparse has been part of Python since 2.7.
+                (remove (cut string=? "python-argparse" <>)
+                        inputs))
+           (lambda args
+             (match args
+               (((a _ ...) (b _ ...))
+                (string-ci<? a b))))))
+       (guess-requirements source-url wheel-url tarball)))
 
 (define (make-pypi-sexp name version source-url wheel-url home-page synopsis
                         description license)
@@ -247,30 +260,29 @@ VERSION, SOURCE-URL, HOME-PAGE, SYNOPSIS, DESCRIPTION, and LICENSE."
   (call-with-temporary-output-file
    (lambda (temp port)
      (and (url-fetch source-url temp)
-          `(package
-             (name ,(python->package-name name))
-             (version ,version)
-             (source (origin
-                       (method url-fetch)
-
-                       ;; Sometimes 'pypi-uri' doesn't quite work due to mixed
-                       ;; cases in NAME, for instance, as is the case with
-                       ;; "uwsgi".  In that case, fall back to a full URL.
-                       (uri ,(if (equal? (pypi-uri name version) source-url)
-                                 `(pypi-uri ,name version)
-                                 `(string-append
-                                   ,@(factorize-uri source-url version))))
-
-                       (sha256
-                        (base32
-                         ,(guix-hash-url temp)))))
-             (build-system python-build-system)
-             ,@(maybe-inputs 'propagated-inputs
-                             (compute-inputs source-url wheel-url temp))
-             (home-page ,home-page)
-             (synopsis ,synopsis)
-             (description ,description)
-             (license ,(license->symbol license)))))))
+          (let ((inputs (compute-inputs source-url wheel-url temp)))
+            `(package
+               (name ,(python->package-name name))
+               (version ,version)
+               (source (origin
+                         (method url-fetch)
+                         ;; Sometimes 'pypi-uri' doesn't quite work due to mixed
+                         ;; cases in NAME, for instance, as is the case with
+                         ;; "uwsgi".  In that case, fall back to a full URL.
+                         (uri ,(if (equal? (pypi-uri name version) source-url)
+                                   `(pypi-uri ,name version)
+                                   `(string-append
+                                     ,@(factorize-uri source-url version))))
+                         (sha256
+                          (base32
+                           ,(guix-hash-url temp)))))
+               (build-system python-build-system)
+               ,@(maybe-inputs 'propagated-inputs (list-ref inputs 0))
+               ,@(maybe-inputs 'native-inputs (list-ref inputs 1))
+               (home-page ,home-page)
+               (synopsis ,synopsis)
+               (description ,description)
+               (license ,(license->symbol license))))))))
 
 (define (pypi->guix-package package-name)
   "Fetch the metadata for PACKAGE-NAME from pypi.python.org, and return the
diff --git a/tests/pypi.scm b/tests/pypi.scm
index 7f0ad23..cbf7066 100644
--- a/tests/pypi.scm
+++ b/tests/pypi.scm
@@ -68,6 +68,14 @@ baz > 13.37")
         \"baz (>13.37)\"
       ]
     }
+  ],
+  \"test_requires\": [
+    {
+      \"requires\": [
+        \"test-bar\",
+        \"test-baz (>42.42)\"
+      ]
+    }
   ]
 }")
 
@@ -169,6 +177,10 @@ baz > 13.37")
           ('quasiquote
            (("python-bar" ('unquote 'python-bar))
             ("python-baz" ('unquote 'python-baz)))))
+         ('native-inputs
+          ('quasiquote
+           (("python-test-bar" ('unquote 'python-test-bar))
+            ("python-test-baz" ('unquote 'python-test-baz)))))
          ('home-page "http://example.com")
          ('synopsis "summary")
          ('description "summary")
-- 
2.6.2

^ permalink raw reply related	[flat|nested] 19+ messages in thread

* [PATCH 4/5] import: pypi: Compute test requirements when reading requirements files.
  2016-07-16 15:23 [PATCH 0/5] Improve the pypi updater Cyril Roelandt
                   ` (2 preceding siblings ...)
  2016-07-16 15:23 ` [PATCH 3/5] import: pypi: Compute test requirements when using wheels Cyril Roelandt
@ 2016-07-16 15:23 ` Cyril Roelandt
  2016-07-22 21:30   ` Ludovic Courtès
  2016-07-16 15:23 ` [PATCH 5/5] import: pypi: Add more tests Cyril Roelandt
                   ` (2 subsequent siblings)
  6 siblings, 1 reply; 19+ messages in thread
From: Cyril Roelandt @ 2016-07-16 15:23 UTC (permalink / raw)
  To: guix-devel

* guix/import/pypi.scm (guess-requirement-from-source): Read
test-requirements.txt as well as requirements.txt.
* tests/pypi.scm ("pypi->guix-package"): Update accordingly.
---
 guix/import/pypi.scm | 20 +++++++++++++++-----
 tests/pypi.scm       | 11 +++++++++++
 2 files changed, 26 insertions(+), 5 deletions(-)

diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm
index f43d7d5..917713d 100644
--- a/guix/import/pypi.scm
+++ b/guix/import/pypi.scm
@@ -210,16 +210,26 @@ cannot determine package dependencies"))
     (let ((dirname (tarball-directory source-url)))
       (if (string? dirname)
           (let* ((req-file (string-append dirname "/requirements.txt"))
-                 (exit-code (system* "tar" "xf" tarball req-file)))
+                 (test-req-file (string-append dirname "/test-requirements.txt"))
+                 (exit-code (system* "tar" "xf" tarball req-file))
+                 (test-exit-code (system* "tar" "xf" tarball test-req-file)))
             ;; TODO: support more formats.
-            (if (zero? exit-code)
+            (if (or (zero? exit-code)
+                    (zero? test-exit-code))
                 (dynamic-wind
                   (const #t)
                   (lambda ()
-                    (list (read-requirements req-file)
-                          '()))
+                    (list (if (zero? exit-code)
+                              (read-requirements req-file)
+                              '())
+                          (if (zero? test-exit-code)
+                              (read-requirements test-req-file)
+                              '())))
                   (lambda ()
-                    (delete-file req-file)
+                    (when (zero? exit-code)
+                          (delete-file req-file))
+                    (when (zero? test-exit-code)
+                          (delete-file test-req-file))
                     (rmdir dirname)))
                 (begin
                   (warning (_ "'tar xf' failed with exit code ~a\n")
diff --git a/tests/pypi.scm b/tests/pypi.scm
index cbf7066..7a38d6e 100644
--- a/tests/pypi.scm
+++ b/tests/pypi.scm
@@ -59,6 +59,10 @@
 bar
 baz > 13.37")
 
+(define test-test-requirements
+"test-bar
+test-baz > 42.42")
+
 (define test-metadata
   "{
   \"run_requires\": [
@@ -96,6 +100,9 @@ baz > 13.37")
                  (with-output-to-file "foo-1.0.0/requirements.txt"
                    (lambda ()
                      (display test-requirements)))
+                 (with-output-to-file "foo-1.0.0/test-requirements.txt"
+                   (lambda ()
+                     (display test-test-requirements)))
                  (system* "tar" "czvf" file-name "foo-1.0.0/")
                  (delete-file-recursively "foo-1.0.0")
                  (set! test-source-hash
@@ -118,6 +125,10 @@ baz > 13.37")
           ('quasiquote
            (("python-bar" ('unquote 'python-bar))
             ("python-baz" ('unquote 'python-baz)))))
+         ('native-inputs
+          ('quasiquote
+           (("python-test-bar" ('unquote 'python-test-bar))
+            ("python-test-baz" ('unquote 'python-test-baz)))))
          ('home-page "http://example.com")
          ('synopsis "summary")
          ('description "summary")
-- 
2.6.2

^ permalink raw reply related	[flat|nested] 19+ messages in thread

* [PATCH 5/5] import: pypi: Add more tests.
  2016-07-16 15:23 [PATCH 0/5] Improve the pypi updater Cyril Roelandt
                   ` (3 preceding siblings ...)
  2016-07-16 15:23 ` [PATCH 4/5] import: pypi: Compute test requirements when reading requirements files Cyril Roelandt
@ 2016-07-16 15:23 ` Cyril Roelandt
  2016-07-22 21:31   ` Ludovic Courtès
  2016-07-17 16:30 ` [PATCH 0/5] Improve the pypi updater Hartmut Goebel
  2016-07-22 21:22 ` Ludovic Courtès
  6 siblings, 1 reply; 19+ messages in thread
From: Cyril Roelandt @ 2016-07-16 15:23 UTC (permalink / raw)
  To: guix-devel

* tests/pypi.scm ("pypi->guix-package, no requirements", "pypi->guix-package,
no test requirements", "pypi->guix-package, no runtime requirements"): New
tests.
---
 tests/pypi.scm | 137 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 137 insertions(+)

diff --git a/tests/pypi.scm b/tests/pypi.scm
index 7a38d6e..dd516c5 100644
--- a/tests/pypi.scm
+++ b/tests/pypi.scm
@@ -85,6 +85,143 @@ test-baz > 42.42")
 
 (test-begin "pypi")
 
+(test-assert "pypi->guix-package, no requirements"
+  ;; Replace network resources with sample data.
+  (mock ((guix import utils) url-fetch
+         (lambda (url file-name)
+           (match url
+             ("https://pypi.python.org/pypi/foo/json"
+              (with-output-to-file file-name
+                (lambda ()
+                  (display test-json))))
+             ("https://example.com/foo-1.0.0.tar.gz"
+               (begin
+                 (mkdir "foo-1.0.0")
+                 ;; Do not create any requirements files.
+                 (system* "tar" "czvf" file-name "foo-1.0.0/")
+                 (delete-file-recursively "foo-1.0.0")
+                 (set! test-source-hash
+                       (call-with-input-file file-name port-sha256))))
+             ("https://example.com/foo-1.0.0-py2.py3-none-any.whl" #f)
+             (_ (error "Unexpected URL: " url)))))
+    (match (pypi->guix-package "foo")
+      (('package
+         ('name "python-foo")
+         ('version "1.0.0")
+         ('source ('origin
+                    ('method 'url-fetch)
+                    ('uri (string-append "https://example.com/foo-"
+                                         version ".tar.gz"))
+                    ('sha256
+                     ('base32
+                      (? string? hash)))))
+         ('build-system 'python-build-system)
+         ('home-page "http://example.com")
+         ('synopsis "summary")
+         ('description "summary")
+         ('license 'lgpl2.0))
+       (string=? (bytevector->nix-base32-string
+                  test-source-hash)
+                 hash))
+      (x
+       (pk 'fail x #f)))))
+
+(test-assert "pypi->guix-package, no test requirements"
+  ;; Replace network resources with sample data.
+  (mock ((guix import utils) url-fetch
+         (lambda (url file-name)
+           (match url
+             ("https://pypi.python.org/pypi/foo/json"
+              (with-output-to-file file-name
+                (lambda ()
+                  (display test-json))))
+             ("https://example.com/foo-1.0.0.tar.gz"
+               (begin
+                 (mkdir "foo-1.0.0")
+                  ;; Create only the runtime requirements.
+                 (with-output-to-file "foo-1.0.0/requirements.txt"
+                   (lambda ()
+                     (display test-requirements)))
+                 (system* "tar" "czvf" file-name "foo-1.0.0/")
+                 (delete-file-recursively "foo-1.0.0")
+                 (set! test-source-hash
+                       (call-with-input-file file-name port-sha256))))
+             ("https://example.com/foo-1.0.0-py2.py3-none-any.whl" #f)
+             (_ (error "Unexpected URL: " url)))))
+    (match (pypi->guix-package "foo")
+      (('package
+         ('name "python-foo")
+         ('version "1.0.0")
+         ('source ('origin
+                    ('method 'url-fetch)
+                    ('uri (string-append "https://example.com/foo-"
+                                         version ".tar.gz"))
+                    ('sha256
+                     ('base32
+                      (? string? hash)))))
+         ('build-system 'python-build-system)
+         ('propagated-inputs
+          ('quasiquote
+           (("python-bar" ('unquote 'python-bar))
+            ("python-baz" ('unquote 'python-baz)))))
+         ('home-page "http://example.com")
+         ('synopsis "summary")
+         ('description "summary")
+         ('license 'lgpl2.0))
+       (string=? (bytevector->nix-base32-string
+                  test-source-hash)
+                 hash))
+      (x
+       (pk 'fail x #f)))))
+
+(test-assert "pypi->guix-package, no runtime requirements"
+  ;; Replace network resources with sample data.
+  (mock ((guix import utils) url-fetch
+         (lambda (url file-name)
+           (match url
+             ("https://pypi.python.org/pypi/foo/json"
+              (with-output-to-file file-name
+                (lambda ()
+                  (display test-json))))
+             ("https://example.com/foo-1.0.0.tar.gz"
+               (begin
+                 (mkdir "foo-1.0.0")
+                 ;; Create only the test requirements.
+                 (with-output-to-file "foo-1.0.0/test-requirements.txt"
+                   (lambda ()
+                     (display test-test-requirements)))
+                 (system* "tar" "czvf" file-name "foo-1.0.0/")
+                 (delete-file-recursively "foo-1.0.0")
+                 (set! test-source-hash
+                       (call-with-input-file file-name port-sha256))))
+             ("https://example.com/foo-1.0.0-py2.py3-none-any.whl" #f)
+             (_ (error "Unexpected URL: " url)))))
+    (match (pypi->guix-package "foo")
+      (('package
+         ('name "python-foo")
+         ('version "1.0.0")
+         ('source ('origin
+                    ('method 'url-fetch)
+                    ('uri (string-append "https://example.com/foo-"
+                                         version ".tar.gz"))
+                    ('sha256
+                     ('base32
+                      (? string? hash)))))
+         ('build-system 'python-build-system)
+         ('native-inputs
+          ('quasiquote
+           (("python-test-bar" ('unquote 'python-test-bar))
+            ("python-test-baz" ('unquote 'python-test-baz)))))
+         ('home-page "http://example.com")
+         ('synopsis "summary")
+         ('description "summary")
+         ('license 'lgpl2.0))
+       (string=? (bytevector->nix-base32-string
+                  test-source-hash)
+                 hash))
+      (x
+       (pk 'fail x #f)))))
+
 (test-assert "pypi->guix-package"
   ;; Replace network resources with sample data.
   (mock ((guix import utils) url-fetch
-- 
2.6.2

^ permalink raw reply related	[flat|nested] 19+ messages in thread

* Re: [PATCH 0/5] Improve the pypi updater.
  2016-07-16 15:23 [PATCH 0/5] Improve the pypi updater Cyril Roelandt
                   ` (4 preceding siblings ...)
  2016-07-16 15:23 ` [PATCH 5/5] import: pypi: Add more tests Cyril Roelandt
@ 2016-07-17 16:30 ` Hartmut Goebel
  2016-07-22 21:22 ` Ludovic Courtès
  6 siblings, 0 replies; 19+ messages in thread
From: Hartmut Goebel @ 2016-07-17 16:30 UTC (permalink / raw)
  To: guix-devel

Am 16.07.2016 um 17:23 schrieb Cyril Roelandt:
> and a huge one: we should now be able to
> automatically compute the test requirements of our Python packages!
Could you please add some documentation to the manual, too? Thanks.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: [PATCH 0/5] Improve the pypi updater.
  2016-07-16 15:23 [PATCH 0/5] Improve the pypi updater Cyril Roelandt
                   ` (5 preceding siblings ...)
  2016-07-17 16:30 ` [PATCH 0/5] Improve the pypi updater Hartmut Goebel
@ 2016-07-22 21:22 ` Ludovic Courtès
  2016-07-23 14:53   ` Cyril Roelandt
  6 siblings, 1 reply; 19+ messages in thread
From: Ludovic Courtès @ 2016-07-22 21:22 UTC (permalink / raw)
  To: Cyril Roelandt; +Cc: guix-devel

Hello!

Cyril Roelandt <tipecaml@gmail.com> skribis:

> A couple of minor improvements, and a huge one: we should now be able to
> automatically compute the test requirements of our Python packages!

Neat!  So the test dependencies go to ‘inputs’ instead of
‘propagated-inputs’, right?  That can definitely save time and avoid
errors.

Is there an example package that illustrates this?

> * guix/import/pypi.scm (maybe-inputs): Pass INPUT-TYPE as a parameter.
> * guix/import/pypi.scm (make-pypi-sexp): Use this parameter.

Nitpick: no need to repeat the file name.

> * tests/pypi.scm ("pypi->guix-package", "pypi->guix-package, wheels"): Update
> accordingly.

LGTM!

Thanks,
Ludo’.

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: [PATCH 2/5] import: pypi: Remove setuptools from the inputs.
  2016-07-16 15:23 ` [PATCH 2/5] import: pypi: Remove setuptools from the inputs Cyril Roelandt
@ 2016-07-22 21:24   ` Ludovic Courtès
  2016-07-22 21:43     ` Leo Famulari
  0 siblings, 1 reply; 19+ messages in thread
From: Ludovic Courtès @ 2016-07-22 21:24 UTC (permalink / raw)
  To: Cyril Roelandt; +Cc: guix-devel

Cyril Roelandt <tipecaml@gmail.com> skribis:

> * guix/import/pypi.scm (compute-inputs): remove python-setuptools from the
> inputs.
> * tests/pypi.scm ("pypi->guix-package", "pypi->guix-package, wheels"): Update
> accordingly.

IIRC python-setuptools is needed for Python 2.x only, right?  I keep
forgetting what the deal is.

Ludo’.

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: [PATCH 3/5] import: pypi: Compute test requirements when using wheels.
  2016-07-16 15:23 ` [PATCH 3/5] import: pypi: Compute test requirements when using wheels Cyril Roelandt
@ 2016-07-22 21:28   ` Ludovic Courtès
  0 siblings, 0 replies; 19+ messages in thread
From: Ludovic Courtès @ 2016-07-22 21:28 UTC (permalink / raw)
  To: Cyril Roelandt; +Cc: guix-devel

Cyril Roelandt <tipecaml@gmail.com> skribis:

> * guix/import/pypi.scm (read-wheel-metadata): Read test requirements from the
> metadata included in the wheel.
> * guix/import/pypi.scm (guess-requirement-from-source): Return an empty list
> for test requirements.
> * guix/import/pypi.scm (compute-inputs): Return both the runtime requirements
> and the test requirements.
> * guix/import.pypi.scm (make-pypi-sexp): Compute native inputs.
> * tests/pypi.scm ("pypi->guix-package, wheels"): Update accordingly.

[...]

>                            (requirements (if run_requires
>                                              (hash-ref (list-ref run_requires 0)
>                                                         "requires")
> -                                            '())))
> -                     (map (lambda (r)
> -                            (python->package-name (clean-requirement r)))
> -                          requirements)))))
> +                                            '()))
> +                          (test-requirements (if test_requires
> +                                                 (hash-ref (list-ref test_requires 0)
> +                                                            "requires")
> +                                                 '())))

Maybe for a subsequent patch: these two occurrences could be written as

  (match test_requires
    ((head _ ...)
     (hash-ref head "requires"))
    (_
     '()))

LGTM, thanks!

Ludo’.

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: [PATCH 4/5] import: pypi: Compute test requirements when reading requirements files.
  2016-07-16 15:23 ` [PATCH 4/5] import: pypi: Compute test requirements when reading requirements files Cyril Roelandt
@ 2016-07-22 21:30   ` Ludovic Courtès
  2016-07-23 14:48     ` Cyril Roelandt
  0 siblings, 1 reply; 19+ messages in thread
From: Ludovic Courtès @ 2016-07-22 21:30 UTC (permalink / raw)
  To: Cyril Roelandt; +Cc: guix-devel

Cyril Roelandt <tipecaml@gmail.com> skribis:

> * guix/import/pypi.scm (guess-requirement-from-source): Read
> test-requirements.txt as well as requirements.txt.
> * tests/pypi.scm ("pypi->guix-package"): Update accordingly.

[...]

>            (let* ((req-file (string-append dirname "/requirements.txt"))
> -                 (exit-code (system* "tar" "xf" tarball req-file)))
> +                 (test-req-file (string-append dirname "/test-requirements.txt"))
> +                 (exit-code (system* "tar" "xf" tarball req-file))
> +                 (test-exit-code (system* "tar" "xf" tarball test-req-file)))
>              ;; TODO: support more formats.
> -            (if (zero? exit-code)
> +            (if (or (zero? exit-code)
> +                    (zero? test-exit-code))

This seems to suggest that this could be factorized somehow.  Maybe
unpack once and read the two files at once?

Ludo’.

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: [PATCH 5/5] import: pypi: Add more tests.
  2016-07-16 15:23 ` [PATCH 5/5] import: pypi: Add more tests Cyril Roelandt
@ 2016-07-22 21:31   ` Ludovic Courtès
  0 siblings, 0 replies; 19+ messages in thread
From: Ludovic Courtès @ 2016-07-22 21:31 UTC (permalink / raw)
  To: Cyril Roelandt; +Cc: guix-devel

Cyril Roelandt <tipecaml@gmail.com> skribis:

> * tests/pypi.scm ("pypi->guix-package, no requirements", "pypi->guix-package,
> no test requirements", "pypi->guix-package, no runtime requirements"): New
> tests.

Perfect.  :-)

Thank you!

Ludo'.

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: [PATCH 2/5] import: pypi: Remove setuptools from the inputs.
  2016-07-22 21:24   ` Ludovic Courtès
@ 2016-07-22 21:43     ` Leo Famulari
  2016-07-23 14:46       ` Cyril Roelandt
  0 siblings, 1 reply; 19+ messages in thread
From: Leo Famulari @ 2016-07-22 21:43 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: guix-devel

On Fri, Jul 22, 2016 at 11:24:52PM +0200, Ludovic Courtès wrote:
> Cyril Roelandt <tipecaml@gmail.com> skribis:
> 
> > * guix/import/pypi.scm (compute-inputs): remove python-setuptools from the
> > inputs.
> > * tests/pypi.scm ("pypi->guix-package", "pypi->guix-package, wheels"): Update
> > accordingly.
> 
> IIRC python-setuptools is needed for Python 2.x only, right?  I keep
> forgetting what the deal is.

In my experience, it is more often required for Python 2.x, but
sometimes it is required by Python 3.x packages, and sometimes not for
Python 2.x packages.

I'd be happy to get a more systematic explanation, or to be shown wrong.
A simple rule would be great :)

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: [PATCH 2/5] import: pypi: Remove setuptools from the inputs.
  2016-07-22 21:43     ` Leo Famulari
@ 2016-07-23 14:46       ` Cyril Roelandt
  2016-07-25 21:07         ` Ludovic Courtès
  0 siblings, 1 reply; 19+ messages in thread
From: Cyril Roelandt @ 2016-07-23 14:46 UTC (permalink / raw)
  To: Leo Famulari, Ludovic Courtès; +Cc: guix-devel

On 07/22/2016 11:43 PM, Leo Famulari wrote:
> In my experience, it is more often required for Python 2.x, but
> sometimes it is required by Python 3.x packages, and sometimes not for
> Python 2.x packages.
> 
> I'd be happy to get a more systematic explanation, or to be shown wrong.
> A simple rule would be great :)
> 

Yes, we usually do not need to include python-setuptools in the native
inputs of our Python 3 packages, we just add it to the Python 2 version.

Cyril.

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: [PATCH 4/5] import: pypi: Compute test requirements when reading requirements files.
  2016-07-22 21:30   ` Ludovic Courtès
@ 2016-07-23 14:48     ` Cyril Roelandt
  2016-07-25 21:12       ` Ludovic Courtès
  0 siblings, 1 reply; 19+ messages in thread
From: Cyril Roelandt @ 2016-07-23 14:48 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: guix-devel

On 07/22/2016 11:30 PM, Ludovic Courtès wrote:
> This seems to suggest that this could be factorized somehow.  Maybe
> unpack once and read the two files at once?

The problem is that both files might not be there, and unzip will return
a non-zero exit code if any of them is missing, so it seems easier to
just run unzip twice. WDYT?

Cyril.

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: [PATCH 0/5] Improve the pypi updater.
  2016-07-22 21:22 ` Ludovic Courtès
@ 2016-07-23 14:53   ` Cyril Roelandt
  2016-07-25 21:14     ` Ludovic Courtès
  0 siblings, 1 reply; 19+ messages in thread
From: Cyril Roelandt @ 2016-07-23 14:53 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: guix-devel

On 07/22/2016 11:22 PM, Ludovic Courtès wrote:
> Neat!  So the test dependencies go to ‘inputs’ instead of
> ‘propagated-inputs’, right?  That can definitely save time and avoid
> errors.
> 

Indeed, the runtime dependencies are propagated inputs, and the test
dependencies are native inputs. This is what we currently do when
writing packages "by hand".

> Is there an example package that illustrates this?

Anything from OpenStack :-D

Cyril.

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: [PATCH 2/5] import: pypi: Remove setuptools from the inputs.
  2016-07-23 14:46       ` Cyril Roelandt
@ 2016-07-25 21:07         ` Ludovic Courtès
  0 siblings, 0 replies; 19+ messages in thread
From: Ludovic Courtès @ 2016-07-25 21:07 UTC (permalink / raw)
  To: Cyril Roelandt; +Cc: guix-devel

Cyril Roelandt <tipecaml@gmail.com> skribis:

> On 07/22/2016 11:43 PM, Leo Famulari wrote:
>> In my experience, it is more often required for Python 2.x, but
>> sometimes it is required by Python 3.x packages, and sometimes not for
>> Python 2.x packages.
>> 
>> I'd be happy to get a more systematic explanation, or to be shown wrong.
>> A simple rule would be great :)
>> 
>
> Yes, we usually do not need to include python-setuptools in the native
> inputs of our Python 3 packages, we just add it to the Python 2 version.

OK, thanks for explaining.  This patch looks good!

Ludo’.

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: [PATCH 4/5] import: pypi: Compute test requirements when reading requirements files.
  2016-07-23 14:48     ` Cyril Roelandt
@ 2016-07-25 21:12       ` Ludovic Courtès
  0 siblings, 0 replies; 19+ messages in thread
From: Ludovic Courtès @ 2016-07-25 21:12 UTC (permalink / raw)
  To: Cyril Roelandt; +Cc: guix-devel

Cyril Roelandt <tipecaml@gmail.com> skribis:

> On 07/22/2016 11:30 PM, Ludovic Courtès wrote:
>> This seems to suggest that this could be factorized somehow.  Maybe
>> unpack once and read the two files at once?
>
> The problem is that both files might not be there, and unzip will return
> a non-zero exit code if any of them is missing, so it seems easier to
> just run unzip twice. WDYT?

OK, I see.  Then what about adding a procedure like:

  (define (file-from-zip-archive archive file)
    "Return the contents of FILE from ARCHIVE as a string, or #f if FILE
  could not be found in ARCHIVE or extraction failed."
    ;; … invoke unzip in temporary dir, check return code, then:
    (call-with-input-file (string-append tempdir "/" file)
      get-string-all))

That would move the bits about exit codes and all that out of the main
logic.

WDYT?

Thanks,
Ludo’.

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: [PATCH 0/5] Improve the pypi updater.
  2016-07-23 14:53   ` Cyril Roelandt
@ 2016-07-25 21:14     ` Ludovic Courtès
  0 siblings, 0 replies; 19+ messages in thread
From: Ludovic Courtès @ 2016-07-25 21:14 UTC (permalink / raw)
  To: Cyril Roelandt; +Cc: guix-devel

Cyril Roelandt <tipecaml@gmail.com> skribis:

> On 07/22/2016 11:22 PM, Ludovic Courtès wrote:
>> Neat!  So the test dependencies go to ‘inputs’ instead of
>> ‘propagated-inputs’, right?  That can definitely save time and avoid
>> errors.
>> 
>
> Indeed, the runtime dependencies are propagated inputs, and the test
> dependencies are native inputs. This is what we currently do when
> writing packages "by hand".

OK.

>> Is there an example package that illustrates this?
>
> Anything from OpenStack :-D

Like ‘guix import pypi hacking’ or other packages found in
openstack.scm, right?

Thanks,
Ludo’.

^ permalink raw reply	[flat|nested] 19+ messages in thread

end of thread, other threads:[~2016-07-25 21:14 UTC | newest]

Thread overview: 19+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2016-07-16 15:23 [PATCH 0/5] Improve the pypi updater Cyril Roelandt
2016-07-16 15:23 ` [PATCH 1/5] import: pypi: Move runtime dependencies to the propagated-inputs Cyril Roelandt
2016-07-16 15:23 ` [PATCH 2/5] import: pypi: Remove setuptools from the inputs Cyril Roelandt
2016-07-22 21:24   ` Ludovic Courtès
2016-07-22 21:43     ` Leo Famulari
2016-07-23 14:46       ` Cyril Roelandt
2016-07-25 21:07         ` Ludovic Courtès
2016-07-16 15:23 ` [PATCH 3/5] import: pypi: Compute test requirements when using wheels Cyril Roelandt
2016-07-22 21:28   ` Ludovic Courtès
2016-07-16 15:23 ` [PATCH 4/5] import: pypi: Compute test requirements when reading requirements files Cyril Roelandt
2016-07-22 21:30   ` Ludovic Courtès
2016-07-23 14:48     ` Cyril Roelandt
2016-07-25 21:12       ` Ludovic Courtès
2016-07-16 15:23 ` [PATCH 5/5] import: pypi: Add more tests Cyril Roelandt
2016-07-22 21:31   ` Ludovic Courtès
2016-07-17 16:30 ` [PATCH 0/5] Improve the pypi updater Hartmut Goebel
2016-07-22 21:22 ` Ludovic Courtès
2016-07-23 14:53   ` Cyril Roelandt
2016-07-25 21:14     ` Ludovic Courtès

Code repositories for project(s) associated with this external index

	https://git.savannah.gnu.org/cgit/guix.git

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.