From mboxrd@z Thu Jan 1 00:00:00 1970 From: Maxim Cournoyer Subject: bug#24450: [PATCH] bug#24450: pypi importer outputs strange character series in optional dependency case. Date: Fri, 29 Mar 2019 00:34:43 -0400 Message-ID: <87tvfm1eos.fsf@gmail.com> References: <87h99fipj1.fsf@we.make.ritual.n0.is> Mime-Version: 1.0 Content-Type: multipart/signed; boundary="==-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" Return-path: Received: from eggs.gnu.org ([209.51.188.92]:59280) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1h9jFX-0001eH-0e for bug-guix@gnu.org; Fri, 29 Mar 2019 00:36:21 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1h9jEN-0002oI-PZ for bug-guix@gnu.org; Fri, 29 Mar 2019 00:35:09 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:49655) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1h9jEM-0002nk-JV for bug-guix@gnu.org; Fri, 29 Mar 2019 00:35:03 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1h9jEM-0006K9-DX for bug-guix@gnu.org; Fri, 29 Mar 2019 00:35:02 -0400 Sender: "Debbugs-submit" Resent-Message-ID: In-Reply-To: <87h99fipj1.fsf@we.make.ritual.n0.is> (ng0's message of "Fri, 16 Sep 2016 20:00:02 +0000") List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+gcggb-bug-guix=m.gmane.org@gnu.org Sender: "bug-Guix" To: ng0 Cc: 24450@debbugs.gnu.org --==-=-= Content-Type: multipart/mixed; boundary="=-=-=" --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hello, ng0 writes: > I think this should not happen with pypi import: > > (inputs > `(("python-certifi=3D=3D2016.2.28" > ,python-certifi=3D=3D2016.2.28) > ("python-dateutil=3D=3D2.5.3" > ,python-dateutil=3D=3D2.5.3) > ("python-flask-babel=3D=3D0.11.1" > ,python-flask-babel=3D=3D0.11.1) > ("python-flask=3D=3D0.11.1" ,python-flask=3D=3D0.11.1) > ("python-lxml=3D=3D3.6.0" ,python-lxml=3D=3D3.6.0) > ("python-ndg-httpsclient=3D=3D0.4.1" > ,python-ndg-httpsclient=3D=3D0.4.1) > ("python-pyasn1-modules=3D=3D0.0.8" > ,python-pyasn1-modules=3D=3D0.0.8) > ("python-pyasn1=3D=3D0.1.9" ,python-pyasn1=3D=3D0.1.9) > ("python-pygments=3D=3D2.1.3" > ,python-pygments=3D=3D2.1.3) > ("python-pyopenssl=3D=3D0.15.1" > ,python-pyopenssl=3D=3D0.15.1) > ("python-pyyaml=3D=3D3.11" ,python-pyyaml=3D=3D3.11) > ("python-requests[socks]=3D=3D2.10.0" > ,#{python-requests\x5b;socks\x5d;=3D=3D2.10.0}#) > ("python-setuptools" ,python-setuptools))) > > > I can understand the version numbers, I can also understand the optional > socks building/module of the python-requests, but why does it read like > Gobbledygook? Can't we improve the output here? > > For version numbers, this is not a format which happened recently which > is exclusive for python build system right? This is just bad formated > because of the pypi query. > I will first try and not pin the application to these version numbers, > maybe itjustworks=E2=84=A2. > > > To reproduce: "guix import pypi searx" The following patches fix this, and more! --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0001-import-pypi-Do-not-consider-requirements.txt-files.patch Content-Transfer-Encoding: quoted-printable From=2054e44b7397f17910d95dbdb233d23e5c97c095aa Mon Sep 17 00:00:00 2001 From: Maxim Cournoyer Date: Thu, 28 Mar 2019 00:26:00 -0400 Subject: [PATCH 1/7] import: pypi: Do not consider requirements.txt files. * guix/import/pypi.scm (guess-requirements): Update comment. [guess-requirements-from-source]: Do not attempt to parse the file requirements.txt. Streamline logic. =2D-- guix/import/pypi.scm | 35 +++++++++++++---------------------- tests/pypi.scm | 23 +++++++++++------------ 2 files changed, 24 insertions(+), 34 deletions(-) diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm index 3a20fc4b9b..8269aa61d7 100644 =2D-- a/guix/import/pypi.scm +++ b/guix/import/pypi.scm @@ -206,35 +206,26 @@ cannot determine package dependencies")) (call-with-temporary-directory (lambda (dir) (let* ((pypi-name (string-take dirname (string-rindex dirname= #\-))) =2D (req-files (list (string-append dirname "/requiremen= ts.txt") =2D (string-append dirname "/" pypi-nam= e ".egg-info" =2D "/requires.txt"))) =2D (exit-codes (map (lambda (file-name) =2D (parameterize ((current-error-por= t (%make-void-port "rw+")) =2D (current-output-po= rt (%make-void-port "rw+"))) =2D (system* "tar" "xf" tarball "-C= " dir file-name))) =2D req-files))) =2D ;; Only one of these files needs to exist. =2D (if (any zero? exit-codes) =2D (match (find-files dir) =2D ((file . _) =2D (read-requirements file)) =2D (() =2D (warning (G_ "No requirements file found.\n")))) + (requires.txt (string-append dirname "/" pypi-name + ".egg-info" "/requires.tx= t")) + (exit-code (parameterize ((current-error-port (%make-v= oid-port "rw+")) + (current-output-port (%make-= void-port "rw+"))) + (system* "tar" "xf" tarball "-C" dir requ= ires.txt)))) + (if (zero? exit-code) + (read-requirements (string-append dir "/" requires.txt)) (begin =2D (warning (G_ "Failed to extract requirements files\= n")) + (warning + (G_ "Failed to extract file: ~a from source.~%") + requires.txt) '()))))) '()))) =20 =2D ;; First, try to compute the requirements using the wheel, since that = is the =2D ;; most reliable option. If a wheel is not provided for this package, = try =2D ;; getting them by reading either the "requirements.txt" file or the =2D ;; "requires.txt" from the egg-info directory from the source tarball.= Note =2D ;; that "requirements.txt" is not mandatory, so this is likely to fail. + ;; First, try to compute the requirements using the wheel, else, fallbac= k to + ;; reading the "requires.txt" from the egg-info directory from the source + ;; tarball. (or (guess-requirements-from-wheel) (guess-requirements-from-source))) =20 =2D (define (compute-inputs source-url wheel-url tarball) "Given the SOURCE-URL of an already downloaded TARBALL, return a list of name/variable pairs describing the required inputs of this package. Also diff --git a/tests/pypi.scm b/tests/pypi.scm index 6daa44a6e7..335be42644 100644 =2D-- a/tests/pypi.scm +++ b/tests/pypi.scm @@ -23,7 +23,7 @@ #:use-module (gcrypt hash) #:use-module (guix tests) #:use-module (guix build-system python) =2D #:use-module ((guix build utils) #:select (delete-file-recursively whi= ch)) + #:use-module ((guix build utils) #:select (delete-file-recursively which= mkdir-p)) #:use-module (srfi srfi-64) #:use-module (ice-9 match)) =20 @@ -55,11 +55,10 @@ (define test-source-hash "") =20 =2D(define test-requirements =2D"# A comment =2D # A comment after a space +(define test-requires.txt "\ bar =2Dbaz > 13.37") +baz > 13.37 +") =20 (define test-metadata "{ @@ -107,10 +106,10 @@ baz > 13.37") (match url ("https://example.com/foo-1.0.0.tar.gz" (begin =2D (mkdir "foo-1.0.0") =2D (with-output-to-file "foo-1.0.0/requirements.txt" + (mkdir-p "foo-1.0.0/foo.egg-info/") + (with-output-to-file "foo-1.0.0/foo.egg-info/requires.tx= t" (lambda () =2D (display test-requirements))) + (display test-requires.txt))) (system* "tar" "czvf" file-name "foo-1.0.0/") (delete-file-recursively "foo-1.0.0") (set! test-source-hash @@ -157,11 +156,11 @@ baz > 13.37") (lambda (url file-name) (match url ("https://example.com/foo-1.0.0.tar.gz" =2D (begin =2D (mkdir "foo-1.0.0") =2D (with-output-to-file "foo-1.0.0/requirements.txt" + (begin + (mkdir-p "foo-1.0.0/foo.egg-info/") + (with-output-to-file "foo-1.0.0/foo.egg-info/requires.txt" (lambda () =2D (display test-requirements))) + (display test-requires.txt))) (system* "tar" "czvf" file-name "foo-1.0.0/") (delete-file-recursively "foo-1.0.0") (set! test-source-hash =2D-=20 2.20.1 --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0002-import-pypi-Do-not-parse-optional-requirements-from-.patch Content-Transfer-Encoding: quoted-printable From=205f79b0502f62bd1dacc8ea143c1dbd9ef7cfc29d Mon Sep 17 00:00:00 2001 From: Maxim Cournoyer Date: Thu, 28 Mar 2019 00:26:00 -0400 Subject: [PATCH 2/7] import: pypi: Do not parse optional requirements from source. * guix/import/pypi.scm: Export PARSE-REQUIRES.TXT. (guess-requirements): Move the READ-REQUIREMENTS procedure to the top level, and rename it to PARSE-REQUIRES.TXT. Move the CLEAN-REQUIREMENT and COMMEN= T? functions inside the READ-REQUIREMENTS procedure. (parse-requires.txt): Add a SECTION-HEADER? predicate, and use it to prevent parsing optional requirements. * tests/pypi.scm (test-requires-with-sections): New variable. ("parse-requires.txt, with sections"): New test. ("pypi->guix-package"): Mute tar output to stdout. =2D-- guix/import/pypi.scm | 76 +++++++++++++++++++++++++++----------------- tests/pypi.scm | 21 ++++++++++-- 2 files changed, 65 insertions(+), 32 deletions(-) diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm index 8269aa61d7..91e987e9f1 100644 =2D-- a/guix/import/pypi.scm +++ b/guix/import/pypi.scm @@ -47,7 +47,8 @@ #:use-module (guix upstream) #:use-module ((guix licenses) #:prefix license:) #:use-module (guix build-system python) =2D #:export (guix-package->pypi-name + #:export (parse-requires.txt + guix-package->pypi-name pypi-recursive-import pypi->guix-package %pypi-updater)) @@ -117,6 +118,49 @@ package definition." ((package-inputs ...) `((propagated-inputs (,'quasiquote ,package-inputs)))))) =20 +(define (clean-requirement s) + ;; Given a requirement LINE, as can be found in a setuptools requires.txt + ;; file, remove everything other than the actual name of the required + ;; package, and return it. + (string-take s (or (string-index s (lambda (chr) + (member chr '(#\space #\> #\=3D #\<= )))) + (string-length s)))) + +(define (parse-requires.txt requires.txt) + "Given REQUIRES.TXT, a Setuptools requires.txt file, return a list of +requirement names." + ;; This is a very incomplete parser, which job is to select the non-opti= onal + ;; dependencies and strip them out of any version information. + ;; Alternatively, we could implement a PEG parser with the (ice-9 peg) + ;; library and the requirements grammar defined by PEP-0508 + ;; (https://www.python.org/dev/peps/pep-0508/). + + (define (comment? line) + ;; Return #t if the given LINE is a comment, #f otherwise. + (eq? (string-ref (string-trim line) 0) #\#)) + + (define (section-header? line) + ;; Return #t if the given LINE is a section header, #f otherwise. + (let ((trimmed-line (string-trim line))) + (and (not (string-null? trimmed-line)) + (eq? (string-ref trimmed-line 0) #\[)))) + + (call-with-input-file requires.txt + (lambda (port) + (let loop ((result '())) + (let ((line (read-line port))) + ;; Stop when a section is encountered, as sections contains opti= onal + ;; (extra) requirements. Non-optional requirements must appear + ;; before any section is defined. + (if (or (eof-object? line) (section-header? line)) + (reverse result) + (cond + ((or (string-null? line) (comment? line)) + (loop result)) + (else + (loop (cons (clean-requirement line) + result)))))))))) + (define (guess-requirements source-url wheel-url tarball) "Given SOURCE-URL, WHEEL-URL and a TARBALL of the package, return a list of the required packages specified in the requirements.txt file. TARBALL = will @@ -139,34 +183,6 @@ be extracted in a temporary directory." cannot determine package dependencies")) #f))))) =20 =2D (define (clean-requirement s) =2D ;; Given a requirement LINE, as can be found in a Python requirement= s.txt =2D ;; file, remove everything other than the actual name of the required =2D ;; package, and return it. =2D (string-take s =2D (or (string-index s (lambda (chr) (member chr '(#\space #\> #\=3D = #\<)))) =2D (string-length s)))) =2D =2D (define (comment? line) =2D ;; Return #t if the given LINE is a comment, #f otherwise. =2D (eq? (string-ref (string-trim line) 0) #\#)) =2D =2D (define (read-requirements requirements-file) =2D ;; Given REQUIREMENTS-FILE, a Python requirements.txt file, return a= list =2D ;; of name/variable pairs describing the requirements. =2D (call-with-input-file requirements-file =2D (lambda (port) =2D (let loop ((result '())) =2D (let ((line (read-line port))) =2D (if (eof-object? line) =2D result =2D (cond =2D ((or (string-null? line) (comment? line)) =2D (loop result)) =2D (else =2D (loop (cons (clean-requirement line) =2D result)))))))))) =2D (define (read-wheel-metadata wheel-archive) ;; Given WHEEL-ARCHIVE, a ZIP Python wheel archive, return the package= 's ;; requirements. @@ -212,7 +228,7 @@ cannot determine package dependencies")) (current-output-port (%make-= void-port "rw+"))) (system* "tar" "xf" tarball "-C" dir requ= ires.txt)))) (if (zero? exit-code) =2D (read-requirements (string-append dir "/" requires.tx= t)) + (parse-requires.txt (string-append dir "/" requires.txt= )) (begin (warning (G_ "Failed to extract file: ~a from source.~%") diff --git a/tests/pypi.scm b/tests/pypi.scm index 335be42644..e4b7142311 100644 =2D-- a/tests/pypi.scm +++ b/tests/pypi.scm @@ -60,6 +60,15 @@ bar baz > 13.37 ") =20 +(define test-requires-with-sections "\ +# A comment +foo ~=3D 3 +bar !=3D 2 + +[test] +pytest (>=3D2.5.0) +") + (define test-metadata "{ \"run_requires\": [ @@ -99,6 +108,12 @@ baz > 13.37 (uri (list "https://bitheap.org/cram/cram-0.7.tar.gz" (pypi-uri "cram" "0.7")))))))) =20 +(test-equal "parse-requires.txt, with sections" + '("foo" "bar") + (mock ((ice-9 ports) call-with-input-file + call-with-input-string) + (parse-requires.txt test-requires-with-sections))) + (test-assert "pypi->guix-package" ;; Replace network resources with sample data. (mock ((guix import utils) url-fetch @@ -110,7 +125,8 @@ baz > 13.37 (with-output-to-file "foo-1.0.0/foo.egg-info/requires.tx= t" (lambda () (display test-requires.txt))) =2D (system* "tar" "czvf" file-name "foo-1.0.0/") + (parameterize ((current-output-port (%make-void-port "rw= +"))) + (system* "tar" "czvf" file-name "foo-1.0.0/")) (delete-file-recursively "foo-1.0.0") (set! test-source-hash (call-with-input-file file-name port-sha256)))) @@ -161,7 +177,8 @@ baz > 13.37 (with-output-to-file "foo-1.0.0/foo.egg-info/requires.txt" (lambda () (display test-requires.txt))) =2D (system* "tar" "czvf" file-name "foo-1.0.0/") + (parameterize ((current-output-port (%make-void-port "rw+"= ))) + (system* "tar" "czvf" file-name "foo-1.0.0/")) (delete-file-recursively "foo-1.0.0") (set! test-source-hash (call-with-input-file file-name port-sha256)))) =2D-=20 2.20.1 --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0003-import-pypi-Improve-parsing-of-requirement-specifica.patch Content-Transfer-Encoding: quoted-printable From=200c62b541a3e8925b5ca31fe55dbe7536cf95151f Mon Sep 17 00:00:00 2001 From: Maxim Cournoyer Date: Thu, 28 Mar 2019 00:26:01 -0400 Subject: [PATCH 3/7] import: pypi: Improve parsing of requirement specifications. The previous solution was fragile and could leave unwanted characters in a requirement name, such as '[' or ']'. Partially fixes issue #33047 (see: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D33047). * guix/import/pypi.scm (use-modules): Export SPECIFICATION->REQUIREMENT-NAME (%requirement-name-regexp): New variable. (clean-requirement): Rename to... (specification->requirement-name): this, which now uses %requirement-name-regexp to select the requirement name from the requirement specification. (parse-requires.txt): Adapt. =2D-- guix/import/pypi.scm | 43 ++++++++++++++++++++++++++++++++++--------- tests/pypi.scm | 12 ++++++++++++ 2 files changed, 46 insertions(+), 9 deletions(-) diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm index 91e987e9f1..efb5939c78 100644 =2D-- a/guix/import/pypi.scm +++ b/guix/import/pypi.scm @@ -48,6 +48,7 @@ #:use-module ((guix licenses) #:prefix license:) #:use-module (guix build-system python) #:export (parse-requires.txt + specification->requirement-name guix-package->pypi-name pypi-recursive-import pypi->guix-package @@ -118,13 +119,37 @@ package definition." ((package-inputs ...) `((propagated-inputs (,'quasiquote ,package-inputs)))))) =20 =2D(define (clean-requirement s) =2D ;; Given a requirement LINE, as can be found in a setuptools requires.= txt =2D ;; file, remove everything other than the actual name of the required =2D ;; package, and return it. =2D (string-take s (or (string-index s (lambda (chr) =2D (member chr '(#\space #\> #\=3D #= \<)))) =2D (string-length s)))) +(define %requirement-name-regexp + ;; Regexp to match the requirement name in a requirement specification. + + ;; Some grammar, taken from PEP-0508 (see: + ;; https://www.python.org/dev/peps/pep-0508/). + + ;; The unified rule can be expressed as: + ;; specification =3D wsp* ( url_req | name_req ) wsp* + + ;; where url_req is: + ;; url_req =3D name wsp* extras? wsp* urlspec wsp+ quoted_marker? + + ;; and where name_req is: + ;; name_req =3D name wsp* extras? wsp* versionspec? wsp* quoted_marker? + + ;; Thus, we need only matching NAME, which is expressed as: + ;; identifer_end =3D letterOrDigit | (('-' | '_' | '.' )* letterOrDigit) + ;; identifier =3D letterOrDigit identifier_end* + ;; name =3D identifier + (let* ((letter-or-digit "[A-Za-z0-9]") + (identifier-end (string-append "(" letter-or-digit "|" + "[-_.]*" letter-or-digit ")")) + (identifier (string-append "^" letter-or-digit identifier-end "*"= )) + (name identifier)) + (make-regexp name))) + +(define (specification->requirement-name spec) + "Given a specification SPEC, return the requirement name." + (match:substring + (or (regexp-exec %requirement-name-regexp spec) + (error (G_ "Could not extract requirement name in spec:") spec)))) =20 (define (parse-requires.txt requires.txt) "Given REQUIRES.TXT, a Setuptools requires.txt file, return a list of @@ -158,7 +183,7 @@ requirement names." ((or (string-null? line) (comment? line)) (loop result)) (else =2D (loop (cons (clean-requirement line) + (loop (cons (specification->requirement-name line) result)))))))))) =20 (define (guess-requirements source-url wheel-url tarball) @@ -200,7 +225,7 @@ cannot determine package dependencies")) (hash-ref (list-ref run_requir= es 0) "requires") '()))) =2D (map clean-requirement requirements))))) + (map specification->requirement-name requirements))))) (lambda () (delete-file json-file) (rmdir dirname)))))) diff --git a/tests/pypi.scm b/tests/pypi.scm index e4b7142311..82d6bba8dd 100644 =2D-- a/tests/pypi.scm +++ b/tests/pypi.scm @@ -55,6 +55,14 @@ (define test-source-hash "") =20 +(define test-specifications + '("Fizzy [foo, bar]" + "PickyThing<1.6,>1.9,!=3D1.9.6,<2.0a0,=3D=3D2.4c1" + "SomethingWithMarker[foo]>1.0;python_version<\"2.7\"" + "requests [security,tests] >=3D 2.8.1, =3D=3D 2.8.* ; python_version <= \"2.7\"" + "pip @ https://github.com/pypa/pip/archive/1.3.1.zip#\ +sha1=3Dda9234ee9982d4bbb3c72346a6de940a148ea686")) + (define test-requires.txt "\ bar baz > 13.37 @@ -108,6 +116,10 @@ pytest (>=3D2.5.0) (uri (list "https://bitheap.org/cram/cram-0.7.tar.gz" (pypi-uri "cram" "0.7")))))))) =20 +(test-equal "specification->requirement-name" + '("Fizzy" "PickyThing" "SomethingWithMarker" "requests" "pip") + (map specification->requirement-name test-specifications)) + (test-equal "parse-requires.txt, with sections" '("foo" "bar") (mock ((ice-9 ports) call-with-input-file =2D-=20 2.20.1 --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0004-import-pypi-Deduplicate-requirements.patch Content-Transfer-Encoding: quoted-printable From=2076e4a3150f8126e0b952c6129b6e1371afba80c0 Mon Sep 17 00:00:00 2001 From: Maxim Cournoyer Date: Thu, 28 Mar 2019 00:26:01 -0400 Subject: [PATCH 4/7] import: pypi: Deduplicate requirements. * guix/import/pypi.scm (parse-requires.txt): Remove potential duplicates. =2D-- guix/import/pypi.scm | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm index efb5939c78..a90be67bb0 100644 =2D-- a/guix/import/pypi.scm +++ b/guix/import/pypi.scm @@ -178,7 +178,11 @@ requirement names." ;; (extra) requirements. Non-optional requirements must appear ;; before any section is defined. (if (or (eof-object? line) (section-header? line)) =2D (reverse result) + ;; Duplicates can occur, since the same requirement can be + ;; listed multiple times with different conditional markers,= e.g. + ;; pytest >=3D 3 ; python_version >=3D "3.3" + ;; pytest < 3 ; python_version < "3.3" + (reverse (delete-duplicates result)) (cond ((or (string-null? line) (comment? line)) (loop result)) =2D-=20 2.20.1 --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0005-import-pypi-Support-more-types-of-archives.patch Content-Transfer-Encoding: quoted-printable From=2073e27235cac1275ba7671fd2364325cf5788cb3c Mon Sep 17 00:00:00 2001 From: Maxim Cournoyer Date: Thu, 28 Mar 2019 00:26:02 -0400 Subject: [PATCH 5/7] import: pypi: Support more types of archives. This change enables the PyPI importer to look for requirements in a source archive of a different type than "tar.gz" or "tar.bz2". * guix/import/pypi.scm: (guess-requirements)[tarball-directory]: Rename to.= .. [archive-root-directory]: this. Use COMPRESSED-FILED? to determine if an archive is supported or not. [guess-requirements-from-source]: Adapt to use the new method, and use unzip to extract ZIP archives. (guess-requirements): Rename the TARBALL argument to ARCHIVE, to denote the archive format is no longer bound specifically to the Tar format. =2D-- guix/import/pypi.scm | 47 ++++++++++++++++++++++---------------------- 1 file changed, 24 insertions(+), 23 deletions(-) diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm index a90be67bb0..8e93653717 100644 =2D-- a/guix/import/pypi.scm +++ b/guix/import/pypi.scm @@ -190,27 +190,24 @@ requirement names." (loop (cons (specification->requirement-name line) result)))))))))) =20 =2D(define (guess-requirements source-url wheel-url tarball) =2D "Given SOURCE-URL, WHEEL-URL and a TARBALL of the package, return a li= st =2Dof the required packages specified in the requirements.txt file. TARBAL= L will +(define (guess-requirements source-url wheel-url archive) + "Given SOURCE-URL, WHEEL-URL and a ARCHIVE of the package, return a list +of the required packages specified in the requirements.txt file. ARCHIVE = will be extracted in a temporary directory." =20 =2D (define (tarball-directory url) =2D ;; Given the URL of the package's tarball, return the name of the di= rectory + (define (archive-root-directory url) + ;; Given the URL of the package's archive, return the name of the dire= ctory ;; that will be created upon decompressing it. If the filetype is not ;; supported, return #f. =2D ;; TODO: Support more archive formats. =2D (let ((basename (substring url (+ 1 (string-rindex url #\/))))) =2D (cond =2D ((string-suffix? ".tar.gz" basename) =2D (string-drop-right basename 7)) =2D ((string-suffix? ".tar.bz2" basename) =2D (string-drop-right basename 8)) =2D (else + (if (compressed-file? url) + (let ((root-directory (file-sans-extension (basename url)))) + (if (string=3D? "tar" (file-extension root-directory)) + (file-sans-extension root-directory) + root-directory)) (begin =2D (warning (G_ "Unsupported archive format: \ =2Dcannot determine package dependencies")) =2D #f))))) + (warning (G_ "Unsupported archive format (~a): \ +cannot determine package dependencies") (file-extension url)) + #f))) =20 (define (read-wheel-metadata wheel-archive) ;; Given WHEEL-ARCHIVE, a ZIP Python wheel archive, return the package= 's @@ -246,16 +243,20 @@ cannot determine package dependencies")) =20 (define (guess-requirements-from-source) ;; Return the package's requirements by guessing them from the source. =2D (let ((dirname (tarball-directory source-url))) + (let ((dirname (archive-root-directory source-url)) + (extension (file-extension source-url))) (if (string? dirname) (call-with-temporary-directory (lambda (dir) (let* ((pypi-name (string-take dirname (string-rindex dirname= #\-))) (requires.txt (string-append dirname "/" pypi-name ".egg-info" "/requires.tx= t")) =2D (exit-code (parameterize ((current-error-port (%make= -void-port "rw+")) =2D (current-output-port (%mak= e-void-port "rw+"))) =2D (system* "tar" "xf" tarball "-C" dir re= quires.txt)))) + (exit-code + (parameterize ((current-error-port (%make-void-port "= rw+")) + (current-output-port (%make-void-port = "rw+"))) + (if (string=3D? "zip" extension) + (system* "unzip" archive "-d" dir requires.txt) + (system* "tar" "xf" archive "-C" dir requires.t= xt))))) (if (zero? exit-code) (parse-requires.txt (string-append dir "/" requires.txt= )) (begin @@ -271,13 +272,13 @@ cannot determine package dependencies")) (or (guess-requirements-from-wheel) (guess-requirements-from-source))) =20 =2D(define (compute-inputs source-url wheel-url tarball) =2D "Given the SOURCE-URL of an already downloaded TARBALL, return a list = of +(define (compute-inputs source-url wheel-url archive) + "Given the SOURCE-URL of an already downloaded ARCHIVE, return a list of name/variable pairs describing the required inputs of this package. Also return the unaltered list of upstream dependency names." (let ((dependencies (remove (cut string=3D? "argparse" <>) =2D (guess-requirements source-url wheel-url tarball)))) + (guess-requirements source-url wheel-url archive)))) (values (sort (map (lambda (input) (let ((guix-name (python->package-name input))) =2D-=20 2.20.1 --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0006-import-pypi-Parse-wheel-METADATA-instead-of-metadata.patch Content-Transfer-Encoding: quoted-printable From=20fb0547ef225103c0f8355a7eccc41e0d028f6563 Mon Sep 17 00:00:00 2001 From: Maxim Cournoyer Date: Thu, 28 Mar 2019 00:26:03 -0400 Subject: [PATCH 6/7] import: pypi: Parse wheel METADATA instead of metadata.json. With newer Wheel releases, there is no more metadata.json file; the METADATA file should be used instead (see: https://github.com/pypa/wheel/issues/195). This change updates our PyPI importer so that it uses the later. * guix/import/pypi.scm (define-module): Remove unnecessary modules and expo= rt the PARSE-WHEEL-METADATA method. (parse-wheel-metadata): Add method. (guess-requirements): Use it. * tests/pypi.scm (test-metadata): Test it. =2D-- guix/import/pypi.scm | 66 +++++++++++++++++++++++++++++--------------- tests/pypi.scm | 60 ++++++++++++++++++++++++++++++---------- 2 files changed, 89 insertions(+), 37 deletions(-) diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm index 8e93653717..c520213b6a 100644 =2D-- a/guix/import/pypi.scm +++ b/guix/import/pypi.scm @@ -21,9 +21,7 @@ ;;; along with GNU Guix. If not, see . =20 (define-module (guix import pypi) =2D #:use-module (ice-9 binary-ports) #:use-module (ice-9 match) =2D #:use-module (ice-9 pretty-print) #:use-module (ice-9 regex) #:use-module (ice-9 receive) #:use-module ((ice-9 rdelim) #:select (read-line)) @@ -31,9 +29,6 @@ #:use-module (srfi srfi-26) #:use-module (srfi srfi-34) #:use-module (srfi srfi-35) =2D #:use-module (rnrs bytevectors) =2D #:use-module (json) =2D #:use-module (web uri) #:use-module (guix ui) #:use-module (guix utils) #:use-module ((guix build utils) @@ -48,6 +43,7 @@ #:use-module ((guix licenses) #:prefix license:) #:use-module (guix build-system python) #:export (parse-requires.txt + parse-wheel-metadata specification->requirement-name guix-package->pypi-name pypi-recursive-import @@ -190,6 +186,37 @@ requirement names." (loop (cons (specification->requirement-name line) result)))))))))) =20 +(define (parse-wheel-metadata metadata) + "Given METADATA, a Wheel metadata file, return a list of requirement nam= es." + ;; METADATA is a RFC-2822-like, header based file. + + (define (requires-dist-header? line) + ;; Return #t if the given LINE is a Requires-Dist header. + (regexp-match? (string-match "^Requires-Dist: " line))) + + (define (requires-dist-value line) + (string-drop line (string-length "Requires-Dist: "))) + + (define (extra? line) + ;; Return #t if the given LINE is an "extra" requirement. + (regexp-match? (string-match "extra =3D=3D " line))) + + (call-with-input-file metadata + (lambda (port) + (let loop ((requirements '())) + (let ((line (read-line port))) + ;; Stop at the first 'Provides-Extra' section: the non-optional + ;; requirements appear before the optional ones. + (if (eof-object? line) + (reverse (delete-duplicates requirements)) + (cond + ((and (requires-dist-header? line) (not (extra? line))) + (loop (cons (specification->requirement-name + (requires-dist-value line)) + requirements))) + (else + (loop requirements))))))))) + (define (guess-requirements source-url wheel-url archive) "Given SOURCE-URL, WHEEL-URL and a ARCHIVE of the package, return a list of the required packages specified in the requirements.txt file. ARCHIVE = will @@ -211,25 +238,18 @@ cannot determine package dependencies") (file-extensi= on url)) =20 (define (read-wheel-metadata wheel-archive) ;; Given WHEEL-ARCHIVE, a ZIP Python wheel archive, return the package= 's =2D ;; requirements. + ;; requirements, or #f if the metadata file contained therein couldn't= be + ;; extracted. (let* ((dirname (wheel-url->extracted-directory wheel-url)) =2D (json-file (string-append dirname "/metadata.json"))) =2D (and (zero? (system* "unzip" "-q" wheel-archive json-file)) =2D (dynamic-wind =2D (const #t) =2D (lambda () =2D (call-with-input-file json-file =2D (lambda (port) =2D (let* ((metadata (json->scm port)) =2D (run_requires (hash-ref metadata "run_requires= ")) =2D (requirements (if run_requires =2D (hash-ref (list-ref run_requ= ires 0) =2D "requires") =2D '()))) =2D (map specification->requirement-name requirements))= ))) =2D (lambda () =2D (delete-file json-file) =2D (rmdir dirname)))))) + (metadata (string-append dirname "/METADATA"))) + (call-with-temporary-directory + (lambda (dir) + (if (zero? (system* "unzip" "-q" wheel-archive "-d" dir metadata)) + (parse-wheel-metadata (string-append dir "/" metadata)) + (begin + (warning + (G_ "Failed to extract file: ~a from wheel.~%") metadata) + #f)))))) =20 (define (guess-requirements-from-wheel) ;; Return the package's requirements using the wheel, or #f if an error diff --git a/tests/pypi.scm b/tests/pypi.scm index 82d6bba8dd..ca8cb5f6de 100644 =2D-- a/tests/pypi.scm +++ b/tests/pypi.scm @@ -21,6 +21,7 @@ #:use-module (guix import pypi) #:use-module (guix base32) #:use-module (gcrypt hash) + #:use-module (guix memoization) #:use-module (guix tests) #:use-module (guix build-system python) #:use-module ((guix build utils) #:select (delete-file-recursively which= mkdir-p)) @@ -77,17 +78,33 @@ bar !=3D 2 pytest (>=3D2.5.0) ") =20 =2D(define test-metadata =2D "{ =2D \"run_requires\": [ =2D { =2D \"requires\": [ =2D \"bar\", =2D \"baz (>13.37)\" =2D ] =2D } =2D ] =2D}") +(define test-metadata "\ +Classifier: Programming Language :: Python :: 3.7 +Requires-Dist: baz ~=3D 3 +Requires-Dist: bar !=3D 2 +Provides-Extra: test +pytest (>=3D2.5.0) +") + +(define test-metadata-with-extras " +Classifier: Programming Language :: Python :: 3.7 +Requires-Python: >=3D2.7, !=3D3.0.*, !=3D3.1.*, !=3D3.2.*, !=3D3.3.* +Requires-Dist: wrapt (<2,>=3D1) +Requires-Dist: bar + +Provides-Extra: dev +Requires-Dist: tox ; extra =3D=3D 'dev' +Requires-Dist: bumpversion (<1) ; extra =3D=3D 'dev' +") + +;;; Provides-Extra can appear before Requires-Dist. +(define test-metadata-with-extras-jedi "\ +Requires-Python: >=3D2.7, !=3D3.0.*, !=3D3.1.*, !=3D3.2.*, !=3D3.3.* +Provides-Extra: testing +Requires-Dist: parso (>=3D0.3.0) +Provides-Extra: testing +Requires-Dist: pytest (>=3D3.1.0); extra =3D=3D 'testing' +") =20 (test-begin "pypi") =20 @@ -126,6 +143,18 @@ pytest (>=3D2.5.0) call-with-input-string) (parse-requires.txt test-requires-with-sections))) =20 +(test-equal "parse-wheel-metadata, with extras" + '("wrapt" "bar") + (mock ((ice-9 ports) call-with-input-file + call-with-input-string) + (parse-wheel-metadata test-metadata-with-extras))) + +(test-equal "parse-wheel-metadata, with extras - Jedi" + '("parso") + (mock ((ice-9 ports) call-with-input-file + call-with-input-string) + (parse-wheel-metadata test-metadata-with-extras-jedi))) + (test-assert "pypi->guix-package" ;; Replace network resources with sample data. (mock ((guix import utils) url-fetch @@ -188,7 +217,7 @@ pytest (>=3D2.5.0) (mkdir-p "foo-1.0.0/foo.egg-info/") (with-output-to-file "foo-1.0.0/foo.egg-info/requires.txt" (lambda () =2D (display test-requires.txt))) + (display "wrong data to make sure we're testing wheel= s "))) (parameterize ((current-output-port (%make-void-port "rw+"= ))) (system* "tar" "czvf" file-name "foo-1.0.0/")) (delete-file-recursively "foo-1.0.0") @@ -197,13 +226,13 @@ pytest (>=3D2.5.0) ("https://example.com/foo-1.0.0-py2.py3-none-any.whl" (begin (mkdir "foo-1.0.0.dist-info") =2D (with-output-to-file "foo-1.0.0.dist-info/metadata.json" + (with-output-to-file "foo-1.0.0.dist-info/METADATA" (lambda () (display test-metadata))) (let ((zip-file (string-append file-name ".zip"))) ;; zip always adds a "zip" extension to the file it cre= ates, ;; so we need to rename it. =2D (system* "zip" zip-file "foo-1.0.0.dist-info/metadata= .json") + (system* "zip" zip-file "foo-1.0.0.dist-info/METADATA") (rename-file zip-file file-name)) (delete-file-recursively "foo-1.0.0.dist-info"))) (_ (error "Unexpected URL: " url))))) @@ -215,6 +244,9 @@ pytest (>=3D2.5.0) (string-length test-json))) ("https://example.com/foo-1.0.0-py2.py3-none-any.whl" #= f) (_ (error "Unexpected URL: " url))))) + ;; Not clearing the memoization cache here would mean return= ing the value + ;; computed in the previous test. + (invalidate-memoization! pypi->guix-package) (match (pypi->guix-package "foo") (('package ('name "python-foo") =2D-=20 2.20.1 --=-=-= Content-Type: text/x-patch; charset=utf-8 Content-Disposition: attachment; filename=0007-import-pypi-Include-optional-test-inputs-as-native-i.patch Content-Transfer-Encoding: quoted-printable From=20ea0f24eb7b19c57ebb24ec48ba776b240bccfc99 Mon Sep 17 00:00:00 2001 From: Maxim Cournoyer Date: Thu, 28 Mar 2019 23:12:26 -0400 Subject: [PATCH 7/7] import: pypi: Include optional test inputs as native-inputs. * guix/import/pypi.scm (maybe-inputs): Add INPUT-TYPE argument, and use it. (test-section?): New predicate. (parse-requires.txt): Collect the optional test inputs, and return them as = the second element of the returned list. (parse-wheel-metadata): Likewise. (guess-requirements): Adapt, and hide unzip output. (make-pypi-sexp): Likewise, and include the test inputs requirements as nat= ive inputs in the returned package expression. * tests/pypi.scm (test-requires.txt): Include a test section in the test-requires.txt data. (test-requires.txt-beaker): New variable. ("parse-requires.txt"): Adapt. ("parse-requires.txt - Beaker"): New test. ("parse-wheel-metadata, with extras"): Adapt. ("parse-wheel-metadata, with extras - Jedi"): Adapt. ("pypi->guix-package, no wheel"): Re-indent, and add the expected native-inputs. ("pypi->guix-package, wheels"): Likewise. ("pypi->guix-package, no usable requirement file."): New test. =2D-- guix/import/pypi.scm | 158 ++++++++++++++++++++++++++++--------------- tests/pypi.scm | 123 +++++++++++++++++++++++++-------- 2 files changed, 199 insertions(+), 82 deletions(-) diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm index c520213b6a..f84ad88e44 100644 =2D-- a/guix/import/pypi.scm +++ b/guix/import/pypi.scm @@ -4,6 +4,7 @@ ;;; Copyright =C2=A9 2015, 2016, 2017 Ludovic Court=C3=A8s ;;; Copyright =C2=A9 2017 Mathieu Othacehe ;;; Copyright =C2=A9 2018 Ricardo Wurmus +;;; Copyright =C2=A9 2019 Maxim Cournoyer ;;; ;;; This file is part of GNU Guix. ;;; @@ -26,6 +27,7 @@ #:use-module (ice-9 receive) #:use-module ((ice-9 rdelim) #:select (read-line)) #:use-module (srfi srfi-1) + #:use-module (srfi srfi-11) #:use-module (srfi srfi-26) #:use-module (srfi srfi-34) #:use-module (srfi srfi-35) @@ -106,14 +108,15 @@ package on PyPI." ((name version _ ...) (string-append name "-" version ".dist-info")))) =20 =2D(define (maybe-inputs package-inputs) +(define (maybe-inputs package-inputs input-type) "Given a list of PACKAGE-INPUTS, tries to generate the 'inputs' field of= a =2Dpackage definition." +package definition. INPUT-TYPE, a symbol, is used to populate the name of +the input field." (match package-inputs (() '()) ((package-inputs ...) =2D `((propagated-inputs (,'quasiquote ,package-inputs)))))) + `((,input-type (,'quasiquote ,package-inputs)))))) =20 (define %requirement-name-regexp ;; Regexp to match the requirement name in a requirement specification. @@ -147,11 +150,21 @@ package definition." (or (regexp-exec %requirement-name-regexp spec) (error (G_ "Could not extract requirement name in spec:") spec)))) =20 +(define (test-section? name) + "Return #t if the section name contains 'test' or 'dev'." + (any (cut string-contains-ci name <>) + '("test" "dev"))) + (define (parse-requires.txt requires.txt) =2D "Given REQUIRES.TXT, a Setuptools requires.txt file, return a list of =2Drequirement names." =2D ;; This is a very incomplete parser, which job is to select the non-op= tional =2D ;; dependencies and strip them out of any version information. + "Given REQUIRES.TXT, a Setuptools requires.txt file, return a pair of re= quirements. + +The first element of the pair contains the required dependencies while the +second the optional test dependencies. Note that currently, optional, +non-test dependencies are omitted since these can be difficult or expensiv= e to +satisfy." + + ;; This is a very incomplete parser, which job is to read in the require= ment + ;; specification lines, and strip them out of any version information. ;; Alternatively, we could implement a PEG parser with the (ice-9 peg) ;; library and the requirements grammar defined by PEP-0508 ;; (https://www.python.org/dev/peps/pep-0508/). @@ -168,57 +181,89 @@ requirement names." =20 (call-with-input-file requires.txt (lambda (port) =2D (let loop ((result '())) + (let loop ((required-deps '()) + (test-deps '()) + (inside-test-section? #f) + (optional? #f)) (let ((line (read-line port))) =2D ;; Stop when a section is encountered, as sections contains op= tional =2D ;; (extra) requirements. Non-optional requirements must appear =2D ;; before any section is defined. =2D (if (or (eof-object? line) (section-header? line)) + (if (eof-object? line) ;; Duplicates can occur, since the same requirement can be ;; listed multiple times with different conditional markers,= e.g. ;; pytest >=3D 3 ; python_version >=3D "3.3" ;; pytest < 3 ; python_version < "3.3" =2D (reverse (delete-duplicates result)) + (map (compose reverse delete-duplicates) + (list required-deps test-deps)) (cond ((or (string-null? line) (comment? line)) =2D (loop result)) =2D (else + (loop required-deps test-deps inside-test-section? optiona= l?)) + ((section-header? line) + ;; Encountering a section means that all the requirements + ;; listed below are optional. Since we want to pick only t= he + ;; test dependencies from the optional dependencies, we mu= st + ;; track those separately. + (loop required-deps test-deps (test-section? line) #t)) + (inside-test-section? + (loop required-deps + (cons (specification->requirement-name line) + test-deps) + inside-test-section? optional?)) + ((not optional?) (loop (cons (specification->requirement-name line) =2D result)))))))))) + required-deps) + test-deps inside-test-section? optional?)) + (optional? + ;; Skip optional items. + (loop required-deps test-deps inside-test-section? optiona= l?)) + (else + (warning (G_ "parse-requires.txt reached an unexpected \ +condition on line ~a~%") line))))))))) =20 (define (parse-wheel-metadata metadata) =2D "Given METADATA, a Wheel metadata file, return a list of requirement n= ames." + "Given METADATA, a Wheel metadata file, return a pair of requirements. + +The first element of the pair contains the required dependencies while the= second the optional +test dependencies. Note that currently, optional, non-test dependencies a= re +omitted since these can be difficult or expensive to satisfy." ;; METADATA is a RFC-2822-like, header based file. =20 (define (requires-dist-header? line) ;; Return #t if the given LINE is a Requires-Dist header. =2D (regexp-match? (string-match "^Requires-Dist: " line))) + (string-match "^Requires-Dist: " line)) =20 (define (requires-dist-value line) (string-drop line (string-length "Requires-Dist: "))) =20 (define (extra? line) ;; Return #t if the given LINE is an "extra" requirement. =2D (regexp-match? (string-match "extra =3D=3D " line))) + (string-match "extra =3D=3D '(.*)'" line)) + + (define (test-requirement? line) + (let ((extra-label (match:substring (extra? line) 1))) + (and extra-label (test-section? extra-label)))) =20 (call-with-input-file metadata (lambda (port) =2D (let loop ((requirements '())) + (let loop ((required-deps '()) + (test-deps '())) (let ((line (read-line port))) =2D ;; Stop at the first 'Provides-Extra' section: the non-optional =2D ;; requirements appear before the optional ones. (if (eof-object? line) =2D (reverse (delete-duplicates requirements)) + (map (compose reverse delete-duplicates) + (list required-deps test-deps)) (cond ((and (requires-dist-header? line) (not (extra? line))) (loop (cons (specification->requirement-name (requires-dist-value line)) =2D requirements))) + required-deps) + test-deps)) + ((and (requires-dist-header? line) (test-requirement? line)) + (loop required-deps + (cons (specification->requirement-name (requires-dis= t-value line)) + test-deps))) (else =2D (loop requirements))))))))) + (loop required-deps test-deps))))))))) ;skip line =20 (define (guess-requirements source-url wheel-url archive) =2D "Given SOURCE-URL, WHEEL-URL and a ARCHIVE of the package, return a li= st + "Given SOURCE-URL, WHEEL-URL and an ARCHIVE of the package, return a list of the required packages specified in the requirements.txt file. ARCHIVE = will be extracted in a temporary directory." =20 @@ -244,7 +289,10 @@ cannot determine package dependencies") (file-extensio= n url)) (metadata (string-append dirname "/METADATA"))) (call-with-temporary-directory (lambda (dir) =2D (if (zero? (system* "unzip" "-q" wheel-archive "-d" dir metadat= a)) + (if (zero? + (parameterize ((current-error-port (%make-void-port "rw+")) + (current-output-port (%make-void-port "rw+"))) + (system* "unzip" wheel-archive "-d" dir metadata))) (parse-wheel-metadata (string-append dir "/" metadata)) (begin (warning @@ -283,32 +331,38 @@ cannot determine package dependencies") (file-extensi= on url)) (warning (G_ "Failed to extract file: ~a from source.~%") requires.txt) =2D '()))))) =2D '()))) + (list '() '())))))) + (list '() '())))) =20 ;; First, try to compute the requirements using the wheel, else, fallbac= k to ;; reading the "requires.txt" from the egg-info directory from the source =2D ;; tarball. + ;; archive. (or (guess-requirements-from-wheel) (guess-requirements-from-source))) =20 (define (compute-inputs source-url wheel-url archive) =2D "Given the SOURCE-URL of an already downloaded ARCHIVE, return a list = of =2Dname/variable pairs describing the required inputs of this package. Also =2Dreturn the unaltered list of upstream dependency names." =2D (let ((dependencies =2D (remove (cut string=3D? "argparse" <>) =2D (guess-requirements source-url wheel-url archive)))) =2D (values (sort =2D (map (lambda (input) =2D (let ((guix-name (python->package-name input))) =2D (list guix-name (list 'unquote (string->symbol gui= x-name))))) =2D dependencies) =2D (lambda args =2D (match args =2D (((a _ ...) (b _ ...)) =2D (string-ci) deps)) + + (define (requirement->package-name/sort deps) + (sort + (map (lambda (input) + (let ((guix-name (python->package-name input))) + (list guix-name (list 'unquote (string->symbol guix-name))))) + deps) + (lambda args + (match args + (((a _ ...) (b _ ...)) + (string-cipackage-name/sort strip-argparse)) + + (map process-requirements (guess-requirements source-url wheel-url archi= ve))) =20 (define (make-pypi-sexp name version source-url wheel-url home-page synops= is description license) @@ -317,15 +371,13 @@ VERSION, SOURCE-URL, HOME-PAGE, SYNOPSIS, DESCRIPTION= , and LICENSE." (call-with-temporary-output-file (lambda (temp port) (and (url-fetch source-url temp) =2D (receive (input-package-names upstream-dependency-names) =2D (compute-inputs source-url wheel-url temp) =2D (values + (match (compute-inputs source-url wheel-url temp) + ((required-inputs test-inputs) `(package (name ,(python->package-name name)) (version ,version) (source (origin (method url-fetch) =2D ;; Sometimes 'pypi-uri' doesn't quite work due t= o mixed ;; cases in NAME, for instance, as is the case w= ith ;; "uwsgi". In that case, fall back to a full U= RL. @@ -334,12 +386,12 @@ VERSION, SOURCE-URL, HOME-PAGE, SYNOPSIS, DESCRIPTION= , and LICENSE." (base32 ,(guix-hash-url temp))))) (build-system python-build-system) =2D ,@(maybe-inputs input-package-names) + ,@(maybe-inputs required-inputs 'propagated-inputs) + ,@(maybe-inputs test-inputs 'native-inputs) (home-page ,home-page) (synopsis ,synopsis) (description ,description) =2D (license ,(license->symbol license))) =2D upstream-dependency-names)))))) + (license ,(license->symbol license))))))))) =20 (define pypi->guix-package (memoize diff --git a/tests/pypi.scm b/tests/pypi.scm index ca8cb5f6de..aa08e2cb54 100644 =2D-- a/tests/pypi.scm +++ b/tests/pypi.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright =C2=A9 2014 David Thompson ;;; Copyright =C2=A9 2016 Ricardo Wurmus +;;; Copyright =C2=A9 2019 Maxim Cournoyer ;;; ;;; This file is part of GNU Guix. ;;; @@ -65,11 +66,6 @@ sha1=3Dda9234ee9982d4bbb3c72346a6de940a148ea686")) =20 (define test-requires.txt "\ =2Dbar =2Dbaz > 13.37 =2D") =2D =2D(define test-requires-with-sections "\ # A comment foo ~=3D 3 bar !=3D 2 @@ -78,12 +74,25 @@ bar !=3D 2 pytest (>=3D2.5.0) ") =20 +;; Beaker contains only optional dependencies. +(define test-requires.txt-beaker "\ +[crypto] +pycryptopp>=3D0.5.12 + +[cryptography] +cryptography + +[testsuite] +Mock +coverage +") + (define test-metadata "\ Classifier: Programming Language :: Python :: 3.7 Requires-Dist: baz ~=3D 3 Requires-Dist: bar !=3D 2 Provides-Extra: test =2Dpytest (>=3D2.5.0) +Requires-Dist: pytest (>=3D2.5.0) ; extra =3D=3D 'test' ") =20 (define test-metadata-with-extras " @@ -137,25 +146,31 @@ Requires-Dist: pytest (>=3D3.1.0); extra =3D=3D 'test= ing' '("Fizzy" "PickyThing" "SomethingWithMarker" "requests" "pip") (map specification->requirement-name test-specifications)) =20 =2D(test-equal "parse-requires.txt, with sections" =2D '("foo" "bar") +(test-equal "parse-requires.txt" + (list '("foo" "bar") '("pytest")) (mock ((ice-9 ports) call-with-input-file call-with-input-string) =2D (parse-requires.txt test-requires-with-sections))) + (parse-requires.txt test-requires.txt))) + +(test-equal "parse-requires.txt - Beaker" + (list '() '("Mock" "coverage")) + (mock ((ice-9 ports) call-with-input-file + call-with-input-string) + (parse-requires.txt test-requires.txt-beaker))) =20 (test-equal "parse-wheel-metadata, with extras" =2D '("wrapt" "bar") + (list '("wrapt" "bar") '("tox" "bumpversion")) (mock ((ice-9 ports) call-with-input-file call-with-input-string) (parse-wheel-metadata test-metadata-with-extras))) =20 (test-equal "parse-wheel-metadata, with extras - Jedi" =2D '("parso") + (list '("parso") '("pytest")) (mock ((ice-9 ports) call-with-input-file call-with-input-string) (parse-wheel-metadata test-metadata-with-extras-jedi))) =20 =2D(test-assert "pypi->guix-package" +(test-assert "pypi->guix-package, no wheel" ;; Replace network resources with sample data. (mock ((guix import utils) url-fetch (lambda (url file-name) @@ -195,7 +210,10 @@ Requires-Dist: pytest (>=3D3.1.0); extra =3D=3D 'testi= ng' ('propagated-inputs ('quasiquote (("python-bar" ('unquote 'python-bar)) =2D ("python-baz" ('unquote 'python-baz))))) + ("python-foo" ('unquote 'python-foo))))) + ('native-inputs + ('quasiquote + (("python-pytest" ('unquote 'python-pytest))))) ('home-page "http://example.com") ('synopsis "summary") ('description "summary") @@ -216,25 +234,25 @@ Requires-Dist: pytest (>=3D3.1.0); extra =3D=3D 'test= ing' (begin (mkdir-p "foo-1.0.0/foo.egg-info/") (with-output-to-file "foo-1.0.0/foo.egg-info/requires.txt" =2D (lambda () =2D (display "wrong data to make sure we're testing whe= els "))) + (lambda () + (display "wrong data to make sure we're testing wheels= "))) (parameterize ((current-output-port (%make-void-port "rw+"= ))) (system* "tar" "czvf" file-name "foo-1.0.0/")) =2D (delete-file-recursively "foo-1.0.0") =2D (set! test-source-hash =2D (call-with-input-file file-name port-sha256)))) + (delete-file-recursively "foo-1.0.0") + (set! test-source-hash + (call-with-input-file file-name port-sha256)))) ("https://example.com/foo-1.0.0-py2.py3-none-any.whl" =2D (begin =2D (mkdir "foo-1.0.0.dist-info") =2D (with-output-to-file "foo-1.0.0.dist-info/METADATA" =2D (lambda () =2D (display test-metadata))) =2D (let ((zip-file (string-append file-name ".zip"))) =2D ;; zip always adds a "zip" extension to the file it c= reates, =2D ;; so we need to rename it. =2D (system* "zip" zip-file "foo-1.0.0.dist-info/METADATA= ") =2D (rename-file zip-file file-name)) =2D (delete-file-recursively "foo-1.0.0.dist-info"))) + (begin + (mkdir "foo-1.0.0.dist-info") + (with-output-to-file "foo-1.0.0.dist-info/METADATA" + (lambda () + (display test-metadata))) + (let ((zip-file (string-append file-name ".zip"))) + ;; zip always adds a "zip" extension to the file it crea= tes, + ;; so we need to rename it. + (system* "zip" "-q" zip-file "foo-1.0.0.dist-info/METADA= TA") + (rename-file zip-file file-name)) + (delete-file-recursively "foo-1.0.0.dist-info"))) (_ (error "Unexpected URL: " url))))) (mock ((guix http-client) http-fetch (lambda (url . rest) @@ -262,6 +280,9 @@ Requires-Dist: pytest (>=3D3.1.0); extra =3D=3D 'testin= g' ('quasiquote (("python-bar" ('unquote 'python-bar)) ("python-baz" ('unquote 'python-baz))))) + ('native-inputs + ('quasiquote + (("python-pytest" ('unquote 'python-pytest))))) ('home-page "http://example.com") ('synopsis "summary") ('description "summary") @@ -272,4 +293,48 @@ Requires-Dist: pytest (>=3D3.1.0); extra =3D=3D 'testi= ng' (x (pk 'fail x #f)))))) =20 +(test-assert "pypi->guix-package, no usable requirement file." + ;; Replace network resources with sample data. + (mock ((guix import utils) url-fetch + (lambda (url file-name) + (match url + ("https://example.com/foo-1.0.0.tar.gz" + (set! test-source-hash + (call-with-input-file file-name port-sha256)) + #t) + ("https://example.com/foo-1.0.0-py2.py3-none-any.whl" #t) + (_ (error "Unexpected URL: " url))))) + (mock ((guix http-client) http-fetch + (lambda (url . rest) + (match url + ("https://pypi.org/pypi/foo/json" + (values (open-input-string test-json) + (string-length test-json))) + ("https://example.com/foo-1.0.0-py2.py3-none-any.whl" #= f) + (_ (error "Unexpected URL: " url))))) + ;; Not clearing the memoization cache here would mean return= ing the value + ;; computed in the previous test. + (invalidate-memoization! pypi->guix-package) + (match (pypi->guix-package "foo") + (('package + ('name "python-foo") + ('version "1.0.0") + ('source ('origin + ('method 'url-fetch) + ('uri ('pypi-uri "foo" 'version)) + ('sha256 + ('base32 + (? string? hash))))) + ('build-system 'python-build-system) + ('home-page "http://example.com") + ('synopsis "summary") + ('description "summary") + ('license 'license:lgpl2.0)) + (string=3D? (bytevector->nix-base32-string + test-source-hash) + hash)) + (x + (pk 'fail x #f))) + ))) + (test-end "pypi") =2D-=20 2.20.1 --=-=-= Content-Type: text/plain Thanks, Maxim --=-=-=-- --==-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEJ9WGpPiQCFQyn/CfEmDkZILmNWIFAlydoGMACgkQEmDkZILm NWJMOA/+Nk3rtALS/r5Su8aNXIZfC5z0p5Zk3WBc/FSOkZ0e+56qdiiLLQ6bEBrT xEmr23LqIiHfwU2H3H+mpZCKWDmqBb0edLW0u6cKRRO0eU5T55pEA71CyQj8c+mu UHNyFGm6y/hmaMKPVsyP8p3rhNl8TIPJe2vFXnN4LWaCJaj3E0X6Ge2fsnNTgzXk zpe5eU1vAFNfYOaf7uc658cfIPFzTEWgBfHASsfuKTLE34ZNB9HOxwskTNd1CvlN USjs3kSFRGQl2q0tXQzrxt9eNwQzcXWMk61bCYtDrZ4d87dhENKd2EAivMgGLGq/ 0poA4ijhb/zfmK7RNtHItivipzaZ+xAOpIEhcfwlmz2WxIZfV4eKkpLEB1q6QGAQ 0OKUmzAPHdYvNcHkhhAZIZl195JZ11HlrpTX5gUMqjL11d4ReuhlqNlAnynt8+tJ 8WgjVemJ7dhn8oZVLcXELBdFH01DZnbvF/8kheAyO6/dhTzKbxua+mPbNTpMn0w4 Ipe94BiF5DlaDffvWfmTebR9j+x7j8WSQeezc0GiNQuV8wOXKdVGsm+B66MdCpOH ioQ8B/62Tu/9ul4L+hkaZCBrVM/JI8W+SAeAn+CMQ0Zye20POpwn1xXK5SCn1FRU wXoFrEGE1L7yG+B406fRiABr711AItzTO9MvZQQmEquZBCN0qqI= =2rRk -----END PGP SIGNATURE----- --==-=-=--