From mboxrd@z Thu Jan 1 00:00:00 1970 From: Maxim Cournoyer Subject: bug#24450: [PATCH] bug#24450: pypi importer outputs strange character series in optional dependency case. Date: Sun, 31 Mar 2019 10:40:34 -0400 Message-ID: <87ftr32jkt.fsf_-_@gmail.com> References: <87h99fipj1.fsf@we.make.ritual.n0.is> <87tvfm1eos.fsf@gmail.com> <877ech5cvd.fsf_-_@kwak.i-did-not-set--mail-host-address--so-tickle-me> Mime-Version: 1.0 Content-Type: multipart/signed; boundary="==-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" Return-path: Received: from eggs.gnu.org ([209.51.188.92]:50558) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hAbdw-0003ot-2q for bug-guix@gnu.org; Sun, 31 Mar 2019 10:41:05 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hAbdu-0001IA-FP for bug-guix@gnu.org; Sun, 31 Mar 2019 10:41:04 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:53074) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1hAbdt-0001Hg-TE for bug-guix@gnu.org; Sun, 31 Mar 2019 10:41:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1hAbdt-00083Y-L0 for bug-guix@gnu.org; Sun, 31 Mar 2019 10:41:01 -0400 Sender: "Debbugs-submit" Resent-Message-ID: In-Reply-To: <877ech5cvd.fsf_-_@kwak.i-did-not-set--mail-host-address--so-tickle-me> (T460s laptop's message of "Fri, 29 Mar 2019 22:12:38 -0400") List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+gcggb-bug-guix=m.gmane.org@gnu.org Sender: "bug-Guix" To: ng0 Cc: 24450@debbugs.gnu.org --==-=-= Content-Type: multipart/mixed; boundary="=-=-=" --=-=-= Content-Type: text/plain Hello! I've yet another commit to add on the top of the current stack of patches already sent. This one makes it so that the source archive is searched for the '[...].egg-info/requires.txt' file rather than check the more common, fixed location. This makes the importer more useful, as some packages such as robotframework-sshlibrary use a "src" directory in their hierarchy which would cause the previous scheme to not find requires.txt. --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0008-import-pypi-Scan-source-archive-to-find-requires.txt.patch Content-Transfer-Encoding: quoted-printable From=20cfde6e09f8f8c692fe252d76ed27e8c50a9e5377 Mon Sep 17 00:00:00 2001 From: Maxim Cournoyer Date: Sat, 30 Mar 2019 23:13:26 -0400 Subject: [PATCH] import: pypi: Scan source archive to find requires.txt fil= e. * guix/import/pypi.scm (use-modules): Use invoke from (guix build utils). (guess-requirements)[archive-root-directory]: Remove procedure. [guess-requirements-from-wheel]: Re-ident. [guess-requirements-from-source]: Search for the requires.txt file in the archive instead of using a static, expected location. * tests/pypi.scm ("pypi->guix-package, no wheel"): Mock the requires.txt at= a non-standard location to test the new feature. ("pypi->guix-package, no usable requirement file."): Adapt. =2D-- guix/import/pypi.scm | 65 ++++++++++++++++++-------------------------- tests/pypi.scm | 17 +++++++----- 2 files changed, 37 insertions(+), 45 deletions(-) diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm index 099768f0c8..a2ce14b192 100644 =2D-- a/guix/import/pypi.scm +++ b/guix/import/pypi.scm @@ -36,7 +36,8 @@ #:use-module ((guix build utils) #:select ((package-name->name+version . hyphen-package-name->name+version) =2D find-files)) + find-files + invoke)) #:use-module (guix import utils) #:use-module ((guix download) #:prefix download:) #:use-module (guix import json) @@ -267,19 +268,6 @@ omitted since these can be difficult or expensive to s= atisfy." of the required packages specified in the requirements.txt file. ARCHIVE = will be extracted in a temporary directory." =20 =2D (define (archive-root-directory url) =2D ;; Given the URL of the package's archive, return the name of the di= rectory =2D ;; that will be created upon decompressing it. If the filetype is not =2D ;; supported, return #f. =2D (if (compressed-file? url) =2D (let ((root-directory (file-sans-extension (basename url)))) =2D (if (string=3D? "tar" (file-extension root-directory)) =2D (file-sans-extension root-directory) =2D root-directory)) =2D (begin =2D (warning (G_ "Unsupported archive format (~a): \ =2Dcannot determine package dependencies") (file-extension url)) =2D #f))) =20 (define (read-wheel-metadata wheel-archive) ;; Given WHEEL-ARCHIVE, a ZIP Python wheel archive, return the package= 's @@ -305,33 +293,34 @@ cannot determine package dependencies") (file-extensi= on url)) (call-with-temporary-output-file (lambda (temp port) (if wheel-url =2D (and (url-fetch wheel-url temp) =2D (read-wheel-metadata temp)) =2D #f)))) + (and (url-fetch wheel-url temp) + (read-wheel-metadata temp)) + #f)))) =20 (define (guess-requirements-from-source) ;; Return the package's requirements by guessing them from the source. =2D (let ((dirname (archive-root-directory source-url)) =2D (extension (file-extension source-url))) =2D (if (string? dirname) =2D (call-with-temporary-directory =2D (lambda (dir) =2D (let* ((pypi-name (string-take dirname (string-rindex dirna= me #\-))) =2D (requires.txt (string-append dirname "/" pypi-name =2D ".egg-info" "/requires.= txt")) =2D (exit-code =2D (parameterize ((current-error-port (%make-void-port= "rw+")) =2D (current-output-port (%make-void-por= t "rw+"))) =2D (if (string=3D? "zip" extension) =2D (system* "unzip" archive "-d" dir requires.tx= t) =2D (system* "tar" "xf" archive "-C" dir requires= .txt))))) =2D (if (zero? exit-code) =2D (parse-requires.txt (string-append dir "/" requires.t= xt)) =2D (begin =2D (warning =2D (G_ "Failed to extract file: ~a from source.~%") =2D requires.txt) =2D (list '() '())))))) + (if (compressed-file? source-url) + (call-with-temporary-directory + (lambda (dir) + (parameterize ((current-error-port (%make-void-port "rw+")) + (current-output-port (%make-void-port "rw+"))) + (if (string=3D? "zip" (file-extension source-url)) + (invoke "unzip" archive "-d" dir) + (invoke "tar" "xf" archive "-C" dir))) + (let ((requires.txt-files + (find-files dir (lambda (abs-file-name _) + (string-match "\\.egg-info/requires.txt$" + abs-file-name))))) + (if (> (length requires.txt-files) 0) + (begin + (parse-requires.txt (first requires.txt-files))) + (begin (warning (G_ "Cannot guess requirements from sourc= e archive:\ + no requires.txt file found.~%")) + (list '() '())))))) + (begin + (warning (G_ "Unsupported archive format; \ +cannot determine package dependencies from source archive: ~a~%") + (basename source-url)) (list '() '())))) =20 ;; First, try to compute the requirements using the wheel, else, fallbac= k to diff --git a/tests/pypi.scm b/tests/pypi.scm index aa08e2cb54..ad188df16c 100644 =2D-- a/tests/pypi.scm +++ b/tests/pypi.scm @@ -177,8 +177,9 @@ Requires-Dist: pytest (>=3D3.1.0); extra =3D=3D 'testin= g' (match url ("https://example.com/foo-1.0.0.tar.gz" (begin =2D (mkdir-p "foo-1.0.0/foo.egg-info/") =2D (with-output-to-file "foo-1.0.0/foo.egg-info/requires.= txt" + ;; Unusual requires.txt location should still be found. + (mkdir-p "foo-1.0.0/src/bizarre.egg-info") + (with-output-to-file "foo-1.0.0/src/bizarre.egg-info/req= uires.txt" (lambda () (display test-requires.txt))) (parameterize ((current-output-port (%make-void-port "rw= +"))) @@ -299,10 +300,13 @@ Requires-Dist: pytest (>=3D3.1.0); extra =3D=3D 'test= ing' (lambda (url file-name) (match url ("https://example.com/foo-1.0.0.tar.gz" + (mkdir-p "foo-1.0.0/foo.egg-info/") + (parameterize ((current-output-port (%make-void-port "rw+"))) + (system* "tar" "czvf" file-name "foo-1.0.0/")) + (delete-file-recursively "foo-1.0.0") (set! test-source-hash =2D (call-with-input-file file-name port-sha256)) =2D #t) =2D ("https://example.com/foo-1.0.0-py2.py3-none-any.whl" #t) + (call-with-input-file file-name port-sha256))) + ("https://example.com/foo-1.0.0-py2.py3-none-any.whl" #f) (_ (error "Unexpected URL: " url))))) (mock ((guix http-client) http-fetch (lambda (url . rest) @@ -334,7 +338,6 @@ Requires-Dist: pytest (>=3D3.1.0); extra =3D=3D 'testin= g' test-source-hash) hash)) (x =2D (pk 'fail x #f))) =2D ))) + (pk 'fail x #f)))))) =20 (test-end "pypi") =2D-=20 2.20.1 --=-=-= Content-Type: text/plain Thanks, Maxim --=-=-=-- --==-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEJ9WGpPiQCFQyn/CfEmDkZILmNWIFAlyg0WIACgkQEmDkZILm NWIEMA/9HsMlrdzTwtllz/jPaAM13pPbyvI1sx1uwQi6AYessiZYBr/vGiFEBBTB FJJUQ9Sz0XQOTW3wOitgTRz1kr55c17cv80IvwWYhTZLGZ+O2F29hFpp38OucRS7 d4XeZSJwBSn9ccYgytebd0Xtkp340HHeLOHnNTnA600TLwTq/6YXP4hLi5hiU7p1 zIOHe3Nze9dk1B30tuivqwvIVkSMnnPQyCmPSE+1vhzczVQ+m+f5PxxmSbP6Kznx KF7Rlm+Ltjup0JfiKeglNiqLYYOKRpYuWM+oMzWSJgl1zA/MzDRZCnbs30TD/XKd dkixB0Hmazer2BBRJoNSgCKu/Mx8hZg6ML/NcgrbmJnUd4Sw4r/M2My/Iq1D6RLA Kkj7gzoqqowy+PcsU+/l+XrLdvT0wd6VR2x+DwY+y3Fiv8DSQRWI0mXxl9V4M4St gF27p1RhfyUhIQo801AtJ+Tin/N5cYik8orRnARHnWNXVy7Iqz1RGr4MIqGheS0l 8GMOMVeLskAqagLynhWY4fz0teGCnaGAjVyzda7R1tYBkWzsioLugzsDoIftpEA8 p0TBiVEu0/BAWlL/BYdWyOIbkPQ05xV+Yad+QvECS3uu4/MckAcEwLMQpPjqKP8O J5tRiRMRkEP1Dxl7oBuKC1IOHDhGvUOuIqFKqmyGRcBInwTC5Is= =H7lm -----END PGP SIGNATURE----- --==-=-=--