From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id YNACMW1X1GAcywAAgWs5BA (envelope-from ) for ; Thu, 24 Jun 2021 11:59:09 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id 8D/GLG1X1GBVTwAA1q6Kng (envelope-from ) for ; Thu, 24 Jun 2021 09:59:09 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id C4DF025219 for ; Thu, 24 Jun 2021 11:59:08 +0200 (CEST) Received: from localhost ([::1]:46344 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lwM8Z-00022m-TD for larch@yhetil.org; Thu, 24 Jun 2021 05:59:07 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:57620) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lwM8T-00022c-Qg for bug-guix@gnu.org; Thu, 24 Jun 2021 05:59:01 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:59179) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lwM8T-0006Ai-Hp for bug-guix@gnu.org; Thu, 24 Jun 2021 05:59:01 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lwM8T-0002E0-IN for bug-guix@gnu.org; Thu, 24 Jun 2021 05:59:01 -0400 X-Loop: help-debbugs@gnu.org Subject: bug#49168: =?UTF-8?Q?=E2=80=98guix?= import =?UTF-8?Q?pypi=E2=80=99?= misses package dependencies Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Thu, 24 Jun 2021 09:59:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 49168 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Leo Prikler Received: via spool by 49168-submit@debbugs.gnu.org id=B49168.16245286838485 (code B ref 49168); Thu, 24 Jun 2021 09:59:01 +0000 Received: (at 49168) by debbugs.gnu.org; 24 Jun 2021 09:58:03 +0000 Received: from localhost ([127.0.0.1]:42492 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lwM7W-0002Cn-Nf for submit@debbugs.gnu.org; Thu, 24 Jun 2021 05:58:03 -0400 Received: from eggs.gnu.org ([209.51.188.92]:59728) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lwM7U-0002C1-Mm for 49168@debbugs.gnu.org; Thu, 24 Jun 2021 05:58:01 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:41640) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lwM7N-0005Br-P9; Thu, 24 Jun 2021 05:57:53 -0400 Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=48426 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lwM7N-0002N1-GG; Thu, 24 Jun 2021 05:57:53 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <877dim5psf.fsf@inria.fr> <992e746ed43c5ae7def7b35996ed44743fef85bf.camel@student.tugraz.at> <87eecu3wqh.fsf@inria.fr> <87mtrgyaxe.fsf@gnu.org> <7da007493c9bcea075a76dc01d6e85b457c5d4e3.camel@student.tugraz.at> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 6 Messidor an 229 de la =?UTF-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Thu, 24 Jun 2021 11:57:51 +0200 In-Reply-To: <7da007493c9bcea075a76dc01d6e85b457c5d4e3.camel@student.tugraz.at> (Leo Prikler's message of "Wed, 23 Jun 2021 16:17:11 +0200") Message-ID: <87a6nftwio.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: 49168@debbugs.gnu.org Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: "bug-Guix" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1624528749; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:in-reply-to:in-reply-to:references:references: list-id:list-help:list-unsubscribe:list-subscribe:list-post; bh=+1GqS2H2IPHY2X+20cZRt5wpouQctSY8t6bJ42k2erE=; b=hl6p58Zm2HB498AhTHQU+D/UxmFoEfGMZiSGM6RLItncclJ2kPEBP9YjXRODSpnBL1nwdd smXRHlD/1TFr9UlcTVIOf37WCpLUl/IRVuX2be5Nk96NbHwOC+TYqCZL621KID1OWWhQp/ EpPSaWyRcIYxwtgQnaVx7IldyUveyDuJvYx9iE27Nb1NMR8XerQFBwBXRi7u/M1GGuwk+8 ZRyjd/RH3o8qPrwp20cJdW/H4+jrJyg4HT78mve/NVRIjqJmv6L0D4otGR5FT9WGGJQMHi gWlqrfGw3sbmi7kRgYB4P91C2DNOn8Qb2zvWA/xxU8v3boXa8wzgXma+5fN8cw== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1624528749; a=rsa-sha256; cv=none; b=hkbUdF8ycLjRsjfokEBv8zDZCDIwNhWDL0W1ZVygf1H9qvRE6OgHv4kHvEch339rT4XC5S 3J8SYsD9XXwlp6KsjoCH3ecqwnX88ks0FYOpE1eO93iyLop4ecs4sbwHCy6xS9kAXICCP8 E7PQ+WfQYrey0vF3qozpFr8ThoHeX/YCQ94RJ1pyYonDJFVnF5AyJSFt9O8QcHoO5OQCTB tE0Q0/1nR4hlS/aBE8zEtUPWnXA9kNA/IrDBpzU6fKcB1gAJ30Vipvo0UDuzgBq/oEklOw l2A62L7GkHOWuxRo4cCxQoMn8U3Ja8Kh4cpb4j17qJqUYRCwfsYstbQpatEt7w== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; spf=pass (aspmx1.migadu.com: domain of bug-guix-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=bug-guix-bounces@gnu.org X-Migadu-Spam-Score: -1.43 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of bug-guix-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=bug-guix-bounces@gnu.org X-Migadu-Queue-Id: C4DF025219 X-Spam-Score: -1.43 X-Migadu-Scanner: scn1.migadu.com X-TUID: 3kXThagasw05 --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi, Leo Prikler skribis: > Am Mittwoch, den 23.06.2021, 15:20 +0200 schrieb Ludovic Court=C3=A8s: >> Hi, >>=20 >> Leo Prikler skribis: >>=20 >> > Am Dienstag, den 22.06.2021, 14:33 +0200 schrieb Ludovic Court=C3=A8s: >>=20 >> [...] >>=20 >> > > Actually >> > > < >> > > https://files.pythonhosted.org/packages/fe/9d/4e15b2e74044ee051b6939= c1b3ff716b0106e8f72d78eab8e08212eab44c/tablib-3.0.0.tar.gz >> > > does not have a =E2=80=98requirements.txt=E2=80=99 file, and >> > > < >> > > https://files.pythonhosted.org/packages/16/85/078fc037b15aa1120d6a02= 87ec9d092d93d632ab01a0e7a3e69b4733da5e/tablib-3.0.0-py3-none-any.whl >> > > doesn=E2=80=99t have much metadata, so I don=E2=80=99t even get wher= e were get >> > > that >> > > info. >> > It does, but it's well hidden in the src tree. I peeked into the >> > guix >> > import code to find it. >>=20 >> Indeed. The tarball above has =E2=80=98tests/requirements.txt=E2=80=99: >>=20 >> --8<---------------cut here---------------start------------->8--- >> pytest >> pytest-cov >> MarkupPy >> odfpy >> openpyxl>=3D2.6.0 >> pandas >> pyyaml >> tabulate >> xlrd >> xlwt >> --8<---------------cut here---------------end--------------->8--- >>=20 >> There are no optional dependencies in that file, though. Or were you >> looking at something else? > The importer and I are looking at something else: > > $ tar xfv tablib-3.0.0.tar.gz tablib-3.0.0/src/tablib.egg- > info/requires.txt | xargs cat > > [all] > markuppy > odfpy > openpyxl>=3D2.6.0 > pandas > pyyaml > tabulate > xlrd > xlwt > > [cli] > tabulate > > [html] > markuppy > > [ods] > odfpy > > [pandas] > pandas > > [xls] > xlrd > xlwt > > [xlsx] > openpyxl>=3D2.6.0 > > [yaml] > pyyaml Oooh, I see. So I came up with the following patch, which adds a flag for optional dependencies, based on =E2=80=98requires.txt=E2=80=99. Unfortunately, it has no effect for =E2=80=98guix import pypi tablib=E2=80= =99 because we only look at Wheel info in that case, as per: ;; First, try to compute the requirements using the wheel, else, fallback= to ;; reading the "requires.txt" from the egg-info directory from the source ;; archive. (or (guess-requirements-from-wheel) (guess-requirements-from-source)) AFAICS, wheels don=E2=80=99t provide that info, do they? Why does the importer favor .whl in the first place? Is it supposed to be more accurate or more widespread or something? Thoughts? Thanks, Ludo=E2=80=99. --=-=-= Content-Type: text/x-patch; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm index 6731d50891..be01cc99d2 100644 --- a/guix/import/pypi.scm +++ b/guix/import/pypi.scm @@ -1,7 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright =C2=A9 2014 David Thompson ;;; Copyright =C2=A9 2015 Cyril Roelandt -;;; Copyright =C2=A9 2015, 2016, 2017, 2019, 2020 Ludovic Court=C3=A8s +;;; Copyright =C2=A9 2015, 2016, 2017, 2019, 2020, 2021 Ludovic Court=C3= =A8s ;;; Copyright =C2=A9 2017 Mathieu Othacehe ;;; Copyright =C2=A9 2018 Ricardo Wurmus ;;; Copyright =C2=A9 2019 Maxim Cournoyer @@ -232,10 +232,9 @@ the input field." "Given REQUIRES.TXT, a path to a Setuptools requires.txt file, return a = list of lists of requirements. =20 -The first list contains the required dependencies while the second the -optional test dependencies. Note that currently, optional, non-test -dependencies are omitted since these can be difficult or expensive to -satisfy." +The first list contains the required dependencies, the second contains test +dependencies, and the third one contains optional dependencies. Note that +optional, non-test dependencies can be difficult or expensive to satisfy." =20 (define (comment? line) ;; Return #t if the given LINE is a comment, #f otherwise. @@ -249,6 +248,7 @@ satisfy." (lambda (port) (let loop ((required-deps '()) (test-deps '()) + (optional-deps '()) (inside-test-section? #f) (optional? #f)) (let ((line (read-line port))) @@ -259,27 +259,33 @@ satisfy." ;; pytest >=3D 3 ; python_version >=3D "3.3" ;; pytest < 3 ; python_version < "3.3" (map (compose reverse delete-duplicates) - (list required-deps test-deps))) + (list required-deps test-deps optional-deps))) ((or (string-null? line) (comment? line)) - (loop required-deps test-deps inside-test-section? optional?)) + (loop required-deps test-deps optional-deps + inside-test-section? optional?)) ((section-header? line) ;; Encountering a section means that all the requirements ;; listed below are optional. Since we want to pick only the ;; test dependencies from the optional dependencies, we must ;; track those separately. - (loop required-deps test-deps (test-section? line) #t)) + (loop required-deps test-deps optional-deps + (test-section? line) #t)) (inside-test-section? (loop required-deps (cons (specification->requirement-name line) test-deps) + optional-deps inside-test-section? optional?)) ((not optional?) (loop (cons (specification->requirement-name line) required-deps) - test-deps inside-test-section? optional?)) + test-deps optional-deps + inside-test-section? optional?)) (optional? - ;; Skip optional items. - (loop required-deps test-deps inside-test-section? optional?)) + (loop required-deps test-deps + (cons (specification->requirement-name line) + optional-deps) + inside-test-section? optional?)) (else (warning (G_ "parse-requires.txt reached an unexpected \ condition on line ~a~%") line)))))))) @@ -314,7 +320,8 @@ returned value." (cond ((eof-object? line) (map (compose reverse delete-duplicates) - (list required-deps test-deps))) + (list required-deps test-deps + '()))) ;XXX: no known optional dependen= cies ((and (requires-dist-header? line) (not (extra? line))) (loop (cons (specification->requirement-name (requires-dist-value line)) @@ -392,7 +399,7 @@ cannot determine package dependencies from source archi= ve: ~a~%") (or (guess-requirements-from-wheel) (guess-requirements-from-source))) =20 -(define (compute-inputs source-url wheel-url archive) +(define* (compute-inputs source-url wheel-url archive) "Given the SOURCE-URL and WHEEL-URL of an already downloaded ARCHIVE, re= turn a pair of lists, each consisting of a list of name/variable pairs, for the propagated inputs and the native inputs, respectively. Also @@ -419,17 +426,19 @@ return the unaltered list of upstream dependency name= s." (values (map process-requirements dependencies) (concatenate dependencies)))) =20 -(define (make-pypi-sexp name version source-url wheel-url home-page synops= is - description license) +(define* (make-pypi-sexp name version source-url wheel-url home-page synop= sis + description license + #:key optional-dependencies?) "Return the `package' s-expression for a python package with the given N= AME, -VERSION, SOURCE-URL, HOME-PAGE, SYNOPSIS, DESCRIPTION, and LICENSE." +VERSION, SOURCE-URL, HOME-PAGE, SYNOPSIS, DESCRIPTION, and LICENSE. When +OPTIONAL-DEPENDENCIES? is true, include optional dependencies." (call-with-temporary-output-file (lambda (temp port) (and (url-fetch source-url temp) (receive (guix-dependencies upstream-dependencies) (compute-inputs source-url wheel-url temp) (match guix-dependencies - ((required-inputs native-inputs) + ((required-inputs native-inputs optional-inputs) (when (string-suffix? ".zip" source-url) (set! native-inputs (cons '("unzip" ,unzip) @@ -462,7 +471,12 @@ VERSION, SOURCE-URL, HOME-PAGE, SYNOPSIS, DESCRIPTION,= and LICENSE." (base32 ,(guix-hash-url temp))))) (build-system python-build-system) - ,@(maybe-inputs required-inputs 'propagated-inputs) + ,@(maybe-inputs + (append required-inputs + (if optional-dependencies? + optional-inputs ;TODO: emit a comment + '())) + 'propagated-inputs) ,@(maybe-inputs native-inputs 'native-inputs) (home-page ,home-page) (synopsis ,synopsis) @@ -472,9 +486,10 @@ VERSION, SOURCE-URL, HOME-PAGE, SYNOPSIS, DESCRIPTION,= and LICENSE." =20 (define pypi->guix-package (memoize - (lambda* (package-name #:key repo version) + (lambda* (package-name #:key repo version optional-dependencies?) "Fetch the metadata for PACKAGE-NAME from pypi.org, and return the -`package' s-expression corresponding to that package, or #f on failure." +`package' s-expression corresponding to that package, or #f on failure. W= hen +OPTIONAL-DEPENDENCIES? is true, include optional dependencies." (let* ((project (pypi-fetch package-name)) (info (and project (pypi-project-info project)))) (and project @@ -493,11 +508,17 @@ VERSION, SOURCE-URL, HOME-PAGE, SYNOPSIS, DESCRIPTION= , and LICENSE." (project-info-summary info) (project-info-summary info) (string->license - (project-info-license info))))))))) + (project-info-license info)) + #:optional-dependencies? + optional-dependencies?))))))) =20 -(define (pypi-recursive-import package-name) +(define* (pypi-recursive-import package-name #:key optional-dependencies?) (recursive-import package-name - #:repo->guix-package pypi->guix-package + #:repo->guix-package + (lambda args + (apply pypi->guix-package + #:optional-dependencies? optional-dependencie= s? + args)) #:guix-name python->package-name)) =20 (define (string->license str) diff --git a/guix/scripts/import/pypi.scm b/guix/scripts/import/pypi.scm index 33167174e2..58ae07f802 100644 --- a/guix/scripts/import/pypi.scm +++ b/guix/scripts/import/pypi.scm @@ -46,6 +46,8 @@ Import and convert the PyPI package for PACKAGE-NAME.\n")) (display (G_ " -r, --recursive import packages recursively")) (display (G_ " + -O, --optional include optional dependencies")) + (display (G_ " -V, --version display version information and exit")) (newline) (show-bug-report-information)) @@ -62,6 +64,9 @@ Import and convert the PyPI package for PACKAGE-NAME.\n")) (option '(#\r "recursive") #f #f (lambda (opt name arg result) (alist-cons 'recursive #t result))) + (option '(#\O "optional") #f #f + (lambda (opt name arg result) + (alist-cons 'optional-dependencies? #t result))) %standard-import-options)) =20 @@ -85,6 +90,9 @@ Import and convert the PyPI package for PACKAGE-NAME.\n")) value) (_ #f)) (reverse opts)))) + (define optional-dependencies? + (assoc-ref opts 'optional-dependencies?)) + (match args ((package-name) (if (assoc-ref opts 'recursive) @@ -94,9 +102,13 @@ Import and convert the PyPI package for PACKAGE-NAME.\n= ")) `(define-public ,(string->symbol name) ,pkg)) (_ #f)) - (pypi-recursive-import package-name)) + (pypi-recursive-import package-name + #:optional-dependencies? + optional-dependencies?)) ;; Single import - (let ((sexp (pypi->guix-package package-name))) + (let ((sexp (pypi->guix-package package-name + #:optional-dependencies? + optional-dependencies?))) (unless sexp (leave (G_ "failed to download meta-data for package '~a'~%= ") package-name)) --=-=-=--