bug#49168: ‘guix import pypi’ misses package dependencies

bug#49168: ‘guix import pypi’ misses package dependencies

[Ludovic Courtès]

Hi!

‘guix import pypi -r django-import-export’ does not include all the
inputs:

--8<---------------cut here---------------start------------->8---
$ guix build -L /tmp/mods python-django-import-export

[...]

starting phase `build'
running "python setup.py" with command "build" and parameters ()
Download error on https://pypi.org/simple/setuptools_scm/: [Errno -2] Name or service not known -- Some packages may not be found!
Download error on https://pypi.org/simple/setuptools-scm/: [Errno -2] Name or service not known -- Some packages may not be found!
Couldn't find index page for 'setuptools_scm' (maybe misspelled?)
Download error on https://pypi.org/simple/: [Errno -2] Name or service not known -- Some packages may not be found!
No local packages or working download links found for setuptools_scm
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "setup.py", line 5, in <module>
    setup(
  File "/gnu/store/9w9jvy3bgjg4qaqmrij01nbppiccqr7c-python-3.8.2/lib/python3.8/site-packages/setuptools/__init__.py", line 144, in setup
    _install_setup_requires(attrs)
  File "/gnu/store/9w9jvy3bgjg4qaqmrij01nbppiccqr7c-python-3.8.2/lib/python3.8/site-packages/setuptools/__init__.py", line 139, in _install_setup_requires
    dist.fetch_build_eggs(dist.setup_requires)
  File "/gnu/store/9w9jvy3bgjg4qaqmrij01nbppiccqr7c-python-3.8.2/lib/python3.8/site-packages/setuptools/dist.py", line 716, in fetch_build_eggs
    resolved_dists = pkg_resources.working_set.resolve(
  File "/gnu/store/9w9jvy3bgjg4qaqmrij01nbppiccqr7c-python-3.8.2/lib/python3.8/site-packages/pkg_resources/__init__.py", line 780, in resolve
    dist = best[req.key] = env.best_match(
  File "/gnu/store/9w9jvy3bgjg4qaqmrij01nbppiccqr7c-python-3.8.2/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1065, in best_match
    return self.obtain(req, installer)
  File "/gnu/store/9w9jvy3bgjg4qaqmrij01nbppiccqr7c-python-3.8.2/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1077, in obtain
    return installer(requirement)
  File "/gnu/store/9w9jvy3bgjg4qaqmrij01nbppiccqr7c-python-3.8.2/lib/python3.8/site-packages/setuptools/dist.py", line 786, in fetch_build_egg
    return cmd.easy_install(req)
  File "/gnu/store/9w9jvy3bgjg4qaqmrij01nbppiccqr7c-python-3.8.2/lib/python3.8/site-packages/setuptools/command/easy_install.py", line 673, in easy_install
    raise DistutilsError(msg)
distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('setuptools_scm')
command "python" "-c" "import setuptools, tokenize;__file__='setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\\r\\n', '\\n');f.close();exec(compile(code, __file__, 'exec'))" "build" failed with status 1
builder for `/gnu/store/74ph75dfy1i9hrx9n327j32ma2zv4hhd-python-tablib-3.0.0.drv' failed with exit code 1
build of /gnu/store/74ph75dfy1i9hrx9n327j32ma2zv4hhd-python-tablib-3.0.0.drv failed
View build log at '/var/log/guix/drvs/74/ph75dfy1i9hrx9n327j32ma2zv4hhd-python-tablib-3.0.0.drv.bz2'.
cannot build derivation `/gnu/store/6xk0jk0qnjsazqgjihmpafsx3wfmxjf0-python-django-import-export-2.5.0.drv': 1 dependencies couldn't be built
guix build: error: build of `/gnu/store/6xk0jk0qnjsazqgjihmpafsx3wfmxjf0-python-django-import-export-2.5.0.drv' failed
$ cat /tmp/mods/django.scm
(define-module (django)
  #:use-module (guix)
  #:use-module (guix build-system python)
  #:use-module (gnu packages python)
  #:use-module (gnu packages python-xyz)
  #:use-module (gnu packages django)
  #:use-module ((guix licenses) #:prefix license:))

(define-public python-tablib
  (package
    (name "python-tablib")
    (version "3.0.0")
    (source
      (origin
        (method url-fetch)
        (uri (pypi-uri "tablib" version))
        (sha256
          (base32
            "03f1z6jq6rf67gwhbm9ma4rydm8h447a5nh5lcs5l8jg8l4aqg7q"))))
    (build-system python-build-system)
    (home-page "https://tablib.readthedocs.io")
    (synopsis
      "Format agnostic tabular data library (XLS, JSON, YAML, CSV)")
    (description
      "Format agnostic tabular data library (XLS, JSON, YAML, CSV)")
    (license license:expat)))

(define-public python-django-import-export
  (package
    (name "python-django-import-export")
    (version "2.5.0")
    (source
      (origin
        (method url-fetch)
        (uri (pypi-uri "django-import-export" version))
        (sha256
          (base32
            "0j0gsj18bkkmbn262qj2syla80vncvqn49bllwxvcgw0zhxh1763"))))
    (build-system python-build-system)
    (propagated-inputs
      `(("python-diff-match-patch"
         ,python-diff-match-patch)
        ("python-django" ,python-django)
        ("python-tablib" ,python-tablib)))
    (home-page
      "https://github.com/django-import-export/django-import-export")
    (synopsis
      "Django application and library for importing and exporting data with included admin integration.")
    (description
      "Django application and library for importing and exporting data with included admin integration.")
    (license license:bsd-3)))

--8<---------------cut here---------------end--------------->8---

Here, some of tablib’s dependencies are missing; the same problem seems
to happen with python-django-import-export itself.

Are we failing to parse some of the PyPI metadata?

Thanks,
Ludo’.

bug#49168: ‘guix import pypi’ misses package dependencies

[Leo Prikler]

Am Dienstag, den 22.06.2021, 09:20 +0200 schrieb Ludovic Courtès:
> Hi!
> 
> ‘guix import pypi -r django-import-export’ does not include all the
> inputs:
> 
> --8<---------------cut here---------------start------------->8---
> $ guix build -L /tmp/mods python-django-import-export
> 
> [...]
> 
> starting phase `build'
> running "python setup.py" with command "build" and parameters ()
> Download error on https://pypi.org/simple/setuptools_scm/: [Errno -2]
> Name or service not known -- Some packages may not be found!
> Download error on https://pypi.org/simple/setuptools-scm/: [Errno -2]
> Name or service not known -- Some packages may not be found!
> Couldn't find index page for 'setuptools_scm' (maybe misspelled?)
> Download error on https://pypi.org/simple/: [Errno -2] Name or
> service not known -- Some packages may not be found!
> No local packages or working download links found for setuptools_scm
> Traceback (most recent call last):
>   File "<string>", line 1, in <module>
>   File "setup.py", line 5, in <module>
>     setup(
>   File "/gnu/store/9w9jvy3bgjg4qaqmrij01nbppiccqr7c-python-
> 3.8.2/lib/python3.8/site-packages/setuptools/__init__.py", line 144,
> in setup
>     _install_setup_requires(attrs)
>   File "/gnu/store/9w9jvy3bgjg4qaqmrij01nbppiccqr7c-python-
> 3.8.2/lib/python3.8/site-packages/setuptools/__init__.py", line 139,
> in _install_setup_requires
>     dist.fetch_build_eggs(dist.setup_requires)
>   File "/gnu/store/9w9jvy3bgjg4qaqmrij01nbppiccqr7c-python-
> 3.8.2/lib/python3.8/site-packages/setuptools/dist.py", line 716, in
> fetch_build_eggs
>     resolved_dists = pkg_resources.working_set.resolve(
>   File "/gnu/store/9w9jvy3bgjg4qaqmrij01nbppiccqr7c-python-
> 3.8.2/lib/python3.8/site-packages/pkg_resources/__init__.py", line
> 780, in resolve
>     dist = best[req.key] = env.best_match(
>   File "/gnu/store/9w9jvy3bgjg4qaqmrij01nbppiccqr7c-python-
> 3.8.2/lib/python3.8/site-packages/pkg_resources/__init__.py", line
> 1065, in best_match
>     return self.obtain(req, installer)
>   File "/gnu/store/9w9jvy3bgjg4qaqmrij01nbppiccqr7c-python-
> 3.8.2/lib/python3.8/site-packages/pkg_resources/__init__.py", line
> 1077, in obtain
>     return installer(requirement)
>   File "/gnu/store/9w9jvy3bgjg4qaqmrij01nbppiccqr7c-python-
> 3.8.2/lib/python3.8/site-packages/setuptools/dist.py", line 786, in
> fetch_build_egg
>     return cmd.easy_install(req)
>   File "/gnu/store/9w9jvy3bgjg4qaqmrij01nbppiccqr7c-python-
> 3.8.2/lib/python3.8/site-
> packages/setuptools/command/easy_install.py", line 673, in
> easy_install
>     raise DistutilsError(msg)
> distutils.errors.DistutilsError: Could not find suitable distribution
> for Requirement.parse('setuptools_scm')
> command "python" "-c" "import setuptools,
> tokenize;__file__='setup.py';f=getattr(tokenize, 'open',
> open)(__file__);code=f.read().replace('\\r\\n',
> '\\n');f.close();exec(compile(code, __file__, 'exec'))" "build"
> failed with status 1
> builder for `/gnu/store/74ph75dfy1i9hrx9n327j32ma2zv4hhd-python-
> tablib-3.0.0.drv' failed with exit code 1
> build of /gnu/store/74ph75dfy1i9hrx9n327j32ma2zv4hhd-python-tablib-
> 3.0.0.drv failed
> View build log at
> '/var/log/guix/drvs/74/ph75dfy1i9hrx9n327j32ma2zv4hhd-python-tablib-
> 3.0.0.drv.bz2'.
> cannot build derivation `/gnu/store/6xk0jk0qnjsazqgjihmpafsx3wfmxjf0-
> python-django-import-export-2.5.0.drv': 1 dependencies couldn't be
> built
> guix build: error: build of
> `/gnu/store/6xk0jk0qnjsazqgjihmpafsx3wfmxjf0-python-django-import-
> export-2.5.0.drv' failed
> $ cat /tmp/mods/django.scm
> (define-module (django)
>   #:use-module (guix)
>   #:use-module (guix build-system python)
>   #:use-module (gnu packages python)
>   #:use-module (gnu packages python-xyz)
>   #:use-module (gnu packages django)
>   #:use-module ((guix licenses) #:prefix license:))
> 
> (define-public python-tablib
>   (package
>     (name "python-tablib")
>     (version "3.0.0")
>     (source
>       (origin
>         (method url-fetch)
>         (uri (pypi-uri "tablib" version))
>         (sha256
>           (base32
>             "03f1z6jq6rf67gwhbm9ma4rydm8h447a5nh5lcs5l8jg8l4aqg7q")))
> )
>     (build-system python-build-system)
>     (home-page "https://tablib.readthedocs.io")
>     (synopsis
>       "Format agnostic tabular data library (XLS, JSON, YAML, CSV)")
>     (description
>       "Format agnostic tabular data library (XLS, JSON, YAML, CSV)")
>     (license license:expat)))
> 
> (define-public python-django-import-export
>   (package
>     (name "python-django-import-export")
>     (version "2.5.0")
>     (source
>       (origin
>         (method url-fetch)
>         (uri (pypi-uri "django-import-export" version))
>         (sha256
>           (base32
>             "0j0gsj18bkkmbn262qj2syla80vncvqn49bllwxvcgw0zhxh1763")))
> )
>     (build-system python-build-system)
>     (propagated-inputs
>       `(("python-diff-match-patch"
>          ,python-diff-match-patch)
>         ("python-django" ,python-django)
>         ("python-tablib" ,python-tablib)))
>     (home-page
>       "https://github.com/django-import-export/django-import-export")
>     (synopsis
>       "Django application and library for importing and exporting
> data with included admin integration.")
>     (description
>       "Django application and library for importing and exporting
> data with included admin integration.")
>     (license license:bsd-3)))
> 
> --8<---------------cut here---------------end--------------->8---
> 
> Here, some of tablib’s dependencies are missing; the same problem
> seems
> to happen with python-django-import-export itself.
> 
> Are we failing to parse some of the PyPI metadata?
I don't think so, rather it appears as though the requires.txt is
somehow malformed or ill-suited for importing.  It basically marks all
inputs as optional.

Perhaps we could instruct the importer to emit comments for optional
inputs like so
;; Optional inputs by section
;; -- [SECTION]
;; ("python-some-package" ,python-some-package)
;; ("python-some-other-package" ,python-some-other-package) 
Maybe add a command line switch to toggle these comments or a way of
assuming some section, e.g. [cli] to be the thing we want to build for
a package.

Regards,
Leo

bug#49168: ‘guix import pypi’ misses package dependencies

[Ludovic Courtès]

Hi,

Leo Prikler <leo.prikler@student.tugraz.at> skribis:

>> Here, some of tablib’s dependencies are missing; the same problem
>> seems
>> to happen with python-django-import-export itself.
>> 
>> Are we failing to parse some of the PyPI metadata?
> I don't think so, rather it appears as though the requires.txt is
> somehow malformed or ill-suited for importing.  It basically marks all
> inputs as optional.

Oh.  So that’s first and foremost a bug of the PyPI package, right?

Actually
<https://files.pythonhosted.org/packages/fe/9d/4e15b2e74044ee051b6939c1b3ff716b0106e8f72d78eab8e08212eab44c/tablib-3.0.0.tar.gz>
does not have a ‘requirements.txt’ file, and
<https://files.pythonhosted.org/packages/16/85/078fc037b15aa1120d6a0287ec9d092d93d632ab01a0e7a3e69b4733da5e/tablib-3.0.0-py3-none-any.whl>
doesn’t have much metadata, so I don’t even get where were get that
info.

> Perhaps we could instruct the importer to emit comments for optional
> inputs like so
> ;; Optional inputs by section
> ;; -- [SECTION]
> ;; ("python-some-package" ,python-some-package)
> ;; ("python-some-other-package" ,python-some-other-package) 
> Maybe add a command line switch to toggle these comments or a way of
> assuming some section, e.g. [cli] to be the thing we want to build for
> a package.

A CLI switch to include optional dependencies would be nice.

As for emitting comments, the (not-so-) pretty printer written for ‘guix
style’¹ could come in handy for importers and more generally in any
place where we need to emit comments.

Thanks,
Ludo’.

¹ https://issues.guix.gnu.org/49169

bug#49168: ‘guix import pypi’ misses package dependencies

[Leo Prikler]

Am Dienstag, den 22.06.2021, 14:33 +0200 schrieb Ludovic Courtès:
> Hi,
> 
> Leo Prikler <leo.prikler@student.tugraz.at> skribis:
> 
> > > Here, some of tablib’s dependencies are missing; the same problem
> > > seems
> > > to happen with python-django-import-export itself.
> > > 
> > > Are we failing to parse some of the PyPI metadata?
> > I don't think so, rather it appears as though the requires.txt is
> > somehow malformed or ill-suited for importing.  It basically marks
> > all
> > inputs as optional.
> 
> Oh.  So that’s first and foremost a bug of the PyPI package, right?
> 
> Actually
> <
> https://files.pythonhosted.org/packages/fe/9d/4e15b2e74044ee051b6939c1b3ff716b0106e8f72d78eab8e08212eab44c/tablib-3.0.0.tar.gz
> >
> does not have a ‘requirements.txt’ file, and
> <
> https://files.pythonhosted.org/packages/16/85/078fc037b15aa1120d6a0287ec9d092d93d632ab01a0e7a3e69b4733da5e/tablib-3.0.0-py3-none-any.whl
> >
> doesn’t have much metadata, so I don’t even get where were get that
> info.
It does, but it's well hidden in the src tree.  I peeked into the guix
import code to find it.

> > Perhaps we could instruct the importer to emit comments for
> > optional
> > inputs like so
> > ;; Optional inputs by section
> > ;; -- [SECTION]
> > ;; ("python-some-package" ,python-some-package)
> > ;; ("python-some-other-package" ,python-some-other-package) 
> > Maybe add a command line switch to toggle these comments or a way
> > of
> > assuming some section, e.g. [cli] to be the thing we want to build
> > for
> > a package.
> 
> A CLI switch to include optional dependencies would be nice.
> 
> As for emitting comments, the (not-so-) pretty printer written for
> ‘guix
> style’¹ could come in handy for importers and more generally in any
> place where we need to emit comments.
Indeed, I wasn't thinking about that when I wrote this reply, but I see
what you mean.  Should we strive to have such a pretty-printer in Guile
as well or do we keep it to ourselves for now? :)

> Thanks,
> Ludo’.
> 
> ¹ https://issues.guix.gnu.org/49169

bug#49168: ‘guix import pypi’ misses package dependencies

[Maxim Cournoyer]

Hello,

Leo Prikler <leo.prikler@student.tugraz.at> writes:

> Am Dienstag, den 22.06.2021, 14:33 +0200 schrieb Ludovic Courtès:
>> Hi,
>> 
>> Leo Prikler <leo.prikler@student.tugraz.at> skribis:
>> 
>> > > Here, some of tablib’s dependencies are missing; the same problem
>> > > seems
>> > > to happen with python-django-import-export itself.
>> > > 
>> > > Are we failing to parse some of the PyPI metadata?
>> > I don't think so, rather it appears as though the requires.txt is
>> > somehow malformed or ill-suited for importing.  It basically marks
>> > all
>> > inputs as optional.
>> 
>> Oh.  So that’s first and foremost a bug of the PyPI package, right?

Sounds like so, if the inputs are required but marked as optional.

>> > Perhaps we could instruct the importer to emit comments for
>> > optional
>> > inputs like so
>> > ;; Optional inputs by section
>> > ;; -- [SECTION]
>> > ;; ("python-some-package" ,python-some-package)
>> > ;; ("python-some-other-package" ,python-some-other-package) 
>> > Maybe add a command line switch to toggle these comments or a way
>> > of
>> > assuming some section, e.g. [cli] to be the thing we want to build
>> > for
>> > a package.
>> 
>> A CLI switch to include optional dependencies would be nice.

+1.

>> As for emitting comments, the (not-so-) pretty printer written for
>> ‘guix
>> style’¹ could come in handy for importers and more generally in any
>> place where we need to emit comments.
> Indeed, I wasn't thinking about that when I wrote this reply, but I see
> what you mean.  Should we strive to have such a pretty-printer in Guile
> as well or do we keep it to ourselves for now? :)

This sounds good too, although I haven't read about that new fancy
printer yet.

Maxim

bug#49168: ‘guix import pypi’ misses package dependencies

[Ludovic Courtès]

Hi,

Leo Prikler <leo.prikler@student.tugraz.at> skribis:

> Am Dienstag, den 22.06.2021, 14:33 +0200 schrieb Ludovic Courtès:

[...]

>> Actually
>> <
>> https://files.pythonhosted.org/packages/fe/9d/4e15b2e74044ee051b6939c1b3ff716b0106e8f72d78eab8e08212eab44c/tablib-3.0.0.tar.gz
>> >
>> does not have a ‘requirements.txt’ file, and
>> <
>> https://files.pythonhosted.org/packages/16/85/078fc037b15aa1120d6a0287ec9d092d93d632ab01a0e7a3e69b4733da5e/tablib-3.0.0-py3-none-any.whl
>> >
>> doesn’t have much metadata, so I don’t even get where were get that
>> info.
> It does, but it's well hidden in the src tree.  I peeked into the guix
> import code to find it.

Indeed.  The tarball above has ‘tests/requirements.txt’:

--8<---------------cut here---------------start------------->8---
pytest
pytest-cov
MarkupPy
odfpy
openpyxl>=2.6.0
pandas
pyyaml
tabulate
xlrd
xlwt
--8<---------------cut here---------------end--------------->8---

There are no optional dependencies in that file, though.  Or were you
looking at something else?

>> As for emitting comments, the (not-so-) pretty printer written for
>> ‘guix
>> style’¹ could come in handy for importers and more generally in any
>> place where we need to emit comments.
> Indeed, I wasn't thinking about that when I wrote this reply, but I see
> what you mean.  Should we strive to have such a pretty-printer in Guile
> as well or do we keep it to ourselves for now? :)

Let’s nurture it in Guix and move it to Guile when it looks better.
Likewise, we should have support for comments directly in (ice-9 read).

Ludo’.

bug#49168: ‘guix import pypi’ misses package dependencies

[Leo Prikler]

Am Mittwoch, den 23.06.2021, 15:20 +0200 schrieb Ludovic Courtès:
> Hi,
> 
> Leo Prikler <leo.prikler@student.tugraz.at> skribis:
> 
> > Am Dienstag, den 22.06.2021, 14:33 +0200 schrieb Ludovic Courtès:
> 
> [...]
> 
> > > Actually
> > > <
> > > https://files.pythonhosted.org/packages/fe/9d/4e15b2e74044ee051b6939c1b3ff716b0106e8f72d78eab8e08212eab44c/tablib-3.0.0.tar.gz
> > > does not have a ‘requirements.txt’ file, and
> > > <
> > > https://files.pythonhosted.org/packages/16/85/078fc037b15aa1120d6a0287ec9d092d93d632ab01a0e7a3e69b4733da5e/tablib-3.0.0-py3-none-any.whl
> > > doesn’t have much metadata, so I don’t even get where were get
> > > that
> > > info.
> > It does, but it's well hidden in the src tree.  I peeked into the
> > guix
> > import code to find it.
> 
> Indeed.  The tarball above has ‘tests/requirements.txt’:
> 
> --8<---------------cut here---------------start------------->8---
> pytest
> pytest-cov
> MarkupPy
> odfpy
> openpyxl>=2.6.0
> pandas
> pyyaml
> tabulate
> xlrd
> xlwt
> --8<---------------cut here---------------end--------------->8---
> 
> There are no optional dependencies in that file, though.  Or were you
> looking at something else?
The importer and I are looking at something else:

--8<---------------cut here---------------start------------->8---
$ tar xfv tablib-3.0.0.tar.gz tablib-3.0.0/src/tablib.egg-
info/requires.txt | xargs cat

[all]
markuppy
odfpy
openpyxl>=2.6.0
pandas
pyyaml
tabulate
xlrd
xlwt

[cli]
tabulate

[html]
markuppy

[ods]
odfpy

[pandas]
pandas

[xls]
xlrd
xlwt

[xlsx]
openpyxl>=2.6.0

[yaml]
pyyaml
--8<---------------cut here---------------end--------------->8---


> > > As for emitting comments, the (not-so-) pretty printer written
> > > for
> > > ‘guix
> > > style’¹ could come in handy for importers and more generally in
> > > any
> > > place where we need to emit comments.
> > Indeed, I wasn't thinking about that when I wrote this reply, but I
> > see
> > what you mean.  Should we strive to have such a pretty-printer in
> > Guile
> > as well or do we keep it to ourselves for now? :)
> 
> Let’s nurture it in Guix and move it to Guile when it looks better.
> Likewise, we should have support for comments directly in (ice-9
> read).
Sounds good to me.

bug#49168: ‘guix import pypi’ misses package dependencies

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 2496 bytes --]

Hi,

Leo Prikler <leo.prikler@student.tugraz.at> skribis:

> Am Mittwoch, den 23.06.2021, 15:20 +0200 schrieb Ludovic Courtès:
>> Hi,
>> 
>> Leo Prikler <leo.prikler@student.tugraz.at> skribis:
>> 
>> > Am Dienstag, den 22.06.2021, 14:33 +0200 schrieb Ludovic Courtès:
>> 
>> [...]
>> 
>> > > Actually
>> > > <
>> > > https://files.pythonhosted.org/packages/fe/9d/4e15b2e74044ee051b6939c1b3ff716b0106e8f72d78eab8e08212eab44c/tablib-3.0.0.tar.gz
>> > > does not have a ‘requirements.txt’ file, and
>> > > <
>> > > https://files.pythonhosted.org/packages/16/85/078fc037b15aa1120d6a0287ec9d092d93d632ab01a0e7a3e69b4733da5e/tablib-3.0.0-py3-none-any.whl
>> > > doesn’t have much metadata, so I don’t even get where were get
>> > > that
>> > > info.
>> > It does, but it's well hidden in the src tree.  I peeked into the
>> > guix
>> > import code to find it.
>> 
>> Indeed.  The tarball above has ‘tests/requirements.txt’:
>> 
>> --8<---------------cut here---------------start------------->8---
>> pytest
>> pytest-cov
>> MarkupPy
>> odfpy
>> openpyxl>=2.6.0
>> pandas
>> pyyaml
>> tabulate
>> xlrd
>> xlwt
>> --8<---------------cut here---------------end--------------->8---
>> 
>> There are no optional dependencies in that file, though.  Or were you
>> looking at something else?
> The importer and I are looking at something else:
>
> $ tar xfv tablib-3.0.0.tar.gz tablib-3.0.0/src/tablib.egg-
> info/requires.txt | xargs cat
>
> [all]
> markuppy
> odfpy
> openpyxl>=2.6.0
> pandas
> pyyaml
> tabulate
> xlrd
> xlwt
>
> [cli]
> tabulate
>
> [html]
> markuppy
>
> [ods]
> odfpy
>
> [pandas]
> pandas
>
> [xls]
> xlrd
> xlwt
>
> [xlsx]
> openpyxl>=2.6.0
>
> [yaml]
> pyyaml

Oooh, I see.

So I came up with the following patch, which adds a flag for optional
dependencies, based on ‘requires.txt’.

Unfortunately, it has no effect for ‘guix import pypi tablib’ because we
only look at Wheel info in that case, as per:

  ;; First, try to compute the requirements using the wheel, else, fallback to
  ;; reading the "requires.txt" from the egg-info directory from the source
  ;; archive.
  (or (guess-requirements-from-wheel)
      (guess-requirements-from-source))

AFAICS, wheels don’t provide that info, do they?

Why does the importer favor .whl in the first place?  Is it supposed to
be more accurate or more widespread or something?

Thoughts?

Thanks,
Ludo’.


[-- Attachment #2: Type: text/x-patch, Size: 10848 bytes --]

diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm
index 6731d50891..be01cc99d2 100644
--- a/guix/import/pypi.scm
+++ b/guix/import/pypi.scm
@@ -1,7 +1,7 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2014 David Thompson <davet@gnu.org>
 ;;; Copyright © 2015 Cyril Roelandt <tipecaml@gmail.com>
-;;; Copyright © 2015, 2016, 2017, 2019, 2020 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2015, 2016, 2017, 2019, 2020, 2021 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2017 Mathieu Othacehe <m.othacehe@gmail.com>
 ;;; Copyright © 2018 Ricardo Wurmus <rekado@elephly.net>
 ;;; Copyright © 2019 Maxim Cournoyer <maxim.cournoyer@gmail.com>
@@ -232,10 +232,9 @@ the input field."
   "Given REQUIRES.TXT, a path to a Setuptools requires.txt file, return a list
 of lists of requirements.
 
-The first list contains the required dependencies while the second the
-optional test dependencies.  Note that currently, optional, non-test
-dependencies are omitted since these can be difficult or expensive to
-satisfy."
+The first list contains the required dependencies, the second contains test
+dependencies, and the third one contains optional dependencies.  Note that
+optional, non-test dependencies can be difficult or expensive to satisfy."
 
   (define (comment? line)
     ;; Return #t if the given LINE is a comment, #f otherwise.
@@ -249,6 +248,7 @@ satisfy."
     (lambda (port)
       (let loop ((required-deps '())
                  (test-deps '())
+                 (optional-deps '())
                  (inside-test-section? #f)
                  (optional? #f))
         (let ((line (read-line port)))
@@ -259,27 +259,33 @@ satisfy."
             ;; pytest >= 3 ; python_version >= "3.3"
             ;; pytest < 3 ; python_version < "3.3"
             (map (compose reverse delete-duplicates)
-                 (list required-deps test-deps)))
+                 (list required-deps test-deps optional-deps)))
            ((or (string-null? line) (comment? line))
-            (loop required-deps test-deps inside-test-section? optional?))
+            (loop required-deps test-deps optional-deps
+                  inside-test-section? optional?))
            ((section-header? line)
             ;; Encountering a section means that all the requirements
             ;; listed below are optional. Since we want to pick only the
             ;; test dependencies from the optional dependencies, we must
             ;; track those separately.
-            (loop required-deps test-deps (test-section? line) #t))
+            (loop required-deps test-deps optional-deps
+                  (test-section? line) #t))
            (inside-test-section?
             (loop required-deps
                   (cons (specification->requirement-name line)
                         test-deps)
+                  optional-deps
                   inside-test-section? optional?))
            ((not optional?)
             (loop (cons (specification->requirement-name line)
                         required-deps)
-                  test-deps inside-test-section? optional?))
+                  test-deps optional-deps
+                  inside-test-section? optional?))
            (optional?
-            ;; Skip optional items.
-            (loop required-deps test-deps inside-test-section? optional?))
+            (loop required-deps test-deps
+                  (cons (specification->requirement-name line)
+                        optional-deps)
+                  inside-test-section? optional?))
            (else
             (warning (G_ "parse-requires.txt reached an unexpected \
 condition on line ~a~%") line))))))))
@@ -314,7 +320,8 @@ returned value."
           (cond
            ((eof-object? line)
             (map (compose reverse delete-duplicates)
-                 (list required-deps test-deps)))
+                 (list required-deps test-deps
+                       '())))             ;XXX: no known optional dependencies
            ((and (requires-dist-header? line) (not (extra? line)))
             (loop (cons (specification->requirement-name
                          (requires-dist-value line))
@@ -392,7 +399,7 @@ cannot determine package dependencies from source archive: ~a~%")
   (or (guess-requirements-from-wheel)
       (guess-requirements-from-source)))
 
-(define (compute-inputs source-url wheel-url archive)
+(define* (compute-inputs source-url wheel-url archive)
   "Given the SOURCE-URL and WHEEL-URL of an already downloaded ARCHIVE, return
 a pair of lists, each consisting of a list of name/variable pairs, for the
 propagated inputs and the native inputs, respectively.  Also
@@ -419,17 +426,19 @@ return the unaltered list of upstream dependency names."
     (values (map process-requirements dependencies)
             (concatenate dependencies))))
 
-(define (make-pypi-sexp name version source-url wheel-url home-page synopsis
-                        description license)
+(define* (make-pypi-sexp name version source-url wheel-url home-page synopsis
+                         description license
+                         #:key optional-dependencies?)
   "Return the `package' s-expression for a python package with the given NAME,
-VERSION, SOURCE-URL, HOME-PAGE, SYNOPSIS, DESCRIPTION, and LICENSE."
+VERSION, SOURCE-URL, HOME-PAGE, SYNOPSIS, DESCRIPTION, and LICENSE.  When
+OPTIONAL-DEPENDENCIES? is true, include optional dependencies."
   (call-with-temporary-output-file
    (lambda (temp port)
      (and (url-fetch source-url temp)
           (receive (guix-dependencies upstream-dependencies)
               (compute-inputs source-url wheel-url temp)
             (match guix-dependencies
-              ((required-inputs native-inputs)
+              ((required-inputs native-inputs optional-inputs)
                (when (string-suffix? ".zip" source-url)
                  (set! native-inputs (cons
                                      '("unzip" ,unzip)
@@ -462,7 +471,12 @@ VERSION, SOURCE-URL, HOME-PAGE, SYNOPSIS, DESCRIPTION, and LICENSE."
                        (base32
                         ,(guix-hash-url temp)))))
                    (build-system python-build-system)
-                   ,@(maybe-inputs required-inputs 'propagated-inputs)
+                   ,@(maybe-inputs
+                      (append required-inputs
+                              (if optional-dependencies?
+                                  optional-inputs ;TODO: emit a comment
+                                  '()))
+                      'propagated-inputs)
                    ,@(maybe-inputs native-inputs 'native-inputs)
                    (home-page ,home-page)
                    (synopsis ,synopsis)
@@ -472,9 +486,10 @@ VERSION, SOURCE-URL, HOME-PAGE, SYNOPSIS, DESCRIPTION, and LICENSE."
 
 (define pypi->guix-package
   (memoize
-   (lambda* (package-name #:key repo version)
+   (lambda* (package-name #:key repo version optional-dependencies?)
      "Fetch the metadata for PACKAGE-NAME from pypi.org, and return the
-`package' s-expression corresponding to that package, or #f on failure."
+`package' s-expression corresponding to that package, or #f on failure.  When
+OPTIONAL-DEPENDENCIES? is true, include optional dependencies."
      (let* ((project (pypi-fetch package-name))
             (info    (and project (pypi-project-info project))))
        (and project
@@ -493,11 +508,17 @@ VERSION, SOURCE-URL, HOME-PAGE, SYNOPSIS, DESCRIPTION, and LICENSE."
                               (project-info-summary info)
                               (project-info-summary info)
                               (string->license
-                               (project-info-license info)))))))))
+                               (project-info-license info))
+                              #:optional-dependencies?
+                              optional-dependencies?)))))))
 
-(define (pypi-recursive-import package-name)
+(define* (pypi-recursive-import package-name #:key optional-dependencies?)
   (recursive-import package-name
-                    #:repo->guix-package pypi->guix-package
+                    #:repo->guix-package
+                    (lambda args
+                      (apply pypi->guix-package
+                             #:optional-dependencies? optional-dependencies?
+                             args))
                     #:guix-name python->package-name))
 
 (define (string->license str)
diff --git a/guix/scripts/import/pypi.scm b/guix/scripts/import/pypi.scm
index 33167174e2..58ae07f802 100644
--- a/guix/scripts/import/pypi.scm
+++ b/guix/scripts/import/pypi.scm
@@ -46,6 +46,8 @@ Import and convert the PyPI package for PACKAGE-NAME.\n"))
   (display (G_ "
   -r, --recursive        import packages recursively"))
   (display (G_ "
+  -O, --optional         include optional dependencies"))
+  (display (G_ "
   -V, --version          display version information and exit"))
   (newline)
   (show-bug-report-information))
@@ -62,6 +64,9 @@ Import and convert the PyPI package for PACKAGE-NAME.\n"))
          (option '(#\r "recursive") #f #f
                  (lambda (opt name arg result)
                    (alist-cons 'recursive #t result)))
+         (option '(#\O "optional") #f #f
+                 (lambda (opt name arg result)
+                   (alist-cons 'optional-dependencies? #t result)))
          %standard-import-options))
 
 \f
@@ -85,6 +90,9 @@ Import and convert the PyPI package for PACKAGE-NAME.\n"))
                              value)
                             (_ #f))
                            (reverse opts))))
+    (define optional-dependencies?
+      (assoc-ref opts 'optional-dependencies?))
+
     (match args
       ((package-name)
        (if (assoc-ref opts 'recursive)
@@ -94,9 +102,13 @@ Import and convert the PyPI package for PACKAGE-NAME.\n"))
                    `(define-public ,(string->symbol name)
                       ,pkg))
                   (_ #f))
-                (pypi-recursive-import package-name))
+                (pypi-recursive-import package-name
+                                       #:optional-dependencies?
+                                       optional-dependencies?))
            ;; Single import
-           (let ((sexp (pypi->guix-package package-name)))
+           (let ((sexp (pypi->guix-package package-name
+                                           #:optional-dependencies?
+                                           optional-dependencies?)))
              (unless sexp
                (leave (G_ "failed to download meta-data for package '~a'~%")
                       package-name))

bug#49168: ‘guix import pypi’ misses package dependencies

[Maxim Cournoyer]

Hi,

Ludovic Courtès <ludo@gnu.org> writes:

[...]

> So I came up with the following patch, which adds a flag for optional
> dependencies, based on ‘requires.txt’.
>
> Unfortunately, it has no effect for ‘guix import pypi tablib’ because we
> only look at Wheel info in that case, as per:
>
>   ;; First, try to compute the requirements using the wheel, else, fallback to
>   ;; reading the "requires.txt" from the egg-info directory from the source
>   ;; archive.
>   (or (guess-requirements-from-wheel)
>       (guess-requirements-from-source))
>
> AFAICS, wheels don’t provide that info, do they?
>
> Why does the importer favor .whl in the first place?  Is it supposed to
> be more accurate or more widespread or something?

Yes, the METADATA file from the binary wheel is a better place to look
than the source egg-info requires.txt file.  In my commit 01589acc5e1, I
simplified a comment that used used to read as:

-  ;; First, try to compute the requirements using the wheel, since that is the
-  ;; most reliable option. If a wheel is not provided for this package, try
-  ;; getting them by reading either the "requirements.txt" file or the
-  ;; "requires.txt" from the egg-info directory from the source tarball. Note
-  ;; that "requirements.txt" is not mandatory, so this is likely to fail.
+  ;; First, try to compute the requirements using the wheel, else, fallback to
+  ;; reading the "requires.txt" from the egg-info directory from the source
+  ;; tarball.

The wheel (.whl) binary format is well specified as PEP 427 [0] and is
what pip primarily uses for installing Python packages, making it a very
reliable source of metadata.  The Python egg is the predecessor of the
wheel, and can be considered obsolete, which explains why it's used as a
fallback.

Perhaps it'd be best to raise the issue to the package maintainers and
have them specify their metadata correctly?

Having the code you wrote to allow importing optional dependencies is
still a nice (optional) option to have though.  It was originally left
out based on comments from Ricardo that it wouldn't make a good default
due to raising the packaging effort.

HTH,

Maxim

[0]  https://www.python.org/dev/peps/pep-0427/

bug#49168: ‘guix import pypi’ misses package dependencies

[Ludovic Courtès]

Hello,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:

[...]

>> Why does the importer favor .whl in the first place?  Is it supposed to
>> be more accurate or more widespread or something?
>
> Yes, the METADATA file from the binary wheel is a better place to look
> than the source egg-info requires.txt file.  In my commit 01589acc5e1, I
> simplified a comment that used used to read as:
>
> -  ;; First, try to compute the requirements using the wheel, since that is the
> -  ;; most reliable option. If a wheel is not provided for this package, try
> -  ;; getting them by reading either the "requirements.txt" file or the
> -  ;; "requires.txt" from the egg-info directory from the source tarball. Note
> -  ;; that "requirements.txt" is not mandatory, so this is likely to fail.
> +  ;; First, try to compute the requirements using the wheel, else, fallback to
> +  ;; reading the "requires.txt" from the egg-info directory from the source
> +  ;; tarball.
>
> The wheel (.whl) binary format is well specified as PEP 427 [0] and is
> what pip primarily uses for installing Python packages, making it a very
> reliable source of metadata.  The Python egg is the predecessor of the
> wheel, and can be considered obsolete, which explains why it's used as a
> fallback.

Oh, I see.

> Perhaps it'd be best to raise the issue to the package maintainers and
> have them specify their metadata correctly?

Going back to the example at the beginning of this thread, what ‘guix
import pypi tablib’ produces is missing ‘python-setuptools-scm’.
Indeed, ‘METADATA’ doesn’t mention it.

Is it really a bug on their side, or is it something peculiar about Guix
packaging?  Perhaps ‘python-setuptools-scm’ should be provided more or
less by default?

> Having the code you wrote to allow importing optional dependencies is
> still a nice (optional) option to have though.  It was originally left
> out based on comments from Ricardo that it wouldn't make a good default
> due to raising the packaging effort.

OK.  Looking at <https://www.python.org/dev/peps/pep-0345/>, the
‘METADATA’ format apparently doesn’t support optional dependencies
anyway (which makes sense, because Wheels are a binary format), so
perhaps that idea was misguided.

(However ‘METADATA’ defines ‘Requires-External’, which the importer
could usefully interpret!)

WDYT?

Ludo’.

bug#49168: ‘guix import pypi’ misses package dependencies

[Leo Prikler]

Am Freitag, den 25.06.2021, 16:51 +0200 schrieb Ludovic Courtès:
> Hello,
> 
> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
> 
> > Ludovic Courtès <ludo@gnu.org> writes:
> 
> [...]
> 
> > > Why does the importer favor .whl in the first place?  Is it
> > > supposed to
> > > be more accurate or more widespread or something?
> > 
> > Yes, the METADATA file from the binary wheel is a better place to
> > look
> > than the source egg-info requires.txt file.  In my commit
> > 01589acc5e1, I
> > simplified a comment that used used to read as:
> > 
> > -  ;; First, try to compute the requirements using the wheel, since
> > that is the
> > -  ;; most reliable option. If a wheel is not provided for this
> > package, try
> > -  ;; getting them by reading either the "requirements.txt" file or
> > the
> > -  ;; "requires.txt" from the egg-info directory from the source
> > tarball. Note
> > -  ;; that "requirements.txt" is not mandatory, so this is likely
> > to fail.
> > +  ;; First, try to compute the requirements using the wheel, else,
> > fallback to
> > +  ;; reading the "requires.txt" from the egg-info directory from
> > the source
> > +  ;; tarball.
> > 
> > The wheel (.whl) binary format is well specified as PEP 427 [0] and
> > is
> > what pip primarily uses for installing Python packages, making it a
> > very
> > reliable source of metadata.  The Python egg is the predecessor of
> > the
> > wheel, and can be considered obsolete, which explains why it's used
> > as a
> > fallback.
> 
> Oh, I see.
> 
> > Perhaps it'd be best to raise the issue to the package maintainers
> > and
> > have them specify their metadata correctly?
> 
> Going back to the example at the beginning of this thread, what ‘guix
> import pypi tablib’ produces is missing ‘python-setuptools-scm’.
> Indeed, ‘METADATA’ doesn’t mention it.
> 
> Is it really a bug on their side, or is it something peculiar about
> Guix
> packaging?  Perhaps ‘python-setuptools-scm’ should be provided more
> or
> less by default?
Perhaps both?  It probably depends on how setuptools are used, but this
use appears internal to the mechanisms of setuptools itself and not
something, that requires.txt is concerned about.  In particular, I
think it is only relevant to the setup.py script and can probably be
detected by the presence of a certain form within it.

I'm not sure on the number of packages, that use this package vs. those
that don't, but I personally don't think we should make it an input in
every package.  It is probably already a native-input to those that
need it (or at least probably should be if it's propagated instead).

Of course we would also have to look at the METADATA of other packages
and check whether they actually mention python-setuptools-scm or are
also unaware of the dependency.

Regards,
Leo

bug#49168: ‘guix import pypi’ misses package dependencies

[Maxim Cournoyer]

Hello,

Ludovic Courtès <ludo@gnu.org> writes:

[...]

>> Perhaps it'd be best to raise the issue to the package maintainers and
>> have them specify their metadata correctly?
>
> Going back to the example at the beginning of this thread, what ‘guix
> import pypi tablib’ produces is missing ‘python-setuptools-scm’.
> Indeed, ‘METADATA’ doesn’t mention it.
>
> Is it really a bug on their side, or is it something peculiar about Guix
> packaging?  Perhaps ‘python-setuptools-scm’ should be provided more or
> less by default?

Hmm, sorry, I had lost track of what the original missing dependencies
were.  In this case (setuptools-scm), it is special in that it extends
the setuptools build system; it must be loaded early by setuptools, so
perhaps that explains why the dependency doesn't show as required in
METADATA.  I think going forward with the revised PEP 517 build system [0]
will alleviate such issues, by allowing to declare the build system and
its dependencies in a TOML file.

In the mean time, the statu quo seems the best option to me (manually
specify the setuptools extensions as native-inputs for the python
packages that need it).  We should look forward adjusting our importer
to the latest PEP 517 trends (along the PEP 517 build system patches
awaiting review for core-updates!), which are quickly becoming
ubiquitous in the Python world.

[0]  https://www.python.org/dev/peps/pep-0517/

HTH,

Maxim

bug#49168: ‘guix import pypi’ misses package dependencies

[Ludovic Courtès]

Hi Maxim,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> Hmm, sorry, I had lost track of what the original missing dependencies
> were.  In this case (setuptools-scm), it is special in that it extends
> the setuptools build system; it must be loaded early by setuptools, so
> perhaps that explains why the dependency doesn't show as required in
> METADATA.  I think going forward with the revised PEP 517 build system [0]
> will alleviate such issues, by allowing to declare the build system and
> its dependencies in a TOML file.
>
> In the mean time, the statu quo seems the best option to me (manually
> specify the setuptools extensions as native-inputs for the python
> packages that need it).  We should look forward adjusting our importer
> to the latest PEP 517 trends (along the PEP 517 build system patches
> awaiting review for core-updates!), which are quickly becoming
> ubiquitous in the Python world.
>
> [0]  https://www.python.org/dev/peps/pep-0517/

Alright, makes sense to me!

I’m not sure I have much to contribute to the review of the PEP 517
patches, being ignorant about these things, but I fully support the
effort.  :-)

In the meantime I guess we can close this bug?

Thanks,
Ludo’.

bug#49168: ‘guix import pypi’ misses package dependencies

[zimoun]

Hi,

On lun., 28 juin 2021 at 11:39, Ludovic Courtès <ludo@gnu.org> wrote:
> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

>> In the mean time, the statu quo seems the best option to me (manually
>> specify the setuptools extensions as native-inputs for the python
>> packages that need it).  We should look forward adjusting our importer
>> to the latest PEP 517 trends (along the PEP 517 build system patches
>> awaiting review for core-updates!), which are quickly becoming
>> ubiquitous in the Python world.

> In the meantime I guess we can close this bug?

Do we wait the merge of the branch 'wip-python-pep517' to close this
report?  Or can we close it now?


Cheers,
simon