bug#69997: Should ‘guix import pypi’ get dependencies from pyproject files?

bug#69997: Should ‘guix import pypi’ get dependencies from pyproject files?

[Ludovic Courtès]

Hello Python team!

Should ‘guix import pypi’ attempt to get dependency information from
‘pyproject.toml’, in addition to ‘requirements.txt’ and wheel ‘METADATA’
as it already does?

It might be more complicated than we’d like: in some cases, that file
seems to be used as a “trampoline” to Poetry.  For instance, in
python-pypugjs, the ‘requires’ bit delegates everything to Poetry:

--8<---------------cut here---------------start------------->8---
[tool.poetry]
name = "pypugjs"
version = "5.9.4"
description = ""
authors = ["Andy Grabow <andy@freilandkiwis.de>"]
license = "MIT"

[tool.poetry.dependencies]
python = "^3.8"
Jinja2 = "^3.1.1"
Mako = "^1.1.3"
tornado = "^6.0.4"
six = "^1.15.0"
coverage = "^6.3.2"
nose = "^1.3.7"
Flask = "^2.1.1"
charset-normalizer = "^2.1.0"
flake8 = "^4.0.1"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
--8<---------------cut here---------------end--------------->8---

Thoughts?

Ludo’.

bug#69997: Should ‘guix import pypi’ get dependencies from pyproject files?

[Sharlatan Hellseher]

[-- Attachment #1: Type: text/plain, Size: 408 bytes --]

Hi Lido!

> Should ‘guix import pypi’ attempt to get dependency
> information fromn‘pyproject.toml’, in addition to
> ‘requirements.txt’ and wheel ‘METADATA’ as it already does?

It's quite a common practice in modern Python just to include
 pyproject.toml, that fact makes importing long chains problematic.

It would be nice to have common yaml/toml parser for that task.

Oleg

[-- Attachment #2: Type: text/html, Size: 706 bytes --]

bug#69997: Should ‘guix import pypi’ get dependencies from pyproject files?

[Tanguy LE CARROUR]

Hi Ludo’,


Quoting Ludovic Courtès (2024-03-25 12:06:51)
> Should ‘guix import pypi’ attempt to get dependency information from
> ‘pyproject.toml’, in addition to ‘requirements.txt’ and wheel ‘METADATA’
> as it already does?
> 
> It might be more complicated than we’d like: in some cases, that file
> seems to be used as a “trampoline” to Poetry.  For instance, in
> python-pypugjs, the ‘requires’ bit delegates everything to Poetry:

Short answer: no! 😁

I’m pretty sure you know everything that I’m about to write, but better
say it out loud…

For a "standard modern" project managed with Poetry, the Python source
package contains `PKG-INFO` and `pyproject.toml ` that both contain
the run time dependencies. The wheel package only contains `METADATA` that
lists the dependencies. The source only contains a `pyproject.toml`.
To make the installed package as small as possible, tests files and
uncompiled assets are not (should not be) included.
From a Guix stand point, it’s better to build from source to be able to
run the test suite.

For the `python-pypugjs` you used as an example, we build from source,
so I guess the question does not arise. If we were to use the packages
available on PyPI, what I said above is *NOT* confirmed 😱:
- wheel (`.whl`) only contains `METADATA` with the dependencies; **BUT**
- source (`.tar.gz`) contains `PKG-INFO` (without dependency information),
  `pyproject.toml` (with dep’) and `setup.py` (also with dep’).

… "fun" fact, the information in `pyproject.toml` are **NOT** the same as
the one in `setup.py`!? 🤯 `pyproject.toml` says that `nose` is a run time
dependency (which it is not), but `setup.py` properly lists it in `tests_require`.

So, my answer would be: do not import from PyPI! Yes, I know, it’s radical! 😅
But if you have to, rely on the wheel’s `METADATA` file.

I hope this make sense. … I’m not really sure any more! 😅

Regards,

-- 
Tanguy

bug#69997: Should ‘guix import pypi’ get dependencies from pyproject files?

[Ludovic Courtès]

Hello!

Tanguy LE CARROUR <tanguy@bioneland.org> skribis:

> I’m pretty sure you know everything that I’m about to write, but better
> say it out loud…

Nope, I know nothing (I’ve just been told about using ‘pyproject.toml’
and it seemed to kinda make sense. :-))

> For a "standard modern" project managed with Poetry, the Python source
> package contains `PKG-INFO` and `pyproject.toml ` that both contain
> the run time dependencies. The wheel package only contains `METADATA` that
> lists the dependencies. The source only contains a `pyproject.toml`.
> To make the installed package as small as possible, tests files and
> uncompiled assets are not (should not be) included.
> From a Guix stand point, it’s better to build from source to be able to
> run the test suite.
>
> For the `python-pypugjs` you used as an example, we build from source,
> so I guess the question does not arise. If we were to use the packages
> available on PyPI, what I said above is *NOT* confirmed 😱:
> - wheel (`.whl`) only contains `METADATA` with the dependencies; **BUT**
> - source (`.tar.gz`) contains `PKG-INFO` (without dependency information),
>   `pyproject.toml` (with dep’) and `setup.py` (also with dep’).
>
> … "fun" fact, the information in `pyproject.toml` are **NOT** the same as
> the one in `setup.py`!? 🤯 `pyproject.toml` says that `nose` is a run time
> dependency (which it is not), but `setup.py` properly lists it in `tests_require`.

Oh my, such a mess.

> So, my answer would be: do not import from PyPI! Yes, I know, it’s radical! 😅
> But if you have to, rely on the wheel’s `METADATA` file.
>
> I hope this make sense. … I’m not really sure any more! 😅

It does!

But then I mean, we could offer, say, ‘guix import upstream https://…’,
and that thing could parse ‘setup.py’ or similar to produce a package
definition from that.  Maybe that’s what you had in mind: import
straight from upstream rather than via PyPI?

Thanks,
Ludo’.

bug#69997: Should ‘guix import pypi’ get dependencies from pyproject files?

[Tanguy LE CARROUR]

Hi,


Quoting Ludovic Courtès (2024-03-26 17:04:52)
> Tanguy LE CARROUR <tanguy@bioneland.org> skribis:
> > So, my answer would be: do not import from PyPI! Yes, I know, it’s radical! 😅
> > But if you have to, rely on the wheel’s `METADATA` file.
> >
> > I hope this make sense. … I’m not really sure any more! 😅
> 
> It does!
> 
> But then I mean, we could offer, say, ‘guix import upstream https://…’,
> and that thing could parse ‘setup.py’ or similar to produce a package
> definition from that.

I’m not against a good-old-generic-solution®, but this one might be
a bit… too generic. It contains no mention to Python, so the next logical step
would be `guix import URL`. Not that I would not like it, though! 😁

So I would say… let’s wait and see what the others think. In the
meantime, I’ll have to dive deeper in the PEP and the actual importer
code.


> Maybe that’s what you had in mind: import straight from upstream rather than via PyPI?

I actually had nothing in mind! I blacked out after sending this email.
Too many emotions at once, I guess! I had no clue it was such a mess. 😅

-- 
Tanguy

bug#69997: Should ‘guix import pypi’ get dependencies from pyproject files?

[Tanguy LE CARROUR]

Hi,

Quoting Tanguy LE CARROUR (2024-03-26 17:55:23)
> Quoting Ludovic Courtès (2024-03-26 17:04:52)
> > Tanguy LE CARROUR <tanguy@bioneland.org> skribis:
> > > So, my answer would be: do not import from PyPI! Yes, I know, it’s radical! 😅
> > > But if you have to, rely on the wheel’s `METADATA` file.
> > >
> > > I hope this make sense. … I’m not really sure any more! 😅
> > 
> > It does!
> > 
> > But then I mean, we could offer, say, ‘guix import upstream https://…’,
> > and that thing could parse ‘setup.py’ or similar to produce a package
> > definition from that.
> […]
> So I would say… let’s wait and see what the others think. In the
> meantime, I’ll have to dive deeper in the PEP and the actual importer
> code.

According to PEP 427 [1] a.k.a. Binary distribution format [2], if you
go for packaged/PyPI then we should go for `METADATA`.

[1]: https://peps.python.org/pep-0427/
[2]: https://packaging.python.org/en/latest/specifications/binary-distribution-format/#the-dist-info-directory

But, as stated earlier, we should build from source, to make sure we can
run the test suite. Active projects should slowly migrate to PEP 517 [3]
`pyproject.toml`. But, this is not a solution! 😱 This is actually yet
another problem! 😵

[3]: https://peps.python.org/pep-0517/

Each build system relies on it’s own file organization. For instance, Poetry
looks for a `[tool.poetry.dependencies]` section in the file. So the
importer should be "build system aware", which leads us to… `guix import poetry URL`!?
Not really generic any more! 😞

I guess we should sleep on it…

-- 
Tanguy

bug#69997: Should ‘guix import pypi’ get dependencies from pyproject files?

[Ludovic Courtès]

Hi,

Tanguy LE CARROUR <tanguy@bioneland.org> skribis:

> Quoting Ludovic Courtès (2024-03-26 17:04:52)

[...]

>> But then I mean, we could offer, say, ‘guix import upstream https://…’,
>> and that thing could parse ‘setup.py’ or similar to produce a package
>> definition from that.
>
> I’m not against a good-old-generic-solution®, but this one might be
> a bit… too generic. It contains no mention to Python, so the next logical step
> would be `guix import URL`. Not that I would not like it, though! 😁

Well, this has been on my mind for a long time.  Such a tool could
provide at least a useful package skeleton even for software using CMake
or Autotools.

> So I would say… let’s wait and see what the others think. In the
> meantime, I’ll have to dive deeper in the PEP and the actual importer
> code.

Looks like consensus among you Pythonistas has yet to be reached
regarding whether ‘pyproject.toml’ data would be a useful addition.  :-)

We’ll see!

Ludo’.

PS: I hear more and more long-time Python developers dismayed by the sad
    state of packaging and code evolution in Python.  In Guile land, we
    say: refugees welcome!  Come discover a great language and a great
    community (together with their own set of problems).

bug#69997: Should ‘guix import pypi’ get dependencies from pyproject files?

[Tanguy LE CARROUR]

Hi Ludo’,


Quoting Ludovic Courtès (2024-03-28 19:09:49)
> Tanguy LE CARROUR <tanguy@bioneland.org> skribis:
> 
> > Quoting Ludovic Courtès (2024-03-26 17:04:52)
> 
> [...]
> 
> >> But then I mean, we could offer, say, ‘guix import upstream https://…’,
> >> and that thing could parse ‘setup.py’ or similar to produce a package
> >> definition from that.
> >
> > I’m not against a good-old-generic-solution®, but this one might be
> > a bit… too generic. It contains no mention to Python, so the next logical step
> > would be `guix import URL`. Not that I would not like it, though! 😁
> 
> Well, this has been on my mind for a long time.  Such a tool could
> provide at least a useful package skeleton even for software using CMake
> or Autotools.
> 
> > So I would say… let’s wait and see what the others think. In the
> > meantime, I’ll have to dive deeper in the PEP and the actual importer
> > code.
> 
> Looks like consensus among you Pythonistas has yet to be reached
> regarding whether ‘pyproject.toml’ data would be a useful addition.  :-)

I have to admit that the Zen of Python [1] "There should be one-- and preferably only one --
obvious way to do it." has been recently difficult to follow packaging-wise!

[1]: https://peps.python.org/pep-0020

Even Poetry, the one I’ve been using for years, as made questionable
decisions [2] and I have to admit that I had a look at the competitors.

[2]: https://python-poetry.org/docs/faq/#why-does-poetry-not-adhere-to-semantic-versioning


> PS: I hear more and more long-time Python developers dismayed by the sad
>     state of packaging and code evolution in Python.  In Guile land, we
>     say: refugees welcome!  Come discover a great language and a great
>     community (together with their own set of problems).

Thanks for the kind invitation, but… I have an acute form of parens-itis.
Seeing more that a pair of parenthesis on a single line make my eyes bleed! 😅
And… `#` is for comments, every other use is complete heresy and those
who go against the creed should suffer! … isn’t that what the parentheses
are for?! 😉

Any way, thanks again for caring about Python!

Regards,

-- 
Tanguy

bug#69997: Should ‘guix import pypi’ get dependencies from pyproject files?

[Ludovic Courtès]

Tanguy LE CARROUR <tanguy@bioneland.org> skribis:

>> PS: I hear more and more long-time Python developers dismayed by the sad
>>     state of packaging and code evolution in Python.  In Guile land, we
>>     say: refugees welcome!  Come discover a great language and a great
>>     community (together with their own set of problems).
>
> Thanks for the kind invitation, but… I have an acute form of parens-itis.
> Seeing more that a pair of parenthesis on a single line make my eyes bleed! 😅
> And… `#` is for comments, every other use is complete heresy and those
> who go against the creed should suffer! … isn’t that what the parentheses
> are for?! 😉

Worry not! As part of our refugees-welcome effort, “we” have put
together tools and guides to help you feel at home:

  https://hg.sr.ht/~arnebab/wisp
  https://www.draketo.de/py2guile

:-)

Ludo’.

bug#69997: Should ‘guix import pypi’ get dependencies from pyproject files?

[Tanguy LE CARROUR]

Quoting Ludovic Courtès (2024-03-29 10:06:31)
> Tanguy LE CARROUR <tanguy@bioneland.org> skribis:
> 
> >> PS: I hear more and more long-time Python developers dismayed by the sad
> >>     state of packaging and code evolution in Python.  In Guile land, we
> >>     say: refugees welcome!  Come discover a great language and a great
> >>     community (together with their own set of problems).
> >
> > Thanks for the kind invitation, but… I have an acute form of parens-itis.
> > Seeing more that a pair of parenthesis on a single line make my eyes bleed! 😅
> > And… `#` is for comments, every other use is complete heresy and those
> > who go against the creed should suffer! … isn’t that what the parentheses
> > are for?! 😉
> 
> Worry not! As part of our refugees-welcome effort, “we” have put
> together tools and guides to help you feel at home:
> 
>   https://hg.sr.ht/~arnebab/wisp
>   https://www.draketo.de/py2guile

😵… 😱… 🤯!

-- 
Tanguy

bug#69997: Should ‘guix import pypi’ get dependencies from pyproject files?

[Lars-Dominik Braun]

Hey Ludo,

> Should ‘guix import pypi’ attempt to get dependency information from
> ‘pyproject.toml’, in addition to ‘requirements.txt’ and wheel ‘METADATA’
> as it already does?

yes it should. It’s the next logical step after having a
pyproject-build-system. The python-team branch (not sure whether Ricardo
had time to merge it yet) adds a TOML parser to Guix as a first step.

> It might be more complicated than we’d like: in some cases, that file
> seems to be used as a “trampoline” to Poetry.  For instance, in
> python-pypugjs, the ‘requires’ bit delegates everything to Poetry:

The file is always a trampoline to other build systems, like setuptools
or poetry. That’s (unfortunately) by design, see the [build-system]
section.

> [tool.poetry.dependencies]
> python = "^3.8"
> Jinja2 = "^3.1.1"
> Mako = "^1.1.3"
> tornado = "^6.0.4"
> six = "^1.15.0"
> coverage = "^6.3.2"
> nose = "^1.3.7"
> Flask = "^2.1.1"
> charset-normalizer = "^2.1.0"
> flake8 = "^4.0.1"

That’s unfortunate, because the
specification includes a common field for dependencies:
https://packaging.python.org/en/latest/specifications/pyproject-toml/#dependencies-optional-dependencies
I guess we’ll have to deal with these idiosyncracies as well :(

Lars

bug#69997: Should ‘guix import pypi’ get dependencies from pyproject files?

[Lars-Dominik Braun]

[-- Attachment #1: Type: text/plain, Size: 381 bytes --]

Hi,

> Should ‘guix import pypi’ attempt to get dependency information from
> ‘pyproject.toml’, in addition to ‘requirements.txt’ and wheel ‘METADATA’
> as it already does?

attached patches allow parsing the standardized pyproject.toml fields
for dependencies. This won’t work for poetry (we need a different
version parser for that), but it’s a start.

Lars


[-- Attachment #2: 0001-import-pypi-Support-extracting-dependencies-from-pyp.patch --]
[-- Type: text/plain, Size: 11994 bytes --]

From c2e7e07ad407613233edbb7ebfcc6f0c7c0bcc25 Mon Sep 17 00:00:00 2001
Message-ID: <c2e7e07ad407613233edbb7ebfcc6f0c7c0bcc25.1734278914.git.lars@6xq.net>
From: Lars-Dominik Braun <lars@6xq.net>
Date: Sun, 15 Dec 2024 13:22:00 +0100
Subject: [PATCH 1/4] import: pypi: Support extracting dependencies from
 pyproject.toml.

* guix/import/pypi.scm (guess-requirements): Support extracting dependencies from pyproject.toml.
* tests/pypi.scm: ("pypi->guix-package, no requires.txt, but wheel."):
Renamed from "pypi->guix-package, wheels", remove requires.txt file,
because the current implementation cannot detect invalid files.
("pypi->guix-package, no usable requirement file, no wheel."): Renamed
from "pypi->guix-package, no usable requirement file.".
(test-pyproject.toml): New variable.
("pypi->guix-package, no wheel, no requires.txt, but pyproject.toml"): New test.
("pypi->guix-package, no wheel, but requires.txt and pyproject.toml"): Ditto.

Change-Id: Ib525750eb6ff4139a8209420042b28ae3c850764
---
 guix/import/pypi.scm |  74 +++++++++++++++++++++++--------
 tests/pypi.scm       | 101 ++++++++++++++++++++++++++++++++++++++++---
 2 files changed, 152 insertions(+), 23 deletions(-)

diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm
index 7b9f54a200..7915d65d23 100644
--- a/guix/import/pypi.scm
+++ b/guix/import/pypi.scm
@@ -57,6 +57,7 @@ (define-module (guix import pypi)
   #:use-module (guix import utils)
   #:use-module (guix import json)
   #:use-module (json)
+  #:use-module (guix build toml)
   #:use-module (guix packages)
   #:use-module (guix upstream)
   #:use-module ((guix licenses) #:prefix license:)
@@ -386,7 +387,42 @@ (define (guess-requirements source-url wheel-url archive)
        (if wheel-url
            (and (url-fetch wheel-url temp)
                 (read-wheel-metadata temp))
-           #f))))
+           (list '() '())))))
+
+  (define (guess-requirements-from-pyproject.toml dir)
+    (let* ((pyproject.toml-files (find-files dir (lambda (abs-file-name _)
+                                          (string-match "/pyproject.toml$"
+                                          abs-file-name))))
+          (pyproject.toml (match pyproject.toml-files
+                            (()
+                              (warning (G_ "Cannot guess requirements from \
+pyproject.toml file, because it does not exist.~%"))
+                              '())
+                            (else (parse-toml-file (first pyproject.toml-files)))))
+          (pyproject-build-requirements
+           (or (recursive-assoc-ref pyproject.toml '("build-system" "requires")) '()))
+          (pyproject-dependencies
+           (or (recursive-assoc-ref pyproject.toml '("project" "dependencies")) '()))
+          ;; This is more of a convention, since optional-dependencies is a table of arbitrary values.
+          (pyproject-test-dependencies
+           (or (recursive-assoc-ref pyproject.toml '("project" "optional-dependencies" "test")) '())))
+      (if (null? pyproject.toml)
+        #f
+        (list (map specification->requirement-name pyproject-dependencies)
+              (map specification->requirement-name
+                   (append pyproject-build-requirements
+                           pyproject-test-dependencies))))))
+
+  (define (guess-requirements-from-requires.txt dir)
+    (let ((requires.txt-files (find-files dir (lambda (abs-file-name _)
+		                                          (string-match "\\.egg-info/requires.txt$"
+                                                  abs-file-name)))))
+     (match requires.txt-files
+       (()
+        (warning (G_ "Cannot guess requirements from source archive: \
+no requires.txt file found.~%"))
+        #f)
+       (else (parse-requires.txt (first requires.txt-files))))))
 
   (define (guess-requirements-from-source)
     ;; Return the package's requirements by guessing them from the source.
@@ -398,27 +434,29 @@ (define (guess-requirements source-url wheel-url archive)
              (if (string=? "zip" (file-extension source-url))
                  (invoke "unzip" archive "-d" dir)
                  (invoke "tar" "xf" archive "-C" dir)))
-           (let ((requires.txt-files
-                  (find-files dir (lambda (abs-file-name _)
-		                    (string-match "\\.egg-info/requires.txt$"
-                                                  abs-file-name)))))
-             (match requires.txt-files
-               (()
-                (warning (G_ "Cannot guess requirements from source archive:\
- no requires.txt file found.~%"))
-                (list '() '()))
-               (else (parse-requires.txt (first requires.txt-files)))))))
+               (list (guess-requirements-from-pyproject.toml dir)
+                     (guess-requirements-from-requires.txt dir))))
         (begin
           (warning (G_ "Unsupported archive format; \
 cannot determine package dependencies from source archive: ~a~%")
                    (basename source-url))
-          (list '() '()))))
-
-  ;; First, try to compute the requirements using the wheel, else, fallback to
-  ;; reading the "requires.txt" from the egg-info directory from the source
-  ;; archive.
-  (or (guess-requirements-from-wheel)
-      (guess-requirements-from-source)))
+          (list #f #f))))
+
+  (define (merge a b)
+    "Given lists A and B with two iteams each, combine A1 and B1, as well as A2 and B2."
+    (match (list a b)
+      (((first-propagated first-native) (second-propagated second-native))
+       (list (append first-propagated second-propagated) (append first-native second-native)))))
+
+  ;; requires.txt and the metadata of a wheel contain redundant information,
+  ;; so fetch only one of them, preferring requires.txt from the source
+  ;; distribution, which we always fetch, since the source tarball also
+  ;; contains pyproject.toml.
+  (match (guess-requirements-from-source)
+    ((from-pyproject.toml #f)
+      (merge (or from-pyproject.toml '(() ())) (or (guess-requirements-from-wheel) '(() ()))))
+    ((from-pyproject.toml from-requires.txt)
+      (merge (or from-pyproject.toml '(() ())) from-requires.txt))))
 
 (define (compute-inputs source-url wheel-url archive)
   "Given the SOURCE-URL and WHEEL-URL of an already downloaded ARCHIVE, return
diff --git a/tests/pypi.scm b/tests/pypi.scm
index c9aee34d8b..fe00e429b7 100644
--- a/tests/pypi.scm
+++ b/tests/pypi.scm
@@ -112,6 +112,20 @@ (define test-requires.txt-beaker "\
 coverage
 ")
 
+(define test-pyproject.toml "\
+[build-system]
+requires = [\"dummy-build-dep-a\", \"dummy-build-dep-b\"]
+
+[project]
+dependencies = [
+  \"dummy-dep-a\",
+  \"dummy-dep-b\",
+]
+
+[project.optional-dependencies]
+test = [\"dummy-test-dep-a\", \"dummy-test-dep-b\"]
+")
+
 (define test-metadata "\
 Classifier: Programming Language :: Python :: 3.7
 Requires-Dist: baz ~= 3
@@ -325,13 +339,90 @@ (define-syntax-rule (with-pypi responses body ...)
         (x
          (pk 'fail x #f))))))
 
+(test-assert "pypi->guix-package, no wheel, no requires.txt, but pyproject.toml"
+  (let ((tarball (pypi-tarball
+                  "foo-1.0.0"
+                  `(("pyproject.toml" ,test-pyproject.toml))))
+        (twice (lambda (lst) (append lst lst))))
+    (with-pypi (twice `(("/foo-1.0.0.tar.gz" 200 ,(file-dump tarball))
+                        ("/foo-1.0.0-py2.py3-none-any.whl" 404 "")
+                        ("/foo/json" 200 ,(lambda (port)
+                                            (display (foo-json) port)))))
+      ;; Not clearing the memoization cache here would mean returning the value
+      ;; computed in the previous test.
+      (invalidate-memoization! pypi->guix-package)
+      (match (pypi->guix-package "foo")
+        (`(package
+            (name "python-foo")
+            (version "1.0.0")
+            (source (origin
+                      (method url-fetch)
+                      (uri (pypi-uri "foo" version))
+                      (sha256
+                       (base32 ,(? string? hash)))))
+            (build-system pyproject-build-system)
+            (propagated-inputs (list python-dummy-dep-a python-dummy-dep-b))
+            (native-inputs (list python-dummy-build-dep-a python-dummy-build-dep-b
+                                 python-dummy-test-dep-a python-dummy-test-dep-b))
+            (home-page "http://example.com")
+            (synopsis "summary")
+            (description "summary.")
+            (license license:lgpl2.0))
+         (and (string=? default-sha256/base32 hash)
+              (equal? (pypi->guix-package "foo" #:version "1.0.0")
+                      (pypi->guix-package "foo"))
+              (guard (c ((error? c) #t))
+                (pypi->guix-package "foo" #:version "42"))))
+        (x
+         (pk 'fail x #f))))))
+
+(test-assert "pypi->guix-package, no wheel, but requires.txt and pyproject.toml"
+  (let ((tarball (pypi-tarball
+                  "foo-1.0.0"
+                  `(("foo-1.0.0/pyproject.toml" ,test-pyproject.toml)
+                    ("foo-1.0.0/bizarre.egg-info/requires.txt"
+                     ,test-requires.txt))))
+        (twice (lambda (lst) (append lst lst))))
+    (with-pypi (twice `(("/foo-1.0.0.tar.gz" 200 ,(file-dump tarball))
+                        ("/foo-1.0.0-py2.py3-none-any.whl" 404 "")
+                        ("/foo/json" 200 ,(lambda (port)
+                                            (display (foo-json) port)))))
+      ;; Not clearing the memoization cache here would mean returning the value
+      ;; computed in the previous test.
+      (invalidate-memoization! pypi->guix-package)
+      (match (pypi->guix-package "foo")
+        (`(package
+            (name "python-foo")
+            (version "1.0.0")
+            (source (origin
+                      (method url-fetch)
+                      (uri (pypi-uri "foo" version))
+                      (sha256
+                       (base32 ,(? string? hash)))))
+            (build-system pyproject-build-system)
+            ;; Information from requires.txt and pyproject.toml is combined.
+            (propagated-inputs (list python-bar python-dummy-dep-a python-dummy-dep-b
+                                     python-foo))
+            (native-inputs (list python-dummy-build-dep-a python-dummy-build-dep-b
+                                 python-dummy-test-dep-a python-dummy-test-dep-b
+                                 python-pytest))
+            (home-page "http://example.com")
+            (synopsis "summary")
+            (description "summary.")
+            (license license:lgpl2.0))
+         (and (string=? default-sha256/base32 hash)
+              (equal? (pypi->guix-package "foo" #:version "1.0.0")
+                      (pypi->guix-package "foo"))
+              (guard (c ((error? c) #t))
+                (pypi->guix-package "foo" #:version "42"))))
+        (x
+         (pk 'fail x #f))))))
+
 (test-skip (if (which "zip") 0 1))
-(test-assert "pypi->guix-package, wheels"
+(test-assert "pypi->guix-package, no requires.txt, but wheel."
   (let ((tarball (pypi-tarball
                   "foo-1.0.0"
-                  '(("foo-1.0.0/foo.egg-info/requires.txt"
-                     "wrong data \
-to make sure we're testing wheels"))))
+                  '(("foo-1.0.0/foo.egg-info/.empty" ""))))
         (wheel (wheel-file "foo-1.0.0"
                            `(("METADATA" ,test-metadata)))))
     (with-pypi `(("/foo-1.0.0.tar.gz" 200 ,(file-dump tarball))
@@ -362,7 +453,7 @@ (define-syntax-rule (with-pypi responses body ...)
         (x
          (pk 'fail x #f))))))
 
-(test-assert "pypi->guix-package, no usable requirement file."
+(test-assert "pypi->guix-package, no usable requirement file, no wheel."
   (let ((tarball (pypi-tarball "foo-1.0.0"
                                '(("foo.egg-info/.empty" "")))))
     (with-pypi `(("/foo-1.0.0.tar.gz" 200 ,(file-dump tarball))

base-commit: cfd4f56f75a20b6732d463180d211f796c9032e5
-- 
2.45.2


[-- Attachment #3: 0002-import-pypi-Add-python-wheel-to-native-inputs-if-set.patch --]
[-- Type: text/plain, Size: 1797 bytes --]

From 0abdb392bf10a99291114fc7e162a3845f25c696 Mon Sep 17 00:00:00 2001
Message-ID: <0abdb392bf10a99291114fc7e162a3845f25c696.1734278914.git.lars@6xq.net>
In-Reply-To: <c2e7e07ad407613233edbb7ebfcc6f0c7c0bcc25.1734278914.git.lars@6xq.net>
References: <c2e7e07ad407613233edbb7ebfcc6f0c7c0bcc25.1734278914.git.lars@6xq.net>
From: Lars-Dominik Braun <lars@6xq.net>
Date: Sun, 15 Dec 2024 13:30:59 +0100
Subject: [PATCH 2/4] import: pypi: Add python-wheel to native inputs if
 setuptools is used.

* guix/import/pypi.scm (compute-inputs): Add missing python-wheel if
necessary.

Change-Id: Iedad213a6684856e48349289c4d9beba953f396b
---
 guix/import/pypi.scm | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm
index 7915d65d23..52ec6e4ee6 100644
--- a/guix/import/pypi.scm
+++ b/guix/import/pypi.scm
@@ -470,12 +470,18 @@ (define (compute-inputs source-url wheel-url archive)
                          (type type))))
                 (sort deps string-ci<?)))
 
+  (define (add-missing-native-inputs inputs)
+    ;; setuptools cannot build wheels without the python-wheel.
+    (if (member "setuptools" inputs)
+      (cons "wheel" inputs)
+      inputs))
+
   ;; TODO: Record version number ranges in <upstream-input>.
   (let ((dependencies (guess-requirements source-url wheel-url archive)))
     (match dependencies
       ((propagated native)
        (append (requirements->upstream-inputs propagated 'propagated)
-               (requirements->upstream-inputs native 'native))))))
+               (requirements->upstream-inputs (add-missing-native-inputs native) 'native))))))
 
 (define* (pypi-package-inputs pypi-package #:optional version)
   "Return the list of <upstream-input> for PYPI-PACKAGE.  This procedure
-- 
2.45.2


[-- Attachment #4: 0003-import-pypi-Default-to-setuptools-as-build-system-in.patch --]
[-- Type: text/plain, Size: 2240 bytes --]

From 0c9708bf7b387f2100cdf375353982fbca9b364e Mon Sep 17 00:00:00 2001
Message-ID: <0c9708bf7b387f2100cdf375353982fbca9b364e.1734278914.git.lars@6xq.net>
In-Reply-To: <c2e7e07ad407613233edbb7ebfcc6f0c7c0bcc25.1734278914.git.lars@6xq.net>
References: <c2e7e07ad407613233edbb7ebfcc6f0c7c0bcc25.1734278914.git.lars@6xq.net>
From: Lars-Dominik Braun <lars@6xq.net>
Date: Sun, 15 Dec 2024 16:56:53 +0100
Subject: [PATCH 3/4] import: pypi: Default to setuptools as build system
 input.

* guix/import/pypi.scm (guess-requirements): Default to setuptools if
pyproject.toml does not exist.

Change-Id: I600bd0a44342847878e3a2a7041bd7e7c7d30769
---
 guix/import/pypi.scm | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm
index 52ec6e4ee6..bba7361307 100644
--- a/guix/import/pypi.scm
+++ b/guix/import/pypi.scm
@@ -448,15 +448,21 @@ (define (guess-requirements source-url wheel-url archive)
       (((first-propagated first-native) (second-propagated second-native))
        (list (append first-propagated second-propagated) (append first-native second-native)))))
 
+  (define default-pyproject.toml-dependencies
+    ;; If there is no pyproject.toml, we assume it’s an old-style setuptools-based project.
+    '(() ("setuptools")))
+
   ;; requires.txt and the metadata of a wheel contain redundant information,
   ;; so fetch only one of them, preferring requires.txt from the source
   ;; distribution, which we always fetch, since the source tarball also
   ;; contains pyproject.toml.
   (match (guess-requirements-from-source)
     ((from-pyproject.toml #f)
-      (merge (or from-pyproject.toml '(() ())) (or (guess-requirements-from-wheel) '(() ()))))
+      (merge (or from-pyproject.toml default-pyproject.toml-dependencies)
+             (or (guess-requirements-from-wheel) '(() ()))))
     ((from-pyproject.toml from-requires.txt)
-      (merge (or from-pyproject.toml '(() ())) from-requires.txt))))
+      (merge (or from-pyproject.toml default-pyproject.toml-dependencies)
+             from-requires.txt))))
 
 (define (compute-inputs source-url wheel-url archive)
   "Given the SOURCE-URL and WHEEL-URL of an already downloaded ARCHIVE, return
-- 
2.45.2


[-- Attachment #5: 0004-import-pypi-Move-deduplication-to-final-processing-s.patch --]
[-- Type: text/plain, Size: 2987 bytes --]

From 8ab434690c870deb95bfbf61adc60a6a38d084bb Mon Sep 17 00:00:00 2001
Message-ID: <8ab434690c870deb95bfbf61adc60a6a38d084bb.1734278914.git.lars@6xq.net>
In-Reply-To: <c2e7e07ad407613233edbb7ebfcc6f0c7c0bcc25.1734278914.git.lars@6xq.net>
References: <c2e7e07ad407613233edbb7ebfcc6f0c7c0bcc25.1734278914.git.lars@6xq.net>
From: Lars-Dominik Braun <lars@6xq.net>
Date: Sun, 15 Dec 2024 17:02:44 +0100
Subject: [PATCH 4/4] import: pypi: Move deduplication to final processing
 step.

* guix/import/pypi.scm (parse-requires.txt): Remove deduplication.
(parse-wheel-metadata): Remove deduplication.
(compute-inputs): Instead do it here on all the collected inputs.

Change-Id: I2504cc693e9bf2e4cc44fd37b5823904dbaaa925
---
 guix/import/pypi.scm | 16 ++++++----------
 1 file changed, 6 insertions(+), 10 deletions(-)

diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm
index bba7361307..530b7d6879 100644
--- a/guix/import/pypi.scm
+++ b/guix/import/pypi.scm
@@ -283,12 +283,7 @@ (define (parse-requires.txt requires.txt)
         (let ((line (read-line port)))
           (cond
            ((eof-object? line)
-            ;; Duplicates can occur, since the same requirement can be
-            ;; listed multiple times with different conditional markers, e.g.
-            ;; pytest >= 3 ; python_version >= "3.3"
-            ;; pytest < 3 ; python_version < "3.3"
-            (map (compose reverse delete-duplicates)
-                 (list required-deps test-deps)))
+            (list required-deps test-deps))
            ((or (string-null? line) (comment? line))
             (loop required-deps test-deps inside-test-section? optional?))
            ((section-header? line)
@@ -342,8 +337,7 @@ (define (parse-wheel-metadata metadata)
         (let ((line (read-line port)))
           (cond
            ((eof-object? line)
-            (map (compose reverse delete-duplicates)
-                 (list required-deps test-deps)))
+            (list required-deps test-deps))
            ((and (requires-dist-header? line) (not (extra? line)))
             (loop (cons (specification->requirement-name
                          (requires-dist-value line))
@@ -486,8 +480,10 @@ (define (compute-inputs source-url wheel-url archive)
   (let ((dependencies (guess-requirements source-url wheel-url archive)))
     (match dependencies
       ((propagated native)
-       (append (requirements->upstream-inputs propagated 'propagated)
-               (requirements->upstream-inputs (add-missing-native-inputs native) 'native))))))
+       (append (requirements->upstream-inputs (delete-duplicates propagated)
+                                              'propagated)
+               (requirements->upstream-inputs (delete-duplicates (add-missing-native-inputs native))
+                                              'native))))))
 
 (define* (pypi-package-inputs pypi-package #:optional version)
   "Return the list of <upstream-input> for PYPI-PACKAGE.  This procedure
-- 
2.45.2

bug#69997: Should ‘guix import pypi’ get dependencies from pyproject files?

[Ludovic Courtès]

Hello,

Lars-Dominik Braun <lars@6xq.net> skribis:

> attached patches allow parsing the standardized pyproject.toml fields
> for dependencies. This won’t work for poetry (we need a different
> version parser for that), but it’s a start.

Excellent!  The code LGTM (I can’t really tell about the Python side of
things).

> +               (list (guess-requirements-from-pyproject.toml dir)
> +                     (guess-requirements-from-requires.txt dir))))

It looks like we still take info from ‘requirements.txt’; is
‘pyproject.toml’ insufficient?

Anyhow, if others on the Python team approve, I guess you can go ahead
and push it.

Thanks,
Ludo’.

bug#69997: Should ‘guix import pypi’ get dependencies from pyproject files?

[Sharlatan Hellseher]

Hi Ludovic,

> It looks like we still take info from ‘requirements.txt’; is
Python packaging ecosystem is not standradisied like in Julia
somteimes you may see all at once setup.cfg, setup.py, pyproject.toml,
 requirements*.txt (covering in separate files dev/install deps),

Poetry changed the game as it become popular but it does not follow
PEP in pyproject.toml and just
created it's own config field in it not copatible with pip.

This patch would help to refresh easely most of the astronomy packages
as they are on
pyproject.toml and failed to be refresh with guix refresh CLI.

Lars, do you have a commit access?

> Anyhow, if others on the Python team approve, I guess you can go ahead
and push it.

I vote for merge as well

Thanks,
Oleg

bug#69997: Should ‘guix import pypi’ get dependencies from pyproject files?

[Lars-Dominik Braun]

Hi,

> It looks like we still take info from ‘requirements.txt’; is
> ‘pyproject.toml’ insufficient?

correct. As a first quality of life improvement I would keep the other
information sources, but as we improve the importer further (by adding
poetry support for example), we need to fall back on them less often
I hope.

> Lars, do you have a commit access?

I have, but please go ahead and merge it into master. My available time
slots to work on Guix are quite limited, so merging my patches – even
though I have commit access – speeds things up.

Lars

bug#69997: Should ‘guix import pypi’ get dependencies from pyproject files?

[Sharlatan Hellseher]

[-- Attachment #1: Type: text/plain, Size: 137 bytes --]

Hi,

As it's pypi importer which improving refresh as well, may
it be pushed to master or we need it on team branch first?

Thanks,
Oleg

[-- Attachment #2: Type: text/html, Size: 300 bytes --]

bug#69997: Should ‘guix import pypi’ get dependencies from pyproject files?

[Sharlatan Hellseher]

[-- Attachment #1: Type: text/plain, Size: 361 bytes --]


Hi,

Pushed as

a3ffb920f1 * master import: pypi: Move deduplication to final processing step.
c904350a81 * import: pypi: Default to setuptools as build system input.
d7890af335 * import: pypi: Add python-wheel to native inputs if setuptools is used.
8bb3bb19c2 * import: pypi: Support extracting dependencies from pyproject.toml.

to master.

--
Thanks,
Oleg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]