bug#69997: Should ‘guix import pypi’ get dependencies from pyproject files?

bug#69997: Should ‘guix import pypi’ get dependencies from pyproject files?

[Ludovic Courtès]

Hello Python team!

Should ‘guix import pypi’ attempt to get dependency information from
‘pyproject.toml’, in addition to ‘requirements.txt’ and wheel ‘METADATA’
as it already does?

It might be more complicated than we’d like: in some cases, that file
seems to be used as a “trampoline” to Poetry.  For instance, in
python-pypugjs, the ‘requires’ bit delegates everything to Poetry:

--8<---------------cut here---------------start------------->8---
[tool.poetry]
name = "pypugjs"
version = "5.9.4"
description = ""
authors = ["Andy Grabow <andy@freilandkiwis.de>"]
license = "MIT"

[tool.poetry.dependencies]
python = "^3.8"
Jinja2 = "^3.1.1"
Mako = "^1.1.3"
tornado = "^6.0.4"
six = "^1.15.0"
coverage = "^6.3.2"
nose = "^1.3.7"
Flask = "^2.1.1"
charset-normalizer = "^2.1.0"
flake8 = "^4.0.1"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
--8<---------------cut here---------------end--------------->8---

Thoughts?

Ludo’.

bug#69997: Should ‘guix import pypi’ get dependencies from pyproject files?

[Sharlatan Hellseher]

[-- Attachment #1: Type: text/plain, Size: 408 bytes --]

Hi Lido!

> Should ‘guix import pypi’ attempt to get dependency
> information fromn‘pyproject.toml’, in addition to
> ‘requirements.txt’ and wheel ‘METADATA’ as it already does?

It's quite a common practice in modern Python just to include
 pyproject.toml, that fact makes importing long chains problematic.

It would be nice to have common yaml/toml parser for that task.

Oleg

[-- Attachment #2: Type: text/html, Size: 706 bytes --]

bug#69997: Should ‘guix import pypi’ get dependencies from pyproject files?

[Tanguy LE CARROUR]

Hi Ludo’,


Quoting Ludovic Courtès (2024-03-25 12:06:51)
> Should ‘guix import pypi’ attempt to get dependency information from
> ‘pyproject.toml’, in addition to ‘requirements.txt’ and wheel ‘METADATA’
> as it already does?
> 
> It might be more complicated than we’d like: in some cases, that file
> seems to be used as a “trampoline” to Poetry.  For instance, in
> python-pypugjs, the ‘requires’ bit delegates everything to Poetry:

Short answer: no! 😁

I’m pretty sure you know everything that I’m about to write, but better
say it out loud…

For a "standard modern" project managed with Poetry, the Python source
package contains `PKG-INFO` and `pyproject.toml ` that both contain
the run time dependencies. The wheel package only contains `METADATA` that
lists the dependencies. The source only contains a `pyproject.toml`.
To make the installed package as small as possible, tests files and
uncompiled assets are not (should not be) included.
From a Guix stand point, it’s better to build from source to be able to
run the test suite.

For the `python-pypugjs` you used as an example, we build from source,
so I guess the question does not arise. If we were to use the packages
available on PyPI, what I said above is *NOT* confirmed 😱:
- wheel (`.whl`) only contains `METADATA` with the dependencies; **BUT**
- source (`.tar.gz`) contains `PKG-INFO` (without dependency information),
  `pyproject.toml` (with dep’) and `setup.py` (also with dep’).

… "fun" fact, the information in `pyproject.toml` are **NOT** the same as
the one in `setup.py`!? 🤯 `pyproject.toml` says that `nose` is a run time
dependency (which it is not), but `setup.py` properly lists it in `tests_require`.

So, my answer would be: do not import from PyPI! Yes, I know, it’s radical! 😅
But if you have to, rely on the wheel’s `METADATA` file.

I hope this make sense. … I’m not really sure any more! 😅

Regards,

-- 
Tanguy

bug#69997: Should ‘guix import pypi’ get dependencies from pyproject files?

[Ludovic Courtès]

Hello!

Tanguy LE CARROUR <tanguy@bioneland.org> skribis:

> I’m pretty sure you know everything that I’m about to write, but better
> say it out loud…

Nope, I know nothing (I’ve just been told about using ‘pyproject.toml’
and it seemed to kinda make sense. :-))

> For a "standard modern" project managed with Poetry, the Python source
> package contains `PKG-INFO` and `pyproject.toml ` that both contain
> the run time dependencies. The wheel package only contains `METADATA` that
> lists the dependencies. The source only contains a `pyproject.toml`.
> To make the installed package as small as possible, tests files and
> uncompiled assets are not (should not be) included.
> From a Guix stand point, it’s better to build from source to be able to
> run the test suite.
>
> For the `python-pypugjs` you used as an example, we build from source,
> so I guess the question does not arise. If we were to use the packages
> available on PyPI, what I said above is *NOT* confirmed 😱:
> - wheel (`.whl`) only contains `METADATA` with the dependencies; **BUT**
> - source (`.tar.gz`) contains `PKG-INFO` (without dependency information),
>   `pyproject.toml` (with dep’) and `setup.py` (also with dep’).
>
> … "fun" fact, the information in `pyproject.toml` are **NOT** the same as
> the one in `setup.py`!? 🤯 `pyproject.toml` says that `nose` is a run time
> dependency (which it is not), but `setup.py` properly lists it in `tests_require`.

Oh my, such a mess.

> So, my answer would be: do not import from PyPI! Yes, I know, it’s radical! 😅
> But if you have to, rely on the wheel’s `METADATA` file.
>
> I hope this make sense. … I’m not really sure any more! 😅

It does!

But then I mean, we could offer, say, ‘guix import upstream https://…’,
and that thing could parse ‘setup.py’ or similar to produce a package
definition from that.  Maybe that’s what you had in mind: import
straight from upstream rather than via PyPI?

Thanks,
Ludo’.

bug#69997: Should ‘guix import pypi’ get dependencies from pyproject files?

[Tanguy LE CARROUR]

Hi,


Quoting Ludovic Courtès (2024-03-26 17:04:52)
> Tanguy LE CARROUR <tanguy@bioneland.org> skribis:
> > So, my answer would be: do not import from PyPI! Yes, I know, it’s radical! 😅
> > But if you have to, rely on the wheel’s `METADATA` file.
> >
> > I hope this make sense. … I’m not really sure any more! 😅
> 
> It does!
> 
> But then I mean, we could offer, say, ‘guix import upstream https://…’,
> and that thing could parse ‘setup.py’ or similar to produce a package
> definition from that.

I’m not against a good-old-generic-solution®, but this one might be
a bit… too generic. It contains no mention to Python, so the next logical step
would be `guix import URL`. Not that I would not like it, though! 😁

So I would say… let’s wait and see what the others think. In the
meantime, I’ll have to dive deeper in the PEP and the actual importer
code.


> Maybe that’s what you had in mind: import straight from upstream rather than via PyPI?

I actually had nothing in mind! I blacked out after sending this email.
Too many emotions at once, I guess! I had no clue it was such a mess. 😅

-- 
Tanguy

bug#69997: Should ‘guix import pypi’ get dependencies from pyproject files?

[Tanguy LE CARROUR]

Hi,

Quoting Tanguy LE CARROUR (2024-03-26 17:55:23)
> Quoting Ludovic Courtès (2024-03-26 17:04:52)
> > Tanguy LE CARROUR <tanguy@bioneland.org> skribis:
> > > So, my answer would be: do not import from PyPI! Yes, I know, it’s radical! 😅
> > > But if you have to, rely on the wheel’s `METADATA` file.
> > >
> > > I hope this make sense. … I’m not really sure any more! 😅
> > 
> > It does!
> > 
> > But then I mean, we could offer, say, ‘guix import upstream https://…’,
> > and that thing could parse ‘setup.py’ or similar to produce a package
> > definition from that.
> […]
> So I would say… let’s wait and see what the others think. In the
> meantime, I’ll have to dive deeper in the PEP and the actual importer
> code.

According to PEP 427 [1] a.k.a. Binary distribution format [2], if you
go for packaged/PyPI then we should go for `METADATA`.

[1]: https://peps.python.org/pep-0427/
[2]: https://packaging.python.org/en/latest/specifications/binary-distribution-format/#the-dist-info-directory

But, as stated earlier, we should build from source, to make sure we can
run the test suite. Active projects should slowly migrate to PEP 517 [3]
`pyproject.toml`. But, this is not a solution! 😱 This is actually yet
another problem! 😵

[3]: https://peps.python.org/pep-0517/

Each build system relies on it’s own file organization. For instance, Poetry
looks for a `[tool.poetry.dependencies]` section in the file. So the
importer should be "build system aware", which leads us to… `guix import poetry URL`!?
Not really generic any more! 😞

I guess we should sleep on it…

-- 
Tanguy

bug#69997: Should ‘guix import pypi’ get dependencies from pyproject files?

[Ludovic Courtès]

Hi,

Tanguy LE CARROUR <tanguy@bioneland.org> skribis:

> Quoting Ludovic Courtès (2024-03-26 17:04:52)

[...]

>> But then I mean, we could offer, say, ‘guix import upstream https://…’,
>> and that thing could parse ‘setup.py’ or similar to produce a package
>> definition from that.
>
> I’m not against a good-old-generic-solution®, but this one might be
> a bit… too generic. It contains no mention to Python, so the next logical step
> would be `guix import URL`. Not that I would not like it, though! 😁

Well, this has been on my mind for a long time.  Such a tool could
provide at least a useful package skeleton even for software using CMake
or Autotools.

> So I would say… let’s wait and see what the others think. In the
> meantime, I’ll have to dive deeper in the PEP and the actual importer
> code.

Looks like consensus among you Pythonistas has yet to be reached
regarding whether ‘pyproject.toml’ data would be a useful addition.  :-)

We’ll see!

Ludo’.

PS: I hear more and more long-time Python developers dismayed by the sad
    state of packaging and code evolution in Python.  In Guile land, we
    say: refugees welcome!  Come discover a great language and a great
    community (together with their own set of problems).

bug#69997: Should ‘guix import pypi’ get dependencies from pyproject files?

[Tanguy LE CARROUR]

Hi Ludo’,


Quoting Ludovic Courtès (2024-03-28 19:09:49)
> Tanguy LE CARROUR <tanguy@bioneland.org> skribis:
> 
> > Quoting Ludovic Courtès (2024-03-26 17:04:52)
> 
> [...]
> 
> >> But then I mean, we could offer, say, ‘guix import upstream https://…’,
> >> and that thing could parse ‘setup.py’ or similar to produce a package
> >> definition from that.
> >
> > I’m not against a good-old-generic-solution®, but this one might be
> > a bit… too generic. It contains no mention to Python, so the next logical step
> > would be `guix import URL`. Not that I would not like it, though! 😁
> 
> Well, this has been on my mind for a long time.  Such a tool could
> provide at least a useful package skeleton even for software using CMake
> or Autotools.
> 
> > So I would say… let’s wait and see what the others think. In the
> > meantime, I’ll have to dive deeper in the PEP and the actual importer
> > code.
> 
> Looks like consensus among you Pythonistas has yet to be reached
> regarding whether ‘pyproject.toml’ data would be a useful addition.  :-)

I have to admit that the Zen of Python [1] "There should be one-- and preferably only one --
obvious way to do it." has been recently difficult to follow packaging-wise!

[1]: https://peps.python.org/pep-0020

Even Poetry, the one I’ve been using for years, as made questionable
decisions [2] and I have to admit that I had a look at the competitors.

[2]: https://python-poetry.org/docs/faq/#why-does-poetry-not-adhere-to-semantic-versioning


> PS: I hear more and more long-time Python developers dismayed by the sad
>     state of packaging and code evolution in Python.  In Guile land, we
>     say: refugees welcome!  Come discover a great language and a great
>     community (together with their own set of problems).

Thanks for the kind invitation, but… I have an acute form of parens-itis.
Seeing more that a pair of parenthesis on a single line make my eyes bleed! 😅
And… `#` is for comments, every other use is complete heresy and those
who go against the creed should suffer! … isn’t that what the parentheses
are for?! 😉

Any way, thanks again for caring about Python!

Regards,

-- 
Tanguy

bug#69997: Should ‘guix import pypi’ get dependencies from pyproject files?

[Ludovic Courtès]

Tanguy LE CARROUR <tanguy@bioneland.org> skribis:

>> PS: I hear more and more long-time Python developers dismayed by the sad
>>     state of packaging and code evolution in Python.  In Guile land, we
>>     say: refugees welcome!  Come discover a great language and a great
>>     community (together with their own set of problems).
>
> Thanks for the kind invitation, but… I have an acute form of parens-itis.
> Seeing more that a pair of parenthesis on a single line make my eyes bleed! 😅
> And… `#` is for comments, every other use is complete heresy and those
> who go against the creed should suffer! … isn’t that what the parentheses
> are for?! 😉

Worry not! As part of our refugees-welcome effort, “we” have put
together tools and guides to help you feel at home:

  https://hg.sr.ht/~arnebab/wisp
  https://www.draketo.de/py2guile

:-)

Ludo’.

bug#69997: Should ‘guix import pypi’ get dependencies from pyproject files?

[Tanguy LE CARROUR]

Quoting Ludovic Courtès (2024-03-29 10:06:31)
> Tanguy LE CARROUR <tanguy@bioneland.org> skribis:
> 
> >> PS: I hear more and more long-time Python developers dismayed by the sad
> >>     state of packaging and code evolution in Python.  In Guile land, we
> >>     say: refugees welcome!  Come discover a great language and a great
> >>     community (together with their own set of problems).
> >
> > Thanks for the kind invitation, but… I have an acute form of parens-itis.
> > Seeing more that a pair of parenthesis on a single line make my eyes bleed! 😅
> > And… `#` is for comments, every other use is complete heresy and those
> > who go against the creed should suffer! … isn’t that what the parentheses
> > are for?! 😉
> 
> Worry not! As part of our refugees-welcome effort, “we” have put
> together tools and guides to help you feel at home:
> 
>   https://hg.sr.ht/~arnebab/wisp
>   https://www.draketo.de/py2guile

😵… 😱… 🤯!

-- 
Tanguy

bug#69997: Should ‘guix import pypi’ get dependencies from pyproject files?

[Lars-Dominik Braun]

Hey Ludo,

> Should ‘guix import pypi’ attempt to get dependency information from
> ‘pyproject.toml’, in addition to ‘requirements.txt’ and wheel ‘METADATA’
> as it already does?

yes it should. It’s the next logical step after having a
pyproject-build-system. The python-team branch (not sure whether Ricardo
had time to merge it yet) adds a TOML parser to Guix as a first step.

> It might be more complicated than we’d like: in some cases, that file
> seems to be used as a “trampoline” to Poetry.  For instance, in
> python-pypugjs, the ‘requires’ bit delegates everything to Poetry:

The file is always a trampoline to other build systems, like setuptools
or poetry. That’s (unfortunately) by design, see the [build-system]
section.

> [tool.poetry.dependencies]
> python = "^3.8"
> Jinja2 = "^3.1.1"
> Mako = "^1.1.3"
> tornado = "^6.0.4"
> six = "^1.15.0"
> coverage = "^6.3.2"
> nose = "^1.3.7"
> Flask = "^2.1.1"
> charset-normalizer = "^2.1.0"
> flake8 = "^4.0.1"

That’s unfortunate, because the
specification includes a common field for dependencies:
https://packaging.python.org/en/latest/specifications/pyproject-toml/#dependencies-optional-dependencies
I guess we’ll have to deal with these idiosyncracies as well :(

Lars