Packaging Python projects managed with Poetry

Packaging Python projects managed with Poetry

[Tanguy Le Carrour]

Hi Guix,

I've been happily working with Poetry to manage my Python projects, but
now, for the first time, I would like to package one of those projects
for Guix.

The Python packages I build do not contain any tests or specs, because
to me, they don't belong there. But, I need those tests to make sure
that my package works with the versions of the dependencies available on
Guix.

The problem is that the source code that I fetch from the git repository
contains the test, but does not contain a `setup.py` file –because Poetry
does not use it!—, and the `python-build-system` fails.

I haven't wrap my head around this yet and I'm not sure what would be
the proper way to do it? Write a `python-poetry-build-system`? I hope not!
Just put the d**n tests in the Python package? This would look like a
failure to me! :-(

Any thought, help, guidance welcome! Thanks! :-)

-- 
Tanguy

Re: Packaging Python projects managed with Poetry

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1326 bytes --]


Tanguy Le Carrour <tanguy@bioneland.org> writes:

> I've been happily working with Poetry to manage my Python projects, but
> now, for the first time, I would like to package one of those projects
> for Guix.
>
> The Python packages I build do not contain any tests or specs, because
> to me, they don't belong there. But, I need those tests to make sure
> that my package works with the versions of the dependencies available on
> Guix.
>
> The problem is that the source code that I fetch from the git repository
> contains the test, but does not contain a `setup.py` file –because Poetry
> does not use it!—, and the `python-build-system` fails.
>
> I haven't wrap my head around this yet and I'm not sure what would be
> the proper way to do it? Write a `python-poetry-build-system`? I hope not!
> Just put the d**n tests in the Python package? This would look like a
> failure to me! :-(
>
> Any thought, help, guidance welcome! Thanks! :-)

My first thought, is what would it require for the existing
python-build-system to detect and support building the things you
describe?

I haven't used Poetry myself, have you got a project that can be used as
an example?

It looks like the python-build-system already has some functionality
that's dependent on the use-setuptools? argument.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Packaging Python projects managed with Poetry

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 1433 bytes --]

Hi,

On Thu, 22 Oct 2020 17:15:20 +0200
Tanguy Le Carrour <tanguy@bioneland.org> wrote:

> does not contain a `setup.py` file –because Poetry does not use it!—, and
>the `python-build-system` fails.
> I haven't wrap my head around this yet and I'm not sure what would be
> the proper way to do it?

>Write a `python-poetry-build-system`? I hope not!

Why not?

According to https://github.com/python-poetry/poetry they took inspiration
from existing build systems like cargo, and they just replaced setup.py by
pyproject.toml.

So what you could do is create a poetry-build-system that is just like
python-build-system (probably even inherits from it) but uses "poetry"
instead of "python setup.py".

If the author of a package replaces the build system used in his actual
project, he has to expect to also have to replace the build-system reference
in the guix package.  Why is that weird?

Or you could try to add it to the existing python-build-system--but the
poetry website doesn't sound like it's designed like that (it rather sounds
like they want to replace all other python build systems).

> Just put the d**n tests in the Python package? This would look like a
> failure to me! :-(

If the end user doesn't need the tests, the tests shouldn't make it into the
derivation of your package.  But they are there while the package is building
the derivation--so just run the tests then.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Packaging Python projects managed with Poetry

[Tanguy Le Carrour]

Hi Danny,

Thank you for your answer!


Le 10/22, Danny Milosavljevic a écrit :
> On Thu, 22 Oct 2020 17:15:20 +0200
> Tanguy Le Carrour <tanguy@bioneland.org> wrote:
> 
> > does not contain a `setup.py` file –because Poetry does not use it!—, and
> >the `python-build-system` fails.
> > I haven't wrap my head around this yet and I'm not sure what would be
> > the proper way to do it?
> 
> >Write a `python-poetry-build-system`? I hope not!
> 
> Why not?
> 
> According to https://github.com/python-poetry/poetry they took inspiration
> from existing build systems like cargo, and they just replaced setup.py by
> pyproject.toml.
> 
> So what you could do is create a poetry-build-system that is just like
> python-build-system (probably even inherits from it) but uses "poetry"
> instead of "python setup.py".
> 
> If the author of a package replaces the build system used in his actual
> project, he has to expect to also have to replace the build-system reference
> in the guix package.  Why is that weird?

Oh, when I said "hope", I didn't mean to say "weird", but "out of my
comfort zone"! But I guess it would be a good way to learn more about
Guix's internals. So, wouldn't be so bad after all!

I'll give it a try.


> Or you could try to add it to the existing python-build-system--but the
> poetry website doesn't sound like it's designed like that (it rather sounds
> like they want to replace all other python build systems).

Actually, they don't "replace" all other systems, they build on them.
Internally, it uses `pip` and `virtualenv`. And there is competition out
there! To be faire, I should also package `hatch` [1] and `pipenv` [2].
But first, I have to make it right with `poetry`.

[1]: https://github.com/ofek/hatch
[2]: https://pipenv.pypa.io/en/latest


> > Just put the d**n tests in the Python package? This would look like a
> > failure to me! :-(
> 
> If the end user doesn't need the tests, the tests shouldn't make it into the
> derivation of your package.  But they are there while the package is building
> the derivation--so just run the tests then.

And… this is where the end of my comprehension of Guix is reached! ^_^'
The only thing I want to make sure is that the tests don't end up on the
the end user's system, the one that runs `guix install project-managed-with-poetry`.
But I guess the tests (contained in the source) will remain on the builder's system.
At least until they run `guix gc`.

Thanks for your help!

-- 
Tanguy

Re: Packaging Python projects managed with Poetry

[Tanguy Le Carrour]

Hi Christopher,

Thanks for your answer!


Le 10/22, Christopher Baines a écrit :
> Tanguy Le Carrour <tanguy@bioneland.org> writes:
> > I've been happily working with Poetry to manage my Python projects, but
> > now, for the first time, I would like to package one of those projects
> > for Guix.
> >
> > The Python packages I build do not contain any tests or specs, because
> > to me, they don't belong there. But, I need those tests to make sure
> > that my package works with the versions of the dependencies available on
> > Guix.
> >
> > The problem is that the source code that I fetch from the git repository
> > contains the test, but does not contain a `setup.py` file –because Poetry
> > does not use it!—, and the `python-build-system` fails.
> >
> > I haven't wrap my head around this yet and I'm not sure what would be
> > the proper way to do it? Write a `python-poetry-build-system`? I hope not!
> > Just put the d**n tests in the Python package? This would look like a
> > failure to me! :-(
> >
> > Any thought, help, guidance welcome! Thanks! :-)
> 
> My first thought, is what would it require for the existing
> python-build-system to detect and support building the things you
> describe?

Mmmm… I guess it would require fixing some of the phases, like stopping
it from trying to patch `setup.py`.
The tests would have to be run from the source directory, but the command
could be anything: `pytest`, `nosetest`, `invoke test`, `mamba`…

Then, instead of running `python setup.py`, one should run `poetry build`
and `pip install dist/name-of-the-package`.

So I guess Danny is right and a poetry-build-system would make sense.


> I haven't used Poetry myself, have you got a project that can be used as
> an example?

```
$ git clone https://github.com/tlc28/test-poetry.git
$ cd test-poetry/
$ poetry build
$ pip install dist/test_poetry-0.1.0-py3-none-any.whl
```

> It looks like the python-build-system already has some functionality
> that's dependent on the use-setuptools? argument.

I guess I'll have to dig into it! Thanks for pointing out!

-- 
Tanguy