Prototype for using Guix with python packages

Prototype for using Guix with python packages

[Christopher Baines]

[-- Attachment #1.1.1: Type: text/plain, Size: 3262 bytes --]

For a few months now, I've been assembling a prototype for how packages
could be produced for software released as Python source distributions
(sdists) [1].

I wanted to build something that was as useful as any similar tool. The
functionality offered by pip [2] was something I looked at, as through
it you can use a large amount of software from the main PyPI [3],
including many different versions of the same project and use the
setuptools feature of extras [4] which allow for optional dependencies.

This differs from the functionality currently offered by the python
packages found in the Guix repository currently, as the number is much
smaller than available on PyPI, and typically only one release is available.

I've now managed to build a system which in a very automated way can
acquire sdists, determine there dependencies, use a full dependency
resolver to determine exactly what version of each dependency to use
across the entire build graph, and convert that graph in to Guix package
records, which can then be built.

Due to the wonderful power of Guix, this should be reasonably easy to
try out. I have imported a piece of software called sentry (as it was
something I was interested in using at the time) and all of its
dependencies in to the repository at [5]. You can find the instructions
here [5] for generating a package for sentry from this information.

My plan was to work towards something that could be run automatically to
stay up to date with a PyPI, so I tried to even automatically solve as
many issues as possible, e.g.  dependency looks, so if you look at the
build graph for sentry (attached, and the command to generate it is here
[5]), you should be able to see that the tests are run for setuptools,
something that is not even done for the package in the Guix repository.

I don't view this as a good long term approach, as I'm really hoping
that maybe one day, Guix could help to reduce the specific nature in
which software written in different languages is packaged/released. But
until that happens, I think something like this could be very useful at
getting some of the value that Guix provides quickly, without lots of
manual effort.

The code is very rough as I took a lot of shortcuts to get things
working, and I have lots of ideas for things I would like to change and
improve, but as I have been working on this for so long, I thought it
was time to publish it.

I will also try to look at how what I have been doing could be used to
improve the Guix PyPI importer and the Python build system, as well as
submit the packages that are currently just in the guix-env.scm file in
the repository (pyguile, libsolv) for inclusion in Guix.

I'm quite excited by the possibilities offered by approaches like this,
and was wondering if anyone has feedback, questions, opinions, or if
anyone is working on something similar?

Thanks,

Chris

1: https://docs.python.org/2/distutils/sourcedist.html
2: https://pypi.python.org/pypi/pip
3: https://pypi.python.org/pypi
4:
http://setuptools.readthedocs.io/en/latest/setuptools.html#declaring-extras-optional-features-with-their-own-dependencies
5: http://git.cbaines.net/guix-python-integration-project/about/

[-- Attachment #1.1.2: sentry-graph.pdf --]
[-- Type: application/pdf, Size: 75204 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 955 bytes --]

Re: Prototype for using Guix with python packages

[Ludovic Courtès]

Hello!

Christopher Baines <mail@cbaines.net> skribis:

> For a few months now, I've been assembling a prototype for how packages
> could be produced for software released as Python source distributions
> (sdists) [1].

Woow, quite an achievement!  Do you know how many of the automatically
generated packages built from source flawlessly?

IIRC, ‘guix import pypi’ currently produces templates that can require
extra tweaks, although that was improved by reading metadata from
Wheels.

How does sdist metadata differ from PyPI or Wheels metadata?  Is it
generally more complete, or of better quality?

> I will also try to look at how what I have been doing could be used to
> improve the Guix PyPI importer and the Python build system, as well as
> submit the packages that are currently just in the guix-env.scm file in
> the repository (pyguile, libsolv) for inclusion in Guix.

That would be awesome!  Anything towards making the output of ‘guix
import’ work out of the box would be great.

> I'm quite excited by the possibilities offered by approaches like this,
> and was wondering if anyone has feedback, questions, opinions, or if
> anyone is working on something similar?

That reminds me of the “recursive importer” that has been discussed a
few times and is currently implemented (but not in master yet) for the
CRAN and NPM importers.  Perhaps your work could be somehow integrated
to get a recursive PyPI/sdist importer?

An idea would be to automatically put all the imported packages into,
say, ~/.cache/guix/python, and add that to GUIX_PACKAGE_PATH.  Then we
could run “guix import pypi --update” to update it, or something like
that.

Thanks for sharing!

Ludo’.

Re: Prototype for using Guix with python packages

[Christopher Baines]

[-- Attachment #1.1: Type: text/plain, Size: 3955 bytes --]

On 07/09/16 15:01, Ludovic Courtès wrote:
>> For a few months now, I've been assembling a prototype for how packages
>> could be produced for software released as Python source distributions
>> (sdists) [1].
> 
> Woow, quite an achievement!  Do you know how many of the automatically
> generated packages built from source flawlessly?

Not exactly, I disabled quite a few test suites, and made some
requirements stricter in some cases among other things.

> IIRC, ‘guix import pypi’ currently produces templates that can require
> extra tweaks, although that was improved by reading metadata from
> Wheels.
> 
> How does sdist metadata differ from PyPI or Wheels metadata?  Is it
> generally more complete, or of better quality?

sdists normally contain a .egg-info directory (e.g. sentry.egg-info),
which contains a requires.txt file, which as far as I can tell contains
the same information as the metadata.json in a wheel.

I use that, as it means I can get the data without having to rely on the
existence of a wheel.

Its still really poor quality though, as it does not include
dependencies required to run the tests, nor build system dependencies
(e.g. setuptools or otherwise) which may be found in a different file [1].

1: https://www.python.org/dev/peps/pep-0518/

>> I will also try to look at how what I have been doing could be used to
>> improve the Guix PyPI importer and the Python build system, as well as
>> submit the packages that are currently just in the guix-env.scm file in
>> the repository (pyguile, libsolv) for inclusion in Guix.
> 
> That would be awesome!  Anything towards making the output of ‘guix
> import’ work out of the box would be great.

I think the next steps towards this would be:
 - Switch from downloading wheels to just using the requires.txt in the
.egg-info directory
 - Add support for https://www.python.org/dev/peps/pep-0518/
 - Identify when PBR (Python Build Reasonableness) is in use, as it may
mean that the test requirements are available (in test-requirements.txt)

In my attempt to do this in a very automated way, I've only done the
first point, but I build the package and parse the build log for signs
of missing dependencies if the build fails, and then repeat the build.
This is done first with the tests disabled, and then if it eventually
builds, again with the tests enabled. I'm not sure if you would want
guix import to do this?

>> I'm quite excited by the possibilities offered by approaches like this,
>> and was wondering if anyone has feedback, questions, opinions, or if
>> anyone is working on something similar?
> 
> That reminds me of the “recursive importer” that has been discussed a
> few times and is currently implemented (but not in master yet) for the
> CRAN and NPM importers.  Perhaps your work could be somehow integrated
> to get a recursive PyPI/sdist importer?
> 
> An idea would be to automatically put all the imported packages into,
> say, ~/.cache/guix/python, and add that to GUIX_PACKAGE_PATH.  Then we
> could run “guix import pypi --update” to update it, or something like
> that.

The main reason why I didn't just improve the importer is that I was
looking for a way to collaborate around getting multiple versions of
python packages building and working, and as far as I am aware, Guix
only contains multiple versions of the same piece of software in some
special cases?

I have implemented a recursive importer, but, I'm not trying to generate
Guix package records, but instead records describing the versioned
requirements, as from this, build graphs can be generated. I don't like
this approach in that it duplicates a few packages in the Guix
repository, but having multiple versions can be really useful in some
circumstances.

Thanks for your feedback Ludo, and it would be really interesting to
hear your thoughts on the multiple version issue.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 955 bytes --]

Re: Prototype for using Guix with python packages

[Ludovic Courtès]

Hi,

Christopher Baines <mail@cbaines.net> skribis:

> On 07/09/16 15:01, Ludovic Courtès wrote:
>>> For a few months now, I've been assembling a prototype for how packages
>>> could be produced for software released as Python source distributions
>>> (sdists) [1].
>> 
>> Woow, quite an achievement!  Do you know how many of the automatically
>> generated packages built from source flawlessly?
>
> Not exactly, I disabled quite a few test suites, and made some
> requirements stricter in some cases among other things.

OK, not too bad.

>> IIRC, ‘guix import pypi’ currently produces templates that can require
>> extra tweaks, although that was improved by reading metadata from
>> Wheels.
>> 
>> How does sdist metadata differ from PyPI or Wheels metadata?  Is it
>> generally more complete, or of better quality?
>
> sdists normally contain a .egg-info directory (e.g. sentry.egg-info),
> which contains a requires.txt file, which as far as I can tell contains
> the same information as the metadata.json in a wheel.
>
> I use that, as it means I can get the data without having to rely on the
> existence of a wheel.

‘guix import’ uses Wheels data when available, and otherwise falls back
to requirements.txt.  AIUI, the rationale was that Wheels dependency
information that is more detailed/accurate than requirements.txt:

  https://lists.gnu.org/archive/html/guix-devel/2016-01/msg00822.html

I’m Ccing Cyril who knows this better.  :-)

> I think the next steps towards this would be:
>  - Switch from downloading wheels to just using the requires.txt in the
> .egg-info directory
>  - Add support for https://www.python.org/dev/peps/pep-0518/
>  - Identify when PBR (Python Build Reasonableness) is in use, as it may
> mean that the test requirements are available (in test-requirements.txt)
>
> In my attempt to do this in a very automated way, I've only done the
> first point, but I build the package and parse the build log for signs
> of missing dependencies if the build fails, and then repeat the build.
> This is done first with the tests disabled, and then if it eventually
> builds, again with the tests enabled. I'm not sure if you would want
> guix import to do this?

Anything that improves the quality of what ‘guix import pypi’ produces
would be welcome.

> The main reason why I didn't just improve the importer is that I was
> looking for a way to collaborate around getting multiple versions of
> python packages building and working, and as far as I am aware, Guix
> only contains multiple versions of the same piece of software in some
> special cases?

That’s right: we keep multiple versions only when people or packages
expect to be able to choose among them (e.g., the various GnuPG series,
GCC, Python 2.x vs. 3.x).  That’s usually a case-by-case decision,
because every additional version that is kept entails additional
maintenance work.

The same would apply here: we could have multiple versions of some
packages, and only one version of the majority of them.

What do you think of this approach in the Python context?

Thanks,
Ludo’.

Re: Prototype for using Guix with python packages

[Christopher Baines]

[-- Attachment #1.1: Type: text/plain, Size: 2639 bytes --]

On 11/09/16 14:43, Ludovic Courtès wrote:
>> The main reason why I didn't just improve the importer is that I was
>> looking for a way to collaborate around getting multiple versions of
>> python packages building and working, and as far as I am aware, Guix
>> only contains multiple versions of the same piece of software in some
>> special cases?
> 
> That’s right: we keep multiple versions only when people or packages
> expect to be able to choose among them (e.g., the various GnuPG series,
> GCC, Python 2.x vs. 3.x).  That’s usually a case-by-case decision,
> because every additional version that is kept entails additional
> maintenance work.
> 
> The same would apply here: we could have multiple versions of some
> packages, and only one version of the majority of them.
> 
> What do you think of this approach in the Python context?

I've been quite lucky, as I have managed to stay away from using pip and
other Python specific packaging things (by using Debian packages), but,
lots of people do use these tools.

In the future, I would love to be able to use Guix when writing Python,
and to encourage others to do similarly. Ignoring myself and
generalising, most people who use Python use pip and pip though using
pypi.python.org allows installing one of many available versions of a
particular package. Therefore, I would think that most people who
consider using Guix for a similar use case in which they use pip, would
expect to be able to choose among multiple versions of every package.

I don't think adding every release of every bit of free software on
pypi.python.org in to Guix is feasible, or something that I would want
to propose doing, but I still would really like something that gives
Guix equal (or greater) utility with respect to Python packages.

Which is why I started thinking, and then developing a way of
automatically importing releases from pypi.python.org in to a Guix
package repository that depends on Guix.

All of this is still very uncertain in my mind, in particular, I see
Guix as a viable alternative to lots of domain specific package
managers, which I would really like to avoid when writing software (e.g.
pip, npm gem) but I'm not certain that being a viable alternative to
these tools is within the project scope?

The other main uncertainty in my mind is whether the approach I have
been trying is the best one? I'm getting more confident that this is
possible, although there are still technical problems (for example,
adding new versions, without breaking existing versions of other
packages) that I haven't solved yet.



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 955 bytes --]

Re: Prototype for using Guix with python packages

[Ludovic Courtès]

Hi,

Christopher Baines <mail@cbaines.net> skribis:

> All of this is still very uncertain in my mind, in particular, I see
> Guix as a viable alternative to lots of domain specific package
> managers, which I would really like to avoid when writing software (e.g.
> pip, npm gem) but I'm not certain that being a viable alternative to
> these tools is within the project scope?

Yes and no.  Clearly how NPM and, to a lesser extent, PyPI approach
packaging is very different from how Debian or Guix approach it.

They tend to be very large repos with little or no review, and little
consideration for computing freedom in the case of NPM; they tend to be
mostly automated things that move very fast.

As discussed in another thread, maybe what we can do is provide “core”
packages in Guix proper.  There could be auto-importers for the rest,
but users would have to trust those other repos, which isn’t great.

Then again, importing and maintaining PyPI packages (with the help of
‘guix import’ and ‘guix refresh’) sounds more reasonable than importing
and maintaining NPM stuff.

> The other main uncertainty in my mind is whether the approach I have
> been trying is the best one? I'm getting more confident that this is
> possible, although there are still technical problems (for example,
> adding new versions, without breaking existing versions of other
> packages) that I haven't solved yet.

I think there’s no definite answer; we’ll probably have to explore
different possibilities.

Ludo’.