Re: Prototype for using Guix with python packages

[ludo@gnu.org (Ludovic Courtès)]

Hi,

Christopher Baines <mail@cbaines.net> skribis:

> On 07/09/16 15:01, Ludovic Courtès wrote:
>>> For a few months now, I've been assembling a prototype for how packages
>>> could be produced for software released as Python source distributions
>>> (sdists) [1].
>> 
>> Woow, quite an achievement!  Do you know how many of the automatically
>> generated packages built from source flawlessly?
>
> Not exactly, I disabled quite a few test suites, and made some
> requirements stricter in some cases among other things.

OK, not too bad.

>> IIRC, ‘guix import pypi’ currently produces templates that can require
>> extra tweaks, although that was improved by reading metadata from
>> Wheels.
>> 
>> How does sdist metadata differ from PyPI or Wheels metadata?  Is it
>> generally more complete, or of better quality?
>
> sdists normally contain a .egg-info directory (e.g. sentry.egg-info),
> which contains a requires.txt file, which as far as I can tell contains
> the same information as the metadata.json in a wheel.
>
> I use that, as it means I can get the data without having to rely on the
> existence of a wheel.

‘guix import’ uses Wheels data when available, and otherwise falls back
to requirements.txt.  AIUI, the rationale was that Wheels dependency
information that is more detailed/accurate than requirements.txt:

  https://lists.gnu.org/archive/html/guix-devel/2016-01/msg00822.html

I’m Ccing Cyril who knows this better.  :-)

> I think the next steps towards this would be:
>  - Switch from downloading wheels to just using the requires.txt in the
> .egg-info directory
>  - Add support for https://www.python.org/dev/peps/pep-0518/
>  - Identify when PBR (Python Build Reasonableness) is in use, as it may
> mean that the test requirements are available (in test-requirements.txt)
>
> In my attempt to do this in a very automated way, I've only done the
> first point, but I build the package and parse the build log for signs
> of missing dependencies if the build fails, and then repeat the build.
> This is done first with the tests disabled, and then if it eventually
> builds, again with the tests enabled. I'm not sure if you would want
> guix import to do this?

Anything that improves the quality of what ‘guix import pypi’ produces
would be welcome.

> The main reason why I didn't just improve the importer is that I was
> looking for a way to collaborate around getting multiple versions of
> python packages building and working, and as far as I am aware, Guix
> only contains multiple versions of the same piece of software in some
> special cases?

That’s right: we keep multiple versions only when people or packages
expect to be able to choose among them (e.g., the various GnuPG series,
GCC, Python 2.x vs. 3.x).  That’s usually a case-by-case decision,
because every additional version that is kept entails additional
maintenance work.

The same would apply here: we could have multiple versions of some
packages, and only one version of the majority of them.

What do you think of this approach in the Python context?

Thanks,
Ludo’.