From mboxrd@z Thu Jan 1 00:00:00 1970 From: Christopher Baines Subject: Re: Prototype for using Guix with python packages Date: Sat, 10 Sep 2016 14:09:58 +0100 Message-ID: References: <4174c012-8e2b-d6f2-8387-031e309b7daf@cbaines.net> <87fupbn713.fsf@gnu.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="6vS34UdqIqvO2V2OAJMXKTC8sMaEdd6fE" Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:45605) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bii2t-00062S-77 for guix-devel@gnu.org; Sat, 10 Sep 2016 09:10:12 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bii2o-0004lf-MJ for guix-devel@gnu.org; Sat, 10 Sep 2016 09:10:11 -0400 Received: from mira.cbaines.net ([2a01:7e00::f03c:91ff:fe69:8da9]:60561) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bii2n-0004jt-ST for guix-devel@gnu.org; Sat, 10 Sep 2016 09:10:06 -0400 In-Reply-To: <87fupbn713.fsf@gnu.org> List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: =?UTF-8?Q?Ludovic_Court=c3=a8s?= Cc: guix-devel@gnu.org This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --6vS34UdqIqvO2V2OAJMXKTC8sMaEdd6fE Content-Type: multipart/mixed; boundary="SKspq8SsoAPnKIAkA41A6mPBO6RU268WR"; protected-headers="v1" From: Christopher Baines To: =?UTF-8?Q?Ludovic_Court=c3=a8s?= Cc: guix-devel@gnu.org Message-ID: Subject: Re: Prototype for using Guix with python packages References: <4174c012-8e2b-d6f2-8387-031e309b7daf@cbaines.net> <87fupbn713.fsf@gnu.org> In-Reply-To: <87fupbn713.fsf@gnu.org> --SKspq8SsoAPnKIAkA41A6mPBO6RU268WR Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On 07/09/16 15:01, Ludovic Court=C3=A8s wrote: >> For a few months now, I've been assembling a prototype for how package= s >> could be produced for software released as Python source distributions= >> (sdists) [1]. >=20 > Woow, quite an achievement! Do you know how many of the automatically > generated packages built from source flawlessly? Not exactly, I disabled quite a few test suites, and made some requirements stricter in some cases among other things. > IIRC, =E2=80=98guix import pypi=E2=80=99 currently produces templates t= hat can require > extra tweaks, although that was improved by reading metadata from > Wheels. >=20 > How does sdist metadata differ from PyPI or Wheels metadata? Is it > generally more complete, or of better quality? sdists normally contain a .egg-info directory (e.g. sentry.egg-info), which contains a requires.txt file, which as far as I can tell contains the same information as the metadata.json in a wheel. I use that, as it means I can get the data without having to rely on the existence of a wheel. Its still really poor quality though, as it does not include dependencies required to run the tests, nor build system dependencies (e.g. setuptools or otherwise) which may be found in a different file [1]= =2E 1: https://www.python.org/dev/peps/pep-0518/ >> I will also try to look at how what I have been doing could be used to= >> improve the Guix PyPI importer and the Python build system, as well as= >> submit the packages that are currently just in the guix-env.scm file i= n >> the repository (pyguile, libsolv) for inclusion in Guix. >=20 > That would be awesome! Anything towards making the output of =E2=80=98= guix > import=E2=80=99 work out of the box would be great. I think the next steps towards this would be: - Switch from downloading wheels to just using the requires.txt in the =2Eegg-info directory - Add support for https://www.python.org/dev/peps/pep-0518/ - Identify when PBR (Python Build Reasonableness) is in use, as it may mean that the test requirements are available (in test-requirements.txt) In my attempt to do this in a very automated way, I've only done the first point, but I build the package and parse the build log for signs of missing dependencies if the build fails, and then repeat the build. This is done first with the tests disabled, and then if it eventually builds, again with the tests enabled. I'm not sure if you would want guix import to do this? >> I'm quite excited by the possibilities offered by approaches like this= , >> and was wondering if anyone has feedback, questions, opinions, or if >> anyone is working on something similar? >=20 > That reminds me of the =E2=80=9Crecursive importer=E2=80=9D that has be= en discussed a > few times and is currently implemented (but not in master yet) for the > CRAN and NPM importers. Perhaps your work could be somehow integrated > to get a recursive PyPI/sdist importer? >=20 > An idea would be to automatically put all the imported packages into, > say, ~/.cache/guix/python, and add that to GUIX_PACKAGE_PATH. Then we > could run =E2=80=9Cguix import pypi --update=E2=80=9D to update it, or = something like > that. The main reason why I didn't just improve the importer is that I was looking for a way to collaborate around getting multiple versions of python packages building and working, and as far as I am aware, Guix only contains multiple versions of the same piece of software in some special cases? I have implemented a recursive importer, but, I'm not trying to generate Guix package records, but instead records describing the versioned requirements, as from this, build graphs can be generated. I don't like this approach in that it duplicates a few packages in the Guix repository, but having multiple versions can be really useful in some circumstances. Thanks for your feedback Ludo, and it would be really interesting to hear your thoughts on the multiple version issue. --SKspq8SsoAPnKIAkA41A6mPBO6RU268WR-- --6vS34UdqIqvO2V2OAJMXKTC8sMaEdd6fE Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQKOBAEBCgB4BQJX1AYqXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQzRTg5RUVFNzQ1OEU3MjBEOTc1NEUwQjI1 RTI4QTMzQjBCODRGNTc3ERxtYWlsQGNiYWluZXMubmV0AAoJEF4oozsLhPV3cbMQ AKN1T1Wf33I9m4hbHD7eVoimX134UEG2qSmIB+hFRJrmidnXAqbdOlqUnXwDrIs0 0DYi06508SW5V+bsIKPBBoTKS2oEpAMhX3yqp6q2zCMnOcBK/eTMPAsdq7sXm7T5 lZkjwkUOfECC5l8yyocZUgd9vPsIV+5ZNXtG5eFeBBeo+NdkhwfJMRLo3u1Sf6Dy x+FLYb4cSHxVquZMwc5o6WbaNYruz2r+8acgpeSN8gebaKvZGfvtbVUYRg+8ZuUO 8226m+dQhulWd5yqp0rZurKbqItY4l0HCxDGg/Nc5kFerrQYMPUa1C5tuPtlSDE7 vNlGx7uMBhO/fbJgUGoRMT/pOFKquap690SBOWXUxS83pjg4/gn4vKgdnM2vC8Bs dyXTUu55yZpeMP9e7tLWZRjXhZSxjZ0AY80RaJcQqhwd3QdnH3rN7DijEAnqAiQA q1M9zDahxJq+2IZxc/eWBN+340umiSH2jHiJKPzPdTs3YXMWA+tqe7lOel+3pJO+ tDC/yYmd07wADCYUxmIYFHnA6zhs4qCU9RMJG4c5bgWPipKZIUDy79rYtyseQjdy FpK3uuF2nn0Nn4SSfJp+f9N49w7U9XGY0JVT8LvAlYyGXrDzqMJH6Pnak8jYWZrM A2Z2kwMOxpjwFojiKNgqJ7wIayYvhZDoAnKcbfFtByp1 =a7Dr -----END PGP SIGNATURE----- --6vS34UdqIqvO2V2OAJMXKTC8sMaEdd6fE--