From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id SEWQBbcbDGCpQQAA0tVLHw (envelope-from ) for ; Sat, 23 Jan 2021 12:51:03 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id 0OKRAbcbDGBJLAAAB5/wlQ (envelope-from ) for ; Sat, 23 Jan 2021 12:51:03 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 8EDCB9402B1 for ; Sat, 23 Jan 2021 12:51:02 +0000 (UTC) Received: from localhost ([::1]:54984 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l3INZ-0002yq-BR for larch@yhetil.org; Sat, 23 Jan 2021 07:51:01 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:40516) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l3I7f-0005es-7v for guix-devel@gnu.org; Sat, 23 Jan 2021 07:34:35 -0500 Received: from mout-p-202.mailbox.org ([2001:67c:2050::465:202]:56736) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_CHACHA20_POLY1305:256) (Exim 4.90_1) (envelope-from ) id 1l3I7U-0002fK-AZ for guix-devel@gnu.org; Sat, 23 Jan 2021 07:34:34 -0500 Received: from smtp1.mailbox.org (smtp1.mailbox.org [IPv6:2001:67c:2050:105:465:1:1:0]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mout-p-202.mailbox.org (Postfix) with ESMTPS id 4DNFtd5kj3zQlRy; Sat, 23 Jan 2021 13:34:17 +0100 (CET) X-Virus-Scanned: amavisd-new at heinlein-support.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6xq.net; s=MBO0001; t=1611405255; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=C5zaGTWp+V3bzKFM4Ne7XPSN7sscihsIg2KPVJak7C4=; b=vd36YVTxLOdL3k3HV9nv49n9sttc+ZgioSvcO6G3a5VIV8/nfnNjwBrDk1HeBSjQThhqZ0 dzZlBIEsbz94X3P963Pnv4LYpsbgwZJjISa2EoDM4/XBzCT/CYa2rD6+NzzNMlZa0dwLmm bWIx7Z0bCZiXABfhWc/I2sCbCbiG3RY0xTaQhQAL1z7ZfWQRSjNiIu7JxKsOR6qPdOrhAY jR02goRGKuYqqg1/olygmZE2xKYBq6/wNl0oIwox4O1L8QKXTuUIVJEF7GaLbUxTZqSCIy BUpGJYgDZUKWP4VGkyAqDSxPL3YgKg8WNaQePQWme0mjrqH+qmEgPKAmhjTJ0A== Received: from smtp1.mailbox.org ([80.241.60.240]) by gerste.heinlein-support.de (gerste.heinlein-support.de [91.198.250.173]) (amavisd-new, port 10030) with ESMTP id FSeITIfp0LGS; Sat, 23 Jan 2021 13:34:11 +0100 (CET) Date: Sat, 23 Jan 2021 13:34:09 +0100 From: Lars-Dominik Braun To: Tanguy LE CARROUR Subject: Re: Questions regarding Python packaging Message-ID: References: <20201108142717.lmud5h4gh44vtjc6@melmoth> <1609946775.8blxygrg9p.astroid@rafflesia.none> <1611303651.35tpgtn1z1.astroid@melmoth.none> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="nHGOwjDfkGutFLan" Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <1611303651.35tpgtn1z1.astroid@melmoth.none> X-MBO-SPAM-Probability: X-Rspamd-Score: -6.63 / 15.00 / 15.00 X-Rspamd-Queue-Id: 5E22B1848 X-Rspamd-UID: b26ede Received-SPF: pass client-ip=2001:67c:2050::465:202; envelope-from=lars@6xq.net; helo=mout-p-202.mailbox.org X-Spam_score_int: -19 X-Spam_score: -2.0 X-Spam_bar: -- X-Spam_report: (-2.0 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_SBL_A=0.1 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Guix Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-Spam-Score: -0.25 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=6xq.net header.s=MBO0001 header.b=vd36YVTx; dmarc=fail reason="SPF not aligned (relaxed)" header.from=6xq.net (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Queue-Id: 8EDCB9402B1 X-Spam-Score: -0.25 X-Migadu-Scanner: scn0.migadu.com X-TUID: m3ffFVFLqDZa --nHGOwjDfkGutFLan Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit Hi Tanguy, > Done! :-) > I've eventually succeeded in ("properly") packaging a software managed > with Poetry. And I've learned quite a lot on the way! oh, I see. I’ve actually been trying to replace python-build-system with a python-build based build. Attached is my current work in progress. I cannot quite build python-build, because I’m lacking support for python-flit, but I think the general idea is clear: Remove pip and setuptools from python (saves almost 20MiB from the closure and avoids weird conflicts between python’s setuptools and python-setuptools) and turn them into (almost) ordinary packages. Then use setuptools to bootstrap itself, bootstrap python-build with setuptools and use python-build to build evrey other packages using python-build-system. Cheers, Lars --nHGOwjDfkGutFLan Content-Type: text/plain; charset=utf-8 Content-Disposition: attachment; filename="wip-pep-517-build.diff" Content-Transfer-Encoding: 8bit diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm index d08e23936c..a9843bd9cb 100644 --- a/gnu/packages/python-xyz.scm +++ b/gnu/packages/python-xyz.scm @@ -1123,14 +1123,14 @@ other machines, such as over the network.") (define-public python-setuptools (package (name "python-setuptools") - (version "41.0.1") + (version "51.3.1") (source (origin (method url-fetch) - (uri (pypi-uri "setuptools" version ".zip")) + (uri (pypi-uri "setuptools" version)) (sha256 (base32 - "04sns22y2hhsrwfy1mha2lgslvpjsjsz8xws7h2rh5a7ylkd28m2")) + "07qk6ykhq1q4k33x1m96r39f9qpfzp2xfl5ly9zww0180gssrvqc")) (modules '((guix build utils))) (snippet '(begin @@ -1143,7 +1143,26 @@ other machines, such as over the network.") ;; FIXME: Tests require pytest, which itself relies on setuptools. ;; One could bootstrap with an internal untested setuptools. (arguments - `(#:tests? #f)) + `(#:tests? #f + #:need-setuptools #f + #:python-build-variant #f + #:phases (modify-phases %standard-phases + (replace 'build + (lambda _ + (invoke "python" "setup.py" "build"))) + (replace 'install + (lambda* (#:key outputs #:allow-other-keys) + (let ((out (assoc-ref outputs "out"))) + (invoke "python" "setup.py" "install" "--prefix" out "--single-version-externally-managed" "--root=/")))) + ;; Use this setuptools’ sources to bootstrap themselves. + (add-before 'build 'set-PYTHONPATH + (lambda _ + (format #t "current working dir ~s~%" (getcwd)) + (setenv "PYTHONPATH" + (string-append ".:" (getenv "PYTHONPATH"))) + #t))))) + ;; Not required when not building a wheel + ;(propagated-inputs `(("python-wheel" ,python-wheel))) (home-page "https://pypi.org/project/setuptools/") (synopsis "Library designed to facilitate packaging Python projects") @@ -1822,7 +1841,7 @@ Python file, so it can be easily copied into your project.") (package (inherit python-six) (name "python-six-bootstrap") - (native-inputs `()) + (native-inputs '()) (arguments `(#:tests? #f)))) (define-public python2-six-bootstrap @@ -5675,7 +5694,8 @@ by pycodestyle.") ;; NOTE: Any value works, the variable just has to be present. (setenv "SKIP_ONLINE" "1") #t))))) - (native-inputs `(("unzip" ,unzip))) + (native-inputs + `(("unzip" ,unzip))) (home-page "https://bitbucket.org/pypa/distlib") (synopsis "Distribution utilities") (description "Distlib is a library which implements low-level functions that @@ -15272,7 +15292,7 @@ protocols.") (package (inherit python-attrs) (name "python-attrs-bootstrap") - (native-inputs `()) + (native-inputs '()) (arguments `(#:tests? #f)))) (define-public python2-attrs-bootstrap @@ -23451,3 +23471,292 @@ Qt applications.") "Pivy provides python bindings for Coin, a 3D graphics library with an Application Programming Interface based on the Open Inventor 2.1 API.") (license license:isc))) + +(define-public python-build + (package + (name "python-build") + (version "0.1.0") + (source + (origin + (method url-fetch) + (uri (pypi-uri "build" version)) + (sha256 + (base32 + "1d6m21lijwm04g50nwgsgj7x3vhblzw7jv05ah8psqgzk20bbch8")))) + (build-system python-build-system) + (arguments `(#:python-build-variant ,python-build-bootstrap)) + (propagated-inputs + `(("python-packaging" ,python-packaging) + ("python-pep517" ,python-pep517) + ("python-toml" ,python-toml) + ;; For fallback option. + ("python-setuptools" ,python-setuptools))) + (native-inputs + `(("python-filelock" ,python-filelock) + ("python-pytest" ,python-pytest) + ("python-pytest-cov" ,python-pytest-cov) + ("python-pytest-mock" ,python-pytest-mock) + ("python-pytest-xdist" ,python-pytest-xdist))) + (home-page "https://pypa-build.readthedocs.io/") + (synopsis + "A simple, correct PEP517 package builder") + (description + "A simple, correct PEP517 package builder") + (license license:expat))) + +(define-public python-pep517-bootstrap + (package + (inherit python-pep517) + (name "python-pep517-bootstrap") + (arguments `(#:tests? #f)) + (native-inputs `()))) + +(define-public python-build-bootstrap + (package + (name "python-build-bootstrap") + (version "0.1.0") + (source + (origin + (method url-fetch) + (uri (pypi-uri "build" version)) + (sha256 + (base32 + "1d6m21lijwm04g50nwgsgj7x3vhblzw7jv05ah8psqgzk20bbch8")))) + (build-system python-build-system) + (arguments + `(#:tests? #f + #:python-build-variant #f + #:phases (modify-phases %standard-phases + (replace 'build + (lambda _ + (invoke "python" "setup.py" "build"))) + (replace 'install + (lambda* (#:key outputs #:allow-other-keys) + (let ((out (assoc-ref outputs "out"))) + (invoke "python" "setup.py" "install" "--prefix" out "--single-version-externally-managed" "--root=/"))))))) + (propagated-inputs + `(("python-pep517" ,python-pep517-source-bootstrap) + ("python-packaging" ,python-packaging-source-bootstrap))) + (home-page "https://pypa-build.readthedocs.io/") + (synopsis + "A simple, correct PEP517 package builder") + (description + "A simple, correct PEP517 package builder") + (license license:expat))) + +(define-public python-pep517-source-bootstrap + (package + (name "python-pep517-source-bootstrap") + (version "0.9.1") + (source + (origin + (method url-fetch) + (uri (pypi-uri "pep517" version)) + (sha256 + (base32 + "0zqidxah03qpnp6zkg3zd1kmd5f79hhdsfmlc0cldaniy80qddxf")))) + (build-system python-build-system) + (arguments + `(#:tests? #f + #:python-build-variant #f + #:phases (modify-phases %standard-phases +(add-after 'unpack 'patch + (lambda _ + (substitute* "setup.py" + (("distutils\\.core") "setuptools")) + #t)) + (replace 'build + (lambda _ + (invoke "python" "setup.py" "build"))) + (replace 'install + (lambda* (#:key outputs #:allow-other-keys) + (let ((out (assoc-ref outputs "out"))) + (invoke "python" "setup.py" "install" "--prefix" out "--single-version-externally-managed" "--root=/"))))))) + (propagated-inputs + `(("python-toml" ,python-toml-source-bootstrap) + ("python-wheel" ,python-wheel-bootstrap) + )) + (home-page "https://github.com/pypa/pep517") + (synopsis "Wrappers to build Python packages using PEP 517 hooks") + (description + "Wrappers to build Python packages using PEP 517 hooks.") + (license license:expat))) + +(define-public python-toml-source-bootstrap + (package + (name "python-toml-source-bootstrap") + (version "0.10.1") + (source + (origin + (method url-fetch) + (uri (pypi-uri "toml" version)) + (sha256 + (base32 + "03wbqm5cn685cwx2664hjdpz370njl7lf0yal8s0dkp5w4mn2swj")))) + (build-system python-build-system) + (arguments + `(#:tests? #f + #:python-build-variant #f + #:phases (modify-phases %standard-phases + (replace 'build + (lambda _ + (invoke "python" "setup.py" "build"))) + (replace 'install + (lambda* (#:key outputs #:allow-other-keys) + (let ((out (assoc-ref outputs "out"))) + (invoke "python" "setup.py" "install" "--prefix" out "--single-version-externally-managed" "--root=/"))))))) + (home-page "https://github.com/uiri/toml") + (synopsis "Library for TOML") + (description + "@code{toml} is a library for parsing and creating Tom's Obvious, Minimal +Language (TOML) configuration files.") + (license license:expat))) + +(define-public python-packaging-source-bootstrap + (package + (name "python-packaging-source-bootstrap") + (version "20.0") + (source + (origin + (method url-fetch) + (uri (pypi-uri "packaging" version)) + ;; XXX: The URL in the patch file is wrong, it should be + ;; . + (patches (search-patches "python-packaging-test-arch.patch")) + (sha256 + (base32 + "1y2ip3a4ykkpgnwgn85j6hkspcl0cg3mzms97f40mk57vwqq67gy")))) + (build-system python-build-system) + (arguments + `(#:tests? #f + #:python-build-variant #f + #:phases (modify-phases %standard-phases + (replace 'build + (lambda _ + (invoke "python" "setup.py" "build"))) + (replace 'install + (lambda* (#:key outputs #:allow-other-keys) + (let ((out (assoc-ref outputs "out"))) + (invoke "python" "setup.py" "install" "--prefix" out "--single-version-externally-managed" "--root=/"))))))) + (propagated-inputs + `(("python-pyparsing" ,python-pyparsing-bootstrap) + ("python-six" ,python-six-bootstrap2) + )) + (home-page "https://github.com/pypa/packaging") + (synopsis "Core utilities for Python packages") + (description "Packaging is a Python module for dealing with Python packages. +It offers an interface for working with package versions, names, and dependency +information.") + ;; From 'LICENSE': This software is made available under the terms of + ;; *either* of the licenses found in LICENSE.APACHE or LICENSE.BSD. + ;; Contributions to this software is made under the terms of *both* these + ;; licenses. + (license (list license:asl2.0 license:bsd-2)))) + +(define-public python-pyparsing-bootstrap + (package + (name "python-pyparsing-bootstrap") + (version "2.4.6") + (source + (origin + (method url-fetch) + (uri (pypi-uri "pyparsing" version)) + (sha256 + (base32 "17wn5zlijc9m9zj26gy3f541y7smpj8rfhl51d025c2gm210b0sc")))) + (build-system python-build-system) + (arguments + `(#:tests? #f + #:python-build-variant #f + #:phases (modify-phases %standard-phases + (replace 'build + (lambda _ + (invoke "python" "setup.py" "build"))) + (replace 'install + (lambda* (#:key outputs #:allow-other-keys) + (let ((out (assoc-ref outputs "out"))) + (invoke "python" "setup.py" "install" "--prefix" out "--single-version-externally-managed" "--root=/"))))))) + (home-page "https://github.com/pyparsing/pyparsing") + (synopsis "Python parsing class library") + (description + "The pyparsing module is an alternative approach to creating and +executing simple grammars, vs. the traditional lex/yacc approach, or the use +of regular expressions. The pyparsing module provides a library of classes +that client code uses to construct the grammar directly in Python code.") + (license license:expat))) + +(define-public python-six-bootstrap2 + (package + (name "python-six-bootstrap") + (version "1.14.0") + (source + (origin + (method url-fetch) + (uri (pypi-uri "six" version)) + (sha256 + (base32 + "02lw67hprv57hyg3cfy02y3ixjk3nzwc0dx3c4ynlvkfwkfdnsr3")))) + (build-system python-build-system) + (arguments + `(#:tests? #f + #:python-build-variant #f + #:phases (modify-phases %standard-phases + (replace 'build + (lambda _ + (invoke "python" "setup.py" "build"))) + (replace 'install + (lambda* (#:key outputs #:allow-other-keys) + (let ((out (assoc-ref outputs "out"))) + (invoke "python" "setup.py" "install" "--prefix" out "--single-version-externally-managed" "--root=/"))))))) + (home-page "https://pypi.org/project/six/") + (synopsis "Python 2 and 3 compatibility utilities") + (description + "Six is a Python 2 and 3 compatibility library. It provides utility +functions for smoothing over the differences between the Python versions with +the goal of writing Python code that is compatible on both Python versions. +Six supports every Python version since 2.5. It is contained in only one +Python file, so it can be easily copied into your project.") + (license license:x11))) + +(define-public python-wheel-bootstrap + (package + (name "python-wheel-bootstrap") + (version "0.33.6") + (source + (origin + (method url-fetch) + (uri (pypi-uri "wheel" version)) + (sha256 + (base32 + "0ii6f34rvpjg3nmw4bc2h7fhdsy38y1h93hghncfs5akfrldmj8h")))) + (build-system python-build-system) + (arguments + `(#:tests? #f + #:python-build-variant #f + #:phases (modify-phases %standard-phases +(add-after 'unpack 'patch-enable-entrypoints + (lambda _ + ;; python-wheel tells setuptools to not install entry point scripts + ;; by default. Stop doing that, so wheels built contain all data + ;; required. + (substitute* "wheel/bdist_wheel.py" + (("(install_scripts\\.no_ep = )True" all assignment) (string-append assignment "False"))) + #t)) + (replace 'build + (lambda _ + (invoke "python" "setup.py" "build"))) + (replace 'install + (lambda* (#:key outputs #:allow-other-keys) + (let ((out (assoc-ref outputs "out"))) + (invoke "python" "setup.py" "install" "--prefix" out "--single-version-externally-managed" "--root=/"))))))) + (home-page "https://bitbucket.org/pypa/wheel/") + (synopsis "Format for built Python packages") + (description + "A wheel is a ZIP-format archive with a specially formatted filename and +the @code{.whl} extension. It is designed to contain all the files for a PEP +376 compatible install in a way that is very close to the on-disk format. Many +packages will be properly installed with only the @code{Unpack} step and the +unpacked archive preserves enough information to @code{Spread} (copy data and +scripts to their final locations) at any later time. Wheel files can be +installed with a newer @code{pip} or with wheel's own command line utility.") + (license license:expat))) + diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm index fa9bf10e07..57221a91ea 100644 --- a/gnu/packages/python.scm +++ b/gnu/packages/python.scm @@ -100,6 +100,7 @@ #:use-module (guix utils) #:use-module (guix build-system gnu) #:use-module (guix build-system trivial) + #:use-module (guix build-system copy) #:use-module (srfi srfi-1) #:use-module (srfi srfi-26)) @@ -153,7 +154,7 @@ (list "--enable-shared" ;allow embedding "--with-system-expat" ;for XML support "--with-system-ffi" ;build ctypes - "--with-ensurepip=install" ;install pip and setuptools + "--with-ensurepip=no" ;do not install pip and setuptools "--enable-unicode=ucs4" ;; Prevent the installed _sysconfigdata.py from retaining a reference @@ -874,4 +875,12 @@ Python code natively, including C extensions.") license:bsd-3 ; lib_pypy/cffi/_pycparser/ply/ (license:non-copyleft "http://www.unicode.org/copyright.html"))))) +(define* (pypi-uri name version #:optional (extension ".tar.gz")) + "Return a URI string for the Python package hosted on the Python Package +Index (PyPI) corresponding to NAME and VERSION. EXTENSION is the file name +extension, such as '.tar.gz'." + (string-append "https://files.pythonhosted.org/packages/source/" + (string-take name 1) "/" name "/" + name "-" version extension)) + diff --git a/gnu/packages/xml.scm b/gnu/packages/xml.scm index 930cd795f5..2af62d59df 100644 --- a/gnu/packages/xml.scm +++ b/gnu/packages/xml.scm @@ -64,6 +64,7 @@ #:use-module (gnu packages perl) #:use-module (gnu packages perl-check) #:use-module (gnu packages python) + #:use-module (gnu packages python-xyz) #:use-module (gnu packages tls) #:use-module (gnu packages web) #:use-module ((guix licenses) #:prefix license:) @@ -2391,6 +2392,7 @@ The central program included in this package is @code{onsgmls}, which replaces (base32 "15h7d41v48q31hzjay7qzixdv531hnga3h35hksk7x52pgqcrkz7")))) (build-system python-build-system) + (native-inputs `(("python-setuptools" ,python-setuptools))) (home-page "https://github.com/sissaschool/elementpath") (synopsis @@ -2416,13 +2418,20 @@ because lxml.etree already has it's own implementation of XPath 1.0.") (base32 "01nvb5j8vs9nk4z5s3250b1m22b4d08kffa36if3g1mdygdrvxpg")))) (build-system python-build-system) (arguments - `(#:phases (modify-phases %standard-phases - (replace 'check + `(#:phases (modify-phases %standard-phases + ;; Expects setup.py to be called from source root. + (add-after 'unpack 'patch-get-base-dir (lambda _ - (invoke "make" "test")))))) + (substitute* "versioninfo.py" + (("os.path.abspath\\(os.path.dirname\\(sys.argv\\[0\\]\\)\\)") "'.'")) + #t)) + (replace 'check + (lambda _ + (invoke "make" "test")))))) (inputs `(("libxml2" ,libxml2) ("libxslt" ,libxslt))) + (native-inputs `(("python-setuptools" ,python-setuptools))) (home-page "https://lxml.de/") (synopsis "Python XML processing library") (description diff --git a/guix/build-system/python.scm b/guix/build-system/python.scm index e39c06528e..6a9f40faa4 100644 --- a/guix/build-system/python.scm +++ b/guix/build-system/python.scm @@ -70,6 +70,16 @@ extension, such as '.tar.gz'." (let ((python (resolve-interface '(gnu packages python)))) (module-ref python 'python-2))) +(define (default-python-build) + "Return the default Python 2 package." + (let ((python (resolve-interface '(gnu packages python-xyz)))) + (module-ref python 'python-build-bootstrap))) + +(define (default-python-setuptools) + "Return the default Python 2 package." + (let ((python (resolve-interface '(gnu packages python-xyz)))) + (module-ref python 'python-setuptools))) + (define* (package-with-explicit-python python old-prefix new-prefix #:key variant-property) "Return a procedure of one argument, P. The procedure creates a package with @@ -138,11 +148,13 @@ pre-defined variants." (define* (lower name #:key source inputs native-inputs outputs system target (python (default-python)) + (python-build-variant (default-python-build)) + (need-setuptools #t) #:allow-other-keys #:rest arguments) "Return a bag for NAME." (define private-keywords - '(#:source #:target #:python #:inputs #:native-inputs)) + '(#:source #:target #:python #:python-build-variant #:need-setuptools #:inputs #:native-inputs)) (and (not target) ;XXX: no cross-compilation (bag @@ -156,6 +168,15 @@ pre-defined variants." ;; Keep the standard inputs of 'gnu-build-system'. ,@(standard-packages))) (build-inputs `(("python" ,python) + ,@(if python-build-variant + `(("python-build" ,python-build-variant)) + '()) + ;; Most packages still use setuptools and *all* of our + ;; packages assume it’s available, so include it + ;; except when bootstrapping. + ,@(if need-setuptools + `(("python-setuptools" ,(default-python-setuptools))) + '()) ,@native-inputs)) (outputs outputs) (build python-build) @@ -164,8 +185,6 @@ pre-defined variants." (define* (python-build store name inputs #:key (tests? #t) - (test-target "test") - (use-setuptools? #t) (configure-flags ''()) (phases '(@ (guix build python-build-system) %standard-phases)) @@ -191,9 +210,7 @@ provides a 'setup.py' file as its build system." source)) #:configure-flags ,configure-flags #:system ,system - #:test-target ,test-target #:tests? ,tests? - #:use-setuptools? ,use-setuptools? #:phases ,phases #:outputs %outputs #:search-paths ',(map search-path-specification->sexp diff --git a/guix/build/python-build-system.scm b/guix/build/python-build-system.scm index 09bd8465c8..750c2d33f7 100644 --- a/guix/build/python-build-system.scm +++ b/guix/build/python-build-system.scm @@ -104,47 +104,18 @@ ;; and the scripts defined in entry-points will always be created. -(define setuptools-shim - ;; Run setup.py with "setuptools" being imported, which will patch - ;; "distutils". This is needed for packages using "distutils" instead of - ;; "setuptools" since the former does not understand the - ;; "--single-version-externally-managed" flag. - ;; Python code taken from pip 9.0.1 pip/utils/setuptools_build.py - (string-append - "import setuptools, tokenize;__file__='setup.py';" - "f=getattr(tokenize, 'open', open)(__file__);" - "code=f.read().replace('\\r\\n', '\\n');" - "f.close();" - "exec(compile(code, __file__, 'exec'))")) - -(define (call-setuppy command params use-setuptools?) - (if (file-exists? "setup.py") - (begin - (format #t "running \"python setup.py\" with command ~s and parameters ~s~%" - command params) - (if use-setuptools? - (apply invoke "python" "-c" setuptools-shim - command params) - (apply invoke "python" "./setup.py" command params))) - (error "no setup.py found"))) - -(define* (build #:key use-setuptools? #:allow-other-keys) +(define* (build #:key outputs #:allow-other-keys) "Build a given Python package." - (call-setuppy "build" '() use-setuptools?) + ;; XXX: use "wheel" output instead? + (mkdir-p "dist") + (invoke "python" "-m" "build" "--outdir" "dist" "--no-isolation" "--wheel" ".") #t) -(define* (check #:key tests? test-target use-setuptools? #:allow-other-keys) +(define* (check #:key tests? #:allow-other-keys) "Run the test suite of a given Python package." (if tests? - ;; Running `setup.py test` creates an additional .egg-info directory in - ;; build/lib in some cases, e.g. if the source is in a sub-directory - ;; (given with `package_dir`). This will by copied to the output, too, - ;; so we need to remove. - (let ((before (find-files "build" "\\.egg-info$" #:directories? #t))) - (call-setuppy test-target '() use-setuptools?) - (let* ((after (find-files "build" "\\.egg-info$" #:directories? #t)) - (inter (lset-difference string=? after before))) - (for-each delete-file-recursively inter))) + ;; XXX: Choose testing util based on native inputs? + (format #t "fixme") (format #t "test suite not run~%")) #t) @@ -172,18 +143,38 @@ when running checks after installing the package." (if old-path (string-append ":" old-path) ""))) #t)) -(define* (install #:key outputs (configure-flags '()) use-setuptools? - #:allow-other-keys) +(define* (install #:key inputs outputs (configure-flags '()) #:allow-other-keys) "Install a given Python package." - (let* ((out (assoc-ref outputs "out")) - (params (append (list (string-append "--prefix=" out)) - (if use-setuptools? - ;; distutils does not accept these flags - (list "--single-version-externally-managed" - "--root=/") - '()) - configure-flags))) - (call-setuppy "install" params use-setuptools?) + (let* ((site-dir (site-packages inputs outputs)) + (out (assoc-ref outputs "out")) + (wheels (find-files "dist" "\\.whl$"))) + (define (extract file) + ;; Use Python’s zipfile to avoid extra dependency + ;; XXX: have to move data files according to PEP 517 + (invoke "python3" "-m" "zipfile" "-e" file site-dir)) + + (for-each extract wheels) + + ;; for scripts + (mkdir-p (string-append out "/bin")) + (let ((datadirs (find-files site-dir "\\.data$" #:directories? #t))) + (for-each (lambda (d) + (for-each (lambda (f) + (rename-file (string-append d "/scripts/" f) (string-append out "/bin/" (basename f))) + ;; ZIP does not save/restore permissions, make executable, XXX: f might not be a file, but directory with subdirectories + (chmod (string-append out "/bin/" (basename f)) #o755) + (substitute* (string-append out "/bin/" (basename f)) + (("#!python") + (string-append "#!" (assoc-ref inputs "python") "/bin/python")))) + (scandir (string-append d "/scripts") (negate (cut member <> '("." "..")))))) datadirs)) + #t)) + +(define* (compile-bytecode #:key inputs outputs (configure-flags '()) #:allow-other-keys) + "Compile installed byte-code in site-packages." + (let ((site-dir (site-packages inputs outputs))) + ;; XXX: replace 0 with max allowed threads + (invoke "python3" "-m" "compileall" "-j" "0" site-dir) + ;; XXX: We could compile with -O and -OO too here, at the cost of more space. #t)) (define* (wrap #:key inputs outputs #:allow-other-keys) @@ -214,23 +205,6 @@ when running checks after installing the package." bindirs) #t)) -(define* (rename-pth-file #:key name inputs outputs #:allow-other-keys) - "Rename easy-install.pth to NAME.pth to avoid conflicts between packages -installed with setuptools." - ;; Even if the "easy-install.pth" is not longer created, we kept this phase. - ;; There still may be packages creating an "easy-install.pth" manually for - ;; some good reason. - (let* ((out (assoc-ref outputs "out")) - (python (assoc-ref inputs "python")) - (site-packages (string-append out "/lib/python" - (python-version python) - "/site-packages")) - (easy-install-pth (string-append site-packages "/easy-install.pth")) - (new-pth (string-append site-packages "/" name ".pth"))) - (when (file-exists? easy-install-pth) - (rename-file easy-install-pth new-pth)) - #t)) - (define* (ensure-no-mtimes-pre-1980 #:rest _) "Ensure that there are no mtimes before 1980-01-02 in the source tree." ;; Rationale: patch-and-repack creates tarballs with timestamps at the POSIX @@ -244,6 +218,13 @@ installed with setuptools." #t)) #t)) +(define* (set-SOURCE-DATE-EPOCH #:rest _) + "Set the 'SOURCE_DATE_EPOCH' environment variable. This is used by tools +that incorporate timestamps as a way to tell them to use a fixed timestamp. +See https://reproducible-builds.org/specs/source-date-epoch/." + (setenv "SOURCE_DATE_EPOCH" "315619200") ;; python-wheel respects this variable and sets pre-1980 times on files in zip files, which is unsupported + #t) + (define* (enable-bytecode-determinism #:rest _) "Improve determinism of pyc files." ;; Use deterministic hashes for strings, bytes, and datetime objects. @@ -260,14 +241,15 @@ installed with setuptools." (add-after 'unpack 'ensure-no-mtimes-pre-1980 ensure-no-mtimes-pre-1980) (add-after 'ensure-no-mtimes-pre-1980 'enable-bytecode-determinism enable-bytecode-determinism) + (replace 'set-SOURCE-DATE-EPOCH set-SOURCE-DATE-EPOCH) (delete 'bootstrap) (delete 'configure) ;not needed (replace 'build build) (delete 'check) ;moved after the install phase (replace 'install install) (add-after 'install 'check check) - (add-after 'install 'wrap wrap) - (add-before 'strip 'rename-pth-file rename-pth-file))) + (add-before 'check 'compile-bytecode compile-bytecode) + (add-after 'install 'wrap wrap))) (define* (python-build #:key inputs (phases %standard-phases) #:allow-other-keys #:rest args) --nHGOwjDfkGutFLan--