Questions regarding Python packaging

Questions regarding Python packaging

[Tanguy Le Carrour]

Hi Guix!

I have some general questions regarding Python packaging, that are not
directly related to the "poetry build system" I'm currently working on.

I've just learned, by accident (working on `python-keyring` [1]), that
`python setup.py install` was somehow deprecated in favor of tools like
`pep517` or `build`.

So, I've tried packaging `python-keyring` with those two…

`pep517` keeps on trying to download dependencies, which won't work.

`build` crashes with "ZIP does not support timestamps before 1980",
which, I guess is related to the fact that everything in the store is
timestamped to January 1st 1970.

Does anyone have a opinion on Python packaging and how it should be done?
Any idea how I can circumvent the timestamps problem? Is this fish too
big for me?!

Any help or advice welcome! Thanks!

-- 
Tanguy

[1]: https://github.com/jaraco/keyring/issues/469
     Keyring package version is set to 0.0.0, this might be related to
     the fact that, upstream, they build it with `python -m pep517.build .`,
     not with `python setup.py install`… but it could also not be
     related at all! But in order to be sure, I have to try!

Re: Questions regarding Python packaging

[Leo Famulari]

On Sun, Nov 08, 2020 at 03:27:17PM +0100, Tanguy Le Carrour wrote:
> `pep517` keeps on trying to download dependencies, which won't work.

That usually means that the software is missing some dependencies. If
you think they should be available in the build environment,
double-check that they are, and then look into how the software is
looking for them.

Also, Python software seems to change a lot, and quickly — make sure you
are using the correct version of all the dependencies.

> `build` crashes with "ZIP does not support timestamps before 1980",
> which, I guess is related to the fact that everything in the store is
> timestamped to January 1st 1970.

Are you using the python-build-system? It should handle this problem
with its 'ensure-no-mtimes-pre-1980' phase.

> Does anyone have a opinion on Python packaging and how it should be done?
> Any idea how I can circumvent the timestamps problem? Is this fish too
> big for me?!

In general, Python packaging is on the easy side but, when it gets
complicated (e.g. due to unusual design of the upstream build scripts),
it can be really complicated.

> Any help or advice welcome! Thanks!

If will help if you share your package definition.

Re: Questions regarding Python packaging

[Tanguy Le Carrour]

Hi Leo,


Le 11/08, Leo Famulari a écrit :
> On Sun, Nov 08, 2020 at 03:27:17PM +0100, Tanguy Le Carrour wrote:
> > `pep517` keeps on trying to download dependencies, which won't work.
>
> That usually means that the software is missing some dependencies. If
> you think they should be available in the build environment,
> double-check that they are, and then look into how the software is
> looking for them.

I'll check and double check!


> > `build` crashes with "ZIP does not support timestamps before 1980",
> > which, I guess is related to the fact that everything in the store is
> > timestamped to January 1st 1970.
> 
> Are you using the python-build-system? It should handle this problem
> with its 'ensure-no-mtimes-pre-1980' phase.

Good to know, thanks! I'll read more about this 'ensure-no-mtimes-pre-1980'
phase.


> If will help if you share your package definition.

I'll work a bit more on it based on your answers guys and post a WIP to
guix-devel.

Regards

-- 
Tanguy

Re: Questions regarding Python packaging

[Michael Rohleder]

[-- Attachment #1: Type: text/plain, Size: 994 bytes --]

Hello Tanguy,

Tanguy Le Carrour <tanguy@bioneland.org> writes:
> I've just learned, by accident (working on `python-keyring` [1]), that
> `python setup.py install` was somehow deprecated in favor of tools like
> `pep517` or `build`.
>
> So, I've tried packaging `python-keyring` with those two…

apropos python-keyring:

I also tried to update that package (in order to package "pantalaimon"
which needs this) [1]
The problem with this thing is, it wants to install an egginfo with
version 0.0.0, no matter if build with setup.py or poetry [2]

I tried debugging^Whacking this a while (eg. updateing setuptools, as
suggested in the ticket), but couldn't make it work.
If you find a solution, I'd be interested ;)


Footnotes:
[1] https://rohleder.de/gitweb/?p=guix.git;a=blob;f=mroh/guix/packages/python.scm;h=0c847795375ce08ca878727f15fd1a1b156df6f7;hb=HEAD#l533
[2] https://github.com/jaraco/keyring/issues/469

-- 
There are no winners in life, only survivors.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 511 bytes --]

Re: Questions regarding Python packaging

[Tanguy Le Carrour]

Hi Michael,


Le 11/08, Michael Rohleder a écrit :
> Tanguy Le Carrour <tanguy@bioneland.org> writes:
> > I've just learned, by accident (working on `python-keyring` [1]), that
> > `python setup.py install` was somehow deprecated in favor of tools like
> > `pep517` or `build`.
> >
> > So, I've tried packaging `python-keyring` with those two…
>
> apropos python-keyring:
>
> I also tried to update that package (in order to package "pantalaimon"
> which needs this) [1]
> The problem with this thing is, it wants to install an egginfo with
> version 0.0.0, no matter if build with setup.py or poetry [2]
>
> I tried debugging^Whacking this a while (eg. updateing setuptools, as
> suggested in the ticket), but couldn't make it work.
> If you find a solution, I'd be interested ;)
>
> Footnotes:
> [1] https://rohleder.de/gitweb/?p=guix.git;a=blob;f=mroh/guix/packages/python.scm;h=0c847795375ce08ca878727f15fd1a1b156df6f7;hb=HEAD#l533
> [2] https://github.com/jaraco/keyring/issues/469

Good to know that I'm not alone, thanks. I'll keep you posted!

-- 
Tanguy

Re: Questions regarding Python packaging

[Hartmut Goebel]

Hi,

seems like another messages of mine, regarding the first thread  about a 
poetry build system, did not make it to the list.

Am 08.11.20 um 15:27 schrieb Tanguy Le Carrour:
> I've just learned, by accident (working on `python-keyring` [1]), that
> `python setup.py install` was somehow deprecated

This statement is not exactly true - well, depending on interpretation 
of "somehow". I've not set seen an official deprecation.

It's true that users are encouraged to use pip for installing packages. 
But the official Python Packaging Tutorial [1] is still based on 
setuptools (not even recommending a setup.cfg file). Thus setuptools 
will be around for quite some more time, as will "python setup.py install".

In the future Python world, any build-tool can be specified in 
"pyproject.toml". User will then call "pip install", and pip will 
(AFAIU) call a Python function (aka entry-point) specified in that file. 
(If this file does not exist, setuptools are assumed). For our 
python-build-system, we would use "pip wheel" (for phase build) and "pip 
install" (for phase install).

So, if we switch to "pip wheel" and "pip install", different python 
build systems could share a common base, just redefining some 
dependencies (setuptools, poetry, build, ...) and some tool-dependent 
flags. Is this the direction you are working towards?

[1] https://packaging.python.org/tutorials/packaging-projects/


> in favor of tools like`pep517` or `build`.

Thanks for point to these, both are new to me.

"build" sounds interesting, esp. for guix: "It is a simple build tool 
and does not perform any dependency management." This would help us 
spliting dependency management and build phase. Anyhow, it's quite new 
(half an year old) and implements a PEP 517 package builder - and PEP 
517 (defining the build system in pyproject.toml) is not yet adopted widely.

"pep517" seems o be a library used for "build". Its high-level interface 
has been deprecated in favor for "build".

I as just about to write "So, while this might be one road to go, this 
is not of much use for us yet.". Anyhow, this might be a good base for 
pep517 based packages. On the other hand: Maybe we'd better stick with 
"pip wheel" and "pip install", see above.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Questions regarding Python packaging

[Tanguy Le Carrour]

Hi Hartmut,


Le 11/09, Hartmut Goebel a écrit :
> seems like another messages of mine, regarding the first thread  about a
> poetry build system, did not make it to the list.
> 
> Am 08.11.20 um 15:27 schrieb Tanguy Le Carrour:
> > I've just learned, by accident (working on `python-keyring` [1]), that
> > `python setup.py install` was somehow deprecated
> 
> This statement is not exactly true - well, depending on interpretation of
> "somehow". I've not set seen an official deprecation.

Neither did I! But things are changing (fast) and it seems that I'm
always the last one to know! ^_^'


> It's true that users are encouraged to use pip for installing packages. But
> the official Python Packaging Tutorial [1] is still based on setuptools (not
> even recommending a setup.cfg file). Thus setuptools will be around for
> quite some more time, as will "python setup.py install".
> 
> [1] https://packaging.python.org/tutorials/packaging-projects/
>
> In the future Python world, any build-tool can be specified in
> "pyproject.toml". User will then call "pip install", and pip will (AFAIU)
> call a Python function (aka entry-point) specified in that file. (If this
> file does not exist, setuptools are assumed). For our python-build-system,
> we would use "pip wheel" (for phase build) and "pip install" (for phase
> install).
> 
> So, if we switch to "pip wheel" and "pip install", different python build
> systems could share a common base, just redefining some dependencies
> (setuptools, poetry, build, ...) and some tool-dependent flags. Is this the
> direction you are working towards?

I cannot say that, at the moment, I'm working in any particular direction!
… I'm just trying to make "it" work and learn a bit about Guix packaging
and Python along the way.

Actually, I'm not even yet sure that Poetry needs a dedicated build system,
as it also relies on a build-backend (defined in `pyproject.toml`) which
just happened to be `poetry.core.masonry`, but could be another one… I
guess, I'm not sure yet.

So, definitively a WIP!


> > in favor of tools like`pep517` or `build`.
> 
> Thanks for point to these, both are new to me.
> 
> "build" sounds interesting, esp. for guix: "It is a simple build tool and
> does not perform any dependency management." This would help us spliting
> dependency management and build phase. Anyhow, it's quite new (half an year
> old) and implements a PEP 517 package builder - and PEP 517 (defining the
> build system in pyproject.toml) is not yet adopted widely.
> 
> "pep517" seems o be a library used for "build". Its high-level interface has
> been deprecated in favor for "build".
> 
> I as just about to write "So, while this might be one road to go, this is
> not of much use for us yet.". Anyhow, this might be a good base for pep517
> based packages. On the other hand: Maybe we'd better stick with "pip wheel"
> and "pip install", see above.

+1! I'll try to make those packages who need a special build system
(which might be the case for `keyring`) work and see for a more
general "new" `python-build-system` later! And if I happen to learn
something on the way… great! :-)

Regards

-- 
Tanguy

Re: Questions regarding Python packaging

[Hartmut Goebel]

Hi Tanguy,

Am 10.11.20 um 09:47 schrieb Tanguy Le Carrour:

> Actually, I'm not even yet sure that Poetry needs a dedicated build system,
> as it also relies on a build-backend (defined in `pyproject.toml`) which
> just happened to be `poetry.core.masonry`, but could be another one… I
> guess, I'm not sure yet.

This sounds like it is meant to be build and installed using one of the 
PEP-517 compatible tools, e.g. "pip wheel" & "pip install".


> +1! I'll try to make those packages who need a special build system
> (which might be the case for `keyring`) work and see for a more
> general "new" `python-build-system` later! And if I happen to learn
> something on the way… great! :-)

Sounds good :-)

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Questions regarding Python packaging

[Lars-Dominik Braun]

Hi Tanguy,

> So, I've tried packaging `python-keyring` with those two…
> 
> `pep517` keeps on trying to download dependencies, which won't work.
> 
> `build` crashes with "ZIP does not support timestamps before 1980",
> which, I guess is related to the fact that everything in the store is
> timestamped to January 1st 1970.
have you been looking into a python-build-system using `build`[1]? I’ve
had the same issue with egg versions set to 0.0.0 and think in the long
run moving to a PEP 517-style build is the way forward.

Cheers,
Lars

[1] https://github.com/pypa/build

Re: Questions regarding Python packaging

[Tanguy LE CARROUR]

Hi Lars,


Excerpts from Lars-Dominik Braun's message of January 5, 2021 11:25 am:
> Hi Tanguy,
> 
>> So, I've tried packaging `python-keyring` with those two…
>> 
>> `pep517` keeps on trying to download dependencies, which won't work.
>> 
>> `build` crashes with "ZIP does not support timestamps before 1980",
>> which, I guess is related to the fact that everything in the store is
>> timestamped to January 1st 1970.
> have you been looking into a python-build-system using `build`[1]? I’ve
> had the same issue with egg versions set to 0.0.0 and think in the long
> run moving to a PEP 517-style build is the way forward.

I agree! Unfortunately, I haven't had much time (so far) to work on it! :-(

I'll revive the thread as soon as I've made progress…

Regards!

-- 
Tanguy

Re: Questions regarding Python packaging

[Tanguy LE CARROUR]

Hi Guix, hi Lars,


Excerpts from Tanguy LE CARROUR's message of January 6, 2021 4:32 pm:
> Excerpts from Lars-Dominik Braun's message of January 5, 2021 11:25 am:
>>> So, I've tried packaging `python-keyring` with those two…
>>> 
>>> `pep517` keeps on trying to download dependencies, which won't work.
>>> 
>>> `build` crashes with "ZIP does not support timestamps before 1980",
>>> which, I guess is related to the fact that everything in the store is
>>> timestamped to January 1st 1970.
>> have you been looking into a python-build-system using `build`[1]? I’ve
>> had the same issue with egg versions set to 0.0.0 and think in the long
>> run moving to a PEP 517-style build is the way forward.
> 
> I agree! Unfortunately, I haven't had much time (so far) to work on it! :-(
> 
> I'll revive the thread as soon as I've made progress…

Done! :-)
I've eventually succeeded in ("properly") packaging a software managed
with Poetry. And I've learned quite a lot on the way!

Following Efraim's presentation on Guix Day, I've added a `guix.scm`
file to one of my pet projects. Any Guix user should be able to build it.

```bash
$ git clone https://gitlab.com/TanguyEE/nowty.git
Cloning into 'nowty'...
[…]

$ cd nowty/
$ guix build --file=guix.scm
[…]
successfully built /gnu/store/[…]-nowty-0.1.0a0+2d656ef.drv
/gnu/store/[…]-nowty-0.1.0a0+2d656ef

$ /gnu/store/[…]-nowty-0.1.0a0+2d656ef/bin/nowty
'High Hopes' from album 'Pray for the Wicked' by 'Panic! at the Disco'.
```

Regarding the package definition:
- the `build` phase updates the package version in `pyproject.toml` ;
- the `install` phase uses `pip` which in turn uses the build system
  from `poetry-core`.

I know that the phase names don't make much sense! … but it's a
work in progress ! :-)

Any comment welcome.

Regards,

-- 
Tanguy

Re: Questions regarding Python packaging

[Lars-Dominik Braun]

[-- Attachment #1: Type: text/plain, Size: 793 bytes --]

Hi Tanguy,

> Done! :-)
> I've eventually succeeded in ("properly") packaging a software managed
> with Poetry. And I've learned quite a lot on the way!
oh, I see. I’ve actually been trying to replace python-build-system with
a python-build based build. Attached is my current work in progress. I
cannot quite build python-build, because I’m lacking support for
python-flit, but I think the general idea is clear: Remove pip and
setuptools from python (saves almost 20MiB from the closure and avoids
weird conflicts between python’s setuptools and python-setuptools) and
turn them into (almost) ordinary packages. Then use setuptools to
bootstrap itself, bootstrap python-build with setuptools and use
python-build to build evrey other packages using python-build-system.

Cheers,
Lars


[-- Attachment #2: wip-pep-517-build.diff --]
[-- Type: text/plain, Size: 30148 bytes --]

diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm
index d08e23936c..a9843bd9cb 100644
--- a/gnu/packages/python-xyz.scm
+++ b/gnu/packages/python-xyz.scm
@@ -1123,14 +1123,14 @@ other machines, such as over the network.")
 (define-public python-setuptools
   (package
     (name "python-setuptools")
-    (version "41.0.1")
+    (version "51.3.1")
     (source
      (origin
       (method url-fetch)
-      (uri (pypi-uri "setuptools" version ".zip"))
+      (uri (pypi-uri "setuptools" version))
       (sha256
        (base32
-        "04sns22y2hhsrwfy1mha2lgslvpjsjsz8xws7h2rh5a7ylkd28m2"))
+        "07qk6ykhq1q4k33x1m96r39f9qpfzp2xfl5ly9zww0180gssrvqc"))
       (modules '((guix build utils)))
       (snippet
        '(begin
@@ -1143,7 +1143,26 @@ other machines, such as over the network.")
     ;; FIXME: Tests require pytest, which itself relies on setuptools.
     ;; One could bootstrap with an internal untested setuptools.
     (arguments
-     `(#:tests? #f))
+     `(#:tests? #f
+       #:need-setuptools #f
+       #:python-build-variant #f
+       #:phases (modify-phases %standard-phases
+                  (replace 'build
+                           (lambda _
+                             (invoke "python" "setup.py" "build")))
+                  (replace 'install
+                           (lambda* (#:key outputs #:allow-other-keys)
+                             (let ((out (assoc-ref outputs "out")))
+                             (invoke "python" "setup.py" "install" "--prefix" out "--single-version-externally-managed" "--root=/"))))
+                  ;; Use this setuptools’ sources to bootstrap themselves.
+                  (add-before 'build 'set-PYTHONPATH
+                    (lambda _
+                      (format #t "current working dir ~s~%" (getcwd))
+                      (setenv "PYTHONPATH"
+                              (string-append ".:" (getenv "PYTHONPATH")))
+                      #t)))))
+    ;; Not required when not building a wheel
+    ;(propagated-inputs `(("python-wheel" ,python-wheel)))
     (home-page "https://pypi.org/project/setuptools/")
     (synopsis
      "Library designed to facilitate packaging Python projects")
@@ -1822,7 +1841,7 @@ Python file, so it can be easily copied into your project.")
   (package
     (inherit python-six)
     (name "python-six-bootstrap")
-    (native-inputs `())
+    (native-inputs '())
     (arguments `(#:tests? #f))))
 
 (define-public python2-six-bootstrap
@@ -5675,7 +5694,8 @@ by pycodestyle.")
              ;; NOTE: Any value works, the variable just has to be present.
              (setenv "SKIP_ONLINE" "1")
              #t)))))
-    (native-inputs `(("unzip" ,unzip)))
+    (native-inputs
+      `(("unzip" ,unzip)))
     (home-page "https://bitbucket.org/pypa/distlib")
     (synopsis "Distribution utilities")
     (description "Distlib is a library which implements low-level functions that
@@ -15272,7 +15292,7 @@ protocols.")
   (package
     (inherit python-attrs)
     (name "python-attrs-bootstrap")
-    (native-inputs `())
+    (native-inputs '())
     (arguments `(#:tests? #f))))
 
 (define-public python2-attrs-bootstrap
@@ -23451,3 +23471,292 @@ Qt applications.")
       "Pivy provides python bindings for Coin, a 3D graphics library with an
 Application Programming Interface based on the Open Inventor 2.1 API.")
     (license license:isc)))
+
+(define-public python-build
+  (package
+    (name "python-build")
+    (version "0.1.0")
+    (source
+      (origin
+        (method url-fetch)
+        (uri (pypi-uri "build" version))
+        (sha256
+          (base32
+            "1d6m21lijwm04g50nwgsgj7x3vhblzw7jv05ah8psqgzk20bbch8"))))
+    (build-system python-build-system)
+    (arguments `(#:python-build-variant ,python-build-bootstrap))
+    (propagated-inputs
+      `(("python-packaging" ,python-packaging)
+        ("python-pep517" ,python-pep517)
+        ("python-toml" ,python-toml)
+        ;; For fallback option.
+        ("python-setuptools" ,python-setuptools)))
+    (native-inputs
+      `(("python-filelock" ,python-filelock)
+        ("python-pytest" ,python-pytest)
+        ("python-pytest-cov" ,python-pytest-cov)
+        ("python-pytest-mock" ,python-pytest-mock)
+        ("python-pytest-xdist" ,python-pytest-xdist)))
+    (home-page "https://pypa-build.readthedocs.io/")
+    (synopsis
+      "A simple, correct PEP517 package builder")
+    (description
+      "A simple, correct PEP517 package builder")
+    (license license:expat)))
+
+(define-public python-pep517-bootstrap
+               (package
+                 (inherit python-pep517)
+                 (name "python-pep517-bootstrap")
+                 (arguments `(#:tests? #f))
+                 (native-inputs `())))
+
+(define-public python-build-bootstrap
+  (package
+    (name "python-build-bootstrap")
+    (version "0.1.0")
+    (source
+      (origin
+        (method url-fetch)
+        (uri (pypi-uri "build" version))
+        (sha256
+          (base32
+            "1d6m21lijwm04g50nwgsgj7x3vhblzw7jv05ah8psqgzk20bbch8"))))
+    (build-system python-build-system)
+    (arguments
+     `(#:tests? #f
+       #:python-build-variant #f
+       #:phases (modify-phases %standard-phases
+                  (replace 'build
+                           (lambda _
+                             (invoke "python" "setup.py" "build")))
+                  (replace 'install
+                           (lambda* (#:key outputs #:allow-other-keys)
+                             (let ((out (assoc-ref outputs "out")))
+                             (invoke "python" "setup.py" "install" "--prefix" out "--single-version-externally-managed" "--root=/")))))))
+    (propagated-inputs
+      `(("python-pep517" ,python-pep517-source-bootstrap)
+        ("python-packaging" ,python-packaging-source-bootstrap)))
+    (home-page "https://pypa-build.readthedocs.io/")
+    (synopsis
+      "A simple, correct PEP517 package builder")
+    (description
+      "A simple, correct PEP517 package builder")
+    (license license:expat)))
+
+(define-public python-pep517-source-bootstrap
+  (package
+    (name "python-pep517-source-bootstrap")
+    (version "0.9.1")
+    (source
+      (origin
+        (method url-fetch)
+        (uri (pypi-uri "pep517" version))
+        (sha256
+         (base32
+          "0zqidxah03qpnp6zkg3zd1kmd5f79hhdsfmlc0cldaniy80qddxf"))))
+    (build-system python-build-system)
+    (arguments
+     `(#:tests? #f
+       #:python-build-variant #f
+       #:phases (modify-phases %standard-phases
+(add-after 'unpack 'patch
+           (lambda _
+             (substitute* "setup.py"
+               (("distutils\\.core") "setuptools"))
+             #t))
+                  (replace 'build
+                           (lambda _
+                             (invoke "python" "setup.py" "build")))
+                  (replace 'install
+                           (lambda* (#:key outputs #:allow-other-keys)
+                             (let ((out (assoc-ref outputs "out")))
+                             (invoke "python" "setup.py" "install" "--prefix" out "--single-version-externally-managed" "--root=/")))))))
+    (propagated-inputs
+     `(("python-toml" ,python-toml-source-bootstrap)
+       ("python-wheel" ,python-wheel-bootstrap)
+       ))
+    (home-page "https://github.com/pypa/pep517")
+    (synopsis "Wrappers to build Python packages using PEP 517 hooks")
+    (description
+     "Wrappers to build Python packages using PEP 517 hooks.")
+    (license license:expat)))
+
+(define-public python-toml-source-bootstrap
+  (package
+    (name "python-toml-source-bootstrap")
+    (version "0.10.1")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (pypi-uri "toml" version))
+       (sha256
+        (base32
+         "03wbqm5cn685cwx2664hjdpz370njl7lf0yal8s0dkp5w4mn2swj"))))
+    (build-system python-build-system)
+    (arguments
+     `(#:tests? #f
+       #:python-build-variant #f
+       #:phases (modify-phases %standard-phases
+                  (replace 'build
+                           (lambda _
+                             (invoke "python" "setup.py" "build")))
+                  (replace 'install
+                           (lambda* (#:key outputs #:allow-other-keys)
+                             (let ((out (assoc-ref outputs "out")))
+                             (invoke "python" "setup.py" "install" "--prefix" out "--single-version-externally-managed" "--root=/")))))))
+    (home-page "https://github.com/uiri/toml")
+    (synopsis "Library for TOML")
+    (description
+     "@code{toml} is a library for parsing and creating Tom's Obvious, Minimal
+Language (TOML) configuration files.")
+    (license license:expat)))
+
+(define-public python-packaging-source-bootstrap
+  (package
+    (name "python-packaging-source-bootstrap")
+    (version "20.0")
+    (source
+      (origin
+        (method url-fetch)
+        (uri (pypi-uri "packaging" version))
+        ;; XXX: The URL in the patch file is wrong, it should be
+        ;; <https://github.com/pypa/packaging/pull/256>.
+        (patches (search-patches "python-packaging-test-arch.patch"))
+        (sha256
+         (base32
+          "1y2ip3a4ykkpgnwgn85j6hkspcl0cg3mzms97f40mk57vwqq67gy"))))
+    (build-system python-build-system)
+    (arguments
+     `(#:tests? #f
+       #:python-build-variant #f
+       #:phases (modify-phases %standard-phases
+                  (replace 'build
+                           (lambda _
+                             (invoke "python" "setup.py" "build")))
+                  (replace 'install
+                           (lambda* (#:key outputs #:allow-other-keys)
+                             (let ((out (assoc-ref outputs "out")))
+                             (invoke "python" "setup.py" "install" "--prefix" out "--single-version-externally-managed" "--root=/")))))))
+    (propagated-inputs
+     `(("python-pyparsing" ,python-pyparsing-bootstrap)
+       ("python-six" ,python-six-bootstrap2)
+       ))
+    (home-page "https://github.com/pypa/packaging")
+    (synopsis "Core utilities for Python packages")
+    (description "Packaging is a Python module for dealing with Python packages.
+It offers an interface for working with package versions, names, and dependency
+information.")
+    ;; From 'LICENSE': This software is made available under the terms of
+    ;; *either* of the licenses found in LICENSE.APACHE or LICENSE.BSD.
+    ;; Contributions to this software is made under the terms of *both* these
+    ;; licenses.
+    (license (list license:asl2.0 license:bsd-2))))
+
+(define-public python-pyparsing-bootstrap
+  (package
+    (name "python-pyparsing-bootstrap")
+    (version "2.4.6")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (pypi-uri "pyparsing" version))
+       (sha256
+        (base32 "17wn5zlijc9m9zj26gy3f541y7smpj8rfhl51d025c2gm210b0sc"))))
+    (build-system python-build-system)
+    (arguments
+     `(#:tests? #f
+       #:python-build-variant #f
+       #:phases (modify-phases %standard-phases
+                  (replace 'build
+                           (lambda _
+                             (invoke "python" "setup.py" "build")))
+                  (replace 'install
+                           (lambda* (#:key outputs #:allow-other-keys)
+                             (let ((out (assoc-ref outputs "out")))
+                             (invoke "python" "setup.py" "install" "--prefix" out "--single-version-externally-managed" "--root=/")))))))
+    (home-page "https://github.com/pyparsing/pyparsing")
+    (synopsis "Python parsing class library")
+    (description
+     "The pyparsing module is an alternative approach to creating and
+executing simple grammars, vs. the traditional lex/yacc approach, or the use
+of regular expressions.  The pyparsing module provides a library of classes
+that client code uses to construct the grammar directly in Python code.")
+    (license license:expat)))
+
+(define-public python-six-bootstrap2
+  (package
+    (name "python-six-bootstrap")
+    (version "1.14.0")
+    (source
+     (origin
+      (method url-fetch)
+      (uri (pypi-uri "six" version))
+      (sha256
+       (base32
+        "02lw67hprv57hyg3cfy02y3ixjk3nzwc0dx3c4ynlvkfwkfdnsr3"))))
+    (build-system python-build-system)
+    (arguments
+     `(#:tests? #f
+       #:python-build-variant #f
+       #:phases (modify-phases %standard-phases
+                  (replace 'build
+                           (lambda _
+                             (invoke "python" "setup.py" "build")))
+                  (replace 'install
+                           (lambda* (#:key outputs #:allow-other-keys)
+                             (let ((out (assoc-ref outputs "out")))
+                             (invoke "python" "setup.py" "install" "--prefix" out "--single-version-externally-managed" "--root=/")))))))
+    (home-page "https://pypi.org/project/six/")
+    (synopsis "Python 2 and 3 compatibility utilities")
+    (description
+     "Six is a Python 2 and 3 compatibility library.  It provides utility
+functions for smoothing over the differences between the Python versions with
+the goal of writing Python code that is compatible on both Python versions.
+Six supports every Python version since 2.5.  It is contained in only one
+Python file, so it can be easily copied into your project.")
+    (license license:x11)))
+
+(define-public python-wheel-bootstrap
+  (package
+    (name "python-wheel-bootstrap")
+    (version "0.33.6")
+    (source
+      (origin
+        (method url-fetch)
+        (uri (pypi-uri "wheel" version))
+        (sha256
+         (base32
+          "0ii6f34rvpjg3nmw4bc2h7fhdsy38y1h93hghncfs5akfrldmj8h"))))
+    (build-system python-build-system)
+    (arguments
+     `(#:tests? #f
+       #:python-build-variant #f
+       #:phases (modify-phases %standard-phases
+(add-after 'unpack 'patch-enable-entrypoints
+           (lambda _
+             ;; python-wheel tells setuptools to not install entry point scripts
+             ;; by default. Stop doing that, so wheels built contain all data
+             ;; required.
+             (substitute* "wheel/bdist_wheel.py"
+               (("(install_scripts\\.no_ep = )True" all assignment) (string-append assignment "False")))
+             #t))
+                  (replace 'build
+                           (lambda _
+                             (invoke "python" "setup.py" "build")))
+                  (replace 'install
+                           (lambda* (#:key outputs #:allow-other-keys)
+                             (let ((out (assoc-ref outputs "out")))
+                             (invoke "python" "setup.py" "install" "--prefix" out "--single-version-externally-managed" "--root=/")))))))
+    (home-page "https://bitbucket.org/pypa/wheel/")
+    (synopsis "Format for built Python packages")
+    (description
+     "A wheel is a ZIP-format archive with a specially formatted filename and
+the @code{.whl} extension.  It is designed to contain all the files for a PEP
+376 compatible install in a way that is very close to the on-disk format.  Many
+packages will be properly installed with only the @code{Unpack} step and the
+unpacked archive preserves enough information to @code{Spread} (copy data and
+scripts to their final locations) at any later time.  Wheel files can be
+installed with a newer @code{pip} or with wheel's own command line utility.")
+    (license license:expat)))
+
diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index fa9bf10e07..57221a91ea 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -100,6 +100,7 @@
   #:use-module (guix utils)
   #:use-module (guix build-system gnu)
   #:use-module (guix build-system trivial)
+  #:use-module (guix build-system copy)
   #:use-module (srfi srfi-1)
   #:use-module (srfi srfi-26))
 
@@ -153,7 +154,7 @@
        (list "--enable-shared"                    ;allow embedding
              "--with-system-expat"                ;for XML support
              "--with-system-ffi"                  ;build ctypes
-             "--with-ensurepip=install"           ;install pip and setuptools
+             "--with-ensurepip=no"                ;do not install pip and setuptools
              "--enable-unicode=ucs4"
 
              ;; Prevent the installed _sysconfigdata.py from retaining a reference
@@ -874,4 +875,12 @@ Python code natively, including C extensions.")
                    license:bsd-3 ; lib_pypy/cffi/_pycparser/ply/
                    (license:non-copyleft
                     "http://www.unicode.org/copyright.html")))))
+(define* (pypi-uri name version #:optional (extension ".tar.gz"))
+  "Return a URI string for the Python package hosted on the Python Package
+Index (PyPI) corresponding to NAME and VERSION.  EXTENSION is the file name
+extension, such as '.tar.gz'."
+  (string-append "https://files.pythonhosted.org/packages/source/"
+                 (string-take name 1) "/" name "/"
+                 name "-" version extension))
+
 
diff --git a/gnu/packages/xml.scm b/gnu/packages/xml.scm
index 930cd795f5..2af62d59df 100644
--- a/gnu/packages/xml.scm
+++ b/gnu/packages/xml.scm
@@ -64,6 +64,7 @@
   #:use-module (gnu packages perl)
   #:use-module (gnu packages perl-check)
   #:use-module (gnu packages python)
+  #:use-module (gnu packages python-xyz)
   #:use-module (gnu packages tls)
   #:use-module (gnu packages web)
   #:use-module ((guix licenses) #:prefix license:)
@@ -2391,6 +2392,7 @@ The central program included in this package is @code{onsgmls}, which replaces
         (base32
          "15h7d41v48q31hzjay7qzixdv531hnga3h35hksk7x52pgqcrkz7"))))
     (build-system python-build-system)
+    (native-inputs `(("python-setuptools" ,python-setuptools)))
     (home-page
      "https://github.com/sissaschool/elementpath")
     (synopsis
@@ -2416,13 +2418,20 @@ because lxml.etree already has it's own implementation of XPath 1.0.")
         (base32 "01nvb5j8vs9nk4z5s3250b1m22b4d08kffa36if3g1mdygdrvxpg"))))
     (build-system python-build-system)
     (arguments
-     `(#:phases (modify-phases %standard-phases
-                  (replace 'check
+      `(#:phases (modify-phases %standard-phases
+                                ;; Expects setup.py to be called from source root.
+         (add-after 'unpack 'patch-get-base-dir
                     (lambda _
-                      (invoke "make" "test"))))))
+                      (substitute* "versioninfo.py"
+                                   (("os.path.abspath\\(os.path.dirname\\(sys.argv\\[0\\]\\)\\)") "'.'"))
+                      #t))
+         (replace 'check
+                  (lambda _
+                    (invoke "make" "test"))))))
     (inputs
      `(("libxml2" ,libxml2)
        ("libxslt" ,libxslt)))
+    (native-inputs `(("python-setuptools" ,python-setuptools)))
     (home-page "https://lxml.de/")
     (synopsis "Python XML processing library")
     (description
diff --git a/guix/build-system/python.scm b/guix/build-system/python.scm
index e39c06528e..6a9f40faa4 100644
--- a/guix/build-system/python.scm
+++ b/guix/build-system/python.scm
@@ -70,6 +70,16 @@ extension, such as '.tar.gz'."
   (let ((python (resolve-interface '(gnu packages python))))
     (module-ref python 'python-2)))
 
+(define (default-python-build)
+  "Return the default Python 2 package."
+  (let ((python (resolve-interface '(gnu packages python-xyz))))
+    (module-ref python 'python-build-bootstrap)))
+
+(define (default-python-setuptools)
+  "Return the default Python 2 package."
+  (let ((python (resolve-interface '(gnu packages python-xyz))))
+    (module-ref python 'python-setuptools)))
+
 (define* (package-with-explicit-python python old-prefix new-prefix
                                        #:key variant-property)
   "Return a procedure of one argument, P.  The procedure creates a package with
@@ -138,11 +148,13 @@ pre-defined variants."
 (define* (lower name
                 #:key source inputs native-inputs outputs system target
                 (python (default-python))
+                (python-build-variant (default-python-build))
+				(need-setuptools #t)
                 #:allow-other-keys
                 #:rest arguments)
   "Return a bag for NAME."
   (define private-keywords
-    '(#:source #:target #:python #:inputs #:native-inputs))
+    '(#:source #:target #:python #:python-build-variant #:need-setuptools #:inputs #:native-inputs))
 
   (and (not target)                               ;XXX: no cross-compilation
        (bag
@@ -156,6 +168,15 @@ pre-defined variants."
                         ;; Keep the standard inputs of 'gnu-build-system'.
                         ,@(standard-packages)))
          (build-inputs `(("python" ,python)
+                         ,@(if python-build-variant
+							`(("python-build" ,python-build-variant))
+							'())
+						 ;; Most packages still use setuptools and *all* of our
+						 ;; packages assume it’s available, so include it
+						 ;; except when bootstrapping.
+						 ,@(if need-setuptools
+							 `(("python-setuptools" ,(default-python-setuptools)))
+							 '())
                          ,@native-inputs))
          (outputs outputs)
          (build python-build)
@@ -164,8 +185,6 @@ pre-defined variants."
 (define* (python-build store name inputs
                        #:key
                        (tests? #t)
-                       (test-target "test")
-                       (use-setuptools? #t)
                        (configure-flags ''())
                        (phases '(@ (guix build python-build-system)
                                    %standard-phases))
@@ -191,9 +210,7 @@ provides a 'setup.py' file as its build system."
                                   source))
                      #:configure-flags ,configure-flags
                      #:system ,system
-                     #:test-target ,test-target
                      #:tests? ,tests?
-                     #:use-setuptools? ,use-setuptools?
                      #:phases ,phases
                      #:outputs %outputs
                      #:search-paths ',(map search-path-specification->sexp
diff --git a/guix/build/python-build-system.scm b/guix/build/python-build-system.scm
index 09bd8465c8..750c2d33f7 100644
--- a/guix/build/python-build-system.scm
+++ b/guix/build/python-build-system.scm
@@ -104,47 +104,18 @@
 ;; and the scripts defined in entry-points will always be created.
 
 
-(define setuptools-shim
-  ;; Run setup.py with "setuptools" being imported, which will patch
-  ;; "distutils". This is needed for packages using "distutils" instead of
-  ;; "setuptools" since the former does not understand the
-  ;; "--single-version-externally-managed" flag.
-  ;; Python code taken from pip 9.0.1 pip/utils/setuptools_build.py
-  (string-append
-   "import setuptools, tokenize;__file__='setup.py';"
-   "f=getattr(tokenize, 'open', open)(__file__);"
-   "code=f.read().replace('\\r\\n', '\\n');"
-   "f.close();"
-   "exec(compile(code, __file__, 'exec'))"))
-
-(define (call-setuppy command params use-setuptools?)
-  (if (file-exists? "setup.py")
-      (begin
-         (format #t "running \"python setup.py\" with command ~s and parameters ~s~%"
-                command params)
-         (if use-setuptools?
-             (apply invoke "python" "-c" setuptools-shim
-                    command params)
-             (apply invoke "python" "./setup.py" command params)))
-      (error "no setup.py found")))
-
-(define* (build #:key use-setuptools? #:allow-other-keys)
+(define* (build #:key outputs #:allow-other-keys)
   "Build a given Python package."
-  (call-setuppy "build" '() use-setuptools?)
+  ;; XXX: use "wheel" output instead?
+  (mkdir-p "dist")
+  (invoke "python" "-m" "build" "--outdir" "dist" "--no-isolation" "--wheel" ".")
   #t)
 
-(define* (check #:key tests? test-target use-setuptools? #:allow-other-keys)
+(define* (check #:key tests? #:allow-other-keys)
   "Run the test suite of a given Python package."
   (if tests?
-      ;; Running `setup.py test` creates an additional .egg-info directory in
-      ;; build/lib in some cases, e.g. if the source is in a sub-directory
-      ;; (given with `package_dir`). This will by copied to the output, too,
-      ;; so we need to remove.
-      (let ((before (find-files "build" "\\.egg-info$" #:directories? #t)))
-        (call-setuppy test-target '() use-setuptools?)
-        (let* ((after (find-files "build" "\\.egg-info$" #:directories? #t))
-               (inter (lset-difference string=? after before)))
-          (for-each delete-file-recursively inter)))
+    ;; XXX: Choose testing util based on native inputs?
+      (format #t "fixme")
       (format #t "test suite not run~%"))
   #t)
 
@@ -172,18 +143,38 @@ when running checks after installing the package."
                            (if old-path (string-append ":" old-path) "")))
     #t))
 
-(define* (install #:key outputs (configure-flags '()) use-setuptools?
-                  #:allow-other-keys)
+(define* (install #:key inputs outputs (configure-flags '()) #:allow-other-keys)
   "Install a given Python package."
-  (let* ((out (assoc-ref outputs "out"))
-         (params (append (list (string-append "--prefix=" out))
-                         (if use-setuptools?
-                             ;; distutils does not accept these flags
-                             (list "--single-version-externally-managed"
-                                    "--root=/")
-                             '())
-                         configure-flags)))
-    (call-setuppy "install" params use-setuptools?)
+  (let* ((site-dir (site-packages inputs outputs))
+         (out (assoc-ref outputs "out"))
+         (wheels (find-files "dist" "\\.whl$")))
+	(define (extract file)
+      ;; Use Python’s zipfile to avoid extra dependency
+      ;; XXX: have to move data files according to PEP 517
+      (invoke "python3" "-m" "zipfile" "-e" file site-dir))
+
+    (for-each extract wheels)
+
+    ;; for scripts
+    (mkdir-p (string-append out "/bin"))
+	(let ((datadirs (find-files site-dir "\\.data$" #:directories? #t)))
+      (for-each (lambda (d)
+                  (for-each (lambda (f)
+                              (rename-file (string-append d "/scripts/" f) (string-append out "/bin/" (basename f)))
+                              ;; ZIP does not save/restore permissions, make executable, XXX: f might not be a file, but directory with subdirectories
+                              (chmod (string-append out "/bin/" (basename f)) #o755)
+                              (substitute* (string-append out "/bin/" (basename f))
+               (("#!python")
+                (string-append "#!" (assoc-ref inputs "python") "/bin/python"))))
+                            (scandir (string-append d "/scripts") (negate (cut member <> '("." "..")))))) datadirs))
+    #t))
+
+(define* (compile-bytecode #:key inputs outputs (configure-flags '()) #:allow-other-keys)
+  "Compile installed byte-code in site-packages."
+  (let ((site-dir (site-packages inputs outputs)))
+    ;; XXX: replace 0 with max allowed threads
+    (invoke "python3" "-m" "compileall" "-j" "0" site-dir)
+    ;; XXX: We could compile with -O and -OO too here, at the cost of more space.
     #t))
 
 (define* (wrap #:key inputs outputs #:allow-other-keys)
@@ -214,23 +205,6 @@ when running checks after installing the package."
               bindirs)
     #t))
 
-(define* (rename-pth-file #:key name inputs outputs #:allow-other-keys)
-  "Rename easy-install.pth to NAME.pth to avoid conflicts between packages
-installed with setuptools."
-  ;; Even if the "easy-install.pth" is not longer created, we kept this phase.
-  ;; There still may be packages creating an "easy-install.pth" manually for
-  ;; some good reason.
-  (let* ((out (assoc-ref outputs "out"))
-         (python (assoc-ref inputs "python"))
-         (site-packages (string-append out "/lib/python"
-                                       (python-version python)
-                                       "/site-packages"))
-         (easy-install-pth (string-append site-packages "/easy-install.pth"))
-         (new-pth (string-append site-packages "/" name ".pth")))
-    (when (file-exists? easy-install-pth)
-      (rename-file easy-install-pth new-pth))
-    #t))
-
 (define* (ensure-no-mtimes-pre-1980 #:rest _)
   "Ensure that there are no mtimes before 1980-01-02 in the source tree."
   ;; Rationale: patch-and-repack creates tarballs with timestamps at the POSIX
@@ -244,6 +218,13 @@ installed with setuptools."
                #t))
     #t))
 
+(define* (set-SOURCE-DATE-EPOCH #:rest _)
+  "Set the 'SOURCE_DATE_EPOCH' environment variable.  This is used by tools
+that incorporate timestamps as a way to tell them to use a fixed timestamp.
+See https://reproducible-builds.org/specs/source-date-epoch/."
+  (setenv "SOURCE_DATE_EPOCH" "315619200") ;; python-wheel respects this variable and sets pre-1980 times on files in zip files, which is unsupported
+  #t)
+
 (define* (enable-bytecode-determinism #:rest _)
   "Improve determinism of pyc files."
   ;; Use deterministic hashes for strings, bytes, and datetime objects.
@@ -260,14 +241,15 @@ installed with setuptools."
     (add-after 'unpack 'ensure-no-mtimes-pre-1980 ensure-no-mtimes-pre-1980)
     (add-after 'ensure-no-mtimes-pre-1980 'enable-bytecode-determinism
       enable-bytecode-determinism)
+    (replace 'set-SOURCE-DATE-EPOCH set-SOURCE-DATE-EPOCH)
     (delete 'bootstrap)
     (delete 'configure)                 ;not needed
     (replace 'build build)
     (delete 'check)                     ;moved after the install phase
     (replace 'install install)
     (add-after 'install 'check check)
-    (add-after 'install 'wrap wrap)
-    (add-before 'strip 'rename-pth-file rename-pth-file)))
+    (add-before 'check 'compile-bytecode compile-bytecode)
+    (add-after 'install 'wrap wrap)))
 
 (define* (python-build #:key inputs (phases %standard-phases)
                        #:allow-other-keys #:rest args)

Re: Questions regarding Python packaging

[Tanguy LE CARROUR]

Hi Lars,


Excerpts from Lars-Dominik Braun's message of January 23, 2021 1:34 pm:
>> Done! :-)
>> I've eventually succeeded in ("properly") packaging a software managed
>> with Poetry. And I've learned quite a lot on the way!
> oh, I see. I’ve actually been trying to replace python-build-system with
> a python-build based build. Attached is my current work in progress. I
> cannot quite build python-build, ,

?!
My `python-build` seems to work:
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=45931


> because I’m lacking support for python-flit

I also had a problem with `python-flit`, but it was when I was working
on `python-typer`:
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=45935

This is why I didn't build it from the source.


> but I think the general idea is clear: Remove pip and
> setuptools from python (saves almost 20MiB from the closure and avoids
> weird conflicts between python’s setuptools and python-setuptools) and
> turn them into (almost) ordinary packages. Then use setuptools to
> bootstrap itself, bootstrap python-build with setuptools and use
> python-build to build evrey other packages using python-build-system.

Wow, the rest is way out of my comfort zone! But I'll read it carefully
and try to help if I can!

Best regards,

-- 
Tanguy

Re: Questions regarding Python packaging

[Ryan Prior]

[-- Attachment #1: Type: text/plain, Size: 837 bytes --]

On January 23, 2021, Lars-Dominik Braun <lars@6xq.net> wrote:
> [...] Remove pip and
> setuptools from python (saves almost 20MiB from the closure and avoids
> weird conflicts between python’s setuptools and python-setuptools) and
> turn them into (almost) ordinary packages. 

I think if we do that then Python will need a treatment similar to GCC,
where we don't expose the package and instead offer a compound package
(could be called "python-toolchain" or just "python") which includes pip
and setuptools. The last decades of python packaging have trained people
that when you install python you get pip and setuptools as well, I think
we should not try to be too clever and violate that assumption.

Also, for what it's worth, we already have python-minimal which doesn't
have pip, and it's only  

[-- Attachment #2: Type: text/html, Size: 3156 bytes --]

Re: Questions regarding Python packaging

[Lars-Dominik Braun]

Hi Ryan,

> I think if we do that then Python will need a treatment similar to GCC,
> where we don't expose the package and instead offer a compound package
> (could be called "python-toolchain" or just "python") which includes pip
> and setuptools. The last decades of python packaging have trained people
> that when you install python you get pip and setuptools as well, I think
> we should not try to be too clever and violate that assumption.
this change is actually more about moving forward and adding PEP 517
support to python-build-system. Being able to demote setuptools and pip
to ordinary packages is merely a side-effect, because they’re not
essential any more.

Your mail seems to be incomplete, it stopped after:
> Also, for what it's worth, we already have python-minimal which doesn't
> have pip, and it's only  

Cheers,
Lars

Re: Questions regarding Python packaging

[Ryan Prior]

[-- Attachment #1: Type: text/plain, Size: 1464 bytes --]


On January 25, 2021, Lars-Dominik Braun <lars@6xq.net> wrote:
> Being able to demote setuptools and pip
> to ordinary packages is merely a side-effect, because they’re not
> essential any more.

I didn't read all of PEP 517, does it deprecate bundling pip with
Python? My understanding was that it just gives maintainers more options
and makes setuptools less of a de-facto standard. I'll read it more
thoroughly later and see if I missed some important info there.

Anyhow, even if a PEP says so, people's expectations to be able to use
pip will not disappear suddenly. It's part of the expected interface to
Python & I imagine removing it has the potential to cause much
confusion.

> Your mail seems to be incomplete, it stopped after:
> > Also, for what it's worth, we already have python-minimal which
> doesn't
> > have pip, and it's only

Ah darn, I'd meant to write it's only a little bit smaller, and I'm
interested to see where we can slim down the Python closure but others
have managed to do so without cutting out pip & setuptools.

All this being said, I'm not a huge fan of pip & setuptools, I like Guix
because it frees me from having to think so much about such things, so I
hope this doesn't come off as a defense of the tools themselves. I mean
to defend the many Python users who have been trained with a certain set
of expectations about the interfaces a Python package provides.

[-- Attachment #2: Type: text/html, Size: 3816 bytes --]

Re: Questions regarding Python packaging

[Hartmut Goebel]

Am 23.01.21 um 13:34 schrieb Lars-Dominik Braun:
> Remove pip and
> setuptools from python (saves almost 20MiB from the closure

When doing to we need to be carefully. pip is expected to be available 
after installing "python". So when removing pip and setuptool, we would 
need some "python/bootstrap" package without pip and setuptools and some 
"python" package still including both.


-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Questions regarding Python packaging

[Lars-Dominik Braun]

Hi everyone,

just a quick reminder that an updated version (includes
python-toolchain) of this proposal is still looking for a code review or
further discussion. So if you feel confident about touching
python-build-system, please have a look at
https://issues.guix.gnu.org/46848#1

I’d be nice to get this into core-updates before the next merge.
Otherwise we’ll have to keep adding workarounds (see for example
python-testpath in master) to Python packages not using setuptools as
their build system.

Cheers,
Lars

Re: Questions regarding Python packaging

[Tanguy LE CARROUR]

Hi Lars,


Excerpts from Lars-Dominik Braun's message of May 17, 2021 8:24 am:
> just a quick reminder that an updated version (includes
> python-toolchain) of this proposal is still looking for a code review or
> further discussion. So if you feel confident about touching
> python-build-system, please have a look at
> https://issues.guix.gnu.org/46848#1
> 
> I’d be nice to get this into core-updates before the next merge.
> Otherwise we’ll have to keep adding workarounds (see for example
> python-testpath in master) to Python packages not using setuptools as
> their build system.

Sorry if I'm (very) late, but apprently this hasn't made it to master
yet, so… What the status? Do you still need a willing-but-maybe-not-qualified
person to review or discuss your patch?

Regards,

-- 
Tanguy

Re: Questions regarding Python packaging

[Lars-Dominik Braun]

Hi Tanguy,

(cross-posting this to the issue itself too)

> Sorry if I'm (very) late, but apprently this hasn't made it to master
> yet, so… What the status? Do you still need a willing-but-maybe-not-qualified
> person to review or discuss your patch?
the patch set works, I can build many Python packages, although some
require changes. Still, multiple things need to be done before merging
is possible imo:

1) Validate the general idea of using pypa-build is viable and
   sustainable in the long run – ideally through review by someone else
   than me. We can’t touch python-build-system every week to solve
   structural issues, so it needs to be bullet-proof.
2) Figure out how to run testing code. Currently python-build-system
   just picks pytest, if available – not sure this is the best option we
   have. How do we deal with other test systems? How do we pass options?
3) Determine the fate of Python 2, which is probably broken through this
   patch set. Shall we remove it entirely? Is it worth to keep support?
4) Iron out minor details like including pip in the python package or
   create a new python-toolchain package? What do we include in that
   meta-package? pip? virtualenv? …?
5) Fix my awkward Scheme code, especially regarding unpacking of the
   built wheels. Should we be using Python’s unzip module or can be
   assumed unzip is available in the build environment? (Should we add
   it?)

I’m by no means a Python packaging expert, so any help would be
appreciated, even if it’s just a question or thumbs-up/thumbs-down on my
ideas.

Cheers,
Lars

Removal of Python 2?

[Hartmut Goebel]

Am 06.06.21 um 21:44 schrieb Lars-Dominik Braun:
> 3) Determine the fate of Python 2, which is probably broken through this
>     patch set. Shall we remove it entirely? Is it worth to keep support?

Python 2 is dead, dead, dead like the parrot and end-of-prolonged life 
as of more than 1 1/2 years. Anyhow, there might still be quite some 
software not ported to Python 3 after 10 years. So I'm afraid we need to 
keep Python 2.

Other opinions?

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Removal of Python 2?

[Konrad Hinsen]

Hi Hartmut,

> Python 2 is dead, dead, dead like the parrot and end-of-prolonged life 
> as of more than 1 1/2 years. Anyhow, there might still be quite some 
> software not ported to Python 3 after 10 years. So I'm afraid we need to 
> keep Python 2.

At this time, more than 2500 packages still depend on Python 2, and
that's not just Python libraries. The list includes packages such as
TeXlive, LLVM, Rust, OpenJDK, Calibre, OCaml, qemu, etc.

For many of these packages, Python 2 is used only as part of the build
procedure. Which is run in a highly protective sandbox. So I don't think
we need to worry much about having Python 2 in Guix. To discourage
people from installing it by accident, we could make it a hidden
package.

Cheers,
  Konrad.

Re: Removal of Python 2?

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 1378 bytes --]

Hi,

Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:

>> Python 2 is dead, dead, dead like the parrot and end-of-prolonged life 
>> as of more than 1 1/2 years. Anyhow, there might still be quite some 
>> software not ported to Python 3 after 10 years. So I'm afraid we need to 
>> keep Python 2.
>
> At this time, more than 2500 packages still depend on Python 2, and
> that's not just Python libraries. The list includes packages such as
> TeXlive, LLVM, Rust, OpenJDK, Calibre, OCaml, qemu, etc.
>
> For many of these packages, Python 2 is used only as part of the build
> procedure. Which is run in a highly protective sandbox. So I don't think
> we need to worry much about having Python 2 in Guix. To discourage
> people from installing it by accident, we could make it a hidden
> package.

I agree.

What we could do is start removing ‘python2-’ packages, especially those
with a non-negligible maintenance cost (numpy, scipy, matplotlib, etc.).
We can move them to the Guix-Past channel if there’s interest.

The attached script lists unused ‘python2-’ leaf packages:

--8<---------------cut here---------------start------------->8---
$ guix repl -- unused-python2-packages.scm | wc -l
288
--8<---------------cut here---------------end--------------->8---

Are these good candidates for removal?

Thoughts?

Ludo’.


[-- Attachment #2: the script --]
[-- Type: text/plain, Size: 667 bytes --]

(use-modules (gnu packages)
             (guix)
             (guix graph)
             (guix scripts graph))

(define all
  (fold-packages cons '() #:select? (const #t)))

(with-store store
  (let ((back-edges (run-with-store store
                      (node-back-edges %bag-node-type all))))
    (define (unused-python2? package)
      (and (string-prefix? "python2-" (package-name package))
           (zero? (node-reachable-count (list package)
                                        back-edges))))
    (for-each (lambda (package)
                (when (unused-python2? package)
                  (format #t "~a~%" (package-name package))))
              all)))

Re: Removal of Python 2?

[zimoun]

Hi,

On Wed, 23 Jun 2021 at 17:27, Ludovic Courtès <ludo@gnu.org> wrote:

> What we could do is start removing ‘python2-’ packages, especially those
> with a non-negligible maintenance cost (numpy, scipy, matplotlib, etc.).
> We can move them to the Guix-Past channel if there’s interest.

Starting with e13ab46c2052609c4ad792aeb304a752b50d034e, I remove the
broken leaf Python2 packages.  I find them using "guix weather" and
"guix refresh -l".  More are coming. ;-)

Cheers,
simon

Re: Removal of Python 2?

[Konrad Hinsen]

Hi Ludo and Simon,

Ludovic Courtès <ludo@gnu.org> writes:

> What we could do is start removing ‘python2-’ packages, especially those
> with a non-negligible maintenance cost (numpy, scipy, matplotlib, etc.).
> We can move them to the Guix-Past channel if there’s interest.

Yes, that sounds like a good plan. With the long-term goal of moving
python2 itself to guix-past, if it ever ceases to be an important
dependency (which I am not yet convinced of).

> The attached script lists unused ‘python2-’ leaf packages:
...
> Are these good candidates for removal?

These and more: once they are removed, there will be a new set of leaf
packages.

It might be worth checking if there is a Python 3 equivalent before
removing a library. But in the majority of cases there is.


zimoun <zimon.toutoune@gmail.com> writes:

> Starting with e13ab46c2052609c4ad792aeb304a752b50d034e, I remove the
> broken leaf Python2 packages.  I find them using "guix weather" and
> "guix refresh -l".  More are coming. ;-)

It certainly makes sense to remove broken packages, but for the rest I'd
say guix-past is a better destination than the trashcan.

Cheers,
  Konrad

Re: Removal of Python 2?

[Ryan Prior]

On Tuesday, June 22nd, 2021 at 6:53 AM, Hartmut Goebel <h.goebel@crazy-compilers.com> wrote:

> Am 06.06.21 um 21:44 schrieb Lars-Dominik Braun:
>
> > 3.  Determine the fate of Python 2, which is probably broken through this
> >
> >     patch set. Shall we remove it entirely? Is it worth to keep support?
>
> Python 2 is dead, dead, dead like the parrot and end-of-prolonged life
>
> as of more than 1 1/2 years. Anyhow, there might still be quite some
>
> software not ported to Python 3 after 10 years. So I'm afraid we need to
>
> keep Python 2.


Python 2 is no longer supported, but the Tauthon project [1] is continuing support and backporting features from Python 3. We should consider packaging Tauthon in Guix and updating packages to depend on it. The Python project has promised there will never be a Python 2.8 release [2] but considering the huge body of extant Python 2 code, if we can get consensus within Guix I think there would be value in continuing to support python2 packages via Tauthon.

[1] https://github.com/naftaliharris/tauthon
[2] https://www.python.org/dev/peps/pep-0404/

Re: Removal of Python 2?

[Konrad Hinsen]

Hi Ryan,

> Python 2 is no longer supported, but the Tauthon project [1] is
> continuing support and backporting features from Python 3. We should

Didn't know about Tauthon, thanks for the pointer!

> consider packaging Tauthon in Guix and updating packages to depend on

Yes, that sounds like an interesting strategy to explore. One potential
obstacle is that most of the Python2-dependent package are out of
maintenance as well. But then, many of them don't need any maintenance.

If we introduce Tauthon, we should probably untie its dependents from
the Python (3) version, i.e. stop deriving them from Python 3 package
definitions. The latter will ever more evolve in an incompatible way.

Cheers,
  Konrad.

Re: Questions regarding Python packaging

[Hartmut Goebel]

[-- Attachment #1: Type: text/plain, Size: 3927 bytes --]

Hi Lars,

sorry for being late for commenting on this (the time I can spend on 
guix is rather limited atm).

Here are some general remarks on this patch-set (in order of appearance):

  *

    Not installing pip by default might break some user's environments.
    Anyhow, since using pip in guix is not such a good idea anyway, this
    should be okay.

  *

    "use-setuptools" is gone. There are still about 10 packages with
    "#:use-setuptools #f" - which means they are (expected to be)
    incompatible with setuptools for some reason. You might want to
    check whether these packages actually still can't be packages with
    setuptools.

  *

    setuptools-shim has been removed. I don't think this is a good idea,
    since this peace of code enforces packages to be actually build with
    setuptools instead of old distutils. This code is still in current
    pip, so I assume it is still required.

    (This shim ensures setuptools is used, even if setup.py only imports
    distutils. And setuptools is required for some options like
    ""--single-version-externally-managed" - as the comment for the shim
    says.)

  *

    set-SOURCE-DATE-EPOCH: Please keep the verbose rational. It's much
    more helpful than the new one-line comment.

  *

    set-SOURCE-DATE-EPOCH: This implementation makes the code depend on
    wheel and wheel being used for installation.

  *

    Why has rename-pth-file been removed? Are you sure .pth-files are
    never created anymore nowerdays?

  *

    python-hashbang: Isn't this done already by the normal
    "patch-shebangs" phase after install in  gnu-build-system? (BTW:
    these are called *she*bangs).

  *

    I suggest to have phase compile-bytecode still honor older versions
    of python


> 1) Validate the general idea of using pypa-build is viable and
>     sustainable in the long run – ideally through review by someone else
>     than me. We can’t touch python-build-system every week to solve
>     structural issues, so it needs to be bullet-proof.

pypa bulld is where the PyPA is pushing towards. Anyhow, as of today, as 
far as I can see, adoption is low.

> 2) Figure out how to run testing code. Currently python-build-system
>     just picks pytest, if available – not sure this is the best option we
>     have. How do we deal with other test systems? How do we pass options?

AFAIK fhere is no standard way for running tests in python. pytest seems 
to be the most modern test-system. Anyhow packages still use nose or tox 
(which again might run pytest or nose, with parameters fetched from 
tox.ini). So I'm afraid, there is no general rule.

Did the PyPA publish some recommendations or PEP on this?

> 4) Iron out minor details like including pip in the python package or
>     create a new python-toolchain package? What do we include in that
>     meta-package? pip? virtualenv? …?

As I Python developer I nowerdays would expect pip and venv (which is 
part of the std-lib - but not the virualenv, which is a separate module) 
to be availalbe when installing "python". Anyhow I could live with pip 
being a separate package.

"python-toolchain" sounds oversized for me. Would this include the 
C-compiler, too (which one? maybe I want to build cross). I'd rather not 
have such a package.

> 5) Fix my awkward Scheme code, especially regarding unpacking of the
>     built wheels. Should we be using Python’s unzip module or can be
>     assumed unzip is available in the build environment? (Should we add
>     it?)
The gnu-build-system already provides the "unzip" binary (used in phase 
"unpack"). So we could simply use this. Otherwise I recommend using the 
Python zip module, as this is what is used for creating the zip-archives 
:-)

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |


[-- Attachment #2: Type: text/html, Size: 5886 bytes --]

Re: Questions regarding Python packaging

[Lars-Dominik Braun]

Hi Hartmut,

> sorry for being late for commenting on this (the time I can spend on 
> guix is rather limited atm).
no problem, same thing on my side.

>   *
> 
>     Not installing pip by default might break some user's environments.
>     Anyhow, since using pip in guix is not such a good idea anyway, this
>     should be okay.
True. We could rename python→python-minimal-interpreteronly (or similar;
given that python-minimal already exists) and python-toolchain→python to
work around that.

>   *
> 
>     "use-setuptools" is gone. There are still about 10 packages with
>     "#:use-setuptools #f" - which means they are (expected to be)
>     incompatible with setuptools for some reason. You might want to
>     check whether these packages actually still can't be packages with
>     setuptools.
Yeah, I’ve seen those, but the number was too small to bother for now.
I’ll have a look later.

>   *
> 
>     setuptools-shim has been removed. I don't think this is a good idea,
>     since this peace of code enforces packages to be actually build with
>     setuptools instead of old distutils. This code is still in current
>     pip, so I assume it is still required.
> 
>     (This shim ensures setuptools is used, even if setup.py only imports
>     distutils. And setuptools is required for some options like
>     ""--single-version-externally-managed" - as the comment for the shim
>     says.)
Is this relevant though? I doubt many packages are still importing
distutils and the few that do can be patched.

>   *
> 
>     set-SOURCE-DATE-EPOCH: Please keep the verbose rational. It's much
>     more helpful than the new one-line comment.
You mean the one from the now-removed ensure-no-mtimes-pre-1980? Sure.

>   *
> 
>     set-SOURCE-DATE-EPOCH: This implementation makes the code depend on
>     wheel and wheel being used for installation.
Technically it depends on the wheel builder understanding
SOURCE_DATE_EPOCH (not necessarily python-wheel). I’d say that’s
acceptable and it’d be preferable to fix build systems not respecting
this variable imo.


>   *
> 
>     Why has rename-pth-file been removed? Are you sure .pth-files are
>     never created anymore nowerdays?
Given that easy-install has been deprecated I think it’s safe to remove
this phase and flag any packages creating this easy-install.pth as
broken. (There are, however, legitimate packages creating files like
ruamel.yaml-0.15.83-py3.8-nspkg.pth.)

>   *
> 
>     python-hashbang: Isn't this done already by the normal
>     "patch-shebangs" phase after install in  gnu-build-system? (BTW:
>     these are called *she*bangs).
Afaik the function patch-shebang expects a leading slash and thus it
does not replace this “special” shebang (see
https://www.python.org/dev/peps/pep-0427/#installing-a-wheel-distribution-1-0-py32-none-any-whl;
Spread, point 3).

>   *
> 
>     I suggest to have phase compile-bytecode still honor older versions
>     of python
I’m not sure what you mean. compileall is also part of Python 2.

> pypa bulld is where the PyPA is pushing towards. Anyhow, as of today, as 
> far as I can see, adoption is low.
Of pypa build? That is true.

> AFAIK fhere is no standard way for running tests in python. pytest seems 
> to be the most modern test-system. Anyhow packages still use nose or tox 
> (which again might run pytest or nose, with parameters fetched from 
> tox.ini). So I'm afraid, there is no general rule.
> 
> Did the PyPA publish some recommendations or PEP on this?
I’m not aware of any accepted PEP’s. There is a discussion about the
removal of `python setup.py test`:
https://github.com/pypa/setuptools/issues/931
And a proposal for pyproject.toml going nowhere:
https://discuss.python.org/t/proposal-for-tests-entry-point-in-pyproject-toml/2077/2

> As I Python developer I nowerdays would expect pip and venv (which is 
> part of the std-lib - but not the virualenv, which is a separate module) 
> to be availalbe when installing "python". Anyhow I could live with pip 
> being a separate package.
If we keep setuptools/pip bundled, we don’t have to do any of this
pypa-build dance. We could also modernize python-build-system around
`pip install` and just be done with it. (I don’t have a proof-of-concept
for that yet.)

> "python-toolchain" sounds oversized for me. Would this include the 
> C-compiler, too (which one? maybe I want to build cross). I'd rather not 
> have such a package.
See suggestion above wrt renaming.

> The gnu-build-system already provides the "unzip" binary (used in phase 
> "unpack"). So we could simply use this. Otherwise I recommend using the 
> Python zip module, as this is what is used for creating the zip-archives 
> :-)
I’m using Python’s zipfile module already.

Cheers,
Lars

Re: Questions regarding Python packaging

[Hartmut Goebel]

Hi Lars-Dominik,

Am 28.06.21 um 13:59 schrieb Lars-Dominik Braun:*
>>      Not installing pip by default might break some user's environments.
>>      Anyhow, since using pip in guix is not such a good idea anyway, this
>>      should be okay.
> True. We could rename python→python-minimal-interpreteronly (or similar;
> given that python-minimal already exists) and python-toolchain→python to
> work around that.

What should be the use of having a package without pip? Anything else 
than saving a few KB?


>> [setuptools-shim has been removed]
> Is this relevant though? I doubt many packages are still importing
> distutils and the few that do can be patched.

Was I wrote: This code is still in pip, so I assume it is still relevant.

I don't think patching is a good idea. It requires effort (implementing, 
reviewing), which can be saved by keeping exisiting and tested code.


>>      set-SOURCE-DATE-EPOCH: This implementation makes the code depend on
>>      wheel and wheel being used for installation.
> Technically it depends on the wheel builder understanding
> SOURCE_DATE_EPOCH (not necessarily python-wheel). I’d say that’s
> acceptable and it’d be preferable to fix build systems not respecting
> this variable imo.

For this case please change the comment not not referring to wheel in 
this way. More something like "we expect the builder to support 
SOURCE_DATE_EPOCH, like wheel does"

Anyhow, *m not actually convinced that we should throw away the old 
code. I can imagine in the next cuple of years quite some new 
build-systems to arrive, most of which will most probably not support 
SOURCE_DATE_EPOCH in the beginning, and thus making package's life harder.


>
>>      Why has rename-pth-file been removed? Are you sure .pth-files are
>>      never created anymore nowerdays?
> Given that easy-install has been deprecated I think it’s safe to remove
> this phase and flag any packages creating this easy-install.pth as
> broken. (There are, however, legitimate packages creating files like
> ruamel.yaml-0.15.83-py3.8-nspkg.pth.)

What exaclty do you mean with "flag as broken"? Will anybody (you? ;-) 
verify *all* current packages to not be "broken" prior to merging this 
change?

Anyhow, again, I'm not convinced we should remove this phase now. 
.pth-file are deprecated only, but still supported. By removing this 
phase we might create conflict cased we can not foresee. And I would 
keep it even if one analyzes none of the current packages is "broken" - 
just do be on the save side fpr avoiding user trouble. (These issues 
will show up at the user, and are hard to track down, since noone will 
think about .pth files)


>
>>      python-hashbang: Isn't this done already by the normal
>>      "patch-shebangs" phase after install in  gnu-build-system? (BTW:
>>      these are called *she*bangs).
> Afaik the function patch-shebang expects a leading slash and thus it
> does not replace this “special” shebang (see
> https://www.python.org/dev/peps/pep-0427/#installing-a-wheel-distribution-1-0-py32-none-any-whl;
> Spread, point 3).

IC. Please add a comment to make this clear (e.g. "handle shebang of 
scripts generated by wheel missing leading slash")

>>    *
>>
>>      I suggest to have phase compile-bytecode still honor older versions
>>      of python
> I’m not sure what you mean. compileall is also part of Python 2.

The old code did not compile the source for Python <3.7. Please see the 
comment of the old code for rational.


>> As I Python developer I nowerdays would expect pip and venv (which is
>> part of the std-lib - but not the virualenv, which is a separate module)
>> to be availalbe when installing "python". Anyhow I could live with pip
>> being a separate package.
> If we keep setuptools/pip bundled, we don’t have to do any of this
> pypa-build dance. We could also modernize python-build-system around
> `pip install` and just be done with it. (I don’t have a proof-of-concept
> for that yet.)

AFAIK this might not be true if other build systems not using setuptools 
at all might show up. And isn't this the main reason for all your work?


>
>> The gnu-build-system already provides the "unzip" binary (used in phase
>> "unpack"). So we could simply use this. Otherwise I recommend using the
>> Python zip module, as this is what is used for creating the zip-archives
>> :-)
> I’m using Python’s zipfile module already.
Fine, so you can safely remove the respective comment ;-)

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Questions regarding Python packaging

[Lars-Dominik Braun]

Hi Hartmut,

> What should be the use of having a package without pip? Anything else 
> than saving a few KB?
saving some space and unvendoring components that we also have separate
packages for. (As I understand it, ensurepip, which installs both pip and
setuptools, is merely a convenience tool if you install Python from
source yourself – not for distributions.)

> AFAIK this might not be true if other build systems not using setuptools 
> at all might show up. And isn't this the main reason for all your work?
No, try

	git clone https://github.com/pypa/pep517.git
	cd pep517
	pip wheel --use-pep517 -v .

which has no setup.py and uses flit instead. pip can build it, because
it supports PEP 517-based builds. As I said, if we decide to keep it
bundled with our python package, there’s no good reason to choose
pypa-build.

Cheers,
Lars

[bug#46848] Questions regarding Python packaging

[Lars-Dominik Braun]

[-- Attachment #1: Type: text/plain, Size: 890 bytes --]

Hi again,

> No, try
> 
> 	git clone https://github.com/pypa/pep517.git
> 	cd pep517
> 	pip wheel --use-pep517 -v .
> 
> which has no setup.py and uses flit instead. pip can build it, because
> it supports PEP 517-based builds. As I said, if we decide to keep it
> bundled with our python package, there’s no good reason to choose
> pypa-build.
I now have a proof of concept for a pip-based python-build-system, see
1st patch. The changeset is quite small, but it does not handle testing
at all right now.

Note that alot of setuptools-based packages lack a dependency on
python-wheel (2nd patch; pardon the whitespace errors), which is
required when using pyproject.toml (and also declared there, but our
Python importer cannot read that file yet). But – as expected – that’s
really the only issue I’ve encountered while rebuilding alot of Python
packages so far.

Cheers,
Lars


[-- Attachment #2: 0001-dirty-build-Build-Python-packages-using-pip.patch --]
[-- Type: text/x-diff, Size: 7300 bytes --]

From a7c6750917f5dc2e1eaf34520f7e6b0e3d5e0d3c Mon Sep 17 00:00:00 2001
From: Lars-Dominik Braun <lars@6xq.net>
Date: Tue, 6 Jul 2021 14:13:51 +0200
Subject: [PATCH 1/2] dirty: build: Build Python packages using pip.
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

* guix/build/python-build-system.scm …
---
 guix/build/python-build-system.scm | 109 +++++++++++------------------
 1 file changed, 42 insertions(+), 67 deletions(-)

diff --git a/guix/build/python-build-system.scm b/guix/build/python-build-system.scm
index 8ade1d5911..e4b63e131e 100644
--- a/guix/build/python-build-system.scm
+++ b/guix/build/python-build-system.scm
@@ -109,30 +109,6 @@
 ;; and the scripts defined in entry-points will always be created.
 
 
-(define setuptools-shim
-  ;; Run setup.py with "setuptools" being imported, which will patch
-  ;; "distutils". This is needed for packages using "distutils" instead of
-  ;; "setuptools" since the former does not understand the
-  ;; "--single-version-externally-managed" flag.
-  ;; Python code taken from pip 9.0.1 pip/utils/setuptools_build.py
-  (string-append
-   "import setuptools, tokenize;__file__='setup.py';"
-   "f=getattr(tokenize, 'open', open)(__file__);"
-   "code=f.read().replace('\\r\\n', '\\n');"
-   "f.close();"
-   "exec(compile(code, __file__, 'exec'))"))
-
-(define (call-setuppy command params use-setuptools?)
-  (if (file-exists? "setup.py")
-      (begin
-         (format #t "running \"python setup.py\" with command ~s and parameters ~s~%"
-                command params)
-         (if use-setuptools?
-             (apply invoke "python" "-c" setuptools-shim
-                    command params)
-             (apply invoke "python" "./setup.py" command params)))
-      (error "no setup.py found")))
-
 (define* (sanity-check #:key tests? inputs outputs #:allow-other-keys)
   "Ensure packages depending on this package via setuptools work properly,
 their advertised endpoints work and their top level modules are importable
@@ -142,26 +118,6 @@ without errors."
     (with-directory-excursion "/tmp"
       (invoke "python" sanity-check.py (site-packages inputs outputs)))))
 
-(define* (build #:key use-setuptools? #:allow-other-keys)
-  "Build a given Python package."
-  (call-setuppy "build" '() use-setuptools?)
-  #t)
-
-(define* (check #:key tests? test-target use-setuptools? #:allow-other-keys)
-  "Run the test suite of a given Python package."
-  (if tests?
-      ;; Running `setup.py test` creates an additional .egg-info directory in
-      ;; build/lib in some cases, e.g. if the source is in a sub-directory
-      ;; (given with `package_dir`). This will by copied to the output, too,
-      ;; so we need to remove.
-      (let ((before (find-files "build" "\\.egg-info$" #:directories? #t)))
-        (call-setuppy test-target '() use-setuptools?)
-        (let* ((after (find-files "build" "\\.egg-info$" #:directories? #t))
-               (inter (lset-difference string=? after before)))
-          (for-each delete-file-recursively inter)))
-      (format #t "test suite not run~%"))
-  #t)
-
 (define (python-version python)
   (let* ((version     (last (string-split python #\-)))
          (components  (string-split version #\.))
@@ -195,31 +151,40 @@ running checks after installing the package."
                                 "/bin:"
                                 (getenv "PATH"))))
 
-(define* (install #:key inputs outputs (configure-flags '()) use-setuptools?
+;; Some packages expect 'build and 'check exist. If they don’t replacing them
+;; or adding phases before/after will fail. Preserve them as dummy-phases.
+(define* (build #:key outputs (configure-flags '()) use-setuptools?
+                  #:allow-other-keys)
+    #t)
+
+(define* (check #:key outputs (configure-flags '()) use-setuptools?
+                  #:allow-other-keys)
+    #t)
+
+(define* (install #:key outputs (configure-flags '()) use-setuptools?
                   #:allow-other-keys)
   "Install a given Python package."
-  (let* ((out (python-output outputs))
-         (python (assoc-ref inputs "python"))
-         (major-minor (map string->number
-                           (take (string-split (python-version python) #\.) 2)))
-         (<3.7? (match major-minor
-                   ((major minor)
-                    (or (< major 3) (and (= major 3) (< minor 7))))))
-         (params (append (list (string-append "--prefix=" out)
-                               "--no-compile")
-                         (if use-setuptools?
-                             ;; distutils does not accept these flags
-                             (list "--single-version-externally-managed"
-                                   "--root=/")
-                             '())
-                         configure-flags)))
-    (call-setuppy "install" params use-setuptools?)
-    ;; Rather than produce potentially non-reproducible .pyc files on Pythons
-    ;; older than 3.7, whose 'compileall' module lacks the
-    ;; '--invalidation-mode' option, do not generate any.
-    (unless <3.7?
-      (invoke "python" "-m" "compileall" "--invalidation-mode=unchecked-hash"
-              out))))
+  (let* ((out (assoc-ref outputs "out")))
+    (setenv "HOME" "/tmp") ; silence warning
+    (invoke "pip" "install"
+            (string-append "--prefix=" out)
+            "--root=/"
+            ;; Hide pip’s own version check. Nothing we can do anyway.
+            "--disable-pip-version-check"
+            ;; Do not search and install dependencies.
+            "--no-deps"
+            ;; Do not search PyPi.
+            "--no-index"
+            ;; Don’t reinstall build dependencies into virtal environent,
+            ;; instead use local ones. Otherwise build process will fail
+            ;; finding dependencies.
+            "--no-build-isolation"
+            ;; Ignore installed packages, important for packages bundled by
+            ;; Python, like setuptools.
+            "--ignore-installed"
+            ;; Library is in the current directory.
+            ".")
+    #t))
 
 (define* (wrap #:key inputs outputs #:allow-other-keys)
   (define (list-of-files dir)
@@ -267,6 +232,15 @@ installed with setuptools."
                  (utime file early-1980 early-1980))
                #t))))
 
+;; python-wheel respects this virable and passes an invalid early timestamp to
+;; Python’s zip module if not redefined to a later date.
+(define* (set-SOURCE-DATE-EPOCH #:rest _)
+  "Set the 'SOURCE_DATE_EPOCH' environment variable.  This is used by tools
+that incorporate timestamps as a way to tell them to use a fixed timestamp.
+See https://reproducible-builds.org/specs/source-date-epoch/."
+  (setenv "SOURCE_DATE_EPOCH" "315619200")
+  #t)
+
 (define* (enable-bytecode-determinism #:rest _)
   "Improve determinism of pyc files."
   ;; Use deterministic hashes for strings, bytes, and datetime objects.
@@ -297,6 +271,7 @@ by Cython."
       enable-bytecode-determinism)
     (add-after 'enable-bytecode-determinism 'ensure-no-cythonized-files
       ensure-no-cythonized-files)
+    (replace 'set-SOURCE-DATE-EPOCH set-SOURCE-DATE-EPOCH)
     (delete 'bootstrap)
     (delete 'configure)                 ;not needed
     (replace 'build build)
-- 
2.31.1


[-- Attachment #3: 0002-dirty-Fix-build-errors.patch --]
[-- Type: text/x-diff, Size: 15164 bytes --]

From 7e6d12aac228f369a1f09870529dc003a8ad356e Mon Sep 17 00:00:00 2001
From: Lars-Dominik Braun <lars@6xq.net>
Date: Tue, 6 Jul 2021 14:14:21 +0200
Subject: [PATCH 2/2] dirty: Fix build errors.

---
 gnu/packages/build-tools.scm   |  2 ++
 gnu/packages/check.scm         | 16 ++++++++----
 gnu/packages/python-build.scm  | 47 ++++++++++++++++++++++++++++++++++
 gnu/packages/python-crypto.scm |  9 ++++---
 gnu/packages/python-xyz.scm    | 44 ++++++++++++++++---------------
 gnu/packages/time.scm          |  4 ++-
 6 files changed, 92 insertions(+), 30 deletions(-)

diff --git a/gnu/packages/build-tools.scm b/gnu/packages/build-tools.scm
index dc2411801c..6c29fb9bdc 100644
--- a/gnu/packages/build-tools.scm
+++ b/gnu/packages/build-tools.scm
@@ -50,6 +50,7 @@
   #:use-module (gnu packages pretty-print)
   #:use-module (gnu packages protobuf)
   #:use-module (gnu packages python)
+  #:use-module (gnu packages python-build)
   #:use-module (gnu packages python-crypto)
   #:use-module (gnu packages python-web)
   #:use-module (gnu packages python-xyz)
@@ -237,6 +238,7 @@ files and generates build instructions for the Ninja build system.")
                   (delete 'wrap))))
     (inputs `(("ninja" ,ninja)))
     (propagated-inputs `(("python" ,python)))
+    (native-inputs `(("python-wheel" ,python-wheel)))
     (home-page "https://mesonbuild.com/")
     (synopsis "Build system designed to be fast and user-friendly")
     (description
diff --git a/gnu/packages/check.scm b/gnu/packages/check.scm
index 11117b88e4..12f17c41c6 100644
--- a/gnu/packages/check.scm
+++ b/gnu/packages/check.scm
@@ -975,6 +975,7 @@ standard library.")
        ("python-mock" ,python-mock)
        ("python-pytest" ,python-pytest-bootstrap)
        ("python-setuptools-scm" ,python-setuptools-scm)
+       ("python-wheel" ,python-wheel)
        ("python-toml" ,python-toml)
        ("python-xmlschema" ,python-xmlschema)))
     (home-page "https://docs.pytest.org/en/latest/")
@@ -1028,7 +1029,8 @@ and many external plugins.")
     (name "python-pytest-bootstrap")
     (native-inputs `(("python-iniconfig" ,python-iniconfig)
                      ("python-setuptools-scm" ,python-setuptools-scm)
-                     ("python-toml" ,python-toml)))
+                     ("python-toml" ,python-toml)
+                     ("python-wheel" ,python-wheel)))
     (arguments `(#:tests? #f))
     (properties `((python2-variant . ,(delay python2-pytest-bootstrap))))))
 
@@ -1113,7 +1115,8 @@ supports coverage of subprocesses.")
                           (format #t "test suite not run~%"))
                       #t)))))
     (native-inputs
-     `(("python-setuptools-scm" ,python-setuptools-scm)))
+     `(("python-setuptools-scm" ,python-setuptools-scm)
+	   ("python-wheel" ,python-wheel)))
     (home-page "https://github.com/pytest-dev/pytest-runner")
     (synopsis "Invoke py.test as a distutils command")
     (description
@@ -1236,7 +1239,8 @@ same arguments.")
               (when tests?
                 (invoke "py.test" "-v")))))))
     (native-inputs
-     `(("python-setuptools-scm" ,python-setuptools-scm)))
+     `(("python-setuptools-scm" ,python-setuptools-scm)
+	   ("python-wheel" ,python-wheel)))
     (propagated-inputs
      `(("python-execnet" ,python-execnet)
        ("python-pytest" ,python-pytest)
@@ -1302,7 +1306,8 @@ timeout has been exceeded.")
     (build-system python-build-system)
     (native-inputs
      `(("python-pytest" ,python-pytest)
-       ("python-setuptools-scm" ,python-setuptools-scm)))
+       ("python-setuptools-scm" ,python-setuptools-scm)
+	   ("python-wheel" ,python-wheel)))
     (home-page
      "https://github.com/pytest-dev/pytest-forked")
     (synopsis
@@ -2486,7 +2491,8 @@ create data based on random numbers and yet remain repeatable.")
     (build-system python-build-system)
     (native-inputs
      `(("python-mock" ,python-mock)
-       ("python-pytest" ,python-pytest)))
+       ("python-pytest" ,python-pytest)
+	   ("python-wheel" ,python-wheel)))
     (propagated-inputs
      `(("python-six" ,python-six)
        ("python-dateutil" ,python-dateutil)))
diff --git a/gnu/packages/python-build.scm b/gnu/packages/python-build.scm
index 140629ca43..913d1b04ce 100644
--- a/gnu/packages/python-build.scm
+++ b/gnu/packages/python-build.scm
@@ -89,6 +89,52 @@ installed with a newer @code{pip} or with wheel's own command line utility.")
 Language (TOML) configuration files.")
     (license license:expat)))
 
+(define-public python-pytoml
+  (package
+    (name "python-pytoml")
+    (version "0.1.21")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (pypi-uri "pytoml" version))
+       (sha256
+        (base32
+         "1rv1byiw82k7mj6aprcrqi2vdabs801y97xhfnrz7kxds34ggv4f"))))
+    (build-system python-build-system)
+    (home-page "https://github.com/avakar/pytoml")
+    (synopsis "Parser for TOML")
+    (description "This package provides a Python parser for TOML-0.4.0.")
+    (license license:expat)))
+
+(define-public python-flit-core
+  (package
+    (name "python-flit-core")
+    (version "3.2.0")
+    (source
+      (origin
+        (method url-fetch)
+        (uri (pypi-uri "flit-core" version))
+        (sha256
+          (base32
+            "0cclv7v4cmzi457bzqsx9ig8ir1ha3ip8h1kx8qfy95wbmfg51zz"))))
+    (build-system python-build-system)
+	(arguments
+     `(#:phases
+       (modify-phases %standard-phases
+         ;; Requirements refer to a specific version of dependencies,
+         ;; which are too old. So we patch to refer to any later version.
+         (add-after 'unpack 'use-local-sourcetree
+           (lambda _
+             (setenv "GUIX_PYTHONPATH" (string-append (getenv "GUIX_PYTHONPATH") ":" (getcwd))))))))
+    (propagated-inputs
+      `(("python-toml" ,python-toml)))
+    (home-page "https://github.com/takluyver/flit")
+    (synopsis
+      "Distribution-building parts of Flit. See flit package for more information")
+    (description
+      "Distribution-building parts of Flit. See flit package for more information")
+    (license #f)))
+
 (define-public python-pep517-bootstrap
   (hidden-package
    (package
@@ -104,6 +150,7 @@ Language (TOML) configuration files.")
      (build-system python-build-system)
      (arguments
       `(#:tests? #f))                     ;to avoid circular dependencies
+	 (native-inputs `(("python-flit-core" ,python-flit-core)))
      (propagated-inputs
       `(("python-toml" ,python-toml)
         ("python-wheel" ,python-wheel)))
diff --git a/gnu/packages/python-crypto.scm b/gnu/packages/python-crypto.scm
index 39b194b25e..e40a60a9ca 100644
--- a/gnu/packages/python-crypto.scm
+++ b/gnu/packages/python-crypto.scm
@@ -124,7 +124,8 @@ with what is used by the Bitcoin network.")
     (build-system python-build-system)
     (native-inputs
      `(("python-pycparser" ,python-pycparser)
-       ("python-pytest" ,python-pytest)))
+       ("python-pytest" ,python-pytest)
+	   ("python-wheel" ,python-wheel)))
     (propagated-inputs
      `(("python-cffi" ,python-cffi)
        ("python-six" ,python-six)))
@@ -544,7 +545,8 @@ is used by the Requests library to verify HTTPS requests.")
        ("python-hypothesis" ,python-hypothesis)
        ("python-pretend" ,python-pretend)
        ("python-pytz" ,python-pytz)
-       ("python-pytest" ,python-pytest)))
+       ("python-pytest" ,python-pytest)
+	   ("python-wheel" ,python-wheel)))
     (home-page "https://github.com/pyca/cryptography")
     (synopsis "Cryptographic recipes and primitives for Python")
     (description
@@ -611,7 +613,8 @@ message digests and key derivation functions.")
      `(("libfaketime" ,libfaketime)
        ("python-flaky" ,python-flaky)
        ("python-pretend" ,python-pretend)
-       ("python-pytest" ,python-pytest)))
+       ("python-pytest" ,python-pytest)
+	   ("python-wheel" ,python-wheel)))
     (home-page "https://github.com/pyca/pyopenssl")
     (synopsis "Python wrapper module around the OpenSSL library")
     (description
diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm
index 7a2b9eecc6..59dbe45a4f 100644
--- a/gnu/packages/python-xyz.scm
+++ b/gnu/packages/python-xyz.scm
@@ -2307,6 +2307,7 @@ and is not compatible with JSON.")
              (invoke "python" "bootstrap.py" "build/scons" "DEVELOPER=guix")
              (chdir "build/scons")
              #t)))))
+    (native-inputs `(("python-wheel" ,python-wheel)))
     (home-page "https://scons.org/")
     (synopsis "Software construction tool written in Python")
     (description
@@ -3161,7 +3162,8 @@ e.g. filters, callbacks and errbacks can all be promises.")
     (native-inputs
      `(("python-mock" ,python-mock)
        ("python-pytest" ,python-pytest)
-       ("python-setuptools-scm" ,python-setuptools-scm)))
+       ("python-setuptools-scm" ,python-setuptools-scm)
+	   ("python-wheel" ,python-wheel)))
     (propagated-inputs
      `(("python-appdirs" ,python-appdirs)
        ("python-distlib" ,python-distlib/next)
@@ -6498,6 +6500,7 @@ child application and control it as if a human were typing commands.")
                (base32
                 "0ahlrxxkx2xhmxskx57gc96w3bdndflxx30304ihvm7ds136nny8"))))
     (build-system python-build-system)
+    (native-inputs `(("python-wheel" ,python-wheel)))
     (home-page "https://github.com/pypa/setuptools_scm/")
     (synopsis "Manage Python package versions in SCM metadata")
     (description
@@ -8284,7 +8287,8 @@ PEP 8.")
     (native-inputs
      `(("python-mock" ,python-mock)
        ("python-pytest" ,python-pytest)
-       ("python-testpath" ,python-testpath)))
+       ("python-testpath" ,python-testpath)
+       ,@(package-native-inputs python-pep517-bootstrap)))
     (properties `((python2-variant . ,(delay python2-pep517))))))
 
 ;; Skip the tests so we don't create a cyclical dependency with pytest.
@@ -8752,6 +8756,7 @@ output.")
         (sha256
           (base32 "1p9p7mn8x2j9psc4jxab98897v4i9s4fliyfw8rp8v4bx1n7pjj2"))))
     (build-system python-build-system)
+    (native-inputs `(("python-poetry-core" ,python-poetry-core)))
     (home-page "https://github.com/sdispater/crashtest")
     (synopsis "Manage Python errors with ease")
     (description
@@ -9891,20 +9896,7 @@ Jupyter Notebook format and Python APIs for working with notebooks.")
         (base32
          "0lc4si3xb7hza424414rdqdc3vng3kcrph8jbvjqb32spqddf3f7"))))
     (build-system python-build-system)
-    ;; The package does not come with a setup.py file, so we have to generate
-    ;; one ourselves.
-    (arguments
-     `(#:tests? #f
-       #:phases
-       (modify-phases %standard-phases
-         (add-after 'unpack 'create-setup.py
-           (lambda _
-             (call-with-output-file "setup.py"
-               (lambda (port)
-                 (format port "\
-from setuptools import setup
-setup(name='entrypoints', version='~a', py_modules=['entrypoints'])
-" ,version))))))))
+    (native-inputs `(("python-flit" ,python-flit)))
     (home-page "https://github.com/takluyver/entrypoints")
     (synopsis "Discover and load entry points from installed Python packages")
     (description "Entry points are a way for Python packages to advertise
@@ -11176,7 +11168,8 @@ library as well as on the command line.")
        "1c35qyhvy27q9ih9n899f3h4sdnpgq027dbiilly2qb5cvgarchm"))))
    (build-system python-build-system)
    (native-inputs
-    `(("python-setuptools-scm" ,python-setuptools-scm)))
+    `(("python-setuptools-scm" ,python-setuptools-scm)
+      ("python-wheel" ,python-wheel)))
    (synopsis "Plugin and hook calling mechanism for Python")
    (description "Pluggy is an extraction of the plugin manager as used by
 Pytest but stripped of Pytest specific details.")
@@ -14657,7 +14650,8 @@ than during a preprocessing step).")
            "17xbrgi23l87yg6h0qcknssp2q812miiy33qw6v45v5gx0jwv5xh"))))
     (build-system python-build-system)
     (propagated-inputs
-     `(("python-setuptools-scm" ,python-setuptools-scm)))
+     `(("python-setuptools-scm" ,python-setuptools-scm)
+	   ("python-wheel" ,python-wheel)))
     (home-page "https://github.com/marcelm/xopen/")
     (synopsis "Open compressed files transparently")
     (description "This module provides an @code{xopen} function that works like
@@ -15194,7 +15188,8 @@ and bit flag values.")
                     (lambda _
                       (invoke "pytest"))))))
     (native-inputs
-     `(("python-coverage" ,python-coverage)
+     `(("python-wheel" ,python-wheel)
+       ("python-coverage" ,python-coverage)
        ("python-hypothesis" ,python-hypothesis)
        ("python-pympler" ,python-pympler)
        ("python-pytest" ,python-pytest)
@@ -15215,7 +15210,7 @@ protocols.")
   (package
     (inherit python-attrs)
     (name "python-attrs-bootstrap")
-    (native-inputs `())
+    (native-inputs `(("python-wheel" ,python-wheel)))
     (arguments `(#:tests? #f))))
 
 (define-public python2-attrs-bootstrap
@@ -15606,6 +15601,7 @@ in other versions.")
         (base32
          "0ckzngs3scaa1mcfmsi1w40a1l8cxxnncscrxzjjwjyisx8z0fmw"))))
     (build-system python-build-system)
+    (native-inputs `(("python-wheel" ,python-wheel)))
     (home-page "https://github.com/RonnyPfannschmidt/iniconfig")
     (synopsis "Simple INI-file parser")
     (description "The @code{iniconfig} package provides a small and simple
@@ -20784,7 +20780,8 @@ register custom encoders and decoders.")
     (native-inputs
      `(("double-conversion" ,double-conversion)
        ("python-setuptools-scm" ,python-setuptools-scm)
-       ("python-pytest" ,python-pytest)))
+       ("python-pytest" ,python-pytest)
+	   ("python-wheel" ,python-wheel)))
     (home-page "https://github.com/ultrajson/ultrajson")
     (synopsis "Ultra fast JSON encoder and decoder for Python")
     (description
@@ -20980,6 +20977,11 @@ the syntactic logic to configure and launch jobs in an execution environment.")
     (build-system python-build-system)
     (arguments
      `(#:tests? #f)) ; XXX: Check requires network access.
+	(propagated-inputs
+	  `(("python-docutils" ,python-docutils)
+		("python-pytoml" ,python-pytoml)
+		("python-requests" ,python-requests)
+		("python-flit-core" ,python-flit-core)))
     (home-page "https://flit.readthedocs.io/")
     (synopsis
      "Simple packaging tool for simple packages")
diff --git a/gnu/packages/time.scm b/gnu/packages/time.scm
index 9b006a5438..60a0584202 100644
--- a/gnu/packages/time.scm
+++ b/gnu/packages/time.scm
@@ -46,6 +46,7 @@
   #:use-module (gnu packages compression)
   #:use-module (gnu packages perl)
   #:use-module (gnu packages python)
+  #:use-module (gnu packages python-build)
   #:use-module (gnu packages python-xyz))
 
 (define-public time
@@ -194,7 +195,8 @@ Pendulum instances.")
     (native-inputs
      `(("python-pytest" ,python-pytest)
        ("python-pytest-cov" ,python-pytest-cov)
-       ("python-setuptools-scm" ,python-setuptools-scm)))
+       ("python-setuptools-scm" ,python-setuptools-scm)
+	   ("python-wheel" ,python-wheel)))
     (propagated-inputs
      `(("python-six" ,python-six)))
     (home-page "https://dateutil.readthedocs.io/en/stable/")
-- 
2.31.1

Re: Questions regarding Python packaging

[Hartmut Goebel]

[-- Attachment #1: Type: text/plain, Size: 816 bytes --]

Am 29.06.21 um 09:20 schrieb Lars-Dominik Braun:
>> AFAIK this might not be true if other build systems not using setuptools
>> at all might show up. And isn't this the main reason for all your work?
> No, try

Sorry, I've been inprecise on this:

There might still be quite some packages out there importing plain, old 
distutils (and not setuptools) in their setup.py. These are what I meant 
with "other build systems not using setuptools". For these setup.py to 
understand the options we (and pip) need for installation, "import 
distutils" has to be hacked to actually become "import setuptools" - 
which is what setuptools-shim does

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |


[-- Attachment #2: Type: text/html, Size: 1511 bytes --]

[bug#46848] Questions regarding Python packaging

[Lars-Dominik Braun]

Hi,

> > What should be the use of having a package without pip? Anything else 
> > than saving a few KB?
> saving some space and unvendoring components that we also have separate
> packages for. (As I understand it, ensurepip, which installs both pip and
> setuptools, is merely a convenience tool if you install Python from
> source yourself – not for distributions.)
there is another crucial one: Collisions. Right now our python-pip and
python-setuptools packages are rather useless, because they collide with
the Python-bundled one. I’m trying to update Jupyter currently and
python-jupyter-packaging depends on a newer setuptools (v46ish) than our
python package provides (v41). Adding python-setuptools (v52) to the
propagated inputs (yes, it’s a runtime dependency), would be the natural
solution, but the resulting setuptools package in the build environment
for packages depending on python-jupyter-packaging is a mix between v42
and v52 and does not work.

Cheers,
Lars

Re: Questions regarding Python packaging

[Tanguy LE CARROUR]

Excerpts from Tanguy LE CARROUR's message of January 22, 2021 9:38 am:
> Excerpts from Tanguy LE CARROUR's message of January 6, 2021 4:32 pm:
>> Excerpts from Lars-Dominik Braun's message of January 5, 2021 11:25 am:
>>>> So, I've tried packaging `python-keyring` with those two…
>>>> 
>>>> `pep517` keeps on trying to download dependencies, which won't work.
>>>> 
>>>> `build` crashes with "ZIP does not support timestamps before 1980",
>>>> which, I guess is related to the fact that everything in the store is
>>>> timestamped to January 1st 1970.
>>> have you been looking into a python-build-system using `build`[1]? I’ve
>>> had the same issue with egg versions set to 0.0.0 and think in the long
>>> run moving to a PEP 517-style build is the way forward.
>> 
>> I agree! Unfortunately, I haven't had much time (so far) to work on it! :-(
>> 
>> I'll revive the thread as soon as I've made progress…
> 
> Done! :-)
> I've eventually succeeded in ("properly") packaging a software managed
> with Poetry. And I've learned quite a lot on the way!

Just for the sake of having it documented somewhere, here is the
relevant part of the `guix.scm` file:

```
(define-public nowty
  (package
    (name "nowty")
    (version (string-append %pyproject-version "+" %git-commit))
    (source (local-file %source-dir #:recursive? #t))
    (build-system python-build-system)
    (arguments
     `(#:phases
       (modify-phases %standard-phases
        (replace 'build
         (lambda* (#:key #:allow-other-keys)
           (substitute* "pyproject.toml"
            (((string-append "version = \"" ,%pyproject-version "\"" ))
             (string-append "version = \"" ,version "\"")))))
        (replace 'install
         (lambda* (#:key outputs #:allow-other-keys)
          (let ((out (assoc-ref outputs "out")))
           (invoke "python" "-m" "pip" "install"
                   "--no-dependencies" "--no-build-isolation" "--prefix" out "."))))
        (replace 'check
         (lambda* (#:key inputs outputs #:allow-other-keys)
          (add-installed-pythonpath inputs outputs)
          (invoke "python" "-m" "invoke" "test.unit"))))))
    (native-inputs
     `(("python-invoke" ,python-invoke-patched)
       ("python-mamba" ,python-mamba)
       ("python-poetry-core" ,python-poetry-core)
       ("python-robber" ,python-robber)
       ("python-termcolor" ,python-termcolor)))
    (propagated-inputs
     `(("python-mpd2" ,python-mpd2)
       ("python-typer" ,python-typer)))
    (synopsis "Music notification daemon for MPD")
    (description "A music notification daemon that monitors songs being played by MPD
and displays notifications with the song's details.")
    (home-page "http://projects.bioneland.org/nowty")
    (license license:gpl3+)))
```

-- 
Tanguy

Re: Questions regarding Python packaging

[Maxim Cournoyer]

Hello,

Tanguy LE CARROUR <tanguy@bioneland.org> writes:

> Excerpts from Tanguy LE CARROUR's message of January 22, 2021 9:38 am:
>> Excerpts from Tanguy LE CARROUR's message of January 6, 2021 4:32 pm:
>>> Excerpts from Lars-Dominik Braun's message of January 5, 2021 11:25 am:
>>>>> So, I've tried packaging `python-keyring` with those two…
>>>>> 
>>>>> `pep517` keeps on trying to download dependencies, which won't work.
>>>>> 
>>>>> `build` crashes with "ZIP does not support timestamps before 1980",
>>>>> which, I guess is related to the fact that everything in the store is
>>>>> timestamped to January 1st 1970.
>>>> have you been looking into a python-build-system using `build`[1]? I’ve
>>>> had the same issue with egg versions set to 0.0.0 and think in the long
>>>> run moving to a PEP 517-style build is the way forward.
>>> 
>>> I agree! Unfortunately, I haven't had much time (so far) to work on it! :-(
>>> 
>>> I'll revive the thread as soon as I've made progress…
>> 
>> Done! :-)
>> I've eventually succeeded in ("properly") packaging a software managed
>> with Poetry. And I've learned quite a lot on the way!
>
> Just for the sake of having it documented somewhere, here is the
> relevant part of the `guix.scm` file:
>
> ```
> (define-public nowty
>   (package
>     (name "nowty")
>     (version (string-append %pyproject-version "+" %git-commit))
>     (source (local-file %source-dir #:recursive? #t))
>     (build-system python-build-system)
>     (arguments
>      `(#:phases
>        (modify-phases %standard-phases
>         (replace 'build
>          (lambda* (#:key #:allow-other-keys)
>            (substitute* "pyproject.toml"
>             (((string-append "version = \"" ,%pyproject-version "\"" ))
>              (string-append "version = \"" ,version "\"")))))
>         (replace 'install
>          (lambda* (#:key outputs #:allow-other-keys)
>           (let ((out (assoc-ref outputs "out")))
>            (invoke "python" "-m" "pip" "install"
>                    "--no-dependencies" "--no-build-isolation" "--prefix" out "."))))
>         (replace 'check
>          (lambda* (#:key inputs outputs #:allow-other-keys)
>           (add-installed-pythonpath inputs outputs)
>           (invoke "python" "-m" "invoke" "test.unit"))))))
>     (native-inputs
>      `(("python-invoke" ,python-invoke-patched)
>        ("python-mamba" ,python-mamba)
>        ("python-poetry-core" ,python-poetry-core)
>        ("python-robber" ,python-robber)
>        ("python-termcolor" ,python-termcolor)))
>     (propagated-inputs
>      `(("python-mpd2" ,python-mpd2)
>        ("python-typer" ,python-typer)))
>     (synopsis "Music notification daemon for MPD")
>     (description "A music notification daemon that monitors songs being played by MPD
> and displays notifications with the song's details.")
>     (home-page "http://projects.bioneland.org/nowty")
>     (license license:gpl3+)))
> ```

As another PEP 517 package build example, you may want to look at
python-isort on core-updates (commit
812a2931de553d12c01b0a4d53d03613b09adaaf).  Here's the definition:

--8<---------------cut here---------------start------------->8---
(define-public python-isort
  (package
    (name "python-isort")
    (version "5.7.0")
    (source
     (origin
       (method git-fetch)
       (uri (git-reference
             ;; Tests pass only from the Github sources
             (url "https://github.com/timothycrosley/isort")
             (commit version)))
       (file-name (git-file-name name version))
       (modules '((guix build utils)))
       (snippet '(for-each delete-file (find-files "." "\\.whl$")))
       (sha256
        (base32
         "0phq4s911mjjdyr5h5siz93jnpkqb2qgphgcfk6axncgxr8i7vi1"))))
    (build-system python-build-system)
    (arguments
     `(#:phases
       (modify-phases %standard-phases
         ;; A foretaste of what our future python-build-system will need to
         ;; do.
         (replace 'build
           (lambda _
             (invoke "python" "-m" "build" "--wheel" "--no-isolation" ".")))
         (replace 'install
           (lambda* (#:key outputs #:allow-other-keys)
             (let ((out (assoc-ref outputs "out"))
                   (whl (car (find-files "dist" "\\.whl$"))))
               (invoke "pip" "--no-cache-dir" "--no-input"
                       "install" "--no-deps" "--prefix" out whl))))
         (add-after 'install 'install-example-plugins
           (lambda* (#:key outputs #:allow-other-keys)
             (let ((out (assoc-ref outputs "out")))
               ;; Patch to use the core poetry API.
               (substitute* '("example_isort_formatting_plugin/pyproject.toml"
                              "example_shared_isort_profile/pyproject.toml")
                 (("poetry>=0.12")
                  "poetry-core>=1.0.0")
                 (("poetry.masonry.api")
                  "poetry.core.masonry.api"))
               ;; Build the example plugins.
               (for-each (lambda (source-directory)
                           (invoke "python" "-m" "build" "--wheel"
                                   "--no-isolation" "--outdir=dist"
                                   source-directory))
                         '("example_isort_formatting_plugin"
                           "example_shared_isort_profile"))
               ;; Install them to temporary storage, for the test.
               (setenv "HOME" (getcwd))
               (let ((example-whls (find-files "dist" "^example.*\\.whl$")))
                 (apply invoke "pip" "--no-cache-dir" "--no-input"
                        "install"  "--user" "--no-deps" example-whls)))))
         (replace 'check
           (lambda* (#:key tests? inputs outputs #:allow-other-keys)
             (when tests?
               (let ((bin (string-append (assoc-ref outputs "out") "/bin")))
                 (setenv "PATH" (string-append (getenv "PATH") ":" bin)))
               (add-installed-pythonpath inputs outputs)
               (invoke "pytest" "-vv" "tests/unit/"
                       "--ignore=tests/unit/test_deprecated_finders.py")))))))
    (native-inputs
     `(("git" ,git-minimal)
       ("python-black" ,python-black)
       ("python-colorama" ,python-colorama)
       ("python-hypothesmith" ,python-hypothesmith)
       ("python-libcst" ,python-libcst-minimal)
       ("python-poetry-core" ,python-poetry-core)
       ("python-pylama" ,python-pylama)
       ("python-pypa-build" ,python-pypa-build)
       ("python-pytest-mock" ,python-pytest-mock)
       ("python-pytest" ,python-pytest)))
    (home-page "https://github.com/PyCQA/isort")
    (synopsis "Python utility/library to sort python imports")
    (description "@code{python-isort} is a python utility/library to sort
imports alphabetically, and automatically separated into sections.  It
provides a command line utility, a python library and plugins for various
editors.")
    (license license:expat)))
--8<---------------cut here---------------end--------------->8---

That's an ad-hoc solution pending an update to our Python build system
phases.

HTH!

Maxim

Re: Questions regarding Python packaging

[Tanguy LE CARROUR]

Excerpts from Lars-Dominik Braun's message of January 5, 2021 11:25 am:
> Hi Tanguy,
> 
>> So, I've tried packaging `python-keyring` with those two…
>> 
>> `pep517` keeps on trying to download dependencies, which won't work.
>> 
>> `build` crashes with "ZIP does not support timestamps before 1980",
>> which, I guess is related to the fact that everything in the store is
>> timestamped to January 1st 1970.
> have you been looking into a python-build-system using `build`[1]? I’ve
> had the same issue with egg versions set to 0.0.0 and think in the long
> run moving to a PEP 517-style build is the way forward.

Actually —and I almost forgot!—, Sébastien found a solution to our "0.0.0" problem:
https://github.com/jaraco/keyring/issues/469#issuecomment-735695476

-- 
Tanguy