Python feature branch

Python feature branch

[John Kehayias]

Hello,

Below is the cover letter I sent for a series of patches to fix python-yubikey-manager which along the way updates/fixes more. (Apologies for the double send to Python team but I wanted the bug number first for this message and to include them in any larger discussion.) The series is here: <https://issues.guix.gnu.org/63139>

The deepest change is a package that affects pyproject-build-system. I thought I would send this here as well for wider visibility on this series as well as to garner any ready to go Python changes that can't be pushed to master directly.

So, please feel free to help gather any other patches that should go on a Python updates feature branch. I would like to keep this for changes that are less controversial/quicker changes in order to fix the breakages introduced from core-updates, namely poetry and python-yubikey-manager (an important leaf for some). However, since this series will rebuild all of pyproject-build-system using packages, there is the potential for lots of breakage. In that case, we can separate out the newer python-pypa-build package just for where it is needed an not update the build-system now.

Thanks everyone!
John

Cover for #63139:

Hi Guix and Python team,

Here is a patch series where the original goal was to fix python-yubikey-manager on core-updates and then ballooned to a bit more. This should be done in a feature branch for Python.

Mostly this was to fix/update the needed dependencies, though it may be possible to do this in a more minimal way just for that package fix. Anyway, I tried to generate this series in a way that each patch continues to fix things, but due to the complicated dependencies this may not be perfect.

A few notes:

1. Most of the series is pretty trivial, quick fixes/updates, some new packages.

2. What isn't is a few cases of failing tests which weren't immediately obvious to me and likely were some network access and/or build environment details. Some could be worked around maybe if someone wants to try (e.g. in python-virtualenv). I did enable more tests along the way though (like for poetry), so on the whole I think this is a step forward.

3. The dependents tend to be maybe 10s, a few in the hundreds, and then about 3k for python-filelock. Until we get to pyproject-build-system updates:

4. I believe it was poetry that needed a newer python-pypa-build module, which then touches all pyproject-build-system (about 6k packages). This isn't strictly necessary as we could have a newer and separate package for leafs to use rather than in the build system as well, but I figured might as well do it sooner rather than later. At least the packages up to python-yubikey-manager built with this along with some random others.

5. On that note, I did not complete this change as I wanted some feedback on the bootstraping. I've added python-pyproject-hooks which should deprecate pep517, but currently it also needs python-pypa-build. I've made the older python-pypa-build a -bootstrap package to build this and the newer version of itself as well. So I did not deprecate pep517 yet.

   Also, python-wheel was a propagated-input in pep517 which is not needed in pyproject-hooks. However, I saw at least some packages will then need that as an input to build; so I kept it for pyproject-hooks to ease testing. It should be removed and added as an input as needed (no idea if that is just a few or a lot of the tree).

Okay, I think those are my notes. We should see what other things are ready to be made into this feature branch for Python. One brought to my attention recently is <https://issues.guix.gnu.org/63044> though I have not looked at it.

Thanks!
John

John Kehayias (20):
  gnu: Add python-installer.
  gnu: Add python-pyproject-hooks.
  gnu: Add python-rapidfuzz.
  gnu: python-crashtest: Update to 0.4.1.
  gnu: python-cleo: Update to 2.0.1.
  gnu: Add python-deepdiff.
  gnu: python-platformdirs: Update to 3.2.0.
  gnu: python-filelock: Update to 3.12.0.
  gnu: python-distlib: Update to 0.3.6.
  gnu: python-virtualenv: Update to 20.22.0.
  gnu: python-pkginfo: Update to 1.9.6.
  gnu: python-jsonschema: Update to 4.17.3.
  gnu: python-dulwich: Update to 0.21.3.
  gnu: Update python-pypa-build to 1.0.0.
  gnu: poetry: Fix build.
  gnu: Add python-poetry-plugin-export.
  gnu: python-pyscard: Update to 2.0.7.
  gnu: python-fido2: Update to 1.1.1 and enable tests.
  gnu: Add python-makefun.
  gnu: python-yubikey-manager: Update to 5.1.0 and enable tests.

gnu/packages/python-build.scm   |  80 +++++++-
gnu/packages/python-xyz.scm     | 319 +++++++++++++++++++++++++-------
gnu/packages/security-token.scm |  61 +++---
3 files changed, 370 insertions(+), 90 deletions(-)


base-commit: aecc6e70587f8412cbbb9b2c13141de4f534518e

Re: Python feature branch

[Sharlatan Hellseher]

[-- Attachment #1: Type: text/plain, Size: 159 bytes --]

Hi,

I've fixed the build and updated chain of inputs for python-astropy,
posting here for wider awareness.

<https://issues.guix.gnu.org/63136>

Thanks,
Oleg

[-- Attachment #2: Type: text/html, Size: 425 bytes --]

Re: Python feature branch

[Andreas Enge]

Hello,

I wanted to set up automatic building on cuirass for the Python updates
branch, but was not sure which one it is:
$ git branch -a | grep python
  remotes/origin/python-updates
  remotes/origin/wip-python-graphviz
  remotes/origin/wip-python-mne
  remotes/origin/wip-python-pep517

Some of these have recent commits, others not; it would be nice if you
could clean them up and agree on which one to build.

Since we do not have enough aarch64 build power, I would have the branch
built only by the other architectures in the first place; we could add
aarch64 to build out the branch once we think that it is ready to be merged.

Andreas

Re: Python feature branch

[Lars-Dominik Braun]

Hi Andreas,

> I wanted to set up automatic building on cuirass for the Python updates
> branch, but was not sure which one it is:
> $ git branch -a | grep python
>   remotes/origin/python-updates
>   remotes/origin/wip-python-graphviz
>   remotes/origin/wip-python-mne
>   remotes/origin/wip-python-pep517

I doubt there is one yet. Per #63139 I was going to set up
“python-team”, which seems to be the naming consesus right now.

> Some of these have recent commits, others not; it would be nice if you
> could clean them up and agree on which one to build.

@Ricardo, Vivien: Can you confirm wip-python-graphviz and wip-python-mne
are stale/unused?

Cheers,
Lars

Re: Python feature branch

[Ricardo Wurmus]

Lars-Dominik Braun <lars@6xq.net> writes:

> @Ricardo, Vivien: Can you confirm wip-python-graphviz and wip-python-mne
> are stale/unused?

Yes, I don’t use them and don’t even remember them.  IIRC
wip-python-graphviz has long been merged, so it’s fine to delete it.

Thanks!

-- 
Ricardo

Re: Python feature branch

[John Kehayias]

Hi all,

On Mon, May 08, 2023 at 07:28 PM, Lars-Dominik Braun wrote:

> Hi Andreas,
>
>> I wanted to set up automatic building on cuirass for the Python updates
>> branch, but was not sure which one it is:
>> $ git branch -a | grep python
>>   remotes/origin/python-updates
>>   remotes/origin/wip-python-graphviz
>>   remotes/origin/wip-python-mne
>>   remotes/origin/wip-python-pep517
>
> I doubt there is one yet. Per #63139 I was going to set up
> “python-team”, which seems to be the naming consesus right now.
>

Right, I don't think we have a naming convention, but as per some
other threads we want to solidify this new process over the
staging/core-updates monolithic branches. We could add this as part of
the patch naming convention to ease finding patches and using QA for
testing/building. A straightforward "python-team" or "python-updates"
in general sounds good to me.

By the way, I'll comment in #63139, but I'll be away for several weeks
starting at the end of this week. I'll see what progress I can chip
away at before then and leave it in all these capable hands.

Thanks!
John