[bug#74715] Request for merging "python-team" branch

[bug#74715] Request for merging "python-team" branch

[Sharlatan Hellseher]

[-- Attachment #1: Type: text/plain, Size: 410 bytes --]


Hi Guix!

I think it's a time to review and merge python-team branch.

It will bring refactored pyproject-build-system, refreshed collection of
packages from python-build module, refreshed python-numpy (1.24),
default python-pytest (8+), fresh python-flask, some clean up from
python-nose which is deprecated for few years.

Please test and report for any issues of the package you
love/need/demand.

--
Oleg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Request for merging "python-team" branch

[Sharlatan Hellseher]

[-- Attachment #1: Type: text/plain, Size: 34 bytes --]


CC Python team for wider spread.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: bug#74715: Request for merging "python-team" branch

[Ludovic Courtès]

Hey there,

https://qa.guix.gnu.org/ says it’s your turn to merge!  :-)

Ludo’.

Re: bug#74715: Request for merging "python-team" branch

[jgart]

[-- Attachment #1: Type: text/plain, Size: 79 bytes --]

Hi Sharlatan,

Guix cheerleader here. Go for it! Merge ahoy!
🦜🦆
 LGTM

[-- Attachment #2: Type: text/html, Size: 297 bytes --]

Re: bug#74715: Request for merging "python-team" branch

[Ian Eure]

[-- Attachment #1: Type: text/plain, Size: 3679 bytes --]

Hi all,

Since this merge landed, the builds for several Python packages in my personal channel broke.  Any package using pyproject-build-system for a Python project using setuptools seems to be affected.  This python-manhole package[1] is an example.  It's  about as simple as they can get, with no inputs or custom build steps.  It's failing with:

------------------------------------------------------------
starting phase `build'
Using 'setuptools.build_meta' to build wheels, auto-detected '#f', override '#f'.
Prepending '[]' to sys.path, auto-detected '#f', override '#f'.
Traceback (most recent call last):
  File "<string>", line 6, in <module>
  File "/gnu/store/jjcka1g6sk2cvwx8nm4fdwpdq3vll0v0-python-3.10.7/lib/python3.10/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'setuptools'
error: in phase 'build': uncaught exception:
%exception #<&invoke-error program: "python" arguments: ("-c" "import sys, importlib, json\nbackend_path = json.loads (sys.argv[1]) or []\nbackend_path.extend (sys.path)\nsys.path = backend_path\nconfig_settings = json.loads (sys.argv[4])\nbuilder = importlib.import_module(sys.argv[2])\nbuilder.build_wheel(sys.argv[3], config_settings=config_settings)" "[]" "setuptools.build_meta" "dist" "{}") exit-status: 1 term-signal: #f stop-signal: #f> 
phase `build' failed after 0.0 seconds
command "python" "-c" "import sys, importlib, json\nbackend_path = json.loads (sys.argv[1]) or []\nbackend_path.extend (sys.path)\nsys.path = backend_path\nconfig_settings = json.loads (sys.argv[4])\nbuilder = importlib.import_module(sys.argv[2])\nbuilder.build_wheel(sys.argv[3], config_settings=config_settings)" "[]" "setuptools.build_meta" "dist" "{}" failed with status 1
build process 10 exited with status 256
------------------------------------------------------------

Since it's complaining about setuptools, I thought I might need to add that to the native-inputs.  Alas, that fails with a different error:

------------------------------------------------------------
starting phase `build'
Using 'setuptools.build_meta' to build wheels, auto-detected '#f', override '#f'.
Prepending '[]' to sys.path, auto-detected '#f', override '#f'.
usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: -c --help [cmd1 cmd2 ...]
   or: -c --help-commands
   or: -c cmd --help

error: invalid command 'bdist_wheel'
------------------------------------------------------------

Is there somewhere I can find out how to fix these packages for the updated pyproject-build-system?  Should they be getting switched to python-build-system?

Noting also that the PyPI module for `guix import' hardcodes the pyproject-build-system, so it will generate unbuildable definitions for any Python project which uses setuptools.

Thanks,

  -- Ian

[1]: https://codeberg.org/ieure/atomized-guix/src/branch/main/atomized/packages/python-xyz.scm#L22

On Fri, Dec 13, 2024, at 5:00 PM, jgart wrote:
> Hi Sharlatan,
> 
> Guix cheerleader here. Go for it! Merge ahoy!
> 🦜🦆
> LGTM
> 
> 

[-- Attachment #2: Type: text/html, Size: 5834 bytes --]

Re: bug#74715: Request for merging "python-team" branch

[Kaelyn via Development of GNU Guix and the GNU System distribution.]

Hi Ian,

On Saturday, December 14th, 2024 at 5:00 PM, Ian Eure <ian@retrospec.tv> wrote:

> Hi all,
> 
> Since this merge landed, the builds for several Python packages in my personal channel broke.  Any package using pyproject-build-system for a Python project using setuptools seems to be affected.  This python-manhole package[1] is an example.  It's  about as simple as they can get, with no inputs or custom build steps.  It's failing with:
> 
> ------------------------------------------------------------
> starting phase `build'
> Using 'setuptools.build_meta' to build wheels, auto-detected '#f', override '#f'.
> Prepending '[]' to sys.path, auto-detected '#f', override '#f'.
> Traceback (most recent call last):
>   File "<string>", line 6, in <module>
>   File "/gnu/store/jjcka1g6sk2cvwx8nm4fdwpdq3vll0v0-python-3.10.7/lib/python3.10/importlib/__init__.py", line 126, in import_module
> return _bootstrap._gcd_import(name[level:], package, level)
>   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
>   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
>   File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
>   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
>   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
>   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
>   File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
> ModuleNotFoundError: No module named 'setuptools'
> error: in phase 'build': uncaught exception:
> %exception #<&invoke-error program: "python" arguments: ("-c" "import sys, importlib, json\nbackend_path = json.loads (sys.argv[1]) or []\nbackend_path.extend (sys.path)\nsys.path = backend_path\nconfig_settings = json.loads (sys.argv[4])\nbuilder = importlib.import_module(sys.argv[2])\nbuilder.build_wheel(sys.argv[3], config_settings=config_settings)" "[]" "setuptools.build_meta" "dist" "{}") exit-status: 1 term-signal: #f stop-signal: #f> 
> phase `build' failed after 0.0 seconds
> command "python" "-c" "import sys, importlib, json\nbackend_path = json.loads (sys.argv[1]) or []\nbackend_path.extend (sys.path)\nsys.path = backend_path\nconfig_settings = json.loads (sys.argv[4])\nbuilder = importlib.import_module(sys.argv[2])\nbuilder.build_wheel(sys.argv[3], config_settings=config_settings)" "[]" "setuptools.build_meta" "dist" "{}" failed with status 1
> build process 10 exited with status 256
> ------------------------------------------------------------
> 
> Since it's complaining about setuptools, I thought I might need to add that to the native-inputs.  Alas, that fails with a different error:
> 
> ------------------------------------------------------------
> starting phase `build'
> Using 'setuptools.build_meta' to build wheels, auto-detected '#f', override '#f'.
> Prepending '[]' to sys.path, auto-detected '#f', override '#f'.
> usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
>    or: -c --help [cmd1 cmd2 ...]
>    or: -c --help-commands
>    or: -c cmd --help
> 
> error: invalid command 'bdist_wheel'
> ------------------------------------------------------------

I don't have any comment about the pyproject-build-system issue in general, but I've done enough maintenance of Python packages in general to be familiar with this error. It should be as simple as adding `python-wheel` to the native-inputs--it is the module that allows setuptools to generate .whl files, and I'd be surprised if additional packages were needed (I've encountered having to add explicit setuptools and wheel dependencies to requirements.txt files for pip when changing the build environment of packages such as changing distros or Python versions). HTH!

Cheers,
Kaelyn

> Is there somewhere I can find out how to fix these packages for the updated pyproject-build-system?  Should they be getting switched to python-build-system?
> 
> Noting also that the PyPI module for `guix import' hardcodes the pyproject-build-system, so it will generate unbuildable definitions for any Python project which uses setuptools.
> 
> Thanks,
> 
>   -- Ian
> 
> [1]: https://codeberg.org/ieure/atomized-guix/src/branch/main/atomized/packages/python-xyz.scm#L22
> 
> On Fri, Dec 13, 2024, at 5:00 PM, jgart wrote:
> 
> > Hi Sharlatan,
> > 
> > Guix cheerleader here. Go for it! Merge ahoy!
> > 🦜🦆
> > LGTM
> > 
> > 
> 
>

Re: bug#74715: Request for merging "python-team" branch

[Lars-Dominik Braun]

Hi Ian,

> Since this merge landed, the builds for several Python packages in my personal channel broke.  Any package using pyproject-build-system for a Python project using setuptools seems to be affected.

as Sharlatan Hellseher wrote in https://issues.guix.gnu.org/issue/74715#4,
you need to add python-setuptools and python-wheel to your
setuptools-based packages. The default python toolchain used by
pyproject-build-system (python-sans-pip-wrapper from
gnu/packages/python.scm) does not include these packages any more,
since they are technically not required and declaring them as *real*
inputs allows using different versions of these packages more easily
for packages, which require them. Plus there are quite a few packages,
which build using different build systems nowadays.

The python importer should probably be updated to read pyproject.toml
and parse the [build-system] table (there is a toml parser in Guix now,
so this should be easy).

Lars

Re: bug#74715: Request for merging "python-team" branch

[Sharlatan Hellseher]

[-- Attachment #1: Type: text/plain, Size: 86 bytes --]

Hi,

Maybe we may put some news about changes in pyproject build system?

Thanks
Oleg

[-- Attachment #2: Type: text/html, Size: 224 bytes --]

Re: bug#74715: Request for merging "python-team" branch

[Leo Famulari]

On Sun, Dec 15, 2024 at 10:29:03AM +0000, Sharlatan Hellseher wrote:
> Hi,
> 
> Maybe we may put some news about changes in pyproject build system?

I think that's a great idea.

Re: bug#74715: Request for merging "python-team" branch

[Ian Eure]

Hi Lars-Dominik,

On Sun, Dec 15, 2024, at 6:28 AM, Lars-Dominik Braun wrote:
> Hi Ian,
>
>> Since this merge landed, the builds for several Python packages in my personal channel broke.  Any package using pyproject-build-system for a Python project using setuptools seems to be affected.
>
> as Sharlatan Hellseher wrote in https://issues.guix.gnu.org/issue/74715#4,
> you need to add python-setuptools and python-wheel to your
> setuptools-based packages. The default python toolchain used by
> pyproject-build-system (python-sans-pip-wrapper from
> gnu/packages/python.scm) does not include these packages any more,
> since they are technically not required and declaring them as *real*
> inputs allows using different versions of these packages more easily
> for packages, which require them. Plus there are quite a few packages,
> which build using different build systems nowadays.
>

Thanks, this worked for me.  I skimmed the related bug, but missed this comment.  I think the docs for pyproject-build-system are likely the best place for this, as they already mention some of the setuptools/pyproject interaction.  I sent a patch (#74899) with some draft language, let me know what you think.

> The python importer should probably be updated to read pyproject.toml
> and parse the [build-system] table (there is a toml parser in Guix now,
> so this should be easy).
>

Would it be helpful to open a bug about this?

Thanks,

  -- Ian

Re: bug#74715: Request for merging "python-team" branch

[Cayetano Santos]

[-- Attachment #1: Type: text/plain, Size: 438 bytes --]


>dim. 15 déc. 2024 at 17:53, "Ian Eure" <ian@retrospec.tv> wrote:

> Would it be helpful to open a bug about this?

I confirm, this line

    (inputs (list python-setuptools python-wheel))

fixes the issue for me.

However, ‘guix lint’ complaints about python-setuptools not necessary.

--
Cayetano Santos
GnuPG Key:   https://meta.sr.ht/~csantosb.pgp
FingerPrint: CCB8 1842 F9D7 058E CD67 377A BF5C DF4D F6BF 6682

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 259 bytes --]

Re: bug#74715: Request for merging "python-team" branch

[Lars-Dominik Braun]

Hi,

> Would it be helpful to open a bug about this?

there is https://issues.guix.gnu.org/69997, which includes patches to
add initial pyproject.toml support to `guix import pypi`.

Lars

bug#74715: Request for merging "python-team" branch

[Sharlatan Hellseher]

[-- Attachment #1: Type: text/plain, Size: 439 bytes --]


Hi,

python-team branch has been rebased, conflicts resolved (python-redis
and python-pydevd) and merged to master.

Long lists of rebuild are in CI right now:

- <https://ci.guix.gnu.org/eval/1903757>
- <https://ci.guix.gnu.org/eval/1903820>

Post merge fix-up might be required like adding missing build time
dependencies (python-setuptools, python-wheel) for
pyproject-build-system packages not refreshed for a while.

--
Thanks,
Oleg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]