problem trying to install python module in python virtual environment on top of Guix

problem trying to install python module in python virtual environment on top of Guix

[Andy Tai]

Hi, on a Guix system I tried to install python dependencies in a
python virtual environment.   I installed Guix packages
python-toolchain and python-virtualenv, and use the basic python
commands virtualenv and pip3 to go ahead and install dependencies in a
typical python manner.

I got one error in a dependencies that tried to build C modules which
involves using cmake in this python environment: (not the system cmake
but something python stuff installed as part of its process):

Building wheels for collected packages: llama-cpp-python
  Building wheel for llama-cpp-python (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for llama-cpp-python (pyproject.toml) did not run
successfully.
  │ exit code: 1
  ╰─> [9 lines of output]
      Traceback (most recent call last):
        File "/tmp/pip-build-env-44zbk4iu/overlay/lib/python3.10/site-packages/skbuild/setuptools_wrap.py",
line 645, in setup
          cmkr = cmaker.CMaker(cmake_executable)
        File "/tmp/pip-build-env-44zbk4iu/overlay/lib/python3.10/site-packages/skbuild/cmaker.py",
line 148, in __init__
          self.cmake_version = get_cmake_version(self.cmake_executable)
        File "/tmp/pip-build-env-44zbk4iu/overlay/lib/python3.10/site-packages/skbuild/cmaker.py",
line 105, in get_cmake_version
          raise SKBuildError(msg) from err

      Problem with the CMake installation, aborting build. CMake
executable is /tmp/pip-build-env-44zbk4iu/overlay/lib/python3.10/site-packages/cmake/data/bin/cmake
      [end of output]

  note: This error originates from a subprocess, and is likely not a
problem with pip.
  ERROR: Failed building wheel for llama-cpp-python
Failed to build llama-cpp-python
ERROR: Could not build wheels for llama-cpp-python, which is required
to install pyproject.toml-based projects

[notice] A new release of pip is available: 23.1.2 -> 23.2.1
[notice] To update, run: pip install --upgrade pip


wonders if anyone knows if the above can be attempted on top of Guix;
not sure if the C build environment can be seen properly under Python
virtualenv


Also the Python tools packaged in Guix seem rather old, and tat can be
part of the issue here.

Re: problem trying to install python module in python virtual environment on top of Guix

[Maxim Cournoyer]

Hi Andy,

Andy Tai <atai@atai.org> writes:

> Hi, on a Guix system I tried to install python dependencies in a
> python virtual environment.   I installed Guix packages
> python-toolchain and python-virtualenv, and use the basic python
> commands virtualenv and pip3 to go ahead and install dependencies in a
> typical python manner.
>
> I got one error in a dependencies that tried to build C modules which
> involves using cmake in this python environment: (not the system cmake
> but something python stuff installed as part of its process):
>
> Building wheels for collected packages: llama-cpp-python
>   Building wheel for llama-cpp-python (pyproject.toml) ... error
>   error: subprocess-exited-with-error
>
>   × Building wheel for llama-cpp-python (pyproject.toml) did not run
> successfully.
>   │ exit code: 1
>   ╰─> [9 lines of output]
>       Traceback (most recent call last):
>         File "/tmp/pip-build-env-44zbk4iu/overlay/lib/python3.10/site-packages/skbuild/setuptools_wrap.py",
> line 645, in setup
>           cmkr = cmaker.CMaker(cmake_executable)
>         File "/tmp/pip-build-env-44zbk4iu/overlay/lib/python3.10/site-packages/skbuild/cmaker.py",
> line 148, in __init__
>           self.cmake_version = get_cmake_version(self.cmake_executable)
>         File "/tmp/pip-build-env-44zbk4iu/overlay/lib/python3.10/site-packages/skbuild/cmaker.py",
> line 105, in get_cmake_version
>           raise SKBuildError(msg) from err
>
>       Problem with the CMake installation, aborting build. CMake
> executable is /tmp/pip-build-env-44zbk4iu/overlay/lib/python3.10/site-packages/cmake/data/bin/cmake
>       [end of output]
>
>   note: This error originates from a subprocess, and is likely not a
> problem with pip.
>   ERROR: Failed building wheel for llama-cpp-python
> Failed to build llama-cpp-python
> ERROR: Could not build wheels for llama-cpp-python, which is required
> to install pyproject.toml-based projects
>
> [notice] A new release of pip is available: 23.1.2 -> 23.2.1
> [notice] To update, run: pip install --upgrade pip
>
>
> wonders if anyone knows if the above can be attempted on top of Guix;
> not sure if the C build environment can be seen properly under Python
> virtualenv

I've had success in the past using the --emulate-fhs option in a
containerized shell, but the better option would be to package what you
need in Guix proper.

> Also the Python tools packaged in Guix seem rather old, and tat can be
> part of the issue here.

I'm sure the Python team would welcome new members :-).

-- 
Thanks,
Maxim

Re: problem trying to install python module in python virtual environment on top of Guix

[Georgios Athanasiou]

Hi Andy,

I think that it may be possible to build the package with a little 
effort. You'll have to emulate the FHS as Maxim suggested. But you'll 
also have to expose the standard C/C++ libraries (they are currently 
hidden, see https://issues.guix.gnu.org/63267).

Save the following Scheme code as gcc-unhidden.scm in a directory:

(define-module (gcc-unhidden)
   #:use-module (gnu packages gcc)
   #:use-module (guix packages)
   #:use-module (srfi srfi-1))

(define-public gcc-12-unhidden
   (package
    (inherit gcc-12)
    (properties (alist-delete 'hidden? (package-properties gcc-12)))))

That will make the gcc:lib output available. You can use it in a Guix 
container along with the necessary tools. This is how I'd do it:

guix shell --load-path=. --container --emulate-fhs --network \
      bash coreutils gcc-toolchain gcc:lib make python

You can now create a Python virtual environment inside the Guix 
container and continue as usual.

Kind regards,

George

Re: problem trying to install python module in python virtual environment on top of Guix

[Georgios Athanasiou]

Hi Andy,

I think that it may be possible to build the package with a little 
effort. You'll have to emulate the FHS as Maxim suggested. But you'll 
also have to expose the standard C/C++ libraries (they are currently 
hidden, see https://issues.guix.gnu.org/63267).

Save the following Scheme code as gcc-unhidden.scm in a directory:

(define-module (gcc-unhidden)
   #:use-module (gnu packages gcc)
   #:use-module (guix packages)
   #:use-module (srfi srfi-1))

(define-public gcc-12-unhidden
   (package
    (inherit gcc-12)
    (properties (alist-delete 'hidden? (package-properties gcc-12)))))

That will make the gcc:lib output available. You can use it in a Guix 
container along with the necessary tools. This is how I'd do it:

guix shell --load-path=. --container --emulate-fhs --network \
      bash coreutils gcc-toolchain gcc:lib make python

You can now create a Python virtual environment inside the Guix 
container and continue as usual.

Kind regards,

George