Problem loading numpy and other on a venv in a guix system

Problem loading numpy and other on a venv in a guix system

[mprodrigues]

Hello,

I've been using a guix shell with a couple of python packages and it 
works fine,
however I'd also like to keep the option of using a virtualenv if 
possible, the
thing is I'm having problem running modules like numpy, that depend on 
non
python dependencies. The error I get when I try to load numpy in a venv 
is:

Importing the numpy C-extensions failed. This error can happen for many 
reasons,
often due to issues with your setup or how NumPy was installed.

and in the end:

Original error was: libz.so.1: cannot open shared object file: No such 
file or
directory

I have tried installing zlib directly, also tried to add my default 
profile libs
path to the venv PATH, but to no avail until now.  How can I solve this? 
Or
maybe someone can point me in the direction that I have to take to 
properly
debug this.

Best regards,
Mateus Rodrigues

Re: Problem loading numpy and other on a venv in a guix system

[Wojtek Kosior via]

[-- Attachment #1: Type: text/plain, Size: 2202 bytes --]

Hi Mateus!

Indeed, when using virtualenv's numpy, this

    python3 -c 'import numpy'

fails with the error you posted. I seem to have been able to make numpy
happy by instead running Python like

    LD_LIBRARY_PATH=/gnu/store/8qv5kb2fgm4c3bf70zcg9l6hkf3qzpw9-zlib-1.2.11/lib/ python3 -c 'import numpy'

If you use the `--container` and `--emulate-fhs` options to
`guix shell`, it is also possible to instead do

    LD_LIBRARY_PATH=/usr/lib/ python3 -c 'import numpy'

The `libz` guix package still needs to be included among the packages
passed to `guix shell` in this last case.

Also, given you're using virtualenv, you might find the `--network`
option to `guix shell` useful :)

I hope there isn't any dumb mistake in my solution... Good luck :)

Wojtek

-- (sig_start)
website: https://koszko.org/koszko.html
PGP: https://koszko.org/key.gpg
fingerprint: E972 7060 E3C5 637C 8A4F  4B42 4BC5 221C 5A79 FD1A

Meet Kraków saints!           #27: blessed Maria Klemensa Staszewska
Poznaj świętych krakowskich!  #27: błogosławiona Maria Klemensa Staszewska
https://pl.wikipedia.org/wiki/Maria_Klemensa_Staszewska
-- (sig_end)


On Tue, 29 Nov 2022 16:42:14 +0000
mprodrigues@posteo.net wrote:

> Hello,
> 
> I've been using a guix shell with a couple of python packages and it 
> works fine,
> however I'd also like to keep the option of using a virtualenv if 
> possible, the
> thing is I'm having problem running modules like numpy, that depend on 
> non
> python dependencies. The error I get when I try to load numpy in a venv 
> is:
> 
> Importing the numpy C-extensions failed. This error can happen for many 
> reasons,
> often due to issues with your setup or how NumPy was installed.
> 
> and in the end:
> 
> Original error was: libz.so.1: cannot open shared object file: No such 
> file or
> directory
> 
> I have tried installing zlib directly, also tried to add my default 
> profile libs
> path to the venv PATH, but to no avail until now.  How can I solve this? 
> Or
> maybe someone can point me in the direction that I have to take to 
> properly
> debug this.
> 
> Best regards,
> Mateus Rodrigues
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: Problem loading numpy and other on a venv in a guix system

[mprodrigues]

Hi Wojtek,

thanks for your help, indeed setting LD_LIBRARY_PATH solves the issue, 
but it is less than ideal as I have to first
check exactly the PATH to set, and if I update guix this will also 
change I believe, still it is not so bad. I'd still like
to understand why this happens in the first place.

Best regards,
Mateus Rodrigues

On 29.11.2022 19:33, Wojtek Kosior wrote:
> Hi Mateus!
> 
> Indeed, when using virtualenv's numpy, this
> 
>     python3 -c 'import numpy'
> 
> fails with the error you posted. I seem to have been able to make numpy
> happy by instead running Python like
> 
> 
> LD_LIBRARY_PATH=/gnu/store/8qv5kb2fgm4c3bf70zcg9l6hkf3qzpw9-zlib-1.2.11/lib/
> python3 -c 'import numpy'
> 
> If you use the `--container` and `--emulate-fhs` options to
> `guix shell`, it is also possible to instead do
> 
>     LD_LIBRARY_PATH=/usr/lib/ python3 -c 'import numpy'
> 
> The `libz` guix package still needs to be included among the packages
> passed to `guix shell` in this last case.
> 
> Also, given you're using virtualenv, you might find the `--network`
> option to `guix shell` useful :)
> 
> I hope there isn't any dumb mistake in my solution... Good luck :)
> 
> Wojtek
> 
> -- (sig_start)
> website: https://koszko.org/koszko.html
> PGP: https://koszko.org/key.gpg
> fingerprint: E972 7060 E3C5 637C 8A4F  4B42 4BC5 221C 5A79 FD1A
> 
> Meet Kraków saints!           #27: blessed Maria Klemensa Staszewska
> Poznaj świętych krakowskich!  #27: błogosławiona Maria Klemensa 
> Staszewska
> https://pl.wikipedia.org/wiki/Maria_Klemensa_Staszewska
> -- (sig_end)
> 
> 
> On Tue, 29 Nov 2022 16:42:14 +0000
> mprodrigues@posteo.net wrote:
> 
>> Hello,
>> 
>> I've been using a guix shell with a couple of python packages and it
>> works fine,
>> however I'd also like to keep the option of using a virtualenv if
>> possible, the
>> thing is I'm having problem running modules like numpy, that depend on
>> non
>> python dependencies. The error I get when I try to load numpy in a 
>> venv
>> is:
>> 
>> Importing the numpy C-extensions failed. This error can happen for 
>> many
>> reasons,
>> often due to issues with your setup or how NumPy was installed.
>> 
>> and in the end:
>> 
>> Original error was: libz.so.1: cannot open shared object file: No such
>> file or
>> directory
>> 
>> I have tried installing zlib directly, also tried to add my default
>> profile libs
>> path to the venv PATH, but to no avail until now.  How can I solve 
>> this?
>> Or
>> maybe someone can point me in the direction that I have to take to
>> properly
>> debug this.
>> 
>> Best regards,
>> Mateus Rodrigues
>> 

Re: Problem loading numpy and other on a venv in a guix system

[Wojtek Kosior via]

[-- Attachment #1: Type: text/plain, Size: 6588 bytes --]

Hello,


> Hi Wojtek,
> 
> thanks for your help, indeed setting LD_LIBRARY_PATH solves the issue, 
> but it is less than ideal as I have to first
> check exactly the PATH to set, and if I update guix this will also 
> change I believe, still it is not so bad. I'd still like
> to understand why this happens in the first place.
> 
> Best regards,
> Mateus Rodrigues

I see.

I don't think I realize all the nuances of dynamic library loading
myself. But I can try to explain what I know :)

In C (also to some extent C++) world, libraries can be either linked
statically into a program (i.e. included in its executable) or linked
dynamically (i.e. so that they are found in the filesystem when program
gets executed). There are pros and cons to both approaches.

With dynamically-linked libraries (also called shared libraries), it is
possible to write a program so that it doesn't load them immediately
upon start. It is possible to instead have program's code call a C
library function that loads a shared library *by name*, together with
other shared libraries it depends on. This is what the Python
interpreted process tries to do when told to load Numpy's C extension.

Under a typical GNU/Linux system, shared libraries are searched for in
some predefined locations (like `/lib/x86_64-linux-gnu/` which is where
`libz.so.1` resides in my Debian OS). AFAIK, these locations are
compiled into the C library and cannot be *changed*. However, they can
be *supplemented* by paths in the `LD_LIBRARY_PATH` variable which are
searched by the C library *before* the standard locations.

Now, Guix does things differently. It does not keep its libraries in
standard locations like `/lib`. It cannot - the assumption is that
files needed by programs are kept under `/gnu/store`. So, AFAIU, the C
library in Guix is *modified* not to assume any standard directories
for shared libraries. Instead, all programs have the exact locations of
their shared libraries specified directly in their executables. This is
possible with the help of a (perhaps a bit obscure) feature called
`RPATH`.

So we already know that Guix' own packages make up for the lack of
standard shared library directories. How about binaries like those
from PyPI that were not compiled for Guix? They don't have the RPATH
set and they expect libc to be able to resolve some of their
dependencies. In particular, Python C extensions devs used to be rather
safe to assume that zlib (a dependency of CPython) would be present and
resolvable in the user's operating system and need not be bundled with
the extension.

If we think about it, an analogous problem appears for many languages,
not just C. And it seems Guix solves it slightly differently there. For
example, Python library locations are specified in Guix-prepared
variable `GUIX_PYTHONPATH`. I believe that works more reliably than C's
`RPATH` (although I had some issues with loading of Python libs, too).

Back to your problem - symlinks to shared libraries are also placed in
the `lib/` subdirectory of a Guix profile. For example, after doing

    guix install zlib

I got a `libz.so.1` inside `~/.guix-profile/lib`. Perhaps this will
prove helpful to you? :)

Wojtek

-- (sig_start)
website: https://koszko.org/koszko.html
PGP: https://koszko.org/key.gpg
fingerprint: E972 7060 E3C5 637C 8A4F  4B42 4BC5 221C 5A79 FD1A

Meet Kraków saints!           #39: blessed Narcyz Turchan
Poznaj świętych krakowskich!  #39: błogosławiony Narcyz Turchan
https://pl.wikipedia.org/wiki/Narcyz_Turchan
-- (sig_end)


On Wed, 30 Nov 2022 09:42:34 +0000
mprodrigues@posteo.net wrote:

> Hi Wojtek,
> 
> thanks for your help, indeed setting LD_LIBRARY_PATH solves the issue, 
> but it is less than ideal as I have to first
> check exactly the PATH to set, and if I update guix this will also 
> change I believe, still it is not so bad. I'd still like
> to understand why this happens in the first place.
> 
> Best regards,
> Mateus Rodrigues
> 
> On 29.11.2022 19:33, Wojtek Kosior wrote:
> > Hi Mateus!
> > 
> > Indeed, when using virtualenv's numpy, this
> > 
> >     python3 -c 'import numpy'
> > 
> > fails with the error you posted. I seem to have been able to make numpy
> > happy by instead running Python like
> > 
> > 
> > LD_LIBRARY_PATH=/gnu/store/8qv5kb2fgm4c3bf70zcg9l6hkf3qzpw9-zlib-1.2.11/lib/
> > python3 -c 'import numpy'
> > 
> > If you use the `--container` and `--emulate-fhs` options to
> > `guix shell`, it is also possible to instead do
> > 
> >     LD_LIBRARY_PATH=/usr/lib/ python3 -c 'import numpy'
> > 
> > The `libz` guix package still needs to be included among the packages
> > passed to `guix shell` in this last case.
> > 
> > Also, given you're using virtualenv, you might find the `--network`
> > option to `guix shell` useful :)
> > 
> > I hope there isn't any dumb mistake in my solution... Good luck :)
> > 
> > Wojtek
> > 
> > -- (sig_start)
> > website: https://koszko.org/koszko.html
> > PGP: https://koszko.org/key.gpg
> > fingerprint: E972 7060 E3C5 637C 8A4F  4B42 4BC5 221C 5A79 FD1A
> > 
> > Meet Kraków saints!           #27: blessed Maria Klemensa Staszewska
> > Poznaj świętych krakowskich!  #27: błogosławiona Maria Klemensa 
> > Staszewska
> > https://pl.wikipedia.org/wiki/Maria_Klemensa_Staszewska
> > -- (sig_end)
> > 
> > 
> > On Tue, 29 Nov 2022 16:42:14 +0000
> > mprodrigues@posteo.net wrote:
> >   
> >> Hello,
> >> 
> >> I've been using a guix shell with a couple of python packages and it
> >> works fine,
> >> however I'd also like to keep the option of using a virtualenv if
> >> possible, the
> >> thing is I'm having problem running modules like numpy, that depend on
> >> non
> >> python dependencies. The error I get when I try to load numpy in a 
> >> venv
> >> is:
> >> 
> >> Importing the numpy C-extensions failed. This error can happen for 
> >> many
> >> reasons,
> >> often due to issues with your setup or how NumPy was installed.
> >> 
> >> and in the end:
> >> 
> >> Original error was: libz.so.1: cannot open shared object file: No such
> >> file or
> >> directory
> >> 
> >> I have tried installing zlib directly, also tried to add my default
> >> profile libs
> >> path to the venv PATH, but to no avail until now.  How can I solve 
> >> this?
> >> Or
> >> maybe someone can point me in the direction that I have to take to
> >> properly
> >> debug this.
> >> 
> >> Best regards,
> >> Mateus Rodrigues
> >>   



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: Problem loading numpy and other on a venv in a guix system

[zimoun]

Hi,

On Tue, 29 Nov 2022 at 16:42, mprodrigues@posteo.net wrote:

> Importing the numpy C-extensions failed. This error can happen for many 
> reasons,
> often due to issues with your setup or how NumPy was installed.

Well, I am not sure to understand your configuration.  You are
installing some packages (say python and python-numpy) using “guix
shell”  and then installing others using pip loaded with virtualenv.
Right?


Cheers,
simon

Re: Problem loading numpy and other on a venv in a guix system

[Wojtek Kosior via]

[-- Attachment #1: Type: text/plain, Size: 1736 bytes --]

> Hi,
> 
> On Tue, 29 Nov 2022 at 16:42, mprodrigues@posteo.net wrote:
> 
> > Importing the numpy C-extensions failed. This error can happen for many 
> > reasons,
> > often due to issues with your setup or how NumPy was installed.  
> 
> Well, I am not sure to understand your configuration.  You are
> installing some packages (say python and python-numpy) using “guix
> shell”  and then installing others using pip loaded with virtualenv.
> Right?
> 
> 
> Cheers,
> simon
> 

I believe Mateus did something like

    guix shell python python-virtualenv coreutils

followed by

    virtualenv -p python3 somedirectory
    . ./somedirectory/bin/activate
    pip install numpy

This is more or less what I used to reproduce the issue

Wojtek

-- (sig_start)
website: https://koszko.org/koszko.html
PGP: https://koszko.org/key.gpg
fingerprint: E972 7060 E3C5 637C 8A4F  4B42 4BC5 221C 5A79 FD1A

Meet Kraków saints!           #47: saint Stanisław ze Szczepanowa
Poznaj świętych krakowskich!  #47: święty Stanisław ze Szczepanowa
https://pl.wikipedia.org/wiki/Stanisław_ze_Szczepanowa
-- (sig_end)


On Wed, 30 Nov 2022 09:13:03 +0100
zimoun <zimon.toutoune@gmail.com> wrote:

> Hi,
> 
> On Tue, 29 Nov 2022 at 16:42, mprodrigues@posteo.net wrote:
> 
> > Importing the numpy C-extensions failed. This error can happen for many 
> > reasons,
> > often due to issues with your setup or how NumPy was installed.  
> 
> Well, I am not sure to understand your configuration.  You are
> installing some packages (say python and python-numpy) using “guix
> shell”  and then installing others using pip loaded with virtualenv.
> Right?
> 
> 
> Cheers,
> simon
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: Problem loading numpy and other on a venv in a guix system

[mprodrigues]

Hello,

On 30.11.2022 09:47, Wojtek Kosior wrote:

> I believe Mateus did something like
> 
>     guix shell python python-virtualenv coreutils
> 
> followed by
> 
>     virtualenv -p python3 somedirectory
>     . ./somedirectory/bin/activate
>     pip install numpy
> 
> This is more or less what I used to reproduce the issue

not quite, but I guess it doesn't make all that much difference to 
reproduce the error.
I use either guix shell, and install all python packages through guix, 
or the venv, and
install all modules through pip, not both at the same time. Although, 
maybe using both
at the same time would not be bad.

Best regards,
Mateus Rodrigues

Re: Problem loading numpy and other on a venv in a guix system

[zimoun]

Hi,

On Wed, 30 Nov 2022 at 09:46, mprodrigues@posteo.net wrote:

>> This is more or less what I used to reproduce the issue
>
> not quite, but I guess it doesn't make all that much difference to 
> reproduce the error.

I think it makes difference. :-)

> I use either guix shell, and install all python packages through guix, 
> or the venv, and
> install all modules through pip, not both at the same time. Although, 
> maybe using both
> at the same time would not be bad.

From my experience, you cannot mix both “guix shell” and “pip“ easily.
For what it is worth, try to use both at the same time is the beginning
of all sort of annoyances.  See Wojtek message [1].

1: <https://yhetil.org/guix/20221130122110.708c7e20@koszkonutek-tmp.pl.eu.org>


Cheers,
simon

Re: Problem loading numpy and other on a venv in a guix system

[zimoun]

Hi,

On Wed, 30 Nov 2022 at 09:47, Wojtek Kosior via <help-guix@gnu.org> wrote:

      cd /tmp/
>     guix shell python python-virtualenv coreutils
>     virtualenv -p python3 somedirectory
>     . ./somedirectory/bin/activate
>     pip install numpy
      python3 -c 'import numpy'

leads to,

--8<---------------cut here---------------start------------->8---
Traceback (most recent call last):
  File "/tmp/somedirectory/lib/python3.9/site-packages/numpy/core/__init__.py", line 23, in <module>
    from . import multiarray
  File "/tmp/somedirectory/lib/python3.9/site-packages/numpy/core/multiarray.py", line 10, in <module>
    from . import overrides
  File "/tmp/somedirectory/lib/python3.9/site-packages/numpy/core/overrides.py", line 6, in <module>
    from numpy.core._multiarray_umath import (
ImportError: libz.so.1: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/somedirectory/lib/python3.9/site-packages/numpy/__init__.py", line 140, in <module>
    from . import core
  File "/tmp/somedirectory/lib/python3.9/site-packages/numpy/core/__init__.py", line 49, in <module>
    raise ImportError(msg)
ImportError: 

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.9 from "/tmp/somedirectory/bin/python3"
  * The NumPy version is: "1.23.5"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: libz.so.1: cannot open shared object file: No such file or directory
--8<---------------cut here---------------end--------------->8---

That’s because the Numpy from pip is linked to system-wide library,

--8<---------------cut here---------------start------------->8---
$ ldd somedirectory/lib/python3.9/site-packages/numpy/core/_multiarray_umath.cpython-39-x86_64-linux-gnu.so
	linux-vdso.so.1 (0x00007ffde12e0000)
	libopenblas64_p-r0-742d56dc.3.20.so => /tmp/somedirectory/lib/python3.9/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-742d56dc.3.20.so (0x00007f6f1223d000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f6f120dd000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f6f120ba000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f6f11ec8000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f6f1481a000)
	libgfortran-040039e1.so.5.0.0 => /tmp/somedirectory/lib/python3.9/site-packages/numpy/core/../../numpy.libs/libgfortran-040039e1.so.5.0.0 (0x00007f6f11a3b000)
	libquadmath-96973f99.so.0.0.0 => /tmp/somedirectory/lib/python3.9/site-packages/numpy/core/../../numpy.libs/libquadmath-96973f99.so.0.0.0 (0x00007f6f117fc000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f6f117e0000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f6f117c5000)
--8<---------------cut here---------------end--------------->8---

and they are not found.  Well, if you are running Guix on foreign
distro, you can manually fix the issue with this “missing” libz library
by setting LD_LIBRARY_PATH=/lib/x86_64-linux-gnu/.  But you get another
issue:

--8<---------------cut here---------------start------------->8---
$ LD_LIBRARY_PATH=/lib/x86_64-linux-gnu/ python3 -c 'import numpy'
python3: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /gnu/store/lvip6h5pamjwmvnkwg60sjb63ph8698k-python-3.9.9/lib/libpython3.9.so.1.0)
python3: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /gnu/store/lvip6h5pamjwmvnkwg60sjb63ph8698k-python-3.9.9/lib/libpython3.9.so.1.0)
--8<---------------cut here---------------end--------------->8---

Well, I do not know if the Guix package “python-virtualenv” is working
in “guix shell”.


Cheers,
simon