* Problem loading numpy and other on a venv in a guix system @ 2022-11-29 16:42 mprodrigues 2022-11-29 18:33 ` Wojtek Kosior via 2022-11-30 8:13 ` zimoun 0 siblings, 2 replies; 9+ messages in thread From: mprodrigues @ 2022-11-29 16:42 UTC (permalink / raw) To: help-guix Hello, I've been using a guix shell with a couple of python packages and it works fine, however I'd also like to keep the option of using a virtualenv if possible, the thing is I'm having problem running modules like numpy, that depend on non python dependencies. The error I get when I try to load numpy in a venv is: Importing the numpy C-extensions failed. This error can happen for many reasons, often due to issues with your setup or how NumPy was installed. and in the end: Original error was: libz.so.1: cannot open shared object file: No such file or directory I have tried installing zlib directly, also tried to add my default profile libs path to the venv PATH, but to no avail until now. How can I solve this? Or maybe someone can point me in the direction that I have to take to properly debug this. Best regards, Mateus Rodrigues ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Problem loading numpy and other on a venv in a guix system 2022-11-29 16:42 Problem loading numpy and other on a venv in a guix system mprodrigues @ 2022-11-29 18:33 ` Wojtek Kosior via 2022-11-30 9:42 ` mprodrigues 2022-11-30 8:13 ` zimoun 1 sibling, 1 reply; 9+ messages in thread From: Wojtek Kosior via @ 2022-11-29 18:33 UTC (permalink / raw) To: mprodrigues; +Cc: help-guix [-- Attachment #1: Type: text/plain, Size: 2202 bytes --] Hi Mateus! Indeed, when using virtualenv's numpy, this python3 -c 'import numpy' fails with the error you posted. I seem to have been able to make numpy happy by instead running Python like LD_LIBRARY_PATH=/gnu/store/8qv5kb2fgm4c3bf70zcg9l6hkf3qzpw9-zlib-1.2.11/lib/ python3 -c 'import numpy' If you use the `--container` and `--emulate-fhs` options to `guix shell`, it is also possible to instead do LD_LIBRARY_PATH=/usr/lib/ python3 -c 'import numpy' The `libz` guix package still needs to be included among the packages passed to `guix shell` in this last case. Also, given you're using virtualenv, you might find the `--network` option to `guix shell` useful :) I hope there isn't any dumb mistake in my solution... Good luck :) Wojtek -- (sig_start) website: https://koszko.org/koszko.html PGP: https://koszko.org/key.gpg fingerprint: E972 7060 E3C5 637C 8A4F 4B42 4BC5 221C 5A79 FD1A Meet Kraków saints! #27: blessed Maria Klemensa Staszewska Poznaj świętych krakowskich! #27: błogosławiona Maria Klemensa Staszewska https://pl.wikipedia.org/wiki/Maria_Klemensa_Staszewska -- (sig_end) On Tue, 29 Nov 2022 16:42:14 +0000 mprodrigues@posteo.net wrote: > Hello, > > I've been using a guix shell with a couple of python packages and it > works fine, > however I'd also like to keep the option of using a virtualenv if > possible, the > thing is I'm having problem running modules like numpy, that depend on > non > python dependencies. The error I get when I try to load numpy in a venv > is: > > Importing the numpy C-extensions failed. This error can happen for many > reasons, > often due to issues with your setup or how NumPy was installed. > > and in the end: > > Original error was: libz.so.1: cannot open shared object file: No such > file or > directory > > I have tried installing zlib directly, also tried to add my default > profile libs > path to the venv PATH, but to no avail until now. How can I solve this? > Or > maybe someone can point me in the direction that I have to take to > properly > debug this. > > Best regards, > Mateus Rodrigues > [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 228 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Problem loading numpy and other on a venv in a guix system 2022-11-29 18:33 ` Wojtek Kosior via @ 2022-11-30 9:42 ` mprodrigues 2022-11-30 11:21 ` Wojtek Kosior via 0 siblings, 1 reply; 9+ messages in thread From: mprodrigues @ 2022-11-30 9:42 UTC (permalink / raw) To: Wojtek Kosior; +Cc: help-guix Hi Wojtek, thanks for your help, indeed setting LD_LIBRARY_PATH solves the issue, but it is less than ideal as I have to first check exactly the PATH to set, and if I update guix this will also change I believe, still it is not so bad. I'd still like to understand why this happens in the first place. Best regards, Mateus Rodrigues On 29.11.2022 19:33, Wojtek Kosior wrote: > Hi Mateus! > > Indeed, when using virtualenv's numpy, this > > python3 -c 'import numpy' > > fails with the error you posted. I seem to have been able to make numpy > happy by instead running Python like > > > LD_LIBRARY_PATH=/gnu/store/8qv5kb2fgm4c3bf70zcg9l6hkf3qzpw9-zlib-1.2.11/lib/ > python3 -c 'import numpy' > > If you use the `--container` and `--emulate-fhs` options to > `guix shell`, it is also possible to instead do > > LD_LIBRARY_PATH=/usr/lib/ python3 -c 'import numpy' > > The `libz` guix package still needs to be included among the packages > passed to `guix shell` in this last case. > > Also, given you're using virtualenv, you might find the `--network` > option to `guix shell` useful :) > > I hope there isn't any dumb mistake in my solution... Good luck :) > > Wojtek > > -- (sig_start) > website: https://koszko.org/koszko.html > PGP: https://koszko.org/key.gpg > fingerprint: E972 7060 E3C5 637C 8A4F 4B42 4BC5 221C 5A79 FD1A > > Meet Kraków saints! #27: blessed Maria Klemensa Staszewska > Poznaj świętych krakowskich! #27: błogosławiona Maria Klemensa > Staszewska > https://pl.wikipedia.org/wiki/Maria_Klemensa_Staszewska > -- (sig_end) > > > On Tue, 29 Nov 2022 16:42:14 +0000 > mprodrigues@posteo.net wrote: > >> Hello, >> >> I've been using a guix shell with a couple of python packages and it >> works fine, >> however I'd also like to keep the option of using a virtualenv if >> possible, the >> thing is I'm having problem running modules like numpy, that depend on >> non >> python dependencies. The error I get when I try to load numpy in a >> venv >> is: >> >> Importing the numpy C-extensions failed. This error can happen for >> many >> reasons, >> often due to issues with your setup or how NumPy was installed. >> >> and in the end: >> >> Original error was: libz.so.1: cannot open shared object file: No such >> file or >> directory >> >> I have tried installing zlib directly, also tried to add my default >> profile libs >> path to the venv PATH, but to no avail until now. How can I solve >> this? >> Or >> maybe someone can point me in the direction that I have to take to >> properly >> debug this. >> >> Best regards, >> Mateus Rodrigues >> ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Problem loading numpy and other on a venv in a guix system 2022-11-30 9:42 ` mprodrigues @ 2022-11-30 11:21 ` Wojtek Kosior via 0 siblings, 0 replies; 9+ messages in thread From: Wojtek Kosior via @ 2022-11-30 11:21 UTC (permalink / raw) To: mprodrigues; +Cc: help-guix [-- Attachment #1: Type: text/plain, Size: 6588 bytes --] Hello, > Hi Wojtek, > > thanks for your help, indeed setting LD_LIBRARY_PATH solves the issue, > but it is less than ideal as I have to first > check exactly the PATH to set, and if I update guix this will also > change I believe, still it is not so bad. I'd still like > to understand why this happens in the first place. > > Best regards, > Mateus Rodrigues I see. I don't think I realize all the nuances of dynamic library loading myself. But I can try to explain what I know :) In C (also to some extent C++) world, libraries can be either linked statically into a program (i.e. included in its executable) or linked dynamically (i.e. so that they are found in the filesystem when program gets executed). There are pros and cons to both approaches. With dynamically-linked libraries (also called shared libraries), it is possible to write a program so that it doesn't load them immediately upon start. It is possible to instead have program's code call a C library function that loads a shared library *by name*, together with other shared libraries it depends on. This is what the Python interpreted process tries to do when told to load Numpy's C extension. Under a typical GNU/Linux system, shared libraries are searched for in some predefined locations (like `/lib/x86_64-linux-gnu/` which is where `libz.so.1` resides in my Debian OS). AFAIK, these locations are compiled into the C library and cannot be *changed*. However, they can be *supplemented* by paths in the `LD_LIBRARY_PATH` variable which are searched by the C library *before* the standard locations. Now, Guix does things differently. It does not keep its libraries in standard locations like `/lib`. It cannot - the assumption is that files needed by programs are kept under `/gnu/store`. So, AFAIU, the C library in Guix is *modified* not to assume any standard directories for shared libraries. Instead, all programs have the exact locations of their shared libraries specified directly in their executables. This is possible with the help of a (perhaps a bit obscure) feature called `RPATH`. So we already know that Guix' own packages make up for the lack of standard shared library directories. How about binaries like those from PyPI that were not compiled for Guix? They don't have the RPATH set and they expect libc to be able to resolve some of their dependencies. In particular, Python C extensions devs used to be rather safe to assume that zlib (a dependency of CPython) would be present and resolvable in the user's operating system and need not be bundled with the extension. If we think about it, an analogous problem appears for many languages, not just C. And it seems Guix solves it slightly differently there. For example, Python library locations are specified in Guix-prepared variable `GUIX_PYTHONPATH`. I believe that works more reliably than C's `RPATH` (although I had some issues with loading of Python libs, too). Back to your problem - symlinks to shared libraries are also placed in the `lib/` subdirectory of a Guix profile. For example, after doing guix install zlib I got a `libz.so.1` inside `~/.guix-profile/lib`. Perhaps this will prove helpful to you? :) Wojtek -- (sig_start) website: https://koszko.org/koszko.html PGP: https://koszko.org/key.gpg fingerprint: E972 7060 E3C5 637C 8A4F 4B42 4BC5 221C 5A79 FD1A Meet Kraków saints! #39: blessed Narcyz Turchan Poznaj świętych krakowskich! #39: błogosławiony Narcyz Turchan https://pl.wikipedia.org/wiki/Narcyz_Turchan -- (sig_end) On Wed, 30 Nov 2022 09:42:34 +0000 mprodrigues@posteo.net wrote: > Hi Wojtek, > > thanks for your help, indeed setting LD_LIBRARY_PATH solves the issue, > but it is less than ideal as I have to first > check exactly the PATH to set, and if I update guix this will also > change I believe, still it is not so bad. I'd still like > to understand why this happens in the first place. > > Best regards, > Mateus Rodrigues > > On 29.11.2022 19:33, Wojtek Kosior wrote: > > Hi Mateus! > > > > Indeed, when using virtualenv's numpy, this > > > > python3 -c 'import numpy' > > > > fails with the error you posted. I seem to have been able to make numpy > > happy by instead running Python like > > > > > > LD_LIBRARY_PATH=/gnu/store/8qv5kb2fgm4c3bf70zcg9l6hkf3qzpw9-zlib-1.2.11/lib/ > > python3 -c 'import numpy' > > > > If you use the `--container` and `--emulate-fhs` options to > > `guix shell`, it is also possible to instead do > > > > LD_LIBRARY_PATH=/usr/lib/ python3 -c 'import numpy' > > > > The `libz` guix package still needs to be included among the packages > > passed to `guix shell` in this last case. > > > > Also, given you're using virtualenv, you might find the `--network` > > option to `guix shell` useful :) > > > > I hope there isn't any dumb mistake in my solution... Good luck :) > > > > Wojtek > > > > -- (sig_start) > > website: https://koszko.org/koszko.html > > PGP: https://koszko.org/key.gpg > > fingerprint: E972 7060 E3C5 637C 8A4F 4B42 4BC5 221C 5A79 FD1A > > > > Meet Kraków saints! #27: blessed Maria Klemensa Staszewska > > Poznaj świętych krakowskich! #27: błogosławiona Maria Klemensa > > Staszewska > > https://pl.wikipedia.org/wiki/Maria_Klemensa_Staszewska > > -- (sig_end) > > > > > > On Tue, 29 Nov 2022 16:42:14 +0000 > > mprodrigues@posteo.net wrote: > > > >> Hello, > >> > >> I've been using a guix shell with a couple of python packages and it > >> works fine, > >> however I'd also like to keep the option of using a virtualenv if > >> possible, the > >> thing is I'm having problem running modules like numpy, that depend on > >> non > >> python dependencies. The error I get when I try to load numpy in a > >> venv > >> is: > >> > >> Importing the numpy C-extensions failed. This error can happen for > >> many > >> reasons, > >> often due to issues with your setup or how NumPy was installed. > >> > >> and in the end: > >> > >> Original error was: libz.so.1: cannot open shared object file: No such > >> file or > >> directory > >> > >> I have tried installing zlib directly, also tried to add my default > >> profile libs > >> path to the venv PATH, but to no avail until now. How can I solve > >> this? > >> Or > >> maybe someone can point me in the direction that I have to take to > >> properly > >> debug this. > >> > >> Best regards, > >> Mateus Rodrigues > >> [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 228 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Problem loading numpy and other on a venv in a guix system 2022-11-29 16:42 Problem loading numpy and other on a venv in a guix system mprodrigues 2022-11-29 18:33 ` Wojtek Kosior via @ 2022-11-30 8:13 ` zimoun 2022-11-30 8:47 ` Wojtek Kosior via 1 sibling, 1 reply; 9+ messages in thread From: zimoun @ 2022-11-30 8:13 UTC (permalink / raw) To: mprodrigues, help-guix Hi, On Tue, 29 Nov 2022 at 16:42, mprodrigues@posteo.net wrote: > Importing the numpy C-extensions failed. This error can happen for many > reasons, > often due to issues with your setup or how NumPy was installed. Well, I am not sure to understand your configuration. You are installing some packages (say python and python-numpy) using “guix shell” and then installing others using pip loaded with virtualenv. Right? Cheers, simon ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Problem loading numpy and other on a venv in a guix system 2022-11-30 8:13 ` zimoun @ 2022-11-30 8:47 ` Wojtek Kosior via 2022-11-30 9:46 ` mprodrigues 2022-11-30 12:37 ` zimoun 0 siblings, 2 replies; 9+ messages in thread From: Wojtek Kosior via @ 2022-11-30 8:47 UTC (permalink / raw) To: zimoun; +Cc: mprodrigues, help-guix [-- Attachment #1: Type: text/plain, Size: 1736 bytes --] > Hi, > > On Tue, 29 Nov 2022 at 16:42, mprodrigues@posteo.net wrote: > > > Importing the numpy C-extensions failed. This error can happen for many > > reasons, > > often due to issues with your setup or how NumPy was installed. > > Well, I am not sure to understand your configuration. You are > installing some packages (say python and python-numpy) using “guix > shell” and then installing others using pip loaded with virtualenv. > Right? > > > Cheers, > simon > I believe Mateus did something like guix shell python python-virtualenv coreutils followed by virtualenv -p python3 somedirectory . ./somedirectory/bin/activate pip install numpy This is more or less what I used to reproduce the issue Wojtek -- (sig_start) website: https://koszko.org/koszko.html PGP: https://koszko.org/key.gpg fingerprint: E972 7060 E3C5 637C 8A4F 4B42 4BC5 221C 5A79 FD1A Meet Kraków saints! #47: saint Stanisław ze Szczepanowa Poznaj świętych krakowskich! #47: święty Stanisław ze Szczepanowa https://pl.wikipedia.org/wiki/Stanisław_ze_Szczepanowa -- (sig_end) On Wed, 30 Nov 2022 09:13:03 +0100 zimoun <zimon.toutoune@gmail.com> wrote: > Hi, > > On Tue, 29 Nov 2022 at 16:42, mprodrigues@posteo.net wrote: > > > Importing the numpy C-extensions failed. This error can happen for many > > reasons, > > often due to issues with your setup or how NumPy was installed. > > Well, I am not sure to understand your configuration. You are > installing some packages (say python and python-numpy) using “guix > shell” and then installing others using pip loaded with virtualenv. > Right? > > > Cheers, > simon > [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 228 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Problem loading numpy and other on a venv in a guix system 2022-11-30 8:47 ` Wojtek Kosior via @ 2022-11-30 9:46 ` mprodrigues 2022-11-30 12:07 ` zimoun 2022-11-30 12:37 ` zimoun 1 sibling, 1 reply; 9+ messages in thread From: mprodrigues @ 2022-11-30 9:46 UTC (permalink / raw) To: Wojtek Kosior; +Cc: zimoun, help-guix Hello, On 30.11.2022 09:47, Wojtek Kosior wrote: > I believe Mateus did something like > > guix shell python python-virtualenv coreutils > > followed by > > virtualenv -p python3 somedirectory > . ./somedirectory/bin/activate > pip install numpy > > This is more or less what I used to reproduce the issue not quite, but I guess it doesn't make all that much difference to reproduce the error. I use either guix shell, and install all python packages through guix, or the venv, and install all modules through pip, not both at the same time. Although, maybe using both at the same time would not be bad. Best regards, Mateus Rodrigues ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Problem loading numpy and other on a venv in a guix system 2022-11-30 9:46 ` mprodrigues @ 2022-11-30 12:07 ` zimoun 0 siblings, 0 replies; 9+ messages in thread From: zimoun @ 2022-11-30 12:07 UTC (permalink / raw) To: mprodrigues, Wojtek Kosior; +Cc: help-guix Hi, On Wed, 30 Nov 2022 at 09:46, mprodrigues@posteo.net wrote: >> This is more or less what I used to reproduce the issue > > not quite, but I guess it doesn't make all that much difference to > reproduce the error. I think it makes difference. :-) > I use either guix shell, and install all python packages through guix, > or the venv, and > install all modules through pip, not both at the same time. Although, > maybe using both > at the same time would not be bad. From my experience, you cannot mix both “guix shell” and “pip“ easily. For what it is worth, try to use both at the same time is the beginning of all sort of annoyances. See Wojtek message [1]. 1: <https://yhetil.org/guix/20221130122110.708c7e20@koszkonutek-tmp.pl.eu.org> Cheers, simon ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Problem loading numpy and other on a venv in a guix system 2022-11-30 8:47 ` Wojtek Kosior via 2022-11-30 9:46 ` mprodrigues @ 2022-11-30 12:37 ` zimoun 1 sibling, 0 replies; 9+ messages in thread From: zimoun @ 2022-11-30 12:37 UTC (permalink / raw) To: Wojtek Kosior; +Cc: mprodrigues, help-guix Hi, On Wed, 30 Nov 2022 at 09:47, Wojtek Kosior via <help-guix@gnu.org> wrote: cd /tmp/ > guix shell python python-virtualenv coreutils > virtualenv -p python3 somedirectory > . ./somedirectory/bin/activate > pip install numpy python3 -c 'import numpy' leads to, --8<---------------cut here---------------start------------->8--- Traceback (most recent call last): File "/tmp/somedirectory/lib/python3.9/site-packages/numpy/core/__init__.py", line 23, in <module> from . import multiarray File "/tmp/somedirectory/lib/python3.9/site-packages/numpy/core/multiarray.py", line 10, in <module> from . import overrides File "/tmp/somedirectory/lib/python3.9/site-packages/numpy/core/overrides.py", line 6, in <module> from numpy.core._multiarray_umath import ( ImportError: libz.so.1: cannot open shared object file: No such file or directory During handling of the above exception, another exception occurred: Traceback (most recent call last): File "<string>", line 1, in <module> File "/tmp/somedirectory/lib/python3.9/site-packages/numpy/__init__.py", line 140, in <module> from . import core File "/tmp/somedirectory/lib/python3.9/site-packages/numpy/core/__init__.py", line 49, in <module> raise ImportError(msg) ImportError: IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE! Importing the numpy C-extensions failed. This error can happen for many reasons, often due to issues with your setup or how NumPy was installed. We have compiled some common reasons and troubleshooting tips at: https://numpy.org/devdocs/user/troubleshooting-importerror.html Please note and check the following: * The Python version is: Python3.9 from "/tmp/somedirectory/bin/python3" * The NumPy version is: "1.23.5" and make sure that they are the versions you expect. Please carefully study the documentation linked above for further help. Original error was: libz.so.1: cannot open shared object file: No such file or directory --8<---------------cut here---------------end--------------->8--- That’s because the Numpy from pip is linked to system-wide library, --8<---------------cut here---------------start------------->8--- $ ldd somedirectory/lib/python3.9/site-packages/numpy/core/_multiarray_umath.cpython-39-x86_64-linux-gnu.so linux-vdso.so.1 (0x00007ffde12e0000) libopenblas64_p-r0-742d56dc.3.20.so => /tmp/somedirectory/lib/python3.9/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-742d56dc.3.20.so (0x00007f6f1223d000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f6f120dd000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f6f120ba000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f6f11ec8000) /lib64/ld-linux-x86-64.so.2 (0x00007f6f1481a000) libgfortran-040039e1.so.5.0.0 => /tmp/somedirectory/lib/python3.9/site-packages/numpy/core/../../numpy.libs/libgfortran-040039e1.so.5.0.0 (0x00007f6f11a3b000) libquadmath-96973f99.so.0.0.0 => /tmp/somedirectory/lib/python3.9/site-packages/numpy/core/../../numpy.libs/libquadmath-96973f99.so.0.0.0 (0x00007f6f117fc000) libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f6f117e0000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f6f117c5000) --8<---------------cut here---------------end--------------->8--- and they are not found. Well, if you are running Guix on foreign distro, you can manually fix the issue with this “missing” libz library by setting LD_LIBRARY_PATH=/lib/x86_64-linux-gnu/. But you get another issue: --8<---------------cut here---------------start------------->8--- $ LD_LIBRARY_PATH=/lib/x86_64-linux-gnu/ python3 -c 'import numpy' python3: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /gnu/store/lvip6h5pamjwmvnkwg60sjb63ph8698k-python-3.9.9/lib/libpython3.9.so.1.0) python3: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /gnu/store/lvip6h5pamjwmvnkwg60sjb63ph8698k-python-3.9.9/lib/libpython3.9.so.1.0) --8<---------------cut here---------------end--------------->8--- Well, I do not know if the Guix package “python-virtualenv” is working in “guix shell”. Cheers, simon ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2022-11-30 12:41 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-11-29 16:42 Problem loading numpy and other on a venv in a guix system mprodrigues 2022-11-29 18:33 ` Wojtek Kosior via 2022-11-30 9:42 ` mprodrigues 2022-11-30 11:21 ` Wojtek Kosior via 2022-11-30 8:13 ` zimoun 2022-11-30 8:47 ` Wojtek Kosior via 2022-11-30 9:46 ` mprodrigues 2022-11-30 12:07 ` zimoun 2022-11-30 12:37 ` zimoun
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).