Hello, > Hi Wojtek, > > thanks for your help, indeed setting LD_LIBRARY_PATH solves the issue, > but it is less than ideal as I have to first > check exactly the PATH to set, and if I update guix this will also > change I believe, still it is not so bad. I'd still like > to understand why this happens in the first place. > > Best regards, > Mateus Rodrigues I see. I don't think I realize all the nuances of dynamic library loading myself. But I can try to explain what I know :) In C (also to some extent C++) world, libraries can be either linked statically into a program (i.e. included in its executable) or linked dynamically (i.e. so that they are found in the filesystem when program gets executed). There are pros and cons to both approaches. With dynamically-linked libraries (also called shared libraries), it is possible to write a program so that it doesn't load them immediately upon start. It is possible to instead have program's code call a C library function that loads a shared library *by name*, together with other shared libraries it depends on. This is what the Python interpreted process tries to do when told to load Numpy's C extension. Under a typical GNU/Linux system, shared libraries are searched for in some predefined locations (like `/lib/x86_64-linux-gnu/` which is where `libz.so.1` resides in my Debian OS). AFAIK, these locations are compiled into the C library and cannot be *changed*. However, they can be *supplemented* by paths in the `LD_LIBRARY_PATH` variable which are searched by the C library *before* the standard locations. Now, Guix does things differently. It does not keep its libraries in standard locations like `/lib`. It cannot - the assumption is that files needed by programs are kept under `/gnu/store`. So, AFAIU, the C library in Guix is *modified* not to assume any standard directories for shared libraries. Instead, all programs have the exact locations of their shared libraries specified directly in their executables. This is possible with the help of a (perhaps a bit obscure) feature called `RPATH`. So we already know that Guix' own packages make up for the lack of standard shared library directories. How about binaries like those from PyPI that were not compiled for Guix? They don't have the RPATH set and they expect libc to be able to resolve some of their dependencies. In particular, Python C extensions devs used to be rather safe to assume that zlib (a dependency of CPython) would be present and resolvable in the user's operating system and need not be bundled with the extension. If we think about it, an analogous problem appears for many languages, not just C. And it seems Guix solves it slightly differently there. For example, Python library locations are specified in Guix-prepared variable `GUIX_PYTHONPATH`. I believe that works more reliably than C's `RPATH` (although I had some issues with loading of Python libs, too). Back to your problem - symlinks to shared libraries are also placed in the `lib/` subdirectory of a Guix profile. For example, after doing guix install zlib I got a `libz.so.1` inside `~/.guix-profile/lib`. Perhaps this will prove helpful to you? :) Wojtek -- (sig_start) website: https://koszko.org/koszko.html PGP: https://koszko.org/key.gpg fingerprint: E972 7060 E3C5 637C 8A4F 4B42 4BC5 221C 5A79 FD1A Meet Kraków saints! #39: blessed Narcyz Turchan Poznaj świętych krakowskich! #39: błogosławiony Narcyz Turchan https://pl.wikipedia.org/wiki/Narcyz_Turchan -- (sig_end) On Wed, 30 Nov 2022 09:42:34 +0000 mprodrigues@posteo.net wrote: > Hi Wojtek, > > thanks for your help, indeed setting LD_LIBRARY_PATH solves the issue, > but it is less than ideal as I have to first > check exactly the PATH to set, and if I update guix this will also > change I believe, still it is not so bad. I'd still like > to understand why this happens in the first place. > > Best regards, > Mateus Rodrigues > > On 29.11.2022 19:33, Wojtek Kosior wrote: > > Hi Mateus! > > > > Indeed, when using virtualenv's numpy, this > > > > python3 -c 'import numpy' > > > > fails with the error you posted. I seem to have been able to make numpy > > happy by instead running Python like > > > > > > LD_LIBRARY_PATH=/gnu/store/8qv5kb2fgm4c3bf70zcg9l6hkf3qzpw9-zlib-1.2.11/lib/ > > python3 -c 'import numpy' > > > > If you use the `--container` and `--emulate-fhs` options to > > `guix shell`, it is also possible to instead do > > > > LD_LIBRARY_PATH=/usr/lib/ python3 -c 'import numpy' > > > > The `libz` guix package still needs to be included among the packages > > passed to `guix shell` in this last case. > > > > Also, given you're using virtualenv, you might find the `--network` > > option to `guix shell` useful :) > > > > I hope there isn't any dumb mistake in my solution... Good luck :) > > > > Wojtek > > > > -- (sig_start) > > website: https://koszko.org/koszko.html > > PGP: https://koszko.org/key.gpg > > fingerprint: E972 7060 E3C5 637C 8A4F 4B42 4BC5 221C 5A79 FD1A > > > > Meet Kraków saints! #27: blessed Maria Klemensa Staszewska > > Poznaj świętych krakowskich! #27: błogosławiona Maria Klemensa > > Staszewska > > https://pl.wikipedia.org/wiki/Maria_Klemensa_Staszewska > > -- (sig_end) > > > > > > On Tue, 29 Nov 2022 16:42:14 +0000 > > mprodrigues@posteo.net wrote: > > > >> Hello, > >> > >> I've been using a guix shell with a couple of python packages and it > >> works fine, > >> however I'd also like to keep the option of using a virtualenv if > >> possible, the > >> thing is I'm having problem running modules like numpy, that depend on > >> non > >> python dependencies. The error I get when I try to load numpy in a > >> venv > >> is: > >> > >> Importing the numpy C-extensions failed. This error can happen for > >> many > >> reasons, > >> often due to issues with your setup or how NumPy was installed. > >> > >> and in the end: > >> > >> Original error was: libz.so.1: cannot open shared object file: No such > >> file or > >> directory > >> > >> I have tried installing zlib directly, also tried to add my default > >> profile libs > >> path to the venv PATH, but to no avail until now. How can I solve > >> this? > >> Or > >> maybe someone can point me in the direction that I have to take to > >> properly > >> debug this. > >> > >> Best regards, > >> Mateus Rodrigues > >>