* bug#30229: Python modules installed by pip in virtualenv can't find shared objects.
@ 2018-01-23 11:49 Fis Trivial
2018-01-23 12:00 ` Ricardo Wurmus
2018-01-24 15:01 ` Ludovic Courtès
0 siblings, 2 replies; 6+ messages in thread
From: Fis Trivial @ 2018-01-23 11:49 UTC (permalink / raw)
To: 30229
When a python module needs to load a dynamic shared object, it looks in the
path provided by *LD_LIBRARY_PATH*(1), but guix doesn't modify this environment
variable to export the needed path for python.
* Backtrace
In my case, it's lightgbm (installed by pip), needs libgomp.so from gcc:lib for
openmp support. Here is the backtrace:
--8<---------------cut here---------------start------------->8---
Traceback (most recent call last):
File "ex1.py", line 5, in <module>
import lightgbm as lgb
File "/home/fis/Workspace/tianchi/medical_treatment/lib/python3.5/site-packages/lightgbm/__init__.py", line 8, in <module>
from .basic import Booster, Dataset
File "/home/fis/Workspace/tianchi/medical_treatment/lib/python3.5/site-packages/lightgbm/basic.py", line 32, in <module>
_LIB = _load_lib()
File "/home/fis/Workspace/tianchi/medical_treatment/lib/python3.5/site-packages/lightgbm/basic.py", line 27, in _load_lib
lib = ctypes.cdll.LoadLibrary(lib_path[0])
File "/gnu/store/jb3n0bsdpkhvyb8y70jyr8fcx8fqssr9-python-3.5.3/lib/python3.5/ctypes/__init__.py", line 425, in LoadLibrary
return self._dlltype(name)
File "/gnu/store/jb3n0bsdpkhvyb8y70jyr8fcx8fqssr9-python-3.5.3/lib/python3.5/ctypes/__init__.py", line 347, in __init__
self._handle = _dlopen(self._name, mode)
OSError: libgomp.so.1: cannot open shared object file: No such file or directory
--8<---------------cut here---------------end--------------->8---
* Reproduce
To install lightgbm, simply use `pip install lightgbm`(in virtualenv).
Then in python shell:
import lightgbm as lgb
* Ad-hoc
export LD_LIBRARY_PATH=~/.guix-profile/lib:$LD_LIBRARY_PATH
[1]: https://stackoverflow.com/a/1100016
^ permalink raw reply [flat|nested] 6+ messages in thread* bug#30229: Python modules installed by pip in virtualenv can't find shared objects.
2018-01-23 11:49 bug#30229: Python modules installed by pip in virtualenv can't find shared objects Fis Trivial
@ 2018-01-23 12:00 ` Ricardo Wurmus
2018-01-23 13:21 ` Fis Trivial
2021-11-24 23:39 ` zimoun
2018-01-24 15:01 ` Ludovic Courtès
1 sibling, 2 replies; 6+ messages in thread
From: Ricardo Wurmus @ 2018-01-23 12:00 UTC (permalink / raw)
To: Fis Trivial; +Cc: 30229
Hi,
> When a python module needs to load a dynamic shared object, it looks in the
> path provided by *LD_LIBRARY_PATH*(1), but guix doesn't modify this environment
> variable to export the needed path for python.
We cannot set this environment variable by default lest we break other
packages that may be installed. LD_LIBRARY_PATH is dangerous as it
tells the runtime linker to prefer libraries in the specified
directories.
For Guix packages we use different means to embed store paths in
binaries, which are looked up at runtime. For binaries that’s achieved
with RUNPATH; for others we patch the sources to ensure that libraries
are not looked up merely by name but by absolute path.
In your particular case (installing packages without Guix) I think the
best way is to manually set LD_LIBRARY_PATH on demand, or to set
LD_PRELOAD to the specific libraries that are required.
In general, though, I recommend using Guix for package management and
development instead of virtualenv and pip.
--
Ricardo
GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC
https://elephly.net
^ permalink raw reply [flat|nested] 6+ messages in thread
* bug#30229: Python modules installed by pip in virtualenv can't find shared objects.
2018-01-23 12:00 ` Ricardo Wurmus
@ 2018-01-23 13:21 ` Fis Trivial
2021-11-24 23:39 ` zimoun
1 sibling, 0 replies; 6+ messages in thread
From: Fis Trivial @ 2018-01-23 13:21 UTC (permalink / raw)
To: Ricardo Wurmus; +Cc: 30229@debbugs.gnu.org
>
> In general, though, I recommend using Guix for package management and
> development instead of virtualenv and pip.
>
I understand the importance for having a self-contained dependency graph.
After trying guix a few months, I came to realize that, at the a theoretical
level, the environment variables are not treated as part of the dependency
graph in the Guix world. Dangling environment variables happens, and can not
be solved by installing everything with guix(1).
I really hope that we can come up with some methods to solve these problems
in a systematical way. I will keep reporting bugs caused by environment
variables, hoping that it might be helpful in future development.
[1]: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=30093
^ permalink raw reply [flat|nested] 6+ messages in thread
* bug#30229: Python modules installed by pip in virtualenv can't find shared objects.
2018-01-23 12:00 ` Ricardo Wurmus
2018-01-23 13:21 ` Fis Trivial
@ 2021-11-24 23:39 ` zimoun
2022-01-13 14:59 ` zimoun
1 sibling, 1 reply; 6+ messages in thread
From: zimoun @ 2021-11-24 23:39 UTC (permalink / raw)
To: Ricardo Wurmus; +Cc: 30229, Fis Trivial
Hi,
On Tue, 23 Jan 2018 at 13:00, Ricardo Wurmus <rekado@elephly.net> wrote:
>> When a python module needs to load a dynamic shared object, it looks in the
>> path provided by *LD_LIBRARY_PATH*(1), but guix doesn't modify this environment
>> variable to export the needed path for python.
>
> We cannot set this environment variable by default lest we break other
> packages that may be installed. LD_LIBRARY_PATH is dangerous as it
> tells the runtime linker to prefer libraries in the specified
> directories.
>
> For Guix packages we use different means to embed store paths in
> binaries, which are looked up at runtime. For binaries that’s achieved
> with RUNPATH; for others we patch the sources to ensure that libraries
> are not looked up merely by name but by absolute path.
>
> In your particular case (installing packages without Guix) I think the
> best way is to manually set LD_LIBRARY_PATH on demand, or to set
> LD_PRELOAD to the specific libraries that are required.
>
> In general, though, I recommend using Guix for package management and
> development instead of virtualenv and pip.
Regarding the improvements of ’guix import pypi’ since 2018, and because
tweaking LD_LIBRARY_PATH is dangerous, I do not see what could be the
next action to solve this.
Therefore, I propose to close it. WDYT?
Cheers,
simon
^ permalink raw reply [flat|nested] 6+ messages in thread
* bug#30229: Python modules installed by pip in virtualenv can't find shared objects.
2021-11-24 23:39 ` zimoun
@ 2022-01-13 14:59 ` zimoun
0 siblings, 0 replies; 6+ messages in thread
From: zimoun @ 2022-01-13 14:59 UTC (permalink / raw)
To: Ricardo Wurmus; +Cc: Fis Trivial, 30229-done
Hi,
On Thu, 25 Nov 2021 at 00:39, zimoun <zimon.toutoune@gmail.com> wrote:
> On Tue, 23 Jan 2018 at 13:00, Ricardo Wurmus <rekado@elephly.net> wrote:
>
>>> When a python module needs to load a dynamic shared object, it looks in the
>>> path provided by *LD_LIBRARY_PATH*(1), but guix doesn't modify this
>>> environment
>>> variable to export the needed path for python.
>>
>> We cannot set this environment variable by default lest we break other
>> packages that may be installed. LD_LIBRARY_PATH is dangerous as it
>> tells the runtime linker to prefer libraries in the specified
>> directories.
>>
>> For Guix packages we use different means to embed store paths in
>> binaries, which are looked up at runtime. For binaries that’s achieved
>> with RUNPATH; for others we patch the sources to ensure that libraries
>> are not looked up merely by name but by absolute path.
>>
>> In your particular case (installing packages without Guix) I think the
>> best way is to manually set LD_LIBRARY_PATH on demand, or to set
>> LD_PRELOAD to the specific libraries that are required.
>>
>> In general, though, I recommend using Guix for package management and
>> development instead of virtualenv and pip.
>
> Regarding the improvements of ’guix import pypi’ since 2018, and because
> tweaking LD_LIBRARY_PATH is dangerous, I do not see what could be the
> next action to solve this.
>
> Therefore, I propose to close it. WDYT?
After 7 weeks of delay, I am closing.
Cheers,
simon
^ permalink raw reply [flat|nested] 6+ messages in thread
* bug#30229: Python modules installed by pip in virtualenv can't find shared objects.
2018-01-23 11:49 bug#30229: Python modules installed by pip in virtualenv can't find shared objects Fis Trivial
2018-01-23 12:00 ` Ricardo Wurmus
@ 2018-01-24 15:01 ` Ludovic Courtès
1 sibling, 0 replies; 6+ messages in thread
From: Ludovic Courtès @ 2018-01-24 15:01 UTC (permalink / raw)
To: Fis Trivial; +Cc: 30229
Hi,
Fis Trivial <ybbs.daans@hotmail.com> skribis:
> In my case, it's lightgbm (installed by pip), needs libgomp.so from gcc:lib for
> openmp support. Here is the backtrace:
>
> Traceback (most recent call last):
> File "ex1.py", line 5, in <module>
> import lightgbm as lgb
> File "/home/fis/Workspace/tianchi/medical_treatment/lib/python3.5/site-packages/lightgbm/__init__.py", line 8, in <module>
> from .basic import Booster, Dataset
> File "/home/fis/Workspace/tianchi/medical_treatment/lib/python3.5/site-packages/lightgbm/basic.py", line 32, in <module>
> _LIB = _load_lib()
> File "/home/fis/Workspace/tianchi/medical_treatment/lib/python3.5/site-packages/lightgbm/basic.py", line 27, in _load_lib
> lib = ctypes.cdll.LoadLibrary(lib_path[0])
> File "/gnu/store/jb3n0bsdpkhvyb8y70jyr8fcx8fqssr9-python-3.5.3/lib/python3.5/ctypes/__init__.py", line 425, in LoadLibrary
> return self._dlltype(name)
> File "/gnu/store/jb3n0bsdpkhvyb8y70jyr8fcx8fqssr9-python-3.5.3/lib/python3.5/ctypes/__init__.py", line 347, in __init__
> self._handle = _dlopen(self._name, mode)
> OSError: libgomp.so.1: cannot open shared object file: No such file or directory
I’d suggest hard-coding the absolute file name of libgomp.so.1 in
lightgbm/basic.py.
This is what we do, for instance, for Guile packages that use the
foreign function interface (FFI) and dynamically load shared libraries
(see ‘guile-gdbm-ffi’ as an example.)
HTH,
Ludo’.
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2022-01-13 15:16 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2018-01-23 11:49 bug#30229: Python modules installed by pip in virtualenv can't find shared objects Fis Trivial
2018-01-23 12:00 ` Ricardo Wurmus
2018-01-23 13:21 ` Fis Trivial
2021-11-24 23:39 ` zimoun
2022-01-13 14:59 ` zimoun
2018-01-24 15:01 ` Ludovic Courtès
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).