Re: [RFC/PATCH] python: search parent lib directory for libnotmuch.so

[Justus Winter <4winter@informatik.uni-hamburg.de>]

Hi Jed,

Quoting Jed Brown (2013-04-09 04:47:26)
> If libnotmuch.so is installed to a path that is not searched by
> dlopen(3), we must import it using an absolute path because the Python
> bindings do not have the luxury of RPATH linking.  So strip off the
> trailing directories from the install location and try CDLL with an
> absolute path.

May I ask why you cannot use LD_LIBRARY_PATH? I too install libnotmuch
to a non-standard location as unprivileged user and to make this
library available I add its path to LD_LIBRARY_PATH. It seems to work
just fine:

teythoon@thinkbox ~ % python -c 'import notmuch; print("\n".join(l.strip() for l in open("/proc/self/maps") if "notmuch" in l))'
7f4214339000-7f4214357000 r-xp 00000000 00:17 2120750                    /home/teythoon/.local/stow/notmuch/lib/libnotmuch.so.3.0.0
7f4214357000-7f4214557000 ---p 0001e000 00:17 2120750                    /home/teythoon/.local/stow/notmuch/lib/libnotmuch.so.3.0.0
7f4214557000-7f4214559000 rw-p 0001e000 00:17 2120750                    /home/teythoon/.local/stow/notmuch/lib/libnotmuch.so.3.0.0
teythoon@thinkbox ~ % python3.2 -c 'import notmuch; print("\n".join(l.strip() for l in open("/proc/self/maps") if "notmuch" in l))'
7fa5ecea4000-7fa5ecec2000 r-xp 00000000 00:17 2120750                    /home/teythoon/.local/stow/notmuch/lib/libnotmuch.so.3.0.0
7fa5ecec2000-7fa5ed0c2000 ---p 0001e000 00:17 2120750                    /home/teythoon/.local/stow/notmuch/lib/libnotmuch.so.3.0.0
7fa5ed0c2000-7fa5ed0c4000 rw-p 0001e000 00:17 2120750                    /home/teythoon/.local/stow/notmuch/lib/libnotmuch.so.3.0.0
teythoon@thinkbox ~ % env | grep '\.local'
PATH=/home/teythoon/.local/bin:/usr/local/stow/enlightenment/bin:/usr/local/bin:/usr/bin:/bin:/usr/games
LOCAL=/home/teythoon/.local
PYTHONPATH=/home/teythoon/.local/lib/python2.7/site-packages
LD_LIBRARY_PATH=/home/teythoon/.local/lib
PKG_CONFIG_PATH=/home/teythoon/.local/lib/pkgconfig
teythoon@thinkbox ~ % unset LD_LIBRARY_PATH
teythoon@thinkbox ~ % python -c 'import notmuch; print("\n".join(l.strip() for l in open("/proc/self/maps") if "notmuch" in l))'
7fd30b43c000-7fd30b457000 r-xp 00000000 fe:00 4241                       /usr/lib/libnotmuch.so.3.0.0
7fd30b457000-7fd30b657000 ---p 0001b000 fe:00 4241                       /usr/lib/libnotmuch.so.3.0.0
7fd30b657000-7fd30b658000 rw-p 0001b000 fe:00 4241                       /usr/lib/libnotmuch.so.3.0.0
teythoon@thinkbox ~ % python3.2 -c 'import notmuch; print("\n".join(l.strip() for l in open("/proc/self/maps") if "notmuch" in l))'
7fadb5704000-7fadb571f000 r-xp 00000000 fe:00 4241                       /usr/lib/libnotmuch.so.3.0.0
7fadb571f000-7fadb591f000 ---p 0001b000 fe:00 4241                       /usr/lib/libnotmuch.so.3.0.0
7fadb591f000-7fadb5920000 rw-p 0001b000 fe:00 4241                       /usr/lib/libnotmuch.so.3.0.0

> ---
> This is sort of a hack, but I don't know a less intrusive way to get
> libnotmuch.so from somewhere dlopen(3) doesn't already search.
> 
> The absolute path version won't do the right thing in case of 'setup.py
> develop', otherwise we could use it in all cases.  It may still make
> sense to make the absolute path version take precedence.

Well, if something like this is necessary and wanted (opinions
anyone?) at least let's not hardcode the assumption about the
directory layout but just walk up the tree until we find notmuch.so.3
or lib/notmuch.so.3. This way the bindings will find the correct
library even when they are included directly from within the source
tree.

Otoh, adding such behavior might be 'surprising' and lead to many
problems down the road like spurious bug reports just because the
magic library finder locates a rogue libnotmuch.so.3 somewhere.

> An alternative would be to find libnotmuch.so using the notmuch
> executable.

How would you do that? fork'ing and exec'ing is not an option (and I'm
not sure what one could achieve by exec'ing it, but how else would you
talk to the dynamic linker?) , and poking around in binaries to get
their rpath isn't either in my opinion. And you would have to locate
the 'right' binary in the first place?

Cheers,
Justus