* [PATCH] Fix shared library loading in Python bindings. @ 2013-05-26 3:53 Julian Berman 2013-05-30 12:26 ` Justus Winter 0 siblings, 1 reply; 3+ messages in thread From: Julian Berman @ 2013-05-26 3:53 UTC (permalink / raw) To: notmuch; +Cc: Julian Berman Specifically, fixes loading on OS X, where libnotmuch will be a dylib.: --- bindings/python/notmuch/globals.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/bindings/python/notmuch/globals.py b/bindings/python/notmuch/globals.py index c7632c3..5e08e73 100644 --- a/bindings/python/notmuch/globals.py +++ b/bindings/python/notmuch/globals.py @@ -18,11 +18,12 @@ Copyright 2010 Sebastian Spaeth <Sebastian@SSpaeth.de> """ from ctypes import CDLL, Structure, POINTER +from ctypes.util import find_library #----------------------------------------------------------------------------- #package-global instance of the notmuch library try: - nmlib = CDLL("libnotmuch.so.3") + nmlib = CDLL(find_library("libnotmuch")) except: raise ImportError("Could not find shared 'notmuch' library.") -- 1.8.3 ^ permalink raw reply related [flat|nested] 3+ messages in thread
* Re: [PATCH] Fix shared library loading in Python bindings. 2013-05-26 3:53 [PATCH] Fix shared library loading in Python bindings Julian Berman @ 2013-05-30 12:26 ` Justus Winter 2013-05-30 16:06 ` Tomi Ollila 0 siblings, 1 reply; 3+ messages in thread From: Justus Winter @ 2013-05-30 12:26 UTC (permalink / raw) To: Julian Berman, notmuch [-- Attachment #1: Type: text/plain, Size: 5209 bytes --] Hi Julian :) Quoting Julian Berman (2013-05-26 05:53:38) > Specifically, fixes loading on OS X, where libnotmuch will be > a dylib.: > --- > bindings/python/notmuch/globals.py | 3 ++- > 1 file changed, 2 insertions(+), 1 deletion(-) > > diff --git a/bindings/python/notmuch/globals.py b/bindings/python/notmuch/globals.py > index c7632c3..5e08e73 100644 > --- a/bindings/python/notmuch/globals.py > +++ b/bindings/python/notmuch/globals.py > @@ -18,11 +18,12 @@ Copyright 2010 Sebastian Spaeth <Sebastian@SSpaeth.de> > """ > > from ctypes import CDLL, Structure, POINTER > +from ctypes.util import find_library > > #----------------------------------------------------------------------------- > #package-global instance of the notmuch library > try: > - nmlib = CDLL("libnotmuch.so.3") > + nmlib = CDLL(find_library("libnotmuch")) Does this work for you on Darwin? On my box (Debian/Linux) I have to use "notmuch" instead of "libnotmuch" to get anything from find_library: In [5]: print ctypes.util.find_library("notmuch") libnotmuch.so.3 In [6]: print ctypes.util.find_library("libnotmuch") None Then again, find_library is different for every architecture under the sun... > except: > raise ImportError("Could not find shared 'notmuch' library.") I'm leaning towards declining the patch in it's current form. If the bindings do not work on OS X, we need to find another solution. There are two reasons for this: 1. find_library was once used, but was removed since it is (has?) been problematic wrt to LD_LIBRARY_PATH usage: % git show f378f458 commit f378f45893bb4263402008b2abd8aab522901fb6 Author: Cedric Cabessa <ced@ryick.net> Date: Mon Apr 5 03:03:51 2010 +0200 find_library does not read LD_LIBRARY_PATH, but CDLL does. diff --git a/cnotmuch/globals.py b/cnotmuch/globals.py index ef2686f..fa20ae8 100644 --- a/cnotmuch/globals.py +++ b/cnotmuch/globals.py @@ -3,17 +3,17 @@ from ctypes.util import find_library #----------------------------------------------------------------------------- #package-global instance of the notmuch library -#TODO: lazy load this on first access? -so = find_library('notmuch') -if so is None: - raise ImportError("Could not find shared 'notmuch' library.") -nmlib = CDLL(so) +try: + nmlib = CDLL("libnotmuch.so.1") +except: + raise ImportError("Could not find shared 'notmuch' library.") [...] As a heavy LD_LIBRARY_PATH user I don't want to break this. So I tried to test whether find_library now respects LD_LIBRARY_PATH or not: teythoon@thinkbox ~ % echo $LD_LIBRARY_PATH /home/teythoon/.local/lib teythoon@thinkbox ~ % strace -e trace=open python -c 'import ctypes; ctypes.CDLL("libnotmuch.so.3")' 2>&1 | grep notmuch open("/home/teythoon/.local/lib/libnotmuch.so.3", O_RDONLY) = 3 That's how it's done now and indeed it finds my libnotmuch. teythoon@thinkbox ~ % strace -f -e trace=stat,open python -c 'import ctypes.util; ctypes.util.find_library("notmuch")' 2>&1 | grep notmuch Nothing. Funny. Let's see: teythoon@thinkbox ~ % strace -f -e trace=fork,execve,clone python -c 'import ctypes.util; print ctypes.util.find_library("notmuch")' execve("/usr/bin/python", ["python", "-c", "import ctypes.util; print ctypes"...], [/* 63 vars */]) = 0 clone(Process 12000 attached child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f0734a649d0) = 12000 [pid 12000] execve("/bin/sh", ["sh", "-c", "/sbin/ldconfig -p 2>/dev/null"], [/* 63 vars */]) = 0 [pid 12000] clone(Process 12001 attached child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f119d5479d0) = 12001 [pid 12001] execve("/sbin/ldconfig", ["/sbin/ldconfig", "-p"], [/* 63 vars */]) = 0 Process 12001 detached [pid 12000] --- SIGCHLD (Child exited) @ 0 (0) --- Process 12000 detached --- SIGCHLD (Child exited) @ 0 (0) --- libnotmuch.so.3 So it also prints libnotmuch.so.3, but only because the version installed from the Debian archive is also libnotmuch.so.3: teythoon@thinkbox ~ % /sbin/ldconfig -p | grep notmuch libnotmuch.so.3 (libc6,x86-64) => /usr/lib/libnotmuch.so.3 So I guess *if* I had a libnotmuch.so.2 in my ldconfig(8) cache (-p prints the cache), find_library would have returned libnotmuch.so.2 and not my libnotmuch.so.3. So your patch would most likely break this kind of setup. 2. Uh. I actually looked at /usr/lib/python2.7/ctypes/util.py and I almost got eye cancer from just looking briefly at it... The implementation of find_library differs for various operating systems and most (all?) of them use fork/exec some program (even gcc in some cases) to look up the libraries. I'd like to avoid this if possible, maybe even if this means handling OS X as a special case in our bindings. For reference, our current solution does not require any fork/exec calls: teythoon@thinkbox ~ % strace -f -e trace=execve,fork,clone python -c 'import ctypes; ctypes.CDLL("libnotmuch.so.3")' execve("/usr/bin/python", ["python", "-c", "import ctypes; ctypes.CDLL(\"libn"...], [/* 63 vars */]) = 0 Thoughts anyone? Cheers, Justus [-- Attachment #2: signature --] [-- Type: application/pgp-signature, Size: 836 bytes --] -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) iQIcBAABCAAGBQJRp0VcAAoJENMeiILK3jZYJwMP/R4VQYUmEDFYHX7xuiLL4zoC r+afAVY4OE2u2c9bncT54QairXuDGlLs8wmpDwkfZeiZ1N8OXeEwDlcMZYZ8BpQD /5LW0Gcf4u91UIsQ6t3wQLj6slvnMEQmLgyWbHpZIjFMbi/sMF7akJ+Ov/eZf0Si /9ZTrXteoUWw7kw9NhIqyxRC9/WHB9TwSdCVj7QPRBxK78B3+OxoX7DIp40QB3eZ MPiaOWRqV/6PrEmM1CxIRAEb5EoNB5LdgMM+0Ip6DtoEoKGbt3E7BbBk//tVTtaZ OYTCDEDtUfv0bfKL/VHVyihFZDJjDN1H+VgJZ4Sai3rjME1k/HphT/ZiBSO8b995 tpsc3yBB76kWEnHd1wf6DFJTLiKDwhm46V9WeNZiZ9auUm4RfcHozejPIooog9Bo +qubc9r7FhZpV7qrh9+PfS5MPf7qXb+Q5BpZMU30Cqksy1BwrPtmWJ6dDqucI+pR /9LxMTxfm8Qub8Fy811nOuG8vcKuL0GxvN/cSM2oqgceDiTXqwCWDO6y3jO3+HBf yChjoPioIdxfsyRv3UWjxVI0am3lkN0lKHp7tUbCxank9Yaj1yW/DOQhvpt5pNLe NvuNfYPG+Gk0IIsdOWM+R7EBkkPZmnMGdIJ+fsKPxgFZegeYJRP0FejHtw/HoWhG R9uuna4ewRUTTFnhxxLU =IPQn -----END PGP SIGNATURE----- ^ permalink raw reply related [flat|nested] 3+ messages in thread
* Re: [PATCH] Fix shared library loading in Python bindings. 2013-05-30 12:26 ` Justus Winter @ 2013-05-30 16:06 ` Tomi Ollila 0 siblings, 0 replies; 3+ messages in thread From: Tomi Ollila @ 2013-05-30 16:06 UTC (permalink / raw) To: Justus Winter, Julian Berman, notmuch On Thu, May 30 2013, Justus Winter <4winter@informatik.uni-hamburg.de> wrote: > Hi Julian :) > > Quoting Julian Berman (2013-05-26 05:53:38) >> Specifically, fixes loading on OS X, where libnotmuch will be >> a dylib.: >> --- >> bindings/python/notmuch/globals.py | 3 ++- >> 1 file changed, 2 insertions(+), 1 deletion(-) >> >> diff --git a/bindings/python/notmuch/globals.py b/bindings/python/notmuch/globals.py >> index c7632c3..5e08e73 100644 >> --- a/bindings/python/notmuch/globals.py >> +++ b/bindings/python/notmuch/globals.py >> @@ -18,11 +18,12 @@ Copyright 2010 Sebastian Spaeth <Sebastian@SSpaeth.de> >> """ >> >> from ctypes import CDLL, Structure, POINTER >> +from ctypes.util import find_library >> >> #----------------------------------------------------------------------------- >> #package-global instance of the notmuch library >> try: >> - nmlib = CDLL("libnotmuch.so.3") >> + nmlib = CDLL(find_library("libnotmuch")) > > Does this work for you on Darwin? On my box (Debian/Linux) I have to > use "notmuch" instead of "libnotmuch" to get anything from > find_library: > > In [5]: print ctypes.util.find_library("notmuch") > libnotmuch.so.3 > > In [6]: print ctypes.util.find_library("libnotmuch") > None > > Then again, find_library is different for every architecture under the > sun... > >> except: >> raise ImportError("Could not find shared 'notmuch' library.") > > I'm leaning towards declining the patch in it's current form. If the > bindings do not work on OS X, we need to find another solution. There > are two reasons for this: > > 1. find_library was once used, but was removed since it is (has?) been > problematic wrt to LD_LIBRARY_PATH usage: > > % git show f378f458 > commit f378f45893bb4263402008b2abd8aab522901fb6 > Author: Cedric Cabessa <ced@ryick.net> > Date: Mon Apr 5 03:03:51 2010 +0200 > > find_library does not read LD_LIBRARY_PATH, but CDLL does. > > diff --git a/cnotmuch/globals.py b/cnotmuch/globals.py > index ef2686f..fa20ae8 100644 > --- a/cnotmuch/globals.py > +++ b/cnotmuch/globals.py > @@ -3,17 +3,17 @@ from ctypes.util import find_library > > #----------------------------------------------------------------------------- > #package-global instance of the notmuch library > -#TODO: lazy load this on first access? > -so = find_library('notmuch') > -if so is None: > - raise ImportError("Could not find shared 'notmuch' library.") > -nmlib = CDLL(so) > +try: > + nmlib = CDLL("libnotmuch.so.1") > +except: > + raise ImportError("Could not find shared 'notmuch' library.") > [...] > > As a heavy LD_LIBRARY_PATH user I don't want to break this. So I tried > to test whether find_library now respects LD_LIBRARY_PATH or not: > > teythoon@thinkbox ~ % echo $LD_LIBRARY_PATH > /home/teythoon/.local/lib > teythoon@thinkbox ~ % strace -e trace=open python -c 'import ctypes; ctypes.CDLL("libnotmuch.so.3")' 2>&1 | grep notmuch > open("/home/teythoon/.local/lib/libnotmuch.so.3", O_RDONLY) = 3 > > That's how it's done now and indeed it finds my libnotmuch. > > teythoon@thinkbox ~ % strace -f -e trace=stat,open python -c 'import ctypes.util; ctypes.util.find_library("notmuch")' 2>&1 | grep notmuch > > Nothing. Funny. Let's see: > > teythoon@thinkbox ~ % strace -f -e trace=fork,execve,clone python -c 'import ctypes.util; print ctypes.util.find_library("notmuch")' > execve("/usr/bin/python", ["python", "-c", "import ctypes.util; print ctypes"...], [/* 63 vars */]) = 0 > clone(Process 12000 attached > child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f0734a649d0) = 12000 > [pid 12000] execve("/bin/sh", ["sh", "-c", "/sbin/ldconfig -p 2>/dev/null"], [/* 63 vars */]) = 0 > [pid 12000] clone(Process 12001 attached > child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f119d5479d0) = 12001 > [pid 12001] execve("/sbin/ldconfig", ["/sbin/ldconfig", "-p"], [/* 63 vars */]) = 0 > Process 12001 detached > [pid 12000] --- SIGCHLD (Child exited) @ 0 (0) --- > Process 12000 detached > --- SIGCHLD (Child exited) @ 0 (0) --- > libnotmuch.so.3 > > So it also prints libnotmuch.so.3, but only because the version > installed from the Debian archive is also libnotmuch.so.3: > > teythoon@thinkbox ~ % /sbin/ldconfig -p | grep notmuch > libnotmuch.so.3 (libc6,x86-64) => /usr/lib/libnotmuch.so.3 > > So I guess *if* I had a libnotmuch.so.2 in my ldconfig(8) cache (-p > prints the cache), find_library would have returned libnotmuch.so.2 > and not my libnotmuch.so.3. So your patch would most likely break this > kind of setup. > > 2. Uh. I actually looked at /usr/lib/python2.7/ctypes/util.py and I > almost got eye cancer from just looking briefly at it... The > implementation of find_library differs for various operating systems > and most (all?) of them use fork/exec some program (even gcc in some > cases) to look up the libraries. I'd like to avoid this if possible, > maybe even if this means handling OS X as a special case in our > bindings. For reference, our current solution does not require any > fork/exec calls: > > teythoon@thinkbox ~ % strace -f -e trace=execve,fork,clone python -c 'import ctypes; ctypes.CDLL("libnotmuch.so.3")' > execve("/usr/bin/python", ["python", "-c", "import ctypes; ctypes.CDLL(\"libn"...], [/* 63 vars */]) = 0 > > Thoughts anyone? Tried this: >>> from ctypes import CDLL >>> x = CDLL("foobnar.so") Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python2.6/ctypes/__init__.py", line 353, in __init__ self._handle = _dlopen(self._name, mode) OSError: foobnar.so: cannot open shared object file: No such file or directory Looked into /usr/lib/python2.6/ctypes/__init__.py -- _dlopen() uses dlopen() from _ctypes module, which is in /usr/lib/python2.6/lib-dynload/_ctypes.so I'd guess this does pretty much the same as dlopen(3) does in Linux (or Mac -- whatever it does there) Could something like try: nmlib = CDLL("libnotmuch.so.3") except OSError: try: nmlib = CDLL("something_macosx_understands") except: ImportError("Could not find shared 'notmuch' library.") be used ? Just before sending this email I came up with this page: https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man3/dlopen.3.html The 'SEARCHING' section has quite a few words how the file is found. > Cheers, > Justus Tomi ^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2013-05-30 16:06 UTC | newest] Thread overview: 3+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2013-05-26 3:53 [PATCH] Fix shared library loading in Python bindings Julian Berman 2013-05-30 12:26 ` Justus Winter 2013-05-30 16:06 ` Tomi Ollila
Code repositories for project(s) associated with this public inbox https://yhetil.org/notmuch.git/ This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).