[PATCH] Fix shared library loading in Python bindings.

[PATCH] Fix shared library loading in Python bindings.

[Julian Berman]

Specifically, fixes loading on OS X, where libnotmuch will be
a dylib.:
---
 bindings/python/notmuch/globals.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/bindings/python/notmuch/globals.py b/bindings/python/notmuch/globals.py
index c7632c3..5e08e73 100644
--- a/bindings/python/notmuch/globals.py
+++ b/bindings/python/notmuch/globals.py
@@ -18,11 +18,12 @@ Copyright 2010 Sebastian Spaeth <Sebastian@SSpaeth.de>
 """
 
 from ctypes import CDLL, Structure, POINTER
+from ctypes.util import find_library
 
 #-----------------------------------------------------------------------------
 #package-global instance of the notmuch library
 try:
-    nmlib = CDLL("libnotmuch.so.3")
+    nmlib = CDLL(find_library("libnotmuch"))
 except:
     raise ImportError("Could not find shared 'notmuch' library.")
 
-- 
1.8.3

Re: [PATCH] Fix shared library loading in Python bindings.

[Justus Winter]

[-- Attachment #1: Type: text/plain, Size: 5209 bytes --]

Hi Julian :)

Quoting Julian Berman (2013-05-26 05:53:38)
> Specifically, fixes loading on OS X, where libnotmuch will be
> a dylib.:
> ---
>  bindings/python/notmuch/globals.py | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/bindings/python/notmuch/globals.py b/bindings/python/notmuch/globals.py
> index c7632c3..5e08e73 100644
> --- a/bindings/python/notmuch/globals.py
> +++ b/bindings/python/notmuch/globals.py
> @@ -18,11 +18,12 @@ Copyright 2010 Sebastian Spaeth <Sebastian@SSpaeth.de>
>  """
>  
>  from ctypes import CDLL, Structure, POINTER
> +from ctypes.util import find_library
>  
>  #-----------------------------------------------------------------------------
>  #package-global instance of the notmuch library
>  try:
> -    nmlib = CDLL("libnotmuch.so.3")
> +    nmlib = CDLL(find_library("libnotmuch"))

Does this work for you on Darwin? On my box (Debian/Linux) I have to
use "notmuch" instead of "libnotmuch" to get anything from
find_library:

In [5]: print ctypes.util.find_library("notmuch")
libnotmuch.so.3

In [6]: print ctypes.util.find_library("libnotmuch")
None

Then again, find_library is different for every architecture under the
sun...

>  except:
>      raise ImportError("Could not find shared 'notmuch' library.")

I'm leaning towards declining the patch in it's current form. If the
bindings do not work on OS X, we need to find another solution. There
are two reasons for this:

1. find_library was once used, but was removed since it is (has?) been
problematic wrt to LD_LIBRARY_PATH usage:

% git show f378f458
commit f378f45893bb4263402008b2abd8aab522901fb6
Author: Cedric Cabessa <ced@ryick.net>
Date:   Mon Apr 5 03:03:51 2010 +0200

    find_library does not read LD_LIBRARY_PATH, but CDLL does.

diff --git a/cnotmuch/globals.py b/cnotmuch/globals.py
index ef2686f..fa20ae8 100644
--- a/cnotmuch/globals.py
+++ b/cnotmuch/globals.py
@@ -3,17 +3,17 @@ from ctypes.util import find_library
 
 #-----------------------------------------------------------------------------
 #package-global instance of the notmuch library
-#TODO: lazy load this on first access?
-so = find_library('notmuch')
-if so is None:
-  raise ImportError("Could not find shared 'notmuch' library.")
-nmlib = CDLL(so)
+try:
+    nmlib = CDLL("libnotmuch.so.1")
+except:
+    raise ImportError("Could not find shared 'notmuch' library.")
[...]

As a heavy LD_LIBRARY_PATH user I don't want to break this. So I tried
to test whether find_library now respects LD_LIBRARY_PATH or not:

teythoon@thinkbox ~ % echo $LD_LIBRARY_PATH
/home/teythoon/.local/lib
teythoon@thinkbox ~ % strace -e trace=open python -c 'import ctypes; ctypes.CDLL("libnotmuch.so.3")' 2>&1 | grep notmuch
open("/home/teythoon/.local/lib/libnotmuch.so.3", O_RDONLY) = 3

That's how it's done now and indeed it finds my libnotmuch.

teythoon@thinkbox ~ % strace -f -e trace=stat,open python -c 'import ctypes.util; ctypes.util.find_library("notmuch")' 2>&1 | grep notmuch

Nothing. Funny. Let's see:

teythoon@thinkbox ~ % strace -f -e trace=fork,execve,clone python -c 'import ctypes.util; print ctypes.util.find_library("notmuch")'
execve("/usr/bin/python", ["python", "-c", "import ctypes.util; print ctypes"...], [/* 63 vars */]) = 0
clone(Process 12000 attached
child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f0734a649d0) = 12000
[pid 12000] execve("/bin/sh", ["sh", "-c", "/sbin/ldconfig -p 2>/dev/null"], [/* 63 vars */]) = 0
[pid 12000] clone(Process 12001 attached
child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f119d5479d0) = 12001
[pid 12001] execve("/sbin/ldconfig", ["/sbin/ldconfig", "-p"], [/* 63 vars */]) = 0
Process 12001 detached
[pid 12000] --- SIGCHLD (Child exited) @ 0 (0) ---
Process 12000 detached
--- SIGCHLD (Child exited) @ 0 (0) ---
libnotmuch.so.3

So it also prints libnotmuch.so.3, but only because the version
installed from the Debian archive is also libnotmuch.so.3:

teythoon@thinkbox ~ % /sbin/ldconfig -p | grep notmuch
        libnotmuch.so.3 (libc6,x86-64) => /usr/lib/libnotmuch.so.3

So I guess *if* I had a libnotmuch.so.2 in my ldconfig(8) cache (-p
prints the cache), find_library would have returned libnotmuch.so.2
and not my libnotmuch.so.3. So your patch would most likely break this
kind of setup.

2. Uh. I actually looked at /usr/lib/python2.7/ctypes/util.py and I
almost got eye cancer from just looking briefly at it... The
implementation of find_library differs for various operating systems
and most (all?) of them use fork/exec some program (even gcc in some
cases) to look up the libraries. I'd like to avoid this if possible,
maybe even if this means handling OS X as a special case in our
bindings. For reference, our current solution does not require any
fork/exec calls:

teythoon@thinkbox ~ % strace -f -e trace=execve,fork,clone python -c 'import ctypes; ctypes.CDLL("libnotmuch.so.3")' 
execve("/usr/bin/python", ["python", "-c", "import ctypes; ctypes.CDLL(\"libn"...], [/* 63 vars */]) = 0

Thoughts anyone?

Cheers,
Justus

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iQIcBAABCAAGBQJRp0VcAAoJENMeiILK3jZYJwMP/R4VQYUmEDFYHX7xuiLL4zoC
r+afAVY4OE2u2c9bncT54QairXuDGlLs8wmpDwkfZeiZ1N8OXeEwDlcMZYZ8BpQD
/5LW0Gcf4u91UIsQ6t3wQLj6slvnMEQmLgyWbHpZIjFMbi/sMF7akJ+Ov/eZf0Si
/9ZTrXteoUWw7kw9NhIqyxRC9/WHB9TwSdCVj7QPRBxK78B3+OxoX7DIp40QB3eZ
MPiaOWRqV/6PrEmM1CxIRAEb5EoNB5LdgMM+0Ip6DtoEoKGbt3E7BbBk//tVTtaZ
OYTCDEDtUfv0bfKL/VHVyihFZDJjDN1H+VgJZ4Sai3rjME1k/HphT/ZiBSO8b995
tpsc3yBB76kWEnHd1wf6DFJTLiKDwhm46V9WeNZiZ9auUm4RfcHozejPIooog9Bo
+qubc9r7FhZpV7qrh9+PfS5MPf7qXb+Q5BpZMU30Cqksy1BwrPtmWJ6dDqucI+pR
/9LxMTxfm8Qub8Fy811nOuG8vcKuL0GxvN/cSM2oqgceDiTXqwCWDO6y3jO3+HBf
yChjoPioIdxfsyRv3UWjxVI0am3lkN0lKHp7tUbCxank9Yaj1yW/DOQhvpt5pNLe
NvuNfYPG+Gk0IIsdOWM+R7EBkkPZmnMGdIJ+fsKPxgFZegeYJRP0FejHtw/HoWhG
R9uuna4ewRUTTFnhxxLU
=IPQn
-----END PGP SIGNATURE-----

Re: [PATCH] Fix shared library loading in Python bindings.

[Tomi Ollila]

On Thu, May 30 2013, Justus Winter <4winter@informatik.uni-hamburg.de> wrote:

> Hi Julian :)
>
> Quoting Julian Berman (2013-05-26 05:53:38)
>> Specifically, fixes loading on OS X, where libnotmuch will be
>> a dylib.:
>> ---
>>  bindings/python/notmuch/globals.py | 3 ++-
>>  1 file changed, 2 insertions(+), 1 deletion(-)
>> 
>> diff --git a/bindings/python/notmuch/globals.py b/bindings/python/notmuch/globals.py
>> index c7632c3..5e08e73 100644
>> --- a/bindings/python/notmuch/globals.py
>> +++ b/bindings/python/notmuch/globals.py
>> @@ -18,11 +18,12 @@ Copyright 2010 Sebastian Spaeth <Sebastian@SSpaeth.de>
>>  """
>>  
>>  from ctypes import CDLL, Structure, POINTER
>> +from ctypes.util import find_library
>>  
>>  #-----------------------------------------------------------------------------
>>  #package-global instance of the notmuch library
>>  try:
>> -    nmlib = CDLL("libnotmuch.so.3")
>> +    nmlib = CDLL(find_library("libnotmuch"))
>
> Does this work for you on Darwin? On my box (Debian/Linux) I have to
> use "notmuch" instead of "libnotmuch" to get anything from
> find_library:
>
> In [5]: print ctypes.util.find_library("notmuch")
> libnotmuch.so.3
>
> In [6]: print ctypes.util.find_library("libnotmuch")
> None
>
> Then again, find_library is different for every architecture under the
> sun...
>
>>  except:
>>      raise ImportError("Could not find shared 'notmuch' library.")
>
> I'm leaning towards declining the patch in it's current form. If the
> bindings do not work on OS X, we need to find another solution. There
> are two reasons for this:
>
> 1. find_library was once used, but was removed since it is (has?) been
> problematic wrt to LD_LIBRARY_PATH usage:
>
> % git show f378f458
> commit f378f45893bb4263402008b2abd8aab522901fb6
> Author: Cedric Cabessa <ced@ryick.net>
> Date:   Mon Apr 5 03:03:51 2010 +0200
>
>     find_library does not read LD_LIBRARY_PATH, but CDLL does.
>
> diff --git a/cnotmuch/globals.py b/cnotmuch/globals.py
> index ef2686f..fa20ae8 100644
> --- a/cnotmuch/globals.py
> +++ b/cnotmuch/globals.py
> @@ -3,17 +3,17 @@ from ctypes.util import find_library
>  
>  #-----------------------------------------------------------------------------
>  #package-global instance of the notmuch library
> -#TODO: lazy load this on first access?
> -so = find_library('notmuch')
> -if so is None:
> -  raise ImportError("Could not find shared 'notmuch' library.")
> -nmlib = CDLL(so)
> +try:
> +    nmlib = CDLL("libnotmuch.so.1")
> +except:
> +    raise ImportError("Could not find shared 'notmuch' library.")
> [...]
>
> As a heavy LD_LIBRARY_PATH user I don't want to break this. So I tried
> to test whether find_library now respects LD_LIBRARY_PATH or not:
>
> teythoon@thinkbox ~ % echo $LD_LIBRARY_PATH
> /home/teythoon/.local/lib
> teythoon@thinkbox ~ % strace -e trace=open python -c 'import ctypes; ctypes.CDLL("libnotmuch.so.3")' 2>&1 | grep notmuch
> open("/home/teythoon/.local/lib/libnotmuch.so.3", O_RDONLY) = 3
>
> That's how it's done now and indeed it finds my libnotmuch.
>
> teythoon@thinkbox ~ % strace -f -e trace=stat,open python -c 'import ctypes.util; ctypes.util.find_library("notmuch")' 2>&1 | grep notmuch
>
> Nothing. Funny. Let's see:
>
> teythoon@thinkbox ~ % strace -f -e trace=fork,execve,clone python -c 'import ctypes.util; print ctypes.util.find_library("notmuch")'
> execve("/usr/bin/python", ["python", "-c", "import ctypes.util; print ctypes"...], [/* 63 vars */]) = 0
> clone(Process 12000 attached
> child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f0734a649d0) = 12000
> [pid 12000] execve("/bin/sh", ["sh", "-c", "/sbin/ldconfig -p 2>/dev/null"], [/* 63 vars */]) = 0
> [pid 12000] clone(Process 12001 attached
> child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f119d5479d0) = 12001
> [pid 12001] execve("/sbin/ldconfig", ["/sbin/ldconfig", "-p"], [/* 63 vars */]) = 0
> Process 12001 detached
> [pid 12000] --- SIGCHLD (Child exited) @ 0 (0) ---
> Process 12000 detached
> --- SIGCHLD (Child exited) @ 0 (0) ---
> libnotmuch.so.3
>
> So it also prints libnotmuch.so.3, but only because the version
> installed from the Debian archive is also libnotmuch.so.3:
>
> teythoon@thinkbox ~ % /sbin/ldconfig -p | grep notmuch
>         libnotmuch.so.3 (libc6,x86-64) => /usr/lib/libnotmuch.so.3
>
> So I guess *if* I had a libnotmuch.so.2 in my ldconfig(8) cache (-p
> prints the cache), find_library would have returned libnotmuch.so.2
> and not my libnotmuch.so.3. So your patch would most likely break this
> kind of setup.
>
> 2. Uh. I actually looked at /usr/lib/python2.7/ctypes/util.py and I
> almost got eye cancer from just looking briefly at it... The
> implementation of find_library differs for various operating systems
> and most (all?) of them use fork/exec some program (even gcc in some
> cases) to look up the libraries. I'd like to avoid this if possible,
> maybe even if this means handling OS X as a special case in our
> bindings. For reference, our current solution does not require any
> fork/exec calls:
>
> teythoon@thinkbox ~ % strace -f -e trace=execve,fork,clone python -c 'import ctypes; ctypes.CDLL("libnotmuch.so.3")' 
> execve("/usr/bin/python", ["python", "-c", "import ctypes; ctypes.CDLL(\"libn"...], [/* 63 vars */]) = 0
>
> Thoughts anyone?

Tried this:

>>> from ctypes import CDLL
>>> x = CDLL("foobnar.so")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.6/ctypes/__init__.py", line 353, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: foobnar.so: cannot open shared object file: No such file or
directory

Looked into /usr/lib/python2.6/ctypes/__init__.py -- _dlopen() uses
dlopen() from _ctypes module, which is in 
/usr/lib/python2.6/lib-dynload/_ctypes.so

I'd guess this does pretty much the same as dlopen(3) does in Linux
(or Mac -- whatever it does there)

Could something like

try:
    nmlib = CDLL("libnotmuch.so.3")
except OSError:
    try:
        nmlib = CDLL("something_macosx_understands")
    except:
        ImportError("Could not find shared 'notmuch' library.")


be used ?

Just before sending this email I came up with this page:

https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man3/dlopen.3.html

The 'SEARCHING' section has quite a few words how the file is found.

> Cheers,
> Justus


Tomi