Re: [PATCH] Fix shared library loading in Python bindings.

[Justus Winter <4winter@informatik.uni-hamburg.de>]

[-- Attachment #1: Type: text/plain, Size: 5209 bytes --]

Hi Julian :)

Quoting Julian Berman (2013-05-26 05:53:38)
> Specifically, fixes loading on OS X, where libnotmuch will be
> a dylib.:
> ---
>  bindings/python/notmuch/globals.py | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/bindings/python/notmuch/globals.py b/bindings/python/notmuch/globals.py
> index c7632c3..5e08e73 100644
> --- a/bindings/python/notmuch/globals.py
> +++ b/bindings/python/notmuch/globals.py
> @@ -18,11 +18,12 @@ Copyright 2010 Sebastian Spaeth <Sebastian@SSpaeth.de>
>  """
>  
>  from ctypes import CDLL, Structure, POINTER
> +from ctypes.util import find_library
>  
>  #-----------------------------------------------------------------------------
>  #package-global instance of the notmuch library
>  try:
> -    nmlib = CDLL("libnotmuch.so.3")
> +    nmlib = CDLL(find_library("libnotmuch"))

Does this work for you on Darwin? On my box (Debian/Linux) I have to
use "notmuch" instead of "libnotmuch" to get anything from
find_library:

In [5]: print ctypes.util.find_library("notmuch")
libnotmuch.so.3

In [6]: print ctypes.util.find_library("libnotmuch")
None

Then again, find_library is different for every architecture under the
sun...

>  except:
>      raise ImportError("Could not find shared 'notmuch' library.")

I'm leaning towards declining the patch in it's current form. If the
bindings do not work on OS X, we need to find another solution. There
are two reasons for this:

1. find_library was once used, but was removed since it is (has?) been
problematic wrt to LD_LIBRARY_PATH usage:

% git show f378f458
commit f378f45893bb4263402008b2abd8aab522901fb6
Author: Cedric Cabessa <ced@ryick.net>
Date:   Mon Apr 5 03:03:51 2010 +0200

    find_library does not read LD_LIBRARY_PATH, but CDLL does.

diff --git a/cnotmuch/globals.py b/cnotmuch/globals.py
index ef2686f..fa20ae8 100644
--- a/cnotmuch/globals.py
+++ b/cnotmuch/globals.py
@@ -3,17 +3,17 @@ from ctypes.util import find_library
 
 #-----------------------------------------------------------------------------
 #package-global instance of the notmuch library
-#TODO: lazy load this on first access?
-so = find_library('notmuch')
-if so is None:
-  raise ImportError("Could not find shared 'notmuch' library.")
-nmlib = CDLL(so)
+try:
+    nmlib = CDLL("libnotmuch.so.1")
+except:
+    raise ImportError("Could not find shared 'notmuch' library.")
[...]

As a heavy LD_LIBRARY_PATH user I don't want to break this. So I tried
to test whether find_library now respects LD_LIBRARY_PATH or not:

teythoon@thinkbox ~ % echo $LD_LIBRARY_PATH
/home/teythoon/.local/lib
teythoon@thinkbox ~ % strace -e trace=open python -c 'import ctypes; ctypes.CDLL("libnotmuch.so.3")' 2>&1 | grep notmuch
open("/home/teythoon/.local/lib/libnotmuch.so.3", O_RDONLY) = 3

That's how it's done now and indeed it finds my libnotmuch.

teythoon@thinkbox ~ % strace -f -e trace=stat,open python -c 'import ctypes.util; ctypes.util.find_library("notmuch")' 2>&1 | grep notmuch

Nothing. Funny. Let's see:

teythoon@thinkbox ~ % strace -f -e trace=fork,execve,clone python -c 'import ctypes.util; print ctypes.util.find_library("notmuch")'
execve("/usr/bin/python", ["python", "-c", "import ctypes.util; print ctypes"...], [/* 63 vars */]) = 0
clone(Process 12000 attached
child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f0734a649d0) = 12000
[pid 12000] execve("/bin/sh", ["sh", "-c", "/sbin/ldconfig -p 2>/dev/null"], [/* 63 vars */]) = 0
[pid 12000] clone(Process 12001 attached
child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f119d5479d0) = 12001
[pid 12001] execve("/sbin/ldconfig", ["/sbin/ldconfig", "-p"], [/* 63 vars */]) = 0
Process 12001 detached
[pid 12000] --- SIGCHLD (Child exited) @ 0 (0) ---
Process 12000 detached
--- SIGCHLD (Child exited) @ 0 (0) ---
libnotmuch.so.3

So it also prints libnotmuch.so.3, but only because the version
installed from the Debian archive is also libnotmuch.so.3:

teythoon@thinkbox ~ % /sbin/ldconfig -p | grep notmuch
        libnotmuch.so.3 (libc6,x86-64) => /usr/lib/libnotmuch.so.3

So I guess *if* I had a libnotmuch.so.2 in my ldconfig(8) cache (-p
prints the cache), find_library would have returned libnotmuch.so.2
and not my libnotmuch.so.3. So your patch would most likely break this
kind of setup.

2. Uh. I actually looked at /usr/lib/python2.7/ctypes/util.py and I
almost got eye cancer from just looking briefly at it... The
implementation of find_library differs for various operating systems
and most (all?) of them use fork/exec some program (even gcc in some
cases) to look up the libraries. I'd like to avoid this if possible,
maybe even if this means handling OS X as a special case in our
bindings. For reference, our current solution does not require any
fork/exec calls:

teythoon@thinkbox ~ % strace -f -e trace=execve,fork,clone python -c 'import ctypes; ctypes.CDLL("libnotmuch.so.3")' 
execve("/usr/bin/python", ["python", "-c", "import ctypes; ctypes.CDLL(\"libn"...], [/* 63 vars */]) = 0

Thoughts anyone?

Cheers,
Justus

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iQIcBAABCAAGBQJRp0VcAAoJENMeiILK3jZYJwMP/R4VQYUmEDFYHX7xuiLL4zoC
r+afAVY4OE2u2c9bncT54QairXuDGlLs8wmpDwkfZeiZ1N8OXeEwDlcMZYZ8BpQD
/5LW0Gcf4u91UIsQ6t3wQLj6slvnMEQmLgyWbHpZIjFMbi/sMF7akJ+Ov/eZf0Si
/9ZTrXteoUWw7kw9NhIqyxRC9/WHB9TwSdCVj7QPRBxK78B3+OxoX7DIp40QB3eZ
MPiaOWRqV/6PrEmM1CxIRAEb5EoNB5LdgMM+0Ip6DtoEoKGbt3E7BbBk//tVTtaZ
OYTCDEDtUfv0bfKL/VHVyihFZDJjDN1H+VgJZ4Sai3rjME1k/HphT/ZiBSO8b995
tpsc3yBB76kWEnHd1wf6DFJTLiKDwhm46V9WeNZiZ9auUm4RfcHozejPIooog9Bo
+qubc9r7FhZpV7qrh9+PfS5MPf7qXb+Q5BpZMU30Cqksy1BwrPtmWJ6dDqucI+pR
/9LxMTxfm8Qub8Fy811nOuG8vcKuL0GxvN/cSM2oqgceDiTXqwCWDO6y3jO3+HBf
yChjoPioIdxfsyRv3UWjxVI0am3lkN0lKHp7tUbCxank9Yaj1yW/DOQhvpt5pNLe
NvuNfYPG+Gk0IIsdOWM+R7EBkkPZmnMGdIJ+fsKPxgFZegeYJRP0FejHtw/HoWhG
R9uuna4ewRUTTFnhxxLU
=IPQn
-----END PGP SIGNATURE-----