Re: SWIG (and particularly Python) bindings

[Adrian Perez de Castro <aperez@igalia.com>]

[-- Attachment #1.1: Type: text/plain, Size: 2425 bytes --]

On Tue, 29 Dec 2009 04:16:43 -0500, Ben wrote:

> Regardless, I thought it might be nice to have access to the notmuch
> backend from a language other than C (preferably my high-level language
> of choice, python) [...]

Funny, I was just doing the same: a Python binding. Haha, so now we have
two just-backed Python bindings. What should we do?

> [...] To this end, I took a few hours today acquainting
> myself with SWIG and produced these bindings for Python. Unfortunately,
> it doesn't appear that SWIG has particularly good support for
> object-oriented C [...]

I already used SWIG sometimes in the past (and did not like it a lot), so
my binding is using Cython [*] (which is exactly like Pyrex plus some extra
features), so the binding is partly manual.

> While the bindings are currently in the form of a patch to notmuch
> (creating a top-level swig directory in the source tree), they could
> certainly be moved out-of-tree if the powers that be don't feel it
> appropriate to include them. [...]

Same here, see attached patch. It is currently unfinished, and I was just
about to add support for iterating notmuch_threads_t and other similar
structures. I can also publish a Git repo with the entire branch, just
drop me a line if you want me to do that.

> [...] Unfortunately, the build system is currently almost entirely
> independent from the notmuch build system. If  these are to be
> included in-tree, I would be curious to hear people have
> to say about how we might integrate it into the sup build system.
                                                  ^^^
(Mmmh, I suppose you mean "notmuch build system" there :P)

Mine is a little more cooked, as I have added a distutils "setup.py"
script. The bad news is that Python modules need to be compiled as
relocatable object files (-fPIC to teh rescue!), and the linker will
refuse to link the generated code with "notmuch.a" -- so I am instructing
distutils to compile *all* sources again. Not nice.

BTW, I think that if more bindings start to appear, Notmuch might be built
as a shared library, to avoid duplicating it everywhere. One option may be
using *just* libtool but not the rest of auto-foo tools (for the record:
I agree with Carl that they are slow and wicked).

Regards,


[*] http://www.cython.org/
-- 
Adrian Perez de Castro <aperez@igalia.com>
Igalia - Free Software Engineering

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.2: notmuch-python-wip.patch --]
[-- Type: text/x-patch, Size: 17456 bytes --]

 Makefile              |    1 +
 python/.gitignore     |    2 +
 python/Makefile       |    6 +
 python/Makefile.local |   15 ++
 python/notmuch.pyx    |  397 +++++++++++++++++++++++++++++++++++++++++++++++++
 python/pyutil.h       |   16 ++
 python/setup.py       |   89 +++++++++++
 7 files changed, 526 insertions(+), 0 deletions(-)

diff --git a/Makefile b/Makefile
index 021fdb8..081d670 100644
--- a/Makefile
+++ b/Makefile
@@ -37,6 +37,7 @@ include Makefile.config
 
 include lib/Makefile.local
 include compat/Makefile.local
+include python/Makefile.local
 include Makefile.local
 
 # The user has not set any verbosity, default to quiet mode and inform the
diff --git a/python/.gitignore b/python/.gitignore
new file mode 100644
index 0000000..7f0efa8
--- /dev/null
+++ b/python/.gitignore
@@ -0,0 +1,2 @@
+notmuch.c
+build/
diff --git a/python/Makefile b/python/Makefile
new file mode 100644
index 0000000..e1e5c43
--- /dev/null
+++ b/python/Makefile
@@ -0,0 +1,6 @@
+
+all: python
+
+%:
+	make -C .. $@
+
diff --git a/python/Makefile.local b/python/Makefile.local
new file mode 100644
index 0000000..140a701
--- /dev/null
+++ b/python/Makefile.local
@@ -0,0 +1,15 @@
+dir=python
+
+python: $(dir)/build/.stamp
+	(cd $(dir) && python setup.py build)
+	touch $@
+
+$(dir)/build/.stamp: lib/notmuch.a
+
+clean: clean-python
+
+clean-python:
+	$(RM) -r $(dir)/build
+
+.PHONY: clean-python python
+
diff --git a/python/notmuch.pyx b/python/notmuch.pyx
new file mode 100644
index 0000000..f38b719
--- /dev/null
+++ b/python/notmuch.pyx
@@ -0,0 +1,397 @@
+#! /usr/bin/env python
+# -*- coding: utf-8 -*-
+# vim: fenc=utf-8 ft=pyrex
+#
+# Copyright © 2009 Adrian Perez <aperez@igalia.com>
+#
+# Distributed under terms of the GPLv3 license.
+#
+
+cdef extern from "talloc.h":
+	void* talloc_init(char *fmt, ...)
+	int   talloc_free(void *ctx)
+
+
+cdef extern from "pyutil.h":
+	#
+	# Utility macros
+	#
+	char** pyutil_alloc_strv(void *ctx, unsigned nitems)
+
+
+cdef extern from "notmuch.h":
+	#
+	# Return status handling
+	#
+	ctypedef enum notmuch_status_t:
+		NOTMUCH_STATUS_SUCCESS
+		NOTMUCH_STATUS_OUT_OF_MEMORY
+		NOTMUCH_STATUS_READONLY_DATABASE
+		NOTMUCH_STATUS_XAPIAN_EXCEPTION
+		NOTMUCH_STATUS_FILE_ERROR
+		NOTMUCH_STATUS_FILE_NOT_EMAIL
+		NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID
+		NOTMUCH_STATUS_NULL_POINTER
+		NOTMUCH_STATUS_TAG_TOO_LONG
+		NOTMUCH_STATUS_UNBALANCED_FREEZE_THAW
+
+	char* notmuch_status_to_string(notmuch_status_t status)
+
+	#
+	# notmuch_database_* -> notmuch.Database
+	#
+	ctypedef enum notmuch_database_mode_t:
+		NOTMUCH_DATABASE_MODE_READ_ONLY
+		NOTMUCH_DATABASE_MODE_READ_WRITE
+
+	ctypedef enum notmuch_sort_t:
+		NOTMUCH_SORT_OLDEST_FIRST
+		NOTMUCH_SORT_NEWEST_FIRST
+		NOTMUCH_SORT_MESSAGE_ID
+
+	ctypedef struct notmuch_database_t
+	ctypedef struct notmuch_messages_t
+	ctypedef struct notmuch_message_t
+	ctypedef struct notmuch_threads_t
+	ctypedef struct notmuch_thread_t
+	ctypedef struct notmuch_query_t
+	ctypedef struct notmuch_tags_t
+	ctypedef int time_t
+
+	int notmuch_threads_has_more(notmuch_threads_t *threads)
+	void notmuch_threads_advance(notmuch_threads_t *threads)
+	void notmuch_threads_destroy(notmuch_threads_t *threads)
+	notmuch_thread_t* notmuch_threads_get(notmuch_threads_t *threads)
+
+	void   notmuch_thread_destroy(notmuch_thread_t *thread)
+	char*  notmuch_thread_get_authors(notmuch_thread_t *thread)
+	char*  notmuch_thread_get_subject(notmuch_thread_t *thread)
+	char*  notmuch_thread_get_thread_id(notmuch_thread_t *thread)
+	time_t notmuch_thread_get_oldest_date(notmuch_thread_t *thread)
+	time_t notmuch_thread_get_newest_date(notmuch_thread_t *thread)
+	int    notmuch_thread_get_total_messages(notmuch_thread_t *thread)
+	int    notmuch_thread_get_matched_messages(notmuch_thread_t *thread)
+	notmuch_tags_t*     notmuch_thread_get_tags(notmuch_thread_t *thread)
+	notmuch_messages_t* notmuch_thread_get_toplevel_messages(notmuch_thread_t *thread)
+
+	notmuch_database_t* notmuch_database_create(char *path)
+	notmuch_database_t* notmuch_database_open(char *path, notmuch_database_mode_t mode)
+	char* notmuch_database_get_path(notmuch_database_t *db)
+	void  notmuch_database_close(notmuch_database_t *db)
+
+	time_t notmuch_database_get_timestamp(notmuch_database_t *db, char *key)
+	notmuch_status_t notmuch_database_set_timestamp(
+			notmuch_database_t *db, char *key, time_t timestamp)
+
+	notmuch_status_t notmuch_database_add_message(
+			notmuch_database_t *db, char *filename, notmuch_message_t **mesage)
+
+	notmuch_message_t* notmuch_database_find_message(
+			notmuch_database_t *db, char *message_id)
+
+	notmuch_tags_t* notmuch_database_get_all_tags(notmuch_database_t *db)
+
+	notmuch_query_t* notmuch_query_create(notmuch_database_t *db, char *qstring)
+
+	void notmuch_query_destroy(notmuch_query_t *query)
+	void notmuch_query_set_sort(notmuch_query_t *query, notmuch_sort_t sort)
+	unsigned notmuch_query_count_messages(notmuch_query_t *query)
+	notmuch_threads_t*  notmuch_query_search_threads(notmuch_query_t *query)
+	notmuch_messages_t* notmuch_query_search_messages(notmuch_query_t *query)
+
+	char* notmuch_message_get_message_id(notmuch_message_t *msg)
+	char* notmuch_message_get_thread_id(notmuch_message_t *msg)
+	char* notmuch_message_get_filename(notmuch_message_t *msg)
+	char* notmuch_message_get_header(notmuch_message_t *msg, char *name)
+	notmuch_status_t notmuch_message_add_tag(notmuch_message_t *msg, char *tag)
+	notmuch_status_t notmuch_message_remove_tag(notmuch_message_t *msg, char *tag)
+	void notmuch_message_remove_all_tags(notmuch_message_t *msg)
+	void notmuch_message_destroy(notmuch_message_t *msg)
+	void notmuch_message_freeze(notmuch_message_t *msg)
+	void notmuch_message_thaw(notmuch_message_t *msg)
+
+
+cdef extern from "notmuch-client.h":
+	#
+	# notmuch_config_* -> notmuch.Config
+	#
+	ctypedef struct notmuch_config_t
+
+	notmuch_config_t* notmuch_config_open(void *ctx, char *filename, int *is_new_ret)
+	void   notmuch_config_close(notmuch_config_t *cfg)
+	int    notmuch_config_save(notmuch_config_t *cfg)
+	char*  notmuch_config_get_database_path(notmuch_config_t *cfg)
+	void   notmuch_config_set_database_path(notmuch_config_t *cfg, char *path)
+	char*  notmuch_config_get_user_name(notmuch_config_t *cfg)
+	void   notmuch_config_set_user_name(notmuch_config_t *cfg, char *name)
+	char*  notmuch_config_get_user_primary_email(notmuch_config_t *cfg)
+	void   notmuch_config_set_user_primary_email(notmuch_config_t *cfg, char *email)
+	char** notmuch_config_get_user_other_email(notmuch_config_t *cfg, size_t *length)
+	void   notmuch_config_set_user_other_email(notmuch_config_t *cfg, char **other_email, size_t length)
+
+	#
+	# Miscellaneous
+	#
+	int debugger_is_active()
+
+
+#
+# Import needed Python built-in modules
+#
+from datetime import datetime, date
+
+#
+# Miscellaneous functions and information
+#
+debugger_active = bool(debugger_is_active())
+
+
+#
+# notmuch_database_* -> notmuch.Database
+#
+
+
+#
+# notmuch_config_* -> notmuch.Config
+#
+cdef class Config:
+	"""Handles the Notmuch configuration file.
+	"""
+	cdef notmuch_config_t * _cfg
+	cdef void * _ctx
+	cdef object _filename
+
+	def __init__(self, filename="~/.notmuch-config"):
+		"""Open a Notmuch configuration file.
+		"""
+		cdef int newret
+		self._ctx = talloc_init("notmuch.Config")
+		self._cfg = notmuch_config_open(self._ctx, filename, &newret)
+		self._filename = filename
+
+	def __dealloc__(self):
+		notmuch_config_close(self._cfg)
+		talloc_free(self._ctx)
+
+	property filename:
+		"""File name containing the configuration (string)"""
+		def __get__(self):
+			return self._filename
+
+	property database_path:
+		"""Path to the Notmuch database (string)"""
+		def __get__(self):
+			return notmuch_config_get_database_path(self._cfg)
+		def __set__(self, path):
+			notmuch_config_set_database_path(self._cfg, path)
+
+	property user_name:
+		"""User name (string)"""
+		def __get__(self):
+			return notmuch_config_get_user_name(self._cfg)
+		def __set__(self, name):
+			notmuch_config_set_user_name(self._cfg, name)
+
+	property user_primary_email:
+		"""Primary e-mail address of the user (string)"""
+		def __get__(self):
+			return notmuch_config_get_user_primary_email(self._cfg)
+		def __set__(self, email):
+			notmuch_config_set_user_primary_email(self._cfg, email)
+
+	property other_email:
+		"""List of other e-mail addresses of the user (tuple of strings)"""
+		def __get__(self):
+			cdef size_t length
+			cdef size_t i
+			cdef char **emails
+			emails = notmuch_config_get_user_other_email(self._cfg, &length)
+
+			result = []
+			for i from 0 <= i < length:
+				result.append(emails[i])
+
+			# XXX We do not want the result to be modifiable, because the property
+			#     must be assigned as a whole, and not just modified only in the
+			#     Python side of the world.
+			return tuple(result)
+
+		def __set__(self, emaillist):
+			cdef size_t length = len(emaillist)
+			cdef char **emails = pyutil_alloc_strv(self._ctx, length)
+			cdef size_t i
+
+			for i from 0 <= i < len(emaillist):
+				emails[i] = emaillist[i]
+			notmuch_config_set_user_other_email(self._cfg, emails, len(emaillist))
+
+
+	def save(self):
+		"""Save the Notmuch configuration"""
+		notmuch_config_save(self._cfg)
+
+
+cdef class Database
+
+
+cdef class Message:
+	cdef notmuch_message_t *_msg
+
+	def __cinit__(self, object messageptr):
+		# XXX Counterpart of bogus cast
+		self._msg = <notmuch_message_t*> messageptr
+
+	def __dealloc__(self):
+		notmuch_message_destroy(self._msg)
+
+	property message_id:
+		def __get__(self):
+			return notmuch_message_get_message_id(self._msg)
+
+	property thread_id:
+		def __get__(self):
+			return notmuch_message_get_thread_id(self._msg)
+
+	property filename:
+		def __get__(self):
+			return notmuch_message_get_filename(self._msg)
+
+	def get_header(self, name):
+		return notmuch_message_get_header(self._msg, name)
+
+	def add_tag(self, tag):
+		cdef notmuch_status_t ret = notmuch_message_add_tag(self._msg, tag)
+		if ret != NOTMUCH_STATUS_SUCCESS:
+			raise ValueError(notmuch_status_to_string(ret))
+
+	def remove_tag(self, tag):
+		cdef notmuch_status_t ret = notmuch_message_remove_tag(self._msg, tag)
+		if ret != NOTMUCH_STATUS_SUCCESS:
+			raise ValueError(notmuch_status_to_string(ret))
+
+	def remove_all_tags(self):
+		notmuch_message_remove_all_tags(self._msg)
+
+	def freeze(self):
+		notmuch_message_freeze(self._msg)
+
+	def thaw(self):
+		notmuch_message_thaw(self._msg)
+
+
+cdef class Thread:
+	cdef notmuch_thread_t *_thread
+
+	def __cinit__(self, object threadptr):
+		self._thread = <notmuch_thread_t*> threadptr
+
+	def __dealloc__(self):
+		notmuch_thread_destroy(self._thread)
+
+	property authors:
+		def __get__(self):
+			return notmuch_thread_get_authors(self._thread)
+
+	property subject:
+		def __get__(self):
+			return notmuch_thread_get_subject(self._thread)
+
+	property thread_id:
+		def __get__(self):
+			return notmuch_thread_get_thread_id(self._thread)
+
+	property oldest_date:
+		def __get__(self):
+			return datetime.fromtimestamp(notmuch_thread_get_oldest_date(self._thread))
+
+	property newest_date:
+		def __get__(self):
+			return datetime.fromtimestamp(notmuch_thread_get_newest_date(self._thread))
+
+	property total_messages:
+		def __get__(self):
+			return notmuch_thread_get_total_messages(self._thread)
+
+	property matched_messages:
+		def __get__(self):
+			return notmuch_thread_get_matched_messages(self._thread)
+
+	def __len__(self):
+		return self.matched_messages
+
+
+cdef class Query:
+	cdef notmuch_query_t *_query
+
+	def __cinit__(self, Database db not None, qs):
+		self._query = notmuch_query_create(db._db, qs)
+
+	def __dealloc__(self):
+		notmuch_query_destroy(self._query)
+
+	def __len__(self):
+		return notmuch_query_count_messages(self._query)
+
+	def sort(self, notmuch_sort_t ordering):
+		notmuch_query_set_sort(self._query, ordering)
+		return self
+
+
+
+cdef class Database:
+	cdef notmuch_database_t *_db
+
+	def __init__(self, path=None, readonly=True):
+		cdef notmuch_database_mode_t mode = NOTMUCH_DATABASE_MODE_READ_WRITE
+		if path is None:
+			path = Config().database_path
+		if readonly:
+			mode = NOTMUCH_DATABASE_MODE_READ_ONLY
+		self._db = notmuch_database_open(path, mode)
+
+	def __dealloc__(self):
+		notmuch_database_close(self._db)
+
+	def get_timestamp(self, key):
+		cdef time_t ts = notmuch_database_get_timestamp(self._db, key)
+		return datetime.fromtimestamp(ts)
+
+	def set_timestamp(self, key, timestamp):
+		cdef time_t ts
+		if isinstance(timestamp, date):
+			ts = int(timestamp.strftime("%s"))
+		elif isinstance(timestamp, float):
+			ts = <time_t> timestamp
+		elif isinstance(timestamp, int):
+			ts = timestamp
+		else:
+			raise ValueError("Numeric timestamp or datetime.date object expected")
+
+	def add_message(self, filename):
+		cdef notmuch_message_t *message = NULL
+		cdef notmuch_status_t   status
+		status = notmuch_database_add_message(self._db, filename, &message)
+		if status == NOTMUCH_STATUS_SUCCESS:
+			# XXX This cast seems bogus, it may work, though
+			return Message(<object> message)
+		else:
+			if message != NULL:
+				notmuch_message_destroy(message)
+			raise ValueError(notmuch_status_to_string(status))
+
+	def find_message(self, message_id):
+		cdef notmuch_message_t *message
+		message = notmuch_database_find_message(self._db, message_id)
+		if message == NULL:
+			raise KeyError(message_id)
+		return Message(<object> message)
+
+	property path:
+		"""Database path"""
+		def __get__(self):
+			return notmuch_database_get_path(self._db)
+
+	def query(self, qstring):
+		return Query(self, qstring)
+
+
diff --git a/python/pyutil.h b/python/pyutil.h
new file mode 100644
index 0000000..64d93bf
--- /dev/null
+++ b/python/pyutil.h
@@ -0,0 +1,16 @@
+/*
+ * pyutil.h
+ * Copyright (C) 2009 Adrian Perez <aperez@igalia.com>
+ *
+ * Distributed under terms of the GPLv3 license.
+ */
+
+#ifndef __pyutil_h__
+#define __pyutil_h__
+
+#include <talloc.h>
+
+#define pyutil_alloc_strv(_ctx, _n)  talloc_array ((_ctx), char*, (_n))
+
+#endif /* !__pyutil_h__ */
+
diff --git a/python/setup.py b/python/setup.py
new file mode 100644
index 0000000..ffd43b2
--- /dev/null
+++ b/python/setup.py
@@ -0,0 +1,89 @@
+#! /usr/bin/env python
+# -*- coding: utf-8 -*-
+# vim:fenc=utf-8
+#
+# Copyright © 2009 Adrian Perez <aperez@igalia.com>
+#
+# Distributed under terms of the GPLv3 license.
+
+from distutils.core import setup
+from distutils.extension import Extension
+from Cython.Distutils import build_ext
+import commands
+
+
+class FlagOMatic(dict):
+    _KEYS = ("extra_link_args", "include_dirs", "library_dirs", "libraries")
+
+    def __init__(self, *arg, **kw):
+        super(FlagOMatic, self).__init__(*arg, **kw)
+        for key in self._KEYS:
+            self[key] = set(self.get(key, []))
+
+    extra_link_args = property(lambda self: self["extra_link_args"])
+    include_dirs    = property(lambda self: self["include_dirs"])
+    library_dirs    = property(lambda self: self["library_dirs"])
+    libraries       = property(lambda self: self["libraries"])
+
+    _FLAG_MAP = {
+            "-I": "include_dirs",
+            "-L": "library_dirs",
+            "-l": "libraries",
+        }
+
+    def add_compiler_flags(self, text):
+        for token in text.split():
+            key = self._FLAG_MAP.get(token[:2], "extra_link_args")
+            if key == "extra_link_args":
+                self.extra_link_args.add(token)
+            else:
+                self[key].add(token[2:])
+
+    def add_pkgconfig_flags(self, *modules):
+        self.add_command_output_flags(
+                "pkg-config --libs --cflags %s" % " ".join(modules))
+
+    def add_command_output_flags(self, command):
+        self.add_compiler_flags(commands.getoutput(command))
+
+    @property
+    def kwargs(self):
+        return dict((k, list(v)) for k, v in self.iteritems())
+
+
+# Gather compiler flags
+#
+flags = FlagOMatic(
+       #extra_link_args = ("../lib/notmuch.a",),
+        include_dirs    = ("..", "../lib", "../compat"))
+
+flags.add_pkgconfig_flags("gmime-2.4", "talloc")
+flags.add_command_output_flags("xapian-config --cxxflags --libs")
+
+# We are building a extension module
+#
+import os
+
+srcs = ["notmuch.pyx"]
+srcs.extend(
+    map(lambda s: "../"+s,
+        filter(lambda s: s.endswith(".c"),
+            os.listdir(".."))))
+srcs.extend(
+    map(lambda s: "../lib/"+s,
+        filter(lambda s: s.endswith(".cc") or s.endswith(".c"),
+            os.listdir("../lib"))))
+
+ext_modules = [Extension("notmuch", srcs, **flags.kwargs)]
+
+
+# And now for the easy part :-)
+#
+setup(
+    name         = "notmuch",
+    author       = "Adrian Perez",
+    author_email = "aperez@igalia.com",
+    cmdclass     = {"build_ext": build_ext},
+    ext_modules  = ext_modules,
+)
+

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]