unofficial mirror of notmuch@notmuchmail.org
 help / color / mirror / code / Atom feed
* Python3 cffi bindings
@ 2019-10-08 21:03 Floris Bruynooghe
  2019-10-08 21:03 ` [PATCH] Introduce CFFI-based python bindings Floris Bruynooghe
                   ` (3 more replies)
  0 siblings, 4 replies; 19+ messages in thread
From: Floris Bruynooghe @ 2019-10-08 21:03 UTC (permalink / raw)
  To: notmuch

Hi all,

IIRC there was a thread in August about another attempt at bringing
the CFFI-based bindings on board as a Python3-only version.  I
believe there was a desire to re-name things but my searching-fu is
failing me and I can no longer find the email thread.

Anyway, I found the code, checked things work, updated tests on new
python versions, added a very basic intergration with the test
framework and squashed the commits.  Otherwise the attached patch
is just a plain dump of the current state so interested people have
at least a copy of the code again which can be made to work.

IIRC this probably wants to be renamed to "notmuch2" instead of
"notdb".  Otherwise I'm pretty sure this doesn't cover all the
current features either.

So maybe this can be used as a start to figure out how to merge
this if there's still an interest in this.

Cheers,
Floris

^ permalink raw reply	[flat|nested] 19+ messages in thread

* [PATCH] Introduce CFFI-based python bindings
  2019-10-08 21:03 Python3 cffi bindings Floris Bruynooghe
@ 2019-10-08 21:03 ` Floris Bruynooghe
  2019-10-08 22:24 ` Python3 cffi bindings David Bremner
                   ` (2 subsequent siblings)
  3 siblings, 0 replies; 19+ messages in thread
From: Floris Bruynooghe @ 2019-10-08 21:03 UTC (permalink / raw)
  To: notmuch; +Cc: Floris Bruynooghe

From: Floris Bruynooghe <flub@google.com>

This introduces CFFI-based Python3-only bindings.
The bindings aim at:
- Better performance on pypy
- Easier to use Python-C interface
- More "pythonic"
  - The API should not allow invalid operations
  - Use native object protocol where possible
- Memory safety; whatever you do from python, it should not coredump.
---
 AUTHORS                                     |   1 +
 bindings/python-cffi/MANIFEST.in            |   2 +
 bindings/python-cffi/notdb/__init__.py      |  62 ++
 bindings/python-cffi/notdb/_base.py         | 238 +++++++
 bindings/python-cffi/notdb/_build.py        | 300 +++++++++
 bindings/python-cffi/notdb/_database.py     | 705 ++++++++++++++++++++
 bindings/python-cffi/notdb/_errors.py       | 112 ++++
 bindings/python-cffi/notdb/_message.py      | 691 +++++++++++++++++++
 bindings/python-cffi/notdb/_query.py        |  83 +++
 bindings/python-cffi/notdb/_tags.py         | 338 ++++++++++
 bindings/python-cffi/notdb/_thread.py       | 190 ++++++
 bindings/python-cffi/setup.py               |  22 +
 bindings/python-cffi/tests/conftest.py      | 142 ++++
 bindings/python-cffi/tests/test_base.py     | 116 ++++
 bindings/python-cffi/tests/test_database.py | 326 +++++++++
 bindings/python-cffi/tests/test_message.py  | 226 +++++++
 bindings/python-cffi/tests/test_tags.py     | 177 +++++
 bindings/python-cffi/tests/test_thread.py   | 102 +++
 bindings/python-cffi/tox.ini                |  16 +
 test/T391-pytest.sh                         |  15 +
 20 files changed, 3864 insertions(+)
 create mode 100644 bindings/python-cffi/MANIFEST.in
 create mode 100644 bindings/python-cffi/notdb/__init__.py
 create mode 100644 bindings/python-cffi/notdb/_base.py
 create mode 100644 bindings/python-cffi/notdb/_build.py
 create mode 100644 bindings/python-cffi/notdb/_database.py
 create mode 100644 bindings/python-cffi/notdb/_errors.py
 create mode 100644 bindings/python-cffi/notdb/_message.py
 create mode 100644 bindings/python-cffi/notdb/_query.py
 create mode 100644 bindings/python-cffi/notdb/_tags.py
 create mode 100644 bindings/python-cffi/notdb/_thread.py
 create mode 100644 bindings/python-cffi/setup.py
 create mode 100644 bindings/python-cffi/tests/conftest.py
 create mode 100644 bindings/python-cffi/tests/test_base.py
 create mode 100644 bindings/python-cffi/tests/test_database.py
 create mode 100644 bindings/python-cffi/tests/test_message.py
 create mode 100644 bindings/python-cffi/tests/test_tags.py
 create mode 100644 bindings/python-cffi/tests/test_thread.py
 create mode 100644 bindings/python-cffi/tox.ini
 create mode 100755 test/T391-pytest.sh

diff --git a/AUTHORS b/AUTHORS
index 5fe5006f..6a05b441 100644
--- a/AUTHORS
+++ b/AUTHORS
@@ -28,3 +28,4 @@ ideas, inspiration, testing or feedback):
 Martin Krafft
 Keith Packard
 Jamey Sharp
+Google LLC
diff --git a/bindings/python-cffi/MANIFEST.in b/bindings/python-cffi/MANIFEST.in
new file mode 100644
index 00000000..9ef81f24
--- /dev/null
+++ b/bindings/python-cffi/MANIFEST.in
@@ -0,0 +1,2 @@
+include MANIFEST.in
+include tox.ini
diff --git a/bindings/python-cffi/notdb/__init__.py b/bindings/python-cffi/notdb/__init__.py
new file mode 100644
index 00000000..67051df5
--- /dev/null
+++ b/bindings/python-cffi/notdb/__init__.py
@@ -0,0 +1,62 @@
+"""Pythonic API to the notmuch database.
+
+Creating Objects
+================
+
+Only the :class:`Database` object is meant to be created by the user.
+All other objects should be created from this initial object.  Users
+should consider their signatures implementation details.
+
+Errors
+======
+
+All errors occuring due to errors from the underlying notmuch database
+are subclasses of the :exc:`NotmuchError`.  Due to memory management
+it is possible to try and use an object after it has been freed.  In
+this case a :exc:`ObjectDestoryedError` will be raised.
+
+Memory Management
+=================
+
+Libnotmuch uses a hierarchical memory allocator, this means all
+objects have a strict parent-child relationship and when the parent is
+freed all the children are freed as well.  This has some implications
+for these Python bindings as parent objects need to be kept alive.
+This is normally schielded entirely from the user however and the
+Python objects automatically make sure the right references are kept
+alive.  It is however the reason the :class:`BaseObject` exists as it
+defines the API all Python objects need to implement to work
+correctly.
+
+Collections and Containers
+==========================
+
+Libnotmuch exposes nearly all collections of things as iterators only.
+In these python bindings they have sometimes been exposed as
+:class:`collections.abc.Container` instances or subclasses of this
+like :class:`collections.abc.Set` or :class:`collections.abc.Mapping`
+etc.  This gives a more natural API to work with, e.g. being able to
+treat tags as sets.  However it does mean that the
+:meth:`__contains__`, :meth:`__len__` and frieds methods on these are
+usually more and essentially O(n) rather than O(1) as you might
+usually expect from Python containers.
+"""
+
+from notdb import _capi
+from notdb._base import *
+from notdb._database import *
+from notdb._errors import *
+from notdb._message import *
+from notdb._tags import *
+from notdb._thread import *
+
+
+NOTMUCH_TAG_MAX = _capi.lib.NOTMUCH_TAG_MAX
+del _capi
+
+
+# Re-home all the objects to the package.  This leaves __qualname__ intact.
+for x in locals().copy().values():
+    if hasattr(x, '__module__'):
+        x.__module__ = __name__
+del x
diff --git a/bindings/python-cffi/notdb/_base.py b/bindings/python-cffi/notdb/_base.py
new file mode 100644
index 00000000..acb64413
--- /dev/null
+++ b/bindings/python-cffi/notdb/_base.py
@@ -0,0 +1,238 @@
+import abc
+import collections.abc
+
+from notdb import _capi as capi
+from notdb import _errors as errors
+
+
+__all__ = ['NotmuchObject', 'BinString']
+
+
+class NotmuchObject(metaclass=abc.ABCMeta):
+    """Base notmuch object syntax.
+
+    This base class exists to define the memory management handling
+    required to use the notmuch library.  It is meant as an interface
+    definition rather than a base class, though you can use it as a
+    base class to ensure you don't forget part of the interface.  It
+    only concerns you if you are implementing this package itself
+    rather then using it.
+
+    libnotmuch uses a hierarchical memory allocator, where freeing the
+    memory of a parent object also frees the memory of all child
+    objects.  To make this work seamlessly in Python this package
+    keeps references to parent objects which makes them stay alive
+    correctly under normal circumstances.  When an object finally gets
+    deleted the :meth:`__del__` method will be called to free the
+    memory.
+
+    However during some peculiar situations, e.g. interpreter
+    shutdown, it is possible for the :meth:`__del__` method to have
+    been called, whele there are still references to an object.  This
+    could result in child objects asking their memeory to be freed
+    after the parent has already freed the memory, making things
+    rather unhappy as double frees are not taken lightly in C.  To
+    handle this case all objects need to follow the same protocol to
+    destroy themselves, see :meth:`destroy`.
+
+    Once an object has been destroyed trying to use it should raise
+    the :exc:`ObjectDestroyedError` exception.  For this see also the
+    convenience :class:`MemoryPointer` descriptor in this module which
+    can be used as a pointer to libnotmuch memory.
+    """
+
+    @abc.abstractmethod
+    def __init__(self, parent, *args, **kwargs):
+        """Create a new object.
+
+        Other then for the toplevel :class:`Database` object
+        constructors are only ever called by internal code and not by
+        the user.  Per convention their signature always takes the
+        parent object as first argument.  Feel free to make the rest
+        of the signature match the object's requirement.  The object
+        needs to keep a reference to the parent, so it can check the
+        parent is still alive.
+        """
+
+    @property
+    @abc.abstractmethod
+    def alive(self):
+        """Whether the object is still alive.
+
+        This indicates whether the object is still alive.  The first
+        thing this needs to check is whether the parent object is
+        still alive, if it is not then this object can not be alive
+        either.  If the parent is alive then it depends on whether the
+        memory for this object has been freed yet or not.
+        """
+
+    def __del__(self):
+        self._destroy()
+
+    @abc.abstractmethod
+    def _destroy(self):
+        """Destroy the object, freeing all memory.
+
+        This method needs to destory the object on the
+        libnotmuch-level.  It must ensure it's not been destroyed by
+        it's parent object yet before doing so.  It also must be
+        idempotent.
+        """
+
+
+class MemoryPointer:
+    """Data Descriptor to handle accessing libnotmuch pointers.
+
+    Most :class:`NotmuchObject` instances will have one or more CFFI
+    pointers to C-objects.  Once an object is destroyed this pointer
+    should no longer be used and a :exc:`ObjectDestroyedError`
+    exception should be raised on trying to access it.  This
+    descriptor simplifies implementing this, allowing the creation of
+    an attribute which can be assigned to, but when accessed when the
+    stored value is *None* it will raise the
+    :exc:`ObjectDestroyedError` exception::
+
+       class SomeOjb:
+           _ptr = MemoryPointer()
+
+           def __init__(self, ptr):
+               self._ptr = ptr
+
+           def destroy(self):
+               somehow_free(self._ptr)
+               self._ptr = None
+
+           def do_something(self):
+               return some_libnotmuch_call(self._ptr)
+    """
+
+    def __get__(self, instance, owner):
+        try:
+            val = getattr(instance, self.attr_name, None)
+        except AttributeError:
+            # We're not on 3.6+ and self.attr_name does not exist
+            self.__set_name__(instance, 'dummy')
+            val = getattr(instance, self.attr_name, None)
+        if val is None:
+            raise errors.ObjectDestroyedError()
+        return val
+
+    def __set__(self, instance, value):
+        try:
+            setattr(instance, self.attr_name, value)
+        except AttributeError:
+            # We're not on 3.6+ and self.attr_name does not exist
+            self.__set_name__(instance, 'dummy')
+            setattr(instance, self.attr_name, value)
+
+    def __set_name__(self, instance, name):
+        self.attr_name = '_memptr_{}_{:x}'.format(name, id(instance))
+
+
+class BinString(str):
+    """A str subclass with binary data.
+
+    Most data in libnotmuch should be valid ASCII or valid UTF-8.
+    However since it is a C library these are represented as
+    bytestrings intead which means on an API level we can not
+    guarantee that decoding this to UTF-8 will both succeed and be
+    lossless.  This string type converts bytes to unicode in a lossy
+    way, but also makes the raw bytes available.
+
+    This object is a normal unicode string for most intents and
+    purposes, but you can get the original bytestring back by calling
+    ``bytes()`` on it.
+    """
+
+    def __new__(cls, data, encoding='utf-8', errors='ignore'):
+        if not isinstance(data, bytes):
+            data = bytes(data, encoding=encoding)
+        strdata = str(data, encoding=encoding, errors=errors)
+        inst = super().__new__(cls, strdata)
+        inst._bindata = data
+        return inst
+
+    @classmethod
+    def from_cffi(cls, cdata):
+        """Create a new string from a CFFI cdata pointer."""
+        return cls(capi.ffi.string(cdata))
+
+    def __bytes__(self):
+        return self._bindata
+
+
+class NotmuchIter(NotmuchObject, collections.abc.Iterator):
+    """An iterator for libnotmuch iterators.
+
+    It is tempting to use a generator function instead, but this would
+    not correctly respect the :class:`NotmuchObject` memory handling
+    protocol and in some unsuspecting cornercases cause memory
+    trouble.  You probably want to sublcass this in order to wrap the
+    value returned by :meth:`__next__`.
+
+    :param parent: The parent object.
+    :type parent: NotmuchObject
+    :param iter_p: The CFFI pointer to the C iterator.
+    :type iter_p: cffi.cdata
+    :param fn_destory: The CFFI notmuch_*_destroy function.
+    :param fn_valid: The CFFI notmuch_*_valid function.
+    :param fn_get: The CFFI notmuch_*_get function.
+    :param fn_next: The CFFI notmuch_*_move_to_next function.
+    """
+    _iter_p = MemoryPointer()
+
+    def __init__(self, parent, iter_p,
+                 *, fn_destroy, fn_valid, fn_get, fn_next):
+        self._parent = parent
+        self._iter_p = iter_p
+        self._fn_destroy = fn_destroy
+        self._fn_valid = fn_valid
+        self._fn_get = fn_get
+        self._fn_next = fn_next
+
+    def __del__(self):
+        self._destroy()
+
+    @property
+    def alive(self):
+        if not self._parent.alive:
+            return False
+        try:
+            self._iter_p
+        except errors.ObjectDestroyedError:
+            return False
+        else:
+            return True
+
+    def _destroy(self):
+        if self.alive:
+            try:
+                self._fn_destroy(self._iter_p)
+            except errors.ObjectDestroyedError:
+                pass
+        self._iter_p = None
+
+    def __iter__(self):
+        """Return the iterator itself.
+
+        Note that as this is an iterator and not a container this will
+        not return a new iterator.  Thus any elements already consumed
+        will not be yielded by the :meth:`__next__` method anymore.
+        """
+        return self
+
+    def __next__(self):
+        if not self._fn_valid(self._iter_p):
+            self._destroy()
+            raise StopIteration()
+        obj_p = self._fn_get(self._iter_p)
+        self._fn_next(self._iter_p)
+        return obj_p
+
+    def __repr__(self):
+        try:
+            self._iter_p
+        except errors.ObjectDestroyedError:
+            return '<NotmuchIter (exhausted)>'
+        else:
+            return '<NotmuchIter>'
diff --git a/bindings/python-cffi/notdb/_build.py b/bindings/python-cffi/notdb/_build.py
new file mode 100644
index 00000000..97ad7808
--- /dev/null
+++ b/bindings/python-cffi/notdb/_build.py
@@ -0,0 +1,300 @@
+import cffi
+
+
+ffibuilder = cffi.FFI()
+ffibuilder.set_source(
+    'notdb._capi',
+    r"""
+    #include <stdlib.h>
+    #include <time.h>
+    #include <notmuch.h>
+
+    #if LIBNOTMUCH_MAJOR_VERSION < 5
+        #error libnotmuch version not supported by notdb
+    #endif
+    """,
+    libraries=['notmuch'],
+)
+ffibuilder.cdef(
+    r"""
+    void free(void *ptr);
+    typedef int... time_t;
+
+    #define LIBNOTMUCH_MAJOR_VERSION ...
+    #define LIBNOTMUCH_MINOR_VERSION ...
+    #define LIBNOTMUCH_MICRO_VERSION ...
+
+    #define NOTMUCH_TAG_MAX ...
+
+    typedef enum _notmuch_status {
+        NOTMUCH_STATUS_SUCCESS = 0,
+        NOTMUCH_STATUS_OUT_OF_MEMORY,
+        NOTMUCH_STATUS_READ_ONLY_DATABASE,
+        NOTMUCH_STATUS_XAPIAN_EXCEPTION,
+        NOTMUCH_STATUS_FILE_ERROR,
+        NOTMUCH_STATUS_FILE_NOT_EMAIL,
+        NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID,
+        NOTMUCH_STATUS_NULL_POINTER,
+        NOTMUCH_STATUS_TAG_TOO_LONG,
+        NOTMUCH_STATUS_UNBALANCED_FREEZE_THAW,
+        NOTMUCH_STATUS_UNBALANCED_ATOMIC,
+        NOTMUCH_STATUS_UNSUPPORTED_OPERATION,
+        NOTMUCH_STATUS_UPGRADE_REQUIRED,
+        NOTMUCH_STATUS_PATH_ERROR,
+        NOTMUCH_STATUS_ILLEGAL_ARGUMENT,
+        NOTMUCH_STATUS_LAST_STATUS
+    } notmuch_status_t;
+    typedef enum {
+        NOTMUCH_DATABASE_MODE_READ_ONLY = 0,
+        NOTMUCH_DATABASE_MODE_READ_WRITE
+    } notmuch_database_mode_t;
+    typedef int notmuch_bool_t;
+    typedef enum _notmuch_message_flag {
+        NOTMUCH_MESSAGE_FLAG_MATCH,
+        NOTMUCH_MESSAGE_FLAG_EXCLUDED,
+        NOTMUCH_MESSAGE_FLAG_GHOST,
+    } notmuch_message_flag_t;
+    typedef enum {
+        NOTMUCH_SORT_OLDEST_FIRST,
+        NOTMUCH_SORT_NEWEST_FIRST,
+        NOTMUCH_SORT_MESSAGE_ID,
+        NOTMUCH_SORT_UNSORTED
+    } notmuch_sort_t;
+    typedef enum {
+        NOTMUCH_EXCLUDE_FLAG,
+        NOTMUCH_EXCLUDE_TRUE,
+        NOTMUCH_EXCLUDE_FALSE,
+        NOTMUCH_EXCLUDE_ALL
+    } notmuch_exclude_t;
+
+    // These are fully opaque types for us, we only ever use pointers.
+    typedef struct _notmuch_database notmuch_database_t;
+    typedef struct _notmuch_query notmuch_query_t;
+    typedef struct _notmuch_threads notmuch_threads_t;
+    typedef struct _notmuch_thread notmuch_thread_t;
+    typedef struct _notmuch_messages notmuch_messages_t;
+    typedef struct _notmuch_message notmuch_message_t;
+    typedef struct _notmuch_tags notmuch_tags_t;
+    typedef struct _notmuch_string_map_iterator notmuch_message_properties_t;
+    typedef struct _notmuch_directory notmuch_directory_t;
+    typedef struct _notmuch_filenames notmuch_filenames_t;
+    typedef struct _notmuch_config_list notmuch_config_list_t;
+
+    const char *
+    notmuch_status_to_string (notmuch_status_t status);
+
+    notmuch_status_t
+    notmuch_database_create_verbose (const char *path,
+                                     notmuch_database_t **database,
+                                     char **error_message);
+    notmuch_status_t
+    notmuch_database_create (const char *path, notmuch_database_t **database);
+    notmuch_status_t
+    notmuch_database_open_verbose (const char *path,
+                                   notmuch_database_mode_t mode,
+                                   notmuch_database_t **database,
+                                   char **error_message);
+    notmuch_status_t
+    notmuch_database_open (const char *path,
+                           notmuch_database_mode_t mode,
+                           notmuch_database_t **database);
+    notmuch_status_t
+    notmuch_database_close (notmuch_database_t *database);
+    notmuch_status_t
+    notmuch_database_destroy (notmuch_database_t *database);
+    const char *
+    notmuch_database_get_path (notmuch_database_t *database);
+    unsigned int
+    notmuch_database_get_version (notmuch_database_t *database);
+    notmuch_bool_t
+    notmuch_database_needs_upgrade (notmuch_database_t *database);
+    notmuch_status_t
+    notmuch_database_begin_atomic (notmuch_database_t *notmuch);
+    notmuch_status_t
+    notmuch_database_end_atomic (notmuch_database_t *notmuch);
+    unsigned long
+    notmuch_database_get_revision (notmuch_database_t *notmuch,
+                                   const char **uuid);
+    notmuch_status_t
+    notmuch_database_add_message (notmuch_database_t *database,
+                                  const char *filename,
+                                  notmuch_message_t **message);
+    notmuch_status_t
+    notmuch_database_remove_message (notmuch_database_t *database,
+                                     const char *filename);
+    notmuch_status_t
+    notmuch_database_find_message (notmuch_database_t *database,
+                                   const char *message_id,
+                                   notmuch_message_t **message);
+    notmuch_status_t
+    notmuch_database_find_message_by_filename (notmuch_database_t *notmuch,
+                                               const char *filename,
+                                               notmuch_message_t **message);
+    notmuch_tags_t *
+    notmuch_database_get_all_tags (notmuch_database_t *db);
+
+    notmuch_query_t *
+    notmuch_query_create (notmuch_database_t *database,
+                          const char *query_string);
+    const char *
+    notmuch_query_get_query_string (const notmuch_query_t *query);
+    notmuch_database_t *
+    notmuch_query_get_database (const notmuch_query_t *query);
+    void
+    notmuch_query_set_omit_excluded (notmuch_query_t *query,
+                                     notmuch_exclude_t omit_excluded);
+    void
+    notmuch_query_set_sort (notmuch_query_t *query, notmuch_sort_t sort);
+    notmuch_sort_t
+    notmuch_query_get_sort (const notmuch_query_t *query);
+    notmuch_status_t
+    notmuch_query_add_tag_exclude (notmuch_query_t *query, const char *tag);
+    notmuch_status_t
+    notmuch_query_search_threads (notmuch_query_t *query,
+                                  notmuch_threads_t **out);
+    notmuch_status_t
+    notmuch_query_search_messages (notmuch_query_t *query,
+                                   notmuch_messages_t **out);
+    notmuch_status_t
+    notmuch_query_count_messages (notmuch_query_t *query, unsigned int *count);
+    notmuch_status_t
+    notmuch_query_count_threads (notmuch_query_t *query, unsigned *count);
+    void
+    notmuch_query_destroy (notmuch_query_t *query);
+
+    notmuch_bool_t
+    notmuch_threads_valid (notmuch_threads_t *threads);
+    notmuch_thread_t *
+    notmuch_threads_get (notmuch_threads_t *threads);
+    void
+    notmuch_threads_move_to_next (notmuch_threads_t *threads);
+    void
+    notmuch_threads_destroy (notmuch_threads_t *threads);
+
+    const char *
+    notmuch_thread_get_thread_id (notmuch_thread_t *thread);
+    notmuch_messages_t *
+    notmuch_message_get_replies (notmuch_message_t *message);
+    int
+    notmuch_thread_get_total_messages (notmuch_thread_t *thread);
+    notmuch_messages_t *
+    notmuch_thread_get_toplevel_messages (notmuch_thread_t *thread);
+    notmuch_messages_t *
+    notmuch_thread_get_messages (notmuch_thread_t *thread);
+    int
+    notmuch_thread_get_matched_messages (notmuch_thread_t *thread);
+    const char *
+    notmuch_thread_get_authors (notmuch_thread_t *thread);
+    const char *
+    notmuch_thread_get_subject (notmuch_thread_t *thread);
+    time_t
+    notmuch_thread_get_oldest_date (notmuch_thread_t *thread);
+    time_t
+    notmuch_thread_get_newest_date (notmuch_thread_t *thread);
+    notmuch_tags_t *
+    notmuch_thread_get_tags (notmuch_thread_t *thread);
+    void
+    notmuch_thread_destroy (notmuch_thread_t *thread);
+
+    notmuch_bool_t
+    notmuch_messages_valid (notmuch_messages_t *messages);
+    notmuch_message_t *
+    notmuch_messages_get (notmuch_messages_t *messages);
+    void
+    notmuch_messages_move_to_next (notmuch_messages_t *messages);
+    void
+    notmuch_messages_destroy (notmuch_messages_t *messages);
+    notmuch_tags_t *
+    notmuch_messages_collect_tags (notmuch_messages_t *messages);
+
+    const char *
+    notmuch_message_get_message_id (notmuch_message_t *message);
+    const char *
+    notmuch_message_get_thread_id (notmuch_message_t *message);
+    const char *
+    notmuch_message_get_filename (notmuch_message_t *message);
+    notmuch_filenames_t *
+    notmuch_message_get_filenames (notmuch_message_t *message);
+    notmuch_bool_t
+    notmuch_message_get_flag (notmuch_message_t *message,
+                              notmuch_message_flag_t flag);
+    void
+    notmuch_message_set_flag (notmuch_message_t *message,
+                              notmuch_message_flag_t flag,
+                              notmuch_bool_t value);
+    time_t
+    notmuch_message_get_date  (notmuch_message_t *message);
+    const char *
+    notmuch_message_get_header (notmuch_message_t *message,
+                                const char *header);
+    notmuch_tags_t *
+    notmuch_message_get_tags (notmuch_message_t *message);
+    notmuch_status_t
+    notmuch_message_add_tag (notmuch_message_t *message, const char *tag);
+    notmuch_status_t
+    notmuch_message_remove_tag (notmuch_message_t *message, const char *tag);
+    notmuch_status_t
+    notmuch_message_remove_all_tags (notmuch_message_t *message);
+    notmuch_status_t
+    notmuch_message_maildir_flags_to_tags (notmuch_message_t *message);
+    notmuch_status_t
+    notmuch_message_tags_to_maildir_flags (notmuch_message_t *message);
+    notmuch_status_t
+    notmuch_message_freeze (notmuch_message_t *message);
+    notmuch_status_t
+    notmuch_message_thaw (notmuch_message_t *message);
+    notmuch_status_t
+    notmuch_message_get_property (notmuch_message_t *message,
+                                  const char *key, const char **value);
+    notmuch_status_t
+    notmuch_message_add_property (notmuch_message_t *message,
+                                  const char *key, const char *value);
+    notmuch_status_t
+    notmuch_message_remove_property (notmuch_message_t *message,
+                                     const char *key, const char *value);
+    notmuch_status_t
+    notmuch_message_remove_all_properties (notmuch_message_t *message,
+                                           const char *key);
+    notmuch_message_properties_t *
+    notmuch_message_get_properties (notmuch_message_t *message,
+                                    const char *key, notmuch_bool_t exact);
+    notmuch_bool_t
+    notmuch_message_properties_valid (notmuch_message_properties_t
+                                          *properties);
+    void
+    notmuch_message_properties_move_to_next (notmuch_message_properties_t
+                                                 *properties);
+    const char *
+    notmuch_message_properties_key (notmuch_message_properties_t *properties);
+    const char *
+    notmuch_message_properties_value (notmuch_message_properties_t
+                                          *properties);
+    void
+    notmuch_message_properties_destroy (notmuch_message_properties_t
+                                            *properties);
+    void
+    notmuch_message_destroy (notmuch_message_t *message);
+
+    notmuch_bool_t
+    notmuch_tags_valid (notmuch_tags_t *tags);
+    const char *
+    notmuch_tags_get (notmuch_tags_t *tags);
+    void
+    notmuch_tags_move_to_next (notmuch_tags_t *tags);
+    void
+    notmuch_tags_destroy (notmuch_tags_t *tags);
+
+    notmuch_bool_t
+    notmuch_filenames_valid (notmuch_filenames_t *filenames);
+    const char *
+    notmuch_filenames_get (notmuch_filenames_t *filenames);
+    void
+    notmuch_filenames_move_to_next (notmuch_filenames_t *filenames);
+    void
+    notmuch_filenames_destroy (notmuch_filenames_t *filenames);
+    """
+)
+
+
+if __name__ == '__main__':
+    ffibuilder.compile(verbose=True)
diff --git a/bindings/python-cffi/notdb/_database.py b/bindings/python-cffi/notdb/_database.py
new file mode 100644
index 00000000..d414082a
--- /dev/null
+++ b/bindings/python-cffi/notdb/_database.py
@@ -0,0 +1,705 @@
+import collections
+import configparser
+import enum
+import functools
+import os
+import pathlib
+import weakref
+
+import notdb._base as base
+import notdb._capi as capi
+import notdb._errors as errors
+import notdb._message as message
+import notdb._query as querymod
+import notdb._tags as tags
+
+
+__all__ = ['Database', 'AtomicContext', 'DbRevision']
+
+
+def _config_pathname():
+    """Return the path of the configuration file.
+
+    :rtype: pathlib.Path
+    """
+    cfgfname = os.getenv('NOTMUCH_CONFIG', '~/.notmuch-config')
+    return pathlib.Path(os.path.expanduser(cfgfname))
+
+
+class Mode(enum.Enum):
+    READ_ONLY = capi.lib.NOTMUCH_DATABASE_MODE_READ_ONLY
+    READ_WRITE = capi.lib.NOTMUCH_DATABASE_MODE_READ_WRITE
+
+
+class QuerySortOrder(enum.Enum):
+    OLDEST_FIRST = capi.lib.NOTMUCH_SORT_OLDEST_FIRST
+    NEWEST_FIRST = capi.lib.NOTMUCH_SORT_NEWEST_FIRST
+    MESSAGE_ID = capi.lib.NOTMUCH_SORT_MESSAGE_ID
+    UNSORTED = capi.lib.NOTMUCH_SORT_UNSORTED
+
+
+class QueryExclude(enum.Enum):
+    TRUE = capi.lib.NOTMUCH_EXCLUDE_TRUE
+    FLAG = capi.lib.NOTMUCH_EXCLUDE_FLAG
+    FALSE = capi.lib.NOTMUCH_EXCLUDE_FALSE
+    ALL = capi.lib.NOTMUCH_EXCLUDE_ALL
+
+
+class Database(base.NotmuchObject):
+    """Toplevel access to notmuch.
+
+    A :class:`Database` can be opened read-only or read-write.
+    Modifications are not atomic by default, use :meth:`begin_atomic`
+    for atomic updates.  If the underlying database has been modified
+    outside of this class a :exc:`XapianError` will be raised and the
+    instance must be closed and a new one created.
+
+    You can use an instance of this class as a context-manager.
+
+    :cvar MODE: The mode a database can be opened with, an enumeration
+       of ``READ_ONLY`` and ``READ_WRITE``
+    :cvar SORT: The sort order for search results, ``OLDEST_FIRST``,
+       ``NEWEST_FIRST``, ``MESSAGE_ID`` or ``UNSORTED``.
+    :cvar EXCLUDE: Which messages to exclude from queries, ``TRUE``,
+       ``FLAG``, ``FALSE`` or ``ALL``.  See the query documentation
+       for details.
+    :cvar AddedMessage: A namedtuple ``(msg, dup)`` used by
+       :meth:`add` as return value.
+    :cvar STR_MODE_MAP: A map mapping strings to :attr:`MODE` items.
+       This is used to implement the ``ro`` and ``rw`` string
+       variants.
+
+    :ivar closed: Boolean indicating if the database is closed or
+       still open.
+
+    :param path: The directory of where the database is stored.  If
+       ``None`` the location will be read from the user's
+       configuration file, respecting the ``NOTMUCH_CONFIG``
+       environment variable if set.
+    :type path: str, bytes, os.PathLike or pathlib.Path
+    :param mode: The mode to open the database in.  One of
+       :attr:`MODE.READ_ONLY` OR :attr:`MODE.READ_WRITE`.  For
+       convenience you can also use the strings ``ro`` for
+       :attr:`MODE.READ_ONLY` and ``rw`` for :attr:`MODE.READ_WRITE`.
+    :type mode: :attr:`MODE` or str.
+
+    :raises KeyError: if an unknown mode string is used.
+    :raises OSError: or subclasses if the configuration file can not
+       be opened.
+    :raises configparser.Error: or subclasses if the configuration
+       file can not be parsed.
+    :raises NotmuchError: or subclasses for other failures.
+    """
+
+    MODE = Mode
+    SORT = QuerySortOrder
+    EXCLUDE = QueryExclude
+    AddedMessage = collections.namedtuple('AddedMessage', ['msg', 'dup'])
+    _db_p = base.MemoryPointer()
+    STR_MODE_MAP = {
+        'ro': MODE.READ_ONLY,
+        'rw': MODE.READ_WRITE,
+    }
+
+    def __init__(self, path=None, mode=MODE.READ_ONLY):
+        if isinstance(mode, str):
+            mode = self.STR_MODE_MAP[mode]
+        self.mode = mode
+        if path is None:
+            path = self.default_path()
+        if not hasattr(os, 'PathLike') and isinstance(path, pathlib.Path):
+            path = bytes(path)
+        db_pp = capi.ffi.new('notmuch_database_t **')
+        cmsg = capi.ffi.new('char**')
+        ret = capi.lib.notmuch_database_open_verbose(os.fsencode(path),
+                                                     mode.value, db_pp, cmsg)
+        if cmsg[0]:
+            msg = capi.ffi.string(cmsg[0]).decode(errors='replace')
+            capi.lib.free(cmsg[0])
+        else:
+            msg = None
+        if ret != capi.lib.NOTMUCH_STATUS_SUCCESS:
+            raise errors.NotmuchError(ret, msg)
+        self._db_p = db_pp[0]
+        self.closed = False
+
+    @classmethod
+    def create(cls, path=None):
+        """Create and open database in READ_WRITE mode.
+
+        This is creates a new notmuch database and returns an opened
+        instance in :attr:`MODE.READ_WRITE` mode.
+
+        :param path: The directory of where the database is stored.  If
+           ``None`` the location will be read from the user's
+           configuration file, respecting the ``NOTMUCH_CONFIG``
+           environment variable if set.
+        :type path: str, bytes or os.PathLike
+
+        :raises OSError: or subclasses if the configuration file can not
+           be opened.
+        :raises configparser.Error: or subclasses if the configuration
+           file can not be parsed.
+        :raises NotmuchError: if the config file does not have the
+           database.path setting.
+        :raises FileError: if the database already exists.
+
+        :returns: The newly created instance.
+        """
+        if path is None:
+            path = cls.default_path()
+        if not hasattr(os, 'PathLike') and isinstance(path, pathlib.Path):
+            path = bytes(path)
+        db_pp = capi.ffi.new('notmuch_database_t **')
+        cmsg = capi.ffi.new('char**')
+        ret = capi.lib.notmuch_database_create_verbose(os.fsencode(path),
+                                                       db_pp, cmsg)
+        if cmsg[0]:
+            msg = capi.ffi.string(cmsg[0]).decode(errors='replace')
+            capi.lib.free(cmsg[0])
+        else:
+            msg = None
+        if ret != capi.lib.NOTMUCH_STATUS_SUCCESS:
+            raise errors.NotmuchError(ret, msg)
+
+        # Now close the db and let __init__ open it.  Inefficient but
+        # creating is not a hot loop while this allows us to have a
+        # clean API.
+        ret = capi.lib.notmuch_database_destroy(db_pp[0])
+        if ret != capi.lib.NOTMUCH_STATUS_SUCCESS:
+            raise errors.NotmuchError(ret)
+        return cls(path, cls.MODE.READ_WRITE)
+
+    @staticmethod
+    def default_path(cfg_path=None):
+        """Return the path of the user's default database.
+
+        This reads the user's configuration file and returns the
+        default path of the database.
+
+        :param cfg_path: The pathname of the notmuch configuration file.
+           If not specified tries to use the pathname provided in the
+           :env:`NOTMUCH_CONFIG` environment variable and falls back
+           to :file:`~/.notmuch-config.
+        :type cfg_path: str, bytes, os.PathLike or pathlib.Path.
+
+        :returns: The path of the database, which does not necessarily
+           exists.
+        :rtype: pathlib.Path
+        :raises OSError: or subclasses if the configuration file can not
+           be opened.
+        :raises configparser.Error: or subclasses if the configuration
+           file can not be parsed.
+        :raises NotmuchError if the config file does not have the
+           database.path setting.
+        """
+        if not cfg_path:
+            cfg_path = _config_pathname()
+        if not hasattr(os, 'PathLike') and isinstance(cfg_path, pathlib.Path):
+            cfg_path = bytes(cfg_path)
+        parser = configparser.ConfigParser()
+        with open(cfg_path) as fp:
+            parser.read_file(fp)
+        try:
+            return pathlib.Path(parser.get('database', 'path'))
+        except configparser.Error:
+            raise errors.NotmuchError(
+                'No database.path setting in {}'.format(cfg_path))
+
+    def __del__(self):
+        self._destroy()
+
+    @property
+    def alive(self):
+        try:
+            self._db_p
+        except errors.ObjectDestroyedError:
+            return False
+        else:
+            return True
+
+    def _destroy(self):
+        try:
+            ret = capi.lib.notmuch_database_destroy(self._db_p)
+        except errors.ObjectDestroyedError:
+            ret = capi.lib.NOTMUCH_STATUS_SUCCESS
+        else:
+            self._db_p = None
+        if ret != capi.lib.NOTMUCH_STATUS_SUCCESS:
+            raise errors.NotmuchError(ret)
+
+    def close(self):
+        """Close the notmuch database.
+
+        Once closed most operations will fail.  This can still be
+        useful however to explicitly close a database which is opened
+        read-write as this would otherwise stop other processes from
+        reading the database while it is open.
+
+        :raises ObjectDestroyedError: if used after destroyed.
+        """
+        ret = capi.lib.notmuch_database_close(self._db_p)
+        if ret != capi.lib.NOTMUCH_STATUS_SUCCESS:
+            raise errors.NotmuchError(ret)
+        self.closed = True
+
+    def __enter__(self):
+        return self
+
+    def __exit__(self, exc_type, exc_value, traceback):
+        self.close()
+
+    @property
+    def path(self):
+        """The pathname of the notmuch database.
+
+        This is returned as a :class:`pathlib.Path` instance.
+
+        :raises ObjectDestroyedError: if used after destoryed.
+        """
+        try:
+            return self._cache_path
+        except AttributeError:
+            ret = capi.lib.notmuch_database_get_path(self._db_p)
+            self._cache_path = pathlib.Path(os.fsdecode(capi.ffi.string(ret)))
+            return self._cache_path
+
+    @property
+    def version(self):
+        """The database format version.
+
+        This is a positive integer.
+
+        :raises ObjectDestroyedError: if used after destoryed.
+        """
+        try:
+            return self._cache_version
+        except AttributeError:
+            ret = capi.lib.notmuch_database_get_version(self._db_p)
+            self._cache_version = ret
+            return ret
+
+    @property
+    def needs_upgrade(self):
+        """Whether the database should be upgraded.
+
+        If *True* the database can be upgraded using :meth:`upgrade`.
+        Not doing so may result in some operations raising
+        :exc:`UpgradeRequiredError`.
+
+        A read-only database will never be upgradable.
+
+        :raises ObjectDestroyedError: if used after destoryed.
+        """
+        ret = capi.lib.notmuch_database_needs_upgrade(self._db_p)
+        return bool(ret)
+
+    def upgrade(self, progress_cb=None):
+        """Upgrade the database to the latest version.
+
+        Upgrade the database, optionally with a progress callback
+        which should be a callable which will be called with a
+        floating point number in the range of [0.0 .. 1.0].
+        """
+        raise NotImplementedError
+
+    def atomic(self):
+        """Return a context manager to perform atomic operations.
+
+        The returned context manager can be used to perform atomic
+        operations on the database.
+
+        .. note:: Unlinke a traditional RDBMS transaction this does
+           not imply durability, it only ensures the changes are
+           performed atomically.
+
+        :raises ObjectDestroyedError: if used after destoryed.
+        """
+        ctx = AtomicContext(self, '_db_p')
+        return ctx
+
+    def revision(self):
+        """The currently committed revision in the database.
+
+        Returned as a ``(revision, uuid)`` namedtuple.
+
+        :raises ObjectDestroyedError: if used after destoryed.
+        """
+        raw_uuid = capi.ffi.new('char**')
+        rev = capi.lib.notmuch_database_get_revision(self._db_p, raw_uuid)
+        return DbRevision(rev, capi.ffi.string(raw_uuid[0]))
+
+    def get_directory(self, path):
+        raise NotImplementedError
+
+    def add(self, filename, *, sync_flags=False):
+        """Add a message to the database.
+
+        Add a new message to the notmuch database.  The message is
+        referred to by the pathname of the maildir file.  If the
+        message ID of the new message already exists in the database,
+        this adds ``pathname`` to the list of list of files for the
+        existing message.
+
+        :param filename: The path of the file containing the message.
+        :type filename: str, bytes, os.PathLike or pathlib.Path.
+        :param sync_flags: Whether to sync the known maildir flags to
+           notmuch tags.  See :meth:`Message.flags_to_tags` for
+           details.
+
+        :returns: A tuple where the first item is the newly inserted
+           messages as a :class:`Message` instance, and the second
+           item is a boolean indicating if the message inserted was a
+           duplicate.  This is the namedtuple ``AddedMessage(msg,
+           dup)``.
+        :rtype: Database.AddedMessage
+
+        If an exception is raised, no message was added.
+
+        :raises XapianError: A Xapian exception occurred.
+        :raises FileError: The file referred to by ``pathname`` could
+           not be opened.
+        :raises FileNotEmailError: The file referreed to by
+           ``pathname`` is not recognised as an email message.
+        :raises ReadOnlyDatabaseError: The database is opened in
+           READ_ONLY mode.
+        :raises UpgradeRequiredError: The database must be upgraded
+           first.
+        :raises ObjectDestroyedError: if used after destoryed.
+        """
+        if not hasattr(os, 'PathLike') and isinstance(filename, pathlib.Path):
+            filename = bytes(filename)
+        msg_pp = capi.ffi.new('notmuch_message_t **')
+        ret = capi.lib.notmuch_database_add_message(self._db_p,
+                                                    os.fsencode(filename),
+                                                    msg_pp)
+        ok = [capi.lib.NOTMUCH_STATUS_SUCCESS,
+              capi.lib.NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID]
+        if ret not in ok:
+            raise errors.NotmuchError(ret)
+        msg = message.Message(self, msg_pp[0], db=self)
+        if sync_flags:
+            msg.tags.from_maildir_flags()
+        return self.AddedMessage(
+            msg, ret == capi.lib.NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID)
+
+    def remove(self, filename):
+        """Remove a message from the notmuch database.
+
+        Removing a message which is not in the database is just a
+        silent nop-operation.
+
+        :param filename: The pathname of the file containing the
+           message to be removed.
+        :type filename: str, bytes, os.PathLike or pathlib.Path.
+
+        :returns: True if the message is still in the database.  This
+           can happen when multiple files contain the same message ID.
+           The true/false distinction is fairly arbitrary, but think
+           of it as ``dup = db.remove_message(name); if dup: ...``.
+        :rtype: bool
+
+        :raises XapianError: A Xapian exception ocurred.
+        :raises ReadOnlyDatabaseError: The database is opened in
+           READ_ONLY mode.
+        :raises UpgradeRequiredError: The database must be upgraded
+           first.
+        :raises ObjectDestroyedError: if used after destoryed.
+        """
+        if not hasattr(os, 'PathLike') and isinstance(filename, pathlib.Path):
+            filename = bytes(filename)
+        ret = capi.lib.notmuch_database_remove_message(self._db_p,
+                                                       os.fsencode(filename))
+        ok = [capi.lib.NOTMUCH_STATUS_SUCCESS,
+              capi.lib.NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID]
+        if ret not in ok:
+            raise errors.NotmuchError(ret)
+        if ret == capi.lib.NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID:
+            return True
+        else:
+            return False
+
+    def find(self, msgid):
+        """Return the message matching the given message ID.
+
+        If a message with the given message ID is found a
+        :class:`Message` instance is returned.  Otherwise a
+        :exc:`LookupError` is raised.
+
+        :param msgid: The message ID to look for.
+        :type msgid: str
+
+        :returns: The message instance.
+        :rtype: Message
+
+        :raises LookupError: If no message was found.
+        :raises OutOfMemoryError: When there is no memory to allocate
+           the message instance.
+        :raises XapianError: A Xapian exception ocurred.
+        :raises ObjectDestroyedError: if used after destoryed.
+        """
+        msg_pp = capi.ffi.new('notmuch_message_t **')
+        ret = capi.lib.notmuch_database_find_message(self._db_p,
+                                                     msgid.encode(), msg_pp)
+        if ret != capi.lib.NOTMUCH_STATUS_SUCCESS:
+            raise errors.NotmuchError(ret)
+        msg_p = msg_pp[0]
+        if msg_p == capi.ffi.NULL:
+            raise LookupError
+        msg = message.Message(self, msg_p, db=self)
+        return msg
+
+    def get(self, filename):
+        """Return the :class:`Message` given a pathname.
+
+        If a message with the given pathname exists in the database
+        return the :class:`Message` instance for the message.
+        Otherwise raise a :exc:`LookupError` exception.
+
+        :param filename: The pathname of the message.
+        :type filename: str, bytes, os.PathLike or pathlib.Path
+
+        :returns: The message instance.
+        :rtype: Message
+
+        :raises LookupError: If no message was found.  This is also
+           a subclass of :exc:`KeyError`.
+        :raises OutOfMemoryError: When there is no memory to allocate
+           the message instance.
+        :raises XapianError: A Xapian exception ocurred.
+        :raises ObjectDestroyedError: if used after destoryed.
+        """
+        if not hasattr(os, 'PathLike') and isinstance(filename, pathlib.Path):
+            filename = bytes(filename)
+        msg_pp = capi.ffi.new('notmuch_message_t **')
+        ret = capi.lib.notmuch_database_find_message_by_filename(
+            self._db_p, os.fsencode(filename), msg_pp)
+        if ret != capi.lib.NOTMUCH_STATUS_SUCCESS:
+            raise errors.NotmuchError(ret)
+        msg_p = msg_pp[0]
+        if msg_p == capi.ffi.NULL:
+            raise LookupError
+        msg = message.Message(self, msg_p, db=self)
+        return msg
+
+    @property
+    def tags(self):
+        """Return an immutable set with all tags used in this database.
+
+        This returns an immutable set-like object implementing the
+        collections.abc.Set Abstract Base Class.  Due to the
+        underlying libnotmuch implementation some operations have
+        different performance characteristics then plain set objects.
+        Mainly any lookup operation is O(n) rather then O(1).
+
+        Normal usage treats tags as UTF-8 encoded unicode strings so
+        they are exposed to Python as normal unicode string objects.
+        If you need to handle tags stored in libnotmuch which are not
+        valid unicode do check the :class:`ImmutableTagSet` docs for
+        how to handle this.
+
+        :rtype: ImmutableTagSet
+
+        :raises ObjectDestroyedError: if used after destoryed.
+        """
+        try:
+            ref = self._cached_tagset
+        except AttributeError:
+            tagset = None
+        else:
+            tagset = ref()
+        if tagset is None:
+            tagset = tags.ImmutableTagSet(
+                self, '_db_p', capi.lib.notmuch_database_get_all_tags)
+            self._cached_tagset = weakref.ref(tagset)
+        return tagset
+
+    def _create_query(self, query, *,
+                      omit_excluded=EXCLUDE.TRUE,
+                      sort=SORT.UNSORTED,  # Check this default
+                      exclude_tags=None):
+        """Create an internal query object.
+
+        :raises OutOfMemoryError: if no memory is available to
+           allocate the query.
+        """
+        if isinstance(query, str):
+            query = query.encode('utf-8')
+        query_p = capi.lib.notmuch_query_create(self._db_p, query)
+        if query_p == capi.ffi.NULL:
+            raise errors.OutOfMemoryError()
+        capi.lib.notmuch_query_set_omit_excluded(query_p, omit_excluded.value)
+        capi.lib.notmuch_query_set_sort(query_p, sort.value)
+        if exclude_tags is not None:
+            for tag in exclude_tags:
+                if isinstance(tag, str):
+                    tag = str.encode('utf-8')
+                capi.lib.notmuch_query_add_tag_exclude(query_p, tag)
+        return querymod.Query(self, query_p)
+
+    def messages(self, query, *,
+                 omit_excluded=EXCLUDE.TRUE,
+                 sort=SORT.UNSORTED,  # Check this default
+                 exclude_tags=None):
+        """Search the database for messages.
+
+        :returns: An iterator over the messages found.
+        :rtype: MessageIter
+
+        :raises OutOfMemoryError: if no memory is available to
+           allocate the query.
+        :raises ObjectDestroyedError: if used after destoryed.
+        """
+        query = self._create_query(query,
+                                   omit_excluded=omit_excluded,
+                                   sort=sort,
+                                   exclude_tags=exclude_tags)
+        return query.messages()
+
+    def count_messages(self, query, *,
+                       omit_excluded=EXCLUDE.TRUE,
+                       sort=SORT.UNSORTED,  # Check this default
+                       exclude_tags=None):
+        """Search the database for messages.
+
+        :returns: An iterator over the messages found.
+        :rtype: MessageIter
+
+        :raises ObjectDestroyedError: if used after destoryed.
+        """
+        query = self._create_query(query,
+                                   omit_excluded=omit_excluded,
+                                   sort=sort,
+                                   exclude_tags=exclude_tags)
+        return query.count_messages()
+
+    def threads(self,  query, *,
+                omit_excluded=EXCLUDE.TRUE,
+                sort=SORT.UNSORTED,  # Check this default
+                exclude_tags=None):
+        query = self._create_query(query,
+                                   omit_excluded=omit_excluded,
+                                   sort=sort,
+                                   exclude_tags=exclude_tags)
+        return query.threads()
+
+    def count_threads(self, query, *,
+                      omit_excluded=EXCLUDE.TRUE,
+                      sort=SORT.UNSORTED,  # Check this default
+                      exclude_tags=None):
+        query = self._create_query(query,
+                                   omit_excluded=omit_excluded,
+                                   sort=sort,
+                                   exclude_tags=exclude_tags)
+        return query.count_threads()
+
+    def status_string(self):
+        raise NotImplementedError
+
+    def __repr__(self):
+        return 'Database(path={self.path}, mode={self.mode})'.format(self=self)
+
+
+class AtomicContext:
+    """Context manager for atomic support.
+
+    This supports the notmuch_database_begin_atomic and
+    notmuch_database_end_atomic API calls.  The object can not be
+    directly instantiated by the user, only via ``Database.atomic``.
+    It does keep a reference to the :class:`Database` instance to keep
+    the C memory alive.
+
+    :raises XapianError: When this is raised at enter time the atomic
+       section is not active.  When it is raised at exit time the
+       atomic section is still active and you may need to try using
+       :meth:`force_end`.
+    :raises ObjectDestroyedError: if used after destoryed.
+    """
+
+    def __init__(self, db, ptr_name):
+        self._db = db
+        self._ptr = lambda: getattr(db, ptr_name)
+
+    def __del__(self):
+        self._destroy()
+
+    @property
+    def alive(self):
+        return self.parent.alive
+
+    def _destroy(self):
+        pass
+
+    def __enter__(self):
+        ret = capi.lib.notmuch_database_begin_atomic(self._ptr())
+        if ret != capi.lib.NOTMUCH_STATUS_SUCCESS:
+            raise errors.NotmuchError(ret)
+        return self
+
+    def __exit__(self, exc_type, exc_value, traceback):
+        ret = capi.lib.notmuch_database_end_atomic(self._ptr())
+        if ret != capi.lib.NOTMUCH_STATUS_SUCCESS:
+            raise errors.NotmuchError(ret)
+
+    def force_end(self):
+        """Force ending the atomic section.
+
+        This can only be called once __exit__ has been called.  It
+        will attept to close the atomic section (again).  This is
+        useful if the original exit raised an exception and the atomic
+        section is still open.  But things are pretty ugly by now.
+
+        :raises XapianError: If exiting fails, the atomic section is
+           not ended.
+        :raises UnbalancedAtomicError: If the database was currently
+           not in an atomic section.
+        :raises ObjectDestroyedError: if used after destoryed.
+        """
+        ret = capi.lib.notmuch_database_end_atomic(self._ptr())
+        if ret != capi.lib.NOTMUCH_STATUS_SUCCESS:
+            raise errors.NotmuchError(ret)
+
+
+@functools.total_ordering
+class DbRevision:
+    """A database revision.
+
+    The database revision number increases monotonically with each
+    commit to the database.  Which means user-visible changes can be
+    ordered.  This object is sortable with other revisions.  It
+    carries the UUID of the database to ensure it is only ever
+    compared with revisions from the same database.
+    """
+
+    def __init__(self, rev, uuid):
+        self._rev = rev
+        self._uuid = uuid
+
+    @property
+    def rev(self):
+        """The revision number, a positive integer."""
+        return self._rev
+
+    @property
+    def uuid(self):
+        """The UUID of the database, consider this opaque."""
+        return self._uuid
+
+    def __eq__(self, other):
+        if isinstance(other, self.__class__):
+            if self.uuid != other.uuid:
+                return False
+            return self.rev == other.rev
+        else:
+            return NotImplemented
+
+    def __lt__(self, other):
+        if self.__class__ is other.__class__:
+            if self.uuid != other.uuid:
+                return False
+            return self.rev < other.rev
+        else:
+            return NotImplemented
+
+    def __repr__(self):
+        return 'DbRevision(rev={self.rev}, uuid={self.uuid})'.format(self=self)
diff --git a/bindings/python-cffi/notdb/_errors.py b/bindings/python-cffi/notdb/_errors.py
new file mode 100644
index 00000000..924e722f
--- /dev/null
+++ b/bindings/python-cffi/notdb/_errors.py
@@ -0,0 +1,112 @@
+from notdb import _capi as capi
+
+
+class NotmuchError(Exception):
+    """Base exception for errors originating from the notmuch library.
+
+    Usually this will have two attributes:
+
+    :status: This is a numeric status code corresponding to the error
+       code in the notmuch library.  This is normally fairly
+       meaningless, it can also often be ``None``.  This exists mostly
+       to easily create new errors from notmuch status codes and
+       should not normally be used by users.
+
+    :message: A user-facing message for the error.  This can
+       occasionally also be ``None``.  Usually you'll want to call
+       ``str()`` on the error object instead to get a sensible
+       message.
+    """
+
+    @classmethod
+    def exc_type(cls, status):
+        """Return correct exception type for notmuch status."""
+        types = {
+            capi.lib.NOTMUCH_STATUS_OUT_OF_MEMORY:
+                OutOfMemoryError,
+            capi.lib.NOTMUCH_STATUS_READ_ONLY_DATABASE:
+                ReadOnlyDatabaseError,
+            capi.lib.NOTMUCH_STATUS_XAPIAN_EXCEPTION:
+                XapianError,
+            capi.lib.NOTMUCH_STATUS_FILE_ERROR:
+                FileError,
+            capi.lib.NOTMUCH_STATUS_FILE_NOT_EMAIL:
+                FileNotEmailError,
+            capi.lib.NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID:
+                DuplicateMessageIdError,
+            capi.lib.NOTMUCH_STATUS_NULL_POINTER:
+                NullPointerError,
+            capi.lib.NOTMUCH_STATUS_TAG_TOO_LONG:
+                TagTooLongError,
+            capi.lib.NOTMUCH_STATUS_UNBALANCED_FREEZE_THAW:
+                UnbalancedFreezeThawError,
+            capi.lib.NOTMUCH_STATUS_UNBALANCED_ATOMIC:
+                UnbalancedAtomicError,
+            capi.lib.NOTMUCH_STATUS_UNSUPPORTED_OPERATION:
+                UnsupportedOperationError,
+            capi.lib.NOTMUCH_STATUS_UPGRADE_REQUIRED:
+                UpgradeRequiredError,
+            capi.lib.NOTMUCH_STATUS_PATH_ERROR:
+                PathError,
+            capi.lib.NOTMUCH_STATUS_ILLEGAL_ARGUMENT:
+                IllegalArgumentError,
+        }
+        return types[status]
+
+    def __new__(cls, *args, **kwargs):
+        """Return the correct subclass based on status."""
+        # This is simplistic, but the actual __init__ will fail if the
+        # signature is wrong anyway.
+        if args:
+            status = args[0]
+        else:
+            status = kwargs.get('status', None)
+        if status and cls == NotmuchError:
+            exc = cls.exc_type(status)
+            return exc.__new__(exc, *args, **kwargs)
+        else:
+            return super().__new__(cls)
+
+    def __init__(self, status=None, message=None):
+        self.status = status
+        self.message = message
+
+    def __str__(self):
+        if self.message:
+            return self.message
+        elif self.status:
+            return capi.lib.notmuch_status_to_string(self.status)
+        else:
+            return 'Unknown error'
+
+
+class OutOfMemoryError(NotmuchError): pass
+class ReadOnlyDatabaseError(NotmuchError): pass
+class XapianError(NotmuchError): pass
+class FileError(NotmuchError): pass
+class FileNotEmailError(NotmuchError): pass
+class DuplicateMessageIdError(NotmuchError): pass
+class NullPointerError(NotmuchError): pass
+class TagTooLongError(NotmuchError): pass
+class UnbalancedFreezeThawError(NotmuchError): pass
+class UnbalancedAtomicError(NotmuchError): pass
+class UnsupportedOperationError(NotmuchError): pass
+class UpgradeRequiredError(NotmuchError): pass
+class PathError(NotmuchError): pass
+class IllegalArgumentError(NotmuchError): pass
+
+
+class ObjectDestroyedError(NotmuchError):
+    """The object has already been destoryed and it's memory freed.
+
+    This occurs when :meth:`destroy` has been called on the object but
+    you still happen to have access to the object.  This should not
+    normally occur since you should never call :meth:`destroy` by
+    hand.
+    """
+
+    def __str__(self):
+        if self.message:
+            return self.message
+        else:
+            return 'Memory already freed'
diff --git a/bindings/python-cffi/notdb/_message.py b/bindings/python-cffi/notdb/_message.py
new file mode 100644
index 00000000..9b2b037f
--- /dev/null
+++ b/bindings/python-cffi/notdb/_message.py
@@ -0,0 +1,691 @@
+import collections
+import contextlib
+import os
+import pathlib
+import weakref
+
+import notdb._base as base
+import notdb._capi as capi
+import notdb._errors as errors
+import notdb._tags as tags
+
+
+__all__ = ['Message']
+
+
+class Message(base.NotmuchObject):
+    """An email message stored in the notmuch database.
+
+    This should not be directly created, instead it will be returned
+    by calling methods on :class:`Database`.  A message keeps a
+    reference to the database object since the database object can not
+    be released while the message is in use.
+
+    Note that this represents a message in the notmuch database.  For
+    full email functionality you may want to use the :mod:`email`
+    package from Python's standard library.  You could e.g. create
+    this as such::
+
+       notmuch_msg = db.get_message(msgid)  # or from a query
+       parser = email.parser.BytesParser(policy=email.policy.default)
+       with notmuch_msg.path.open('rb) as fp:
+           email_msg = parser.parse(fp)
+
+    Most commonly the functionality provided by notmuch is sufficient
+    to read email however.
+
+    Messages are considered equal when they have the same message ID.
+    This is how libnotmuch treats messages as well, the
+    :meth:`pathnames` function returns multiple results for
+    duplicates.
+
+    :param parent: The parent object.  This is probably one off a
+       :class:`Database`, :class:`Thread` or :class:`Query`.
+    :type parent: NotmuchObject
+    :param db: The database instance this message is associated with.
+       This could be the same as the parent.
+    :type db: Database
+    :param msg_p: The C pointer to the ``notmuch_message_t``.
+    :type msg_p: <cdata>
+
+    :param dup: Whether the message was a duplicate on insertion.
+
+    :type dup: None or bool
+    """
+    _msg_p = base.MemoryPointer()
+
+    def __init__(self, parent, msg_p, *, db):
+        self._parent = parent
+        self._msg_p = msg_p
+        self._db = db
+
+    @property
+    def alive(self):
+        if not self._parent.alive:
+            return False
+        try:
+            self._msg_p
+        except errors.ObjectDestroyedError:
+            return False
+        else:
+            return True
+
+    def __del__(self):
+        self._destroy()
+
+    def _destroy(self):
+        if self.alive:
+            capi.lib.notmuch_message_destroy(self._msg_p)
+        self._msg_p = None
+
+    @property
+    def messageid(self):
+        """The message ID as a string.
+
+        The message ID is decoded with the ignore error handler.  This
+        is fine as long as the message ID is well formed.  If it is
+        not valid ASCII then this will be lossy.  So if you need to be
+        able to write the exact same message ID back you should use
+        :attr:`messageidb`.
+
+        Note that notmuch will decode the message ID value and thus
+        strip off the surrounding ``<`` and ``>`` characters.  This is
+        different from Python's :mod:`email` package behaviour which
+        leaves these characters in place.
+
+        :returns: The message ID.
+        :rtype: :class:`BinString`, this is a normal str but calling
+           bytes() on it will return the original bytes used to create
+           it.
+
+        :raises ObjectDestroyedError: if used after destoryed.
+        """
+        ret = capi.lib.notmuch_message_get_message_id(self._msg_p)
+        return base.BinString(capi.ffi.string(ret))
+
+    @property
+    def threadid(self):
+        """The thread ID.
+
+        The thread ID is decoded with the surrogateescape error
+        handler so that it is possible to reconstruct the original
+        thread ID if it is not valid UTF-8.
+
+        :returns: The thread ID.
+        :rtype: :class:`BinString`, this is a normal str but calling
+           bytes() on it will return the original bytes used to create
+           it.
+
+        :raises ObjectDestroyedError: if used after destoryed.
+        """
+        ret = capi.lib.notmuch_message_get_thread_id(self._msg_p)
+        return base.BinString(capi.ffi.string(ret))
+
+    @property
+    def path(self):
+        """A pathname of the message as a pathlib.Path instance.
+
+        If multiple files in the database contain the same message ID
+        this will be just one of the files, chosen at random.
+
+        :raises ObjectDestroyedError: if used after destoryed.
+        """
+        ret = capi.lib.notmuch_message_get_filename(self._msg_p)
+        return pathlib.Path(os.fsdecode(capi.ffi.string(ret)))
+
+    @property
+    def pathb(self):
+        """A pathname of the message as a bytes object.
+
+        See :attr:`path` for details, this is the same but does return
+        the path as a bytes object which is faster but less convenient.
+
+        :raises ObjectDestroyedError: if used after destoryed.
+        """
+        ret = capi.lib.notmuch_message_get_filename(self._msg_p)
+        return capi.ffi.string(ret)
+
+    def filenames(self):
+        """Return an iterator of all files for this message.
+
+        If multiple files contained the same message ID they will all
+        be returned here.  The files are returned as intances of
+        :class:`pathlib.Path`.
+
+        :returns: Iterator yielding :class:`pathlib.Path` instances.
+        :rtype: iter
+
+        :raises ObjectDestroyedError: if used after destoryed.
+        """
+        fnames_p = capi.lib.notmuch_message_get_filenames(self._msg_p)
+        return PathIter(self, fnames_p)
+
+    def filenamesb(self):
+        """Return an iterator of all files for this message.
+
+        This is like :meth:`pathnames` but the files are returned as
+        byte objects instead.
+
+        :returns: Iterator yielding :class:`bytes` instances.
+        :rtype: iter
+
+        :raises ObjectDestroyedError: if used after destoryed.
+        """
+        fnames_p = capi.lib.notmuch_message_get_filenames(self._msg_p)
+        return FilenamesIter(self, fnames_p)
+
+    @property
+    def ghost(self):
+        """Indicates whether this message is a ghost message.
+
+        A ghost message if a message which we know exists, but it has
+        no files or content associated with it.  This can happen if
+        it was referenced by some other message.  Only the
+        :attr:`messageid` and :attr:`threadid` attributes are valid
+        for it.
+
+        :raises ObjectDestroyedError: if used after destoryed.
+        """
+        ret = capi.lib.notmuch_message_get_flag(
+            self._msg_p, capi.lib.NOTMUCH_MESSAGE_FLAG_GHOST)
+        return bool(ret)
+
+    @property
+    def excluded(self):
+        """Indicates whether this message was excluded from the query.
+
+        When a message is created from a search, sometimes messages
+        that where excluded by the search query could still be
+        returned by it, e.g. because they are part of a thread
+        matching the query.  the :meth:`Database.query` method allows
+        these messages to be flagged, which results in this property
+        being set to *True*.
+
+        :raises ObjectDestroyedError: if used after destoryed.
+        """
+        ret = capi.lib.notmuch_message_get_flag(
+            self._msg_p, capi.lib.NOTMUCH_MESSAGE_FLAG_EXCLUDED)
+        return bool(ret)
+
+    @property
+    def date(self):
+        """The message date as an integer.
+
+        The time the message was sent as an integer number of seconds
+        since the *epoch*, 1 Jan 1970.  This is derived from the
+        message's header, you can get the original header value with
+        :meth:`header`.
+
+        :raises ObjectDestroyedError: if used after destoryed.
+        """
+        return capi.lib.notmuch_message_get_date(self._msg_p)
+
+    def header(self, name):
+        """Return the value of the named header.
+
+        Returns the header from notmuch, some common headers are
+        stored in the database, others are read from the file.
+        Headers are returned with their newlines stripped and
+        collapsed concatenated together if they occur multiple times.
+        You may be better off using the standard library email
+        package's ``email.message_from_file(msg.path.open())`` if that
+        is not sufficient for you.
+
+        :param header: Case-insensitive header name to retrieve.
+        :type header: str or bytes
+
+        :returns: The header value, an empty string if the header is
+           not present.
+        :rtype: str
+
+        :raises LookupError: if the header is not present.
+        :raises NullPointerError: For unexpected notmuch errors.
+        :raises ObjectDestroyedError: if used after destoryed.
+        """
+        # The returned is supposedly guaranteed to be UTF-8.  Header
+        # names must be ASCII as per RFC x822.
+        if isinstance(name, str):
+            name = name.encode('ascii')
+        ret = capi.lib.notmuch_message_get_header(self._msg_p, name)
+        if ret == capi.ffi.NULL:
+            raise errors.NullPointerError()
+        hdr = capi.ffi.string(ret)
+        if not hdr:
+            raise LookupError
+        return hdr.decode(encoding='utf-8')
+
+    @property
+    def tags(self):
+        """The tags associated with the message.
+
+        This behaves as a set.  But removing and adding items to the
+        set removes and adds them to the message in the database.
+
+        :raises ReadOnlyDatabaseError: When manipulating tags on a
+           database opened in read-only mode.
+        :raises ObjectDestroyedError: if used after destoryed.
+        """
+        try:
+            ref = self._cached_tagset
+        except AttributeError:
+            tagset = None
+        else:
+            tagset = ref()
+        if tagset is None:
+            tagset = tags.MutableTagSet(
+                self, '_msg_p', capi.lib.notmuch_message_get_tags)
+            self._cached_tagset = weakref.ref(tagset)
+        return tagset
+
+    @contextlib.contextmanager
+    def frozen(self):
+        """Context manager to freeze the message state.
+
+        This allows you to perform atomic tag updates::
+
+           with msg.frozen():
+               msg.tags.clear()
+               msg.tags.add('foo')
+
+        Using This would ensure the message never ends up with no tags
+        applied at all.
+
+        It is safe to nest calls to this context manager.
+
+        :raises ReadOnlyDatabaseError: if the database is opened in
+           read-only mode.
+        :raises UnbalancedFreezeThawError: if you somehow managed to
+           call __exit__ of this context manager more than once.  Why
+           did you do that?
+        :raises ObjectDestroyedError: if used after destoryed.
+        """
+        ret = capi.lib.notmuch_message_freeze(self._msg_p)
+        if ret != capi.lib.NOTMUCH_STATUS_SUCCESS:
+            raise errors.NotmuchError(ret)
+        self._frozen = True
+        try:
+            yield
+        except Exception:
+            # Only way to "rollback" these changes is to destroy
+            # ourselves and re-create.  Behold.
+            msgid = self.messageid
+            self._destroy()
+            with contextlib.suppress(Exception):
+                new = self._db.find(msgid)
+                self._msg_p = new._msg_p
+                new._msg_p = None
+                del new
+            raise
+        else:
+            ret = capi.lib.notmuch_message_thaw(self._msg_p)
+            if ret != capi.lib.NOTMUCH_STATUS_SUCCESS:
+                raise errors.NotmuchError(ret)
+            self._frozen = False
+
+    @property
+    def properties(self):
+        """A map of arbitrary key-value pairs associated with the message.
+
+        Be aware that properties may be used by other extensions to
+        store state in.  So delete or modify with care.
+
+        The properties map is somewhat special.  It is essentially a
+        multimap-like structure where each key can have multiple
+        values.  Therefore accessing a single item using
+        :meth:`PropertiesMap.get` or :meth:`PropertiesMap.__getitem__`
+        will only return you the *first* item if there are multiple
+        and thus are only recommended if you know there to be only one
+        value.
+
+        Instead the map has an additional :meth:`PropertiesMap.all`
+        method which can be used to retrieve all properties of a given
+        key.  This method also allows iterating of a a subset of the
+        keys starting with a given prefix.
+        """
+        try:
+            ref = self._cached_props
+        except AttributeError:
+            props = None
+        else:
+            props = ref()
+        if props is None:
+            props = PropertiesMap(self, '_msg_p')
+            self._cached_props = weakref.ref(props)
+        return props
+
+    def replies(self):
+        """Return an iterator of all replies to this message.
+
+        This method will only work if the message was created from a
+        thread.  Otherwise it will yield no results.
+
+        :returns: An iterator yielding :class:`Message` instances.
+        :rtype: MessageIter
+        """
+        # The notmuch_messages_valid call accepts NULL and this will
+        # become an empty iterator, raising StopIteration immediately.
+        # Hence no return value checking here.
+        msgs_p = capi.lib.notmuch_message_get_replies(self._msg_p)
+        return MessageIter(self, msgs_p, db=self._db)
+
+    def __hash__(self):
+        return hash(self.messageid)
+
+    def __eq__(self, other):
+        if isinstance(other, self.__class__):
+            return self.messageid == other.messageid
+
+
+class FilenamesIter(base.NotmuchIter):
+    """Iterator for binary filenames objects."""
+
+    def __init__(self, parent, iter_p):
+        super().__init__(parent, iter_p,
+                         fn_destroy=capi.lib.notmuch_filenames_destroy,
+                         fn_valid=capi.lib.notmuch_filenames_valid,
+                         fn_get=capi.lib.notmuch_filenames_get,
+                         fn_next=capi.lib.notmuch_filenames_move_to_next)
+
+    def __next__(self):
+        fname = super().__next__()
+        return capi.ffi.string(fname)
+
+
+class PathIter(FilenamesIter):
+    """Iterator for pathlib.Path objects."""
+
+    def __next__(self):
+        fname = super().__next__()
+        return pathlib.Path(os.fsdecode(fname))
+
+
+class PropertiesMap(base.NotmuchObject, collections.abc.MutableMapping):
+    """A mutable mapping to manage properties.
+
+    Both keys and values of properties are supposed to be UTF-8
+    strings in libnotmuch.  However since the uderlying API uses
+    bytestrings you can use either str or bytes to represent keys and
+    all returned keys and values use :class:`BinString`.
+
+    Also be aware that ``iter(this_map)`` will return duplicate keys,
+    while the :class:`collections.abc.KeysView` returned by
+    :meth:`keys` is a :class:`collections.abc.Set` subclass.  This
+    means the former will yield duplicate keys while the latter won't.
+    It also means ``len(list(iter(this_map)))`` could be different
+    than ``len(this_map.keys())``.  ``len(this_map)`` will correspond
+    with the lenght of the default iterator.
+
+    Be aware that libnotmuch exposes all of this as iterators, so
+    quite a few operations have O(n) performance instead of the usual
+    O(1).
+    """
+    Property = collections.namedtuple('Property', ['key', 'value'])
+    _marker = object()
+
+    def __init__(self, msg, ptr_name):
+        self._msg = msg
+        self._ptr = lambda: getattr(msg, ptr_name)
+
+    @property
+    def alive(self):
+        if not self._msg.alive:
+            return False
+        try:
+            self._ptr
+        except errors.ObjectDestroyedError:
+            return False
+        else:
+            return True
+
+    def _destroy(self):
+        pass
+
+    def __iter__(self):
+        """Return an iterator which iterates over the keys.
+
+        Be aware that a single key may have multiple values associated
+        with it, if so it will appear multiple times here.
+        """
+        iter_p = capi.lib.notmuch_message_get_properties(self._ptr(), b'', 0)
+        return PropertiesKeyIter(self, iter_p)
+
+    def __len__(self):
+        iter_p = capi.lib.notmuch_message_get_properties(self._ptr(), b'', 0)
+        it = base.NotmuchIter(
+            self, iter_p,
+            fn_destroy=capi.lib.notmuch_message_properties_destroy,
+            fn_valid=capi.lib.notmuch_message_properties_valid,
+            fn_get=capi.lib.notmuch_message_properties_key,
+            fn_next=capi.lib.notmuch_message_properties_move_to_next,
+        )
+        return len(list(it))
+
+    def __getitem__(self, key):
+        """Return **the first** peroperty associated with a key."""
+        if isinstance(key, str):
+            key = key.encode('utf-8')
+        value_pp = capi.ffi.new('char**')
+        ret = capi.lib.notmuch_message_get_property(self._ptr(), key, value_pp)
+        if ret != capi.lib.NOTMUCH_STATUS_SUCCESS:
+            raise errors.NotmuchError(ret)
+        if value_pp[0] == capi.ffi.NULL:
+            raise KeyError
+        return base.BinString.from_cffi(value_pp[0])
+
+    def keys(self):
+        """Return a :class:`collections.abc.KeysView` for this map.
+
+        Even when keys occur multiple times this is a subset of set()
+        so will only contain them once.
+        """
+        return collections.abc.KeysView({k: None for k in self})
+
+    def items(self):
+        """Return a :class:`collections.abc.ItemsView` for this map.
+
+        The ItemsView treats a ``(key, value)`` pair as unique, so
+        dupcliate ``(key, value)`` pairs will be merged together.
+        However duplicate keys with different values will be returned.
+        """
+        items = set()
+        props_p = capi.lib.notmuch_message_get_properties(self._ptr(), b'', 0)
+        while capi.lib.notmuch_message_properties_valid(props_p):
+            key = capi.lib.notmuch_message_properties_key(props_p)
+            value = capi.lib.notmuch_message_properties_value(props_p)
+            items.add((base.BinString.from_cffi(key),
+                       base.BinString.from_cffi(value)))
+            capi.lib.notmuch_message_properties_move_to_next(props_p)
+        capi.lib.notmuch_message_properties_destroy(props_p)
+        return PropertiesItemsView(items)
+
+    def values(self):
+        """Return a :class:`collecions.abc.ValuesView` for this map.
+
+        All unique property values are included in the view.
+        """
+        values = set()
+        props_p = capi.lib.notmuch_message_get_properties(self._ptr(), b'', 0)
+        while capi.lib.notmuch_message_properties_valid(props_p):
+            value = capi.lib.notmuch_message_properties_value(props_p)
+            values.add(base.BinString.from_cffi(value))
+            capi.lib.notmuch_message_properties_move_to_next(props_p)
+        capi.lib.notmuch_message_properties_destroy(props_p)
+        return PropertiesValuesView(values)
+
+    def __setitem__(self, key, value):
+        """Add a key-value pair to the properties.
+
+        You may prefer to use :meth:`add` for clarity since this
+        method usually implies implicit overwriting of an existing key
+        if it exists, while for properties this is not the case.
+        """
+        self.add(key, value)
+
+    def add(self, key, value):
+        """Add a key-value pair to the properties."""
+        if isinstance(key, str):
+            key = key.encode('utf-8')
+        if isinstance(value, str):
+            value = value.encode('utf-8')
+        ret = capi.lib.notmuch_message_add_property(self._ptr(), key, value)
+        if ret != capi.lib.NOTMUCH_STATUS_SUCCESS:
+            raise errors.NotmuchError(ret)
+
+    def __delitem__(self, key):
+        """Remove all properties with this key."""
+        if isinstance(key, str):
+            key = key.encode('utf-8')
+        ret = capi.lib.notmuch_message_remove_all_properties(self._ptr(), key)
+        if ret != capi.lib.NOTMUCH_STATUS_SUCCESS:
+            raise errors.NotmuchError(ret)
+
+    def remove(self, key, value):
+        """Remove a key-value pair from the properties."""
+        if isinstance(key, str):
+            key = key.encode('utf-8')
+        if isinstance(value, str):
+            value = value.encode('utf-8')
+        ret = capi.lib.notmuch_message_remove_property(self._ptr(), key, value)
+        if ret != capi.lib.NOTMUCH_STATUS_SUCCESS:
+            raise errors.NotmuchError(ret)
+
+    def pop(self, key, default=_marker):
+        try:
+            value = self[key]
+        except KeyError:
+            if default is self._marker:
+                raise
+            else:
+                return default
+        else:
+            self.remove(key, value)
+            return value
+
+    def popitem(self):
+        try:
+            key = next(iter(self))
+        except StopIteration:
+            raise KeyError
+        value = self.pop(key)
+        return (key, value)
+
+    def clear(self):
+        ret = capi.lib.notmuch_message_remove_all_properties(self._ptr(),
+                                                             capi.ffi.NULL)
+        if ret != capi.lib.NOTMUCH_STATUS_SUCCESS:
+            raise errors.NotmuchError(ret)
+
+    def getall(self, prefix='', *, exact=False):
+        """Return an iterator yielding all properties for a given key prefix.
+
+        The returned iterator yields all peroperties which start with
+        a given key prefix as ``(key, value)`` namedtuples.  If called
+        with ``exact=True`` then only properties which exactly match
+        the prefix are returned, those a key longer than the prefix
+        will not be included.
+
+        :param prefix: The prefix of the key.
+        """
+        if isinstance(prefix, str):
+            prefix = prefix.encode('utf-8')
+        props_p = capi.lib.notmuch_message_get_properties(self._ptr(),
+                                                          prefix, exact)
+        return PropertiesIter(self, props_p)
+
+
+class PropertiesKeyIter(base.NotmuchIter):
+
+    def __init__(self, parent, iter_p):
+        super().__init__(
+            parent,
+            iter_p,
+            fn_destroy=capi.lib.notmuch_message_properties_destroy,
+            fn_valid=capi.lib.notmuch_message_properties_valid,
+            fn_get=capi.lib.notmuch_message_properties_key,
+            fn_next=capi.lib.notmuch_message_properties_move_to_next)
+
+    def __next__(self):
+        item = super().__next__()
+        return base.BinString.from_cffi(item)
+
+
+class PropertiesIter(base.NotmuchIter):
+
+    def __init__(self, parent, iter_p):
+        super().__init__(
+            parent,
+            iter_p,
+            fn_destroy=capi.lib.notmuch_message_properties_destroy,
+            fn_valid=capi.lib.notmuch_message_properties_valid,
+            fn_get=capi.lib.notmuch_message_properties_key,
+            fn_next=capi.lib.notmuch_message_properties_move_to_next,
+        )
+
+    def __next__(self):
+        if not self._fn_valid(self._iter_p):
+            self._destroy()
+            raise StopIteration
+        key = capi.lib.notmuch_message_properties_key(self._iter_p)
+        value = capi.lib.notmuch_message_properties_value(self._iter_p)
+        capi.lib.notmuch_message_properties_move_to_next(self._iter_p)
+        return PropertiesMap.Property(base.BinString.from_cffi(key),
+                                      base.BinString.from_cffi(value))
+
+
+class PropertiesItemsView(collections.abc.Set):
+
+    __slots__ = ('_items',)
+
+    def __init__(self, items):
+        self._items = items
+
+    @classmethod
+    def _from_iterable(self, it):
+        return set(it)
+
+    def __len__(self):
+        return len(self._items)
+
+    def __contains__(self, item):
+        return item in self._items
+
+    def __iter__(self):
+        yield from self._items
+
+
+collections.abc.ItemsView.register(PropertiesItemsView)
+
+
+class PropertiesValuesView(collections.abc.Set):
+
+    __slots__ = ('_values',)
+
+    def __init__(self, values):
+        self._values = values
+
+    def __len__(self):
+        return len(self._values)
+
+    def __contains__(self, value):
+        return value in self._values
+
+    def __iter__(self):
+        yield from self._values
+
+
+collections.abc.ValuesView.register(PropertiesValuesView)
+
+
+class MessageIter(base.NotmuchIter):
+
+    def __init__(self, parent, msgs_p, *, db):
+        self._db = db
+        super().__init__(parent, msgs_p,
+                         fn_destroy=capi.lib.notmuch_messages_destroy,
+                         fn_valid=capi.lib.notmuch_messages_valid,
+                         fn_get=capi.lib.notmuch_messages_get,
+                         fn_next=capi.lib.notmuch_messages_move_to_next)
+
+    def __next__(self):
+        msg_p = super().__next__()
+        return Message(self, msg_p, db=self._db)
diff --git a/bindings/python-cffi/notdb/_query.py b/bindings/python-cffi/notdb/_query.py
new file mode 100644
index 00000000..613aaf12
--- /dev/null
+++ b/bindings/python-cffi/notdb/_query.py
@@ -0,0 +1,83 @@
+from notdb import _base as base
+from notdb import _capi as capi
+from notdb import _errors as errors
+from notdb import _message as message
+from notdb import _thread as thread
+
+
+__all__ = []
+
+
+class Query(base.NotmuchObject):
+    """Private, minimal query object.
+
+    This is not meant for users and is not a full implementation of
+    the query API.  It is only an intermediate used internally to
+    match libnotmuch's memory management.
+    """
+    _query_p = base.MemoryPointer()
+
+    def __init__(self, db, query_p):
+        self._db = db
+        self._query_p = query_p
+
+    @property
+    def alive(self):
+        if not self._db.alive:
+            return False
+        try:
+            self._query_p
+        except errors.ObjectDestroyedError:
+            return False
+        else:
+            return True
+
+    def __del__(self):
+        self._destroy()
+
+    def _destroy(self):
+        if self.alive:
+            capi.lib.notmuch_query_destroy(self._query_p)
+        self._query_p = None
+
+    @property
+    def query(self):
+        """The query string as seen by libnotmuch."""
+        q = capi.lib.notmuch_query_get_query_string(self._query_p)
+        return base.BinString.from_cffi(q)
+
+    def messages(self):
+        """Return an iterator over all the messages found by the query.
+
+        This executes the query and returns an iterator over the
+        :class:`Message` objects found.
+        """
+        msgs_pp = capi.ffi.new('notmuch_messages_t**')
+        ret = capi.lib.notmuch_query_search_messages(self._query_p, msgs_pp)
+        if ret != capi.lib.NOTMUCH_STATUS_SUCCESS:
+            raise errors.NotmuchError(ret)
+        return message.MessageIter(self, msgs_pp[0], db=self._db)
+
+    def count_messages(self):
+        """Return the number of messages matching this query."""
+        count_p = capi.ffi.new('unsigned int *')
+        ret = capi.lib.notmuch_query_count_messages(self._query_p, count_p)
+        if ret != capi.lib.NOTMUCH_STATUS_SUCCESS:
+            raise errors.NotmuchError(ret)
+        return count_p[0]
+
+    def threads(self):
+        """Return an iterator over all the threads found by the query."""
+        threads_pp = capi.ffi.new('notmuch_threads_t **')
+        ret = capi.lib.notmuch_query_search_threads(self._query_p, threads_pp)
+        if ret != capi.lib.NOTMUCH_STATUS_SUCCESS:
+            raise errors.NotmuchError(ret)
+        return thread.ThreadIter(self, threads_pp[0], db=self._db)
+
+    def count_threads(self):
+        """Return the number of threads matching this query."""
+        count_p = capi.ffi.new('unsigned int *')
+        ret = capi.lib.notmuch_query_count_threads(self._query_p, count_p)
+        if ret != capi.lib.NOTMUCH_STATUS_SUCCESS:
+            raise errors.NotmuchError(ret)
+        return count_p[0]
diff --git a/bindings/python-cffi/notdb/_tags.py b/bindings/python-cffi/notdb/_tags.py
new file mode 100644
index 00000000..a25a2264
--- /dev/null
+++ b/bindings/python-cffi/notdb/_tags.py
@@ -0,0 +1,338 @@
+import collections.abc
+
+import notdb._base as base
+import notdb._capi as capi
+import notdb._errors as errors
+
+
+__all__ = ['ImmutableTagSet', 'MutableTagSet', 'TagsIter']
+
+
+class ImmutableTagSet(base.NotmuchObject, collections.abc.Set):
+    """The tags associated with a message thread or whole database.
+
+    Both a thread as well as the database expose the union of all tags
+    in messages associated with them.  This exposes these as a
+    :class:`collections.abc.Set` object.
+
+    Note that due to the underlying notmuch API the performance of the
+    implementation is not the same as you would expect from normal
+    sets.  E.g. the :meth:`__contains__` and :meth:`__len__` are O(n)
+    rather then O(1).
+
+    Tags are internally stored as bytestrings but normally exposed as
+    unicode strings using the UTF-8 encoding and the *ignore* decoder
+    error handler.  However the :meth:`iter` method can be used to
+    return tags as bytestrings or using a different error handler.
+
+    Note that when doing arithmetic operations on tags, this class
+    will return a plain normal set as it is no longer associated with
+    the message.
+
+    :param parent: the parent object
+    :param ptr_name: the name of the attribute on the parent which will
+       return the memory pointer.  This allows this object to
+       access the pointer via the parent's descriptor and thus
+       trigger :class:`MemoryPointer`'s memory safety.
+    :param cffi_fn: the callable CFFI wrapper to retrieve the tags
+       iter.  This can be one of notmuch_database_get_all_tags,
+       notmuch_thread_get_tags or notmuch_message_get_tags.
+    """
+
+    def __init__(self, parent, ptr_name, cffi_fn):
+        self._parent = parent
+        self._ptr = lambda: getattr(parent, ptr_name)
+        self._cffi_fn = cffi_fn
+
+    def __del__(self):
+        self._destroy()
+
+    @property
+    def alive(self):
+        return self._parent.alive
+
+    def _destroy(self):
+        pass
+
+    @classmethod
+    def _from_iterable(cls, it):
+        return set(it)
+
+    def __iter__(self):
+        """Return an iterator over the tags.
+
+        Tags are yielded as unicode strings, decoded using the
+        "ignore" error handler.
+
+        :raises NullPointerError: If the iterator can not be created.
+        """
+        return self.iter(encoding='utf-8', errors='ignore')
+
+    def iter(self, *, encoding=None, errors='strict'):
+        """Aternate iterator constructor controlling string decoding.
+
+        Tags are stored as bytes in the notmuch database, in Python
+        it's easier to work with unicode strings and thus is what the
+        normal iterator returns.  However this method allows you to
+        specify how you would like to get the tags, defaulting to the
+        bytestring representation instead of unicode strings.
+
+        :param encoding: Which codec to use.  The default *None* does not
+           decode at all and will return the unmodified bytes.
+           Otherwise this is passed on to :func:`str.decode`.
+        :param errors: If using a codec, this is the error handler.
+           See :func:`str.decode` to which this is passed on.
+
+        :raises NullPointerError: When things do not go as planned.
+        """
+        # self._cffi_fn should point either to
+        # notmuch_database_get_all_tags, notmuch_thread_get_tags or
+        # notmuch_message_get_tags.  nothmuch.h suggests these never
+        # fail, let's handle NULL anyway.
+        tags_p = self._cffi_fn(self._ptr())
+        if tags_p == capi.ffi.NULL:
+            raise errors.NullPointerError()
+        tags = TagsIter(self, tags_p, encoding=encoding, errors=errors)
+        return tags
+
+    def __len__(self):
+        return sum(1 for t in self)
+
+    def __contains__(self, tag):
+        if isinstance(tag, str):
+            tag = tag.encode()
+        for msg_tag in self.iter():
+            if tag == msg_tag:
+                return True
+        else:
+            return False
+
+    def __eq__(self, other):
+        return tuple(sorted(self.iter())) == tuple(sorted(other.iter()))
+
+    def __hash__(self):
+        return hash(tuple(self.iter()))
+
+    def __repr__(self):
+        return '<{name} object at 0x{addr:x} tags={{{tags}}}>'.format(
+            name=self.__class__.__name__,
+            addr=id(self),
+            tags=', '.join(repr(t) for t in self))
+
+
+class MutableTagSet(ImmutableTagSet, collections.abc.MutableSet):
+    """The tags associated with a message.
+
+    This is a :class:`collections.abc.MutableSet` object which can be
+    used to manipulate the tags of a message.
+
+    Note that due to the underlying notmuch API the performance of the
+    implementation is not the same as you would expect from normal
+    sets.  E.g. the ``in`` operator and variants are O(n) rather then
+    O(1).
+
+    Tags are bytestrings and calling ``iter()`` will return an
+    iterator yielding bytestrings.  However the :meth:`iter` method
+    can be used to return tags as unicode strings, while all other
+    operations accept either byestrings or unicode strings.  In case
+    unicode strings are used they will be encoded using utf-8 before
+    being passed to notmuch.
+    """
+
+    # Since we subclass ImmutableTagSet we inherit a __hash__.  But we
+    # are mutable, setting it to None will make the Python machinary
+    # recognise us as unhashable.
+    __hash__ = None
+
+    def add(self, tag):
+        """Add a tag to the message.
+
+        :param tag: The tag to add.
+        :type tag: str or bytes.  A str will be encoded using UTF-8.
+
+        :param sync_flags: Whether to sync the maildir flags with the
+           new set of tags.  Leaving this as *None* respects the
+           configuration set in the database, while *True* will always
+           sync and *False* will never sync.
+        :param sync_flags: NoneType or bool
+
+        :raises TypeError: If the tag is not a valid type.
+        :raises TagTooLongError: If the added tag exceeds the maximum
+           lenght, see ``notmuch_cffi.NOTMUCH_TAG_MAX``.
+        :raises ReadOnlyDatabaseError: If the database is opened in
+           read-only mode.
+        """
+        if isinstance(tag, str):
+            tag = tag.encode()
+        if not isinstance(tag, bytes):
+            raise TypeError('Not a valid type for a tag: {}'.format(type(tag)))
+        ret = capi.lib.notmuch_message_add_tag(self._ptr(), tag)
+        if ret != capi.lib.NOTMUCH_STATUS_SUCCESS:
+            raise errors.NotmuchError(ret)
+
+    def discard(self, tag):
+        """Remove a tag from the message.
+
+        :param tag: The tag to remove.
+        :type tag: str of bytes.  A str will be encoded using UTF-8.
+        :param sync_flags: Whether to sync the maildir flags with the
+           new set of tags.  Leaving this as *None* respects the
+           configuration set in the database, while *True* will always
+           sync and *False* will never sync.
+        :param sync_flags: NoneType or bool
+
+        :raises TypeError: If the tag is not a valid type.
+        :raises TagTooLongError: If the tag exceeds the maximum
+           lenght, see ``notmuch_cffi.NOTMUCH_TAG_MAX``.
+        :raises ReadOnlyDatabaseError: If the database is opened in
+           read-only mode.
+        """
+        if isinstance(tag, str):
+            tag = tag.encode()
+        if not isinstance(tag, bytes):
+            raise TypeError('Not a valid type for a tag: {}'.format(type(tag)))
+        ret = capi.lib.notmuch_message_remove_tag(self._ptr(), tag)
+        if ret != capi.lib.NOTMUCH_STATUS_SUCCESS:
+            raise errors.NotmuchError(ret)
+
+    def clear(self):
+        """Remove all tags from the message.
+
+        :raises ReadOnlyDatabaseError: If the database is opened in
+           read-only mode.
+        """
+        ret = capi.lib.notmuch_message_remove_all_tags(self._ptr())
+        if ret != capi.lib.NOTMUCH_STATUS_SUCCESS:
+            raise errors.NotmuchError(ret)
+
+    def from_maildir_flags(self):
+        """Update the tags based on the state in the message's maildir flags.
+
+        This function examines the filenames of 'message' for maildir
+        flags, and adds or removes tags on 'message' as follows when
+        these flags are present:
+
+        Flag    Action if present
+        ----    -----------------
+        'D'     Adds the "draft" tag to the message
+        'F'     Adds the "flagged" tag to the message
+        'P'     Adds the "passed" tag to the message
+        'R'     Adds the "replied" tag to the message
+        'S'     Removes the "unread" tag from the message
+
+        For each flag that is not present, the opposite action
+        (add/remove) is performed for the corresponding tags.
+
+        Flags are identified as trailing components of the filename
+        after a sequence of ":2,".
+
+        If there are multiple filenames associated with this message,
+        the flag is considered present if it appears in one or more
+        filenames. (That is, the flags from the multiple filenames are
+        combined with the logical OR operator.)
+        """
+        ret = capi.lib.notmuch_message_maildir_flags_to_tags(self._ptr())
+        if ret != capi.lib.NOTMUCH_STATUS_SUCCESS:
+            raise errors.NotmuchError(ret)
+
+    def to_maildir_flags(self):
+        """Update the message's maildir flags based on the notmuch tags.
+
+        If the message's filename is in a maildir directory, that is a
+        directory named ``new`` or ``cur``, and has a valid maildir
+        filename then the flags will be added as such:
+
+        'D' if the message has the "draft" tag
+        'F' if the message has the "flagged" tag
+        'P' if the message has the "passed" tag
+        'R' if the message has the "replied" tag
+        'S' if the message does not have the "unread" tag
+
+        Any existing flags unmentioned in the list above will be
+        preserved in the renaming.
+
+        Also, if this filename is in a directory named "new", rename it to
+        be within the neighboring directory named "cur".
+
+        In case there are multiple files associated with the message
+        all filenames will get the same logic applied.
+        """
+        ret = capi.lib.notmuch_message_tags_to_maildir_flags(self._ptr())
+        if ret != capi.lib.NOTMUCH_STATUS_SUCCESS:
+            raise errors.NotmuchError(ret)
+
+
+class TagsIter(base.NotmuchObject, collections.abc.Iterator):
+    """Iterator over tags.
+
+    This is only an interator, not a container so calling
+    :meth:`__iter__` does not return a new, replenished iterator but
+    only itself.
+
+    :param parent: The parent object to keep alive.
+    :param tags_p: The CFFI pointer to the C-level tags iterator.
+    :param encoding: Which codec to use.  The default *None* does not
+       decode at all and will return the unmodified bytes.
+       Otherwise this is passed on to :func:`str.decode`.
+    :param errors: If using a codec, this is the error handler.
+       See :func:`str.decode` to which this is passed on.
+
+    :raises ObjectDestoryedError: if used after destroyed.
+    """
+    _tags_p = base.MemoryPointer()
+
+    def __init__(self, parent, tags_p, *, encoding=None, errors='strict'):
+        self._parent = parent
+        self._tags_p = tags_p
+        self._encoding = encoding
+        self._errors = errors
+
+    def __del__(self):
+        self._destroy()
+
+    @property
+    def alive(self):
+        if not self._parent.alive:
+            return False
+        try:
+            self._tags_p
+        except errors.ObjectDestroyedError:
+            return False
+        else:
+            return True
+
+    def _destroy(self):
+        if self.alive:
+            try:
+                capi.lib.notmuch_tags_destroy(self._tags_p)
+            except errors.ObjectDestroyedError:
+                pass
+        self._tags_p = None
+
+    def __iter__(self):
+        """Return the iterator itself.
+
+        Note that as this is an iterator and not a container this will
+        not return a new iterator.  Thus any elements already consumed
+        will not be yielded by the :meth:`__next__` method anymore.
+        """
+        return self
+
+    def __next__(self):
+        if not capi.lib.notmuch_tags_valid(self._tags_p):
+            self._destroy()
+            raise StopIteration()
+        tag_p = capi.lib.notmuch_tags_get(self._tags_p)
+        tag = capi.ffi.string(tag_p)
+        if self._encoding:
+            tag = tag.decode(encoding=self._encoding, errors=self._errors)
+        capi.lib.notmuch_tags_move_to_next(self._tags_p)
+        return tag
+
+    def __repr__(self):
+        try:
+            self._tags_p
+        except errors.ObjectDestroyedError:
+            return '<TagsIter (exhausted)>'
+        else:
+            return '<TagsIter>'
diff --git a/bindings/python-cffi/notdb/_thread.py b/bindings/python-cffi/notdb/_thread.py
new file mode 100644
index 00000000..e1ef6b07
--- /dev/null
+++ b/bindings/python-cffi/notdb/_thread.py
@@ -0,0 +1,190 @@
+import collections.abc
+import weakref
+
+from notdb import _base as base
+from notdb import _capi as capi
+from notdb import _errors as errors
+from notdb import _message as message
+from notdb import _tags as tags
+
+
+__all__ = ['Thread']
+
+
+class Thread(base.NotmuchObject, collections.abc.Iterable):
+    _thread_p = base.MemoryPointer()
+
+    def __init__(self, parent, thread_p, *, db):
+        self._parent = parent
+        self._thread_p = thread_p
+        self._db = db
+
+    @property
+    def alive(self):
+        if not self._parent.alive:
+            return False
+        try:
+            self._thread_p
+        except errors.ObjectDestroyedError:
+            return False
+        else:
+            return True
+
+    def __del__(self):
+        self._destroy()
+
+    def _destroy(self):
+        if self.alive:
+            capi.lib.notmuch_thread_destroy(self._thread_p)
+        self._thread_p = None
+
+    @property
+    def threadid(self):
+        """The thread ID as a :class:`BinString`.
+
+        :raises ObjectDestroyedError: if used after destoryed.
+        """
+        ret = capi.lib.notmuch_thread_get_thread_id(self._thread_p)
+        return base.BinString.from_cffi(ret)
+
+    def __len__(self):
+        """Return the number of messages in the thread.
+
+        :raises ObjectDestroyedError: if used after destoryed.
+        """
+        return capi.lib.notmuch_thread_get_total_messages(self._thread_p)
+
+    def toplevel(self):
+        """Return an iterator of the toplevel messages.
+
+        :returns: An iterator yielding :class:`Message` instances.
+
+        :raises ObjectDestroyedError: if used after destoryed.
+        """
+        msgs_p = capi.lib.notmuch_thread_get_toplevel_messages(self._thread_p)
+        return message.MessageIter(self, msgs_p, db=self._db)
+
+    def __iter__(self):
+        """Return an iterator over all the messages in the thread.
+
+        :returns: An iterator yielding :class:`Message` instances.
+
+        :raises ObjectDestroyedError: if used after destoryed.
+        """
+        msgs_p = capi.lib.notmuch_thread_get_messages(self._thread_p)
+        return message.MessageIter(self, msgs_p, db=self._db)
+
+    @property
+    def matched(self):
+        """The number of messages in this thread which matched the query.
+
+        Of the messages in the thread this gives the count of messages
+        which did directly match the search query which this thread
+        originates from.
+
+        :raises ObjectDestroyedError: if used after destoryed.
+        """
+        return capi.lib.notmuch_thread_get_matched_messages(self._thread_p)
+
+    @property
+    def authors(self):
+        """A comma-separated string of all authors in the thread.
+
+        Authors of messages which matched the query the thread was
+        retrieved from will be at the head of the string, ordered by
+        date of their messages.  Following this will be the authors of
+        the other messages in the thread, also ordered by date of
+        their messages.  Both groups of authors are separated by the
+        ``|`` character.
+
+        :returns: The stringified list of authors.
+        :rtype: BinString
+
+        :raises ObjectDestroyedError: if used after destoryed.
+        """
+        ret = capi.lib.notmuch_thread_get_authors(self._thread_p)
+        return base.BinString.from_cffi(ret)
+
+    @property
+    def subject(self):
+        """The subject of the thread, taken from the first message.
+
+        The thread's subject is taken to be the subject of the first
+        message according to query sort order.
+
+        :returns: The thread's subject.
+        :rtype: BinString
+
+        :raises ObjectDestroyedError: if used after destoryed.
+        """
+        ret = capi.lib.notmuch_thread_get_subject(self._thread_p)
+        return base.BinString.from_cffi(ret)
+
+    @property
+    def first(self):
+        """Return the date of the oldest message in the thread.
+
+        The time the first message was sent as an integer number of
+        seconds since the *epoch*, 1 Jan 1970.
+
+        :raises ObjectDestroyedError: if used after destoryed.
+        """
+        return capi.lib.notmuch_thread_get_oldest_date(self._thread_p)
+
+    @property
+    def last(self):
+        """Return the date of the newest message in the thread.
+
+        The time the last message was sent as an integer number of
+        seconds since the *epoch*, 1 Jan 1970.
+
+        :raises ObjectDestroyedError: if used after destoryed.
+        """
+        return capi.lib.notmuch_thread_get_newest_date(self._thread_p)
+
+    @property
+    def tags(self):
+        """Return an immutable set with all tags used in this thread.
+
+        This returns an immutable set-like object implementing the
+        collections.abc.Set Abstract Base Class.  Due to the
+        underlying libnotmuch implementation some operations have
+        different performance characteristics then plain set objects.
+        Mainly any lookup operation is O(n) rather then O(1).
+
+        Normal usage treats tags as UTF-8 encoded unicode strings so
+        they are exposed to Python as normal unicode string objects.
+        If you need to handle tags stored in libnotmuch which are not
+        valid unicode do check the :class:`ImmutableTagSet` docs for
+        how to handle this.
+
+        :rtype: ImmutableTagSet
+
+        :raises ObjectDestroyedError: if used after destoryed.
+        """
+        try:
+            ref = self._cached_tagset
+        except AttributeError:
+            tagset = None
+        else:
+            tagset = ref()
+        if tagset is None:
+            tagset = tags.ImmutableTagSet(
+                self, '_thread_p', capi.lib.notmuch_thread_get_tags)
+            self._cached_tagset = weakref.ref(tagset)
+        return tagset
+
+
+class ThreadIter(base.NotmuchIter):
+
+    def __init__(self, parent, threads_p, *, db):
+        self._db = db
+        super().__init__(parent, threads_p,
+                         fn_destroy=capi.lib.notmuch_threads_destroy,
+                         fn_valid=capi.lib.notmuch_threads_valid,
+                         fn_get=capi.lib.notmuch_threads_get,
+                         fn_next=capi.lib.notmuch_threads_move_to_next)
+
+    def __next__(self):
+        thread_p = super().__next__()
+        return Thread(self, thread_p, db=self._db)
diff --git a/bindings/python-cffi/setup.py b/bindings/python-cffi/setup.py
new file mode 100644
index 00000000..7baf63cf
--- /dev/null
+++ b/bindings/python-cffi/setup.py
@@ -0,0 +1,22 @@
+import setuptools
+
+
+setuptools.setup(
+    name='notdb',
+    version='0.1',
+    description='Pythonic bindings for the notmuch mail database using CFFI',
+    author='Floris Bruynooghe',
+    author_email='flub@devork.be',
+    setup_requires=['cffi>=1.0.0'],
+    install_requires=['cffi>=1.0.0'],
+    packages=setuptools.find_packages(exclude=['tests']),
+    cffi_modules=['notdb/_build.py:ffibuilder'],
+    classifiers=[
+        'Development Status :: 3 - Alpha',
+        'Intended Audience :: Developers',
+        'License :: OSI Approved :: GNU General Public License (GPL)',
+        'Programming Language :: Python :: 3',
+        'Topic :: Communications :: Email',
+        'Topic :: Software Development :: Libraries',
+    ],
+)
diff --git a/bindings/python-cffi/tests/conftest.py b/bindings/python-cffi/tests/conftest.py
new file mode 100644
index 00000000..1b7bbc35
--- /dev/null
+++ b/bindings/python-cffi/tests/conftest.py
@@ -0,0 +1,142 @@
+import email.message
+import mailbox
+import pathlib
+import socket
+import subprocess
+import textwrap
+import time
+
+import pytest
+
+
+@pytest.fixture(scope='function')
+def tmppath(tmpdir):
+    """The tmpdir fixture wrapped in pathlib.Path."""
+    return pathlib.Path(str(tmpdir))
+
+
+@pytest.fixture
+def notmuch(maildir):
+    """Return a function which runs notmuch commands on our test maildir.
+
+    This uses the notmuch-config file created by the ``maildir``
+    fixture.
+    """
+    def run(*args):
+        """Run a notmuch comand.
+
+        This function runs with a timeout error as many notmuch
+        commands may block if multiple processes are trying to open
+        the database in write-mode.  It is all too easy to
+        accidentally do this in the unittests.
+        """
+        cfg_fname = maildir.path / 'notmuch-config'
+        cmd = ['notmuch'] + list(args)
+        print('Invoking: {}'.format(' '.join(cmd)))
+        proc = subprocess.run(cmd,
+                              timeout=5,
+                              env={'NOTMUCH_CONFIG': str(cfg_fname)})
+        proc.check_returncode()
+    return run
+
+
+@pytest.fixture
+def maildir(tmppath):
+    """A basic test interface to a valid maildir directory.
+
+    This creates a valid maildir and provides a simple mechanism to
+    deliver test emails to it.  It also writes a notmuch-config file
+    in the top of the maildir.
+    """
+    cur = tmppath / 'cur'
+    cur.mkdir()
+    new = tmppath / 'new'
+    new.mkdir()
+    tmp = tmppath / 'tmp'
+    tmp.mkdir()
+    cfg_fname = tmppath/'notmuch-config'
+    with cfg_fname.open('w') as fp:
+        fp.write(textwrap.dedent("""\
+            [database]
+            path={tmppath!s}
+            [user]
+            name=Some Hacker
+            primary_email=dst@example.com
+            [new]
+            tags=unread;inbox;
+            ignore=
+            [search]
+            exclude_tags=deleted;spam;
+            [maildir]
+            synchronize_flags=true
+            [crypto]
+            gpg_path=gpg
+            """.format(tmppath=tmppath)))
+    return MailDir(tmppath)
+
+
+class MailDir:
+    """An interface around a correct maildir."""
+
+    def __init__(self, path):
+        self._path = pathlib.Path(path)
+        self.mailbox = mailbox.Maildir(str(path))
+        self._idcount = 0
+
+    @property
+    def path(self):
+        """The pathname of the maildir."""
+        return self._path
+
+    def _next_msgid(self):
+        """Return a new unique message ID."""
+        msgid = '{}@{}'.format(self._idcount, socket.getfqdn())
+        self._idcount += 1
+        return msgid
+
+    def deliver(self,
+                subject='Test mail',
+                body='This is a test mail',
+                to='dst@example.com',
+                frm='src@example.com',
+                headers=None,
+                new=False,      # Move to new dir or cur dir?
+                keywords=None,  # List of keywords or labels
+                seen=False,     # Seen flag (cur dir only)
+                replied=False,  # Replied flag (cur dir only)
+                flagged=False):  # Flagged flag (cur dir only)
+        """Deliver a new mail message in the mbox.
+
+        This does only adds the message to maildir, does not insert it
+        into the notmuch database.
+
+        :returns: A tuple of (msgid, pathname).
+        """
+        msgid = self._next_msgid()
+        when = time.time()
+        msg = email.message.EmailMessage()
+        msg.add_header('Received', 'by MailDir; {}'.format(time.ctime(when)))
+        msg.add_header('Message-ID', '<{}>'.format(msgid))
+        msg.add_header('Date', time.ctime(when))
+        msg.add_header('From', frm)
+        msg.add_header('To', to)
+        msg.add_header('Subject', subject)
+        if headers:
+            for h, v in headers:
+                msg.add_header(h, v)
+        msg.set_content(body)
+        mdmsg = mailbox.MaildirMessage(msg)
+        if not new:
+            mdmsg.set_subdir('cur')
+        if flagged:
+            mdmsg.add_flag('F')
+        if replied:
+            mdmsg.add_flag('R')
+        if seen:
+            mdmsg.add_flag('S')
+        boxid = self.mailbox.add(mdmsg)
+        basename = boxid
+        if mdmsg.get_info():
+            basename += mailbox.Maildir.colon + mdmsg.get_info()
+        msgpath = self.path / mdmsg.get_subdir() / basename
+        return (msgid, msgpath)
diff --git a/bindings/python-cffi/tests/test_base.py b/bindings/python-cffi/tests/test_base.py
new file mode 100644
index 00000000..b6d3d62c
--- /dev/null
+++ b/bindings/python-cffi/tests/test_base.py
@@ -0,0 +1,116 @@
+import pytest
+
+from notdb import _base as base
+from notdb import _errors as errors
+
+
+class TestNotmuchObject:
+
+    def test_no_impl_methods(self):
+        class Object(base.NotmuchObject):
+            pass
+        with pytest.raises(TypeError):
+            Object()
+
+    def test_impl_methods(self):
+
+        class Object(base.NotmuchObject):
+
+            def __init__(self):
+                pass
+
+            @property
+            def alive(self):
+                pass
+
+            def _destroy(self, parent=False):
+                pass
+
+        Object()
+
+    def test_del(self):
+        destroyed = False
+
+        class Object(base.NotmuchObject):
+
+            def __init__(self):
+                pass
+
+            @property
+            def alive(self):
+                pass
+
+            def _destroy(self, parent=False):
+                nonlocal destroyed
+                destroyed = True
+
+        o = Object()
+        o.__del__()
+        assert destroyed
+
+
+class TestMemoryPointer:
+
+    @pytest.fixture
+    def obj(self):
+        class Cls:
+            ptr = base.MemoryPointer()
+        return Cls()
+
+    def test_unset(self, obj):
+        with pytest.raises(errors.ObjectDestroyedError):
+            obj.ptr
+
+    def test_set(self, obj):
+        obj.ptr = 'some'
+        assert obj.ptr == 'some'
+
+    def test_cleared(self, obj):
+        obj.ptr = 'some'
+        obj.ptr
+        obj.ptr = None
+        with pytest.raises(errors.ObjectDestroyedError):
+            obj.ptr
+
+    def test_two_instances(self, obj):
+        obj2 = obj.__class__()
+        obj.ptr = 'foo'
+        obj2.ptr = 'bar'
+        assert obj.ptr != obj2.ptr
+
+
+class TestBinString:
+
+    def test_type(self):
+        s = base.BinString(b'foo')
+        assert isinstance(s, str)
+
+    def test_init_bytes(self):
+        s = base.BinString(b'foo')
+        assert s == 'foo'
+
+    def test_init_str(self):
+        s = base.BinString('foo')
+        assert s == 'foo'
+
+    def test_bytes(self):
+        s = base.BinString(b'foo')
+        assert bytes(s) == b'foo'
+
+    def test_invalid_utf8(self):
+        s = base.BinString(b'\x80foo')
+        assert s == 'foo'
+        assert bytes(s) == b'\x80foo'
+
+    def test_errors(self):
+        s = base.BinString(b'\x80foo', errors='replace')
+        assert s == '�foo'
+        assert bytes(s) == b'\x80foo'
+
+    def test_encoding(self):
+        # pound sign: '£' == '\u00a3' latin-1: b'\xa3', utf-8: b'\xc2\xa3'
+        with pytest.raises(UnicodeDecodeError):
+            base.BinString(b'\xa3', errors='strict')
+        s = base.BinString(b'\xa3', encoding='latin-1', errors='strict')
+        assert s == '£'
+        assert bytes(s) == b'\xa3'
diff --git a/bindings/python-cffi/tests/test_database.py b/bindings/python-cffi/tests/test_database.py
new file mode 100644
index 00000000..02de0f41
--- /dev/null
+++ b/bindings/python-cffi/tests/test_database.py
@@ -0,0 +1,326 @@
+import collections
+import configparser
+import os
+import pathlib
+
+import pytest
+
+import notdb
+import notdb._errors as errors
+import notdb._database as dbmod
+import notdb._message as message
+
+
+@pytest.fixture
+def db(maildir):
+    with dbmod.Database.create(maildir.path) as db:
+        yield db
+
+
+class TestDefaultDb:
+    """Tests for reading the default database.
+
+    The error cases are fairly undefined, some relevant Python error
+    will come out if you give it a bad filename or if the file does
+    not parse correctly.  So we're not testing this too deeply.
+    """
+
+    def test_config_pathname_default(self, monkeypatch):
+        monkeypatch.delenv('NOTMUCH_CONFIG', raising=False)
+        user = pathlib.Path('~/.notmuch-config').expanduser()
+        assert dbmod._config_pathname() == user
+
+    def test_config_pathname_env(self, monkeypatch):
+        monkeypatch.setenv('NOTMUCH_CONFIG', '/some/random/path')
+        assert dbmod._config_pathname() == pathlib.Path('/some/random/path')
+
+    def test_default_path_nocfg(self, monkeypatch, tmppath):
+        monkeypatch.setenv('NOTMUCH_CONFIG', str(tmppath/'foo'))
+        with pytest.raises(FileNotFoundError):
+            dbmod.Database.default_path()
+
+    def test_default_path_cfg_is_dir(self, monkeypatch, tmppath):
+        monkeypatch.setenv('NOTMUCH_CONFIG', str(tmppath))
+        with pytest.raises(IsADirectoryError):
+            dbmod.Database.default_path()
+
+    def test_default_path_parseerr(self, monkeypatch, tmppath):
+        cfg = tmppath / 'notmuch-config'
+        with cfg.open('w') as fp:
+            fp.write('invalid')
+        monkeypatch.setenv('NOTMUCH_CONFIG', str(cfg))
+        with pytest.raises(configparser.Error):
+            dbmod.Database.default_path()
+
+    def test_default_path_parse(self, monkeypatch, tmppath):
+        cfg = tmppath / 'notmuch-config'
+        with cfg.open('w') as fp:
+            fp.write('[database]\n')
+            fp.write('path={!s}'.format(tmppath))
+        monkeypatch.setenv('NOTMUCH_CONFIG', str(cfg))
+        assert dbmod.Database.default_path() == tmppath
+
+    def test_default_path_param(self, monkeypatch, tmppath):
+        cfg_dummy = tmppath / 'dummy'
+        monkeypatch.setenv('NOTMUCH_CONFIG', str(cfg_dummy))
+        cfg_real = tmppath / 'notmuch_config'
+        with cfg_real.open('w') as fp:
+            fp.write('[database]\n')
+            fp.write('path={!s}'.format(cfg_real/'mail'))
+        assert dbmod.Database.default_path(cfg_real) == cfg_real/'mail'
+
+
+class TestCreate:
+
+    def test_create(self, tmppath, db):
+        assert tmppath.joinpath('.notmuch/xapian/').exists()
+
+    def test_create_already_open(self, tmppath, db):
+        with pytest.raises(errors.NotmuchError):
+            db.create(tmppath)
+
+    def test_create_existing(self, tmppath, db):
+        with pytest.raises(errors.FileError):
+            dbmod.Database.create(path=tmppath)
+
+    def test_close(self, db):
+        db.close()
+
+    def test_del_noclose(self, db):
+        del db
+
+    def test_close_del(self, db):
+        db.close()
+        del db
+
+    def test_closed_attr(self, db):
+        assert not db.closed
+        db.close()
+        assert db.closed
+
+    def test_ctx(self, db):
+        with db as ctx:
+            assert ctx is db
+            assert not db.closed
+        assert db.closed
+
+    def test_path(self, db, tmppath):
+        assert db.path == tmppath
+
+    def test_version(self, db):
+        assert db.version > 0
+
+    def test_needs_upgrade(self, db):
+        assert db.needs_upgrade in (True, False)
+
+
+class TestAtomic:
+
+    def test_exit_early(self, db):
+        with pytest.raises(errors.UnbalancedAtomicError):
+            with db.atomic() as ctx:
+                ctx.force_end()
+
+    def test_exit_late(self, db):
+        with db.atomic() as ctx:
+            pass
+        with pytest.raises(errors.UnbalancedAtomicError):
+            ctx.force_end()
+
+
+class TestRevision:
+
+    def test_single_rev(self, db):
+        r = db.revision()
+        assert isinstance(r, dbmod.DbRevision)
+        assert isinstance(r.rev, int)
+        assert isinstance(r.uuid, bytes)
+        assert r is r
+        assert r == r
+        assert r <= r
+        assert r >= r
+        assert not r < r
+        assert not r > r
+
+    def test_diff_db(self, tmppath):
+        dbpath0 = tmppath.joinpath('db0')
+        dbpath0.mkdir()
+        dbpath1 = tmppath.joinpath('db1')
+        dbpath1.mkdir()
+        db0 = dbmod.Database.create(path=dbpath0)
+        db1 = dbmod.Database.create(path=dbpath1)
+        r_db0 = db0.revision()
+        r_db1 = db1.revision()
+        assert r_db0 != r_db1
+        assert r_db0.uuid != r_db1.uuid
+
+    def test_cmp(self, db, maildir):
+        rev0 = db.revision()
+        _, pathname = maildir.deliver()
+        db.add(pathname, sync_flags=False)
+        rev1 = db.revision()
+        assert rev0 < rev1
+        assert rev0 <= rev1
+        assert not rev0 > rev1
+        assert not rev0 >= rev1
+        assert not rev0 == rev1
+        assert rev0 != rev1
+
+    # XXX add tests for revisions comparisons
+
+class TestMessages:
+
+    def test_add_message(self, db, maildir):
+        msgid, pathname = maildir.deliver()
+        msg, dup = db.add(pathname, sync_flags=False)
+        assert isinstance(msg, message.Message)
+        assert msg.path == pathname
+        assert msg.messageid == msgid
+
+    def test_add_message_str(self, db, maildir):
+        msgid, pathname = maildir.deliver()
+        msg, dup = db.add(str(pathname), sync_flags=False)
+
+    def test_add_message_bytes(self, db, maildir):
+        msgid, pathname = maildir.deliver()
+        msg, dup = db.add(os.fsencode(bytes(pathname)), sync_flags=False)
+
+    def test_remove_message(self, db, maildir):
+        msgid, pathname = maildir.deliver()
+        msg, dup = db.add(pathname, sync_flags=False)
+        assert db.find(msgid)
+        dup = db.remove(pathname)
+        with pytest.raises(LookupError):
+            db.find(msgid)
+
+    def test_remove_message_str(self, db, maildir):
+        msgid, pathname = maildir.deliver()
+        msg, dup = db.add(pathname, sync_flags=False)
+        assert db.find(msgid)
+        dup = db.remove(str(pathname))
+        with pytest.raises(LookupError):
+            db.find(msgid)
+
+    def test_remove_message_bytes(self, db, maildir):
+        msgid, pathname = maildir.deliver()
+        msg, dup = db.add(pathname, sync_flags=False)
+        assert db.find(msgid)
+        dup = db.remove(os.fsencode(bytes(pathname)))
+        with pytest.raises(LookupError):
+            db.find(msgid)
+
+    def test_find_message(self, db, maildir):
+        msgid, pathname = maildir.deliver()
+        msg0, dup = db.add(pathname, sync_flags=False)
+        msg1 = db.find(msgid)
+        assert isinstance(msg1, message.Message)
+        assert msg1.messageid == msgid == msg0.messageid
+        assert msg1.path == pathname == msg0.path
+
+    def test_find_message_notfound(self, db):
+        with pytest.raises(LookupError):
+            db.find('foo')
+
+    def test_get_message(self, db, maildir):
+        msgid, pathname = maildir.deliver()
+        msg0, _ = db.add(pathname, sync_flags=False)
+        msg1 = db.get(pathname)
+        assert isinstance(msg1, message.Message)
+        assert msg1.messageid == msgid == msg0.messageid
+        assert msg1.path == pathname == msg0.path
+
+    def test_get_message_str(self, db, maildir):
+        msgid, pathname = maildir.deliver()
+        db.add(pathname, sync_flags=False)
+        msg = db.get(str(pathname))
+        assert msg.messageid == msgid
+
+    def test_get_message_bytes(self, db, maildir):
+        msgid, pathname = maildir.deliver()
+        db.add(pathname, sync_flags=False)
+        msg = db.get(os.fsencode(bytes(pathname)))
+        assert msg.messageid == msgid
+
+
+class TestTags:
+    # We just want to test this behaves like a set at a hight level.
+    # The set semantics are tested in detail in the test_tags module.
+
+    def test_type(self, db):
+        assert isinstance(db.tags, collections.abc.Set)
+
+    def test_none(self, db):
+        itags = iter(db.tags)
+        with pytest.raises(StopIteration):
+            next(itags)
+        assert len(db.tags) == 0
+        assert not db.tags
+
+    def test_some(self, db, maildir):
+        _, pathname = maildir.deliver()
+        msg, _ = db.add(pathname, sync_flags=False)
+        msg.tags.add('hello')
+        itags = iter(db.tags)
+        assert next(itags) == 'hello'
+        with pytest.raises(StopIteration):
+            next(itags)
+        assert 'hello' in msg.tags
+
+    def test_cache(self, db):
+        assert db.tags is db.tags
+
+    def test_iters(self, db):
+        i1 = iter(db.tags)
+        i2 = iter(db.tags)
+        assert i1 is not i2
+
+
+class TestQuery:
+
+    @pytest.fixture
+    def db(self, maildir, notmuch):
+        """Return a read-only notdb.Database.
+
+        The database will have 3 messages, 2 threads.
+        """
+        msgid, _ = maildir.deliver(body='foo')
+        maildir.deliver(body='bar')
+        maildir.deliver(body='baz',
+                        headers=[('In-Reply-To', '<{}>'.format(msgid))])
+        notmuch('new')
+        with dbmod.Database(maildir.path, 'rw') as db:
+            yield db
+
+    def test_count_messages(self, db):
+        assert db.count_messages('*') == 3
+
+    def test_messages_type(self, db):
+        msgs = db.messages('*')
+        assert isinstance(msgs, collections.abc.Iterator)
+
+    def test_message_no_results(self, db):
+        msgs = db.messages('not_a_matching_query')
+        with pytest.raises(StopIteration):
+            next(msgs)
+
+    def test_message_match(self, db):
+        msgs = db.messages('*')
+        msg = next(msgs)
+        assert isinstance(msg, notdb.Message)
+
+    def test_count_threads(self, db):
+        assert db.count_threads('*') == 2
+
+    def test_threads_type(self, db):
+        threads = db.threads('*')
+        assert isinstance(threads, collections.abc.Iterator)
+
+    def test_threads_no_match(self, db):
+        threads = db.threads('not_a_matching_query')
+        with pytest.raises(StopIteration):
+            next(threads)
+
+    def test_threads_match(self, db):
+        threads = db.threads('*')
+        thread = next(threads)
+        assert isinstance(thread, notdb.Thread)
diff --git a/bindings/python-cffi/tests/test_message.py b/bindings/python-cffi/tests/test_message.py
new file mode 100644
index 00000000..56d06f34
--- /dev/null
+++ b/bindings/python-cffi/tests/test_message.py
@@ -0,0 +1,226 @@
+import collections.abc
+import time
+import pathlib
+
+import pytest
+
+import notdb
+
+
+class TestMessage:
+    MaildirMsg = collections.namedtuple('MaildirMsg', ['msgid', 'path'])
+
+    @pytest.fixture
+    def maildir_msg(self, maildir):
+        msgid, path = maildir.deliver()
+        return self.MaildirMsg(msgid, path)
+
+    @pytest.fixture
+    def db(self, maildir):
+        with notdb.Database.create(maildir.path) as db:
+            yield db
+
+    @pytest.fixture
+    def msg(self, db, maildir_msg):
+        msg, dup = db.add(maildir_msg.path, sync_flags=False)
+        yield msg
+
+    def test_type(self, msg):
+        assert isinstance(msg, notdb.NotmuchObject)
+        assert isinstance(msg, notdb.Message)
+
+    def test_alive(self, msg):
+        assert msg.alive
+
+    def test_hash(self, msg):
+        assert hash(msg)
+
+    def test_eq(self, db, msg):
+        copy = db.get(msg.path)
+        assert msg == copy
+
+    def test_messageid_type(self, msg):
+        assert isinstance(msg.messageid, str)
+        assert isinstance(msg.messageid, notdb.BinString)
+        assert isinstance(bytes(msg.messageid), bytes)
+
+    def test_messageid(self, msg, maildir_msg):
+        assert msg.messageid == maildir_msg.msgid
+
+    def test_messageid_find(self, db, msg):
+        copy = db.find(msg.messageid)
+        assert msg.messageid == copy.messageid
+
+    def test_threadid_type(self, msg):
+        assert isinstance(msg.threadid, str)
+        assert isinstance(msg.threadid, notdb.BinString)
+        assert isinstance(bytes(msg.threadid), bytes)
+
+    def test_path_type(self, msg):
+        assert isinstance(msg.path, pathlib.Path)
+
+    def test_path(self, msg, maildir_msg):
+        assert msg.path == maildir_msg.path
+
+    def test_pathb_type(self, msg):
+        assert isinstance(msg.pathb, bytes)
+
+    def test_pathb(self, msg, maildir_msg):
+        assert msg.path == maildir_msg.path
+
+    def test_filenames_type(self, msg):
+        ifn = msg.filenames()
+        assert isinstance(ifn, collections.abc.Iterator)
+
+    def test_filenames(self, msg):
+        ifn = msg.filenames()
+        fn = next(ifn)
+        assert fn == msg.path
+        assert isinstance(fn, pathlib.Path)
+        with pytest.raises(StopIteration):
+            next(ifn)
+        assert list(msg.filenames()) == [msg.path]
+
+    def test_filenamesb_type(self, msg):
+        ifn = msg.filenamesb()
+        assert isinstance(ifn, collections.abc.Iterator)
+
+    def test_filenamesb(self, msg):
+        ifn = msg.filenamesb()
+        fn = next(ifn)
+        assert fn == msg.pathb
+        assert isinstance(fn, bytes)
+        with pytest.raises(StopIteration):
+            next(ifn)
+        assert list(msg.filenamesb()) == [msg.pathb]
+
+    def test_ghost_no(self, msg):
+        assert not msg.ghost
+
+    def test_date(self, msg):
+        # XXX Someone seems to treat things as local time instead of
+        #     UTC or the other way around.
+        now = int(time.time())
+        assert abs(now - msg.date) < 3600*24
+
+    def test_header(self, msg):
+        assert msg.header('from') == 'src@example.com'
+
+    def test_header_not_present(self, msg):
+        with pytest.raises(LookupError):
+            msg.header('foo')
+
+    def test_freeze(self, msg):
+        with msg.frozen():
+            msg.tags.add('foo')
+            msg.tags.add('bar')
+            msg.tags.discard('foo')
+        assert 'foo' not in msg.tags
+        assert 'bar' in msg.tags
+
+    def test_freeze_err(self, msg):
+        msg.tags.add('foo')
+        try:
+            with msg.frozen():
+                msg.tags.clear()
+                raise Exception('oops')
+        except Exception:
+            assert 'foo' in msg.tags
+        else:
+            pytest.fail('Context manager did not raise')
+
+    def test_replies_type(self, msg):
+        assert isinstance(msg.replies(), collections.abc.Iterator)
+
+    def test_replies(self, msg):
+        with pytest.raises(StopIteration):
+            next(msg.replies())
+
+
+class TestProperties:
+
+    @pytest.fixture
+    def props(self, maildir):
+        msgid, path = maildir.deliver()
+        with notdb.Database.create(maildir.path) as db:
+            msg, dup = db.add(path, sync_flags=False)
+            yield msg.properties
+
+    def test_type(self, props):
+        assert isinstance(props, collections.abc.MutableMapping)
+
+    def test_add_single(self, props):
+        props['foo'] = 'bar'
+        assert props['foo'] == 'bar'
+        props.add('bar', 'baz')
+        assert props['bar'] == 'baz'
+
+    def test_add_dup(self, props):
+        props.add('foo', 'bar')
+        props.add('foo', 'baz')
+        assert props['foo'] == 'bar'
+        assert (set(props.getall('foo', exact=True))
+                == {('foo', 'bar'), ('foo', 'baz')})
+
+    def test_len(self, props):
+        props.add('foo', 'a')
+        props.add('foo', 'b')
+        props.add('bar', 'a')
+        assert len(props) == 3
+        assert len(props.keys()) == 2
+        assert len(props.values()) == 2
+        assert len(props.items()) == 3
+
+    def test_del(self, props):
+        props.add('foo', 'a')
+        props.add('foo', 'b')
+        del props['foo']
+        with pytest.raises(KeyError):
+            props['foo']
+
+    def test_remove(self, props):
+        props.add('foo', 'a')
+        props.add('foo', 'b')
+        props.remove('foo', 'a')
+        assert props['foo'] == 'b'
+
+    def test_view_abcs(self, props):
+        assert isinstance(props.keys(), collections.abc.KeysView)
+        assert isinstance(props.values(), collections.abc.ValuesView)
+        assert isinstance(props.items(), collections.abc.ItemsView)
+
+    def test_pop(self, props):
+        props.add('foo', 'a')
+        props.add('foo', 'b')
+        val = props.pop('foo')
+        assert val == 'a'
+
+    def test_pop_default(self, props):
+        with pytest.raises(KeyError):
+            props.pop('foo')
+        assert props.pop('foo', 'default') == 'default'
+
+    def test_popitem(self, props):
+        props.add('foo', 'a')
+        assert props.popitem() == ('foo', 'a')
+        with pytest.raises(KeyError):
+            props.popitem()
+
+    def test_clear(self, props):
+        props.add('foo', 'a')
+        props.clear()
+        assert len(props) == 0
+
+    def test_getall(self, props):
+        props.add('foo', 'a')
+        assert set(props.getall('foo')) == {('foo', 'a')}
+
+    def test_getall_prefix(self, props):
+        props.add('foo', 'a')
+        props.add('foobar', 'b')
+        assert set(props.getall('foo')) == {('foo', 'a'), ('foobar', 'b')}
+
+    def test_getall_exact(self, props):
+        props.add('foo', 'a')
+        props.add('foobar', 'b')
+        assert set(props.getall('foo', exact=True)) == {('foo', 'a')}
diff --git a/bindings/python-cffi/tests/test_tags.py b/bindings/python-cffi/tests/test_tags.py
new file mode 100644
index 00000000..0cb42d89
--- /dev/null
+++ b/bindings/python-cffi/tests/test_tags.py
@@ -0,0 +1,177 @@
+"""Tests for the behaviour of immutable and mutable tagsets.
+
+This module tests the Pythonic behaviour of the sets.
+"""
+
+import collections
+import subprocess
+import textwrap
+
+import pytest
+
+from notdb import _database as database
+from notdb import _tags as tags
+
+
+class TestImmutable:
+
+    @pytest.fixture
+    def tagset(self, maildir, notmuch):
+        """An non-empty immutable tagset.
+
+        This will have the default new mail tags: inbox, unread.
+        """
+        maildir.deliver()
+        notmuch('new')
+        with database.Database(maildir.path) as db:
+            yield db.tags
+
+    def test_type(self, tagset):
+        assert isinstance(tagset, tags.ImmutableTagSet)
+        assert isinstance(tagset, collections.abc.Set)
+
+    def test_hash(self, tagset, maildir, notmuch):
+        h0 = hash(tagset)
+        notmuch('tag', '+foo', '*')
+        with database.Database(maildir.path) as db:
+            h1 = hash(db.tags)
+        assert h0 != h1
+
+    def test_eq(self, tagset):
+        assert tagset == tagset
+
+    def test_neq(self, tagset, maildir, notmuch):
+        notmuch('tag', '+foo', '*')
+        with database.Database(maildir.path) as db:
+            assert tagset != db.tags
+
+    def test_contains(self, tagset):
+        print(tuple(tagset))
+        assert 'unread' in tagset
+        assert 'foo' not in tagset
+
+    def test_iter(self, tagset):
+        expected = sorted(['unread', 'inbox'])
+        found = []
+        for tag in tagset:
+            assert isinstance(tag, str)
+            found.append(tag)
+        assert expected == sorted(found)
+
+    def test_special_iter(self, tagset):
+        expected = sorted([b'unread', b'inbox'])
+        found = []
+        for tag in tagset.iter():
+            assert isinstance(tag, bytes)
+            found.append(tag)
+        assert expected == sorted(found)
+
+    def test_special_iter_codec(self, tagset):
+        for tag in tagset.iter(encoding='ascii', errors='surrogateescape'):
+            assert isinstance(tag, str)
+
+    def test_len(self, tagset):
+        assert len(tagset) == 2
+
+    def test_and(self, tagset):
+        common = tagset & {'unread'}
+        assert isinstance(common, set)
+        assert isinstance(common, collections.abc.Set)
+        assert common == {'unread'}
+
+    def test_or(self, tagset):
+        res = tagset | {'foo'}
+        assert isinstance(res, set)
+        assert isinstance(res, collections.abc.Set)
+        assert res == {'unread', 'inbox', 'foo'}
+
+    def test_sub(self, tagset):
+        res = tagset - {'unread'}
+        assert isinstance(res, set)
+        assert isinstance(res, collections.abc.Set)
+        assert res == {'inbox'}
+
+    def test_rsub(self, tagset):
+        res = {'foo', 'unread'} - tagset
+        assert isinstance(res, set)
+        assert isinstance(res, collections.abc.Set)
+        assert res == {'foo'}
+
+    def test_xor(self, tagset):
+        res = tagset ^ {'unread', 'foo'}
+        assert isinstance(res, set)
+        assert isinstance(res, collections.abc.Set)
+        assert res == {'inbox', 'foo'}
+
+    def test_rxor(self, tagset):
+        res = {'unread', 'foo'} ^ tagset
+        assert isinstance(res, set)
+        assert isinstance(res, collections.abc.Set)
+        assert res == {'inbox', 'foo'}
+
+
+class TestMutableTagset:
+
+    @pytest.fixture
+    def tagset(self, maildir, notmuch):
+        """An non-empty mutable tagset.
+
+        This will have the default new mail tags: inbox, unread.
+        """
+        _, pathname = maildir.deliver()
+        notmuch('new')
+        with database.Database(maildir.path,
+                               mode=database.Mode.READ_WRITE) as db:
+            msg = db.get(pathname)
+            yield msg.tags
+
+    def test_type(self, tagset):
+        assert isinstance(tagset, collections.abc.MutableSet)
+        assert isinstance(tagset, tags.MutableTagSet)
+
+    def test_hash(self, tagset):
+        assert not isinstance(tagset, collections.abc.Hashable)
+        with pytest.raises(TypeError):
+            hash(tagset)
+
+    def test_add(self, tagset):
+        assert 'foo' not in tagset
+        tagset.add('foo')
+        assert 'foo' in tagset
+
+    def test_discard(self, tagset):
+        assert 'inbox' in tagset
+        tagset.discard('inbox')
+        assert 'inbox' not in tagset
+
+    def test_discard_not_present(self, tagset):
+        assert 'foo' not in tagset
+        tagset.discard('foo')
+
+    def test_clear(self, tagset):
+        assert len(tagset) > 0
+        tagset.clear()
+        assert len(tagset) == 0
+
+    def test_from_maildir_flags(self, maildir, notmuch):
+        _, pathname = maildir.deliver(flagged=True)
+        notmuch('new')
+        with database.Database(maildir.path,
+                               mode=database.Mode.READ_WRITE) as db:
+            msg = db.get(pathname)
+            msg.tags.discard('flagged')
+            msg.tags.from_maildir_flags()
+            assert 'flagged' in msg.tags
+
+    def test_to_maildir_flags(self, maildir, notmuch):
+        _, pathname = maildir.deliver(flagged=True)
+        notmuch('new')
+        with database.Database(maildir.path,
+                               mode=database.Mode.READ_WRITE) as db:
+            msg = db.get(pathname)
+            flags = msg.path.name.split(',')[-1]
+            assert 'F' in flags
+            msg.tags.discard('flagged')
+            msg.tags.to_maildir_flags()
+            flags = msg.path.name.split(',')[-1]
+            assert 'F' not in flags
diff --git a/bindings/python-cffi/tests/test_thread.py b/bindings/python-cffi/tests/test_thread.py
new file mode 100644
index 00000000..366bd8a5
--- /dev/null
+++ b/bindings/python-cffi/tests/test_thread.py
@@ -0,0 +1,102 @@
+import collections.abc
+import time
+
+import pytest
+
+import notdb
+
+
+@pytest.fixture
+def thread(maildir, notmuch):
+    """Return a single thread with one matched message."""
+    msgid, _ = maildir.deliver(body='foo')
+    maildir.deliver(body='bar',
+                    headers=[('In-Reply-To', '<{}>'.format(msgid))])
+    notmuch('new')
+    with notdb.Database(maildir.path) as db:
+        yield next(db.threads('foo'))
+
+
+def test_type(thread):
+    assert isinstance(thread, notdb.Thread)
+    assert isinstance(thread, collections.abc.Iterable)
+
+
+def test_threadid(thread):
+    assert isinstance(thread.threadid, notdb.BinString)
+    assert thread.threadid
+
+
+def test_len(thread):
+    assert len(thread) == 2
+
+
+def test_toplevel_type(thread):
+    assert isinstance(thread.toplevel(), collections.abc.Iterator)
+
+
+def test_toplevel(thread):
+    msgs = thread.toplevel()
+    assert isinstance(next(msgs), notdb.Message)
+    with pytest.raises(StopIteration):
+        next(msgs)
+
+
+def test_toplevel_reply(thread):
+    msg = next(thread.toplevel())
+    assert isinstance(next(msg.replies()), notdb.Message)
+
+
+def test_iter(thread):
+    msgs = list(iter(thread))
+    assert len(msgs) == len(thread)
+    for msg in msgs:
+        assert isinstance(msg, notdb.Message)
+
+
+def test_matched(thread):
+    assert thread.matched == 1
+
+
+def test_authors_type(thread):
+    assert isinstance(thread.authors, notdb.BinString)
+
+
+def test_authors(thread):
+    assert thread.authors == 'src@example.com'
+
+
+def test_subject(thread):
+    assert thread.subject == 'Test mail'
+
+
+def test_first(thread):
+    # XXX Someone seems to treat things as local time instead of
+    #     UTC or the other way around.
+    now = int(time.time())
+    assert abs(now - thread.first) < 3600*24
+
+
+def test_last(thread):
+    # XXX Someone seems to treat things as local time instead of
+    #     UTC or the other way around.
+    now = int(time.time())
+    assert abs(now - thread.last) < 3600*24
+
+
+def test_first_last(thread):
+    # Sadly we only have second resolution so these will always be the
+    # same time in our tests.
+    assert thread.first <= thread.last
+
+
+def test_tags_type(thread):
+    assert isinstance(thread.tags, notdb.ImmutableTagSet)
+
+
+def test_tags_cache(thread):
+    assert thread.tags is thread.tags
+
+
+def test_tags(thread):
+    assert 'inbox' in thread.tags
diff --git a/bindings/python-cffi/tox.ini b/bindings/python-cffi/tox.ini
new file mode 100644
index 00000000..d6b87987
--- /dev/null
+++ b/bindings/python-cffi/tox.ini
@@ -0,0 +1,16 @@
+[pytest]
+minversion = 3.0
+addopts = -ra --cov=notdb --cov=tests
+
+[tox]
+envlist = py35,py36,py37,pypy35,pypy36
+
+[testenv]
+deps =
+     cffi
+     pytest
+     pytest-cov
+commands = pytest --cov={envsitepackagesdir}/notdb {posargs}
+
+[testenv:pypy35]
+basepython = pypy3.5
diff --git a/test/T391-pytest.sh b/test/T391-pytest.sh
new file mode 100755
index 00000000..3729417e
--- /dev/null
+++ b/test/T391-pytest.sh
@@ -0,0 +1,15 @@
+#!/usr/bin/env bash
+test_description="python bindings (pytest)"
+. $(dirname "$0")/test-lib.sh || exit 1
+
+test_require_external_prereq ${NOTMUCH_PYTHON}
+
+# for bin in ${NOTMUCH_PYTEST_PYTHONS}; do
+# test_begin_subtest "pytest ($bin)"
+# PYTHONPATH="$NOTMUCH_SRCDIR/bindings/python${PYTHONPATH:+:$PYTHONPATH}" \
+#           test_expect_success "$bin -m pytest $NOTMUCH_SRCDIR/bindings/python"
+# done
+PYTHONPATH="$NOTMUCH_SRCDIR/bindings/python${PYTHONPATH:+:$PYTHONPATH}" \
+          ${NOTMUCH_PYTHON} -m pytest $NOTMUCH_SRCDIR/bindings/python-cffi
+
+test_done
-- 
2.23.0

^ permalink raw reply related	[flat|nested] 19+ messages in thread

* Re: Python3 cffi bindings
  2019-10-08 21:03 Python3 cffi bindings Floris Bruynooghe
  2019-10-08 21:03 ` [PATCH] Introduce CFFI-based python bindings Floris Bruynooghe
@ 2019-10-08 22:24 ` David Bremner
  2019-10-09 18:34   ` Floris Bruynooghe
  2019-10-14 12:40 ` David Bremner
  2019-11-04 10:27 ` Gaute Hope
  3 siblings, 1 reply; 19+ messages in thread
From: David Bremner @ 2019-10-08 22:24 UTC (permalink / raw)
  To: Floris Bruynooghe, notmuch

Floris Bruynooghe <flub@devork.be> writes:
> Anyway, I found the code, checked things work, updated tests on new
> python versions, added a very basic intergration with the test
> framework and squashed the commits.  Otherwise the attached patch
> is just a plain dump of the current state so interested people have
> at least a copy of the code again which can be made to work.

I think you missed the attachement. Other than that, sounds interesting ;).

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Python3 cffi bindings
  2019-10-08 22:24 ` Python3 cffi bindings David Bremner
@ 2019-10-09 18:34   ` Floris Bruynooghe
  0 siblings, 0 replies; 19+ messages in thread
From: Floris Bruynooghe @ 2019-10-09 18:34 UTC (permalink / raw)
  To: David Bremner, notmuch

On Tue 08 Oct 2019 at 19:24 -0300, David Bremner wrote:

> Floris Bruynooghe <flub@devork.be> writes:
>> Anyway, I found the code, checked things work, updated tests on new
>> python versions, added a very basic intergration with the test
>> framework and squashed the commits.  Otherwise the attached patch
>> is just a plain dump of the current state so interested people have
>> at least a copy of the code again which can be made to work.
>
> I think you missed the attachement. Other than that, sounds interesting ;).

I used git send-email as per the contributing nodes, so it was supposed
to be a followup email.  It does seem like I got a message back saying
the patch mail is being held in the moderator queue as it's too large...
Perhaps you have moderation powers?

Cheers,
Floris

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Python3 cffi bindings
  2019-10-08 21:03 Python3 cffi bindings Floris Bruynooghe
  2019-10-08 21:03 ` [PATCH] Introduce CFFI-based python bindings Floris Bruynooghe
  2019-10-08 22:24 ` Python3 cffi bindings David Bremner
@ 2019-10-14 12:40 ` David Bremner
  2019-10-14 12:42   ` David Bremner
  2019-11-04 10:27 ` Gaute Hope
  3 siblings, 1 reply; 19+ messages in thread
From: David Bremner @ 2019-10-14 12:40 UTC (permalink / raw)
  To: Floris Bruynooghe, notmuch

Floris Bruynooghe <flub@devork.be> writes:

> IIRC this probably wants to be renamed to "notmuch2" instead of
> "notdb".  Otherwise I'm pretty sure this doesn't cover all the
> current features either.

Yes, I think notmuch2 was the least disliked suggestion. 

> So maybe this can be used as a start to figure out how to merge
> this if there's still an interest in this.

I'm interested. The blocker for me at the moment is getting the tests
working without tox / venvs.  I'm hoping we can test with the system
python and the built, but not installed module. I guess we need to build
it so that the C extension part is loaded. The shim in
T391-python-cffi.sh doesn't work for me, it doesn't manage to set
PYTHONPATH so that notdb is importable.

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Python3 cffi bindings
  2019-10-14 12:40 ` David Bremner
@ 2019-10-14 12:42   ` David Bremner
  2019-10-17 17:35     ` Floris Bruynooghe
  0 siblings, 1 reply; 19+ messages in thread
From: David Bremner @ 2019-10-14 12:42 UTC (permalink / raw)
  To: Floris Bruynooghe, notmuch

David Bremner <david@tethera.net> writes:

> The shim in
> T391-python-cffi.sh doesn't work for me, it doesn't manage to set
> PYTHONPATH so that notdb is importable.

I should have mentioned that if I manually set python path with
something like

$ PYTHONPATH=`pwd`/build/lib.linux-x86_64-3.7:$PYTHONPATH pytest-3

it works OK.  Is there a simple/reliable way of calculating the path
lib.linux-x86_64-3.7?

d

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Python3 cffi bindings
  2019-10-14 12:42   ` David Bremner
@ 2019-10-17 17:35     ` Floris Bruynooghe
  2019-10-20 12:22       ` David Bremner
  0 siblings, 1 reply; 19+ messages in thread
From: Floris Bruynooghe @ 2019-10-17 17:35 UTC (permalink / raw)
  To: David Bremner, notmuch

On Mon 14 Oct 2019 at 09:42 -0300, David Bremner wrote:

> David Bremner <david@tethera.net> writes:
>
>> The shim in
>> T391-python-cffi.sh doesn't work for me, it doesn't manage to set
>> PYTHONPATH so that notdb is importable.

Ah yes, I tested this shim while activating a venv with the extension
installed using `pip -e .`.

> I should have mentioned that if I manually set python path with
> something like
>
> $ PYTHONPATH=`pwd`/build/lib.linux-x86_64-3.7:$PYTHONPATH pytest-3
>
> it works OK.  Is there a simple/reliable way of calculating the path
> lib.linux-x86_64-3.7?

It is possible to run this without installing, but it does need a build
step since cffi (in the mode used - which is the recommended mode) needs
to build an extension module.  I did something like this, using my
debian testing system-installed python

$ export PYTHONPATH=$(pwd)/bindings/python-cffi
$ pushd bindings/python-cffi
$ python3 notdb/_build.py  # creates notdb/_capi.cpython-37m-x86_64-linux-gnu.so
$ popd
$ pushd test
$ ./T391-pytest.sh

Does that more or less work?  One problem with this is that it will pick
up the system-wide installed notmuch though.  I guess the way to change
this is by tweaking CFLAGS=-I... LDFLAGS=-L... or so when building?  But
than you also have the whole RPATH/LD_LIBRARY_PATH stuff going on as
well.  Does notmuch abstract any of this away already for it's test
suite?

Cheers,
Floris

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Python3 cffi bindings
  2019-10-17 17:35     ` Floris Bruynooghe
@ 2019-10-20 12:22       ` David Bremner
  2019-10-22 16:32         ` David Bremner
  0 siblings, 1 reply; 19+ messages in thread
From: David Bremner @ 2019-10-20 12:22 UTC (permalink / raw)
  To: Floris Bruynooghe, notmuch

[-- Attachment #1: Type: text/plain, Size: 2120 bytes --]

Floris Bruynooghe <flub@devork.be> writes:
>
> It is possible to run this without installing, but it does need a build
> step since cffi (in the mode used - which is the recommended mode) needs
> to build an extension module.  I did something like this, using my
> debian testing system-installed python
>
> $ export PYTHONPATH=$(pwd)/bindings/python-cffi
> $ pushd bindings/python-cffi
> $ python3 notdb/_build.py  # creates notdb/_capi.cpython-37m-x86_64-linux-gnu.so
> $ popd
> $ pushd test
> $ ./T391-pytest.sh

Yes, I think I arrived at a similar place, except

1) using "python3 setup.py --build-lib build/stage" to build. I'm not
sure which is better, I think it will depend a bit on when we try to get
out of tree builds working. It is a bit nicer to have the build output
out of tree, but then I have to copy the tests.

2) instead of changing PYTHONPATH, use "python3 -m pytest", which picks
up the module in the current directory. 

> Does that more or less work?  One problem with this is that it will pick
> up the system-wide installed notmuch though.  I guess the way to change
> this is by tweaking CFLAGS=-I... LDFLAGS=-L... or so when building?  But
> than you also have the whole RPATH/LD_LIBRARY_PATH stuff going on as
> well.  Does notmuch abstract any of this away already for it's test
> suite?

The LD_LIBRARY_PATH is already set by the test harness, as is PATH (to
find notmuch). It looks like your function notmuch is not respecting
PATH (see attached log). if I hack something like

diff --git a/bindings/python-cffi/tests/conftest.py b/bindings/python-cffi/tests/conftest.py
index 1b7bbc35..ac17397c 100644
--- a/bindings/python-cffi/tests/conftest.py
+++ b/bindings/python-cffi/tests/conftest.py
@@ -31,7 +31,7 @@ def notmuch(maildir):
         accidentally do this in the unittests.
         """
         cfg_fname = maildir.path / 'notmuch-config'
-        cmd = ['notmuch'] + list(args)
+        cmd = ['../../notmuch'] + list(args)
         print('Invoking: {}'.format(' '.join(cmd)))
         proc = subprocess.run(cmd,

then the tests pass, but this is obviously not a good solution.


[-- Attachment #2: test.output.xz --]
[-- Type: application/x-xz, Size: 5512 bytes --]

^ permalink raw reply related	[flat|nested] 19+ messages in thread

* Re: Python3 cffi bindings
  2019-10-20 12:22       ` David Bremner
@ 2019-10-22 16:32         ` David Bremner
  2019-10-25  9:57           ` Floris Bruynooghe
  0 siblings, 1 reply; 19+ messages in thread
From: David Bremner @ 2019-10-22 16:32 UTC (permalink / raw)
  To: Floris Bruynooghe, notmuch

David Bremner <david@tethera.net> writes:

> The LD_LIBRARY_PATH is already set by the test harness, as is PATH (to
> find notmuch). It looks like your function notmuch is not respecting
> PATH (see attached log). if I hack something like
>
> diff --git a/bindings/python-cffi/tests/conftest.py b/bindings/python-cffi/tests/conftest.py
> index 1b7bbc35..ac17397c 100644
> --- a/bindings/python-cffi/tests/conftest.py
> +++ b/bindings/python-cffi/tests/conftest.py
> @@ -31,7 +31,7 @@ def notmuch(maildir):
>          accidentally do this in the unittests.
>          """
>          cfg_fname = maildir.path / 'notmuch-config'
> -        cmd = ['notmuch'] + list(args)
> +        cmd = ['../../notmuch'] + list(args)
>          print('Invoking: {}'.format(' '.join(cmd)))
>          proc = subprocess.run(cmd,
>

I think I figured it out. Your 'run' function completely overrides the
environment. But just adding PATH back seems to do the trick. I'm not
sure if this is the most idomatic change, but it works:

diff --git a/bindings/python-cffi/tests/conftest.py b/bindings/python-cffi/tests/conftest.py
index 1b7bbc35..6a81aa18 100644
--- a/bindings/python-cffi/tests/conftest.py
+++ b/bindings/python-cffi/tests/conftest.py
@@ -33,9 +33,11 @@ def notmuch(maildir):
         cfg_fname = maildir.path / 'notmuch-config'
         cmd = ['notmuch'] + list(args)
         print('Invoking: {}'.format(' '.join(cmd)))
                               proc = subprocess.run(cmd,
                               timeout=5,
-                              env={'NOTMUCH_CONFIG': str(cfg_fname)})
+                              env={'PATH':os.environ["PATH"],'NOTMUCH_CONFIG': str(cfg_fname)})
         proc.check_returncode()
     return run
 

^ permalink raw reply related	[flat|nested] 19+ messages in thread

* Re: Python3 cffi bindings
  2019-10-22 16:32         ` David Bremner
@ 2019-10-25  9:57           ` Floris Bruynooghe
  0 siblings, 0 replies; 19+ messages in thread
From: Floris Bruynooghe @ 2019-10-25  9:57 UTC (permalink / raw)
  To: David Bremner, notmuch

On Tue 22 Oct 2019 at 13:32 -0300, David Bremner wrote:

> David Bremner <david@tethera.net> writes:
>
>> The LD_LIBRARY_PATH is already set by the test harness, as is PATH (to
>> find notmuch). It looks like your function notmuch is not respecting
>> PATH (see attached log). if I hack something like
>>
>> diff --git a/bindings/python-cffi/tests/conftest.py b/bindings/python-cffi/tests/conftest.py
>> index 1b7bbc35..ac17397c 100644
>> --- a/bindings/python-cffi/tests/conftest.py
>> +++ b/bindings/python-cffi/tests/conftest.py
>> @@ -31,7 +31,7 @@ def notmuch(maildir):
>>          accidentally do this in the unittests.
>>          """
>>          cfg_fname = maildir.path / 'notmuch-config'
>> -        cmd = ['notmuch'] + list(args)
>> +        cmd = ['../../notmuch'] + list(args)
>>          print('Invoking: {}'.format(' '.join(cmd)))
>>          proc = subprocess.run(cmd,
>>
>
> I think I figured it out. Your 'run' function completely overrides the
> environment. But just adding PATH back seems to do the trick. I'm not
> sure if this is the most idomatic change, but it works:
>
> diff --git a/bindings/python-cffi/tests/conftest.py b/bindings/python-cffi/tests/conftest.py
> index 1b7bbc35..6a81aa18 100644
> --- a/bindings/python-cffi/tests/conftest.py
> +++ b/bindings/python-cffi/tests/conftest.py
> @@ -33,9 +33,11 @@ def notmuch(maildir):
>          cfg_fname = maildir.path / 'notmuch-config'
>          cmd = ['notmuch'] + list(args)
>          print('Invoking: {}'.format(' '.join(cmd)))
>                                proc = subprocess.run(cmd,
>                                timeout=5,
> -                              env={'NOTMUCH_CONFIG': str(cfg_fname)})
> +                              env={'PATH':os.environ["PATH"],'NOTMUCH_CONFIG': str(cfg_fname)})
>          proc.check_returncode()
>      return run
>  

This seems reasonable, perhaps even a "env = os.environ.copy();
env['NOTMUCH_CONFIG'] = src(cfg_fname)" is better here so that
LD_LIBRARY_PATH and anything else is kept around.

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Python3 cffi bindings
  2019-10-08 21:03 Python3 cffi bindings Floris Bruynooghe
                   ` (2 preceding siblings ...)
  2019-10-14 12:40 ` David Bremner
@ 2019-11-04 10:27 ` Gaute Hope
  2019-11-16 16:44   ` David Bremner
  2019-11-17 17:14   ` Floris Bruynooghe
  3 siblings, 2 replies; 19+ messages in thread
From: Gaute Hope @ 2019-11-04 10:27 UTC (permalink / raw)
  To: Floris Bruynooghe; +Cc: notmuch

Hi,

I just checked out the wip/cffi branch on git.notmuch.org with the
purpose of porting Lieer (https://github.com/gauteh/lieer). There
seems to be some missing functionality: `Database.get_directory()`
specifically. I also ran into a couple of warning when building
(included below).

Thanks for your work.

By the way, it does not seem that the API is very far from the
previous python API. If it is close enough, perhaps it is possible to
get away with a bug version bump in the bindings rather than creating
a new package. I understand the need for a new package, but it would
be nice if we could avoid the future confusion of two python binding
packages (if at all possible).

Regards, Gaute

~/dev/notm/notmuch/bindings/python-cffi wip/cffi
notm ❯ python setup.py build
warning: no previously-included files found matching 'setup.pyc'
warning: no previously-included files matching 'yacctab.*' found under
directory 'tests'
warning: no previously-included files matching 'lextab.*' found under
directory 'tests'
warning: no previously-included files matching 'yacctab.*' found under
directory 'examples'
warning: no previously-included files matching 'lextab.*' found under
directory 'examples'
zip_safe flag not set; analyzing archive contents...
pycparser.ply.__pycache__.lex.cpython-37: module references __file__
pycparser.ply.__pycache__.lex.cpython-37: module MAY be using
inspect.getsourcefile
pycparser.ply.__pycache__.yacc.cpython-37: module references __file__
pycparser.ply.__pycache__.yacc.cpython-37: module MAY be using
inspect.getsourcefile
pycparser.ply.__pycache__.yacc.cpython-37: module MAY be using inspect.stack
pycparser.ply.__pycache__.ygen.cpython-37: module references __file__

Installed /home/gauteh/dev/notm/notmuch/bindings/python-cffi/.eggs/pycparser-2.19-py3.7.egg
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.7
creating build/lib.linux-x86_64-3.7/notdb
copying notdb/_message.py -> build/lib.linux-x86_64-3.7/notdb
copying notdb/__init__.py -> build/lib.linux-x86_64-3.7/notdb
copying notdb/_database.py -> build/lib.linux-x86_64-3.7/notdb
copying notdb/_errors.py -> build/lib.linux-x86_64-3.7/notdb
copying notdb/_tags.py -> build/lib.linux-x86_64-3.7/notdb
copying notdb/_thread.py -> build/lib.linux-x86_64-3.7/notdb
copying notdb/_query.py -> build/lib.linux-x86_64-3.7/notdb
copying notdb/_build.py -> build/lib.linux-x86_64-3.7/notdb
copying notdb/_base.py -> build/lib.linux-x86_64-3.7/notdb
running build_ext
generating cffi module 'build/temp.linux-x86_64-3.7/notdb._capi.c'
creating build/temp.linux-x86_64-3.7
building 'notdb._capi' extension
creating build/temp.linux-x86_64-3.7/build
creating build/temp.linux-x86_64-3.7/build/temp.linux-x86_64-3.7
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3
-Wall -fPIC -I../../lib
-I/home/gauteh/.pyenv/versions/3.7.4/envs/notm/include
-I/home/gauteh/.pyenv/versions/3.7.4/include/python3.7m -c
build/temp.linux-x86_64-3.7/notdb._capi.c -o
build/temp.linux-x86_64-3.7/build/temp.linux-x86_64-3.7/notdb._capi.o
build/temp.linux-x86_64-3.7/notdb._capi.c: In function
‘_cffi_d_notmuch_database_add_message’:
build/temp.linux-x86_64-3.7/notdb._capi.c:980:3: warning:
‘notmuch_database_add_message’ is deprecated: function deprecated as
of libnotmuch 5.1 [-Wdeprecated-declarations]
   return notmuch_database_add_message(x0, x1, x2);
   ^~~~~~
In file included from build/temp.linux-x86_64-3.7/notdb._capi.c:495:0:
../../lib/notmuch.h:637:1: note: declared here
 notmuch_database_add_message (notmuch_database_t *database,
 ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
build/temp.linux-x86_64-3.7/notdb._capi.c: In function
‘_cffi_f_notmuch_database_add_message’:
build/temp.linux-x86_64-3.7/notdb._capi.c:1033:3: warning:
‘notmuch_database_add_message’ is deprecated: function deprecated as
of libnotmuch 5.1 [-Wdeprecated-declarations]
   { result = notmuch_database_add_message(x0, x1, x2); }
   ^
In file included from build/temp.linux-x86_64-3.7/notdb._capi.c:495:0:
../../lib/notmuch.h:637:1: note: declared here
 notmuch_database_add_message (notmuch_database_t *database,
 ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
gcc -pthread -shared -L/home/gauteh/.pyenv/versions/3.7.4/lib
-L/home/gauteh/.pyenv/versions/3.7.4/lib
build/temp.linux-x86_64-3.7/build/temp.linux-x86_64-3.7/notdb._capi.o
-L../../lib -lnotmuch -o
build/lib.linux-x86_64-3.7/notdb/_capi.abi3.so

On Tue, Oct 8, 2019 at 11:03 PM Floris Bruynooghe <flub@devork.be> wrote:
>
> Hi all,
>
> IIRC there was a thread in August about another attempt at bringing
> the CFFI-based bindings on board as a Python3-only version.  I
> believe there was a desire to re-name things but my searching-fu is
> failing me and I can no longer find the email thread.
>
> Anyway, I found the code, checked things work, updated tests on new
> python versions, added a very basic intergration with the test
> framework and squashed the commits.  Otherwise the attached patch
> is just a plain dump of the current state so interested people have
> at least a copy of the code again which can be made to work.
>
> IIRC this probably wants to be renamed to "notmuch2" instead of
> "notdb".  Otherwise I'm pretty sure this doesn't cover all the
> current features either.
>
> So maybe this can be used as a start to figure out how to merge
> this if there's still an interest in this.
>
> Cheers,
> Floris
>
>
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> https://notmuchmail.org/mailman/listinfo/notmuch

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Python3 cffi bindings
  2019-11-04 10:27 ` Gaute Hope
@ 2019-11-16 16:44   ` David Bremner
  2019-12-04 20:18     ` Tomi Ollila
  2019-11-17 17:14   ` Floris Bruynooghe
  1 sibling, 1 reply; 19+ messages in thread
From: David Bremner @ 2019-11-16 16:44 UTC (permalink / raw)
  To: Gaute Hope, Floris Bruynooghe; +Cc: notmuch

Gaute Hope <eg@gaute.vetsj.com> writes:

>
> By the way, it does not seem that the API is very far from the
> previous python API. If it is close enough, perhaps it is possible to
> get away with a bug version bump in the bindings rather than creating
> a new package. I understand the need for a new package, but it would
> be nice if we could avoid the future confusion of two python binding
> packages (if at all possible).
>

I'm not in a good position to judge how similar the APIs are.  It does
seem like there are at least some breaking changes, and we usually try
to make things smooth for people upgrading by deprecating interfaces
before removing them completely. On the other hand our previous concern
for supporting python pre 3.6 (I think. Maybe 3.5?) seems less and less
worrying (except maybe for people using old CentOS like things).

d

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Python3 cffi bindings
  2019-11-04 10:27 ` Gaute Hope
  2019-11-16 16:44   ` David Bremner
@ 2019-11-17 17:14   ` Floris Bruynooghe
  2019-11-17 18:01     ` Gaute Hope
  2020-10-08  8:13     ` Gaute Hope
  1 sibling, 2 replies; 19+ messages in thread
From: Floris Bruynooghe @ 2019-11-17 17:14 UTC (permalink / raw)
  To: Gaute Hope; +Cc: notmuch

Hi Gaute,

Thanks for trying this out!

On Mon 04 Nov 2019 at 11:27 +0100, Gaute Hope wrote:
> I just checked out the wip/cffi branch on git.notmuch.org with the
> purpose of porting Lieer (https://github.com/gauteh/lieer). There
> seems to be some missing functionality: `Database.get_directory()`
> specifically.

Yeah, I didn't add that yet because I don't fully understand how it
should be used.  Specifically I don't know where one might get a
pathname from to pass to .get_directory() and thus whether the API would
be cleaner to just return a reasonable directory object from whatever
location that might be.  Maybe notmuch_database_get_path() is the only
entrypoint here and you can get further by listing files and directories
from it?  But maybe people then use the filesystem directly to find a
directory and create the directories ad-hoc.

I grepped lieer but I think you only use it in one place?  And if I
understand it correctly you only do this to check if your mailstore/cwd
is inside the notmuch database.  I.e. this is equivalent to checking if
your mailstore/cwd has notmuch2.Database.path as prefix which you could
easily do directly rather than using the FileError exception from
.get_directory().

So is anyone else aware of some code which uses db.get_directory() to
give an idea of how and why this is used?

> I also ran into a couple of warning when building
> (included below).

Thanks for pointing these out.  I guess if the bindings are in the main
repo only the latest library version can be supported without any
further concerns.

> By the way, it does not seem that the API is very far from the
> previous python API. If it is close enough, perhaps it is possible to
> get away with a bug version bump in the bindings rather than creating
> a new package. I understand the need for a new package, but it would
> be nice if we could avoid the future confusion of two python binding
> packages (if at all possible).

While I'm glad to hear that you think a migration wouldn't be to painful
for you I am very weary of knowingly breaking APIs.  I'd rather have
people have an easy migration rather than unexpected breakage after an
upgrade.


Cheers,
Floris

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Python3 cffi bindings
  2019-11-17 17:14   ` Floris Bruynooghe
@ 2019-11-17 18:01     ` Gaute Hope
  2020-10-08  8:13     ` Gaute Hope
  1 sibling, 0 replies; 19+ messages in thread
From: Gaute Hope @ 2019-11-17 18:01 UTC (permalink / raw)
  To: Floris Bruynooghe; +Cc: notmuch

On Sun, Nov 17, 2019 at 6:14 PM Floris Bruynooghe <flub@devork.be> wrote:
>
> Hi Gaute,
>
> Thanks for trying this out!
>
> On Mon 04 Nov 2019 at 11:27 +0100, Gaute Hope wrote:
> > I just checked out the wip/cffi branch on git.notmuch.org with the
> > purpose of porting Lieer (https://github.com/gauteh/lieer). There
> > seems to be some missing functionality: `Database.get_directory()`
> > specifically.
>
> Yeah, I didn't add that yet because I don't fully understand how it
> should be used.  Specifically I don't know where one might get a
> pathname from to pass to .get_directory() and thus whether the API would
> be cleaner to just return a reasonable directory object from whatever
> location that might be.  Maybe notmuch_database_get_path() is the only
> entrypoint here and you can get further by listing files and directories
> from it?  But maybe people then use the filesystem directly to find a
> directory and create the directories ad-hoc.

If I understand correctly then these are the directories known to
notmuch db, so may not correspond to filesystem. Lieer do not modify
Directory objects directly, but others might.

> I grepped lieer but I think you only use it in one place?  And if I
> understand it correctly you only do this to check if your mailstore/cwd
> is inside the notmuch database.  I.e. this is equivalent to checking if
> your mailstore/cwd has notmuch2.Database.path as prefix which you could
> easily do directly rather than using the FileError exception from
> .get_directory().

Yes, I think that would work here. I need the path of the directory
later (for the path:.. query). Seems that the current python API
removes the leading path of the database for the argument to
notmuch_database_get_directory(..) -- at least if the notmuch API docs
are correct. I had some reported issues with symlinked directories and
absolute paths, but I don't think any of that would be influenced by
changes like these.

Regards, Gaute

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Python3 cffi bindings
  2019-11-16 16:44   ` David Bremner
@ 2019-12-04 20:18     ` Tomi Ollila
  0 siblings, 0 replies; 19+ messages in thread
From: Tomi Ollila @ 2019-12-04 20:18 UTC (permalink / raw)
  Cc: notmuch

On Sat, Nov 16 2019, David Bremner wrote:

> Gaute Hope <eg@gaute.vetsj.com> writes:
>
>>
>> By the way, it does not seem that the API is very far from the
>> previous python API. If it is close enough, perhaps it is possible to
>> get away with a bug version bump in the bindings rather than creating
>> a new package. I understand the need for a new package, but it would
>> be nice if we could avoid the future confusion of two python binding
>> packages (if at all possible).
>>
>
> I'm not in a good position to judge how similar the APIs are.  It does
> seem like there are at least some breaking changes, and we usually try
> to make things smooth for people upgrading by deprecating interfaces
> before removing them completely. On the other hand our previous concern
> for supporting python pre 3.6 (I think. Maybe 3.5?) seems less and less
> worrying (except maybe for people using old CentOS like things).


Currently such a recent Linux distribution as Ubuntu 16.04 LTS is not new
enough to be used as is when compiling latest notmuch from git (or 0.29),
as GMIME 3.0 is required (Ubuntu 16.04 ships GMIME 2.6 and Python 3.5).

GMIME 2.6 was pretty easy to compile, GMIME 3 is a bit harder...

Ubuntu 16.04 LTS ships python 3.5 -- and compiling later Pythons is easier
than GMIME 3 (done both, GMIME 3 on CentOS 6, Python 3.7 on CentOS 7).

So, IMO not supporting python pre 3.6 is fine, as we already reguire gmime 3 
and probably soon xapian 1.4 -- distros that ship gmime3 most probably 
already ship python 3.6...


Tomi


>
> d

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Python3 cffi bindings
  2019-11-17 17:14   ` Floris Bruynooghe
  2019-11-17 18:01     ` Gaute Hope
@ 2020-10-08  8:13     ` Gaute Hope
  2020-10-14 20:23       ` Floris Bruynooghe
  1 sibling, 1 reply; 19+ messages in thread
From: Gaute Hope @ 2020-10-08  8:13 UTC (permalink / raw)
  To: Floris Bruynooghe; +Cc: notmuch


[-- Attachment #1.1: Type: text/plain, Size: 2523 bytes --]

Hi Floris and others,

I made another attempt at porting lieer to notmuch2, but I am missing the
get_directory method still. Any plans to look at it?

Regards, Gaute

On Sun, Nov 17, 2019 at 6:14 PM Floris Bruynooghe <flub@devork.be> wrote:

> Hi Gaute,
>
> Thanks for trying this out!
>
> On Mon 04 Nov 2019 at 11:27 +0100, Gaute Hope wrote:
> > I just checked out the wip/cffi branch on git.notmuch.org with the
> > purpose of porting Lieer (https://github.com/gauteh/lieer). There
> > seems to be some missing functionality: `Database.get_directory()`
> > specifically.
>
> Yeah, I didn't add that yet because I don't fully understand how it
> should be used.  Specifically I don't know where one might get a
> pathname from to pass to .get_directory() and thus whether the API would
> be cleaner to just return a reasonable directory object from whatever
> location that might be.  Maybe notmuch_database_get_path() is the only
> entrypoint here and you can get further by listing files and directories
> from it?  But maybe people then use the filesystem directly to find a
> directory and create the directories ad-hoc.
>
> I grepped lieer but I think you only use it in one place?  And if I
> understand it correctly you only do this to check if your mailstore/cwd
> is inside the notmuch database.  I.e. this is equivalent to checking if
> your mailstore/cwd has notmuch2.Database.path as prefix which you could
> easily do directly rather than using the FileError exception from
> .get_directory().
>
> So is anyone else aware of some code which uses db.get_directory() to
> give an idea of how and why this is used?
>
> > I also ran into a couple of warning when building
> > (included below).
>
> Thanks for pointing these out.  I guess if the bindings are in the main
> repo only the latest library version can be supported without any
> further concerns.
>
> > By the way, it does not seem that the API is very far from the
> > previous python API. If it is close enough, perhaps it is possible to
> > get away with a bug version bump in the bindings rather than creating
> > a new package. I understand the need for a new package, but it would
> > be nice if we could avoid the future confusion of two python binding
> > packages (if at all possible).
>
> While I'm glad to hear that you think a migration wouldn't be to painful
> for you I am very weary of knowingly breaking APIs.  I'd rather have
> people have an easy migration rather than unexpected breakage after an
> upgrade.
>
>
> Cheers,
> Floris
>

[-- Attachment #1.2: Type: text/html, Size: 3224 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]



^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Python3 cffi bindings
  2020-10-08  8:13     ` Gaute Hope
@ 2020-10-14 20:23       ` Floris Bruynooghe
  2020-10-16  7:19         ` Gaute Hope
  0 siblings, 1 reply; 19+ messages in thread
From: Floris Bruynooghe @ 2020-10-14 20:23 UTC (permalink / raw)
  To: Gaute Hope; +Cc: notmuch

Hi Gaute,

On Thu 08 Oct 2020 at 10:13 +0200, Gaute Hope wrote:
> I made another attempt at porting lieer to notmuch2, but I am missing the
> get_directory method still. Any plans to look at it?

Would indeed be good to add this sometime.  I'm still curious to how you
use it though to make sure we make a good API.  I only found
https://github.com/gauteh/lieer/blob/394d8c1a574fd57e63390e92a6e73363808ebac5/lieer/local.py#L280
and it seems you only use the `.path` attribute.  Is this correct or did
I miss anything?

Cheers,
Floris

>
> Regards, Gaute
>
> On Sun, Nov 17, 2019 at 6:14 PM Floris Bruynooghe <flub@devork.be> wrote:
>
>> Hi Gaute,
>>
>> Thanks for trying this out!
>>
>> On Mon 04 Nov 2019 at 11:27 +0100, Gaute Hope wrote:
>> > I just checked out the wip/cffi branch on git.notmuch.org with the
>> > purpose of porting Lieer (https://github.com/gauteh/lieer). There
>> > seems to be some missing functionality: `Database.get_directory()`
>> > specifically.
>>
>> Yeah, I didn't add that yet because I don't fully understand how it
>> should be used.  Specifically I don't know where one might get a
>> pathname from to pass to .get_directory() and thus whether the API would
>> be cleaner to just return a reasonable directory object from whatever
>> location that might be.  Maybe notmuch_database_get_path() is the only
>> entrypoint here and you can get further by listing files and directories
>> from it?  But maybe people then use the filesystem directly to find a
>> directory and create the directories ad-hoc.
>>
>> I grepped lieer but I think you only use it in one place?  And if I
>> understand it correctly you only do this to check if your mailstore/cwd
>> is inside the notmuch database.  I.e. this is equivalent to checking if
>> your mailstore/cwd has notmuch2.Database.path as prefix which you could
>> easily do directly rather than using the FileError exception from
>> .get_directory().
>>
>> So is anyone else aware of some code which uses db.get_directory() to
>> give an idea of how and why this is used?
>>
>> > I also ran into a couple of warning when building
>> > (included below).
>>
>> Thanks for pointing these out.  I guess if the bindings are in the main
>> repo only the latest library version can be supported without any
>> further concerns.
>>
>> > By the way, it does not seem that the API is very far from the
>> > previous python API. If it is close enough, perhaps it is possible to
>> > get away with a bug version bump in the bindings rather than creating
>> > a new package. I understand the need for a new package, but it would
>> > be nice if we could avoid the future confusion of two python binding
>> > packages (if at all possible).
>>
>> While I'm glad to hear that you think a migration wouldn't be to painful
>> for you I am very weary of knowingly breaking APIs.  I'd rather have
>> people have an easy migration rather than unexpected breakage after an
>> upgrade.
>>
>>
>> Cheers,
>> Floris
>>

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Python3 cffi bindings
  2020-10-14 20:23       ` Floris Bruynooghe
@ 2020-10-16  7:19         ` Gaute Hope
  2020-10-16  7:24           ` Gaute Hope
  0 siblings, 1 reply; 19+ messages in thread
From: Gaute Hope @ 2020-10-16  7:19 UTC (permalink / raw)
  To: Floris Bruynooghe; +Cc: notmuch


[-- Attachment #1.1: Type: text/plain, Size: 928 bytes --]

On Wed, Oct 14, 2020 at 10:24 PM Floris Bruynooghe <flub@devork.be> wrote:

> Hi Gaute,
>
> On Thu 08 Oct 2020 at 10:13 +0200, Gaute Hope wrote:
> > I made another attempt at porting lieer to notmuch2, but I am missing the
> > get_directory method still. Any plans to look at it?
>
> Would indeed be good to add this sometime.  I'm still curious to how you
> use it though to make sure we make a good API.  I only found
>
> https://github.com/gauteh/lieer/blob/394d8c1a574fd57e63390e92a6e73363808ebac5/lieer/local.py#L280
> and it seems you only use the `.path` attribute.  Is this correct or did
> I miss anything?
>

That is correct, as well as relying on an exception if the input directory
is not in the notmuch database. I also use `db.get_path()` to figure out
the relative path w.r.t to the database root, for use in `path:` queries (
https://github.com/gauteh/lieer/blob/master/lieer/gmailieer.py#L315).

Regards, Gaute

[-- Attachment #1.2: Type: text/html, Size: 1497 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]



^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Python3 cffi bindings
  2020-10-16  7:19         ` Gaute Hope
@ 2020-10-16  7:24           ` Gaute Hope
  0 siblings, 0 replies; 19+ messages in thread
From: Gaute Hope @ 2020-10-16  7:24 UTC (permalink / raw)
  To: Floris Bruynooghe; +Cc: notmuch


[-- Attachment #1.1: Type: text/plain, Size: 1202 bytes --]

On Fri, Oct 16, 2020 at 9:19 AM Gaute Hope <eg@gaute.vetsj.com> wrote:

> On Wed, Oct 14, 2020 at 10:24 PM Floris Bruynooghe <flub@devork.be> wrote:
>
>> Hi Gaute,
>>
>> On Thu 08 Oct 2020 at 10:13 +0200, Gaute Hope wrote:
>> > I made another attempt at porting lieer to notmuch2, but I am missing
>> the
>> > get_directory method still. Any plans to look at it?
>>
>> Would indeed be good to add this sometime.  I'm still curious to how you
>> use it though to make sure we make a good API.  I only found
>>
>> https://github.com/gauteh/lieer/blob/394d8c1a574fd57e63390e92a6e73363808ebac5/lieer/local.py#L280
>> and it seems you only use the `.path` attribute.  Is this correct or did
>> I miss anything?
>>
>
> That is correct, as well as relying on an exception if the input directory
> is not in the notmuch database. I also use `db.get_path()` to figure out
> the relative path w.r.t to the database root, for use in `path:` queries (
> https://github.com/gauteh/lieer/blob/master/lieer/gmailieer.py#L315).
>
>
Re-read the rest of the old thread, and I think you are right that I only
need to use `db.get_path()` to do what I need. So that makes this issue
less critical for me I guess.

-- gaute

[-- Attachment #1.2: Type: text/html, Size: 2068 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]



^ permalink raw reply	[flat|nested] 19+ messages in thread

end of thread, other threads:[~2020-10-16  7:24 UTC | newest]

Thread overview: 19+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-10-08 21:03 Python3 cffi bindings Floris Bruynooghe
2019-10-08 21:03 ` [PATCH] Introduce CFFI-based python bindings Floris Bruynooghe
2019-10-08 22:24 ` Python3 cffi bindings David Bremner
2019-10-09 18:34   ` Floris Bruynooghe
2019-10-14 12:40 ` David Bremner
2019-10-14 12:42   ` David Bremner
2019-10-17 17:35     ` Floris Bruynooghe
2019-10-20 12:22       ` David Bremner
2019-10-22 16:32         ` David Bremner
2019-10-25  9:57           ` Floris Bruynooghe
2019-11-04 10:27 ` Gaute Hope
2019-11-16 16:44   ` David Bremner
2019-12-04 20:18     ` Tomi Ollila
2019-11-17 17:14   ` Floris Bruynooghe
2019-11-17 18:01     ` Gaute Hope
2020-10-08  8:13     ` Gaute Hope
2020-10-14 20:23       ` Floris Bruynooghe
2020-10-16  7:19         ` Gaute Hope
2020-10-16  7:24           ` Gaute Hope

Code repositories for project(s) associated with this public inbox

	https://yhetil.org/notmuch.git/

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).