* DRAFT Introduce CFFI-based Python bindings
@ 2017-11-28 20:46 Floris Bruynooghe
2017-11-28 20:46 ` [PATCH] Introduce CFFI-based python bindings Floris Bruynooghe
2017-11-28 23:59 ` DRAFT Introduce CFFI-based Python bindings David Bremner
0 siblings, 2 replies; 9+ messages in thread
From: Floris Bruynooghe @ 2017-11-28 20:46 UTC (permalink / raw)
To: notmuch
Hi all,
Here are the beginnings off CFFI-based Python bindings, rather
than the ctypes-based ones currently available. I started this
work in order to get faster bindings on pypy since a script of
mine was running slower on pypy than CPython. Initially aiming
for a drop-in replacement of the existing bindings I ended up
abandoning this to help enforce correct usage of the API.
The benefits of this approach are:
- More "Pythonic" API, e.g. tags behave like sets, iterators
which get consumed can easily be re-created as is usual with
collections, avoid allowing invalid combinations of args and
calls on a Python-API level.
- CFFI, this works on both CPython and PyPy, on the latter it
is (supposed to be) a lot faster as the JIT can cross the
boundary between C and Python code where it otherwise has
extra overheads to emulate the C-Python API. Additionally
it makes it safer to use compared to ctypes, it works on the
API level using the compiler to figure out the correct details
of the platform. Compared to ctypes which only works on the
ABI level and you need to rely on knowing the layout of code
when writing the Python bindings.
Additionaly I belive these bindings fix a memory safety issue,
certain situations in my test-suite would lead to coredump which
is not something which should be possible from within Python.
I believe I have seen similar reports in the list archives so
am not the only one seeing these. Sadly these are hard to
isolate and I have not managed to re-create this in a nice
minimal example, however I believe the root cause is that in
some situations, mostly interpreter shutdown, the __del__
method can have been called while there are still references
to the object and while child-objects are still alive. This
effectively results in double-frees as the child object frees
memory already freed by the parent. These bindings solve this
by adding the .alive property and using this to check parent
objects are still alive before destroying themselves. This is
somewhat expensive, but works and is easy to implement.
Lastly there are some downsides to the choices I made:
- I ended up going squarely for CPython 3.6+. Choosing Python
3 allowed better API design, e.g. with keyword-only parameters
etc. Choosing CPython 3.4+ restricts the madness that can
happen with __del__ and gives some newer (tho now unused)
features in weakref.finalizer.
- This is no longer drop-in compatible.
- I haven't got to a stage where my initial goal of speed has
been proven yet.
In theory I think it's possible to create a CFFI-based drop-in
replacement to the bindings, only adding the memory-safety fixes
and keeping the Python 2.7 compatibility. It would then be
possible to build the API proposed in these bindings on top of
this, but once I was making these bindings safer it felt strange
to still allow the API to be misused.
There are a lot of details about this which can be discussed,
also many finer implementation points and even just getting the
proposed API right (you'll notice large gaps for now). But
this mail is already too long. I look forward to your comments
and feedback on the approach taken and on whether some form
of this could make it into the main repo.
Lastly a small note on the AUTHORS file patch, due to my own
unfortunate choice of employer I have strict rules to follow
on how to submit patches. One of which is to add this line if
an AUTHORS file exists. Given clearly not everyone is listed
here though maybe this is not appropriate. I would also rather
receive email on flub@devork.be rather than the address I have
to use in the git commits.
Kind Regards,
Floris
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH] Introduce CFFI-based python bindings
2017-11-28 20:46 DRAFT Introduce CFFI-based Python bindings Floris Bruynooghe
@ 2017-11-28 20:46 ` Floris Bruynooghe
2017-11-28 23:59 ` DRAFT Introduce CFFI-based Python bindings David Bremner
1 sibling, 0 replies; 9+ messages in thread
From: Floris Bruynooghe @ 2017-11-28 20:46 UTC (permalink / raw)
To: notmuch; +Cc: Floris Bruynooghe
From: Floris Bruynooghe <flub@google.com>
This introduces the beginnings of new CFFI-based Python bindings.
The bindings aim at:
- Better performance on pypy
- Easier to use Python-C interface
- More "pythonic"
- The API should not allow invalid operations
- Use native object protocol where possible
- Memory safety; whatever you do from python, it should not coredump.
---
AUTHORS | 1 +
bindings/python-cffi/notdb/__init__.py | 52 +++
bindings/python-cffi/notdb/_base.py | 141 +++++++
bindings/python-cffi/notdb/_build.py | 140 +++++++
bindings/python-cffi/notdb/_database.py | 577 ++++++++++++++++++++++++++++
bindings/python-cffi/notdb/_errors.py | 114 ++++++
bindings/python-cffi/notdb/_message.py | 285 ++++++++++++++
bindings/python-cffi/notdb/_tags.py | 319 +++++++++++++++
bindings/python-cffi/setup.py | 13 +
bindings/python-cffi/tests/conftest.py | 138 +++++++
bindings/python-cffi/tests/test_base.py | 116 ++++++
bindings/python-cffi/tests/test_database.py | 274 +++++++++++++
bindings/python-cffi/tests/test_tags.py | 141 +++++++
13 files changed, 2311 insertions(+)
create mode 100644 bindings/python-cffi/notdb/__init__.py
create mode 100644 bindings/python-cffi/notdb/_base.py
create mode 100644 bindings/python-cffi/notdb/_build.py
create mode 100644 bindings/python-cffi/notdb/_database.py
create mode 100644 bindings/python-cffi/notdb/_errors.py
create mode 100644 bindings/python-cffi/notdb/_message.py
create mode 100644 bindings/python-cffi/notdb/_tags.py
create mode 100644 bindings/python-cffi/setup.py
create mode 100644 bindings/python-cffi/tests/conftest.py
create mode 100644 bindings/python-cffi/tests/test_base.py
create mode 100644 bindings/python-cffi/tests/test_database.py
create mode 100644 bindings/python-cffi/tests/test_tags.py
diff --git a/AUTHORS b/AUTHORS
index 6d0f2de8..9ce9ce6f 100644
--- a/AUTHORS
+++ b/AUTHORS
@@ -28,3 +28,4 @@ ideas, inspiration, testing or feedback):
Martin Krafft
Keith Packard
Jamey Sharp
+Google LLC
diff --git a/bindings/python-cffi/notdb/__init__.py b/bindings/python-cffi/notdb/__init__.py
new file mode 100644
index 00000000..340f4388
--- /dev/null
+++ b/bindings/python-cffi/notdb/__init__.py
@@ -0,0 +1,52 @@
+"""Pythonic API to the notmuch database.
+
+Creating Objects
+================
+
+Only the :class:`Database` object is meant to be created by the user.
+All other objects should be created from this initial object. Users
+should consider their signatures implementation details.
+
+Errors
+======
+
+All errors occuring due to errors from the underlying notmuch database
+are subclasses of the :exc:`NotmuchError`. Due to memory management
+it is possible to try and use an object after it has been freed. In
+this case a :exc:`ObjectDestoryedError` will be raised.
+
+Memory Management
+=================
+
+xxx
+"""
+
+from notdb import _capi
+from notdb._database import AtomicContext
+from notdb._database import Database
+from notdb._database import DbRevision
+from notdb._errors import NotmuchError
+from notdb._errors import OutOfMemoryError
+from notdb._errors import ReadOnlyDatabaseError
+from notdb._errors import XapianError
+from notdb._errors import FileError
+from notdb._errors import FileNotEmailError
+from notdb._errors import DuplicateMessageIdError
+from notdb._errors import NullPointerError
+from notdb._errors import TagTooLongError
+from notdb._errors import UnbalancedFreezeThawError
+from notdb._errors import UnbalancedAtomicError
+from notdb._errors import UnsupportedOperationError
+from notdb._errors import UpgradeRequiredError
+from notdb._errors import PathError
+from notdb._errors import IllegalArgumentError
+from notdb._errors import NoSuchHeaderError
+from notdb._errors import NoMessageError
+from notdb._errors import ObjectDestroyedError
+from notdb._message import Message
+from notdb._tags import ImmutableTagSet
+from notdb._tags import MutableTagSet
+from notdb._tags import TagsIter
+
+
+NOTMUCH_TAG_MAX = _capi.lib.NOTMUCH_TAG_MAX
diff --git a/bindings/python-cffi/notdb/_base.py b/bindings/python-cffi/notdb/_base.py
new file mode 100644
index 00000000..8229b1a6
--- /dev/null
+++ b/bindings/python-cffi/notdb/_base.py
@@ -0,0 +1,141 @@
+import abc
+
+from notdb import _errors as errors
+
+
+class NotmuchObject(metaclass=abc.ABCMeta):
+ """Base notmuch object syntax.
+
+ This base class exists to define the memory management handling
+ required to use the notmuch library. It is meant as an interface
+ definition rather than a base class, though you can use it as a
+ base class to ensure you don't forget part of the interface. It
+ only concerns you if you are implementing this package itself
+ rather then using it.
+
+ libnotmuch uses a hierarchical memory allocator, where freeing the
+ memory of a parent object also frees the memory of all child
+ objects. To make this work seamlessly in Python this package
+ keeps references to parent objects which makes them stay alive
+ correctly under normal circumstances. When an object finally gets
+ deleted the :meth:`__del__` method will be called to free the
+ memory.
+
+ However during some peculiar situations, e.g. interpreter
+ shutdown, it is possible for the :meth:`__del__` method to have
+ been called, whele there are still references to an object. This
+ could result in child objects asking their memeory to be freed
+ after the parent has already freed the memory, making things
+ rather unhappy as double frees are not taken lightly in C. To
+ handle this case all objects need to follow the same protocol to
+ destroy themselves, see :meth:`destroy`.
+
+ Once an object has been destroyed trying to use it should raise
+ the :exc:`ObjectDestroyedError` exception. For this see also the
+ convenience :class:`MemoryPointer` descriptor in this module which
+ can be used as a pointer to libnotmuch memory.
+ """
+
+ @abc.abstractmethod
+ def __init__(self, parent, *args, **kwargs):
+ """Create a new object.
+
+ Other then for the toplevel :class:`Database` object
+ constructors are only ever called by internal code and not by
+ the user. Per convention their signature always takes the
+ parent object as first argument. Feel free to make the rest
+ of the signature match the object's requirement. The object
+ needs to keep a reference to the parent, so it can check the
+ parent is still alive.
+ """
+
+ @property
+ @abc.abstractmethod
+ def alive(self):
+ """Whether the object is still alive.
+
+ This indicates whether the object is still alive. The first
+ thing this needs to check is whether the parent object is
+ still alive, if it is not then this object can not be alive
+ either. If the parent is alive then it depends on whether the
+ memory for this object has been freed yet or not.
+ """
+
+ def __del__(self):
+ self.destroy()
+
+ @abc.abstractmethod
+ def destroy(self):
+ """Destroy the object, freeing all memory.
+
+ This method needs to destory the object on the
+ libnotmuch-level. It must ensure it's not been destroyed by
+ it's parent object yet before doing so. It also must be
+ idempotent.
+ """
+
+
+class MemoryPointer:
+ """Data Descriptor to handle accessing libnotmuch pointers.
+
+ Most :class:`NotmuchObject` instances will have one or more CFFI
+ pointers to C-objects. Once an object is destroyed this pointer
+ should no longer be used and a :exc:`ObjectDestroyedError`
+ exception should be raised on trying to access it. This
+ descriptor simplifies implementing this, allowing the creation of
+ an attribute which can be assigned to, but when accessed when the
+ stored value is *None* it will raise the
+ :exc:`ObjectDestroyedError` exception::
+
+ class SomeOjb:
+ _ptr = MemoryPointer()
+
+ def __init__(self, ptr):
+ self._ptr = ptr
+
+ def destroy(self):
+ somehow_free(self._ptr)
+ self._ptr = None
+
+ def do_something(self):
+ return some_libnotmuch_call(self._ptr)
+ """
+
+ def __get__(self, instance, owner):
+ val = getattr(instance, self.attr_name, None)
+ if val is None:
+ raise errors.ObjectDestroyedError()
+ return val
+
+ def __set__(self, instance, value):
+ setattr(instance, self.attr_name, value)
+
+ def __set_name__(self, instance, name):
+ self.attr_name = '_memptr_{}_{:x}'.format(name, id(instance))
+
+
+class BinString(str):
+ """A str subclass with binary data.
+
+ Most data in libnotmuch should be valid ASCII or valid UTF-8.
+ However since it is a C library these are represented as
+ bytestrings intead which means on an API level we can not
+ guarantee that decoding this to UTF-8 will both succeed and be
+ lossless. This string type converts bytes to unicode in a lossy
+ way, but also makes the raw bytes available.
+
+ This object is a normal unicode string for most intents and
+ purposes, but you can get the original bytestring back by calling
+ ``bytes()`` on it.
+ """
+
+ def __new__(cls, data, encoding='utf-8', errors='ignore'):
+ if not isinstance(data, bytes):
+ data = bytes(data, encoding=encoding)
+ strdata = str(data, encoding=encoding, errors=errors)
+ inst = super().__new__(cls, strdata)
+ inst._bindata = data
+ return inst
+
+ def __bytes__(self):
+ return self._bindata
diff --git a/bindings/python-cffi/notdb/_build.py b/bindings/python-cffi/notdb/_build.py
new file mode 100644
index 00000000..affce989
--- /dev/null
+++ b/bindings/python-cffi/notdb/_build.py
@@ -0,0 +1,140 @@
+import cffi
+
+
+ffibuilder = cffi.FFI()
+ffibuilder.set_source(
+ 'notdb._capi',
+ r"""
+ #include <stdlib.h>
+ #include <notmuch.h>
+ """,
+ libraries=['notmuch'],
+)
+ffibuilder.cdef(
+ r"""
+ void free(void *ptr);
+
+ #define NOTMUCH_TAG_MAX ...
+
+ typedef enum _notmuch_status {
+ NOTMUCH_STATUS_SUCCESS = 0,
+ NOTMUCH_STATUS_OUT_OF_MEMORY,
+ NOTMUCH_STATUS_READ_ONLY_DATABASE,
+ NOTMUCH_STATUS_XAPIAN_EXCEPTION,
+ NOTMUCH_STATUS_FILE_ERROR,
+ NOTMUCH_STATUS_FILE_NOT_EMAIL,
+ NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID,
+ NOTMUCH_STATUS_NULL_POINTER,
+ NOTMUCH_STATUS_TAG_TOO_LONG,
+ NOTMUCH_STATUS_UNBALANCED_FREEZE_THAW,
+ NOTMUCH_STATUS_UNBALANCED_ATOMIC,
+ NOTMUCH_STATUS_UNSUPPORTED_OPERATION,
+ NOTMUCH_STATUS_UPGRADE_REQUIRED,
+ NOTMUCH_STATUS_PATH_ERROR,
+ NOTMUCH_STATUS_ILLEGAL_ARGUMENT,
+ NOTMUCH_STATUS_LAST_STATUS
+ } notmuch_status_t;
+ typedef enum {
+ NOTMUCH_DATABASE_MODE_READ_ONLY = 0,
+ NOTMUCH_DATABASE_MODE_READ_WRITE
+ } notmuch_database_mode_t;
+ typedef int notmuch_bool_t;
+
+ // These are fully opaque types for us, we only ever use pointers.
+ typedef struct _notmuch_database notmuch_database_t;
+ typedef struct _notmuch_query notmuch_query_t;
+ typedef struct _notmuch_threads notmuch_threads_t;
+ typedef struct _notmuch_thread notmuch_thread_t;
+ typedef struct _notmuch_messages notmuch_messages_t;
+ typedef struct _notmuch_message notmuch_message_t;
+ typedef struct _notmuch_tags notmuch_tags_t;
+ typedef struct _notmuch_directory notmuch_directory_t;
+ typedef struct _notmuch_filenames notmuch_filenames_t;
+ typedef struct _notmuch_config_list notmuch_config_list_t;
+
+ const char *
+ notmuch_status_to_string (notmuch_status_t status);
+
+ notmuch_status_t
+ notmuch_database_create_verbose (const char *path,
+ notmuch_database_t **database,
+ char **error_message);
+ notmuch_status_t
+ notmuch_database_create (const char *path, notmuch_database_t **database);
+ notmuch_status_t
+ notmuch_database_open_verbose (const char *path,
+ notmuch_database_mode_t mode,
+ notmuch_database_t **database,
+ char **error_message);
+ notmuch_status_t
+ notmuch_database_open (const char *path,
+ notmuch_database_mode_t mode,
+ notmuch_database_t **database);
+ notmuch_status_t
+ notmuch_database_close (notmuch_database_t *database);
+ notmuch_status_t
+ notmuch_database_destroy (notmuch_database_t *database);
+ const char *
+ notmuch_database_get_path (notmuch_database_t *database);
+ unsigned int
+ notmuch_database_get_version (notmuch_database_t *database);
+ notmuch_bool_t
+ notmuch_database_needs_upgrade (notmuch_database_t *database);
+ notmuch_status_t
+ notmuch_database_begin_atomic (notmuch_database_t *notmuch);
+ notmuch_status_t
+ notmuch_database_end_atomic (notmuch_database_t *notmuch);
+ unsigned long
+ notmuch_database_get_revision (notmuch_database_t *notmuch,
+ const char **uuid);
+ notmuch_status_t
+ notmuch_database_add_message (notmuch_database_t *database,
+ const char *filename,
+ notmuch_message_t **message);
+ notmuch_status_t
+ notmuch_database_remove_message (notmuch_database_t *database,
+ const char *filename);
+ notmuch_status_t
+ notmuch_database_find_message (notmuch_database_t *database,
+ const char *message_id,
+ notmuch_message_t **message);
+ notmuch_status_t
+ notmuch_database_find_message_by_filename (notmuch_database_t *notmuch,
+ const char *filename,
+ notmuch_message_t **message);
+ notmuch_tags_t *
+ notmuch_database_get_all_tags (notmuch_database_t *db);
+
+ const char *
+ notmuch_message_get_message_id (notmuch_message_t *message);
+ const char *
+ notmuch_message_get_filename (notmuch_message_t *message);
+ notmuch_tags_t *
+ notmuch_message_get_tags (notmuch_message_t *message);
+ notmuch_status_t
+ notmuch_message_add_tag (notmuch_message_t *message, const char *tag);
+ notmuch_status_t
+ notmuch_message_remove_tag (notmuch_message_t *message, const char *tag);
+ notmuch_status_t
+ notmuch_message_remove_all_tags (notmuch_message_t *message);
+ notmuch_status_t
+ notmuch_message_maildir_flags_to_tags (notmuch_message_t *message);
+ notmuch_status_t
+ notmuch_message_tags_to_maildir_flags (notmuch_message_t *message);
+ void
+ notmuch_message_destroy (notmuch_message_t *message);
+
+ notmuch_bool_t
+ notmuch_tags_valid (notmuch_tags_t *tags);
+ const char *
+ notmuch_tags_get (notmuch_tags_t *tags);
+ void
+ notmuch_tags_move_to_next (notmuch_tags_t *tags);
+ void
+ notmuch_tags_destroy (notmuch_tags_t *tags);
+ """
+)
+
+
+if __name__ == '__main__':
+ ffibuilder.compile(verbose=True)
diff --git a/bindings/python-cffi/notdb/_database.py b/bindings/python-cffi/notdb/_database.py
new file mode 100644
index 00000000..596c7710
--- /dev/null
+++ b/bindings/python-cffi/notdb/_database.py
@@ -0,0 +1,577 @@
+import collections
+import configparser
+import enum
+import functools
+import os
+import pathlib
+
+import notdb._base as base
+import notdb._capi as capi
+import notdb._errors as errors
+import notdb._message as message
+import notdb._tags as tags
+
+
+def _config_pathname():
+ """Return the path of the configuration file.
+
+ :rtype: pathlib.Path
+ """
+ cfgfname = os.getenv('NOTMUCH_CONFIG', '~/.notmuch-config')
+ return pathlib.Path(os.path.expanduser(cfgfname))
+
+
+class Mode(enum.Enum):
+ READ_ONLY = capi.lib.NOTMUCH_DATABASE_MODE_READ_ONLY
+ READ_WRITE = capi.lib.NOTMUCH_DATABASE_MODE_READ_WRITE
+
+
+class Database(base.NotmuchObject):
+ """Toplevel access to notmuch.
+
+ A :class:`Database` can be opened read-only or read-write.
+ Modifications are not atomic by default, use :meth:`begin_atomic`
+ for atomic updates. If the underlying database has been modified
+ outside of this class a :exc:`XapianError` will be raised and the
+ instance must be closed and a new one created.
+
+ You can use an instance of this class as a context-manager.
+
+ :cvar MODE: The mode a database can be opened with, an enumeration
+ of ``READ_ONLY`` and ``READ_WRITE``
+ :cvar AddedMessage: A namedtuple ``(msg, dup)`` used by
+ :meth:`add` as return value.
+
+ :ivar closed: Boolean indicating if the database is closed or
+ still open.
+
+ :param path: The directory of where the database is stored. If
+ ``None`` the location will be read from the user's
+ configuration file, respecting the ``NOTMUCH_CONFIG``
+ environment variable if set.
+ :type path: str, bytes or os.PathLike
+ :param mode: The mode to open the database in. One of
+ :attr:`MODE.READ_ONLY` OR :attr:`MODE.READ_WRITE`.
+ :type mode: :attr:`MODE`.
+
+ :raises OSError: or subclasses if the configuration file can not
+ be opened.
+ :raises configparser.Error: or subclasses if the configuration
+ file can not be parsed.
+ :raises NotmuchError: or subclasses for other failures.
+ """
+
+ MODE = Mode
+ AddedMessage = collections.namedtuple('AddedMessage', ['msg', 'dup'])
+ _db_p = base.MemoryPointer()
+
+ def __init__(self, path=None, mode=MODE.READ_ONLY):
+ self.mode = mode
+ if path is None:
+ path = self.default_path()
+ db_pp = capi.ffi.new('notmuch_database_t **')
+ cmsg = capi.ffi.new('char**')
+ ret = capi.lib.notmuch_database_open_verbose(os.fsencode(path),
+ mode.value, db_pp, cmsg)
+ if cmsg[0]:
+ msg = capi.ffi.string(cmsg[0]).decode(errors='replace')
+ capi.lib.free(cmsg[0])
+ else:
+ msg = None
+ if ret != capi.lib.NOTMUCH_STATUS_SUCCESS:
+ raise errors.NotmuchError(ret, msg)
+ self._db_p = db_pp[0]
+ self.closed = False
+
+ @classmethod
+ def create(cls, path=None):
+ """Create and open database in READ_WRITE mode.
+
+ This is creates a new notmuch database and returns an opened
+ instance in :attr:`MODE.READ_WRITE` mode.
+
+ :param path: The directory of where the database is stored. If
+ ``None`` the location will be read from the user's
+ configuration file, respecting the ``NOTMUCH_CONFIG``
+ environment variable if set.
+ :type path: str, bytes or os.PathLike
+
+ :raises OSError: or subclasses if the configuration file can not
+ be opened.
+ :raises configparser.Error: or subclasses if the configuration
+ file can not be parsed.
+ :raises NotmuchError: if the config file does not have the
+ database.path setting.
+ :raises FileError: if the database already exists.
+
+ :returns: The newly created instance.
+ """
+ if path is None:
+ path = cls.default_path()
+ db_pp = capi.ffi.new('notmuch_database_t **')
+ cmsg = capi.ffi.new('char**')
+ ret = capi.lib.notmuch_database_create_verbose(os.fsencode(path),
+ db_pp, cmsg)
+ if cmsg[0]:
+ msg = capi.ffi.string(cmsg[0]).decode(errors='replace')
+ capi.lib.free(cmsg[0])
+ else:
+ msg = None
+ if ret != capi.lib.NOTMUCH_STATUS_SUCCESS:
+ raise errors.NotmuchError(ret, msg)
+
+ # Now close the db and let __init__ open it. Inefficient but
+ # creating is not a hot loop while this allows us to have a
+ # clean API.
+ ret = capi.lib.notmuch_database_destroy(db_pp[0])
+ if ret != capi.lib.NOTMUCH_STATUS_SUCCESS:
+ raise errors.NotmuchError(ret)
+ return cls(path, cls.MODE.READ_WRITE)
+
+ @staticmethod
+ def default_path(cfg_path=None):
+ """Return the path of the user's default database.
+
+ This reads the user's configuration file and returns the
+ default path of the database.
+
+ :param cfg_path: The pathname of the notmuch configuration file.
+ If not specified tries to use the pathname provided in the
+ :env:`NOTMUCH_CONFIG` environment variable and falls back
+ to :file:`~/.notmuch-config.
+ :type cfg_path: str, bytes or os.PathLike.
+
+ :returns: The path of the database, which does not necessarily
+ exists.
+ :rtype: pathlib.Path
+ :raises OSError: or subclasses if the configuration file can not
+ be opened.
+ :raises configparser.Error: or subclasses if the configuration
+ file can not be parsed.
+ :raises NotmuchError if the config file does not have the
+ database.path setting.
+ """
+ if not cfg_path:
+ cfg_path = _config_pathname()
+ parser = configparser.ConfigParser()
+ with open(cfg_path) as fp:
+ parser.read_file(fp)
+ try:
+ return pathlib.Path(parser.get('database', 'path'))
+ except configparser.Error:
+ raise errors.NotmuchError(
+ 'No database.path setting in {}'.format(cfg_path))
+
+ def __del__(self):
+ self.destroy()
+
+ @property
+ def alive(self):
+ try:
+ self._db_p
+ except errors.ObjectDestroyedError:
+ return False
+ else:
+ return True
+
+ def destroy(self):
+ try:
+ ret = capi.lib.notmuch_database_destroy(self._db_p)
+ except errors.ObjectDestroyedError:
+ ret = capi.lib.NOTMUCH_STATUS_SUCCESS
+ else:
+ self._db_p = None
+ if ret != capi.lib.NOTMUCH_STATUS_SUCCESS:
+ raise errors.NotmuchError(ret)
+
+ def close(self):
+ """Close the notmuch database.
+
+ Once closed most operations will fail.
+
+ :raises ObjectDestroyedError: if used after destroyed.
+ """
+ ret = capi.lib.notmuch_database_close(self._db_p)
+ if ret != capi.lib.NOTMUCH_STATUS_SUCCESS:
+ raise errors.NotmuchError(ret)
+ self.closed = True
+
+ def __enter__(self):
+ return self
+
+ def __exit__(self, exc_type, exc_value, traceback):
+ self.close()
+
+ @property
+ def path(self):
+ """The pathname of the notmuch database.
+
+ This is returned as a :class:`pathlib.Path` instance.
+
+ :raises ObjectDestroyedError: if used after destoryed.
+ """
+ try:
+ return self._cache_path
+ except AttributeError:
+ ret = capi.lib.notmuch_database_get_path(self._db_p)
+ self._cache_path = pathlib.Path(os.fsdecode(capi.ffi.string(ret)))
+ return self._cache_path
+
+ @property
+ def version(self):
+ """The database format version.
+
+ This is a positive integer.
+
+ :raises ObjectDestroyedError: if used after destoryed.
+ """
+ try:
+ return self._cache_version
+ except AttributeError:
+ ret = capi.lib.notmuch_database_get_version(self._db_p)
+ self._cache_version = ret
+ return ret
+
+ @property
+ def needs_upgrade(self):
+ """Whether the database should be upgraded.
+
+ If *True* the database can be upgraded using :meth:`upgrade`.
+ Not doing so may result in some operations raising
+ :exc:`UpgradeRequiredError`.
+
+ A read-only database will never be upgradable.
+
+ :raises ObjectDestroyedError: if used after destoryed.
+ """
+ ret = capi.lib.notmuch_database_needs_upgrade(self._db_p)
+ return bool(ret)
+
+ def upgrade(self, progress_cb=None):
+ """Upgrade the database to the latest version.
+
+ Upgrade the database, optionally with a progress callback
+ which should be a callable which will be called with a
+ floating point number in the range of [0.0 .. 1.0].
+ """
+ raise NotImplementedError
+
+ def atomic(self):
+ """Return a context manager to perform atomic operations.
+
+ The returned context manager can be used to perform atomic
+ operations on the database.
+
+ .. note:: Unlinke a traditional RDBMS transaction this does
+ not imply durability, it only ensures the changes are
+ performed atomically.
+
+ :raises ObjectDestroyedError: if used after destoryed.
+ """
+ ctx = AtomicContext(self, '_db_p')
+ return ctx
+
+ def revision(self):
+ """The currently committed revision in the database.
+
+ Returned as a ``(revision, uuid)`` namedtuple.
+
+ :raises ObjectDestroyedError: if used after destoryed.
+ """
+ raw_uuid = capi.ffi.new('char**')
+ rev = capi.lib.notmuch_database_get_revision(self._db_p, raw_uuid)
+ return DbRevision(rev, capi.ffi.string(raw_uuid[0]))
+
+ def get_directory(self, path):
+ raise NotImplementedError
+
+ def add(self, pathname, *, sync_flags=False):
+ """Add a message to the database.
+
+ Add a new message to the notmuch database. The message is
+ referred to by the pathname of the maildir file. If the
+ message ID of the new message already exists in the database,
+ this adds ``pathname`` to the list of list of files for the
+ existing message.
+
+ :param pathname: The path of the file containing the message.
+ :type pathname: str, bytes or os.PathLike
+ :param sync_flags: Whether to sync the known maildir flags to
+ notmuch tags. See :meth:`Message.flags_to_tags` for
+ details.
+
+ :returns: A tuple where the first item is the newly inserted
+ messages as a :class:`Message` instance, and the second
+ item is a boolean indicating if the message inserted was a
+ duplicate. This is the namedtuple ``AddedMessage(msg,
+ dup)``.
+ :rtype: Database.AddedMessage
+
+ If an exception is raised, no message was added.
+
+ :raises XapianError: A Xapian exception occurred.
+ :raises FileError: The file referred to by ``pathname`` could
+ not be opened.
+ :raises FileNotEmailError: The file referreed to by
+ ``pathname`` is not recognised as an email message.
+ :raises ReadOnlyDatabaseError: The database is opened in
+ READ_ONLY mode.
+ :raises UpgradeRequiredError: The database must be upgraded
+ first.
+ :raises ObjectDestroyedError: if used after destoryed.
+ """
+ msg_pp = capi.ffi.new('notmuch_message_t **')
+ ret = capi.lib.notmuch_database_add_message(self._db_p,
+ os.fsencode(pathname),
+ msg_pp)
+ ok = [capi.lib.NOTMUCH_STATUS_SUCCESS,
+ capi.lib.NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID]
+ if ret not in ok:
+ raise errors.NotmuchError(ret)
+ msg = message.Message(self, msg_pp[0])
+ if sync_flags:
+ msg.flags_to_tags()
+ return self.AddedMessage(
+ msg, ret == capi.lib.NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID)
+
+ def remove(self, pathname):
+ """Remove a message from the notmuch database.
+
+ Removing a message which is not in the database is just a
+ silent nop-operation.
+
+ :param pathname: The pathname of the file containing the
+ message to be removed.
+ :type pathname: str, bytes or os.PathLike
+
+ :returns: True if the message is still in the database. This
+ can happen when multiple files contain the same message ID.
+ The true/false distinction is fairly arbitrary, but think
+ of it as ``dup = db.remove_message(name); if dup: ...``.
+ :rtype: bool
+
+ :raises XapianError: A Xapian exception ocurred.
+ :raises ReadOnlyDatabaseError: The database is opened in
+ READ_ONLY mode.
+ :raises UpgradeRequiredError: The database must be upgraded
+ first.
+ :raises ObjectDestroyedError: if used after destoryed.
+ """
+ ret = capi.lib.notmuch_database_remove_message(self._db_p,
+ os.fsencode(pathname))
+ ok = [capi.lib.NOTMUCH_STATUS_SUCCESS,
+ capi.lib.NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID]
+ if ret not in ok:
+ raise errors.NotmuchError(ret)
+ if ret == capi.lib.NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID:
+ return True
+ else:
+ return False
+
+ def find(self, msgid):
+ """Return the message matching the given message ID.
+
+ If a message with the given message ID is found a
+ :class:`Message` instance is returned. Otherwise a
+ :exc:`NoMessageError` is raised.
+
+ :param msgid: The message ID to look for.
+ :type msgid: str
+
+ :returns: The message instance.
+ :rtype: Message
+
+ :raises NoMessageError: If no message was found.
+ :raises OutOfMemoryError: When there is no memory to allocate
+ the message instance.
+ :raises XapianError: A Xapian exception ocurred.
+ :raises ObjectDestroyedError: if used after destoryed.
+ """
+ msg_pp = capi.ffi.new('notmuch_message_t **')
+ ret = capi.lib.notmuch_database_find_message(self._db_p,
+ msgid.encode(), msg_pp)
+ if ret != capi.lib.NOTMUCH_STATUS_SUCCESS:
+ raise errors.NotmuchError(ret)
+ msg_p = msg_pp[0]
+ if msg_p == capi.ffi.NULL:
+ raise errors.NoMessageError
+ msg = message.Message(self, msg_p)
+ return msg
+
+ def get(self, pathname):
+ """Return the :class:`Message` given a pathname.
+
+ If a message with the given pathname exists in the database
+ return the :class:`Message` instance for the message.
+ Otherwise raise a :exc:`NoMessageError` exception.
+
+ :param pathname: The pathname of the message.
+ :type pathname: str, bytes or os.PathLike
+
+ :returns: The message instance.
+ :rtype: Message
+
+ :raises NoMessageError: If no message was found. This is also
+ a subclass of :exc:`KeyError`.
+ :raises OutOfMemoryError: When there is no memory to allocate
+ the message instance.
+ :raises XapianError: A Xapian exception ocurred.
+ :raises ObjectDestroyedError: if used after destoryed.
+ """
+ msg_pp = capi.ffi.new('notmuch_message_t **')
+ ret = capi.lib.notmuch_database_find_message_by_filename(
+ self._db_p, os.fsencode(pathname), msg_pp)
+ if ret != capi.lib.NOTMUCH_STATUS_SUCCESS:
+ raise errors.NoMessageError(ret)
+ msg_p = msg_pp[0]
+ if msg_p == capi.ffi.NULL:
+ raise errors.NoMessageError
+ msg = message.Message(self, msg_p)
+ return msg
+
+ @property
+ def tags(self):
+ """Return an immutable set with all tags used in this database.
+
+ This returns an immutable set-like object implementing
+ matching the collections.abc.Set Abstract Base Class. Due to
+ the underlying libnotmuch implementation some operations have
+ different performance characteristics then plain set objects.
+ Mainly any lookup operation is O(n) rather then O(1).
+
+ Normal usage treats tags as UTF-8 encoded unicode strings so
+ they are exposed to Python as normal unicode string objects.
+ If you need to handle tags stored in libnotmuch which are not
+ valid unicode do check the :class:`ImmutableTagSet` docs for
+ how to handle this.
+
+ :rtype: ImmutableTagSet
+
+ :raises ObjectDestroyedError: if used after destoryed.
+ """
+ # By caching the tagset this creates a circular reference.
+ # This is fine on CPython 3.4+. We could improve this by using
+ # weakref.finalizer instead of __del__.
+ try:
+ return self._cached_tagset
+ except AttributeError:
+ self._cached_tagset = tags.ImmutableTagSet(
+ self, '_db_p', capi.lib.notmuch_database_get_all_tags)
+ return self._cached_tagset
+
+ def create_query(self, querystring):
+ raise NotImplementedError
+
+ def status_string(self):
+ raise NotImplementedError
+
+ def __repr__(self):
+ return f'Database(path={self.path}, mode={self.mode})'
+
+
+class AtomicContext:
+ """Context manager for atomic support.
+
+ This supports the notmuch_database_begin_atomic and
+ notmuch_database_end_atomic API calls. The object can not be
+ directly instantiated by the user, only via ``Database.atomic``.
+ It does keep a reference to the :class:`Database` instance to keep
+ the C memory alive.
+
+ :raises XapianError: When this is raised at enter time the atomic
+ section is not active. When it is raised at exit time the
+ atomic section is still active and you may need to try using
+ :meth:`force_end`.
+ :raises ObjectDestroyedError: if used after destoryed.
+ """
+
+ def __init__(self, db, ptr_name):
+ self._db = db
+ self._ptr = lambda: getattr(db, ptr_name)
+ self._entered = False
+
+ def __del__(self):
+ self.destroy()
+
+ @property
+ def alive(self):
+ return self.parent.alive
+
+ def destroy(self):
+ pass
+
+ def __enter__(self):
+ ret = capi.lib.notmuch_database_begin_atomic(self._ptr())
+ if ret != capi.lib.NOTMUCH_STATUS_SUCCESS:
+ raise errors.NotmuchError(ret)
+ self._entered = True
+ return self
+
+ def __exit__(self, exc_type, exc_value, traceback):
+ self._entered = False
+ ret = capi.lib.notmuch_database_end_atomic(self._ptr())
+ if ret != capi.lib.NOTMUCH_STATUS_SUCCESS:
+ raise errors.NotmuchError(ret)
+
+ def force_end(self):
+ """Force ending the atomic section.
+
+ This can only be called once __exit__ has been called. It
+ will attept to close the atomic section (again). This is
+ useful if the original exit raised an exception and the atomic
+ section is still open.
+
+ :raises XapianError: If exiting fails, the atomic section is
+ not ended.
+ :raises UnbalancedAtomicError: If the database was currently
+ not in an atomic section.
+ :raises ObjectDestroyedError: if used after destoryed.
+ """
+ ret = capi.lib.notmuch_database_end_atomic(self._ptr())
+ if ret != capi.lib.NOTMUCH_STATUS_SUCCESS:
+ raise errors.NotmuchError(ret)
+
+
+@functools.total_ordering
+class DbRevision:
+ """A database revision.
+
+ The database revision number increases monotonically with each
+ commit to the database. Which means user-visible changes can be
+ ordered. This object is sortable with other revisions. It
+ carries the UUID of the database to ensure it is only ever
+ compared with revisions from the same database.
+ """
+
+ def __init__(self, rev, uuid):
+ self._rev = rev
+ self._uuid = uuid
+
+ @property
+ def rev(self):
+ """The revision number, a positive integer."""
+ return self._rev
+
+ @property
+ def uuid(self):
+ """The UUID of the database, consider this opaque."""
+ return self._uuid
+
+ def __eq__(self, other):
+ if isinstance(other, self.__class__):
+ if self.uuid != other.uuid:
+ return False
+ return self.rev == other.rev
+ else:
+ return NotImplemented
+
+ def __lt__(self, other):
+ if self.__class__ is other.__class__:
+ if self.uuid != other.uuid:
+ return False
+ return self.rev < other.rev
+ else:
+ return NotImplemented
+
+ def __repr__(self):
+ return f'DbRevision(rev={self.rev}, uuid={self.uuid})'
diff --git a/bindings/python-cffi/notdb/_errors.py b/bindings/python-cffi/notdb/_errors.py
new file mode 100644
index 00000000..e6a468c1
--- /dev/null
+++ b/bindings/python-cffi/notdb/_errors.py
@@ -0,0 +1,114 @@
+from notdb import _capi as capi
+
+
+class NotmuchError(Exception):
+ """Base exception for errors originating from the notmuch library.
+
+ Usually this will have two attributes:
+
+ :status: This is a numeric status code corresponding to the error
+ code in the notmuch library. This is normally fairly
+ meaningless, it can also often be ``None``. This exists mostly
+ to easily create new errors from notmuch status codes and
+ should not normally be used by users.
+
+ :message: A user-facing message for the error. This can
+ occasionally also be ``None``. Usually you'll want to call
+ ``str()`` on the error object instead to get a sensible
+ message.
+ """
+
+ @classmethod
+ def exc_type(cls, status):
+ """Return correct exception type for notmuch status."""
+ types = {
+ capi.lib.NOTMUCH_STATUS_OUT_OF_MEMORY:
+ OutOfMemoryError,
+ capi.lib.NOTMUCH_STATUS_READ_ONLY_DATABASE:
+ ReadOnlyDatabaseError,
+ capi.lib.NOTMUCH_STATUS_XAPIAN_EXCEPTION:
+ XapianError,
+ capi.lib.NOTMUCH_STATUS_FILE_ERROR:
+ FileError,
+ capi.lib.NOTMUCH_STATUS_FILE_NOT_EMAIL:
+ FileNotEmailError,
+ capi.lib.NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID:
+ DuplicateMessageIdError,
+ capi.lib.NOTMUCH_STATUS_NULL_POINTER:
+ NullPointerError,
+ capi.lib.NOTMUCH_STATUS_TAG_TOO_LONG:
+ TagTooLongError,
+ capi.lib.NOTMUCH_STATUS_UNBALANCED_FREEZE_THAW:
+ UnbalancedFreezeThawError,
+ capi.lib.NOTMUCH_STATUS_UNBALANCED_ATOMIC:
+ UnbalancedAtomicError,
+ capi.lib.NOTMUCH_STATUS_UNSUPPORTED_OPERATION:
+ UnsupportedOperationError,
+ capi.lib.NOTMUCH_STATUS_UPGRADE_REQUIRED:
+ UpgradeRequiredError,
+ capi.lib.NOTMUCH_STATUS_PATH_ERROR:
+ PathError,
+ capi.lib.NOTMUCH_STATUS_ILLEGAL_ARGUMENT:
+ IllegalArgumentError,
+ }
+ return types[status]
+
+ def __new__(cls, *args, **kwargs):
+ """Return the correct subclass based on status."""
+ # This is simplistic, but the actual __init__ will fail if the
+ # signature is wrong anyway.
+ if args:
+ status = args[0]
+ else:
+ status = kwargs.get('status', None)
+ if status and cls == NotmuchError:
+ exc = cls.exc_type(status)
+ return exc.__new__(exc, *args, **kwargs)
+ else:
+ return super().__new__(cls)
+
+ def __init__(self, status=None, message=None):
+ self.status = status
+ self.message = message
+
+ def __str__(self):
+ if self.message:
+ return self.message
+ elif self.status:
+ return capi.lib.notmuch_status_to_string(self.status)
+ else:
+ return 'Unknown error'
+
+
+class OutOfMemoryError(NotmuchError): pass
+class ReadOnlyDatabaseError(NotmuchError): pass
+class XapianError(NotmuchError): pass
+class FileError(NotmuchError): pass
+class FileNotEmailError(NotmuchError): pass
+class DuplicateMessageIdError(NotmuchError): pass
+class NullPointerError(NotmuchError): pass
+class TagTooLongError(NotmuchError): pass
+class UnbalancedFreezeThawError(NotmuchError): pass
+class UnbalancedAtomicError(NotmuchError): pass
+class UnsupportedOperationError(NotmuchError): pass
+class UpgradeRequiredError(NotmuchError): pass
+class PathError(NotmuchError): pass
+class IllegalArgumentError(NotmuchError): pass
+class NoSuchHeaderError(NotmuchError): pass
+class NoMessageError(NotmuchError, KeyError): pass
+
+
+class ObjectDestroyedError(NotmuchError):
+ """The object has already been destoryed and it's memory freed.
+
+ This occurs when :meth:`destroy` has been called on the object but
+ you still happen to have access to the object. This should not
+ normally occur since you should never call :meth:`destroy` by
+ hand.
+ """
+
+ def __str__(self):
+ if self.message:
+ return self.message
+ else:
+ return 'Memory already freed'
diff --git a/bindings/python-cffi/notdb/_message.py b/bindings/python-cffi/notdb/_message.py
new file mode 100644
index 00000000..251607bb
--- /dev/null
+++ b/bindings/python-cffi/notdb/_message.py
@@ -0,0 +1,285 @@
+import os
+import pathlib
+
+import notdb._base as base
+import notdb._capi as capi
+import notdb._errors as errors
+import notdb._tags as tags
+
+
+class Message(base.NotmuchObject):
+ """An email message stored in the notmuch database.
+
+ This should not be directly created, instead it will be returned
+ by calling methods on :class:`Database`. A message keeps a
+ reference to the database object since the database object can not
+ be released while the message is in use.
+
+ Note that this represents a message in the notmuch database. For
+ full email functionality you may want to use the :mod:`email`
+ package from Python's standard library. You could e.g. create
+ this as such::
+
+ notmuch_msg = db.get_message(msgid) # or from a query
+ parser = email.parser.BytesParser(policy=email.policy.default)
+ with notmuch_msg.path.open('rb) as fp:
+ email_msg = parser.parse(fp)
+
+ Most commonly the functionality provided by notmuch is sufficient
+ to read email however.
+
+ :param db: The database instance this message is associated with.
+ :type db: Database
+ :param msg_p: The C pointer to the ``notmuch_message_t``.
+ :type msg_p: <cdata>
+
+ :param dup: Whether the message was a duplicate on insertion.
+
+ :type dup: None or bool
+ """
+ _msg_p = base.MemoryPointer()
+
+ def __init__(self, db, msg_p):
+ self._db = db
+ self._msg_p = msg_p
+
+ @property
+ def alive(self):
+ if not self._db.alive:
+ return False
+ try:
+ self._msg_p
+ except errors.ObjectDestroyedError:
+ return False
+ else:
+ return True
+
+ def __del__(self):
+ """Destroy the message, freeing the memory.
+
+ Note that when an owning object, like the containing query, is
+ destroyed the messages also get destoryed.
+ """
+ self.destroy()
+
+ def destroy(self):
+ """Destroy the object and all children.
+
+ This will destroy the object, freeing all memory for it and
+ it's children. You should not normally need to call this, it
+ will be called automatically by Python garbage collection.
+
+ The main reason for it's existence is for parent objects being
+ able to destroy their children, this is required when Python's
+ garbage collection does not guarantee ordered deletion,
+ e.g. at intepreter shutdown.
+
+ :param parent: Used by the parent to indicate it has already
+ destroyed itself, and thus all it's children. In this case
+ this object only marks itself as already destroyed to avoid
+ double freeing memory.
+ """
+ if self.alive:
+ capi.lib.notmuch_message_destroy(self._msg_p)
+ self._msg_p = None
+
+ @property
+ def messageid(self):
+ """The message ID as a string.
+
+ The message ID is decoded with the ignore error handler. This
+ is fine as long as the message ID is well formed. If it is
+ not valid ASCII then this will be lossy. So if you need to be
+ able to write the exact same message ID back you should use
+ :attr:`messageidb`.
+
+ Note that notmuch will decode the message ID value and thus
+ strip off the surrounding ``<`` and ``>`` characters. This is
+ different from Python's :mod:`email` package behaviour which
+ leaves these characters in place.
+
+ :returns: The message ID.
+ :rtype: :class:`BinString`, this is a normal str but calling
+ bytes() on it will return the original bytes used to create
+ it.
+
+ :raises ObjectDestroyedError: if used after destoryed.
+ """
+ ret = capi.lib.notmuch_message_get_message_id(self._msg_p)
+ return base.BinString(capi.ffi.string(ret))
+
+ @property
+ def threadid(self):
+ """The thread ID.
+
+ The thread ID is decoded with the surrogateescape error
+ handler so that it is possible to reconstruct the original
+ thread ID if it is not valid UTF-8.
+ """
+ raise NotImplementedError
+
+ @property
+ def path(self):
+ """A pathname of the message as a pathlib.Path instance.
+
+ If multiple files in the database contain the same message ID
+ this will be just one of the files, chosen at random.
+
+ :raises ObjectDestroyedError: if used after destoryed.
+ """
+ ret = capi.lib.notmuch_message_get_filename(self._msg_p)
+ return pathlib.Path(os.fsdecode(capi.ffi.string(ret)))
+
+ @property
+ def pathb(self):
+ """A pathname of the message as a bytes object.
+
+ See :attr:`path` for details, this is the same but does return
+ the path as a bytes object which is faster but less convenient.
+
+ :raises ObjectDestroyedError: if used after destoryed.
+ """
+ ret = capi.lib.notmuch_message_get_filename(self._msg_p)
+ return capi.ffi.string(ret)
+
+ def pathnames(self):
+ """Return an iterator of all files for this message.
+
+ If multiple files contained the same message ID they will all
+ be returned here. The files are returned as intances of
+ :class:`pathlib.Path`.
+
+ :raises ObjectDestroyedError: if used after destoryed.
+ """
+ raise NotImplementedError
+
+ def pathnamesb(self):
+ """Return an iterator of all files for this message.
+
+ This is like :meth:`pathnames` but the files are returned as
+ byte objects instead.
+
+ :raises ObjectDestroyedError: if used after destoryed.
+ """
+ raise NotImplementedError
+
+ @property
+ def ghost(self):
+ """Indicates whether this message is a ghost message.
+
+ A ghost message if a message which we know exists, but it has
+ no files or content associated with it. This can happen if
+ it was referenced by some other message. Only the
+ :attr:`messageid` and :attr:`threadid` attributes are valid
+ for it.
+ """
+ raise NotImplementedError
+
+ @property
+ def date(self):
+ """The message date.
+
+ XXX Figure out which format to provide this in.
+ """
+ raise NotImplementedError
+
+ def header(self, name):
+ """Return the value of the named header.
+
+ Returns the header from notmuch, some common headers are
+ stored in the database, others are read from the file.
+ Headers are returned with their newlines stripped and
+ collapsed concatenated together if they occur multiple times.
+ You may be better off using the standard library email
+ package's ``email.message_from_file(msg.path.open())`` if that
+ is not sufficient for you.
+
+ :param header: Case-insensitive header name to retrieve.
+ :type header: str
+
+ :returns: The header value, an empty string if the header is
+ not present.
+ :rtype: str
+
+ :raises NoSuchHeaderError: if the header is not present.
+ """
+ raise NotImplementedError
+
+ @property
+ def tags(self):
+ """The tags associated with the message.
+
+ This behaves as a set. But removing and adding items to the
+ set removes and adds them to the message in the database.
+ """
+ # By caching the tagset this creates a circular reference.
+ # This is fine on CPython 3.4+. We could improve this by using
+ # weakref.finalizer instead of __del__.
+ try:
+ return self._cached_tagset
+ except AttributeError:
+ self._cached_tagset = tags.MutableTagSet(
+ self, '_msg_p', capi.lib.notmuch_message_get_tags)
+ return self._cached_tagset
+
+ def tags_to_flags(self):
+ """Sync the notmuch tags to maildir flags.
+
+ This will rename the pathname of the message so that the
+ maildir flags match the current set of notmuch tags. The
+ mappings are:
+
+ flag tag
+ ----- --------------
+ ``D`` ``draft``
+ ``F`` ``flagged``
+ ``P`` ``passed``
+ ``R`` ``replied``
+ ``S`` not ``unread``
+
+ Any other flags are preserved in the renaming. If the
+ existing flag format is invalid, e.g. flags repeated, not in
+ ASCII order file not ending in ``:2,``, the file is not
+ renamed.
+ """
+ raise NotImplementedError
+
+ def flags_to_tags(self):
+ """Sync the maildir flags to notmuch tags.
+
+ This synchronizes the opposite way as described in
+ :meth:`rags_to_flags`.
+ """
+ raise NotImplementedError
+
+ def frozen(self):
+ """Context manager to freeze the message state.
+
+ This allows you to perform atomic tag updates::
+
+ with msg.frozen():
+ msg.tags.clear()
+ msg.tags.add('foo')
+
+ Using This would ensure the message never ends up with no tags
+ applied at all.
+
+ It is safe to nest calls to this context manager.
+ """
+ raise NotImplementedError
+
+ @property
+ def properties(self):
+ """A map of arbitrary key-value pairs associated with the message.
+
+ Be aware that properties may be used by other extensions to
+ store state in. So delete or modify with care.
+ """
+ raise NotImplementedError
+
+
+class MessageProperties:
+ # XXX This will need to decide what to do with encoding. Easiest
+ # is to store bytes and leave it to the user to call .encode()
+ # .decode().
+ pass
diff --git a/bindings/python-cffi/notdb/_tags.py b/bindings/python-cffi/notdb/_tags.py
new file mode 100644
index 00000000..1955688d
--- /dev/null
+++ b/bindings/python-cffi/notdb/_tags.py
@@ -0,0 +1,319 @@
+import collections.abc
+
+import notdb._base as base
+import notdb._capi as capi
+import notdb._errors as errors
+
+
+class ImmutableTagSet(base.NotmuchObject, collections.abc.Set):
+ """The tags associated with a message thread or whole database.
+
+ Both a thread as well as the database expose the union of all tags
+ in messages associated with them. This exposes these as a
+ :class:`collections.abc.Set` object.
+
+ Note that due to the underlying notmuch API the performance of the
+ implementation is not the same as you would expect from normal
+ sets. E.g. the :meth:`__contains__` and :meth:`__len__` are O(n)
+ rather then O(1).
+
+ Tags are internally stored as bytestrings but normally exposed as
+ unicode strings using the UTF-8 encoding and the *ignore* decoder
+ error handler. However the :meth:`iter` method can be used to
+ return tags as bytestrings or using a different error handler.
+
+ :param parent: the parent object
+ :param ptr_name: the name of the attribute on the parent which will
+ return the memory pointer. This allows this object to
+ access the pointer via the parent's descriptor and thus
+ trigger :class:`MemoryPointer`'s memory safety.
+ :param cffi_fn: the callable CFFI wrapper to retrieve the tags
+ iter. This can be one of notmuch_database_get_all_tags,
+ notmuch_thread_get_tags or notmuch_message_get_tags.
+ """
+
+ def __init__(self, parent, ptr_name, cffi_fn):
+ self._parent = parent
+ self._ptr = lambda: getattr(parent, ptr_name)
+ self._cffi_fn = cffi_fn
+
+ def __del__(self):
+ self.destroy()
+
+ @property
+ def alive(self):
+ return self._parent.alive
+
+ def destroy(self):
+ pass
+
+ def __iter__(self):
+ """Return an iterator over the tags.
+
+ Tags are yielded as unicode strings, decoded using the
+ "ignore" error handler.
+
+ :raises NullPointerError: If the iterator can not be created.
+ """
+ return self.iter(encoding='utf-8', errors='ignore')
+
+ def iter(self, *, encoding=None, errors='strict'):
+ """Aternate iterator constructor controlling string decoding.
+
+ Tags are stored as bytes in the notmuch database, in Python
+ it's easier to work with unicode strings and thus is what the
+ normal iterator returns. However this method allows you to
+ specify how you would like to get the tags, defaulting to the
+ bytestring representation instead of unicode strings.
+
+ :param encoding: Which codec to use. The default *None* does not
+ decode at all and will return the unmodified bytes.
+ Otherwise this is passed on to :func:`str.decode`.
+ :param errors: If using a codec, this is the error handler.
+ See :func:`str.decode` to which this is passed on.
+
+ :raises NullPointerError: When things do not go as planned.
+ """
+ # self._cffi_fn should point either to
+ # notmuch_database_get_all_tags, notmuch_thread_get_tags or
+ # notmuch_message_get_tags. nothmuch.h suggests these never
+ # fail, let's handle NULL anyway.
+ tags_p = self._cffi_fn(self._ptr())
+ if tags_p == capi.ffi.NULL:
+ raise errors.NullPointerError()
+ tags = TagsIter(self, tags_p, encoding=encoding, errors=errors)
+ return tags
+
+ def __len__(self):
+ return sum(1 for t in self)
+
+ def __contains__(self, tag):
+ if isinstance(tag, str):
+ tag = tag.encode()
+ for msg_tag in self.iter():
+ if tag == msg_tag:
+ return True
+ else:
+ return False
+
+ def __eq__(self, other):
+ return tuple(sorted(self.iter())) == tuple(sorted(other.iter()))
+
+ def __hash__(self):
+ return hash(tuple(self.iter()))
+
+
+class MutableTagSet(ImmutableTagSet, collections.abc.MutableSet):
+ """The tags associated with a message.
+
+ This is a :class:`collections.abc.MutableSet` object which can be
+ used to manipulate the tags of a message.
+
+ Note that due to the underlying notmuch API the performance of the
+ implementation is not the same as you would expect from normal
+ sets. E.g. the ``in`` operator and variants are O(n) rather then
+ O(1).
+
+ Tags are bytestrings and calling ``iter()`` will return an
+ iterator yielding bytestrings. However the :meth:`iter` method
+ can be used to return tags as unicode strings, while all other
+ operations accept either byestrings or unicode strings. In case
+ unicode strings are used they will be encoded using utf-8 before
+ being passed to notmuch.
+ """
+
+ # Since we subclass ImmutableTagSet we inherit a __hash__. But we
+ # are mutable, setting it to None will make the Python machinary
+ # recognise us as unhashable.
+ __hash__ = None
+
+ def add(self, tag):
+ """Add a tag to the message.
+
+ :param tag: The tag to add.
+ :type tag: str or bytes. A str will be encoded using UTF-8.
+
+ :param sync_flags: Whether to sync the maildir flags with the
+ new set of tags. Leaving this as *None* respects the
+ configuration set in the database, while *True* will always
+ sync and *False* will never sync.
+ :param sync_flags: NoneType or bool
+
+ :raises TypeError: If the tag is not a valid type.
+ :raises TagTooLongError: If the added tag exceeds the maximum
+ lenght, see ``notmuch_cffi.NOTMUCH_TAG_MAX``.
+ :raises ReadOnlyDatabaseError: If the database is opened in
+ read-only mode.
+ """
+ if isinstance(tag, str):
+ tag = tag.encode()
+ if not isinstance(tag, bytes):
+ raise TypeError('Not a valid type for a tag: {}'.format(type(tag)))
+ ret = capi.lib.notmuch_message_add_tag(self._ptr(), tag)
+ if ret != capi.lib.NOTMUCH_STATUS_SUCCESS:
+ raise errors.NotmuchError(ret)
+
+ def discard(self, tag):
+ """Remove a tag from the message.
+
+ :param tag: The tag to remove.
+ :type tag: str of bytes. A str will be encoded using UTF-8.
+ :param sync_flags: Whether to sync the maildir flags with the
+ new set of tags. Leaving this as *None* respects the
+ configuration set in the database, while *True* will always
+ sync and *False* will never sync.
+ :param sync_flags: NoneType or bool
+
+ :raises TypeError: If the tag is not a valid type.
+ :raises TagTooLongError: If the tag exceeds the maximum
+ lenght, see ``notmuch_cffi.NOTMUCH_TAG_MAX``.
+ :raises ReadOnlyDatabaseError: If the database is opened in
+ read-only mode.
+ """
+ if isinstance(tag, str):
+ tag = tag.encode()
+ if not isinstance(tag, bytes):
+ raise TypeError('Not a valid type for a tag: {}'.format(type(tag)))
+ ret = capi.lib.notmuch_message_remove_tag(self._ptr(), tag)
+ if ret != capi.lib.NOTMUCH_STATUS_SUCCESS:
+ raise errors.NotmuchError(ret)
+
+ def clear(self):
+ """Remove all tags from the message.
+
+ :raises ReadOnlyDatabaseError: If the database is opened in
+ read-only mode.
+ """
+ ret = capi.lib.notmuch_message_remove_all_tags(self._ptr())
+ if ret != capi.lib.NOTMUCH_STATUS_SUCCESS:
+ raise errors.NoMessageError(ret)
+
+ def from_maildir_flags(self):
+ """Update the tags based on the state in the message's maildir flags.
+
+ This function examines the filenames of 'message' for maildir
+ flags, and adds or removes tags on 'message' as follows when
+ these flags are present:
+
+ Flag Action if present
+ ---- -----------------
+ 'D' Adds the "draft" tag to the message
+ 'F' Adds the "flagged" tag to the message
+ 'P' Adds the "passed" tag to the message
+ 'R' Adds the "replied" tag to the message
+ 'S' Removes the "unread" tag from the message
+
+ For each flag that is not present, the opposite action
+ (add/remove) is performed for the corresponding tags.
+
+ Flags are identified as trailing components of the filename
+ after a sequence of ":2,".
+
+ If there are multiple filenames associated with this message,
+ the flag is considered present if it appears in one or more
+ filenames. (That is, the flags from the multiple filenames are
+ combined with the logical OR operator.)
+ """
+ ret = capi.lib.notmuch_message_maildir_flags_to_tags(self._ptr())
+ if ret != capi.lib.NOTMUCH_STATUS_SUCCESS:
+ raise errors.NotmuchError(ret)
+
+ def to_maildir_flags(self):
+ """Update the message's maildir flags based on the notmuch tags.
+
+ If the message's filename is in a maildir directory, that is a
+ directory named ``new`` or ``cur``, and has a valid maildir
+ filename then the flags will be added as such:
+
+ 'D' if the message has the "draft" tag
+ 'F' if the message has the "flagged" tag
+ 'P' if the message has the "passed" tag
+ 'R' if the message has the "replied" tag
+ 'S' if the message does not have the "unread" tag
+
+ Any existing flags unmentioned in the list above will be
+ preserved in the renaming.
+
+ Also, if this filename is in a directory named "new", rename it to
+ be within the neighboring directory named "cur".
+
+ In case there are multiple files associated with the message
+ all filenames will get the same logic applied.
+ """
+ ret = capi.lib.notmuch_message_tags_to_maildir_flags(self._ptr())
+ if ret != capi.lib.NOTMUCH_STATUS_SUCCESS:
+ raise errors.NotmuchError(ret)
+
+
+class TagsIter(base.NotmuchObject):
+ """Iterator over tags.
+
+ This is only an interator, not a container so calling
+ :meth:`__iter__` does not return a new, replenished iterator but
+ only itself.
+
+ :param parent: The parent object to keep alive.
+ :param tags_p: The CFFI pointer to the C-level tags iterator.
+ :param encoding: Which codec to use. The default *None* does not
+ decode at all and will return the unmodified bytes.
+ Otherwise this is passed on to :func:`str.decode`.
+ :param errors: If using a codec, this is the error handler.
+ See :func:`str.decode` to which this is passed on.
+
+ :raises ObjectDestoryedError: if used after destroyed.
+ """
+ _tags_p = base.MemoryPointer()
+
+ def __init__(self, parent, tags_p, *, encoding=None, errors='strict'):
+ self._parent = parent
+ self._tags_p = tags_p
+ self._encoding = encoding
+ self._errors = errors
+
+ def __del__(self):
+ self.destroy()
+
+ @property
+ def alive(self):
+ if not self._parent.alive:
+ return False
+ try:
+ self._tags_p
+ except errors.ObjectDestroyedError:
+ return False
+ else:
+ return True
+
+ def destroy(self):
+ if self.alive:
+ try:
+ capi.lib.notmuch_tags_destroy(self._tags_p)
+ except errors.ObjectDestroyedError:
+ pass
+ self._tags_p = None
+
+ def __iter__(self):
+ """Return the iterator itself.
+
+ Note that as this is an iterator and not a container this will
+ not return a new iterator. Thus any elements already consumed
+ will not be yielded by the :meth:`__next__` method anymore.
+ """
+ return self
+
+ def __next__(self):
+ if not capi.lib.notmuch_tags_valid(self._tags_p):
+ self.destroy()
+ raise StopIteration()
+ tag_p = capi.lib.notmuch_tags_get(self._tags_p)
+ tag = capi.ffi.string(tag_p)
+ if self._encoding:
+ tag = tag.decode(encoding=self._encoding, errors=self._errors)
+ capi.lib.notmuch_tags_move_to_next(self._tags_p)
+ return tag
+
+ def __repr__(self):
+ if self._tags_p is None:
+ return '<TagsIter (exhausted)>'
+ else:
+ return '<TagsIter>'
diff --git a/bindings/python-cffi/setup.py b/bindings/python-cffi/setup.py
new file mode 100644
index 00000000..9a8693d6
--- /dev/null
+++ b/bindings/python-cffi/setup.py
@@ -0,0 +1,13 @@
+import setuptools
+
+
+setuptools.setup(
+ name='notdb',
+ version='0.1',
+ description='Pythonic bindings for the notmuch mail database using CFFI',
+ author='Floris Bruynooghe',
+ author_email='flub@devork.be',
+ setup_requires=['cffi>=1.0.0'],
+ cffi_modules=['notdb/_build.py:ffibuilder'],
+ install_requires=['cffi>=1.0.0'],
+)
diff --git a/bindings/python-cffi/tests/conftest.py b/bindings/python-cffi/tests/conftest.py
new file mode 100644
index 00000000..b9173762
--- /dev/null
+++ b/bindings/python-cffi/tests/conftest.py
@@ -0,0 +1,138 @@
+import email.message
+import mailbox
+import pathlib
+import socket
+import subprocess
+import textwrap
+import time
+
+import pytest
+
+
+@pytest.fixture(scope='function')
+def tmppath(tmpdir):
+ """The tmpdir fixture wrapped in pathlib.Path."""
+ return pathlib.Path(str(tmpdir))
+
+
+@pytest.fixture
+def notmuch(maildir):
+ """Return a function which runs notmuch commands on our test maildir.
+
+ This uses the notmuch-config file created by the ``maildir``
+ fixture.
+ """
+ def run(*args):
+ """Run a notmuch comand.
+
+ This function runs with a timeout error as many notmuch
+ commands may block if multiple processes are trying to open
+ the database in write-mode. It is all too easy to
+ accidentally do this in the unittests.
+ """
+ cfg_fname = maildir.path / 'notmuch-config'
+ cmd = ['notmuch'] + list(args)
+ print('Invoking: {}'.format(' '.join(cmd)))
+ proc = subprocess.run(cmd,
+ timeout=5,
+ env={'NOTMUCH_CONFIG': str(cfg_fname)})
+ proc.check_returncode()
+ return run
+
+
+@pytest.fixture
+def maildir(tmppath):
+ """A basic test interface to a valid maildir directory.
+
+ This creates a valid maildir and provides a simple mechanism to
+ deliver test emails to it. It also writes a notmuch-config file
+ in the top of the maildir.
+ """
+ cur = tmppath / 'cur'
+ cur.mkdir()
+ new = tmppath / 'new'
+ new.mkdir()
+ tmp = tmppath / 'tmp'
+ tmp.mkdir()
+ cfg_fname = tmppath/'notmuch-config'
+ with cfg_fname.open('w') as fp:
+ fp.write(textwrap.dedent(f"""\
+ [database]
+ path={tmppath!s}
+ [user]
+ name=Some Hacker
+ primary_email=dst@example.com
+ [new]
+ tags=unread;inbox;
+ ignore=
+ [search]
+ exclude_tags=deleted;spam;
+ [maildir]
+ synchronize_flags=true
+ [crypto]
+ gpg_path=gpg
+ """))
+ return MailDir(tmppath)
+
+
+class MailDir:
+ """An interface around a correct maildir."""
+
+ def __init__(self, path):
+ self._path = pathlib.Path(path)
+ self.mailbox = mailbox.Maildir(path)
+ self._idcount = 0
+
+ @property
+ def path(self):
+ """The pathname of the maildir."""
+ return self._path
+
+ def _next_msgid(self):
+ """Return a new unique message ID."""
+ msgid = '{}@{}'.format(self._idcount, socket.getfqdn())
+ self._idcount += 1
+ return msgid
+
+ def deliver(self,
+ subject='Test mail',
+ body='This is a test mail',
+ to='dst@example.com',
+ frm='src@example.com',
+ new=False, # Move to new dir or cur dir?
+ keywords=None, # List of keywords or labels
+ seen=False, # Seen flag (cur dir only)
+ replied=False, # Replied flag (cur dir only)
+ flagged=False): # Flagged flag (cur dir only)
+ """Deliver a new mail message in the mbox.
+
+ This does only adds the message to maildir, does not insert it
+ into the notmuch database.
+
+ :returns: A tuple of (msgid, pathname).
+ """
+ msgid = self._next_msgid()
+ when = time.time()
+ msg = email.message.EmailMessage()
+ msg.add_header('Received', 'by MailDir; {}'.format(time.ctime(when)))
+ msg.add_header('Message-ID', f'<{msgid}>') # header encoder bug?
+ msg.add_header('Date', time.ctime(when))
+ msg.add_header('From', frm)
+ msg.add_header('To', to)
+ msg.add_header('Subject', subject)
+ msg.set_content(body)
+ mdmsg = mailbox.MaildirMessage(msg)
+ if not new:
+ mdmsg.set_subdir('cur')
+ if flagged:
+ mdmsg.add_flag('F')
+ if replied:
+ mdmsg.add_flag('R')
+ if seen:
+ mdmsg.add_flag('S')
+ boxid = self.mailbox.add(mdmsg)
+ basename = boxid
+ if mdmsg.get_info():
+ basename += mailbox.Maildir.colon + mdmsg.get_info()
+ msgpath = self.path / mdmsg.get_subdir() / basename
+ return (msgid, msgpath)
diff --git a/bindings/python-cffi/tests/test_base.py b/bindings/python-cffi/tests/test_base.py
new file mode 100644
index 00000000..03df990f
--- /dev/null
+++ b/bindings/python-cffi/tests/test_base.py
@@ -0,0 +1,116 @@
+import pytest
+
+from notdb import _base as base
+from notdb import _errors as errors
+
+
+class TestNotmuchObject:
+
+ def test_no_impl_methods(self):
+ class Object(base.NotmuchObject):
+ pass
+ with pytest.raises(TypeError):
+ Object()
+
+ def test_impl_methods(self):
+
+ class Object(base.NotmuchObject):
+
+ def __init__(self):
+ pass
+
+ @property
+ def alive(self):
+ pass
+
+ def destroy(self, parent=False):
+ pass
+
+ Object()
+
+ def test_del(self):
+ destroyed = False
+
+ class Object(base.NotmuchObject):
+
+ def __init__(self):
+ pass
+
+ @property
+ def alive(self):
+ pass
+
+ def destroy(self, parent=False):
+ nonlocal destroyed
+ destroyed = True
+
+ o = Object()
+ o.__del__()
+ assert destroyed
+
+
+class TestMemoryPointer:
+
+ @pytest.fixture
+ def obj(self):
+ class Cls:
+ ptr = base.MemoryPointer()
+ return Cls()
+
+ def test_unset(self, obj):
+ with pytest.raises(errors.ObjectDestroyedError):
+ obj.ptr
+
+ def test_set(self, obj):
+ obj.ptr = 'some'
+ assert obj.ptr == 'some'
+
+ def test_cleared(self, obj):
+ obj.ptr = 'some'
+ obj.ptr
+ obj.ptr = None
+ with pytest.raises(errors.ObjectDestroyedError):
+ obj.ptr
+
+ def test_two_instances(self, obj):
+ obj2 = obj.__class__()
+ obj.ptr = 'foo'
+ obj2.ptr = 'bar'
+ assert obj.ptr != obj2.ptr
+
+
+class TestBinString:
+
+ def test_type(self):
+ s = base.BinString(b'foo')
+ assert isinstance(s, str)
+
+ def test_init_bytes(self):
+ s = base.BinString(b'foo')
+ assert s == 'foo'
+
+ def test_init_str(self):
+ s = base.BinString('foo')
+ assert s == 'foo'
+
+ def test_bytes(self):
+ s = base.BinString(b'foo')
+ assert bytes(s) == b'foo'
+
+ def test_invalid_utf8(self):
+ s = base.BinString(b'\x80foo')
+ assert s == 'foo'
+ assert bytes(s) == b'\x80foo'
+
+ def test_errors(self):
+ s = base.BinString(b'\x80foo', errors='replace')
+ assert s == '�foo'
+ assert bytes(s) == b'\x80foo'
+
+ def test_encoding(self):
+ # pound sign: '£' == '\u00a3' latin-1: b'\xa3', utf-8: b'\xc2\xa3'
+ with pytest.raises(UnicodeDecodeError):
+ base.BinString(b'\xa3', errors='strict')
+ s = base.BinString(b'\xa3', encoding='latin-1', errors='strict')
+ assert s == '£'
+ assert bytes(s) == b'\xa3'
diff --git a/bindings/python-cffi/tests/test_database.py b/bindings/python-cffi/tests/test_database.py
new file mode 100644
index 00000000..9adec03e
--- /dev/null
+++ b/bindings/python-cffi/tests/test_database.py
@@ -0,0 +1,274 @@
+import collections
+import configparser
+import os
+import pathlib
+
+import pytest
+
+import notdb._errors as errors
+import notdb._database as dbmod
+import notdb._message as message
+
+
+@pytest.fixture
+def db(maildir):
+ with dbmod.Database.create(maildir.path) as db:
+ yield db
+
+
+class TestDefaultDb:
+ """Tests for reading the default database.
+
+ The error cases are fairly undefined, some relevant Python error
+ will come out if you give it a bad filename or if the file does
+ not parse correctly. So we're not testing this too deeply.
+ """
+
+ def test_config_pathname_default(self, monkeypatch):
+ monkeypatch.delenv('NOTMUCH_CONFIG', raising=False)
+ user = pathlib.Path('~/.notmuch-config').expanduser()
+ assert dbmod._config_pathname() == user
+
+ def test_config_pathname_env(self, monkeypatch):
+ monkeypatch.setenv('NOTMUCH_CONFIG', '/some/random/path')
+ assert dbmod._config_pathname() == pathlib.Path('/some/random/path')
+
+ def test_default_path_nocfg(self, monkeypatch, tmppath):
+ monkeypatch.setenv('NOTMUCH_CONFIG', str(tmppath/'foo'))
+ with pytest.raises(FileNotFoundError):
+ dbmod.Database.default_path()
+
+ def test_default_path_cfg_is_dir(self, monkeypatch, tmppath):
+ monkeypatch.setenv('NOTMUCH_CONFIG', str(tmppath))
+ with pytest.raises(IsADirectoryError):
+ dbmod.Database.default_path()
+
+ def test_default_path_parseerr(self, monkeypatch, tmppath):
+ cfg = tmppath / 'notmuch-config'
+ with cfg.open('w') as fp:
+ fp.write('invalid')
+ monkeypatch.setenv('NOTMUCH_CONFIG', str(cfg))
+ with pytest.raises(configparser.Error):
+ dbmod.Database.default_path()
+
+ def test_default_path_parse(self, monkeypatch, tmppath):
+ cfg = tmppath / 'notmuch-config'
+ with cfg.open('w') as fp:
+ fp.write('[database]\n')
+ fp.write('path={!s}'.format(tmppath))
+ monkeypatch.setenv('NOTMUCH_CONFIG', str(cfg))
+ assert dbmod.Database.default_path() == tmppath
+
+ def test_default_path_param(self, monkeypatch, tmppath):
+ cfg_dummy = tmppath / 'dummy'
+ monkeypatch.setenv('NOTMUCH_CONFIG', str(cfg_dummy))
+ cfg_real = tmppath / 'notmuch_config'
+ with cfg_real.open('w') as fp:
+ fp.write('[database]\n')
+ fp.write('path={!s}'.format(cfg_real/'mail'))
+ assert dbmod.Database.default_path(cfg_real) == cfg_real/'mail'
+
+
+class TestCreate:
+
+ def test_create(self, tmppath, db):
+ assert tmppath.joinpath('.notmuch/xapian/').exists()
+
+ def test_create_already_open(self, tmppath, db):
+ with pytest.raises(errors.NotmuchError):
+ db.create(tmppath)
+
+ def test_create_existing(self, tmppath, db):
+ with pytest.raises(errors.FileError):
+ dbmod.Database.create(path=tmppath)
+
+ def test_close(self, db):
+ db.close()
+
+ def test_del_noclose(self, db):
+ del db
+
+ def test_close_del(self, db):
+ db.close()
+ del db
+
+ def test_closed_attr(self, db):
+ assert not db.closed
+ db.close()
+ assert db.closed
+
+ def test_ctx(self, db):
+ with db as ctx:
+ assert ctx is db
+ assert not db.closed
+ assert db.closed
+
+ def test_path(self, db, tmppath):
+ assert db.path == tmppath
+
+ def test_version(self, db):
+ assert db.version > 0
+
+ def test_needs_upgrade(self, db):
+ assert db.needs_upgrade in (True, False)
+
+
+class TestAtomic:
+
+ def test_exit_early(self, db):
+ with pytest.raises(errors.UnbalancedAtomicError):
+ with db.atomic() as ctx:
+ ctx.force_end()
+
+ def test_exit_late(self, db):
+ with db.atomic() as ctx:
+ pass
+ with pytest.raises(errors.UnbalancedAtomicError):
+ ctx.force_end()
+
+
+class TestRevision:
+
+ def test_single_rev(self, db):
+ r = db.revision()
+ assert isinstance(r, dbmod.DbRevision)
+ assert isinstance(r.rev, int)
+ assert isinstance(r.uuid, bytes)
+ assert r is r
+ assert r == r
+ assert r <= r
+ assert r >= r
+ assert not r < r
+ assert not r > r
+
+ def test_diff_db(self, tmppath):
+ dbpath0 = tmppath.joinpath('db0')
+ dbpath0.mkdir()
+ dbpath1 = tmppath.joinpath('db1')
+ dbpath1.mkdir()
+ db0 = dbmod.Database.create(path=dbpath0)
+ db1 = dbmod.Database.create(path=dbpath1)
+ r_db0 = db0.revision()
+ r_db1 = db1.revision()
+ assert r_db0 != r_db1
+ assert r_db0.uuid != r_db1.uuid
+
+ def test_cmp(self, db, maildir):
+ rev0 = db.revision()
+ _, pathname = maildir.deliver()
+ db.add(pathname, sync_flags=False)
+ rev1 = db.revision()
+ assert rev0 < rev1
+ assert rev0 <= rev1
+ assert not rev0 > rev1
+ assert not rev0 >= rev1
+ assert not rev0 == rev1
+ assert rev0 != rev1
+
+ # XXX add tests for revisions comparisons
+
+class TestMessages:
+
+ def test_add_message(self, db, maildir):
+ msgid, pathname = maildir.deliver()
+ msg, dup = db.add(pathname, sync_flags=False)
+ assert isinstance(msg, message.Message)
+ assert msg.path == pathname
+ assert msg.messageid == msgid
+
+ def test_add_message_str(self, db, maildir):
+ msgid, pathname = maildir.deliver()
+ msg, dup = db.add(str(pathname), sync_flags=False)
+
+ def test_add_message_bytes(self, db, maildir):
+ msgid, pathname = maildir.deliver()
+ msg, dup = db.add(os.fsencode(pathname), sync_flags=False)
+
+ def test_remove_message(self, db, maildir):
+ msgid, pathname = maildir.deliver()
+ msg, dup = db.add(pathname, sync_flags=False)
+ assert db.find(msgid)
+ dup = db.remove(pathname)
+ with pytest.raises(errors.NoMessageError):
+ db.find(msgid)
+
+ def test_remove_message_str(self, db, maildir):
+ msgid, pathname = maildir.deliver()
+ msg, dup = db.add(pathname, sync_flags=False)
+ assert db.find(msgid)
+ dup = db.remove(str(pathname))
+ with pytest.raises(errors.NoMessageError):
+ db.find(msgid)
+
+ def test_remove_message_bytes(self, db, maildir):
+ msgid, pathname = maildir.deliver()
+ msg, dup = db.add(pathname, sync_flags=False)
+ assert db.find(msgid)
+ dup = db.remove(os.fsencode(pathname))
+ with pytest.raises(errors.NoMessageError):
+ db.find(msgid)
+
+ def test_find_message(self, db, maildir):
+ msgid, pathname = maildir.deliver()
+ msg0, dup = db.add(pathname, sync_flags=False)
+ msg1 = db.find(msgid)
+ assert isinstance(msg1, message.Message)
+ assert msg1.messageid == msgid == msg0.messageid
+ assert msg1.path == pathname == msg0.path
+
+ def test_find_message_notfound(self, db):
+ with pytest.raises(errors.NoMessageError):
+ db.find('foo')
+
+ def test_get_message(self, db, maildir):
+ msgid, pathname = maildir.deliver()
+ msg0, _ = db.add(pathname, sync_flags=False)
+ msg1 = db.get(pathname)
+ assert isinstance(msg1, message.Message)
+ assert msg1.messageid == msgid == msg0.messageid
+ assert msg1.path == pathname == msg0.path
+
+ def test_get_message_str(self, db, maildir):
+ msgid, pathname = maildir.deliver()
+ db.add(pathname, sync_flags=False)
+ msg = db.get(str(pathname))
+ assert msg.messageid == msgid
+
+ def test_get_message_bytes(self, db, maildir):
+ msgid, pathname = maildir.deliver()
+ db.add(pathname, sync_flags=False)
+ msg = db.get(os.fsencode(pathname))
+ assert msg.messageid == msgid
+
+
+class TestTags:
+ # We just want to test this behaves like a set at a hight level.
+ # The set semantics are tested in detail in the test_tags module.
+
+ def test_type(self, db):
+ assert isinstance(db.tags, collections.abc.Set)
+
+ def test_none(self, db):
+ itags = iter(db.tags)
+ with pytest.raises(StopIteration):
+ next(itags)
+ assert len(db.tags) == 0
+ assert not db.tags
+
+ def test_some(self, db, maildir):
+ _, pathname = maildir.deliver()
+ msg, _ = db.add(pathname, sync_flags=False)
+ msg.tags.add('hello')
+ itags = iter(db.tags)
+ assert next(itags) == 'hello'
+ with pytest.raises(StopIteration):
+ next(itags)
+ assert 'hello' in msg.tags
+
+ def test_cache(self, db):
+ assert db.tags is db.tags
+
+ def test_iters(self, db):
+ i1 = iter(db.tags)
+ i2 = iter(db.tags)
+ assert i1 is not i2
diff --git a/bindings/python-cffi/tests/test_tags.py b/bindings/python-cffi/tests/test_tags.py
new file mode 100644
index 00000000..e39a4931
--- /dev/null
+++ b/bindings/python-cffi/tests/test_tags.py
@@ -0,0 +1,141 @@
+"""Tests for the behaviour of immutable and mutable tagsets.
+
+This module tests the Pythonic behaviour of the sets.
+"""
+
+import collections
+import subprocess
+import textwrap
+
+import pytest
+
+from notdb import _database as database
+from notdb import _tags as tags
+
+
+class TestImmutable:
+
+ @pytest.fixture
+ def tagset(self, maildir, notmuch):
+ """An non-empty immutable tagset.
+
+ This will have the default new mail tags: inbox, unread.
+ """
+ maildir.deliver()
+ notmuch('new')
+ with database.Database(maildir.path) as db:
+ yield db.tags
+
+ def test_type(self, tagset):
+ assert isinstance(tagset, tags.ImmutableTagSet)
+ assert isinstance(tagset, collections.abc.Set)
+
+ def test_hash(self, tagset, maildir, notmuch):
+ h0 = hash(tagset)
+ notmuch('tag', '+foo', '*')
+ with database.Database(maildir.path) as db:
+ h1 = hash(db.tags)
+ assert h0 != h1
+
+ def test_eq(self, tagset):
+ assert tagset == tagset
+
+ def test_neq(self, tagset, maildir, notmuch):
+ notmuch('tag', '+foo', '*')
+ with database.Database(maildir.path) as db:
+ assert tagset != db.tags
+
+ def test_contains(self, tagset):
+ print(tuple(tagset))
+ assert 'unread' in tagset
+ assert 'foo' not in tagset
+
+ def test_iter(self, tagset):
+ expected = sorted(['unread', 'inbox'])
+ found = []
+ for tag in tagset:
+ assert isinstance(tag, str)
+ found.append(tag)
+ assert expected == sorted(found)
+
+ def test_special_iter(self, tagset):
+ expected = sorted([b'unread', b'inbox'])
+ found = []
+ for tag in tagset.iter():
+ assert isinstance(tag, bytes)
+ found.append(tag)
+ assert expected == sorted(found)
+
+ def test_special_iter_codec(self, tagset):
+ for tag in tagset.iter(encoding='ascii', errors='surrogateescape'):
+ assert isinstance(tag, str)
+
+ def test_len(self, tagset):
+ assert len(tagset) == 2
+
+
+class TestMutableTagset:
+
+ @pytest.fixture
+ def tagset(self, maildir, notmuch):
+ """An non-empty mutable tagset.
+
+ This will have the default new mail tags: inbox, unread.
+ """
+ _, pathname = maildir.deliver()
+ notmuch('new')
+ with database.Database(maildir.path,
+ mode=database.Mode.READ_WRITE) as db:
+ msg = db.get(pathname)
+ yield msg.tags
+
+ def test_type(self, tagset):
+ assert isinstance(tagset, collections.abc.MutableSet)
+ assert isinstance(tagset, tags.MutableTagSet)
+
+ def test_hash(self, tagset):
+ assert not isinstance(tagset, collections.abc.Hashable)
+ with pytest.raises(TypeError):
+ hash(tagset)
+
+ def test_add(self, tagset):
+ assert 'foo' not in tagset
+ tagset.add('foo')
+ assert 'foo' in tagset
+
+ def test_discard(self, tagset):
+ assert 'inbox' in tagset
+ tagset.discard('inbox')
+ assert 'inbox' not in tagset
+
+ def test_discard_not_present(self, tagset):
+ assert 'foo' not in tagset
+ tagset.discard('foo')
+
+ def test_clear(self, tagset):
+ assert len(tagset) > 0
+ tagset.clear()
+ assert len(tagset) == 0
+
+ def test_from_maildir_flags(self, maildir, notmuch):
+ _, pathname = maildir.deliver(flagged=True)
+ notmuch('new')
+ with database.Database(maildir.path,
+ mode=database.Mode.READ_WRITE) as db:
+ msg = db.get(pathname)
+ msg.tags.discard('flagged')
+ msg.tags.from_maildir_flags()
+ assert 'flagged' in msg.tags
+
+ def test_to_maildir_flags(self, maildir, notmuch):
+ _, pathname = maildir.deliver(flagged=True)
+ notmuch('new')
+ with database.Database(maildir.path,
+ mode=database.Mode.READ_WRITE) as db:
+ msg = db.get(pathname)
+ flags = msg.path.name.split(',')[-1]
+ assert 'F' in flags
+ msg.tags.discard('flagged')
+ msg.tags.to_maildir_flags()
+ flags = msg.path.name.split(',')[-1]
+ assert 'F' not in flags
--
2.15.0.417.g466bffb3ac-goog
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: DRAFT Introduce CFFI-based Python bindings
2017-11-28 20:46 DRAFT Introduce CFFI-based Python bindings Floris Bruynooghe
2017-11-28 20:46 ` [PATCH] Introduce CFFI-based python bindings Floris Bruynooghe
@ 2017-11-28 23:59 ` David Bremner
2017-11-29 9:56 ` Patrick Totzke
2017-11-29 20:44 ` Floris Bruynooghe
1 sibling, 2 replies; 9+ messages in thread
From: David Bremner @ 2017-11-28 23:59 UTC (permalink / raw)
To: Floris Bruynooghe, notmuch
Floris Bruynooghe <flub@devork.be> writes:
>
> Lastly there are some downsides to the choices I made:
> - I ended up going squarely for CPython 3.6+. Choosing Python
> 3 allowed better API design, e.g. with keyword-only parameters
> etc. Choosing CPython 3.4+ restricts the madness that can
> happen with __del__ and gives some newer (tho now unused)
> features in weakref.finalizer.
> - This is no longer drop-in compatible.
> - I haven't got to a stage where my initial goal of speed has
> been proven yet.
I guess you'll have to convince the maintainers / users of alot and afew
that this makes sense before we go much further. I'd point out that
Debian stable is only at python 3.5, so that makes me a bit wary of this
(being able to run the test suite on debian stable and similar aged
distros useful for me, and I suspect other developers).
I know there are issues with memory management in the current bindings,
so that may be a strong reason to push to python 3.6; it seems to need
more investigation at the moment.
d
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: DRAFT Introduce CFFI-based Python bindings
2017-11-28 23:59 ` DRAFT Introduce CFFI-based Python bindings David Bremner
@ 2017-11-29 9:56 ` Patrick Totzke
2017-11-29 21:26 ` Floris Bruynooghe
2017-11-30 23:57 ` Florian Klink
2017-11-29 20:44 ` Floris Bruynooghe
1 sibling, 2 replies; 9+ messages in thread
From: Patrick Totzke @ 2017-11-29 9:56 UTC (permalink / raw)
To: notmuch
[-- Attachment #1: Type: text/plain, Size: 1771 bytes --]
Quoting David Bremner (2017-11-28 23:59:26)
> Floris Bruynooghe <flub@devork.be> writes:
>
> >
> > Lastly there are some downsides to the choices I made:
> > - I ended up going squarely for CPython 3.6+. Choosing Python
> > 3 allowed better API design, e.g. with keyword-only parameters
> > etc. Choosing CPython 3.4+ restricts the madness that can
> > happen with __del__ and gives some newer (tho now unused)
> > features in weakref.finalizer.
> > - This is no longer drop-in compatible.
> > - I haven't got to a stage where my initial goal of speed has
> > been proven yet.
>
> I guess you'll have to convince the maintainers / users of alot and afew
> that this makes sense before we go much further. I'd point out that
> Debian stable is only at python 3.5, so that makes me a bit wary of this
> (being able to run the test suite on debian stable and similar aged
> distros useful for me, and I suspect other developers).
>
> I know there are issues with memory management in the current bindings,
> so that may be a strong reason to push to python 3.6; it seems to need
> more investigation at the moment.
>
> d
I am generally in favour of modernizing the notmuch python bindings,
especially when it comes to memory management and exception handling.
At the moment, the alot interface officially only supports python v2.7
but our dependencies have now mostly been updated and we are working on
port to python 3, see here: https://github.com/pazz/alot/pull/1055
@Floris, you are welcome to join #alot on freenode if you want to
discuss details on that.
You mention that your new API breaks compatibility with the existing
ones. Do you have some demo code that uses the new API for reference?
Cheers,
P
[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]
-----BEGIN PGP SIGNATURE-----
iF0EABECAB0WIQS3g41l/ugN7R3KSU2UNANn19bFqgUCWh6EXQAKCRCUNANn19bF
qqeyAKCcnwchlZZGP3hKE1ZAiaAndQ+DkACgqVD69PJtKCbWw+1bqmi/Jx98viA=
=lZpT
-----END PGP SIGNATURE-----
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: DRAFT Introduce CFFI-based Python bindings
2017-11-28 23:59 ` DRAFT Introduce CFFI-based Python bindings David Bremner
2017-11-29 9:56 ` Patrick Totzke
@ 2017-11-29 20:44 ` Floris Bruynooghe
2017-12-03 12:37 ` Tomi Ollila
1 sibling, 1 reply; 9+ messages in thread
From: Floris Bruynooghe @ 2017-11-29 20:44 UTC (permalink / raw)
To: notmuch
David Bremner <david@tethera.net> writes:
> Floris Bruynooghe <flub@devork.be> writes:
>
>>
>> Lastly there are some downsides to the choices I made:
>> - I ended up going squarely for CPython 3.6+. Choosing Python
>> 3 allowed better API design, e.g. with keyword-only parameters
>> etc. Choosing CPython 3.4+ restricts the madness that can
>> happen with __del__ and gives some newer (tho now unused)
>> features in weakref.finalizer.
>> - This is no longer drop-in compatible.
>> - I haven't got to a stage where my initial goal of speed has
>> been proven yet.
>
> I guess you'll have to convince the maintainers / users of alot and afew
> that this makes sense before we go much further. I'd point out that
> Debian stable is only at python 3.5, so that makes me a bit wary of this
> (being able to run the test suite on debian stable and similar aged
> distros useful for me, and I suspect other developers).
>
> I know there are issues with memory management in the current bindings,
> so that may be a strong reason to push to python 3.6; it seems to need
> more investigation at the moment.
So on earlier Python versions, sure this is possible at not too much
cost.
- Python 3.4+ would just cost the use of some f-strings. Not major, was
just nice to use.
- Python <3.4 afaik would only need a tweak to the Database.tags and
Message.tags properties. I *think* swapping the caching of these
using a weakref should suffice and not break the brittle
Python-libnotmuch memory management.
Mind you I think Python 3.0-3.3 are pretty old and not much point in
supporting them. But this would also apply for 2.7 support.
- Python 2.7 is probably the worst, in that keyword-only arguments would
be gone. If python 2.7 is required I'd be much keener to have another
go at a drop-in replacement with the memory safety features and then
build the "notdb" API on top off it. But for that to be worth it
people need to be convinced enough that maintaining a CFFI version is
nicer than a ctypes version I guess.
Kind regards,
Floris
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: DRAFT Introduce CFFI-based Python bindings
2017-11-29 9:56 ` Patrick Totzke
@ 2017-11-29 21:26 ` Floris Bruynooghe
2017-11-30 23:57 ` Florian Klink
1 sibling, 0 replies; 9+ messages in thread
From: Floris Bruynooghe @ 2017-11-29 21:26 UTC (permalink / raw)
To: notmuch
Patrick Totzke <patricktotzke@gmail.com> writes:
> Quoting David Bremner (2017-11-28 23:59:26)
>> Floris Bruynooghe <flub@devork.be> writes:
>>
>> >
>> > Lastly there are some downsides to the choices I made:
>> > - I ended up going squarely for CPython 3.6+. Choosing Python
>> > 3 allowed better API design, e.g. with keyword-only parameters
>> > etc. Choosing CPython 3.4+ restricts the madness that can
>> > happen with __del__ and gives some newer (tho now unused)
>> > features in weakref.finalizer.
>> > - This is no longer drop-in compatible.
>> > - I haven't got to a stage where my initial goal of speed has
>> > been proven yet.
>>
>> I guess you'll have to convince the maintainers / users of alot and afew
>> that this makes sense before we go much further. I'd point out that
>> Debian stable is only at python 3.5, so that makes me a bit wary of this
>> (being able to run the test suite on debian stable and similar aged
>> distros useful for me, and I suspect other developers).
>>
>> I know there are issues with memory management in the current bindings,
>> so that may be a strong reason to push to python 3.6; it seems to need
>> more investigation at the moment.
>>
>> d
>
>
> I am generally in favour of modernizing the notmuch python bindings,
> especially when it comes to memory management and exception handling.
>
> At the moment, the alot interface officially only supports python v2.7
> but our dependencies have now mostly been updated and we are working on
> port to python 3, see here: https://github.com/pazz/alot/pull/1055
>
> @Floris, you are welcome to join #alot on freenode if you want to
> discuss details on that.
>
> You mention that your new API breaks compatibility with the existing
> ones. Do you have some demo code that uses the new API for reference?
Short, untested, example which works with what's posted:
db = notdb.Database.create()
# or
db = notdb.Database(path=None, mode=notdb.Database.MODE.READ_WRITE)
print(db.path) -> pathlib.Path (a py34 dependency)
if 'unread' in db.tags: # tags behaves like a set
print('unread mail!')
with db.atomic():
msg = db.add('/path/to/file')
mdg = db.get('/path/to/file')
msg = db.find('some-msgid')
db.remove('path/to/other/file')
# sorry, don't have a query interface yet
assert 'unread' in msg.tags
for tag in msg.tags:
print(f'a tag: {tag}')
with msg.frozen(): # Message.frozen() not yet implemented
msg.tags.clear() # all set operations supported
msg.tags.add('atag')
msg.tags_to_flags()
I imagine the query interface would be something like:
with db.query('tag:atag') as query:
print(f'results: {query.count}')
for msg in query:
print(msg.path)
But to be honest I've been spending most time on getting the
memory-safety figured (which I hope I finally did) so far and I think
the tag handling is so far the nicest thing to show off. They're
completely normal Python sets with no special behaviour at all (well,
that's not true - there's the binary interface, see the code posted for
this).
Actually, this last point is kind of important and I failed to mention
it before too. The existing Python bindings convert many bytes from
libnotmuch to Python strings, that is unicode on Python 3. For many it
uses b'bytes'.decode('utf-8', errors='ignore') which is a sane default
if you want to display things. But if you need to round-trip a tag and
store it again you might be changing the tag. I've not found the right
way to handle binary data (e.g. also needed for messageid) everywhere
yet but for tags I've gone with:
for tag in iter(msg.tags): # iter() normally called implicitly by for loop
print(f'All unicode this: {tag}')
for tag in msg.tags.iter(encoding=None):
other_msg.tags.add(tag) # This passes pure bytes around, loses nothing
# What I've used for message ID for now is a "BinString" type
print(f'All just unicode: {msg.messageid}')
binary_msgid = bytes(msg.messageid) # No lossy conversion
This BinString stuff is somewhat hacky, not sure how sane that is. The
second iterator on tags feels somewhat cleaner. Likewise tags could be
BinString as well instead of plain str.
Cheers,
Floris
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: DRAFT Introduce CFFI-based Python bindings
2017-11-29 9:56 ` Patrick Totzke
2017-11-29 21:26 ` Floris Bruynooghe
@ 2017-11-30 23:57 ` Florian Klink
2017-12-01 22:47 ` Floris Bruynooghe
1 sibling, 1 reply; 9+ messages in thread
From: Florian Klink @ 2017-11-30 23:57 UTC (permalink / raw)
To: Floris Bruynooghe; +Cc: notmuch
[-- Attachment #1: Type: text/plain, Size: 1366 bytes --]
>> I guess you'll have to convince the maintainers / users of alot and afew
>> that this makes sense before we go much further. I'd point out that
>> Debian stable is only at python 3.5, so that makes me a bit wary of this
>> (being able to run the test suite on debian stable and similar aged
>> distros useful for me, and I suspect other developers).
>>
>> I know there are issues with memory management in the current bindings,
>> so that may be a strong reason to push to python 3.6; it seems to need
>> more investigation at the moment.
>
>I am generally in favour of modernizing the notmuch python bindings,
>especially when it comes to memory management and exception handling.
>
>At the moment, the alot interface officially only supports python v2.7
>but our dependencies have now mostly been updated and we are working on
>port to python 3, see here: https://github.com/pazz/alot/pull/1055
afew maintainer here ;-)
I'm also very much in favor of a more modern and pythonic interface, and would
gladly support retiring python 2, moving to the new interface.
I had a quick glimpse on the code, and would like to do some annotations. I fear
it's a bit awkward to do this inside the huge patch, which might already have
changed, and send back via email.
Did you publish a changeset to github, or somewhere else where I could comment
on it?
Cheers,
Florian
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: DRAFT Introduce CFFI-based Python bindings
2017-11-30 23:57 ` Florian Klink
@ 2017-12-01 22:47 ` Floris Bruynooghe
0 siblings, 0 replies; 9+ messages in thread
From: Floris Bruynooghe @ 2017-12-01 22:47 UTC (permalink / raw)
To: Florian Klink; +Cc: notmuch
Florian Klink <flokli@flokli.de> writes:
>>> I guess you'll have to convince the maintainers / users of alot and afew
>>> that this makes sense before we go much further. I'd point out that
>>> Debian stable is only at python 3.5, so that makes me a bit wary of this
>>> (being able to run the test suite on debian stable and similar aged
>>> distros useful for me, and I suspect other developers).
>>>
>>> I know there are issues with memory management in the current bindings,
>>> so that may be a strong reason to push to python 3.6; it seems to need
>>> more investigation at the moment.
>>
>>I am generally in favour of modernizing the notmuch python bindings,
>>especially when it comes to memory management and exception handling.
>>
>>At the moment, the alot interface officially only supports python v2.7
>>but our dependencies have now mostly been updated and we are working on
>>port to python 3, see here: https://github.com/pazz/alot/pull/1055
>
> afew maintainer here ;-)
>
> I'm also very much in favor of a more modern and pythonic interface, and would
> gladly support retiring python 2, moving to the new interface.
>
> I had a quick glimpse on the code, and would like to do some annotations. I fear
> it's a bit awkward to do this inside the huge patch, which might already have
> changed, and send back via email.
> Did you publish a changeset to github, or somewhere else where I could comment
> on it?
I now also pushed it to github.com/flub/notmuch in the now, it's in the
cffi branch and all code is in bindings/notmuch-cffi. Not sure if this
helps with commenting, I can make a pull-request to somewhere if you
want - just let me know where.
Cheers,
Floris
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: DRAFT Introduce CFFI-based Python bindings
2017-11-29 20:44 ` Floris Bruynooghe
@ 2017-12-03 12:37 ` Tomi Ollila
0 siblings, 0 replies; 9+ messages in thread
From: Tomi Ollila @ 2017-12-03 12:37 UTC (permalink / raw)
To: Floris Bruynooghe, notmuch
On Wed, Nov 29 2017, Floris Bruynooghe wrote:
> David Bremner <david@tethera.net> writes:
>
>> Floris Bruynooghe <flub@devork.be> writes:
>>
>>>
>>> Lastly there are some downsides to the choices I made:
>>> - I ended up going squarely for CPython 3.6+. Choosing Python
>>> 3 allowed better API design, e.g. with keyword-only parameters
>>> etc. Choosing CPython 3.4+ restricts the madness that can
>>> happen with __del__ and gives some newer (tho now unused)
>>> features in weakref.finalizer.
>>> - This is no longer drop-in compatible.
>>> - I haven't got to a stage where my initial goal of speed has
>>> been proven yet.
>>
>> I guess you'll have to convince the maintainers / users of alot and afew
>> that this makes sense before we go much further. I'd point out that
>> Debian stable is only at python 3.5, so that makes me a bit wary of this
>> (being able to run the test suite on debian stable and similar aged
>> distros useful for me, and I suspect other developers).
>>
>> I know there are issues with memory management in the current bindings,
>> so that may be a strong reason to push to python 3.6; it seems to need
>> more investigation at the moment.
>
> So on earlier Python versions, sure this is possible at not too much
> cost.
>
> - Python 3.4+ would just cost the use of some f-strings. Not major, was
> just nice to use.
> - Python <3.4 afaik would only need a tweak to the Database.tags and
> Message.tags properties. I *think* swapping the caching of these
> using a weakref should suffice and not break the brittle
> Python-libnotmuch memory management.
> Mind you I think Python 3.0-3.3 are pretty old and not much point in
> supporting them. But this would also apply for 2.7 support.
> - Python 2.7 is probably the worst, in that keyword-only arguments would
> be gone. If python 2.7 is required I'd be much keener to have another
> go at a drop-in replacement with the memory safety features and then
> build the "notdb" API on top off it. But for that to be worth it
> people need to be convinced enough that maintaining a CFFI version is
> nicer than a ctypes version I guess.
IMO Python 3.4+ would be OK, if python 2 support can be dropped.
Even Ubuntu 14.04 has python 3.4. One notable distribution that has
Python 3.3 by default is RHEL 7, but there seems to be quite a few
packaged alternatives available...
>
> Kind regards,
> Floris
Tomi
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2017-12-03 12:37 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-11-28 20:46 DRAFT Introduce CFFI-based Python bindings Floris Bruynooghe
2017-11-28 20:46 ` [PATCH] Introduce CFFI-based python bindings Floris Bruynooghe
2017-11-28 23:59 ` DRAFT Introduce CFFI-based Python bindings David Bremner
2017-11-29 9:56 ` Patrick Totzke
2017-11-29 21:26 ` Floris Bruynooghe
2017-11-30 23:57 ` Florian Klink
2017-12-01 22:47 ` Floris Bruynooghe
2017-11-29 20:44 ` Floris Bruynooghe
2017-12-03 12:37 ` Tomi Ollila
Code repositories for project(s) associated with this public inbox
https://yhetil.org/notmuch.git/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).