Unhandled Xapian exception

Unhandled Xapian exception

[Sebastian Spaeth]

Hi Carl,

dme complained that my python bindings abort with
Xapian::DatabaseModifiedException when doing a
Database.find_message('id'). But libnotmuch.so terminates before python
has even a chance to catch an execption, and I think it boils down to this:

http://git.notmuchmail.org/git/notmuch/blob/ec6d78acf12d5c8fe6d10d091adee6516bf48d8a:/lib/database.cc#l276

find_message() which calls:
find_doc_ids_for_term() in lib/database.cc which contains:

 *begin = notmuch->xapian_db->postlist_begin (term);
   *end = notmuch->xapian_db->postlist_end (term);

without doing any catching. According to Olly Betts this can possibly
throw such an exception when the Database has been modified.

I propose to try..catch this code block and rather than returning VOID
it could return NOTMUCH_STATUS_SUCCESS or NOTMUCH_XAPIAN_EXCEPTION.
Not sure how "notmuch_database_find_message" would notify the caller of
such an exception situation though. The only possible failure value is
NULL (which also means did not find such a message).

Sebastian

Re: Unhandled Xapian exception

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 2344 bytes --]

On Fri, 23 Apr 2010 13:21:56 +0200, "Sebastian Spaeth" <Sebastian@SSpaeth.de> wrote:
> I propose to try..catch this code block and rather than returning VOID
> it could return NOTMUCH_STATUS_SUCCESS or NOTMUCH_XAPIAN_EXCEPTION.
> Not sure how "notmuch_database_find_message" would notify the caller of
> such an exception situation though. The only possible failure value is
> NULL (which also means did not find such a message).

We haven't been doing fine-grained exception handling in the library so
far. Mostly because the assumption was that there was nothing we could
do---when I first wrote the code I assumed that Xapian exceptions would
be entirely exceptional.

So I just fixed the top-level entry point here to handle this exception
in our standard way, (print the message, then return NULL or
NOTMUCH_STATUS_XAPIAN_EXCEPTION).

While at this, I audited all notmuch_database calls to ensure they
follow this scheme. (I have not yet audited all libnotmuch entry
points.)

Meanwhile, we've recently discovered that there is a Xapian exception
that is not all that exceptional. Since database readers don't lock the
database, it's not hard to have one or more readers accessing the
database when a writer comes along and modifies the database. This
triggers the readers to subsequently fail with a DatabaseModified
exception.

I don't know how conceivable it would be to fix that at the Xapian
level. It might be quite nice if opening a database in read-only mode
gave access to a snapshot of the database as it exists at that time. But
that might be a feature that's entirely unreasonable to implement.

Otherwise, we might want to start supporting more clever handling of the
exception. For example, the high-level application might want to retry
an operation if it fails due to a DatabaseModified exception. To support
this, you would want your python bindings to have a layer that would
catch the C++ exception and re-throw a python exception.

And to support that, we would need a different scheme in the
library. Basically to just document that all calls might throw an
exception and then not catch and print anything. That would at least be
much simpler in the library. Then the top-level "notmuch" application
could just have a C++ wrapper for main() that would catch and print the
exception message.

What do you think?

-Carl

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Unhandled Xapian exception

[Sebastian Spaeth]

On 2010-04-24, Carl Worth wrote:
> On Fri, 23 Apr 2010 13:21:56 +0200, "Sebastian Spaeth" <Sebastian@SSpaeth.de> wrote:
> > I propose to try..catch this code block and rather than returning VOID
> > it could return NOTMUCH_STATUS_SUCCESS or NOTMUCH_XAPIAN_EXCEPTION.
> > Not sure how "notmuch_database_find_message" would notify the caller of
> > such an exception situation though. The only possible failure value is
> > NULL (which also means did not find such a message).

> And to support that, we would need a different scheme in the
> library. Basically to just document that all calls might throw an
> exception and then not catch and print anything. That would at least be
> much simpler in the library. Then the top-level "notmuch" application
> could just have a C++ wrapper for main() that would catch and print the
> exception message.

I think the python API would not be able to catch a C++ exception at the
library level. I think the proper fix would be to guard the high level
lib/* functions (our public API) and modify the API as needed to notify
us.

E.g for msg* = notmuch_database_find_message(id) (which I used to test
the existence of a message and where NULL can now also mean that someone
modified the database and which renders that test somewhat useless), I
would propose that we make it:

NOTMUCH_STATUS status = notmuch_database_find_message(id, msg*)

where *msg will be filled with the message if it exists and xapian
exceptions are notified through the status code.

Does this make sense?

Sebastian

Re: Unhandled Xapian exception

[David Edmondson]

On Sat, 24 Apr 2010 07:38:03 -0700, Carl Worth <cworth@cworth.org> wrote:
> Otherwise, we might want to start supporting more clever handling of the
> exception. For example, the high-level application might want to retry
> an operation if it fails due to a DatabaseModified exception.

This is what I would like to do (and thought that I was!) in notsync.

dme.
-- 
David Edmondson, http://dme.org