Re: notmuch's idea of concurrency / failing an invocation

[Austin Clements <amdragon@mit.edu>]

[-- Attachment #1: Type: text/plain, Size: 3314 bytes --]

On Fri, Jan 28, 2011 at 4:45 AM, Thomas Schwinge <thomas@schwinge.name>wrote:

> On Fri, 28 Jan 2011 15:10:01 +1000, Carl Worth <cworth@cworth.org> wrote:
> > On Thu, 27 Jan 2011 17:20:21 -0500, Austin Clements <amdragon@mit.edu>
> wrote:
> > > I'm looking into breaking notmuch new up into small transactions.  It
> > > wouldn't be much a leap from there to simply close and reopen the
> database
> > > between transactions if another task wants to use it, which would
> release
> > > the lock and let the queued notmuch task have the database for a bit.
> >
> > That sounds like something very useful to pursue. Please continue!
>
> Ack!  And actually -- I just wondered about that: what happens if
> ``notmuch new'' has executed notmuch_database_add_message for a new
> message M, but then is killed before adding any tags and finishing up
> (and supposing that the DB isn't in an invalid state now).  This process
> of adding M to the DB and applying any tags isn't one single transaction
> currently, right?  (And this is exactly what you're working on
> chainging?)  Am I right that what currently happens is that upon the next
> ``notmuch new'' run, notmuch will not reconsider M (given that it already
> is present in the DB), but continue with the next messages -- thus
> leaving M without any tags?  This isn't a very likely scenario, but still
> a possible one.


There are quite a few bugs like this.  In fact, last night I added a test
that interrupts notmuch new (for real, not SIGINT) after every database
write, and on each interrupted database snapshot, re-runs notmuch new to
completion, then checks that the database winds up in the correct state.
 There are dozens of interruption points where it doesn't, many of which are
permanent, even if you force notmuch new to rescan the maildir.

> Another advantage that can happen with queueing (wherever it occurs) is
> > to allow a client to be very responsive without waiting for an operation
> > to complete. Though that can of course be band if the operation isn't
> > reliably committed.
>
> (Obviously this can only work as long as we don't immediatelly need the
> operation's result; think ``notmuch show''.)
>
> So, if the DB has the functionality to internally queue and immediatelly
> acknowledge transactions, and only later (reliably) commit them, wouldn't
> that be fine indeed?  For example, ``notmuch tag'' then wouldn't have to
> wait for the DB to be writable.  (And note that I have no idea whether
> Xapian supports such things.)  But on the other hand we would like to
> immediatelly display the requested change in the UI, right?
>

This would be fantastic, if the client could indicate the difference between
a "pending" change and a "committed" change as you suggest below.  I don't
think having the database lie about its commit state is the right way to do
this, though (nor should the client lie about this, thus the "pending"
display).  A better way would be for the client to update the display to
"pending", start the notmuch operation asynchronously, have the notmuch
operation block and queue up on the database lock, then have the client
update the display to "committed" when the asynchronous operation returns.
 No weird database operations or transactional semantics and the client side
is fairly straightforward.

[-- Attachment #2: Type: text/html, Size: 4222 bytes --]