notmuch's idea of concurrency / failing an invocation

notmuch's idea of concurrency / failing an invocation

[Thomas Schwinge]

[-- Attachment #1: Type: text/plain, Size: 1935 bytes --]

Hallo!

Stepping away from the current code base -- what is notmuch's original
idea of concurrency?  That is, all of us probably know that one:

    A Xapian exception occurred opening database: Unable to get write
      lock on /home/thomas/Mail-schwinge.name-thomas/.notmuch/xapian:
      already locked

I recently saw that one while using the Emacs UI (that one tried to
remove a unread tag or similar), and in parallel a delivery to the
notmuch DB was going on.

Apparently the DB we're using doesn't allow for simultaneous writing
(even though it can't even possibly have been dangerous in this case).

Which is the original idea here?  Is it that...

  * each and every client should catch these kinds of errors, and retry,
    or eventually give up at some point, and report the status to the
    user; or is it that...

  * notmuch internally should catch these concurrency cases, and retry,
    or eventually give up at some point (``notmuch --maximum-wait=30s tag
    [...]''), and fail as seen above?

This one is an obvious temporary error due to a concurrency situation.
Wouldn't the latter suggestion be preferable here?  I guess that in most
cases the DB isn't locked for long periods of time, and thus the
concurrency situation would decline quickly.


One difficulty I see is judging which errors are temporary and which are
permanent -- which is obvious in a lot of cases (concurrent DB access,
memory starved or any other OS resource), but may not be, for example in
case of I/O errors (is ``disk full'' a permanent error?).  And then, for
some of these cases, waiting does make sense (concurrent DB access, as
suggested above), and for other (temporary?) errors it doesn't make (a
lot of) sense (out of memory: only sensible thing is to abort, and have
the caller re-try, or disk full: waiting for some free space may be worth
it, or it may be not).


Grüße,
 Thomas

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: notmuch's idea of concurrency / failing an invocation

[micah anderson]

[-- Attachment #1: Type: text/plain, Size: 1691 bytes --]

On Thu, 27 Jan 2011 19:20:00 +0100, Thomas Schwinge <thomas@schwinge.name> wrote:
> Stepping away from the current code base -- what is notmuch's original
> idea of concurrency?  That is, all of us probably know that one:
> 
>     A Xapian exception occurred opening database: Unable to get write
>       lock on /home/thomas/Mail-schwinge.name-thomas/.notmuch/xapian:
>       already locked
> 
> I recently saw that one while using the Emacs UI (that one tried to
> remove a unread tag or similar), and in parallel a delivery to the
> notmuch DB was going on.

Due to my harddisk in my laptop being slow (5400RPM), my notmuch
database growing, and perhaps some fragmentation somewhere, this has
become *incredibly* annoying for me. I am checking email every 30
minutes, and I'm nicing and ionicing the processes so I can use my
machine, but while those processes are running, I'm effectively locked
out of a good portion of my email. 

Usually, I switch to another task until my disk light has ceased being
solid, because the update time is too slow for me to wait. 

Now that folders are making it in, the two remaining features that are
driving me nuts with notmuch is this one and the
verification/decryption/encryption process (replying to an encrypted
message is 12 distinct steps for me, which is discouraging me from doing
that at all). 

I really don't want to complain, because I have no time to help in these
areas,  rather I'm interested  to know  if anyone  has any  pointers for
making this less annoying, and I'm  hoping that at some point I can free
up time to help. Perhaps I need to dump/restore my notmuch DB? Or index
less mail?

micah

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: notmuch's idea of concurrency / failing an invocation

[Jameson Rollins]

[-- Attachment #1: Type: text/plain, Size: 1011 bytes --]

On Thu, 27 Jan 2011 13:40:25 -0500, micah anderson <micah@riseup.net> wrote:
> Due to my harddisk in my laptop being slow (5400RPM), my notmuch
> database growing, and perhaps some fragmentation somewhere, this has
> become *incredibly* annoying for me. I am checking email every 30
> minutes, and I'm nicing and ionicing the processes so I can use my
> machine, but while those processes are running, I'm effectively locked
> out of a good portion of my email. 

I also have a very slow disk, but this is very rarely a problem for me.
I retrieve mail every 10 minutes, and the corresponding notmuch new
usually takes a minute or so.  I really haven't found it to be much of a
bother to just wait it out.

One of the suggested ways to develop around this problem would be a
notmuch daemon that would queue database modification requests.  I don't
think anyone has been working on this yet, but if this is a big problem
for you guys, you might start looking into putting one together.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: notmuch's idea of concurrency / failing an invocation

[Austin Clements]

[-- Attachment #1: Type: text/plain, Size: 1596 bytes --]

I'm looking into breaking notmuch new up into small transactions.  It
wouldn't be much a leap from there to simply close and reopen the database
between transactions if another task wants to use it, which would release
the lock and let the queued notmuch task have the database for a bit.  It
seems silly to have a daemon when all of notmuch's state is already on disk
and queue on a lock is as good as a queue in a daemon, but without the
accompanying architectural shenanigans.

On Thu, Jan 27, 2011 at 3:35 PM, Jameson Rollins <jrollins@finestructure.net
> wrote:

> On Thu, 27 Jan 2011 13:40:25 -0500, micah anderson <micah@riseup.net>
> wrote:
> > Due to my harddisk in my laptop being slow (5400RPM), my notmuch
> > database growing, and perhaps some fragmentation somewhere, this has
> > become *incredibly* annoying for me. I am checking email every 30
> > minutes, and I'm nicing and ionicing the processes so I can use my
> > machine, but while those processes are running, I'm effectively locked
> > out of a good portion of my email.
>
> I also have a very slow disk, but this is very rarely a problem for me.
> I retrieve mail every 10 minutes, and the corresponding notmuch new
> usually takes a minute or so.  I really haven't found it to be much of a
> bother to just wait it out.
>
> One of the suggested ways to develop around this problem would be a
> notmuch daemon that would queue database modification requests.  I don't
> think anyone has been working on this yet, but if this is a big problem
> for you guys, you might start looking into putting one together.
>
> jamie.

[-- Attachment #2: Type: text/html, Size: 1997 bytes --]

Re: notmuch's idea of concurrency / failing an invocation

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 2961 bytes --]

On Thu, 27 Jan 2011 19:20:00 +0100, Thomas Schwinge <thomas@schwinge.name> wrote:
> Which is the original idea here?  Is it that...

There's no original idea yet. It's essentially an unsolved problem right now.

>   * each and every client should catch these kinds of errors, and retry,
>     or eventually give up at some point, and report the status to the
>     user; or is it that...
> 
>   * notmuch internally should catch these concurrency cases, and retry,
>     or eventually give up at some point (``notmuch --maximum-wait=30s tag
>     [...]''), and fail as seen above?

Some people have actually already done work solutions in one way or
another. Here are a few of the messages I found in my "outstanding
notmuch mail to read"[*] queue:

    James Vasile patched the emacs interface to call notmuch
    asynchronously and to repeatedly call it if it fails (he also
    wonders if it should have some sort of timeout):

	id:"87vddnlxos.wl%james@hackervisions.org"

    James also wrote a shell script that repeatedly calls the notmuch
    binary as necessary (and he wonders if this retrying should happen
    inside notmuch itself):

	id:"87pr3sw43a.fsf@hackervisions.org"


    "servilio" wrote a new "notmuch repl" command which can accept
    notmuch operations expressed in text form on stdin, and then
    interpret and execute them. That's a good start on a notmuch daemon:

	id:"AANLkTi=7eCt0=NqUiJFrGDcaZ17LOd3qNNqN1-ASwYzr@mail.gmail.com"

I'm not sure yet which approach (or approaches) we want. But I would
love to see some of the limitations described in the messages above
addressed. That would definitely make some of the patches more
acceptable.

-Carl

[*] And yes, my queue really does span a year(!) or so. That's
embarrassing. I'm committed to making progress on this queue and staying
up-to-date with new patches, so I've made a couple of recent changes:

1. I'm now processing the queue largely in reverse-chronological
   order. The idea here is that I can stay on top of new posts, while
   also making progress on previously-sent items.

   This does mean that you can hack my workflow by replying to an old
   thread, (and thereby bringing it back to my attention). Please feel
   free to do that---ideally by mentioning any new information such as
   "these patches are now rebased <here>" or "I've tested these patches
   in daily use for X months and they still apply fine to master" or
   similar.

2. I've date-limited my saved search for my notmuch queue to show a
   small number of messages. This is a cheap psychological hack. If the
   number on the queue is too large it makes me hesitant to even look at
   it. But with a small number, it's easier to make progress since the
   end is apparently in sight.

   Of course, once I reduce my date-limited queue to 0, I'll extend the
   date back into the past and try to keep working through things.

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: notmuch's idea of concurrency / failing an invocation

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 1115 bytes --]

On Thu, 27 Jan 2011 17:20:21 -0500, Austin Clements <amdragon@mit.edu> wrote:
> I'm looking into breaking notmuch new up into small transactions.  It
> wouldn't be much a leap from there to simply close and reopen the database
> between transactions if another task wants to use it, which would release
> the lock and let the queued notmuch task have the database for a bit.

That sounds like something very useful to pursue. Please continue!

> It seems silly to have a daemon when all of notmuch's state is already on disk
> and queue on a lock is as good as a queue in a daemon, but without the
> accompanying architectural shenanigans.

It would definitely be nice to avoid the complexity inherent in having a
daemon, but how do you imagine "queue on a lock" to work? We don't have
anything like that in place now.

Another advantage that can happen with queueing (wherever it occurs) is
to allow a client to be very responsive without waiting for an operation
to complete. Though that can of course be band if the operation isn't
reliably committed.

-Carl

-- 
carl.d.worth@intel.com

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: notmuch's idea of concurrency / failing an invocation

[Thomas Schwinge]

[-- Attachment #1: Type: text/plain, Size: 4232 bytes --]

Hallo!

On Fri, 28 Jan 2011 15:10:01 +1000, Carl Worth <cworth@cworth.org> wrote:
> On Thu, 27 Jan 2011 17:20:21 -0500, Austin Clements <amdragon@mit.edu> wrote:
> > I'm looking into breaking notmuch new up into small transactions.  It
> > wouldn't be much a leap from there to simply close and reopen the database
> > between transactions if another task wants to use it, which would release
> > the lock and let the queued notmuch task have the database for a bit.
> 
> That sounds like something very useful to pursue. Please continue!

Ack!  And actually -- I just wondered about that: what happens if
``notmuch new'' has executed notmuch_database_add_message for a new
message M, but then is killed before adding any tags and finishing up
(and supposing that the DB isn't in an invalid state now).  This process
of adding M to the DB and applying any tags isn't one single transaction
currently, right?  (And this is exactly what you're working on
chainging?)  Am I right that what currently happens is that upon the next
``notmuch new'' run, notmuch will not reconsider M (given that it already
is present in the DB), but continue with the next messages -- thus
leaving M without any tags?  This isn't a very likely scenario, but still
a possible one.

> > It seems silly to have a daemon when all of notmuch's state is already on disk
> > and queue on a lock is as good as a queue in a daemon, but without the
> > accompanying architectural shenanigans.

Ack, too.  A daemon seems one abstraction layer too much.  (But I'm not
actively opposed either, if someone has a valid use for such a scheme.)

> It would definitely be nice to avoid the complexity inherent in having a
> daemon, but how do you imagine "queue on a lock" to work? We don't have
> anything like that in place now.

I suppose what he means is trying to get the lock, and if that fails wait
a bit / wait until it is available again.

Actually, as a next step, wouldn't it also be possible to add some
heuristic to avoid ``notmuch new'' (being a low-priority task) blocking
some interactive user (UI; high-priority task)?  But we can pursue such
schemes as soon as the basic infrastructure is in place.

> Another advantage that can happen with queueing (wherever it occurs) is
> to allow a client to be very responsive without waiting for an operation
> to complete. Though that can of course be band if the operation isn't
> reliably committed.

(Obviously this can only work as long as we don't immediatelly need the
operation's result; think ``notmuch show''.)

So, if the DB has the functionality to internally queue and immediatelly
acknowledge transactions, and only later (reliably) commit them, wouldn't
that be fine indeed?  For example, ``notmuch tag'' then wouldn't have to
wait for the DB to be writable.  (And note that I have no idea whether
Xapian supports such things.)  But on the other hand we would like to
immediatelly display the requested change in the UI, right?

What notmuch-show.el:notmuch-show-remove-tag currently does is *not*
re-asking the DB for a message's current tags after having removed a
specific one, but instead it interprets the tag removal command itself --
which is easy enough in this case, and rather unlikely to ever yield
different results, at least unless there's another process operating on
the DB concurrently.

Otherwise, the other way round, the client could maintain a list of to-do
items, to which actions are added if the DB is currently busy, and this
list is periodically worked on in order to get it empty.  For example,
tag changes that are in this list, but not yet committed in the DB could
be displayed in another color in the UI.  But doing so would shift the
responsibility to the UI, which should be in the DB, in my humble
opinion.  (Actually, this issue feels similar to the one who should be
doing the re-trying in case the DB is busy: the UI, or the notmuch
process itself, which we're discussing in another thread.)


As you can guess I'm not very much into DBs, and neither too much into
concurrent systems, so if my ideas don't make sense, please feel free to
refer me to literature.


Grüße,
 Thomas

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: notmuch's idea of concurrency / failing an invocation

[Mike Kelly]

On Fri, 28 Jan 2011 10:45:19 +0100, Thomas Schwinge <thomas@schwinge.name> wrote:
> > It would definitely be nice to avoid the complexity inherent in having a
> > daemon, but how do you imagine "queue on a lock" to work? We don't have
> > anything like that in place now.
>
> I suppose what he means is trying to get the lock, and if that fails wait
> a bit / wait until it is available again.
>
> Actually, as a next step, wouldn't it also be possible to add some
> heuristic to avoid ``notmuch new'' (being a low-priority task) blocking
> some interactive user (UI; high-priority task)?  But we can pursue such
> schemes as soon as the basic infrastructure is in place.

Couldn't we pretty much get the desired behavior by using flock(2)?
Basically, take out a LOCK_EX when we need to write, and a LOCK_SH when
we only need to read. Using the blocking form, things should pretty much
just queue up and take their turn, right?

I'm not familiar with Xapian, but if it doesn't give us something we
could use this sort of locking on, couldn't we just add some
/path/to/mail/.notmuch.lock file that we open to hold a lock on?

We already have to specify if we want a read-only or read-write database
handle in notmuch_database_open, so it seems like it'd be easy enough to
hook in there.

-- 
Mike Kelly

Re: notmuch's idea of concurrency / failing an invocation

[Austin Clements]

[-- Attachment #1: Type: text/plain, Size: 3314 bytes --]

On Fri, Jan 28, 2011 at 4:45 AM, Thomas Schwinge <thomas@schwinge.name>wrote:

> On Fri, 28 Jan 2011 15:10:01 +1000, Carl Worth <cworth@cworth.org> wrote:
> > On Thu, 27 Jan 2011 17:20:21 -0500, Austin Clements <amdragon@mit.edu>
> wrote:
> > > I'm looking into breaking notmuch new up into small transactions.  It
> > > wouldn't be much a leap from there to simply close and reopen the
> database
> > > between transactions if another task wants to use it, which would
> release
> > > the lock and let the queued notmuch task have the database for a bit.
> >
> > That sounds like something very useful to pursue. Please continue!
>
> Ack!  And actually -- I just wondered about that: what happens if
> ``notmuch new'' has executed notmuch_database_add_message for a new
> message M, but then is killed before adding any tags and finishing up
> (and supposing that the DB isn't in an invalid state now).  This process
> of adding M to the DB and applying any tags isn't one single transaction
> currently, right?  (And this is exactly what you're working on
> chainging?)  Am I right that what currently happens is that upon the next
> ``notmuch new'' run, notmuch will not reconsider M (given that it already
> is present in the DB), but continue with the next messages -- thus
> leaving M without any tags?  This isn't a very likely scenario, but still
> a possible one.


There are quite a few bugs like this.  In fact, last night I added a test
that interrupts notmuch new (for real, not SIGINT) after every database
write, and on each interrupted database snapshot, re-runs notmuch new to
completion, then checks that the database winds up in the correct state.
 There are dozens of interruption points where it doesn't, many of which are
permanent, even if you force notmuch new to rescan the maildir.

> Another advantage that can happen with queueing (wherever it occurs) is
> > to allow a client to be very responsive without waiting for an operation
> > to complete. Though that can of course be band if the operation isn't
> > reliably committed.
>
> (Obviously this can only work as long as we don't immediatelly need the
> operation's result; think ``notmuch show''.)
>
> So, if the DB has the functionality to internally queue and immediatelly
> acknowledge transactions, and only later (reliably) commit them, wouldn't
> that be fine indeed?  For example, ``notmuch tag'' then wouldn't have to
> wait for the DB to be writable.  (And note that I have no idea whether
> Xapian supports such things.)  But on the other hand we would like to
> immediatelly display the requested change in the UI, right?
>

This would be fantastic, if the client could indicate the difference between
a "pending" change and a "committed" change as you suggest below.  I don't
think having the database lie about its commit state is the right way to do
this, though (nor should the client lie about this, thus the "pending"
display).  A better way would be for the client to update the display to
"pending", start the notmuch operation asynchronously, have the notmuch
operation block and queue up on the database lock, then have the client
update the display to "committed" when the asynchronous operation returns.
 No weird database operations or transactional semantics and the client side
is fairly straightforward.

[-- Attachment #2: Type: text/html, Size: 4222 bytes --]

Re: notmuch's idea of concurrency / failing an invocation

[Austin Clements]

[-- Attachment #1: Type: text/plain, Size: 1584 bytes --]

On Fri, Jan 28, 2011 at 10:36 AM, Mike Kelly <pioto@pioto.org> wrote:

> On Fri, 28 Jan 2011 10:45:19 +0100, Thomas Schwinge <thomas@schwinge.name>
> wrote:
> > > It would definitely be nice to avoid the complexity inherent in having
> a
> > > daemon, but how do you imagine "queue on a lock" to work? We don't have
> > > anything like that in place now.
> >
> > I suppose what he means is trying to get the lock, and if that fails wait
> > a bit / wait until it is available again.
> >
> > Actually, as a next step, wouldn't it also be possible to add some
> > heuristic to avoid ``notmuch new'' (being a low-priority task) blocking
> > some interactive user (UI; high-priority task)?  But we can pursue such
> > schemes as soon as the basic infrastructure is in place.
>
> Couldn't we pretty much get the desired behavior by using flock(2)?
> Basically, take out a LOCK_EX when we need to write, and a LOCK_SH when
> we only need to read. Using the blocking form, things should pretty much
> just queue up and take their turn, right?
>
> I'm not familiar with Xapian, but if it doesn't give us something we
> could use this sort of locking on, couldn't we just add some
> /path/to/mail/.notmuch.lock file that we open to hold a lock on?
>

Yes, exactly.  All of this.  Unfortunately, Xapian doesn't expose the
ability to block on the lock (see the fcntl call in backends/flint_lock.cc,
which is hard-coded to the non-blocking F_SETLK instead of F_SETLKW), so
we'd either need a new Xapian option, or we would just have to wrap our own
flock/fcntl lock around things as you suggest.

[-- Attachment #2: Type: text/html, Size: 2063 bytes --]

Re: notmuch's idea of concurrency / failing an invocation

[Austin Clements]

Actually, this is trivial to play with.  Here's a stop-gap wrapper
script for people having trouble with Xapian locking,

#!/bin/bash

NOTMUCH_BIN="/path/to/notmuch"
MAIL_DIR="/path/to/mailroot"

(
    case "$1" in
        setup|help)
            ;;
        search|show|count|reply|dump|search-tags|part)
            flock -s 200;;
        *)
            flock -x 200;;
    esac
    "$NOTMUCH_BIN" "$@" 200>&-
) 200>"$MAIL_DIR"/.notmuch/lock

On Fri, Jan 28, 2011 at 11:57 AM, Austin Clements <amdragon@mit.edu> wrote:
>
> On Fri, Jan 28, 2011 at 10:36 AM, Mike Kelly <pioto@pioto.org> wrote:
>>
>> On Fri, 28 Jan 2011 10:45:19 +0100, Thomas Schwinge <thomas@schwinge.name> wrote:
>> > > It would definitely be nice to avoid the complexity inherent in having a
>> > > daemon, but how do you imagine "queue on a lock" to work? We don't have
>> > > anything like that in place now.
>> >
>> > I suppose what he means is trying to get the lock, and if that fails wait
>> > a bit / wait until it is available again.
>> >
>> > Actually, as a next step, wouldn't it also be possible to add some
>> > heuristic to avoid ``notmuch new'' (being a low-priority task) blocking
>> > some interactive user (UI; high-priority task)?  But we can pursue such
>> > schemes as soon as the basic infrastructure is in place.
>>
>> Couldn't we pretty much get the desired behavior by using flock(2)?
>> Basically, take out a LOCK_EX when we need to write, and a LOCK_SH when
>> we only need to read. Using the blocking form, things should pretty much
>> just queue up and take their turn, right?
>>
>> I'm not familiar with Xapian, but if it doesn't give us something we
>> could use this sort of locking on, couldn't we just add some
>> /path/to/mail/.notmuch.lock file that we open to hold a lock on?
>
> Yes, exactly.  All of this.  Unfortunately, Xapian doesn't expose the ability to block on the lock (see the fcntl call in backends/flint_lock.cc, which is hard-coded to the non-blocking F_SETLK instead of F_SETLKW), so we'd either need a new Xapian option, or we would just have to wrap our own flock/fcntl lock around things as you suggest.

Re: notmuch's idea of concurrency / failing an invocation

[Stewart Smith]

On Thu, 27 Jan 2011 13:40:25 -0500, micah anderson <micah@riseup.net> wrote:
> Due to my harddisk in my laptop being slow (5400RPM), my notmuch
> database growing, and perhaps some fragmentation somewhere, this has
> become *incredibly* annoying for me. I am checking email every 30
> minutes, and I'm nicing and ionicing the processes so I can use my
> machine, but while those processes are running, I'm effectively locked
> out of a good portion of my email. 

I used to use spinning rust and also noticed things were slow. This
is in fact mostly not xapian - but rather crawling the Maildir. I
improved this early on in notmuch history by reducing the number of
seeks needed when traversing the Maildir hierarchy (e.g. stat in
i-node order, which is roughly on-disk order).

I'm about at the point where I'm going to take my git mail store
experiments and get them really to work (and everyone will have to use
'notmuch cat' or the like to access the messages) which should provide
both great storage efficiency, much faster backups of your Maildir as
well as having way fewer paths to traverse checking for new mail.

-- 
Stewart Smith

Re: notmuch's idea of concurrency / failing an invocation

[Mike Kelly]

[-- Attachment #1: Type: text/plain, Size: 619 bytes --]

On Fri, 28 Jan 2011 11:57:34 -0500
Austin Clements <amdragon@mit.edu> wrote:

> Yes, exactly.  All of this.  Unfortunately, Xapian doesn't expose the
> ability to block on the lock (see the fcntl call in
> backends/flint_lock.cc, which is hard-coded to the non-blocking
> F_SETLK instead of F_SETLKW), so we'd either need a new Xapian
> option, or we would just have to wrap our own flock/fcntl lock around
> things as you suggest.

Hrm. Do you know if Xapian upstream would be open to a patch to support
optional blocking locks? We can't be the only ones hitting these sorts
of issues.

-- 
Mike Kelly

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: notmuch's idea of concurrency / failing an invocation

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 752 bytes --]

On 01/28/2011 08:05 PM, Stewart Smith wrote:
> I'm about at the point where I'm going to take my git mail store
> experiments and get them really to work (and everyone will have to use
> 'notmuch cat' or the like to access the messages)

Would this hypothetical git-based mail store retain the atomicity and
lockless concurrent-access of a maildir?  That is, could it be used in a
server environment?

> which should provide
> both great storage efficiency, much faster backups of your Maildir as
> well as having way fewer paths to traverse checking for new mail.

when you say "backups of your Maildir" do you mean "backups of your
git-based mail store" ?  or is this somehow a literal Maildir stored in git?

Intrigued,

	--dkg


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 1030 bytes --]

Re: notmuch's idea of concurrency / failing an invocation

[Stewart Smith]

On Sat, 29 Jan 2011 19:14:27 -0500, Daniel Kahn Gillmor <dkg@fifthhorseman.net> wrote:
> On 01/28/2011 08:05 PM, Stewart Smith wrote:
> > I'm about at the point where I'm going to take my git mail store
> > experiments and get them really to work (and everyone will have to use
> > 'notmuch cat' or the like to access the messages)
> 
> Would this hypothetical git-based mail store retain the atomicity and
> lockless concurrent-access of a maildir?  That is, could it be used in a
> server environment?

My idea is that it would be... at least with the experiments conducted
so far.

> > which should provide
> > both great storage efficiency, much faster backups of your Maildir as
> > well as having way fewer paths to traverse checking for new mail.
> 
> when you say "backups of your Maildir" do you mean "backups of your
> git-based mail store" ?  or is this somehow a literal Maildir stored in git?

I'll write more "soon" when there is more code behind it... and I figure
out a good upgrade path to something that is also self-consistently sane.

-- 
Stewart Smith

Re: notmuch's idea of concurrency / failing an invocation

[micah anderson]

[-- Attachment #1: Type: text/plain, Size: 1138 bytes --]

On Thu, 27 Jan 2011 12:35:16 -0800, Jameson Rollins <jrollins@finestructure.net> wrote:
> On Thu, 27 Jan 2011 13:40:25 -0500, micah anderson <micah@riseup.net> wrote:
> > Due to my harddisk in my laptop being slow (5400RPM), my notmuch
> > database growing, and perhaps some fragmentation somewhere, this has
> > become *incredibly* annoying for me. I am checking email every 30
> > minutes, and I'm nicing and ionicing the processes so I can use my
> > machine, but while those processes are running, I'm effectively locked
> > out of a good portion of my email. 
> 
> I also have a very slow disk, but this is very rarely a problem for me.
> I retrieve mail every 10 minutes, and the corresponding notmuch new
> usually takes a minute or so.  I really haven't found it to be much of a
> bother to just wait it out.

Sometimes I can have several thousand messages to add/remove from the
database. I know this is probably unusal, but for me its not due to
system emails. I suppose if I checked my email more frequently, I'd have
less to process on each run, but thats more side-stepping the
concurrency issue.

micah

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: notmuch's idea of concurrency / failing an invocation

[Austin Clements]

Now that I've split notmuch new up in to lots of small transactions, I
think the database locking issue is quite approachable.  Here's a
proposed locking protocol where a notmuch operation that wants to
modify the database blocks if there's another operation in progress
(instead of immediately failing like now), but indicates to the
in-progress operation that, when convenient, it should temporarily
abdicate the database.

Add a file to the .notmuch directory, say "lock", which we'll use for
fcntl locks (fcntl locks have nice properties, like automatic cleanup
on process exit and NFS compatibility).

To open the database for write,
1. Acquire an exclusive lock on byte 0 of the lock file (in blocking mode)
2. Acquire an exclusive lock on byte 1 of the lock file
3. Release the lock on byte 0
4. Open the Xapian database

When it's convenient to abdicate the lock, test if there are pending
operations by testing for a lock on byte 0 of the lock file using
F_GETLK.  If there's no lock on byte 0, just continue without
releasing the database.  Otherwise,
1. Close the Xapian database
2. Release the lock on byte 1
3. Re-lock and re-open the database.

In effect, this acts like one lock, since byte 1 is only ever acquired
while byte 0 is held, but splitting it across two locks like this lets
us "peek" at the waiter queue and see if someone is waiting.

Some possible extensions: We may want a timeout for how long to wait
for the lock (in case the lock holder gets wedged).  We could work
around DatabaseModified exceptions by having readers do essentially
the same thing as writers, but take the locks in shared mode.  Readers
wouldn't proceed in parallel with writers, but long-running writers
would relinquish the lock, so this isn't so bad.  Finally, concurrent
notmuch new's should probably be serialized (instead of repeatedly
abdicating to each other), so it may make sense to have an additional
"notmuch new lock".

On Thu, Jan 27, 2011 at 5:20 PM, Austin Clements <amdragon@mit.edu> wrote:
> I'm looking into breaking notmuch new up into small transactions.  It
> wouldn't be much a leap from there to simply close and reopen the database
> between transactions if another task wants to use it, which would release
> the lock and let the queued notmuch task have the database for a bit.  It
> seems silly to have a daemon when all of notmuch's state is already on disk
> and queue on a lock is as good as a queue in a daemon, but without the
> accompanying architectural shenanigans.