Re: Notmuch performance problems on OSX

[Olly Betts <olly@survex.com>]

On 2010-01-14, Oliver Charles wrote:
> I've installed the latest notmuch from Git at this time of writing,
> along with Xapian from SVN head. However, just tagging a single thread
> with only one message seems to take too long:

One difference between OS X and other systems is that OS X supports the
F_FULLSYNC ioctl, and other systems don't (currently, at least AFAIK)
and Xapian uses that if it is available to ensure that changes have
actually made it to disk:

http://trac.xapian.org/ticket/288

On other systems, it uses fdatasync() or fsync(), which typically just
ensure that the data has left the OS - it can sit in disk controller or
drive caches for potentially seconds longer.  This call happens once
per table for every (explicit or implicit) flush on a database.

I can see an issue here which is that currently Xapian writes the base
file for the table, then syncs it, then does the next table.  I bet it
would be more efficient to write them all and then sync them all,
especially with F_FULLSYNC.

I'll take a look at doing that, and have created a ticket for it:

http://trac.xapian.org/ticket/426

If after that this is still causing problems, it should probably be made
configurable what (if any) flushing is done.  If you're on a UPS-backed
server, you probably don't need such paranoia.

Cheers,
    Olly