Notmuch performance problems on OSX

Notmuch performance problems on OSX

[Oliver Charles]

Actually, significant performance problems. Ho ho ho. (sorry)

I've installed the latest notmuch from Git at this time of writing,
along with Xapian from SVN head. However, just tagging a single thread
with only one message seems to take too long:

$ time notmuch tag +dissertation thread:7dc536441e6deade4256a46d46451221

real	0m0.812s
user	0m0.022s
sys	0m0.037s

And tagging all my messages is really horrible:

$ time notmuch tag +foobar tag:inbox

real	0m5.076s
user	0m3.688s
sys	0m0.105s

Here is what my notmuch binary links with:

$ otool -L /usr/local/bin/notmuch
/usr/local/bin/notmuch:
	/usr/local/Cellar/gmime/2.4.0/lib/libgmime-2.4.2.dylib (compatibility
version 7.0.0, current version 7.0.0)
	/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.3)
	/usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
	/usr/local/Cellar/glib/2.20.5/lib/libgobject-2.0.0.dylib
(compatibility version 2001.0.0, current version 2001.5.0)
	/usr/local/Cellar/glib/2.20.5/lib/libglib-2.0.0.dylib (compatibility
version 2001.0.0, current version 2001.5.0)
	/usr/local/Cellar/gettext/0.17/lib/libintl.8.dylib (compatibility
version 9.0.0, current version 9.2.0)
	/usr/local/Cellar/xapian-svn/HEAD/lib/libxapian-1.1.3.dylib
(compatibility version 4.0.0, current version 4.0.0)
	/usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.4.0)
	/usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version 111.1.4)

That xapian-svn was built from svn HEAD right now, so I'm assuming it
contains the #250 fix (http://trac.xapian.org/changeset/13808)

Any ideas?

-- 
    Oliver Charles / aCiD2

Re: Notmuch performance problems on OSX

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 2728 bytes --]

Hi Oliver, welcome to notmuch!

On Thu, 14 Jan 2010 15:30:48 +0000, Oliver Charles <oliver.g.charles@googlemail.com> wrote:
> I've installed the latest notmuch from Git at this time of writing,
> along with Xapian from SVN head. However, just tagging a single thread
> with only one message seems to take too long:
> 
> $ time notmuch tag +dissertation thread:7dc536441e6deade4256a46d46451221
> 
> real	0m0.812s
> user	0m0.022s
> sys	0m0.037s

Things work quite a bit faster than that on my machine:

$ time notmuch tag +foo id:5641883d1001140730l22832715ld6bdc95c9938d314@mail.gmail.com

real	0m0.024s
user	0m0.012s
sys	0m0.004s

But that could just be system differences.

> And tagging all my messages is really horrible:
> 
> $ time notmuch tag +foobar tag:inbox
> 
> real	0m5.076s
> user	0m3.688s
> sys	0m0.105s

For this operation, I can't really compare. How many messages are you
tagging? Here's that operation for me with 525 messages in my inbox:

$ time notmuch tag +foobar tag:inbox

real	0m1.551s
user	0m1.504s
sys	0m0.016s

> That xapian-svn was built from svn HEAD right now, so I'm assuming it
> contains the #250 fix (http://trac.xapian.org/changeset/13808)

Which I think means that things could have been even *much* slower
before. ;-)

The Xapian defect #250 was just one, initial (and obvious) performance
problem. [Though, as I mentioned in a previous thread, if you're using a
Xapian flint database, (look for .notmuch/xapian/iamflint), then you
won't get the benefit of the Xapian fix until you rebuild your notmuch
database from scratch with a current notmuch.]

Once you've verified that you've got the #250 fix functional, there
could still be lots of performance bugs. And it would be time to start
profiling.

Perhaps the "notmuch daemon" idea (which we've proposed earlier for
other reasons) could help reduce overhead from reading the database and
writing it back out again. So that might be one avenue to explore for
fixing things.

I have no idea what OS X does, but Linux keeps my notmuch database in
its buffer cache so I can do these operations without even touching
"disk" (which is actually an SSD anyway, which also helps). I just
tried, and was able to get the single-message tag operation to be 3
times slower by dropping the cache:

$ sudo sh -c "echo 3 > /proc/sys/vm/drop_caches "
$ time notmuch tag +foo id:5641883d1001140730l22832715ld6bdc95c9938d314@mail.gmail.com

real	0m0.062s
user	0m0.000s
sys	0m0.020s

But again, whatever the performance problem might be, the first step
would be to examine some profiles. (And I'm clueless, myself as to what
profiling tools might be available for OS X.)

-Carl

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Notmuch performance problems on OSX

[Olly Betts]

On 2010-01-14, Oliver Charles wrote:
> I've installed the latest notmuch from Git at this time of writing,
> along with Xapian from SVN head. However, just tagging a single thread
> with only one message seems to take too long:

One difference between OS X and other systems is that OS X supports the
F_FULLSYNC ioctl, and other systems don't (currently, at least AFAIK)
and Xapian uses that if it is available to ensure that changes have
actually made it to disk:

http://trac.xapian.org/ticket/288

On other systems, it uses fdatasync() or fsync(), which typically just
ensure that the data has left the OS - it can sit in disk controller or
drive caches for potentially seconds longer.  This call happens once
per table for every (explicit or implicit) flush on a database.

I can see an issue here which is that currently Xapian writes the base
file for the table, then syncs it, then does the next table.  I bet it
would be more efficient to write them all and then sync them all,
especially with F_FULLSYNC.

I'll take a look at doing that, and have created a ticket for it:

http://trac.xapian.org/ticket/426

If after that this is still causing problems, it should probably be made
configurable what (if any) flushing is done.  If you're on a UPS-backed
server, you probably don't need such paranoia.

Cheers,
    Olly

Re: Notmuch performance problems on OSX

[Oliver Charles]

On Thu, Jan 14, 2010 at 11:16 PM, Carl Worth <cworth@cworth.org> wrote:
> Hi Oliver, welcome to notmuch!
>
> On Thu, 14 Jan 2010 15:30:48 +0000, Oliver Charles <oliver.g.charles@googlemail.com> wrote:
>> I've installed the latest notmuch from Git at this time of writing,
>> along with Xapian from SVN head. However, just tagging a single thread
>> with only one message seems to take too long:
>>
>> $ time notmuch tag +dissertation thread:7dc536441e6deade4256a46d46451221
>>
>> real  0m0.812s
>> user  0m0.022s
>> sys   0m0.037s
>
> Things work quite a bit faster than that on my machine:
>
> $ time notmuch tag +foo id:5641883d1001140730l22832715ld6bdc95c9938d314@mail.gmail.com
>
> real    0m0.024s
> user    0m0.012s
> sys     0m0.004s
>
> But that could just be system differences.

Possibly, though my machine is more than capable of what it's doing -
a Q6600 with 4gb RAM

>> And tagging all my messages is really horrible:
>>
>> $ time notmuch tag +foobar tag:inbox
>>
>> real  0m5.076s
>> user  0m3.688s
>> sys   0m0.105s
>
> For this operation, I can't really compare. How many messages are you
> tagging? Here's that operation for me with 525 messages in my inbox:

A few thousand (4k, I believe)

>> That xapian-svn was built from svn HEAD right now, so I'm assuming it
>> contains the #250 fix (http://trac.xapian.org/changeset/13808)
>
> Which I think means that things could have been even *much* slower
> before. ;-)
>
> The Xapian defect #250 was just one, initial (and obvious) performance
> problem. [Though, as I mentioned in a previous thread, if you're using a
> Xapian flint database, (look for .notmuch/xapian/iamflint), then you
> won't get the benefit of the Xapian fix until you rebuild your notmuch
> database from scratch with a current notmuch.]

I didn't know about this need to rebuild, but I tried that and didn't
have any more success sadly.

> Once you've verified that you've got the #250 fix functional, there
> could still be lots of performance bugs. And it would be time to start
> profiling.
>
> [...]

I'm pressed for time at the moment, but in a few weeks I might have
some time to investigate here...

-- 
    Oliver Charles / aCiD2

Re: Notmuch performance problems on OSX

[Oliver Charles]

I just upgraded to xapian-core HEAD and notmuch master tip today, in
desparation to get away from GMail. Sadly it's still taking at least
0.7s to tag a single thread (with one message). I'm really eager to
solve this, could anyone give me any pointers on how I could go about
profiling it or finding the cause of this problem?
-- 
    Oliver Charles / aCiD2

Re: Notmuch performance problems on OSX

[Olly Betts]

On 2010-02-09, Oliver Charles wrote:
> I just upgraded to xapian-core HEAD and notmuch master tip today, in
> desparation to get away from GMail. Sadly it's still taking at least
> 0.7s to tag a single thread (with one message). I'm really eager to
> solve this, could anyone give me any pointers on how I could go about
> profiling it or finding the cause of this problem?

The first thing to try is disabling use of F_FULLFSYNC.  You'll need to
run this command in the xapian-core source tree to comment out the F_FULLFSYNC
code:

perl -pi -e 's/^#ifdef F_FULLFSYNC/#if 0/' backends/*/*_io.h

Then run "make" and "make install".

This makes you a bit more vulnerable to power failures, but no worse than
a typical Unix system.  There's some background here:

http://lists.apple.com/archives/Darwin-dev/2005/Feb/msg00072.html

Assuming that helps, then (a) you have a workaround, and (b) we'll know for
sure it is F_FULLFSYNC to blame.

I've created a ticket for a change to Xapian which should help here, but
not had a chance to work on it yet:

http://trac.xapian.org/ticket/426

Cheers,
    Olly

Re: Notmuch performance problems on OSX

[Oliver Charles]

On Tue, Feb 9, 2010 at 10:09 PM, Olly Betts <olly@survex.com> wrote:
> On 2010-02-09, Oliver Charles wrote:
>> I just upgraded to xapian-core HEAD and notmuch master tip today, in
>> desparation to get away from GMail. Sadly it's still taking at least
>> 0.7s to tag a single thread (with one message). I'm really eager to
>> solve this, could anyone give me any pointers on how I could go about
>> profiling it or finding the cause of this problem?
>
> The first thing to try is disabling use of F_FULLFSYNC.  You'll need to
> run this command in the xapian-core source tree to comment out the F_FULLFSYNC
> code:
>
> perl -pi -e 's/^#ifdef F_FULLFSYNC/#if 0/' backends/*/*_io.h
>
> Then run "make" and "make install".

$ time notmuch tag +inbox thread:6e66368b7887184c6d4c63653211b3f2

real    0m0.067s
user    0m0.036s
sys     0m0.028s

Now this looks a little bit more usable!

> Assuming that helps, then (a) you have a workaround, and (b) we'll know for
> sure it is F_FULLFSYNC to blame.

Looks like this is the case.

> I've created a ticket for a change to Xapian which should help here, but
> not had a chance to work on it yet:
>
> http://trac.xapian.org/ticket/426

I will add my info there and follow the ticket if I can.

--
   Oliver Charles / aCiD2

(Olly, sorry about the double email - in all my excitement I didn't
hit reply all :))

Re: Notmuch performance problems on OSX

[Stewart Smith]

On Fri, 15 Jan 2010 03:58:50 +0000 (UTC), Olly Betts <olly@survex.com> wrote:
> One difference between OS X and other systems is that OS X supports the
> F_FULLSYNC ioctl, and other systems don't (currently, at least AFAIK)
> and Xapian uses that if it is available to ensure that changes have
> actually made it to disk:
> 
> http://trac.xapian.org/ticket/288
> 
> On other systems, it uses fdatasync() or fsync(), which typically just
> ensure that the data has left the OS - it can sit in disk controller or
> drive caches for potentially seconds longer.  This call happens once
> per table for every (explicit or implicit) flush on a database.

At least if you OS and file system don't hate you (e.g. XFS on Linux),
then fsync() really does flush the drive cache.

Also keep in mind that the OSX file system (HFS+) was great for
1985. It's essentially single threaded :/

-- 
Stewart Smith