* Notmuch performance problems on OSX
@ 2010-01-14 15:30 Oliver Charles
2010-01-14 23:16 ` Carl Worth
2010-01-15 3:58 ` Olly Betts
0 siblings, 2 replies; 8+ messages in thread
From: Oliver Charles @ 2010-01-14 15:30 UTC (permalink / raw)
To: notmuch
Actually, significant performance problems. Ho ho ho. (sorry)
I've installed the latest notmuch from Git at this time of writing,
along with Xapian from SVN head. However, just tagging a single thread
with only one message seems to take too long:
$ time notmuch tag +dissertation thread:7dc536441e6deade4256a46d46451221
real 0m0.812s
user 0m0.022s
sys 0m0.037s
And tagging all my messages is really horrible:
$ time notmuch tag +foobar tag:inbox
real 0m5.076s
user 0m3.688s
sys 0m0.105s
Here is what my notmuch binary links with:
$ otool -L /usr/local/bin/notmuch
/usr/local/bin/notmuch:
/usr/local/Cellar/gmime/2.4.0/lib/libgmime-2.4.2.dylib (compatibility
version 7.0.0, current version 7.0.0)
/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.3)
/usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
/usr/local/Cellar/glib/2.20.5/lib/libgobject-2.0.0.dylib
(compatibility version 2001.0.0, current version 2001.5.0)
/usr/local/Cellar/glib/2.20.5/lib/libglib-2.0.0.dylib (compatibility
version 2001.0.0, current version 2001.5.0)
/usr/local/Cellar/gettext/0.17/lib/libintl.8.dylib (compatibility
version 9.0.0, current version 9.2.0)
/usr/local/Cellar/xapian-svn/HEAD/lib/libxapian-1.1.3.dylib
(compatibility version 4.0.0, current version 4.0.0)
/usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.4.0)
/usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version 111.1.4)
That xapian-svn was built from svn HEAD right now, so I'm assuming it
contains the #250 fix (http://trac.xapian.org/changeset/13808)
Any ideas?
--
Oliver Charles / aCiD2
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Notmuch performance problems on OSX
2010-01-14 15:30 Notmuch performance problems on OSX Oliver Charles
@ 2010-01-14 23:16 ` Carl Worth
2010-01-18 18:35 ` Oliver Charles
2010-01-15 3:58 ` Olly Betts
1 sibling, 1 reply; 8+ messages in thread
From: Carl Worth @ 2010-01-14 23:16 UTC (permalink / raw)
To: Oliver Charles, notmuch
[-- Attachment #1: Type: text/plain, Size: 2728 bytes --]
Hi Oliver, welcome to notmuch!
On Thu, 14 Jan 2010 15:30:48 +0000, Oliver Charles <oliver.g.charles@googlemail.com> wrote:
> I've installed the latest notmuch from Git at this time of writing,
> along with Xapian from SVN head. However, just tagging a single thread
> with only one message seems to take too long:
>
> $ time notmuch tag +dissertation thread:7dc536441e6deade4256a46d46451221
>
> real 0m0.812s
> user 0m0.022s
> sys 0m0.037s
Things work quite a bit faster than that on my machine:
$ time notmuch tag +foo id:5641883d1001140730l22832715ld6bdc95c9938d314@mail.gmail.com
real 0m0.024s
user 0m0.012s
sys 0m0.004s
But that could just be system differences.
> And tagging all my messages is really horrible:
>
> $ time notmuch tag +foobar tag:inbox
>
> real 0m5.076s
> user 0m3.688s
> sys 0m0.105s
For this operation, I can't really compare. How many messages are you
tagging? Here's that operation for me with 525 messages in my inbox:
$ time notmuch tag +foobar tag:inbox
real 0m1.551s
user 0m1.504s
sys 0m0.016s
> That xapian-svn was built from svn HEAD right now, so I'm assuming it
> contains the #250 fix (http://trac.xapian.org/changeset/13808)
Which I think means that things could have been even *much* slower
before. ;-)
The Xapian defect #250 was just one, initial (and obvious) performance
problem. [Though, as I mentioned in a previous thread, if you're using a
Xapian flint database, (look for .notmuch/xapian/iamflint), then you
won't get the benefit of the Xapian fix until you rebuild your notmuch
database from scratch with a current notmuch.]
Once you've verified that you've got the #250 fix functional, there
could still be lots of performance bugs. And it would be time to start
profiling.
Perhaps the "notmuch daemon" idea (which we've proposed earlier for
other reasons) could help reduce overhead from reading the database and
writing it back out again. So that might be one avenue to explore for
fixing things.
I have no idea what OS X does, but Linux keeps my notmuch database in
its buffer cache so I can do these operations without even touching
"disk" (which is actually an SSD anyway, which also helps). I just
tried, and was able to get the single-message tag operation to be 3
times slower by dropping the cache:
$ sudo sh -c "echo 3 > /proc/sys/vm/drop_caches "
$ time notmuch tag +foo id:5641883d1001140730l22832715ld6bdc95c9938d314@mail.gmail.com
real 0m0.062s
user 0m0.000s
sys 0m0.020s
But again, whatever the performance problem might be, the first step
would be to examine some profiles. (And I'm clueless, myself as to what
profiling tools might be available for OS X.)
-Carl
[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Notmuch performance problems on OSX
2010-01-14 23:16 ` Carl Worth
@ 2010-01-18 18:35 ` Oliver Charles
2010-02-09 20:46 ` Oliver Charles
0 siblings, 1 reply; 8+ messages in thread
From: Oliver Charles @ 2010-01-18 18:35 UTC (permalink / raw)
To: Carl Worth; +Cc: notmuch
On Thu, Jan 14, 2010 at 11:16 PM, Carl Worth <cworth@cworth.org> wrote:
> Hi Oliver, welcome to notmuch!
>
> On Thu, 14 Jan 2010 15:30:48 +0000, Oliver Charles <oliver.g.charles@googlemail.com> wrote:
>> I've installed the latest notmuch from Git at this time of writing,
>> along with Xapian from SVN head. However, just tagging a single thread
>> with only one message seems to take too long:
>>
>> $ time notmuch tag +dissertation thread:7dc536441e6deade4256a46d46451221
>>
>> real 0m0.812s
>> user 0m0.022s
>> sys 0m0.037s
>
> Things work quite a bit faster than that on my machine:
>
> $ time notmuch tag +foo id:5641883d1001140730l22832715ld6bdc95c9938d314@mail.gmail.com
>
> real 0m0.024s
> user 0m0.012s
> sys 0m0.004s
>
> But that could just be system differences.
Possibly, though my machine is more than capable of what it's doing -
a Q6600 with 4gb RAM
>> And tagging all my messages is really horrible:
>>
>> $ time notmuch tag +foobar tag:inbox
>>
>> real 0m5.076s
>> user 0m3.688s
>> sys 0m0.105s
>
> For this operation, I can't really compare. How many messages are you
> tagging? Here's that operation for me with 525 messages in my inbox:
A few thousand (4k, I believe)
>> That xapian-svn was built from svn HEAD right now, so I'm assuming it
>> contains the #250 fix (http://trac.xapian.org/changeset/13808)
>
> Which I think means that things could have been even *much* slower
> before. ;-)
>
> The Xapian defect #250 was just one, initial (and obvious) performance
> problem. [Though, as I mentioned in a previous thread, if you're using a
> Xapian flint database, (look for .notmuch/xapian/iamflint), then you
> won't get the benefit of the Xapian fix until you rebuild your notmuch
> database from scratch with a current notmuch.]
I didn't know about this need to rebuild, but I tried that and didn't
have any more success sadly.
> Once you've verified that you've got the #250 fix functional, there
> could still be lots of performance bugs. And it would be time to start
> profiling.
>
> [...]
I'm pressed for time at the moment, but in a few weeks I might have
some time to investigate here...
--
Oliver Charles / aCiD2
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Notmuch performance problems on OSX
2010-01-18 18:35 ` Oliver Charles
@ 2010-02-09 20:46 ` Oliver Charles
2010-02-09 22:09 ` Olly Betts
0 siblings, 1 reply; 8+ messages in thread
From: Oliver Charles @ 2010-02-09 20:46 UTC (permalink / raw)
To: Carl Worth; +Cc: notmuch
I just upgraded to xapian-core HEAD and notmuch master tip today, in
desparation to get away from GMail. Sadly it's still taking at least
0.7s to tag a single thread (with one message). I'm really eager to
solve this, could anyone give me any pointers on how I could go about
profiling it or finding the cause of this problem?
--
Oliver Charles / aCiD2
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Notmuch performance problems on OSX
2010-02-09 20:46 ` Oliver Charles
@ 2010-02-09 22:09 ` Olly Betts
2010-02-09 22:16 ` Oliver Charles
0 siblings, 1 reply; 8+ messages in thread
From: Olly Betts @ 2010-02-09 22:09 UTC (permalink / raw)
To: notmuch
On 2010-02-09, Oliver Charles wrote:
> I just upgraded to xapian-core HEAD and notmuch master tip today, in
> desparation to get away from GMail. Sadly it's still taking at least
> 0.7s to tag a single thread (with one message). I'm really eager to
> solve this, could anyone give me any pointers on how I could go about
> profiling it or finding the cause of this problem?
The first thing to try is disabling use of F_FULLFSYNC. You'll need to
run this command in the xapian-core source tree to comment out the F_FULLFSYNC
code:
perl -pi -e 's/^#ifdef F_FULLFSYNC/#if 0/' backends/*/*_io.h
Then run "make" and "make install".
This makes you a bit more vulnerable to power failures, but no worse than
a typical Unix system. There's some background here:
http://lists.apple.com/archives/Darwin-dev/2005/Feb/msg00072.html
Assuming that helps, then (a) you have a workaround, and (b) we'll know for
sure it is F_FULLFSYNC to blame.
I've created a ticket for a change to Xapian which should help here, but
not had a chance to work on it yet:
http://trac.xapian.org/ticket/426
Cheers,
Olly
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Notmuch performance problems on OSX
2010-02-09 22:09 ` Olly Betts
@ 2010-02-09 22:16 ` Oliver Charles
0 siblings, 0 replies; 8+ messages in thread
From: Oliver Charles @ 2010-02-09 22:16 UTC (permalink / raw)
To: Olly Betts; +Cc: notmuch
On Tue, Feb 9, 2010 at 10:09 PM, Olly Betts <olly@survex.com> wrote:
> On 2010-02-09, Oliver Charles wrote:
>> I just upgraded to xapian-core HEAD and notmuch master tip today, in
>> desparation to get away from GMail. Sadly it's still taking at least
>> 0.7s to tag a single thread (with one message). I'm really eager to
>> solve this, could anyone give me any pointers on how I could go about
>> profiling it or finding the cause of this problem?
>
> The first thing to try is disabling use of F_FULLFSYNC. You'll need to
> run this command in the xapian-core source tree to comment out the F_FULLFSYNC
> code:
>
> perl -pi -e 's/^#ifdef F_FULLFSYNC/#if 0/' backends/*/*_io.h
>
> Then run "make" and "make install".
$ time notmuch tag +inbox thread:6e66368b7887184c6d4c63653211b3f2
real 0m0.067s
user 0m0.036s
sys 0m0.028s
Now this looks a little bit more usable!
> Assuming that helps, then (a) you have a workaround, and (b) we'll know for
> sure it is F_FULLFSYNC to blame.
Looks like this is the case.
> I've created a ticket for a change to Xapian which should help here, but
> not had a chance to work on it yet:
>
> http://trac.xapian.org/ticket/426
I will add my info there and follow the ticket if I can.
--
Oliver Charles / aCiD2
(Olly, sorry about the double email - in all my excitement I didn't
hit reply all :))
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Notmuch performance problems on OSX
2010-01-14 15:30 Notmuch performance problems on OSX Oliver Charles
2010-01-14 23:16 ` Carl Worth
@ 2010-01-15 3:58 ` Olly Betts
2010-02-16 3:53 ` Stewart Smith
1 sibling, 1 reply; 8+ messages in thread
From: Olly Betts @ 2010-01-15 3:58 UTC (permalink / raw)
To: notmuch
On 2010-01-14, Oliver Charles wrote:
> I've installed the latest notmuch from Git at this time of writing,
> along with Xapian from SVN head. However, just tagging a single thread
> with only one message seems to take too long:
One difference between OS X and other systems is that OS X supports the
F_FULLSYNC ioctl, and other systems don't (currently, at least AFAIK)
and Xapian uses that if it is available to ensure that changes have
actually made it to disk:
http://trac.xapian.org/ticket/288
On other systems, it uses fdatasync() or fsync(), which typically just
ensure that the data has left the OS - it can sit in disk controller or
drive caches for potentially seconds longer. This call happens once
per table for every (explicit or implicit) flush on a database.
I can see an issue here which is that currently Xapian writes the base
file for the table, then syncs it, then does the next table. I bet it
would be more efficient to write them all and then sync them all,
especially with F_FULLSYNC.
I'll take a look at doing that, and have created a ticket for it:
http://trac.xapian.org/ticket/426
If after that this is still causing problems, it should probably be made
configurable what (if any) flushing is done. If you're on a UPS-backed
server, you probably don't need such paranoia.
Cheers,
Olly
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Notmuch performance problems on OSX
2010-01-15 3:58 ` Olly Betts
@ 2010-02-16 3:53 ` Stewart Smith
0 siblings, 0 replies; 8+ messages in thread
From: Stewart Smith @ 2010-02-16 3:53 UTC (permalink / raw)
To: Olly Betts, notmuch
On Fri, 15 Jan 2010 03:58:50 +0000 (UTC), Olly Betts <olly@survex.com> wrote:
> One difference between OS X and other systems is that OS X supports the
> F_FULLSYNC ioctl, and other systems don't (currently, at least AFAIK)
> and Xapian uses that if it is available to ensure that changes have
> actually made it to disk:
>
> http://trac.xapian.org/ticket/288
>
> On other systems, it uses fdatasync() or fsync(), which typically just
> ensure that the data has left the OS - it can sit in disk controller or
> drive caches for potentially seconds longer. This call happens once
> per table for every (explicit or implicit) flush on a database.
At least if you OS and file system don't hate you (e.g. XFS on Linux),
then fsync() really does flush the drive cache.
Also keep in mind that the OSX file system (HFS+) was great for
1985. It's essentially single threaded :/
--
Stewart Smith
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2010-02-16 3:53 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2010-01-14 15:30 Notmuch performance problems on OSX Oliver Charles
2010-01-14 23:16 ` Carl Worth
2010-01-18 18:35 ` Oliver Charles
2010-02-09 20:46 ` Oliver Charles
2010-02-09 22:09 ` Olly Betts
2010-02-09 22:16 ` Oliver Charles
2010-01-15 3:58 ` Olly Betts
2010-02-16 3:53 ` Stewart Smith
Code repositories for project(s) associated with this public inbox
https://yhetil.org/notmuch.git/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).