notmuch painfully slow

notmuch painfully slow

[Xavier Maillard]

Hi,

as time goes, notmuch is really becoming slower and slower. It may be
normal as it has more and more messages to "track" but I am not sure it
is that normal give these numbers:

- notmuch version: notmuch 0.5-133-gc509598
- notmuch count: 18475
- uname: Linux kcals 2.6.33.4-smp #2 SMP Wed May 12 22:47:36 CDT 2010
         i686 Intel(R) Atom(TM) CPU N270   @ 1.60GHz GenuineIntel GNU/Linux
- Load average: 0.00 0.00 0.00 1/144 28377
- Memory:
             total       used       free     shared    buffers     cached
Mem:           992        881        111          0        253        500
-/+ buffers/cache:        127        865
Swap:          506          0        505

This is on a Slackware GNU/linux system on a Eee PC 901 (thus SSD
storage). I am using my computer *exclusively* in the virtual linux
console (tty) and 99% of my tasks are done through GNU Emacs (dunno if
is of any interest for this problem though).

My sync script is taking really long time to achieve, updating one mail
at a time is also really slow (several seconds). In the worst cases,
updating my collection can take upto 15 minutes and updating just one
message can take up to the minute too !

What is happening here ? Is there any maintenance task I should do onto
my mail collection ?

Thank you for your help.

/Xavier

Re: notmuch painfully slow

[Pieter Praet]

On Fri, 22 Apr 2011 16:14:52 +0200, Xavier Maillard <xma@gnu.org> wrote:
> Hi,
> 
> as time goes, notmuch is really becoming slower and slower. It may be
> normal as it has more and more messages to "track" but I am not sure it
> is that normal give these numbers:
> 
> - notmuch version: notmuch 0.5-133-gc509598
> - notmuch count: 18475
> - uname: Linux kcals 2.6.33.4-smp #2 SMP Wed May 12 22:47:36 CDT 2010
>          i686 Intel(R) Atom(TM) CPU N270   @ 1.60GHz GenuineIntel GNU/Linux
> - Load average: 0.00 0.00 0.00 1/144 28377
> - Memory:
>              total       used       free     shared    buffers     cached
> Mem:           992        881        111          0        253        500
> -/+ buffers/cache:        127        865
> Swap:          506          0        505
> 
> This is on a Slackware GNU/linux system on a Eee PC 901 (thus SSD
> storage). I am using my computer *exclusively* in the virtual linux
> console (tty) and 99% of my tasks are done through GNU Emacs (dunno if
> is of any interest for this problem though).
> 
> My sync script is taking really long time to achieve, updating one mail
> at a time is also really slow (several seconds). In the worst cases,
> updating my collection can take upto 15 minutes and updating just one
> message can take up to the minute too !
> 
> What is happening here ? Is there any maintenance task I should do onto
> my mail collection ?
> 
> Thank you for your help.

Have you switched your Xapian database from flint to chert yet? This
really makes a huge difference in responsiveness.

> /Xavier
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch


Peace

-Pieter

Re: notmuch painfully slow

[Florian Friesdorf]

[-- Attachment #1: Type: text/plain, Size: 754 bytes --]

On Fri, 22 Apr 2011 22:18:09 +0200, Pieter Praet <pieter@praet.org> wrote:
> Have you switched your Xapian database from flint to chert yet? This
> really makes a huge difference in responsiveness.

Is this the switch from xapian 1.0.x to 1.2.x or are further steps
necessary?

Are there instructions on how to do the switch? For the mailing list
archive a link would be great - a quick search for 'xapian flint chert'
was unsuccessful.

In case there are no instructions yet, I'm happy to conclude what is
gathered here and add it to notmuch's webpage.

-- 
Florian Friesdorf <flo@chaoflow.net>
  GPG FPR: 7A13 5EEE 1421 9FC2 108D  BAAF 38F8 99A3 0C45 F083
Jabber/XMPP: flo@chaoflow.net
IRC: chaoflow on freenode,ircnet,blafasel,OFTC

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: notmuch painfully slow

[Simon Fondrie-Teitler]

[-- Attachment #1: Type: text/plain, Size: 816 bytes --]

On Sat, 23 Apr 2011 07:18:52 +0200, Florian Friesdorf <flo@chaoflow.net> wrote:
> On Fri, 22 Apr 2011 22:18:09 +0200, Pieter Praet <pieter@praet.org> wrote:
> Is this the switch from xapian 1.0.x to 1.2.x or are further steps
> necessary?
> 
> Are there instructions on how to do the switch? For the mailing list
> archive a link would be great - a quick search for 'xapian flint chert'
> was unsuccessful.
Notmuch has been very slow for me, and I decided to give cert a try. A
search turned up this message:
http://notmuchmail.org/pipermail/notmuch/2010/001105.html 
as well as: http://notmuchmail.org/pipermail/notmuch/2011/004048.html

It seems you have to rebuild the database with "XAPIAN_PREFER_CHERT=1
notmuch new"

I have done this, and it does seemed to have sped up notmuch. 

  --Simon

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: notmuch painfully slow

[Xavier Maillard]

Hi Pieter,

On Fri, 22 Apr 2011 22:18:09 +0200, Pieter Praet <pieter@praet.org> wrote:
> On Fri, 22 Apr 2011 16:14:52 +0200, Xavier Maillard <xma@gnu.org> wrote:
> >
> > as time goes, notmuch is really becoming slower and slower. It may be
> > normal as it has more and more messages to "track" but I am not sure it
> > is that normal give these numbers:

[SNIP]

> > What is happening here ? Is there any maintenance task I should do onto
> > my mail collection ?
> > 
> > Thank you for your help.
> 
> Have you switched your Xapian database from flint to chert yet? This
> really makes a huge difference in responsiveness.

Obviously, no, I did not. I even did not know about that ;) What's the
story with this and what's the way to do this exactly ?

Thank you.

/Xavier

Re: notmuch painfully slow

[Pieter Praet]

On Sat, 23 Apr 2011 07:18:52 +0200, Florian Friesdorf <flo@chaoflow.net> wrote:
> [...]
> Is this the switch from xapian 1.0.x to 1.2.x or are further steps
> necessary?
> 
> Are there instructions on how to do the switch? For the mailing list
> archive a link would be great - a quick search for 'xapian flint chert'
> was unsuccessful.
> 
> In case there are no instructions yet, I'm happy to conclude what is
> gathered here and add it to notmuch's webpage.
> 
> -- 
> Florian Friesdorf <flo@chaoflow.net>
>   GPG FPR: 7A13 5EEE 1421 9FC2 108D  BAAF 38F8 99A3 0C45 F083
> Jabber/XMPP: flo@chaoflow.net
> IRC: chaoflow on freenode,ircnet,blafasel,OFTC
Non-text part: application/pgp-signature

On Sat, 23 Apr 2011 09:20:53 +0200, Xavier Maillard <xma@gnu.org> wrote:
> [...]
> Obviously, no, I did not. I even did not know about that ;) What's the
> story with this and what's the way to do this exactly ?
> 
> Thank you.
> 
> /Xavier


Both Carl [1] and Sebastian [2] posted about this a little while ago.

Bottom line: Chert is the new db format (replacing Flint) as of Xapian
1.2.x [3], and should -in most cases- be alot faster, whilst consuming
less disk space.

If .notmuch/xapian contains a file called "iamflint", you're still on
the old format. Otherwise it's called (surprise) "iamchert". Perhaps
somewhat confusingly, "flintlock" remains though.

Switching your notmuch db is straightforward:
>> notmuch dump ${tagdump}
>> rm -rf .notmuch/xapian
>> notmuch new
>> notmuch restore ${tagdump}
>> mv .notmuch/xapian .notmuch/xapian-fat
>> xapian-compact --no-renumber .notmuch/xapian-fat .notmuch/xapian
>> rm -rf .notmuch/xapian-fat

The last 3 lines are entirely optional and will cause writes to be a
bit more expensive in the short term, but will significantly reduce
the db's size, which your SSD will surely appreciate.


Peace

-Pieter

[1] id:"87ocl1lut1.fsf@yoom.home.cworth.org"
[2] id:"87lj0iqa3c.fsf@SSpaeth.de"
[3] http://www.archlinux.org/packages/?q=xapian-core

Re: notmuch painfully slow

[Florian Friesdorf]

[-- Attachment #1: Type: text/plain, Size: 346 bytes --]


thank you all for your answers!

I am now also on a compacted chert database and started a "performance"
section:

http://notmuchmail.org/performance/

-- 
Florian Friesdorf <flo@chaoflow.net>
  GPG FPR: 7A13 5EEE 1421 9FC2 108D  BAAF 38F8 99A3 0C45 F083
Jabber/XMPP: flo@chaoflow.net
IRC: chaoflow on freenode,ircnet,blafasel,OFTC

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: notmuch painfully slow

[Xavier Maillard]

Hi,

On Tue, 26 Apr 2011 22:04:13 +0200, Florian Friesdorf <flo@chaoflow.net> wrote:
> 
> thank you all for your answers!

I second that and thanks to your performance page, I finally switched to
chert format database. That's day and night !

Thank you !

/Xavier