From: Tom Bulli <mrbulli@yahoo.com>
To: Felipe Contreras <felipe.contreras@gmail.com>
Cc: "notmuch@notmuchmail.org" <notmuch@notmuchmail.org>
Subject: Re: Notmuch indexing 21 million emails
Date: Wed, 23 Nov 2011 09:20:33 -0800 (PST) [thread overview]
Message-ID: <1322068833.15983.YahooMailNeo@web36504.mail.mud.yahoo.com> (raw)
In-Reply-To: <CAMP44s2zf-OpN4u+zv1g+q_nrgkMHUGLUT7cVFZMJ=Xdy-9WdA@mail.gmail.com>
I have been able to speed that up with the code below - basically increase "XAPIAN_FLUSH_THRESHOLD" based on the total virtual memory divided by the avg. size of an email times 2 (just to be safe). It seems to be faster since it does less xapian updates. However, I have a nagging feeling that ""XAPIAN_FLUSH_THRESHOLD" could even be higher since I don't see any increase in used memory (via "top -d 1"). The server in question has eight CPU cores and 8GB RAM, running Debian squeeze on a 32bit architecture (I know - but it is what it is :) ).
# Assume an average size of 120KB per email
# and use at most half the virtual memory
XFT=$(($(free -otk | awk '/^Total/ {print $2}') / 240))
# Keep more index info in memory before flushing to disk
[ $XFT -lt 10000 ] && XFT=10000
su - archive -c "export XAPIAN_FLUSH_THRESHOLD=$XFT; notmuch new --verbose"
----- Original Message -----
> From: Felipe Contreras <felipe.contreras@gmail.com>
> To: Tom Bulli <mrbulli@yahoo.com>
> Cc: "notmuch@notmuchmail.org" <notmuch@notmuchmail.org>
> Sent: Wednesday, November 23, 2011 10:40 AM
> Subject: Re: Notmuch indexing 21 million emails
>
> On Tue, Nov 22, 2011 at 5:02 AM, Tom Bulli <mrbulli@yahoo.com> wrote:
>> I have a project where I need to search about 21 emails - and decided to
> use "notmuch" for it. The system is a Debian Squeeze, the notmuch
> version is "0.8-1~bpo60+1" from "kyria's" private
> repository.
>>
>> I am running the "notmuch new" for approx. 4 days now - and
> according to "not,uch count" it has indexed about 4.5 million emails.
>>
>> Is this expected performance? Is there any way to speed that up?
>
> It would be nice to run something like this with OProfile (or perf)
> and see if there's some obvious fixes.
>
> --
> Felipe Contreras
>
prev parent reply other threads:[~2011-11-23 17:20 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-11-22 3:02 Notmuch indexing 21 million emails Tom Bulli
2011-11-23 3:20 ` Austin Clements
2011-11-23 15:40 ` Felipe Contreras
2011-11-23 17:20 ` Tom Bulli [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://notmuchmail.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1322068833.15983.YahooMailNeo@web36504.mail.mud.yahoo.com \
--to=mrbulli@yahoo.com \
--cc=felipe.contreras@gmail.com \
--cc=notmuch@notmuchmail.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://yhetil.org/notmuch.git/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).