From: Austin Clements <amdragon@MIT.EDU>
To: Petter Reinholdtsen <pere@hungry.com>
Cc: notmuch@notmuchmail.org
Subject: Re: 'notmuch new' leaking memory and getting slower over time?
Date: Tue, 22 Nov 2011 21:48:18 -0500 [thread overview]
Message-ID: <20111123024818.GI9351@mit.edu> (raw)
In-Reply-To: <2flfwhht87d.fsf@diskless.uio.no>
Quoth Petter Reinholdtsen on Nov 21 at 11:35 pm:
> The indexing took 36 hours. At the start it claimed it would take 10
> hours, and it continued to underestimate the amount of time left until
> the very end. It claimed to have 1 hour left when I checked before I
> went to bed, and claimed to have 15 minutes left when I woke up 6-7
> hours later.
notmuch new does a simple linear extrapolation based on how many files
it's examined and how many there are total. This is doomed to
undershoot at least because indexing becomes slower as the database
grows (B-tree insertion is O(log N), fragmentation will increase over
time, posting lists will get longer...).
I'm not sure much can be done about the estimate at the beginning,
short of throwing in some fudge factor, but the estimates later in the
process would be much more accurate if it used a sliding window,
rather than measuring from the beginning.
> Shortly before the indexing finished, the notmuch process was using 1.2
> GiB of resident memory according to top. Is the process leaking memory?
It's possible this is just memory fragmentation, but it definitely
sounds like a leak. talloc has some tools for tracking down leaks and
it would be good to heap profile notmuch new, but to my knowledge
nobody's applied these tools.
next prev parent reply other threads:[~2011-11-23 2:46 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-11-21 22:35 'notmuch new' leaking memory and getting slower over time? Petter Reinholdtsen
2011-11-22 21:50 ` David Bremner
2011-11-23 2:48 ` Austin Clements [this message]
2012-12-10 11:50 ` David Bremner
2012-12-10 12:39 ` David Bremner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://notmuchmail.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20111123024818.GI9351@mit.edu \
--to=amdragon@mit.edu \
--cc=notmuch@notmuchmail.org \
--cc=pere@hungry.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://yhetil.org/notmuch.git/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).