From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by olra.theworths.org (Postfix) with ESMTP id 48066431FBC for ; Sun, 22 Nov 2009 00:37:31 -0800 (PST) X-Virus-Scanned: Debian amavisd-new at olra.theworths.org Received: from olra.theworths.org ([127.0.0.1]) by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 07CYxhR9p5Wi for ; Sun, 22 Nov 2009 00:37:30 -0800 (PST) Received: from vuizook.err.no (vuizook.err.no [85.19.221.46]) by olra.theworths.org (Postfix) with ESMTP id 51C1C431FAE for ; Sun, 22 Nov 2009 00:37:30 -0800 (PST) Received: from cha92-13-88-165-248-19.fbx.proxad.net ([88.165.248.19] helo=jigen) by vuizook.err.no with esmtps (TLS-1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.69) (envelope-from ) id 1NC7wk-0006Ts-2P; Sun, 22 Nov 2009 09:37:28 +0100 Received: from mh by jigen with local (Exim 4.69) (envelope-from ) id 1NC7w6-0002OZ-93; Sun, 22 Nov 2009 09:36:46 +0100 Date: Sun, 22 Nov 2009 09:36:46 +0100 From: Mike Hommey To: Brett Viren Message-ID: <20091122083646.GA8420@glandium.org> References: <20091121145111.GB19397@excalibur.local> <87fx874xj5.fsf@yoom.home.cworth.org> <46263c600911211436s5826015eqc5fc18a4164245cb@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <46263c600911211436s5826015eqc5fc18a4164245cb@mail.gmail.com> X-GPG-Fingerprint: A479 A824 265C B2A5 FC54 8D1E DE4B DA2C 54FD 2A58 User-Agent: Mutt/1.5.20 (2009-06-14) Cc: notmuch@notmuchmail.org Subject: Re: 25 minutes load time with emacs -f notmuch X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Nov 2009 08:37:31 -0000 On Sat, Nov 21, 2009 at 05:36:18PM -0500, Brett Viren wrote: > On Sat, Nov 21, 2009 at 12:07 PM, Carl Worth wrote: > > > Though, frankly, I think we need to fix "notmuch new" to do much better > > than 40 files/sec. > > Just a "me too". > > Processed 130871 total files in 38m 7s (57 files/sec.). > Added 102723 new messages to the database (not much, really). > > This was ~2GB of mail on a 2.5GHz CPU. That seems pretty reasonable > to me but I'd like to rerun the "notmuch new" under google perftools > to see if there are any obvious bottlenecks that might be cleaned up. FWIW, my 90k+ messages mailbox was imported at a pace of 130 files/sec, and my CPU is "only" 2.2GHz, but I have a SSD. A good share of the bottlenecks is "simply" I/O. Don't forget having a lot of small files sucks I/O wise, as files are most likely spread all over the disk. A good test, if you have enough memory, would be to put your mailbox in a tmpfs, and see how fast that imports. Mike