From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by olra.theworths.org (Postfix) with ESMTP id 357BE431FD0 for ; Tue, 12 Jul 2011 13:10:09 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at olra.theworths.org X-Spam-Flag: NO X-Spam-Score: -0.7 X-Spam-Level: X-Spam-Status: No, score=-0.7 tagged_above=-999 required=5 tests=[RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled Received: from olra.theworths.org ([127.0.0.1]) by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id mC1Zd2qgNlf6 for ; Tue, 12 Jul 2011 13:10:08 -0700 (PDT) Received: from mail-vw0-f53.google.com (mail-vw0-f53.google.com [209.85.212.53]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by olra.theworths.org (Postfix) with ESMTPS id B11BC431FB6 for ; Tue, 12 Jul 2011 13:10:08 -0700 (PDT) Received: by vws13 with SMTP id 13so4139244vws.26 for ; Tue, 12 Jul 2011 13:10:08 -0700 (PDT) Received: by 10.220.7.79 with SMTP id c15mr119732vcc.3.1310501407819; Tue, 12 Jul 2011 13:10:07 -0700 (PDT) Received: from compy.jasonwoof.org (c-98-216-100-90.hsd1.ma.comcast.net [98.216.100.90]) by mx.google.com with ESMTPS id bd5sm2384994vcb.11.2011.07.12.13.10.05 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 12 Jul 2011 13:10:05 -0700 (PDT) Date: Tue, 12 Jul 2011 16:10:02 -0400 From: Jason Woofenden To: Notmuch Mail Subject: Re: Slowness (search opens every email file?) Message-ID: <20110712201002.GA15019@compy.jasonwoof.org> Mail-Followup-To: Notmuch Mail References: <20110711190721.GA5386@compy.jasonwoof.org> <20110711215858.GA18884@brick.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20110711215858.GA18884@brick.lan> User-Agent: Mutt/1.5.21 (2010-09-15) X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Jul 2011 20:10:10 -0000 On 2011-07-11 10:58PM, Patrick Totzke wrote: > Hi Jason, > On Mon, Jul 11, 2011 at 03:07:21PM -0400, Jason Woofenden wrote: > > notmuch search tag:foo is slow! > > > yes, i've just used the vim ui for the first time and i agreee, its sluggish, > searching for * takes a while. It's not the vim ui that's the bottleneck. The underlying notmuch search command is slow. I gave examples in my last e-mail: >> notmuch search tag:foo took 0.5 seconds for 32 threads >> >> notmuch search foo took 6.4 seconds for 130 threads > > (when my e-mail files are not already in the disk cache) > > > > I saw on my activity monitor applet that it was using mostly i/o, > > and started to wonder if it was opening every e-mail. I little work > > with strace and searching revealed that this command was opening > > many many e-mail files from my maildir(s). I spent a little while > > digging around in the notmuch source, and didn't see where it was > > opening the email files. > > I cannot reproduce this. I'm no expert, but at least the output of > strace vim -c ":NotMuch" 2>log :NotMuch just shows the mailboxes. that's fast. It's showing the contents that is slow. And it's not vim, it's because the notmuch command is slow. Try this in a terminal: strace notmuch search tag:flagged 2>&1 | grep 'open(.*/cur/' Of course change the tag if you don't have flagged messages. > A quick browse through notmuch.vim tells me that > 1) it doesn't use notmuchs json output, i think it should, as iirc > this api is considered 'more stable' and is easier to parse than the > default output. More importantly, I like this idea. I did some work earlier on improving the message parsing in the vim ui. Might be better to use the json. I'll look into json parsing in vim. > 2) the output of notmuch is copied into a list. This will of course > be slow if your query matches a lot of messages. Could this be done > by asyncronously writing to the buffer somehow? I'm pretty sure vim doesn't do asynchronous anything. That came up in a vim vs emacs article I read. Take care, - Jason