Displaying tags with many messages very slow in Emacs

Displaying tags with many messages very slow in Emacs

[webservice]

Hello,

I am planning to have some tags with more than 10k messages, and I am experiencing very slow list display in emacs.

Installation on OS X 10.6, using brew: notmuch 0.9, chert database format. Emacs 24.0.90.1 on iMac, intel 2.8 Ghz i7, 8Gb ram.

Testing 'notmuch new' on a Maildir folder with about 30000 messages (about one fourth of my archive).

At beginning notmuch was indexing about 200 msgs/sec. The more it indexed, the slower it became. After about 20000 msgs it slowed down to 70 msgs/sec. I am not sure if that is a behaviour due to Xapian.

Anyway, after initial indexing, starting notmuch from emacs, I have:

"Welcome to notmuch. You have 34,366 messages."

Once I start browsing the inbox (34,208), Emacs displays the initial screenful, but the whole list takes quite a while to load, about 2 minutes. After that time, in the status bar I see "Top of 2M".

That happens every time I am browsing the inbox, in order to assign proper tags.

The real issue, though, is that scrolling down the list has a noticeable delay, enough to make it basically unusable.

Is there something I do not know that I should do to improve that speed (either from within Emacs, or in notmuch config file?

I would be very grateful for any advice or pointer on how to speed up the notmuch experience.


Thank you,

Sam

Re: Displaying tags with many messages very slow in Emacs

[James Vasile]

On August 20th, I sent a message to this list (Message-ID:
<87vctrq4or.fsf@hackervisions.org>) that limits the view to a small
number threads.  You can set the number of threads (or turn it off).  I
did this because I never want to page through more than a screen or two
of threads.  If I need more results, I filter my search or choose better
search terms.  This speeds things up considerably on my (admittedly old)
machine.

When I posted the patch, Daniel Schoepe suggested a few improvements.  I
didn't make those improvements, but I suppose I should.

Best,
James

Re: Displaying tags with many messages very slow in Emacs

[bnt]

On 2011, Oct 31, at 13:47 , James Vasile wrote:

> On August 20th, I sent a message to this list (Message-ID:
> <87vctrq4or.fsf@hackervisions.org>) that limits the view to a small
> number threads.  You can set the number of threads (or turn it off).  I
> did this because I never want to page through more than a screen or two
> of threads.  If I need more results, I filter my search or choose better
> search terms.  This speeds things up considerably on my (admittedly old)
> machine.
> 
> When I posted the patch, Daniel Schoepe suggested a few improvements.  I
> didn't make those improvements, but I suppose I should.
> 
> Best,
> James


Hi James,

Thank you for the pointer.

I saw that the two patches (yours, and the "Do not query on notmuch-search exit" one) are not in git yet:

http://git.notmuchmail.org/git/notmuch/history/HEAD:/emacs/notmuch.el

(The last commit according to that page was on 2011-06-23).

I am not too proficient with the patch utility. Is there some kind of scheduled git commits timeline?

If not, I guess it will be a good time to learn patching.

Thank you once again,

Sam

Re: Displaying tags with many messages very slow in Emacs

[Daniel Schoepe]

[-- Attachment #1: Type: text/plain, Size: 1110 bytes --]

On Mon, 31 Oct 2011 16:43:22 +0100, bnt <webservice@bontempi.net> wrote:
> I am not too proficient with the patch utility. Is there some kind of
> scheduled git commits timeline?

Unfortunately, there's a huge number of outstanding patches at the
moment, so it's impossible to say when/if it will be applied.

> If not, I guess it will be a good time to learn patching.

There's no need to use the patch binary directly when the patches are
sent in a format git understands, which is generally the case here. To
apply patches from a thread, you can use `git am patches.mbox' where
patches.mbox contains the mails with the patches.

If you're using the emacs UI for notmuch, you can open only those
messages with the patches you want in a thread and pipe them to an mbox
file like this:

C-u | cat > ~/patches.mbox

If there are no conflicts due to more recent changes to the repository,
git am patches.mbox will apply them using the commit messages from the
mails.

FWIW: There's another more recent series of patches that accomplishes a
similar goal: id:"cover.1319833617.git.jani@nikula.org"

Cheers,
Daniel

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: Displaying tags with many messages very slow in Emacs

[bnt]

On 2011, Oct 31, at 19:25 , Daniel Schoepe wrote:

> On Mon, 31 Oct 2011 16:43:22 +0100, bnt <webservice@bontempi.net> wrote:
>> I am not too proficient with the patch utility. Is there some kind of
>> scheduled git commits timeline?
> 
> Unfortunately, there's a huge number of outstanding patches at the
> moment, so it's impossible to say when/if it will be applied.
> 
>> If not, I guess it will be a good time to learn patching.
> 
> There's no need to use the patch binary directly when the patches are
> sent in a format git understands, which is generally the case here. To
> apply patches from a thread, you can use `git am patches.mbox' where
> patches.mbox contains the mails with the patches.
> 
> If you're using the emacs UI for notmuch, you can open only those
> messages with the patches you want in a thread and pipe them to an mbox
> file like this:
> 
> C-u | cat > ~/patches.mbox
> 
> If there are no conflicts due to more recent changes to the repository,
> git am patches.mbox will apply them using the commit messages from the
> mails.
> 
> FWIW: There's another more recent series of patches that accomplishes a
> similar goal: id:"cover.1319833617.git.jani@nikula.org"
> 
> Cheers,
> Daniel


Thank you so much for the explanation, and thanks to Jani for those patches.

I will work my way into applying them.

As just a plain user, I was wondering if it would be possible to have a "dev" or "untested" branch in the git repo, where one could just get all the latest patches? 

Of course it would not be desirable if it would just complicate the developers life.

Thank you so much once again,

Sam