notmuch-hello buffer slow doe to slow query

notmuch-hello buffer slow doe to slow query

[Nils Dagsson Moskopp]

[-- Attachment #1: Type: text/plain, Size: 1160 bytes --]

Hello,

If notmuch-hello includes a saved search with a slow query, switching to
a notmuch-hello buffer is very slow due to notmuch-mode updating counts
for search results. mjw1009 suggested "(setq notmuch-hello-auto-refresh
nil)", which stops the counting and works around the problem.

Fundamentally, the problem is a slow query. On my laptop (Thinkpad T60),
many things are pretty much instant, even though I have a HDD, no SSD:

> ; time notmuch count 'tag:inbox and tag:list'
> 25452
> 0.02user 0.00system 0:00.03elapsed 72%CPU (0avgtext+0avgdata 3852maxresident)k
> 0inputs+0outputs (0major+1135minor)pagefaults 0swaps

However, from-queries take their time:

> ; time notmuch count 'not tag:replied and to:nils@dieweltistgarnichtso.net'
> 5328
> 0.10user 0.15system 0:14.14elapsed 1%CPU (0avgtext+0avgdata 3472maxresident)k
> 157544inputs+0outputs (0major+1039minor)pagefaults 0swaps

mjw1009 can reproduce if the from-query contains an "@" and thinks the
problem may be "something deeper down in notmuch (actually probably in
xapian)".


Greetings,
-- 
Nils Dagsson Moskopp // erlehmann
<http://dieweltistgarnichtso.net>

[-- Attachment #2: Type: application/pgp-signature, Size: 212 bytes --]

Re: notmuch-hello buffer slow doe to slow query

[David Bremner]

[-- Attachment #1: Type: text/plain, Size: 1095 bytes --]

Nils Dagsson Moskopp <nils@dieweltistgarnichtso.net> writes:

>
> mjw1009 can reproduce if the from-query contains an "@" and thinks the
> problem may be "something deeper down in notmuch (actually probably in
> xapian)".

My own experiments contradict that (see attached patch for a test to
run). The exact address is about twice as fast to count on both of my
machines, which matches the amount of I/O. FWIW, my SSD based laptop is
about 10x faster (compared to a similarly specced desktop with HD) for
both of them, which kindof reinforces how I/O bound this is.  

Replacing 'count' with 'search' gives pretty similar relative behaviour,
although much slower since it seems to do about 10x I/O.  

Finally after an initial "notmuch tag +fromlinus
from:torvalds@linux-foundation.org", "notmuch count tag:fromlinus" seems
to be about 5-10x faster than "notmuch count
from:torvalds@linux-foundation.org".  So that might be a workaround for
some people.

To sum up, I think the underlying question is whether the amount of I/O
that count does can be reduced when using non-boolean query terms.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-WIP-new-performance-test-for-notmuch-count.patch --]
[-- Type: text/x-diff, Size: 1118 bytes --]

From d3fbbe7ac0a1fb5b5691674226a3a3d000e15a1c Mon Sep 17 00:00:00 2001
From: David Bremner <david@tethera.net>
Date: Sat, 12 Apr 2014 11:50:22 -0300
Subject: [PATCH] WIP: new performance test for notmuch count

Probably not suitable for production because of sudo
---
 performance-test/T03-count.sh | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)
 create mode 100755 performance-test/T03-count.sh

diff --git a/performance-test/T03-count.sh b/performance-test/T03-count.sh
new file mode 100755
index 0000000..e74437e
--- /dev/null
+++ b/performance-test/T03-count.sh
@@ -0,0 +1,23 @@
+#!/bin/bash
+
+test_description='counting'
+
+. ./perf-test-lib.sh
+
+time_start
+
+for i in $(seq 1 10); do
+    echo 3 | sudo tee /proc/sys/vm/drop_caches
+    notmuch --version > /dev/null
+    time_run "from:torvalds@linux-foundation.org  " \
+	"notmuch count from:torvalds@linux-foundation.org"
+
+    echo 3 | sudo tee /proc/sys/vm/drop_caches
+    notmuch --version > /dev/null
+
+    time_run "from:linux-foundation.org           " \
+	"notmuch count from:linux-foundation.org"
+done
+
+
+time_done
-- 
1.9.1