[PATCH] perf-test: add test for reindexing after removal/renaming msgs

[PATCH] perf-test: add test for reindexing after removal/renaming msgs

[David Bremner]

At this point it is a bit tricky to measure the performance increase
from the new message deletion code, since the same commit (9ad19e4)
that improved the performance also seems to have fixed a bug with an
uncaught Xapian exception triggered by this test.
---
 performance-test/T03-reindex.sh | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)

diff --git a/performance-test/T03-reindex.sh b/performance-test/T03-reindex.sh
index 8db52a33..afa11df8 100755
--- a/performance-test/T03-reindex.sh
+++ b/performance-test/T03-reindex.sh
@@ -10,4 +10,32 @@ time_run 'reindex *' "notmuch reindex '*'"
 time_run 'reindex *' "notmuch reindex '*'"
 time_run 'reindex *' "notmuch reindex '*'"
 
+manifest=$(mktemp manifestXXXXXX)
+
+find mail -type f ! -path 'mail/.notmuch/*' | sed -n '1~4 p' > $manifest
+# arithmetic context is to eat extra whitespace on e.g. some BSDs
+count=$((`wc -l < $manifest`))
+
+perl -nle 'rename $_, "$_.renamed"' $manifest
+
+time_run "reindex ($count mv)" "notmuch reindex '*'"
+
+perl -nle 'rename "$_.renamed", $_' $manifest
+
+time_run "reindex ($count mv back)" "notmuch reindex '*'"
+
+xargs tar cf backup.tar < $manifest
+
+perl -nle 'unlink $_; unlink $_.copy' $manifest
+
+time_run "reindex ($count rm)" "notmuch reindex '*'"
+
+tar xf backup.tar
+
+time_run "reindex ($count restore)" "notmuch reindex '*'"
+
+perl -nle 'link $_, "$_.copy"' $manifest
+
+time_run "reindex ($count cp)" "notmuch reindex '*'"
+
 time_done
-- 
2.30.2

Re: [PATCH] perf-test: add test for reindexing after removal/renaming msgs

[David Bremner]

David Bremner <david@tethera.net> writes:

> At this point it is a bit tricky to measure the performance increase
> from the new message deletion code, since the same commit (9ad19e4)
> that improved the performance also seems to have fixed a bug with an
> uncaught Xapian exception triggered by this test.

Actually 9ad19e4 just delays the bug in question to the medium corpus
size. I'll have to see if I can trigger it without the
performance-corpus.

d

[PATCH] test: add regression test for large number of deletions

[David Bremner]

Prior to 9ad19e4454f there was an unhandled Xapian exception when
reindexing after a large number of deletes.  This test was used for
bisection, and will subsequently serve as a regression test.
---

It turns out my original assessment of the situation was
correct. Whatever the problem was, the delete performance optimization
changes fixed it.

 test/T700-reindex.sh | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/test/T700-reindex.sh b/test/T700-reindex.sh
index f51130e8..3b0f893e 100755
--- a/test/T700-reindex.sh
+++ b/test/T700-reindex.sh
@@ -83,4 +83,11 @@ notmuch reindex '*'
 notmuch search '*' | notmuch_search_sanitize > OUTPUT
 test_expect_equal_file EXPECTED OUTPUT
 
+
+test_begin_subtest "reindex after removing corpus"
+tar cf backup.tar mail/cur
+find mail/cur -type f -delete
+test_expect_success "notmuch reindex '*'"
+tar xf backup.tar
+
 test_done
-- 
2.30.2

Re: [PATCH] perf-test: add test for reindexing after removal/renaming msgs

[David Bremner]

David Bremner <david@tethera.net> writes:

> At this point it is a bit tricky to measure the performance increase
> from the new message deletion code, since the same commit (9ad19e4)
> that improved the performance also seems to have fixed a bug with an
> uncaught Xapian exception triggered by this test.

series applied to master, with the s/xargs tar xf/xargs tar uf/
adjustment previously applied for performance-test/T00-new.sh

d