* [PATCH] perf-test: add test for reindexing after removal/renaming msgs
@ 2021-04-24 17:46 David Bremner
2021-04-24 18:59 ` David Bremner
2021-05-23 12:02 ` [PATCH] perf-test: add test for reindexing after removal/renaming msgs David Bremner
0 siblings, 2 replies; 4+ messages in thread
From: David Bremner @ 2021-04-24 17:46 UTC (permalink / raw)
To: notmuch; +Cc: David Bremner
At this point it is a bit tricky to measure the performance increase
from the new message deletion code, since the same commit (9ad19e4)
that improved the performance also seems to have fixed a bug with an
uncaught Xapian exception triggered by this test.
---
performance-test/T03-reindex.sh | 28 ++++++++++++++++++++++++++++
1 file changed, 28 insertions(+)
diff --git a/performance-test/T03-reindex.sh b/performance-test/T03-reindex.sh
index 8db52a33..afa11df8 100755
--- a/performance-test/T03-reindex.sh
+++ b/performance-test/T03-reindex.sh
@@ -10,4 +10,32 @@ time_run 'reindex *' "notmuch reindex '*'"
time_run 'reindex *' "notmuch reindex '*'"
time_run 'reindex *' "notmuch reindex '*'"
+manifest=$(mktemp manifestXXXXXX)
+
+find mail -type f ! -path 'mail/.notmuch/*' | sed -n '1~4 p' > $manifest
+# arithmetic context is to eat extra whitespace on e.g. some BSDs
+count=$((`wc -l < $manifest`))
+
+perl -nle 'rename $_, "$_.renamed"' $manifest
+
+time_run "reindex ($count mv)" "notmuch reindex '*'"
+
+perl -nle 'rename "$_.renamed", $_' $manifest
+
+time_run "reindex ($count mv back)" "notmuch reindex '*'"
+
+xargs tar cf backup.tar < $manifest
+
+perl -nle 'unlink $_; unlink $_.copy' $manifest
+
+time_run "reindex ($count rm)" "notmuch reindex '*'"
+
+tar xf backup.tar
+
+time_run "reindex ($count restore)" "notmuch reindex '*'"
+
+perl -nle 'link $_, "$_.copy"' $manifest
+
+time_run "reindex ($count cp)" "notmuch reindex '*'"
+
time_done
--
2.30.2
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [PATCH] perf-test: add test for reindexing after removal/renaming msgs
2021-04-24 17:46 [PATCH] perf-test: add test for reindexing after removal/renaming msgs David Bremner
@ 2021-04-24 18:59 ` David Bremner
2021-04-24 23:02 ` [PATCH] test: add regression test for large number of deletions David Bremner
2021-05-23 12:02 ` [PATCH] perf-test: add test for reindexing after removal/renaming msgs David Bremner
1 sibling, 1 reply; 4+ messages in thread
From: David Bremner @ 2021-04-24 18:59 UTC (permalink / raw)
To: notmuch
David Bremner <david@tethera.net> writes:
> At this point it is a bit tricky to measure the performance increase
> from the new message deletion code, since the same commit (9ad19e4)
> that improved the performance also seems to have fixed a bug with an
> uncaught Xapian exception triggered by this test.
Actually 9ad19e4 just delays the bug in question to the medium corpus
size. I'll have to see if I can trigger it without the
performance-corpus.
d
^ permalink raw reply [flat|nested] 4+ messages in thread
* [PATCH] test: add regression test for large number of deletions
2021-04-24 18:59 ` David Bremner
@ 2021-04-24 23:02 ` David Bremner
0 siblings, 0 replies; 4+ messages in thread
From: David Bremner @ 2021-04-24 23:02 UTC (permalink / raw)
To: David Bremner, notmuch
Prior to 9ad19e4454f there was an unhandled Xapian exception when
reindexing after a large number of deletes. This test was used for
bisection, and will subsequently serve as a regression test.
---
It turns out my original assessment of the situation was
correct. Whatever the problem was, the delete performance optimization
changes fixed it.
test/T700-reindex.sh | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/test/T700-reindex.sh b/test/T700-reindex.sh
index f51130e8..3b0f893e 100755
--- a/test/T700-reindex.sh
+++ b/test/T700-reindex.sh
@@ -83,4 +83,11 @@ notmuch reindex '*'
notmuch search '*' | notmuch_search_sanitize > OUTPUT
test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "reindex after removing corpus"
+tar cf backup.tar mail/cur
+find mail/cur -type f -delete
+test_expect_success "notmuch reindex '*'"
+tar xf backup.tar
+
test_done
--
2.30.2
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [PATCH] perf-test: add test for reindexing after removal/renaming msgs
2021-04-24 17:46 [PATCH] perf-test: add test for reindexing after removal/renaming msgs David Bremner
2021-04-24 18:59 ` David Bremner
@ 2021-05-23 12:02 ` David Bremner
1 sibling, 0 replies; 4+ messages in thread
From: David Bremner @ 2021-05-23 12:02 UTC (permalink / raw)
To: notmuch
David Bremner <david@tethera.net> writes:
> At this point it is a bit tricky to measure the performance increase
> from the new message deletion code, since the same commit (9ad19e4)
> that improved the performance also seems to have fixed a bug with an
> uncaught Xapian exception triggered by this test.
series applied to master, with the s/xargs tar xf/xargs tar uf/
adjustment previously applied for performance-test/T00-new.sh
d
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2021-05-23 12:02 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-04-24 17:46 [PATCH] perf-test: add test for reindexing after removal/renaming msgs David Bremner
2021-04-24 18:59 ` David Bremner
2021-04-24 23:02 ` [PATCH] test: add regression test for large number of deletions David Bremner
2021-05-23 12:02 ` [PATCH] perf-test: add test for reindexing after removal/renaming msgs David Bremner
Code repositories for project(s) associated with this public inbox
https://yhetil.org/notmuch.git/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).