unofficial mirror of notmuch@notmuchmail.org
 help / color / mirror / code / Atom feed
* [PATCH] perf-test: add test for reindexing after removal/renaming msgs
@ 2021-04-24 17:46 David Bremner
  2021-04-24 18:59 ` David Bremner
  2021-05-23 12:02 ` [PATCH] perf-test: add test for reindexing after removal/renaming msgs David Bremner
  0 siblings, 2 replies; 4+ messages in thread
From: David Bremner @ 2021-04-24 17:46 UTC (permalink / raw)
  To: notmuch; +Cc: David Bremner

At this point it is a bit tricky to measure the performance increase
from the new message deletion code, since the same commit (9ad19e4)
that improved the performance also seems to have fixed a bug with an
uncaught Xapian exception triggered by this test.
---
 performance-test/T03-reindex.sh | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)

diff --git a/performance-test/T03-reindex.sh b/performance-test/T03-reindex.sh
index 8db52a33..afa11df8 100755
--- a/performance-test/T03-reindex.sh
+++ b/performance-test/T03-reindex.sh
@@ -10,4 +10,32 @@ time_run 'reindex *' "notmuch reindex '*'"
 time_run 'reindex *' "notmuch reindex '*'"
 time_run 'reindex *' "notmuch reindex '*'"
 
+manifest=$(mktemp manifestXXXXXX)
+
+find mail -type f ! -path 'mail/.notmuch/*' | sed -n '1~4 p' > $manifest
+# arithmetic context is to eat extra whitespace on e.g. some BSDs
+count=$((`wc -l < $manifest`))
+
+perl -nle 'rename $_, "$_.renamed"' $manifest
+
+time_run "reindex ($count mv)" "notmuch reindex '*'"
+
+perl -nle 'rename "$_.renamed", $_' $manifest
+
+time_run "reindex ($count mv back)" "notmuch reindex '*'"
+
+xargs tar cf backup.tar < $manifest
+
+perl -nle 'unlink $_; unlink $_.copy' $manifest
+
+time_run "reindex ($count rm)" "notmuch reindex '*'"
+
+tar xf backup.tar
+
+time_run "reindex ($count restore)" "notmuch reindex '*'"
+
+perl -nle 'link $_, "$_.copy"' $manifest
+
+time_run "reindex ($count cp)" "notmuch reindex '*'"
+
 time_done
-- 
2.30.2

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [PATCH] perf-test: add test for reindexing after removal/renaming msgs
  2021-04-24 17:46 [PATCH] perf-test: add test for reindexing after removal/renaming msgs David Bremner
@ 2021-04-24 18:59 ` David Bremner
  2021-04-24 23:02   ` [PATCH] test: add regression test for large number of deletions David Bremner
  2021-05-23 12:02 ` [PATCH] perf-test: add test for reindexing after removal/renaming msgs David Bremner
  1 sibling, 1 reply; 4+ messages in thread
From: David Bremner @ 2021-04-24 18:59 UTC (permalink / raw)
  To: notmuch

David Bremner <david@tethera.net> writes:

> At this point it is a bit tricky to measure the performance increase
> from the new message deletion code, since the same commit (9ad19e4)
> that improved the performance also seems to have fixed a bug with an
> uncaught Xapian exception triggered by this test.

Actually 9ad19e4 just delays the bug in question to the medium corpus
size. I'll have to see if I can trigger it without the
performance-corpus.

d

^ permalink raw reply	[flat|nested] 4+ messages in thread

* [PATCH] test: add regression test for large number of deletions
  2021-04-24 18:59 ` David Bremner
@ 2021-04-24 23:02   ` David Bremner
  0 siblings, 0 replies; 4+ messages in thread
From: David Bremner @ 2021-04-24 23:02 UTC (permalink / raw)
  To: David Bremner, notmuch

Prior to 9ad19e4454f there was an unhandled Xapian exception when
reindexing after a large number of deletes.  This test was used for
bisection, and will subsequently serve as a regression test.
---

It turns out my original assessment of the situation was
correct. Whatever the problem was, the delete performance optimization
changes fixed it.

 test/T700-reindex.sh | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/test/T700-reindex.sh b/test/T700-reindex.sh
index f51130e8..3b0f893e 100755
--- a/test/T700-reindex.sh
+++ b/test/T700-reindex.sh
@@ -83,4 +83,11 @@ notmuch reindex '*'
 notmuch search '*' | notmuch_search_sanitize > OUTPUT
 test_expect_equal_file EXPECTED OUTPUT
 
+
+test_begin_subtest "reindex after removing corpus"
+tar cf backup.tar mail/cur
+find mail/cur -type f -delete
+test_expect_success "notmuch reindex '*'"
+tar xf backup.tar
+
 test_done
-- 
2.30.2

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [PATCH] perf-test: add test for reindexing after removal/renaming msgs
  2021-04-24 17:46 [PATCH] perf-test: add test for reindexing after removal/renaming msgs David Bremner
  2021-04-24 18:59 ` David Bremner
@ 2021-05-23 12:02 ` David Bremner
  1 sibling, 0 replies; 4+ messages in thread
From: David Bremner @ 2021-05-23 12:02 UTC (permalink / raw)
  To: notmuch

David Bremner <david@tethera.net> writes:

> At this point it is a bit tricky to measure the performance increase
> from the new message deletion code, since the same commit (9ad19e4)
> that improved the performance also seems to have fixed a bug with an
> uncaught Xapian exception triggered by this test.

series applied to master, with the s/xargs tar xf/xargs tar uf/
adjustment previously applied for performance-test/T00-new.sh

d

^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2021-05-23 12:02 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-04-24 17:46 [PATCH] perf-test: add test for reindexing after removal/renaming msgs David Bremner
2021-04-24 18:59 ` David Bremner
2021-04-24 23:02   ` [PATCH] test: add regression test for large number of deletions David Bremner
2021-05-23 12:02 ` [PATCH] perf-test: add test for reindexing after removal/renaming msgs David Bremner

unofficial mirror of notmuch@notmuchmail.org

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://yhetil.org/notmuch/0 notmuch/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 notmuch notmuch/ https://yhetil.org/notmuch \
		notmuch@notmuchmail.org
	public-inbox-index notmuch

Example config snippet for mirrors.
Newsgroups are available over NNTP:
	nntp://news.yhetil.org/yhetil.mail.notmuch.general
	nntp://news.gmane.io/gmane.mail.notmuch.general


code repositories for project(s) associated with this inbox:

	notmuch.git.git (no URL configured)

AGPL code for this site: git clone http://ou63pmih66umazou.onion/public-inbox.git