* [Patch v2 01/11] perf-test: propagate non-zero returns from /usr/bin/time
2012-12-06 3:01 Performance test improvements, v2 david
@ 2012-12-06 3:01 ` david
2012-12-06 3:01 ` [Patch v2 02/11] perf-test: add argument parsing for performance tests david
` (10 subsequent siblings)
11 siblings, 0 replies; 15+ messages in thread
From: david @ 2012-12-06 3:01 UTC (permalink / raw)
To: notmuch; +Cc: David Bremner
From: David Bremner <bremner@debian.org>
Unlike in the correctness tests, the most common cause of non-zero
return seems to be the user interrupting, so killing the run seems
like the friendly thing to do.
---
performance-test/perf-test-lib.sh | 2 ++
1 file changed, 2 insertions(+)
diff --git a/performance-test/perf-test-lib.sh b/performance-test/perf-test-lib.sh
index 1399d05..fb15028 100644
--- a/performance-test/perf-test-lib.sh
+++ b/performance-test/perf-test-lib.sh
@@ -51,7 +51,9 @@ time_run () {
if test "$verbose" != "t"; then exec 4>test.output 3>&4; fi
if ! eval >&3 "/usr/bin/time -f '%e\t%U\t%S\t%M\t%I\t%O' $2" ; then
test_failure=$(($test_failure + 1))
+ return 1
fi
+ return 0
}
time_done () {
--
1.7.10.4
^ permalink raw reply related [flat|nested] 15+ messages in thread* [Patch v2 02/11] perf-test: add argument parsing for performance tests
2012-12-06 3:01 Performance test improvements, v2 david
2012-12-06 3:01 ` [Patch v2 01/11] perf-test: propagate non-zero returns from /usr/bin/time david
@ 2012-12-06 3:01 ` david
2012-12-06 3:01 ` [Patch v2 03/11] perf-test: add corpus size to output, compact I/O stats david
` (9 subsequent siblings)
11 siblings, 0 replies; 15+ messages in thread
From: david @ 2012-12-06 3:01 UTC (permalink / raw)
To: notmuch; +Cc: David Bremner
From: David Bremner <bremner@debian.org>
This patch just sets (non-exported) variables. The variable $debug is
already used, and $corpus_size will be used in following commits.
---
performance-test/perf-test-lib.sh | 25 +++++++++++++++++++++++++
1 file changed, 25 insertions(+)
diff --git a/performance-test/perf-test-lib.sh b/performance-test/perf-test-lib.sh
index fb15028..3a4a23d 100644
--- a/performance-test/perf-test-lib.sh
+++ b/performance-test/perf-test-lib.sh
@@ -1,5 +1,30 @@
. ./version.sh
+corpus_size=large
+
+while test "$#" -ne 0
+do
+ case "$1" in
+ -d|--debug)
+ debug=t;
+ shift
+ ;;
+ -s|--small)
+ corpus_size=small;
+ shift
+ ;;
+ -m|--medium)
+ corpus_size=medium;
+ shift
+ ;;
+ -l|--large)
+ corpus_size=large;
+ shift
+ ;;
+ *)
+ echo "error: unknown performance test option '$1'" >&2; exit 1 ;;
+ esac
+done
. ../test/test-lib-common.sh
set -e
--
1.7.10.4
^ permalink raw reply related [flat|nested] 15+ messages in thread* [Patch v2 03/11] perf-test: add corpus size to output, compact I/O stats
2012-12-06 3:01 Performance test improvements, v2 david
2012-12-06 3:01 ` [Patch v2 01/11] perf-test: propagate non-zero returns from /usr/bin/time david
2012-12-06 3:01 ` [Patch v2 02/11] perf-test: add argument parsing for performance tests david
@ 2012-12-06 3:01 ` david
2012-12-06 3:01 ` [Patch v2 04/11] perf-test: optionally print description for each group of tests david
` (8 subsequent siblings)
11 siblings, 0 replies; 15+ messages in thread
From: david @ 2012-12-06 3:01 UTC (permalink / raw)
To: notmuch; +Cc: David Bremner
From: David Bremner <bremner@debian.org>
Austin suggested a while ago that the corpus size be printed in the
header. In the end it seems the corpus will be fixed per test script,
so this suggestion indeed makes sense.
The tabbing was wrapping on my usual 80 column terminal, so I joined
the input and output columns together.
---
performance-test/perf-test-lib.sh | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/performance-test/perf-test-lib.sh b/performance-test/perf-test-lib.sh
index 3a4a23d..e399d3f 100644
--- a/performance-test/perf-test-lib.sh
+++ b/performance-test/perf-test-lib.sh
@@ -67,14 +67,14 @@ add_email_corpus ()
}
print_header () {
- printf "[v%4s] Wall(s)\tUsr(s)\tSys(s)\tRes(K)\tIn(512B)\tOut(512B)\n" \
- ${PERFTEST_VERSION}
+ printf "[v%4s %6s] Wall(s)\tUsr(s)\tSys(s)\tRes(K)\tIn/Out(512B)\n" \
+ ${PERFTEST_VERSION} ${corpus_size}
}
time_run () {
printf "%-22s" "$1"
if test "$verbose" != "t"; then exec 4>test.output 3>&4; fi
- if ! eval >&3 "/usr/bin/time -f '%e\t%U\t%S\t%M\t%I\t%O' $2" ; then
+ if ! eval >&3 "/usr/bin/time -f '%e\t%U\t%S\t%M\t%I/%O' $2" ; then
test_failure=$(($test_failure + 1))
return 1
fi
--
1.7.10.4
^ permalink raw reply related [flat|nested] 15+ messages in thread* [Patch v2 04/11] perf-test: optionally print description for each group of tests
2012-12-06 3:01 Performance test improvements, v2 david
` (2 preceding siblings ...)
2012-12-06 3:01 ` [Patch v2 03/11] perf-test: add corpus size to output, compact I/O stats david
@ 2012-12-06 3:01 ` david
2012-12-11 4:31 ` Austin Clements
2012-12-06 3:01 ` [Patch v2 05/11] perf-test: cache unpacked corpus david
` (7 subsequent siblings)
11 siblings, 1 reply; 15+ messages in thread
From: david @ 2012-12-06 3:01 UTC (permalink / raw)
To: notmuch; +Cc: David Bremner
From: David Bremner <bremner@debian.org>
Output from tests is indented slightly in the same style as the
correctness tests.
---
performance-test/perf-test-lib.sh | 9 +++++++--
1 file changed, 7 insertions(+), 2 deletions(-)
diff --git a/performance-test/perf-test-lib.sh b/performance-test/perf-test-lib.sh
index e399d3f..415bf15 100644
--- a/performance-test/perf-test-lib.sh
+++ b/performance-test/perf-test-lib.sh
@@ -67,12 +67,12 @@ add_email_corpus ()
}
print_header () {
- printf "[v%4s %6s] Wall(s)\tUsr(s)\tSys(s)\tRes(K)\tIn/Out(512B)\n" \
+ printf "[v%4s %6s] Wall(s)\tUsr(s)\tSys(s)\tRes(K)\tIn/Out(512B)\n" \
${PERFTEST_VERSION} ${corpus_size}
}
time_run () {
- printf "%-22s" "$1"
+ printf " %-22s" "$1"
if test "$verbose" != "t"; then exec 4>test.output 3>&4; fi
if ! eval >&3 "/usr/bin/time -f '%e\t%U\t%S\t%M\t%I/%O' $2" ; then
test_failure=$(($test_failure + 1))
@@ -92,3 +92,8 @@ time_done () {
cd -P "$test" || error "Cannot setup test environment"
test_failure=0
+
+if [[ -n "test_description" ]]; then
+ echo
+ echo $(basename "$0"): "Benchmarking ${test_description}"
+fi
--
1.7.10.4
^ permalink raw reply related [flat|nested] 15+ messages in thread* Re: [Patch v2 04/11] perf-test: optionally print description for each group of tests
2012-12-06 3:01 ` [Patch v2 04/11] perf-test: optionally print description for each group of tests david
@ 2012-12-11 4:31 ` Austin Clements
0 siblings, 0 replies; 15+ messages in thread
From: Austin Clements @ 2012-12-11 4:31 UTC (permalink / raw)
To: david, notmuch; +Cc: David Bremner
On Wed, 05 Dec 2012, david@tethera.net wrote:
> From: David Bremner <bremner@debian.org>
>
> Output from tests is indented slightly in the same style as the
> correctness tests.
> ---
> performance-test/perf-test-lib.sh | 9 +++++++--
> 1 file changed, 7 insertions(+), 2 deletions(-)
>
> diff --git a/performance-test/perf-test-lib.sh b/performance-test/perf-test-lib.sh
> index e399d3f..415bf15 100644
> --- a/performance-test/perf-test-lib.sh
> +++ b/performance-test/perf-test-lib.sh
> @@ -67,12 +67,12 @@ add_email_corpus ()
> }
>
> print_header () {
> - printf "[v%4s %6s] Wall(s)\tUsr(s)\tSys(s)\tRes(K)\tIn/Out(512B)\n" \
> + printf "[v%4s %6s] Wall(s)\tUsr(s)\tSys(s)\tRes(K)\tIn/Out(512B)\n" \
> ${PERFTEST_VERSION} ${corpus_size}
> }
>
> time_run () {
> - printf "%-22s" "$1"
> + printf " %-22s" "$1"
> if test "$verbose" != "t"; then exec 4>test.output 3>&4; fi
> if ! eval >&3 "/usr/bin/time -f '%e\t%U\t%S\t%M\t%I/%O' $2" ; then
> test_failure=$(($test_failure + 1))
> @@ -92,3 +92,8 @@ time_done () {
>
> cd -P "$test" || error "Cannot setup test environment"
> test_failure=0
> +
> +if [[ -n "test_description" ]]; then
I think you meant "$test_description". Though, it seems like it would
be confusing if $test_description wasn't set and there was no output.
Maybe this shouldn't be conditional and below should be
${test_description:-UNNAMED} or something?
> + echo
> + echo $(basename "$0"): "Benchmarking ${test_description}"
> +fi
> --
> 1.7.10.4
>
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch
^ permalink raw reply [flat|nested] 15+ messages in thread
* [Patch v2 05/11] perf-test: cache unpacked corpus
2012-12-06 3:01 Performance test improvements, v2 david
` (3 preceding siblings ...)
2012-12-06 3:01 ` [Patch v2 04/11] perf-test: optionally print description for each group of tests david
@ 2012-12-06 3:01 ` david
2012-12-06 3:01 ` [Patch v2 06/11] perf-test: add caching of xapian database david
` (6 subsequent siblings)
11 siblings, 0 replies; 15+ messages in thread
From: david @ 2012-12-06 3:01 UTC (permalink / raw)
To: notmuch; +Cc: David Bremner
From: David Bremner <bremner@debian.org>
Unpacking is not really the expensive step (compared to the initial
notmuch new), but this is a pre-requisite to caching the database.
---
performance-test/.gitignore | 1 +
performance-test/Makefile.local | 2 +-
performance-test/perf-test-lib.sh | 51 ++++++++++++++++++++++++-------------
3 files changed, 36 insertions(+), 18 deletions(-)
diff --git a/performance-test/.gitignore b/performance-test/.gitignore
index 53f2697..796ed01 100644
--- a/performance-test/.gitignore
+++ b/performance-test/.gitignore
@@ -1 +1,2 @@
tmp.*/
+corpus/
diff --git a/performance-test/Makefile.local b/performance-test/Makefile.local
index 5d2acbd..63e4c3d 100644
--- a/performance-test/Makefile.local
+++ b/performance-test/Makefile.local
@@ -29,4 +29,4 @@ $(TXZFILE):
download-corpus:
wget -O ${TXZFILE} ${DEFAULT_URL}
-CLEAN := $(CLEAN) $(dir)/tmp.*
+CLEAN := $(CLEAN) $(dir)/tmp.* $(dir)/corpus
diff --git a/performance-test/perf-test-lib.sh b/performance-test/perf-test-lib.sh
index 415bf15..0c10718 100644
--- a/performance-test/perf-test-lib.sh
+++ b/performance-test/perf-test-lib.sh
@@ -39,33 +39,50 @@ add_email_corpus ()
{
rm -rf ${MAIL_DIR}
- case "$1" in
- --small)
- arg="mail/enron/bailey-s"
+ case "$corpus_size" in
+ small)
+ mail_subdir="mail/enron/bailey-s"
+ check_for="${TEST_DIRECTORY}/corpus/$mail_subdir"
;;
- --medium)
- arg="mail/notmuch-archive"
+ medium)
+ mail_subdir="mail/notmuch-archive"
+ check_for="${TEST_DIRECTORY}/corpus/$mail_subdir"
;;
*)
- arg=mail
+ mail_subdir=mail
+ check_for="${TEST_DIRECTORY}/corpus/$mail_subdir/enron/wolfe-j"
esac
- if command -v pixz > /dev/null; then
- XZ=pixz
- else
- XZ=xz
+ MAIL_CORPUS="${TEST_DIRECTORY}/corpus/$mail_subdir"
+ args=()
+ if [ ! -d "$check_for" ] ; then
+ args+=("notmuch-email-corpus/$mail_subdir")
fi
- printf "Unpacking corpus\n"
- tar --checkpoint=.5000 --extract --strip-components=1 \
- --directory ${TMP_DIRECTORY} \
- --use-compress-program ${XZ} \
- --file ../download/notmuch-email-corpus-${PERFTEST_VERSION}.tar.xz \
- notmuch-email-corpus/"$arg"
+ if [[ ${#args[@]} > 0 ]]; then
+ if command -v pixz > /dev/null; then
+ XZ=pixz
+ else
+ XZ=xz
+ fi
+
+ printf "Unpacking corpus\n"
+ mkdir -p "${TEST_DIRECTORY}/corpus"
+
+ tar --checkpoint=.5000 --extract --strip-components=1 \
+ --directory ${TEST_DIRECTORY}/corpus \
+ --use-compress-program ${XZ} \
+ --file ../download/notmuch-email-corpus-${PERFTEST_VERSION}.tar.xz \
+ "${args[@]}"
+
+ printf "\n"
- printf "\n"
+ fi
+
+ cp -lr $MAIL_CORPUS $MAIL_DIR
}
+
print_header () {
printf "[v%4s %6s] Wall(s)\tUsr(s)\tSys(s)\tRes(K)\tIn/Out(512B)\n" \
${PERFTEST_VERSION} ${corpus_size}
--
1.7.10.4
^ permalink raw reply related [flat|nested] 15+ messages in thread* [Patch v2 06/11] perf-test: add caching of xapian database
2012-12-06 3:01 Performance test improvements, v2 david
` (4 preceding siblings ...)
2012-12-06 3:01 ` [Patch v2 05/11] perf-test: cache unpacked corpus david
@ 2012-12-06 3:01 ` david
2012-12-06 3:01 ` [Patch v2 07/11] perf-test: update README david
` (5 subsequent siblings)
11 siblings, 0 replies; 15+ messages in thread
From: david @ 2012-12-06 3:01 UTC (permalink / raw)
To: notmuch; +Cc: David Bremner
From: David Bremner <bremner@debian.org>
The caching and uncaching seem to be necessarily manual, as timing the
initial notmuch new is one of our goals with this suite.
---
performance-test/.gitignore | 1 +
performance-test/Makefile.local | 2 +-
performance-test/basic | 5 +++++
performance-test/perf-test-lib.sh | 17 +++++++++++++++++
4 files changed, 24 insertions(+), 1 deletion(-)
diff --git a/performance-test/.gitignore b/performance-test/.gitignore
index 796ed01..6421a9a 100644
--- a/performance-test/.gitignore
+++ b/performance-test/.gitignore
@@ -1,2 +1,3 @@
tmp.*/
corpus/
+notmuch.cache.*/
diff --git a/performance-test/Makefile.local b/performance-test/Makefile.local
index 63e4c3d..3834e4d 100644
--- a/performance-test/Makefile.local
+++ b/performance-test/Makefile.local
@@ -29,4 +29,4 @@ $(TXZFILE):
download-corpus:
wget -O ${TXZFILE} ${DEFAULT_URL}
-CLEAN := $(CLEAN) $(dir)/tmp.* $(dir)/corpus
+CLEAN := $(CLEAN) $(dir)/tmp.* $(dir)/corpus $(dir)/notmuch.cache.*
diff --git a/performance-test/basic b/performance-test/basic
index 9d015ee..41a7ff1 100755
--- a/performance-test/basic
+++ b/performance-test/basic
@@ -2,11 +2,16 @@
. ./perf-test-lib.sh
+uncache_database
+
add_email_corpus
print_header
time_run 'initial notmuch new' 'notmuch new'
+
+cache_database
+
time_run 'second notmuch new' 'notmuch new'
time_run 'dump *' 'notmuch dump > tags.out'
time_run 'restore *' 'notmuch restore < tags.out'
diff --git a/performance-test/perf-test-lib.sh b/performance-test/perf-test-lib.sh
index 0c10718..0f9796d 100644
--- a/performance-test/perf-test-lib.sh
+++ b/performance-test/perf-test-lib.sh
@@ -35,6 +35,8 @@ then
exit 1
fi
+DB_CACHE_DIR=${TEST_DIRECTORY}/notmuch.cache.$corpus_size
+
add_email_corpus ()
{
rm -rf ${MAIL_DIR}
@@ -80,8 +82,23 @@ add_email_corpus ()
fi
cp -lr $MAIL_CORPUS $MAIL_DIR
+
+ if [ -d $DB_CACHE_DIR ]; then
+ cp -r $DB_CACHE_DIR ${MAIL_DIR}/.notmuch
+ fi
+}
+
+cache_database () {
+ if [ -d $MAIL_DIR/.notmuch ]; then
+ cp -r $MAIL_DIR/.notmuch $DB_CACHE_DIR
+ else
+ echo "Warning: No database found to cache"
+ fi
}
+uncache_database () {
+ rm -rf $DB_CACHE_DIR
+}
print_header () {
printf "[v%4s %6s] Wall(s)\tUsr(s)\tSys(s)\tRes(K)\tIn/Out(512B)\n" \
--
1.7.10.4
^ permalink raw reply related [flat|nested] 15+ messages in thread* [Patch v2 07/11] perf-test: update README
2012-12-06 3:01 Performance test improvements, v2 david
` (5 preceding siblings ...)
2012-12-06 3:01 ` [Patch v2 06/11] perf-test: add caching of xapian database david
@ 2012-12-06 3:01 ` david
2012-12-06 3:01 ` [Patch v2 08/11] perf-test: bump corpus version to 0.3 david
` (4 subsequent siblings)
11 siblings, 0 replies; 15+ messages in thread
From: david @ 2012-12-06 3:01 UTC (permalink / raw)
To: notmuch; +Cc: David Bremner
From: David Bremner <bremner@debian.org>
Describe new argument parsing and mention cache handling routines.
---
performance-test/README | 22 +++++++++++++---------
1 file changed, 13 insertions(+), 9 deletions(-)
diff --git a/performance-test/README b/performance-test/README
index 36f1dac..fd5c977 100644
--- a/performance-test/README
+++ b/performance-test/README
@@ -42,17 +42,21 @@ available performance tests.
Alternately, you can run a specific subset of tests by simply invoking
one of the executable scripts in this directory, (such as ./basic).
+Each test script supports the following arguments
+
+--small / --medium / --large Choose corpus size.
+--debug Enable debugging. In particular don't delete
+ temporary directories.
Writing tests
-------------
-Have a look at "basic" for an example.
-
-add_email_corpus takes arguments "--small" and "--medium" for when you
-want smaller subsets of the corpus to check.
-
-time_done does the cleanup; comment it out or define "$debug" to leave
-the temporary files around.
+Have a look at "basic" for an example. Sourcing "perf-test-lib.sh" is
+mandatory. Utility functions include
-Currently there is no option processing (e.g. --debug) in the
-performance tests.
+- 'add_email_corpus' unpacks a set of messages and adds them to the database.
+- 'cache_database': makes a snapshot of the current database
+- 'uncache_database': forces the next 'add_email_corpus' to rebuild the
+ database.
+- 'time_done' does the cleanup; comment it out or pass --debug to the
+ script to leave the temporary files around.
--
1.7.10.4
^ permalink raw reply related [flat|nested] 15+ messages in thread* [Patch v2 08/11] perf-test: bump corpus version to 0.3
2012-12-06 3:01 Performance test improvements, v2 david
` (6 preceding siblings ...)
2012-12-06 3:01 ` [Patch v2 07/11] perf-test: update README david
@ 2012-12-06 3:01 ` david
2012-12-06 3:01 ` [Patch v2 09/11] perf-test: unpack tags david
` (3 subsequent siblings)
11 siblings, 0 replies; 15+ messages in thread
From: david @ 2012-12-06 3:01 UTC (permalink / raw)
To: notmuch; +Cc: David Bremner
From: David Bremner <bremner@debian.org>
The new version ships with some tags, and an updated archive of the
notmuch mailing list.
---
performance-test/download/notmuch-email-corpus-0.2.tar.xz.asc | 9 ---------
performance-test/download/notmuch-email-corpus-0.3.tar.xz.asc | 9 +++++++++
performance-test/version.sh | 2 +-
3 files changed, 10 insertions(+), 10 deletions(-)
delete mode 100644 performance-test/download/notmuch-email-corpus-0.2.tar.xz.asc
create mode 100644 performance-test/download/notmuch-email-corpus-0.3.tar.xz.asc
diff --git a/performance-test/download/notmuch-email-corpus-0.2.tar.xz.asc b/performance-test/download/notmuch-email-corpus-0.2.tar.xz.asc
deleted file mode 100644
index c8b4b3d..0000000
--- a/performance-test/download/notmuch-email-corpus-0.2.tar.xz.asc
+++ /dev/null
@@ -1,9 +0,0 @@
------BEGIN PGP SIGNATURE-----
-Version: GnuPG v1.4.12 (GNU/Linux)
-
-iJwEAAECAAYFAlCsvx0ACgkQTiiN/0Um85kZAwP9GgOQ22jK8mr5X4pT/mB8EjSH
-QbndlxxbRrP0ChTqjBQoD3IsTHjNL7W572BfXb/MNo94R/iIQ7yTHCDVNuwBhvKd
-7qgIuW2FUS1uTfJRP5KBNf8JPuin+6wqGe8/+y/iOs+XJSdiYg1ElS49Ntnpg0yl
-btImgEcxTxQ2qfzDS1g=
-=iuZR
------END PGP SIGNATURE-----
diff --git a/performance-test/download/notmuch-email-corpus-0.3.tar.xz.asc b/performance-test/download/notmuch-email-corpus-0.3.tar.xz.asc
new file mode 100644
index 0000000..f109e81
--- /dev/null
+++ b/performance-test/download/notmuch-email-corpus-0.3.tar.xz.asc
@@ -0,0 +1,9 @@
+-----BEGIN PGP SIGNATURE-----
+Version: GnuPG v1.4.12 (GNU/Linux)
+
+iJwEAAECAAYFAlC9a90ACgkQTiiN/0Um85nAMAP+LCWdKzolcl/KW+JcCd0Dk+9v
+0vvtBVEhBes0TbK6iWrxCV2OIuYG/RhnFlJTZ4MjgaTRxzDubpC+JktaJdLmIQUN
+B7ZIDMjFduCwmtyLiuu/00CjxJKUXm7vx+ULGpvp0uxFE/vaqGP997BHwBjjfBVm
+YX6BlLX1SV6TfENkuRE=
+=Mks5
+-----END PGP SIGNATURE-----
diff --git a/performance-test/version.sh b/performance-test/version.sh
index d9270b1..afafc73 100644
--- a/performance-test/version.sh
+++ b/performance-test/version.sh
@@ -1,3 +1,3 @@
# this should be both a valid Makefile fragment and valid POSIX(ish) shell.
-PERFTEST_VERSION=0.2
+PERFTEST_VERSION=0.3
--
1.7.10.4
^ permalink raw reply related [flat|nested] 15+ messages in thread* [Patch v2 09/11] perf-test: unpack tags.
2012-12-06 3:01 Performance test improvements, v2 david
` (7 preceding siblings ...)
2012-12-06 3:01 ` [Patch v2 08/11] perf-test: bump corpus version to 0.3 david
@ 2012-12-06 3:01 ` david
2012-12-06 3:01 ` [Patch v2 10/11] perf-test: split basic into 00-new and 01-dump-restore david
` (2 subsequent siblings)
11 siblings, 0 replies; 15+ messages in thread
From: david @ 2012-12-06 3:01 UTC (permalink / raw)
To: notmuch; +Cc: David Bremner
From: David Bremner <bremner@debian.org>
There is only one set of tags, independant of the size of message
corpus chosen.
---
performance-test/perf-test-lib.sh | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/performance-test/perf-test-lib.sh b/performance-test/perf-test-lib.sh
index 0f9796d..ca08026 100644
--- a/performance-test/perf-test-lib.sh
+++ b/performance-test/perf-test-lib.sh
@@ -56,7 +56,13 @@ add_email_corpus ()
esac
MAIL_CORPUS="${TEST_DIRECTORY}/corpus/$mail_subdir"
+ TAG_CORPUS="${TEST_DIRECTORY}/corpus/tags"
+
args=()
+ if [ ! -d "$TAG_CORPUS" ] ; then
+ args+=("notmuch-email-corpus/tags")
+ fi
+
if [ ! -d "$check_for" ] ; then
args+=("notmuch-email-corpus/$mail_subdir")
fi
@@ -81,6 +87,7 @@ add_email_corpus ()
fi
+ cp -lr $TAG_CORPUS $TMP_DIRECTORY/corpus.tags
cp -lr $MAIL_CORPUS $MAIL_DIR
if [ -d $DB_CACHE_DIR ]; then
--
1.7.10.4
^ permalink raw reply related [flat|nested] 15+ messages in thread* [Patch v2 10/11] perf-test: split basic into 00-new and 01-dump-restore
2012-12-06 3:01 Performance test improvements, v2 david
` (8 preceding siblings ...)
2012-12-06 3:01 ` [Patch v2 09/11] perf-test: unpack tags david
@ 2012-12-06 3:01 ` david
2012-12-11 4:50 ` Austin Clements
2012-12-06 3:01 ` [Patch v2 11/11] perf-test: use nmbug tags in dump-restore tests david
2012-12-11 4:52 ` Performance test improvements, v2 Austin Clements
11 siblings, 1 reply; 15+ messages in thread
From: david @ 2012-12-06 3:01 UTC (permalink / raw)
To: notmuch; +Cc: David Bremner
From: David Bremner <bremner@debian.org>
We rely on subsequent "notmuch new" invocations being relatively fast
rather than trying to detect if notmuch-new needs to be run.
---
performance-test/00-new | 21 +++++++++++++++++++++
performance-test/01-dump-restore | 15 +++++++++++++++
performance-test/README | 8 ++++++--
performance-test/basic | 20 --------------------
performance-test/notmuch-perf-test | 3 ++-
5 files changed, 44 insertions(+), 23 deletions(-)
create mode 100755 performance-test/00-new
create mode 100755 performance-test/01-dump-restore
delete mode 100755 performance-test/basic
diff --git a/performance-test/00-new b/performance-test/00-new
new file mode 100755
index 0000000..02db874
--- /dev/null
+++ b/performance-test/00-new
@@ -0,0 +1,21 @@
+#!/bin/bash
+
+test_description='notmuch new'
+
+. ./perf-test-lib.sh
+
+uncache_database
+
+add_email_corpus
+
+print_header
+
+time_run 'initial notmuch new' 'notmuch new'
+
+cache_database
+
+for i in $(seq 2 6); do
+ time_run "notmuch new #$i" 'notmuch new'
+done
+
+time_done
diff --git a/performance-test/01-dump-restore b/performance-test/01-dump-restore
new file mode 100755
index 0000000..09444e3
--- /dev/null
+++ b/performance-test/01-dump-restore
@@ -0,0 +1,15 @@
+#!/bin/bash
+
+test_description='Dump and restore'
+
+. ./perf-test-lib.sh
+
+add_email_corpus
+
+print_header
+
+time_run 'notmuch new' 'notmuch new'
+time_run 'dump *' 'notmuch dump > tags.out'
+time_run 'restore *' 'notmuch restore < tags.out'
+
+time_done
diff --git a/performance-test/README b/performance-test/README
index fd5c977..35489a9 100644
--- a/performance-test/README
+++ b/performance-test/README
@@ -51,8 +51,8 @@ Each test script supports the following arguments
Writing tests
-------------
-Have a look at "basic" for an example. Sourcing "perf-test-lib.sh" is
-mandatory. Utility functions include
+Have a look at "01-dump-restore" for an example. Sourcing
+"perf-test-lib.sh" is mandatory. Utility functions include
- 'add_email_corpus' unpacks a set of messages and adds them to the database.
- 'cache_database': makes a snapshot of the current database
@@ -60,3 +60,7 @@ mandatory. Utility functions include
database.
- 'time_done' does the cleanup; comment it out or pass --debug to the
script to leave the temporary files around.
+
+Scripts are run in the order specified in notmuch-perf-test. In the
+future this order might be chosen automatically so please follow the
+convention of starting the name with two digits to specify the order.
diff --git a/performance-test/basic b/performance-test/basic
deleted file mode 100755
index 41a7ff1..0000000
--- a/performance-test/basic
+++ /dev/null
@@ -1,20 +0,0 @@
-#!/bin/bash
-
-. ./perf-test-lib.sh
-
-uncache_database
-
-add_email_corpus
-
-print_header
-
-time_run 'initial notmuch new' 'notmuch new'
-
-cache_database
-
-time_run 'second notmuch new' 'notmuch new'
-time_run 'dump *' 'notmuch dump > tags.out'
-time_run 'restore *' 'notmuch restore < tags.out'
-time_run 'tag * +new_tag' "notmuch tag +new_tag '*'"
-
-time_done
diff --git a/performance-test/notmuch-perf-test b/performance-test/notmuch-perf-test
index 1bea345..f93d8a4 100755
--- a/performance-test/notmuch-perf-test
+++ b/performance-test/notmuch-perf-test
@@ -17,7 +17,8 @@ fi
cd $(dirname "$0")
TESTS="
- basic
+ 00-new
+ 01-dump-restore
"
for test in $TESTS; do
--
1.7.10.4
^ permalink raw reply related [flat|nested] 15+ messages in thread* Re: [Patch v2 10/11] perf-test: split basic into 00-new and 01-dump-restore
2012-12-06 3:01 ` [Patch v2 10/11] perf-test: split basic into 00-new and 01-dump-restore david
@ 2012-12-11 4:50 ` Austin Clements
0 siblings, 0 replies; 15+ messages in thread
From: Austin Clements @ 2012-12-11 4:50 UTC (permalink / raw)
To: david, notmuch; +Cc: David Bremner
On Wed, 05 Dec 2012, david@tethera.net wrote:
> From: David Bremner <bremner@debian.org>
>
> We rely on subsequent "notmuch new" invocations being relatively fast
> rather than trying to detect if notmuch-new needs to be run.
> ---
> performance-test/00-new | 21 +++++++++++++++++++++
> performance-test/01-dump-restore | 15 +++++++++++++++
> performance-test/README | 8 ++++++--
> performance-test/basic | 20 --------------------
> performance-test/notmuch-perf-test | 3 ++-
> 5 files changed, 44 insertions(+), 23 deletions(-)
> create mode 100755 performance-test/00-new
> create mode 100755 performance-test/01-dump-restore
> delete mode 100755 performance-test/basic
>
> diff --git a/performance-test/00-new b/performance-test/00-new
> new file mode 100755
> index 0000000..02db874
> --- /dev/null
> +++ b/performance-test/00-new
> @@ -0,0 +1,21 @@
> +#!/bin/bash
> +
> +test_description='notmuch new'
> +
> +. ./perf-test-lib.sh
> +
> +uncache_database
> +
> +add_email_corpus
> +
> +print_header
> +
> +time_run 'initial notmuch new' 'notmuch new'
> +
> +cache_database
> +
> +for i in $(seq 2 6); do
> + time_run "notmuch new #$i" 'notmuch new'
> +done
> +
> +time_done
> diff --git a/performance-test/01-dump-restore b/performance-test/01-dump-restore
> new file mode 100755
> index 0000000..09444e3
> --- /dev/null
> +++ b/performance-test/01-dump-restore
> @@ -0,0 +1,15 @@
> +#!/bin/bash
> +
> +test_description='Dump and restore'
> +
> +. ./perf-test-lib.sh
> +
> +add_email_corpus
> +
> +print_header
> +
> +time_run 'notmuch new' 'notmuch new'
> +time_run 'dump *' 'notmuch dump > tags.out'
> +time_run 'restore *' 'notmuch restore < tags.out'
> +
> +time_done
> diff --git a/performance-test/README b/performance-test/README
> index fd5c977..35489a9 100644
> --- a/performance-test/README
> +++ b/performance-test/README
> @@ -51,8 +51,8 @@ Each test script supports the following arguments
> Writing tests
> -------------
>
> -Have a look at "basic" for an example. Sourcing "perf-test-lib.sh" is
> -mandatory. Utility functions include
> +Have a look at "01-dump-restore" for an example. Sourcing
> +"perf-test-lib.sh" is mandatory. Utility functions include
>
> - 'add_email_corpus' unpacks a set of messages and adds them to the database.
> - 'cache_database': makes a snapshot of the current database
> @@ -60,3 +60,7 @@ mandatory. Utility functions include
> database.
> - 'time_done' does the cleanup; comment it out or pass --debug to the
> script to leave the temporary files around.
> +
> +Scripts are run in the order specified in notmuch-perf-test. In the
> +future this order might be chosen automatically so please follow the
> +convention of starting the name with two digits to specify the order.
> diff --git a/performance-test/basic b/performance-test/basic
> deleted file mode 100755
> index 41a7ff1..0000000
> --- a/performance-test/basic
> +++ /dev/null
> @@ -1,20 +0,0 @@
> -#!/bin/bash
> -
> -. ./perf-test-lib.sh
> -
> -uncache_database
> -
> -add_email_corpus
> -
> -print_header
> -
> -time_run 'initial notmuch new' 'notmuch new'
> -
> -cache_database
> -
> -time_run 'second notmuch new' 'notmuch new'
> -time_run 'dump *' 'notmuch dump > tags.out'
> -time_run 'restore *' 'notmuch restore < tags.out'
> -time_run 'tag * +new_tag' "notmuch tag +new_tag '*'"
Did you mean to lose the tag-everything test?
> -
> -time_done
> diff --git a/performance-test/notmuch-perf-test b/performance-test/notmuch-perf-test
> index 1bea345..f93d8a4 100755
> --- a/performance-test/notmuch-perf-test
> +++ b/performance-test/notmuch-perf-test
> @@ -17,7 +17,8 @@ fi
> cd $(dirname "$0")
>
> TESTS="
> - basic
> + 00-new
> + 01-dump-restore
> "
>
> for test in $TESTS; do
> --
> 1.7.10.4
>
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch
^ permalink raw reply [flat|nested] 15+ messages in thread
* [Patch v2 11/11] perf-test: use nmbug tags in dump-restore tests
2012-12-06 3:01 Performance test improvements, v2 david
` (9 preceding siblings ...)
2012-12-06 3:01 ` [Patch v2 10/11] perf-test: split basic into 00-new and 01-dump-restore david
@ 2012-12-06 3:01 ` david
2012-12-11 4:52 ` Performance test improvements, v2 Austin Clements
11 siblings, 0 replies; 15+ messages in thread
From: david @ 2012-12-06 3:01 UTC (permalink / raw)
To: notmuch; +Cc: David Bremner
From: David Bremner <bremner@debian.org>
This makes the tag set a bit less trivial.
Note that if you use the small corpus, this is not so interesting (and
is also a bit noisy) since the messages will not be found. In the
future this could be checked for.
---
performance-test/01-dump-restore | 1 +
1 file changed, 1 insertion(+)
diff --git a/performance-test/01-dump-restore b/performance-test/01-dump-restore
index 09444e3..7bcf821 100755
--- a/performance-test/01-dump-restore
+++ b/performance-test/01-dump-restore
@@ -9,6 +9,7 @@ add_email_corpus
print_header
time_run 'notmuch new' 'notmuch new'
+time_run 'load nmbug tags' 'notmuch restore --accumulate < corpus.tags/nmbug.sup-dump'
time_run 'dump *' 'notmuch dump > tags.out'
time_run 'restore *' 'notmuch restore < tags.out'
--
1.7.10.4
^ permalink raw reply related [flat|nested] 15+ messages in thread* Re: Performance test improvements, v2
2012-12-06 3:01 Performance test improvements, v2 david
` (10 preceding siblings ...)
2012-12-06 3:01 ` [Patch v2 11/11] perf-test: use nmbug tags in dump-restore tests david
@ 2012-12-11 4:52 ` Austin Clements
11 siblings, 0 replies; 15+ messages in thread
From: Austin Clements @ 2012-12-11 4:52 UTC (permalink / raw)
To: david, notmuch
This series LGTM except the few nits I pointed out.
On Wed, 05 Dec 2012, david@tethera.net wrote:
> This obsoletes the several tests and loose patches posted in thread
>
> id:1354583824-10520-1-git-send-email-david@tethera.net
>
> The main changes introduced here are caching, and some non-trivial
> tags are used in the dump restore tests.
>
> Caching now makes it sane to split the tests into several files, and
> the output is tidied up a bit for the multi-file case.
>
>
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch
^ permalink raw reply [flat|nested] 15+ messages in thread