Performance test improvements, v2

Performance test improvements, v2

[david]

This obsoletes the several tests and loose patches posted in thread

     id:1354583824-10520-1-git-send-email-david@tethera.net

The main changes introduced here are caching, and some non-trivial
tags are used in the dump restore tests.

Caching now makes it sane to split the tests into several files, and
the output is tidied up a bit for the multi-file case.

[Patch v2 01/11] perf-test: propagate non-zero returns from /usr/bin/time

[david]

From: David Bremner <bremner@debian.org>

Unlike in the correctness tests, the most common cause of non-zero
return seems to be the user interrupting, so killing the run seems
like the friendly thing to do.
---
 performance-test/perf-test-lib.sh |    2 ++
 1 file changed, 2 insertions(+)

diff --git a/performance-test/perf-test-lib.sh b/performance-test/perf-test-lib.sh
index 1399d05..fb15028 100644
--- a/performance-test/perf-test-lib.sh
+++ b/performance-test/perf-test-lib.sh
@@ -51,7 +51,9 @@ time_run () {
     if test "$verbose" != "t"; then exec 4>test.output 3>&4; fi
     if ! eval >&3 "/usr/bin/time -f '%e\t%U\t%S\t%M\t%I\t%O' $2" ; then
 	test_failure=$(($test_failure + 1))
+	return 1
     fi
+    return 0
 }
 
 time_done () {
-- 
1.7.10.4

[Patch v2 02/11] perf-test: add argument parsing for performance tests

[david]

From: David Bremner <bremner@debian.org>

This patch just sets (non-exported) variables. The variable $debug is
already used, and $corpus_size will be used in following commits.
---
 performance-test/perf-test-lib.sh |   25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/performance-test/perf-test-lib.sh b/performance-test/perf-test-lib.sh
index fb15028..3a4a23d 100644
--- a/performance-test/perf-test-lib.sh
+++ b/performance-test/perf-test-lib.sh
@@ -1,5 +1,30 @@
 . ./version.sh
 
+corpus_size=large
+
+while test "$#" -ne 0
+do
+	case "$1" in
+	-d|--debug)
+		debug=t;
+		shift
+		;;
+	-s|--small)
+		corpus_size=small;
+		shift
+		;;
+	-m|--medium)
+		corpus_size=medium;
+		shift
+		;;
+	-l|--large)
+		corpus_size=large;
+		shift
+		;;
+	*)
+		echo "error: unknown performance test option '$1'" >&2; exit 1 ;;
+	esac
+done
 . ../test/test-lib-common.sh
 
 set -e
-- 
1.7.10.4

[Patch v2 03/11] perf-test: add corpus size to output, compact I/O stats

[david]

From: David Bremner <bremner@debian.org>

Austin suggested a while ago that the corpus size be printed in the
header. In the end it seems the corpus will be fixed per test script,
so this suggestion indeed makes sense.

The tabbing was wrapping on my usual 80 column terminal, so I joined
the input and output columns together.
---
 performance-test/perf-test-lib.sh |    6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/performance-test/perf-test-lib.sh b/performance-test/perf-test-lib.sh
index 3a4a23d..e399d3f 100644
--- a/performance-test/perf-test-lib.sh
+++ b/performance-test/perf-test-lib.sh
@@ -67,14 +67,14 @@ add_email_corpus ()
 }
 
 print_header () {
-    printf "[v%4s]               Wall(s)\tUsr(s)\tSys(s)\tRes(K)\tIn(512B)\tOut(512B)\n" \
-	   ${PERFTEST_VERSION}
+    printf "[v%4s %6s]        Wall(s)\tUsr(s)\tSys(s)\tRes(K)\tIn/Out(512B)\n" \
+	   ${PERFTEST_VERSION} ${corpus_size}
 }
 
 time_run () {
     printf "%-22s" "$1"
     if test "$verbose" != "t"; then exec 4>test.output 3>&4; fi
-    if ! eval >&3 "/usr/bin/time -f '%e\t%U\t%S\t%M\t%I\t%O' $2" ; then
+    if ! eval >&3 "/usr/bin/time -f '%e\t%U\t%S\t%M\t%I/%O' $2" ; then
 	test_failure=$(($test_failure + 1))
 	return 1
     fi
-- 
1.7.10.4

[Patch v2 04/11] perf-test: optionally print description for each group of tests

[david]

From: David Bremner <bremner@debian.org>

Output from tests is indented slightly in the same style as the
correctness tests.
---
 performance-test/perf-test-lib.sh |    9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/performance-test/perf-test-lib.sh b/performance-test/perf-test-lib.sh
index e399d3f..415bf15 100644
--- a/performance-test/perf-test-lib.sh
+++ b/performance-test/perf-test-lib.sh
@@ -67,12 +67,12 @@ add_email_corpus ()
 }
 
 print_header () {
-    printf "[v%4s %6s]        Wall(s)\tUsr(s)\tSys(s)\tRes(K)\tIn/Out(512B)\n" \
+    printf "[v%4s %6s]          Wall(s)\tUsr(s)\tSys(s)\tRes(K)\tIn/Out(512B)\n" \
 	   ${PERFTEST_VERSION} ${corpus_size}
 }
 
 time_run () {
-    printf "%-22s" "$1"
+    printf "  %-22s" "$1"
     if test "$verbose" != "t"; then exec 4>test.output 3>&4; fi
     if ! eval >&3 "/usr/bin/time -f '%e\t%U\t%S\t%M\t%I/%O' $2" ; then
 	test_failure=$(($test_failure + 1))
@@ -92,3 +92,8 @@ time_done () {
 
 cd -P "$test" || error "Cannot setup test environment"
 test_failure=0
+
+if [[ -n "test_description" ]]; then
+    echo
+    echo $(basename "$0"): "Benchmarking ${test_description}"
+fi
-- 
1.7.10.4

[Patch v2 05/11] perf-test: cache unpacked corpus

[david]

From: David Bremner <bremner@debian.org>

Unpacking is not really the expensive step (compared to the initial
notmuch new), but this is a pre-requisite to caching the database.
---
 performance-test/.gitignore       |    1 +
 performance-test/Makefile.local   |    2 +-
 performance-test/perf-test-lib.sh |   51 ++++++++++++++++++++++++-------------
 3 files changed, 36 insertions(+), 18 deletions(-)

diff --git a/performance-test/.gitignore b/performance-test/.gitignore
index 53f2697..796ed01 100644
--- a/performance-test/.gitignore
+++ b/performance-test/.gitignore
@@ -1 +1,2 @@
 tmp.*/
+corpus/
diff --git a/performance-test/Makefile.local b/performance-test/Makefile.local
index 5d2acbd..63e4c3d 100644
--- a/performance-test/Makefile.local
+++ b/performance-test/Makefile.local
@@ -29,4 +29,4 @@ $(TXZFILE):
 download-corpus:
 	wget -O ${TXZFILE} ${DEFAULT_URL}
 
-CLEAN := $(CLEAN) $(dir)/tmp.*
+CLEAN := $(CLEAN) $(dir)/tmp.* $(dir)/corpus
diff --git a/performance-test/perf-test-lib.sh b/performance-test/perf-test-lib.sh
index 415bf15..0c10718 100644
--- a/performance-test/perf-test-lib.sh
+++ b/performance-test/perf-test-lib.sh
@@ -39,33 +39,50 @@ add_email_corpus ()
 {
     rm -rf ${MAIL_DIR}
 
-    case "$1" in
-	--small)
-	    arg="mail/enron/bailey-s"
+    case "$corpus_size" in
+	small)
+	    mail_subdir="mail/enron/bailey-s"
+	    check_for="${TEST_DIRECTORY}/corpus/$mail_subdir"
 	    ;;
-	--medium)
-	    arg="mail/notmuch-archive"
+	medium)
+	    mail_subdir="mail/notmuch-archive"
+	    check_for="${TEST_DIRECTORY}/corpus/$mail_subdir"
 	    ;;
 	*)
-	    arg=mail
+	    mail_subdir=mail
+	    check_for="${TEST_DIRECTORY}/corpus/$mail_subdir/enron/wolfe-j"
     esac
 
-    if command -v pixz > /dev/null; then
-	XZ=pixz
-    else
-	XZ=xz
+    MAIL_CORPUS="${TEST_DIRECTORY}/corpus/$mail_subdir"
+    args=()
+    if [ ! -d "$check_for" ] ; then
+	args+=("notmuch-email-corpus/$mail_subdir")
     fi
 
-    printf "Unpacking corpus\n"
-    tar --checkpoint=.5000 --extract --strip-components=1 \
-	--directory ${TMP_DIRECTORY} \
-	--use-compress-program ${XZ} \
-	--file ../download/notmuch-email-corpus-${PERFTEST_VERSION}.tar.xz \
-	notmuch-email-corpus/"$arg"
+    if [[ ${#args[@]} > 0 ]]; then
+	if command -v pixz > /dev/null; then
+	    XZ=pixz
+	else
+	    XZ=xz
+	fi
+
+	printf "Unpacking corpus\n"
+	mkdir -p "${TEST_DIRECTORY}/corpus"
+
+	tar --checkpoint=.5000 --extract --strip-components=1 \
+	    --directory ${TEST_DIRECTORY}/corpus \
+	    --use-compress-program ${XZ} \
+	    --file ../download/notmuch-email-corpus-${PERFTEST_VERSION}.tar.xz \
+	    "${args[@]}"
+
+	printf "\n"
 
-    printf "\n"
+    fi
+
+    cp -lr $MAIL_CORPUS $MAIL_DIR
 }
 
+
 print_header () {
     printf "[v%4s %6s]          Wall(s)\tUsr(s)\tSys(s)\tRes(K)\tIn/Out(512B)\n" \
 	   ${PERFTEST_VERSION} ${corpus_size}
-- 
1.7.10.4

[Patch v2 06/11] perf-test: add caching of xapian database

[david]

From: David Bremner <bremner@debian.org>

The caching and uncaching seem to be necessarily manual, as timing the
initial notmuch new is one of our goals with this suite.
---
 performance-test/.gitignore       |    1 +
 performance-test/Makefile.local   |    2 +-
 performance-test/basic            |    5 +++++
 performance-test/perf-test-lib.sh |   17 +++++++++++++++++
 4 files changed, 24 insertions(+), 1 deletion(-)

diff --git a/performance-test/.gitignore b/performance-test/.gitignore
index 796ed01..6421a9a 100644
--- a/performance-test/.gitignore
+++ b/performance-test/.gitignore
@@ -1,2 +1,3 @@
 tmp.*/
 corpus/
+notmuch.cache.*/
diff --git a/performance-test/Makefile.local b/performance-test/Makefile.local
index 63e4c3d..3834e4d 100644
--- a/performance-test/Makefile.local
+++ b/performance-test/Makefile.local
@@ -29,4 +29,4 @@ $(TXZFILE):
 download-corpus:
 	wget -O ${TXZFILE} ${DEFAULT_URL}
 
-CLEAN := $(CLEAN) $(dir)/tmp.* $(dir)/corpus
+CLEAN := $(CLEAN) $(dir)/tmp.* $(dir)/corpus $(dir)/notmuch.cache.*
diff --git a/performance-test/basic b/performance-test/basic
index 9d015ee..41a7ff1 100755
--- a/performance-test/basic
+++ b/performance-test/basic
@@ -2,11 +2,16 @@
 
 . ./perf-test-lib.sh
 
+uncache_database
+
 add_email_corpus
 
 print_header
 
 time_run 'initial notmuch new' 'notmuch new'
+
+cache_database
+
 time_run 'second notmuch new' 'notmuch new'
 time_run 'dump *' 'notmuch dump > tags.out'
 time_run 'restore *' 'notmuch restore < tags.out'
diff --git a/performance-test/perf-test-lib.sh b/performance-test/perf-test-lib.sh
index 0c10718..0f9796d 100644
--- a/performance-test/perf-test-lib.sh
+++ b/performance-test/perf-test-lib.sh
@@ -35,6 +35,8 @@ then
 	exit 1
 fi
 
+DB_CACHE_DIR=${TEST_DIRECTORY}/notmuch.cache.$corpus_size
+
 add_email_corpus ()
 {
     rm -rf ${MAIL_DIR}
@@ -80,8 +82,23 @@ add_email_corpus ()
     fi
 
     cp -lr $MAIL_CORPUS $MAIL_DIR
+
+    if [ -d $DB_CACHE_DIR ]; then
+	cp -r $DB_CACHE_DIR ${MAIL_DIR}/.notmuch
+    fi
+}
+
+cache_database () {
+    if [ -d $MAIL_DIR/.notmuch ]; then
+	cp -r $MAIL_DIR/.notmuch $DB_CACHE_DIR
+    else
+	echo "Warning: No database found to cache"
+    fi
 }
 
+uncache_database () {
+    rm -rf $DB_CACHE_DIR
+}
 
 print_header () {
     printf "[v%4s %6s]          Wall(s)\tUsr(s)\tSys(s)\tRes(K)\tIn/Out(512B)\n" \
-- 
1.7.10.4

[Patch v2 07/11] perf-test: update README

[david]

From: David Bremner <bremner@debian.org>

Describe new argument parsing and mention cache handling routines.
---
 performance-test/README |   22 +++++++++++++---------
 1 file changed, 13 insertions(+), 9 deletions(-)

diff --git a/performance-test/README b/performance-test/README
index 36f1dac..fd5c977 100644
--- a/performance-test/README
+++ b/performance-test/README
@@ -42,17 +42,21 @@ available performance tests.
 
 Alternately, you can run a specific subset of tests by simply invoking
 one of the executable scripts in this directory, (such as ./basic).
+Each test script supports the following arguments
+
+--small / --medium / --large	Choose corpus size.
+--debug				Enable debugging. In particular don't delete
+				temporary directories.
 
 Writing tests
 -------------
 
-Have a look at "basic" for an example.
-
-add_email_corpus takes arguments "--small" and "--medium" for when you
-want smaller subsets of the corpus to check.
-
-time_done does the cleanup; comment it out or define "$debug" to leave
-the temporary files around.
+Have a look at "basic" for an example. Sourcing "perf-test-lib.sh" is
+mandatory.  Utility functions include
 
-Currently there is no option processing (e.g. --debug) in the
-performance tests.
+- 'add_email_corpus' unpacks a set of messages and adds them to the database.
+- 'cache_database': makes a snapshot of the current database
+- 'uncache_database': forces the next 'add_email_corpus' to rebuild the
+  database.
+- 'time_done' does the cleanup; comment it out or pass --debug to the
+  script to leave the temporary files around.
-- 
1.7.10.4

[Patch v2 08/11] perf-test: bump corpus version to 0.3

[david]

From: David Bremner <bremner@debian.org>

The new version ships with some tags, and an updated archive of the
notmuch mailing list.
---
 performance-test/download/notmuch-email-corpus-0.2.tar.xz.asc |    9 ---------
 performance-test/download/notmuch-email-corpus-0.3.tar.xz.asc |    9 +++++++++
 performance-test/version.sh                                   |    2 +-
 3 files changed, 10 insertions(+), 10 deletions(-)
 delete mode 100644 performance-test/download/notmuch-email-corpus-0.2.tar.xz.asc
 create mode 100644 performance-test/download/notmuch-email-corpus-0.3.tar.xz.asc

diff --git a/performance-test/download/notmuch-email-corpus-0.2.tar.xz.asc b/performance-test/download/notmuch-email-corpus-0.2.tar.xz.asc
deleted file mode 100644
index c8b4b3d..0000000
--- a/performance-test/download/notmuch-email-corpus-0.2.tar.xz.asc
+++ /dev/null
@@ -1,9 +0,0 @@
------BEGIN PGP SIGNATURE-----
-Version: GnuPG v1.4.12 (GNU/Linux)
-
-iJwEAAECAAYFAlCsvx0ACgkQTiiN/0Um85kZAwP9GgOQ22jK8mr5X4pT/mB8EjSH
-QbndlxxbRrP0ChTqjBQoD3IsTHjNL7W572BfXb/MNo94R/iIQ7yTHCDVNuwBhvKd
-7qgIuW2FUS1uTfJRP5KBNf8JPuin+6wqGe8/+y/iOs+XJSdiYg1ElS49Ntnpg0yl
-btImgEcxTxQ2qfzDS1g=
-=iuZR
------END PGP SIGNATURE-----
diff --git a/performance-test/download/notmuch-email-corpus-0.3.tar.xz.asc b/performance-test/download/notmuch-email-corpus-0.3.tar.xz.asc
new file mode 100644
index 0000000..f109e81
--- /dev/null
+++ b/performance-test/download/notmuch-email-corpus-0.3.tar.xz.asc
@@ -0,0 +1,9 @@
+-----BEGIN PGP SIGNATURE-----
+Version: GnuPG v1.4.12 (GNU/Linux)
+
+iJwEAAECAAYFAlC9a90ACgkQTiiN/0Um85nAMAP+LCWdKzolcl/KW+JcCd0Dk+9v
+0vvtBVEhBes0TbK6iWrxCV2OIuYG/RhnFlJTZ4MjgaTRxzDubpC+JktaJdLmIQUN
+B7ZIDMjFduCwmtyLiuu/00CjxJKUXm7vx+ULGpvp0uxFE/vaqGP997BHwBjjfBVm
+YX6BlLX1SV6TfENkuRE=
+=Mks5
+-----END PGP SIGNATURE-----
diff --git a/performance-test/version.sh b/performance-test/version.sh
index d9270b1..afafc73 100644
--- a/performance-test/version.sh
+++ b/performance-test/version.sh
@@ -1,3 +1,3 @@
 # this should be both a valid Makefile fragment and valid POSIX(ish) shell.
 
-PERFTEST_VERSION=0.2
+PERFTEST_VERSION=0.3
-- 
1.7.10.4

[Patch v2 09/11] perf-test: unpack tags.

[david]

From: David Bremner <bremner@debian.org>

There is only one set of tags, independant of the size of message
corpus chosen.
---
 performance-test/perf-test-lib.sh |    7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/performance-test/perf-test-lib.sh b/performance-test/perf-test-lib.sh
index 0f9796d..ca08026 100644
--- a/performance-test/perf-test-lib.sh
+++ b/performance-test/perf-test-lib.sh
@@ -56,7 +56,13 @@ add_email_corpus ()
     esac
 
     MAIL_CORPUS="${TEST_DIRECTORY}/corpus/$mail_subdir"
+    TAG_CORPUS="${TEST_DIRECTORY}/corpus/tags"
+
     args=()
+    if [ ! -d "$TAG_CORPUS" ] ; then
+	args+=("notmuch-email-corpus/tags")
+    fi
+
     if [ ! -d "$check_for" ] ; then
 	args+=("notmuch-email-corpus/$mail_subdir")
     fi
@@ -81,6 +87,7 @@ add_email_corpus ()
 
     fi
 
+    cp -lr $TAG_CORPUS $TMP_DIRECTORY/corpus.tags
     cp -lr $MAIL_CORPUS $MAIL_DIR
 
     if [ -d $DB_CACHE_DIR ]; then
-- 
1.7.10.4

[Patch v2 10/11] perf-test: split basic into 00-new and 01-dump-restore

[david]

From: David Bremner <bremner@debian.org>

We rely on subsequent "notmuch new" invocations being relatively fast
rather than trying to detect if notmuch-new needs to be run.
---
 performance-test/00-new            |   21 +++++++++++++++++++++
 performance-test/01-dump-restore   |   15 +++++++++++++++
 performance-test/README            |    8 ++++++--
 performance-test/basic             |   20 --------------------
 performance-test/notmuch-perf-test |    3 ++-
 5 files changed, 44 insertions(+), 23 deletions(-)
 create mode 100755 performance-test/00-new
 create mode 100755 performance-test/01-dump-restore
 delete mode 100755 performance-test/basic

diff --git a/performance-test/00-new b/performance-test/00-new
new file mode 100755
index 0000000..02db874
--- /dev/null
+++ b/performance-test/00-new
@@ -0,0 +1,21 @@
+#!/bin/bash
+
+test_description='notmuch new'
+
+. ./perf-test-lib.sh
+
+uncache_database
+
+add_email_corpus
+
+print_header
+
+time_run 'initial notmuch new' 'notmuch new'
+
+cache_database
+
+for i in $(seq 2 6); do
+    time_run "notmuch new #$i" 'notmuch new'
+done
+
+time_done
diff --git a/performance-test/01-dump-restore b/performance-test/01-dump-restore
new file mode 100755
index 0000000..09444e3
--- /dev/null
+++ b/performance-test/01-dump-restore
@@ -0,0 +1,15 @@
+#!/bin/bash
+
+test_description='Dump and restore'
+
+. ./perf-test-lib.sh
+
+add_email_corpus
+
+print_header
+
+time_run 'notmuch new' 'notmuch new'
+time_run 'dump *' 'notmuch dump > tags.out'
+time_run 'restore *' 'notmuch restore < tags.out'
+
+time_done
diff --git a/performance-test/README b/performance-test/README
index fd5c977..35489a9 100644
--- a/performance-test/README
+++ b/performance-test/README
@@ -51,8 +51,8 @@ Each test script supports the following arguments
 Writing tests
 -------------
 
-Have a look at "basic" for an example. Sourcing "perf-test-lib.sh" is
-mandatory.  Utility functions include
+Have a look at "01-dump-restore" for an example. Sourcing
+"perf-test-lib.sh" is mandatory.  Utility functions include
 
 - 'add_email_corpus' unpacks a set of messages and adds them to the database.
 - 'cache_database': makes a snapshot of the current database
@@ -60,3 +60,7 @@ mandatory.  Utility functions include
   database.
 - 'time_done' does the cleanup; comment it out or pass --debug to the
   script to leave the temporary files around.
+
+Scripts are run in the order specified in notmuch-perf-test. In the
+future this order might be chosen automatically so please follow the
+convention of starting the name with two digits to specify the order.
diff --git a/performance-test/basic b/performance-test/basic
deleted file mode 100755
index 41a7ff1..0000000
--- a/performance-test/basic
+++ /dev/null
@@ -1,20 +0,0 @@
-#!/bin/bash
-
-. ./perf-test-lib.sh
-
-uncache_database
-
-add_email_corpus
-
-print_header
-
-time_run 'initial notmuch new' 'notmuch new'
-
-cache_database
-
-time_run 'second notmuch new' 'notmuch new'
-time_run 'dump *' 'notmuch dump > tags.out'
-time_run 'restore *' 'notmuch restore < tags.out'
-time_run 'tag * +new_tag' "notmuch tag +new_tag '*'"
-
-time_done
diff --git a/performance-test/notmuch-perf-test b/performance-test/notmuch-perf-test
index 1bea345..f93d8a4 100755
--- a/performance-test/notmuch-perf-test
+++ b/performance-test/notmuch-perf-test
@@ -17,7 +17,8 @@ fi
 cd $(dirname "$0")
 
 TESTS="
-  basic
+  00-new
+  01-dump-restore
 "
 
 for test in $TESTS; do
-- 
1.7.10.4

[Patch v2 11/11] perf-test: use nmbug tags in dump-restore tests

[david]

From: David Bremner <bremner@debian.org>

This makes the tag set a bit less trivial.

Note that if you use the small corpus, this is not so interesting (and
is also a bit noisy) since the messages will not be found. In the
future this could be checked for.
---
 performance-test/01-dump-restore |    1 +
 1 file changed, 1 insertion(+)

diff --git a/performance-test/01-dump-restore b/performance-test/01-dump-restore
index 09444e3..7bcf821 100755
--- a/performance-test/01-dump-restore
+++ b/performance-test/01-dump-restore
@@ -9,6 +9,7 @@ add_email_corpus
 print_header
 
 time_run 'notmuch new' 'notmuch new'
+time_run 'load nmbug tags' 'notmuch restore --accumulate < corpus.tags/nmbug.sup-dump'
 time_run 'dump *' 'notmuch dump > tags.out'
 time_run 'restore *' 'notmuch restore < tags.out'
 
-- 
1.7.10.4

Re: [Patch v2 04/11] perf-test: optionally print description for each group of tests

[Austin Clements]

On Wed, 05 Dec 2012, david@tethera.net wrote:
> From: David Bremner <bremner@debian.org>
>
> Output from tests is indented slightly in the same style as the
> correctness tests.
> ---
>  performance-test/perf-test-lib.sh |    9 +++++++--
>  1 file changed, 7 insertions(+), 2 deletions(-)
>
> diff --git a/performance-test/perf-test-lib.sh b/performance-test/perf-test-lib.sh
> index e399d3f..415bf15 100644
> --- a/performance-test/perf-test-lib.sh
> +++ b/performance-test/perf-test-lib.sh
> @@ -67,12 +67,12 @@ add_email_corpus ()
>  }
>  
>  print_header () {
> -    printf "[v%4s %6s]        Wall(s)\tUsr(s)\tSys(s)\tRes(K)\tIn/Out(512B)\n" \
> +    printf "[v%4s %6s]          Wall(s)\tUsr(s)\tSys(s)\tRes(K)\tIn/Out(512B)\n" \
>  	   ${PERFTEST_VERSION} ${corpus_size}
>  }
>  
>  time_run () {
> -    printf "%-22s" "$1"
> +    printf "  %-22s" "$1"
>      if test "$verbose" != "t"; then exec 4>test.output 3>&4; fi
>      if ! eval >&3 "/usr/bin/time -f '%e\t%U\t%S\t%M\t%I/%O' $2" ; then
>  	test_failure=$(($test_failure + 1))
> @@ -92,3 +92,8 @@ time_done () {
>  
>  cd -P "$test" || error "Cannot setup test environment"
>  test_failure=0
> +
> +if [[ -n "test_description" ]]; then

I think you meant "$test_description".  Though, it seems like it would
be confusing if $test_description wasn't set and there was no output.
Maybe this shouldn't be conditional and below should be
${test_description:-UNNAMED} or something?

> +    echo
> +    echo $(basename "$0"): "Benchmarking ${test_description}"
> +fi
> -- 
> 1.7.10.4
>
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

Re: [Patch v2 10/11] perf-test: split basic into 00-new and 01-dump-restore

[Austin Clements]

On Wed, 05 Dec 2012, david@tethera.net wrote:
> From: David Bremner <bremner@debian.org>
>
> We rely on subsequent "notmuch new" invocations being relatively fast
> rather than trying to detect if notmuch-new needs to be run.
> ---
>  performance-test/00-new            |   21 +++++++++++++++++++++
>  performance-test/01-dump-restore   |   15 +++++++++++++++
>  performance-test/README            |    8 ++++++--
>  performance-test/basic             |   20 --------------------
>  performance-test/notmuch-perf-test |    3 ++-
>  5 files changed, 44 insertions(+), 23 deletions(-)
>  create mode 100755 performance-test/00-new
>  create mode 100755 performance-test/01-dump-restore
>  delete mode 100755 performance-test/basic
>
> diff --git a/performance-test/00-new b/performance-test/00-new
> new file mode 100755
> index 0000000..02db874
> --- /dev/null
> +++ b/performance-test/00-new
> @@ -0,0 +1,21 @@
> +#!/bin/bash
> +
> +test_description='notmuch new'
> +
> +. ./perf-test-lib.sh
> +
> +uncache_database
> +
> +add_email_corpus
> +
> +print_header
> +
> +time_run 'initial notmuch new' 'notmuch new'
> +
> +cache_database
> +
> +for i in $(seq 2 6); do
> +    time_run "notmuch new #$i" 'notmuch new'
> +done
> +
> +time_done
> diff --git a/performance-test/01-dump-restore b/performance-test/01-dump-restore
> new file mode 100755
> index 0000000..09444e3
> --- /dev/null
> +++ b/performance-test/01-dump-restore
> @@ -0,0 +1,15 @@
> +#!/bin/bash
> +
> +test_description='Dump and restore'
> +
> +. ./perf-test-lib.sh
> +
> +add_email_corpus
> +
> +print_header
> +
> +time_run 'notmuch new' 'notmuch new'
> +time_run 'dump *' 'notmuch dump > tags.out'
> +time_run 'restore *' 'notmuch restore < tags.out'
> +
> +time_done
> diff --git a/performance-test/README b/performance-test/README
> index fd5c977..35489a9 100644
> --- a/performance-test/README
> +++ b/performance-test/README
> @@ -51,8 +51,8 @@ Each test script supports the following arguments
>  Writing tests
>  -------------
>  
> -Have a look at "basic" for an example. Sourcing "perf-test-lib.sh" is
> -mandatory.  Utility functions include
> +Have a look at "01-dump-restore" for an example. Sourcing
> +"perf-test-lib.sh" is mandatory.  Utility functions include
>  
>  - 'add_email_corpus' unpacks a set of messages and adds them to the database.
>  - 'cache_database': makes a snapshot of the current database
> @@ -60,3 +60,7 @@ mandatory.  Utility functions include
>    database.
>  - 'time_done' does the cleanup; comment it out or pass --debug to the
>    script to leave the temporary files around.
> +
> +Scripts are run in the order specified in notmuch-perf-test. In the
> +future this order might be chosen automatically so please follow the
> +convention of starting the name with two digits to specify the order.
> diff --git a/performance-test/basic b/performance-test/basic
> deleted file mode 100755
> index 41a7ff1..0000000
> --- a/performance-test/basic
> +++ /dev/null
> @@ -1,20 +0,0 @@
> -#!/bin/bash
> -
> -. ./perf-test-lib.sh
> -
> -uncache_database
> -
> -add_email_corpus
> -
> -print_header
> -
> -time_run 'initial notmuch new' 'notmuch new'
> -
> -cache_database
> -
> -time_run 'second notmuch new' 'notmuch new'
> -time_run 'dump *' 'notmuch dump > tags.out'
> -time_run 'restore *' 'notmuch restore < tags.out'
> -time_run 'tag * +new_tag' "notmuch tag +new_tag '*'"

Did you mean to lose the tag-everything test?

> -
> -time_done
> diff --git a/performance-test/notmuch-perf-test b/performance-test/notmuch-perf-test
> index 1bea345..f93d8a4 100755
> --- a/performance-test/notmuch-perf-test
> +++ b/performance-test/notmuch-perf-test
> @@ -17,7 +17,8 @@ fi
>  cd $(dirname "$0")
>  
>  TESTS="
> -  basic
> +  00-new
> +  01-dump-restore
>  "
>  
>  for test in $TESTS; do
> -- 
> 1.7.10.4
>
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

Re: Performance test improvements, v2

[Austin Clements]

This series LGTM except the few nits I pointed out.

On Wed, 05 Dec 2012, david@tethera.net wrote:
> This obsoletes the several tests and loose patches posted in thread
>
>      id:1354583824-10520-1-git-send-email-david@tethera.net
>
> The main changes introduced here are caching, and some non-trivial
> tags are used in the dump restore tests.
>
> Caching now makes it sane to split the tests into several files, and
> the output is tidied up a bit for the multi-file case.
>
>
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch