v2 of valgrind based memory tests

v2 of valgrind based memory tests

[david]

These obsolete
      
      id:1355196820-29734-1-git-send-email-david@tethera.net

I tried to follow the suggestions of 

  id:20121216191121.GH6187@mit.edu

pretty closely.

diff --git a/performance-test/M00-new b/performance-test/M00-new
index 733e9b0..99c3f52 100755
--- a/performance-test/M00-new
+++ b/performance-test/M00-new
@@ -9,6 +9,7 @@ uncache_database
 
 memory_start
 
+# run 'notmuch new' a second time, to test different code paths
 memory_run "notmuch new" "notmuch new"
 
 memory_done
diff --git a/performance-test/Makefile.local b/performance-test/Makefile.local
index 357d800..73aa963 100644
--- a/performance-test/Makefile.local
+++ b/performance-test/Makefile.local
@@ -4,7 +4,6 @@ dir := performance-test
 
 include $(dir)/version.sh
 
-# these two are just make sure dir is expanded at the right time.
 TIME_TEST_SCRIPT := ${dir}/notmuch-time-test
 MEMORY_TEST_SCRIPT := ${dir}/notmuch-memory-test
 
@@ -17,11 +16,11 @@ perf-test: time-test memory-test
 
 time-test: setup-perf-test all
 	@echo
-	$(TIME_TEST_SCRIPT) $(TEST_OPTIONS)
+	$(TIME_TEST_SCRIPT) $(OPTIONS)
 
 memory-test: setup-perf-test all
 	@echo
-	$(MEMORY_TEST_SCRIPT) $(TEST_OPTIONS)
+	$(MEMORY_TEST_SCRIPT) $(OPTIONS)
 
 
 .PHONY: download-corpus setup-perf-test
diff --git a/performance-test/README b/performance-test/README
index 7eaf5f7..996724c 100644
--- a/performance-test/README
+++ b/performance-test/README
@@ -1,7 +1,7 @@
 Performance Tests
 -----------------
 
-This directory contains two kinds of performance tests, time tests,
+This directory contains two kinds of performance tests: time tests,
 and memory tests. The former use gnu time, and the latter use
 valgrind.
 
@@ -12,7 +12,7 @@ In addition to having notmuch, you need:
 
 - gpg
 - gnu tar
-- gnu time (for the time tests).
+- gnu time (for the time tests)
 - xz. Some speedup can be gotten by installing "pixz", but this is
   probably only worthwhile if you are debugging the tests.
 - valgrind (for the memory tests)
@@ -59,13 +59,13 @@ supports the following arguments
 				temporary directories.
 
 When using the make targets, you can pass arguments to all test
-scripts by defining the make variable TEST_OPTIONS.
+scripts by defining the make variable OPTIONS.
 
 Writing tests
 -------------
 
-Have a look at "T01-dump-restore" for an example time test and and
-"M00-new" for an example memory tests. In both cases sourcing
+Have a look at "T01-dump-restore" for an example time test and
+"M00-new" for an example memory test. In both cases sourcing
 "perf-test-lib.sh" is mandatory.
 
 Basics:
diff --git a/performance-test/perf-test-lib.sh b/performance-test/perf-test-lib.sh
index 79eb2c5..10d05e0 100644
--- a/performance-test/perf-test-lib.sh
+++ b/performance-test/perf-test-lib.sh
@@ -89,11 +89,10 @@ add_email_corpus ()
 
     cp -lr $TAG_CORPUS $TMP_DIRECTORY/corpus.tags
     cp -lr $MAIL_CORPUS $MAIL_DIR
-
 }
 
-notmuch_new_with_cache () {
-
+notmuch_new_with_cache ()
+{
     if [ -d $DB_CACHE_DIR ]; then
 	cp -r $DB_CACHE_DIR ${MAIL_DIR}/.notmuch
     else
@@ -102,8 +101,8 @@ notmuch_new_with_cache () {
     fi
 }
 
-time_start () {
-
+time_start ()
+{
     add_email_corpus
 
     print_header
@@ -111,17 +110,19 @@ time_start () {
     notmuch_new_with_cache time_run
 }
 
-memory_start () {
-
+memory_start ()
+{
     add_email_corpus
 
-    _timestamp=$(printf "%x" $(date +"%s"))
-    log_dir=$(mktemp -d "${TEST_DIRECTORY}/log.$(basename $0)-$corpus_size-${_timestamp}-XXXXXX")
+    local timestamp=$(date +%Y%m%dT%H%M%S)
+    log_dir="${TEST_DIRECTORY}/log.$(basename $0)-$corpus_size-${timestamp}"
+    mkdir -p ${log_dir}
 
     notmuch_new_with_cache memory_run
 }
 
-memory_run () {
+memory_run ()
+{
     test_count=$(($test_count+1))
 
     log_file=$log_dir/$test_count.log
@@ -134,11 +135,13 @@ memory_run () {
     echo
 }
 
-memory_done () {
+memory_done ()
+{
     time_done
 }
 
-cache_database () {
+cache_database ()
+{
     if [ -d $MAIL_DIR/.notmuch ]; then
 	cp -r $MAIL_DIR/.notmuch $DB_CACHE_DIR
     else
@@ -146,16 +149,18 @@ cache_database () {
     fi
 }
 
-uncache_database () {
+uncache_database ()
+{
     rm -rf $DB_CACHE_DIR
 }
 
-print_header () {
+print_header ()
+{
     printf "\t\t\tWall(s)\tUsr(s)\tSys(s)\tRes(K)\tIn/Out(512B)\n"
-
 }
 
-time_run () {
+time_run ()
+{
     printf "  %-22s" "$1"
     test_count=$(($test_count+1))
     if test "$verbose" != "t"; then exec 4>test.output 3>&4; fi
@@ -166,7 +171,8 @@ time_run () {
     return 0
 }
 
-time_done () {
+time_done ()
+{
     if [ "$test_failure" = "0" ]; then
 	rm -rf "$remove_tmp"
 	exit 0

[Patch v2 1/4] perf-test: remove redundant "initial notmuch new"

[david]

From: David Bremner <bremner@debian.org>

The initial notmuch-new and caching are now done automatically by
time_start
---
 performance-test/00-new |    4 ----
 1 file changed, 4 deletions(-)

diff --git a/performance-test/00-new b/performance-test/00-new
index 6f0b50c..553bb8b 100755
--- a/performance-test/00-new
+++ b/performance-test/00-new
@@ -8,10 +8,6 @@ uncache_database
 
 time_start
 
-time_run 'initial notmuch new' 'notmuch new'
-
-cache_database
-
 for i in $(seq 2 6); do
     time_run "notmuch new #$i" 'notmuch new'
 done
-- 
1.7.10.4

[Patch v2 2/4] perf-test: rename current tests as "time tests"

[david]

From: David Bremner <bremner@debian.org>

This is almost entirely renaming files, except for updating a few
references to those file names, and changing the makefile target.

A new set of memory tests will be run separately because they take
much longer.
---
 performance-test/00-new            |   15 ---------------
 performance-test/01-dump-restore   |   13 -------------
 performance-test/02-tag            |   14 --------------
 performance-test/Makefile.local    |    2 +-
 performance-test/README            |    9 +++++----
 performance-test/T00-new           |   15 +++++++++++++++
 performance-test/T01-dump-restore  |   13 +++++++++++++
 performance-test/T02-tag           |   14 ++++++++++++++
 performance-test/notmuch-perf-test |   27 ---------------------------
 performance-test/notmuch-time-test |   27 +++++++++++++++++++++++++++
 10 files changed, 75 insertions(+), 74 deletions(-)
 delete mode 100755 performance-test/00-new
 delete mode 100755 performance-test/01-dump-restore
 delete mode 100755 performance-test/02-tag
 create mode 100755 performance-test/T00-new
 create mode 100755 performance-test/T01-dump-restore
 create mode 100755 performance-test/T02-tag
 delete mode 100755 performance-test/notmuch-perf-test
 create mode 100755 performance-test/notmuch-time-test

diff --git a/performance-test/00-new b/performance-test/00-new
deleted file mode 100755
index 553bb8b..0000000
--- a/performance-test/00-new
+++ /dev/null
@@ -1,15 +0,0 @@
-#!/bin/bash
-
-test_description='notmuch new'
-
-. ./perf-test-lib.sh
-
-uncache_database
-
-time_start
-
-for i in $(seq 2 6); do
-    time_run "notmuch new #$i" 'notmuch new'
-done
-
-time_done
diff --git a/performance-test/01-dump-restore b/performance-test/01-dump-restore
deleted file mode 100755
index b2ff940..0000000
--- a/performance-test/01-dump-restore
+++ /dev/null
@@ -1,13 +0,0 @@
-#!/bin/bash
-
-test_description='dump and restore'
-
-. ./perf-test-lib.sh
-
-time_start
-
-time_run 'load nmbug tags' 'notmuch restore --accumulate < corpus.tags/nmbug.sup-dump'
-time_run 'dump *' 'notmuch dump > tags.out'
-time_run 'restore *' 'notmuch restore < tags.out'
-
-time_done
diff --git a/performance-test/02-tag b/performance-test/02-tag
deleted file mode 100755
index 78ceccc..0000000
--- a/performance-test/02-tag
+++ /dev/null
@@ -1,14 +0,0 @@
-#!/bin/bash
-
-test_description='tagging'
-
-. ./perf-test-lib.sh
-
-time_start
-
-time_run 'tag * +new_tag' "notmuch tag +new_tag '*'"
-time_run 'tag * +existing_tag' "notmuch tag +new_tag '*'"
-time_run 'tag * -existing_tag' "notmuch tag -new_tag '*'"
-time_run 'tag * -missing_tag' "notmuch tag -new_tag '*'"
-
-time_done
diff --git a/performance-test/Makefile.local b/performance-test/Makefile.local
index 3834e4d..57beb44 100644
--- a/performance-test/Makefile.local
+++ b/performance-test/Makefile.local
@@ -10,7 +10,7 @@ SIGFILE := ${TXZFILE}.asc
 TEST_SCRIPT := ${dir}/notmuch-perf-test
 DEFAULT_URL :=  http://notmuchmail.org/releases/${CORPUS_NAME}
 
-perf-test: setup-perf-test all
+time-test: setup-perf-test all
 	$(TEST_SCRIPT) $(OPTIONS)
 
 .PHONY: download-corpus setup-perf-test
diff --git a/performance-test/README b/performance-test/README
index 1481660..d1fb6de 100644
--- a/performance-test/README
+++ b/performance-test/README
@@ -36,8 +36,8 @@ for a list of mirrors.
 Running tests
 -------------
 
-The easiest way to run performance tests is to say "make perf-test", (or
-simply run the notmuch-perf-test script). Either command will run all
+The easiest way to run performance tests is to say "make time-test", (or
+simply run the notmuch-time-test script). Either command will run all
 available performance tests.
 
 Alternately, you can run a specific subset of tests by simply invoking
@@ -51,7 +51,7 @@ Each test script supports the following arguments
 Writing tests
 -------------
 
-Have a look at "01-dump-restore" for an example. Sourcing
+Have a look at "T01-dump-restore" for an example. Sourcing
 "perf-test-lib.sh" is mandatory.  Utility functions include
 
 - 'add_email_corpus' unpacks a set of messages and adds them to the database.
@@ -65,4 +65,5 @@ Have a look at "01-dump-restore" for an example. Sourcing
 
 Scripts are run in the order specified in notmuch-perf-test. In the
 future this order might be chosen automatically so please follow the
-convention of starting the name with two digits to specify the order.
+convention of starting the name with 'T' followed by two digits to
+specify the order.
diff --git a/performance-test/T00-new b/performance-test/T00-new
new file mode 100755
index 0000000..553bb8b
--- /dev/null
+++ b/performance-test/T00-new
@@ -0,0 +1,15 @@
+#!/bin/bash
+
+test_description='notmuch new'
+
+. ./perf-test-lib.sh
+
+uncache_database
+
+time_start
+
+for i in $(seq 2 6); do
+    time_run "notmuch new #$i" 'notmuch new'
+done
+
+time_done
diff --git a/performance-test/T01-dump-restore b/performance-test/T01-dump-restore
new file mode 100755
index 0000000..b2ff940
--- /dev/null
+++ b/performance-test/T01-dump-restore
@@ -0,0 +1,13 @@
+#!/bin/bash
+
+test_description='dump and restore'
+
+. ./perf-test-lib.sh
+
+time_start
+
+time_run 'load nmbug tags' 'notmuch restore --accumulate < corpus.tags/nmbug.sup-dump'
+time_run 'dump *' 'notmuch dump > tags.out'
+time_run 'restore *' 'notmuch restore < tags.out'
+
+time_done
diff --git a/performance-test/T02-tag b/performance-test/T02-tag
new file mode 100755
index 0000000..78ceccc
--- /dev/null
+++ b/performance-test/T02-tag
@@ -0,0 +1,14 @@
+#!/bin/bash
+
+test_description='tagging'
+
+. ./perf-test-lib.sh
+
+time_start
+
+time_run 'tag * +new_tag' "notmuch tag +new_tag '*'"
+time_run 'tag * +existing_tag' "notmuch tag +new_tag '*'"
+time_run 'tag * -existing_tag' "notmuch tag -new_tag '*'"
+time_run 'tag * -missing_tag' "notmuch tag -new_tag '*'"
+
+time_done
diff --git a/performance-test/notmuch-perf-test b/performance-test/notmuch-perf-test
deleted file mode 100755
index fc39d8a..0000000
--- a/performance-test/notmuch-perf-test
+++ /dev/null
@@ -1,27 +0,0 @@
-#!/usr/bin/env bash
-
-# Run tests
-#
-# Copyright (c) 2005 Junio C Hamano
-#
-# Adapted from a Makefile to a shell script by Carl Worth (2010)
-
-if [ ${BASH_VERSINFO[0]} -lt 4 ]; then
-    echo "Error: The notmuch test suite requires a bash version >= 4.0"
-    echo "due to use of associative arrays within the test suite."
-    echo "Please try again with a newer bash (or help us fix the"
-    echo "test suite to be more portable). Thanks."
-    exit 1
-fi
-
-cd $(dirname "$0")
-
-TESTS="
-  00-new
-  01-dump-restore
-  02-tag
-"
-
-for test in $TESTS; do
-    ./$test "$@"
-done
diff --git a/performance-test/notmuch-time-test b/performance-test/notmuch-time-test
new file mode 100755
index 0000000..54a208f
--- /dev/null
+++ b/performance-test/notmuch-time-test
@@ -0,0 +1,27 @@
+#!/usr/bin/env bash
+
+# Run tests
+#
+# Copyright (c) 2005 Junio C Hamano
+#
+# Adapted from a Makefile to a shell script by Carl Worth (2010)
+
+if [ ${BASH_VERSINFO[0]} -lt 4 ]; then
+    echo "Error: The notmuch test suite requires a bash version >= 4.0"
+    echo "due to use of associative arrays within the test suite."
+    echo "Please try again with a newer bash (or help us fix the"
+    echo "test suite to be more portable). Thanks."
+    exit 1
+fi
+
+cd $(dirname "$0")
+
+TESTS="
+  T00-new
+  T01-dump-restore
+  T02-tag
+"
+
+for test in $TESTS; do
+    ./$test "$@"
+done
-- 
1.7.10.4

[Patch v2 3/4] perf-test: initial version of memory test infrastructure.

[david]

From: David Bremner <bremner@debian.org>

The idea is run some code under valgrind --leak-check=full and report
a summary, leaving the user to peruse the log file if they want.

We go to some lengths to preserve the log files from accidental
overwriting; the full corpus takes about 3 hours to run under valgrind
on my machine.

The naming of the log directories may be slightly controversial; in
the unlikely event of two runs in less than a second, the log will be
overwritten. A previous version with mktemp+timestamp was dismissed as
overkill; just mktemp alone does not sort nicely.

One new test is included, to check notmuch new for memory leaks.
---
 performance-test/.gitignore       |    1 +
 performance-test/M00-new          |   15 ++++++++
 performance-test/Makefile.local   |   16 ++++++--
 performance-test/README           |   57 +++++++++++++++++++---------
 performance-test/perf-test-lib.sh |   75 ++++++++++++++++++++++++++++---------
 5 files changed, 126 insertions(+), 38 deletions(-)
 create mode 100755 performance-test/M00-new

diff --git a/performance-test/.gitignore b/performance-test/.gitignore
index 6421a9a..f3f9be4 100644
--- a/performance-test/.gitignore
+++ b/performance-test/.gitignore
@@ -1,3 +1,4 @@
 tmp.*/
+log.*/
 corpus/
 notmuch.cache.*/
diff --git a/performance-test/M00-new b/performance-test/M00-new
new file mode 100755
index 0000000..99c3f52
--- /dev/null
+++ b/performance-test/M00-new
@@ -0,0 +1,15 @@
+#!/bin/bash
+
+test_description='notmuch new'
+
+. ./perf-test-lib.sh
+
+# ensure initial 'notmuch new' is run by memory_start
+uncache_database
+
+memory_start
+
+# run 'notmuch new' a second time, to test different code paths
+memory_run "notmuch new" "notmuch new"
+
+memory_done
diff --git a/performance-test/Makefile.local b/performance-test/Makefile.local
index 57beb44..73aa963 100644
--- a/performance-test/Makefile.local
+++ b/performance-test/Makefile.local
@@ -4,14 +4,24 @@ dir := performance-test
 
 include $(dir)/version.sh
 
+TIME_TEST_SCRIPT := ${dir}/notmuch-time-test
+MEMORY_TEST_SCRIPT := ${dir}/notmuch-memory-test
+
 CORPUS_NAME := notmuch-email-corpus-$(PERFTEST_VERSION).tar.xz
 TXZFILE := ${dir}/download/${CORPUS_NAME}
 SIGFILE := ${TXZFILE}.asc
-TEST_SCRIPT := ${dir}/notmuch-perf-test
 DEFAULT_URL :=  http://notmuchmail.org/releases/${CORPUS_NAME}
 
+perf-test: time-test memory-test
+
 time-test: setup-perf-test all
-	$(TEST_SCRIPT) $(OPTIONS)
+	@echo
+	$(TIME_TEST_SCRIPT) $(OPTIONS)
+
+memory-test: setup-perf-test all
+	@echo
+	$(MEMORY_TEST_SCRIPT) $(OPTIONS)
+
 
 .PHONY: download-corpus setup-perf-test
 
@@ -29,4 +39,4 @@ $(TXZFILE):
 download-corpus:
 	wget -O ${TXZFILE} ${DEFAULT_URL}
 
-CLEAN := $(CLEAN) $(dir)/tmp.* $(dir)/corpus $(dir)/notmuch.cache.*
+CLEAN := $(CLEAN) $(dir)/tmp.* $(dir)/log.* $(dir)/corpus $(dir)/notmuch.cache.*
diff --git a/performance-test/README b/performance-test/README
index d1fb6de..996724c 100644
--- a/performance-test/README
+++ b/performance-test/README
@@ -1,3 +1,10 @@
+Performance Tests
+-----------------
+
+This directory contains two kinds of performance tests: time tests,
+and memory tests. The former use gnu time, and the latter use
+valgrind.
+
 Pre-requisites
 --------------
 
@@ -5,9 +12,10 @@ In addition to having notmuch, you need:
 
 - gpg
 - gnu tar
-- gnu time
+- gnu time (for the time tests)
 - xz. Some speedup can be gotten by installing "pixz", but this is
   probably only worthwhile if you are debugging the tests.
+- valgrind (for the memory tests)
 
 Getting set up to run tests:
 ----------------------------
@@ -36,34 +44,47 @@ for a list of mirrors.
 Running tests
 -------------
 
-The easiest way to run performance tests is to say "make time-test", (or
-simply run the notmuch-time-test script). Either command will run all
-available performance tests.
-
-Alternately, you can run a specific subset of tests by simply invoking
-one of the executable scripts in this directory, (such as ./basic).
-Each test script supports the following arguments
+The easiest way to run performance tests is to say "make perf-test".
+This will run all time and memory tests.  Be aware that the memory
+tests are quite time consuming when run on the full corpus, and that
+depending on your interests it may be more sensible to run "make
+time-test" or "make memory-test".  You can also invoke one of the
+scripts notmuch-time-test or notmuch-memory-test or run a more
+specific subset of tests by simply invoking one of the executable
+scripts in this directory, (such as ./T00-new).  Each test script
+supports the following arguments
 
 --small / --medium / --large	Choose corpus size.
 --debug				Enable debugging. In particular don't delete
 				temporary directories.
 
+When using the make targets, you can pass arguments to all test
+scripts by defining the make variable OPTIONS.
+
 Writing tests
 -------------
 
-Have a look at "T01-dump-restore" for an example. Sourcing
-"perf-test-lib.sh" is mandatory.  Utility functions include
+Have a look at "T01-dump-restore" for an example time test and
+"M00-new" for an example memory test. In both cases sourcing
+"perf-test-lib.sh" is mandatory.
 
-- 'add_email_corpus' unpacks a set of messages and adds them to the database.
-- 'cache_database': makes a snapshot of the current database
-- 'uncache_database': forces the next 'add_email_corpus' to rebuild the
-  database.
-- 'time_start' unpacks the mail corpus and calls notmuch new if it
+Basics:
+
+- '(time|memory)_start' unpacks the mail corpus and calls notmuch new if it
    cannot find a cache of the appropriate corpus.
-- 'time_done' does the cleanup; comment it out or pass --debug to the
+- '(time|memory)_run' runs the command under time or valgrind. Currently
+  "memory_run" does not support i/o redirection in the command.
+- '(time|memory)_done' does the cleanup; comment it out or pass --debug to the
   script to leave the temporary files around.
 
+Utility functions include
+
+- 'add_email_corpus' unpacks a set of messages and tags
+- 'cache_database': makes a snapshot of the current database
+- 'uncache_database': forces the next '(time|memory)_start' to rebuild the
+  database.
+
 Scripts are run in the order specified in notmuch-perf-test. In the
 future this order might be chosen automatically so please follow the
-convention of starting the name with 'T' followed by two digits to
-specify the order.
+convention of starting the name with 'T' or 'M' followed by two digits
+to specify the order.
diff --git a/performance-test/perf-test-lib.sh b/performance-test/perf-test-lib.sh
index 3a7ef2b..10d05e0 100644
--- a/performance-test/perf-test-lib.sh
+++ b/performance-test/perf-test-lib.sh
@@ -89,24 +89,59 @@ add_email_corpus ()
 
     cp -lr $TAG_CORPUS $TMP_DIRECTORY/corpus.tags
     cp -lr $MAIL_CORPUS $MAIL_DIR
-
 }
 
-time_start () {
-
-    add_email_corpus
-
-    print_header
-
+notmuch_new_with_cache ()
+{
     if [ -d $DB_CACHE_DIR ]; then
 	cp -r $DB_CACHE_DIR ${MAIL_DIR}/.notmuch
     else
-	time_run 'Initial notmuch new' "notmuch new"
+	"$1" 'Initial notmuch new' "notmuch new"
 	cache_database
     fi
 }
 
-cache_database () {
+time_start ()
+{
+    add_email_corpus
+
+    print_header
+
+    notmuch_new_with_cache time_run
+}
+
+memory_start ()
+{
+    add_email_corpus
+
+    local timestamp=$(date +%Y%m%dT%H%M%S)
+    log_dir="${TEST_DIRECTORY}/log.$(basename $0)-$corpus_size-${timestamp}"
+    mkdir -p ${log_dir}
+
+    notmuch_new_with_cache memory_run
+}
+
+memory_run ()
+{
+    test_count=$(($test_count+1))
+
+    log_file=$log_dir/$test_count.log
+
+    printf "[ %d ]\t%s\n" $test_count "$1"
+
+    valgrind --leak-check=full --log-file="$log_file" $2
+
+    awk '/LEAK SUMMARY/,/suppressed/ { sub(/^==[0-9]*==/," "); print }' "$log_file"
+    echo
+}
+
+memory_done ()
+{
+    time_done
+}
+
+cache_database ()
+{
     if [ -d $MAIL_DIR/.notmuch ]; then
 	cp -r $MAIL_DIR/.notmuch $DB_CACHE_DIR
     else
@@ -114,17 +149,20 @@ cache_database () {
     fi
 }
 
-uncache_database () {
+uncache_database ()
+{
     rm -rf $DB_CACHE_DIR
 }
 
-print_header () {
-    printf "[v%4s %6s]          Wall(s)\tUsr(s)\tSys(s)\tRes(K)\tIn/Out(512B)\n" \
-	   ${PERFTEST_VERSION} ${corpus_size}
+print_header ()
+{
+    printf "\t\t\tWall(s)\tUsr(s)\tSys(s)\tRes(K)\tIn/Out(512B)\n"
 }
 
-time_run () {
+time_run ()
+{
     printf "  %-22s" "$1"
+    test_count=$(($test_count+1))
     if test "$verbose" != "t"; then exec 4>test.output 3>&4; fi
     if ! eval >&3 "/usr/bin/time -f '%e\t%U\t%S\t%M\t%I/%O' $2" ; then
 	test_failure=$(($test_failure + 1))
@@ -133,7 +171,8 @@ time_run () {
     return 0
 }
 
-time_done () {
+time_done ()
+{
     if [ "$test_failure" = "0" ]; then
 	rm -rf "$remove_tmp"
 	exit 0
@@ -144,6 +183,8 @@ time_done () {
 
 cd -P "$test" || error "Cannot setup test environment"
 test_failure=0
+test_count=0
 
-echo
-echo $(basename "$0"): "Testing ${test_description:-notmuch performance}"
+printf "\n%-55s [%s %s]\n"  \
+    "$(basename "$0"): Testing ${test_description:-notmuch performance}" \
+    "${PERFTEST_VERSION}"  "${corpus_size}"
-- 
1.7.10.4

[Patch v2 4/4] perf-test: add memory leak test for dump restore

[david]

From: David Bremner <bremner@debian.org>

In id:87vcc2q5n2.fsf@nikula.org, Jani points out a memory leak in the
current version of the sup restore code. Among other things, this test
is intended to verify a fix for that leak.
---
 performance-test/M01-dump-restore |   15 +++++++++++++++
 1 file changed, 15 insertions(+)
 create mode 100755 performance-test/M01-dump-restore

diff --git a/performance-test/M01-dump-restore b/performance-test/M01-dump-restore
new file mode 100755
index 0000000..be5894a
--- /dev/null
+++ b/performance-test/M01-dump-restore
@@ -0,0 +1,15 @@
+#!/bin/bash
+
+test_description='dump and restore'
+
+. ./perf-test-lib.sh
+
+memory_start
+
+memory_run 'load nmbug tags' 'notmuch restore --accumulate --input=corpus.tags/nmbug.sup-dump'
+memory_run 'dump *' 'notmuch dump --output=tags.sup'
+memory_run 'restore *' 'notmuch restore --input=tags.sup'
+memory_run 'dump --format=batch-tag *' 'notmuch dump --format=batch-tag --output=tags.bt'
+memory_run 'restore --format=batch-tag *' 'notmuch restore --format=batch-tag --input=tags.bt'
+
+memory_done
-- 
1.7.10.4

Re: v2 of valgrind based memory tests

[Austin Clements]

On Mon, 24 Dec 2012, david@tethera.net wrote:
> These obsolete
>       
>       id:1355196820-29734-1-git-send-email-david@tethera.net
>
> I tried to follow the suggestions of 
>
>   id:20121216191121.GH6187@mit.edu
>
> pretty closely.

LGTM.  In the long run, I wonder if we want to separate the time and
memory tests like this or if some tests would make sense in either mode.

Re: v2 of valgrind based memory tests

[David Bremner]

Austin Clements <amdragon@MIT.EDU> writes:

> On Mon, 24 Dec 2012, david@tethera.net wrote:
>
> LGTM.  In the long run, I wonder if we want to separate the time and
> memory tests like this or if some tests would make sense in either mode.

pushed. At the moment, the split into time versus memory tests is a
simple way to separate out "heavy weight" tests, but this could be done
more explicitely.

d