simplify and fix performance tests

simplify and fix performance tests

[david]

Jani noticed the alleged file notmuch-memory-test was missing.
Austin grumbled about explicitly adding tests to a list.
This series should give them both something else to grumble about.
 

[PATCH 1/4] perf-test: rename memory tests

[david]

From: David Bremner <bremner@debian.org>

The common suffix will help running them all automatically.
---
 performance-test/M00-new             | 15 ---------------
 performance-test/M00-new.sh          | 15 +++++++++++++++
 performance-test/M01-dump-restore    | 15 ---------------
 performance-test/M01-dump-restore.sh | 15 +++++++++++++++
 4 files changed, 30 insertions(+), 30 deletions(-)
 delete mode 100755 performance-test/M00-new
 create mode 100755 performance-test/M00-new.sh
 delete mode 100755 performance-test/M01-dump-restore
 create mode 100755 performance-test/M01-dump-restore.sh

diff --git a/performance-test/M00-new b/performance-test/M00-new
deleted file mode 100755
index 99c3f52..0000000
--- a/performance-test/M00-new
+++ /dev/null
@@ -1,15 +0,0 @@
-#!/bin/bash
-
-test_description='notmuch new'
-
-. ./perf-test-lib.sh
-
-# ensure initial 'notmuch new' is run by memory_start
-uncache_database
-
-memory_start
-
-# run 'notmuch new' a second time, to test different code paths
-memory_run "notmuch new" "notmuch new"
-
-memory_done
diff --git a/performance-test/M00-new.sh b/performance-test/M00-new.sh
new file mode 100755
index 0000000..99c3f52
--- /dev/null
+++ b/performance-test/M00-new.sh
@@ -0,0 +1,15 @@
+#!/bin/bash
+
+test_description='notmuch new'
+
+. ./perf-test-lib.sh
+
+# ensure initial 'notmuch new' is run by memory_start
+uncache_database
+
+memory_start
+
+# run 'notmuch new' a second time, to test different code paths
+memory_run "notmuch new" "notmuch new"
+
+memory_done
diff --git a/performance-test/M01-dump-restore b/performance-test/M01-dump-restore
deleted file mode 100755
index be5894a..0000000
--- a/performance-test/M01-dump-restore
+++ /dev/null
@@ -1,15 +0,0 @@
-#!/bin/bash
-
-test_description='dump and restore'
-
-. ./perf-test-lib.sh
-
-memory_start
-
-memory_run 'load nmbug tags' 'notmuch restore --accumulate --input=corpus.tags/nmbug.sup-dump'
-memory_run 'dump *' 'notmuch dump --output=tags.sup'
-memory_run 'restore *' 'notmuch restore --input=tags.sup'
-memory_run 'dump --format=batch-tag *' 'notmuch dump --format=batch-tag --output=tags.bt'
-memory_run 'restore --format=batch-tag *' 'notmuch restore --format=batch-tag --input=tags.bt'
-
-memory_done
diff --git a/performance-test/M01-dump-restore.sh b/performance-test/M01-dump-restore.sh
new file mode 100755
index 0000000..be5894a
--- /dev/null
+++ b/performance-test/M01-dump-restore.sh
@@ -0,0 +1,15 @@
+#!/bin/bash
+
+test_description='dump and restore'
+
+. ./perf-test-lib.sh
+
+memory_start
+
+memory_run 'load nmbug tags' 'notmuch restore --accumulate --input=corpus.tags/nmbug.sup-dump'
+memory_run 'dump *' 'notmuch dump --output=tags.sup'
+memory_run 'restore *' 'notmuch restore --input=tags.sup'
+memory_run 'dump --format=batch-tag *' 'notmuch dump --format=batch-tag --output=tags.bt'
+memory_run 'restore --format=batch-tag *' 'notmuch restore --format=batch-tag --input=tags.bt'
+
+memory_done
-- 
1.8.2.rc2

[PATCH 2/4] perf-test: rename time tests to have .sh suffix

[david]

From: David Bremner <bremner@debian.org>

This will simplify notmuch time tests
---
 performance-test/T00-new             | 15 ---------------
 performance-test/T00-new.sh          | 15 +++++++++++++++
 performance-test/T01-dump-restore    | 13 -------------
 performance-test/T01-dump-restore.sh | 13 +++++++++++++
 performance-test/T02-tag             | 14 --------------
 performance-test/T02-tag.sh          | 14 ++++++++++++++
 6 files changed, 42 insertions(+), 42 deletions(-)
 delete mode 100755 performance-test/T00-new
 create mode 100755 performance-test/T00-new.sh
 delete mode 100755 performance-test/T01-dump-restore
 create mode 100755 performance-test/T01-dump-restore.sh
 delete mode 100755 performance-test/T02-tag
 create mode 100755 performance-test/T02-tag.sh

diff --git a/performance-test/T00-new b/performance-test/T00-new
deleted file mode 100755
index 553bb8b..0000000
--- a/performance-test/T00-new
+++ /dev/null
@@ -1,15 +0,0 @@
-#!/bin/bash
-
-test_description='notmuch new'
-
-. ./perf-test-lib.sh
-
-uncache_database
-
-time_start
-
-for i in $(seq 2 6); do
-    time_run "notmuch new #$i" 'notmuch new'
-done
-
-time_done
diff --git a/performance-test/T00-new.sh b/performance-test/T00-new.sh
new file mode 100755
index 0000000..553bb8b
--- /dev/null
+++ b/performance-test/T00-new.sh
@@ -0,0 +1,15 @@
+#!/bin/bash
+
+test_description='notmuch new'
+
+. ./perf-test-lib.sh
+
+uncache_database
+
+time_start
+
+for i in $(seq 2 6); do
+    time_run "notmuch new #$i" 'notmuch new'
+done
+
+time_done
diff --git a/performance-test/T01-dump-restore b/performance-test/T01-dump-restore
deleted file mode 100755
index b2ff940..0000000
--- a/performance-test/T01-dump-restore
+++ /dev/null
@@ -1,13 +0,0 @@
-#!/bin/bash
-
-test_description='dump and restore'
-
-. ./perf-test-lib.sh
-
-time_start
-
-time_run 'load nmbug tags' 'notmuch restore --accumulate < corpus.tags/nmbug.sup-dump'
-time_run 'dump *' 'notmuch dump > tags.out'
-time_run 'restore *' 'notmuch restore < tags.out'
-
-time_done
diff --git a/performance-test/T01-dump-restore.sh b/performance-test/T01-dump-restore.sh
new file mode 100755
index 0000000..b2ff940
--- /dev/null
+++ b/performance-test/T01-dump-restore.sh
@@ -0,0 +1,13 @@
+#!/bin/bash
+
+test_description='dump and restore'
+
+. ./perf-test-lib.sh
+
+time_start
+
+time_run 'load nmbug tags' 'notmuch restore --accumulate < corpus.tags/nmbug.sup-dump'
+time_run 'dump *' 'notmuch dump > tags.out'
+time_run 'restore *' 'notmuch restore < tags.out'
+
+time_done
diff --git a/performance-test/T02-tag b/performance-test/T02-tag
deleted file mode 100755
index 78ceccc..0000000
--- a/performance-test/T02-tag
+++ /dev/null
@@ -1,14 +0,0 @@
-#!/bin/bash
-
-test_description='tagging'
-
-. ./perf-test-lib.sh
-
-time_start
-
-time_run 'tag * +new_tag' "notmuch tag +new_tag '*'"
-time_run 'tag * +existing_tag' "notmuch tag +new_tag '*'"
-time_run 'tag * -existing_tag' "notmuch tag -new_tag '*'"
-time_run 'tag * -missing_tag' "notmuch tag -new_tag '*'"
-
-time_done
diff --git a/performance-test/T02-tag.sh b/performance-test/T02-tag.sh
new file mode 100755
index 0000000..78ceccc
--- /dev/null
+++ b/performance-test/T02-tag.sh
@@ -0,0 +1,14 @@
+#!/bin/bash
+
+test_description='tagging'
+
+. ./perf-test-lib.sh
+
+time_start
+
+time_run 'tag * +new_tag' "notmuch tag +new_tag '*'"
+time_run 'tag * +existing_tag' "notmuch tag +new_tag '*'"
+time_run 'tag * -existing_tag' "notmuch tag -new_tag '*'"
+time_run 'tag * -missing_tag' "notmuch tag -new_tag '*'"
+
+time_done
-- 
1.8.2.rc2

[PATCH 3/4] perf-test: run all appropriately named time tests

[david]

From: David Bremner <bremner@debian.org>

This avoids hassle with manually adding every test to the master list.
---
 performance-test/notmuch-time-test | 10 ++--------
 1 file changed, 2 insertions(+), 8 deletions(-)

diff --git a/performance-test/notmuch-time-test b/performance-test/notmuch-time-test
index 54a208f..7113efb 100755
--- a/performance-test/notmuch-time-test
+++ b/performance-test/notmuch-time-test
@@ -16,12 +16,6 @@ fi
 
 cd $(dirname "$0")
 
-TESTS="
-  T00-new
-  T01-dump-restore
-  T02-tag
-"
-
-for test in $TESTS; do
-    ./$test "$@"
+for test in T*.sh; do
+    ./"$test" "$@"
 done
-- 
1.8.2.rc2

[PATCH 4/4] perf-test: add notmuch-memory-test

[david]

From: David Bremner <bremner@debian.org>

Somehow this file was not added to the patch set which split the tests
into time and memory tests. Take advantage of the the "new" way of
running tests to avoid listing the explicitly.
---
 performance-test/notmuch-memory-test | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)
 create mode 100755 performance-test/notmuch-memory-test

diff --git a/performance-test/notmuch-memory-test b/performance-test/notmuch-memory-test
new file mode 100755
index 0000000..3cf28c7
--- /dev/null
+++ b/performance-test/notmuch-memory-test
@@ -0,0 +1,21 @@
+#!/usr/bin/env bash
+
+# Run tests
+#
+# Copyright (c) 2005 Junio C Hamano
+#
+# Adapted from a Makefile to a shell script by Carl Worth (2010)
+
+if [ ${BASH_VERSINFO[0]} -lt 4 ]; then
+    echo "Error: The notmuch test suite requires a bash version >= 4.0"
+    echo "due to use of associative arrays within the test suite."
+    echo "Please try again with a newer bash (or help us fix the"
+    echo "test suite to be more portable). Thanks."
+    exit 1
+fi
+
+cd $(dirname "$0")
+
+for test in M*.sh; do
+    ./"$test" "$@"
+done
-- 
1.8.2.rc2

Re: simplify and fix performance tests

[Tomi Ollila]

On Wed, May 15 2013, david@tethera.net wrote:

> Jani noticed the alleged file notmuch-memory-test was missing.
> Austin grumbled about explicitly adding tests to a list.
> This series should give them both something else to grumble about.

This test series looks good to me. It is consistent what was there
before and easies future development.

If there is desire to handle the cases where executed test
scripts exit nonzero that can be done in future commits
(in any case).

+1

Tomi

Re: simplify and fix performance tests

[Austin Clements]

LGTM.

Quoth david@tethera.net on May 15 at  8:02 am:
> Jani noticed the alleged file notmuch-memory-test was missing.
> Austin grumbled about explicitly adding tests to a list.
> This series should give them both something else to grumble about.

Re: simplify and fix performance tests

[David Bremner]

david@tethera.net writes:

> Jani noticed the alleged file notmuch-memory-test was missing.
> Austin grumbled about explicitly adding tests to a list.
> This series should give them both something else to grumble about.

Pushed,

d