* v4 of Performance tests @ 2012-11-25 15:02 david 2012-11-25 15:02 ` [Patch v4 1/2] test: factor out part of test-lib.sh into test-lib-common.sh david ` (2 more replies) 0 siblings, 3 replies; 8+ messages in thread From: david @ 2012-11-25 15:02 UTC (permalink / raw) To: notmuch This version adds an optional makefile target to download the corpus. Tomi reviewed it on IRC and didn't complain. ^ permalink raw reply [flat|nested] 8+ messages in thread
* [Patch v4 1/2] test: factor out part of test-lib.sh into test-lib-common.sh 2012-11-25 15:02 v4 of Performance tests david @ 2012-11-25 15:02 ` david 2012-11-25 15:02 ` [Patch v4 2/2] test: initial performance testing infrastructure david 2012-11-25 21:19 ` v4 of Performance tests Tomi Ollila 2 siblings, 0 replies; 8+ messages in thread From: david @ 2012-11-25 15:02 UTC (permalink / raw) To: notmuch; +Cc: David Bremner From: David Bremner <bremner@debian.org> The idea is to use some of the simpler parts of the test suite infrastructure to help run performance tests. --- test/test-lib-common.sh | 147 +++++++++++++++++++++++++++++++++++++++++++++++ test/test-lib.sh | 129 +---------------------------------------- 2 files changed, 148 insertions(+), 128 deletions(-) create mode 100644 test/test-lib-common.sh diff --git a/test/test-lib-common.sh b/test/test-lib-common.sh new file mode 100644 index 0000000..e1eaa5a --- /dev/null +++ b/test/test-lib-common.sh @@ -0,0 +1,147 @@ +# +# Copyright (c) 2005 Junio C Hamano +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see http://www.gnu.org/licenses/ . + +# This file contains common code to be used by both the regular +# (correctness) tests and the performance tests. + +find_notmuch_path () +{ + dir="$1" + + while [ -n "$dir" ]; do + bin="$dir/notmuch" + if [ -x "$bin" ]; then + echo "$dir" + return + fi + dir="$(dirname "$dir")" + if [ "$dir" = "/" ]; then + break + fi + done +} + +# Test the binaries we have just built. The tests are kept in +# test/ subdirectory and are run in 'trash directory' subdirectory. +TEST_DIRECTORY=$(pwd) +notmuch_path=`find_notmuch_path "$TEST_DIRECTORY"` +if test -n "$valgrind" +then + make_symlink () { + test -h "$2" && + test "$1" = "$(readlink "$2")" || { + # be super paranoid + if mkdir "$2".lock + then + rm -f "$2" && + ln -s "$1" "$2" && + rm -r "$2".lock + else + while test -d "$2".lock + do + say "Waiting for lock on $2." + sleep 1 + done + fi + } + } + + make_valgrind_symlink () { + # handle only executables + test -x "$1" || return + + base=$(basename "$1") + symlink_target=$TEST_DIRECTORY/../$base + # do not override scripts + if test -x "$symlink_target" && + test ! -d "$symlink_target" && + test "#!" != "$(head -c 2 < "$symlink_target")" + then + symlink_target=$TEST_DIRECTORY/valgrind.sh + fi + case "$base" in + *.sh|*.perl) + symlink_target=$TEST_DIRECTORY/unprocessed-script + esac + # create the link, or replace it if it is out of date + make_symlink "$symlink_target" "$GIT_VALGRIND/bin/$base" || exit + } + + # override notmuch executable in TEST_DIRECTORY/.. + GIT_VALGRIND=$TEST_DIRECTORY/valgrind + mkdir -p "$GIT_VALGRIND"/bin + make_valgrind_symlink $TEST_DIRECTORY/../notmuch + OLDIFS=$IFS + IFS=: + for path in $PATH + do + ls "$path"/notmuch 2> /dev/null | + while read file + do + make_valgrind_symlink "$file" + done + done + IFS=$OLDIFS + PATH=$GIT_VALGRIND/bin:$PATH + GIT_EXEC_PATH=$GIT_VALGRIND/bin + export GIT_VALGRIND + test -n "$notmuch_path" && MANPATH="$notmuch_path/man:$MANPATH" +else # normal case + if test -n "$notmuch_path" + then + PATH="$notmuch_path:$PATH" + MANPATH="$notmuch_path/man:$MANPATH" + fi +fi +export PATH MANPATH + +# Test repository +test="tmp.$(basename "$0" .sh)" +test -n "$root" && test="$root/$test" +case "$test" in +/*) TMP_DIRECTORY="$test" ;; + *) TMP_DIRECTORY="$TEST_DIRECTORY/$test" ;; +esac +test ! -z "$debug" || remove_tmp=$TMP_DIRECTORY +rm -fr "$test" || { + GIT_EXIT_OK=t + echo >&5 "FATAL: Cannot prepare test area" + exit 1 +} + +# A temporary home directory is needed by at least: +# - emacs/"Sending a message via (fake) SMTP" +# - emacs/"Reply within emacs" +# - crypto/emacs_deliver_message +export HOME="${TMP_DIRECTORY}/home" +mkdir -p "${HOME}" + +MAIL_DIR="${TMP_DIRECTORY}/mail" +export GNUPGHOME="${TMP_DIRECTORY}/gnupg" +export NOTMUCH_CONFIG="${TMP_DIRECTORY}/notmuch-config" + +mkdir -p "${test}" +mkdir -p "${MAIL_DIR}" + +cat <<EOF >"${NOTMUCH_CONFIG}" +[database] +path=${MAIL_DIR} + +[user] +name=Notmuch Test Suite +primary_email=test_suite@notmuchmail.org +other_email=test_suite_other@notmuchmail.org;test_suite@otherdomain.org +EOF diff --git a/test/test-lib.sh b/test/test-lib.sh index e7638be..467b83c 100644 --- a/test/test-lib.sh +++ b/test/test-lib.sh @@ -1070,134 +1070,7 @@ test_init_ () { } -find_notmuch_path () -{ - dir="$1" - - while [ -n "$dir" ]; do - bin="$dir/notmuch" - if [ -x "$bin" ]; then - echo "$dir" - return - fi - dir="$(dirname "$dir")" - if [ "$dir" = "/" ]; then - break - fi - done -} - -# Test the binaries we have just built. The tests are kept in -# test/ subdirectory and are run in 'trash directory' subdirectory. -TEST_DIRECTORY=$(pwd) -notmuch_path=`find_notmuch_path "$TEST_DIRECTORY"` -if test -n "$valgrind" -then - make_symlink () { - test -h "$2" && - test "$1" = "$(readlink "$2")" || { - # be super paranoid - if mkdir "$2".lock - then - rm -f "$2" && - ln -s "$1" "$2" && - rm -r "$2".lock - else - while test -d "$2".lock - do - say "Waiting for lock on $2." - sleep 1 - done - fi - } - } - - make_valgrind_symlink () { - # handle only executables - test -x "$1" || return - - base=$(basename "$1") - symlink_target=$TEST_DIRECTORY/../$base - # do not override scripts - if test -x "$symlink_target" && - test ! -d "$symlink_target" && - test "#!" != "$(head -c 2 < "$symlink_target")" - then - symlink_target=$TEST_DIRECTORY/valgrind.sh - fi - case "$base" in - *.sh|*.perl) - symlink_target=$TEST_DIRECTORY/unprocessed-script - esac - # create the link, or replace it if it is out of date - make_symlink "$symlink_target" "$GIT_VALGRIND/bin/$base" || exit - } - - # override notmuch executable in TEST_DIRECTORY/.. - GIT_VALGRIND=$TEST_DIRECTORY/valgrind - mkdir -p "$GIT_VALGRIND"/bin - make_valgrind_symlink $TEST_DIRECTORY/../notmuch - OLDIFS=$IFS - IFS=: - for path in $PATH - do - ls "$path"/notmuch 2> /dev/null | - while read file - do - make_valgrind_symlink "$file" - done - done - IFS=$OLDIFS - PATH=$GIT_VALGRIND/bin:$PATH - GIT_EXEC_PATH=$GIT_VALGRIND/bin - export GIT_VALGRIND - test -n "$notmuch_path" && MANPATH="$notmuch_path/man:$MANPATH" -else # normal case - if test -n "$notmuch_path" - then - PATH="$notmuch_path:$PATH" - MANPATH="$notmuch_path/man:$MANPATH" - fi -fi -export PATH MANPATH - -# Test repository -test="tmp.$(basename "$0" .sh)" -test -n "$root" && test="$root/$test" -case "$test" in -/*) TMP_DIRECTORY="$test" ;; - *) TMP_DIRECTORY="$TEST_DIRECTORY/$test" ;; -esac -test ! -z "$debug" || remove_tmp=$TMP_DIRECTORY -rm -fr "$test" || { - GIT_EXIT_OK=t - echo >&5 "FATAL: Cannot prepare test area" - exit 1 -} - -# A temporary home directory is needed by at least: -# - emacs/"Sending a message via (fake) SMTP" -# - emacs/"Reply within emacs" -# - crypto/emacs_deliver_message -export HOME="${TMP_DIRECTORY}/home" -mkdir -p "${HOME}" - -MAIL_DIR="${TMP_DIRECTORY}/mail" -export GNUPGHOME="${TMP_DIRECTORY}/gnupg" -export NOTMUCH_CONFIG="${TMP_DIRECTORY}/notmuch-config" - -mkdir -p "${test}" -mkdir -p "${MAIL_DIR}" - -cat <<EOF >"${NOTMUCH_CONFIG}" -[database] -path=${MAIL_DIR} - -[user] -name=Notmuch Test Suite -primary_email=test_suite@notmuchmail.org -other_email=test_suite_other@notmuchmail.org;test_suite@otherdomain.org -EOF +. ./test-lib-common.sh emacs_generate_script -- 1.7.10.4 ^ permalink raw reply related [flat|nested] 8+ messages in thread
* [Patch v4 2/2] test: initial performance testing infrastructure 2012-11-25 15:02 v4 of Performance tests david 2012-11-25 15:02 ` [Patch v4 1/2] test: factor out part of test-lib.sh into test-lib-common.sh david @ 2012-11-25 15:02 ` david 2012-11-25 21:40 ` Austin Clements 2012-11-25 21:19 ` v4 of Performance tests Tomi Ollila 2 siblings, 1 reply; 8+ messages in thread From: david @ 2012-11-25 15:02 UTC (permalink / raw) To: notmuch; +Cc: David Bremner From: David Bremner <bremner@debian.org> This is not near as fancy as as the unit tests, on the theory that the code should typically be crashing when performance tuning. Nonetheless, there is plenty of room for improvement. Several more of the pieces of the test infrastructure (e.g. the option parsing) could be factored out into test/test-lib-common.sh --- Makefile | 3 +- performance-test/.gitignore | 1 + performance-test/Makefile | 7 +++ performance-test/Makefile.local | 32 ++++++++++ performance-test/README | 50 +++++++++++++++ performance-test/basic | 15 +++++ performance-test/download/.gitignore | 2 + .../download/notmuch-email-corpus-0.2.tar.xz.asc | 9 +++ performance-test/notmuch-perf-test | 25 ++++++++ performance-test/perf-test-lib.sh | 65 ++++++++++++++++++++ performance-test/version.sh | 3 + 11 files changed, 211 insertions(+), 1 deletion(-) create mode 100644 performance-test/.gitignore create mode 100644 performance-test/Makefile create mode 100644 performance-test/Makefile.local create mode 100644 performance-test/README create mode 100755 performance-test/basic create mode 100644 performance-test/download/.gitignore create mode 100644 performance-test/download/notmuch-email-corpus-0.2.tar.xz.asc create mode 100755 performance-test/notmuch-perf-test create mode 100644 performance-test/perf-test-lib.sh create mode 100644 performance-test/version.sh diff --git a/Makefile b/Makefile index bb9c316..5decbea 100644 --- a/Makefile +++ b/Makefile @@ -3,7 +3,8 @@ all: # List all subdirectories here. Each contains its own Makefile.local -subdirs = compat completion emacs lib man parse-time-string util test +subdirs := compat completion emacs lib man parse-time-string +subdirs := $(subdirs) performance-test util test # We make all targets depend on the Makefiles themselves. global_deps = Makefile Makefile.config Makefile.local \ diff --git a/performance-test/.gitignore b/performance-test/.gitignore new file mode 100644 index 0000000..53f2697 --- /dev/null +++ b/performance-test/.gitignore @@ -0,0 +1 @@ +tmp.*/ diff --git a/performance-test/Makefile b/performance-test/Makefile new file mode 100644 index 0000000..de492a7 --- /dev/null +++ b/performance-test/Makefile @@ -0,0 +1,7 @@ +# See Makefile.local for the list of files to be compiled in this +# directory. +all: + $(MAKE) -C .. all + +.DEFAULT: + $(MAKE) -C .. $@ diff --git a/performance-test/Makefile.local b/performance-test/Makefile.local new file mode 100644 index 0000000..1114ec1 --- /dev/null +++ b/performance-test/Makefile.local @@ -0,0 +1,32 @@ +# -*- makefile -*- + +dir := performance-test + +include $(dir)/version.sh + +CORPUS_NAME := notmuch-email-corpus-$(PERFTEST_VERSION).tar.xz +TXZFILE := ${dir}/download/${CORPUS_NAME} +SIGFILE := ${TXZFILE}.asc +TEST_SCRIPT := ${dir}/notmuch-perf-test +DEFAULT_URL := http://notmuchmail.org/releases/${CORPUS_NAME} + +perf-test: setup-perf-test all + $(TEST_SCRIPT) $(OPTIONS) + +.PHONY: download-corpus setup-perf-test + +# Note that this intentionally does not depend on download-corpus. +setup-perf-test: $(TXZFILE) + gpg --verify $(SIGFILE) + +$(TXZFILE): + @printf "\nPlease download ${TXZFILE}.\n\n" + @printf "\t%% make download-corpus\n\n" + @echo or see http://notmuchmail.org/corpus for download locations + @echo + @false + +download-corpus: + wget -O ${TXZFILE} ${DEFAULT_URL} + +CLEAN := $(CLEAN) $(dir)/tmp.* diff --git a/performance-test/README b/performance-test/README new file mode 100644 index 0000000..239d2fb --- /dev/null +++ b/performance-test/README @@ -0,0 +1,50 @@ +Pre-requisites +-------------- + +In addition to having notmuch, you need: + +- gpg +- gnu tar +- gnu time +- xz. Some speedup can be gotten by installing "pixz", but this is + probably only worthwhile if you are debugging the tests. + +Getting set up to run tests: +---------------------------- + +First, you need to get the corpus. + +It should work to run + + % make download-corpus + +In case that fails or is too slow, check + + http://notmuchmail.org/corpus + +for a list of mirrors. + +Running tests +------------- + +The easiest way to run performance tests is to say "make perf-test", (or +simply run the notmuch-perf-test script). Either command will run all +available performance tests. + +Alternately, you can run a specific subset of tests by simply invoking +one of the executable scripts in this directory, (such as ./basic). + + +Writing tests +------------- + +Have a look at "basic" for an example. + +add_email_corpus takes arguments "--small" and "--medium" for when you +want smaller corpuses to check. + +time_done does the cleanup; comment it out or define "$debug" to leave +the temporary files around. + +Currently there is no option processing (e.g. --debug) in the +performance tests. diff --git a/performance-test/basic b/performance-test/basic new file mode 100755 index 0000000..9d015ee --- /dev/null +++ b/performance-test/basic @@ -0,0 +1,15 @@ +#!/bin/bash + +. ./perf-test-lib.sh + +add_email_corpus + +print_header + +time_run 'initial notmuch new' 'notmuch new' +time_run 'second notmuch new' 'notmuch new' +time_run 'dump *' 'notmuch dump > tags.out' +time_run 'restore *' 'notmuch restore < tags.out' +time_run 'tag * +new_tag' "notmuch tag +new_tag '*'" + +time_done diff --git a/performance-test/download/.gitignore b/performance-test/download/.gitignore new file mode 100644 index 0000000..7b09234 --- /dev/null +++ b/performance-test/download/.gitignore @@ -0,0 +1,2 @@ +*.tar.gz +*.tar.xz diff --git a/performance-test/download/notmuch-email-corpus-0.2.tar.xz.asc b/performance-test/download/notmuch-email-corpus-0.2.tar.xz.asc new file mode 100644 index 0000000..c8b4b3d --- /dev/null +++ b/performance-test/download/notmuch-email-corpus-0.2.tar.xz.asc @@ -0,0 +1,9 @@ +-----BEGIN PGP SIGNATURE----- +Version: GnuPG v1.4.12 (GNU/Linux) + +iJwEAAECAAYFAlCsvx0ACgkQTiiN/0Um85kZAwP9GgOQ22jK8mr5X4pT/mB8EjSH +QbndlxxbRrP0ChTqjBQoD3IsTHjNL7W572BfXb/MNo94R/iIQ7yTHCDVNuwBhvKd +7qgIuW2FUS1uTfJRP5KBNf8JPuin+6wqGe8/+y/iOs+XJSdiYg1ElS49Ntnpg0yl +btImgEcxTxQ2qfzDS1g= +=iuZR +-----END PGP SIGNATURE----- diff --git a/performance-test/notmuch-perf-test b/performance-test/notmuch-perf-test new file mode 100755 index 0000000..1bea345 --- /dev/null +++ b/performance-test/notmuch-perf-test @@ -0,0 +1,25 @@ +#!/usr/bin/env bash + +# Run tests +# +# Copyright (c) 2005 Junio C Hamano +# +# Adapted from a Makefile to a shell script by Carl Worth (2010) + +if [ ${BASH_VERSINFO[0]} -lt 4 ]; then + echo "Error: The notmuch test suite requires a bash version >= 4.0" + echo "due to use of associative arrays within the test suite." + echo "Please try again with a newer bash (or help us fix the" + echo "test suite to be more portable). Thanks." + exit 1 +fi + +cd $(dirname "$0") + +TESTS=" + basic +" + +for test in $TESTS; do + ./$test "$@" +done diff --git a/performance-test/perf-test-lib.sh b/performance-test/perf-test-lib.sh new file mode 100644 index 0000000..11ccc77 --- /dev/null +++ b/performance-test/perf-test-lib.sh @@ -0,0 +1,65 @@ +. ./version.sh + +. ../test/test-lib-common.sh + +set -e + +if ! test -x ../notmuch +then + echo >&2 'You do not seem to have built notmuch yet.' + exit 1 +fi + +add_email_corpus () +{ + rm -rf ${MAIL_DIR} + + arg="" + case "$1" in + --small) + arg="/enron/bailey-s" + ;; + --medium) + arg="/notmuch-archive" + ;; + esac + + if command -v pixz > /dev/null; then + XZ=pixz + else + XZ=xz + fi + + printf "Unpacking corpus\n" + tar --checkpoint=.5000 --extract --strip-components=1 \ + --directory ${TMP_DIRECTORY} \ + --use-compress-program ${XZ} \ + --file ../download/notmuch-email-corpus-${PERFTEST_VERSION}.tar.xz \ + notmuch-email-corpus/mail"$arg" + + printf "\n" +} + +print_header () { + printf " Wall(s)\tUsr(s)\tSys(s)\tRes(K)\tIn\tOut\n" +} + +time_run () { + printf "%-22s" "$1" + if test "$verbose" != "t"; then exec 4>test.output 3>&4; fi + if ! eval >&3 "/usr/bin/time -f '%e\t%U\t%S\t%M\t%I\t%O' $2" ; then + test_failure=$(($test_failure + 1)) + fi +} + +time_done () { + if [ "$test_failure" = "0" ]; then + rm -rf "$remove_tmp" + exit 0 + else + exit 1 + fi +} + +cd -P "$test" || error "Cannot setup test environment" +test_failure=0 diff --git a/performance-test/version.sh b/performance-test/version.sh new file mode 100644 index 0000000..d9270b1 --- /dev/null +++ b/performance-test/version.sh @@ -0,0 +1,3 @@ +# this should be both a valid Makefile fragment and valid POSIX(ish) shell. + +PERFTEST_VERSION=0.2 -- 1.7.10.4 ^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [Patch v4 2/2] test: initial performance testing infrastructure 2012-11-25 15:02 ` [Patch v4 2/2] test: initial performance testing infrastructure david @ 2012-11-25 21:40 ` Austin Clements 2012-11-26 0:05 ` David Bremner 0 siblings, 1 reply; 8+ messages in thread From: Austin Clements @ 2012-11-25 21:40 UTC (permalink / raw) To: david; +Cc: notmuch, David Bremner Quoth david@tethera.net on Nov 25 at 11:02 am: > From: David Bremner <bremner@debian.org> > > This is not near as fancy as as the unit tests, on the theory that > the code should typically be crashing when performance tuning. > Nonetheless, there is plenty of room for improvement. Several more of > the pieces of the test infrastructure (e.g. the option parsing) could > be factored out into test/test-lib-common.sh > --- > Makefile | 3 +- > performance-test/.gitignore | 1 + > performance-test/Makefile | 7 +++ > performance-test/Makefile.local | 32 ++++++++++ > performance-test/README | 50 +++++++++++++++ > performance-test/basic | 15 +++++ > performance-test/download/.gitignore | 2 + > .../download/notmuch-email-corpus-0.2.tar.xz.asc | 9 +++ > performance-test/notmuch-perf-test | 25 ++++++++ > performance-test/perf-test-lib.sh | 65 ++++++++++++++++++++ > performance-test/version.sh | 3 + > 11 files changed, 211 insertions(+), 1 deletion(-) > create mode 100644 performance-test/.gitignore > create mode 100644 performance-test/Makefile > create mode 100644 performance-test/Makefile.local > create mode 100644 performance-test/README > create mode 100755 performance-test/basic > create mode 100644 performance-test/download/.gitignore > create mode 100644 performance-test/download/notmuch-email-corpus-0.2.tar.xz.asc > create mode 100755 performance-test/notmuch-perf-test > create mode 100644 performance-test/perf-test-lib.sh > create mode 100644 performance-test/version.sh > > diff --git a/Makefile b/Makefile > index bb9c316..5decbea 100644 > --- a/Makefile > +++ b/Makefile > @@ -3,7 +3,8 @@ > all: > > # List all subdirectories here. Each contains its own Makefile.local > -subdirs = compat completion emacs lib man parse-time-string util test > +subdirs := compat completion emacs lib man parse-time-string > +subdirs := $(subdirs) performance-test util test += ? > > # We make all targets depend on the Makefiles themselves. > global_deps = Makefile Makefile.config Makefile.local \ > diff --git a/performance-test/.gitignore b/performance-test/.gitignore > new file mode 100644 > index 0000000..53f2697 > --- /dev/null > +++ b/performance-test/.gitignore > @@ -0,0 +1 @@ > +tmp.*/ > diff --git a/performance-test/Makefile b/performance-test/Makefile > new file mode 100644 > index 0000000..de492a7 > --- /dev/null > +++ b/performance-test/Makefile > @@ -0,0 +1,7 @@ > +# See Makefile.local for the list of files to be compiled in this > +# directory. > +all: > + $(MAKE) -C .. all > + > +.DEFAULT: > + $(MAKE) -C .. $@ > diff --git a/performance-test/Makefile.local b/performance-test/Makefile.local > new file mode 100644 > index 0000000..1114ec1 > --- /dev/null > +++ b/performance-test/Makefile.local > @@ -0,0 +1,32 @@ > +# -*- makefile -*- > + > +dir := performance-test > + > +include $(dir)/version.sh > + > +CORPUS_NAME := notmuch-email-corpus-$(PERFTEST_VERSION).tar.xz Would it make sense to split out the different size corpora so a user could, say, only download the small one? > +TXZFILE := ${dir}/download/${CORPUS_NAME} > +SIGFILE := ${TXZFILE}.asc > +TEST_SCRIPT := ${dir}/notmuch-perf-test > +DEFAULT_URL := http://notmuchmail.org/releases/${CORPUS_NAME} > + > +perf-test: setup-perf-test all > + $(TEST_SCRIPT) $(OPTIONS) > + > +.PHONY: download-corpus setup-perf-test > + > +# Note that this intentionally does not depend on download-corpus. > +setup-perf-test: $(TXZFILE) > + gpg --verify $(SIGFILE) > + > +$(TXZFILE): > + @printf "\nPlease download ${TXZFILE}.\n\n" "\nPlease download ${TXZFILE} using\n\n"? > + @printf "\t%% make download-corpus\n\n" > + @echo or see http://notmuchmail.org/corpus for download locations > + @echo > + @false > + > +download-corpus: > + wget -O ${TXZFILE} ${DEFAULT_URL} > + > +CLEAN := $(CLEAN) $(dir)/tmp.* > diff --git a/performance-test/README b/performance-test/README > new file mode 100644 > index 0000000..239d2fb > --- /dev/null > +++ b/performance-test/README > @@ -0,0 +1,50 @@ > +Pre-requisites > +-------------- > + > +In addition to having notmuch, you need: > + > +- gpg > +- gnu tar > +- gnu time > +- xz. Some speedup can be gotten by installing "pixz", but this is > + probably only worthwhile if you are debugging the tests. > + > +Getting set up to run tests: > +---------------------------- > + > +First, you need to get the corpus. > + > +It should work to run > + > + % make download-corpus > + > +In case that fails or is too slow, check > + > + http://notmuchmail.org/corpus > + > +for a list of mirrors. > + > +Running tests > +------------- > + > +The easiest way to run performance tests is to say "make perf-test", (or > +simply run the notmuch-perf-test script). Either command will run all > +available performance tests. > + > +Alternately, you can run a specific subset of tests by simply invoking > +one of the executable scripts in this directory, (such as ./basic). > + > + > +Writing tests > +------------- > + > +Have a look at "basic" for an example. > + > +add_email_corpus takes arguments "--small" and "--medium" for when you > +want smaller corpuses to check. "corpora"? > + > +time_done does the cleanup; comment it out or define "$debug" to leave > +the temporary files around. > + > +Currently there is no option processing (e.g. --debug) in the > +performance tests. > diff --git a/performance-test/basic b/performance-test/basic > new file mode 100755 > index 0000000..9d015ee > --- /dev/null > +++ b/performance-test/basic > @@ -0,0 +1,15 @@ > +#!/bin/bash > + > +. ./perf-test-lib.sh > + > +add_email_corpus > + > +print_header > + > +time_run 'initial notmuch new' 'notmuch new' > +time_run 'second notmuch new' 'notmuch new' > +time_run 'dump *' 'notmuch dump > tags.out' > +time_run 'restore *' 'notmuch restore < tags.out' > +time_run 'tag * +new_tag' "notmuch tag +new_tag '*'" > + > +time_done > diff --git a/performance-test/download/.gitignore b/performance-test/download/.gitignore > new file mode 100644 > index 0000000..7b09234 > --- /dev/null > +++ b/performance-test/download/.gitignore > @@ -0,0 +1,2 @@ > +*.tar.gz > +*.tar.xz > diff --git a/performance-test/download/notmuch-email-corpus-0.2.tar.xz.asc b/performance-test/download/notmuch-email-corpus-0.2.tar.xz.asc > new file mode 100644 > index 0000000..c8b4b3d > --- /dev/null > +++ b/performance-test/download/notmuch-email-corpus-0.2.tar.xz.asc > @@ -0,0 +1,9 @@ > +-----BEGIN PGP SIGNATURE----- > +Version: GnuPG v1.4.12 (GNU/Linux) > + > +iJwEAAECAAYFAlCsvx0ACgkQTiiN/0Um85kZAwP9GgOQ22jK8mr5X4pT/mB8EjSH > +QbndlxxbRrP0ChTqjBQoD3IsTHjNL7W572BfXb/MNo94R/iIQ7yTHCDVNuwBhvKd > +7qgIuW2FUS1uTfJRP5KBNf8JPuin+6wqGe8/+y/iOs+XJSdiYg1ElS49Ntnpg0yl > +btImgEcxTxQ2qfzDS1g= > +=iuZR > +-----END PGP SIGNATURE----- > diff --git a/performance-test/notmuch-perf-test b/performance-test/notmuch-perf-test > new file mode 100755 > index 0000000..1bea345 > --- /dev/null > +++ b/performance-test/notmuch-perf-test > @@ -0,0 +1,25 @@ > +#!/usr/bin/env bash > + > +# Run tests > +# > +# Copyright (c) 2005 Junio C Hamano > +# > +# Adapted from a Makefile to a shell script by Carl Worth (2010) > + > +if [ ${BASH_VERSINFO[0]} -lt 4 ]; then > + echo "Error: The notmuch test suite requires a bash version >= 4.0" > + echo "due to use of associative arrays within the test suite." > + echo "Please try again with a newer bash (or help us fix the" > + echo "test suite to be more portable). Thanks." > + exit 1 > +fi > + > +cd $(dirname "$0") > + > +TESTS=" > + basic > +" > + > +for test in $TESTS; do > + ./$test "$@" > +done > diff --git a/performance-test/perf-test-lib.sh b/performance-test/perf-test-lib.sh > new file mode 100644 > index 0000000..11ccc77 > --- /dev/null > +++ b/performance-test/perf-test-lib.sh > @@ -0,0 +1,65 @@ > +. ./version.sh > + > +. ../test/test-lib-common.sh > + > +set -e > + > +if ! test -x ../notmuch > +then > + echo >&2 'You do not seem to have built notmuch yet.' > + exit 1 > +fi > + > +add_email_corpus () > +{ > + rm -rf ${MAIL_DIR} > + > + arg="" > + case "$1" in > + --small) > + arg="/enron/bailey-s" > + ;; > + --medium) > + arg="/notmuch-archive" > + ;; > + esac > + > + if command -v pixz > /dev/null; then > + XZ=pixz > + else > + XZ=xz > + fi > + > + printf "Unpacking corpus\n" > + tar --checkpoint=.5000 --extract --strip-components=1 \ > + --directory ${TMP_DIRECTORY} \ > + --use-compress-program ${XZ} \ > + --file ../download/notmuch-email-corpus-${PERFTEST_VERSION}.tar.xz \ > + notmuch-email-corpus/mail"$arg" I'm a bit confused by this. What happens if you don't specify --small or --medium? Is the "large"/default corpus just the combined small and medium corpora? Would be worth a comment, at least. This probably doesn't matter now, but I wonder if we want to unpack on first use to somewhere not test-specific and then cp -rl the corpus into the test directory. I haven't tried unpacking the corpus yet, but if you're running tests repeatedly to compare results, or running more than one performance test, it seems like a full decompress and unpack could get onerous. > + > + printf "\n" > +} > + > +print_header () { > + printf " Wall(s)\tUsr(s)\tSys(s)\tRes(K)\tIn\tOut\n" Should the header include which corpus size is in use? This would be important when emailing around performance results. > +} > + > +time_run () { > + printf "%-22s" "$1" > + if test "$verbose" != "t"; then exec 4>test.output 3>&4; fi > + if ! eval >&3 "/usr/bin/time -f '%e\t%U\t%S\t%M\t%I\t%O' $2" ; then > + test_failure=$(($test_failure + 1)) > + fi > +} > + > +time_done () { > + if [ "$test_failure" = "0" ]; then > + rm -rf "$remove_tmp" > + exit 0 > + else > + exit 1 > + fi > +} > + > +cd -P "$test" || error "Cannot setup test environment" > +test_failure=0 > diff --git a/performance-test/version.sh b/performance-test/version.sh > new file mode 100644 > index 0000000..d9270b1 > --- /dev/null > +++ b/performance-test/version.sh > @@ -0,0 +1,3 @@ > +# this should be both a valid Makefile fragment and valid POSIX(ish) shell. > + > +PERFTEST_VERSION=0.2 ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [Patch v4 2/2] test: initial performance testing infrastructure 2012-11-25 21:40 ` Austin Clements @ 2012-11-26 0:05 ` David Bremner 2012-11-26 3:29 ` Austin Clements 0 siblings, 1 reply; 8+ messages in thread From: David Bremner @ 2012-11-26 0:05 UTC (permalink / raw) To: Austin Clements; +Cc: notmuch Austin Clements <amdragon@MIT.EDU> writes: >> +subdirs := compat completion emacs lib man parse-time-string >> +subdirs := $(subdirs) performance-test util test > += ? > Sure. >> +CORPUS_NAME := notmuch-email-corpus-$(PERFTEST_VERSION).tar.xz > > Would it make sense to split out the different size corpora so a user > could, say, only download the small one? Currently the choice of test is local to given test file; one doing something particularly intense (or just lots of repetitions) might want to only use a subset. So I'm not sure if separate downloading of smaller corpora makes sense. This is all hypothetical at the moment, since the one test file uses the full corpus. > "\nPlease download ${TXZFILE} using\n\n"? OK >> +add_email_corpus takes arguments "--small" and "--medium" for when you >> +want smaller corpuses to check. > > "corpora"? reworded to say ,---- | add_email_corpus takes arguments "--small" and "--medium" for when you | want smaller subsets of the corpus to check. `---- > > I'm a bit confused by this. What happens if you don't specify --small > or --medium? Is the "large"/default corpus just the combined small > and medium corpora? Would be worth a comment, at least. Hopefully the README makes this clear(er) now? > This probably doesn't matter now, but I wonder if we want to unpack on > first use to somewhere not test-specific and then cp -rl the corpus > into the test directory. I haven't tried unpacking the corpus yet, > but if you're running tests repeatedly to compare results, or running > more than one performance test, it seems like a full decompress and > unpack could get onerous. Hmm. On my machine it is 10s for the copy versus 45s for a full unpack. For some reason I tested with "cp -a" which is incredibly slow, so I thought there was no loss. For comparison the basic test takes about 10 minutes on the same machine. In any case this can wait until we have a second test file and a second call to add_mail_corpus, adding caching now would not help. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [Patch v4 2/2] test: initial performance testing infrastructure 2012-11-26 0:05 ` David Bremner @ 2012-11-26 3:29 ` Austin Clements 2012-11-26 12:44 ` David Bremner 0 siblings, 1 reply; 8+ messages in thread From: Austin Clements @ 2012-11-26 3:29 UTC (permalink / raw) To: David Bremner; +Cc: notmuch Quoth David Bremner on Nov 25 at 8:05 pm: > Austin Clements <amdragon@MIT.EDU> writes: > >> +add_email_corpus takes arguments "--small" and "--medium" for when you > >> +want smaller corpuses to check. > > > > "corpora"? > > reworded to say > > ,---- > | add_email_corpus takes arguments "--small" and "--medium" for when you > | want smaller subsets of the corpus to check. > `---- That's clearer. > > > > I'm a bit confused by this. What happens if you don't specify --small > > or --medium? Is the "large"/default corpus just the combined small > > and medium corpora? Would be worth a comment, at least. > > Hopefully the README makes this clear(er) now? The README definitely helps. Might still be worth a comment in the code since it took me some thinking to realize it would do something reasonable when given no argument. Perhaps above the initial assignment of arg, # With no argument, use the entire (combined) corpus to acknowledge that this is a legitimate and intentional code path? > > This probably doesn't matter now, but I wonder if we want to unpack on > > first use to somewhere not test-specific and then cp -rl the corpus > > into the test directory. I haven't tried unpacking the corpus yet, > > but if you're running tests repeatedly to compare results, or running > > more than one performance test, it seems like a full decompress and > > unpack could get onerous. > > Hmm. On my machine it is 10s for the copy versus 45s for a full > unpack. For some reason I tested with "cp -a" which is incredibly slow, > so I thought there was no loss. For comparison the basic test takes > about 10 minutes on the same machine. > > In any case this can wait until we have a second test file and a second > call to add_mail_corpus, adding caching now would not help. It would help (a little) if you run basic multiple times. I think it's completely reasonable to leave it as is for now and see if caching would help down the road. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [Patch v4 2/2] test: initial performance testing infrastructure 2012-11-26 3:29 ` Austin Clements @ 2012-11-26 12:44 ` David Bremner 0 siblings, 0 replies; 8+ messages in thread From: David Bremner @ 2012-11-26 12:44 UTC (permalink / raw) To: Austin Clements; +Cc: notmuch Austin Clements <amdragon@MIT.EDU> writes: > > The README definitely helps. Might still be worth a comment in the > code since it took me some thinking to realize it would do something > reasonable when given no argument. Perhaps above the initial > assignment of arg, I decided to fix the code, rather than comment it ;). Now every code path makes an explicit, non-empty assignment to $arg I have pushed that slightly updated version of the patches. d ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: v4 of Performance tests 2012-11-25 15:02 v4 of Performance tests david 2012-11-25 15:02 ` [Patch v4 1/2] test: factor out part of test-lib.sh into test-lib-common.sh david 2012-11-25 15:02 ` [Patch v4 2/2] test: initial performance testing infrastructure david @ 2012-11-25 21:19 ` Tomi Ollila 2 siblings, 0 replies; 8+ messages in thread From: Tomi Ollila @ 2012-11-25 21:19 UTC (permalink / raw) To: david, notmuch On Sun, Nov 25 2012, david@tethera.net wrote: > This version adds an optional makefile target to download the corpus. > > Tomi reviewed it on IRC and didn't complain. +1 Tomi ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2012-11-26 12:44 UTC | newest] Thread overview: 8+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2012-11-25 15:02 v4 of Performance tests david 2012-11-25 15:02 ` [Patch v4 1/2] test: factor out part of test-lib.sh into test-lib-common.sh david 2012-11-25 15:02 ` [Patch v4 2/2] test: initial performance testing infrastructure david 2012-11-25 21:40 ` Austin Clements 2012-11-26 0:05 ` David Bremner 2012-11-26 3:29 ` Austin Clements 2012-11-26 12:44 ` David Bremner 2012-11-25 21:19 ` v4 of Performance tests Tomi Ollila
Code repositories for project(s) associated with this public inbox https://yhetil.org/notmuch.git/ This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).