[PATCH 0/4] set test prereqs (Emacs, GDB, GPG) v3

[PATCH 0/4] set test prereqs (Emacs, GDB, GPG) v3

[Pieter Praet]

Rebased the "set test prereqs (Emacs, GDB, GPG)" [1] patch series
to Jameson's release-candidate/0.6 branch (0c0b4172).

The GDB dependency for the atomicity tests was dropped since
Austin has already included this in his amdragon/atomic-new-v4
branch [2], which has been included in release-candidate/0.6.

Also, some crypto tests which depend not only on GPG but Emacs as
well have been updated accordingly.

Jameson, could this be merged in? It helps prevent false
negatives in the absence of Emacs/GnuPG/GDB.


[1] id:"1305274636-19975-1-git-send-email-pieter@praet.org"
[2] id:"BANLkTikEhvhroyJNRN9RLwBNz_DbOLfzBQ@mail.gmail.com"

[PATCH 1/4] test: add 'GnuPG' prereq to dependent 'crypto' tests

[Pieter Praet]

Adds a new test that checks for the presence of 'gpg',
and adds that test as a prereq to all subsequent tests
that rely on GnuPG.

This causes tests with unmet dependencies to be skipped.

Signed-off-by: Pieter Praet <pieter@praet.org>
---
 test/crypto |   33 +++++++++++++++++++--------------
 1 files changed, 19 insertions(+), 14 deletions(-)

diff --git a/test/crypto b/test/crypto
index 8e92016..59f7989 100755
--- a/test/crypto
+++ b/test/crypto
@@ -7,6 +7,11 @@
 test_description='PGP/MIME signature verification and decryption'
 . ./test-lib.sh
 
+# GnuPG is a prereq.
+test_expect_success "prereq: GnuPG is present" "which gpg" \
+    && test_set_prereq GPG
+
+
 add_gnupg_home ()
 {
     local output
@@ -31,7 +36,7 @@ FINGERPRINT=$(gpg --no-tty --list-secret-keys --with-colons --fingerprint | grep
 # although I can't figure out why
 add_email_corpus
 
-test_expect_success 'emacs delivery of signed message' \
+test_expect_success GPG 'emacs delivery of signed message' \
 'emacs_deliver_message \
     "test signed message 001" \
     "This is a test signed message." \
@@ -64,7 +69,7 @@ expected='[[[{"id": "XXXXX",
  {"id": 3,
  "content-type": "application/pgp-signature"}]}]},
  []]]]'
-test_expect_equal \
+test_expect_equal GPG \
     "$output" \
     "$expected"
 
@@ -99,7 +104,7 @@ expected='[[[{"id": "XXXXX",
  {"id": 3,
  "content-type": "application/pgp-signature"}]}]},
  []]]]'
-test_expect_equal \
+test_expect_equal GPG \
     "$output" \
     "$expected"
 
@@ -132,7 +137,7 @@ expected='[[[{"id": "XXXXX",
  {"id": 3,
  "content-type": "application/pgp-signature"}]}]},
  []]]]'
-test_expect_equal \
+test_expect_equal GPG \
     "$output" \
     "$expected"
 mv "${GNUPGHOME}"{.bak,}
@@ -141,7 +146,7 @@ mv "${GNUPGHOME}"{.bak,}
 cat <<EOF >TESTATTACHMENT
 This is a test file.
 EOF
-test_expect_success 'emacs delivery of encrypted message with attachment' \
+test_expect_success GPG 'emacs delivery of encrypted message with attachment' \
 'emacs_deliver_message \
     "test encrypted message 001" \
     "This is a test encrypted message.\n" \
@@ -176,7 +181,7 @@ Non-text part: application/octet-stream
 \fpart}
 \fbody}
 \fmessage}'
-test_expect_equal \
+test_expect_equal GPG \
     "$output" \
     "$expected"
 
@@ -211,7 +216,7 @@ expected='[[[{"id": "XXXXX",
  "content-type": "application/octet-stream",
  "filename": "TESTATTACHMENT"}]}]}]},
  []]]]'
-test_expect_equal \
+test_expect_equal GPG \
     "$output" \
     "$expected"
 
@@ -222,7 +227,7 @@ output=$(notmuch show --format=json --part=4 --decrypt subject:"test encrypted m
 expected='{"id": 4,
  "content-type": "text/plain",
  "content": "This is a test encrypted message.\n"}'
-test_expect_equal \
+test_expect_equal GPG \
     "$output" \
     "$expected"
 
@@ -232,7 +237,7 @@ notmuch show \
     --part=5 \
     --decrypt \
     subject:"test encrypted message 001" >OUTPUT
-test_expect_equal_file OUTPUT TESTATTACHMENT
+test_expect_equal_file GPG OUTPUT TESTATTACHMENT
 
 test_begin_subtest "decryption failure with missing key"
 mv "${GNUPGHOME}"{,.bak}
@@ -259,12 +264,12 @@ expected='[[[{"id": "XXXXX",
  {"id": 3,
  "content-type": "application/octet-stream"}]}]},
  []]]]'
-test_expect_equal \
+test_expect_equal GPG \
     "$output" \
     "$expected"
 mv "${GNUPGHOME}"{.bak,}
 
-test_expect_success 'emacs delivery of encrypted + signed message' \
+test_expect_success GPG 'emacs delivery of encrypted + signed message' \
 'emacs_deliver_message \
     "test encrypted message 002" \
     "This is another test encrypted message.\n" \
@@ -299,7 +304,7 @@ expected='[[[{"id": "XXXXX",
  "content-type": "text/plain",
  "content": "This is another test encrypted message.\n"}]}]},
  []]]]'
-test_expect_equal \
+test_expect_equal GPG \
     "$output" \
     "$expected"
 
@@ -313,7 +318,7 @@ On 01 Jan 2000 12:00:00 -0000, Notmuch Test Suite <test_suite@notmuchmail.org> w
 Non-text part: multipart/encrypted
 Non-text part: application/pgp-encrypted
 > This is another test encrypted message.'
-test_expect_equal \
+test_expect_equal GPG \
     "$output" \
     "$expected"
 
@@ -354,7 +359,7 @@ expected='[[[{"id": "XXXXX",
  {"id": 3,
  "content-type": "application/pgp-signature"}]}]},
  []]]]'
-test_expect_equal \
+test_expect_equal GPG \
     "$output" \
     "$expected"
 
-- 
1.7.4.1

[PATCH 2/4] test: add 'Emacs' prereq to dependent 'crypto' tests

[Pieter Praet]

Adds a new test that checks for the presence of 'emacs',
and adds that test as a prereq to all subsequent tests
that rely on Emacs.

This causes tests with unmet dependencies to be skipped.

Signed-off-by: Pieter Praet <pieter@praet.org>
---
 test/crypto |   17 ++++++++++++++---
 1 files changed, 14 insertions(+), 3 deletions(-)

diff --git a/test/crypto b/test/crypto
index 59f7989..c69307d 100755
--- a/test/crypto
+++ b/test/crypto
@@ -7,10 +7,21 @@
 test_description='PGP/MIME signature verification and decryption'
 . ./test-lib.sh
 
+# Emacs is a prereq.
+test_expect_success "prereq: Emacs is present" "which emacs" \
+    && test_set_prereq EMACS
+
 # GnuPG is a prereq.
 test_expect_success "prereq: GnuPG is present" "which gpg" \
     && test_set_prereq GPG
 
+# Some tests have multiple prereqs, but the test_expect_* functions
+# accept only a single argument as prereq tag, and using test_have_prereq
+# in and around tests causes various errors for me, so a dirty workaround
+# will have to do for the time being.
+test_have_prereq EMACS && test_have_prereq GPG \
+    && test_set_prereq EMACS+GPG
+
 
 add_gnupg_home ()
 {
@@ -36,7 +47,7 @@ FINGERPRINT=$(gpg --no-tty --list-secret-keys --with-colons --fingerprint | grep
 # although I can't figure out why
 add_email_corpus
 
-test_expect_success GPG 'emacs delivery of signed message' \
+test_expect_success EMACS+GPG 'emacs delivery of signed message' \
 'emacs_deliver_message \
     "test signed message 001" \
     "This is a test signed message." \
@@ -146,7 +157,7 @@ mv "${GNUPGHOME}"{.bak,}
 cat <<EOF >TESTATTACHMENT
 This is a test file.
 EOF
-test_expect_success GPG 'emacs delivery of encrypted message with attachment' \
+test_expect_success EMACS+GPG 'emacs delivery of encrypted message with attachment' \
 'emacs_deliver_message \
     "test encrypted message 001" \
     "This is a test encrypted message.\n" \
@@ -269,7 +280,7 @@ test_expect_equal GPG \
     "$expected"
 mv "${GNUPGHOME}"{.bak,}
 
-test_expect_success GPG 'emacs delivery of encrypted + signed message' \
+test_expect_success EMACS+GPG 'emacs delivery of encrypted + signed message' \
 'emacs_deliver_message \
     "test encrypted message 002" \
     "This is another test encrypted message.\n" \
-- 
1.7.4.1

[PATCH 3/4] test: add 'Emacs' prereq to dependent 'emacs' tests

[Pieter Praet]

Adds a new test that checks for the presence of 'emacs',
and adds that test as a prereq to all subsequent tests
that rely on Emacs.

This causes tests with unmet dependencies to be skipped.

Signed-off-by: Pieter Praet <pieter@praet.org>
---
 test/emacs |   43 ++++++++++++++++++++++++-------------------
 1 files changed, 24 insertions(+), 19 deletions(-)

diff --git a/test/emacs b/test/emacs
index da8ba68..10138a6 100755
--- a/test/emacs
+++ b/test/emacs
@@ -2,65 +2,70 @@
 test_description="emacs interface"
 . test-lib.sh
 
+# Emacs is a prereq.
+test_expect_success "prereq: Emacs is present" "which emacs" \
+    && test_set_prereq EMACS
+
+
 EXPECTED=../emacs.expected-output
 
 add_email_corpus
 
 test_begin_subtest "Basic notmuch-hello view in emacs"
 test_emacs '(notmuch-hello) (princ (buffer-string))' >OUTPUT
-test_expect_equal_file OUTPUT $EXPECTED/notmuch-hello
+test_expect_equal_file EMACS OUTPUT $EXPECTED/notmuch-hello
 
 test_begin_subtest "Saved search with 0 results"
 test_emacs '(setq notmuch-show-empty-saved-searches t) (setq notmuch-saved-searches '\''(("inbox" . "tag:inbox") ("unread" . "tag:unread") ("empty" . "tag:doesnotexist"))) (notmuch-hello) (princ (buffer-string))' >OUTPUT
-test_expect_equal_file OUTPUT $EXPECTED/notmuch-hello-with-empty
+test_expect_equal_file EMACS OUTPUT $EXPECTED/notmuch-hello-with-empty
 
 test_begin_subtest "No saved searches displayed (all with 0 results)"
 test_emacs '(setq notmuch-saved-searches '\''(("empty" . "tag:doesnotexist"))) (notmuch-hello) (princ (buffer-string))' >OUTPUT
-test_expect_equal_file OUTPUT $EXPECTED/notmuch-hello-no-saved-searches
+test_expect_equal_file EMACS OUTPUT $EXPECTED/notmuch-hello-no-saved-searches
 
 test_begin_subtest "Basic notmuch-search view in emacs"
 test_emacs '(notmuch-search "tag:inbox") (notmuch-test-wait) (princ (buffer-string))' >OUTPUT
-test_expect_equal_file OUTPUT $EXPECTED/notmuch-search-tag-inbox
+test_expect_equal_file EMACS OUTPUT $EXPECTED/notmuch-search-tag-inbox
 
 test_begin_subtest "Navigation of notmuch-hello to search results"
 test_emacs '(notmuch-hello) (goto-char (point-min)) (re-search-forward "inbox") (widget-button-press (point)) (notmuch-test-wait) (princ (buffer-string))' >OUTPUT
-test_expect_equal_file OUTPUT $EXPECTED/notmuch-hello-view-inbox
+test_expect_equal_file EMACS OUTPUT $EXPECTED/notmuch-hello-view-inbox
 
 test_begin_subtest "Basic notmuch-show view in emacs"
 maildir_storage_thread=$(notmuch search --output=threads id:20091117190054.GU3165@dottiness.seas.harvard.edu)
 test_emacs "(notmuch-show \"$maildir_storage_thread\") (princ (buffer-string))" >OUTPUT
-test_expect_equal_file OUTPUT $EXPECTED/notmuch-show-thread-maildir-storage
+test_expect_equal_file EMACS OUTPUT $EXPECTED/notmuch-show-thread-maildir-storage
 
 test_begin_subtest "Navigation of notmuch-search to thread view"
 test_emacs '(notmuch-search "tag:inbox") (notmuch-test-wait) (goto-char (point-min)) (re-search-forward "Working with Maildir") (notmuch-search-show-thread) (notmuch-test-wait) (princ (buffer-string))' >OUTPUT
-test_expect_equal_file OUTPUT $EXPECTED/notmuch-show-thread-maildir-storage
+test_expect_equal_file EMACS OUTPUT $EXPECTED/notmuch-show-thread-maildir-storage
 
 test_begin_subtest "Add tag from search view"
 os_x_darwin_thread=$(notmuch search --output=threads id:ddd65cda0911171950o4eea4389v86de9525e46052d3@mail.gmail.com)
 test_emacs "(notmuch-search \"$os_x_darwin_thread\") (notmuch-test-wait) (notmuch-search-add-tag \"tag-from-search-view\")"
 output=$(notmuch search $os_x_darwin_thread | notmuch_search_sanitize)
-test_expect_equal "$output" "thread:XXX   2009-11-18 [4/4] Jjgod Jiang, Alexander Botero-Lowry; [notmuch] Mac OS X/Darwin compatibility issues (inbox tag-from-search-view unread)"
+test_expect_equal EMACS "$output" "thread:XXX   2009-11-18 [4/4] Jjgod Jiang, Alexander Botero-Lowry; [notmuch] Mac OS X/Darwin compatibility issues (inbox tag-from-search-view unread)"
 
 test_begin_subtest "Remove tag from search view"
 test_emacs "(notmuch-search \"$os_x_darwin_thread\") (notmuch-test-wait) (notmuch-search-remove-tag \"tag-from-search-view\")"
 output=$(notmuch search $os_x_darwin_thread | notmuch_search_sanitize)
-test_expect_equal "$output" "thread:XXX   2009-11-18 [4/4] Jjgod Jiang, Alexander Botero-Lowry; [notmuch] Mac OS X/Darwin compatibility issues (inbox unread)"
+test_expect_equal EMACS "$output" "thread:XXX   2009-11-18 [4/4] Jjgod Jiang, Alexander Botero-Lowry; [notmuch] Mac OS X/Darwin compatibility issues (inbox unread)"
 
 test_begin_subtest "Add tag from notmuch-show view"
 test_emacs "(notmuch-show \"$os_x_darwin_thread\") (notmuch-show-add-tag \"tag-from-show-view\")"
 output=$(notmuch search $os_x_darwin_thread | notmuch_search_sanitize)
-test_expect_equal "$output" "thread:XXX   2009-11-18 [4/4] Jjgod Jiang, Alexander Botero-Lowry; [notmuch] Mac OS X/Darwin compatibility issues (inbox tag-from-show-view unread)"
+test_expect_equal EMACS "$output" "thread:XXX   2009-11-18 [4/4] Jjgod Jiang, Alexander Botero-Lowry; [notmuch] Mac OS X/Darwin compatibility issues (inbox tag-from-show-view unread)"
 
 test_begin_subtest "Remove tag from notmuch-show view"
 test_emacs "(notmuch-show \"$os_x_darwin_thread\") (notmuch-show-remove-tag \"tag-from-show-view\")"
 output=$(notmuch search $os_x_darwin_thread | notmuch_search_sanitize)
-test_expect_equal "$output" "thread:XXX   2009-11-18 [4/4] Jjgod Jiang, Alexander Botero-Lowry; [notmuch] Mac OS X/Darwin compatibility issues (inbox unread)"
+test_expect_equal EMACS "$output" "thread:XXX   2009-11-18 [4/4] Jjgod Jiang, Alexander Botero-Lowry; [notmuch] Mac OS X/Darwin compatibility issues (inbox unread)"
 
 test_begin_subtest "Message with .. in Message-Id:"
 add_message [id]=123..456@example '[subject]="Message with .. in Message-Id"'
 test_emacs '(notmuch-search "id:\"123..456@example\"") (notmuch-test-wait) (notmuch-search-add-tag "search-add") (notmuch-search-add-tag "search-remove") (notmuch-search-remove-tag "search-remove") (notmuch-show "id:\"123..456@example\"") (notmuch-test-wait) (notmuch-show-add-tag "show-add") (notmuch-show-add-tag "show-remove") (notmuch-show-remove-tag "show-remove")'
 output=$(notmuch search 'id:"123..456@example"' | notmuch_search_sanitize)
-test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Message with .. in Message-Id (inbox search-add show-add)"
+test_expect_equal EMACS "$output" "thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Message with .. in Message-Id (inbox search-add show-add)"
 
 test_begin_subtest "Sending a message via (fake) SMTP"
 
@@ -89,12 +94,12 @@ Content-Type: text/plain; charset=us-ascii
 
 This is a test that messages are sent via SMTP
 EOF
-test_expect_equal_file OUTPUT EXPECTED
+test_expect_equal_file EMACS OUTPUT EXPECTED
 
 test_begin_subtest "Verify that sent messages are saved/searchable (via FCC)"
 notmuch new > /dev/null
 output=$(notmuch search 'subject:"testing message sent via SMTP"' | notmuch_search_sanitize)
-test_expect_equal "$output" "thread:XXX   1974-03-29 [1/1] Notmuch Test Suite; Testing message sent via SMTP (inbox)"
+test_expect_equal EMACS "$output" "thread:XXX   1974-03-29 [1/1] Notmuch Test Suite; Testing message sent via SMTP (inbox)"
 
 test_begin_subtest "Reply within emacs"
 # We sed away everything before the ^From in the output to avoid getting
@@ -110,22 +115,22 @@ Fcc: $(pwd)/mail/sent
 On Fri, 29 Mar 1974 10:00:00 -0000, Notmuch Test Suite <test_suite@notmuchmail.org> wrote:
 > This is a test that messages are sent via SMTP
 EOF
-test_expect_equal_file OUTPUT EXPECTED
+test_expect_equal_file EMACS OUTPUT EXPECTED
 
 test_begin_subtest "Save attachment from within emacs using notmuch-show-save-attachments"
 # save as archive to test that Emacs does not re-compress .gz
 echo ./attachment1.gz | test_emacs '(notmuch-show "id:cf0c4d610911171136h1713aa59w9cf9aa31f052ad0a@mail.gmail.com") (notmuch-show-save-attachments)' > /dev/null 2>&1
-test_expect_equal_file "$EXPECTED/attachment" attachment1.gz
+test_expect_equal_file EMACS "$EXPECTED/attachment" attachment1.gz
 
 test_begin_subtest "Save attachment from within emacs using notmuch-show-save-part"
 # save as archive to test that Emacs does not re-compress .gz
 echo ./attachment2.gz | test_emacs '(notmuch-show-save-part "id:cf0c4d610911171136h1713aa59w9cf9aa31f052ad0a@mail.gmail.com" 5)' > /dev/null 2>&1
-test_expect_equal_file "$EXPECTED/attachment" attachment2.gz
+test_expect_equal_file EMACS "$EXPECTED/attachment" attachment2.gz
 
 test_begin_subtest "View raw message within emacs"
 first_line=$(head -n1 $EXPECTED/raw-message-cf0c4d-52ad0a)
 test_emacs '(notmuch-show "id:cf0c4d610911171136h1713aa59w9cf9aa31f052ad0a@mail.gmail.com") (notmuch-show-view-raw-message) (princ (buffer-string))' | sed -ne "/$first_line/,\$ p" >OUTPUT
-test_expect_equal_file OUTPUT $EXPECTED/raw-message-cf0c4d-52ad0a
+test_expect_equal_file EMACS OUTPUT $EXPECTED/raw-message-cf0c4d-52ad0a
 
 test_begin_subtest "Hiding/showing signature in notmuch-show view"
 maildir_storage_thread=$(notmuch search --output=threads id:20091117190054.GU3165@dottiness.seas.harvard.edu)
@@ -135,6 +140,6 @@ test_emacs "(notmuch-show \"$maildir_storage_thread\")
 	    (search-backward \"Click/Enter to hide.\")
 	    (button-activate (button-at (point)))
 	    (princ (buffer-string))" >OUTPUT
-test_expect_equal_file OUTPUT $EXPECTED/notmuch-show-thread-maildir-storage
+test_expect_equal_file EMACS OUTPUT $EXPECTED/notmuch-show-thread-maildir-storage
 
 test_done
-- 
1.7.4.1

[PATCH 4/4] test: add 'Emacs' prereq to dependent 'emacs-large-search-buffer' tests

[Pieter Praet]

Adds a new test that checks for the presence of 'emacs',
and adds that test as a prereq to all subsequent tests
that rely on Emacs.

This causes tests with unmet dependencies to be skipped.

Signed-off-by: Pieter Praet <pieter@praet.org>
---
 test/emacs-large-search-buffer |    9 +++++++--
 1 files changed, 7 insertions(+), 2 deletions(-)

diff --git a/test/emacs-large-search-buffer b/test/emacs-large-search-buffer
index c78ce33..063f121 100755
--- a/test/emacs-large-search-buffer
+++ b/test/emacs-large-search-buffer
@@ -2,6 +2,11 @@
 test_description="Emacs with large search results buffer"
 . test-lib.sh
 
+# Emacs is a prereq.
+test_expect_success "prereq: Emacs is present" "which emacs" \
+    && test_set_prereq EMACS
+
+
 x=xxxxxxxxxx # 10
 x=$x$x$x$x$x$x$x$x$x$x # 100
 x=$x$x$x$x$x$x$x$x$x # 900
@@ -23,11 +28,11 @@ expected="$(notmuch search '*' | sed -e 's/^thread:[0-9a-f]*  //' -e 's/;//' -e
 End of search results."
 
 output=$(test_emacs '(notmuch-search "*") (notmuch-test-wait) (princ (buffer-string))' | sed -e s',  *, ,g' -e 's/xxx*/[BLOB]/g')
-test_expect_equal "$output" "$expected"
+test_expect_equal EMACS "$output" "$expected"
 
 test_begin_subtest "Ensure that emacs doesn't drop error messages"
 output=$(test_emacs '(notmuch-search "--this-option-does-not-exist") (notmuch-test-wait) (princ (buffer-string))')
-test_expect_equal "$output" "Error: Unexpected output from notmuch search:
+test_expect_equal EMACS "$output" "Error: Unexpected output from notmuch search:
 Unrecognized option: --this-option-does-not-exist
 End of search results. (process returned 1)"
 
-- 
1.7.4.1

Re: [PATCH 0/4] set test prereqs (Emacs, GDB, GPG) v3

[Austin Clements]

This looks good to me, and should be merged sooner rather than later
because it touches a lot of lines.

If atomic-new-v4 doesn't happen to get merged before this, then
id:"1305206080-17461-1-git-send-email-amdragon@mit.edu" and
id:"1305206110-17511-1-git-send-email-amdragon@mit.edu" should get
merged, since they fix test-lib problems with dependencies (they're
included in the atomic-new-v4 branch because that uses dependencies,
but aren't related to atomicity).

Quoth Pieter Praet on Jun 02 at  2:03 pm:
> Rebased the "set test prereqs (Emacs, GDB, GPG)" [1] patch series
> to Jameson's release-candidate/0.6 branch (0c0b4172).
> 
> The GDB dependency for the atomicity tests was dropped since
> Austin has already included this in his amdragon/atomic-new-v4
> branch [2], which has been included in release-candidate/0.6.
> 
> Also, some crypto tests which depend not only on GPG but Emacs as
> well have been updated accordingly.
> 
> Jameson, could this be merged in? It helps prevent false
> negatives in the absence of Emacs/GnuPG/GDB.
> 
> 
> [1] id:"1305274636-19975-1-git-send-email-pieter@praet.org"
> [2] id:"BANLkTikEhvhroyJNRN9RLwBNz_DbOLfzBQ@mail.gmail.com"