unofficial mirror of notmuch@notmuchmail.org
 help / color / mirror / code / Atom feed
* [PATCH 01/10] test: fix argument order of test_expect_equal_file in few tests
@ 2011-06-28  4:45 Dmitry Kurochkin
  2011-06-28  4:45 ` [PATCH 02/10] test: do not set `message-signature' in test_emacs Dmitry Kurochkin
                   ` (8 more replies)
  0 siblings, 9 replies; 22+ messages in thread
From: Dmitry Kurochkin @ 2011-06-28  4:45 UTC (permalink / raw)
  To: notmuch

Few Emacs tests had test_expect_equal_file arguments in the wrong
order: the first argument should be the test output and the
second one should be the expected.
---
 test/emacs |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/test/emacs b/test/emacs
index 6f82b08..f4ff3f1 100755
--- a/test/emacs
+++ b/test/emacs
@@ -198,12 +198,12 @@ test_expect_equal_file OUTPUT EXPECTED
 test_begin_subtest "Save attachment from within emacs using notmuch-show-save-attachments"
 # save as archive to test that Emacs does not re-compress .gz
 echo ./attachment1.gz | test_emacs '(notmuch-show "id:cf0c4d610911171136h1713aa59w9cf9aa31f052ad0a@mail.gmail.com") (notmuch-show-save-attachments)' > /dev/null 2>&1
-test_expect_equal_file "$EXPECTED/attachment" attachment1.gz
+test_expect_equal_file attachment1.gz "$EXPECTED/attachment"
 
 test_begin_subtest "Save attachment from within emacs using notmuch-show-save-part"
 # save as archive to test that Emacs does not re-compress .gz
 echo ./attachment2.gz | test_emacs '(notmuch-show-save-part "id:cf0c4d610911171136h1713aa59w9cf9aa31f052ad0a@mail.gmail.com" 5)' > /dev/null 2>&1
-test_expect_equal_file "$EXPECTED/attachment" attachment2.gz
+test_expect_equal_file attachment2.gz "$EXPECTED/attachment"
 
 test_begin_subtest "View raw message within emacs"
 first_line=$(head -n1 $EXPECTED/raw-message-cf0c4d-52ad0a)
-- 
1.7.5.4

^ permalink raw reply related	[flat|nested] 22+ messages in thread
* [PATCH 02/10] test: do not set `message-signature' in test_emacs
  2011-06-28  4:45 [PATCH 01/10] test: fix argument order of test_expect_equal_file in few tests Dmitry Kurochkin
@ 2011-06-28  4:45 ` Dmitry Kurochkin
  2011-06-28  4:45 ` [PATCH 03/10] test: cleanup test_emacs Dmitry Kurochkin
                   ` (7 subsequent siblings)
  8 siblings, 0 replies; 22+ messages in thread
From: Dmitry Kurochkin @ 2011-06-28  4:45 UTC (permalink / raw)
  To: notmuch

It is no longer needed since tests are run in a temporary home
directory instead of the user's one.
---
 test/test-lib.sh |    3 ---
 1 files changed, 0 insertions(+), 3 deletions(-)

diff --git a/test/test-lib.sh b/test/test-lib.sh
index cc20f41..3ec388c 100755
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -855,8 +855,6 @@ fi
 #
 # notmuch-test-wait	Function for tests to use to wait for process completion
 #
-# message-signature	Avoiding appending user's signature on messages
-#
 # set-frame-width	80 columns (avoids crazy 10-column default of --batch)
 
 emacs \$BATCH --no-init-file --no-site-file \
@@ -865,7 +863,6 @@ emacs \$BATCH --no-init-file --no-site-file \
 	--eval "(defun notmuch-test-wait ()
 			(while (get-buffer-process (current-buffer))
 				(sleep-for 0.1)))" \
-	--eval "(setq message-signature nil)" \
 	--eval "(progn (set-frame-width (window-frame (get-buffer-window)) 80) \$@)"
 EOF
 	chmod a+x ./run_emacs
-- 
1.7.5.4

^ permalink raw reply related	[flat|nested] 22+ messages in thread
* [PATCH 03/10] test: cleanup test_emacs
  2011-06-28  4:45 [PATCH 01/10] test: fix argument order of test_expect_equal_file in few tests Dmitry Kurochkin
  2011-06-28  4:45 ` [PATCH 02/10] test: do not set `message-signature' in test_emacs Dmitry Kurochkin
@ 2011-06-28  4:45 ` Dmitry Kurochkin
  2011-06-28  4:45 ` [PATCH 04/10] test: wrap and indent test_emacs calls Dmitry Kurochkin
                   ` (6 subsequent siblings)
  8 siblings, 0 replies; 22+ messages in thread
From: Dmitry Kurochkin @ 2011-06-28  4:45 UTC (permalink / raw)
  To: notmuch

Move auxiliary function definition and configuration from command
line to test-lib.el.
---
 test/test-lib.el |    8 ++++++++
 test/test-lib.sh |    9 +--------
 2 files changed, 9 insertions(+), 8 deletions(-)

diff --git a/test/test-lib.el b/test/test-lib.el
index 9439996..344a02e 100644
--- a/test/test-lib.el
+++ b/test/test-lib.el
@@ -20,6 +20,14 @@
 ;;
 ;; Authors: Dmitry Kurochkin <dmitry.kurochkin@gmail.com>
 
+;; avoid crazy 10-column default of --batch
+(set-frame-width (window-frame (get-buffer-window)) 80)
+
+(defun notmuch-test-wait ()
+  "Wait for process completion."
+  (while (get-buffer-process (current-buffer))
+    (sleep-for 0.1)))
+
 (defun visible-buffer-string ()
   "Same as `buffer-string', but excludes invisible text."
   (visible-buffer-substring (point-min) (point-max)))
diff --git a/test/test-lib.sh b/test/test-lib.sh
index 3ec388c..5f61960 100755
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -852,18 +852,11 @@ fi
 # --directory		Ensure that the local elisp sources are found
 #
 # --load		Force loading of notmuch.el and test-lib.el
-#
-# notmuch-test-wait	Function for tests to use to wait for process completion
-#
-# set-frame-width	80 columns (avoids crazy 10-column default of --batch)
 
 emacs \$BATCH --no-init-file --no-site-file \
 	--directory ../../emacs --load notmuch.el \
 	--directory .. --load test-lib.el \
-	--eval "(defun notmuch-test-wait ()
-			(while (get-buffer-process (current-buffer))
-				(sleep-for 0.1)))" \
-	--eval "(progn (set-frame-width (window-frame (get-buffer-window)) 80) \$@)"
+	--eval "(progn \$@)"
 EOF
 	chmod a+x ./run_emacs
 	./run_emacs "$@"
-- 
1.7.5.4

^ permalink raw reply related	[flat|nested] 22+ messages in thread
* [PATCH 04/10] test: wrap and indent test_emacs calls
  2011-06-28  4:45 [PATCH 01/10] test: fix argument order of test_expect_equal_file in few tests Dmitry Kurochkin
  2011-06-28  4:45 ` [PATCH 02/10] test: do not set `message-signature' in test_emacs Dmitry Kurochkin
  2011-06-28  4:45 ` [PATCH 03/10] test: cleanup test_emacs Dmitry Kurochkin
@ 2011-06-28  4:45 ` Dmitry Kurochkin
  2011-06-28  4:45 ` [PATCH 05/10] test: save buffer content to file instead of printing it in Emacs tests Dmitry Kurochkin
                   ` (5 subsequent siblings)
  8 siblings, 0 replies; 22+ messages in thread
From: Dmitry Kurochkin @ 2011-06-28  4:45 UTC (permalink / raw)
  To: notmuch

Most test_emacs calls have long arguments that consist of many
expressions.  Putting them on a single line makes it hard to read
and produces poor diff when they are changed.  The patch puts
every expression in test_emacs calls on a separate line.
---
 test/emacs       |  124 ++++++++++++++++++++++++++++++++++++++++++++----------
 test/test-lib.sh |   14 ++++++-
 2 files changed, 114 insertions(+), 24 deletions(-)

diff --git a/test/emacs b/test/emacs
index f4ff3f1..fddb4a1 100755
--- a/test/emacs
+++ b/test/emacs
@@ -7,32 +7,51 @@ EXPECTED=../emacs.expected-output
 add_email_corpus
 
 test_begin_subtest "Basic notmuch-hello view in emacs"
-test_emacs '(notmuch-hello) (princ (buffer-string))' >OUTPUT
+test_emacs '(notmuch-hello)
+	    (princ (buffer-string))' >OUTPUT
 test_expect_equal_file OUTPUT $EXPECTED/notmuch-hello
 
 test_begin_subtest "Saved search with 0 results"
-test_emacs '(setq notmuch-show-empty-saved-searches t) (setq notmuch-saved-searches '\''(("inbox" . "tag:inbox") ("unread" . "tag:unread") ("empty" . "tag:doesnotexist"))) (notmuch-hello) (princ (buffer-string))' >OUTPUT
+test_emacs '(setq notmuch-show-empty-saved-searches t)
+	    (setq notmuch-saved-searches
+		  '\''(("inbox" . "tag:inbox")
+		       ("unread" . "tag:unread")
+		       ("empty" . "tag:doesnotexist")))
+	    (notmuch-hello)
+	    (princ (buffer-string))' >OUTPUT
 test_expect_equal_file OUTPUT $EXPECTED/notmuch-hello-with-empty
 
 test_begin_subtest "No saved searches displayed (all with 0 results)"
-test_emacs '(setq notmuch-saved-searches '\''(("empty" . "tag:doesnotexist"))) (notmuch-hello) (princ (buffer-string))' >OUTPUT
+test_emacs '(setq notmuch-saved-searches
+		  '\''(("empty" . "tag:doesnotexist")))
+	    (notmuch-hello)
+	    (princ (buffer-string))' >OUTPUT
 test_expect_equal_file OUTPUT $EXPECTED/notmuch-hello-no-saved-searches
 
 test_begin_subtest "Basic notmuch-search view in emacs"
-test_emacs '(notmuch-search "tag:inbox") (notmuch-test-wait) (princ (buffer-string))' >OUTPUT
+test_emacs '(notmuch-search "tag:inbox")
+	    (notmuch-test-wait)
+	    (princ (buffer-string))' >OUTPUT
 test_expect_equal_file OUTPUT $EXPECTED/notmuch-search-tag-inbox
 
 test_begin_subtest "Navigation of notmuch-hello to search results"
-test_emacs '(notmuch-hello) (goto-char (point-min)) (re-search-forward "inbox") (widget-button-press (point)) (notmuch-test-wait) (princ (buffer-string))' >OUTPUT
+test_emacs '(notmuch-hello)
+	    (goto-char (point-min))
+	    (re-search-forward "inbox")
+	    (widget-button-press (point))
+	    (notmuch-test-wait)
+	    (princ (buffer-string))' >OUTPUT
 test_expect_equal_file OUTPUT $EXPECTED/notmuch-hello-view-inbox
 
 test_begin_subtest "Basic notmuch-show view in emacs"
 maildir_storage_thread=$(notmuch search --output=threads id:20091117190054.GU3165@dottiness.seas.harvard.edu)
-test_emacs "(notmuch-show \"$maildir_storage_thread\") (princ (buffer-string))" >OUTPUT
+test_emacs "(notmuch-show \"$maildir_storage_thread\")
+	    (princ (buffer-string))" >OUTPUT
 test_expect_equal_file OUTPUT $EXPECTED/notmuch-show-thread-maildir-storage
 
 test_begin_subtest "notmuch-show for message with invalid From"
-add_message "[subject]=\"message-with-invalid-from\"" "[from]=\"\\\"Invalid \\\" From\\\" <test_suite@notmuchmail.org>\""
+add_message "[subject]=\"message-with-invalid-from\"" \
+	    "[from]=\"\\\"Invalid \\\" From\\\" <test_suite@notmuchmail.org>\""
 thread=$(notmuch search --output=threads subject:message-with-invalid-from)
 output=$(test_emacs "(notmuch-show \"$thread\") (princ (buffer-string))")
 test_expect_equal "$output" \
@@ -44,33 +63,54 @@ Date: Tue, 05 Jan 2001 15:43:57 -0000
 This is just a test message (#1)'
 
 test_begin_subtest "Navigation of notmuch-search to thread view"
-test_emacs '(notmuch-search "tag:inbox") (notmuch-test-wait) (goto-char (point-min)) (re-search-forward "Working with Maildir") (notmuch-search-show-thread) (notmuch-test-wait) (princ (buffer-string))' >OUTPUT
+test_emacs '(notmuch-search "tag:inbox")
+	    (notmuch-test-wait)
+	    (goto-char (point-min))
+	    (re-search-forward "Working with Maildir")
+	    (notmuch-search-show-thread)
+	    (notmuch-test-wait)
+	    (princ (buffer-string))' >OUTPUT
 test_expect_equal_file OUTPUT $EXPECTED/notmuch-show-thread-maildir-storage
 
 test_begin_subtest "Add tag from search view"
 os_x_darwin_thread=$(notmuch search --output=threads id:ddd65cda0911171950o4eea4389v86de9525e46052d3@mail.gmail.com)
-test_emacs "(notmuch-search \"$os_x_darwin_thread\") (notmuch-test-wait) (notmuch-search-add-tag \"tag-from-search-view\")"
+test_emacs "(notmuch-search \"$os_x_darwin_thread\")
+	    (notmuch-test-wait)
+	    (notmuch-search-add-tag \"tag-from-search-view\")"
 output=$(notmuch search $os_x_darwin_thread | notmuch_search_sanitize)
 test_expect_equal "$output" "thread:XXX   2009-11-18 [4/4] Jjgod Jiang, Alexander Botero-Lowry; [notmuch] Mac OS X/Darwin compatibility issues (inbox tag-from-search-view unread)"
 
 test_begin_subtest "Remove tag from search view"
-test_emacs "(notmuch-search \"$os_x_darwin_thread\") (notmuch-test-wait) (notmuch-search-remove-tag \"tag-from-search-view\")"
+test_emacs "(notmuch-search \"$os_x_darwin_thread\")
+	    (notmuch-test-wait)
+	    (notmuch-search-remove-tag \"tag-from-search-view\")"
 output=$(notmuch search $os_x_darwin_thread | notmuch_search_sanitize)
 test_expect_equal "$output" "thread:XXX   2009-11-18 [4/4] Jjgod Jiang, Alexander Botero-Lowry; [notmuch] Mac OS X/Darwin compatibility issues (inbox unread)"
 
 test_begin_subtest "Add tag from notmuch-show view"
-test_emacs "(notmuch-show \"$os_x_darwin_thread\") (notmuch-show-add-tag \"tag-from-show-view\")"
+test_emacs "(notmuch-show \"$os_x_darwin_thread\")
+	    (notmuch-show-add-tag \"tag-from-show-view\")"
 output=$(notmuch search $os_x_darwin_thread | notmuch_search_sanitize)
 test_expect_equal "$output" "thread:XXX   2009-11-18 [4/4] Jjgod Jiang, Alexander Botero-Lowry; [notmuch] Mac OS X/Darwin compatibility issues (inbox tag-from-show-view unread)"
 
 test_begin_subtest "Remove tag from notmuch-show view"
-test_emacs "(notmuch-show \"$os_x_darwin_thread\") (notmuch-show-remove-tag \"tag-from-show-view\")"
+test_emacs "(notmuch-show \"$os_x_darwin_thread\")
+	    (notmuch-show-remove-tag \"tag-from-show-view\")"
 output=$(notmuch search $os_x_darwin_thread | notmuch_search_sanitize)
 test_expect_equal "$output" "thread:XXX   2009-11-18 [4/4] Jjgod Jiang, Alexander Botero-Lowry; [notmuch] Mac OS X/Darwin compatibility issues (inbox unread)"
 
 test_begin_subtest "Message with .. in Message-Id:"
 add_message [id]=123..456@example '[subject]="Message with .. in Message-Id"'
-test_emacs '(notmuch-search "id:\"123..456@example\"") (notmuch-test-wait) (notmuch-search-add-tag "search-add") (notmuch-search-add-tag "search-remove") (notmuch-search-remove-tag "search-remove") (notmuch-show "id:\"123..456@example\"") (notmuch-test-wait) (notmuch-show-add-tag "show-add") (notmuch-show-add-tag "show-remove") (notmuch-show-remove-tag "show-remove")'
+test_emacs '(notmuch-search "id:\"123..456@example\"")
+	    (notmuch-test-wait)
+	    (notmuch-search-add-tag "search-add")
+	    (notmuch-search-add-tag "search-remove")
+	    (notmuch-search-remove-tag "search-remove")
+	    (notmuch-show "id:\"123..456@example\"")
+	    (notmuch-test-wait)
+	    (notmuch-show-add-tag "show-add")
+	    (notmuch-show-add-tag "show-remove")
+	    (notmuch-show-remove-tag "show-remove")'
 output=$(notmuch search 'id:"123..456@example"' | notmuch_search_sanitize)
 test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Message with .. in Message-Id (inbox search-add show-add)"
 
@@ -83,7 +123,18 @@ mkdir -p mail/sent/tmp
 
 ../smtp-dummy sent_message &
 smtp_dummy_pid=$!
-test_emacs "(setq message-send-mail-function 'message-smtpmail-send-it) (setq smtpmail-smtp-server \"localhost\") (setq smtpmail-smtp-service \"25025\") (notmuch-hello) (notmuch-mua-mail) (message-goto-to) (insert \"user@example.com\nDate: Fri, 29 Mar 1974 10:00:00 -0000\") (message-goto-subject) (insert \"Testing message sent via SMTP\") (message-goto-body) (insert \"This is a test that messages are sent via SMTP\") (message-send-and-exit)" >/dev/null 2>&1
+test_emacs "(setq message-send-mail-function 'message-smtpmail-send-it)
+	    (setq smtpmail-smtp-server \"localhost\")
+	    (setq smtpmail-smtp-service \"25025\")
+	    (notmuch-hello)
+	    (notmuch-mua-mail)
+	    (message-goto-to)
+	    (insert \"user@example.com\nDate: Fri, 29 Mar 1974 10:00:00 -0000\")
+	    (message-goto-subject)
+	    (insert \"Testing message sent via SMTP\")
+	    (message-goto-body)
+	    (insert \"This is a test that messages are sent via SMTP\")
+	    (message-send-and-exit)" >/dev/null 2>&1
 wait ${smtp_dummy_pid}
 
 sed \
@@ -109,7 +160,9 @@ output=$(notmuch search 'subject:"testing message sent via SMTP"' | notmuch_sear
 test_expect_equal "$output" "thread:XXX   1974-03-29 [1/1] Notmuch Test Suite; Testing message sent via SMTP (inbox)"
 
 test_begin_subtest "notmuch-fcc-dirs set to nil"
-test_emacs "(setq notmuch-fcc-dirs nil) (notmuch-mua-mail) (princ (buffer-string))" > OUTPUT
+test_emacs "(setq notmuch-fcc-dirs nil)
+	    (notmuch-mua-mail)
+	    (princ (buffer-string))" > OUTPUT
 cat <<EOF >EXPECTED
 From: Notmuch Test Suite <test_suite@notmuchmail.org>
 To: 
@@ -124,7 +177,9 @@ mkdir -p mail/sent-string/new
 mkdir -p mail/sent-string/tmp
 
 test_begin_subtest "notmuch-fcc-dirs set to a string"
-test_emacs "(setq notmuch-fcc-dirs \"sent-string\") (notmuch-mua-mail) (princ (buffer-string))" > OUTPUT
+test_emacs "(setq notmuch-fcc-dirs \"sent-string\")
+	    (notmuch-mua-mail)
+	    (princ (buffer-string))" > OUTPUT
 cat <<EOF >EXPECTED
 From: Notmuch Test Suite <test_suite@notmuchmail.org>
 To: 
@@ -143,7 +198,11 @@ mkdir -p mail/failure/new
 mkdir -p mail/failure/tmp
 
 test_begin_subtest "notmuch-fcc-dirs set to a list (with match)"
-test_emacs "(setq notmuch-fcc-dirs '((\"notmuchmail.org\" . \"sent-list-match\") (\".*\" . \"failure\"))) (notmuch-mua-mail) (princ (buffer-string))" > OUTPUT
+test_emacs "(setq notmuch-fcc-dirs
+		  '((\"notmuchmail.org\" . \"sent-list-match\")
+		    (\".*\" . \"failure\")))
+	    (notmuch-mua-mail)
+	    (princ (buffer-string))" > OUTPUT
 cat <<EOF >EXPECTED
 From: Notmuch Test Suite <test_suite@notmuchmail.org>
 To: 
@@ -159,7 +218,11 @@ mkdir -p mail/sent-list-catch-all/new
 mkdir -p mail/sent-list-catch-all/tmp
  
 test_begin_subtest "notmuch-fcc-dirs set to a list (catch-all)"
-test_emacs "(setq notmuch-fcc-dirs '((\"example.com\" . \"failure\") (\".*\" . \"sent-list-catch-all\"))) (notmuch-mua-mail) (princ (buffer-string))" > OUTPUT
+test_emacs "(setq notmuch-fcc-dirs
+		  '((\"example.com\" . \"failure\")
+		    (\".*\" . \"sent-list-catch-all\")))
+	    (notmuch-mua-mail)
+	    (princ (buffer-string))" > OUTPUT
 cat <<EOF >EXPECTED
 From: Notmuch Test Suite <test_suite@notmuchmail.org>
 To: 
@@ -170,7 +233,11 @@ EOF
 test_expect_equal_file OUTPUT EXPECTED
 
 test_begin_subtest "notmuch-fcc-dirs set to a list (no match)"
-test_emacs "(setq notmuch-fcc-dirs '((\"example.com\" . \"failure\") (\"nomatchhere.net\" . \"failure\"))) (notmuch-mua-mail) (princ (buffer-string))" > OUTPUT
+test_emacs "(setq notmuch-fcc-dirs
+		  '((\"example.com\" . \"failure\")
+		    (\"nomatchhere.net\" . \"failure\")))
+	    (notmuch-mua-mail)
+	    (princ (buffer-string))" > OUTPUT
 cat <<EOF >EXPECTED
 From: Notmuch Test Suite <test_suite@notmuchmail.org>
 To: 
@@ -182,7 +249,12 @@ test_expect_equal_file OUTPUT EXPECTED
 test_begin_subtest "Reply within emacs"
 # We sed away everything before the ^From in the output to avoid getting
 # confused by messages such as "Parsing /home/cworth/.mailrc... done"
-test_emacs '(notmuch-search "subject:\"testing message sent via SMTP\"") (notmuch-test-wait) (notmuch-search-reply-to-thread) (princ (buffer-string))' | sed -ne '/^From/,$ p' | sed -e 's/^In-Reply-To: <.*>$/In-Reply-To: <XXX>/' >OUTPUT
+test_emacs '(notmuch-search "subject:\"testing message sent via SMTP\"")
+	    (notmuch-test-wait)
+	    (notmuch-search-reply-to-thread)
+	    (princ (buffer-string))' |
+sed -ne '/^From/,$ p' |
+sed -e 's/^In-Reply-To: <.*>$/In-Reply-To: <XXX>/' >OUTPUT
 cat <<EOF >EXPECTED
 From: Notmuch Test Suite <test_suite@notmuchmail.org>
 To: user@example.com
@@ -197,17 +269,23 @@ test_expect_equal_file OUTPUT EXPECTED
 
 test_begin_subtest "Save attachment from within emacs using notmuch-show-save-attachments"
 # save as archive to test that Emacs does not re-compress .gz
-echo ./attachment1.gz | test_emacs '(notmuch-show "id:cf0c4d610911171136h1713aa59w9cf9aa31f052ad0a@mail.gmail.com") (notmuch-show-save-attachments)' > /dev/null 2>&1
+echo ./attachment1.gz |
+test_emacs '(notmuch-show "id:cf0c4d610911171136h1713aa59w9cf9aa31f052ad0a@mail.gmail.com")
+	    (notmuch-show-save-attachments)' > /dev/null 2>&1
 test_expect_equal_file attachment1.gz "$EXPECTED/attachment"
 
 test_begin_subtest "Save attachment from within emacs using notmuch-show-save-part"
 # save as archive to test that Emacs does not re-compress .gz
-echo ./attachment2.gz | test_emacs '(notmuch-show-save-part "id:cf0c4d610911171136h1713aa59w9cf9aa31f052ad0a@mail.gmail.com" 5)' > /dev/null 2>&1
+echo ./attachment2.gz |
+test_emacs '(notmuch-show-save-part "id:cf0c4d610911171136h1713aa59w9cf9aa31f052ad0a@mail.gmail.com" 5)' > /dev/null 2>&1
 test_expect_equal_file attachment2.gz "$EXPECTED/attachment"
 
 test_begin_subtest "View raw message within emacs"
 first_line=$(head -n1 $EXPECTED/raw-message-cf0c4d-52ad0a)
-test_emacs '(notmuch-show "id:cf0c4d610911171136h1713aa59w9cf9aa31f052ad0a@mail.gmail.com") (notmuch-show-view-raw-message) (princ (buffer-string))' | sed -ne "/$first_line/,\$ p" >OUTPUT
+test_emacs '(notmuch-show "id:cf0c4d610911171136h1713aa59w9cf9aa31f052ad0a@mail.gmail.com")
+	    (notmuch-show-view-raw-message)
+	    (princ (buffer-string))' |
+sed -ne "/$first_line/,\$ p" >OUTPUT
 test_expect_equal_file OUTPUT $EXPECTED/raw-message-cf0c4d-52ad0a
 
 test_begin_subtest "Hiding/showing signature in notmuch-show view"
diff --git a/test/test-lib.sh b/test/test-lib.sh
index 5f61960..ad1506c 100755
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -394,7 +394,19 @@ emacs_deliver_message ()
     mkdir -p "$MAIL_DIR"/sent/{cur,new,tmp}
     ../smtp-dummy sent_message &
     smtp_dummy_pid=$!
-    test_emacs "(setq message-send-mail-function 'message-smtpmail-send-it) (setq smtpmail-smtp-server \"localhost\") (setq smtpmail-smtp-service \"25025\") (notmuch-hello) (notmuch-mua-mail) (message-goto-to) (insert \"test_suite@notmuchmail.org\nDate: 01 Jan 2000 12:00:00 -0000\") (message-goto-subject) (insert \"${subject}\") (message-goto-body) (insert \"${body}\") $@ (message-send-and-exit)" >/dev/null 2>&1
+    test_emacs "(setq message-send-mail-function 'message-smtpmail-send-it)
+		(setq smtpmail-smtp-server \"localhost\")
+		(setq smtpmail-smtp-service \"25025\")
+		(notmuch-hello)
+		(notmuch-mua-mail)
+		(message-goto-to)
+		(insert \"test_suite@notmuchmail.org\nDate: 01 Jan 2000 12:00:00 -0000\")
+		(message-goto-subject)
+		(insert \"${subject}\")
+		(message-goto-body)
+		(insert \"${body}\")
+		$@
+		(message-send-and-exit)" >/dev/null 2>&1
     wait ${smtp_dummy_pid}
     notmuch new >/dev/null
 }
-- 
1.7.5.4

^ permalink raw reply related	[flat|nested] 22+ messages in thread
* [PATCH 05/10] test: save buffer content to file instead of printing it in Emacs tests
  2011-06-28  4:45 [PATCH 01/10] test: fix argument order of test_expect_equal_file in few tests Dmitry Kurochkin
                   ` (2 preceding siblings ...)
  2011-06-28  4:45 ` [PATCH 04/10] test: wrap and indent test_emacs calls Dmitry Kurochkin
@ 2011-06-28  4:45 ` Dmitry Kurochkin
  2012-01-23  4:26   ` [PATCH] test/emacs-large-search-buffer: correct typo (EXPEXTED -> EXPECTED) Pieter Praet
  2011-06-28  4:45 ` [PATCH 06/10] test: remove some sed(1) calls in Emacs tests Dmitry Kurochkin
                   ` (4 subsequent siblings)
  8 siblings, 1 reply; 22+ messages in thread
From: Dmitry Kurochkin @ 2011-06-28  4:45 UTC (permalink / raw)
  To: notmuch

Before the change, the common Emacs test scheme was to print
buffer content to stdout and redirect it to a file or capture it
in a shell variable.  This does not work if we switch to using
emacsclient(1) for running the tests, because you can not print
to the stdout in this case. (Actually, you can print to stdout
from Emacs server, but you can not capture the output on
emacsclient(1)).

The patch introduces new Emacs test auxiliary functions:
`test-output' and `test-visible-output'.  These functions are
used to save buffer content to a file directly from Emacs.  For
most tests the changes are trivial, because Emacs stdout output
was redirected to a file anyway.  But some tests captured the
output in a shell variable and compare it with the expected
output using test_expect_equal.  These tests are changed to use
files and test_expect_equal_file instead.

Note: even if we do not switch Emacs tests to emacsclient(1), the
patch makes tests cleaner and is an improvement.
---
 test/emacs                     |   71 ++++++++++++++++++++-------------------
 test/emacs-large-search-buffer |   23 +++++++++----
 test/test-lib.el               |   10 ++++++
 3 files changed, 62 insertions(+), 42 deletions(-)

diff --git a/test/emacs b/test/emacs
index fddb4a1..8f8b841 100755
--- a/test/emacs
+++ b/test/emacs
@@ -8,7 +8,7 @@ add_email_corpus
 
 test_begin_subtest "Basic notmuch-hello view in emacs"
 test_emacs '(notmuch-hello)
-	    (princ (buffer-string))' >OUTPUT
+	    (test-output)'
 test_expect_equal_file OUTPUT $EXPECTED/notmuch-hello
 
 test_begin_subtest "Saved search with 0 results"
@@ -18,20 +18,20 @@ test_emacs '(setq notmuch-show-empty-saved-searches t)
 		       ("unread" . "tag:unread")
 		       ("empty" . "tag:doesnotexist")))
 	    (notmuch-hello)
-	    (princ (buffer-string))' >OUTPUT
+	    (test-output)'
 test_expect_equal_file OUTPUT $EXPECTED/notmuch-hello-with-empty
 
 test_begin_subtest "No saved searches displayed (all with 0 results)"
 test_emacs '(setq notmuch-saved-searches
 		  '\''(("empty" . "tag:doesnotexist")))
 	    (notmuch-hello)
-	    (princ (buffer-string))' >OUTPUT
+	    (test-output)'
 test_expect_equal_file OUTPUT $EXPECTED/notmuch-hello-no-saved-searches
 
 test_begin_subtest "Basic notmuch-search view in emacs"
 test_emacs '(notmuch-search "tag:inbox")
 	    (notmuch-test-wait)
-	    (princ (buffer-string))' >OUTPUT
+	    (test-output)'
 test_expect_equal_file OUTPUT $EXPECTED/notmuch-search-tag-inbox
 
 test_begin_subtest "Navigation of notmuch-hello to search results"
@@ -40,27 +40,30 @@ test_emacs '(notmuch-hello)
 	    (re-search-forward "inbox")
 	    (widget-button-press (point))
 	    (notmuch-test-wait)
-	    (princ (buffer-string))' >OUTPUT
+	    (test-output)'
 test_expect_equal_file OUTPUT $EXPECTED/notmuch-hello-view-inbox
 
 test_begin_subtest "Basic notmuch-show view in emacs"
 maildir_storage_thread=$(notmuch search --output=threads id:20091117190054.GU3165@dottiness.seas.harvard.edu)
 test_emacs "(notmuch-show \"$maildir_storage_thread\")
-	    (princ (buffer-string))" >OUTPUT
+	    (test-output)"
 test_expect_equal_file OUTPUT $EXPECTED/notmuch-show-thread-maildir-storage
 
 test_begin_subtest "notmuch-show for message with invalid From"
 add_message "[subject]=\"message-with-invalid-from\"" \
 	    "[from]=\"\\\"Invalid \\\" From\\\" <test_suite@notmuchmail.org>\""
 thread=$(notmuch search --output=threads subject:message-with-invalid-from)
-output=$(test_emacs "(notmuch-show \"$thread\") (princ (buffer-string))")
-test_expect_equal "$output" \
-'"Invalid " From" <test_suite@notmuchmail.org> (2001-01-05) (inbox)
+test_emacs "(notmuch-show \"$thread\")
+	    (test-output)"
+cat <<EOF >EXPECTED
+"Invalid " From" <test_suite@notmuchmail.org> (2001-01-05) (inbox)
 Subject: message-with-invalid-from
 To: Notmuch Test Suite <test_suite@notmuchmail.org>
 Date: Tue, 05 Jan 2001 15:43:57 -0000
 
-This is just a test message (#1)'
+This is just a test message (#1)
+EOF
+test_expect_equal_file OUTPUT EXPECTED
 
 test_begin_subtest "Navigation of notmuch-search to thread view"
 test_emacs '(notmuch-search "tag:inbox")
@@ -69,7 +72,7 @@ test_emacs '(notmuch-search "tag:inbox")
 	    (re-search-forward "Working with Maildir")
 	    (notmuch-search-show-thread)
 	    (notmuch-test-wait)
-	    (princ (buffer-string))' >OUTPUT
+	    (test-output)'
 test_expect_equal_file OUTPUT $EXPECTED/notmuch-show-thread-maildir-storage
 
 test_begin_subtest "Add tag from search view"
@@ -162,7 +165,7 @@ test_expect_equal "$output" "thread:XXX   1974-03-29 [1/1] Notmuch Test Suite; T
 test_begin_subtest "notmuch-fcc-dirs set to nil"
 test_emacs "(setq notmuch-fcc-dirs nil)
 	    (notmuch-mua-mail)
-	    (princ (buffer-string))" > OUTPUT
+	    (test-output)"
 cat <<EOF >EXPECTED
 From: Notmuch Test Suite <test_suite@notmuchmail.org>
 To: 
@@ -179,7 +182,7 @@ mkdir -p mail/sent-string/tmp
 test_begin_subtest "notmuch-fcc-dirs set to a string"
 test_emacs "(setq notmuch-fcc-dirs \"sent-string\")
 	    (notmuch-mua-mail)
-	    (princ (buffer-string))" > OUTPUT
+	    (test-output)"
 cat <<EOF >EXPECTED
 From: Notmuch Test Suite <test_suite@notmuchmail.org>
 To: 
@@ -202,7 +205,7 @@ test_emacs "(setq notmuch-fcc-dirs
 		  '((\"notmuchmail.org\" . \"sent-list-match\")
 		    (\".*\" . \"failure\")))
 	    (notmuch-mua-mail)
-	    (princ (buffer-string))" > OUTPUT
+	    (test-output)"
 cat <<EOF >EXPECTED
 From: Notmuch Test Suite <test_suite@notmuchmail.org>
 To: 
@@ -222,7 +225,7 @@ test_emacs "(setq notmuch-fcc-dirs
 		  '((\"example.com\" . \"failure\")
 		    (\".*\" . \"sent-list-catch-all\")))
 	    (notmuch-mua-mail)
-	    (princ (buffer-string))" > OUTPUT
+	    (test-output)"
 cat <<EOF >EXPECTED
 From: Notmuch Test Suite <test_suite@notmuchmail.org>
 To: 
@@ -237,7 +240,7 @@ test_emacs "(setq notmuch-fcc-dirs
 		  '((\"example.com\" . \"failure\")
 		    (\"nomatchhere.net\" . \"failure\")))
 	    (notmuch-mua-mail)
-	    (princ (buffer-string))" > OUTPUT
+	    (test-output)"
 cat <<EOF >EXPECTED
 From: Notmuch Test Suite <test_suite@notmuchmail.org>
 To: 
@@ -252,9 +255,9 @@ test_begin_subtest "Reply within emacs"
 test_emacs '(notmuch-search "subject:\"testing message sent via SMTP\"")
 	    (notmuch-test-wait)
 	    (notmuch-search-reply-to-thread)
-	    (princ (buffer-string))' |
-sed -ne '/^From/,$ p' |
-sed -e 's/^In-Reply-To: <.*>$/In-Reply-To: <XXX>/' >OUTPUT
+	    (test-output)'
+sed -i -ne '/^From/,$ p' OUTPUT
+sed -i -e 's/^In-Reply-To: <.*>$/In-Reply-To: <XXX>/' OUTPUT
 cat <<EOF >EXPECTED
 From: Notmuch Test Suite <test_suite@notmuchmail.org>
 To: user@example.com
@@ -284,8 +287,8 @@ test_begin_subtest "View raw message within emacs"
 first_line=$(head -n1 $EXPECTED/raw-message-cf0c4d-52ad0a)
 test_emacs '(notmuch-show "id:cf0c4d610911171136h1713aa59w9cf9aa31f052ad0a@mail.gmail.com")
 	    (notmuch-show-view-raw-message)
-	    (princ (buffer-string))' |
-sed -ne "/$first_line/,\$ p" >OUTPUT
+	    (test-output)'
+sed -i -ne "/$first_line/,\$ p" OUTPUT
 test_expect_equal_file OUTPUT $EXPECTED/raw-message-cf0c4d-52ad0a
 
 test_begin_subtest "Hiding/showing signature in notmuch-show view"
@@ -295,7 +298,7 @@ test_emacs "(notmuch-show \"$maildir_storage_thread\")
 	    (button-activate (button-at (point)))
 	    (search-backward \"Click/Enter to hide.\")
 	    (button-activate (button-at (point)))
-	    (princ (buffer-string))" >OUTPUT
+	    (test-output)"
 test_expect_equal_file OUTPUT $EXPECTED/notmuch-show-thread-maildir-storage
 
 test_begin_subtest "Detection and hiding of top-post quoting of message"
@@ -323,7 +326,7 @@ Q: Why is top-posting such a bad thing?
 A: Top-posting.
 Q: What is the most annoying thing in e-mail?"'
 test_emacs "(notmuch-show \"top-posting\")
-	    (princ (visible-buffer-string))" >OUTPUT
+	    (test-visible-output)"
 echo "Notmuch Test Suite <test_suite@notmuchmail.org> (2001-01-05) (inbox)
 Subject: The problem with top-posting
 To: Notmuch Test Suite <test_suite@notmuchmail.org>
@@ -346,19 +349,17 @@ Thanks for the advice! I will be sure to put it to good use.
 test_expect_equal_file OUTPUT EXPECTED
 
 test_begin_subtest "Hiding message in notmuch-show view"
-output=$(test_emacs '(notmuch-show "id:f35dbb950911171438k5df6eb56k77b6c0944e2e79ae@mail.gmail.com")
-		     (notmuch-show-toggle-message)
-		     (princ (visible-buffer-string))')
-expected=$(cat $EXPECTED/notmuch-show-thread-with-hidden-messages)
-test_expect_equal "$output" "$expected"
+test_emacs '(notmuch-show "id:f35dbb950911171438k5df6eb56k77b6c0944e2e79ae@mail.gmail.com")
+	    (notmuch-show-toggle-message)
+	    (test-visible-output)'
+test_expect_equal_file OUTPUT $EXPECTED/notmuch-show-thread-with-hidden-messages
 
 test_begin_subtest "Hiding message with visible citation in notmuch-show view"
-output=$(test_emacs '(notmuch-show "id:f35dbb950911171438k5df6eb56k77b6c0944e2e79ae@mail.gmail.com")
-		     (search-forward "Click/Enter to show.")
-		     (button-activate (button-at (point)))
-		     (notmuch-show-toggle-message)
-		     (princ (visible-buffer-string))')
-expected=$(cat $EXPECTED/notmuch-show-thread-with-hidden-messages)
-test_expect_equal "$output" "$expected"
+test_emacs '(notmuch-show "id:f35dbb950911171438k5df6eb56k77b6c0944e2e79ae@mail.gmail.com")
+	    (search-forward "Click/Enter to show.")
+	    (button-activate (button-at (point)))
+	    (notmuch-show-toggle-message)
+	    (test-visible-output)'
+test_expect_equal_file OUTPUT $EXPECTED/notmuch-show-thread-with-hidden-messages
 
 test_done
diff --git a/test/emacs-large-search-buffer b/test/emacs-large-search-buffer
index c78ce33..6095e9d 100755
--- a/test/emacs-large-search-buffer
+++ b/test/emacs-large-search-buffer
@@ -19,16 +19,25 @@ done
 notmuch new > /dev/null
 
 test_begin_subtest "Ensure that emacs doesn't drop results"
-expected="$(notmuch search '*' | sed -e 's/^thread:[0-9a-f]*  //' -e 's/;//' -e  's/xx*/[BLOB]/')
-End of search results."
+notmuch search '*' > EXPEXTED
+sed -i -e 's/^thread:[0-9a-f]*  //' -e 's/;//' -e 's/xx*/[BLOB]/' EXPEXTED
+echo 'End of search results.' >> EXPEXTED
 
-output=$(test_emacs '(notmuch-search "*") (notmuch-test-wait) (princ (buffer-string))' | sed -e s',  *, ,g' -e 's/xxx*/[BLOB]/g')
-test_expect_equal "$output" "$expected"
+test_emacs '(notmuch-search "*")
+	    (notmuch-test-wait)
+	    (test-output)'
+sed -i -e s',  *, ,g' -e 's/xxx*/[BLOB]/g' OUTPUT
+test_expect_equal_file OUTPUT EXPEXTED
 
 test_begin_subtest "Ensure that emacs doesn't drop error messages"
-output=$(test_emacs '(notmuch-search "--this-option-does-not-exist") (notmuch-test-wait) (princ (buffer-string))')
-test_expect_equal "$output" "Error: Unexpected output from notmuch search:
+test_emacs '(notmuch-search "--this-option-does-not-exist")
+	    (notmuch-test-wait)
+	    (test-output)'
+cat <<EOF >EXPEXTED
+Error: Unexpected output from notmuch search:
 Unrecognized option: --this-option-does-not-exist
-End of search results. (process returned 1)"
+End of search results. (process returned 1)
+EOF
+test_expect_equal_file OUTPUT EXPEXTED
 
 test_done
diff --git a/test/test-lib.el b/test/test-lib.el
index 344a02e..4e7f5cf 100644
--- a/test/test-lib.el
+++ b/test/test-lib.el
@@ -28,6 +28,16 @@
   (while (get-buffer-process (current-buffer))
     (sleep-for 0.1)))
 
+(defun test-output (&optional filename)
+  "Save current buffer to file FILENAME.  Default FILENAME is OUTPUT."
+  (write-region (point-min) (point-max) (or filename "OUTPUT")))
+
+(defun test-visible-output (&optional filename)
+  "Save visible text in current buffer to file FILENAME.  Default
+FILENAME is OUTPUT."
+  (let ((text (visible-buffer-string)))
+    (with-temp-file (or filename "OUTPUT") (insert text))))
+
 (defun visible-buffer-string ()
   "Same as `buffer-string', but excludes invisible text."
   (visible-buffer-substring (point-min) (point-max)))
-- 
1.7.5.4

^ permalink raw reply related	[flat|nested] 22+ messages in thread
* [PATCH 06/10] test: remove some sed(1) calls in Emacs tests
  2011-06-28  4:45 [PATCH 01/10] test: fix argument order of test_expect_equal_file in few tests Dmitry Kurochkin
                   ` (3 preceding siblings ...)
  2011-06-28  4:45 ` [PATCH 05/10] test: save buffer content to file instead of printing it in Emacs tests Dmitry Kurochkin
@ 2011-06-28  4:45 ` Dmitry Kurochkin
  2011-06-28  4:45 ` [PATCH 07/10] test: use emacs_deliver_message in Emacs SMTP send test Dmitry Kurochkin
                   ` (3 subsequent siblings)
  8 siblings, 0 replies; 22+ messages in thread
From: Dmitry Kurochkin @ 2011-06-28  4:45 UTC (permalink / raw)
  To: notmuch

Few Emacs tests used sed(1) to remove unexpected output in the
beginning to avoid getting confused by messages such as "Parsing
/home/cworth/.mailrc... done".  This is no longer needed since
tests are run in a temporary home directory instead of the user's
one.  So remove these sed(1) calls.
---
 test/emacs |    5 -----
 1 files changed, 0 insertions(+), 5 deletions(-)

diff --git a/test/emacs b/test/emacs
index 8f8b841..4592005 100755
--- a/test/emacs
+++ b/test/emacs
@@ -250,13 +250,10 @@ EOF
 test_expect_equal_file OUTPUT EXPECTED
 
 test_begin_subtest "Reply within emacs"
-# We sed away everything before the ^From in the output to avoid getting
-# confused by messages such as "Parsing /home/cworth/.mailrc... done"
 test_emacs '(notmuch-search "subject:\"testing message sent via SMTP\"")
 	    (notmuch-test-wait)
 	    (notmuch-search-reply-to-thread)
 	    (test-output)'
-sed -i -ne '/^From/,$ p' OUTPUT
 sed -i -e 's/^In-Reply-To: <.*>$/In-Reply-To: <XXX>/' OUTPUT
 cat <<EOF >EXPECTED
 From: Notmuch Test Suite <test_suite@notmuchmail.org>
@@ -284,11 +281,9 @@ test_emacs '(notmuch-show-save-part "id:cf0c4d610911171136h1713aa59w9cf9aa31f052
 test_expect_equal_file attachment2.gz "$EXPECTED/attachment"
 
 test_begin_subtest "View raw message within emacs"
-first_line=$(head -n1 $EXPECTED/raw-message-cf0c4d-52ad0a)
 test_emacs '(notmuch-show "id:cf0c4d610911171136h1713aa59w9cf9aa31f052ad0a@mail.gmail.com")
 	    (notmuch-show-view-raw-message)
 	    (test-output)'
-sed -i -ne "/$first_line/,\$ p" OUTPUT
 test_expect_equal_file OUTPUT $EXPECTED/raw-message-cf0c4d-52ad0a
 
 test_begin_subtest "Hiding/showing signature in notmuch-show view"
-- 
1.7.5.4

^ permalink raw reply related	[flat|nested] 22+ messages in thread
* [PATCH 07/10] test: use emacs_deliver_message in Emacs SMTP send test
  2011-06-28  4:45 [PATCH 01/10] test: fix argument order of test_expect_equal_file in few tests Dmitry Kurochkin
                   ` (4 preceding siblings ...)
  2011-06-28  4:45 ` [PATCH 06/10] test: remove some sed(1) calls in Emacs tests Dmitry Kurochkin
@ 2011-06-28  4:45 ` Dmitry Kurochkin
  2011-06-28  4:45 ` [PATCH 08/10] test: set variables using `let' instead of `setq' in Emacs tests Dmitry Kurochkin
                   ` (2 subsequent siblings)
  8 siblings, 0 replies; 22+ messages in thread
From: Dmitry Kurochkin @ 2011-06-28  4:45 UTC (permalink / raw)
  To: notmuch

Minor changes to expected results of other Emacs tests were
needed because the message Date header changed.
---
 test/emacs |   34 +++++++++-------------------------
 1 files changed, 9 insertions(+), 25 deletions(-)

diff --git a/test/emacs b/test/emacs
index 4592005..409e033 100755
--- a/test/emacs
+++ b/test/emacs
@@ -118,28 +118,12 @@ output=$(notmuch search 'id:"123..456@example"' | notmuch_search_sanitize)
 test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Message with .. in Message-Id (inbox search-add show-add)"
 
 test_begin_subtest "Sending a message via (fake) SMTP"
-
-# Before we can send a message, we have to prepare the FCC maildir
-mkdir -p mail/sent/cur
-mkdir -p mail/sent/new
-mkdir -p mail/sent/tmp
-
-../smtp-dummy sent_message &
-smtp_dummy_pid=$!
-test_emacs "(setq message-send-mail-function 'message-smtpmail-send-it)
-	    (setq smtpmail-smtp-server \"localhost\")
-	    (setq smtpmail-smtp-service \"25025\")
-	    (notmuch-hello)
-	    (notmuch-mua-mail)
-	    (message-goto-to)
-	    (insert \"user@example.com\nDate: Fri, 29 Mar 1974 10:00:00 -0000\")
-	    (message-goto-subject)
-	    (insert \"Testing message sent via SMTP\")
-	    (message-goto-body)
-	    (insert \"This is a test that messages are sent via SMTP\")
-	    (message-send-and-exit)" >/dev/null 2>&1
-wait ${smtp_dummy_pid}
-
+emacs_deliver_message \
+    'Testing message sent via SMTP' \
+    'This is a test that messages are sent via SMTP' \
+    '(message-goto-to)
+     (kill-whole-line)
+     (insert "To: user@example.com\n")'
 sed \
     -e s',^User-Agent: Notmuch/.* Emacs/.*,User-Agent: Notmuch/XXX Emacs/XXX,' \
     -e s',^Message-ID: <.*>$,Message-ID: <XXX>,' < sent_message >OUTPUT
@@ -147,7 +131,7 @@ cat <<EOF >EXPECTED
 From: Notmuch Test Suite <test_suite@notmuchmail.org>
 To: user@example.com
 Subject: Testing message sent via SMTP
-Date: Fri, 29 Mar 1974 10:00:00 -0000
+Date: 01 Jan 2000 12:00:00 -0000
 User-Agent: Notmuch/XXX Emacs/XXX
 Message-ID: <XXX>
 MIME-Version: 1.0
@@ -160,7 +144,7 @@ test_expect_equal_file OUTPUT EXPECTED
 test_begin_subtest "Verify that sent messages are saved/searchable (via FCC)"
 notmuch new > /dev/null
 output=$(notmuch search 'subject:"testing message sent via SMTP"' | notmuch_search_sanitize)
-test_expect_equal "$output" "thread:XXX   1974-03-29 [1/1] Notmuch Test Suite; Testing message sent via SMTP (inbox)"
+test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; Testing message sent via SMTP (inbox)"
 
 test_begin_subtest "notmuch-fcc-dirs set to nil"
 test_emacs "(setq notmuch-fcc-dirs nil)
@@ -262,7 +246,7 @@ Subject: Re: Testing message sent via SMTP
 In-Reply-To: <XXX>
 Fcc: $(pwd)/mail/sent
 --text follows this line--
-On Fri, 29 Mar 1974 10:00:00 -0000, Notmuch Test Suite <test_suite@notmuchmail.org> wrote:
+On 01 Jan 2000 12:00:00 -0000, Notmuch Test Suite <test_suite@notmuchmail.org> wrote:
 > This is a test that messages are sent via SMTP
 EOF
 test_expect_equal_file OUTPUT EXPECTED
-- 
1.7.5.4

^ permalink raw reply related	[flat|nested] 22+ messages in thread
* [PATCH 08/10] test: set variables using `let' instead of `setq' in Emacs tests
  2011-06-28  4:45 [PATCH 01/10] test: fix argument order of test_expect_equal_file in few tests Dmitry Kurochkin
                   ` (5 preceding siblings ...)
  2011-06-28  4:45 ` [PATCH 07/10] test: use emacs_deliver_message in Emacs SMTP send test Dmitry Kurochkin
@ 2011-06-28  4:45 ` Dmitry Kurochkin
  2012-01-23 19:05   ` Xavier Maillard
  2011-06-28  4:45 ` [PATCH 09/10] test: generate run_emacs script once on test startup Dmitry Kurochkin
  2011-06-28  4:45 ` [PATCH 10/10] test: use emacsclient(1) for Emacs tests Dmitry Kurochkin
  8 siblings, 1 reply; 22+ messages in thread
From: Dmitry Kurochkin @ 2011-06-28  4:45 UTC (permalink / raw)
  To: notmuch

Using `setq' for setting variables in Emacs tests affect other
tests that may run in the same Emacs environment.  Currently it
works because each test is run in a separate Emacs instance.  But
in the future multiple tests will run in a single Emacs instance.
The patch changes all variables to use `let', so the scope of the
change is limited to a single test.
---
 test/emacs       |   70 +++++++++++++++++++++++++++++------------------------
 test/test-lib.sh |   27 ++++++++++----------
 2 files changed, 52 insertions(+), 45 deletions(-)

diff --git a/test/emacs b/test/emacs
index 409e033..3d42645 100755
--- a/test/emacs
+++ b/test/emacs
@@ -1,4 +1,10 @@
 #!/usr/bin/env bash
+
+# Note: do not use `setq' for setting variables in Emacs tests because
+# it affects other tests that may run in the same Emacs instance.  Use
+# `let' instead so the scope of the changed variables is limited to a
+# single test.
+
 test_description="emacs interface"
 . test-lib.sh
 
@@ -12,20 +18,20 @@ test_emacs '(notmuch-hello)
 test_expect_equal_file OUTPUT $EXPECTED/notmuch-hello
 
 test_begin_subtest "Saved search with 0 results"
-test_emacs '(setq notmuch-show-empty-saved-searches t)
-	    (setq notmuch-saved-searches
-		  '\''(("inbox" . "tag:inbox")
-		       ("unread" . "tag:unread")
-		       ("empty" . "tag:doesnotexist")))
-	    (notmuch-hello)
-	    (test-output)'
+test_emacs '(let ((notmuch-show-empty-saved-searches t)
+		  (notmuch-saved-searches
+		   '\''(("inbox" . "tag:inbox")
+			("unread" . "tag:unread")
+			("empty" . "tag:doesnotexist"))))
+	      (notmuch-hello)
+	      (test-output))'
 test_expect_equal_file OUTPUT $EXPECTED/notmuch-hello-with-empty
 
 test_begin_subtest "No saved searches displayed (all with 0 results)"
-test_emacs '(setq notmuch-saved-searches
-		  '\''(("empty" . "tag:doesnotexist")))
-	    (notmuch-hello)
-	    (test-output)'
+test_emacs '(let ((notmuch-saved-searches
+		   '\''(("empty" . "tag:doesnotexist"))))
+	      (notmuch-hello)
+	      (test-output))'
 test_expect_equal_file OUTPUT $EXPECTED/notmuch-hello-no-saved-searches
 
 test_begin_subtest "Basic notmuch-search view in emacs"
@@ -147,9 +153,9 @@ output=$(notmuch search 'subject:"testing message sent via SMTP"' | notmuch_sear
 test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; Testing message sent via SMTP (inbox)"
 
 test_begin_subtest "notmuch-fcc-dirs set to nil"
-test_emacs "(setq notmuch-fcc-dirs nil)
-	    (notmuch-mua-mail)
-	    (test-output)"
+test_emacs "(let ((notmuch-fcc-dirs nil))
+	      (notmuch-mua-mail)
+	      (test-output))"
 cat <<EOF >EXPECTED
 From: Notmuch Test Suite <test_suite@notmuchmail.org>
 To: 
@@ -164,9 +170,9 @@ mkdir -p mail/sent-string/new
 mkdir -p mail/sent-string/tmp
 
 test_begin_subtest "notmuch-fcc-dirs set to a string"
-test_emacs "(setq notmuch-fcc-dirs \"sent-string\")
-	    (notmuch-mua-mail)
-	    (test-output)"
+test_emacs "(let ((notmuch-fcc-dirs \"sent-string\"))
+	      (notmuch-mua-mail)
+	      (test-output))"
 cat <<EOF >EXPECTED
 From: Notmuch Test Suite <test_suite@notmuchmail.org>
 To: 
@@ -185,11 +191,11 @@ mkdir -p mail/failure/new
 mkdir -p mail/failure/tmp
 
 test_begin_subtest "notmuch-fcc-dirs set to a list (with match)"
-test_emacs "(setq notmuch-fcc-dirs
-		  '((\"notmuchmail.org\" . \"sent-list-match\")
-		    (\".*\" . \"failure\")))
-	    (notmuch-mua-mail)
-	    (test-output)"
+test_emacs "(let ((notmuch-fcc-dirs
+		   '((\"notmuchmail.org\" . \"sent-list-match\")
+		     (\".*\" . \"failure\"))))
+	      (notmuch-mua-mail)
+	      (test-output))"
 cat <<EOF >EXPECTED
 From: Notmuch Test Suite <test_suite@notmuchmail.org>
 To: 
@@ -205,11 +211,11 @@ mkdir -p mail/sent-list-catch-all/new
 mkdir -p mail/sent-list-catch-all/tmp
  
 test_begin_subtest "notmuch-fcc-dirs set to a list (catch-all)"
-test_emacs "(setq notmuch-fcc-dirs
-		  '((\"example.com\" . \"failure\")
-		    (\".*\" . \"sent-list-catch-all\")))
-	    (notmuch-mua-mail)
-	    (test-output)"
+test_emacs "(let ((notmuch-fcc-dirs
+		   '((\"example.com\" . \"failure\")
+		     (\".*\" . \"sent-list-catch-all\"))))
+	      (notmuch-mua-mail)
+	      (test-output))"
 cat <<EOF >EXPECTED
 From: Notmuch Test Suite <test_suite@notmuchmail.org>
 To: 
@@ -220,11 +226,11 @@ EOF
 test_expect_equal_file OUTPUT EXPECTED
 
 test_begin_subtest "notmuch-fcc-dirs set to a list (no match)"
-test_emacs "(setq notmuch-fcc-dirs
-		  '((\"example.com\" . \"failure\")
-		    (\"nomatchhere.net\" . \"failure\")))
-	    (notmuch-mua-mail)
-	    (test-output)"
+test_emacs "(let ((notmuch-fcc-dirs
+		   '((\"example.com\" . \"failure\")
+		     (\"nomatchhere.net\" . \"failure\"))))
+	      (notmuch-mua-mail)
+	      (test-output))"
 cat <<EOF >EXPECTED
 From: Notmuch Test Suite <test_suite@notmuchmail.org>
 To: 
diff --git a/test/test-lib.sh b/test/test-lib.sh
index ad1506c..7c8a86f 100755
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -394,19 +394,20 @@ emacs_deliver_message ()
     mkdir -p "$MAIL_DIR"/sent/{cur,new,tmp}
     ../smtp-dummy sent_message &
     smtp_dummy_pid=$!
-    test_emacs "(setq message-send-mail-function 'message-smtpmail-send-it)
-		(setq smtpmail-smtp-server \"localhost\")
-		(setq smtpmail-smtp-service \"25025\")
-		(notmuch-hello)
-		(notmuch-mua-mail)
-		(message-goto-to)
-		(insert \"test_suite@notmuchmail.org\nDate: 01 Jan 2000 12:00:00 -0000\")
-		(message-goto-subject)
-		(insert \"${subject}\")
-		(message-goto-body)
-		(insert \"${body}\")
-		$@
-		(message-send-and-exit)" >/dev/null 2>&1
+    test_emacs \
+	"(let ((message-send-mail-function 'message-smtpmail-send-it)
+	       (smtpmail-smtp-server \"localhost\")
+	       (smtpmail-smtp-service \"25025\"))
+	   (notmuch-hello)
+	   (notmuch-mua-mail)
+	   (message-goto-to)
+	   (insert \"test_suite@notmuchmail.org\nDate: 01 Jan 2000 12:00:00 -0000\")
+	   (message-goto-subject)
+	   (insert \"${subject}\")
+	   (message-goto-body)
+	   (insert \"${body}\")
+	   $@
+	   (message-send-and-exit))" >/dev/null 2>&1
     wait ${smtp_dummy_pid}
     notmuch new >/dev/null
 }
-- 
1.7.5.4

^ permalink raw reply related	[flat|nested] 22+ messages in thread
* [PATCH 09/10] test: generate run_emacs script once on test startup
  2011-06-28  4:45 [PATCH 01/10] test: fix argument order of test_expect_equal_file in few tests Dmitry Kurochkin
                   ` (6 preceding siblings ...)
  2011-06-28  4:45 ` [PATCH 08/10] test: set variables using `let' instead of `setq' in Emacs tests Dmitry Kurochkin
@ 2011-06-28  4:45 ` Dmitry Kurochkin
  2011-06-28  4:45 ` [PATCH 10/10] test: use emacsclient(1) for Emacs tests Dmitry Kurochkin
  8 siblings, 0 replies; 22+ messages in thread
From: Dmitry Kurochkin @ 2011-06-28  4:45 UTC (permalink / raw)
  To: notmuch

Instead of generating auxiliary run_emacs script every time
test_emacs is run, do it once in the beginning of the test.

Also, use absolute paths in the script to make it more robust.
---
 test/test-lib.sh |   17 +++++++++++------
 1 files changed, 11 insertions(+), 6 deletions(-)

diff --git a/test/test-lib.sh b/test/test-lib.sh
index 7c8a86f..48bace2 100755
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -839,11 +839,11 @@ test_done () {
 	fi
 }
 
-test_emacs () {
+emacs_generate_script () {
 	# Construct a little test script here for the benefit of the user,
 	# (who can easily run "run_emacs" to get the same emacs environment
 	# for investigating any failures).    
-	cat <<EOF > run_emacs
+	cat <<EOF >"$TMP_DIRECTORY/run_emacs"
 #!/bin/sh
 export PATH=$PATH
 export NOTMUCH_CONFIG=$NOTMUCH_CONFIG
@@ -867,12 +867,15 @@ fi
 # --load		Force loading of notmuch.el and test-lib.el
 
 emacs \$BATCH --no-init-file --no-site-file \
-	--directory ../../emacs --load notmuch.el \
-	--directory .. --load test-lib.el \
+	--directory "$TMP_DIRECTORY/../../emacs" --load notmuch.el \
+	--directory "$TMP_DIRECTORY/.." --load test-lib.el \
 	--eval "(progn \$@)"
 EOF
-	chmod a+x ./run_emacs
-	./run_emacs "$@"
+	chmod a+x "$TMP_DIRECTORY/run_emacs"
+}
+
+test_emacs () {
+	"$TMP_DIRECTORY/run_emacs" "$@"
 }
 
 
@@ -1000,6 +1003,8 @@ primary_email=test_suite@notmuchmail.org
 other_email=test_suite_other@notmuchmail.org;test_suite@otherdomain.org
 EOF
 
+emacs_generate_script
+
 
 # Use -P to resolve symlinks in our working directory so that the cwd
 # in subprocesses like git equals our $PWD (for pathname comparisons).
-- 
1.7.5.4

^ permalink raw reply related	[flat|nested] 22+ messages in thread
* [PATCH 10/10] test: use emacsclient(1) for Emacs tests
  2011-06-28  4:45 [PATCH 01/10] test: fix argument order of test_expect_equal_file in few tests Dmitry Kurochkin
                   ` (7 preceding siblings ...)
  2011-06-28  4:45 ` [PATCH 09/10] test: generate run_emacs script once on test startup Dmitry Kurochkin
@ 2011-06-28  4:45 ` Dmitry Kurochkin
  2011-06-28  4:56   ` [PATCH] " Dmitry Kurochkin
                     ` (2 more replies)
  8 siblings, 3 replies; 22+ messages in thread
From: Dmitry Kurochkin @ 2011-06-28  4:45 UTC (permalink / raw)
  To: notmuch

Before the change, every Emacs test ran in a separate Emacs
instance.  Starting Emacs many times wastes considerable time and
it gets worse as the test suite grows.  The patch solves this by
using a single Emacs server and emacsclient(1) to run multiple
tests.  Emacs server is started on the first test_emacs call and
stopped when test_done is called.  We take care not to leave
orphan Emacs processes behind when test is terminated by whatever
reason: Emacs server runs a watchdog that periodically checks
that the test is still running.

Some tests need to provide user input.  Before the change, this
was done using echo(1) to Emacs stdin.  This no longer works and
instead `standard-input' variable is set accordingly to make
`read' return the appropriate string.
---
 test/emacs       |   10 +++++-----
 test/test-lib.el |   13 +++++++++++++
 test/test-lib.sh |   24 +++++++++++++-----------
 3 files changed, 31 insertions(+), 16 deletions(-)

diff --git a/test/emacs b/test/emacs
index 3d42645..0f5f99c 100755
--- a/test/emacs
+++ b/test/emacs
@@ -259,15 +259,15 @@ test_expect_equal_file OUTPUT EXPECTED
 
 test_begin_subtest "Save attachment from within emacs using notmuch-show-save-attachments"
 # save as archive to test that Emacs does not re-compress .gz
-echo ./attachment1.gz |
-test_emacs '(notmuch-show "id:cf0c4d610911171136h1713aa59w9cf9aa31f052ad0a@mail.gmail.com")
-	    (notmuch-show-save-attachments)' > /dev/null 2>&1
+test_emacs '(let ((standard-input "\"attachment1.gz\""))
+	      (notmuch-show "id:cf0c4d610911171136h1713aa59w9cf9aa31f052ad0a@mail.gmail.com")
+	      (notmuch-show-save-attachments))' > /dev/null 2>&1
 test_expect_equal_file attachment1.gz "$EXPECTED/attachment"
 
 test_begin_subtest "Save attachment from within emacs using notmuch-show-save-part"
 # save as archive to test that Emacs does not re-compress .gz
-echo ./attachment2.gz |
-test_emacs '(notmuch-show-save-part "id:cf0c4d610911171136h1713aa59w9cf9aa31f052ad0a@mail.gmail.com" 5)' > /dev/null 2>&1
+test_emacs '(let ((standard-input "\"attachment2.gz\""))
+	      (notmuch-show-save-part "id:cf0c4d610911171136h1713aa59w9cf9aa31f052ad0a@mail.gmail.com" 5))' > /dev/null 2>&1
 test_expect_equal_file attachment2.gz "$EXPECTED/attachment"
 
 test_begin_subtest "View raw message within emacs"
diff --git a/test/test-lib.el b/test/test-lib.el
index 4e7f5cf..a783936 100644
--- a/test/test-lib.el
+++ b/test/test-lib.el
@@ -23,6 +23,12 @@
 ;; avoid crazy 10-column default of --batch
 (set-frame-width (window-frame (get-buffer-window)) 80)
 
+;; `read-file-name' by default uses `completing-read' function to read
+;; user input.  It does not respect `standard-input' variable which we
+;; use in tests to provide user input.  So replace it with a plain
+;; `read' call.
+(setq read-file-name-function (lambda (&rest _) (read)))
+
 (defun notmuch-test-wait ()
   "Wait for process completion."
   (while (get-buffer-process (current-buffer))
@@ -51,3 +57,10 @@ FILENAME is OUTPUT."
 	  (setq str (concat str (buffer-substring start next-pos))))
 	(setq start next-pos)))
     str))
+
+(defun orphan-watchdog (pid)
+  "Periodically check that the process with id PID is still
+running, quit if it terminated."
+  (if (not (process-attributes pid))
+      (kill-emacs)
+    (run-at-time "1 min" nil 'orphan-watchdog pid)))
diff --git a/test/test-lib.sh b/test/test-lib.sh
index 48bace2..e8f8629 100755
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -829,6 +829,8 @@ test_done () {
 
 	echo
 
+	[ -n "$EMACS_SERVER" ] && test_emacs '(kill-emacs)'
+
 	if [ "$test_failure" = "0" ]; then
 	    if [ "$test_broken" = "0" ]; then	    
 		rm -rf "$remove_tmp"
@@ -848,16 +850,8 @@ emacs_generate_script () {
 export PATH=$PATH
 export NOTMUCH_CONFIG=$NOTMUCH_CONFIG
 
-# We assume that the user will give a command-line argument only if
-# wanting to run in batch mode.
-if [ \$# -gt 0 ]; then
-	BATCH=--batch
-fi
-
 # Here's what we are using here:
 #
-# --batch:		Quit after given commands and print all (messages)
-#
 # --no-init-file	Don't load users ~/.emacs
 #
 # --no-site-file	Don't load the site-wide startup stuff
@@ -866,16 +860,24 @@ fi
 #
 # --load		Force loading of notmuch.el and test-lib.el
 
-emacs \$BATCH --no-init-file --no-site-file \
+emacs --no-init-file --no-site-file \
 	--directory "$TMP_DIRECTORY/../../emacs" --load notmuch.el \
 	--directory "$TMP_DIRECTORY/.." --load test-lib.el \
-	--eval "(progn \$@)"
+	"\$@"
 EOF
 	chmod a+x "$TMP_DIRECTORY/run_emacs"
 }
 
 test_emacs () {
-	"$TMP_DIRECTORY/run_emacs" "$@"
+	if [ -z "$EMACS_SERVER" ]; then
+		EMACS_SERVER="notmuch-server-$$"
+		"$TMP_DIRECTORY/run_emacs" \
+			--daemon \
+			--eval "(setq server-name \"$EMACS_SERVER\")" \
+			--eval "(orphan-watchdog $$)" || return
+	fi
+
+	emacsclient --socket-name="$EMACS_SERVER" --eval "(progn $@)"
 }
 
 
-- 
1.7.5.4

^ permalink raw reply related	[flat|nested] 22+ messages in thread
* [PATCH] test: use emacsclient(1) for Emacs tests
  2011-06-28  4:45 ` [PATCH 10/10] test: use emacsclient(1) for Emacs tests Dmitry Kurochkin
@ 2011-06-28  4:56   ` Dmitry Kurochkin
  2011-06-28 22:14   ` [PATCH 10/10] " Carl Worth
  2012-01-23 19:08   ` Xavier Maillard
  2 siblings, 0 replies; 22+ messages in thread
From: Dmitry Kurochkin @ 2011-06-28  4:56 UTC (permalink / raw)
  To: notmuch

Before the change, every Emacs test ran in a separate Emacs
instance.  Starting Emacs many times wastes considerable time and
it gets worse as the test suite grows.  The patch solves this by
using a single Emacs server and emacsclient(1) to run multiple
tests.  Emacs server is started on the first test_emacs call and
stopped when test_done is called.  We take care not to leave
orphan Emacs processes behind when test is terminated by whatever
reason: Emacs server runs a watchdog that periodically checks
that the test is still running.

Some tests need to provide user input.  Before the change, this
was done using echo(1) to Emacs stdin.  This no longer works and
instead `standard-input' variable is set accordingly to make
`read' return the appropriate string.
---

Amended patch with a more appropriate Emacs server name.  Sorry,
for inconvenience, should go to sleep.

Regards,
  Dmitry

 test/emacs       |   10 +++++-----
 test/test-lib.el |   13 +++++++++++++
 test/test-lib.sh |   24 +++++++++++++-----------
 3 files changed, 31 insertions(+), 16 deletions(-)

diff --git a/test/emacs b/test/emacs
index 3d42645..0f5f99c 100755
--- a/test/emacs
+++ b/test/emacs
@@ -259,15 +259,15 @@ test_expect_equal_file OUTPUT EXPECTED
 
 test_begin_subtest "Save attachment from within emacs using notmuch-show-save-attachments"
 # save as archive to test that Emacs does not re-compress .gz
-echo ./attachment1.gz |
-test_emacs '(notmuch-show "id:cf0c4d610911171136h1713aa59w9cf9aa31f052ad0a@mail.gmail.com")
-	    (notmuch-show-save-attachments)' > /dev/null 2>&1
+test_emacs '(let ((standard-input "\"attachment1.gz\""))
+	      (notmuch-show "id:cf0c4d610911171136h1713aa59w9cf9aa31f052ad0a@mail.gmail.com")
+	      (notmuch-show-save-attachments))' > /dev/null 2>&1
 test_expect_equal_file attachment1.gz "$EXPECTED/attachment"
 
 test_begin_subtest "Save attachment from within emacs using notmuch-show-save-part"
 # save as archive to test that Emacs does not re-compress .gz
-echo ./attachment2.gz |
-test_emacs '(notmuch-show-save-part "id:cf0c4d610911171136h1713aa59w9cf9aa31f052ad0a@mail.gmail.com" 5)' > /dev/null 2>&1
+test_emacs '(let ((standard-input "\"attachment2.gz\""))
+	      (notmuch-show-save-part "id:cf0c4d610911171136h1713aa59w9cf9aa31f052ad0a@mail.gmail.com" 5))' > /dev/null 2>&1
 test_expect_equal_file attachment2.gz "$EXPECTED/attachment"
 
 test_begin_subtest "View raw message within emacs"
diff --git a/test/test-lib.el b/test/test-lib.el
index 4e7f5cf..a783936 100644
--- a/test/test-lib.el
+++ b/test/test-lib.el
@@ -23,6 +23,12 @@
 ;; avoid crazy 10-column default of --batch
 (set-frame-width (window-frame (get-buffer-window)) 80)
 
+;; `read-file-name' by default uses `completing-read' function to read
+;; user input.  It does not respect `standard-input' variable which we
+;; use in tests to provide user input.  So replace it with a plain
+;; `read' call.
+(setq read-file-name-function (lambda (&rest _) (read)))
+
 (defun notmuch-test-wait ()
   "Wait for process completion."
   (while (get-buffer-process (current-buffer))
@@ -51,3 +57,10 @@ FILENAME is OUTPUT."
 	  (setq str (concat str (buffer-substring start next-pos))))
 	(setq start next-pos)))
     str))
+
+(defun orphan-watchdog (pid)
+  "Periodically check that the process with id PID is still
+running, quit if it terminated."
+  (if (not (process-attributes pid))
+      (kill-emacs)
+    (run-at-time "1 min" nil 'orphan-watchdog pid)))
diff --git a/test/test-lib.sh b/test/test-lib.sh
index 48bace2..bcf18ec 100755
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -829,6 +829,8 @@ test_done () {
 
 	echo
 
+	[ -n "$EMACS_SERVER" ] && test_emacs '(kill-emacs)'
+
 	if [ "$test_failure" = "0" ]; then
 	    if [ "$test_broken" = "0" ]; then	    
 		rm -rf "$remove_tmp"
@@ -848,16 +850,8 @@ emacs_generate_script () {
 export PATH=$PATH
 export NOTMUCH_CONFIG=$NOTMUCH_CONFIG
 
-# We assume that the user will give a command-line argument only if
-# wanting to run in batch mode.
-if [ \$# -gt 0 ]; then
-	BATCH=--batch
-fi
-
 # Here's what we are using here:
 #
-# --batch:		Quit after given commands and print all (messages)
-#
 # --no-init-file	Don't load users ~/.emacs
 #
 # --no-site-file	Don't load the site-wide startup stuff
@@ -866,16 +860,24 @@ fi
 #
 # --load		Force loading of notmuch.el and test-lib.el
 
-emacs \$BATCH --no-init-file --no-site-file \
+emacs --no-init-file --no-site-file \
 	--directory "$TMP_DIRECTORY/../../emacs" --load notmuch.el \
 	--directory "$TMP_DIRECTORY/.." --load test-lib.el \
-	--eval "(progn \$@)"
+	"\$@"
 EOF
 	chmod a+x "$TMP_DIRECTORY/run_emacs"
 }
 
 test_emacs () {
-	"$TMP_DIRECTORY/run_emacs" "$@"
+	if [ -z "$EMACS_SERVER" ]; then
+		EMACS_SERVER="notmuch-test-suite-$$"
+		"$TMP_DIRECTORY/run_emacs" \
+			--daemon \
+			--eval "(setq server-name \"$EMACS_SERVER\")" \
+			--eval "(orphan-watchdog $$)" || return
+	fi
+
+	emacsclient --socket-name="$EMACS_SERVER" --eval "(progn $@)"
 }
 
 
-- 
1.7.5.4

^ permalink raw reply related	[flat|nested] 22+ messages in thread
* Re: [PATCH 10/10] test: use emacsclient(1) for Emacs tests
  2011-06-28  4:45 ` [PATCH 10/10] test: use emacsclient(1) for Emacs tests Dmitry Kurochkin
  2011-06-28  4:56   ` [PATCH] " Dmitry Kurochkin
@ 2011-06-28 22:14   ` Carl Worth
  2011-06-28 22:22     ` Dmitry Kurochkin
  2012-01-23 19:08   ` Xavier Maillard
  2 siblings, 1 reply; 22+ messages in thread
From: Carl Worth @ 2011-06-28 22:14 UTC (permalink / raw)
  To: Dmitry Kurochkin, notmuch

[-- Attachment #1: Type: text/plain, Size: 1382 bytes --]

On Tue, 28 Jun 2011 08:45:11 +0400, Dmitry Kurochkin <dmitry.kurochkin@gmail.com> wrote:
> Before the change, every Emacs test ran in a separate Emacs
> instance.  Starting Emacs many times wastes considerable time and
> it gets worse as the test suite grows.  The patch solves this by
> using a single Emacs server and emacsclient(1) to run multiple
> tests. 

Great, great stuff, Dmitry!

I've pushed everything earlier than this patch in this series.

And I'd be fine pushing this one as well, (if Austin cares strongly
about not polling, I'll invite him to improve things from here).

Except that things don't actually work for me with this patch applied.

I'm not getting consistent results from the test suite, (I have seen
both "2 tests failed" and "5 tests failed"). Here are the failures from
a recent run with 5 failures:

 FAIL   decryption, --format=text
 FAIL   decryption, --format=json
 FAIL   decryption, --format=json, --part=4
 FAIL   decrypt attachment (--part=5 --format=raw)
 FAIL   decryption failure with missing key

In each case the actual output was either empty, or an empty JSON
array. In some cases notmuch also gave an error:

Error: search term did not match precisely one message.

Are you not getting these same failures?

Let me know what else I can do to investigate this.

-Carl

-- 
carl.d.worth@intel.com

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

^ permalink raw reply	[flat|nested] 22+ messages in thread
* Re: [PATCH 10/10] test: use emacsclient(1) for Emacs tests
  2011-06-28 22:14   ` [PATCH 10/10] " Carl Worth
@ 2011-06-28 22:22     ` Dmitry Kurochkin
  2011-06-29  0:15       ` Carl Worth
  0 siblings, 1 reply; 22+ messages in thread
From: Dmitry Kurochkin @ 2011-06-28 22:22 UTC (permalink / raw)
  To: Carl Worth, notmuch

On Tue, 28 Jun 2011 15:14:04 -0700, Carl Worth <cworth@cworth.org> wrote:
> On Tue, 28 Jun 2011 08:45:11 +0400, Dmitry Kurochkin <dmitry.kurochkin@gmail.com> wrote:
> > Before the change, every Emacs test ran in a separate Emacs
> > instance.  Starting Emacs many times wastes considerable time and
> > it gets worse as the test suite grows.  The patch solves this by
> > using a single Emacs server and emacsclient(1) to run multiple
> > tests. 
> 
> Great, great stuff, Dmitry!
> 
> I've pushed everything earlier than this patch in this series.
> 
> And I'd be fine pushing this one as well, (if Austin cares strongly
> about not polling, I'll invite him to improve things from here).
> 
> Except that things don't actually work for me with this patch applied.
> 
> I'm not getting consistent results from the test suite, (I have seen
> both "2 tests failed" and "5 tests failed"). Here are the failures from
> a recent run with 5 failures:
> 
>  FAIL   decryption, --format=text
>  FAIL   decryption, --format=json
>  FAIL   decryption, --format=json, --part=4
>  FAIL   decrypt attachment (--part=5 --format=raw)
>  FAIL   decryption failure with missing key
> 
> In each case the actual output was either empty, or an empty JSON
> array. In some cases notmuch also gave an error:
> 
> Error: search term did not match precisely one message.
> 
> Are you not getting these same failures?
> 

Ouch.  I never saw these.  Do you get these in crypto tests only?

Regards,
  Dmitry

> Let me know what else I can do to investigate this.
> 
> -Carl
> 
> -- 
> carl.d.worth@intel.com

^ permalink raw reply	[flat|nested] 22+ messages in thread
* Re: [PATCH 10/10] test: use emacsclient(1) for Emacs tests
  2011-06-28 22:22     ` Dmitry Kurochkin
@ 2011-06-29  0:15       ` Carl Worth
  2011-06-29  7:34         ` Austin Clements
  0 siblings, 1 reply; 22+ messages in thread
From: Carl Worth @ 2011-06-29  0:15 UTC (permalink / raw)
  To: Dmitry Kurochkin, notmuch

[-- Attachment #1: Type: text/plain, Size: 1250 bytes --]

On Wed, 29 Jun 2011 02:22:31 +0400, Dmitry Kurochkin <dmitry.kurochkin@gmail.com> wrote:
> Ouch.  I never saw these.  Do you get these in crypto tests only?

Yes.

I just found the bug. The crypto tests use emacs_deliver_message to
generate signed/encrypted messages, but that function doesn't call
increment_mtime between running emacs and running "notmuch new".

That means that if the emacs_deliver_message code is fast enough, (less
than 1 second), and if the test suite is being run on a lame filesystem
without sub-second timestamp granularity (including ext3fs) that the
"notmuch new" invocation will not see the mail[*].

I've added a call to increment_mtime and the failures go away. So I've
pushed the patch to use emacsclient now.

Thanks very much for helping our test suite run faster!

-Carl

[*] And, yes, this does mean that there are race conditions under which
mail can be delivered and notmuch won't see it, (unless subsequent mails
are delivered to the same directory). Austin has a plan to fix the use
of the mtime timestamp in notmuch, (never storing the current mtime in
the database if it's the same as the current time), that should
hopefully close this race window.

-- 
carl.d.worth@intel.com

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

^ permalink raw reply	[flat|nested] 22+ messages in thread
* Re: [PATCH 10/10] test: use emacsclient(1) for Emacs tests
  2011-06-29  0:15       ` Carl Worth
@ 2011-06-29  7:34         ` Austin Clements
  2011-06-29 14:20           ` Carl Worth
  0 siblings, 1 reply; 22+ messages in thread
From: Austin Clements @ 2011-06-29  7:34 UTC (permalink / raw)
  To: Carl Worth; +Cc: notmuch

On Tue, Jun 28, 2011 at 8:15 PM, Carl Worth <cworth@cworth.org> wrote:
> That means that if the emacs_deliver_message code is fast enough, (less
> than 1 second), and if the test suite is being run on a lame filesystem
> without sub-second timestamp granularity (including ext3fs) that the
> "notmuch new" invocation will not see the mail[*].

Or, really, any file system, since notmuch doesn't currently take
advantage of sub-second mtimes.

> [*] And, yes, this does mean that there are race conditions under which
> mail can be delivered and notmuch won't see it, (unless subsequent mails
> are delivered to the same directory). Austin has a plan to fix the use
> of the mtime timestamp in notmuch, (never storing the current mtime in
> the database if it's the same as the current time), that should
> hopefully close this race window.

Posted.

^ permalink raw reply	[flat|nested] 22+ messages in thread
* Re: [PATCH 10/10] test: use emacsclient(1) for Emacs tests
  2011-06-29  7:34         ` Austin Clements
@ 2011-06-29 14:20           ` Carl Worth
  0 siblings, 0 replies; 22+ messages in thread
From: Carl Worth @ 2011-06-29 14:20 UTC (permalink / raw)
  To: Austin Clements; +Cc: notmuch

[-- Attachment #1: Type: text/plain, Size: 641 bytes --]

On Wed, 29 Jun 2011 03:34:58 -0400, Austin Clements <amdragon@mit.edu> wrote:
> Or, really, any file system, since notmuch doesn't currently take
> advantage of sub-second mtimes.

Oh, ouch. I hadn't realized that I'd codified the 1-second granularity
mtime into the notmuch API.

Oh well, we could extend that with a floating-point API as needed.

> > Austin has a plan to fix the use of the mtime timestamp in notmuch,
> > (never storing the current mtime in the database if it's the same as
> > the current time), that should hopefully close this race window.
> 
> Posted.

Thanks!

-Carl

-- 
carl.d.worth@intel.com

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

^ permalink raw reply	[flat|nested] 22+ messages in thread
* [PATCH] test/emacs-large-search-buffer: correct typo (EXPEXTED -> EXPECTED)
  2011-06-28  4:45 ` [PATCH 05/10] test: save buffer content to file instead of printing it in Emacs tests Dmitry Kurochkin
@ 2012-01-23  4:26   ` Pieter Praet
  2012-01-23 11:48     ` David Bremner
  0 siblings, 1 reply; 22+ messages in thread
From: Pieter Praet @ 2012-01-23  4:26 UTC (permalink / raw)
  To: Dmitry Kurochkin; +Cc: Notmuch Mail

introduced in commit 3b24b396
---
 test/emacs-large-search-buffer |   12 ++++++------
 1 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/test/emacs-large-search-buffer b/test/emacs-large-search-buffer
index 6095e9d..4351e33 100755
--- a/test/emacs-large-search-buffer
+++ b/test/emacs-large-search-buffer
@@ -19,25 +19,25 @@ done
 notmuch new > /dev/null
 
 test_begin_subtest "Ensure that emacs doesn't drop results"
-notmuch search '*' > EXPEXTED
-sed -i -e 's/^thread:[0-9a-f]*  //' -e 's/;//' -e 's/xx*/[BLOB]/' EXPEXTED
-echo 'End of search results.' >> EXPEXTED
+notmuch search '*' > EXPECTED
+sed -i -e 's/^thread:[0-9a-f]*  //' -e 's/;//' -e 's/xx*/[BLOB]/' EXPECTED
+echo 'End of search results.' >> EXPECTED
 
 test_emacs '(notmuch-search "*")
 	    (notmuch-test-wait)
 	    (test-output)'
 sed -i -e s',  *, ,g' -e 's/xxx*/[BLOB]/g' OUTPUT
-test_expect_equal_file OUTPUT EXPEXTED
+test_expect_equal_file OUTPUT EXPECTED
 
 test_begin_subtest "Ensure that emacs doesn't drop error messages"
 test_emacs '(notmuch-search "--this-option-does-not-exist")
 	    (notmuch-test-wait)
 	    (test-output)'
-cat <<EOF >EXPEXTED
+cat <<EOF >EXPECTED
 Error: Unexpected output from notmuch search:
 Unrecognized option: --this-option-does-not-exist
 End of search results. (process returned 1)
 EOF
-test_expect_equal_file OUTPUT EXPEXTED
+test_expect_equal_file OUTPUT EXPECTED
 
 test_done
-- 
1.7.8.1

^ permalink raw reply related	[flat|nested] 22+ messages in thread
* Re: [PATCH] test/emacs-large-search-buffer: correct typo (EXPEXTED -> EXPECTED)
  2012-01-23  4:26   ` [PATCH] test/emacs-large-search-buffer: correct typo (EXPEXTED -> EXPECTED) Pieter Praet
@ 2012-01-23 11:48     ` David Bremner
  0 siblings, 0 replies; 22+ messages in thread
From: David Bremner @ 2012-01-23 11:48 UTC (permalink / raw)
  To: Pieter Praet; +Cc: Notmuch Mail

On Mon, 23 Jan 2012 05:26:10 +0100, Pieter Praet <pieter@praet.org> wrote:
> introduced in commit 3b24b396
> ---
>  test/emacs-large-search-buffer |   12 ++++++------
>  1 files changed, 6 insertions(+), 6 deletions(-)

pushed

d

^ permalink raw reply	[flat|nested] 22+ messages in thread
* Re: [PATCH 08/10] test: set variables using `let' instead of `setq' in Emacs tests
  2011-06-28  4:45 ` [PATCH 08/10] test: set variables using `let' instead of `setq' in Emacs tests Dmitry Kurochkin
@ 2012-01-23 19:05   ` Xavier Maillard
  0 siblings, 0 replies; 22+ messages in thread
From: Xavier Maillard @ 2012-01-23 19:05 UTC (permalink / raw)
  To: Dmitry Kurochkin, notmuch

On Tue, 28 Jun 2011 08:45:09 +0400, Dmitry Kurochkin <dmitry.kurochkin@gmail.com> wrote:
> Using `setq' for setting variables in Emacs tests affect other
> tests that may run in the same Emacs environment.  Currently it
> works because each test is run in a separate Emacs instance.  But
> in the future multiple tests will run in a single Emacs instance.
> The patch changes all variables to use `let', so the scope of the
> change is limited to a single test.

+1 (definetely)

/Xavier

^ permalink raw reply	[flat|nested] 22+ messages in thread
* Re: [PATCH 10/10] test: use emacsclient(1) for Emacs tests
  2011-06-28  4:45 ` [PATCH 10/10] test: use emacsclient(1) for Emacs tests Dmitry Kurochkin
  2011-06-28  4:56   ` [PATCH] " Dmitry Kurochkin
  2011-06-28 22:14   ` [PATCH 10/10] " Carl Worth
@ 2012-01-23 19:08   ` Xavier Maillard
  2012-01-25  0:47     ` Pieter Praet
  2012-01-25  1:01     ` David Bremner
  2 siblings, 2 replies; 22+ messages in thread
From: Xavier Maillard @ 2012-01-23 19:08 UTC (permalink / raw)
  To: Dmitry Kurochkin, notmuch

On Tue, 28 Jun 2011 08:45:11 +0400, Dmitry Kurochkin <dmitry.kurochkin@gmail.com> wrote:
> Before the change, every Emacs test ran in a separate Emacs
> instance.  Starting Emacs many times wastes considerable time and
> it gets worse as the test suite grows.  The patch solves this by
> using a single Emacs server and emacsclient(1) to run multiple
> tests.  Emacs server is started on the first test_emacs call and
> stopped when test_done is called.  We take care not to leave
> orphan Emacs processes behind when test is terminated by whatever
> reason: Emacs server runs a watchdog that periodically checks
> that the test is still running.
> 
> Some tests need to provide user input.  Before the change, this
> was done using echo(1) to Emacs stdin.  This no longer works and
> instead `standard-input' variable is set accordingly to make
> `read' return the appropriate string.

I really like this too. 

+1

/Xavier

^ permalink raw reply	[flat|nested] 22+ messages in thread
* Re: [PATCH 10/10] test: use emacsclient(1) for Emacs tests
  2012-01-23 19:08   ` Xavier Maillard
@ 2012-01-25  0:47     ` Pieter Praet
  2012-01-25  1:01     ` David Bremner
  1 sibling, 0 replies; 22+ messages in thread
From: Pieter Praet @ 2012-01-25  0:47 UTC (permalink / raw)
  To: Xavier Maillard, Dmitry Kurochkin, notmuch

On Mon, 23 Jan 2012 20:08:59 +0100, Xavier Maillard <xavier@maillard.im> wrote:
> On Tue, 28 Jun 2011 08:45:11 +0400, Dmitry Kurochkin <dmitry.kurochkin@gmail.com> wrote:
> > Before the change, every Emacs test ran in a separate Emacs
> > instance.  Starting Emacs many times wastes considerable time and
> > it gets worse as the test suite grows.  The patch solves this by
> > using a single Emacs server and emacsclient(1) to run multiple
> > tests.  Emacs server is started on the first test_emacs call and
> > stopped when test_done is called.  We take care not to leave
> > orphan Emacs processes behind when test is terminated by whatever
> > reason: Emacs server runs a watchdog that periodically checks
> > that the test is still running.
> > 
> > Some tests need to provide user input.  Before the change, this
> > was done using echo(1) to Emacs stdin.  This no longer works and
> > instead `standard-input' variable is set accordingly to make
> > `read' return the appropriate string.
> 
> I really like this too. 
> 
> +1
> 

Actually, these have all been pushed already, around June 2011.
See commits e4fc21e8..a854d06e.

I necrobumped this thread to keep my typo fix [1] in proper context.

Which leaves me to ponder: should these kinds of fixes be sent in reply
to the original patch (and potentially cause confusion), or would it be
better to just start a new thread?


> /Xavier
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch


Peace

-- 
Pieter

[1] id:"1327292770-9528-1-git-send-email-pieter@praet.org"

^ permalink raw reply	[flat|nested] 22+ messages in thread
* Re: [PATCH 10/10] test: use emacsclient(1) for Emacs tests
  2012-01-23 19:08   ` Xavier Maillard
  2012-01-25  0:47     ` Pieter Praet
@ 2012-01-25  1:01     ` David Bremner
  1 sibling, 0 replies; 22+ messages in thread
From: David Bremner @ 2012-01-25  1:01 UTC (permalink / raw)
  To: Xavier Maillard, Dmitry Kurochkin, notmuch

On Mon, 23 Jan 2012 20:08:59 +0100, Xavier Maillard <xavier@maillard.im> wrote:
> 
> I really like this too. 
> 
> +1
> 

I appreciate your enthusiasm, but, it seems like both of these old
patches are applied to master some time ago.

You can use e.g. git shortlog --author=Dmitry  to have a look.

d

^ permalink raw reply	[flat|nested] 22+ messages in thread
end of thread, other threads:[~2012-01-25  1:01 UTC | newest]

Thread overview: 22+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2011-06-28  4:45 [PATCH 01/10] test: fix argument order of test_expect_equal_file in few tests Dmitry Kurochkin
2011-06-28  4:45 ` [PATCH 02/10] test: do not set `message-signature' in test_emacs Dmitry Kurochkin
2011-06-28  4:45 ` [PATCH 03/10] test: cleanup test_emacs Dmitry Kurochkin
2011-06-28  4:45 ` [PATCH 04/10] test: wrap and indent test_emacs calls Dmitry Kurochkin
2011-06-28  4:45 ` [PATCH 05/10] test: save buffer content to file instead of printing it in Emacs tests Dmitry Kurochkin
2012-01-23  4:26   ` [PATCH] test/emacs-large-search-buffer: correct typo (EXPEXTED -> EXPECTED) Pieter Praet
2012-01-23 11:48     ` David Bremner
2011-06-28  4:45 ` [PATCH 06/10] test: remove some sed(1) calls in Emacs tests Dmitry Kurochkin
2011-06-28  4:45 ` [PATCH 07/10] test: use emacs_deliver_message in Emacs SMTP send test Dmitry Kurochkin
2011-06-28  4:45 ` [PATCH 08/10] test: set variables using `let' instead of `setq' in Emacs tests Dmitry Kurochkin
2012-01-23 19:05   ` Xavier Maillard
2011-06-28  4:45 ` [PATCH 09/10] test: generate run_emacs script once on test startup Dmitry Kurochkin
2011-06-28  4:45 ` [PATCH 10/10] test: use emacsclient(1) for Emacs tests Dmitry Kurochkin
2011-06-28  4:56   ` [PATCH] " Dmitry Kurochkin
2011-06-28 22:14   ` [PATCH 10/10] " Carl Worth
2011-06-28 22:22     ` Dmitry Kurochkin
2011-06-29  0:15       ` Carl Worth
2011-06-29  7:34         ` Austin Clements
2011-06-29 14:20           ` Carl Worth
2012-01-23 19:08   ` Xavier Maillard
2012-01-25  0:47     ` Pieter Praet
2012-01-25  1:01     ` David Bremner

Code repositories for project(s) associated with this public inbox

	https://yhetil.org/notmuch.git/

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).