test/emacs: reduce use of smtp-dummy, notmuch-hello

test/emacs: reduce use of smtp-dummy, notmuch-hello

[David Bremner]

In addition to the timestamp related failures in the test suite,
there were some intermittent problems [1] that turned out to be failure of
the emacs_deliver_message function.  In this series we try to make
mail delivery from emacs more robust (mainly replace actual delivery
with fcc)

[1]: https://buildd.debian.org/fetch.cgi?pkg=notmuch&arch=i386&ver=0.17%7Erc3-1&stamp=1386412394&file=log

As I write this, the crypto tests have run successfully for 9155
times. Of course that doesn't really prove anything, but it is twice
as many as the previously longest interval before failure.

[PATCH 1/2] test: add emacs_fcc_message that does not use smtp-dummy

[David Bremner]

Most of the tests previously using emacs_deliver_message do not use
the actual transmitted message, so we replace it with a simpler (and
presumably more reliable function) that only saves (and indexes) an
fcc copy of the message.
---
 test/crypto      | 10 +++-------
 test/emacs       |  4 ++--
 test/json        |  2 +-
 test/sexp        |  2 +-
 test/test-lib.sh | 29 +++++++++++++++++++++++++++++
 5 files changed, 36 insertions(+), 11 deletions(-)

diff --git a/test/crypto b/test/crypto
index 9e5ff4f..477b397 100755
--- a/test/crypto
+++ b/test/crypto
@@ -28,12 +28,8 @@ add_gnupg_home
 # get key fingerprint
 FINGERPRINT=$(gpg --no-tty --list-secret-keys --with-colons --fingerprint | grep '^fpr:' | cut -d: -f10)
 
-# for some reason this is needed for emacs_deliver_message to work,
-# although I can't figure out why
-add_email_corpus
-
 test_expect_success 'emacs delivery of signed message' \
-'emacs_deliver_message \
+'emacs_fcc_message \
     "test signed message 001" \
     "This is a test signed message." \
     "(mml-secure-message-sign)"'
@@ -143,7 +139,7 @@ cat <<EOF >TESTATTACHMENT
 This is a test file.
 EOF
 test_expect_success 'emacs delivery of encrypted message with attachment' \
-'emacs_deliver_message \
+'emacs_fcc_message \
     "test encrypted message 001" \
     "This is a test encrypted message.\n" \
     "(mml-attach-file \"TESTATTACHMENT\") (mml-secure-message-encrypt)"'
@@ -270,7 +266,7 @@ test_expect_equal_json \
 mv "${GNUPGHOME}"{.bak,}
 
 test_expect_success 'emacs delivery of encrypted + signed message' \
-'emacs_deliver_message \
+'emacs_fcc_message \
     "test encrypted message 002" \
     "This is another test encrypted message.\n" \
     "(mml-secure-message-sign-encrypt)"'
diff --git a/test/emacs b/test/emacs
index 1b1ebe3..863219d 100755
--- a/test/emacs
+++ b/test/emacs
@@ -771,7 +771,7 @@ test_expect_equal_file OUTPUT EXPECTED
 
 test_begin_subtest "Do not call notmuch for non-inlinable application/mpeg parts"
 id='message-with-application/mpeg-attachment@notmuchmail.org'
-emacs_deliver_message \
+emacs_fcc_message \
     'Message with application/mpeg attachment' \
     '' \
     "(message-goto-eoh)
@@ -786,7 +786,7 @@ test_expect_equal $(notmuch_counter_value) 1
 
 test_begin_subtest "Do not call notmuch for non-inlinable audio/mpeg parts"
 id='message-with-audio/mpeg-attachment@notmuchmail.org'
-emacs_deliver_message \
+emacs_fcc_message \
     'Message with audio/mpeg attachment' \
     '' \
     "(message-goto-eoh)
diff --git a/test/json b/test/json
index e07a290..c1cf649 100755
--- a/test/json
+++ b/test/json
@@ -38,7 +38,7 @@ test_expect_equal_json "$output" "[[[{\"id\": \"${gen_msg_id}\", \"match\": true
 test_begin_subtest "Show message: json, inline attachment filename"
 subject='json-show-inline-attachment-filename'
 id="json-show-inline-attachment-filename@notmuchmail.org"
-emacs_deliver_message \
+emacs_fcc_message \
     "$subject" \
     'This is a test message with inline attachment with a filename' \
     "(mml-attach-file \"$TEST_DIRECTORY/README\" nil nil \"inline\")
diff --git a/test/sexp b/test/sexp
index be815e1..667e319 100755
--- a/test/sexp
+++ b/test/sexp
@@ -29,7 +29,7 @@ test_expect_equal "$output" "((((:id \"${gen_msg_id}\" :match t :excluded nil :f
 test_begin_subtest "Show message: sexp, inline attachment filename"
 subject='sexp-show-inline-attachment-filename'
 id="sexp-show-inline-attachment-filename@notmuchmail.org"
-emacs_deliver_message \
+emacs_fcc_message \
     "$subject" \
     'This is a test message with inline attachment with a filename' \
     "(mml-attach-file \"$TEST_DIRECTORY/README\" nil nil \"inline\")
diff --git a/test/test-lib.sh b/test/test-lib.sh
index 53cb947..66c0720 100644
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -470,6 +470,35 @@ emacs_deliver_message ()
     notmuch new >/dev/null
 }
 
+# Pretend to deliver a message with emacs. Really save it to a file
+# and add it to the database
+#
+# Uses emacs to generate and deliver a message to the mail store.
+# Accepts arbitrary extra emacs/elisp functions to modify the message
+# before sending, which is useful to doing things like attaching files
+# to the message and encrypting/signing.
+emacs_fcc_message ()
+{
+    local subject="$1"
+    local body="$2"
+    shift 2
+    # before we can send a message, we have to prepare the FCC maildir
+    mkdir -p "$MAIL_DIR"/sent/{cur,new,tmp}
+
+    test_emacs \
+	"(let ((message-send-mail-function (lambda () t)))
+	   (notmuch-mua-mail)
+	   (message-goto-to)
+	   (insert \"test_suite@notmuchmail.org\nDate: 01 Jan 2000 12:00:00 -0000\")
+	   (message-goto-subject)
+	   (insert \"${subject}\")
+	   (message-goto-body)
+	   (insert \"${body}\")
+	   $@
+	   (message-send-and-exit))" || return 1
+    notmuch new >/dev/null
+}
+
 # Generate a corpus of email and add it to the database.
 #
 # This corpus is fixed, (it happens to be 50 messages from early in
-- 
1.8.4.3

[PATCH 2/2] test: remove call to notmuch-hello from emacs_deliver_message

[David Bremner]

There is an obscure bug in notmuch-hello that very occasionally causes
emacs_deliver_message to fail. Since it it doesn't serve any actual
purpose in the function we delete it, and leave tracking down the the
bug for another day.
---
 test/test-lib.sh | 1 -
 1 file changed, 1 deletion(-)

diff --git a/test/test-lib.sh b/test/test-lib.sh
index 66c0720..1bb5610 100644
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -452,7 +452,6 @@ emacs_deliver_message ()
                (mail-host-address \"example.com\")
 	       (smtpmail-smtp-server \"localhost\")
 	       (smtpmail-smtp-service \"25025\"))
-	   (notmuch-hello)
 	   (notmuch-mua-mail)
 	   (message-goto-to)
 	   (insert \"test_suite@notmuchmail.org\nDate: 01 Jan 2000 12:00:00 -0000\")
-- 
1.8.4.3

Re: test/emacs: reduce use of smtp-dummy, notmuch-hello

[Tomi Ollila]

On Sun, Dec 15 2013, David Bremner <david@tethera.net> wrote:

> In addition to the timestamp related failures in the test suite,
> there were some intermittent problems [1] that turned out to be failure of
> the emacs_deliver_message function.  In this series we try to make
> mail delivery from emacs more robust (mainly replace actual delivery
> with fcc)
>
> [1]: https://buildd.debian.org/fetch.cgi?pkg=notmuch&arch=i386&ver=0.17%7Erc3-1&stamp=1386412394&file=log
>
> As I write this, the crypto tests have run successfully for 9155
> times. Of course that doesn't really prove anything, but it is twice
> as many as the previously longest interval before failure.

series LGTM.

Tomi

(sent using fingerterm on N9, review quality may vary ;/)

Re: [PATCH 1/2] test: add emacs_fcc_message that does not use smtp-dummy

[Mark Walters]

Hi

> Most of the tests previously using emacs_deliver_message do not use
> the actual transmitted message, so we replace it with a simpler (and
> presumably more reliable function) that only saves (and indexes) an
> fcc copy of the message.

I get a test failure with this patch, but I have a tentative fix. It may
be a mis-configuration in some other part of my setup. 

My computer has a hostname of e4300 (which perhaps should be a FQDN?)
Anyway when running with this patch I get an error for the crypto test
as

 FAIL   reply to encrypted message
        --- crypto.13.expected  2013-12-22 11:32:24.117104674 +0000
        +++ crypto.13.output    2013-12-22 11:32:24.117104674 +0000
        @@ -1,5 +1,7 @@
         From: Notmuch Test Suite <test_suite@notmuchmail.org>
         Subject: Re: test encrypted message 002
        +       <871u15xgs8.fsf@e4300.i-did-not-set--mail-host-address--so-tickle-me>
        +       <871u15xgs8.fsf@e4300.i-did-not-set--mail-host-address--so-tickle-me>
         
         On 01 Jan 2000 12:00:00 -0000, Notmuch Test Suite <test_suite@notmuchmail.org> wrote:
         > This is another test encrypted message.

If I add the line

(mail-host-address \"example.com\"))

inside the let in emacs_fcc_message below then it passes. All tests also
pass on current master.

Best wishes

Mark



>  test/crypto      | 10 +++-------
>  test/emacs       |  4 ++--
>  test/json        |  2 +-
>  test/sexp        |  2 +-
>  test/test-lib.sh | 29 +++++++++++++++++++++++++++++
>  5 files changed, 36 insertions(+), 11 deletions(-)
>
> diff --git a/test/crypto b/test/crypto
> index 9e5ff4f..477b397 100755
> --- a/test/crypto
> +++ b/test/crypto
> @@ -28,12 +28,8 @@ add_gnupg_home
>  # get key fingerprint
>  FINGERPRINT=$(gpg --no-tty --list-secret-keys --with-colons --fingerprint | grep '^fpr:' | cut -d: -f10)
>  
> -# for some reason this is needed for emacs_deliver_message to work,
> -# although I can't figure out why
> -add_email_corpus
> -
>  test_expect_success 'emacs delivery of signed message' \
> -'emacs_deliver_message \
> +'emacs_fcc_message \
>      "test signed message 001" \
>      "This is a test signed message." \
>      "(mml-secure-message-sign)"'
> @@ -143,7 +139,7 @@ cat <<EOF >TESTATTACHMENT
>  This is a test file.
>  EOF
>  test_expect_success 'emacs delivery of encrypted message with attachment' \
> -'emacs_deliver_message \
> +'emacs_fcc_message \
>      "test encrypted message 001" \
>      "This is a test encrypted message.\n" \
>      "(mml-attach-file \"TESTATTACHMENT\") (mml-secure-message-encrypt)"'
> @@ -270,7 +266,7 @@ test_expect_equal_json \
>  mv "${GNUPGHOME}"{.bak,}
>  
>  test_expect_success 'emacs delivery of encrypted + signed message' \
> -'emacs_deliver_message \
> +'emacs_fcc_message \
>      "test encrypted message 002" \
>      "This is another test encrypted message.\n" \
>      "(mml-secure-message-sign-encrypt)"'
> diff --git a/test/emacs b/test/emacs
> index 1b1ebe3..863219d 100755
> --- a/test/emacs
> +++ b/test/emacs
> @@ -771,7 +771,7 @@ test_expect_equal_file OUTPUT EXPECTED
>  
>  test_begin_subtest "Do not call notmuch for non-inlinable application/mpeg parts"
>  id='message-with-application/mpeg-attachment@notmuchmail.org'
> -emacs_deliver_message \
> +emacs_fcc_message \
>      'Message with application/mpeg attachment' \
>      '' \
>      "(message-goto-eoh)
> @@ -786,7 +786,7 @@ test_expect_equal $(notmuch_counter_value) 1
>  
>  test_begin_subtest "Do not call notmuch for non-inlinable audio/mpeg parts"
>  id='message-with-audio/mpeg-attachment@notmuchmail.org'
> -emacs_deliver_message \
> +emacs_fcc_message \
>      'Message with audio/mpeg attachment' \
>      '' \
>      "(message-goto-eoh)
> diff --git a/test/json b/test/json
> index e07a290..c1cf649 100755
> --- a/test/json
> +++ b/test/json
> @@ -38,7 +38,7 @@ test_expect_equal_json "$output" "[[[{\"id\": \"${gen_msg_id}\", \"match\": true
>  test_begin_subtest "Show message: json, inline attachment filename"
>  subject='json-show-inline-attachment-filename'
>  id="json-show-inline-attachment-filename@notmuchmail.org"
> -emacs_deliver_message \
> +emacs_fcc_message \
>      "$subject" \
>      'This is a test message with inline attachment with a filename' \
>      "(mml-attach-file \"$TEST_DIRECTORY/README\" nil nil \"inline\")
> diff --git a/test/sexp b/test/sexp
> index be815e1..667e319 100755
> --- a/test/sexp
> +++ b/test/sexp
> @@ -29,7 +29,7 @@ test_expect_equal "$output" "((((:id \"${gen_msg_id}\" :match t :excluded nil :f
>  test_begin_subtest "Show message: sexp, inline attachment filename"
>  subject='sexp-show-inline-attachment-filename'
>  id="sexp-show-inline-attachment-filename@notmuchmail.org"
> -emacs_deliver_message \
> +emacs_fcc_message \
>      "$subject" \
>      'This is a test message with inline attachment with a filename' \
>      "(mml-attach-file \"$TEST_DIRECTORY/README\" nil nil \"inline\")
> diff --git a/test/test-lib.sh b/test/test-lib.sh
> index 53cb947..66c0720 100644
> --- a/test/test-lib.sh
> +++ b/test/test-lib.sh
> @@ -470,6 +470,35 @@ emacs_deliver_message ()
>      notmuch new >/dev/null
>  }
>  
> +# Pretend to deliver a message with emacs. Really save it to a file
> +# and add it to the database
> +#
> +# Uses emacs to generate and deliver a message to the mail store.
> +# Accepts arbitrary extra emacs/elisp functions to modify the message
> +# before sending, which is useful to doing things like attaching files
> +# to the message and encrypting/signing.
> +emacs_fcc_message ()
> +{
> +    local subject="$1"
> +    local body="$2"
> +    shift 2
> +    # before we can send a message, we have to prepare the FCC maildir
> +    mkdir -p "$MAIL_DIR"/sent/{cur,new,tmp}
> +
> +    test_emacs \
> +	"(let ((message-send-mail-function (lambda () t)))
> +	   (notmuch-mua-mail)
> +	   (message-goto-to)
> +	   (insert \"test_suite@notmuchmail.org\nDate: 01 Jan 2000 12:00:00 -0000\")
> +	   (message-goto-subject)
> +	   (insert \"${subject}\")
> +	   (message-goto-body)
> +	   (insert \"${body}\")
> +	   $@
> +	   (message-send-and-exit))" || return 1
> +    notmuch new >/dev/null
> +}
> +
>  # Generate a corpus of email and add it to the database.
>  #
>  # This corpus is fixed, (it happens to be 50 messages from early in
> -- 
> 1.8.4.3
>
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH 1/2] test: add emacs_fcc_message that does not use smtp-dummy

[David Bremner]

Mark Walters <markwalters1009@gmail.com> writes:
>
> If I add the line
>
> (mail-host-address \"example.com\"))
>
> inside the let in emacs_fcc_message below then it passes. All tests also
> pass on current master.

I pushed the version suggested by Mark to the release branch. 

Depending on the response to 

     id:87bo089y0t.fsf@maritornes.cs.unb.ca

I'll either wait for a fix for the other intmittent test failure, or tag
another release candidate.

d