[PATCH] test: new: Fix intermittent test failures with --debug

[PATCH] test: new: Fix intermittent test failures with --debug

[Ethan Glasser-Camp]

Although messages are created in a particular order, it seems that
when they are created on a tmpfs, they do not always come back in the
same order, leading to the same files being ignored but being output
in a different order. This causes the test to fail because the outputs
being compared are the same.

Fix the failures by sorting the output of notmuch --debug and
comparing this to a hand-sorted version of its output.

Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
---
 test/new |   10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/test/new b/test/new
index cc2af72..587aa11 100755
--- a/test/new
+++ b/test/new
@@ -189,17 +189,17 @@ touch "${MAIL_DIR}"/.git # change .git's mtime for notmuch new to rescan.
 mkdir -p "${MAIL_DIR}"/one/two/three/.git
 notmuch new > /dev/null # ensure that files/folders will be printed in ASCII order.
 touch "${MAIL_DIR}"/{one,one/two,one/two/three}/ignored_file
-output=$(NOTMUCH_NEW --debug 2>&1)
+output=$(NOTMUCH_NEW --debug 2>&1 | sort)
 test_expect_equal "$output" \
 "(D) add_files_recursive, pass 1: explicitly ignoring ${MAIL_DIR}/.git
 (D) add_files_recursive, pass 1: explicitly ignoring ${MAIL_DIR}/one/two/three/.git
-(D) add_files_recursive, pass 2: explicitly ignoring ${MAIL_DIR}/one/two/three/.git
-(D) add_files_recursive, pass 2: explicitly ignoring ${MAIL_DIR}/one/two/three/ignored_file
-(D) add_files_recursive, pass 2: explicitly ignoring ${MAIL_DIR}/one/two/ignored_file
-(D) add_files_recursive, pass 2: explicitly ignoring ${MAIL_DIR}/one/ignored_file
 (D) add_files_recursive, pass 2: explicitly ignoring ${MAIL_DIR}/.git
 (D) add_files_recursive, pass 2: explicitly ignoring ${MAIL_DIR}/.ignored_hidden_file
 (D) add_files_recursive, pass 2: explicitly ignoring ${MAIL_DIR}/ignored_file
+(D) add_files_recursive, pass 2: explicitly ignoring ${MAIL_DIR}/one/ignored_file
+(D) add_files_recursive, pass 2: explicitly ignoring ${MAIL_DIR}/one/two/ignored_file
+(D) add_files_recursive, pass 2: explicitly ignoring ${MAIL_DIR}/one/two/three/.git
+(D) add_files_recursive, pass 2: explicitly ignoring ${MAIL_DIR}/one/two/three/ignored_file
 No new mail."
 
 
-- 
1.7.9.5

Re: [PATCH] test: new: Fix intermittent test failures with --debug

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 672 bytes --]

On Sun, Oct 21 2012, Ethan Glasser-Camp <ethan.glasser.camp@gmail.com> wrote:
> Although messages are created in a particular order, it seems that
> when they are created on a tmpfs, they do not always come back in the
> same order, leading to the same files being ignored but being output
> in a different order. This causes the test to fail because the outputs
> being compared are the same.
>
> Fix the failures by sorting the output of notmuch --debug and
> comparing this to a hand-sorted version of its output.

This seems to fix a failing test that I'm seeing right now, at least in
the couple of times that I've re-run the tests after applying this
patch.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: [PATCH] test: new: Fix intermittent test failures with --debug

[Austin Clements]

LGTM.

This is closely related to another problem in the test suite that
causes frequent failures for me when I run on tmpfs: generate_message
creates messages with the same time stamp by default, which means that
the output of date-sorted notmuch search is technically
non-deterministic.  I suspect this doesn't cause problems most of the
time or for most people because messages get added in the order
they're read from the file system (of course, this isn't guaranteed in
general and doesn't hold on tmpfs), which assigns them monotonically
increasing docids, which stabilizes the search results (though I don't
think this is guaranteed either).

Quoth Ethan Glasser-Camp on Oct 21 at  9:14 am:
> Although messages are created in a particular order, it seems that
> when they are created on a tmpfs, they do not always come back in the
> same order, leading to the same files being ignored but being output
> in a different order. This causes the test to fail because the outputs
> being compared are the same.
> 
> Fix the failures by sorting the output of notmuch --debug and
> comparing this to a hand-sorted version of its output.
> 
> Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
> ---
>  test/new |   10 +++++-----
>  1 file changed, 5 insertions(+), 5 deletions(-)
> 
> diff --git a/test/new b/test/new
> index cc2af72..587aa11 100755
> --- a/test/new
> +++ b/test/new
> @@ -189,17 +189,17 @@ touch "${MAIL_DIR}"/.git # change .git's mtime for notmuch new to rescan.
>  mkdir -p "${MAIL_DIR}"/one/two/three/.git
>  notmuch new > /dev/null # ensure that files/folders will be printed in ASCII order.
>  touch "${MAIL_DIR}"/{one,one/two,one/two/three}/ignored_file
> -output=$(NOTMUCH_NEW --debug 2>&1)
> +output=$(NOTMUCH_NEW --debug 2>&1 | sort)
>  test_expect_equal "$output" \
>  "(D) add_files_recursive, pass 1: explicitly ignoring ${MAIL_DIR}/.git
>  (D) add_files_recursive, pass 1: explicitly ignoring ${MAIL_DIR}/one/two/three/.git
> -(D) add_files_recursive, pass 2: explicitly ignoring ${MAIL_DIR}/one/two/three/.git
> -(D) add_files_recursive, pass 2: explicitly ignoring ${MAIL_DIR}/one/two/three/ignored_file
> -(D) add_files_recursive, pass 2: explicitly ignoring ${MAIL_DIR}/one/two/ignored_file
> -(D) add_files_recursive, pass 2: explicitly ignoring ${MAIL_DIR}/one/ignored_file
>  (D) add_files_recursive, pass 2: explicitly ignoring ${MAIL_DIR}/.git
>  (D) add_files_recursive, pass 2: explicitly ignoring ${MAIL_DIR}/.ignored_hidden_file
>  (D) add_files_recursive, pass 2: explicitly ignoring ${MAIL_DIR}/ignored_file
> +(D) add_files_recursive, pass 2: explicitly ignoring ${MAIL_DIR}/one/ignored_file
> +(D) add_files_recursive, pass 2: explicitly ignoring ${MAIL_DIR}/one/two/ignored_file
> +(D) add_files_recursive, pass 2: explicitly ignoring ${MAIL_DIR}/one/two/three/.git
> +(D) add_files_recursive, pass 2: explicitly ignoring ${MAIL_DIR}/one/two/three/ignored_file
>  No new mail."
>  
>  

Re: [PATCH] test: new: Fix intermittent test failures with --debug

[David Bremner]

Ethan Glasser-Camp <ethan.glasser.camp@gmail.com> writes:
>
> Fix the failures by sorting the output of notmuch --debug and
> comparing this to a hand-sorted version of its output.

Pushed. Good to see the debug flag being put to use.

d

P.S. somebody also requested a quiet flag for notmuch new, if someone is
looking for an easy project.