Failing test cases

Failing test cases

[Jason White]

I just tried to build the Debian package from the latest Git master branch,
but this could not be completed due to failures of test cases 067, 068 and
069.

Are others seeing this, or is it a peculiarity of my system (Debian Sid,
x86-64)?

Re: Failing test cases

[Dirk Hohndel]

On Sun, 25 Apr 2010 10:17:44 +1000, Jason White <jason@jasonjgw.net> wrote:
> I just tried to build the Debian package from the latest Git master branch,
> but this could not be completed due to failures of test cases 067, 068 and
> 069.
> 
> Are others seeing this, or is it a peculiarity of my system (Debian Sid,
> x86-64)?

No, this is the expected behavior right now. Carl has the habit of
committing the tests before he commits the code that addresses the issue
/ feature that is tested for. And it turned out that there was a bug in
the original version of that code. I have submitted a fixed version but
that hasn't been pushed, yet.

0.3 with all the latest and greatest fixes and cleanups should be out,
shortly. 

/D

-- 
Dirk Hohndel
Intel Open Source Technology Center

Re: Failing test cases

[Jason White]

In case this is not expected behaviour for the latest code in the Git master
branch, which is current up to 0.3.1:


Testing author reordering;
 Adding parent message...				FAIL
--- test-066.expected	2010-04-27 16:15:48.273607558 -0800
+++ test-066.output	2010-04-27 16:15:48.273607558 -0800
@@ -1 +1 @@
-Added 1 new message to the database.
+Added 1 new message to the database. Removed 1 message.
 Adding initial child message...			FAIL
--- test-067.expected	2010-04-27 16:15:48.441247008 -0800
+++ test-067.output	2010-04-27 16:15:48.441247008 -0800
@@ -1 +1 @@
-Added 1 new message to the database.
+Added 2 new messages to the database.
 Adding second child message...				PASS
 Searching when all three messages match...		PASS
 
 The system is Debian Sid, x86-64 architecture, Xapian 1.0.19-1, with the git
 commit
 commit 417274d698b6718621b9f5dec744ab169499f4e3 which merged in the 0.3.1
 branch.

 If this is all expected, that's fine, but if not, let me know how I can help
 further.

Thank you for a very promising project which is attracting healthy
contributions and clearly developing an active community.

Re: Failing test cases

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 1150 bytes --]

On Wed, 28 Apr 2010 10:26:15 +1000, Jason White <jason@jasonjgw.net> wrote:
> In case this is not expected behaviour for the latest code in the Git master
> branch, which is current up to 0.3.1:

Thanks for the report.

> +++ test-066.output	2010-04-27 16:15:48.273607558 -0800
> @@ -1 +1 @@
> -Added 1 new message to the database.
> +Added 1 new message to the database. Removed 1 message.
>  Adding initial child message...			FAIL
> --- test-067.expected	2010-04-27 16:15:48.441247008 -0800
> +++ test-067.output	2010-04-27 16:15:48.441247008 -0800
> @@ -1 +1 @@
> -Added 1 new message to the database.
> +Added 2 new messages to the database.
>  Adding second child message...				PASS
...
>  If this is all expected, that's fine, but if not, let me know how I can help
>  further.

This is not expected.

The output above is quite strange. It would appear that during test 66
notmuch thought that a message file had bee removed from the
mailstore. But from the test 67 output it looks like the file was not
actually removed, since notmuch then added it again.

Is this behavior repeatable? I've certainly never seen anything like
this yet.

-Carl

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Failing test cases

[Jason White]

Carl Worth <cworth@cworth.org> wrote:
 
> Is this behavior repeatable? I've certainly never seen anything like
> this yet.

It seems to be repeatable here, even after running git clean -d -f -x,
followed by make && make test

Is anyone else seeing this?

Re: Failing test cases

[Olly Betts]

On 2010-04-28, Jason White wrote:
> It seems to be repeatable here, even after running git clean -d -f -x,
> followed by make && make test
>
> Is anyone else seeing this?

Yes, I got the same failure trying to rebuild the notmuch 0.3.1 Debian package
with Xapian 1.2.0 (building in a clean Debian sid chroot on x86-64).

I'm hoping to get 1.2.x in for Debian's next stable release, so it would be
good to work out what's going on.

Cheers,
    Olly

Re: Failing test cases

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 958 bytes --]

On Fri, 30 Apr 2010 01:23:16 +0000 (UTC), Olly Betts <olly@survex.com> wrote:
> On 2010-04-28, Jason White wrote:
> > It seems to be repeatable here, even after running git clean -d -f -x,
> > followed by make && make test
> >
> > Is anyone else seeing this?
> 
> Yes, I got the same failure trying to rebuild the notmuch 0.3.1 Debian package
> with Xapian 1.2.0 (building in a clean Debian sid chroot on x86-64).
> 
> I'm hoping to get 1.2.x in for Debian's next stable release, so it would be
> good to work out what's going on.

Here's a little update.

I wasn't getting this failure because I was building against Xapian
1.0.18, (while using a clean Debian sid chroot now installs Xapian
1.0.20).

I installed Xapian 1.0.20 (from the libxapian-dev package in sid) and am
now getting this failure.

I'm investigating more closely now, but I thought I'd at least share
that much information.

-Carl

-- 
carl.d.worth@intel.com

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Failing test cases

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 2400 bytes --]

On Tue, 04 May 2010 11:21:27 -0700, Carl Worth <cworth@cworth.org> wrote:
> I installed Xapian 1.0.20 (from the libxapian-dev package in sid) and am
> now getting this failure.
> 
> I'm investigating more closely now, but I thought I'd at least share
> that much information.

Here are my conclusions based on some additional investigation:

What's happening here is that notmuch is creating a Xapian TermIterator,
iterating over some of the results, and then modifying the database such
that the terms being iterated over are affected. Specifically, in this
test case, notmuch is inserting a new term before the term being pointed
at by the iterator.

With Xapian 1.0.18 incrementing the iterator from this state causes it
to point to the next term. However, with Xapian 1.0.20 this same
increment causes the iterator to point at the same term again.

I was able to put a little workaround into notmuch-new.c that avoids the
bug (see below). But this isn't entirely satisfactory for several
reasons:

1. We need to know what the intended semantics of the Xapian iterators
   really are. If this behavior is just a bug in Xapian, then that
   should be fixed.

   But if the iterators are to have explicitly undefined behavior in
   cases like this, then notmuch is going to need to carefully copy
   contents out of Xapian iterators when creating notmuch iterators (at
   some performance cost).

2. Working around the bug in notmuch-new.c doesn't fix the notmuch
   library itself, (so that python bindings, etc. will remain
   broken). So we really do need a lower-level fix for this.

To properly do the lower-level fix for (2), we'll need to know the
answer to (1). Olly, do you have an answer on the intended semantics of
Xapian iterators?

-Carl

diff --git a/notmuch-new.c b/notmuch-new.c
index 8818728..928bb94 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -380,7 +380,11 @@ add_files_recursive (notmuch_database_t *notmuch,
 	if (notmuch_filenames_valid (db_files) &&
 	    strcmp (notmuch_filenames_get (db_files), entry->d_name) == 0)
 	{
-	    notmuch_filenames_move_to_next (db_files);
+	    while (notmuch_filenames_valid (db_files) &&
+		   strcmp (notmuch_filenames_get (db_files), entry->d_name) <= 0)
+	    {
+		notmuch_filenames_move_to_next (db_files);
+	    }
 	    continue;
 	}
 


-- 
carl.d.worth@intel.com

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Failing test cases

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 500 bytes --]

On Tue, 04 May 2010 12:09:56 -0700, Carl Worth <cworth@cworth.org> wrote:
> To properly do the lower-level fix for (2), we'll need to know the
> answer to (1). Olly, do you have an answer on the intended semantics of
> Xapian iterators?

Just to close the loop on this one. Olly identified the behavior as a
bug in Xapian 1.0.20 and released a new 1.0.20-2 package (into Debian at
least).

So with an updated Xapian, people should no longer be seeing this
failure from the notmuch test suite.

-Carl

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]