Inconsistent query results

Inconsistent query results

[Kirill A. Shutemov]

[-- Attachment #1: Type: text/plain, Size: 356 bytes --]

Hello,

I found that on particular queries notmuch return different results if run
the query few times. Re-initialing the query or db doesn't help.

I've attached test case along with corpus of messages.

Unpack the archive and run `make' there. It will initialize the notmuch
database for the corpus, build and run the test-case.

-- 
 Kirill A. Shutemov

[-- Attachment #2: test.tar.xz --]
[-- Type: application/x-xz, Size: 170804 bytes --]

Re: Inconsistent query results

[David Bremner]

[-- Attachment #1: Type: text/plain, Size: 2873 bytes --]

"Kirill A. Shutemov" <kirill@shutemov.name> writes:

> Hello,
>
> I found that on particular queries notmuch return different results if run
> the query few times. Re-initialing the query or db doesn't help.
>
> I've attached test case along with corpus of messages.
>
> Unpack the archive and run `make' there. It will initialize the notmuch
> database for the corpus, build and run the test-case.

Thanks for the report. I don't yet understand where the bug is, but I
think it's safe to say it's not in your code. I made a somewhat simpler
test case that displays the same problem (at the end).

I'm also fairly sure this is different than the exclude related bug I
recently fixed in notmuch, since running your test under
"NOTMUCH_DEBUG_QUERY=yes ./test" shows the same xapian query is used
both times.

One thing I noticed is that if run both your test case and mine under
valgrind, I get a report of some uninitialized memory. The reports are
similar in both cases, here is part of the report from my test case

==11180== Conditional jump or move depends on uninitialised value(s)
==11180==    at 0x5F5D0B1: OrPostList::check(unsigned int, double, bool&) (orpostlist.cc:198)
==11180==    by 0x5F4E171: check_helper (multiandpostlist.h:97)
==11180==    by 0x5F4E171: MultiAndPostList::find_next_match(double) (multiandpostlist.cc:217)
==11180==    by 0x5F44C3F: skip_to_handling_prune (branchpostlist.h:98)
==11180==    by 0x5F44C3F: AndNotPostList::advance_to_next_match(double, Xapian::PostingIterator::Internal*) (andnotpostlist.cc:50)
==11180==    by 0x5F4E317: next_helper (multiandpostlist.h:76)
==11180==    by 0x5F4E317: MultiAndPostList::next(double) (multiandpostlist.cc:238)
==11180==    by 0x5F4FACC: next_handling_prune (branchpostlist.h:85)
==11180==    by 0x5F4FACC: MultiMatch::get_mset(unsigned int, unsigned int, unsigned int, Xapian::MSet&, Xapian::Weight::Internal&, Xapian::MatchDecider const*, Xapian::KeyMaker const*) (multimatch.cc:570)
==11180==    by 0x5E485CE: Xapian::Enquire::Internal::get_mset(unsigned int, unsigned int, unsigned int, Xapian::RSet const*, Xapian::MatchDecider const*) const (omenquire.cc:581)
==11180==    by 0x5E48913: Xapian::Enquire::get_mset(unsigned int, unsigned int, unsigned int, Xapian::RSet const*, Xapian::MatchDecider const*) const (omenquire.cc:939)
==11180==    by 0x4E52FB8: _notmuch_query_count_documents (query.cc:679)
==11180==    by 0x108A99: doit (test.c:17)
==11180==    by 0x108B0C: main (test.c:28)
==11180== 
count1: 2, count2: 2

Notice that under valgrind the counts match, which strongly suggests
that whatever is going on here is related to a memory error.

In case someone on the xapian list wants to play with this, you can grab
Kirill's test corpus and driver from

wget http://notmuchmail.org/pipermail/notmuch/attachments/20170308/fa83965a/attachment-0001.xz
tar Jxvf attachment-0001.xz


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: test.c --]
[-- Type: text/x-csrc, Size: 1220 bytes --]

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#include <notmuch.h>

const char *query = "tag:unread to:linux-mm@kvack.org -(tag:unread (shutemov or tag:followed))";
//const char *query = "tag:unread to:linux-mm@kvack.org -(shutemov or tag:followed)";

int main(int argc, char **argv)
{
    notmuch_database_t *db;
    notmuch_query_t *q;
    notmuch_threads_t *threads;
    int count1 = 0, count2 = 0;

    notmuch_database_open(DB_PATH, 0, &db);
    q = notmuch_query_create(db, query);

    for (notmuch_query_search_threads_st(q, &threads);
	    notmuch_threads_valid(threads);
	    notmuch_threads_move_to_next(threads)) {
	count1++;
    }

    notmuch_threads_destroy(threads);
    notmuch_query_destroy(q);
    notmuch_database_close(db);

    notmuch_database_open(DB_PATH, 0, &db);
    q = notmuch_query_create(db, query);

    for (notmuch_query_search_threads_st(q, &threads);
	    notmuch_threads_valid(threads);
	    notmuch_threads_move_to_next(threads)) {
	count2++;
    }

    notmuch_threads_destroy(threads);
    notmuch_query_destroy(q);
    notmuch_database_close(db);

    printf("count1: %d, count2: %d\n", count1, count2);
    if (count1 != count2)
	printf("WTF?\n");
    return 0;
}

Re: Inconsistent query results

[David Bremner]

[-- Attachment #1: Type: text/plain, Size: 379 bytes --]

David Bremner <david@tethera.net> writes:

>
> Thanks for the report. I don't yet understand where the bug is, but I
> think it's safe to say it's not in your code. I made a somewhat simpler
> test case that displays the same problem (at the end).
>

Oops, I cleverly wiped out my modified test case test-unpacking the
attachment. Luckily I still had it in a buffer. Here it is


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: test2.c --]
[-- Type: text/x-csrc, Size: 874 bytes --]

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <assert.h>
#include <notmuch.h>

const char *query = "tag:unread to:linux-mm@kvack.org -(tag:unread (shutemov or tag:followed))";
//const char *query = "tag:unread to:linux-mm@kvack.org -(shutemov or tag:followed)";

static void
doit(unsigned int *out) {
  notmuch_database_t *db;
  notmuch_query_t *q;

  assert (NOTMUCH_STATUS_SUCCESS == notmuch_database_open(DB_PATH, 0, &db));
  assert (q = notmuch_query_create(db, query));
  assert (NOTMUCH_STATUS_SUCCESS == notmuch_query_count_messages_st (q, out));
  notmuch_query_destroy(q);
  notmuch_database_close(db);
  notmuch_database_destroy(db);
}

 
int main(int argc, char **argv)
{
  unsigned int count1 = 0, count2 = 0;

    doit(&count1);
    doit(&count2);
    printf("count1: %d, count2: %d\n", count1, count2); 

    return (count1 != count2);
}

Re: Inconsistent query results

[Olly Betts]

On Wed, Mar 08, 2017 at 10:32:56PM -0400, David Bremner wrote:
> "Kirill A. Shutemov" <kirill@shutemov.name> writes:
> > I found that on particular queries notmuch return different results if run
> > the query few times. Re-initialing the query or db doesn't help.
> 
> Thanks for the report. I don't yet understand where the bug is, but I
> think it's safe to say it's not in your code. I made a somewhat simpler
> test case that displays the same problem (at the end).

It's a bug in Xapian - I've committed a fix to master (commit
fa12a83957e97349aa6e2a6c0896faf210dfe4b4) which I'll backport for 1.4.4 and
1.2.25 (it also affects 1.2.x).

To trigger it you need an AND operator which is unweighted (e.g. being on the
right side of AND_NOT here) with a subquery which uses the passed max weight
value (the obvious case is another operator, OR in this case).

(David already confirmed on IRC that the fix applied solves this for him.)

Cheers,
    Olly

Re: Inconsistent query results

[Kirill A. Shutemov]

On Fri, Mar 10, 2017 at 02:56:03AM +0000, Olly Betts wrote:
> On Wed, Mar 08, 2017 at 10:32:56PM -0400, David Bremner wrote:
> > "Kirill A. Shutemov" <kirill@shutemov.name> writes:
> > > I found that on particular queries notmuch return different results if run
> > > the query few times. Re-initialing the query or db doesn't help.
> > 
> > Thanks for the report. I don't yet understand where the bug is, but I
> > think it's safe to say it's not in your code. I made a somewhat simpler
> > test case that displays the same problem (at the end).
> 
> It's a bug in Xapian - I've committed a fix to master (commit
> fa12a83957e97349aa6e2a6c0896faf210dfe4b4) which I'll backport for 1.4.4 and
> 1.2.25 (it also affects 1.2.x).
> 
> To trigger it you need an AND operator which is unweighted (e.g. being on the
> right side of AND_NOT here) with a subquery which uses the passed max weight
> value (the obvious case is another operator, OR in this case).
> 
> (David already confirmed on IRC that the fix applied solves this for him.)

Verified. Thanks!

-- 
 Kirill A. Shutemov

Inconsistent query results

[Nicolas Bock]

Hi,

When I run the following query I get no results:

    $ notmuch search --output=files folder:"mail_account/Deleted
Items" and not tag:deleted

When I run that same query in emacs/notmuch I find one file:

    query: folder:"mail_account/Deleted Items" and not tag:deleted
    file: /home/nbock/Mail/mail_account/Deleted
Items/cur/5d5aca30-2ef6-11ea-9e47-28f10e4fd9ec,U=984:2,S

Am I missing something?

Best,

Nick

Re: Inconsistent query results

[Jani Nikula]

On Sun, 05 Jan 2020, Nicolas Bock <nicolasbock@gmail.com> wrote:
> When I run the following query I get no results:
>
>     $ notmuch search --output=files folder:"mail_account/Deleted
> Items" and not tag:deleted
>
> When I run that same query in emacs/notmuch I find one file:
>
>     query: folder:"mail_account/Deleted Items" and not tag:deleted
>     file: /home/nbock/Mail/mail_account/Deleted
> Items/cur/5d5aca30-2ef6-11ea-9e47-28f10e4fd9ec,U=984:2,S
>
> Am I missing something?

Yes, quotes. ;)

Emacs does the quoting for you, but in shell you need to ensure the
double quotes go all the way to Xapian. Try wrapping the folder search
in single quotes: 'folder:"mail_account/Deleted Items"'.

You can use the NOTMUCH_DEBUG_QUERY=1 environment variable to debug
queries. You can see the difference.

BR,
Jani.

Re: Inconsistent query results

[Nicolas Bock]

On Sun, Jan 5, 2020 at 8:41 AM Jani Nikula <jani@nikula.org> wrote:
>
> On Sun, 05 Jan 2020, Nicolas Bock <nicolasbock@gmail.com> wrote:
> > When I run the following query I get no results:
> >
> >     $ notmuch search --output=files folder:"mail_account/Deleted
> > Items" and not tag:deleted
> >
> > When I run that same query in emacs/notmuch I find one file:
> >
> >     query: folder:"mail_account/Deleted Items" and not tag:deleted
> >     file: /home/nbock/Mail/mail_account/Deleted
> > Items/cur/5d5aca30-2ef6-11ea-9e47-28f10e4fd9ec,U=984:2,S
> >
> > Am I missing something?
>
> Yes, quotes. ;)

Ah, thanks :)

> Emacs does the quoting for you, but in shell you need to ensure the
> double quotes go all the way to Xapian. Try wrapping the folder search
> in single quotes: 'folder:"mail_account/Deleted Items"'.

That seems to work better, however, now I get

    $ notmuch search --output=files 'folder:"mail_account/Deleted
Items"' and not tag:deleted
    /home/nbock/Mail/mail_account/Deleted
Items/cur/5d5aca30-2ef6-11ea-9e47-28f10e4fd9ec,U=984:2,S
    /home/nbock/Mail/mail_account/Archives/2020/cur/5d5b7b2e-2ef6-11ea-9e47-28f10e4fd9ec,U=620:2,S

How come there are two files with identical names in two separate
folders? What am I missing now? :)

> You can use the NOTMUCH_DEBUG_QUERY=1 environment variable to debug
> queries. You can see the difference.
>
> BR,
> Jani.