problem with folder: search in python bindings

problem with folder: search in python bindings

[Jameson Rollins]

[-- Attachment #1.1: Type: text/plain, Size: 1365 bytes --]

Hey, folks.  I'm seeing something peculiar with the python bindings.
It seems that I am not able to get any results when doing "folder:"
searches with the python bindings.  I'm using a version of notmuch that
includes the folder index patch.

Attached is a script that demonstrates the problem.  It generates a test
repository, then does a couple of searches over it.  When running the
script, I get the following output:

0$ ./python-folder-search-test
Found 2 total files (that's not much mail).
Note: Ignoring non-mail file: /home/jrollins/tmp/tmp.1dMO5vnweJ/config
Processed 2 total files in almost no time.
Added 1 new message to the database.
== notmuch search '*':
thread:0000000000000001  Yest. 23:26 [1/1] Jameson Rollins; python documentation (inbox signed unread)
== notmuch search 'folder:sent':
thread:0000000000000001  Yest. 23:26 [1/1] Jameson Rollins; python documentation (inbox signed unread)
== python search '*':
Jameson Rollins <jrollins@finestructure.net> (2011-03-14) (inbox signed unread) (-1) replies
== python search 'folder:sent':
0$ 

You can see that cli notmuch returns search results when searching for
"folder:sent".  The python bindings, on the other hand, return nothing.

Anyone have any idea what could be going on here?  I'm not familiar with
how the python bindings work, unfortunately.

jamie.


[-- Attachment #1.2: script to test python folder: search --]
[-- Type: application/octet-stream, Size: 742 bytes --]

#!/bin/bash

dir=$(mktemp -d)
trap "rm -rf $dir" EXIT

mkdir "$dir"/sent

notmuch search --output=files to:notmuch from:jrollins \
    | head -1 \
    | xargs -I'{}' cp '{}' "$dir"/sent/

export NOTMUCH_CONFIG="$dir"/config

cat <<EOF >"$NOTMUCH_CONFIG"
[database]
path=$dir
EOF

notmuch new

echo "== notmuch search '*':"
notmuch search '*'

echo "== notmuch search 'folder:sent':"
notmuch search 'folder:sent'

echo "== python search '*':"
python -c "import notmuch
db = notmuch.Database()
q = notmuch.Query(db, '*')
for msg in q.search_messages():
    print msg
"

echo "== python search 'folder:sent':"
python -c "import notmuch
db = notmuch.Database()
q = notmuch.Query(db, 'folder:sent')
for msg in q.search_messages():
    print msg
"

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: problem with folder: search in python bindings

[Sebastian Spaeth]

[-- Attachment #1: Type: text/plain, Size: 884 bytes --]

On Tue, 15 Mar 2011 11:30:30 -0700, Jameson Rollins <jrollins@finestructure.net> wrote:
> Hey, folks.  I'm seeing something peculiar with the python bindings.
> It seems that I am not able to get any results when doing "folder:"
> searches with the python bindings.  I'm using a version of notmuch that
> includes the folder index patch.

Interesting, this is not at all what I am seeing in the current
bindings:

cd ~/src/notmuch/bindings/python;./notmuch.py count 'folder:sent'
1957
/usr/local/bin/notmuch count  'folder:sent'
1957

> Anyone have any idea what could be going on here?  I'm not familiar with
> how the python bindings work, unfortunately.

The folder: stuff came after my latest work on the bindings, so I
haven't really tested things with that. But I don't see why it should
work any other than other queries too.

Some quotation issues at work perhaps?

Sebastian

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: problem with folder: search in python bindings

[Jameson Rollins]

[-- Attachment #1: Type: text/plain, Size: 610 bytes --]

On Wed, 16 Mar 2011 10:25:29 +0100, Sebastian Spaeth <Sebastian@SSpaeth.de> wrote:
> Interesting, this is not at all what I am seeing in the current
> bindings:
> 
> cd ~/src/notmuch/bindings/python;./notmuch.py count 'folder:sent'
> 1957
> /usr/local/bin/notmuch count  'folder:sent'
> 1957

Hey, Sebastian.  Can you share your notmuch.py script?  I want to
compare yours with what I'm doing in my python script.  I don't think
it's a quoting issue, since the quoting I'm using seems to work fine for
other search terms.  It's only with "folder:" searches that I'm seeing
the problem.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: problem with folder: search in python bindings

[Sebastian Spaeth]

[-- Attachment #1: Type: text/plain, Size: 337 bytes --]

On Thu, 17 Mar 2011 09:19:29 -0700, Jameson Rollins <jrollins@finestructure.net> wrote:
> Hey, Sebastian.  Can you share your notmuch.py script?

No, it's my secret plan for earning my pension :).
You already got it:
http://git.notmuchmail.org/git/notmuch/blob/HEAD:/bindings/python/notmuch.py

Let me know what you find out.

Sebastian

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: problem with folder: search in python bindings

[Jameson Rollins]

[-- Attachment #1: Type: text/plain, Size: 1720 bytes --]

On Mon, 21 Mar 2011 16:29:57 +0100, Sebastian Spaeth <Sebastian@SSpaeth.de> wrote:
> On Thu, 17 Mar 2011 09:19:29 -0700, Jameson Rollins <jrollins@finestructure.net> wrote:
> > Hey, Sebastian.  Can you share your notmuch.py script?
> 
> No, it's my secret plan for earning my pension :).
> You already got it:
> http://git.notmuchmail.org/git/notmuch/blob/HEAD:/bindings/python/notmuch.py

Thanks, Sebastian.  Very convenient of you to include it in the repo!

> Let me know what you find out.

So right off the bat I'm seeing a big discrepancy between the results
From notmuch and notmuch.py:

servo:~/src/notmuch/git [personal] 0$ notmuch count from:jrollins
34358
servo:~/src/notmuch/git [personal] 0$ ./bindings/python/notmuch.py count from:jrollins
34358
servo:~/src/notmuch/git [personal] 0$ notmuch count folder:sent
635
servo:~/src/notmuch/git [personal] 0$ ./bindings/python/notmuch.py count folder:sent
7
servo:~/src/notmuch/git [personal] 0$ 

As you can see, the two interfaces are returning the same results for
the "from:jrollins" search, but the binary is returning 635 results for
the "folder:sent" search while the python interface is returning 7.
That's a big discrepancy.  This must be related to the same problem I
was seeing with my test case.

I'm running notmuch.py from my "personal" branch, which is the same
branch that the binary has been compiled from.  My database has also
recently been rebuild from the same branch.

I really have no idea what could be going on here.  It's a stranger
result even than the one I was seeing before, since it is actually
returning *some* results.

Any suggestions about how we might go about debugging this?

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: problem with folder: search in python bindings

[Jesse Rosenthal]

Hmm, I'm getting the same results from the python implementation.

On Mon, 21 Mar 2011 09:32:52 -0700, Jameson Rollins <jrollins@finestructure.net> wrote:
> Any suggestions about how we might go about debugging this?

One thing to try would be to run it step by step in an interactive session. I.e.,

>>> import notmuch as nm
>>> db = nm.Database("/where/ever/it/is")
>>> qry = nm.Query(db, "folder:sent")

Then, two things to look at. What's the output of

>>> qry.count_messages()

and what's the output of 

>>> len(qry.search_messages())

If you're still getting weird results on both, try dumping the search
results into a list

>>> msgs = [m for m in qry.search_messages()]

and take a look at them. Anything special about those messages?

Best,
Jesse

Re: problem with folder: search in python bindings

[Jameson Rollins]

[-- Attachment #1: Type: text/plain, Size: 3098 bytes --]

On Mon, 21 Mar 2011 12:45:06 -0400, Jesse Rosenthal <jrosenthal@jhu.edu> wrote:
> One thing to try would be to run it step by step in an interactive session. I.e.,

Hey, Jesse.  So I'm getting different results interactively!:

>>> import notmuch
>>> db = notmuch.Database("/home/jrollins/.mail")
>>> query = notmuch.Query(db, "folder:sent")
>>> query.count_messages()
9L
>>> len(query.search_messages())
9
>>> 

This same search returns 635 results with the binary, 7 results with the
notmuch.py script, and now 9 in this interactive session.  This is going
to be a pain to debug.

> >>> msgs = [m for m in qry.search_messages()]
> 
> and take a look at them. Anything special about those messages?

So I'm seeing quite a few strange things here.

>>> for m in msgs:
...     print m.get_message_id()
...     print ' ', m.get_filename()
... 
87tyewtobx.fsf@gogo.home
  /home/jrollins/.mail/mailboxes/snapper/new/1300725908.Vfd05I469d8M318600.snapper
871v20bfij.fsf@servo.finestructure.net
  /home/jrollins/.mail/mailboxes/snapper/new/1300725184.Vfd05I469d6M959989.snapper
87y64dk9da.fsf@servo.finestructure.net
  /home/jrollins/.mail/mailboxes/snapper/new/1300378797.Vfd05I469d8M808571.snapper
87aagv2z92.fsf@SSpaeth.de
  /home/jrollins/.mail/mailboxes/snapper/new/1300267540.Vfd05I46a17M964837.snapper
87tyf4xmm1.fsf@servo.finestructure.net
  /home/jrollins/.mail/mailboxes/snapper/new/1300213852.Vfd05I469d8M923454.snapper
1290682750-30283-3-git-send-email-dme@dme.org
  /home/jrollins/.mail/archive/2010/cur/1290698042_0.5764.servo,U=333629,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,
8763426kd5.fsf@steelpick.2x.cz
  /home/jrollins/.mail/archive/2010/cur/1270740662_1.29713.servo,U=264838,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,
87636feg10.fsf@servo.finestructure.net
  /home/jrollins/.mail/archive/2010/cur/1265129567.M505745P5400Q1.servo:2,S
87eil3ehjh.fsf@servo.finestructure.net
  /home/jrollins/.mail/archive/2010/cur/1265127608.M500761P12582Q1.servo:2,S
>>> 

First of all, only messages that were sent by me are in my sent folder.
There are no messages in there that are sent by anyone else.  But this
search is returning messages that are from people other than me.  That
makes no sense.

Second, strangely, all of the messages are ones that are sent to the
notmuch list.  Not sure what to make of that except that the messages
that were sent from me to the list are also in my received mail so are
actually duplicated in my mail store.  I have plenty of other messages
that fall into that same condition, though.

It's going to take a lot more digging before I can identify what is
common about these messages and different from all the messages that are
not returned.  For what it's worth, it's not *just* that they were sent
to the list, since the binary is counting 35 messages that are in the
sent folder and sent to the list:

servo:~ 0$ notmuch count folder:sent and to:notmuch
35
servo:~ 0$ 

I'll keep poking at this, but others can test this and report what they
see I would really appreciate it.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: problem with folder: search in python bindings

[Jesse Rosenthal]

On Mon, 21 Mar 2011 10:34:29 -0700, Jameson Rollins <jrollins@finestructure.net> wrote:
> It's going to take a lot more digging before I can identify what is
> common about these messages and different from all the messages that are
> not returned.  

One guess: do those messages happen to have "folder:sent" in the
body? If so, that might point to the possibility that the python
bindings are using a different libnotmuch. Perhaps they have a different
search path (and maybe there's a debian installed library around that
they're finding?).

Re: problem with folder: search in python bindings

[Jameson Rollins]

[-- Attachment #1: Type: text/plain, Size: 1131 bytes --]

On Mon, 21 Mar 2011 13:42:09 -0400, Jesse Rosenthal <jrosenthal@jhu.edu> wrote:
> On Mon, 21 Mar 2011 10:34:29 -0700, Jameson Rollins <jrollins@finestructure.net> wrote:
> > It's going to take a lot more digging before I can identify what is
> > common about these messages and different from all the messages that are
> > not returned.  
> 
> One guess: do those messages happen to have "folder:sent" in the
> body? If so, that might point to the possibility that the python
> bindings are using a different libnotmuch. Perhaps they have a different
> search path (and maybe there's a debian installed library around that
> they're finding?).

Oh good call, Jesse.  Bonus points for you!  That was the problem.  Damn
it.  In retrospect that's so obvious, though, isn't it!  I had purged
the notmuch binary package, but had accidentally left libnotmuch1 and
libnotmuch-dev installed.  Removing those packages and pointing to my
locally installed libraries fixed the problem.

Sorry for spamming the list with what, in retrospect, I should have been
able to figure out on my own.  Thanks again, Jesse.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: problem with folder: search in python bindings

[Sebastian Spaeth]

[-- Attachment #1: Type: text/plain, Size: 723 bytes --]

On Mon, 21 Mar 2011 11:01:24 -0700, Jameson Rollins <jrollins@finestructure.net> wrote:
> Oh good call, Jesse.  Bonus points for you!  That was the problem.  Damn
> it.  In retrospect that's so obvious, though, isn't it!  I had purged
> the notmuch binary package, but had accidentally left libnotmuch1 and
> libnotmuch-dev installed.  Removing those packages and pointing to my
> locally installed libraries fixed the problem.

Pfeww, I was getting very scared that the python bindings have serious
flaws that I am not aware of. Don't ever do that again to me :-).

Glad you could sort it out. Unfortunately it is not easy to tease out
the path of the libnotmuch being used, otherwise we could debug-output
it.

Sebastian

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: problem with folder: search in python bindings

[Jameson Rollins]

[-- Attachment #1: Type: text/plain, Size: 384 bytes --]

On Tue, 22 Mar 2011 10:35:17 +0100, Sebastian Spaeth <Sebastian@SSpaeth.de> wrote:
> Pfeww, I was getting very scared that the python bindings have serious
> flaws that I am not aware of. Don't ever do that again to me :-).

In the future, we should make sure that all new features are included
since the beginning.  That way we'll avoid all future library
incompatibilities.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]