Re: nbook: a notmuch based address book written in python

[Patrick Totzke <patricktotzke@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 5337 bytes --]

Quoting Suvayu Ali (2012-10-08 10:34:29)
> Hi Patrick,
> 
> Sorry for the very late reply; I got distracted with some personal
> matters.
> 
> On Tue, Sep 25, 2012 at 11:44:57AM +0100, Patrick Totzke wrote:
> > Hey Suvayu, welcome to notmuch!
> > 
> > I hope you are aware that there are already a few search based abook tools
> > around for notmuch (listed in the wiki, albeit hidden in the emacs docs):
> > http://notmuchmail.org/emacstips/#index14h2
> > I personally use nottoomuch-addresses.sh, which apparently does some advanced
> > caching voodoo for speed.
> > 
> 
> I wasn't aware of either of them, thanks for pointing them out.  I'll
> take a look for inspiration and ideas.
> 
> > But to your tool; practice test:
> > I wasn't able to use wildcards or simply prefixes of names. This is essential
> > if you want to use it for tabcompleting contacts in a MUA.
> 
> Since the idea was inspired by the completion on the Gmail web
> interface, I already do a partial search so wildcards should not be
> necessary.

Not sure what you mean here: If I compose a mail using gmails web interface
and type a prefix of someone's name I will get this contect as a suggestion.
My point was that using your tool, I did not get a contact suggested
for all prefixes.

> > The time lookups take seems to depend on how many matches there are:
> > 
> > -------------------------------
> > time nbook Suvayu
> > 1 unique email addresses found for `Suvayu'
> > fatkasuvayu+linux@gmail.com     Suvayu Ali
> > 
> > nbook Suvayu  0.04s user 0.01s system 95% cpu 0.050 total
> > -------------------------------
> > time nbook Justus
> > ...
> > 
> > nbook Justus  0.21s user 0.07s system 11% cpu 2.484 total
> > -------------------------------
> 
> Yes, I noticed this too when I searched for the more common names.  Not
> sure how to get around this though.

I think this is a conceptual problem with your algorithm:
You look up *all* messages and add a name to your result-list
if it matches. This means you go through some condidate
as often as you index contains mails from/to him.
What one really wants is to ask the database to do something like
  "SELECT name,email from RECIPIENTS_OR_SENDER"
where RECIPIENTS_OR_SENDER is some imaginary list that stores
a set of contacts.

Bottom line: One would have to change the layout of the underlying
database (not likely) or do regularly update some cache
and only work on that. This is what some of the mentioned tools do if i'm not mistaken.

> > And If I look for my own name, this takes over a minute,
> > eventually dying. This could be an issue with libnotmuch though.
> > Possibly, your algorithm takes very long and then reads from an initially
> > opened Database object again, which was invalidated by concurrent writes of other processes..
> > 
> > -------------------------------
> > [~] time nbook Patrick                     
> > 
> > Error opening /home/pazz/mail/gmail/[Google Mail].All Mail/cur/1330682270_0.12958.megatron,U=8766,FMD5=66ff6a8bc18a8a3ac4b311daa93d358a:2,S: Too many open files
> > Traceback (most recent call last):
> >   File "/home/pazz/bin/nbook", line 167, in <module>
> >   File "/home/pazz/bin/nbook", line 71, in __init__
> >   File "/home/pazz/.local/lib/python2.7/site-packages/notmuch/message.py", line 233, in get_header
> > notmuch.errors.NullPointerError
> > Error in sys.excepthook:
> > Traceback (most recent call last):
> >   File "/usr/lib/python2.7/dist-packages/apport_python_hook.py", line 66, in apport_excepthook
> > ImportError: No module named fileutils
> > 
> > Original exception was:
> > Traceback (most recent call last):
> >   File "/home/pazz/bin/nbook", line 167, in <module>
> >   File "/home/pazz/bin/nbook", line 71, in __init__
> >   File "/home/pazz/.local/lib/python2.7/site-packages/notmuch/message.py", line 233, in get_header
> > notmuch.errors.NullPointerError
> > nbook Patrick  3.20s user 5.47s system 12% cpu 1:11.65 total
> > ------------------------------------
> > 
> 
> Yes someone else pointed this out too.  Again I'm not sure how to
> proceed here.  I had a quick look at this last week and it seemed to me
> the limitation comes from within the python bindings for notmuch.  Do
> you have any ideas?

As mentioned before, I think you invalidate the Database object concurrently
while your long-running algorithm goes through all messages.
Xapian doesn't handle concurrent access to the index like a normal™ database would.
This means you are notified by this error that some changes were detected.
Maybe the error message should be more telling here though. Teythoon?

> > Anyway, have fun hacking notmuch! If you are looking for a related project to bring in your python skills
> > I could think of one or two :D
> 
> That would be wonderful.  To give you my background, I'm a graduate
> student in physics and I have to do a lot of C/C++ and python
> programming for my research.  Contributing to FOSS projects seems like a
> wonderful way to learn to collaborate and clean programming (we
> physicists tend to be sloppy programmers :-p).

https://github.com/teythoon/afew
https://github.com/pazz/alot
http://excess.org/urwid/

I'm sure patches will be welcome to any of the above :)
Best,
/p

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iEYEABECAAYFAlB5ncsACgkQlDQDZ9fWxaofuwCbBIrFTCAEoimDW+oZLkLIOp5+
hFsAnjPfXjLw2idZX33ykZMrhQ5KXSp/
=1H/w
-----END PGP SIGNATURE-----