Info about notmuch database

Info about notmuch database

[piuttosto]

Hello!
I like notmuch a lot, so I'm writing a (conceptually) similar software
about addressbook: it will scan all your emails, storing email addresses
in a xapian database (you can think of it as little brother database[1] on
steroids)
The part that I'd like to re-implement is "notmuch new": it seems that
in the xapian db there is not only informations about each mail, but
also the mtime of each directory. My impression is this being "chaotic",
but probably I am just missing the point.

So, here's the question: how is the db "structured"? is there any
documentation to look at?

[1] http://www.spinnaker.de/lbdb/

Info about notmuch database

[boyska]

Hello!
I like notmuch a lot, so I'm writing a (conceptually) similar software
about addressbook: it will scan all your emails, storing email 
addresses
in a xapian database (you can think of it as little brother database[1] 
on
steroids)
The part that I'd like to re-implement is "notmuch new": it seems that
in the xapian db there is not only informations about each mail, but
also the mtime of each directory. My impression is this being 
"chaotic",
but probably I am just missing the point.

So, here's the question: how is the db "structured"? is there any
documentation to look at?

[1] http://www.spinnaker.de/lbdb/

-- 
boyska
GPG: 0x520CE393

Re: Info about notmuch database

[Thomas Jost]

[-- Attachment #1: Type: text/plain, Size: 1274 bytes --]

On Wed, 04 Jan 2012 15:49:19 +0000, boyska <piuttosto@logorroici.org> wrote:
> Hello!
> I like notmuch a lot, so I'm writing a (conceptually) similar software
> about addressbook: it will scan all your emails, storing email 
> addresses
> in a xapian database (you can think of it as little brother database[1] 
> on
> steroids)
> The part that I'd like to re-implement is "notmuch new": it seems that
> in the xapian db there is not only informations about each mail, but
> also the mtime of each directory. My impression is this being 
> "chaotic",
> but probably I am just missing the point.
> 
> So, here's the question: how is the db "structured"? is there any
> documentation to look at?
> 
> [1] http://www.spinnaker.de/lbdb/
> 
> -- 
> boyska
> GPG: 0x520CE393

There's a description of the DB "schema" in lib/database.cc in the
notmuch source code. But you may also consider just using libnotmuch
instead, if that's enough for what you want to do.

Also: why Xapian? I'm already using something similar I wrote with
Python, storing everything in a dictionary, using Pickle to save that to
disk: 162 lines of code and 45 kb of data are enough to store my
addressbook and have completion in Emacs...

Regards,

-- 
Thomas/Schnouki

[-- Attachment #2: Type: application/pgp-signature, Size: 489 bytes --]

Re: Info about notmuch database

[boyska]

On Thu, Jan 05, 2012 at 04:04:22PM +0100, Thomas Jost wrote:
> On Wed, 04 Jan 2012 15:49:19 +0000, boyska <piuttosto@logorroici.org> wrote:
> > Hello!
> > I like notmuch a lot, so I'm writing a (conceptually) similar software
> > about addressbook: it will scan all your emails, storing email 
> > addresses
> > in a xapian database (you can think of it as little brother database[1] 
> > on
> > steroids)
> > The part that I'd like to re-implement is "notmuch new": it seems that
> > in the xapian db there is not only informations about each mail, but
> > also the mtime of each directory. My impression is this being 
> > "chaotic",
> > but probably I am just missing the point.
> > 
> > So, here's the question: how is the db "structured"? is there any
> > documentation to look at?
> > 
> > [1] http://www.spinnaker.de/lbdb/
> > 
> > -- 
> > boyska
> > GPG: 0x520CE393
> 
> There's a description of the DB "schema" in lib/database.cc in the
> notmuch source code. But you may also consider just using libnotmuch
> instead, if that's enough for what you want to do.

thanks, found it, much clearer now.
But I really can't understand why not just putting these things on a
separate file :) atomic consistency issues?

> Also: why Xapian? I'm already using something similar I wrote with
> Python, storing everything in a dictionary, using Pickle to save that to
> disk: 162 lines of code and 45 kb of data are enough to store my
> addressbook and have completion in Emacs...

dictionary approach is fine to manage a "manual" addressbook, where you
store addresses. But what I want is an _automatic_ addressbook, like the
lbdb one, which just indexes all seen emails.
The grep approach is better from this point of view, but still not
advanced enough for me.
For example, I'd like to store "cooccorrences": if some email is used in
the same mail of some other, then it must contain a relationship; for
example, your email should be correlated to the notmuch mailinglist,
because you wrote to it. (they should be 0-weighted xapian term).  Also,
I want to give more importance to email addresses which are frequently
seen, and much less to not-so-frequently seen. Xapian makes these really
easy, so the question is "why not using it?" ;)

Re: Info about notmuch database

[Thomas Jost]

[-- Attachment #1: Type: text/plain, Size: 2198 bytes --]

On Thu, 5 Jan 2012 16:38:07 +0100, boyska <piuttosto@logorroici.org> wrote:
> > There's a description of the DB "schema" in lib/database.cc in the
> > notmuch source code. But you may also consider just using libnotmuch
> > instead, if that's enough for what you want to do.
> 
> thanks, found it, much clearer now.
> But I really can't understand why not just putting these things on a
> separate file :) atomic consistency issues?

I doubt it's for consistency (see commit 824dad76), more likely it's
because people should use libnotmuch rather than directly hacking into
the DB ;)


> > Also: why Xapian? I'm already using something similar I wrote with
> > Python, storing everything in a dictionary, using Pickle to save that to
> > disk: 162 lines of code and 45 kb of data are enough to store my
> > addressbook and have completion in Emacs...
> 
> dictionary approach is fine to manage a "manual" addressbook, where you
> store addresses. But what I want is an _automatic_ addressbook, like the
> lbdb one, which just indexes all seen emails.

That's what my little script does too: index emails and how many times
they appear in the DB so that completion shows more frequently used ones
first. The indexing is done after running "notmuch new", when running my
auto-tagging script. I'm too lazy to maintain a "manual" addressbook
correctly :)

> The grep approach is better from this point of view, but still not
> advanced enough for me.
> For example, I'd like to store "cooccorrences": if some email is used in
> the same mail of some other, then it must contain a relationship; for
> example, your email should be correlated to the notmuch mailinglist,
> because you wrote to it. (they should be 0-weighted xapian term).  Also,
> I want to give more importance to email addresses which are frequently
> seen, and much less to not-so-frequently seen. Xapian makes these really
> easy, so the question is "why not using it?" ;)

Nice ideas, and Xapian is probably a good choice for doing that kind of
stuff :)

Do you plan to use this addressbook with notmuch-address.el, or will it
be a standalone program?

Regards,

-- 
Thomas/Schnouki

[-- Attachment #2: Type: application/pgp-signature, Size: 489 bytes --]

Re: Info about notmuch database

[boyska]

On Thu, Jan 05, 2012 at 05:35:55PM +0100, Thomas Jost wrote:
> On Thu, 5 Jan 2012 16:38:07 +0100, boyska <piuttosto@logorroici.org> wrote:
> > > There's a description of the DB "schema" in lib/database.cc in the
> > > notmuch source code. But you may also consider just using libnotmuch
> > > instead, if that's enough for what you want to do.
> > 
> > thanks, found it, much clearer now.
> > But I really can't understand why not just putting these things on a
> > separate file :) atomic consistency issues?
> 
> I doubt it's for consistency (see commit 824dad76), more likely it's
> because people should use libnotmuch rather than directly hacking into
> the DB ;)

Fine; I'll probably keep the whole output of "find" as the data of a
SINGLE entry, instead of one entry for directory. This just seems easier
to me.

> Do you plan to use this addressbook with notmuch-address.el, or will it
> be a standalone program?

It will be a standalone program, meant to be used with mutt-query [1].
So just call "notmany thomas" on commandline, and your email will
appear.
I don't use emacs, so I won't write an emacs tool (nor I know how
notmuch-address.el works), but I am trying to keep library and UI
separate, so writing a wrapper suitable for emacs is possible, and
probably very easy.

[1] http://wiki.mutt.org/?QueryCommand

Re: Info about notmuch database

[Sebastian Spaeth]

[-- Attachment #1: Type: text/plain, Size: 606 bytes --]

On Thu, 05 Jan 2012 16:04:22 +0100, Thomas Jost <schnouki@schnouki.net> wrote:
> There's a description of the DB "schema" in lib/database.cc in the
> notmuch source code. But you may also consider just using libnotmuch
> instead, if that's enough for what you want to do.
> 
> Also: why Xapian? I'm already using something similar I wrote with
> Python, storing everything in a dictionary, using Pickle to save that to
> disk: 162 lines of code and 45 kb of data are enough to store my
> addressbook and have completion in Emacs...

Ohh, that sounds nice. Is that public somewhere?

Sebastian

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: Info about notmuch database

[Thomas Jost]

[-- Attachment #1: Type: text/plain, Size: 869 bytes --]

On Sun, 08 Jan 2012 13:59:30 +0100, Sebastian Spaeth <Sebastian@SSpaeth.de> wrote:
> On Thu, 05 Jan 2012 16:04:22 +0100, Thomas Jost <schnouki@schnouki.net> wrote:
> > There's a description of the DB "schema" in lib/database.cc in the
> > notmuch source code. But you may also consider just using libnotmuch
> > instead, if that's enough for what you want to do.
> > 
> > Also: why Xapian? I'm already using something similar I wrote with
> > Python, storing everything in a dictionary, using Pickle to save that to
> > disk: 162 lines of code and 45 kb of data are enough to store my
> > addressbook and have completion in Emacs...
> 
> Ohh, that sounds nice. Is that public somewhere?
> 
> Sebastian

https://github.com/Schnouki/dotfiles/blob/master/notmuch/addrbook.py

Maybe I should add more comments in it :)

Regards,

-- 
Thomas/Schnouki

[-- Attachment #2: Type: application/pgp-signature, Size: 489 bytes --]

Re: Info about notmuch database

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 1054 bytes --]

On Mon, 2 Jan 2012 17:04:17 +0100, piuttosto@logorroici.org wrote:
> I like notmuch a lot, so I'm writing a (conceptually) similar software
> about addressbook: it will scan all your emails, storing email addresses
> in a xapian database (you can think of it as little brother database[1] on
> steroids)
> The part that I'd like to re-implement is "notmuch new": it seems that
> in the xapian db there is not only informations about each mail, but
> also the mtime of each directory. My impression is this being "chaotic",
> but probably I am just missing the point.

Hi.  I think this is a neat idea.  However, it has be wondering: would
it be possible to just extend the notmuch database itself to support
this?  Could notmuch just index all the email addresses in such a way
that they would be easily retrievable?  Maybe through an "addresses" sub
command or something?  Obviously having notmuch handle this
functionality natively would get rid of the need to reinvent a lot of
the stuff you're trying to reinvent now (such as "notmuch new").

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: Info about notmuch database

[Bryant, Daniel B.]

I currently use a little Vala utility that Sebastian Spaeth wrote in order to use what's already stored in the notmuch database as a simple addressbook. It's fast and works extremely well for me.

See the section "Address lookup while composing" at http://notmuchmail.org/emacstips/ for links to that and a couple other similar implementations.

Maybe one of these suits your needs or could be used as a starting point for the functionality you're trying to implement?

Dan

----- Original Message -----
From: notmuch-bounces@notmuchmail.org <notmuch-bounces@notmuchmail.org>
To: piuttosto@logorroici.org <piuttosto@logorroici.org>; notmuch@notmuchmail.org <notmuch@notmuchmail.org>
Sent: Wed Jan 11 23:26:42 2012
Subject: Re: Info about notmuch database

On Mon, 2 Jan 2012 17:04:17 +0100, piuttosto@logorroici.org wrote:
> I like notmuch a lot, so I'm writing a (conceptually) similar software
> about addressbook: it will scan all your emails, storing email addresses
> in a xapian database (you can think of it as little brother database[1] on
> steroids)
> The part that I'd like to re-implement is "notmuch new": it seems that
> in the xapian db there is not only informations about each mail, but
> also the mtime of each directory. My impression is this being "chaotic",
> but probably I am just missing the point.

Hi.  I think this is a neat idea.  However, it has be wondering: would
it be possible to just extend the notmuch database itself to support
this?  Could notmuch just index all the email addresses in such a way
that they would be easily retrievable?  Maybe through an "addresses" sub
command or something?  Obviously having notmuch handle this
functionality natively would get rid of the need to reinvent a lot of
the stuff you're trying to reinvent now (such as "notmuch new").

jamie.