notmuch for documents

notmuch for documents

[Jameson Rollins]

[-- Attachment #1: Type: text/plain, Size: 4038 bytes --]

A little while ago on #notmuch, madduck and ojwb mentioned that they
thought notmuch was overly focused on mail.  At the time I thought this
was a silly criticism and defended notmuch as doing what it does
*really* well and that we shouldn't expect notmuch to be all things to
all people.

Yesterday, however, I had the profound realization that madduck and ojwb
are right.

Notmuch stores database entries for email messages.  However, these
messages are nothing more than simple rfc5322 [0] structured documents.
They include nothing more than headers and a text body.

Imagine now that I have a collection of ebooks, each stored in a single
rfc5322-formatted text file:

------------------------------------------------------------
From: Italo Calvino <italo@calvino.com>
Subject: If on a winter's night a traveler
Date: 1979

You are about to begin reading Italo Calvino's new novel,
...
------------------------------------------------------------

I store them all in a directory.  I now create a NOTMUCH_CONFIG with a
database.path that points to that directory, and run notmuch.  Notmuch
works *out of the box* (almost) perfectly to index my collection of
ebooks.  All the notmuch commands work exactly as expected.  I can
search through the bodies, search the titles, search for an author,
search for a publication date, etc.  The emacs interface even works as
expected.  Try it: it really works!  There are only a couple of very
little things that are a little funky:

  * the "headers" in my ebooks aren't exactly intuitive ("From" instead
    of "Author", "Subject" instead of "Title", etc.) and there are some
    missing headers ("Publisher").  I also had to format some of them in
    a strange way (I had to add "<italo@calvino.com>" in the "From"
    field in order to get it to index properly for some reason).

  * The documentation keeps referring to "messages", even though my
    documents are books.  And there are some subcommands that don't seem
    to make sense ("reply" to a book?).

But that's it!  Everything else works as a perfect ebook indexer.  I can
of course even add tags to my books.  Beautiful.  It's really quite
incredible how well it works for this out of the box.  The only other
issue is that my ebooks don't come in rfc5322-formatted files.  I have
to translate them for notmuch to work.

So what would have to be tweaked in notmuch to make it work even better
as an ebook indexer?

  * add some sort of translator to extract the "headers" and "body" from
    my non-rfc5322-formatted ebook files

  * allow me to specify which "headers" from my ebooks I want indexed
    ("Author", "Publisher", etc.)

  * tweak notmuch show to just open the ebook itself in an ebook reader
    instead of outputting it to stdout

  * tweak the documentation

Those are not very big changes.  And yet, with these changes notmuch can
now work for *many* other large classes of structured documents.

Another real world example:

I have hundreds of scientific journal articles on my computer.  They are
all pdf files and each has a corresponding bibtex entry in a flat text
file.  If notmuch could read the headers from the bibtex file and the
body from the text in the pdf (ps2ascii), notmuch would work *perfectly*
as an indexer for my scientific journal articles.

So what do people think about this idea?  Does it make sense to look
into extending notmuch to handle non-mail documents?  We definitely
would *not* want to compromise notmuch as a mail indexer/reader.
Notmuch is the best damn mail system there ever was and we wouldn't want
to mess with that.  Does abstracting everything in notmuch from
"messages" -> "documents" hurt it as a mail system?  What if just the
back-end were abstracted, to allow for different front-ends for
different classes of documents, i.e. "messages", "articles", "books",
"rss feeds", etc.?  Are there any big problems with this proposal that
I'm overlooking?

I'm very interested to hear what others think about this idea.

jamie.

[0] http://tools.ietf.org/html/rfc5322

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: notmuch for documents

[Darren McGuicken]

[-- Attachment #1: Type: text/plain, Size: 1553 bytes --]

On Sat, 06 Nov 2010 16:12:17 -0400, Jameson Rollins <jrollins@finestructure.net> wrote:
> Notmuch is the best damn mail system there ever was and we wouldn't
> want to mess with that.  Does abstracting everything in notmuch from
> "messages" -> "documents" hurt it as a mail system?  What if just the
> back-end were abstracted, to allow for different front-ends for
> different classes of documents, i.e. "messages", "articles", "books",
> "rss feeds", etc.?  Are there any big problems with this proposal that
> I'm overlooking?
> 
> I'm very interested to hear what others think about this idea.

Absolutely 100% in agreement - the reason I use emacs for everything I
possibly can from web browsing to writing to coding to mailing is
because it's all just (occasionally arbitrarily formatted) text and all
text can be dealt with across buffers in exactly the same manner.  The
ability to use a single indexer and search interface for any of that
same text gets a big +1 from me.

I recently started using feed2imap in order to get notmuch tagging and
searching for rss, I'm sure I'm not alone.  I was just wondering how on
earth to go about adding headers to my sets of org-mode formatted notes,
for instance, so that notmuch could pick them up and index them for me.

In fact, the less of a distinction between the types of 'document' I'm
dealing with the better.  Maybe different document types get different
default keybindings - I may not want to 'reply' to an ebook but I
absolutely may want to forward it to someone?

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: notmuch for documents

[Jameson Rollins]

[-- Attachment #1: Type: text/plain, Size: 499 bytes --]

On Sat, 06 Nov 2010 20:40:10 +0000, Darren McGuicken <mailing-notmuch@fernseed.info> wrote:
> I recently started using feed2imap in order to get notmuch tagging and
> searching for rss, I'm sure I'm not alone.

Hey, Darren.  Total side note, but you might be interested in the recent
message from Kristoffer about his nice rss2message utility, "sluk", that
works great for translating rss feeds into notmuch-indexable messages:

http://github.com/krl/sluk/
id:"87d3rgzdyj.fsf@rymdkoloni.se"

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: notmuch for documents

[Nicolás Reynolds]

[-- Attachment #1: Type: text/plain, Size: 842 bytes --]

El 06/11/10 04:59, Jameson Rollins dijo:
> On Sat, 06 Nov 2010 20:40:10 +0000, Darren McGuicken <mailing-notmuch@fernseed.info> wrote:
> > I recently started using feed2imap in order to get notmuch tagging and
> > searching for rss, I'm sure I'm not alone.
> 
> Hey, Darren.  Total side note, but you might be interested in the recent
> message from Kristoffer about his nice rss2message utility, "sluk", that
> works great for translating rss feeds into notmuch-indexable messages:
> 
> http://github.com/krl/sluk/
> id:"87d3rgzdyj.fsf@rymdkoloni.se"

thanks to both of you, i was just looking for this a few weeks ago! :)


-- 
Salud!
Nicolás Reynolds,
xmpp:fauno@kiwwwi.com.ar
omb:http://identi.ca/fauno
blog:http://selfdandi.com.ar/
gnu/linux user #455044

http://librecultivo.org.ar
http://parabolagnulinux.org

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: notmuch for documents

[Darren McGuicken]

[-- Attachment #1: Type: text/plain, Size: 625 bytes --]

On Sat, 06 Nov 2010 16:59:17 -0400, Jameson Rollins <jrollins@finestructure.net> wrote:
> Hey, Darren.  Total side note, but you might be interested in the recent
> message from Kristoffer about his nice rss2message utility, "sluk", that
> works great for translating rss feeds into notmuch-indexable messages:

Thanks for the heads up Jamie - I saw this come through but at the time
it didn't seem to support most of the feeds I was reading.  I didn't
look too closely since feed2imap worked for me.  I'll check out the
latest version and see if that has changed.  I do like the idea of
notmuch-to-feed as a companion tool.

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: notmuch for documents

[Darren McGuicken]

[-- Attachment #1: Type: text/plain, Size: 1213 bytes --]

On Sat, 06 Nov 2010 16:12:17 -0400, Jameson Rollins <jrollins@finestructure.net> wrote:
> But that's it!  Everything else works as a perfect ebook indexer.  I
> can of course even add tags to my books.  Beautiful.  It's really
> quite incredible how well it works for this out of the box.  The only
> other issue is that my ebooks don't come in rfc5322-formatted files.
> I have to translate them for notmuch to work.

I've now had a chance to play with this a little and while indexing,
tagging and searching all seem to work as expected, I am getting the
error 'Stack overflow in regexp matcher' when I try to view any of the
ebooks which either leaves the buffer basically useless (no notmuch key
shortcuts will work) or leads to a full segfault in emacs (23.1.1).

The trace begins:

Debugger entered--Lisp error: (error "Stack overflow in regexp matcher")
  re-search-forward("\\(^[^>]+\\)\n>" nil t)
  notmuch-wash-tidy-citations(0)
  run-hook-with-args(notmuch-wash-tidy-citations 0)
  notmuch-show-insert-part-text/plain((:body ((:content "The Project
  Gutenberg EBook of The Adventures of Sherlock Holmes\nby Sir Art...

The contents of the ':content' part appears to be the complete text of
the novel.

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: notmuch for documents

[Darren McGuicken]

[-- Attachment #1: Type: text/plain, Size: 582 bytes --]

On Sat, 06 Nov 2010 23:28:21 +0000, Darren McGuicken <mailing-notmuch@fernseed.info> wrote:
> I've now had a chance to play with this a little and while indexing,
> tagging and searching all seem to work as expected, I am getting the
> error 'Stack overflow in regexp matcher' when I try to view any of the
> ebooks which either leaves the buffer basically useless (no notmuch key
> shortcuts will work) or leads to a full segfault in emacs (23.1.1).

Hmm, looks like Michal's patch back in July fixes this behaviour:

     id:"1279279955-3110-1-git-send-email-sojkam1@fel.cvut.cz"

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: notmuch for documents

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 842 bytes --]

On Sat, 06 Nov 2010 16:12:17 -0400, Jameson Rollins <jrollins@finestructure.net> wrote:
> expected.  Try it: it really works!  There are only a couple of very
> little things that are a little funky:

Hey, that's pretty cool. I'm glad it's working so well for you.

>   * allow me to specify which "headers" from my ebooks I want indexed
>     ("Author", "Publisher", etc.)

Fortunately, that's a feature we're already planning to add (quite
soon!) even just to better support the indexing of email.

> So what do people think about this idea?  Does it make sense to look
> into extending notmuch to handle non-mail documents? 

I'm cautious about making any big changes to support a much wider use
case. But where little changes can make a big improvement here, I'm
ready to listen.

-Carl

-- 
carl.d.worth@intel.com

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]