Quick thoughts on a notmuch daemon

Quick thoughts on a notmuch daemon

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 1165 bytes --]

Here's a little idea in case someone wants to write some code before I
get around to it.

Two current problems:

  1. You can't read mail (updating tags, etc.) at the same time new mail
     is being incorporated to the database. So if you have "notmuch new"
     as a cron job, then you will sometimes see "failed to obtain write
     lock" in emacs.

  2. Lots of people want to interact with a notmuch database on a remote
     server, for obvious reasons.

A simple solution would be a notmuch daemon that can accept commands on
stdin, (in basically the exact same form as the current notmuch
command-line interface). If the daemon does the job of periodically
incorporating new mail, then the only command necessary to solve (1)
above would be the tag command.

But with the other commands in place, (such as search and show), then
this could operate over ssh and all that would be necessary is one
additional command not in the current command line, (for "give me this
file"). And then the client could just maintain a local cache of mail
files as needed.

So that might be a nice project for somebody, and would likely be very
useful for many people.

-Carl

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Quick thoughts on a notmuch daemon

[David Bremner]

On Thu, 03 Dec 2009 14:27:05 -0800, Carl Worth <cworth@cworth.org> wrote:
> But with the other commands in place, (such as search and show), then
> this could operate over ssh and all that would be necessary is one
> additional command not in the current command line, (for "give me this
> file"). And then the client could just maintain a local cache of mail
> files as needed.

It might be a bit blue sky, but if this daemon could (optionally) talk
IMAP and translate tags into folders on the fly, this would be very
useful for people who need imap access to their mail as well as from an
custom notmuch client.

Of course it would be nice to be able to just drop a back-end into some 
well tested server that dealt with authentication and the imap protocol.

As long as we're tossing out projects :)

d

Re: Quick thoughts on a notmuch daemon

[Marten Veldthuis]

On Mon, 07 Dec 2009 15:02:03 -0400, David Bremner <david@tethera.net> wrote:
> 
> On Thu, 03 Dec 2009 14:27:05 -0800, Carl Worth <cworth@cworth.org> wrote:
> It might be a bit blue sky, but if this daemon could (optionally) talk
> IMAP and translate tags into folders on the fly, this would be very
> useful for people who need imap access to their mail as well as from an
> custom notmuch client.

For me, my IMAP needs are pretty much limited to my iPhone. I'm making
do for now, but to make notmuch viable in the long term, what I'd need
is:

 * notmuch shouldn't choke on mails I had in notmuch's database, and
   then marked read or deleted on my iPhone (which renames them in the
   maildir). This is coming with the moving/deleting patches.
 * notmuch should sync back read/unread state to maildir
 * notmuch should move read stuff out of my inbox. It would be
   acceptable if it moved everything into a designated archive folder
   unless it had the 'inbox' tag assigned, in which case it moved it
   there. Note that we have the moving/deleting patches, then this could
   even be done as a script and some searches.

With this, my inbox would be usable from my iPhone.

-- 
- Marten

Re: Quick thoughts on a notmuch daemon

[micah anderson]

[-- Attachment #1: Type: text/plain, Size: 5498 bytes --]

Excerpts from Marten Veldthuis's message of Mon Dec 07 17:55:24 -0500 2009:
> On Mon, 07 Dec 2009 15:02:03 -0400, David Bremner <david@tethera.net> wrote:
> > 
> > On Thu, 03 Dec 2009 14:27:05 -0800, Carl Worth <cworth@cworth.org> wrote:
> > It might be a bit blue sky, but if this daemon could (optionally) talk
> > IMAP and translate tags into folders on the fly, this would be very
> > useful for people who need imap access to their mail as well as from an
> > custom notmuch client.
> 
> For me, my IMAP needs are pretty much limited to my iPhone. I'm making
> do for now, but to make notmuch viable in the long term, what I'd need
> is:
> 
>  * notmuch shouldn't choke on mails I had in notmuch's database, and
>    then marked read or deleted on my iPhone (which renames them in the
>    maildir). This is coming with the moving/deleting patches.
>  * notmuch should sync back read/unread state to maildir
>  * notmuch should move read stuff out of my inbox. It would be
>    acceptable if it moved everything into a designated archive folder
>    unless it had the 'inbox' tag assigned, in which case it moved it
>    there. Note that we have the moving/deleting patches, then this could
>    even be done as a script and some searches.
> 
> With this, my inbox would be usable from my iPhone.

I dont have an iPhone, but this would be a *killer* feature for me, and
I suspect for others as well. 

I have two fundamental problems with label-state clients (sup, notmuch)
that I am still trying to come to terms with, this bluesky feature would
change that radically.

The first issue is that I do not have unlimited IMAP storage space on
the server, in fact I have a quota. I know, lots of people have
effectively unlimited space,and others POP/fetchmail things so its not
stored on the server...but lots of us still do have upper limits that we
have to deal with and many of us do use IMAP, for good reasons. With
label-state clients (sup, notmuch) you have to accept the fact that your
mail is going to grow on the IMAP server indefinitely. This is not a
good thing for those of us with quotas, but it is also a bad thing for
some IMAP servers who choke to death on very large IMAP stores (or
consume an awful lot of memory dealing with them). Or you have to setup
kludgy archive operations to periodically deal with the disk space
issue.

The second mind-bending problem is that sometimes I actually do have to
use mutt, sometimes webmail (for various reasons, but one important one
in the early stages of a new email client is that notmuch just doesn't
have support for certain operations such as inline/pgp-mime handling of
emails, which means I need to open those emails in other clients that do
support that). Other people likely are going to need to use things like
iphones or blackberries. Indeed, using other clients is not an
unreasonable thing for people to do. However, switching between notmuch
and these clients is. Because the message state is stored in a DB which
is only useful for notmuch, its a dreadful nightmare to do anything
outside of the notmuch world.

Most importantly, having all of your email state in a notmuch database
format means that it can only be parsed by those tools, and is no longer
in a standard format. I think it is great that 'notmuch restore' can
deal with the sup database format, which may signal the dawn of a new
label-state standard....but still the problem is the glue to the
underlying maildir structure which provides a lot of useful information
contained in reasonably well-defined ways is totally thrown out the
window. 

I've switched mail clients enough over the past few years to know that
switching itself is a major pain. I've settled on Maildir as my storage
method of choice and in a way it is a 'client'. I can serve it via IMAP
if I need, I can read it with different mail clients and maintain state
across those mail clients, there are tools to manipulate and convert
maildirs that have matured over the years. If I switch to an
'all-in-one' label-state mail client, like notmuch, I want to be sure
that in 2 years from now if I happen to decide to change to something
else (I'm hoping this wont EVER happen because notmuch is very
promising, but I need to be honest based on past experiences here), then
I'm going to want to make sure that all of my mail is marked as read,
deleted mail is deleted and even though I was using labels for
organizational purposes in notmuch, things are still organized in some
way appropriate to folders on the filesystem.

The way it is now, if I switch to notmuch and then try to switch back,
my life is a total mess because all of the state is contained within the
notmuch database, that is frightening for the long-term, but it also
makes me very worried about simple corruption of that data store. If I
lose that state, I'm totally screwed. At least in a maildir scenario, if
I got corruption in one place it might cause me to lose one email, but
if I get corruption in my notmuch database, I had better have a recent
backup or I am screwed. 

The important part is gluing the maildir capabilities in there, and not
getting distracted by the transport (let offlineimap and its ilk handle
this, maybe some eager python hacker might find a way to add this to
offlineimap?!).

I like to think that this concept of label-state, indexable
storage-backed mail clients is the dawn of a new age, just as maildir
was to mbox, but I'm still scared because there is no mb2md equivalent
yet.

micah

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Quick thoughts on a notmuch daemon

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 3948 bytes --]

On Mon, 07 Dec 2009 18:51:58 -0500, micah anderson <micah@riseup.net> wrote:
> I've switched mail clients enough over the past few years to know that
> switching itself is a major pain.

Absolutely.

One concept in notmuch (compared to sup) is to (eventually) avoid people
having to go through that pain by their current mail client becoming
"notmuch enabled". For me, I had always liked composing email in emacs,
so I just have to add notmuch support to the existing emacs
message-mode.

Hopefully people working on other email interfaces will do similar
things, (would be great to have Anjal and Thunderbird get some notmuch
support for example).

I definitely didn't like that with sup, to get all the global-search and
tagging features one had to accept the curses UI as well.

>                                            If I switch to an
> 'all-in-one' label-state mail client, like notmuch, I want to be sure
> that in 2 years from now if I happen to decide to change to something
> else (I'm hoping this wont EVER happen because notmuch is very
> promising, but I need to be honest based on past experiences here), then
> I'm going to want to make sure that all of my mail is marked as read,
> deleted mail is deleted and even though I was using labels for
> organizational purposes in notmuch, things are still organized in some
> way appropriate to folders on the filesystem.

I appreciate your caution before making the commitment to a mail
client. I think that's very wise. [*]

And I think that for the case of "I'm giving up on notmuch and want to
switch to something else" you can be quite assured that notmuch will
allow you to take care of everything you need. At a system level, it's
really an almost trivial matter to implement everything you describe
above as small shell scripts based on "notmuch search" commands. And I
can only imagine support for that kind of thing getting better all the
time.

> The way it is now, if I switch to notmuch and then try to switch back,
> my life is a total mess because all of the state is contained within the
> notmuch database, that is frightening for the long-term, but it also
> makes me very worried about simple corruption of that data store. If I
> lose that state, I'm totally screwed. At least in a maildir scenario, if
> I got corruption in one place it might cause me to lose one email, but
> if I get corruption in my notmuch database, I had better have a recent
> backup or I am screwed. 

I highly recommend making very regular backups of the output of "notmuch
dump". It's a quick process to run, and the file it creates is not
large. It's also nicely sorted so that if you are doing some kind of
incremental backup, that should work well.

> I like to think that this concept of label-state, indexable
> storage-backed mail clients is the dawn of a new age, just as maildir
> was to mbox, but I'm still scared because there is no mb2md equivalent
> yet.

I think we're really close to where you could write a notmuch2md script
as a very tiny shell script:

for tag in $(notmuch search --for=tags *); do
	notmuch search --for=messages --output=maildir:/some/dir/$tag tag:$tag
done

Being able to do that kind of scriptable output is pretty powerful. And
we're not missing anything fundamental in the system to be able to
support that, (just a little bit of support for the new command-line
options like --for and --output).

-Carl

[*] I know I went through a similarly cautious evaluation when switching
away from CVS. CVS made the transition away so painful that I was
determined to choose a system that satisfied me on two criteria:

1. The system seemed well-designed enough that I could imagine it
   surviving "forever".

2. That when realities exceeded my imagination, the system wouldn't make
   it hard for me to switch away.

So far I'm still quite happy with git on both points.

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Quick thoughts on a notmuch daemon

[Michiel Buddingh']

On Thu, 03 Dec 2009 14:27:05 -0800, Carl Worth <cworth@cworth.org> wrote:
> A simple solution would be a notmuch daemon that can accept commands on
> stdin, (in basically the exact same form as the current notmuch
> command-line interface). If the daemon does the job of periodically
> incorporating new mail, then the only command necessary to solve (1)
> above would be the tag command.

If you add a second pipe for notmuch to broadcast information about
events (such as new mail being indexed) you could farm out most of the
logic that will increasingly clutter up notmuch-new.c to an external
daemon.

Just the mailid and path would be enough for people to implement their
own tagging based on directory, Maildir flags or (for all I care)
Bayesian content filtering with relative ease.

Just a thought
-- 
Michiel Buddingh'

Re: Quick thoughts on a notmuch daemon

[Ruben Pollan]

[-- Attachment #1: Type: text/plain, Size: 992 bytes --]

On 14:27, Thu 03 Dec 09, Carl Worth wrote:
> A simple solution would be a notmuch daemon that can accept commands on
> stdin, (in basically the exact same form as the current notmuch
> command-line interface). If the daemon does the job of periodically
> incorporating new mail, then the only command necessary to solve (1)
> above would be the tag command.

I like the idea. I didn't liked to fork for each command, so I started to play
with the library for create a UI. But with a demon like that I guess will be
nicer to use it than to call directly to the library.

Why use stdin? Why not sockets? With them at could be possible to use several 
concurrent clients with the same server.
(I really love moc for play music, and one of its greatest features is that)


-- 
Rubén Pollán  | jabber:meskio@jabber.org
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
         No vamos a reivindicar nada,
            no vamos a pedir nada.
            Tomaremos, okuparemos.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Quick thoughts on a notmuch daemon

[martin f krafft]

[-- Attachment #1: Type: text/plain, Size: 2959 bytes --]

also sprach Carl Worth <cworth@cworth.org> [2009.12.08.2001 +1300]:
> One concept in notmuch (compared to sup) is to (eventually) avoid people
> having to go through that pain by their current mail client becoming
> "notmuch enabled". For me, I had always liked composing email in emacs,
> so I just have to add notmuch support to the existing emacs
> message-mode.
> 
> Hopefully people working on other email interfaces will do similar
> things, (would be great to have Anjal and Thunderbird get some notmuch
> support for example).
> 
> I definitely didn't like that with sup, to get all the global-search and
> tagging features one had to accept the curses UI as well.

I am a bit late to the party, but re: this thread [0], I would
suggest to go the way of a fuse filesystem. That's effectively
a daemon, but one which also bridges a chasm between notmuch and all
kinds of existing mail tools, including IMAP servers, by way of the
standard filesystem interface.

0. http://notmuchmail.org/pipermail/notmuch/2009/000782.html

I don't want to harp on this too much right now for I have not yet
fully understood notmuch, but the basic idea would be that you'd
have ~/mail provided by notmuch-fuse-daemon, and there'd be a tool
like notmuch-fuse-cli with which you can add virtual folders, e.g.

  notmuch-fuse-cli new debianmail 'from:debian OR to:debian'

and that would create ~/mail/debianmail with mode 555 (since you
cannot write the results of a search) containing a Maildir with all
messages matching the query.

The benefit of this would be that I could use mutt, evolution, or an
IMAP server, or vi and shell tools to manipulate my mail without any
modifications to those tools.

There could be a separate hierarchy for tags, e.g. ~/mail/TAGS/foo
and ~/mail/TAGS/bar/baz matching on explicit tags (and maybe
~/mail/TAGS/notmuch with mode 555 for implicit tags). Writing mail
to those directories effectively adds tags, unlinking removes them.
~/mail/TAGS/UNTAGGED holds untagged mail for easier reference.

In addition to all of this, fuse could be used to index new messages
directly as they are delivered into ~/mail, rather than running
'notmuch new' regularly.

These ideas are not new, and I've written about them before:

http://madduck.net/blog/2007.07.24:a-user-space-filesystem-for-mail-labeling/

notmuch seems an excellent base for implementing such a filesystem.
I will try to make time before LCA to get up to speed on fuse, then
maybe Carl and Micah and I (and whoever else will be in Wellington)
can hack this up in a few hours and over a few beers.

If this resonates, or you want to work on this too, let's hear from
you!

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
"no problem is so formidable
 that you can't just walk away from it."
                                                          -- c. schulz
 
spamtraps: madduck.bogus@madduck.net

[-- Attachment #2: Digital signature (see http://martin-krafft.net/gpg/) --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Quick thoughts on a notmuch daemon

[Mike Hommey]

On Fri, Jan 08, 2010 at 03:56:20PM +1300, martin f krafft wrote:
> These ideas are not new, and I've written about them before:
> 
> http://madduck.net/blog/2007.07.24:a-user-space-filesystem-for-mail-labeling/
> 
> notmuch seems an excellent base for implementing such a filesystem.
> I will try to make time before LCA to get up to speed on fuse, then
> maybe Carl and Micah and I (and whoever else will be in Wellington)
> can hack this up in a few hours and over a few beers.
> 
> If this resonates, or you want to work on this too, let's hear from
> you!

I'm in \o_ (though I won't be in Wellington). I've been thinking about a
fuse filesystem on top of notmuch for a while.

Mike

Re: Quick thoughts on a notmuch daemon

[martin f krafft]

[-- Attachment #1: Type: text/plain, Size: 822 bytes --]

also sprach Mike Hommey <mh+notmuch@glandium.org> [2010.01.08.2106 +1300]:
> I'm in \o_ (though I won't be in Wellington). I've been thinking
> about a fuse filesystem on top of notmuch for a while.

Grand news to see you interested! A FUSE filesystem is <25 functions
to implement, and each function is basically an entity of its own
and thus highly parallisable. Once we agreed on a general mapping
between filesystem i/o and notmuch interaction, 25 of us can write
a function each and be done. How's that for collaboration? ;)

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
"courage is not the absence of fear, but the decision
 that something else is more important than fear."
                                                  -- ambrose redmoon
 
spamtraps: madduck.bogus@madduck.net

[-- Attachment #2: Digital signature (see http://martin-krafft.net/gpg/) --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Quick thoughts on a notmuch daemon

[Mike Hommey]

On Fri, Jan 08, 2010 at 10:03:21PM +1300, martin f krafft wrote:
> also sprach Mike Hommey <mh+notmuch@glandium.org> [2010.01.08.2106 +1300]:
> > I'm in \o_ (though I won't be in Wellington). I've been thinking
> > about a fuse filesystem on top of notmuch for a while.
> 
> Grand news to see you interested! A FUSE filesystem is <25 functions
> to implement, and each function is basically an entity of its own
> and thus highly parallisable. Once we agreed on a general mapping
> between filesystem i/o and notmuch interaction, 25 of us can write
> a function each and be done. How's that for collaboration? ;)

FYI, I have a good experience writing fuse filesystems, both with
high-level and low-level APIs. I'd avise to use the low-level API, which
allows for better performance.

Mike

Re: Quick thoughts on a notmuch daemon

[martin f krafft]

[-- Attachment #1: Type: text/plain, Size: 754 bytes --]

also sprach Mike Hommey <mh+notmuch@glandium.org> [2010.01.08.2220 +1300]:
> FYI, I have a good experience writing fuse filesystems, both with
> high-level and low-level APIs. I'd avise to use the low-level API,
> which allows for better performance.

I don't have any experience with FUSE yet, but the examples in
/usr/share/doc/libfuse-dev/examples/ look trivial. This is where
I would start, one function at a time. If you have a better
suggestion, I'd love to hear it; or to clone your repo! ;)

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
be careful of reading health books -- you might die of a misprint.
                                                         -- mark twain
 
spamtraps: madduck.bogus@madduck.net

[-- Attachment #2: Digital signature (see http://martin-krafft.net/gpg/) --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Quick thoughts on a notmuch daemon

[Mike Hommey]

On Fri, Jan 08, 2010 at 11:26:31PM +1300, martin f krafft wrote:
> also sprach Mike Hommey <mh+notmuch@glandium.org> [2010.01.08.2220 +1300]:
> > FYI, I have a good experience writing fuse filesystems, both with
> > high-level and low-level APIs. I'd avise to use the low-level API,
> > which allows for better performance.
> 
> I don't have any experience with FUSE yet, but the examples in
> /usr/share/doc/libfuse-dev/examples/ look trivial. This is where
> I would start, one function at a time. If you have a better
> suggestion, I'd love to hear it; or to clone your repo! ;)

As I said above, there are 2 sets of APIs in FUSE.

The high-level API sends the full path for the file being accessed for
every system call. And except for specific cases such as read(), write()
or readdir() you have nothing else to identify the file you are referring
to, which means you have to parse the path, and find the proper file
accordingly.
In notmuch case, that would mean doing a search for most system calls.
Try to imagine how many syscalls that are not read(), write() or
readdir() mutt does when opening a Maildir.

The low-level API, otoh, uses inode numbers extensively (again, except
for read, write and readdir). The lookup call is responsible for resolving
the paths, given an inode and a name. Its results are cached by the kernel.
So, for example reading foo/bar from your fuse mount point will lookup
foo in the inode 1 (FUSE_ROOT_ID) and then do another lookup for bar in
the first result.
One of the problems with this API is that the inode number type is
unsigned long, which means you can't necessarily map real inode numbers,
which can be 64 bits. And even if it could, afaik, there is no quick way
to get a file from its inode, sadly.

All in all, in the high-level API case, that means we would need lookups
caching badly, and in the low-level API case, some fast way to map on
one hand virtual directories with inodes numbers, and on the other hand,
real files with inode numbers.

Some quick thoughts, about the whole thing:
- We will need to be careful about deduplication: if you copy a file
  from one directory to another, you don't want to have the copy in the
  underlying Maildir. But as you won't know until the file is totally
  written and closed...
- We should probably allow extra files to be stored in the virtual
  Maildir (for example, courierimap stores stuff in a Maildir)
- We may not need a client program at all, the "search directories"
  configuration could be handled via extended file attributes.

I also had another not quite unrelated idea a while ago, that could have
its value here: a generic data store, very much like the git object
database (an idea would be to have the git object datastore be a special
case of this generic data store, for possibly interesting compatibility),
which would allow for better storage of the messages: if the maildir is
exposed via fuse, why would you need a raw maildir for ? It would also
allow easier deduplication of messages that are different but not quite:
- Mailing list replies you get both directly and from the mailing
  list software, their headers have differences, but the files are mostly
  equivalent
- Mail quotes are found in both the original message and its response.

Mike

Re: Quick thoughts on a notmuch daemon

[Ruben Pollan]

[-- Attachment #1: Type: text/plain, Size: 1320 bytes --]

I started to code something base on your idea of a notmuch daemon. You can find
it on:
git://gitorious.org/notsomuch/notmuch.git
On the server branch.

The idea is to use unix named sockets to intercomunicate between the daemon and the
client. And threads on the server to handle every request. The implementation is 
no great, it's a fast hack. It can only handle one request per connection and breaks 
some times on concurrent request. But I hope helps to see the idea.

I implemented both, daemon and client in the same binary. So you can still run
as before:
$ notmuch search inbox
If the daemon is already running (so the socket is in MAILDIR_PATH/.notmuch/socket) 
it will connect to it and ask for the search. If is not running will fork
creating it and send it the search.

Up to now the comunication between daemon and client is with the same syntax of
notmuch. But I think will be a nice idea to use JSON (or some other
computer-friendly syntax) and convert it to human readable on the client.

What do you think about that approach? Will it fit on what you imagined or is it
to far?

I'm not sure if that is adding to much complexity to notmuch or is a good idea.

-- 
Rubén Pollán  | jabber:meskio@jabber.org
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Nos vamos a Croatan.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]