FastMail uses xapian for real time search, can we do the same?

FastMail uses xapian for real time search, can we do the same?

[Edward Betts]

I've tried using 'notmuch insert' with mutt-kz, the mutt fork that talks to
the notmuch database. Both keep trying to grab the write lock at the same time.

I tag a mail in mutt-kz and it writes the new tag to the database, at the same
time a new mail is coming in. Either 'notmuch insert' has the lock and
my mail client pauses while it retries or notmuch-insert isn't able to get the
write lock and just saves the mail to the maildir and it doesn't get added
until I later run 'notmuch new'.

I guess many people here are using the Emacs client and the problem is partly
avoided because there is a separate thread in the background trying to grab
the write lock and save tag changes. Is that correct?

I wonder if the solution is real time search? xapian can search in multiple
databases. New mails and tag changes can be written to a new database, there
could be one database for inserting new mails and another for changes to the
tags on existing mails. Then there are no problems with write locks. These
databases will be much smaller, so the writes should be faster. There can be a
process that runs once a day to merge the databases.

FastMail have implemented real time search for email with xapian.

Blog post: https://blog.fastmail.com/2014/12/01/email-search-system/

Implementation: https://github.com/brong/cyrus-imapd/tree/fastmail

Is there any enthusiasm for adding real time search to notmuch?
-- 
Edward.

Re: FastMail uses xapian for real time search, can we do the same?

[David Bremner]

Edward Betts <edward@4angle.com> writes:

>
> Is there any enthusiasm for adding real time search to notmuch?

It sounds interesting. I do wonder if just better (read, any) retry
handling might give most of the user benefits with fewer moving parts.

I think the real reason that the locking failures bother me less is just
a workflow one: I don't have notmuch-insert / notmuch-new running in the
background, but rather fetch a few few times a day. I'm not saying
everyone should read mail like I do, just explaining why it's less of a
priority for me personally than some other issues.

d