From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by arlo.cworth.org (Postfix) with ESMTP id 49C5F6DE00F1 for ; Thu, 24 Mar 2016 03:40:47 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at cworth.org X-Spam-Flag: NO X-Spam-Score: -1.083 X-Spam-Level: X-Spam-Status: No, score=-1.083 tagged_above=-999 required=5 tests=[AWL=-1.073, T_RP_MATCHES_RCVD=-0.01] autolearn=disabled Received: from arlo.cworth.org ([127.0.0.1]) by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kMPWFhRXwnlV for ; Thu, 24 Mar 2016 03:40:38 -0700 (PDT) Received: from 4angle.com (4angle.com [82.145.46.9]) by arlo.cworth.org (Postfix) with ESMTPS id B56036DE0005 for ; Thu, 24 Mar 2016 03:40:38 -0700 (PDT) Received: from cpc91242-cmbg18-2-0-cust1730.5-4.cable.virginm.net ([82.8.134.195] helo=x230) by 4angle.com with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86) (envelope-from ) id 1aj2gr-0008Bj-QV for notmuch@notmuchmail.org; Thu, 24 Mar 2016 10:40:33 +0000 Received: from edward by x230 with local (Exim 4.87) (envelope-from ) id 1aj2gr-0006sc-8s for notmuch@notmuchmail.org; Thu, 24 Mar 2016 10:40:33 +0000 Date: Thu, 24 Mar 2016 10:40:33 +0000 From: Edward Betts To: notmuch@notmuchmail.org Subject: FastMail uses xapian for real time search, can we do the same? Message-ID: <20160324104033.GA23082@4angle.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.24 (2015-08-30) X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Mar 2016 10:40:47 -0000 I've tried using 'notmuch insert' with mutt-kz, the mutt fork that talks to the notmuch database. Both keep trying to grab the write lock at the same time. I tag a mail in mutt-kz and it writes the new tag to the database, at the same time a new mail is coming in. Either 'notmuch insert' has the lock and my mail client pauses while it retries or notmuch-insert isn't able to get the write lock and just saves the mail to the maildir and it doesn't get added until I later run 'notmuch new'. I guess many people here are using the Emacs client and the problem is partly avoided because there is a separate thread in the background trying to grab the write lock and save tag changes. Is that correct? I wonder if the solution is real time search? xapian can search in multiple databases. New mails and tag changes can be written to a new database, there could be one database for inserting new mails and another for changes to the tags on existing mails. Then there are no problems with write locks. These databases will be much smaller, so the writes should be faster. There can be a process that runs once a day to merge the databases. FastMail have implemented real time search for email with xapian. Blog post: https://blog.fastmail.com/2014/12/01/email-search-system/ Implementation: https://github.com/brong/cyrus-imapd/tree/fastmail Is there any enthusiasm for adding real time search to notmuch? -- Edward.