From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by olra.theworths.org (Postfix) with ESMTP id 3A712431FBC for ; Sun, 14 Feb 2010 16:29:24 -0800 (PST) X-Virus-Scanned: Debian amavisd-new at olra.theworths.org X-Spam-Flag: NO X-Spam-Score: -0.721 X-Spam-Level: X-Spam-Status: No, score=-0.721 tagged_above=-999 required=5 tests=[AWL=-0.722, BAYES_50=0.001] autolearn=ham Received: from olra.theworths.org ([127.0.0.1]) by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id LdXSoGrFFfL8 for ; Sun, 14 Feb 2010 16:29:23 -0800 (PST) Received: from kaylee.flamingspork.com (kaylee.flamingspork.com [74.207.245.61]) by olra.theworths.org (Postfix) with ESMTP id 2C829431FAE for ; Sun, 14 Feb 2010 16:29:23 -0800 (PST) Received: from willster (localhost [127.0.0.1]) by kaylee.flamingspork.com (Postfix) with ESMTPS id D2F746393 for ; Mon, 15 Feb 2010 00:26:23 +0000 (UTC) Received: from flamingspork.com (localhost.localdomain [127.0.0.1]) by willster (Postfix) with ESMTPS id C7DD51023D8F for ; Mon, 15 Feb 2010 11:29:18 +1100 (EST) Date: Mon, 15 Feb 2010 11:29:14 +1100 From: Stewart Smith To: notmuch@notmuchmail.org Message-ID: <20100215002914.GA22402@flamingspork.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.20 (2009-06-14) Subject: Mail in git X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Feb 2010 00:29:24 -0000 So... I sketched this out in my head at LCA... and it's taken a bit of time to actually properly try it. The problem is: A simple 'find ~/Maildir` takes 10 minutes, and if you write the output to a file, it's 88MB+ there's "only" about 900,000 entries there. But this means 900,000 files, which is a non-trivial amount. Some mail folders are quite large too. Some of this problem could just be solved by using notmuch a bit differently (folder per month for example). However... this is a one-way change and going back would be very tricky. There's also the backup problem. Iterating through ~1million inodes takes a *LONG* time. Restoring it takes even longer (think about writing all that data to the file system journal). Historically, if i'm running a backup, I couldn't really use my laptop, it'd be saturated with disk IO performing the file system dump. It would also take many hours. Restoring from backup? about 8hrs. An observation is that mail never changes. It may be reclassified (and that's what notmuch is for), but it never changes. We really just want a way to store and access many many many small blobs of data that never change. It turns out git is pretty good at that. Underneath, we could just use it as an object store (a simple git-hash-object and git-cat-file test confirmed this to be pretty simple to do). even better is since a lot of mail is fairly similar, to use delta compression between mail messages to reduce the storage space. Git is pretty good at that too. A few giant git packs will be much quicker to backup and restore than 1million files. So... I wrote a script to test it.... $ time perl /home/stewart/evenless.pl /home/stewart/Maildir/ real 841m41.491s user 491m3.200s sys 261m58.080s Which goes from a 15GB Maildir to a 3.7GB git repo. The algorithm of evenless.pl is basically: 1 get next directory entry 2 if is directory, recurse into it 3 write item to git (git hash-object -w) 4 add item to tree object 5 if number of items written = 1000 5.1 make pack of last 1000 items 6 goto 1 $ git count-objects -v count: 479 size: 27680 in-pack: 873109 packs: 1084 size-pack: 3746219 prune-packable: 0 garbage: 0 If i did a "git checkout", about 8 hours later i'd have a directory tree exactly the same as my maildir. Why didn't I just git-add everything? I didn't exactly feel like creating another giant copy of my mail (that also takes a long time). What about adding more mail to the archive? So the way I think is that you use a Maildir for day to day mail (e.g. delivery) and every so often you run some magic command that takes old mail out of the Maildir and stores it in the git repo. Next step? Make notmuch be able to read mail out of it and add it to an index (oh, and some kind of verification and error checking about creating the git repo). -- Stewart Smith