On Thu, 26 Nov 2009 11:16:21 -0800, Carl Worth <cworth@cworth.org> wrote:
> Clearly, some experimenting is needed. Dominik, if you can share the
> large file, (with either me alone or with the whole list), a pointer to
> where we could download it would be appreciated.

Dominik replied to me privately and described a way for me to create a
file that replicates the bug. Here's a recipe I came up with from his
description:

	mkdir tmp
	cd tmp/
	echo [database]$'\n'path=mail > notmuch-config
	mkdir mail
	echo From: Me$'\n'To: You$'\n'Subject: uuencode$'\n' > mail/msg
	dd if=/dev/urandom of=blob bs=1024 count=10240
	uuencode blob < blob >> mail/msg
	NOTMUCH_CONFIG=notmuch-config notmuch new

So that's a 10MB blob of random data which uuencodes to a ~14MB mail
file. And notmuch (before a patch I just pushed) chews on it for quite a
while, consuming several hundred MB of memory and resulting finally in a
76MB Xapian database (with chert).

I'm not sure if there is a Xapian bug there or not, (or perhaps a bug in
how notmuch is using Xapian to generate the terms for this large of an
email message).

But the thing that's obvious to me is that indexing encoded data like
this doesn't make any sense at all. So I've just pushed a set of patches
to notmuch to make it detect uuencoded data within a mail message and
ignore it.

Of course, I also pushed a set of tests to the test suite for this, (and
some new "notmuch search" tests while I was at it).

-Carl