On Sat, 28 Nov 2009 10:02:03 -0600, "Jeffrey C. Ollie" <jeff@ocjtech.us> wrote:
> This is the beginning of a test suite.  It uses the Check[1] unit
> testing framework to handle the testing.  There's one basic test of
> the SHA1 routines, obviously many more will need to be added.

Hi Jeffrey,

Thanks very much for this contribution. I'm always highly in favor of
improving our testing infrastructure. And a good test suite was on of
the first items I put on the notmuch/TODO list.

However, it's not clear to me that this kind of C-based unit testing
will be able to easily support much of what I want from testing for
notmuch. It seems that most of the bugs I encounter, or features that I
want to test, require a lot of state to be setup that would be really
painful to do in C code.

For example, a recent bug I hit was:

	After moving an (old) existing file into the mail store,
	"notmuch new" won't pick it up.

And I don't see an easy way to test something like that from a typical
"check" test. Meanwhile, I can't see a lot of use from the kinds of
things that *could* be checked from a check test. We could easily check
little things like that after calling notmuch_message_add_tag that
notmuch_message_get_tags has the same tag in its list. But if we have to
do higher-level testing anyway, (for cases like the bug above), then
it's easy to cover fine-grained stuff like this there as well.

For the recent rename-support work I did I wrote the attached
notmuch-test script. It does the kind of high-level state setup I
described above, (and actually contains a test for precisely the bug I
described above).

I haven't added this to the repository yet since it needs a bunch of
work before being usable. Here are some things it needs:

  * Automated verification that tests are working, (currently I'm just
    manually verifying that the results are as expected---obviously
    that's not practical in the long term).

  * Modularization so that each test can be maintained as a small,
    independent snippet.

  * The ability to run an individual test in isolation without running
    the whole suite.

  * A fix in notmuch to get rid of the stupid sleep calls that are
    slowing down the current notmuch-test. (One approach would be to
    stop using mtime altogether. That might slow things down
    unacceptably in general use, but might be just fine for the test
    suite where the mail store can generally be quite tiny. Another
    approach would be to use a filesystem like ext4 with sub-second
    mtime support).

Anyway, just wanted to share some thoughts and some preliminary code for
a notmuch test suite.

-Carl