Re: unit test framework

[Christian Ohler <ohler+emacs@fastmail.net>]

joakim@verona.se, 2010-04-13:

> Stefan Monnier <monnier@IRO.UMontreal.CA> writes:
> 
>>> I write try to write unit tests for my elisp, and it would be much
>>> easier if there was a canonical unit test framework in Emacs.
>> Yes, that's the intention.  We should pick one and run with it.
>> The choice doesn't have to be "the very best", but it should aim for:
>> - simple&clean code, easy to maintain&extend.
>> - well maintained with a maintainer interested in taking a lead helping
>>   us integrate it into Emacs.
>>
>>
>>         Stefan
> 
> I've now tested "elunit" and converted to "ert". So far I prefer "ert"
> because its available in ELPA, and the report looks like the ones I'm
> used to get with JUnit or something similar.
> 
> OTOH none of the packages reported exactly where in my code my errors
> where. "ert" was slightly better tough.
> 
> Just my €0.02, to get the thread going again. Again, to quote Stefan, we
> should just pick something and run with it. I'm sure Emacs will
> improve from it.

I was very busy and missed this whole thread.  Sorry.  I assume no 
conclusion has been reached?

It seems that ert has a few supporters already.  I had a look at the 
other frameworks, and while things like mocking support and perhaps 
elk-test's fringe markers could be nice to add on top, I am fairly 
confident that ert has the most useful core, by which I mean basic 
functionality like defining and running tests and presenting failures in 
a (somewhat) comprehensible way.

For example, it has a `should' macro to express assertions that 
generalizes the `assert-eq', `assert-equal' etc. macros in elk-test, and 
installs a debugger hook rather than catching errors with condition-case 
so that it can capture backtraces of errors in tests.  Features like 
these add a little bit of complexity to ert's code, but they simplify 
testing and debugging other code, so they are worthwhile.

Unlike some other frameworks, ert does not dictate any particular style 
of testing; its deftest is very similar to defun, and tests can be 
stateful or functional as needed.  It was designed for Emacs Lisp and 
fits nicely into it rather than imitating frameworks from other 
environments.

It does not yet include utilities for testing Emacs state such as window 
layouts and buffer contents with text properties, or user input, but 
such utilities are orthogonal to the basic test execution framework and 
can be added incrementally.  I think Lennart has some of this already in 
his fork.

I don't know if the code qualifies as well-maintained; I haven't touched 
it in a while, since no bugs have been reported, but I haven't been very 
good at accepting patches to add features, either.  (Sorry Yamato-san, 
I'll look at your patch today.)  On the other hand, the other frameworks 
have been dormant for long periods as well.


I have some spare time now and can offer to work on integrating ert into 
Emacs; any suggestions where to start?  Where does a file like ert.el 
belong?  Where should the corresponding tests go?  Should I try to 
change makefiles to let tests run automatically during build?

My employer has a blanket copyright assignment.  Please contact me 
off-list if this needs further discussion.

Christian.