unit test framework

unit test framework

[Masatake YAMATO]

Hi,

Do you have a plan to include an elisp library for unit
testing in Emacs-24?

I've tried some and I'm using ert.el written by Christian M. Ohler.

     http://github.com/masatake/ert/blob/master/ert.el

Masatake YAMATO

Re: unit test framework

[Lennart Borgman]

On Thu, Mar 18, 2010 at 8:39 AM, Masatake YAMATO <yamato@redhat.com> wrote:
> Hi,
>
> Do you have a plan to include an elisp library for unit
> testing in Emacs-24?
>
> I've tried some and I'm using ert.el written by Christian M. Ohler.
>
>     http://github.com/masatake/ert/blob/master/ert.el


I am using that in nXhtml with some additions to test fontifications
and commands.

Re: unit test framework

[Leo]

On 2010-03-18 09:31 +0000, Lennart Borgman wrote:
> On Thu, Mar 18, 2010 at 8:39 AM, Masatake YAMATO <yamato@redhat.com> wrote:
>> Hi,
>>
>> Do you have a plan to include an elisp library for unit
>> testing in Emacs-24?
>>
>> I've tried some and I'm using ert.el written by Christian M. Ohler.
>>
>>     http://github.com/masatake/ert/blob/master/ert.el
>
>
> I am using that in nXhtml with some additions to test fontifications
> and commands.

I have been thinking about using a unit test framework even for my own
.emacs which has grown to a few thousand lines. But there are so many of
them available and I don't know which one is good. Could you share why
you use ert.el? Thanks.

Leo

Re: unit test framework

[Lennart Borgman]

On Thu, Mar 18, 2010 at 11:07 AM, Leo <sdl.web@gmail.com> wrote:
> On 2010-03-18 09:31 +0000, Lennart Borgman wrote:
>> On Thu, Mar 18, 2010 at 8:39 AM, Masatake YAMATO <yamato@redhat.com> wrote:
>>> Hi,
>>>
>>> Do you have a plan to include an elisp library for unit
>>> testing in Emacs-24?
>>>
>>> I've tried some and I'm using ert.el written by Christian M. Ohler.
>>>
>>>     http://github.com/masatake/ert/blob/master/ert.el
>>
>>
>> I am using that in nXhtml with some additions to test fontifications
>> and commands.
>
> I have been thinking about using a unit test framework even for my own
> .emacs which has grown to a few thousand lines. But there are so many of
> them available and I don't know which one is good. Could you share why
> you use ert.el? Thanks.

At the time I looked at it ert seemed to be the best. I do not
remember why now and maybe it is not so important since there are
alternatives now. I will not be switching now because the
modifications I have made to ert is really what I need. (If however
another unit framework gets included in Emacs I will try to switch.)

I think those who have been writing new frameworks should be those
that answers why they have done so instead of trying to make ert
better fit their needs.

Re: unit test framework

[Masatake YAMATO]

> On 2010-03-18 09:31 +0000, Lennart Borgman wrote:
>> On Thu, Mar 18, 2010 at 8:39 AM, Masatake YAMATO <yamato@redhat.com> wrote:
>>> Hi,
>>>
>>> Do you have a plan to include an elisp library for unit
>>> testing in Emacs-24?
>>>
>>> I've tried some and I'm using ert.el written by Christian M. Ohler.
>>>
>>>     http://github.com/masatake/ert/blob/master/ert.el
>>
>>
>> I am using that in nXhtml with some additions to test fontifications
>> and commands.
> 
> I have been thinking about using a unit test framework even for my own
> .emacs which has grown to a few thousand lines. But there are so many of
> them available and I don't know which one is good. Could you share why
> you use ert.el? Thanks.

I'm using it because it supports a use case, testing an elisp with "emacs --batch".
The emacs process run for unit testing can tell the test result via exit status;
0 is for successful, 1 is for failure. With ert.el it is possible to integrate 
running unit test to "make check".

If you are interested in, see the new docuemnt for ``ert-run-tests-batch'' in

   http://github.com/masatake/ert/commit/5f1bc45c5c54e1f95e9e8ba787f3f66b33ddc583

Masatake YAMATO

Re: unit test framework

[Stefan Monnier]

> Do you have a plan to include an elisp library for unit
> testing in Emacs-24?

It's not in the current plan, but I'm in favor of adding such
a thing, yes.  I simply don't have the time and energy to delve much
into it.

> I've tried some and I'm using ert.el written by Christian M. Ohler.
>      http://github.com/masatake/ert/blob/master/ert.el

If other people can chime in on the various options and form
a consensus, that would be wonderful.


        Stefan


PS: One thing for which I'd like to be able to use such a test framework, is
to link tests<->bug numbers<->commits.

Re: unit test framework

[Lennart Borgman]

On Thu, Mar 18, 2010 at 3:22 PM, Stefan Monnier
<monnier@iro.umontreal.ca> wrote:
>> Do you have a plan to include an elisp library for unit
>> testing in Emacs-24?
>
> It's not in the current plan, but I'm in favor of adding such
> a thing, yes.  I simply don't have the time and energy to delve much
> into it.
>
>> I've tried some and I'm using ert.el written by Christian M. Ohler.
>>      http://github.com/masatake/ert/blob/master/ert.el
>
> If other people can chime in on the various options and form
> a consensus, that would be wonderful.


I asked long ago for a comparision on EmacsWiki:

  http://www.emacswiki.org/cgi-bin/wiki/UnitTesting

However there have not been any interest in that. If have cc:ed the
unit test framework authors, maybe they want to say something.

I am not sure which of them have signed papers:

  ert: Christian Ohler
  elunit: Phil Hagelberg
  elk-test: Nikolaj Schumacher
  etest: Phil Jackson
  el-mock: rubikitch


>        Stefan
>
>
> PS: One thing for which I'd like to be able to use such a test framework, is
> to link tests<->bug numbers<->commits.
>
>
>

Re: unit test framework

[rubikitch]

From: Lennart Borgman <lennart.borgman@gmail.com>
Subject: Re: unit test framework
Date: Thu, 18 Mar 2010 15:56:51 +0100

Hi,

> I asked long ago for a comparision on EmacsWiki:
> 
>   http://www.emacswiki.org/cgi-bin/wiki/UnitTesting
> 
> However there have not been any interest in that. If have cc:ed the
> unit test framework authors, maybe they want to say something.
> 
> I am not sure which of them have signed papers:
> 
>   ert: Christian Ohler
>   elunit: Phil Hagelberg
>   elk-test: Nikolaj Schumacher
>   etest: Phil Jackson
>   el-mock: rubikitch

I am the author of el-expectations and el-mock.

el-expectations is focused on simpleness and readability.
No assert-* functions, no setup and teardown functions.
It is the same philosophy as expectations in Ruby.
http://expectations.rubyforge.org/files/README.html

el-mock is a mock framework.

Cheers,
--
rubikitch

Re: unit test framework

[Phil Hagelberg]

On Thu, Mar 18, 2010 at 7:56 AM, Lennart Borgman
<lennart.borgman@gmail.com> wrote:
> I asked long ago for a comparision on EmacsWiki:
>
>  http://www.emacswiki.org/cgi-bin/wiki/UnitTesting
>
> However there have not been any interest in that. If have cc:ed the
> unit test framework authors, maybe they want to say something.
>
> I am not sure which of them have signed papers:
>
>  ert: Christian Ohler
>  elunit: Phil Hagelberg
>  elk-test: Nikolaj Schumacher
>  etest: Phil Jackson
>  el-mock: rubikitch

elunit is deprecated in favour of ert, so it should not be considered
for inclusion. When I was investigating the options (~2 years ago),
ert was the clear winner, though I don't remember the details clearly.
Mostly it had to do with having the clearest failure reports. Beyond
that there were not a lot of differences; unit testing is a pretty
straightforward task.

Things may have changed since then.

-Phil

Re: unit test framework

[Nikolaj Schumacher]

On 2010-03-18 15:56 , Lennart Borgman wrote:

 >    elk-test: Nikolaj Schumacher

elk-test might look more complicated than it is.  Most of the code is 
for the user interface.  The testing framework itself is just a thin 
wrapper around Emacs' error signaling with better readability.

While
   (assert (equal "yy" foo))
causes
   (cl-assertion-failed (equal "yy" foo))
the macro
   (assert-equal "yy" foo)
will cause the more readable:
   (error "assert-equal for <foo> failed: expected <yy>, was <xx>")

`assert-that' can take care of the formatting for custom tests, and 
`assert-error' fails when a specific error isn't thrown.

The nice thing about these macros is that they work in any evaluation 
context.  I'd vote to include such macros at the least.

The
   (deftest "test name" ...
macro just provides an name anchor for running individual tests.  And 
the test runner will just catch errors and display them as text, 
highlight where the error occurred, and mark the buffer as "red".  This 
GUI is more designed for test-driven development than automatic 
regression tests, but the macros would work fine for that, I suppose.


kind regards,
Nikolaj Schumacher

Re: unit test framework

[joakim]

Lennart Borgman <lennart.borgman@gmail.com> writes:

> On Thu, Mar 18, 2010 at 3:22 PM, Stefan Monnier
> <monnier@iro.umontreal.ca> wrote:
>>> Do you have a plan to include an elisp library for unit
>>> testing in Emacs-24?
>>
>> It's not in the current plan, but I'm in favor of adding such
>> a thing, yes.  I simply don't have the time and energy to delve much
>> into it.
>>
>>> I've tried some and I'm using ert.el written by Christian M. Ohler.
>>>      http://github.com/masatake/ert/blob/master/ert.el>
>> If other people can chime in on the various options and form
>> a consensus, that would be wonderful.
>
>
> I asked long ago for a comparision on EmacsWiki:
>
>   http://www.emacswiki.org/cgi-bin/wiki/UnitTesting
>
> However there have not been any interest in that. If have cc:ed the
> unit test framework authors, maybe they want to say something.
>
> I am not sure which of them have signed papers:
>
>   ert: Christian Ohler
>   elunit: Phil Hagelberg
>   elk-test: Nikolaj Schumacher
>   etest: Phil Jackson
>   el-mock: rubikitch
>
>
>>        Stefan
>>
>>
>> PS: One thing for which I'd like to be able to use such a test framework, is
>> to link tests<->bug numbers<->commits.

I write try to write unit tests for my elisp, and it would be much
easier if there was a canonical unit test framework in Emacs.

I have written some tests for CEDET, which doesnt really use a
framework, and should be included in Emacs(I havent looked at the merged
version yet). I have also written some tests in elunit for other
packages.

I'm currently writing some code that would benefit from simulation of
keyboard input, in particular code that uses read, read-number,
completing-read etc. Some of the contenders listed above surely does
this, right?

Anyway, maybe we should make a list of requirements for a unit test
package for inclusion. Heres my take:

- possibility to generate unit tests reports in test runs during "make"

- at least rudimentary user input simulation, to test read, read-number,
completing-read etc.

- some way of isolating test runs from normal operations

- good error reporting, so that you dont have to spend hours in edebug needlessly

- convenient way of defining and running tests, in group or one-by-one



-- 
Joakim Verona

Re: unit test framework

[Lennart Borgman]

On Tue, Mar 23, 2010 at 10:02 AM,  <joakim@verona.se> wrote:
>
> I write try to write unit tests for my elisp, and it would be much
> easier if there was a canonical unit test framework in Emacs.
>
> I have written some tests for CEDET, which doesnt really use a
> framework, and should be included in Emacs(I havent looked at the merged
> version yet). I have also written some tests in elunit for other
> packages.
>
> I'm currently writing some code that would benefit from simulation of
> keyboard input, in particular code that uses read, read-number,
> completing-read etc. Some of the contenders listed above surely does
> this, right?
>
> Anyway, maybe we should make a list of requirements for a unit test
> package for inclusion. Heres my take:
>
> - possibility to generate unit tests reports in test runs during "make"
>
> - at least rudimentary user input simulation, to test read, read-number,
> completing-read etc.


I have written some extensions to ert.el that allows running commands
in the test function (ie calling relevant hooks etc).


> - some way of isolating test runs from normal operations


I have some small things that starts a new Emacs instance for the tests.


> - good error reporting, so that you dont have to spend hours in edebug needlessly


ert.el have good interactive reporting (that needs some usability
enhancements, like menu entries), but not so much for saving to file
(or maybe I have just missed that part).


> - convenient way of defining and running tests, in group or one-by-one


ert.el does it by regexp matching the names of the test functions.

Re: unit test framework

[Masatake YAMATO]

> On Tue, Mar 23, 2010 at 10:02 AM,  <joakim@verona.se> wrote:
>>
>> I write try to write unit tests for my elisp, and it would be much
>> easier if there was a canonical unit test framework in Emacs.
>>
>> I have written some tests for CEDET, which doesnt really use a
>> framework, and should be included in Emacs(I havent looked at the merged
>> version yet). I have also written some tests in elunit for other
>> packages.
>>
>> I'm currently writing some code that would benefit from simulation of
>> keyboard input, in particular code that uses read, read-number,
>> completing-read etc. Some of the contenders listed above surely does
>> this, right?
>>
>> Anyway, maybe we should make a list of requirements for a unit test
>> package for inclusion. Heres my take:
>>
>> - possibility to generate unit tests reports in test runs during "make"

I've extended ert.el for the purpose.

     http://github.com/masatake/ert/commit/5f1bc45c5c54e1f95e9e8ba787f3f66b33ddc583

The patch was improved based on the comment from the original author, 
Christian M. Ohler. I asked him to merge my patch to his source tree, but I have got 
no response yet.

>> - at least rudimentary user input simulation, to test read, read-number,
>> completing-read etc.
> 
> 
> I have written some extensions to ert.el that allows running commands
> in the test function (ie calling relevant hooks etc).
> 
> 
>> - some way of isolating test runs from normal operations
> 
> 
> I have some small things that starts a new Emacs instance for the tests.
>
> 
> 
>> - good error reporting, so that you dont have to spend hours in edebug needlessly
> 
> 
> ert.el have good interactive reporting (that needs some usability
> enhancements, like menu entries), but not so much for saving to file
> (or maybe I have just missed that part).
> 
> 
>> - convenient way of defining and running tests, in group or one-by-one
> 
> 
> ert.el does it by regexp matching the names of the test functions.
> 
> 

Masatake YAMATO

Re: unit test framework

[Stefan Monnier]

> I write try to write unit tests for my elisp, and it would be much
> easier if there was a canonical unit test framework in Emacs.

Yes, that's the intention.  We should pick one and run with it.
The choice doesn't have to be "the very best", but it should aim for:
- simple&clean code, easy to maintain&extend.
- well maintained with a maintainer interested in taking a lead helping
  us integrate it into Emacs.


        Stefan

Re: unit test framework

[joakim]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>> I write try to write unit tests for my elisp, and it would be much
>> easier if there was a canonical unit test framework in Emacs.
>
> Yes, that's the intention.  We should pick one and run with it.
> The choice doesn't have to be "the very best", but it should aim for:
> - simple&clean code, easy to maintain&extend.
> - well maintained with a maintainer interested in taking a lead helping
>   us integrate it into Emacs.
>
>
>         Stefan

I've now tested "elunit" and converted to "ert". So far I prefer "ert"
because its available in ELPA, and the report looks like the ones I'm
used to get with JUnit or something similar.

OTOH none of the packages reported exactly where in my code my errors
where. "ert" was slightly better tough.

Just my €0.02, to get the thread going again. Again, to quote Stefan, we
should just pick something and run with it. I'm sure Emacs will
improve from it.

-- 
Joakim Verona

Re: unit test framework

[Christian Ohler]

joakim@verona.se, 2010-04-13:

> Stefan Monnier <monnier@IRO.UMontreal.CA> writes:
> 
>>> I write try to write unit tests for my elisp, and it would be much
>>> easier if there was a canonical unit test framework in Emacs.
>> Yes, that's the intention.  We should pick one and run with it.
>> The choice doesn't have to be "the very best", but it should aim for:
>> - simple&clean code, easy to maintain&extend.
>> - well maintained with a maintainer interested in taking a lead helping
>>   us integrate it into Emacs.
>>
>>
>>         Stefan
> 
> I've now tested "elunit" and converted to "ert". So far I prefer "ert"
> because its available in ELPA, and the report looks like the ones I'm
> used to get with JUnit or something similar.
> 
> OTOH none of the packages reported exactly where in my code my errors
> where. "ert" was slightly better tough.
> 
> Just my €0.02, to get the thread going again. Again, to quote Stefan, we
> should just pick something and run with it. I'm sure Emacs will
> improve from it.

I was very busy and missed this whole thread.  Sorry.  I assume no 
conclusion has been reached?

It seems that ert has a few supporters already.  I had a look at the 
other frameworks, and while things like mocking support and perhaps 
elk-test's fringe markers could be nice to add on top, I am fairly 
confident that ert has the most useful core, by which I mean basic 
functionality like defining and running tests and presenting failures in 
a (somewhat) comprehensible way.

For example, it has a `should' macro to express assertions that 
generalizes the `assert-eq', `assert-equal' etc. macros in elk-test, and 
installs a debugger hook rather than catching errors with condition-case 
so that it can capture backtraces of errors in tests.  Features like 
these add a little bit of complexity to ert's code, but they simplify 
testing and debugging other code, so they are worthwhile.

Unlike some other frameworks, ert does not dictate any particular style 
of testing; its deftest is very similar to defun, and tests can be 
stateful or functional as needed.  It was designed for Emacs Lisp and 
fits nicely into it rather than imitating frameworks from other 
environments.

It does not yet include utilities for testing Emacs state such as window 
layouts and buffer contents with text properties, or user input, but 
such utilities are orthogonal to the basic test execution framework and 
can be added incrementally.  I think Lennart has some of this already in 
his fork.

I don't know if the code qualifies as well-maintained; I haven't touched 
it in a while, since no bugs have been reported, but I haven't been very 
good at accepting patches to add features, either.  (Sorry Yamato-san, 
I'll look at your patch today.)  On the other hand, the other frameworks 
have been dormant for long periods as well.


I have some spare time now and can offer to work on integrating ert into 
Emacs; any suggestions where to start?  Where does a file like ert.el 
belong?  Where should the corresponding tests go?  Should I try to 
change makefiles to let tests run automatically during build?

My employer has a blanket copyright assignment.  Please contact me 
off-list if this needs further discussion.

Christian.

Re: unit test framework

[Lennart Borgman]

On Sat, Jun 12, 2010 at 8:06 AM, Christian Ohler
<ohler+emacs@fastmail.net> wrote:
>
> It does not yet include utilities for testing Emacs state such as window
> layouts and buffer contents with text properties, or user input, but such
> utilities are orthogonal to the basic test execution framework and can be
> added incrementally.  I think Lennart has some of this already in his fork.


My additions are for running commands (i.e. running the necessary
hooks in hopefully correct order) and for testing of fontification.
There is also a little bit for adding a test input file for a test.
And a bit for keeping track of the test name.

But I suggest you review my code and tweak it to your liking. Please
contact me off list.


> I have some spare time now and can offer to work on integrating ert into
> Emacs; any suggestions where to start?  Where does a file like ert.el
> belong?  Where should the corresponding tests go?


There is already a sub-directory for tests, "emacs/test". To my
surprise the elisp files for test also is there. I suggest these
should be somewhere like "emacs/lisp/test" instead and that
"emacs/test" will be for data files etc instead.

An objection to this might be that the tests can't be run without the
data. Yes, but the code in the elisp files can serve as examples for
those writing tests.

Re: unit test framework

[Masatake YAMATO]

Christian,

> I don't know if the code qualifies as well-maintained; I haven't
> touched it in a while, since no bugs have been reported, but I haven't
> been very good at accepting patches to add features, either.  (Sorry
> Yamato-san, I'll look at your patch today.)  On the other hand, the
> other frameworks have been dormant for long periods as well.

I've confirmed that my patch is include in you tree. Thanks.
As I wrote repeatedly supporting batch mode is a big advantage of
ert.

Masatake YAMATO

Re: unit test framework

[Stefan Monnier]

> I have some spare time now and can offer to work on integrating ert into
> Emacs; any suggestions where to start?

Try integrating it, provide some example tests in emacs/test.

> Where does a file like ert.el belong?

In emacs/lisp/emacs-lisp, I think.

> Where should the corresponding tests go?

We've been using emacs/test for the (very) few tests we have, so that's
where they should go.

> Should I try to change makefiles to let tests run automatically
> during build?

I don't think so.  You can just try and make "cd test; make" do
something useful.  We can later on decide whether running tests shojuld
be performed as part of routine compilations.


        Stefan

Re: unit test framework

[Philip Jackson]

At Thu, 18 Mar 2010 15:56:51 +0100,
Lennart Borgman wrote:

>>> Do you have a plan to include an elisp library for unit
>>> testing in Emacs-24?

> I asked long ago for a comparision on EmacsWiki:
> 
>   http://www.emacswiki.org/cgi-bin/wiki/UnitTesting
> 
> However there have not been any interest in that. If have cc:ed the
> unit test framework authors, maybe they want to say something.
> 
> I am not sure which of them have signed papers:
> 
>   ert: Christian Ohler
>   elunit: Phil Hagelberg
>   elk-test: Nikolaj Schumacher
>   etest: Phil Jackson
>   el-mock: rubikitch

I wrote etest:

  http://www.shellarchive.co.uk/content/etest.html
  http://www.flickr.com/photos/phil-jackson/2702854411/

I've signed the papers, would be happy for its inclusion but have to
admit I've not done much of a comparison with the other tools.

Cheers,
Phil

Re: unit test framework

[Philip Jackson]

At Thu, 18 Mar 2010 10:22:42 -0400,
Stefan Monnier wrote:
> 
> > Do you have a plan to include an elisp library for unit
> > testing in Emacs-24?
> 
> It's not in the current plan, but I'm in favor of adding such
> a thing, yes.  I simply don't have the time and energy to delve much
> into it.
> 
> > I've tried some and I'm using ert.el written by Christian M. Ohler.
> >      http://github.com/masatake/ert/blob/master/ert.el
> 
> If other people can chime in on the various options and form
> a consensus, that would be wonderful.

It looks like ert has the most support. How about running with it?

Cheers,
Phil

Re: unit test framework

[Christian Ohler]

Philip Jackson, 2010-06-28:

> At Thu, 18 Mar 2010 10:22:42 -0400,
> Stefan Monnier wrote:
>>> Do you have a plan to include an elisp library for unit
>>> testing in Emacs-24?
>> It's not in the current plan, but I'm in favor of adding such
>> a thing, yes.  I simply don't have the time and energy to delve much
>> into it.
>>
>>> I've tried some and I'm using ert.el written by Christian M. Ohler.
>>>      http://github.com/masatake/ert/blob/master/ert.el
>> If other people can chime in on the various options and form
>> a consensus, that would be wonderful.
> 
> It looks like ert has the most support. How about running with it?

I think that is pretty much the plan.  I volunteered to do the necessary 
work and have been cleaning up the code.

Unfortunately, I still don't have a copyright assignment; assign at gnu 
dot org has yet to respond to the request that I sent and re-sent 
several times over the last few weeks.  Does anyone have a way to find 
out what's going on there?

Christian.

Re: unit test framework

[Glenn Morris]

Christian Ohler wrote:

> Unfortunately, I still don't have a copyright assignment; assign at
> gnu dot org has yet to respond to the request that I sent and re-sent
> several times over the last few weeks.  Does anyone have a way to find
> out what's going on there?

What are you trying to do? If you are just trying to get started, I
can send you the first (electronic) form to fill in. If you have
already done that and are waiting to get a paper form, I am afraid I
cannot help. Sometimes assign@gnu does seem to get busy.

Re: unit test framework

[Christian Ohler]

Glenn Morris, 2010-07-10:

> What are you trying to do? If you are just trying to get started, I
> can send you the first (electronic) form to fill in. If you have
> already done that and are waiting to get a paper form, I am afraid I
> cannot help. Sometimes assign@gnu does seem to get busy.

Thanks, I have the electronic form.  The paper form is what I'm waiting 
for, sorry for not being clear.  I guess I'll just keep waiting.

Christian.