Re: testing framework and package.el

[Sebastian Rose <sebastian_rose@gmx.de>]

Christian Ohler <ohler+emacs@fastmail.net> writes:
>  On 4/10/10 6:22, Sebastian Rose wrote:
> I suspect it would be easier if the convention was that a file /some/path/foo.el
> had its tests in /some/path/test/foo.el.  The convention of having a top-level
> src/ and test/ split makes sense in Java because of its package file layout and
> visibility rules, but Emacs Lisp has no such constraints.  Has anyone else
> thought about the pros and cons of these different conventions?  Is there a deep
> reason why Emacs currently has test/ as a separate top-level directory?


This would place many "test" directoies inside the source tree.  They
might get in the way of builds.

Emacs conventions are indeed naming conventions:  Start function names
whith the name of the "package", name the file like it, (provide
'PACKAGE) etc.


> I just rewrote the test buffer functionality.  The design is now more orthogonal
> and flexible, but reading input from files will require an explicit call to
> insert-file-contents' or similar.

That's not a big hurdle I guess.


> I see.  Comparing directory trees may be important to org-mode, but overall, it
> seems like a feature that would be used by the tests of only very few packages.
> I think you're better off writing some utility functions on top of ERT, as you
> have already started doing for other things.  Not every test-related utility
> function belongs in ERT itself; it's fine to build stuff on top.


OK.


> The `ert-info' feature that I just implemented allows you to add custom context
> information that ERT will display on test failures.  It's not documented in the
> manual yet, but it makes it easier to build the kind of test utilities that you
> may want, so please check it out.
>
>
>>> One limitations of my approach is that we really need some kind of
>>> "with-test-buffer" rather than "with-temp-buffer" that keeps the buffer around
>>> for inspection if the test fails, but this is item 2.) on your list already.
>>> Another limitation is that the explainer for `equal' is not as pretty as ediff,
>>> but OTOH, as soon as you want to compare text properties, ediff can't help
>>> anymore (AFAIK), while there is an explainer for
>>> ert-equal-including-properties'.
>> Also, I believe ediff is too interactive for a simple test run (diff's
>> exit status is 0 if no difference is found - don't know yet if and to
>> how ediff could be used non-interactively).
> I have a rough idea now how to extend ERT's "explanation" feature to allow
> interactive invocation of ediff to visualize differences; this seems like a good
> way to solve this.  Tests would simply use (should (equal string1 string2)), and
> the explainer for `equal' would take care of the rest, putting a button in the
> results buffer that allows the user to invoke ediff.  string1 and string2 could
> be the result of calling `buffer-string' on different buffers, of course.


+1



> OK, that makes sense.  It is important to write actual tests, though; ideas for
> improvement that come from experience with real-world problems carry much more
> weight than thought experiments.
>
> Christian.


Yes, true.



  Sebastian