testing framework and package.el

testing framework and package.el

[Sebastian Rose]

Hi,


is there a chance to get a testing framework on elpa.gnu.org on the
short run?

Ert [1] is used in nxhtml, simple to use and documented.
Emt [2] is used in Org for one file, but comes without any
documentation and seems complicated (lot's of files and directories).
There are some more on [3].


Are there any other frameworks used for current emacs development?


Having one such framework on elpa.gnu.org could be a step to somewhat
standardized coding and serve as recommendation.

I just started to write a little test for an Org-mode function I
modified and would like to add more tests in the future.  In some cases
tests are even better than explanations or docs.  For now, I decided to
use ert [1] because of it's simplicity and documentation.  But I'd
rather switch to something widely used.




Sebastian



[1] http://www.emacswiki.org/emacs/ErtTestLibrary
[2] http://www.panix.com/~tehom/my-code/
[3] http://www.emacswiki.org/emacs/UnitTesting

Re: testing framework and package.el

[Stefan Monnier]

> is there a chance to get a testing framework on elpa.gnu.org on the
> short run?
> Ert [1] is used in nxhtml, simple to use and documented.

IIRC we've agreed to try and install ERT, either on elpa or in Emacs
itself, and then try and move the few tests we already have
scattered about to use ERT.  But I haven't heard much about it recently.


        Stefan

Re: testing framework and package.el

[Sebastian Rose]

Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> is there a chance to get a testing framework on elpa.gnu.org on the
>> short run?
>> Ert [1] is used in nxhtml, simple to use and documented.
>
> IIRC we've agreed to try and install ERT, either on elpa or in Emacs
> itself, and then try and move the few tests we already have
> scattered about to use ERT.  But I haven't heard much about it recently.
>
>
>         Stefan


I'll go with ERT then.  I did already ;)

The most important fact with tests is, that they are gathered and
written down somewhere.  One could switch the framework at any time.



  Sebastian

Re: testing framework and package.el

[Sebastian Rose]

Stefan Monnier <monnier@iro.umontreal.ca> writes:
> IIRC we've agreed to try and install ERT, either on elpa or in Emacs
> itself, and then try and move the few tests we already have
> scattered about to use ERT.  But I haven't heard much about it recently.

Would be nice to merge it into the core.  How would one distribute the
tests otherwise?





  Sebastian

Re: testing framework and package.el

[Christian Ohler]

On 28/09/10 8:57, Stefan Monnier wrote:
>> is there a chance to get a testing framework on elpa.gnu.org on the
>> short run?
>> Ert [1] is used in nxhtml, simple to use and documented.
>
> IIRC we've agreed to try and install ERT, either on elpa or in Emacs
> itself, and then try and move the few tests we already have
> scattered about to use ERT.  But I haven't heard much about it recently.

Incidentally, I finished a major overhaul of ERT's manual a few days 
ago.  Over the next few weeks, I should be able to find some time to put 
together a patch to integrate it into Emacs.  If anyone is interested, 
you can already review the code at http://github.com/ohler/ert and send 
me comments to speed up the integration process.

I think it makes more sense to have ERT in Emacs proper rather than as 
an ELPA package.  This way, tests for Emacs itself can use ERT.  Ulf 
Jasper's tests in the test/ directory would be the first candidates.

Christian.

Re: testing framework and package.el

[Sebastian Rose]

Christian Ohler <ohler+emacs@fastmail.net> writes:
> On 28/09/10 8:57, Stefan Monnier wrote:
>>> is there a chance to get a testing framework on elpa.gnu.org on the
>>> short run?
>>> Ert [1] is used in nxhtml, simple to use and documented.
>>
>> IIRC we've agreed to try and install ERT, either on elpa or in Emacs
>> itself, and then try and move the few tests we already have
>> scattered about to use ERT.  But I haven't heard much about it recently.
>
> Incidentally, I finished a major overhaul of ERT's manual a few days ago.  Over
> the next few weeks, I should be able to find some time to put together a patch
> to integrate it into Emacs.  If anyone is interested, you can already review the
> code at http://github.com/ohler/ert and send me comments to speed up the
> integration process.
>
> I think it makes more sense to have ERT in Emacs proper rather than as an ELPA
> package.  This way, tests for Emacs itself can use ERT.  Ulf Jasper's tests in
> the test/ directory would be the first candidates.
>
> Christian.


Yes, that would make sense.


Christian, I have a few ideas/feature requests.


1.)  Make testing a matter of one key-stroke

To ease testing, I've written two little commands, as you can see on
http://github.com/SebastianRose/org-test

  `org-test-test-current-defun' and
  `org-test-test-buffer-file'

The idea is to search the directory structure from the current source
file upwards for a directory named "tests/" if it exists.  Else ask the
user.  Similar to what `add-change-log-entry' does.

Below that directory, a tree like the source tree exists:

project
   +-- lisp/
   |     +-- a.el
   |     `-- b/
   |         +-- b.el
   |
   `-- tests/
         +-- a.el/
         |     +-- tests.el
         |     `-- a-defun.el
         `-- b/
             +-- b.el/
                   +-- tests.el
                   `-- b-defun.el

If this setup exists, when editing defun-x in lisp/a.el,
`M-x org-test-test-current-defun' will load tests/a.el/defun-x.el
(fallback: tests.el there) and execute all tests with selector
"^a-defun".
`M-x org-test-test-buffer-file' in that same source file will load all
*.el files in tests/a.el/ and execute all ERT tests for selector "^a".

This would   a) recommend a certain structure for testing (what ever the
structure will be) and   b) make testing a key-stroke.



2.)  Create temp buffers for output

One thing I miss is a way to create output buffers automagically (I
didn't look deeper into the sources yet).

I know Lennart Borgman has written something for nXhtml.  It's called
ert2.el and part of the nXhtml distribution (but I did not comprehend
what's going on there yet - my elisp is badbadbad :).  Maybe Lennart
will help here or even care for that addition?


3.)  Compare output to control files

ediff the contents of those output buffers with control files.  We need
this feature to test org's exports.  No one will find the time to sit
there and search hundreds of output files for errors "by hand" (once
tests exist).



Thanks for all your efforts

   Sebastian

Re: testing framework and package.el

[Christian Ohler]

  On 2/10/10 13:53, Sebastian Rose wrote:
> 1.) Make testing a matter of one key-stroke
> To ease testing, I've written two little commands, as you can see on
> http://github.com/SebastianRose/org-test
>
>    `org-test-test-current-defun' and
>    `org-test-test-buffer-file'
>
> The idea is to search the directory structure from the current source
> file upwards for a directory named "tests/" if it exists.  Else ask the
> user.  Similar to what `add-change-log-entry' does.
>
> Below that directory, a tree like the source tree exists:
>
> project
>     +-- lisp/
>     |     +-- a.el
>     |     `-- b/
>     |         +-- b.el
>     |
>     `-- tests/
>           +-- a.el/
>           |     +-- tests.el
>           |     `-- a-defun.el
>           `-- b/
>               +-- b.el/
>                     +-- tests.el
>                     `-- b-defun.el
>
> If this setup exists, when editing defun-x in lisp/a.el,
> `M-x org-test-test-current-defun' will load tests/a.el/defun-x.el
> (fallback: tests.el there) and execute all tests with selector
> "^a-defun".
> `M-x org-test-test-buffer-file' in that same source file will load all
> *.el files in tests/a.el/ and execute all ERT tests for selector "^a".
>
> This would   a) recommend a certain structure for testing (what ever the
> structure will be) and   b) make testing a key-stroke.
This structure looks like it is optimized for many test files for each 
file of non-test code.  Right now, Emacs has 1735 .el files, and 35 
files total in the tests/ directory.  This ratio is not likely to invert 
any time soon.  The structure that you describe could work, but isn't it 
overkill?


> 2.)  Create temp buffers for output
>
> One thing I miss is a way to create output buffers automagically (I
> didn't look deeper into the sources yet).
>
> I know Lennart Borgman has written something for nXhtml.  It's called
> ert2.el and part of the nXhtml distribution (but I did not comprehend
> what's going on there yet - my elisp is badbadbad :).  Maybe Lennart
> will help here or even care for that addition?
Lennart's code is already integrated in ERT, obsoleting ert2.  I 
remember that I wasn't quite happy with his design, but didn't have time 
to fix it, and since it solved an important problem, I included it as an 
experimental addition in ert-exp.el.


> 3.)  Compare output to control files
>
> ediff the contents of those output buffers with control files.  We need
> this feature to test org's exports.  No one will find the time to sit
> there and search hundreds of output files for errors "by hand" (once
> tests exist).
What do you mean by "sitting there and searching hundreds of output 
files for errors"?  Do you have an example of tests where ERT's error 
reporting is too unspecific to be useful?

I remember writing some tests roughly like this:

(ert-deftest foo ()
   (with-temp-buffer
     (insert "initial buffer contents")
     (... invoke code under test ...)
     (should (equal (ert-filter-string (buffer-string) ...)
                    "expected final buffer contents"))))

Are you sure that you want the initial/expected buffer contents in their 
own files?  Such a split between the inputs and the invocation of the 
code under test seems cumbersome.  I guess if your inputs/expected 
outputs are complex enough, moving them out into files can make sense.  
Can you point me to some examples where you think the data is too 
complex to be in-line?

One limitations of my approach is that we really need some kind of 
"with-test-buffer" rather than "with-temp-buffer" that keeps the buffer 
around for inspection if the test fails, but this is item 2.) on your 
list already.  Another limitation is that the explainer for `equal' is 
not as pretty as ediff, but OTOH, as soon as you want to compare text 
properties, ediff can't help anymore (AFAIK), while there is an 
explainer for `ert-equal-including-properties'.

Text properties (e.g., font lock properties) are important to many kinds 
of tests.  Both ediff and using files are problematic in such cases 
because they don't support text properties.  Do text properties never 
come up in your context?

Christian.

Re: testing framework and package.el

[Sebastian Rose]

Christian Ohler <ohler+emacs@fastmail.net> writes:
>  On 2/10/10 13:53, Sebastian Rose wrote:
>> Below that directory, a tree like the source tree exists:
>>
>> project
>>     +-- lisp/
>>     |     +-- a.el
>>     |     `-- b/
>>     |         +-- b.el
>>     |
>>     `-- tests/
>>           +-- a.el/
>>           |     +-- tests.el
>>           |     `-- a-defun.el
>>           `-- b/
>>               +-- b.el/
>>                     +-- tests.el
>>                     `-- b-defun.el
>>
>> If this setup exists, when editing defun-x in lisp/a.el,
>> `M-x org-test-test-current-defun' will load tests/a.el/defun-x.el
>> (fallback: tests.el there) and execute all tests with selector
>> "^a-defun".
>> `M-x org-test-test-buffer-file' in that same source file will load all
>> *.el files in tests/a.el/ and execute all ERT tests for selector "^a".
>>
>> This would   a) recommend a certain structure for testing (what ever the
>> structure will be) and   b) make testing a key-stroke.
> This structure looks like it is optimized for many test files for each file of
> non-test code.  Right now, Emacs has 1735 .el files, and 35 files total in the
> tests/ directory.  This ratio is not likely to invert any time soon.  The
> structure that you describe could work, but isn't it overkill?


Might be overkill, yes.  All I wrote here was just after the first
glance at ERT and few experiments.  You're not the only one who says so
:)
I'll drop that probably.  A structure like that could be enforced later
if needed.  We will start with a very small project.  I want to make it
easy to jump to the test-file for a given source-file and execute tests
for current defun automatically.  This could be done based on such a
structure (i.e. conventions).

I guess reading more of ERT's code I'll find other ways to find the file
a certain test (or tests matching a selector) is defined.


>> 2.)  Create temp buffers for output
>>
>> One thing I miss is a way to create output buffers automagically (I
>> didn't look deeper into the sources yet).
>>
>> I know Lennart Borgman has written something for nXhtml.  It's called
>> ert2.el and part of the nXhtml distribution (but I did not comprehend
>> what's going on there yet - my elisp is badbadbad :).  Maybe Lennart
>> will help here or even care for that addition?
> Lennart's code is already integrated in ERT, obsoleting ert2.  I remember that I
> wasn't quite happy with his design, but didn't have time to fix it, and since it
> solved an important problem, I included it as an experimental addition in
> ert-exp.el.

OK, good.  Found that, thanks.  From skimming over that file I guess
that's what I was looking for and maybe even a bit more.


>> 3.)  Compare output to control files
>>
>> ediff the contents of those output buffers with control files.  We need
>> this feature to test org's exports.  No one will find the time to sit
>> there and search hundreds of output files for errors "by hand" (once
>> tests exist).
> What do you mean by "sitting there and searching hundreds of output files for
> errors"?  Do you have an example of tests where ERT's error reporting is too
> unspecific to be useful?
>
> I remember writing some tests roughly like this:
>
> (ert-deftest foo ()
>   (with-temp-buffer
>     (insert "initial buffer contents")
>     (... invoke code under test ...)
>     (should (equal (ert-filter-string (buffer-string) ...)
>                    "expected final buffer contents"))))
>
> Are you sure that you want the initial/expected buffer contents in their own
> files?  Such a split between the inputs and the invocation of the code under
> test seems cumbersome.  I guess if your inputs/expected outputs are complex
> enough, moving them out into files can make sense.  Can you point me to some
> examples where you think the data is too complex to be in-line?


Org's exports to XHTML, LaTeX etc.  We could compare them to control
files.  The publishing mechanism might create quite big files and even
entire file/directory trees.  Currently we publish our stuff and detect
regressions when clicking a link that points nowhere or validating a
XHTML page (spot checking).



> One limitations of my approach is that we really need some kind of
> "with-test-buffer" rather than "with-temp-buffer" that keeps the buffer around
> for inspection if the test fails, but this is item 2.) on your list already.
> Another limitation is that the explainer for `equal' is not as pretty as ediff,
> but OTOH, as soon as you want to compare text properties, ediff can't help
> anymore (AFAIK), while there is an explainer for
> ert-equal-including-properties'.


Also, I believe ediff is too interactive for a simple test run (diff's
exit status is 0 if no difference is found - don't know yet if and to
how ediff could be used non-interactively).


> Text properties (e.g., font lock properties) are important to many kinds of
> tests.  Both ediff and using files are problematic in such cases because they
> don't support text properties.  Do text properties never come up in your
> context?


Some so some so I guess.  I didn't start to write actual tests yet.
This will be done by many people (I hope) and there will be many
different types of tests (I hope).  Some will involve text properties,
too.



Thank's for your comments!  I think I have enough to fill my spare time
this week :)


  Sebastian

Re: testing framework and package.el

[Masatake YAMATO]

Hi,

> On 28/09/10 8:57, Stefan Monnier wrote:
>>> is there a chance to get a testing framework on elpa.gnu.org on the
>>> short run?
>>> Ert [1] is used in nxhtml, simple to use and documented.
>>
>> IIRC we've agreed to try and install ERT, either on elpa or in Emacs
>> itself, and then try and move the few tests we already have
>> scattered about to use ERT.  But I haven't heard much about it
>> recently.
> 
> Incidentally, I finished a major overhaul of ERT's manual a few days
> ago.  Over the next few weeks, I should be able to find some time to
> put together a patch to integrate it into Emacs.  If anyone is
> interested, you can already review the code at
> http://github.com/ohler/ert and send me comments to speed up the
> integration process.

I read the manual and I've had three small comments.


1. When explaining about the name of test, the read may want
   to know how the name is used in ert. My understanding is
   it can be used to select the tests to run.

2. I would like to know the difference between `Failed' and `unexpected'.

3. I would like to see the example of `should-error'.

Following the patch tells the place where I got above comments.

diff --git a/ert.texinfo b/ert.texinfo
index 3543738..1f65426 100644
--- a/ert.texinfo
+++ b/ert.texinfo
@@ -125,7 +125,8 @@ better error reporting.  @xref{The @code{should} Macro}.
 The names of tests can be chosen arbitrarily --- they are in a
 namespace separate from functions and variables --- but should follow
 the usual Emacs Lisp convention of having a prefix that indicates
-which package they belong to.
+which package they belong to. The names can be used to select the
+tests to run. @xref{Test Selectors}.
 
 The empty parentheses @code{()} in the first line don't currently have
 any meaning and are reserved for future extension.  They also make
@@ -214,6 +215,7 @@ shows details about each test that had an unexpected result.  In the
 example above, there are two failures, both due to failed
 @code{should} forms.  @xref{Understanding Explanations}, for more
 details.
+@c ??? Difference between `Failed' and `unexpected'.
 
 In the ERT results buffer, @kbd{TAB} and @kbd{S-TAB} cycle between
 buttons.  Each name of a function or macro in this buffer is a button;
@@ -357,6 +359,8 @@ checks that the form called within it signals an error.  There is no
 @code{should-not-error} macro since tests that signal an error fail
 anyway, so @code{should-not-error} is effectively the default.
 
+@c An example of should-error
+
 @xref{Understanding Explanations}, for more details on what
 @code{should} reports.
 

Masatake YAMATO

Re: testing framework and package.el

[Christian Ohler]

  On 4/10/10 6:22, Sebastian Rose wrote:
> A structure like that could be enforced later
> if needed.  We will start with a very small project.  I want to make it
> easy to jump to the test-file for a given source-file and execute tests
> for current defun automatically.  This could be done based on such a
> structure (i.e. conventions).
>
> I guess reading more of ERT's code I'll find other ways to find the file
> a certain test (or tests matching a selector) is defined.
I just started experimenting with a directory layout where ERT's code is 
in lisp/emacs-lisp/*.el and the tests are in test/lisp/emacs-lisp/*.el.  
It is indeed very clumsy to jump back and forth between the tests and 
the code under test.  (Putting the tests in test/emacs-lisp/*.el would 
make no difference.)

I suspect it would be easier if the convention was that a file 
/some/path/foo.el had its tests in /some/path/test/foo.el.  The 
convention of having a top-level src/ and test/ split makes sense in 
Java because of its package file layout and visibility rules, but Emacs 
Lisp has no such constraints.  Has anyone else thought about the pros 
and cons of these different conventions?  Is there a deep reason why 
Emacs currently has test/ as a separate top-level directory?


>> Lennart's code is already integrated in ERT, obsoleting ert2.  I remember that I
>> wasn't quite happy with his design, but didn't have time to fix it, and since it
>> solved an important problem, I included it as an experimental addition in
>> ert-exp.el.
> OK, good.  Found that, thanks.  From skimming over that file I guess
> that's what I was looking for and maybe even a bit more.
I just rewrote the test buffer functionality.  The design is now more 
orthogonal and flexible, but reading input from files will require an 
explicit call to `insert-file-contents' or similar.


> Org's exports to XHTML, LaTeX etc. We could compare them to control
> files.  The publishing mechanism might create quite big files and even
> entire file/directory trees.  Currently we publish our stuff and detect
> regressions when clicking a link that points nowhere or validating a
> XHTML page (spot checking).
I see.  Comparing directory trees may be important to org-mode, but 
overall, it seems like a feature that would be used by the tests of only 
very few packages.  I think you're better off writing some utility 
functions on top of ERT, as you have already started doing for other 
things.  Not every test-related utility function belongs in ERT itself; 
it's fine to build stuff on top.

The `ert-info' feature that I just implemented allows you to add custom 
context information that ERT will display on test failures.  It's not 
documented in the manual yet, but it makes it easier to build the kind 
of test utilities that you may want, so please check it out.


>> One limitations of my approach is that we really need some kind of
>> "with-test-buffer" rather than "with-temp-buffer" that keeps the buffer around
>> for inspection if the test fails, but this is item 2.) on your list already.
>> Another limitation is that the explainer for `equal' is not as pretty as ediff,
>> but OTOH, as soon as you want to compare text properties, ediff can't help
>> anymore (AFAIK), while there is an explainer for
>> ert-equal-including-properties'.
> Also, I believe ediff is too interactive for a simple test run (diff's
> exit status is 0 if no difference is found - don't know yet if and to
> how ediff could be used non-interactively).
I have a rough idea now how to extend ERT's "explanation" feature to 
allow interactive invocation of ediff to visualize differences; this 
seems like a good way to solve this.  Tests would simply use (should 
(equal string1 string2)), and the explainer for `equal' would take care 
of the rest, putting a button in the results buffer that allows the user 
to invoke ediff.  string1 and string2 could be the result of calling 
`buffer-string' on different buffers, of course.


>> Text properties (e.g., font lock properties) are important to many kinds of
>> tests.  Both ediff and using files are problematic in such cases because they
>> don't support text properties.  Do text properties never come up in your
>> context?
> Some so some so I guess.  I didn't start to write actual tests yet.
> This will be done by many people (I hope) and there will be many
> different types of tests (I hope).  Some will involve text properties,
> too.
OK, that makes sense.  It is important to write actual tests, though; 
ideas for improvement that come from experience with real-world problems 
carry much more weight than thought experiments.

Christian.

Re: testing framework and package.el

[Chong Yidong]

Christian Ohler <ohler+emacs@fastmail.net> writes:

> I suspect it would be easier if the convention was that a file
> /some/path/foo.el had its tests in /some/path/test/foo.el.  The
> convention of having a top-level src/ and test/ split makes sense in
> Java because of its package file layout and visibility rules, but
> Emacs Lisp has no such constraints.  Has anyone else thought about the
> pros and cons of these different conventions?  Is there a deep reason
> why Emacs currently has test/ as a separate top-level directory?

Because the test files are not to be distributed with the tarball.

Re: testing framework and package.el

[Eric Schulte]

Christian Ohler <ohler+emacs@fastmail.net> writes:

>  On 4/10/10 6:22, Sebastian Rose wrote:
>> A structure like that could be enforced later
>> if needed.  We will start with a very small project.  I want to make it
>> easy to jump to the test-file for a given source-file and execute tests
>> for current defun automatically.  This could be done based on such a
>> structure (i.e. conventions).
>>
>> I guess reading more of ERT's code I'll find other ways to find the file
>> a certain test (or tests matching a selector) is defined.
> I just started experimenting with a directory layout where ERT's code
> is in lisp/emacs-lisp/*.el and the tests are in
> test/lisp/emacs-lisp/*.el.  It is indeed very clumsy to jump back and
> forth between the tests and the code under test.  (Putting the tests
> in test/emacs-lisp/*.el would make no difference.)
>
> I suspect it would be easier if the convention was that a file
> /some/path/foo.el had its tests in /some/path/test/foo.el.  The
> convention of having a top-level src/ and test/ split makes sense in
> Java because of its package file layout and visibility rules, but
> Emacs Lisp has no such constraints.  Has anyone else thought about the
> pros and cons of these different conventions?  Is there a deep reason
> why Emacs currently has test/ as a separate top-level directory?
>

I've been working with Sebastian on a branch of the Org-mode repository
which includes a test suite [1].  Included in this setup is some
functionality for jumping between e-lisp files and their related test
files.  This uses an old utility I wrote long ago [2] which allows the
definition of "jumping" functions defined using a simple schema.  In the
org-mode repository the tests and elisp are organized as

elisp file in

  /lisp/org-foo.el

related tests can be in either of the following

  /testing/lisp/test-org-foo.el
  /testing/lisp/org-foo.el/test.el
  /testing/lisp/org-foo.el/tests.el

the following is used to define our jumping function in this setup [3].
Maybe this would be helpful for navigating the Emacs tests as well.

Cheers -- Eric

Footnotes: 
[1]  http://repo.or.cz/w/org-mode.git/shortlog/refs/heads/combined-testing

[2]  http://github.com/eschulte/jump.el
       I was just learning elisp when I wrote this,
       so please don't judge me by the code :)

[3]  example jump schema
--8<---------------cut here---------------start------------->8---
;;; Navigation Functions
(defjump 'org-test-jump
  '(("lisp/\\1.el" . "testing/lisp/test-\\1.el")
    ("lisp/\\1.el" . "testing/lisp/\\1.el/test.*.el")
    ("contrib/lisp/\\1.el" . "testing/contrib/lisp/test-\\1.el")
    ("contrib/lisp/\\1.el" . "testing/contrib/lisp/\\1.el/test.*.el")
    ("testing/lisp/test-\\1.el" . "lisp/\\1.el")
    ("testing/lisp/\\1.el" . "lisp/\\1.el/test.*.el")
    ("testing/contrib/lisp/test-\\1.el" . "contrib/lisp/\\1.el")
    ("testing/contrib/lisp/test-\\1.el" . "contrib/lisp/\\1.el/test.*.el"))
  (concat org-base-dir "/")
  "Jump between org-mode files and their tests.")
--8<---------------cut here---------------end--------------->8---

Re: testing framework and package.el

[Sebastian Rose]

Christian Ohler <ohler+emacs@fastmail.net> writes:
>  On 4/10/10 6:22, Sebastian Rose wrote:
> I suspect it would be easier if the convention was that a file /some/path/foo.el
> had its tests in /some/path/test/foo.el.  The convention of having a top-level
> src/ and test/ split makes sense in Java because of its package file layout and
> visibility rules, but Emacs Lisp has no such constraints.  Has anyone else
> thought about the pros and cons of these different conventions?  Is there a deep
> reason why Emacs currently has test/ as a separate top-level directory?


This would place many "test" directoies inside the source tree.  They
might get in the way of builds.

Emacs conventions are indeed naming conventions:  Start function names
whith the name of the "package", name the file like it, (provide
'PACKAGE) etc.


> I just rewrote the test buffer functionality.  The design is now more orthogonal
> and flexible, but reading input from files will require an explicit call to
> insert-file-contents' or similar.

That's not a big hurdle I guess.


> I see.  Comparing directory trees may be important to org-mode, but overall, it
> seems like a feature that would be used by the tests of only very few packages.
> I think you're better off writing some utility functions on top of ERT, as you
> have already started doing for other things.  Not every test-related utility
> function belongs in ERT itself; it's fine to build stuff on top.


OK.


> The `ert-info' feature that I just implemented allows you to add custom context
> information that ERT will display on test failures.  It's not documented in the
> manual yet, but it makes it easier to build the kind of test utilities that you
> may want, so please check it out.
>
>
>>> One limitations of my approach is that we really need some kind of
>>> "with-test-buffer" rather than "with-temp-buffer" that keeps the buffer around
>>> for inspection if the test fails, but this is item 2.) on your list already.
>>> Another limitation is that the explainer for `equal' is not as pretty as ediff,
>>> but OTOH, as soon as you want to compare text properties, ediff can't help
>>> anymore (AFAIK), while there is an explainer for
>>> ert-equal-including-properties'.
>> Also, I believe ediff is too interactive for a simple test run (diff's
>> exit status is 0 if no difference is found - don't know yet if and to
>> how ediff could be used non-interactively).
> I have a rough idea now how to extend ERT's "explanation" feature to allow
> interactive invocation of ediff to visualize differences; this seems like a good
> way to solve this.  Tests would simply use (should (equal string1 string2)), and
> the explainer for `equal' would take care of the rest, putting a button in the
> results buffer that allows the user to invoke ediff.  string1 and string2 could
> be the result of calling `buffer-string' on different buffers, of course.


+1



> OK, that makes sense.  It is important to write actual tests, though; ideas for
> improvement that come from experience with real-world problems carry much more
> weight than thought experiments.
>
> Christian.


Yes, true.



  Sebastian

Re: testing framework and package.el

[Christian Ohler]

  On 5/10/10 2:18, Chong Yidong wrote:
> Christian Ohler<ohler+emacs@fastmail.net>  writes:
>> Is there a deep reason
>> why Emacs currently has test/ as a separate top-level directory?
> Because the test files are not to be distributed with the tarball.
Why not?  Isn't it true that whenever someone builds Emacs from source, 
it's possible that they are using a configuration (GCC version, 
configure flags, etc.) that hasn't been tested extensively, and it would 
make sense for them to run Emacs' self-tests before they waste their 
time installing an unstable Emacs?  This seems to apply especially to 
the pretest tarballs, but also to the regular distribution to some extent.

Christian.

Re: testing framework and package.el

[Lennart Borgman]

On Tue, Oct 5, 2010 at 3:17 AM, Christian Ohler
<ohler+emacs@fastmail.net> wrote:
>  On 5/10/10 2:18, Chong Yidong wrote:
>>
>> Christian Ohler<ohler+emacs@fastmail.net>  writes:
>>>
>>> Is there a deep reason
>>> why Emacs currently has test/ as a separate top-level directory?
>>
>> Because the test files are not to be distributed with the tarball.
>
> Why not?  Isn't it true that whenever someone builds Emacs from source, it's
> possible that they are using a configuration (GCC version, configure flags,
> etc.) that hasn't been tested extensively, and it would make sense for them
> to run Emacs' self-tests before they waste their time installing an unstable
> Emacs?  This seems to apply especially to the pretest tarballs, but also to
> the regular distribution to some extent.

I agree with Christian.

Re: testing framework and package.el

[Sebastian Rose]

Christian Ohler <ohler+emacs@fastmail.net> writes:
>  On 5/10/10 2:18, Chong Yidong wrote:
>> Christian Ohler<ohler+emacs@fastmail.net>  writes:
>>> Is there a deep reason
>>> why Emacs currently has test/ as a separate top-level directory?
>> Because the test files are not to be distributed with the tarball.
> Why not?  Isn't it true that whenever someone builds Emacs from source, it's
> possible that they are using a configuration (GCC version, configure flags,
> etc.) that hasn't been tested extensively, and it would make sense for them to
> run Emacs' self-tests before they waste their time installing an unstable Emacs?
> This seems to apply especially to the pretest tarballs, but also to the regular
> distribution to some extent.
>
> Christian.


Tests should be part of the repo, yes.

But for tarballs?

  Sebastian

Re: testing framework and package.el

[Christian Ohler]

  On 4/10/10 22:32, Masatake YAMATO wrote:
> I read the manual and I've had three small comments.
> 1. When explaining about the name of test, the read may want
>     to know how the name is used in ert. My understanding is
>     it can be used to select the tests to run.
>
> 2. I would like to know the difference between `Failed' and `unexpected'.
>
> 3. I would like to see the example of `should-error'.

Thanks for your feedback.  I expanded the manual to explain these 
things.  See 
http://github.com/ohler/ert/commit/908d834a08ae4b16bb7efc4318cb81df955649db 
for the changes that I made.

Christian.

Re: testing framework and package.el

[Christian Ohler]

  On 5/10/10 11:20, Sebastian Rose wrote:
> Christian Ohler<ohler+emacs@fastmail.net>  writes:
>>   On 5/10/10 2:18, Chong Yidong wrote:
>>> Because the test files are not to be distributed with the tarball.
>> Why not?  Isn't it true that whenever someone builds Emacs from source, it's
>> possible that they are using a configuration (GCC version, configure flags,
>> etc.) that hasn't been tested extensively, and it would make sense for them to
>> run Emacs' self-tests before they waste their time installing an unstable Emacs?
>> This seems to apply especially to the pretest tarballs, but also to the regular
>> distribution to some extent.
>>
>> Christian.
> Tests should be part of the repo, yes.
>
> But for tarballs?

What do you use the tarballs for?  For building Emacs from source.  
Thus, what I said above applies.

Christian.

Re: testing framework and package.el

[Stephen J. Turnbull]

Christian Ohler writes:

 >   On 5/10/10 2:18, Chong Yidong wrote:
 > > Christian Ohler<ohler+emacs@fastmail.net>  writes:
 > >> Is there a deep reason
 > >> why Emacs currently has test/ as a separate top-level directory?
 > > Because the test files are not to be distributed with the tarball.

 > Why not?

Conceptually, tests should be developed separately from the core, just
as the Lisp Reference is.  Of course, in practice it's not that easy,
but I think it's a good idea to structure the tests and the reference
at the same level as the core sources.

XEmacs experience with its tests has shown that having the tests
subsidiary to the source tree leads to regressions.  Specifically,
changing the source code leads to changes in behavior, and these
changes are "sanctified" by writing tests after the fact.  But these
tests are *not* run against older versions, leading to unplanned
divergence in behavior.  Occasionally these divergences are annoying
to users or third-party libraries.

I think as a matter of convenience both Reference and tests should be
distributed with (ie, in) the core tarball.

Re: testing framework and package.el

[Eric Schulte]

On the relation between tests and core code, how should regular Emacs
code standards be applied to tests.  For example
- need my tests compile cleanly w/o warnings
- may I require 'cl-seq in my testing files
- must I ensure that every test file name is unique in the first X many
  characters

I'd think one driving factor should be how we anticipate the tests being
used, e.g. will they mainly be run for one project at a time to as part
of the development process, or will the entire Emacs tests suite
(assuming one exists at some point in the future) often be run en mass?

Best -- Eric

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> Christian Ohler writes:
>
>  >   On 5/10/10 2:18, Chong Yidong wrote:
>  > > Christian Ohler<ohler+emacs@fastmail.net>  writes:
>  > >> Is there a deep reason
>  > >> why Emacs currently has test/ as a separate top-level directory?
>  > > Because the test files are not to be distributed with the tarball.
>
>  > Why not?
>
> Conceptually, tests should be developed separately from the core, just
> as the Lisp Reference is.  Of course, in practice it's not that easy,
> but I think it's a good idea to structure the tests and the reference
> at the same level as the core sources.
>
> XEmacs experience with its tests has shown that having the tests
> subsidiary to the source tree leads to regressions.  Specifically,
> changing the source code leads to changes in behavior, and these
> changes are "sanctified" by writing tests after the fact.  But these
> tests are *not* run against older versions, leading to unplanned
> divergence in behavior.  Occasionally these divergences are annoying
> to users or third-party libraries.
>
> I think as a matter of convenience both Reference and tests should be
> distributed with (ie, in) the core tarball.

Re: testing framework and package.el

[Stefan Monnier]

> On the relation between tests and core code, how should regular Emacs
> code standards be applied to tests.  For example
> - need my tests compile cleanly w/o warnings
> - may I require 'cl-seq in my testing files
> - must I ensure that every test file name is unique in the first X many
>   characters

I think that given the current situation, tests written in LOLCODE or
Agda are better than no tests at all,


        Stefan

Re: testing framework and package.el

[Christian Ohler]

  On 5/10/10 14:09, Stephen J. Turnbull wrote:
> Christian Ohler writes:
>
>   >    On 5/10/10 2:18, Chong Yidong wrote:
>   >  >  Christian Ohler<ohler+emacs@fastmail.net>   writes:
>   >  >>  Is there a deep reason
>   >  >>  why Emacs currently has test/ as a separate top-level directory?
>   >  >  Because the test files are not to be distributed with the tarball.
>
>   >  Why not?
>
> Conceptually, tests should be developed separately from the core, just
> as the Lisp Reference is.  Of course, in practice it's not that easy,
> but I think it's a good idea to structure the tests and the reference
> at the same level as the core sources.
>
> XEmacs experience with its tests has shown that having the tests
> subsidiary to the source tree leads to regressions.  Specifically,
> changing the source code leads to changes in behavior, and these
> changes are "sanctified" by writing tests after the fact.  But these
> tests are *not* run against older versions, leading to unplanned
> divergence in behavior.  Occasionally these divergences are annoying
> to users or third-party libraries.

AIUI, XEmacs has its tests in a tests/ directory next to lisp/, in the 
same repository.  Are you saying this has proven to be a bad idea?

The most frequent use cases of the testing system are probably browsing 
existing tests, writing new tests, running tests against locally a 
modified Emacs, and maintaining tests.  We should lay things out in such 
a way that these actions are as easy as possible.  I expect that running 
the current tests against older Emacs versions is something that we will 
want to do much less frequently, so we should not optimize for this use 
case at the expense of the others.

Anyway, I don't think the directory layout matters much as long as there 
are very few tests.  I went with a lightweight solution and put the 
tests all in one directory test/automated/ like XEmacs does.  (I'm 
referring to the current state of the ERT repository here, this code 
isn't installed yet.)  I think the primary use cases would benefit from 
having the tests right next to the corresponding sources rather than in 
a separate tree, but I'm not going to fight over it.  (Emacs functions 
for jumping from a file in lisp/ to a corresponding file in test/ solve 
only part of the problem since they only help when viewing or editing 
the files, but not while operating on them with shell commands, VC, or 
other tools.)

We can always move the files around later if we decide on a different 
convention.

Christian.

Re: testing framework and package.el

[Stephen J. Turnbull]

Christian Ohler writes:

 > AIUI, XEmacs has its tests in a tests/ directory next to lisp/, in the 
 > same repository.  Are you saying this has proven to be a bad idea?

Not bad; suboptimal.  There are two kinds of tests, tests of the LISP
language (including Emacs extensions described in the Lispref), and
tests of the internal implementation.

The former should be used for all versions of X?Emacs Lisp, and
therefore are conceptually separate from the sources.  They often need
to be versioned, but usually a boundp or fboundp test is good enough
for that.

The latter, of course, is implementation-specific and tied to the
sources.

In XEmacs practice, the two are mixed together, and as bugs are fixed
and features are added the tests for different versions have diverged,
even where the features are the same.  This means that, for example,
it's possible (and not uncommon, in fact) for 21.4 to be missing
regression tests that are present in 21.5 for 21.4 features.

 > a way that these actions are as easy as possible.  I expect that
 > running the current tests against older Emacs versions is something
 > that we will want to do much less frequently, so we should not
 > optimize for this use case at the expense of the others.

What's so hard about prepending "../" to a few Makefile variables?

You're right that changing the convention is not too hard for the
framework, but getting tests to be backward compatible is going to be
annoying if you don't plan for it from the beginning.  In practice it
very likely won't happen, and that's unfortunate.

Re: testing framework and package.el

[Lennart Borgman]

On Tue, Oct 12, 2010 at 5:02 AM, Stephen J. Turnbull <stephen@xemacs.org> wrote:
>
> You're right that changing the convention is not too hard for the
> framework, but getting tests to be backward compatible is going to be
> annoying if you don't plan for it from the beginning.  In practice it
> very likely won't happen, and that's unfortunate.

I don't understand. Is there any other way supporting backward
compatibility of the test than actually running them from time to time
in older versions? Why does it matter very much where they are
located?

Re: testing framework and package.el

[Stephen J. Turnbull]

Lennart Borgman writes:

 > I don't understand. Is there any other way supporting backward
 > compatibility of the test than actually running them from time to time
 > in older versions?

Not that I know of.

 > Why does it matter very much where they are located?

Because it affects how likely they are to be run with older versions
in my experience.  If they're in the source tree, it doesn't happen
(ie, people only run the tests that come with the particular version
of the source tree).  Backward compatibility is lost because they
diverge, and then nobody wants to expend a lot of effort making them
backward compatible when nobody will run them with old versions
anyway.

Re: testing framework and package.el

[Christian Ohler]

  On 12/10/10 14:02, Stephen J. Turnbull wrote:
> Christian Ohler writes:
>
>   >  AIUI, XEmacs has its tests in a tests/ directory next to lisp/, in the
>   >  same repository.  Are you saying this has proven to be a bad idea?
>
> Not bad; suboptimal.  There are two kinds of tests, tests of the LISP
> language (including Emacs extensions described in the Lispref), and
> tests of the internal implementation.
>
> The former should be used for all versions of X?Emacs Lisp, and
> therefore are conceptually separate from the sources.  They often need
> to be versioned, but usually a boundp or fboundp test is good enough
> for that.
>
> The latter, of course, is implementation-specific and tied to the
> sources.
>
> In XEmacs practice, the two are mixed together, and as bugs are fixed
> and features are added the tests for different versions have diverged,
> even where the features are the same.  This means that, for example,
> it's possible (and not uncommon, in fact) for 21.4 to be missing
> regression tests that are present in 21.5 for 21.4 features.

I see, thanks for clarifying.  Is it really mainly ELisp language 
features where this applies?  Any test that I write now could make sense 
for the previous Emacs version, or across all Emacsen, if it's a shared 
feature.  Probably we don't care about compatibility or regressions of 
specific UI features too much, so sharing such tests is perhaps not 
worth the trouble.  Still, do you think there's a strict boundary 
between shared tests and implementation-specific tests, or a continuum?  
If it's a continuum, perhaps `ert-deftest' should have a :since keyword 
that takes some kind of structured value that specified the Emacs 
versions that are supposed to support the feature being tested.  OTOH, 
if tests really mostly fall in one of two categories, a simpler solution 
would be sufficient.

>   >  a way that these actions are as easy as possible.  I expect that
>   >  running the current tests against older Emacs versions is something
>   >  that we will want to do much less frequently, so we should not
>   >  optimize for this use case at the expense of the others.
>
> What's so hard about prepending "../" to a few Makefile variables?

I'm not sure I understand your question.  I was trying to say that it 
would be easier to write and maintain tests if the tests for X.el were 
in X-tests.el in the same directory (or perhaps in a subdirectory 
tests/X.el), rather than in ../../test/*/*/X-tests.el, which is quite a 
bit further removed, and clumsy to navigate to (you can provide Emacs 
commands that help with this, but you really also need shell aliases, 
etc., and it's never going to be as simple as having code and tests next 
to each other).  While browsing and editing the tests, you would benefit 
from having the tests right where the source code is; and since running 
them against an older Emacs version is comparatively rare, the more 
complicated find expression that you need to collect the test files 
makes little difference.

> You're right that changing the convention is not too hard for the
> framework, but getting tests to be backward compatible is going to be
> annoying if you don't plan for it from the beginning.  In practice it
> very likely won't happen, and that's unfortunate.

I think this is a good time to discuss concrete proposals to solve this 
problem.

Christian.

Re: testing framework and package.el

[Christian Ohler]

  On 12/10/10 20:41, Lennart Borgman wrote:
> Is there any other way supporting backward
> compatibility of the test than actually running them from time to time
> in older versions?

Rather than running the tests of Emacs version N in historical versions 
before N, we can run them in future versions after N.  This is actually 
what happens naturally as the code evolves (assuming we run "make check" 
regularly), so it's very simple and requires no added effort.  (Of 
course, this doesn't work for tests written after the fact.)  As long as 
we don't touch a given test, we know that it's compatible.  If we have 
to modify it to keep it passing, we know that there was an incompatible 
change.

Christian.

Re: testing framework and package.el

[Lennart Borgman]

On Tue, Oct 12, 2010 at 3:39 PM, Stephen J. Turnbull <stephen@xemacs.org> wrote:
>
>  > Why does it matter very much where they are located?
>
> Because it affects how likely they are to be run with older versions
> in my experience.  If they're in the source tree, it doesn't happen
> (ie, people only run the tests that come with the particular version
> of the source tree).  Backward compatibility is lost because they
> diverge, and then nobody wants to expend a lot of effort making them
> backward compatible when nobody will run them with old versions
> anyway.

Wouldn't it be better than to setup the tests so that it is easy to
run them with old versions of Emacs - regardless of where the tests
are located?

And of course run it with old versions of Emacs to assure that this is
not broken. (Some tests might be, but the test structure should not be
broken.)

Re: testing framework and package.el

[Stephen J. Turnbull]

Lennart Borgman writes:

 > Wouldn't it be better than to setup the tests so that it is easy to
 > run them with old versions of Emacs - regardless of where the tests
 > are located?

Better than what?  As Christian points out, typically there are shell
scripts/.bat files and Makefiles that need to be fixed up.

What I am suggesting is simply that it makes sense to locate the
common test suite as a sibling to the source checkouts:

/usr/local/src/Emacsen -+- trunk    -+- etc
                        |            +- lisp
                        |            +- src
                        +- emacs-23 -+- etc
                        |            +- lisp
                        |            +- src
                        +- tests    -+- automated
                                     +- crashers
                                     +- ui

instead of having a separate copy

/usr/local/src/Emacsen -+- trunk    -+- etc
                        |            +- lisp
                        |            +- src
                        |            +- tests -+- automated
                                               +- crashers
                                               +- ui
                        +- emacs-23 -+- etc
                                     +- lisp
                                     +- src
                                     +- tests -+- automated
                                               +- crashers
                                               +- ui

This makes it easy to share the tests, as you only need to write the
one set of scripts, Makefiles, etc.  The tests will always be in the
same relative position to your sources (and most important, Makefiles).

 > And of course run it with old versions of Emacs to assure that this is
 > not broken. (Some tests might be, but the test structure should not be
 > broken.)

Yes, that's the theory that everybody agrees to AFAIK.  The practice
is that if there's any inconvenience, it probably won't happen.

Re: testing framework and package.el

[Stephen J. Turnbull]

Christian Ohler writes:

 > As long as we don't touch a given test, we know that it's
 > compatible.  If we have to modify it to keep it passing, we know
 > that there was an incompatible change.

You are forgetting new tests for old features.  These *must* be run
against old versions to ensure that the current version of the feature
is still compatible with the old one.

Re: testing framework and package.el

[Lennart Borgman]

On Tue, Oct 12, 2010 at 7:37 PM, Stephen J. Turnbull <stephen@xemacs.org> wrote:
> Lennart Borgman writes:
>
>  > Wouldn't it be better than to setup the tests so that it is easy to
>  > run them with old versions of Emacs - regardless of where the tests
>  > are located?
>
> Better than what?  As Christian points out, typically there are shell
> scripts/.bat files and Makefiles that need to be fixed up.

What I suggested was to avoid that situation. Make sure that nothing
in the tests are dependent on the test location.

Re: testing framework and package.el

[Stephen J. Turnbull]

Christian Ohler writes:

 > I see, thanks for clarifying.  Is it really mainly ELisp language 
 > features where this applies?

It applies to everything in the Lispref and User Guide.  We don't have
tests for UI features in most cases (although there is a suite of
tests to make sure the keyboard works correctly, but this is a real
PITA to exercise because it cannot really be done automatically,
except to some extent on X).  I would think Emacs and most packages
would also lack UI tests.

 > worth the trouble.  Still, do you think there's a strict boundary 
 > between shared tests and implementation-specific tests, or a continuum?  

As I wrote before, in my experience, f?boundp is sufficient in most
cases to check for version- or implementation-specific tests.

The implementation-specific tests are unusual, and mostly happen when
functionality is partially moved from Lisp to C or vice-versa,
resulting in the creation of an -internal version of some function.
Once again, (fboundp 'foo-internal) will usually catch that.

 > >   >  a way that these actions are as easy as possible.  I expect that
 > >   >  running the current tests against older Emacs versions is something
 > >   >  that we will want to do much less frequently, so we should not
 > >   >  optimize for this use case at the expense of the others.
 > >
 > > What's so hard about prepending "../" to a few Makefile variables?
 > 
 > I'm not sure I understand your question.

In practice, our tests are either run from within XEmacs with
M-x test-emacs-test-file, or from a Makefile.  Fix those two places
(by prepending "../") and you're golden.

 > I was trying to say that it would be easier to write and maintain
 > tests if the tests for X.el were in X-tests.el in the same
 > directory

That may be true, but in practice I don't know any large applications
that are organized that way.  Do you?

Re: testing framework and package.el

[Lennart Borgman]

On Tue, Oct 12, 2010 at 7:58 PM, Stephen J. Turnbull <stephen@xemacs.org> wrote:
>
> In practice, our tests are either run from within XEmacs with
> M-x test-emacs-test-file, or from a Makefile.  Fix those two places
> (by prepending "../") and you're golden.

Are there any reasons to use makefiles when we already have a built Emacs?

Re: testing framework and package.el

[Stefan Monnier]

> /usr/local/src/Emacsen -+- trunk    -+- etc
>                         |            +- lisp
>                         |            +- src
>                         +- emacs-23 -+- etc
>                         |            +- lisp
>                         |            +- src
>                         +- tests    -+- automated
>                                      +- crashers
>                                      +- ui

> instead of having a separate copy

> /usr/local/src/Emacsen -+- trunk    -+- etc
>                         |            +- lisp
>                         |            +- src
>                         |            +- tests -+- automated
>                                                +- crashers
>                                                +- ui
>                         +- emacs-23 -+- etc
>                                      +- lisp
>                                      +- src
>                                      +- tests -+- automated
>                                                +- crashers
>                                                +- ui

That makes a lot of sense to me (except for the unacceptable use of
a capital letter in a filename, which I assume you only put there as
a form of provocation).

We should indeed turn the current `test' subdirectory into a new
Bzr branch.  If/when we need tests that are really
implementation-specific we will see how to deal with them.


        Stefan

Re: testing framework and package.el

[Stephen J. Turnbull]

Stefan Monnier writes:

 > That makes a lot of sense to me (except for the unacceptable use of
 > a capital letter in a filename, which I assume you only put there as
 > a form of provocation).

Excuse me?  /usr/local/src is *my* property; Emacs or any other
application can make rules about its internal structure, but the name
of *my* Emacs workspace is none of your business. :-)

Note that it matters that this is *my* repo, because it might just be
me that likes this setup.  If other developers are *not* in the habit
of keeping all their Emacs workspaces in one directory, then my
proposal makes less sense (because both Emacs and the build scripts
may need extra, possibly fragile, logic to find the tests).  (I would
still favor it, but I couldn't say that people who organize their
branches differently, and prefer a different location for "tests", are
wrong.)  By the same logic, although I'm glad you agree with me (or
vice versa, if you prefer ;-), I think you should wait for a bit more
discussion before pronouncing.

Also note that you don't have to structure the shared repo the same
way, although in Bazaar that seems like the most logical way to do
it.

Lastly, nested branches will show up at some point (in the next year
or so), and then it will be possible to nest a common test directory
in the Emacs source directory, with attendant simplification of
Makefile and Emacs' test-file-finding logic.

Re: testing framework and package.el

[Christian Ohler]

  On 13/10/10 4:41, Stephen J. Turnbull wrote:
> Christian Ohler writes:
>
>   >  As long as we don't touch a given test, we know that it's
>   >  compatible.  If we have to modify it to keep it passing, we know
>   >  that there was an incompatible change.
>
> You are forgetting new tests for old features.

I was referring to them as "tests written after the fact", sorry for not 
being clearer.

> These *must* be run
> against old versions to ensure that the current version of the feature
> is still compatible with the old one.

Yes; but this only needs to be done once per test.  The old version is 
not going to change.  This is assuming that tests get merged into all 
branches where changes are made, including bugfix branches.  I assume 
this is the part that you want to avoid.

Christian.

Re: testing framework and package.el

[Christian Ohler]

  On 13/10/10 4:58, Stephen J. Turnbull wrote:
> Christian Ohler writes:
>
>   >  I see, thanks for clarifying.  Is it really mainly ELisp language
>   >  features where this applies?
>
> It applies to everything in the Lispref and User Guide.  We don't have
> tests for UI features in most cases (although there is a suite of
> tests to make sure the keyboard works correctly, but this is a real
> PITA to exercise because it cannot really be done automatically,
> except to some extent on X).  I would think Emacs and most packages
> would also lack UI tests.

ERT has some UI tests.  (Maybe just one, I don't remember.)

>   >  worth the trouble.  Still, do you think there's a strict boundary
>   >  between shared tests and implementation-specific tests, or a continuum?
>
> As I wrote before, in my experience, f?boundp is sufficient in most
> cases to check for version- or implementation-specific tests.
>
> The implementation-specific tests are unusual, and mostly happen when
> functionality is partially moved from Lisp to C or vice-versa,
> resulting in the creation of an -internal version of some function.
> Once again, (fboundp 'foo-internal) will usually catch that.

OK, that sounds good.

>   >  I was trying to say that it would be easier to write and maintain
>   >  tests if the tests for X.el were in X-tests.el in the same
>   >  directory
>
> That may be true, but in practice I don't know any large applications
> that are organized that way.  Do you?

Yes.  Chromium is a fairly large code base that seems to have .cc, .h 
and _unittest.cc files next to each other, see, e.g., 
http://src.chromium.org/viewvc/chrome/trunk/src/remoting/base/ .  I know 
about other proprietary examples but didn't look any further for 
open-source ones.

Christian.

Re: testing framework and package.el

[Christian Ohler]

  On 13/10/10 11:36, Stefan Monnier wrote:
>> /usr/local/src/Emacsen -+- trunk    -+- etc
>>                          |            +- lisp
>>                          |            +- src
>>                          +- emacs-23 -+- etc
>>                          |            +- lisp
>>                          |            +- src
>>                          +- tests    -+- automated
>>                                       +- crashers
>>                                       +- ui
>
> That makes a lot of sense to me (except for the unacceptable use of
> a capital letter in a filename, which I assume you only put there as
> a form of provocation).
>
> We should indeed turn the current `test' subdirectory into a new
> Bzr branch.  If/when we need tests that are really
> implementation-specific we will see how to deal with them.

IIUC, this design requires developers who want to add features and write 
tests for them to check out two different bzr branches and submit two 
different patches.  Isn't that a rather high bar that will be 
inconvenient for developers and discourage them from writing tests?  
Earlier in this thread, you said:

> I think that given the current situation, tests written in LOLCODE or
> Agda are better than no tests at all,

This sounded to me like you would like the bar for contributing tests to 
be low.


As an alternative that avoids this problem, we could maintain the tests 
in Emacs trunk, but our makefiles could have a test_src variable that 
defaults to tests, and that we can set to ../trunk/tests when running 
tests in the emacs-23 directory in Stephen's layout.

Christian.

Re: testing framework and package.el

[Stephen J. Turnbull]

Christian Ohler writes:
 >   On 13/10/10 4:41, Stephen J. Turnbull wrote:
 > > Christian Ohler writes:
 > >
 > >   >  As long as we don't touch a given test, we know that it's
 > >   >  compatible.  If we have to modify it to keep it passing, we know
 > >   >  that there was an incompatible change.
 > >
 > > You are forgetting new tests for old features.
 > 
 > I was referring to them as "tests written after the fact", sorry for not 
 > being clearer.
 > 
 > > These *must* be run against old versions to ensure that the
 > > current version of the feature is still compatible with the old
 > > one.
 > 
 > Yes; but this only needs to be done once per test.

That assumes that the test doesn't change, and that the test-writer
remembers to add/fix the test in old versions of Emacs.  If she
forgets, then it doesn't need to be done at all, you just have to live
with any divergences that *would* have been cuaght by the test.

 > The old version is not going to change.  This is assuming that
 > tests get merged into all branches where changes are made,
 > including bugfix branches.  I assume this is the part that you want
 > to avoid.

Yes, indeed.  I might do it, but I might forget.  I doubt very many
contributors would be willing to do so at all, unless reminded
aggressively.  But that sort of implies a "no commits without tests"
policy, and I don't think Emacs is ready for that.

Re: testing framework and package.el

[Stephen J. Turnbull]

Christian Ohler writes:

 > IIUC, this design requires developers who want to add features and write 
 > tests for them to check out two different bzr branches and submit two 
 > different patches.

Currently, yes.  With nested branches, the checkout will be implicit.

 > Isn't that a rather high bar that will be inconvenient for
 > developers and discourage them from writing tests?

In my experience, people who actually write tests (unless there's a
firm requirement that tests be submitted before committing) are hard
to discourage.  The only job programmers like less than writing tests
is writing docs. :-)

The real problem is getting those tests backported.

 > As an alternative that avoids this problem, we could maintain the tests 
 > in Emacs trunk, but our makefiles could have a test_src variable that 
 > defaults to tests, and that we can set to ../trunk/tests when running 
 > tests in the emacs-23 directory in Stephen's layout.

Which still requires serious developers to have multiple checkouts.

Re: testing framework and package.el

[Christian Ohler]

  On 14/10/10 1:19, Stephen J. Turnbull wrote:
> Christian Ohler writes:
>
>   >  IIUC, this design requires developers who want to add features and write
>   >  tests for them to check out two different bzr branches and submit two
>   >  different patches.
>
> Currently, yes.  With nested branches, the checkout will be implicit.

Let's find a solution that works well now rather than speculating on a 
future bzr feature that may be delayed, may be slow in its initial 
implementation, or may turn out not to work the way we think it will.  
Once nested branches are implemented, we can learn how they work, and 
then split the repositories if it still makes sense.


>   >  Isn't that a rather high bar that will be inconvenient for
>   >  developers and discourage them from writing tests?
>
> In my experience, people who actually write tests (unless there's a
> firm requirement that tests be submitted before committing) are hard
> to discourage.

We want more than this; we want those who don't write tests to start 
doing so.  Putting significant barriers in their way makes it much less 
likely that Emacs will gain a useful test suite.


> The real problem is getting those tests backported.

You mean, running a test suite in an older Emacs version, and adding 
fboundp checks to the tests that test missing features, is harder than 
writing the test suite in the first place?  I don't expect that will be 
true.


>   >  As an alternative that avoids this problem, we could maintain the tests
>   >  in Emacs trunk, but our makefiles could have a test_src variable that
>   >  defaults to tests, and that we can set to ../trunk/tests when running
>   >  tests in the emacs-23 directory in Stephen's layout.
>
> Which still requires serious developers to have multiple checkouts.

You are right that developers who want to run tests in multiple versions 
of Emacs will need the code for all those versions, and the code for the 
tests.  There is no way around that.  However, including the tests in 
Emacs' trunk  saves them one checkout, and means less hassle submitting 
Emacs bugfixes together with additional tests.  Also, casual 
contributors often won't have multiple Emacs versions checked out, and 
they are worth catering for.  It seems more likely that they would 
contribute tests if they can do so without the overhead of multiple 
workspaces.

The solution of adding a makefile variable like test_src that allows 
running the test suite from a different workspace seems to solve the 
problem you described, and is much simpler than splitting the 
repository.  Do you think it has significant drawbacks?

Christian.

Re: testing framework and package.el

[Christian Ohler]

  Thanks for the explanation.  I agree with your overall point that it 
can be useful to run the current version of the test suite on an older 
version of Emacs.  If it's as simple as adding a makefile variable, we 
should implement it.

Christian.

Re: testing framework and package.el

[Stefan Monnier]

> There is no way around that.  However, including the tests in Emacs' trunk
> saves them one checkout, and means less hassle submitting Emacs bugfixes
> together with additional tests.

Actually, to submit a bug report and a test, you definitely do not need
to checkout the trunk.  So that case might be easier if the tests are
placed in their own repository than if they're placed in the Emacs
repository, whose `checkout' is known to be problematic for some people.

In any case, this issue doesn't matter nearly as much as installing ERT
and starting to write actual tests.


        Stefan

Re: testing framework and package.el

[Stephen J. Turnbull]

Christian Ohler writes:

 > > In my experience, people who actually write tests (unless there's a
 > > firm requirement that tests be submitted before committing) are hard
 > > to discourage.
 > 
 > We want more than this; we want those who don't write tests to start 
 > doing so.  Putting significant barriers in their way makes it much less 
 > likely that Emacs will gain a useful test suite.

You're excessively optimistic, IMO.

Since I can't prove that, I'm out of this thread.