Re: testing framework and package.el

["Stephen J. Turnbull" <stephen@xemacs.org>]

Christian Ohler writes:

 > AIUI, XEmacs has its tests in a tests/ directory next to lisp/, in the 
 > same repository.  Are you saying this has proven to be a bad idea?

Not bad; suboptimal.  There are two kinds of tests, tests of the LISP
language (including Emacs extensions described in the Lispref), and
tests of the internal implementation.

The former should be used for all versions of X?Emacs Lisp, and
therefore are conceptually separate from the sources.  They often need
to be versioned, but usually a boundp or fboundp test is good enough
for that.

The latter, of course, is implementation-specific and tied to the
sources.

In XEmacs practice, the two are mixed together, and as bugs are fixed
and features are added the tests for different versions have diverged,
even where the features are the same.  This means that, for example,
it's possible (and not uncommon, in fact) for 21.4 to be missing
regression tests that are present in 21.5 for 21.4 features.

 > a way that these actions are as easy as possible.  I expect that
 > running the current tests against older Emacs versions is something
 > that we will want to do much less frequently, so we should not
 > optimize for this use case at the expense of the others.

What's so hard about prepending "../" to a few Makefile variables?

You're right that changing the convention is not too hard for the
framework, but getting tests to be backward compatible is going to be
annoying if you don't plan for it from the beginning.  In practice it
very likely won't happen, and that's unfortunate.