emacs test suite?

emacs test suite?

[Robert Anderson]

When I help identify and fix bugs, I like to be able to provide a
"push-button" test case that demonstrates the problem, and
prevents regression once the problem has been fixed.

When I run "make check" for GNU emacs, it tells me there are no
tests for emacs yet.

Have there been any initiatives to develop a testing system for
emacs?

Bob

Re: emacs test suite?

[Thien-Thi Nguyen]

"Robert Anderson" <RWA@sbcglobal.net> writes:

   When I help identify and fix bugs, I like to be able to provide a
   "push-button" test case that demonstrates the problem, and
   prevents regression once the problem has been fixed.

this approach is also useful for new code.

   When I run "make check" for GNU emacs, it tells me there are no
   tests for emacs yet.

   Have there been any initiatives to develop a testing system for
   emacs?

that's what the ftp site is for, no?  (half-smiley.)

i'm in favor of such a system.  for current-column munging (in the presence of
variable-width fonts), i have a local testing setup but it is not generalized.
this could be because testing interactive programs typically requires a driver
program external to the one under test, and that starts stepping into debugger
territory (where programmers' disparate debugging styles is nonregularizable).

however, having said that, perhaps someone could build a system on top of
screen(1), for text mode; and something analogous, for X.

thi

Re: emacs test suite?

[Robert Anderson]

--- Original Message ---
From: Thien-Thi Nguyen <ttn@glug.org>
To: "Robert Anderson" <RWA@sbcglobal.net>
CC: emacs-devel@gnu.org
Subject: Re: emacs test suite?

>"Robert Anderson" <RWA@sbcglobal.net> writes:
>
>   When I help identify and fix bugs, I like to be able to provide a
>   "push-button" test case that demonstrates the problem, and
>   prevents regression once the problem has been fixed.
>
>this approach is also useful for new code.
>
>   When I run "make check" for GNU emacs, it tells me there are no
>   tests for emacs yet.
>
>   Have there been any initiatives to develop a testing system for
>   emacs?
>
>that's what the ftp site is for, no?  (half-smiley.)
>
>i'm in favor of such a system.  for current-column munging (in
the presence of
>variable-width fonts), i have a local testing setup but it is
not generalized.
>this could be because testing interactive programs typically
requires a driver
>program external to the one under test, and that starts stepping
into debugger
>territory (where programmers' disparate debugging styles is
nonregularizable).
>
>however, having said that, perhaps someone could build a system
on top of
>screen(1), for text mode; and something analogous, for X.
>
>thi

[sorry for the screwed up quoting, my mailer is horrible.]

It depends on what you are trying to test, I suppose.  I think
there is a broad class of "textual transformation" tests that
would be trivial to implement as batch jobs called from a simple
script harness.

In fact, I've made a very simple sketch of how such a thing would
work, which verifies the cc-mode filling bug in CVS HEAD (and
also demonstrates that it works "as expected" for 21.1.1 and 20.7.1.)

There is a driver script which runs GNU find on a subtree looking
for programs which have the name, by convention, "run-test". 
Each command "run-test" accepts an argument "--description" which
is spits out a one-line description to be displayed while the
test is running, as well as an option to specify a full path to
the emacs binary to test.  Each program "run-test" returns 0 for
pass and non-zero for fail.

I suspect from perusing the mailing list that this sort of thing
might be valuable, if people would use it.

Bob

Re: emacs test suite?

[Robert W Anderson]

Thien-Thi Nguyen wrote:

>"Robert Anderson" <RWA@sbcglobal.net> writes:
>
>   When I help identify and fix bugs, I like to be able to provide a
>   "push-button" test case that demonstrates the problem, and
>   prevents regression once the problem has been fixed.
>
>this approach is also useful for new code.
>
>   When I run "make check" for GNU emacs, it tells me there are no
>   tests for emacs yet.
>
>   Have there been any initiatives to develop a testing system for
>   emacs?
>
>that's what the ftp site is for, no?  (half-smiley.)
>
That went over my head.  What ftp site?

Bob

Re: emacs test suite?

[Thien-Thi Nguyen]

   From: "Robert Anderson" <RWA@sbcglobal.net>
   Date: Thu, 16 Jan 2003 19:25:07 -0500

   I suspect from perusing the mailing list that this sort of thing
   might be valuable, if people would use it.

value arises from the meeting of the eye and the train-track.  the
tree-walk you describe sounds agreeable, so ... did i miss a link to
posted code?  see also "ttn-do cron-walk":

  http://www.glug.org/people/ttn/software/ttn-do/dist/

how do people feel about using scheme scripts for emacs development?

thi

Re: emacs test suite?

[Richard Stallman]

    When I run "make check" for GNU emacs, it tells me there are no
    tests for emacs yet.

    Have there been any initiatives to develop a testing system for
    emacs?

I don't remember any.  Anyway, nobody has been looking at it recently.

Many, perhaps most, Emacs bug reports involve interactive input or
incorrect display or both.  It's hard to make test cases for such
problems.  Given this, is a testing system still useful?

Re: emacs test suite?

[Richard Stallman]

    In fact, I've made a very simple sketch of how such a thing would
    work, which verifies the cc-mode filling bug in CVS HEAD (and
    also demonstrates that it works "as expected" for 21.1.1 and 20.7.1.)

If you can find a useful subset of bugs that you can test in this way,
it seems worth pursuing.

I am concerned, though, about the time it might take to enter bugs in
the tests.  I can barely find time to fix some bugs--I certainly
cannot write tests for them too.  And I would hate to see the scarce
time of people like Kim Storm and Stefan Monnier diverted into writing
tests instead of fixing bugs and adding features.

If we can find volunteers who like writing tests, then we could build
up a set of tests.

Re: emacs test suite?

[Richard Stallman]

    >however, having said that, perhaps someone could build a system
    on top of
    >screen(1), for text mode; and something analogous, for X.

That is a bizarre idea, but it might work.