Need for a Emacs regression test suite

Need for a Emacs regression test suite

[dhruva]

Hi,
 The various changes getting into Emacs and number of things breaking
sporadically, IMO we need to have a regression suite. A collection
batch mode scripts and macros to test the GUI invocation.
 It will take time to build it but would be useful. We can catch
issues during big merges much earlier in the cycle.
 We could start small and build up the suite. What does the core team feel?

-dky

-- 
Dhruva Krishnamurthy
Contents reflect my personal views only!

Re: Need for a Emacs regression test suite

[Lennart Borgman (gmail)]

dhruva wrote:
> Hi,
>  The various changes getting into Emacs and number of things breaking
> sporadically, IMO we need to have a regression suite. A collection
> batch mode scripts and macros to test the GUI invocation.
>  It will take time to build it but would be useful. We can catch
> issues during big merges much earlier in the cycle.
>  We could start small and build up the suite. What does the core team feel?
> 
> -dky

There is a page about unit testing on EmacsWiki that might help:

   http://www.emacswiki.org/cgi-bin/wiki/UnitTesting

Need for a Emacs regression test suite

[Stephen J. Turnbull]

dhruva writes:

 > Hi,
 >  The various changes getting into Emacs and number of things breaking
 > sporadically, IMO we need to have a regression suite. A collection
 > batch mode scripts and macros to test the GUI invocation.
 >  It will take time to build it but would be useful. We can catch
 > issues during big merges much earlier in the cycle.
 >  We could start small and build up the suite. What does the core team feel?

FWIW, XEmacs has a testing framework, the authors are well-documented
and few; most of the code is FSF-assigned.  The tests themselves have
a wider variety of authors as you would expect, but most files list
only an FSF copyright.  In 2001, Emacs passed more than half of the
tests without any adjustments to the tests, a few tweaks to the code
were needed.  I don't know about now, but it should be a good head
start.

I'm not willing to do the deskwork of checking assignment status of
authors (and don't have access to FSF assignment files anyway), but
would help tracking down authors who have not assigned or have
ambiguous records, as well as encourage them to do paperwork if
necessary.

Re: Need for a Emacs regression test suite

[David Kastrup]

dhruva <dhruvakm@gmail.com> writes:

> On 9/9/07, Richard Stallman <rms@gnu.org> wrote:
>>      The various changes getting into Emacs and number of things breaking
>>     sporadically, IMO we need to have a regression suite. A collection
>>     batch mode scripts and macros to test the GUI invocation.
>>
>> I don't see how that is possible, but if you know a way to do it,
>> please do it.
>
> I will start looking into the XEmacs testing framework. Once I have
> something working on my local box, I will get back. Let us delay the
> legalities till we get the technicalities sorted out.

Stephen has already outlined the parts of the legal issues he sees and
just what kind of help he is willing to provide getting them solved
(thanks for that!).

I would suggest that you are careful not to invest too much
Emacs-specific work that might be lost ultimately due to legal
reasons, though this warning might appear somewhat premature at the
moment.  In any way, it would seem important to evaluate the testing
framework and see how well it applies to Emacs and how useful it is,
and for that, it certainly makes sense to write some tests and see how
easy it is, and how useful the results.

If it applies suitably well (and from Stephen's description that half
of the tests get passed by Emacs this seems very much so), it is
important that we get the legalities solved for the framework.  That
is really indispensible.  Getting papers for the tests themselves
could then be done on a case-by-case basis: it would not preclude us
from writing our own tests or embedding those that are properly
accounted for.

It certainly seems like something we should pursue.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Need for a Emacs regression test suite

[Lennart Borgman (gmail)]

David Kastrup wrote:
> I would suggest that you are careful not to invest too much
> Emacs-specific work that might be lost ultimately due to legal
> reasons, though this warning might appear somewhat premature at the
> moment.

Could not the testing framework be used by developers even if there are 
some legal problems with it?

Re: Need for a Emacs regression test suite

[David Kastrup]

"Lennart Borgman (gmail)" <lennart.borgman@gmail.com> writes:

> David Kastrup wrote:
>> I would suggest that you are careful not to invest too much
>> Emacs-specific work that might be lost ultimately due to legal
>> reasons, though this warning might appear somewhat premature at the
>> moment.
>
> Could not the testing framework be used by developers even if there
> are some legal problems with it?

Using third-party tools should not constitute a problem when they are
licensed under appropriate terms.  But in this case the main modus of
operandi means extending the test case base, and that implies an
investment of work.  So there is some desire to not make this sort of
work entangled with code where it is not clear that we will always be
able to use it, or distribute it.

The GNU project has always tried to keep core components of the GNU
system copyrighted by the FSF.  If a testing suite becomes an integral
part of developing Emacs, something that we rely on, it would be
desirable to have this the case here as well.

That does not mean that it would be impossible to work with it
otherwise, but if we have a chance of ensuring this, it would be
better to do so.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Need for a Emacs regression test suite

[Richard Stallman]

    >      The various changes getting into Emacs and number of things breaking
    >     sporadically, IMO we need to have a regression suite. A collection
    >     batch mode scripts and macros to test the GUI invocation.
    >
    > I don't see how that is possible, but if you know a way to do it,
    > please do it.

    I will start looking into the XEmacs testing framework.

Is this a test of programming constructs or a test for GUI usage?
Regression tests for programming constructs are a usual thing, and I
have no doubt that is possible for Emacs.  I think it would be good
for someone to find the people we should ask for papers, so that we
can use the one in XEmacs.

But the suggestion was "to test the GUI invocation".  I don't see how
regression tests for that could possibly work.  Does XEmacs' testing
framework apply to user input operations?

Re: Need for a Emacs regression test suite

[Stephen J. Turnbull]

Richard Stallman writes:

 > But the suggestion was "to test the GUI invocation".  I don't see how
 > regression tests for that could possibly work.  Does XEmacs' testing
 > framework apply to user input operations?

Not currently.  I don't know about Windows and Carbon, but it would
not be impossible to develop a framework to generate X events to send
to X windows at known positions.  (Ie, you would arrange for a frame
to be created at 0,0, then you can send mouse clicks to the menubar
and things like that, also keystroke events.)  Since Windows and
Carbon are designed for operation only the local console an entirely
different approach would have to be taken, I suppose.

These tests can easily be done in LISP, since there is an xlib package
and an X window manager, both written entirely in LISP.

Re: Need for a Emacs regression test suite

[Davis Herring]

>  > But the suggestion was "to test the GUI invocation".  I don't see how
>  > regression tests for that could possibly work.  Does XEmacs' testing
>  > framework apply to user input operations?
>
> Not currently.  I don't know about Windows and Carbon, but it would
> not be impossible to develop a framework to generate X events to send
> to X windows at known positions.  (Ie, you would arrange for a frame
> to be created at 0,0, then you can send mouse clicks to the menubar
> and things like that, also keystroke events.)  Since Windows and
> Carbon are designed for operation only the local console an entirely
> different approach would have to be taken, I suppose.
>
> These tests can easily be done in LISP, since there is an xlib package
> and an X window manager, both written entirely in LISP.

See also java.awt.Robot:
http://java.sun.com/j2se/1.5.0/docs/api/java/awt/Robot.html.  That page
says that it uses the XTEST extension on X, and I have seen it work on
Windows (so there is presumably a painless way to do it).

Davis

-- 
This product is sold by volume, not by mass.  If it appears too dense or
too sparse, it is because mass-energy conversion has occurred during
shipping.

Re: Need for a Emacs regression test suite

[Mathias Dahl]

> Not currently.  I don't know about Windows and Carbon, but it would
> not be impossible to develop a framework to generate X events to send
> to X windows at known positions.  (Ie, you would arrange for a frame
> to be created at 0,0, then you can send mouse clicks to the menubar
> and things like that, also keystroke events.)

You mean like xnee (http://www.gnu.org/software/xnee/):

 Xnee is a suite of programs that can record, replay and distribute
user actions under the X11 environment. Think of it as a robot that
can imitate the job you just did. Xnee can be used to:

    Automate tests 	e.g nightly automatic regression tests
    Demonstrate programs 	e.g record and replay a user interaction
with your system
    Distribute actions 	e.g teacher's actions can be sent to all
student's desktops
    Record and replay 'macro' 	e.g Tie a key modifier combination (e.g
using xkeymouse) to replay a recorded session. The macro is WM- and
program independent.
    Retype a file 	e.g When testing an editor, Xnee can type in the
content of a file. This can be compared with the original and thereby

Re: Need for a Emacs regression test suite

[Stephen J. Turnbull]

Mathias Dahl writes:

 > You mean like xnee (http://www.gnu.org/software/xnee/):

Maybe; it depends on how X-specific the scripts end up being.

Re: Need for a Emacs regression test suite

[Richard Stallman]

    Not currently.  I don't know about Windows and Carbon, but it would
    not be impossible to develop a framework to generate X events to send
    to X windows at known positions.  (Ie, you would arrange for a frame
    to be created at 0,0, then you can send mouse clicks to the menubar
    and things like that, also keystroke events.)

Indeed, that would do the job, if we can also control the frame size
and font, in such a way that we can reliably associate window
positions with actions.  That is a good idea.

    These tests can easily be done in LISP, since there is an xlib package
    and an X window manager, both written entirely in LISP.

To use them for this, maybe we would need to install them in Emacs.
The xlib package certainly sounds interesting to install.
Can you tell us where to find it, and the author's name and email?

Re: Need for a Emacs regression test suite

[Stephen J. Turnbull]

Richard Stallman writes:

 >     These tests can easily be done in LISP, since there is an xlib package
 >     and an X window manager, both written entirely in LISP.
 > 
 > To use them for this, maybe we would need to install them in Emacs.
 > The xlib package certainly sounds interesting to install.
 > Can you tell us where to find it, and the author's name and email?

Both the xlib package and the xwem (window manager) are by Zajcev
Evgeny <zevlg@yandex.org>.  They are available from XEmacs CVS
(currently down due to problems at dotsrc.org, I'm afraid, but should
be ok soon) or by FTP from ftp://ftp.xemacs.org/pub/xemacs/packages/.

I'm sorry, but upstream is not obvious from the READMEs.  I suspect
it's a personal Arch repo and he checks into XEmacs CVS as the primary
distribution point, but you'll have to ask him.

Re: Need for a Emacs regression test suite

[Richard Stallman]

I wrote to Zajcev Evgeny.  Thanks.

Re: Need for a Emacs regression test suite

[Stephen J. Turnbull]

dhruva writes:

 > I request Stephen to provide me some links that can help me understand
 > and get it up and running for Emacs. Hope it is hackable with average
 > Elisp knowledge.

Basically, just get a copy of the XEmacs sources and the test suite is
self-contained, in the tests/automated directory.

The testing framework itself is in the file test-harness.el.  The
tests, which are more or less organized according to subsystem, are in
files named MODULE-tests.el.  (These files are never byte-compiled to
.elcs; instead, the tests are run both uncompiled and compiled, with
compilation being done on the fly.)

The only link I can give off hand is mailto:stephen@xemacs.org. :-)  I
will certainly contribute to any work on the test suite to make it
work with Emacs, because that is an excellent way to identify, and
with luck in many cases eliminate, gratuitous differences between the
two.

There may be some information in the Internals manual, but in fact
writing most tests is "mere scripting".  We have some macros such as
Assert to actually run a test and check for correct values, CheckError
to run a test of invalid code and check that it raises the correct
error condition, and so on.  If you can write recipe to identify a
bug, then you just wrap it in one of those macros.  Eg, to test
addition you could do

  (Assert (= (+ 1 1) 2))

and you're done.

Re: Need for a Emacs regression test suite

[dhruva]

Hi,

On 9/11/07, Stephen J. Turnbull <stephen@xemacs.org> wrote:
> The only link I can give off hand is mailto:stephen@xemacs.org. :-)  I

That is the best link, nothing can be better than interacting with a
responsive author/maintainer, thank you for offering to help.

I have downloaded the sources (wish XEmacs had a Hg/GIT repo, I would
have updated directly from the repo) and gave it a try. Emacs does not
have "display-error", I will create an Emacs variant of it and
continue. I have started playing around with it and will bug you when
I get stuck.

For the UI part, I do not have an answer and no experience in X. I was
wondering if we can have a start time flag which can map the keyboard
and mouse to a port instead of the real mouse and keyboard. We can
have a program that can feed the events to that port. The 'select'
will be listening on that port (instead of keyboard/mouse). Looks like
it is theoretically possible but not sure if practically feasible.

-dky

-- 
Dhruva Krishnamurthy
Contents reflect my personal views only!

Re: Need for a Emacs regression test suite

[Stephen J. Turnbull]

dhruva writes:

 > For the UI part, I do not have an answer and no experience in X. I was
 > wondering if we can have a start time flag which can map the keyboard
 > and mouse to a port instead of the real mouse and keyboard.

Not necessary.  See Mathias's post for an existing solution for X11,
and other posts mention that Java can do it for Windows.

FYI, the basic point for X11 is that even for a server on a local
console, the X protocol is used over a standard channel (usually a
network or Unix domain socket), and is basically public information to
any process with physical access to that channel.  (This is also why X
is insecure.)

Re: Need for a Emacs regression test suite

[Eli Zaretskii]

> Date: Tue, 11 Sep 2007 09:20:58 +0530
> From: dhruva <dhruvakm@gmail.com>
> Cc: rms@gnu.org, emacs-devel@gnu.org
> 
> For the UI part, I do not have an answer and no experience in X. I was
> wondering if we can have a start time flag which can map the keyboard
> and mouse to a port instead of the real mouse and keyboard. We can
> have a program that can feed the events to that port. The 'select'
> will be listening on that port (instead of keyboard/mouse). Looks like
> it is theoretically possible but not sure if practically feasible.

I'd suggest first to crack the problem of testing the non-interactive
parts of Emacs, and postponing the UI part for later.  After all, most
of Emacs code -- the parts written in Lisp, and a significant portion
of C primitives -- are not related to the display engine, so having a
way to regression-test them alone would be a huge win.

I'm not saying that testing display and input events is not important,
just that the first approximation to a test suite does not need to
solve all the problems at once, in order to be useful.  Especially
since the solution to automated UI exercising will probably end up
being platform-specific, so will need to be tackled separately for
each platform.