emacs test suite

emacs test suite

[Robert Anderson]

Re: ftp site.  Now I get it. That form of testing is invaluable,
but by itself, leaves a lot to be desired.

And no, you didn't miss my link.  I was waiting to see if anyone
would respond favorably, so as not to waste time on something for
which there is no interest.

My next question would be:  What is the best way to share this
kind of contribution?  There is a bit of a bootstrapping problem
in getting it to a state where people see that it should
certainly go in the development sources.  But if it's not there,
it's not likely to grow and become that useful.

If emacs used a distributed version control system, this would
not be a problem. (for which the only GPL solution right now is
arch, btw).  Oh well.

I guess for now I will publish a tarball, and we can coordinate
via email for adding tests.

Here it is:  http://rwa.homelinux.net/emacs/emacs-test.tar.gz

Thanks,
Bob

Re: emacs test suite

[Thien-Thi Nguyen]

"Robert Anderson" <RWA@sbcglobal.net> writes:

   Here it is:  http://rwa.homelinux.net/emacs/emacs-test.tar.gz

i just tried to grab this but it is no longer there.  where is it now?
(thanks for accomodating my slow schedule.)

thi

emacs test suite

[Robert Anderson]

The link to my (larval) testing scripts is now:

http://rwa.homelinux.net/emacs

The higher version numbers are newer.  I'll be exploring the
issue of testability for a bit.

Bob

Re: emacs test suite

[Robert Anderson]

--- Original Message ---
From: Richard Stallman <rms@gnu.org>
To: "Robert Anderson" <RWA@sbcglobal.net>
Subject: Re: emacs test suite

>    If emacs used a distributed version control system, this would
>    not be a problem.
>
>We use CVS--isn't that suitable?

It's reasonable, but has some limitations.  For example, I am
potentially interested in maintaining a set of tests, but you are
not convinced of the utility of them and so you'd rather not have
them in your CVS - so you don't give me write access.

That's fine, and perfectly reasonable.  However, that puts me in
a bad spot, because although I can follow your work by updating
from CVS, I cannot "commit" my work (the tests) if it involves
changes to any files or dirs in your CVS repo.  So I am stuck,
and likely to give up, because I have hit a bottleneck in CVS
which only allows me to work with your sources if you have given
me write access to your repository.  I can make simple local
changes, but I can't version control my own work.

If instead I had a branch that I could store locally (this is the
"distributed" part), I could version control my own work,
continue to incorporate your changes, and as it became more
useful, you could at some point decide that you'd like to merge
my branch into your sources - and since I've been updating from
your repo, that would be a relatively smooth process (as opposed
to what CVS would allow:  for me to import into a local CVS repo
at some moment in time, and develop from there.  Then, when ready
to merge, all of the work that had occured in the main repo from
the time I imported would have to be carefully recreated to
prevent regressions from the merge).

Not to mention the performance boon of development from local
copies of the repo...

Bob

Re: emacs test suite

[Eli Zaretskii]

On Sat, 18 Jan 2003, Robert Anderson wrote:

> >We use CVS--isn't that suitable?
> 
> It's reasonable, but has some limitations.  For example, I am
> potentially interested in maintaining a set of tests, but you are
> not convinced of the utility of them and so you'd rather not have
> them in your CVS - so you don't give me write access.

We have CVS branches for that; a couple of them already exist for 
specialized features people work on.  So I don't see any problem here.

> If instead I had a branch that I could store locally (this is the
> "distributed" part), I could version control my own work,
> continue to incorporate your changes, and as it became more
> useful, you could at some point decide that you'd like to merge
> my branch into your sources

That's cool, but why do you need the branch to be local?  Why not start a 
branch in the Emacs CVS?  If you agree not to check in changes into the 
trunk (assuming we don't want them on the trunk, about which I'm unsure, 
see below), I don't see any problems granting you write access to the CVS 
tree.  It's Richard's decision, but I don't see why would he refuse.

Moreover, I don't even see why would we request that the changes be on a 
branch.  A test suite by definition is mostly orthogonal to the sources 
being tested.  I expect it to be in a separate directory, with only minor 
influence on the files in other directories (perhaps some simple change 
in some Makefile.in or so).

Therefore, even a branch does not seem to be necessary.

Am I missing something?

Re: emacs test suite

[Robert Anderson]

On Sat, 2003-01-18 at 22:00, Eli Zaretskii wrote:
> 
> On Sat, 18 Jan 2003, Robert Anderson wrote:
> 
> > >We use CVS--isn't that suitable?
> > 
> > It's reasonable, but has some limitations.  For example, I am
> > potentially interested in maintaining a set of tests, but you are
> > not convinced of the utility of them and so you'd rather not have
> > them in your CVS - so you don't give me write access.
> 
> We have CVS branches for that; a couple of them already exist for 
> specialized features people work on.  So I don't see any problem here.
> 
> > If instead I had a branch that I could store locally (this is the
> > "distributed" part), I could version control my own work,
> > continue to incorporate your changes, and as it became more
> > useful, you could at some point decide that you'd like to merge
> > my branch into your sources
> 
> That's cool, but why do you need the branch to be local?

Well, I don't "need" it to be - this is only a matter of convenience. 
But it would be nice if this didn't even require discussion or Richard
having to make a decision apriori about whether my work is credible
before he has even seen it.  If the branch was local, I would just get
to work, and everyone could worry about these decisions later when it
was more than vaporware, saving valuable coordinate effort.

  Why not start a 
> branch in the Emacs CVS?  If you agree not to check in changes into the 
> trunk (assuming we don't want them on the trunk, about which I'm unsure, 
> see below), I don't see any problems granting you write access to the CVS 
> tree.  It's Richard's decision, but I don't see why would he refuse.
> 
> Moreover, I don't even see why would we request that the changes be on a 
> branch.  A test suite by definition is mostly orthogonal to the sources 
> being tested.  I expect it to be in a separate directory, with only minor 
> influence on the files in other directories (perhaps some simple change 
> in some Makefile.in or so).
> 
> Therefore, even a branch does not seem to be necessary.
> 
> Am I missing something?

Only that, were the branch local, this discussion would be unnecessary -
I would just start working and only "bother" you guys and take your
valuable time when it became more mission critical, i.e., there was code
suitable for a merge.  For the test suite in particular, you are right
that inherent orthogonality makes it less of an issue.

On a more general note, let me lurk for awhile and get a feel for what
kinds of bugs might be easily addressable with test cases before we get
into write access or branching decisions.

Bob

Re: emacs test suite

[Jason Rumney]

Robert Anderson <rwa@alumni.princeton.edu> writes:

> > That's cool, but why do you need the branch to be local?
> 
> Well, I don't "need" it to be - this is only a matter of convenience. 
> But it would be nice if this didn't even require discussion or Richard
> having to make a decision apriori about whether my work is credible
> before he has even seen it.  If the branch was local, I would just get
> to work, and everyone could worry about these decisions later when it
> was more than vaporware, saving valuable coordinate effort.

Read the section of the CVS manual about vendor branches. Perhaps
that would give the functionality you want.

Re: emacs test suite

[Kim F. Storm]

Jason Rumney <jasonr@gnu.org> writes:

> Robert Anderson <rwa@alumni.princeton.edu> writes:
> 
> > > That's cool, but why do you need the branch to be local?
> > 
> > Well, I don't "need" it to be - this is only a matter of convenience. 
> > But it would be nice if this didn't even require discussion or Richard
> > having to make a decision apriori about whether my work is credible
> > before he has even seen it.  If the branch was local, I would just get
> > to work, and everyone could worry about these decisions later when it
> > was more than vaporware, saving valuable coordinate effort.
> 
> Read the section of the CVS manual about vendor branches. Perhaps
> that would give the functionality you want.

If we keep the test suite in a separate directory tree, e.g. emacs/test,
I don't see *any* reason to make CVS branches to place the test suite in
CVS; like the emacs/admin directory, we don't have to distribute the
test scripts with emacs, unless we want to do that.

But if we get a substancial amount of tests, I think that advising the
user to run the test suite after building/installing emacs on a new
system would be a good thing.

Something simple like "make test" would be a good interface.

In addition, I believe that all of us write small scripts / sample
code to test and debug various things, so if we had a test/manual
directory below test/, we could put our own informal tests there --
and maybe our "auto-test experts" could pick up that work and
integrate it in the auto tests.

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: emacs test suite

[Eli Zaretskii]

> From: storm@cua.dk (Kim F. Storm)
> Date: 19 Jan 2003 18:55:45 +0100
> 
> Something simple like "make test" would be a good interface.

We already have "make check", a standard GNU target.  It just doesn't
do anything useful yet.