call for more ert tests

call for more ert tests

[Glenn Morris]

There are a lot of bugs in Emacs. [1]
To me, too many of these feel like breakage of things that used to work
before, because of implementation changes, or whatever.

One thing that could help reduce this is more unit tests.
If you haven't used it, ERT makes it pretty easy to write tests.
Of course, many aspects of Emacs's behaviour are not easy to test (GUI
stuff, etc.), but many are. See test/automated/ for examples. [2]

For example, package.el seems like something that should have a test
suite.

So if you fix a bug, please consider adding a unit test to make sure it
does not come back. Or if you rewrite a lisp package, consider adding
tests at the same time to check that obvious functionality still works.

I know writing tests is maybe not as interesting as writing shiny new
features, but I think it will save work in the long run.


[1] E.g. http://debbugs.gnu.org/rrd/emacs.html

[2] I think it is a bit embarrassing that the limited test suite that we
have has been broken for months, though. Bugs 13064, 13662.

Re: call for more ert tests

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Date: Mon, 24 Jun 2013 13:31:50 -0400
> 
> One thing that could help reduce this is more unit tests.
> If you haven't used it, ERT makes it pretty easy to write tests.
> Of course, many aspects of Emacs's behaviour are not easy to test (GUI
> stuff, etc.), but many are. See test/automated/ for examples. [2]
> 
> For example, package.el seems like something that should have a test
> suite.
> 
> So if you fix a bug, please consider adding a unit test to make sure it
> does not come back. Or if you rewrite a lisp package, consider adding
> tests at the same time to check that obvious functionality still works.
> 
> I know writing tests is maybe not as interesting as writing shiny new
> features, but I think it will save work in the long run.

IMO, unless we require every new feature to come with a test and a
report that no regressions were found by running the existing tests,
we will never get any better testability than what we have now.

Re: call for more ert tests

[Lennart Borgman]

[-- Attachment #1: Type: text/plain, Size: 1146 bytes --]

On Mon, Jun 24, 2013 at 8:21 PM, Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Glenn Morris <rgm@gnu.org>
> > Date: Mon, 24 Jun 2013 13:31:50 -0400
> >
> > One thing that could help reduce this is more unit tests.
> > If you haven't used it, ERT makes it pretty easy to write tests.
> > Of course, many aspects of Emacs's behaviour are not easy to test (GUI
> > stuff, etc.), but many are. See test/automated/ for examples. [2]
> >
> > For example, package.el seems like something that should have a test
> > suite.
> >
> > So if you fix a bug, please consider adding a unit test to make sure it
> > does not come back. Or if you rewrite a lisp package, consider adding
> > tests at the same time to check that obvious functionality still works.
> >
> > I know writing tests is maybe not as interesting as writing shiny new
> > features, but I think it will save work in the long run.
>
> IMO, unless we require every new feature to come with a test and a
> report that no regressions were found by running the existing tests,
> we will never get any better testability than what we have now.
>
>
Maybe writing tests for all bugs that shows up?

[-- Attachment #2: Type: text/html, Size: 2104 bytes --]

Re: call for more ert tests

[Andreas Röhler]

Am 24.06.2013 20:24, schrieb Lennart Borgman:
> On Mon, Jun 24, 2013 at 8:21 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>
>>> From: Glenn Morris <rgm@gnu.org>
>>> Date: Mon, 24 Jun 2013 13:31:50 -0400
>>>
>>> One thing that could help reduce this is more unit tests.
>>> If you haven't used it, ERT makes it pretty easy to write tests.
>>> Of course, many aspects of Emacs's behaviour are not easy to test (GUI
>>> stuff, etc.), but many are. See test/automated/ for examples. [2]
>>>
>>> For example, package.el seems like something that should have a test
>>> suite.
>>>
>>> So if you fix a bug, please consider adding a unit test to make sure it
>>> does not come back. Or if you rewrite a lisp package, consider adding
>>> tests at the same time to check that obvious functionality still works.
>>>
>>> I know writing tests is maybe not as interesting as writing shiny new
>>> features, but I think it will save work in the long run.
>>
>> IMO, unless we require every new feature to come with a test and a
>> report that no regressions were found by running the existing tests,
>> we will never get any better testability than what we have now.
>>
>>
> Maybe writing tests for all bugs that shows up?
>

That's good. I'm keeping such a thing for python-mode.el

http://bazaar.launchpad.net/~python-mode-devs/python-mode/python-mode/view/head:/test/py-bug-numbered-tests.el

What's nice about: if some regression occurs, the bug-number leads to the reports and helps fixing.

However, IMO a dual system is needed. Having a test for all bugs will be slow from a certain amount.
Need to condense that again into some more structured.

BTW as all the tests must not be part of Emacs itself, what about dropping the copyright assignment policy, make tests rely on GPL?
AFAIS are a lot of hackers around which might help then.

Andreas

Re: call for more ert tests

[Stefan Merten]

[-- Attachment #1: Type: text/plain, Size: 389 bytes --]

Hi!

4 hours ago Andreas Röhler wrote:
> However, IMO a dual system is needed. Having a test for all bugs will be slow from a certain amount.
> Need to condense that again into some more structured.

You could tag the tests somehow. If there is a set of common tags then
you can easily prevent those tagged 'longrunning' or 'broken' from
running.


						Grüße

						Stefan

[-- Attachment #2: Type: application/pgp-signature, Size: 307 bytes --]

Re: call for more ert tests

[Andreas Röhler]

Am 01.07.2013 18:14, schrieb Stefan Merten:
> Hi!
>
> 4 hours ago Andreas Röhler wrote:
>> However, IMO a dual system is needed. Having a test for all bugs will be slow from a certain amount.
>> Need to condense that again into some more structured.
>
> You could tag the tests somehow. If there is a set of common tags then
> you can easily prevent those tagged 'longrunning' or 'broken' from
> running.
>
>
> 						Grüße
>
> 						Stefan
>

Okay, thanks pointing at.

Let's outline though a little bit the matter: When having a list of tests pointing at bug-reports, several of this tests will have similarities. Maybe some of them might be 
dropped again as redundant. From others a more smart test might derived, aiming at a couple of possible errors at once.
IMHO from a certain level it pays to reflect a more intelligent, abstract test-writing you would do from one bug to another.

Cheers,

Andreas

Re: call for more ert tests

[Eli Zaretskii]

> Date: Mon, 01 Jul 2013 13:35:54 +0200
> From: Andreas Röhler <andreas.roehler@online.de>
> 
> BTW as all the tests must not be part of Emacs itself, what about dropping the copyright assignment policy, make tests rely on GPL?

I can't ever stop fighting the GPL, can you?

Re: call for more ert tests

[Eli Zaretskii]

> Date: Mon, 01 Jul 2013 19:37:26 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> > Date: Mon, 01 Jul 2013 13:35:54 +0200
> > From: Andreas Röhler <andreas.roehler@online.de>
> > 
> > BTW as all the tests must not be part of Emacs itself, what about dropping the copyright assignment policy, make tests rely on GPL?
> 
> I can't ever stop fighting the GPL, can you?
  ^

I meant "you", of course.  Sorry for the typo.

Re: call for more ert tests

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > > I can't ever stop fighting the GPL, can you?
 >   ^
 > 
 > I meant "you", of course.  Sorry for the typo.

Oh, don't apologize for that type.  It was a great WTF? moment!

Re: call for more ert tests

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Date: Tue, 02 Jul 2013 03:44:24 +0900
> Cc: andreas.roehler@online.de, emacs-devel@gnu.org
> 
> Eli Zaretskii writes:
> 
>  > > I can't ever stop fighting the GPL, can you?
>  >   ^
>  > 
>  > I meant "you", of course.  Sorry for the typo.
> 
> Oh, don't apologize for that type.  It was a great WTF? moment!

Yes, for me too.  Some bot on my machine thought it was April 1.

Re: call for more ert tests

[Dmitry Gutov]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Mon, 01 Jul 2013 13:35:54 +0200
>> From: Andreas Röhler <andreas.roehler@online.de>
>> 
>> BTW as all the tests must not be part of Emacs itself, what about dropping the copyright assignment policy, make tests rely on GPL?
>
> You can't ever stop fighting the GPL, can you?

Why do you say that? He's suggesting to rely on GPL, not put it aside.

Which IMO makes sense, allowing us to at least accept tests from random
contributors, even for large features, with no friction.

I'm not sure how useful would that be in the current situation, but in
the event of adopting stricter testing guidelines, it could definitely
help.

Re: call for more ert tests

[Sebastian Wiesner]

2013/6/24 Eli Zaretskii <eliz@gnu.org>:
>> From: Glenn Morris <rgm@gnu.org>
>> Date: Mon, 24 Jun 2013 13:31:50 -0400
>>
>> One thing that could help reduce this is more unit tests.
>> If you haven't used it, ERT makes it pretty easy to write tests.
>> Of course, many aspects of Emacs's behaviour are not easy to test (GUI
>> stuff, etc.), but many are. See test/automated/ for examples. [2]
>>
>> For example, package.el seems like something that should have a test
>> suite.
>>
>> So if you fix a bug, please consider adding a unit test to make sure it
>> does not come back. Or if you rewrite a lisp package, consider adding
>> tests at the same time to check that obvious functionality still works.
>>
>> I know writing tests is maybe not as interesting as writing shiny new
>> features, but I think it will save work in the long run.
>
> IMO, unless we require every new feature to come with a test and a
> report that no regressions were found by running the existing tests,
> we will never get any better testability than what we have now.

Then this is probably a good policy, isn't it?

Re: call for more ert tests

[Eli Zaretskii]

> Date: Mon, 24 Jun 2013 20:33:41 +0200
> From: Sebastian Wiesner <lunaryorn@gmail.com>
> Cc: Glenn Morris <rgm@gnu.org>, emacs-devel@gnu.org
> 
> > IMO, unless we require every new feature to come with a test and a
> > report that no regressions were found by running the existing tests,
> > we will never get any better testability than what we have now.
> 
> Then this is probably a good policy, isn't it?

Which policy?

Re: call for more ert tests

[Sebastian Wiesner]

2013/6/24 Eli Zaretskii <eliz@gnu.org>:
>> Date: Mon, 24 Jun 2013 20:33:41 +0200
>> From: Sebastian Wiesner <lunaryorn@gmail.com>
>> Cc: Glenn Morris <rgm@gnu.org>, emacs-devel@gnu.org
>>
>> > IMO, unless we require every new feature to come with a test and a
>> > report that no regressions were found by running the existing tests,
>> > we will never get any better testability than what we have now.
>>
>> Then this is probably a good policy, isn't it?
>
> Which policy?

Well, requiring tests for new features or fixed bugs, and confirmation
that there are no regressions from the existing tests.

Re: call for more ert tests

[Eli Zaretskii]

> Date: Mon, 24 Jun 2013 20:55:28 +0200
> From: Sebastian Wiesner <lunaryorn@gmail.com>
> Cc: emacs-devel@gnu.org
> 
> 2013/6/24 Eli Zaretskii <eliz@gnu.org>:
> >> Date: Mon, 24 Jun 2013 20:33:41 +0200
> >> From: Sebastian Wiesner <lunaryorn@gmail.com>
> >> Cc: Glenn Morris <rgm@gnu.org>, emacs-devel@gnu.org
> >>
> >> > IMO, unless we require every new feature to come with a test and a
> >> > report that no regressions were found by running the existing tests,
> >> > we will never get any better testability than what we have now.
> >>
> >> Then this is probably a good policy, isn't it?
> >
> > Which policy?
> 
> Well, requiring tests for new features or fixed bugs, and confirmation
> that there are no regressions from the existing tests.

I believe it's good, obviously.  The problem is with introducing it
without losing too many contributors.

Re: call for more ert tests

[Lennart Borgman]

[-- Attachment #1: Type: text/plain, Size: 1187 bytes --]

On Mon, Jun 24, 2013 at 9:16 PM, Eli Zaretskii <eliz@gnu.org> wrote:

> > Date: Mon, 24 Jun 2013 20:55:28 +0200
> > From: Sebastian Wiesner <lunaryorn@gmail.com>
> > Cc: emacs-devel@gnu.org
> >
> > 2013/6/24 Eli Zaretskii <eliz@gnu.org>:
> > >> Date: Mon, 24 Jun 2013 20:33:41 +0200
> > >> From: Sebastian Wiesner <lunaryorn@gmail.com>
> > >> Cc: Glenn Morris <rgm@gnu.org>, emacs-devel@gnu.org
> > >>
> > >> > IMO, unless we require every new feature to come with a test and a
> > >> > report that no regressions were found by running the existing tests,
> > >> > we will never get any better testability than what we have now.
> > >>
> > >> Then this is probably a good policy, isn't it?
> > >
> > > Which policy?
> >
> > Well, requiring tests for new features or fixed bugs, and confirmation
> > that there are no regressions from the existing tests.
>
> I believe it's good, obviously.  The problem is with introducing it
> without losing too many contributors.
>
> Perhaps people will be more willing to write tests when fixing bugs
because then you will nearly immediately see the benefit of it. (Both for
writing the bug fix and assuring that other people do not break it again.)

[-- Attachment #2: Type: text/html, Size: 3633 bytes --]

Re: call for more ert tests

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> Well, requiring tests for new features or fixed bugs, and confirmation
>> that there are no regressions from the existing tests.
>
> I believe it's good, obviously.  The problem is with introducing it
> without losing too many contributors.

I wonder about the possibility of making Emacs testing an "interesting"
task. Something like a test framework which is complete and expressive
enough to allow describing testable conditions on a very high level way.

Having Lisp and an almost unlimited capability for state introspection
looks like a good start.

Re: call for more ert tests

[John Wiegley]

>>>>> Óscar Fuentes <ofv@wanadoo.es> writes:

> I wonder about the possibility of making Emacs testing an "interesting"
> task. Something like a test framework which is complete and expressive
> enough to allow describing testable conditions on a very high level way.

One thing that would help with that is to allow structural pattern matching.
For example, say I have a function `foo` which returns a list '(a b c), but
all I care about is that `b` is `b`.  Then I could write:

    (shouldBe (foo) '(_ b _))

`shouldBe` then raises an exception if the pattern fails to match.

John

Re: call for more ert tests

[Leo Liu]

On 2013-06-25 03:59 +0800, John Wiegley wrote:
> One thing that would help with that is to allow structural pattern matching.
> For example, say I have a function `foo` which returns a list '(a b c), but
> all I care about is that `b` is `b`.  Then I could write:
>
>     (shouldBe (foo) '(_ b _))
>
> `shouldBe` then raises an exception if the pattern fails to match.

Use pcase?

Leo

Re: call for more ert tests

[John Wiegley]

>>>>> Leo Liu <sdl.web@gmail.com> writes:

> On 2013-06-25 03:59 +0800, John Wiegley wrote:
>> One thing that would help with that is to allow structural pattern
>> matching.  For example, say I have a function `foo` which returns a list
>> '(a b c), but all I care about is that `b` is `b`.  Then I could write:
>> 
>> (shouldBe (foo) '(_ b _))
>> 
>> `shouldBe` then raises an exception if the pattern fails to match.

> Use pcase?

pcase does not imply an assertion of equality, which the above does.  I'm not
capturing the second element of the list in a variable named 'b', I'm
asserting that the second element is equal to the symbol 'b'.

John

Re: call for more ert tests

[Stefan Monnier]

>>> One thing that would help with that is to allow structural pattern
>>> matching.  For example, say I have a function `foo` which returns a list
>>> '(a b c), but all I care about is that `b` is `b`.  Then I could write:
>>> 
>>> (shouldBe (foo) '(_ b _))
>>> 
>>> `shouldBe` then raises an exception if the pattern fails to match.

>> Use pcase?

> pcase does not imply an assertion of equality, which the above does.  I'm not
> capturing the second element of the list in a variable named 'b', I'm
> asserting that the second element is equal to the symbol 'b'.

I think he means that it should be easy to make a shouldbe-pcase macro
that provides this kind of behavior using using pcase (the pattern
would look like `(,_ b ,_) instead of the one you wrote).


        Stefan

Re: call for more ert tests

[Stephen J. Turnbull]

Eli Zaretskii <eliz@gnu.org> writes:

 >> Well, requiring tests for new features or fixed bugs, and confirmation
 >> that there are no regressions from the existing tests.
 >
 > I believe it's good, obviously.  The problem is with introducing it
 > without losing too many contributors.

It's not obviously good.  It helped kill Bazaar development, for one
thing.  While that may be an exaggeration, it's clear that the high
standards for getting patches applied discouraged many potential
contributors, and the "patch pilot" program was extremely useful in
getting contribution rates up.  Whether a patch pilot program would be
a good idea for Emacs is a separate question, but note that the core
dev on patch pilot duty spent 25% of their time (full-time employees
of Canonical, most of them) on that task.  Not to mention the time the
contributors spent on meeting the Bazaar quality standards.

Does Emacs really need that high quality?  After trying to get
standards raised at XEmacs (where, being Number 2, we have to try
harder), I've come to the conclusion that for Emacsen, worse is
better.  The factors that led Richard Gabriel to that conclusion are
quite strong in this area of application.  I think rather than
pressure people about testing, code reviews are more important.  Teach
people to habitually write good, maintainable code.  That's the cake,
tests are the icing.

That said, I encourage the core developers to lead by example,
creating tests, even trivial ones, whenever possible.  Today's general
programming culture is very sympathetic to testing, even to the point
of "write tests first".  So new developers will pick up the idea from
what the leaders do.  At XEmacs, regulars (even those not associated
with core development) do submit tests regularly without being asked.
When people post a good replication recipe, we try to put it in the
test suite immediately, crediting the poster, not the committer.  (At
Emacs this might occasionally raise assignment issues, but remember,
it's all for freedom.)  So this practice works.

I also encourage you to get your buildbots green.  It's hard to stay
green in an active project, and as somebody mentioned about the XEmacs
bot for Gnus, some platforms will stay red for a long time because
nobody cares enough.  But I would guess that in XEmacs modal
turnaround for 'bot failures is 3-6 hours, with the tail stretching
out because of sleep and $DAYJOB periods.  (Part of this is because
you get lazy: testing only one platform, or committing "obviously
correct" fixes without testing locally, and the 'bot makes your
mistake obvious.  But laziness gets corrected by social pressure when
there are lots of active committers.)  Once the bots get green,
developers take pride in keeping them that way.

Óscar Fuentes writes:

 > I wonder about the possibility of making Emacs testing an "interesting"
 > task. Something like a test framework which is complete and expressive
 > enough to allow describing testable conditions on a very high level way.

That's going to be a very hard task.  Testing is inherently boring
work, exploring the things that shouldn't happen, and the burden is
heavy, because writing good tests requires good specs, which we rarely
have.  So designing those specs (and maybe even writing them down) is
a hidden burden of testing.

 > Having Lisp and an almost unlimited capability for state introspection
 > looks like a good start.

Lisp is good.  Unlimited state introspection sucks.  It's very easy to
write a test that fixes the implementation in stone when you use
introspection.  Good tests require that you distinguish between
language essentials (externally visible behavior) and implementation
accidents (internal state).  Sometimes you do want to test
intermediate results in an implementation to help debug maintenance
errors on the implementation, but these tests should be marked as
"subject to internal API changes".  This is the kind of thing that
makes testing a hard problem.

John Weigley writes:

 > One thing that would help with that is to allow structural pattern
 > matching.

It helps a little, but as Oscar says, we have Lisp.  Extracting values
from a structure should be part of the API for the structure, and
fixed patterns don't help when fuzzing or iterating over all values in
a sledgehammer test -- constructing the pattern becomes a PITA and
unreadable, even with backquotes.

Also, testing that reasonable input produces correct results (the
common case for such pattern-matchers) isn't so important.  More
important is providing facilities to test safety, that is to express
tests for correct behavior under stress (ie, whether Emacs signals an
erroneous condition, and whether the right signal is raised), and for
helping the buildbot stay green in the presence of bad behavior whose
fix is unknown and stays that way for a while (an ExpectedFailure
condition that people see as a failure when running the test, but the
'bot perceives as a pass == SNAFU).

Re: call for more ert tests

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Date: Tue, 25 Jun 2013 11:31:50 +0900
> Cc: emacs-devel@gnu.org
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
>  >> Well, requiring tests for new features or fixed bugs, and confirmation
>  >> that there are no regressions from the existing tests.
>  >
>  > I believe it's good, obviously.  The problem is with introducing it
>  > without losing too many contributors.
> 
> It's not obviously good.  It helped kill Bazaar development, for one
> thing.

There are opposite examples, like GDB and GCC.  I don't think they are
anywhere near dying.

So clearly other factors are at work here, and they at least sometimes
are stronger than this one, in determining whether a project will
continue its development and get enough contributions.

Re: call for more ert tests

[Lars Magne Ingebrigtsen]

Óscar Fuentes <ofv@wanadoo.es> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>>
>> I believe it's good, obviously.  The problem is with introducing it
>> without losing too many contributors.

I think the bar to contributing to Emacs is high enough as it is without
adding further requirements.

> I wonder about the possibility of making Emacs testing an "interesting"
> task. Something like a test framework which is complete and expressive
> enough to allow describing testable conditions on a very high level way.

ert is fine, but, I think, somewhat misguided.  It allows us to test the
functions we have Lisp interfaces for, but not the deep internal C
bits.  And that's kinda backward.

When I write Lisp code, I'm testing it interactively all the time.  What
should this function return?  Does it return what I'm expecting?  No?
*hack hack*  Now?  Yes.  Done.

You really need a test harness more for languages where you don't have
that extremely rapid and natural write-test-write cycle, because nobody
can test stuff that requires a compile (or startup) cycle as
thoroughly as that.  For instance, when I wrote the
:max-width/:max-height code the other day, the result depends on six
different things: image width/height, and the four
:width/:height/:max-width/:max-height parameters.

If I could have called that function from Lisp directly, I would have
tested a much larger variety of combinations than I ended up doing.

So my feeling about adding more ert tests is: Meh.  More work to write,
more work to maintain, doesn't really give us that much.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: call for more ert tests

[Juanma Barranquero]

On Tue, Jun 25, 2013 at 1:06 PM, Lars Magne Ingebrigtsen <larsi@gnus.org> wrote:

> I think the bar to contributing to Emacs is high enough as it is without
> adding further requirements.

Agreed.

> ert is fine, but, I think, somewhat misguided.  It allows us to test the
> functions we have Lisp interfaces for, but not the deep internal C
> bits.  And that's kinda backward.

Tests for C internals would be nice, yes.

> When I write Lisp code, I'm testing it interactively all the time.  What
> should this function return?  Does it return what I'm expecting?  No?
> *hack hack*  Now?  Yes.  Done.

Tests as specifications are great, but they do not cease to be useful
once the code is ready. Regression tests exist because it is easy to
break unexpected things with apparently unrelated changes. Nobody can
test every possible outcome by hand. So if you spend the time writing
the test, someone will likely benefit from it in the long run.

> If I could have called that function from Lisp directly, I would have
> tested a much larger variety of combinations than I ended up doing.

You could have written a Lisp API for it. Even if you #ifdef 0
afterwards, it wouldn't be time wasted IMHO.

> So my feeling about adding more ert tests is: Meh.  More work to write,
> more work to maintain, doesn't really give us that much.

With this I disagree.

   J

Re: call for more ert tests

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Date: Tue, 25 Jun 2013 13:06:18 +0200
> 
> > Eli Zaretskii <eliz@gnu.org> writes:
> >>
> >> I believe it's good, obviously.  The problem is with introducing it
> >> without losing too many contributors.
> 
> I think the bar to contributing to Emacs is high enough as it is without
> adding further requirements.

I don't know why you are saying this.  Please elaborate by comparing
with other projects whose bar is significantly lower.

For me, the single most important issue for contributing is how easy
or hard is it to find the reason for the problem(s) that cause my
itch, and how easy or hard is it to make a change that I can convince
myself is right.  I can tell you that in my experience, Emacs is not
much harder in this regard than other projects, like GDB or Guile (to
pick just 2 random examples).

In any case, I didn't say that we actually should require tests as
part of any contribution.  All I said was that if we don't, the
chances of us getting anywhere with the test suite are extremely low.
In fact, I believe that only an appearance of a very dedicated person
who would do the job is the only alternative to a strict policy of
requiring tests with each changeset.  How probable is that such an
alternative materializes any time soon, in your opinion?

> ert is fine, but, I think, somewhat misguided.  It allows us to test the
> functions we have Lisp interfaces for, but not the deep internal C
> bits.  And that's kinda backward.
> 
> When I write Lisp code, I'm testing it interactively all the time.  What
> should this function return?  Does it return what I'm expecting?  No?
> *hack hack*  Now?  Yes.  Done.

I agree that testing C-level internals is a must.  But most of Emacs
is Lisp code, and if you examine the bug reports, I think you will
find that most of them are about Lisp code.  So testing Lisp is also
important.  It's not just the API that is being tested; it's the
behavior under different inputs and user options; in short, how well
the API keeps its side of the contract.

Re: call for more ert tests

[Lars Magne Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> I think the bar to contributing to Emacs is high enough as it is without
>> adding further requirements.
>
> I don't know why you are saying this.  Please elaborate by comparing
> with other projects whose bar is significantly lower.

You want names of projects that have a lower bar for contributions?  Or
what it is that makes the bar higher for Emacs than with these other
projects?

1) The Linux kernel.  2) Both the Emacs Lisp language and the
sorta-kinda C layer are novel ideas for most novices; the copyright
assignment paperwork; the unfamiliar VC; the many unfamiliar concepts
that Emacs has compared to most other projects (buffers, encodings); the
interaction between the Lisp bits and the C bits; the way the C bits
aren't compiled with -Wall, so you get little help from the compiler...
You may be used to all the Emacs internals, but poking around in Emacs
is pretty daunting compared to simple stuff like the Linux kernel.

> In fact, I believe that only an appearance of a very dedicated person
> who would do the job is the only alternative to a strict policy of
> requiring tests with each changeset.  How probable is that such an
> alternative materializes any time soon, in your opinion?

Unlikely, and I don't favour a huge set of automated tests, anyway, so
I'm happy about that.

If I decide that shr should render <li> differently than I originally
thought it should, I'm very happy that I don't have a slew of unit tests
for the HTML rendering.  Because then I'm done after changing the <li>
rendering, and I don't have to touch up however many ert tests that this
affects.

None of us have unlimited time to spend on this stuff, and what doesn't
help us doesn't help us, no matter what the current orthodoxies
surrounding testing says.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: call for more ert tests

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Date: Tue, 25 Jun 2013 21:18:48 +0200
> Cc: emacs-devel@gnu.org
> 
> 1) The Linux kernel.  2) Both the Emacs Lisp language and the
> sorta-kinda C layer are novel ideas for most novices; the copyright
> assignment paperwork; the unfamiliar VC; the many unfamiliar concepts
> that Emacs has compared to most other projects (buffers, encodings); the
> interaction between the Lisp bits and the C bits; the way the C bits
> aren't compiled with -Wall, so you get little help from the compiler...
> You may be used to all the Emacs internals, but poking around in Emacs
> is pretty daunting compared to simple stuff like the Linux kernel.

I'm surprised you can say that an OS kernel is easier than Emacs.  As
for an extension language, many packages have that.

Bottom line, I'm not convinced.

Re: call for more ert tests

[Sebastian Wiesner]

2013/6/25 Lars Magne Ingebrigtsen <larsi@gnus.org>:
> Eli Zaretskii <eliz@gnu.org> writes:
>> In fact, I believe that only an appearance of a very dedicated person
>> who would do the job is the only alternative to a strict policy of
>> requiring tests with each changeset.  How probable is that such an
>> alternative materializes any time soon, in your opinion?
>
> Unlikely, and I don't favour a huge set of automated tests, anyway, so
> I'm happy about that.
>
> If I decide that shr should render <li> differently than I originally
> thought it should, I'm very happy that I don't have a slew of unit tests
> for the HTML rendering.  Because then I'm done after changing the <li>
> rendering, and I don't have to touch up however many ert tests that this
> affects.

Actually, you aren't done… you are just out-sourcing your work, i.e.
regression testing, to your users.  In the absence of tests, you will
never realize that your quick change of "li" rendering accidentally
broke "a" rendering until the first user complains and sends you back
to the keyboard in order to debug the bug which you introduced.

Of couse, to you as a developer it's easier to just use your users as
testers, but that's probably not fair.  I as a user of your package
wouldn't like it… which is why I write tests for my own Elisp
packages.

Re: call for more ert tests

[Lars Magne Ingebrigtsen]

Sebastian Wiesner <lunaryorn@gmail.com> writes:

> Actually, you aren't done… you are just out-sourcing your work, i.e.
> regression testing, to your users.  In the absence of tests, you will
> never realize that your quick change of "li" rendering accidentally
> broke "a" rendering until the first user complains and sends you back
> to the keyboard in order to debug the bug which you introduced.

Sounds unlikely.  When I fix a the rendering, the page is there right in
the buffer as I'm coding, and <a> mess-ups would be pretty immediately
obvious.

And if I miss it, somebody will tell me, as you say.

So in this scenario, having to fiddle with the test cases to make the
change "pass" means more work for me, for a very unlikely gain, which
(in the unlikely case of me making an error!!!) would be discovered by
somebody else.

So there's nothing to gain here for the Emacs developers.

And I think that people writing other highly interactive and non-vital
stuff has started to find this out in general the last few years.  On
Hacker News there's constantly people popping up who've seen the light
-- having too many automatic tests means ossifying the software.  The
TDD nightmare is almost over now, and Emacs survived without gaining too
many useless automatic tests while the fad lasted.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: call for more ert tests

[Ted Zlatanov]

On Tue, 25 Jun 2013 22:44:48 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> And I think that people writing other highly interactive and non-vital
LMI> stuff has started to find this out in general the last few years.  On
LMI> Hacker News there's constantly people popping up who've seen the light
LMI> -- having too many automatic tests means ossifying the software.

In effect writing tests is like buying insurance.  Not everyone wants
it, not everyone needs it.  Flood insurance for a mountaintop house is
probably a bad investment, but in a flood zone it makes sense.

I think it should be up to the developer to decide if the cost of tests
is justified for their package.  For the Emacs core, the maintainers
have to make that decision as part of accepting a contribution.

Ted

Re: call for more ert tests

[Juanma Barranquero]

[OFFTOPICLY OFFTOPIC]

On Fri, Jun 28, 2013 at 5:01 PM, Ted Zlatanov <tzz@lifelogs.com> wrote:

> Flood insurance for a mountaintop house is
> probably a bad investment, but in a flood zone it makes sense.

A friend of mine once had her home flooded, and it was a flat at floor 12.

The guy in the flat just over hers left the windows fully open in the
middle of a really big tropical storm (not a hurricane, just lots &
lots of water). That was in São Paulo, and having lived there for a
while, I have every reason to believe my friends' account.

   J

Re: call for more ert tests

[Dmitry Gutov]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Tue, 25 Jun 2013 22:44:48 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 
>
> LMI> And I think that people writing other highly interactive and non-vital
> LMI> stuff has started to find this out in general the last few years.  On
> LMI> Hacker News there's constantly people popping up who've seen the light
> LMI> -- having too many automatic tests means ossifying the software.

Writing lots of poor (e.g. implementation-dependent) test will do that
for you. That doesn't mean that any such application won't benefit from
having at least some tests.

> In effect writing tests is like buying insurance.  Not everyone wants
> it, not everyone needs it.  Flood insurance for a mountaintop house is
> probably a bad investment, but in a flood zone it makes sense.

If you go along with this analogy to pick the kinds of tests to write
and pieces of code to explicitly test, I'll agree.

But automated testing in general can be useful for most kinds of
projects, at least to the degree that it replaces the need for manual
testing of nuts and bolts in that new feature you're writing.

> For the Emacs core, the maintainers
> have to make that decision as part of accepting a contribution.

It would be nice to annouce such a policy. Want this complex,
non-essential feature? Tests, please!

Re: call for more ert tests

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 659 bytes --]

On Tue, Jun 25 2013, Sebastian Wiesner wrote:

> Actually, you aren't done… you are just out-sourcing your work, i.e.
> regression testing, to your users.  In the absence of tests, you will
> never realize that your quick change of "li" rendering accidentally
> broke "a" rendering until the first user complains and sends you back
> to the keyboard in order to debug the bug which you introduced.

+1 billions

As for Emacs, anybody who tried to contribute to something like org-mode
can experience that and the pain going with it very easily.

-- 
Julien Danjou
;; Free Software hacker ; freelance consultant
;; http://julien.danjou.info

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 835 bytes --]

Re: call for more ert tests

[Stephen J. Turnbull]

Lars Magne Ingebrigtsen writes:

 > If I decide that shr should render <li> differently than I
 > originally thought it should, I'm very happy that I don't have a
 > slew of unit tests for the HTML rendering.  Because then I'm done
 > after changing the <li> rendering, and I don't have to touch up
 > however many ert tests that this affects.

This is a terrible example, of course, because modern specs for HTML
emphasize that rendering isn't defined by the HTML spec.  Pixel-
perfect rendering isn't an invariant if you're only interested in
ensuring that the semantic content is conveyed, and so shouldn't be
tested.  OTOH, I think it's quite reasonable to ask that in
"reasonably wide" windows, text not be rendered outside of the window
boundaries, and I would be happy to have a test for that.  YMMV.

 > None of us have unlimited time to spend on this stuff, and what
 > doesn't help us doesn't help us, no matter what the current
 > orthodoxies surrounding testing says.

Ah, "orthodoxy".  The current orthodoxies (or some of them, anyway)
are reasonable disciplines for novice developers.  It's unfortunate
that many teachers engrave them in stone, but that's not what Eli is
suggesting.

Re: call for more ert tests

[Glenn Morris]

Eli Zaretskii wrote:

> IMO, unless we require every new feature to come with a test and a
> report that no regressions were found by running the existing tests,
> we will never get any better testability than what we have now.

I think that is a tough goal, and also somewhat pessimistic ("unless we
force everybody to be in complete compliance with X, we will never get
any more X"). Maybe we could try to move gradually in this direction
until it naturally becomes the accepted convention? But maybe you are
right and incremental improvement is impossible. I guess we will see...

Re: call for more ert tests

[Noah Lavine]

[-- Attachment #1: Type: text/plain, Size: 1221 bytes --]

I'm afraid I probably won't have time to do this myself, but one idea I've
heard is to use Emacs' interactivity to make testing easier.

For instance, you could code in a split-window view where one window has
your code and another has test results that are constantly updated to match
the code.

It would be even better if Emacs could figure out how to test on
"representative" inputs itself, but I can't figure out how to do that. It
seems like quite a hard problem in general, so it would probably on work in
some cases.

Best,
Noah Lavine


On Mon, Jun 24, 2013 at 12:46 PM, Glenn Morris <rgm@gnu.org> wrote:

> Eli Zaretskii wrote:
>
> > IMO, unless we require every new feature to come with a test and a
> > report that no regressions were found by running the existing tests,
> > we will never get any better testability than what we have now.
>
> I think that is a tough goal, and also somewhat pessimistic ("unless we
> force everybody to be in complete compliance with X, we will never get
> any more X"). Maybe we could try to move gradually in this direction
> until it naturally becomes the accepted convention? But maybe you are
> right and incremental improvement is impossible. I guess we will see...
>
>

[-- Attachment #2: Type: text/html, Size: 1677 bytes --]

Re: call for more ert tests

[Rüdiger Sonderfeld]

On Monday 24 June 2013 21:21:50 Eli Zaretskii wrote:
> IMO, unless we require every new feature to come with a test and a
> report that no regressions were found by running the existing tests,
> we will never get any better testability than what we have now.

I think one problem is that it's hard to write tests for many Emacs features 
and the results might not really help to improve the code quality.  E.g., the 
new browser eww.  There could certainly be some tests for lower level 
features.  But a real test coverage would require tests for the rendered 
layout.  Which would be hard to write and even worse probably breaking all the 
time when rendering is improved and thus the output changes.

Therefore having a policy to demand tests for new features might be counter 
productive.  On the other hand maintainers should at least ask for test cases 
when they think a new feature could benefit from tests.  Which in my 
experience, as somebody who is relatively new to contributing to Emacs, does 
not seem to happen at all.

An automated test report would certainly improve things.  The Gnus folks seem 
to use http://buildbot.net/.

Regards
Rüdiger

Re: call for more ert tests

[Eli Zaretskii]

> From: Rüdiger Sonderfeld <ruediger@c-plusplus.de>
> Cc: Glenn Morris <rgm@gnu.org>
> Date: Tue, 25 Jun 2013 19:18:25 +0200
> 
> But a real test coverage would require tests for the rendered 
> layout.  Which would be hard to write and even worse probably breaking all the 
> time when rendering is improved and thus the output changes.

The fact that the "correct" results change with development is a fact
of life in every project, not just in Emacs.  It is detected very
simply, when the test fails.  Then you analyze the failure, understand
that the expected results are wrong, and modify them to follow suit.
Case closed.

> Therefore having a policy to demand tests for new features might be counter 
> productive.  On the other hand maintainers should at least ask for test cases 
> when they think a new feature could benefit from tests.  Which in my 
> experience, as somebody who is relatively new to contributing to Emacs, does 
> not seem to happen at all.

If you don't have good test coverage, you cannot regress-test, so you
can never be sure that a change you just made doesn't break something.
It's as simple as that, and I'm sure you know it.

Re: call for more ert tests

[Dmitry Gutov]

Eli Zaretskii <eliz@gnu.org> writes:

> The fact that the "correct" results change with development is a fact
> of life in every project, not just in Emacs.  It is detected very
> simply, when the test fails.  Then you analyze the failure, understand
> that the expected results are wrong, and modify them to follow suit.
> Case closed.

> If you don't have good test coverage, you cannot regress-test, so you
> can never be sure that a change you just made doesn't break something.
> It's as simple as that, and I'm sure you know it.

+1 on both counts.

I do believe that some packages and pieces of code may be exempt from
requiring tests, like code affecting how things look (don't need to go
far for examples), and select new packages with active maintainers,
where the code and features change a lot.

Requiring tests to accompany most "functional" bugfixes would be a solid
principle, though.

But anyway, all of that comes second to having an unbroken suite of
current tests, working integration server and email notifications for
committers who break tests.

Re: call for more ert tests

[Andreas Röhler]

Am 24.06.2013 20:21, schrieb Eli Zaretskii:
>> From: Glenn Morris <rgm@gnu.org>
>> Date: Mon, 24 Jun 2013 13:31:50 -0400
>>
>> One thing that could help reduce this is more unit tests.
>> If you haven't used it, ERT makes it pretty easy to write tests.
>> Of course, many aspects of Emacs's behaviour are not easy to test (GUI
>> stuff, etc.), but many are. See test/automated/ for examples. [2]
>>
>> For example, package.el seems like something that should have a test
>> suite.
>>
>> So if you fix a bug, please consider adding a unit test to make sure it
>> does not come back. Or if you rewrite a lisp package, consider adding
>> tests at the same time to check that obvious functionality still works.
>>
>> I know writing tests is maybe not as interesting as writing shiny new
>> features, but I think it will save work in the long run.
>
> IMO, unless we require every new feature to come with a test and a
> report that no regressions were found by running the existing tests,
> we will never get any better testability than what we have now.
>
>

Maybe a new feature must not come with a test the very day, but expect it next - before any other new feature.

Also some core-tests should be obligatory for all code checked in.
Just no check-ins before these tests are passed.

Re: call for more ert tests

[Ted Zlatanov]

On Mon, 01 Jul 2013 13:45:25 +0200 Andreas Röhler <andreas.roehler@online.de> wrote: 

AR> Also some core-tests should be obligatory for all code checked in.
AR> Just no check-ins before these tests are passed.

It should be up to the maintainers to decide.  Let's not try to force
these strict rules on the Emacs developer community.

Ted

Re: call for more ert tests

[Andreas Röhler]

Am 01.07.2013 14:43, schrieb Ted Zlatanov:
> On Mon, 01 Jul 2013 13:45:25 +0200 Andreas Röhler <andreas.roehler@online.de> wrote:
>
> AR> Also some core-tests should be obligatory for all code checked in.
> AR> Just no check-ins before these tests are passed.
>
> It should be up to the maintainers to decide.  Let's not try to force
> these strict rules on the Emacs developer community.
>
> Ted
>
>
>

It's just a suggestion. From my own experience and also if you make aware all the bug-reports resulting from just a typo.
The one who thinks: "such a trivial change might never fail" might save time for the moment - it will be paid many-fold by others.
BTW the time I checked in something without running tests are long ago - might report a long list of surprises instead.
But must confess: seldom all tests pass ;)

Cheers,

Andreas

Re: call for more ert tests

[David Engster]

Glenn Morris writes:
> [2] I think it is a bit embarrassing that the limited test suite that we
> have has been broken for months, though. Bugs 13064, 13662.

Couldn't we run the test suite automatically on the Hydra server, so
that this gets reported as a broken build?

-David

Re: call for more ert tests

[Glenn Morris]

David Engster wrote:

> Glenn Morris writes:
>> [2] I think it is a bit embarrassing that the limited test suite that we
>> have has been broken for months, though. Bugs 13064, 13662.
>
> Couldn't we run the test suite automatically on the Hydra server, so
> that this gets reported as a broken build?

I guess so. But since we already know the testsuite is broken (and
haven't done anything about fixing it), we would not immediately gain
anything by this.

Also, hydra builds has been completely broken for weeks. I'm trying to
get it fixed, but it's not clear what the problem is.

http://lists.gnu.org/archive/html/hydra-users/2013-06/msg00014.html

And before it broke we had 4/7 failing builds for a long time, and
no-one seemed very motivated to get them fixed. True, the only real
Emacs issue is the x86_64-darwin one (the rest are hydra problems that
it is hard for us to fix), but I think it reflects the fact that we
don't really seem to care about unit tests that much.

Re: call for more ert tests

[David Engster]

Glenn Morris writes:
> David Engster wrote:
>> Glenn Morris writes:
>>> [2] I think it is a bit embarrassing that the limited test suite that we
>>> have has been broken for months, though. Bugs 13064, 13662.
>>
>> Couldn't we run the test suite automatically on the Hydra server, so
>> that this gets reported as a broken build?
>
> I guess so. But since we already know the testsuite is broken (and
> haven't done anything about fixing it), we would not immediately gain
> anything by this.

Not immediately, no. But at worst, you can still deactivate the broken
tests, so that you will at least notice new regressions quickly. It's
better than nothing.

> Also, hydra builds has been completely broken for weeks. I'm trying to
> get it fixed, but it's not clear what the problem is.
>
> http://lists.gnu.org/archive/html/hydra-users/2013-06/msg00014.html
>
> And before it broke we had 4/7 failing builds for a long time, and
> no-one seemed very motivated to get them fixed. True, the only real
> Emacs issue is the x86_64-darwin one (the rest are hydra problems that
> it is hard for us to fix), but I think it reflects the fact that we
> don't really seem to care about unit tests that much.

Yes, if these things do not get fixed right away, they tend to stay
broken for a long time, especially when they affect other platforms like
Darwin. It's the same thing on the Gnus Buildbot, where the XEmacs
builds are often broken for weeks, because no one cares enough to fix
them.

However, the Gnus Buildbot does at least send a mail to the one who
committed the breakage. It'd be nice if Hydra could do that as well
(when it's working again, that is).

BTW, if the Hydra things turns out to be not fixable, my offer still
stands to set up a Buildbot for Emacs, like I did for CEDET and Gnus,
but someone would have to donate space on a server with the necessary
horsepower.

-David

Re: call for more ert tests

[Daniel Hackney]

Glenn Morris <rgm <at> gnu.org> writes:
> For example, package.el seems like something that should have a test
> suite.

Funny you should mention package.el. My big refactoring of package.el comes 
with a whole pile of ERT-based tests!

Re: call for more ert tests

[Stefan Merten]

[-- Attachment #1: Type: text/plain, Size: 4301 bytes --]

Hi all!

Being a relatively new Emacs developer my impressions might be
helpful for this debate. I'm subscribed to all(?) lists but usually
check only the subject lines for things interesting me. Also I'm
maintaining rst.el and so far did not do anything else in Emacs.

2 days ago Glenn Morris wrote:
> There are a lot of bugs in Emacs. [1]
> To me, too many of these feel like breakage of things that used to work
> before, because of implementation changes, or whatever.

I share this impression. And I think it's a shame that this is so.

> One thing that could help reduce this is more unit tests.

Definitely. Together with a build bot running tests automatically.

> If you haven't used it, ERT makes it pretty easy to write tests.

Some time ago I started to care about rst.el really. I wanted to do a
lot of changes in code I did not wrote myself and today this is
nothing to do without good test support. I found ert and it did most
of what I needed. It definitely is a very good foundation.

> Of course, many aspects of Emacs's behaviour are not easy to test

In rst.el I tested a lot of functionality which operated on buffers.
Thus I wanted to know that a buffer contained a certain content after
running a function. I wrote a little extension based on ert which
allowed such tests easily. I even suggested this code here but the ert
maintainer did not like it and suggested a different approach I never
checked out.

I think this can be done in a more generalized way. For instance the
next thing I was thinking of was to check for correct fontification -
since this is one of the major features in rst.el. I already thought
about ways of extending ert for this.

IMHO ert is a perfect foundation but needs extensions to test standard
cases where you are not (only) interested in the return values of a
function call.

> (GUI
> stuff, etc.), but many are. See test/automated/ for examples. [2]

Indeed they are. Often when I browse through the documented API I
wonder why there are no unit tests for it.

> I know writing tests is maybe not as interesting as writing shiny new
> features, but I think it will save work in the long run.

Well, I love writing tests. May be there are more like me?

Yesterday Stephen J Turnbull wrote:
> Does Emacs really need that high quality?

Definitely. Emacs is probably the oldest Free Software project and
IMHO it's a shame that the traffic on the development list is that
high because of bugs. IMHO a good part of this could be prevented by
suitable unit tests. Interactive testing is nice but definitely not
enough.

> That said, I encourage the core developers to lead by example,
> creating tests, even trivial ones, whenever possible.

One obvious task would be to have unit tests for the API specified in
the info documentation. There are also good chances that the
documentation can be improved by this.

> Today's general
> programming culture is very sympathetic to testing, even to the point
> of "write tests first".

Definitely. Frankly speaking I was shocked when I entered the Emacs
community and saw that in the end there are (close to) *no* tests.

> Testing is inherently boring
> work, exploring the things that shouldn't happen, and the burden is
> heavy, because writing good tests requires good specs, which we rarely
> have.  So designing those specs (and maybe even writing them down) is
> a hidden burden of testing.

I disagree that testing is boring. To me it's interesting to catch all
those corner cases in tests and improve the specification or even the
implementation as a side effect. May be this is a tester approach
(which I'm not ;-) ).

Yesterday Lars Magne Ingebrigtsen wrote:
> ert is fine, but, I think, somewhat misguided.  It allows us to test the
> functions we have Lisp interfaces for, but not the deep internal C
> bits.  And that's kinda backward.

May be I do not understand Lars' intentions here, but what is testing
the C code good for at all? I mean there is an API - or rather: lots
of APIs - and *this* is what needs testing. The C code to me looks
like an implementation detail I'd not write tests for. Or at least
separate those tests clearly from the API tests.


						Grüße

						Stefan

[-- Attachment #2: Type: application/pgp-signature, Size: 307 bytes --]

Re: call for more ert tests

[Bozhidar Batsov]

[-- Attachment #1: Type: text/plain, Size: 5401 bytes --]

Hi guys,  

Here's my two cents - having tests is good as long as there is someone to constantly run them. I think it'd be great of Emacs started running tests automatically after each commit with the help of some integration server (like Travis, Jenkins or something similar). It'd be easier to spot problems that way and people would be aided in developing some good programming habits.  

On a related note at some point we might want to start using an issue tracker alongside the mailing list. Seems a bit odd to attach everyone to each discussion - some discussions should be of interest to everyone indeed, but most - won't be. Issue trackers help to keep discussions focused.  

--  
Cheers,
Bozhidar


On Wednesday, June 26, 2013 at 12:22 PM, Stefan Merten wrote:

> Hi all!
>  
> Being a relatively new Emacs developer my impressions might be
> helpful for this debate. I'm subscribed to all(?) lists but usually
> check only the subject lines for things interesting me. Also I'm
> maintaining rst.el and so far did not do anything else in Emacs.
>  
> 2 days ago Glenn Morris wrote:
> > There are a lot of bugs in Emacs. [1]
> > To me, too many of these feel like breakage of things that used to work
> > before, because of implementation changes, or whatever.
> >  
>  
>  
> I share this impression. And I think it's a shame that this is so.
>  
> > One thing that could help reduce this is more unit tests.
>  
> Definitely. Together with a build bot running tests automatically.
>  
> > If you haven't used it, ERT makes it pretty easy to write tests.
>  
> Some time ago I started to care about rst.el really. I wanted to do a
> lot of changes in code I did not wrote myself and today this is
> nothing to do without good test support. I found ert and it did most
> of what I needed. It definitely is a very good foundation.
>  
> > Of course, many aspects of Emacs's behaviour are not easy to test
>  
> In rst.el I tested a lot of functionality which operated on buffers.
> Thus I wanted to know that a buffer contained a certain content after
> running a function. I wrote a little extension based on ert which
> allowed such tests easily. I even suggested this code here but the ert
> maintainer did not like it and suggested a different approach I never
> checked out.
>  
> I think this can be done in a more generalized way. For instance the
> next thing I was thinking of was to check for correct fontification -
> since this is one of the major features in rst.el. I already thought
> about ways of extending ert for this.
>  
> IMHO ert is a perfect foundation but needs extensions to test standard
> cases where you are not (only) interested in the return values of a
> function call.
>  
> > (GUI
> > stuff, etc.), but many are. See test/automated/ for examples. [2]
> >  
>  
>  
> Indeed they are. Often when I browse through the documented API I
> wonder why there are no unit tests for it.
>  
> > I know writing tests is maybe not as interesting as writing shiny new
> > features, but I think it will save work in the long run.
> >  
>  
>  
> Well, I love writing tests. May be there are more like me?
>  
> Yesterday Stephen J Turnbull wrote:
> > Does Emacs really need that high quality?
>  
>  
> Definitely. Emacs is probably the oldest Free Software project and
> IMHO it's a shame that the traffic on the development list is that
> high because of bugs. IMHO a good part of this could be prevented by
> suitable unit tests. Interactive testing is nice but definitely not
> enough.
>  
> > That said, I encourage the core developers to lead by example,
> > creating tests, even trivial ones, whenever possible.
> >  
>  
>  
> One obvious task would be to have unit tests for the API specified in
> the info documentation. There are also good chances that the
> documentation can be improved by this.
>  
> > Today's general
> > programming culture is very sympathetic to testing, even to the point
> > of "write tests first".
> >  
>  
>  
> Definitely. Frankly speaking I was shocked when I entered the Emacs
> community and saw that in the end there are (close to) *no* tests.
>  
> > Testing is inherently boring
> > work, exploring the things that shouldn't happen, and the burden is
> > heavy, because writing good tests requires good specs, which we rarely
> > have. So designing those specs (and maybe even writing them down) is
> > a hidden burden of testing.
> >  
>  
>  
> I disagree that testing is boring. To me it's interesting to catch all
> those corner cases in tests and improve the specification or even the
> implementation as a side effect. May be this is a tester approach
> (which I'm not ;-) ).
>  
> Yesterday Lars Magne Ingebrigtsen wrote:
> > ert is fine, but, I think, somewhat misguided. It allows us to test the
> > functions we have Lisp interfaces for, but not the deep internal C
> > bits. And that's kinda backward.
> >  
>  
>  
> May be I do not understand Lars' intentions here, but what is testing
> the C code good for at all? I mean there is an API - or rather: lots
> of APIs - and *this* is what needs testing. The C code to me looks
> like an implementation detail I'd not write tests for. Or at least
> separate those tests clearly from the API tests.
>  
>  
> Grüße
>  
> Stefan  


[-- Attachment #2: Type: text/html, Size: 7362 bytes --]

Re: call for more ert tests

[Eli Zaretskii]

> Date: Wed, 26 Jun 2013 15:17:57 +0300
> From: Bozhidar Batsov <bozhidar@batsov.com>
> Cc: emacs-devel@gnu.org
> 
> Here's my two cents - having tests is good as long as there is someone to constantly run them. I think it'd be great of Emacs started running tests automatically after each commit with the help of some integration server (like Travis, Jenkins or something similar). It'd be easier to spot problems that way and people would be aided in developing some good programming habits.  

You cannot run tests that aren't there, can you?

> On a related note at some point we might want to start using an issue tracker alongside the mailing list.

We already have an issue tracker, see http://debbugs.gnu.org/.

Re: call for more ert tests

[Bozhidar Batsov]

[-- Attachment #1: Type: text/plain, Size: 1124 bytes --]

On Wednesday, June 26, 2013 at 6:52 PM, Eli Zaretskii wrote:
> > Date: Wed, 26 Jun 2013 15:17:57 +0300
> > From: Bozhidar Batsov <bozhidar@batsov.com (mailto:bozhidar@batsov.com)>
> > Cc: emacs-devel@gnu.org (mailto:emacs-devel@gnu.org)
> > 
> > Here's my two cents - having tests is good as long as there is someone to constantly run them. I think it'd be great of Emacs started running tests automatically after each commit with the help of some integration server (like Travis, Jenkins or something similar). It'd be easier to spot problems that way and people would be aided in developing some good programming habits. 
> 
> You cannot run tests that aren't there, can you?
Indeed you cannot. 
> 
> > On a related note at some point we might want to start using an issue tracker alongside the mailing list.
> 
> We already have an issue tracker, see http://debbugs.gnu.org/. 
I'm aware of its existence, but it's kind of spartan and not particularly easy to navigate/search/etc. It looks to me like a web interface to a bug mailing list (which I guess it is). I was thinking of something more like bugzilla, trac, etc. 

[-- Attachment #2: Type: text/html, Size: 2072 bytes --]

Re: call for more ert tests

[Eli Zaretskii]

> Date: Wed, 26 Jun 2013 19:03:42 +0300
> From: Bozhidar Batsov <bozhidar@batsov.com>
> Cc: smerten@oekonux.de, emacs-devel@gnu.org
> 
> > We already have an issue tracker, see http://debbugs.gnu.org/. 
> I'm aware of its existence, but it's kind of spartan and not particularly easy to navigate/search/etc.

To have decent search facilities, use
http://debbugs.gnu.org/cgi/search.cgi and not the "simple" search
higher on the page.

What hides behind "etc."?

Re: call for more ert tests

[Dmitry Gutov]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Wed, 26 Jun 2013 19:03:42 +0300
>> From: Bozhidar Batsov <bozhidar@batsov.com>
>> Cc: smerten@oekonux.de, emacs-devel@gnu.org
>> 
>> > We already have an issue tracker, see http://debbugs.gnu.org/. 
>> I'm aware of its existence, but it's kind of spartan and not particularly easy to navigate/search/etc.
>
> To have decent search facilities, use
> http://debbugs.gnu.org/cgi/search.cgi and not the "simple" search
> higher on the page.
>
> What hides behind "etc."?

My answer would be: being able to comment, tag and close the issues
without having to use an email client and look up the exact way you do
those things via inline text and special email addresses.

These things are not hard when you know them, but for a casual user,
debbugs is quite intimidating.

Re: call for more ert tests

[Michael Albinus]

Dmitry Gutov <dgutov@yandex.ru> writes:

>> To have decent search facilities, use
>> http://debbugs.gnu.org/cgi/search.cgi and not the "simple" search
>> higher on the page.
>>
>> What hides behind "etc."?
>
> My answer would be: being able to comment, tag and close the issues
> without having to use an email client and look up the exact way you do
> those things via inline text and special email addresses.
>
> These things are not hard when you know them, but for a casual user,
> debbugs is quite intimidating.

There is the debbugs package from the ELPA archive.

Best regards, Michael.

Re: call for more ert tests

[Dmitry Gutov]

On 26.06.2013 23:10, Michael Albinus wrote:
> There is the debbugs package from the ELPA archive.

Thanks, I'm aware of it. But it's not something a user would know about 
from looking at e.g. http://debbugs.gnu.org/cgi/bugreport.cgi?bug=12345

Re: call for more ert tests

[Michael Albinus]

Dmitry Gutov <dgutov@yandex.ru> writes:

>> There is the debbugs package from the ELPA archive.
>
> Thanks, I'm aware of it. But it's not something a user would know
> about from looking at
> e.g. http://debbugs.gnu.org/cgi/bugreport.cgi?bug=12345

Sure. That's why I advertise it occasionally :-)

Best regards, Michael.

Re: call for more ert tests

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Cc: Bozhidar Batsov <bozhidar@batsov.com>,  smerten@oekonux.de,  emacs-devel@gnu.org
> Date: Wed, 26 Jun 2013 23:01:19 +0400
> 
> > What hides behind "etc."?
> 
> My answer would be: being able to comment, tag and close the issues
> without having to use an email client and look up the exact way you do
> those things via inline text and special email addresses.

What does this have to do with debbugs being a bug tracker, but not an
issue tracker?

Re: call for more ert tests

[Teemu Likonen]

[-- Attachment #1: Type: text/plain, Size: 663 bytes --]

Dmitry Gutov [2013-06-26 23:01:19 +04:00] wrote:

> My answer would be: being able to comment, tag and close the issues
> without having to use an email client and look up the exact way you do
> those things via inline text and special email addresses.
>
> These things are not hard when you know them, but for a casual user,
> debbugs is quite intimidating.

Debian has this nice "bts" tool. Examples:

    bts show <number>
    bts select status:open
    bts done [version]
    bts merge <number> <number> ...
    bts tag <number> = tag tag tag


So,

    $ sudo apt-get install devscripts
    $ alias bts-emacs='bts --bts-server=debbugs.gnu.org'
    $ man bts

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: call for more ert tests

[Eli Zaretskii]

> From: Stefan Merten <smerten@oekonux.de>
> Comments: In-reply-to Glenn Morris <rgm@gnu.org>
> 	message dated "Mon, 24 Jun 2013 13:31:50 -0400."
> Date: Wed, 26 Jun 2013 11:22:24 +0200
> 
> May be I do not understand Lars' intentions here, but what is testing
> the C code good for at all? I mean there is an API - or rather: lots
> of APIs - and *this* is what needs testing. The C code to me looks
> like an implementation detail I'd not write tests for. Or at least
> separate those tests clearly from the API tests.

You seem to think that every C code in Emacs is part or a subroutine
of some Lisp API.  But that is false: e.g., the display engine is
implemented almost entirely in C.  If you have no test harness for
testing that, you cannot be sure, for example, that some change didn't
break cursor positioning (which happened a lot of time and will most
probably happen again, by a sheer coincidence -- or maybe due to
something else ;-).

Other important parts of Emacs that are implemented almost entirely in
C include:

  . GC

  . keyboard input

  . interaction with subprocesses

  . file I/O

There's probably more; these are just off the top of my head.

Re: call for more ert tests

[Stefan Merten]

[-- Attachment #1: Type: text/plain, Size: 1636 bytes --]

Hi!

3 days ago Eli Zaretskii wrote:
>> From: Stefan Merten <smerten@oekonux.de>
>> May be I do not understand Lars' intentions here, but what is testing
>> the C code good for at all? I mean there is an API - or rather: lots
>> of APIs - and *this* is what needs testing. The C code to me looks
>> like an implementation detail I'd not write tests for. Or at least
>> separate those tests clearly from the API tests.
> 
> You seem to think that every C code in Emacs is part or a subroutine
> of some Lisp API.  But that is false:

Of course. Sorry for my ignorance. My focus is on the interface
exposed in Lisp but that is of course only part of the story if you
really want a good test coverage.

> e.g., the display engine is
> implemented almost entirely in C.  If you have no test harness for
> testing that, you cannot be sure, for example, that some change didn't
> break cursor positioning

Sure.

> Other important parts of Emacs that are implemented almost entirely in
> C include:
> 
>   . GC
> 
>   . keyboard input
> 
>   . interaction with subprocesses
> 
>   . file I/O
> 
> There's probably more; these are just off the top of my head.

That would call for different test suites then. Also you hardly can
test all this by ert tests but need other frameworks for the tests.

Also the tests in an repository need to be organized or otherwise
you'll end up in chaos soon. For instance at the moment for rst.el I
have 22 lisp files with ert tests making up for nearly 10000 lines of
code. Yes, this is about the double of the lines rst.el has...


						Grüße

						Stefan

[-- Attachment #2: Type: application/pgp-signature, Size: 307 bytes --]