burden of maintainance

burden of maintainance

[Andreas Röhler]

Hi folks,

as the burden of maintainance was mentioned: from reading the 
bug-reports got the impression, a more strict test-regime might reduce that.

If a bug shows up, the first question should be: how it could survive 
the tests?
Current commit-policy seems still a bit away from that.

Will not being in favor of formalistic code-coverage technics,

Andreas

Re: burden of maintainance

[Eli Zaretskii]

> Date: Thu, 01 Oct 2015 10:10:18 +0200
> From: Andreas Röhler <andreas.roehler@online.de>
> 
> as the burden of maintainance was mentioned: from reading the 
> bug-reports got the impression, a more strict test-regime might reduce that.
> 
> If a bug shows up, the first question should be: how it could survive 
> the tests?
> Current commit-policy seems still a bit away from that.
> 
> Will not being in favor of formalistic code-coverage technics,

Suggestions for how to improve our test suite without alienating
potential contributors are welcome.

Re: burden of maintainance

[Przemysław Wojnowski]

> as the burden of maintainance was mentioned: from reading the
> bug-reports got the impression, a more strict test-regime might reduce
> that.
>
> If a bug shows up, the first question should be: how it could survive
> the tests?

+1

In previous project I joined a team that couldn't do any release in 4
years. I've introduced automated tests and refactoring among other
things and after 2 years we were releasing 4 times a year, with 3 times
less defects, found much sooner in release cycle and they were easier
to fix.

Some people here work in academia so maybe don't have such experiences,
but in software industry automated tests are a standard. Projects
without (or with weak) tests are replaced with those having strong
tests.

Re: burden of maintainance

[Andreas Röhler]

Am 01.10.2015 um 18:03 schrieb Eli Zaretskii:
>> Date: Thu, 01 Oct 2015 10:10:18 +0200
>> From: Andreas Röhler <andreas.roehler@online.de>
>>
>> as the burden of maintainance was mentioned: from reading the
>> bug-reports got the impression, a more strict test-regime might reduce that.
>>
>> If a bug shows up, the first question should be: how it could survive
>> the tests?
>> Current commit-policy seems still a bit away from that.
>>
>> Will not being in favor of formalistic code-coverage technics,
> Suggestions for how to improve our test suite without alienating
> potential contributors are welcome.
>
>

What about saying: no checkin before the tests passed?

Coverage and quality of tests should be an integral part of developing.

WRT to maintainance there are also redundancy, complexity which might be 
worked on.

Keeping an eye at the number of symbols - too many blow up the language, 
make in harder for beginners than needed.

Re: burden of maintainance

[Eli Zaretskii]

> Date: Thu, 01 Oct 2015 20:46:15 +0200
> From: Andreas Röhler <andreas.roehler@online.de>
> CC: Eli Zaretskii <eliz@gnu.org>
> 
> > Suggestions for how to improve our test suite without alienating
> > potential contributors are welcome.
> 
> What about saying: no checkin before the tests passed?

That is OK, but what to do if some tests fail for many moons before
they are fixed?  We cannot stop development because of that.

> Coverage and quality of tests should be an integral part of developing.

That's the hard part.  Our current coverage is quite low (my
impression; it would be good to have some tool that can measure
that).  Worse, for interactive features (and there are a lot of them),
we lack the infrastructure for writing automated tests.  Also, I don't
quite see who will write tests for large portions of the C code, given
how few people are even prepared to work on that.  The result is that
getting closer to good coverage is a huge job.

> WRT to maintainance there are also redundancy, complexity which might be 
> worked on.

Alas, there are wildly different views on what is and isn't complex.

> Keeping an eye at the number of symbols - too many blow up the language, 
> make in harder for beginners than needed.

Which symbols did you have in mind?

Re: burden of maintainance

[Andreas Röhler]

Am 01.10.2015 um 21:09 schrieb Eli Zaretskii:
>> Date: Thu, 01 Oct 2015 20:46:15 +0200
>> From: Andreas Röhler <andreas.roehler@online.de>
>> CC: Eli Zaretskii <eliz@gnu.org>
>>
>>> Suggestions for how to improve our test suite without alienating
>>> potential contributors are welcome.
>> What about saying: no checkin before the tests passed?
> That is OK, but what to do if some tests fail for many moons before
> they are fixed?  We cannot stop development because of that.

Let's understand tests as valid ones and checkins as non-trivial.

It would suffice to put intelligent test-writing into the focus - where 
it's not yet or not yet to extend it deserves.


>
>> Coverage and quality of tests should be an integral part of developing.
> That's the hard part.  Our current coverage is quite low (my
> impression; it would be good to have some tool that can measure
> that).  Worse, for interactive features (and there are a lot of them),
> we lack the infrastructure for writing automated tests.

Let's note that.

>    Also, I don't
> quite see who will write tests for large portions of the C code, given
> how few people are even prepared to work on that.

Estimating one hour invested in writing the right tests might save 
ten-fold or more of development time afterwards.


>    The result is that
> getting closer to good coverage is a huge job.
>
>> WRT to maintainance there are also redundancy, complexity which might be
>> worked on.
> Alas, there are wildly different views on what is and isn't complex.
>
>> Keeping an eye at the number of symbols - too many blow up the language,
>> make in harder for beginners than needed.
> Which symbols did you have in mind?
>
>
>

Not a special one at the moment. Just got the feeling the collection of 
used symbols were growing and growing last years.

Re: burden of maintainance

[Eli Zaretskii]

> Date: Thu, 01 Oct 2015 22:18:16 +0200
> From: Andreas Röhler <andreas.roehler@online.de>
> CC: Eli Zaretskii <eliz@gnu.org>
> 
> It would suffice to put intelligent test-writing into the focus - where 
> it's not yet or not yet to extend it deserves.

This is a volunteer effort.  Nothing gets done by putting some issues
into the focus: you cannot tell anyone to do something because it's in
focus.  The only way things get moved in Emacs is when volunteers step
forward and take tasks upon themselves.

So the issue at hand is how to attract contributors to writing tests.
It's not a decision-making problem, it's a leadership and
team-management problem.

> >    Also, I don't
> > quite see who will write tests for large portions of the C code, given
> > how few people are even prepared to work on that.
> 
> Estimating one hour invested in writing the right tests might save 
> ten-fold or more of development time afterwards.

I agree.  But it's not enough to agree, there's a need to actually do
it.

> Not a special one at the moment. Just got the feeling the collection of 
> used symbols were growing and growing last years.

My take of this is that many people here disagree with you about this
being a bad thing.

Re: burden of maintainance

[Helmut Eller]

On Thu, Oct 01 2015, Eli Zaretskii wrote:

> Suggestions for how to improve our test suite without alienating
> potential contributors are welcome.

For a start, fix bug#16853.

Second, make it easy to run the test suite in non-batch mode as it's
impossible to test display related issues in batch mode alone.  Perhaps
use Screen or Xvfb to run the display code without physically displaying
anything.

Third, add a construct like HANDLER-BIND in Common Lisp that runs the
condition handler without unwinding the stack.  ERT rebinds `debugger'
for this purpose but IMO it's very ugly and obviously interferes with
the normal debugger.

Helmut

Re: burden of maintainance

[Eli Zaretskii]

> From: Helmut Eller <eller.helmut@gmail.com>
> Date: Fri, 02 Oct 2015 09:51:43 +0200
> 
> On Thu, Oct 01 2015, Eli Zaretskii wrote:
> 
> > Suggestions for how to improve our test suite without alienating
> > potential contributors are welcome.
> 
> For a start, fix bug#16853.

Why that particular bug?  We have lots of open bugs; what's special
about that one.

> Second, make it easy to run the test suite in non-batch mode

It's already possible, I do that quite a lot.  What doesn't work for
you?

> as it's impossible to test display related issues in batch mode
> alone.  Perhaps use Screen or Xvfb to run the display code without
> physically displaying anything.

These are X- and Unix-specific, AFAIK.  And even on those systems,
we'll need a lot of infrastructure built on top of that, before it
will be reasonably practical to write display-related tests.  I hope
someone will volunteer to do that.

In any case, the problem is not with actually displaying something,
the problem is how to compare that with some "expected results".
(Assuming you were talking about testing the display engine.)

Re: burden of maintainance

[Helmut Eller]

On Fri, Oct 02 2015, Eli Zaretskii wrote:

>> From: Helmut Eller <eller.helmut@gmail.com>
>> Date: Fri, 02 Oct 2015 09:51:43 +0200
>> 
>> On Thu, Oct 01 2015, Eli Zaretskii wrote:
>> 
>> > Suggestions for how to improve our test suite without alienating
>> > potential contributors are welcome.
>> 
>> For a start, fix bug#16853.
>
> Why that particular bug?  We have lots of open bugs; what's special
> about that one.

I thought you were interested in suggestion to improve something but as
usual you aren't.  My mistake; should know that my now.

>> Second, make it easy to run the test suite in non-batch mode
>
> It's already possible, I do that quite a lot.  What doesn't work for
> you?

bug#16853.

Helmut

Re: burden of maintainance

[Eli Zaretskii]

> From: Helmut Eller <eller.helmut@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Fri, 02 Oct 2015 10:56:32 +0200
> 
> On Fri, Oct 02 2015, Eli Zaretskii wrote:
> 
> >> From: Helmut Eller <eller.helmut@gmail.com>
> >> Date: Fri, 02 Oct 2015 09:51:43 +0200
> >> 
> >> On Thu, Oct 01 2015, Eli Zaretskii wrote:
> >> 
> >> > Suggestions for how to improve our test suite without alienating
> >> > potential contributors are welcome.
> >> 
> >> For a start, fix bug#16853.
> >
> > Why that particular bug?  We have lots of open bugs; what's special
> > about that one.
> 
> I thought you were interested in suggestion to improve something but as
> usual you aren't.  My mistake; should know that my now.

I thought you'd learn to behave in a civilized manner, instead of
being your usual rude self.  My mistake; should have known better and
refrained to take the bait.

> >> Second, make it easy to run the test suite in non-batch mode
> >
> > It's already possible, I do that quite a lot.  What doesn't work for
> > you?
> 
> bug#16853.

You publicly asked me to shut up and stop waisting your time about
that bug, so someone else will have to do that in my stead.

Re: burden of maintainance

[Phillip Lord]

Przemysław Wojnowski <esperanto@cumego.com> writes:

>> as the burden of maintainance was mentioned: from reading the
>> bug-reports got the impression, a more strict test-regime might reduce
>> that.
>>
>> If a bug shows up, the first question should be: how it could survive
>> the tests?
>
> +1
>
> In previous project I joined a team that couldn't do any release in 4
> years. I've introduced automated tests and refactoring among other
> things and after 2 years we were releasing 4 times a year, with 3 times
> less defects, found much sooner in release cycle and they were easier
> to fix.
>
> Some people here work in academia so maybe don't have such experiences,
> but in software industry automated tests are a standard. Projects
> without (or with weak) tests are replaced with those having strong
> tests.


Well some of us work in academia and still have this experience anyway,
which is why I write tests for most of my own Emacs packages.

WRT Emacs, I think, that testing would be a good thing but there are a
couple of hurdles to overcome. First, most of Emacs doesn't have tests,
simply because it predates widespread use of testing. Secondly, Emacs
use of global state (current-buffer!) can make testing quite difficult;
combined with the reality that some of the functions are pretty large,
and will be difficult to retrofit tests onto, I think this is quite an
issue. And, thirdly, testing of interactive programs is difficult
anyway; the difficulty of writing tests in this environment is much
higher, and the payback lower. 

We can have more than one maintainer. Someone who wanted to just
concentrate on getting a nightly build infrastructure running, with
email to emacs-diffs, and then starting to add fixtures and some other
frameworks to ert would be valuable addition to the process.

Phil

Re: burden of maintainance

[Michael Albinus]

Helmut Eller <eller.helmut@gmail.com> writes:

>>> Second, make it easy to run the test suite in non-batch mode
>>
>> It's already possible, I do that quite a lot.  What doesn't work for
>> you?
>
> bug#16853.

You can always run one test interactively, by pressing "r". It's not a
fix of your problem, but your problem is not a show-stopper.

> Helmut

Best regards, Michael.

Re: burden of maintainance

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Thu, 01 Oct 2015 10:10:18 +0200
>> From: Andreas Röhler <andreas.roehler@online.de>
>> 
>> as the burden of maintainance was mentioned: from reading the 
>> bug-reports got the impression, a more strict test-regime might reduce that.
>> 
>> If a bug shows up, the first question should be: how it could survive 
>> the tests?
>> Current commit-policy seems still a bit away from that.
>> 
>> Will not being in favor of formalistic code-coverage technics,
>
> Suggestions for how to improve our test suite without alienating
> potential contributors are welcome.


I think for many potential contributors a richer test suite is likely to
lessen alienation rather than increase it.

Still, you asked for ways to improve it. Here is my suggestion:

[phillord@...src/large/emacs]$ make test
make: Nothing to be done for `test'.


I would rename "check" to "test" and add "test" to .PHONY. I didn't
even know that Emacs had an automated test suite until one of the
elongated threads that Eric inspired here about six months ago.

Phil

Re: burden of maintainance

[Michael Albinus]

phillip.lord@russet.org.uk (Phillip Lord) writes:

> Someone who wanted to just concentrate on getting a nightly build
> infrastructure running, with email to emacs-diffs, and then starting
> to add fixtures and some other frameworks to ert would be valuable
> addition to the process.

What's wrong with hydra and the emacs-buildstatus ML?

<http://hydra.nixos.org/jobset/gnu/emacs-trunk>

<http://lists.gnu.org/archive/html/emacs-buildstatus>

> Phil

Best regards, Michael.

Re: burden of maintainance

[Artur Malabarba]

2015-10-02 12:58 GMT+01:00 Phillip Lord <phillip.lord@russet.org.uk>:
> Still, you asked for ways to improve it. Here is my suggestion:
>
> [phillord@...src/large/emacs]$ make test
> make: Nothing to be done for `test'.

Good point. Would anyone object to adding a trivial test entry to the makefile?

Re: burden of maintainance

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Cc: Andreas Röhler <andreas.roehler@online.de>,
>   <emacs-devel@gnu.org>
> Date: Fri, 02 Oct 2015 12:58:48 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Suggestions for how to improve our test suite without alienating
> > potential contributors are welcome.
> 
> I think for many potential contributors a richer test suite is likely to
> lessen alienation rather than increase it.

I meant if we require each contributor to accompany each commit with a
test, that could prevent quite a few from contributing.

> Still, you asked for ways to improve it. Here is my suggestion:
> 
> [phillord@...src/large/emacs]$ make test
> make: Nothing to be done for `test'.
> 
> I would rename "check" to "test" and add "test" to .PHONY.

Feel free to make such a change, and TIA.

Re: burden of maintainance

[Przemysław Wojnowski]

>>> Suggestions for how to improve our test suite without alienating
>>> potential contributors are welcome.
I would say that without tests they are alienated. It's much harder to 
contribute anything without test coverage.

>> What about saying: no checkin before the tests passed?
 >
 > That is OK, but what to do if some tests fail for many moons before
 > they are fixed?  We cannot stop development because of that.

Everyone should run tests before push.
Even though they sometimes fail for various reasons (someone forgot,
different env). In such case the person is notified and should fix that
before doing any other work. Usually if it cannot be done within a few
hours (a day) the commit is reverted or test marked as ignored
(sometimes it's a faulty test).

I've seen that "procedure" in many projects and it worked fine.
Of course it is not welded and can be customized to suite Emacs
development.

 > Worse, for interactive features (and there are a lot of them),
> we lack the infrastructure for writing automated tests.
Can you give an example? Of course not everything can be tested, but
many things can.

IMHO it's a matter of restructuring the code. If interactive functions
try to do work by themselves then it is hard, but if they only take
input and pass to another component (for example like in MVC pattern)
then it's much easier.

> Also, I don't
> quite see who will write tests for large portions of the C code, given
> how few people are even prepared to work on that.  The result is that
> getting closer to good coverage is a huge job.
Everything starts somewhere and a good start would be to add a C
testing library and enabling it in tests.

This would at least enable those willing to write tests. Currently they
can't, even if they would like to.

Re: burden of maintainance

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Przemysław Wojnowski <esperanto@cumego.com>
> Date: Fri, 2 Oct 2015 15:54:08 +0200
> 
> >>> Suggestions for how to improve our test suite without alienating
> >>> potential contributors are welcome.
> I would say that without tests they are alienated. It's much harder to 
> contribute anything without test coverage.

That's a misunderstanding.  I explained what I meant in a later
message.

> >> What about saying: no checkin before the tests passed?
>  >
>  > That is OK, but what to do if some tests fail for many moons before
>  > they are fixed?  We cannot stop development because of that.
> 
> Everyone should run tests before push.
> Even though they sometimes fail for various reasons (someone forgot,
> different env). In such case the person is notified and should fix that
> before doing any other work. Usually if it cannot be done within a few
> hours (a day) the commit is reverted or test marked as ignored
> (sometimes it's a faulty test).
> 
> I've seen that "procedure" in many projects and it worked fine.

Me too.  But we are not talking about theoretical feasibility.

> Of course it is not welded and can be customized to suite Emacs
> development.

I'm okay with that, if everyone agrees.  You should know, though, that
Emacs development up until now is VERY different from such projects:
we have no strictly codified process for committing and patch reviews,
never had one.  Introducing such mandatory procedures is a big change.

>  > Worse, for interactive features (and there are a lot of them),
> > we lack the infrastructure for writing automated tests.
> Can you give an example? Of course not everything can be tested, but
> many things can.

We are mis-communicating again: I meant features which display
something or involve interactive keyboard input.

> a good start would be to add a C testing library and enabling it in
> tests.

I agree.  I hope someone will volunteer to make this happen.

Re: burden of maintainance

[Phillip Lord]

Michael Albinus <michael.albinus@gmx.de> writes:

> phillip.lord@russet.org.uk (Phillip Lord) writes:
>
>> Someone who wanted to just concentrate on getting a nightly build
>> infrastructure running, with email to emacs-diffs, and then starting
>> to add fixtures and some other frameworks to ert would be valuable
>> addition to the process.
>
> What's wrong with hydra and the emacs-buildstatus ML?
>
> <http://hydra.nixos.org/jobset/gnu/emacs-trunk>

Is is running the tests?

From the logs it doesn't appear to be. Which is unfortunate because
although I can ./configure;make from trunk but at the moment the tests
on trunk are failing. At least on my machine.

Does it do ELPA as well? And branches would be good.

> <http://lists.gnu.org/archive/html/emacs-buildstatus>

Didn't know about this. My apologies!

Phil

Re: burden of maintainance

[Andreas Röhler]

Am 02.10.2015 um 15:36 schrieb Eli Zaretskii:
> I meant if we require each contributor to accompany each commit with a
> test, that could prevent quite a few from contributing.

Not just require, but let's say: for the time being,
every committer should be encuraged to asked the related part for test 
coverage.

Notably not stupid tests for all and everything, but the core parts.

If there is no test, writing the test before any other proceeding should 
save a lot of time.

Re: burden of maintainance

[Eli Zaretskii]

> Date: Fri, 02 Oct 2015 17:00:01 +0200
> From: Andreas Röhler <andreas.roehler@online.de>
> CC: emacs-devel@gnu.org
> 
> Am 02.10.2015 um 15:36 schrieb Eli Zaretskii:
> > I meant if we require each contributor to accompany each commit with a
> > test, that could prevent quite a few from contributing.
> 
> Not just require, but let's say: for the time being,
> every committer should be encuraged to asked the related part for test 
> coverage.

That we already do, although we could do it more.

> If there is no test, writing the test before any other proceeding should 
> save a lot of time.

I agree.  The problem is in practice, not in principle.

Re: burden of maintainance

[Michael Albinus]

phillip.lord@russet.org.uk (Phillip Lord) writes:

>> What's wrong with hydra and the emacs-buildstatus ML?
>>
>> <http://hydra.nixos.org/jobset/gnu/emacs-trunk>
>
> Is is running the tests?
>
> From the logs it doesn't appear to be. Which is unfortunate because
> although I can ./configure;make from trunk but at the moment the tests
> on trunk are failing. At least on my machine.

Yes, the ERT tests run on hydra. Check the "coverage" job of a test run,
for example <http://hydra.nixos.org/build/26534891>. In the log file,
you see also the output of a "make check" call.

> Does it do ELPA as well? And branches would be good.

No, it doesn't check ELPA packages. I'm sure that such a contribution
from a volunteer would be welcome.

Branches are not checked, except the emacs-24 branch while we were in
pretest. This has been stalled meanwhile, because nothing happens in
this branch now.

> Phil

Best regards, Michael.

Re: burden of maintainance

[Tassilo Horn]

Eli Zaretskii <eliz@gnu.org> writes:

>> Can you give an example? Of course not everything can be tested, but
>> many things can.
>
> We are mis-communicating again: I meant features which display
> something or involve interactive keyboard input.

Emacs is in a better position here than most other tools because it has
keyboard macros and can easily inspect the things it displays, e.g., the
text properties or overlays at point.

I can think of something like "ERT keyboard-macro tests" where you would
basically record a keyboard macro and during that, you have some way to
add some assertions before resuming recording.  The whole recorded thing
could then be stored as an ERT test.

Bye,
Tassilo

Re: burden of maintainance

[Andreas Röhler]

Am 02.10.2015 um 17:16 schrieb Eli Zaretskii:
>> Date: Fri, 02 Oct 2015 17:00:01 +0200
>> From: Andreas Röhler <andreas.roehler@online.de>
>> CC: emacs-devel@gnu.org
>>
>> Am 02.10.2015 um 15:36 schrieb Eli Zaretskii:
>>> I meant if we require each contributor to accompany each commit with a
>>> test, that could prevent quite a few from contributing.
>> Not just require, but let's say: for the time being,
>> every committer should be encuraged to asked the related part for test
>> coverage.
> That we already do, although we could do it more.
>
>> If there is no test, writing the test before any other proceeding should
>> save a lot of time.
> I agree.  The problem is in practice, not in principle.
>
>

What about creating a co-maintainance WRT code-quality?

Re: burden of maintainance

[Eli Zaretskii]

> Date: Fri, 02 Oct 2015 19:19:31 +0200
> From: Andreas Röhler <andreas.roehler@online.de>
> CC: phillip.lord@russet.org.uk, emacs-devel@gnu.org
> 
> >> If there is no test, writing the test before any other proceeding should
> >> save a lot of time.
> > I agree.  The problem is in practice, not in principle.
> >
> >
> 
> What about creating a co-maintainance WRT code-quality?

Good idea.  We will need a clear description of the job and one or
more volunteers.

Re: burden of maintainance

[Eli Zaretskii]

> From: Tassilo Horn <tsdh@gnu.org>
> Cc: Przemysław Wojnowski <esperanto@cumego.com>,
>   andreas.roehler@online.de,  emacs-devel@gnu.org
> Date: Fri, 02 Oct 2015 19:18:08 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Can you give an example? Of course not everything can be tested, but
> >> many things can.
> >
> > We are mis-communicating again: I meant features which display
> > something or involve interactive keyboard input.
> 
> Emacs is in a better position here than most other tools because it has
> keyboard macros and can easily inspect the things it displays, e.g., the
> text properties or overlays at point.

We may be mis-communicating here.  Because I cannot see how examining
text properties or overlays can test whether their display on screen
is correct.  What am I missing?  Can you show an example of what you
had in mind?

If we are talking about testing the display engine, it should be
possible and relatively easy to provide Lisp access to the "glyph
matrices", which are data structures the display engine builds that
describe what _should_ be on the screen.  You can see one specialized
example of that in the function bidi-resolved-levels which I wrote for
testing the bidirectional display.

But doing the above is only half the way.  The other half is the
terminal-specific back-end, either X, w32, NS, or TTY, which actually
converts the glyph matrices to draw commands and delivers the results
to the glass.  To test that, we need some way of "sniffing" the
results of this drawing, which requires terminal-specific
infrastructure.

We then need some "language" to describe the expected results to be
displayed on the screen.  For TTY, this is almost trivial, but for
sophisticated GUI display features it's more complicated (think of
fonts, stretches of white space, composed characters, fringe bitmaps,
tool-bar icons, tooltips, etc.).  We also need a way of describing the
results of cursor movement.

When all of these are implemented, then yes, writing display tests
should be relatively easy.  But today all this infrastructure doesn't
exist.  We don't even have its detailed design.

> I can think of something like "ERT keyboard-macro tests" where you would
> basically record a keyboard macro and during that, you have some way to
> add some assertions before resuming recording.  The whole recorded thing
> could then be stored as an ERT test.

For testing keyboard input, we need more than that.  For example,
consider the simple feature of input history -- you'd need a way to
inject M-p, M-n, etc., and a way of recording the text that is
displayed as result and returned to the caller.

And please don't forget that keyboard input is only one kind of events
supported by Emacs.  There are others, starting with mouse.  We'd need
ways to simulate or trigger all of them.

Re: burden of maintainance

[joakim]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Tassilo Horn <tsdh@gnu.org>
>> Cc: Przemysław Wojnowski <esperanto@cumego.com>,
>>   andreas.roehler@online.de,  emacs-devel@gnu.org
>> Date: Fri, 02 Oct 2015 19:18:08 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> Can you give an example? Of course not everything can be tested, but
>> >> many things can.
>> >
>> > We are mis-communicating again: I meant features which display
>> > something or involve interactive keyboard input.
>> 
>> Emacs is in a better position here than most other tools because it has
>> keyboard macros and can easily inspect the things it displays, e.g., the
>> text properties or overlays at point.
>
> We may be mis-communicating here.  Because I cannot see how examining
> text properties or overlays can test whether their display on screen
> is correct.  What am I missing?  Can you show an example of what you
> had in mind?

As a data point, there is an interesting GUI test toolkit called Sikuli,
It uses the OpenCV computer vision toolkit to analyze that the display
matches expectations. Sikuli is free software if I recall correctly.


>
> If we are talking about testing the display engine, it should be
> possible and relatively easy to provide Lisp access to the "glyph
> matrices", which are data structures the display engine builds that
> describe what _should_ be on the screen.  You can see one specialized
> example of that in the function bidi-resolved-levels which I wrote for
> testing the bidirectional display.
>
> But doing the above is only half the way.  The other half is the
> terminal-specific back-end, either X, w32, NS, or TTY, which actually
> converts the glyph matrices to draw commands and delivers the results
> to the glass.  To test that, we need some way of "sniffing" the
> results of this drawing, which requires terminal-specific
> infrastructure.
>
> We then need some "language" to describe the expected results to be
> displayed on the screen.  For TTY, this is almost trivial, but for
> sophisticated GUI display features it's more complicated (think of
> fonts, stretches of white space, composed characters, fringe bitmaps,
> tool-bar icons, tooltips, etc.).  We also need a way of describing the
> results of cursor movement.
>
> When all of these are implemented, then yes, writing display tests
> should be relatively easy.  But today all this infrastructure doesn't
> exist.  We don't even have its detailed design.
>
>> I can think of something like "ERT keyboard-macro tests" where you would
>> basically record a keyboard macro and during that, you have some way to
>> add some assertions before resuming recording.  The whole recorded thing
>> could then be stored as an ERT test.
>
> For testing keyboard input, we need more than that.  For example,
> consider the simple feature of input history -- you'd need a way to
> inject M-p, M-n, etc., and a way of recording the text that is
> displayed as result and returned to the caller.
>
> And please don't forget that keyboard input is only one kind of events
> supported by Emacs.  There are others, starting with mouse.  We'd need
> ways to simulate or trigger all of them.
>
>

-- 
Joakim Verona

Re: burden of maintainance

[Andreas Röhler]

Am 02.10.2015 um 20:08 schrieb Eli Zaretskii:
>> Date: Fri, 02 Oct 2015 19:19:31 +0200
>> From: Andreas Röhler <andreas.roehler@online.de>
>> CC: phillip.lord@russet.org.uk, emacs-devel@gnu.org
>>
>>>> If there is no test, writing the test before any other proceeding should
>>>> save a lot of time.
>>> I agree.  The problem is in practice, not in principle.
>>>
>>>
>> What about creating a co-maintainance WRT code-quality?
> Good idea.  We will need a clear description of the job and one or
> more volunteers.
>
>

Also considerably pay-off should be gained by reducing redundancy - just 
reading org-mode related comment in thread "Should we just start dumping 
cl-lib?"

Re: burden of maintainance

[Tassilo Horn]

Eli Zaretskii <eliz@gnu.org> writes:

>> > We are mis-communicating again: I meant features which display
>> > something or involve interactive keyboard input.
>> 
>> Emacs is in a better position here than most other tools because it
>> has keyboard macros and can easily inspect the things it displays,
>> e.g., the text properties or overlays at point.
>
> We may be mis-communicating here.  Because I cannot see how examining
> text properties or overlays can test whether their display on screen
> is correct.  What am I missing?  Can you show an example of what you
> had in mind?

No, if something is actually displayed correctly cannot be tested like
so.  What I had in mind were things like testing if fontification works
properly by checking if the 'face property is set to some expected face
when point is one some keyword.

> If we are talking about testing the display engine, it should be
> possible and relatively easy to provide Lisp access to the "glyph
> matrices", which are data structures the display engine builds that
> describe what _should_ be on the screen.  You can see one specialized
> example of that in the function bidi-resolved-levels which I wrote for
> testing the bidirectional display.
>
> But doing the above is only half the way.  The other half is the
> terminal-specific back-end, either X, w32, NS, or TTY, which actually
> converts the glyph matrices to draw commands and delivers the results
> to the glass.  To test that, we need some way of "sniffing" the
> results of this drawing, which requires terminal-specific
> infrastructure.
>
> We then need some "language" to describe the expected results to be
> displayed on the screen.  For TTY, this is almost trivial, but for
> sophisticated GUI display features it's more complicated (think of
> fonts, stretches of white space, composed characters, fringe bitmaps,
> tool-bar icons, tooltips, etc.).  We also need a way of describing the
> results of cursor movement.
>
> When all of these are implemented, then yes, writing display tests
> should be relatively easy.  But today all this infrastructure doesn't
> exist.  We don't even have its detailed design.

That's obviously not what I've head in mind.  But I don't think that's
an area where we need a special focus on tests.  At least I have the
feeling that there are only few bugs wrt. the display engine.

>> I can think of something like "ERT keyboard-macro tests" where you
>> would basically record a keyboard macro and during that, you have
>> some way to add some assertions before resuming recording.  The whole
>> recorded thing could then be stored as an ERT test.
>
> For testing keyboard input, we need more than that.  For example,
> consider the simple feature of input history -- you'd need a way to
> inject M-p, M-n, etc., and a way of recording the text that is
> displayed as result and returned to the caller.
>
> And please don't forget that keyboard input is only one kind of events
> supported by Emacs.  There are others, starting with mouse.  We'd need
> ways to simulate or trigger all of them.

Yes, eventually.  But it would at least be a first step.  And this
hypothetical recorder could also be exposed to the user for creating
replayable recipes for bugs they report.  Where a tester would add
assertions, the user would just write what he's expecting, and whoever
fixes the bug could easily turn the string into assertions, and voila,
there's the regression test for the bug.

Bye,
Tassilo

Re: burden of maintainance

[Chad Brown]

> On 02 Oct 2015, at 11:24, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Tassilo Horn <tsdh@gnu.org>
>> Cc: Przemysław Wojnowski <esperanto@cumego.com>,
>>  andreas.roehler@online.de,  emacs-devel@gnu.org
>> Date: Fri, 02 Oct 2015 19:18:08 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>>>> Can you give an example? Of course not everything can be tested, but
>>>> many things can.
>>> 
>>> We are mis-communicating again: I meant features which display
>>> something or involve interactive keyboard input.
>> 
>> Emacs is in a better position here than most other tools because it has
>> keyboard macros and can easily inspect the things it displays, e.g., the
>> text properties or overlays at point.
> 
> We may be mis-communicating here.  Because I cannot see how examining
> text properties or overlays can test whether their display on screen
> is correct.  What am I missing?  Can you show an example of what you
> had in mind?

I believe that Tassilo is talking about tests in the vein of “does the buffer report containing the expected characters, in the expected order, with the expected properties, etc”, rather than “are the expected pixels on the display the expected colors at the expected times”. Similarly, tools exist to automate testing with a synthetic mouse and keyboard.

I had some experience with this kind of testing a long while back (roughly, 1995-2000). There are tools that can be used to automate tests of GUIs, but they were (I think still are, but I’m not current) fairly deeply complicated - that is, they use their own fairly extensive language and idiom. It would take significant effort to learn these test harnesses, especially if one wanted to drive them from elisp (unless the SotA has changed more than I think - which is possible).

That said, we found that we could get quite a lot of benefit out of automated testing that basically ignored the gui. I suspect that emacs could see roughly the same - while there are issues that require testing the gui, there are also a great many that don’t. Looking over the recent archives for emacs-devel, it seems like a majority of topics would fall into this “can ignore the gui” category. This might be enough to get emacs started toward a more test-driven development model.

Hope that helps,
~Chad

Re: burden of maintainance

[Tassilo Horn]

Andreas Röhler <andreas.roehler@online.de> writes:

> Also considerably pay-off should be gained by reducing redundancy -
> just reading org-mode related comment in thread "Should we just start
> dumping cl-lib?"

Org, Gnus, AUCTeX and friends define org/gnus/TeX-* duplicates of
standard functions because for compatibility with older emacs versions
and XEmacs.

Bye,
Tassilo

Re: burden of maintainance

[Eli Zaretskii]

> From: Tassilo Horn <tsdh@gnu.org>
> Cc: esperanto@cumego.com,  andreas.roehler@online.de,  emacs-devel@gnu.org
> Date: Fri, 02 Oct 2015 21:16:58 +0200
> 
> > When all of these are implemented, then yes, writing display tests
> > should be relatively easy.  But today all this infrastructure doesn't
> > exist.  We don't even have its detailed design.
> 
> That's obviously not what I've head in mind.  But I don't think that's
> an area where we need a special focus on tests.  At least I have the
> feeling that there are only few bugs wrt. the display engine.

I wish this were true.  Bugs are still being reported, albeit at a low
frequency.

But a good test suite is required not only to find existing bugs, it
is also required to be sure changes we make to introduce new features
don't introduce bugs in what was previously correct behavior.

Re: burden of maintainance

[Xue Fuqiao]

On Fri, Oct 2, 2015 at 7:58 PM, Phillip Lord <phillip.lord@russet.org.uk> wrote:
> [phillord@...src/large/emacs]$ make test
> make: Nothing to be done for `test'.
>
>
> I would rename "check" to "test" and add "test" to .PHONY. I didn't
> even know that Emacs had an automated test suite until one of the
> elongated threads that Eric inspired here about six months ago.

"make check" is part of the GNU Coding Standards.  See:

* https://www.gnu.org/prep/standards/html_node/Standard-Targets.html#Standard-Targets
* https://www.gnu.org/software/automake/manual/html_node/Standard-Targets.html#Standard-Targets
* https://www.gnu.org/software/make/manual/html_node/Standard-Targets.html#Standard-Targets

So IMHO an alias (instead of renaming "check" to "test") would be better.

Re: burden of maintainance

[Eric Ludlam]

On 10/02/2015 09:36 AM, Eli Zaretskii wrote:
>> From: phillip.lord@russet.org.uk (Phillip Lord)
>> Cc: Andreas Röhler <andreas.roehler@online.de>,
>>    <emacs-devel@gnu.org>
>> Date: Fri, 02 Oct 2015 12:58:48 +0100
>>
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>>> Suggestions for how to improve our test suite without alienating
>>> potential contributors are welcome.
>>
>> I think for many potential contributors a richer test suite is likely to
>> lessen alienation rather than increase it.
>
> I meant if we require each contributor to accompany each commit with a
> test, that could prevent quite a few from contributing.

CEDET on SF has a pretty extensive test suite, and I still include fixes 
from folks without a test, usually adding a little test along the way. 
Fortunately for me, many of the test harnesses I've written make it 
trivial to add new points to it.   For smart completion for example, you 
can edit a test input file with a new location to try completion, and 
stick the answer next to it in a comment for that language.  The SRecode 
tests are usually just writing a template and adding a name to a list to 
try out new features there.  No need to write complex logic stuff.

A side effect is that new contributors who come by have been adding 
tests without me asking, which is super awesome!   A large part of that 
is having a rich test suite to copy from to make new tests easier.

So, that is my recommendation.  If more tests are desired in a 
particular area, a smart harness will allow a new feature to be tested 
quickly.  The best case is it can be tested by adding a text file 
somewhere that can be pulled in and processed.

The opposite is the eieio test suite which is long and all about complex 
lisp structures to test it out.  That kind of infrastructure is a bit 
harder to simplify.

Eric

Re: burden of maintainance

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Looking for how to improve testing is useful,
but it is tangential to finding a new maintainer team,
and that is what we really need now.  Let's focus mainly
on that.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: burden of maintainance

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > What about creating a co-maintainance WRT code-quality?

Let's discuss questions like this AFTER we have a new maintainer team
that can make decisions about them.


-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: burden of maintainance

[Przemysław Wojnowski]

> "make check" is part of the GNU Coding Standards.  See:
Maybe it should be changed to conform to the rest of the world, making
it intuitive at the same time.

IMHO the name should reveal its intent and "check" doesn't tell what is
it checking (maybe whether environment is ready for compilation?).

"make test" is understandable and intuitive, especially to anyone who
has any experience building software in other environments.

Re: burden of maintainance

[Andreas Röhler]

Am 02.10.2015 um 22:39 schrieb Tassilo Horn:
> Andreas Röhler <andreas.roehler@online.de> writes:
>
>> Also considerably pay-off should be gained by reducing redundancy -
>> just reading org-mode related comment in thread "Should we just start
>> dumping cl-lib?"
> Org, Gnus, AUCTeX and friends define org/gnus/TeX-* duplicates of
> standard functions because for compatibility with older emacs versions
> and XEmacs.
>
>

Okay, understand the endeavour, but does this make sense?

If the modes don't agree with core-policy, wouldn't it be better to 
reach a solution for all by discussion and decision-making?

Cheers,

Andreas

Re: burden of maintainance

[Xue Fuqiao]

On Sat, Oct 3, 2015 at 2:42 PM, Przemysław Wojnowski
<esperanto@cumego.com> wrote:
>> "make check" is part of the GNU Coding Standards.  See:
>
> Maybe it should be changed to conform to the rest of the world, making
> it intuitive at the same time.
>
> IMHO the name should reveal its intent and "check" doesn't tell what is
> it checking (maybe whether environment is ready for compilation?).

I think (but I'm not sure) most users who can compile software with
`configure' followed by `make' know that `configure' scans the build
environment and tests whether environment is ready for compilation.

> "make test" is understandable and intuitive, especially to anyone who
> has any experience building software in other environments.

I agree that "make test" is intuitive.  I'll start a discussion on
bug-standards@gnu.org.

Re: burden of maintainance

[Eli Zaretskii]

> Cc: Eli Zaretskii <eliz@gnu.org>, Andreas Röhler
>  <andreas.roehler@online.de>, Emacs-devel <emacs-devel@gnu.org>
> From: Przemysław Wojnowski <esperanto@cumego.com>
> Date: Sat, 3 Oct 2015 08:42:58 +0200
> 
> > "make check" is part of the GNU Coding Standards.  See:
> Maybe it should be changed to conform to the rest of the world, making
> it intuitive at the same time.
> 
> IMHO the name should reveal its intent and "check" doesn't tell what is
> it checking (maybe whether environment is ready for compilation?).
> 
> "make test" is understandable and intuitive, especially to anyone who
> has any experience building software in other environments.

If they both do the same, we have the best of all worlds, no?  So why
continue arguing?

Re: burden of maintainance

[Tassilo Horn]

Andreas Röhler <andreas.roehler@online.de> writes:

>>> Also considerably pay-off should be gained by reducing redundancy -
>>> just reading org-mode related comment in thread "Should we just start
>>> dumping cl-lib?"
>> Org, Gnus, AUCTeX and friends define org/gnus/TeX-* duplicates of
>> standard functions because for compatibility with older emacs versions
>> and XEmacs.
>
> Okay, understand the endeavour, but does this make sense?

Well, as long as you want to support more than just the very recent GNU
Emacs versions, I don't see a better approach.  But yes, it's
problematic.

> If the modes don't agree with core-policy, wouldn't it be better to
> reach a solution for all by discussion and decision-making?

Yes, in theory, and I think we are very careful in decision-making.
However, package developers cannot decide which kind and version of
emacs their users are using.  Most of them use the version that ships
with their distro/OS.  For example, OSX still ships emacs 22 and that
won't change because AFAIK they don't ship GPLv3 software (and therefore
they also ship an ancient version of bash).

Long story short: keeping backward compatibility is horrible, horrible,
horrible, and the world would be a better place if just everybody would
use the current emacs version. ;-)

Bye,
Tassilo