Re: Trunk still not open

[Eli Zaretskii <eliz@gnu.org>]

> Date: Sat, 15 Mar 2014 08:22:39 +0200
> From: Dmitry Gutov <dgutov@yandex.ru>
> CC: stephen@xemacs.org, emacs-devel@gnu.org
> 
> On 14.03.2014 17:29, Eli Zaretskii wrote:
> 
> > Not necessarily.  You will know how to test a function, but without
> > some overview docs, you will have no idea how to test a complex
> > feature that is built of several functions and variables, and relies
> > on some internals on top of that.  Also, internal functions many times
> > don't have doc strings -- a practice that Emacs development accepts as
> > valid.
> 
> If I'm a "testing engineer" in a separate department from people who 
> wrote the code (not the case here), then yes, I might like an overview 
> document to understand the code, or write tests for it.
> 
> "No code without tests" policy assumes that someone actually can write 
> the code changes, and they need to supply the tests that fail without that.
> 
> I.e. you both seem to be referring to some different kind of testing: 
> non-automated, performed by third parties?

At least I didn't, and I'm quite sure Stephen didn't, either.

Here's what I had in mind: It is quite rare in Emacs nowadays to write
code to implement a new feature entirely from scratch.  Usually, you
either enhance an existing feature or fix a bug in an existing
feature.  IOW, you are adding code on top of existing implementation.
Therefore, writing a good test requires you to know what the existing
code should do, because a good test will test both that you fixed
whatever problem you wanted to fix, and that you didn't break the
existing code where you didn't mean to modify existing behavior.

Therefore, you need to know what was expected of the code which you
modified.  This is exacerbated in Emacs by the lack of good coverage
by existing tests, so in many cases you will be writing a test for
modified implementation without having a test for the existing
implementation.  In those cases, your _only_ source to learn what the
existing implementation tried to do will be the docs.

> If I write automated tests, I'll have to get familiar with the code 
> either way, and it's often its own best documentation.

It's not always possible to get familiar with the code enough to
understand how it was supposed to work and what effects it was
supposed to produce, without investing unduly large or even
impractical effort.  Many times, you fix a bug or add a feature whose
effects are localized enough to make it unnecessary to study too much
surrounding or supporting code.  Sometimes, the code you modify use
some infrastructure which you understand only vaguely, but writing a
test for your changes requires to know some specific detail about
those other parts.  Etc. etc. -- I'm sure that if you dig deep enough
into your own experience, you will recall such incidents.  They all
require one to read the docs and try gleaning from there what the code
in question was supposed to do.

As another data point, consider the bug reports which say "the docs
says FOO should work such-and-such, but in fact I see a different
behavior".  IOW, people use the docs as a kind of contract, and
rightfully so.  Documenting the behavior goes a long way towards
explaining users what they should expect.

> Lisp packages often put the intro or overview in the Commentary, though.

Yes, but it's rare to have there enough stuff to cover the ground.
And if that overview is good enough, copying it into the manual and
adding the necessary markup is almost trivial.