From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: "Stephen J. Turnbull" Newsgroups: gmane.emacs.devel Subject: Re: call for more ert tests Date: Tue, 25 Jun 2013 11:31:50 +0900 Message-ID: <87d2ra2a5l.fsf@uwakimon.sk.tsukuba.ac.jp> References: <838v1zjrnl.fsf@gnu.org> <8361x3jqsy.fsf@gnu.org> <8338s7jp53.fsf@gnu.org> <87bo6vb8uo.fsf@wanadoo.es> NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Trace: ger.gmane.org 1372127528 10837 80.91.229.3 (25 Jun 2013 02:32:08 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Tue, 25 Jun 2013 02:32:08 +0000 (UTC) Cc: emacs-devel@gnu.org To: =?utf-8?Q?=C3=93scar?= Fuentes Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Tue Jun 25 04:32:08 2013 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1UrJ37-0000ON-GH for ged-emacs-devel@m.gmane.org; Tue, 25 Jun 2013 04:32:05 +0200 Original-Received: from localhost ([::1]:41475 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1UrJ37-0001gp-1U for ged-emacs-devel@m.gmane.org; Mon, 24 Jun 2013 22:32:05 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:50012) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1UrJ33-0001gX-DZ for emacs-devel@gnu.org; Mon, 24 Jun 2013 22:32:02 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1UrJ31-0003S3-Fb for emacs-devel@gnu.org; Mon, 24 Jun 2013 22:32:01 -0400 Original-Received: from mgmt2.sk.tsukuba.ac.jp ([130.158.97.224]:54676) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1UrJ30-0003RL-UQ for emacs-devel@gnu.org; Mon, 24 Jun 2013 22:31:59 -0400 Original-Received: from uwakimon.sk.tsukuba.ac.jp (uwakimon.sk.tsukuba.ac.jp [130.158.99.156]) by mgmt2.sk.tsukuba.ac.jp (Postfix) with ESMTP id D5DF4970976; Tue, 25 Jun 2013 11:31:50 +0900 (JST) Original-Received: by uwakimon.sk.tsukuba.ac.jp (Postfix, from userid 1000) id 949DC11F841; Tue, 25 Jun 2013 11:31:50 +0900 (JST) In-Reply-To: <87bo6vb8uo.fsf@wanadoo.es> X-Mailer: VM undefined under 21.5 (beta32) "habanero" b0d40183ac79 XEmacs Lucid (x86_64-unknown-linux) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-Received-From: 130.158.97.224 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Xref: news.gmane.org gmane.emacs.devel:161002 Archived-At: Eli Zaretskii writes: >> Well, requiring tests for new features or fixed bugs, and confirmation >> that there are no regressions from the existing tests. > > I believe it's good, obviously. The problem is with introducing it > without losing too many contributors. It's not obviously good. It helped kill Bazaar development, for one thing. While that may be an exaggeration, it's clear that the high standards for getting patches applied discouraged many potential contributors, and the "patch pilot" program was extremely useful in getting contribution rates up. Whether a patch pilot program would be a good idea for Emacs is a separate question, but note that the core dev on patch pilot duty spent 25% of their time (full-time employees of Canonical, most of them) on that task. Not to mention the time the contributors spent on meeting the Bazaar quality standards. Does Emacs really need that high quality? After trying to get standards raised at XEmacs (where, being Number 2, we have to try harder), I've come to the conclusion that for Emacsen, worse is better. The factors that led Richard Gabriel to that conclusion are quite strong in this area of application. I think rather than pressure people about testing, code reviews are more important. Teach people to habitually write good, maintainable code. That's the cake, tests are the icing. That said, I encourage the core developers to lead by example, creating tests, even trivial ones, whenever possible. Today's general programming culture is very sympathetic to testing, even to the point of "write tests first". So new developers will pick up the idea from what the leaders do. At XEmacs, regulars (even those not associated with core development) do submit tests regularly without being asked. When people post a good replication recipe, we try to put it in the test suite immediately, crediting the poster, not the committer. (At Emacs this might occasionally raise assignment issues, but remember, it's all for freedom.) So this practice works. I also encourage you to get your buildbots green. It's hard to stay green in an active project, and as somebody mentioned about the XEmacs bot for Gnus, some platforms will stay red for a long time because nobody cares enough. But I would guess that in XEmacs modal turnaround for 'bot failures is 3-6 hours, with the tail stretching out because of sleep and $DAYJOB periods. (Part of this is because you get lazy: testing only one platform, or committing "obviously correct" fixes without testing locally, and the 'bot makes your mistake obvious. But laziness gets corrected by social pressure when there are lots of active committers.) Once the bots get green, developers take pride in keeping them that way. =C3=93scar Fuentes writes: > I wonder about the possibility of making Emacs testing an "interesting" > task. Something like a test framework which is complete and expressive > enough to allow describing testable conditions on a very high level way. That's going to be a very hard task. Testing is inherently boring work, exploring the things that shouldn't happen, and the burden is heavy, because writing good tests requires good specs, which we rarely have. So designing those specs (and maybe even writing them down) is a hidden burden of testing. > Having Lisp and an almost unlimited capability for state introspection > looks like a good start. Lisp is good. Unlimited state introspection sucks. It's very easy to write a test that fixes the implementation in stone when you use introspection. Good tests require that you distinguish between language essentials (externally visible behavior) and implementation accidents (internal state). Sometimes you do want to test intermediate results in an implementation to help debug maintenance errors on the implementation, but these tests should be marked as "subject to internal API changes". This is the kind of thing that makes testing a hard problem. John Weigley writes: > One thing that would help with that is to allow structural pattern > matching. It helps a little, but as Oscar says, we have Lisp. Extracting values from a structure should be part of the API for the structure, and fixed patterns don't help when fuzzing or iterating over all values in a sledgehammer test -- constructing the pattern becomes a PITA and unreadable, even with backquotes. Also, testing that reasonable input produces correct results (the common case for such pattern-matchers) isn't so important. More important is providing facilities to test safety, that is to express tests for correct behavior under stress (ie, whether Emacs signals an erroneous condition, and whether the right signal is raised), and for helping the buildbot stay green in the presence of bad behavior whose fix is unknown and stays that way for a while (an ExpectedFailure condition that people see as a failure when running the test, but the 'bot perceives as a pass =3D=3D SNAFU).