From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: Christian Ohler Newsgroups: gmane.emacs.devel Subject: Re: testing framework and package.el Date: Sun, 03 Oct 2010 21:51:18 +1100 Message-ID: <4CA86026.1020000@fastmail.net> References: <87tyladfnf.fsf@gmx.de> <4CA5DBA2.4050102@fastmail.net> <87tyl5i737.fsf@gmx.de> NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Trace: dough.gmane.org 1286103110 25824 80.91.229.12 (3 Oct 2010 10:51:50 GMT) X-Complaints-To: usenet@dough.gmane.org NNTP-Posting-Date: Sun, 3 Oct 2010 10:51:50 +0000 (UTC) Cc: Stefan Monnier , emacs-devel Mailinglist To: Sebastian Rose Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Sun Oct 03 12:51:47 2010 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([199.232.76.165]) by lo.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1P2MAP-0006ZC-Bv for ged-emacs-devel@m.gmane.org; Sun, 03 Oct 2010 12:51:41 +0200 Original-Received: from localhost ([127.0.0.1]:47581 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1P2MAO-0006Tn-Pd for ged-emacs-devel@m.gmane.org; Sun, 03 Oct 2010 06:51:40 -0400 Original-Received: from [140.186.70.92] (port=41253 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1P2MAB-0006Rf-32 for emacs-devel@gnu.org; Sun, 03 Oct 2010 06:51:28 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1P2MA9-0007QP-H1 for emacs-devel@gnu.org; Sun, 03 Oct 2010 06:51:26 -0400 Original-Received: from out5.smtp.messagingengine.com ([66.111.4.29]:49037) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1P2MA9-0007Pk-Dp for emacs-devel@gnu.org; Sun, 03 Oct 2010 06:51:25 -0400 Original-Received: from compute3.internal (compute3.nyi.mail.srv.osa [10.202.2.43]) by gateway1.messagingengine.com (Postfix) with ESMTP id 95FB6631; Sun, 3 Oct 2010 06:51:23 -0400 (EDT) Original-Received: from frontend1.messagingengine.com ([10.202.2.160]) by compute3.internal (MEProxy); Sun, 03 Oct 2010 06:51:23 -0400 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=messagingengine.com; h=message-id:date:from:mime-version:to:cc:subject:references:in-reply-to:content-type:content-transfer-encoding; s=smtpout; bh=WO7hzJZJHTihNsLa7LksmrOgxxg=; b=Wj/9paac4BAEixrAtL4WesZX/eOzrxb9PHk7mVYJYfR8KWpFyv//ExB/VoBxlqPEyz8B2ICMl6hhHakC/PP1Zny1kG9n17zxk4bGa2eWrHvqTCXn04zEP8PgrO+WYupQCVAGmHN5Yj8SuloGM6U8E6BSc2Fl8maFnrcx/se8JsA= X-Sasl-enc: soSwBAJh4GC58Vy7MlQ3/TlOp1WJYhDWufXXz4AqGjCz 1286103081 Original-Received: from [192.168.43.184] (unknown [120.21.100.107]) by mail.messagingengine.com (Postfix) with ESMTPSA id 5BCA1402209; Sun, 3 Oct 2010 06:51:19 -0400 (EDT) In-Reply-To: <87tyl5i737.fsf@gmx.de> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Xref: news.gmane.org gmane.emacs.devel:131280 Archived-At: On 2/10/10 13:53, Sebastian Rose wrote: > 1.) Make testing a matter of one key-stroke > To ease testing, I've written two little commands, as you can see on > http://github.com/SebastianRose/org-test > > `org-test-test-current-defun' and > `org-test-test-buffer-file' > > The idea is to search the directory structure from the current source > file upwards for a directory named "tests/" if it exists. Else ask the > user. Similar to what `add-change-log-entry' does. > > Below that directory, a tree like the source tree exists: > > project > +-- lisp/ > | +-- a.el > | `-- b/ > | +-- b.el > | > `-- tests/ > +-- a.el/ > | +-- tests.el > | `-- a-defun.el > `-- b/ > +-- b.el/ > +-- tests.el > `-- b-defun.el > > If this setup exists, when editing defun-x in lisp/a.el, > `M-x org-test-test-current-defun' will load tests/a.el/defun-x.el > (fallback: tests.el there) and execute all tests with selector > "^a-defun". > `M-x org-test-test-buffer-file' in that same source file will load all > *.el files in tests/a.el/ and execute all ERT tests for selector "^a". > > This would a) recommend a certain structure for testing (what ever the > structure will be) and b) make testing a key-stroke. This structure looks like it is optimized for many test files for each file of non-test code. Right now, Emacs has 1735 .el files, and 35 files total in the tests/ directory. This ratio is not likely to invert any time soon. The structure that you describe could work, but isn't it overkill? > 2.) Create temp buffers for output > > One thing I miss is a way to create output buffers automagically (I > didn't look deeper into the sources yet). > > I know Lennart Borgman has written something for nXhtml. It's called > ert2.el and part of the nXhtml distribution (but I did not comprehend > what's going on there yet - my elisp is badbadbad :). Maybe Lennart > will help here or even care for that addition? Lennart's code is already integrated in ERT, obsoleting ert2. I remember that I wasn't quite happy with his design, but didn't have time to fix it, and since it solved an important problem, I included it as an experimental addition in ert-exp.el. > 3.) Compare output to control files > > ediff the contents of those output buffers with control files. We need > this feature to test org's exports. No one will find the time to sit > there and search hundreds of output files for errors "by hand" (once > tests exist). What do you mean by "sitting there and searching hundreds of output files for errors"? Do you have an example of tests where ERT's error reporting is too unspecific to be useful? I remember writing some tests roughly like this: (ert-deftest foo () (with-temp-buffer (insert "initial buffer contents") (... invoke code under test ...) (should (equal (ert-filter-string (buffer-string) ...) "expected final buffer contents")))) Are you sure that you want the initial/expected buffer contents in their own files? Such a split between the inputs and the invocation of the code under test seems cumbersome. I guess if your inputs/expected outputs are complex enough, moving them out into files can make sense. Can you point me to some examples where you think the data is too complex to be in-line? One limitations of my approach is that we really need some kind of "with-test-buffer" rather than "with-temp-buffer" that keeps the buffer around for inspection if the test fails, but this is item 2.) on your list already. Another limitation is that the explainer for `equal' is not as pretty as ediff, but OTOH, as soon as you want to compare text properties, ediff can't help anymore (AFAIK), while there is an explainer for `ert-equal-including-properties'. Text properties (e.g., font lock properties) are important to many kinds of tests. Both ediff and using files are problematic in such cases because they don't support text properties. Do text properties never come up in your context? Christian.