From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: Christian Ohler Newsgroups: gmane.emacs.devel Subject: Re: testing framework and package.el Date: Tue, 05 Oct 2010 01:06:35 +0930 Message-ID: <4CA9F483.8020000@fastmail.net> References: <87tyladfnf.fsf@gmx.de> <4CA5DBA2.4050102@fastmail.net> <87tyl5i737.fsf@gmx.de> <4CA86026.1020000@fastmail.net> <87vd5jvw1q.fsf@gmx.de> NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Trace: dough.gmane.org 1286206822 21900 80.91.229.12 (4 Oct 2010 15:40:22 GMT) X-Complaints-To: usenet@dough.gmane.org NNTP-Posting-Date: Mon, 4 Oct 2010 15:40:22 +0000 (UTC) Cc: Stefan Monnier , emacs-devel Mailinglist To: Sebastian Rose Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Mon Oct 04 17:40:20 2010 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([199.232.76.165]) by lo.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1P2n9E-0007yo-1G for ged-emacs-devel@m.gmane.org; Mon, 04 Oct 2010 17:40:16 +0200 Original-Received: from localhost ([127.0.0.1]:47837 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1P2n9D-0000Co-Dc for ged-emacs-devel@m.gmane.org; Mon, 04 Oct 2010 11:40:15 -0400 Original-Received: from [140.186.70.92] (port=50294 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1P2n69-0007gJ-OU for emacs-devel@gnu.org; Mon, 04 Oct 2010 11:37:20 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1P2n5p-0000Me-Oy for emacs-devel@gnu.org; Mon, 04 Oct 2010 11:37:03 -0400 Original-Received: from out5.smtp.messagingengine.com ([66.111.4.29]:32914) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1P2n5p-0000KL-Kw for emacs-devel@gnu.org; Mon, 04 Oct 2010 11:36:45 -0400 Original-Received: from compute3.internal (compute3.nyi.mail.srv.osa [10.202.2.43]) by gateway1.messagingengine.com (Postfix) with ESMTP id B816BCC; Mon, 4 Oct 2010 11:36:43 -0400 (EDT) Original-Received: from frontend1.messagingengine.com ([10.202.2.160]) by compute3.internal (MEProxy); Mon, 04 Oct 2010 11:36:43 -0400 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=messagingengine.com; h=message-id:date:from:mime-version:to:cc:subject:references:in-reply-to:content-type:content-transfer-encoding; s=smtpout; bh=i8uC77S5TT2K3dsKfqI3LPq0y3Q=; b=DzX21Oxh26ZHT69lFITz7FwpdG3DgB/GO3ps1pF18HCMspVBEGmR3es5g18L0yaOFQhNMnfZzm9R+Yn45JMAB05ZpCPu2p8+lhAI0YeGVLvT8IzqwmjL5yiISGCVvQlUuX+5PT4bboH6MbtBo+/E0oddFELYUGWNjl0NAIxwB6c= X-Sasl-enc: bNDHSxhLke9jWXrZDQaqw/OWeW30h4MwylDpaNIfhnN/ 1286206601 Original-Received: from ohler-macbookpro.local (17.8.70.115.static.exetel.com.au [115.70.8.17]) by mail.messagingengine.com (Postfix) with ESMTPSA id 9A0C6409A07; Mon, 4 Oct 2010 11:36:40 -0400 (EDT) In-Reply-To: <87vd5jvw1q.fsf@gmx.de> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Xref: news.gmane.org gmane.emacs.devel:131324 Archived-At: On 4/10/10 6:22, Sebastian Rose wrote: > A structure like that could be enforced later > if needed. We will start with a very small project. I want to make it > easy to jump to the test-file for a given source-file and execute tests > for current defun automatically. This could be done based on such a > structure (i.e. conventions). > > I guess reading more of ERT's code I'll find other ways to find the file > a certain test (or tests matching a selector) is defined. I just started experimenting with a directory layout where ERT's code is in lisp/emacs-lisp/*.el and the tests are in test/lisp/emacs-lisp/*.el. It is indeed very clumsy to jump back and forth between the tests and the code under test. (Putting the tests in test/emacs-lisp/*.el would make no difference.) I suspect it would be easier if the convention was that a file /some/path/foo.el had its tests in /some/path/test/foo.el. The convention of having a top-level src/ and test/ split makes sense in Java because of its package file layout and visibility rules, but Emacs Lisp has no such constraints. Has anyone else thought about the pros and cons of these different conventions? Is there a deep reason why Emacs currently has test/ as a separate top-level directory? >> Lennart's code is already integrated in ERT, obsoleting ert2. I remember that I >> wasn't quite happy with his design, but didn't have time to fix it, and since it >> solved an important problem, I included it as an experimental addition in >> ert-exp.el. > OK, good. Found that, thanks. From skimming over that file I guess > that's what I was looking for and maybe even a bit more. I just rewrote the test buffer functionality. The design is now more orthogonal and flexible, but reading input from files will require an explicit call to `insert-file-contents' or similar. > Org's exports to XHTML, LaTeX etc. We could compare them to control > files. The publishing mechanism might create quite big files and even > entire file/directory trees. Currently we publish our stuff and detect > regressions when clicking a link that points nowhere or validating a > XHTML page (spot checking). I see. Comparing directory trees may be important to org-mode, but overall, it seems like a feature that would be used by the tests of only very few packages. I think you're better off writing some utility functions on top of ERT, as you have already started doing for other things. Not every test-related utility function belongs in ERT itself; it's fine to build stuff on top. The `ert-info' feature that I just implemented allows you to add custom context information that ERT will display on test failures. It's not documented in the manual yet, but it makes it easier to build the kind of test utilities that you may want, so please check it out. >> One limitations of my approach is that we really need some kind of >> "with-test-buffer" rather than "with-temp-buffer" that keeps the buffer around >> for inspection if the test fails, but this is item 2.) on your list already. >> Another limitation is that the explainer for `equal' is not as pretty as ediff, >> but OTOH, as soon as you want to compare text properties, ediff can't help >> anymore (AFAIK), while there is an explainer for >> ert-equal-including-properties'. > Also, I believe ediff is too interactive for a simple test run (diff's > exit status is 0 if no difference is found - don't know yet if and to > how ediff could be used non-interactively). I have a rough idea now how to extend ERT's "explanation" feature to allow interactive invocation of ediff to visualize differences; this seems like a good way to solve this. Tests would simply use (should (equal string1 string2)), and the explainer for `equal' would take care of the rest, putting a button in the results buffer that allows the user to invoke ediff. string1 and string2 could be the result of calling `buffer-string' on different buffers, of course. >> Text properties (e.g., font lock properties) are important to many kinds of >> tests. Both ediff and using files are problematic in such cases because they >> don't support text properties. Do text properties never come up in your >> context? > Some so some so I guess. I didn't start to write actual tests yet. > This will be done by many people (I hope) and there will be many > different types of tests (I hope). Some will involve text properties, > too. OK, that makes sense. It is important to write actual tests, though; ideas for improvement that come from experience with real-world problems carry much more weight than thought experiments. Christian.