From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: "Neil Jerram" Newsgroups: gmane.lisp.guile.devel,gmane.lisp.guile.user Subject: "Pace is nothing without guile" Date: Sun, 13 Jul 2008 18:06:17 +0100 Message-ID: <49dd78620807131006m4efc2a96v2d9778b6c4a501b4@mail.gmail.com> NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Trace: ger.gmane.org 1215968812 11483 80.91.229.12 (13 Jul 2008 17:06:52 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Sun, 13 Jul 2008 17:06:52 +0000 (UTC) To: guile-devel , guile-user@gnu.org Original-X-From: guile-devel-bounces+guile-devel=m.gmane.org@gnu.org Sun Jul 13 19:07:39 2008 Return-path: Envelope-to: guile-devel@m.gmane.org Original-Received: from lists.gnu.org ([199.232.76.165]) by lo.gmane.org with esmtp (Exim 4.50) id 1KI52u-0002j1-5D for guile-devel@m.gmane.org; Sun, 13 Jul 2008 19:07:37 +0200 Original-Received: from localhost ([127.0.0.1]:45014 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1KI51s-0004sL-Pw for guile-devel@m.gmane.org; Sun, 13 Jul 2008 13:06:32 -0400 Original-Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1KI51n-0004sC-R5 for guile-devel@gnu.org; Sun, 13 Jul 2008 13:06:27 -0400 Original-Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1KI51m-0004ri-4I for guile-devel@gnu.org; Sun, 13 Jul 2008 13:06:27 -0400 Original-Received: from [199.232.76.173] (port=42476 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1KI51l-0004rf-TH for guile-devel@gnu.org; Sun, 13 Jul 2008 13:06:25 -0400 Original-Received: from rv-out-0708.google.com ([209.85.198.242]:4111) by monty-python.gnu.org with esmtp (Exim 4.60) (envelope-from ) id 1KI51i-0005hi-Ng for guile-devel@gnu.org; Sun, 13 Jul 2008 13:06:25 -0400 Original-Received: by rv-out-0708.google.com with SMTP id k29so5169868rvb.6 for ; Sun, 13 Jul 2008 10:06:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:mime-version:content-type:content-transfer-encoding :content-disposition; bh=x3jyRH2epBK9MQ5qHXG6/5Jf7vsCGbeQHSR5jcA/vsE=; b=CXNCy7U/HoGxeUb6aHzXer0vWmjyoEGVVUrIt/OE6lMYCdhB4NGfLUbsPbcIo30LZc 9fKqmsMjs/y6+l07Lwd8DiPBkkd0snWUaMmJbzIJ79aqrdCbjnxo7N5iy6IjUUbuxIXg 3h1AeL0GogCVWa6dGvPTEXo55Y72bq1yWxGUc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=googlemail.com; s=gamma; h=message-id:date:from:to:subject:mime-version:content-type :content-transfer-encoding:content-disposition; b=Q2A/fqCuvhZGzzgESBpwOawnw0Py9CsGioWO2ZEcpCRPzazt58Bt2967Kq1TUTcOhm HOV+OIzZOCpx5/xKdCA1x4h4Grts+aAlTvB0hWFPxbGECHL0w3FbVgjwkIs+2KH/hxGs g0s4Sl8QpoN4eEM6Ohs0sHr28/KGaRREZIUW0= Original-Received: by 10.114.158.1 with SMTP id g1mr4480818wae.111.1215968777244; Sun, 13 Jul 2008 10:06:17 -0700 (PDT) Original-Received: by 10.114.197.8 with HTTP; Sun, 13 Jul 2008 10:06:17 -0700 (PDT) Content-Disposition: inline X-detected-kernel: by monty-python.gnu.org: Linux 2.6 (newer, 2) X-BeenThere: guile-devel@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Developers list for Guile, the GNU extensibility library" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Original-Sender: guile-devel-bounces+guile-devel=m.gmane.org@gnu.org Errors-To: guile-devel-bounces+guile-devel=m.gmane.org@gnu.org Xref: news.gmane.org gmane.lisp.guile.devel:7373 gmane.lisp.guile.user:6651 Archived-At: ... That's a comment from coverage of the current England v South Africa cricket match (http://uk.cricinfo.com/talk/content/current/multimedia/360921.html). But is Guile nothing without pace? Well obviously it isn't "nothing", but I think Guile is perceived, among both Scheme implementations and free scripting languages, as being a bit slow, and I think that a large part of the reason for this is that we have no systematic benchmarking. So this email is about systematic performance data. I was wondering what benchmarks we could run to get good coverage of all Guile's function, and suddenly thought "of course, the test suite!" The test suite should, by definition, provide coverage of everything that we care about. Therefore I think that we should be able to start collecting a lot of useful performance data by implementing a version of "make check" that measures and stores off the time that each test takes to run. What I'd like input/advice on, is exactly how we store and collate such data. I think the system should ideally support - arbitrary later analysis of the collected data - correlation of the result for a specific test with the exact source code of that test at the time it was run... - ...and hence, being able to work out (later) that the results changed because the content of the test changed - anyone running the tests and uploading data, not just Guile core developers - associating a set of results with the relevant information about the machine that they were obtained on (CPUs, RAM) in such a way that the information is trustable, but without invading the privacy of the uploader. So how do we do that? Perhaps the test content identification could be done by its Git (SHA-1) hash - together with the path of the repo containing that version. And I imagine that the form of the results could be a file containing lines like: ("numbers.test" SHA1-HASH REPO-PATH DATE+TIME MACHINE-INFO MEASURED-DURATION) That would allow sets of results to be concatenated for later analysis. But I'm not sure what the relevant MACHINE-INFO is and how to represent that. Any thoughts / comments / ideas? Thanks for reading! Neil