From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: Neil Jerram Newsgroups: gmane.lisp.guile.user,gmane.lisp.guile.devel Subject: Benchmarks and performance tracking Date: Wed, 16 Apr 2008 21:50:21 +0100 Message-ID: <87r6d5o776.fsf@ossau.uklinux.net> NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Trace: ger.gmane.org 1208379625 11087 80.91.229.12 (16 Apr 2008 21:00:25 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Wed, 16 Apr 2008 21:00:25 +0000 (UTC) To: Guile Development , Guile User List Original-X-From: guile-user-bounces+guile-user=m.gmane.org@gnu.org Wed Apr 16 23:00:49 2008 connect(): Connection refused Return-path: Envelope-to: guile-user@m.gmane.org Original-Received: from lists.gnu.org ([199.232.76.165]) by lo.gmane.org with esmtp (Exim 4.50) id 1JmEbB-0007BU-TD for guile-user@m.gmane.org; Wed, 16 Apr 2008 22:51:22 +0200 Original-Received: from localhost ([127.0.0.1] helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1JmEaX-0007yh-9G for guile-user@m.gmane.org; Wed, 16 Apr 2008 16:50:41 -0400 Original-Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1JmEaL-0007vB-Ll for guile-user@gnu.org; Wed, 16 Apr 2008 16:50:29 -0400 Original-Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1JmEaI-0007sS-7k for guile-user@gnu.org; Wed, 16 Apr 2008 16:50:28 -0400 Original-Received: from [199.232.76.173] (helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1JmEaF-0007qR-Nm; Wed, 16 Apr 2008 16:50:23 -0400 Original-Received: from mail3.uklinux.net ([80.84.72.33]) by monty-python.gnu.org with esmtp (Exim 4.60) (envelope-from ) id 1JmEaF-00070J-8r; Wed, 16 Apr 2008 16:50:23 -0400 Original-Received: from arudy (host86-145-183-175.range86-145.btcentralplus.com [86.145.183.175]) by mail3.uklinux.net (Postfix) with ESMTP id 85DF41F67CB; Wed, 16 Apr 2008 21:50:22 +0100 (BST) Original-Received: from laruns (laruns [192.168.0.10]) by arudy (Postfix) with ESMTP id 661E23800D; Wed, 16 Apr 2008 21:50:21 +0100 (BST) User-Agent: Gnus/5.110006 (No Gnus v0.6) Emacs/21.4 (gnu/linux) X-detected-kernel: by monty-python.gnu.org: Linux 2.4-2.6 X-BeenThere: guile-user@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: General Guile related discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Original-Sender: guile-user-bounces+guile-user=m.gmane.org@gnu.org Errors-To: guile-user-bounces+guile-user=m.gmane.org@gnu.org Xref: news.gmane.org gmane.lisp.guile.user:6538 gmane.lisp.guile.devel:7171 Archived-At: I think it would be good if we could track Guile's performance better, and how it changes over time. But... 1. We don't currently have many benchmarks. There are just 3 in the repo, and they're all pretty trivial. 2. I have no experience in, and no immediate feel for, how we manage the output from benchmarks. I'm thinking of things like - needing to run benchmarks multiple times, in order to average out effects like the code being cached in memory - different people running the benchmarks in different environments (architectures, chip speeds etc.), and whether the results from those environments can be sensibly merged or collated - how best to present results, and show how they change over time (i.e. with Guile releases). (And I imagine there are other problems and subtleties that I haven't even thought of...) So, any help or advice on these points would be much appreciated! If anyone has some interesting benchmark code that they'd like to contribute, or wants to write new benchmarks, please do so. Or if there are standard Scheme benchmarks that we could incorporate, please point them out. And then if anyone has expertise on actually running benchmarks regularly, and collating and presenting the results, and would like to contribute that, please go ahead! Many thanks, Neil