From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: Neil Jerram Newsgroups: gmane.lisp.guile.devel Subject: Re: our benchmark-suite Date: Sat, 28 Apr 2012 23:09:35 +0200 Message-ID: <87aa1vzhps.fsf@neil-laptop.ossau.uklinux.net> References: <87y5pmx0ow.fsf@pobox.com> <874ns77dh6.fsf@gnu.org> NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Trace: dough.gmane.org 1335647410 13071 80.91.229.3 (28 Apr 2012 21:10:10 GMT) X-Complaints-To: usenet@dough.gmane.org NNTP-Posting-Date: Sat, 28 Apr 2012 21:10:10 +0000 (UTC) Cc: guile-devel@gnu.org To: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) Original-X-From: guile-devel-bounces+guile-devel=m.gmane.org@gnu.org Sat Apr 28 23:10:09 2012 Return-path: Envelope-to: guile-devel@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1SOEu8-0008MK-G2 for guile-devel@m.gmane.org; Sat, 28 Apr 2012 23:10:08 +0200 Original-Received: from localhost ([::1]:47792 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1SOEu7-0000lO-Pq for guile-devel@m.gmane.org; Sat, 28 Apr 2012 17:10:07 -0400 Original-Received: from eggs.gnu.org ([208.118.235.92]:39690) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1SOEu4-0000jU-GA for guile-devel@gnu.org; Sat, 28 Apr 2012 17:10:05 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1SOEu2-0007e6-HX for guile-devel@gnu.org; Sat, 28 Apr 2012 17:10:04 -0400 Original-Received: from out1.ip09ir2.opaltelecom.net ([62.24.128.245]:56542) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1SOEu2-0007aR-BS; Sat, 28 Apr 2012 17:10:02 -0400 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AgAFAH1cnE9OlXc8/2dsb2JhbABEFoVSrDCBCIIJAQEEASMzIxAIAxoCBSECAg8BBCU3iAgJB6ZAkh2BL45bgRgElw+KMYUAgmo X-IronPort-AV: E=Sophos;i="4.75,498,1330905600"; d="scan'208";a="513059178" Original-Received: from host-78-149-119-60.as13285.net (HELO arudy) ([78.149.119.60]) by out1.ip09ir2.opaltelecom.net with ESMTP; 28 Apr 2012 22:09:39 +0100 Original-Received: from neil-laptop.ossau.uklinux.net.ossau.homelinux.net (AToulouse-158-1-62-197.w90-50.abo.wanadoo.fr [90.50.197.197]) by arudy (Postfix) with ESMTPA id 163A038071; Sat, 28 Apr 2012 22:09:37 +0100 (BST) In-Reply-To: <874ns77dh6.fsf@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\= \=\?utf-8\?Q\?s\?\= message of "Wed, 25 Apr 2012 22:39:33 +0200") User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.4 (gnu/linux) X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 62.24.128.245 X-BeenThere: guile-devel@gnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Developers list for Guile, the GNU extensibility library" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guile-devel-bounces+guile-devel=m.gmane.org@gnu.org Original-Sender: guile-devel-bounces+guile-devel=m.gmane.org@gnu.org Xref: news.gmane.org gmane.lisp.guile.devel:14318 Archived-At: ludo@gnu.org (Ludovic Court=C3=A8s) writes: >> My proposal is to rebase the iteration count in 0-reference.bm to run >> for 0.5s on some modern machine, and adjust all benchmarks to match, >> removing those benchmarks that do not measure anything useful. > > Sounds good. However, adjusting iteration counts of the benchmarks > themselves should be done rarely, as it breaks performance tracking like > . > >> Finally we should perhaps enable automatic scaling of the iteration >> count. What do folks think about that? >> >> On the positive side, all of our benchmarks are very clear that they are >> a time per number of iterations, and so this change should not affect >> users that measure time per iteration. > > If the reported time is divided by the global iteration count, then > automatic scaling of the global iteration count would be good, yes. For http://ossau.homelinux.net/~neil I do still have all of the raw data including iteration counts, so I could easily implement dividing by the iteration count, and hence allow for future iteration count changes. Is there any downside from doing that? (I don't think so.) Regards, Neil