From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: Andy Wingo Newsgroups: gmane.lisp.guile.devel Subject: Re: our benchmark-suite Date: Wed, 16 May 2012 19:01:49 +0200 Message-ID: <8762bwhxfm.fsf@pobox.com> References: <87y5pmx0ow.fsf@pobox.com> <874ns77dh6.fsf@gnu.org> NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Trace: dough.gmane.org 1337187733 21181 80.91.229.3 (16 May 2012 17:02:13 GMT) X-Complaints-To: usenet@dough.gmane.org NNTP-Posting-Date: Wed, 16 May 2012 17:02:13 +0000 (UTC) Cc: guile-devel@gnu.org To: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) Original-X-From: guile-devel-bounces+guile-devel=m.gmane.org@gnu.org Wed May 16 19:02:12 2012 Return-path: Envelope-to: guile-devel@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1SUhc1-0004HV-4M for guile-devel@m.gmane.org; Wed, 16 May 2012 19:02:09 +0200 Original-Received: from localhost ([::1]:48360 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1SUhc0-0002Bv-In for guile-devel@m.gmane.org; Wed, 16 May 2012 13:02:08 -0400 Original-Received: from eggs.gnu.org ([208.118.235.92]:46743) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1SUhbs-0002Aq-GT for guile-devel@gnu.org; Wed, 16 May 2012 13:02:06 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1SUhbq-0004It-1L for guile-devel@gnu.org; Wed, 16 May 2012 13:02:00 -0400 Original-Received: from a-pb-sasl-sd.pobox.com ([74.115.168.62]:35266 helo=sasl.smtp.pobox.com) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1SUhbp-0004I3-Nk; Wed, 16 May 2012 13:01:57 -0400 Original-Received: from sasl.smtp.pobox.com (unknown [127.0.0.1]) by a-pb-sasl-sd.pobox.com (Postfix) with ESMTP id 2D6EA9207; Wed, 16 May 2012 13:01:54 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=pobox.com; h=from:to:cc :subject:references:date:in-reply-to:message-id:mime-version :content-type:content-transfer-encoding; s=sasl; bh=zN9Q1q950mkh 9S/YZcF60nkJuoY=; b=sZmDAVomOUiHHbq4qB8CLSKIzjlUe+vYwvBLgNPUGy5r YzxjloLtNr7ST3JpH5s9DYbfEJWwUP7TQ0snIxGWS02ry0lruDzsIXaTHZHLn6ny 8qmJKG0TlNol4fSFANVMEgcmMtDaWrlSFH41v1xREpsuJWDXS+8cIMZvrsga1W4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=pobox.com; h=from:to:cc :subject:references:date:in-reply-to:message-id:mime-version :content-type:content-transfer-encoding; q=dns; s=sasl; b=ocKde2 E7kDvxFmc68Q2AQdn+UGYCQAbo/GPFuTBVMikIV2veHXsIDUVJvMfWba+bWJGMAM FGhsx0a/UzMIf16b/c3tLlorYxqywwWVgycfkU13i4wtl9T1MHgF6LpX51EVIj59 QK/JM3f4mUQpgcOL0l46andgifLKm+BcPtrow= Original-Received: from a-pb-sasl-sd.pobox.com (unknown [127.0.0.1]) by a-pb-sasl-sd.pobox.com (Postfix) with ESMTP id 248419205; Wed, 16 May 2012 13:01:54 -0400 (EDT) Original-Received: from badger (unknown [85.50.188.176]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by a-pb-sasl-sd.pobox.com (Postfix) with ESMTPSA id 4FC1291FB; Wed, 16 May 2012 13:01:52 -0400 (EDT) In-Reply-To: <874ns77dh6.fsf@gnu.org> ("Ludovic =?utf-8?Q?Court=C3=A8s=22'?= =?utf-8?Q?s?= message of "Wed, 25 Apr 2012 22:39:33 +0200") User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.4 (gnu/linux) X-Pobox-Relay-ID: D5822988-9F78-11E1-BE82-E981AF15ED39-02397024!a-pb-sasl-sd.pobox.com X-detected-operating-system: by eggs.gnu.org: Solaris 10 (beta) X-Received-From: 74.115.168.62 X-BeenThere: guile-devel@gnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Developers list for Guile, the GNU extensibility library" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guile-devel-bounces+guile-devel=m.gmane.org@gnu.org Original-Sender: guile-devel-bounces+guile-devel=m.gmane.org@gnu.org Xref: news.gmane.org gmane.lisp.guile.devel:14472 Archived-At: Howdy! On Wed 25 Apr 2012 22:39, ludo@gnu.org (Ludovic Court=C3=A8s) writes: >> So, those are the problems: benchmarks running for inappropriate, >> inconsistent durations; > > I don=E2=80=99t really see such a problem. It doesn=E2=80=99t matter to = me if > =E2=80=98arithmetic.bm=E2=80=99 takes 2mn while =E2=80=98vlists.bm=E2=80= =99 takes 40s, since I=E2=80=99m not > comparing them. Running a benchmark for 2 minutes is not harmful to the results, but it is a bit needless. One second is enough. However, running a benchmark for just a few milliseconds is not very interesting: ;; ("if.bm: if--then: executing then" 330000 real 0.011994627 real/it= eration 3.63473545454545e-8 run/iteration 3.62829060606061e-8 core/iteratio= n 9.61427360606058e-10 gc 0.0) That's 12 milliseconds. The jitter there is too much. >> inappropriate benchmarks; > > I agree that things like =E2=80=98if.bm=E2=80=99 are not very relevant no= w. But there > are also appropriate benchmarks, and benchmarks are always better than > wild guess. ;-) Agreed :-) >> and benchmarks being optimized out. > > That should be fixed. In what way? It would make those benchmarks different. Thesis: anything for which you would want to turn off the optimizer is not a good benchmark anyway. See also: http://www.azulsystems.com/presentations/art-of-java-benchmarking >> My proposal is to rebase the iteration count in 0-reference.bm to run >> for 0.5s on some modern machine, and adjust all benchmarks to match, >> removing those benchmarks that do not measure anything useful. > > Sounds good. However, adjusting iteration counts of the benchmarks > themselves should be done rarely, as it breaks performance tracking like > . I think we've established that this isn't the case -- modulo the effect that such a change would have on GC (process image size, etc) >> Finally we should perhaps enable automatic scaling of the iteration >> count. What do folks think about that? >> >> On the positive side, all of our benchmarks are very clear that they are >> a time per number of iterations, and so this change should not affect >> users that measure time per iteration. > > If the reported time is divided by the global iteration count, then > automatic scaling of the global iteration count would be good, yes. OK, will do. Speak now or be surprised by a commit! ;-) Andy --=20 http://wingolog.org/