From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: Neil Jerram Newsgroups: gmane.lisp.guile.devel Subject: Benchmarking queries Date: Sat, 19 Sep 2009 01:59:09 +0100 Message-ID: <87my4rda1u.fsf@ossau.uklinux.net> NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Trace: ger.gmane.org 1253322018 23800 80.91.229.12 (19 Sep 2009 01:00:18 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Sat, 19 Sep 2009 01:00:18 +0000 (UTC) To: Guile Development Original-X-From: guile-devel-bounces+guile-devel=m.gmane.org@gnu.org Sat Sep 19 03:00:11 2009 Return-path: Envelope-to: guile-devel@m.gmane.org Original-Received: from lists.gnu.org ([199.232.76.165]) by lo.gmane.org with esmtp (Exim 4.50) id 1MooJ7-00039u-7r for guile-devel@m.gmane.org; Sat, 19 Sep 2009 03:00:11 +0200 Original-Received: from localhost ([127.0.0.1]:55643 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1MooJ6-0002Za-PT for guile-devel@m.gmane.org; Fri, 18 Sep 2009 21:00:08 -0400 Original-Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1MooIt-0002Wk-5g for guile-devel@gnu.org; Fri, 18 Sep 2009 20:59:55 -0400 Original-Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1MooIp-0002Tk-6y for guile-devel@gnu.org; Fri, 18 Sep 2009 20:59:54 -0400 Original-Received: from [199.232.76.173] (port=55644 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1MooIo-0002TZ-Kw for guile-devel@gnu.org; Fri, 18 Sep 2009 20:59:50 -0400 Original-Received: from mail3.uklinux.net ([80.84.72.33]:39640) by monty-python.gnu.org with esmtp (Exim 4.60) (envelope-from ) id 1MooIn-0007Rn-LZ for guile-devel@gnu.org; Fri, 18 Sep 2009 20:59:50 -0400 Original-Received: from arudy (host86-147-112-99.range86-147.btcentralplus.com [86.147.112.99]) by mail3.uklinux.net (Postfix) with ESMTP id E9FDD1F66FF for ; Sat, 19 Sep 2009 01:59:10 +0100 (BST) Original-Received: from arudy (arudy [127.0.0.1]) by arudy (Postfix) with ESMTP id B7AA638023 for ; Sat, 19 Sep 2009 01:59:09 +0100 (BST) User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.1 (gnu/linux) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.4-2.6 X-BeenThere: guile-devel@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Developers list for Guile, the GNU extensibility library" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Original-Sender: guile-devel-bounces+guile-devel=m.gmane.org@gnu.org Errors-To: guile-devel-bounces+guile-devel=m.gmane.org@gnu.org Xref: news.gmane.org gmane.lisp.guile.devel:9356 Archived-At: --=-=-= I had to make some changes to the benchmark code, to persuade benchmark-guile to run to completion within a reasonable time. Full patch is below for review; there are 3 particular points/queries. 1. The files with an Emacs "coding: latin-1" comment can only be compiled if I add an extra comment "coding: latin1". In other words it seems that Guile (=> libunistring) only understands latin1 without the hyphen. Is this expected, and is there anything we can do about it? 2. I had to divide the number of iterations for most of the read.bm benchmarks by 10. Has the reader become massively slower than when read.bm was first written? 3. In `(defmacro benchmark ...)' in benchmark/lib.scm I had to change `,run-benchmark' to `run-benchmark'. This makes sense, and the `,xxx' trick was always a bit of a hack (for when we did not have fully hygienic macros) - but I'm wondering if there is now any nice way of writing a defmacro like this - i.e. which uses a procedure defined in the same module - that will work for both 1.8 and 1.9/2.0 Guile. One solution is to export the procedure. Another could be to use eval-case/eval-when to evaluate the required definition during compilation, or to call defmacro differently at compile and load time. But I see that we only have eval-case in 1.8, and only eval-when in 1.9. Should we add eval-case to 1.9, so that we have a mechanism for addressing cases like this where the 1.8 and 1.9/2.0 code has to be different? Regards, Neil --=-=-= Content-Type: text/x-diff; charset=iso-8859-1 Content-Disposition: attachment; filename=0001-Benchmarking.patch Content-Transfer-Encoding: quoted-printable >From 488e19f0e568758ac204dce2999350b168b508e9 Mon Sep 17 00:00:00 2001 From: Neil Jerram Date: Sat, 19 Sep 2009 01:23:48 +0100 Subject: [PATCH] Benchmarking --- benchmark-suite/benchmarks/bytevectors.bm | 2 +- benchmark-suite/benchmarks/chars.bm | 2 +- benchmark-suite/benchmarks/read.bm | 8 +- benchmark-suite/benchmarks/srfi-13.bm | 2 +- benchmark-suite/lib.scm | 2 +- benchmark-suite/results/neil-arudy | 93 +++++++++++++++++++++++++= ++++ 6 files changed, 101 insertions(+), 8 deletions(-) create mode 100644 benchmark-suite/results/neil-arudy diff --git a/benchmark-suite/benchmarks/bytevectors.bm b/benchmark-suite/be= nchmarks/bytevectors.bm index a686a08..06f23ef 100644 --- a/benchmark-suite/benchmarks/bytevectors.bm +++ b/benchmark-suite/benchmarks/bytevectors.bm @@ -1,4 +1,4 @@ -;;; -*- mode: scheme; coding: latin-1; -*- +;;; coding: latin1 -*- mode: scheme; coding: latin-1; -*- ;;; R6RS Byte Vectors. ;;; ;;; Copyright 2009 Ludovic Court=E8s diff --git a/benchmark-suite/benchmarks/chars.bm b/benchmark-suite/benchmar= ks/chars.bm index dc6ad94..6ae0e60 100644 --- a/benchmark-suite/benchmarks/chars.bm +++ b/benchmark-suite/benchmarks/chars.bm @@ -1,4 +1,4 @@ -;;; -*- mode: scheme; coding: latin-1; -*- +;;; coding: latin1 -*- mode: scheme; coding: latin-1; -*- ;;; chars.bm ;;; ;;; Copyright (C) 2009 Free Software Foundation, Inc. diff --git a/benchmark-suite/benchmarks/read.bm b/benchmark-suite/benchmark= s/read.bm index f11ca68..7bc8150 100644 --- a/benchmark-suite/benchmarks/read.bm +++ b/benchmark-suite/benchmarks/read.bm @@ -49,14 +49,14 @@ (benchmark "_IONBF" 5 ;; this one is very slow (exercise-read (list _IONBF))) =20 - (benchmark "_IOLBF" 100 + (benchmark "_IOLBF" 10 (exercise-read (list _IOLBF))) =20 - (benchmark "_IOFBF 4096" 100 + (benchmark "_IOFBF 4096" 10 (exercise-read (list _IOFBF 4096))) =20 - (benchmark "_IOFBF 8192" 100 + (benchmark "_IOFBF 8192" 10 (exercise-read (list _IOFBF 8192))) =20 - (benchmark "_IOFBF 16384" 100 + (benchmark "_IOFBF 16384" 10 (exercise-read (list _IOFBF 16384)))) diff --git a/benchmark-suite/benchmarks/srfi-13.bm b/benchmark-suite/benchm= arks/srfi-13.bm index e648e2a..c6afd7e 100644 --- a/benchmark-suite/benchmarks/srfi-13.bm +++ b/benchmark-suite/benchmarks/srfi-13.bm @@ -1,4 +1,4 @@ -;;; -*- mode: scheme; coding: latin-1; -*- +;;; coding: latin1 -*- mode: scheme; coding: latin-1; -*- ;;; srfi-13.bm ;;; ;;; Copyright (C) 2009 Free Software Foundation, Inc. diff --git a/benchmark-suite/lib.scm b/benchmark-suite/lib.scm index 65253c5..4a2bf38 100644 --- a/benchmark-suite/lib.scm +++ b/benchmark-suite/lib.scm @@ -325,7 +325,7 @@ =20 ;;; A short form for benchmarks. (defmacro benchmark (name iterations body . rest) - `(,run-benchmark ,name ,iterations (lambda () ,body ,@rest))) + `(run-benchmark ,name ,iterations (lambda () ,body ,@rest))) =20 ;;;; BENCHMARK NAMES diff --git a/benchmark-suite/results/neil-arudy b/benchmark-suite/results/n= eil-arudy new file mode 100644 index 0000000..67da90a --- /dev/null +++ b/benchmark-suite/results/neil-arudy @@ -0,0 +1,93 @@ +Benchmarking /home/neil/SW/Guile/git/meta/guile ... +with GUILE_LOAD_PATH=3D/home/neil/SW/Guile/git/benchmark-suite +;; running guile version 1.9.3 +;; calibrating the benchmarking framework... +;; framework time per iteration: 5.7220458984375e-7 +("0-reference.bm: reference benchmark for iteration counts" 330000 user 0.= 2 benchmark 0.0111724853515625 bench/interp 0.0111724853515625 gc 0.0) +("bytevectors.bm: ref/set!: bytevector-u8-ref" 1000000 user 0.72 benchmark= 0.14779541015625 bench/interp 0.14779541015625 gc 0.0) +("bytevectors.bm: ref/set!: bytevector-u16-ref (foreign)" 1000000 user 0.8= 2 benchmark 0.24779541015625 bench/interp 0.24779541015625 gc 0.0) +("bytevectors.bm: ref/set!: bytevector-u16-ref (native)" 1000000 user 0.77= benchmark 0.19779541015625 bench/interp 0.19779541015625 gc 0.0) +("bytevectors.bm: ref/set!: bytevector-u16-native-ref" 1000000 user 0.72 b= enchmark 0.14779541015625 bench/interp 0.14779541015625 gc 0.0) +("bytevectors.bm: ref/set!: bytevector-u32-ref (foreign)" 1000000 user 1.4= 6 benchmark 0.88779541015625 bench/interp 0.88779541015625 gc 0.0) +("bytevectors.bm: ref/set!: bytevector-u32-ref (native)" 1000000 user 0.79= benchmark 0.21779541015625 bench/interp 0.21779541015625 gc 0.0) +("bytevectors.bm: ref/set!: bytevector-u32-native-ref" 1000000 user 0.74 b= enchmark 0.16779541015625 bench/interp 0.16779541015625 gc 0.0) +("bytevectors.bm: ref/set!: bytevector-u64-ref (foreign)" 1000000 user 2.9= 5 benchmark 2.37779541015625 bench/interp 2.37779541015625 gc 0.0) +("bytevectors.bm: ref/set!: bytevector-u64-ref (native)" 1000000 user 2.37= benchmark 1.79779541015625 bench/interp 1.79779541015625 gc 0.0) +("bytevectors.bm: ref/set!: bytevector-u64-native-ref" 1000000 user 0.71 b= enchmark 0.13779541015625 bench/interp 0.13779541015625 gc 0.0) +("bytevectors.bm: lists: bytevector->u8-list" 2000 user 9.62 benchmark 9.6= 1885559082031 bench/interp 9.61885559082031 gc 0.0) +("bytevectors.bm: lists: bytevector->uint-list 16-bit" 2000 user 4.67 benc= hmark 4.66885559082031 bench/interp 4.66885559082031 gc 0.0) +("bytevectors.bm: lists: bytevector->uint-list 64-bit" 2000 user 6.08 benc= hmark 6.07885559082031 bench/interp 6.07885559082031 gc 0.0) +("bytevectors.bm: SRFI-4: u8vector-ref" 1000000 user 1.89 benchmark 1.3177= 9541015625 bench/interp 1.31779541015625 gc 0.0) +("bytevectors.bm: SRFI-4: u16vector-ref" 1000000 user 1.9 benchmark 1.3277= 9541015625 bench/interp 1.32779541015625 gc 0.0) +("bytevectors.bm: SRFI-4: u32vector-ref" 1000000 user 2.1 benchmark 1.5277= 9541015625 bench/interp 1.52779541015625 gc 0.0) +("bytevectors.bm: SRFI-4: u64vector-ref" 1000000 user 1.89 benchmark 1.317= 79541015625 bench/interp 1.31779541015625 gc 0.0) +("chars.bm: chars: char" 1000000 user 0.63 benchmark 0.05779541015625 benc= h/interp 0.05779541015625 gc 0.0) +("chars.bm: chars: octal" 1000000 user 0.64 benchmark 0.06779541015625 ben= ch/interp 0.06779541015625 gc 0.0) +("chars.bm: chars: char? eq" 1000000 user 1.27 benchmark 0.69779541015625 = bench/interp 0.69779541015625 gc 0.0) +("chars.bm: chars: char=3D?" 1000000 user 1.55 benchmark 0.97779541015625 = bench/interp 0.97779541015625 gc 0.0) +("chars.bm: chars: charinteger" 1000000 user 1.28 benchmark 0.7077954101= 5625 bench/interp 0.70779541015625 gc 0.0) +("chars.bm: chars: char-alphabetic?" 1000000 user 1.45 benchmark 0.8777954= 1015625 bench/interp 0.87779541015625 gc 0.0) +("chars.bm: chars: char-numeric?" 1000000 user 1.46 benchmark 0.8877954101= 5625 bench/interp 0.88779541015625 gc 0.0) +("continuations.bm: call/cc" 300 user 0.05 benchmark 0.0498283386230469 be= nch/interp 0.0498283386230469 gc 0.0) +("if.bm: if--then-else: executing then" 330000 user 0.24 benchmark 0= .0511724853515625 bench/interp 0.0511724853515625 gc 0.0) +("if.bm: if--then-else: executing else" 330000 user 0.23 benchmark 0= .0411724853515625 bench/interp 0.0411724853515625 gc 0.0) +("if.bm: if--then: executing then" 330000 user 0.23 benchmark 0.0411= 724853515625 bench/interp 0.0411724853515625 gc 0.0) +("if.bm: if--then: executing else" 330000 user 0.23 benchmark 0.0411= 724853515625 bench/interp 0.0411724853515625 gc 0.0) +("if.bm: if--then-else: executing then" 330000 user 0.24 benchmark 0= .0511724853515625 bench/interp 0.0511724853515625 gc 0.0) +("if.bm: if--then-else: executing else" 330000 user 0.24 benchmark 0= .0511724853515625 bench/interp 0.0511724853515625 gc 0.0) +("if.bm: if--then: executing then" 330000 user 0.24 benchmark 0.0511= 724853515625 bench/interp 0.0511724853515625 gc 0.0) +("if.bm: if--then: executing else" 330000 user 0.24 benchmark 0.0511= 724853515625 bench/interp 0.0511724853515625 gc 0.0) +("if.bm: if--then-else: executing then" 330000 user 0.23 benchmark 0= .0411724853515625 bench/interp 0.0411724853515625 gc 0.0) +("if.bm: if--then-else: executing else" 330000 user 0.23 benchmark 0= .0411724853515625 bench/interp 0.0411724853515625 gc 0.0) +("if.bm: if--then: executing then" 330000 user 0.23 benchmark 0.0411= 724853515625 bench/interp 0.0411724853515625 gc 0.0) +("if.bm: if--then: executing else" 330000 user 0.23 benchmark 0.0411= 724853515625 bench/interp 0.0411724853515625 gc 0.0) +("logand.bm: bignum" 130000 user 0.48 benchmark 0.405613403320312 bench/in= terp 0.405613403320312 gc 0.0) +("read.bm: read: _IONBF" 5 user 13.56 benchmark 13.5599971389771 bench/int= erp 13.5599971389771 gc 0.0) +("read.bm: read: _IOLBF" 10 user 22.1 benchmark 22.0999942779541 bench/int= erp 22.0999942779541 gc 0.0) +("read.bm: read: _IOFBF 4096" 10 user 22.09 benchmark 22.0899942779541 ben= ch/interp 22.0899942779541 gc 0.0) +("read.bm: read: _IOFBF 8192" 10 user 22.06 benchmark 22.0599942779541 ben= ch/interp 22.0599942779541 gc 0.0) +("read.bm: read: _IOFBF 16384" 10 user 22.07 benchmark 22.0699942779541 be= nch/interp 22.0699942779541 gc 0.0) +("srfi-13.bm: strings: predicates: string?" 1190000 user 3.11 benchmark 2.= 42907653808594 bench/interp 2.42907653808594 gc 0.0) +("srfi-13.bm: strings: predicates: null?" 969000 user 2.66 benchmark 2.105= 53375244141 bench/interp 2.10553375244141 gc 0.0) +("srfi-13.bm: strings: predicates: any" 94000 user 1.93 benchmark 1.876212= 76855469 bench/interp 1.87621276855469 gc 0.0) +("srfi-13.bm: strings: predicates: every" 94000 user 1.6 benchmark 1.54621= 276855469 bench/interp 1.54621276855469 gc 0.0) +("srfi-13.bm: strings: constructors: string" 5000 user 2.13 benchmark 2.12= 713897705078 bench/interp 2.12713897705078 gc 0.0) +("srfi-13.bm: strings: constructors: list->" 4500 user 0.3 benchmark 0.297= 425079345703 bench/interp 0.297425079345703 gc 0.0) +("srfi-13.bm: strings: constructors: reverse-list->" 5000 user 0.6 benchma= rk 0.597138977050781 bench/interp 0.597138977050781 gc 0.0) +("srfi-13.bm: strings: constructors: make" 22000 user 0.53 benchmark 0.517= 411499023438 bench/interp 0.517411499023438 gc 0.0) +("srfi-13.bm: strings: constructors: tabulate" 17000 user 0.55 benchmark 0= .540272521972656 bench/interp 0.540272521972656 gc 0.0) +("srfi-13.bm: strings: constructors: join" 5500 user 0.5 benchmark 0.49685= 2874755859 bench/interp 0.496852874755859 gc 0.0) +("srfi-13.bm: strings: list/string: ->list" 7300 user 2.49 benchmark 2.485= 82290649414 bench/interp 2.48582290649414 gc 0.0) +("srfi-13.bm: strings: list/string: split" 60000 user 1.93 benchmark 1.895= 66772460937 bench/interp 1.89566772460937 gc 0.0) +("srfi-13.bm: strings: selection: ref" 660 user 1.59 benchmark 1.589622344= 9707 bench/interp 1.5896223449707 gc 0.0) +("srfi-13.bm: strings: selection: copy" 1100 user 0.05 benchmark 0.0493705= 749511719 bench/interp 0.0493705749511719 gc 0.0) +("srfi-13.bm: strings: selection: pad" 6800 user 0.17 benchmark 0.16610900= 8789063 bench/interp 0.166109008789063 gc 0.0) +("srfi-13.bm: strings: selection: trim trim-right trim-both" 60000 user 2.= 66 benchmark 2.62566772460938 bench/interp 2.62566772460938 gc 0.0) +("srfi-13.bm: strings: modification: set!" 3000 user 1.41 benchmark 1.4082= 8338623047 bench/interp 1.40828338623047 gc 0.0) +("srfi-13.bm: strings: modification: sub-move!" 230000 user 1.95 benchmark= 1.81839294433594 bench/interp 1.81839294433594 gc 0.0) +("srfi-13.bm: strings: modification: fill!" 230000 user 2.35 benchmark 2.2= 1839294433594 bench/interp 2.21839294433594 gc 0.0) +("srfi-13.bm: strings: modification: comparison: compare compare-ci" 14000= 0 user 2.02 benchmark 1.93989135742188 bench/interp 1.93989135742188 gc 0.0) +("srfi-13.bm: strings: modification: comparison: hash hash-ci" 1000 user 0= .29 benchmark 0.289427795410156 bench/interp 0.289427795410156 gc 0.0) +("srfi-13.bm: strings: searching: prefix-length suffix-length" 270 user 0.= 2 benchmark 0.199845504760742 bench/interp 0.199845504760742 gc 0.0) +("srfi-13.bm: strings: searching: prefix? suffix?" 270 user 0.21 benchmark= 0.209845504760742 bench/interp 0.209845504760742 gc 0.0) +("srfi-13.bm: strings: searching: index index-right rindex" 100000 user 3.= 53 benchmark 3.47277954101562 bench/interp 3.47277954101562 gc 0.0) +("srfi-13.bm: strings: searching: skip skip-right?" 100000 user 2.14 bench= mark 2.08277954101563 bench/interp 2.08277954101563 gc 0.0) +("srfi-13.bm: strings: searching: count" 10000 user 9.42 benchmark 9.41427= 795410156 bench/interp 9.41427795410156 gc 0.0) +("srfi-13.bm: strings: searching: contains contains-ci" 34000 user 2.32 be= nchmark 2.30054504394531 bench/interp 2.30054504394531 gc 0.0) +("srfi-13.bm: strings: searching: upcase downcase upcase! downcase!" 600 u= ser 0.22 benchmark 0.219656677246094 bench/interp 0.219656677246094 gc 0.0) +("srfi-13.bm: strings: readers: read token, method 1" 1200 user 1.32 bench= mark 1.31931335449219 bench/interp 1.31931335449219 gc 0.0) +("srfi-13.bm: strings: readers: read token, method 2" 1200 user 2.09 bench= mark 2.08931335449219 bench/interp 2.08931335449219 gc 0.0) +("subr.bm: subr invocation: simple subr" 700000 user 0.46 benchmark 0.0594= 56787109375 bench/interp 0.059456787109375 gc 0.0) +("subr.bm: subr invocation: generic subr" 700000 user 1.78 benchmark 1.379= 45678710938 bench/interp 1.37945678710938 gc 0.0) +("subr.bm: subr invocation: generic subr with rest arg" 700000 user 1.49 b= enchmark 1.08945678710937 bench/interp 1.08945678710937 gc 0.0) +("subr.bm: subr invocation: generic subr with rest arg and 3+ parameters" = 700000 user 1.97 benchmark 1.56945678710937 bench/interp 1.56945678710937 g= c 0.0) +("subr.bm: subr application: simple subr" 700000 user 0.98 benchmark 0.579= 456787109375 bench/interp 0.579456787109375 gc 0.0) +("subr.bm: subr application: generic subr" 700000 user 1.81 benchmark 1.40= 945678710938 bench/interp 1.40945678710938 gc 0.0) +("subr.bm: subr application: generic subr with rest arg" 700000 user 1.53 = benchmark 1.12945678710938 bench/interp 1.12945678710938 gc 0.0) +("subr.bm: subr application: generic subr with rest arg and 3+ parameters"= 700000 user 1.98 benchmark 1.57945678710937 bench/interp 1.57945678710937 = gc 0.0) +("uniform-vector-read.bm: uniform-vector-read!: uniform-vector-write" 500 = user 0.11 benchmark 0.109713897705078 bench/interp 0.109713897705078 gc 0.0) +("uniform-vector-read.bm: uniform-vector-read!: uniform-vector-read!" 500 = user 0.01 benchmark 0.00971389770507813 bench/interp 0.00971389770507813 gc= 0.0) +("uniform-vector-read.bm: uniform-vector-read!: string port" 5000 user 3.0= 5 benchmark 3.04713897705078 bench/interp 3.04713897705078 gc 0.0) --=20 1.5.6.5 --=-=-=--