Benchmarking queries

Benchmarking queries

[Neil Jerram]

[-- Attachment #1: Type: text/plain, Size: 1491 bytes --]

I had to make some changes to the benchmark code, to persuade
benchmark-guile to run to completion within a reasonable time.  Full
patch is below for review; there are 3 particular points/queries.

1. The files with an Emacs "coding: latin-1" comment can only be
compiled if I add an extra comment "coding: latin1".  In other words it
seems that Guile (=> libunistring) only understands latin1 without the
hyphen.  Is this expected, and is there anything we can do about it?

2. I had to divide the number of iterations for most of the read.bm
benchmarks by 10.  Has the reader become massively slower than when
read.bm was first written?

3. In `(defmacro benchmark ...)' in benchmark/lib.scm I had to change
`,run-benchmark' to `run-benchmark'.  This makes sense, and the `,xxx'
trick was always a bit of a hack (for when we did not have fully
hygienic macros) - but I'm wondering if there is now any nice way of
writing a defmacro like this - i.e. which uses a procedure defined in
the same module - that will work for both 1.8 and 1.9/2.0 Guile.

One solution is to export the procedure.

Another could be to use eval-case/eval-when to evaluate the required
definition during compilation, or to call defmacro differently at
compile and load time.  But I see that we only have eval-case in 1.8,
and only eval-when in 1.9.  Should we add eval-case to 1.9, so that we
have a mechanism for addressing cases like this where the 1.8 and
1.9/2.0 code has to be different?

Regards,
      Neil


[-- Attachment #2: 0001-Benchmarking.patch --]
[-- Type: text/x-diff, Size: 15035 bytes --]

From 488e19f0e568758ac204dce2999350b168b508e9 Mon Sep 17 00:00:00 2001
From: Neil Jerram <neil@ossau.uklinux.net>
Date: Sat, 19 Sep 2009 01:23:48 +0100
Subject: [PATCH] Benchmarking

---
 benchmark-suite/benchmarks/bytevectors.bm |    2 +-
 benchmark-suite/benchmarks/chars.bm       |    2 +-
 benchmark-suite/benchmarks/read.bm        |    8 +-
 benchmark-suite/benchmarks/srfi-13.bm     |    2 +-
 benchmark-suite/lib.scm                   |    2 +-
 benchmark-suite/results/neil-arudy        |   93 +++++++++++++++++++++++++++++
 6 files changed, 101 insertions(+), 8 deletions(-)
 create mode 100644 benchmark-suite/results/neil-arudy

diff --git a/benchmark-suite/benchmarks/bytevectors.bm b/benchmark-suite/benchmarks/bytevectors.bm
index a686a08..06f23ef 100644
--- a/benchmark-suite/benchmarks/bytevectors.bm
+++ b/benchmark-suite/benchmarks/bytevectors.bm
@@ -1,4 +1,4 @@
-;;; -*- mode: scheme; coding: latin-1; -*-
+;;; coding: latin1 -*- mode: scheme; coding: latin-1; -*-
 ;;; R6RS Byte Vectors.
 ;;;
 ;;; Copyright 2009  Ludovic Courtès <ludo@gnu.org>
diff --git a/benchmark-suite/benchmarks/chars.bm b/benchmark-suite/benchmarks/chars.bm
index dc6ad94..6ae0e60 100644
--- a/benchmark-suite/benchmarks/chars.bm
+++ b/benchmark-suite/benchmarks/chars.bm
@@ -1,4 +1,4 @@
-;;; -*- mode: scheme; coding: latin-1; -*-
+;;; coding: latin1 -*- mode: scheme; coding: latin-1; -*-
 ;;; chars.bm
 ;;;
 ;;; Copyright (C) 2009  Free Software Foundation, Inc.
diff --git a/benchmark-suite/benchmarks/read.bm b/benchmark-suite/benchmarks/read.bm
index f11ca68..7bc8150 100644
--- a/benchmark-suite/benchmarks/read.bm
+++ b/benchmark-suite/benchmarks/read.bm
@@ -49,14 +49,14 @@
   (benchmark "_IONBF" 5  ;; this one is very slow
     (exercise-read (list _IONBF)))
 
-  (benchmark "_IOLBF" 100
+  (benchmark "_IOLBF" 10
     (exercise-read (list _IOLBF)))
 
-  (benchmark "_IOFBF 4096" 100
+  (benchmark "_IOFBF 4096" 10
     (exercise-read (list _IOFBF 4096)))
 
-  (benchmark "_IOFBF 8192" 100
+  (benchmark "_IOFBF 8192" 10
     (exercise-read (list _IOFBF 8192)))
 
-  (benchmark "_IOFBF 16384" 100
+  (benchmark "_IOFBF 16384" 10
     (exercise-read (list _IOFBF 16384))))
diff --git a/benchmark-suite/benchmarks/srfi-13.bm b/benchmark-suite/benchmarks/srfi-13.bm
index e648e2a..c6afd7e 100644
--- a/benchmark-suite/benchmarks/srfi-13.bm
+++ b/benchmark-suite/benchmarks/srfi-13.bm
@@ -1,4 +1,4 @@
-;;; -*- mode: scheme; coding: latin-1; -*-
+;;; coding: latin1 -*- mode: scheme; coding: latin-1; -*-
 ;;; srfi-13.bm
 ;;;
 ;;; Copyright (C) 2009  Free Software Foundation, Inc.
diff --git a/benchmark-suite/lib.scm b/benchmark-suite/lib.scm
index 65253c5..4a2bf38 100644
--- a/benchmark-suite/lib.scm
+++ b/benchmark-suite/lib.scm
@@ -325,7 +325,7 @@
 
 ;;; A short form for benchmarks.
 (defmacro benchmark (name iterations body . rest)
-  `(,run-benchmark ,name ,iterations (lambda () ,body ,@rest)))
+  `(run-benchmark ,name ,iterations (lambda () ,body ,@rest)))
 
 \f
 ;;;; BENCHMARK NAMES
diff --git a/benchmark-suite/results/neil-arudy b/benchmark-suite/results/neil-arudy
new file mode 100644
index 0000000..67da90a
--- /dev/null
+++ b/benchmark-suite/results/neil-arudy
@@ -0,0 +1,93 @@
+Benchmarking /home/neil/SW/Guile/git/meta/guile ...
+with GUILE_LOAD_PATH=/home/neil/SW/Guile/git/benchmark-suite
+;; running guile version 1.9.3
+;; calibrating the benchmarking framework...
+;; framework time per iteration: 5.7220458984375e-7
+("0-reference.bm: reference benchmark for iteration counts" 330000 user 0.2 benchmark 0.0111724853515625 bench/interp 0.0111724853515625 gc 0.0)
+("bytevectors.bm: ref/set!: bytevector-u8-ref" 1000000 user 0.72 benchmark 0.14779541015625 bench/interp 0.14779541015625 gc 0.0)
+("bytevectors.bm: ref/set!: bytevector-u16-ref (foreign)" 1000000 user 0.82 benchmark 0.24779541015625 bench/interp 0.24779541015625 gc 0.0)
+("bytevectors.bm: ref/set!: bytevector-u16-ref (native)" 1000000 user 0.77 benchmark 0.19779541015625 bench/interp 0.19779541015625 gc 0.0)
+("bytevectors.bm: ref/set!: bytevector-u16-native-ref" 1000000 user 0.72 benchmark 0.14779541015625 bench/interp 0.14779541015625 gc 0.0)
+("bytevectors.bm: ref/set!: bytevector-u32-ref (foreign)" 1000000 user 1.46 benchmark 0.88779541015625 bench/interp 0.88779541015625 gc 0.0)
+("bytevectors.bm: ref/set!: bytevector-u32-ref (native)" 1000000 user 0.79 benchmark 0.21779541015625 bench/interp 0.21779541015625 gc 0.0)
+("bytevectors.bm: ref/set!: bytevector-u32-native-ref" 1000000 user 0.74 benchmark 0.16779541015625 bench/interp 0.16779541015625 gc 0.0)
+("bytevectors.bm: ref/set!: bytevector-u64-ref (foreign)" 1000000 user 2.95 benchmark 2.37779541015625 bench/interp 2.37779541015625 gc 0.0)
+("bytevectors.bm: ref/set!: bytevector-u64-ref (native)" 1000000 user 2.37 benchmark 1.79779541015625 bench/interp 1.79779541015625 gc 0.0)
+("bytevectors.bm: ref/set!: bytevector-u64-native-ref" 1000000 user 0.71 benchmark 0.13779541015625 bench/interp 0.13779541015625 gc 0.0)
+("bytevectors.bm: lists: bytevector->u8-list" 2000 user 9.62 benchmark 9.61885559082031 bench/interp 9.61885559082031 gc 0.0)
+("bytevectors.bm: lists: bytevector->uint-list 16-bit" 2000 user 4.67 benchmark 4.66885559082031 bench/interp 4.66885559082031 gc 0.0)
+("bytevectors.bm: lists: bytevector->uint-list 64-bit" 2000 user 6.08 benchmark 6.07885559082031 bench/interp 6.07885559082031 gc 0.0)
+("bytevectors.bm: SRFI-4: u8vector-ref" 1000000 user 1.89 benchmark 1.31779541015625 bench/interp 1.31779541015625 gc 0.0)
+("bytevectors.bm: SRFI-4: u16vector-ref" 1000000 user 1.9 benchmark 1.32779541015625 bench/interp 1.32779541015625 gc 0.0)
+("bytevectors.bm: SRFI-4: u32vector-ref" 1000000 user 2.1 benchmark 1.52779541015625 bench/interp 1.52779541015625 gc 0.0)
+("bytevectors.bm: SRFI-4: u64vector-ref" 1000000 user 1.89 benchmark 1.31779541015625 bench/interp 1.31779541015625 gc 0.0)
+("chars.bm: chars: char" 1000000 user 0.63 benchmark 0.05779541015625 bench/interp 0.05779541015625 gc 0.0)
+("chars.bm: chars: octal" 1000000 user 0.64 benchmark 0.06779541015625 bench/interp 0.06779541015625 gc 0.0)
+("chars.bm: chars: char? eq" 1000000 user 1.27 benchmark 0.69779541015625 bench/interp 0.69779541015625 gc 0.0)
+("chars.bm: chars: char=?" 1000000 user 1.55 benchmark 0.97779541015625 bench/interp 0.97779541015625 gc 0.0)
+("chars.bm: chars: char<?" 1000000 user 1.54 benchmark 0.96779541015625 bench/interp 0.96779541015625 gc 0.0)
+("chars.bm: chars: char-ci=?" 1000000 user 1.55 benchmark 0.97779541015625 bench/interp 0.97779541015625 gc 0.0)
+("chars.bm: chars: char-ci<? " 1000000 user 1.56 benchmark 0.98779541015625 bench/interp 0.98779541015625 gc 0.0)
+("chars.bm: chars: char->integer" 1000000 user 1.28 benchmark 0.70779541015625 bench/interp 0.70779541015625 gc 0.0)
+("chars.bm: chars: char-alphabetic?" 1000000 user 1.45 benchmark 0.87779541015625 bench/interp 0.87779541015625 gc 0.0)
+("chars.bm: chars: char-numeric?" 1000000 user 1.46 benchmark 0.88779541015625 bench/interp 0.88779541015625 gc 0.0)
+("continuations.bm: call/cc" 300 user 0.05 benchmark 0.0498283386230469 bench/interp 0.0498283386230469 gc 0.0)
+("if.bm: if-<expr>-then-else: executing then" 330000 user 0.24 benchmark 0.0511724853515625 bench/interp 0.0511724853515625 gc 0.0)
+("if.bm: if-<expr>-then-else: executing else" 330000 user 0.23 benchmark 0.0411724853515625 bench/interp 0.0411724853515625 gc 0.0)
+("if.bm: if-<expr>-then: executing then" 330000 user 0.23 benchmark 0.0411724853515625 bench/interp 0.0411724853515625 gc 0.0)
+("if.bm: if-<expr>-then: executing else" 330000 user 0.23 benchmark 0.0411724853515625 bench/interp 0.0411724853515625 gc 0.0)
+("if.bm: if-<iloc>-then-else: executing then" 330000 user 0.24 benchmark 0.0511724853515625 bench/interp 0.0511724853515625 gc 0.0)
+("if.bm: if-<iloc>-then-else: executing else" 330000 user 0.24 benchmark 0.0511724853515625 bench/interp 0.0511724853515625 gc 0.0)
+("if.bm: if-<iloc>-then: executing then" 330000 user 0.24 benchmark 0.0511724853515625 bench/interp 0.0511724853515625 gc 0.0)
+("if.bm: if-<iloc>-then: executing else" 330000 user 0.24 benchmark 0.0511724853515625 bench/interp 0.0511724853515625 gc 0.0)
+("if.bm: if-<bool>-then-else: executing then" 330000 user 0.23 benchmark 0.0411724853515625 bench/interp 0.0411724853515625 gc 0.0)
+("if.bm: if-<bool>-then-else: executing else" 330000 user 0.23 benchmark 0.0411724853515625 bench/interp 0.0411724853515625 gc 0.0)
+("if.bm: if-<bool>-then: executing then" 330000 user 0.23 benchmark 0.0411724853515625 bench/interp 0.0411724853515625 gc 0.0)
+("if.bm: if-<bool>-then: executing else" 330000 user 0.23 benchmark 0.0411724853515625 bench/interp 0.0411724853515625 gc 0.0)
+("logand.bm: bignum" 130000 user 0.48 benchmark 0.405613403320312 bench/interp 0.405613403320312 gc 0.0)
+("read.bm: read: _IONBF" 5 user 13.56 benchmark 13.5599971389771 bench/interp 13.5599971389771 gc 0.0)
+("read.bm: read: _IOLBF" 10 user 22.1 benchmark 22.0999942779541 bench/interp 22.0999942779541 gc 0.0)
+("read.bm: read: _IOFBF 4096" 10 user 22.09 benchmark 22.0899942779541 bench/interp 22.0899942779541 gc 0.0)
+("read.bm: read: _IOFBF 8192" 10 user 22.06 benchmark 22.0599942779541 bench/interp 22.0599942779541 gc 0.0)
+("read.bm: read: _IOFBF 16384" 10 user 22.07 benchmark 22.0699942779541 bench/interp 22.0699942779541 gc 0.0)
+("srfi-13.bm: strings: predicates: string?" 1190000 user 3.11 benchmark 2.42907653808594 bench/interp 2.42907653808594 gc 0.0)
+("srfi-13.bm: strings: predicates: null?" 969000 user 2.66 benchmark 2.10553375244141 bench/interp 2.10553375244141 gc 0.0)
+("srfi-13.bm: strings: predicates: any" 94000 user 1.93 benchmark 1.87621276855469 bench/interp 1.87621276855469 gc 0.0)
+("srfi-13.bm: strings: predicates: every" 94000 user 1.6 benchmark 1.54621276855469 bench/interp 1.54621276855469 gc 0.0)
+("srfi-13.bm: strings: constructors: string" 5000 user 2.13 benchmark 2.12713897705078 bench/interp 2.12713897705078 gc 0.0)
+("srfi-13.bm: strings: constructors: list->" 4500 user 0.3 benchmark 0.297425079345703 bench/interp 0.297425079345703 gc 0.0)
+("srfi-13.bm: strings: constructors: reverse-list->" 5000 user 0.6 benchmark 0.597138977050781 bench/interp 0.597138977050781 gc 0.0)
+("srfi-13.bm: strings: constructors: make" 22000 user 0.53 benchmark 0.517411499023438 bench/interp 0.517411499023438 gc 0.0)
+("srfi-13.bm: strings: constructors: tabulate" 17000 user 0.55 benchmark 0.540272521972656 bench/interp 0.540272521972656 gc 0.0)
+("srfi-13.bm: strings: constructors: join" 5500 user 0.5 benchmark 0.496852874755859 bench/interp 0.496852874755859 gc 0.0)
+("srfi-13.bm: strings: list/string: ->list" 7300 user 2.49 benchmark 2.48582290649414 bench/interp 2.48582290649414 gc 0.0)
+("srfi-13.bm: strings: list/string: split" 60000 user 1.93 benchmark 1.89566772460937 bench/interp 1.89566772460937 gc 0.0)
+("srfi-13.bm: strings: selection: ref" 660 user 1.59 benchmark 1.5896223449707 bench/interp 1.5896223449707 gc 0.0)
+("srfi-13.bm: strings: selection: copy" 1100 user 0.05 benchmark 0.0493705749511719 bench/interp 0.0493705749511719 gc 0.0)
+("srfi-13.bm: strings: selection: pad" 6800 user 0.17 benchmark 0.166109008789063 bench/interp 0.166109008789063 gc 0.0)
+("srfi-13.bm: strings: selection: trim trim-right trim-both" 60000 user 2.66 benchmark 2.62566772460938 bench/interp 2.62566772460938 gc 0.0)
+("srfi-13.bm: strings: modification: set!" 3000 user 1.41 benchmark 1.40828338623047 bench/interp 1.40828338623047 gc 0.0)
+("srfi-13.bm: strings: modification: sub-move!" 230000 user 1.95 benchmark 1.81839294433594 bench/interp 1.81839294433594 gc 0.0)
+("srfi-13.bm: strings: modification: fill!" 230000 user 2.35 benchmark 2.21839294433594 bench/interp 2.21839294433594 gc 0.0)
+("srfi-13.bm: strings: modification: comparison: compare compare-ci" 140000 user 2.02 benchmark 1.93989135742188 bench/interp 1.93989135742188 gc 0.0)
+("srfi-13.bm: strings: modification: comparison: hash hash-ci" 1000 user 0.29 benchmark 0.289427795410156 bench/interp 0.289427795410156 gc 0.0)
+("srfi-13.bm: strings: searching: prefix-length suffix-length" 270 user 0.2 benchmark 0.199845504760742 bench/interp 0.199845504760742 gc 0.0)
+("srfi-13.bm: strings: searching: prefix? suffix?" 270 user 0.21 benchmark 0.209845504760742 bench/interp 0.209845504760742 gc 0.0)
+("srfi-13.bm: strings: searching: index index-right rindex" 100000 user 3.53 benchmark 3.47277954101562 bench/interp 3.47277954101562 gc 0.0)
+("srfi-13.bm: strings: searching: skip skip-right?" 100000 user 2.14 benchmark 2.08277954101563 bench/interp 2.08277954101563 gc 0.0)
+("srfi-13.bm: strings: searching: count" 10000 user 9.42 benchmark 9.41427795410156 bench/interp 9.41427795410156 gc 0.0)
+("srfi-13.bm: strings: searching: contains contains-ci" 34000 user 2.32 benchmark 2.30054504394531 bench/interp 2.30054504394531 gc 0.0)
+("srfi-13.bm: strings: searching: upcase downcase upcase! downcase!" 600 user 0.22 benchmark 0.219656677246094 bench/interp 0.219656677246094 gc 0.0)
+("srfi-13.bm: strings: readers: read token, method 1" 1200 user 1.32 benchmark 1.31931335449219 bench/interp 1.31931335449219 gc 0.0)
+("srfi-13.bm: strings: readers: read token, method 2" 1200 user 2.09 benchmark 2.08931335449219 bench/interp 2.08931335449219 gc 0.0)
+("subr.bm: subr invocation: simple subr" 700000 user 0.46 benchmark 0.059456787109375 bench/interp 0.059456787109375 gc 0.0)
+("subr.bm: subr invocation: generic subr" 700000 user 1.78 benchmark 1.37945678710938 bench/interp 1.37945678710938 gc 0.0)
+("subr.bm: subr invocation: generic subr with rest arg" 700000 user 1.49 benchmark 1.08945678710937 bench/interp 1.08945678710937 gc 0.0)
+("subr.bm: subr invocation: generic subr with rest arg and 3+ parameters" 700000 user 1.97 benchmark 1.56945678710937 bench/interp 1.56945678710937 gc 0.0)
+("subr.bm: subr application: simple subr" 700000 user 0.98 benchmark 0.579456787109375 bench/interp 0.579456787109375 gc 0.0)
+("subr.bm: subr application: generic subr" 700000 user 1.81 benchmark 1.40945678710938 bench/interp 1.40945678710938 gc 0.0)
+("subr.bm: subr application: generic subr with rest arg" 700000 user 1.53 benchmark 1.12945678710938 bench/interp 1.12945678710938 gc 0.0)
+("subr.bm: subr application: generic subr with rest arg and 3+ parameters" 700000 user 1.98 benchmark 1.57945678710937 bench/interp 1.57945678710937 gc 0.0)
+("uniform-vector-read.bm: uniform-vector-read!: uniform-vector-write" 500 user 0.11 benchmark 0.109713897705078 bench/interp 0.109713897705078 gc 0.0)
+("uniform-vector-read.bm: uniform-vector-read!: uniform-vector-read!" 500 user 0.01 benchmark 0.00971389770507813 bench/interp 0.00971389770507813 gc 0.0)
+("uniform-vector-read.bm: uniform-vector-read!: string port" 5000 user 3.05 benchmark 3.04713897705078 bench/interp 3.04713897705078 gc 0.0)
-- 
1.5.6.5

Re: Benchmarking queries

[Mike Gran]

On Sat, 2009-09-19 at 01:59 +0100, Neil Jerram wrote:

> 1. The files with an Emacs "coding: latin-1" comment can only be
> compiled if I add an extra comment "coding: latin1".  In other words it
> seems that Guile (=> libunistring) only understands latin1 without the
> hyphen.  Is this expected, and is there anything we can do about it?

In the string_abstraction2 branch, I'd added functionality to translate
some common EMACS names for encodings into the proper linunistring names
for encodings, but, that hasn't made its way into master.  For latin-1,
the name libunistring expects is ISO-8859-1.  So it was my bad for not
changing latin-1 to iso-8859-1 when I merged it from string_abstraction2
to master.

I haven't put the Emacs aliases into master yet because I want that list
to be in scheme code in ice-9 instead of in C code.  I haven't really
thought through the details of how that should be done.

Thanks,

Mike

Re: Benchmarking queries

[Neil Jerram]

Mike Gran <spk121@yahoo.com> writes:

> On Sat, 2009-09-19 at 01:59 +0100, Neil Jerram wrote:
>
>> 1. The files with an Emacs "coding: latin-1" comment can only be
>> compiled if I add an extra comment "coding: latin1".  In other words it
>> seems that Guile (=> libunistring) only understands latin1 without the
>> hyphen.  Is this expected, and is there anything we can do about it?
>
> In the string_abstraction2 branch, I'd added functionality to translate
> some common EMACS names for encodings into the proper linunistring names
> for encodings, but, that hasn't made its way into master.  For latin-1,
> the name libunistring expects is ISO-8859-1.  So it was my bad for not
> changing latin-1 to iso-8859-1 when I merged it from string_abstraction2
> to master.

So do you mean that

;;; -*- mode: scheme; coding: iso-8859-1; -*-

should work for both Emacs and Guile?  In any case

;;; coding: latin1 -*- mode: scheme; coding: latin-1; -*-

doesn't seem so bad, so I think I'll commit for the time being with
this.

> I haven't put the Emacs aliases into master yet because I want that list
> to be in scheme code in ice-9 instead of in C code.  I haven't really
> thought through the details of how that should be done.

No problem; thanks for the explanation!

     Neil

Re: Benchmarking queries

[Mike Gran]

> So do you mean that
> 
> ;;; -*- mode: scheme; coding: iso-8859-1; -*-
> 
> should work for both Emacs and Guile?

Yes.  In http://www.iana.org/assignments/character-sets the names
of encodings labeled as "preferred MIME name" are likely to be
understood by libunistring.  There is some overlap with the
emacs encodings.  Until I get some aliases in there, the 
encodings "utf-8" and "iso-8859-1" through "iso-8859-15" are
good choices that will work in both Emacs and Guile.

Thanks,

Mike