unofficial mirror of guile-devel@gnu.org 
 help / color / mirror / Atom feed
* benchmarking framework
@ 2002-07-20  2:13 Dirk Herrmann
  2002-07-20  8:44 ` Neil Jerram
                   ` (3 more replies)
  0 siblings, 4 replies; 7+ messages in thread
From: Dirk Herrmann @ 2002-07-20  2:13 UTC (permalink / raw)


Hi folks,

I have taken the freedom to add a benchmarking framework to guile.  It is
a simple adaptation from the testing framework.  A lot of code has been
copied from there.  Well, maybe at some time it would be better to extract
the common parts into a module.

Whatever, here's a short introduction:  benchmarks are placed in the
directory benchmark-suite/benchmarks.  They are ordinary scheme files, but
with the .bm extension.  At least these files are detected automatically.
Others can also be specified explicitly.

Within the benchmark files, you can introduce benchmarks with the
(benchmark ...) macro.  The syntax is as follows:
  (benchmark NAME ITERATION body...)
where NAME is a name for the benchmark, similar to the names in the
test-suite (btw, you also have with-benchmark-prefix...), ITERATION is the
number of times the body shall be executed.  And the body finally is the
code to be benchmarked.  Example:
  (define bignum (1- (expt 2 128)))
  (let* ((i 0))
    (benchmark "bignum" 130000
      (logand i bignum)
      (set! i (+ i 1))))
This will run the statements (logand i bignum) and (set! i (+ i 1))
130000 times (this, however, can be scaled - see below).  The time for
this is measured and written out.  The execution counts in the small
examples are chosen such that on my machine this results in an execution
time of about one second for each benchmark.

You start the benchmarking with the command ./benchmark-guile.  You have
the same options as with ./check-guile, except for --flag-unresolved,
which is test specific, and --test-suite is renamed to --benchmark-suite.  
There is an additional option --iteration-factor NUM, which allows to
scale the execution time for benchmarks:  As can be seen from the exmample
above, every benchmark is given an iteration count, which indicates how
often the benchmark is to be executed.  With the option --iteration-factor
NUM you can increase or decrease the execution count of the benchmarks and
thus influence the time needed for performing the benchmarks.  For
example, running the benchmark suite with --iteration-factor 0.5 will
about halven the execution time, since all benchmark's bodies are executed
about half as often.

Results are written to a log file (this file contains a lot of data) and
to the console (still a lot of data, but not quite as much).  The values
have the following meaning:
  total:     total execution time (this is what the unix time command
             reports as real time).
  user:      user time (this is reported as user time also by the unix
             time command)
  system:    system time, similar to the unix time command
  frame:     this is the part of the user time that is consumed by the
             benchmarking framework itself.  You can think of this as
             the time that would still be consumed, even if the
             benchmarking code itself was empty.  This value does not
             include the time for garbage collection.
  benchmark: This is the part of the user time that is actually spent
             within the benchmarking code.  That is, the time needed
             for the benchmarking framework is subtracted.  This value,
             however, includes all garbage collection times.
  user/interp: This is the part of the user time that is spent in the
             interpreter (and not in garbage collection)
  bench/interp: This is the part of the benchmark time that is spent in
             the interpreter (and not in garbage collection).  This
             value is most probably the one you are interested in, except
             if you are doing some garbage collection checks.
  gc:        The time spent in garbage collection.
However, there are some caveats when using the values:  The frame time is
estimated based on running an empty benchmark during startup and measuring
that time.  This can be somewhat inaccurate.  This value, however, is
then used to compute the benchmark and bench/interp times.  I don't know
about the accuracy of the other values as reported by (times) and
(gc-run-time).

Best regards, 
Dirk Herrmann


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: benchmarking framework
  2002-07-20  2:13 benchmarking framework Dirk Herrmann
@ 2002-07-20  8:44 ` Neil Jerram
  2002-07-21 12:03   ` Dirk Herrmann
  2002-07-20 12:13 ` Marius Vollmer
                   ` (2 subsequent siblings)
  3 siblings, 1 reply; 7+ messages in thread
From: Neil Jerram @ 2002-07-20  8:44 UTC (permalink / raw)
  Cc: guile-devel, guile-user

>>>>> "Dirk" == Dirk Herrmann <dirk@sallust.ida.ing.tu-bs.de> writes:

    Dirk> I have taken the freedom to add a benchmarking framework to
    Dirk> guile.

My results, in case they are interesting ...

Benchmarking /home/neil/Guile/cvs/guile-core/pre-inst-guile ...
with GUILE_LOAD_PATH=/home/neil/Guile/cvs/guile-core/benchmark-suite
;; frame time per iteration: 1.1444091796875e-6
("continuations.bm: call/cc" 300 "user:" 0.72 "benchmark:" 0.719656677246094 "bench/interp:" -3.4332275390625e-4 "gc:" 0.72)
("if.bm: if-<expr>-then-else: executing then" 330000 "user:" 0.71 "benchmark:" 0.332344970703125 "bench/interp:" 0.042344970703125 "gc:" 0.29)
("if.bm: if-<expr>-then-else: executing else" 330000 "user:" 0.73 "benchmark:" 0.352344970703125 "bench/interp:" 0.082344970703125 "gc:" 0.27)
("if.bm: if-<expr>-then: executing then" 330000 "user:" 0.71 "benchmark:" 0.332344970703125 "bench/interp:" 0.042344970703125 "gc:" 0.29)
("if.bm: if-<expr>-then: executing else" 330000 "user:" 0.72 "benchmark:" 0.342344970703125 "bench/interp:" 0.072344970703125 "gc:" 0.27)
("if.bm: if-<iloc>-then-else: executing then" 330000 "user:" 0.72 "benchmark:" 0.342344970703125 "bench/interp:" 0.062344970703125 "gc:" 0.28)
("if.bm: if-<iloc>-then-else: executing else" 330000 "user:" 0.71 "benchmark:" 0.332344970703125 "bench/interp:" 0.042344970703125 "gc:" 0.29)
("if.bm: if-<iloc>-then: executing then" 330000 "user:" 0.72 "benchmark:" 0.342344970703125 "bench/interp:" 0.062344970703125 "gc:" 0.28)
("if.bm: if-<iloc>-then: executing else" 330000 "user:" 0.7 "benchmark:" 0.322344970703125 "bench/interp:" 0.032344970703125 "gc:" 0.29)
("if.bm: if-<bool>-then-else: executing then" 330000 "user:" 0.7 "benchmark:" 0.322344970703125 "bench/interp:" 0.032344970703125 "gc:" 0.29)
("if.bm: if-<bool>-then-else: executing else" 330000 "user:" 0.7 "benchmark:" 0.322344970703125 "bench/interp:" 0.052344970703125 "gc:" 0.27)
("if.bm: if-<bool>-then: executing then" 330000 "user:" 0.69 "benchmark:" 0.312344970703125 "bench/interp:" 0.032344970703125 "gc:" 0.28)
("if.bm: if-<bool>-then: executing else" 330000 "user:" 0.69 "benchmark:" 0.312344970703125 "bench/interp:" 0.052344970703125 "gc:" 0.26)
("logand.bm: bignum" 130000 "user:" 0.88 "benchmark:" 0.731226806640625 "bench/interp:" 0.211226806640625 "gc:" 0.52)

Did you consider using symbols rather than strings for "user:",
"benchmark:" and so on?  I think that would be more readable, and
marginally easier to process programmatically as well.

Also, from the PHB department :-), is there a single headline number
that could usefully be boiled down from these results?

Regards,
        Neil


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: benchmarking framework
  2002-07-20  2:13 benchmarking framework Dirk Herrmann
  2002-07-20  8:44 ` Neil Jerram
@ 2002-07-20 12:13 ` Marius Vollmer
  2002-07-20 12:46 ` Han-Wen
  2002-07-20 19:11 ` Rob Browning
  3 siblings, 0 replies; 7+ messages in thread
From: Marius Vollmer @ 2002-07-20 12:13 UTC (permalink / raw)
  Cc: guile-devel, guile-user

Dirk Herrmann <dirk@sallust.ida.ing.tu-bs.de> writes:

> I have taken the freedom to add a benchmarking framework to guile.

Excellent! :-)

_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel


^ permalink raw reply	[flat|nested] 7+ messages in thread

* benchmarking framework
  2002-07-20  2:13 benchmarking framework Dirk Herrmann
  2002-07-20  8:44 ` Neil Jerram
  2002-07-20 12:13 ` Marius Vollmer
@ 2002-07-20 12:46 ` Han-Wen
  2002-07-20 14:01   ` Neil Jerram
  2002-07-20 19:11 ` Rob Browning
  3 siblings, 1 reply; 7+ messages in thread
From: Han-Wen @ 2002-07-20 12:46 UTC (permalink / raw)
  Cc: guile-devel, guile-user

dirk@sallust.ida.ing.tu-bs.de writes:
> I have taken the freedom to add a benchmarking framework to guile.  It is
> a simple adaptation from the testing framework.  A lot of code has been
> copied from there.  Well, maybe at some time it would be better to extract
> the common parts into a module.
> 
> Whatever, here's a short introduction:  benchmarks are placed in the
> directory benchmark-suite/benchmarks.  They are ordinary scheme files, but
> with the .bm extension.  At least these files are detected automatically.
> Others can also be specified explicitly.

there is a nice suite of scheme GC benchmarks in 
http://www.ccs.neu.edu/home/will/GC/sourcecode.html,
however, is it required that authors of these also have disclaimers at gnu?

-- 

Han-Wen Nienhuys   |   hanwen@cs.uu.nl   |   http://www.cs.uu.nl/~hanwen 

_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: benchmarking framework
  2002-07-20 12:46 ` Han-Wen
@ 2002-07-20 14:01   ` Neil Jerram
  0 siblings, 0 replies; 7+ messages in thread
From: Neil Jerram @ 2002-07-20 14:01 UTC (permalink / raw)
  Cc: Dirk Herrmann, guile-devel, guile-user

>>>>> "Han-Wen" == Han-Wen  <hanwen@cs.uu.nl> writes:

    Han-Wen> there is a nice suite of scheme GC benchmarks in 
    Han-Wen> http://www.ccs.neu.edu/home/will/GC/sourcecode.html,
    Han-Wen> however, is it required that authors of these also have disclaimers at gnu?

For distribution, yes.  However, much cooler would be not to
distribute, but instead to write a .bm file that, if connected to the
Internet, downloaded the benchmarks and then ran them.

Anyone up for this challenge?

        Neil


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: benchmarking framework
  2002-07-20  2:13 benchmarking framework Dirk Herrmann
                   ` (2 preceding siblings ...)
  2002-07-20 12:46 ` Han-Wen
@ 2002-07-20 19:11 ` Rob Browning
  3 siblings, 0 replies; 7+ messages in thread
From: Rob Browning @ 2002-07-20 19:11 UTC (permalink / raw)
  Cc: guile-devel, guile-user

Dirk Herrmann <dirk@sallust.ida.ing.tu-bs.de> writes:

> I have taken the freedom to add a benchmarking framework to guile.
> It is a simple adaptation from the testing framework.  A lot of code
> has been copied from there.  Well, maybe at some time it would be
> better to extract the common parts into a module.

That's great!  I had actually started that here too, but hadn't gotten
nearly as far.  I had also started porting some of the stalin
benchmarks to guile, so I'll look back at those.  I think they're
GPLed, but I don't think we own the copyrights, so I'm not sure
whether or not they can go in without talking to Siskind.

-- 
Rob Browning
rlb @defaultvalue.org, @linuxdevel.com, and @debian.org
Previously @cs.utexas.edu
GPG=1C58 8B2C FB5E 3F64 EA5C  64AE 78FE E5FE F0CB A0AD

_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: benchmarking framework
  2002-07-20  8:44 ` Neil Jerram
@ 2002-07-21 12:03   ` Dirk Herrmann
  0 siblings, 0 replies; 7+ messages in thread
From: Dirk Herrmann @ 2002-07-21 12:03 UTC (permalink / raw)
  Cc: guile-devel, guile-user

On 20 Jul 2002, Neil Jerram wrote:

> Did you consider using symbols rather than strings for "user:",
> "benchmark:" and so on?  I think that would be more readable, and
> marginally easier to process programmatically as well.

You are right, I have changed "user:" to 'user and so on.  I would have
liked to keep the colon for readability, but the printed form of 'user: is
even worse to read.

> Also, from the PHB department :-), is there a single headline number
> that could usefully be boiled down from these results?

I don't know which kind of information would be most useful here:  One
might like to see the total execution time of the whole benchmarking suite
including the framework code, or one might want to see the summarized
execution time of the benchmarks only.  Both would be easy to add.  So,
what should be used for a summary - maybe even some different solution?

Best regards,
Dirk


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel


^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2002-07-21 12:03 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2002-07-20  2:13 benchmarking framework Dirk Herrmann
2002-07-20  8:44 ` Neil Jerram
2002-07-21 12:03   ` Dirk Herrmann
2002-07-20 12:13 ` Marius Vollmer
2002-07-20 12:46 ` Han-Wen
2002-07-20 14:01   ` Neil Jerram
2002-07-20 19:11 ` Rob Browning

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).