unofficial mirror of guile-devel@gnu.org 
 help / color / mirror / Atom feed
From: Dirk Herrmann <dirk@sallust.ida.ing.tu-bs.de>
Subject: benchmarking framework
Date: Sat, 20 Jul 2002 04:13:30 +0200 (CEST)	[thread overview]
Message-ID: <Pine.GSO.4.05.10207200332430.5632-100000@sallust.ida.ing.tu-bs.de> (raw)

Hi folks,

I have taken the freedom to add a benchmarking framework to guile.  It is
a simple adaptation from the testing framework.  A lot of code has been
copied from there.  Well, maybe at some time it would be better to extract
the common parts into a module.

Whatever, here's a short introduction:  benchmarks are placed in the
directory benchmark-suite/benchmarks.  They are ordinary scheme files, but
with the .bm extension.  At least these files are detected automatically.
Others can also be specified explicitly.

Within the benchmark files, you can introduce benchmarks with the
(benchmark ...) macro.  The syntax is as follows:
  (benchmark NAME ITERATION body...)
where NAME is a name for the benchmark, similar to the names in the
test-suite (btw, you also have with-benchmark-prefix...), ITERATION is the
number of times the body shall be executed.  And the body finally is the
code to be benchmarked.  Example:
  (define bignum (1- (expt 2 128)))
  (let* ((i 0))
    (benchmark "bignum" 130000
      (logand i bignum)
      (set! i (+ i 1))))
This will run the statements (logand i bignum) and (set! i (+ i 1))
130000 times (this, however, can be scaled - see below).  The time for
this is measured and written out.  The execution counts in the small
examples are chosen such that on my machine this results in an execution
time of about one second for each benchmark.

You start the benchmarking with the command ./benchmark-guile.  You have
the same options as with ./check-guile, except for --flag-unresolved,
which is test specific, and --test-suite is renamed to --benchmark-suite.  
There is an additional option --iteration-factor NUM, which allows to
scale the execution time for benchmarks:  As can be seen from the exmample
above, every benchmark is given an iteration count, which indicates how
often the benchmark is to be executed.  With the option --iteration-factor
NUM you can increase or decrease the execution count of the benchmarks and
thus influence the time needed for performing the benchmarks.  For
example, running the benchmark suite with --iteration-factor 0.5 will
about halven the execution time, since all benchmark's bodies are executed
about half as often.

Results are written to a log file (this file contains a lot of data) and
to the console (still a lot of data, but not quite as much).  The values
have the following meaning:
  total:     total execution time (this is what the unix time command
             reports as real time).
  user:      user time (this is reported as user time also by the unix
             time command)
  system:    system time, similar to the unix time command
  frame:     this is the part of the user time that is consumed by the
             benchmarking framework itself.  You can think of this as
             the time that would still be consumed, even if the
             benchmarking code itself was empty.  This value does not
             include the time for garbage collection.
  benchmark: This is the part of the user time that is actually spent
             within the benchmarking code.  That is, the time needed
             for the benchmarking framework is subtracted.  This value,
             however, includes all garbage collection times.
  user/interp: This is the part of the user time that is spent in the
             interpreter (and not in garbage collection)
  bench/interp: This is the part of the benchmark time that is spent in
             the interpreter (and not in garbage collection).  This
             value is most probably the one you are interested in, except
             if you are doing some garbage collection checks.
  gc:        The time spent in garbage collection.
However, there are some caveats when using the values:  The frame time is
estimated based on running an empty benchmark during startup and measuring
that time.  This can be somewhat inaccurate.  This value, however, is
then used to compute the benchmark and bench/interp times.  I don't know
about the accuracy of the other values as reported by (times) and
(gc-run-time).

Best regards, 
Dirk Herrmann


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel


             reply	other threads:[~2002-07-20  2:13 UTC|newest]

Thread overview: 7+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2002-07-20  2:13 Dirk Herrmann [this message]
2002-07-20  8:44 ` benchmarking framework Neil Jerram
2002-07-21 12:03   ` Dirk Herrmann
2002-07-20 12:13 ` Marius Vollmer
2002-07-20 12:46 ` Han-Wen
2002-07-20 14:01   ` Neil Jerram
2002-07-20 19:11 ` Rob Browning

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://www.gnu.org/software/guile/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=Pine.GSO.4.05.10207200332430.5632-100000@sallust.ida.ing.tu-bs.de \
    --to=dirk@sallust.ida.ing.tu-bs.de \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).