hsbencher
Benchmark frameworks are usually very specific to the
host language/environment. Hence they are usually about as reusable
as compiler passes (that is, not).
Nevertheless, hsbencher is an attempt at a reusable benchmark
framework. It knows fairly little about what the benchmarks do, and
is mostly concerned with defining and iterating through
configuration spaces (e.g. varying the number of threads), and
managing the data that results.
Benchmark data is stored in simple text files, and optionally
uploaded via pluggable backend packages such as hsbencher-fusion,
which uploads to Google Fusion Tables.
hsbencher attempts to stradle the divide between language-specific
and language-agnostic by having an extensible set of BuildMethods.
As shipped, hsbencher knows a little about cabal, ghc, and less
about Make, but it can be taught more.
The general philosophy is to have benchmarks follow a simple
protocol, for example printing out a line SELFTIMED: 3.3 if they
wish to report their own timing, in seconds. The focus is on benchmarks that
run long enough to run in their own process. This is typical of
parallelism benchmarks and different than the fine-grained
benchmarks that are well supported by Criterion.
hsbencher is used by creating a script or executable that imports HSBencher
and provides a list of benchmarks, each of which is decorated with its
parameter space. Below is a minimal example that creates a two-configuration
parameter space:
The output would appear as in this gist:
https://gist.github.com/rrnewton/5667800
More examples can be found here:
https://github.com/rrnewton/HSBencher/tree/master/hsbencher/example
ChangeLog:
(1.3.8) Added --skipto and --runid arguments
(1.3.4) Added ability to prune benchmarks with patterns on command line.
(1.4.2) Breaking changes, don't use Benchmark constructor directly. Use mkBenchmark.
(1.5) New columns in schema.
(1.8) Backend plugins, hsbencher-fusion package factored out.
(1.15) Add systemCleaner field to Config
(1.17) Add cpu affinity control; lspci off by default
(1.18) Fix COMPILE_FLAGS upload; upload unwords of COMPILE_FLAGS to avoid quotes in output.
(1.19.1) Change the semantics of naked runtime args to conjunction, not disjunction.
Also allow filtering of benchmarks by BenchSpace as well.
(1.20) Add --bindir command line argument.
