This document may describe some future capabilities of pbench. We are working as fast as we can to make the code catch up to the documentation and vice-versa, but things may not be exactly as described here. If you find something not working as described here, please let us know: it may be a bug in the documentation, a bug in the code or a feature that we want to implement but we haven’t quite gotten to yet.
Pbench is a harness that allows data collection from a variety of tools while running a benchmark. Pbench has some built-in script that run some common benchmarks, but the data collection can be run separately as well with a benchmark that is not built-in to pbench, or a pbench script can be written for the benchmark. Such contributions are more than welcome!
Convenience links for some places of interest (N.B. The results directory has changed - see Accessing results on the web for the details):
[2015-01-23 Fri] In preparation for moving the incoming directory to bigger storage, older directories in the pbench incoming area were archived (tarred/compressed/moved to new area). The list of these directories can be found here.
Here is the minimum set of commands to get from 0 to a crawl:
wget -O /etc/yum.repos.d/pbench.repo http://repohost.example.com/repo/yum.repos.d/pbench.repo yum install pbench-agent -y . /etc/profile.d/pbench-agent.sh # or log out and log back in register-tool-set user-benchmark -C test1 -- ./your_cmd.sh move-results
Visit the Results directory to see the results: assuming you ran the above on a host named “myhost”, the results will be in its “myhost/user_benchmark_test1_<yyyy.mm.ddTHH.MM.SS>” subidrectory.
For explanations, and if crawling is not enough, see subsequent sections.
Visit
http://repohost.example.com/repo/yum.repos.d
and copy the pbench.repo file to /etc/yum.repos.d on the SUT[fn:8]. If the SUT
consists of multiple hosts, the file (and pbench) should be installed on all
of them. Here is a command that you can execute (as root) on the SUT to accomplish
this:
wget -O /etc/yum.repos.d/pbench.repo http://repohost.example.com/repo/yum.repos.d/pbench.repo
Then
yum install pbench-agent
will install the pbench agent in /opt/pbench-agent. Before you do
anything else, you need to source the file
/etc/profile.d/pbench-agent.sh. It sets PATH appropriately and
defines an env variable _PBENCH_AGENT_CONFIG to point to the default pbench
configuration file (see Customizing below). Alternatively, logging out
and logging back in will source the script automatically.
Since the package (as well as the benchmark and tools RPMS that are in the repo) gets updated fairly often, I have found it necessary to clean the yum cache in order for yum to see the new package, although you might want to try updating without cleaning the cache first and if yum reports no packages to update, then try again after cleaning the cache:
yum clean expire-cache yum update pbench-agent
If you try the above update and encounter problems (e.g. the pbench scripts, or the config file, are not found even after you log out and log back in), then try the following workaround:
yum reinstall pbench-agent
That should reestablish the missing symlink.
The workaround should not be necessary if your currently installed release is 0.31-95 or later.
In desperate situations, removing the pbench-agent package and reinstalling
after cleaning the cache should clear up any problems:
yum erase pbench-agent yum clean expire-cache yum install pbench-agent
If you upgrade to a release later than -102, please also ensure that you clear out your tools and reregister them after the upgrade: the label handling has changed. For example:
clear-tools register-tool-set --label=ceph-server --remote=my-ceph-server register-tool-set --label=kvmhost --remote=my-kvmhost register-tool-set --label=kvmguest --remote=my-kvmguest
All of the commands take a --help option and produce a terse
usage message.
The default set of tools for data collection can be enabled with
register-tool-set
You can then run a built-in benchmark by invoking its pbench script - pbench will install the benchmark if necessary[fn:1]:
pbench_fio
When the benchmark finishes, the tools will be stopped as well. The results can be collected and shipped to the standard storage location[fn:2] with
move-results
or
copy-results
If you want to run something that is not already packaged up as a benchmark script,
you may be able to use the user-benchmark script: it takes a command as argument,
starts the collection tools, invokes the command, stops the collection tools and
postprocesses the results. So the workflow becomes:
register-tool-set user-benchmark --config=foo -- myscript.sh move-results
Apart from having to register the collection tools on all the hosts, the rest
is the same: user-benchmark will start the collection tools on all the hosts,
run myscript.sh, stop the tools and run the postprocessing phase, gathering up
all the remote results to the local host (the local host may be just a controller,
not running any collection tools itself, or it may be part of the set of hosts where
the benchmark is run, with collection tools running).
The underlying assumption is that myscript.sh will run your
benchmark on all the relevant hosts and will copy all the results into
the standard directory which postprocessing will copy over to the
controller host. user-benchmark calls the script in its command-line
arguments (everything after the – is just execed by user-benchmark)
and redirects its stdout to a file in that directory:
$benchmark_run_dir/result.txt.
The benchmark scripts source the base script (/opt/pbench-agent/base)
which sets a bunch of defaults:
pbench_run=/var/lib/pbench pbench_log=/var/lib/pbench/pbench.log date=`date --utc "+%Y.%m.%dT%H.%M.%S"` hostname=`hostname -s` [email protected] results_repo_dir=/pbench/public_html/incoming ssh_opts='-o StrictHostKeyChecking=no'
These are now specified in the config file
/opt/pbench-agent/config/pbench.conf.
The configured default set of tools (what you would get by running
register-tool-set) is:
- sar, iostat, mpstat, pidstat, proc-vmstat, proc-interrupts, perf
In addition, there are tools that can be added to the default set
with register-tool:
- blktrace, cpuacct, dm-cache, docker, kvmstat, kvmtrace, lockstat, numastat, perf, porc-sched_debug, proc-vmstat, qemu-migrate, rabbit, strace, sysfs, systemtap, tcpdump, turbostat, virsh-migrate, vmstat
There is a default group of tools (that’s what register-tool-set uses), but
tools can be registered in other groups using the --group option of register-tool.
The group can then be started and stopped using start-tools and stop-tools
using their --group option.
Additional tools can be registered:
register-tool --name blktrace
or unregistered (e.g. some people prefer to run without perf):
unregister-tool --name perf
Note that perf is run in a “low overhead” mode with options “record -a –freq=100”, but if you want to run it differently, you can always unregister it and register it again with different options:
unregister --name=perf register-tool --name=perf -- --record-opts="record -a --freq=200"
Tools can be also be registered, started and stopped on remote hosts
(see the --remote option described in Running pbench collection tools with an arbitrary benchmark below for more on this)
Note that in many of these scripts the default tool group is hard-wired: if you want them to run a different tool group, you need to edit the script[fn:4].
This section is needed as preparation for the Second steps section below.
Pbench uses a bunch of utility scripts to do common operations. There
is a common set of options for some of these: --name to specify a
tool, --group to specify a tool group, --with-options to list or
pass options to a tool, --remote to operate on a remote host
(see entries in the FAQ section below for more
details on these options).
The first set is for registering and unregistering tools and getting some information about them:
list-tools- list the tools in the default group or in the specified group; with the –name option, list the groups that the named tool is in. TBD: how do you list all available tools whether in a group or not?
register-tool-set- call
register-toolon each tool in the default list. register-tool- add a tool to a tool group (possibly remotely).
unregister-tool- remove a tool from a tool group (possibly remotely).
clear-tools- remove a tool or all tools from a specified tool group (including remotely).
The second set is for controlling the running of tools –
start-tools and stop-tools, as well as postprocess-tools below,
take --group, --dir and --iteration options: which group of
tools to start/stop/postprocess, which directory to use to stash
results and a label to apply to this set of results. kill-tools is
used to make sure that all running tools are stopped: having a bunch
of tools from earlier runs still running has been know to happen and
is the cause of many problems (slowdowns in particular):
start-tools- start a group of tools, stashing the results in the
directory specified by
--dir. stop-tools- stop a group of tools.
kill-tools- make sure that no tools are running to pollute the environment.
The third set is for handling the results and doing cleanup:
postprocess-tools- run all the relevant postprocessing scripts on the tool output - this step also gathers up tool output from remote hosts to the local host in preparation for copying it to the results repository.
clear-results- start with a clean slate.
copy-results- copy results to the results repo.
move-results- move the results to the results repo and delete them from the local host.
edit-prefix- change the directory structure of the results (see the Accessing results on the web section below for details).
cleanup- clean up the pbench run directory - after this step, you will need to register any tools again.
register-tool-set, register-tool and unregister-tool can also
take a --remote option (see Benchmark scripts options).
pbench_fio iterates over a couple test types: read, randread
and a bunch of block sizes), with each iteration doing the following:
- create a benchmark_results directory
- start the collection tools
- run the benchmark
- stop the collection tools
- postprocess the collection tools data
The tools are started with an invocation of start-tools like this:
start-tools --group=$group --iteration=$iteration --dir=$benchmark_tools_dir
where the group is usually “default” but can be changed to taste as
described above, iteration is a benchmark-specific tag that
disambiguates the separate iterations in a run (e.g. for pbench_fio
it is a combination of a count, the test type, the block size and a
device name), and the benchmark_tools_dir specifies where the collection
results are going to end up (see the Results structure section for much
more detail on this).
The stop invocation is exactly parallel, as is the postprocessing invocation:
stop-tools --group=$group --iteration=$iteration --dir=$benchmark_tools_dir postprocess-tools --group=$group --iteration=$iteration --dir=$benchmark_tools_dir
Generally speaking, benchmark scripts do not take any pbench-specific
options except --config (see
register-tool --name=blktrace [--remote=foo] -- --devices=/dev/sda,/dev/sdb
There is no default and leaving it empty causes errors in postprocessing (this should be flagged).
Note that move-results, copy-results and clear-results always
assume that the run directory is the default /var/lib/pbench.
move-results and copy-results now (starting with pbench version 0.31-108gf016ed6)
take a --prefix option. This is explained in the Accessing results on the web section
below.
Note also that start/stop/postprocess-tools must be called with exactly the same
arguments. The built-in benchmark scripts do that already, but if you go your own way,
make sure to follow this dictum.
--dir- specify the run directory for all the collections tools. This argument
must be used by
start/stop/postrprocess-tools, so that all the results files are in known places:start-tools --dir=/var/lib/pbench/foo stop-tools --dir=/var/lib/pbench/foo postprocess-tools --dir=/var/lib/pbench/foo --remote- specify a remote host on which a collection tools (or set of collection tools)
is to be registered:
register-tool --name=<tool> --remote=<host>
If you want to take advantage of pbench’s data collection and other goodies, but your benchmark is not part of the set above (see <a href=”*Available benchmark scripts”>Available benchmark scripts), or you want to run it differently so that the pre-packaged script does not work for you, that’s no problem (but, if possible, heed the WARNING above). The various pbench phases can be run separately and you can fit your benchmark into the appropriate slot:
group=default benchmark_tools_dir=TBD register-tool-set --group=$group start-tools --group=$group --iteration=$iteration --dir=$benchmark_tools_dir <run your benchmark> stop-tools --group=$group --iteration=$iteration --dir=$benchmark_tools_dir postprocess-tools --group=$group --iteration=$iteration --dir=$benchmark_tools_dir copy-results
Often, multiple experiments (or “iterations”) are run as part of a single run. The modified flow then looks like this:
group=default
experiments="exp1 exp2 exp3"
benchmark_tools_dir=TBD
register-tool-set --group=$group
for exp in $experiments ;do
start-tools --group=$group --iteration=$exp
<run the experiment>
stop-tools --group=$group --iteration=$exp
postprocess-tools --group=$group --iteration=$exp
done
copy-results
Alternatively, you may be able to use the user-benchmark script as follows:
user-benchmark --config="specjbb2005-4-JVMs" -- my_benchmark.sh
which is going to run my_benchmark.sh in the <run your benchmark>
slot above. Iterations and such are your responsibility.
user-benchmark can also be used for a somewhat more specialized
scenario: sometimes you just want to run the collection tools for a
short time while your benchmark is running to get an idea of how the
system looks. The idea here is to use user-benchmark to run a sleep
of the appropriate duration in parallel with your benchmark:
user-benchmark --config="specjbb2005-4-JVMs" -- sleep 10
will start data collection, sleep for 10 seconds, then stop data collection and gather up the results. The config argument is a tag to distinguish this data collection from any other: you will probably want to make sure it’s unique.
This works well for one-off scenarios, but for repeated usage on well defined phase changes you might want to investigate Triggers.
Note that from latest version onwards, we would like to have a file at
http://pbench.example.com/pbench-archive-host where the FQDN
of the pbench web-server lies and the results would be pushed here. Currently it
is archivehost.example.com. This would mean, if in future, we would like
to change the central server settings, we wouldn’t want the users to upgrade to a latest
version of pbench. Rather, just change the FQDN in this hosted file and then new results
would automatically be pushed to the updated location.
Usually, a multihost benchmark is run using a host that acts as the “controller” of the run. There is a set of hosts on which data collection is to be performed while the benchmark is running. The controller may or may not be itself part of that set. In what follows, we assume that the controller has password-less ssh access to the relevant hosts.
The recommended way to run your workload is to use the generic user-benchmark script.
The workflow in that case is:
- Register the collection tools on each host in the set:
for host in $hosts ;do
register-tool-set --remote=$host
- Invoke
user-benchmarkwith your workload generator as argument: that will start the collection tools on all the hosts and then run your workload generator; when that finished, it will stop the collection tools on all the hosts and then run the postprocessing phase which will gather the data from all the remote hosts and run the postprocessing tools on everything. - Run
copy-resultsormove-resultsto upload the data to the results server.
If you cannot use the user-benchmark script, then the process becomes more manual.
The workflow is:
- Register the collection tools on each host as above.
- Invoke
start-toolson the controller: that will start data collection on all of the remote hosts. - Run the workload generator.
- Invoke
stop-toolson the controller: that will stop data collection on all of the remote hosts. - Invoke
postprocess-toolson the controller: that will gather all the data from the remotes and run the postprocessing tools on all the data. - Run
copy-resultsormove-resultsto upload the data to the results server.
Some characteristics[fn:3] of pbench are specified in config files and can be customized by adding your own config file to override the default settings.
TBD
The move-results script removes the results directory (assumed to be
within the /var/lib/pbench hierarchy) after copying it the results
repo. But if there are previous results present (perhaps because
move-results was never invoked, or perhaps because copy-results
was invoked instead), move-results will copy all of them: you
probably do not want that.
It’s a good idea in general to invoke clear-results, which cleans
/var/lib/pbench, before running your benchmark.
If you interrupt a built-in benchmark script (or your own script perhaps),
the collection tools are not going to be stopped. If you don’t stop them
explicitly, they can severely affect subsequent runs that you make. So it
is strongly recommended that you invoke kill-tools before you start your
run:
kill-tools --group=$group
This tool will delete the tools.$group file on the local host as well
as on all the remote hosts specified therein. After doing that, you
will need to re-register all the tools that you want to use. In
combination with clear-results, this tool creates a blank slate
where you can start from scratch. You probably don’t want to call
this much, but it may be useful in certain isolated cases.
Some tools have required options[fn:9] and you have to specify
them when you register the tool. One example is the blktrace tool
which requires a --devices=/dev/sda,dev/sdb= argument. register-tool-set
knows about such options for the default set of tools, but with other
tools, you are on your own.
The trouble is that registration does not invoke the tool and does not
know what options are required. So the best thing to do is invoke the
tool with --help: the --help option may or may not be recognized
by any particular tool, but either way you should get a usage message
that labels required options. You can then register the tool by using
an invocation similar to:
register-tool --name=blktrace -- --devices=/dev/sda,/dev/sdb
If you use the tool scripts explicitly, specify --dir=/var/lib/pbench/<run-id>
so that all the data are collected in the specified directory. Also, save any data
that your benchmark produces inside that directory: that way, move-results
can move everything to the results warehouse.
Make the <run-id> as detailed as possible to disambiguate results. The built-in
benchmark scripts use the following form: <benchmark>_<config>_<ts>, e.g
fio_bagl-16-4-ceph_2014.12.15T15.58.51
where the <config> part (bagl-16-4-ceph) comes from the --config option and
can be as detailed as you want to make it.
If you are running multihost benchmarks, we strongly encourage you to set up the
tool collections using --remote. Choose a driver host (which might or might not
participate in the tool data collection: in the first case, you register tools locally
as well as remotely; in the second, you just register them remotely) and run everything
from it. During the data collection phase, everything will be pulled off the remotes and
copied to the driver host, so it can be moved to the results repo as a single unit.
Consider also using --label to label sets of hosts - see Using =–label= for more information.
When you register remotes, --label can be used to give a meaningful
label to the results subdirectories that come from remote hosts. For
example, use =–label=server” (or client, or vm, or capsule or
whatever else is appropriate for your use case).
This section describes how to get to your results using a web browser. It describes
how move-results moves the results from your local controller to a centralized
location and what happens there. It also describes the --prefix option to move-results
(and copy-results) and a utility script, edit-prefix, that allows you to change how
the results are viewed.
N.B. This section applies to the pbench RPM version 0.31-108gf016ed6 and later. If you are using an earlier version, please upgrade at your earliest convenience.
The canonical place is
http://resultshost.example.com/results/
There are subdirectories there for each controller host (the host on
which move-results was executed) and underneath those, there are
subdirectories for each pbench run.
The leaves of the hierarchy are actually symlinks that point to the
corresponding results directory in the old, flat incoming/
hierarchy. Direct access to incoming/ is now deprecated (and will
eventually go away).
The advantage is that the results/ hierarchy can be manipulated to
change one’s view of the results[fn:10], while leaving the incoming/
hierarchy intact, so that tools manipulating it can assume a fixed
structure.
In the interim, a simple script is running once an hour creating any
missing links from results/ to incoming/. It will be turned off
eventually after everybody has upgraded to this or a later version
of pbench.
In order to make move-results more robust, it now packages up the
results in a tarball, computes an MD5 checksum, copies the tarball
to an archive area, checks that the MD5 checksum is still correct
and then deletes the results from one’s local host.
The tarball is unpacked into the incoming/ hierarchy by a cron script
which runs every minute (so there might be a short delay before the results
are available), and plants a symlink to the results directory in the results/
hierarchy.
Using the --prefix= option to move-results affects where that
symlink is planted (and that’s the only thing it affects). For
example, if your controller host is alphaville and the results name
is fio__2015.03.30T13.33.15, normally move-results would unpack
the tarball under incoming/alphaville/fio__2015.03.30T13.33.15 and
plant a symlink pointing to that at results/alphaville/fio__2015.03.30T13.33.15.
But if you wanted to group all your fio results under results/alphaville/fio, you
could instead say
move-results --prefix=fio
which would plant the link at results/alphaville/fio/fio__2015.03.30T13.33.15
instead of planting it at results/alphaville/fio__2015.03.30T13.33.15.
What if you forget to use --prefix when calling move-prefix? Or
you want to reorganize further, perhaps pushing a set of results
further down in the results/ hierarchy?
You can do that with edit-prefix. For example, continuing the example
above, suppose you want to push a bunch of results from fio/ down another
level, perhaps to group all the fio results on a particular platform together:
edit-prefix --prefix=fio/dl980 fio/fio__2015.03.30T13.33.15 ...
would do that. The arguments must exist in the appropriate place in
the results hierarchy and the symlink at the leaf must point to an
existing result in the incoming/ hierarchy. The links are then moved,
using the new prefix, to a different place in the results/ hierarchy.
edit-prefix works similarly to move-results: it sends instruction to
the centralized results repository which are executed by a cron script
running once a minute; so it may take a bit before the change takes effect.
Andrew writes:
- All of the benchmark scripts use var/lib/pbench/$benchmark-$config-$date/$iteration/reference-result/tools-$tool_group
- This allows for 1-N iterations and 1-N samples per iteration. For example, user-benchmark uses var/lib/pbench/user-benchmark-$config-$date/1/reference-result
- A self-explanatory example of the above mentioned hierarchical pattern is as follows:
fio__2015.01.15T19.45.10/ --> $benchmark-$config-$date ├── 1-read-4KiB --> $iteration │ └── reference-result --> reference-result/ │ │ │ └── tools-default --> tools-$tool_group/ │ ├── cmds │ ├── iostat │ ├── mpstat │ ├── perf │ ├── pidstat │ ├── proc-interrupts │ ├── proc-vmstat │ ├── sar │ └── turbostat
reference-results- This is calculated (based on standard deviation) as the best result from all the iterations, after the tests have ended. This is just a sym-link to one of the iterations, so as to make it easier for the user take a quick look at the results.
Postprocessing now produces CSV files of the results. Each row consists of a timestamp and a series of measures. The first row is a header row with the labels.
The CSV files are directly used by the Javascript library that allows users to view graphs. The library runs in the client browser and pulls the CSV file from the server. If that file is large, there might be a substantial delay in the rendering of the graphs. In certain cases, large files have caused browsers to explode. The only known method to avoid that currently is to reduce the sampling frequency and therefore make the files smaller. This is unsatisfactory and we are working to mitigate this problem
Andrew writes:
To understand how data is arranged, you have to understand the different requirements users & benchmarks might have:
The simplest use case is when a user just wants to get tool data for a single measurement. For example, a user may run:
register-tool-set dir=/var/lib/pbench/mytest start-tools --dir=$dir my-benchmark-script.sh stop-tools --dir=$dir postprocess-tools --dir=$dir move-results(the “my-benchmark-script.sh” above could be substituted by simply waiting until whatever thing is happening is done, or a “sleep <x>”, etc)
The hierarchy is then pretty simple:
/var/lib/pbench/my-testis the base directory for this test, and the tool data is intools-$tool_group. Since they used the default tool group (they did not specify an alternative), it’s “tools-default”. The base directory is where a user should put any data regarding the workload (benchmark result). So, in general, when processing a test result, the benchmark data is in ./mytest, and the tool data for this benchmark is in ./mytest/tools-$tool_group/. These two are always tightly coupled to ensure the tool data is always included in the benchmark result.In the case above, the user has total control over the –dir name. The “tools-default” is a fixed name, which originates from “tools-$tool_group”. This should not change. “./<dir>/tools-*” should always be recognizable by other postprocessing scripts as the tools data for test <dir>. If a user wants to identify this result uniquely, the upper directory should be used, for example:
a first test:
dir=/var/lib/pbench/mytest-using-containers start-tools --dir=$dir my-benchmark-script.sh --use-docker stop-tools --dir=$dir postprocess-tools --dir=$dir move-resultsand then a second test:
dir=/var/lib/pbench/mytest-using-VMs start-tools --dir=$dir my-benchmark-script.sh --use-vms stop-tools --dir=$dir postprocess-tools --dir=$dir move-resultsWhen a user uses a built-in pbench benchmark, the directory hierarchy is maintained [and optionally expanded], but some of the directory names (or rather a portion of the name) is under the control of the pbench benchmark script. This is to maintain consistency across the pbench benchmark scripts. The pbench benchmark scripts should include a date in the base directory name and include contents from the –config option.
Since many benchmarks actually have several measurements, an extra level of directory is added to accommodate this. Instead of /var/lib/pbench/<mytest>/tools-default, we usually end up with /var/lib/pbench/<mytest>/<test-iteration[s]>/tools-default.
There are actually multiple reasons for the ./<test-iteration[s]>/ addition, as there are many reasons to have more than one test execution for any given benchmark. These include (but are not limited to):
- a benchmark simply has multiple different tests.
- a pbench benchmark script often tries to execute several benchmark configurations, varying things like load levels & different benchmark options, so the user does not have script these themselves.
- benchmarks may need multiple samples of the exact same benchmark configuration to compute standard deviation.
An example of (1) is SPECcpu, where there are several completely different tests, and they each should get their own result sub-directory (./<test-iteration-X/), with its own tools-$tool_group subdirectory. The “main” directory (/var/lib/pbench/<mytest>) includes the overall result, and generally where any report generated would reside.
An example of (2) is uperf, where by default this script runs several configurations, varying message size, number of instances, and protocol type. This can produce dozens of different results, all of which need to be organized properly. Each unique configuration uses a unique ./<iteration>/ directory under the main directory, each with their own tools-$tool_group subdir.
An example of (3) is dbench, where by default 5 samples of the same test are taken, Each of these results are kept in a ./<iteration>/ subdir. After the end of the tests, the dbench script computes the standard deviation and even creates a symlink, “reference-result”, to the 1 iteration-dir that it’s result closest to the sdtdev.
More than one of these uses for iterations can also be used. In fact, uperf, uses iterations for both varying benchmark options (like message sizes), but for each of those unique configurations, multiple samples are taken to compute a standard deviation. This then involves two levels of subdirs for the iterations. So, in this case, we have a hierarchy like:
/var/lib/pbench/<my-test> /var/lib/pbench/<my-test>/1-tcp-stream-1024k-1instance/ /var/lib/pbench/<my-test>/1-tcp-stream-1024k-1instance/sample1/So, in summary:
- tool data is always in a subdir of where the benchmark result is kept. The tool subdir starts with “tools-”
- A benchmark result dir can be as high up as var/lib/pbench/<mytest>, or it can be 1 or two levels deeper, depending on the need for multiple test runs. Some kind of benchmark summary should always be in /var/lib/pbench/<mytest>.
I will cover remote tools in another comment section.
When pbench tools are registered remotely, the structure described above is followed on each host
Post-processing collects all the remote results locally. The results
from each remote host are pushed down one level in the hierarchy, with
the name of the host (augmented by the value of the --label option if
applicable) providing the extra directory level at the top.
In addition, if local results are present, they are also pushed down
one level in the hierarchy with the name of the local host providing
the extra directory level at the top (this happens in the purely local
case as well, for uniformity’s sake). Again, the value of the
--label option is used to augment the name if applicable.
Triggers are groups of tools that are started and stopped on specific events.
They are registered with register-tool-trigger using the --start-trigger
and --stop-trigger options. The output of the benchmark is piped into the
tool-trigger tool which detects the conditions for starting and stopping
the specified group of tools.
There are some commands specifically for triggers:
register-tool-trigger- register start and stop triggers for a tool group.
list-triggers- list triggers and their start/stop criteria.
tool-trigger- this is a Perl script that looks for the start-trigger and end-trigger markers in the benchmark’s output, starting and stopping the appropriate group of tools when it finds the corresponding marker.
As an example, pbench_dbench uses three groups of tools: warmup, measurement
and cleanup. It registers these groups as triggers using
register-tool-trigger --group=warmup --start-trigger="warmup" --stop-trigger="execute" register-tool-trigger --group=measurement --start-trigger="execute" --stop-trigger="cleanup" register-tool-trigger --group=cleanup --start-trigger="cleanup" --stop-trigger="Operation"
It then pipes the output of the benchmark into tool-trigger:
$benchmark_bin --machine-readable --directory=$dir --timelimit=$runtime
--warmup=$warmup --loadfile $loadfile $client |
tee $benchmark_results_dir/result.txt |
tool-trigger "$iteration" "$benchmark_results_dir" no
tool-trigger will then start the warmup group when it encounters the
string “warmup” in the benchmark’s output and stop it when it
encounters “execute”. It will also start the measurement group when it
encounters “execute” and stop it when it encounters “cleanup” - and so
on.
Obviously, the start/stop conditions will have to be chosen with some care to ensure correct actions.
This option is recognized by register-tool and unregister-tool: it
specifies the name of the tool that is to be (un)registered. list-tools
with the --name option list all the groups that contain the named tool[fn:7].
This option is recognized by the benchmark scripts (see Available benchmark scripts above) which use it as a tag for the directory where the benchmark is going to run. The default value is empty. The run directory for the benchmark is constructed this way:
${pbench_run}/${benchmark}_${config}_${date}
where $pbench_run and $date are set by the /opt/pbench-agent/base script
and $benchmark is set to the obvious value by the benchmark script; e.g. a
fio run with config=foo would run in the directory
/var/lib/pbench/fio_foo_2014.11.10T15.47.04.
This option is recognized by start-tools, stop-tools,
tool-trigger and postprocess-tools. It specifies the directory
where the tools are going to stash their data. The default value is /tmp.
Each group then uses it as a prefix for its own stash, which has the form
$dir/tools-$group. Part of the stash is the set of cmds to start and stop
the tools - they are stored in $dir/tools-$group/cmds. The output of the
tool is in $dir/tools-$group/$tool.txt.
This option has to be specified identically for each command when
invoking these commands (actually, each of the commands should be invoked
with the identical set of all options, not just --dir.)
If you use these tools explicitly (i.e. you don’t use one of the
benchmark scripts), it is highly recommended that you specify this
option explicitly and not rely on the /tmp default. For one, you
should make sure that different iterations of your benchmark use a
different value for this option, otherwise later results will
overwrite earlier ones.
N.B. If you want to run move-results or copy-results after the
end of the run, your resuls should be under /var/lib/pbench:
move/copy-results does not know anything about your choice for this
option; it only looks in /var/lib/pbench for results to upload. So
if you are planning to use move/copy-results, make sure that the
specified directory is a subdirectory of /var/lib/pbench.
pbench can register tools on remote hosts, start them and stop them remotely and gather up the results from the remote hosts for post-processing. The model is that one has a controller or orchestrator and a bunch of remote hosts that participate in the benchmark run.
The pbench setup is as follows: register-tool-set or register-tool
is called on the controller with the --remote option, once for each
remote host:
for remote in $remotes ;do
register-tool-set --remote=$remote --label=foo --group=$group
done
That has two effects: it adds a stanza for the tool to
the appropriate tools.$group file on the remote host and it also adds
a stanza like this to the controller tools.$group file:
remote@<host>:<label>
The label is optionally specified with --label and is empty by default.
When start-tools is called on the controller, it starts the local
collection (if any), but it also interprets the above stanzas and
starts the appropriate tools on the remote hosts. Similarly for
stop-tools and postprocess-tools.
TBD
Tool scripts are mostly boilerplate: they need to take a standard set of commands (–install, –start, –stop, –postprocess) and a standard set of options (–iteration, –group, –dir, –interval, –options). Consequently, the easiest thing to do is to take an existing script and modify it slightly to call the tool of your choice. I describe here the case of turbostat.
There are some tools that timestamp each output stanza; there are others that do not. In the former case, make sure to use whatever option the tool requires to include such timestamps (e.g. vmstat -t on RHEL6 or RHEL7 - but strangely not on Fedora 20 - will produce such timestamps).
There are some tools that are included in the default installation - others need to be installed separately. Turbostat is not always installed by default, so the tool script installs the package (which is named differently on RHEL6 and RHEL7) if necessary. In some cases (e.g. the sysstat tools), we provide an RPM in the pbench repo and the tool script makes sure to install that if necessary.
The only other knowledge required is where the tool executable resides (usually /usr/bin/<tool> or /usr/local/bin/<tool> - /usr/bin/turbostat in this case) and the default options to pass to the tool (which can be modified by passing –options to the tool script).
So here are the non-boilerplate portions of the turbostat tool
script. The first interesting part is to set tool_bin to point to
the binary:
# Defaults tool=$script_name tool_bin=/usr/bin/$tool
This only works if the script is named the same as the tool, which
is encouraged. If the installed location of your tool is not /usr/bin,
then adjust accordingly.
Since turbostat does not provide a timestamp option, we define a datalog script to add timestamps (no need for that for vmstat e.g.) and use that as the tool command:
case "$script_name" in
turbostat)
tool_cmd="$script_path/datalog/$tool-datalog $interval $tool_output_file"
;;
esac
The datalog script uses the log-timestamp pbench utility to timestamp the
output. It will then be up to the postprocessing script to tease out the data
appropriately.
The last interesting part dispatches on the command - the install is turbostat-specific,
but the rest is boilerplate: --start just executes the tool_cmd as defined above
and stashes away the pid, so that --stop can kill the command later; --postprocess
calls the separate post-processing script (see below):
release=$(awk '{x=$7; split(x, a, "."); print a[1];}' /etc/redhat-release)
case $release in
6)
pkg=cpupowerutils
;;
7)
pkg=kernel-tools
;;
*)
# better be installed already
;;
esac
case "$mode" in
install)
if [ ! -e $tool_bin ]; then
yum install $pkg
echo $script_name is installed
else
echo $script_name is installed
fi
start)
mkdir -p $tool_output_dir
echo "$tool_cmd" >$tool_cmd_file
debug_log "$script_name: running $tool_cmd"
$tool_cmd >>"$tool_output_file" & echo $! >$tool_pid_file
wait
;;
stop)
pid=`cat "$tool_pid_file"`
debug_log "stopping $script_name"
kill $pid && /bin/rm "$tool_pid_file"
;;
postprocess)
debug_log "postprocessing $script_name"
$script_path/postprocess/$script_name-postprocess $tool_output_dir
;;
esac
Finally, there is the post-processing tool: the simplest thing to do is nothing. That’s currently the case for the turbostat post-processing tool, but ideally it should produce a JSON file with the data points and an HTML file that uses the nv3 library to plot the data graphically in a browser. See the postprocess directory for examples, e.g. the iostat postprocessing tool.
TBD
Running
user_benchmark -- sleep 60
will start whatever data collections are specified in the default tool group, then sleep for 60 seconds. At the end of that period, it will stop the running collections tools and postprocess the collected data. Running move-results afterwards will move the results to the results server as usual.
pbench is a set of building blocks, so it allows you to use it in many different ways, but it also makes certain assumptions which if not satisfied, lead to problems.
Let’s assume that you want to run a number of iozone experiments, each with different
parameters. Your script probably contains a loop, running one experiment each time around.
If you can change your script so that it executes one experiment specified by an argument,
then the best way is to use the user-benchmark script:
register-tool-set
for exp in experiment1 experiment2 experiment3 ;do
user-benchmark --config $exp -- my-script.sh $exp
done
move-results
The results are going to end up in directories named /var/lib/pbench/user-benchmark_$exp_$ts
for each experiment (unfortunately, the timestamp will be recalculated at the beginning of
each user-benchmark invocation), before being uploaded to the results server.
Alternatively, you can modify your script so that each experiment is wrapped with start/stop/postprocess-tools and then call move-results at the end:
register-tool-set
for exp in experiment1 experiment2 experiment3 ;do
start-tools
<run the experiment>
stop-tools
postprocess-tools
done
move-results
[fn:8] “System under test”.
[fn:1] The current version of pbench-agent yum installs prebuilt RPMs of various common benchmarks: dbench, fio, iozone, linpack, smallfile and uperf, as well as the most recent version of the sysstat tools. We are planning to add more benchmarks to the list: iperf, netperf, streams, maybe the phoronix benchmarks. If you want some other benchmark (AIM7?), let us know.
[fn:2] The standard storage location currently is http://resultshost.example.com/incoming but it is subject to change without notice.
[fn:3] Only a few such characteristics exist today, but the plan is to move more hardwired things into the config files from the scripts. If you need to override some setting and have to modify scripts in order to do so, let us know: that’s a good candidate for the config file.
[fn:4] That will be handled by a configuration file in the future.
[fn:5] It is probably better to bundle these options in a configuration file, but that’s still WIP.
[fn:6] There is work-in-progress to provide a higher-level interface for this.
[fn:7] A list of available tools in a specific group can be obtained with the
--group option of list-tools; unfortunately, there is no option to list
all available tools - the current workaround is to check the contents of
/opt/pbench-agent/tool-scripts.
[fn:9] Yes, I know: it’s an oxymoron.
[fn:10] E.g. A performance engineer was NFS-mounting the incoming/
hierarchy, grouping his results under separate subdirectories for fio, iozone
and smallfile, and grouping them further under thematically created
subdirectories (“baremetal results for this configuration”, “virtual host
results under that configuration” etc.), primarily because having them all in
a single directory was slow, as well as confusing. There were two problems
with this approach which motivated the prefix approach described above. One
was that the NFS export of the FUSE mount of the gluster volume that houses
the result is extremetly flakey. The other is that the incoming/ hierarchy
is modified, which makes the writing of tools to extract data harder: they
have to figure out arbitrary changes, instead of being able to assume a fixed
structure.