user-benchmark should support arbitrary pre-test and post-test operations

[jeremyeder]

To avoid everyone writing wrapper scripts that wrap user-benchmark.

For example, we will want to sleep for N seconds before and after the test to gather "steady state" metrics with pbench.

    pre-test:  echo 3 > /proc/sys/vm/drop_caches ; sleep 300
    post-test:  sleep 60 ; cat /proc/diskstats

I'm not sure what the user-benchmark UI would look like though :/

[atheurer]

Can't this simply be done with the script supplied to user-benchmark? Like:
pbench-user-benchmark --config=blah -- ./my-script

and my-script has the pre-test, the benchmark command, and the post test in it?

[jeremyeder]

That's specifically what we want to avoid.

[jeremyeder]

Basically everyone has to do that. To me that means it should be part of the tooling.

[portante]

@jeremyeder what steps are common enough that the pattern can be captured in pbench? Can you give some examples on how you think this should work?

[jeremyeder]

Well at a basic level...

sleep $SLEEP
user-benchmark -C test1 -- ./script.sh
sleep $SLEEP

but really the sleep lines could be any string of commands.

[jeremyeder]

Another use-case would be to start and stop collectd service pre/post test.

[atheurer]

If pbench-user-benchmark could do the following, would that be ok:

user-benchmark -C test1 --pre=/path-to-pre.sh --post=path-to-post.sh -- ./script.sh

[jeremyeder]

/cc @ofthecurerh

[portante]

@atheurer, @jeremyeder, I'd like to have collectd be in the start/stop tools framework directly, so that we can record the notion that collectd tooling was used in with pbench metadata. When indexed, we can key off of collectd data to generate graphs in grafana.

A start/stop collectd tool would look up the collectd configuration and record what that hosts collectd configuration was at the time of the run so that we can later reference where to find the data. Alternatively, we could have start/stop collectd tool be given the grafana host which will have the data.

[jeremyeder]

@ofthecurerh is going to look deeper into this. I like the idea of collectd being "just another tool to start", and the collectd metadata being the key to graph generation...nice!

[atheurer]

Agreed collectd should be just like any other tool.

[ashishkamra]

discuss further in the Tools meeting

[ashishkamra]

@jeremyeder do we still need this enhancement in pbench? if yes, what's the priority in your opinion?