Export Data to time series DB - Influx or Prom

[timothysc]

Ideally it would be nice if I had an export feature to push * data to a time series database such as influx or prometheus such that I could create my own plots and cross functional analysis using grafana.

/cc @jeremyeder

[portante]

@timothysc, we are planning on that, we'll start with indexing sar data first, since we have the tools for that, along with the run metadata (PR #122).

[jeremyeder]

The issue is we needed it today. Yesterday. Last year actually. I was able to get browbeat up and running against openshift in half a day :/

[atheurer]

So browbeat deploys collectd, and collectd pushes its data to a time-series database?
For the benchmarks that are run via browbeat, do all of their results & stats get imported in to the same time-series database?

Could you point us to a grafana dashboard (or whatever else is used analyze the result of the workload) where you ran some workloads against openshift? I'd like to better understand how this works, like how the results are summarized in the grafana dashboard, how the workload stats are correlated with the tool data, etc. Thanks.

[jtaleric]

On Tue, Mar 8, 2016 at 3:53 PM, Andrew Theurer [email protected]
wrote:

So browbeat deploys collectd, and collectd pushes its data to a
time-series database?

Yes, Browbeat deploys collectd (installs and configures) and points it to a
TSDB (currently now Carbon).

For the benchmarks that are run via browbeat, do all of their results &
stats get imported in to the same time-series database?

For Benchmarks we use the snapshot mechanism in Grafana to pull the data
from a given period of time.

Could you point us to a grafana dashboard (or whatever else is used
analyze the result of the workload) where you ran some workloads against
openshift? I'd like to better understand how this works, like how the
results are summarized in the grafana dashboard, how the workload stats are
correlated with the tool data, etc. Thanks.

—
Reply to this email directly or view it on GitHub
#180 (comment)
.

Joe Talerico
Mobile : 919 760 3476

[jeremyeder]

It's not a complete pbench replacement. It would need a lot of work to become so.
http://perf-infra.ec2.breakage.org:3000/dashboard/db/openshift-dashboard

[atheurer]

We can put a higher priority on the tools which natively support exporting to a TSDB. Performance Co-Pilot should be available in the next release of pbench. We can possibly do collectd by next release as well. Are there other tools which you would rather see support this feature before either of these two?

@jtaleric when you use the snaphot feature, is that essentially pulling some stats from tools to get an idea of various throughputs (network, disk, etc) or pulling from the benchmarks?

[jtaleric]

On Tue, Mar 8, 2016 at 4:29 PM, Andrew Theurer [email protected]
wrote:

We can put a higher priority on the tools which natively support exporting
to a TSDB. Performance Co-Pilot should be available in the next release of
pbench. We can possibly do collectd by next release as well. Are there
other tools which you would rather see support this feature before either
of these two?

@jtaleric https://github.com/jtaleric when you use the snaphot feature,
is that essentially pulling some stats from tools to get an idea of various
throughputs (network, disk, etc) or pulling from the benchmarks?

Pulling from Grafana, we use a python lib to build graphs from the
benchmark(s) data.

Some of the benchmarks we have implemented provide their own graphing
tools... However some like PerfKit Benchmarker do not (unless you want to
use google).

Joe

—
Reply to this email directly or view it on GitHub
#180 (comment)
.

Joe Talerico
Mobile : 919 760 3476

[jeremyeder]

My personal challenge is that neither PCP nor collectd have any native support for docker or kubernetes. So regardless of the choice, we have to write plugins. I had a conversation with the OpenShift ops team and they've been in close contact with the PCP team. I need another conversation with them. Their initial feedback was that developing PCP plugins was way too difficult. Tomorrow I will investigate what they did instead.

[atheurer]

And this was my biggest fear for jumping on a specific tool for our measurement and analysis.

[portante]

Working to integrate browbeat like functionality into pbench is a pretty straight-forward process, and we have a clear path to it.

@arcolife is working on indexing sar data into ElasticSearch, from his sarjitsu project, which includes dashboards for the graphs. @atheurer, and @ndokos, are currently working on indexing more and more of the metadata from pbench itself, so that we have the time windows for when runs begin and end, for each iteration of each sample. With that data indexed, we'll then be able to algorithmically generate Grafana templates for the related data for all the hosts involved in pbench.

I think we have the infrastructure in pbench for identifying runs as objects with all the necessary metadata to reason about them and correlate them with log data, or metric data, etc. We just need help getting all the pipes hooked up.

[jeremyeder]

Commit to timeline please.

[jtaleric]

On Wed, Mar 9, 2016 at 7:09 AM, Jeremy Eder [email protected]
wrote:

Commit to timeline please.

+1

I have suggested something like this a long time ago (12/1/14, you can see
a email status where I reference the first incarnation of the Graphite
dash-boarding), and it has yet to be looked at or implemented. The lack of
a timeline forced us to look at alternatives... We didn't want to build graphing/collecting metrics into Browbeat. If you look at our commits we used to have
integration with pbench, it just didn't fit into our workflow so we had to
move on without it.

—
Reply to this email directly or view it on GitHub
#180 (comment)
.

Joe Talerico
Mobile : 919 760 3476

[ashishkamra]

discuss in the Tools meeting

[ashishkamra]

decided not to take up for enhancement after discussion in the Tools meeting on 11/14