Prometheus and Grafana integration in NannyML

[bgalvao]

Hey! I would love to contribute to this feature, but I could not find any Pull Request associated with this topic. If there is, I would love to check it out.

If there isn't, I would love to hear your ideas on how to implement this. In broad lines, I am thinking of

• instrumenting with OpenTelemetry
• add a subcommand to the nml CLI to build a docker image with the instrumented code...

Does this sound good? Does this sound bad? I would love to discuss.

[nnansters]

Hey Bernardo!

Thanks for reaching out! We love the idea! So much that in fact, we've already implemented some of this 😄

As a first way of integrating with NannyML, we've implemented an export of our results to databases such as PostgreSQL, MySQL, SQL server, ... You can then have Grafana connect to that database and retrieve the metrics for there. Even better news: we have some fully containerized examples for you to browse at https://github.com/NannyML/examples.

I had thought about going the OpenTelemetry and Prometheus route as well, but it seems like a much more specialized use case so I prioritized the good old database instead. This approach does have some additional operational complexity as it requires running an additional database. Just having NannyML expose metrics might be a lot simpler to deploy in a context like Kubernetes.

Are there any reasons for you to look at Prometheus and OpenTelemetry specifically? Or does the database export take care of your current needs?

As for the implementation suggestion: I think a subcommand for nml is the preferred path indeed. There is a complication however.
Since performing the actual number crunching might take a while, it would be better if NannyML did not trigger its calculations upon a scrape by Prometheus. I think it would be desirable to have NannyML run at its own pace (scheduling is possible, check the incremental regression example) and store the results somewhere, e.g. in memory SQLite or such. Upon scraping we can then simply return the latest precalculated results.

This is not a huge deal, but it probably requires communication between threads etc., so might be more complex then at first sight.

Love to get your thoughts on this!

[bgalvao]

Hi Niels, it's been a while and I am back on this topic.

About the database writer

Yes indeed, the database writer has been very helpful! I've hooked up to a MSSQL database and connected SuperSet to it to query the results. My first feedback would be to add a chunking parameter field to the runs table (I've had to add a hack to the model name as <model_name>/<chunking_param>), for all three of them: chunk_number, chunk_period and chunk_size.

Otherwise, this information is never saved and is lost. And it could be convenient to see and query according to these fields!

---

About the prometheus exporter approach

After thinking about this and having experienced the database writer approach written just above, the Prom approach could be... really pointless. I think I brought it up in order to have a "kubernetic" approach.

In light of ML monitoring problems - as in to day, non-standard software monitoring - for model performance monitoring, it is necessary to keep the complete history of data that the model is supposed to make predictions for. This is by principle not appropriate for a prom scrape. It is better to query the database straight from Grafana.

So right now, perhaps as you have already concluded, I don't see any motivation for to create a Prom approach for model performance monitoring. I am open to being wrong though 😄