run_benchmarks is referenced in the documentation and the codebase but is not available

[rw1nkler]

The documentation of the benchmark_main points to run_benchmarks, also run_benchmarks.py is mentioned in the codebase, but the tool seems to be unavailable in the repository. Is this part of the documentation up to date? Is the tool still needed/used somewhere?

If so, would it be possible to update the documentation about it?

CC @proppy

[lpawelcz]

Hi @proppy do you happen to know anything about this run_benchmarks.py?
I'm asking with regards to your comment and effort concentrated on reducing bazel usage in flows for dashboard generation (#1137) and Vizier parameter evaluation (#1160).

[proppy]

maybe @cdleary can shed more light on the history behind this script?

[cdleary]

Hasn't been open sourced yet because of two main complications:

• it's tied to the internal revision control system (it sees how your change compares to baseline which is the canonical HEAD so has to have a facility for run-at-head)
• it pulls in internal targets to benchmark which makes the bidirectional-import/export-to-github rewrite slightly more complicated

@allight and @meheff I think were chatting about how big of a lift it would be to make it work with the git based flow so we could properly OSS it -- presumably we'd need to virtualize the VCS parts, any other major parts I'm not thinking of?