Are the tf_cnn_benchmarks deprecated and replaced by MLPerf?

[vilmara]

Hi @tfboyd, are the tf_cnn_benchmarks deprecated and replaced by MLPerf?. You have mentioned at #407 that "tf_cnn_benchmarks is deprecated going into TF 2.0. It is still a great tool pre-TF 2.0. The new "perf tools" are different and focus on end user performance". For "the new perf tools" do you mean MLPerf training benchmarks?.

I am testing the tf_cnn_benchmarks master branch with the TF version tf-nightly-gpu 2.1.0.dev20191112 and it is throwing the error ImportError No module named contrib.data.python.ops. I see also the repo hasn't been updated for a while (the last update was 7 months ago).

Could you please provide further comments and recommendations on what benchmarks we should use according to the test goal, and maybe updating the README file with that info?

[tfboyd]

I moved to a new team. I seem them as deprecated. Before I moved I focused is on the official models the TensorFlow/models/official because that pushed the high level apis to be faster.

…

On Thu, Nov 14, 2019, 8:38 AM Vilmara ***@***.***> wrote: Hi @tfboyd <https://github.com/tfboyd>, are the tf_cnn_benchmarks deprecated and replaced by MLPerf?. You have mentioned at #407 <#407> that "tf_cnn_benchmarks is deprecated going into TF 2.0. It is still a great tool pre-TF 2.0. The new "perf tools" are different and focus on end user performance". For "the new perf tools" do you mean MLPerf training benchmarks?. I am testing the tf_cnn_benchmarks master branch with the TF version tf-nightly-gpu 2.1.0.dev20191112 and it is throwing the error ImportError No module named contrib.data.python.ops. I see also the repo hasn't been updated for a while (the last update was 7 months ago). Could you please provide further comments and recommendations on what benchmarks we should use according to the test goal, and maybe updating the README file with that info? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#436?email_source=notifications&email_token=AFTF5MSCAFKGAXVQE63EQ43QTV5I3A5CNFSM4JNOMCGKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HZL6YQQ>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFTF5MXMGMYZIARR72ZXHFLQTV5I3ANCNFSM4JNOMCGA> .

[vilmara]

@tfboyd thanks for your prompt reply, do you know if there is somebody else still administrating this benchmark repo?

[tfboyd]

Reed would know.

…

On Thu, Nov 14, 2019, 8:50 AM Vilmara ***@***.***> wrote: @tfboyd <https://github.com/tfboyd> thanks for your prompt reply, do you know if there is somebody else still administrating this benchmark repo? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#436?email_source=notifications&email_token=AFTF5MRIFTFI2PPPPXTSAL3QTV6U7A5CNFSM4JNOMCGKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEECQFFA#issuecomment-553976468>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFTF5MW32TJ4A6O7BC7R5MDQTV6U7ANCNFSM4JNOMCGA> .

[vilmara]

thanks @tfboyd. Hi @reedwm, could you please assist?

[reedwm]

Yes, tf_cnn_benchmarks is no longer maintained. I just updated the repo to support TF2, but will not do any further updates. It is now recommended to use the TensorFlow Official
Models

[vilmara]

hi @reedwm. Does the tf_cnn_benchmark.py script work with TensorFlow Official Models?. Or do you recommend getting rid of this repo at all?

[reedwm]

tf_cnn_benchmarks.py does not work with the TensorFlow Official Models. Each official model has their own Python script you need to run. I recommend not using this repo anymore (except for PerfZero), and using the Official Models instead.

[vilmara]

@reedwm duly noted, thanks!