Codecov Report

Attention: Patch coverage is 0% with 1 line in your changes missing coverage. Please review.

Project coverage is 70.64%. Comparing base (32ae062) to head (912bda0).

@@            Coverage Diff             @@
##           master      #45      +/-   ##
==========================================
- Coverage   70.72%   70.64%   -0.08%     
==========================================
  Files          57       57              
  Lines        3593     3591       -2     
==========================================
- Hits         2541     2537       -4     
- Misses       1052     1054       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Here is some feedback with respect to the notebook structure and the contents of the wandb report.
(the numbers in the list correspond to the same numbered points in this here: #11 (comment))

1. Looks good!
2. First panel: Show metrics for a normal training run (with an rtx8000 GPU, ~1-4 CPUs), steps per second, cpu utilization, GPU utilization , RAM, VRAM. Could also be nice to display the # of CPUs, the CPU type, the GPU type, # of gpus, etc.
3. Second panel: Within the same job, show a comparison of the metrics above with / without training. (algorithm=no_op vs algorithm=example)
   3.1: Assuming that the dataloading is the bottleneck, make a plot that compares the throughput with different num_workers for the same number of CPUs (all run within the same job, the current interactive job.)
   3.2: Make a plot that compares the throughput of runs with the no-op algorithm with a different number of CPUs with either a fixed num_workers, or a fixed num_workers per cpu ratio.
   3.3: Given this better configuration for num_workers and n_cpus, show similar panels as in step 2, showing a comparison between A) previous parameters (lower bound), B) New, optimized parameters, and C) Optimized parameters without training (upper bound).
4. Comparing GPU vs CPU training:
   • The current panel looks good. The content will have to be updated to use the optimized # of cpus / num_workers from step 3.
   • Would be nice to add another panel showing the same comparison for training a small fcnetnetwork on MNIST. There, I suspect that the difference between GPU / CPU throughput shouldn't be that large.
5. Comparing different types of GPUS
   • The panels look good! The content will have to be updated to use the optimized # of cpus / num_workers from step 3.
6. GPU Utilization
   • Also show the GPU utilization %, mem usage % in addition to the metrics in 2.