Merge branch 'master' into master

cm-mlops/script/app-image-classification-tvm-onnx-py/README.md

@@ -112,19 +112,9 @@ ___
              - CM script: [get-cuda](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-cuda)
     * `_llvm`
       - Workflow:
-    * **`_tvm`** (default)
-      - Workflow:
-        1. ***Read "deps" on other CM scripts***
-           * get,tvm
-             * CM names: `--adr.['tvm']...`
-             - CM script: [get-tvm](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-tvm)
 
     </details>
 
-
-#### Default variations
-
-`_tvm`
 #### Default environment
 
 <details>
@@ -159,6 +149,9 @@ ___
        - CM script: [get-ml-model-resnet50](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-ml-model-resnet50)
      * get,generic-python-lib,_onnxruntime
        - CM script: [get-generic-python-lib](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-generic-python-lib)
+     * get,tvm
+       * CM names: `--adr.['tvm']...`
+       - CM script: [get-tvm](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-tvm)
   1. Run "preprocess" function from customize.py
   1. Read "prehook_deps" on other CM scripts from [meta](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/app-image-classification-tvm-onnx-py/_cm.json)
   1. ***Run native script if exists***

cm-mlops/script/app-mlperf-inference-reference/README.md

@@ -466,6 +466,26 @@ ___
              - CM script: [get-generic-python-lib](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-generic-python-lib)
            * get,generic-python-lib,_toml
              - CM script: [get-generic-python-lib](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-generic-python-lib)
+    * `_sdxl`
+      - Environment variables:
+        - *CM_MODEL*: `sdxl`
+      - Workflow:
+        1. ***Read "deps" on other CM scripts***
+           * get,generic-python-lib,_package.diffusers
+             * CM names: `--adr.['diffusers']...`
+             - CM script: [get-generic-python-lib](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-generic-python-lib)
+           * get,generic-python-lib,_package.transformers
+             * CM names: `--adr.['transformers']...`
+             - CM script: [get-generic-python-lib](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-generic-python-lib)
+           * get,generic-python-lib,_package.accelerate
+             * CM names: `--adr.['accelerate']...`
+             - CM script: [get-generic-python-lib](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-generic-python-lib)
+           * get,generic-python-lib,_package.torchmetrics
+             * CM names: `--adr.['torchmetrics']...`
+             - CM script: [get-generic-python-lib](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-generic-python-lib)
+           * get,generic-python-lib,_package.torch-fidelity
+             * CM names: `--adr.['torch-fidelity']...`
+             - CM script: [get-generic-python-lib](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-generic-python-lib)
 
     </details>
 
@@ -535,6 +555,7 @@ ___
 * `--target_latency=value`  &rarr;  `CM_MLPERF_LOADGEN_TARGET_LATENCY=value`
 * `--target_qps=value`  &rarr;  `CM_MLPERF_LOADGEN_TARGET_QPS=value`
 * `--test_query_count=value`  &rarr;  `CM_TEST_QUERY_COUNT=value`
+* `--threads=value`  &rarr;  `CM_NUM_THREADS=value`
 
 **Above CLI flags can be used in the Python CM API as follows:**
 
@@ -642,6 +663,7 @@ ___
        * `if (CM_MODEL in ['gptj-99', 'gptj-99.9'])`
        * CM names: `--adr.['ml-model', 'gptj-model', 'gpt-j-model']...`
        - CM script: [get-ml-model-gptj](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-ml-model-gptj)
+       - CM script: [get-ml-model-stable-diffusion](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-ml-model-stable-diffusion)
      * get,ml-model,object-detection,resnext50,fp32,_pytorch-weights
        * `if (CM_MLPERF_BACKEND  == pytorch AND CM_MLPERF_IMPLEMENTATION  == nvidia AND CM_MODEL  == retinanet)`
        * CM names: `--adr.['ml-model', 'retinanet-model']...`
@@ -650,6 +672,10 @@ ___
        * `if (CM_MODEL in ['bert-99', 'bert-99.9']) AND (CM_MLPERF_CUSTOM_MODEL_PATH  != on)`
        * CM names: `--adr.['ml-model', 'bert-model']...`
        - CM script: [get-ml-model-bert-large-squad](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-ml-model-bert-large-squad)
+     * get,ml-model,stable-diffusion,text-to-image,sdxl
+       * `if (CM_MODEL  == sdxl) AND (CM_MLPERF_CUSTOM_MODEL_PATH  != on)`
+       * CM names: `--adr.['ml-model', 'sdxl-model']...`
+       - CM script: [get-ml-model-stable-diffusion](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-ml-model-stable-diffusion)
      * get,ml-model,medical-imaging,3d-unet
        * `if (CM_MODEL in ['3d-unet-99', '3d-unet-99.9'])`
        * CM names: `--adr.['ml-model', '3d-unet-model']...`
@@ -662,7 +688,7 @@ ___
        * `if (CM_MODEL in ['dlrm-99', 'dlrm-99.9'])`
        * CM names: `--adr.['ml-model', 'dlrm-model']...`
        - CM script: [get-ml-model-dlrm-terabyte](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-ml-model-dlrm-terabyte)
-     * get,dataset,image-classification,imagenet,preprocessed,_default
+     * get,dataset,image-classification,imagenet,preprocessed
        * `if (CM_MODEL  == resnet50) AND (CM_MLPERF_VISION_DATASET_OPTION  != True)`
        * CM names: `--adr.['imagenet-preprocessed']...`
        - CM script: [get-preprocessed-dataset-imagenet](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-preprocessed-dataset-imagenet)
@@ -687,6 +713,9 @@ ___
      * get,dataset-aux,squad-vocab
        * `if (CM_MODEL in ['bert-99', 'bert-99.9'])`
        - CM script: [get-dataset-squad-vocab](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-dataset-squad-vocab)
+     * get,dataset,coco2014,_validation
+       * `if (CM_MODEL  == sdxl)`
+       - CM script: [get-dataset-coco2014](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-dataset-coco2014)
      * get,dataset,kits19,preprocessed
        * `if (CM_MODEL in ['3d-unet-99', '3d-unet-99.9'])`
        - CM script: [get-preprocessed-dataset-kits19](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-preprocessed-dataset-kits19)

cm-mlops/script/app-mlperf-inference/README.md

@@ -143,6 +143,13 @@ ___
              * `if (CM_SKIP_RUN  != True)`
              * CM names: `--adr.['cpp-mlperf-inference', 'mlperf-inference-implementation']...`
              - CM script: [app-mlperf-inference-cpp](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/app-mlperf-inference-cpp)
+    * `_kilt`
+      - Workflow:
+        1. ***Read "posthook_deps" on other CM scripts***
+           * reproduce,mlperf,inference,kilt
+             * `if (CM_SKIP_RUN  != True)`
+             * CM names: `--adr.['kilt', 'kilt-harness', 'mlperf-inference-implementation']...`
+             - CM script: [reproduce-mlperf-inference-kilt](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/reproduce-mlperf-inference-kilt)
     * `_nvidia`
       - Environment variables:
         - *CM_MLPERF_IMPLEMENTATION*: `nvidia`
@@ -221,6 +228,10 @@ ___
       - Environment variables:
         - *CM_MLPERF_BACKEND*: `deepsparse`
       - Workflow:
+    * `_glow`
+      - Environment variables:
+        - *CM_MLPERF_BACKEND*: `glow`
+      - Workflow:
     * `_ncnn`
       - Environment variables:
         - *CM_MLPERF_BACKEND*: `ncnn`
@@ -273,6 +284,10 @@ ___
       - Environment variables:
         - *CM_MLPERF_DEVICE*: `gpu`
       - Workflow:
+    * `_qaic`
+      - Environment variables:
+        - *CM_MLPERF_DEVICE*: `qaic`
+      - Workflow:
     * `_rocm`
       - Environment variables:
         - *CM_MLPERF_DEVICE*: `rocm`
@@ -375,6 +390,15 @@ ___
              * `if (CM_MLPERF_LOADGEN_MODE in ['accuracy', 'all'] AND CM_MLPERF_ACCURACY_RESULTS_DIR  == on) AND (CM_MLPERF_IMPLEMENTATION not in ['nvidia-original', 'reference'])`
              * CM names: `--adr.['mlperf-accuracy-script', 'librispeech-accuracy-script']...`
              - CM script: [process-mlperf-accuracy](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/process-mlperf-accuracy)
+    * `_sdxl`
+      - Environment variables:
+        - *CM_MODEL*: `sdxl`
+      - Workflow:
+        1. ***Read "post_deps" on other CM scripts***
+           * run,accuracy,mlperf,_coco2014
+             * `if (CM_MLPERF_LOADGEN_MODE in ['accuracy', 'all'] AND CM_MLPERF_ACCURACY_RESULTS_DIR  == on) AND (CM_MLPERF_IMPLEMENTATION not in ['nvidia-original', 'reference'])`
+             * CM names: `--adr.['mlperf-accuracy-script', 'coco2014-accuracy-script']...`
+             - *Warning: no scripts found*
 
     </details>
 
@@ -393,7 +417,8 @@ ___
         - *CM_MLPERF_QUANTIZATION*: `False`
         - *CM_MLPERF_MODEL_PRECISION*: `float32`
       - Workflow:
-    * **`_fp32`** (default)
+    * **`_float32`** (default)
+      - Aliases: `_fp32`
       - Environment variables:
         - *CM_MLPERF_QUANTIZATION*: `False`
         - *CM_MLPERF_MODEL_PRECISION*: `float32`
@@ -571,7 +596,7 @@ ___
 
 #### Default variations
 
-`_cpu,_fp32,_offline,_onnxruntime,_reference,_resnet50,_test`
+`_cpu,_float32,_offline,_onnxruntime,_reference,_resnet50,_test`
 
 #### Input description
 

cm-mlops/script/calibrate-model-for.qaic/README.md

@@ -102,27 +102,10 @@ ___
 
     * `_first.#`
       - Workflow:
-    * `_resnet50`
-      - Environment variables:
-        - *CM_CALIBRATE_IMAGENET*: `yes`
-        - *CM_QAIC_COMPILER_ARGS*: ``
-        - *CM_QAIC_COMPILER_PARAMS*: `-output-node-name=ArgMax -profiling-threads=8`
-        - *CM_QAIC_OUTPUT_NODE_NAME*: `-output-node-name=ArgMax`
-        - *CM_QAIC_MODEL_TO_CONVERT*: `calibrate_resnet50_tf`
-      - Workflow:
     * `_resnet50,tf`
       - Environment variables:
         - *CM_QAIC_MODEL_TO_CONVERT*: `calibrate_resnet50_tf`
       - Workflow:
-    * `_retinanet`
-      - Environment variables:
-        - *CM_CALIBRATE_OPENIMAGES*: `yes`
-        - *CM_QAIC_COMPILER_ARGS*: ``
-        - *CM_QAIC_NODE_PRECISION_FILE_PATH*: `retinanet/node-precision.yaml`
-        - *CM_QAIC_COMPILER_PARAMS*: `-enable-channelwise -profiling-threads=32 -onnx-define-symbol=batch_size,<<<CM_QAIC_MODEL_BATCH_SIZE>>> -node-precision-info=<<<CM_QAIC_NODE_PRECISION_FILE_PATH>>>`
-        - *CM_IMAGE_ORDER_FILE_PATH*: `retinanet/openimages_cal_images_list.txt`
-        - *CM_QAIC_MODEL_TO_CONVERT*: `calibrate_retinanet_no_nms_mlperf`
-      - Workflow:
 
     </details>
 
@@ -134,10 +117,12 @@ ___
     * `_bs.#`
       - Environment variables:
         - *CM_QAIC_MODEL_BATCH_SIZE*: `#`
+        - *CM_CREATE_INPUT_BATCH*: `yes`
       - Workflow:
-    * **`_bs.1`** (default)
+    * `_bs.1`
       - Environment variables:
         - *CM_QAIC_MODEL_BATCH_SIZE*: `1`
+        - *CM_CREATE_INPUT_BATCH*: `yes`
       - Workflow:
 
     </details>
@@ -155,19 +140,51 @@ ___
     </details>
 
 
-  * Group "**model-framework**"
+  * Group "**model**"
     <details>
     <summary>Click here to expand this section.</summary>
 
-    * `_tf`
+    * `_bert-99`
+      - Environment variables:
+        - *CM_CALIBRATE_SQUAD*: `yes`
+        - *CM_QAIC_COMPILER_ARGS*: ``
+        - *CM_QAIC_COMPILER_PARAMS*: `-onnx-define-symbol=batch_size,1 -onnx-define-symbol=seg_length,<<<CM_DATASET_SQUAD_TOKENIZED_MAX_SEQ_LENGTH>>> -input-list-file=<<<CM_DATASET_SQUAD_TOKENIZED_PACKED_FILENAMES_FILE>>> -num-histogram-bins=512 -profiling-threads=4`
+        - *CM_QAIC_MODEL_TO_CONVERT*: `calibrate_bert_mlperf`
+      - Workflow:
+        1. ***Read "deps" on other CM scripts***
+           * get,preprocessed,dataset,squad,_packed,_pickle
+             * CM names: `--adr.['squad-preprocessed', 'preprocessed-dataset']...`
+             - CM script: [get-preprocessed-dataset-squad](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-preprocessed-dataset-squad)
+    * `_resnet50`
+      - Environment variables:
+        - *CM_CALIBRATE_IMAGENET*: `yes`
+        - *CM_QAIC_COMPILER_ARGS*: ``
+        - *CM_QAIC_COMPILER_PARAMS*: `-output-node-name=ArgMax -profiling-threads=8`
+        - *CM_QAIC_OUTPUT_NODE_NAME*: `-output-node-name=ArgMax`
+        - *CM_QAIC_MODEL_TO_CONVERT*: `calibrate_resnet50_tf`
+      - Workflow:
+    * `_retinanet`
+      - Environment variables:
+        - *CM_CALIBRATE_OPENIMAGES*: `yes`
+        - *CM_QAIC_COMPILER_ARGS*: ``
+        - *CM_QAIC_NODE_PRECISION_FILE_PATH*: `retinanet/node-precision.yaml`
+        - *CM_QAIC_COMPILER_PARAMS*: `-enable-channelwise -profiling-threads=32 -onnx-define-symbol=batch_size,<<<CM_QAIC_MODEL_BATCH_SIZE>>> -node-precision-info=<<<CM_QAIC_NODE_PRECISION_FILE_PATH>>>`
+        - *CM_IMAGE_ORDER_FILE_PATH*: `retinanet/openimages_cal_images_list.txt`
+        - *CM_QAIC_MODEL_TO_CONVERT*: `calibrate_retinanet_no_nms_mlperf`
       - Workflow:
 
     </details>
 
 
-#### Default variations
+  * Group "**model-framework**"
+    <details>
+    <summary>Click here to expand this section.</summary>
+
+    * `_tf`
+      - Workflow:
+
+    </details>
 
-`_bs.1`
 #### Default environment
 
 <details>
@@ -196,9 +213,13 @@ ___
        * `if (CM_CALIBRATE_IMAGENET  == on)`
        * CM names: `--adr.['imagenet-cal', 'preprocessed-dataset']...`
        - CM script: [get-preprocessed-dataset-imagenet](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-preprocessed-dataset-imagenet)
+     * get,dataset,calibration,squad,_pickle,_width.384
+       * CM names: `--adr.['squad-cal', 'preprocessed-dataset']...`
+       - *Warning: no scripts found*
      * get,ml-model
        * CM names: `--adr.['model-src']...`
        - CM script: [get-ml-model-3d-unet-kits19](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-ml-model-3d-unet-kits19)
+       - CM script: [get-ml-model-abtf-ssd-pytorch](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-ml-model-abtf-ssd-pytorch)
        - CM script: [get-ml-model-bert-base-squad](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-ml-model-bert-base-squad)
        - CM script: [get-ml-model-bert-large-squad](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-ml-model-bert-large-squad)
        - CM script: [get-ml-model-dlrm-terabyte](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-ml-model-dlrm-terabyte)
@@ -211,6 +232,7 @@ ___
        - CM script: [get-ml-model-retinanet](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-ml-model-retinanet)
        - CM script: [get-ml-model-retinanet-nvidia](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-ml-model-retinanet-nvidia)
        - CM script: [get-ml-model-rnnt](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-ml-model-rnnt)
+       - CM script: [get-ml-model-stable-diffusion](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-ml-model-stable-diffusion)
        - CM script: [get-ml-model-tiny-resnet](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-ml-model-tiny-resnet)
        - CM script: [get-ml-model-using-imagenet-from-model-zoo](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-ml-model-using-imagenet-from-model-zoo)
   1. ***Run "preprocess" function from [customize.py](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/calibrate-model-for.qaic/customize.py)***
@@ -230,7 +252,6 @@ ___
 * `CM_QAIC_*`
 #### New environment keys auto-detected from customize
 
-* `CM_QAIC_EXEC_PATH`
 * `CM_QAIC_MODEL_PROFILE_WITH_PATH`
 ___
 ### Maintainers