regular updates (#1062)

README.md

@@ -17,7 +17,7 @@ CM is being developed by the [MLCommons Task Force on Automation and Reproducibi
 based on the feedback from the [research community](https://www.youtube.com/watch?v=7zpeIVwICa4), Google, AMD, Neural Magic, Nvidia, Qualcomm, Dell, HPE, Red Hat,
 Intel, TTA, One Stop Systems, ACM and [other organizations and individual contributors](https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md).
 
-The goal is to gradually convert all ad-hoc README files, papers, reports, Juputer notebooks and containers 
+The goal is to help the community gradually convert all ad-hoc README files, papers, reports, Juputer notebooks and containers 
 into [portable and reusable automation recipes (CM scripts)](https://github.com/mlcommons/ck/blob/master/cm-mlops/script)
 that find and call existing scripts and tools via English-like language based on tags 
 and glue them together via extensible JSON/YAML meta descriptions and simple Python logic.
@@ -31,7 +31,7 @@ cm pull repo mlcommons@ck
 cm run script "python app image-classification onnx" --input={some image.jpg}
 ```
 
-Such approach requires minimal learning curve and minimal changes to existing projects while helping 
+Such approach requires minimal learning curve and minimal or no changes to existing projects while helping 
 to dramatically reduce time to understand how to run and customize numerous AI/ML projects 
 across diverse and continuously changing models, datasets, software and hardware from different vendors.
 
@@ -43,7 +43,7 @@ environments, generating command lines, processing logs and sharing results*: se
 Please check this [Getting Started tutorial](docs/getting-started.md) to understand
 how CM works and start using it.
 
-Note that this project is under heavy developments - feel free to get in touch
+Note that this project is under heavy development - feel free to get in touch
 via [public Discord server](https://discord.gg/JjWNWXKxwT) if you have questions, 
 suggestions and feature requests.
 

cm-mlops/automation/script/module.py

@@ -3674,12 +3674,12 @@ def enable_or_skip_script(meta, env):
         if key in env:
             value = str(env[key]).lower()
 
-            if value in ["yes", "on", "true", "1"]:
-                if value in (meta[key] + ["yes", "on", "true", "1"]):
-                    continue
-            elif set(meta[key]) & set(["yes", "on", "true", "1"]):
+            if set(meta[key]) & set(["yes", "on", "true", "1"]):
                 if value not in ["no", "off", "false", "0"]:
                     continue
+            elif set(meta[key]) & set(["no", "off", "false", "0"]):
+                if value in ["no", "off", "false", "0"]:
+                    continue
             elif value in meta[key]:
                 continue
         return False

cm-mlops/script/README.md

@@ -1 +1,19 @@
-[ [Read about CM scripts here](../README.md) ]
+### About
+
+These automation recipes wrap native MLOps and DevOps scripts and tools to unify them and run them 
+via a simple, common and human-readable [CM interface](https://github.com/mlcommons/ck) 
+on any platform with any software stack.
+
+They are being developed and maintained by the [MLCommons Task Force on Automation and Reproducibility](../../docs/taskforce.md)
+with [great contributions](../../CONTRIBUTING.md) from the community and important feedback from Google, AMD, Neural Magic, Nvidia, Qualcomm, Dell, HPE, 
+Red Hat, Intel, TTA, One Stop Systems, ACM and other organizations.
+
+See the automatically generated catalog [here](https://github.com/mlcommons/ck/blob/master/docs/list_of_scripts.md).
+
+### Copyright
+
+2022-2024 [MLCommons](https://mlcommons.org)
+
+### License
+
+[Apache 2.0](LICENSE.md)

cm-mlops/script/app-image-classification-onnx-py/_cm.yaml

@@ -48,3 +48,6 @@ variations:
   cuda:
     env:
       USE_CUDA: yes
+
+input_mapping:
+  input: CM_IMAGE

cm-mlops/script/install-llvm-src/_cm.json

@@ -127,7 +127,7 @@
         "CM_GIT_CHECKOUT_TAG": "#"
       }
     },
-    "for-intel-mlperf-inference": {
+    "for-intel-mlperf-inference-v3.1-bert": {
       "base": [
         "tag.llvmorg-15.0.7",
         "clang",

cm-mlops/script/install-onednn-from.src/_cm.json

@@ -72,7 +72,7 @@
         "CM_GIT_CHECKOUT": "#"
       }
     },
-    "for-intel-mlperf-inference": {
+    "for-intel-mlperf-inference-v3.1-bert": {
       "adr": {
         "conda-package": {
           "tags": "_name.bert-pt"

cm-mlops/script/install-pytorch-from.src/_cm.json

@@ -98,7 +98,7 @@
         }
       }
     },
-    "for-intel-mlperf-inference": {
+    "for-intel-mlperf-inference-v3.1-bert": {
       "base": [
         "tag.v1.12.0"
       ],
@@ -134,7 +134,7 @@
           "version": "3.8"
         },
         {
-          "tags": "install,llvm,src,_tag.llvmorg-15.0.7,_runtimes.libcxx:libcxxabi:openmp,_clang,_release,_for-intel-mlperf-inference"
+          "tags": "install,llvm,src,_tag.llvmorg-15.0.7,_runtimes.libcxx:libcxxabi:openmp,_clang,_release,_for-intel-mlperf-inference-v3.1-bert"
         },
         {
           "names": [

cm-mlops/script/install-transformers-from.src/_cm.json

@@ -82,7 +82,7 @@
         "CM_GIT_CHECKOUT": "#"
       }
     },
-    "for-intel-mlperf-inference": {
+    "for-intel-mlperf-inference-v3.1-bert": {
       "adr": {
         "conda-package": {
           "tags": "_name.bert-pt"

cm-mlops/script/reproduce-mlperf-inference-intel/_cm.yaml

@@ -85,24 +85,6 @@ deps:
     names:
     - mlperf-logging
 
-  - tags: get,conda,_name.bert-pt
-  - tags: install,llvm,src,_tag.llvmorg-15.0.7,_runtimes.libcxx:libcxxabi:openmp,_clang,_release,_for-intel-mlperf-inference
-  - tags: get,generic-sys-util,_libffi7
-  - tags: get,generic,conda-package,_package.python
-    names:
-     - conda-package
-     - python
-    version: "3.8"
-  - names:
-     - conda-package
-     - ncurses
-    tags: get,generic,conda-package,_package.ncurses,_source.conda-forge
-
-  - tags: get,generic-sys-util,_numactl
-  - tags: get,generic,conda-package,_package.jemalloc,_source.conda-forge
-    names:
-     - conda-package
-     - jemalloc
 
   ########################################################################
   # Install ResNet50 model (ONNX) and ImageNet
@@ -191,11 +173,6 @@ variations:
     env:
       CM_MLPERF_BACKEND: pytorch
       CM_MLPERF_BACKEND_LIB_NAMESPEC: pytorch
-    deps:
-    - tags: get,pytorch,from.src,_for-intel-mlperf-inference
-    - tags: install,onednn,from.src,_for-intel-mlperf-inference
-
-
 
   bs.#:
     env:
@@ -241,14 +218,34 @@ variations:
 
 
   bert_:
-    deps:
-    - tags: install,transformers,from.src,_for-intel-mlperf-inference
     env:
       CM_BENCHMARK: STANDALONE_BERT
       dataset_squad_tokenized_max_seq_length: 384
       loadgen_buffer_size: 10833
       loadgen_dataset_size: 10833
 
+  bert_,pytorch:
+    - tags: get,conda,_name.bert-pt
+    - tags: install,llvm,src,_tag.llvmorg-15.0.7,_runtimes.libcxx:libcxxabi:openmp,_clang,_release,_for-intel-mlperf-inference-v3.1-bert
+    - tags: get,generic-sys-util,_libffi7
+    - tags: get,generic,conda-package,_package.python
+      names:
+       - conda-package
+       - python
+      version: "3.8"
+    - names:
+       - conda-package
+       - ncurses
+      tags: get,generic,conda-package,_package.ncurses,_source.conda-forge
+
+    - tags: get,generic-sys-util,_numactl
+    - tags: get,generic,conda-package,_package.jemalloc,_source.conda-forge
+      names:
+       - conda-package
+       - jemalloc
+    - tags: get,pytorch,from.src,_for-intel-mlperf-inference-v3.1-bert
+    - tags: install,onednn,from.src,_for-intel-mlperf-inference-v3.1-bert
+    - tags: install,transformers,from.src,_for-intel-mlperf-inference-v3.1-bert
 
   standalone:
     group: network-mode

cm-mlops/script/run-all-mlperf-models/run-mobilenet-models.sh

@@ -27,7 +27,7 @@ POWER=""
 extra_option=" --adr.mlperf-inference-implementation.compressed_dataset=on"
 extra_option=""
 extra_tags=""
-#extra_tags=",_only-fp32"
+extra_tags=",_only-fp32"
 
 
 #Add your run commands here...

docs/getting-started.md

@@ -1,13 +1,160 @@
-[ [Back to index](README.md) ]
+[ [Back to documentation](README.md) ]
 
-TBD
+*Under development*
 
-For example, the following commands should prepare and run image classification 
-with ONNX on Linux, Windows and MacOS in a native environment:
+# CM Getting Started Guide
+
+## Image classification example
+
+One of the goals of the [MLCommons CM workflow automation framework (CM)](https://github.com/mlcommons/ck?tab=readme-ov-file#about) 
+is to provide a common, simple and human readable interface to run and manage complex software projects and benchmarks 
+on any platform with any software stack in a unified and automated way.
+
+This tutorial explains how CM works and should help you start using it with existing projects 
+or to modularize and unify your own projects.
+
+Let us test CM to run image classification from the command line on any platform with Windows, Linux and MacOS.
+
+### Installing CM
+
+CM is implemented as a [very small Python library](https://github.com/mlcommons/ck/tree/master/cm/cmind) 
+with `cm` and `cmr` front-ends and minimal dependencies (Python 3+, git and wget) 
+that can be installed via PIP:
 
 
 ```bash
 pip install cmind
+```
+
+You may need to re-login to update the PATH to `cm` and `cmr` front-ends.
+
+Note that CM can be also installed from virtual environment (required in Ubuntu 23.04+) and inside containers.
+You can check a detailed guide to install CM on different platforms [here](installation.md).
+
+### Pulling some repository with embedded CM interface
+
+Let's now pull a Git repository that has embedded CM interface 
+(note that if your Git repository doesn't have CM interface embedded,
+CM will automatically initialize one):
+
+```bash
 cm pull repo mlcommons@ck
-cmr "python app image-classification onnx"
 ```
+
+CM will pull GitHub repository from `https://github.com/mlcommons/ck` to the `CM/repos` directory in your local HOME directory.
+You can use flag `--url=https://github.com/mlcommons/ck` instead of `mlcommons@ck` to pull any Git repository.
+
+CM will then check if this repository has a CM interface by checking the [`cmr.yaml`](https://github.com/mlcommons/ck/blob/master/cmr.yaml) 
+file in the root directory of this repository (abbreviation for `C`ollective `M`ind `R`epository):
+
+```yaml
+git: true
+alias: mlcommons@ck
+uid: a4705959af8e447a
+version: 1.5.4
+prefix: cm-mlops
+```
+
+Note that this file will be automatically generated if it doesn't exist in your repository.
+
+While working on modularizing, unifying and automating MLPerf benchmarks,
+we decided to embed a CM interface to this development repository 
+in the [cm-mlops directory](https://github.com/mlcommons/ck/tree/master/cm-mlops)
+
+The `prefix` in `cmr.yaml` tells CM to search for the CM interface in some sub-directory of a given repository
+to avoid altering the original structure of software projects.
+
+### Using CM interface to run a given software project
+
+You can now invoke a human-friendly CM command to run your project such as image classification
+(we will show how to use Python API later):
+
+```bash
+cm run script "python app image-classification onnx"
+```
+
+CM will recursively walk through all pulled or downloaded repositories in your home `CM/repos` directory
+and search for matching tags `python,app,image-classification,onnx` in all `_cm.yaml` or `_cm.json`
+files in a `script` sub-directory of all repositories.
+
+In our case, CM will find 1 match in 
+the [`cm-mlops/script/app-image-classification-onnx-py/_cm.yaml`](https://github.com/mlcommons/ck/blob/master/cm-mlops/script/app-image-classification-onnx-py/_cm.yaml).
+
+This file tells CM how to prepare environment variables, paths and command lines 
+to run a native script or tool on any platform.
+
+
+#### 
+
+
+
+
+
+
+
+
+
+
+
+
+#### Using inputs and environment variables
+
+env
+
+const
+
+
+default_env
+
+input_mapping
+
+
+
+#### Using variations
+
+using the same code/script/tool but altering it's behavior and sub-dependencies
+
+CUDA
+
+_cuda
+
+
+#### Reporting issues
+
+The community helped us test this example on many platforms but if you still encounter
+some issues, please report them [here](https://github.com/mlcommons/ck/issues) - CM is not a magic (yet)
+and our concept is to collaboratively extend CM workflows to gradually improve their portability and reproducibility
+across diverse software and hardware.
+
+
+
+#### Debugging CM interface
+
+
+#### Extending CM interface
+
+
+### Reusing automation recipes
+
+
+
+### Adding CM interface to your own project
+
+
+
+### Using CM with containers
+
+
+### Using CM GUI
+
+
+### Running MLPerf and other projects via CM 
+
+Recent examples from MLPerf and ML, compiler and systems conferences
+
+
+### Participating in collaborative developments
+
+This is a community project being developed by the [MLCommons Task Force on Automation and Reproducibility](taskforce.md)
+based on your feedback - please join our [public Discord server](https://discord.gg/JjWNWXKxwT) if you 
+would like to help with developments or have questions, suggestions and feature requests.

docs/taskforce.md

@@ -4,8 +4,12 @@
 
 ## Mission
 
-Develop a common methodology, interface and workflow to run, reproduce, customize and optimize different implementations of MLPerf benchmarks 
-in a unified way across diverse and rapidly evolving models, data sets, software and hardware from different vendors.
+* Develop a common interface and workflow to run, reproduce, customize and optimize different implementations of MLPerf benchmarks 
+  in a unified and automated way across diverse models, data sets, software and hardware from different vendors: 
+  <a href="https://github.com/mlcommons/ck">Collective Mind (MLCommons CM)</a>.
+* Gradually add the MLCommons CM interface to [all MLPerf inference submissions](https://github.com/mlcommons/ck/issues/1052) starting from v3.1.
+* Continuously encode MLPerf rules and best practices in the MLCommons CM interface and workflows for MLPerf
+  to reduce the burden for submitters.
 
 ## Chairs and Tech Leads