Skip to content

Commit

Permalink
add code snippet count to find-snippets
Browse files Browse the repository at this point in the history
  • Loading branch information
@sdgilley
sdgilley committed Mar 1, 2024
1 parent 2ce405c commit be63ef9
Show file tree
Hide file tree
Showing 5 changed files with 2,341 additions and 29 deletions.
3 changes: 3 additions & 0 deletions GitHub/CODEOWNERS.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -175,6 +175,9 @@
/cli/jobs/spark/serverless-spark-standalone-user-identity.yaml @sdgilley @msakande @Blackmist @ssalgadodev @lgayhardt @fbsolo-ms1
/cli/jobs/spark/spark-job-component.yml @sdgilley @msakande @Blackmist @ssalgadodev @lgayhardt @fbsolo-ms1
/cli/misc.sh @sdgilley @msakande @Blackmist @ssalgadodev @lgayhardt @fbsolo-ms1
/cli/monitoring/advanced-model-monitoring.yaml @sdgilley @msakande @Blackmist @ssalgadodev @lgayhardt @fbsolo-ms1
/cli/monitoring/model-monitoring-with-collected-data.yaml @sdgilley @msakande @Blackmist @ssalgadodev @lgayhardt @fbsolo-ms1
/cli/monitoring/out-of-box-monitoring.yaml @sdgilley @msakande @Blackmist @ssalgadodev @lgayhardt @fbsolo-ms1
/cli/resources/compute/cluster-basic.yml @sdgilley @msakande @Blackmist @ssalgadodev @lgayhardt @fbsolo-ms1
/cli/resources/compute/cluster-location.yml @sdgilley @msakande @Blackmist @ssalgadodev @lgayhardt @fbsolo-ms1
/cli/resources/compute/cluster-low-priority.yml @sdgilley @msakande @Blackmist @ssalgadodev @lgayhardt @fbsolo-ms1
Expand Down
19 changes: 18 additions & 1 deletion GitHub/find-snippets.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@

found = pd.DataFrame(columns=['ref_file', 'from_file'])
dict_list = []
dict_list2 = []
branches = []
# Record the start time
start_time = datetime.now()
Expand All @@ -45,7 +46,10 @@
# Get the file content
file_content = content_file.decoded_content
lines = file_content.decode().splitlines()

blocks = []
count = 0
code_type = None
inside_code_block = False
for line in lines:
# snippets have ~\azureml-examples in them. Find all snippets in this file.
match_snippet = re.findall(r'\(~\/azureml-examples[^)]*\)|source="~\/azureml-examples[^"]*"', line)
Expand All @@ -56,6 +60,19 @@
if branch == az_ml_branch: #PRs are merged into main, so only these files are relevant
row_dict = {'ref_file': ref_file, 'from_file': file}
dict_list.append(row_dict)
# count lines in code snippets
blocks, inside_code_block, count, code_type = h.count_code_lines(line, blocks, inside_code_block, count, code_type)
# done looking through lines of this file
if inside_code_block:
print(f"{file}: Warning: A code block started but did not end.")
print(f" The last code block type was {code_type} and had {count} lines.")
if blocks:
# this file has code blocks. add info to the dictionary
for block in blocks:
dict_list2.append({'file': file, 'type': block[0], 'lines': block[1]})

code_counts = pd.DataFrame.from_dict(dict_list2)
code_counts.to_csv("code-counts.csv", index=False)

found = pd.DataFrame.from_dict(dict_list)
branches = pd.DataFrame(branches)
Expand Down
57 changes: 30 additions & 27 deletions GitHub/refs-found.csv
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,13 @@ cli/assets/data/cloud-folder-wasbs.yml,reference-yaml-data.md
cli/assets/data/cloud-folder.yml,reference-yaml-data.md
cli/assets/data/local-file.yml,reference-yaml-data.md
cli/assets/data/local-folder.yml,reference-yaml-data.md
cli/assets/environment/docker-context.yml,how-to-manage-environments-v2.md
cli/assets/environment/docker-context.yml,reference-yaml-environment.md
cli/assets/environment/docker-image-plus-conda.yaml,how-to-manage-environments-v2.md
cli/assets/environment/docker-context.yml,how-to-manage-environments-v2.md
cli/assets/environment/docker-image-plus-conda.yaml,reference-yaml-environment.md
cli/assets/environment/docker-image.yml,how-to-identity-based-service-authentication.md
cli/assets/environment/docker-image-plus-conda.yaml,how-to-manage-environments-v2.md
cli/assets/environment/docker-image.yml,how-to-manage-environments-v2.md
cli/assets/environment/docker-image.yml,reference-yaml-environment.md
cli/assets/environment/docker-image.yml,how-to-identity-based-service-authentication.md
cli/assets/model/local-file.yml,reference-yaml-model.md
cli/assets/model/local-file.yml,how-to-manage-models.md
cli/assets/model/local-mlflow.yml,reference-yaml-model.md
Expand All @@ -41,8 +41,8 @@ cli/endpoints/batch/deploy-models/custom-outputs-parquet/endpoint.yml,how-to-dep
cli/endpoints/batch/deploy-models/heart-classifier-mlflow/deploy-and-run.sh,how-to-mlflow-batch.md
cli/endpoints/batch/deploy-models/heart-classifier-mlflow/deployment-custom/code/batch_driver.py,how-to-mlflow-batch.md
cli/endpoints/batch/deploy-models/heart-classifier-mlflow/deployment-custom/deployment.yml,how-to-mlflow-batch.md
cli/endpoints/batch/deploy-models/heart-classifier-mlflow/deployment-simple/deployment.yml,reference-yaml-deployment-batch.md
cli/endpoints/batch/deploy-models/heart-classifier-mlflow/deployment-simple/deployment.yml,how-to-mlflow-batch.md
cli/endpoints/batch/deploy-models/heart-classifier-mlflow/deployment-simple/deployment.yml,reference-yaml-deployment-batch.md
cli/endpoints/batch/deploy-models/heart-classifier-mlflow/endpoint.yml,how-to-mlflow-batch.md
cli/endpoints/batch/deploy-models/heart-classifier-mlflow/heart-dataset-unlabeled.yml,how-to-mlflow-batch.md
cli/endpoints/batch/deploy-models/huggingface-text-summarization/code/batch_driver.py,how-to-nlp-processing-batch.md
Expand All @@ -66,10 +66,10 @@ cli/endpoints/batch/deploy-models/mnist-classifier/deployment-torch/code/batch_d
cli/endpoints/batch/deploy-models/mnist-classifier/deployment-torch/deployment.yml,how-to-use-batch-model-deployments.md
cli/endpoints/batch/deploy-models/mnist-classifier/deployment-torch/deployment.yml,reference-yaml-deployment-batch.md
cli/endpoints/batch/deploy-models/mnist-classifier/deployment-torch/deployment.yml,how-to-batch-scoring-script.md
cli/endpoints/batch/deploy-models/mnist-classifier/deployment-torch/environment/conda.yaml,how-to-use-batch-model-deployments.md
cli/endpoints/batch/deploy-models/mnist-classifier/deployment-torch/environment/conda.yaml,how-to-batch-scoring-script.md
cli/endpoints/batch/deploy-models/mnist-classifier/endpoint.yml,how-to-use-batch-model-deployments.md
cli/endpoints/batch/deploy-models/mnist-classifier/deployment-torch/environment/conda.yaml,how-to-use-batch-model-deployments.md
cli/endpoints/batch/deploy-models/mnist-classifier/endpoint.yml,reference-yaml-endpoint-batch.md
cli/endpoints/batch/deploy-models/mnist-classifier/endpoint.yml,how-to-use-batch-model-deployments.md
cli/endpoints/batch/deploy-models/openai-embeddings/code/batch_driver.py,how-to-use-batch-model-openai-embeddings.md
cli/endpoints/batch/deploy-models/openai-embeddings/deploy-and-run.sh,how-to-use-batch-model-openai-embeddings.md
cli/endpoints/batch/deploy-models/openai-embeddings/deployment.yml,how-to-use-batch-model-openai-embeddings.md
Expand All @@ -84,13 +84,13 @@ cli/endpoints/batch/deploy-pipelines/batch-scoring-with-preprocessing/environmen
cli/endpoints/batch/deploy-pipelines/batch-scoring-with-preprocessing/inputs.yml,how-to-use-batch-scoring-pipeline.md
cli/endpoints/batch/deploy-pipelines/batch-scoring-with-preprocessing/pipeline-job.yml,how-to-use-batch-scoring-pipeline.md
cli/endpoints/batch/deploy-pipelines/batch-scoring-with-preprocessing/pipeline.yml,how-to-use-batch-scoring-pipeline.md
cli/endpoints/batch/deploy-pipelines/hello-batch/deploy-and-run.sh,how-to-use-batch-pipeline-deployments.md
cli/endpoints/batch/deploy-pipelines/hello-batch/deploy-and-run.sh,how-to-use-batch-pipeline-from-job.md
cli/endpoints/batch/deploy-pipelines/hello-batch/deploy-and-run.sh,how-to-use-batch-pipeline-deployments.md
cli/endpoints/batch/deploy-pipelines/hello-batch/deployment-from-job.yml,how-to-use-batch-pipeline-from-job.md
cli/endpoints/batch/deploy-pipelines/hello-batch/deployment.yml,reference-yaml-deployment-batch.md
cli/endpoints/batch/deploy-pipelines/hello-batch/deployment.yml,how-to-use-batch-pipeline-deployments.md
cli/endpoints/batch/deploy-pipelines/hello-batch/endpoint.yml,how-to-use-batch-pipeline-deployments.md
cli/endpoints/batch/deploy-pipelines/hello-batch/deployment.yml,reference-yaml-deployment-batch.md
cli/endpoints/batch/deploy-pipelines/hello-batch/endpoint.yml,how-to-use-batch-pipeline-from-job.md
cli/endpoints/batch/deploy-pipelines/hello-batch/endpoint.yml,how-to-use-batch-pipeline-deployments.md
cli/endpoints/batch/deploy-pipelines/hello-batch/hello-component/hello.yml,how-to-use-batch-pipeline-deployments.md
cli/endpoints/batch/deploy-pipelines/hello-batch/pipeline-job.yml,how-to-use-batch-pipeline-from-job.md
cli/endpoints/batch/deploy-pipelines/training-with-components/deploy-and-run.sh,how-to-use-batch-training-pipeline.md
Expand All @@ -105,42 +105,42 @@ cli/endpoints/batch/deploy-pipelines/training-with-components/environment/xgboos
cli/endpoints/batch/deploy-pipelines/training-with-components/inputs.yml,how-to-use-batch-training-pipeline.md
cli/endpoints/online/custom-container/tfserving/half-plus-two/tfserving-deployment.yml,how-to-deploy-custom-container.md
cli/endpoints/online/custom-container/tfserving/half-plus-two/tfserving-endpoint.yml,how-to-deploy-custom-container.md
cli/endpoints/online/deploy-with-packages/custom-model/deploy.sh,how-to-package-models-moe.md
cli/endpoints/online/deploy-with-packages/custom-model/deploy.sh,how-to-package-models.md
cli/endpoints/online/deploy-with-packages/custom-model/deploy.sh,how-to-package-models-moe.md
cli/endpoints/online/deploy-with-packages/custom-model/deployment.yml,how-to-package-models-moe.md
cli/endpoints/online/deploy-with-packages/custom-model/environment/conda.yaml,how-to-package-models-moe.md
cli/endpoints/online/deploy-with-packages/custom-model/environment/conda.yaml,how-to-package-models.md
cli/endpoints/online/deploy-with-packages/custom-model/environment/sklearn-regression-env.yml,how-to-package-models.md
cli/endpoints/online/deploy-with-packages/custom-model/environment/sklearn-regression-env.yml,how-to-package-models-moe.md
cli/endpoints/online/deploy-with-packages/custom-model/environment/sklearn-regression-env.yml,how-to-package-models.md
cli/endpoints/online/deploy-with-packages/custom-model/package-external.yml,how-to-package-models.md
cli/endpoints/online/deploy-with-packages/custom-model/package-moe.yml,how-to-package-models.md
cli/endpoints/online/deploy-with-packages/custom-model/package-moe.yml,how-to-package-models-moe.md
cli/endpoints/online/deploy-with-packages/custom-model/package-moe.yml,how-to-package-models.md
cli/endpoints/online/deploy-with-packages/custom-model/sample-request.json,how-to-package-models-moe.md
cli/endpoints/online/deploy-with-packages/mlflow-model/deploy.sh,concept-package-models.md
cli/endpoints/online/deploy-with-packages/mlflow-model/deploy.sh,how-to-package-models-app-service.md
cli/endpoints/online/deploy-with-packages/mlflow-model/package-external.yml,how-to-package-models-app-service.md
cli/endpoints/online/deploy-with-packages/mlflow-model/sample-request.json,how-to-package-models-app-service.md
cli/endpoints/online/deploy-with-packages/registry-model/deploy.sh,how-to-package-models.md
cli/endpoints/online/deploy-with-packages/registry-model/package.yml,how-to-package-models.md
cli/endpoints/online/managed/managed-identities/1-sai-create-endpoint.yml,how-to-access-resources-from-endpoints-managed-identities.md
cli/endpoints/online/managed/managed-identities/1-sai-create-endpoint.yml,reference-yaml-endpoint-online.md
cli/endpoints/online/managed/managed-identities/1-sai-create-endpoint.yml,how-to-access-resources-from-endpoints-managed-identities.md
cli/endpoints/online/managed/managed-identities/1-uai-create-endpoint.yml,reference-yaml-endpoint-online.md
cli/endpoints/online/managed/managed-identities/1-uai-create-endpoint.yml,how-to-access-resources-from-endpoints-managed-identities.md
cli/endpoints/online/managed/managed-identities/2-sai-deployment.yml,reference-yaml-deployment-managed-online.md
cli/endpoints/online/managed/managed-identities/2-sai-deployment.yml,how-to-access-resources-from-endpoints-managed-identities.md
cli/endpoints/online/managed/managed-identities/2-uai-deployment.yml,reference-yaml-deployment-managed-online.md
cli/endpoints/online/managed/managed-identities/2-sai-deployment.yml,reference-yaml-deployment-managed-online.md
cli/endpoints/online/managed/managed-identities/2-uai-deployment.yml,how-to-access-resources-from-endpoints-managed-identities.md
cli/endpoints/online/managed/sample/blue-deployment.yml,how-to-deploy-automl-endpoint.md
cli/endpoints/online/managed/managed-identities/2-uai-deployment.yml,reference-yaml-deployment-managed-online.md
cli/endpoints/online/managed/sample/blue-deployment.yml,reference-yaml-deployment-managed-online.md
cli/endpoints/online/managed/sample/blue-deployment.yml,how-to-deploy-online-endpoints.md
cli/endpoints/online/managed/sample/blue-deployment.yml,how-to-deploy-automl-endpoint.md
cli/endpoints/online/managed/sample/blue-deployment.yml,how-to-safely-rollout-online-endpoints.md
cli/endpoints/online/managed/sample/blue-deployment.yml,reference-yaml-deployment-managed-online.md
cli/endpoints/online/managed/sample/endpoint.yml,how-to-deploy-automl-endpoint.md
cli/endpoints/online/managed/sample/endpoint.yml,how-to-deploy-online-endpoints.md
cli/endpoints/online/managed/sample/endpoint.yml,reference-yaml-endpoint-online.md
cli/endpoints/online/managed/sample/endpoint.yml,how-to-deploy-online-endpoints.md
cli/endpoints/online/managed/sample/endpoint.yml,how-to-safely-rollout-online-endpoints.md
cli/endpoints/online/managed/sample/green-deployment.yml,reference-yaml-deployment-managed-online.md
cli/endpoints/online/model-1/onlinescoring/score.py,how-to-inference-server-http.md
cli/endpoints/online/model-1/onlinescoring/score.py,how-to-deploy-online-endpoints.md
cli/endpoints/online/model-1/onlinescoring/score.py,how-to-inference-server-http.md
cli/endpoints/online/model-1/onlinescoring/score_managedidentity.py,how-to-access-resources-from-endpoints-managed-identities.md
cli/endpoints/online/model-1/sample-request.json,how-to-access-resources-from-endpoints-managed-identities.md
cli/endpoints/online/ncd/create-endpoint.yaml,how-to-deploy-mlflow-models-online-endpoints.md
Expand Down Expand Up @@ -199,8 +199,8 @@ cli/jobs/pipelines/automl/image-multiclass-classification-fridge-items-pipeline/
cli/jobs/pipelines/automl/image-multilabel-classification-fridge-items-pipeline/pipeline.yml,reference-automl-images-cli-multilabel-classification.md
cli/jobs/pipelines/automl/image-object-detection-task-fridge-items-pipeline/pipeline.yml,reference-automl-images-cli-object-detection.md
cli/jobs/pipelines/cifar-10/job.yml,reference-yaml-job-pipeline.md
cli/jobs/pipelines/iris-batch-prediction-using-parallel/pipeline.yml,reference-yaml-job-parallel.md
cli/jobs/pipelines/iris-batch-prediction-using-parallel/pipeline.yml,how-to-use-parallel-job-in-pipeline.md
cli/jobs/pipelines/iris-batch-prediction-using-parallel/pipeline.yml,reference-yaml-job-parallel.md
cli/jobs/single-step/pytorch/cifar-distributed/job.yml,reference-yaml-job-command.md
cli/jobs/single-step/r/iris/job.yml,reference-yaml-job-command.md
cli/jobs/single-step/scikit-learn/iris/job-sweep.yml,reference-yaml-job-sweep.md
Expand All @@ -214,6 +214,9 @@ cli/jobs/spark/serverless-spark-standalone-user-identity.yaml,reference-yaml-job
cli/jobs/spark/spark-job-component.yml,reference-yaml-component-spark.md
cli/misc.sh,how-to-deploy-automl-endpoint.md
cli/misc.sh,how-to-configure-cli.md
cli/monitoring/advanced-model-monitoring.yaml,how-to-monitor-model-performance.md
cli/monitoring/model-monitoring-with-collected-data.yaml,how-to-monitor-model-performance.md
cli/monitoring/out-of-box-monitoring.yaml,how-to-monitor-model-performance.md
cli/resources/compute/cluster-basic.yml,reference-yaml-compute-aml.md
cli/resources/compute/cluster-location.yml,how-to-create-attach-compute-cluster.md
cli/resources/compute/cluster-location.yml,reference-yaml-compute-aml.md
Expand All @@ -223,8 +226,8 @@ cli/resources/compute/cluster-minimal.yml,reference-yaml-compute-aml.md
cli/resources/compute/cluster-ssh-password.yml,reference-yaml-compute-aml.md
cli/resources/compute/cluster-system-identity.yml,how-to-identity-based-service-authentication.md
cli/resources/compute/cluster-user-identity.yml,how-to-identity-based-service-authentication.md
cli/resources/compute/instance-basic.yml,how-to-create-compute-instance.md
cli/resources/compute/instance-basic.yml,reference-yaml-compute-instance.md
cli/resources/compute/instance-basic.yml,how-to-create-compute-instance.md
cli/resources/compute/instance-minimal.yml,reference-yaml-compute-instance.md
cli/resources/compute/instance-schedule.yml,how-to-create-compute-instance.md
cli/resources/compute/vm-attach.yml,reference-yaml-compute-vm.md
Expand All @@ -242,27 +245,27 @@ cli/resources/datastore/file.yml,reference-yaml-datastore-files.md
cli/resources/registry/registry-storage-options.yml,reference-yaml-registry.md
cli/resources/registry/registry.yml,reference-yaml-registry.md
cli/resources/workspace/basic.yml,reference-yaml-workspace.md
cli/resources/workspace/cmk.yml,reference-yaml-workspace.md
cli/resources/workspace/cmk.yml,how-to-manage-workspace-cli.md
cli/resources/workspace/cmk.yml,reference-yaml-workspace.md
cli/resources/workspace/hbi.yml,reference-yaml-workspace.md
cli/resources/workspace/mvnet-allow-internet-outbound.yml,reference-yaml-workspace.md
cli/resources/workspace/mvnet-allow-only-approved-outbound.yml,reference-yaml-workspace.md
cli/resources/workspace/privatelink.yml,how-to-manage-workspace-cli.md
cli/resources/workspace/privatelink.yml,how-to-configure-private-link.md
cli/resources/workspace/privatelink.yml,reference-yaml-workspace.md
cli/resources/workspace/privatelink.yml,how-to-manage-workspace-cli.md
cli/resources/workspace/with-existing-resources.yml,how-to-manage-workspace-cli.md
cli/resources/workspace/with-existing-resources.yml,reference-yaml-workspace.md
cli/schedules/cron-job-schedule.yml,how-to-schedule-pipeline-job.md
cli/schedules/cron-job-schedule.yml,reference-yaml-schedule.md
cli/schedules/cron-job-schedule.yml,reference-yaml-monitor.md
cli/schedules/cron-with-settings-job-schedule.yml,how-to-schedule-pipeline-job.md
cli/schedules/recurrence-job-schedule.yml,how-to-schedule-pipeline-job.md
cli/schedules/recurrence-job-schedule.yml,reference-yaml-schedule.md
cli/schedules/recurrence-job-schedule.yml,reference-yaml-monitor.md
cli/schedules/schedule.sh,how-to-schedule-data-import.md
cli/schedules/recurrence-job-schedule.yml,reference-yaml-schedule.md
cli/schedules/schedule.sh,how-to-schedule-pipeline-job.md
cli/setup.sh,how-to-configure-cli.md
cli/schedules/schedule.sh,how-to-schedule-data-import.md
cli/setup.sh,how-to-deploy-automl-endpoint.md
cli/setup.sh,how-to-configure-cli.md
cli/train.sh,how-to-train-model.md
deploy-arm-templates-az-cli.sh,how-to-train-model.md
deploy-arm-templates-az-cli.sh,how-to-deploy-online-endpoints.md
Expand Down Expand Up @@ -298,9 +301,9 @@ sdk/python/featurestore_sample/notebooks/sdk_only/4. Enable online store and run
sdk/python/featurestore_sample/notebooks/sdk_only/4. Enable online store and run online inference.ipynb,tutorial-online-materialization-inference.md
sdk/python/featurestore_sample/notebooks/sdk_only/5. Develop a feature set with custom source.ipynb,tutorial-develop-feature-set-with-custom-source.md
sdk/python/jobs/automl-standalone-jobs/automl-classification-task-bankmarketing/automl-classification-task-bankmarketing-serverless.ipynb,how-to-use-serverless-compute.md
sdk/python/jobs/automl-standalone-jobs/automl-image-object-detection-task-fridge-items/automl-image-object-detection-task-fridge-items.ipynb,how-to-auto-train-image-models.md
sdk/python/jobs/automl-standalone-jobs/automl-image-object-detection-task-fridge-items/automl-image-object-detection-task-fridge-items.ipynb,tutorial-auto-train-image-models.md
sdk/python/jobs/automl-standalone-jobs/automl-image-object-detection-task-fridge-items/automl-image-object-detection-task-fridge-items.ipynb,how-to-prepare-datasets-for-automl-images.md
sdk/python/jobs/automl-standalone-jobs/automl-image-object-detection-task-fridge-items/automl-image-object-detection-task-fridge-items.ipynb,how-to-auto-train-image-models.md
sdk/python/jobs/automl-standalone-jobs/automl-image-object-detection-task-fridge-items/automl-image-object-detection-task-fridge-items.ipynb,reference-automl-images-schema.md
sdk/python/jobs/configuration.ipynb,how-to-auto-train-forecast.md
sdk/python/jobs/configuration.ipynb,how-to-train-model.md
Expand Down
Loading

0 comments on commit be63ef9

Please sign in to comment.