Restructure examples for website (#884)

**Pull Request Checklist** - [x] Fixes #883 - [x] Tests ~added~ updated - [x] Documentation/examples added - [x] [Good commit messages](https://cbea.ms/git-commit/) and/or PR title **Description of PR** Currently, all our `.py` examples are dumped into the `examples/workflows` folder, or the `examples/workflows/upstream` folder, making it very difficult to surface features and exemplify usage. This PR enables any kind of recursive directory structure in `/examples/workflows`, which will have the yaml generated by `test_examples.py`. The `generate.py` script has also been updated to copy over the arbitrary structure. The `mkdocs.yml` file is still manipulated by hand, so you should add the examples in the order you want there. Also NB, you need to add an `__init__.py` file for each folder in `examples/workflows` so it can be detected as a module. At the moment, I've added these folders in `examples/workflows`: - artifacts - dags - experimental - loops - scripts - steps - upstream - use-cases And labelled the rest of the examples as ~`Unsorted`~ `Miscellaneous` in the readthedocs site. * Also renamed branch references as argo upstream moved to a `main` branch, which broke the API fetch. --------- Signed-off-by: Elliot Gunton <[email protected]>
argoproj-labs · Dec 5, 2023 · ec7e5ed · ec7e5ed
1 parent 9aad50b
commit ec7e5ed
Show file tree

Hide file tree

Showing 270 changed files with 330 additions and 469 deletions.
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -88,14 +88,14 @@ In order to add a new workflow test to test Hera functionality, do the following
 
 ### Upstream Hera examples
 
-Tests that correspond to any [upstream Argo Workflow examples](https://github.com/argoproj/argo-workflows/tree/master/examples) should live in `examples/workflows/upstream/*.py`. These tests exist to ensure that Hera has complete parity with Argo Workflows and also to catch any regressions that might happen.
+Tests that correspond to any [upstream Argo Workflow examples](https://github.com/argoproj/argo-workflows/tree/main/examples) should live in `examples/workflows/upstream/*.py`. These tests exist to ensure that Hera has complete parity with Argo Workflows and also to catch any regressions that might happen.
 
 In order to add a new workflow test to test Hera functionality, do the following -
 
 * Create a new file under `examples/workflows/upstream` that corresponds with the name of the upstream example yaml
   file. If the yaml file has a hyphen, your python file name should replace those with an underscore. eg. if you are
   trying to replicate
-  [archive-location.yaml](https://github.com/argoproj/argo-workflows/blob/master/examples/archive-location.yaml) your
+  [archive-location.yaml](https://github.com/argoproj/argo-workflows/blob/main/examples/archive-location.yaml) your
   python file should be called `archive_location.py`
 * Define your new workflow. Make sure that the target workflow you wish to export and test against is named `w`
 * Run tests using `make test`. Hera tests will generate a golden copy of the output YAML with the name

diff --git a/Makefile b/Makefile
@@ -109,7 +109,7 @@ init-files:
 	$(MAKE) format
 
 .PHONY: examples
-examples:  ## Generate all the examples
+examples:  ## Generate documentation files for examples
 	@(cd docs && poetry run python generate.py)
 
 .PHONY: regenerate-test-data

diff --git a/SECURITY.md b/SECURITY.md
@@ -22,5 +22,5 @@ of course).
 
 Since Hera Workflows is built for Argo Workflows, this section offers the Argo Workflows security docs:
 
-- see [docs/security.md](https://github.com/argoproj/argo-workflows/blob/master/docs/security.md) for information about
+- see [docs/security.md](https://github.com/argoproj/argo-workflows/blob/main/docs/security.md) for information about
   securing your Argo Workflows instance
diff --git a/docs/examples/about-upstream.md b/docs/examples/about-upstream.md
@@ -0,0 +1,6 @@
+# Upstream Examples
+
+The "Upstream" collection contains examples
+[directly from the Argo Workflows repository](https://github.com/argoproj/argo-workflows/tree/6e97c7d/examples), such as
+the [DAG Diamond example](dag_diamond.md), to demonstrate how the YAML spec maps to Hera classes. These examples are
+generated and compared in Hera's CI/CD pipeline to ensure that all the examples are possible to recreate in Hera.
diff --git a/docs/examples/about-use-cases.md b/docs/examples/about-use-cases.md
@@ -0,0 +1,4 @@
+# Use-Cases and Integrations Examples
+
+These examples show how to use third-party technologies in your Workflows, along with use-cases to show more in-depth
+uses of Hera.
diff --git a/docs/examples/workflows-examples.md b/docs/examples/workflows-examples.md
diff --git a/docs/examples/workflows/artifact.md → .../examples/workflows/artifacts/artifact.md b/docs/examples/workflows/artifact.md → .../examples/workflows/artifacts/artifact.md
diff --git a/...xamples/workflows/artifact_with_fanout.md → ...rkflows/artifacts/artifact_with_fanout.md b/...xamples/workflows/artifact_with_fanout.md → ...rkflows/artifacts/artifact_with_fanout.md
diff --git a/docs/examples/workflows/callable_coinflip.md b/docs/examples/workflows/callable_coinflip.md
diff --git a/docs/examples/workflows/complex_expr.md b/docs/examples/workflows/complex_expr.md
@@ -4,7 +4,7 @@
 
 This example showcases how to string together complex expressions.
 This example is a python replica of
-https://github.com/argoproj/argo-workflows/blob/master/examples/expression-reusing-verbose-snippets.yaml
+https://github.com/argoproj/argo-workflows/blob/main/examples/expression-reusing-verbose-snippets.yaml
 
 
 === "Hera"

diff --git a/...xamples/workflows/any_success_all_fail.md → ...es/workflows/dags/any_success_all_fail.md b/...xamples/workflows/any_success_all_fail.md → ...es/workflows/dags/any_success_all_fail.md
diff --git a/docs/examples/workflows/callable_dag.md → docs/examples/workflows/dags/callable_dag.md b/docs/examples/workflows/callable_dag.md → docs/examples/workflows/dags/callable_dag.md
diff --git a/.../workflows/callable_dag_with_param_get.md → ...flows/dags/callable_dag_with_param_get.md b/.../workflows/callable_dag_with_param_get.md → ...flows/dags/callable_dag_with_param_get.md
@@ -76,7 +76,7 @@
           - -m
           - hera.workflows.runner
           - -e
-          - examples.workflows.callable_dag_with_param_get:hello_with_output
+          - examples.workflows.dags.callable_dag_with_param_get:hello_with_output
           command:
           - python
           image: python:3.8

diff --git a/docs/examples/workflows/complex_deps.md → docs/examples/workflows/dags/complex_deps.md b/docs/examples/workflows/complex_deps.md → docs/examples/workflows/dags/complex_deps.md
diff --git a/docs/examples/workflows/conditional.md → docs/examples/workflows/dags/conditional.md b/docs/examples/workflows/conditional.md → docs/examples/workflows/dags/conditional.md
diff --git a/...rkflows/dag_conditional_on_task_status.md → ...ws/dags/dag_conditional_on_task_status.md b/...rkflows/dag_conditional_on_task_status.md → ...ws/dags/dag_conditional_on_task_status.md
diff --git a/...s/workflows/dag_conditional_parameters.md → ...kflows/dags/dag_conditional_parameters.md b/...s/workflows/dag_conditional_parameters.md → ...kflows/dags/dag_conditional_parameters.md
diff --git a/...ws/dag_diamond_with_callable_container.md → ...gs/dag_diamond_with_callable_container.md b/...ws/dag_diamond_with_callable_container.md → ...gs/dag_diamond_with_callable_container.md
diff --git a/...s/dag_diamond_with_callable_decorators.md → ...s/dag_diamond_with_callable_decorators.md b/...s/dag_diamond_with_callable_decorators.md → ...s/dag_diamond_with_callable_decorators.md
diff --git a/...flows/dag_diamond_with_callable_script.md → .../dags/dag_diamond_with_callable_script.md b/...flows/dag_diamond_with_callable_script.md → .../dags/dag_diamond_with_callable_script.md
diff --git a/...mples/workflows/dag_with_param_passing.md → .../workflows/dags/dag_with_param_passing.md b/...mples/workflows/dag_with_param_passing.md → .../workflows/dags/dag_with_param_passing.md
diff --git a/...s/dag_with_script_output_param_passing.md → ...s/dag_with_script_output_param_passing.md b/...s/dag_with_script_output_param_passing.md → ...s/dag_with_script_output_param_passing.md
diff --git a/...orkflows/dag_with_script_param_passing.md → ...ows/dags/dag_with_script_param_passing.md b/...orkflows/dag_with_script_param_passing.md → ...ows/dags/dag_with_script_param_passing.md
diff --git a/docs/examples/workflows/on_exit.md → docs/examples/workflows/dags/on_exit.md b/docs/examples/workflows/on_exit.md → docs/examples/workflows/dags/on_exit.md
diff --git a/...ipt_annotations_artifact_custom_volume.md → ...ipt_annotations_artifact_custom_volume.md b/...ipt_annotations_artifact_custom_volume.md → ...ipt_annotations_artifact_custom_volume.md
@@ -140,7 +140,7 @@ This example will reuse the outputs volume across script steps.
           - -m
           - hera.workflows.runner
           - -e
-          - examples.workflows.script_annotations_artifact_custom_volume:output_artifact_empty_dir
+          - examples.workflows.experimental.script_annotations_artifact_custom_volume:output_artifact_empty_dir
           command:
           - python
           env:
@@ -166,7 +166,7 @@ This example will reuse the outputs volume across script steps.
           - -m
           - hera.workflows.runner
           - -e
-          - examples.workflows.script_annotations_artifact_custom_volume:use_artifact
+          - examples.workflows.experimental.script_annotations_artifact_custom_volume:use_artifact
           command:
           - python
           env:
@@ -187,7 +187,7 @@ This example will reuse the outputs volume across script steps.
           - -m
           - hera.workflows.runner
           - -e
-          - examples.workflows.script_annotations_artifact_custom_volume:output_artifact_existing_vol
+          - examples.workflows.experimental.script_annotations_artifact_custom_volume:output_artifact_existing_vol
           command:
           - python
           env:
@@ -210,7 +210,7 @@ This example will reuse the outputs volume across script steps.
           - -m
           - hera.workflows.runner
           - -e
-          - examples.workflows.script_annotations_artifact_custom_volume:use_artifact_existing_vol
+          - examples.workflows.experimental.script_annotations_artifact_custom_volume:use_artifact_existing_vol
           command:
           - python
           env:

diff --git a/...ws/script_annotations_artifact_loaders.md → ...al/script_annotations_artifact_loaders.md b/...ws/script_annotations_artifact_loaders.md → ...al/script_annotations_artifact_loaders.md
@@ -96,7 +96,7 @@
           - -m
           - hera.workflows.runner
           - -e
-          - examples.workflows.script_annotations_artifact_loaders:output_dict_artifact
+          - examples.workflows.experimental.script_annotations_artifact_loaders:output_dict_artifact
           command:
           - python
           env:
@@ -120,7 +120,7 @@
           - -m
           - hera.workflows.runner
           - -e
-          - examples.workflows.script_annotations_artifact_loaders:artifact_loaders
+          - examples.workflows.experimental.script_annotations_artifact_loaders:artifact_loaders
           command:
           - python
           env:

diff --git a/..._annotations_artifact_outputs_defaults.md → ..._annotations_artifact_outputs_defaults.md b/..._annotations_artifact_outputs_defaults.md → ..._annotations_artifact_outputs_defaults.md
@@ -97,7 +97,7 @@ This example will reuse the outputs volume across script steps.
           - -m
           - hera.workflows.runner
           - -e
-          - examples.workflows.script_annotations_artifact_outputs_defaults:output_artifact
+          - examples.workflows.experimental.script_annotations_artifact_outputs_defaults:output_artifact
           command:
           - python
           env:
@@ -117,7 +117,7 @@ This example will reuse the outputs volume across script steps.
           - -m
           - hera.workflows.runner
           - -e
-          - examples.workflows.script_annotations_artifact_outputs_defaults:use_artifact
+          - examples.workflows.experimental.script_annotations_artifact_outputs_defaults:use_artifact
           command:
           - python
           env:

diff --git a/...ws/script_annotations_artifact_passing.md → ...al/script_annotations_artifact_passing.md b/...ws/script_annotations_artifact_passing.md → ...al/script_annotations_artifact_passing.md
@@ -93,7 +93,7 @@ This example will reuse the outputs volume across script steps.
           - -m
           - hera.workflows.runner
           - -e
-          - examples.workflows.script_annotations_artifact_passing:output_artifact
+          - examples.workflows.experimental.script_annotations_artifact_passing:output_artifact
           command:
           - python
           env:
@@ -113,7 +113,7 @@ This example will reuse the outputs volume across script steps.
           - -m
           - hera.workflows.runner
           - -e
-          - examples.workflows.script_annotations_artifact_passing:use_artifact
+          - examples.workflows.experimental.script_annotations_artifact_passing:use_artifact
           command:
           - python
           env:

diff --git a/...es/workflows/script_annotations_inputs.md → ...experimental/script_annotations_inputs.md b/...es/workflows/script_annotations_inputs.md → ...experimental/script_annotations_inputs.md
@@ -105,7 +105,7 @@
           - -m
           - hera.workflows.runner
           - -e
-          - examples.workflows.script_annotations_inputs:output_dict_artifact
+          - examples.workflows.experimental.script_annotations_inputs:output_dict_artifact
           command:
           - python
           env:
@@ -137,7 +137,7 @@
           - -m
           - hera.workflows.runner
           - -e
-          - examples.workflows.script_annotations_inputs:echo_all
+          - examples.workflows.experimental.script_annotations_inputs:echo_all
           command:
           - python
           env:

diff --git a/...s/workflows/script_annotations_outputs.md → ...xperimental/script_annotations_outputs.md b/...s/workflows/script_annotations_outputs.md → ...xperimental/script_annotations_outputs.md
@@ -80,7 +80,7 @@
           - -m
           - hera.workflows.runner
           - -e
-          - examples.workflows.script_annotations_outputs:script_param_artifact_in_function_signature_and_return_type
+          - examples.workflows.experimental.script_annotations_outputs:script_param_artifact_in_function_signature_and_return_type
           command:
           - python
           env:

diff --git a/docs/examples/workflows/dynamic_fanout.md → ...xamples/workflows/loops/dynamic_fanout.md b/docs/examples/workflows/dynamic_fanout.md → ...xamples/workflows/loops/dynamic_fanout.md
diff --git a/...les/workflows/dynamic_fanout_container.md → ...rkflows/loops/dynamic_fanout_container.md b/...les/workflows/dynamic_fanout_container.md → ...rkflows/loops/dynamic_fanout_container.md
diff --git a/.../workflows/dynamic_fanout_extra_kwargs.md → ...lows/loops/dynamic_fanout_extra_kwargs.md b/.../workflows/dynamic_fanout_extra_kwargs.md → ...lows/loops/dynamic_fanout_extra_kwargs.md
diff --git a/...xamples/workflows/dynamic_fanout_fanin.md → ...s/workflows/loops/dynamic_fanout_fanin.md b/...xamples/workflows/dynamic_fanout_fanin.md → ...s/workflows/loops/dynamic_fanout_fanin.md
diff --git a/.../workflows/dynamic_fanout_json_payload.md → ...lows/loops/dynamic_fanout_json_payload.md b/.../workflows/dynamic_fanout_json_payload.md → ...lows/loops/dynamic_fanout_json_payload.md
diff --git a/docs/examples/workflows/script_loops_maps.md → ...ples/workflows/loops/script_loops_maps.md b/docs/examples/workflows/script_loops_maps.md → ...ples/workflows/loops/script_loops_maps.md
diff --git a/docs/examples/workflows/with_sequence.md → ...examples/workflows/loops/with_sequence.md b/docs/examples/workflows/with_sequence.md → ...examples/workflows/loops/with_sequence.md
diff --git a/docs/examples/workflows/callable_script.md → ...ples/workflows/scripts/callable_script.md b/docs/examples/workflows/callable_script.md → ...ples/workflows/scripts/callable_script.md
@@ -157,7 +157,7 @@
           - -m
           - hera.workflows.runner
           - -e
-          - examples.workflows.callable_script:my_function
+          - examples.workflows.scripts.callable_script:my_function
           command:
           - python
           env:
@@ -174,7 +174,7 @@
           - -m
           - hera.workflows.runner
           - -e
-          - examples.workflows.callable_script:str_function
+          - examples.workflows.scripts.callable_script:str_function
           command:
           - python
           env:
@@ -191,7 +191,7 @@
           - -m
           - hera.workflows.runner
           - -e
-          - examples.workflows.callable_script:another_function
+          - examples.workflows.scripts.callable_script:another_function
           command:
           - python
           env:
@@ -213,7 +213,7 @@
           - -m
           - hera.workflows.runner
           - -e
-          - examples.workflows.callable_script:function_kebab
+          - examples.workflows.scripts.callable_script:function_kebab
           command:
           - python
           env:
@@ -230,7 +230,7 @@
           - -m
           - hera.workflows.runner
           - -e
-          - examples.workflows.callable_script:function_kebab_object
+          - examples.workflows.scripts.callable_script:function_kebab_object
           command:
           - python
           env:

diff --git a/docs/examples/workflows/coinflip.md → docs/examples/workflows/scripts/coinflip.md b/docs/examples/workflows/coinflip.md → docs/examples/workflows/scripts/coinflip.md
diff --git a/...ples/workflows/default_param_overwrite.md → ...kflows/scripts/default_param_overwrite.md b/...ples/workflows/default_param_overwrite.md → ...kflows/scripts/default_param_overwrite.md
diff --git a/...ples/workflows/script_artifact_passing.md → ...kflows/scripts/script_artifact_passing.md b/...ples/workflows/script_artifact_passing.md → ...kflows/scripts/script_artifact_passing.md
diff --git a/docs/examples/workflows/script_auto_infer.md → ...es/workflows/scripts/script_auto_infer.md b/docs/examples/workflows/script_auto_infer.md → ...es/workflows/scripts/script_auto_infer.md
diff --git a/docs/examples/workflows/script_variations.md → ...es/workflows/scripts/script_variations.md b/docs/examples/workflows/script_variations.md → ...es/workflows/scripts/script_variations.md
diff --git a/...s/workflows/script_with_default_params.md → ...ows/scripts/script_with_default_params.md b/...s/workflows/script_with_default_params.md → ...ows/scripts/script_with_default_params.md
diff --git a/...orkflows/script_with_image_pull_policy.md → .../scripts/script_with_image_pull_policy.md b/...orkflows/script_with_image_pull_policy.md → .../scripts/script_with_image_pull_policy.md
diff --git a/...amples/workflows/script_with_resources.md → ...orkflows/scripts/script_with_resources.md b/...amples/workflows/script_with_resources.md → ...orkflows/scripts/script_with_resources.md
diff --git a/docs/examples/workflows/callable_steps.md → ...xamples/workflows/steps/callable_steps.md b/docs/examples/workflows/callable_steps.md → ...xamples/workflows/steps/callable_steps.md
diff --git a/docs/examples/workflows/steps_types.md → docs/examples/workflows/steps/steps_types.md b/docs/examples/workflows/steps_types.md → docs/examples/workflows/steps/steps_types.md
diff --git a/...orkflows/steps_with_callable_container.md → ...ws/steps/steps_with_callable_container.md b/...orkflows/steps_with_callable_container.md → ...ws/steps/steps_with_callable_container.md
diff --git a/docs/examples/workflows/upstream/archive_location.md b/docs/examples/workflows/upstream/archive_location.md
@@ -3,7 +3,7 @@
 ## Note
 
 This example is a replication of an Argo Workflow example in Hera.
-The upstream example can be [found here](https://github.com/argoproj/argo-workflows/blob/master/examples/archive-location.yaml).
+The upstream example can be [found here](https://github.com/argoproj/argo-workflows/blob/main/examples/archive-location.yaml).
 
 
 

diff --git a/docs/examples/workflows/upstream/arguments_artifacts.md b/docs/examples/workflows/upstream/arguments_artifacts.md
@@ -3,7 +3,7 @@
 ## Note
 
 This example is a replication of an Argo Workflow example in Hera.
-The upstream example can be [found here](https://github.com/argoproj/argo-workflows/blob/master/examples/arguments-artifacts.yaml).
+The upstream example can be [found here](https://github.com/argoproj/argo-workflows/blob/main/examples/arguments-artifacts.yaml).
 
 
 

diff --git a/docs/examples/workflows/upstream/arguments_parameters.md b/docs/examples/workflows/upstream/arguments_parameters.md
@@ -3,7 +3,7 @@
 ## Note
 
 This example is a replication of an Argo Workflow example in Hera.
-The upstream example can be [found here](https://github.com/argoproj/argo-workflows/blob/master/examples/arguments-parameters.yaml).
+The upstream example can be [found here](https://github.com/argoproj/argo-workflows/blob/main/examples/arguments-parameters.yaml).
 
 
 

diff --git a/docs/examples/workflows/upstream/arguments_parameters_from_configmap.md b/docs/examples/workflows/upstream/arguments_parameters_from_configmap.md
@@ -3,7 +3,7 @@
 ## Note
 
 This example is a replication of an Argo Workflow example in Hera.
-The upstream example can be [found here](https://github.com/argoproj/argo-workflows/blob/master/examples/arguments-parameters-from-configmap.yaml).
+The upstream example can be [found here](https://github.com/argoproj/argo-workflows/blob/main/examples/arguments-parameters-from-configmap.yaml).
 
 
 

diff --git a/docs/examples/workflows/upstream/artifact_disable_archive.md b/docs/examples/workflows/upstream/artifact_disable_archive.md
@@ -3,7 +3,7 @@
 ## Note
 
 This example is a replication of an Argo Workflow example in Hera.
-The upstream example can be [found here](https://github.com/argoproj/argo-workflows/blob/master/examples/artifact-disable-archive.yaml).
+The upstream example can be [found here](https://github.com/argoproj/argo-workflows/blob/main/examples/artifact-disable-archive.yaml).
 
 
 

diff --git a/docs/examples/workflows/upstream/artifact_gc_workflow.md b/docs/examples/workflows/upstream/artifact_gc_workflow.md
@@ -3,7 +3,7 @@
 ## Note
 
 This example is a replication of an Argo Workflow example in Hera.
-The upstream example can be [found here](https://github.com/argoproj/argo-workflows/blob/master/examples/artifact-gc-workflow.yaml).
+The upstream example can be [found here](https://github.com/argoproj/argo-workflows/blob/main/examples/artifact-gc-workflow.yaml).
 
 
 

diff --git a/docs/examples/workflows/upstream/artifact_passing.md b/docs/examples/workflows/upstream/artifact_passing.md
@@ -3,7 +3,7 @@
 ## Note
 
 This example is a replication of an Argo Workflow example in Hera.
-The upstream example can be [found here](https://github.com/argoproj/argo-workflows/blob/master/examples/artifact-passing.yaml).
+The upstream example can be [found here](https://github.com/argoproj/argo-workflows/blob/main/examples/artifact-passing.yaml).
-Original file line number
+Diff line change
@@ Expand Up / @@ -3,7 +3,7 @@ @@
     ## Note
     This example is a replication of an Argo Workflow example in Hera.
-    The upstream example can be [found here](https://github.com/argoproj/argo-workflows/blob/master/examples/archive-location.yaml).
+    The upstream example can be [found here](https://github.com/argoproj/argo-workflows/blob/main/examples/archive-location.yaml).
@@ Expand Down @@