From 4bffb517978e85d9fcb05c31de82c4218bacafc1 Mon Sep 17 00:00:00 2001 From: Dustin Black Date: Fri, 13 Dec 2024 15:20:00 +0100 Subject: [PATCH 1/8] update and correct some links --- docs/arcaflow/getting-started.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/arcaflow/getting-started.md b/docs/arcaflow/getting-started.md index 7e6835d..c7f742d 100644 --- a/docs/arcaflow/getting-started.md +++ b/docs/arcaflow/getting-started.md @@ -138,6 +138,8 @@ input-->steps.stressng.starting input-->steps.pcp.starting ``` +[Try some other basic example workflows »](https://github.com/arcalot/arcaflow-workflows/tree/main/basic-examples){ .md-button } + [Learn more about running workflows »](/arcaflow/running/){ .md-button } ## Writing Workflows @@ -709,9 +711,9 @@ debug_logs: '' Congratulations, you are now an Arcaflow user! Here are some things you can do next to start working with plugins and workflows: -- [See our repositories of community-supported plugins »](https://github.com/orgs/arcalot/repositories?q=%22arcaflow-plugin-%22) +- [See our catalog of community-supported plugins »](https://github.com/arcalot/arcaflow-plugin-catalog) - [Get our latest plugin container builds from quay.io »](https://quay.io/arcalot) -- [Experiment with more advanced example workflows »](https://github.com/arcalot/arcaflow-workflows/advanced-examples/) +- [Experiment with more advanced example workflows »](https://github.com/arcalot/arcaflow-workflows/tree/main/advanced-examples) ## Keep learning From 78c188e1f7edf9c22d64aeffcce0a9b9ad1f89be Mon Sep 17 00:00:00 2001 From: Dustin Black Date: Fri, 13 Dec 2024 15:49:20 +0100 Subject: [PATCH 2/8] updates, tweaks, and link fixes for getting started guide --- docs/arcaflow/getting-started.md | 55 ++++++++++++++++++-------------- 1 file changed, 31 insertions(+), 24 deletions(-) diff --git a/docs/arcaflow/getting-started.md b/docs/arcaflow/getting-started.md index c7f742d..af5c6ca 100644 --- a/docs/arcaflow/getting-started.md +++ b/docs/arcaflow/getting-started.md @@ -176,15 +176,18 @@ input: ``` Next we will define the steps of the workflow. The steps are to be deployed as container -images, where the `src` field defines the image and tag. The `arcaflow-plugin-utilities` -plugin has multiple steps available, so we indicate with the `step: uuid` field which -step we want to run. The `arcaflow-plugin-example` plugin has only one step, so the -`step` field is not required. The `uuidgen` step requires no input, so we pass an empty -object `{}` to it. The `example` plugin requires an input object of `name` with -`_type` and `nick` fields. We statically set the value of the `_type` field in the -input object of the step, and then we use the -[Arcaflow expression language](/arcaflow/workflows/expressions/) to reference the -workflow input value for `nickname` as the input to the plugin's `nick` field. +images, where the `src` field defines the image and tag. + +For the `uuidgen` step, the `arcaflow-plugin-utilities` plugin has multiple *steps* +available, so we indicate with the `step: uuid` field which step we want to run. The +`uuidgen` step requires no input, so we pass an empty object `{}` to it. + +The `arcaflow-plugin-example` plugin has only one step, so the `step` field is not +required. This plugin requires an input object of `name` with `_type` and `nick` fields. +We statically set the value of the `_type` field in the input object of the step, and +then we use the [Arcaflow expression language](/arcaflow/workflows/expressions/) to +reference the workflow input value for `nickname` as the input to the plugin's `nick` +field. ```yaml title="workflow.yaml (excerpt)" ... @@ -227,7 +230,7 @@ outputs: example: !expr $.steps.example.outputs.success ``` -Our final workflow looks like this: +Our final complete workflow looks like this: ```yaml title="workflow.yaml" version: v0.2.0 @@ -267,13 +270,13 @@ outputs: example: !expr $.steps.example.outputs.success.message ``` -We will create an input file to satisfy the input schema of the workflow: +We will then create an input file to satisfy the input schema of the workflow: ```yaml title="input.yaml" nickname: Arcalot ``` -We will also create a configuration file, setting the container deployer to Podman and +And we will also create a configuration file, setting the container deployer to Podman and the log levels to `error`: ```yaml title="config.yaml" @@ -289,14 +292,15 @@ deployers: And now we can run our new workflow: -!!! tip - The default workflow file is `workflow.yaml` so we don't need to specifiy it here - explicitly. - ```bash arcaflow --config config.yaml --input input.yaml ``` +!!! tip + + The default workflow file is `workflow.yaml` so we don't need to specifiy + `--workflow workflow.yaml` here explicitly. + ```yaml title="example workflow output YAML" output_data: example: Hello, Arcalot! @@ -374,18 +378,18 @@ output_id: success ## Running Plugins + Workflow steps are run via plugins, which are delivered as containers. The Arcalot -community maintains an ever-growing list of -[official plugins](https://github.com/orgs/arcalot/repositories?q=%22arcaflow-plugin-%22), -which are version-controlled and hosted in our -[Quay.io repository](https://quay.io/arcalot). +community maintains an ever-growing list of [official +plugins](https://github.com/arcalot/arcaflow-plugin-catalog), which are +version-controlled and hosted in our [Quay.io repository](https://quay.io/arcalot). Plugins are designed to run independent of an Arcaflow workflow. All plugins have schema definitions for their inputs and outputs, and they perform data validation against those schemas when run. Plugins also have one or more steps, and when there are multiple steps we always need to specify which step we want to run. -!!! tip +!!! tip "Did you know?" Plugin **steps** are the fundamental building blocks for workflows. Let's take a look at the schema for the example plugin. Passing the `--schema` parameter @@ -667,10 +671,13 @@ debug_logs: '' {! https://raw.githubusercontent.com/arcalot/arcaflow-plugin-getting-started-example/main/arcaflow_plugin_getting_started_example/getting_started_example_plugin.py !} ``` -[Learn more about writing Python plugins »](/arcaflow/plugins/python/first.md){ .md-button } +[Learn more about writing Python plugins »](/arcaflow/plugins/python/first){ .md-button } -Next, let's create a `Dockerfile` and build a container image: +Next, let's create a `Dockerfile` and build a container image. This example uses a +container base image that is maintaned by the Arcalot community, which installs +dependencies like python. It copies in our python plugin code and installs its +dependencies, and the container entrypoint is set to the plugin python script. ```Dockerfile title="Dockerfile" {! https://raw.githubusercontent.com/arcalot/arcaflow-plugin-getting-started-example/main/Dockerfile !} @@ -704,7 +711,7 @@ output_data: debug_logs: '' ``` -[Learn more about Packaging plugins](/arcaflow/plugins/packaging.md){ .md-button } +[Learn more about Packaging plugins](/arcaflow/plugins/packaging){ .md-button } ## Next steps From 612f59b2abbddb291804bf6f9f7e3fa0aec602b1 Mon Sep 17 00:00:00 2001 From: Dustin Black Date: Fri, 13 Dec 2024 16:12:28 +0100 Subject: [PATCH 3/8] add info about !ordisabled with wait_for --- docs/arcaflow/workflows/flow-control.md | 45 +++++++++++++++++++++++++ 1 file changed, 45 insertions(+) diff --git a/docs/arcaflow/workflows/flow-control.md b/docs/arcaflow/workflows/flow-control.md index ab9135f..053a4b1 100644 --- a/docs/arcaflow/workflows/flow-control.md +++ b/docs/arcaflow/workflows/flow-control.md @@ -188,6 +188,51 @@ outputs: plugin_output: !ordisabled $.steps.my_step.outputs.success ``` +It is also possible to use `!ordisabled` as part of the `wait_for` flow control. An +explicit relationshp between steps in a workflow in this case becomes a oneof condition +for either the requiested step output or otherwise the disabled state of the step. +```yaml title="workflow.yaml" +version: v0.2.0 +input: + root: RootObject + objects: + RootObject: + id: RootObject + properties: + step_1_enabled: + type: + type_id: bool + step_1_input: + type: + type_id: integer + step_2_enabled: + type: + type_id: bool + step_2_input: + type: + type_id: integer +steps: + my_first_step: + plugin: + deployment_type: image + src: path/to/my_plugin:1 + input: + param_1: !expr $.input.step_1_input + enabled: !expr $.input.step_1_enabled + my_second_step: + plugin: + deployment_type: image + src: path/to/my_plugin:1 + input: + param_1: !expr $.input.step_2_input + enabled: !expr $.input.step_2_enabled + wait_for: !ordisabled $.steps.my_first_step.outputs +outputs: + workflow_success: + first_step_output: !ordisabled $.steps.my_first_step.outputs.success + second_step_output: !ordisabled $.steps.my_second_step.outputs.success +``` + #### Alternative methods For handling disabled steps, `!oneof` and `!ordisabled` are the recommended methods because they cause output failure From c57992ac7dbeb308c235f30b95f55882b5294b82 Mon Sep 17 00:00:00 2001 From: Dustin Black Date: Fri, 13 Dec 2024 16:20:46 +0100 Subject: [PATCH 4/8] restructure --- docs/arcaflow/workflows/flow-control.md | 94 +++++++++++++------------ 1 file changed, 49 insertions(+), 45 deletions(-) diff --git a/docs/arcaflow/workflows/flow-control.md b/docs/arcaflow/workflows/flow-control.md index 053a4b1..66c6843 100644 --- a/docs/arcaflow/workflows/flow-control.md +++ b/docs/arcaflow/workflows/flow-control.md @@ -188,51 +188,6 @@ outputs: plugin_output: !ordisabled $.steps.my_step.outputs.success ``` -It is also possible to use `!ordisabled` as part of the `wait_for` flow control. An -explicit relationshp between steps in a workflow in this case becomes a oneof condition -for either the requiested step output or otherwise the disabled state of the step. -```yaml title="workflow.yaml" -version: v0.2.0 -input: - root: RootObject - objects: - RootObject: - id: RootObject - properties: - step_1_enabled: - type: - type_id: bool - step_1_input: - type: - type_id: integer - step_2_enabled: - type: - type_id: bool - step_2_input: - type: - type_id: integer -steps: - my_first_step: - plugin: - deployment_type: image - src: path/to/my_plugin:1 - input: - param_1: !expr $.input.step_1_input - enabled: !expr $.input.step_1_enabled - my_second_step: - plugin: - deployment_type: image - src: path/to/my_plugin:1 - input: - param_1: !expr $.input.step_2_input - enabled: !expr $.input.step_2_enabled - wait_for: !ordisabled $.steps.my_first_step.outputs -outputs: - workflow_success: - first_step_output: !ordisabled $.steps.my_first_step.outputs.success - second_step_output: !ordisabled $.steps.my_second_step.outputs.success -``` - #### Alternative methods For handling disabled steps, `!oneof` and `!ordisabled` are the recommended methods because they cause output failure @@ -331,6 +286,55 @@ outputs: plugin_output: $.steps.my_step.outputs.success ``` +### Oneof methods with step flow control + +It is also possible to use oneof conditions as part of the `wait_for` flow control. An +explicit relationshp between steps in a workflow in this case becomes a oneof condition +for either the requested step output or otherwise the disabled state of the step. + +An example using `!ordisabled`: +```yaml title="workflow.yaml" +version: v0.2.0 +input: + root: RootObject + objects: + RootObject: + id: RootObject + properties: + step_1_enabled: + type: + type_id: bool + step_1_input: + type: + type_id: integer + step_2_enabled: + type: + type_id: bool + step_2_input: + type: + type_id: integer +steps: + my_first_step: + plugin: + deployment_type: image + src: path/to/my_plugin:1 + input: + param_1: !expr $.input.step_1_input + enabled: !expr $.input.step_1_enabled + my_second_step: + plugin: + deployment_type: image + src: path/to/my_plugin:1 + input: + param_1: !expr $.input.step_2_input + enabled: !expr $.input.step_2_enabled + wait_for: !ordisabled $.steps.my_first_step.outputs +outputs: + workflow_success: + first_step_output: !ordisabled $.steps.my_first_step.outputs.success + second_step_output: !ordisabled $.steps.my_second_step.outputs.success +``` + ## Foreach Loops Foreach loops allow for running a sub-workflow with iterative inputs from a parent workflow. A sub-workflow is a complete Arcaflow workflow file with its own input and output schemas as described in this section. The inputs for the sub-workflow are provided as a list, where each list item is an object that matches the sub-workflow input schema. From 3455389af7ecc2ad7d132c89e9db99cafb7d6de9 Mon Sep 17 00:00:00 2001 From: Dustin Black Date: Fri, 13 Dec 2024 16:45:25 +0100 Subject: [PATCH 5/8] add missing flow control info --- docs/arcaflow/workflows/flow-control.md | 53 +++++++++++++++++++++++++ 1 file changed, 53 insertions(+) diff --git a/docs/arcaflow/workflows/flow-control.md b/docs/arcaflow/workflows/flow-control.md index 66c6843..7022c6b 100644 --- a/docs/arcaflow/workflows/flow-control.md +++ b/docs/arcaflow/workflows/flow-control.md @@ -2,6 +2,59 @@ Flow control allows the workflow author to build a workflow with a decision tree based on supported flow logic. These flow control operations are not implemented by plugins, but are part of the workflow engine itself. +## Implicit Step Relationships + +Any time the input of a step relies on the output of another step via an [Arcaflow +expression](http://127.0.0.1:8000/arcaflow/workflows/expressions/), an implicit step +relationship is established. In this case, the Arcaflow engine holds the execution of the dependent step until the output from the supplier step is available. + +```yaml title="workflow.yaml" +version: v0.2.0 +steps: + step_a: + plugin: + deployment_type: image + src: quay.io/some/container/image + input: + some: + key: !expr $.input.some_value + step_b: + plugin: + deployment_type: image + src: quay.io/some/container/image + input: + some: + key: !expr $.steps.step_a.outputs.success.some_value +``` + +## Explicit Step Relationships + +Sometimes it is important to serialize workflow steps even if they do not have a data +passing relationship. An example may be running a series of benchmarks where you want to +ensure that you get valid results without one step interfereing with another. In this +case, you can use the `wait_for` option of the step to provide an expression or a oneof +condition. + +```yaml title="workflow.yaml" +version: v0.2.0 +steps: + step_a: + plugin: + deployment_type: image + src: quay.io/some/container/image + input: + some: + key: !expr $.input.some_value + step_b: + plugin: + deployment_type: image + src: quay.io/some/container/image + input: + some: + key: !expr $.input.some_other_value + wait_for: !expr $.steps.step_a.outputs +``` + ## Conditional Step Execution Conditional step execution can be achieved with the `enabled` input. From 106f6470733f3274156140ad26c7fc5a04d2794f Mon Sep 17 00:00:00 2001 From: Dustin Black Date: Wed, 8 Jan 2025 12:06:18 +0100 Subject: [PATCH 6/8] updates from PR feedback and link fix --- docs/arcaflow/workflows/flow-control.md | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/docs/arcaflow/workflows/flow-control.md b/docs/arcaflow/workflows/flow-control.md index 7022c6b..e01457c 100644 --- a/docs/arcaflow/workflows/flow-control.md +++ b/docs/arcaflow/workflows/flow-control.md @@ -5,8 +5,9 @@ Flow control allows the workflow author to build a workflow with a decision tree ## Implicit Step Relationships Any time the input of a step relies on the output of another step via an [Arcaflow -expression](http://127.0.0.1:8000/arcaflow/workflows/expressions/), an implicit step -relationship is established. In this case, the Arcaflow engine holds the execution of the dependent step until the output from the supplier step is available. +expression](/arcaflow/workflows/expressions/), an implicit step relationship is +established. In this case, the Arcaflow engine holds the execution of the dependent step +until the output from the supplier step is available. ```yaml title="workflow.yaml" version: v0.2.0 @@ -31,9 +32,9 @@ steps: Sometimes it is important to serialize workflow steps even if they do not have a data passing relationship. An example may be running a series of benchmarks where you want to -ensure that you get valid results without one step interfereing with another. In this -case, you can use the `wait_for` option of the step to provide an expression or a oneof -condition. +ensure that you get valid results without one step interfering with another. In this +case, you can use the `wait_for` option of the step to provide an expression or a [oneof +condition](#conditional-step-execution). ```yaml title="workflow.yaml" version: v0.2.0 @@ -341,9 +342,10 @@ outputs: ### Oneof methods with step flow control -It is also possible to use oneof conditions as part of the `wait_for` flow control. An -explicit relationshp between steps in a workflow in this case becomes a oneof condition -for either the requested step output or otherwise the disabled state of the step. +It is also possible to use [oneof conditions](#conditional-step-execution) as part of +the `wait_for` flow control. An explicit relationship between steps in a workflow in +this case becomes a oneof condition, creating an OR relationship that allows the step to +run when either the requested step output is reached OR the step is disabled. An example using `!ordisabled`: ```yaml title="workflow.yaml" From a02c64e37e9c6a776634838d3e64dd7648d9cc81 Mon Sep 17 00:00:00 2001 From: Dustin Black Date: Wed, 8 Jan 2025 12:19:08 +0100 Subject: [PATCH 7/8] clean up line lengths --- docs/arcaflow/workflows/flow-control.md | 161 ++++++++++++++---------- 1 file changed, 95 insertions(+), 66 deletions(-) diff --git a/docs/arcaflow/workflows/flow-control.md b/docs/arcaflow/workflows/flow-control.md index e01457c..97e5ec5 100644 --- a/docs/arcaflow/workflows/flow-control.md +++ b/docs/arcaflow/workflows/flow-control.md @@ -1,6 +1,8 @@ # Using Flow Control Mechanics -Flow control allows the workflow author to build a workflow with a decision tree based on supported flow logic. These flow control operations are not implemented by plugins, but are part of the workflow engine itself. +Flow control allows the workflow author to build a workflow with a decision tree based +on supported flow logic. These flow control operations are not implemented by plugins, +but are part of the workflow engine itself. ## Implicit Step Relationships @@ -60,9 +62,14 @@ steps: Conditional step execution can be achieved with the `enabled` input. -The value passed into the `enabled` field should be an expression that resolves as a boolean. So it should either reference a path to a boolean value, or it should be a binary comparison. +The value passed into the `enabled` field should be an expression that resolves as a +boolean. So it should either reference a path to a boolean value, or it should be a +binary comparison. -A situation that would benefit from conditional step execution includes a step that uploads the result to a remote server. For development runs you may not want to upload the results, so you can create a `bool` field in the input to toggle the upload, then you can reference that field in the `enabled` field of the upload step. +A situation that would benefit from conditional step execution includes a step that +uploads the result to a remote server. For development runs you may not want to upload +the results, so you can create a `bool` field in the input to toggle the upload, then +you can reference that field in the `enabled` field of the upload step. Example: ```yaml title="workflow.yaml" @@ -92,16 +99,17 @@ outputs: plugin_output: !expr $.steps.my_step.outputs.success ``` -But oh no! The workflow fails with the error `all outputs marked as unresolvable` when the step is disabled. -This happens because the workflow instructed the step not to run, and therefore the output `workflow_success` -cannot get the information it needs. See the next section for methods of handling this situation. +But oh no! The workflow fails with the error `all outputs marked as unresolvable` when +the step is disabled. This happens because the workflow instructed the step not to run, +and therefore the output `workflow_success` cannot get the information it needs. See the +next section for methods of handling this situation. #### Graceful handling of disabled steps -When steps are disabled, they no longer emit their step output. To handle this, the workflow output (or a -following step) must have an OR dependency on both the step's disable output and on another step that has -opposite enable logic. -An OR dependency means that the step will resolve when any of the nodes it depends on resolve, unlike the +When steps are disabled, they no longer emit their step output. To handle this, the +workflow output (or a following step) must have an OR dependency on both the step's +disable output and on another step that has opposite enable logic. An OR dependency +means that the step will resolve when any of the nodes it depends on resolve, unlike the default AND logic that waits for all dependencies to resolve. @@ -140,8 +148,8 @@ stateDiagram-v2 or_disabled --> [*] ``` -The way this works is that either of the two paths can create a valid output, allowing the workflow to continue past -the disabled steps. +The way this works is that either of the two paths can create a valid output, allowing +the workflow to continue past the disabled steps. ##### How to do this in a workflow @@ -154,16 +162,21 @@ There are two tags that create the described OR dependency. ##### How to use `!oneof` -To use `!oneof` for graceful handling of disabled steps, the oneof should depend on ONE OF the success output and -the disabled output, which will output either the success output or the disabled output. +To use `!oneof` for graceful handling of disabled steps, the oneof should depend on ONE +OF the success output and the disabled output, which will output either the success +output or the disabled output. -The oneof tag is a method of creating a schema `one_of_string` type from values present in the workflow. +The oneof tag is a method of creating a schema `one_of_string` type from values present +in the workflow. The syntax of `!oneof` is: -- Following the tag `!oneof`, create a new YAML section (map) by starting an indented new line. That section should contain two properties: - - discriminator: A string that specifies what the oneof discriminator should be. The discriminator specifies which option was emitted. - - options: A YAML section (map) that contains all options. The keys are the discriminator values, and the values should be valid expressions. +- Following the tag `!oneof`, create a new YAML section (map) by starting an indented + new line. That section should contain two properties: + - discriminator: A string that specifies what the oneof discriminator should be. The + discriminator specifies which option was emitted. + - options: A YAML section (map) that contains all options. The keys are the + discriminator values, and the values should be valid expressions. Example: ```yaml title="workflow.yaml" @@ -199,12 +212,12 @@ outputs: ##### How to use `!ordisabled` -Arcaflow provides workflow developers an easy way to handle disabled steps that is as simple as changing the -expression tag from `!expr` to `!ordisabled`. +Arcaflow provides workflow developers an easy way to handle disabled steps that is as +simple as changing the expression tag from `!expr` to `!ordisabled`. -This is a special case of `!oneof` with the discriminator set to `result`, and the options set to `enabled` and -`disabled`. The enabled output is the expression provided, and the disabled output is automatically generated from -the expression provided. +This is a special case of `!oneof` with the discriminator set to `result`, and the +options set to `enabled` and `disabled`. The enabled output is the expression provided, +and the disabled output is automatically generated from the expression provided. The input must be in one of the following formats: - `$.steps.step_name.outputs` @@ -244,13 +257,15 @@ outputs: #### Alternative methods -For handling disabled steps, `!oneof` and `!ordisabled` are the recommended methods because they cause output failure -when the step fails. However, if it is acceptable for the workflow to succeed when a step crashes, the optional tags -could be the right feature for your workflow. +For handling disabled steps, `!oneof` and `!ordisabled` are the recommended methods +because they cause output failure when the step fails. However, if it is acceptable for +the workflow to succeed when a step crashes, the optional tags could be the right +feature for your workflow. -The alterative methods are the "optional" family of expression tags. The `oneof` tags instructed the workflow to -build a `oneof_string` type with OR dependencies, requiring one of the options to have an output for the oneof to -resolve. Meanwhile, the "optional" family of tags can resolve without an output by utilizing optional object property +The alterative methods are the "optional" family of expression tags. The `oneof` tags +instructed the workflow to build a `oneof_string` type with OR dependencies, requiring +one of the options to have an output for the oneof to resolve. Meanwhile, the "optional" +family of tags can resolve without an output by utilizing optional object property fields. | Tag | Description | @@ -287,21 +302,24 @@ outputs: plugin_output: !wait-optional $.steps.my_step.outputs.success ``` -If `my_step` is enabled and has an output, it will be present in the `plugin_output` field of the `workflow_success` -output. Otherwise, the `workflow_success` output will not include the `plugin_output` field. The output will wait -for the plugin to finish, if enabled. +If `my_step` is enabled and has an output, it will be present in the `plugin_output` +field of the `workflow_success` output. Otherwise, the `workflow_success` output will +not include the `plugin_output` field. The output will wait for the plugin to finish, if +enabled. ##### How to use `!soft-optional` !!! warning This feature is not recommended for most use cases. -The `!soft-optional` tag creates the loosest type of dependency. The output will __not__ wait for the step to finish -or fail. In the event that the step referenced with the `!soft-optional` tag does not finish by the time the other -dependencies resolve, or if it crashes or is disabled, the field will be left out of the output. +The `!soft-optional` tag creates the loosest type of dependency. The output will __not__ +wait for the step to finish or fail. In the event that the step referenced with the +`!soft-optional` tag does not finish by the time the other dependencies resolve, or if +it crashes or is disabled, the field will be left out of the output. -The example used in `!wait-optional` would not work with `!soft-optional` because the output would immediately -resolve without the `plugin_output` field. A second dependency is required to ensure the output does not resolve immediately. +The example used in `!wait-optional` would not work with `!soft-optional` because the +output would immediately resolve without the `plugin_output` field. A second dependency +is required to ensure the output does not resolve immediately. ```yaml title="workflow.yaml" version: v0.2.0 @@ -392,12 +410,19 @@ outputs: ## Foreach Loops -Foreach loops allow for running a sub-workflow with iterative inputs from a parent workflow. A sub-workflow is a complete Arcaflow workflow file with its own input and output schemas as described in this section. The inputs for the sub-workflow are provided as a list, where each list item is an object that matches the sub-workflow input schema. +Foreach loops allow for running a sub-workflow with iterative inputs from a parent +workflow. A sub-workflow is a complete Arcaflow workflow file with its own input and +output schemas as described in this section. The inputs for the sub-workflow are +provided as a list, where each list item is an object that matches the sub-workflow +input schema. !!! tip - A complete functional example is available in the [arcaflow-workflows](https://github.com/arcalot/arcaflow-workflows/tree/main/examples/sub-workflow-foreach) repository. + A complete functional example is available in the + [arcaflow-workflows](https://github.com/arcalot/arcaflow-workflows/tree/main/examples/sub-workflow-foreach) + repository. -In the parent workflow file, the author can define an input schema with the list that will contain the input object that will be passed to the sub-workflow. For example: +In the parent workflow file, the author can define an input schema with the list that +will contain the input object that will be passed to the sub-workflow. For example: ```yaml title="workflow.yaml" input: @@ -424,13 +449,17 @@ input: type_id: string ``` -Then in the `steps` section of the workflow, the sub-workflow can be defined as a step with the `loop` list object from above passed to its input. +Then in the `steps` section of the workflow, the sub-workflow can be defined as a step +with the `loop` list object from above passed to its input. The parameters for the sub-workflow step are: - `kind` - The type of loop (currently only foreach is supported) - - `items` - A list of objects to pass to the sub-workflow (the [expression language](expressions.md) allows to pass this from the input schema per the above example) - - `workflow` - The file name for the sub-workflow (this should be in the same directory as the parent workflow) + - `items` - A list of objects to pass to the sub-workflow (the [expression + language](expressions.md) allows to pass this from the input schema per the above + example) + - `workflow` - The file name for the sub-workflow (this should be in the same + directory as the parent workflow) - `parallelism` - The number of sub-workflow loop iterations that will run in parallel ```yaml title="workflow.yaml" @@ -442,7 +471,8 @@ steps: parallelism: 1 ``` -The input yaml file for the above parent workflow would provide the list of objects to loop over as in this example: +The input yaml file for the above parent workflow would provide the list of objects to +loop over as in this example: ```yaml title="input.yaml" loop: @@ -498,11 +528,11 @@ outputs: ### Reduce Repetition with `bindConstants()` -The builtin function [`bindConstants()`](expressions.md#functions) allows you to -avoid repeating input variables for a `foreach` subworkflow. In the example -below, the input variable `name`'s value is repeated across each iteration in -this input. This results in a more repetitive input and schema definition. This -section will show you how to simplify it. +The builtin function [`bindConstants()`](expressions.md#functions) allows you to avoid +repeating input variables for a `foreach` subworkflow. In the example below, the input +variable `name`'s value is repeated across each iteration in this input. This results in +a more repetitive input and schema definition. This section will show you how to +simplify it. #### Workflow and Input Before `bindConstants()` @@ -596,9 +626,9 @@ outputs: #### Reduced Repetition Workflow -Here we restructure the input, factoring out the repeated `name` and `ratio` -entries in the list and placing them into a single field; we will use -`bindConstants()` to construct the `foreach` list with repeated entries. +Here we restructure the input, factoring out the repeated `name` and `ratio` entries in +the list and placing them into a single field; we will use `bindConstants()` to +construct the `foreach` list with repeated entries. ```yaml title="input.yaml" repeated_inputs: @@ -611,16 +641,14 @@ iterations: - loop_id: 4 ``` -To use the generated values from `bindConstants()`, a new schema representing -these bound values must be added to the input schema section of our -`subworkflow.yaml`, `input`. This new schema's ID will be the ID of the schema -that defines the items in your list, in this case `SubRootObject` and the -schema name that defines your repeated inputs, in this case `RepeatedValues`, -for more information see -[Generated Schema Names](schemas.md#generated-combined-schema-names). This -creates our new schema ID, `SubRootObject__RepeatedValues`. You are required -to use this schema ID because it is generated from the names of your other -schemas. +To use the generated values from `bindConstants()`, a new schema representing these +bound values must be added to the input schema section of our `subworkflow.yaml`, +`input`. This new schema's ID will be the ID of the schema that defines the items in +your list, in this case `SubRootObject` and the schema name that defines your repeated +inputs, in this case `RepeatedValues`, for more information see [Generated Schema +Names](schemas.md#generated-combined-schema-names). This creates our new schema ID, +`SubRootObject__RepeatedValues`. You are required to use this schema ID because it is +generated from the names of your other schemas. ```yaml title="workflow.yaml" steps: @@ -631,8 +659,9 @@ steps: parallelism: 1 ``` -To use `bindConstants()` with an `outputSchema` in your workflow, you need to -reference the schema of the list items returned by `bindConstants()`, see -[Generated Schema Name](schemas.md#generated-combined-schema-names). +To use `bindConstants()` with an `outputSchema` in your workflow, you need to reference +the schema of the list items returned by `bindConstants()`, see [Generated Schema +Names](schemas.md#generated-combined-schema-names). -See the [full workflow](https://github.com/arcalot/arcaflow-workflows/blob/492e30ffbea6ce902e6e7ec050c4d1be307b6d73/basic-examples/bind-constants/workflow.yaml#L28). +See the [full +workflow](https://github.com/arcalot/arcaflow-workflows/blob/492e30ffbea6ce902e6e7ec050c4d1be307b6d73/basic-examples/bind-constants/workflow.yaml#L28). From 909f08f047d4b687936e40f8c498cacbb07a9ca3 Mon Sep 17 00:00:00 2001 From: Dustin Black Date: Wed, 8 Jan 2025 12:23:36 +0100 Subject: [PATCH 8/8] update links --- docs/arcaflow/workflows/flow-control.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/arcaflow/workflows/flow-control.md b/docs/arcaflow/workflows/flow-control.md index 97e5ec5..a2ee16b 100644 --- a/docs/arcaflow/workflows/flow-control.md +++ b/docs/arcaflow/workflows/flow-control.md @@ -7,9 +7,9 @@ but are part of the workflow engine itself. ## Implicit Step Relationships Any time the input of a step relies on the output of another step via an [Arcaflow -expression](/arcaflow/workflows/expressions/), an implicit step relationship is -established. In this case, the Arcaflow engine holds the execution of the dependent step -until the output from the supplier step is available. +expression](expressions.md), an implicit step relationship is established. In this case, +the Arcaflow engine holds the execution of the dependent step until the output from the +supplier step is available. ```yaml title="workflow.yaml" version: v0.2.0 @@ -664,4 +664,4 @@ the schema of the list items returned by `bindConstants()`, see [Generated Schem Names](schemas.md#generated-combined-schema-names). See the [full -workflow](https://github.com/arcalot/arcaflow-workflows/blob/492e30ffbea6ce902e6e7ec050c4d1be307b6d73/basic-examples/bind-constants/workflow.yaml#L28). +workflow](https://github.com/arcalot/arcaflow-workflows/blob/main/basic-examples/bind-constants/workflow.yaml).