Skip to content

Commit

Permalink
Address review comment.
Browse files Browse the repository at this point in the history
Signed-off-by: Revital Sur <[email protected]>
  • Loading branch information
revit13 committed Oct 1, 2024
1 parent 400a016 commit d3381ab
Show file tree
Hide file tree
Showing 3 changed files with 5 additions and 8 deletions.
1 change: 1 addition & 0 deletions kfp/doc/simple_transform_pipeline.md
Original file line number Diff line number Diff line change
Expand Up @@ -220,6 +220,7 @@ export KFP_TOLERATIONS='[{"key": "key","operator": "Equal", "value1": "value", "
export KFP_NODE_SELECTOR='{"label_key":"cloud.google.com/gke-accelerator","label_value":"nvidia-tesla-p4"}'

```
In KFP v1, setting `KFP_TOLERATIONS` will apply to the Ray pods, overriding any tolerations specified in the `ray_head_options` and `ray_worker_options` pipeline parameters if they are present.

## Compiling a pipeline <a name = "compilation"></a>

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ def _add_tolerations() -> None:
if tolerations != "":
print(f"Note: Applying Tolerations {tolerations} to kfp and ray pods")

# Add Tolerations as env var so it can used when creating the ray cluster
# Add Tolerations as env var so it can be used when creating the ray cluster
component.add_env_variable(k8s_client.V1EnvVar(name="KFP_TOLERATIONS", value=tolerations))

tolerations = json.loads(tolerations)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -42,13 +42,9 @@ def _add_tolerations() -> None:
try:
tolerations = os.getenv("KFP_TOLERATIONS", "")
if tolerations != "":
print(f"Note: Applying Tolerations {tolerations} to kfp and ray pods")

# Add Tolerations as env var so it can used when creating the ray cluster
task.add_pod_annotation("ray_tolerations", tolerations)
kubernetes.use_field_path_as_env(
task, env_name="KFP_TOLERATIONS", field_path="metadata.annotations['ray_tolerations']"
)
# TODO: apply the tolerations defined as env vars to ray pods.
# Currently they can be specified in the pipeline params:
# ray_head_options and ray_worker_options.

tolerations = json.loads(tolerations)
for toleration in tolerations:
Expand Down

0 comments on commit d3381ab

Please sign in to comment.