[Bug] path issues when running superworkflow pipeline sample for kfp v2

[juancappi]

Search before asking

• I searched the issues and found no similar issues.

Component

KFP workflows

What happened + What you expected to happen

When running super-pipeline example from examples/kfp-pipelines/superworkflows/ray/kfp_v2/ (only after fixing the issues proposed in PR #906), the build fails due to incorrect relative path computation:

FileNotFoundError: [Errno 2] No such file or directory: '../../../../kfp/kfp_ray_components/createRayClusterComponent.yaml'

Reproduction script

make workflow-build
...
Traceback (most recent call last):
  File "[...]/git/ai-foundation/data-prep-kit/examples/kfp-pipelines/superworkflows/ray/kfp_v2/superpipeline_noop_docId_v2_wf.py", line 18, in <module>
    from universal.doc_id.kfp_ray.doc_id_wf import doc_id
  File "[...]/git/ai-foundation/data-prep-kit/transforms/universal/doc_id/kfp_ray/doc_id_wf.py", line 94, in <module>
    create_ray_op = comp.load_component_from_file(component_spec_path + "createRayClusterComponent.yaml")
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[...]/git/ai-foundation/data-prep-kit/transforms/venv/lib/python3.11/site-packages/kfp/components/load_yaml_utilities.py", line 53, in load_component_from_file
    with open(file_path, 'r') as component_stream:
         ^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '../../../../kfp/kfp_ray_components/createRayClusterComponent.yaml'
make[2]: *** [.workflows.compile-pipeline] Error 1
make[1]: *** [superpipeline_noop_docId_v2_wf.yaml] Error 2
make: *** [workflow-build] Error 2

If I update the relative path in doc_id_wf.py to include the extra level ../:

data-prep-kit/transforms/universal/doc_id/kfp_ray/doc_id_wf.py

Line 28 in 6a06d87

component_spec_path = "../../../../kfp/kfp_ray_components/"

The super-pipeline build works, but then the build of kfp pipeline for that individual transform will fail.

Anything else

This depends on fix from #906

OS

MacOS (limited support)

Python

3.11.x

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[juancappi]

@roytman I see you have worked on some of the affected files. If you are not the right person, maybe you can help finding the right person to work with.
CC @touma-I

[revit13]

@juancappi This probably due to one of the refactoring changes that was made in the repo. A possible solution is to add to the examples/kfp-pipelines/superworkflows/ray/kfp_v2/README.md file a comment about the need to change the component_spec_path with the correct path as you mentioned just for the purpose of compiling the super pipelines.