Import error gets buried by pipeline ValueError

[merelcht]

Description

When I have a wrong import in my pipelines.py file I see an error message about the pipeline not being found and the registry. When digging deeper I realised the import error is there all the way at the top, but much less visible.

Context

Steps to Reproduce

1. Create a Kedro project with pipeline (e.g. spaceflights)
2. In the pipeline.py file add a random import, e.g. import bla
3. Run kedro run

Expected Result

I would see clearly that there's something wrong with my imports and see the error:

ImportError: cannot import name 'bla' from 'wcc_workshop.pipelines.data_processing.nodes'    

Actual Result

I see:

ValueError: Failed to find the pipeline named 'data_processing'. It needs to be generated and returned by the 'register_pipelines' function.

Full stacktrace:

gitpod /workspace/wcc-workshop (main) $ kedro run --pipeline=data_processing
[01/24/25 17:04:49] INFO     Using '/workspace/.pyenv_mirror/user/current/lib/python3.12/site-packages/kedro/framework/project/rich_logging.yml' as logging configuration.                     __init__.py:270
[01/24/25 17:04:49] INFO     Kedro project wcc-workshop                                                                                                                                         session.py:329
[01/24/25 17:04:51] WARNING  /workspace/.pyenv_mirror/user/current/lib/python3.12/site-packages/kedro/framework/project/__init__.py:450: UserWarning: An error occurred while importing the    warnings.py:112
                             'wcc_workshop.pipelines.data_processing' module. Nothing defined therein will be returned by 'find_pipelines'.                                                                   
                                                                                                                                                                                                              
                             Traceback (most recent call last):                                                                                                                                               
                               File "/workspace/.pyenv_mirror/user/current/lib/python3.12/site-packages/kedro/framework/project/__init__.py", line 442, in find_pipelines                                     
                                 pipeline_module = importlib.import_module(pipeline_module_name)                                                                                                              
                                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                              
                               File "/home/gitpod/.pyenv/versions/3.12.7/lib/python3.12/importlib/__init__.py", line 90, in import_module                                                                     
                                 return _bootstrap._gcd_import(name[level:], package, level)                                                                                                                  
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                  
                               File "<frozen importlib._bootstrap>", line 1387, in _gcd_import                                                                                                                
                               File "<frozen importlib._bootstrap>", line 1360, in _find_and_load                                                                                                             
                               File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked                                                                                                    
                               File "<frozen importlib._bootstrap>", line 935, in _load_unlocked                                                                                                              
                               File "<frozen importlib._bootstrap_external>", line 995, in exec_module                                                                                                        
                               File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed                                                                                                   
                               File "/workspace/wcc-workshop/src/wcc_workshop/pipelines/data_processing/__init__.py", line 6, in <module>                                                                     
                                 from .pipeline import create_pipeline                                                                                                                                        
                               File "/workspace/wcc-workshop/src/wcc_workshop/pipelines/data_processing/pipeline.py", line 7, in <module>                                                                     
                                 from .nodes import bla, preprocess_companies, preprocess_shuttles, create_model_table                                                                                        
                             ImportError: cannot import name 'bla' from 'wcc_workshop.pipelines.data_processing.nodes'                                                                                        
                             (/workspace/wcc-workshop/src/wcc_workshop/pipelines/data_processing/nodes.py)                                                                                                    
                                                                                                                                                                                                              
                               warnings.warn(                                                                                                                                                                 
                                                                                                                                                                                                              
                    INFO     Kedro is sending anonymous usage data with the sole purpose of improving the product. No personal data or IP addresses are stored on our side. If you want to opt   plugin.py:233
                             out, set the `KEDRO_DISABLE_TELEMETRY` or `DO_NOT_TRACK` environment variables, or create a `.telemetry` file in the current working directory with the contents                 
                             `consent: false`. Read more at https://docs.kedro.org/en/stable/configuration/telemetry.html                                                                                     
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /workspace/.pyenv_mirror/user/current/lib/python3.12/site-packages/kedro/framework/session/sessi │
│ on.py:346 in run                                                                                 │
│                                                                                                  │
│   343 │   │   name = pipeline_name or "__default__"                                              │
│   344 │   │                                                                                      │
│   345 │   │   try:                                                                               │
│ ❱ 346 │   │   │   pipeline = pipelines[name]                                                     │
│   347 │   │   except KeyError as exc:                                                            │
│   348 │   │   │   raise ValueError(                                                              │
│   349 │   │   │   │   f"Failed to find the pipeline named '{name}'. "                            │
│                                                                                                  │
│ /workspace/.pyenv_mirror/user/current/lib/python3.12/site-packages/kedro/framework/project/__ini │
│ t__.py:167 in inner                                                                              │
│                                                                                                  │
│   164 │                                                                                          │
│   165 │   def inner(self: Any, *args: Any, **kwargs: Any) -> Any:                                │
│   166 │   │   self._load_data()                                                                  │
│ ❱ 167 │   │   return func(self._content, *args, **kwargs)                                        │
│   168 │                                                                                          │
│   169 │   return inner                                                                           │
│   170                                                                                            │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
KeyError: 'data_processing'

The above exception was the direct cause of the following exception:

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /workspace/.pyenv_mirror/user/3.12.7/bin/kedro:8 in <module>                                     │
│                                                                                                  │
│   5 from kedro.framework.cli import main                                                         │
│   6 if __name__ == '__main__':                                                                   │
│   7 │   sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])                         │
│ ❱ 8 │   sys.exit(main())                                                                         │
│   9                                                                                              │
│                                                                                                  │
│ /workspace/.pyenv_mirror/user/current/lib/python3.12/site-packages/kedro/framework/cli/cli.py:26 │
│ 3 in main                                                                                        │
│                                                                                                  │
│   260 │   cli_collection = KedroCLI(                                                             │
│   261 │   │   project_path=_find_kedro_project(Path.cwd()) or Path.cwd()                         │
│   262 │   )                                                                                      │
│ ❱ 263 │   cli_collection()                                                                       │
│   264                                                                                            │
│                                                                                                  │
│ /workspace/.pyenv_mirror/user/current/lib/python3.12/site-packages/click/core.py:1161 in         │
│ __call__                                                                                         │
│                                                                                                  │
│ /workspace/.pyenv_mirror/user/current/lib/python3.12/site-packages/kedro/framework/cli/cli.py:16 │
│ 3 in main                                                                                        │
│                                                                                                  │
│   160 │   │   )                                                                                  │
│   161 │   │                                                                                      │
│   162 │   │   try:                                                                               │
│ ❱ 163 │   │   │   super().main(                                                                  │
│   164 │   │   │   │   args=args,                                                                 │
│   165 │   │   │   │   prog_name=prog_name,                                                       │
│   166 │   │   │   │   complete_var=complete_var,                                                 │
│                                                                                                  │
│ /workspace/.pyenv_mirror/user/current/lib/python3.12/site-packages/click/core.py:1082 in main    │
│                                                                                                  │
│ /workspace/.pyenv_mirror/user/current/lib/python3.12/site-packages/click/core.py:1697 in invoke  │
│                                                                                                  │
│ /workspace/.pyenv_mirror/user/current/lib/python3.12/site-packages/click/core.py:1443 in invoke  │
│                                                                                                  │
│ /workspace/.pyenv_mirror/user/current/lib/python3.12/site-packages/click/core.py:788 in invoke   │
│                                                                                                  │
│ /workspace/.pyenv_mirror/user/current/lib/python3.12/site-packages/kedro/framework/cli/project.p │
│ y:228 in run                                                                                     │
│                                                                                                  │
│   225 │   with KedroSession.create(                                                              │
│   226 │   │   env=env, conf_source=conf_source, extra_params=params                              │
│   227 │   ) as session:                                                                          │
│ ❱ 228 │   │   return session.run(                                                                │
│   229 │   │   │   tags=tuple_tags,                                                               │
│   230 │   │   │   runner=runner_obj(is_async=is_async),                                          │
│   231 │   │   │   node_names=tuple_node_names,                                                   │
│                                                                                                  │
│ /workspace/.pyenv_mirror/user/current/lib/python3.12/site-packages/kedro/framework/session/sessi │
│ on.py:348 in run                                                                                 │
│                                                                                                  │
│   345 │   │   try:                                                                               │
│   346 │   │   │   pipeline = pipelines[name]                                                     │
│   347 │   │   except KeyError as exc:                                                            │
│ ❱ 348 │   │   │   raise ValueError(                                                              │
│   349 │   │   │   │   f"Failed to find the pipeline named '{name}'. "                            │
│   350 │   │   │   │   f"It needs to be generated and returned "                                  │
│   351 │   │   │   │   f"by the 'register_pipelines' function."                                   │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: Failed to find the pipeline named 'data_processing'. It needs to be generated and returned by the 'register_pipelines' function.

Your Environment

• Kedro version used (pip show kedro or kedro -V):
• Python version used (python -V):
• Operating system and version:

[noklam]

I think it's the same issue that we always have with find_pipelines, which now has a raise_errors=False default and can be turned on for raising error explicitly.

[merelcht]

@noklam you are right. I also realised the above scenario only happens if your run command only tries to execute the faulty pipeline. So if there's another pipeline it can wrong you won't get the register_pipelines error. I'm closing this.