Please let us debug tasks with ipdb

[GuruComposer]

Description

Please add a feature to allow debugging tasks with ipdb.

Context:
If I put this statement in a task that I am testing: import ipdb; ipdb.set_trace(), and then run the following command in my terminal: airflow tasks test <dag_name> <task_id> <time_stamp>, I should be able to pop into the ipdb shell.

However, this seems to have broken with Airflow 2.3. Specifically, this no longer works when I upgraded from Astro Runtime 4 to Astro Runtime 5.

I am no longer seeing the ipdb shell, but instead, getting an error.

Here is the error I've recently started to get after upgrading to Airflow 2.3.

Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/airflow/__main__.py", line 38, in main
    args.func(args)
  File "/usr/local/lib/python3.9/site-packages/airflow/cli/cli_parser.py", line 51, in command
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/airflow/utils/cli.py", line 99, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/airflow/cli/commands/task_command.py", line 547, in task_test
    ti.run(ignore_task_deps=True, ignore_ti_state=True, test_mode=True)
  File "/usr/local/lib/python3.9/site-packages/airflow/utils/session.py", line 71, in wrapper
    return func(*args, session=session, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1795, in run
    self._run_raw_task(
  File "/usr/local/lib/python3.9/site-packages/airflow/utils/session.py", line 68, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1463, in _run_raw_task
    self._execute_task_with_callbacks(context, test_mode)
  File "/usr/local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1610, in _execute_task_with_callbacks
    result = self._execute_task(context, task_orig)
  File "/usr/local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1671, in _execute_task
    result = execute_callable(context=context)
  File "/usr/local/lib/python3.9/site-packages/airflow/operators/python.py", line 171, in execute
    return_value = self.execute_callable()
  File "/usr/local/lib/python3.9/site-packages/airflow/operators/python.py", line 189, in execute_callable
    return self.python_callable(*self.op_args, **self.op_kwargs)
  File "/usr/local/airflow/dags/factory/legacy_tasks.py", line 28, in get_files
    local_files_path, files = report_object.get_files()
  File "/usr/local/airflow/dags/factory/legacy_tasks.py", line 28, in get_files
    local_files_path, files = report_object.get_files()
  File "/usr/local/lib/python3.9/bdb.py", line 88, in trace_dispatch
    return self.dispatch_line(frame)
  File "/usr/local/lib/python3.9/bdb.py", line 112, in dispatch_line
    self.user_line(frame)
  File "/usr/local/lib/python3.9/pdb.py", line 262, in user_line
    self.interaction(frame, None)
  File "/home/astro/.local/lib/python3.9/site-packages/IPython/core/debugger.py", line 336, in interaction
    OldPdb.interaction(self, frame, traceback)
  File "/usr/local/lib/python3.9/pdb.py", line 357, in interaction
    self._cmdloop()
  File "/usr/local/lib/python3.9/pdb.py", line 322, in _cmdloop
    self.cmdloop()
  File "/usr/local/lib/python3.9/cmd.py", line 126, in cmdloop
    line = input(self.prompt)
TypeError: an integer is required (got type NoneType)

If you suspect this is an IPython 8.5.0 bug, please report it at:
    https://github.com/ipython/ipython/issues
or send an email to the mailing list at [email protected]

You can print a more detailed traceback right now with "%tb", or use "%debug"
to interactively debug it.

Extra-detailed tracebacks for bug-reporting purposes can be enabled via:
    %config Application.verbose_crash=True

Use case/motivation

It would be very helpful to be able to debug tasks with an interactive debugger.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[uranusjr]

Could you provide context to this? What did you try, and what prevents it from being successful? Are there any errors? We don’t disallow what you want to do, so it’s impossible to tell what Airflow is supposed to do to satisfy you, based on what you have put down in the issue description.

[GuruComposer]

Hi @uranusjr, okay, I've updated this issue. Is the context sufficient?

[potiuk]

Please avoid creating duplicate issues with the same "reason". As commented in #26349 - this is an environmental problem of yours. Just tested and it works for me. Please follow the advices I gave you on bisecting and searching for similar issues. There is not much we can do about it, without having access to your environment. You are about the only person in the world that can investigate it.

[potiuk]

Also. this is very likely IPython problem (they even wrote about it in the error message). IPython 8.5.0 that you are using has been released 6 days ago, so this is very likely it somehow triggers the error. I'd advice you to start "bisecting" with downgrading iPython and seeing if it works and if it does, then you should definitely raise a bug to iPython team not here (as they actually even advised you to do),

[potiuk]

Also - more than likely downgrading iPython is good workaround for your problem until they solve it too.

[kaxil]

This was fixed by #26806