Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AIP-81 Central API Communication Mechanism for CLI and Connection Command Integration #45300

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

bugraoz93
Copy link
Collaborator

@bugraoz93 bugraoz93 commented Dec 30, 2024

closes: #42561 and discarding dynamic generation of requests: #42562

Summary
This PR introduces a central API communication mechanism for the CLI and adopts one of the broadest endpoint ranges in the API, focusing on connection commands. After several iterations to automate API call generation and handle edge cases, I couldn't achieve a fully dynamic request generation process. Instead, I landed on a structure similar to TaskSDK <-> ExecutionAPI. This approach simplifies reviews, maintains consistency across clients, and opens the door for better automation in future iterations, possibly in 3.1 or 3.2.

Implementation Details

  • Added a central API communication mechanism for the CLI.
  • Integrated connection_command.py with unit tests.
  • Implemented most of the necessary CLI operations.
  • Data model generation from RestAPI to CLI is handled using uv.
  • Added decorators to cascade generic processes to operations.
  • Created a testing framework for CLI testing, with an example implementation for connection command.
  • Pending Work
  • Some minor features related to the connection command are still in progress and will be added in follow-up updates before 3.0.

Note: There are two missing functionalities in this PR for the connection command. Those will be included with follow-up tasks.

Notes on task_command
While working on this, I realized that task_command behaves more like a local_command than a remote_command. This needs further discussion, especially regarding its integration with TaskSDK. From the CLI perspective, depending on TaskSDK seems cleaner than directly calling ExecutionAPI, it avoids duplication and keeps the structure consistent. Even if the CLI calls TaskSDK, the two-hop process to reach the ExecutionAPI feels a bit redundant. What do you think, @ashb (to all, please jump into discussion, I tagged Ash to follow up the previous discussion started on this in Slack)?


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@boring-cyborg boring-cyborg bot added area:CLI area:UI Related to UI/UX. For Frontend Developers. labels Dec 30, 2024
@bugraoz93 bugraoz93 added the area:API Airflow's REST/HTTP API label Dec 30, 2024
@bugraoz93 bugraoz93 requested review from ashb, potiuk and kaxil December 30, 2024 21:06
@jscheffl
Copy link
Contributor

jscheffl commented Jan 1, 2025

Note: As PR #45312 has been merged, the code formatting rules have changed for new UI. Please rebase and re-run pre-commit checks to ensure that formatting in folder airflow/ui is adjusted.

@bugraoz93 bugraoz93 force-pushed the feat/42561/cli-central-api-communication branch from 8446931 to 776cf69 Compare January 2, 2025 17:31
@bugraoz93 bugraoz93 changed the title Central API Communication Mechanism for CLI and Connection Command Integration AIP-81 Central API Communication Mechanism for CLI and Connection Command Integration Jan 2, 2025
@bugraoz93 bugraoz93 force-pushed the feat/42561/cli-central-api-communication branch from 776cf69 to fde9b2a Compare January 2, 2025 18:38
… most remote commands, connection_command.py integrated with client, data model generation from ResAPI to CLI is added using uv similar to TaskSDK
@bugraoz93 bugraoz93 force-pushed the feat/42561/cli-central-api-communication branch from fde9b2a to 449e9ab Compare January 3, 2025 17:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:API Airflow's REST/HTTP API area:CLI area:UI Related to UI/UX. For Frontend Developers.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

AIP-81 Central API Communication Mechanism for CLI
3 participants