Skip to content

Commit

Permalink
docs: add public documentation for declarative oauth (airbytehq#52041)
Browse files Browse the repository at this point in the history
Co-authored-by: Oleksandr Bazarnov <[email protected]>
  • Loading branch information
bnchrch and bazarnov authored Feb 4, 2025
1 parent 30521ca commit 90557c2
Show file tree
Hide file tree
Showing 16 changed files with 1,592 additions and 425 deletions.
2 changes: 1 addition & 1 deletion docs/connector-development/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ If you need support along the way, visit the [Slack channel](https://airbytehq.s
| Tool | Description |
| ----------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| [Connector Builder](./connector-builder-ui/overview.md) | We recommend Connector Builder for developing a connector for an API source. If you’re using Airbyte Cloud, no local developer environment is required to create a new connection with the Connector Builder because you configure it directly in the Airbyte web UI. This tool guides you through creating and testing a connection. Refer to our [tutorial](./connector-builder-ui/tutorial.mdx) on the Connector Builder to guide you through the basics. |
| [Low Code Connector Development Kit (CDK)](./config-based/low-code-cdk-overview.md) | This framework lets you build source connectors for HTTP API sources. The Low-code CDK is a declarative framework that allows you to describe the connector using a [YAML schema](./schema-reference) without writing Python code. It’s flexible enough to include [custom Python components](./config-based/advanced-topics.md#custom-components) in conjunction with this method if necessary. |
| [Low Code Connector Development Kit (CDK)](./config-based/low-code-cdk-overview.md) | This framework lets you build source connectors for HTTP API sources. The Low-code CDK is a declarative framework that allows you to describe the connector using a [YAML schema](./schema-reference) without writing Python code. It’s flexible enough to include [custom Python components](./config-based/advanced-topics/custom-components.md) in conjunction with this method if necessary. |
| [Python Connector Development Kit (CDK)](./cdk-python/basic-concepts.md) | While this method provides the most flexibility to developers, it also requires the most code and maintenance. This library provides classes that work out-of-the-box for most scenarios you’ll encounter along with the generators to make the connector scaffolds for you. We maintain an [in-depth guide](./tutorials/custom-python-connector/0-getting-started.md) to building a connector using the Python CDK. |
| [Java CDK](./tutorials/building-a-java-destination.md) | If you're bulding a source or a destination against a traditional database (not an HTTP API, not a vector database), you should use the Java CDK instead. |

Expand Down
321 changes: 0 additions & 321 deletions docs/connector-development/config-based/advanced-topics.md

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Component Schema Reference

A JSON schema representation of the relationships between the components that can be used in the YAML configuration can be found [here](https://github.com/airbytehq/airbyte-python-cdk/blob/main/airbyte_cdk/sources/declarative/declarative_component_schema.yaml).
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
# Custom Components

:::info
Please help us improve the low code CDK! If you find yourself needing to build a custom component,please [create a feature request issue](https://github.com/airbytehq/airbyte/issues/new?assignees=&labels=type%2Fenhancement%2C+%2Cneeds-triage%2C+area%2Flow-code%2Fcomponents&template=feature-request.md&title=Low%20Code%20Feature:). If appropriate, we'll add it directly to the framework (or you can submit a PR)!

If an issue already exist for the missing feature you need, please upvote or comment on it so we can prioritize the issue accordingly.
:::

Any built-in components can be overloaded by a custom Python class.
To create a custom component, define a new class in a new file in the connector's module.
The class must implement the interface of the component it is replacing. For instance, a pagination strategy must implement `airbyte_cdk.sources.declarative.requesters.paginators.strategies.pagination_strategy.PaginationStrategy`.
The class must also be a dataclass where each field represents an argument to configure from the yaml file, and an `InitVar` named parameters.

For example:

```
@dataclass
class MyPaginationStrategy(PaginationStrategy):
my_field: Union[InterpolatedString, str]
parameters: InitVar[Mapping[str, Any]]
def __post_init__(self, parameters: Mapping[str, Any]):
pass
def next_page_token(self, response: requests.Response, last_records: List[Mapping[str, Any]]) -> Optional[Any]:
pass
def reset(self):
pass
```

This class can then be referred from the yaml file by specifying the type of custom component and using its fully qualified class name:

```yaml
pagination_strategy:
type: "CustomPaginationStrategy"
class_name: "my_connector_module.MyPaginationStrategy"
my_field: "hello world"
```
### Custom Components that pass fields to child components
There are certain scenarios where a child subcomponent might rely on a field defined on a parent component. For regular components, we perform this propagation of fields from the parent component to the child automatically.
However, custom components do not support this behavior. If you have a child subcomponent of your custom component that falls under this use case, you will see an error message like:
```
Error creating component 'DefaultPaginator' with parent custom component source_example.components.CustomRetriever: Please provide DefaultPaginator.$parameters.url_base
```
When you receive this error, you can address this by defining the missing field within the `$parameters` block of the child component.

```yaml
paginator:
type: "DefaultPaginator"
<...>
$parameters:
url_base: "https://example.com"
```
Loading

0 comments on commit 90557c2

Please sign in to comment.