Skip to content

Commit

Permalink
Add a tutorial for using a locally generated catalog (airbytehq#23448)
Browse files Browse the repository at this point in the history
  • Loading branch information
bnchrch authored Feb 24, 2023
1 parent 7da6a3b commit c3f657f
Show file tree
Hide file tree
Showing 2 changed files with 66 additions and 7 deletions.
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
# Testing The Local Catalog

## Purpose of this document
This document describes how to
1. Generate the OSS catalog in the `airbyte` repo
1. Move the OSS catalog to the `airbyte-platform` repo
1. Point the platform to the new catalog
1. Run the platform

## Why you might need to
1. You've added/updated/deleted a generally available connector and want to test it in the platform UI
1. You've added/updated/deleted a generally available connector and want to test it in the platform API

## Steps

### 1. Generate the OSS Catalog
In the `airbyte` repo run
```sh
SUB_BUILD=ALL_CONNECTORS ./gradlew :airbyte-config:specs:generateOssConnectorCatalog
```

### 2. Move the OSS catalog to platform
In the `airbyte` repo run
```sh
cp airbyte-config/init/src/main/resources/seed/oss_catalog.json <PATH_TO_PLATFORM_REPO>/airbyte-config/init/src/main/resources/seed/local_dev_catalog.json
```

### 3. Point the platform to the new catalog
In the `airbyte-platform` repo modify `.env` add the environment variable
```sh
LOCAL_CONNECTOR_CATALOG_PATH=seed/local_dev_catalog.json
```

### 4. Build the platform
In the `airbyte-platform` run
```sh
SUB_BUILD=PLATFORM ./gradlew build -x test
```

### 4. Run platform
In the `airbyte-platform` run
```sh
VERSION=dev docker-compose up
```

Success! Your new catalog should now be loaded into the database.
27 changes: 20 additions & 7 deletions docusaurus/sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ module.exports = {
label: 'On AWS EC2',
id: 'deploying-airbyte/on-aws-ec2',
},

{
type: 'doc',
label: 'On Azure',
Expand Down Expand Up @@ -196,12 +196,12 @@ module.exports = {
'troubleshooting/README',
'troubleshooting/on-deploying',
'troubleshooting/new-connection',
'troubleshooting/running-sync',
'troubleshooting/running-sync',
],
},
{
type: 'category',
label: 'Build a connector',
label: 'Build a connector',
items: [
{
type: 'doc',
Expand Down Expand Up @@ -249,7 +249,7 @@ module.exports = {
items: [
'connector-development/config-based/understanding-the-yaml-file/request-options',
'connector-development/config-based/understanding-the-yaml-file/authentication',
'connector-development/config-based/understanding-the-yaml-file/error-handling',
'connector-development/config-based/understanding-the-yaml-file/error-handling',
]
},
'connector-development/config-based/understanding-the-yaml-file/incremental-syncs',
Expand All @@ -259,7 +259,7 @@ module.exports = {
'connector-development/config-based/understanding-the-yaml-file/reference',
]
},
'connector-development/config-based/advanced-topics',
'connector-development/config-based/advanced-topics',
]
},
'connector-development/tutorials/cdk-speedrun',
Expand Down Expand Up @@ -301,8 +301,21 @@ module.exports = {
},
'connector-development/cdk-faros-js',
'connector-development/airbyte101',
'connector-development/testing-connectors/README',
'connector-development/testing-connectors/connector-acceptance-tests-reference',
{
type: 'category',
label: 'Testing Connectors',
link: {
type: 'doc',
id: 'connector-development/testing-connectors/README',
},
items: [
'connector-development/testing-connectors/connector-acceptance-tests-reference',
'connector-development/testing-connectors/legacy-standard-source-tests',
'connector-development/testing-connectors/source-acceptance-tests-reference',
'connector-development/testing-connectors/standard-source-tests',
'connector-development/testing-connectors/testing-a-local-catalog-in-development',
]
},
'connector-development/connector-specification-reference',
'connector-development/best-practices',
'connector-development/ux-handbook',
Expand Down

0 comments on commit c3f657f

Please sign in to comment.