Skip to content

Commit

Permalink
Rename airbyte-config module (airbytehq#24885)
Browse files Browse the repository at this point in the history
* rename airbyte-config module

* Automated Commit - Formatting Changes

* sanity

* update import

* update import

* update script

* update script

* update script

* update script

* Automated Change

* Automated Change

* Automated Change

* Automated Change

* update awsdatalake icon

* point slash commands to new path

* sanity

* Automated Commit - Formatting Changes

* sanity

* Automated Change

* Automated Change

* sanity

---------

Co-authored-by: rodireich <[email protected]>
  • Loading branch information
rodireich and rodireich authored Apr 6, 2023
1 parent 3ba15b5 commit 0bab175
Show file tree
Hide file tree
Showing 510 changed files with 50,158 additions and 705 deletions.
2 changes: 1 addition & 1 deletion .env
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
# Variables with defaults have been omitted to avoid duplication of defaults.
# The only exception to the non-default rule are env vars related to scaling.
#
# See https://github.com/airbytehq/airbyte/blob/master/airbyte-config/config-models/src/main/java/io/airbyte/config/Configs.java
# See https://github.com/airbytehq/airbyte/blob/master/airbyte-config-oss/config-models-oss/src/main/java/io/airbyte/config/Configs.java
# for the latest environment variables.
#
# # Contributors - please organise this env file according to the above linked file.
Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/deploy-oss-catalog.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ on:
branches:
- master
paths:
- airbyte-config/init/src/main/resources/seed/**
- airbyte-config-oss/init-oss/src/main/resources/seed/**

workflow_dispatch:

Expand All @@ -29,12 +29,12 @@ jobs:
distribution: "zulu"
java-version: "17"
- name: Generate catalog
run: SUB_BUILD=ALL_CONNECTORS ./gradlew :airbyte-config:specs:generateOssConnectorCatalog
run: SUB_BUILD=ALL_CONNECTORS ./gradlew :airbyte-config-oss:specs-oss:generateOssConnectorCatalog
- name: Upload catalog to GCS
shell: bash
run: |
gcs_bucket_name="prod-airbyte-cloud-connector-metadata-service"
catalog_path="airbyte-config/init/src/main/resources/seed/oss_catalog.json"
catalog_path="airbyte-config-oss/init-oss/src/main/resources/seed/oss_catalog.json"
gsutil -h "Cache-Control:public, max-age=10" cp "$catalog_path" "gs://$gcs_bucket_name/oss_catalog.json"
- name: Trigger Cloud catalog generation
uses: peter-evans/repository-dispatch@v2
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/gradle.yml
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ jobs:
- 'tools/**'
- '*.gradle'
- 'deps.toml'
- 'airbyte-config/**'
- 'airbyte-config-oss/**'
cdk:
- 'airbyte-cdk/**'
cli:
Expand Down Expand Up @@ -327,7 +327,7 @@ jobs:
- name: Process Resources
uses: Wandalen/wretry.action@master
with:
command: SUB_BUILD=CONNECTORS_BASE ./gradlew :airbyte-config:init:processResources --scan
command: SUB_BUILD=CONNECTORS_BASE ./gradlew :airbyte-config-oss:init-oss:processResources --scan
attempt_limit: 3
attempt_delay: 5000 # in ms

Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/publish-command.yml
Original file line number Diff line number Diff line change
Expand Up @@ -334,7 +334,7 @@ jobs:
if: github.event.inputs.auto-bump-version == 'true' && success()
run: |
connector="airbyte/${{ env.IMAGE_NAME }}"
definitionpath=./airbyte-config/init/src/main/resources/seed/
definitionpath=./airbyte-config-oss/init-oss/src/main/resources/seed/
sourcecheck=$(yq e ".. | select(has(\"dockerRepository\")) | select(.dockerRepository == \"$connector\")" "$definitionpath"source_definitions.yaml)
destcheck=$(yq e ".. | select(has(\"dockerRepository\")) | select(.dockerRepository == \"$connector\")" "$definitionpath"destination_definitions.yaml)
if [[ (-z "$sourcecheck" && -z "$destcheck") ]]
Expand All @@ -344,7 +344,7 @@ jobs:
if: github.event.inputs.auto-bump-version == 'true' && success()
run: |
connector="airbyte/${{ env.IMAGE_NAME }}"
definitionpath=./airbyte-config/init/src/main/resources/seed/
definitionpath=./airbyte-config-oss/init-oss/src/main/resources/seed/
sourcename=$(yq e ".[] | select(has(\"dockerRepository\")) | select(.dockerRepository == \"$connector\") | .name" "$definitionpath"source_definitions.yaml)
destname=$(yq e ".[] | select(has(\"dockerRepository\")) | select(.dockerRepository == \"$connector\") | .name" "$definitionpath"destination_definitions.yaml)
if [ -z "$sourcename" ]
Expand All @@ -355,7 +355,7 @@ jobs:
if: github.event.inputs.auto-bump-version == 'true' && success()
uses: Wandalen/wretry.action@master
with:
command: ./gradlew :airbyte-config:init:processResources
command: ./gradlew :airbyte-config-oss:init-oss:processResources
attempt_limit: 3
attempt_delay: 5000 # in ms
- name: git config
Expand Down
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ docs/SUMMARY.md
**/specs_secrets_mask.yaml

# Files generated for uploading to GCS
airbyte-config/**/resources/seed/oss_catalog.json
airbyte-config-oss/**/resources/seed/oss_catalog.json

# Output Files generated by scripts
lowcode_connector_names.txt
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ This module uses `jsonschema2pojo` to generate Java config objects from [json sc
```
- Run the following command under the project root:
```sh
SUB_BUILD=PLATFORM ./gradlew airbyte-config:config-models:generateJsonSchema2Pojo
SUB_BUILD=PLATFORM ./gradlew airbyte-config-oss:config-models-oss:generateJsonSchema2Pojo
```
The generated file is under:
```
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ jsonSchema2Pojo {
source = files("${sourceSets.main.output.resourcesDir}/types")
targetDirectory = new File(project.buildDir, 'generated/src/gen/java/')

targetPackage = 'io.airbyte.config'
targetPackage = 'io.airbyte.configoss'
useLongIntegers = true

removeOldOutput = true
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
* Copyright (c) 2023 Airbyte, Inc., all rights reserved.
*/

package io.airbyte.config;
package io.airbyte.configoss;

import java.io.File;

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
* Copyright (c) 2023 Airbyte, Inc., all rights reserved.
*/

package io.airbyte.config;
package io.airbyte.configoss;

import io.airbyte.validation.json.AbstractSchemaValidator;
import java.nio.file.Path;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
* Copyright (c) 2023 Airbyte, Inc., all rights reserved.
*/

package io.airbyte.config;
package io.airbyte.configoss;

import java.util.Optional;

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
* Copyright (c) 2023 Airbyte, Inc., all rights reserved.
*/

package io.airbyte.config;
package io.airbyte.configoss;

import io.airbyte.commons.json.JsonSchemas;
import java.io.File;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
* Copyright (c) 2023 Airbyte, Inc., all rights reserved.
*/

package io.airbyte.config;
package io.airbyte.configoss;

/**
* This interface defines the general variables for configuring Airbyte.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
* Copyright (c) 2023 Airbyte, Inc., all rights reserved.
*/

package io.airbyte.config;
package io.airbyte.configoss;

import java.util.Map;
import java.util.Optional;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,15 @@
* Copyright (c) 2023 Airbyte, Inc., all rights reserved.
*/

package io.airbyte.config.helpers;
package io.airbyte.configoss.helpers;

import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.JsonNode;
import com.google.common.collect.Iterables;
import io.airbyte.commons.json.Jsons;
import io.airbyte.config.State;
import io.airbyte.config.StateType;
import io.airbyte.config.StateWrapper;
import io.airbyte.configoss.State;
import io.airbyte.configoss.StateType;
import io.airbyte.configoss.StateWrapper;
import io.airbyte.protocol.models.AirbyteStateMessage;
import io.airbyte.protocol.models.AirbyteStateMessage.AirbyteStateType;
import java.util.List;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,16 @@
* Copyright (c) 2023 Airbyte, Inc., all rights reserved.
*/

package io.airbyte.config.helpers;
package io.airbyte.configoss.helpers;

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.util.ClassUtil;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.Preconditions;
import io.airbyte.commons.json.Jsons;
import io.airbyte.commons.yaml.Yamls;
import io.airbyte.config.StandardDestinationDefinition;
import io.airbyte.config.StandardSourceDefinition;
import io.airbyte.configoss.StandardDestinationDefinition;
import io.airbyte.configoss.StandardSourceDefinition;
import java.util.AbstractMap.SimpleImmutableEntry;
import java.util.ArrayList;
import java.util.HashSet;
Expand All @@ -24,7 +24,7 @@
* human-friendly yaml to processing friendly formats i.e. Java models or JSON. As this class
* performs validation, it is recommended to use this class to deal with plain lists. An example of
* such lists are Airbyte's master definition lists, which can be seen in the resources folder of
* the airbyte-config/seed module.
* the airbyte-config-oss/seed module.
*
* In addition to usual deserialization validations, we check: 1) The given list contains no
* duplicate names. 2) The given list contains no duplicate ids.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
"$schema": http://json-schema.org/draft-07/schema#
"$id": https://github.com/airbytehq/airbyte/blob/master/airbyte-config/models/src/main/resources/types/ActorDefinitionResourceRequirements.yaml
"$id": https://github.com/airbytehq/airbyte/blob/master/airbyte-config-oss/models-oss/src/main/resources/types/ActorDefinitionResourceRequirements.yaml
title: ActorDefinitionResourceRequirements
description: actor definition specific resource requirements
type: object
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
"$schema": http://json-schema.org/draft-07/schema#
"$id": https://github.com/airbytehq/airbyte/blob/master/airbyte-config/models/src/main/resources/types/ActorType.yaml
"$id": https://github.com/airbytehq/airbyte/blob/master/airbyte-config-oss/models-oss/src/main/resources/types/ActorType.yaml
title: ActorType
description: enum that describes different types of actors
type: string
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
"$schema": http://json-schema.org/draft-07/schema#
"$id": https://github.com/airbytehq/airbyte/blob/master/airbyte-config/models/src/main/resources/types/AllowedHosts.yaml
"$id": https://github.com/airbytehq/airbyte/blob/master/airbyte-config-oss/models-oss/src/main/resources/types/AllowedHosts.yaml
title: AllowedHosts
description: A connector's allowed hosts. If present, the platform will limit communication to only hosts which are listed in `AllowedHosts.hosts`.
type: object
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
"$schema": http://json-schema.org/draft-07/schema#
"$id": https://github.com/airbytehq/airbyte/blob/master/airbyte-config/models/src/main/resources/types/CombinedConnectorCatalog.yaml
"$id": https://github.com/airbytehq/airbyte/blob/master/airbyte-config-oss/models-oss/src/main/resources/types/CombinedConnectorCatalog.yaml
title: CombinedConnectorCatalog
description: Used to provide the connector catalog from a remote source
type: object
Expand All @@ -11,8 +11,8 @@ properties:
destinations:
type: array
items:
existingJavaType: io.airbyte.config.StandardDestinationDefinition
existingJavaType: io.airbyte.configoss.StandardDestinationDefinition
sources:
type: array
items:
existingJavaType: io.airbyte.config.StandardSourceDefinition
existingJavaType: io.airbyte.configoss.StandardSourceDefinition
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
"$schema": http://json-schema.org/draft-07/schema#
"$id": https://github.com/airbytehq/airbyte/blob/master/airbyte-config/models/src/main/resources/types/ConnectorJobOutput.yaml
"$id": https://github.com/airbytehq/airbyte/blob/master/airbyte-config-oss/models-oss/src/main/resources/types/ConnectorJobOutput.yaml
title: ConnectorJobOutput
description: connector command job output
type: object
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
"$schema": http://json-schema.org/draft-07/schema#
"$id": https://github.com/airbytehq/airbyte/blob/master/airbyte-config/models/src/main/resources/types/DataType.yaml
"$id": https://github.com/airbytehq/airbyte/blob/master/airbyte-config-oss/models-oss/src/main/resources/types/DataType.yaml
title: DataType
description: standard data types.
type: string
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
"$schema": http://json-schema.org/draft-07/schema#
"$id": https://github.com/airbytehq/airbyte/blob/master/airbyte-config/models/src/main/resources/types/DestinationConnection.yaml
"$id": https://github.com/airbytehq/airbyte/blob/master/airbyte-config-oss/models-oss/src/main/resources/types/DestinationConnection.yaml
title: DestinationConnection
description: information required for connection to a destination.
type: object
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
"$schema": http://json-schema.org/draft-07/schema#
"$id": https://github.com/airbytehq/airbyte/blob/master/airbyte-config/models/src/main/resources/types/DestinationOAuthParameter.yaml
"$id": https://github.com/airbytehq/airbyte/blob/master/airbyte-config-oss/models-oss/src/main/resources/types/DestinationOAuthParameter.yaml
title: DestinationOAuthParameter
description: OAuth parameters used when connecting to destination
type: object
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
"$schema": http://json-schema.org/draft-07/schema#
"$id": https://github.com/airbytehq/airbyte/blob/master/airbyte-config/models/src/main/resources/types/DestinationSyncMode.yaml
"$id": https://github.com/airbytehq/airbyte/blob/master/airbyte-config-oss/models-oss/src/main/resources/types/DestinationSyncMode.yaml
title: DestinationSyncMode
description: destination sync modes.
type: string
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
"$schema": http://json-schema.org/draft-07/schema#
"$id": https://github.com/airbytehq/airbyte/blob/master/airbyte-config/models/src/main/resources/types/DockerImageSpec.yaml
"$id": https://github.com/airbytehq/airbyte/blob/master/airbyte-config-oss/models-oss/src/main/resources/types/DockerImageSpec.yaml
title: DockerImageSpec
description: docker image name and the connector specification associated with it
type: object
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
"$schema": http://json-schema.org/draft-07/schema#
"$id": https://github.com/airbytehq/airbyte/blob/master/airbyte-config/models/src/main/resources/types/FailureReason.yaml
"$id": https://github.com/airbytehq/airbyte/blob/master/airbyte-config-oss/models-oss/src/main/resources/types/FailureReason.yaml
title: FailureSummary
type: object
required:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
"$schema": http://json-schema.org/draft-07/schema#
"$id": https://github.com/airbytehq/airbyte/blob/master/airbyte-config/models/src/main/resources/types/JobGetSpecConfig.yaml
"$id": https://github.com/airbytehq/airbyte/blob/master/airbyte-config-oss/models-oss/src/main/resources/types/JobGetSpecConfig.yaml
title: JobGetSpecConfig
description: job check get spec
type: object
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
"$schema": http://json-schema.org/draft-07/schema#
"$id": https://github.com/airbytehq/airbyte/blob/master/airbyte-config/models/src/main/resources/types/JobSyncConfig.yaml
"$id": https://github.com/airbytehq/airbyte/blob/master/airbyte-config-oss/models-oss/src/main/resources/types/JobSyncConfig.yaml
title: JobSyncConfig
description: job sync config
type: object
Expand Down Expand Up @@ -40,11 +40,11 @@ properties:
sourceResourceRequirements:
type: object
description: optional resource requirements to use in source container - this is used instead of `resourceRequirements` for the source container
existingJavaType: io.airbyte.config.ResourceRequirements
existingJavaType: io.airbyte.configoss.ResourceRequirements
destinationResourceRequirements:
type: object
description: optional resource requirements to use in dest container - this is used instead of `resourceRequirements` for the dest container
existingJavaType: io.airbyte.config.ResourceRequirements
existingJavaType: io.airbyte.configoss.ResourceRequirements
operationSequence:
description: Sequence of configurations of operations to apply as part of the sync
type: array
Expand All @@ -57,7 +57,7 @@ properties:
resourceRequirements:
type: object
description: optional resource requirements to run sync workers - this is used for containers other than the source/dest containers
existingJavaType: io.airbyte.config.ResourceRequirements
existingJavaType: io.airbyte.configoss.ResourceRequirements
isSourceCustomConnector:
description: determine if the source running image is a custom connector.
type: boolean
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
"$schema": http://json-schema.org/draft-07/schema#
"$id": https://github.com/airbytehq/airbyte/blob/master/airbyte-config/models/src/main/resources/types/JobType.yaml
"$id": https://github.com/airbytehq/airbyte/blob/master/airbyte-config-oss/models-oss/src/main/resources/types/JobType.yaml
title: JobType
description: enum that describes the different types of jobs that the platform runs.
type: string
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
"$schema": http://json-schema.org/draft-07/schema#
"$id": https://github.com/airbytehq/airbyte/blob/master/airbyte-config/models/src/main/resources/types/NamespaceDefinitionType.yaml
"$id": https://github.com/airbytehq/airbyte/blob/master/airbyte-config-oss/models-oss/src/main/resources/types/NamespaceDefinitionType.yaml
title: Namespace Definition
description: Method used for computing final namespace in destination
type: string
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
"$schema": http://json-schema.org/draft-07/schema#
"$id": https://github.com/airbytehq/airbyte/blob/master/airbyte-config/models/src/main/resources/types/NormalizationDestinationDefinitionConfig.yaml
"$id": https://github.com/airbytehq/airbyte/blob/master/airbyte-config-oss/models-oss/src/main/resources/types/NormalizationDestinationDefinitionConfig.yaml
title: NormalizationDestinationDefinitionConfig
description: describes a normalization config for destination definition
type: object
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
"$schema": http://json-schema.org/draft-07/schema#
"$id": https://github.com/airbytehq/airbyte/blob/master/airbyte-config/models/src/main/resources/types/NotificationType.yaml
"$id": https://github.com/airbytehq/airbyte/blob/master/airbyte-config-oss/models-oss/src/main/resources/types/NotificationType.yaml
title: NotificationType
description: Type of notification
type: string
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
"$schema": http://json-schema.org/draft-07/schema#
"$id": https://github.com/airbytehq/airbyte/blob/master/airbyte-config/models/src/main/resources/types/OperatorDbt.yaml
"$id": https://github.com/airbytehq/airbyte/blob/master/airbyte-config-oss/models-oss/src/main/resources/types/OperatorDbt.yaml
title: OperatorDbt
description: Settings for a DBT operator
type: object
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
"$schema": http://json-schema.org/draft-07/schema#
"$id": https://github.com/airbytehq/airbyte/blob/master/airbyte-config/models/src/main/resources/types/OperatorNormalization.yaml
"$id": https://github.com/airbytehq/airbyte/blob/master/airbyte-config-oss/models-oss/src/main/resources/types/OperatorNormalization.yaml
title: OperatorNormalization
description: Settings for a normalization operator
type: object
Expand Down
Loading

0 comments on commit 0bab175

Please sign in to comment.