[feature](iceberg) support s3 tables #47115
Draft
+220
−0
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
This pull request introduces several changes to the project, primarily focusing on adding support for S3 tables in the Iceberg catalog and updating dependencies. The most significant changes include updating the
pom.xml
files, adding a new catalog type for S3 tables, and implementing a custom AWS credentials provider.Dependency Updates:
fe/fe-core/pom.xml
: Added dependencies forsoftware.amazon.awssdk:s3tables
,software.amazon.s3tables:s3-tables-catalog-for-iceberg
, andsoftware.amazon.awssdk:sdk-core
.fe/pom.xml
: Updateddoris.hive.catalog.shade.version
to2.1.3
,iceberg.version
to1.6.1
, and addeds3tables.catalog.version
set to0.1.4
. [1] [2] [3]Code Changes for S3 Tables Support:
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalCatalog.java
: AddedICEBERG_S3_TABLES
constant.fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalCatalogFactory.java
: Added a case forICEBERG_S3_TABLES
to create anIcebergS3TablesExternalCatalog
instance.fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergS3TablesExternalCatalog.java
: Implemented theIcebergS3TablesExternalCatalog
class to support S3 tables.fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/s3tables/CustomAwsCredentialsProvider.java
: Implemented a custom AWS credentials provider.Additional Changes:
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/source/IcebergScanNode.java
: Added a case forICEBERG_S3_TABLES
in theIcebergScanNode
constructor.fe/fe-core/src/main/java/org/apache/doris/persist/gson/GsonUtils.java
: RegisteredIcebergS3TablesExternalCatalog
for Gson serialization. [1] [2]fe/fe-core/src/test/java/org/apache/doris/datasource/s3tables/S3TablesTest.java
: Added a test class for S3 tables catalog functionality.These changes collectively enhance the project's capability to manage S3 tables within the Iceberg catalog, along with updating necessary dependencies to ensure compatibility and functionality.
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)