Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Enhance bigtable-beam-import pipeline to import multiple hbase table snapshots into bigtable #4072

Conversation

prathapreddy123
Copy link

Feature request to enhance the bigtable-beam-import pipeline to support following features:

  • Importing multiple snapshots in the same pipeline into corresponding BigTable tables
  • Handle parallelization of Skewed regions to enhance performance
  • Configuring additional BigTable configuration

Fixes #4070 ☕️

@prathapreddy123 prathapreddy123 requested a review from a team as a code owner May 5, 2023 21:30
@product-auto-label product-auto-label bot added size: xl Pull request size is extra large. api: bigtable Issues related to the googleapis/java-bigtable-hbase API. labels May 5, 2023
Comment on lines +211 to +213
return providedPath == null
? SnapshotUtils.getNamedDirectory(hbaseSnapshotsPath, "restore")
: SnapshotUtils.appendCurrentTimestamp(providedPath);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should be more unique to avoid collisions (rather than minute timestamp, it should use. a uuid)

@igorbernstein2
Copy link
Collaborator

Continuing work on this in #4358

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigtable Issues related to the googleapis/java-bigtable-hbase API. size: xl Pull request size is extra large.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants