Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable Topic Compaction #98

Open
5 tasks
lepsalex opened this issue Jul 30, 2020 · 0 comments
Open
5 tasks

Enable Topic Compaction #98

lepsalex opened this issue Jul 30, 2020 · 0 comments
Labels
low Low priority to be worked on. tech-debt

Comments

@lepsalex
Copy link
Contributor

Need to check on: https://cwiki.apache.org/confluence/display/KAFKA/KIP-280%3A+Enhanced+log+compaction

Before enabling compaction we need to ensure that we are passing our event UTC time to some yet to be developed custom header for compaction to use to determine order otherwise order is not guaranteed.

Once we confirm that Kafka now supports compaction based on timestamp we need to:

  • Upgrade Kafka to version supporting this
  • Update Relay to pass a custom compaction timestamp based on event UTC time
  • Enable and test compaction in dev by wiping existing topics, running multiple test runs, then re-indexing from compacted topics and assessing data consistency
  • Repeat process in QA (this has multiple consumers/producers per topic and is a better test)
  • Migrate prod by rerunning relay-splitter into compacted topics, replacing the existing workflow and task topics
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
low Low priority to be worked on. tech-debt
Projects
None yet
Development

No branches or pull requests

1 participant