You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Before enabling compaction we need to ensure that we are passing our event UTC time to some yet to be developed custom header for compaction to use to determine order otherwise order is not guaranteed.
Once we confirm that Kafka now supports compaction based on timestamp we need to:
Upgrade Kafka to version supporting this
Update Relay to pass a custom compaction timestamp based on event UTC time
Enable and test compaction in dev by wiping existing topics, running multiple test runs, then re-indexing from compacted topics and assessing data consistency
Repeat process in QA (this has multiple consumers/producers per topic and is a better test)
Migrate prod by rerunning relay-splitter into compacted topics, replacing the existing workflow and task topics
The text was updated successfully, but these errors were encountered:
Need to check on: https://cwiki.apache.org/confluence/display/KAFKA/KIP-280%3A+Enhanced+log+compaction
Before enabling compaction we need to ensure that we are passing our event UTC time to some yet to be developed custom header for compaction to use to determine order otherwise order is not guaranteed.
Once we confirm that Kafka now supports compaction based on timestamp we need to:
The text was updated successfully, but these errors were encountered: