Skip to content
This repository has been archived by the owner on Jan 20, 2025. It is now read-only.

PSC-STM-A3: Add Storage for PSC Stream Timepoint #242

Open
tiredpixel opened this issue Feb 28, 2024 · 0 comments
Open

PSC-STM-A3: Add Storage for PSC Stream Timepoint #242

tiredpixel opened this issue Feb 28, 2024 · 0 comments
Assignees

Comments

@tiredpixel
Copy link
Contributor

When using the PSC Stream API, it requires either:

  • A timepoint pointer of the last record seen through the API, which will then return records seen since here
  • Nothing sent through, which will obtain any new records from now
stream_client.read_stream(timepoint:)

This timepoint is important to ensure that if there is any restart of requests (eg after a timeout or failure), that no records were lost in the meantime.

This needs to be:

  • Retrieved when starting up, to read latest timepoint
  • Stored after a new record has been processed

A simple key/value pair is sufficient for storage and retrieval of this, so potentially Redis would be a good choice if that is available. However, note that if this is on a single EC2 instance it is a single point of failure.

Estimate: 6 hours

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Development

No branches or pull requests

1 participant