Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(mu): add task recovery for pending tasks #1080

Open
jfrain99 opened this issue Dec 5, 2024 · 0 comments
Open

feat(mu): add task recovery for pending tasks #1080

jfrain99 opened this issue Dec 5, 2024 · 0 comments

Comments

@jfrain99
Copy link
Contributor

jfrain99 commented Dec 5, 2024

Background

#1074 added an exponential backoff to the MU retry mechanism. The retry mechanism attempts to process a message, and on an error, it adds it back to the queue after a (now exponential) timeout. Because of this change, tasks can sit in these timeouts for up to 2 days. Before, they could only wait 5 seconds.

Problem

This greatly increases the chances of tasks being in a retry delay when a MU cycles. These messages are lost! We need to make it so that tasks waiting to be retried are not lost on a cycle / restart.

Proposed Solution

Currently, tasks are added to a sqlite table (tasks) and removed from the database when they are dequeued and begin processing. Instead, we can add a status column to the tasks table that marks whether a task has be dequeued. Then, when the message is successfully processed (or runs out of retries), the task is removed from the table. When recovering the table on cycle, all pending statuses should be overwritten.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant