SQS back off strategy

[nolde]

How would one go about using Lift to implement a strategy similar to the ones described in the articles below?

https://aws.amazon.com/blogs/compute/using-amazon-sqs-dead-letter-queues-to-replay-messages/
https://medium.com/@andy.tarpley/highly-reliable-processing-with-amazon-sqs-and-lambda-bcbfb0262490

In summary, it uses multiple dead letters queues to create a handling back off, as each new queue level has different visibility timeouts and delivery delays.

[mnapoli]

Oh that's very interesting indeed, I love the idea 👍

Currently the queue construct in Lift doesn't implement any of that, and I don't think there's an easy/quick way to do this. I wonder if this would warrant a new construct though 🤔 That sounds like extra work/complexity, for little benefit. But maybe I'm wrong.

Are there scenarios where this setup would be very interesting?

Also I wonder if a simpler setup wouldn't be enough, with 3 queues:

• primary queue (1 retry, ~30s delay between retries)
• backoff queue (0 retry, 15min delay)
• DLQ

?

[nolde]

A common use case is 3rd party service throttling. If you handling depends on some 3rd party service, and you have a sudden burst of usage, having this setup allows the system to safely back off and retry without your direct intervention.

Another use case is handling simultaneous updates across the system. In my case, I have different handlers listening to different events within the system, and sometimes they try to update a record and the same time as a user or another handler tried to do the same. When that happens, I usually just let it fail the conditional constraints and push it in a reconciliation queue, with a new type of handling that basically refreshes the event payload with fresh data from the database.

Anyway, any setup that would benefit from exponential backoff can use this setup. I have been using it with great success.

In my specific case, 3 queues are indeed enough; I've never had the need for more than that. On the backoff queue, I usually allow at least 1 retry, using the receive wait time to create the 1st rate reducing level, and a larger visibility timeout for a 2nd level (it is supposed to be exponential).

I am thinking about migrating to Lift after getting sick of managing the queues using CloudFormation myself =). So, if you or anyone else have an example of how this would look in Lift, I would appreciate it!