fix: add pagination to deposit fetching

[Jiloc]

Description

Closes: #1148

Changes

• add pagination handling to get_deposits
• add pagination_timeout to EmilyClient - it causes the client to stop fetching deposits - but still return the ones alrealdy fetched - when the time passed > timeout.
• added pagination_timeout to the configs - currently set to 15

Testing Information

• Added integration tests to test pagination and timeout on get_deposits

Checklist:

• I have performed a self-review of my code
• My changes generate no new warnings
• New and existing unit tests pass locally with my changes

[Jiloc]

The idea of using the the BlockHeight to filter deposits already processed, may not work. The BlockHeight is not guaranteed to remain consistent in Emily (also currently they are returned from newest to oldest). In the event of a reorg, requests could be updated with a different associated block. This could result in requests being modified to have a lower height than the previous one. If we only consider requests with a BlockHeight greater than or equal to a certain value, it could lead to signers skipping requests after a reorg. A potential solution could be to add a created_at parameter to Dynamo and use that value for sorting and ignoring.

[Jiloc]

I'd propose to decide on a strategy for sorting for Emily and implement a filtering mechanism on the signers side in a separate PR

[cylewitruk]

The idea of using the the BlockHeight to filter deposits already processed, may not work. The BlockHeight is not guaranteed to remain consistent in Emily (also currently they are returned from newest to oldest). In the event of a reorg, requests could be updated with a different associated block. This could result in requests being modified to have a lower height than the previous one. If we only consider requests with a BlockHeight greater than or equal to a certain value, it could lead to signers skipping requests after a reorg. A potential solution could be to add a created_at parameter to Dynamo and use that value for sorting and ignoring.

If I got to choose, I would probably have opted for a streaming protocol backed by durable topic subscriptions (at least that's what they're called in the message broker world..). For example MQTT over Websockets (AWS version), instead of polling. That would obsolete the above challenges.

[Jiloc]

The idea of using the the BlockHeight to filter deposits already processed, may not work. The BlockHeight is not guaranteed to remain consistent in Emily (also currently they are returned from newest to oldest). In the event of a reorg, requests could be updated with a different associated block. This could result in requests being modified to have a lower height than the previous one. If we only consider requests with a BlockHeight greater than or equal to a certain value, it could lead to signers skipping requests after a reorg. A potential solution could be to add a created_at parameter to Dynamo and use that value for sorting and ignoring.

If I got to choose, I would probably have opted for a streaming protocol backed by durable topic subscriptions (at least that's what they're called in the message broker world..). For example MQTT over Websockets (AWS version), instead of polling. That would obsolete the above challenges.

I love the idea and believe this is the cleanest way to go for the signers. We'll have to design it for a future milestone though, it will require a different infra because of the usual limitations of the current setup...
We still need to support basic pagination, filtering and ordering in the REST API though. That will help in the short term with the signers and for every other user downstream (e.g. the bridge) for the long.