Skip to content

Commit

Permalink
Add more details to SQS queue
Browse files Browse the repository at this point in the history
  • Loading branch information
ha7315 committed Jan 8, 2025
1 parent d103eba commit e80afa6
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion modules/postgres-etl/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,4 +97,4 @@ Contains environment variables for container `pg_restore` within **Load** ECS ta

#### SQS Integration for ETL process

The SQS queues are used to watch the S3 bucket in the Extract environment. We only need it in the `extract` module. The main queue `postgres-etl-s3` receives messages about the data extraction events and a dead-letter queue `postgres-etl-s3-dlq` handles messages that cannot be processed successfully.
The SQS queues are used to watch the S3 bucket in the Extract environment. We only need it in the `extract` module. The main queue `postgres-etl-s3` receives messages about the data extraction events and a dead-letter queue `postgres-etl-s3-dlq` handles messages that cannot be processed successfully. Another component of the ETL process polls the `postgres-etl-s3` SQS queue for messages. When a message is received, it triggers the next steps in the ETL process such as processing the extracted data using Kubernetes jobs and Step Functions.

0 comments on commit e80afa6

Please sign in to comment.