Assemble CARs aws lambda function

[vasco-santos]

We currently write to S3 all CARs we receive under a /raw namespace. Some of them are already complete, while others are partial CARs with a partial DAG of a given upload.

Work triggered in the context of #19 , but also valuable for other directions.

Flow

Lambda function triggers on S3 write assemble-cars-lambda

• verify if /raw namespace trigger
• perform CAR completeness optimistic validations
  • is CAR file complete? we can rely on S3 metadata
• if CAR is complete: write to /complete namespace
• else: does CAR file have a DagPB root with know acceptable size?
  • get list of CARs in same directory and verify if their total size is bigger than known size
  • Get all CARs in the S3 directory
  • Validate CAR is complete through all the links
  • Join CARs and write to /complete namespace

Metrics

TBD

• Successful warm cache requests
• Length distribution of warm cache content

Notes

• We need to look into limitations of AWS Lambda memory for "acceptable" size config