Documentation

Overview

S3 built-in replication mechanism works only on a 1:1 basis, i.e. you can only configure it between one source bucket and one target bucket, meaning we cannot have a master bucket from which to replicate the artifacts automatically to all other buckets. In the Web Platform squad we started to develop an alternative solution to overcome this problem. Lambda-based replication We can use bucket events to trigger a lambda execution which will download the artifact it addressed on event and upload to multiple other buckets. The Amazon S3 notification feature enables you to receive notifications when certain events happen in your bucket. To enable notifications, you must first add a notification configuration identifying the events you want Amazon S3 to publish, and the destinations where you want Amazon S3 to send the event notifications. LambdaConfigurations is a property of the Amazon S3 NotificationConfiguration property that describes the AWS Lambda (Lambda) functions to invoke and the events for which to invoke them.

Implementation

The master (main) bucket is the one that always talk to slave (replicated) buckets through s3-replicator lambda function. To do this following configuration needs to be done: Add a tag to master bucket to get target S3 buckets names.

"Tags": [
  {
      "Key": "ReplicatedTargetBuckets",
      "Value": {
          "bucket1@ap-southeast-1 bucket2@eu-central-1"
      }
  }
]

• Name (Not ARN) of target buckets should be added followed by @ and name of the region
• If replicating to more then one region, the bucketName@region should be space separated

Add Lambda notification to master bucket

"NotificationConfiguration": {
    "LambdaConfigurations": [
         {
             "Event": "s3:ObjectCreated:Put",
             "Function": { "Ref": "S3ReplicatorFunctionName" }
         },
         {
             "Event": "s3:ObjectRemoved:Delete",
             "Function": { "Ref": "S3ReplicatorFunctionName" }
         }
     ]
 }

• S3ReplicatorFunctionName is the s3-replicator lambda function name(Not ARN)

Add permission to master bucket to access to your lambda function

• BucketName is the master bucket name

After configuring these, the lambda function will be listening to CreateDelete events happening to master bucket and automatically replicate those changes to slave buckets. No actual change needed to be done on lambda function.

User Guide The step by step user guide can be found on wiki page user Guide