feat: initial implementation

.github/workflows/deploy.yml

@@ -0,0 +1,35 @@
+---
+name: Deploy
+on: [workflow_dispatch]
+env:
+  AWS_REGION: us-east-2
+  LAMBDA: assemble-cars
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v2
+      - name: Use Node.js Current
+        uses: actions/setup-node@v2
+        with:
+          node-version: 16.x
+      - name: Restore cached dependencies
+        uses: actions/cache@v2
+        with:
+          path: node_modules
+          key: node-modules-${{ hashFiles('package.json') }}
+      - name: Install dependencies
+        run: npm install
+      - name: Lint code
+        run: npm run lint
+      - name: Build
+        run: npm run build
+      - name: Configure AWS credentials
+        uses: aws-actions/configure-aws-credentials@v1
+        with:
+          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
+          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+          aws-region: ${{ env.AWS_REGION }}
+      - name: Deploy
+        run: npm run deploy

.github/workflows/test.yml

@@ -0,0 +1,20 @@
+name: ci
+on:
+  push:
+    branches:
+      - master
+  pull_request:
+    branches:
+      - '**'
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    name: Test
+    steps:
+      - uses: actions/checkout@v2
+      - uses: actions/setup-node@v2
+        with:
+          node-version: '14'
+      - uses: bahmutov/npm-install@v1
+      - run: npm run test

README.md

@@ -1 +1,40 @@
 # assemble-cars-lambda
+
+The `assemble-cars-lambda` will trigger on writes to a given S3 bucket and assemble partial raw CAR files into complete CAR files.
+
+Once a CAR is written, the lambda function will inspect its completeness, as well as if its wrapping directory as all the CARs of the DAG. If we can traverse all the DAG, the partial CARs will be joined and stored on a `/complete` namespace.
+
+## Assumptions
+
+There are a few assumptions in place and this lambda function will assemble CARs if:
+- S3 Object Metadata has a "Complete" structure 
+- S3 Object has a DagPB encoded root with a known size __acceptable__ (100MB) and the S3 directory for that root CID already has all the DAG chunks
+
+## AWS Setup
+
+This project already comes with a script for build and deploy (which is also integrated with Github Actions). However it needs:
+- Project creation in AWS
+  - It needs a role policy with S3 `s3:GetObject`, `s3:PutObject` and `s3:ListBucket` privileges
+- Secrets setup in Repo secrets for automatic deploy
+- Environment variables setup
+
+## Development and AWS bootstrap
+
+Set environment variables in a `.env.local` file in the root directory of this project:
+
+```env
+AWS_ACCESS_KEY_ID=""
+AWS_SECRET_ACCESS_KEY=""
+AWS_REGION=""
+AWS_BUCKET=""
+NAME="dotstorage-assemble-cars"
+```
+
+```sh
+npm run init
+```
+
+## What's next?
+
+- CBOR
+- What to do with really large CAR files ?

aws/lambda-policy.json

@@ -0,0 +1,28 @@
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Effect": "Allow",
+      "Action": "logs:CreateLogGroup",
+      "Resource": "arn:aws:logs:::*"
+    },
+    {
+      "Effect": "Allow",
+      "Action": [
+        "logs:CreateLogStream",
+        "logs:PutLogEvents"
+      ],
+      "Resource": [
+        "arn:aws:logs:::::*"
+      ]
+    },
+    {
+      "Effect": "Allow",
+      "Action": [
+        "s3:GetObject",
+        "s3:ListBucket"
+      ],
+      "Resource": "arn:aws:s3:::*"
+    }
+  ]
+}

aws/role-trust-policy.json

@@ -0,0 +1,12 @@
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Effect": "Allow",
+      "Principal": {
+        "Service": "lambda.amazonaws.com"
+      },
+      "Action": "sts:AssumeRole"
+    }
+  ]
+}

aws/s3-trigger.json

@@ -0,0 +1,21 @@
+{
+  "LambdaFunctionConfigurations": [
+    {
+      "Id": "assemble-cars-notification",
+      "LambdaFunctionArn": "TODO",
+      "Events": [
+        "s3:ObjectCreated:*"
+      ],
+      "Filter": {
+        "Key": {
+          "FilterRules": [
+            {
+              "Name": "prefix",
+              "Value": "raw"
+            }
+          ]
+        }
+      }
+    }
+  ]
+}