Skip to content

Latest commit

 

History

History
1097 lines (665 loc) · 41.6 KB

API.md

File metadata and controls

1097 lines (665 loc) · 41.6 KB

Turbo Layers for CDK

NPM PyPI Maven Central Go Nuget License

Speed up deployment of Lambda functions by creating dependency layers in AWS instead of locally.

  • ⛓️ Easily separate dependency deployment from Lambda code deployment
  • 🔁 Never re-package dependencies just because of a small code change
  • ☁️ Never download another single dependency package locally again
  • 🏋️ Never upload oversized code packages again
  • 🌎 Edit your code in the browser -- no more "deployment package too large to enable inline code editing"
  • ❌ Uninstall Docker from your laptop and extend your battery life
  • ☕ Take shorter coffee breaks when deploying

Supported Lambda runtimes:

  • 🐍 Python
  • 📜 Node.js
  • 💎 Ruby
  • ☕ Java

Benchmark

Below are synth and deploy times for a simple Python function with PythonFunction compared to Turbo Layers. The benchmark ran three times and the best time were taken for each step.

💤 PythonFunction 🚀 Turbo Layers 💤 5x PythonFunction 🚀 5x Functions w/ Shared Turbo Layer
Initial Synth 1:21 0:06 2:43 0:06
Initial Deploy 1:18 2:05 2:10 2:06
Code Change Synth 0:31 0:06 1:21 0:06
Code Change Deploy 0:49 0:29 1:19 0:36
New Dependency Synth 0:33 0:06 1:30 0:06
New Dependency Deploy 0:52 1:50 1:31 1:50

As you can see, code changes synth much faster and deploy a bit faster too. Dependency changes take longer to deploy, but are assumed to be way less frequent than code changes. The more dependencies your function uses, the better the results will be.

To run the benchmark yourself use:

npm run bundle && npm run benchmark

API

The best way to browse API documentation is on Constructs Hub. It is available in all supported programming languages.

Installation

  1. Confirm you're using CDK v2
  2. Install the appropriate package
    1. Python 
      pip install cloudsnorkel.cdk-turbo-layers
    2. TypeScript or JavaScript 
      npm i @cloudsnorkel/cdk-turbo-layers
    3. Java 
      <dependency>
<groupId>com.cloudsnorkel</groupId>
<artifactId>cdk.turbo-layers</artifactId>
</dependency>
    4. Go 
      go get github.com/CloudSnorkel/cdk-turbo-layers-go/cloudsnorkelcdkturbolayers
    5. .NET 
      dotnet add package CloudSnorkel.Cdk.TurboLayers

Example

 const packager = new PythonDependencyPackager(this, 'Packager', {
    runtime: lambda.Runtime.PYTHON_3_9,
    type: DependencyPackagerType.LAMBDA,
});
new Function(this, 'Function with inline requirements', {
    handler: 'index.handler',
    code: lambda.Code.fromInline('def handler(event, context):\n  import requests'),
    runtime: lambda.Runtime.PYTHON_3_9,
    // this will create a layer from with requests and Scrapy in a Lambda function instead of locally
    layers: [packager.layerFromInline('inline requirements', ['requests', 'Scrapy'])],
});
new Function(this, 'Function with external source and requirements', {
    handler: 'index.handler',
    code: lambda.Code.fromAsset('lambda-src'),
    runtime: lambda.Runtime.PYTHON_3_9,
    // this will read pyproject.toml and poetry.lock and create a layer from the requirements in a Lambda function instead of locally
    layers: [packager.layerFromPoetry('poetry requirements', 'lambda-src')],
});

Older Implementations

API Reference

Constructs

JavaDependencyPackager

  • Implements: aws-cdk-lib.aws_iam.IGrantable, aws-cdk-lib.aws_ec2.IConnectable

Packager for creating Lambda layers for Java dependencies in AWS.

Nothing is done locally so this doesn't require Docker and doesn't upload huge files to S3.

Initializers 

import { JavaDependencyPackager } from '@cloudsnorkel/cdk-turbo-layers'

new JavaDependencyPackager(scope: Construct, id: string, props?: DependencyPackagerProps)
Name Type Description
scope constructs.Construct No description.
id string No description.
props DependencyPackagerProps No description.
scopeRequired
  • Type: constructs.Construct
idRequired
  • Type: string
propsOptional

Methods

Name Description
toString Returns a string representation of this construct.
layerFromMaven Create a layer for dependencies defined in pom.xml installed with Maven.
toString 
public toString(): string

Returns a string representation of this construct.

layerFromMaven 
public layerFromMaven(id: string, path: string, props?: LayerProps): LayerVersion

Create a layer for dependencies defined in pom.xml installed with Maven.

idRequired
  • Type: string
pathRequired
  • Type: string
propsOptional

Static Functions

Name Description
isConstruct Checks if x is a construct.
isConstruct 
import { JavaDependencyPackager } from '@cloudsnorkel/cdk-turbo-layers'

JavaDependencyPackager.isConstruct(x: any)

Checks if x is a construct.

xRequired
  • Type: any

Any object.

Properties

Name Type Description
node constructs.Node The tree node.
connections aws-cdk-lib.aws_ec2.Connections The network connections associated with this resource.
grantPrincipal aws-cdk-lib.aws_iam.IPrincipal The principal to grant permissions to.
nodeRequired 
public readonly node: Node;
  • Type: constructs.Node

The tree node.

connectionsRequired 
public readonly connections: Connections;
  • Type: aws-cdk-lib.aws_ec2.Connections

The network connections associated with this resource.

grantPrincipalRequired 
public readonly grantPrincipal: IPrincipal;
  • Type: aws-cdk-lib.aws_iam.IPrincipal

The principal to grant permissions to.

NodejsDependencyPackager

  • Implements: aws-cdk-lib.aws_iam.IGrantable, aws-cdk-lib.aws_ec2.IConnectable

Packager for creating Lambda layers for Node.js dependencies in AWS. Nothing is done locally so this doesn't require Docker, doesn't download any packages and doesn't upload huge files to S3.

Initializers 

import { NodejsDependencyPackager } from '@cloudsnorkel/cdk-turbo-layers'

new NodejsDependencyPackager(scope: Construct, id: string, props?: DependencyPackagerProps)
Name Type Description
scope constructs.Construct No description.
id string No description.
props DependencyPackagerProps No description.
scopeRequired
  • Type: constructs.Construct
idRequired
  • Type: string
propsOptional

Methods

Name Description
toString Returns a string representation of this construct.
layerFromInline Create a layer for dependencies passed as an argument and installed with npm.
layerFromPackageJson Create a layer for dependencies defined in package.json and (optionally) package-lock.json and installed with npm.
layerFromYarn Create a layer for dependencies defined in package.json and yarn.lock and installed with yarn.
toString 
public toString(): string

Returns a string representation of this construct.

layerFromInline 
public layerFromInline(id: string, libraries: string[], props?: LayerProps): LayerVersion

Create a layer for dependencies passed as an argument and installed with npm.

idRequired
  • Type: string
librariesRequired
  • Type: string[]
propsOptional
layerFromPackageJson 
public layerFromPackageJson(id: string, path: string, props?: LayerProps): LayerVersion

Create a layer for dependencies defined in package.json and (optionally) package-lock.json and installed with npm.

idRequired
  • Type: string
pathRequired
  • Type: string
propsOptional
layerFromYarn 
public layerFromYarn(id: string, path: string, props?: LayerProps): LayerVersion

Create a layer for dependencies defined in package.json and yarn.lock and installed with yarn.

idRequired
  • Type: string
pathRequired
  • Type: string
propsOptional

Static Functions

Name Description
isConstruct Checks if x is a construct.
isConstruct 
import { NodejsDependencyPackager } from '@cloudsnorkel/cdk-turbo-layers'

NodejsDependencyPackager.isConstruct(x: any)

Checks if x is a construct.

xRequired
  • Type: any

Any object.

Properties

Name Type Description
node constructs.Node The tree node.
connections aws-cdk-lib.aws_ec2.Connections The network connections associated with this resource.
grantPrincipal aws-cdk-lib.aws_iam.IPrincipal The principal to grant permissions to.
nodeRequired 
public readonly node: Node;
  • Type: constructs.Node

The tree node.

connectionsRequired 
public readonly connections: Connections;
  • Type: aws-cdk-lib.aws_ec2.Connections

The network connections associated with this resource.

grantPrincipalRequired 
public readonly grantPrincipal: IPrincipal;
  • Type: aws-cdk-lib.aws_iam.IPrincipal

The principal to grant permissions to.

PythonDependencyPackager

  • Implements: aws-cdk-lib.aws_iam.IGrantable, aws-cdk-lib.aws_ec2.IConnectable

Packager for creating Lambda layers for Python dependencies in AWS.

Nothing is done locally so this doesn't require Docker, doesn't download any packages and doesn't upload huge files to S3.

Initializers 

import { PythonDependencyPackager } from '@cloudsnorkel/cdk-turbo-layers'

new PythonDependencyPackager(scope: Construct, id: string, props?: DependencyPackagerProps)
Name Type Description
scope constructs.Construct No description.
id string No description.
props DependencyPackagerProps No description.
scopeRequired
  • Type: constructs.Construct
idRequired
  • Type: string
propsOptional

Methods

Name Description
toString Returns a string representation of this construct.
layerFromInline Create a layer for dependencies passed as an argument and installed with pip.
layerFromPipenv Create a layer for dependencies defined in Pipfile and (optionally) Pipfile.lock and installed with pipenv.
layerFromPoetry Create a layer for dependencies defined in pyproject.toml and (optionally) poetry.lock and installed with poetry.
layerFromRequirementsTxt Create a layer for dependencies defined in requirements.txt and installed with pip.
toString 
public toString(): string

Returns a string representation of this construct.

layerFromInline 
public layerFromInline(id: string, requirements: string[], props?: LayerProps): LayerVersion

Create a layer for dependencies passed as an argument and installed with pip.

idRequired
  • Type: string
requirementsRequired
  • Type: string[]
propsOptional
layerFromPipenv 
public layerFromPipenv(id: string, path: string, props?: LayerProps): LayerVersion

Create a layer for dependencies defined in Pipfile and (optionally) Pipfile.lock and installed with pipenv.

idRequired
  • Type: string
pathRequired
  • Type: string
propsOptional
layerFromPoetry 
public layerFromPoetry(id: string, path: string, props?: LayerProps): LayerVersion

Create a layer for dependencies defined in pyproject.toml and (optionally) poetry.lock and installed with poetry.

idRequired
  • Type: string
pathRequired
  • Type: string
propsOptional
layerFromRequirementsTxt 
public layerFromRequirementsTxt(id: string, path: string, props?: LayerProps): LayerVersion

Create a layer for dependencies defined in requirements.txt and installed with pip.

idRequired
  • Type: string
pathRequired
  • Type: string
propsOptional

Static Functions

Name Description
isConstruct Checks if x is a construct.
isConstruct 
import { PythonDependencyPackager } from '@cloudsnorkel/cdk-turbo-layers'

PythonDependencyPackager.isConstruct(x: any)

Checks if x is a construct.

xRequired
  • Type: any

Any object.

Properties

Name Type Description
node constructs.Node The tree node.
connections aws-cdk-lib.aws_ec2.Connections The network connections associated with this resource.
grantPrincipal aws-cdk-lib.aws_iam.IPrincipal The principal to grant permissions to.
nodeRequired 
public readonly node: Node;
  • Type: constructs.Node

The tree node.

connectionsRequired 
public readonly connections: Connections;
  • Type: aws-cdk-lib.aws_ec2.Connections

The network connections associated with this resource.

grantPrincipalRequired 
public readonly grantPrincipal: IPrincipal;
  • Type: aws-cdk-lib.aws_iam.IPrincipal

The principal to grant permissions to.

RubyDependencyPackager

  • Implements: aws-cdk-lib.aws_iam.IGrantable, aws-cdk-lib.aws_ec2.IConnectable

Packager for creating Lambda layers for Ruby dependencies in AWS.

Nothing is done locally so this doesn't require Docker, doesn't download any packages and doesn't upload huge files to S3.

Initializers 

import { RubyDependencyPackager } from '@cloudsnorkel/cdk-turbo-layers'

new RubyDependencyPackager(scope: Construct, id: string, props?: DependencyPackagerProps)
Name Type Description
scope constructs.Construct No description.
id string No description.
props DependencyPackagerProps No description.
scopeRequired
  • Type: constructs.Construct
idRequired
  • Type: string
propsOptional

Methods

Name Description
toString Returns a string representation of this construct.
layerFromBundler Create a layer for dependencies defined in Gemfile and (optionally) Gemfile.lock and installed with Bundler.
toString 
public toString(): string

Returns a string representation of this construct.

layerFromBundler 
public layerFromBundler(id: string, path: string, props?: LayerProps): LayerVersion

Create a layer for dependencies defined in Gemfile and (optionally) Gemfile.lock and installed with Bundler.

idRequired
  • Type: string
pathRequired
  • Type: string
propsOptional

Static Functions

Name Description
isConstruct Checks if x is a construct.
isConstruct 
import { RubyDependencyPackager } from '@cloudsnorkel/cdk-turbo-layers'

RubyDependencyPackager.isConstruct(x: any)

Checks if x is a construct.

xRequired
  • Type: any

Any object.

Properties

Name Type Description
node constructs.Node The tree node.
connections aws-cdk-lib.aws_ec2.Connections The network connections associated with this resource.
grantPrincipal aws-cdk-lib.aws_iam.IPrincipal The principal to grant permissions to.
nodeRequired 
public readonly node: Node;
  • Type: constructs.Node

The tree node.

connectionsRequired 
public readonly connections: Connections;
  • Type: aws-cdk-lib.aws_ec2.Connections

The network connections associated with this resource.

grantPrincipalRequired 
public readonly grantPrincipal: IPrincipal;
  • Type: aws-cdk-lib.aws_iam.IPrincipal

The principal to grant permissions to.

Structs

DependencyPackagerProps

Initializer 

import { DependencyPackagerProps } from '@cloudsnorkel/cdk-turbo-layers'

const dependencyPackagerProps: DependencyPackagerProps = { ... }

Properties

Name Type Description
architecture aws-cdk-lib.aws_lambda.Architecture Target Lambda architecture.
logRemovalPolicy aws-cdk-lib.RemovalPolicy Removal policy for logs of image builds.
logRetention aws-cdk-lib.aws_logs.RetentionDays The number of days log events are kept in CloudWatch Logs.
preinstallCommands string[] Additional commands to run before installing packages.
runtime aws-cdk-lib.aws_lambda.Runtime Target Lambda runtime.
subnetSelection aws-cdk-lib.aws_ec2.SubnetSelection VPC subnets used for packager.
type DependencyPackagerType Type of dependency packager.
vpc aws-cdk-lib.aws_ec2.IVpc VPC used for packager.
architectureOptional 
public readonly architecture: Architecture;
  • Type: aws-cdk-lib.aws_lambda.Architecture

Target Lambda architecture.

Packages will be installed for this architecture so make sure it fits your Lambda functions.

logRemovalPolicyOptional 
public readonly logRemovalPolicy: RemovalPolicy;
  • Type: aws-cdk-lib.RemovalPolicy
  • Default: RemovalPolicy.DESTROY

Removal policy for logs of image builds.

If deployment fails on the custom resource, try setting this to RemovalPolicy.RETAIN. This way the CodeBuild logs can still be viewed, and you can see why the build failed.

We try to not leave anything behind when removed. But sometimes a log staying behind is useful.

logRetentionOptional 
public readonly logRetention: RetentionDays;
  • Type: aws-cdk-lib.aws_logs.RetentionDays
  • Default: logs.RetentionDays.ONE_MONTH

The number of days log events are kept in CloudWatch Logs.

When updating this property, unsetting it doesn't remove the log retention policy. To remove the retention policy, set the value to INFINITE.

preinstallCommandsOptional 
public readonly preinstallCommands: string[];
  • Type: string[]
  • Default: []

Additional commands to run before installing packages.

Use this to authenticate your package repositories like CodeArtifact.

runtimeOptional 
public readonly runtime: Runtime;
  • Type: aws-cdk-lib.aws_lambda.Runtime

Target Lambda runtime.

Packages will be installed for this runtime so make sure it fits your Lambda functions.

subnetSelectionOptional 
public readonly subnetSelection: SubnetSelection;
  • Type: aws-cdk-lib.aws_ec2.SubnetSelection
  • Default: default subnets, if VPC is used

VPC subnets used for packager.

typeOptional 
public readonly type: DependencyPackagerType;

Type of dependency packager.

Use Lambda for speed and CodeBuild for complex dependencies that require building native extensions.

vpcOptional 
public readonly vpc: IVpc;
  • Type: aws-cdk-lib.aws_ec2.IVpc
  • Default: no VPC

VPC used for packager.

Use this if your package repositories are only available from within a VPC.

LayerProps

Initializer 

import { LayerProps } from '@cloudsnorkel/cdk-turbo-layers'

const layerProps: LayerProps = { ... }

Properties

Name Type Description
alwaysRebuild boolean Always rebuild the layer, even when the dependencies definition files haven't changed.
alwaysRebuildOptional 
public readonly alwaysRebuild: boolean;
  • Type: boolean
  • Default: false

Always rebuild the layer, even when the dependencies definition files haven't changed.

Enums

DependencyPackagerType

Type of dependency packager.

This affects timeouts and capabilities of the packager.

Members

Name Description
LAMBDA Use Lambda function to package dependencies.
CODEBUILD Use CodeBuild to package dependencies.
LAMBDA

Use Lambda function to package dependencies.

It is much faster than the alternative, but limited to 15 minutes and can't build native extensions.

CODEBUILD

Use CodeBuild to package dependencies.

It is capable of everything your local machine can do, but takes a little longer to startup.