Aws-cdk: Lambda "build" assets

[eladb]

We would like to leverage the aws-lambda-builders project (part of SAM CLI) in order to allow CDK users to build AWS Lambda bundles.

Requirement (API sketch):

new lambda.Function(this, 'MyFunction', {
  code: lambda.Code.build('/path/to/lambda/handler'),
  runtime: lambda.Runtime.Xxx,
  handler: 'foo.bar'
});

Ideally, this is all the user should need to specify. The lambda.Code.build class should be able to deduce most of the information needed in order to tell the toolkit to invoke aws-lambda-builders via it's JSON RPC protocol when assets are prepared (similar to any other asset).

Notes:

• __Installation experience__: it might okay if users would have to manually install aws-lambda-builders through pip install. We should consider if the toolkit can do this automatically upon first use?
• SAM CLI uses heuristics to determine various options. The CDK should be able to use similar heuristics.

[eladb]

Copy @jfuss

[jfuss]

Additional context:

• The aws-lambda-builders library only builds locally. For functions that need native binaries/ dependencies, the build process will need to be run a 'lambda like environment'. SAM CLI uses docker-lambda's build images for this (SAM CLI code for building in an image and the relevant Container class).
• aws-lambda-builders is not yet on a minor version (currently 0.0.5 as of Dec 20, 2018). We do have a protocol version and you should consider doing protocol version checks to ensure compatibility and/or pin to a certain library version.
• Python2.7 is finally dying Jan 1, 2020. It is likely that we will stop supporting Py2.7 at some point before this. What this means is pip install might not be enough. On most systems, pip defaults to the system python which still tends to be Python2.7 (which makes me super sad). Once Py2.7 support is dropped, customers may not be able to install and would require them to download/install Python3.6 or greater. The moral of this is, aws-lambda-builders might need an installer that is easy to run which will setup the library correctly and lowering the bar for entry on installing. This could open the door for an easier installation with CDK as well.

[sam-goodwin]

Working with aws-lambda-builders to extract the logic within SAM that maps a function runtime to a build workflow, e.g. taking a java8 function and determining if it's a maven or gradle project by looking for a pom.xml or build.gradle file. This wil enable us to provide the following experience:

const myFunction = new lambda.JavaFunction(stack, 'MyJavaFunction', {
  projectPath: './lambda'
});

[sam-goodwin]

I am currently using the asset's bind function as a hook to inspect the runtime property of Function to determine which lambda-builder workflow to run. It enables a familiar experience, only requiring you to use lambda.Code.build instead of asset or file:

const gradle = new lambda.Function(stack, 'Gradle', {
  code: lambda.Code.build('./gradle-project'),
  runtime: lambda.Runtime.Java8,
  handler: 'com.example.Main:handle'
});

It's simple, but lambda.Code.bind takes a Construct instead of a Function, so I'm not sure if I can safely assume I'll always have access to the function's runtime . Is there a use-case here to pass anything other than a Function to bind?

I like the idea of a type-safe JvmFunction because it enables the injection of hooks specific to an ecosystem, but I'm wondering how we should draw the lines between the CDK's responsibility and that of tools like aws-lambda-builders. I have a prototype integrating lambda-builders as an Asset, where I have duplicated logic from SAM to detects things like gradle vs maven for java. The following PR proposes centralizing that logic in a common tool for both CDK and SAM, but we could also solve it with an explicit parameter:

const myFunction = new lambda.JvmFunction(stack, 'MyJavaFunction', {
  projectPath: './lambda',
  dependencyManager: 'maven'
});

Which begs the question: what information should be injectable as props and what information should be inferred by the tool?

[eladb]

Is there a use-case here to pass anything other than a Function to bind?

Yes, for example, Layers also use the same Code mechanism. Maybe we should define an interface :-)

Which begs the question: what information should be injectable as props and what information should be inferred by the tool?

I think the default behavior should be to infer as much as possible, but I really like the option of letting users specify many of the project settings here (like dependencies etc) if they wish.

[sam-goodwin]

Layers changes things up a bit because they aren't specific to a single runtime: compatibleRuntimes?: Runtime[] with default All.

I see the following options for proceeding:

1. Pass the runtime/build information to lambda.Code.build so it doesn't need to inspect the Layer/Function. This would mean we are redundantly passing runtime information to both the Layer/Function and the Code asset.

const fn = new lambda.Function(this, 'F', {
  runtime: lambda.Runtime.Java8,
  code: lambda.Code.build({
    path: './path',
    language: lambda.Runtime.Java8,
    // maybe we only need to pass the family, not the specific language and infer the rest?
    // language: lambda.RuntimeFamily.Java,
  })
});
const fn = new lambda.Layer(this, 'F', {
  code: lambda.Code.build({
    path: './path',
    language: lambda.Runtime.Java8
  })
});

1. Only support builds in higher-level constructs such as PythonFunction or JvmLayer.

const fn = new lambda.PythonFunction(this, 'F', {
  path: './path',
  version: '3.7'
});
const layer = new lambda.NodeLayer(this, 'L', {
  path: './path',
});

1. Ditch lambda.Code.build and instead have various options such as JvmCode, NodeCode:

const fn = new lambda.Function(this, 'F', {
  runtime: lambda.Runtime.Java8,
  code: new JvmCode('./path')
});
const layer = new lambda.Layer(this, 'L', {
  code: new NodeCode('./path'),
  compatibleRuntimes: [
    lambda.Runtime.NodeJs,
    lambda.Runtime.NodeJs810,
    // etc.
  ]
});

// the static method could still be useful for discoverability and consistency.
// again, requires redundant 'runtime' information to be passed
const fn = new lambda.Function(this, 'F', {
  runtime: lambda.Runtime.Java8,
  code: lambda.Code.java('./path')
});

I'm leaning towards a layered approach incorporating option 2. and 3. - provide high-level Function constructs for each target we support, built on top of a set of Code classes so developers can drop down and customize where they see fit.

[zoonman]

Hey guys, is there any progress on the issue for typescript version?

[andreimcristof]

Hi there, is there any news on this feature?
I wrote all my provisioning code with Typescript, and just found out that the packaging of the lambdas must be done manually - I thought that there would be some equivalent of "sam build".
Is there any way to actually invoke "sam build" on the synthesized cloudformation template, so that sam cli takes over the packaging?

Thank you

[nija-at]

Unfortunately, we don't yet have a plan or update on this feature at this time.

@andreimcristof - this issue is only focusing on using the lambda builders tooling, i.e., equivalent of sam build. Packaging and deployment would still be handled by the CDK.

If you're only looking to invoke sam build on the synthesized template (found in cdk.out/ folder), this should be possible by passing in the right values to the --base-dir and --template parameters of the sam build command.
At this point, it should be possible to run sam package followed by sam deploy on the output. However, it would not be possible to bring the generated template and build directory to work with cdk deploy.

[eladb]

I think at a minimum we should provide guidance on how to use the SAM build capabilities as a pre synthesis step in order to produce lambda bundle zip files that can later be referenced as file assets for Lambda code.

[andreimcristof]

...At this point, it should be possible to run sam package followed by sam deploy on the output. However, it would not be possible to bring the generated template and build directory to work with cdk deploy.

@nija-at but can sam build and the cdk not play nice with eachother? Meaning: before initializing the cdk infra, I trigger a sam build, and customise the path of the cdk lambdas to read from the sam build output? Can that not work? because, if I understood correctly, all that the cdk needs for the functions, is a path where to read the zips from:

return new lambda.Function(scope, `${funcHandler}_lambda`, {
    runtime: lambda.Runtime.NODEJS_10_X,
    handler: funcHandler,
    code: lambda.Code.fromAsset(path.join(__dirname, 'crud')),
  } as lambda.FunctionProps);

... so then, I just configure the .fromAsset to read from the sam build output?
ah, exactly what @eladb suggested. No, wait. I'm missing something. @eladb - the sam build cannot be pre-synthesis, as it needs the template. the synth _generates_ the template. The way I underrstand it, this would happen like this:

• run synthesis so the Cloud Formation template is generated
• run sam build by specifying template path
• add a step in the script to copy asset functions from sam build output into the cdk.out folder -> ? is this correct?
• run cdk deploy.

[coderbyheart]

I had the same problem and solved it by having a separate stack for storing the lambda archives on S3, and before the stack is deployed building and uploading the lambdas so they can be referenced in CDK from the bucket: https://coderbyheart.com/how-i-package-typescript-lambdas-for-aws/

[eladb]

Addressed by #5532 and #9182

[polothy]

@eladb are there plans to make a aws-lambda-go or aws-lambda-golang package? Maybe I'm supposed to use Bundling Asset Code and its easy enough for Go?

[eladb]

@eladb are there plans to make a aws-lambda-go or aws-lambda-golang package? Maybe I'm supposed to use Bundling Asset Code and its easy enough for Go?

No concrete plans but more than happy to take contributions! Bundling is the right way.

[polothy]

Thanks for the reply! I might just do that, though it'll be a while before I have time :)