Aws-sam-cli: sam build - reuse lambda deployment packages, ideally one-per-manifest

[adilosa]

Describe your idea/feature/enhancement

sam build appears to re-create identical Lambda deployment packages for each of my Python functions. That is, all dependencies and code files are included – even handler files for Lambdas other than the Lambda being built.

This adds increased build time as the same deployment package is built and uploaded over and over.

Proposal

It would be great if sam build could reuse Lambda deployment packages when the environment between Lambda functions is the same. Since only one manifest is supported right now, this should result in one deployment package.

Ideally, multiple manifests could be provided so Lambdas can have different dependencies. Then sam build would produce one deployment package per manifest, rather than re-building an uber deployment package for every lambda.

Things to consider:

1. Will this require any updates to the SAM Spec
   Perhaps, to add a way to specify which manifest a Lambda function should use

[adilosa]

Looks like this can be somewhat mitigated by:

• Creating multiple source folders with different requirements.txt files
• Storing common dependencies in a Lambda Layer so they don't need to be built into every handler (pending #840)

That said, it still seems like one deployment package per handler function could be unnecessary – even though their size can be mitigated with Layers.

[sriram-mv]

Looks like you have captured mitigation stratergies which make sense.

it still seems like one deployment package per handler function could be unnecessary – even though their size can be mitigated with Layers.

Could you explain a bit more on this, this is in essence the deployment artifacts that need to be uploaded for lambda to work.

[adilosa]

Sure, I'm structuring my repo like this:

.
+-- some_lambdas
|   +-- __init__.py   # contains 3 Lambda handler functions
|   +-- requirements.txt
+-- some_lambdas_2
|   +-- __init__.py
|   +-- fourth_handler.py
|   +-- fifth_handler.py
|   +-- requirements.txt
+-- template.yml

Here I have 5 Lambda handler functions in 2 directories. In theory, this requires 2 deployment packages – one for some_lambdas and one for some_lambdas_2 – with the Lambda functions having different entry points.

However, since there are 5 AWS::Serverless::Function resources in my SAM template, sam build will accordingly build 5 deployment packages. Resulting in 3 copies of the some_lambdas environment and 2 copies of the some_lambdas_2 environment.

This causes build time to scale linearly with the number of Lambdas in my template, rather than the minimum number of deployment packages.

As a more concrete example, say I have an Api Gateway and ~25 Lambda functions whose handlers are all in the same folder, and they all require requests. sam build would rebuild and upload this environment 25 times on every build, whereas a user doing this by hand would likely only produce one ZIP.

By using Layers, I can significantly reduce build time by making the "expensive" ZIP with all the dependencies only once. There is still a ZIP uploaded for each handler, but since it only contains the handler code it becomes negligibly small.

[beck3905]

I've run into this issue too. sam build commands are taking longer and longer as I add Lambda functions to my SAM template. I've been looking for ways to only generate a single lambda package, but it doesn't look like its possible unless I want to do the dependency packaging myself and skip sam build altogether. Has there been any movement on this?

[mariaines]

Hi there @sriram-mv, any update on this? We are also running into this problem. Considering skipping sam build as @beck3905 suggested but we'd prefer to avoid that if possible.

[ItayGoren]

I have the same problem. I even tried to upload the lambda's zip to s3 on my own and specify the location at the CodeUrl using Bucket and Key. But sam still scans for dependencies for each lambda...

[beck3905]

I ended up building a Lambda Layer (AWS::Serverless::LayerVersion) with all my dependencies and then I don't need to run sam build at all. But, I do have to package my requirements and code into the Lambda layer ContentUri folder locally and then sam deploy will package and upload the Lambda layer for me. It's a little more than I would like to manage myself (I wish the same aws-sam-cli code that sam build uses to install dependencies into the .aws-sam folder would also work for AWS::Serverless::LayerVersions too. But, at least I only have a single package to be uploaded to S3 and then my Lambda functions are quite small.

[uguracikgozz]

is there any movement or update about that ? It's killing that waiting a small changes causes a huge deployment process..

[mndeveci]

With SAM CLI 1.6.0 release, we added support for de-duplicated builds, which will build same CodeUri for multiple functions once. And since packaging uses md5 of the resources that have been packed, packaging (S3 upload) happens only once for all the functions that uses same CodeUri.

And with the SAM CLI 1.9.0 release, we also added cached and parallel builds which will speed up build command, especially if you are using a layer for your dependencies.

Pelase re-open this issue (or create a new one), if you encounter other problems.

Thanks!