Chalice: Implement layers

[onurmatik]

AWS recently announced "layers" for sharing code and libraries among lambda functions:
https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html

This eases maintenance when sharing custom code and reduces deployment time when using libraries that are not pure Python.

Existing chalicelib and vendor directories can be deployed as layers. Also, a configuration option would be useful for cross-project sharing of resources via layers among Chalice projects.

[jamesls]

Interesting idea on having chalicelib and vendor as layers. What I was thinking was maybe we could have all your dependencies outside of app.py and chalicelib/ in a single layer. That has several benefits:

• You only have to update that layer if your requirements.txt or vendor/ directory change.
• Subsequent app deployments are only your application code, which would significantly speed up deployments if your app code is much smaller than the code for your dependencies.

The cross project sharing I'd have to think about. We've definitely ran into cases where a "conceptual" serverless app consists of multiple chalice apps, and it would be really helpful to be able to share/reuse across multiple chalice apps.

I think the next step here would be to get a more fleshed out proposal/spec for layers support.

[owenbrown]

Thank you both for working on this exciting project!

To weigh in, our team has would love to be able to share code, as a layer, across several chalice applications.

To be specific about our use case, we build chatbots. Different chatbots have different backend functions. But they all use the same "core", which is a cross platform (Actions on Google, Facebook, Web-client) SDK. Layers seemed like a good way to share the core engine with the different agents.

Another use case we have is for analytics. We have many functions that write to AWS RDS. We're a Python shop, and we use SQLAlchemy as an ORM to connect to RDS. Uploading SQLAlchemy with every deploy takes a while. Getting this into a layer would speed our deploys considerably.

If there are better ways to share code across Chalice applications, please message me - owen at starbutter dot com!

[sciurus]

Seperate from the larger effort to convert chalice into a layer, it would be helpful if chalice let you declare existing layers to be loaded by the lambda it deploys. For instance, so you could enable the datadog layer and then send metrics to datadog from your chalice application.

[cedricyau]

+1 for adding Lambda Layer option to config.json

[cedricyau]

One option that might be simpler is to give the option of using layers in place of requirements.txt and vendor/. In this case, the Lambda Layers that have already been uploaded could be referenced in config.json.

Also, in the interim, is there a way we can disable code inspection to allow Layers to be used via aws cli lambda update-function-configuration --layers? That would allow us to use numpy or SQLAlchemy in app.py or chalicelib/ without having to zip and transfer those packages with each deploy.

[cedricyau]

Well, it turns out that if you don't use requirements.txt, the code doesn't get tested. I was able to create a basic app.py that imported pandas and returned a simple stringified dataframe. All I needed to do was use aws lambda update-function-configuration --function-name <function_name> --layers <insert_pandas_layer_arn_here>

[skwashd]

2 different features have been rolled into one ticket because they both use lambda layers. They're different things.

The idea for dependencies in layers as outlined by @jamesls is really neat. That would help speed up deployments amongst other things.

The other is the use case outlined by @sciurus, to allow developers to specify their own layers. I have multiple places where I could use this.

Does it make sense to split these 2 issues out and track them separately?

I'm happy to see what I can do to contribute to either task.

[jarrodb]

I hope this is not hijacking -- is it possible to use the layers in a local development? Right now I'm just downloading the layer package and unzipping it in the project directory (ugly).

[kapilt]

you can reference extant layers in chalice function configuration. i.e. chalice supports layers constructed out of band for lambda functions.

[phillprice]

What's the current status of this is? Is the best practice workaround to unzip a package and remove items that exist in other layers manually then rezip before deployment?

[amarjandu]

@phillprice you can take a look at how this project uses it https://github.com/HumanCellAtlas/data-store/commit/5d006f93cfb1bc1d6ca9998ff620052b2f1179e8

If there is a change between what a current (old) deployment layer uses and what a new deployment uses, there will be a new layer uploaded by extracting the dependencies from chalice package .

:)

[jamesls]

Closing out old issue, chalice now supports automatically generating layers for your app dependencies as well as the ability to add external layers to your lambda function.