Aws-cdk: No example for deploying a python lambda with external libraries

[kjpgit]

I am trying to deploy some python lambdas that have dependencies like "requests", "boto3", "mysql-connector", etc.

I don't see any examples that add runtime libraries to requirements.txt, nor do I see a way to have the CDK package these during build / deploy time. Is this within the scope of the CDK?

[fitzee]

Have you tried using lambda.Code.fromAsset(<directory>, <handler-name>) as the value for your function's FunctionProps code property?

<directory> is a local directory relative to the CDK project and would contain all the files required by your function, e.g. app.py, requirements.txt etc. These will be bundled and uploaded to S3.

[kjpgit]

$ ls cdk.out/asset.d0d655f9cdd3f8e0326bb37d7f16d0718590270ccdf81c275143ba521a68f92e/ -l
total 8
-rw-rw-r-- 1 ubuntu ubuntu 69 Aug 15 14:28 handler.py
-rw-rw-r-- 1 ubuntu ubuntu 20 Aug 15 14:28 requirements.txt

That's not what I want.
I found the example python project (lambda-cron) very confusing because if I see a requirements.txt in the root of a project, I expect that to be the run-time dependencies of the code business logic -- not that of the deploy tool. It's made even more confusing by having pytest in the example requirements.txt.

I understand it's not the CDKs job to fix the python build process, since a standard process doesn't even exist and that's python's fault.

But if the CDK is just a cloudformation generator, and not actually "building" python projects for Lambda (e.g. downloading libraries from requirements.txt), then the examples should probably put all the CDK stuff in a deploy/ subdir and leave the normal python project structure unchanged, so people aren't totally confused.

[pepastach]

Here's a custom AssetCode implementation that I use to package and deploy lambda functions with dependencies: https://gitlab.com/josef.stach/aws-cdk-lambda-asset/

[NGL321]

Hey @kjpgit,

I was wondering if the package @pepastach commented was helpful to you in this?

If it was not, I can look further into your issue. My initial impression tells me best bet is to follow the lambda standard of adding the handler and all 3rd party libraries to a .zip file and then linking that file.
If that does not work this might warrant a new Feature Request issue.

Let me know what you think! 😸

[kjpgit]

So the point of this ticket was that this example: https://github.com/aws-samples/aws-cdk-examples/tree/master/python/lambda-cron/ , which was the only example I found for cdk+python+lambda, was very unhelpful for me to learn what the CDK does or what its scope is.

It would be much more clear if there was a subdir for the lambda handler code, which contains a requirements.txt for the runtime dependencies (like boto or requests; best practice per AWS is to deploy your own version of boto and not rely on the lambda platform copy), and another subdir for the deploy / cdk code, which would have a different requirements.txt for the build/deploy dependencies, e.g. CDK.

This separation would ensure people understand that the CDK python modules and venv are not required or used during runtime. And best practice is to not package unnecessary libraries for lambda, it just makes your cold start slower.

Runtime dependencies are different from build time dependencies. That example just has a single, combined "dependencies" file which leaves the reader mystified as to its purpose.

[alukach]

@kjpgit @NGL321 I agree with the theme of this issue, it's not clear how to manage external Python dependencies with cdk. I also agree that build dependencies and application dependencies are often kept separate. Can we please re-open this and figure out what the best way is to:

1) Handle external dependencies (i.e. dependencies specified in either requirements.txt or setup.py that are installed within a virtual environment somewhere on my machine)
2) How we can best educate CDK Python users of the best practice approach

Thanks

[alukach]

A simple example that has a handler.py that calls and returns request.get("https://github.com").body would be helpful.

[garnaat]

I agree that we are lacking a good example of a more realistic Lambda which includes dependencies. I think it would be best to create an issue in the examples repo pointing to this thread. I will work on a better example and update that new issue. Then we can iterate on that till we have an approach that feels right.

[kjpgit]

I wrote an example here: https://github.com/kjpgit/techdemo/tree/master/cdk_python_lambda
That's based on our real-world usage, on 3 projects.

[JamieMcKernanKaizen]

Thanks @kjpgit

I figured it out after posting a question on StackOverflow.

[esteban-uo]

I just published a simple example to deal with it, hope it helps someone else https://github.com/esteban-uo/aws-cdk-python-application-dependencies (based on https://stackoverflow.com/a/61248003)

[dardanbekteshi]

There is now an official CDK module for handling the Lambda requirements.txt by automatically installing them: https://docs.aws.amazon.com/cdk/api/latest/docs/aws-lambda-python-readme.html

[JamieMcKernanKaizen]

There is now an official CDK module for handling the Lambda requirements.txt by automatically installing them: https://docs.aws.amazon.com/cdk/api/latest/docs/aws-lambda-python-readme.html

This looks really promising! Thanks for sharing.

[esteban-uo]

Unfortunately relies on docker, but thx!

[JamieMcKernanKaizen]

@esteban-uo

Unfortunately relies on docker, but thx!

I think it's required here...

If requirements.txt or Pipfile exists at the entry path, the construct will handle installing all required modules in a Lambda compatible Docker container according to the runtime.

But later down, I don't know if it will need it for lambda layers...

You may create a python-based lambda layer with PythonLayerVersion. If PythonLayerVersion detects a requirements.txt or Pipfile at the entry path, then PythonLayerVersion will include the dependencies inline with your code in the layer.

[velles]

I found this one - https://docs.aws.amazon.com/cdk/api/latest/docs/aws-lambda-readme.html#bundling-asset-code helpful. It does rely on Docker, but otherwise, it's pretty clean for using the requirement manifest.