Chalice: Support to upload custom binaries

[fawzyj]

Add support to upload binaries to the lambda function.
This may be a custom command or at least a detailed documentation on how to achieve that.

Our application uses external libraries and binaries not instalable through pip.

[JamieCressey]

I guess it depends on your use case, but custom libraries will inherently be supported once #21 is implemented, and depending on what you mean by binaries, S3 may be a better storage location for those.

[chrismccracken]

I'd really like to see resolution on #21 along with this. https://github.com/Miserlou/lambda-packages has some great Lambda-compatible libs ready to use, in particular I'm itching to use psycopg2 with an RDS datastore in Chalice.

[jamesls]

Does anyone know if the many linux wheel type works on Lambda? Haven't had a chance to try it out, but it would be nice if we could download the appropriate wheels when building your deployment package. That way we'd support any project that created linux wheels.

[stannie]

I'm also looking for a way to add binaries, e.g. I need PyNaCl, which relies on libsodium, a portable C library. In another Lambda I created by hand, I added it with pip install pynacl (which compiles libsodium) and then added all files in ./env/lib/python2.7/site-packages/ to the bundle I deploy on Lambda. Or is this already supported?

[jamesls]

@stannie that works (with a few caveats), but it's not officially documented as a way to customize what goes in the zip file. Perhaps providing a more "official" way to do this would be helpful. Here's what I propose:

Add the ability for users to provide scripts that are run as part of the deploy process. The main benefit here is that you can explicitly capture how you're modifying the venv dir:

#!/bin/bash
# This would be hooks/building-zip
# It would pass in the zip dir as an argument:

ZIP_DIR="$1"
MY_PREBUILT_BINARIES="linux-binaries"  # These could be dynamically fetched/generated

cp -r "$MY_PREBUILT_BINARIES/*"  "$ZIP_DIR/"

# If you wanted to, for some reason, you could also remove things from the deployed zip
rm $ZIP_DIR/unnecessary-file.py

Rationale

• Clearly separates code that's part of your app that's maintained by you (i.e chalicelib/*) vs. code that's 3rd party dependencies that need special handling (they're not on pip, or you need a library that's built in a specific way, etc).
• Still allows users to git clone your app and have chalice deploy work.

Another option I've considered is just having an "overrides" or maybe "vendor" directory, that at deploy time will get merged onto the venv/ directory. That way you still cleanly separate 3rd party vendored code from your own code. The main downside is that the content is static so you'd have to pre-generate/pre-download content and place it in vendor/ before running chalice deploy.

[stannie]

Ability for users to provide scripts that are run as part of the deploy process would work for me.

Alternatively you could pass the zip file, and with a zip -u9 bundle.zip $MY_PREBUILT_BINARIES/* the binaries, built on an EC2 instance, will be added.

[jamesls]

Some of the feedback I've had so far pointed out that the most common use case is taking prebuilt binaries and adding them to the deployment package. Modifying/removing data from the package is less common.

Given that, I think it makes more sense to just support a directory that we'll automatically overlay into the deployment package. I've implemented that here: https://github.com/awslabs/chalice/issues/182

[bitblit]

Just adding my upvote on this one as well - Can't wait to see #182 merged in and released so I can get rid of my weirdo hacks for adding Lambda-Compatible numpy and PyCrypt!

[jamesls]

182 has been merged in https://github.com/awslabs/chalice/commit/7329b27fbe2e1ee5019cc2845b9329e6ad857c67

[baileyspace]

I'm trying to install moviepy and numpy is a dependency. With the addition of putting the pre-compiled numpy package in the vendor/ directory are there any other steps I need to perform in order to get numpy/moviepy running? I'm still getting "Unable to import module 'app'" after deploy.

[baileyspace]

Actually, I am getting an error in CloudWatch:

Unable to import module 'app': Importing the multiarray numpy extension module failed. Most likely you are trying to import a failed build of numpy. If you're working with a numpy git repo, trygit clean -xdf(removes all files not under version control). Otherwise reinstall numpy.

I've gotten the same error message from an install that I created on an ec2 instance as well as from the version in this repo https://github.com/Miserlou/lambda-packages/tree/master/lambda_packages.