Chalice: Unable to deploy to aws with 3rd party (tarball) dependencies

[malikasinger1]

I'm an expert node.js developer, have worked with serverless framework and firebase functions
I recently started working with Chalice framework, it is nice and easy to get start but something is wrong with dependencies, I'm still unable to deploy it with dependencies

it all starts with when I made a hello world program and run locally with chalice local command and then deploy on aws with chalice deploy it worked quite fine
then i tried to include pyrebase(https://github.com/thisbejim/Pyrebase#database) a python library for firebase

code is here for reference:

from chalice import Chalice
app = Chalice(app_name='abc')

import pyrebase
# api keys are replaced with xxxxx
config = {
    "apiKey": "xxxxxxxxxxx",
    "authDomain": "xxxxxx.firebaseapp.com",
    "databaseURL": "https://xxx.firebaseio.com",
    "storageBucket": "xxxxx.appspot.com",
}
firebase = pyrebase.initialize_app(config).database()

@app.route('/webhook', methods=['POST'])
def myFunction():
    # This is the JSON body which is sent in POST request.
    jsonBody = app.current_request.json_body

    if jsonBody.get('result').get('action') == 'wellcome':
        print('wellcome action detected')

        return {"speech": "Wellcome to the bot, say book hotel to book it any time"}

    elif jsonBody.get('result').get('action') == 'bookHotel':
        print('hotel booking action detected')

        users = firebase.child("users").get()
        print(users.val())

        data = {
            "name": jsonBody.get('result').get('parameters').get('number'),
            "color": jsonBody.get('result').get('parameters').get('color')
        }
        print("dictionary made ", data)

        results = firebase.child("users").push(data)
        print("result: ")
        print(results)
        return{"speech": " hotel booking done"}
    else:
        print('No Action detected')
        return{"speech": "No Action Detected"}

it runs locally very fine, but when i deployed it on aws (after doing pip freeze > requirements.txt) it rejected with a message 'charmap' codec can't decode byte 0x81 in position 191: character maps to <undefined>

then i learnt from issue #491 that doing pip freeze may end with this problem, and ofcourse my environment was loaded with alot of things - i didnt understand why because i deployed flask code on heroku with same environment loaded with lots of extra things not related to my flask code but it didnt throw any error and code was running correctly

Then i created a new environment of python 3.6 with nothing already install (using anaconda navigator since i'm using anaconda distribution of python) just to make sure nothing extra is installed in the new environment i do pip freeze just after creating environment and it was only two:

certifi==2016.2.28
wincertstore==0.2

i memorized these two for removal in case of any problem

then i installed the pyrebase in the environment

(C:\Users\AdMin\Anaconda3\envs\new_env) E:\Projects\class project\chalice-apiai-webhook>pip install pyrebase
Collecting pyrebase
  Using cached Pyrebase-3.0.27-py3-none-any.whl
Collecting pycryptodome==3.4.3 (from pyrebase)
  Using cached pycryptodome-3.4.3-cp36-cp36m-win_amd64.whl
Collecting python-jwt==2.0.1 (from pyrebase)
  Using cached python_jwt-2.0.1-py2.py3-none-any.whl
Collecting gcloud==0.17.0 (from pyrebase)
Collecting requests==2.11.1 (from pyrebase)
  Using cached requests-2.11.1-py2.py3-none-any.whl
Collecting requests-toolbelt==0.7.0 (from pyrebase)
  Using cached requests_toolbelt-0.7.0-py2.py3-none-any.whl
Collecting oauth2client==3.0.0 (from pyrebase)
Collecting jws>=0.1.3 (from python-jwt==2.0.1->pyrebase)
Collecting httplib2>=0.9.1 (from gcloud==0.17.0->pyrebase)
Collecting googleapis-common-protos (from gcloud==0.17.0->pyrebase)
Collecting six (from gcloud==0.17.0->pyrebase)
  Using cached six-1.11.0-py2.py3-none-any.whl
Collecting protobuf!=3.0.0.b2.post1,>=3.0.0b2 (from gcloud==0.17.0->pyrebase)
  Using cached protobuf-3.4.0-py2.py3-none-any.whl
Collecting pyasn1>=0.1.7 (from oauth2client==3.0.0->pyrebase)
  Using cached pyasn1-0.3.5-py2.py3-none-any.whl
Collecting rsa>=3.1.4 (from oauth2client==3.0.0->pyrebase)
  Using cached rsa-3.4.2-py2.py3-none-any.whl
Collecting pyasn1-modules>=0.0.5 (from oauth2client==3.0.0->pyrebase)
  Using cached pyasn1_modules-0.1.4-py2.py3-none-any.whl
Requirement already satisfied: setuptools in c:\users\admin\anaconda3\envs\new_env\lib\site-packages (from protobuf!=3.0.0.b2.post1,>=3.0.0b2->gcloud==0.17.0->pyrebase)
Installing collected packages: pycryptodome, jws, python-jwt, pyasn1, rsa, six, httplib2, pyasn1-modules, oauth2client, protobuf, googleapis-common-protos, gcloud, requests, requests-toolbelt, pyrebase
Successfully installed gcloud-0.17.0 googleapis-common-protos-1.5.2 httplib2-0.10.3 jws-0.1.3 oauth2client-3.0.0 protobuf-3.4.0 pyasn1-0.3.5 pyasn1-modules-0.1.4 pycryptodome-3.4.3 pyrebase-3.0.27 python-jwt-2.0.1 requests-2.11.1 requests-toolbelt-0.7.0 rsa-3.4.2 six-1.11.0

(C:\Users\AdMin\Anaconda3\envs\new_env) E:\Projects\class project\chalice-apiai-webhook>

and again pip freeze > requirement.txt
then requirements.txt become:

certifi==2016.2.28
gcloud==0.17.0
googleapis-common-protos==1.5.2
httplib2==0.10.3
jws==0.1.3
oauth2client==3.0.0
protobuf==3.4.0
pyasn1==0.3.5
pyasn1-modules==0.1.4
pycryptodome==3.4.3
Pyrebase==3.0.27
python-jwt==2.0.1
requests==2.11.1
requests-toolbelt==0.7.0
rsa==3.4.2
six==1.11.0
wincertstore==0.2

then i deploy again using chalice deploy:

Updating lambda function...
Creating deployment package.

Could not install dependencies:
googleapis-common-protos==1.5.2
pycryptodome==3.4.3
You will have to build these yourself and vendor them in
the chalice vendor folder.

Your deployment will continue but may not work correctly
if missing dependencies are not present. For more information:
http://chalice.readthedocs.io/en/latest/topics/packaging.html

Updating IAM policy.
Sending changes to lambda.
API Gateway rest API already found.
Deploying to: api
https://p5s68hp275.execute-api.us-east-1.amazonaws.com/api/

when i call the url https://p5s68hp275.execute-api.us-east-1.amazonaws.com/api/webhook via postman tool it says: {"message": "Internal server error"}

when i check logs it says:
2017-09-19 15:13:46.966000 577087 module initialization error: module 'pyrebase' has no attribute 'initialize_app'
2017-09-19 15:13:46.966000 577087 module initialization error: module 'pyrebase' has no attribute 'initialize_app'
2017-09-19 15:13:46.966000 577087 module initialization error: module 'pyrebase' has no attribute 'initialize_app'

then i realized there are two ways to provide 3rd party dependencies, one is requirement.txt and other is vendor folder
(reference doc: http://chalice.readthedocs.io/en/latest/topics/packaging.html#rd-party-packages)
and also realized that there are two things one is wheel (.whl) and other is tarball (.tar.gz) and i read somewhere that the problem i'm suffering from is mostly come with the package in which tarball involve, and in this case we have to download the source using pip download packege-name and have to build the it manually (tarball -> wheel) with command pip wheel packege-name.tar.gz

well I'm not sure with it, but tried this method and download the source of pyrebase:

19-Sep-17  07:01 PM    <DIR>          .
19-Sep-17  07:01 PM    <DIR>          ..
19-Sep-17  07:01 PM           458,001 gcloud-0.17.0.tar.gz
19-Sep-17  07:01 PM            28,426 googleapis-common-protos-1.5.2.tar.gz
19-Sep-17  07:01 PM           204,500 httplib2-0.10.3.tar.gz
19-Sep-17  07:01 PM             8,104 jws-0.1.3.tar.gz
19-Sep-17  07:01 PM            77,208 oauth2client-3.0.0.tar.gz
19-Sep-17  07:01 PM           375,727 protobuf-3.4.0-py2.py3-none-any.whl
19-Sep-17  07:01 PM            63,452 pyasn1-0.3.5-py2.py3-none-any.whl
19-Sep-17  07:01 PM            60,445 pyasn1_modules-0.1.4-py2.py3-none-any.whl
19-Sep-17  07:01 PM         7,544,723 pycryptodome-3.4.3-cp36-cp36m-win_amd64.whl
19-Sep-17  07:01 PM             9,561 Pyrebase-3.0.27-py3-none-any.whl
19-Sep-17  07:01 PM             8,755 python_jwt-2.0.1-py2.py3-none-any.whl
19-Sep-17  07:01 PM           514,827 requests-2.11.1-py2.py3-none-any.whl
19-Sep-17  07:01 PM            52,341 requests_toolbelt-0.7.0-py2.py3-none-any.whl
19-Sep-17  07:01 PM            46,918 rsa-3.4.2-py2.py3-none-any.whl
19-Sep-17  07:01 PM           478,810 setuptools-36.5.0-py2.py3-none-any.whl
19-Sep-17  07:01 PM            10,702 six-1.11.0-py2.py3-none-any.whl
              16 File(s)      9,942,500 bytes
               2 Dir(s)  25,223,860,224 bytes free

build it manually with pip wheel Pyrebase-3.0.27-py3-none-any.whl
now i can see some additional files, it looks like all .tar.gz now have its .whl representation:

19-Sep-17  07:04 PM    <DIR>          .
19-Sep-17  07:04 PM    <DIR>          ..
19-Sep-17  07:04 PM           642,294 gcloud-0.17.0-cp36-none-any.whl
19-Sep-17  07:01 PM           458,001 gcloud-0.17.0.tar.gz
19-Sep-17  07:01 PM            28,426 googleapis-common-protos-1.5.2.tar.gz
19-Sep-17  07:04 PM            59,967 googleapis_common_protos-1.5.2-cp36-none-any.whl
19-Sep-17  07:04 PM            85,518 httplib2-0.10.3-cp36-none-any.whl
19-Sep-17  07:01 PM           204,500 httplib2-0.10.3.tar.gz
19-Sep-17  07:04 PM            11,818 jws-0.1.3-cp36-none-any.whl
19-Sep-17  07:01 PM             8,104 jws-0.1.3.tar.gz
19-Sep-17  07:04 PM           107,384 oauth2client-3.0.0-cp36-none-any.whl
19-Sep-17  07:01 PM            77,208 oauth2client-3.0.0.tar.gz
19-Sep-17  07:01 PM           375,727 protobuf-3.4.0-py2.py3-none-any.whl
19-Sep-17  07:01 PM            63,452 pyasn1-0.3.5-py2.py3-none-any.whl
19-Sep-17  07:01 PM            60,445 pyasn1_modules-0.1.4-py2.py3-none-any.whl
19-Sep-17  07:01 PM         7,544,723 pycryptodome-3.4.3-cp36-cp36m-win_amd64.whl
19-Sep-17  07:01 PM             9,561 Pyrebase-3.0.27-py3-none-any.whl
19-Sep-17  07:01 PM             8,755 python_jwt-2.0.1-py2.py3-none-any.whl
19-Sep-17  07:01 PM           514,827 requests-2.11.1-py2.py3-none-any.whl
19-Sep-17  07:01 PM            52,341 requests_toolbelt-0.7.0-py2.py3-none-any.whl
19-Sep-17  07:01 PM            46,918 rsa-3.4.2-py2.py3-none-any.whl
19-Sep-17  07:01 PM           478,810 setuptools-36.5.0-py2.py3-none-any.whl
19-Sep-17  07:01 PM            10,702 six-1.11.0-py2.py3-none-any.whl
              21 File(s)     10,849,481 bytes
               2 Dir(s)  25,222,946,816 bytes free

and put it in vendor folder
now my folder structure is:

and then again tried to deploy with hope that this solution will work,
project deployed with no error but the problem is same when i hit the url with postman tool it says internal server error and logs says the problem is with pyrebase

please some words for the problem

[stealthycoin]

A couple things here some of which are critical and some of which are not:

As a more minor point requirements.txt does not need have all dependencies frozen into it. chalice deploy will explore all the dependencies and attempt to package them up so your requirements.txt can just be:

pyrebase==3.0.27

And it will bundle up all the packages required on a deploy, this would mean you may get different versions of transient dependencies, which is fine if you don't care about their versions. You can of course freeze all the dependencies as you have done if you would like to pin all the transitive dependencies to get a known set of dependencies that won't change out from under you. Just wanted to make sure you were aware of that behavior.

The main issue here is explained by the error message

Could not install dependencies:
googleapis-common-protos==1.5.2
pycryptodome==3.4.3
You will have to build these yourself and vendor them in
the chalice vendor folder.

The packages googleapis-common-protos and pycryptodome have C extensions and do not have wheel files on PyPi that allow for automatic packaging on Chalice's part. So you have to build them yourself.
Another issue is that googleapis-common-protos looks tricky to install. It depends on you having protoc installed according to the installation instructions: here and it does not have wheels available on PyPi so you do need to build and package it yourself on the architecture you intend to run it on, in this case Amazon Linux since that is what Lambda runs on. There are is a tutorial on how to do that instructions on how to do that located here. You would have to take the extra steps of making sure that the machine you do the build on has protoc installed and is able to build that package though by following the instructions in their readme.

You do not need to put everything in the vendor directory, and in fact putting wheel files in the vendor directory is not going to work properly. You can read the packaging documentation that I linked below which explains how to vendor a package.

So in summary, build those two packages on an EC2 instance running Amazon Linux and vendor them according to the documentation I linked below. Let me know if you have any more questions.

Related documentation:
http://chalice.readthedocs.io/en/latest/topics/packaging.html
http://chalice.readthedocs.io/en/latest/topics/packaging.html#cryptography-example

[malikasinger1]

i already read those documents, not much helpfull, can you please refer to some example?

[stealthycoin]

Where in the example do you get stuck? There is a step by step example that I linked that illustrates how to install cryptography 1.9 (which did not have wheels yet). It would be the exact same process just instead of cryptography it would be the two that were listed in the error message
googleapis-common-protos==1.5.2 and pycryptodome==3.4.3.

[malikasinger1]

please have a look of this image:

i have followed the instruction in document e.g: download googleapis-common-protos==1.5.2 and pycryptodome==3.4.3 build it with pip wheel <filename> and put it in vendor folder

it is still not working, however it is not giving me any error or warning on deploy

PS E:\Projects\class project\wheel build\helloworld> chalice deploy
Updating lambda function...
Regen deployment package...
Updating IAM policy.
Sending changes to lambda.
API Gateway rest API already found.
Deploying to: helloworld
https://622ty8a1xf.execute-api.us-east-1.amazonaws.com/helloworld/

when i calling the url https://622ty8a1xf.execute-api.us-east-1.amazonaws.com/helloworld/routeone it is saying {"message": "Internal server error"}

chalice logs are saying

PS E:\Projects\class project\wheel build\helloworld> chalice logs
2017-09-23 21:55:44.062000 497974 Unable to import module 'app': No module named 'Crypto'
2017-09-23 21:58:24.542000 a17dd9 Unable to import module 'app': No module named 'Crypto'
2017-09-23 22:02:57.239000 e40aab Unable to import module 'app': No module named 'Crypto'
2017-09-23 22:03:13.701000 e40aab Unable to import module 'app': No module named 'Crypto'
2017-09-23 22:03:14.741000 e40aab Unable to import module 'app': No module named 'Crypto'
PS E:\Projects\class project\wheel build\helloworld>

let me know if i'm making any mistake in making vendor folder,

thank you for your support
looking forward to listen from you
Inzamam Malik.

[stealthycoin]

Right so the issue is you are putting the whl files directly in the vendor directory.

From the example walkthrough there is a step where you unzip the wheel files into the vendor directory:

Download the cryptography wheel file from the Amazon Linux instance and unzip it into the vendor/ directory in the root directory of your Chalice app.

So you would download the file googleapis-common-protos-somestuffhere.whl from the Amazon Linux machine you did the build on and UNZIP the contents of that wheel file into the vendor directory. If you look at the example I linked it shows what the vendor directory should look like after unzipping a whl file into it. There should be a directory with the same name as the package, and then another directory with the same name as the package and a .{version}.dist-info extension at the end which has some metadata files in it.

[malikasinger1]

how can i

Download the cryptography(googleapis-common-protos-somestuffhere.whl in my case) wheel file from the Amazon Linux instance

since i didn't put anything in the Amazon Linux Instance (did i put any thing? i dont think)

[stealthycoin]

You will have to do that for any packages with C extensions since you need to compile them on a system that is compatible with lambda. I see in your screenshot you have one .whl file PyYAML-3.12-cp36-cp36m-win_amd64.whl that won't work on lambda. The suffix win_amd64 implies this means it was compiled on windows, and hence it will have windows specific binaries in the wheel.

The error you got initially is because Chalice can't package the dependencies for AWS Lambda from your current machine because there are C dependencies that need compiling, and the maintainers of that dependency did not add wheel files for manylinux1 or linux_x86 both of which work on lambda.

So you have to do the work instead of the maintainers which means you need to get an EC2 instance spun up that is Amazon Linux (the same as what Lambda runs on) and compile those dependencies on that box. That will generate a wheel file that is compatible with Lambda.

In your case you need to build pycryptodome since it has C extensions, and no wheel for manylinux1 (hopefully that will change in the future) but for now you need to build this yourself on a compatible machine. Which consists of sshing to an Amazon Linux EC2 instance and building the package.

At its most basic this means doing:
$ pip install pycryptodome==3.4.7
$ pip wheel pycryptodome-3.4.7.tar.gz

Which should give you
pycryptodome-3.4.7-cp36-cp36m-linux_x86_64.whl if you are on python 3.6. You would then need to download that file from the EC2 instance and unzip it into the vendor directory in your chalice app.

$ unizp pycryptodome-3.4.7-cp36-cp36m-linux_x86_64.whl -d vendor

Which should make your vendor directory look like this:

$ ls vendor
Crypto  
pycryptodome-3.4.7.dist-info

So there is your Crypto that it was complaining isn't found. The build also complained about googleapis-common-protos==1.5.2 not being included so we can do the same process on that.

$ pip download googleapis-common-protos==1.5.2
$ pip wheel googleapis-common-protos-1.5.2.tar.gz

This one is a little different because it builds out two wheel files:
googleapis_common_protos-1.5.2-py3-none-any.whl and protobuf-3.4.0-cp36-cp36m-manylinux1_x86_64.whl. If you take both of these and again download them and unzip into the vendor directory you should get the following vendor directory:

$ ls vendor
Crypto  
googleapis_common_protos-1.5.2.dist-info        
protobuf-3.4.0.dist-info        
pycryptodome-3.4.7.dist-info
google  
googleapis_common_protos-1.5.2-py3.6-nspkg.pth  
protobuf-3.4.0-py3.6-nspkg.pth

[malikasinger1]

so that was the point i was missing from the beginning that i need the linux machine to build, of course you are right because i worked on C in childhood it builds for that machine that it is building on,
Thank you very much for pointing the problem out, it was really helpful for me

let me know one last thing if i use ubuntu distribution of linux for building, will it work on lambda or not?
since i never used EC2 so i'm not willing to invest time on that and prefer to make a virtual machine using vmware or something similar,

i can make what ever machine e.g: dabian, ubuntu, fedora, blackbox
please let me know which one will work
looking forward to listen from you

[stealthycoin]

Im not sure what image you could use that would be binary compatible with Amazon Linux, its based on CentOS and RHEL so maybe one of those would work? Not sure as I have never tired it myself.

My next suggestion if you wanted to do something local and not spin up an EC2 instance would be to use docker and this repo to make a manylinux1_x86_64 wheel which will be compatible with Amazon Linux.

[jamesls]

Closing out issue. Let us know if you're still having issues.