Chalice: {"errorMessage": "Unable to import module 'app'"}

[loretoparisi]

This error could be related to this issue https://github.com/awslabs/chalice/issues/105
My app.py is importing a 3rdparty module like

from chalice import Chalice
from empythy import EmpathyMachines;

that I have successfully installed via pip install --upgrade -r requirements.txt as explained in the related issue, but it seems it has not been deployed correctly by chalice on the lambda machine.
My virtualenv looks like this

(env) admin@macbookproloreto:~/Developmemt/ParisiLabs/ML/tutorials/lambda-sentiment/env$ tree -d -L 4
.
├── bin
├── include
│   └── python2.7 -> /System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7
└── lib
    └── python2.7
        ├── config -> /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/config
        ├── distutils
        ├── encodings -> /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings
        ├── lib-dynload -> /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload
        └── site-packages
            ├── empythy
            ├── empythy-0.5.6.dist-info
            ├── nltk
            ├── nltk-3.2.1.dist-info
            ├── numpy
            ├── numpy-1.11.1.dist-info
            ├── pip
            ├── pip-8.1.2.dist-info
            ├── pkg_resources
            ├── scikit_learn-0.17.1.dist-info
            ├── scipy
            ├── scipy-0.18.0.dist-info
            ├── setuptools
            ├── setuptools-25.2.0.dist-info
            ├── sklearn
            ├── wheel
            └── wheel-0.29.0.dist-info

[jamesls]

I think this is the same root cause as #105. We don't have support for C extensions.

Another way you can see this error is if you turn on app.debug = True and try to import the module in your view function. This is what I get, which is the expected error right now until we can support C extensions:

{
    "errorMessage": "/var/task/sklearn/__check_build/_check_build.so: invalid ELF header\n___________________________________________________________________________\nContents of /var/task/sklearn/__check_build:\nsetup.pyc                 setup.py                  _check_build.so\n__init__.pyc              __init__.py\n___________________________________________________________________________\nIt seems that scikit-learn has not been built correctly.\n\nIf you have installed scikit-learn from source, please do not forget\nto build the package before using it: run `python setup.py install` or\n`make` in the source directory.\n\nIf you have used an installer, please check that it is suited for your\nPython version, your operating system and your platform.",
    "errorType": "ImportError",
    "stackTrace": [
        [
            "/var/task/chalice/__init__.py",
            192,
            "__call__",
            "response = view_function(*function_args)"
        ],
        [
            "/var/task/app.py",
            9,
            "index",
            "from empythy import EmpathyMachines"
        ],
        [
            "/var/task/empythy/__init__.py",
            2,
            "<module>",
            "from EmpathyMachines import EmpathyMachines"
        ],
        [
            "/var/task/empythy/EmpathyMachines.py",
            4,
            "<module>",
            "from sklearn.base import BaseEstimator, TransformerMixin"
        ],
        [
            "/var/task/sklearn/__init__.py",
            56,
            "<module>",
            "from . import __check_build"
        ],
        [
            "/var/task/sklearn/__check_build/__init__.py",
            46,
            "<module>",
            "raise_build_error(e)"
        ],
        [
            "/var/task/sklearn/__check_build/__init__.py",
            41,
            "raise_build_error",
            "%s\"\"\" % (e, local_dir, ''.join(dir_content).strip(), msg))"
        ]
    ]
}

I'm going to update our docs to state the limitations we have right now for python packages to help others that might run into this issue.

[loretoparisi]

@jamesls Ok so basically we cannot use numpy at this moment via chalice.

[fidiego]

I'm seeing this same error when I try to use andymccurdy/redis-py.

[wittfabian]

I have found this article about the limitations of C libraries for Python modules.
https://serverlesscode.com/post/deploy-scikitlearn-on-lamba/
Is this a possibility?

[fidiego]

@wittfabian
would be interested in learning how to build other libraries with the methods outlined
specifically, py-redis

[onurmatik]

Basically the idea to make this work is to compile the required non-python extensions in the same environment as the environment that Lambda runs, which is an Amazon Linux instance, and include them in the zipped package.

I don't know how this can be integrated into chalice but here are 2 related links:
http://stackoverflow.com/questions/34749806/using-moviepy-scipy-and-numpy-in-amazon-lambda
http://www.perrygeo.com/running-python-with-compiled-code-on-aws-lambda.html

[loretoparisi]

@onurmatik so it should work as in the python lambda implementation. The best way would be to use requirements.txt and setup.py, but a local zipped folder will work as well. What about combine virtualenv and zip the while env?

[jamesls]

@fidiego redis-py doesn't have any required C deps (as far as I know) so adding it to your requirements.txt should work. Here's what I tried:

$ cat requirements.txt
redis==2.10.5
$ cat app.py
import inspect
from chalice import Chalice
from chalice import BadRequestError
import redis

app = Chalice(app_name='redispy')


@app.route('/')
def index():
    return {'filename': inspect.getfile(redis)}

$ chalice deploy
...
$ http https://vt7kr8t4y8.execute-api.us-west-2.amazonaws.com/dev/
HTTP/1.1 200 OK
...

{
    "filename": "/var/task/redis/__init__.py"
}

Let me know if that's not working for you.

[jamesls]

As for numpy/binary file support I have a few ideas for how to making working with both C extensions as well as custom binaries easier.

I see two ways to go (maybe it's worth doing both):

First one: Provide a hook into adding arbitrary data into the deployment zip file we send to lambda. That way as we're adding files to the zip file we can give you a chance to inject any prebuilt binaries. I think that supporting a "on-build" hook would make this possible, something generic like:

#!/bin/bash
# This would be hooks/building-zip
# It would pass in several arguments:

ZIP_DIR="$1"
MY_PREBUILT_BINARIES="linux-binaries"

cp -r "$MY_PREBUILT_BINARIES/*"  "$ZIP_DIR/"

# If you wanted to, for some reason, you could also remove things from the deployed zip
rm $ZIP_DIR/unnecessary-file.py

chalice deploy would call the hooks/building-zip script if it existed.

This is probably more useful to address #42.

The second option is to try to leverage the manylinux1 wheels. I'm not exactly sure how to plumb this in, but many libraries are starting to provide manylinux wheel files (numpy, sklearn)

The latest version of pip let's you override the platform/arch to use, so the idea would be to ask it to use manlinux wheels's when building out the virtualenv to send to lambda. I'd have to experiment with this approach more to see if I could get it working, but the benefit of this approach would be that it would "just work", you wouldn't have to do anything (provided your 3rd party librarys have manylinux1 wheels).

[jamesls]

182 has been merged in https://github.com/awslabs/chalice/commit/7329b27fbe2e1ee5019cc2845b9329e6ad857c67