Aws-sam-cli: [python] 3rd party modules

[enghwa]

currently, for 3rd party module, we have to do pip install -t . on the same directory as template.yaml and the main python application file. This pollutes the working directory when you have a lot of modules. Is there a better way to handle this?
maybe on the file zipping of a subdirectory?

[sanathkr]

You can write a script with a task runner that monitors source code and creates a build directory where you have your source code as well as dependencies flattened. Set CodeUri in SAM file to point point to the build directory

This script will follow the guidelines given here (http://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html), but do this every time source code changes.

I know Nodejs has pretty robust task runners like `gulp. I see quite a few for Python here - https://wiki.python.org/moin/ConfigurationAndBuildTools

One request: If you happen to setup a task runner, can you contribute the script back to this repository under samples folder? That would help everyone.

[kpx-dev]

I just ran into this issue as well for my Python project. I have this script based on your above suggestion:

rm -rf dist
mkdir -p dist/
cp app.py dist/

# install packages
pip freeze > requirements.txt
pip install -r requirements.txt -t dist/

And in my template.yaml, I have this line: CodeUri: dist

Is this something good enough to contribute to the samples dir?

[kpx-dev]

Actually the above approach I don't get live code loading, always have to re-install packages. Will look for diff approach...

./scripts/build && sam local generate-event api --method GET --path /products | sam local invoke networth

[pansen]

please have a look on my example. for me this works perfect, though i would like to know if i can avoid creating this zip all the time, just to run my handler locally.

following that condition in runtime.go:258 i assumed it must be zip, maybe there's another trick.

func (r *Runtime) Invoke(event string) (io.Reader, io.Reader, error) {

    log.Printf("Invoking %s (%s)\n", r.Function.Handler, r.Name)

    // If the CodeUri has been specified as a .jar or .zip file, unzip it on the fly
    if strings.HasSuffix(r.Cwd, ".jar") || strings.HasSuffix(r.Cwd, ".zip") {

[PaulMaddox]

I just pushed an update to some of this logic in a pull request that was merged this morning.
Specifically, this was added:

https://github.com/awslabs/aws-sam-local/blob/develop/runtime.go#L276-L278

You should be able to specify a directory name in your SAM template, such as:

CodeUri: ./some-directory

Can you give that a go? It should avoid you having to zip each time.

[mikeluby]

Doesn't this still require dependencies to be pip installed into the working directory? I don't think the issue is with creating the zip (at least not for my usage) but it's more about not polluting the working directory with 3rd party modules.

This update would still require a build script to copy the working directory and 3rd party modules into some sort of build directory

[PaulMaddox]

Good point - i'll merge your PR as it is.

[randyjohnston]

Could you please clarify how to set up template.yml for Python module with dependencies? The example template.yml does not make sense and references a non-existent zip file.

[heitorlessa]

I'll make a proper PR as soon as I get time to clean comments up and test in other OSes, but if that helps here's what I came up with (tested in OSX only) to have dependencies installed + hot-reloading experience similar to NodeJS.

Essentially it takes advantage of hard links and clones the entire source code/folders (lib, etc.) onto a folder called dev. All you have to do is to change your SAM template to point to <source_code>/dev.

Makefile - Hardcoded for Python 3.6 and pip :

# VirtualEnv vars
SERVICE ?= users
VENV := ${SERVICE}/.venv
VENV_LIBS := $(VENV)/lib/python3.6/site-packages

# help credit (https://gist.github.com/prwhite/8168133)
help: ## Show this help

    @fgrep -h "##" $(MAKEFILE_LIST) | fgrep -v fgrep | sed -e 's/\\$$//' | sed -e 's/##//'

target: help

    exit 0

clean: ## -> Deletes current virtual env environment

    $(info "[-] Who needs all that anyway? Destroying environment....")
    rm -rf ./$(VENV)
    rm -rf ./$(SERVICE)/dev
    rm -rf ./*.zip

build-dev: _check_service_definition ## -> creates dev folder and install dependencies (requirements.txt) into dev/ folder

    echo "[+] Cloning ${SERVICE} directory structure to ${SERVICE}/dev"
    rsync -a -f "+ */" -f "- *" -f "- dev/" ${SERVICE}/ ${SERVICE}/dev/
    echo "[+] Cloning source files from ${SERVICE} to ${SERVICE}/dev"
    find ${SERVICE} -type f \
            -not -name "*.pyc" \
            -not -name "*__pycache__" \
            -not -name "requirements.txt" \
            -not -name "event.json" \
            -not -name "dev" | cut -d '/' -f2- > .results.txt
    @while read line; do \
        ln -f ${SERVICE}/$$line ${SERVICE}/dev/$$line; \
    done < .results.txt
    echo "[+] Installing dependencies into dev/"
    pip3.6 install \
        --isolated \
        --disable-pip-version-check \
        -Ur $(SERVICE)/requirements.txt -t ${SERVICE}/dev/

_check_service_definition:

    echo "[*] Checking whether service $(SERVICE) exists..."

# SERVICE="<name_of_service>" must be passed as ARG for target or else fail
ifndef SERVICE
    echo "[!] SERVICE argument not defined...FAIL"
    echo "[*] Try running 'make <target> SERVICE=<service_name>'"
    exit 1
endif

ifeq ($(wildcard $(SERVICE)/.),)
    echo "[!] $(SERVICE) folder doesn't exist"
    exit 1
endif

ifeq ($(wildcard $(SERVICE)/requirements.txt),)
    echo "[!] Pip requirements file missing from $(SERVICE) folder..."
    exit 1
endif

run-dev: ## -> Run SAM Local API GW
    sam local start-api

Folder structure - Sample with multiple private packages (vendor/db, vendor/dispatcher) and external dependencies (boto3, requirements, etc.)

.
├── Makefile
├── bootstrap_dynamodb.py
├── setup.cfg
├── template.yaml
└── users
    ├── requirements.txt
    ├── users.py
    ├── something_else.py
    └── vendor
        ├── db
        │   └── __init__.py
        ├── dispatcher
        │   ├── __init__.py
        ├── helper
        │   └── __init__.py
        └── users
            ├── __init__.py

SAM

    UsersFunction:
      Type: AWS::Serverless::Function
      Properties:
        CodeUri: ./users/dev/
        Handler: users.lambda_handler
        Runtime: python3.6
        Environment:
          Variables:
            TABLE_NAME: "users"
        Events:
          CatchAll:
            Type: Api
            Properties:
              Path: /{proxy+}
              Method: any

Usage

make build-dev SERVICE="users"

Where users is where my source code is as it's a users service sample.

[liamdawson]

CodeUri helps to build alternate workflows, sure, but I really wouldn't consider this issue closed. What would really help would be some kind of pre-run hook for development purposes.

[J-Rojas]

I've used the suggested workarounds here, however I'm running into a problem where Python modules that use native libraries will not run when the host machine OS is different than the Docker container's.

I'm running on MacOS and my Python modules are installed via pip into a build directory, then the SAM template file uses the build directory as the source path which contains all lambda code and Python modules.

I receive the following error when executing sam local invoke because the Docker container cannot locate a Linux compatible Python library within the modules installed on my MacOS host.

module initialization error: Cannot load native module 'Crypto.Hash._SHA256'

Are there any known workarounds for this? Is it possible to run some custom install scripts within the SAM Docker container to install Python packages natively?

NOTE: The Serverless SDK solves this problem with the serverless-python-requirements plugin that uses a Docker container to pip install all of the Python modules for final packaging and deployment.
https://www.npmjs.com/package/serverless-python-requirements

[heitorlessa]

Hi @J-Rojas, as you're on Mac, give this a try:

sam init --location gh:aws-samples/cookiecutter-aws-sam-python

Answer y when asked about Include Experimental Makefile. You'll see the instructions in the Readme under Appendix on how to do this.

In essence, you will simply run this command and problem will be solved as we will build all your deps under a Docker container similar to the Lambda environment so the compilation will use the correct target system ;-)

make package SERVICE="first_function" DOCKER=1

Where the value of SERVICE is where your python project is. The folder will be structured in a way to allow you to have multiple functions, etc. This is a more advanced approach but happy to provide more details if you need.

[J-Rojas]

@heitorlessa Thanks for the advice. I see this Makefile with your command packages the project with Docker into a zip file. You gave advice in an earlier post on how to configure the Makefile for hot-reloading. I'd like to be able to make code changes while testing. Is there a way to do this while leveraging Docker? Maybe it's just a matter of packaging the Python modules in Docker, unzipping the zip file and then using the build command for hot-reloading?

[S-Cardenas]

@heitorlessa Using sam init --location gh:aws-samples/cookiecutter-aws-sam-python what would be the workflow for installing 3rd party pip packages? It is not evident from the fact that there is both a requirements.txt file and a Pipfile.

[heitorlessa]

Hi Stefan - If you haven’t selected the experimental Makefile (I would if
you’re on Mac/Linux) I had left comments on README for that, otherwise
simply use pip.

If you used the Makefile all of this is taken care with one command that
will also provide hot code reload: make build SERVICE=“folder name where
the code is”

Before I created the Makefile I used the following in a script:

pipenv install
pipenv lock -r > requirements.txt
pip install -r requirements.txt -t

Hope that makes sense and I can revisit the README after in 2 weeks time

PS: I will add logic to remove requirements.txt if one selects the Makefile
to make things less confusing as that is redundant
On Fri, 25 May 2018 at 18:17, Stefan Cardenas [e-mail]
wrote:

@heitorlessa https://github.com/heitorlessa Using sam init --location
gh:aws-samples/cookiecutter-aws-sam-python what would be the workflow for
installing 3rd party pip packages? It is not evident from the fact that
there is both a requirements.txt file and a Pipfile.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/awslabs/aws-sam-cli/issues/53#issuecomment-392108321,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADL4BJDuO34_DpaGeX6FFXoyRZ5VaYoDks5t2C57gaJpZM4O4TrT
.

[S-Cardenas]

@heitorlessa Thanks for the quick response. I was able to deduce the same steps you mentioned and got the desired results before I saw your message. It would be nice if the docs made the process more painfully obvious :)

I'd like to contribute to this repo seeing as I use docker and lambda daily. Any suggestions as to where to begin or what areas need the most immediate work?

[john-aws]

Hi, I've read this thread a couple of times and I'm no clearer on how to avoid Python packages polluting my source folder. I'd like them to be elsewhere, for example under a packages/ subfolder, so it's easy to git ignore. Is this possible? Thanks.

[jfuss]

@john-aws This is a pretty old thread. We recently released sam build (late last year). Using that should give you what you are looking for.

[FelixTx]

So is there a way of running a function with dependencies without building it before ?
ie using sam local start-api -t template.yaml without sam build before, without getting Unable to import module 'app': No module named <dependency> apart from copying all dependencies next to the app.py

[Rubyj]

@john-aws This is a pretty old thread. We recently released sam build (late last year). Using that should give you what you are looking for.

Hi @john-aws I am trying to do a sam deploy which works, but then when I try to run the lambda function in aws console I am receiving the missing dependency error. Can you explain how sam build should fix this even though it only deploys to a hidden directory? I still find myself having to drag my third party dependencies into the same folder as my source code before executing sam deploy

[jfuss]

@Rubyj Not sure I understand your question. sam build creates an artifact based on your requirements.txt. sam package will then zip that up for you and sam deploy pushes out to AWS.

If you need specific help, you should engage with us and the community in our Slack channel (details in the README of the repo).

[Rubyj]

@jfuss It seems that your comment https://github.com/awslabs/aws-sam-cli/issues/125#issuecomment-467516880 solved my problem. The dependency error (while already using build) was due to passing in the template into my package command. Now it works!

[Rubyj]

@jfuss @FelixTx Do we have a good solution for installing local third party libraries during a sam container build?

[erickisos]

@Rubyj Not sure I understand your question. sam build creates an artifact based on your requirements.txt. sam package will then zip that up for you and sam deploy pushes out to AWS.

If you need specific help, you should engage with us and the community in our Slack channel (details in the README of the repo).

Until yesterday, i used to make a simple

sam build
sam package
sam deploy

to release my apps, but now on none of my laptops sam put the python requirements in the deployed package, so i end with ImportErrors in all of my lambdas.

Is there a change with newer versions of sam-cli?

How can i solve this, or just i need to implement my own deploy-scripts?

[rakcy]

Performing the sam build before invoking is often too slow for my case. A solution I have was to imitate some parts of sam build (this is specifically for python):

1. The template file I used have the option: CodeUri: sam_test
2. In order for the dependencies to be mounted by docker container, I used rsync to create hardlinks to sam_test:

rsync -ar --link-dest="$PWD/.venv/lib/python3.6/site-packages/" .venv/lib/python3.6/site-packages/ sam_test

1. Next I put the lambda files to the sam_test:

ln -f app_* sam_test

1. After this preparation, I can use sam local ... directly and it would still support hot-reloading. It also doesn't matter that much if I run the rsync scripts again since it's fast for it to just compare files and only create hardlinks for missing files.