Aws-sam-cli: sam build can't resolve dependencies - tensorflow

[pasppro]

Description

Error when building projects containing tensorflow on a Mac:

Error: PythonPipBuilder:ResolveDependencies - {tensorflow==2.2.0(wheel), grpcio==1.29.0(wheel)}

This could be the same as Issue #1840 but I'm not using PyCharm - just the command line.

Steps to reproduce

(Assumes python 3.6 is already installed)

1. sam init (Project sam-bld-err, choose python 3.6 runtime and Hello World template)
2. Create a python virtual environment within the project
   cd sam-bld-err
   virtualenv -p /Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6 venv_be
   (Adjust path to python 3.6 as needed)
3. Activate virtual env and install tensorflow
   source ./venv_be/bin/activate
   pip install tensorflow==2.2.0
4. Update project requirements
   pip freeze > ./hello_world/requirements.txt
5. Build the project
   sam build or sam build --use-container (Same results for each)

Observed result

(Sorry - Lots of debug output!)

sam build --debug
Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
'build' command is called
No Parameters detected in the template
2 resources found in the template
Found Serverless function with name='HelloWorldFunction' and CodeUri='hello_world/'
No Parameters detected in the template
Building function 'HelloWorldFunction'
Loading workflow module 'aws_lambda_builders.workflows'
Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
Registering workflow 'GoDepBuilder' with capability 'Capability(language='go', dependency_manager='dep', application_framework=None)'
Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'
Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
Registering workflow 'CustomMakeBuilder' with capability 'Capability(language='provided', dependency_manager=None, application_framework=None)'
Found workflow 'PythonPipBuilder' to support capabilities 'Capability(language='python', dependency_manager='pip', application_framework=None)'
Running workflow 'PythonPipBuilder'
Running PythonPipBuilder:ResolveDependencies
calling pip download -r /Users/philsweeney/Documents/AWSProjects/sam-bld-error/hello_world/requirements.txt --dest /var/folders/s5/_787jljn6l14fhg2w4wn4d_w0000gn/T/tmplzw2zx5z
Full dependency closure: {tensorflow==2.2.0(wheel), markdown==3.2.2(wheel), setuptools==47.1.1(wheel), tensorboard==2.2.2(wheel), tensorboard-plugin-wit==1.6.0.post3(wheel), scipy==1.4.1(wheel), chardet==3.0.4(wheel), google-auth-oauthlib==0.4.1(wheel), h5py==2.10.0(wheel), opt-einsum==3.2.1(wheel), certifi==2020.4.5.1(wheel), oauthlib==3.1.0(wheel), protobuf==3.12.2(wheel), google-auth==1.16.0(wheel), zipp==3.1.0(wheel), six==1.15.0(wheel), tensorflow-estimator==2.2.0(wheel), importlib-metadata==1.6.0(wheel), termcolor==1.1.0(sdist), gast==0.3.3(wheel), wrapt==1.12.1(sdist), cachetools==4.1.0(wheel), grpcio==1.29.0(wheel), pyasn1==0.4.8(wheel), google-pasta==0.2.0(wheel), requests-oauthlib==1.3.0(wheel), urllib3==1.25.9(wheel), pyasn1-modules==0.2.8(wheel), absl-py==0.9.0(sdist), astunparse==1.6.3(wheel), keras-preprocessing==1.1.2(wheel), idna==2.9(wheel), requests==2.23.0(wheel), rsa==4.0(wheel), wheel==0.34.2(wheel), numpy==1.18.4(wheel), werkzeug==1.0.1(wheel)}
initial compatible: {markdown==3.2.2(wheel), setuptools==47.1.1(wheel), tensorboard==2.2.2(wheel), tensorboard-plugin-wit==1.6.0.post3(wheel), chardet==3.0.4(wheel), google-auth-oauthlib==0.4.1(wheel), opt-einsum==3.2.1(wheel), certifi==2020.4.5.1(wheel), oauthlib==3.1.0(wheel), google-auth==1.16.0(wheel), zipp==3.1.0(wheel), six==1.15.0(wheel), tensorflow-estimator==2.2.0(wheel), importlib-metadata==1.6.0(wheel), gast==0.3.3(wheel), cachetools==4.1.0(wheel), pyasn1==0.4.8(wheel), google-pasta==0.2.0(wheel), requests-oauthlib==1.3.0(wheel), urllib3==1.25.9(wheel), pyasn1-modules==0.2.8(wheel), astunparse==1.6.3(wheel), keras-preprocessing==1.1.2(wheel), idna==2.9(wheel), requests==2.23.0(wheel), rsa==4.0(wheel), wheel==0.34.2(wheel), werkzeug==1.0.1(wheel)}
initial incompatible: {protobuf==3.12.2(wheel), tensorflow==2.2.0(wheel), termcolor==1.1.0(sdist), wrapt==1.12.1(sdist), numpy==1.18.4(wheel), grpcio==1.29.0(wheel), absl-py==0.9.0(sdist), scipy==1.4.1(wheel), h5py==2.10.0(wheel)}
Downloading missing wheels: {tensorflow==2.2.0(wheel), termcolor==1.1.0(sdist), absl-py==0.9.0(sdist), h5py==2.10.0(wheel), protobuf==3.12.2(wheel), wrapt==1.12.1(sdist), numpy==1.18.4(wheel), grpcio==1.29.0(wheel), scipy==1.4.1(wheel)}
calling pip download --only-binary=:all: --no-deps --platform manylinux1_x86_64 --implementation cp --abi cp36m --dest /var/folders/s5/_787jljn6l14fhg2w4wn4d_w0000gn/T/tmplzw2zx5z tensorflow==2.2.0
calling pip download --only-binary=:all: --no-deps --platform manylinux1_x86_64 --implementation cp --abi cp36m --dest /var/folders/s5/_787jljn6l14fhg2w4wn4d_w0000gn/T/tmplzw2zx5z termcolor==1.1.0
calling pip download --only-binary=:all: --no-deps --platform manylinux1_x86_64 --implementation cp --abi cp36m --dest /var/folders/s5/_787jljn6l14fhg2w4wn4d_w0000gn/T/tmplzw2zx5z absl-py==0.9.0
calling pip download --only-binary=:all: --no-deps --platform manylinux1_x86_64 --implementation cp --abi cp36m --dest /var/folders/s5/_787jljn6l14fhg2w4wn4d_w0000gn/T/tmplzw2zx5z h5py==2.10.0
calling pip download --only-binary=:all: --no-deps --platform manylinux1_x86_64 --implementation cp --abi cp36m --dest /var/folders/s5/_787jljn6l14fhg2w4wn4d_w0000gn/T/tmplzw2zx5z protobuf==3.12.2
calling pip download --only-binary=:all: --no-deps --platform manylinux1_x86_64 --implementation cp --abi cp36m --dest /var/folders/s5/_787jljn6l14fhg2w4wn4d_w0000gn/T/tmplzw2zx5z wrapt==1.12.1
calling pip download --only-binary=:all: --no-deps --platform manylinux1_x86_64 --implementation cp --abi cp36m --dest /var/folders/s5/_787jljn6l14fhg2w4wn4d_w0000gn/T/tmplzw2zx5z numpy==1.18.4
calling pip download --only-binary=:all: --no-deps --platform manylinux1_x86_64 --implementation cp --abi cp36m --dest /var/folders/s5/_787jljn6l14fhg2w4wn4d_w0000gn/T/tmplzw2zx5z grpcio==1.29.0
calling pip download --only-binary=:all: --no-deps --platform manylinux1_x86_64 --implementation cp --abi cp36m --dest /var/folders/s5/_787jljn6l14fhg2w4wn4d_w0000gn/T/tmplzw2zx5z scipy==1.4.1
compatible wheels after second download pass: {markdown==3.2.2(wheel), setuptools==47.1.1(wheel), tensorboard==2.2.2(wheel), tensorboard-plugin-wit==1.6.0.post3(wheel), scipy==1.4.1(wheel), chardet==3.0.4(wheel), google-auth-oauthlib==0.4.1(wheel), h5py==2.10.0(wheel), opt-einsum==3.2.1(wheel), certifi==2020.4.5.1(wheel), oauthlib==3.1.0(wheel), protobuf==3.12.2(wheel), google-auth==1.16.0(wheel), zipp==3.1.0(wheel), six==1.15.0(wheel), tensorflow-estimator==2.2.0(wheel), importlib-metadata==1.6.0(wheel), gast==0.3.3(wheel), cachetools==4.1.0(wheel), pyasn1==0.4.8(wheel), google-pasta==0.2.0(wheel), requests-oauthlib==1.3.0(wheel), urllib3==1.25.9(wheel), pyasn1-modules==0.2.8(wheel), astunparse==1.6.3(wheel), keras-preprocessing==1.1.2(wheel), idna==2.9(wheel), requests==2.23.0(wheel), rsa==4.0(wheel), wheel==0.34.2(wheel), numpy==1.18.4(wheel), werkzeug==1.0.1(wheel)}
Build missing wheels from sdists (C compiling True): {termcolor==1.1.0(sdist), wrapt==1.12.1(sdist), absl-py==0.9.0(sdist)}
calling pip wheel --no-deps --wheel-dir /var/folders/s5/_787jljn6l14fhg2w4wn4d_w0000gn/T/tmplzw2zx5z /var/folders/s5/_787jljn6l14fhg2w4wn4d_w0000gn/T/tmplzw2zx5z/termcolor-1.1.0.tar.gz
calling pip wheel --no-deps --wheel-dir /var/folders/s5/_787jljn6l14fhg2w4wn4d_w0000gn/T/tmplzw2zx5z /var/folders/s5/_787jljn6l14fhg2w4wn4d_w0000gn/T/tmplzw2zx5z/wrapt-1.12.1.tar.gz
calling pip wheel --no-deps --wheel-dir /var/folders/s5/_787jljn6l14fhg2w4wn4d_w0000gn/T/tmplzw2zx5z /var/folders/s5/_787jljn6l14fhg2w4wn4d_w0000gn/T/tmplzw2zx5z/absl-py-0.9.0.tar.gz
compatible after building wheels (no C compiling): {markdown==3.2.2(wheel), setuptools==47.1.1(wheel), tensorboard==2.2.2(wheel), tensorboard-plugin-wit==1.6.0.post3(wheel), scipy==1.4.1(wheel), chardet==3.0.4(wheel), google-auth-oauthlib==0.4.1(wheel), h5py==2.10.0(wheel), opt-einsum==3.2.1(wheel), certifi==2020.4.5.1(wheel), oauthlib==3.1.0(wheel), protobuf==3.12.2(wheel), google-auth==1.16.0(wheel), zipp==3.1.0(wheel), six==1.15.0(wheel), tensorflow-estimator==2.2.0(wheel), importlib-metadata==1.6.0(wheel), termcolor==1.1.0(wheel), gast==0.3.3(wheel), cachetools==4.1.0(wheel), pyasn1==0.4.8(wheel), google-pasta==0.2.0(wheel), requests-oauthlib==1.3.0(wheel), urllib3==1.25.9(wheel), pyasn1-modules==0.2.8(wheel), absl-py==0.9.0(wheel), astunparse==1.6.3(wheel), keras-preprocessing==1.1.2(wheel), idna==2.9(wheel), requests==2.23.0(wheel), rsa==4.0(wheel), wheel==0.34.2(wheel), numpy==1.18.4(wheel), werkzeug==1.0.1(wheel)}
Build missing wheels from sdists (C compiling False): {wrapt==1.12.1(sdist)}
calling pip wheel --no-deps --wheel-dir /var/folders/s5/_787jljn6l14fhg2w4wn4d_w0000gn/T/tmplzw2zx5z /var/folders/s5/_787jljn6l14fhg2w4wn4d_w0000gn/T/tmplzw2zx5z/wrapt-1.12.1.tar.gz
compatible after building wheels (C compiling): {markdown==3.2.2(wheel), setuptools==47.1.1(wheel), tensorboard==2.2.2(wheel), tensorboard-plugin-wit==1.6.0.post3(wheel), scipy==1.4.1(wheel), chardet==3.0.4(wheel), google-auth-oauthlib==0.4.1(wheel), h5py==2.10.0(wheel), opt-einsum==3.2.1(wheel), certifi==2020.4.5.1(wheel), oauthlib==3.1.0(wheel), protobuf==3.12.2(wheel), google-auth==1.16.0(wheel), zipp==3.1.0(wheel), six==1.15.0(wheel), tensorflow-estimator==2.2.0(wheel), importlib-metadata==1.6.0(wheel), termcolor==1.1.0(wheel), gast==0.3.3(wheel), wrapt==1.12.1(wheel), cachetools==4.1.0(wheel), pyasn1==0.4.8(wheel), google-pasta==0.2.0(wheel), requests-oauthlib==1.3.0(wheel), urllib3==1.25.9(wheel), pyasn1-modules==0.2.8(wheel), absl-py==0.9.0(wheel), astunparse==1.6.3(wheel), keras-preprocessing==1.1.2(wheel), idna==2.9(wheel), requests==2.23.0(wheel), rsa==4.0(wheel), wheel==0.34.2(wheel), numpy==1.18.4(wheel), werkzeug==1.0.1(wheel)}
Final compatible: {markdown==3.2.2(wheel), setuptools==47.1.1(wheel), tensorboard==2.2.2(wheel), tensorboard-plugin-wit==1.6.0.post3(wheel), scipy==1.4.1(wheel), chardet==3.0.4(wheel), google-auth-oauthlib==0.4.1(wheel), h5py==2.10.0(wheel), opt-einsum==3.2.1(wheel), certifi==2020.4.5.1(wheel), oauthlib==3.1.0(wheel), protobuf==3.12.2(wheel), google-auth==1.16.0(wheel), zipp==3.1.0(wheel), six==1.15.0(wheel), tensorflow-estimator==2.2.0(wheel), importlib-metadata==1.6.0(wheel), termcolor==1.1.0(wheel), gast==0.3.3(wheel), wrapt==1.12.1(wheel), cachetools==4.1.0(wheel), pyasn1==0.4.8(wheel), google-pasta==0.2.0(wheel), requests-oauthlib==1.3.0(wheel), urllib3==1.25.9(wheel), pyasn1-modules==0.2.8(wheel), absl-py==0.9.0(wheel), astunparse==1.6.3(wheel), keras-preprocessing==1.1.2(wheel), idna==2.9(wheel), requests==2.23.0(wheel), rsa==4.0(wheel), wheel==0.34.2(wheel), numpy==1.18.4(wheel), werkzeug==1.0.1(wheel)}
Final incompatible: {tensorflow==2.2.0(wheel), h5py==2.10.0(wheel), protobuf==3.12.2(wheel), wrapt==1.12.1(wheel), numpy==1.18.4(wheel), grpcio==1.29.0(wheel), scipy==1.4.1(wheel)}
Final missing wheels: {tensorflow==2.2.0(wheel), grpcio==1.29.0(wheel)}
PythonPipBuilder:ResolveDependencies failed
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 42, in execute
requirements_path=self.manifest_path,
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 137, in build_dependencies
self._dependency_builder.build_site_packages(requirements_path, artifacts_dir_path, scratch_dir_path)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 201, in build_site_packages
raise MissingDependencyError(packages_without_wheels)
aws_lambda_builders.workflows.python_pip.packager.MissingDependencyError: {tensorflow==2.2.0(wheel), grpcio==1.29.0(wheel)}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/aws_lambda_builders/workflow.py", line 269, in run
action.execute()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 45, in execute
raise ActionFailedError(str(ex))
aws_lambda_builders.actions.ActionFailedError: {tensorflow==2.2.0(wheel), grpcio==1.29.0(wheel)}

Build Failed
Sending Telemetry: {'metrics': [{'commandRun': {'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build', 'duration': 50308, 'exitReason': 'WorkflowFailedError', 'exitCode': 1, 'requestId': '199413b8-0cd6-41f2-8750-b2189bccb2fa', 'installationId': '27166426-9c3e-4c34-93bc-1f3346ed5909', 'sessionId': '08d96ae3-9148-4850-ad9c-87e77e262c0d', 'executionEnvironment': 'CLI', 'pyversion': '3.6.8', 'samcliVersion': '0.52.0'}}]}
HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)
Error: PythonPipBuilder:ResolveDependencies - {tensorflow==2.2.0(wheel), grpcio==1.29.0(wheel)}

Expected result

I expected a successful build.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: MacOS 10.14.4
2. sam --version: SAM CLI, version 0.52.0

[divnrio]

I am currently having the exact same issue with those libraries and SAM. I thought I would check in to see if you had managed to discover the cause, and identified a solution. If not, I will persist and if I manage to find the solution myself - I will be sure to post it here.

[pasppro]

@divnrio I haven't been able to find a work around. Please let me know if you do!

[jackyzha0]

Also having this issue, following!

[pasppro]

Submitted 17 days ago, and no activity that I can see. Not being able to build a project is a real drawback to using the SAM platform.

[jfuss]

@pasppro You will need to build tensorflow separately and either vend in your function code or within a layer.

sam build tried to build something that is known to be compatible with Lambda. We first search for a whl. If one is not available, we attempt to build from source. These failures are commonly seen when the package has c-bindings or dependencies that require c-bindings (can be seen by other "can't resolve dependencies" issues in the repo.

[pasppro]

@jfuss Thanks! I think I see. sam build needs a wheel for tensorflow 2.2.0, but there isn't one. So, I'll need to create one(?)

Can you point me to an example of how to include my local build in the project? I've looked at the Layer examples here, but it's not exactly clear.

[jfuss]

@pasppro Sorry this took me a while to respond. I was digging into some of what sam build does for python (more details below).

To answer the "how to include to include it": The easiest way I can think of is doing something like pip download --only-binary=:all: --platform manylinux2014_x86_64 --implementation cp --abi cp36m --dest <directory> tensorflow should work (though I have not tested but this is what sam build does). The can be in your function code directory. This should make in import-able then (that is more or less what we are doing under the hood). You can also put this into a directory outside of the function, say myproject/tensorflow-layer/ and then create a Layer resouce that points the ContentUri to that directory, and Ref that resource within the Layers property of the Function.

So over the weeks, we have seen a number of "we can't build this X dependency". Nearly all of these have been related to python. Our builder logic claims Lambda only supports the following platforms (whl specifically). So build generally works if a package creates a manylinux1_x86_64 wheel, but packages like Tensorflow are using a newer platform wheel, manylinux2010, and some only package manylinux2014. Reading through PEP 513 (manylinux1) and PEP 599 (manylinux2014), it seems the part we really care about is the glibc versions. PEP 599 states that if the platform supports manylinux2014, then it will support earlier versions as they are meant to be upper bounds. I did some testing on the Python3.8 and Python2.7 runtimes and looks like we meet the min glibc versions of 2.17 or greater of manylinux2014.

So what does this mean?
The tldr: We and should expand scope to manylinux2014.

I have a patch locally that will download the manylinux2014 whls (or if manylinux2014 is not available to download either manylinux2010 or manylinux1). This will require an AWS Lambda Builders release and then an update in SAM CLI. I want to do a little more testing but going to try and get a patch out to that repo.

[pasppro]

@jfuss Thanks! You're the BEST!

[remarkablerocket]

Thanks for looking into this issue, it's been causing us some headaches with another dependency!

It seems glibc-2.17 is available in Amazon Linux as well as Amazon Linux 2, so presumably Python 3.7 will also be fine after this update?

I saw there's a PR out for the lambda builder already. Is there an ETA for when this might be fixed in both packages @jfuss ?

[dan-mba]

There is a PR #2116 that will incorporate the updated Lambda Builder release into SAM

[jfuss]

This was released in v1.0.0.

Note: You will need to update pip to at least 19.3 to support manylinux2014 downloading, otherwise sam build will still fail.

Closing

[tianchu]

@jfuss This issue still persist with SAM CLI v1.0.0 when building with the --use-container option, because the SAM CLI built into amazon/aws-sam-cli-build-image-pythonXX docker images is still 0.53.0.

Steps to reproduce:

docker pull amazon/aws-sam-cli-build-image-python3.7:latest
docker run -it --rm amazon/aws-sam-cli-build-image-python3.7:latest
sam --version # returns "SAM CLI, version 0.53.0"

[therealjeffg]

@tianchu thanks for reporting this, we're going to track it in a separate issue:

https://github.com/awslabs/aws-sam-cli/issues/2134