Aws-sam-cli: Error: PythonPipBuilder:ResolveDependencies - {pyrsistent==0.16.0(sdist)}

[brysontyrrell]

Description

Possibly related to #1840 but discovered while using new features in 0.51.0.

I implemented Metadata feature for Layers to allow sam build to install requirements.

    Metadata:
      BuildMethod: python3.8

When running sam build --use-container on the template I am getting this error:

Error: PythonPipBuilder:ResolveDependencies - {pyrsistent==0.16.0(sdist)}

The Layer triggering this error is installing the following packages:

jsonschema==3.2.0
PyJWT==1.7.1

If I downgrade from 0.51.0 to 0.50.0 which does not include the updates for supporting Metadata on Lambda Functions the build command completes without error.

Steps to reproduce

Install SAM CLI 0.51.0.

Add Metadata build option as shown above to a Lambda Layer in a template.

Build the template.

Observed result

This test template contains multiple Layers - only one is triggering the error.

 sam build --use-container --debug
Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
'build' command is called
Starting Build inside a container
No Parameters detected in the template
4 resources found in the template
No Parameters detected in the template
Building layer 'AwsXraySdk'
Loading workflow module 'aws_lambda_builders.workflows'
Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
Registering workflow 'GoDepBuilder' with capability 'Capability(language='go', dependency_manager='dep', application_framework=None)'
Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'
Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
Registering workflow 'CustomMakeBuilder' with capability 'Capability(language='provided', dependency_manager=None, application_framework=None)'
Found workflow 'PythonPipBuilder' to support capabilities 'Capability(language='python', dependency_manager='pip', application_framework=None)'
Running workflow 'PythonPipBuilder'
Running PythonPipBuilder:ResolveDependencies
calling pip download -r /Users/brysontyrrell/Development/Eir/src/layers/aws-xray-sdk/requirements.txt --dest /var/folders/h8/y0kv0zw94912x9fhc48v46rm0000gn/T/tmp3b7k058p
Full dependency closure: {aws-xray-sdk==2.5.0(wheel), wrapt==1.12.1(sdist), python-dateutil==2.8.1(wheel), urllib3==1.25.9(wheel), jmespath==0.10.0(wheel), future==0.18.2(sdist), zipp==3.1.0(wheel), jsonpickle==1.4.1(wheel), docutils==0.15.2(wheel), importlib-metadata==1.6.0(wheel), botocore==1.16.15(wheel), six==1.15.0(wheel)}
initial compatible: {aws-xray-sdk==2.5.0(wheel), python-dateutil==2.8.1(wheel), urllib3==1.25.9(wheel), botocore==1.16.15(wheel), jmespath==0.10.0(wheel), zipp==3.1.0(wheel), docutils==0.15.2(wheel), importlib-metadata==1.6.0(wheel), jsonpickle==1.4.1(wheel), six==1.15.0(wheel)}
initial incompatible: {future==0.18.2(sdist), wrapt==1.12.1(sdist)}
Downloading missing wheels: {future==0.18.2(sdist), wrapt==1.12.1(sdist)}
calling pip download --only-binary=:all: --no-deps --platform manylinux1_x86_64 --implementation cp --abi cp38 --dest /var/folders/h8/y0kv0zw94912x9fhc48v46rm0000gn/T/tmp3b7k058p future==0.18.2
calling pip download --only-binary=:all: --no-deps --platform manylinux1_x86_64 --implementation cp --abi cp38 --dest /var/folders/h8/y0kv0zw94912x9fhc48v46rm0000gn/T/tmp3b7k058p wrapt==1.12.1
compatible wheels after second download pass: {aws-xray-sdk==2.5.0(wheel), python-dateutil==2.8.1(wheel), urllib3==1.25.9(wheel), jmespath==0.10.0(wheel), zipp==3.1.0(wheel), jsonpickle==1.4.1(wheel), docutils==0.15.2(wheel), importlib-metadata==1.6.0(wheel), botocore==1.16.15(wheel), six==1.15.0(wheel)}
Build missing wheels from sdists (C compiling True): {future==0.18.2(sdist), wrapt==1.12.1(sdist)}
calling pip wheel --no-deps --wheel-dir /var/folders/h8/y0kv0zw94912x9fhc48v46rm0000gn/T/tmp3b7k058p /var/folders/h8/y0kv0zw94912x9fhc48v46rm0000gn/T/tmp3b7k058p/future-0.18.2.tar.gz
calling pip wheel --no-deps --wheel-dir /var/folders/h8/y0kv0zw94912x9fhc48v46rm0000gn/T/tmp3b7k058p /var/folders/h8/y0kv0zw94912x9fhc48v46rm0000gn/T/tmp3b7k058p/wrapt-1.12.1.tar.gz
compatible after building wheels (no C compiling): {aws-xray-sdk==2.5.0(wheel), python-dateutil==2.8.1(wheel), urllib3==1.25.9(wheel), jmespath==0.10.0(wheel), future==0.18.2(wheel), zipp==3.1.0(wheel), jsonpickle==1.4.1(wheel), docutils==0.15.2(wheel), importlib-metadata==1.6.0(wheel), botocore==1.16.15(wheel), six==1.15.0(wheel)}
Build missing wheels from sdists (C compiling False): {wrapt==1.12.1(sdist)}
calling pip wheel --no-deps --wheel-dir /var/folders/h8/y0kv0zw94912x9fhc48v46rm0000gn/T/tmp3b7k058p /var/folders/h8/y0kv0zw94912x9fhc48v46rm0000gn/T/tmp3b7k058p/wrapt-1.12.1.tar.gz
compatible after building wheels (C compiling): {aws-xray-sdk==2.5.0(wheel), wrapt==1.12.1(wheel), python-dateutil==2.8.1(wheel), urllib3==1.25.9(wheel), jmespath==0.10.0(wheel), future==0.18.2(wheel), zipp==3.1.0(wheel), jsonpickle==1.4.1(wheel), docutils==0.15.2(wheel), importlib-metadata==1.6.0(wheel), botocore==1.16.15(wheel), six==1.15.0(wheel)}
Final compatible: {aws-xray-sdk==2.5.0(wheel), wrapt==1.12.1(wheel), python-dateutil==2.8.1(wheel), urllib3==1.25.9(wheel), jmespath==0.10.0(wheel), future==0.18.2(wheel), zipp==3.1.0(wheel), jsonpickle==1.4.1(wheel), docutils==0.15.2(wheel), importlib-metadata==1.6.0(wheel), botocore==1.16.15(wheel), six==1.15.0(wheel)}
Final incompatible: {wrapt==1.12.1(wheel)}
Final missing wheels: set()
PythonPipBuilder:ResolveDependencies succeeded
Running PythonPipBuilder:CopySource
PythonPipBuilder:CopySource succeeded
Building layer 'CredStash'
Loading workflow module 'aws_lambda_builders.workflows'
Found workflow 'PythonPipBuilder' to support capabilities 'Capability(language='python', dependency_manager='pip', application_framework=None)'
Running workflow 'PythonPipBuilder'
Running PythonPipBuilder:ResolveDependencies
calling pip download -r /Users/brysontyrrell/Development/Eir/src/layers/credstash/requirements.txt --dest /var/folders/h8/y0kv0zw94912x9fhc48v46rm0000gn/T/tmpcrdfa_oz
Full dependency closure: {boto3==1.13.15(wheel), python-dateutil==2.8.1(wheel), six==1.15.0(wheel), urllib3==1.25.9(wheel), jmespath==0.10.0(wheel), s3transfer==0.3.3(wheel), cffi==1.14.0(wheel), cryptography==2.9.2(wheel), pycparser==2.20(wheel), docutils==0.15.2(wheel), botocore==1.16.15(wheel), credstash==1.17.1(wheel)}
initial compatible: {boto3==1.13.15(wheel), python-dateutil==2.8.1(wheel), urllib3==1.25.9(wheel), jmespath==0.10.0(wheel), credstash==1.17.1(wheel), s3transfer==0.3.3(wheel), pycparser==2.20(wheel), docutils==0.15.2(wheel), botocore==1.16.15(wheel), six==1.15.0(wheel)}
initial incompatible: {cffi==1.14.0(wheel), cryptography==2.9.2(wheel)}
Downloading missing wheels: {cffi==1.14.0(wheel), cryptography==2.9.2(wheel)}
calling pip download --only-binary=:all: --no-deps --platform manylinux1_x86_64 --implementation cp --abi cp38 --dest /var/folders/h8/y0kv0zw94912x9fhc48v46rm0000gn/T/tmpcrdfa_oz cffi==1.14.0
calling pip download --only-binary=:all: --no-deps --platform manylinux1_x86_64 --implementation cp --abi cp38 --dest /var/folders/h8/y0kv0zw94912x9fhc48v46rm0000gn/T/tmpcrdfa_oz cryptography==2.9.2
compatible wheels after second download pass: {boto3==1.13.15(wheel), python-dateutil==2.8.1(wheel), six==1.15.0(wheel), urllib3==1.25.9(wheel), botocore==1.16.15(wheel), jmespath==0.10.0(wheel), s3transfer==0.3.3(wheel), pycparser==2.20(wheel), cryptography==2.9.2(wheel), docutils==0.15.2(wheel), cffi==1.14.0(wheel), credstash==1.17.1(wheel)}
Build missing wheels from sdists (C compiling True): set()
compatible after building wheels (no C compiling): {boto3==1.13.15(wheel), python-dateutil==2.8.1(wheel), six==1.15.0(wheel), urllib3==1.25.9(wheel), botocore==1.16.15(wheel), jmespath==0.10.0(wheel), s3transfer==0.3.3(wheel), pycparser==2.20(wheel), cryptography==2.9.2(wheel), docutils==0.15.2(wheel), cffi==1.14.0(wheel), credstash==1.17.1(wheel)}
Build missing wheels from sdists (C compiling False): set()
compatible after building wheels (C compiling): {boto3==1.13.15(wheel), python-dateutil==2.8.1(wheel), six==1.15.0(wheel), urllib3==1.25.9(wheel), botocore==1.16.15(wheel), jmespath==0.10.0(wheel), s3transfer==0.3.3(wheel), pycparser==2.20(wheel), cryptography==2.9.2(wheel), docutils==0.15.2(wheel), cffi==1.14.0(wheel), credstash==1.17.1(wheel)}
Final compatible: {boto3==1.13.15(wheel), python-dateutil==2.8.1(wheel), six==1.15.0(wheel), urllib3==1.25.9(wheel), botocore==1.16.15(wheel), jmespath==0.10.0(wheel), s3transfer==0.3.3(wheel), pycparser==2.20(wheel), cryptography==2.9.2(wheel), docutils==0.15.2(wheel), cffi==1.14.0(wheel), credstash==1.17.1(wheel)}
Final incompatible: {cffi==1.14.0(wheel), cryptography==2.9.2(wheel)}
Final missing wheels: set()
PythonPipBuilder:ResolveDependencies succeeded
Running PythonPipBuilder:CopySource
PythonPipBuilder:CopySource succeeded
Building layer 'EirSharedLayer'
Loading workflow module 'aws_lambda_builders.workflows'
Found workflow 'PythonPipBuilder' to support capabilities 'Capability(language='python', dependency_manager='pip', application_framework=None)'
Running workflow 'PythonPipBuilder'
Running PythonPipBuilder:ResolveDependencies
calling pip download -r /Users/brysontyrrell/Development/Eir/src/layers/eir_shared/requirements.txt --dest /var/folders/h8/y0kv0zw94912x9fhc48v46rm0000gn/T/tmp3ndwceht
Full dependency closure: {pyrsistent==0.16.0(sdist), six==1.15.0(wheel), pyjwt==1.7.1(wheel), jsonschema==3.2.0(wheel), attrs==19.3.0(wheel), setuptools==46.4.0(wheel)}
initial compatible: {pyjwt==1.7.1(wheel), six==1.15.0(wheel), attrs==19.3.0(wheel), jsonschema==3.2.0(wheel), setuptools==46.4.0(wheel)}
initial incompatible: {pyrsistent==0.16.0(sdist)}
Downloading missing wheels: {pyrsistent==0.16.0(sdist)}
calling pip download --only-binary=:all: --no-deps --platform manylinux1_x86_64 --implementation cp --abi cp38 --dest /var/folders/h8/y0kv0zw94912x9fhc48v46rm0000gn/T/tmp3ndwceht pyrsistent==0.16.0
compatible wheels after second download pass: {pyjwt==1.7.1(wheel), six==1.15.0(wheel), attrs==19.3.0(wheel), jsonschema==3.2.0(wheel), setuptools==46.4.0(wheel)}
Build missing wheels from sdists (C compiling True): {pyrsistent==0.16.0(sdist)}
calling pip wheel --no-deps --wheel-dir /var/folders/h8/y0kv0zw94912x9fhc48v46rm0000gn/T/tmp3ndwceht /var/folders/h8/y0kv0zw94912x9fhc48v46rm0000gn/T/tmp3ndwceht/pyrsistent-0.16.0.tar.gz
compatible after building wheels (no C compiling): {pyjwt==1.7.1(wheel), six==1.15.0(wheel), attrs==19.3.0(wheel), jsonschema==3.2.0(wheel), setuptools==46.4.0(wheel)}
Build missing wheels from sdists (C compiling False): {pyrsistent==0.16.0(sdist)}
calling pip wheel --no-deps --wheel-dir /var/folders/h8/y0kv0zw94912x9fhc48v46rm0000gn/T/tmp3ndwceht /var/folders/h8/y0kv0zw94912x9fhc48v46rm0000gn/T/tmp3ndwceht/pyrsistent-0.16.0.tar.gz
compatible after building wheels (C compiling): {pyjwt==1.7.1(wheel), six==1.15.0(wheel), attrs==19.3.0(wheel), jsonschema==3.2.0(wheel), setuptools==46.4.0(wheel)}
Final compatible: {pyjwt==1.7.1(wheel), six==1.15.0(wheel), attrs==19.3.0(wheel), jsonschema==3.2.0(wheel), setuptools==46.4.0(wheel)}
Final incompatible: {pyrsistent==0.16.0(wheel)}
Final missing wheels: {pyrsistent==0.16.0(sdist)}
PythonPipBuilder:ResolveDependencies failed
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 39, in execute
    package_builder.build_dependencies(
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 137, in build_dependencies
    self._dependency_builder.build_site_packages(requirements_path, artifacts_dir_path, scratch_dir_path)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 201, in build_site_packages
    raise MissingDependencyError(packages_without_wheels)
aws_lambda_builders.workflows.python_pip.packager.MissingDependencyError: {pyrsistent==0.16.0(sdist)}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/aws_lambda_builders/workflow.py", line 269, in run
    action.execute()
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 45, in execute
    raise ActionFailedError(str(ex))
aws_lambda_builders.actions.ActionFailedError: {pyrsistent==0.16.0(sdist)}

Build Failed
Sending Telemetry: {'metrics': [{'commandRun': {'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build', 'duration': 30920, 'exitReason': 'WorkflowFailedError', 'exitCode': 1, 'requestId': 'da2f0647-65be-412e-93c3-44f21977ba04', 'installationId': '03c2f651-de98-424a-814d-b8f65a7593e7', 'sessionId': '959761b2-d6ba-462a-86fa-f4983464dcf3', 'executionEnvironment': 'CLI', 'pyversion': '3.8.0', 'samcliVersion': '0.51.0'}}]}
HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)
Error: PythonPipBuilder:ResolveDependencies - {pyrsistent==0.16.0(sdist)}

Expected result

The Layer is successfully built.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: macOS 10.15.4
2. sam --version: 0.51.0
3. Python: 3.8.0

Add --debug flag to command you are running

Workaround

Downgrade to SAM CLI 0.50.0.

[sriram-mv]

This seems like an inherent issue with the python-pip builder thats present in aws-lambda-builders. Curious that this doesnt work on the container though.

[c2tarun]

Hi, @brysontyrrell,
Fix for this is release today in v0.52.0. I am closing this issue, please re-open if you have question or still experiencing some issues.

Thanks

[rp21buzz]

@c2tarun
I am still facing this issue with --use-container in 0.52.0 but for package boto3

[GarimaBathla]

@c2tarun I upgraded to latest sam version

gb186037$ sam --version
SAM CLI, version 0.53.0

And I am seeing same error

Error: PythonPipBuilder:ResolveDependencies - {pyrsistent==0.16.0(sdist)}

I have manually installed pyrsistent but it ain't working.

I cannot use container builds , what is the fix here?

[silverfoxf7]

@GarimaBathla try installing pip install wheel accord. https://stackoverflow.com/questions/57569718/sam-build-only-completes-successfully-with-use-container-parameter – worked for me