Aws-cdk: Support question : Pipenv is not able to download aws cdk core package

[olivier-schmitt]

**Note: for support questions, please first reference our [documentation]

Pipenv is not mentioned in cdk documentation

• I'm submitting a ...

  • [ ] :question: support request => Please see note at the top of this template.

• What is the current behavior?
  If the current behavior is a :beetle:bug:beetle:: Please provide the steps to reproduce

Use the following Pipfile:

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]

[packages]
aws-cdk.core="*"

[requires]
python_version = "3.6"

Create a virtual env with "pipenv --python 3.6" then "pipenv install"

The package is not found but if I remove aws-cdk and add for instance botot3 package in the requires section, it works perfectly.

BTW, a "pip install aws-cdk.core" works perfectly too.

So it seems Pipenv does not like the way the package aws-cdk.core is defined or there is a special name to use.

• What is the expected behavior (or behavior of feature suggested)?

Pipenv finding the package like any package available on pypi.

• Please tell us about your environment:

  • CDK CLI Version: not installed
  • Module Version: core latest
  • OS: [OSX Mojave ]
  • Language: [Python ]

• Other information (e.g. detailed explanation, stacktraces, related issues, suggestions

Spreading Javascript or Typescript code all over the place in the Python documentation is just incredibly misleading.
I'm a bit disappointing by this because I was used to boto3 experience which such an incredible library and has an incredible documentation.

[vaneek]

I too would like to see better support for Pipenv and to be more Pythonic in general. 👍

[garnaat]

The CDK consists of many individual packages, basically one for each service plus some things like core. Each of these are separate Python modules. So, there is no package called aws-cdk on PyPI. That's why pipenv is not able to install aws-cdk. This would be a problem with or without pipenv. You should determine which CDK modules you need for your application and install each module.

I'm not aware of anything that would preclude the use of pipenv with CDK although the current examples use virtualenv. This is mainly because virtualenv is built into Python 3.6+ and does not require any additional software to be installed. I think it would be useful to provide some pipenv-based examples, as well. If there are specific issues using pipenv I would recommend creating new issues for each one.

I agree that the Typescript examples that appear in the README files for Python modules are confusing. We have an existing issue to track this and it is on our roadmap to fix. We aren't able to provide a estimate of when that will happen but I encourage you to up vote the issue.

[brainstorm]

@garnaat I just got started on AWS CDK for Python right now and I concur with @vaneek and @olivier-schmitt's opinions: As a python dev, I would have expected a pip-installable aws_cdk "global" metapackage that installs all the aws_cdk.* dependencies.

I do prefer to have all the CDK-AWS dependencies pre-installed (like other AWS SDKs do: aws-sdk-java-v2, aws-sdk-go-v2, etc..), than dealing with this during my onboarding into CDK:

$ cdk ls
Traceback (most recent call last):
  File "app.py", line 5, in <module>
    from hello.hello_stack import MyStack
  File "/Users/romanvg/dev/umccr/infrastructure/cdk/hello/hello_stack.py", line 1, in <module>
    from aws_cdk import (
ImportError: cannot import name 'aws_iam' from 'aws_cdk' (unknown location)

Instead, I found myself going through the following sequence of commands, fixing the import errors while reading the excellent CDK workshop docs, which don't seem to mention this caveat explicitly:

$ pip install aws_cdk.aws_iam
$ pip install aws_cdk.aws_sqs
$ pip install aws_cdk.aws_sns
$ pip install aws_cdk.aws_sns_subscriptions

Also, the auto-creation of a virtualenv can potentially clash with third party virtualenv systems like Conda/Anaconda, heavily used in scientific environments:

https://www.anaconda.com/
https://docs.conda.io/en/latest/miniconda.html

Just my 2 cents as I explore CDK with a "fresh pair of eyes" ;)

[brainstorm]

Correction: the aforementioned packages are in requirements.txt (ultimately listed in setup.py) from the tutorial as expected. My main point still stands, though: an aws-cdk global metapackage is desirable to have PULA when working with CDK for Python as a python dev.

[brainstorm]

@garnaat I just found one of the reasons why you might not be providing such an aws-cdk metapackage in one line:

pip search aws-cdk.aws | awk '{print $1}' > reqs.txt && pip install -r reqs.txt

And that seems to be inconsistent version pinning across aws-cdk python subcomponents?:

ERROR: aws-cdk-cx-api 1.6.1 has requirement jsii~=0.15.2, but you'll have jsii 0.11.3 which is incompatible.
ERROR: aws-cdk-core 1.6.1 has requirement jsii~=0.15.2, but you'll have jsii 0.11.3 which is incompatible.
ERROR: aws-cdk-region-info 1.6.1 has requirement jsii~=0.15.2, but you'll have jsii 0.11.3 which is incompatible.
ERROR: aws-cdk-aws-iam 1.6.1 has requirement jsii~=0.15.2, but you'll have jsii 0.11.3 which is incompatible.
ERROR: aws-cdk-aws-cloudwatch 1.6.1 has requirement jsii~=0.15.2, but you'll have jsii 0.11.3 which is incompatible.
ERROR: aws-cdk-aws-logs 1.6.1 has requirement jsii~=0.15.2, but you'll have jsii 0.11.3 which is incompatible.
ERROR: aws-cdk-aws-ssm 1.6.1 has requirement jsii~=0.15.2, but you'll have jsii 0.11.3 which is incompatible.
ERROR: aws-cdk-aws-ec2 1.6.1 has requirement jsii~=0.15.2, but you'll have jsii 0.11.3 which is incompatible.
ERROR: aws-cdk-aws-route53 1.6.1 has requirement jsii~=0.15.2, but you'll have jsii 0.11.3 which is incompatible.
ERROR: aws-cdk-aws-events 1.6.1 has requirement jsii~=0.15.2, but you'll have jsii 0.11.3 which is incompatible.
ERROR: aws-cdk-assets 1.6.1 has requirement jsii~=0.15.2, but you'll have jsii 0.11.3 which is incompatible.
ERROR: aws-cdk-aws-kms 1.6.1 has requirement jsii~=0.15.2, but you'll have jsii 0.11.3 which is incompatible.
ERROR: aws-cdk-aws-s3 1.6.1 has requirement jsii~=0.15.2, but you'll have jsii 0.11.3 which is incompatible.

[Taity045]

@garnaat I just found one of the reasons why you might not be providing such an aws-cdk metapackage in one line:

pip search aws-cdk.aws | awk '{print $1}' > reqs.txt && pip install -r reqs.txt

And that seems to be inconsistent version pinning across aws-cdk python subcomponents?:

ERROR: aws-cdk-cx-api 1.6.1 has requirement jsii~=0.15.2, but you'll have jsii 0.11.3 which is incompatible.
ERROR: aws-cdk-core 1.6.1 has requirement jsii~=0.15.2, but you'll have jsii 0.11.3 which is incompatible.
ERROR: aws-cdk-region-info 1.6.1 has requirement jsii~=0.15.2, but you'll have jsii 0.11.3 which is incompatible.
ERROR: aws-cdk-aws-iam 1.6.1 has requirement jsii~=0.15.2, but you'll have jsii 0.11.3 which is incompatible.
ERROR: aws-cdk-aws-cloudwatch 1.6.1 has requirement jsii~=0.15.2, but you'll have jsii 0.11.3 which is incompatible.
ERROR: aws-cdk-aws-logs 1.6.1 has requirement jsii~=0.15.2, but you'll have jsii 0.11.3 which is incompatible.
ERROR: aws-cdk-aws-ssm 1.6.1 has requirement jsii~=0.15.2, but you'll have jsii 0.11.3 which is incompatible.
ERROR: aws-cdk-aws-ec2 1.6.1 has requirement jsii~=0.15.2, but you'll have jsii 0.11.3 which is incompatible.
ERROR: aws-cdk-aws-route53 1.6.1 has requirement jsii~=0.15.2, but you'll have jsii 0.11.3 which is incompatible.
ERROR: aws-cdk-aws-events 1.6.1 has requirement jsii~=0.15.2, but you'll have jsii 0.11.3 which is incompatible.
ERROR: aws-cdk-assets 1.6.1 has requirement jsii~=0.15.2, but you'll have jsii 0.11.3 which is incompatible.
ERROR: aws-cdk-aws-kms 1.6.1 has requirement jsii~=0.15.2, but you'll have jsii 0.11.3 which is incompatible.
ERROR: aws-cdk-aws-s3 1.6.1 has requirement jsii~=0.15.2, but you'll have jsii 0.11.3 which is incompatible.

How did you resolve this ?