Aws-cdk: (core): app.synth 1.75.0 performance degradation

[mannickutd]

The bug appears in the slow down of the cdk commands 'cdk ls, cdk diff, cdk deploy'. When killing the process the traceback would include the app.synth() function in the code.

Reproduction Steps

Replicating the environment with the latest cdk library 1.75.0 our existing cdk python code base performance goes from several minutes to deploy to 10s of minutes to deploy. It appears that any command that runs the app.synth() (app = core.App()) function in the codebase degrades the performance. There is no error thrown, eventually the function completes.

Environment

• CDK CLI Version : 1.75.0
• Framework Version: Python
• Node.js Version:
• OS : Locally mac, but also duplicated on build machines ubuntu-20.04 and ubuntu-18.04
• Language (Version): Python 3.7.9

pip freeze

attrs==20.3.0
aws-adfs==1.24.4
aws-cdk.assets==1.75.0
aws-cdk.aws-apigateway==1.75.0
aws-cdk.aws-apigatewayv2==1.75.0
aws-cdk.aws-applicationautoscaling==1.75.0
aws-cdk.aws-autoscaling==1.75.0
aws-cdk.aws-autoscaling-common==1.75.0
aws-cdk.aws-autoscaling-hooktargets==1.75.0
aws-cdk.aws-batch==1.75.0
aws-cdk.aws-certificatemanager==1.75.0
aws-cdk.aws-cloudformation==1.75.0
aws-cdk.aws-cloudfront==1.75.0
aws-cdk.aws-cloudwatch==1.75.0
aws-cdk.aws-codebuild==1.75.0
aws-cdk.aws-codecommit==1.75.0
aws-cdk.aws-codeguruprofiler==1.75.0
aws-cdk.aws-codepipeline==1.75.0
aws-cdk.aws-cognito==1.75.0
aws-cdk.aws-ec2==1.75.0
aws-cdk.aws-ecr==1.75.0
aws-cdk.aws-ecr-assets==1.75.0
aws-cdk.aws-ecs==1.75.0
aws-cdk.aws-ecs-patterns==1.75.0
aws-cdk.aws-efs==1.75.0
aws-cdk.aws-elasticloadbalancing==1.75.0
aws-cdk.aws-elasticloadbalancingv2==1.75.0
aws-cdk.aws-events==1.75.0
aws-cdk.aws-events-targets==1.75.0
aws-cdk.aws-iam==1.75.0
aws-cdk.aws-kinesis==1.75.0
aws-cdk.aws-kinesisfirehose==1.75.0
aws-cdk.aws-kms==1.75.0
aws-cdk.aws-lambda==1.75.0
aws-cdk.aws-logs==1.75.0
aws-cdk.aws-route53==1.75.0
aws-cdk.aws-route53-targets==1.75.0
aws-cdk.aws-s3==1.75.0
aws-cdk.aws-s3-assets==1.75.0
aws-cdk.aws-sam==1.75.0
aws-cdk.aws-secretsmanager==1.75.0
aws-cdk.aws-servicediscovery==1.75.0
aws-cdk.aws-sns==1.75.0
aws-cdk.aws-sns-subscriptions==1.75.0
aws-cdk.aws-sqs==1.75.0
aws-cdk.aws-ssm==1.75.0
aws-cdk.aws-stepfunctions==1.75.0
aws-cdk.cloud-assembly-schema==1.75.0
aws-cdk.core==1.75.0
aws-cdk.custom-resources==1.75.0
aws-cdk.cx-api==1.75.0
aws-cdk.region-info==1.75.0
awscli==1.18.185
boto3==1.14.37
botocore==1.19.25
cattrs==1.1.1
Cerberus==1.3.1
certifi==2020.6.20
cffi==1.14.0
chardet==3.0.4
click==7.1.2
colorama==0.4.3
configparser==5.0.0
constructs==3.2.44
cryptography==2.9.2
docutils==0.15.2
fido2==0.8.1
idna==2.10
jmespath==0.10.0
jsii==1.15.0
lxml==4.5.1
publication==0.0.3
pyasn1==0.4.8
pycparser==2.20
pykerberos==1.2.1
pyOpenSSL==19.1.0
python-dateutil==2.8.1
PyYAML==5.1.2
requests==2.24.0
requests-kerberos==0.12.0
rsa==3.4.2
s3transfer==0.3.3
six==1.15.0
typing-extensions==3.7.4.2
urllib3==1.25.9

---

Other

The previous version that worked fine
cdk version 1.74.0
pip freeze

---

attrs==19.3.0
aws-adfs==1.24.4
aws-cdk.assets==1.74.0
aws-cdk.aws-apigateway==1.74.0
aws-cdk.aws-apigatewayv2==1.74.0
aws-cdk.aws-applicationautoscaling==1.74.0
aws-cdk.aws-autoscaling==1.74.0
aws-cdk.aws-autoscaling-common==1.74.0
aws-cdk.aws-autoscaling-hooktargets==1.74.0
aws-cdk.aws-batch==1.74.0
aws-cdk.aws-certificatemanager==1.74.0
aws-cdk.aws-cloudformation==1.74.0
aws-cdk.aws-cloudfront==1.74.0
aws-cdk.aws-cloudwatch==1.74.0
aws-cdk.aws-codebuild==1.74.0
aws-cdk.aws-codecommit==1.74.0
aws-cdk.aws-codeguruprofiler==1.74.0
aws-cdk.aws-codepipeline==1.74.0
aws-cdk.aws-cognito==1.74.0
aws-cdk.aws-ec2==1.74.0
aws-cdk.aws-ecr==1.74.0
aws-cdk.aws-ecr-assets==1.74.0
aws-cdk.aws-ecs==1.74.0
aws-cdk.aws-ecs-patterns==1.74.0
aws-cdk.aws-efs==1.74.0
aws-cdk.aws-elasticloadbalancing==1.74.0
aws-cdk.aws-elasticloadbalancingv2==1.74.0
aws-cdk.aws-events==1.74.0
aws-cdk.aws-events-targets==1.74.0
aws-cdk.aws-iam==1.74.0
aws-cdk.aws-kinesis==1.74.0
aws-cdk.aws-kinesisfirehose==1.74.0
aws-cdk.aws-kms==1.74.0
aws-cdk.aws-lambda==1.74.0
aws-cdk.aws-logs==1.74.0
aws-cdk.aws-route53==1.74.0
aws-cdk.aws-route53-targets==1.74.0
aws-cdk.aws-s3==1.74.0
aws-cdk.aws-s3-assets==1.74.0
aws-cdk.aws-sam==1.74.0
aws-cdk.aws-secretsmanager==1.74.0
aws-cdk.aws-servicediscovery==1.74.0
aws-cdk.aws-sns==1.74.0
aws-cdk.aws-sns-subscriptions==1.74.0
aws-cdk.aws-sqs==1.74.0
aws-cdk.aws-ssm==1.74.0
aws-cdk.aws-stepfunctions==1.74.0
aws-cdk.cloud-assembly-schema==1.74.0
aws-cdk.core==1.74.0
aws-cdk.custom-resources==1.74.0
aws-cdk.cx-api==1.74.0
aws-cdk.region-info==1.74.0
awscli==1.18.185
boto3==1.16.25
botocore==1.19.25
cattrs==1.0.0
Cerberus==1.3.1
certifi==2020.6.20
cffi==1.14.0
chardet==3.0.4
click==7.1.2
colorama==0.4.3
configparser==5.0.0
constructs==3.0.28
cryptography==2.9.2
docutils==0.15.2
fido2==0.8.1
idna==2.10
jmespath==0.10.0
jsii==1.8.0
lxml==4.5.1
publication==0.0.3
pyasn1==0.4.8
pycparser==2.20
pykerberos==1.2.1
pyOpenSSL==19.1.0
python-dateutil==2.8.1
PyYAML==5.1.2
requests==2.24.0
requests-kerberos==0.12.0
rsa==3.4.2
s3transfer==0.3.3
six==1.15.0
typing-extensions==3.7.4.2
urllib3==1.26.2

---

---

This is :bug: Bug Report

[rix0rrr]

It would help if I could see the code base that is causing this behavior. It's not open by any chance?

[NetaNir]

Probably related https://github.com/aws/jsii/pull/2298. The next release of the cdk will include the jsii patch

[RomainMuller]

In fact, this can likely be fixed without having to have a CDK release. Once the fix has been verified, we can release a new version of the jsii runtime libraries, which should fix all clients (they'd simply need to make sure their dependencies are re-resolved so the latest runtime is downloaded)

[rix0rrr]

@NetaNir, @RomainMuller, I can only see https://github.com/aws/jsii/pull/2298 causing this if there's a lot of back-and-forth chatter between the Node and Python sides, which app.synth() typically should not trigger. All the work there happens Node-side.

@mannickutd can you set the environment variable JSII_DEBUG=true and run again so we can ascertain that there's a lot of communication between the Node and Python sides and that's what's causing the issue?

[rix0rrr]

Okay I've heard @RomainMuller is fairly sure this is the issue, so perhaps no need to paste the debug log.