Aws-cdk: Custom::CDKBucketDeployment failed to create

[dstroppa]

• I'm submitting a ...

  • [X] :beetle: bug report
  • [ ] :rocket: feature request
  • [ ] :books: construct library gap
  • [ ] :phone: security issue or vulnerability => Please see policy
  • [ ] :question: support request => Please see note at the top of this template.

• What is the current behavior?
  If the current behavior is a :beetle:bug:beetle:: Please provide the steps to reproduce

I'm using a Custom::CDKBucketDeployment resource to deploy a static website to a S3 bucket:

// A CDK helper that takes the defined source directory, compresses it, and uploads it to the destination s3 bucket.
    new s3deploy.BucketDeployment(this, "DeployWebsite", {
      source: s3deploy.Source.asset(webAppRoot),
      destinationKeyPrefix: "web/",
      destinationBucket: bucket,
      retainOnDelete: false
    });

When running the cdk deploy command this fails with:

9/10 | 3:50:56 PM | CREATE_FAILED        | Custom::CDKBucketDeployment                     | DeployWebsite/CustomResource/Default (DeployWebsiteCustomResourceD116527B) Failed to create resource. Command '['python3', '/var/task/aws', 's3', 'cp', 's3://cdktoolkit-stagingbucket-198n7ki0ng3ti/assets/MythicalMysfitsWebsiteDeployWebsiteAsset1A926195/2299535aebe61fbe992dffa2c950144f29f963029c9a5b7758b93409fd4e379a.zip', '/tmp/tmpfxnjcuck/archive.zip']' returned non-zero exit status 1.
        new CustomResource (/home/ec2-user/environment/workshop/cdk/node_modules/@aws-cdk/aws-cloudformation/lib/custom-resource.ts:92:21)
        \_ new BucketDeployment (/home/ec2-user/environment/workshop/cdk/node_modules/@aws-cdk/aws-s3-deployment/lib/bucket-deployment.ts:89:5)
        \_ new WebApplicationStack (/home/ec2-user/environment/workshop/cdk/lib/web-application-stack.ts:50:5)
        \_ Object.<anonymous> (/home/ec2-user/environment/workshop/cdk/bin/cdk.ts:7:1)
        \_ Module._compile (internal/modules/cjs/loader.js:776:30)
        \_ Module.m._compile (/home/ec2-user/environment/workshop/cdk/node_modules/ts-node/src/index.ts:473:23)
        \_ Module._extensions..js (internal/modules/cjs/loader.js:787:10)
        \_ Object.require.extensions.(anonymous function) [as .ts] (/home/ec2-user/environment/workshop/cdk/node_modules/ts-node/src/index.ts:476:12)
        \_ Module.load (internal/modules/cjs/loader.js:653:32)
        \_ tryModuleLoad (internal/modules/cjs/loader.js:593:12)
        \_ Function.Module._load (internal/modules/cjs/loader.js:585:3)
        \_ Function.Module.runMain (internal/modules/cjs/loader.js:829:12)
        \_ Object.<anonymous> (/home/ec2-user/environment/workshop/cdk/node_modules/ts-node/src/bin.ts:158:12)
        \_ Module._compile (internal/modules/cjs/loader.js:776:30)
        \_ Object.Module._extensions..js (internal/modules/cjs/loader.js:787:10)
        \_ Module.load (internal/modules/cjs/loader.js:653:32)
        \_ tryModuleLoad (internal/modules/cjs/loader.js:593:12)
        \_ Function.Module._load (internal/modules/cjs/loader.js:585:3)
        \_ Function.Module.runMain (internal/modules/cjs/loader.js:829:12)
        \_ findNodeScript.then.existing (/home/ec2-user/.nvm/versions/node/v10.16.0/lib/node_modules/npm/node_modules/libnpx/index.js:268:14)

• What is the expected behavior (or behavior of feature suggested)?
  Expected behavior is that the static website content is deployed onto the S3 bucket.

• What is the motivation / use case for changing the behavior or adding this feature?
  N/A

• Please tell us about your environment:

  • CDK CLI Version: 1.4.0
  • Module Version: 1.4.0
  • OS: all
  • Language: TypeScript (but I expect all to be affected)

• Other information (e.g. detailed explanation, stacktraces, related issues, suggestions how to fix, links for us to have context, eg. associated pull-request, stackoverflow, gitter, etc)

  • Downgrading the @aws-cdk/aws-s3-deployment module to version 1.3.0 fixes the issue.
  • The full code I'm using is at https://github.com/aws-samples/aws-modern-application-workshop/tree/python-cdk/module-1/cdk

[tetsuya-zama]

I've faced almost same problem on typescript aws-cdk. @aws-cdk/[email protected] is OK, but 1.4.0 fails to creating Custom Resource.

In my cloudwatch logs, following error is written.

[INFO]  2019-08-14T22:28:22.879Z    755cf3a1-def9-448e-a29b-74a016a8e842    | aws s3 cp s3://cdktoolkit-stagingbucket-1mv12w5d3n5e4/assets/S3DeploymentInspectionStackDeployWebsiteAssetA4106BD7/65824631bfe4ddf945b765352ff8e862e4211f3148a4c563c9959db18a788e82.zip /tmp/tmpjxj14jwa/archive.zip
Traceback (most recent call last):
File "/var/task/aws", line 27, in <module>
sys.exit(main())
File "/var/task/aws", line 23, in main
return awscli.clidriver.main()
File "/var/task/awscli/clidriver.py", line 58, in main
driver = create_clidriver()
File "/var/task/awscli/clidriver.py", line 68, in create_clidriver
event_hooks=session.get_component('event_emitter'))
File "/var/task/awscli/plugin.py", line 44, in load_plugins
modules = _import_plugins(plugin_mapping)
File "/var/task/awscli/plugin.py", line 61, in _import_plugins
module = __import__(path, fromlist=[module])
File "/var/task/awscli/handlers.py", line 27, in <module>
from awscli.customizations.cloudformation import initialize as cloudformation_init
File "/var/task/awscli/customizations/cloudformation/__init__.py", line 13, in <module>
from awscli.customizations.cloudformation.package import PackageCommand
File "/var/task/awscli/customizations/cloudformation/package.py", line 22, in <module>
from awscli.customizations.cloudformation.artifact_exporter import Template
File "/var/task/awscli/customizations/cloudformation/artifact_exporter.py", line 26, in <module>
from awscli.customizations.cloudformation.yamlhelper import yaml_dump, \
File "/var/task/awscli/customizations/cloudformation/yamlhelper.py", line 14, in <module>
import yaml
ModuleNotFoundError: No module named 'yaml'
[ERROR] 2019-08-14T22:28:32.189Z    755cf3a1-def9-448e-a29b-74a016a8e842    Command '['python3', '/var/task/aws', 's3', 'cp', 's3://cdktoolkit-stagingbucket-1mv12w5d3n5e4/assets/S3DeploymentInspectionStackDeployWebsiteAssetA4106BD7/65824631bfe4ddf945b765352ff8e862e4211f3148a4c563c9959db18a788e82.zip', '/tmp/tmpjxj14jwa/archive.zip']' returned non-zero exit status 1.

According to this, there must be no pyyaml module or something.

[dehli]

I'm also experiencing this in 1.4.0 and it was previously working when I used 1.3.0. Gonna investigate.

After investigating, it seems like the biggest change to the CDKBucketDeployment was https://github.com/aws/aws-cdk/pull/3213. Nothing in that PR stood out to me though. For now I just went back to 1.3 and was able to deploy again.

[eladb]

Thanks for reporting. I’ll look into this today.

[eladb]

Confirming this is a bug, we have a repro.

[hoegertn]

@eladb any plans for a v1.4.1 with this fix?

[carlosrfernandez]

@eladb any plans for a v1.4.1 with this fix?

Wouldn't mind this at all

[jewelsjacobs]

Ditto 😄

[schof]

I'm using the 1.3.0 version and it appears to be working but it's super slow. Taking 10+ minutes to deploy a handful of files. Was there also a performance issue that was addressed?

[jewelsjacobs]

Just wanted to reference this issue about downgrading to 1.3.0:
For me, due to https://github.com/aws/aws-cdk/issues/3711 downgrading as a workaround was not an option.

Since my use case is using a pipeline to build static files and publish them to a website directory, I am creating the website s3 bucket before the project I build my files with and then adding an aws s3 sync ... in the buildspec.yaml in the project Im building my files with.

[KnisterPeter]

This blocks us from creating deployments right now. It should be marked as critical.
I would welcome a 1.4.1 release as well.

[floydding]

Hi do we have a timeline for 1.4.1? thanks!

[KnisterPeter]

@eladb @skinny85 As mentioned in #3711 we could not downgrade and this is really critical for the use of the CDK. Any update from you on a bugfix release?

[dehli]

@KnisterPeter 1.5.0 was merged in (https://github.com/aws/aws-cdk/commit/c020efa2c21bcc7411c06640ada627b06fa86390) so I"m sure that release will come out shortly.

Edit: And it's live 🎉

[carlosrfernandez]

Awesome!

[mcalello]

I'm encountering this same error in python on 1.22.0 in an enterprise environment where all new IAM roles require a boundary policy - so I have to pre-create the role and supply it to BucketDeployment.

Is there a minimum set of permissions I need to include in this role to prevent this error?

[MrArnoldPalmer]

@mcalello you can see the policies added to the role and check that you're pre-existing one has these available. If the role is mutable the construct should be adding these already.