Aws-cdk: custom-resources integ tests failing

[MrArnoldPalmer]

yarn integ currently fails in the @aws-cdk/custom-resources module with a cloudformation error.

Reproduction Steps

yarn install
cd packages/@aws-cdk/custom-resources
../../../scripts/buildup
yarn run integ

Error Log

 5/9 | 9:09:36 PM | CREATE_FAILED        | AWS::Lambda::Function | AWS679f53fac002430cb0da5b7982bd2287 (AWS679f53fac002430cb0da5b7982bd22872D164C4C) Uploaded file must be a non-empty zip (Service: AWSLambdaInternal; Status Code: 400; Error Code: InvalidParameterValueException; Request ID: 7ad19cda-8432-4743-8473-7f4b292b8003)
        new Function (/Users/valinm/dev/aws-cdk/packages/@aws-cdk/aws-lambda/lib/function.js:71:26)
        \_ SingletonFunction.ensureLambda (/Users/valinm/dev/aws-cdk/packages/@aws-cdk/aws-lambda/lib/singleton-lambda.js:37:16)
        \_ new SingletonFunction (/Users/valinm/dev/aws-cdk/packages/@aws-cdk/aws-lambda/lib/singleton-lambda.js:19:36)
        \_ new AwsCustomResource (/Users/valinm/dev/aws-cdk/packages/@aws-cdk/custom-resources/lib/aws-custom-resource/aws-custom-resource.js:102:26)
        \_ Object.<anonymous> (/Users/valinm/dev/aws-cdk/packages/@aws-cdk/custom-resources/test/aws-custom-resource/integ.aws-custom-resource.js:11:20)
        \_ Module._compile (internal/modules/cjs/loader.js:776:30)
        \_ Object.Module._extensions..js (internal/modules/cjs/loader.js:787:10)
        \_ Module.load (internal/modules/cjs/loader.js:653:32)
        \_ tryModuleLoad (internal/modules/cjs/loader.js:593:12)
        \_ Function.Module._load (internal/modules/cjs/loader.js:585:3)
        \_ Function.Module.runMain (internal/modules/cjs/loader.js:829:12)
        \_ startup (internal/bootstrap/node.js:283:19)
        \_ bootstrapNodeJSCore (internal/bootstrap/node.js:622:3)

Environment

• **CLI Version : 1.27.0 (current master)
• **Framework Version: 1.27.0 (current master)
• **OS : MacOS 10.14
• **Language : Typescript

Other

As far as I can tell, the code in custom-resources where this is coming from is pointing to an asset that has valid content. This could be an issue with assets.

recent assets change though not immediately clear if this is causing this.
https://github.com/aws/aws-cdk/commit/d5575926b25cf0df8ea651041c426814153b6647

---

This is :bug: Bug Report

[jogold]

The problem is here:
https://github.com/aws/aws-cdk/blob/8da9e1eb07e5cc60b421112df543b4bb9db703e8/packages/cdk-assets/lib/private/handlers/files.ts#L58

Here this.workDir is cdk.out and this.asset.source.path is an absolute path to the asset. This result in a non-existant path and an empty zip file uploaded to S3.

The cdk.out comes from here stack.assembly.directory:
https://github.com/aws/aws-cdk/blob/8da9e1eb07e5cc60b421112df543b4bb9db703e8/packages/aws-cdk/lib/api/deploy-stack.ts#L132

I would say that using this.asset.source.path in cdk-assets/lib/private/handlers/files.ts would fix this since it's an absolute path already... but from the JSDoc it's expected to be relative...

Note also that once you deployed it with an empty zip file it will always reuse that empty file unless you manually remove it from S3.
https://github.com/aws/aws-cdk/blob/8da9e1eb07e5cc60b421112df543b4bb9db703e8/packages/cdk-assets/lib/private/handlers/files.ts#L38-L41

@rix0rrr

[rix0rrr]

Oops

[rix0rrr]

Thanks for debugging @jogold !

[rix0rrr]

Here this.workDir is cdk.out and this.asset.source.path is an absolute path to the asset.

Aha. So I assumed (without verifying) that path.join() would do the sane thing if the 2nd argument was absolute (like Python's os.path.join()), which it totally doesn't.

[rix0rrr]

Cool. The solution seems to be to use path.resolve() everywhere path.join() is being used.

[rix0rrr]

What I don't understand is why the path in the asset file would be absolute. That actually shouldn't be happening.

[jogold]

What I don't understand is why the path in the asset file would be absolute. That actually shouldn't be happening.

I find this also weird. Moreover, it's the absolute path to original asset path and not the staged path. Looks like there's something wrong with the asset staging...

[rix0rrr]

Running the command that @MrArnoldPalmer posted above doesn't reproduce it for me. @jogold do you know how to repro?

[rix0rrr]

This might do it: https://github.com/aws/aws-cdk/blob/master/packages/@aws-cdk/assets/lib/staging.ts#L62

[jogold]

@rix0rrr when you log this.assets here
https://github.com/aws/aws-cdk/blob/d4e17b1e2891e82a6797c8cb0454638cb43ec794/packages/%40aws-cdk/cx-api/lib/cloudformation-artifact.ts#L73

you get this:

[
  {
    "path": "asset.4317eb4c8ccce73f91d75018d965c243c175c64101884e0f202d4a280e23313f",
    "id": "4317eb4c8ccce73f91d75018d965c243c175c64101884e0f202d4a280e23313f",
    "packaging": "zip",
    "sourceHash": "4317eb4c8ccce73f91d75018d965c243c175c64101884e0f202d4a280e23313f",
    "s3BucketParameter": "AssetParameters4317eb4c8ccce73f91d75018d965c243c175c64101884e0f202d4a280e23313fS3Bucket71E9DB75",
    "s3KeyParameter": "AssetParameters4317eb4c8ccce73f91d75018d965c243c175c64101884e0f202d4a280e23313fS3VersionKeyADF76D6F",
    "artifactHashParameter": "AssetParameters4317eb4c8ccce73f91d75018d965c243c175c64101884e0f202d4a280e23313fArtifactHash7FE12709"
  }
]

the asset is correctly staged during an integ test and the path is relative

but when you log stack.assets here before the call:
https://github.com/aws/aws-cdk/blob/d5575926b25cf0df8ea651041c426814153b6647/packages/aws-cdk/lib/api/deploy-stack.ts#L105

you get

[
  {
    "path": "/c/Users/Jonathan/Documents/dev/aws-cdk/packages/@aws-cdk/custom-resources/lib/aws-custom-resource/runtime",
    "id": "4317eb4c8ccce73f91d75018d965c243c175c64101884e0f202d4a280e23313f",
    "packaging": "zip",
    "sourceHash": "4317eb4c8ccce73f91d75018d965c243c175c64101884e0f202d4a280e23313f",
    "s3BucketParameter": "AssetParameters4317eb4c8ccce73f91d75018d965c243c175c64101884e0f202d4a280e23313fS3Bucket71E9DB75",
    "s3KeyParameter": "AssetParameters4317eb4c8ccce73f91d75018d965c243c175c64101884e0f202d4a280e23313fS3VersionKeyADF76D6F",
    "artifactHashParameter": "AssetParameters4317eb4c8ccce73f91d75018d965c243c175c64101884e0f202d4a280e23313fArtifactHash7FE12709"
  }
]

somehow the absolute path of the asset reappears

[jogold]

Rebuilding/installing everything again to make sure I can reproduce. I am on 8da9e1eb07e5cc60b421112df543b4bb9db703e8.

[jogold]

On #6742 the path.resolve() hides the problem. This doesn't explain why this.asset.source.path is not a relative path to the staged asset...

https://github.com/aws/aws-cdk/blob/1579da3ee19ed12398f159daaf8868c888efe7e7/packages/cdk-assets/lib/private/handlers/files.ts#L58

[jogold]

Found it!

https://github.com/aws/aws-cdk/blob/a75f711aea3dac83c6feec885b7df9b14a39486e/tools/cdk-integ-tools/bin/cdk-integ.ts#L34

Staging is disabled for integ test during deploy only. When the integ test runs ls first to determine the test stack its not disabled so when you log the asset at this point you get a relative path and then when it deploys its an absolute path. Maybe both set of args (for deploy and ls in the integ test) should be identical... because this creates staged files that are never used.

This explains the path.resolve() fix. But maybe the logic should be based on the fact that staging is enabled or not (or at least explained in a comment).

[mattchrist]

I'm still unable to run integration tests for custom-resources as of 50280092ab2c0a2c8e19177c4a70f8a7c0f3c5fb

18/28 | 1:28:44 PM | CREATE_FAILED | AWS::Lambda::Function | com.amazonaws.cdk.custom-resources.s3assert-provider/s3assert-provider/framework-isComplete (comamazonawscdkcustomresourcess3assertproviderframeworkisComplete63829575) Uploaded file must be a non-empty zip (Service: AWSLambdaInternal; Status Code: 400; Error Code: InvalidParameterValueException; Request ID: e9fc4e5a-45f8-482e-9e43-3ed95ab4c6ba)

[jogold]

@mattchrist did you manually remove the empty zip file from S3 before retrying?

[mattchrist]

@jogold I ran git clean -fdx, would that do it?

edit: of course not. I'll try to clean the s3 bucket

[mattchrist]

@jogold That did the trick, thank you!