Aws-cdk: Obscure error message 'Bucket named 'bucket-name' exists, but not in account [object Object]. Wrong account?'

[pmarrone]

When running CDK deploy trying to deploy assets in preexisting CDK buckets, created by the bootstrap process without the necessary s3:getBucketLocation permission, the deployment fails with a message like
Obscure error message 'Bucket named 'bucket-name' exists, but not in account [object Object]. Wrong account?'
Notice that the bucket does exist in the account, it's just a permission issue.

Reproduction Steps

Theoretical yet, but it should happen when trying to make a deploy to an account which already contains the assets being deployed in S3 from a previous bootstrap (at least there's where we saw it happening).

It's very likely that this can be reproduce by doing the same as in https://github.com/aws/aws-cdk/issues/4039, but only removing the s3:getBucketLocation permission.

Error Log

'Bucket named 'bucket-name' exists, but not in account [object Object]. Wrong account?'

Environment

• CLI Version :
• **Framework Version: 1.1.0
• **OS : Linux x64
• **Language : English

Other

Checking the code in https://github.com/aws/aws-cdk/blob/a75f711aea3dac83c6feec885b7df9b14a39486e/packages/cdk-assets/lib/private/handlers/files.ts you can see that when there is a permissions error, this message will be shown. Also, account is sent as an object and it stringifies into [objest Object]

---

This is :bug: Bug Report

[iliapolo]

Hi @pmarrone

I'm not sure i follow, are you running the deploy with an assumed role? Can you share how exactly? Are you runing the cdk commands from within an ec2 instance?

[iliapolo]

Hi @pmarrone - scratch my previous message, I understand what's happening. Indeed, the message is not clear enough.

Thanks for reporting this!

[pmarrone]

Hi @iliapolo . I'm trying to find how to reproduce this. Our problem was the following:
We deploy a codepipeline using the cdk with a role with admin privileges. One of the codepipe steps invokes the cdk itself to redeploy changed assets. Before, this step was run with admin rights (:).
While tightening the permissions for this step, the s3 privileges were reduced to the following

new iam.PolicyStatement({
   resources: ['arn:aws:s3:::cdktoolkit-stagingbudket-*'],
   actions: ['s3:*Object', 's3:ListBucket'],
})

This was not enough as it triggers the described error. It appears that this error response happens when the role has these permissions but lacks the s3:getBucketLocation permission.

This seems to be caused by the call to

async function bucketOwnership(s3: AWS.S3, bucket: string): Promise<BucketOwnership> {
  try {
    await s3.getBucketLocation({ Bucket: bucket }).promise();
    return BucketOwnership.MINE;
  } catch (e) {
    if (e.code === 'NoSuchBucket') { return BucketOwnership.DOES_NOT_EXIST; }
    if (['AccessDenied', 'AllAccessDisabled'].includes(e.code)) { return BucketOwnership.SOMEONE_ELSES_OR_NO_ACCESS; }
    throw e;
  }
}

in https://github.com/aws/aws-cdk/blob/a75f711aea3dac83c6feec885b7df9b14a39486e/packages/cdk-assets/lib/private/handlers/files.ts
Trying to check the bucket access using those permissions will fail with SOMEONE_ELSES_OR_NO_ACCESS value returned, and will be handled with the error message described above.

      case BucketOwnership.SOMEONE_ELSES_OR_NO_ACCESS:
        throw new Error(`Bucket named '${destination.bucketName}' exists, but not in account ${account}. Wrong account?`);

[iliapolo]

Yeah gotcha. So basically would a nicer error message be suffice? In addition to documenting the necessary permissions as mentioned here: https://github.com/aws/aws-cdk/issues/4039

Did I understand correctly?

[iliapolo]

@pmarrone btw i managed to reproduce this easily by simply assuming a role from the command line using aws sts assume-role.

[pmarrone]

Nice. I was thinking if this makes sense. The required permission triggers the error using an action that does not seem to be used elsewhere, just to check the access to the bucket (say, it requires an extra permission to check that the permission exists even when it is not the permission that is required to perform operations on that bucket). Shouldn't this check be attempted only when an operation fails on the S3 bucket, such as publishing the asset? I failed to understand the ownership check.
A nicer error message would be a start, but I wonder if the proper error message would be better displayed when trying to perform the actual publish operation on the bucket.
Does it make sense?

[iliapolo]

Yeah i see what you're saying. Definitely makes sense. Ok, i'm prioritizing this bug and we will address it promptly. Keep an eye out here in the issue.

@rix0rrr Can I have your take on this? It looks like this getLocation operation was recently added, but can actually break existing deployments since it requires a change to the role they might using. Is it just a pre-emptive check?

[cnilak]

Any update on this? I am still getting the same error, I even updated the bucket policy as below::
{
"Version": "2012-10-17",
"Id": "AccessControl",
"Statement": [
{
"Sid": "UpdateBucketPolicy",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::{...acount id..}:root"
},
"Action": "s3:",
"Resource": [
"arn:aws:s3:::bucketnamefordemo",
"arn:aws:s3:::bucketnamefordemo/"
]
}
]
}

[oscarnevarezleal]

I can confirm the error is related to a lack of permissions as @pmarrone mentioned. I had a task definition complaining about the same problem. The policy below worked for my situation.

taskDefinition.addToTaskRolePolicy(
      new iam.PolicyStatement({
        resources: ["arn:aws:s3:::cdktoolkit-stagingbucket-*"],
        actions: ["s3:*Object", "s3:ListBucket", 's3:getBucketLocation'],
      })
    );

[moravcik]

Thank you @oscarnevarezleal, I was missing the s3:getBucketLocation action in my case

[iliapolo]

The obscure error message was resolved in https://github.com/aws/aws-cdk/pull/7022/. As far as the permissions, GetBucketLocation is a required action for the deployment. I've created this issue to follow up on documenting these permissions: https://github.com/aws/aws-cdk/issues/9863