Aws-cdk: [aws-eks] `ec2:DescribeVpcs` permission is missing when creating clusters

[eladb]

Today I've also gotten the same error but on the ec2:DescribeVpcs permission. This was not the case on Friday because I rolled out an EKS cluster with a CDK build done from the master branch. Adding the DescribeVpcs permission solved the problem, same as the PR in #8859

Seems like AWS is changing stuff on their end that requires more permissions?

_Originally posted by @jargelo in https://github.com/aws/aws-cdk/issues/8574#issuecomment-654241149_

[jargelo]

Thanks @eladb , I was about to raise a new issue for this since I wasn't sure you seen my comment.

[stefanolczak]

I've got the issue when deploying CDK app from scratch ( VPC and EKS as seperate stacks ). Deployment of EKS stack failed with the same error. During redeploy it worked fine. I am using CDK 1.60

[iliapolo]

@stefanolczak Can you share the code? Thanks

[stefanolczak]

#!/usr/bin/env python3

from aws_cdk import (
    core,
    aws_ec2,
    aws_eks
)


class VpcStack(core.Stack):

    def __init__(self, app: core.App) -> None:
        super().__init__(app, 'vpc-stack')

        self.vpc = aws_ec2.Vpc(
            scope=self,
            id='vpc'
        )


class EksStack(core.Stack):
    def __init__(self, app: core.App, vpc: aws_ec2.Vpc) -> None:
        super().__init__(app, 'eks-stack')

        aws_eks.Cluster(
            scope=self,
            id='eks',
            cluster_name='eks-cluster',
            default_capacity=0,
            version=aws_eks.KubernetesVersion.V1_17,
            vpc=vpc,
            endpoint_access=aws_eks.EndpointAccess.PRIVATE
        )


app = core.App()
vpc_stack = VpcStack(app)
EksStack(app, vpc_stack.vpc)

app.synth()

The issue reproduces only on empty AWS account or on AWS account where CDK or EKS wasn't used for a while. I'm investigating it more why it does matter.

[stefanolczak]

I'm almost sure I have once seen other issue ( also with IAM policy ) related to deploying EKS that was fixed by redeploy so I'm guessing there is some global problem in that matter.

I found the error message from mentioned issue from the past:

Error: Caller does not have permission to perform `iam:listAttachedRolePolicies`

[sandeep-balouria]

Upgraded to 1.60.0
Tried deploying to fresh aws account and failed with this

Failed to create resource. IAM role's policy must include the ec2:DescribeVpcs action

redeploy goes through fine.

[kossmoboleat]

@eladb This issue might have to be reopened. I've had the same issue with CDK 1.66.0 yesterday when trying to deploy a stack from scratch too.

The first try to deploy fails with the mentioned error message but a re-deploy works. Seems the CDK cannot clean up all created stacks/resources and then those are re-used and it succeeds.

I've also checked and the creation role now has the necessary permissions. So I'm speculating the policy is added to late with the changes in the MR when deploying from scratch.

In our stack we're using a shared VPC with another AWS account that we're importing like so:

    this.vpc = Vpc.fromLookup(this, this.vpcName + '-VPC', {
      tags: {
        Name: this.vpcName
      }
    });

And then using it like so to create the EKS cluster (which fails the first time around):

    const clusterAdmin = new Role(this, 'AdminRole', {
      assumedBy: new AccountRootPrincipal()
    });
    const clusterProps: ClusterProps = {
      version: KubernetesVersion.V1_17,
      outputClusterName: true,
      kubectlEnabled: true,
      defaultCapacity: 1,
      defaultCapacityInstance: new InstanceType("t2.small"),
      mastersRole: clusterAdmin,
      vpc: this.vpc
    };

    return new Cluster(this, name, clusterProps);

[iliapolo]

Hi @kossmoboleat - Thanks, yeah looks like there is still something here. Re-opening and we'll investigate.

[scalasm]

I've got the same issue with CDK 1.67.0 . If you need a simple test project, just use this on GitHub.

You can see the file log in aws-cdk-issue-9027.log . I can confirm what @kossmoboleat: if I run 'cdk deploy' just after this error, then the deploy is succesful :|

[PerArneng]

I also ran in to this problem when setting up a test cluster. https://github.com/PerArneng/eks-cdk-test/tree/issue_9027 . I ran it against a blank account and then it worked on the second attempt. I also got an email from AWS saying that i was granted access to AWS resources You recently requested an AWS Service that required additional validation. Your request has now been validated for AWS Europe (Stockholm) region(s)

Edit: In the branch above the instance type should be 't3.nano' because 't2.nano' is not available in that region

[aakrem]

Still happening with 1.76.0 and redeploying the stack fixed it

[iliapolo]

Managed to reproduce this on a blank account as well, and also got the email that @PerArneng mentioned.

I believe the error is coming from a validation that EKS makes on the cluster role, and not necessarily from an operation that requires this action.

What happens is that the cluster role we pass gets created by default like so:

https://github.com/aws/aws-cdk/blob/8216738dc4a496a4114bad46b4a11bc1e205d8bc/packages/%40aws-cdk/aws-eks/lib/cluster.ts#L911-L916

This AmazonEKSClusterPolicy does contain the ec2:DescribeVpcs action, but I suspect that on a blank account, granting usage of a managed policy is what requires this additional validation.

I'll do some internal investigation to see what the best path forward here.

[kay-is]

I got the same problem.

A deployment takes 30minutes, so re-deploying isn't a viable workaround for me.

Any other solutions?