Cluster-api-provider-aws: failed to reconcile load balancers - not authorized to perform: ec2:DescribeAccountAttributes

[joejulian]

/kind bug

What steps did you take and what happened:

clusterawsadm alpha bootstrap create-stack
unset AWS_CREDENTIALS AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY && 
for key in $(aws iam list-access-keys --user-name bootstrapper.cluster-api-provider-aws.sigs.k8s.io | jq -r '.AccessKeyMetadata[].AccessKeyId'); do
aws iam delete-access-key --user-name bootstrapper.cluster-api-provider-aws.sigs.k8s.io --access-key-id $key
done &&
export AWS_CREDENTIALS=$(aws iam create-access-key \
  --user-name bootstrapper.cluster-api-provider-aws.sigs.k8s.io)
export AWS_ACCESS_KEY_ID=$(echo $AWS_CREDENTIALS | jq .AccessKey.AccessKeyId -r)
export AWS_SECRET_ACCESS_KEY=$(echo $AWS_CREDENTIALS | jq .AccessKey.SecretAccessKey -r)
cmd/clusterctl/examples/aws/generate-yaml.sh -f
clusterctl create cluster -v 3 --bootstrap-type kind --provider aws -m out/machines.yaml -c out/cluster.yaml -p out/provider-components-dev.yaml -a out/addons.yaml

provider-components uses an image I built from master and pushed to quay for cluster-api-aws-controller.

aws-provider-system/aws-provider-controller-manager-0[manager]: E0306 20:56:34.913901       1 controller.go:134] Error reconciling cluster object demo1; failed to reconcile load balancers for cluster "demo1": failed to create classic load balancer: &{demo1-apiserver  Internet-facing [subnet-087fb7230f4a8ad68] [sg-0fb4f29972f803e69] [0xc00013f830] 0xc00013f890 {10m0s} map[kubernetes.io/cluster/demo1:owned sigs.k8s.io/cluster-api-provider-aws/managed:true sigs.k8s.io/cluster-api-provider-aws/role:apiserver]}: AccessDenied: User: arn:aws:iam::572115487488:user/bootstrapper.cluster-api-provider-aws.sigs.k8s.io is not authorized to perform: ec2:DescribeAccountAttributes

What did you expect to happen:
I expected a usable kubernetes cluster. :)

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• Cluster-api-provider-aws version: master@44dedbb
• Kubernetes version: (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.1", GitCommit:"eec55b9ba98609a46fee712359c7b5b365bdd920", GitTreeState:"clean", BuildDate:"2018-12-13T10:39:04Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}

• OS (e.g. from /etc/os-release):

NAME="Arch Linux"
PRETTY_NAME="Arch Linux"
ID=arch
BUILD_ID=rolling
ANSI_COLOR="0;36"
HOME_URL="https://www.archlinux.org/"
DOCUMENTATION_URL="https://wiki.archlinux.org/"
SUPPORT_URL="https://bbs.archlinux.org/"
BUG_REPORT_URL="https://bugs.archlinux.org/"

[randomvariable]

Not necessary, but would love to see the related CloudTrail events, including the ones surrounding it for this if possible.

[chuckha]

@randomvariable can you expand a little bit on this comment? What's not necessary?

[randomvariable]

I mean, not necessary for close out of this issue as a PR is incoming, I'm more curious as to what triggered it. We don't call it directly, so it's AWS calling it on our behalf as part of another request. CloudTrail should reveal the culprit.

[randomvariable]

For additional context, we've had two instances of this in the past:

489 dealt with a case on completely new accounts that ELBv1 will call iam:CreateServiceLinkedRole to create a aws-service-role/elasticloadbalancing.amazonaws.com/AWSServiceRoleForElasticLoadBalancing role in the user's account if it doesn't exist. Normally done on the first ever request to create a load balancer in an account, and never again.

198 allows EC2 to call iam:PassRole as part of ec2:RunInstances.

[detiber]

/lifecycle active