Kops: Unable to create elb with guestbook-go example

[chrislovecnm]

An odd problem occurring on master with IAM permissions. I am uncertain if my account is monkeyed but I am unable to run guestbook-go and create a LoadBalancer service. I am getting the error:

  11m       51s 8   service-controller      Normal  CreatingLoadBalancer        Creating load balancer
  51s       51s 1   service-controller      Warning CreatingLoadBalancerFailed  Error creating load balancer (will retry): Failed to create load balancer for service default/guestbook: UnauthorizedOperation: You are not authorized to perform this operation.
        status code: 403, request id: 0c8a48da-c360-4272-b606-97d4192cd2e

The same error is popping up in the k-c-m logs. I am on

$ kops version
Version 1.7.1-beta.2 (git-bd3daf41d)

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.3", GitCommit:"2c2fe6e8278a5db2d15a013987b53968c743f2a1", GitTreeState:"clean", BuildDate:"2017-08-03T07:00:21Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.2", GitCommit:"922a86cfcd65915a9b2f69f3f193b8907d741d9c", GitTreeState:"clean", BuildDate:"2017-07-21T08:08:00Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}



md5-fa671b541a0a294c22972260c7a709f0



 kops create cluster --zones us-east-2c --name $CLUSTER_NAME --yes

I have validated that the IAM permissions look correct, and I have manually created an ELB in the account. So I am uncertain what is going on. Filing this issue to see if someone has a chance to recreate.

[AaronLuo3296]

I also met the same problem while deploying docker-node-helloworld. I'm getting error:

46m 1m 15 service-controller Normal CreatingLoadBalancer Creating load balancer
26m 1m 6 service-controller Warning CreatingLoadBalancerFailed (combined from similar events): Error creating load balancer (will retry): Failed to create load balancer for service default/hello-node-svc: UnauthorizedOperation: You are not authorized to perform this operation.
status code: 403, request id: 0ea2e38b-e511-45be-aeea-b0b9b6d58405

$ kops version
Version 1.7.1-beta.2 (git-bd3daf41)

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.5", GitCommit:"17d7182a7ccbb167074be7a87f0a68bd00d58d97", GitTreeState:"clean", BuildDate:"2017-08-31T19:32:26Z", GoVersion:"go1.9", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.2", GitCommit:"922a86cfcd65915a9b2f69f3f193b8907d741d9c", GitTreeState:"clean", BuildDate:"2017-07-21T08:08:00Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

I deployed a HA cluster with kops:
$ kops create cluster --node-count 3 --zones ap-southeast-2a,ap-southeast-2b,ap-southeast-2c --master-zones ap-southeast-2a,ap-southeast-2b,ap-southeast-2c --node-size t2.micro --master-size t2.micro --topology public ${NAME}

kops user have AmazonEC2FullAccess permissions which includes ELB full access permission. Actually kops has successful created the kubernetes cluster, which has created a ELB.

BTW, in my kops environment, I didn't use public domain, but use a gossip-based cluster.

[AaronLuo3296]

I found it's not the problem of permission of "kops" user in AWS, but the role created by kops. when deploying service, kubernetes process in master-node try to create external ELB with permissions assuming "masters" role (in my cluster the role name is masters.cluster.k8s.local).

When I checked CloudTrail in AWS, there are errors ( I hide my account information with xxxxx):
{
"eventVersion": "1.05",
"userIdentity": {
"type": "AssumedRole",
"principalId": "AROAIGRBKTLAHXOYWWBLS:i-09af3630e77776e11",
"arn": "arn:aws:sts::4525780xxxxx:assumed-role/masters.cluster.k8s.local/i-09af3630e77776e11",
"accountId": "4525780xxxxx",
"accessKeyId": "ASIAJIRXD7VTRUxxxxx",
"sessionContext": {
"attributes": {
"mfaAuthenticated": "false",
"creationDate": "2017-09-06T00:23:58Z"
},
"sessionIssuer": {
"type": "Role",
"principalId": "AROAIGRBKTLAHXOYWWBLS",
"arn": "arn:aws:iam::4525780xxxxx:role/masters.cluster.k8s.local",
"accountId": "4525780xxxxx",
"userName": "masters.cluster.k8s.local"
}
}
},
"eventTime": "2017-09-06T05:04:47Z",
"eventSource": "ec2.amazonaws.com",
"eventName": "CreateSecurityGroup",
"awsRegion": "ap-southeast-2",
"sourceIPAddress": "13.55.131.212",
"userAgent": "aws-sdk-go/1.6.10 (go1.8.3; linux; amd64)",
"errorCode": "Client.UnauthorizedOperation",
"errorMessage": "You are not authorized to perform this operation.",
"requestParameters": {
"groupName": "k8s-elb-a782db2db92bf11e795570aa5169ac26",
"groupDescription": "Security group for Kubernetes ELB a782db2db92bf11e795570aa5169ac26 (default/hello-world-svc)",
"vpcId": "vpc-d82c73bc"
},
"responseElements": null,
"requestID": "48d51964-f7b8-4224-be52-f97afcf393dd",
"eventID": "3420e5fc-83ae-414e-bef4-d6fd4318bef8",
"eventType": "AwsApiCall",
"recipientAccountId": "4525780xxxxx"
}

I checked the permission of the role (arn:aws:iam::4525780xxxxx:role/masters.cluster.k8s.local):
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:Describe"
],
"Resource": [
""
]
},
{
"Effect": "Allow",
"Action": [
"ec2:CreateRoute",
"ec2:CreateTags",
"ec2:CreateVolume",
"ec2:DeleteVolume",
"ec2:ModifyInstanceAttribute"
],
"Resource": [
""
]
},
{
"Effect": "Allow",
"Action": [
"ec2:"
],
"Resource": [
""
],
"Condition": {
"StringEquals": {
"ec2:ResourceTag/KubernetesCluster": "cluster.k8s.local"
}
}
},
{
"Effect": "Allow",
"Action": [
"ecr:GetAuthorizationToken",
"ecr:BatchCheckLayerAvailability",
"ecr:GetDownloadUrlForLayer",
"ecr:GetRepositoryPolicy",
"ecr:DescribeRepositories",
"ecr:ListImages",
"ecr:BatchGetImage"
],
"Resource": [
""
]
},
{
"Effect": "Allow",
"Action": [
"elasticloadbalancing:"
],
"Resource": [
""
]
},
{
"Effect": "Allow",
"Action": [
"autoscaling:DescribeAutoScalingGroups",
"autoscaling:DescribeAutoScalingInstances",
"autoscaling:SetDesiredCapacity",
"autoscaling:TerminateInstanceInAutoScalingGroup"
],
"Resource": [
""
]
},
{
"Effect": "Allow",
"Action": [
"route53:ListHostedZones"
],
"Resource": [
""
]
},
{
"Effect": "Allow",
"Action": [
"s3:GetBucketLocation",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::clusters.kubernetes"
]
},
{
"Effect": "Allow",
"Action": [
"s3:List"
],
"Resource": [
"arn:aws:s3:::clusters.kubernetes/cluster.k8s.local",
"arn:aws:s3:::clusters.kubernetes/cluster.k8s.local/"
]
},
{
"Effect": "Allow",
"Action": [
"s3:Get"
],
"Resource": "arn:aws:s3:::clusters.kubernetes/cluster.k8s.local/"
}
]
}

Seems that there's no permission of security group for this role. Is that the reason of this problem?

[AaronLuo3296]

I've verified that it's the problem of security group for this role. After I added the following permission for this role, it works. I've also verified guestbook, it works too.
{
"Effect": "Allow",
"Action": [
"ec2:CreateSecurityGroup",
"ec2:DescribeSecurityGroups",
"ec2:AuthorizeSecurityGroupIngress",
"ec2:AuthorizeSecurityGroupEgress",
"ec2:DeleteSecurityGroup",
"ec2:RevokeSecurityGroupEgress",
"ec2:RevokeSecurityGroupIngress"
],
"Resource": [
"*"
]

[chrislovecnm]

@justinsb you thought we missed an IAM perm for classic ELB's?

[chrislovecnm]

Should be fixed

[bjethwan]

Which release of kops was this fixed?
I am facing the same issue with kops 1.10.0