Eksctl: "cannot list Kubernetes Services: Unauthorized" - which AWS permissions?

[fabboe]

Hi all,
this might relate to #1070 (and the issues that are referenced in that one).

But actually in my case I'm the owner/creator respectively assume the role of the owner.

My versions:

$ eksctl version
[ℹ]  version.Info{BuiltAt:"", GitCommit:"", GitTag:"0.2.1"}

$ aws --version
aws-cli/1.16.208 Python/3.6.8 Linux/4.15.0-54-generic botocore/1.12.198

$ kubectl version  
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.1", GitCommit:"4485c6f18cee9a5d3c3b4e523bd27972b1b53892", GitTreeState:"clean", BuildDate:"2019-07-18T09:18:22Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}

I create the cluster in this way:

aws sts assume-role --role-arn arn:aws:iam::123:role/CreateEKSClustersCloudFormation --duration 3600

eksctl create cluster \
--name test \                                           
--node-type p2.xlarge \
--nodes 4 \
--nodes-min 2 \
--nodes-max 4 \
--cfn-role-arn arn:aws:iam::123:role/CreateEKSClustersCloudFormation \
--ssh-access \
--ssh-public-key ~/.ssh/id_rsa.pub \
--region eu-central-1



md5-b0555d941c7ba60ddcbdc903795c301f



2019-08-02T09:29:35+02:00 [ℹ]  cleaning up LoadBalancer services
2019-08-02T09:29:35+02:00 [▶]  DEBUG: Request eks/DescribeCluster Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /clusters/test HTTP/1.1
Host: eks.eu-central-1.amazonaws.com
User-Agent: eksctl/{"BuiltAt":"","GitTag":"0.2.1"} aws-sdk-go/1.19.18 (go1.12.6; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=AWS_ACCESS_KEY/20190802/eu-central-1/eks/aws4_request, SignedHeaders=content-type;host;x-amz-date, Signature=134iyasd8t123156
Content-Type: application/json
X-Amz-Date: 20190802T072935Z
Accept-Encoding: gzip


-----------------------------------------------------
2019-08-02T09:29:35+02:00 [▶]  DEBUG: Response eks/DescribeCluster Details:
---[ RESPONSE ]--------------------------------------
HTTP/2.0 200 OK
Content-Length: 2268
Content-Type: application/json
Date: Fri, 02 Aug 2019 07:29:35 GMT
X-Amz-Apigw-Id: dyDu6F7rFiAFoeQ=
X-Amzn-Requestid: 471ee689-b4f7-812h-a618-57ahsd87tg12
X-Amzn-Trace-Id: Root=1-5d43e65f-d98hy12973y87tgasd


-----------------------------------------------------
2019-08-02T09:29:35+02:00 [▶]  {"cluster":{"name":"test","arn":"arn:aws:eks:eu-central-1:ARN_ID:cluster/test","createdAt":1564721940.931,"version":"1.11","endpoint":"https://1234.sk1.eu-central-1.eks.amazonaws.com","roleArn":"arn:aws:iam::ARN_ID:role/eksctl-test-cluster-ServiceRole-JABSDY1235","resourcesVpcConfig":{"subnetIds":["subnet-1","subnet-2","subnet-3","subnet-4","subnet-5","subnet-6"],"securityGroupIds":["sg-1"],"vpcId":"vpc-1","endpointPublicAccess":true,"endpointPrivateAccess":false},"logging":{"clusterLogging":[{"types":["api","audit","authenticator","controllerManager","scheduler"],"enabled":false}]},"identity":null,"status":"ACTIVE","certificateAuthority":{"data":"datastr="},"clientRequestToken":null,"platformVersion":"eks.4"}}
2019-08-02T09:29:35+02:00 [▶]  cluster = {
  Arn: "arn:aws:eks:eu-central-1:ARN_ID:cluster/test",
  CertificateAuthority: {
    Data: "datastr="
  },
  CreatedAt: 2019-08-02 04:59:00 +0000 UTC,
  Endpoint: "https://1234.sk1.eu-central-1.eks.amazonaws.com",
  Logging: {
    ClusterLogging: [{
        Enabled: false,
        Types: [
          "api",
          "audit",
          "authenticator",
          "controllerManager",
          "scheduler"
        ]
      }]
  },
  Name: "test",
  PlatformVersion: "eks.4",
  ResourcesVpcConfig: {
    EndpointPrivateAccess: false,
    EndpointPublicAccess: true,
    SecurityGroupIds: ["sg-1"],
    SubnetIds: [
      "subnet-1",
      "subnet-2",
      "subnet-3",
      "subnet-4",
      "subnet-5",
      "subnet-6"
    ],
    VpcId: "vpc-1"
  },
  RoleArn: "arn:aws:iam::ARN_ID:role/eksctl-test-cluster-ServiceRole-JABSDY1235",
  Status: "ACTIVE",
  Version: "1.11"
}
2019-08-02T09:29:35+02:00 [✖]  cannot list Kubernetes Services: Unauthorized

Now - which part shows me the conflict here? And what AWS permissions are the ones to "list Kubernetes Services"? Then I can ask for them.

Apart from this eksctl I do seem to have necessary permissions, as right now I do the remaining cluster cleanup of EC2 instances, gateways, network interaces and VPCs on my AWS (UI) of the same user (without the assume role but that my be implicit on the UI?).

Thanks for your help,
Fabian

[errordeveloper]

Hi, it is indeed to do with https://github.com/weaveworks/eksctl/issues/1070.

[fabboe]

I see. Which concrete permissions are actually lacking? Are those not part of the AmazonEKSServicePolicy?
Pardon my ignorance but the referenced naming of permissions ("Service list/delete permissions") doesn't map to concrete AWS permissions according to my Google skills.

[errordeveloper]

This is about access to the Kubernetes API, which is defined by IAM identity mappings in kube-system:ConfigMap/aws-auth. This essentially means you are not the owner of that cluster, another IAM user created it and they should add you to kube-system:ConfigMap/aws-auth, otherwise you can delete the cluster via CloudFormation. I'm going to close this, f you have other concerns please do leave a comment on #1070.