Amazon-vpc-cni-k8s: EC2 instances don't get assigned with private IPs

[marcincuber]

I have implemented OIDC provider and integrated new IAM role with the service account. I was using following docs. I also restricted access to Amazon EC2 Instance Profile Credentials.

After updating everything and training to add new nodes I am getting an error:

rpc error: code = Unknown desc = failed to set up sandbox container "b86cc85dc26426bd0cd5655b1cb270465e3e04c3c86699c554f9aa8d10e409ec" network for pod "external-secrets-54cbfc45b7-587wt": NetworkPlugin cni failed to set up pod "external-secrets-54cbfc45b7-587wt_default" network: add cmd: failed to assign an IP address to container

Simply, nodes are not being assigned with private IPs. I am using version 1.5.3 of the vpc CNI. I verified that service account it correctly configured with IAM role:

$ k exec aws-node-2ssc2 -n kube-system env |grep AWS                                                   [16:16:08]
AWS_ROLE_ARN=arn:aws:iam::00000000:role/ceng-eks-test-aws-node
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
AWS_VPC_K8S_CNI_LOGLEVEL=DEBUG
AWS_VPC_K8S_CNI_EXTERNALSNAT=true

The role is also using the correct policy.

I found error log:

2019-09-13T15:31:31.302Z [ERROR]    Failed to CreateNetworkInterface WebIdentityErr: failed to retrieve credentials
caused by: InvalidIdentityToken: OpenIDConnect provider's HTTPS certificate doesn't match configured thumbprint
    status code: 400, request id: 8fa06a43-d63b-11e9-80ee-c35ff5f09261
2019-09-13T15:31:31.302Z [ERROR]    Failed to increase pool size due to not able to allocate ENI AllocENI: failed to create ENI: failed to create network interface: WebIdentityErr: failed to retrieve credentials
caused by: InvalidIdentityToken: OpenIDConnect provider's HTTPS certificate doesn't match configured thumbprint
    status code: 400, request id: 8fa06a43-d63b-11e9-80ee-c35ff5f09261

May I ask for help on how to further debug this or can propose a solution?

[jqmichael]

Could you provide the content of the token that's getting generated?
/var/run/secrets/eks.amazonaws.com/serviceaccount/token

[marcincuber]

Hi, I managed to solve the issue. So I am creating an oidc provider using terraform:

resource "aws_iam_openid_connect_provider" "cluster" {
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = []
  url             = aws_eks_cluster.cluster.identity.0.oidc.0.issuer
}

and so thumbprint_list is empty. Note that if you create the same oidc provider in the console, it will be populated with thumbprint_list and there will be a single value.
It looks like with terraform you don't get the same behaviour and so everything that uses this provider will fail.

You have to go through the process described in https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc_verify-thumbprint.html#thumbstep2 to retrive the thumbprint and put it into thumbprint_list.

That solved my issue. If someone can suggest how to obtain the thumbprint of oidc provider using terraform that would be heaven.

[jqmichael]

@marcincuber
https://github.com/aws/containers-roadmap/issues/23#issuecomment-530548586

[jqmichael]

Closed the issue.