Amazon-vpc-cni-k8s: Nodes are not joining in aws eks

[trojanops]

I have launched cluster using aws eks successfully. applied aws-auth but nodes are not joining. I checked log message of a node and found this -

Dec  4 08:09:02 ip-10-0-8-187 kubelet: E1204 08:09:02.760634    3542 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:461: Failed to list *v1.Node: Unauthorized
Dec  4 08:09:03 ip-10-0-8-187 kubelet: W1204 08:09:03.296102    3542 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
Dec  4 08:09:03 ip-10-0-8-187 kubelet: E1204 08:09:03.296217    3542 kubelet.go:2130] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Dec  4 08:09:03 ip-10-0-8-187 kubelet: E1204 08:09:03.459361    3542 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:452: Failed to list *v1.Service: Unauthorized

I am not sure about this. I have attached eks full access to these instance node roles.

[Deepak-Vohra]

Has the ConfigMap for authentication been created?

[sbkg0002]

We have the same issue with the official AMI from 20190109. The ConfigMap was successfull.

[Deepak-Vohra]

Has the kubeconfig been created using the settings generated by EKS cluster? What does kubectl get nodes list?

[sbkg0002]

In our case the tags on the nodes had a typo..

[sasvirco]

I have the same problem, what was the typo?

[Deepak-Vohra]

sasvirco What does kubectl get nodes list?

[sasvirco]

in my case was a wrong arn in the config map, kubectl get nodes list did not show anything as nodes were unable to join, once I did update the configMap with the proper arn, they joined promptly.

[Deepak-Vohra]

sasvirco thanks for the update.

[sbkg0002]

@sasvirco the mandatory tags had a custom suffix added. That way EKS can't find the nodes.

[paulkendobnc]

### I had the same issue and spent a few hours investigating, only to fin out that my new instance type "m5a.2xlarge" was the cause!!! I switched back to "m5.2xlarge" and everything is ok. very weird.

[tabern]

@trojanops this may be an issue with the IAM authenticator? Please open a ticket with AWS support so we can troubleshoot.

[elliottb303]

Adding on re: tags, the typo is a newb mistake. Digging for more complicated reasons was rough. Node has to be 'owned' by a certain cluster. The nodes will only join a cluster its supposed to. I overlooked this but there was not a lot of docs on it. In my case using terraform. Make sure variables match. This is node tag naming parent cluster to join

  tag {
    key = "kubernetes.io/cluster/${var.eks_cluster_name}-${terraform.workspace}"
    value = "owned"
    propagate_at_launch = true
  }