Amazon-eks-ami: Worker nodes not attaching after following the official guides step by step

[ZeeCoder]

What happened:
I've created a cluster, in several ways, and try to add a node group to it (with a single instance for simplicity), but no matter which way I do it, the node group never manages to attach the nodes.

I've followed all the different ways the EKS guide describes setting up a cluster:
https://docs.aws.amazon.com/eks/latest/userguide/getting-started.html

I tried eksctl first, then setting up resources with cloudformation, then setting up everything via the console manually, the result is always the same. 🤔

EC2 instances are created, but they always get an "Unauthorised" error in the logs.
A snippet of the errors:

Jun 04 20:27:13 ip-192-168-38-68.eu-west-2.compute.internal kubelet[3787]: W0604 20:27:13.438742    3787 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d
Jun 04 20:27:13 ip-192-168-38-68.eu-west-2.compute.internal kubelet[3787]: E0604 20:27:13.456986    3787 kubelet.go:2272] node "ip-192-168-38-68.eu-west-2.compute.internal" not found
Jun 04 20:27:13 ip-192-168-38-68.eu-west-2.compute.internal kubelet[3787]: I0604 20:27:13.480612    3787 kubelet_node_status.go:286] Setting node annotation to enable volume controller attach/detach
Jun 04 20:27:13 ip-192-168-38-68.eu-west-2.compute.internal kubelet[3787]: I0604 20:27:13.480650    3787 kubelet_node_status.go:334] Adding node label from cloud provider: beta.kubernetes.io/instance-type=t3.medium
Jun 04 20:27:13 ip-192-168-38-68.eu-west-2.compute.internal kubelet[3787]: I0604 20:27:13.480658    3787 kubelet_node_status.go:345] Adding node label from cloud provider: failure-domain.beta.kubernetes.io/zone=eu-west-2a
Jun 04 20:27:13 ip-192-168-38-68.eu-west-2.compute.internal kubelet[3787]: I0604 20:27:13.480670    3787 kubelet_node_status.go:349] Adding node label from cloud provider: failure-domain.beta.kubernetes.io/region=eu-west-2
Jun 04 20:27:13 ip-192-168-38-68.eu-west-2.compute.internal kubelet[3787]: I0604 20:27:13.482624    3787 kubelet_node_status.go:72] Attempting to register node ip-192-168-38-68.eu-west-2.compute.internal
Jun 04 20:27:13 ip-192-168-38-68.eu-west-2.compute.internal kubelet[3787]: E0604 20:27:13.510883    3787 kubelet_node_status.go:94] Unable to register node "ip-192-168-38-68.eu-west-2.compute.internal" with API server: Unauthorized
Jun 04 20:27:13 ip-192-168-38-68.eu-west-2.compute.internal kubelet[3787]: E0604 20:27:13.557148    3787 kubelet.go:2272] node "ip-192-168-38-68.eu-west-2.compute.internal" not found
Jun 04 20:27:13 ip-192-168-38-68.eu-west-2.compute.internal kubelet[3787]: E0604 20:27:13.580712    3787 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:450: Failed to list *v1.Service: Unauthorized
Jun 04 20:27:13 ip-192-168-38-68.eu-west-2.compute.internal kubelet[3787]: E0604 20:27:13.657342    3787 kubelet.go:2272] node "ip-192-168-38-68.eu-west-2.compute.internal" not found
Jun 04 20:27:13 ip-192-168-38-68.eu-west-2.compute.internal kubelet[3787]: E0604 20:27:13.757534    3787 kubelet.go:2272] node "ip-192-168-38-68.eu-west-2.compute.internal" not found
Jun 04 20:27:13 ip-192-168-38-68.eu-west-2.compute.internal kubelet[3787]: E0604 20:27:13.857708    3787 kubelet.go:2272] node "ip-192-168-38-68.eu-west-2.compute.internal" not found
Jun 04 20:27:13 ip-192-168-38-68.eu-west-2.compute.internal kubelet[3787]: E0604 20:27:13.957956    3787 kubelet.go:2272] node "ip-192-168-38-68.eu-west-2.compute.internal" not found
Jun 04 20:27:14 ip-192-168-38-68.eu-west-2.compute.internal kubelet[3787]: E0604 20:27:14.005202    3787 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1beta1.CSIDriver: Unauthorized
Jun 04 20:27:14 ip-192-168-38-68.eu-west-2.compute.internal kubelet[3787]: E0604 20:27:14.058161    3787 kubelet.go:2272] node "ip-192-168-38-68.eu-west-2.compute.internal" not found
Jun 04 20:27:14 ip-192-168-38-68.eu-west-2.compute.internal kubelet[3787]: E0604 20:27:14.126128    3787 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1beta1.RuntimeClass: Unauthorized
Jun 04 20:27:14 ip-192-168-38-68.eu-west-2.compute.internal kubelet[3787]: E0604 20:27:14.158372    3787 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Jun 04 20:27:14 ip-192-168-38-68.eu-west-2.compute.internal kubelet[3787]: E0604 20:27:14.158450    3787 kubelet.go:2272] node "ip-192-168-38-68.eu-west-2.compute.internal" not found
Jun 04 20:27:14 ip-192-168-38-68.eu-west-2.compute.internal kubelet[3787]: E0604 20:27:14.196491    3787 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Unauthorized
Jun 04 20:27:14 ip-192-168-38-68.eu-west-2.compute.internal kubelet[3787]: E0604 20:27:14.250720    3787 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:459: Failed to list *v1.Node: Unauthorized
Jun 04 20:27:14 ip-192-168-38-68.eu-west-2.compute.internal kubelet[3787]: E0604 20:27:14.258569    3787 kubelet.go:2272] node "ip-192-168-38-68.eu-west-2.compute.internal" not found
Jun 04 20:27:14 ip-192-168-38-68.eu-west-2.compute.internal kubelet[3787]: E0604 20:27:14.358715    3787 kubelet.go:2272] node "ip-192-168-38-68.eu-west-2.compute.internal" not found
Jun 04 20:27:14 ip-192-168-38-68.eu-west-2.compute.internal kubelet[3787]: E0604 20:27:14.458901    3787 kubelet.go:2272] node "ip-192-168-38-68.eu-west-2.compute.internal" not found
Jun 04 20:27:14 ip-192-168-38-68.eu-west-2.compute.internal kubelet[3787]: E0604 20:27:14.473451    3787 controller.go:135] failed to ensure node lease exists, will retry in 7s, error: Unauthorized
Jun 04 20:27:14 ip-192-168-38-68.eu-west-2.compute.internal kubelet[3787]: E0604 20:27:14.559077    3787 kubelet.go:2272] node "ip-192-168-38-68.eu-west-2.compute.internal" not found
Jun 04 20:27:14 ip-192-168-38-68.eu-west-2.compute.internal kubelet[3787]: E0604 20:27:14.613885    3787 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:450: Failed to list *v1.Service: Unauthorized
Jun 04 20:27:14 ip-192-168-38-68.eu-west-2.compute.internal kubelet[3787]: E0604 20:27:14.659285    3787 kubelet.go:2272] node "ip-192-168-38-68.eu-west-2.compute.internal" not found
Jun 04 20:27:14 ip-192-168-38-68.eu-west-2.compute.internal kubelet[3787]: E0604 20:27:14.759476    3787 kubelet.go:2272] node "ip-192-168-38-68.eu-west-2.compute.internal" not found
Jun 04 20:27:14 ip-192-168-38-68.eu-west-2.compute.internal kubelet[3787]: E0604 20:27:14.859632    3787 kubelet.go:2272] node "ip-192-168-38-68.eu-west-2.compute.internal" not found
Jun 04 20:27:14 ip-192-168-38-68.eu-west-2.compute.internal kubelet[3787]: E0604 20:27:14.959814    3787 kubelet.go:2272] node "ip-192-168-38-68.eu-west-2.compute.internal" not found
Jun 04 20:27:15 ip-192-168-38-68.eu-west-2.compute.internal kubelet[3787]: E0604 20:27:15.035684    3787 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1beta1.CSIDriver: Unauthorized
Jun 04 20:27:15 ip-192-168-38-68.eu-west-2.compute.internal kubelet[3787]: E0604 20:27:15.059961    3787 kubelet.go:2272] node "ip-192-168-38-68.eu-west-2.compute.internal" not found
Jun 04 20:27:15 ip-192-168-38-68.eu-west-2.compute.internal kubelet[3787]: E0604 20:27:15.153981    3787 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1beta1.RuntimeClass: Unauthorized
Jun 04 20:27:15 ip-192-168-38-68.eu-west-2.compute.internal kubelet[3787]: E0604 20:27:15.160101    3787 kubelet.go:2272] node "ip-192-168-38-68.eu-west-2.compute.internal" not found
Jun 04 20:27:15 ip-192-168-38-68.eu-west-2.compute.internal kubelet[3787]: E0604 20:27:15.225109    3787 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Unauthorized
Jun 04 20:27:15 ip-192-168-38-68.eu-west-2.compute.internal kubelet[3787]: E0604 20:27:15.260240    3787 kubelet.go:2272] node "ip-192-168-38-68.eu-west-2.compute.internal" not found
Jun 04 20:27:15 ip-192-168-38-68.eu-west-2.compute.internal kubelet[3787]: E0604 20:27:15.278214    3787 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:459: Failed to list *v1.Node: Unauthorized
Jun 04 20:27:15 ip-192-168-38-68.eu-west-2.compute.internal kubelet[3787]: E0604 20:27:15.360409    3787 kubelet.go:2272] node "ip-192-168-38-68.eu-west-2.compute.internal" not found
Jun 04 20:27:15 ip-192-168-38-68.eu-west-2.compute.internal kubelet[3787]: E0604 20:27:15.460592    3787 kubelet.go:2272] node "ip-192-168-38-68.eu-west-2.compute.internal" not found
Jun 04 20:27:15 ip-192-168-38-68.eu-west-2.compute.internal kubelet[3787]: E0604 20:27:15.560773    3787 kubelet.go:2272] node "ip-192-168-38-68.eu-west-2.compute.internal" not found
Jun 04 20:27:15 ip-192-168-38-68.eu-west-2.compute.internal kubelet[3787]: E0604 20:27:15.641682    3787 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:450: Failed to list *v1.Service: Unauthorized
Jun 04 20:27:15 ip-192-168-38-68.eu-west-2.compute.internal kubelet[3787]: E0604 20:27:15.660921    3787 kubelet.go:2272] node "ip-192-168-38-68.eu-west-2.compute.internal" not found
Jun 04 20:27:15 ip-192-168-38-68.eu-west-2.compute.internal kubelet[3787]: E0604 20:27:15.761080    3787 kubelet.go:2272] node "ip-192-168-38-68.eu-west-2.compute.internal" not found
Jun 04 20:27:15 ip-192-168-38-68.eu-west-2.compute.internal kubelet[3787]: E0604 20:27:15.861270    3787 kubelet.go:2272] node "ip-192-168-38-68.eu-west-2.compute.internal" not found
Jun 04 20:27:15 ip-192-168-38-68.eu-west-2.compute.internal kubelet[3787]: E0604 20:27:15.961456    3787 kubelet.go:2272] node "ip-192-168-38-68.eu-west-2.compute.internal" not found
Jun 04 20:27:16 ip-192-168-38-68.eu-west-2.compute.internal kubelet[3787]: E0604 20:27:16.061786    3787 kubelet.go:2272] node "ip-192-168-38-68.eu-west-2.compute.internal" not found
Jun 04 20:27:16 ip-192-168-38-68.eu-west-2.compute.internal kubelet[3787]: E0604 20:27:16.064455    3787 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1beta1.CSIDriver: Unauthorized
Jun 04 20:27:16 ip-192-168-38-68.eu-west-2.compute.internal kubelet[3787]: E0604 20:27:16.162366    3787 kubelet.go:2272] node "ip-192-168-38-68.eu-west-2.compute.internal" not found

I've also read that some people didn't have the aws auth config applied, but that didn't help for me either.
I did set up a NodeInstanceRole, which has all the required permissions as outlined here:
https://docs.aws.amazon.com/eks/latest/userguide/worker_node_IAM_role.html

Then following this guide, I applied the aws-auth to the cluster:
https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html
(Using the role's arn.)

The node instance is publicly accessible, as I managed to SSH into it as well.

Tags seem correct as well, but that was done by the console anyway, so should be correct.

What you expected to happen:
I expected to have a working EKS cluster with a couple nodes by following the documentation.

How to reproduce it (as minimally and precisely as possible):
Easiest way is if you just use the recommended eksctl way of creating the cluster with node groups.

Anything else we need to know?:

Environment:

• AWS Region: eu-west-2
• Instance Type(s): Tried with medium, large, micro, all the same
• EKS Platform version: eks1
• Kubernetes version: 1.16
• AMI Version: latest one (Not sure exactly as I've deleted the ec2 instance)
• Kernel: latest one (Not sure exactly as I've deleted the ec2 instance)
• Release information: latest one (Not sure exactly as I've deleted the ec2 instance)

[heybronson]

Are your nodes using an InstanceProfile with the NodeInstanceRole that you've created?

[ZeeCoder]

I can't see anything like that when I create the node groups on the console, so I don't know.

The only thing I see is on the role itself, it lists an instance profile arn.

[ZeeCoder]

I've just tested and every time I create a role, I get an instance profile ARN, but not sure what that means.

[ZeeCoder]

I've just used the same eksctl command on my personal account (which didn't have anything on it), and that worked out of the box.
Not sure what the difference is.

[ZeeCoder]

Doesn't seem like it's the IAM roles.
I've copied over the policies as well as tags (adjusted to real cluster- and node names) from the cluster from my personal account to the one that's not working.

[heybronson]

Is this with EKS Managed Nodegroups or are you creating these worker nodes manually?

Is the instance profile used by your worker nodes also listed in your aws-auth configmap?

[ZeeCoder]

Managed node group that I'm trying to add through the console.
I've set up an "InstanceNodeRole" following the amazon guides, I've updated the config map and I've selected that role when creating the nodegroup.

In the console I can see a NodeCreationError (if my memory serves), and in the node instance logs I can see the unauthenticated logs.

[ZeeCoder]

I'm in contact with someone from AWS as part of our "developer" level support.
I'll feed back here anything tangible.

[ZeeCoder]

Figured out with the help of amazon support.

For everything to work, you need to have STS enabled in the region you're deploying.
They'll add this to the guide, but essentially what you need to do is:

• Go to IAM
• Go to "Account Settings"
• Under the "Endpoints" section, you'll need to make sure the region has an "Active" STS status.

That's it, once I've enabled that everything worked out of the box as it did on my personal acc. 👌

[Viswa88]

@ZeeCoder It's worked & saved my time