Eksctl: create cluster hangs on waiting for nodes to become ready

Created on 24 Oct 2019 · 44Comments · Source: weaveworks/eksctl

What happened?
When creating a new cluster, nodes fail to join the cluster. eksctl times out waiting for nodes after 25minutes.

What you expected to happen?
A cluster stood up and ready.

How to reproduce it?
Create new aws account, create iam with admin access and api credentials.
export aws credentials with envvars (aws_access_key... etc).
run following command:

eksctl create cluster \                                                                                                                                                                      adp-ip 2.3.8
--name my-super-cluster \
--vpc-cidr 10.180.0.0/16 \
--zones us-west-2a,us-west-2b,us-west-2c \
--nodegroup-name standard-1-14 \
--node-type m5n.large \
--nodes 3 \
--nodes-min 3 \
--nodes-max 6 \
--region us-west-2 \
--version 1.14 \
--asg-access \
--external-dns-access \
--node-ami auto

Anything else we need to know?
What OS are you using, are you using a downloaded binary or did you compile eksctl, what type of AWS credentials are you using (i.e. default/named profile, MFA) - please don't include actual credentials though!
Using mac os catalina.
eksctl installed through curl one liner
I've tried this with eksctl v 0.7.0 too. also failed.

Versions
Please paste in the output of these commands:

$ eksctl version
[ℹ]  version.Info{BuiltAt:"", GitCommit:"", GitTag:"0.6.0"}
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.4", GitCommit:"67d2fcf276fcd9cf743ad4be9a9ef5828adc082f", GitTreeState:"clean", BuildDate:"2019-09-18T14:51:13Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.7-eks-e9b1d0", GitCommit:"e9b1d0551216e1e8ace5ee4ca50161df34325ec2", GitTreeState:"clean", BuildDate:"2019-09-21T08:33:01Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

Logs
Include the output of the command line when running eksctl. If possible, eksctl should be run with debug logs. For example:
eksctl get clusters -v 4
Make sure you redact any sensitive information before posting.
If the output is long, please consider a Gist.

eksctl create cluster \                                                                                                                                                                      adp-ip 2.3.8
--name my-super-cluster \
--vpc-cidr 10.180.0.0/16 \
--zones us-west-2a,us-west-2b,us-west-2c \
--nodegroup-name standard-1-14 \
--node-type m5n.large \
--nodes 3 \
--nodes-min 3 \
--nodes-max 6 \
--region us-west-2 \
--version 1.14 \
--asg-access \
--external-dns-access \
--node-ami auto
[ℹ]  using region us-west-2
[ℹ]  subnets for us-west-2a - public:10.180.0.0/19 private:10.180.96.0/19
[ℹ]  subnets for us-west-2b - public:10.180.32.0/19 private:10.180.128.0/19
[ℹ]  subnets for us-west-2c - public:10.180.64.0/19 private:10.180.160.0/19
[ℹ]  nodegroup "standard-1-14" will use "ami-05d586e6f773f6abf" [AmazonLinux2/1.14]
[ℹ]  using Kubernetes version 1.14
[ℹ]  creating EKS cluster "my-super-cluster" in "us-west-2" region
[ℹ]  will create 2 separate CloudFormation stacks for cluster itself and the initial nodegroup
[ℹ]  if you encounter any issues, check CloudFormation console or try 'eksctl utils describe-stacks --region=us-west-2 --name=my-super-cluster'
[ℹ]  CloudWatch logging will not be enabled for cluster "my-super-cluster" in "us-west-2"
[ℹ]  you can enable it with 'eksctl utils update-cluster-logging --region=us-west-2 --name=my-super-cluster'
[ℹ]  2 sequential tasks: { create cluster control plane "my-super-cluster", create nodegroup "standard-1-14" }
[ℹ]  building cluster stack "eksctl-my-super-cluster-cluster"
[ℹ]  deploying stack "eksctl-my-super-cluster-cluster"
[ℹ]  building nodegroup stack "eksctl-my-super-cluster-nodegroup-standard-1-14"
[ℹ]  deploying stack "eksctl-my-super-cluster-nodegroup-standard-1-14"
[✔]  all EKS cluster resource for "my-super-cluster" had been created
[✔]  saved kubeconfig as "/Users/ivanp/.kube/config"
[ℹ]  adding role "arn:aws:iam::xxxxxxxx:role/eksctl-my-super-cluster-nodegroup-sta-NodeInstanceRole-CG2DJOYVDFWV" to auth ConfigMap
[ℹ]  nodegroup "standard-1-14" has 0 node(s)
[ℹ]  waiting for at least 3 node(s) to become ready in "standard-1-14"
[✖]  timed out (after 25m0s) waiting for at least 3 nodes to join the cluster and become ready in "standard-1-14"

kinbug needs-investigation

Source

i5okie

👍14

Most helpful comment

i too have the same issue. do we have any solution ??

cool-raj on 24 Jan 2020

👍5

All 44 comments

Quick update:
I've tried creating kubernetes 1.13 cluster with otherwise same values.
I've tried creating a cluster without specifying node group name. I've tried small node-type.

Last thing I've tried was spinning up a new ubuntu 18.04 ec2 instance, installing kubectl, latest eksctl, and running the above command there. the end result was the same. cluster and node group cloudformation stacks successfully finished creating everything. But nodes never joined the cluster.
I did not encounter any errors, or otherwise reasons or clues as to what is causing this issue.

i5okie on 24 Oct 2019

A similar issue found here as well. One node is joined while the other won't