Terraform-aws-eks: worker nodes unable to connect to the cluster
I'm submitting a...
- [ ] bug report
- [ ] feature request
- [ * ] support request
- [ ] kudos, thank you, warm fuzzy
What is the current behavior?
I am having issues getting the worker nodes to connect to my eks cluster. when i logged in to the worker node i am getting the error below. This is also using a public subnet from my vpc. aws_auth gets created with no issues and ec2 instances are also using the right instance profile. the version im using is Version":"5.0.0"
Jun 30 03:08:19 ip-10-100-5-7 kubelet: E0630 03:08:19.149994 4420 kubelet.go:2266] node "ip-10-100-5-7.us-west-2.compute.internal" not found
Jun 30 03:08:19 ip-10-100-5-7 kubelet: E0630 03:08:19.250192 4420 kubelet.go:2266] node "ip-10-100-5-7.us-west-2.compute.internal" not found
Jun 30 03:08:19 ip-10-100-5-7 kubelet: E0630 03:08:19.350403 4420 kubelet.go:2266] node "ip-10-100-5-7.us-west-2.compute.internal" not found
Jun 30 03:08:19 ip-10-100-5-7 kubelet: E0630 03:08:19.450605 4420 kubelet.go:2266] node "ip-10-100-5-7.us-west-2.compute.internal" not found
Jun 30 03:08:19 ip-10-100-5-7 kubelet: E0630 03:08:19.550837 4420 kubelet.go:2266] node "ip-10-100-5-7.us-west-2.compute.internal" not found
Jun 30 03:08:19 ip-10-100-5-7 kubelet: E0630 03:08:19.651064 4420 kubelet.go:2266] node "ip-10-100-5-7.us-west-2.compute.internal" not found
Jun 30 03:08:19 ip-10-100-5-7 kubelet: E0630 03:08:19.751273 4420 kubelet.go:2266] node "ip-10-100-5-7.us-west-2.compute.internal" not found
Jun 30 03:08:19 ip-10-100-5-7 kubelet: E0630 03:08:19.770595 4420 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Unauthorized
Jun 30 03:08:19 ip-10-100-5-7 kubelet: E0630 03:08:19.851962 4420 kubelet.go:2266] node "ip-10-100-5-7.us-west-2.compute.internal" not found
Jun 30 03:08:19 ip-10-100-5-7 kubelet: E0630 03:08:19.900817 4420 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Unauthorized
Jun 30 03:08:19 ip-10-100-5-7 kubelet: E0630 03:08:19.900859 4420 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Unauthorized
Jun 30 03:08:19 ip-10-100-5-7 kubelet: E0630 03:08:19.952483 4420 kubelet.go:2266] node "ip-10-100-5-7.us-west-2.compute.internal" not found
Jun 30 03:08:20 ip-10-100-5-7 kubelet: E0630 03:08:20.052714 4420 kubelet.go:2266] node "ip-10-100-5-7.us-west-2.compute.internal" not found
Jun 30 03:08:20 ip-10-100-5-7 kubelet: E0630 03:08:20.152924 4420 kubelet.go:2266] node "ip-10-100-5-7.us-west-2.compute.internal" not found
Jun 30 03:08:20 ip-10-100-5-7 kubelet: E0630 03:08:20.253136 4420 kubelet.go:2266] node "ip-10-100-5-7.us-west-2.compute.internal" not found
Jun 30 03:08:20 ip-10-100-5-7 kubelet: E0630 03:08:20.353369 4420 kubelet.go:2266] node "ip-10-100-5-7.us-west-2.compute.internal" not found
Jun 30 03:08:20 ip-10-100-5-7 kubelet: E0630 03:08:20.453600 4420 kubelet.go:2266] node "ip-10-100-5-7.us-west-2.compute.internal" not found
Jun 30 03:08:20 ip-10-100-5-7 kubelet: E0630 03:08:20.553818 4420 kubelet.go:2266] node "ip-10-100-5-7.us-west-2.compute.internal" not found
Jun 30 03:08:20 ip-10-100-5-7 kubelet: E0630 03:08:20.654032 4420 kubelet.go:2266] node "ip-10-100-5-7.us-west-2.compute.internal" not found
Jun 30 03:08:20 ip-10-100-5-7 kubelet: E0630 03:08:20.754252 4420 kubelet.go:2266] node "ip-10-100-5-7.us-west-2.compute.internal" not found
Jun 30 03:08:20 ip-10-100-5-7 kubelet: E0630 03:08:20.854445 4420 kubelet.go:2266] node "ip-10-100-5-7.us-west-2.compute.internal" not found
Jun 30 03:08:20 ip-10-100-5-7 kubelet: E0630 03:08:20.897784 4420 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Unauthorized
Jun 30 03:08:20 ip-10-100-5-7 kubelet: E0630 03:08:20.954949 4420 kubelet.go:2266] node "ip-10-100-5-7.us-west-2.compute.internal" not found
If this is a bug, how to reproduce? Please include a code sample if relevant.
Here is my main.tf
variable "vpc_id" {
default = "vpc-001670fb"
}
variable "subnets" {
default = ["subnet-0a4c92b3", "subnet-0e414456"]
type = list(string)
}
variable "vpc_env_name" {
default = "production"
}
variable "cluster_version" {
default = "1.13"
}
variable "cluster_name" {
default = "eks-cluster-us-west-2-01"
}
variable "office_sg" {
default = ["sg-0a92c8b"]
type = list(string)
}
variable "map_roles" {
description = "Additional IAM roles to add to the aws-auth configmap."
type = list(map(string))
default = [
{
role_arn = "arn:aws:iam:::role/devops"
username = "devops"
group = "system:masters"
},
]
}
provider "aws" {
region = "us-west-2"
profile = "sa-terraform"
}
terraform {
backend "s3" {
bucket = "eks-cluster-operations-terraform"
key = "eks-clusters-terraform-module/eks-cluster-us-west-2-01"
region = "us-west-2"
profile = "sa-terraform"
}
}
module "eks-clusters-terraform-module" {
source = "terraform-aws-modules/eks/aws"
cluster_name = var.cluster_name
cluster_version = "1.13"
subnets = var.subnets
vpc_id = var.vpc_id
worker_additional_security_group_ids = var.office_sg
manage_aws_auth = "true"
map_roles = var.map_roles
kubeconfig_aws_authenticator_env_variables = {
AWS_PROFILE = "devops"
}
tags = {
Environment = "shared"
}
worker_groups = [
{
name = "generalPurpose_2a"
instance_type = "t2.medium"
asg_min_size = 1
asg_max_size = 2
autoscaling_enabled = true
protect_scale_in = true
asg_desired_capacity = 1
key_name = ""
},
{
name = "generalPurpose_2b"
instance_type = "t2.medium"
asg_min_size = 1
asg_max_size = 2
autoscaling_enabled = true
protect_scale_in = true
asg_desired_capacity = 1
key_name = ""
}
]
}
What's the expected behavior?
Are you able to fix this problem and submit a PR? Link here if you have already.
Environment details
- Affected module version:
- OS: OSX
- Terraform version:
Terraform v0.12.3 - provider.aws v2.17.0
- provider.local v1.3.0
- provider.null v2.1.2
- provider.template v2.1.2
Any other relevant info
opencatersf
All 13 comments
Updating with the AMI i am using: amazon-eks-node-1.13-v20190614
opencatersf
on 30 Jun 2019
I think this is your problem: Failed to list *v1.Pod: Unauthorized
Check that your aws-auth configmap was applied and has the correct rolearn
max-rocket-internet
on 1 Jul 2019
@max-rocket-internet config map looks good to me
apiVersion: v1
kind: ConfigMap
metadata:
name: aws-auth
namespace: kube-system
data:
mapRoles: |
- rolearn: arn:aws:iam:::role/eks-cluster-us-west-2-0120190701201308460600000006
username: system:node:{{EC2PrivateDNSName}}
groups:
- system:bootstrappers
- system:nodes
- rolearn: arn:aws:iam:::role/devops
username: devops
groups:
- system:masters
mapUsers: |
mapAccounts: |
opencatersf
on 1 Jul 2019
i also checked the kubeconfig inside the worker nodes seems right.
apiVersion: v1
clusters:
- cluster:
certificate-authority: /etc/kubernetes/pki/ca.crt
server: https://140922E6BD597ABCDEC9E72A3CE9CE.yl4.us-west-2.eks.amazonaws.com
name: kubernetes
contexts: - context:
cluster: kubernetes
user: kubelet
name: kubelet
current-context: kubelet
kind: Config
preferences: {}
users: - name: kubelet
user:
exec:
apiVersion: client.authentication.k8s.io/v1alpha1
args:
- token
- -i
- eks-cluster-us-west-2-01
command: /usr/bin/aws-iam-authenticator
env: null
opencatersf
on 1 Jul 2019
I tried running this on public subnets now but same issue. seems like
Jul 1 21:03:15 ip-10-72-171-192 kubelet: I0701 21:03:15.275924 4431 kubelet_node_status.go:279] Setting node annotation to enable volume controller attach/detach
Jul 1 21:03:15 ip-10-72-171-192 kubelet: I0701 21:03:15.276524 4431 kubelet_node_status.go:327] Adding node label from cloud provider: beta.kubernetes.io/instance-type=t2.medium
Jul 1 21:03:15 ip-10-72-171-192 kubelet: I0701 21:03:15.276873 4431 kubelet_node_status.go:338] Adding node label from cloud provider: failure-domain.beta.kubernetes.io/zone=us-west-2b
Jul 1 21:03:15 ip-10-72-171-192 kubelet: I0701 21:03:15.277221 4431 kubelet_node_status.go:342] Adding node label from cloud provider: failure-domain.beta.kubernetes.io/region=us-west-2
opencatersf
on 2 Jul 2019
i got this figured out. looks like the aws_auth configmap is not applying automatically, I had to apply it manually in order for the workers to join the cluster. is this a bug?
opencatersf
on 2 Jul 2019
Same here, but still haven't figured out how to re-apply configmap/aws-auth so the EC2 instance can join...
The command below did not work:
kubectl get -n kube-system configmap/aws-auth -o yaml | kubectl replace -f -
davi5e
on 10 Jul 2019
@davi5e are you connected to the cluster? what is the output of your kubectl cluster-info
renperez
on 10 Jul 2019
$ kubectl cluster-info
Kubernetes master is running at https://FD682B64C213C073209FDA092F6CCBCC.gr7.us-east-1.eks.amazonaws.com
CoreDNS is running at https://FD682B64C213C073209FDA092F6CCBCC.gr7.us-east-1.eks.amazonaws.com/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
And then
$ kubectl cluster-info dump
{
"kind": "NodeList",
"apiVersion": "v1",
"metadata": {
"selfLink": "/api/v1/nodes",
"resourceVersion": "3358"
},
"items": []
}
{
"kind": "EventList",
"apiVersion": "v1",
"metadata": {
"selfLink": "/api/v1/namespaces/kube-system/events",
"resourceVersion": "3358"
},
"items": [
{
"metadata": {
"name": "coredns-7f66c6c4b9-8l76x.15b0645a88cb1314",
"namespace": "kube-system",
"selfLink": "/api/v1/namespaces/kube-system/events/coredns-7f66c6c4b9-8l76x.15b0645a88cb1314",
"uid": "b397dc60-a3f1-11e9-82e6-0e594b7607e6",
"resourceVersion": "3231",
"creationTimestamp": "2019-07-11T15:36:50Z"
},
"involvedObject": {
"kind": "Pod",
"namespace": "kube-system",
"name": "coredns-7f66c6c4b9-8l76x",
"uid": "b3961b28-a3f1-11e9-82e6-0e594b7607e6",
"apiVersion": "v1",
"resourceVersion": "194"
},
"reason": "FailedScheduling",
"message": "no nodes available to schedule pods",
"source": {
"component": "default-scheduler"
},
"firstTimestamp": "2019-07-11T15:36:50Z",
"lastTimestamp": "2019-07-11T16:16:53Z",
"count": 46,
"type": "Warning",
"eventTime": null,
"reportingComponent": "",
"reportingInstance": ""
},
{
"metadata": {
"name": "coredns-7f66c6c4b9-9pd82.15b0645a881c9f97",
"namespace": "kube-system",
"selfLink": "/api/v1/namespaces/kube-system/events/coredns-7f66c6c4b9-9pd82.15b0645a881c9f97",
"uid": "b3962d3b-a3f1-11e9-82e6-0e594b7607e6",
"resourceVersion": "3232",
"creationTimestamp": "2019-07-11T15:36:50Z"
},
"involvedObject": {
"kind": "Pod",
"namespace": "kube-system",
"name": "coredns-7f66c6c4b9-9pd82",
"uid": "b3955020-a3f1-11e9-82e6-0e594b7607e6",
"apiVersion": "v1",
"resourceVersion": "192"
},
"reason": "FailedScheduling",
"message": "no nodes available to schedule pods",
"source": {
"component": "default-scheduler"
},
"firstTimestamp": "2019-07-11T15:36:50Z",
"lastTimestamp": "2019-07-11T16:16:53Z",
"count": 44,
"type": "Warning",
"eventTime": null,
"reportingComponent": "",
"reportingInstance": ""
},
{
"metadata": {
"name": "coredns-7f66c6c4b9.15b0645a881988c8",
"namespace": "kube-system",
"selfLink": "/api/v1/namespaces/kube-system/events/coredns-7f66c6c4b9.15b0645a881988c8",
"uid": "b3960f75-a3f1-11e9-82e6-0e594b7607e6",
"resourceVersion": "196",
"creationTimestamp": "2019-07-11T15:36:50Z"
},
"involvedObject": {
"kind": "ReplicaSet",
"namespace": "kube-system",
"name": "coredns-7f66c6c4b9",
"uid": "b3937942-a3f1-11e9-82e6-0e594b7607e6",
"apiVersion": "apps/v1",
"resourceVersion": "189"
},
"reason": "SuccessfulCreate",
"message": "Created pod: coredns-7f66c6c4b9-9pd82",
"source": {
"component": "replicaset-controller"
},
"firstTimestamp": "2019-07-11T15:36:50Z",
"lastTimestamp": "2019-07-11T15:36:50Z",
"count": 1,
"type": "Normal",
"eventTime": null,
"reportingComponent": "",
"reportingInstance": ""
},
{
"metadata": {
"name": "coredns-7f66c6c4b9.15b0645a88707d67",
"namespace": "kube-system",
"selfLink": "/api/v1/namespaces/kube-system/events/coredns-7f66c6c4b9.15b0645a88707d67",
"uid": "b3978326-a3f1-11e9-82e6-0e594b7607e6",
"resourceVersion": "202",
"creationTimestamp": "2019-07-11T15:36:50Z"
},
"involvedObject": {
"kind": "ReplicaSet",
"namespace": "kube-system",
"name": "coredns-7f66c6c4b9",
"uid": "b3937942-a3f1-11e9-82e6-0e594b7607e6",
"apiVersion": "apps/v1",
"resourceVersion": "189"
},
"reason": "SuccessfulCreate",
"message": "Created pod: coredns-7f66c6c4b9-8l76x",
"source": {
"component": "replicaset-controller"
},
"firstTimestamp": "2019-07-11T15:36:50Z",
"lastTimestamp": "2019-07-11T15:36:50Z",
"count": 1,
"type": "Normal",
"eventTime": null,
"reportingComponent": "",
"reportingInstance": ""
},
{
"metadata": {
"name": "coredns.15b0645a8768c7b6",
"namespace": "kube-system",
"selfLink": "/api/v1/namespaces/kube-system/events/coredns.15b0645a8768c7b6",
"uid": "b3945813-a3f1-11e9-82e6-0e594b7607e6",
"resourceVersion": "190",
"creationTimestamp": "2019-07-11T15:36:50Z"
},
"involvedObject": {
"kind": "Deployment",
"namespace": "kube-system",
"name": "coredns",
"uid": "b3928c68-a3f1-11e9-82e6-0e594b7607e6",
"apiVersion": "apps/v1",
"resourceVersion": "188"
},
"reason": "ScalingReplicaSet",
"message": "Scaled up replica set coredns-7f66c6c4b9 to 2",
"source": {
"component": "deployment-controller"
},
"firstTimestamp": "2019-07-11T15:36:50Z",
"lastTimestamp": "2019-07-11T15:36:50Z",
"count": 1,
"type": "Normal",
"eventTime": null,
"reportingComponent": "",
"reportingInstance": ""
},
{
"metadata": {
"name": "kube-controller-manager.15b06459c54df18d",
"namespace": "kube-system",
"selfLink": "/api/v1/namespaces/kube-system/events/kube-controller-manager.15b06459c54df18d",
"uid": "b1a36e38-a3f1-11e9-82e6-0e594b7607e6",
"resourceVersion": "127",
"creationTimestamp": "2019-07-11T15:36:47Z"
},
"involvedObject": {
"kind": "Endpoints",
"namespace": "kube-system",
"name": "kube-controller-manager",
"uid": "b1a2c886-a3f1-11e9-82e6-0e594b7607e6",
"apiVersion": "v1",
"resourceVersion": "126"
},
"reason": "LeaderElection",
"message": "ip-172-16-171-189.ec2.internal_abc76c8b-a3f1-11e9-9d03-0e594b7607e6 became leader",
"source": {
"component": "kube-controller-manager"
},
"firstTimestamp": "2019-07-11T15:36:47Z",
"lastTimestamp": "2019-07-11T15:36:47Z",
"count": 1,
"type": "Normal",
"eventTime": null,
"reportingComponent": "",
"reportingInstance": ""
},
{
"metadata": {
"name": "kube-scheduler.15b0645973b3d581",
"namespace": "kube-system",
"selfLink": "/api/v1/namespaces/kube-system/events/kube-scheduler.15b0645973b3d581",
"uid": "b0d2814a-a3f1-11e9-82e6-0e594b7607e6",
"resourceVersion": "45",
"creationTimestamp": "2019-07-11T15:36:46Z"
},
"involvedObject": {
"kind": "Endpoints",
"namespace": "kube-system",
"name": "kube-scheduler",
"uid": "b0d1d808-a3f1-11e9-82e6-0e594b7607e6",
"apiVersion": "v1",
"resourceVersion": "41"
},
"reason": "LeaderElection",
"message": "ip-172-16-171-189.ec2.internal_a3861477-a3f1-11e9-a0fe-0e594b7607e6 became leader",
"source": {
"component": "default-scheduler"
},
"firstTimestamp": "2019-07-11T15:36:46Z",
"lastTimestamp": "2019-07-11T15:36:46Z",
"count": 1,
"type": "Normal",
"eventTime": null,
"reportingComponent": "",
"reportingInstance": ""
}
]
}
{
"kind": "ReplicationControllerList",
"apiVersion": "v1",
"metadata": {
"selfLink": "/api/v1/namespaces/kube-system/replicationcontrollers",
"resourceVersion": "3359"
},
"items": []
}
{
"kind": "ServiceList",
"apiVersion": "v1",
"metadata": {
"selfLink": "/api/v1/namespaces/kube-system/services",
"resourceVersion": "3359"
},
"items": [
{
"metadata": {
"name": "kube-dns",
"namespace": "kube-system",
"selfLink": "/api/v1/namespaces/kube-system/services/kube-dns",
"uid": "b3961696-a3f1-11e9-82e6-0e594b7607e6",
"resourceVersion": "195",
"creationTimestamp": "2019-07-11T15:36:50Z",
"labels": {
"eks.amazonaws.com/component": "kube-dns",
"k8s-app": "kube-dns",
"kubernetes.io/cluster-service": "true",
"kubernetes.io/name": "CoreDNS"
},
"annotations": {
"kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{\"prometheus.io/port\":\"9153\",\"prometheus.io/scrape\":\"true\"},\"labels\":{\"eks.amazonaws.com/component\":\"kube-dns\",\"k8s-app\":\"kube-dns\",\"kubernetes.io/cluster-service\":\"true\",\"kubernetes.io/name\":\"CoreDNS\"},\"name\":\"kube-dns\",\"namespace\":\"kube-system\"},\"spec\":{\"clusterIP\":\"172.20.0.10\",\"ports\":[{\"name\":\"dns\",\"port\":53,\"protocol\":\"UDP\"},{\"name\":\"dns-tcp\",\"port\":53,\"protocol\":\"TCP\"}],\"selector\":{\"k8s-app\":\"kube-dns\"}}}\n",
"prometheus.io/port": "9153",
"prometheus.io/scrape": "true"
}
},
"spec": {
"ports": [
{
"name": "dns",
"protocol": "UDP",
"port": 53,
"targetPort": 53
},
{
"name": "dns-tcp",
"protocol": "TCP",
"port": 53,
"targetPort": 53
}
],
"selector": {
"k8s-app": "kube-dns"
},
"clusterIP": "172.20.0.10",
"type": "ClusterIP",
"sessionAffinity": "None"
},
"status": {
"loadBalancer": {}
}
}
]
}
{
"kind": "DaemonSetList",
"apiVersion": "apps/v1",
"metadata": {
"selfLink": "/apis/apps/v1/namespaces/kube-system/daemonsets",
"resourceVersion": "3359"
},
"items": [
{
"metadata": {
"name": "aws-node",
"namespace": "kube-system",
"selfLink": "/apis/apps/v1/namespaces/kube-system/daemonsets/aws-node",
"uid": "b385dd37-a3f1-11e9-82e6-0e594b7607e6",
"resourceVersion": "180",
"generation": 1,
"creationTimestamp": "2019-07-11T15:36:50Z",
"labels": {
"k8s-app": "aws-node"
},
"annotations": {
"deprecated.daemonset.template.generation": "1",
"kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"extensions/v1beta1\",\"kind\":\"DaemonSet\",\"metadata\":{\"annotations\":{},\"labels\":{\"k8s-app\":\"aws-node\"},\"name\":\"aws-node\",\"namespace\":\"kube-system\"},\"spec\":{\"selector\":{\"matchLabels\":{\"k8s-app\":\"aws-node\"}},\"template\":{\"metadata\":{\"annotations\":{\"scheduler.alpha.kubernetes.io/critical-pod\":\"\"},\"labels\":{\"k8s-app\":\"aws-node\"}},\"spec\":{\"affinity\":{\"nodeAffinity\":{\"requiredDuringSchedulingIgnoredDuringExecution\":{\"nodeSelectorTerms\":[{\"matchExpressions\":[{\"key\":\"beta.kubernetes.io/os\",\"operator\":\"In\",\"values\":[\"linux\"]},{\"key\":\"beta.kubernetes.io/arch\",\"operator\":\"In\",\"values\":[\"amd64\"]}]}]}}},\"containers\":[{\"env\":[{\"name\":\"AWS_VPC_K8S_CNI_LOGLEVEL\",\"value\":\"DEBUG\"},{\"name\":\"MY_NODE_NAME\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"spec.nodeName\"}}},{\"name\":\"WATCH_NAMESPACE\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.namespace\"}}}],\"image\":\"602401143452.dkr.ecr.us-east-1.amazonaws.com/amazon-k8s-cni:v1.4.1\",\"imagePullPolicy\":\"Always\",\"name\":\"aws-node\",\"ports\":[{\"containerPort\":61678,\"name\":\"metrics\"}],\"resources\":{\"requests\":{\"cpu\":\"10m\"}},\"securityContext\":{\"privileged\":true},\"volumeMounts\":[{\"mountPath\":\"/host/opt/cni/bin\",\"name\":\"cni-bin-dir\"},{\"mountPath\":\"/host/etc/cni/net.d\",\"name\":\"cni-net-dir\"},{\"mountPath\":\"/host/var/log\",\"name\":\"log-dir\"},{\"mountPath\":\"/var/run/docker.sock\",\"name\":\"dockersock\"}]}],\"hostNetwork\":true,\"priorityClassName\":\"system-node-critical\",\"serviceAccountName\":\"aws-node\",\"tolerations\":[{\"operator\":\"Exists\"}],\"volumes\":[{\"hostPath\":{\"path\":\"/opt/cni/bin\"},\"name\":\"cni-bin-dir\"},{\"hostPath\":{\"path\":\"/etc/cni/net.d\"},\"name\":\"cni-net-dir\"},{\"hostPath\":{\"path\":\"/var/log\"},\"name\":\"log-dir\"},{\"hostPath\":{\"path\":\"/var/run/docker.sock\"},\"name\":\"dockersock\"}]}},\"updateStrategy\":{\"type\":\"RollingUpdate\"}}}\n"
}
},
"spec": {
"selector": {
"matchLabels": {
"k8s-app": "aws-node"
}
},
"template": {
"metadata": {
"creationTimestamp": null,
"labels": {
"k8s-app": "aws-node"
},
"annotations": {
"scheduler.alpha.kubernetes.io/critical-pod": ""
}
},
"spec": {
"volumes": [
{
"name": "cni-bin-dir",
"hostPath": {
"path": "/opt/cni/bin",
"type": ""
}
},
{
"name": "cni-net-dir",
"hostPath": {
"path": "/etc/cni/net.d",
"type": ""
}
},
{
"name": "log-dir",
"hostPath": {
"path": "/var/log",
"type": ""
}
},
{
"name": "dockersock",
"hostPath": {
"path": "/var/run/docker.sock",
"type": ""
}
}
],
"containers": [
{
"name": "aws-node",
"image": "602401143452.dkr.ecr.us-east-1.amazonaws.com/amazon-k8s-cni:v1.4.1",
"ports": [
{
"name": "metrics",
"hostPort": 61678,
"containerPort": 61678,
"protocol": "TCP"
}
],
"env": [
{
"name": "AWS_VPC_K8S_CNI_LOGLEVEL",
"value": "DEBUG"
},
{
"name": "MY_NODE_NAME",
"valueFrom": {
"fieldRef": {
"apiVersion": "v1",
"fieldPath": "spec.nodeName"
}
}
},
{
"name": "WATCH_NAMESPACE",
"valueFrom": {
"fieldRef": {
"apiVersion": "v1",
"fieldPath": "metadata.namespace"
}
}
}
],
"resources": {
"requests": {
"cpu": "10m"
}
},
"volumeMounts": [
{
"name": "cni-bin-dir",
"mountPath": "/host/opt/cni/bin"
},
{
"name": "cni-net-dir",
"mountPath": "/host/etc/cni/net.d"
},
{
"name": "log-dir",
"mountPath": "/host/var/log"
},
{
"name": "dockersock",
"mountPath": "/var/run/docker.sock"
}
],
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "File",
"imagePullPolicy": "Always",
"securityContext": {
"privileged": true,
"procMount": "Default"
}
}
],
"restartPolicy": "Always",
"terminationGracePeriodSeconds": 30,
"dnsPolicy": "ClusterFirst",
"serviceAccountName": "aws-node",
"serviceAccount": "aws-node",
"hostNetwork": true,
"securityContext": {},
"affinity": {
"nodeAffinity": {
"requiredDuringSchedulingIgnoredDuringExecution": {
"nodeSelectorTerms": [
{
"matchExpressions": [
{
"key": "beta.kubernetes.io/os",
"operator": "In",
"values": [
"linux"
]
},
{
"key": "beta.kubernetes.io/arch",
"operator": "In",
"values": [
"amd64"
]
}
]
}
]
}
}
},
"schedulerName": "default-scheduler",
"tolerations": [
{
"operator": "Exists"
}
],
"priorityClassName": "system-node-critical"
}
},
"updateStrategy": {
"type": "RollingUpdate",
"rollingUpdate": {
"maxUnavailable": 1
}
},
"revisionHistoryLimit": 10
},
"status": {
"currentNumberScheduled": 0,
"numberMisscheduled": 0,
"desiredNumberScheduled": 0,
"numberReady": 0,
"observedGeneration": 1
}
},
{
"metadata": {
"name": "kube-proxy",
"namespace": "kube-system",
"selfLink": "/apis/apps/v1/namespaces/kube-system/daemonsets/kube-proxy",
"uid": "b39e3d0f-a3f1-11e9-82e6-0e594b7607e6",
"resourceVersion": "213",
"generation": 1,
"creationTimestamp": "2019-07-11T15:36:50Z",
"labels": {
"eks.amazonaws.com/component": "kube-proxy",
"k8s-app": "kube-proxy"
},
"annotations": {
"deprecated.daemonset.template.generation": "1",
"kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"extensions/v1beta1\",\"kind\":\"DaemonSet\",\"metadata\":{\"annotations\":{},\"labels\":{\"eks.amazonaws.com/component\":\"kube-proxy\",\"k8s-app\":\"kube-proxy\"},\"name\":\"kube-proxy\",\"namespace\":\"kube-system\"},\"spec\":{\"selector\":{\"matchLabels\":{\"k8s-app\":\"kube-proxy\"}},\"template\":{\"metadata\":{\"annotations\":{\"scheduler.alpha.kubernetes.io/critical-pod\":\"\"},\"labels\":{\"k8s-app\":\"kube-proxy\"}},\"spec\":{\"affinity\":{\"nodeAffinity\":{\"requiredDuringSchedulingIgnoredDuringExecution\":{\"nodeSelectorTerms\":[{\"matchExpressions\":[{\"key\":\"beta.kubernetes.io/os\",\"operator\":\"In\",\"values\":[\"linux\"]},{\"key\":\"beta.kubernetes.io/arch\",\"operator\":\"In\",\"values\":[\"amd64\"]}]}]}}},\"containers\":[{\"command\":[\"/bin/sh\",\"-c\",\"kube-proxy --v=2 --config=/var/lib/kube-proxy-config/config\"],\"image\":\"602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/kube-proxy:v1.12.6\",\"name\":\"kube-proxy\",\"resources\":{\"requests\":{\"cpu\":\"100m\"}},\"securityContext\":{\"privileged\":true},\"volumeMounts\":[{\"mountPath\":\"/var/log\",\"name\":\"varlog\",\"readOnly\":false},{\"mountPath\":\"/run/xtables.lock\",\"name\":\"xtables-lock\",\"readOnly\":false},{\"mountPath\":\"/lib/modules\",\"name\":\"lib-modules\",\"readOnly\":true},{\"mountPath\":\"/var/lib/kube-proxy/\",\"name\":\"kubeconfig\"},{\"mountPath\":\"/var/lib/kube-proxy-config/\",\"name\":\"config\"}]}],\"hostNetwork\":true,\"priorityClassName\":\"system-node-critical\",\"serviceAccountName\":\"kube-proxy\",\"tolerations\":[{\"effect\":\"NoExecute\",\"operator\":\"Exists\"},{\"effect\":\"NoSchedule\",\"operator\":\"Exists\"}],\"volumes\":[{\"hostPath\":{\"path\":\"/var/log\"},\"name\":\"varlog\"},{\"hostPath\":{\"path\":\"/run/xtables.lock\",\"type\":\"FileOrCreate\"},\"name\":\"xtables-lock\"},{\"hostPath\":{\"path\":\"/lib/modules\"},\"name\":\"lib-modules\"},{\"configMap\":{\"name\":\"kube-proxy\"},\"name\":\"kubeconfig\"},{\"configMap\":{\"name\":\"kube-proxy-config\"},\"name\":\"config\"}]}},\"updateStrategy\":{\"rollingUpdate\":{\"maxUnavailable\":\"10%\"},\"type\":\"RollingUpdate\"}}}\n"
}
},
"spec": {
"selector": {
"matchLabels": {
"k8s-app": "kube-proxy"
}
},
"template": {
"metadata": {
"creationTimestamp": null,
"labels": {
"k8s-app": "kube-proxy"
},
"annotations": {
"scheduler.alpha.kubernetes.io/critical-pod": ""
}
},
"spec": {
"volumes": [
{
"name": "varlog",
"hostPath": {
"path": "/var/log",
"type": ""
}
},
{
"name": "xtables-lock",
"hostPath": {
"path": "/run/xtables.lock",
"type": "FileOrCreate"
}
},
{
"name": "lib-modules",
"hostPath": {
"path": "/lib/modules",
"type": ""
}
},
{
"name": "kubeconfig",
"configMap": {
"name": "kube-proxy",
"defaultMode": 420
}
},
{
"name": "config",
"configMap": {
"name": "kube-proxy-config",
"defaultMode": 420
}
}
],
"containers": [
{
"name": "kube-proxy",
"image": "602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/kube-proxy:v1.12.6",
"command": [
"/bin/sh",
"-c",
"kube-proxy --v=2 --config=/var/lib/kube-proxy-config/config"
],
"resources": {
"requests": {
"cpu": "100m"
}
},
"volumeMounts": [
{
"name": "varlog",
"mountPath": "/var/log"
},
{
"name": "xtables-lock",
"mountPath": "/run/xtables.lock"
},
{
"name": "lib-modules",
"readOnly": true,
"mountPath": "/lib/modules"
},
{
"name": "kubeconfig",
"mountPath": "/var/lib/kube-proxy/"
},
{
"name": "config",
"mountPath": "/var/lib/kube-proxy-config/"
}
],
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "File",
"imagePullPolicy": "IfNotPresent",
"securityContext": {
"privileged": true,
"procMount": "Default"
}
}
],
"restartPolicy": "Always",
"terminationGracePeriodSeconds": 30,
"dnsPolicy": "ClusterFirst",
"serviceAccountName": "kube-proxy",
"serviceAccount": "kube-proxy",
"hostNetwork": true,
"securityContext": {},
"affinity": {
"nodeAffinity": {
"requiredDuringSchedulingIgnoredDuringExecution": {
"nodeSelectorTerms": [
{
"matchExpressions": [
{
"key": "beta.kubernetes.io/os",
"operator": "In",
"values": [
"linux"
]
},
{
"key": "beta.kubernetes.io/arch",
"operator": "In",
"values": [
"amd64"
]
}
]
}
]
}
}
},
"schedulerName": "default-scheduler",
"tolerations": [
{
"operator": "Exists",
"effect": "NoExecute"
},
{
"operator": "Exists",
"effect": "NoSchedule"
}
],
"priorityClassName": "system-node-critical"
}
},
"updateStrategy": {
"type": "RollingUpdate",
"rollingUpdate": {
"maxUnavailable": "10%"
}
},
"revisionHistoryLimit": 10
},
"status": {
"currentNumberScheduled": 0,
"numberMisscheduled": 0,
"desiredNumberScheduled": 0,
"numberReady": 0,
"observedGeneration": 1
}
}
]
}
{
"kind": "DeploymentList",
"apiVersion": "apps/v1",
"metadata": {
"selfLink": "/apis/apps/v1/namespaces/kube-system/deployments",
"resourceVersion": "3359"
},
"items": [
{
"metadata": {
"name": "coredns",
"namespace": "kube-system",
"selfLink": "/apis/apps/v1/namespaces/kube-system/deployments/coredns",
"uid": "b3928c68-a3f1-11e9-82e6-0e594b7607e6",
"resourceVersion": "207",
"generation": 1,
"creationTimestamp": "2019-07-11T15:36:50Z",
"labels": {
"eks.amazonaws.com/component": "coredns",
"k8s-app": "kube-dns",
"kubernetes.io/name": "CoreDNS"
},
"annotations": {
"deployment.kubernetes.io/revision": "1",
"kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"extensions/v1beta1\",\"kind\":\"Deployment\",\"metadata\":{\"annotations\":{},\"labels\":{\"eks.amazonaws.com/component\":\"coredns\",\"k8s-app\":\"kube-dns\",\"kubernetes.io/name\":\"CoreDNS\"},\"name\":\"coredns\",\"namespace\":\"kube-system\"},\"spec\":{\"replicas\":2,\"selector\":{\"matchLabels\":{\"eks.amazonaws.com/component\":\"coredns\",\"k8s-app\":\"kube-dns\"}},\"strategy\":{\"rollingUpdate\":{\"maxUnavailable\":1},\"type\":\"RollingUpdate\"},\"template\":{\"metadata\":{\"labels\":{\"eks.amazonaws.com/component\":\"coredns\",\"k8s-app\":\"kube-dns\"}},\"spec\":{\"affinity\":{\"nodeAffinity\":{\"requiredDuringSchedulingIgnoredDuringExecution\":{\"nodeSelectorTerms\":[{\"matchExpressions\":[{\"key\":\"beta.kubernetes.io/os\",\"operator\":\"In\",\"values\":[\"linux\"]},{\"key\":\"beta.kubernetes.io/arch\",\"operator\":\"In\",\"values\":[\"amd64\"]}]}]}}},\"containers\":[{\"args\":[\"-conf\",\"/etc/coredns/Corefile\"],\"image\":\"602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/coredns:v1.2.2\",\"imagePullPolicy\":\"IfNotPresent\",\"livenessProbe\":{\"failureThreshold\":5,\"httpGet\":{\"path\":\"/health\",\"port\":8080,\"scheme\":\"HTTP\"},\"initialDelaySeconds\":60,\"successThreshold\":1,\"timeoutSeconds\":5},\"name\":\"coredns\",\"ports\":[{\"containerPort\":53,\"name\":\"dns\",\"protocol\":\"UDP\"},{\"containerPort\":53,\"name\":\"dns-tcp\",\"protocol\":\"TCP\"},{\"containerPort\":9153,\"name\":\"metrics\",\"protocol\":\"TCP\"}],\"resources\":{\"limits\":{\"memory\":\"170Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"70Mi\"}},\"securityContext\":{\"allowPrivilegeEscalation\":false,\"capabilities\":{\"add\":[\"NET_BIND_SERVICE\"],\"drop\":[\"all\"]},\"readOnlyRootFilesystem\":true},\"volumeMounts\":[{\"mountPath\":\"/etc/coredns\",\"name\":\"config-volume\",\"readOnly\":true}]}],\"dnsPolicy\":\"Default\",\"priorityClassName\":\"system-cluster-critical\",\"serviceAccountName\":\"coredns\",\"tolerations\":[{\"effect\":\"NoSchedule\",\"key\":\"node-role.kubernetes.io/master\"},{\"key\":\"CriticalAddonsOnly\",\"operator\":\"Exists\"}],\"volumes\":[{\"configMap\":{\"items\":[{\"key\":\"Corefile\",\"path\":\"Corefile\"}],\"name\":\"coredns\"},\"name\":\"config-volume\"}]}}}}\n"
}
},
"spec": {
"replicas": 2,
"selector": {
"matchLabels": {
"eks.amazonaws.com/component": "coredns",
"k8s-app": "kube-dns"
}
},
"template": {
"metadata": {
"creationTimestamp": null,
"labels": {
"eks.amazonaws.com/component": "coredns",
"k8s-app": "kube-dns"
}
},
"spec": {
"volumes": [
{
"name": "config-volume",
"configMap": {
"name": "coredns",
"items": [
{
"key": "Corefile",
"path": "Corefile"
}
],
"defaultMode": 420
}
}
],
"containers": [
{
"name": "coredns",
"image": "602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/coredns:v1.2.2",
"args": [
"-conf",
"/etc/coredns/Corefile"
],
"ports": [
{
"name": "dns",
"containerPort": 53,
"protocol": "UDP"
},
{
"name": "dns-tcp",
"containerPort": 53,
"protocol": "TCP"
},
{
"name": "metrics",
"containerPort": 9153,
"protocol": "TCP"
}
],
"resources": {
"limits": {
"memory": "170Mi"
},
"requests": {
"cpu": "100m",
"memory": "70Mi"
}
},
"volumeMounts": [
{
"name": "config-volume",
"readOnly": true,
"mountPath": "/etc/coredns"
}
],
"livenessProbe": {
"httpGet": {
"path": "/health",
"port": 8080,
"scheme": "HTTP"
},
"initialDelaySeconds": 60,
"timeoutSeconds": 5,
"periodSeconds": 10,
"successThreshold": 1,
"failureThreshold": 5
},
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "File",
"imagePullPolicy": "IfNotPresent",
"securityContext": {
"capabilities": {
"add": [
"NET_BIND_SERVICE"
],
"drop": [
"all"
]
},
"readOnlyRootFilesystem": true,
"allowPrivilegeEscalation": false,
"procMount": "Default"
}
}
],
"restartPolicy": "Always",
"terminationGracePeriodSeconds": 30,
"dnsPolicy": "Default",
"serviceAccountName": "coredns",
"serviceAccount": "coredns",
"securityContext": {},
"affinity": {
"nodeAffinity": {
"requiredDuringSchedulingIgnoredDuringExecution": {
"nodeSelectorTerms": [
{
"matchExpressions": [
{
"key": "beta.kubernetes.io/os",
"operator": "In",
"values": [
"linux"
]
},
{
"key": "beta.kubernetes.io/arch",
"operator": "In",
"values": [
"amd64"
]
}
]
}
]
}
}
},
"schedulerName": "default-scheduler",
"tolerations": [
{
"key": "node-role.kubernetes.io/master",
"effect": "NoSchedule"
},
{
"key": "CriticalAddonsOnly",
"operator": "Exists"
}
],
"priorityClassName": "system-cluster-critical"
}
},
"strategy": {
"type": "RollingUpdate",
"rollingUpdate": {
"maxUnavailable": 1,
"maxSurge": 1
}
},
"revisionHistoryLimit": 10,
"progressDeadlineSeconds": 2147483647
},
"status": {
"observedGeneration": 1,
"replicas": 2,
"updatedReplicas": 2,
"unavailableReplicas": 2,
"conditions": [
{
"type": "Available",
"status": "False",
"lastUpdateTime": "2019-07-11T15:36:50Z",
"lastTransitionTime": "2019-07-11T15:36:50Z",
"reason": "MinimumReplicasUnavailable",
"message": "Deployment does not have minimum availability."
}
]
}
}
]
}
{
"kind": "ReplicaSetList",
"apiVersion": "apps/v1",
"metadata": {
"selfLink": "/apis/apps/v1/namespaces/kube-system/replicasets",
"resourceVersion": "3359"
},
"items": [
{
"metadata": {
"name": "coredns-7f66c6c4b9",
"namespace": "kube-system",
"selfLink": "/apis/apps/v1/namespaces/kube-system/replicasets/coredns-7f66c6c4b9",
"uid": "b3937942-a3f1-11e9-82e6-0e594b7607e6",
"resourceVersion": "205",
"generation": 1,
"creationTimestamp": "2019-07-11T15:36:50Z",
"labels": {
"eks.amazonaws.com/component": "coredns",
"k8s-app": "kube-dns",
"pod-template-hash": "7f66c6c4b9"
},
"annotations": {
"deployment.kubernetes.io/desired-replicas": "2",
"deployment.kubernetes.io/max-replicas": "3",
"deployment.kubernetes.io/revision": "1"
},
"ownerReferences": [
{
"apiVersion": "apps/v1",
"kind": "Deployment",
"name": "coredns",
"uid": "b3928c68-a3f1-11e9-82e6-0e594b7607e6",
"controller": true,
"blockOwnerDeletion": true
}
]
},
"spec": {
"replicas": 2,
"selector": {
"matchLabels": {
"eks.amazonaws.com/component": "coredns",
"k8s-app": "kube-dns",
"pod-template-hash": "7f66c6c4b9"
}
},
"template": {
"metadata": {
"creationTimestamp": null,
"labels": {
"eks.amazonaws.com/component": "coredns",
"k8s-app": "kube-dns",
"pod-template-hash": "7f66c6c4b9"
}
},
"spec": {
"volumes": [
{
"name": "config-volume",
"configMap": {
"name": "coredns",
"items": [
{
"key": "Corefile",
"path": "Corefile"
}
],
"defaultMode": 420
}
}
],
"containers": [
{
"name": "coredns",
"image": "602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/coredns:v1.2.2",
"args": [
"-conf",
"/etc/coredns/Corefile"
],
"ports": [
{
"name": "dns",
"containerPort": 53,
"protocol": "UDP"
},
{
"name": "dns-tcp",
"containerPort": 53,
"protocol": "TCP"
},
{
"name": "metrics",
"containerPort": 9153,
"protocol": "TCP"
}
],
"resources": {
"limits": {
"memory": "170Mi"
},
"requests": {
"cpu": "100m",
"memory": "70Mi"
}
},
"volumeMounts": [
{
"name": "config-volume",
"readOnly": true,
"mountPath": "/etc/coredns"
}
],
"livenessProbe": {
"httpGet": {
"path": "/health",
"port": 8080,
"scheme": "HTTP"
},
"initialDelaySeconds": 60,
"timeoutSeconds": 5,
"periodSeconds": 10,
"successThreshold": 1,
"failureThreshold": 5
},
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "File",
"imagePullPolicy": "IfNotPresent",
"securityContext": {
"capabilities": {
"add": [
"NET_BIND_SERVICE"
],
"drop": [
"all"
]
},
"readOnlyRootFilesystem": true,
"allowPrivilegeEscalation": false,
"procMount": "Default"
}
}
],
"restartPolicy": "Always",
"terminationGracePeriodSeconds": 30,
"dnsPolicy": "Default",
"serviceAccountName": "coredns",
"serviceAccount": "coredns",
"securityContext": {},
"affinity": {
"nodeAffinity": {
"requiredDuringSchedulingIgnoredDuringExecution": {
"nodeSelectorTerms": [
{
"matchExpressions": [
{
"key": "beta.kubernetes.io/os",
"operator": "In",
"values": [
"linux"
]
},
{
"key": "beta.kubernetes.io/arch",
"operator": "In",
"values": [
"amd64"
]
}
]
}
]
}
}
},
"schedulerName": "default-scheduler",
"tolerations": [
{
"key": "node-role.kubernetes.io/master",
"effect": "NoSchedule"
},
{
"key": "CriticalAddonsOnly",
"operator": "Exists"
}
],
"priorityClassName": "system-cluster-critical"
}
}
},
"status": {
"replicas": 2,
"fullyLabeledReplicas": 2,
"observedGeneration": 1
}
}
]
}
{
"kind": "PodList",
"apiVersion": "v1",
"metadata": {
"selfLink": "/api/v1/namespaces/kube-system/pods",
"resourceVersion": "3359"
},
"items": [
{
"metadata": {
"name": "coredns-7f66c6c4b9-8l76x",
"generateName": "coredns-7f66c6c4b9-",
"namespace": "kube-system",
"selfLink": "/api/v1/namespaces/kube-system/pods/coredns-7f66c6c4b9-8l76x",
"uid": "b3961b28-a3f1-11e9-82e6-0e594b7607e6",
"resourceVersion": "3307",
"creationTimestamp": "2019-07-11T15:36:50Z",
"labels": {
"eks.amazonaws.com/component": "coredns",
"k8s-app": "kube-dns",
"pod-template-hash": "7f66c6c4b9"
},
"ownerReferences": [
{
"apiVersion": "apps/v1",
"kind": "ReplicaSet",
"name": "coredns-7f66c6c4b9",
"uid": "b3937942-a3f1-11e9-82e6-0e594b7607e6",
"controller": true,
"blockOwnerDeletion": true
}
]
},
"spec": {
"volumes": [
{
"name": "config-volume",
"configMap": {
"name": "coredns",
"items": [
{
"key": "Corefile",
"path": "Corefile"
}
],
"defaultMode": 420
}
},
{
"name": "coredns-token-tfssw",
"secret": {
"secretName": "coredns-token-tfssw",
"defaultMode": 420
}
}
],
"containers": [
{
"name": "coredns",
"image": "602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/coredns:v1.2.2",
"args": [
"-conf",
"/etc/coredns/Corefile"
],
"ports": [
{
"name": "dns",
"containerPort": 53,
"protocol": "UDP"
},
{
"name": "dns-tcp",
"containerPort": 53,
"protocol": "TCP"
},
{
"name": "metrics",
"containerPort": 9153,
"protocol": "TCP"
}
],
"resources": {
"limits": {
"memory": "170Mi"
},
"requests": {
"cpu": "100m",
"memory": "70Mi"
}
},
"volumeMounts": [
{
"name": "config-volume",
"readOnly": true,
"mountPath": "/etc/coredns"
},
{
"name": "coredns-token-tfssw",
"readOnly": true,
"mountPath": "/var/run/secrets/kubernetes.io/serviceaccount"
}
],
"livenessProbe": {
"httpGet": {
"path": "/health",
"port": 8080,
"scheme": "HTTP"
},
"initialDelaySeconds": 60,
"timeoutSeconds": 5,
"periodSeconds": 10,
"successThreshold": 1,
"failureThreshold": 5
},
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "File",
"imagePullPolicy": "IfNotPresent",
"securityContext": {
"capabilities": {
"add": [
"NET_BIND_SERVICE"
],
"drop": [
"all"
]
},
"readOnlyRootFilesystem": true,
"allowPrivilegeEscalation": false,
"procMount": "Default"
}
}
],
"restartPolicy": "Always",
"terminationGracePeriodSeconds": 30,
"dnsPolicy": "Default",
"serviceAccountName": "coredns",
"serviceAccount": "coredns",
"securityContext": {},
"affinity": {
"nodeAffinity": {
"requiredDuringSchedulingIgnoredDuringExecution": {
"nodeSelectorTerms": [
{
"matchExpressions": [
{
"key": "beta.kubernetes.io/os",
"operator": "In",
"values": [
"linux"
]
},
{
"key": "beta.kubernetes.io/arch",
"operator": "In",
"values": [
"amd64"
]
}
]
}
]
}
}
},
"schedulerName": "default-scheduler",
"tolerations": [
{
"key": "node-role.kubernetes.io/master",
"effect": "NoSchedule"
},
{
"key": "CriticalAddonsOnly",
"operator": "Exists"
},
{
"key": "node.kubernetes.io/not-ready",
"operator": "Exists",
"effect": "NoExecute",
"tolerationSeconds": 300
},
{
"key": "node.kubernetes.io/unreachable",
"operator": "Exists",
"effect": "NoExecute",
"tolerationSeconds": 300
}
],
"priorityClassName": "system-cluster-critical",
"priority": 2000000000
},
"status": {
"phase": "Pending",
"conditions": [
{
"type": "PodScheduled",
"status": "False",
"lastProbeTime": "2019-07-11T16:17:53Z",
"lastTransitionTime": "2019-07-11T15:36:50Z",
"reason": "Unschedulable",
"message": "no nodes available to schedule pods"
}
],
"qosClass": "Burstable"
}
},
{
"metadata": {
"name": "coredns-7f66c6c4b9-9pd82",
"generateName": "coredns-7f66c6c4b9-",
"namespace": "kube-system",
"selfLink": "/api/v1/namespaces/kube-system/pods/coredns-7f66c6c4b9-9pd82",
"uid": "b3955020-a3f1-11e9-82e6-0e594b7607e6",
"resourceVersion": "3306",
"creationTimestamp": "2019-07-11T15:36:50Z",
"labels": {
"eks.amazonaws.com/component": "coredns",
"k8s-app": "kube-dns",
"pod-template-hash": "7f66c6c4b9"
},
"ownerReferences": [
{
"apiVersion": "apps/v1",
"kind": "ReplicaSet",
"name": "coredns-7f66c6c4b9",
"uid": "b3937942-a3f1-11e9-82e6-0e594b7607e6",
"controller": true,
"blockOwnerDeletion": true
}
]
},
"spec": {
"volumes": [
{
"name": "config-volume",
"configMap": {
"name": "coredns",
"items": [
{
"key": "Corefile",
"path": "Corefile"
}
],
"defaultMode": 420
}
},
{
"name": "coredns-token-tfssw",
"secret": {
"secretName": "coredns-token-tfssw",
"defaultMode": 420
}
}
],
"containers": [
{
"name": "coredns",
"image": "602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/coredns:v1.2.2",
"args": [
"-conf",
"/etc/coredns/Corefile"
],
"ports": [
{
"name": "dns",
"containerPort": 53,
"protocol": "UDP"
},
{
"name": "dns-tcp",
"containerPort": 53,
"protocol": "TCP"
},
{
"name": "metrics",
"containerPort": 9153,
"protocol": "TCP"
}
],
"resources": {
"limits": {
"memory": "170Mi"
},
"requests": {
"cpu": "100m",
"memory": "70Mi"
}
},
"volumeMounts": [
{
"name": "config-volume",
"readOnly": true,
"mountPath": "/etc/coredns"
},
{
"name": "coredns-token-tfssw",
"readOnly": true,
"mountPath": "/var/run/secrets/kubernetes.io/serviceaccount"
}
],
"livenessProbe": {
"httpGet": {
"path": "/health",
"port": 8080,
"scheme": "HTTP"
},
"initialDelaySeconds": 60,
"timeoutSeconds": 5,
"periodSeconds": 10,
"successThreshold": 1,
"failureThreshold": 5
},
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "File",
"imagePullPolicy": "IfNotPresent",
"securityContext": {
"capabilities": {
"add": [
"NET_BIND_SERVICE"
],
"drop": [
"all"
]
},
"readOnlyRootFilesystem": true,
"allowPrivilegeEscalation": false,
"procMount": "Default"
}
}
],
"restartPolicy": "Always",
"terminationGracePeriodSeconds": 30,
"dnsPolicy": "Default",
"serviceAccountName": "coredns",
"serviceAccount": "coredns",
"securityContext": {},
"affinity": {
"nodeAffinity": {
"requiredDuringSchedulingIgnoredDuringExecution": {
"nodeSelectorTerms": [
{
"matchExpressions": [
{
"key": "beta.kubernetes.io/os",
"operator": "In",
"values": [
"linux"
]
},
{
"key": "beta.kubernetes.io/arch",
"operator": "In",
"values": [
"amd64"
]
}
]
}
]
}
}
},
"schedulerName": "default-scheduler",
"tolerations": [
{
"key": "node-role.kubernetes.io/master",
"effect": "NoSchedule"
},
{
"key": "CriticalAddonsOnly",
"operator": "Exists"
},
{
"key": "node.kubernetes.io/not-ready",
"operator": "Exists",
"effect": "NoExecute",
"tolerationSeconds": 300
},
{
"key": "node.kubernetes.io/unreachable",
"operator": "Exists",
"effect": "NoExecute",
"tolerationSeconds": 300
}
],
"priorityClassName": "system-cluster-critical",
"priority": 2000000000
},
"status": {
"phase": "Pending",
"conditions": [
{
"type": "PodScheduled",
"status": "False",
"lastProbeTime": "2019-07-11T16:17:53Z",
"lastTransitionTime": "2019-07-11T15:36:50Z",
"reason": "Unschedulable",
"message": "no nodes available to schedule pods"
}
],
"qosClass": "Burstable"
}
}
]
}
==== START logs for container coredns of pod kube-system/coredns-7f66c6c4b9-8l76x ====
==== END logs for container coredns of pod kube-system/coredns-7f66c6c4b9-8l76x ====
==== START logs for container coredns of pod kube-system/coredns-7f66c6c4b9-9pd82 ====
==== END logs for container coredns of pod kube-system/coredns-7f66c6c4b9-9pd82 ====
{
"kind": "EventList",
"apiVersion": "v1",
"metadata": {
"selfLink": "/api/v1/namespaces/default/events",
"resourceVersion": "3359"
},
"items": []
}
{
"kind": "ReplicationControllerList",
"apiVersion": "v1",
"metadata": {
"selfLink": "/api/v1/namespaces/default/replicationcontrollers",
"resourceVersion": "3359"
},
"items": []
}
{
"kind": "ServiceList",
"apiVersion": "v1",
"metadata": {
"selfLink": "/api/v1/namespaces/default/services",
"resourceVersion": "3360"
},
"items": [
{
"metadata": {
"name": "kubernetes",
"namespace": "default",
"selfLink": "/api/v1/namespaces/default/services/kubernetes",
"uid": "b057a1a4-a3f1-11e9-82e6-0e594b7607e6",
"resourceVersion": "32",
"creationTimestamp": "2019-07-11T15:36:45Z",
"labels": {
"component": "apiserver",
"provider": "kubernetes"
}
},
"spec": {
"ports": [
{
"name": "https",
"protocol": "TCP",
"port": 443,
"targetPort": 443
}
],
"clusterIP": "172.20.0.1",
"type": "ClusterIP",
"sessionAffinity": "None"
},
"status": {
"loadBalancer": {}
}
}
]
}
{
"kind": "DaemonSetList",
"apiVersion": "apps/v1",
"metadata": {
"selfLink": "/apis/apps/v1/namespaces/default/daemonsets",
"resourceVersion": "3360"
},
"items": []
}
{
"kind": "DeploymentList",
"apiVersion": "apps/v1",
"metadata": {
"selfLink": "/apis/apps/v1/namespaces/default/deployments",
"resourceVersion": "3360"
},
"items": []
}
{
"kind": "ReplicaSetList",
"apiVersion": "apps/v1",
"metadata": {
"selfLink": "/apis/apps/v1/namespaces/default/replicasets",
"resourceVersion": "3360"
},
"items": []
}
{
"kind": "PodList",
"apiVersion": "v1",
"metadata": {
"selfLink": "/api/v1/namespaces/default/pods",
"resourceVersion": "3360"
},
"items": []
}
Cluster info dumped to standard output
So, I figured out what was causing my problem, but I don't know _why_...
Once I compared the EKS basic example with the VPC module I used as a template I noticed that EC2 endpoints were a really bad idea and now the nodes join just fine.
In the VPC (using terraform-aws-modules/vpc/aws v2.7.0) I just removed the following:
# VPC Endpoint for EC2
enable_ec2_endpoint = false
ec2_endpoint_private_dns_enabled = false
ec2_endpoint_security_group_ids = []
EDIT:
I also had to remove ECR DKR since the nodes were stuck in "Not Ready"
# VPC Endpoint for ECR DKR
enable_ecr_dkr_endpoint = false
ecr_dkr_endpoint_private_dns_enabled = false
ecr_dkr_endpoint_security_group_ids = []
So this may be circumstantial but I found that when following the basic example, the created VPC lacks enable_dns_hostnames = true, which, to my understanding makes it impossible to resolve hostnames like ip-10-100-5-7.us-west-2.compute.internal. AWS docs on it are somewhat muddy (as usual) but I think this setting enables VPC-internal resolution of assigned private and public IPs. Enabling it seems to work with some delay - I was able to dig/ssh to a worker node from a bastion host in the same VPC but a public subnet, using its hostname rather than the IP.
While I was able to resolve a worker's hostname from the bastion, kubernetes continued to fail to do so - same error in logs, which went like:
Jul 24 11:18:46 ip-10-0-1-36.eu-west-1.compute.internal kubelet[4355]: E0724 11:18:46.602539 4355 kubelet.go:2236] node "ip-10-0-1-36.eu-west-1.compute.internal" not found
Jul 24 11:18:46 ip-10-0-1-36.eu-west-1.compute.internal kubelet[4355]: E0724 11:18:46.682540 4355 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:442: Failed to list *v1.Service: Unauthorized
Jul 24 11:18:46 ip-10-0-1-36.eu-west-1.compute.internal kubelet[4355]: E0724 11:18:46.703201 4355 kubelet.go:2236] node "ip-10-0-1-36.eu-west-1.compute.internal" not found
Jul 24 11:18:46 ip-10-0-1-36.eu-west-1.compute.internal kubelet[4355]: E0724 11:18:46.803935 4355 kubelet.go:2236] node "ip-10-0-1-36.eu-west-1.compute.internal" not found
Jul 24 11:18:46 ip-10-0-1-36.eu-west-1.compute.internal kubelet[4355]: E0724 11:18:46.900234 4355 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Unauthorized
Jul 24 11:18:46 ip-10-0-1-36.eu-west-1.compute.internal kubelet[4355]: E0724 11:18:46.900771 4355 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Node: Unauthorized
however after correcting and applying terraform (i previously downscaled the workers and enabled DNS resolution in the VPC manually), it updated the workers launch template and the cluster, after which nodes started being discovered.
There are 2 possibilities then:
1) Not enabling DNS resolution may break node registration (yesterday out of my 2 attempts it worked once and didn't work once, today it failed once, then i turned DNS resolution on)
2) DNS resolution may have nothing to do with it and I just got lucky when doing update-in-place with TF today. However, when I destroyed and recreated the cluster, it found the nodes immediately.
I cannot run many experiments like that because the full create-destroy cycle takes around 25 minutes.
_Seriously, if you can, just use K8s on GCP._
karolinepauls
on 24 Jul 2019
So to keep it short, one actionable item from my comment above is the question:
Why is DNS resolution not enabled by default in the example? Could this interfere with node discovery or cluster configuration?
It's worth noting AWS docs state:
Your VPC must have DNS hostname and DNS resolution support. Otherwise, your worker nodes cannot register with your cluster. For more information, see Using DNS with Your VPC in the Amazon VPC User Guide.
karolinepauls
on 24 Jul 2019
Why is DNS resolution not enabled by default in the example?
It is now: https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/examples/basic/main.tf#L96
Will close this as there's been no updates. Also issues just like this have been covered in detail so I think there's already enough info available.
max-rocket-internet
on 30 Oct 2019
Related issues
swapzero
路
4Comments
mojochao
路
3Comments
max-rocket-internet
路
4Comments
linxcat
路
4Comments
maganuk
路
4Comments
Most helpful comment
So to keep it short, one actionable item from my comment above is the question:
Why is DNS resolution not enabled by default in the example? Could this interfere with node discovery or cluster configuration?
It's worth noting AWS docs state: