kubeadm init (offline mode) stuck on pulling images
BUG REPORT
Versions
v1.10.0
kubeadm version (use kubeadm version):
v1.10.0
Environment:
- Kubernetes version 1.10.0
- Cloud provider or hardware configuration: On premise (OFFLINE)
- OS (e.g. from /etc/os-release): Centos
- Kernel (e.g.
uname -a): 3.10.0-514 - Others: Docker images are pre-loaded
What happened?
Running the same process on the internet works flawless, however on offline mode
my kubeadm init process is stuck on pulling images.
On the /var/log/messages I can see a lots of retries to connect to 127.0.0.1:6443 while
this port doesn't exists on the system.
The following docker containers exists when running docker ps:
controller-mananger
etcd
4 x pause
What you expected to happen?
To run exactly as if I were online and succeed with cluster deployment
How to reproduce it (as minimally and precisely as possible)?
- Create a local RPM repository that have the v1.10.0 packages
- Set your Centos to work with that repository
- Download all the docker containers that v1.10.0 requires and load them to your machine
- Go offline completely
- Disable DNS resolving
geostant
All 27 comments
/assign @chuckha - please triage.
timothysc
on 4 Apr 2018
@geostant Would you be able to provide a list of docker images downloaded prior to going offline?
chuckha
on 6 Apr 2018
@chuckha
I got a script that download the relevant images and save them to tar files before I transfer them
to an offline server.
These are the images based on installed kubectl, kubelet and kubeadm version v1.10.0:
k8s.gcr.io/kube-proxy-amd64 v1.10.0 bfc21aadc7d3 12 days ago 97 MB
k8s.gcr.io/kube-scheduler-amd64 v1.10.0 704ba848e69a 12 days ago 50.4 MB
k8s.gcr.io/kube-controller-manager-amd64 v1.10.0 ad86dbed1555 12 days ago 148 MB
k8s.gcr.io/kube-apiserver-amd64 v1.10.0 af20925d51a3 12 days ago 225 MB
weaveworks/weave-npc 2.2.1 26d868a4eb75 3 weeks ago 46.6 MB
weaveworks/weave-kube 2.2.1 86e2da7dd27b 3 weeks ago 93.8 MB
k8s.gcr.io/etcd-amd64 3.1.12 52920ad46f5b 4 weeks ago 193 MB
k8s.gcr.io/kubernetes-dashboard-amd64 v1.8.3 0c60bcf89900 7 weeks ago 102 MB
k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64 1.14.8 c2ce1ffb51ed 3 months ago 40.9 MB
k8s.gcr.io/k8s-dns-sidecar-amd64 1.14.8 6f7f2dc7fab5 3 months ago 42.2 MB
k8s.gcr.io/k8s-dns-kube-dns-amd64 1.14.8 80cc5ea4b547 3 months ago 50.5 MB
k8s.gcr.io/pause-amd64 3.1 da86e6ba6ca1 3 months ago 742 kB
After I save them to tars and move them to the offline server,
I load them to my docker engine and run kubeadm init --kubernetes-version v1.10.0
geostant
on 8 Apr 2018
Some logs that I've extracted from the offline system (I had to type it word by word):
localhost kubelet: E0408 certificate_manager.go:299] Failed while requesting a signed certificate from the master: cannot create certificate signing request: Port https://xxx.xxx.xxx.xxx:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dail tcp xxx.xxx.xxx.xxx:6443: getsockopt: connection refused
localhost kubelet: W0408 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
localhost kubelet: E0408 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
localhost kubelet: E0408 event.go:209] Unable to wrtie event: 'Post https://xxx.xxx.xxx.xxx:6443/api/v1/namespaces/default/events: dual tcp xxx.xxx.xxx.xxx:6443: getsockopt: connection refused'
Obviously the main issue is port 6443 which is down
I've 2 console screens - one of the kubeadm and the second with ss -tuna | grep -i listen where I obviously missing 6443 in the port list.
SELINUX is disabled, firewalld is disabled and of course running as root.
The only ports that I can see up and listening (in addition to 22) are:
127.0.0.1:2379
127.0.0.1:10252
127.0.0.1:2380
127.0.0.1:10248
:::10250
:::10255
If you run docker ps -a has the apiserver container ever been started?
geoffgarside
on 9 Apr 2018
@geoffgarside There is a "pasue" place holder for the api server
I got the following containers:
k8s.gcr.io/kube-controller-manager:v1.10.0 | k8s_kube-controller-manager
k8s.gcr.io/pause-amd:3.1 | k8s_POD_kube-controller-manager
k8s.gcr.io/pause-amd:3.1 | k8s_POD_kube-apiserver
k8s.gcr.io/pause-amd:3.1 | k8s_POD_etcd
k8s.gcr.io/pause-amd:3.1 | k8s_POD_kube-scheduler
I'm going to try and run the same process outside on AWS on a machine that isn't connected to the internet (will connect thru VPN) to try and simulate the environment inside our isolated network
geostant
on 9 Apr 2018
Note that the etcd container is not running. It should run first. Is there any clue for this?
EDIT: just find in the issue body that you said etcd is running :) I didn't see it in https://github.com/kubernetes/kubeadm/issues/742#comment-379721844
xiangpengzhao
on 9 Apr 2018
I've done testing a similar isolated environment on an internet machine where the network connection was physically disconnected - tldr - it failed with the same results as I had before.
However this time I can provide a very detailed logs and process as it's a machine on the internet, so here it goes:
Command history
1 - Installed docker
yum -y install yum-utils
yum-config-manager --add-repo https://yum.dockerproject.org/repo/main/centos/7
yum search --showduplicates docker-engine
yum -y --nogpgcheck install docker-engine-1.12.6-1.el7.centos.x86_64
systemctl enable docker && systemctl start docker
2 - Loading docker images for offline use (Downloaded with docker pull the following images)
k8s.gcr.io/kube-proxy-amd64 v1.10.0 bfc21aadc7d3 2 weeks ago 97.04 MB
k8s.gcr.io/kube-apiserver-amd64 v1.10.0 704ba848e69a 2 weeks ago 50.39 MB
k8s.gcr.io/kube-controller-manager-amd64 v1.10.0 ad86dbed1555 2 weeks ago 147.8 MB
k8s.gcr.io/kube-scheduler-amd64 v1.10.0 af20925d51a3 2 weeks ago 225 MB
weaveworks/weave-npc 2.2.1 26d868a4eb75 4 weeks ago 46.55 MB
weaveworks/weave-kube 2.2.1 86e2da7dd27b 4 weeks ago 93.82 MB
k8s.gcr.io/etcd-amd64 3.1.12 52920ad46f5b 4 weeks ago 193.2 MB
k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64 1.14.8 c2ce1ffb51ed 3 months ago 40.95 MB
k8s.gcr.io/k8s-dns-sidecar-amd64 1.14.8 6f7f2dc7fab5 3 months ago 42.21 MB
k8s.gcr.io/k8s-dns-kube-dns-amd64 1.14.8 80cc5ea4b547 3 months ago 50.45 MB
k8s.gcr.io/pause-amd64 3.1 da86e6ba6ca1 3 months ago 742.5 kB
3 - Installed kubectl, kubeadm, kubelet
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF
setenforce 0
yum install -y kubelet kubeadm kubectl
systemctl enable kubelet && systemctl start kubelet
cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system
sed -i "s/cgroup-driver=systemd/cgroup-driver=cgroupfs/g" /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
systemctl daemon-reload
systemctl restart kubelet
4 - Initiated the kubeadm init process
kubeadm init --kubernetes-version v1.10.0
Logs from kubeadm
[root@localhost vagrant]# kubeadm init --kubernetes-version v1.10.0
[init] Using Kubernetes version: v1.10.0
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks.
[WARNING FileExisting-crictl]: crictl not found in system path
Suggestion: go get github.com/kubernetes-incubator/cri-tools/cmd/crictl
[preflight] Starting the kubelet service
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [localhost.localdomain kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.0.2.15]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated etcd/ca certificate and key.
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [localhost] and IPs [127.0.0.1]
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [localhost.localdomain] and IPs [10.0.2.15]
[certificates] Generated etcd/healthcheck-client certificate and key.
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] Waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests".
[init] This might take a minute or longer if the control plane images have to be pulled.
^C
Logs from /var/log/messages
Apr 11 07:55:56 localhost kubelet: E0411 07:55:56.539063 2154 event.go:209] Unable to write event: 'Post https://10.0.2.15:6443/api/v1/namespaces/default/events: dial tcp 10.0.2.15:6443: getsockopt: connection refused' (may retry after sleeping)
Apr 11 07:55:56 localhost kubelet: E0411 07:55:56.826758 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:55:56 localhost kubelet: E0411 07:55:56.828918 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:55:56 localhost kubelet: E0411 07:55:56.838127 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:55:57 localhost kubelet: E0411 07:55:57.827728 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:55:57 localhost kubelet: E0411 07:55:57.829479 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:55:57 localhost kubelet: E0411 07:55:57.838789 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:55:57 localhost kubelet: W0411 07:55:57.953851 2154 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
Apr 11 07:55:57 localhost kubelet: E0411 07:55:57.954052 2154 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Apr 11 07:55:58 localhost kubelet: I0411 07:55:58.405846 2154 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
Apr 11 07:55:58 localhost kubelet: I0411 07:55:58.410046 2154 kubelet_node_status.go:82] Attempting to register node localhost.localdomain
Apr 11 07:55:58 localhost kubelet: E0411 07:55:58.410567 2154 kubelet_node_status.go:106] Unable to register node "localhost.localdomain" with API server: Post https://10.0.2.15:6443/api/v1/nodes: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:55:58 localhost kubelet: E0411 07:55:58.828765 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:55:58 localhost kubelet: E0411 07:55:58.830926 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:55:58 localhost kubelet: E0411 07:55:58.839906 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:55:59 localhost kubelet: E0411 07:55:59.830148 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:55:59 localhost kubelet: E0411 07:55:59.832330 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:55:59 localhost kubelet: E0411 07:55:59.841021 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:00 localhost kubelet: I0411 07:56:00.569142 2154 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
Apr 11 07:56:00 localhost kubelet: W0411 07:56:00.574019 2154 status_manager.go:461] Failed to get status for pod "kube-scheduler-localhost.localdomain_kube-system(31cf0ccbee286239d451edb6fb511513)": Get https://10.0.2.15:6443/api/v1/namespaces/kube-system/pods/kube-scheduler-localhost.localdomain: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:00 localhost kubelet: E0411 07:56:00.831335 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:00 localhost kubelet: E0411 07:56:00.833088 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:00 localhost kubelet: E0411 07:56:00.841839 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:00 localhost kubelet: I0411 07:56:00.874022 2154 kuberuntime_manager.go:513] Container {Name:kube-scheduler Image:k8s.gcr.io/kube-scheduler-amd64:v1.10.0 Command:[kube-scheduler --address=127.0.0.1 --leader-elect=true --kubeconfig=/etc/kubernetes/scheduler.conf] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[] Requests:map[cpu:{i:{value:100 scale:-3} d:{Dec:<nil>} s:100m Format:DecimalSI}]} VolumeMounts:[{Name:kubeconfig ReadOnly:true MountPath:/etc/kubernetes/scheduler.conf SubPath: MountPropagation:<nil>}] VolumeDevices:[] LivenessProbe:&Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:10251,Host:127.0.0.1,Scheme:HTTP,HTTPHeaders:[],},TCPSocket:nil,},InitialDelaySeconds:15,TimeoutSeconds:15,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:8,} ReadinessProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} is dead, but RestartPolicy says that we should restart it.
Apr 11 07:56:00 localhost kubelet: I0411 07:56:00.874267 2154 kuberuntime_manager.go:757] checking backoff for container "kube-scheduler" in pod "kube-scheduler-localhost.localdomain_kube-system(31cf0ccbee286239d451edb6fb511513)"
Apr 11 07:56:00 localhost kubelet: I0411 07:56:00.874459 2154 kuberuntime_manager.go:767] Back-off 1m20s restarting failed container=kube-scheduler pod=kube-scheduler-localhost.localdomain_kube-system(31cf0ccbee286239d451edb6fb511513)
Apr 11 07:56:00 localhost kubelet: E0411 07:56:00.874537 2154 pod_workers.go:186] Error syncing pod 31cf0ccbee286239d451edb6fb511513 ("kube-scheduler-localhost.localdomain_kube-system(31cf0ccbee286239d451edb6fb511513)"), skipping: failed to "StartContainer" for "kube-scheduler" with CrashLoopBackOff: "Back-off 1m20s restarting failed container=kube-scheduler pod=kube-scheduler-localhost.localdomain_kube-system(31cf0ccbee286239d451edb6fb511513)"
Apr 11 07:56:01 localhost kubelet: E0411 07:56:01.832867 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:01 localhost kubelet: E0411 07:56:01.835118 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:01 localhost kubelet: E0411 07:56:01.846720 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:01 localhost systemd-udevd: inotify_add_watch(7, /dev/dm-9, 10) failed: No such file or directory
Apr 11 07:56:01 localhost systemd-udevd: inotify_add_watch(7, /dev/dm-9, 10) failed: No such file or directory
Apr 11 07:56:02 localhost systemd-udevd: inotify_add_watch(7, /dev/dm-9, 10) failed: No such file or directory
Apr 11 07:56:02 localhost systemd-udevd: inotify_add_watch(7, /dev/dm-9, 10) failed: No such file or directory
Apr 11 07:56:02 localhost systemd-udevd: inotify_add_watch(7, /dev/dm-9, 10) failed: No such file or directory
Apr 11 07:56:02 localhost systemd-udevd: inotify_add_watch(7, /dev/dm-9, 10) failed: No such file or directory
Apr 11 07:56:02 localhost kubelet: I0411 07:56:02.415358 2154 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
Apr 11 07:56:02 localhost kubelet: I0411 07:56:02.416629 2154 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
Apr 11 07:56:02 localhost kubelet: W0411 07:56:02.426564 2154 pod_container_deletor.go:77] Container "47563ed29f82ae74af4568d80e50a306f9b349cc1efdc12cdd7105d307d50061" not found in pod's containers
Apr 11 07:56:02 localhost kubelet: W0411 07:56:02.427533 2154 status_manager.go:461] Failed to get status for pod "kube-apiserver-localhost.localdomain_kube-system(96bb966b9c3212b8695ff3a763ac0e2c)": Get https://10.0.2.15:6443/api/v1/namespaces/kube-system/pods/kube-apiserver-localhost.localdomain: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:02 localhost kubelet: I0411 07:56:02.432448 2154 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
Apr 11 07:56:02 localhost kubelet: I0411 07:56:02.433369 2154 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
Apr 11 07:56:02 localhost kubelet: W0411 07:56:02.437887 2154 pod_container_deletor.go:77] Container "8566a3d9cd0f677bea2d2c9b0492684652db58f4cd673624365f7cef7609292a" not found in pod's containers
Apr 11 07:56:02 localhost kubelet: I0411 07:56:02.437965 2154 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
Apr 11 07:56:02 localhost kubelet: W0411 07:56:02.449831 2154 status_manager.go:461] Failed to get status for pod "kube-scheduler-localhost.localdomain_kube-system(31cf0ccbee286239d451edb6fb511513)": Get https://10.0.2.15:6443/api/v1/namespaces/kube-system/pods/kube-scheduler-localhost.localdomain: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:02 localhost kubelet: W0411 07:56:02.452606 2154 pod_container_deletor.go:77] Container "a41320b112b8bd2ec75d86e9f5c8d713523823f8a530036d59fdcbf79b36dd54" not found in pod's containers
Apr 11 07:56:02 localhost kubelet: I0411 07:56:02.727614 2154 kuberuntime_manager.go:513] Container {Name:kube-apiserver Image:k8s.gcr.io/kube-apiserver-amd64:v1.10.0 Command:[kube-apiserver --service-account-key-file=/etc/kubernetes/pki/sa.pub --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota --enable-bootstrap-token-auth=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-group-headers=X-Remote-Group --service-cluster-ip-range=10.96.0.0/12 --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --insecure-port=0 --requestheader-username-headers=X-Remote-User --requestheader-allowed-names=front-proxy-client --advertise-address=10.0.2.15 --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --allow-privileged=true --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/pki/apiserver.key --secure-port=6443 --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --client-ca-file=/etc/kubernetes/pki/ca.crt --authorization-mode=Node,RBAC --etcd-servers=https://127.0.0.1:2379 --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[] Requests:map[cpu:{i:{value:250 scale:-3} d:{Dec:<nil>} s:250m Format:DecimalSI}]} VolumeMounts:[{Name:k8s-certs ReadOnly:true MountPath:/etc/kubernetes/pki SubPath: MountPropagation:<nil>} {Name:ca-certs ReadOnly:true MountPath:/etc/ssl/certs SubPath: MountPropagation:<nil>} {Name:ca-certs-etc-pki ReadOnly:true MountPath:/etc/pki SubPath: MountPropagation:<nil>}] VolumeDevices:[] LivenessProbe:&Probe
Apr 11 07:56:02 localhost kubelet: {Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:6443,Host:10.0.2.15,Scheme:HTTPS,HTTPHeaders:[],},TCPSocket:nil,},InitialDelaySeconds:15,TimeoutSeconds:15,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:8,} ReadinessProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} is dead, but RestartPolicy says that we should restart it.
Apr 11 07:56:02 localhost kubelet: I0411 07:56:02.727816 2154 kuberuntime_manager.go:757] checking backoff for container "kube-apiserver" in pod "kube-apiserver-localhost.localdomain_kube-system(96bb966b9c3212b8695ff3a763ac0e2c)"
Apr 11 07:56:02 localhost kubelet: I0411 07:56:02.728065 2154 kuberuntime_manager.go:767] Back-off 1m20s restarting failed container=kube-apiserver pod=kube-apiserver-localhost.localdomain_kube-system(96bb966b9c3212b8695ff3a763ac0e2c)
Apr 11 07:56:02 localhost kubelet: E0411 07:56:02.728141 2154 pod_workers.go:186] Error syncing pod 96bb966b9c3212b8695ff3a763ac0e2c ("kube-apiserver-localhost.localdomain_kube-system(96bb966b9c3212b8695ff3a763ac0e2c)"), skipping: failed to "StartContainer" for "kube-apiserver" with CrashLoopBackOff: "Back-off 1m20s restarting failed container=kube-apiserver pod=kube-apiserver-localhost.localdomain_kube-system(96bb966b9c3212b8695ff3a763ac0e2c)"
Apr 11 07:56:02 localhost kubelet: I0411 07:56:02.749317 2154 kuberuntime_manager.go:513] Container {Name:kube-scheduler Image:k8s.gcr.io/kube-scheduler-amd64:v1.10.0 Command:[kube-scheduler --address=127.0.0.1 --leader-elect=true --kubeconfig=/etc/kubernetes/scheduler.conf] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[] Requests:map[cpu:{i:{value:100 scale:-3} d:{Dec:<nil>} s:100m Format:DecimalSI}]} VolumeMounts:[{Name:kubeconfig ReadOnly:true MountPath:/etc/kubernetes/scheduler.conf SubPath: MountPropagation:<nil>}] VolumeDevices:[] LivenessProbe:&Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:10251,Host:127.0.0.1,Scheme:HTTP,HTTPHeaders:[],},TCPSocket:nil,},InitialDelaySeconds:15,TimeoutSeconds:15,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:8,} ReadinessProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} is dead, but RestartPolicy says that we should restart it.
Apr 11 07:56:02 localhost kubelet: I0411 07:56:02.749599 2154 kuberuntime_manager.go:757] checking backoff for container "kube-scheduler" in pod "kube-scheduler-localhost.localdomain_kube-system(31cf0ccbee286239d451edb6fb511513)"
Apr 11 07:56:02 localhost kubelet: I0411 07:56:02.749776 2154 kuberuntime_manager.go:767] Back-off 1m20s restarting failed container=kube-scheduler pod=kube-scheduler-localhost.localdomain_kube-system(31cf0ccbee286239d451edb6fb511513)
Apr 11 07:56:02 localhost kubelet: E0411 07:56:02.749837 2154 pod_workers.go:186] Error syncing pod 31cf0ccbee286239d451edb6fb511513 ("kube-scheduler-localhost.localdomain_kube-system(31cf0ccbee286239d451edb6fb511513)"), skipping: failed to "StartContainer" for "kube-scheduler" with CrashLoopBackOff: "Back-off 1m20s restarting failed container=kube-scheduler pod=kube-scheduler-localhost.localdomain_kube-system(31cf0ccbee286239d451edb6fb511513)"
Apr 11 07:56:02 localhost kubelet: E0411 07:56:02.842509 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:02 localhost kubelet: E0411 07:56:02.842567 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:02 localhost kubelet: E0411 07:56:02.848053 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:02 localhost kubelet: E0411 07:56:02.850446 2154 eviction_manager.go:246] eviction manager: failed to get get summary stats: failed to get node info: node "localhost.localdomain" not found
Apr 11 07:56:02 localhost kubelet: W0411 07:56:02.955897 2154 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
Apr 11 07:56:02 localhost kubelet: E0411 07:56:02.956102 2154 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Apr 11 07:56:03 localhost kubelet: I0411 07:56:03.437277 2154 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
Apr 11 07:56:03 localhost kubelet: I0411 07:56:03.438582 2154 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
Apr 11 07:56:03 localhost kubelet: W0411 07:56:03.446002 2154 status_manager.go:461] Failed to get status for pod "kube-apiserver-localhost.localdomain_kube-system(96bb966b9c3212b8695ff3a763ac0e2c)": Get https://10.0.2.15:6443/api/v1/namespaces/kube-system/pods/kube-apiserver-localhost.localdomain: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:03 localhost kubelet: I0411 07:56:03.745662 2154 kuberuntime_manager.go:513] Container {Name:kube-scheduler Image:k8s.gcr.io/kube-scheduler-amd64:v1.10.0 Command:[kube-scheduler --address=127.0.0.1 --leader-elect=true --kubeconfig=/etc/kubernetes/scheduler.conf] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[] Requests:map[cpu:{i:{value:100 scale:-3} d:{Dec:<nil>} s:100m Format:DecimalSI}]} VolumeMounts:[{Name:kubeconfig ReadOnly:true MountPath:/etc/kubernetes/scheduler.conf SubPath: MountPropagation:<nil>}] VolumeDevices:[] LivenessProbe:&Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:10251,Host:127.0.0.1,Scheme:HTTP,HTTPHeaders:[],},TCPSocket:nil,},InitialDelaySeconds:15,TimeoutSeconds:15,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:8,} ReadinessProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} is dead, but RestartPolicy says that we should restart it.
Apr 11 07:56:03 localhost kubelet: I0411 07:56:03.745902 2154 kuberuntime_manager.go:757] checking backoff for container "kube-scheduler" in pod "kube-scheduler-localhost.localdomain_kube-system(31cf0ccbee286239d451edb6fb511513)"
Apr 11 07:56:03 localhost kubelet: I0411 07:56:03.746283 2154 kuberuntime_manager.go:767] Back-off 1m20s restarting failed container=kube-scheduler pod=kube-scheduler-localhost.localdomain_kube-system(31cf0ccbee286239d451edb6fb511513)
Apr 11 07:56:03 localhost kubelet: E0411 07:56:03.746354 2154 pod_workers.go:186] Error syncing pod 31cf0ccbee286239d451edb6fb511513 ("kube-scheduler-localhost.localdomain_kube-system(31cf0ccbee286239d451edb6fb511513)"), skipping: failed to "StartContainer" for "kube-scheduler" with CrashLoopBackOff: "Back-off 1m20s restarting failed container=kube-scheduler pod=kube-scheduler-localhost.localdomain_kube-system(31cf0ccbee286239d451edb6fb511513)"
Apr 11 07:56:03 localhost kubelet: I0411 07:56:03.746720 2154 kuberuntime_manager.go:513] Container {Name:kube-apiserver Image:k8s.gcr.io/kube-apiserver-amd64:v1.10.0 Command:[kube-apiserver --service-account-key-file=/etc/kubernetes/pki/sa.pub --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota --enable-bootstrap-token-auth=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-group-headers=X-Remote-Group --service-cluster-ip-range=10.96.0.0/12 --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --insecure-port=0 --requestheader-username-headers=X-Remote-User --requestheader-allowed-names=front-proxy-client --advertise-address=10.0.2.15 --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --allow-privileged=true --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/pki/apiserver.key --secure-port=6443 --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --client-ca-file=/etc/kubernetes/pki/ca.crt --authorization-mode=Node,RBAC --etcd-servers=https://127.0.0.1:2379 --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[] Requests:map[cpu:{i:{value:250 scale:-3} d:{Dec:<nil>} s:250m Format:DecimalSI}]} VolumeMounts:[{Name:k8s-certs ReadOnly:true MountPath:/etc/kubernetes/pki SubPath: MountPropagation:<nil>} {Name:ca-certs ReadOnly:true MountPath:/etc/ssl/certs SubPath: MountPropagation:<nil>} {Name:ca-certs-etc-pki ReadOnly:true MountPath:/etc/pki SubPath: MountPropagation:<nil>}] VolumeDevices:[] LivenessProbe:&Probe
Apr 11 07:56:03 localhost kubelet: {Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:6443,Host:10.0.2.15,Scheme:HTTPS,HTTPHeaders:[],},TCPSocket:nil,},InitialDelaySeconds:15,TimeoutSeconds:15,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:8,} ReadinessProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} is dead, but RestartPolicy says that we should restart it.
Apr 11 07:56:03 localhost kubelet: I0411 07:56:03.747020 2154 kuberuntime_manager.go:757] checking backoff for container "kube-apiserver" in pod "kube-apiserver-localhost.localdomain_kube-system(96bb966b9c3212b8695ff3a763ac0e2c)"
Apr 11 07:56:03 localhost kubelet: I0411 07:56:03.747227 2154 kuberuntime_manager.go:767] Back-off 1m20s restarting failed container=kube-apiserver pod=kube-apiserver-localhost.localdomain_kube-system(96bb966b9c3212b8695ff3a763ac0e2c)
Apr 11 07:56:03 localhost kubelet: E0411 07:56:03.747280 2154 pod_workers.go:186] Error syncing pod 96bb966b9c3212b8695ff3a763ac0e2c ("kube-apiserver-localhost.localdomain_kube-system(96bb966b9c3212b8695ff3a763ac0e2c)"), skipping: failed to "StartContainer" for "kube-apiserver" with CrashLoopBackOff: "Back-off 1m20s restarting failed container=kube-apiserver pod=kube-apiserver-localhost.localdomain_kube-system(96bb966b9c3212b8695ff3a763ac0e2c)"
Apr 11 07:56:03 localhost kubelet: E0411 07:56:03.843565 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:03 localhost kubelet: E0411 07:56:03.845764 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:03 localhost kubelet: E0411 07:56:03.848683 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:04 localhost kubelet: E0411 07:56:04.844987 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:04 localhost kubelet: E0411 07:56:04.847413 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:04 localhost kubelet: E0411 07:56:04.850004 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:05 localhost kubelet: I0411 07:56:05.410854 2154 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
Apr 11 07:56:05 localhost kubelet: I0411 07:56:05.414730 2154 kubelet_node_status.go:82] Attempting to register node localhost.localdomain
Apr 11 07:56:05 localhost kubelet: E0411 07:56:05.415227 2154 kubelet_node_status.go:106] Unable to register node "localhost.localdomain" with API server: Post https://10.0.2.15:6443/api/v1/nodes: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:05 localhost kubelet: E0411 07:56:05.846425 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:05 localhost kubelet: E0411 07:56:05.849635 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:05 localhost kubelet: E0411 07:56:05.851590 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:06 localhost kubelet: I0411 07:56:06.489612 2154 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
Apr 11 07:56:06 localhost kubelet: E0411 07:56:06.540459 2154 event.go:209] Unable to write event: 'Post https://10.0.2.15:6443/api/v1/namespaces/default/events: dial tcp 10.0.2.15:6443: getsockopt: connection refused' (may retry after sleeping)
Apr 11 07:56:06 localhost kubelet: I0411 07:56:06.795534 2154 kuberuntime_manager.go:513] Container {Name:kube-scheduler Image:k8s.gcr.io/kube-scheduler-amd64:v1.10.0 Command:[kube-scheduler --address=127.0.0.1 --leader-elect=true --kubeconfig=/etc/kubernetes/scheduler.conf] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[] Requests:map[cpu:{i:{value:100 scale:-3} d:{Dec:<nil>} s:100m Format:DecimalSI}]} VolumeMounts:[{Name:kubeconfig ReadOnly:true MountPath:/etc/kubernetes/scheduler.conf SubPath: MountPropagation:<nil>}] VolumeDevices:[] LivenessProbe:&Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:10251,Host:127.0.0.1,Scheme:HTTP,HTTPHeaders:[],},TCPSocket:nil,},InitialDelaySeconds:15,TimeoutSeconds:15,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:8,} ReadinessProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} is dead, but RestartPolicy says that we should restart it.
Apr 11 07:56:06 localhost kubelet: I0411 07:56:06.795729 2154 kuberuntime_manager.go:757] checking backoff for container "kube-scheduler" in pod "kube-scheduler-localhost.localdomain_kube-system(31cf0ccbee286239d451edb6fb511513)"
Apr 11 07:56:06 localhost kubelet: I0411 07:56:06.795913 2154 kuberuntime_manager.go:767] Back-off 1m20s restarting failed container=kube-scheduler pod=kube-scheduler-localhost.localdomain_kube-system(31cf0ccbee286239d451edb6fb511513)
Apr 11 07:56:06 localhost kubelet: E0411 07:56:06.795965 2154 pod_workers.go:186] Error syncing pod 31cf0ccbee286239d451edb6fb511513 ("kube-scheduler-localhost.localdomain_kube-system(31cf0ccbee286239d451edb6fb511513)"), skipping: failed to "StartContainer" for "kube-scheduler" with CrashLoopBackOff: "Back-off 1m20s restarting failed container=kube-scheduler pod=kube-scheduler-localhost.localdomain_kube-system(31cf0ccbee286239d451edb6fb511513)"
Apr 11 07:56:06 localhost kubelet: E0411 07:56:06.849610 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:06 localhost kubelet: E0411 07:56:06.851609 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:06 localhost kubelet: E0411 07:56:06.854017 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:07 localhost kubelet: E0411 07:56:07.850555 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:07 localhost kubelet: E0411 07:56:07.852656 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:07 localhost kubelet: E0411 07:56:07.854625 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:07 localhost kubelet: W0411 07:56:07.958155 2154 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
Apr 11 07:56:07 localhost kubelet: E0411 07:56:07.958398 2154 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Apr 11 07:56:08 localhost kubelet: E0411 07:56:08.335657 2154 certificate_manager.go:299] Failed while requesting a signed certificate from the master: cannot create certificate signing request: Post https://10.0.2.15:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:08 localhost kubelet: E0411 07:56:08.851894 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:08 localhost kubelet: E0411 07:56:08.853769 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:08 localhost kubelet: E0411 07:56:08.856241 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:09 localhost kubelet: E0411 07:56:09.852897 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:09 localhost kubelet: E0411 07:56:09.855072 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:09 localhost kubelet: E0411 07:56:09.857083 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:10 localhost kubelet: E0411 07:56:10.853865 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:10 localhost kubelet: E0411 07:56:10.855987 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:10 localhost kubelet: E0411 07:56:10.857855 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:11 localhost kubelet: E0411 07:56:11.854818 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:11 localhost kubelet: E0411 07:56:11.856735 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:11 localhost kubelet: E0411 07:56:11.858505 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:12 localhost kubelet: I0411 07:56:12.062221 2154 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
Apr 11 07:56:12 localhost kubelet: I0411 07:56:12.367930 2154 kuberuntime_manager.go:513] Container {Name:kube-apiserver Image:k8s.gcr.io/kube-apiserver-amd64:v1.10.0 Command:[kube-apiserver --service-account-key-file=/etc/kubernetes/pki/sa.pub --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota --enable-bootstrap-token-auth=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-group-headers=X-Remote-Group --service-cluster-ip-range=10.96.0.0/12 --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --insecure-port=0 --requestheader-username-headers=X-Remote-User --requestheader-allowed-names=front-proxy-client --advertise-address=10.0.2.15 --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --allow-privileged=true --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/pki/apiserver.key --secure-port=6443 --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --client-ca-file=/etc/kubernetes/pki/ca.crt --authorization-mode=Node,RBAC --etcd-servers=https://127.0.0.1:2379 --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[] Requests:map[cpu:{i:{value:250 scale:-3} d:{Dec:<nil>} s:250m Format:DecimalSI}]} VolumeMounts:[{Name:k8s-certs ReadOnly:true MountPath:/etc/kubernetes/pki SubPath: MountPropagation:<nil>} {Name:ca-certs ReadOnly:true MountPath:/etc/ssl/certs SubPath: MountPropagation:<nil>} {Name:ca-certs-etc-pki ReadOnly:true MountPath:/etc/pki SubPath: MountPropagation:<nil>}] VolumeDevices:[] LivenessProbe:&Probe
Apr 11 07:56:12 localhost kubelet: {Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:6443,Host:10.0.2.15,Scheme:HTTPS,HTTPHeaders:[],},TCPSocket:nil,},InitialDelaySeconds:15,TimeoutSeconds:15,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:8,} ReadinessProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} is dead, but RestartPolicy says that we should restart it.
Apr 11 07:56:12 localhost kubelet: I0411 07:56:12.368126 2154 kuberuntime_manager.go:757] checking backoff for container "kube-apiserver" in pod "kube-apiserver-localhost.localdomain_kube-system(96bb966b9c3212b8695ff3a763ac0e2c)"
Apr 11 07:56:12 localhost kubelet: I0411 07:56:12.368343 2154 kuberuntime_manager.go:767] Back-off 1m20s restarting failed container=kube-apiserver pod=kube-apiserver-localhost.localdomain_kube-system(96bb966b9c3212b8695ff3a763ac0e2c)
Apr 11 07:56:12 localhost kubelet: E0411 07:56:12.368396 2154 pod_workers.go:186] Error syncing pod 96bb966b9c3212b8695ff3a763ac0e2c ("kube-apiserver-localhost.localdomain_kube-system(96bb966b9c3212b8695ff3a763ac0e2c)"), skipping: failed to "StartContainer" for "kube-apiserver" with CrashLoopBackOff: "Back-off 1m20s restarting failed container=kube-apiserver pod=kube-apiserver-localhost.localdomain_kube-system(96bb966b9c3212b8695ff3a763ac0e2c)"
Apr 11 07:56:12 localhost kubelet: I0411 07:56:12.415697 2154 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
Apr 11 07:56:12 localhost kubelet: I0411 07:56:12.419306 2154 kubelet_node_status.go:82] Attempting to register node localhost.localdomain
Apr 11 07:56:12 localhost kubelet: E0411 07:56:12.419973 2154 kubelet_node_status.go:106] Unable to register node "localhost.localdomain" with API server: Post https://10.0.2.15:6443/api/v1/nodes: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:12 localhost kubelet: E0411 07:56:12.850802 2154 eviction_manager.go:246] eviction manager: failed to get get summary stats: failed to get node info: node "localhost.localdomain" not found
Apr 11 07:56:12 localhost kubelet: E0411 07:56:12.855462 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:12 localhost kubelet: E0411 07:56:12.857664 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:12 localhost kubelet: E0411 07:56:12.859273 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:12 localhost kubelet: W0411 07:56:12.960273 2154 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
Apr 11 07:56:12 localhost kubelet: E0411 07:56:12.960542 2154 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Apr 11 07:56:13 localhost kubelet: E0411 07:56:13.856529 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:13 localhost kubelet: E0411 07:56:13.858560 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:13 localhost kubelet: E0411 07:56:13.860337 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:14 localhost kubelet: E0411 07:56:14.857618 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:14 localhost kubelet: E0411 07:56:14.859666 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:14 localhost kubelet: E0411 07:56:14.861451 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:15 localhost kubelet: E0411 07:56:15.858689 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:15 localhost kubelet: E0411 07:56:15.860825 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:15 localhost kubelet: E0411 07:56:15.862536 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:16 localhost kubelet: E0411 07:56:16.541721 2154 event.go:209] Unable to write event: 'Post https://10.0.2.15:6443/api/v1/namespaces/default/events: dial tcp 10.0.2.15:6443: getsockopt: connection refused' (may retry after sleeping)
Apr 11 07:56:16 localhost kubelet: E0411 07:56:16.859815 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:16 localhost kubelet: E0411 07:56:16.861874 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:16 localhost kubelet: E0411 07:56:16.863646 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:17 localhost kubelet: E0411 07:56:17.860771 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:17 localhost kubelet: E0411 07:56:17.862948 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:17 localhost kubelet: E0411 07:56:17.864818 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:17 localhost kubelet: W0411 07:56:17.962610 2154 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
Apr 11 07:56:17 localhost kubelet: E0411 07:56:17.962889 2154 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Apr 11 07:56:18 localhost kubelet: E0411 07:56:18.862415 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:18 localhost kubelet: E0411 07:56:18.865620 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:18 localhost kubelet: E0411 07:56:18.867696 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:19 localhost kubelet: I0411 07:56:19.420564 2154 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
Apr 11 07:56:19 localhost kubelet: I0411 07:56:19.425231 2154 kubelet_node_status.go:82] Attempting to register node localhost.localdomain
Apr 11 07:56:19 localhost kubelet: E0411 07:56:19.425877 2154 kubelet_node_status.go:106] Unable to register node "localhost.localdomain" with API server: Post https://10.0.2.15:6443/api/v1/nodes: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:19 localhost kubelet: E0411 07:56:19.863292 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:19 localhost kubelet: E0411 07:56:19.866471 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:19 localhost kubelet: E0411 07:56:19.868765 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:20 localhost kubelet: E0411 07:56:20.864831 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:20 localhost kubelet: E0411 07:56:20.867278 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:20 localhost kubelet: E0411 07:56:20.869603 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:21 localhost kubelet: I0411 07:56:21.568884 2154 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
Apr 11 07:56:21 localhost kubelet: E0411 07:56:21.866334 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:21 localhost kubelet: E0411 07:56:21.868555 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:21 localhost kubelet: E0411 07:56:21.870455 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:21 localhost kubelet: I0411 07:56:21.873921 2154 kuberuntime_manager.go:513] Container {Name:kube-scheduler Image:k8s.gcr.io/kube-scheduler-amd64:v1.10.0 Command:[kube-scheduler --address=127.0.0.1 --leader-elect=true --kubeconfig=/etc/kubernetes/scheduler.conf] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[] Requests:map[cpu:{i:{value:100 scale:-3} d:{Dec:<nil>} s:100m Format:DecimalSI}]} VolumeMounts:[{Name:kubeconfig ReadOnly:true MountPath:/etc/kubernetes/scheduler.conf SubPath: MountPropagation:<nil>}] VolumeDevices:[] LivenessProbe:&Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:10251,Host:127.0.0.1,Scheme:HTTP,HTTPHeaders:[],},TCPSocket:nil,},InitialDelaySeconds:15,TimeoutSeconds:15,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:8,} ReadinessProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} is dead, but RestartPolicy says that we should restart it.
Apr 11 07:56:21 localhost kubelet: I0411 07:56:21.874078 2154 kuberuntime_manager.go:757] checking backoff for container "kube-scheduler" in pod "kube-scheduler-localhost.localdomain_kube-system(31cf0ccbee286239d451edb6fb511513)"
Apr 11 07:56:21 localhost kubelet: I0411 07:56:21.874256 2154 kuberuntime_manager.go:767] Back-off 1m20s restarting failed container=kube-scheduler pod=kube-scheduler-localhost.localdomain_kube-system(31cf0ccbee286239d451edb6fb511513)
Apr 11 07:56:21 localhost kubelet: E0411 07:56:21.874311 2154 pod_workers.go:186] Error syncing pod 31cf0ccbee286239d451edb6fb511513 ("kube-scheduler-localhost.localdomain_kube-system(31cf0ccbee286239d451edb6fb511513)"), skipping: failed to "StartContainer" for "kube-scheduler" with CrashLoopBackOff: "Back-off 1m20s restarting failed container=kube-scheduler pod=kube-scheduler-localhost.localdomain_kube-system(31cf0ccbee286239d451edb6fb511513)"
Apr 11 07:56:22 localhost kubelet: E0411 07:56:22.851229 2154 eviction_manager.go:246] eviction manager: failed to get get summary stats: failed to get node info: node "localhost.localdomain" not found
Apr 11 07:56:22 localhost kubelet: E0411 07:56:22.867534 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:22 localhost kubelet: E0411 07:56:22.869716 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:22 localhost kubelet: E0411 07:56:22.871978 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:22 localhost kubelet: W0411 07:56:22.965767 2154 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
Apr 11 07:56:22 localhost kubelet: E0411 07:56:22.966159 2154 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Apr 11 07:56:23 localhost kubelet: E0411 07:56:23.868720 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:23 localhost kubelet: E0411 07:56:23.870708 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:23 localhost kubelet: E0411 07:56:23.872672 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:24 localhost kubelet: E0411 07:56:24.869817 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:24 localhost kubelet: E0411 07:56:24.871767 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:24 localhost kubelet: E0411 07:56:24.873465 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:25 localhost kubelet: I0411 07:56:25.568829 2154 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
Apr 11 07:56:25 localhost kubelet: E0411 07:56:25.871365 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:25 localhost kubelet: E0411 07:56:25.873861 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:25 localhost kubelet: E0411 07:56:25.874383 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:25 localhost kubelet: I0411 07:56:25.876799 2154 kuberuntime_manager.go:513] Container {Name:kube-apiserver Image:k8s.gcr.io/kube-apiserver-amd64:v1.10.0 Command:[kube-apiserver --service-account-key-file=/etc/kubernetes/pki/sa.pub --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota --enable-bootstrap-token-auth=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-group-headers=X-Remote-Group --service-cluster-ip-range=10.96.0.0/12 --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --insecure-port=0 --requestheader-username-headers=X-Remote-User --requestheader-allowed-names=front-proxy-client --advertise-address=10.0.2.15 --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --allow-privileged=true --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/pki/apiserver.key --secure-port=6443 --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --client-ca-file=/etc/kubernetes/pki/ca.crt --authorization-mode=Node,RBAC --etcd-servers=https://127.0.0.1:2379 --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[] Requests:map[cpu:{i:{value:250 scale:-3} d:{Dec:<nil>} s:250m Format:DecimalSI}]} VolumeMounts:[{Name:k8s-certs ReadOnly:true MountPath:/etc/kubernetes/pki SubPath: MountPropagation:<nil>} {Name:ca-certs ReadOnly:true MountPath:/etc/ssl/certs SubPath: MountPropagation:<nil>} {Name:ca-certs-etc-pki ReadOnly:true MountPath:/etc/pki SubPath: MountPropagation:<nil>}] VolumeDevices:[] LivenessProbe:&Probe
Apr 11 07:56:25 localhost kubelet: {Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:6443,Host:10.0.2.15,Scheme:HTTPS,HTTPHeaders:[],},TCPSocket:nil,},InitialDelaySeconds:15,TimeoutSeconds:15,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:8,} ReadinessProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} is dead, but RestartPolicy says that we should restart it.
Apr 11 07:56:25 localhost kubelet: I0411 07:56:25.876986 2154 kuberuntime_manager.go:757] checking backoff for container "kube-apiserver" in pod "kube-apiserver-localhost.localdomain_kube-system(96bb966b9c3212b8695ff3a763ac0e2c)"
Apr 11 07:56:25 localhost kubelet: I0411 07:56:25.877220 2154 kuberuntime_manager.go:767] Back-off 1m20s restarting failed container=kube-apiserver pod=kube-apiserver-localhost.localdomain_kube-system(96bb966b9c3212b8695ff3a763ac0e2c)
Apr 11 07:56:25 localhost kubelet: E0411 07:56:25.877286 2154 pod_workers.go:186] Error syncing pod 96bb966b9c3212b8695ff3a763ac0e2c ("kube-apiserver-localhost.localdomain_kube-system(96bb966b9c3212b8695ff3a763ac0e2c)"), skipping: failed to "StartContainer" for "kube-apiserver" with CrashLoopBackOff: "Back-off 1m20s restarting failed container=kube-apiserver pod=kube-apiserver-localhost.localdomain_kube-system(96bb966b9c3212b8695ff3a763ac0e2c)"
Apr 11 07:56:26 localhost dockerd: time="2018-04-11T07:56:26.237450964Z" level=warning msg="containerd: unable to save c0e6a47a4542e655b9898f47edca7a2821fdffcb2166d1612d69b2e09c2bd747:f0029628e844877f3141efa603de703652d7afe36e5eb96d9749213af78c0336 starttime: open /proc/3751/stat: no such file or directory"
Apr 11 07:56:26 localhost dockerd: time="2018-04-11T07:56:26.2376524Z" level=info msg="containerd: c0e6a47a4542e655b9898f47edca7a2821fdffcb2166d1612d69b2e09c2bd747:f0029628e844877f3141efa603de703652d7afe36e5eb96d9749213af78c0336 (pid 3751) has become an orphan, killing it"
Apr 11 07:56:26 localhost kubelet: I0411 07:56:26.426499 2154 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
Apr 11 07:56:26 localhost kubelet: I0411 07:56:26.431015 2154 kubelet_node_status.go:82] Attempting to register node localhost.localdomain
Apr 11 07:56:26 localhost kubelet: E0411 07:56:26.432083 2154 kubelet_node_status.go:106] Unable to register node "localhost.localdomain" with API server: Post https://10.0.2.15:6443/api/v1/nodes: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:26 localhost kubelet: E0411 07:56:26.542913 2154 event.go:209] Unable to write event: 'Post https://10.0.2.15:6443/api/v1/namespaces/default/events: dial tcp 10.0.2.15:6443: getsockopt: connection refused' (may retry after sleeping)
Apr 11 07:56:26 localhost kubelet: E0411 07:56:26.872640 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:26 localhost kubelet: E0411 07:56:26.874873 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:26 localhost kubelet: E0411 07:56:26.877183 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:27 localhost kubelet: E0411 07:56:27.873986 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:27 localhost kubelet: E0411 07:56:27.875867 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:27 localhost kubelet: E0411 07:56:27.877709 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:27 localhost kubelet: W0411 07:56:27.968791 2154 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
Apr 11 07:56:27 localhost kubelet: E0411 07:56:27.969188 2154 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Apr 11 07:56:28 localhost kubelet: E0411 07:56:28.875505 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:28 localhost kubelet: E0411 07:56:28.879513 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:28 localhost kubelet: E0411 07:56:28.879684 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:29 localhost kubelet: E0411 07:56:29.877131 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:29 localhost kubelet: E0411 07:56:29.880193 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:29 localhost kubelet: E0411 07:56:29.883221 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:30 localhost kubelet: E0411 07:56:30.878545 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:30 localhost kubelet: E0411 07:56:30.880853 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:30 localhost kubelet: E0411 07:56:30.883894 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:31 localhost kubelet: I0411 07:56:31.568868 2154 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
Apr 11 07:56:31 localhost kubelet: E0411 07:56:31.879690 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:31 localhost kubelet: E0411 07:56:31.881975 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:31 localhost kubelet: E0411 07:56:31.884641 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:32 localhost kubelet: E0411 07:56:32.851511 2154 eviction_manager.go:246] eviction manager: failed to get get summary stats: failed to get node info: node "localhost.localdomain" not found
Apr 11 07:56:32 localhost kubelet: E0411 07:56:32.880445 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:32 localhost kubelet: E0411 07:56:32.882714 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:32 localhost kubelet: E0411 07:56:32.885467 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:32 localhost kubelet: W0411 07:56:32.971004 2154 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
Apr 11 07:56:32 localhost kubelet: E0411 07:56:32.971373 2154 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Apr 11 07:56:33 localhost kubelet: I0411 07:56:33.432507 2154 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
Apr 11 07:56:33 localhost kubelet: I0411 07:56:33.436207 2154 kubelet_node_status.go:82] Attempting to register node localhost.localdomain
Apr 11 07:56:33 localhost kubelet: E0411 07:56:33.436747 2154 kubelet_node_status.go:106] Unable to register node "localhost.localdomain" with API server: Post https://10.0.2.15:6443/api/v1/nodes: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:33 localhost kubelet: E0411 07:56:33.881305 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:33 localhost kubelet: E0411 07:56:33.883558 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:33 localhost kubelet: E0411 07:56:33.886256 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:34 localhost kubelet: E0411 07:56:34.882131 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:34 localhost kubelet: E0411 07:56:34.884182 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:34 localhost kubelet: E0411 07:56:34.886924 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:35 localhost kubelet: E0411 07:56:35.882975 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:35 localhost kubelet: E0411 07:56:35.885004 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:35 localhost kubelet: E0411 07:56:35.887668 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:36 localhost dockerd: time="2018-04-11T07:56:36.243447325Z" level=warning msg="containerd: unable to save c0e6a47a4542e655b9898f47edca7a2821fdffcb2166d1612d69b2e09c2bd747:8c8e2994da23e716e0831ad405cbc23f4375501d6d5d77463cef595af379d40a starttime: open /proc/3773/stat: no such file or directory"
Apr 11 07:56:36 localhost dockerd: time="2018-04-11T07:56:36.243674476Z" level=info msg="containerd: c0e6a47a4542e655b9898f47edca7a2821fdffcb2166d1612d69b2e09c2bd747:8c8e2994da23e716e0831ad405cbc23f4375501d6d5d77463cef595af379d40a (pid 3773) has become an orphan, killing it"
Apr 11 07:56:36 localhost kubelet: E0411 07:56:36.544414 2154 event.go:209] Unable to write event: 'Post https://10.0.2.15:6443/api/v1/namespaces/default/events: dial tcp 10.0.2.15:6443: getsockopt: connection refused' (may retry after sleeping)
Apr 11 07:56:36 localhost kubelet: I0411 07:56:36.568820 2154 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
Apr 11 07:56:36 localhost kubelet: I0411 07:56:36.874969 2154 kuberuntime_manager.go:513] Container {Name:kube-scheduler Image:k8s.gcr.io/kube-scheduler-amd64:v1.10.0 Command:[kube-scheduler --address=127.0.0.1 --leader-elect=true --kubeconfig=/etc/kubernetes/scheduler.conf] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[] Requests:map[cpu:{i:{value:100 scale:-3} d:{Dec:<nil>} s:100m Format:DecimalSI}]} VolumeMounts:[{Name:kubeconfig ReadOnly:true MountPath:/etc/kubernetes/scheduler.conf SubPath: MountPropagation:<nil>}] VolumeDevices:[] LivenessProbe:&Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:10251,Host:127.0.0.1,Scheme:HTTP,HTTPHeaders:[],},TCPSocket:nil,},InitialDelaySeconds:15,TimeoutSeconds:15,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:8,} ReadinessProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} is dead, but RestartPolicy says that we should restart it.
Apr 11 07:56:36 localhost kubelet: I0411 07:56:36.875190 2154 kuberuntime_manager.go:757] checking backoff for container "kube-scheduler" in pod "kube-scheduler-localhost.localdomain_kube-system(31cf0ccbee286239d451edb6fb511513)"
Apr 11 07:56:36 localhost kubelet: I0411 07:56:36.875392 2154 kuberuntime_manager.go:767] Back-off 1m20s restarting failed container=kube-scheduler pod=kube-scheduler-localhost.localdomain_kube-system(31cf0ccbee286239d451edb6fb511513)
Apr 11 07:56:36 localhost kubelet: E0411 07:56:36.875452 2154 pod_workers.go:186] Error syncing pod 31cf0ccbee286239d451edb6fb511513 ("kube-scheduler-localhost.localdomain_kube-system(31cf0ccbee286239d451edb6fb511513)"), skipping: failed to "StartContainer" for "kube-scheduler" with CrashLoopBackOff: "Back-off 1m20s restarting failed container=kube-scheduler pod=kube-scheduler-localhost.localdomain_kube-system(31cf0ccbee286239d451edb6fb511513)"
Apr 11 07:56:36 localhost kubelet: E0411 07:56:36.883713 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:36 localhost kubelet: E0411 07:56:36.885616 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:36 localhost kubelet: E0411 07:56:36.888297 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:37 localhost kubelet: E0411 07:56:37.884514 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:37 localhost kubelet: E0411 07:56:37.886507 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:37 localhost kubelet: E0411 07:56:37.888889 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:37 localhost kubelet: W0411 07:56:37.973190 2154 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
Apr 11 07:56:37 localhost kubelet: E0411 07:56:37.973426 2154 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Apr 11 07:56:38 localhost kubelet: I0411 07:56:38.568807 2154 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
Apr 11 07:56:38 localhost kubelet: I0411 07:56:38.874368 2154 kuberuntime_manager.go:513] Container {Name:kube-apiserver Image:k8s.gcr.io/kube-apiserver-amd64:v1.10.0 Command:[kube-apiserver --service-account-key-file=/etc/kubernetes/pki/sa.pub --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota --enable-bootstrap-token-auth=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-group-headers=X-Remote-Group --service-cluster-ip-range=10.96.0.0/12 --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --insecure-port=0 --requestheader-username-headers=X-Remote-User --requestheader-allowed-names=front-proxy-client --advertise-address=10.0.2.15 --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --allow-privileged=true --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/pki/apiserver.key --secure-port=6443 --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --client-ca-file=/etc/kubernetes/pki/ca.crt --authorization-mode=Node,RBAC --etcd-servers=https://127.0.0.1:2379 --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[] Requests:map[cpu:{i:{value:250 scale:-3} d:{Dec:<nil>} s:250m Format:DecimalSI}]} VolumeMounts:[{Name:k8s-certs ReadOnly:true MountPath:/etc/kubernetes/pki SubPath: MountPropagation:<nil>} {Name:ca-certs ReadOnly:true MountPath:/etc/ssl/certs SubPath: MountPropagation:<nil>} {Name:ca-certs-etc-pki ReadOnly:true MountPath:/etc/pki SubPath: MountPropagation:<nil>}] VolumeDevices:[] LivenessProbe:&Probe
Apr 11 07:56:38 localhost kubelet: {Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:6443,Host:10.0.2.15,Scheme:HTTPS,HTTPHeaders:[],},TCPSocket:nil,},InitialDelaySeconds:15,TimeoutSeconds:15,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:8,} ReadinessProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} is dead, but RestartPolicy says that we should restart it.
Apr 11 07:56:38 localhost kubelet: I0411 07:56:38.874643 2154 kuberuntime_manager.go:757] checking backoff for container "kube-apiserver" in pod "kube-apiserver-localhost.localdomain_kube-system(96bb966b9c3212b8695ff3a763ac0e2c)"
Apr 11 07:56:38 localhost kubelet: I0411 07:56:38.874911 2154 kuberuntime_manager.go:767] Back-off 1m20s restarting failed container=kube-apiserver pod=kube-apiserver-localhost.localdomain_kube-system(96bb966b9c3212b8695ff3a763ac0e2c)
Apr 11 07:56:38 localhost kubelet: E0411 07:56:38.874981 2154 pod_workers.go:186] Error syncing pod 96bb966b9c3212b8695ff3a763ac0e2c ("kube-apiserver-localhost.localdomain_kube-system(96bb966b9c3212b8695ff3a763ac0e2c)"), skipping: failed to "StartContainer" for "kube-apiserver" with CrashLoopBackOff: "Back-off 1m20s restarting failed container=kube-apiserver pod=kube-apiserver-localhost.localdomain_kube-system(96bb966b9c3212b8695ff3a763ac0e2c)"
Apr 11 07:56:38 localhost kubelet: E0411 07:56:38.892167 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:38 localhost kubelet: E0411 07:56:38.892337 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:38 localhost kubelet: E0411 07:56:38.892498 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:39 localhost kubelet: E0411 07:56:39.893201 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:39 localhost kubelet: E0411 07:56:39.896675 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:39 localhost kubelet: E0411 07:56:39.898289 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:40 localhost kubelet: E0411 07:56:40.335915 2154 certificate_manager.go:299] Failed while requesting a signed certificate from the master: cannot create certificate signing request: Post https://10.0.2.15:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:40 localhost kubelet: I0411 07:56:40.437592 2154 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
Apr 11 07:56:40 localhost kubelet: I0411 07:56:40.442417 2154 kubelet_node_status.go:82] Attempting to register node localhost.localdomain
Apr 11 07:56:40 localhost kubelet: E0411 07:56:40.443144 2154 kubelet_node_status.go:106] Unable to register node "localhost.localdomain" with API server: Post https://10.0.2.15:6443/api/v1/nodes: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:40 localhost kubelet: I0411 07:56:40.568903 2154 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
Apr 11 07:56:40 localhost kubelet: E0411 07:56:40.894182 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:40 localhost kubelet: E0411 07:56:40.897439 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:40 localhost kubelet: E0411 07:56:40.899111 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:41 localhost kubelet: E0411 07:56:41.895892 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:41 localhost kubelet: E0411 07:56:41.898437 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:41 localhost kubelet: E0411 07:56:41.900191 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:42 localhost kubelet: E0411 07:56:42.851866 2154 eviction_manager.go:246] eviction manager: failed to get get summary stats: failed to get node info: node "localhost.localdomain" not found
Apr 11 07:56:42 localhost kubelet: E0411 07:56:42.896997 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:42 localhost kubelet: E0411 07:56:42.899121 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:42 localhost kubelet: E0411 07:56:42.901288 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:42 localhost kubelet: W0411 07:56:42.975164 2154 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
Apr 11 07:56:42 localhost kubelet: E0411 07:56:42.975384 2154 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Apr 11 07:56:43 localhost kubelet: E0411 07:56:43.897830 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:43 localhost kubelet: E0411 07:56:43.900419 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:43 localhost kubelet: E0411 07:56:43.902366 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:44 localhost kubelet: E0411 07:56:44.899124 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:44 localhost kubelet: E0411 07:56:44.901410 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost.localdomain&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
Apr 11 07:56:44 localhost kubelet: E0411 07:56:44.903843 2154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
❤2
Is this the same as kubernetes/kubernetes#61277, is your etcd container crashing like @ChristianCandelaria describes?
geoffgarside
on 11 Apr 2018
What are the specs of this machine? Plenty of resources to spare? Does it meet the minimum requirements listed here https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#before-you-begin ?
I get the exact same situation when I use a vm with .5gB of ram and when i increase it to 2gB everything works fine.
chuckha
on 11 Apr 2018
👍1
@geoffgarside
Seems that ETCD doesn't even start based on the container list that I've published above
There is a "placeholser" (?) for ETCD
k8s.gcr.io/pause-amd:3.1 | k8s_POD_etcd
I've noticed that on your link there is a suggestions to start ETCD seperatly
I'm going to try that.
geostant
on 15 Apr 2018
@chuckha
Plenty of resources
22 Cores (Xeon)
512G RAM
15T NVMe disks
geostant
on 15 Apr 2018
When the kubeinit timeout (after a very long time - left it overnight)
I can see the following:
Unortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
- Either there is no internet connection, or imagePullPolicy is set to "Never",
so the kubelet cannot pull or find the following control plane images:
- k8s.gcr.io/kube-apiserver-amd64:v1.10.0
- k8s.gcr.io/kube-controller-manager-amd64:v1.10.0
- k8s.gcr.io/kube-scheduler-amd64:v1.10.0
- k8s.gcr.io/etcd-amd64:3.1.12 (only if no external etcd endpoints are configured)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
couldn't initialize a Kubernetes cluster
So systemctl status kubelet shows:
active (running)
And journalctl -xeu kubelet shows:
similar logs as I published before
geostant
on 15 Apr 2018
@geostant if you run docker ps -a it should show the exited containers as well, usually there will be a etcd container which has been terminated, though it looks like its been shutdown intentionally rather than crashing. I'm not sure if perhaps the livenessProbe is failing, thinking the service is dead, then restarting it. Running the etcd service externally presumably isn't using a livenessProbe. I've tried commenting out the livenessProbe but that still isn't working, so I suspect there to be some network issue somewhere that means etcd is running, but nothing can talk to the 127.0.0.1:2379 address.
geoffgarside
on 15 Apr 2018
@geoffgarside
Your latest response was fruitful !
I found etcd in exit status with docker ps -a (silly me that I didn't think about that before)
I've checked the logs of the container and found a very interesting line:
etcdmain: listen tcp 192.168.1.111:2380: bind: cannot assign requested address
This is not the IP address of my machine, it's actually 192.168.1.2
Any idea if I can force bind address to etcd container ?
Thanks
geostant
on 15 Apr 2018
I think that I got something... I'm trying it now
setting external yaml config file with:
```apiVersion: kubeadm.k8s.io/v1alpha1
kind: MasterConfiguration
kubernetesVersion: v1.10.0
etcd:
extraArgs:
'listen-peer-urls': 'http://127.0.0.1:2380'
geostant
on 15 Apr 2018
Ah, you've got the localhost lookup/binding issue. I found that in my searching, but its not fixed my little pi cluster :( glad you've got yours working :)
geoffgarside
on 15 Apr 2018
@geoffgarside
Not so fast - I'm waiting for several minutes to the init to finish
On ETCD logs I can see that it binds now to 127.0.0.1 and the port is now being served (ss -tuna | grep -i listen)
geostant
on 15 Apr 2018
/var/log/messages still show connection refuse on port 6443
geostant
on 15 Apr 2018
I've been trying this command to try and debug the liveness probe, I'm assuming the liveness probe is executed in the running container, and not using a separate container
docker exec $(docker ps --format "{{.ID}}" --filter="label=io.kubernetes.container.name=etcd") /bin/sh -ec "ETCDCTL_API=3 etcdctl --endpoints=127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt --key=/etc/kubernetes/pki/etcd/healthcheck-client.key get foo; echo \$?"
I'm probably just going around in circles and missing something very basic
geoffgarside
on 15 Apr 2018
apiserver container logs show:
container_linux.go:247: starting container process caused "exec: \"kube-apiserver\": executable file not found in $PATH"
geostant
on 15 Apr 2018
Thats worrying, might be worth deleting and re-pulling your api server container images down again.
geoffgarside
on 15 Apr 2018
I'm happy to inform that kubernetes cluster works great
After bypassing the localhost dns issue I had still the issue of
container_linux.go:247: starting container process caused "exec: \"kube-apiserver\": executable file not found in $PATH"
which then lead to go inside the container and actually search for the command kube-apiserver which yield zero results.
Then I've searched for kube* and found kube-scheduler.
Only then I understood that on the process to bring into the isolated network the tar images I've mistakenly swapped between 2 images.
Fixed that and now I've a working cluster that runs spark 2.3.0 flawless*
- on linux machines as with windows spark-submit fails (but that for another issue).
geostant
on 17 Apr 2018
@Geostant, how you were able bypass the local host.
I'm happy to inform that kubernetes cluster works great
After bypassing the localhost dns issue I had still the issue of
container_linux.go:247: starting container process caused "exec: \"kube-apiserver\": executable file not found in $PATH"
kasim457
on 1 Nov 2018
@Geostant, how you were able bypass the local host.
I'm happy to inform that kubernetes cluster works great
After bypassing the localhost dns issue I had still the issue of
container_linux.go:247: starting container process caused "exec: \"kube-apiserver\": executable file not found in $PATH"
kasim457
on 1 Nov 2018
Worker mode not able to launch weave-net and kuBe-proxy daemonsets on worker node. Nodes tries to pull image from internet ( gcr.io). How to bypass the request. I have all images locally available and also created internal registry service as cotrainer.
kasim457
on 1 Nov 2018
Was this page helpful?
0 / 5 - 0 ratings
Related issues
mattmoyer
路
4Comments
jbrandes
路
4Comments
jessfraz
路
3Comments
shenshouer
路
3Comments
kvaps
路
3Comments
Most helpful comment
I've done testing a similar isolated environment on an internet machine where the network connection was physically disconnected - tldr - it failed with the same results as I had before.
However this time I can provide a very detailed logs and process as it's a machine on the internet, so here it goes:
Command history
1 - Installed docker
2 - Loading docker images for offline use (Downloaded with docker pull the following images)
3 - Installed kubectl, kubeadm, kubelet
4 - Initiated the kubeadm init process
Logs from kubeadm
Logs from /var/log/messages