Rke: FATA[0073] Failed to bring up Worker Plane: [Failed to verify healthcheck: Failed to check http://localhost:10256/healthz for service [kube-proxy]

Created on 31 Mar 2019  路  9Comments  路  Source: rancher/rke

Tried to update the kubernetes version from "v1.12.3-rancher1-1" to "v1.13.5-rancher1-2" on my node running rancher v2.1.8 and got this error when running rke up.

RKE version:

v0.2.1 running on Ubuntu 18.04

rancher-cluster.yml file:

ssh_agent_auth: true
ignore_docker_version: true
kubernetes_version: "v1.13.5-rancher1-2"

nodes:
  - address: 1.1.1.1
    user: rke
    role: [controlplane,worker,etcd]
    ssh_key_path: ~/.ssh/rancher_server1

services:
  etcd:
    snapshot: true
    creation: 6h
    retention: 24h

Type/provider of hosts:

VPS

Docker version: (from the node)

Client:
 Version:           18.06.3-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        d7080c1
 Built:             Wed Feb 20 02:26:51 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.3-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       d7080c1
  Built:            Wed Feb 20 02:28:17 2019
  OS/Arch:          linux/amd64
  Experimental:     false

Operating system and kernel: (from the node)

uname -r : 3.10.0-862.14.4.el7.x86_64

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

Steps to Reproduce:

rke up --config rancher-cluster.yml

Results:

INFO[0000] Initiating Kubernetes cluster                
INFO[0000] [certificates] Generating admin certificates and kubeconfig 
INFO[0000] Successfully Deployed state file at [./rancher-cluster.rkestate] 
INFO[0000] Building Kubernetes cluster                  
INFO[0000] [dialer] Setup tunnel for host [1.1.1.1] 
INFO[0000] [network] No hosts added existing cluster, skipping port check 
INFO[0000] [certificates] Deploying kubernetes certificates to Cluster nodes 
INFO[0006] [reconcile] Rebuilding and updating local kube config 
INFO[0006] Successfully Deployed local admin kubeconfig at [./kube_config_rancher-cluster.yml] 
INFO[0006] [reconcile] host [1.1.1.1] is active master on the cluster 
INFO[0006] [certificates] Successfully deployed kubernetes certificates to Cluster nodes 
INFO[0006] [reconcile] Reconciling cluster state        
INFO[0006] [reconcile] Check etcd hosts to be deleted   
INFO[0006] [reconcile] Check etcd hosts to be added     
INFO[0006] [reconcile] Rebuilding and updating local kube config 
INFO[0006] Successfully Deployed local admin kubeconfig at [./kube_config_rancher-cluster.yml] 
INFO[0006] [reconcile] host [1.1.1.1] is active master on the cluster 
INFO[0006] [reconcile] Reconciled cluster state successfully 
INFO[0006] Pre-pulling kubernetes images                
INFO[0006] Kubernetes images pulled successfully        
INFO[0006] [etcd] Building up etcd plane..              
INFO[0006] [etcd] Saving snapshot [etcd-rolling-snapshots] on host [1.1.1.1] 
INFO[0006] [remove/etcd-rolling-snapshots] Successfully removed container on host [1.1.1.1] 
INFO[0007] [etcd] Successfully started [etcd-rolling-snapshots] container on host [1.1.1.1] 
INFO[0012] [certificates] Successfully started [rke-bundle-cert] container on host [1.1.1.1] 
INFO[0013] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [1.1.1.1] 
INFO[0013] [etcd] Successfully started [rke-log-linker] container on host [1.1.1.1] 
INFO[0014] [remove/rke-log-linker] Successfully removed container on host [1.1.1.1] 
INFO[0014] [etcd] Successfully started etcd plane.. Checking etcd cluster health 
INFO[0014] [controlplane] Building up Controller Plane.. 
INFO[0014] [sidekick] Sidekick container already created on host [1.1.1.1] 
INFO[0014] [healthcheck] Start Healthcheck on service [kube-apiserver] on host [1.1.1.1] 
INFO[0015] [healthcheck] service [kube-apiserver] on host [1.1.1.1] is healthy 
INFO[0015] [controlplane] Successfully started [rke-log-linker] container on host [1.1.1.1] 
INFO[0015] [remove/rke-log-linker] Successfully removed container on host [1.1.1.1] 
INFO[0016] [healthcheck] Start Healthcheck on service [kube-controller-manager] on host [1.1.1.1] 
INFO[0016] [healthcheck] service [kube-controller-manager] on host [1.1.1.1] is healthy 
INFO[0016] [controlplane] Successfully started [rke-log-linker] container on host [1.1.1.1] 
INFO[0017] [remove/rke-log-linker] Successfully removed container on host [1.1.1.1] 
INFO[0017] [healthcheck] Start Healthcheck on service [kube-scheduler] on host [1.1.1.1] 
INFO[0017] [healthcheck] service [kube-scheduler] on host [1.1.1.1] is healthy 
INFO[0018] [controlplane] Successfully started [rke-log-linker] container on host [1.1.1.1] 
INFO[0018] [remove/rke-log-linker] Successfully removed container on host [1.1.1.1] 
INFO[0018] [controlplane] Successfully started Controller Plane.. 
INFO[0018] [authz] Creating rke-job-deployer ServiceAccount 
INFO[0018] [authz] rke-job-deployer ServiceAccount created successfully 
INFO[0018] [authz] Creating system:node ClusterRoleBinding 
INFO[0018] [authz] system:node ClusterRoleBinding created successfully 
INFO[0018] Successfully Deployed state file at [./rancher-cluster.rkestate] 
INFO[0018] [state] Saving full cluster state to Kubernetes 
INFO[0019] [state] Successfully Saved full cluster state to Kubernetes ConfigMap: cluster-state 
INFO[0019] [worker] Building up Worker Plane..          
INFO[0019] [sidekick] Sidekick container already created on host [1.1.1.1] 
INFO[0019] [healthcheck] Start Healthcheck on service [kubelet] on host [1.1.1.1] 
INFO[0019] [healthcheck] service [kubelet] on host [1.1.1.1] is healthy 
INFO[0020] [worker] Successfully started [rke-log-linker] container on host [1.1.1.1] 
INFO[0020] [remove/rke-log-linker] Successfully removed container on host [1.1.1.1] 
INFO[0020] [healthcheck] Start Healthcheck on service [kube-proxy] on host [1.1.1.1] 
FATA[0073] [workerPlane] Failed to bring up Worker Plane: [Failed to verify healthcheck: Failed to check http://localhost:10256/healthz for service [kube-proxy] on host [1.1.1.1]: Get http://localhost:10256/healthz: Unable to access the service on localhost:10256. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), log: I0331 14:24:20.146218   32236 proxier.go:659] Stale udp service kube-system/kube-dns:dns -> 10.43.0.10]

running docker logs kube-proxy on the node:

+ grep -q cloud-provider=azure
+ echo kube-proxy --v=2 --healthz-bind-address=127.0.0.1 --cluster-cidr=10.42.0.0/16 --hostname-override=1.1.1.1 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml
+ '[' kube-proxy = kubelet ']'
+ exec kube-proxy --v=2 --healthz-bind-address=127.0.0.1 --cluster-cidr=10.42.0.0/16 --hostname-override=1.1.1.1 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml
I0331 13:41:38.013557   32236 flags.go:33] FLAG: --alsologtostderr="false"
I0331 13:41:38.013643   32236 flags.go:33] FLAG: --application-metrics-count-limit="100"
I0331 13:41:38.013652   32236 flags.go:33] FLAG: --azure-container-registry-config=""
I0331 13:41:38.013660   32236 flags.go:33] FLAG: --bind-address="0.0.0.0"
I0331 13:41:38.013671   32236 flags.go:33] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
I0331 13:41:38.013678   32236 flags.go:33] FLAG: --cleanup="false"
I0331 13:41:38.013686   32236 flags.go:33] FLAG: --cleanup-iptables="false"
I0331 13:41:38.013692   32236 flags.go:33] FLAG: --cleanup-ipvs="true"
I0331 13:41:38.013698   32236 flags.go:33] FLAG: --cloud-provider-gce-lb-src-cidrs="130.211.0.0/22,209.85.152.0/22,209.85.204.0/22,35.191.0.0/16"
I0331 13:41:38.013708   32236 flags.go:33] FLAG: --cluster-cidr="10.42.0.0/16"
I0331 13:41:38.013714   32236 flags.go:33] FLAG: --config=""
I0331 13:41:38.013720   32236 flags.go:33] FLAG: --config-sync-period="15m0s"
I0331 13:41:38.013727   32236 flags.go:33] FLAG: --conntrack-max="0"
I0331 13:41:38.013735   32236 flags.go:33] FLAG: --conntrack-max-per-core="32768"
I0331 13:41:38.013741   32236 flags.go:33] FLAG: --conntrack-min="131072"
I0331 13:41:38.013747   32236 flags.go:33] FLAG: --conntrack-tcp-timeout-close-wait="1h0m0s"
I0331 13:41:38.013754   32236 flags.go:33] FLAG: --conntrack-tcp-timeout-established="24h0m0s"
I0331 13:41:38.013760   32236 flags.go:33] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
I0331 13:41:38.013767   32236 flags.go:33] FLAG: --containerd="unix:///var/run/containerd.sock"
I0331 13:41:38.013773   32236 flags.go:33] FLAG: --default-not-ready-toleration-seconds="300"
I0331 13:41:38.013779   32236 flags.go:33] FLAG: --default-unreachable-toleration-seconds="300"
I0331 13:41:38.013785   32236 flags.go:33] FLAG: --docker="unix:///var/run/docker.sock"
I0331 13:41:38.013792   32236 flags.go:33] FLAG: --docker-env-metadata-whitelist=""
I0331 13:41:38.013797   32236 flags.go:33] FLAG: --docker-only="false"
I0331 13:41:38.013803   32236 flags.go:33] FLAG: --docker-root="/var/lib/docker"
I0331 13:41:38.013809   32236 flags.go:33] FLAG: --docker-tls="false"
I0331 13:41:38.013815   32236 flags.go:33] FLAG: --docker-tls-ca="ca.pem"
I0331 13:41:38.013821   32236 flags.go:33] FLAG: --docker-tls-cert="cert.pem"
I0331 13:41:38.013827   32236 flags.go:33] FLAG: --docker-tls-key="key.pem"
I0331 13:41:38.013833   32236 flags.go:33] FLAG: --enable-load-reader="false"
I0331 13:41:38.013839   32236 flags.go:33] FLAG: --event-storage-age-limit="default=0"
I0331 13:41:38.013845   32236 flags.go:33] FLAG: --event-storage-event-limit="default=0"
I0331 13:41:38.013851   32236 flags.go:33] FLAG: --feature-gates=""
I0331 13:41:38.013865   32236 flags.go:33] FLAG: --global-housekeeping-interval="1m0s"
I0331 13:41:38.013872   32236 flags.go:33] FLAG: --healthz-bind-address="127.0.0.1"
I0331 13:41:38.013878   32236 flags.go:33] FLAG: --healthz-port="10256"
I0331 13:41:38.013885   32236 flags.go:33] FLAG: --help="false"
I0331 13:41:38.013891   32236 flags.go:33] FLAG: --hostname-override="1.1.1.1"
I0331 13:41:38.013897   32236 flags.go:33] FLAG: --housekeeping-interval="10s"
I0331 13:41:38.013903   32236 flags.go:33] FLAG: --iptables-masquerade-bit="14"
I0331 13:41:38.013909   32236 flags.go:33] FLAG: --iptables-min-sync-period="0s"
I0331 13:41:38.013915   32236 flags.go:33] FLAG: --iptables-sync-period="30s"
I0331 13:41:38.013921   32236 flags.go:33] FLAG: --ipvs-exclude-cidrs="[]"
I0331 13:41:38.013938   32236 flags.go:33] FLAG: --ipvs-min-sync-period="0s"
I0331 13:41:38.013944   32236 flags.go:33] FLAG: --ipvs-scheduler=""
I0331 13:41:38.013950   32236 flags.go:33] FLAG: --ipvs-sync-period="30s"
I0331 13:41:38.013956   32236 flags.go:33] FLAG: --kube-api-burst="10"
I0331 13:41:38.013962   32236 flags.go:33] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
I0331 13:41:38.013968   32236 flags.go:33] FLAG: --kube-api-qps="5"
I0331 13:41:38.013976   32236 flags.go:33] FLAG: --kubeconfig="/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml"
I0331 13:41:38.013983   32236 flags.go:33] FLAG: --log-backtrace-at=":0"
I0331 13:41:38.013992   32236 flags.go:33] FLAG: --log-cadvisor-usage="false"
I0331 13:41:38.013998   32236 flags.go:33] FLAG: --log-dir=""
I0331 13:41:38.014004   32236 flags.go:33] FLAG: --log-file=""
I0331 13:41:38.014010   32236 flags.go:33] FLAG: --log-flush-frequency="5s"
I0331 13:41:38.014015   32236 flags.go:33] FLAG: --logtostderr="true"
I0331 13:41:38.014021   32236 flags.go:33] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
I0331 13:41:38.014028   32236 flags.go:33] FLAG: --masquerade-all="false"
I0331 13:41:38.014034   32236 flags.go:33] FLAG: --master=""
I0331 13:41:38.014040   32236 flags.go:33] FLAG: --mesos-agent="127.0.0.1:5051"
I0331 13:41:38.014046   32236 flags.go:33] FLAG: --mesos-agent-timeout="10s"
I0331 13:41:38.014052   32236 flags.go:33] FLAG: --metrics-bind-address="127.0.0.1:10249"
I0331 13:41:38.014058   32236 flags.go:33] FLAG: --metrics-port="10249"
I0331 13:41:38.014064   32236 flags.go:33] FLAG: --nodeport-addresses="[]"
I0331 13:41:38.014079   32236 flags.go:33] FLAG: --oom-score-adj="-999"
I0331 13:41:38.014086   32236 flags.go:33] FLAG: --profiling="false"
I0331 13:41:38.014092   32236 flags.go:33] FLAG: --proxy-mode=""
I0331 13:41:38.014100   32236 flags.go:33] FLAG: --proxy-port-range=""
I0331 13:41:38.014107   32236 flags.go:33] FLAG: --resource-container="/kube-proxy"
I0331 13:41:38.014113   32236 flags.go:33] FLAG: --skip-headers="false"
I0331 13:41:38.014119   32236 flags.go:33] FLAG: --stderrthreshold="2"
I0331 13:41:38.014125   32236 flags.go:33] FLAG: --storage-driver-buffer-duration="1m0s"
I0331 13:41:38.014131   32236 flags.go:33] FLAG: --storage-driver-db="cadvisor"
I0331 13:41:38.014137   32236 flags.go:33] FLAG: --storage-driver-host="localhost:8086"
I0331 13:41:38.014144   32236 flags.go:33] FLAG: --storage-driver-password="root"
I0331 13:41:38.014149   32236 flags.go:33] FLAG: --storage-driver-secure="false"
I0331 13:41:38.014155   32236 flags.go:33] FLAG: --storage-driver-table="stats"
I0331 13:41:38.014161   32236 flags.go:33] FLAG: --storage-driver-user="root"
I0331 13:41:38.014167   32236 flags.go:33] FLAG: --udp-timeout="250ms"
I0331 13:41:38.014173   32236 flags.go:33] FLAG: --v="2"
I0331 13:41:38.014179   32236 flags.go:33] FLAG: --version="false"
I0331 13:41:38.014194   32236 flags.go:33] FLAG: --vmodule=""
I0331 13:41:38.014201   32236 flags.go:33] FLAG: --write-config-to=""
W0331 13:41:38.014216   32236 server.go:198] WARNING: all flags other than --config, --write-config-to, and --cleanup are deprecated. Please begin using a config file ASAP.
I0331 13:41:38.014279   32236 feature_gate.go:206] feature gates: &{map[]}
W0331 13:41:38.018733   32236 proxier.go:480] Failed to read file /lib/modules/3.10.0-862.14.4.el7.x86_64/modules.builtin with error open /lib/modules/3.10.0-862.14.4.el7.x86_64/modules.builtin: no such file or directory. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0331 13:41:38.020031   32236 proxier.go:493] Failed to load kernel module ip_vs with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0331 13:41:38.021056   32236 proxier.go:493] Failed to load kernel module ip_vs_rr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0331 13:41:38.022087   32236 proxier.go:493] Failed to load kernel module ip_vs_wrr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0331 13:41:38.023080   32236 proxier.go:493] Failed to load kernel module ip_vs_sh with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0331 13:41:38.025974   32236 proxier.go:493] Failed to load kernel module nf_conntrack_ipv4 with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0331 13:41:38.042021   32236 server_others.go:295] Flag proxy-mode="" unknown, assuming iptables proxy
I0331 13:41:38.058282   32236 server_others.go:148] Using iptables Proxier.
I0331 13:41:38.058520   32236 server_others.go:178] Tearing down inactive rules.
I0331 13:41:38.099543   32236 iptables.go:200] Could not connect to D-Bus system bus: dial unix /var/run/dbus/system_bus_socket: connect: no such file or directory
I0331 13:41:38.099582   32236 server.go:483] Version: v1.13.5
I0331 13:41:38.105439   32236 server.go:509] Running in resource-only container "/kube-proxy"
I0331 13:41:38.106197   32236 conntrack.go:52] Setting nf_conntrack_max to 262144
I0331 13:41:38.108528   32236 config.go:202] Starting service config controller
I0331 13:41:38.108562   32236 controller_utils.go:1027] Waiting for caches to sync for service config controller
I0331 13:41:38.108774   32236 config.go:102] Starting endpoints config controller
I0331 13:41:38.108784   32236 controller_utils.go:1027] Waiting for caches to sync for endpoints config controller
I0331 13:41:38.209003   32236 controller_utils.go:1034] Caches are synced for service config controller
I0331 13:41:38.209042   32236 controller_utils.go:1034] Caches are synced for endpoints config controller
I0331 13:41:38.209177   32236 service.go:309] Adding new service port "cattle-system/rancher:http" at 10.43.30.198:80/TCP
I0331 13:41:38.209220   32236 service.go:309] Adding new service port "default/kubernetes:https" at 10.43.0.1:443/TCP
I0331 13:41:38.209241   32236 service.go:309] Adding new service port "ingress-nginx/default-http-backend:" at 10.43.128.22:80/TCP
I0331 13:41:38.209261   32236 service.go:309] Adding new service port "kube-system/kube-dns:dns" at 10.43.0.10:53/UDP
I0331 13:41:38.209276   32236 service.go:309] Adding new service port "kube-system/kube-dns:dns-tcp" at 10.43.0.10:53/TCP
I0331 13:41:38.209330   32236 service.go:309] Adding new service port "kube-system/metrics-server:" at 10.43.93.234:443/TCP
I0331 13:41:38.209356   32236 service.go:309] Adding new service port "kube-system/tiller-deploy:tiller" at 10.43.160.62:44134/TCP
I0331 13:41:40.947589   32236 proxier.go:659] Stale udp service kube-system/kube-dns:dns -> 10.43.0.10
W0331 14:23:56.834674   32236 reflector.go:256] k8s.io/client-go/informers/factory.go:132: watch of *v1.Service ended with: too old resource version: 27821492 (27828632)
I0331 14:24:20.146218   32236 proxier.go:659] Stale udp service kube-system/kube-dns:dns -> 10.43.0.10
Done kinbug
Source
picture vitorarantes
👍2

Most helpful comment

I checked /etc/hosts and find problem with 127.0.0.1 and localhost and it helped me

picture Negashev on 14 Jun 2019
👍2 👎1

All 9 comments

Same problem here, but with rke 0.1.18 and it's stable kubernetes (v1.13.4-rancher1-1)

[root@ran-wor4 ~]# docker logs kube-proxy
+ echo kube-proxy --v=2 --healthz-bind-address=0.0.0.0 --cluster-cidr=10.42.0.0/16 --hostname-override=192.168.88.107 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml
+ grep -q cloud-provider=azure
+ '[' kube-proxy = kubelet ']'
+ exec kube-proxy --v=2 --healthz-bind-address=0.0.0.0 --cluster-cidr=10.42.0.0/16 --hostname-override=192.168.88.107 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml
I0504 12:24:11.152256   26911 flags.go:33] FLAG: --alsologtostderr="false"
I0504 12:24:11.152334   26911 flags.go:33] FLAG: --application-metrics-count-limit="100"
I0504 12:24:11.152341   26911 flags.go:33] FLAG: --azure-container-registry-config=""
I0504 12:24:11.152346   26911 flags.go:33] FLAG: --bind-address="0.0.0.0"
I0504 12:24:11.152354   26911 flags.go:33] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
I0504 12:24:11.152362   26911 flags.go:33] FLAG: --cleanup="false"
I0504 12:24:11.152373   26911 flags.go:33] FLAG: --cleanup-iptables="false"
I0504 12:24:11.152380   26911 flags.go:33] FLAG: --cleanup-ipvs="true"
I0504 12:24:11.152387   26911 flags.go:33] FLAG: --cloud-provider-gce-lb-src-cidrs="130.211.0.0/22,209.85.152.0/22,209.85.204.0/22,35.191.0.0/16"
I0504 12:24:11.152395   26911 flags.go:33] FLAG: --cluster-cidr="10.42.0.0/16"
I0504 12:24:11.152400   26911 flags.go:33] FLAG: --config=""
I0504 12:24:11.152403   26911 flags.go:33] FLAG: --config-sync-period="15m0s"
I0504 12:24:11.152409   26911 flags.go:33] FLAG: --conntrack-max="0"
I0504 12:24:11.152414   26911 flags.go:33] FLAG: --conntrack-max-per-core="32768"
I0504 12:24:11.152419   26911 flags.go:33] FLAG: --conntrack-min="131072"
I0504 12:24:11.152423   26911 flags.go:33] FLAG: --conntrack-tcp-timeout-close-wait="1h0m0s"
I0504 12:24:11.152427   26911 flags.go:33] FLAG: --conntrack-tcp-timeout-established="24h0m0s"
I0504 12:24:11.152431   26911 flags.go:33] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
I0504 12:24:11.152435   26911 flags.go:33] FLAG: --containerd="unix:///var/run/containerd.sock"
I0504 12:24:11.152440   26911 flags.go:33] FLAG: --default-not-ready-toleration-seconds="300"
I0504 12:24:11.152444   26911 flags.go:33] FLAG: --default-unreachable-toleration-seconds="300"
I0504 12:24:11.152448   26911 flags.go:33] FLAG: --docker="unix:///var/run/docker.sock"
I0504 12:24:11.152452   26911 flags.go:33] FLAG: --docker-env-metadata-whitelist=""
I0504 12:24:11.152456   26911 flags.go:33] FLAG: --docker-only="false"
I0504 12:24:11.152460   26911 flags.go:33] FLAG: --docker-root="/var/lib/docker"
I0504 12:24:11.152464   26911 flags.go:33] FLAG: --docker-tls="false"
I0504 12:24:11.152468   26911 flags.go:33] FLAG: --docker-tls-ca="ca.pem"
I0504 12:24:11.152472   26911 flags.go:33] FLAG: --docker-tls-cert="cert.pem"
I0504 12:24:11.152476   26911 flags.go:33] FLAG: --docker-tls-key="key.pem"
I0504 12:24:11.152480   26911 flags.go:33] FLAG: --enable-load-reader="false"
I0504 12:24:11.152483   26911 flags.go:33] FLAG: --event-storage-age-limit="default=0"
I0504 12:24:11.152487   26911 flags.go:33] FLAG: --event-storage-event-limit="default=0"
I0504 12:24:11.152491   26911 flags.go:33] FLAG: --feature-gates=""
I0504 12:24:11.152497   26911 flags.go:33] FLAG: --global-housekeeping-interval="1m0s"
I0504 12:24:11.152501   26911 flags.go:33] FLAG: --healthz-bind-address="0.0.0.0"
I0504 12:24:11.152509   26911 flags.go:33] FLAG: --healthz-port="10256"
I0504 12:24:11.152516   26911 flags.go:33] FLAG: --help="false"
I0504 12:24:11.152523   26911 flags.go:33] FLAG: --hostname-override="192.168.88.107"
I0504 12:24:11.152530   26911 flags.go:33] FLAG: --housekeeping-interval="10s"
I0504 12:24:11.152538   26911 flags.go:33] FLAG: --iptables-masquerade-bit="14"
I0504 12:24:11.152542   26911 flags.go:33] FLAG: --iptables-min-sync-period="0s"
I0504 12:24:11.152546   26911 flags.go:33] FLAG: --iptables-sync-period="30s"
I0504 12:24:11.152550   26911 flags.go:33] FLAG: --ipvs-exclude-cidrs="[]"
I0504 12:24:11.152557   26911 flags.go:33] FLAG: --ipvs-min-sync-period="0s"
I0504 12:24:11.152561   26911 flags.go:33] FLAG: --ipvs-scheduler=""
I0504 12:24:11.152565   26911 flags.go:33] FLAG: --ipvs-sync-period="30s"
I0504 12:24:11.152569   26911 flags.go:33] FLAG: --kube-api-burst="10"
I0504 12:24:11.152573   26911 flags.go:33] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
I0504 12:24:11.152577   26911 flags.go:33] FLAG: --kube-api-qps="5"
I0504 12:24:11.152583   26911 flags.go:33] FLAG: --kubeconfig="/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml"
I0504 12:24:11.152589   26911 flags.go:33] FLAG: --log-backtrace-at=":0"
I0504 12:24:11.152600   26911 flags.go:33] FLAG: --log-cadvisor-usage="false"
I0504 12:24:11.152608   26911 flags.go:33] FLAG: --log-dir=""
I0504 12:24:11.152615   26911 flags.go:33] FLAG: --log-file=""
I0504 12:24:11.152619   26911 flags.go:33] FLAG: --log-flush-frequency="5s"
I0504 12:24:11.152624   26911 flags.go:33] FLAG: --logtostderr="true"
I0504 12:24:11.152629   26911 flags.go:33] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
I0504 12:24:11.152634   26911 flags.go:33] FLAG: --masquerade-all="false"
I0504 12:24:11.152638   26911 flags.go:33] FLAG: --master=""
I0504 12:24:11.152642   26911 flags.go:33] FLAG: --mesos-agent="127.0.0.1:5051"
I0504 12:24:11.152646   26911 flags.go:33] FLAG: --mesos-agent-timeout="10s"
I0504 12:24:11.152650   26911 flags.go:33] FLAG: --metrics-bind-address="127.0.0.1:10249"
I0504 12:24:11.152654   26911 flags.go:33] FLAG: --metrics-port="10249"
I0504 12:24:11.152658   26911 flags.go:33] FLAG: --nodeport-addresses="[]"
I0504 12:24:11.152670   26911 flags.go:33] FLAG: --oom-score-adj="-999"
I0504 12:24:11.152674   26911 flags.go:33] FLAG: --profiling="false"
I0504 12:24:11.152678   26911 flags.go:33] FLAG: --proxy-mode=""
I0504 12:24:11.152684   26911 flags.go:33] FLAG: --proxy-port-range=""
I0504 12:24:11.152689   26911 flags.go:33] FLAG: --resource-container="/kube-proxy"
I0504 12:24:11.152693   26911 flags.go:33] FLAG: --skip-headers="false"
I0504 12:24:11.152697   26911 flags.go:33] FLAG: --stderrthreshold="2"
I0504 12:24:11.152700   26911 flags.go:33] FLAG: --storage-driver-buffer-duration="1m0s"
I0504 12:24:11.152704   26911 flags.go:33] FLAG: --storage-driver-db="cadvisor"
I0504 12:24:11.152708   26911 flags.go:33] FLAG: --storage-driver-host="localhost:8086"
I0504 12:24:11.152712   26911 flags.go:33] FLAG: --storage-driver-password="root"
I0504 12:24:11.152716   26911 flags.go:33] FLAG: --storage-driver-secure="false"
I0504 12:24:11.152720   26911 flags.go:33] FLAG: --storage-driver-table="stats"
I0504 12:24:11.152724   26911 flags.go:33] FLAG: --storage-driver-user="root"
I0504 12:24:11.152728   26911 flags.go:33] FLAG: --udp-timeout="250ms"
I0504 12:24:11.152732   26911 flags.go:33] FLAG: --v="2"
I0504 12:24:11.152736   26911 flags.go:33] FLAG: --version="false"
I0504 12:24:11.152744   26911 flags.go:33] FLAG: --vmodule=""
I0504 12:24:11.152748   26911 flags.go:33] FLAG: --write-config-to=""
W0504 12:24:11.152764   26911 server.go:198] WARNING: all flags other than --config, --write-config-to, and --cleanup are deprecated. Please begin using a config file ASAP.
I0504 12:24:11.152795   26911 feature_gate.go:206] feature gates: &{map[]}
W0504 12:24:12.715830   26911 proxier.go:480] Failed to read file /lib/modules/3.10.0-957.12.1.el7.x86_64/modules.builtin with error open /lib/modules/3.10.0-957.12.1.el7.x86_64/modules.builtin: no such file or directory. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0504 12:24:15.637427   26911 proxier.go:493] Failed to load kernel module ip_vs with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0504 12:24:15.638502   26911 proxier.go:493] Failed to load kernel module ip_vs_rr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0504 12:24:15.639182   26911 proxier.go:493] Failed to load kernel module ip_vs_wrr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0504 12:24:15.639818   26911 proxier.go:493] Failed to load kernel module ip_vs_sh with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0504 12:24:15.640459   26911 proxier.go:493] Failed to load kernel module nf_conntrack_ipv4 with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0504 12:24:27.832280   26911 server_others.go:295] Flag proxy-mode="" unknown, assuming iptables proxy
I0504 12:24:52.478478   26911 server_others.go:148] Using iptables Proxier.
I0504 12:24:52.479341   26911 server_others.go:178] Tearing down inactive rules.
I0504 12:24:52.970600   26911 iptables.go:200] Could not connect to D-Bus system bus: dial unix /var/run/dbus/system_bus_socket: connect: no such file or directory
I0504 12:24:52.970924   26911 server.go:483] Version: v1.13.5
I0504 12:24:52.978367   26911 server.go:509] Running in resource-only container "/kube-proxy"
I0504 12:24:52.979098   26911 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_max' to 131072
I0504 12:24:52.979130   26911 conntrack.go:52] Setting nf_conntrack_max to 131072
I0504 12:24:52.979376   26911 mount_linux.go:165] Detected OS without systemd
I0504 12:24:52.979537   26911 conntrack.go:83] Setting conntrack hashsize to 32768
I0504 12:24:52.981966   26911 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
I0504 12:24:52.982028   26911 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
I0504 12:24:52.982675   26911 config.go:202] Starting service config controller
I0504 12:24:52.982701   26911 controller_utils.go:1027] Waiting for caches to sync for service config controller
I0504 12:24:53.580575   26911 config.go:102] Starting endpoints config controller
I0504 12:24:53.580612   26911 controller_utils.go:1027] Waiting for caches to sync for endpoints config controller
I0504 12:24:53.584015   26911 controller_utils.go:1034] Caches are synced for service config controller
I0504 12:24:53.584137   26911 proxier.go:645] Not syncing iptables until Services and Endpoints have been received from master
I0504 12:24:53.680773   26911 controller_utils.go:1034] Caches are synced for endpoints config controller
I0504 12:24:53.680915   26911 service.go:309] Adding new service port "default/kubernetes:https" at 10.43.0.1:443/TCP
tuxpeople picture tuxpeople on 4 May 2019

I found that my problem was related to the hoster, the node described on
my issue was running on a VPS from Leaseweb and since I updated to the v1.13.5-rancher1-2 it's throwing this error, I also did try a fresh install, disabled the firewall, but always stuck on this error...

I tested the same steps on a bare metal server from another hoster and everything worked fine.

vitorarantes picture vitorarantes on 8 May 2019

I'm now stuck in another problem (#1333), I think this here has been solved by upgrading the installed packages on my CentOS, but I'm not sure.

tuxpeople picture tuxpeople on 8 May 2019

we have same problem with following error:

FATA[0164] [workerPlane] Failed to bring up Worker Plane: [Failed to verify healthcheck: Failed to check http://localhost:10256/healthz for service [kube-proxy] on host [192.168.0.10]: Get http://localhost:10256/healthz: Unable to access the service on localhost:10256. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), log: I0606 18:50:16.872465 9625 service.go:309] Adding new service port "default/kubernetes:https" at 10.43.0.1:443/TCP]

The thing is it was working fine. I'm not sure, but after we made some changes on OS and hardware we got this issue.

amirbagh75 picture amirbagh75 on 6 Jun 2019
👍1

+1

Negashev picture Negashev on 13 Jun 2019

I checked /etc/hosts and find problem with 127.0.0.1 and localhost and it helped me

Negashev picture Negashev on 14 Jun 2019
👍2 👎1

I checked /etc/hosts and find problem with 127.0.0.1 and localhost and it helped me

Sorry for the late reply, my problem was solved also by adding 127.0.0.1 to /etc/hosts

vitorarantes picture vitorarantes on 3 Jul 2019
👎1

The same problems. When I upgrade hardware and change OS then get this issues.
*Docker version: *

Client: Docker Engine - Community
Version: 19.03.11
API version: 1.40
Go version: go1.13.10
Git commit: 42e35e61f3
Built: Mon Jun 1 09:13:48 2020
OS/Arch: linux/amd64
Experimental: false

Server: Docker Engine - Community
Engine:
Version: 19.03.11
API version: 1.40 (minimum version 1.12)
Go version: go1.13.10
Git commit: 42e35e61f3
Built: Mon Jun 1 09:12:26 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.2.13
GitCommit: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc:
Version: 1.0.0-rc10
GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
docker-init:
Version: 0.18.0
GitCommit: fec3683

rke version

WARN[0000] This is not an officially supported version (v1.2.0-rc2) of RKE. Please download the latest official release at https://github.com/rancher/rke/releases/latest
INFO[0000] Running RKE version: v1.2.0-rc2
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.6", GitCommit:"d32e40e20d167e103faf894261614c5b45c44198", GitTreeState:"clean", BuildDate:"2020-05-20T13:08:34Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

problem _rke up_

FATA[0106] [workerPlane] Failed to bring up Worker Plane: [Failed to verify healthcheck: Failed to check http://localhost:10256/healthz for service [kube-proxy] on host [10.0.1.165]: Get http://localhost:10256/healthz: Unable to access the service on localhost:10256. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), log: E0616 06:37:48.711514 6456 node.go:124] Failed to retrieve node info: an error on the server ("") has prevented the request from succeeding (get nodes 10.0.1.165)]

log kube-proxy

W0616 06:37:19.657246 6456 server.go:213] WARNING: all flags other than --config, --write-config-to, and --cleanup are deprecated. Please begin using a config file ASAP.
I0616 06:37:19.657325 6456 feature_gate.go:243] feature gates: &{map[]}
W0616 06:37:19.670786 6456 proxier.go:608] Failed to load kernel module ip_vs with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0616 06:37:19.671839 6456 proxier.go:608] Failed to load kernel module ip_vs_rr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0616 06:37:19.672847 6456 proxier.go:608] Failed to load kernel module ip_vs_wrr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0616 06:37:19.673786 6456 proxier.go:608] Failed to load kernel module ip_vs_sh with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0616 06:37:19.680426 6456 server_others.go:324] Unknown proxy mode "", assuming iptables proxy
E0616 06:37:48.711514 6456 node.go:124] Failed to retrieve node info: an error on the server ("") has prevented the request from succeeding (get nodes 10.0.1.165)
E0616 06:38:18.929199 6456 node.go:124] Failed to retrieve node info: an error on the server ("") has prevented the request from succeeding (get nodes 10.0.1.165)
E0616 06:38:50.013884 6456 node.go:124] Failed to retrieve node info: an error on the server ("") has prevented the request from succeeding (get nodes 10.0.1.165)
E0616 06:39:23.207300 6456 node.go:124] Failed to retrieve node info: an error on the server ("") has prevented the request from succeeding (get nodes 10.0.1.165)
E0616 06:40:01.113167 6456 node.go:124] Failed to retrieve node info: an error on the server ("") has prevented the request from succeeding (get nodes 10.0.1.165)
E0616 06:40:46.794879 6456 node.go:124] Failed to retrieve node info: an error on the server ("") has prevented the request from succeeding (get nodes 10.0.1.165)
I0616 06:40:46.795090 6456 server_others.go:140] can't determine this node's IP, assuming 127.0.0.1; if this is incorrect, please set the --bind-address flag
I0616 06:40:46.795175 6456 server_others.go:145] Using iptables Proxier.
I0616 06:40:46.798885 6456 server.go:571] Version: v1.17.6
I0616 06:40:46.802072 6456 conntrack.go:52] Setting nf_conntrack_max to 131072
I0616 06:40:46.807847 6456 config.go:313] Starting service config controller
I0616 06:40:46.807899 6456 shared_informer.go:197] Waiting for caches to sync for service config
I0616 06:40:46.807951 6456 config.go:131] Starting endpoints config controller
I0616 06:40:46.808082 6456 shared_informer.go:197] Waiting for caches to sync for endpoints config

khiemnd5 picture khiemnd5 on 16 Jun 2020

===
Update:
I must change OS centos 7.6 => ubuntu 18.04 and reinstall then ok.

khiemnd5 picture khiemnd5 on 23 Jun 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

bootc picture bootc  路  18Comments
Skipper0707 picture Skipper0707  路  16Comments
gknepper picture gknepper  路  19Comments
freeloop914 picture freeloop914  路  16Comments
iljaweis picture iljaweis  路  20Comments