Origin: Openshift Origin: failed to run Kubelet: failed to create kubelet:

Created on 14 Nov 2017 · 12Comments · Source: openshift/origin

[provide a description of the issue]

Per instruction, https://docs.openshift.org/latest/getting_started/administrators.html#downloading-the-binary, I downloaded and untared, openshift-origin-server-v3.6.1-008f2d5-linux-64bit.tar.gz and installed at CentOS 7.

I got errors:

./openshift start

......
W1114 18:12:05.650854 9288 kubelet_network.go:70] Hairpin mode set to "promiscuous-bridge" but kubenet is not enabled, falling back to "hairpin-veth"
I1114 18:12:05.650908 9288 kubelet.go:494] Hairpin mode set to "hairpin-veth"
W1114 18:12:05.664043 9288 cni.go:157] Unable to update cni config: No networks found in /etc/cni/net.d
I1114 18:12:05.769751 9288 docker_service.go:184] Docker cri networking managed by kubernetes.io/no-op
F1114 18:12:05.794043 9288 node.go:281] failed to run Kubelet: failed to create kubelet: misconfiguration: kubelet cgroup driver: "systemd" is different from docker cgroup driver: "cgroupfs"

Version

[provide output of the openshift version or oc version command]

./openshift version
openshift v3.6.1+008f2d5
kubernetes v1.6.1+5115d708d7
etcd 3.2.1

docker version
Client:
Version: 17.11.0-ce-rc3
API version: 1.34
Go version: go1.8.3
Git commit: 5b4af4f
Built: Wed Nov 8 03:04:32 2017
OS/Arch: linux/amd64

Server:
Version: 17.11.0-ce-rc3
API version: 1.34 (minimum version 1.12)
Go version: go1.8.3
Git commit: 5b4af4f
Built: Wed Nov 8 03:07:05 2017
OS/Arch: linux/amd64
Experimental: false

oc version
oc v3.6.1+008f2d5
kubernetes v1.6.1+5115d708d7
features: Basic-Auth GSSAPI Kerberos SPNEGO

Steps To Reproduce

[step 1]
Download openshift origin at https://github.com/openshift/origin/releases/download/v3.6.1/openshift-origin-server-v3.6.1-008f2d5-linux-64bit.tar.gz
[step 2]

Untar the tar ball. Run sudo ./openshift start

Current Result

W1114 18:12:05.650854 9288 kubelet_network.go:70] Hairpin mode set to "promiscuous-bridge" but kubenet is not enabled, falling back to "hairpin-veth"
I1114 18:12:05.650908 9288 kubelet.go:494] Hairpin mode set to "hairpin-veth"
W1114 18:12:05.664043 9288 cni.go:157] Unable to update cni config: No networks found in /etc/cni/net.d
I1114 18:12:05.769751 9288 docker_service.go:184] Docker cri networking managed by kubernetes.io/no-op
F1114 18:12:05.794043 9288 node.go:281] failed to run Kubelet: failed to create kubelet: misconfiguration: kubelet cgroup driver: "systemd" is different from docker cgroup driver: "cgroupfs"

Expected Result

Error

Additional Information

[try to run $ oc adm diagnostics (or oadm diagnostics) command if possible]
[if you are reporting issue related to builds, provide build logs with BUILD_LOGLEVEL=5]
[consider attaching output of the $ oc get all -o json -n <namespace> command to the issue]
[visit https://docs.openshift.org/latest/welcome/index.html]

componeninstall kinquestion prioritP2

Source

garyyang6

Most helpful comment

This worked for me,

cat << EOF > /etc/docker/daemon.json
→ {
→  "exec-opts": ["native.cgroupdriver=systemd"]
→ }
→ EOF

restarted docker and ran
sudo ./openshift start

Chandru-B on 17 Mar 2018

👍13

All 12 comments

Please see https://github.com/openshift/origin/issues/16974

pweil- on 14 Nov 2017

Solution #16974 does not work.

I changed it to match Docker's.

vi /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

update KUBELET_CGROUP_ARGS=--cgroup-driver=systemd to KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs

grep KUBELET_CGROUP_ARGS 10-kubeadm.conf
Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs"

ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_CGROUP_ARGS $KUBELET_CERTIFICATE_ARGS $KUBELET_EXTRA_ARGS

docker info |grep -i cgroup
Cgroup Driver: cgroupfs

systemctl daemon-reload
service kubelet restart

./openshift start
W1114 20:26:47.521319 17156 start_master.go:297] Warning: assetConfig.loggingPublicURL: Invalid value: "": required to view aggregated container logs in the console, master start will continue.
W1114 20:26:47.521412 17156 start_master.go:297] Warning: assetConfig.metricsPublicURL: Invalid value: "": required to view cluster metrics in the console, master start will continue.
W1114 20:26:47.521425 17156 start_master.go:297] Warning: auditConfig.auditFilePath: Required value: audit can not be logged to a separate file, master start will continue.
I1114 20:26:47.544184 17156 plugins.go:101] No cloud provider specified.
2017-11-14 20:26:47.595757 I | etcdserver/api/v3rpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.104.6.127:4001: getsockopt: connection refused"; Reconnecting to {10.104.6.127:4001 }
E1114 20:26:47.595859 17156 controllermanager.go:337] Server isn't healthy yet. Waiting a little while.
2017-11-14 20:26:47.595907 I | etcdserver/api/v3rpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.104.6.127:4001: getsockopt: connection refused"; Reconnecting to {10.104.6.127:4001 }
2017-11-14 20:26:47.595957 I | etcdserver/api/v3rpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.104.6.127:4001: getsockopt: connection refused"; Reconnecting to {10.104.6.127:4001 }
2017-11-14 20:26:47.596004 I | etcdserver/api/v3rpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.104.6.127:4001: getsockopt: connection refused"; Reconnecting to {10.104.6.127:4001 }
2017-11-14 20:26:47.617441 I | etcdserver/api/v3rpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.104.6.127:4001: getsockopt: connection refused"; Reconnecting to {10.104.6.127:4001 }
I1114 20:26:47.861440 17156 start_master.go:529] Starting master on 0.0.0.0:8443 (v3.6.1+008f2d5)
I1114 20:26:47.861464 17156 start_master.go:530] Public master address is https://10.104.6.127:8443
I1114 20:26:47.861484 17156 start_master.go:534] Using images from "openshift/origin-:v3.6.1"
2017-11-14 20:26:47.861582 I | embed: peerTLS: cert = openshift.local.config/master/etcd.server.crt, key = openshift.local.config/master/etcd.server.key, ca = openshift.local.config/master/ca.crt, trusted-ca = , client-cert-auth = true
2017-11-14 20:26:47.862490 I | embed: listening for peers on https://0.0.0.0:7001
2017-11-14 20:26:47.862548 I | embed: listening for client requests on 0.0.0.0:4001
2017-11-14 20:26:47.865511 I | etcdserver: name = openshift.local
2017-11-14 20:26:47.865525 I | etcdserver: data dir = openshift.local.etcd
2017-11-14 20:26:47.865533 I | etcdserver: member dir = openshift.local.etcd/member
2017-11-14 20:26:47.865540 I | etcdserver: heartbeat = 100ms
2017-11-14 20:26:47.865546 I | etcdserver: election = 1000ms
2017-11-14 20:26:47.865552 I | etcdserver: snapshot count = 100000
2017-11-14 20:26:47.865567 I | etcdserver: advertise client URLs = https://10.104.6.127:4001
2017-11-14 20:26:47.895923 I | etcdserver: restarting member a7340362e2996c30 in cluster cf86d7c1b2833ba9 at commit index 1187
2017-11-14 20:26:47.896041 I | raft: a7340362e2996c30 became follower at term 21
2017-11-14 20:26:47.896063 I | raft: newRaft a7340362e2996c30 [peers: [], term: 21, commit: 1187, applied: 0, lastindex: 1187, lastterm: 21]
2017-11-14 20:26:47.919587 W | auth: simple token is not cryptographically signed
2017-11-14 20:26:47.922793 I | etcdserver: starting server... [version: 3.2.1, cluster version: to_be_decided]
2017-11-14 20:26:47.922828 I | embed: ClientTLS: cert = openshift.local.config/master/etcd.server.crt, key = openshift.local.config/master/etcd.server.key, ca = openshift.local.config/master/ca.crt, trusted-ca = , client-cert-auth = true
2017-11-14 20:26:47.925319 I | etcdserver/api/v3rpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.104.6.127:4001: getsockopt: connection refused"; Reconnecting to {10.104.6.127:4001 }
2017-11-14 20:26:47.925562 I | etcdserver/membership: added member a7340362e2996c30 [https://10.104.6.127:7001] to cluster cf86d7c1b2833ba9
2017-11-14 20:26:47.925674 N | etcdserver/membership: set the initial cluster version to 3.2
2017-11-14 20:26:47.925719 I | etcdserver/api: enabled capabilities for version 3.2
2017-11-14 20:26:48.096468 I | raft: a7340362e2996c30 is starting a new election at term 21
2017-11-14 20:26:48.096563 I | raft: a7340362e2996c30 became candidate at term 22
2017-11-14 20:26:48.096589 I | raft: a7340362e2996c30 received MsgVoteResp from a7340362e2996c30 at term 22
2017-11-14 20:26:48.096609 I | raft: a7340362e2996c30 became leader at term 22
2017-11-14 20:26:48.096622 I | raft: raft.node: a7340362e2996c30 elected leader a7340362e2996c30 at term 22
2017-11-14 20:26:48.097433 I | etcdserver: published {Name:openshift.local ClientURLs:[https://10.104.6.127:4001]} to cluster cf86d7c1b2833ba9
I1114 20:26:48.097468 17156 run.go:85] Started etcd at 10.104.6.127:4001
2017-11-14 20:26:48.098248 I | embed: ready to serve client requests
2017-11-14 20:26:48.098751 I | embed: serving client requests on [::]:4001
2017-11-14 20:26:48.298679 I | etcdserver/api/v3rpc: Failed to dial [::]:4001: connection error: desc = "transport: remote error: tls: bad certificate"; please retry.
2017-11-14 20:26:48.299820 I | etcdserver/api/v3rpc: Failed to dial [::]:4001: connection error: desc = "transport: remote error: tls: bad certificate"; please retry.
2017-11-14 20:26:48.305434 I | etcdserver/api/v3rpc: Failed to dial [::]:4001: connection error: desc = "transport: remote error: tls: bad certificate"; please retry.
2017-11-14 20:26:48.307315 I | etcdserver/api/v3rpc: Failed to dial [::]:4001: connection error: desc = "transport: remote error: tls: bad certificate"; please retry.
2017-11-14 20:26:48.307368 I | etcdserver/api/v3rpc: Failed to dial [::]:4001: connection error: desc = "transport: remote error: tls: bad certificate"; please retry.
2017-11-14 20:26:48.307407 I | etcdserver/api/v3rpc: Failed to dial [::]:4001: connection error: desc = "transport: remote error: tls: bad certificate"; please retry.
2017-11-14 20:26:48.307444 I | etcdserver/api/v3rpc: Failed to dial [::]:4001: connection error: desc = "transport: remote error: tls: bad certificate"; please retry.
2017-11-14 20:26:48.307633 I | etcdserver/api/v3rpc: Failed to dial [::]:4001: connection error: desc = "transport: remote error: tls: bad certificate"; please retry.
I1114 20:26:48.308900 17156 run_components.go:91] Using default project node label selector:
I1114 20:26:48.317753 17156 clusterquotamapping.go:160] Starting ClusterQuotaMappingController controller
I1114 20:26:48.318126 17156 master.go:182] Starting OAuth2 API at /oauth
I1114 20:26:48.318141 17156 master.go:190] Starting Web Console /console/
E1114 20:26:48.333055 17156 reflector.go:201] github.com/openshift/origin/pkg/authorization/generated/informers/internalversion/factory.go:45: Failed to list *authorization.ClusterPolicyBinding: Get https://10.104.6.127:8443/apis/authorization.openshift.io/v1/clusterpolicybindings?resourceVersion=0: dial tcp 10.104.6.127:8443: getsockopt: connection refused
E1114 20:26:48.333798 17156 reflector.go:201] github.com/openshift/origin/pkg/authorization/generated/informers/internalversion/factory.go:45: Failed to list *authorization.ClusterPolicy: Get https://10.104.6.127:8443/apis/authorization.openshift.io/v1/clusterpolicies?resourceVersion=0: dial tcp 10.104.6.127:8443: getsockopt: connection refused
E1114 20:26:48.333870 17156 reflector.go:201] github.com/openshift/origin/pkg/authorization/generated/informers/internalversion/factory.go:45: Failed to list *authorization.PolicyBinding: Get https://10.104.6.127:8443/apis/authorization.openshift.io/v1/policybindings?resourceVersion=0: dial tcp 10.104.6.127:8443: getsockopt: connection refused
E1114 20:26:48.333937 17156 reflector.go:201] github.com/openshift/origin/pkg/authorization/generated/informers/internalversion/factory.go:45: Failed to list *authorization.Policy: Get https://10.104.6.127:8443/apis/authorization.openshift.io/v1/policies?resourceVersion=0: dial tcp 10.104.6.127:8443: getsockopt: connection refused
E1114 20:26:48.334012 17156 reflector.go:201] github.com/openshift/origin/pkg/quota/generated/informers/internalversion/factory.go:45: Failed to list *quota.ClusterResourceQuota: Get https://10.104.6.127:8443/apis/quota.openshift.io/v1/clusterresourcequotas?resourceVersion=0: dial tcp 10.104.6.127:8443: getsockopt: connection refused
E1114 20:26:48.642161 17156 controllermanager.go:337] Server isn't healthy yet. Waiting a little while.
W1114 20:26:48.934271 17156 genericapiserver.go:295] Skipping API autoscaling/v2alpha1 because it has no resources.
W1114 20:26:49.108531 17156 genericapiserver.go:295] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
I1114 20:26:49.272056 17156 openshift_apiserver.go:237] Starting Origin API at /apis/user.openshift.io/v1
I1114 20:26:49.274906 17156 openshift_apiserver.go:237] Starting Origin API at /apis/image.openshift.io/v1
I1114 20:26:49.276019 17156 openshift_apiserver.go:237] Starting Origin API at /apis/template.openshift.io/v1
I1114 20:26:49.277340 17156 openshift_apiserver.go:237] Starting Origin API at /apis/security.openshift.io/v1
I1114 20:26:49.278069 17156 openshift_apiserver.go:237] Starting Origin API at /apis/project.openshift.io/v1
I1114 20:26:49.280928 17156 openshift_apiserver.go:237] Starting Origin API at /apis/build.openshift.io/v1
I1114 20:26:49.282884 17156 openshift_apiserver.go:237] Starting Origin API at /apis/apps.openshift.io/v1
I1114 20:26:49.480116 17156 openshift_apiserver.go:237] Starting Origin API at /apis/authorization.openshift.io/v1
I1114 20:26:49.482871 17156 openshift_apiserver.go:237] Starting Origin API at /apis/oauth.openshift.io/v1
I1114 20:26:49.484164 17156 openshift_apiserver.go:237] Starting Origin API at /apis/quota.openshift.io/v1
I1114 20:26:49.486888 17156 openshift_apiserver.go:237] Starting Origin API at /apis/network.openshift.io/v1
I1114 20:26:49.488031 17156 openshift_apiserver.go:237] Starting Origin API at /apis/route.openshift.io/v1
I1114 20:26:49.919536 17156 openshift_apiserver.go:243] Started Origin API at /oapi/v1
E1114 20:26:49.947160 17156 controllermanager.go:337] Server isn't healthy yet. Waiting a little while.
E1114 20:26:49.947241 17156 reflector.go:201] github.com/openshift/origin/pkg/quota/generated/informers/internalversion/factory.go:45: Failed to list *quota.ClusterResourceQuota: Get https://10.104.6.127:8443/apis/quota.openshift.io/v1/clusterresourcequotas?resourceVersion=0: dial tcp 10.104.6.127:8443: getsockopt: connection refused
E1114 20:26:49.947342 17156 reflector.go:201] github.com/openshift/origin/pkg/authorization/generated/informers/internalversion/factory.go:45: Failed to list *authorization.Policy: Get https://10.104.6.127:8443/apis/authorization.openshift.io/v1/policies?resourceVersion=0: dial tcp 10.104.6.127:8443: getsockopt: connection refused
E1114 20:26:49.947449 17156 reflector.go:201] github.com/openshift/origin/pkg/authorization/generated/informers/internalversion/factory.go:45: Failed to list *authorization.PolicyBinding: Get https://10.104.6.127:8443/apis/authorization.openshift.io/v1/policybindings?resourceVersion=0: dial tcp 10.104.6.127:8443: getsockopt: connection refused
E1114 20:26:49.947513 17156 reflector.go:201] github.com/openshift/origin/pkg/authorization/generated/informers/internalversion/factory.go:45: Failed to list *authorization.ClusterPolicy: Get https://10.104.6.127:8443/apis/authorization.openshift.io/v1/clusterpolicies?resourceVersion=0: dial tcp 10.104.6.127:8443: getsockopt: connection refused
E1114 20:26:49.947572 17156 reflector.go:201] github.com/openshift/origin/pkg/authorization/generated/informers/internalversion/factory.go:45: Failed to list *authorization.ClusterPolicyBinding: Get https://10.104.6.127:8443/apis/authorization.openshift.io/v1/clusterpolicybindings?resourceVersion=0: dial tcp 10.104.6.127:8443: getsockopt: connection refused
[restful] 2017/11/14 20:26:50 log.go:30: [restful/swagger] listing is available at https://10.104.6.127:8443/swaggerapi
[restful] 2017/11/14 20:26:50 log.go:30: [restful/swagger] https://10.104.6.127:8443/swaggerui/ is mapped to folder /swagger-ui/
I1114 20:26:50.459540 17156 serve.go:86] Serving securely on 0.0.0.0:8443
W1114 20:26:50.496834 17156 lease_endpoint_reconciler.go:176] Resetting endpoints for master service "kubernetes" to [10.104.6.127]
W1114 20:26:50.570565 17156 run_components.go:60] Binding DNS on port 8053 instead of 53, which may not be resolvable from all clients
I1114 20:26:50.570884 17156 logs.go:41] skydns: ready for queries on cluster.local. for tcp4://0.0.0.0:8053 [rcache 0]
I1114 20:26:50.570898 17156 logs.go:41] skydns: ready for queries on cluster.local. for udp4://0.0.0.0:8053 [rcache 0]
E1114 20:26:50.597709 17156 controllermanager.go:337] Server isn't healthy yet. Waiting a little while.
I1114 20:26:50.671922 17156 run_components.go:86] DNS listening at 0.0.0.0:8053
I1114 20:26:51.448452 17156 docker.go:364] Connecting to docker on unix:///var/run/docker.sock
I1114 20:26:51.448480 17156 docker.go:384] Start docker client with request timeout=2m0s
W1114 20:26:51.452960 17156 cni.go:157] Unable to update cni config: No networks found in /etc/cni/net.d
I1114 20:26:51.490948 17156 node_config.go:367] DNS Bind to 10.104.6.127:53
I1114 20:26:51.490974 17156 start_node.go:345] Starting node sf-docker01.corp.wagerworks.com (v3.6.1+008f2d5)
I1114 20:26:51.492790 17156 start_node.go:354] Connecting to API server https://10.104.6.127:8443
I1114 20:26:51.493026 17156 docker.go:364] Connecting to docker on unix:///var/run/docker.sock
I1114 20:26:51.493042 17156 docker.go:384] Start docker client with request timeout=2m0s
I1114 20:26:51.494487 17156 node.go:134] Connecting to Docker at unix:///var/run/docker.sock
I1114 20:26:51.520004 17156 feature_gate.go:144] feature gates: map[]
I1114 20:26:51.521375 17156 manager.go:143] cAdvisor running in container: "/user.slice"
I1114 20:26:51.524423 17156 node.go:348] Using iptables Proxier.
W1114 20:26:51.531871 17156 node.go:488] Failed to retrieve node info: nodes "sf-docker01.corp.wagerworks.com" not found
W1114 20:26:51.531973 17156 proxier.go:309] invalid nodeIP, initializing kube-proxy with 127.0.0.1 as nodeIP
W1114 20:26:51.531984 17156 proxier.go:314] clusterCIDR not specified, unable to distinguish between internal and external traffic
I1114 20:26:51.532007 17156 node.go:380] Tearing down userspace rules.
I1114 20:26:51.692198 17156 node.go:480] Started Kubernetes Proxy on 0.0.0.0
I1114 20:26:51.693463 17156 node.go:316] Starting DNS on 10.104.6.127:53
I1114 20:26:51.693599 17156 logs.go:41] skydns: ready for queries on cluster.local. for tcp://10.104.6.127:53 [rcache 0]
I1114 20:26:51.693617 17156 logs.go:41] skydns: ready for queries on cluster.local. for udp://10.104.6.127:53 [rcache 0]
W1114 20:26:51.708162 17156 manager.go:151] unable to connect to Rkt api service: rkt: cannot tcp Dial rkt api service: dial tcp [::1]:15441: getsockopt: connection refused
I1114 20:26:51.744723 17156 fs.go:117] Filesystem partitions: map[/dev/mapper/centos-root:{mountpoint:/var/lib/docker/devicemapper major:253 minor:0 fsType:xfs blockSize:0} /dev/sda1:{mountpoint:/boot major:8 minor:1 fsType:xfs blockSize:0}]
I1114 20:26:51.747229 17156 manager.go:198] Machine: {NumCores:1 CpuFrequency:2533423 MemoryCapacity:16658931712 MachineID:197dcaf983ac43d49bc33a715706d364 SystemUUID:423C41D5-41C2-B4C5-FB04-E3E643ABDDC6 BootID:a883e9e3-59c1-46c4-a086-1aa638600889 Filesystems:[{Device:/dev/mapper/centos-root DeviceMajor:253 DeviceMinor:0 Capacity:47724642304 Type:vfs Inodes:28275344 HasInodes:true} {Device:/dev/sda1 DeviceMajor:8 DeviceMinor:1 Capacity:520794112 Type:vfs Inodes:512000 HasInodes:true}] DiskMap:map[2:0:{Name:fd0 Major:2 Minor:0 Size:4096 Scheduler:deadline} 8:0:{Name:sda Major:8 Minor:0 Size:53687091200 Scheduler:deadline} 253:0:{Name:dm-0 Major:253 Minor:0 Size:47747956736 Scheduler:none} 253:1:{Name:dm-1 Major:253 Minor:1 Size:5368709120 Scheduler:none} 253:2:{Name:dm-2 Major:253 Minor:2 Size:107374182400 Scheduler:none}] NetworkDevices:[{Name:ens160 MacAddress:00:50:56:bc:74:5e Speed:10000 Mtu:1500} {Name:virbr0 MacAddress:52:54:00:4c:ea:cb Speed:0 Mtu:1500} {Name:virbr0-nic MacAddress:52:54:00:4c:ea:cb Speed:0 Mtu:1500} {Name:virbr1 MacAddress:52:54:00:fa:ae:1e Speed:0 Mtu:1500} {Name:virbr1-nic MacAddress:52:54:00:fa:ae:1e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:17179402240 Cores:[{Id:0 Threads:[0] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]}] Caches:[{Size:12582912 Type:Unified Level:3}]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
I1114 20:26:51.756883 17156 start_master.go:715] Started serviceaccount-token controller
I1114 20:26:51.764977 17156 manager.go:204] Version: {KernelVersion:3.10.0-693.5.2.el7.x86_64 ContainerOsVersion:CentOS Linux 7 (Core) DockerVersion:17.11.0-ce-rc3 DockerAPIVersion:1.34 CadvisorVersion: CadvisorRevision:}
I1114 20:26:51.765707 17156 server.go:509] --cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to /
W1114 20:26:51.769951 17156 container_manager_linux.go:217] Running with swap on is not supported, please disable swap! This will be a fatal error by default starting in K8s v1.6! In the meantime, you can opt-in to making this a fatal error by enabling --experimental-fail-swap-on.
I1114 20:26:51.770304 17156 container_manager_linux.go:244] container manager verified user specified cgroup-root exists: /
I1114 20:26:51.770323 17156 container_manager_linux.go:249] Creating Container Manager object based on Node Config: {RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:docker CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:systemd ProtectKernelDefaults:false EnableCRI:true NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:memory.available Operator:LessThan Value:{Quantity:100Mi Percentage:0} GracePeriod:0s MinReclaim:}]} ExperimentalQOSReserved:map[]}
I1114 20:26:51.770527 17156 kubelet.go:265] Watching apiserver
W1114 20:26:51.886300 17156 kubelet_network.go:70] Hairpin mode set to "promiscuous-bridge" but kubenet is not enabled, falling back to "hairpin-veth"
I1114 20:26:51.886342 17156 kubelet.go:494] Hairpin mode set to "hairpin-veth"
W1114 20:26:51.902336 17156 cni.go:157] Unable to update cni config: No networks found in /etc/cni/net.d
I1114 20:26:52.030094 17156 start_master.go:783] Started "openshift.io/serviceaccount-pull-secrets"
I1114 20:26:52.038075 17156 docker_service.go:184] Docker cri networking managed by kubernetes.io/no-op
F1114 20:26:52.065947 17156 node.go:281] failed to run Kubelet: failed to create kubelet: misconfiguration: kubelet cgroup driver: "systemd" is different from docker cgroup driver: "cgroupfs"

./openshift version
openshift v3.6.1+008f2d5
kubernetes v1.6.1+5115d708d7
etcd 3.2.1

docker version
Client:
Version: 17.11.0-ce-rc3
API version: 1.34
Go version: go1.8.3
Git commit: 5b4af4f
Built: Wed Nov 8 03:04:32 2017
OS/Arch: linux/amd64

Server:
Version: 17.11.0-ce-rc3
API version: 1.34 (minimum version 1.12)
Go version: go1.8.3
Git commit: 5b4af4f
Built: Wed Nov 8 03:07:05 2017
OS/Arch: linux/amd64
Experimental: false

oc version
oc v3.6.1+008f2d5
kubernetes v1.6.1+5115d708d7
features: Basic-Auth GSSAPI Kerberos SPNEGO

kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.3", GitCommit:"f0efb3cb883751c5ffdbe6d515f3cb4fbe7b7acd", GitTreeState:"clean", BuildDate:"2017-11-08T18:27:48Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

garyyang6 on 14 Nov 2017

Hi @garyyang6. From your comment above it looks like you are not modifying a node config but another unit file. Please note my comment here: https://github.com/openshift/origin/issues/16974#issuecomment-339006432. It also looks like you are attempting to run the kubelet by itself but also starting an all-in-one openshift server which includes the kubelet.

If you'd like to try out the all-in-one I would recommend updating the docker daemon setting. Alternatively you may write out the standard config (see openshift start --help), modify the node config, and start the node and master separately.

pweil- on 14 Nov 2017

Hi @pweil. I read #16974. But, I do not understand your comments. Please tell me how to modify the node config. I did not mean to run kubelet by itself and also starting an all-in-one openshift server. But, what should I do? I stopped kubelet, "systemctl stop kubelet" and restart openshif. But, I still get same error..

garyyang6 on 14 Nov 2017

@garyyang6 Please try updating the docker daemon config to use the systemd driver. Then you should be able to perform the openshift start command to start the all-in-one.

The alternative method (using the binary) would be openshift start --write-config, modify the node-config.yaml to include the kubelet arguments section and pass the argument there, then running openshift start node and openshift start master passing in the configuration locations from --write-config. That's a little more involved if you're just looking to try out the all-in-one :smile:

@sdodson this is the second time this has come up in the last week. First was on Ubuntu but the desc of this one notes Centos 7. Do you know if something has changed?

pweil- on 14 Nov 2017

@pweil: How to "updating the docker daemon config to use the systemd driver"?

garyyang6 on 14 Nov 2017

@pweil How to use openshift start --write-config to modify the node-config.yaml and pass what value of kubelet arguments? Please give me specific command. Thanks.

garyyang6 on 14 Nov 2017

@pweil: How to "updating the docker daemon config to use the systemd driver"?

To update the docker daemon config you need to pass --exec-opt native.cgroupdriver=systemd to the docker daemon. Update your systemd unit file to pass this argument and restart the docker with systemctl. From there try your openshift start command again.

For more info on this docker parameter please refer to the upstream documentation: https://docs.docker.com/engine/reference/commandline/dockerd/#docker-runtime-execution-options

pweil- on 15 Nov 2017

@pweil- I started docker with the command, dockerd --exec-opt native.cgroupdriver=systemd. And restarted openshift. But, I got "getsockopt: connection refused". I cannot hit the openshif origin page.

./openshift start
W1115 17:52:34.943310 W1115 17:52:34.943498 W1115 17:52:34.943512 I1115 17:52:34.966025 2017-11-15 17:52:35.007674 2017-11-15 17:52:35.007747 2017-11-15 17:52:35.007793 2017-11-15 17:52:35.007840 E1115 17:52:35.007905 2017-11-15 17:52:35.035496 I1115 17:52:35.328465 I1115 17:52:35.328501 I1115 17:52:35.328534 2017-11-15 17:52:35.328671 2017-11-15 17:52:35.330166 2017-11-15 17:52:35.330245 2017-11-15 17:52:35.335840 2017-11-15 17:52:35.335862 2017-11-15 17:52:35.335874 2017-11-15 17:52:35.335883 2017-11-15 17:52:35.335892 2017-11-15 17:52:35.335902 2017-11-15 17:52:35.335921 2017-11-15 17:52:35.385982 2017-11-15 17:52:35.386102 2017-11-15 17:52:35.386123 2017-11-15 17:52:35.420983 2017-11-15 17:52:35.422448 2017-11-15 17:52:35.425367 2017-11-15 17:52:35.425404 2017-11-15 17:52:35.428015 2017-11-15 17:52:35.428141 2017-11-15 17:52:35.428191 E1115 17:52:36.009139 2017-11-15 17:52:36.286587 2017-11-15 17:52:36.286682 2017-11-15 17:52:36.286714 2017-11-15 17:52:36.286736 2017-11-15 17:52:36.286749 2017-11-15 17:52:36.287304 I1115 17:52:36.287339 2017-11-15 17:52:36.288156 2017-11-15 17:52:36.288685 2017-11-15 17:52:36.539973 2017-11-15 17:52:36.540180 2017-11-15 17:52:36.542774 2017-11-15 17:52:36.542818 2017-11-15 17:52:36.542854 2017-11-15 17:52:36.550348 2017-11-15 17:52:36.550416 2017-11-15 17:52:36.551941 I1115 17:52:36.556899 I1115 17:52:36.579084 I1115 17:52:36.581073 I1115 17:52:36.581091 E1115 17:52:36.595429 E1115 17:52:36.595496 E1115 17:52:36.595559 E1115 17:52:36.595625 E1115 17:52:36.595682 W1115 17:52:36.924533 W1115 17:52:37.058477 I1115 17:52:37.218594 I1115 17:52:37.219876 I1115 17:52:37.222561 I1115 17:52:37.223848 I1115 17:52:37.225158 I1115 17:52:37.227121 I1115 17:52:37.423171 I1115 17:52:37.424021 I1115 17:52:37.426660 I1115 17:52:37.430855 I1115 17:52:37.433812 I1115 17:52:37.574259 E1115 17:52:37.684944 I1115 17:52:37.782821 E1115 17:52:37.872250 E1115 17:52:37.929480 E1115 17:52:37.982890 E1115 17:52:37.982974 E1115 17:52:37.983039 E1115 17:52:38.122606 [restful] 2017/11/15 [restful] 2017/11/15 I1115 17:52:38.453433 W1115 17:52:38.491931 W1115 17:52:38.563626 I1115 17:52:38.564102 I1115 17:52:38.564117 I1115 17:52:38.665541 E1115 17:52:39.018226 I1115 17:52:39.494491 I1115 17:52:39.494518 W1115 17:52:39.500746 I1115 17:52:39.535200 I1115 17:52:39.535227 I1115 17:52:39.537152 I1115 17:52:39.537190 I1115 17:52:39.537203 I1115 17:52:39.538600 I1115 17:52:39.560591 I1115 17:52:39.560795 I1115 17:52:39.578057 W1115 17:52:39.580587 W1115 17:52:39.580676 W1115 17:52:39.580687 I1115 17:52:39.580704 W1115 17:52:39.607612 I1115 17:52:39.636780 I1115 17:52:39.640010 I1115 17:52:39.641057 I1115 17:52:39.650867 W1115 17:52:39.658520 I1115 17:52:39.658616 I1115 17:52:39.658634 I1115 17:52:39.658834 W1115 17:52:39.701505 I1115 17:52:39.701564 W1115 17:52:39.713957 I1115 17:52:39.833770 I1115 17:52:39.834838 I1115 17:52:39.834981 I1115 17:52:39.834998 I1115 17:52:39.835853 I1115 17:52:39.857578 I1115 17:52:39.903663 I1115 17:52:39.905727 I1115 17:52:39.907243 I1115 17:52:39.907282 I1115 17:52:39.908129 E1115 17:52:39.910684 I1115 17:52:39.911159 E1115 17:52:39.963089 E1115 17:52:39.963152 I1115 17:52:40.099961 E1115 17:52:40.178336 I1115 17:52:40.184147 I1115 17:52:40.184184 I1115 17:52:40.184205 I1115 17:52:40.184231 W1115 17:52:40.184473 W1115 17:52:40.184483 W1115 17:52:40.185542 W1115 17:52:40.185552 I1115 17:52:40.185579 I1115 17:52:40.298135 E1115 17:52:40.300475 I1115 17:52:40.300491 W1115 17:52:40.300517 I1115 17:52:40.300528 I1115 17:52:40.300712 I1115 17:52:40.300944 I1115 17:52:40.307491 I1115 17:52:40.308418 I1115 17:52:40.463224 E1115 17:52:40.588819 I1115 17:52:40.595727 I1115 17:52:40.615942 I1115 17:52:40.627493 I1115 17:52:40.674701 I1115 17:52:40.674830 I1115 17:52:40.723376 W1115 17:52:40.723403 I1115 17:52:40.723521 E1115 17:52:40.740294 I1115 17:52:40.740376 W1115 17:52:40.740389 I1115 17:52:40.740505 I1115 17:52:40.792620 I1115 17:52:40.806215 I1115 17:52:40.821992 I1115 17:52:40.822157 I1115 17:52:40.880262 I1115 17:52:40.944700 I1115 17:52:40.998513 I1115 17:52:40.998562 I1115 17:52:41.004943 I1115 17:52:41.005122 I1115 17:52:41.005144 I1115 17:52:41.005155 I1115 17:52:41.005189 I1115 17:52:41.005221 I1115 17:52:41.005254 I1115 17:52:41.025638 I1115 17:52:41.025778 I1115 17:52:41.071024 W1115 17:52:41.071060 W1115 17:52:41.071073 I1115 17:52:41.087013 I1115 17:52:41.087144 I1115 17:52:41.087224 I1115 17:52:41.102170 I1115 17:52:41.102297 I1115 17:52:41.115088 I1115 17:52:41.115221 I1115 17:52:41.127914 I1115 17:52:41.128042 I1115 17:52:41.143382 W1115 17:52:41.168513 I1115 17:52:41.179771 W1115 17:52:41.179796 W1115 17:52:41.179807 W1115 17:52:41.179814 I1115 17:52:41.192598 I1115 17:52:41.192727 I1115 17:52:41.205350 I1115 17:52:41.218399 I1115 17:52:41.218530 I1115 17:52:41.384429 I1115 17:52:41.430656 I1115 17:52:41.453448 I1115 17:52:41.453586 I1115 17:52:41.465669 I1115 17:52:41.465793 I1115 17:52:41.477721 I1115 17:52:41.477840 I1115 17:52:41.489654 I1115 17:52:41.489774 I1115 17:52:41.501280 W1115 17:52:41.546894 I1115 17:52:41.546935 I1115 17:52:41.547127 I1115 17:52:41.590304 W1115 17:52:41.590332 I1115 17:52:41.590458 I1115 17:52:41.609986 I1115 17:52:41.622897 I1115 17:52:41.623147 E1115 17:52:41.638640 W1115 17:52:41.638669 I1115 17:52:41.662645 I1115 17:52:41.662670 I1115 17:52:41.722166 I1115 17:52:41.722247 E1115 17:52:41.726142 E1115 17:52:41.726157 I1115 17:52:41.786215 I1115 17:52:41.787968 I1115 17:52:41.806392 I1115 17:52:41.809745 W1115 17:52:41.809843 I1115 17:52:41.809891 I1115 17:52:41.812718 I1115 17:52:41.813148 I1115 17:52:41.897842 8336 start_master.go:297] Warning: assetConfig.loggingPublicURL: Invalid value: "": required to view aggregated container logs in the console, master start will continue.
8336 start_master.go:297] Warning: assetConfig.metricsPublicURL: Invalid value: "": required to view cluster metrics in the console, master start will continue.
8336 start_master.go:297] Warning: auditConfig.auditFilePath: Required value: audit can not be logged to a separate file, master start will continue.
8336 plugins.go:101] No cloud provider specified.
I | etcdserver/api/v3rpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.104.6.127:4001: getsockopt: connection refused"; Reconnecting to {10.104.6.127:4001 }
I | etcdserver/api/v3rpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.104.6.127:4001: getsockopt: connection refused"; Reconnecting to {10.104.6.127:4001 }
I | etcdserver/api/v3rpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.104.6.127:4001: getsockopt: connection refused"; Reconnecting to {10.104.6.127:4001 }
I | etcdserver/api/v3rpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.104.6.127:4001: getsockopt: connection refused"; Reconnecting to {10.104.6.127:4001 }
8336 controllermanager.go:337] Server isn't healthy yet. Waiting a little while.
I | etcdserver/api/v3rpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.104.6.127:4001: getsockopt: connection refused"; Reconnecting to {10.104.6.127:4001 }
8336 start_master.go:529] Starting master on 0.0.0.0:8443 (v3.6.1+008f2d5)
8336 start_master.go:530] Public master address is https://10.104.6.127:8443
8336 start_master.go:534] Using images from "openshift/origin-:v3.6.1"
I | embed: peerTLS: cert = openshift.local.config/master/etcd.server.crt, key = openshift.local.config/master/etcd.server.key, ca = openshift.local.config/master/ca.crt, trusted-ca = , client-cert-auth = true
I | embed: listening for peers on https://0.0.0.0:7001
I | embed: listening for client requests on 0.0.0.0:4001
I | etcdserver: name = openshift.local
I | etcdserver: data dir = openshift.local.etcd
I | etcdserver: member dir = openshift.local.etcd/member
I | etcdserver: heartbeat = 100ms
I | etcdserver: election = 1000ms
I | etcdserver: snapshot count = 100000
I | etcdserver: advertise client URLs = https://10.104.6.127:4001
I | etcdserver: restarting member a7340362e2996c30 in cluster cf86d7c1b2833ba9 at commit index 1231
I | raft: a7340362e2996c30 became follower at term 23
I | raft: newRaft a7340362e2996c30 [peers: [], term: 23, commit: 1231, applied: 0, lastindex: 1231, lastterm: 23]
I | etcdserver/api/v3rpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.104.6.127:4001: getsockopt: connection refused"; Reconnecting to {10.104.6.127:4001 }
W | auth: simple token is not cryptographically signed
I | etcdserver: starting server... [version: 3.2.1, cluster version: to_be_decided]
I | embed: ClientTLS: cert = openshift.local.config/master/etcd.server.crt, key = openshift.local.config/master/etcd.server.key, ca = openshift.local.config/master/ca.crt, trusted-ca = , client-cert-auth = true
I | etcdserver/membership: added member a7340362e2996c30 [https://10.104.6.127:7001] to cluster cf86d7c1b2833ba9
N | etcdserver/membership: set the initial cluster version to 3.2
I | etcdserver/api: enabled capabilities for version 3.2
8336 controllermanager.go:337] Server isn't healthy yet. Waiting a little while.
I | raft: a7340362e2996c30 is starting a new election at term 23
I | raft: a7340362e2996c30 became candidate at term 24
I | raft: a7340362e2996c30 received MsgVoteResp from a7340362e2996c30 at term 24
I | raft: a7340362e2996c30 became leader at term 24
I | raft: raft.node: a7340362e2996c30 elected leader a7340362e2996c30 at term 24
I | etcdserver: published {Name:openshift.local ClientURLs:[https://10.104.6.127:4001]} to cluster cf86d7c1b2833ba9
8336 run.go:85] Started etcd at 10.104.6.127:4001
I | embed: ready to serve client requests
I | embed: serving client requests on [::]:4001
I | etcdserver/api/v3rpc: Failed to dial [::]:4001: connection error: desc = "transport: remote error: tls: bad certificate"; please retry.
I | etcdserver/api/v3rpc: Failed to dial [::]:4001: connection error: desc = "transport: remote error: tls: bad certificate"; please retry.
I | etcdserver/api/v3rpc: Failed to dial [::]:4001: connection error: desc = "transport: remote error: tls: bad certificate"; please retry.
I | etcdserver/api/v3rpc: Failed to dial [::]:4001: connection error: desc = "transport: remote error: tls: bad certificate"; please retry.
I | etcdserver/api/v3rpc: Failed to dial [::]:4001: connection error: desc = "transport: remote error: tls: bad certificate"; please retry.
I | etcdserver/api/v3rpc: Failed to dial [::]:4001: connection error: desc = "transport: remote error: tls: bad certificate"; please retry.
I | etcdserver/api/v3rpc: Failed to dial [::]:4001: connection error: desc = "transport: remote error: tls: bad certificate"; please retry.
I | etcdserver/api/v3rpc: Failed to dial [::]:4001: connection error: desc = "transport: remote error: tls: bad certificate"; please retry.
8336 run_components.go:91] Using default project node label selector:
8336 clusterquotamapping.go:160] Starting ClusterQuotaMappingController controller
8336 master.go:182] Starting OAuth2 API at /oauth
8336 master.go:190] Starting Web Console /console/
8336 reflector.go:201] github.com/openshift/origin/pkg/authorization/generated/informers/internalversion/factory.go:45: Failed to list authorization.ClusterPolicyBinding: Get https://10.104.6.127:8443/apis/authorization.openshift.io/v1/clusterpolicybindings?resourceVersion=0: dial tcp 10.104.6.127:8443: getsockopt: connection refused
8336 reflector.go:201] github.com/openshift/origin/pkg/authorization/generated/informers/internalversion/factory.go:45: Failed to list *authorization.ClusterPolicy: Get https://10.104.6.127:8443/apis/authorization.openshift.io/v1/clusterpolicies?resourceVersion=0: dial tcp 10.104.6.127:8443: getsockopt: connection refused
8336 reflector.go:201] github.com/openshift/origin/pkg/authorization/generated/informers/internalversion/factory.go:45: Failed to list *authorization.PolicyBinding: Get https://10.104.6.127:8443/apis/authorization.openshift.io/v1/policybindings?resourceVersion=0: dial tcp 10.104.6.127:8443: getsockopt: connection refused
8336 reflector.go:201] github.com/openshift/origin/pkg/authorization/generated/informers/internalversion/factory.go:45: Failed to list *authorization.Policy: Get https://10.104.6.127:8443/apis/authorization.openshift.io/v1/policies?resourceVersion=0: dial tcp 10.104.6.127:8443: getsockopt: connection refused
8336 reflector.go:201] github.com/openshift/origin/pkg/quota/generated/informers/internalversion/factory.go:45: Failed to list *quota.ClusterResourceQuota: Get https://10.104.6.127:8443/apis/quota.openshift.io/v1/clusterresourcequotas?resourceVersion=0: dial tcp 10.104.6.127:8443: getsockopt: connection refused
8336 genericapiserver.go:295] Skipping API autoscaling/v2alpha1 because it has no resources.
8336 genericapiserver.go:295] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
8336 openshift_apiserver.go:237] Starting Origin API at /apis/oauth.openshift.io/v1
8336 openshift_apiserver.go:237] Starting Origin API at /apis/security.openshift.io/v1
8336 openshift_apiserver.go:237] Starting Origin API at /apis/build.openshift.io/v1
8336 openshift_apiserver.go:237] Starting Origin API at /apis/quota.openshift.io/v1
8336 openshift_apiserver.go:237] Starting Origin API at /apis/route.openshift.io/v1
8336 openshift_apiserver.go:237] Starting Origin API at /apis/apps.openshift.io/v1
8336 openshift_apiserver.go:237] Starting Origin API at /apis/authorization.openshift.io/v1
8336 openshift_apiserver.go:237] Starting Origin API at /apis/project.openshift.io/v1
8336 openshift_apiserver.go:237] Starting Origin API at /apis/network.openshift.io/v1
8336 openshift_apiserver.go:237] Starting Origin API at /apis/user.openshift.io/v1
8336 openshift_apiserver.go:237] Starting Origin API at /apis/image.openshift.io/v1
8336 openshift_apiserver.go:237] Starting Origin API at /apis/template.openshift.io/v1
8336 controllermanager.go:337] Server isn't healthy yet. Waiting a little while.
8336 openshift_apiserver.go:243] Started Origin API at /oapi/v1
8336 reflector.go:201] github.com/openshift/origin/pkg/authorization/generated/informers/internalversion/factory.go:45: Failed to list *authorization.ClusterPolicy: Get https://10.104.6.127:8443/apis/authorization.openshift.io/v1/clusterpolicies?resourceVersion=0: dial tcp 10.104.6.127:8443: getsockopt: connection refused
8336 reflector.go:201] github.com/openshift/origin/pkg/authorization/generated/informers/internalversion/factory.go:45: Failed to list *authorization.ClusterPolicyBinding: Get https://10.104.6.127:8443/apis/authorization.openshift.io/v1/clusterpolicybindings?resourceVersion=0: dial tcp 10.104.6.127:8443: getsockopt: connection refused
8336 reflector.go:201] github.com/openshift/origin/pkg/quota/generated/informers/internalversion/factory.go:45: Failed to list *quota.ClusterResourceQuota: Get https://10.104.6.127:8443/apis/quota.openshift.io/v1/clusterresourcequotas?resourceVersion=0: dial tcp 10.104.6.127:8443: getsockopt: connection refused
8336 reflector.go:201] github.com/openshift/origin/pkg/authorization/generated/informers/internalversion/factory.go:45: Failed to list *authorization.Policy: Get https://10.104.6.127:8443/apis/authorization.openshift.io/v1/policies?resourceVersion=0: dial tcp 10.104.6.127:8443: getsockopt: connection refused
8336 reflector.go:201] github.com/openshift/origin/pkg/authorization/generated/informers/internalversion/factory.go:45: Failed to list *authorization.PolicyBinding: Get https://10.104.6.127:8443/apis/authorization.openshift.io/v1/policybindings?resourceVersion=0: dial tcp 10.104.6.127:8443: getsockopt: connection refused
8336 controllermanager.go:337] Server isn't healthy yet. Waiting a little while.
17:52:38 log.go:30: [restful/swagger] listing is available at https://10.104.6.127:8443/swaggerapi
17:52:38 log.go:30: [restful/swagger] https://10.104.6.127:8443/swaggerui/ is mapped to folder /swagger-ui/
8336 serve.go:86] Serving securely on 0.0.0.0:8443
8336 lease_endpoint_reconciler.go:176] Resetting endpoints for master service "kubernetes" to [10.104.6.127]
8336 run_components.go:60] Binding DNS on port 8053 instead of 53, which may not be resolvable from all clients
8336 logs.go:41] skydns: ready for queries on cluster.local. for tcp4://0.0.0.0:8053 [rcache 0]
8336 logs.go:41] skydns: ready for queries on cluster.local. for udp4://0.0.0.0:8053 [rcache 0]
8336 run_components.go:86] DNS listening at 0.0.0.0:8053
8336 controllermanager.go:337] Server isn't healthy yet. Waiting a little while.
8336 docker.go:364] Connecting to docker on unix:///var/run/docker.sock
8336 docker.go:384] Start docker client with request timeout=2m0s
8336 cni.go:157] Unable to update cni config: No networks found in /etc/cni/net.d
8336 node_config.go:367] DNS Bind to 10.104.6.127:53
8336 start_node.go:345] Starting node sf-docker01.corp.wagerworks.com (v3.6.1+008f2d5)
8336 start_node.go:354] Connecting to API server https://10.104.6.127:8443
8336 docker.go:364] Connecting to docker on unix:///var/run/docker.sock
8336 docker.go:384] Start docker client with request timeout=2m0s
8336 node.go:134] Connecting to Docker at unix:///var/run/docker.sock
8336 feature_gate.go:144] feature gates: map[]
8336 manager.go:143] cAdvisor running in container: "/user.slice"
8336 node.go:348] Using iptables Proxier.
8336 node.go:488] Failed to retrieve node info: nodes "sf-docker01.corp.wagerworks.com" not found
8336 proxier.go:309] invalid nodeIP, initializing kube-proxy with 127.0.0.1 as nodeIP
8336 proxier.go:314] clusterCIDR not specified, unable to distinguish between internal and external traffic
8336 node.go:380] Tearing down userspace rules.
8336 manager.go:151] unable to connect to Rkt api service: rkt: cannot tcp Dial rkt api service: dial tcp [::1]:15441: getsockopt: connection refused
8336 fs.go:117] Filesystem partitions: map[/dev/mapper/centos-root:{mountpoint:/var/lib/docker/devicemapper major:253 minor:0 fsType:xfs blockSize:0} /dev/sda1:{mountpoint:/boot major:8 minor:1 fsType:xfs blockSize:0}]
8336 manager.go:198] Machine: {NumCores:1 CpuFrequency:2533423 MemoryCapacity:16658931712 MachineID:197dcaf983ac43d49bc33a715706d364 SystemUUID:423C41D5-41C2-B4C5-FB04-E3E643ABDDC6 BootID:a883e9e3-59c1-46c4-a086-1aa638600889 Filesystems:[{Device:/dev/mapper/centos-root DeviceMajor:253 DeviceMinor:0 Capacity:47724642304 Type:vfs Inodes:28311952 HasInodes:true} {Device:/dev/sda1 DeviceMajor:8 DeviceMinor:1 Capacity:520794112 Type:vfs Inodes:512000 HasInodes:true}] DiskMap:map[253:0:{Name:dm-0 Major:253 Minor:0 Size:47747956736 Scheduler:none} 253:1:{Name:dm-1 Major:253 Minor:1 Size:5368709120 Scheduler:none} 253:2:{Name:dm-2 Major:253 Minor:2 Size:107374182400 Scheduler:none} 2:0:{Name:fd0 Major:2 Minor:0 Size:4096 Scheduler:deadline} 8:0:{Name:sda Major:8 Minor:0 Size:53687091200 Scheduler:deadline}] NetworkDevices:[{Name:ens160 MacAddress:00:50:56:bc:74:5e Speed:10000 Mtu:1500} {Name:virbr0 MacAddress:52:54:00:4c:ea:cb Speed:0 Mtu:1500} {Name:virbr0-nic MacAddress:52:54:00:4c:ea:cb Speed:0 Mtu:1500} {Name:virbr1 MacAddress:52:54:00:fa:ae:1e Speed:0 Mtu:1500} {Name:virbr1-nic MacAddress:52:54:00:fa:ae:1e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:17179402240 Cores:[{Id:0 Threads:[0] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]}] Caches:[{Size:12582912 Type:Unified Level:3}]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
8336 manager.go:204] Version: {KernelVersion:3.10.0-693.5.2.el7.x86_64 ContainerOsVersion:CentOS Linux 7 (Core) DockerVersion:17.11.0-ce-rc3 DockerAPIVersion:1.34 CadvisorVersion: CadvisorRevision:}
8336 server.go:509] --cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to /
8336 container_manager_linux.go:217] Running with swap on is not supported, please disable swap! This will be a fatal error by default starting in K8s v1.6! In the meantime, you can opt-in to making this a fatal error by enabling --experimental-fail-swap-on.
8336 container_manager_linux.go:244] container manager verified user specified cgroup-root exists: /
8336 container_manager_linux.go:249] Creating Container Manager object based on Node Config: {RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:docker CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:systemd ProtectKernelDefaults:false EnableCRI:true NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:memory.available Operator:LessThan Value:{Quantity:100Mi Percentage:0} GracePeriod:0s MinReclaim:}]} ExperimentalQOSReserved:map[]}
8336 kubelet.go:265] Watching apiserver
8336 kubelet_network.go:70] Hairpin mode set to "promiscuous-bridge" but kubenet is not enabled, falling back to "hairpin-veth"
8336 kubelet.go:494] Hairpin mode set to "hairpin-veth"
8336 cni.go:157] Unable to update cni config: No networks found in /etc/cni/net.d
8336 node.go:480] Started Kubernetes Proxy on 0.0.0.0
8336 node.go:316] Starting DNS on 10.104.6.127:53
8336 logs.go:41] skydns: ready for queries on cluster.local. for tcp://10.104.6.127:53 [rcache 0]
8336 logs.go:41] skydns: ready for queries on cluster.local. for udp://10.104.6.127:53 [rcache 0]
8336 docker_service.go:184] Docker cri networking managed by kubernetes.io/no-op
8336 docker_service.go:201] Setting cgroupDriver to systemd
8336 remote_runtime.go:41] Connecting to runtime service /var/run/dockershim.sock
8336 kuberuntime_manager.go:171] Container runtime docker initialized, version: 17.11.0-ce-rc3, apiVersion: 1.34.0
8336 server.go:869] Started kubelet v1.6.1+5115d708d7
8336 server.go:127] Starting to listen on 0.0.0.0:10250
8336 server.go:298] Adding debug handlers to kubelet server.
8336 kubelet.go:1170] Image garbage collection failed: unable to find data for container /
8336 kubelet_node_status.go:253] Setting node annotation to enable volume controller attach/detach
8336 kubelet.go:1669] Failed to check if disk space is available for the runtime: failed to get fs info for "runtime": unable to find data for container /
8336 kubelet.go:1677] Failed to check if disk space is available on the root partition: failed to get fs info for "root": unable to find data for container /
8336 start_master.go:715] Started serviceaccount-token controller
8336 event.go:259] Could not construct reference to: '&v1.Node{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"sf-docker01.corp.wagerworks.com", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{sec:0, nsec:0, loc:(time.Location)(nil)}}, DeletionTimestamp:(v1.Time)(nil), DeletionGracePeriodSeconds:(int64)(nil), Labels:map[string]string{"beta.kubernetes.io/arch":"amd64", "kubernetes.io/hostname":"sf-docker01.corp.wagerworks.com", "beta.kubernetes.io/os":"linux"}, Annotations:map[string]string{"volumes.kubernetes.io/controller-managed-attach-detach":"true"}, OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.NodeSpec{PodCIDR:"", ExternalID:"sf-docker01.corp.wagerworks.com", ProviderID:"", Unschedulable:false, Taints:[]v1.Taint(nil)}, Status:v1.NodeStatus{Capacity:v1.ResourceList{"pods":resource.Quantity{i:resource.int64Amount{value:10, scale:0}, d:resource.infDecAmount{Dec:(inf.Dec)(nil)}, s:"", Format:"DecimalSI"}, "memory":resource.Quantity{i:resource.int64Amount{value:16658931712, scale:0}, d:resource.infDecAmount{Dec:(inf.Dec)(nil)}, s:"", Format:"BinarySI"}, "cpu":resource.Quantity{i:resource.int64Amount{value:1000, scale:-3}, d:resource.infDecAmount{Dec:(inf.Dec)(nil)}, s:"", Format:"DecimalSI"}}, Allocatable:v1.ResourceList{"memory":resource.Quantity{i:resource.int64Amount{value:16554074112, scale:0}, d:resource.infDecAmount{Dec:(inf.Dec)(nil)}, s:"", Format:"BinarySI"}, "cpu":resource.Quantity{i:resource.int64Amount{value:1000, scale:-3}, d:resource.infDecAmount{Dec:(inf.Dec)(nil)}, s:"", Format:"DecimalSI"}, "pods":resource.Quantity{i:resource.int64Amount{value:10, scale:0}, d:resource.infDecAmount{Dec:(inf.Dec)(nil)}, s:"", Format:"DecimalSI"}}, Phase:"", Conditions:[]v1.NodeCondition{v1.NodeCondition{Type:"OutOfDisk", Status:"False", LastHeartbeatTime:v1.Time{Time:time.Time{sec:63646365159, nsec:963052559, loc:(time.Location)(0xc000180)}}, LastTransitionTime:v1.Time{Time:time.Time{sec:63646365159, nsec:963052559, loc:(time.Location)(0xc000180)}}, Reason:"KubeletHasSufficientDisk", Message:"kubelet has sufficient disk space available"}, v1.NodeCondition{Type:"MemoryPressure", Status:"False", LastHeartbeatTime:v1.Time{Time:time.Time{sec:63646365159, nsec:963192617, loc:(time.Location)(0xc000180)}}, LastTransitionTime:v1.Time{Time:time.Time{sec:63646365159, nsec:963192617, loc:(time.Location)(0xc000180)}}, Reason:"KubeletHasSufficientMemory", Message:"kubelet has sufficient memory available"}, v1.NodeCondition{Type:"DiskPressure", Status:"False", LastHeartbeatTime:v1.Time{Time:time.Time{sec:63646365159, nsec:963206987, loc:(time.Location)(0xc000180)}}, LastTransitionTime:v1.Time{Time:time.Time{sec:63646365159, nsec:963206987, loc:(time.Location)(0xc000180)}}, Reason:"KubeletHasNoDiskPressure", Message:"kubelet has no disk pressure"}, v1.NodeCondition{Type:"Ready", Status:"False", LastHeartbeatTime:v1.Time{Time:time.Time{sec:63646365159, nsec:963212152, loc:(time.Location)(0xc000180)}}, LastTransitionTime:v1.Time{Time:time.Time{sec:63646365159, nsec:963212152, loc:(time.Location)(0xc000180)}}, Reason:"KubeletNotReady", Message:"container runtime is down,PLEG is not healthy: pleg was last seen active 2562047h47m16.854775807s ago; threshold is 3m0s,network state unknown"}}, Addresses:[]v1.NodeAddress{v1.NodeAddress{Type:"LegacyHostIP", Address:"10.104.6.127"}, v1.NodeAddress{Type:"InternalIP", Address:"10.104.6.127"}, v1.NodeAddress{Type:"Hostname", Address:"sf-docker01.corp.wagerworks.com"}}, DaemonEndpoints:v1.NodeDaemonEndpoints{KubeletEndpoint:v1.DaemonEndpoint{Port:10250}}, NodeInfo:v1.NodeSystemInfo{MachineID:"197dcaf983ac43d49bc33a715706d364", SystemUUID:"423C41D5-41C2-B4C5-FB04-E3E643ABDDC6", BootID:"a883e9e3-59c1-46c4-a086-1aa638600889", KernelVersion:"3.10.0-693.5.2.el7.x86_64", OSImage:"CentOS Linux 7 (Core)", ContainerRuntimeVersion:"docker://17.11.0-ce-rc3", KubeletVersion:"v1.6.1+5115d708d7", KubeProxyVersion:"v1.6.1+5115d708d7", OperatingSystem:"linux", Architecture:"amd64"}, Images:[]v1.ContainerImage{v1.ContainerImage{Names:[]string{"centos@sha256:26f74cefad82967f97f3eeeef88c1b6262f9b42bc96f2ad61d6f3fdf544759b8", "centos:latest"}, SizeBytes:192503276}}, VolumesInUse:[]v1.UniqueVolumeName(nil), VolumesAttached:[]v1.AttachedVolume(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'NodeAllocatableEnforced' 'Updated Node Allocatable limit across pods'
8336 fs_resource_analyzer.go:66] Starting FS ResourceAnalyzer
8336 status_manager.go:140] Starting to sync pod status with apiserver
8336 kubelet.go:1749] Starting kubelet main sync loop.
8336 kubelet.go:1760] skipping pod synchronization - [container runtime is down PLEG is not healthy: pleg was last seen active 2562047h47m16.854775807s ago; threshold is 3m0s]
8336 container_manager_linux.go:750] CPUAccounting not enabled for pid: 8161
8336 container_manager_linux.go:753] MemoryAccounting not enabled for pid: 8161
8336 container_manager_linux.go:750] CPUAccounting not enabled for pid: 8336
8336 container_manager_linux.go:753] MemoryAccounting not enabled for pid: 8336
8336 volume_manager.go:249] Starting Kubelet Volume Manager
8336 kubelet_node_status.go:253] Setting node annotation to enable volume controller attach/detach
8336 factory.go:330] devicemapper filesystem stats will not be reported: usage of thin_ls is disabled to preserve iops
8336 factory.go:342] Registering Docker factory
8336 manager.go:247] Registration of the rkt container factory failed: unable to communicate with Rkt api service: rkt: cannot tcp Dial rkt api service: dial tcp [::1]:15441: getsockopt: connection refused
8336 factory.go:54] Registering systemd factory
8336 factory.go:86] Registering Raw factory
8336 manager.go:1106] Started watching for new ooms in manager
8336 oomparser.go:185] oomparser using systemd
8336 manager.go:288] Starting recovery of all containers
8336 manager.go:293] Recovery completed
8336 eviction_manager.go:214] eviction manager: unexpected err: failed GetNode: node 'sf-docker01.corp.wagerworks.com' not found
8336 kubelet_node_status.go:77] Attempting to register node sf-docker01.corp.wagerworks.com
8336 kubelet_node_status.go:80] Successfully registered node sf-docker01.corp.wagerworks.com
8336 start_master.go:783] Started "openshift.io/build-config-change"
8336 start_master.go:783] Started "openshift.io/deploymentconfig"
8336 factory.go:84] Starting deploymentconfig controller
8336 start_master.go:783] Started "resourcequota"
8336 start_master.go:765] "tokencleaner" is skipped
8336 resource_quota_controller.go:240] Starting resource quota controller
8336 util.go:45] Metric for serviceaccount_controller already registered
8336 start_master.go:783] Started "serviceaccount"
8336 start_master.go:765] "openshift.io/templateinstance" is skipped
8336 serviceaccounts_controller.go:122] Starting ServiceAccount controller
8336 start_master.go:783] Started "openshift.io/cluster-quota-reconciliation"
8336 start_master.go:783] Started "endpoint"
8336 start_master.go:783] Started "replicaset"
8336 replica_set.go:155] Starting ReplicaSet controller
8336 start_master.go:783] Started "openshift.io/serviceaccount-pull-secrets"
8336 imagestream_controller.go:65] Starting image stream controller
8336 start_master.go:783] Started "openshift.io/image-import"
8336 start_master.go:783] Started "openshift.io/cluster-quota-mapping"
8336 start_master.go:783] Started "openshift.io/origin-to-rbac"
8336 generic.go:33] Starting OriginRoleBindingToRBACRoleBindingController controller
8336 scheduled_image_controller.go:59] Starting scheduled import controller
8336 clusterquotamapping.go:160] Starting ClusterQuotaMappingController controller
8336 generic.go:33] Starting OriginClusterRoleToRBACClusterRoleController controller
8336 generic.go:33] Starting OriginClusterRoleBindingToRBACClusterRoleBindingController controller
8336 generic.go:33] Starting OriginRoleToRBACRoleController controller
8336 start_master.go:783] Started "statefuleset"
8336 stateful_set.go:144] Starting statefulset controller
8336 start_master.go:783] Started "openshift.io/unidling"
8336 core.go:33] NodeController failed parsing cluster CIDR : invalid CIDR address:
8336 core.go:38] NodeController failed parsing service CIDR : invalid CIDR address:
8336 nodecontroller.go:219] Sending events to api server.
8336 taint_controller.go:157] Sending events to api server.
8336 start_master.go:783] Started "node"
8336 start_master.go:783] Started "replicationcontroller"
8336 replication_controller.go:150] Starting RC Manager
8336 start_master.go:783] Started "horizontalpodautoscaling"
8336 horizontal.go:140] Starting HPA Controller
8336 start_master.go:783] Started "cronjob"
8336 cronjob_controller.go:95] Starting CronJob Manager
8336 start_master.go:783] Started "openshift.io/origin-namespace"
8336 shared_informer.go:298] resyncPeriod 120000000000 is smaller than resyncCheckPeriod 600000000000 and the informer has already started. Changing it to 600000000000
8336 start_master.go:783] Started "openshift.io/service-serving-cert"
8336 start_master.go:765] "bootstrapsigner" is skipped
8336 core.go:80] ServiceLoadBalancer controller will not start - no cloud provider configured
8336 start_master.go:780] Skipping "service"
8336 start_master.go:783] Started "daemonset"
8336 daemoncontroller.go:199] Starting Daemon Sets controller manager
8336 start_master.go:783] Started "job"
8336 start_master.go:783] Started "disruption"
8336 disruption.go:269] Starting disruption controller
8336 start_master.go:783] Started "openshift.io/build"
8336 start_master.go:783] Started "openshift.io/deploymenttrigger"
8336 start_master.go:783] Started "openshift.io/image-trigger"
8336 image_trigger_controller.go:214] Starting trigger controller
8336 start_master.go:783] Started "deployment"
8336 deployment_controller.go:147] Starting deployment controller
8336 start_master.go:783] Started "attachdetach"
8336 attach_detach_controller.go:239] Starting Attach Detach Controller
8336 start_master.go:783] Started "openshift.io/deployer"
8336 factory.go:83] Starting deployer controller
8336 start_master.go:783] Started "podgc"
8336 shared_informer.go:298] resyncPeriod 300000000000 is smaller than resyncCheckPeriod 600000000000 and the informer has already started. Changing it to 600000000000
8336 start_master.go:783] Started "namespace"
8336 namespace_controller.go:184] Starting the NamespaceController
8336 start_master.go:783] Started "garbagecollector"
8336 start_master.go:765] "ttl" is skipped
8336 garbagecollector.go:122] Garbage Collector: Initializing
8336 start_master.go:783] Started "openshift.io/sdn"
8336 start_master.go:783] Started "openshift.io/ingress-ip"
8336 start_master.go:783] Started "openshift.io/scheduler"
8336 certificates.go:38] Failed to start certificate controller: open : no such file or directory
8336 start_master.go:780] Skipping "certificatesigningrequests"
8336 start_master.go:783] Started "persistentvolume-binder"
8336 start_master.go:786] Started Origin Controllers
8336 build_controller.go:164] Starting build controller
8336 factory.go:90] Deployer controller caches are synced. Starting workers.
8336 actual_state_of_world.go:473] Failed to set statusUpdateNeeded to needed true because nodeName="sf-docker01.corp.wagerworks.com" does not exist
8336 actual_state_of_world.go:487] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true because nodeName="sf-docker01.corp.wagerworks.com" does not exist
8336 disruption.go:277] Sending events to api server.
8336 buildconfig_controller.go:185] Starting buildconfig controller
8336 factory.go:91] deploymentconfig controller caches are synced. Starting workers.
8336 nodecontroller.go:633] Initializing eviction metric for zone:
8336 nodecontroller.go:956] Missing timestamp for Node sf-docker01.corp.wagerworks.com. Assuming now as a timestamp.
8336 nodecontroller.go:872] NodeController detected that zone is now in state Normal.
8336 taint_controller.go:180] Starting NoExecuteTaintManager
8336 event.go:217] Event(v1.ObjectReference{Kind:"Node", Namespace:"", Name:"sf-docker01.corp.wagerworks.com", UID:"c63997ab-ca2d-11e7-8223-005056bc745e", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RegisteredNode' Node sf-docker01.corp.wagerworks.com event: Registered Node sf-docker01.corp.wagerworks.com in NodeController
8336 garbagecollector.go:127] Garbage Collector: All resource monitors have synced. Proceeding to collect garbage

I1115 17:54:36.935503 8336 logs.go:41] http: TLS handshake error from 10.104.13.31:53313: tls: first record does not look like a TLS handshake
I1115 17:54:43.192745 8336 logs.go:41] http: TLS handshake error from 10.104.13.31:53315: tls: first record does not look like a TLS handshake
I1115 17:57:39.915443 8336 image_gc_manager.go:270] [imageGCManager]: Disk usage on "/dev/mapper/centos-root" (/) is at 86% which is over the high threshold (85%). Trying to free 2386627788 bytes
I1115 17:57:39.942274 8336 image_gc_manager.go:335] [imageGCManager]: Removing image "sha256:328edcd84f1bbf868bc88e4ae37afe421ef19be71890f59b4b2d8ba48414b84d" to free 192503276 bytes
E1115 17:57:39.993833 8336 remote_image.go:124] RemoveImage "sha256:328edcd84f1bbf868bc88e4ae37afe421ef19be71890f59b4b2d8ba48414b84d" from image service failed: rpc error: code = 2 desc = Error response from daemon: {"message":"reference does not exist"}
E1115 17:57:39.993932 8336 kuberuntime_image.go:126] Remove image "sha256:328edcd84f1bbf868bc88e4ae37afe421ef19be71890f59b4b2d8ba48414b84d" failed: rpc error: code = 2 desc = Error response from daemon: {"message":"reference does not exist"}
E1115 17:57:39.993962 8336 kubelet.go:1170] Image garbage collection failed: wanted to free 2386627788, but freed 0 space with errors in image deletion: rpc error: code = 2 desc = Error response from daemon: {"message":"reference does not exist"}
W1115 17:57:40.190936 8336 container_manager_linux.go:750] CPUAccounting not enabled for pid: 8161
W1115 17:57:40.190958 8336 container_manager_linux.go:753] MemoryAccounting not enabled for pid: 8161
W1115 17:57:40.191051 8336 container_manager_linux.go:750] CPUAccounting not enabled for pid: 8336
W1115 17:57:40.191060 8336 container_manager_linux.go:753] MemoryAccounting not enabled for pid: 8336
2017-11-15 18:02:34.998158 I | mvcc: store.index: compact 965
2017-11-15 18:02:35.001035 I | mvcc: finished scheduled compaction at 965 (took 2.093112ms)
I1115 18:02:35.001534 8336 compact.go:159] etcd: compacted rev (965), endpoints ([https://10.104.6.127:4001])
I1115 18:02:40.016879 8336 image_gc_manager.go:270] [imageGCManager]: Disk usage on "/dev/mapper/centos-root" (/) is at 86% which is over the high threshold (85%). Trying to free 2386816204 bytes
I1115 18:02:40.028177 8336 image_gc_manager.go:335] [imageGCManager]: Removing image "sha256:328edcd84f1bbf868bc88e4ae37afe421ef19be71890f59b4b2d8ba48414b84d" to free 192503276 bytes
E1115 18:02:40.032506 8336 remote_image.go:124] RemoveImage "sha256:328edcd84f1bbf868bc88e4ae37afe421ef19be71890f59b4b2d8ba48414b84d" from image service failed: rpc error: code = 2 desc = Error response from daemon: {"message":"reference does not exist"}
E1115 18:02:40.032557 8336 kuberuntime_image.go:126] Remove image "sha256:328edcd84f1bbf868bc88e4ae37afe421ef19be71890f59b4b2d8ba48414b84d" failed: rpc error: code = 2 desc = Error response from daemon: {"message":"reference does not exist"}
E1115 18:02:40.032588 8336 kubelet.go:1170] Image garbage collection failed: wanted to free 2386816204, but freed 0 space with errors in image deletion: rpc error: code = 2 desc = Error response from daemon: {"message":"reference does not exist"}
W1115 18:02:40.191922 8336 container_manager_linux.go:750] CPUAccounting not enabled for pid: 8161
W1115 18:02:40.191942 8336 container_manager_linux.go:753] MemoryAccounting not enabled for pid: 8161
W1115 18:02:40.192016 8336 container_manager_linux.go:750] CPUAccounting not enabled for pid: 8336
W1115 18:02:40.192025 8336 container_manager_linux.go:753] MemoryAccounting not enabled for pid: 8336
I1115 18:03:18.633993 8336 logs.go:41] http: TLS handshake error from 10.104.13.31:58829: tls: first record does not look like a TLS handshake

garyyang6 on 15 Nov 2017

This is using Docker CE 17.11.0-ce-rc3 from docker, just use docker from centos or fedora.

sdodson on 15 Nov 2017

@sdodson I removed Docker CE and installed docker from centos. But, I still got errors.

systemctl start docker
[root@sf-docker01 ~]# docker version
Client:
Version: 1.12.6
API version: 1.24
Package version: docker-1.12.6-61.git85d7426.el7.centos.x86_64
Go version: go1.8.3
Git commit: 85d7426/1.12.6
Built: Tue Oct 24 15:40:21 2017
OS/Arch: linux/amd64

Server:
Version: 1.12.6
API version: 1.24
Package version: docker-1.12.6-61.git85d7426.el7.centos.x86_64
Go version: go1.8.3
Git commit: 85d7426/1.12.6
Built: Tue Oct 24 15:40:21 2017
OS/Arch: linux/amd64

./openshift version
openshift v3.6.1+008f2d5
kubernetes v1.6.1+5115d708d7
etcd 3.2.1

./openshift start
W1115 20:49:23.884487 2453 start_master.go:297] Warning: assetConfig.loggingPublicURL: Invalid value: "": required to view aggregated container logs in the console, master start will continue.
W1115 20:49:23.884586 2453 start_master.go:297] Warning: assetConfig.metricsPublicURL: Invalid value: "": required to view cluster metrics in the console, master start will continue.
W1115 20:49:23.884604 2453 start_master.go:297] Warning: auditConfig.auditFilePath: Required value: audit can not be logged to a separate file, master start will continue.
I1115 20:49:23.897646 2453 plugins.go:101] No cloud provider specified.
E1115 20:49:23.973168 2453 controllermanager.go:337] Server isn't healthy yet. Waiting a little while.
2017-11-15 20:49:23.973253 I | etcdserver/api/v3rpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.104.6.127:4001: getsockopt: connection refused"; Reconnecting to {10.104.6.127:4001 }
2017-11-15 20:49:23.973316 I | etcdserver/api/v3rpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.104.6.127:4001: getsockopt: connection refused"; Reconnecting to {10.104.6.127:4001 }
2017-11-15 20:49:23.973365 I | etcdserver/api/v3rpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.104.6.127:4001: getsockopt: connection refused"; Reconnecting to {10.104.6.127:4001 }
2017-11-15 20:49:23.973409 I | etcdserver/api/v3rpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.104.6.127:4001: getsockopt: connection refused"; Reconnecting to {10.104.6.127:4001 }
2017-11-15 20:49:23.994063 I | etcdserver/api/v3rpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.104.6.127:4001: getsockopt: connection refused"; Reconnecting to {10.104.6.127:4001 }
I1115 20:49:24.106700 2453 start_master.go:529] Starting master on 0.0.0.0:8443 (v3.6.1+008f2d5)
I1115 20:49:24.106722 2453 start_master.go:530] Public master address is https://10.104.6.127:8443
I1115 20:49:24.106750 2453 start_master.go:534] Using images from "openshift/origin-:v3.6.1"
2017-11-15 20:49:24.107078 I | embed: peerTLS: cert = openshift.local.config/master/etcd.server.crt, key = openshift.local.config/master/etcd.server.key, ca = openshift.local.config/master/ca.crt, trusted-ca = , client-cert-auth = true
2017-11-15 20:49:24.108337 I | embed: listening for peers on https://0.0.0.0:7001
2017-11-15 20:49:24.108399 I | embed: listening for client requests on 0.0.0.0:4001
2017-11-15 20:49:24.110047 I | etcdserver/api/v3rpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 10.104.6.127:4001: getsockopt: connection refused"; Reconnecting to {10.104.6.127:4001 }
2017-11-15 20:49:24.290027 I | etcdserver: name = openshift.local
2017-11-15 20:49:24.290056 I | etcdserver: data dir = openshift.local.etcd
2017-11-15 20:49:24.290064 I | etcdserver: member dir = openshift.local.etcd/member
2017-11-15 20:49:24.290070 I | etcdserver: heartbeat = 100ms
2017-11-15 20:49:24.290077 I | etcdserver: election = 1000ms
2017-11-15 20:49:24.290083 I | etcdserver: snapshot count = 100000
2017-11-15 20:49:24.290102 I | etcdserver: advertise client URLs = https://10.104.6.127:4001
2017-11-15 20:49:24.482581 I | etcdserver: restarting member a7340362e2996c30 in cluster cf86d7c1b2833ba9 at commit index 2580
2017-11-15 20:49:24.482787 I | raft: a7340362e2996c30 became follower at term 24
2017-11-15 20:49:24.482809 I | raft: newRaft a7340362e2996c30 [peers: [], term: 24, commit: 2580, applied: 0, lastindex: 2580, lastterm: 24]
2017-11-15 20:49:24.487709 I | mvcc: restore compact to 1399
2017-11-15 20:49:24.491187 W | auth: simple token is not cryptographically signed
2017-11-15 20:49:24.494537 I | etcdserver: starting server... [version: 3.2.1, cluster version: to_be_decided]
2017-11-15 20:49:24.494579 I | embed: ClientTLS: cert = openshift.local.config/master/etcd.server.crt, key = openshift.local.config/master/etcd.server.key, ca = openshift.local.config/master/ca.crt, trusted-ca = , client-cert-auth = true
2017-11-15 20:49:24.516140 I | etcdserver/membership: added member a7340362e2996c30 [https://10.104.6.127:7001] to cluster cf86d7c1b2833ba9
2017-11-15 20:49:24.516323 N | etcdserver/membership: set the initial cluster version to 3.2
2017-11-15 20:49:24.516372 I | etcdserver/api: enabled capabilities for version 3.2
2017-11-15 20:49:24.584251 I | raft: a7340362e2996c30 is starting a new election at term 24
2017-11-15 20:49:24.584349 I | raft: a7340362e2996c30 became candidate at term 25
2017-11-15 20:49:24.584375 I | raft: a7340362e2996c30 received MsgVoteResp from a7340362e2996c30 at term 25
2017-11-15 20:49:24.584394 I | raft: a7340362e2996c30 became leader at term 25
2017-11-15 20:49:24.584406 I | raft: raft.node: a7340362e2996c30 elected leader a7340362e2996c30 at term 25
2017-11-15 20:49:24.585321 I | etcdserver: published {Name:openshift.local ClientURLs:[https://10.104.6.127:4001]} to cluster cf86d7c1b2833ba9
I1115 20:49:24.585384 2453 run.go:85] Started etcd at 10.104.6.127:4001
2017-11-15 20:49:24.586441 I | embed: ready to serve client requests
2017-11-15 20:49:24.586934 I | embed: serving client requests on [::]:4001
2017-11-15 20:49:24.759900 I | etcdserver/api/v3rpc: Failed to dial [::]:4001: connection error: desc = "transport: remote error: tls: bad certificate"; please retry.
2017-11-15 20:49:24.759979 I | etcdserver/api/v3rpc: Failed to dial [::]:4001: connection error: desc = "transport: remote error: tls: bad certificate"; please retry.
2017-11-15 20:49:24.760016 I | etcdserver/api/v3rpc: Failed to dial [::]:4001: connection error: desc = "transport: remote error: tls: bad certificate"; please retry.
2017-11-15 20:49:24.760053 I | etcdserver/api/v3rpc: Failed to dial [::]:4001: connection error: desc = "transport: remote error: tls: bad certificate"; please retry.
2017-11-15 20:49:24.761935 I | etcdserver/api/v3rpc: Failed to dial [::]:4001: connection error: desc = "transport: remote error: tls: bad certificate"; please retry.
2017-11-15 20:49:24.762388 I | etcdserver/api/v3rpc: Failed to dial [::]:4001: connection error: desc = "transport: remote error: tls: bad certificate"; please retry.
2017-11-15 20:49:24.762427 I | etcdserver/api/v3rpc: Failed to dial [::]:4001: connection error: desc = "transport: remote error: tls: bad certificate"; please retry.
2017-11-15 20:49:24.762465 I | etcdserver/api/v3rpc: Failed to dial [::]:4001: connection error: desc = "transport: remote error: tls: bad certificate"; please retry.
I1115 20:49:24.763667 2453 run_components.go:91] Using default project node label selector:
I1115 20:49:24.768432 2453 clusterquotamapping.go:160] Starting ClusterQuotaMappingController controller
I1115 20:49:24.768774 2453 master.go:182] Starting OAuth2 API at /oauth
I1115 20:49:24.768788 2453 master.go:190] Starting Web Console /console/
E1115 20:49:24.776355 2453 reflector.go:201] github.com/openshift/origin/pkg/quota/generated/informers/internalversion/factory.go:45: Failed to list *quota.ClusterResourceQuota: Get https://10.104.6.127:8443/apis/quota.openshift.io/v1/clusterresourcequotas?resourceVersion=0: dial tcp 10.104.6.127:8443: getsockopt: connection refused
E1115 20:49:24.783067 2453 reflector.go:201] github.com/openshift/origin/pkg/authorization/generated/informers/internalversion/factory.go:45: Failed to list *authorization.ClusterPolicyBinding: Get https://10.104.6.127:8443/apis/authorization.openshift.io/v1/clusterpolicybindings?resourceVersion=0: dial tcp 10.104.6.127:8443: getsockopt: connection refused
E1115 20:49:24.783142 2453 reflector.go:201] github.com/openshift/origin/pkg/authorization/generated/informers/internalversion/factory.go:45: Failed to list *authorization.ClusterPolicy: Get https://10.104.6.127:8443/apis/authorization.openshift.io/v1/clusterpolicies?resourceVersion=0: dial tcp 10.104.6.127:8443: getsockopt: connection refused
E1115 20:49:24.783207 2453 reflector.go:201] github.com/openshift/origin/pkg/authorization/generated/informers/internalversion/factory.go:45: Failed to list *authorization.PolicyBinding: Get https://10.104.6.127:8443/apis/authorization.openshift.io/v1/policybindings?resourceVersion=0: dial tcp 10.104.6.127:8443: getsockopt: connection refused
E1115 20:49:24.783282 2453 reflector.go:201] github.com/openshift/origin/pkg/authorization/generated/informers/internalversion/factory.go:45: Failed to list *authorization.Policy: Get https://10.104.6.127:8443/apis/authorization.openshift.io/v1/policies?resourceVersion=0: dial tcp 10.104.6.127:8443: getsockopt: connection refused
W1115 20:49:25.098646 2453 genericapiserver.go:295] Skipping API autoscaling/v2alpha1 because it has no resources.
E1115 20:49:25.239431 2453 controllermanager.go:337] Server isn't healthy yet. Waiting a little while.
W1115 20:49:25.242764 2453 genericapiserver.go:295] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
I1115 20:49:25.495723 2453 openshift_apiserver.go:237] Starting Origin API at /apis/security.openshift.io/v1
I1115 20:49:25.496423 2453 openshift_apiserver.go:237] Starting Origin API at /apis/project.openshift.io/v1
I1115 20:49:25.499347 2453 openshift_apiserver.go:237] Starting Origin API at /apis/image.openshift.io/v1
I1115 20:49:25.501277 2453 openshift_apiserver.go:237] Starting Origin API at /apis/apps.openshift.io/v1
I1115 20:49:25.502507 2453 openshift_apiserver.go:237] Starting Origin API at /apis/template.openshift.io/v1
I1115 20:49:25.509275 2453 openshift_apiserver.go:237] Starting Origin API at /apis/authorization.openshift.io/v1
I1115 20:49:25.512515 2453 openshift_apiserver.go:237] Starting Origin API at /apis/oauth.openshift.io/v1
I1115 20:49:25.722776 2453 openshift_apiserver.go:237] Starting Origin API at /apis/build.openshift.io/v1
I1115 20:49:25.723992 2453 openshift_apiserver.go:237] Starting Origin API at /apis/quota.openshift.io/v1
I1115 20:49:25.726809 2453 openshift_apiserver.go:237] Starting Origin API at /apis/network.openshift.io/v1
I1115 20:49:25.728257 2453 openshift_apiserver.go:237] Starting Origin API at /apis/route.openshift.io/v1
I1115 20:49:25.730177 2453 openshift_apiserver.go:237] Starting Origin API at /apis/user.openshift.io/v1
I1115 20:49:25.929423 2453 openshift_apiserver.go:243] Started Origin API at /oapi/v1
E1115 20:49:26.171211 2453 reflector.go:201] github.com/openshift/origin/pkg/authorization/generated/informers/internalversion/factory.go:45: Failed to list *authorization.Policy: Get https://10.104.6.127:8443/apis/authorization.openshift.io/v1/policies?resourceVersion=0: dial tcp 10.104.6.127:8443: getsockopt: connection refused
E1115 20:49:26.216256 2453 reflector.go:201] github.com/openshift/origin/pkg/authorization/generated/informers/internalversion/factory.go:45: Failed to list *authorization.PolicyBinding: Get https://10.104.6.127:8443/apis/authorization.openshift.io/v1/policybindings?resourceVersion=0: dial tcp 10.104.6.127:8443: getsockopt: connection refused
E1115 20:49:26.216342 2453 reflector.go:201] github.com/openshift/origin/pkg/authorization/generated/informers/internalversion/factory.go:45: Failed to list *authorization.ClusterPolicy: Get https://10.104.6.127:8443/apis/authorization.openshift.io/v1/clusterpolicies?resourceVersion=0: dial tcp 10.104.6.127:8443: getsockopt: connection refused
E1115 20:49:26.216416 2453 reflector.go:201] github.com/openshift/origin/pkg/authorization/generated/informers/internalversion/factory.go:45: Failed to list *authorization.ClusterPolicyBinding: Get https://10.104.6.127:8443/apis/authorization.openshift.io/v1/clusterpolicybindings?resourceVersion=0: dial tcp 10.104.6.127:8443: getsockopt: connection refused
E1115 20:49:26.216482 2453 reflector.go:201] github.com/openshift/origin/pkg/quota/generated/informers/internalversion/factory.go:45: Failed to list *quota.ClusterResourceQuota: Get https://10.104.6.127:8443/apis/quota.openshift.io/v1/clusterresourcequotas?resourceVersion=0: dial tcp 10.104.6.127:8443: getsockopt: connection refused
E1115 20:49:26.295296 2453 controllermanager.go:337] Server isn't healthy yet. Waiting a little while.
I1115 20:49:26.320734 2453 trace.go:76] Trace[1313190296]: "cacher *user.Group: List" (started: 2017-11-15 20:49:24.768539611 +0000 UTC) (total time: 1.552147608s):
Trace[1313190296]: [1.552147608s] [1.552147608s] END
[restful] 2017/11/15 20:49:26 log.go:30: [restful/swagger] listing is available at https://10.104.6.127:8443/swaggerapi
[restful] 2017/11/15 20:49:26 log.go:30: [restful/swagger] https://10.104.6.127:8443/swaggerui/ is mapped to folder /swagger-ui/
I1115 20:49:26.855537 2453 serve.go:86] Serving securely on 0.0.0.0:8443
W1115 20:49:26.888630 2453 lease_endpoint_reconciler.go:176] Resetting endpoints for master service "kubernetes" to [10.104.6.127]
W1115 20:49:26.967408 2453 run_components.go:60] Binding DNS on port 8053 instead of 53, which may not be resolvable from all clients
I1115 20:49:26.967928 2453 logs.go:41] skydns: ready for queries on cluster.local. for tcp4://0.0.0.0:8053 [rcache 0]
I1115 20:49:26.967961 2453 logs.go:41] skydns: ready for queries on cluster.local. for udp4://0.0.0.0:8053 [rcache 0]
E1115 20:49:26.974886 2453 controllermanager.go:337] Server isn't healthy yet. Waiting a little while.
I1115 20:49:27.069423 2453 run_components.go:86] DNS listening at 0.0.0.0:8053
I1115 20:49:27.945384 2453 docker.go:364] Connecting to docker on unix:///var/run/docker.sock
I1115 20:49:27.945411 2453 docker.go:384] Start docker client with request timeout=2m0s
W1115 20:49:27.962367 2453 cni.go:157] Unable to update cni config: No networks found in /etc/cni/net.d
I1115 20:49:28.019354 2453 start_master.go:715] Started serviceaccount-token controller
I1115 20:49:28.071419 2453 node_config.go:367] DNS Bind to 10.104.6.127:53
I1115 20:49:28.071449 2453 start_node.go:345] Starting node sf-docker01.corp.wagerworks.com (v3.6.1+008f2d5)
I1115 20:49:28.073226 2453 start_node.go:354] Connecting to API server https://10.104.6.127:8443
I1115 20:49:28.073540 2453 docker.go:364] Connecting to docker on unix:///var/run/docker.sock
I1115 20:49:28.073553 2453 docker.go:384] Start docker client with request timeout=2m0s
I1115 20:49:28.112000 2453 start_master.go:783] Started "replicationcontroller"
I1115 20:49:28.112167 2453 replication_controller.go:150] Starting RC Manager
I1115 20:49:28.125219 2453 node.go:134] Connecting to Docker at unix:///var/run/docker.sock
I1115 20:49:28.166274 2453 feature_gate.go:144] feature gates: map[]
I1115 20:49:28.167787 2453 manager.go:143] cAdvisor running in container: "/"
W1115 20:49:28.194721 2453 manager.go:151] unable to connect to Rkt api service: rkt: cannot tcp Dial rkt api service: dial tcp [::1]:15441: getsockopt: connection refused
I1115 20:49:28.223925 2453 start_master.go:783] Started "openshift.io/unidling"
I1115 20:49:28.228213 2453 node.go:348] Using iptables Proxier.
W1115 20:49:28.235445 2453 proxier.go:314] clusterCIDR not specified, unable to distinguish between internal and external traffic
I1115 20:49:28.235478 2453 node.go:380] Tearing down userspace rules.
I1115 20:49:28.251378 2453 fs.go:117] Filesystem partitions: map[/dev/mapper/centos-root:{mountpoint:/ major:253 minor:0 fsType:xfs blockSize:0} /dev/sda1:{mountpoint:/boot major:8 minor:1 fsType:xfs blockSize:0}]
I1115 20:49:28.254492 2453 manager.go:198] Machine: {NumCores:1 CpuFrequency:2533423 MemoryCapacity:16658931712 MachineID:197dcaf983ac43d49bc33a715706d364 SystemUUID:423C41D5-41C2-B4C5-FB04-E3E643ABDDC6 BootID:7b767de6-9ef5-4d25-ab53-1fc9eb044482 Filesystems:[{Device:/dev/mapper/centos-root DeviceMajor:253 DeviceMinor:0 Capacity:47724642304 Type:vfs Inodes:37744848 HasInodes:true} {Device:/dev/sda1 DeviceMajor:8 DeviceMinor:1 Capacity:520794112 Type:vfs Inodes:512000 HasInodes:true}] DiskMap:map[253:0:{Name:dm-0 Major:253 Minor:0 Size:47747956736 Scheduler:none} 253:1:{Name:dm-1 Major:253 Minor:1 Size:5368709120 Scheduler:none} 253:2:{Name:dm-2 Major:253 Minor:2 Size:107374182400 Scheduler:none} 2:0:{Name:fd0 Major:2 Minor:0 Size:4096 Scheduler:deadline} 8:0:{Name:sda Major:8 Minor:0 Size:53687091200 Scheduler:deadline}] NetworkDevices:[{Name:ens160 MacAddress:00:50:56:bc:74:5e Speed:10000 Mtu:1500} {Name:virbr0 MacAddress:52:54:00:4c:ea:cb Speed:0 Mtu:1500} {Name:virbr0-nic MacAddress:52:54:00:4c:ea:cb Speed:0 Mtu:1500} {Name:virbr1 MacAddress:52:54:00:fa:ae:1e Speed:0 Mtu:1500} {Name:virbr1-nic MacAddress:52:54:00:fa:ae:1e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:17179402240 Cores:[{Id:0 Threads:[0] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]}] Caches:[{Size:12582912 Type:Unified Level:3}]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
I1115 20:49:28.258353 2453 start_master.go:783] Started "disruption"
I1115 20:49:28.258556 2453 disruption.go:269] Starting disruption controller
I1115 20:49:28.292696 2453 manager.go:204] Version: {KernelVersion:3.10.0-693.5.2.el7.x86_64 ContainerOsVersion:CentOS Linux 7 (Core) DockerVersion:1.12.6 DockerAPIVersion:1.24 CadvisorVersion: CadvisorRevision:}
I1115 20:49:28.293489 2453 server.go:509] --cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to /
W1115 20:49:28.308661 2453 container_manager_linux.go:217] Running with swap on is not supported, please disable swap! This will be a fatal error by default starting in K8s v1.6! In the meantime, you can opt-in to making this a fatal error by enabling --experimental-fail-swap-on.
I1115 20:49:28.309221 2453 container_manager_linux.go:244] container manager verified user specified cgroup-root exists: /
I1115 20:49:28.309243 2453 container_manager_linux.go:249] Creating Container Manager object based on Node Config: {RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:docker CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:systemd ProtectKernelDefaults:false EnableCRI:true NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:memory.available Operator:LessThan Value:{Quantity:100Mi Percentage:0} GracePeriod:0s MinReclaim:}]} ExperimentalQOSReserved:map[]}
I1115 20:49:28.309447 2453 kubelet.go:265] Watching apiserver
W1115 20:49:28.353031 2453 kubelet_network.go:70] Hairpin mode set to "promiscuous-bridge" but kubenet is not enabled, falling back to "hairpin-veth"
I1115 20:49:28.353075 2453 kubelet.go:494] Hairpin mode set to "hairpin-veth"
W1115 20:49:28.367684 2453 cni.go:157] Unable to update cni config: No networks found in /etc/cni/net.d
I1115 20:49:28.407580 2453 start_master.go:783] Started "persistentvolume-binder"
I1115 20:49:28.521120 2453 docker_service.go:184] Docker cri networking managed by kubernetes.io/no-op
I1115 20:49:28.527861 2453 start_master.go:783] Started "deployment"
W1115 20:49:28.527899 2453 start_master.go:765] "ttl" is skipped
I1115 20:49:28.528134 2453 deployment_controller.go:147] Starting deployment controller
I1115 20:49:28.573558 2453 docker_service.go:201] Setting cgroupDriver to systemd
I1115 20:49:28.616148 2453 start_master.go:783] Started "daemonset"
I1115 20:49:28.616409 2453 daemoncontroller.go:199] Starting Daemon Sets controller manager
I1115 20:49:28.668712 2453 remote_runtime.go:41] Connecting to runtime service /var/run/dockershim.sock
I1115 20:49:28.699230 2453 node.go:480] Started Kubernetes Proxy on 0.0.0.0
I1115 20:49:28.731034 2453 node.go:316] Starting DNS on 10.104.6.127:53
I1115 20:49:28.731317 2453 logs.go:41] skydns: ready for queries on cluster.local. for tcp://10.104.6.127:53 [rcache 0]
I1115 20:49:28.731342 2453 logs.go:41] skydns: ready for queries on cluster.local. for udp://10.104.6.127:53 [rcache 0]
I1115 20:49:28.738931 2453 kuberuntime_manager.go:171] Container runtime docker initialized, version: 1.12.6, apiVersion: 1.24.0
I1115 20:49:28.768239 2453 server.go:869] Started kubelet v1.6.1+5115d708d7
I1115 20:49:28.768281 2453 server.go:127] Starting to listen on 0.0.0.0:10250
I1115 20:49:28.769347 2453 server.go:298] Adding debug handlers to kubelet server.
E1115 20:49:28.772394 2453 kubelet.go:1170] Image garbage collection failed: unable to find data for container /
I1115 20:49:28.883221 2453 start_master.go:783] Started "openshift.io/build"
I1115 20:49:28.942406 2453 start_master.go:783] Started "openshift.io/build-config-change"
I1115 20:49:28.974502 2453 start_master.go:783] Started "endpoint"
I1115 20:49:29.062148 2453 start_master.go:783] Started "openshift.io/deploymentconfig"
I1115 20:49:29.062354 2453 factory.go:84] Starting deploymentconfig controller
I1115 20:49:29.090761 2453 fs_resource_analyzer.go:66] Starting FS ResourceAnalyzer
I1115 20:49:29.090831 2453 status_manager.go:140] Starting to sync pod status with apiserver
I1115 20:49:29.090865 2453 kubelet.go:1749] Starting kubelet main sync loop.
I1115 20:49:29.090893 2453 kubelet.go:1760] skipping pod synchronization - [container runtime is down PLEG is not healthy: pleg was last seen active 2562047h47m16.854775807s ago; threshold is 3m0s]
W1115 20:49:29.091197 2453 container_manager_linux.go:750] CPUAccounting not enabled for pid: 2155
W1115 20:49:29.091209 2453 container_manager_linux.go:753] MemoryAccounting not enabled for pid: 2155
W1115 20:49:29.091322 2453 container_manager_linux.go:750] CPUAccounting not enabled for pid: 2453
W1115 20:49:29.091336 2453 container_manager_linux.go:753] MemoryAccounting not enabled for pid: 2453
I1115 20:49:29.091363 2453 volume_manager.go:249] Starting Kubelet Volume Manager
I1115 20:49:29.183621 2453 start_master.go:783] Started "resourcequota"
I1115 20:49:29.183842 2453 resource_quota_controller.go:240] Starting resource quota controller
E1115 20:49:29.188112 2453 factory.go:330] devicemapper filesystem stats will not be reported: usage of thin_ls is disabled to preserve iops
I1115 20:49:29.188359 2453 factory.go:342] Registering Docker factory
W1115 20:49:29.188378 2453 manager.go:247] Registration of the rkt container factory failed: unable to communicate with Rkt api service: rkt: cannot tcp Dial rkt api service: dial tcp [::1]:15441: getsockopt: connection refused
I1115 20:49:29.188386 2453 factory.go:54] Registering systemd factory
I1115 20:49:29.188553 2453 factory.go:86] Registering Raw factory
I1115 20:49:29.188715 2453 manager.go:1106] Started watching for new ooms in manager
I1115 20:49:29.198961 2453 kubelet_node_status.go:253] Setting node annotation to enable volume controller attach/detach
I1115 20:49:29.220173 2453 oomparser.go:185] oomparser using systemd
I1115 20:49:29.220867 2453 manager.go:288] Starting recovery of all containers
I1115 20:49:29.226180 2453 manager.go:293] Recovery completed
I1115 20:49:29.287009 2453 kubelet_node_status.go:77] Attempting to register node sf-docker01.corp.wagerworks.com
W1115 20:49:29.288875 2453 shared_informer.go:298] resyncPeriod 300000000000 is smaller than resyncCheckPeriod 600000000000 and the informer has already started. Changing it to 600000000000
I1115 20:49:29.288916 2453 start_master.go:783] Started "namespace"
I1115 20:49:29.289071 2453 namespace_controller.go:184] Starting the NamespaceController
I1115 20:49:29.356021 2453 kubelet_node_status.go:128] Node sf-docker01.corp.wagerworks.com was previously registered
I1115 20:49:29.356060 2453 kubelet_node_status.go:80] Successfully registered node sf-docker01.corp.wagerworks.com
E1115 20:49:29.388446 2453 certificates.go:38] Failed to start certificate controller: open : no such file or directory
W1115 20:49:29.388505 2453 start_master.go:780] Skipping "certificatesigningrequests"
I1115 20:49:29.511505 2453 start_master.go:783] Started "attachdetach"
I1115 20:49:29.511710 2453 attach_detach_controller.go:239] Starting Attach Detach Controller
I1115 20:49:29.551996 2453 start_master.go:783] Started "openshift.io/sdn"
I1115 20:49:29.624623 2453 start_master.go:783] Started "openshift.io/ingress-ip"
I1115 20:49:29.624654 2453 start_master.go:783] Started "openshift.io/cluster-quota-mapping"
I1115 20:49:29.624793 2453 clusterquotamapping.go:160] Starting ClusterQuotaMappingController controller
I1115 20:49:29.638505 2453 start_master.go:783] Started "statefuleset"
W1115 20:49:29.638536 2453 core.go:80] ServiceLoadBalancer controller will not start - no cloud provider configured
W1115 20:49:29.638544 2453 start_master.go:780] Skipping "service"
I1115 20:49:29.638729 2453 stateful_set.go:144] Starting statefulset controller
I1115 20:49:29.686031 2453 start_master.go:783] Started "garbagecollector"
I1115 20:49:29.686191 2453 garbagecollector.go:122] Garbage Collector: Initializing
I1115 20:49:29.709348 2453 start_master.go:783] Started "openshift.io/deployer"
I1115 20:49:29.709496 2453 factory.go:83] Starting deployer controller
I1115 20:49:29.737101 2453 start_master.go:783] Started "openshift.io/image-trigger"
I1115 20:49:29.737355 2453 start_master.go:783] Started "openshift.io/scheduler"
I1115 20:49:29.737575 2453 image_trigger_controller.go:214] Starting trigger controller
I1115 20:49:29.752648 2453 start_master.go:783] Started "replicaset"
W1115 20:49:29.752673 2453 start_master.go:765] "tokencleaner" is skipped
I1115 20:49:29.752850 2453 replica_set.go:155] Starting ReplicaSet controller
E1115 20:49:29.767341 2453 util.go:45] Metric for serviceaccount_controller already registered
I1115 20:49:29.767416 2453 start_master.go:783] Started "serviceaccount"
I1115 20:49:29.767534 2453 serviceaccounts_controller.go:122] Starting ServiceAccount controller
I1115 20:49:29.816114 2453 start_master.go:783] Started "openshift.io/cluster-quota-reconciliation"
I1115 20:49:29.816888 2453 start_master.go:783] Started "openshift.io/origin-to-rbac"
I1115 20:49:29.817037 2453 generic.go:33] Starting OriginRoleBindingToRBACRoleBindingController controller
I1115 20:49:29.817250 2453 generic.go:33] Starting OriginClusterRoleToRBACClusterRoleController controller
I1115 20:49:29.817283 2453 generic.go:33] Starting OriginClusterRoleBindingToRBACClusterRoleBindingController controller
I1115 20:49:29.817310 2453 generic.go:33] Starting OriginRoleToRBACRoleController controller
I1115 20:49:29.831098 2453 start_master.go:783] Started "cronjob"
I1115 20:49:29.831261 2453 cronjob_controller.go:95] Starting CronJob Manager
I1115 20:49:29.847992 2453 start_master.go:783] Started "horizontalpodautoscaling"
I1115 20:49:29.848154 2453 horizontal.go:140] Starting HPA Controller
I1115 20:49:29.861819 2453 start_master.go:783] Started "openshift.io/origin-namespace"
W1115 20:49:29.861846 2453 start_master.go:765] "openshift.io/templateinstance" is skipped
I1115 20:49:29.875529 2453 start_master.go:783] Started "podgc"
I1115 20:49:29.888851 2453 start_master.go:783] Started "job"
I1115 20:49:29.976549 2453 start_master.go:783] Started "openshift.io/serviceaccount-pull-secrets"
W1115 20:49:30.063183 2453 shared_informer.go:298] resyncPeriod 120000000000 is smaller than resyncCheckPeriod 600000000000 and the informer has already started. Changing it to 600000000000
I1115 20:49:30.100731 2453 start_master.go:783] Started "openshift.io/service-serving-cert"
I1115 20:49:30.114005 2453 start_master.go:783] Started "openshift.io/deploymenttrigger"
I1115 20:49:30.126805 2453 imagestream_controller.go:65] Starting image stream controller
I1115 20:49:30.137205 2453 start_master.go:783] Started "openshift.io/image-import"
W1115 20:49:30.137226 2453 start_master.go:765] "bootstrapsigner" is skipped
W1115 20:49:30.137238 2453 core.go:33] NodeController failed parsing cluster CIDR : invalid CIDR address:
W1115 20:49:30.137250 2453 core.go:38] NodeController failed parsing service CIDR : invalid CIDR address:
I1115 20:49:30.137393 2453 scheduled_image_controller.go:59] Starting scheduled import controller
I1115 20:49:30.149650 2453 nodecontroller.go:219] Sending events to api server.
I1115 20:49:30.149802 2453 taint_controller.go:157] Sending events to api server.
I1115 20:49:30.149874 2453 start_master.go:783] Started "node"
I1115 20:49:30.149885 2453 start_master.go:786] Started Origin Controllers
E1115 20:49:30.246118 2453 actual_state_of_world.go:473] Failed to set statusUpdateNeeded to needed true because nodeName="sf-docker01.corp.wagerworks.com" does not exist
E1115 20:49:30.246143 2453 actual_state_of_world.go:487] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true because nodeName="sf-docker01.corp.wagerworks.com" does not exist
I1115 20:49:30.246380 2453 buildconfig_controller.go:185] Starting buildconfig controller
I1115 20:49:30.249184 2453 factory.go:90] Deployer controller caches are synced. Starting workers.
I1115 20:49:30.267426 2453 factory.go:91] deploymentconfig controller caches are synced. Starting workers.
I1115 20:49:30.268364 2453 disruption.go:277] Sending events to api server.
I1115 20:49:30.268591 2453 nodecontroller.go:633] Initializing eviction metric for zone:
W1115 20:49:30.268662 2453 nodecontroller.go:956] Missing timestamp for Node sf-docker01.corp.wagerworks.com. Assuming now as a timestamp.
I1115 20:49:30.268697 2453 nodecontroller.go:872] NodeController detected that zone is now in state Normal.
I1115 20:49:30.269146 2453 taint_controller.go:180] Starting NoExecuteTaintManager
I1115 20:49:30.269304 2453 event.go:217] Event(v1.ObjectReference{Kind:"Node", Namespace:"", Name:"sf-docker01.corp.wagerworks.com", UID:"c63997ab-ca2d-11e7-8223-005056bc745e", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RegisteredNode' Node sf-docker01.corp.wagerworks.com event: Registered Node sf-docker01.corp.wagerworks.com in NodeController
I1115 20:49:30.284372 2453 build_controller.go:164] Starting build controller
I1115 20:49:30.290763 2453 garbagecollector.go:127] Garbage Collector: All resource monitors have synced. Proceeding to collect garbage
I1115 20:50:35.919083 2453 logs.go:41] http: TLS handshake error from 10.104.13.31:58806: tls: first record does not look like a TLS handshake
I1115 20:50:38.132550 2453 logs.go:41] http: TLS handshake error from 10.104.13.31:58807: tls: first record does not look like a TLS handshake

garyyang6 on 15 Nov 2017

This worked for me,

cat << EOF > /etc/docker/daemon.json
→ {
→  "exec-opts": ["native.cgroupdriver=systemd"]
→ }
→ EOF

restarted docker and ran
sudo ./openshift start

Chandru-B on 17 Mar 2018

👍13

Was this page helpful?

0 / 5 - 0 ratings

Related issues

privileged parameter is not taken into account for DeploymentConfig

akram · 3Comments

RBAC diagnostics output is not useful under cluster role aggregation

enj · 4Comments

MethodNotAllowed with Generic web hook

theone4ever · 3Comments

Integrated docker-registry.default.svc:5000 is not working as expected

rkrmishra · 4Comments

Cannot import images from private docker registry - importing images from registry "localhost:5000" is forbidden

davivd · 4Comments