Minikube: hello-minikube: Warning FailedCreatePodSandBox kubelet, minikube Failed create pod sandbox.

Created on 20 Sep 2018 · 10Comments · Source: kubernetes/minikube

I use the following command to build deployment

kubectl run hello-minikube --image=k8s.gcr.io/echoserver:1.10 --port=8080

I described the pod and saw
Normal SuccessfulMountVolume 12s kubelet, minikube MountVolume.SetUp succeeded for volume "default-token-mxld5"
Warning FailedCreatePodSandBox kubelet, minikube Failed create pod sandbox.

How can I solve this problem? Thank you

causfirewall-or-proxy causpoor-internet ckube-addon-manager kinbug

Source

matrixoneken

Most helpful comment

I try the following command and it works!
minikube ssh \ "docker pull anjia0532/google-containers.pause-amd64:3.1 docker tag anjia0532/google-containers.pause-amd64:3.1 k8s.gcr.io/pause-amd64:3.1"

liboyue23 on 5 Oct 2018

👍2 ❤1 🎉1 😄1

All 10 comments

Sorry to hear about that. Do you mind sharing more information about your environment so that it can be replicated? Can you attach the full output for:

minikube ssh "journalctl -u kubelet"
kubectl describe pod hello-minikube
minikube logs

Also, please mention:

The "minikube start" command line used
Host operating system
VM driver
Version of minikube

Thanks!

tstromberg on 20 Sep 2018

I run minikube version: v0.25.0 on Chinese win 10 pro with irtualBox 5.2.8 r121009

matrixoneken on 21 Sep 2018

kubectl describe pod hello-minikube
Name: hello-minikube-cbf7c5d59-q9c2g
Namespace: default
Node: minikube/192.168.99.101
Start Time: Fri, 21 Sep 2018 10:53:40 +0800
Labels: pod-template-hash=769371815
run=hello-minikube
Annotations:
Status: Pending
IP:
Controlled By: ReplicaSet/hello-minikube-cbf7c5d59
Containers:
hello-minikube:
Container ID:
Image: k8s.gcr.io/echoserver:1.10
Image ID:
Port: 8080/TCP
Host Port: 0/TCP
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Environment:
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-ww895 (ro)
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
default-token-ww895:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-ww895
Optional: false
QoS Class: BestEffort
Node-Selectors:
Tolerations:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 10s default-scheduler Successfully assigned hello-minikube-cbf7c5d59-q9c2g to minikube
Normal SuccessfulMountVolume 10s kubelet, minikube MountVolume.SetUp succeeded for volume "default-token-ww895"
Warning FailedCreatePodSandBox kubelet, minikube Failed create pod sandbox.

matrixoneken on 21 Sep 2018

minikube logs
-- Logs begin at Fri 2018-09-21 02:50:51 UTC, end at Fri Sep 21 02:51:44 minikube systemd[1]: Starting Localkube...
Sep 21 02:51:44 minikube localkube[3049]: data dir = /var/lib/l Sep 21 02:51:44 minikube localkube[3049]: member dir = Sep 21 02:51:44 minikube localkube[3049]: heartbeat = Sep 21 02:51:44 minikube localkube[3049]: election = 1000ms
Sep 21 02:51:49 minikube localkube[3049]: I0921 02:51:49.668755 Sep 21 02:51:49 minikube localkube[3049]: I0921 02:51:49.668907 Sep 21 02:51:49 minikube localkube[3049]: I0921 02:51:49.669218 Sep 21 02:51:49 minikube localkube[3049]: I0921 02:51:49.669341 Sep 21 02:51:49 minikube localkube[3049]: I0921 02:51:49.669580 Sep 21 02:51:49 minikube localkube[3049]: I0921 02:51:49.669220 Sep 21 02:51:49 minikube localkube[3049]: I0921 02:51:49.670969 Sep 21 02:51:49 minikube localkube[3049]: I0921 02:51:49.671151 Sep 21 02:51:49 minikube localkube[3049]: I0921 02:51:49.673937 Sep 21 02:51:49 minikube localkube[3049]: I0921 02:51:49.673977 Sep 21 02:51:49 minikube localkube[3049]: I0921 02:51:49.703073 Sep 21 02:51:49 minikube localkube[3049]: [+]etcd ok
Sep 21 02:51:49 minikube localkube[3049]: [+]poststarthook/gene Sep 21 02:51:49 minikube localkube[3049]: [+]poststarthook/star Sep 21 02:51:49 minikube localkube[3049]: [+]poststarthook/star Sep 21 02:51:49 minikube localkube[3049]: [-]poststarthook/boot Sep 21 02:51:49 minikube localkube[3049]: [-]poststarthook/ca-registration Sep 21 02:51:49 minikube localkube[3049]: [+]poststarthook/star Sep 21 02:51:49 minikube localkube[3049]: [+]poststarthook/star Sep 21 02:51:49 minikube localkube[3049]: [+]poststarthook/apis Sep 21 02:51:49 minikube localkube[3049]: [+]poststarthook/apis Sep 21 02:51:49 minikube localkube[3049]: [+]poststarthook/apis Sep 21 02:51:49 minikube localkube[3049]: [+]poststarthook/kube Sep 21 02:51:49 minikube localkube[3049]: [-]autoregister-completion Sep 21 02:51:49 minikube localkube[3049]: healthz check Sep 21 02:51:49 minikube localkube[3049]: I0921 02:51:49.769196 Sep 21 02:51:49 minikube localkube[3049]: I0921 02:51:49.770001 Sep 21 02:51:49 minikube localkube[3049]: I0921 02:51:49.771986 Sep 21 02:51:49 minikube localkube[3049]: I0921 02:51:49.772160 Sep 21 02:51:49 minikube localkube[3049]: I0921 02:51:49.772267 Sep 21 02:51:49 minikube localkube[3049]: I0921 02:51:49.872494 Sep 21 02:51:50 minikube localkube[3049]: I0921 02:51:50.420997 Sep 21 02:51:50 minikube localkube[3049]: I0921 02:51:50.429173 Sep 21 02:51:50 minikube localkube[3049]: [+]etcd ok
Sep 21 02:51:50 minikube localkube[3049]: [+]poststarthook/gene Sep 21 02:51:50 minikube localkube[3049]: [+]poststarthook/star Sep 21 02:51:50 minikube localkube[3049]: [+]poststarthook/star Sep 21 02:51:50 minikube localkube[3049]: [-]poststarthook/boot Sep 21 02:51:50 minikube localkube[3049]: [-]poststarthook/ca-registration Sep 21 02:51:50 minikube localkube[3049]: [+]poststarthook/star Sep 21 02:51:50 minikube localkube[3049]: [+]poststarthook/star Sep 21 02:51:50 minikube localkube[3049]: [+]poststarthook/apis Sep 21 02:51:50 minikube localkube[3049]: [+]poststarthook/apis Sep 21 02:51:50 minikube localkube[3049]: [+]poststarthook/apis Sep 21 02:51:50 minikube localkube[3049]: [+]poststarthook/kube Sep 21 02:51:50 minikube localkube[3049]: [-]autoregister-completion Sep 21 02:51:50 minikube localkube[3049]: healthz check Sep 21 02:51:50 minikube localkube[3049]: I0921 02:51:50.889431 Sep 21 02:51:51 minikube localkube[3049]: I0921 02:51:51.422128 Sep 21 02:51:51 minikube localkube[3049]: I0921 02:51:51.431647 Sep 21 02:51:51 minikube localkube[3049]: apiserver is Sep 21 02:51:51 minikube localkube[3049]: Starting controller-manager...
Sep 21 02:51:55 minikube localkube[3049]: E0921 02:51:55.823241 Sep 21 02:51:56 minikube localkube[3049]: I0921 02:51:56.618627 Sep 21 02:51:56 minikube localkube[3049]: I0921 02:51:56.719125 Sep 21 02:51:58 minikube localkube[3049]: I0921 02:51:58.923536 Sep 21 02:51:58 minikube localkube[3049]: I0921 02:51:58.923592 Sep 21 02:51:59 minikube localkube[3049]: I0921 02:51:59.020092 Sep 21 02:51:59 minikube localkube[3049]: W0921 02:51:59.020140 Sep 21 02:51:59 minikube localkube[3049]: I0921 02:51:59.020188 Sep 21 02:51:59 minikube localkube[3049]: I0921 02:51:59.020471 Sep 21 02:52:09 minikube localkube[3049]: I0921 02:52:09.020558 Sep 21 02:52:14 minikube localkube[3049]: E0921 02:52:14.147049 Sep 21 02:52:14 minikube localkube[3049]: E0921 02:52:14.147107 Sep 21 02:52:14 minikube localkube[3049]: E0921 02:52:14.147118 Sep 21 02:52:14 minikube localkube[3049]: E0921 02:52:14.147179 Sep 21 02:52:30 minikube localkube[3049]: E0921 02:52:30.185653 Sep 21 02:52:30 minikube localkube[3049]: E0921 02:52:30.185933 Sep 21 02:52:30 minikube localkube[3049]: E0921 02:52:30.186072 Sep 21 02:52:30 minikube localkube[3049]: E0921 02:52:30.186378 Sep 21 02:52:46 minikube localkube[3049]: E0921 02:52:46.220762 Sep 21 02:52:46 minikube localkube[3049]: E0921 02:52:46.221061 Sep 21 02:52:46 minikube localkube[3049]: E0921 02:52:46.221103 Sep 21 02:52:46 minikube localkube[3049]: E0921 02:52:46.221299 Sep 21 02:52:46 minikube localkube[3049]: apply entries Sep 21 02:52:46 minikube localkube[3049]: avoid queries Sep 21 02:52:54 minikube localkube[3049]: E0921 02:52:54.689179 Sep 21 02:53:02 minikube localkube[3049]: E0921 02:53:02.258994 Sep 21 02:53:02 minikube localkube[3049]: E0921 02:53:02.259028 Sep 21 02:53:02 minikube localkube[3049]: E0921 02:53:02.259039 Sep 21 02:53:02 minikube localkube[3049]: E0921 02:53:02.259080 Sep 21 02:53:18 minikube localkube[3049]: E0921 02:53:18.299997 Sep 21 02:53:18 minikube localkube[3049]: E0921 02:53:18.300030 Sep 21 02:53:18 minikube localkube[3049]: E0921 02:53:18.300040 Sep 21 02:53:18 minikube localkube[3049]: E0921 02:53:18.300081 Sep 21 02:53:34 minikube localkube[3049]: E0921 02:53:34.342080 Sep 21 02:53:34 minikube localkube[3049]: E0921 02:53:34.342110 Sep 21 02:53:34 minikube localkube[3049]: E0921 02:53:34.342120 Sep 21 02:53:34 minikube localkube[3049]: E0921 02:53:34.342164 Sep 21 02:53:39 minikube localkube[3049]: I0921 02:53:39.705003 Sep 21 02:53:39 minikube localkube[3049]: I0921 02:53:39.708838 Sep 21 02:53:39 minikube localkube[3049]: apply entries Sep 21 02:53:39 minikube localkube[3049]: avoid queries Sep 21 02:53:39 minikube localkube[3049]: I0921 02:53:39.948152 Sep 21 02:53:40 minikube localkube[3049]: apply entries Sep 21 02:53:40 minikube localkube[3049]: avoid queries Sep 21 02:53:40 minikube localkube[3049]: I0921 02:53:40.434071 Sep 21 02:53:40 minikube localkube[3049]: I0921 02:53:40.467104 Sep 21 02:53:40 minikube localkube[3049]: Trace[1158976444]: Sep 21 02:53:40 minikube localkube[3049]: I0921 02:53:40.571396 Sep 21 02:53:50 minikube localkube[3049]: E0921 02:53:50.383938 Sep 21 02:53:50 minikube localkube[3049]: E0921 02:53:50.383970 Sep 21 02:53:50 minikube localkube[3049]: E0921 02:53:50.383985 Sep 21 02:53:50 minikube localkube[3049]: E0921 02:53:50.384032 Sep 21 02:53:54 minikube localkube[3049]: E0921 02:53:54.689375 Sep 21 02:53:56 minikube localkube[3049]: E0921 02:53:56.081450 Sep 21 02:53:56 minikube localkube[3049]: E0921 02:53:56.081477 Sep 21 02:53:56 minikube localkube[3049]: E0921 02:53:56.081487 Sep 21 02:53:56 minikube localkube[3049]: E0921 02:53:56.081524 Sep 21 02:54:06 minikube localkube[3049]: E0921 02:54:06.418512 Sep 21 02:54:06 minikube localkube[3049]: E0921 02:54:06.418544 Sep 21 02:54:06 minikube localkube[3049]: E0921 02:54:06.418554 Sep 21 02:54:06 minikube localkube[3049]: E0921 02:54:06.418593 Sep 21 02:54:22 minikube localkube[3049]: E0921 02:54:22.452538 Sep 21 02:54:22 minikube localkube[3049]: E0921 02:54:22.452568 Sep 21 02:54:22 minikube localkube[3049]: E0921 02:54:22.452578 Sep 21 02:54:22 minikube localkube[3049]: E0921 02:54:22.452616 Sep 21 02:54:25 minikube localkube[3049]: E0921 02:54:25.129326 Sep 21 02:54:25 minikube localkube[3049]: E0921 02:54:25.129359 Sep 21 02:54:25 minikube localkube[3049]: E0921 02:54:25.129371 Sep 21 02:54:25 minikube localkube[3049]: E0921 02:54:25.129411 Sep 21 02:54:38 minikube localkube[3049]: E0921 02:54:38.496881 Sep 21 02:54:38 minikube localkube[3049]: E0921 02:54:38.496941 Sep 21 02:54:38 minikube localkube[3049]: E0921 02:54:38.496952 Sep 21 02:54:38 minikube localkube[3049]: E0921 02:54:38.496991 Sep 21 02:54:38 minikube localkube[3049]: apply entries Sep 21 02:54:38 minikube localkube[3049]: avoid queries Sep 21 02:54:54 minikube localkube[3049]: E0921 02:54:54.538197 Sep 21 02:54:54 minikube localkube[3049]: E0921 02:54:54.538228 Sep 21 02:54:54 minikube localkube[3049]: E0921 02:54:54.538236 Sep 21 02:54:54 minikube localkube[3049]: E0921 02:54:54.538273 Sep 21 02:54:54 minikube localkube[3049]: E0921 02:54:54.690087 Sep 21 02:54:54 minikube localkube[3049]: apply entries Sep 21 02:54:54 minikube localkube[3049]: avoid queries Sep 21 02:54:56 minikube localkube[3049]: E0921 02:54:56.132724 Sep 21 02:54:56 minikube localkube[3049]: E0921 02:54:56.132755 Sep 21 02:54:56 minikube localkube[3049]: E0921 02:54:56.132766 Sep 21 02:54:56 minikube localkube[3049]: E0921 02:54:56.132804 2018-09-21 02:55:00 UTC. --
/> peers on http://localhost:2380
client requests on localhost:2379
ocalkube/etcd
/var/lib/localkube/etcd/member
100ms
/> = 10000
URLs = http://localhost:2379
peer URLs = http://localhost:2380
= default=http://localhost:2380
8e9e05c52164694d in cluster cdf818194e3a8c32
became follower at term 0
[peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
became follower at term 1
[version: 3.1.10, cluster version: to_be_decided]
8e9e05c52164694d [http://localhost:2380] to cluster cdf818194e3a8c32
is starting a new election at term 1
became candidate at term 2
received MsgVoteResp from 8e9e05c52164694d at term 2
became leader at term 2
elected leader 8e9e05c52164694d at term 2
ClientURLs:[http://localhost:2379]} to cluster cdf818194e3a8c32
3049 etcd.go:58] Etcd server is ready
initial cluster version to 3.1
client requests
ip address: 10.0.2.15
client requests on 127.0.0.1:2379, this is strongly discouraged!
/> apiserver to be healthy...
3049 server.go:121] Version: v1.9.0
3049 authentication.go:378] AnonymousAuth is not allowed with the AllowAll authorizer. Resetting AnonymousAuth to false. You should use a different authorizer
cluster version to 3.1
for version 3.1
3049 master.go:225] Using reconciler: master-count
02:51:45 log.go:33: [restful/swagger] listing is available at https://10.0.2.15:8443/swaggerapi
02:51:45 log.go:33: [restful/swagger] https://10.0.2.15:8443/swaggerui/ is mapped to folder /swagger-ui/
3049 ready.go:30] Performing healthcheck on https://localhost:8443/healthz
02:51:46 log.go:33: [restful/swagger] listing is available at https://10.0.2.15:8443/swaggerapi
02:51:46 log.go:33: [restful/swagger] https://10.0.2.15:8443/swaggerui/ is mapped to folder /swagger-ui/
3049 serve.go:89] Serving securely on [::]:8443
3049 available_controller.go:262] Starting AvailableConditionController
3049 cache.go:32] Waiting for caches to sync for AvailableConditionController controller
3049 apiservice_controller.go:112] Starting APIServiceRegistrationController
3049 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
3049 controller.go:84] Starting OpenAPI AggregationController
3049 crd_finalizer.go:242] Starting CRDFinalizer
3049 crdregistration_controller.go:110] Starting crd-autoregister controller
3049 controller_utils.go:1019] Waiting for caches to sync for crd-autoregister controller
3049 customresource_discovery_controller.go:152] Starting DiscoveryController
3049 naming_controller.go:274] Starting NamingConditionController
3049 ready.go:49] Got healthcheck response: [+]ping ok
ric-apiserver-start-informers ok
t-apiextensions-informers ok
t-apiextensions-controllers ok
strap-controller failed: reason withheld
failed: reason withheld
t-kube-apiserver-informers ok
t-kube-aggregator-informers ok
ervice-registration-controller ok
ervice-status-available-controller ok
ervice-openapi-controller ok
-apiserver-autoregistration ok
failed: reason withheld
failed
3049 cache.go:39] Caches are synced for AvailableConditionController controller
3049 cache.go:39] Caches are synced for APIServiceRegistrationController controller
3049 controller_utils.go:1026] Caches are synced for crd-autoregister controller
3049 autoregister_controller.go:136] Starting autoregister controller
3049 cache.go:32] Waiting for caches to sync for autoregister controller
3049 cache.go:39] Caches are synced for autoregister controller
3049 ready.go:30] Performing healthcheck on https://localhost:8443/healthz
3049 ready.go:49] Got healthcheck response: [+]ping ok
ric-apiserver-start-informers ok
t-apiextensions-informers ok
t-apiextensions-controllers ok
strap-controller failed: reason withheld
failed: reason withheld
t-kube-apiserver-informers ok
t-kube-aggregator-informers ok
ervice-registration-controller ok
ervice-status-available-controller ok
ervice-openapi-controller ok
-apiserver-autoregistration ok
failed: reason withheld
failed
3049 controller.go:538] quota admission added evaluator for: { endpoints}
3049 ready.go:30] Performing healthcheck on https://localhost:8443/healthz
3049 ready.go:49] Got healthcheck response: ok
ready!
/> controller-manager to be healthy...
3049 controllermanager.go:108] Version: v1.9.0
3049 leaderelection.go:174] attempting to acquire leader lease...
3049 leaderelection.go:184] successfully acquired lease kube-system/kube-controller-manager
3049 event.go:218] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"kube-system", Name:"kube-controller-manager", UID:"4a78e886-bd49-11e8-b2a6-080027f0cc2a", APIVersion:"v1", ResourceVersion:"40", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' minikube became leader
3049 plugins.go:101] No cloud provider specified.
3049 controller_utils.go:1019] Waiting for caches to sync for tokens controller
3049 controllermanager.go:493] Started "replicationcontroller"
3049 controllermanager.go:477] "bootstrapsigner" is disabled
3049 controllermanager.go:493] Started "endpoint"
3049 replica_set.go:183] Starting replicationcontroller controller
3049 controller_utils.go:1019] Waiting for caches to sync for ReplicationController controller
3049 endpoints_controller.go:153] Starting endpoint controller
3049 controller_utils.go:1019] Waiting for caches to sync for endpoint controller
3049 controllermanager.go:493] Started "podgc"
3049 gc_controller.go:76] Starting GC controller
3049 controller_utils.go:1019] Waiting for caches to sync for GC controller
3049 controller_utils.go:1026] Caches are synced for tokens controller
3049 shared_informer.go:310] resyncPeriod 59478255374441 is smaller than resyncCheckPeriod 78276958146727 and the informer has already started. Changing it to 78276958146727
3049 resource_quota_monitor.go:215] QuotaMonitor created object count evaluator for { endpoints}
3049 resource_quota_monitor.go:215] QuotaMonitor created object count evaluator for {apps controllerrevisions}
3049 resource_quota_monitor.go:215] QuotaMonitor created object count evaluator for {autoscaling horizontalpodautoscalers}
3049 resource_quota_monitor.go:215] QuotaMonitor created object count evaluator for {rbac.authorization.k8s.io roles}
3049 resource_quota_monitor.go:215] QuotaMonitor created object count evaluator for {settings.k8s.io podpresets}
3049 resource_quota_monitor.go:215] QuotaMonitor created object count evaluator for {extensions replicasets}
3049 resource_quota_monitor.go:215] QuotaMonitor created object count evaluator for {batch jobs}
3049 shared_informer.go:310] resyncPeriod 71730508848468 is smaller than resyncCheckPeriod 78276958146727 and the informer has already started. Changing it to 78276958146727
3049 resource_quota_monitor.go:215] QuotaMonitor created object count evaluator for { serviceaccounts}
3049 resource_quota_monitor.go:215] QuotaMonitor created object count evaluator for {extensions deployments}
3049 resource_quota_monitor.go:215] QuotaMonitor created object count evaluator for {extensions ingresses}
3049 resource_quota_monitor.go:215] QuotaMonitor created object count evaluator for {apps deployments}
3049 resource_quota_monitor.go:215] QuotaMonitor created object count evaluator for {batch cronjobs}
3049 resource_quota_monitor.go:215] QuotaMonitor created object count evaluator for {extensions daemonsets}
3049 resource_quota_monitor.go:215] QuotaMonitor created object count evaluator for {apps statefulsets}
3049 resource_quota_monitor.go:215] QuotaMonitor created object count evaluator for {rbac.authorization.k8s.io rolebindings}
3049 resource_quota_monitor.go:215] QuotaMonitor created object count evaluator for {apps replicasets}
3049 resource_quota_monitor.go:215] QuotaMonitor created object count evaluator for {apps daemonsets}
3049 resource_quota_monitor.go:215] QuotaMonitor created object count evaluator for {events.k8s.io events}
3049 resource_quota_monitor.go:215] QuotaMonitor created object count evaluator for {networking.k8s.io networkpolicies}
3049 resource_quota_monitor.go:215] QuotaMonitor created object count evaluator for { limitranges}
3049 resource_quota_monitor.go:215] QuotaMonitor created object count evaluator for {policy poddisruptionbudgets}
3049 resource_quota_monitor.go:215] QuotaMonitor created object count evaluator for { podtemplates}
3049 controllermanager.go:493] Started "resourcequota"
3049 resource_quota_controller.go:274] Starting resource quota controller
3049 controller_utils.go:1019] Waiting for caches to sync for resource quota controller
3049 resource_quota_monitor.go:288] QuotaMonitor running
is ready!
/> scheduler to be healthy...
3049 server.go:383] unable to register configz: register config "componentconfig" twice
3049 server.go:551] Version: v1.9.0
3049 server.go:570] starting healthz server on 0.0.0.0:10251
3049 controller_utils.go:1019] Waiting for caches to sync for scheduler controller
3049 controller_utils.go:1026] Caches are synced for scheduler controller
3049 controllermanager.go:493] Started "garbagecollector"
3049 garbagecollector.go:136] Starting garbage collector controller
3049 controller_utils.go:1019] Waiting for caches to sync for garbage collector controller
3049 graph_builder.go:321] GraphBuilder running
3049 controllermanager.go:493] Started "disruption"
3049 disruption.go:288] Starting disruption controller
3049 controller_utils.go:1019] Waiting for caches to sync for disruption controller
3049 controllermanager.go:493] Started "cronjob"
3049 cronjob_controller.go:103] Starting CronJob Manager
3049 controllermanager.go:493] Started "csrcleaner"
3049 controllermanager.go:477] "tokencleaner" is disabled
3049 controllermanager.go:493] Started "clusterrole-aggregation"
3049 cleaner.go:81] Starting CSR cleaner controller
3049 clusterroleaggregation_controller.go:148] Starting ClusterRoleAggregator
3049 controller_utils.go:1019] Waiting for caches to sync for ClusterRoleAggregator controller
ready!
/> kubelet to be healthy...
3049 server.go:182] Version: v1.9.0
3049 feature_gate.go:220] feature gates: &{{} map[]}
3049 server.go:280] --require-kubeconfig is deprecated. Set --kubeconfig without using --require-kubeconfig.
3049 controllermanager.go:493] Started "namespace"
3049 controllermanager.go:493] Started "deployment"
3049 deployment_controller.go:153] Starting deployment controller
3049 controller_utils.go:1019] Waiting for caches to sync for deployment controller
3049 namespace_controller.go:186] Starting namespace controller
3049 controller_utils.go:1019] Waiting for caches to sync for namespace controller
3049 controllermanager.go:493] Started "csrapproving"
3049 certificate_controller.go:113] Starting certificate controller
3049 controller_utils.go:1019] Waiting for caches to sync for certificate controller
3049 core.go:73] Failed to start service controller: WARNING: no cloud provider provided, services of type LoadBalancer will fail
3049 controllermanager.go:490] Skipping "service"
3049 controllermanager.go:490] Skipping "pvc-protection"
3049 controllermanager.go:493] Started "job"
3049 controllermanager.go:490] Skipping "csrsigning"
3049 controllermanager.go:493] Started "persistentvolume-binder"
3049 job_controller.go:138] Starting job controller
3049 controller_utils.go:1019] Waiting for caches to sync for job controller
3049 pv_controller_base.go:264] Starting persistent volume controller
3049 controller_utils.go:1019] Waiting for caches to sync for persistent volume controller
3049 plugins.go:101] No cloud provider specified.
3049 server.go:428] --cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to /
3049 container_manager_linux.go:242] container manager verified user specified cgroup-root exists: /
3049 container_manager_linux.go:247] Creating Container Manager object based on Node Config: {RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:docker CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:nodefs.available Operator:LessThan Value:{Quantity: Percentage:0.1} GracePeriod:0s MinReclaim:} {Signal:nodefs.inodesFree Operator:LessThan Value:{Quantity: Percentage:0.05} GracePeriod:0s MinReclaim:} {Signal:imagefs.available Operator:LessThan Value:{Quantity: Percentage:0.15} GracePeriod:0s MinReclaim:} {Signal:memory.available Operator:LessThan Value:{Quantity:100Mi Percentage:0} GracePeriod:0s MinReclaim:}]} ExperimentalQOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalCPUManagerReconcilePeriod:10s}
3049 container_manager_linux.go:266] Creating device plugin manager: false
3049 kubelet.go:290] Adding manifest path: /etc/kubernetes/manifests
3049 kubelet.go:313] Watching apiserver
3049 controllermanager.go:493] Started "horizontalpodautoscaling"
3049 controllermanager.go:493] Started "ttl"
3049 probe.go:215] Flexvolume plugin directory at /usr/libexec/kubernetes/kubelet-plugins/volume/exec/ does not exist. Recreating.
3049 controllermanager.go:493] Started "attachdetach"
3049 attach_detach_controller.go:258] Starting attach detach controller
3049 controller_utils.go:1019] Waiting for caches to sync for attach detach controller
3049 horizontal.go:128] Starting HPA controller
3049 controller_utils.go:1019] Waiting for caches to sync for HPA controller
3049 ttl_controller.go:116] Starting TTL controller
3049 controller_utils.go:1019] Waiting for caches to sync for TTL controller
3049 kubelet_network.go:139] Hairpin mode set to "promiscuous-bridge" but kubenet is not enabled, falling back to "hairpin-veth"
3049 kubelet.go:571] Hairpin mode set to "hairpin-veth"
3049 client.go:80] Connecting to docker on unix:///var/run/docker.sock
3049 client.go:109] Start docker client with request timeout=2m0s
3049 controllermanager.go:493] Started "replicaset"
3049 controllermanager.go:493] Started "daemonset"
3049 controllermanager.go:490] Skipping "persistentvolume-expander"
3049 controllermanager.go:493] Started "serviceaccount"
3049 controllermanager.go:493] Started "statefulset"
3049 node_controller.go:246] Sending events to api server.
3049 taint_controller.go:158] Sending events to api server.
3049 controllermanager.go:493] Started "node"
3049 core.go:132] Will not configure cloud provider routes for allocate-node-cidrs: false, configure-cloud-routes: true.
3049 controllermanager.go:490] Skipping "route"
3049 controller_utils.go:1019] Waiting for caches to sync for resource quota controller
3049 replica_set.go:183] Starting replicaset controller
3049 controller_utils.go:1019] Waiting for caches to sync for ReplicaSet controller
3049 daemon_controller.go:232] Starting daemon sets controller
3049 controller_utils.go:1019] Waiting for caches to sync for daemon sets controller
3049 serviceaccounts_controller.go:115] Starting service account controller
3049 controller_utils.go:1019] Waiting for caches to sync for service account controller
3049 stateful_set.go:150] Starting stateful set controller
3049 controller_utils.go:1019] Waiting for caches to sync for stateful set controller
3049 node_controller.go:558] Starting node controller
3049 controller_utils.go:1019] Waiting for caches to sync for node controller
3049 docker_service.go:232] Docker cri networking managed by kubernetes.io/no-op
3049 docker_service.go:237] Docker Info: &{ID:FZGD:3YI2:GT45:6MFB:2HFY:6LYK:MYET:2DMS:FI6L:MOEL:G6J7:HBQ5 Containers:0 ContainersRunning:0 ContainersPaused:0 ContainersStopped:0 Images:0 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Native Overlay Diff true]] SystemStatus:[] Plugins:{Volume:[local] Network:[bridge host macvlan null overlay] Authorization:[] Log:[awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:true CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6tables:true Debug:false NFd:20 OomKillDisable:true NGoroutines:27 SystemTime:2018-09-21T02:51:53.740716062Z LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:0 KernelVersion:4.9.64 OperatingSystem:Buildroot 2017.11 OSType:linux Architecture:x86_64 IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:0xc42416ac40 NCPU:2 MemTotal:2097205248 GenericResources:[] DockerRootDir:/var/lib/docker HTTPProxy: HTTPSProxy: NoProxy: Name:minikube Labels:[provider=virtualbox] ExperimentalBuild:false ServerVersion:17.09.0-ce ClusterStore: ClusterAdvertise: Runtimes:map[runc:{Path:docker-runc Args:[]}] DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:[] Nodes:0 Managers:0 Cluster:} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:06b9cb35161009dcb7123345749fef02f7cea8e0 Expected:06b9cb35161009dcb7123345749fef02f7cea8e0} RuncCommit:{ID:3f2f8b84a77f73d38244dd690525642a72156c64 Expected:3f2f8b84a77f73d38244dd690525642a72156c64} InitCommit:{ID:N/A Expected:} SecurityOptions:[name=seccomp,profile=default]}
3049 docker_service.go:250] Setting cgroupDriver to cgroupfs
3049 controller_utils.go:1026] Caches are synced for certificate controller
3049 controller_utils.go:1026] Caches are synced for namespace controller
3049 remote_runtime.go:43] Connecting to runtime service unix:///var/run/dockershim.sock
3049 kuberuntime_manager.go:186] Container runtime docker initialized, version: 17.09.0-ce, apiVersion: 1.32.0
3049 kuberuntime_manager.go:918] updating runtime config through cri with podcidr 10.180.1.0/24
3049 docker_service.go:343] docker cri received runtime config &RuntimeConfig{NetworkConfig:&NetworkConfig{PodCidr:10.180.1.0/24,},}
3049 kubelet_network.go:196] Setting Pod CIDR: -> 10.180.1.0/24
3049 server.go:755] Started kubelet
3049 kubelet.go:1275] Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data for container /
3049 kubelet_node_status.go:273] Setting node annotation to enable volume controller attach/detach
3049 server.go:129] Starting to listen on 0.0.0.0:10250
3049 server.go:299] Adding debug handlers to kubelet server.
3049 controller_utils.go:1026] Caches are synced for TTL controller
3049 controller_utils.go:1026] Caches are synced for HPA controller
3049 fs_resource_analyzer.go:66] Starting FS ResourceAnalyzer
3049 status_manager.go:140] Starting to sync pod status with apiserver
3049 kubelet.go:1767] Starting kubelet main sync loop.
3049 kubelet.go:1778] skipping pod synchronization - [container runtime is down PLEG is not healthy: pleg was last seen active 2562047h47m16.854775807s ago; threshold is 3m0s]
3049 container_manager_linux.go:583] [ContainerManager]: Fail to get rootfs information unable to find data for container /
3049 volume_manager.go:247] Starting Kubelet Volume Manager
3049 controller_utils.go:1026] Caches are synced for service account controller
3049 controller.go:538] quota admission added evaluator for: { serviceaccounts}
3049 controller_utils.go:1026] Caches are synced for ClusterRoleAggregator controller
3049 kubelet_node_status.go:273] Setting node annotation to enable volume controller attach/detach
3049 kubelet_node_status.go:82] Attempting to register node minikube
3049 controller_utils.go:1026] Caches are synced for ReplicationController controller
3049 controller_utils.go:1026] Caches are synced for GC controller
3049 controller_utils.go:1026] Caches are synced for node controller
3049 controller_utils.go:1026] Caches are synced for stateful set controller
3049 taint_controller.go:181] Starting NoExecuteTaintManager
3049 controller_utils.go:1026] Caches are synced for daemon sets controller
3049 controller_utils.go:1026] Caches are synced for ReplicaSet controller
3049 controller_utils.go:1026] Caches are synced for disruption controller
3049 disruption.go:296] Sending events to api server.
3049 controller_utils.go:1026] Caches are synced for deployment controller
3049 kubelet_node_status.go:85] Successfully registered node minikube
3049 actual_state_of_world.go:491] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true because nodeName="minikube" does not exist
3049 kuberuntime_manager.go:918] updating runtime config through cri with podcidr
3049 controller_utils.go:1026] Caches are synced for job controller
3049 docker_service.go:343] docker cri received runtime config &RuntimeConfig{NetworkConfig:&NetworkConfig{PodCidr:,},}
3049 kubelet_network.go:196] Setting Pod CIDR: 10.180.1.0/24 ->
3049 controller_utils.go:1026] Caches are synced for attach detach controller
3049 controller_utils.go:1026] Caches are synced for endpoint controller
3049 controller_utils.go:1026] Caches are synced for resource quota controller
3049 controller_utils.go:1026] Caches are synced for resource quota controller
3049 controller_utils.go:1026] Caches are synced for persistent volume controller
3049 controller_utils.go:1026] Caches are synced for garbage collector controller
3049 garbagecollector.go:145] Garbage collector: all resource monitors have synced. Proceeding to collect garbage
/> /> proxy to be healthy...
3049 server_others.go:138] Using iptables Proxier.
3049 proxier.go:468] clusterCIDR not specified, unable to distinguish between internal and external traffic
3049 server_others.go:171] Tearing down inactive rules.
3049 server.go:430] Version: v1.9.0
3049 healthcheck.go:317] Failed to start node healthz on 0: listen tcp: address 0: missing port in address
3049 config.go:202] Starting service config controller
3049 controller_utils.go:1019] Waiting for caches to sync for service config controller
3049 config.go:102] Starting endpoints config controller
3049 controller_utils.go:1019] Waiting for caches to sync for endpoints config controller
3049 controller_utils.go:1026] Caches are synced for endpoints config controller
3049 controller_utils.go:1026] Caches are synced for service config controller
3049 container_manager_linux.go:583] [ContainerManager]: Fail to get rootfs information unable to find data for container /
3049 container_manager_linux.go:583] [ContainerManager]: Fail to get rootfs information unable to find data for container /
3049 controller_utils.go:1019] Waiting for caches to sync for garbage collector controller
3049 controller_utils.go:1026] Caches are synced for garbage collector controller
3049 reconciler.go:217] operationExecutor.VerifyControllerAttachedVolume started for volume "addons" (UniqueName: "kubernetes.io/host-path/c4c3188325a93a2d7fb1714e1abf1259-addons") pod "kube-addon-manager-minikube" (UID: "c4c3188325a93a2d7fb1714e1abf1259")
3049 reconciler.go:217] operationExecutor.VerifyControllerAttachedVolume started for volume "kubeconfig" (UniqueName: "kubernetes.io/host-path/c4c3188325a93a2d7fb1714e1abf1259-kubeconfig") pod "kube-addon-manager-minikube" (UID: "c4c3188325a93a2d7fb1714e1abf1259")
3049 node_controller.go:611] Initializing eviction metric for zone:
3049 node_controller.go:964] Missing timestamp for Node minikube. Assuming now as a timestamp.
3049 node_controller.go:830] Controller detected that all Nodes are not-Ready. Entering master disruption mode.
3049 event.go:218] Event(v1.ObjectReference{Kind:"Node", Namespace:"", Name:"minikube", UID:"4bf22b26-bd49-11e8-b2a6-080027f0cc2a", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RegisteredNode' Node minikube event: Registered Node minikube in Controller
3049 node_controller.go:857] Controller detected that some Nodes are Ready. Exiting master disruption mode.
3049 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
3049 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
3049 kuberuntime_manager.go:647] createPodSandbox for pod "kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
3049 pod_workers.go:186] Error syncing pod c4c3188325a93a2d7fb1714e1abf1259 ("kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)"), skipping: failed to "CreatePodSandbox" for "kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)" with CreatePodSandboxError: "CreatePodSandbox for pod \"kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)\" failed: rpc error: code = Unknown desc = failed pulling image \"gcr.io/google_containers/pause-amd64:3.0\": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
3049 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
3049 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
3049 kuberuntime_manager.go:647] createPodSandbox for pod "kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
3049 pod_workers.go:186] Error syncing pod c4c3188325a93a2d7fb1714e1abf1259 ("kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)"), skipping: failed to "CreatePodSandbox" for "kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)" with CreatePodSandboxError: "CreatePodSandbox for pod \"kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)\" failed: rpc error: code = Unknown desc = failed pulling image \"gcr.io/google_containers/pause-amd64:3.0\": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
3049 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
3049 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
3049 kuberuntime_manager.go:647] createPodSandbox for pod "kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
3049 pod_workers.go:186] Error syncing pod c4c3188325a93a2d7fb1714e1abf1259 ("kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)"), skipping: failed to "CreatePodSandbox" for "kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)" with CreatePodSandboxError: "CreatePodSandbox for pod \"kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)\" failed: rpc error: code = Unknown desc = failed pulling image \"gcr.io/google_containers/pause-amd64:3.0\": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
took too long [122.859261ms for 1 entries]
with large range/delete range!
3049 healthcheck.go:317] Failed to start node healthz on 0: listen tcp: address 0: missing port in address
3049 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
3049 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
3049 kuberuntime_manager.go:647] createPodSandbox for pod "kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
3049 pod_workers.go:186] Error syncing pod c4c3188325a93a2d7fb1714e1abf1259 ("kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)"), skipping: failed to "CreatePodSandbox" for "kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)" with CreatePodSandboxError: "CreatePodSandbox for pod \"kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)\" failed: rpc error: code = Unknown desc = failed pulling image \"gcr.io/google_containers/pause-amd64:3.0\": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
3049 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
3049 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
3049 kuberuntime_manager.go:647] createPodSandbox for pod "kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
3049 pod_workers.go:186] Error syncing pod c4c3188325a93a2d7fb1714e1abf1259 ("kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)"), skipping: failed to "CreatePodSandbox" for "kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)" with CreatePodSandboxError: "CreatePodSandbox for pod \"kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)\" failed: rpc error: code = Unknown desc = failed pulling image \"gcr.io/google_containers/pause-amd64:3.0\": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
3049 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
3049 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
3049 kuberuntime_manager.go:647] createPodSandbox for pod "kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
3049 pod_workers.go:186] Error syncing pod c4c3188325a93a2d7fb1714e1abf1259 ("kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)"), skipping: failed to "CreatePodSandbox" for "kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)" with CreatePodSandboxError: "CreatePodSandbox for pod \"kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)\" failed: rpc error: code = Unknown desc = failed pulling image \"gcr.io/google_containers/pause-amd64:3.0\": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
3049 controller.go:538] quota admission added evaluator for: {apps deployments}
3049 controller.go:538] quota admission added evaluator for: {extensions replicasets}
took too long [142.630289ms for 1 entries]
with large range/delete range!
3049 event.go:218] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"default", Name:"hello-minikube", UID:"8aff4aba-bd49-11e8-b2a6-080027f0cc2a", APIVersion:"extensions", ResourceVersion:"141", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set hello-minikube-cbf7c5d59 to 1
took too long [334.077761ms for 2 entries]
with large range/delete range!
3049 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"default", Name:"hello-minikube-cbf7c5d59", UID:"8affde6d-bd49-11e8-b2a6-080027f0cc2a", APIVersion:"extensions", ResourceVersion:"142", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: hello-minikube-cbf7c5d59-q9c2g
3049 trace.go:76] Trace[1158976444]: "Create /api/v1/namespaces/default/events" (started: 2018-09-21 02:53:39.949006101 +0000 UTC m=+115.654686456) (total time: 518.073451ms):
[518.035176ms] [518.000698ms] Object stored in database
3049 reconciler.go:217] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-ww895" (UniqueName: "kubernetes.io/secret/8b6e4258-bd49-11e8-b2a6-080027f0cc2a-default-token-ww895") pod "hello-minikube-cbf7c5d59-q9c2g" (UID: "8b6e4258-bd49-11e8-b2a6-080027f0cc2a")
3049 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
3049 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
3049 kuberuntime_manager.go:647] createPodSandbox for pod "kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
3049 pod_workers.go:186] Error syncing pod c4c3188325a93a2d7fb1714e1abf1259 ("kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)"), skipping: failed to "CreatePodSandbox" for "kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)" with CreatePodSandboxError: "CreatePodSandbox for pod \"kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)\" failed: rpc error: code = Unknown desc = failed pulling image \"gcr.io/google_containers/pause-amd64:3.0\": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
3049 healthcheck.go:317] Failed to start node healthz on 0: listen tcp: address 0: missing port in address
3049 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
3049 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "hello-minikube-cbf7c5d59-q9c2g_default(8b6e4258-bd49-11e8-b2a6-080027f0cc2a)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
3049 kuberuntime_manager.go:647] createPodSandbox for pod "hello-minikube-cbf7c5d59-q9c2g_default(8b6e4258-bd49-11e8-b2a6-080027f0cc2a)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
3049 pod_workers.go:186] Error syncing pod 8b6e4258-bd49-11e8-b2a6-080027f0cc2a ("hello-minikube-cbf7c5d59-q9c2g_default(8b6e4258-bd49-11e8-b2a6-080027f0cc2a)"), skipping: failed to "CreatePodSandbox" for "hello-minikube-cbf7c5d59-q9c2g_default(8b6e4258-bd49-11e8-b2a6-080027f0cc2a)" with CreatePodSandboxError: "CreatePodSandbox for pod \"hello-minikube-cbf7c5d59-q9c2g_default(8b6e4258-bd49-11e8-b2a6-080027f0cc2a)\" failed: rpc error: code = Unknown desc = failed pulling image \"gcr.io/google_containers/pause-amd64:3.0\": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
3049 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
3049 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
3049 kuberuntime_manager.go:647] createPodSandbox for pod "kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
3049 pod_workers.go:186] Error syncing pod c4c3188325a93a2d7fb1714e1abf1259 ("kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)"), skipping: failed to "CreatePodSandbox" for "kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)" with CreatePodSandboxError: "CreatePodSandbox for pod \"kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)\" failed: rpc error: code = Unknown desc = failed pulling image \"gcr.io/google_containers/pause-amd64:3.0\": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
3049 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
3049 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
3049 kuberuntime_manager.go:647] createPodSandbox for pod "kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
3049 pod_workers.go:186] Error syncing pod c4c3188325a93a2d7fb1714e1abf1259 ("kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)"), skipping: failed to "CreatePodSandbox" for "kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)" with CreatePodSandboxError: "CreatePodSandbox for pod \"kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)\" failed: rpc error: code = Unknown desc = failed pulling image \"gcr.io/google_containers/pause-amd64:3.0\": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
3049 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
3049 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "hello-minikube-cbf7c5d59-q9c2g_default(8b6e4258-bd49-11e8-b2a6-080027f0cc2a)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
3049 kuberuntime_manager.go:647] createPodSandbox for pod "hello-minikube-cbf7c5d59-q9c2g_default(8b6e4258-bd49-11e8-b2a6-080027f0cc2a)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
3049 pod_workers.go:186] Error syncing pod 8b6e4258-bd49-11e8-b2a6-080027f0cc2a ("hello-minikube-cbf7c5d59-q9c2g_default(8b6e4258-bd49-11e8-b2a6-080027f0cc2a)"), skipping: failed to "CreatePodSandbox" for "hello-minikube-cbf7c5d59-q9c2g_default(8b6e4258-bd49-11e8-b2a6-080027f0cc2a)" with CreatePodSandboxError: "CreatePodSandbox for pod \"hello-minikube-cbf7c5d59-q9c2g_default(8b6e4258-bd49-11e8-b2a6-080027f0cc2a)\" failed: rpc error: code = Unknown desc = failed pulling image \"gcr.io/google_containers/pause-amd64:3.0\": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
3049 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
3049 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
3049 kuberuntime_manager.go:647] createPodSandbox for pod "kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
3049 pod_workers.go:186] Error syncing pod c4c3188325a93a2d7fb1714e1abf1259 ("kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)"), skipping: failed to "CreatePodSandbox" for "kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)" with CreatePodSandboxError: "CreatePodSandbox for pod \"kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)\" failed: rpc error: code = Unknown desc = failed pulling image \"gcr.io/google_containers/pause-amd64:3.0\": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
took too long [156.607868ms for 1 entries]
with large range/delete range!
3049 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
3049 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
3049 kuberuntime_manager.go:647] createPodSandbox for pod "kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
3049 pod_workers.go:186] Error syncing pod c4c3188325a93a2d7fb1714e1abf1259 ("kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)"), skipping: failed to "CreatePodSandbox" for "kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)" with CreatePodSandboxError: "CreatePodSandbox for pod \"kube-addon-manager-minikube_kube-system(c4c3188325a93a2d7fb1714e1abf1259)\" failed: rpc error: code = Unknown desc = failed pulling image \"gcr.io/google_containers/pause-amd64:3.0\": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
3049 healthcheck.go:317] Failed to start node healthz on 0: listen tcp: address 0: missing port in address
took too long [164.951152ms for 1 entries]
with large range/delete range!
3049 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
3049 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "hello-minikube-cbf7c5d59-q9c2g_default(8b6e4258-bd49-11e8-b2a6-080027f0cc2a)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
3049 kuberuntime_manager.go:647] createPodSandbox for pod "hello-minikube-cbf7c5d59-q9c2g_default(8b6e4258-bd49-11e8-b2a6-080027f0cc2a)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
3049 pod_workers.go:186] Error syncing pod 8b6e4258-bd49-11e8-b2a6-080027f0cc2a ("hello-minikube-cbf7c5d59-q9c2g_default(8b6e4258-bd49-11e8-b2a6-080027f0cc2a)"), skipping: failed to "CreatePodSandbox" for "hello-minikube-cbf7c5d59-q9c2g_default(8b6e4258-bd49-11e8-b2a6-080027f0cc2a)" with CreatePodSandboxError: "CreatePodSandbox for pod \"hello-minikube-cbf7c5d59-q9c2g_default(8b6e4258-bd49-11e8-b2a6-080027f0cc2a)\" failed: rpc error: code = Unknown desc = failed pulling image \"gcr.io/google_containers/pause-amd64:3.0\": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"

matrixoneken on 21 Sep 2018

minikube ssh "journalctl -u kubelet"

-- Logs begin at Fri 2018-09-21 02:50:51 UTC, end at Fri 2018-09-21 02:56:15 UTC
[K. --
[K-- No entries --

matrixoneken on 21 Sep 2018

@matrixoneken - OK, you found the issue here:

CreatePodSandboxError: "CreatePodSandbox for pod "hello-minikube-cbf7c5d59-q9c2g_default(8b6e4258-bd49-11e8-b2a6-080027f0cc2a)" failed:
rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0":
Error response from daemon:
Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"

It's trying to download a container from gcr.io, which does not work in China. I did some reading at https://my.oschina.net/go4it/blog/828941 and believe this workaround may help:

minikube ssh \
"docker pull registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0
docker tag registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0 gcr.io/google_containers/pause-amd64:3.0"

I believe then your deployment should work. Please let me know if does!

tstromberg on 21 Sep 2018

@r2d4 mentioned one other possible workaround, but I have not tried it yet:

minikube start \
  —extra-arg=kubelet.pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0

This reinforces to me that we should work on #3144 as soon as possible.

tstromberg on 21 Sep 2018

I run following command
kubectl run helloworld --image=kenganrui/helloworld --port=3000 deployment "helloworld" created
the output kubeclte decribe is following:

Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 1m default-scheduler Successfully assigned nodejs-55458d6994-pdldl to minikube
Normal SuccessfulMountVolume 1m kubelet, minikube MountVolume.SetUp succeeded for volume "default-token-ww895"
Normal Pulling 4s (x3 over 1m) kubelet, minikube pulling image "kenganrui/helloworld"
Normal Pulled 1s (x3 over 25s) kubelet, minikube Successfully pulled image "kenganrui/helloworld"
Normal Created 0s (x3 over 24s) kubelet, minikube Created container
Warning Failed 0s (x3 over 23s) kubelet, minikube Error: failed to start container "nodejs": Error response from daemon: oci runtime error: container_linux.go:265: starting container process caused "exec: "deployment": executable file not found in $PATH"
Warning BackOff 0s kubelet, minikube Back-off restarting failed container

matrixoneken on 21 Sep 2018

The image kenganrui/helloworld can be pulled from docker.hub. And this image can be run in my company kubernetes clusetr. Why can not run in minikube?

matrixoneken on 21 Sep 2018

I try the following command and it works!
minikube ssh \ "docker pull anjia0532/google-containers.pause-amd64:3.1 docker tag anjia0532/google-containers.pause-amd64:3.1 k8s.gcr.io/pause-amd64:3.1"

liboyue23 on 5 Oct 2018

👍2 ❤1 🎉1 😄1

Was this page helpful?

0 / 5 - 0 ratings