Minikube: kube-addon-manager-minikube stuck in ContainerCreating: lookup storage.googleapis.com on 10.0.2.3:53: read udp 10.0.2.15:45378->10.0.2.3:53: i/o timeout

Created on 12 Nov 2017 · 10Comments · Source: kubernetes/minikube

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Please provide the following details:

Environment:

Minikube version (use minikube version): 0.23.0

OS (e.g. from /etc/os-release): Windows 10 version 1709 build 16299.19
VM Driver (e.g. cat ~/.minikube/machines/minikube/config.json | grep DriverName): virtualbox 5.1.30
ISO version (e.g. cat ~/.minikube/machines/minikube/config.json | grep -i ISO or minikube ssh cat /etc/VERSION): 1.8.0
Install tools:
Others:
The above can be generated in one go with the following commands (can be copied and pasted directly into your terminal):

minikube version
echo "";
echo "OS:";
cat /etc/os-release
echo "";
echo "VM driver": 
grep DriverName ~/.minikube/machines/minikube/config.json

echo "";
echo "ISO version";
grep -i ISO ~/.minikube/machines/minikube/config.json

What happened:

Running
minikube start --vm-driver=virtualbox
ends successfully but then:
D:>kubectl get services --all-namespaces
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 10.0.0.1 443/TCP 52m

D:>kubectl get po --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system kube-addon-manager-minikube 0/1 ContainerCreating 0 52m

And "ContainerCreating" status stays forever. So no able to do anything with newly created cluster.

What you expected to happen:
Minikube cluster starts, running and operating correctly.

How to reproduce it (as minimally and precisely as possible):
As I just described

Output of minikube logs (if applicable):

-- Logs begin at Sun 2017-11-12 17:20:31 UTC, end at Sun Nov 12 17:20:42 minikube systemd[1]: Starting Localkube...
Nov 12 17:20:43 minikube localkube[3128]: data dir = /var/lib/l Nov 12 17:20:43 minikube localkube[3128]: member dir = Nov 12 17:20:43 minikube localkube[3128]: heartbeat = Nov 12 17:20:43 minikube localkube[3128]: election = 1000ms
Nov 12 17:20:49 minikube localkube[3128]: I1112 17:20:49.674306 Nov 12 17:20:49 minikube localkube[3128]: I1112 17:20:49.674327 Nov 12 17:20:49 minikube localkube[3128]: I1112 17:20:49.674333 Nov 12 17:20:49 minikube localkube[3128]: I1112 17:20:49.675968 Nov 12 17:20:49 minikube localkube[3128]: I1112 17:20:49.676093 Nov 12 17:20:49 minikube localkube[3128]: I1112 17:20:49.676837 Nov 12 17:20:49 minikube localkube[3128]: I1112 17:20:49.773320 Nov 12 17:20:49 minikube localkube[3128]: I1112 17:20:49.773440 Nov 12 17:20:49 minikube localkube[3128]: I1112 17:20:49.792702 Nov 12 17:20:49 minikube localkube[3128]: I1112 17:20:49.794354 Nov 12 17:20:49 minikube localkube[3128]: I1112 17:20:49.794497 Nov 12 17:20:49 minikube localkube[3128]: I1112 17:20:49.895174 Nov 12 17:20:49 minikube localkube[3128]: I1112 17:20:49.902619 Nov 12 17:20:49 minikube localkube[3128]: I1112 17:20:49.915766 Nov 12 17:20:49 minikube localkube[3128]: [+]etcd ok
Nov 12 17:20:49 minikube localkube[3128]: [+]poststarthook/gene Nov 12 17:20:49 minikube localkube[3128]: [+]poststarthook/star Nov 12 17:20:49 minikube localkube[3128]: [+]poststarthook/star Nov 12 17:20:49 minikube localkube[3128]: [+]poststarthook/boot Nov 12 17:20:49 minikube localkube[3128]: [-]poststarthook/ca-registration Nov 12 17:20:49 minikube localkube[3128]: [+]poststarthook/star Nov 12 17:20:49 minikube localkube[3128]: [+]poststarthook/star Nov 12 17:20:49 minikube localkube[3128]: [+]poststarthook/apis Nov 12 17:20:49 minikube localkube[3128]: [+]poststarthook/apis Nov 12 17:20:49 minikube localkube[3128]: [+]poststarthook/apis Nov 12 17:20:49 minikube localkube[3128]: [+]poststarthook/kube Nov 12 17:20:49 minikube localkube[3128]: [+]autoregister-completion Nov 12 17:20:49 minikube localkube[3128]: healthz check Nov 12 17:20:50 minikube localkube[3128]: I1112 17:20:50.903324 Nov 12 17:20:50 minikube localkube[3128]: I1112 17:20:50.916432 Nov 12 17:20:50 minikube localkube[3128]: apiserver is Nov 12 17:20:50 minikube localkube[3128]: Starting controller-manager...
Nov 12 17:20:54 minikube localkube[3128]: Starting storage-provisioner...
/> peers on http://localhost:2380
client requests on localhost:2379
ocalkube/etcd
/var/lib/localkube/etcd/member
100ms
/> = 10000
URLs = http://localhost:2379
8e9e05c52164694d in cluster cdf818194e3a8c32 at commit index 1325
became follower at term 3
[peers: [], term: 3, commit: 1325, applied: 0, lastindex: 1325, lastterm: 3]
to 711
[version: 3.1.10, cluster version: to_be_decided]
8e9e05c52164694d [http://localhost:2380] to cluster cdf818194e3a8c32
cluster version to 3.1
for version 3.1
is starting a new election at term 3
became candidate at term 4
received MsgVoteResp from 8e9e05c52164694d at term 4
became leader at term 4
elected leader 8e9e05c52164694d at term 4
rel="nofollow noopener" target="_blank" href="Name:default">Name:default ClientURLs:[http://localhost:2379]} to cluster cdf818194e3a8c32
3128 etcd.go:58] Etcd server is ready
ip address: 10.0.2.15
client requests
/> apiserver to be healthy...
3128 server.go:114] Version: v1.8.0
3128 authentication.go:380] AnonymousAuth is not allowed with the AllowAll authorizer. Resetting AnonymousAuth to false. You should use a different authorizer
3128 plugins.go:101] No cloud provider specified.
client requests on 127.0.0.1:2379, this is strongly discouraged!
3128 ready.go:30] Performing healthcheck on https://localhost:8443/healthz
3128 ready.go:40] Error performing healthcheck: Get https://localhost:8443/healthz: dial tcp 127.0.0.1:8443: getsockopt: connection refused
17:20:44 log.go:33: [restful/swagger] listing is available at https://10.0.2.15:8443/swaggerapi
17:20:44 log.go:33: [restful/swagger] https://10.0.2.15:8443/swaggerui/ is mapped to folder /swagger-ui/
3128 ready.go:30] Performing healthcheck on https://localhost:8443/healthz
3128 ready.go:40] Error performing healthcheck: Get https://localhost:8443/healthz: dial tcp 127.0.0.1:8443: getsockopt: connection refused
17:20:45 log.go:33: [restful/swagger] listing is available at https://10.0.2.15:8443/swaggerapi
17:20:45 log.go:33: [restful/swagger] https://10.0.2.15:8443/swaggerui/ is mapped to folder /swagger-ui/
3128 ready.go:30] Performing healthcheck on https://localhost:8443/healthz
3128 ready.go:40] Error performing healthcheck: Get https://localhost:8443/healthz: dial tcp 127.0.0.1:8443: getsockopt: connection refused
3128 ready.go:30] Performing healthcheck on https://localhost:8443/healthz
3128 ready.go:40] Error performing healthcheck: Get https://localhost:8443/healthz: dial tcp 127.0.0.1:8443: getsockopt: connection refused
3128 ready.go:30] Performing healthcheck on https://localhost:8443/healthz
3128 ready.go:40] Error performing healthcheck: Get https://localhost:8443/healthz: dial tcp 127.0.0.1:8443: getsockopt: connection refused
3128 aggregator.go:138] Skipping APIService creation for scheduling.k8s.io/v1alpha1
3128 serve.go:85] Serving securely on 0.0.0.0:8443
3128 available_controller.go:192] Starting AvailableConditionController
3128 cache.go:32] Waiting for caches to sync for AvailableConditionController controller
3128 apiservice_controller.go:112] Starting APIServiceRegistrationController
3128 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
3128 crd_finalizer.go:242] Starting CRDFinalizer
3128 crdregistration_controller.go:112] Starting crd-autoregister controller
3128 controller_utils.go:1041] Waiting for caches to sync for crd-autoregister controller
3128 customresource_discovery_controller.go:152] Starting DiscoveryController
3128 naming_controller.go:277] Starting NamingConditionController
3128 controller.go:84] Starting OpenAPI AggregationController
3128 cache.go:39] Caches are synced for AvailableConditionController controller
3128 cache.go:39] Caches are synced for APIServiceRegistrationController controller
3128 controller_utils.go:1048] Caches are synced for crd-autoregister controller
3128 autoregister_controller.go:136] Starting autoregister controller
3128 cache.go:32] Waiting for caches to sync for autoregister controller
3128 cache.go:39] Caches are synced for autoregister controller
3128 ready.go:30] Performing healthcheck on https://localhost:8443/healthz
3128 ready.go:49] Got healthcheck response: [+]ping ok
ric-apiserver-start-informers ok
t-apiextensions-informers ok
t-apiextensions-controllers ok
strap-controller ok
failed: reason withheld
t-kube-apiserver-informers ok
t-kube-aggregator-informers ok
ervice-registration-controller ok
ervice-status-available-controller ok
ervice-openapi-controller ok
-apiserver-autoregistration ok
ok
failed
3128 ready.go:30] Performing healthcheck on https://localhost:8443/healthz
3128 ready.go:49] Got healthcheck response: ok
ready!
/> controller-manager to be healthy...
3128 controllermanager.go:109] Version: v1.8.0
3128 leaderelection.go:174] attempting to acquire leader lease...
3128 leaderelection.go:184] successfully acquired lease kube-system/kube-controller-manager
3128 event.go:218] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"kube-system", Name:"kube-controller-manager", UID:"e64fba31-c7ca-11e7-85ce-08002730290e", APIVersion:"v1", ResourceVersion:"1260", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' minikube became leader
3128 plugins.go:101] No cloud provider specified.
3128 controller_utils.go:1041] Waiting for caches to sync for tokens controller
3128 controllermanager.go:487] Started "replicationcontroller"
3128 replication_controller.go:151] Starting RC controller
3128 controller_utils.go:1041] Waiting for caches to sync for RC controller
3128 controllermanager.go:487] Started "cronjob"
3128 cronjob_controller.go:98] Starting CronJob Manager
3128 certificates.go:48] Failed to start certificate controller: error reading CA cert file "/etc/kubernetes/ca/ca.pem": open /etc/kubernetes/ca/ca.pem: no such file or directory
3128 controllermanager.go:484] Skipping "csrsigning"
3128 controllermanager.go:471] "bootstrapsigner" is disabled
3128 controllermanager.go:487] Started "persistentvolume-binder"
3128 pv_controller_base.go:259] Starting persistent volume controller
3128 controller_utils.go:1041] Waiting for caches to sync for persistent volume controller
3128 probe.go:215] Flexvolume plugin directory at /usr/libexec/kubernetes/kubelet-plugins/volume/exec/ does not exist. Recreating.
3128 controllermanager.go:487] Started "attachdetach"
3128 controllermanager.go:487] Started "endpoint"
3128 controllermanager.go:487] Started "podgc"
3128 controllermanager.go:487] Started "job"
3128 controllermanager.go:487] Started "deployment"
3128 controllermanager.go:487] Started "replicaset"
3128 shared_informer.go:304] resyncPeriod 64936610479459 is smaller than resyncCheckPeriod 80330752126268 and the informer has already started. Changing it to 80330752126268
3128 controllermanager.go:487] Started "resourcequota"
3128 controllermanager.go:487] Started "horizontalpodautoscaling"
3128 controllermanager.go:487] Started "csrapproving"
3128 controllermanager.go:487] Started "ttl"
3128 core.go:70] Failed to start service controller: WARNING: no cloud provider provided, services of type LoadBalancer will fail.
3128 controllermanager.go:484] Skipping "service"
3128 attach_detach_controller.go:255] Starting attach detach controller
3128 controller_utils.go:1041] Waiting for caches to sync for attach detach controller
3128 endpoints_controller.go:153] Starting endpoint controller
3128 controller_utils.go:1041] Waiting for caches to sync for endpoint controller
3128 gc_controller.go:76] Starting GC controller
3128 controller_utils.go:1041] Waiting for caches to sync for GC controller
3128 job_controller.go:138] Starting job controller
3128 controller_utils.go:1041] Waiting for caches to sync for job controller
3128 deployment_controller.go:151] Starting deployment controller
3128 controller_utils.go:1041] Waiting for caches to sync for deployment controller
3128 replica_set.go:156] Starting replica set controller
3128 controller_utils.go:1041] Waiting for caches to sync for replica set controller
3128 resource_quota_controller.go:238] Starting resource quota controller
3128 controller_utils.go:1041] Waiting for caches to sync for resource quota controller
3128 horizontal.go:145] Starting HPA controller
3128 controller_utils.go:1041] Waiting for caches to sync for HPA controller
3128 certificate_controller.go:109] Starting certificate controller
3128 controller_utils.go:1041] Waiting for caches to sync for certificate controller
3128 ttl_controller.go:116] Starting TTL controller
3128 controller_utils.go:1041] Waiting for caches to sync for TTL controller
3128 core.go:128] Unsuccessful parsing of cluster CIDR : invalid CIDR address:
3128 core.go:131] Will not configure cloud provider routes for allocate-node-cidrs: false, configure-cloud-routes: true.
3128 controllermanager.go:484] Skipping "route"
3128 node_controller.go:249] Sending events to api server.
3128 taint_controller.go:158] Sending events to api server.
3128 controllermanager.go:487] Started "node"
3128 node_controller.go:516] Starting node controller
3128 controller_utils.go:1041] Waiting for caches to sync for node controller
3128 controllermanager.go:487] Started "namespace"
3128 namespace_controller.go:186] Starting namespace controller
3128 controller_utils.go:1041] Waiting for caches to sync for namespace controller
3128 controllermanager.go:487] Started "serviceaccount"
3128 serviceaccounts_controller.go:113] Starting service account controller
3128 controller_utils.go:1041] Waiting for caches to sync for service account controller
3128 controller_utils.go:1048] Caches are synced for tokens controller
is ready!
/> scheduler to be healthy...
3128 server.go:173] unable to register configz: register config "componentconfig" twice
3128 controllermanager.go:487] Started "garbagecollector"
3128 garbagecollector.go:136] Starting garbage collector controller
3128 controller_utils.go:1041] Waiting for caches to sync for garbage collector controller
3128 graph_builder.go:321] GraphBuilder running
3128 controllermanager.go:487] Started "daemonset"
3128 daemon_controller.go:230] Starting daemon sets controller
3128 controller_utils.go:1041] Waiting for caches to sync for daemon sets controller
3128 controllermanager.go:487] Started "disruption"
3128 disruption.go:288] Starting disruption controller
3128 controller_utils.go:1041] Waiting for caches to sync for disruption controller
3128 controllermanager.go:487] Started "statefulset"
3128 controllermanager.go:471] "tokencleaner" is disabled
3128 controllermanager.go:484] Skipping "persistentvolume-expander"
3128 stateful_set.go:146] Starting stateful set controller
3128 controller_utils.go:1041] Waiting for caches to sync for stateful set controller
3128 controller_utils.go:1048] Caches are synced for persistent volume controller
3128 controller_utils.go:1048] Caches are synced for HPA controller
3128 controller_utils.go:1048] Caches are synced for certificate controller
3128 controller_utils.go:1048] Caches are synced for namespace controller
3128 controller_utils.go:1048] Caches are synced for service account controller
3128 controller_utils.go:1048] Caches are synced for RC controller
3128 actual_state_of_world.go:483] Failed to set statusUpdateNeeded to needed true because nodeName="minikube" does not exist
3128 actual_state_of_world.go:497] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true because nodeName="minikube" does not exist
3128 controller_utils.go:1048] Caches are synced for TTL controller
3128 controller_utils.go:1048] Caches are synced for attach detach controller
3128 controller_utils.go:1048] Caches are synced for endpoint controller
3128 controller_utils.go:1048] Caches are synced for GC controller
3128 controller_utils.go:1048] Caches are synced for job controller
3128 controller_utils.go:1048] Caches are synced for deployment controller
3128 controller_utils.go:1048] Caches are synced for replica set controller
3128 controller_utils.go:1048] Caches are synced for resource quota controller
3128 controller_utils.go:1048] Caches are synced for disruption controller
3128 disruption.go:296] Sending events to api server.
3128 controller_utils.go:1048] Caches are synced for stateful set controller
3128 controller_utils.go:1048] Caches are synced for node controller
3128 taint_controller.go:181] Starting NoExecuteTaintManager
3128 node_controller.go:563] Initializing eviction metric for zone:
3128 node_controller.go:916] Missing timestamp for Node minikube. Assuming now as a timestamp.
3128 node_controller.go:832] Controller detected that zone is now in state Normal.
3128 event.go:218] Event(v1.ObjectReference{Kind:"Node", Namespace:"", Name:"minikube", UID:"e7e66bc0-c7ca-11e7-85ce-08002730290e", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RegisteredNode' Node minikube event: Registered Node minikube in Controller
3128 controller_utils.go:1048] Caches are synced for garbage collector controller
3128 garbagecollector.go:145] Garbage collector: all resource monitors have synced. Proceeding to collect garbage
3128 controller_utils.go:1048] Caches are synced for daemon sets controller
3128 controller_utils.go:1041] Waiting for caches to sync for scheduler controller
3128 controller_utils.go:1048] Caches are synced for scheduler controller
3128 leaderelection.go:174] attempting to acquire leader lease...
3128 leaderelection.go:184] successfully acquired lease kube-system/kube-scheduler
3128 event.go:218] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"kube-system", Name:"kube-scheduler", UID:"e7620e19-c7ca-11e7-85ce-08002730290e", APIVersion:"v1", ResourceVersion:"1263", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' minikube became leader
ready!
/> kubelet to be healthy...
3128 feature_gate.go:156] feature gates: map[]
3128 server.go:276] --require-kubeconfig is deprecated. Set --kubeconfig without using --require-kubeconfig.
3128 client.go:75] Connecting to docker on unix:///var/run/docker.sock
3128 client.go:95] Start docker client with request timeout=2m0s
3128 server.go:289] --cloud-provider=auto-detect is deprecated. The desired cloud provider should be set explicitly
3128 manager.go:149] cAdvisor running in container: "/sys/fs/cgroup/cpu,cpuacct/system.slice/localkube.service"
3128 fs.go:139] Filesystem UUIDs: map[07bd0ff2-b5d9-47cc-8101-74728e4da363:/dev/sda2 2017-10-19-17-24-41-00:/dev/sr0 5748075d-7056-4d8b-88a1-f0ee17b5b4a7:/dev/sda1]
3128 fs.go:140] Filesystem partitions: map[tmpfs:{mountpoint:/dev/shm major:0 minor:17 fsType:tmpfs blockSize:0} /dev/sda1:{mountpoint:/mnt/sda1 major:8 minor:1 fsType:ext4 blockSize:0}]
3128 manager.go:216] Machine: {NumCores:2 CpuFrequency:2394456 MemoryCapacity:2097229824 HugePages:[{PageSize:2048 NumPages:0}] MachineID:8de7cbc30b1b4830b9dd7e7d2ccc88fa SystemUUID:7715B0F8-7CB5-46B4-82B1-032A3932BC94 BootID:7f3f38ce-ed28-4e2b-ae7d-f36f457bd401 Filesystems:[{Device:tmpfs DeviceMajor:0 DeviceMinor:17 Capacity:1048612864 Type:vfs Inodes:256009 HasInodes:true} {Device:/dev/sda1 DeviceMajor:8 DeviceMinor:1 Capacity:17293533184 Type:vfs Inodes:9732096 HasInodes:true} {Device:rootfs DeviceMajor:0 DeviceMinor:1 Capacity:0 Type:vfs Inodes:0 HasInodes:true}] DiskMap:map[8:0:{Name:sda Major:8 Minor:0 Size:20971520000 Scheduler:cfq}] NetworkDevices:[{Name:eth0 MacAddress:08:00:27:30:29:0e Speed:1000 Mtu:1500} {Name:eth1 MacAddress:08:00:27:a6:d0:5c Speed:1000 Mtu:1500} {Name:sit0 MacAddress:00:00:00:00 Speed:0 Mtu:1480}] Topology:[{Id:0 Memory:2097229824 Cores:[{Id:0 Threads:[0] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2} {Size:3145728 Type:Unified Level:3}]} {Id:1 Threads:[1] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2} {Size:3145728 Type:Unified Level:3}]}] Caches:[]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
3128 manager.go:222] Version: {KernelVersion:4.9.13 ContainerOsVersion:Buildroot 2017.02 DockerVersion:17.06.0-ce DockerAPIVersion:1.30 CadvisorVersion: CadvisorRevision:}
3128 server.go:422] --cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to /
3128 container_manager_linux.go:252] container manager verified user specified cgroup-root exists: /
3128 container_manager_linux.go:257] Creating Container Manager object based on Node Config: {RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:docker CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:memory.available Operator:LessThan Value:{Quantity:100Mi Percentage:0} GracePeriod:0s MinReclaim:} {Signal:nodefs.available Operator:LessThan Value:{Quantity: Percentage:0.1} GracePeriod:0s MinReclaim:} {Signal:nodefs.inodesFree Operator:LessThan Value:{Quantity: Percentage:0.05} GracePeriod:0s MinReclaim:}]} ExperimentalQOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalCPUManagerReconcilePeriod:10s}
3128 container_manager_linux.go:288] Creating device plugin handler: false
3128 kubelet.go:273] Adding manifest file: /etc/kubernetes/manifests
3128 kubelet.go:283] Watching apiserver
3128 kubelet_network.go:69] Hairpin mode set to "promiscuous-bridge" but kubenet is not enabled, falling back to "hairpin-veth"
3128 kubelet.go:517] Hairpin mode set to "hairpin-veth"
3128 docker_service.go:207] Docker cri networking managed by kubernetes.io/no-op
3128 docker_service.go:224] Setting cgroupDriver to cgroupfs
3128 remote_runtime.go:43] Connecting to runtime service unix:///var/run/dockershim.sock
3128 kuberuntime_manager.go:177] Container runtime docker initialized, version: 17.06.0-ce, apiVersion: 1.30.0
3128 kuberuntime_manager.go:898] updating runtime config through cri with podcidr 10.180.1.0/24
3128 docker_service.go:306] docker cri received runtime config &RuntimeConfig{NetworkConfig:&NetworkConfig{PodCidr:10.180.1.0/24,},}
3128 kubelet_network.go:276] Setting Pod CIDR: -> 10.180.1.0/24
3128 server.go:718] Started kubelet v1.8.0
3128 kubelet.go:1234] Image garbage collection failed once. Stats initialization may not have completed yet: unable to find data for container /
3128 server.go:128] Starting to listen on 0.0.0.0:10250
3128 server.go:296] Adding debug handlers to kubelet server.
3128 fs_resource_analyzer.go:66] Starting FS ResourceAnalyzer
3128 status_manager.go:140] Starting to sync pod status with apiserver
3128 kubelet.go:1768] Starting kubelet main sync loop.
3128 kubelet.go:1779] skipping pod synchronization - [container runtime is down PLEG is not healthy: pleg was last seen active 2562047h47m16.854775807s ago; threshold is 3m0s]
3128 container_manager_linux.go:603] [ContainerManager]: Fail to get rootfs information unable to find data for container /
3128 volume_manager.go:246] Starting Kubelet Volume Manager
3128 factory.go:355] Registering Docker factory
3128 factory.go:89] Registering Rkt factory
3128 factory.go:157] Registering CRI-O factory
3128 factory.go:54] Registering systemd factory
3128 factory.go:86] Registering Raw factory
3128 manager.go:1140] Started watching for new ooms in manager
3128 manager.go:311] Starting recovery of all containers
3128 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
3128 manager.go:316] Recovery completed
3128 rkt.go:56] starting detectRktContainers thread
3128 kubelet_node_status.go:83] Attempting to register node minikube
3128 helpers.go:832] Could not find capacity information for resource ephemeral-storage
3128 helpers.go:843] eviction manager: no observation found for eviction signal allocatableNodeFs.available
3128 kubelet_node_status.go:134] Node minikube was previously registered
3128 kubelet_node_status.go:86] Successfully registered node minikube
3128 kuberuntime_manager.go:898] updating runtime config through cri with podcidr
3128 docker_service.go:306] docker cri received runtime config &RuntimeConfig{NetworkConfig:&NetworkConfig{PodCidr:,},}
3128 kubelet_network.go:276] Setting Pod CIDR: 10.180.1.0/24 ->
/> /> proxy to be healthy...
3128 server_others.go:63] unable to register configz: register config "componentconfig" twice
3128 server_others.go:117] Using iptables Proxier.
3128 proxier.go:473] clusterCIDR not specified, unable to distinguish between internal and external traffic
3128 server_others.go:152] Tearing down inactive rules.
3128 healthcheck.go:317] Failed to start node healthz on 0: listen tcp: address 0: missing port in address
3128 config.go:202] Starting service config controller
3128 controller_utils.go:1041] Waiting for caches to sync for service config controller
3128 config.go:102] Starting endpoints config controller
3128 controller_utils.go:1041] Waiting for caches to sync for endpoints config controller
3128 controller_utils.go:1048] Caches are synced for endpoints config controller
3128 controller_utils.go:1048] Caches are synced for service config controller
/> storage-provisioner to be healthy...
3128 controller.go:407] Starting provisioner controller d71c5464-c7cd-11e7-bf8b-08002730290e!
is ready!
3128 reconciler.go:212] operationExecutor.VerifyControllerAttachedVolume started for volume "addons" (UniqueName: "kubernetes.io/host-path/9831e93c3188555873fdb49f43198eef-addons") pod "kube-addon-manager-minikube" (UID: "9831e93c3188555873fdb49f43198eef")
3128 reconciler.go:212] operationExecutor.VerifyControllerAttachedVolume started for volume "kubeconfig" (UniqueName: "kubernetes.io/host-path/9831e93c3188555873fdb49f43198eef-kubeconfig") pod "kube-addon-manager-minikube" (UID: "9831e93c3188555873fdb49f43198eef")
3128 healthcheck.go:317] Failed to start node healthz on 0: listen tcp: address 0: missing port in address
3128 healthcheck.go:317] Failed to start node healthz on 0: listen tcp: address 0: missing port in address
3128 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": dial tcp: lookup storage.googleapis.com on 10.0.2.3:53: read udp 10.0.2.15:45378->10.0.2.3:53: i/o timeout
3128 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "kube-addon-manager-minikube_kube-system(9831e93c3188555873fdb49f43198eef)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": dial tcp: lookup storage.googleapis.com on 10.0.2.3:53: read udp 10.0.2.15:45378->10.0.2.3:53: i/o timeout
3128 kuberuntime_manager.go:632] createPodSandbox for pod "kube-addon-manager-minikube_kube-system(9831e93c3188555873fdb49f43198eef)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": dial tcp: lookup storage.googleapis.com on 10.0.2.3:53: read udp 10.0.2.15:45378->10.0.2.3:53: i/o timeout
3128 pod_workers.go:182] Error syncing pod 9831e93c3188555873fdb49f43198eef ("kube-addon-manager-minikube_kube-system(9831e93c3188555873fdb49f43198eef)"), skipping: failed to "CreatePodSandbox" for "kube-addon-manager-minikube_kube-system(9831e93c3188555873fdb49f43198eef)" with CreatePodSandboxError: "CreatePodSandbox for pod \"kube-addon-manager-minikube_kube-system(9831e93c3188555873fdb49f43198eef)\" failed: rpc error: code = Unknown desc = failed pulling image \"gcr.io/google_containers/pause-amd64:3.0\": dial tcp: lookup storage.googleapis.com on 10.0.2.3:53: read udp 10.0.2.15:45378->10.0.2.3:53: i/o timeout"
3128 healthcheck.go:317] Failed to start node healthz on 0: listen tcp: address 0: missing port in address
3128 healthcheck.go:317] Failed to start node healthz on 0: listen tcp: address 0: missing port in address
3128 healthcheck.go:317] Failed to start node healthz on 0: listen tcp: address 0: missing port in address
3128 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": dial tcp: lookup storage.googleapis.com on 10.0.2.3:53: read udp 10.0.2.15:36849->10.0.2.3:53: i/o timeout
3128 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "kube-addon-manager-minikube_kube-system(9831e93c3188555873fdb49f43198eef)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": dial tcp: lookup storage.googleapis.com on 10.0.2.3:53: read udp 10.0.2.15:36849->10.0.2.3:53: i/o timeout
3128 kuberuntime_manager.go:632] createPodSandbox for pod "kube-addon-manager-minikube_kube-system(9831e93c3188555873fdb49f43198eef)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": dial tcp: lookup storage.googleapis.com on 10.0.2.3:53: read udp 10.0.2.15:36849->10.0.2.3:53: i/o timeout
3128 pod_workers.go:182] Error syncing pod 9831e93c3188555873fdb49f43198eef ("kube-addon-manager-minikube_kube-system(9831e93c3188555873fdb49f43198eef)"), skipping: failed to "CreatePodSandbox" for "kube-addon-manager-minikube_kube-system(9831e93c3188555873fdb49f43198eef)" with CreatePodSandboxError: "CreatePodSandbox for pod \"kube-addon-manager-minikube_kube-system(9831e93c3188555873fdb49f43198eef)\" failed: rpc error: code = Unknown desc = failed pulling image \"gcr.io/google_containers/pause-amd64:3.0\": dial tcp: lookup storage.googleapis.com on 10.0.2.3:53: read udp 10.0.2.15:36849->10.0.2.3:53: i/o timeout"
3128 healthcheck.go:317] Failed to start node healthz on 0: listen tcp: address 0: missing port in address
3128 healthcheck.go:317] Failed to start node healthz on 0: listen tcp: address 0: missing port in address
3128 healthcheck.go:317] Failed to start node healthz on 0: listen tcp: address 0: missing port in address
3128 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": dial tcp: lookup storage.googleapis.com on 10.0.2.3:53: read udp 10.0.2.15:57585->10.0.2.3:53: i/o timeout
3128 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "kube-addon-manager-minikube_kube-system(9831e93c3188555873fdb49f43198eef)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": dial tcp: lookup storage.googleapis.com on 10.0.2.3:53: read udp 10.0.2.15:57585->10.0.2.3:53: i/o timeout
3128 kuberuntime_manager.go:632] createPodSandbox for pod "kube-addon-manager-minikube_kube-system(9831e93c3188555873fdb49f43198eef)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": dial tcp: lookup storage.googleapis.com on 10.0.2.3:53: read udp 10.0.2.15:57585->10.0.2.3:53: i/o timeout
3128 pod_workers.go:182] Error syncing pod 9831e93c3188555873fdb49f43198eef ("kube-addon-manager-minikube_kube-system(9831e93c3188555873fdb49f43198eef)"), skipping: failed to "CreatePodSandbox" for "kube-addon-manager-minikube_kube-system(9831e93c3188555873fdb49f43198eef)" with CreatePodSandboxError: "CreatePodSandbox for pod \"kube-addon-manager-minikube_kube-system(9831e93c3188555873fdb49f43198eef)\" failed: rpc error: code = Unknown desc = failed pulling image \"gcr.io/google_containers/pause-amd64:3.0\": dial tcp: lookup storage.googleapis.com on 10.0.2.3:53: read udp 10.0.2.15:57585->10.0.2.3:53: i/o timeout"
3128 healthcheck.go:317] Failed to start node healthz on 0: listen tcp: address 0: missing port in address
compact 1603
compaction at 1603 (took 1.362749ms)
3128 healthcheck.go:317] Failed to start node healthz on 0: listen tcp: address 0: missing port in address
3128 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": dial tcp: lookup storage.googleapis.com on 10.0.2.3:53: read udp 10.0.2.15:36306->10.0.2.3:53: i/o timeout
3128 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "kube-addon-manager-minikube_kube-system(9831e93c3188555873fdb49f43198eef)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": dial tcp: lookup storage.googleapis.com on 10.0.2.3:53: read udp 10.0.2.15:36306->10.0.2.3:53: i/o timeout
3128 kuberuntime_manager.go:632] createPodSandbox for pod "kube-addon-manager-minikube_kube-system(9831e93c3188555873fdb49f43198eef)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": dial tcp: lookup storage.googleapis.com on 10.0.2.3:53: read udp 10.0.2.15:36306->10.0.2.3:53: i/o timeout
3128 pod_workers.go:182] Error syncing pod 9831e93c3188555873fdb49f43198eef ("kube-addon-manager-minikube_kube-system(9831e93c3188555873fdb49f43198eef)"), skipping: failed to "CreatePodSandbox" for "kube-addon-manager-minikube_kube-system(9831e93c3188555873fdb49f43198eef)" with CreatePodSandboxError: "CreatePodSandbox for pod \"kube-addon-manager-minikube_kube-system(9831e93c3188555873fdb49f43198eef)\" failed: rpc error: code = Unknown desc = failed pulling image \"gcr.io/google_containers/pause-amd64:3.0\": dial tcp: lookup storage.googleapis.com on 10.0.2.3:53: read udp 10.0.2.15:36306->10.0.2.3:53: i/o timeout"
3128 healthcheck.go:317] Failed to start node healthz on 0: listen tcp: address 0: missing port in address
3128 healthcheck.go:317] Failed to start node healthz on 0: listen tcp: address 0: missing port in address
3128 healthcheck.go:317] Failed to start node healthz on 0: listen tcp: address 0: missing port in address
3128 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": dial tcp: lookup storage.googleapis.com on 10.0.2.3:53: read udp 10.0.2.15:34596->10.0.2.3:53: i/o timeout
3128 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "kube-addon-manager-minikube_kube-system(9831e93c3188555873fdb49f43198eef)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": dial tcp: lookup storage.googleapis.com on 10.0.2.3:53: read udp 10.0.2.15:34596->10.0.2.3:53: i/o timeout
3128 kuberuntime_manager.go:632] createPodSandbox for pod "kube-addon-manager-minikube_kube-system(9831e93c3188555873fdb49f43198eef)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": dial tcp: lookup storage.googleapis.com on 10.0.2.3:53: read udp 10.0.2.15:34596->10.0.2.3:53: i/o timeout
3128 pod_workers.go:182] Error syncing pod 9831e93c3188555873fdb49f43198eef ("kube-addon-manager-minikube_kube-system(9831e93c3188555873fdb49f43198eef)"), skipping: failed to "CreatePodSandbox" for "kube-addon-manager-minikube_kube-system(9831e93c3188555873fdb49f43198eef)" with CreatePodSandboxError: "CreatePodSandbox for pod \"kube-addon-manager-minikube_kube-system(9831e93c3188555873fdb49f43198eef)\" failed: rpc error: code = Unknown desc = failed pulling image \"gcr.io/google_containers/pause-amd64:3.0\": dial tcp: lookup storage.googleapis.com on 10.0.2.3:53: read udp 10.0.2.15:34596->10.0.2.3:53: i/o timeout"
3128 healthcheck.go:317] Failed to start node healthz on 0: listen tcp: address 0: missing port in address
compact 1943
compaction at 1943 (took 1.299136ms)
3128 healthcheck.go:317] Failed to start node healthz on 0: listen tcp: address 0: missing port in address
3128 healthcheck.go:317] Failed to start node healthz on 0: listen tcp: address 0: missing port in address
3128 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": dial tcp: lookup storage.googleapis.com on 10.0.2.3:53: read udp 10.0.2.15:40931->10.0.2.3:53: i/o timeout
3128 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "kube-addon-manager-minikube_kube-system(9831e93c3188555873fdb49f43198eef)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": dial tcp: lookup storage.googleapis.com on 10.0.2.3:53: read udp 10.0.2.15:40931->10.0.2.3:53: i/o timeout
3128 kuberuntime_manager.go:632] createPodSandbox for pod "kube-addon-manager-minikube_kube-system(9831e93c3188555873fdb49f43198eef)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": dial tcp: lookup storage.googleapis.com on 10.0.2.3:53: read udp 10.0.2.15:40931->10.0.2.3:53: i/o timeout
3128 pod_workers.go:182] Error syncing pod 9831e93c3188555873fdb49f43198eef ("kube-addon-manager-minikube_kube-system(9831e93c3188555873fdb49f43198eef)"), skipping: failed to "CreatePodSandbox" for "kube-addon-manager-minikube_kube-system(9831e93c3188555873fdb49f43198eef)" with CreatePodSandboxError: "CreatePodSandbox for pod \"kube-addon-manager-minikube_kube-system(9831e93c3188555873fdb49f43198eef)\" failed: rpc error: code = Unknown desc = failed pulling image \"gcr.io/google_containers/pause-amd64:3.0\": dial tcp: lookup storage.googleapis.com on 10.0.2.3:53: read udp 10.0.2.15:40931->10.0.2.3:53: i/o timeout"
3128 healthcheck.go:317] Failed to start node healthz on 0: listen tcp: address 0: missing port in address
3128 healthcheck.go:317] Failed to start node healthz on 0: listen tcp: address 0: missing port in address
3128 healthcheck.go:317] Failed to start node healthz on 0: listen tcp: address 0: missing port in address
3128 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": dial tcp: lookup storage.googleapis.com on 10.0.2.3:53: read udp 10.0.2.15:47892->10.0.2.3:53: i/o timeout
3128 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "kube-addon-manager-minikube_kube-system(9831e93c3188555873fdb49f43198eef)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": dial tcp: lookup storage.googleapis.com on 10.0.2.3:53: read udp 10.0.2.15:47892->10.0.2.3:53: i/o timeout
3128 kuberuntime_manager.go:632] createPodSandbox for pod "kube-addon-manager-minikube_kube-system(9831e93c3188555873fdb49f43198eef)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": dial tcp: lookup storage.googleapis.com on 10.0.2.3:53: read udp 10.0.2.15:47892->10.0.2.3:53: i/o timeout
3128 pod_workers.go:182] Error syncing pod 9831e93c3188555873fdb49f43198eef ("kube-addon-manager-minikube_kube-system(9831e93c3188555873fdb49f43198eef)"), skipping: failed to "CreatePodSandbox" for "kube-addon-manager-minikube_kube-system(9831e93c3188555873fdb49f43198eef)" with CreatePodSandboxError: "CreatePodSandbox for pod \"kube-addon-manager-minikube_kube-system(9831e93c3188555873fdb49f43198eef)\" failed: rpc error: code = Unknown desc = failed pulling image \"gcr.io/google_containers/pause-amd64:3.0\": dial tcp: lookup storage.googleapis.com on 10.0.2.3:53: read udp 10.0.2.15:47892->10.0.2.3:53: i/o timeout"
compact 2280
compaction at 2280 (took 670.283µs)
3128 healthcheck.go:317] Failed to start node healthz on 0: listen tcp: address 0: missing port in address
3128 healthcheck.go:317] Failed to start node healthz on 0: listen tcp: address 0: missing port in address
3128 healthcheck.go:317] Failed to start node healthz on 0: listen tcp: address 0: missing port in address
3128 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": dial tcp: lookup storage.googleapis.com on 10.0.2.3:53: read udp 10.0.2.15:59126->10.0.2.3:53: i/o timeout
3128 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "kube-addon-manager-minikube_kube-system(9831e93c3188555873fdb49f43198eef)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": dial tcp: lookup storage.googleapis.com on 10.0.2.3:53: read udp 10.0.2.15:59126->10.0.2.3:53: i/o timeout
3128 kuberuntime_manager.go:632] createPodSandbox for pod "kube-addon-manager-minikube_kube-system(9831e93c3188555873fdb49f43198eef)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": dial tcp: lookup storage.googleapis.com on 10.0.2.3:53: read udp 10.0.2.15:59126->10.0.2.3:53: i/o timeout
3128 pod_workers.go:182] Error syncing pod 9831e93c3188555873fdb49f43198eef ("kube-addon-manager-minikube_kube-system(9831e93c3188555873fdb49f43198eef)"), skipping: failed to "CreatePodSandbox" for "kube-addon-manager-minikube_kube-system(9831e93c3188555873fdb49f43198eef)" with CreatePodSandboxError: "CreatePodSandbox for pod \"kube-addon-manager-minikube_kube-system(9831e93c3188555873fdb49f43198eef)\" failed: rpc error: code = Unknown desc = failed pulling image \"gcr.io/google_containers/pause-amd64:3.0\": dial tcp: lookup storage.googleapis.com on 10.0.2.3:53: read udp 10.0.2.15:59126->10.0.2.3:53: i/o timeout"
3128 healthcheck.go:317] Failed to start node healthz on 0: listen tcp: address 0: missing port in address
3128 healthcheck.go:317] Failed to start node healthz on 0: listen tcp: address 0: missing port in address
compact 2620
compaction at 2620 (took 668.118µs)
3128 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": dial tcp: lookup storage.googleapis.com on 10.0.2.3:53: read udp 10.0.2.15:52441->10.0.2.3:53: i/o timeout
3128 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "kube-addon-manager-minikube_kube-system(9831e93c3188555873fdb49f43198eef)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": dial tcp: lookup storage.googleapis.com on 10.0.2.3:53: read udp 10.0.2.15:52441->10.0.2.3:53: i/o timeout
3128 kuberuntime_manager.go:632] createPodSandbox for pod "kube-addon-manager-minikube_kube-system(9831e93c3188555873fdb49f43198eef)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": dial tcp: lookup storage.googleapis.com on 10.0.2.3:53: read udp 10.0.2.15:52441->10.0.2.3:53: i/o timeout
3128 pod_workers.go:182] Error syncing pod 9831e93c3188555873fdb49f43198eef ("kube-addon-manager-minikube_kube-system(9831e93c3188555873fdb49f43198eef)"), skipping: failed to "CreatePodSandbox" for "kube-addon-manager-minikube_kube-system(9831e93c3188555873fdb49f43198eef)" with CreatePodSandboxError: "CreatePodSandbox for pod \"kube-addon-manager-minikube_kube-system(9831e93c3188555873fdb49f43198eef)\" failed: rpc error: code = Unknown desc = failed pulling image \"gcr.io/google_containers/pause-amd64:3.0\": dial tcp: lookup storage.googleapis.com on 10.0.2.3:53: read udp 10.0.2.15:52441->10.0.2.3:53: i/o timeout"
3128 healthcheck.go:317] Failed to start node healthz on 0: listen tcp: address 0: missing port in address
3128 healthcheck.go:317] Failed to start node healthz on 0: listen tcp: address 0: missing port in address
3128 healthcheck.go:317] Failed to start node healthz on 0: listen tcp: address 0: missing port in address
3128 healthcheck.go:317] Failed to start node healthz on 0: listen tcp: address 0: missing port in address
3128 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": dial tcp: lookup storage.googleapis.com on 10.0.2.3:53: read udp 10.0.2.15:34699->10.0.2.3:53: i/o timeout
3128 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "kube-addon-manager-minikube_kube-system(9831e93c3188555873fdb49f43198eef)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": dial tcp: lookup storage.googleapis.com on 10.0.2.3:53: read udp 10.0.2.15:34699->10.0.2.3:53: i/o timeout
3128 kuberuntime_manager.go:632] createPodSandbox for pod "kube-addon-manager-minikube_kube-system(9831e93c3188555873fdb49f43198eef)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": dial tcp: lookup storage.googleapis.com on 10.0.2.3:53: read udp 10.0.2.15:34699->10.0.2.3:53: i/o timeout
3128 pod_workers.go:182] Error syncing pod 9831e93c3188555873fdb49f43198eef ("kube-addon-manager-minikube_kube-system(9831e93c3188555873fdb49f43198eef)"), skipping: failed to "CreatePodSandbox" for "kube-addon-manager-minikube_kube-system(9831e93c3188555873fdb49f43198eef)" with CreatePodSandboxError: "CreatePodSandbox for pod \"kube-addon-manager-minikube_kube-system(9831e93c3188555873fdb49f43198eef)\" failed: rpc error: code = Unknown desc = failed pulling image \"gcr.io/google_containers/pause-amd64:3.0\": dial tcp: lookup storage.googleapis.com on 10.0.2.3:53: read udp 10.0.2.15:34699->10.0.2.3:53: i/o timeout"

Anything else do we need to know:
Looks like it can't get some images from the repository, but i have no limitation in internet access from my laptop.

lifecyclrotten

Source

disparo13

👍5

Most helpful comment

TLDR: Try to ping minikube. If it resolves to an IP, maybe remove the search-server from /etc/resolv.conf.

I experienced the same issue @disparo13 described. I am running:

Ubuntu 16.04.4 (HWE kernel 4.13.0-36)
Docker CE 17.12.1-ce
minikube 0.25.0
--vm-driver=none

on a VPS. After executing minikube start --vm-driver=none only the addon-manager came up (checked with docker ps). I checked the kube-addon-manager container logs and it contained the following:

INFO: == Kubernetes addon manager started at 2018-03-10T15:17:54+0000 with ADDON_CHECK_INTERVAL_SEC=60 ==
Unable to connect to the server: dial tcp 46.XXX.XXX.XXX:8443: i/o timeout
WRN: == Error getting default service account, retry in 0.5 second ==
...

The IP the addon-manager failed connecting to (46.XXX.XXX.XXX) was unknown to me (and surprising, since it should use localhost as far as I can tell). Eventually I tried to ping minikube, which resolved to the same IP:

$ ping minikube
PING minikube.vpsdomain.tld (46.XXX.XXX.XXX) 56(84) bytes of data.

vpsdomain.tld is a domain by my VPS-provider that gets configured as a search-server through DHCP in /etc/resolv.conf by default:

search vpsdomain.tld
nameserver 46.YYY.YYY.YYY
nameserver 46.ZZZ.ZZZ.ZZZ

I have solved my issues by removing the search-domain. (Note: simply removing it from /etc/resolv.conf might not be permanent. Check the documentation for your distribution on how to do this permanently.)

kubectl resolving for minikube instead of using the server specified in kubeconfig (see below) seems like a bug to me.

apiVersion: v1
clusters:
- cluster:
    certificate-authority: /var/lib/localkube/certs/ca.crt
    server: https://localhost:8443
  name: minikube
contexts:
- context:
    cluster: minikube
    user: minikube
  name: minikube
current-context: minikube
kind: Config
preferences: {}
users:
- name: minikube
  user:
    as-user-extra: {}
    client-certificate: /var/lib/localkube/certs/apiserver.crt
    client-key: /var/lib/localkube/certs/apiserver.key

Maybe this helps someone!

@r2d4 can you work with this information? Is there anything else I can provide?

pitkley on 10 Mar 2018

❤1 👍1

All 10 comments

mappedinn on 14 Nov 2017

same

zxcvbn97 on 24 Nov 2017

ximinggao on 4 Dec 2017

This was happening when I had a DNS listener locally on port 53; I moved the local resolver to another port, and the problem went away. I cannot take credit for this -- saw it in some other issue.

mkw on 21 Dec 2017

👍2

@mkw Can you specifically mention how you did that please?

I'm getting :

3047 healthcheck.go:317] Failed to start node healthz on 0: listen tcp: address 0: missing port in address

karan-kapoor90 on 12 Feb 2018

👍1

@karan-kapoor90 I don't remember the error I got, but when researching it I saw it noted that K8s wanted to listen on port 53, and I knew that I had another DNS service listening there. Use netstat -anlv (flags shown for macOS) to see if something else is listening on 53, and if so, move it off.

mkw on 12 Feb 2018

TLDR: Try to ping minikube. If it resolves to an IP, maybe remove the search-server from /etc/resolv.conf.

I experienced the same issue @disparo13 described. I am running:

Ubuntu 16.04.4 (HWE kernel 4.13.0-36)
Docker CE 17.12.1-ce
minikube 0.25.0
--vm-driver=none

on a VPS. After executing minikube start --vm-driver=none only the addon-manager came up (checked with docker ps). I checked the kube-addon-manager container logs and it contained the following:

INFO: == Kubernetes addon manager started at 2018-03-10T15:17:54+0000 with ADDON_CHECK_INTERVAL_SEC=60 ==
Unable to connect to the server: dial tcp 46.XXX.XXX.XXX:8443: i/o timeout
WRN: == Error getting default service account, retry in 0.5 second ==
...

$ ping minikube
PING minikube.vpsdomain.tld (46.XXX.XXX.XXX) 56(84) bytes of data.

vpsdomain.tld is a domain by my VPS-provider that gets configured as a search-server through DHCP in /etc/resolv.conf by default:

search vpsdomain.tld
nameserver 46.YYY.YYY.YYY
nameserver 46.ZZZ.ZZZ.ZZZ

kubectl resolving for minikube instead of using the server specified in kubeconfig (see below) seems like a bug to me.

apiVersion: v1
clusters:
- cluster:
    certificate-authority: /var/lib/localkube/certs/ca.crt
    server: https://localhost:8443
  name: minikube
contexts:
- context:
    cluster: minikube
    user: minikube
  name: minikube
current-context: minikube
kind: Config
preferences: {}
users:
- name: minikube
  user:
    as-user-extra: {}
    client-certificate: /var/lib/localkube/certs/apiserver.crt
    client-key: /var/lib/localkube/certs/apiserver.key

Maybe this helps someone!

@r2d4 can you work with this information? Is there anything else I can provide?

pitkley on 10 Mar 2018

❤1 👍1

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot on 8 Jun 2018

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

fejta-bot on 8 Jul 2018

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

fejta-bot on 7 Aug 2018

Was this page helpful?

0 / 5 - 0 ratings