Minikube: Enabling bidirectional mounts on pod kills docker
Enabling bidirectional mounts on pods kills docker.
deployed kubernetes using minikube with VirtualBox driver with (CPUs=4, Memory=5000MB, Disk=80000MB)
- pod list output
[馃帺锔嶿mrajanna@localhost rook $]kubectl get po -nsmoke-ns-system
NAME READY STATUS RESTARTS AGE
csi-cephfsplugin-cg764 1/2 RunContainerError 0 85s
csi-cephfsplugin-provisioner-0 1/2 ErrImagePull 0 85s
csi-rbdplugin-attacher-0 1/1 Running 0 85s
csi-rbdplugin-provisioner-0 1/3 ImageInspectError 0 85s
csi-rbdplugin-vw4ch 1/2 RunContainerError 0 85s
rook-ceph-agent-kqpzk 1/1 Running 0 85s
rook-ceph-operator-7f5858cb-ngrd5 1/1 Running 0 87s
rook-discover-dqdjh 1/1 Running 0 85s
- docker ps output
$ docker ps
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
- kubectl describe output
kubectl describe po/csi-rbdplugin-vw4ch -nsmoke-ns-system
Name: csi-rbdplugin-vw4ch
Namespace: smoke-ns-system
Priority: 0
PriorityClassName: <none>
Node: minikube/10.0.2.15
Start Time: Wed, 10 Apr 2019 10:12:56 +0530
Labels: app=csi-rbdplugin
controller-revision-hash=8469978b6d
pod-template-generation=1
Annotations: <none>
Status: Running
IP: 10.0.2.15
Controlled By: DaemonSet/csi-rbdplugin
Containers:
driver-registrar:
Container ID: docker://b0c4a874b40af193951955f0673faf1b26b926d490f3500384d2d48922d2d6f3
Image: quay.io/k8scsi/csi-node-driver-registrar:v1.0.2
Image ID: docker-pullable://quay.io/k8scsi/csi-node-driver-registrar@sha256:ffecfbe6ae9f446e5102cbf2c73041d63ccf44bcfd72e2f2a62174a3a185eb69
Port: <none>
Host Port: <none>
Args:
--v=5
--csi-address=/csi/csi.sock
--kubelet-registration-path=/var/lib/kubelet/plugins/rbd.csi.ceph.com/csi.sock
State: Running
Started: Wed, 10 Apr 2019 10:13:03 +0530
Ready: True
Restart Count: 0
Environment:
KUBE_NODE_NAME: (v1:spec.nodeName)
Mounts:
/csi from plugin-dir (rw)
/registration from registration-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from rook-csi-rbd-plugin-sa-token-k2rnr (ro)
csi-rbdplugin:
Container ID: docker://60d0529d7176e3256d142500e2de6120cb2ce60352127ba97baa6599d0b89a1c
Image: quay.io/cephcsi/rbdplugin:v1.0.0
Image ID: docker-pullable://quay.io/cephcsi/rbdplugin@sha256:893f9d27d3abfb3a4a263566f638af6b98c0660ad684f94ecd4e0586a90c05ec
Port: <none>
Host Port: <none>
Args:
--nodeid=$(NODE_ID)
--endpoint=$(CSI_ENDPOINT)
--v=5
--drivername=rbd.csi.ceph.com
--containerized=true
--metadatastorage=k8s_configmap
State: Waiting
Reason: RunContainerError
Last State: Terminated
Reason: ContainerCannotRun
Message: OCI runtime create failed: open /var/run/docker/runtime-runc/moby/60d0529d7176e3256d142500e2de6120cb2ce60352127ba97baa6599d0b89a1c/state.json: no such file or directory: unknown
Exit Code: 128
Started: Wed, 10 Apr 2019 10:13:53 +0530
Finished: Wed, 10 Apr 2019 10:13:53 +0530
Ready: False
Restart Count: 0
Environment:
HOST_ROOTFS: /rootfs
NODE_ID: (v1:spec.nodeName)
POD_NAMESPACE: smoke-ns-system (v1:metadata.namespace)
CSI_ENDPOINT: unix:///csi/csi.sock
Mounts:
/csi from plugin-dir (rw)
/dev from host-dev (rw)
/lib/modules from lib-modules (ro)
/rootfs from host-rootfs (rw)
/sys from host-sys (rw)
/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/ from plugin-mount-dir (rw)
/var/lib/kubelet/pods from pods-mount-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from rook-csi-rbd-plugin-sa-token-k2rnr (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
plugin-dir:
Type: HostPath (bare host directory volume)
Path: /var/lib/kubelet/plugins/rbd.csi.ceph.com
HostPathType: DirectoryOrCreate
plugin-mount-dir:
Type: HostPath (bare host directory volume)
Path: /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/
HostPathType: DirectoryOrCreate
registration-dir:
Type: HostPath (bare host directory volume)
Path: /var/lib/kubelet/plugins_registry/
HostPathType: Directory
pods-mount-dir:
Type: HostPath (bare host directory volume)
Path: /var/lib/kubelet/pods
HostPathType: Directory
host-dev:
Type: HostPath (bare host directory volume)
Path: /dev
HostPathType:
host-rootfs:
Type: HostPath (bare host directory volume)
Path: /
HostPathType:
host-sys:
Type: HostPath (bare host directory volume)
Path: /sys
HostPathType:
lib-modules:
Type: HostPath (bare host directory volume)
Path: /lib/modules
HostPathType:
rook-csi-rbd-plugin-sa-token-k2rnr:
Type: Secret (a volume populated by a Secret)
SecretName: rook-csi-rbd-plugin-sa-token-k2rnr
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/disk-pressure:NoSchedule
node.kubernetes.io/memory-pressure:NoSchedule
node.kubernetes.io/network-unavailable:NoSchedule
node.kubernetes.io/not-ready:NoExecute
node.kubernetes.io/unreachable:NoExecute
node.kubernetes.io/unschedulable:NoSchedule
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m1s default-scheduler Successfully assigned smoke-ns-system/csi-rbdplugin-vw4ch to minikube
Normal Pulling 2m kubelet, minikube pulling image "quay.io/k8scsi/csi-node-driver-registrar:v1.0.2"
Normal Pulled 114s kubelet, minikube Successfully pulled image "quay.io/k8scsi/csi-node-driver-registrar:v1.0.2"
Normal Created 114s kubelet, minikube Created container
Normal Started 114s kubelet, minikube Started container
Normal Pulling 114s kubelet, minikube pulling image "quay.io/cephcsi/rbdplugin:v1.0.0"
Normal Pulled 64s kubelet, minikube Successfully pulled image "quay.io/cephcsi/rbdplugin:v1.0.0"
Normal Created 64s kubelet, minikube Created container
Warning Failed 64s kubelet, minikube Error: failed to start container "csi-rbdplugin": Error response from daemon: OCI runtime create failed: open /var/run/docker/runtime-runc/moby/csi-rbdplugin/state.json: no such file or directory: unknown
- kubelet service status
kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/kubelet.service.d
鈹斺攢10-kubeadm.conf
Active: active (running) since Wed 2019-04-10 04:39:41 UTC; 7min ago
Docs: http://kubernetes.io/docs/
Main PID: 3381 (kubelet)
Tasks: 22 (limit: 4915)
CGroup: /system.slice/kubelet.service
鈹斺攢3381 /usr/bin/kubelet --allow-privileged=true --authorization-mode=Webhook --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --cgroup-driver=cgroupfs --client-ca-file=/var/lib/minikube/certs/ca.crt --cluster-dns=10.96.0.10 --cluster-domain=cluster.local --container-runtime=docker --fail-swap-on=false --hostname-override=minikube --kubeconfig=/etc/kubernetes/kubelet.conf --pod-manifest-path=/etc/kubernetes/manifests
Apr 10 04:46:53 minikube kubelet[3381]: E0410 04:46:53.533805 3381 pod_workers.go:190] Error syncing pod 1c5a8cbb-5b4b-11e9-ac49-08002742bbfb ("csi-cephfsplugin-cg764_smoke-ns-system(1c5a8cbb-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "4c6f8fc54c8b91c59ae60c70b073dfe73939e4bd83344106feffd315661d2473" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-cephfsplugin-cg764_smoke-ns-system(1c5a8cbb-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:46:54 minikube kubelet[3381]: E0410 04:46:54.530571 3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-rbdplugin-attacher-0_smoke-ns-system(1c5609c5-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:46:54 minikube kubelet[3381]: E0410 04:46:54.530706 3381 pod_workers.go:190] Error syncing pod 1c5609c5-5b4b-11e9-ac49-08002742bbfb ("csi-rbdplugin-attacher-0_smoke-ns-system(1c5609c5-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "392b8ee072b7bf80e79df45121440c891aaf8de230aa53964ef24b60bb92a776" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-rbdplugin-attacher-0_smoke-ns-system(1c5609c5-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:46:54 minikube kubelet[3381]: E0410 04:46:54.531381 3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "coredns-86c58d9df4-58lxc_kube-system(b7b01c12-5b4a-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:46:54 minikube kubelet[3381]: E0410 04:46:54.531477 3381 pod_workers.go:190] Error syncing pod b7b01c12-5b4a-11e9-ac49-08002742bbfb ("coredns-86c58d9df4-58lxc_kube-system(b7b01c12-5b4a-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "1fc8c7d721cae6017c0db69dc384eb0dcb08e35a0d7cf345ace875a6d3e9da33" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"coredns-86c58d9df4-58lxc_kube-system(b7b01c12-5b4a-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:46:55 minikube kubelet[3381]: E0410 04:46:55.531816 3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "kube-addon-manager-minikube_kube-system(ef81cb54ca95dd4cf88bfbdf334b363b)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:46:55 minikube kubelet[3381]: E0410 04:46:55.531983 3381 pod_workers.go:190] Error syncing pod ef81cb54ca95dd4cf88bfbdf334b363b ("kube-addon-manager-minikube_kube-system(ef81cb54ca95dd4cf88bfbdf334b363b)"), skipping: failed to "ConfigPodSandbox" for "d296c674f6a728e34637935bb0b34325b87037d70c11eabb5d90a9265f7f60df" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"kube-addon-manager-minikube_kube-system(ef81cb54ca95dd4cf88bfbdf334b363b)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:46:56 minikube kubelet[3381]: E0410 04:46:56.171467 3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:46:56 minikube kubelet[3381]: E0410 04:46:56.174451 3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:46:56 minikube kubelet[3381]: E0410 04:46:56.178389 3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
- journalctl.logs
-- Logs begin at Wed 2019-04-10 04:38:40 UTC, end at Wed 2019-04-10 04:48:41 UTC. --
Apr 10 04:44:01 minikube kubelet[3381]: E0410 04:44:01.531469 3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "rook-ceph-agent-kqpzk_smoke-ns-system(1c3a0bb6-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:01 minikube kubelet[3381]: E0410 04:44:01.531745 3381 pod_workers.go:190] Error syncing pod 1c3a0bb6-5b4b-11e9-ac49-08002742bbfb ("rook-ceph-agent-kqpzk_smoke-ns-system(1c3a0bb6-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "fe4fdce7a437b5a6d8f7973a695632d75fe12959520336b2eb41a32d5e6fe7e3" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"rook-ceph-agent-kqpzk_smoke-ns-system(1c3a0bb6-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:05 minikube kernel: hpet1: lost 319 rtc interrupts
Apr 10 04:44:05 minikube kubelet[3381]: E0410 04:44:05.535691 3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-rbdplugin-vw4ch_smoke-ns-system(1c45ab18-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:05 minikube kubelet[3381]: E0410 04:44:05.536757 3381 pod_workers.go:190] Error syncing pod 1c45ab18-5b4b-11e9-ac49-08002742bbfb ("csi-rbdplugin-vw4ch_smoke-ns-system(1c45ab18-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "10c39abb63b7f6489c04354072eca6b9dee7f2b1c3c48edefb871499fa780fb8" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-rbdplugin-vw4ch_smoke-ns-system(1c45ab18-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:06 minikube kubelet[3381]: E0410 04:44:06.172876 3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:06 minikube kubelet[3381]: E0410 04:44:06.175689 3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:06 minikube kubelet[3381]: E0410 04:44:06.181834 3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:10 minikube kernel: hpet1: lost 317 rtc interrupts
Apr 10 04:44:10 minikube kubelet[3381]: E0410 04:44:10.529817 3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "coredns-86c58d9df4-t8hhp_kube-system(b7af6cca-5b4a-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:10 minikube kubelet[3381]: E0410 04:44:10.529861 3381 pod_workers.go:190] Error syncing pod b7af6cca-5b4a-11e9-ac49-08002742bbfb ("coredns-86c58d9df4-t8hhp_kube-system(b7af6cca-5b4a-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "744911a30bb0370b176f2579eef009177a1a33657396d5739c8eae67ce5bf414" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"coredns-86c58d9df4-t8hhp_kube-system(b7af6cca-5b4a-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.532928 3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "rook-ceph-operator-7f5858cb-ngrd5_smoke-ns-system(1afbd371-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.533038 3381 pod_workers.go:190] Error syncing pod 1afbd371-5b4b-11e9-ac49-08002742bbfb ("rook-ceph-operator-7f5858cb-ngrd5_smoke-ns-system(1afbd371-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "be1ed8b86f84208e51f540c3007bcfdad491fc13f2b495276ca536cd800223a3" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"rook-ceph-operator-7f5858cb-ngrd5_smoke-ns-system(1afbd371-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.533863 3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "storage-provisioner_kube-system(b84450fa-5b4a-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.533918 3381 pod_workers.go:190] Error syncing pod b84450fa-5b4a-11e9-ac49-08002742bbfb ("storage-provisioner_kube-system(b84450fa-5b4a-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "37bf0aa77e594d17d0c4d59678265713bdc73d5c84cd4be305e7a51bd38d4eeb" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"storage-provisioner_kube-system(b84450fa-5b4a-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.877333 3381 remote_image.go:112] PullImage "ceph/ceph:v14.2.0-20190319" from image service failed: rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.877367 3381 kuberuntime_image.go:51] Pull image "ceph/ceph:v14.2.0-20190319" failed: rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.877456 3381 kuberuntime_manager.go:749] container start failed: ErrImagePull: rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.877472 3381 pod_workers.go:190] Error syncing pod 2346a0b2-5b4b-11e9-ac49-08002742bbfb ("rook-ceph-detect-version-mkvzj_smoke-ns(2346a0b2-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "StartContainer" for "version" with ErrImagePull: "rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.878328 3381 remote_image.go:112] PullImage "quay.io/k8scsi/csi-snapshotter:v1.0.1" from image service failed: rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.879367 3381 kuberuntime_image.go:51] Pull image "quay.io/k8scsi/csi-snapshotter:v1.0.1" failed: rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.879508 3381 kuberuntime_manager.go:749] container start failed: ErrImagePull: rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.879876 3381 remote_image.go:87] ImageStatus "quay.io/cephcsi/rbdplugin:v1.0.0" from image service failed: rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.879907 3381 kuberuntime_image.go:87] ImageStatus for image {"quay.io/cephcsi/rbdplugin:v1.0.0"} failed: rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.879938 3381 kuberuntime_manager.go:749] container start failed: ImageInspectError: Failed to inspect image "quay.io/cephcsi/rbdplugin:v1.0.0": rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.879950 3381 remote_image.go:112] PullImage "quay.io/cephcsi/cephfsplugin:v1.0.0" from image service failed: rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.879966 3381 kuberuntime_image.go:51] Pull image "quay.io/cephcsi/cephfsplugin:v1.0.0" failed: rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.880010 3381 kuberuntime_manager.go:749] container start failed: ErrImagePull: rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.880729 3381 pod_workers.go:190] Error syncing pod 1c4d6f79-5b4b-11e9-ac49-08002742bbfb ("csi-rbdplugin-provisioner-0_smoke-ns-system(1c4d6f79-5b4b-11e9-ac49-08002742bbfb)"), skipping: [failed to "StartContainer" for "csi-snapshotter" with ErrImagePull: "rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
Apr 10 04:44:12 minikube kubelet[3381]: , failed to "StartContainer" for "csi-rbdplugin" with ImageInspectError: "Failed to inspect image \"quay.io/cephcsi/rbdplugin:v1.0.0\": rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
Apr 10 04:44:12 minikube kubelet[3381]: ]
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.882122 3381 pod_workers.go:190] Error syncing pod 1c61e899-5b4b-11e9-ac49-08002742bbfb ("csi-cephfsplugin-provisioner-0_smoke-ns-system(1c61e899-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "StartContainer" for "csi-cephfsplugin" with ErrImagePull: "rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
Apr 10 04:44:12 minikube dockerd[2399]: time="2019-04-10T04:44:12Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/8126fe0c8da3e5ffe46df5ca5014dabc83a8f6fefa442b7bc1e3d61c4659a09a/shim.sock" debug=false pid=9076
Apr 10 04:44:12 minikube dockerd[2399]: time="2019-04-10T04:44:12Z" level=info msg="shim reaped" id=8126fe0c8da3e5ffe46df5ca5014dabc83a8f6fefa442b7bc1e3d61c4659a09a
Apr 10 04:44:12 minikube dockerd[2399]: time="2019-04-10T04:44:12.987516753Z" level=error msg="stream copy error: reading from a closed fifo"
Apr 10 04:44:12 minikube dockerd[2399]: time="2019-04-10T04:44:12.987558237Z" level=error msg="stream copy error: reading from a closed fifo"
Apr 10 04:44:13 minikube dockerd[2399]: time="2019-04-10T04:44:13.011173432Z" level=error msg="8126fe0c8da3e5ffe46df5ca5014dabc83a8f6fefa442b7bc1e3d61c4659a09a cleanup: failed to delete container from containerd: no such container"
Apr 10 04:44:13 minikube dockerd[2399]: time="2019-04-10T04:44:13.011222505Z" level=error msg="Handler for POST /v1.38/containers/8126fe0c8da3e5ffe46df5ca5014dabc83a8f6fefa442b7bc1e3d61c4659a09a/start returned error: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/8126fe0c8da3e5ffe46df5ca5014dabc83a8f6fefa442b7bc1e3d61c4659a09a/log.json: no such file or directory): docker-runc did not terminate sucessfully: unknown"
Apr 10 04:44:13 minikube kubelet[3381]: E0410 04:44:13.012419 3381 remote_runtime.go:213] StartContainer "8126fe0c8da3e5ffe46df5ca5014dabc83a8f6fefa442b7bc1e3d61c4659a09a" from runtime service failed: rpc error: code = Unknown desc = failed to start container "8126fe0c8da3e5ffe46df5ca5014dabc83a8f6fefa442b7bc1e3d61c4659a09a": Error response from daemon: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/8126fe0c8da3e5ffe46df5ca5014dabc83a8f6fefa442b7bc1e3d61c4659a09a/log.json: no such file or directory): docker-runc did not terminate sucessfully: unknown
Apr 10 04:44:13 minikube kubelet[3381]: E0410 04:44:13.012487 3381 kuberuntime_manager.go:749] container start failed: RunContainerError: failed to start container "8126fe0c8da3e5ffe46df5ca5014dabc83a8f6fefa442b7bc1e3d61c4659a09a": Error response from daemon: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/8126fe0c8da3e5ffe46df5ca5014dabc83a8f6fefa442b7bc1e3d61c4659a09a/log.json: no such file or directory): docker-runc did not terminate sucessfully: unknown
Apr 10 04:44:13 minikube kubelet[3381]: E0410 04:44:13.012509 3381 pod_workers.go:190] Error syncing pod 1c5a8cbb-5b4b-11e9-ac49-08002742bbfb ("csi-cephfsplugin-cg764_smoke-ns-system(1c5a8cbb-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "StartContainer" for "csi-cephfsplugin" with RunContainerError: "failed to start container \"8126fe0c8da3e5ffe46df5ca5014dabc83a8f6fefa442b7bc1e3d61c4659a09a\": Error response from daemon: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/8126fe0c8da3e5ffe46df5ca5014dabc83a8f6fefa442b7bc1e3d61c4659a09a/log.json: no such file or directory): docker-runc did not terminate sucessfully: unknown"
Apr 10 04:44:13 minikube kubelet[3381]: E0410 04:44:13.106412 3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "rook-ceph-detect-version-mkvzj_smoke-ns(2346a0b2-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:13 minikube kubelet[3381]: E0410 04:44:13.106462 3381 pod_workers.go:190] Error syncing pod 2346a0b2-5b4b-11e9-ac49-08002742bbfb ("rook-ceph-detect-version-mkvzj_smoke-ns(2346a0b2-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "df05f51697b0bae538debec7ba0dc66664485bf2d08b3df84678efcc0a76bbb1" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"rook-ceph-detect-version-mkvzj_smoke-ns(2346a0b2-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:13 minikube kubelet[3381]: E0410 04:44:13.107038 3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-cephfsplugin-provisioner-0_smoke-ns-system(1c61e899-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:13 minikube kubelet[3381]: E0410 04:44:13.107065 3381 pod_workers.go:190] Error syncing pod 1c61e899-5b4b-11e9-ac49-08002742bbfb ("csi-cephfsplugin-provisioner-0_smoke-ns-system(1c61e899-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "c100927f5d927fe9ef5b5c14aec9dc9059d818d91aae56d99189811dacb1ffe4" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-cephfsplugin-provisioner-0_smoke-ns-system(1c61e899-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:13 minikube kubelet[3381]: E0410 04:44:13.107828 3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-cephfsplugin-cg764_smoke-ns-system(1c5a8cbb-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:13 minikube kubelet[3381]: E0410 04:44:13.107858 3381 pod_workers.go:190] Error syncing pod 1c5a8cbb-5b4b-11e9-ac49-08002742bbfb ("csi-cephfsplugin-cg764_smoke-ns-system(1c5a8cbb-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "4c6f8fc54c8b91c59ae60c70b073dfe73939e4bd83344106feffd315661d2473" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-cephfsplugin-cg764_smoke-ns-system(1c5a8cbb-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:13 minikube kubelet[3381]: E0410 04:44:13.108944 3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-rbdplugin-provisioner-0_smoke-ns-system(1c4d6f79-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:13 minikube kubelet[3381]: E0410 04:44:13.109941 3381 pod_workers.go:190] Error syncing pod 1c4d6f79-5b4b-11e9-ac49-08002742bbfb ("csi-rbdplugin-provisioner-0_smoke-ns-system(1c4d6f79-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "a190563612f0e5cf3035ba8192209aac99b53639658f6a460f69207e696cef59" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-rbdplugin-provisioner-0_smoke-ns-system(1c4d6f79-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:13 minikube kubelet[3381]: E0410 04:44:13.530818 3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "rook-ceph-agent-kqpzk_smoke-ns-system(1c3a0bb6-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:13 minikube kubelet[3381]: E0410 04:44:13.530911 3381 pod_workers.go:190] Error syncing pod 1c3a0bb6-5b4b-11e9-ac49-08002742bbfb ("rook-ceph-agent-kqpzk_smoke-ns-system(1c3a0bb6-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "fe4fdce7a437b5a6d8f7973a695632d75fe12959520336b2eb41a32d5e6fe7e3" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"rook-ceph-agent-kqpzk_smoke-ns-system(1c3a0bb6-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:15 minikube kernel: hpet1: lost 320 rtc interrupts
Apr 10 04:44:16 minikube kubelet[3381]: E0410 04:44:16.177455 3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:16 minikube kubelet[3381]: E0410 04:44:16.183538 3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:16 minikube kubelet[3381]: E0410 04:44:16.189990 3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:16 minikube kubelet[3381]: E0410 04:44:16.531697 3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-rbdplugin-vw4ch_smoke-ns-system(1c45ab18-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:16 minikube kubelet[3381]: E0410 04:44:16.531827 3381 pod_workers.go:190] Error syncing pod 1c45ab18-5b4b-11e9-ac49-08002742bbfb ("csi-rbdplugin-vw4ch_smoke-ns-system(1c45ab18-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "10c39abb63b7f6489c04354072eca6b9dee7f2b1c3c48edefb871499fa780fb8" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-rbdplugin-vw4ch_smoke-ns-system(1c45ab18-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:20 minikube kernel: hpet1: lost 317 rtc interrupts
Apr 10 04:44:24 minikube kubelet[3381]: E0410 04:44:24.530788 3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "rook-ceph-detect-version-mkvzj_smoke-ns(2346a0b2-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:24 minikube kubelet[3381]: E0410 04:44:24.530863 3381 pod_workers.go:190] Error syncing pod 2346a0b2-5b4b-11e9-ac49-08002742bbfb ("rook-ceph-detect-version-mkvzj_smoke-ns(2346a0b2-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "df05f51697b0bae538debec7ba0dc66664485bf2d08b3df84678efcc0a76bbb1" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"rook-ceph-detect-version-mkvzj_smoke-ns(2346a0b2-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:24 minikube kubelet[3381]: E0410 04:44:24.530978 3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-cephfsplugin-provisioner-0_smoke-ns-system(1c61e899-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:24 minikube kubelet[3381]: E0410 04:44:24.531023 3381 pod_workers.go:190] Error syncing pod 1c61e899-5b4b-11e9-ac49-08002742bbfb ("csi-cephfsplugin-provisioner-0_smoke-ns-system(1c61e899-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "c100927f5d927fe9ef5b5c14aec9dc9059d818d91aae56d99189811dacb1ffe4" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-cephfsplugin-provisioner-0_smoke-ns-system(1c61e899-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:24 minikube kubelet[3381]: E0410 04:44:24.531825 3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "coredns-86c58d9df4-t8hhp_kube-system(b7af6cca-5b4a-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:24 minikube kubelet[3381]: E0410 04:44:24.531882 3381 pod_workers.go:190] Error syncing pod b7af6cca-5b4a-11e9-ac49-08002742bbfb ("coredns-86c58d9df4-t8hhp_kube-system(b7af6cca-5b4a-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "744911a30bb0370b176f2579eef009177a1a33657396d5739c8eae67ce5bf414" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"coredns-86c58d9df4-t8hhp_kube-system(b7af6cca-5b4a-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:25 minikube kernel: hpet1: lost 318 rtc interrupts
Apr 10 04:44:25 minikube kubelet[3381]: E0410 04:44:25.530237 3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "storage-provisioner_kube-system(b84450fa-5b4a-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:25 minikube kubelet[3381]: E0410 04:44:25.530391 3381 pod_workers.go:190] Error syncing pod b84450fa-5b4a-11e9-ac49-08002742bbfb ("storage-provisioner_kube-system(b84450fa-5b4a-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "37bf0aa77e594d17d0c4d59678265713bdc73d5c84cd4be305e7a51bd38d4eeb" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"storage-provisioner_kube-system(b84450fa-5b4a-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:25 minikube kubelet[3381]: E0410 04:44:25.531630 3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "rook-ceph-operator-7f5858cb-ngrd5_smoke-ns-system(1afbd371-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:25 minikube kubelet[3381]: E0410 04:44:25.531691 3381 pod_workers.go:190] Error syncing pod 1afbd371-5b4b-11e9-ac49-08002742bbfb ("rook-ceph-operator-7f5858cb-ngrd5_smoke-ns-system(1afbd371-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "be1ed8b86f84208e51f540c3007bcfdad491fc13f2b495276ca536cd800223a3" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"rook-ceph-operator-7f5858cb-ngrd5_smoke-ns-system(1afbd371-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:25 minikube kubelet[3381]: E0410 04:44:25.532512 3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-rbdplugin-provisioner-0_smoke-ns-system(1c4d6f79-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:25 minikube kubelet[3381]: E0410 04:44:25.532565 3381 pod_workers.go:190] Error syncing pod 1c4d6f79-5b4b-11e9-ac49-08002742bbfb ("csi-rbdplugin-provisioner-0_smoke-ns-system(1c4d6f79-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "a190563612f0e5cf3035ba8192209aac99b53639658f6a460f69207e696cef59" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-rbdplugin-provisioner-0_smoke-ns-system(1c4d6f79-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:26 minikube kubelet[3381]: E0410 04:44:26.179720 3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:26 minikube kubelet[3381]: E0410 04:44:26.207174 3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:26 minikube kubelet[3381]: E0410 04:44:26.229088 3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:26 minikube kubelet[3381]: E0410 04:44:26.531689 3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "rook-ceph-agent-kqpzk_smoke-ns-system(1c3a0bb6-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:26 minikube kubelet[3381]: E0410 04:44:26.531840 3381 pod_workers.go:190] Error syncing pod 1c3a0bb6-5b4b-11e9-ac49-08002742bbfb ("rook-ceph-agent-kqpzk_smoke-ns-system(1c3a0bb6-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "fe4fdce7a437b5a6d8f7973a695632d75fe12959520336b2eb41a32d5e6fe7e3" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"rook-ceph-agent-kqpzk_smoke-ns-system(1c3a0bb6-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:27 minikube audit: NETFILTER_CFG table=filter family=2 entries=41
Apr 10 04:44:27 minikube audit[9111]: SYSCALL arch=c000003e syscall=54 success=yes exit=0 a0=3 a1=0 a2=40 a3=555d88881830 items=0 ppid=4814 pid=9111 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables-restor" exe="/sbin/xtables-multi" subj=kernel key=(null)
Apr 10 04:44:27 minikube kernel: audit: type=1325 audit(1554871467.238:86): table=filter family=2 entries=41
Apr 10 04:44:27 minikube kernel: audit: type=1300 audit(1554871467.238:86): arch=c000003e syscall=54 success=yes exit=0 a0=3 a1=0 a2=40 a3=555d88881830 items=0 ppid=4814 pid=9111 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables-restor" exe="/sbin/xtables-multi" subj=kernel key=(null)
Apr 10 04:44:27 minikube audit: PROCTITLE proctitle=69707461626C65732D726573746F7265002D2D6E6F666C757368002D2D636F756E74657273
Apr 10 04:44:27 minikube kernel: audit: type=1327 audit(1554871467.238:86): proctitle=69707461626C65732D726573746F7265002D2D6E6F666C757368002D2D636F756E74657273
Apr 10 04:44:27 minikube audit: NETFILTER_CFG table=nat family=2 entries=62
Apr 10 04:44:27 minikube audit[9111]: SYSCALL arch=c000003e syscall=54 success=yes exit=0 a0=3 a1=0 a2=40 a3=555d888855f0 items=0 ppid=4814 pid=9111 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables-restor" exe="/sbin/xtables-multi" subj=kernel key=(null)
Apr 10 04:44:27 minikube kernel: audit: type=1325 audit(1554871467.242:87): table=nat family=2 entries=62
Apr 10 04:44:27 minikube kernel: audit: type=1300 audit(1554871467.242:87): arch=c000003e syscall=54 success=yes exit=0 a0=3 a1=0 a2=40 a3=555d888855f0 items=0 ppid=4814 pid=9111 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables-restor" exe="/sbin/xtables-multi" subj=kernel key=(null)
Apr 10 04:44:27 minikube audit: PROCTITLE proctitle=69707461626C65732D726573746F7265002D2D6E6F666C757368002D2D636F756E74657273
Apr 10 04:44:27 minikube kernel: audit: type=1327 audit(1554871467.242:87): proctitle=69707461626C65732D726573746F7265002D2D6E6F666C757368002D2D636F756E74657273
Apr 10 04:44:27 minikube kubelet[3381]: E0410 04:44:27.533658 3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-rbdplugin-vw4ch_smoke-ns-system(1c45ab18-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:27 minikube kubelet[3381]: E0410 04:44:27.533779 3381 pod_workers.go:190] Error syncing pod 1c45ab18-5b4b-11e9-ac49-08002742bbfb ("csi-rbdplugin-vw4ch_smoke-ns-system(1c45ab18-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "10c39abb63b7f6489c04354072eca6b9dee7f2b1c3c48edefb871499fa780fb8" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-rbdplugin-vw4ch_smoke-ns-system(1c45ab18-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:28 minikube kubelet[3381]: E0410 04:44:28.530741 3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-cephfsplugin-cg764_smoke-ns-system(1c5a8cbb-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:28 minikube kubelet[3381]: E0410 04:44:28.530917 3381 pod_workers.go:190] Error syncing pod 1c5a8cbb-5b4b-11e9-ac49-08002742bbfb ("csi-cephfsplugin-cg764_smoke-ns-system(1c5a8cbb-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "4c6f8fc54c8b91c59ae60c70b073dfe73939e4bd83344106feffd315661d2473" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-cephfsplugin-cg764_smoke-ns-system(1c5a8cbb-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:30 minikube kernel: hpet1: lost 318 rtc interrupts
Apr 10 04:44:30 minikube kubelet[3381]: E0410 04:44:30.531959 3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "rook-discover-dqdjh_smoke-ns-system(1c3c30c1-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:30 minikube kubelet[3381]: E0410 04:44:30.532120 3381 pod_workers.go:190] Error syncing pod 1c3c30c1-5b4b-11e9-ac49-08002742bbfb ("rook-discover-dqdjh_smoke-ns-system(1c3c30c1-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "d459439f566c6bb3db20d21694b6988ba9fd42d0b62a4e3942d9c41227773a6b" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"rook-discover-dqdjh_smoke-ns-system(1c3c30c1-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:32 minikube kubelet[3381]: E0410 04:44:32.530760 3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "kube-controller-manager-minikube_kube-system(17eea6fd9342634d7d40a04d577641fd)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:32 minikube kubelet[3381]: E0410 04:44:32.530916 3381 pod_workers.go:190] Error syncing pod 17eea6fd9342634d7d40a04d577641fd ("kube-controller-manager-minikube_kube-system(17eea6fd9342634d7d40a04d577641fd)"), skipping: failed to "ConfigPodSandbox" for "146305c699f6d8b8260bce39b88f656fb86ca3043b90c653f8f3fb275ab16c85" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"kube-controller-manager-minikube_kube-system(17eea6fd9342634d7d40a04d577641fd)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:32 minikube kubelet[3381]: E0410 04:44:32.532209 3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-rbdplugin-attacher-0_smoke-ns-system(1c5609c5-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:32 minikube kubelet[3381]: E0410 04:44:32.532323 3381 pod_workers.go:190] Error syncing pod 1c5609c5-5b4b-11e9-ac49-08002742bbfb ("csi-rbdplugin-attacher-0_smoke-ns-system(1c5609c5-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "392b8ee072b7bf80e79df45121440c891aaf8de230aa53964ef24b60bb92a776" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-rbdplugin-attacher-0_smoke-ns-system(1c5609c5-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:35 minikube kernel: hpet1: lost 318 rtc interrupts
Apr 10 04:44:36 minikube kubelet[3381]: E0410 04:44:36.174051 3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:36 minikube kubelet[3381]: E0410 04:44:36.177904 3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:36 minikube kubelet[3381]: E0410 04:44:36.187510 3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:36 minikube kubelet[3381]: E0410 04:44:36.530998 3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "rook-ceph-operator-7f5858cb-ngrd5_smoke-ns-system(1afbd371-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:36 minikube kubelet[3381]: E0410 04:44:36.531070 3381 pod_workers.go:190] Error syncing pod 1afbd371-5b4b-11e9-ac49-08002742bbfb ("rook-ceph-operator-7f5858cb-ngrd5_smoke-ns-system(1afbd371-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "be1ed8b86f84208e51f540c3007bcfdad491fc13f2b495276ca536cd800223a3" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"rook-ceph-operator-7f5858cb-ngrd5_smoke-ns-system(1afbd371-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:36 minikube kubelet[3381]: E0410 04:44:36.531773 3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-rbdplugin-provisioner-0_smoke-ns-system(1c4d6f79-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:36 minikube kubelet[3381]: E0410 04:44:36.531809 3381 pod_workers.go:190] Error syncing pod 1c4d6f79-5b4b-11e9-ac49-08002742bbfb ("csi-rbdplugin-provisioner-0_smoke-ns-system(1c4d6f79-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "a190563612f0e5cf3035ba8192209aac99b53639658f6a460f69207e696cef59" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-rbdplugin-provisioner-0_smoke-ns-system(1c4d6f79-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:36 minikube kubelet[3381]: E0410 04:44:36.532157 3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "storage-provisioner_kube-system(b84450fa-5b4a-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:36 minikube kubelet[3381]: E0410 04:44:36.532203 3381 pod_workers.go:190] Error syncing pod b84450fa-5b4a-11e9-ac49-08002742bbfb ("storage-provisioner_kube-system(b84450fa-5b4a-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "37bf0aa77e594d17d0c4d59678265713bdc73d5c84cd4be305e7a51bd38d4eeb" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"storage-provisioner_kube-system(b84450fa-5b4a-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:37 minikube kubelet[3381]: E0410 04:44:37.531047 3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "rook-ceph-detect-version-mkvzj_smoke-ns(2346a0b2-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:37 minikube kubelet[3381]: E0410 04:44:37.531176 3381 pod_workers.go:190] Error syncing pod 2346a0b2-5b4b-11e9-ac49-08002742bbfb ("rook-ceph-detect-version-mkvzj_smoke-ns(2346a0b2-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "df05f51697b0bae538debec7ba0dc66664485bf2d08b3df84678efcc0a76bbb1" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"rook-ceph-detect-version-mkvzj_smoke-ns(2346a0b2-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:38 minikube kubelet[3381]: E0410 04:44:38.535568 3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "kube-scheduler-minikube_kube-system(4b52d75cab61380f07c0c5a69fb371d4)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:38 minikube kubelet[3381]: E0410 04:44:38.535734 3381 pod_workers.go:190] Error syncing pod 4b52d75cab61380f07c0c5a69fb371d4 ("kube-scheduler-minikube_kube-system(4b52d75cab61380f07c0c5a69fb371d4)"), skipping: failed to "ConfigPodSandbox" for "01457429d7542320cf53936ab4023e78fb08cadccd55dde9412ea53ae5161936" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"kube-scheduler-minikube_kube-system(4b52d75cab61380f07c0c5a69fb371d4)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:39 minikube kubelet[3381]: E0410 04:44:39.554986 3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-cephfsplugin-provisioner-0_smoke-ns-system(1c61e899-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:39 minikube kubelet[3381]: E0410 04:44:39.555594 3381 pod_workers.go:190] Error syncing pod 1c61e899-5b4b-11e9-ac49-08002742bbfb ("csi-cephfsplugin-provisioner-0_smoke-ns-system(1c61e899-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "c100927f5d927fe9ef5b5c14aec9dc9059d818d91aae56d99189811dacb1ffe4" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-cephfsplugin-provisioner-0_smoke-ns-system(1c61e899-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:39 minikube kubelet[3381]: E0410 04:44:39.555118 3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "rook-ceph-agent-kqpzk_smoke-ns-system(1c3a0bb6-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:39 minikube kubelet[3381]: E0410 04:44:39.555737 3381 pod_workers.go:190] Error syncing pod 1c3a0bb6-5b4b-11e9-ac49-08002742bbfb ("rook-ceph-agent-kqpzk_smoke-ns-system(1c3a0bb6-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "fe4fdce7a437b5a6d8f7973a695632d75fe12959520336b2eb41a32d5e6fe7e3" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"rook-ceph-agent-kqpzk_smoke-ns-system(1c3a0bb6-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:39 minikube kubelet[3381]: E0410 04:44:39.556946 3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "coredns-86c58d9df4-t8hhp_kube-system(b7af6cca-5b4a-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:39 minikube kubelet[3381]: E0410 04:44:39.557064 3381 pod_workers.go:190] Error syncing pod b7af6cca-5b4a-11e9-ac49-08002742bbfb ("coredns-86c58d9df4-t8hhp_kube-system(b7af6cca-5b4a-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "744911a30bb0370b176f2579eef009177a1a33657396d5739c8eae67ce5bf414" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"coredns-86c58d9df4-t8hhp_kube-system(b7af6cca-5b4a-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:40 minikube kernel: hpet1: lost 318 rtc interrupts
Apr 10 04:44:42 minikube kubelet[3381]: E0410 04:44:42.539856 3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-rbdplugin-vw4ch_smoke-ns-system(1c45ab18-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:42 minikube kubelet[3381]: E0410 04:44:42.540295 3381 pod_workers.go:190] Error syncing pod 1c45ab18-5b4b-11e9-ac49-08002742bbfb ("csi-rbdplugin-vw4ch_smoke-ns-system(1c45ab18-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "10c39abb63b7f6489c04354072eca6b9dee7f2b1c3c48edefb871499fa780fb8" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-rbdplugin-vw4ch_smoke-ns-system(1c45ab18-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:43 minikube kubelet[3381]: E0410 04:44:43.537196 3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-cephfsplugin-cg764_smoke-ns-system(1c5a8cbb-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:43 minikube kubelet[3381]: E0410 04:44:43.537281 3381 pod_workers.go:190] Error syncing pod 1c5a8cbb-5b4b-11e9-ac49-08002742bbfb ("csi-cephfsplugin-cg764_smoke-ns-system(1c5a8cbb-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "4c6f8fc54c8b91c59ae60c70b073dfe73939e4bd83344106feffd315661d2473" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-cephfsplugin-cg764_smoke-ns-system(1c5a8cbb-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:45 minikube kernel: hpet1: lost 320 rtc interrupts
Apr 10 04:44:45 minikube kubelet[3381]: E0410 04:44:45.530651 3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "rook-discover-dqdjh_smoke-ns-system(1c3c30c1-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:45 minikube kubelet[3381]: E0410 04:44:45.530799 3381 pod_workers.go:190] Error syncing pod 1c3c30c1-5b4b-11e9-ac49-08002742bbfb ("rook-discover-dqdjh_smoke-ns-system(1c3c30c1-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "d459439f566c6bb3db20d21694b6988ba9fd42d0b62a4e3942d9c41227773a6b" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"rook-discover-dqdjh_smoke-ns-system(1c3c30c1-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:46 minikube kubelet[3381]: E0410 04:44:46.177820 3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:46 minikube kubelet[3381]: E0410 04:44:46.189574 3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:46 minikube kubelet[3381]: E0410 04:44:46.198624 3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:46 minikube kubelet[3381]: E0410 04:44:46.530807 3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "kube-controller-manager-minikube_kube-system(17eea6fd9342634d7d40a04d577641fd)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:46 minikube kubelet[3381]: E0410 04:44:46.530952 3381 pod_workers.go:190] Error syncing pod 17eea6fd9342634d7d40a04d577641fd ("kube-controller-manager-minikube_kube-system(17eea6fd9342634d7d40a04d577641fd)"), skipping: failed to "ConfigPodSandbox" for "146305c699f6d8b8260bce39b88f656fb86ca3043b90c653f8f3fb275ab16c85" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"kube-controller-manager-minikube_kube-system(17eea6fd9342634d7d40a04d577641fd)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:47 minikube kubelet[3381]: E0410 04:44:47.530410 3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-rbdplugin-attacher-0_smoke-ns-system(1c5609c5-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:47 minikube kubelet[3381]: E0410 04:44:47.530630 3381 pod_workers.go:190] Error syncing pod 1c5609c5-5b4b-11e9-ac49-08002742bbfb ("csi-rbdplugin-attacher-0_smoke-ns-system(1c5609c5-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "392b8ee072b7bf80e79df45121440c891aaf8de230aa53964ef24b60bb92a776" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-rbdplugin-attacher-0_smoke-ns-system(1c5609c5-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:48 minikube kubelet[3381]: E0410 04:44:48.532267 3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-rbdplugin-provisioner-0_smoke-ns-system(1c4d6f79-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:48 minikube kubelet[3381]: E0410 04:44:48.532593 3381 pod_workers.go:190] Error syncing pod 1c4d6f79-5b4b-11e9-ac49-08002742bbfb ("csi-rbdplugin-provisioner-0_smoke-ns-system(1c4d6f79-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "a190563612f0e5cf3035ba8192209aac99b53639658f6a460f69207e696cef59" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-rbdplugin-provisioner-0_smoke-ns-system(1c4d6f79-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:48 minikube kubelet[3381]: E0410 04:44:48.533392 3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "rook-ceph-operator-7f5858cb-ngrd5_smoke-ns-system(1afbd371-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:48 minikube kubelet[3381]: E0410 04:44:48.533490 3381 pod_workers.go:190] Error syncing pod 1afbd371-5b4b-11e9-ac49-08002742bbfb ("rook-ceph-operator-7f5858cb-ngrd5_smoke-ns-system(1afbd371-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "be1ed8b86f84208e51f540c3007bcfdad491fc13f2b495276ca536cd800223a3" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"rook-ceph-operator-7f5858cb-ngrd5_smoke-ns-system(1afbd371-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:49 minikube kubelet[3381]: E0410 04:44:49.536147 3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "storage-provisioner_kube-system(b84450fa-5b4a-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:49 minikube kubelet[3381]: E0410 04:44:49.536487 3381 pod_workers.go:190] Error syncing pod b84450fa-5b4a-11e9-ac49-08002742bbfb ("storage-provisioner_kube-system(b84450fa-5b4a-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "37bf0aa77e594d17d0c4d59678265713bdc73d5c84cd4be305e7a51bd38d4eeb" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"storage-provisioner_kube-system(b84450fa-5b4a-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:50 minikube kernel: hpet1: lost 317 rtc interrupts
Apr 10 04:44:50 minikube kubelet[3381]: E0410 04:44:50.534434 3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "rook-ceph-agent-kqpzk_smoke-ns-system(1c3a0bb6-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:50 minikube kubelet[3381]: E0410 04:44:50.534826 3381 pod_workers.go:190] Error syncing pod 1c3a0bb6-5b4b-11e9-ac49-08002742bbfb ("rook-ceph-agent-kqpzk_smoke-ns-system(1c3a0bb6-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "fe4fdce7a437b5a6d8f7973a695632d75fe12959520336b2eb41a32d5e6fe7e3" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"rook-ceph-agent-kqpzk_smoke-ns-system(1c3a0bb6-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:50 minikube kubelet[3381]: E0410 04:44:50.536539 3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-cephfsplugin-provisioner-0_smoke-ns-system(1c61e899-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:50 minikube kubelet[3381]: E0410 04:44:50.536712 3381 pod_workers.go:190] Error syncing pod 1c61e899-5b4b-11e9-ac49-08002742bbfb ("csi-cephfsplugin-provisioner-0_smoke-ns-system(1c61e899-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "c100927f5d927fe9ef5b5c14aec9dc9059d818d91aae56d99189811dacb1ffe4" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-cephfsplugin-provisioner-0_smoke-ns-system(1c61e899-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:51 minikube kubelet[3381]: E0410 04:44:51.545519 3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "rook-ceph-detect-version-mkvzj_smoke-ns(2346a0b2-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:51 minikube kubelet[3381]: E0410 04:44:51.546282 3381 pod_workers.go:190] Error syncing pod 2346a0b2-5b4b-11e9-ac49-08002742bbfb ("rook-ceph-detect-version-mkvzj_smoke-ns(2346a0b2-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "df05f51697b0bae538debec7ba0dc66664485bf2d08b3df84678efcc0a76bbb1" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"rook-ceph-detect-version-mkvzj_smoke-ns(2346a0b2-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:51 minikube kubelet[3381]: E0410 04:44:51.557738 3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "kube-addon-manager-minikube_kube-system(ef81cb54ca95dd4cf88bfbdf334b363b)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:51 minikube kubelet[3381]: E0410 04:44:51.557931 3381 pod_workers.go:190] Error syncing pod ef81cb54ca95dd4cf88bfbdf334b363b ("kube-addon-manager-minikube_kube-system(ef81cb54ca95dd4cf88bfbdf334b363b)"), skipping: failed to "ConfigPodSandbox" for "d296c674f6a728e34637935bb0b34325b87037d70c11eabb5d90a9265f7f60df" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"kube-addon-manager-minikube_kube-system(ef81cb54ca95dd4cf88bfbdf334b363b)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:53 minikube kubelet[3381]: E0410 04:44:53.535959 3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "kube-scheduler-minikube_kube-system(4b52d75cab61380f07c0c5a69fb371d4)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:53 minikube kubelet[3381]: E0410 04:44:53.536385 3381 pod_workers.go:190] Error syncing pod 4b52d75cab61380f07c0c5a69fb371d4 ("kube-scheduler-minikube_kube-system(4b52d75cab61380f07c0c5a69fb371d4)"), skipping: failed to "ConfigPodSandbox" for "01457429d7542320cf53936ab4023e78fb08cadccd55dde9412ea53ae5161936" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"kube-scheduler-minikube_kube-system(4b52d75cab61380f07c0c5a69fb371d4)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:53 minikube kubelet[3381]: E0410 04:44:53.539699 3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "kube-proxy-gts8g_kube-system(b7e6760f-5b4a-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:53 minikube kubelet[3381]: E0410 04:44:53.540229 3381 pod_workers.go:190] Error syncing pod b7e6760f-5b4a-11e9-ac49-08002742bbfb ("kube-proxy-gts8g_kube-system(b7e6760f-5b4a-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "8dccb241058a7c51d45d93f4a6f58d268d2d8a7b8b6dd9f44a2a24b6000b75b4" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"kube-proxy-gts8g_kube-system(b7e6760f-5b4a-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:54 minikube kubelet[3381]: E0410 04:44:54.530768 3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-rbdplugin-vw4ch_smoke-ns-system(1c45ab18-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:54 minikube kubelet[3381]: E0410 04:44:54.530873 3381 pod_workers.go:190] Error syncing pod 1c45ab18-5b4b-11e9-ac49-08002742bbfb ("csi-rbdplugin-vw4ch_smoke-ns-system(1c45ab18-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "10c39abb63b7f6489c04354072eca6b9dee7f2b1c3c48edefb871499fa780fb8" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-rbdplugin-vw4ch_smoke-ns-system(1c45ab18-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:54 minikube kubelet[3381]: E0410 04:44:54.531639 3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "coredns-86c58d9df4-t8hhp_kube-system(b7af6cca-5b4a-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:54 minikube kubelet[3381]: E0410 04:44:54.532034 3381 pod_workers.go:190] Error syncing pod b7af6cca-5b4a-11e9-ac49-08002742bbfb ("coredns-86c58d9df4-t8hhp_kube-system(b7af6cca-5b4a-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "744911a30bb0370b176f2579eef009177a1a33657396d5739c8eae67ce5bf414" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"coredns-86c58d9df4-t8hhp_kube-system(b7af6cca-5b4a-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:55 minikube kernel: hpet1: lost 319 rtc interrupts
Apr 10 04:44:56 minikube kubelet[3381]: E0410 04:44:56.174345 3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:56 minikube kubelet[3381]: E0410 04:44:56.187741 3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:56 minikube kubelet[3381]: E0410 04:44:56.194960 3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:56 minikube kubelet[3381]: E0410 04:44:56.531670 3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "rook-discover-dqdjh_smoke-ns-system(1c3c30c1-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:56 minikube kubelet[3381]: E0410 04:44:56.532533 3381 pod_workers.go:190] Error syncing pod 1c3c30c1-5b4b-11e9-ac49-08002742bbfb ("rook-discover-dqdjh_smoke-ns-system(1c3c30c1-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "d459439f566c6bb3db20d21694b6988ba9fd42d0b62a4e3942d9c41227773a6b" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"rook-discover-dqdjh_smoke-ns-system(1c3c30c1-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:57 minikube audit: NETFILTER_CFG table=filter family=2 entries=41
Apr 10 04:44:57 minikube audit[9177]: SYSCALL arch=c000003e syscall=54 success=yes exit=0 a0=3 a1=0 a2=40 a3=5608a3e43830 items=0 ppid=4814 pid=9177 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables-restor" exe="/sbin/xtables-multi" subj=kernel key=(null)
Apr 10 04:44:57 minikube audit: PROCTITLE proctitle=69707461626C65732D726573746F7265002D2D6E6F666C757368002D2D636F756E74657273
Apr 10 04:44:57 minikube kernel: audit: type=1325 audit(1554871497.289:88): table=filter family=2 entries=41
Apr 10 04:44:57 minikube kernel: audit: type=1300 audit(1554871497.289:88): arch=c000003e syscall=54 success=yes exit=0 a0=3 a1=0 a2=40 a3=5608a3e43830 items=0 ppid=4814 pid=9177 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables-restor" exe="/sbin/xtables-multi" subj=kernel key=(null)
Apr 10 04:44:57 minikube kernel: audit: type=1327 audit(1554871497.289:88): proctitle=69707461626C65732D726573746F7265002D2D6E6F666C757368002D2D636F756E74657273
Apr 10 04:44:57 minikube audit: NETFILTER_CFG table=nat family=2 entries=62
Madhu-1
All 27 comments
I'm not yet familiar with bi-directional mounts. Any chance you could post instructions to reproduce this bug?
tstromberg
on 11 Apr 2019
https://github.com/rook/rook/blob/master/cluster/examples/kubernetes/ceph/csi/template/cephfs/csi-cephfsplugin.yaml#L71 this is the bi-directional mount used in the pod template
the full template is here https://github.com/rook/rook/blob/master/cluster/examples/kubernetes/ceph/csi/template/cephfs/csi-cephfsplugin.yaml
Madhu-1
on 9 May 2019
I'm seeing something similar maybe. I fired up minikube 1.1.1 and tried to load in a csi driver of my own and odd things start happening immediately after the driver registers. containers fail to start complaining /etc/resolv.conf doesn't exist anymore. logging onto the host, and catting it, says it doesn't exist either. very strange. I confirmed it doesn't do this outside of minikube. Should I file this as a separate issue, or keep it as part of this one?
kfox1111
on 20 Jun 2019
@Madhu-1 Have you enabled the mount propagation?
- https://github.com/kubernetes-csi/docs/blob/735f1ef4adfcb157afce47c64d750b71012c8151/book/src/Setup.md#enabling-mount-propagation
- https://kubernetes.io/docs/concepts/storage/volumes/#mount-propagation
@tstromberg , Is there any way to pass MountFlags=shared to docker systemd unit file inside minikube?
giovanism
on 30 Jun 2019
Still seeing this with minikube 1.2.0.
Reproducer:
minikube start --feature-gates CSIInlineVolume=true
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/csi-driver-image-populator/master/deploy/image/csi-image-csidriverinfo.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/csi-driver-image-populator/master/deploy/image/csi-image-daemonset.yaml
kubectl get pods #shows the pod never fully starting. You start to get weird errors like:
minikube ssh cat /etc/resolv.conf
cat: /etc/resolv.conf: No such file or directory
ssh: Process exited with status 1
kfox1111
on 3 Jul 2019
minikube ssh
sudo su -
mkdir -p /etc/systemd/system/docker.service.d/
(echo '[Service]'; echo 'MountFlags=shared') > /etc/systemd/system/docker.service.d/mount_flags.conf
systemctl daemon-reload
systemctl restart docker
systemctl restart kubelet
systemctl show docker | grep Mount
#MountFlags=shared
exit
exit
#no workie
minikube stop
minikube start
#no workie
#/etc/systemd/system/docker.service.d/mount_flags.conf is missing after reboot.
👍2
Ah.... I found this: https://github.com/opencontainers/runc/issues/1961
inspecting, it looks like it is the issue:
# mount | grep root
rootfs on / type rootfs (rw)
# df -h /
Filesystem Size Used Avail Use% Mounted on
overlay 0 0 0 - /
#
I can confirm it with cri-o too:
minikube start -p crio --container-runtime=cri-o
#confirm that it is cri-o based
minikube -p crio ssh ps ax | grep crio
1988 ? Ssl 0:13 /usr/bin/crio --log-level=debug --insecure-registry 10.96.0.0/12
2860 ? Ssl 0:00 /usr/libexec/crio/conmon --syslog -c 4e35107dd63ff5eef5b0adf9d6eb442f6a88429c40ce702523dae2a22ea99bdf -n k8s_POD_kube-addon-manager-minikube_kube-system_65a31d2b812b11a2035f37c8a742e46
2891 ? Ssl 0:00 /usr/libexec/crio/conmon --syslog -c 085c98ce9552d32c5d8d0966a72685fd6753e547ddbea7f9d93acd3d04ac8b54 -n k8s_kube-addon-manager_kube-addon-manager-minikube_kube-system_65a31d2b812b11a2
#check the mount
minikube -p crio ssh -- df -h /
Filesystem Size Used Avail Use% Mounted on
rootfs 0 0 0 - /
#apply the driver
kubectl apply -f git/csi-driver-image-populator/deploy/image/csi-image-daemonset.yaml
#wait a bit for it to sort of start/crash
minikube -p crio ssh -- df -h /
Filesystem Size Used Avail Use% Mounted on
overlay 0 0 0 - /
Is there any way to switch the root filesystem away from rootfs?
kfox1111
on 18 Jul 2019
I faced the same problem (csi driver not working) with https://github.com/deislabs/secrets-store-csi-driver/issues/49 .
ryysud
on 9 Aug 2019
Is there anything we can do to help with this issue? Provide more context / ... Basically CSI is not working for minikube which seems like a big deal (given minikube tries to be a complete kubernetes setup)?
entropitor
on 12 Aug 2019
More debugging in https://github.com/kubernetes/minikube/issues/3512
Looks like its not necessarily related to rootfs.
Its very odd. Its like mounts intended for the containers are leaking back out to the host.
kfox1111
on 14 Aug 2019
Comparing mounts before and after, it looks like its duplicating a lot of mounts... something really odd is happening.
kfox1111
on 14 Aug 2019
So, I tried looking at a box with working propagation:
findmnt -o TARGET,PROPAGATION | grep -v shared
TARGET PROPAGATION
鈹溾攢/var/lib/docker/containers private
鈹斺攢/var/lib/docker/overlay2 private
It has two private mounts. Tried this on minikube:
mount --bind /var/lib/docker /var/lib/docker
mount --make-private /var/lib/docker
mount --bind /var/lib/docker/containers /var/lib/docker/containers
mount --make-private /var/lib/docker/containers
mount --bind /var/lib/docker/overlay2 /var/lib/docker/overlay2
mount --make-private /var/lib/docker/overlay2
No joy.
Container mounts are still leaking out to the host:
$ mount | grep resolv
/dev/vda1 on /etc/resolv.conf type ext4 (rw,relatime,data=ordered)
Ok. I got it to work!
I had to also remove Environment=DOCKER_RAMDISK=yes from /usr/lib/systemd/system/docker.service
Next I need to unwind a bit and see which of the things above are really required and which are not.
kfox1111
on 14 Aug 2019
@kfox1111 how do you override /usr/lib/systemd/system/docker.service, DOCKER_RAMDISK=yes comes back after reboot for me.
Thanks.
leseb
on 20 Aug 2019
@kfox1111 how do you override
/usr/lib/systemd/system/docker.service,DOCKER_RAMDISK=yescomes back after reboot for me.
I shutdown the vm. I then manually started it in libvirt, attached to the console, logged in. I edited /usr/lib/systemd/system/docker.service. Then systemctl daemon-reload; systemctl restart docker;
Then you: minikube start
It will overwrite the docker.service, but since docker is already running it skips starting it. Everything then came up ok.
kfox1111
on 20 Aug 2019
👍1
Hi,
I am experiencing the same issue, but I am afraid I am abit confused by the thread ^ . Is there any chance someone could summarize the workaround?
JanMatas
on 23 Aug 2019
Its fairly complicated to get the fix in place. You need to extract the iso, extract the initrd from it, update the init script, then rebuild the initrd, and then rebuild the iso from it. overwrite the iso from a previously created minikube, boot the vm not from minikube start but manually, login through the console, update docker.service to remove the environment variable, systemctl daemon-reload and restart the docker service, then minikube start.
The fix should be making it into minikube 1.4.0 so it may be easier to wait unless you really need it now.
kfox1111
on 23 Aug 2019
Any ETA on Minikube 1.4.0? How do you know the fix is making it into 1.4.0 @kfox1111 ?
doprdele
on 28 Aug 2019
There will be a beta ISO out soon
afbjorklund
on 28 Aug 2019
🎉1
@doprdele https://github.com/kubernetes/minikube/pull/5133 merged.
kfox1111
on 28 Aug 2019
@madhu-1I believe this issue is now addressed by minikube v1.4, as it the necessary fixes have been merged. If you still see this issue with minikube v1.4 or higher, please reopen this issue by commenting with /reopen
Thank you for reporting this issue!
tstromberg
on 20 Sep 2019
🎉1
/reopen
Lanayx
on 29 Nov 2019
@Lanayx: You can't reopen an issue/PR unless you authored it or you are a collaborator.
In response to this:
/reopen
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
k8s-ci-robot
on 29 Nov 2019
Hi, I'm getting the same symptoms with minikube 1.5.2 on windows+hyperv, when trying to install rook cluster
Lanayx
on 29 Nov 2019
With minikube-v1.6.0-beta.1 it works for me
Lanayx
on 30 Nov 2019
Was this page helpful?
0 / 5 - 0 ratings
Related issues
jroucheton
路
3Comments
mdkess
路
3Comments
StasPerekrestov
路
3Comments
mrtkp9993
路
3Comments
tstromberg
路
3Comments
Most helpful comment