Minikube: Unable to pull images: /var/lib/docker - input/output error, read-only file system

[poojapatel10]

minikube start
😄 minikube v0.34.1 on darwin (amd64)
🔥 Creating virtualbox VM (CPUs=2, Memory=2048MB, Disk=20000MB) ...
💿 Downloading Minikube ISO ...
184.30 MB / 184.30 MB [============================================] 100.00% 0s
📶 "minikube" IP address is 192.168.99.109
🐳 Configuring Docker as the container runtime ...
✨ Preparing Kubernetes environment ...
💾 Downloading kubelet v1.13.3
💾 Downloading kubeadm v1.13.3
🚜 Pulling images required by Kubernetes v1.13.3 ...
❌ Unable to pull images, which may be OK: running cmd: sudo kubeadm config images pull --config /var/lib/kubeadm.yaml: command failed: sudo kubeadm config images pull --config /var/lib/kubeadm.yaml
stdout: [config/images] Pulled k8s.gcr.io/kube-apiserver:v1.13.3
[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.13.3
[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.13.3

stderr: failed to pull image "k8s.gcr.io/kube-proxy:v1.13.3": output: v1.13.3: Pulling from kube-proxy
73e3e9d78c61: Already exists
0c440f353724: Pulling fs layer
5764bbd93c04: Pulling fs layer
0c440f353724: Verifying Checksum
0c440f353724: Download complete
5764bbd93c04: Verifying Checksum
5764bbd93c04: Download complete
failed to register layer: open /var/lib/docker/image/overlay2/layerdb/tmp/write-set-474937986/diff: read-only file system
, error: exit status 1
: Process exited with status 1
🚀 Launching Kubernetes v1.13.3 using kubeadm ...
💣 Error starting cluster: kubeadm init:
sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data-minikube --ignore-preflight-errors=Port-10250 --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI

[init] Using Kubernetes version: v1.13.3
[preflight] Running pre-flight checks
[WARNING Service-Docker]: docker service is not enabled, please run 'systemctl enable docker.service'
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING DirAvailable--data-minikube]: unable to check if /data/minikube is empty: open /data/minikube: input/output error
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.13.3: output: v1.13.3: Pulling from kube-apiserver
73e3e9d78c61: Pulling fs layer
d261e2f8ca5b: Pulling fs layer
open /var/lib/docker/tmp/GetImageBlob139180537: input/output error
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-controller-manager:v1.13.3: output: v1.13.3: Pulling from kube-controller-manager
73e3e9d78c61: Pulling fs layer
846fc1deb4d7: Pulling fs layer
open /var/lib/docker/tmp/GetImageBlob659853526: input/output error
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-scheduler:v1.13.3: output: v1.13.3: Pulling from kube-scheduler
73e3e9d78c61: Pulling fs layer
da720aab5634: Pulling fs layer
open /var/lib/docker/tmp/GetImageBlob516132413: input/output error
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-proxy:v1.13.3: output: v1.13.3: Pulling from kube-proxy
73e3e9d78c61: Pulling fs layer
0c440f353724: Pulling fs layer
5764bbd93c04: Pulling fs layer
open /var/lib/docker/tmp/GetImageBlob588053185: input/output error
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/pause:3.1: output: 3.1: Pulling from pause
67ddbfb20a22: Pulling fs layer
open /var/lib/docker/tmp/GetImageBlob108475180: input/output error
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/etcd:3.2.24: output: 3.2.24: Pulling from etcd
8c5a7da1afbc: Pulling fs layer
0d363128e48e: Pulling fs layer
1ba5e77f0f6e: Pulling fs layer
open /var/lib/docker/tmp/GetImageBlob280805019: input/output error
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/coredns:1.2.6: output: 1.2.6: Pulling from coredns
2796eccf0de2: Pulling fs layer
6ad5128a7d32: Pulling fs layer
open /var/lib/docker/tmp/GetImageBlob128246048: input/output error
, error: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with --ignore-preflight-errors=...

: Process exited with status 1

[tstromberg]

That's weird. The symptoms indicate that the filesystem inside of virtualbox is corrupt. Do you mind running attaching the output of the following command to this bug?

minikube ssh dmesg

I suspect and hope that this problem isn't repeatable with a new VM, so you can probably workaround it by running minikube delete and minikube start.

Do you mind sharing what version of VirtualBox you are running?

[tstromberg]

Also, the output of these would help:

minikube ssh ls /var/lib/docker/tmp

minikube ssh mount | grep /var/lib/docker/tmp

Thanks again for your help!

[sandy-adi]

@tstromberg I am facing the same issue, here's the information you asked for on my setup:
https://gist.github.com/sandy-adi/52d9b1cc18a197b11c9c945a9ff39a7a

[poojapatel10]

@tstromberg Thank you! for your reply , as you said my VM was crashed so for workaround I was able to resolve this issue by running minikube delete ,brew cask install minikube, curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-darwin-amd64 \ && chmod +x minikube, sudo mv minikube /usr/local/bin and minikube start.

Version of VM:- Oracle VM 6.0

[sfarooq76]

i did the same steps and still fails as above error messages.

😄 minikube v0.34.1 on darwin (amd64)
🔥 Creating virtualbox VM (CPUs=2, Memory=2048MB, Disk=20000MB) ...
💿 Downloading Minikube ISO ...
184.30 MB / 184.30 MB [============================================] 100.00% 0s
📶 "minikube" IP address is 192.168.99.127
🐳 Configuring Docker as the container runtime ...
✨ Preparing Kubernetes environment ...
💾 Downloading kubeadm v1.13.3
💾 Downloading kubelet v1.13.3
🚜 Pulling images required by Kubernetes v1.13.3 ...
❌ Unable to pull images, which may be OK: running cmd: sudo kubeadm config images pull --config /var/lib/kubeadm.yaml: command failed: sudo kubeadm config images pull --config /var/lib/kubeadm.yaml
stdout:
stderr: failed to pull image "k8s.gcr.io/kube-apiserver:v1.13.3": output: Error response from daemon: Get https://k8s.gcr.io/v2/: x509: certificate signed by unknown authority
, error: exit status 1
: Process exited with status 1
🚀 Launching Kubernetes v1.13.3 using kubeadm ...
💣 Error starting cluster: kubeadm init:
sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data-minikube --ignore-preflight-errors=Port-10250 --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI

[init] Using Kubernetes version: v1.13.3
[preflight] Running pre-flight checks
[WARNING Service-Docker]: docker service is not enabled, please run 'systemctl enable docker.service'
[WARNING Swap]: running with swap on is not supported. Please disable swap
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.13.3: output: Error response from daemon: Get https://k8s.gcr.io/v2/: x509: certificate signed by unknown authority
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-controller-manager:v1.13.3: output: Error response from daemon: Get https://k8s.gcr.io/v2/: x509: certificate signed by unknown authority
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-scheduler:v1.13.3: output: Error response from daemon: Get https://k8s.gcr.io/v2/: x509: certificate signed by unknown authority
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-proxy:v1.13.3: output: Error response from daemon: Get https://k8s.gcr.io/v2/: x509: certificate signed by unknown authority
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/pause:3.1: output: Error response from daemon: Get https://k8s.gcr.io/v2/: x509: certificate signed by unknown authority
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/etcd:3.2.24: output: Error response from daemon: Get https://k8s.gcr.io/v2/: x509: certificate signed by unknown authority
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/coredns:1.2.6: output: Error response from daemon: Get https://k8s.gcr.io/v2/: x509: certificate signed by unknown authority
, error: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with --ignore-preflight-errors=...

: Process exited with status 1

😿 Sorry that minikube crashed. If this was unexpected, we would love to hear from you:
👉 https://github.com/kubernetes/minikube/issues/new

[daniel-farina]

Deleting the local config and starting fresh fixed this issue for me:

rm -rf ~/.minikube

Cheers

[Datazource]

using VPN works for me

[tstromberg]

I believe this issue was resolved in the v1.1.0 release. Please try upgrading to the latest release of minikube, and if the same issue occurs, please re-open this bug. You will need to run minikube delete first to remove the old state.

Thank you opening this bug report, and for your patience!

[mwaz]

minikube start --docker-env HTTP_PROXY=http://10.0.2.2:1087 --docker-env HTTPS_PROXY=http://10.0.2.2:1087
fixed it for me