Skaffold: Skaffold example fails with no `space left on device` error

[qnm]

I'm using minikube v0.25.0 on Ubuntu 17.10, and running into a no space left on device error when running skaffold dev:

qnm@glenlivet:~/github/digivizer/skaffold/examples/getting-started$ df -h
Filesystem      Size  Used Avail Use% Mounted on
...
/dev/nvme0n1p5  116G   50G   60G  46% /
...

qnm@glenlivet:~/github/digivizer/skaffold/examples/getting-started$ skaffold dev -v debug
INFO[0000] Skaffold &{Version:v0.2.0 GitVersion: GitCommit:eb0fd0223bb9ec640f1031bb8c9858ef22338e01 GitTreeState:clean BuildDate:2018-03-09T22:47:32Z GoVersion:go1.10 Compiler:gc Platform:linux/amd64} 
INFO[0000] Using kubectl context: minikube              
DEBU[0000] skipPush value not present. defaulting to cluster default true (minikube=true, d4d=true, gke=false) 
Starting build...
Found minikube or Docker for Desktop context, using local docker daemon.
DEBU[0000] Running command: [minikube docker-env --shell none] 
DEBU[0000] Command output: stdout DOCKER_TLS_VERIFY=1
DOCKER_HOST=tcp://192.168.39.140:2376
DOCKER_CERT_PATH=/home/qnm/.minikube/certs
DOCKER_API_VERSION=1.23
, stderr: , err: %!s(<nil>) 
DEBU[0001] Running docker build: context: ., dockerfile: Dockerfile 
Sending build context to Docker daemon  6.144kB
Step 1/5 : FROM golang:1.9.4-alpine3.7
 ---> fb6e10bf973b
Step 2/5 : WORKDIR /go/src/github.com/GoogleCloudPlatform/skaffold/examples/getting-started
 ---> Using cache
 ---> ca0946d716d9
Step 3/5 : CMD ./app
 ---> Using cache
 ---> 69c82d0a2083
Step 4/5 : COPY main.go .
 ---> Using cache
 ---> 7fa3c58a90e4
Step 5/5 : RUN go build -o app main.go
 ---> Using cache
 ---> 5df153534466
Successfully built 5df153534466
Successfully tagged 58c68074b42ab413595abd19743f8d8e:latest
Successfully tagged gcr.io/k8s-skaffold/skaffold-example:5df153534466878a64291424d9504ef313e41e5e1a5ef5ffbb664dd72031e2c0
Build complete.
Starting deploy...
DEBU[0001] Deploying path: [k8s-*] parameters: map[IMAGE_NAME:gcr.io/k8s-skaffold/skaffold-example] 
DEBU[0001] Expanded manifests k8s-pod.yaml              
Deploying k8s-pod.yaml...
DEBU[0001] Running command: [kubectl --context minikube apply -f -] 
DEBU[0001] Command output: stdout pod "getting-started" configured
, stderr: , err: %!s(<nil>) 
Deploy complete.
DEBU[0001] Checking base image golang:1.9.4-alpine3.7 for ONBUILD triggers. 
DEBU[0001] Using registries.d directory /etc/containers/registries.d for sigstore configuration 
DEBU[0001]  No signature storage configuration found for docker.io/library/golang:1.9.4-alpine3.7 
DEBU[0001] Looking for TLS certificates and private keys in /etc/docker/certs.d/docker.io 
DEBU[0001] GET https://registry-1.docker.io/v2/         
INFO[0001] Looking for logs to stream for gcr.io/k8s-skaffold/skaffold-example:5df153534466878a64291424d9504ef313e41e5e1a5ef5ffbb664dd72031e2c0 
DEBU[0001] Found container getting-started with image gcr.io/k8s-skaffold/skaffold-example:5df153534466878a64291424d9504ef313e41e5e1a5ef5ffbb664dd72031e2c0 
INFO[0001] Trying to stream logs from pod: getting-started container: getting-started 
INFO[0001] Waiting for getting-started to be ready      
[getting-started getting-started] Hello world!
DEBU[0002] Ping https://registry-1.docker.io/v2/ err <nil> 
DEBU[0002] Ping https://registry-1.docker.io/v2/ status 401 
[getting-started getting-started] Hello world!
DEBU[0003] GET https://registry-1.docker.io/v2/library/golang/manifests/1.9.4-alpine3.7 
[getting-started getting-started] Hello world!
DEBU[0004] GET https://registry-1.docker.io/v2/library/golang/manifests/sha256:6c542ee4f818338ac6b3347b59189a002b4796a5069729e2fae7201e12042231 
[getting-started getting-started] Hello world!
DEBU[0005] Downloading /v2/library/golang/blobs/sha256:fb6e10bf973b841879e08f77bbc2d2cc4972828c6d5139bdb2c5da93bf8c8937 
DEBU[0005] GET https://registry-1.docker.io/v2/library/golang/blobs/sha256:fb6e10bf973b841879e08f77bbc2d2cc4972828c6d5139bdb2c5da93bf8c8937 
[getting-started getting-started] Hello world!
[getting-started getting-started] Hello world!
DEBU[0007] Found onbuild triggers [] in image golang:1.9.4-alpine3.7 
INFO[0007] Found dependencies for dockerfile [main.go]  
INFO[0007] Added watch for /home/qnm/github/digivizer/skaffold/examples/getting-started/Dockerfile 
ERRO[0007] run: running skaffold steps: running watch: adding watching for deps: starting watch on file /home/qnm/github/digivizer/skaffold/examples/getting-started/Dockerfile: adding watch for /home/qnm/github/digivizer/skaffold/examples/getting-started/Dockerfile: no space left on device 

[r2d4]

Any update on this @qnm? I haven't been able to reproduce this yet.

[nigorice]

I had a similar issue, but restarting my local machine seems to have fixed it.

[nigorice]

I think I found the issue, it's related to the number of inotify watchers: https://github.com/guard/listen/wiki/Increasing-the-amount-of-inotify-watchers.

Hope it helps.

[qnm]

@r2d4 Confirming that the 'inotify' approach suggested by @nigorice solved my issue - thanks!

[r2d4]

We might be able to query and at at minimum warn the user if they approach the limit. Or we can implement a better filewatcher without inotify

[dlorenc]

We have an mtime file watcher now. Closing this out.

[LukeeeeBennett]

I have been getting this same issue with an error about inotifiy watchers. Ingress log shows the no space left error specifically for /var/lib/containers. On minikube in virtualbox on mac os.

I've looked and tried a number of approaches to increase the max watchers but none have worked.

[nkubala]

@LukeeeeBennett this could be related to your minikube docker daemon getting filled up from images on your host. can you try running a docker system prune and see if you're still getting that error?

[LukeeeeBennett]

You're absolutely right. Looks like running my stack locally is going to be a big challenge. Maybe move to remote dev cluster. Not sure how it works with multiple contributors but will google.

[nkubala]

i'd bet you can find a way to make it work locally :) if you have the resources try increasing the VM size for minikube, there's a launch option for that. also, skaffold should be trying to prune its own images on your host when running with minikube so that could also help. otherwise, you could try and get more aggressive with profiles in your skaffold.yaml to move out images you won't be developing on during a session (e.g. if you have 10 services but you're only planning on working on one, get the other 9 deployed and then iterate only on the 1 you want). this should dramatically reduce the files skaffold is monitoring in dev mode.

good luck!