Kind: Kind build node-image leaves docker volumes behind.

[mauilion]

What should be cleaned up or changed:
We should clean up the volumes or warn the user to after the build
Why is this needed:
It sucked up a WHOLE bunch of disk space in a sneaky way.
also Docker system prune --all doesn't catch these! :(

[tao12345666333]

Thanks. This reminds me of the similar things that maybe we should do same thing when deleting a cluster. :thinking:

[BenTheElder]

@mauilion are you building with --type=bazel or the default? kubernetes's dockerized build eats massive disk space over time because of the build images. when I build kubernetes a lot without kind I have to purge docker.

we might be leaking _more_ volumes though, which is absolutely bad and should be fixed, will look into it!

Thanks. This reminds me of the similar things that maybe we should do same thing when deleting a cluster. 🤔

clusters do not leave behind any volumes.

[BenTheElder]

to elaborate on the cluster point: we run nodes with --rm and don't use any named volumes. I tested this one quite a bit, but not the build :( ... testing now.

[tao12345666333]

clusters do not leave behind any volumes.

you are right. :+1: my previous tests may have done some extra work leading to different results. thanks.

[BenTheElder]

just building kubernetes once seems to be creating a lot of volumes (make quick-release, no kind), currently during the kubernetes build my volumes jumped from 12 to 21.

[BenTheElder]

kind build node-image is leaving me with the same count as make quick-release (it adds 10 to whatever other volumes you have, building a few times in a row on the same code seems to leave it at the same 10 instead of adding more)

make clean drops it back down by 10

EDIT: this is with kind from HEAD and kubernetes from 1.14.1, will test with kubernetes at master next

[neolit123]

testing now.

$ docker volume ls | wc -l
203
$ kind build node-image --kube-root=$(pwd)
...
$ docker volume ls | wc -l
213
$ make quick-release
...
$ docker volume ls | wc -l
213

[BenTheElder]

so far i've not managed to leak anything, possibly switching kubernetes versions or something? :confused:

[neolit123]

so far i've not managed to leak anything, possibly switching kubernetes versions or something?

i do that a lot and that might be the issue @mauilion is reporting.
by building multiple k/k SHAs the k8s build process adds the extra volumes for kube-build.

but this doesn't seem like a kind problem, because the kube-build remains after the build.

best bet using default build:

docker system prune --all 
docker volume prune

or use bazel?

$ docker volume prune
...
Total reclaimed space: 74.11GB

yikes.

[mauilion]

mine was just kind build-image --image=mauilion/node:77523 like in the blog post.

I am working on capturing a log info for it so we can see what's up.

[mauilion]

@BenTheElder I can confirm the following:
https://asciinema.org/a/bvDoVH2qxq3RojuzAGwwNQeul

When building a checkout of kubernetes using kind build node-image --image=1:1
I see 10 new volumes created

docker volume ls 
DRIVER              VOLUME NAME
local               02a55c6255da2aff4930b1975083cf08343762b634998b3e528312d64efc5147
local               1cb9deeec52c52e16662041bb3bbb256fef384f3cdd66287515193f48b77c883
local               58685db065ee286a0dc6873162fef1168bcda7c11e1e5628ec2bb5cfe12241ae
local               78794f77a2953b0431fdf6ffb7d483618e34d3d14b10ccf829ebbd9a7e325e6a
local               8bb1347bede3f67a6e4ac916c77bd035fbc7cb0729d1ec98b0cf3fc41290381d
local               94087b32a027dd2b984c6217bd2b13dccc57a5bf85a6e5dcfc49c7ffc6d7ccb5
local               c81b82498a62a52e5191d541b8050dde40ec15e5b6fcd688c47a65cd2fa452cb
local               d1c43ef6243953ea309a9e53926993e783b69b6ca6d2ac53bf0d62ac25f7e51f
local               e8ea3fc9c0ac516885cda866bb1e9282ad0da1df7f05fe94e1fa95444ac976a0
local               eb11403037843088f40950821f4555ce4e8156f1a5c50cb66112ae0b442b4052

My problem is that this happened without my knowledge. I wasn't informed that cruft was left behind.
I think it may make sense when running kind build node-image that we figure a way to clean up these volumes if possible

[neolit123]

i think there was a discussion today on slack about this.
the left-overs are from the k8s build process, it also leaves a bunch of kube-build images on the system.

[mauilion]

ah should I open it against k/k then?

[neolit123]

i can't find a ticket in k/k for this, so seems to be OK to log one.

[BenTheElder]

@mauilion the volumes being left behind (10 + the build image + the data container) is part of intentional caching afaict, it _should_ stay 10 and they go away with make clean in Kubernetes.

The bug seems to be that sometimes it can start to leave behind even more. As far as I know this is a Kubernetes bug.

We _could_ make clean on every build but it would make the next build slower.

I'm not sure there's anything reasonable kind can do about this. IMHO we should open a ticket with Kubernetes and get it fixed there.

[BenTheElder]

We should file a ticket with kubernetes if we can create a reproducer. So far I've been unsuccessful. I've done quite a bit of testing on kind itself vs just building with kubernetes though, and I can't find any differences.

Going to close this here, but if anyone manages to get enough information and file an upstream issue, please cross reference it. I'd be happy to help fix that upstream if anyone can figure out a reliable way to trigger it.