Kind: Accessing node's docker daemon API directly from host

[munnerz]

I've been trying to improve our e2e test times, and have found one of the slowest parts is the juggling/compression/decompression we do of image bundles.

For reference, we currently:

• docker load images from Bazel->host docker
• then docker save to a single .tar.gz containing multiple images
• docker cp to copy that .tar.gz into the kind container
• docker exec -- docker load to load that .tar.gz into the kind container's docker daemon.

This process takes about 1m30s during our test pipeline, with approx. 7-800MB of images (uncompressed).

The initial docker load (1. above) takes <20s, and so it's a shame we have to incur the extra overhead just due to crossing namespace/vm boundaries.

Minikube exposes a minikube docker-env command, which will output a shell script that can be sourced (or run e.g. $(minikube docker-env)) that will configure your local environment with a DOCKER_HOST environment variable allowing access to the docker daemon.

To do this, it also configures the VM's (or in our case, kind container) docker daemon to listen on 0.0.0.0:2376 (which exposes it on all IPs that the VM binds to, solving the port conflict issue with multi-cluster).

It'd be great if kind could expose something like this too, something like kind get docker-info --name=cluster-name --node=node-name

Things I see as problematic:

• we'll also need to assign a random port number and use -p on the node's docker run command to make it possible to access the docker daemon API on e.g. docker for mac, where the containers IP is not directly accessible

• this command/functionality would be docker centric - I'm not too sure what minikube does if you're using some kind of alternative CRI

• we may want to make the behaviour configurable, so that users can disable publishing the docker API on 0.0.0.0 in the kind containers. If we do, we'll need to re-introduce 'nodeLifecycle' hooks (at least internally) so that we can write a custom systemd override file to add -H tcp://0.0.0.0:2376 to the dockerd command.

/cc @BenTheElder

[tao12345666333]

SGTM. :+1: We can add DOCKER_HOST environment variable to node.

[neolit123]

/kind design
/priority important-longterm

[BenTheElder]

[we've discussed this one previously out-of-band, still mulling how to best handle it, listening to any and all input..]

[rohansingh]

I haven't thought it through entirely, but it would be interesting if the Docker endpoint was exposed as a service in kube-system or similar. That would ensure that if you can kubectl port-forward, you can reach the Docker endpoint.

Not sure how to implement this in practice.

[rohansingh]

Having thought about this more, shouldn't it already be possible to run a privileged socat container that exposes the local Docker socket as a service?

[BenTheElder]

possibly, there's a few odd things here including:

• it's probably going to be slower to use this docker than using the docker that the docker runs in
• security :yikes:
• multi-node? which node do we expose? minikube doesn't have this problem

Especially with regard to multi-node I think #28 / other options for making image loading & caching faster will pan out better than this.

[rohansingh]

So after trying to use the new load command from #28, I feel like there's still a legitimate use-case for having access to the Docker API.

My workflow is very similar to what @munnerz originally described, with loading images from Bazel to the host Docker for local development. This uses rules_docker, which has a fast-loading mechanism. This mechanism only loads the layers of an image that are not already present in the local Docker, and hugely speeds up incremental development. It's much faster than repeatedly running kind load docker-image or kind load image-archive repeatedly for the same image or archive.

For the most part, this could be ported to work with kind load image-archive, but it does have a step at the end for applying new tags to the layers that have been loaded, via docker tag. Perhaps there's some way to work around that too, but I guess my point is that there could be a growing universe of Docker operations that are useful for local development.

[BenTheElder]

Other than the rules_bazel case exposing docker on a port is generally not great and won't be faster than using the host and loading.

We also may not continue to use docker vs say containerd inside the container long term, it sounds like the community (SIG Node primarily) is moving toward CRI only, using something like kind load ... lets us continue to keep this working even then and solves the multi-node problem.

Tags can be in the images, so I'm not sure why that's an additional step. Will look into it...

[BenTheElder]

I wonder if we can find a work around for the rules_bazel case. I'm not keen on making the availability of the node docker user facing and for other use cases people definitely should prefer building on the host daemon.

[rohansingh]

Makes sense, especially considering the push to be less Docker-centric.

I'm trying to figure out why tag is an additional step as well. Maybe it's to cover the case where no new layers actually had to be loaded? I'm guessing adding a kind tag command is not on the table? ;)

[BenTheElder]

Not sure :-) I wonder if we can extend the rules to be aware of some fast-ish loading mechanism.

To be clear: we also use the bazel rules in https://github.com/kubernetes/test-infra so I also have this use case. I just want to see if we can avoid awkward dependencies and exposing dockerd on any sort of network :scream:

[BenTheElder]

@munnerz might be able to hack up some more shiny rules to leverage the archive loading :thinking:

[rohansingh]

Awesome. Should we have a separate issue for this specific use-case?

EDIT: And also maybe close this issue if the conclusion is that there will never be direct API access to Docker endpoints.

[rohansingh]

Alright, I was able to modify the approach taken by rules_docker to do an incremental load via kind load image-archive, and even apply the correct tags. Seems to work!

[BenTheElder]

Awesome! I think this would be worth documenting somewhere, I also wonder if we can do it with changing the rules via some shim pretending to be docker and forwarding to kind 🤔

[rohansingh]

Yeah, it's totally doable. To clarify, I didn't actually modify any rules_docker source or fork it. Just copied and modified the incremental loader script and the incremental_load function from layer_tools.

This alternate implementation can then be used by other rules to actually run the loader.

For context, this is all part of a rules_kind that I'm working on to encapsulate kind functionality in Bazel. Currently it brings up a kind cluster and makes the kubeconfig and image loading available to other Bazel rules like rules_k8s.

[munnerz]

@rohansingh do you have a link/repo containing your current implementation of rules_kind, or some of the macros you've built so far? I'd love to have a play with incremental loading of bazel built docker images into kind 😄 it'd make our development workflow much faster!

[rohansingh]

@munnerz It's currently part of a closed-source repo but it's in its own workspace. I'll try to break it out into something public in the coming week.

[BenTheElder]

we don't run docker on the nodes anymore (big improvements switching!), but we should continue to make loading faster, @munnerz we should publish some rules like this somewhere, and we should land #382.