Kind: Support installing kubectl

[tamalsaha]

What would you like to be added:
It will be useful if kind could also kubectl, kustomize etc.

Why is this needed:
kubectl, etc. needs to match the version of k8s. Right now, it is a clumsy manual process and kind of pain to add in a CI env.

Something like https://github.com/nutellinoit/kubenvz could be built into kind, it will be even more awesome.

[BenTheElder]

I tend to use https://github.com/GoogleCloudPlatform/kubectl-dispatcher installed via gcloud components install kubectl as my system kuebctl, it automatically dispatches over multiple versions so you don't have to keep reinstalling.

I'm not sure how we should best match this into kind though, especially since kind is meant to support completely arbitrary k8s commits.

@seans3 is it feasible to drop binaries for specific commits in some directory and have the dispatcher match as closely as possible?

the node images do have kubectl, on linux you can just copy it out to the host with a one-liner, but mac / windows are problematic.

[seans3]

The kubectl-dispatcher currently only matches the version major/minor, falling back to the dispatcher's default version.

Some background: the kubectl dispatcher is kubectl with a simple query/dispatch happening before every kubectl command. The kubectl dispatcher asks the APIServer what version it is, then it executes a version of kubectl (in the same directory) that has the same major/minor version. This version is named kubectl.<MAJOR>.<MINOR>. Example: kubectl.1.15. If the kubectl dispatcher does not find this versioned kubectl, it just drops through to execute at it's default version. This is what the kubectl/gcloud distribution looks like:

$ ls -l /usr/local/google-cloud-sdk/bin/kubectl*
-rwxr-xr-x 1 sean sean 43086784 Feb  5 19:29 /usr/local/google-cloud-sdk/bin/kubectl
-rwxr-xr-x 1 sean sean 39271904 Oct 15  2019 /usr/local/google-cloud-sdk/bin/kubectl.1.13
-rwxr-xr-x 1 sean sean 43119424 Dec 11 05:11 /usr/local/google-cloud-sdk/bin/kubectl.1.14
-rwxr-xr-x 1 sean sean 42993696 Jan 18 16:08 /usr/local/google-cloud-sdk/bin/kubectl.1.15
-rwxr-xr-x 1 sean sean 42889216 Jan 18 16:04 /usr/local/google-cloud-sdk/bin/kubectl.1.16
-rwxr-xr-x 1 sean sean 43491328 Jan 18 16:03 /usr/local/google-cloud-sdk/bin/kubectl.1.17

@BenTheElder to answer your question about dropping binaries into a location: it would have to look like this previous organization. It requires a special kubectl with the dispatch mechanism, and it requires versioned kubectl binaries named according to the convention in the same directory as the kubectl dispatcher.

[seans3]

@BenTheElder Please let me know how I can help. I can probably put a kubectl dispatcher tarball in a google cloud bucket for download.

[BenTheElder]

thanks, I think we might need to consider also placing it somewhere like github, kind is generally workable in china and has an active userbase there but we've found with our nightly builds that GCS cannot be reached behind GFW.

how would the dispatcher to tiebreak kubectl1.15.5 and kubectl1.15.5-something binaries if we drop more in there?

also not sure how we avoid infinite versions.

maybe we focus on installing release versions and just assume that the api is not supposed to break in 1.X. across all ...

[BenTheElder]

I'm not sure when we'll get to this with some other big changes still settling but I really like this idea, this has been a fairly major pain point for me as well.

I'm going to sync with seans3 soon to discuss what it might look like to support installing the dispatcher (and installing more kubectl binaries) because I think that's the least friction way to go, you wouldn't need to switch between versions manually, just make sure they're installed and then it just matches the server :-)

[tamalsaha]

Similar tools for Helm
https://github.com/tokiwong/helm-switcher

[afbjorklund]

We have this in minikube kubectl, which was inspired by microk8s.kubectl.

https://github.com/kubernetes/minikube/blob/master/cmd/minikube/cmd/kubectl.go

Basically we download the correct version to the cache and run it from there.

Here's another nice approach: https://apps.0install.net/kubernetes/kubectl.xml

[aojea]

kind has the kubectl binary inside the node image, and it matches the other kubernetes componentes in the node

 docker exec -it kind-control-plane /usr/bin/kubectl version --kubeconfig /etc/kubernetes/admin.conf
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-30T20:18:51Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-30T20:19:45Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

a quick hack

aojea@host:~/go/src/sigs.k8s.io/kind$ export KUBECTL="docker exec -it kind-control-plane /usr/bin/kubectl --kubeconfig /etc/kubernetes/admin.conf"
aojea@host:~/go/src/sigs.k8s.io/kind$ $KUBECTL get services
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   fd00:10:96::1   <none>        443/TCP   90m

[tao12345666333]

Usually, I will directly copy kubectl to the local after using KIND to start the cluster

(MoeLove) ➜  ~ docker cp kind-control-plane:/kind/bin/kubectl ~/bin/kubectl
(MoeLove) ➜  ~ ~/bin/kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-30T20:18:51Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-30T20:19:45Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

[BenTheElder]

That only works on Linux @aojea @tao12345666333.

We can pretty easily do something like minikube kubectl, however I think the dispatcher is preferable so as to correctly handle multiple contexts & versions without reinventing the wheel.

The biggest trick to consider is that we support unreleased kubernetes versions / arbitrary commits. We'll also need to improve the dispatcher hosting, and determine where to install this on the host.

[aojea]

That only works on Linux @aojea @tao12345666333.

We can pretty easily do something like minikube kubectl, however I think the dispatcher is preferable so as to correctly handle multiple contexts & versions without reinventing the wheel.

The biggest trick to consider is that we support unreleased kubernetes versions / arbitrary commits. We'll also need to improve the dispatcher hosting, and determine where to install this on the host.

Can we wrap with kind the inner kubectl?
kind kubectl --name kind-cluster ...

[BenTheElder]

Not cleanly due to -f ...

On Sat, Jun 6, 2020, 06:22 Antonio Ojea [e-mail] wrote:

That only works on Linux @aojea https://github.com/aojea @tao12345666333
https://github.com/tao12345666333.

We can pretty easily do something like minikube kubectl, however I think
the dispatcher is preferable so as to correctly handle multiple contexts &
versions without reinventing the wheel.

The biggest trick to consider is that we support unreleased kubernetes
versions / arbitrary commits. We'll also need to improve the dispatcher
hosting, and determine where to install this on the host.

Can we wrap with kind the inner kubectl?
kind kubectl --name kind-cluster ...

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes-sigs/kind/issues/1502#issuecomment-640060460,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAHADKZWJYID6XFSV2A7ZGTRVI7H7ANCNFSM4MONV5SQ
.