K3d: [FEATURE]: One kubeconfig.yaml for more than one cluster

[giminni]

Currently if I create two k3d cluster with different names, k3d creates two kubeconfig.yaml files under two different directories.

I wanna have one kubeconfig.yaml file and use kubectx and kubens to switch between them easily.

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: 
<certificate-cluster-dev>
    server: https://localhost:6551
  name: cluster-dev
- cluster:
    certificate-authority-data: 
<certificate-cluster-prod>
    server: https://localhost:6550
  name: cluster-prod
contexts:
- context:
    cluster: cluster-dev
    namespace: development
    user: user-dev
  name: default-dev
- context:
    cluster: cluster-prod
    namespace: production
    user: user-prod
  name: default-prod
current-context: default-prod
kind: Config
preferences: {}
users:
- name: user-dev
  user:
    password: <user-dev password>
    username: admin
- name: user-prod
  user:
    password: <user-prod password>
    username: admin

Implementation perhaps with:
k3d create -a1 6550 -n1 cluster-prod -w1 5 -a2 6551 -n2 cluster-dev -w2 1 --image rancher/k3s:v0.7.0-rc4

[iwilltry42]

Hey @giminni , thanks for submitting this feature request.
What you describe there is creating two clusters at the same time, which is way more complicated than simply merging the configs (because of re-using the same flag for different references multiple times).
If it's a big benefit, I'd rather suggest implementing something like k3d get-kubeconfig --merge cluster-prod cluster-dev which would grab the kubeconfigs for both clusters, rename the cluster in the config from default to the actual k3d cluster name and then run kubectl config view --merge --flatten to merge them.
But that could also be achieved by a simple shell script for now I guess :grimacing:

[giminni]

@iwilltry42 your proposal sounds good

[tanelmae]

If it helps someone here is my shell based workaround: https://gist.github.com/tanelmae/b0dc522ab3984f290b0b25766128f61a

k3ctx <cluster>
or

k3ctx
<pick cluster from what it can find>

[kristoferlundgren]

export KUBECONFIG=file1:file2:file3

... already merges config files.
https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/#set-the-kubeconfig-environment-variable

Thus this one-liner would have worked with kubectx:

export KUBECONFIG=${HOME}/.kube/config:$(k3d get-kubeconfig --all | tr '\n' ':')

(ie, the default config, plus all k3d generated ones, colon separated)

... if k3d did not re-use the name "default" in the config files, regardless of the configured cluster name.
Therefore, "kubectx", when using the one-liner above, sees all k3d clusters as one "default".

I'm quite new to Kubernetes, but my understanding is that if k3d changed the generated config-files from using "default", to the asked-for cluster name. It would perhaps solve the issue and the one-liner above would work for kubectx?

[iwilltry42]

@kristoferlundgren , changing the name from default to the actual k3d cluster name in the config generated by k3s would probably be the best and easiest solution for this combined with your one-liner.
I think it would make working with multiple k3d clusters easier in general.

It's on the list already :+1:

[kristoferlundgren]

@iwilltry42 , Thanks for acknowledging that my idea is feasible!
Do you have a reference to "the list"?
Timeplan / rough estimate?

[iwilltry42]

@kristoferlundgren , I just pushed the change to master (469b56c) and so it will land in the next release :+1:
EDIT: the change here is very simple and simply replaces every occurrence of 'default' in the kubeconfig with the cluster name. So it also replaces the username, etc. This can be improved later on by parsing the kubeconfig file completely and modifying it as a yaml struct.

[kristoferlundgren]

@iwilltry42 Hero of the day!

[iwilltry42]

Landed in k3d v3 (see e.g. #226)