Lens: Lens caching kubeconfig indefinetely

[geruetzel]

Describe the bug
Lens seems to cache the local kubeconfig. Changes to existing clusters in the kubeconfig, like authentication method or credentials are not updated. Only deleting the cluster and re-adding it resolves this issue.

To Reproduce
Steps to reproduce the behavior:

1. Add a cluster to Lens.
2. Check if it works in Lens, if yes, proceed to 3. If no, create a valid kubeconfig.
3. Now change your kubeconfig - just delete parts necessary to connect to the already added cluster. You will see, that regardless of a non functioning kubeconfig, Lens has cached the file and still uses it. Vice versa, if you have a working config and change your authentication details to another working configuration, these changes are unknown to Lens.

Expected behavior
If I change my kubeconfig, I would not expect to have to delete and readd the cluster to refresh the kubeconfig.

Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

• Lens Version: 3.3.1
• OS: OSX
• Installation method: Image from website

Additional context
It is not uncommon to have to change settings in your existing kubeconfig. For example we use id-tokens for authentication on our clusters which expire after a certain time and have to be reissued.

[juv]

I am running into the same issue where my Openshift login credentials are valid for 24 hours and I need to login once per day. It looks like Lens is not aware of the changes to the kubeconfig and thus is apparently reusing the old token credentials. Lens keeps failing with these error logs for me: {"name":"StatusCodeError","statusCode":401,"message":"401 - {\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"Unauthorized\",\"reason\":\"Unauthorized\",\"code\":401}.................

After deleting the cluster from Lens and re-adding the same kubeconfig context, it also works for me (for 24 hours).

[juv]

Possibly related: https://github.com/lensapp/lens/issues/250

[power-craig]

You're correct, It does do this. You can nuke your stored clusters:

rm -rf ~/Library/Application\ Support/Lens/lens-cluster-store.json (on Mac)

I believe this would be in %APP_DATA% on windows

[Nuru]

@juv I would say this is more of a duplicate of #207 than it is related to #250, which is about not being notified that your credentials have expired.

I will say that I have the same issue with some clusters where my kubeconfig gets updated with short-lived credentials several times a day. The failure of Lens to refresh kubeconfig makes it impractical to use for these clusters. While I have argued in other issues that it is unreasonable to ask Lens to handle security concerns that kubectl does not, I have to argue on the other side that it is unreasonable for Lens to fail to do things (like pick up changes in a changed kubeconfig file) that kubectl does.

[jakolehm]

Lens is basically storing a copy of given kubeconfig internally. This will cause trouble if/when both kubectl (outside of Lens) and Lens are trying to refresh the same credentials -> first one most likely wins. I agree that it's not clear to users how kubeconfig is handled internally, maybe there is a way where Lens could use original kubeconfig from ~/.kube/config and only store pasted kubeconfigs internally (it just makes implementation a bit complex).

[Nuru]

@jakolehm worte

... maybe there is a way where Lens could use original kubeconfig from ~/.kube/config and only store pasted kubeconfigs internally.

Lens could also use a tool like nightwatch ;-) to watch the kubeconfig file and reload it when it changes. Or it could do something like detect a kubectl connection failure (which it needs to be doing anyway, #207 #250 #346) and reload the file then before alerting the user.

Or, you know, Lens could just allow users to specify a kubeconfig file and literally use kubectl --kubeconfig=<filename> --cluster=<cluster> --context=<context> proxy for its operations.

[jakolehm]

Lens could also use a tool like fswatch to watch the kubeconfig file and reload it when it changes.

It's not that simple because Lens needs to keep a kubeconfig to a specific context (kubectl outside of Lens might use/switch to a different context). That's the main reason why Lens currently wants to have a private copy of a kubeconfig.

[Nuru]

Lens could also use a tool like fswatch to watch the kubeconfig file and reload it when it changes.

It's not that simple because Lens needs to keep a kubeconfig to a specific context (kubectl outside of Lens might use/switch to a different context). That's the main reason why Lens currently wants to have a private copy of a kubeconfig.

That is not a reason. OMG, Looking at the code, I see Lens _really is_ using kubectl proxy and watching the kubeconfig file, so the only bad behavior by Lens should be due to kubectl proxy caching credentials, for which you need to add some detection and recovery.

@jakolehm You do not need to rely on the kubeconfig context at all ever. It is just a convenience for specifying

• cluster
• user
• namespace

which you can get and store separately when you create the cluster. Lens doesn't care about the context namespace setting anyway, since it is always explicitly setting the namespace based on the view, so Lens' stored Cluster config would be

• path-to-kubecfg-file
• cluster-name
• user-name

and then https://github.com/lensapp/lens/blob/688919b8b1a39b231f20f4ed9904acdd70481c43/src/main/kube-auth-proxy.ts#L48-L53

becomes

    let args = [
      "proxy",
      "-p", this.port.toString(),
      "--kubeconfig", this.cluster.kubeconfigPath(),
      "--accept-hosts", clusterUrl.hostname,
      "--cluster", this.cluster.clusterName,
      "--user", this.cluster.user
    ]

People could still paste in a cluster configuration and you could do what you want with it, including just storing it as a file named whatever you want under ~/Library/Application Support/Lens/ and use that filename in the above code for consistency. Tell users to do it this way (or give them an option when setting up the cluster to create a copy) if they want the current behavior where their cluster configuration is copied rather than referenced, so that it is immune to change.

By default, reference the file they imported the cluster from. Lens would be ignoring changes to the context selection (or definition, for that matter) in the file by specifying [cluster, user, namespace] explicitly to kubectl, and if the user changes the definition of cluster or user in the config file, well, that is exactly what these bugs are asking Lens to do.

This does not immediately solve every problem, as the proxy caches configuration and authentication information itself, but now you have limited the problem to detecting errors and restarting the proxy.

[jakolehm]

That is not a reason. OMG, Looking at the code, I see Lens really is using kubectl proxy and watching the kubeconfig file, so the only bad behavior by Lens should be due to kubectl proxy caching credentials, for which you need to add some detection and recovery.

@Nuru In addition to kubectl proxy Lens is also passing a kubeconfig to Lens terminal.. there a consumer might be any cli app (most of the time it's kubectl / helm). Can we specify all context information as environment variables? And can we trust that all apps are using those (this I might buy, not Lens problem if 3rd party app does not behave properly).

[Nuru]

@jakolehm wrote:

@Nuru In addition to kubectl proxy Lens is also passing a kubeconfig to Lens terminal.. there a consumer might be any cli app (most of the time it's kubectl / helm). Can we specify all context information as environment variables? And can we trust that all apps are using those (this I might buy, not Lens problem if 3rd party app does not behave properly).

I had not thought of that. No, if you want to support arbitrary tools that are using the Kubernetes cli-runtime, you are not going to be able to do it with environment variables. I think the Kubernetes team made a conscious decision not to allow that. I suggest you just use the proxy for everything.

Side note: Other Issues (click to see)

---

My first draft response to this was getting very long because there are so many design issues this raises. I feel there should be a separate set of discussions about those. How/where is the best place/way to do that? Brief outline of things:

1. How do you want Lens to interact with the host's KUBECONFIG in general? Currently it seems Lens is inconsistent at best. See #73
2. How do you want the Lens-launched Terminals to interact with the host KUBECONFIG and each other.
   
   
   
   • Right now, Terminals are not isolated from each other's kubeconfig
   
   
   • Terminals have the host's KUBECONFIG environment variable overwritten, making it harder to access the host's configured clusters
   
   
3. Also, bugs
   
   
   
   • I was able to break Lens by using kubectl in a Terminal tab to change the current context
   
   
   • With connections open to clusters A and B, I was able to get Lens to launch a Terminal configured for a cluster A while viewing cluster B
   
   
4. Security
   
   
   
   • Running a proxy creates a security vulnerability. Probably people are OK with it, but it should be disclosed, but not given so much detail it encourages exploitation.
   
   
   • Thought should go into how to limit access to the proxy. This will probably involve Lens wrapping the proxy in yet another proxy that implements access control that Lens has full control over.
   
   
   • Security is notoriously difficult, so Lens should recruit some expert help.
   
   

---

Proxy everything

Since Lens is using kubectl proxy and needs to make that work, no matter how hard it gets, let us make the most of it. All of that effort can be leveraged by giving every other tool a kubeconfig that points to the proxy. Give each their own private temporary copy if you want to keep them isolated. Give them a shared private copy if you want them to be able to influence each other, say by setting/changing the default namespace.

Proxy kubeconfig file (click to see)

apiVersion: v1
kind: Config
preferences: {}
clusters:
- cluster:
    # replace 18001 with the actual port of the proxy
    server: http://localhost:18001
  name: proxy
contexts:
- context:
    cluster: proxy
    user: proxy
  name: proxy
current-context: proxy
users:
- name: proxy

This works for everything I have tried and has the added advantage of no extra maintenance to keep up with whatever sophisticated strategies might be later developed to manage kubeconfigs and authentication (like MFA support). It does require a couple more lines of changes to kubectl args beyond what has been discussed above. Now you would need

    let args = [
      "proxy",
      "-p", this.port.toString(),
      "--kubeconfig", this.cluster.kubeconfigPath(),
      "--accept-hosts", "^localhost$,^" + clusterUrl.hostname.replace(/\./g,"\\\.") + "$",
      "--reject-paths", "^[^/]",
      "--cluster", this.cluster.clusterName,
      "--user", this.cluster.user
    ]

I do not think the "accept-hosts" filter really provides much security, so you could set it to ".*" if it gives you trouble. You need to set "reject-paths" this way because by default it will not allow "exec" or "attach", which Lens uses extensively.

[jakolehm]

@Nuru good ideas, thanks for sharing those 👍 We did explore the "proxy everything" option when we refactored Lens to use kubectl proxy. If I remember correctly the biggest issue we faced was the fact that proxy internal dns address works only from a "browser". That said it should be possible to use "path based" routing, eg localhost:<port>/<cluster-uuid>/, which most likely works with other kubernetes tools just fine. /cc @nevalla

About side-notes: probably best place to discuss those is Lens slack channel.

[jakolehm]

Fixed in v3.6.