Client-go: Add HTTP proxy setting to kubeconfig

[vjsamuel]

Whilst managing several Kubernetes clusters, there could be potential scenarios where certain clusters require the usage of an HTTP proxy to be accessible. It would be useful to have http-proxy as a per cluster setting on the kube config so that it is easy to talk to clusters that require and don't require the usage of a proxy.

[aakp10]

I've been looking into this issue ,I'd like to work on this

[aakp10]

Does this issue require to add something like this for the http-proxy , where details like the port and the ip on which the proxy server is running can be stored in a struct type and stored in the kubeconfig for the cluster.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[dlipovetsky]

@mikedanese Are you still working on this? This will be useful for the case where a Cluster API control plane needs to reach clusters via proxies..

[dlipovetsky]

Some details on our use case:

We will run Cluster API (CAPI) controllers in a management cluster (in our infrastructure), and create workload clusters in other infrastructures. The CAPI controllers (as of v1alpha2) require access the workload clusters' APIs. However, some workload clusters are deployed in a network that does not allow inbound connections. Therefore we are looking for a way to send requests from the CAPI controllers to the workload clusters' APIs that does not require inbound connections.

In general, there are many ways to solve this problem. We would like to find one that (a) requires no changes to CAPI controller code, and (b) no changes to the workload cluster.

The CAPI controllers use a kubeconfig (stored as a Secrets) to talk to each workload cluster's API. There is one kubeconfig for each workload cluster. If we modify the kubeconfig, we can modify how the CAPI controllers reach the workload cluster, without changing the controller code.

We can update the server property to point to a TCP tunnel. This requires we add the TCP tunnel hostname or IP to the cluster's kube-apiserver server cert's Subject Alt Names (SANs). That's a change we want to avoid making.

We can update the kubeconfig to point to an HTTP/S proxy. This requires no changes to the workload cluster's kube-apiserver server cert.

[dlipovetsky]

sig-apimachinery meeting discussion comments:

• HTTP proxy may need its own set of certificates
• We need to consider interaction between HTTP_PROXY env variable and each kubeconfig's proxy setting
• Is it possible to reuse https://github.com/kubernetes/enhancements/blob/master/keps/sig-api-machinery/20190226-network-proxy.md
• Trade-off: Adding this to kubeconfig makes general use of kubeconfig more complicated.
• Is it possible to run a separate process (configured to use a specific HTTP proxy) to talk to each workload cluster
• Is it possible to provide a separate proxy configuration? (What work is required for CAPI?)

[mikedanese]

I wish I had been to the sig meeting.

• HTTP proxy may need its own set of certificates

Nope. https_proxy are authenticate via whatever is in the cert bundle. http_proxy and https_proxy need to trust the kube-apiserver but the client won't. socks5 doesn't use ssl between client and proxy, and the connection is e2e encrypted to the apiserver.

• Is it possible to reuse https://github.com/kubernetes/enhancements/blob/master/keps/sig-api-machinery/20190226-network-proxy.md

Why would we ask everyone who wants to use a proxy to implement a custom protocol when e.g. ssh supports socks5 tunneling? This seems like a different FR.

• Is it possible to run a separate process (configured to use a specific HTTP proxy) to talk to each workload cluster

Why? We already support this with an env variable. The FR is to add an override to kubeconfig.

[dlipovetsky]

@mikedanese Thanks for following up here. I apologize you missed the meeting, I assumed you'd be there, but in hindsight I should have pinged you about it.

[jacobstr]

Wanted to add my user story here because I think @mikedanese MR would _could_ nicely simplify our own workflows in GKE, but moving parts would still remain in order to make the experience seamless.

We're using private clusters in GKE. Right now, we're having to wrap kubectl config set-context with our own wrapper to _also_ set the https_proxy environment variable out of band from the kubeconfig.

We're typically using gcloud container clusters get-credentials to configure local kubecontexts for our GKE clusters. In addition to using this command to manage my kubecontext, I need to know to construct the https_proxy url accordingly for each cluster.

Every time I switch into a new shell, I need to export this environment variable. Because I work with multiple clusters, no single value works. It also inadvertently affects any other network interactions I might have.

This makes using GKE private clusters slightly more painful than it ought to be. FWIW GKE's own documentation recommends running privoxy to act as an https_proxy to Kubernetes masters.

If https://github.com/kubernetes/kubernetes/pull/81443 gets merged, it may be helpful to make the gcloud/gke folks aware of it, in order to update gcloud container clusters get-credentials accordingly. That might not be straightforward because they're asking users to deploy privoxy themselves, which means GKE has no control / predictability about what address it's actually going to be routable at. E.g. we're managing own DNS records when we spin up our GKE clusters.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[ideasculptor]

• Is it possible to run a separate process (configured to use a specific HTTP proxy) to talk to each workload cluster

Why? We already support this with an env variable. The FR is to add an override to kubeconfig.

For starters, other commands beyond kubectl honor HTTP_PROXY and HTTPS_PROXY environment variables, so forcing an environment-wide setting for a service-specific config value breaks things.

My use case - a private GKE cluster running within a VPC with no outbound internet access. I have a bastion host in the VPC to which I set up an ssh tunnel, and then I use kubectl to access the private address of the GKE endpoint via an HTTP_PROXY (the bastion host runs tiny proxy, and the ssh tunnel connects to it). However, the gcloud cli also honors HTTP_PROXY, but gcloud commands proxied through my bastion do not work. I have no way to separate the proxy config for kubectl from the proxy config for gcloud. Nor can I figure out a NO_PROXY setting which applies to everywhere EXCEPT a couple of subnets.

simply rewriting every kubectl command with HTTP_PROXY=... kubectl ... isn't terribl effective because of the sheer number of tools which handle the running of kubectl without exposing the command that runs it. Similarly, I can't use NO_PROXY because the lack of wildcard support means I have no way to list everything EXCEPT 172.16.0.0/16. There really needs to be a whitelist option to go with the NO_PROXY blacklist. I can probably alias kubectl to a more complete command, but I'll inevitably bump into tools that break because they don't run kubectl via a shell which uses my alias.

[sbussetti]

Why? We already support this with an env variable. The FR is to add an override to kubeconfig.

@mikedanese
The long and short of it is. When you are working in an environment with bastions and many clusters, you currently have to juggle all the proxy addresses yourself if they differ from cluster to cluster and keep them in sync with whatever context you're currently in. It would be "totes amazeballs" if I could keep in my kubeconfig the value I want in HTTPS_PROXY for each cluster. Currently I have to manage all this myself via shell script wrappers and it is not the most fun thing. It also prevents me from distributing kubeconfigs to users with this info pre-populated.

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[dlipovetsky]

There is a PR, but it hasn't been reviewed by the API SIG yet.

/remove-lifecycle rotten

[Siegfriedk]

I'm in the same boat with running a gke private cluster and not being able to route traffic directly to the k8s master (non transitive vpc traffic :/).

@dlipovetsky can you link to the PR? Was not able to find it on this issue page and i looked through https://github.com/kubernetes-sigs as well.

[jasonhadi]

@Siegfriedk I believe the PR is https://github.com/kubernetes/kubernetes/pull/81443

[tillias]

Same here, if you have many external clusters it is hell job to change env variables. Much easier to swap .kube/conf

[alexppg]

I see there's the https://github.com/kubernetes/kubernetes/pull/81443/ MR merged, but I see no docs. How we could use this feature?

[kennethredler]

I see there's the kubernetes/kubernetes#81443 MR merged, but I see no docs. How we could use this feature?

Hopefully the docs come, too. However, to put an answer here to be easy to find in the meantime/for posterity: I think it would be usable by adding a proxy-url field under the cluster in your kubeconfig file. 👍

[alexppg]

Nice, thanks! Do you know if this kind of things get backported or they only enter in the latest binaries?

[liggitt]

This will be 1.19+ only