Rke: Overide server URL in generated kube_config_cluster.yml

[sebastien-prudhomme]

RKE version:

0.17

Use case

I have an external load balancer that setup a virtual IP for all my Kubernetes API endpoints.

When it configures kube_config_cluster.yml, RKE setup the cluster server URL with the first (or the last?) node with the "controlplane" role.

Having an option in cluster.yml config with the URL i want to use would be great.

Thanks

[galal-hussein]

@sebastien-prudhomme RKE currently can't configure kube config with the ip of the load balancer, however rke can configure the certificates SAN to include the load balancer dns:

authentication:
  strategy: x509
  sans:
  - "10.18.160.10"
  - "my-loadbalancer-1234567890.us-west-2.elb.amazonaws.com"

[sebastien-prudhomme]

I yet use the "sans" feature. But each time i do an "rke up", i need to manually correct the kube_config_cluster.yml.

Just need an option as the "cluster_name" one. Wish i know Go to implement it :-)

[lmickh]

Using the SANs option to put the ELB name into the certs works, but if you don't have direct public access to the API server(s) the outputted kubeconfig "doesn't work". The user has to edit the kubeconfig file with the externally accessible server address.

I get that RKE doesn't configure the kube config. It is basically pulling it from the servers after the kube setup process generated. It might make sense to give the user an edited version that "just works". I would imagine the easy target would be to find and replace the server address with the external-hostname arg in the config when it exists. When paired with the sans option already mentioned, it would give the user a working config file.

services:
  kube-api:
    extra_args:  
      external-hostname: my-api-elb.example.com
authentication:
  strategy: x509
  sans:
  - "10.10.10.10"
  - "my-api-elb.example.com"

Would result in a kubeconfig file with:

apiVersion: v1
kind: Config
clusters:
- cluster:
    api-version: v1
    server: "https://my-api-elb.example.com"
    certificate-authority-data: <snip>
  name: "rke-example"

The external-hostname option for the api-server is exactly for this use case. It makes sense to just borrow that value for the kubeconfig file the user can use externally from the cluster in addition to the internal kubeconfig file resulting from the cluster setup.

[willmao]

@lmickh Hi, i used the way you mentioned above by setting the sans for my cloud-internal dns record, after bootstrap the cluster i replaced the generated server url in kube_config_cluster.yml, i can get cluster-info with kubectl, but the kube-apiserver keep printing

I0708 08:37:10.975023       1 logs.go:49] http: TLS handshake error from 100.116.166.3:13232: read tcp 10.81.182.49:6443->100.116.166.3:13232: read: connection reset by peer

seems that apiserver rejects requests from the external load balancer, any idea about how to handle this issue? Thanks in advance!

[lmickh]

@willmao it is hard to tell without additional details. Is your external lb just passing the tcp connection through or trying to handle http(s)? I used the setup above with the external elb just passing the tcp connection through to the API server.

[willmao]

@lmickh Thank you for your reply! The log was triggered by my cloud provider's LB health check servers, in order to improve the efficiency of health check, the health check servers simply send a RST request to the kube-api servers, thus the connection reset by peer logs appear!

[jgreat]

rke currently requires a direct connection to the nodes to test for api connectivity. This feature should change rke to use the defined endpoint for these checks.

INFO[0003] [network] Checking KubeAPI port Control Plane hosts 

[jnicholls]

@jgreat Agreed. For private network setups, this is very important to configure RKE so that it communicates to the API server through a provided external address/domain, e.g. through a load balancer or what have you that is publicly accessible.

Is this on the roadmap by chance?

[jnicholls]

Just looking through the code, it wouldn't be too difficult to refactor parts to support this. However, :6443 is sprinkled all through random places. They clearly didn't design this with modification in mind :D

[jnicholls]

Hopefully the above PR is accepted as a means to support this behavior of honoring the external-hostname argument to the kube-apiserver. I've successfully created clusters with this behavior and all is well.

These semantics may not be what @galal-hussein is looking for, and perhaps would prefer a type change to the cluster config and have a top-level value such as external_kube_api_host or something that intrinsically passes down to the external-hostname extra_args of kube-apiserver.

[wmgroot]

This feature would make private network setups much simpler.

[jnicholls]

@galal-hussein @JasonvanBrackel Who should I connect with to discuss the #1348 PR for further refinement and eventual upstreaming? Thanks in advance!

[iahmad-khan]

I am stuck in the same issue , I have L7 loadbalancer , and the external name is pointing to LB , while LB has a pool with API server:6443 backend.

But it doesnt work.

How can I provide the configuration to API server in RKE.

thanks

[superseb]

A design for this is now in https://github.com/rancher/rke/issues/1682, we can follow up from there.