Cockroach: kubernetes: on premise -> adjusting load-balancer for cockroach cluster

[lwsanty]

kubernetes version 1.10.1

summary
I'm playing with insecure cockroach cluster running on bare metal kubernetes cluster according to this tutorial
https://www.cockroachlabs.com/docs/stable/orchestrate-cockroachdb-with-kubernetes-insecure.html

aim I want to add a load balancer

in tutorials I found only one manual way so far
https://www.cockroachlabs.com/docs/stable/deploy-cockroachdb-on-premises-insecure.html#step-5-set-up-haproxy-load-balancers
but it looks like I need to regenerate config file each time I scale the cluster

however, when I try to access the cluster from client and checking node like here
https://www.cockroachlabs.com/docs/stable/deploy-cockroachdb-on-premises-insecure.html#step-6-test-load-balancing
I have the random one

questions are
1) is there any working manual/tutorial how to use cockroach with kubernetes ingress controllers for loadbalancing?
2) is there a legit way to use client's cockroach gen haproxy and how it behaves during cluster scaling?

Thank you

[bobvawter]

You don't need to use HAProxy when you have Kubernetes; a Service that is configured as a LoadBalancer will do exactly what you want, without having to manage the mappings manually.

You can clone the cockroach-public service, calling it cockroach-external and setting type: LoadBalancer in the spec.

See this reference doc for more on setting up an LB:
https://kubernetes.io/docs/concepts/services-networking/service/#type-loadbalancer

Right now, an Ingress only supports HTTP(S) traffic, per https://github.com/kubernetes/kubernetes/issues/23291, so it would only be useful for exposing the admin UI.

[lwsanty]

@bobvawter thank you for your response!
However I'm a bit confused with Service configured as LoadBalancer, because according to this link

Load Balancer: A kubernetes LoadBalancer service is a service that points to external load balancers that are NOT in your kubernetes cluster, but exist elsewhere. They can work with your pods, assuming that your pods are externally routable. Google and AWS provide this capability natively. In terms of Amazon, this maps directly with ELB and kubernetes when running in AWS can automatically provision and configure an ELB instance for each LoadBalancer service deployed.

but i'm using k8s in bare metal environment and don't have any external balancers.

Please correct me if I'm wrong.

[bobvawter]

Ah, I see. You can use a type: NodePort instead, will will make every one of the nodes in your k8s listen on some TCP port and forward connections inside the cluster where they need to go. In essence, each node in your cluster becomes a load balancer.

https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport

The one minor issue is that you don't have complete control over which port number gets selected (they must be in a reserved range 30000-32767) and you'll have to update your SQL client connection strings to include that port.

[bobvawter]

Additional discussion at: https://gitter.im/cockroachdb/cockroach?at=5ad739c55d7286b43a40122f