Cockroach: Orchestrate a secure CockroachDB cluster with Kubernetes

[jseldess]

We need to identify engineering work necessary to orchestrate a secure CockroachDB cluster with Kubernetes.

@petermattis

[jseldess]

@a-robinson, please fill in more details about why this doesn't work out of the box, when you have time.

[a-robinson]

The main issue here is getting a different cert to each node. Kubernetes has a feature called "Secrets" for things like distributing certs to pods in a cluster, but it isn't integrated (and isn't likely to be soon) with the StatefulSets feature that we're using in a way that lets us distribute a different cert to each pod (see kubernetes/kubernetes#28446 (comment)). All we'd be able to do is give each pod the same cert, which doesn't line up with how CockroachDB's security is currently designed.

Some engineering effort will be needed to work around this limitation. It could mean that we use a different Kubernetes configuration for secure clusters that doesn't use PetSets, or that we stand up a separate service in the cluster for creating and distributing certs. Or that we modify CockroachDB to use symmetric certs on the nodes, but that would require its own conversation. That's what some other services do, for example this moderately popular guide for running secured Consul on Kubernetes.

[jseldess]

It's possible to use the same cert for multiple nodes, so for now, I'm just going to write up a tutorial using that approach. @a-robinson, feel free to leave this open if we expect to make changes to enable node-specific certs with Kubernetes.

[mberhault]

I should have this done this week. the secure config in https://github.com/cockroachdb/cockroach/tree/master/cloud/kubernetes is mostly correct, but I need to finish my improvements to the init container docker image to allow persistent certificate storage (so that you won't have to re-approve certs when nodes move or when you add clients).

[fire]

https://docs.openshift.com/container-platform/3.5/dev_guide/secrets.html#service-serving-certificate-secrets Allows generation of certificates via openshift origin (kubernetes). It looks useful for this case.

[fire]

Note that one service certificate is generated for all the nodes.

If you want a client certificate you'll need to use manual commands.

[fire]

Interesting find. https://github.com/kelseyhightower/certificate-init-container

Per pod certificates.

[fire]

https://kubernetes.io/docs/admin/kubelet-tls-bootstrapping/#approval-controller

This doc describes auto approval for both clients and server (feature gated).

[mberhault]

Certificates (both client and node) work with kubernetes, we just haven't written up the proper docs yet, just the basic ones in https://github.com/cockroachdb/cockroach/tree/master/cloud/kubernetes
There's still some work to do (one big caveat right now is that clients can only start with 1 replica to avoid conflicting certificate requests, once approved they can scale to whatever is desired) and a lot more testing, but the broad lines are there.

[fire]

Is it possible to write instructions of using insecure-experimental-approve-all-kubelet-csrs-for-group?

In kube 1.7 there's a rbac version, but that's not ready yet.

[cuongdo]

@mberhault is this still happening before 1.1?

[jseldess]

Secure k8 docs are next on my list. I do expect them to be done for 1.1.

[mberhault]

Docs issue is: https://github.com/cockroachdb/docs/issues/875
Closing this one.