Docs: Secure deployment on Kubernetes

[jseldess]

We need to expand orchestrate cockroachdb with kubernetes to cover security.

PRs:

• https://github.com/cockroachdb/cockroach/pull/16486
• https://github.com/cockroachdb/cockroach/pull/16740
• https://github.com/cockroachdb/k8s/pull/2

[a-robinson]

Whoops, I meant to post that on https://github.com/cockroachdb/cockroach/issues/13857. Removing my comment here.

[jseldess]

@mberhault, is this now ready for user-facing docs? Last we talked, you were still working out optimizations to cert management.

[mberhault]

There is still at least one big caveat: secure clients need to be started with a single replica, or concurrent certificate requests will cause problems. I started working on fixing that, but then got distracted by the new kubernetes version :)
Still, other than that caveat, it does work and we can probably start documenting it.

[diwakar-s-maurya]

I am new to kubernetes and cockroachdb. Can someone explain what could go wrong if i start the cockroachdb cluster using this yaml? It is similar to this with changes in 'command' label of 'cockroachdb' container.

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: cockroachdb
spec:
  serviceName: "cockroachdb"
  replicas: 2
  template:
    metadata:
      labels:
        app: cockroachdb
    spec:
      # Init containers are run only once in the lifetime of a pod, before
      # it's started up for the first time. It has to exit successfully
      # before the pod's main containers are allowed to start.
      # This particular init container does a DNS lookup for other pods in
      # the set to help determine whether or not a cluster already exists.
      # If any other pods exist, it creates a file in the cockroach-data
      # directory to pass that information along to the primary container that
      # has to decide what command-line flags to use when starting CockroachDB.
      # This only matters when a pod's persistent volume is empty - if it has
      # data from a previous execution, that data will always be used.
      #
      # If your Kubernetes cluster uses a custom DNS domain, you will have
      # to add an additional arg to this pod: "-domain=<your-custom-domain>"
      initContainers:
      - name: bootstrap
        image: cockroachdb/cockroach-k8s-init:0.2
        imagePullPolicy: IfNotPresent
        args:
        - "-on-start=/on-start.sh"
        - "-service=cockroachdb"
        env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: datadir
          mountPath: /cockroach/cockroach-data
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - cockroachdb
              topologyKey: kubernetes.io/hostname
      containers:
      - name: cockroachdb
        image: cockroachdb/cockroach:v1.1.0
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 26257
          name: grpc
        - containerPort: 8080
          name: http
        volumeMounts:
        - name: datadir
          mountPath: /cockroach/cockroach-data
        volumeMounts:
        - name: cockroach-config
          mountPath: /certs
        command:
          - "/bin/bash"
          - "-ecx"
          - |
            # The use of qualified `hostname -f` is crucial:
            # Other nodes aren't able to look up the unqualified hostname.

            ls /certs

            /cockroach/cockroach cert create-node \
              localhost \
              $(hostname -f) \
              cockroachdb-public \
              --certs-dir=/certs \
              --ca-key=/certs/ca.key

            rm /certs/ca.key
            ls /certs

            CRARGS=("start" "--logtostderr" "--certs-dir" "/certs" "--host" "$(hostname -f)" "--http-host" "0.0.0.0" "--cache" "25%" "--max-sql-memory" "25%")
            # We only want to initialize a new cluster (by omitting the join flag)
            # if we're sure that we're the first node (i.e. index 0) and that
            # there aren't any other nodes running as part of the cluster that
            # this is supposed to be a part of (which indicates that a cluster
            # already exists and we should make sure not to create a new one).
            # It's fine to run without --join on a restart if there aren't any
            # other nodes.
            if [ ! "$(hostname)" == "cockroachdb-0" ] || \
               [ -e "/cockroach/cockroach-data/cluster_exists_marker" ]
            then
              # We don't join cockroachdb in order to avoid a node attempting
              # to join itself, which currently doesn't work
              # (https://github.com/cockroachdb/cockroach/issues/9625).
              CRARGS+=("--join" "cockroachdb-public")
            fi
            exec /cockroach/cockroach ${CRARGS[*]}
      # No pre-stop hook is required, a SIGTERM plus some time is all that's
      # needed for graceful shutdown of a node.
      terminationGracePeriodSeconds: 60
      volumes:
      - name: datadir
        persistentVolumeClaim:
          claimName: datadir
      - name: cockroach-config
        secret:
          secretName: cockroachdb-config
  volumeClaimTemplates:
  - metadata:
      name: datadir
    spec:
      accessModes:
        - "ReadWriteOnce"
      resources:
        requests:
          storage: 1Gi

and before deploying it, running this

mkdir securedir
chmod -R 700 securedir

cockroach cert create-ca \
--certs-dir=securedir \
--ca-key=securedir/ca.key

kubectl create secret generic cockroachdb-config --from-file=./securedir/ca.crt --from-file=./securedir/ca.key

[a-robinson]

It's tough to say for sure without you also sharing the output from running that (acquired via, for example, kubectl log cockroachdb-0 or docker log <container id> on the host where the container got scheduled.

However, I wouldn't be shocked if it's related to the permissions on the secrets volume - does it permit removing the ca.key file and writing new files into it?

[a-robinson]

Also, this may be best handled as a separate issue on the https://github.com/cockroachdb/cockroach repo, on StackOverflow tagged with cockroachdb, or in our forum.

[fire]

The docs don't seem to be updated yet.

[jseldess]

@fire, the secure docs just went live. PTAL and let me know if you have any problems or questions.

[fire]

Will try to find some time on the weekend to try it.

[diwakar-s-maurya]

@jseldess Is the helm chart also updated for secure setup?

[jseldess]

@diwakar-s-maurya, I'm not sure. I didn't know about that ;).

@a-robinson, do you know about that help chart and whether it's up-to-date?

[a-robinson]

@diwakar-s-maurya, I have not yet updated the helm chart. Contributions welcome, otherwise I'll get to it soon :)