Consul-helm: Consul and Vault with TLS on Kubernetes

[HofmannZ]

Question

We are setting up Consul and Vault on Kubernetes using both of the official Helm charts.
Our goals are:

• to use Consul as a HA backend for Vault.
• use gossip encryption for both Consul and Vault.
• use TLS encryption on client <> server communication.
• use TLS encryption on Vault <> Consul communication.

NOTE: We would of course first setup ACLs before moving this setup to production.

We have set up our Kubernetes nodes with the labels and taints. And added the affinities, tolerations, and node selectors to the consul-helm and vault-helm configs. Based on the Vault on Kubernetes Reference Architecture.

The recommendation from the consul-helm is to use auto encrypt for distributing the certificates automatically.

This is our current Consul config:

global:
  name: consul
  gossipEncryption:
    secretName: consul-gossip-encryption-key
    secretKey: key
  tls:
    enabled: true
    enableAutoEncrypt: true
    serverAdditionalDNSSANs:
    - "consul.service.consul"
    caCert:
      secretName: ca-certs
      secretKey: "ca.crt"
    caKey:
      secretName: ca-certs
      secretKey: "ca.key"
  replicas: 3
server:
  affinity: |
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              app: consul
              release: "{{ .Release.Name }}"
              component: server
          topologyKey: "kubernetes.io/hostname"
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              app: consul
              release: "{{ .Release.Name }}"
              component: server
          topologyKey: "topology.kubernetes.io/zone"
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              app: vault
              component: server
          topologyKey: "kubernetes.io/hostname"
  tolerations: |
    - key: taint_for_consul_xor_vault
      operator: Equal
      value: "true"
      effect: NoExecute
  nodeSelector: |
    consul_in_k8s: "true"
client:
  tolerations: |
    - key: taint_for_consul_xor_vault
      operator: Equal
      value: "true"
      effect: NoExecute
ui:
  enabled: "-"
  service:
    enabled: false
dns:
  enabled: "-"
ingressGateways:
  enabled: "-"
  gateways:
  - argo-tunnel-example-front-end

Our question is: what is the best practice for providing the client TLS certs from Consul to Vault to enforce secure communication between them. We are currently looking into annotating the Vault server pod: consul.hashicorp.com/connect-inject: "true". This would mean Vault would call Consul over localhost and connect-inject would handle the TLS encryption. We wonder though if this is a production-ready approach or just a hacky setup. We also considered providing our own certificates to Consul and Helm via Kubernetes secrets similar to #74.

We really like Consul and Vault's flexibility but are missing a bit for documentation on using both helm charts together. ✌️

[ishustava]

Hi @HofmannZ

Thanks for creating this issue. I apologize for the confusion and the lack of documentation on this. This is something we're looking to improve in the future.

When you enable auto-encrypt in Consul, the certificates that the server will generate for the Consul clients will be signed by the Consul Connect CA rather than server's CA. In other words, the CA used by the clients will be different from the one you're providing in the global.tls.caCert configuration.

Unfortunately, right now you'd need to manually retrieve that CA from the consul API and save it as secret in Kubernetes, so that you can pass it to Vault. Here is how you can do that:

1. Once you have your Consul cluster running, get CA and create the kube secret from it. Here is a condesend one-liner that I used that worked (Note that you would have to use the right pod name of your Consul servers for the kubectl exec command):

```
kubectl create secret generic consul-client-ca --from-literal=ca="$(kubectl exec consul-server-0 -- curl -sk https://localhost:8501/v1/connect/ca/roots | jq -r .Roots[0].RootCert)"
```

1. Provide that secret to Vault. Here is the simplified Config I used in my testing that worked (Note I did not enable TLS on Vault itself to keep it simple):

server:
  standalone:
    enabled: false
  dataStorage:
    enabled: false
  extraVolumes:
    - type: secret
      name: consul-client-ca
  ha:
    enabled: true
    replicas: 1
    config: |
      ui = true
      listener "tcp" {
        tls_disable = 1
        address = "[::]:8200"
        cluster_address = "[::]:8201"
      }
      storage "consul" {
        path = "vault"
        address = "HOST_IP:8501"
        scheme = "https"
        tls_ca_file = "/vault/userconfig/consul-client-ca/ca"
      }
      service_registration "kubernetes" {}

One caveat to keep in mind here is that when you rotate the Connect CA in consul, you' need to update the secret too. This is something that our Helm chart can do for you in the future (see #474 and hashicorp/consul-k8s#310).

I hope I answered your question!

[r3mattia]

Thank you @ishustava and @HofmannZ for bringing up and providing the solution to this. It worked for me.

[ishustava]

Awesome, thanks for confirming @r3mattia.

@HofmannZ is it OK with you if we close this issue for now as we have those other ones tracking future improvements?

[HofmannZ]

@ishustava - Thanks for the swift response!

When you enable auto-encrypt in Consul, the certificates that the server will generate for the Consul clients will be signed by the Consul Connect CA rather than server's CA. In other words, the CA used by the clients will be different from the one you're providing in the global.tls.caCert configuration.

This really helps, you might want to mention it more prominent in the docs 😉

[PoojaSanaka]

Hi @ishustava,
I'm using Hashicorp consul server managed app on Azure.
I'm unable to make a communication between vault and Consul agent.
I tried the above said configuration.
As I'm using Consul sever managed app I'm unable to run the below command

kubectl create secret generic consul-client-ca --from-literal=ca="$(kubectl exec consul-server-0 -- curl -sk https://localhost:8501/v1/connect/ca/roots | jq -r .Roots[0].RootCert)"

Created k8s secret by command below:

az hcs generate-kubernetes-secret \
  --name consul-sandbox-managed-app \
  --resource-group consul-sandbox-rg-ma | kubectl apply -n consul -f -

Vault config used:

server:
  standalone:
    enabled: false
  dataStorage:
    enabled: false
  extraVolumes:
    - type: secret
      name: consul-sandbox-managed-app-hcs
  ha:
    enabled: true
    replicas: 3
    config: |
      ui = true
      listener "tcp" {
        tls_disable = 1
        address = "[::]:8200"
        cluster_address = "[::]:8201"
      }
      storage "consul" {
        path = "vault"
        address = "HOST_IP:8501"
        scheme = "https"
        tls_ca_file = "/vault/userconfig/consul-sandbox-managed-app-hcs/caCert"
      }
      service_registration "kubernetes" {}

ERROR in vault-0 pod:

WARNING! Unable to read storage migration status.
2021-06-28T20:33:03.912Z [INFO]  proxy environment: http_proxy="" https_proxy="" no_proxy=""
2021-06-28T20:33:03.912Z [WARN]  storage.consul: appending trailing forward slash to path
2021-06-28T20:33:04.921Z [WARN]  storage migration check error: error="Get "https://10.54.0.105:8501/v1/kv/vault/core/migration": x509: certificate signed by unknown authority"

Could you please suggest how to manually retrieve that CA from the consul API and save it as secret in Kubernetes while using Azure managed app

I have also created below :

• Github Issue
• Hashicorp discussion

Thanks
Pooja