Consul-helm: Support ability to configure `consul-webhook-cert-manager` deployment tolerations

[HofmannZ]

Question

We are spinning up Consul on GKE (Kubernetes 1.19) and noticed that the consul-connect-injector-webhook-deployment is not injecting sidecar proxies.

It's giving the following error:

[Info]
2021-01-27 04:05:05.870 CETSuccessfully wrote Consul client CA to: /consul/tls/client/ca/tls.crt
[Info]
2021-01-27 04:05:07.673 CETListening on ":8080"...
[Info]
2021-01-27 04:05:08.075 CETUpdated certificate bundle received. Updating certs...
[Error]
2021-01-27 04:05:08.172 CET2021/01/27 03:05:08 http: TLS handshake error from 10.2.1.1:57248: No certificate available.

Then we cleaned up the cluster and ran it again with logLevel: debug:

Error
2021-01-27 04:46:23.415 CET2021-01-27T03:46:23.415Z [ERROR] Error retrieving CA roots from Consul: err="Get "https://consul-server:8501/v1/agent/connect/ca/roots": dial tcp: lookup consul-server on 192.168.0.10:53: no such host"
Error
2021-01-27 04:46:24.462 CET2021-01-27T03:46:24.461Z [ERROR] Error retrieving CA roots from Consul: err="Get "https://consul-server:8501/v1/agent/connect/ca/roots": dial tcp: lookup consul-server on 192.168.0.10:53: no such host"
Info
2021-01-27 04:46:31.109 CETSuccessfully wrote Consul client CA to: /consul/tls/client/ca/tls.crt
Info
2021-01-27 04:46:32.642 CETListening on ":8080"...
Error
2021-01-27 04:46:32.642 CET2021-01-27T03:46:32.642Z [DEBUG] healthCheckResource: starting reconcile
Info
2021-01-27 04:46:32.941 CETUpdated certificate bundle received. Updating certs...
Error
2021-01-27 04:46:33.245 CET2021-01-27T03:46:33.245Z [DEBUG] healthCheckResource: finished reconcile
Error
2021-01-27 04:46:33.246 CET2021-01-27T03:46:33.245Z [DEBUG] healthCheckController: initial cache sync complete

This time we did not get the error message nor did we get a confirmation message (but we're not sure if we should get one in the first place.)

Our config:

global:
  name: consul
  gossipEncryption:
    secretName: consul-gossip-encryption-key
    secretKey: key
  tls:
    enabled: true
    enableAutoEncrypt: true
    serverAdditionalDNSSANs:
    - "consul.service.consul"
    caCert:
      secretName: ca-certs
      secretKey: "ca.crt"
    caKey:
      secretName: ca-certs
      secretKey: "ca.key"
  # acls:
  #   manageSystemACLs: true
  storageClass: premium-rwo
  replicas: 3
  extraConfig: |
    {
      "telemetry": {
        "prometheus_retention_time": "30s"
      },
      "ui_config": {
        "enabled": true,
        "metrics_provider": "prometheus",
        "metrics_proxy": {
          "base_url": "http://prometheus-server.monitor.svc.cluster.local"
        }
      }
    }

server:
  resources:
    requests:
      cpu: "1"
      memory: 4Gi
    limits:
      cpu: "1"
      memory: 4Gi
  storageClass: premium-rwo
  affinity: |
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              app: consul
              release: consul
              component: server
          topologyKey: "kubernetes.io/hostname"
     podAntiAffinity:
       requiredDuringSchedulingIgnoredDuringExecution:
         - labelSelector:
             matchLabels:
               app: consul
               release: consul
               component: server
           topologyKey: "topology.kubernetes.io/zone"
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              app: vault
              release: vault
              component: server
          topologyKey: "kubernetes.io/hostname"
  tolerations: |
    - key: taint_for_consul_xor_vault
      operator: Equal
      value: "true"
      effect: NoExecute
  nodeSelector: |
    consul_in_k8s: "true"

client:
  enabled: "-"
  grpc: true
  tolerations: |
    - key: taint_for_consul_xor_vault
      operator: Equal
      value: "true"
      effect: NoExecute

ui:
  enabled: "-"
  service:
    enabled: false

dns:
  enabled: "-"

connectInject:
  enabled: "-"
  default: true
  logLevel: debug
  centralConfig:
    enabled: true
    defaultProtocol: "http"
    proxyDefaults: |
      {
        "envoy_prometheus_bind_addr": "0.0.0.0:9102"
      }

controller:
  enabled: "-"

Running with:

NAME                    CHART VERSION   APP VERSION     DESCRIPTION                    
hashicorp/consul        0.29.0          1.9.2           Official HashiCorp Consul Chart

Is there anything we completely missed in our setup or any direction we should take to continue the investigation?

[ishustava]

Hey @HofmannZ

To clarify, when you restarted it the second time and did not get the error message, did connect injection work?

[HofmannZ]

Hi @ishustava - We did not get error messages, but connect inject did not work. 🥴

[ishustava]

Did it just skip it? Meaning your pod started without any injected containers?

Also, I noticed in your config that you need to move extraConfig under the server section for it to take an effect. Also, global.replicas and global.storageClass are being ignored because these values only exist under the server section as well.

[ishustava]

I've just tried on my GKE cluster with your values slightly modified (removed unused values and node affinity, tolerations and stuff since I don't have them set up), and it worked.

My values file ended up looking like this:

$ cat config.yaml
global:
  name: consul
  tls:
    enabled: true
    enableAutoEncrypt: true
connectInject:
  enabled: "-"
  default: true
  logLevel: debug
  centralConfig:
    enabled: true
    defaultProtocol: "http"
    proxyDefaults: |
      {
        "envoy_prometheus_bind_addr": "0.0.0.0:9102"
      }

controller:
  enabled: "-"

And the service I tried to inject looked like so:

$ cat ../examples/static-server.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: static-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: static-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: static-server
  template:
    metadata:
      name: static-server
      labels:
        app: static-server
    spec:
      containers:
        # This name will be the service name in Consul.
        - name: static-server
          image: ishustava/http-echo:latest
          command:
          - "/bin/sh"
          - "-ec"
          - |
            /http-echo -listen=:8080 -text="hello world"
          ports:
            - containerPort: 8080
              name: http
       # If ACLs are enabled, the serviceAccountName must match the Consul service name.
      serviceAccountName: static-server

Do you have a yaml file of the service you tried to inject that you can share, maybe there's a mistake somewhere there?

[HofmannZ]

@ishustava - We will give this a shot in an hour or so. The only big main differences with our config are the non-custom CA certificates. Will update once we have tested this.

[ishustava]

That's true, although I don't expect the custom CA to impact this since the webhook generates its own certificate and CA, and then updates Kubernetes webhook configuration with that information.

Let me know how it goes!

[HofmannZ]

@ishustava - Sorry for the late reply, something urgent took precedence over this issue.

We think we found the issue tho. We are running in a private cluster on GKE. By default, only port 443 and 10250 are allowed by the firewall. See:
https://github.com/hashicorp/consul-k8s/issues/269#issuecomment-635491959
https://github.com/hashicorp/vault-helm/issues/214#issuecomment-592702596

Google actually has some pretty good docs on it, so we're trying that out now:
https://cloud.google.com/kubernetes-engine/docs/how-to/private-clusters#add_firewall_rules

[HofmannZ]

@ishustava - We have enabled the firewall only for certain tainted nodes. We added the tolerations to connectInject config but cannot find a similar option to set tolerations on the consul-webhook-cert-manager deployment.

If it possible to override the tolerations for the consul-webhook-cert-manager deployment?

[ishustava]

Unfortunately, currently, it's not possible. It looks like we're not setting tolerations on that deployment at the moment. It should ideally inherit this from the connectInject tolerations.

Would you be open to contributing this as a PR?

[HofmannZ]

@ishustava - Yes! But would only be able to start contributing after the 12th of Feb (when the IRS e-filing tax season has opened up)

In terms of inheriting, I think it might need its own config to provide enough flexibility. Maybe under global.tls.tolerations or something new like certManager.tolerations?

[ishustava]

Hey @HofmannZ

Thanks so much! You're right. Since it's a separate deployment, it probably makes more sense for it to have a separate section. I like your suggestion of having certManager.tolerations - let's go with that!