Consul-helm: Please provide an example of extraConfig so TLS can work

[vanderhoofen]

https://github.com/hashicorp/consul-helm/blob/ec0de4185681d352f3f7814407002819b2642f34/values.yaml#L70-L73

How exactly should client.extraConfig and client.extraVolumes be set so that we can have TLS enabled from Consul client to a Consul cluster outside of kubernetes?

Could you provide examples?

Trial and error formatting this 'raw JSON payload' via "helm --set client.extraConfig" isn't working out

We want to not have to customize values.yaml by hand but rather just use the chart

[woz5999]

Can you provide an example of the whole helm --set command? It should be sufficient to set that value as a json string wrapped in single quotes.

[vanderhoofen]

Hi Jeff,

Sure.

There's a couple challenges with this consul-helm chart.

1. Missing an easy way to import / utilize existing TLS certs to join a k8s instance with existing consul cluster outside of k8s (must be created manually)

2. The chart requires significant manual clone + edit operations to get started with anything missing in the chart (passed in via extraConfig + extraVolumes)

Example helm install:

• First manually create Kubernetes Secret (consul-certs) containing the 3 TLS Certs + Gossip Key required for "in-k8s" Consul to speak with external Consul cluster. (not shown)

`

git clone https://github.com/hashicorp/consul-helm.git && cd consul-helm

helm install --name consul .
--namespace consul
--set "global.enabled=false"
--set "global.datacenter=noc"
--set "global.image=consul:1.4.0"
--set "server.enabled=false"
--set "dns.enabled=true"
--set "syncCatalog.enabled=true"
--set "client.enabled=true"
--set 'client.join[0]=192.168.250.102'
--set 'client.join[1]=192.168.250.103'
--set 'client.join[2]=192.168.250.104'
--set 'client.join[3]=192.168.250.105'
--set 'client.join[4]=192.168.250.106'
--set "client.extraVolumes[0].type=secret"
--set "client.extraVolumes[0].name=consul-certs"
--set client.extraConfig='{
"ca_path": "/consul/userconfig/consul-certs",
"ca_file": "/consul/userconfig/consul-certs/ca.pem",
"cert_file": "/consul/userconfig/consul-certs/consul-client.pem",
"key_file": "/consul/userconfig/consul-certs/consul-client-key.pem"
}'
`

Helm fails with:
Error: render error in "consul/templates/client-config-configmap.yaml": template: consul/templates/client-config-configmap.yaml:15:14: executing "consul/templates/client-config-configmap.yaml" at <.Values.client.extra...>: wrong type for value; expected string; got []interface {}

This is probably because Helm (go) insists this value in "--set" parameters must be a raw string (albeit JSON). Single quoting it does not protect Helm from choking on the string. I've tried various quoting and escaping methods to no avail.

Bottom line: There seems to be no way to configure TLS or anything else in extraConfig w/o doing a bunch of smelly manual operations (clone, edit, check, install?)

Could you provide an example where this works via helm install w/o unsightly values.yaml edits?

Thanks,

[woz5999]

You might try --set-string or --set-file for client.extraConfig instead of --set.
https://github.com/technosophos/k8s-helm/blob/master/docs/using_helm.md

[vanderhoofen]

--set-string fails with the same error:
Error: render error in "consul/templates/client-config-configmap.yaml": template: consul/templates/client-config-configmap.yaml:15:14: executing "consul/templates/client-config-configmap.yaml" at <.Values.client.extra...>: wrong type for value; expected string; got []interface {}

--set-file requires more custom file operations instead of just tuning --set params

[adilyse]

Hi @vanderhoofen,

I've just merged a fix for how we interpret extraConfig which should enable setting it from the command line. From that PR description:

This enables users to specify extraConfig values using Helm's --set CLI
flag while installing the Helm chart. The format for using this command is:

--set 'client.extraConfig="{"log_level": "DEBUG"}"'

Please feel free to open a new issue if this doesn't solve the issue for you.

[pgbezerra]

Well, I solved this in a easier way.
You don't need to write down your config with --set client.extraConfig.
This chart deploys a configmap named consul-client-config.

Just change the key extra-from-values.json in consul-client-config config map with your file and voilá!

my-values.yml:

---
  client: 
    enabled: "true"
    extraVolumes: 
      - name: "consul-secret"
        load: "false"
        type: "secret"
    join: 
      - "xxx.com"
      - "yyy.com"
  dns: 
    enabled: "true"
  global: 
    datacenter: "datacenter1"
    domain: "mydomain"
    enabled: "false"

config map consul-client-config key: extra-from-values.json:

{
    "verify_outgoing": true,
    "verify_incoming": true,
    "verify_server_hostname": true,
    "ca_file": "/consul/userconfig/consul-secret/ca",
    "key_file": "/consul/userconfig/consul-secret/key",
    "cert_file": "/consul/userconfig/consul-secret/cert",
    "primary_datacenter": "datacenter1",
    "acl_down_policy": "extend-cache",
    "acl_default_policy": "allow",
    "acl_agent_token": "14253987253985329853298",
    "acl_agent_master_token": "14267473457345738"
}
--

[vanderhoofen]

Hi @vanderhoofen,

I've just merged a fix for how we interpret extraConfig which should enable setting it from the command line. From that PR description:

This enables users to specify extraConfig values using Helm's --set CLI
flag while installing the Helm chart. The format for using this command is:

--set 'client.extraConfig="{"log_level": "DEBUG"}"'

Please feel free to open a new issue if this doesn't solve the issue for you.

This is great. Thanks

[svlady]

There is a bit more needed than just providing the extraConfig section. The following steps working for me:

1. Generating certificates and placing them to k8s secrets. There are multiple ways for doing this and I have opted to use own consul helpers:

$ consul tls ca create
$ consul tls cert create -server -additional-dnsname=*.c01-consul-server.devops.svc

$ kubectl create secret generic consul-server-certs \
  --from-file=ca.pem=consul-agent-ca.pem \
  --from-file=consul.pem=dc1-server-consul-0.pem \
  --from-file=consul-key.pem=dc1-server-consul-0-key.pem

$ consul tls cert create -client

$ kubectl create secret generic consul-client-certs \
  --from-file=ca.pem=consul-agent-ca.pem \
  --from-file=consul.pem=dc1-client-consul-0.pem \
  --from-file=consul-key.pem=dc1-client-consul-0-key.pem

1. Adjusting health probes. It's possible to get curl commands to work either, but using consul own probes is a bit more straightforward thanks to env variables. For example in server-statefulset.yaml

...
          livenessProbe:
            exec:
              command:
                - consul
                - operator
                - raft
                - list-peers
            failureThreshold: 2
            initialDelaySeconds: 5
            periodSeconds: 3
            successThreshold: 1
            timeoutSeconds: 5
          readinessProbe:
            exec:
              command:
                - consul
                - members
            failureThreshold: 2
            initialDelaySeconds: 5
            periodSeconds: 3
            successThreshold: 1
            timeoutSeconds: 5
...

and the client-daemonset.yaml can use a similar check.

1. Now, the following additional values will fill the gaps and glue it all together:

global:
  enabled: true
  domain: consul
  datacenter: dc1
  gossipEncryption:
    secretName: consul
    secretKey: gossip-encryption-key

server:
  enabled: true
  replicas: 3
  bootstrapExpect: 3

  affinity: |
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              app: {{ template "consul.name" . }}
              release: "{{ .Release.Name }}"
              component: server
          topologyKey: failure-domain.beta.kubernetes.io/zone

  extraConfig: |
    {
      "ui": true,
      "client_addr": "0.0.0.0",
      "enable_script_checks": false,
      "disable_remote_exec": true,
      "verify_incoming": true,
      "verify_outgoing": true,
      "verify_server_hostname": true,
      "ca_file": "/consul/userconfig/consul-server-certs/ca.pem",
      "cert_file": "/consul/userconfig/consul-server-certs/consul.pem",
      "key_file": "/consul/userconfig/consul-server-certs/consul-key.pem",
      "ports": {
        "http": -1,
        "https": 8500
      }
    }

  extraEnvironmentVars: 
    CONSUL_CACERT: /consul/userconfig/consul-server-certs/ca.pem
    CONSUL_CLIENT_CERT: /consul/userconfig/consul-server-certs/consul.pem
    CONSUL_CLIENT_KEY: /consul/userconfig/consul-server-certs/consul-key.pem
    CONSUL_HTTP_SSL: true
    # CONSUL_HTTP_ADDR: https://localhost:8501

  extraVolumes:
    - type: secret
      name: consul-server-certs
      load: false

client:
  enabled: true
  grpc: true

  extraConfig: |
    {
      "verify_incoming": true,
      "verify_outgoing": true,
      "verify_server_hostname": true,
      "ca_file": "/consul/userconfig/consul-client-certs/ca.pem",
      "cert_file": "/consul/userconfig/consul-client-certs/consul.pem",
      "key_file": "/consul/userconfig/consul-client-certs/consul-key.pem",
      "ports": {
        "http": -1,
        "https": 8500
      }
    }

  extraEnvironmentVars: 
    CONSUL_CACERT: /consul/userconfig/consul-client-certs/ca.pem
    CONSUL_CLIENT_CERT: /consul/userconfig/consul-client-certs/consul.pem
    CONSUL_CLIENT_KEY: /consul/userconfig/consul-client-certs/consul-key.pem
    CONSUL_HTTP_SSL: true
    # CONSUL_HTTP_ADDR: https://localhost:8501

  extraVolumes:
    - type: secret
      name: consul-client-certs
      load: false

dns:
  enabled: true

ui:
  enabled: true
  service:
    enabled: true
    type: NodePort

NB: the affinity rules are not essential here, I have had them in place due to multi-zone cluster setup.

Keep in mind that this configuration is running HTTPS on the same 8500 port, instead of HTTP. Normally, in production you would want to use HTTPS, so this adjustment is fine. If for some reason you need to keep the both HTTP and HTTPS ports available, you can use 8501 for HTTPS and uncomment CONSUL_HTTP_ADDR env variable to point all the tools to the proper API endpoint.

[bsakweson]

I know this thread is old and has been closed but this is the only thread in here that deals with TLS for this chart. It seems as if the consul syncCatalog is designed to work without tls. Once tls is enabled on consul server and client (Agent), syncCatalog seems to stop working. Has anyone been able to get it to work correctly?

[HofmannZ]

Is this still the prefered way for consul/vault on k8s with tls?

[lkysow]

@HofmannZ this ticket is super old so perhaps opening a new one with your specific use-case would be better. Quickly, turning on TLS can be done with:

global:
  tls:
    enabled: true