Consul-helm: How to bootstrap ACL using a well known master token?

[seguidor777]

Hi all,

I am trying to install consul with the managed ACL option and I cannot specify a custom master token.
I have set this option https://www.consul.io/docs/agent/options#acl_tokens_master
However when I install the chart, the ACL init containers fail and show this error
Error getting Kubernetes secret: secrets "consul-consul-client-acl-token" not found

values.yaml

global:
  enabled: true
  image: "consul:1.8.1"
  acls:
    manageSystemACLs: true
datacenter: dc1
gossipEncryption:
  secretName: consul
  keyName: gossip_encryption_key
tls:
  enabled: true
  enableAutoEncrypt: true
  caCert: /consul/userconfig/consul/consul.pem
  caKey: /consul/userconfig/consul/consul-key.pem
server:
  replicas: ${consul_replicas}
  bootstrapExpect: ${consul_replicas}
  extraConfig: |
    {
      "connect": {
        "enabled": true,
        "ca_provider": "vault",
        "ca_config": {
          "address": "https://vault:8200",
          "token": "${consul_vault_token}",
          "root_pki_path": "pki-root",
          "intermediate_pki_path": "pki-services",
          "ca_file": "/consul/userconfig/consul/ca.pem"
        }
      },
      "acl": {
        "tokens": {
          "master": "c1039605-4cb2-4316-bede-e0682d333a6f"
        }
      }
    }
  extraVolumes:
    - type: secret
      name: consul
ui:
  enabled: true

Extract from the logs

consul-consul-mkgcb client-acl-init Error getting Kubernetes secret: secrets "consul-consul-client-acl-token" not found
consul-consul-624xw client-acl-init Error getting Kubernetes secret: secrets "consul-consul-client-acl-token" not found
consul-consul-mkgcb client-acl-init Error getting Kubernetes secret: secrets "consul-consul-client-acl-token" not found
consul-consul-624xw client-acl-init Error getting Kubernetes secret: secrets "consul-consul-client-acl-token" not found
consul-consul-mkgcb client-acl-init Error getting Kubernetes secret: secrets "consul-consul-client-acl-token" not found
consul-consul-624xw client-acl-init Error getting Kubernetes secret: secrets "consul-consul-client-acl-token" not found
consul-consul-mkgcb client-acl-init Error getting Kubernetes secret: secrets "consul-consul-client-acl-token" not found
consul-consul-624xw client-acl-init Error getting Kubernetes secret: secrets "consul-consul-client-acl-token" not found
consul-consul-mkgcb client-acl-init Error getting Kubernetes secret: secrets "consul-consul-client-acl-token" not found
consul-consul-624xw client-acl-init Error getting Kubernetes secret: secrets "consul-consul-client-acl-token" not found
consul-consul-mkgcb client-acl-init Error getting Kubernetes secret: secrets "consul-consul-client-acl-token" not found
consul-consul-624xw client-acl-init Error getting Kubernetes secret: secrets "consul-consul-client-acl-token" not found
consul-consul-mkgcb client-acl-init Error getting Kubernetes secret: secrets "consul-consul-client-acl-token" not found
consul-consul-624xw client-acl-init Error getting Kubernetes secret: secrets "consul-consul-client-acl-token" not found

[lkysow]

Hi Jorge,
I'll preface this response with a disclaimer that the chart doesn't really support pre-specifying the master token right now. I think we need to do more work to make this smooth.

There's a couple of things to address here.

1. If you want to set your master token ahead of time, you also need to create a kubernetes secret and pass this to the global.acls configuration so that the rest of the ACL bootstrapping can occur. To create the secret, you can run kubectl create secret generic consul-acl-bootstrap-token --from-literal=token=c1039605-4cb2-4316-bede-e0682d333a6f.

   Then you can reference this in your helm config:

   global:
     ....
     acls:
       manageSystemACLs: true
       bootstrapToken:
         secretName: consul-acl-bootstrap-token
         secretKey: token
     ...
   

   This will allow the ACL bootstrap job to create the client tokens and other tokens.

2. I see that you're using

   tls:
     enabled: true
     enableAutoEncrypt: true
     caCert: /consul/userconfig/consul/consul.pem
     caKey: /consul/userconfig/consul/consul-key.pem
   

   But caCert and caKey must reference Kubernetes secrets, e.g.

   tls:
   enabled: true
   enableAutoEncrypt: true
   caCert:
     secretName: consul-ca-cert
     secretKey: tls.crt
   caKey:
     secretName: consul-ca-cert
     secretKey: tls.key
   

   To create this secrets, you can run something like kubectl create secret generic consul-ca-cert --from-file=tls.crt=consul-agent-ca.pem --from-file=tls.key=consul-agent-ca-key.pem

3. Your servers also need a token they can use themselves. Otherwise you'll see errors like 2020-08-12T20:10:39.349Z [WARN] agent: Coordinate update blocked by ACLs: accessorID=00000000-0000-0000-0000-000000000002. The ACL bootstrap job usually hands these tokens out to the servers via API, however currently if the bootstrap token is set then we assume the servers already have a token (incorrect in this case). For now I think you'll need to give the servers the bootstrap token in their config file (via acl.tokens.agent) or you'll need to out-of-band make the update token API call to the servers once the cluster is up.

   yaml server: replicas: 1 bootstrapExpect: 1 extraConfig: | { "connect": { "enabled": true }, "acl": { "tokens": { "master": "e7924dd1-dc3f-f644-da54-81a73ba0a178", "agent": "e7924dd1-dc3f-f644-da54-81a73ba0a178" <<<<<<<<<<<<<<<< need this } } }

[seguidor777]

Hi Luke,

Thanks for pointing me to the right direction. The steps that you provided worked like a charm, I already have tried setting the bootstrapToken value and didn't work. Never tried with setting the agent token as well, but that was the missing part.

One last question, should I have the same token for the agent?

[lkysow]

One last question, should I have the same token for the agent?

Just to be clear, are you asking if the master token should be the same as the agent token? If so, the answer is "no". Ideally the agent token has the least amount of privileges it needs. See our ACL guide: https://learn.hashicorp.com/tutorials/consul/access-control-setup-production#create-the-agent-policy. This means ideally you'd bring up Consul, then create a policy and acl token for the servers and the distribute that token. Unfortunately we have a chicken/egg problem here because you want everything to come up at once.

That's why I said this isn't fully supported. I think ideally we'd detect that the servers don't have tokens and create the policies and tokens for them during the acl bootstrap.

[seguidor777]

Are you asking if the master token should be the same as the agent token?

Yes I am

That's why I said this isn't fully supported

I see. In fact, when the agent is assigned a different token, it is not created. Because consul assumes that both the token and its policy already exist.
So it would be enough if consul is installed, then I call the API to create the token policy and the agent token that was specified?

[lkysow]

So it would be enough if consul is installed, then I call the API to create the token policy and the agent token that was specified?

Yes, you could make up an ACL token string and set that in the server config. Then after Consul comes up you could call its API using the master token and create a policy for the server agents. You could then create an ACL token with that policy and in that API call you can actually set the secret id (https://www.consul.io/api-docs/acl/tokens#secretid) for the ACL token which you'd set to the one you made up. (I haven't tested this).

I forgot to ask, what is your use-case here? That will help us prioritize making this easier.

[seguidor777]

Thanks

I forgot to ask, what is your use-case here?

I'm working on a multi-tenant infrastructure and each tenant has consul as a dependency, so I need to automatically generate and provide the master token, initialize the ACL, and once done, run a custom job (I already have this) to register the service mesh using the master token, without having to do any further steps

[seguidor777]

In the end I'll let consul generate the master token and once consul is installed I'll reuse the consul-acl-bootstrap-token secret in my custom k8s job. I think I'd no longer need the token for something else. So the bootstrap process could end here :)

[lkysow]

Okay I think that's a better plan. Should we close this one out then?

[seguidor777]

Yes, no problem. Thanks for taking the time