Consul: agent: Service 'consul' registration blocked by ACLs

[rgruyters]

If you have a question, please direct it to the
consul mailing list if it hasn't been
addressed in either the FAQ or in one
of the Consul Guides.

When filing a bug, please include the following:

consul version for both Client and Server

Server: Consul v0.8.3

consul info for both Client and Server

Server:

agent:
        check_monitors = 0
        check_ttls = 0
        checks = 0
        services = 1
build:
        prerelease =
        revision = ea2a82b
        version = 0.8.3
consul:
        bootstrap = false
        known_datacenters = 1
        leader = true
        leader_addr = 10.234.167.11:8300
        server = true
raft:
        applied_index = 66
        commit_index = 66
        fsm_pending = 0
        last_contact = 0
        last_log_index = 66
        last_log_term = 8
        last_snapshot_index = 0
        last_snapshot_term = 0
        latest_configuration = [{Suffrage:Voter ID:10.234.167.11:8300 Address:10.234.167.11:8300} {Suffrage:Voter ID:10.234.167.10:8300 Address:10.234.167.10:8300}]
        latest_configuration_index = 1
        num_peers = 1
        protocol_version = 2
        protocol_version_max = 3
        protocol_version_min = 0
        snapshot_version_max = 1
        snapshot_version_min = 0
        state = Leader
        term = 8
runtime:
        arch = amd64
        cpu_count = 2
        goroutines = 78
        max_procs = 2
        os = linux
        version = go1.8.1
serf_lan:
        encrypted = true
        event_queue = 0
        event_time = 5
        failed = 0
        health_score = 0
        intent_queue = 0
        left = 0
        member_time = 9
        members = 2
        query_queue = 0
        query_time = 1
serf_wan:
        encrypted = true
        event_queue = 0
        event_time = 1
        failed = 0
        health_score = 0
        intent_queue = 0
        left = 0
        member_time = 5
        members = 2
        query_queue = 0
        query_time = 1

Operating system and Environment details

Both Consul servers are running in a Docker Container. (the official one)

Description of the Issue (and unexpected/desired result)

I'm trying to implement ACL feature on our Consul servers, but some how I can't get it implement it correctly.
Unfortunately the ACL guide on the Hashicorp website doesn't explains this properly. (at least not for me)

Reproduction steps

I have the following configuration on our Consul servers:

{
    "acl_datacenter": "dc1",
    "acl_default_policy": "deny",
    "acl_down_policy": "extend-cache",
    "acl_master_token": "<< uuidgen >>",
    "datacenter": "dc1",
    "encrypt": "<< consul keygen >>",
    "skip_leave_on_interrupt": true
}

Log Fragments or Link to gist

consul    |     2017/05/18 07:53:54 [INFO] consul: cluster leadership acquired
consul    |     2017/05/18 07:53:54 [INFO] consul: New leader elected: <<hostname>>
consul    |     2017/05/18 07:53:54 [INFO] raft: pipelining replication to peer {Voter 10.234.167.10:8300 10.234.167.10:8300}
consul    |     2017/05/18 07:53:54 [ERR] agent: coordinate update error: Permission denied
consul    |     2017/05/18 07:53:55 [WARN] agent: Service 'consul' registration blocked by ACLs
consul    |     2017/05/18 07:53:55 [INFO] agent: Synced node info
consul    |     2017/05/18 07:54:14 [ERR] agent: coordinate update error: Permission denied
consul    |     2017/05/18 07:54:42 [ERR] agent: coordinate update error: Permission denied
consul    |     2017/05/18 07:54:57 [WARN] agent: Service 'consul' registration blocked by ACLs
consul    |     2017/05/18 07:54:57 [INFO] agent: Synced node info
consul    |     2017/05/18 07:55:10 [ERR] agent: coordinate update error: Permission denied
consul    |     2017/05/18 07:55:30 [ERR] agent: coordinate update error: Permission denied

[slackpad]

Hi @rgruyters sorry for the trouble. We've gotten some other feedback that the docs need some more work (and we are looking at tweaking the bootstrap process under #2971). Here's a gist with a fully worked example - https://gist.github.com/slackpad/d89ce0e1cc0802c3c4f2d84932fa3234.

[rgruyters]

@slackpad ah thanks for the explaining this.

[beardedeagle]

@slackpad I tried your steps in that gist, and while it does stop the errors I am seeing in the logs, I cannot do things like consul members or consul info. I either get nothing back at all or I get a 403. For now I have had to entirely disable v8 acl specifics in order to make my quorum run as expected

[slackpad]

@beardedeagle those are covered by ACLs as well with v8 turned on. For consul members you'll need a token with node read privs and consul info requires agent read privs.

[winmasta]

First I have tried to bootstrap ACL with docs from consul.io then I have tried instruction above from gist, result is always the same: 1) on server - there is an error in consul logs "Service 'consul' registration blocked by ACLs" BUT service "consul" is present and visible through the web interface 2) on client - same error about service registration and one more error - "consul: RPC failed to server IP_ADDRESS:8300: rpc error: Permission denied"

[slackpad]

Hi @winmasta those look like you have not configured your acl_agent_token correctly - can to take a look again at the steps in https://www.consul.io/docs/guides/acl.html#acl-agent-token.

[winmasta]

This is how I create master token (on server):

curl -X PUT http://localhost:8500/v1/acl/bootstrap

and server returnes me ACL_MASTER_TOKEN

This is how I create ACL_AGENT_TOKEN:

curl -X PUT -H "X-Consul-Token: ACL_MASTER_TOKEN" -d '{"Name": "Agent Token","Type": "client","Rules": "node \"\" { policy = \"write\" } service \"\" { policy = \"write\" }"}' http://127.0.0.1:8500/v1/acl/create

This is how I put ACL_AGENT_TOKEN to server and clients:

curl -X PUT -H "X-Consul-Token: ACL_MASTER_TOKEN" -d '{"Token": "ACL_AGENT_TOKEN"}' http://127.0.0.1:8500/v1/agent/token/acl_agent_token

Where I did a mistake ?

[winmasta]

How can I get some debug info to put here ?

[slackpad]

Closing this as we've updated the docs since and improved bootstrapping with the new https://www.consul.io/api/acl.html#bootstrap-acls API and the new agent APIs https://www.consul.io/api/agent.html#update-acl-tokens.

@winmasta your agents also need to have https://www.consul.io/docs/agent/options.html#acl_datacenter set. If you are still having trouble can you please open a separate issue?