Consul: Permission denied when trying to register a service in catalog with management ACL token

[ashald]

I have a Consul cluster of 3 nodes with ACL enabled and default policy set to "deny" (as I described in #1708).

I'm trying to register service in catalog with command from docs https://www.consul.io/docs/guides/external.html

$ curl-v  -X PUT -d '{"Datacenter": "aws-dev", "Node": "google", "Address": "www.google.com", "Service": {"Service": "search", "Port": 80}}' 'http://192.168.1.21:8500/v1/catalog/register?token=secret-of-consul

* About to connect() to 192.168.1.21 port 8500 (#0)
*   Trying 192.168.1.21... connected
* Connected to 192.168.1.21(192.168.1.21) port 8500 (#0)
> PUT /v1/catalog/register HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.19.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: 192.168.1.21:8500
> Accept: */*
> Content-Length: 123
> Content-Type: application/x-www-form-urlencoded
>
< HTTP/1.1 403 Forbidden
< Date: Thu, 18 Feb 2016 17:45:30 GMT
< Content-Length: 28
< Content-Type: text/plain; charset=utf-8
<
* Connection #0 to host 192.168.1.21 left intact
* Closing connection #0
rpc error: Permission denied%

In Consul logs:

016/02/18 17:03:05 [ERR] http: Request PUT /v1/catalog/register?token=<hidden>, error: rpc error: Permission denied from=192.168.1.21:44600

Tried with header with the same result.

At first I didn't set up any rules for master ACL token and it didn't work as in example above. The I tried to set it to:

service "" {
    policy = "write"
}

but it didn't work as well.

Am I doing something wrong or it's a bug in Consul?

Is there is way to debug Consul ACLs? Or see resolution process in logs?

[slackpad]

@Ashald this is an interesting bug, and applies mostly to external services because other services are usually managed by the agent which does use ACL Tokens proerly. Tracing this through the code, the the catalog register HTTP endpoint doesn't respect the ?token= parameter passed in, nor does it take the agent's configured ACL Token, so there's no way to get this to work at the moment, unless you do this:

curl -v -X PUT -d '{"ID": "anonymous", "Type": "client", "Rules": "service \"search\" { policy = \"write\" }" }' 'http://127.0.0.1:8500/v1/acl/update?token=secret-of-consul'

This will "fix" it because the anonymous token, which looks like the only one that can go through the register path, will now be able to register the service.

[ashald]

@slackpad thanks for confirming the bug. Well I understand the proposed workaround but hope there will be a fix in upcoming release.

[slackpad]

Hi @Ashald - totally agree! The workaround is not a good one - the right fix is to respect the token that's passed in.

[ashald]

@slackpad can you point me to the place where the issue exists? Maybe I will be able to contribute a PR with the fix.

Btw, are there any plans/schedule for next version release? Just to know whether I need to hurry with PRs etc. :)

[slackpad]

We are actually looking to release early next week, so this may not make it in, but a PR would be appreciated! Here's where the token would need to get parsed and added to the RPC call:

https://github.com/hashicorp/consul/blob/master/command/agent/catalog_endpoint.go#L10

There's a helper in here that will do all the parse magic:

https://github.com/hashicorp/consul/blob/master/command/agent/http.go#L494

Thanks!

[ashald]

@slackpad ok, I understand - code review, code freeze etc. Although I'll try to do the PR.

Last question: what's the best way to get in touch with Consul developers? I tried IRC channel but it doesn't seem to be very active. I'm working on integrating Consul with our infrastructure and thinking about contributing few PRs with additional features that will make Consul more flexible. So I might need to discuss few things while doing that. Sorry for off-topic and thanks for the answers.

[slackpad]

I work for HashiCorp and am one of the Consul developers :-) We try to hang out on IRC when we can, but sometimes it's hard to keep a consistent presence there with everything going on. If you are thinking about new features, GitHub issues is probably the best async way to go, or general questions on the mailing list. I'll try to be around IRC next week if you want to discuss informally (slackpad on there as well).

[edef1c]

This two-line fix seems to do the trick for us, I'd be happy to submit it as a PR if that's helpful…?

--- a/command/agent/catalog_endpoint.go
+++ b/command/agent/catalog_endpoint.go
@@ -19,6 +19,7 @@ func (s *HTTPServer) CatalogRegister(resp http.ResponseWriter, req *http.Request
    if args.Datacenter == "" {
        args.Datacenter = s.agent.config.Datacenter
    }
+   s.parseToken(req, &args.Token)

    // Forward to the servers
    var out struct{}
@@ -40,6 +41,7 @@ func (s *HTTPServer) CatalogDeregister(resp http.ResponseWriter, req *http.Reque
    if args.Datacenter == "" {
        args.Datacenter = s.agent.config.Datacenter
    }
+   s.parseToken(req, &args.Token)

    // Forward to the servers
    var out struct{}

[dvrzalik]

I can confirm that this fix works, is there a chance to include it in the next release?