Consul: connect: cluster bootstrapped with Vault CA cannot issue certs

[tradel]

Overview of the Issue

If you bootstrap a new Consul cluster with Vault as the Connect CA, Consul will fail to generate any certificates because it doesn't correctly initialize the intermediate CA.

If you bootstrap with Consul's internal CA and then switch to Vault after leader election, it works as expected.

Reproduction Steps

1. Start Consul with Vault as the CA.

./consul agent -dev -hcl 'connect {enabled=true, ca_provider="vault", ca_config{token="redacted",root_pki_path="/root",intermediate_pki_path="/inter",address="http://localhost:8200"}}'

1. Try to issue a leaf cert.

curl localhost:8500/v1/agent/connect/ca/leaf/foobar

1. Observe the Consul output.

error issuing cert: Error making API request.

URL: PUT http://localhost:8200/v1/inter/sign/leaf-cert
Code: 404. Errors:

* no handler for route 'inter/sign/leaf-cert'

Consul info for both Client and Server

Confirmed with both 1.5.3 and 1.6.0-beta3.

[i0rek]

@tradel can this issues be closed now?

[seguidor777]

Hi!
Similar issue on v1.6.0 except that I am getting an empty response on the leaf cert and cannot switch back to Consul provider, this is the output:

rpc error making call: error having Vault cross-sign cert: Error making API request.

URL: PUT https://vault.anyns.svc.cluster.local:8200/v1/pki-root/root/sign-self-issued
Code: 403. Errors:

* 1 error occurred:
    * permission denied

Why is still trying to sign the root cert when I am switching to Consul?

[seguidor777]

I was missing these capabilities

path "pki-root/root/sign-self-issued" {
  capabilities = ["update", "sudo"]
}

I had to realize on my own, because I did not see this information in the docs
Now I just switched to Consul and then back to Vault as CA provider, the issue dissapeared, however I do not think this is the best approach

[seguidor777]

New errors when requesting the leaf cert from Vault
This is from the logs

consul-0 consul     2019/08/27 19:12:40 [ERR] http: Request GET /v1/agent/connect/ca/leaf/myservice, error: rpc error making call: EOF from=127.0.0.1:38130

consul-0 consul panic: runtime error: index out of range
consul-0 consul 
consul-0 consul goroutine 2082 [running]:
consul-0 consul github.com/hashicorp/consul/agent/consul.(*ConnectCA).Sign(0xc00016a230, 0xc000f1f110, 0xc00131c420, 0x0, 0x0)
consul-0 consul     /home/circleci/project/consul/agent/consul/connect_ca_endpoint.go:540 +0x172e
consul-0 consul reflect.Value.call(0xc00001a000, 0xc00000ee70, 0x13, 0x2dd8353, 0x4, 0xc000b15588, 0x3, 0x3, 0xc00131c420, 0x16, ...)
consul-0 consul     /usr/local/go/src/reflect/value.go:447 +0x461
consul-0 consul reflect.Value.Call(0xc00001a000, 0xc00000ee70, 0x13, 0xc000b15588, 0x3, 0x3, 0x16, 0x0, 0x2991860)
consul-0 consul     /usr/local/go/src/reflect/value.go:308 +0xa4
consul-0 consul net/rpc.(*service).call(0xc000534380, 0xc0000a9310, 0xc000873940, 0x0, 0xc000295d80, 0xc0008994c0, 0x2991860, 0xc000f1f110, 0x16, 0x25865c0, ...)
consul-0 consul     /usr/local/go/src/net/rpc/server.go:384 +0x14e
consul-0 consul net/rpc.(*Server).ServeRequest(0xc0000a9310, 0x33491c0, 0xc000a26b00, 0x3f800000, 0x30)
consul-0 consul     /usr/local/go/src/net/rpc/server.go:505 +0x23e
consul-0 consul github.com/hashicorp/consul/agent/consul.(*Server).RPC(0xc000487500, 0x2dea1cc, 0xe, 0x2991860, 0xc000f1f0b0, 0x25865c0, 0xc00131c370, 0x30, 0x2b93500)
consul-0 consul     /home/circleci/project/consul/agent/consul/server.go:1113 +0x170
consul-0 consul github.com/hashicorp/consul/agent.(*Agent).RPC(0xc000852b40, 0x2dea1cc, 0xe, 0x2991860, 0xc000f1f0b0, 0x25865c0, 0xc00131c370, 0xc0009de780, 0x1dd)
consul-0 consul     /home/circleci/project/consul/agent/agent.go:1556 +0xdd
consul-0 consul github.com/hashicorp/consul/agent/cache-types.(*ConnectCALeaf).generateNewLeaf(0xc000139020, 0xc000819270, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
consul-0 consul     /home/circleci/project/consul/agent/cache-types/connect_ca_leaf.go:542 +0x486
consul-0 consul github.com/hashicorp/consul/agent/cache-types.(*ConnectCALeaf).Fetch(0xc000139020, 0x0, 0x8bb2c97000, 0x0, 0x32ebc60, 0xc000819270, 0x0, 0x0, 0x0, 0x0, ...)
consul-0 consul     /home/circleci/project/consul/agent/cache-types/connect_ca_leaf.go:325 +0xf9c
consul-0 consul github.com/hashicorp/consul/agent/cache.(*Cache).fetch.func1(0x3310880, 0xc000139020, 0xc000159480, 0x0, 0x0, 0x0, 0x0, 0x32eaf60, 0xc000357eb0, 0x0, ...)
consul-0 consul     /home/circleci/project/consul/agent/cache/cache.go:495 +0xf5
consul-0 consul created by github.com/hashicorp/consul/agent/cache.(*Cache).fetch
consul-0 consul     /home/circleci/project/consul/agent/cache/cache.go:458 +0x467

That is not a permission issue this time, any help please?

[seguidor777]

Issue solved, it was a missconfiguration on the leaf cert from Vault. I would have liked to have a guide on how to configure Vault certs to make them work with Consul

Regards
Jorge Luna

[llchan]

I get the same panic as @seguidor777 above at https://github.com/hashicorp/consul/blob/9be6dfc310ddd54627d698537a98827245185290/agent/consul/connect_ca_endpoint.go#L540
and this should be handled gracefully even if vault is misconfigured.

I'll try to fiddle with the policy/etc to get this to work, and I agree a more detailed laundry list of the required permissions for each of the root/intermediate mounts would be helpful.

[ericbrumfield]

I just ran into this 'error having Vault cross-sign cert: Error making API request.' as well with a 403 response, but for a different reason I believe. In my scenario I was updating the vault token that consul connect ca was using, which expired, AND was also attempting to rotate to a new vault pki mount for both the root and intermediary.

For this scenario, I was able to resolve the error by first updating consul connect ca config with a new and valid vault token. Then I was able to change the vault pki mounts in a second config update, so my issue that got the same error message was when I attempted to change both the token and pki mount in 1 config update. Our problem was not vault policy/permission related.

Our vault cluster has been operational for about 2 months now and this was a new error for me during token/pki mount rotation in a consul connect ca w/ vault setup.

[stale[bot]]

Hey there,
We wanted to check in on this request since it has been inactive for at least 60 days.
If you think this is still an important issue in the latest version of Consul
or its documentation please reply with a comment here which will cause it to stay open for investigation.
If there is still no activity on this issue for 30 more days, we will go ahead and close it.

Feel free to check out the community forum as well!
Thank you!