Consul: API does not serve SSL correctly

[shortdudey123]

I have ca_file, cert_file, and key_file defined, but the HTTPS side of the api is not serving the cert correctly.

Version:

Consul v0.6.4
Consul Protocol: 3 (Understands back to: 1)

Sanitized config:

{
  "bootstrap": false,
  "ca_file": "/etc/ssl/consul/example.com.ca.crt",
  "cert_file": "/etc/ssl/consul/example.com.single.crt",
  "client_addr": "0.0.0.0",
  "data_dir": "/var/lib/consul",
  "datacenter": "dc1",
  "encrypt": "BLAHBLAHBLAHBLAH",
  "key_file": "/etc/ssl/consul/example.com.single.key",
  "ports": {
    "dns": 53,
    "http": 8500,
    "rpc": 8400,
    "serf_lan": 8301,
    "serf_wan": 8302,
    "server": 8300,
    "https": 8501
  },
  "server": true,
  "start_join": [
    "123.456.789.2",
    "123.456.789.3"
  ],
  "ui": true,
  "verify_incoming": true,
  "verify_outgoing": true
}

OpenSSL s_client shows the domain cert being served on top of itself without the chain.

$ echo quit | openssl s_client -connect localhost:8501
depth=0 OU = Domain Control Validated, CN = *.example.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 OU = Domain Control Validated, CN = *.example.com
verify error:num=27:certificate not trusted
verify return:1
depth=0 OU = Domain Control Validated, CN = *.example.com
verify error:num=21:unable to verify the first certificate
verify return:1
140612180817824:error:14094412:SSL routines:SSL3_READ_BYTES:sslv3 alert bad certificate:s3_pkt.c:1259:SSL alert number 42
140612180817824:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:184:
<snip>
$

Based off of https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/consul-tool/RC5RfAKqy00/8JyjeEg8MwAJ

[merri-j]

We are running into this exact same problem as well.

[shortdudey123]

@merri-j glad to know its not just me doing something stupid :p

[jgoldschrafe]

Having just faced what I think is the same problem last night: Consul doesn't automatically append the certificate chain from ca_file to the SSL response. If you append the chain to your cert_file directly, this should work.

[slackpad]

Thanks @jgoldschrafe that sounds right. Can others here confirm if this works?

[compressed]

I recall trying it (appending the CA to the cert), but it didn't work for me. I still received the handshake failure.

[shortdudey123]

Last time i tried, having a bundled file in cert_file did not work

[tniswong]

Also seeing this issue. Bundling the cert file also did not work for me.

[tniswong]

From what I've noticed, seems to only occur when using "verify_incoming": true.

My best guess is that it's trying to verify the incoming HTTPS connection from the browser against the authority from ca_file ...

[err0]

👍 Shouldn't be a tradeoff of having good tls security or a usable UI.

[jpbelanger-mtl]

Same situation here. We ended up putting an HAProxy in front of each node only to server the HTTPS api correctly.

[edgillett]

Same here. Sigh.

[granite-cloud]

Ok, finally I have at least found that this does not in fact work properly. I spent hours trying to understand why the api will not work properly with TLS. I also had already tried bundling the cert which did nothing. I guess I will put behind nginx and terminate tls there , unless there is something new here with this issue?

[granite-cloud]

Just to be absolutely sure since I am running 0.7.1. Is anyone running this version and still seeing this? I have literally debugged this thing to death and there is no escape :). I am asking here in case I have burnt myself out and missed something dumb. I dont have an issue with putting this service behind a load balancer but would be nice if I had an option..

Version

Consul v0.7.1
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)

[atombaby]

I am running 0.7.1 and also get this error message. I have tried making the chain certificate as indicated up-thread, but this didn't change anything for me. The same set of certificates work when installed into nginx.

[kyhavlov]

I looked into this to see if there was any weirdness going on with Consul's TLS config and I think what's happening in this issue is that people are getting burned by setting "verify_incoming": true and then not providing a client cert when connecting.

If verify_incoming is on, you need to have a client key and cert in order to talk to Consul at all, or you'll get SSL errors when connecting:

$ echo quit | openssl s_client -connect localhost:8501

CONNECTED(00000003)
depth=1 C = US, ST = example, L = Testville, O = Hashicorp, OU = test CA, CN = testCA, emailAddress = [email protected]
verify return:1
depth=0 C = US, ST = example, L = Testville, O = Hashicorp, OU = test server, CN = 192.168.1.2, emailAddress = [email protected]
verify return:1
140600432383648:error:14094412:SSL routines:SSL3_READ_BYTES:sslv3 alert bad certificate:s3_pkt.c:1262:SSL alert number 42
140600432383648:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:177:
---
Certificate chain
...

Instead, you need to pass -key and -cert to openssl (and -CAfile too if you're using a self-signed cert):

$ echo quit | openssl s_client -showcerts -CAfile rootCA.pem -key client.key \
     -cert client.crt -connect localhost:8501

CONNECTED(00000003)
depth=1 C = US, ST = example, L = Testville, O = Hashicorp, OU = test CA, CN = testCA, emailAddress = [email protected]
verify return:1
depth=0 C = US, ST = example, L = Testville, O = Hashicorp, OU = test server, CN = 192.168.1.2, emailAddress = [email protected]
verify return:1
---
Certificate chain
...

Or, alternatively:

$ curl -k https://192.168.1.2:8501/v1/catalog/services?pretty --key client.key --cert client.crt
{
    "consul": []
}

[shortdudey123]

@kyhavlov i am not familiar with how to get Chrome to pass a cert request to the UI.
I can understand needing it for api requests, but not the web ui

[slackpad]

I can understand needing it for api requests, but not the web ui

Agree - we just don't currently have an option to configure Consul that way. We will add something to enable this without having to resort to setting verify_incoming to false.

[moofish32]

@shortdudey123 this talks about using pkcs12 and may help you. I have not tried this with the consul configuration described here yet.

[slackpad]

Fixed in #2969 - now you can keep verify_incoming on for RPC, but not require it for HTTPS API access.

[shortdudey123]

@slackpad glad to hear this was finally fixed!
will try it out

[gigenthomas]

I just tested this on Consul v0.8.3. HTTPS API access fails if you don't pass the key and the cert
OpenSSL 1.0.2g 1 Mar 2016

If I don't pass the cert and the key I get the following error:
curl https://aaaaaaa:8080/v1/status/leader

curl: (35) error:14094412:SSL routines:SSL3_READ_BYTES:sslv3 alert bad certificate

[slackpad]

@gigenthomas can you share the verify_ settings you were using? /cc @kyhavlov

[tueder]

Got the same problems here! Does anyone has a hint how to configure consul ssl with a certificate which needs also a a certificate chain? I did no find a way to get consol to serve a proper ssl certificate chain on the https port.

[slackpad]

Kicking this back open for another look.

[gigenthomas]

I believe I have this working now. Will post the details later today. My setup includes an offline rootCA and an issuing CA that issues certs to consul. I have SSL setup and working. FYI - I am not using Cert based authentication, just SSL to access the consul API.

[gigenthomas]

Here's my config. I had to set "verify_incoming": false. I believe setting this to true enables cert auth.
All I need is TLS and don't require cert based auth. Can someone please confirm if the assumptions I made are accurate ?
Thanks in advance !!
{
"bootstrap": true,
"server": true,
"datacenter": "yyyy",
"data_dir": "/var/consul",
"client_addr": "0.0.0.0",
"encrypt":"aaaaaaaaaaa",
"ca_file": "/etc/consul.d/ssl/root.cacert.crt",
"cert_file": "/etc/consul.d/ssl/node.aaaa.pem",
"key_file": "/etc/consul.d/ssl/aaaaate.key",
"ports": {
"https": xxxx
},
"verify_incoming": false,
"verify_incoming_rpc": true,
"verify_outgoing": true,
"log_level": "INFO",
"enable_syslog": true,
"start_join": ["a.a.a.a"],
"bind_addr": "a.a.a.a0",
"acl_master_token" : "aaaaaaaaaaaaaaaaaaaaaa" ,
"acl_datacenter":"yyyy"
}

[tueder]

I can confirm that this working to get Consul TLS without client certh auth.

I also solved my own problem.
In the cert_file i had to inlcude the complete certifacte chain. (Cert - Intermediate Cert - Root Cert) and in the ca-file not only the root certificate, but also the complete chain. (Intermediate Cert - Root Cert).

[slackpad]

@gigenthomas that looks correct to have RPC verification on, but not HTTPS:

"verify_incoming": false,
"verify_incoming_rpc": true,
"verify_outgoing": true,

@tueder thanks for your update, I think that covers the re-open concern, so closing this back out.