Consul: Consul Auto Encrypt: Client Certificate says "x509: certificate signed by unknown authority"

[ashwinkupatkar]

I started with consul version 1.8.3 to try using Auto encrypt feature.on the steps mentioned in https://learn.hashicorp.com/tutorials/consul/tls-encryption-secure and noticed an issue.

After starting the server followed by client. I see client node registered successsfully.

I wanted to use consul connect feature using envoy as the side-car.

I ran the following command

`export CONSUL_HTTP_SSL=true
export CONSUL_HTTP_ADDR=https://127.0.0.1:8501
export CONSUL_CACERT=consul-agent-ca.pem

consul connect envoy -sidecar-for [name of the service] -admin-bind localhost:19001 &`

It gave me below error :

==> Failed looking up sidecar proxy info for [name of the service]: Get "https://127.0.0.1:8501/v1/agent/services": x509: certificate signed by unknown authority

However, if i use operator method to implement TLS between server and client ( i.e manually distributing client certificates),
I do not face this issue and envoy proxy works just fine.

I checked the ca cert file present on server and client and they are identical.

I even add dns and ip san which are present in pem bundle same as when distributed through operator method.

What can be the issue ?

Consul version: 1.8.3 (both server and client)
CA: Built-In Connect CA

[ashwinkupatkar]

Hi @blake,

Can you be able to help me on this issue ? or tag someone from the consul team who might be able to help me here.

Thank you soo much!

[ashwinkupatkar]

Is there anyone, who can help respond on the issue ?

[ashwinkupatkar]

If i generate my Custom RootCA and sign my server certs with it and use auto_encrypt feature, I see that on my client side, it gives

==> Failed looking up sidecar proxy info for <name of service>: Get "https://127.0.0.1:8501/v1/agent/services": remote error: tls: bad certificate

system clocks are synchronised as well before generating RootCA and other certificates on the nodes.

[ashwinkupatkar]

Looks like need to generate custom Connect CA cert with the SAN set to the cluster identifier with the .consul TLD

issue similar to : https://discuss.hashicorp.com/t/trouble-getting-consul-connect-and-envoy-to-work/6415

Need to work on fixing this issue.

[ashwinkupatkar]

How to generate a custom Spiffe certificate for Consul as Connect CA ?

I do not see any document around it,

Any leads ?

[ashwinkupatkar]

I see a similar issue regarding method to generate custom SPIFFE CA certificate here: https://github.com/hashicorp/consul/issues/8492

Can anyone from consul team... would like to enlighten here ?

[ashwinkupatkar]

I am able to generate SPIFFE svid signing certificates as discussed here https://discuss.hashicorp.com/t/trouble-getting-consul-connect-and-envoy-to-work/6415/28

However, regarding consuming the dynamic certs by envoy with the Auto Encrypt method still possess challenge.

How will the envoy proxy pick up the dynamic client certs ? Will I need to create a separate client cert for the envoy proxy purpose ?

[ashwinkupatkar]

Hi @ishustava,

With reference to https://github.com/hashicorp/consul-helm/issues/441

I already had this ticket created before. Today I tried to implement the configs as you mentioned in https://github.com/hashicorp/consul-helm/issues/441

but it does not work for any combination of verify_* setting to false on clients and also on server and still gives me error as below (envoy debug logs):

[2020-09-21 23:28:24.195][17062][debug][pool] [external/envoy/source/common/http/conn_pool_base.cc:337] queueing request due to no available connections [2020-09-21 23:28:24.195][17062][debug][pool] [external/envoy/source/common/http/conn_pool_base.cc:47] creating a new connection [2020-09-21 23:28:24.195][17062][debug][client] [external/envoy/source/common/http/codec_client.cc:34] [C6] connecting [2020-09-21 23:28:24.195][17062][debug][connection] [external/envoy/source/common/network/connection_impl.cc:727] [C6] connecting to 127.0.0.1:8502 [2020-09-21 23:28:24.195][17062][debug][connection] [external/envoy/source/common/network/connection_impl.cc:736] [C6] connection in progress [2020-09-21 23:28:24.195][17062][debug][http2] [external/envoy/source/common/http/http2/codec_impl.cc:970] [C6] updating connection-level initial window size to 268435456 [2020-09-21 23:28:24.195][17062][debug][connection] [external/envoy/source/common/network/connection_impl.cc:592] [C6] connected [2020-09-21 23:28:24.195][17062][debug][client] [external/envoy/source/common/http/codec_client.cc:72] [C6] connected [2020-09-21 23:28:24.196][17062][debug][pool] [external/envoy/source/common/http/conn_pool_base.cc:143] [C6] attaching to next request [2020-09-21 23:28:24.196][17062][debug][pool] [external/envoy/source/common/http/conn_pool_base.cc:68] [C6] creating stream [2020-09-21 23:28:24.196][17062][debug][router] [external/envoy/source/common/router/upstream_request.cc:317] [C0][S1683242336220735818] pool ready 2020-09-21T23:28:24.196Z [WARN] agent: grpc: Server.Serve failed to complete security handshake from "127.0.0.1:33350": tls: first record does not look like a TLS handshake [2020-09-21 23:28:24.196][17062][debug][misc] [external/envoy/source/common/network/io_socket_error_impl.cc:29] Unknown error code 104 details Connection reset by peer [2020-09-21 23:28:24.196][17062][debug][connection] [external/envoy/source/common/network/connection_impl.cc:558] [C6] remote close [2020-09-21 23:28:24.196][17062][debug][connection] [external/envoy/source/common/network/connection_impl.cc:200] [C6] closing socket: 0 [2020-09-21 23:28:24.196][17062][debug][client] [external/envoy/source/common/http/codec_client.cc:91] [C6] disconnect. resetting 1 pending requests [2020-09-21 23:28:24.196][17062][debug][client] [external/envoy/source/common/http/codec_client.cc:114] [C6] request reset [2020-09-21 23:28:24.196][17062][debug][pool] [external/envoy/source/common/http/conn_pool_base.cc:93] [C6] destroying stream: 0 remaining [2020-09-21 23:28:24.196][17062][debug][router] [external/envoy/source/common/router/router.cc:1018] [C0][S1683242336220735818] upstream reset: reset reason connection termination [2020-09-21 23:28:24.196][17062][debug][http] [external/envoy/source/common/http/async_client_impl.cc:96] async http request response headers (end_stream=true): ':status', '200' 'content-type', 'application/grpc' 'grpc-status', '14' 'grpc-message', 'upstream connect error or disconnect/reset before headers. reset reason: connection termination'

It only works if I create a client cert and pass it to the command as description above even in auto-encrypt mode.

You can also see similar issue here: https://github.com/hashicorp/consul/issues/7926

[ashwinkupatkar]

Hi @ishustava

Did you get a chance to look into the issue ?

Will I need to create a separate client cert just for the envoy proxy purpose ?

Thanks

[mariusehr1]

Hi,
Currently facing the same issue, I dont really get how to bootstrap the certificates to sidecars.

Thanks

Marius

[ashwinkupatkar]

Hi @ishustava, @mariusehr1,

Looks like I found a way here for maintaining TLS (rpc and https) and still using auto encrypt mode without having to set verify_incoming=false . We can continue to keep verify_incoming=true for rpc and https .

This is how I experimented:

• Extract the leaf cert of the service using this api : _curl http://127.0.0.1:8500/v1/agent/connect/ca/leaf/_

• Get the CertPEM and the PrivateKeyPEM and put it to a file and assign them to environment vars CONSUL_CLIENT_CERT and CONSUL_CLIENT_KEY

• Pass these vars along with other environment vars (CONSUL_CACERT, CONSUL_HTTP_ADDR, CONSUL_HTTP_SSL, CONSUL_GRPC_ADDR ) and you will see envoy proxy establishes connection to the actual service.

[mariusehr1]

Hi @ashwinkupatkar ,

I made it work indeed this way, thanks a lot.
But it feels like you pretty much only need the CA and its not that great having to manually bootstrap during the init script, but it will do for the time being.

I used the following script in my entrypoint if anyone is interested its used for the web service:

  apk add jq 
  mkdir /certs
  curl -k  https://127.0.0.1:8501/v1/connect/ca/roots | jq -r '.Roots[]."RootCert"' >> /certs/ca.cert
  curl -k  https://127.0.0.1:8501/v1/agent/connect/ca/leaf/web | jq -r '.["PrivateKeyPEM"]' >> /certs/key.pem
  curl -k  https://127.0.0.1:8501/v1/agent/connect/ca/leaf/web | jq -r '.["CertPEM"]' >> /certs/cert.pem
  export CONSUL_CLIENT_CERT=/certs/cert.pem
  export CONSUL_CLIENT_KEY=/certs/key.pem
  export CONSUL_CACERT=/certs/ca.cert

[jsosulska]

Hi all - brief update.

We'll be tracking improvements to this UX in https://github.com/hashicorp/consul/issues/6791, if you are interested, please +1