Envoy: HTTPS unavailable after load private key failed.

Created on 22 Jan 2019 · 13Comments · Source: envoyproxy/envoy

Description:

When I was load a pair of TLS certificate & key to envoy, there's something warning.

[2019-01-21 08:12:08.266][1][info][upstream] source/server/lds_api.cc:80] lds: add/update listener 'ingress_https'
[2019-01-21 08:13:17.399][1][warning][upstream] source/common/config/grpc_mux_impl.cc:226] gRPC config for type.googleapis.com/envoy.api.v2.Listener update rejected: Error adding/updating listener ingress_https: Failed to load private key from
[2019-01-21 08:13:17.399][1][warning][config] bazel-out/k8-opt/bin/source/common/config/_virtual_includes/grpc_mux_subscription_lib/common/config/grpc_mux_subscription_impl.h:70] gRPC config for type.googleapis.com/envoy.api.v2.Listener rejected: Error adding/updating listener ingress_https: Failed to load private key from

It works well when envoy is running, old config was keep working and the new config(which certificate & key has something wrong) will not loaded and warning logs raised.
But after envoy's restart, envoy will not listen HTTPS port any more before remove TLS certificate & key which caused Failed to load private key from <inline>, all of the HTTPS services are not available.

This certificate & key was issued by freessl.cn, I've tested them with nginx and gin.RunTLS(), they're all work very well. envoy's warning was mystifying to me, and it cannot be stopped simply after restart if there's something certificate has problem.

Repro steps:

Add TLS certificate envoy-warning-cert-sample.zip to kubernetes' secret.
Apply Ingress config with TLS

everybody can test this certificate & key, it's just test use, don't worry about security issues.

Versions:

envoy: v1.8.0
contour: v0.8.0

question

Source

exiaohao

Most helpful comment

@PiotrSikora Thanks for your help, I know it's something wrong with CA and issuer.

But on envoy side, a corrupted private key should NOT cause envoy's HTTPS port down after restart, it should keep running without the private key which is corrupted. And logs can more detail, it helps us find out which cert/key is illegal.

exiaohao on 24 Jan 2019

👍4

All 13 comments

I've tested key with OpenSSL:

$ openssl rsa -in server.key -check -noout
4631098988:error:04FFF07E:rsa routines:CRYPTO_internal:iqmp not inverse of q:/BuildRoot/Library/Caches/com.apple.xbs/Sources/libressl/libressl-22.230.1/libressl-2.6/crypto/rsa/rsa_chk.c:199:

exiaohao on 22 Jan 2019

cc @PiotrSikora

mattklein123 on 22 Jan 2019

@exiaohao per the message you pasted, the private key is corrupted:

$ openssl rsa -in server.key -check -noout
RSA key error: iqmp not inverse of q

BoringSSL (and therefore Envoy) won't accept it:

$ bssl s_server -accept 12345 -key server.key -cert server.crt
Failed to load private key: server.key
94140145633672:error:0400006f:RSA routines:OPENSSL_internal:CRT_VALUES_INCORRECT:/path/to/boringssl/crypto/fipsmodule/rsa/rsa.c:747:
94140145633672:error:04000068:RSA routines:OPENSSL_internal:BAD_RSA_PARAMETERS:/path/to/boringssl/crypto/rsa_extra/rsa_asn1.c:193:
94140145633672:error:06000066:public key routines:OPENSSL_internal:DECODE_ERROR:/path/to/boringssl/crypto/evp/p_rsa_asn1.c:146:
94140145633672:error:13000068:PKCS8 routines:OPENSSL_internal:DECODE_ERROR:/path/to/boringssl/crypto/pkcs8/pkcs8_x509.c:128:
94140145633672:error:0900000c:PEM routines:OPENSSL_internal:ASN.1 encoding routines:/path/to/boringssl/crypto/pem/pem_pkey.c:139:
94140145633672:error:10000009:SSL routines:OPENSSL_internal:PEM routines:/path/to/boringssl/ssl/ssl_file.cc:487:

Surprisingly, OpenSSL accepts it (even though it says it's corrupted in the openssl rsa -check):

$ openssl s_server -accept 12345 -key server.key -cert server.crt
Using default temp DH parameters
ACCEPT

There is not much we can do about it on the Envoy side, you should contact your CA and let them know that they produce corrupted private keys (but really, you should be generating private keys yourself, and only let CA generate the public certificate).

PiotrSikora on 23 Jan 2019

🎉3

@PiotrSikora Thanks for your help, I know it's something wrong with CA and issuer.

exiaohao on 24 Jan 2019

👍4

@exiaohao as far as I understand your original message, this works as intended.

When you replace working private key with corrupted private key over xDS, the configuration is rejected and Envoy continues to serve traffic using the last known good configuration.

When your restart (i.e. stop and start) Envoy with configuration depending on the corrupted private key, then Envoy cannot revert to the last known good configuration, since the very first configuration is already broken.

If you expect Envoy to start with all filter chains working, other than the one with corrupted private key, then that's not something that's supposed to work, because you'd have (a) only part of the supplied configuration loaded, leading to unexpected behavior, (b) silent failure, since it's unlikely that you'd notice this if Envoy started and served traffic.

PiotrSikora on 24 Jan 2019

@PiotrSikora yes we'd notice this warning and resolve it asap.
I mean is there a way to minimize the impact after restart with the private key is corrupted(e.g. pass the bad configuration, keep others running). HTTPS services are totally down is unacceptable and leads a terrible affect.

exiaohao on 24 Jan 2019

@exiaohao you should validate the configuration before restarting Envoy with it, i.e.

envoy-static --mode validate -c production.yaml && <whatever you do to restart Envoy>

This way, you won't restart Envoy if your configuration includes corrupted private key (or any other errors, for that matter), leading to the same behavior as xDS, i.e. using the last good know configuration and ignoring the invalid one.

PiotrSikora on 24 Jan 2019

@PiotrSikora It's a good idea, I'll validate the configuration and cert/key before apply.

exiaohao on 24 Jan 2019

I get the same error with Envoy 1.14.1, using SDS.

Error:

type_url: "type.googleapis.com/envoy.api.v2.auth.Secret"
response_nonce: "2020-06-03 18:01:38.723546738 -0700 PDT m=+0.283259137"
error_detail {
  code: 13
  message: "Failed to load private key from <inline>"
}

The key was generated with openssl, using the Istio makefile:

costinm on 4 Jun 2020

More logs from envoy - I started with trace, I don't see any info on why it was rejected.

2020-06-03T18:01:38.724426Z     debug   envoy http      [external/envoy/source/common/http/async_client_impl.cc:96] async http request response headers (end_stream=false):
':status', '200'
'content-type', 'application/grpc'

2020-06-03T18:01:38.724435Z     trace   envoy http2     [external/envoy/source/common/http/http2/codec_impl.cc:515] [C0] about to recv frame type=0, flags=0
2020-06-03T18:01:38.724441Z     trace   envoy http2     [external/envoy/source/common/http/http2/codec_impl.cc:530] [C0] recv frame type=0
2020-06-03T18:01:38.724449Z     trace   envoy http      [external/envoy/source/common/http/async_client_impl.cc:112] async http request response data (length=6061 end_stream=false)
2020-06-03T18:01:38.724463Z     debug   envoy config    [external/envoy/source/common/config/grpc_mux_impl.cc:129] Received gRPC message for type.googleapis.com/envoy.api.v2.auth.Secret at version 2020-06-03 18:01:38.723546738 -0700 PDT m=+0.283259137
2020-06-03T18:01:38.724597Z     debug   envoy config    [external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:638] Secret is updated.
2020-06-03T18:01:38.727934Z     warning envoy config    [external/envoy/source/common/config/grpc_subscription_impl.cc:101] gRPC config for type.googleapis.com/envoy.api.v2.auth.Secret rejected: Failed to load private key from <inline>
2020-06-03T18:01:38.727943Z     debug   envoy init      [external/envoy/source/common/init/watcher_impl.cc:14] target SdsApi default initialized, notifying init manager Cluster xds-grpc

costinm on 4 Jun 2020

@costin can you paste matching certificate? It's hard for me to test it without it.

PiotrSikora on 4 Jun 2020

Hi @PiotrSikora , I ran into the same issue recently. I'm using Envoy 1.12 as an edge proxy to terminate TLS. It's powered by LDS grpc server that dynamically retrieves TLS certificate and builds a listener snapshot. The issue I observed recently is that in case if one of the certs is corrupted, Envoy starts error-ing out with this following error:

Failed to load private key from <inline> and in case if there the server is restarted, the entire cached config is gone, which leads to a hard down of the edge proxy.

Could you please clarify if this is fixed in the latest Envoy versions? Same goes to making the error log message more descriptive, as it's pretty hard to know which one of the hundreds of certs is corrupted.

Also, can I use this command envoy-static --mode validate -c production.yaml && <whatever you do to restart Envoy> for dynamically generated envoy configurations?

nmalika5 on 8 Dec 2020

Hi @PiotrSikora , I ran into the same issue recently. I'm using Envoy 1.12 as an edge proxy to terminate TLS. It's powered by LDS grpc server that dynamically retrieves TLS certificate and builds a listener snapshot. The issue I observed recently is that in case if one of the certs is corrupted, Envoy starts error-ing out with this following error:

Failed to load private key from <inline> and in case if there the server is restarted, the entire cached config is gone, which leads to a hard down of the edge proxy.

Could you please clarify if this is fixed in the latest Envoy versions?

I believe this is fixed if you're using SDS, since then only filter chain(s) with broken TLS certificate(s) won't work.

With LDS alone, I think it's still "broken", since the whole LDS update would be rejected, but there were so many changes to listeners over the past year that I'm not 100% if that's the case. You could always verify it yourself.

In any case, your control plane should verify that the configuration (including TLS certificates) is correct before pushing it out.

Also, you definitely shouldn't be using Envoy v1.12, it reached EOL and there is a ton of bugs fixed since it was released.

Same goes to making the error log message more descriptive, as it's pretty hard to know which one of the hundreds of certs is corrupted.

I agree, but the inlined certificates have no names, and neither do filter chains or listeners, so it's pretty hard to give a more descriptive error.

Also, can I use this command envoy-static --mode validate -c production.yaml && <whatever you do to restart Envoy> for dynamically generated envoy configurations?

I believe this only validates production.yaml and not the dynamic configuration, which could change between the time you verify it and the time you restart Envoy anyway.

PiotrSikora on 12 Dec 2020

Was this page helpful?

0 / 5 - 0 ratings