Aws-app-mesh-roadmap: Bug: no matching filter chain found with TLS configuration

[jessie00chen]

Summary
HTTPS traffic coming into a virtual node (colorteller gateway) cannot talk to the other virtual node (colorteller) after configured TLS.

Steps to Reproduce

1. Setup ALB -> Target group -> colorteller gateway virtual node -> colorteller virtual node
2. Configure Target group healthcheck with HTTPS:9080
3. Configure ACM for the certificate that will be used in virtual node
4. Turn on colorteller gateway virtual node and colorteller virtual node with TLS termination
5. Getting 500 error on colorteller gateway virtual node and the reason is   | 2020-08-07T16:50:10.468-07:00 | [2020-08-07 23:50:10.468][32][debug][conn_handler] [source/server/connection_handler_impl.cc:321] closing connection: no matching filter chain found

Are you currently working around this issue?
Didn't find a workaround

Additional context
Envoy logs on colorteller gateway

2020-08-07T16:50:10.466-07:00 | ':authority', 'xxxxx.us-east-1.elb.amazonaws.com'
-- | --
  | 2020-08-07T16:50:10.466-07:00 | ':path', '/color'
  | 2020-08-07T16:50:10.466-07:00 | ':method', 'GET'
  | 2020-08-07T16:50:10.466-07:00 | 'x-forwarded-for', '73.222.x.x'
  | 2020-08-07T16:50:10.466-07:00 | 'x-forwarded-proto', 'https'
  | 2020-08-07T16:50:10.466-07:00 | 'x-forwarded-port', '443'
  | 2020-08-07T16:50:10.466-07:00 | 'x-amzn-trace-id', 'Root=1-5f2de8b2-a64ad924402992e09b17f294'
  | 2020-08-07T16:50:10.466-07:00 | 'user-agent', 'curl/7.54.0'
  | 2020-08-07T16:50:10.466-07:00 | 'accept', '*/*'
  | 2020-08-07T16:50:10.466-07:00 | [2020-08-07 23:50:10.466][33][debug][http] [source/common/http/conn_manager_impl.cc:1261] [C310][S7613516894816924328] request end stream
  | 2020-08-07T16:50:10.466-07:00 | [2020-08-07 23:50:10.466][33][debug][tracing] [source/extensions/tracers/xray/xray_tracer_impl.cc:160] start sampling strategy.
  | 2020-08-07T16:50:10.466-07:00 | [2020-08-07 23:50:10.466][33][debug][tracing] [source/extensions/tracers/xray/xray_tracer_impl.cc:172] starting generate a segment based on downstream xray header.
  | 2020-08-07T16:50:10.466-07:00 | [2020-08-07 23:50:10.466][33][debug][router] [source/common/router/router.cc:434] [C310][S7613516894816924328] cluster 'cds_ingress_service-dev-colorteller_colorgateway-vn_http_9080' match for URL '/color'
  | 2020-08-07T16:50:10.466-07:00 | [2020-08-07 23:50:10.466][33][debug][router] [source/common/router/router.cc:549] [C310][S7613516894816924328] router decoding headers:
  | 2020-08-07T16:50:10.466-07:00 | ':authority', 'xxxx.us-east-1.elb.amazonaws.com'
  | 2020-08-07T16:50:10.466-07:00 | ':path', '/color'
  | 2020-08-07T16:50:10.466-07:00 | ':method', 'GET'
  | 2020-08-07T16:50:10.466-07:00 | ':scheme', 'http'
  | 2020-08-07T16:50:10.466-07:00 | 'x-forwarded-for', '73.222.x.x'
  | 2020-08-07T16:50:10.466-07:00 | 'x-forwarded-proto', 'https'
  | 2020-08-07T16:50:10.466-07:00 | 'x-forwarded-port', '443'
  | 2020-08-07T16:50:10.466-07:00 | 'x-amzn-trace-id', 'Root=1-5f2de8b2-a64ad924402992e09b17f294; Parent=d77daaa78c0eb259; Sampled=1'
  | 2020-08-07T16:50:10.466-07:00 | 'user-agent', 'curl/7.54.0'
  | 2020-08-07T16:50:10.466-07:00 | 'accept', '*/*'
  | 2020-08-07T16:50:10.467-07:00 | 'x-request-id', 'dc863e68-2d8a-9881-94bf-dcc64fb4b683'
  | 2020-08-07T16:50:10.467-07:00 | 'x-envoy-expected-rq-timeout-ms', '15000'
  | 2020-08-07T16:50:10.467-07:00 | [2020-08-07 23:50:10.466][33][debug][pool] [source/common/http/http1/conn_pool.cc:104] [C14] using existing connection
  | 2020-08-07T16:50:10.467-07:00 | [2020-08-07 23:50:10.466][33][debug][router] [source/common/router/router.cc:1618] [C310][S7613516894816924328] pool ready
  | 2020-08-07T16:50:10.468-07:00 | [2020-08-07 23:50:10.468][32][debug][filter] [source/extensions/filters/listener/original_dst/original_dst.cc:18] original_dst: New connection accepted
  | 2020-08-07T16:50:10.468-07:00 | [2020-08-07 23:50:10.468][32][debug][filter] [source/extensions/filters/listener/tls_inspector/tls_inspector.cc:78] tls inspector: new connection accepted
  | 2020-08-07T16:50:10.468-07:00 | [2020-08-07 23:50:10.468][32][debug][filter] [source/extensions/filters/listener/tls_inspector/tls_inspector.cc:148] tls:onServerName(), requestedServerName: colorteller.appmesh.local
  | 2020-08-07T16:50:10.468-07:00 | [2020-08-07 23:50:10.468][32][debug][conn_handler] [source/server/connection_handler_impl.cc:321] closing connection: no matching filter chain found
  | 2020-08-07T16:50:10.468-07:00 | [2020-08-07 23:50:10.468][33][debug][router] [source/common/router/router.cc:1036] [C310][S7613516894816924328] upstream headers complete: end_stream=false
  | 2020-08-07T16:50:10.468-07:00 | [2020-08-07 23:50:10.468][33][debug][http] [source/common/http/conn_manager_impl.cc:1556] [C310][S7613516894816924328] encoding headers via codec (end_stream=false):
  | 2020-08-07T16:50:10.468-07:00 | ':status', '500'
  | 2020-08-07T16:50:10.468-07:00 | 'x-amzn-trace-id', 'Root=1-5f2de8b2-a64ad924402992e09b17f294'
  | 2020-08-07T16:50:10.468-07:00 | 'date', 'Fri, 07 Aug 2020 23:50:10 GMT'
  | 2020-08-07T16:50:10.468-07:00 | 'content-length', '22'
  | 2020-08-07T16:50:10.468-07:00 | 'content-type', 'text/plain; charset=utf-8'
  | 2020-08-07T16:50:10.468-07:00 | 'x-envoy-upstream-service-time', '1'
  | 2020-08-07T16:50:10.468-07:00 | 'server', 'envoy'

Attachments
If you think you might have additional information that you'd like to include via an attachment, please do - we'll take a look. (Remember to remove any personally-identifiable information.)

[bcelenza]

  | 2020-08-07T16:50:10.468-07:00 | [2020-08-07 23:50:10.468][32][debug][filter] [source/extensions/filters/listener/tls_inspector/tls_inspector.cc:148] tls:onServerName(), requestedServerName: colorteller.appmesh.local
  | 2020-08-07T16:50:10.468-07:00 | [2020-08-07 23:50:10.468][32][debug][conn_handler] [source/server/connection_handler_impl.cc:321] closing connection: no matching filter chain found

This snippet suggests to me that the gateway application is attempting to originate a TLS session (instead of allowing the proxy to do it). Specifically, the Envoy has picked up a Server Name Indication in the TLS session originated by the application.

Can you point me to the source code for the gateway you've used? It should be making a plain-old HTTP request (like this example).

[jessie00chen]

Thanks @bcelenza for your quick response, I have turn the gateway application(from the same example) back to HTTP, and was able to route the request to the colorteller virtual node. I have replaced the colorteller virtual node with my test application. However, the test application is hitting another 500 error due to network traffic goes out to New Relic and Rollbar.

NewRelic::Agent::Agent::InstanceMethods::Connect::WaitOnConnectTimeout: Agent was unable to connect in 10 seconds

&

[Rollbar] Error processing the item: Errno::ECONNRESET, Connection reset by peer - SSL_connect.

Do I have to configure each individual egress on Envoy for external services for TLS? I was reading this issue and seems like with TLS release, it will make the config easier?

[bcelenza]

Hey @jessie00chen,

There are a few options, depending on your requirements for proxying traffic from the New Relic and Rollbar agents:

1. You can model the New Relic and Rollbar endpoints in App Mesh using a Virtual Service with a Virtual Node provider, and have App Mesh negotiate TLS by applying a client policy. For this option you would need to disable TLS on in the agents (for New Relic, looks like there is still a deprecated option to do this, at least in the Go agent). This has the benefit of providing metrics for outbound HTTP calls via the proxy, but you may not need or want those.
2. You can set the Mesh egress filter to ALLOW_ALL, which will allow the agents to communicate and still negotiate TLS. The proxy will still intercept and handle traffic, but will only generate metrics for OSI Layer 4 (connections, bytes over the wire, etc.).
3. If you want the agents to bypass the proxy all together, you may be able to run them as Linux user ID 1337. All outbound communication from processes which use this UID is not proxied by Envoy.

I’d generally recommend approach (3) for this use case, but it at least looks like running the New Relic agent as a custom UID is not an option. If it’s not possible to run the agent as UID 1337, option (2) is a good approach to allow the communication through in a simple manner.

[bcelenza]

We're closing the issue at this time. Additionally, we're planning to make some changes to the egress filter to allow specific destinations as part of #2, so this use case should be improved in the near future.

Please feel free to open a new one if you run into any other challenges.

[jessie00chen]

Thanks @bcelenza!