Envoy: Question: Getting "no matching filter chain found" when enabling proxy_protocol filter

[aalmekhlafi00]

_Description_

I am trying to add client IP transparency to my stack in AWS using proxy protocol.
My setup is:
aws NLB -> Envoy proxy -> Service.
I enabled Proxy Protocol V2 on the NLB as well as I added envoy.listener.proxy_protocol filter to envoy config as following:

static_resources:
    listeners:
        address:
            socket_address:
                address: 0.0.0.0
                port_value: 443
        listener_filters:
        -   name: envoy.listener.tls_inspector
            typed_config: {}
        -   name: envoy.listener.proxy_protocol
        filter_chains:
        -   filter_chain_match:
                source_prefix_ranges:
                -   address_prefix: 127.0.0.0
                    prefix_len: 8
                -   address_prefix: 10.0.0.0
                    prefix_len: 8
            filters:
            -   name: envoy.http_connection_manager
                typed_config:
                    '@type': type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager
                    codec_type: auto
                    use_remote_address: true
                    common_http_protocol_options:
                        idle_timeout: 300s
                    stat_prefix: gateway_ingress_http
                    access_log:
                        name: envoy.file_access_log
                        config:
                            path: /var/log/envoy/access.log
                            format: "[%START_TIME(%d/%m/%YT%H:%M:%S%z %s)%] method=%REQ(:METHOD)%\
                                \ authority=%REQ(:AUTHORITY)% txId=%RESP(INTUIT_TID)%\
                                \ request=%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% alpn=%PROTOCOL%\
                                \ response_flag=%RESPONSE_FLAGS% ssl_protocol=%DOWNSTREAM_TLS_VERSION%\
                                \ ssl_cipher=%DOWNSTREAM_TLS_CIPHER% status=%RESPONSE_CODE%\
                                \ status_details=%RESPONSE_CODE_DETAILS% request_size=%BYTES_RECEIVED%\
                                \ response_size=%BYTES_SENT% envoy_request_id=%REQ(X-REQUEST-ID)%\
                                \ user-agent=%REQ(USER-AGENT)% xFor=%DOWNSTREAM_REMOTE_ADDRESS_WITHOUT_PORT%\
                                \ x-envoy-upstream-service-time=%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%\
                                \ total_time_ms=%DURATION% upstream=%UPSTREAM_HOST%\n"
                    route_config:
                        name: my_cluster
                        virtual_hosts:
                        -   name: my_cluster
                            domains:
                            -   '*'
                            routes:
                            -   match:
                                    prefix: /
                                route:
                                    cluster: my_cluster
                    http_filters:
                    -   name: envoy.router
                        typed_config: {}
            transport_socket:
                name: envoy.transport_sockets.tls
                typed_config:
                    '@type': type.googleapis.com/envoy.api.v2.auth.DownstreamTlsContext
                    common_tls_context:
                        tls_params:
                            tls_maximum_protocol_version: TLSv1_2
                            tls_minimum_protocol_version: TLSv1_2
                        alpn_protocols: h2
                        tls_certificates:
                        -   certificate_chain:
                                filename: /pathto/cert.crt
                            private_key:
                                filename: /pathto/key.key
        -   filter_chain_match:
                server_names:
                -   '*.domain.com'
                -   localhost
                -   '*.elb.us-west-2.amazonaws.com'
                -   '*.elb.us-east-2.amazonaws.com'
            transport_socket:
                name: envoy.transport_sockets.tls
                typed_config:
                    '@type': type.googleapis.com/envoy.api.v2.auth.DownstreamTlsContext
                    common_tls_context:
                        tls_params:
                            tls_maximum_protocol_version: TLSv1_2
                            tls_minimum_protocol_version: TLSv1_2
                        alpn_protocols: h2
                        tls_certificates:
                        -   certificate_chain:
                                filename: /pathto/cert.crt
                            private_key:
                                filename: /pathto/key.key
            filters:
            -   name: envoy.http_connection_manager
                typed_config:
                    '@type': type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager
                    codec_type: auto
                    use_remote_address: true
                    common_http_protocol_options:
                        idle_timeout: 300s
                    stat_prefix: gateway_ingress_http
                    access_log:
                        name: envoy.file_access_log
                        config:
                            path: /var/log/envoy/access.log
                            format: "[%START_TIME(%d/%m/%YT%H:%M:%S%z %s)%] method=%REQ(:METHOD)%\
                                \ authority=%REQ(:AUTHORITY)% txId=%RESP(INTUIT_TID)%\
                                \ request=%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% alpn=%PROTOCOL%\
                                \ response_flag=%RESPONSE_FLAGS% ssl_protocol=%DOWNSTREAM_TLS_VERSION%\
                                \ ssl_cipher=%DOWNSTREAM_TLS_CIPHER% status=%RESPONSE_CODE%\
                                \ status_details=%RESPONSE_CODE_DETAILS% request_size=%BYTES_RECEIVED%\
                                \ response_size=%BYTES_SENT% envoy_request_id=%REQ(X-REQUEST-ID)%\
                                \ user-agent=%REQ(USER-AGENT)% xFor=%DOWNSTREAM_REMOTE_ADDRESS_WITHOUT_PORT%\
                                \ x-envoy-upstream-service-time=%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%\
                                \ total_time_ms=%DURATION% upstream=%UPSTREAM_HOST%\n"
                    route_config:
                        name: my_cluster
                        virtual_hosts:
                        -   name: my_cluster
                            domains:
                            -   '*'
                            routes:
                            -   match:
                                    prefix: /
                                route:
                                    cluster: my_cluster
                    http_filters:
                    -   name: envoy.router
                        typed_config: {}
    clusters:
    -   name: my_cluster
        http2_protocol_options: {}
        connect_timeout: 1s
        type: static
        dns_lookup_family: V4_ONLY
        lb_policy: round_robin
        load_assignment:
            cluster_name: my_cluster
            endpoints:
            -   lb_endpoints:
                -   endpoint:
                        address:
                            socket_address:
                                address: 127.0.0.1
                                port_value: 7000
admin:
    access_log_path: /pathto/admin.log
    address:
        socket_address:
            address: 127.0.0.1
            port_value: 6000

when Making a call from the client ex. curl -v https://hostname/path
Client logs
LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to hostname:443

Envoy logs

[2020-02-11 10:45:58.100][7698][debug][filter] [external/envoy/source/extensions/filters/listener/tls_inspector/tls_inspector.cc:72] tls inspector: new connection accepted
[2020-02-11 10:45:58.100][7698][debug][filter] [external/envoy/source/extensions/filters/listener/proxy_protocol/proxy_protocol.cc:30] proxy_protocol: New connection accepted
[2020-02-11 10:45:58.100][7698][debug][conn_handler] [external/envoy/source/server/connection_handler_impl.cc:330] closing connection: no matching filter chain found

Is there any documentation on how to configure proxy_protocol other than this?

[zuercher]

I'm not super familiar with the proxy protocol, but here are some suggestions:

1. Pretty sure the curl SSL error is because the listener is not configured for TLS. Just having the TLS inspector is not sufficient. See transport_socket in the FilterChain config.
2. If the proxy protocol listener filter is enabled, I believe Envoy will ignore data until it sees a valid proxy protocol header. You can get curl to do that by adding the --haproxy-protocol flag (at least in my version).
3. Finally is that your complete config? Without at least one network filter defined, Envoy will just close the connection even if it gets a proxy protocol request. For example, the HTTP connection manager filter defined in this example would forward requests two hosts (based on DNS names, there are other ways to configure the cluster and service discovery in general).

I was able to proxy requests locally by doing steps 2 / 3 (and using a non-https URL).

[aalmekhlafi00]

Thanks for your response. I updated the question with the complete config.
In the filterChain i only miss use_proxy_proto which the listener should expect a PROXY protocol V1 header if the value is true.

I will try to run curl with --haproxy-protocol flag, although AWS confirmed that proxy protocol headers will be sent when proxy protocol v2 is enabled in the NLB target group

[kflynn]

We ran across this, and dealt with it by _not_ configuring Envoy using use_proxy_proto, but instead switching to the proxy_protocol filter. To make that switch work, we had to _not_ include use_proxy_proto: true in the filter chain, and we had to make sure that the proxy_protocol listener filter came before the tls_inspector filter. Hope that's helpful!

[stale[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted". Thank you for your contributions.