Aws-app-mesh-roadmap: Bug: Virtual Nodes will fail to listen for traffic if there are multiple backends using the same port but with different protocols

Created on 23 Mar 2020 · 2Comments · Source: aws/aws-app-mesh-roadmap

Summary
If a virtual node has two backends that share listening ports, the protocols of both listeners must be the same otherwise App Mesh will generate invalid Envoy configuration.

Steps to Reproduce

Create a virtual service provided by a virtual node, listening for http traffic on port 80.
The virtual service may also be provided a router that ultimately routes to an http virtual node.

http virtual node:

virtualNodeName: http-example-local-node
spec:
listeners:
- portMapping:
    port: 80
    protocol: http
serviceDiscovery:
  dns:
    hostname: http.example.local

http virtual service

virtualServiceName: http.example.local
spec:
provider:
  virtualNode:
    virtualNodeName: http-example-local-node

Create a virtual node listening for http2 traffic on port 80.

http2 virtual node:

virtualNodeName: http2-example-local-node
spec:
listeners:
- portMapping:
    port: 80
    protocol: http2
serviceDiscovery:
  dns:
    hostname: http2.example.local

http2 virtual service

virtualServiceName: http2.example.local
spec:
provider:
  virtualNode:
    virtualNodeName: http2-example-local-node

Create a final virtual node that uses the previous provided virtual nodes as backends:

virtualNodeName: frontend-example-local-node
spec:
backends:
- virtualService:
    virtualServiceName: http.example.local
- virtualService:
    virtualServiceName: http2.example.local
listeners:
- portMapping:
    port: 80
    protocol: http

Observe that the Envoy proxy for frontend-example-local-node will fail to initialize its listeners.

The Envoy logs will display

[warning][config] [source/common/config/grpc_mux_subscription_impl.cc:82] gRPC config for type.googleapis.com/envoy.api.v2.Listener rejected: Error adding/updating listener(s) lds_egress_0.0.0.0_15001: error adding listener '0.0.0.0:15001': multiple filter chains with the same matching rules are defined

meaning that App Mesh sent a Listener resource to the Envoy that looks like this:

name: lds_egress_0.0.0.0_15001
address:
  socketAddress:
    address: 0.0.0.0
    portValue: 15001
filterChains:
- filterChainMatch:
    destinationPort: 80
  filters:
  - name: envoy.http_connection_manager
    config:
      stat_prefix: egress
      rds:
        config_source:
          ads: {}
        route_config_name: rds_egress_http_80
      http_filters:
      - name: envoy.router
      tracing:
        operation_name: EGRESS
- filterChainMatch:
    destinationPort: 80
  filters:
  - name: envoy.http_connection_manager
    config:
      stat_prefix: egress
      rds:
        config_source:
          ads: {}
        route_config_name: rds_egress_http2_80
      http_filters:
      - name: envoy.router
      tracing:
        operation_name: EGRESS
- filterChainMatch:
    destinationPort: 443
    serverNames:
    - "*.amazonaws.com"
  filters:
  - name: envoy.tcp_proxy
    config:
      stat_prefix: egress
      cluster: cds_egress_app-mesh-example_amazonaws
listenerFilters:
- name: envoy.listener.original_dst
- name: envoy.listener.tls_inspector

Note that there are 2 filter chains matches, one for http and one for http2 that both match on port 80. This is ambiguous so Envoy rejects this configuration as well as its other Listener configuration that allows it to listen for ingress traffic.

Are you currently working around this issue?

This can be worked around by using different listening ports for the backend virtual nodes. e.g. the http node can use port 80 and the http2 node can use port 8080.

Bug Working on it High

Source