Envoy: Mirrored traffic has `-shadow` appended to the hostname in the Host header breaking routing in downstream proxies

[domhauton]

Title: Cannot route traffic shadowed by envoy due to addition of -shadow to Host header

Description:
The Host and Authority header are modified when shadowing traffic, even when using host_rewrite to modify Host manually on the route config.

This results in mirrored requested being dropped by reverse proxies further downstream.

Example Config:

Main config ( /etc/envoy/config.yaml ):

static_resources:
  listeners:
  - address:
      socket_address:
        address: 0.0.0.0
        port_value: 8080
    filter_chains:
    - filters:
      - name: envoy.http_connection_manager
        config:
          codec_type: auto
          stat_prefix: ingress_http
          rds:
            route_config_name: local_route
            config_source:
              path: /etc/envoy/config.route.yaml
          http_filters:
          - name: envoy.router
            config: {}
  clusters:
  - name: target-service
    connect_timeout: 15s
    type: LOGICAL_DNS
    dns_lookup_family: V4_ONLY
    lb_policy: round_robin
    hosts:
      - socket_address:
          address: localhost
          port_value: 8081
  - name: mirror-service
    connect_timeout: 15s
    type: LOGICAL_DNS
    dns_lookup_family: V4_ONLY
    lb_policy: round_robin
    hosts:
      - socket_address:
          address: foobar.new-cluster.company.com
          port_value: 443
    tls_context: { sni: foobar.new-cluster.company.com }

node:
  cluster: cluster0
  id: node0

Route config ( /etc/envoy/config.route.yaml ):

version_info: "0"
resources:
- "@type": type.googleapis.com/envoy.api.v2.RouteConfiguration
  name: local_route
  virtual_hosts:
  - name: target-service
    domains:
    - "*"
    routes:
    - match:
        prefix: "/"
      route:
        cluster: target-service
        host_rewrite: foobar.new-cluster.company.com
        request_mirror_policy:
          cluster: mirror-service
          runtime_fraction:
            default_value:
              numerator: 34

What happens:

1. Request comes into envoy for host (target host "foobar.old-cluster.company.com" with Header: "HOST: foobar.old-cluster.company.com")
2. Request is forwarded directly to ip of the server (e.g.: localhost:8081 with Header: "HOST: foobar.new-cluster.company.com")
3. Request is mirrored to mirror service (e.g.: foobar.new-cluster.company.com:443 with Header: "HOST: foobar.new-cluster.company.com-shadow")
4. Request is dropped by downstream ELB as the Host is invalid.

Expected behaviour:

1. Request comes into envoy for host (target host "foobar.old-cluster.company.com" with Header: "HOST: foobar.old-cluster.company.com")
2. Request is forwarded directly to ip of the server (e.g.: localhost:8081 with Header: "HOST: foobar.new-cluster.company.com")
3. Request is mirrored to mirror service (e.g.: foobar.new-cluster.company.com:443 with Header: "HOST: foobar.new-cluster.company.com")
4. Request is routed to mirror service downstream correctly.

Relevant Links:

• Code causing issue: https://github.com/envoyproxy/envoy/blob/b9f798dd1dec86d3ed067c449e3c3a49c57984b1/source/common/router/shadow_writer_impl.cc#L30

Potential Solution:

We could solve this by adding an overriding host_rewrite in the request_mirror_policy config. As an example:

...
        request_mirror_policy:
          cluster: mirror-service
          host_rewrite: "foobar.new-cluster.company.com" # (Default: "" would just append shadow)
          runtime_fraction:
            default_value:
              numerator: 34
...

[derekargueta]

The solution that immediately comes to mind for me is adding an option to the shadow policy resembling something like enable_shadow_host_suffix_append that defaults to true (to protect existing users) but could be set to false to achieve the desired behavior of not adding the suffix. Does this sound appropriate @domhauton ?

[domhauton]

@derekargueta That would be awesome. proto3 does default to false for booleans, so the name may be better as disable_shadow_host_suffix_append.

Would you be happy to receive a PR for this?

[derekargueta]

I think it sounds like a fine feature addition but @envoyproxy/api-shepherds would need to confirm

[mattklein123]

Yup SGTM @domhauton. Thank you!

[domhauton]

For anyone who comes across this before a PR is created: As a temporary workaround we're currently re-routing mirrored traffic through the same envoy instance on a second listener to do a host header rewrite. It will add latency to mirrored requests but as they are mirrored and we ignore the response anyway it doesn't make too much difference. It does add some cpu/memory usage. A nice side-effect is we get better tracing by splitting it this way.

(I'm not prioritising working on this as the workaround is sufficient, albeit slightly inelegant)

[coffenbacher]

Thanks for the workaround idea @domhauton , that is working for me as well.

[leosunmo]

In my current setup, routing the traffic around to a second listener is not possible without a lot of extra infrastructure changes. Is anyone looking at this PR to add a boolean to disable rewrites?
I have however realised that I will probably have to rewrite the Authority header anyway since the staging/testing setup won't be listening for the production Authority header anyway...

Would still be very nice to have this flag for other scenarios.

[brunsgaard]

I ran into this today and created a config that gets the job done.
It can be used on a workaround until a prober fix arrives for this issue.

static_resources:
  listeners:
  - name: http
    address:
      socket_address:
        address: 0.0.0.0
        port_value: 8080
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: collector_mirror_proxy
          route_config:
            virtual_hosts:
            - name: local
              domains:
              - "*"
              routes:
              - match:
                  prefix: "/"
                  headers:
                  - name: :authority
                    suffix_match: "localhost-shadow"
                route:
                  auto_host_rewrite: true
                  cluster: collector-mirror|443

              - match:
                  prefix: "/"
                route:
                  host_rewrite_literal: localhost
                  cluster: collector|8081
                  request_mirror_policies:
                  - cluster: self|8080
                    runtime_fraction:
                      default_value:
                        numerator: 1
          http_filters:
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
  clusters:
  - name: self|8080
    connect_timeout: 0.25s
    type: static
    load_assignment:
      cluster_name: self|8080
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: 127.0.0.1
                port_value: 8080
  - name: collector|8081
    connect_timeout: 0.25s
    type: static
    load_assignment:
      cluster_name: collector|8081
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: 127.0.0.1
                port_value: 8081
  - name: collector-mirror|443
    connect_timeout: 0.25s
    type: logical_dns
    http_protocol_options: {}
    lb_policy: round_robin
    per_connection_buffer_limit_bytes: 32768 # 32 KiB
    load_assignment:
      cluster_name: collector-mirror|443
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: collector.brunsgaard.io
                port_value: 443
    transport_socket:
      name: envoy.transport_sockets.tls
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
        sni: collector.brunsgaard.io