Envoy: Upstream request header modification

[bartebor]

I would like to have a way of modifying upstream request headers _including_ host/:authority.
I can alter headers via request_headers_to_add or lua code, but changing host/:authority is done before virtual host matching and so it changes virtual host picked, or results in 404.
I need virtual host to be picked using downstream request's host/:authority and only then alter headers sent to upstream.

Why do I need it: I would like to multiplex traffic from many services to envoy through single listener with one virtual host per service. Envoy is not aware of all domains (with possible subdomains) of these services, so I would substitute original host with known per service vhost name storing original host in some other header first. Envoy selects exact virtual hosts efficiently with a map and so I would like not to use wildcard domains, or long route chains matching custom headers.

Example of traffic:

Original request      | Request from load balancer to Envoy         | Request from Envoy to upstream
------------------------------------------------------------------------------------------------------
Host: a.example.com   | Host: a,  X-Original-Host: a.example.com    | vhost a -> Host: a.example.com
Host: b.example.com   | Host: b,  X-Original-Host: b.example.com    | vhost b -> Host: b.example.com
Host: a.c.example.com | Host: c,  X-Original-Host: a.c.example.com  | vhost c -> Host: a.c.example.com
Host: b.c.example.com | Host: c,  X-Original-Host: b.c.example.com  | vhost c -> Host: b.c.example.com

Is it possible?

[mattklein123]

I think this might be possible with a custom filter that does an internal redirect. @alyssawilk WDYT?

[alyssawilk]

I'm not 100% sure I'm following the request. Is the problem that by the time the request reaches Envoy it's doing matching based on a, where you want to do a.example.com, or you want to pick on a but forward a.example.com? You can definitely swap the host with a custom filter, but the question of if you need an internal redirect or just a catch-all service and to rewrite authority and clear/reset the cached route depends on what you're aiming for.

[bartebor]

@alyssawilk If I understood you correctly, it is the latter scenario you describe: I want Envoy to match virtual host using a, but forward a.example.com to upstream service.

We hava a (non-Envoy) load balancer handling many services on multiple IPs. This LB should forward traffic to Envoy, which in turn handles multiple upstreams (clusters) in kubernetes cluster. I would like to have a simple, but efficient interface between LB and Envoy, avoiding maintaning multiple listeners, multitude of service domains etc. at the Envoy's layer.

I thought it would be best to assign each application fixed domain for LB to store in host/:authority so Envoy could select virtual host efficiently. At the same time I would like to preserve original domain in a way that upstream services relying on host/:authority continue to work with no changes.

[alyssawilk]

Using HTTP/1 semantics just for simplicity (swap :authority for H2) I think if your LB is fowarding to Envoy with
Host: a
And you want to route select on
Host: a
and then forward
Host: a.example.com
you can just configure Envoy to route based on A, and add a custom filter which on decodeHeaders replaces the authority from the x-original-host header. Alternately if there aren't infinite combinations and permutations you might be able to configure host_rewrite rules and skip the custom filter but I'm not terribly familiar with that feature so unsure if it'd meet your needs.

[bartebor]

@alyssawilk Thank you for your response. Unfortunately neither host_rewrite nor auto_host_rewrite meet my needs, because of using fixed values. If only some kind of host_rewrite_header option existed, I would have what I need. Do you think it is reasonable to implement? If not, I will probably have to follow custom filter path.

[alyssawilk]

Hm, I think it would be generically useful but also somewhat scary from a security perspective. @envoyproxy/senior-maintainers thoughts on this?

[mattklein123]

Couldn't this be done in a custom filter?

[bartebor]

I will try creating custom filter when I find some time, but still it seems that copying/composing upstream headers (including authority) from other headers would be very handy feature to have in vanilla Envoy. I agree that allowing modification of authority header may pose some security risk, but there are other envoy configuration options which may be misused or abused. I think it is administrator's responsibility to configure their Envoy instance in a safe manner in given environment.

[stale[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted". Thank you for your contributions.

[bartebor]

I have finally found some time to tackle this.

It seems that it is not possible to achieve described scenario with custom filter. The problem is that route filter contains whole process of selecting vhost and forwarding request to upstream and must be the last filter on http connection manager's filter list.

Any :authority/host header modification done with custom filter executing before route filter will affect selected vhost. At the same time there is no way to hook into RouteEntryImplBase::finalizeRequestHeaders. Existing header alteration methods either do not allow to modify :authority/host header (request_header_to_add) or use constant strings only (host_rewrite, auto_host_rewrite).

Internal redirect won't work because of above reasons (and it needs separate service to supply 302s, what seems like a not a very good idea in terms of performance).

I decided to add another rewrite option, auto_host_rewrite_header and I will open a PR for evaluation. I made it clear in the docs that using this option requires special care.

[markschmid]

I'm interested in accessing the original Host information at the cluster level, instead of having it rewritten by envoy. Actually I have not found a solution for this so far (is there really no way?).

Will the following do the trick for me? (From the docs: "If header value is empty, host header is left intact.")
auto_host_rewrite_header: ""

For some reason, I have not been able to get it to work, envoy (v1.11.2) fails to start, stating
auto_host_rewrite_header: Cannot find field.

[bartebor]

AFAIK envoy does not normally alter Host information unless you make it to; upstream request will have whatever Host came from downstream. My PR is about doing something opposite: I _want_ to alter Host being sent to cluster endpoint with other header's value. The field you used should contain this other header name, not it's value which is taken from downstream request.

[markschmid]

Thanks for the clarification, it was really helpful and helped me to realize that it is in fact a different component in my stack which has been altering the host information all this time...

[randavidovitz]

@markschmid are you sure that the HOST is not manipulated ?