Envoy: set x-envoy-expected-rq-timeout-ms in header does not work

[shanshanzhu]

Issue Template

Title: One line description

Description:
In the http Request sent to envoy, we set

 {
   x-envoy-expected-rq-timeout-ms: 100000000
 }

In the call stack below, I trunked some sensitive fields in the header but we do see the x-envoy-upstream-rq-timeout-ms gets set originally but was overwritten to default value 15000

Call Stack:

[2018-09-17 23:49:26.195][14][trace][http] source/common/http/http1/codec_impl.cc:404] [C0] headers complete
[2018-09-17 23:49:26.195][14][trace][http] source/common/http/http1/codec_impl.cc:308] [C0] completed header: key=Accept-Language value=en-US,en;q=0.9,zh-CN;q=0.8,zh;q=0.7
[2018-09-17 23:49:26.195][14][debug][http] source/common/http/conn_manager_impl.cc:494] [C0][S16553777963597655641] request headers complete (end_stream=false):
':authority', 'localhost:8080'
'connection', 'keep-alive'
'content-length', '43'
'pragma', 'no-cache'
'cache-control', 'no-cache'
'x-envoy-upstream-rq-timeout-ms', '100000000'
'accept', '*/*'
'referer', 'http://localhost:3000/'
'accept-encoding', 'gzip, deflate, br'
'accept-language', 'en-US,en;q=0.9,zh-CN;q=0.8,zh;q=0.7'

[2018-09-17 23:49:26.195][14][trace][http] source/common/http/conn_manager_impl.cc:723] [C0][S16553777963597655641] decode headers called: filter=0x2e0d310 status=0
[2018-09-17 23:49:26.195][14][trace][http] source/common/http/conn_manager_impl.cc:723] [C0][S16553777963597655641] decode headers called: filter=0x2e0d270 status=0
[2018-09-17 23:49:26.195][14][debug][router] source/common/router/router.cc:252] [C0]
[2018-09-17 23:49:26.195][14][debug][router] source/common/router/router.cc:303] [C0][S16553777963597655641] router decoding headers:
':authority', 'localhost:8080'
':method', 'POST'
':scheme', 'http'
'pragma', 'no-cache'
'cache-control', 'no-cache'
'content-type', 'application/grpc'
'x-grpc-web', '1'
'accept', '*/*'
'referer', 'http://localhost:3000/'
'accept-encoding', 'gzip, deflate, br'
'accept-language', 'en-US,en;q=0.9,zh-CN;q=0.8,zh;q=0.7'
'x-forwarded-proto', 'http'
'te', 'trailers'
'grpc-accept-encoding', 'identity,deflate,gzip'
'x-envoy-expected-rq-timeout-ms', '15000'

[shanshanzhu]

I ended up setting timeout: 1000000000s for each route. It is not very ideal though.

                route:
                  timeout: 1000000000s

[htuch]

@shanshanzhu take a look at https://www.envoyproxy.io/docs/envoy/latest/configuration/http_filters/router_filter#x-envoy-expected-rq-timeout-ms. You need to either set the route timeout as you did or set x-envoy-upstream-rq-timeout-ms.

[stale[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted". Thank you for your contributions.

[hjkatz]

To help those in the future, the use_remote_address setting and x-forwarded-for headers can also impact this. Only trusted sources will have their headers read, and therefore (in our case) when running localhost envoy as a local proxy we want our config to use_remote_address: true to trust the most immediate source (localhost) and all of its headers.