Envoy: Prevent sensitive request headers from being logged in debug level

[nulltrope]

Description:
Hey all, first just some quick background before getting into the issue we're experiencing. We wrote a small service that is called by the ext_authz filter to perform an OIDC Authorization Flow, and verify existing JWTs stored as cookies in request headers.

The JWT cookie contains sensitive information and therefore shouldn't be logged, however we noticed that when envoy is run with -l debug, various components will log the full request headers including the JWT, e.g.:

envoy_1         | [2020-01-10 16:29:21.082][14][debug][http] [source/common/http/conn_manager_impl.cc:246] [C5] new stream
envoy_1         | [2020-01-10 16:29:21.082][14][debug][http] [source/common/http/conn_manager_impl.cc:619] [C5][S6425470214323392008] request headers complete (end_stream=true):
envoy_1         | ':authority', 'localhost:9002'
envoy_1         | ':path', '/say'
envoy_1         | ':method', 'GET'
envoy_1         | 'user-agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:71.0) Gecko/20100101 Firefox/71.0'
envoy_1         | 'accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'
envoy_1         | 'accept-language', 'en-US,en;q=0.5'
envoy_1         | 'accept-encoding', 'gzip, deflate'
envoy_1         | 'dnt', '1'
envoy_1         | 'connection', 'keep-alive'
envoy_1         | 'cookie', 'oidc-service-token=supersecrettoken'
envoy_1         | 'upgrade-insecure-requests', '1'
envoy_1         | 'x-svc-cluster', 'sayer'

and

envoy_1         | [2020-01-10 16:29:21.083][14][debug][router] [source/common/router/router.cc:514] [C5][S6425470214323392008] router decoding headers:
envoy_1         | ':authority', 'localhost:9002'
envoy_1         | ':path', '/say'
envoy_1         | ':method', 'GET'
envoy_1         | ':scheme', 'http'
envoy_1         | 'user-agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:71.0) Gecko/20100101 Firefox/71.0'
envoy_1         | 'accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'
envoy_1         | 'accept-language', 'en-US,en;q=0.5'
envoy_1         | 'accept-encoding', 'gzip, deflate'
envoy_1         | 'dnt', '1'
envoy_1         | 'cookie', 'oidc-service-token=supersecrettoken'
envoy_1         | 'upgrade-insecure-requests', '1'
envoy_1         | 'x-svc-cluster', 'sayer'
envoy_1         | 'x-forwarded-for', '172.22.0.1'
envoy_1         | 'x-forwarded-proto', 'http'
envoy_1         | 'x-envoy-internal', 'true'
envoy_1         | 'x-request-id', '635a3677-6c36-404e-9421-8c7a1f37304e'
envoy_1         | 'x-envoy-expected-rq-timeout-ms', '20000'

Normally we don't run our production envoy's with debug level, however there are instances where we will temporarily set the level to debug when troubleshooting issues etc.

I'm wondering if there is some way to exclude particular headers from being logged in debug level? The only documentation I can seem to find for configuring logging is for access logs, not envoy's application logs.

Thanks!

[mattklein123]

Thanks for raising @nulltrope. In general today we assume that debug/trace logs carry sensitive/trusted information and are primarily used for development, though I think it's reasonable to (optionally) do better here. Marking this as needing a design proposal.

[pzmi]

Hi. I was about to report a similar issue.
I was pointed to a related issue https://github.com/envoyproxy/envoy/issues/4757 and a PR https://github.com/envoyproxy/envoy/pull/9315. However these address only logging proto messages.

For sure we need a possibility of specifying sensitive headers through some configuration as there are a variety of different deployment scenarios and some of them find the full authorization logging useful.
Secondly it would be helpful to use a common headers (or any other sensitive information) logging interface. Bonus points for a generalized log sanitization mechanism through some listeners or visitors. Maybe there's a way of utilizing the proto redaction functionality mentioned earlier.
Also we need a way of enforcing usage of provided tooling as anyone could log sensitive information plain text.

Also another issue regarding sanitization of config_dump https://github.com/envoyproxy/envoy/pull/7365