Contour: Get upstream_connection_options configurable to do workaround for the flaky behavior of envoy

[tapih]

I deployed a Golang HTTP client that makes HTTP requests periodically, about once every 10 seconds, to L7 endpoints which are exposed with envoy. This is done to check if the endpoints are alive as end-to-end testing.

While running it, I experienced the same error as reported in the issue about envoy.
This comment says that the error has something to do with keepalive, and this comment says that this issue is actually solved by configuring the tcp_keepalive under upstream_connection_options.

I think it's better to get the upstream_connection_options fields configurable because some developers seem to have faced this issue.
Any thoughts on this?

[youngnick]

In general, we try not to just lift and copy Envoy structs into HTTPProxy, for three reasons:

• Envoy sometimes changes the way fields or structures in the protobufs work, and we want to avoid needing to track that
• Ideally, we want to make it so that you don't need to know Envoy to use Contour
• Sometimes Envoy fields can have unexpected interactions in the reverse proxy case (as opposed to the mesh proxy case that is the default usecase for Envoy), and we want to try and make it hard for these unexpected interactions to trip you up. The best example of this for me is #1493, where HTTP/2 connection reuse and the way we were using the HTTPConnectionManager combined to make an issue where you could TLS handshake with one domain, and then get routed with another, which was a problem for certificate authentication. This is not an issue when you are running Envoy as a service proxy sidecar, as you simple aren't using the proxy in the same way.

That said, we already set TCP keepalives for the listener itself (see #2633 and #2652 for greater configurability of it), but we don't currently for the upstream.

I think that adding TCP keepalive configurability to the upstreams seems to make sense, and we could probably reuse the same keepalive configuration struct for the listener as well, which would help with #2652.