Contour: configurable default circuit breaking

[phylake]

Contour allows circuit breaking fields such as max connections to be configured via annotations. If unset then the Envoy defaults are used. These defaults are much too low, and were one of the callouts at EnvoyCon with regard to sensible defaults in Envoy.

Providing a way to increase these for all IngressRoutes would be ideal. As operators of Contour we'd like to provide some more sensible defaults rather than go out to dozens of service teams and ask them to provide the annotations.

[stevesloka]

I think this could be good to be added to the new configuration file we're adding to Contour. I think allowing admins to set sane defaults for their cluster, then allowing users to further customize (or not maybe?) would be beneficial.

[stevesloka]

Just to confirm, we'll add the following to the new config file which will set a default on each resource unless overridden:

• MaxConnections
• MaxPendingRequests
• MaxRequests
• MaxRetries

[stevesloka]

Hey @phylake, if we were to set reasonable defaults for Contour, what would you set them to (i.e. not having them be configurable)?

[phylake]

We tripped these circuits at very low load so we increased them to 1,000,000 for MaxConnections and MaxRequests just to rule them out as a source of 503s we're seeing. MaxPendingRequests and MaxRetries seemed fine at 1024.

It's hard to say what reasonable is because these settings should really be fine-tuned per service. That said we're well past the C10k problem so 1024 seems obviously too low. I'd recommend doing what we did so Contour doesn't cause _problems_ by default (i.e. leaving Envoy defaults at levels well under what even antiquated architectures can handle). Services can then opt in to circuit breaking at levels that are reasonable for them.