Contour: Unable to change connect_timeout in Envoy config

[krisdock]

What steps did you take and what happened:
Running into the Envoy issue reported here

Based on this comment, I'd like to update the connect_timeout setting in the envoy.json config file. However, due to the way Contour bootstraps Envoy, this isn't easily configurable.

There is another issue open pertaining to Envoy defaults here that would have the default increased to 15 seconds from 250ms.

There's also another issue pertaining to the configurability of these knobs here.

Environment:

• Contour version: 1.1.0
• Kubernetes version: (use kubectl version): 1.17.2
• Kubernetes installer & version: kubeadm
• Cloud provider or hardware configuration: GCP
• OS (e.g. from /etc/os-release): Ubuntu 18.04

[youngnick]

I'm sorry about #1375, as part of implementing those changes, I looked into the timeouts, and the connect_timeout controls the setting from Envoy to the backend service. Because of this, we left it at the default of 250ms, since we felt that there should not be more than than delay inside the cluster.

Could you explain more what you need? I'll update the original ticket as well.

[krisdock]

just chatted with @youngnick via slack. we've determined the best path forward is to expose the connect_timeout setting as a configurable knob (with clearer naming) rather than changing the default setting. nick is going to look into timing to determine when this may land.

[youngnick]

I think the setting should go into the config file and be called backend_connect_timeout. It also needs explanation that it's used for passive healthchecking, so it shouldn't be set too long.

[youngnick]

Also related to #2045 - ideally we can put documentation godoc-style and have it end up in the config file documentation somehow.

[davecheney]

15 seconds is too long, I'm worried if we allow users to make settings like this we have created a support footgun which allows people to continue to raise their timeouts in _unhealthy_ clusters, potentially masking more serious issues.

Can we compromise and raise the default for everyone? I think 1000ms, 1200ms tops is the most that is reasonably supportable noting that this is the TCP handshake time, not the time it takes for the higher layers to send the initial request, its literally a timer around socket(2) && bind(2)

[youngnick]

I'm okay with that.

[jpeach]

xref #2225

[michmike]

@davecheney while i agree we should change the default to a more sensible number, i don't agree that we should not expose this config knob in contour. the tcp handshake can take longer than 1.2seconds, right? (is the timeout in question here the same one referenced in this article and set to 15 seconds? https://www.envoyproxy.io/docs/envoy/latest/configuration/best_practices/edge.html?highlight=connect_timeout)

@krisdock i didn't see any details above as to why this failed in this specific customer environemt. are there general networking delays? does this fail every time?

[davecheney]

@michmike to clarify, in this class of issues I'm not saying "we will not add this option", instead I offer "is the current default wrong?". if it is, we'll fix the default and all contour users benefit. If the default is reasonable, then we'll look into the underlying issues that are driving the customer to need a higher connect timeout. It might be their application or networking is overloaded and they need a longer timeout, or it could be that their application or networking is overloaded and that is the root problem and that increasing timeouts just leads to further bottlenecks.

At the moment I don't have enough information to tell which case this is.

[michmike]

@davecheney i agree 100%, which is why i asked for more details into this specific customer environment.

[krisdock]

@krisdock i didn't see any details above as to why this failed in this specific customer environemt. are there general networking delays? does this fail every time?

@michmike yes, they're running into a scenario where the handshake appears to be taking longer than the current timeout. it fails every time.

[michmike]

@youngnick what do you think about changing the default to 1200ms and crafting a build of contour that @krisdock can give this user and see if that fixes the issue or not. that would be a good datapoint to have.

[youngnick]

I think we need some more data about how long the connection to the backend takes, and what the connection actually involves. Even 1200ms is purely a guess on connection timings.

I'd like to know:

• what type of service is the Envoy connecting to? Is it HTTP or HTTPS?
• What type of CNI is in use in the cluster?
• lastly, what are the actual timings for connection? I was able to get this number in one of my envoy pods with the following sequence of commands after execing into the pod:

# Exec into an Envoy pod
export CONTOUR_NAMESPACE=projectcontour
kubectl exec -n $CONTOUR_NAMESPACE -it `kubectl get pod -lapp=envoy -n $CONTOUR_NAMESPACE -o name | head -1` bash
apt update
apt install curl
curl -o /dev/null -s -w "%{time_connect} + %{time_starttransfer} = %{time_total}\n" http://*svcname*.*namespace*

That will output the time taken to connect to the service in seconds. Obviously, if it is a HTTPS service, you will need to add -k to the curl command, and use https://.

In my cluster, I get:

root@envoy-25lz4:/# curl -o /dev/null -s -w "%{time_connect}\n" http://httpbin.default
0.006

There's no point setting a new default until we know the value it should be set to.

[krisdock]

@youngnick i've taken these questions to the user.

[davecheney]

@krisdock any updates?

[krisdock]

@davecheney nothing yet. just sent another ping.

[krisdock]

@youngnick

What type of service is the Envoy connecting to? Is it HTTP or HTTPS?

type: ClusterIP
It is Grafana.
HTTPS

What type of CNI is in use in the cluster?

We use weave.

Lastly, what are the actual timings for connection?

curl -o /dev/null -s -k -w "%{time_connect} + %{time_starttransfer} = %{time_total}n" https://grafana.monitoring.cluster.local
0.000 + 0.000 = 0.012
0.000 + 0.000 = 0.039
0.000 + 0.000 = 0.020
0.000 + 0.000 = 0.048

[davecheney]

@krisdock thanks for confirming. This result indicates that connect timeout is _not_ the issue.

[youngnick]

Yes, those results indicate that the connect is taking 12-48ms, well beneath the 250ms default.

[davecheney]

@youngnick did you mean the total time to connect, send the request, stream the response and close the connection is between 12-28ms? The curl results suggest that connect time and the time before starting the transfer (https handshake) are below 1ms

[youngnick]

You are one hundred percent right, yes.

[youngnick]

@krisdock How do you want to proceed with this? I don't think that the connect_timeout parameter tuning will help in this case, so I'm unsure how to help.

[krisdock]

@youngnick i'll get with the user and determine how to move forward.

[michmike]

closing this issue until we get more data. if this ends up being an issue that can be solved by modifying the configuration of Contour (or fixing a bug in Contour), we can reopen and revisit this. thank you!

[paulcreasey]

the connect_timeout controls the setting from Envoy to the backend service. Because of this, we left it at the default of 250ms, since we felt that there should not be more than than delay inside the cluster.

@youngnick @michmike I believe the connect_timeout option is also used when proxying to ExternalName's in which case the assumption that 250ms is sufficient isn't right.

I have been experiencing an issue proxying to an external name in another region (from a european k8s cluster to external service hosted in asia), the connection time as report by curl's time_connect is ~320ms.

I see there are a number of timeout config related issues, but i thought i'd comment here since this is the most specific.

[johanbrandhorst]

With the release of 1.9 and the ability to configure External Authorization services the default, unconfigurable timeout of 0.25s is too short. I strongly encourage the team to expose this knob for users that have an external authorization service implementation that has further downstream dependencies (such as if they're using it to do OAuth).

In the meantime, what is the best workaround to change this? Write my own xDS client?

[jpeach]

Configuring the connect timeout sounds totally reasonable to me.

There's 2 places:

• Global connect timeout in the config file
• per-route timeout for HTTPProxy in TimeoutPolicy

[johanbrandhorst]

With the release of 1.9 and the ability to configure External Authorization services the default, unconfigurable timeout of 0.25s is too short. I strongly encourage the team to expose this knob for users that have an external authorization service implementation that has further downstream dependencies (such as if they're using it to do OAuth).

In the meantime, what is the best workaround to change this? Write my own xDS client?

@jpeach pointed me in the direction of the responseTimeout in the docs which solved my issue, so I no longer need connect_timeout to be configurable. Thanks!

[erwbgy]

@jpeach Is making connect timeout configurable in TimeoutPolicy still planned? Or is there another way to configure this? As per @paulcreasey more time is required when using ExternalName Services outside of the current cluster.

[jpeach]

@jpeach Is making connect timeout configurable in TimeoutPolicy still planned? Or is there another way to configure this? As per @paulcreasey more time is required when using ExternalName Services outside of the current cluster.

AFAIK no-one is working on this. I'd be happy to help anyone who wants to though.

[youngnick]

I think this is okay to add, but it needs documentation about how this really should only be used for ExternalName services. I am worried about side effects of making this timeout too long - I suspect that you could end up with weird problems that would be very hard to find if you did.

In any Kubernetes cluster, if connections between nodes are taking longer than 250ms, something is not right, and papering over that with Contour is not going to help you in the long run.

[HatBat]

We are also having a lot of issues in our product due to the hard-coded 250ms connect_timeout. We get the issue when a new upstream pod starts receiving a lot of traffic for the first time. Then for maybe a second or two connections will not always be able to complete in under 250ms. After that it's fine but we get problems from those failed connections. This is HTTPS and we have seen connections take up to 900ms to complete including the handshakes. We looked at tcpdumps in wireshark.

It would be great if you could expose the connect_timeout parameter so we could change it to any value. But if not then could you at least increase the default value from 250ms to 5 seconds? I say 5s because that is the new default value that Envoy will be using for connect_timeout from version 1.19:
"cluster: added default value of 5 seconds for connect_timeout"

I think it makes little sense for Contour to keep the default at 250ms when Envoy increases it to 5s. Here it was decided to keep connect_timeout at 250ms since that was the default for Envoy. But now it seems that the new default will be 5s so I think it makes sense for Contour to also default to this value.

[youngnick]

Thanks for that info, @HatBat. If Envoy is changing the default connect_timeout, then I guess ~we can change our default too. Obviously the effect is that it will take up to 5s to realise than an endpoint is down if you're using passive health checking, but I guess in most cases, that's probably not an issue.~

Edit: I spent some time reading the Envoy docs and chatting with @wrowe, and the only concern we could come up with is that too high a connect_timeout will mean that, in the event the pod disappears, you could wait up to connect_timeout for it to be marked unhealthy. So I'm reluctant to set the default even at 5sec for our use cases. I propose this:

• I'll do a PR to change the default to 2sec. This is pretty straightforward.
• I'll do a followup PR to add a global timeout setting to the Contour config file. This will let users change the value on a Contour-wide basis, which should hopefully be sufficient for most users. This is a more complex change, as we don't currently have any mechanism to configure cluster-level timeouts. So there's quite a bit of wiring to get a setting from the config file down to the Envoy config generator. I won't add per-cluster values of this setting unless we see some demand for that, as it adds another layer of complexity.

[HatBat]

@youngnick, This sounds like a good proposal that would cover the needs of us and most Contour users I think.