Rancher: [cli] Rancher CLI disconnects after 60 seconds

[martinbutt]

Rancher Versions:
Server: v1.4.1
healthcheck: v0.2.3
ipsec: v0.8.7
network-services: metadata:v0.7.2 / network-manager:v0.4.5
scheduler: v0.6.3

Docker Version:
1.12.6, build 78d1802

OS and where are the hosts located? (cloud, bare metal, etc):
Linux version 4.4.0-63-generic (buildd@lcy01-31) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.4) ) #84-Ubuntu SMP Wed Feb 1 17:20:32 UTC 2017

Setup Details: (single node rancher vs. HA rancher, internal DB vs. external DB)
Single node, external DB

Environment Type: (Cattle/Kubernetes/Swarm/Mesos)
Cattle

Steps to Reproduce:
When using Rancher CLI to connect to a container, an inactive connection is dropped after around a minute.

e.g. rancher exec -ti my-stack/my-container bash

Results:
If I leave this connection, it will drop after 60 seconds. If I do "watch ls", it will stay open. This is particularly problematic if I use the Rancher CLI to exectue a long running script that doesn't have any output for a minute.

I can replicate this with v0.5.0, v0.5.1 and v0.6.0-rc2 of the cli.

My current thought is that this is related to HAProxy dropping the connection, but can't see anything that would cause that.

HAProxy config:

       maxconn   150000 # Total Max Connections. This is dependent on ulimit
       user nobody
       group nogroup
       daemon
       pidfile /var/run/haproxy.pid
       stats socket /var/lib/haproxy_stats.sock
       nosplice
       tune.ssl.default-dh-param   2048

defaults

   mode            http
   retries         3
   maxconn         150000

   timeout connect 5000ms
   timeout client  120000ms
   timeout server  50000ms
   timeout queue   25000ms

   option         clitcpka
   option      dontlognull
   option      redispatch

Has anyone experienced anything similar?

[cpoole]

this is for sure an issue with HA proxy dropping your TCP websockets.

try setting
option http-keep-alive
timeout http-keep-alive 600s

[marksullivancrowd]

thanks @cpoole

timeout tunnel 1h

also fixes it
if you dont have keepalive defined on the haproxy side
think HTTP1.1 should have keepalive enabled by default

https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#4-timeout%20tunnel

"In HTTP, this timeout is used when a connection is upgraded (eg:
when switching to the WebSocket protocol, or forwarding a CONNECT request
to a proxy), or after the first response when no keepalive/close option is
specified."

[mwelch-ptc]

Apologies for re-entering an almost year old ticket, but where does one place the settings described:

option http-keep-alive
timeout http-keep-alive 600s