Envoy: TCP connection pool have a mode where unassigned connections are closed (for TCP proxy)

Created on 12 Sep 2018 · 17Comments · Source: envoyproxy/envoy

Envoy version used are

envoy version: d2fa5f3b4c1f09a8e6fa96ecc8f64555d67a85f6/1.8.0-dev/Clean/RELEASE
envoy version: d2fa5f3b4c1f09a8e6fa96ecc8f64555d67a85f6/1.8.0-dev/Clean/RELEASE

Bug Template

We have noticed that Envoy doesn't always reuse connections from the connection pool and sometimes will open new connections unnecessarily.

Description

We discussed this bug in slack where we were asked to open a github issue.

Repro steps:

Use the following config and have a dummy process listening on port 8080 then run nc -vz localhost 61001 repeatedly. The repro steps are very similar to those in https://github.com/envoyproxy/envoy/issues/4310

Config:

config.yaml

admin:
  access_log_path: /dev/null
  address:
    socket_address:
      address: 127.0.0.1
      port_value: 61003
static_resources:
  clusters:
  - name: 0-service-cluster
    connect_timeout: 0.25s
    type: STATIC
    lb_policy: ROUND_ROBIN
    hosts:
    - socket_address:
        address: 10.255.98.48
        port_value: 8080
    circuit_breakers:
      thresholds:
      - max_connections: 4294967295
  - name: 1-service-cluster
    connect_timeout: 0.25s
    type: STATIC
    lb_policy: ROUND_ROBIN
    hosts:
    - socket_address:
        address: 10.255.98.48
        port_value: 2222
    circuit_breakers:
      thresholds:
      - max_connections: 4294967295
dynamic_resources:
  lds_config:
    path: /etc/cf-assets/envoy_config/listeners.yaml
version_info: "0"
resources:
- '@type': type.googleapis.com/envoy.api.v2.Listener
  name: listener-8080
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 61001
  filter_chains:
  - filters:
    - name: envoy.tcp_proxy
      config:
        stat_prefix: 0-stats
        cluster: 0-service-cluster
    tls_context:
      common_tls_context:
        tls_certificates:
        - certificate_chain:
            inline_string: |
              -----BEGIN CERTIFICATE-----
              MIID9zCCAt+gAwIBAgIRANDiECliiEH6ZkRVhOwFLSgwDQYJKoZIhvcNAQELBQAw
              QzEMMAoGA1UEBhMDVVNBMRYwFAYDVQQKEw1DbG91ZCBGb3VuZHJ5MRswGQYDVQQD
              ExJpbnN0YW5jZUlkZW50aXR5Q0EwHhcNMTgwOTEyMTM1NTQzWhcNMTgwOTEzMTM1
              NTQzWjCByDGBnjA4BgNVBAsTMW9yZ2FuaXphdGlvbjo5MjFjMDhiNi03NzEzLTQy
              OWYtYWRjOC1mZTUxNjU4YzQxYTAwMQYDVQQLEypzcGFjZTo4MDg1MjdhNC05NTk0
              LTQ2MmEtOTM5My0zZTVhZmQyODgwZmUwLwYDVQQLEyhhcHA6NjE3NzZjY2QtYTg3
              Yy00OWIyLTllZjEtOTM3OWFlOGNmZjQzMSUwIwYDVQQDExw0MjFjMmQ3NS04NDFh
              LTRmNGYtNWUzNS04NWNmMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA
              wEOwPkrYy4/2aI7K130EQxsMY6skLxiKt4gV7fAjDAdVc5zEsAqDn96JPgaJYDSh
              rTTXgnYC8ldD6hNfYR1Bp92xKE1mp09q2WcEnYIXUbHCIMR+Atd2ug3IFd9TrN73
              iZR2iRQHNvplh+q47KsWzX1ObGPdZ5PdzYe2Iz+HOEsypEj8S4L7bS8rdETUzvuH
              EQBDuCs6J9PtLqwukHDrqEEc2ETuepUGer+n2kRVSmHAHB/C9jOZeAOBOcSml2uN
              HEkQKeaOYueHocQGbO1Jg1L6j1MbGp3wXdQ8pyLTC6K8grvbLeVhyazPOoDMS7rL
              domLspbvaOgaXUuGxUVrcQIDAQABo2AwXjAOBgNVHQ8BAf8EBAMCA6gwHQYDVR0l
              BBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMBMC0GA1UdEQQmMCSCHDQyMWMyZDc1LTg0
              MWEtNGY0Zi01ZTM1LTg1Y2aHBAr/YjAwDQYJKoZIhvcNAQELBQADggEBAAYtM1RI
              +97uMJcO2H5whDm2P6egOagVF5+l2B25M3jFFql+HC+P1jc9WLt+xDyx5e2NoSYS
              c9ddL6vVImyomZL0+rkBzvhJkdyGMYs9yxWk8fMxXvuPwO2b5UhXPqH7yXWjeKQm
              bo17ppNMkcC6RAXhQ0rq2L6gT+8JLM7YeLVqYB6K0ZISrvvKFidHYs+MR2Hublw6
              UVvp0Rh63b9vMvBhUYknJw2WiaBTOdp/UC7pqsAlSL7OQLV6qaDW3fOOk18oX3F4
              Pl4cRMdDO8gba2ZECCnRdoy6NFdf7+Bm4D3ahh/iGl5DSz/TWbTqFVOzNL1cbEXY
              Ci+ZKVxNDiQQOw4=
              -----END CERTIFICATE-----
              -----BEGIN CERTIFICATE-----
              MIIDKzCCAhOgAwIBAgIRAKQeshAylYEty9gB15bJdNkwDQYJKoZIhvcNAQELBQAw
              OjEMMAoGA1UEBhMDVVNBMRYwFAYDVQQKEw1DbG91ZCBGb3VuZHJ5MRIwEAYDVQQD
              EwlhcHBSb290Q0EwHhcNMTgwOTA2MTQyMjEwWhcNMTkwOTA2MTQyMjEwWjBDMQww
              CgYDVQQGEwNVU0ExFjAUBgNVBAoTDUNsb3VkIEZvdW5kcnkxGzAZBgNVBAMTEmlu
              c3RhbmNlSWRlbnRpdHlDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
              AMYiQBhaW4Cw9FR8qLyN4GgF5V2VUn8B9yIJ3gCsJiObypvIQNDbfTUmnIPgdj3I
              3IVYoOfTAHoeinp7/S+vfY2Q4ad7Yi+PVooTPJSYu+IKOtq9uABO8duwFjQppuaI
              Z6jmZ8AOAc95Ks3+mLqV9W1IxNCqMTgP1iFbIY1fb8oRh1rvysfU7nzHB7cRWvtk
              fHqu7cHITN2Ns2AqeapWSgtAh35AajifOkeDG0XlXbSZALaA4su5ObUBsVaHy8J2
              ZlI4h9qiWSyYB0U9bXzGG8RfneiMr+uVYTRjbQ7cubxJ3ccgYObQYU5dk1lDldkY
              IrijUiwKRfyZnD3tRN3v/3sCAwEAAaMjMCEwDgYDVR0PAQH/BAQDAgEGMA8GA1Ud
              EwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAHbubKOk9auYuQ28uWzQOdDf
              lwV501SK8Mz0Rqq9akp0a3sj1FAxrW6mnW7NBlVkDAT1D3Nm7+F4zfbO2ee37dPM
              EXFLpUFf4tpaxWxEtBEjal3LGbDGuZKT12yjnhTP9hajdd3H/szov+Z9EztX450/
              8BdqIISbWGDDcOWaL+QkpZNI4A5z9MVobw664IJ+cz7nvIzKb6ERh2UZg2WIwkXf
              BG0zDgRVc7pLwur/7E5VxttK315R0A5gAivBG4qdfKPWKZxyabmrqsxVvLg9ZJMJ
              zPB30NFnC9snppeZuE9aHk9m0NM3F8qux3gFtvVemXihlEzrkIP7LsxZDbKQCcE=
              -----END CERTIFICATE-----
          private_key:
            inline_string: |
              -----BEGIN RSA PRIVATE KEY-----
              MIIEogIBAAKCAQEAwEOwPkrYy4/2aI7K130EQxsMY6skLxiKt4gV7fAjDAdVc5zE
              sAqDn96JPgaJYDShrTTXgnYC8ldD6hNfYR1Bp92xKE1mp09q2WcEnYIXUbHCIMR+
              Atd2ug3IFd9TrN73iZR2iRQHNvplh+q47KsWzX1ObGPdZ5PdzYe2Iz+HOEsypEj8
              S4L7bS8rdETUzvuHEQBDuCs6J9PtLqwukHDrqEEc2ETuepUGer+n2kRVSmHAHB/C
              9jOZeAOBOcSml2uNHEkQKeaOYueHocQGbO1Jg1L6j1MbGp3wXdQ8pyLTC6K8grvb
              LeVhyazPOoDMS7rLdomLspbvaOgaXUuGxUVrcQIDAQABAoIBADLg9r/VUL+1zjBn
              MTiLRdh4TPQHMyEfbn4ZDiyUMsIZIrzgjJBFiAWDf3AhjLMf2nUamblVf3wgKcHE
              9KWfd2Nz4FDZAsRd2NnFUExaQ8+Dqeq/9NTNrgaYZQ2ctuj1bC5FUoT9/tbGtp2c
              OOmwcO1aLqmyV8H0OGsOcd2sDx2PGl8v2ZnwSDFWodeZCYDYQlWeSiPYR0nc8q9L
              KcggoDBpVtjZtyh+NTHkdKtL/H/eFrDY0NRYTZchmy7N+d1rfAeh8RPEa9S/gAu9
              KtmJPpR1w9HSCI5XYIML0RQedSzpgJ8eKaalfmyfKNpU/Bqqv1ZU55gUo82GFQD6
              kFfwho0CgYEA/OEoPl+AYv/e9iunSELfx10tyYGKaWhNuLkiqLTC+6LmlC/xjrim
              SPB/bYRuvsL1MPvrRbmtjlsGZ669Uwo2O03DvERQDiRGk1SH1XvjE+ufARgrlRoQ
              3Gqst2uOJRWBbRHTJrrl8c/5DgZXfSgY8VWkDzVFJ8E2Rc8p/VJyVusCgYEAwqMM
              iYYDa4teidTNat1MIfuj1hGwoyJy7wjarYB63+a/DWpG4tjOhZCacFWtUrCKHfvT
              ksFt/6rD4aqoul7xMN+KaOfxOl0TZj0Cd4w8Uju43vm/Qs33VGtsEZTNl/ZMGDYE
              C7CE/m6djBJYp+6GO/LPHBXuEjIYcONWfp8M6BMCgYAsrqwpuT46P/Ppk3UPc6GO
              3IwGmvkV/vQcu7nADZCWXgZYENCh40aehMA/XbMjydcBPMIpo9Fi4fqTv3B8h8Vl
              XSxEy91JYmeGCN2nfyh0kKGb0+V/hweLqt2x2zNWoiIoX4nrwgJ3Y8QiEkTLPJaL
              rhI7DbKWf7FTTHOi6zqA2QKBgDcb62TU6Qr+PxpPgooLdo74rP7jq1vFWT2NCD2v
              7xqOPvIRcVixgmgW9WNxHs4W5kRvFQysb/S/6Q5wP63lS+1Gne3uHkMkL2X3SEeJ
              EVtHRaT8mcbUgn8Im2Su+TdZqVoBkUuuAzn3H+yVFT0XMDKxISsWmn3UaU8tT0Qy
              zHi5AoGAWZKRdffPUZ+G5TK3OtWKr3Xtee9+q+Da5vAPh9TgpyU4pPRS8FBwJVao
              FYkxX7iHIYcabpypcIfapu50MggYzxZcKihqeunWqTvx+FTSf84rkIW6a4IJ3i51
              RzrFNSoKUBth73X26/yHeuAGRttKGV7W2rVaGgly+XDfyyty1IY=
              -----END RSA PRIVATE KEY-----
        tls_params:
          cipher_suites: '[ECDHE-RSA-AES256-GCM-SHA384|ECDHE-RSA-AES128-GCM-SHA256]'
        validation_context:
          trusted_ca:
            inline_string: |
              -----BEGIN CERTIFICATE-----
              MIIDJDCCAgygAwIBAgIRAKh/OvG8JBRVc4nvKhSgJ9EwDQYJKoZIhvcNAQELBQAw
              OzEMMAoGA1UEBhMDVVNBMRYwFAYDVQQKEw1DbG91ZCBGb3VuZHJ5MRMwEQYDVQQD
              EwppbnRlcm5hbENBMB4XDTE4MDkwNTE2MDQwMFoXDTE5MDkwNTE2MDQwMFowOzEM
              MAoGA1UEBhMDVVNBMRYwFAYDVQQKEw1DbG91ZCBGb3VuZHJ5MRMwEQYDVQQDEwpp
              bnRlcm5hbENBMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtCgftcWj
              Pr7TkxnH3+YEh81wU9um5bzjKvnOI3pIoR7TE2nEocOn+7ktGu+tzBKiyeI5Nl6+
              ldRtIARusZKlu/r4ZLYmpvYY2+wrM3T5d9jgQafydLqpbNuoF4BxeVzC6a5w0J3T
              bkqqYMXdIb6BURPJPT2Ockc5e1OJYUAm+3vx4f8ffABWsujFKBSHOBq54DBSNgKJ
              ki93vvvB62/1V75szP5T9hwGL5N+TSvwSISGRjoQfE+AUDjZOvNx6HPYU+GRPBmB
              cfEDwxdTk97ga/vrovYJAj6goxAW90pmLAikcLqfIJKMWVn7LqxVFxJ+y6NnMRzA
              iUW8QRv1U2m16QIDAQABoyMwITAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUw
              AwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAQvsUvfA8gBbMIrWENc66GlCMjwNgXyq2
              sJDi1SIc7IZvOgouXCSQiTLOMRXNjwwE/hf7Hz4AtMS1gkdtAhZPyM+9gRMkXN6d
              XI2GpfFTVVp8fdizd4FRvNnQfPYSBQ/li6XNtyvAJ6ZnKhzHkqfgW8Sz4NYU3F2t
              NoP7Hx8xjj06MyoduwjHU6k/hLKrjnUEa6O0Hw3jzE4Uk1modER4aoevRHB2kSTP
              KW3twTZd+lr8+Tyo7qERLGHAQ1xBTaMCXyYrbITXu8ODgpOioVel6pgVudJOsNiN
              +iUD1rdO+F52RFFplrmIJkmPcRlW4frhsK+sXpROp9yuUPLPHOE8OQ==
              -----END CERTIFICATE-----
          verify_subject_alt_name:
          - gorouter.service.cf.internal
      require_client_certificate: true
- '@type': type.googleapis.com/envoy.api.v2.Listener
  name: listener-2222
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 61002
  filter_chains:
  - filters:
    - name: envoy.tcp_proxy
      config:
        stat_prefix: 1-stats
        cluster: 1-service-cluster
    tls_context:
      common_tls_context:
        tls_certificates:
        - certificate_chain:
            inline_string: |
              -----BEGIN CERTIFICATE-----
              MIID9zCCAt+gAwIBAgIRANDiECliiEH6ZkRVhOwFLSgwDQYJKoZIhvcNAQELBQAw
              QzEMMAoGA1UEBhMDVVNBMRYwFAYDVQQKEw1DbG91ZCBGb3VuZHJ5MRswGQYDVQQD
              ExJpbnN0YW5jZUlkZW50aXR5Q0EwHhcNMTgwOTEyMTM1NTQzWhcNMTgwOTEzMTM1
              NTQzWjCByDGBnjA4BgNVBAsTMW9yZ2FuaXphdGlvbjo5MjFjMDhiNi03NzEzLTQy
              OWYtYWRjOC1mZTUxNjU4YzQxYTAwMQYDVQQLEypzcGFjZTo4MDg1MjdhNC05NTk0
              LTQ2MmEtOTM5My0zZTVhZmQyODgwZmUwLwYDVQQLEyhhcHA6NjE3NzZjY2QtYTg3
              Yy00OWIyLTllZjEtOTM3OWFlOGNmZjQzMSUwIwYDVQQDExw0MjFjMmQ3NS04NDFh
              LTRmNGYtNWUzNS04NWNmMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA
              wEOwPkrYy4/2aI7K130EQxsMY6skLxiKt4gV7fAjDAdVc5zEsAqDn96JPgaJYDSh
              rTTXgnYC8ldD6hNfYR1Bp92xKE1mp09q2WcEnYIXUbHCIMR+Atd2ug3IFd9TrN73
              iZR2iRQHNvplh+q47KsWzX1ObGPdZ5PdzYe2Iz+HOEsypEj8S4L7bS8rdETUzvuH
              EQBDuCs6J9PtLqwukHDrqEEc2ETuepUGer+n2kRVSmHAHB/C9jOZeAOBOcSml2uN
              HEkQKeaOYueHocQGbO1Jg1L6j1MbGp3wXdQ8pyLTC6K8grvbLeVhyazPOoDMS7rL
              domLspbvaOgaXUuGxUVrcQIDAQABo2AwXjAOBgNVHQ8BAf8EBAMCA6gwHQYDVR0l
              BBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMBMC0GA1UdEQQmMCSCHDQyMWMyZDc1LTg0
              MWEtNGY0Zi01ZTM1LTg1Y2aHBAr/YjAwDQYJKoZIhvcNAQELBQADggEBAAYtM1RI
              +97uMJcO2H5whDm2P6egOagVF5+l2B25M3jFFql+HC+P1jc9WLt+xDyx5e2NoSYS
              c9ddL6vVImyomZL0+rkBzvhJkdyGMYs9yxWk8fMxXvuPwO2b5UhXPqH7yXWjeKQm
              bo17ppNMkcC6RAXhQ0rq2L6gT+8JLM7YeLVqYB6K0ZISrvvKFidHYs+MR2Hublw6
              UVvp0Rh63b9vMvBhUYknJw2WiaBTOdp/UC7pqsAlSL7OQLV6qaDW3fOOk18oX3F4
              Pl4cRMdDO8gba2ZECCnRdoy6NFdf7+Bm4D3ahh/iGl5DSz/TWbTqFVOzNL1cbEXY
              Ci+ZKVxNDiQQOw4=
              -----END CERTIFICATE-----
              -----BEGIN CERTIFICATE-----
              MIIDKzCCAhOgAwIBAgIRAKQeshAylYEty9gB15bJdNkwDQYJKoZIhvcNAQELBQAw
              OjEMMAoGA1UEBhMDVVNBMRYwFAYDVQQKEw1DbG91ZCBGb3VuZHJ5MRIwEAYDVQQD
              EwlhcHBSb290Q0EwHhcNMTgwOTA2MTQyMjEwWhcNMTkwOTA2MTQyMjEwWjBDMQww
              CgYDVQQGEwNVU0ExFjAUBgNVBAoTDUNsb3VkIEZvdW5kcnkxGzAZBgNVBAMTEmlu
              c3RhbmNlSWRlbnRpdHlDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
              AMYiQBhaW4Cw9FR8qLyN4GgF5V2VUn8B9yIJ3gCsJiObypvIQNDbfTUmnIPgdj3I
              3IVYoOfTAHoeinp7/S+vfY2Q4ad7Yi+PVooTPJSYu+IKOtq9uABO8duwFjQppuaI
              Z6jmZ8AOAc95Ks3+mLqV9W1IxNCqMTgP1iFbIY1fb8oRh1rvysfU7nzHB7cRWvtk
              fHqu7cHITN2Ns2AqeapWSgtAh35AajifOkeDG0XlXbSZALaA4su5ObUBsVaHy8J2
              ZlI4h9qiWSyYB0U9bXzGG8RfneiMr+uVYTRjbQ7cubxJ3ccgYObQYU5dk1lDldkY
              IrijUiwKRfyZnD3tRN3v/3sCAwEAAaMjMCEwDgYDVR0PAQH/BAQDAgEGMA8GA1Ud
              EwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAHbubKOk9auYuQ28uWzQOdDf
              lwV501SK8Mz0Rqq9akp0a3sj1FAxrW6mnW7NBlVkDAT1D3Nm7+F4zfbO2ee37dPM
              EXFLpUFf4tpaxWxEtBEjal3LGbDGuZKT12yjnhTP9hajdd3H/szov+Z9EztX450/
              8BdqIISbWGDDcOWaL+QkpZNI4A5z9MVobw664IJ+cz7nvIzKb6ERh2UZg2WIwkXf
              BG0zDgRVc7pLwur/7E5VxttK315R0A5gAivBG4qdfKPWKZxyabmrqsxVvLg9ZJMJ
              zPB30NFnC9snppeZuE9aHk9m0NM3F8qux3gFtvVemXihlEzrkIP7LsxZDbKQCcE=
              -----END CERTIFICATE-----
          private_key:
            inline_string: |
              -----BEGIN RSA PRIVATE KEY-----
              MIIEogIBAAKCAQEAwEOwPkrYy4/2aI7K130EQxsMY6skLxiKt4gV7fAjDAdVc5zE
              sAqDn96JPgaJYDShrTTXgnYC8ldD6hNfYR1Bp92xKE1mp09q2WcEnYIXUbHCIMR+
              Atd2ug3IFd9TrN73iZR2iRQHNvplh+q47KsWzX1ObGPdZ5PdzYe2Iz+HOEsypEj8
              S4L7bS8rdETUzvuHEQBDuCs6J9PtLqwukHDrqEEc2ETuepUGer+n2kRVSmHAHB/C
              9jOZeAOBOcSml2uNHEkQKeaOYueHocQGbO1Jg1L6j1MbGp3wXdQ8pyLTC6K8grvb
              LeVhyazPOoDMS7rLdomLspbvaOgaXUuGxUVrcQIDAQABAoIBADLg9r/VUL+1zjBn
              MTiLRdh4TPQHMyEfbn4ZDiyUMsIZIrzgjJBFiAWDf3AhjLMf2nUamblVf3wgKcHE
              9KWfd2Nz4FDZAsRd2NnFUExaQ8+Dqeq/9NTNrgaYZQ2ctuj1bC5FUoT9/tbGtp2c
              OOmwcO1aLqmyV8H0OGsOcd2sDx2PGl8v2ZnwSDFWodeZCYDYQlWeSiPYR0nc8q9L
              KcggoDBpVtjZtyh+NTHkdKtL/H/eFrDY0NRYTZchmy7N+d1rfAeh8RPEa9S/gAu9
              KtmJPpR1w9HSCI5XYIML0RQedSzpgJ8eKaalfmyfKNpU/Bqqv1ZU55gUo82GFQD6
              kFfwho0CgYEA/OEoPl+AYv/e9iunSELfx10tyYGKaWhNuLkiqLTC+6LmlC/xjrim
              SPB/bYRuvsL1MPvrRbmtjlsGZ669Uwo2O03DvERQDiRGk1SH1XvjE+ufARgrlRoQ
              3Gqst2uOJRWBbRHTJrrl8c/5DgZXfSgY8VWkDzVFJ8E2Rc8p/VJyVusCgYEAwqMM
              iYYDa4teidTNat1MIfuj1hGwoyJy7wjarYB63+a/DWpG4tjOhZCacFWtUrCKHfvT
              ksFt/6rD4aqoul7xMN+KaOfxOl0TZj0Cd4w8Uju43vm/Qs33VGtsEZTNl/ZMGDYE
              C7CE/m6djBJYp+6GO/LPHBXuEjIYcONWfp8M6BMCgYAsrqwpuT46P/Ppk3UPc6GO
              3IwGmvkV/vQcu7nADZCWXgZYENCh40aehMA/XbMjydcBPMIpo9Fi4fqTv3B8h8Vl
              XSxEy91JYmeGCN2nfyh0kKGb0+V/hweLqt2x2zNWoiIoX4nrwgJ3Y8QiEkTLPJaL
              rhI7DbKWf7FTTHOi6zqA2QKBgDcb62TU6Qr+PxpPgooLdo74rP7jq1vFWT2NCD2v
              7xqOPvIRcVixgmgW9WNxHs4W5kRvFQysb/S/6Q5wP63lS+1Gne3uHkMkL2X3SEeJ
              EVtHRaT8mcbUgn8Im2Su+TdZqVoBkUuuAzn3H+yVFT0XMDKxISsWmn3UaU8tT0Qy
              zHi5AoGAWZKRdffPUZ+G5TK3OtWKr3Xtee9+q+Da5vAPh9TgpyU4pPRS8FBwJVao
              FYkxX7iHIYcabpypcIfapu50MggYzxZcKihqeunWqTvx+FTSf84rkIW6a4IJ3i51
              RzrFNSoKUBth73X26/yHeuAGRttKGV7W2rVaGgly+XDfyyty1IY=
              -----END RSA PRIVATE KEY-----
        tls_params:
          cipher_suites: '[ECDHE-RSA-AES256-GCM-SHA384|ECDHE-RSA-AES128-GCM-SHA256]'
        validation_context:
          trusted_ca:
            inline_string: |
              -----BEGIN CERTIFICATE-----
              MIIDJDCCAgygAwIBAgIRAKh/OvG8JBRVc4nvKhSgJ9EwDQYJKoZIhvcNAQELBQAw
              OzEMMAoGA1UEBhMDVVNBMRYwFAYDVQQKEw1DbG91ZCBGb3VuZHJ5MRMwEQYDVQQD
              EwppbnRlcm5hbENBMB4XDTE4MDkwNTE2MDQwMFoXDTE5MDkwNTE2MDQwMFowOzEM
              MAoGA1UEBhMDVVNBMRYwFAYDVQQKEw1DbG91ZCBGb3VuZHJ5MRMwEQYDVQQDEwpp
              bnRlcm5hbENBMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtCgftcWj
              Pr7TkxnH3+YEh81wU9um5bzjKvnOI3pIoR7TE2nEocOn+7ktGu+tzBKiyeI5Nl6+
              ldRtIARusZKlu/r4ZLYmpvYY2+wrM3T5d9jgQafydLqpbNuoF4BxeVzC6a5w0J3T
              bkqqYMXdIb6BURPJPT2Ockc5e1OJYUAm+3vx4f8ffABWsujFKBSHOBq54DBSNgKJ
              ki93vvvB62/1V75szP5T9hwGL5N+TSvwSISGRjoQfE+AUDjZOvNx6HPYU+GRPBmB
              cfEDwxdTk97ga/vrovYJAj6goxAW90pmLAikcLqfIJKMWVn7LqxVFxJ+y6NnMRzA
              iUW8QRv1U2m16QIDAQABoyMwITAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUw
              AwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAQvsUvfA8gBbMIrWENc66GlCMjwNgXyq2
              sJDi1SIc7IZvOgouXCSQiTLOMRXNjwwE/hf7Hz4AtMS1gkdtAhZPyM+9gRMkXN6d
              XI2GpfFTVVp8fdizd4FRvNnQfPYSBQ/li6XNtyvAJ6ZnKhzHkqfgW8Sz4NYU3F2t
              NoP7Hx8xjj06MyoduwjHU6k/hLKrjnUEa6O0Hw3jzE4Uk1modER4aoevRHB2kSTP
              KW3twTZd+lr8+Tyo7qERLGHAQ1xBTaMCXyYrbITXu8ODgpOioVel6pgVudJOsNiN
              +iUD1rdO+F52RFFplrmIJkmPcRlW4frhsK+sXpROp9yuUPLPHOE8OQ==
              -----END CERTIFICATE-----
          verify_subject_alt_name:
          - gorouter.service.cf.internal
      require_client_certificate: true

Logs:

Below are the logs with the netstat output every each run of nc -vz localhost 61001

Envoy trace logs interspersed with netstat -anp | grep 8080 output

   2018-09-12T09:56:54.39-0400 [PROXY/0] ERR [2018-09-12 13:56:54.397][69][debug][filter] source/common/tcp_proxy/tcp_proxy.cc:152] [C0] new tcp proxy session
   2018-09-12T09:56:54.39-0400 [PROXY/0] ERR [2018-09-12 13:56:54.397][69][trace][connection] source/common/network/connection_impl.cc:232] [C0] readDisable: enabled=true disable=true
   2018-09-12T09:56:54.39-0400 [PROXY/0] ERR [2018-09-12 13:56:54.397][69][debug][filter] source/common/tcp_proxy/tcp_proxy.cc:294] [C0] Creating connection to cluster 0-service-cluster
   2018-09-12T09:56:54.40-0400 [PROXY/0] ERR [2018-09-12 13:56:54.405][69][debug][pool] source/common/tcp/conn_pool.cc:67] creating a new connection
   2018-09-12T09:56:54.40-0400 [PROXY/0] ERR [2018-09-12 13:56:54.405][69][debug][pool] source/common/tcp/conn_pool.cc:325] [C1] connecting
   2018-09-12T09:56:54.40-0400 [PROXY/0] ERR [2018-09-12 13:56:54.405][69][debug][connection] source/common/network/connection_impl.cc:576] [C1] connecting to 10.255.98.48:8080
   2018-09-12T09:56:54.40-0400 [PROXY/0] ERR [2018-09-12 13:56:54.405][69][debug][connection] source/common/network/connection_impl.cc:585] [C1] connection in progress
   2018-09-12T09:56:54.40-0400 [PROXY/0] ERR [2018-09-12 13:56:54.405][69][debug][pool] source/common/tcp/conn_pool.cc:92] queueing request due to no available connections
   2018-09-12T09:56:54.40-0400 [PROXY/0] ERR [2018-09-12 13:56:54.405][69][debug][main] source/server/connection_handler_impl.cc:218] [C0] new connection
   2018-09-12T09:56:54.40-0400 [PROXY/0] ERR [2018-09-12 13:56:54.406][69][trace][connection] source/common/network/connection_impl.cc:389] [C0] socket event: 2
   2018-09-12T09:56:54.40-0400 [PROXY/0] ERR [2018-09-12 13:56:54.406][69][trace][connection] source/common/network/connection_impl.cc:457] [C0] write ready
   2018-09-12T09:56:54.40-0400 [PROXY/0] ERR [2018-09-12 13:56:54.406][69][debug][connection] source/common/ssl/ssl_socket.cc:131] [C0] handshake error: 5
   2018-09-12T09:56:54.40-0400 [PROXY/0] ERR [2018-09-12 13:56:54.406][69][debug][connection] source/common/network/connection_impl.cc:133] [C0] closing socket: 0
   2018-09-12T09:56:54.40-0400 [PROXY/0] ERR [2018-09-12 13:56:54.406][69][debug][pool] source/common/tcp/conn_pool.cc:196] canceling pending request
   2018-09-12T09:56:54.40-0400 [PROXY/0] ERR [2018-09-12 13:56:54.406][69][debug][main] source/server/connection_handler_impl.cc:51] [C0] adding to cleanup list
   2018-09-12T09:56:54.40-0400 [PROXY/0] ERR [2018-09-12 13:56:54.407][69][trace][main] source/common/event/dispatcher_impl.cc:126] item added to deferred deletion list (size=1)
   2018-09-12T09:56:54.40-0400 [PROXY/0] ERR [2018-09-12 13:56:54.407][69][trace][connection] source/common/network/connection_impl.cc:389] [C1] socket event: 2
   2018-09-12T09:56:54.40-0400 [PROXY/0] ERR [2018-09-12 13:56:54.407][69][trace][connection] source/common/network/connection_impl.cc:457] [C1] write ready
   2018-09-12T09:56:54.40-0400 [PROXY/0] ERR [2018-09-12 13:56:54.407][69][debug][connection] source/common/network/connection_impl.cc:466] [C1] connected
   2018-09-12T09:56:54.40-0400 [PROXY/0] ERR [2018-09-12 13:56:54.407][69][debug][pool] source/common/tcp/conn_pool.cc:244] [C1] moving to ready
   2018-09-12T09:56:54.40-0400 [PROXY/0] ERR [2018-09-12 13:56:54.407][69][trace][main] source/common/event/dispatcher_impl.cc:52] clearing deferred deletion list (size=1)

tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      9/python
tcp        0      0 10.255.98.48:8080       10.255.98.48:44254      ESTABLISHED 9/python
tcp        0      0 10.255.98.48:44254      10.255.98.48:8080       ESTABLISHED -

   2018-09-12T09:59:06.14-0400 [PROXY/0] ERR [2018-09-12 13:59:06.141][69][debug][filter] source/common/tcp_proxy/tcp_proxy.cc:152] [C2] new tcp proxy session
   2018-09-12T09:59:06.14-0400 [PROXY/0] ERR [2018-09-12 13:59:06.141][69][trace][connection] source/common/network/connection_impl.cc:232] [C2] readDisable: enabled=true disable=true
   2018-09-12T09:59:06.14-0400 [PROXY/0] ERR [2018-09-12 13:59:06.141][69][debug][filter] source/common/tcp_proxy/tcp_proxy.cc:294] [C2] Creating connection to cluster 0-service-cluster
   2018-09-12T09:59:06.14-0400 [PROXY/0] ERR [2018-09-12 13:59:06.141][69][debug][pool] source/common/tcp/conn_pool.cc:75] [C1] using existing connection
   2018-09-12T09:59:06.14-0400 [PROXY/0] ERR [2018-09-12 13:59:06.141][69][trace][connection] source/common/network/connection_impl.cc:232] [C2] readDisable: enabled=false disable=false
   2018-09-12T09:59:06.14-0400 [PROXY/0] ERR [2018-09-12 13:59:06.141][69][debug][filter] source/common/tcp_proxy/tcp_proxy.cc:480] TCP:onUpstreamEvent(), requestedServerName:
   2018-09-12T09:59:06.14-0400 [PROXY/0] ERR [2018-09-12 13:59:06.141][69][debug][main] source/server/connection_handler_impl.cc:218] [C2] new connection
   2018-09-12T09:59:06.14-0400 [PROXY/0] ERR [2018-09-12 13:59:06.141][69][trace][connection] source/common/network/connection_impl.cc:389] [C2] socket event: 3
   2018-09-12T09:59:06.14-0400 [PROXY/0] ERR [2018-09-12 13:59:06.141][69][trace][connection] source/common/network/connection_impl.cc:457] [C2] write ready
   2018-09-12T09:59:06.14-0400 [PROXY/0] ERR [2018-09-12 13:59:06.141][69][debug][connection] source/common/ssl/ssl_socket.cc:131] [C2] handshake error: 5
   2018-09-12T09:59:06.14-0400 [PROXY/0] ERR [2018-09-12 13:59:06.141][69][debug][connection] source/common/network/connection_impl.cc:133] [C2] closing socket: 0
   2018-09-12T09:59:06.14-0400 [PROXY/0] ERR [2018-09-12 13:59:06.142][69][debug][connection] source/common/network/connection_impl.cc:98] [C1] closing data_to_write=0 type=0
   2018-09-12T09:59:06.14-0400 [PROXY/0] ERR [2018-09-12 13:59:06.142][69][debug][connection] source/common/network/connection_impl.cc:133] [C1] closing socket: 1
   2018-09-12T09:59:06.14-0400 [PROXY/0] ERR [2018-09-12 13:59:06.142][69][debug][pool] source/common/tcp/conn_pool.cc:107] [C1] client disconnected
   2018-09-12T09:59:06.14-0400 [PROXY/0] ERR [2018-09-12 13:59:06.142][69][trace][main] source/common/event/dispatcher_impl.cc:126] item added to deferred deletion list (size=1)
   2018-09-12T09:59:06.14-0400 [PROXY/0] ERR [2018-09-12 13:59:06.142][69][debug][main] source/server/connection_handler_impl.cc:51] [C2] adding to cleanup list
   2018-09-12T09:59:06.14-0400 [PROXY/0] ERR [2018-09-12 13:59:06.142][69][trace][main] source/common/event/dispatcher_impl.cc:126] item added to deferred deletion list (size=2)
   2018-09-12T09:59:06.14-0400 [PROXY/0] ERR [2018-09-12 13:59:06.142][69][trace][main] source/common/event/dispatcher_impl.cc:52] clearing deferred deletion list (size=2)
   2018-09-12T09:59:06.14-0400 [PROXY/0] ERR [2018-09-12 13:59:06.142][69][debug][pool] source/common/tcp/conn_pool.cc:219] [C1] connection destroyed

tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      9/python
tcp        0      0 10.255.98.48:44254      10.255.98.48:8080       TIME_WAIT   -

   2018-09-12T09:59:15.47-0400 [PROXY/0] ERR [2018-09-12 13:59:15.475][67][debug][filter] source/common/tcp_proxy/tcp_proxy.cc:152] [C3] new tcp proxy session
   2018-09-12T09:59:15.47-0400 [PROXY/0] ERR [2018-09-12 13:59:15.475][67][trace][connection] source/common/network/connection_impl.cc:232] [C3] readDisable: enabled=true disable=true
   2018-09-12T09:59:15.47-0400 [PROXY/0] ERR [2018-09-12 13:59:15.475][67][debug][filter] source/common/tcp_proxy/tcp_proxy.cc:294] [C3] Creating connection to cluster 0-service-cluster
   2018-09-12T09:59:15.47-0400 [PROXY/0] ERR [2018-09-12 13:59:15.476][67][debug][pool] source/common/tcp/conn_pool.cc:67] creating a new connection
   2018-09-12T09:59:15.47-0400 [PROXY/0] ERR [2018-09-12 13:59:15.476][67][debug][pool] source/common/tcp/conn_pool.cc:325] [C4] connecting
   2018-09-12T09:59:15.47-0400 [PROXY/0] ERR [2018-09-12 13:59:15.476][67][debug][connection] source/common/network/connection_impl.cc:576] [C4] connecting to 10.255.98.48:8080
   2018-09-12T09:59:15.47-0400 [PROXY/0] ERR [2018-09-12 13:59:15.476][67][debug][connection] source/common/network/connection_impl.cc:585] [C4] connection in progress
   2018-09-12T09:59:15.47-0400 [PROXY/0] ERR [2018-09-12 13:59:15.476][67][debug][pool] source/common/tcp/conn_pool.cc:92] queueing request due to no available connections
   2018-09-12T09:59:15.47-0400 [PROXY/0] ERR [2018-09-12 13:59:15.477][67][debug][main] source/server/connection_handler_impl.cc:218] [C3] new connection
   2018-09-12T09:59:15.47-0400 [PROXY/0] ERR [2018-09-12 13:59:15.477][67][trace][connection] source/common/network/connection_impl.cc:389] [C3] socket event: 2
   2018-09-12T09:59:15.47-0400 [PROXY/0] ERR [2018-09-12 13:59:15.477][67][trace][connection] source/common/network/connection_impl.cc:457] [C3] write ready
   2018-09-12T09:59:15.47-0400 [PROXY/0] ERR [2018-09-12 13:59:15.477][67][debug][connection] source/common/ssl/ssl_socket.cc:131] [C3] handshake error: 5
   2018-09-12T09:59:15.47-0400 [PROXY/0] ERR [2018-09-12 13:59:15.477][67][debug][connection] source/common/network/connection_impl.cc:133] [C3] closing socket: 0
   2018-09-12T09:59:15.47-0400 [PROXY/0] ERR [2018-09-12 13:59:15.477][67][debug][pool] source/common/tcp/conn_pool.cc:196] canceling pending request
   2018-09-12T09:59:15.47-0400 [PROXY/0] ERR [2018-09-12 13:59:15.477][67][debug][main] source/server/connection_handler_impl.cc:51] [C3] adding to cleanup list
   2018-09-12T09:59:15.47-0400 [PROXY/0] ERR [2018-09-12 13:59:15.477][67][trace][main] source/common/event/dispatcher_impl.cc:126] item added to deferred deletion list (size=1)
   2018-09-12T09:59:15.47-0400 [PROXY/0] ERR [2018-09-12 13:59:15.477][67][trace][connection] source/common/network/connection_impl.cc:389] [C4] socket event: 2
   2018-09-12T09:59:15.47-0400 [PROXY/0] ERR [2018-09-12 13:59:15.477][67][trace][connection] source/common/network/connection_impl.cc:457] [C4] write ready
   2018-09-12T09:59:15.47-0400 [PROXY/0] ERR [2018-09-12 13:59:15.477][67][debug][connection] source/common/network/connection_impl.cc:466] [C4] connected
   2018-09-12T09:59:15.47-0400 [PROXY/0] ERR [2018-09-12 13:59:15.478][67][debug][pool] source/common/tcp/conn_pool.cc:244] [C4] moving to ready
   2018-09-12T09:59:15.47-0400 [PROXY/0] ERR [2018-09-12 13:59:15.478][67][trace][main] source/common/event/dispatcher_impl.cc:52] clearing deferred deletion list (size=1)

tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      9/python
tcp        0      0 10.255.98.48:8080       10.255.98.48:47604      ESTABLISHED 9/python
tcp        0      0 10.255.98.48:47604      10.255.98.48:8080       ESTABLISHED -
tcp        0      0 10.255.98.48:44254      10.255.98.48:8080       TIME_WAIT   -

   2018-09-12T09:59:33.55-0400 [PROXY/0] ERR [2018-09-12 13:59:33.557][76][debug][filter] source/common/tcp_proxy/tcp_proxy.cc:152] [C5] new tcp proxy session
   2018-09-12T09:59:33.55-0400 [PROXY/0] ERR [2018-09-12 13:59:33.557][76][trace][connection] source/common/network/connection_impl.cc:232] [C5] readDisable: enabled=true disable=true
   2018-09-12T09:59:33.55-0400 [PROXY/0] ERR [2018-09-12 13:59:33.557][76][debug][filter] source/common/tcp_proxy/tcp_proxy.cc:294] [C5] Creating connection to cluster 0-service-cluster
   2018-09-12T09:59:33.55-0400 [PROXY/0] ERR [2018-09-12 13:59:33.557][76][debug][pool] source/common/tcp/conn_pool.cc:67] creating a new connection
   2018-09-12T09:59:33.55-0400 [PROXY/0] ERR [2018-09-12 13:59:33.558][76][debug][pool] source/common/tcp/conn_pool.cc:325] [C6] connecting
   2018-09-12T09:59:33.55-0400 [PROXY/0] ERR [2018-09-12 13:59:33.558][76][debug][connection] source/common/network/connection_impl.cc:576] [C6] connecting to 10.255.98.48:8080
   2018-09-12T09:59:33.55-0400 [PROXY/0] ERR [2018-09-12 13:59:33.558][76][debug][connection] source/common/network/connection_impl.cc:585] [C6] connection in progress
   2018-09-12T09:59:33.55-0400 [PROXY/0] ERR [2018-09-12 13:59:33.558][76][debug][pool] source/common/tcp/conn_pool.cc:92] queueing request due to no available connections
   2018-09-12T09:59:33.55-0400 [PROXY/0] ERR [2018-09-12 13:59:33.558][76][debug][main] source/server/connection_handler_impl.cc:218] [C5] new connection
   2018-09-12T09:59:33.55-0400 [PROXY/0] ERR [2018-09-12 13:59:33.559][76][trace][connection] source/common/network/connection_impl.cc:389] [C5] socket event: 2
   2018-09-12T09:59:33.55-0400 [PROXY/0] ERR [2018-09-12 13:59:33.559][76][trace][connection] source/common/network/connection_impl.cc:457] [C5] write ready
   2018-09-12T09:59:33.56-0400 [PROXY/0] ERR [2018-09-12 13:59:33.560][76][debug][connection] source/common/ssl/ssl_socket.cc:131] [C5] handshake error: 5
   2018-09-12T09:59:33.56-0400 [PROXY/0] ERR [2018-09-12 13:59:33.560][76][debug][connection] source/common/network/connection_impl.cc:133] [C5] closing socket: 0
   2018-09-12T09:59:33.56-0400 [PROXY/0] ERR [2018-09-12 13:59:33.560][76][debug][pool] source/common/tcp/conn_pool.cc:196] canceling pending request
   2018-09-12T09:59:33.56-0400 [PROXY/0] ERR [2018-09-12 13:59:33.560][76][debug][main] source/server/connection_handler_impl.cc:51] [C5] adding to cleanup list
   2018-09-12T09:59:33.56-0400 [PROXY/0] ERR [2018-09-12 13:59:33.560][76][trace][main] source/common/event/dispatcher_impl.cc:126] item added to deferred deletion list (size=1)
   2018-09-12T09:59:33.56-0400 [PROXY/0] ERR [2018-09-12 13:59:33.560][76][trace][connection] source/common/network/connection_impl.cc:389] [C6] socket event: 2
   2018-09-12T09:59:33.56-0400 [PROXY/0] ERR [2018-09-12 13:59:33.560][76][trace][connection] source/common/network/connection_impl.cc:457] [C6] write ready
   2018-09-12T09:59:33.56-0400 [PROXY/0] ERR [2018-09-12 13:59:33.560][76][debug][connection] source/common/network/connection_impl.cc:466] [C6] connected
   2018-09-12T09:59:33.56-0400 [PROXY/0] ERR [2018-09-12 13:59:33.561][76][debug][pool] source/common/tcp/conn_pool.cc:244] [C6] moving to ready
   2018-09-12T09:59:33.56-0400 [PROXY/0] ERR [2018-09-12 13:59:33.561][76][trace][main] source/common/event/dispatcher_impl.cc:52] clearing deferred deletion list (size=1)

tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      9/python
tcp        0      0 10.255.98.48:8080       10.255.98.48:47604      ESTABLISHED 9/python
tcp        0      0 10.255.98.48:8080       10.255.98.48:48010      ESTABLISHED -
tcp        0      0 10.255.98.48:48010      10.255.98.48:8080       ESTABLISHED -
tcp        0      0 10.255.98.48:47604      10.255.98.48:8080       ESTABLISHED -
tcp        0      0 10.255.98.48:44254      10.255.98.48:8080       TIME_WAIT   -

   2018-09-12T09:59:48.63-0400 [PROXY/0] ERR [2018-09-12 13:59:48.635][78][debug][filter] source/common/tcp_proxy/tcp_proxy.cc:152] [C7] new tcp proxy session
   2018-09-12T09:59:48.63-0400 [PROXY/0] ERR [2018-09-12 13:59:48.635][78][trace][connection] source/common/network/connection_impl.cc:232] [C7] readDisable: enabled=true disable=true
   2018-09-12T09:59:48.63-0400 [PROXY/0] ERR [2018-09-12 13:59:48.636][78][debug][filter] source/common/tcp_proxy/tcp_proxy.cc:294] [C7] Creating connection to cluster 0-service-cluster
   2018-09-12T09:59:48.63-0400 [PROXY/0] ERR [2018-09-12 13:59:48.636][78][debug][pool] source/common/tcp/conn_pool.cc:67] creating a new connection
   2018-09-12T09:59:48.63-0400 [PROXY/0] ERR [2018-09-12 13:59:48.636][78][debug][pool] source/common/tcp/conn_pool.cc:325] [C8] connecting
   2018-09-12T09:59:48.63-0400 [PROXY/0] ERR [2018-09-12 13:59:48.636][78][debug][connection] source/common/network/connection_impl.cc:576] [C8] connecting to 10.255.98.48:8080
   2018-09-12T09:59:48.63-0400 [PROXY/0] ERR [2018-09-12 13:59:48.636][78][debug][connection] source/common/network/connection_impl.cc:585] [C8] connection in progress
   2018-09-12T09:59:48.63-0400 [PROXY/0] ERR [2018-09-12 13:59:48.636][78][debug][pool] source/common/tcp/conn_pool.cc:92] queueing request due to no available connections
   2018-09-12T09:59:48.63-0400 [PROXY/0] ERR [2018-09-12 13:59:48.636][78][debug][main] source/server/connection_handler_impl.cc:218] [C7] new connection
   2018-09-12T09:59:48.63-0400 [PROXY/0] ERR [2018-09-12 13:59:48.636][78][trace][connection] source/common/network/connection_impl.cc:389] [C7] socket event: 2
   2018-09-12T09:59:48.63-0400 [PROXY/0] ERR [2018-09-12 13:59:48.636][78][trace][connection] source/common/network/connection_impl.cc:457] [C7] write ready
   2018-09-12T09:59:48.63-0400 [PROXY/0] ERR [2018-09-12 13:59:48.636][78][debug][connection] source/common/ssl/ssl_socket.cc:131] [C7] handshake error: 5
   2018-09-12T09:59:48.63-0400 [PROXY/0] ERR [2018-09-12 13:59:48.637][78][debug][connection] source/common/network/connection_impl.cc:133] [C7] closing socket: 0
   2018-09-12T09:59:48.63-0400 [PROXY/0] ERR [2018-09-12 13:59:48.637][78][debug][pool] source/common/tcp/conn_pool.cc:196] canceling pending request
   2018-09-12T09:59:48.63-0400 [PROXY/0] ERR [2018-09-12 13:59:48.637][78][debug][main] source/server/connection_handler_impl.cc:51] [C7] adding to cleanup list
   2018-09-12T09:59:48.63-0400 [PROXY/0] ERR [2018-09-12 13:59:48.637][78][trace][main] source/common/event/dispatcher_impl.cc:126] item added to deferred deletion list (size=1)
   2018-09-12T09:59:48.63-0400 [PROXY/0] ERR [2018-09-12 13:59:48.637][78][trace][connection] source/common/network/connection_impl.cc:389] [C8] socket event: 2
   2018-09-12T09:59:48.63-0400 [PROXY/0] ERR [2018-09-12 13:59:48.637][78][trace][connection] source/common/network/connection_impl.cc:457] [C8] write ready
   2018-09-12T09:59:48.63-0400 [PROXY/0] ERR [2018-09-12 13:59:48.637][78][debug][connection] source/common/network/connection_impl.cc:466] [C8] connected
   2018-09-12T09:59:48.63-0400 [PROXY/0] ERR [2018-09-12 13:59:48.637][78][debug][pool] source/common/tcp/conn_pool.cc:244] [C8] moving to ready
   2018-09-12T09:59:48.63-0400 [PROXY/0] ERR [2018-09-12 13:59:48.637][78][trace][main] source/common/event/dispatcher_impl.cc:52] clearing deferred deletion list (size=1)

tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      9/python
tcp        0      0 10.255.98.48:8080       10.255.98.48:47604      ESTABLISHED 9/python
tcp        0      0 10.255.98.48:48372      10.255.98.48:8080       ESTABLISHED -
tcp        0      0 10.255.98.48:8080       10.255.98.48:48010      ESTABLISHED -
tcp        0      0 10.255.98.48:48010      10.255.98.48:8080       ESTABLISHED -
tcp        0      0 10.255.98.48:47604      10.255.98.48:8080       ESTABLISHED -
tcp        0      0 10.255.98.48:8080       10.255.98.48:48372      ESTABLISHED -
tcp        0      0 10.255.98.48:44254      10.255.98.48:8080       TIME_WAIT   -

   2018-09-12T10:00:02.70-0400 [PROXY/0] ERR [2018-09-12 14:00:02.708][80][debug][filter] source/common/tcp_proxy/tcp_proxy.cc:152] [C9] new tcp proxy session
   2018-09-12T10:00:02.70-0400 [PROXY/0] ERR [2018-09-12 14:00:02.708][80][trace][connection] source/common/network/connection_impl.cc:232] [C9] readDisable: enabled=true disable=true
   2018-09-12T10:00:02.70-0400 [PROXY/0] ERR [2018-09-12 14:00:02.708][80][debug][filter] source/common/tcp_proxy/tcp_proxy.cc:294] [C9] Creating connection to cluster 0-service-cluster
   2018-09-12T10:00:02.70-0400 [PROXY/0] ERR [2018-09-12 14:00:02.708][80][debug][pool] source/common/tcp/conn_pool.cc:67] creating a new connection
   2018-09-12T10:00:02.70-0400 [PROXY/0] ERR [2018-09-12 14:00:02.708][80][debug][pool] source/common/tcp/conn_pool.cc:325] [C10] connecting
   2018-09-12T10:00:02.70-0400 [PROXY/0] ERR [2018-09-12 14:00:02.708][80][debug][connection] source/common/network/connection_impl.cc:576] [C10] connecting to 10.255.98.48:8080
   2018-09-12T10:00:02.70-0400 [PROXY/0] ERR [2018-09-12 14:00:02.708][80][debug][connection] source/common/network/connection_impl.cc:585] [C10] connection in progress
   2018-09-12T10:00:02.70-0400 [PROXY/0] ERR [2018-09-12 14:00:02.708][80][debug][pool] source/common/tcp/conn_pool.cc:92] queueing request due to no available connections
   2018-09-12T10:00:02.70-0400 [PROXY/0] ERR [2018-09-12 14:00:02.708][80][debug][main] source/server/connection_handler_impl.cc:218] [C9] new connection
   2018-09-12T10:00:02.70-0400 [PROXY/0] ERR [2018-09-12 14:00:02.709][80][trace][connection] source/common/network/connection_impl.cc:389] [C9] socket event: 2
   2018-09-12T10:00:02.70-0400 [PROXY/0] ERR [2018-09-12 14:00:02.709][80][trace][connection] source/common/network/connection_impl.cc:457] [C9] write ready
   2018-09-12T10:00:02.70-0400 [PROXY/0] ERR [2018-09-12 14:00:02.709][80][debug][connection] source/common/ssl/ssl_socket.cc:131] [C9] handshake error: 2
   2018-09-12T10:00:02.70-0400 [PROXY/0] ERR [2018-09-12 14:00:02.709][80][trace][connection] source/common/network/connection_impl.cc:389] [C10] socket event: 2
   2018-09-12T10:00:02.70-0400 [PROXY/0] ERR [2018-09-12 14:00:02.709][80][trace][connection] source/common/network/connection_impl.cc:457] [C10] write ready
   2018-09-12T10:00:02.70-0400 [PROXY/0] ERR [2018-09-12 14:00:02.709][80][debug][connection] source/common/network/connection_impl.cc:466] [C10] connected
   2018-09-12T10:00:02.71-0400 [PROXY/0] ERR [2018-09-12 14:00:02.709][80][debug][pool] source/common/tcp/conn_pool.cc:249] [C10] assigning connection
   2018-09-12T10:00:02.71-0400 [PROXY/0] ERR [2018-09-12 14:00:02.710][80][trace][connection] source/common/network/connection_impl.cc:232] [C9] readDisable: enabled=false disable=false
   2018-09-12T10:00:02.71-0400 [PROXY/0] ERR [2018-09-12 14:00:02.710][80][debug][filter] source/common/tcp_proxy/tcp_proxy.cc:480] TCP:onUpstreamEvent(), requestedServerName:
   2018-09-12T10:00:02.71-0400 [PROXY/0] ERR [2018-09-12 14:00:02.710][80][trace][connection] source/common/network/connection_impl.cc:389] [C9] socket event: 2
   2018-09-12T10:00:02.71-0400 [PROXY/0] ERR [2018-09-12 14:00:02.710][80][trace][connection] source/common/network/connection_impl.cc:457] [C9] write ready
   2018-09-12T10:00:02.71-0400 [PROXY/0] ERR [2018-09-12 14:00:02.710][80][debug][connection] source/common/ssl/ssl_socket.cc:131] [C9] handshake error: 2
   2018-09-12T10:00:02.71-0400 [PROXY/0] ERR [2018-09-12 14:00:02.710][80][trace][connection] source/common/network/connection_impl.cc:389] [C9] socket event: 3
   2018-09-12T10:00:02.71-0400 [PROXY/0] ERR [2018-09-12 14:00:02.710][80][trace][connection] source/common/network/connection_impl.cc:457] [C9] write ready
   2018-09-12T10:00:02.71-0400 [PROXY/0] ERR [2018-09-12 14:00:02.710][80][debug][connection] source/common/ssl/ssl_socket.cc:131] [C9] handshake error: 5
   2018-09-12T10:00:02.71-0400 [PROXY/0] ERR [2018-09-12 14:00:02.710][80][debug][connection] source/common/network/connection_impl.cc:133] [C9] closing socket: 0
   2018-09-12T10:00:02.71-0400 [PROXY/0] ERR [2018-09-12 14:00:02.711][80][debug][connection] source/common/network/connection_impl.cc:98] [C10] closing data_to_write=0 type=0
   2018-09-12T10:00:02.71-0400 [PROXY/0] ERR [2018-09-12 14:00:02.711][80][debug][connection] source/common/network/connection_impl.cc:133] [C10] closing socket: 1
   2018-09-12T10:00:02.71-0400 [PROXY/0] ERR [2018-09-12 14:00:02.711][80][debug][pool] source/common/tcp/conn_pool.cc:107] [C10] client disconnected
   2018-09-12T10:00:02.71-0400 [PROXY/0] ERR [2018-09-12 14:00:02.711][80][trace][main] source/common/event/dispatcher_impl.cc:126] item added to deferred deletion list (size=1)
   2018-09-12T10:00:02.71-0400 [PROXY/0] ERR [2018-09-12 14:00:02.711][80][debug][main] source/server/connection_handler_impl.cc:51] [C9] adding to cleanup list
   2018-09-12T10:00:02.71-0400 [PROXY/0] ERR [2018-09-12 14:00:02.711][80][trace][main] source/common/event/dispatcher_impl.cc:126] item added to deferred deletion list (size=2)
   2018-09-12T10:00:02.71-0400 [PROXY/0] ERR [2018-09-12 14:00:02.711][80][trace][main] source/common/event/dispatcher_impl.cc:52] clearing deferred deletion list (size=2)
   2018-09-12T10:00:02.71-0400 [PROXY/0] ERR [2018-09-12 14:00:02.711][80][debug][pool] source/common/tcp/conn_pool.cc:219] [C10] connection destroyed

tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      9/python
tcp        0      0 10.255.98.48:8080       10.255.98.48:47604      ESTABLISHED 9/python
tcp        0      0 10.255.98.48:48372      10.255.98.48:8080       ESTABLISHED -
tcp        0      0 10.255.98.48:8080       10.255.98.48:48010      ESTABLISHED -
tcp        0      0 10.255.98.48:48010      10.255.98.48:8080       ESTABLISHED -
tcp        1      0 10.255.98.48:8080       10.255.98.48:48716      CLOSE_WAIT  -
tcp        0      0 10.255.98.48:47604      10.255.98.48:8080       ESTABLISHED -
tcp        0      0 10.255.98.48:48716      10.255.98.48:8080       FIN_WAIT2   -
tcp        0      0 10.255.98.48:8080       10.255.98.48:48372      ESTABLISHED -

   2018-09-12T10:06:48.78-0400 [PROXY/0] ERR [2018-09-12 14:06:48.779][73][debug][filter] source/common/tcp_proxy/tcp_proxy.cc:152] [C11] new tcp proxy session
   2018-09-12T10:06:48.78-0400 [PROXY/0] ERR [2018-09-12 14:06:48.779][73][trace][connection] source/common/network/connection_impl.cc:232] [C11] readDisable: enabled=true disable=true
   2018-09-12T10:06:48.78-0400 [PROXY/0] ERR [2018-09-12 14:06:48.779][73][debug][filter] source/common/tcp_proxy/tcp_proxy.cc:294] [C11] Creating connection to cluster 0-service-cluster
   2018-09-12T10:06:48.78-0400 [PROXY/0] ERR [2018-09-12 14:06:48.780][73][debug][pool] source/common/tcp/conn_pool.cc:67] creating a new connection
   2018-09-12T10:06:48.78-0400 [PROXY/0] ERR [2018-09-12 14:06:48.780][73][debug][pool] source/common/tcp/conn_pool.cc:325] [C12] connecting
   2018-09-12T10:06:48.78-0400 [PROXY/0] ERR [2018-09-12 14:06:48.780][73][debug][connection] source/common/network/connection_impl.cc:576] [C12] connecting to 10.255.98.48:8080
   2018-09-12T10:06:48.78-0400 [PROXY/0] ERR [2018-09-12 14:06:48.780][73][debug][connection] source/common/network/connection_impl.cc:585] [C12] connection in progress
   2018-09-12T10:06:48.78-0400 [PROXY/0] ERR [2018-09-12 14:06:48.780][73][debug][pool] source/common/tcp/conn_pool.cc:92] queueing request due to no available connections
   2018-09-12T10:06:48.78-0400 [PROXY/0] ERR [2018-09-12 14:06:48.780][73][debug][main] source/server/connection_handler_impl.cc:218] [C11] new connection
   2018-09-12T10:06:48.78-0400 [PROXY/0] ERR [2018-09-12 14:06:48.780][73][trace][connection] source/common/network/connection_impl.cc:389] [C11] socket event: 2
   2018-09-12T10:06:48.78-0400 [PROXY/0] ERR [2018-09-12 14:06:48.781][73][trace][connection] source/common/network/connection_impl.cc:457] [C11] write ready
   2018-09-12T10:06:48.78-0400 [PROXY/0] ERR [2018-09-12 14:06:48.781][73][debug][connection] source/common/ssl/ssl_socket.cc:131] [C11] handshake error: 5
   2018-09-12T10:06:48.78-0400 [PROXY/0] ERR [2018-09-12 14:06:48.781][73][debug][connection] source/common/network/connection_impl.cc:133] [C11] closing socket: 0
   2018-09-12T10:06:48.78-0400 [PROXY/0] ERR [2018-09-12 14:06:48.781][73][debug][pool] source/common/tcp/conn_pool.cc:196] canceling pending request
   2018-09-12T10:06:48.78-0400 [PROXY/0] ERR [2018-09-12 14:06:48.781][73][debug][main] source/server/connection_handler_impl.cc:51] [C11] adding to cleanup list
   2018-09-12T10:06:48.78-0400 [PROXY/0] ERR [2018-09-12 14:06:48.781][73][trace][main] source/common/event/dispatcher_impl.cc:126] item added to deferred deletion list (size=1)
   2018-09-12T10:06:48.78-0400 [PROXY/0] ERR [2018-09-12 14:06:48.781][73][trace][connection] source/common/network/connection_impl.cc:389] [C12] socket event: 2
   2018-09-12T10:06:48.78-0400 [PROXY/0] ERR [2018-09-12 14:06:48.781][73][trace][connection] source/common/network/connection_impl.cc:457] [C12] write ready
   2018-09-12T10:06:48.78-0400 [PROXY/0] ERR [2018-09-12 14:06:48.781][73][debug][connection] source/common/network/connection_impl.cc:466] [C12] connected
   2018-09-12T10:06:48.78-0400 [PROXY/0] ERR [2018-09-12 14:06:48.781][73][debug][pool] source/common/tcp/conn_pool.cc:244] [C12] moving to ready
   2018-09-12T10:06:48.78-0400 [PROXY/0] ERR [2018-09-12 14:06:48.781][73][trace][main] source/common/event/dispatcher_impl.cc:52] clearing deferred deletion list (size=1)

tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      9/python
tcp        0      0 10.255.98.48:8080       10.255.98.48:47604      ESTABLISHED 9/python
tcp        0      0 10.255.98.48:57872      10.255.98.48:8080       ESTABLISHED -
tcp        0      0 10.255.98.48:48372      10.255.98.48:8080       ESTABLISHED -
tcp        0      0 10.255.98.48:8080       10.255.98.48:48010      ESTABLISHED -
tcp        0      0 10.255.98.48:48010      10.255.98.48:8080       ESTABLISHED -
tcp        1      0 10.255.98.48:8080       10.255.98.48:48716      CLOSE_WAIT  -
tcp        0      0 10.255.98.48:8080       10.255.98.48:57872      ESTABLISHED -
tcp        0      0 10.255.98.48:47604      10.255.98.48:8080       ESTABLISHED -
tcp        0      0 10.255.98.48:8080       10.255.98.48:48372      ESTABLISHED -

   2018-09-12T10:07:12.66-0400 [PROXY/0] ERR [2018-09-12 14:07:12.661][73][debug][filter] source/common/tcp_proxy/tcp_proxy.cc:152] [C13] new tcp proxy session
   2018-09-12T10:07:12.66-0400 [PROXY/0] ERR [2018-09-12 14:07:12.661][73][trace][connection] source/common/network/connection_impl.cc:232] [C13] readDisable: enabled=true disable=true
   2018-09-12T10:07:12.66-0400 [PROXY/0] ERR [2018-09-12 14:07:12.661][73][debug][filter] source/common/tcp_proxy/tcp_proxy.cc:294] [C13] Creating connection to cluster 0-service-cluster
   2018-09-12T10:07:12.66-0400 [PROXY/0] ERR [2018-09-12 14:07:12.661][73][debug][pool] source/common/tcp/conn_pool.cc:75] [C12] using existing connection
   2018-09-12T10:07:12.66-0400 [PROXY/0] ERR [2018-09-12 14:07:12.661][73][trace][connection] source/common/network/connection_impl.cc:232] [C13] readDisable: enabled=false disable=false
   2018-09-12T10:07:12.66-0400 [PROXY/0] ERR [2018-09-12 14:07:12.661][73][debug][filter] source/common/tcp_proxy/tcp_proxy.cc:480] TCP:onUpstreamEvent(), requestedServerName:
   2018-09-12T10:07:12.66-0400 [PROXY/0] ERR [2018-09-12 14:07:12.661][73][debug][main] source/server/connection_handler_impl.cc:218] [C13] new connection
   2018-09-12T10:07:12.66-0400 [PROXY/0] ERR [2018-09-12 14:07:12.661][73][trace][connection] source/common/network/connection_impl.cc:389] [C13] socket event: 2
   2018-09-12T10:07:12.66-0400 [PROXY/0] ERR [2018-09-12 14:07:12.661][73][trace][connection] source/common/network/connection_impl.cc:457] [C13] write ready
   2018-09-12T10:07:12.66-0400 [PROXY/0] ERR [2018-09-12 14:07:12.661][73][debug][connection] source/common/ssl/ssl_socket.cc:131] [C13] handshake error: 2
   2018-09-12T10:07:12.66-0400 [PROXY/0] ERR [2018-09-12 14:07:12.662][73][trace][connection] source/common/network/connection_impl.cc:389] [C13] socket event: 3
   2018-09-12T10:07:12.66-0400 [PROXY/0] ERR [2018-09-12 14:07:12.662][73][trace][connection] source/common/network/connection_impl.cc:457] [C13] write ready
   2018-09-12T10:07:12.66-0400 [PROXY/0] ERR [2018-09-12 14:07:12.662][73][debug][connection] source/common/ssl/ssl_socket.cc:131] [C13] handshake error: 5
   2018-09-12T10:07:12.66-0400 [PROXY/0] ERR [2018-09-12 14:07:12.663][73][debug][connection] source/common/network/connection_impl.cc:133] [C13] closing socket: 0
   2018-09-12T10:07:12.66-0400 [PROXY/0] ERR [2018-09-12 14:07:12.663][73][debug][connection] source/common/network/connection_impl.cc:98] [C12] closing data_to_write=0 type=0
   2018-09-12T10:07:12.66-0400 [PROXY/0] ERR [2018-09-12 14:07:12.663][73][debug][connection] source/common/network/connection_impl.cc:133] [C12] closing socket: 1
   2018-09-12T10:07:12.66-0400 [PROXY/0] ERR [2018-09-12 14:07:12.663][73][debug][pool] source/common/tcp/conn_pool.cc:107] [C12] client disconnected
   2018-09-12T10:07:12.66-0400 [PROXY/0] ERR [2018-09-12 14:07:12.663][73][trace][main] source/common/event/dispatcher_impl.cc:126] item added to deferred deletion list (size=1)
   2018-09-12T10:07:12.66-0400 [PROXY/0] ERR [2018-09-12 14:07:12.663][73][debug][main] source/server/connection_handler_impl.cc:51] [C13] adding to cleanup list
   2018-09-12T10:07:12.66-0400 [PROXY/0] ERR [2018-09-12 14:07:12.663][73][trace][main] source/common/event/dispatcher_impl.cc:126] item added to deferred deletion list (size=2)
   2018-09-12T10:07:12.66-0400 [PROXY/0] ERR [2018-09-12 14:07:12.663][73][trace][main] source/common/event/dispatcher_impl.cc:52] clearing deferred deletion list (size=2)
   2018-09-12T10:07:12.66-0400 [PROXY/0] ERR [2018-09-12 14:07:12.663][73][debug][pool] source/common/tcp/conn_pool.cc:219] [C12] connection destroyed

tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      9/python
tcp        0      0 10.255.98.48:8080       10.255.98.48:47604      ESTABLISHED 9/python
tcp        0      0 10.255.98.48:57872      10.255.98.48:8080       FIN_WAIT2   -
tcp        0      0 10.255.98.48:48372      10.255.98.48:8080       ESTABLISHED -
tcp        0      0 10.255.98.48:8080       10.255.98.48:48010      ESTABLISHED -
tcp        0      0 10.255.98.48:48010      10.255.98.48:8080       ESTABLISHED -
tcp        1      0 10.255.98.48:8080       10.255.98.48:48716      CLOSE_WAIT  -
tcp        1      0 10.255.98.48:8080       10.255.98.48:57872      CLOSE_WAIT  -
tcp        0      0 10.255.98.48:47604      10.255.98.48:8080       ESTABLISHED -
tcp        0      0 10.255.98.48:8080       10.255.98.48:48372      ESTABLISHED -

   2018-09-12T10:09:40.13-0400 [PROXY/0] ERR [2018-09-12 14:09:40.135][69][debug][filter] source/common/tcp_proxy/tcp_proxy.cc:152] [C14] new tcp proxy session
   2018-09-12T10:09:40.13-0400 [PROXY/0] ERR [2018-09-12 14:09:40.135][69][trace][connection] source/common/network/connection_impl.cc:232] [C14] readDisable: enabled=true disable=true
   2018-09-12T10:09:40.13-0400 [PROXY/0] ERR [2018-09-12 14:09:40.135][69][debug][filter] source/common/tcp_proxy/tcp_proxy.cc:294] [C14] Creating connection to cluster 0-service-cluster
   2018-09-12T10:09:40.13-0400 [PROXY/0] ERR [2018-09-12 14:09:40.135][69][debug][pool] source/common/tcp/conn_pool.cc:67] creating a new connection
   2018-09-12T10:09:40.13-0400 [PROXY/0] ERR [2018-09-12 14:09:40.136][69][debug][pool] source/common/tcp/conn_pool.cc:325] [C15] connecting
   2018-09-12T10:09:40.13-0400 [PROXY/0] ERR [2018-09-12 14:09:40.136][69][debug][connection] source/common/network/connection_impl.cc:576] [C15] connecting to 10.255.98.48:8080
   2018-09-12T10:09:40.13-0400 [PROXY/0] ERR [2018-09-12 14:09:40.136][69][debug][connection] source/common/network/connection_impl.cc:585] [C15] connection in progress
   2018-09-12T10:09:40.13-0400 [PROXY/0] ERR [2018-09-12 14:09:40.136][69][debug][pool] source/common/tcp/conn_pool.cc:92] queueing request due to no available connections
   2018-09-12T10:09:40.13-0400 [PROXY/0] ERR [2018-09-12 14:09:40.136][69][debug][main] source/server/connection_handler_impl.cc:218] [C14] new connection
   2018-09-12T10:09:40.13-0400 [PROXY/0] ERR [2018-09-12 14:09:40.136][69][trace][connection] source/common/network/connection_impl.cc:389] [C14] socket event: 2
   2018-09-12T10:09:40.13-0400 [PROXY/0] ERR [2018-09-12 14:09:40.136][69][trace][connection] source/common/network/connection_impl.cc:457] [C14] write ready
   2018-09-12T10:09:40.13-0400 [PROXY/0] ERR [2018-09-12 14:09:40.136][69][debug][connection] source/common/ssl/ssl_socket.cc:131] [C14] handshake error: 5
   2018-09-12T10:09:40.13-0400 [PROXY/0] ERR [2018-09-12 14:09:40.136][69][debug][connection] source/common/network/connection_impl.cc:133] [C14] closing socket: 0
   2018-09-12T10:09:40.13-0400 [PROXY/0] ERR [2018-09-12 14:09:40.137][69][debug][pool] source/common/tcp/conn_pool.cc:196] canceling pending request
   2018-09-12T10:09:40.13-0400 [PROXY/0] ERR [2018-09-12 14:09:40.137][69][debug][main] source/server/connection_handler_impl.cc:51] [C14] adding to cleanup list
   2018-09-12T10:09:40.13-0400 [PROXY/0] ERR [2018-09-12 14:09:40.137][69][trace][main] source/common/event/dispatcher_impl.cc:126] item added to deferred deletion list (size=1)
   2018-09-12T10:09:40.13-0400 [PROXY/0] ERR [2018-09-12 14:09:40.137][69][trace][connection] source/common/network/connection_impl.cc:389] [C15] socket event: 2
   2018-09-12T10:09:40.13-0400 [PROXY/0] ERR [2018-09-12 14:09:40.137][69][trace][connection] source/common/network/connection_impl.cc:457] [C15] write ready
   2018-09-12T10:09:40.13-0400 [PROXY/0] ERR [2018-09-12 14:09:40.137][69][debug][connection] source/common/network/connection_impl.cc:466] [C15] connected
   2018-09-12T10:09:40.13-0400 [PROXY/0] ERR [2018-09-12 14:09:40.137][69][debug][pool] source/common/tcp/conn_pool.cc:244] [C15] moving to ready
   2018-09-12T10:09:40.13-0400 [PROXY/0] ERR [2018-09-12 14:09:40.137][69][trace][main] source/common/event/dispatcher_impl.cc:52] clearing deferred deletion list (size=1)

tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      9/python
tcp        0      0 10.255.98.48:8080       10.255.98.48:47604      ESTABLISHED 9/python
tcp        0      0 10.255.98.48:33646      10.255.98.48:8080       ESTABLISHED -
tcp        0      0 10.255.98.48:48372      10.255.98.48:8080       ESTABLISHED -
tcp        0      0 10.255.98.48:8080       10.255.98.48:48010      ESTABLISHED -
tcp        0      0 10.255.98.48:8080       10.255.98.48:33646      ESTABLISHED -
tcp        0      0 10.255.98.48:48010      10.255.98.48:8080       ESTABLISHED -
tcp        1      0 10.255.98.48:8080       10.255.98.48:48716      CLOSE_WAIT  -
tcp        1      0 10.255.98.48:8080       10.255.98.48:57872      CLOSE_WAIT  -
tcp        0      0 10.255.98.48:47604      10.255.98.48:8080       ESTABLISHED -
tcp        0      0 10.255.98.48:8080       10.255.98.48:48372      ESTABLISHED -

   2018-09-12T10:09:45.09-0400 [PROXY/0] ERR [2018-09-12 14:09:45.093][66][debug][filter] source/common/tcp_proxy/tcp_proxy.cc:152] [C16] new tcp proxy session
   2018-09-12T10:09:45.09-0400 [PROXY/0] ERR [2018-09-12 14:09:45.093][66][trace][connection] source/common/network/connection_impl.cc:232] [C16] readDisable: enabled=true disable=true
   2018-09-12T10:09:45.09-0400 [PROXY/0] ERR [2018-09-12 14:09:45.094][66][debug][filter] source/common/tcp_proxy/tcp_proxy.cc:294] [C16] Creating connection to cluster 0-service-cluster
   2018-09-12T10:09:45.09-0400 [PROXY/0] ERR [2018-09-12 14:09:45.094][66][debug][pool] source/common/tcp/conn_pool.cc:67] creating a new connection
   2018-09-12T10:09:45.09-0400 [PROXY/0] ERR [2018-09-12 14:09:45.094][66][debug][pool] source/common/tcp/conn_pool.cc:325] [C17] connecting
   2018-09-12T10:09:45.09-0400 [PROXY/0] ERR [2018-09-12 14:09:45.094][66][debug][connection] source/common/network/connection_impl.cc:576] [C17] connecting to 10.255.98.48:8080
   2018-09-12T10:09:45.09-0400 [PROXY/0] ERR [2018-09-12 14:09:45.094][66][debug][connection] source/common/network/connection_impl.cc:585] [C17] connection in progress
   2018-09-12T10:09:45.09-0400 [PROXY/0] ERR [2018-09-12 14:09:45.095][66][debug][pool] source/common/tcp/conn_pool.cc:92] queueing request due to no available connections
   2018-09-12T10:09:45.09-0400 [PROXY/0] ERR [2018-09-12 14:09:45.095][66][debug][main] source/server/connection_handler_impl.cc:218] [C16] new connection
   2018-09-12T10:09:45.09-0400 [PROXY/0] ERR [2018-09-12 14:09:45.095][66][trace][connection] source/common/network/connection_impl.cc:389] [C16] socket event: 2
   2018-09-12T10:09:45.09-0400 [PROXY/0] ERR [2018-09-12 14:09:45.095][66][trace][connection] source/common/network/connection_impl.cc:457] [C16] write ready
   2018-09-12T10:09:45.09-0400 [PROXY/0] ERR [2018-09-12 14:09:45.095][66][debug][connection] source/common/ssl/ssl_socket.cc:131] [C16] handshake error: 5
   2018-09-12T10:09:45.09-0400 [PROXY/0] ERR [2018-09-12 14:09:45.095][66][debug][connection] source/common/network/connection_impl.cc:133] [C16] closing socket: 0
   2018-09-12T10:09:45.09-0400 [PROXY/0] ERR [2018-09-12 14:09:45.095][66][debug][pool] source/common/tcp/conn_pool.cc:196] canceling pending request
   2018-09-12T10:09:45.09-0400 [PROXY/0] ERR [2018-09-12 14:09:45.095][66][debug][main] source/server/connection_handler_impl.cc:51] [C16] adding to cleanup list
   2018-09-12T10:09:45.09-0400 [PROXY/0] ERR [2018-09-12 14:09:45.095][66][trace][main] source/common/event/dispatcher_impl.cc:126] item added to deferred deletion list (size=1)
   2018-09-12T10:09:45.09-0400 [PROXY/0] ERR [2018-09-12 14:09:45.095][66][trace][connection] source/common/network/connection_impl.cc:389] [C17] socket event: 2
   2018-09-12T10:09:45.09-0400 [PROXY/0] ERR [2018-09-12 14:09:45.095][66][trace][connection] source/common/network/connection_impl.cc:457] [C17] write ready
   2018-09-12T10:09:45.09-0400 [PROXY/0] ERR [2018-09-12 14:09:45.095][66][debug][connection] source/common/network/connection_impl.cc:466] [C17] connected
   2018-09-12T10:09:45.09-0400 [PROXY/0] ERR [2018-09-12 14:09:45.095][66][debug][pool] source/common/tcp/conn_pool.cc:244] [C17] moving to ready
   2018-09-12T10:09:45.09-0400 [PROXY/0] ERR [2018-09-12 14:09:45.096][66][trace][main] source/common/event/dispatcher_impl.cc:52] clearing deferred deletion list (size=1)
   2018-09-12T10:09:47.84-0400 [SSH/0] OUT Successful remote access by 192.168.50.1:39304
   2018-09-12T10:09:47.86-0400 [SSH/0] OUT Remote access ended for 192.168.50.1:39304

tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      9/python
tcp        0      0 10.255.98.48:8080       10.255.98.48:47604      ESTABLISHED 9/python
tcp        0      0 10.255.98.48:33646      10.255.98.48:8080       ESTABLISHED -
tcp        0      0 10.255.98.48:48372      10.255.98.48:8080       ESTABLISHED -
tcp        0      0 10.255.98.48:8080       10.255.98.48:48010      ESTABLISHED -
tcp        0      0 10.255.98.48:8080       10.255.98.48:33646      ESTABLISHED -
tcp        0      0 10.255.98.48:8080       10.255.98.48:33780      ESTABLISHED -
tcp        0      0 10.255.98.48:48010      10.255.98.48:8080       ESTABLISHED -
tcp        1      0 10.255.98.48:8080       10.255.98.48:48716      CLOSE_WAIT  -
tcp        1      0 10.255.98.48:8080       10.255.98.48:57872      CLOSE_WAIT  -
tcp        0      0 10.255.98.48:47604      10.255.98.48:8080       ESTABLISHED -
tcp        0      0 10.255.98.48:33780      10.255.98.48:8080       ESTABLISHED -
tcp        0      0 10.255.98.48:8080       10.255.98.48:48372      ESTABLISHED -


   2018-09-12T10:09:48.77-0400 [PROXY/0] ERR [2018-09-12 14:09:48.769][78][debug][filter] source/common/tcp_proxy/tcp_proxy.cc:152] [C18] new tcp proxy session
   2018-09-12T10:09:48.77-0400 [PROXY/0] ERR [2018-09-12 14:09:48.769][78][trace][connection] source/common/network/connection_impl.cc:232] [C18] readDisable: enabled=true disable=true
   2018-09-12T10:09:48.77-0400 [PROXY/0] ERR [2018-09-12 14:09:48.769][78][debug][filter] source/common/tcp_proxy/tcp_proxy.cc:294] [C18] Creating connection to cluster 0-service-cluster
   2018-09-12T10:09:48.77-0400 [PROXY/0] ERR [2018-09-12 14:09:48.770][78][debug][pool] source/common/tcp/conn_pool.cc:75] [C8] using existing connection
   2018-09-12T10:09:48.77-0400 [PROXY/0] ERR [2018-09-12 14:09:48.770][78][trace][connection] source/common/network/connection_impl.cc:232] [C18] readDisable: enabled=false disable=false
   2018-09-12T10:09:48.77-0400 [PROXY/0] ERR [2018-09-12 14:09:48.770][78][debug][filter] source/common/tcp_proxy/tcp_proxy.cc:480] TCP:onUpstreamEvent(), requestedServerName:
   2018-09-12T10:09:48.77-0400 [PROXY/0] ERR [2018-09-12 14:09:48.770][78][debug][main] source/server/connection_handler_impl.cc:218] [C18] new connection
   2018-09-12T10:09:48.77-0400 [PROXY/0] ERR [2018-09-12 14:09:48.770][78][trace][connection] source/common/network/connection_impl.cc:389] [C18] socket event: 2
   2018-09-12T10:09:48.77-0400 [PROXY/0] ERR [2018-09-12 14:09:48.770][78][trace][connection] source/common/network/connection_impl.cc:457] [C18] write ready
   2018-09-12T10:09:48.77-0400 [PROXY/0] ERR [2018-09-12 14:09:48.771][78][debug][connection] source/common/ssl/ssl_socket.cc:131] [C18] handshake error: 2
   2018-09-12T10:09:48.77-0400 [PROXY/0] ERR [2018-09-12 14:09:48.771][78][trace][connection] source/common/network/connection_impl.cc:389] [C18] socket event: 3
   2018-09-12T10:09:48.77-0400 [PROXY/0] ERR [2018-09-12 14:09:48.772][78][trace][connection] source/common/network/connection_impl.cc:457] [C18] write ready
   2018-09-12T10:09:48.77-0400 [PROXY/0] ERR [2018-09-12 14:09:48.772][78][debug][connection] source/common/ssl/ssl_socket.cc:131] [C18] handshake error: 5
   2018-09-12T10:09:48.77-0400 [PROXY/0] ERR [2018-09-12 14:09:48.772][78][debug][connection] source/common/network/connection_impl.cc:133] [C18] closing socket: 0
   2018-09-12T10:09:48.77-0400 [PROXY/0] ERR [2018-09-12 14:09:48.772][78][debug][connection] source/common/network/connection_impl.cc:98] [C8] closing data_to_write=0 type=0
   2018-09-12T10:09:48.77-0400 [PROXY/0] ERR [2018-09-12 14:09:48.772][78][debug][connection] source/common/network/connection_impl.cc:133] [C8] closing socket: 1
   2018-09-12T10:09:48.77-0400 [PROXY/0] ERR [2018-09-12 14:09:48.772][78][debug][pool] source/common/tcp/conn_pool.cc:107] [C8] client disconnected
   2018-09-12T10:09:48.77-0400 [PROXY/0] ERR [2018-09-12 14:09:48.773][78][trace][main] source/common/event/dispatcher_impl.cc:126] item added to deferred deletion list (size=1)
   2018-09-12T10:09:48.77-0400 [PROXY/0] ERR [2018-09-12 14:09:48.773][78][debug][main] source/server/connection_handler_impl.cc:51] [C18] adding to cleanup list
   2018-09-12T10:09:48.77-0400 [PROXY/0] ERR [2018-09-12 14:09:48.773][78][trace][main] source/common/event/dispatcher_impl.cc:126] item added to deferred deletion list (size=2)
   2018-09-12T10:09:48.77-0400 [PROXY/0] ERR [2018-09-12 14:09:48.773][78][trace][main] source/common/event/dispatcher_impl.cc:52] clearing deferred deletion list (size=2)
   2018-09-12T10:09:48.77-0400 [PROXY/0] ERR [2018-09-12 14:09:48.773][78][debug][pool] source/common/tcp/conn_pool.cc:219] [C8] connection destroyed

tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      9/python
tcp        0      0 10.255.98.48:8080       10.255.98.48:47604      ESTABLISHED 9/python
tcp        0      0 10.255.98.48:33646      10.255.98.48:8080       ESTABLISHED -
tcp        0      0 10.255.98.48:48372      10.255.98.48:8080       FIN_WAIT2   -
tcp        0      0 10.255.98.48:8080       10.255.98.48:48010      ESTABLISHED -
tcp        0      0 10.255.98.48:8080       10.255.98.48:33646      ESTABLISHED -
tcp        0      0 10.255.98.48:8080       10.255.98.48:33780      ESTABLISHED -
tcp        0      0 10.255.98.48:48010      10.255.98.48:8080       ESTABLISHED -
tcp        1      0 10.255.98.48:8080       10.255.98.48:48716      CLOSE_WAIT  -
tcp        1      0 10.255.98.48:8080       10.255.98.48:57872      CLOSE_WAIT  -
tcp        0      0 10.255.98.48:47604      10.255.98.48:8080       ESTABLISHED -
tcp        0      0 10.255.98.48:33780      10.255.98.48:8080       ESTABLISHED -
tcp        1      0 10.255.98.48:8080       10.255.98.48:48372      CLOSE_WAIT  -

bug

Source

jvshahid

Most helpful comment

@zuercher We did some testing with your PR and the reproductions steps outlined in the repo from @rosenhouse. We found that upstream connections are now closed and in the "TIME_WAIT" state waiting to be cleaned up. We did not see any connections in the "ESTABLISHED" state.

@jvshahid @tylerschultz @rosenhouse We also did some testing with this PR and our own bosh-lite test environment, and found that the previously failing CATs test involving the python app is no longer failing.

@zuercher Thank you for making this PR. It solves our issue and we are looking forward to your PR getting merged.

n4wei on 12 Oct 2018

❤3

All 17 comments

For the tcp_proxy, the connection pool should always open a new connection to the upstream because Envoy has no knowledge of what protocol is being used and cannot safely reuse connections. This works correctly for the tcp_proxy when the downstream connection actually exchanges data with the upstream.

I believe what's happening here is that your first downstream connection is closing before the upstream connection is completed and the unused upstream connection ends up in the connection pool's list of ready connections. The second downstream connection is immediately assigned the ready upstream connection and when the downstream connection is closed, so is the upstream.

In the first case, the upstream connection should be terminated as we've no idea if it's safe to reuse it for another downstream connection. Fixing this is going to cause less connection reuse than there is now.

zuercher on 12 Sep 2018

To address the question of determinism, the connection pools are actually maintained per worker thread. So if the one downstream connection is handled by a different worker thread than another, you may see a new connection formed.

zuercher on 12 Sep 2018

For the tcp_proxy, the connection pool should always open a new connection to the upstream because Envoy has no knowledge of what protocol is being used and cannot safely reuse connections. This works correctly for the tcp_proxy when the downstream connection actually exchanges data with the upstream.

That makes sense.

I believe what's happening here is that your first downstream connection is closing before the upstream connection is completed and the unused upstream connection ends up in the connection pool's list of ready connections. The second downstream connection is immediately assigned the ready upstream connection and when the downstream connection is closed, so is the upstream.

That matches my understanding of the logs.

In the first case, the upstream connection should be terminated as we've no idea if it's safe to reuse it for another downstream connection. Fixing this is going to cause less connection reuse than there is now.

Why ? Since the connection hasn't been used yet, it is safe to reuse for future connections, what makes you think otherwise ?

jvshahid on 13 Sep 2018

Most protocols have the client write first, but that's not necessary. If the pending upstream connection sends data expectig there to be a client, Envoy's conn pool will close the connection with an error (and whatever data was sent will be lost).

zuercher on 13 Sep 2018

Do I understand correctly that this bug should remain open or a new one created with a more descriptive title to ensure that tcp connections are closed regardless of whether they were used or not by the downstream connection ?

jvshahid on 14 Sep 2018

We can leave this one open. I can edit the title.

zuercher on 15 Sep 2018

AIUI, the root of this issue is that the test app in question in a single-threaded, single connection python script? It seems that a response including closing the connection following a "Connection: close" response from this app would return the app to a listening state, and solve the problem of a keep-alive connection. I don't believe keep-alives are compatible with having an insufficient number of configured back-end connections. @jvshahid can you try terminating the connection when satisfying a single request, and see if the test cases pass?

wrowe on 2 Oct 2018

It seems that a response including closing the connection following a "Connection: close" response from this app would return the app to a listening state, and solve the problem of a keep-alive connection.

I think this is orthogonal to the issue reported here, I honestly don't know whether the application closes the connection after sending back a response or not. The issue that we originally reported on slack (and didn't do a good job explaining in this github issue) is a result of the following steps:

CF tcp healthcheck checking the envoy port (The tcp healthcheck only checks that something is listening on the given port.
The port check opens the port and immediately closes it. This short lived downstream connection causes Envoy to establish a connection to the upstream application, but doesn't send any data. Additionally, it doesn't close the upstream connection. Instead, the connection is added to a pool of unused connections.
New requests sometimes end up being sent on a new upstream connection. Those requests hang because the application is still waiting to serve a request from the first connection.

We originally opened this issue because Envoy behavior isn't deterministic. Sometimes Envoy uses a connection from the pool and sometimes it doesn't. Per our slack conversation, that isn't expected behavior. In addition, @zuercher pointed out in https://github.com/envoyproxy/envoy/issues/4409#issuecomment-421133011 that the TCP connection shouldn't be added to the pool in the first place, given that the client might have some state attached to the connection and reusing the upstream connection for a new downstream connection could lead to confusing results.

jvshahid on 5 Oct 2018

❤1

I've been playing with this issue in this repo.

When the Envoy listener is configured to terminate TLS, then I see lingering ESTABLISHED connections from Envoy to upstream and the occasional slow request. However when the listener is not doing TLS termination, then everything seems fine.

I didn't see this TLS bit mentioned anywhere else in this thread, just wanted to share it in case it is relevant.

rosenhouse on 8 Oct 2018

@zuercher are you still planning to work on this issue? It is blocking us, and we're considering trying to tackle it... but we haven't made any contributions to Envoy before, and aren't sure quite where to start.

rosenhouse on 8 Oct 2018

I intended to tackle this at some point. I'll try to take a look at it today.

zuercher on 9 Oct 2018

I think I have a working implementation of this fix. I'll post a PR tomorrow.

zuercher on 10 Oct 2018

@rosenhouse Is it possible for you to test with #4684 and see if it fixes your issue?

zuercher on 11 Oct 2018

I’m out for the next few days. Perhaps @tylerschultz could give it s whirl?

rosenhouse on 11 Oct 2018

We're working with @jvshahid's teammates this morning to try it out.