Envoy: questions about upstream connect error or disconnect/reset before headers%

Created on 26 May 2018 · 10Comments · Source: envoyproxy/envoy

Issue Template

Title: One line description

Description:

Describe the issue. Please be detailed. If a feature request, please
describe the desired behaviour, what scenario it enables and how it
would be used.
i want to run an example of envoy. and i typed

$ docker pull envoyproxy/envoy:latest
$ docker run --rm -d -p 10000:10000 envoyproxy/envoy:latest
$ curl -v localhost:10000

but there is an error of upstream connect error or disconnect/reset before headers%

[optional Relevant Links:]

Any extra documentation required to understand the issue.

Bug Template

Title: One line description

Description:

What issue is being seen? Describe what should be happening instead of
the bug, for example: Envoy should not crash, the expected value isn't
returned, etc.

Repro steps:

Include sample requests, environment, etc. All data and inputs
required to reproduce the bug.

i am using docker for mac 18.03.1-ce

Note: The Envoy_collect tool
gathers a tarball with debug logs, config and the following admin
endpoints: /stats, /clusters and /server_info. Please note if there are
privacy concerns, sanitize the data prior to sharing the tarball/pasting.

Admin and Stats Output:

Include the admin output for the following endpoints: /stats,
/clusters, /routes, /server_info. For more information, refer to the
admin endpoint documentation.
cluster.service_google.bind_errors: 0
cluster.service_google.external.upstream_rq_503: 8
cluster.service_google.external.upstream_rq_5xx: 8
cluster.service_google.lb_healthy_panic: 0
cluster.service_google.lb_local_cluster_not_ok: 0
cluster.service_google.lb_recalculate_zone_structures: 0
cluster.service_google.lb_subsets_active: 0
cluster.service_google.lb_subsets_created: 0
cluster.service_google.lb_subsets_fallback: 0
cluster.service_google.lb_subsets_removed: 0
cluster.service_google.lb_subsets_selected: 0
cluster.service_google.lb_zone_cluster_too_small: 0
cluster.service_google.lb_zone_no_capacity_left: 0
cluster.service_google.lb_zone_number_differs: 0
cluster.service_google.lb_zone_routing_all_directly: 0
cluster.service_google.lb_zone_routing_cross_zone: 0
cluster.service_google.lb_zone_routing_sampled: 0
cluster.service_google.max_host_weight: 0
cluster.service_google.membership_change: 1
cluster.service_google.membership_healthy: 1
cluster.service_google.membership_total: 1
cluster.service_google.retry_or_shadow_abandoned: 0
cluster.service_google.ssl.connection_error: 0
cluster.service_google.ssl.fail_verify_cert_hash: 0
cluster.service_google.ssl.fail_verify_error: 0
cluster.service_google.ssl.fail_verify_no_cert: 0
cluster.service_google.ssl.fail_verify_san: 0
cluster.service_google.ssl.handshake: 0
cluster.service_google.ssl.no_certificate: 0
cluster.service_google.ssl.session_reused: 0
cluster.service_google.update_attempt: 382
cluster.service_google.update_empty: 0
cluster.service_google.update_failure: 0
cluster.service_google.update_no_rebuild: 0
cluster.service_google.update_success: 382
cluster.service_google.upstream_cx_active: 0
cluster.service_google.upstream_cx_close_notify: 0
cluster.service_google.upstream_cx_connect_attempts_exceeded: 0
cluster.service_google.upstream_cx_connect_fail: 8
cluster.service_google.upstream_cx_connect_timeout: 7
cluster.service_google.upstream_cx_destroy: 0
cluster.service_google.upstream_cx_destroy_local: 0
cluster.service_google.upstream_cx_destroy_local_with_active_rq: 0
cluster.service_google.upstream_cx_destroy_remote: 0
cluster.service_google.upstream_cx_destroy_remote_with_active_rq: 0
cluster.service_google.upstream_cx_destroy_with_active_rq: 0
cluster.service_google.upstream_cx_http1_total: 8
cluster.service_google.upstream_cx_http2_total: 0
cluster.service_google.upstream_cx_idle_timeout: 0
cluster.service_google.upstream_cx_max_requests: 0
cluster.service_google.upstream_cx_none_healthy: 0
cluster.service_google.upstream_cx_overflow: 0
cluster.service_google.upstream_cx_protocol_error: 0
cluster.service_google.upstream_cx_rx_bytes_buffered: 0
cluster.service_google.upstream_cx_rx_bytes_total: 0
cluster.service_google.upstream_cx_total: 8
cluster.service_google.upstream_cx_tx_bytes_buffered: 0
cluster.service_google.upstream_cx_tx_bytes_total: 0
cluster.service_google.upstream_flow_control_backed_up_total: 0
cluster.service_google.upstream_flow_control_drained_total: 0
cluster.service_google.upstream_flow_control_paused_reading_total: 0
cluster.service_google.upstream_flow_control_resumed_reading_total: 0
cluster.service_google.upstream_rq_503: 8
cluster.service_google.upstream_rq_5xx: 8
cluster.service_google.upstream_rq_active: 0
cluster.service_google.upstream_rq_cancelled: 0
cluster.service_google.upstream_rq_maintenance_mode: 0
cluster.service_google.upstream_rq_pending_active: 0
cluster.service_google.upstream_rq_pending_failure_eject: 8
cluster.service_google.upstream_rq_pending_overflow: 0
cluster.service_google.upstream_rq_pending_total: 8
cluster.service_google.upstream_rq_per_try_timeout: 0
cluster.service_google.upstream_rq_retry: 0
cluster.service_google.upstream_rq_retry_overflow: 0
cluster.service_google.upstream_rq_retry_success: 0
cluster.service_google.upstream_rq_rx_reset: 0
cluster.service_google.upstream_rq_timeout: 0
cluster.service_google.upstream_rq_total: 0
cluster.service_google.upstream_rq_tx_reset: 0
cluster.service_google.version: 0
cluster_manager.active_clusters: 1
cluster_manager.cluster_added: 1
cluster_manager.cluster_modified: 0
cluster_manager.cluster_removed: 0
cluster_manager.warming_clusters: 0
filesystem.flushed_by_timer: 0
filesystem.reopen_failed: 0
filesystem.write_buffered: 0
filesystem.write_completed: 0
filesystem.write_total_buffered: 0
http.admin.downstream_cx_active: 1
http.admin.downstream_cx_destroy: 0
http.admin.downstream_cx_destroy_active_rq: 0
http.admin.downstream_cx_destroy_local: 0
http.admin.downstream_cx_destroy_local_active_rq: 0
http.admin.downstream_cx_destroy_remote: 0
http.admin.downstream_cx_destroy_remote_active_rq: 0
http.admin.downstream_cx_drain_close: 0
http.admin.downstream_cx_http1_active: 1
http.admin.downstream_cx_http1_total: 1
http.admin.downstream_cx_http2_active: 0
http.admin.downstream_cx_http2_total: 0
http.admin.downstream_cx_idle_timeout: 0
http.admin.downstream_cx_protocol_error: 0
http.admin.downstream_cx_rx_bytes_buffered: 83
http.admin.downstream_cx_rx_bytes_total: 83
http.admin.downstream_cx_ssl_active: 0
http.admin.downstream_cx_ssl_total: 0
http.admin.downstream_cx_total: 1
http.admin.downstream_cx_tx_bytes_buffered: 0
http.admin.downstream_cx_tx_bytes_total: 0
http.admin.downstream_cx_websocket_active: 0
http.admin.downstream_cx_websocket_total: 0
http.admin.downstream_flow_control_paused_reading_total: 0
http.admin.downstream_flow_control_resumed_reading_total: 0
http.admin.downstream_rq_1xx: 0
http.admin.downstream_rq_2xx: 0
http.admin.downstream_rq_3xx: 0
http.admin.downstream_rq_4xx: 0
http.admin.downstream_rq_5xx: 0
http.admin.downstream_rq_active: 1
http.admin.downstream_rq_http1_total: 1
http.admin.downstream_rq_http2_total: 0
http.admin.downstream_rq_non_relative_path: 0
http.admin.downstream_rq_response_before_rq_complete: 0
http.admin.downstream_rq_rx_reset: 0
http.admin.downstream_rq_too_large: 0
http.admin.downstream_rq_total: 1
http.admin.downstream_rq_tx_reset: 0
http.admin.downstream_rq_ws_on_non_ws_route: 0
http.admin.rs_too_large: 0
http.async-client.no_cluster: 0
http.async-client.no_route: 0
http.async-client.rq_direct_response: 0
http.async-client.rq_redirect: 0
http.async-client.rq_total: 0
http.ingress_http.downstream_cx_active: 0
http.ingress_http.downstream_cx_destroy: 8
http.ingress_http.downstream_cx_destroy_active_rq: 0
http.ingress_http.downstream_cx_destroy_local: 0
http.ingress_http.downstream_cx_destroy_local_active_rq: 0
http.ingress_http.downstream_cx_destroy_remote: 8
http.ingress_http.downstream_cx_destroy_remote_active_rq: 0
http.ingress_http.downstream_cx_drain_close: 0
http.ingress_http.downstream_cx_http1_active: 0
http.ingress_http.downstream_cx_http1_total: 8
http.ingress_http.downstream_cx_http2_active: 0
http.ingress_http.downstream_cx_http2_total: 0
http.ingress_http.downstream_cx_idle_timeout: 0
http.ingress_http.downstream_cx_protocol_error: 0
http.ingress_http.downstream_cx_rx_bytes_buffered: 0
http.ingress_http.downstream_cx_rx_bytes_total: 636
http.ingress_http.downstream_cx_ssl_active: 0
http.ingress_http.downstream_cx_ssl_total: 0
http.ingress_http.downstream_cx_total: 8
http.ingress_http.downstream_cx_tx_bytes_buffered: 0
http.ingress_http.downstream_cx_tx_bytes_total: 1528
http.ingress_http.downstream_cx_websocket_active: 0
http.ingress_http.downstream_cx_websocket_total: 0
http.ingress_http.downstream_flow_control_paused_reading_total: 0
http.ingress_http.downstream_flow_control_resumed_reading_total: 0
http.ingress_http.downstream_rq_1xx: 0
http.ingress_http.downstream_rq_2xx: 0
http.ingress_http.downstream_rq_3xx: 0
http.ingress_http.downstream_rq_4xx: 0
http.ingress_http.downstream_rq_5xx: 8
http.ingress_http.downstream_rq_active: 0
http.ingress_http.downstream_rq_http1_total: 8
http.ingress_http.downstream_rq_http2_total: 0
http.ingress_http.downstream_rq_non_relative_path: 0
http.ingress_http.downstream_rq_response_before_rq_complete: 0
http.ingress_http.downstream_rq_rx_reset: 0
http.ingress_http.downstream_rq_too_large: 0
http.ingress_http.downstream_rq_total: 8
http.ingress_http.downstream_rq_tx_reset: 0
http.ingress_http.downstream_rq_ws_on_non_ws_route: 0
http.ingress_http.no_cluster: 0
http.ingress_http.no_route: 0
http.ingress_http.rq_direct_response: 0
http.ingress_http.rq_redirect: 0
http.ingress_http.rq_total: 8
http.ingress_http.rs_too_large: 0
http.ingress_http.tracing.client_enabled: 0
http.ingress_http.tracing.health_check: 0
http.ingress_http.tracing.not_traceable: 0
http.ingress_http.tracing.random_sampling: 0
http.ingress_http.tracing.service_forced: 0
listener.0.0.0.0_10000.downstream_cx_active: 0
listener.0.0.0.0_10000.downstream_cx_destroy: 8
listener.0.0.0.0_10000.downstream_cx_total: 8
listener.0.0.0.0_10000.http.ingress_http.downstream_rq_1xx: 0
listener.0.0.0.0_10000.http.ingress_http.downstream_rq_2xx: 0
listener.0.0.0.0_10000.http.ingress_http.downstream_rq_3xx: 0
listener.0.0.0.0_10000.http.ingress_http.downstream_rq_4xx: 0
listener.0.0.0.0_10000.http.ingress_http.downstream_rq_5xx: 8
listener.0.0.0.0_10000.no_filter_chain_match: 0
listener.admin.downstream_cx_active: 1
listener.admin.downstream_cx_destroy: 0
listener.admin.downstream_cx_total: 1
listener.admin.http.admin.downstream_rq_1xx: 0
listener.admin.http.admin.downstream_rq_2xx: 0
listener.admin.http.admin.downstream_rq_3xx: 0
listener.admin.http.admin.downstream_rq_4xx: 0
listener.admin.http.admin.downstream_rq_5xx: 0
listener.admin.no_filter_chain_match: 0
listener_manager.listener_added: 1
listener_manager.listener_create_failure: 0
listener_manager.listener_create_success: 4
listener_manager.listener_modified: 0
listener_manager.listener_removed: 0
listener_manager.total_listeners_active: 1
listener_manager.total_listeners_draining: 0
listener_manager.total_listeners_warming: 0
runtime.admin_overrides_active: 0
runtime.load_error: 0
runtime.load_success: 0
runtime.num_keys: 0
runtime.override_dir_exists: 0
runtime.override_dir_not_exists: 0
server.days_until_first_cert_expiring: 2147483647
server.live: 1
server.memory_allocated: 3310240
server.memory_heap_size: 4194304
server.parent_connections: 0
server.total_connections: 0
server.uptime: 1905
server.version: 3029724
server.watchdog_mega_miss: 0
server.watchdog_miss: 0
stats.overflow: 0
cluster.service_google.upstream_cx_connect_ms: No recorded values
cluster.service_google.upstream_cx_length_ms: P0(nan,0) P25(nan,243.333) P50(nan,250) P75(nan,255) P90(nan,258) P95(nan,259) P99(nan,259.8) P99.9(nan,259.98) P100(nan,260)
http.admin.downstream_cx_length_ms: No recorded values
http.admin.downstream_rq_time: No recorded values
http.ingress_http.downstream_cx_length_ms: P0(nan,3) P25(nan,243.333) P50(nan,250) P75(nan,255) P90(nan,258) P95(nan,259) P99(nan,259.8) P99.9(nan,259.98) P100(nan,260)
http.ingress_http.downstream_rq_time: P0(nan,2) P25(nan,243.333) P50(nan,250) P75(nan,255) P90(nan,258) P95(nan,259) P99(nan,259.8) P99.9(nan,259.98) P100(nan,260)
listener.0.0.0.0_10000.downstream_cx_length_ms: P0(nan,3) P25(nan,243.333) P50(nan,250) P75(nan,255) P90(nan,258) P95(nan,259) P99(nan,259.8) P99.9(nan,259.98) P100(nan,260)
listener.admin.downstream_cx_length_ms: No recorded values

Note: If there are privacy concerns, sanitize the data prior to
sharing.

Config:

Include the config used to configure Envoy.

Logs:

Include the access logs and the Envoy logs.

Note: If there are privacy concerns, sanitize the data prior to
sharing.

Call Stack:

If the Envoy binary is crashing, a call stack is required.
Please refer to the Bazel Stack trace documentation.

question

Source

corn1ng

Most helpful comment

I also experienced similar behaviour when running the standard demo Docker image on Mac OS X. It seemed to happen for any HTTPS upstream. If I manually ran network calls from within the container the requests seem to work okay.

It turns out that in my case the timeouts were too aggressive for the network connection I had. Things were fine with HTTP but the TLS handshake made connecting take too long.

jeteon on 24 Jun 2018

👍9 🎉8

All 10 comments

Hey @corn1ng
I wasn't able to reproduce this myself.

Can you try checking that you're able to connect to google from your container?

For example:

docker run --rm -ti envoyproxy/envoy bash
apt-get update && apt-get install -y curl
curl -IL https://google.com

You should see a 200 OK response.

taion809 on 31 May 2018

Hey @taion809 thanks for your watching,
i am in china so i cant connect to google directly you know .
so i modify Google to baidu . but i also get the 503

corn1ng on 1 Jun 2018

❤4

This sounds like a docker issue to me. I would take a look at your docker networking. Going to go ahead and close this. Reopen if you have further questions.

mattklein123 on 7 Jun 2018

It turns out that in my case the timeouts were too aggressive for the network connection I had. Things were fine with HTTP but the TLS handshake made connecting take too long.

jeteon on 24 Jun 2018

👍9 🎉8

Same issue with me on mac os x. Is there any progress with this issue?

butters-mars on 23 Jul 2018

@taion809

I followed these steps and I am able to get 200 response.

docker run --rm -ti envoyproxy/envoy bash
apt-get update && apt-get install -y curl
curl -IL https://google.com

When I run through the envoy proxy I get upstream error.

vpiduri on 31 Aug 2018

I faced the same problem. I changed the 'connect_timeout' value to a higher value(15s) as per @jeteon's suggestion and it worked for me.

Working yaml for me

admin:
  access_log_path: /tmp/admin_access.log
  address:
    socket_address:
      protocol: TCP
      address: 127.0.0.1
      port_value: 9901
static_resources:
  listeners:
  - name: listener_0
    address:
      socket_address:
        protocol: TCP
        address: 0.0.0.0
        port_value: 10000
    filter_chains:
    - filters:
      - name: envoy.http_connection_manager
        config:
          stat_prefix: ingress_http
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains: ["*"]
              routes:
              - match:
                  prefix: "/"
                route:
                  host_rewrite: www.google.com
                  cluster: service_google
          http_filters:
          - name: envoy.router
  clusters:
  - name: service_google
    connect_timeout: 15.0s
    type: LOGICAL_DNS
    # Comment out the following line to test on v6 networks
    dns_lookup_family: V4_ONLY
    lb_policy: ROUND_ROBIN
    hosts:
      - socket_address:
          address: google.com
          port_value: 443
    tls_context: { sni: www.google.com }

mefarazath on 10 Sep 2018

👍6

I get the same issue: upstream connect error or disconnect/reset before headers

curl to google from container works, 200 OK return

atarlov on 1 Feb 2019

👍1

I got the same issue, is there a solution to this issue.
curl to google or upstream service return 200. Here is complete error:-
upstream connect error or disconnect/reset before headers. reset reason: connection failure
Output of stats command
access_log_file.flushed_by_timer: 2149
access_log_file.reopen_failed: 0
access_log_file.write_buffered: 21
access_log_file.write_completed: 16
access_log_file.write_total_buffered: 223
cluster.eds_cluster.bind_errors: 0
cluster.eds_cluster.circuit_breakers.default.cx_open: 0
cluster.eds_cluster.circuit_breakers.default.cx_pool_open: 0
cluster.eds_cluster.circuit_breakers.default.rq_open: 0
cluster.eds_cluster.circuit_breakers.default.rq_pending_open: 0
cluster.eds_cluster.circuit_breakers.default.rq_retry_open: 0
cluster.eds_cluster.circuit_breakers.high.cx_open: 0
cluster.eds_cluster.circuit_breakers.high.cx_pool_open: 0
cluster.eds_cluster.circuit_breakers.high.rq_open: 0
cluster.eds_cluster.circuit_breakers.high.rq_pending_open: 0
cluster.eds_cluster.circuit_breakers.high.rq_retry_open: 0
cluster.eds_cluster.internal.upstream_rq_200: 5684
cluster.eds_cluster.internal.upstream_rq_2xx: 5684
cluster.eds_cluster.internal.upstream_rq_500: 63
cluster.eds_cluster.internal.upstream_rq_503: 26
cluster.eds_cluster.internal.upstream_rq_504: 4
cluster.eds_cluster.internal.upstream_rq_5xx: 93
cluster.eds_cluster.internal.upstream_rq_completed: 5777
cluster.eds_cluster.lb_healthy_panic: 0
cluster.eds_cluster.lb_local_cluster_not_ok: 0
cluster.eds_cluster.lb_recalculate_zone_structures: 0
cluster.eds_cluster.lb_subsets_active: 0
cluster.eds_cluster.lb_subsets_created: 0
cluster.eds_cluster.lb_subsets_fallback: 0
cluster.eds_cluster.lb_subsets_fallback_panic: 0
cluster.eds_cluster.lb_subsets_removed: 0
cluster.eds_cluster.lb_subsets_selected: 0
cluster.eds_cluster.lb_zone_cluster_too_small: 0
cluster.eds_cluster.lb_zone_no_capacity_left: 0
cluster.eds_cluster.lb_zone_number_differs: 0
cluster.eds_cluster.lb_zone_routing_all_directly: 0
cluster.eds_cluster.lb_zone_routing_cross_zone: 0
cluster.eds_cluster.lb_zone_routing_sampled: 0
cluster.eds_cluster.max_host_weight: 1
cluster.eds_cluster.membership_change: 1
cluster.eds_cluster.membership_degraded: 0
cluster.eds_cluster.membership_healthy: 1
cluster.eds_cluster.membership_total: 1
cluster.eds_cluster.original_dst_host_invalid: 0
cluster.eds_cluster.retry_or_shadow_abandoned: 0
cluster.eds_cluster.update_attempt: 4313
cluster.eds_cluster.update_empty: 0
cluster.eds_cluster.update_failure: 0
cluster.eds_cluster.update_no_rebuild: 4312
cluster.eds_cluster.update_success: 4313
cluster.eds_cluster.upstream_cx_active: 1
cluster.eds_cluster.upstream_cx_close_notify: 95
cluster.eds_cluster.upstream_cx_connect_attempts_exceeded: 0
cluster.eds_cluster.upstream_cx_connect_fail: 9
cluster.eds_cluster.upstream_cx_connect_timeout: 4
cluster.eds_cluster.upstream_cx_destroy: 0
cluster.eds_cluster.upstream_cx_destroy_local: 0
cluster.eds_cluster.upstream_cx_destroy_local_with_active_rq: 4
cluster.eds_cluster.upstream_cx_destroy_remote: 0
cluster.eds_cluster.upstream_cx_destroy_remote_with_active_rq: 17
cluster.eds_cluster.upstream_cx_destroy_with_active_rq: 21
cluster.eds_cluster.upstream_cx_http1_total: 323
cluster.eds_cluster.upstream_cx_http2_total: 0
cluster.eds_cluster.upstream_cx_idle_timeout: 0
cluster.eds_cluster.upstream_cx_max_requests: 0
cluster.eds_cluster.upstream_cx_none_healthy: 0
cluster.eds_cluster.upstream_cx_overflow: 0
cluster.eds_cluster.upstream_cx_pool_overflow: 0
cluster.eds_cluster.upstream_cx_protocol_error: 0
cluster.eds_cluster.upstream_cx_rx_bytes_buffered: 5
cluster.eds_cluster.upstream_cx_rx_bytes_total: 2844939
cluster.eds_cluster.upstream_cx_total: 323
cluster.eds_cluster.upstream_cx_tx_bytes_buffered: 0
cluster.eds_cluster.upstream_cx_tx_bytes_total: 2404647
cluster.eds_cluster.upstream_flow_control_backed_up_total: 0
cluster.eds_cluster.upstream_flow_control_drained_total: 0
cluster.eds_cluster.upstream_flow_control_paused_reading_total: 0
cluster.eds_cluster.upstream_flow_control_resumed_reading_total: 0
cluster.eds_cluster.upstream_internal_redirect_failed_total: 0
cluster.eds_cluster.upstream_internal_redirect_succeeded_total: 0
cluster.eds_cluster.upstream_rq_200: 5684
cluster.eds_cluster.upstream_rq_2xx: 5684
cluster.eds_cluster.upstream_rq_500: 63
cluster.eds_cluster.upstream_rq_503: 26
cluster.eds_cluster.upstream_rq_504: 4
cluster.eds_cluster.upstream_rq_5xx: 93
cluster.eds_cluster.upstream_rq_active: 0
cluster.eds_cluster.upstream_rq_cancelled: 0
cluster.eds_cluster.upstream_rq_completed: 5777
cluster.eds_cluster.upstream_rq_maintenance_mode: 0
cluster.eds_cluster.upstream_rq_pending_active: 0
cluster.eds_cluster.upstream_rq_pending_failure_eject: 9
cluster.eds_cluster.upstream_rq_pending_overflow: 0
cluster.eds_cluster.upstream_rq_pending_total: 323
cluster.eds_cluster.upstream_rq_per_try_timeout: 0
cluster.eds_cluster.upstream_rq_retry: 0
cluster.eds_cluster.upstream_rq_retry_overflow: 0
cluster.eds_cluster.upstream_rq_retry_success: 0
cluster.eds_cluster.upstream_rq_rx_reset: 0
cluster.eds_cluster.upstream_rq_timeout: 4
cluster.eds_cluster.upstream_rq_total: 5768
cluster.eds_cluster.upstream_rq_tx_reset: 0
cluster.eds_cluster.version: 0
cluster.service1.bind_errors: 0
cluster.service1.circuit_breakers.default.cx_open: 0
cluster.service1.circuit_breakers.default.cx_pool_open: 0
cluster.service1.circuit_breakers.default.rq_open: 0
cluster.service1.circuit_breakers.default.rq_pending_open: 0
cluster.service1.circuit_breakers.default.rq_retry_open: 0
cluster.service1.circuit_breakers.high.cx_open: 0
cluster.service1.circuit_breakers.high.cx_pool_open: 0
cluster.service1.circuit_breakers.high.rq_open: 0
cluster.service1.circuit_breakers.high.rq_pending_open: 0
cluster.service1.circuit_breakers.high.rq_retry_open: 0
cluster.service1.external.upstream_rq_503: 19
cluster.service1.external.upstream_rq_5xx: 19
cluster.service1.external.upstream_rq_completed: 19
cluster.service1.http2.header_overflow: 0
cluster.service1.http2.headers_cb_no_stream: 0
cluster.service1.http2.rx_messaging_error: 0
cluster.service1.http2.rx_reset: 0
cluster.service1.http2.too_many_header_frames: 0
cluster.service1.http2.trailers: 0
cluster.service1.http2.tx_reset: 0
cluster.service1.lb_healthy_panic: 6
cluster.service1.lb_local_cluster_not_ok: 0
cluster.service1.lb_recalculate_zone_structures: 0
cluster.service1.lb_subsets_active: 0
cluster.service1.lb_subsets_created: 0
cluster.service1.lb_subsets_fallback: 0
cluster.service1.lb_subsets_fallback_panic: 0
cluster.service1.lb_subsets_removed: 0
cluster.service1.lb_subsets_selected: 0
cluster.service1.lb_zone_cluster_too_small: 0
cluster.service1.lb_zone_no_capacity_left: 0
cluster.service1.lb_zone_number_differs: 0
cluster.service1.lb_zone_routing_all_directly: 0
cluster.service1.lb_zone_routing_cross_zone: 0
cluster.service1.lb_zone_routing_sampled: 0
cluster.service1.max_host_weight: 1
cluster.service1.membership_change: 1
cluster.service1.membership_degraded: 0
cluster.service1.membership_healthy: 1
cluster.service1.membership_total: 1
cluster.service1.original_dst_host_invalid: 0
cluster.service1.retry_or_shadow_abandoned: 0
cluster.service1.update_attempt: 2888
cluster.service1.update_empty: 0
cluster.service1.update_failure: 36
cluster.service1.update_no_rebuild: 165
cluster.service1.update_rejected: 2686
cluster.service1.update_success: 166
cluster.service1.upstream_cx_active: 0
cluster.service1.upstream_cx_close_notify: 0
cluster.service1.upstream_cx_connect_attempts_exceeded: 0
cluster.service1.upstream_cx_connect_fail: 9
cluster.service1.upstream_cx_connect_timeout: 7
cluster.service1.upstream_cx_destroy: 0
cluster.service1.upstream_cx_destroy_local: 0
cluster.service1.upstream_cx_destroy_local_with_active_rq: 0
cluster.service1.upstream_cx_destroy_remote: 0
cluster.service1.upstream_cx_destroy_remote_with_active_rq: 4
cluster.service1.upstream_cx_destroy_with_active_rq: 4
cluster.service1.upstream_cx_http1_total: 0
cluster.service1.upstream_cx_http2_total: 13
cluster.service1.upstream_cx_idle_timeout: 0
cluster.service1.upstream_cx_max_requests: 0
cluster.service1.upstream_cx_none_healthy: 6
cluster.service1.upstream_cx_overflow: 0
cluster.service1.upstream_cx_pool_overflow: 0
cluster.service1.upstream_cx_protocol_error: 0
cluster.service1.upstream_cx_rx_bytes_buffered: 0
cluster.service1.upstream_cx_rx_bytes_total: 0
cluster.service1.upstream_cx_total: 13
cluster.service1.upstream_cx_tx_bytes_buffered: 0
cluster.service1.upstream_cx_tx_bytes_total: 1377
cluster.service1.upstream_flow_control_backed_up_total: 0
cluster.service1.upstream_flow_control_drained_total: 0
cluster.service1.upstream_flow_control_paused_reading_total: 0
cluster.service1.upstream_flow_control_resumed_reading_total: 0
cluster.service1.upstream_internal_redirect_failed_total: 0
cluster.service1.upstream_internal_redirect_succeeded_total: 0
cluster.service1.upstream_rq_503: 19
cluster.service1.upstream_rq_5xx: 19
cluster.service1.upstream_rq_active: 0
cluster.service1.upstream_rq_cancelled: 0
cluster.service1.upstream_rq_completed: 19
cluster.service1.upstream_rq_maintenance_mode: 0
cluster.service1.upstream_rq_pending_active: 0
cluster.service1.upstream_rq_pending_failure_eject: 13
cluster.service1.upstream_rq_pending_overflow: 0
cluster.service1.upstream_rq_pending_total: 13
cluster.service1.upstream_rq_per_try_timeout: 0
cluster.service1.upstream_rq_retry: 0
cluster.service1.upstream_rq_retry_overflow: 0
cluster.service1.upstream_rq_retry_success: 0
cluster.service1.upstream_rq_rx_reset: 0
cluster.service1.upstream_rq_timeout: 0
cluster.service1.upstream_rq_total: 4
cluster.service1.upstream_rq_tx_reset: 0
cluster.service1.version: 3796596507944515212
cluster.service2.bind_errors: 0
cluster.service2.circuit_breakers.default.cx_open: 0
cluster.service2.circuit_breakers.default.cx_pool_open: 0
cluster.service2.circuit_breakers.default.rq_open: 0
cluster.service2.circuit_breakers.default.rq_pending_open: 0
cluster.service2.circuit_breakers.default.rq_retry_open: 0
cluster.service2.circuit_breakers.high.cx_open: 0
cluster.service2.circuit_breakers.high.cx_pool_open: 0
cluster.service2.circuit_breakers.high.rq_open: 0
cluster.service2.circuit_breakers.high.rq_pending_open: 0
cluster.service2.circuit_breakers.high.rq_retry_open: 0
cluster.service2.external.upstream_rq_503: 9
cluster.service2.external.upstream_rq_5xx: 9
cluster.service2.external.upstream_rq_completed: 9
cluster.service2.http2.header_overflow: 0
cluster.service2.http2.headers_cb_no_stream: 0
cluster.service2.http2.rx_messaging_error: 0
cluster.service2.http2.rx_reset: 0
cluster.service2.http2.too_many_header_frames: 0
cluster.service2.http2.trailers: 0
cluster.service2.http2.tx_reset: 0
cluster.service2.lb_healthy_panic: 0
cluster.service2.lb_local_cluster_not_ok: 0
cluster.service2.lb_recalculate_zone_structures: 0
cluster.service2.lb_subsets_active: 0
cluster.service2.lb_subsets_created: 0
cluster.service2.lb_subsets_fallback: 0
cluster.service2.lb_subsets_fallback_panic: 0
cluster.service2.lb_subsets_removed: 0
cluster.service2.lb_subsets_selected: 0
cluster.service2.lb_zone_cluster_too_small: 0
cluster.service2.lb_zone_no_capacity_left: 0
cluster.service2.lb_zone_number_differs: 0
cluster.service2.lb_zone_routing_all_directly: 0
cluster.service2.lb_zone_routing_cross_zone: 0
cluster.service2.lb_zone_routing_sampled: 0
cluster.service2.max_host_weight: 1
cluster.service2.membership_change: 1
cluster.service2.membership_degraded: 0
cluster.service2.membership_healthy: 1
cluster.service2.membership_total: 1
cluster.service2.original_dst_host_invalid: 0
cluster.service2.retry_or_shadow_abandoned: 0
cluster.service2.update_attempt: 2889
cluster.service2.update_empty: 0
cluster.service2.update_failure: 57
cluster.service2.update_no_rebuild: 157
cluster.service2.update_rejected: 2674
cluster.service2.update_success: 158
cluster.service2.upstream_cx_active: 0
cluster.service2.upstream_cx_close_notify: 0
cluster.service2.upstream_cx_connect_attempts_exceeded: 0
cluster.service2.upstream_cx_connect_fail: 9
cluster.service2.upstream_cx_connect_timeout: 8
cluster.service2.upstream_cx_destroy: 0
cluster.service2.upstream_cx_destroy_local: 0
cluster.service2.upstream_cx_destroy_local_with_active_rq: 0
cluster.service2.upstream_cx_destroy_remote: 0
cluster.service2.upstream_cx_destroy_remote_with_active_rq: 0
cluster.service2.upstream_cx_destroy_with_active_rq: 0
cluster.service2.upstream_cx_http1_total: 0
cluster.service2.upstream_cx_http2_total: 9
cluster.service2.upstream_cx_idle_timeout: 0
cluster.service2.upstream_cx_max_requests: 0
cluster.service2.upstream_cx_none_healthy: 0
cluster.service2.upstream_cx_overflow: 0
cluster.service2.upstream_cx_pool_overflow: 0
cluster.service2.upstream_cx_protocol_error: 0
cluster.service2.upstream_cx_rx_bytes_buffered: 0
cluster.service2.upstream_cx_rx_bytes_total: 0
cluster.service2.upstream_cx_total: 9
cluster.service2.upstream_cx_tx_bytes_buffered: 0
cluster.service2.upstream_cx_tx_bytes_total: 0
cluster.service2.upstream_flow_control_backed_up_total: 0
cluster.service2.upstream_flow_control_drained_total: 0
cluster.service2.upstream_flow_control_paused_reading_total: 0
cluster.service2.upstream_flow_control_resumed_reading_total: 0
cluster.service2.upstream_internal_redirect_failed_total: 0
cluster.service2.upstream_internal_redirect_succeeded_total: 0
cluster.service2.upstream_rq_503: 9
cluster.service2.upstream_rq_5xx: 9
cluster.service2.upstream_rq_active: 0
cluster.service2.upstream_rq_cancelled: 0
cluster.service2.upstream_rq_completed: 9
cluster.service2.upstream_rq_maintenance_mode: 0
cluster.service2.upstream_rq_pending_active: 0
cluster.service2.upstream_rq_pending_failure_eject: 9
cluster.service2.upstream_rq_pending_overflow: 0
cluster.service2.upstream_rq_pending_total: 9
cluster.service2.upstream_rq_per_try_timeout: 0
cluster.service2.upstream_rq_retry: 0
cluster.service2.upstream_rq_retry_overflow: 0
cluster.service2.upstream_rq_retry_success: 0
cluster.service2.upstream_rq_rx_reset: 0
cluster.service2.upstream_rq_timeout: 0
cluster.service2.upstream_rq_total: 0
cluster.service2.upstream_rq_tx_reset: 0
cluster.service2.version: 3796596507944515212
cluster.zipkin.bind_errors: 0
cluster.zipkin.circuit_breakers.default.cx_open: 0
cluster.zipkin.circuit_breakers.default.cx_pool_open: 0
cluster.zipkin.circuit_breakers.default.rq_open: 0
cluster.zipkin.circuit_breakers.default.rq_pending_open: 0
cluster.zipkin.circuit_breakers.default.rq_retry_open: 0
cluster.zipkin.circuit_breakers.high.cx_open: 0
cluster.zipkin.circuit_breakers.high.cx_pool_open: 0
cluster.zipkin.circuit_breakers.high.rq_open: 0
cluster.zipkin.circuit_breakers.high.rq_pending_open: 0
cluster.zipkin.circuit_breakers.high.rq_retry_open: 0
cluster.zipkin.internal.upstream_rq_202: 25
cluster.zipkin.internal.upstream_rq_2xx: 25
cluster.zipkin.internal.upstream_rq_completed: 25
cluster.zipkin.lb_healthy_panic: 0
cluster.zipkin.lb_local_cluster_not_ok: 0
cluster.zipkin.lb_recalculate_zone_structures: 0
cluster.zipkin.lb_subsets_active: 0
cluster.zipkin.lb_subsets_created: 0
cluster.zipkin.lb_subsets_fallback: 0
cluster.zipkin.lb_subsets_fallback_panic: 0
cluster.zipkin.lb_subsets_removed: 0
cluster.zipkin.lb_subsets_selected: 0
cluster.zipkin.lb_zone_cluster_too_small: 0
cluster.zipkin.lb_zone_no_capacity_left: 0
cluster.zipkin.lb_zone_number_differs: 0
cluster.zipkin.lb_zone_routing_all_directly: 0
cluster.zipkin.lb_zone_routing_cross_zone: 0
cluster.zipkin.lb_zone_routing_sampled: 0
cluster.zipkin.max_host_weight: 1
cluster.zipkin.membership_change: 1
cluster.zipkin.membership_degraded: 0
cluster.zipkin.membership_healthy: 1
cluster.zipkin.membership_total: 1
cluster.zipkin.original_dst_host_invalid: 0
cluster.zipkin.retry_or_shadow_abandoned: 0
cluster.zipkin.update_attempt: 4313
cluster.zipkin.update_empty: 0
cluster.zipkin.update_failure: 0
cluster.zipkin.update_no_rebuild: 4312
cluster.zipkin.update_success: 4313
cluster.zipkin.upstream_cx_active: 1
cluster.zipkin.upstream_cx_close_notify: 0
cluster.zipkin.upstream_cx_connect_attempts_exceeded: 0
cluster.zipkin.upstream_cx_connect_fail: 0
cluster.zipkin.upstream_cx_connect_timeout: 0
cluster.zipkin.upstream_cx_destroy: 0
cluster.zipkin.upstream_cx_destroy_local: 0
cluster.zipkin.upstream_cx_destroy_local_with_active_rq: 0
cluster.zipkin.upstream_cx_destroy_remote: 0
cluster.zipkin.upstream_cx_destroy_remote_with_active_rq: 0
cluster.zipkin.upstream_cx_destroy_with_active_rq: 0
cluster.zipkin.upstream_cx_http1_total: 16
cluster.zipkin.upstream_cx_http2_total: 0
cluster.zipkin.upstream_cx_idle_timeout: 0
cluster.zipkin.upstream_cx_max_requests: 0
cluster.zipkin.upstream_cx_none_healthy: 0
cluster.zipkin.upstream_cx_overflow: 0
cluster.zipkin.upstream_cx_pool_overflow: 0
cluster.zipkin.upstream_cx_protocol_error: 0
cluster.zipkin.upstream_cx_rx_bytes_buffered: 12
cluster.zipkin.upstream_cx_rx_bytes_total: 2450
cluster.zipkin.upstream_cx_total: 16
cluster.zipkin.upstream_cx_tx_bytes_buffered: 0
cluster.zipkin.upstream_cx_tx_bytes_total: 35104
cluster.zipkin.upstream_flow_control_backed_up_total: 0
cluster.zipkin.upstream_flow_control_drained_total: 0
cluster.zipkin.upstream_flow_control_paused_reading_total: 0
cluster.zipkin.upstream_flow_control_resumed_reading_total: 0
cluster.zipkin.upstream_internal_redirect_failed_total: 0
cluster.zipkin.upstream_internal_redirect_succeeded_total: 0
cluster.zipkin.upstream_rq_202: 25
cluster.zipkin.upstream_rq_2xx: 25
cluster.zipkin.upstream_rq_active: 0
cluster.zipkin.upstream_rq_cancelled: 0
cluster.zipkin.upstream_rq_completed: 25
cluster.zipkin.upstream_rq_maintenance_mode: 0
cluster.zipkin.upstream_rq_pending_active: 0
cluster.zipkin.upstream_rq_pending_failure_eject: 0
cluster.zipkin.upstream_rq_pending_overflow: 0
cluster.zipkin.upstream_rq_pending_total: 16
cluster.zipkin.upstream_rq_per_try_timeout: 0
cluster.zipkin.upstream_rq_retry: 0
cluster.zipkin.upstream_rq_retry_overflow: 0
cluster.zipkin.upstream_rq_retry_success: 0
cluster.zipkin.upstream_rq_rx_reset: 0
cluster.zipkin.upstream_rq_timeout: 0
cluster.zipkin.upstream_rq_total: 25
cluster.zipkin.upstream_rq_tx_reset: 0
cluster.zipkin.version: 0
cluster_manager.active_clusters: 4
cluster_manager.cluster_added: 4
cluster_manager.cluster_modified: 0
cluster_manager.cluster_removed: 0
cluster_manager.cluster_updated: 2
cluster_manager.cluster_updated_via_merge: 0
cluster_manager.update_merge_cancelled: 0
cluster_manager.update_out_of_merge_window: 0
cluster_manager.warming_clusters: 0
http.admin.downstream_cx_active: 2
http.admin.downstream_cx_delayed_close_timeout: 0
http.admin.downstream_cx_destroy: 21
http.admin.downstream_cx_destroy_active_rq: 0
http.admin.downstream_cx_destroy_local: 0
http.admin.downstream_cx_destroy_local_active_rq: 0
http.admin.downstream_cx_destroy_remote: 21
http.admin.downstream_cx_destroy_remote_active_rq: 0
http.admin.downstream_cx_drain_close: 0
http.admin.downstream_cx_http1_active: 1
http.admin.downstream_cx_http1_total: 9
http.admin.downstream_cx_http2_active: 0
http.admin.downstream_cx_http2_total: 0
http.admin.downstream_cx_idle_timeout: 0
http.admin.downstream_cx_overload_disable_keepalive: 0
http.admin.downstream_cx_protocol_error: 0
http.admin.downstream_cx_rx_bytes_buffered: 431
http.admin.downstream_cx_rx_bytes_total: 9082
http.admin.downstream_cx_ssl_active: 0
http.admin.downstream_cx_ssl_total: 0
http.admin.downstream_cx_total: 23
http.admin.downstream_cx_tx_bytes_buffered: 0
http.admin.downstream_cx_tx_bytes_total: 179942
http.admin.downstream_cx_upgrades_active: 0
http.admin.downstream_cx_upgrades_total: 0
http.admin.downstream_flow_control_paused_reading_total: 0
http.admin.downstream_flow_control_resumed_reading_total: 0
http.admin.downstream_rq_1xx: 0
http.admin.downstream_rq_2xx: 13
http.admin.downstream_rq_3xx: 0
http.admin.downstream_rq_4xx: 8
http.admin.downstream_rq_5xx: 0
http.admin.downstream_rq_active: 1
http.admin.downstream_rq_completed: 21
http.admin.downstream_rq_http1_total: 22
http.admin.downstream_rq_http2_total: 0
http.admin.downstream_rq_idle_timeout: 0
http.admin.downstream_rq_non_relative_path: 0
http.admin.downstream_rq_overload_close: 0
http.admin.downstream_rq_response_before_rq_complete: 0
http.admin.downstream_rq_rx_reset: 0
http.admin.downstream_rq_timeout: 0
http.admin.downstream_rq_too_large: 0
http.admin.downstream_rq_total: 22
http.admin.downstream_rq_tx_reset: 0
http.admin.downstream_rq_ws_on_non_ws_route: 0
http.admin.rs_too_large: 0
http.async-client.no_cluster: 0
http.async-client.no_route: 0
http.async-client.rq_direct_response: 0
http.async-client.rq_redirect: 0
http.async-client.rq_reset_after_downstream_response_started: 0
http.async-client.rq_total: 5802
http.ingress_http.downstream_cx_active: 2
http.ingress_http.downstream_cx_delayed_close_timeout: 0
http.ingress_http.downstream_cx_destroy: 22
http.ingress_http.downstream_cx_destroy_active_rq: 0
http.ingress_http.downstream_cx_destroy_local: 0
http.ingress_http.downstream_cx_destroy_local_active_rq: 0
http.ingress_http.downstream_cx_destroy_remote: 22
http.ingress_http.downstream_cx_destroy_remote_active_rq: 0
http.ingress_http.downstream_cx_drain_close: 0
http.ingress_http.downstream_cx_http1_active: 1
http.ingress_http.downstream_cx_http1_total: 13
http.ingress_http.downstream_cx_http2_active: 0
http.ingress_http.downstream_cx_http2_total: 0
http.ingress_http.downstream_cx_idle_timeout: 0
http.ingress_http.downstream_cx_overload_disable_keepalive: 0
http.ingress_http.downstream_cx_protocol_error: 0
http.ingress_http.downstream_cx_rx_bytes_buffered: 452
http.ingress_http.downstream_cx_rx_bytes_total: 10266
http.ingress_http.downstream_cx_ssl_active: 0
http.ingress_http.downstream_cx_ssl_total: 0
http.ingress_http.downstream_cx_total: 24
http.ingress_http.downstream_cx_tx_bytes_buffered: 0
http.ingress_http.downstream_cx_tx_bytes_total: 5884
http.ingress_http.downstream_cx_upgrades_active: 0
http.ingress_http.downstream_cx_upgrades_total: 0
http.ingress_http.downstream_flow_control_paused_reading_total: 0
http.ingress_http.downstream_flow_control_resumed_reading_total: 0
http.ingress_http.downstream_rq_1xx: 0
http.ingress_http.downstream_rq_2xx: 0
http.ingress_http.downstream_rq_3xx: 0
http.ingress_http.downstream_rq_4xx: 0
http.ingress_http.downstream_rq_5xx: 28
http.ingress_http.downstream_rq_active: 0
http.ingress_http.downstream_rq_completed: 28
http.ingress_http.downstream_rq_http1_total: 28
http.ingress_http.downstream_rq_http2_total: 0
http.ingress_http.downstream_rq_idle_timeout: 0
http.ingress_http.downstream_rq_non_relative_path: 0
http.ingress_http.downstream_rq_overload_close: 0
http.ingress_http.downstream_rq_response_before_rq_complete: 0
http.ingress_http.downstream_rq_rx_reset: 0
http.ingress_http.downstream_rq_timeout: 0
http.ingress_http.downstream_rq_too_large: 0
http.ingress_http.downstream_rq_total: 28
http.ingress_http.downstream_rq_tx_reset: 0
http.ingress_http.downstream_rq_ws_on_non_ws_route: 0
http.ingress_http.no_cluster: 0
http.ingress_http.no_route: 0
http.ingress_http.rq_direct_response: 0
http.ingress_http.rq_redirect: 0
http.ingress_http.rq_reset_after_downstream_response_started: 0
http.ingress_http.rq_total: 28
http.ingress_http.rs_too_large: 0
http.ingress_http.tracing.client_enabled: 0
http.ingress_http.tracing.health_check: 0
http.ingress_http.tracing.not_traceable: 0
http.ingress_http.tracing.random_sampling: 28
http.ingress_http.tracing.service_forced: 0
listener.0.0.0.0_10000.downstream_cx_active: 2
listener.0.0.0.0_10000.downstream_cx_destroy: 22
listener.0.0.0.0_10000.downstream_cx_total: 24
listener.0.0.0.0_10000.downstream_pre_cx_active: 0
listener.0.0.0.0_10000.downstream_pre_cx_timeout: 0
listener.0.0.0.0_10000.http.ingress_http.downstream_rq_1xx: 0
listener.0.0.0.0_10000.http.ingress_http.downstream_rq_2xx: 0
listener.0.0.0.0_10000.http.ingress_http.downstream_rq_3xx: 0
listener.0.0.0.0_10000.http.ingress_http.downstream_rq_4xx: 0
listener.0.0.0.0_10000.http.ingress_http.downstream_rq_5xx: 28
listener.0.0.0.0_10000.http.ingress_http.downstream_rq_completed: 28
listener.0.0.0.0_10000.no_filter_chain_match: 0
listener.admin.downstream_cx_active: 2
listener.admin.downstream_cx_destroy: 21
listener.admin.downstream_cx_total: 23
listener.admin.downstream_pre_cx_active: 0
listener.admin.downstream_pre_cx_timeout: 0
listener.admin.http.admin.downstream_rq_1xx: 0
listener.admin.http.admin.downstream_rq_2xx: 13
listener.admin.http.admin.downstream_rq_3xx: 0
listener.admin.http.admin.downstream_rq_4xx: 8
listener.admin.http.admin.downstream_rq_5xx: 0
listener.admin.http.admin.downstream_rq_completed: 21
listener.admin.no_filter_chain_match: 0
listener_manager.listener_added: 1
listener_manager.listener_create_failure: 0
listener_manager.listener_create_success: 2
listener_manager.listener_modified: 0
listener_manager.listener_removed: 0
listener_manager.total_listeners_active: 1
listener_manager.total_listeners_draining: 0
listener_manager.total_listeners_warming: 0
runtime.admin_overrides_active: 0
runtime.deprecated_feature_use: 0
runtime.load_error: 0
runtime.load_success: 0
runtime.num_keys: 0
runtime.override_dir_exists: 0
runtime.override_dir_not_exists: 0
server.concurrency: 2
server.days_until_first_cert_expiring: 2147483647
server.debug_assertion_failures: 0
server.hot_restart_epoch: 0
server.live: 1
server.memory_allocated: 4147504
server.memory_heap_size: 6291456
server.parent_connections: 0
server.total_connections: 2
server.uptime: 299086
server.version: 10854825
server.watchdog_mega_miss: 359
server.watchdog_miss: 547
stats.overflow: 0
tracing.zipkin.reports_dropped: 0
tracing.zipkin.reports_failed: 0
tracing.zipkin.reports_sent: 25
tracing.zipkin.spans_sent: 28
tracing.zipkin.timer_flushed: 13055
cluster.eds_cluster.internal.upstream_rq_time: P0(3,2) P25(3.025,4.07087) P50(3.05,6.04019) P75(3.075,10.4235) P90(3.09,17.0051) P95(3.095,23.5571) P99(3.099,236.54) P99.5(3.0995,371.35) P99.9(3.0999,622.54) P100(3.1,980)
cluster.eds_cluster.upstream_cx_connect_ms: P0(nan,0) P25(nan,1.05202) P50(nan,2.03878) P75(nan,5.03571) P90(nan,39.3) P95(nan,181.5) P99(nan,324.3) P99.5(nan,364.3) P99.9(nan,826.86) P100(nan,830)
cluster.eds_cluster.upstream_cx_length_ms: P0(nan,0) P25(nan,4250) P50(nan,27076.9) P75(nan,56300) P90(nan,354000) P95(nan,374312) P99(nan,386300) P99.5(nan,388983) P99.9(nan,406780) P100(nan,410000)
cluster.eds_cluster.upstream_rq_time: P0(3,2) P25(3.025,4.07087) P50(3.05,6.04019) P75(3.075,10.4235) P90(3.09,17.0051) P95(3.095,23.5571) P99(3.099,236.54) P99.5(3.0995,371.35) P99.9(3.0999,622.54) P100(3.1,980)
cluster.service1.upstream_cx_connect_ms: P0(nan,1) P25(nan,1.05) P50(nan,1.1) P75(nan,3.1) P90(nan,4.06) P95(nan,4.08) P99(nan,4.096) P99.5(nan,4.098) P99.9(nan,4.0996) P100(nan,4.1)
cluster.service1.upstream_cx_length_ms: P0(nan,2) P25(nan,9.0625) P50(nan,242.5) P75(nan,254.375) P90(nan,259.25) P95(nan,263.5) P99(nan,268.7) P99.5(nan,269.35) P99.9(nan,269.87) P100(nan,270)
cluster.service2.upstream_cx_connect_ms: No recorded values
cluster.service2.upstream_cx_length_ms: P0(240,3) P25(242.5,243.125) P50(245,248.75) P75(247.5,254.375) P90(249,257.75) P95(249.5,258.875) P99(249.9,259.775) P99.5(249.95,259.887) P99.9(249.99,259.978) P100(250,260)
cluster.zipkin.internal.upstream_rq_time: P0(nan,6) P25(nan,8.08) P50(nan,11.6667) P75(nan,13.6667) P90(nan,27.6) P95(nan,31.8) P99(nan,35.76) P99.5(nan,35.88) P99.9(nan,35.976) P100(nan,36)
cluster.zipkin.upstream_cx_connect_ms: P0(1,0) P25(1.025,1.04286) P50(1.05,1.1) P75(1.075,2.1) P90(1.09,4.02) P95(1.095,4.06) P99(1.099,4.092) P99.5(1.0995,4.096) P99.9(1.0999,4.0992) P100(1.1,4.1)
cluster.zipkin.upstream_cx_length_ms: P0(nan,15000) P25(nan,15416.7) P50(nan,15833.3) P75(nan,20750) P90(nan,30250) P95(nan,30625) P99(nan,30925) P99.5(nan,30962.5) P99.9(nan,30992.5) P100(nan,31000)
cluster.zipkin.upstream_rq_time: P0(nan,6) P25(nan,8.08) P50(nan,11.6667) P75(nan,13.6667) P90(nan,27.6) P95(nan,31.8) P99(nan,35.76) P99.5(nan,35.88) P99.9(nan,35.976) P100(nan,36)
http.admin.downstream_cx_length_ms: P0(nan,3) P25(nan,11083.3) P50(nan,14250) P75(nan,79750) P90(nan,339000) P95(nan,459500) P99(nan,3.079e+06) P99.5(nan,3.0895e+06) P99.9(nan,3.0979e+06) P100(nan,3.1e+06)
http.admin.downstream_rq_time: P0(nan,0) P25(nan,0) P50(nan,0) P75(nan,1.08) P90(nan,2.1) P95(nan,3.1) P99(nan,6.08) P99.5(nan,6.09) P99.9(nan,6.098) P100(nan,6.1)
http.ingress_http.downstream_cx_length_ms: P0(nan,5) P25(nan,245) P50(nan,13000) P75(nan,54500) P90(nan,308000) P95(nan,439000) P99(nan,587800) P99.5(nan,588900) P99.9(nan,589780) P100(nan,590000)
http.ingress_http.downstream_rq_time: P0(250,0) P25(252.5,2.1) P50(255,242.5) P75(257.5,254) P90(259,258.2) P95(259.5,259.6) P99(259.9,267.2) P99.5(259.95,268.6) P99.9(259.99,269.72) P100(260,270)
listener.0.0.0.0_10000.downstream_cx_length_ms: P0(nan,5) P25(nan,245) P50(nan,13000) P75(nan,54500) P90(nan,308000) P95(nan,439000) P99(nan,587800) P99.5(nan,588900) P99.9(nan,589780) P100(nan,590000)
listener.admin.downstream_cx_length_ms: P0(nan,3) P25(nan,11083.3) P50(nan,14250) P75(nan,79750) P90(nan,339000) P95(nan,459500) P99(nan,3.079e+06) P99.5(nan,3.0895e+06) P99.9(nan,3.0979e+06) P100(nan,3.1e+06)

sumitpathak24 on 19 Apr 2019

I also experienced similar behaviour when running the standard demo Docker image on Mac OS X. It seemed to happen for any HTTPS upstream. If I manually ran network calls from within the container the requests seem to work okay.

It turns out that in my case the timeouts were too aggressive for the network connection I had. Things were fine with HTTP but the TLS handshake made connecting take too long.

after increasing the connect_timeout, my issue too vanished.