Consul-helm: consul connect: 503 after scaling in or pod restarts

Created on 8 Jun 2021 · 15Comments · Source: hashicorp/consul-helm

Overview of the Issue

We have http services running, frontend with 3 pods, backend with 20 pods. frontend is accessing backend via envoy proxy at http://127.0.0.1:10000. After backend pods are removed (scale down, rolling upgrade, manual kubectl delete), frontend pods randomly receive 503s by envoy. If all backend pods are removed, frontend will always receive 503 by envoy. This behavior persists until the consul client running on the same node as the frontend pod receiving the errors is restarted (kubectl delete pod consul-consul-XXXXX).

Reproduction Steps

Steps to reproduce this issue, eg:

Install consul using the below values.yml
Install both below deployments
open a shell in each frontend pod and run watch curl -I 127.0.0.1:10000. Response code is always 200
kubectl delete 10 backend pods
wait for the newly spawned backend pods to fully boot
the above curl requests randomly get a 503 response
kubectl delete consul agent daemonset members, on the nodes of the frontend pods where 503 is received
curl always gets 200

Logs

logs of a later deleted consul agent. backend pods are deleted at 2021-06-08T14:10:52.161Z. consul agent gets deleted at 2021-06-08T14:28:04.910Z.

==> Starting Consul agent...
         Version: '1.9.4'
         Node ID: '0b0a7926-4dbf-5168-2788-67bb4e209be3'
       Node name: 'ip-10-230-226-58.eu-central-1.compute.internal'
      Datacenter: 'dc1' (Segment: '')
          Server: false (Bootstrap: false)
     Client Addr: [0.0.0.0] (HTTP: -1, HTTPS: 8501, gRPC: 8502, DNS: 8600)
    Cluster Addr: 10.230.229.54 (LAN: 8301, WAN: 8302)
         Encrypt: Gossip: false, TLS-Outgoing: true, TLS-Incoming: false, Auto-Encrypt-TLS: true

==> Log data will now stream in as it occurs:

  2021-06-08T13:37:28.661Z [WARN]  agent: Node name "ip-10-230-226-58.eu-central-1.compute.internal" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes.
  2021-06-08T13:37:28.756Z [WARN]  agent.auto_config: Node name "ip-10-230-226-58.eu-central-1.compute.internal" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes.
  2021-06-08T13:37:29.055Z [INFO]  agent.auto_config: automatically upgraded to TLS
  2021-06-08T13:37:29.066Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: ip-10-230-226-58.eu-central-1.compute.internal 10.230.229.54
  2021-06-08T13:37:29.066Z [INFO]  agent.router: Initializing LAN area manager
  2021-06-08T13:37:29.066Z [INFO]  agent.auto_config: auto-config started
  2021-06-08T13:37:29.066Z [WARN]  agent.router.manager: No servers available
  2021-06-08T13:37:29.066Z [WARN]  agent.router.manager: No servers available
  2021-06-08T13:37:29.066Z [INFO]  agent: Started DNS server: address=0.0.0.0:8600 network=udp
  2021-06-08T13:37:29.066Z [INFO]  agent: Started DNS server: address=0.0.0.0:8600 network=tcp
  2021-06-08T13:37:29.066Z [WARN]  agent.cache: handling error in Cache.Notify: cache-type=connect-ca-root error="No known Consul servers" index=14
  2021-06-08T13:37:29.066Z [WARN]  agent.router.manager: No servers available
  2021-06-08T13:37:29.067Z [WARN]  agent.router.manager: No servers available
  2021-06-08T13:37:29.066Z [WARN]  agent.cache: handling error in Cache.Notify: cache-type=connect-ca-root error="No known Consul servers" index=14
  2021-06-08T13:37:29.067Z [WARN]  agent.router.manager: No servers available
  2021-06-08T13:37:29.067Z [WARN]  agent.router.manager: No servers available
  2021-06-08T13:37:29.067Z [WARN]  agent.router.manager: No servers available
  2021-06-08T13:37:29.067Z [WARN]  agent.router.manager: No servers available
  2021-06-08T13:37:29.067Z [WARN]  agent.cache: handling error in Cache.Notify: cache-type=connect-ca-root error="No known Consul servers" index=14
  2021-06-08T13:37:29.067Z [WARN]  agent.router.manager: No servers available
  2021-06-08T13:37:29.067Z [WARN]  agent.router.manager: No servers available
  2021-06-08T13:37:29.067Z [INFO]  agent: Starting server: address=[::]:8501 network=tcp protocol=https
  2021-06-08T13:37:29.067Z [WARN]  agent.cache: handling error in Cache.Notify: cache-type=connect-ca-root error="No known Consul servers" index=14
  2021-06-08T13:37:29.067Z [WARN]  agent.cache: handling error in Cache.Notify: cache-type=connect-ca-root error="No known Consul servers" index=14
  2021-06-08T13:37:29.067Z [WARN]  agent.router.manager: No servers available
  2021-06-08T13:37:29.067Z [INFO]  agent: Retry join is supported for the following discovery methods: cluster=LAN discovery_methods="aliyun aws azure digitalocean gce k8s linode mdns os packet scaleway softlayer tencentcloud triton vsphere"
  2021-06-08T13:37:29.067Z [WARN]  agent.router.manager: No servers available
  2021-06-08T13:37:29.067Z [INFO]  agent: Joining cluster...: cluster=LAN
  2021-06-08T13:37:29.067Z [INFO]  agent: (LAN) joining: lan_addresses=[consul-consul-server-0.consul-consul-server.infra.svc:8301, consul-consul-server-1.consul-consul-server.infra.svc:8301, consul-consul-server-2.consul-consul-server.infra.svc:8301]
  2021-06-08T13:37:29.067Z [INFO]  agent: Started gRPC server: address=[::]:8502 network=tcp
  2021-06-08T13:37:29.067Z [WARN]  agent.cache: handling error in Cache.Notify: cache-type=connect-ca-root error="No known Consul servers" index=14
  2021-06-08T13:37:29.067Z [WARN]  agent: DEPRECATED Backwards compatibility with pre-1.9 metrics enabled. These metrics will be removed in a future version of Consul. Set `telemetry { disable_compat_1.9 = true }` to disable them.
  2021-06-08T13:37:29.067Z [WARN]  agent.cache: handling error in Cache.Notify: cache-type=connect-ca-root error="No known Consul servers" index=14
  2021-06-08T13:37:29.067Z [INFO]  agent: started state syncer
==> Consul agent running!
  2021-06-08T13:37:29.067Z [WARN]  agent.router.manager: No servers available
  2021-06-08T13:37:29.067Z [WARN]  agent.router.manager: No servers available
  2021-06-08T13:37:29.067Z [WARN]  agent.router.manager: No servers available
  2021-06-08T13:37:29.067Z [WARN]  agent.cache: handling error in Cache.Notify: cache-type=connect-ca-root error="No known Consul servers" index=14
  2021-06-08T13:37:29.067Z [WARN]  agent.router.manager: No servers available
  2021-06-08T13:37:29.067Z [WARN]  agent.router.manager: No servers available
  2021-06-08T13:37:29.067Z [ERROR] agent.anti_entropy: failed to sync remote state: error="No known Consul servers"
  2021-06-08T13:37:29.072Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: consul-consul-server-2 10.230.244.254
  2021-06-08T13:37:29.073Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: ip-10-230-226-221.eu-central-1.compute.internal 10.230.232.224
  2021-06-08T13:37:29.073Z [WARN]  agent.client.memberlist.lan: memberlist: Refuting a dead message (from: ip-10-230-226-58.eu-central-1.compute.internal)
  2021-06-08T13:37:29.073Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: ip-10-230-240-149.eu-central-1.compute.internal 10.230.246.35
  2021-06-08T13:37:29.073Z [INFO]  agent.client: adding server: server="consul-consul-server-2 (Addr: tcp/10.230.244.254:8300) (DC: dc1)"
  2021-06-08T13:37:29.073Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: ip-10-230-213-95.eu-central-1.compute.internal 10.230.209.181
  2021-06-08T13:37:29.073Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: consul-consul-server-1 10.230.216.255
  2021-06-08T13:37:29.073Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: consul-consul-server-0 10.230.237.22
  2021-06-08T13:37:29.073Z [INFO]  agent.client: adding server: server="consul-consul-server-1 (Addr: tcp/10.230.216.255:8300) (DC: dc1)"
  2021-06-08T13:37:29.073Z [INFO]  agent.client: adding server: server="consul-consul-server-0 (Addr: tcp/10.230.237.22:8300) (DC: dc1)"
  2021-06-08T13:37:29.092Z [INFO]  agent: (LAN) joined: number_of_nodes=3
  2021-06-08T13:37:29.092Z [INFO]  agent: Join cluster completed. Synced with initial agents: cluster=LAN num_agents=3
  2021-06-08T13:37:29.116Z [INFO]  agent.client.serf.lan: serf: EventMemberUpdate: ip-10-230-226-58.eu-central-1.compute.internal
  2021-06-08T13:37:29.551Z [INFO]  agent: Synced node info
  2021-06-08T13:37:29.560Z [INFO]  agent: Synced service: service=frontend-5bf4898ffc-j4sbp-frontend
  2021-06-08T13:37:29.767Z [INFO]  agent: Synced service: service=frontend-5bf4898ffc-j4sbp-frontend-sidecar-proxy
  2021-06-08T13:37:30.863Z [INFO]  agent: Synced service: service=backend-85c64cc74d-c9tlx-backend
  2021-06-08T13:37:30.958Z [INFO]  agent: Synced service: service=backend-85c64cc74d-c9tlx-backend-sidecar-proxy
  2021-06-08T13:37:32.058Z [INFO]  agent: Synced service: service=frontend-5bf4898ffc-f4wbc-frontend
  2021-06-08T13:37:32.270Z [INFO]  agent: Synced service: service=frontend-5bf4898ffc-f4wbc-frontend-sidecar-proxy
  2021-06-08T13:37:32.494Z [INFO]  agent: Synced check: check=service:frontend-5bf4898ffc-j4sbp-frontend-sidecar-proxy:1
  2021-06-08T13:37:33.273Z [INFO]  agent: Synced check: check=service:frontend-5bf4898ffc-f4wbc-frontend-sidecar-proxy:1
  2021-06-08T13:37:33.497Z [INFO]  agent: Synced service: service=vault-1-vault
  2021-06-08T13:37:33.555Z [INFO]  agent: Synced service: service=vault-1-vault-sidecar-proxy
  2021-06-08T13:37:33.987Z [INFO]  agent.client.serf.lan: serf: EventMemberLeave: ip-10-230-240-149.eu-central-1.compute.internal 10.230.246.35
  2021-06-08T13:37:35.061Z [INFO]  agent: Synced check: check=service:vault-1-vault-sidecar-proxy:1
  2021-06-08T13:37:36.051Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: ip-10-230-252-160.eu-central-1.compute.internal 10.230.242.107
  2021-06-08T13:37:36.067Z [INFO]  agent.client.serf.lan: serf: EventMemberUpdate: ip-10-230-252-160.eu-central-1.compute.internal
  2021-06-08T13:37:37.662Z [INFO]  agent: Synced check: check=service:backend-85c64cc74d-c9tlx-backend-sidecar-proxy:1
  2021-06-08T13:37:38.381Z [INFO]  agent: Synced service: service=backend-85c64cc74d-4vld2-backend
  2021-06-08T13:37:38.465Z [INFO]  agent: Synced service: service=backend-85c64cc74d-4vld2-backend-sidecar-proxy
  2021-06-08T13:37:39.261Z [INFO]  agent: Synced service: service=backend-85c64cc74d-h6sm6-backend
  2021-06-08T13:37:39.557Z [INFO]  agent: Synced service: service=backend-85c64cc74d-h6sm6-backend-sidecar-proxy
  2021-06-08T13:37:39.993Z [INFO]  agent: Synced check: check=service:backend-85c64cc74d-4vld2-backend-sidecar-proxy:1
  2021-06-08T13:37:41.466Z [INFO]  agent: Synced check: check=service:backend-85c64cc74d-h6sm6-backend-sidecar-proxy:1
  2021-06-08T13:37:41.759Z [INFO]  agent: Synced check: check=service:backend-85c64cc74d-4vld2-backend-sidecar-proxy:2
  2021-06-08T13:37:41.772Z [INFO]  agent: Synced check: check=dynaware/backend-85c64cc74d-4vld2-backend/kubernetes-health-check
  2021-06-08T13:37:41.858Z [INFO]  agent: Synced check: check=dynaware/backend-85c64cc74d-4vld2-backend/kubernetes-health-check
  2021-06-08T13:37:43.358Z [INFO]  agent: Synced check: check=service:backend-85c64cc74d-c9tlx-backend-sidecar-proxy:2
  2021-06-08T13:37:43.459Z [INFO]  agent: Synced check: check=dynaware/backend-85c64cc74d-c9tlx-backend/kubernetes-health-check
  2021-06-08T13:37:43.468Z [INFO]  agent: Synced check: check=dynaware/backend-85c64cc74d-c9tlx-backend/kubernetes-health-check
  2021-06-08T13:37:45.168Z [INFO]  agent: Synced check: check=service:backend-85c64cc74d-h6sm6-backend-sidecar-proxy:2
  2021-06-08T13:37:45.250Z [INFO]  agent: Synced check: check=dynaware/backend-85c64cc74d-h6sm6-backend/kubernetes-health-check
  2021-06-08T13:37:45.269Z [INFO]  agent: Synced check: check=dynaware/backend-85c64cc74d-h6sm6-backend/kubernetes-health-check
  2021-06-08T13:37:45.463Z [INFO]  agent.client.serf.lan: serf: EventMemberLeave: ip-10-230-213-95.eu-central-1.compute.internal 10.230.209.181
  2021-06-08T13:37:46.854Z [INFO]  agent: Synced check: check=infra/vault-1-vault/kubernetes-health-check
  2021-06-08T13:37:46.949Z [INFO]  agent: Synced check: check=service:vault-1-vault-sidecar-proxy:2
  2021-06-08T13:37:47.059Z [INFO]  agent: Synced check: check=infra/vault-1-vault/kubernetes-health-check
  2021-06-08T13:37:47.251Z [INFO]  agent: Newer Consul version available: new_version=1.9.5 current_version=1.9.4
  2021-06-08T13:37:47.355Z [INFO]  agent: Synced check: check=umwerk/frontend-5bf4898ffc-f4wbc-frontend/kubernetes-health-check
  2021-06-08T13:37:47.557Z [INFO]  agent: Synced check: check=service:frontend-5bf4898ffc-f4wbc-frontend-sidecar-proxy:2
  2021-06-08T13:37:47.655Z [INFO]  agent: Synced check: check=umwerk/frontend-5bf4898ffc-f4wbc-frontend/kubernetes-health-check
  2021-06-08T13:37:47.674Z [INFO]  agent: Synced check: check=umwerk/frontend-5bf4898ffc-j4sbp-frontend/kubernetes-health-check
  2021-06-08T13:37:47.751Z [INFO]  agent: Synced check: check=service:frontend-5bf4898ffc-j4sbp-frontend-sidecar-proxy:2
  2021-06-08T13:37:47.760Z [INFO]  agent: Synced check: check=umwerk/frontend-5bf4898ffc-j4sbp-frontend/kubernetes-health-check
  2021-06-08T13:37:47.789Z [INFO]  agent.client.memberlist.lan: memberlist: Updating address for left or failed node ip-10-230-240-149.eu-central-1.compute.internal from 10.230.246.35:8301 to 10.230.242.12:8301
  2021-06-08T13:37:47.789Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: ip-10-230-240-149.eu-central-1.compute.internal 10.230.242.12
  2021-06-08T13:37:47.864Z [INFO]  agent.client.serf.lan: serf: EventMemberUpdate: ip-10-230-240-149.eu-central-1.compute.internal
  2021-06-08T13:38:11.483Z [INFO]  agent.client.memberlist.lan: memberlist: Updating address for left or failed node ip-10-230-213-95.eu-central-1.compute.internal from 10.230.209.181:8301 to 10.230.220.116:8301
  2021-06-08T13:38:11.483Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: ip-10-230-213-95.eu-central-1.compute.internal 10.230.220.116
  2021-06-08T13:38:11.865Z [INFO]  agent.client.serf.lan: serf: EventMemberUpdate: ip-10-230-213-95.eu-central-1.compute.internal
  2021-06-08T14:10:52.161Z [INFO]  agent: Deregistered service: service=backend-85c64cc74d-c9tlx-backend
  2021-06-08T14:10:52.163Z [WARN]  agent.cache: handling error in Cache.Notify: cache-type=service-http-checks error="Internal cache failure: service 'backend-85c64cc74d-c9tlx-backend' not in agent state" index=6
  2021-06-08T14:10:52.171Z [INFO]  agent: Synced check: check=service:backend-85c64cc74d-c9tlx-backend-sidecar-proxy:2
  2021-06-08T14:10:52.171Z [WARN]  agent.cache: handling error in Cache.Notify: cache-type=service-http-checks error="Internal cache failure: service 'backend-85c64cc74d-c9tlx-backend' not in agent state" index=6
  2021-06-08T14:10:52.260Z [INFO]  agent: Deregistered service: service=backend-85c64cc74d-c9tlx-backend-sidecar-proxy
  2021-06-08T14:10:52.559Z [INFO]  agent: Deregistered service: service=backend-85c64cc74d-h6sm6-backend
  2021-06-08T14:10:52.559Z [WARN]  agent.cache: handling error in Cache.Notify: cache-type=service-http-checks error="Internal cache failure: service 'backend-85c64cc74d-h6sm6-backend' not in agent state" index=6
  2021-06-08T14:10:52.559Z [WARN]  agent.cache: handling error in Cache.Notify: cache-type=service-http-checks error="Internal cache failure: service 'backend-85c64cc74d-h6sm6-backend' not in agent state" index=6
  2021-06-08T14:10:52.559Z [WARN]  agent.cache: handling error in Cache.Notify: cache-type=service-http-checks error="Internal cache failure: service 'backend-85c64cc74d-h6sm6-backend' not in agent state" index=6
  2021-06-08T14:10:52.560Z [WARN]  agent.cache: handling error in Cache.Notify: cache-type=service-http-checks error="Internal cache failure: service 'backend-85c64cc74d-h6sm6-backend' not in agent state" index=6
  2021-06-08T14:10:52.761Z [INFO]  agent: Synced check: check=service:backend-85c64cc74d-h6sm6-backend-sidecar-proxy:2
  2021-06-08T14:10:53.360Z [INFO]  agent: Deregistered service: service=backend-85c64cc74d-h6sm6-backend-sidecar-proxy
  2021-06-08T14:10:55.555Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.230.237.22:8300 error="rpc error making call: ACL not found"
  2021-06-08T14:10:55.555Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.230.237.22:8300 error="rpc error making call: ACL not found"
  2021-06-08T14:10:57.763Z [INFO]  agent: Synced service: service=backend-85c64cc74d-bbcg8-backend
  2021-06-08T14:10:58.053Z [INFO]  agent: Synced service: service=backend-85c64cc74d-bbcg8-backend-sidecar-proxy
  2021-06-08T14:10:58.287Z [INFO]  agent: Synced service: service=backend-85c64cc74d-4jc92-backend
  2021-06-08T14:10:58.955Z [INFO]  agent: Synced service: service=backend-85c64cc74d-78pb5-backend
  2021-06-08T14:10:59.489Z [INFO]  agent: Synced service: service=backend-85c64cc74d-4jc92-backend-sidecar-proxy
  2021-06-08T14:10:59.507Z [WARN]  agent: Check socket connection failed: check=service:backend-85c64cc74d-4jc92-backend-sidecar-proxy:1 error="dial tcp 10.230.229.55:20000: connect: connection refused"
  2021-06-08T14:10:59.507Z [WARN]  agent: Check is now critical: check=service:backend-85c64cc74d-4jc92-backend-sidecar-proxy:1
  2021-06-08T14:11:00.086Z [INFO]  agent: Synced service: service=backend-85c64cc74d-78pb5-backend-sidecar-proxy
  2021-06-08T14:11:00.453Z [INFO]  agent: Synced check: check=service:backend-85c64cc74d-4jc92-backend-sidecar-proxy:1
  2021-06-08T14:11:01.370Z [WARN]  agent: Check socket connection failed: check=service:backend-85c64cc74d-4jc92-backend-sidecar-proxy:1 error="dial tcp 10.230.229.55:20000: connect: connection refused"
  2021-06-08T14:11:01.370Z [WARN]  agent: Check is now critical: check=service:backend-85c64cc74d-4jc92-backend-sidecar-proxy:1
  2021-06-08T14:11:02.074Z [WARN]  agent: Check socket connection failed: check=service:backend-85c64cc74d-bbcg8-backend-sidecar-proxy:1 error="dial tcp 10.230.238.100:20000: connect: connection refused"
  2021-06-08T14:11:02.074Z [WARN]  agent: Check is now critical: check=service:backend-85c64cc74d-bbcg8-backend-sidecar-proxy:1
  2021-06-08T14:11:02.149Z [INFO]  agent: Synced check: check=service:backend-85c64cc74d-bbcg8-backend-sidecar-proxy:1
  2021-06-08T14:11:05.931Z [WARN]  agent: Check socket connection failed: check=service:backend-85c64cc74d-78pb5-backend-sidecar-proxy:1 error="dial tcp 10.230.234.9:20000: connect: connection refused"
  2021-06-08T14:11:05.931Z [WARN]  agent: Check is now critical: check=service:backend-85c64cc74d-78pb5-backend-sidecar-proxy:1
  2021-06-08T14:11:06.455Z [INFO]  agent: Synced check: check=service:backend-85c64cc74d-78pb5-backend-sidecar-proxy:1
  2021-06-08T14:11:07.290Z [INFO]  agent: Synced check: check=dynaware/backend-85c64cc74d-4jc92-backend/kubernetes-health-check
  2021-06-08T14:11:07.590Z [INFO]  agent: Synced check: check=service:backend-85c64cc74d-4jc92-backend-sidecar-proxy:2
  2021-06-08T14:11:07.792Z [INFO]  agent: Synced check: check=dynaware/backend-85c64cc74d-bbcg8-backend/kubernetes-health-check
  2021-06-08T14:11:07.933Z [INFO]  agent: Synced check: check=dynaware/backend-85c64cc74d-4jc92-backend/kubernetes-health-check
  2021-06-08T14:11:07.972Z [INFO]  agent: Synced check: check=service:backend-85c64cc74d-bbcg8-backend-sidecar-proxy:2
  2021-06-08T14:11:08.216Z [INFO]  agent: Synced check: check=service:backend-85c64cc74d-4jc92-backend-sidecar-proxy:2
  2021-06-08T14:11:08.290Z [INFO]  agent: Synced check: check=dynaware/backend-85c64cc74d-bbcg8-backend/kubernetes-health-check
  2021-06-08T14:11:08.379Z [INFO]  agent: Synced check: check=service:backend-85c64cc74d-bbcg8-backend-sidecar-proxy:2
  2021-06-08T14:11:10.436Z [WARN]  agent: Check socket connection failed: check=service:backend-85c64cc74d-78pb5-backend-sidecar-proxy:1 error="dial tcp 10.230.234.9:20000: connect: connection refused"
  2021-06-08T14:11:10.436Z [WARN]  agent: Check is now critical: check=service:backend-85c64cc74d-78pb5-backend-sidecar-proxy:1
  2021-06-08T14:11:11.286Z [INFO]  agent: Synced check: check=dynaware/backend-85c64cc74d-78pb5-backend/kubernetes-health-check
  2021-06-08T14:11:11.492Z [INFO]  agent: Synced check: check=service:backend-85c64cc74d-78pb5-backend-sidecar-proxy:2
  2021-06-08T14:11:11.664Z [INFO]  agent: Synced check: check=dynaware/backend-85c64cc74d-78pb5-backend/kubernetes-health-check
  2021-06-08T14:11:11.870Z [INFO]  agent: Synced check: check=service:backend-85c64cc74d-78pb5-backend-sidecar-proxy:2
  2021-06-08T14:11:14.613Z [INFO]  agent: Synced check: check=service:backend-85c64cc74d-bbcg8-backend-sidecar-proxy:1
  2021-06-08T14:11:20.983Z [INFO]  agent: Synced check: check=service:backend-85c64cc74d-78pb5-backend-sidecar-proxy:1
  2021-06-08T14:11:21.008Z [INFO]  agent: Synced check: check=service:backend-85c64cc74d-4jc92-backend-sidecar-proxy:1
  2021-06-08T14:11:48.536Z [INFO]  agent: Synced check: check=dynaware/backend-85c64cc74d-bbcg8-backend/kubernetes-health-check
  2021-06-08T14:11:48.548Z [INFO]  agent: Synced check: check=service:backend-85c64cc74d-bbcg8-backend-sidecar-proxy:2
  2021-06-08T14:11:51.971Z [INFO]  agent: Synced check: check=dynaware/backend-85c64cc74d-4jc92-backend/kubernetes-health-check
  2021-06-08T14:11:51.988Z [INFO]  agent: Synced check: check=service:backend-85c64cc74d-4jc92-backend-sidecar-proxy:2
  2021-06-08T14:12:13.665Z [INFO]  agent: Synced check: check=dynaware/backend-85c64cc74d-78pb5-backend/kubernetes-health-check
  2021-06-08T14:12:13.685Z [INFO]  agent: Synced check: check=service:backend-85c64cc74d-78pb5-backend-sidecar-proxy:2
  2021-06-08T14:18:26.641Z [ERROR] agent.client: RPC failed to server: method=ConfigEntry.ResolveServiceConfig server=10.230.237.22:8300 error="rpc error making call: ACL not found"
  2021-06-08T14:18:26.952Z [ERROR] agent.client: RPC failed to server: method=DiscoveryChain.Get server=10.230.216.255:8300 error="rpc error making call: ACL not found"
  2021-06-08T14:18:32.995Z [ERROR] agent.client: RPC failed to server: method=Intention.Match server=10.230.244.254:8300 error="rpc error making call: rpc error making call: ACL not found"
  2021-06-08T14:18:34.090Z [ERROR] agent.client: RPC failed to server: method=DiscoveryChain.Get server=10.230.237.22:8300 error="rpc error making call: ACL not found"
  2021-06-08T14:18:47.611Z [ERROR] agent.client: RPC failed to server: method=ConfigEntry.ResolveServiceConfig server=10.230.216.255:8300 error="rpc error making call: ACL not found"
  2021-06-08T14:18:48.322Z [ERROR] agent.client: RPC failed to server: method=DiscoveryChain.Get server=10.230.244.254:8300 error="rpc error making call: ACL not found"
  2021-06-08T14:18:51.622Z [ERROR] agent.client: RPC failed to server: method=DiscoveryChain.Get server=10.230.237.22:8300 error="rpc error making call: ACL not found"
  2021-06-08T14:19:05.093Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.230.216.255:8300 error="rpc error making call: ACL not found"
  2021-06-08T14:19:07.828Z [ERROR] agent.client: RPC failed to server: method=DiscoveryChain.Get server=10.230.244.254:8300 error="rpc error making call: ACL not found"
  2021-06-08T14:19:09.756Z [ERROR] agent.client: RPC failed to server: method=DiscoveryChain.Get server=10.230.237.22:8300 error="rpc error making call: ACL not found"
  2021-06-08T14:19:13.261Z [ERROR] agent.client: RPC failed to server: method=DiscoveryChain.Get server=10.230.216.255:8300 error="rpc error making call: ACL not found"
  2021-06-08T14:19:20.320Z [ERROR] agent.client: RPC failed to server: method=Intention.Match server=10.230.244.254:8300 error="rpc error making call: rpc error making call: ACL not found"
  2021-06-08T14:19:25.533Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.230.237.22:8300 error="rpc error making call: ACL not found"
  2021-06-08T14:19:42.121Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.230.216.255:8300 error="rpc error making call: ACL not found"
  2021-06-08T14:19:53.406Z [ERROR] agent.client: RPC failed to server: method=DiscoveryChain.Get server=10.230.244.254:8300 error="rpc error making call: ACL not found"
  2021-06-08T14:19:58.051Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.230.237.22:8300 error="rpc error making call: ACL not found"
  2021-06-08T14:20:04.149Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.230.216.255:8300 error="rpc error making call: ACL not found"
  2021-06-08T14:20:30.243Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.230.237.22:8300 error="rpc error making call: ACL not found"
  2021-06-08T14:28:04.910Z [INFO]  agent: Caught: signal=terminated
  2021-06-08T14:28:04.910Z [INFO]  agent: Gracefully shutting down agent...
  2021-06-08T14:28:04.910Z [INFO]  agent.client: client starting leave
  2021-06-08T14:28:05.066Z [INFO]  agent.client.serf.lan: serf: EventMemberLeave: ip-10-230-226-58.eu-central-1.compute.internal 10.230.229.54
  2021-06-08T14:28:08.267Z [INFO]  agent: Graceful exit completed
  2021-06-08T14:28:08.267Z [INFO]  agent: Requesting shutdown
  2021-06-08T14:28:08.270Z [INFO]  agent.client: shutting down client
  2021-06-08T14:28:08.270Z [ERROR] agent.envoy: Error handling ADS stream: error="rpc error: code = Unauthenticated desc = unauthenticated: no config snapshot"
  2021-06-08T14:28:08.272Z [INFO]  agent: consul client down
  2021-06-08T14:28:08.273Z [INFO]  agent: shutdown complete
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=DiscoveryChain.Get server=10.230.216.255:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [INFO]  agent: Stopping server: protocol=DNS address=0.0.0.0:8600 network=tcp
  2021-06-08T14:28:08.272Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.230.216.255:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.272Z [ERROR] agent.client: RPC failed to server: method=DiscoveryChain.Get server=10.230.216.255:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.272Z [ERROR] agent.client: RPC failed to server: method=DiscoveryChain.Get server=10.230.216.255:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.272Z [ERROR] agent.client: RPC failed to server: method=DiscoveryChain.Get server=10.230.216.255:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.272Z [ERROR] agent.client: RPC failed to server: method=DiscoveryChain.Get server=10.230.216.255:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.272Z [ERROR] agent.client: RPC failed to server: method=ConfigEntry.ResolveServiceConfig server=10.230.216.255:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.272Z [ERROR] agent.client: RPC failed to server: method=ConnectCA.Roots server=10.230.216.255:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.272Z [ERROR] agent.client: RPC failed to server: method=DiscoveryChain.Get server=10.230.216.255:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=Intention.Match server=10.230.216.255:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=Intention.Match server=10.230.216.255:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.230.216.255:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.230.216.255:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=DiscoveryChain.Get server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=DiscoveryChain.Get server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=DiscoveryChain.Get server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=DiscoveryChain.Get server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=Intention.Match server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=DiscoveryChain.Get server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.230.237.22:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.230.237.22:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=DiscoveryChain.Get server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=ConfigEntry.ResolveServiceConfig server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=DiscoveryChain.Get server=10.230.237.22:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=DiscoveryChain.Get server=10.230.237.22:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=ConfigEntry.ResolveServiceConfig server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.230.237.22:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=ConnectCA.Roots server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=DiscoveryChain.Get server=10.230.237.22:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=ConnectCA.Roots server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=DiscoveryChain.Get server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.230.237.22:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=ConnectCA.Roots server=10.230.237.22:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=DiscoveryChain.Get server=10.230.237.22:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.230.237.22:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=Intention.Match server=10.230.237.22:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=DiscoveryChain.Get server=10.230.237.22:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=ConnectCA.Roots server=10.230.237.22:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=DiscoveryChain.Get server=10.230.237.22:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.230.237.22:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=DiscoveryChain.Get server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=ConnectCA.Roots server=10.230.216.255:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.276Z [WARN]  agent.cache: handling error in Cache.Notify: cache-type=connect-ca-root error="rpc error making call: EOF" index=14
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=ConfigEntry.ResolveServiceConfig server=10.230.216.255:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=Intention.Match server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=DiscoveryChain.Get server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=DiscoveryChain.Get server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=DiscoveryChain.Get server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=DiscoveryChain.Get server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=DiscoveryChain.Get server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=DiscoveryChain.Get server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=ConnectCA.Roots server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=DiscoveryChain.Get server=10.230.216.255:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=DiscoveryChain.Get server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=ConfigEntry.ResolveServiceConfig server=10.230.216.255:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=DiscoveryChain.Get server=10.230.216.255:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=DiscoveryChain.Get server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=DiscoveryChain.Get server=10.230.216.255:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=DiscoveryChain.Get server=10.230.216.255:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=ConnectCA.Roots server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.230.216.255:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=ConnectCA.Roots server=10.230.216.255:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=ConfigEntry.ResolveServiceConfig server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=ConfigEntry.ResolveServiceConfig server=10.230.216.255:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=DiscoveryChain.Get server=10.230.216.255:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=Intention.Match server=10.230.216.255:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=Intention.Match server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=Intention.Match server=10.230.216.255:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=DiscoveryChain.Get server=10.230.216.255:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=DiscoveryChain.Get server=10.230.216.255:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=ConnectCA.Roots server=10.230.216.255:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=DiscoveryChain.Get server=10.230.216.255:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=DiscoveryChain.Get server=10.230.216.255:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=ConfigEntry.ResolveServiceConfig server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=ConnectCA.Roots server=10.230.216.255:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.272Z [ERROR] agent.client: RPC failed to server: method=DiscoveryChain.Get server=10.230.216.255:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=DiscoveryChain.Get server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.273Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.230.244.254:8300 error="rpc error making call: EOF"
  2021-06-08T14:28:08.274Z [INFO]  agent: Stopping server: protocol=DNS address=0.0.0.0:8600 network=udp
  2021-06-08T14:28:08.277Z [INFO]  agent: Stopping server: address=[::]:8501 network=tcp protocol=https
  2021-06-08T14:28:08.778Z [INFO]  agent: Waiting for endpoints to shut down
  2021-06-08T14:28:08.778Z [INFO]  agent: Endpoints down
  2021-06-08T14:28:08.778Z [INFO]  agent: Exit code: code=0

logs of the above's successor:

==> Starting Consul agent...
         Version: '1.9.4'
         Node ID: '0b0a7926-4dbf-5168-2788-67bb4e209be3'
       Node name: 'ip-10-230-226-58.eu-central-1.compute.internal'
      Datacenter: 'dc1' (Segment: '')
          Server: false (Bootstrap: false)
     Client Addr: [0.0.0.0] (HTTP: -1, HTTPS: 8501, gRPC: 8502, DNS: 8600)
    Cluster Addr: 10.230.225.86 (LAN: 8301, WAN: 8302)
         Encrypt: Gossip: false, TLS-Outgoing: true, TLS-Incoming: false, Auto-Encrypt-TLS: true

==> Log data will now stream in as it occurs:

  2021-06-08T14:28:14.466Z [WARN]  agent: Node name "ip-10-230-226-58.eu-central-1.compute.internal" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes.
  2021-06-08T14:28:14.565Z [WARN]  agent.auto_config: Node name "ip-10-230-226-58.eu-central-1.compute.internal" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes.
  2021-06-08T14:28:14.678Z [INFO]  agent.auto_config: automatically upgraded to TLS
  2021-06-08T14:28:14.679Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: ip-10-230-226-58.eu-central-1.compute.internal 10.230.225.86
  2021-06-08T14:28:14.679Z [INFO]  agent.router: Initializing LAN area manager
  2021-06-08T14:28:14.679Z [INFO]  agent.auto_config: auto-config started
  2021-06-08T14:28:14.680Z [INFO]  agent: Started DNS server: address=0.0.0.0:8600 network=udp
  2021-06-08T14:28:14.680Z [WARN]  agent.router.manager: No servers available
  2021-06-08T14:28:14.680Z [WARN]  agent.router.manager: No servers available
  2021-06-08T14:28:14.681Z [WARN]  agent.router.manager: No servers available
  2021-06-08T14:28:14.681Z [WARN]  agent.router.manager: No servers available
  2021-06-08T14:28:14.681Z [WARN]  agent.router.manager: No servers available
  2021-06-08T14:28:14.681Z [WARN]  agent.router.manager: No servers available
  2021-06-08T14:28:14.681Z [WARN]  agent.router.manager: No servers available
  2021-06-08T14:28:14.681Z [WARN]  agent.router.manager: No servers available
  2021-06-08T14:28:14.682Z [INFO]  agent: Started DNS server: address=0.0.0.0:8600 network=tcp
  2021-06-08T14:28:14.683Z [INFO]  agent: Starting server: address=[::]:8501 network=tcp protocol=https
  2021-06-08T14:28:14.683Z [WARN]  agent: DEPRECATED Backwards compatibility with pre-1.9 metrics enabled. These metrics will be removed in a future version of Consul. Set `telemetry { disable_compat_1.9 = true }` to disable them.
  2021-06-08T14:28:14.683Z [INFO]  agent: started state syncer
==> Consul agent running!
  2021-06-08T14:28:14.683Z [WARN]  agent.cache: handling error in Cache.Notify: cache-type=connect-ca-root error="No known Consul servers" index=14
  2021-06-08T14:28:14.684Z [WARN]  agent.router.manager: No servers available
  2021-06-08T14:28:14.684Z [WARN]  agent.router.manager: No servers available
  2021-06-08T14:28:14.684Z [WARN]  agent.router.manager: No servers available
  2021-06-08T14:28:14.684Z [WARN]  agent.router.manager: No servers available
  2021-06-08T14:28:14.684Z [WARN]  agent.cache: handling error in Cache.Notify: cache-type=connect-ca-root error="No known Consul servers" index=14
  2021-06-08T14:28:14.684Z [WARN]  agent.router.manager: No servers available
  2021-06-08T14:28:14.749Z [INFO]  agent: Started gRPC server: address=[::]:8502 network=tcp
  2021-06-08T14:28:14.749Z [WARN]  agent.router.manager: No servers available
  2021-06-08T14:28:14.750Z [ERROR] agent.anti_entropy: failed to sync remote state: error="No known Consul servers"
  2021-06-08T14:28:14.750Z [WARN]  agent.cache: handling error in Cache.Notify: cache-type=connect-ca-root error="No known Consul servers" index=14
  2021-06-08T14:28:14.750Z [WARN]  agent.cache: handling error in Cache.Notify: cache-type=connect-ca-root error="No known Consul servers" index=14
  2021-06-08T14:28:14.750Z [WARN]  agent.router.manager: No servers available
  2021-06-08T14:28:14.750Z [WARN]  agent.cache: handling error in Cache.Notify: cache-type=connect-ca-root error="No known Consul servers" index=14
  2021-06-08T14:28:14.750Z [WARN]  agent.cache: handling error in Cache.Notify: cache-type=connect-ca-root error="No known Consul servers" index=14
  2021-06-08T14:28:14.750Z [INFO]  agent: Retry join is supported for the following discovery methods: cluster=LAN discovery_methods="aliyun aws azure digitalocean gce k8s linode mdns os packet scaleway softlayer tencentcloud triton vsphere"
  2021-06-08T14:28:14.751Z [INFO]  agent: Joining cluster...: cluster=LAN
  2021-06-08T14:28:14.751Z [INFO]  agent: (LAN) joining: lan_addresses=[consul-consul-server-0.consul-consul-server.infra.svc:8301, consul-consul-server-1.consul-consul-server.infra.svc:8301, consul-consul-server-2.consul-consul-server.infra.svc:8301]
  2021-06-08T14:28:14.751Z [WARN]  agent.router.manager: No servers available
  2021-06-08T14:28:14.751Z [WARN]  agent.cache: handling error in Cache.Notify: cache-type=connect-ca-root error="No known Consul servers" index=14
  2021-06-08T14:28:14.751Z [WARN]  agent.cache: handling error in Cache.Notify: cache-type=connect-ca-root error="No known Consul servers" index=14
  2021-06-08T14:28:14.751Z [WARN]  agent.router.manager: No servers available
  2021-06-08T14:28:14.772Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: ip-10-230-252-160.eu-central-1.compute.internal 10.230.242.107
  2021-06-08T14:28:14.773Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: consul-consul-server-1 10.230.216.255
  2021-06-08T14:28:14.773Z [INFO]  agent.client: adding server: server="consul-consul-server-1 (Addr: tcp/10.230.216.255:8300) (DC: dc1)"
  2021-06-08T14:28:14.773Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: consul-consul-server-2 10.230.244.254
  2021-06-08T14:28:14.773Z [INFO]  agent.client: adding server: server="consul-consul-server-2 (Addr: tcp/10.230.244.254:8300) (DC: dc1)"
  2021-06-08T14:28:14.773Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: ip-10-230-240-149.eu-central-1.compute.internal 10.230.242.12
  2021-06-08T14:28:14.774Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: ip-10-230-213-95.eu-central-1.compute.internal 10.230.220.116
  2021-06-08T14:28:14.774Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: consul-consul-server-0 10.230.237.22
  2021-06-08T14:28:14.774Z [WARN]  agent.client.memberlist.lan: memberlist: Refuting a dead message (from: ip-10-230-226-58.eu-central-1.compute.internal)
  2021-06-08T14:28:14.774Z [INFO]  agent.client: adding server: server="consul-consul-server-0 (Addr: tcp/10.230.237.22:8300) (DC: dc1)"
  2021-06-08T14:28:14.774Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: ip-10-230-226-221.eu-central-1.compute.internal 10.230.232.224
  2021-06-08T14:28:14.806Z [INFO]  agent: (LAN) joined: number_of_nodes=3
  2021-06-08T14:28:14.807Z [INFO]  agent: Join cluster completed. Synced with initial agents: cluster=LAN num_agents=3
  2021-06-08T14:28:14.850Z [INFO]  agent.client.serf.lan: serf: EventMemberUpdate: ip-10-230-226-58.eu-central-1.compute.internal
  2021-06-08T14:28:15.130Z [INFO]  agent: Synced node info
  2021-06-08T14:28:15.486Z [INFO]  agent: Synced service: service=backend-85c64cc74d-4vld2-backend
  2021-06-08T14:28:15.654Z [INFO]  agent: Synced service: service=backend-85c64cc74d-4jc92-backend
  2021-06-08T14:28:15.794Z [INFO]  agent: Synced service: service=backend-85c64cc74d-4vld2-backend-sidecar-proxy
  2021-06-08T14:28:15.986Z [INFO]  agent: Synced service: service=backend-85c64cc74d-4jc92-backend-sidecar-proxy
  2021-06-08T14:28:16.186Z [INFO]  agent: Synced service: service=backend-85c64cc74d-78pb5-backend
  2021-06-08T14:28:16.286Z [INFO]  agent: Synced service: service=backend-85c64cc74d-4jc92-backend-sidecar-proxy
  2021-06-08T14:28:16.487Z [INFO]  agent: Synced service: service=backend-85c64cc74d-78pb5-backend
  2021-06-08T14:28:16.758Z [INFO]  agent: Synced service: service=backend-85c64cc74d-78pb5-backend-sidecar-proxy
  2021-06-08T14:28:16.886Z [INFO]  agent: Synced service: service=backend-85c64cc74d-78pb5-backend-sidecar-proxy
  2021-06-08T14:28:17.286Z [INFO]  agent: Synced service: service=frontend-5bf4898ffc-f4wbc-frontend
  2021-06-08T14:28:17.586Z [INFO]  agent: Synced service: service=frontend-5bf4898ffc-f4wbc-frontend-sidecar-proxy
  2021-06-08T14:28:17.823Z [INFO]  agent: Synced check: check=dynaware/backend-85c64cc74d-4jc92-backend/kubernetes-health-check
  2021-06-08T14:28:17.886Z [INFO]  agent: Synced check: check=service:backend-85c64cc74d-4jc92-backend-sidecar-proxy:2
  2021-06-08T14:28:17.898Z [INFO]  agent: Synced check: check=dynaware/backend-85c64cc74d-4jc92-backend/kubernetes-health-check
  2021-06-08T14:28:18.098Z [INFO]  agent: Synced check: check=dynaware/backend-85c64cc74d-4vld2-backend/kubernetes-health-check
  2021-06-08T14:28:18.109Z [INFO]  agent: Synced check: check=service:backend-85c64cc74d-4vld2-backend-sidecar-proxy:2
  2021-06-08T14:28:18.118Z [INFO]  agent: Synced check: check=dynaware/backend-85c64cc74d-4vld2-backend/kubernetes-health-check
  2021-06-08T14:28:18.286Z [INFO]  agent: Synced check: check=dynaware/backend-85c64cc74d-78pb5-backend/kubernetes-health-check
  2021-06-08T14:28:18.295Z [INFO]  agent: Synced check: check=service:backend-85c64cc74d-78pb5-backend-sidecar-proxy:2
  2021-06-08T14:28:18.486Z [INFO]  agent: Synced check: check=dynaware/backend-85c64cc74d-78pb5-backend/kubernetes-health-check
  2021-06-08T14:28:18.512Z [ERROR] agent.http: Request error: method=PUT url=/v1/agent/check/register from=10.230.251.67:36454 error="ServiceID "backend-85c64cc74d-bbcg8-backend" does not exist"
  2021-06-08T14:28:18.798Z [ERROR] agent.http: Request error: method=PUT url=/v1/agent/check/register from=10.230.251.67:36488 error="ServiceID "vault-1-vault" does not exist"
  2021-06-08T14:28:18.867Z [INFO]  agent: Synced check: check=umwerk/frontend-5bf4898ffc-f4wbc-frontend/kubernetes-health-check
  2021-06-08T14:28:18.878Z [INFO]  agent: Synced check: check=service:frontend-5bf4898ffc-f4wbc-frontend-sidecar-proxy:2
  2021-06-08T14:28:18.887Z [INFO]  agent: Synced check: check=umwerk/frontend-5bf4898ffc-f4wbc-frontend/kubernetes-health-check
  2021-06-08T14:28:18.901Z [ERROR] agent.http: Request error: method=PUT url=/v1/agent/check/register from=10.230.251.67:36504 error="ServiceID "frontend-5bf4898ffc-j4sbp-frontend" does not exist"
  2021-06-08T14:28:19.664Z [INFO]  agent: Synced check: check=service:backend-85c64cc74d-4jc92-backend-sidecar-proxy:1
  2021-06-08T14:28:20.186Z [INFO]  agent: Synced service: service=backend-85c64cc74d-bbcg8-backend
  2021-06-08T14:28:20.294Z [INFO]  agent: Synced service: service=backend-85c64cc74d-bbcg8-backend-sidecar-proxy
  2021-06-08T14:28:22.360Z [INFO]  agent: Synced service: service=vault-1-vault
  2021-06-08T14:28:22.751Z [INFO]  agent: Synced service: service=vault-1-vault-sidecar-proxy
  2021-06-08T14:28:22.886Z [INFO]  agent: Synced check: check=service:backend-85c64cc74d-4vld2-backend-sidecar-proxy:1
  2021-06-08T14:28:23.349Z [INFO]  agent: Synced service: service=frontend-5bf4898ffc-j4sbp-frontend
  2021-06-08T14:28:23.486Z [INFO]  agent: Synced service: service=frontend-5bf4898ffc-j4sbp-frontend-sidecar-proxy
  2021-06-08T14:28:24.496Z [INFO]  agent: Synced check: check=service:backend-85c64cc74d-bbcg8-backend-sidecar-proxy:1
  2021-06-08T14:28:26.349Z [INFO]  agent: Synced check: check=service:backend-85c64cc74d-78pb5-backend-sidecar-proxy:1
  2021-06-08T14:28:26.886Z [INFO]  agent: Synced check: check=service:frontend-5bf4898ffc-f4wbc-frontend-sidecar-proxy:1
  2021-06-08T14:28:29.598Z [INFO]  agent: Synced check: check=service:vault-1-vault-sidecar-proxy:1
  2021-06-08T14:28:30.605Z [INFO]  agent: Synced check: check=service:frontend-5bf4898ffc-j4sbp-frontend-sidecar-proxy:1
  2021-06-08T14:28:41.062Z [INFO]  agent: Newer Consul version available: new_version=1.9.5 current_version=1.9.4
  2021-06-08T14:29:19.250Z [INFO]  agent: Synced check: check=service:backend-85c64cc74d-bbcg8-backend-sidecar-proxy:2
  2021-06-08T14:29:19.286Z [INFO]  agent: Synced check: check=dynaware/backend-85c64cc74d-bbcg8-backend/kubernetes-health-check
  2021-06-08T14:29:19.395Z [INFO]  agent: Synced check: check=dynaware/backend-85c64cc74d-bbcg8-backend/kubernetes-health-check
  2021-06-08T14:29:19.553Z [INFO]  agent: Synced check: check=service:vault-1-vault-sidecar-proxy:2
  2021-06-08T14:29:19.657Z [INFO]  agent: Synced check: check=infra/vault-1-vault/kubernetes-health-check
  2021-06-08T14:29:19.754Z [INFO]  agent: Synced check: check=infra/vault-1-vault/kubernetes-health-check
  2021-06-08T14:29:20.262Z [INFO]  agent: Synced check: check=umwerk/frontend-5bf4898ffc-j4sbp-frontend/kubernetes-health-check
  2021-06-08T14:29:20.271Z [INFO]  agent: Synced check: check=service:frontend-5bf4898ffc-j4sbp-frontend-sidecar-proxy:2
  2021-06-08T14:29:20.359Z [INFO]  agent: Synced check: check=umwerk/frontend-5bf4898ffc-j4sbp-frontend/kubernetes-health-check

Expected behavior

frontend does receive 200 as response code

Environment details

If not already included, please provide the following:

consul-k8s version: hashicorp/consul-k8s:0.25.0
consul-helm version: 0.31.1
values.yaml used to deploy the helm chart:
yaml global: enabled: true name: null domain: consul datacenter: dc1 tls: enabled: true enableAutoEncrypt: true serverAdditionalDNSSANs: - consul - consul.local serverAdditionalIPSANs: - 127.0.0.1 verify: true httpsOnly: true acls: manageSystemACLs: true consulSidecarContainer: resources: requests: memory: "25Mi" cpu: 0 limits: memory: "50Mi" cpu: 0 server: replicas: 3 bootstrapExpect: 3 connect: true client: resources: requests: memory: "250Mi" cpu: "250m" limits: memory: "250Mi" cpu: "250m" ui: enabled: true service: enabled: true type: NodePort annotations: | "alb.ingress.kubernetes.io/healthcheck-path": "/ui/" "alb.ingress.kubernetes.io/healthcheck-protocol": "HTTPS" "alb.ingress.kubernetes.io/backend-protocol": "HTTPS" controller: enabled: true replicas: 1 logLevel: info resources: limits: cpu: 100m memory: 50Mi requests: cpu: 100m memory: 50Mi syncCatalog: enabled: true k8sAllowNamespaces: <ul> <li>"someapp" k8sDenyNamespaces: ["*"] connectInject: enabled: true default: false k8sAllowNamespaces:</li> <li>"someapp" k8sDenyNamespaces: ["*"] healthChecks: enabled: true reconcilePeriod: "1m" cleanupController: reconcilePeriod: "1m" resources: requests: memory: "200Mi" cpu: "500m" limits: memory: "200Mi" cpu: "500m" centralConfig: enabled: true sidecarProxy: resources: requests: memory: 100Mi cpu: 100m limits: memory: 100Mi cpu: 100m ingressGateways: enabled: true defaults: replicas: 2 service: type: LoadBalancer ports: <ul> <li>port: 443 nodePort: null annotations: | "external-dns.alpha.kubernetes.io/hostname": "some.internal.hostname" "service.beta.kubernetes.io/aws-load-balancer-internal": "true" "service.beta.kubernetes.io/aws-load-balancer-ssl-cert": "<a href="arn:something">arn:something</a>" "service.beta.kubernetes.io/aws-load-balancer-backend-protocol": "tcp" gateways:</li> </ul></li> <li>name: ingress-gateway terminatingGateways: enabled: true defaults: replicas: 2 gateways:</li> <li>name: terminating-gateway

deployment manifest of backend:

yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: backend
namespace: someapp
spec:
replicas: 20
selector:
  matchLabels:
    app.kubernetes.io/instance: someinstance
    app.kubernetes.io/name: backend
strategy:
  rollingUpdate:
    maxSurge: 0
    maxUnavailable: 1
  type: RollingUpdate
template:
  metadata:
    annotations:
      consul.hashicorp.com/connect-inject: "true"
      consul.hashicorp.com/connect-service-upstreams: vault:8200, postgresql:5432
    labels:
      app.kubernetes.io/instance: someinstance
      app.kubernetes.io/name: backend
  spec:
    containers:
    - image: backend-image
      imagePullPolicy: Always
      livenessProbe:
        failureThreshold: 3
        httpGet:
          path: /health/liveness
          port: 8081
          scheme: HTTP
        initialDelaySeconds: 60
        periodSeconds: 10
        successThreshold: 1
        timeoutSeconds: 1
      name: backend
      ports:
      - containerPort: 8080
        name: api
        protocol: TCP
      - containerPort: 8081
        name: admin
        protocol: TCP
      readinessProbe:
        failureThreshold: 3
        httpGet:
          path: /health/readiness
          port: 8081
          scheme: HTTP
        initialDelaySeconds: 20
        periodSeconds: 10
        successThreshold: 1
        timeoutSeconds: 1
      resources:
        limits:
          cpu: "2"
          memory: 2Gi
        requests:
          cpu: 100m
          memory: 1Gi
    serviceAccount: backend
    serviceAccountName: backend

deployment manifest of frontend:
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: frontend
namespace: someapp
spec:
replicas: 3
selector:
matchLabels:
app.kubernetes.io/instance: someinstance
app.kubernetes.io/name: frontend
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
type: RollingUpdate
template:
metadata:
annotations:
consul.hashicorp.com/connect-inject: "true"
consul.hashicorp.com/connect-service-upstreams: backend:10000
labels:
app.kubernetes.io/instance: someinstance
app.kubernetes.io/name: frontend
spec:
containers:
- image: frontend-image
imagePullPolicy: Always
livenessProbe:
failureThreshold: 3
httpGet:
path: /health/liveness
port: 8080
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
name: frontend
ports:
- containerPort: 8080
name: http
protocol: TCP
readinessProbe:
failureThreshold: 10
httpGet:
path: /health/readiness
port: 8080
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
cpu: "2"
memory: 2Gi
requests:
cpu: 100m
memory: 1Gi
serviceAccount: frontend
serviceAccountName: frontend

```

Additionally, please provide details regarding the Kubernetes Infrastructure, if known:

Cloud Provider: AWS EKS
consul is installed the kubernetes namespace infra

Additional Context

frontend is exposed to the AWS VPC via an ingress-gateway, which shows the same behavior, if frontend pods are removed.
backend is using vault as a secret store. Same issue there.

Thanks for your help, I'm happy to provide more info if requested.

bug

Source

marcusweinhold

All 15 comments

Hi, upon initial investigation I was unable to reproduce:

CleanShot 2021-06-09 at 16 38 14

I'll look through the logs in more detail.

lkysow on 10 Jun 2021

Thanks for your effort!

Since I'll be debugging on my installation a bit more anyway, is there some log/metric/whatever I should have a closer look at?
Could it be related to lack of CPU/memory for any pod? The backend containers are quite heavy on CPU.

Cheers

marcusweinhold on 10 Jun 2021

Since I'll be debugging on my installation a bit more anyway, is there some log/metric/whatever I should have a closer look at?

If you can do a curl on port 19000 of the frontend pods it would be good to see /clusters?format=json. This is the envoy config. Also the envoy logs would be useful from the frontend pods.

lkysow on 10 Jun 2021

Hi Luke, see the attached file. This is 3 frontend pods, each with envoy config while healthy, while faulty and after "repairing" by deleting the consul clients, as well as each pod's envoy log.

To rule out issues with our own containers, I have reproduced the issue with nginx:alpine for both backend and frontend.

logs-configs.tar.gz

Edit: I forgot to mention, rl6xv was actually not affected by the issue, i got the expected response as soon as backend pods were up.

marcusweinhold on 11 Jun 2021

Hi @marcusweinhold wondering whether is possible you are setting circuit breakers via ServiceDefaults? I was just glancing at the logs and noticed them being set. Maybe that could help us reproduce.

david-yu on 11 Jun 2021

Hi, I have not enabled circuit breakers explicitly. The to services I used for generating the logs above use the global ServiceDefaults, which look like this:

/ $ consul config read -kind service-defaults -name global
{
    "Kind": "service-defaults",
    "Name": "global",
    "Protocol": "tcp",
    "MeshGateway": {},
    "Expose": {},
    "CreateIndex": 616,
    "ModifyIndex": 616
}

Only difference to the productive microservices is "Protocol": "http", but both protocols are affected in my case (http throwing 503, tcp throwing connection reset).

Cheers, Marcus

marcusweinhold on 14 Jun 2021

I have been able to reproduce the issue in a clean minikube cluster:

# launch minikube
minikube start --driver=kvm2 --memory=12g --cpus=6 --container-runtime=containerd

# consul namespace
kubectl apply -f namespace.yml

# install consul
helm upgrade --install consul --namespace infra --values values-minimal.yaml hashicorp/consul

# install backend + frontend
kubectl apply -f svc.yml

# kubectl portforward
kubectl port-forward -n infra service/consul-consul-ui 8443:443

# get token
kubectl -n infra get secrets consul-consul-bootstrap-acl-token -o json | jq -r '.data.token' | base64 -d | xclip

# create intention allow frontend -> backend via gui at https://127.0.0.1:8443

# open a shell in each of the frontend containers and run
watch curl -I localhost:10000

# delete backend pods
eval $(kubectl get pods | grep "^backend" | awk '{print "kubectl delete pod " $1 " &"}')

# delete consul
eval $(kubectl -n infra get pods | grep "consul-consul-..... " | awk '{print "kubectl -n infra delete pod " $1 " &"}')

Sometimes, I had to repeat the delete backend pods step a couple of times, always waited until all backends are running again, but eventually at least one backend pod got into the state I've described.

These are all files used above:
files.tar.gz

marcusweinhold on 15 Jun 2021

👍1

@lkysow any news on this?

marcusweinhold on 23 Jun 2021

@lkysow @david-yu Could it be, that the is is just a cpu bottleneck? I noticed, that it happens less often, if CPU shares are increased for the client in values.yml.

Are there any best practices / recommendations in your docs, which I am overseeing?

marcusweinhold on 30 Jun 2021

Hey @marcusweinhold ! This could be potentially caused because of a CPU bottleneck. It appears as the when the pods are spinning down, they might not be getting de-registered properly. We have updated this workflow a bit in the latest version of Consul 1.10.0 which ships with a newer Helm release v0.32.1. Can you try reproducing the error with the latest version. There is one difference here with the config where you will have to create a Kubernetes service for the frontend and backend deployments. But the remaining workflow should stay the same.

I will try and reproduce this error locally, but i think your hunch is right regarding CPU throttling.

thisisnotashwin on 1 Jul 2021

Thanks for the update. I'll install the new release tomorrow and evaluate over the weekend

marcusweinhold on 1 Jul 2021

hi @thisisnotashwin
the good news: I haven't been able to reproduce the problem on minikube, as described above.

the bad news: in our actual setup, we heavily rely on the service discovery variables set by the injector. I mean <NAME>_CONNECT_SERVICE_PORT as described here. These are no longer present when using the latest the chart 0.32.1. Is this some wrong configuration on my side? I've tried it with the values-minimal.yaml file from the tar above.

edit: I have set the required <NAME>_CONNECT_SERVICE_PORT variables manually, the problem seems to be gone. I'll give you an update next week when I have done more in depth testing.

marcusweinhold on 2 Jul 2021

The environment vars not being set is a regression. Created fix PR: https://github.com/hashicorp/consul-k8s/pull/550.

lkysow on 5 Jul 2021

@marcusweinhold I'll go ahead and close this issue since it is not reproducible. It looks like we have addressed the issue described in your latest comment however. This should be included in our next release of consul-k8s.

david-yu on 8 Jul 2021

👍1

Just FYI, no reopen:
The initial issue is gone after the upgrade.
But now we have something similar as in #866
Thanks for the support everyone!

marcusweinhold on 9 Jul 2021

Was this page helpful?

0 / 5 - 0 ratings