Consul-helm: Meshgateway pods are failing randomly during startup on federated kubernetes

[UlaganathanNamachivayam]

Overview of the Issue

Mesh gateway pods on kubernetes failing randomly during startup on a 4node cluster. When i kill the pod or whenever it gets provisioned on a new node its starting fine.

This behaviour is not consistent and readiness probe is failing due to connection refused on 8443 port. I tried executing the consul connect envoy command manually within the container and it just hangs too. if i keep terminating the pods or to restart then it works once in 2 occasions. But still couldn't get the replicas works on all the nodes.

I have tried changing the resource requests/limits and also probes seconds. Pod is starting fine on one node but failed to do so on another node and throws readinessprobes failure without any logs on the mesh gateway container.

Reproduction Steps

I am not seeing the behaviour on my local cluster when i wanted to reproduce it locally.

Logs

Not seeing any logs on the meshgateway container whenever this issue happens. Only thing i see is there are no logs available for this container.

Expected behavior

Meshgateway replicas should be up and running without any readiness probe failures across any node.

Environment details

AWS
EKS 1.18 version

If not already included, please provide the following:

• consul-k8s version: 0.19.0
• consul-helm version: 0.25.0
• envoy-alpine version: 1.14.4
• consul version: 1.8.4
• values.yaml used to deploy the helm chart: Please find below,

```global:
name: consul-secondary
datacenter: consul-secondary
imageK8S: ${imageK8S}
image: ${image}
imageEnvoy: ${imageEnvoy}

tls:
enabled: true
caCert:
secretName: consul-primary-federation
secretKey: caCert
caKey:
secretName: consul-primary-federation
secretKey: caKey

acls:
manageSystemACLs: true
replicationToken:
secretName: consul-primary-federation
secretKey: replicationToken

federation:
enabled: true
gossipEncryption:
secretName: consul-primary-federation
secretKey: gossipEncryptionKey

connectInject:
enabled: true
default: true
k8sDenyNamespaces: ["kube-system", "kube-public"]
sidecarProxy:
resources:
requests:
memory: 100Mi
cpu: 100m
limits:
memory: 100Mi
cpu: 100m

meshGateway:
enabled: true
service:
enabled: true
type: LoadBalancer
port: 443

ui:
enabled: true
service:
enabled: true
type: LoadBalancer

server:
extraVolumes:
- type: secret
name: consul-federation
items:
- key: serverConfigJSON
path: config.json
load: true

syncCatalog:
enabled: true
toConsul: true
toK8S: false
k8sTag: ${k8stag}
k8sAllowNamespaces: ["*"]
k8sDenyNamespaces: ["kube-system", "kube-public"]

resources:
requests:
memory: "50Mi"
cpu: "50m"
limits:
memory: "50Mi"
cpu: "50m"

[david-yu]

@UlaganathanNamachivayam Thanks for filing, could you also provide log files from the Consul servers as well? There might be something in there that may be point to the underlying issue.

[UlaganathanNamachivayam]

Thank you for your quick response.
PFB the logs. The logs are repetitive of the same errors on all running stateful sets continuously and few times got OOM killed for consul servers and changing the resource requests and limits solved it.

Also, I could see all the nodes has consul-client in running state for DS.

I have removed the node details from below logs,

Logs from consul-server-0:

_2020-11-30T22:01:35.892Z [WARN] agent.server.fsm: EnsureRegistration failed: error="failed inserting node: Error while renaming Node ID: "3c440b15-16e8-e617-2f9e-e2a87b723115": Node name is reserved by node "
2020-12-01T03:52:38.248Z [WARN] agent.router: Non-server in server-only area: non_server= area=lan
2020-12-01T03:52:38.248Z [WARN] agent.router: Non-server in server-only area: non_server= area=lan
2020-12-01T03:52:38.248Z [WARN] agent.router: Non-server in server-only area: non_server= area=lan
2020-12-01T03:59:35.280Z [INFO] agent.server.fsm: snapshot created: duration=47.518µs
2020-12-01T03:59:35.280Z [INFO] agent.server.raft: starting snapshot up to: index=6539428
2020-12-01T03:59:35.280Z [INFO] agent.server.snapshot: creating new snapshot: path=/consul/data/raft/snapshots/256-6539428-1606795175280.tmp
2020-12-01T03:59:35.357Z [INFO] agent.server.snapshot: reaping snapshot: path=/consul/data/raft/snapshots/256-6505680-1606793894497
2020-12-01T03:59:35.358Z [INFO] agent.server.raft: compacting logs: from=6512495 to=6529189
2020-12-01T03:59:35.558Z [INFO] agent.server.raft: snapshot complete up to: index=6539428
2020-12-01T01:38:32.452Z [ERROR] agent.server.rpc: multiplex conn accept failed: conn=from=:33532 error="read tcp :8300->:33532: read: connection reset by peer"
2020-12-01T01:38:33.293Z [INFO] agent.server.memberlist.lan: memberlist: Suspect ip--.compute.internal has failed, no acks received
2020-12-01T01:38:34.654Z [INFO] agent.server.serf.lan: serf: EventMemberUpdate:
2020-12-01T01:38:34.926Z [INFO] agent.server.serf.lan: serf: EventMemberUpdate:__

Latest logs from consul-server-1 when trying to get another pod but it failed also with probe failure:

_2020-12-01T08:40:49.644Z [INFO] agent.server: federation state anti-entropy synced
2020-12-01T08:40:56.172Z [ERROR] agent.server.memberlist.lan: memberlist: Failed fallback ping: read tcp 10.131.54.93:43134->10.131.52.159:8301: read: connection reset by peer
2020-12-01T08:40:56.293Z [INFO] agent.server.memberlist.lan: memberlist: Suspect ip-NODE_IP.eu-west-1.compute.internal has failed, no acks received
2020-12-01T08:40:56.558Z [INFO] agent.server.memberlist.lan: memberlist: Marking ip-NODE_UP.eu-west-1.compute.internal as failed, suspect timeout reached (2 peer confirmations)
2020-12-01T08:40:56.558Z [INFO] agent.server.serf.lan: serf: EventMemberFailed: ip-NODE_IP.eu-west-1.compute.internal 10.131.52.159
2020-12-01T08:40:56.558Z [INFO] agent.server: member failed, marking health critical: member=ip-NODE_IP.eu-west-1.compute.internal
2020-12-01T08:40:56.657Z [INFO] agent.server: federation state anti-entropy synced
2020-12-01T08:40:56.757Z [INFO] agent.server.gateway_locator: new cached locations of mesh gateways: primary=[MY_LB.amazonaws.com:443, MY_LB.amazonaws.com:443] local=[10.131.32.5:8443, 10.131.51.2:8443]
2020-12-01T08:40:58.427Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: ip-NODE_IP.eu-west-1.compute.internal 10.131.52.159
2020-12-01T08:40:58.427Z [INFO] agent.server: member joined, marking health alive: member=ip-NODE_IP.eu-west-1.compute.internal
2020-12-01T08:40:58.749Z [INFO] agent.server: federation state anti-entropy synced
2020-12-01T08:40:59.257Z [INFO] agent.server.gateway_locator: new cached locations of mesh gateways: primary=[MY_LB.elb.region.amazonaws.com:443, MY_LB.amazonaws.com:443] local=[10.131.32.5:8443, 10.131.49.84:8443, 10.131.51.2:8443]_

[UlaganathanNamachivayam]

@david-yu can you update on this issue please.

[thisisnotashwin]

@UlaganathanNamachivayam do the mesh-gateway pods only fail on the secondary cluster or do they appear to fail on the primary Consul cluster as well? During startup, the mesh-gateway pods do fail while the consul servers are starting up and performing leader election. In the case of an install, the pods should eventually become "ready" once the servers are ready. Is this not the behavior you are observing? Do the mesh-gateway pods continue to fail even once the server pods are ready and have completed leader election?

[UlaganathanNamachivayam]

@ashwin-venkatesh Its failing only on secondary clusters. Also, Its not only during startup of mesh gateway pods and the behaviour is not consistent across nodes. if i keep terminating the pods or to restart then it works once in 2 occasions. But still couldn't get the replicas works across all the nodes. I have provided the logs in the above comment.

[lkysow]

So the mesh gateway pods are getting killed by kube because they're failing their liveness checks?
Can you provide the output of kubectl describe pod <mesh-gateway-pod> and kubectl logs -c mesh-gateway -p <mesh-gateway-pd>

[lkysow]

Hi we're going to close this because we haven't heard back.