Consul-helm: Readiness probe errored: rpc error: code = Unknown desc = command error: EOF, stdout: , stderr: , exit code -1

[Sanoj-kumar-DevOps]

I'm unable to run consul on our cluster .It was not working on version 0.29.0 later upgraded to 0.30.0 still I got the same Issue .Please refer the data given below and let me know.

Values:
global:
name: consul
image: hashicorp/consul:1.9.3
datacenter: hcl-stage-dc

gossipEncryption:
secretName: consul-gossip-encryption-key
secretKey: key

x-extraConfig: &extra
extraConfig: |
{ "telemetry" : [
{ "disable_hostname": true,
"prometheus_retention_time": "2m",
"disable_compat_1.9": true }
]}

server:
storage: 10Gi
<<: *extra
resources:
requests:
memory: 100Mi
cpu: 16m

acls:
manageSystemACLs: 'true'

client:
<<: *extra
resources:
requests:
memory: 100Mi
cpu: 16m
limits:
memory: 200Mi
cpu: 64m

ui:
enabled: 'true'

[Sanoj-kumar-DevOps]

Name: consul-server-0
Namespace: consul
Priority: 0
Node: k8s-worker02/10.20.128.102
Start Time: Fri, 05 Mar 2021 09:55:58 +0100
Labels: app=consul
chart=consul-helm
component=server
controller-revision-hash=consul-server-6bcd7c89b7
hasDNS=true
release=consul
statefulset.kubernetes.io/pod-name=consul-server-0
Annotations: consul.hashicorp.com/config-checksum: 7fecb8448709dd8053e1c53f4da3f9651f41b9290723845783e4034b7ea7f86e
consul.hashicorp.com/connect-inject: false
Status: Running
IP: 10.120.69.23
IPs:
IP: 10.120.69.23
Controlled By: StatefulSet/consul-server
Containers:
consul:
Container ID: cri-o://c95244e7ddf16f8f974633472b2fa2b0f31e3c5cf0d691fd39d322a5c60c63e4
Image: hashicorp/consul:1.9.3
Image ID: docker.io/hashicorp/consul@sha256:d3ca55c2cfabad06e715728962d167c8a34d4de616565df2a6543d3678fe893a
Ports: 8500/TCP, 8301/TCP, 8301/UDP, 8302/TCP, 8300/TCP, 8600/TCP, 8600/UDP
Host Ports: 0/TCP, 0/TCP, 0/UDP, 0/TCP, 0/TCP, 0/TCP, 0/UDP
Command:
/bin/sh
-ec
CONSUL_FULLNAME="consul"

  exec /bin/consul agent \
    -advertise="${ADVERTISE_IP}" \
    -bind=0.0.0.0 \
    -bootstrap-expect=3 \
    -client=0.0.0.0 \
    -config-dir=/consul/config \
    -datacenter=hcl-stage-dc \
    -data-dir=/consul/data \
    -domain=consul \
    -hcl="connect { enabled = true }" \
    -ui \
    -retry-join="${CONSUL_FULLNAME}-server-0.${CONSUL_FULLNAME}-server.${NAMESPACE}.svc:8301" \
    -retry-join="${CONSUL_FULLNAME}-server-1.${CONSUL_FULLNAME}-server.${NAMESPACE}.svc:8301" \
    -retry-join="${CONSUL_FULLNAME}-server-2.${CONSUL_FULLNAME}-server.${NAMESPACE}.svc:8301" \
    -serf-lan-port=8301 \
    -server

State:          Running
  Started:      Fri, 05 Mar 2021 09:56:01 +0100
Ready:          False
Restart Count:  0
Limits:
  cpu:     64m
  memory:  200Mi
Requests:
  cpu:      16m
  memory:   100Mi
Readiness:  exec [/bin/sh -ec curl http://127.0.0.1:8500/v1/status/leader \

2>/dev/null | grep -E '".+"'
] delay=5s timeout=5s period=3s #success=1 #failure=2
Environment:
ADVERTISE_IP: (v1:status.podIP)
POD_IP: (v1:status.podIP)
NAMESPACE: consul (v1:metadata.namespace)
Mounts:
/consul/config from config (rw)
/consul/data from data-consul (rw)
/var/run/secrets/kubernetes.io/serviceaccount from consul-server-token-n49xf (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
data-consul:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data-consul-consul-server-0
ReadOnly: false
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: consul-server-config
Optional: false
consul-server-token-n49xf:
Type: Secret (a volume populated by a Secret)
SecretName: consul-server-token-n49xf
Optional: false
QoS Class: Burstable
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s

==============================================
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Unhealthy 2m7s (x1599 over 82m) kubelet Readiness probe errored: rpc error: code = Unknown desc = command error: EOF, stdout: , stderr: , exit code -1

[Sanoj-kumar-DevOps]

NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
consul consul 2 2021-03-05 11:06:25.2699097 +0100 CET deployed consul-0.30.0 1.9.3

[ishustava]

Hey @Sanoj-kumar-DevOps

Could you provide us with a bit more info?

1. The logs of the failing consul servers
2. The Kubernetes distribution and version you are using

[lkysow]

@Sanoj-kumar-DevOps can you please follow our issue template when submitting bugs. This is what it needs to look like so we can effectively respond:

---

When filing a bug, please include the following headings if possible. Any example text in this template can be deleted.

Overview of the Issue

A paragraph or two about the issue you're experiencing.

Reproduction Steps

Steps to reproduce this issue, eg:

1. When running helm install with the following values.yml:

global:
  domain: consul
  datacenter: dc1
server:
  replicas: 1
  bootstrapExpect: 1
connectInject:
  enabled: true
controller:
  enabled: true

1. View error

Logs

Include any relevant logs.

Logs

output from 'kubectl logs' in relevant components

Expected behavior

What was the expected result?

Environment details

If not already included, please provide the following:

• consul-k8s version:
• consul-helm version:
• values.yaml used to deploy the helm chart:

Any other information you can provide about the environment/deployment.

Additional Context

Additional context on the problem.

[Sanoj-kumar-DevOps]

@ishustava Thank you for the quick response .Our applications are running on vmware Ubuntu 20.04.

Please find the attached for logs .

Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-01-14T18:56:46Z", GoVersion:"go1.15.6", Compiler:"gc", Platform:"linux/amd64"}

server-1.txt
server -0.txt
server-2.txt

$ kubectl get pods
NAME READY STATUS RESTARTS AGE
consul-64qkk 0/1 Running 0 9h
consul-n2nzg 0/1 Running 0 9h
consul-pncj2 0/1 Running 0 9h
consul-server-0 0/1 Running 0 10h
consul-server-1 0/1 Running 0 10h
consul-server-2 0/1 Running 0 9h

[ishustava]

It looks like your servers aren't configured with gossip even though you're setting the gossip encryption key.

Could you include a values file that is properly formatted as YAML? I can't tell from yours if some values are ignored because of the indentation or something like that.

[Sanoj-kumar-DevOps]

Attached is a copy of our consul and vault YAML files .Please let me know if you require anything else.
I tried deploying the file without gossipEncryptio unfortunately it didn't work

gossipEncryption:
secretName: consul-gossip-encryption-key
secretKey: key

Consul.yml.txt
Vault.yml.txt

[ndhanushkodi]

Hey @Sanoj-kumar-DevOps I tried to helm install the chart 0.30.0 with your Consul values, after creating the gossip secret using kubectl create secret generic consul-gossip-encryption-key --from-literal=key=$(consul keygen) and it installed successfully on GKE 1.16.15.

So maybe it has to do with the environment you're installing in, or the networking setup between the nodes. Can you confirm that there is network connectivity for the connections described in this diagram between nodes? (i.e if you kubectl exec onto a client pod, you should have tcp connectivity to any server pod ip at port 8300 for rpc, and similarly for lan gossip as well.)

[Sanoj-kumar-DevOps]

Hey @ndhanushkodi , Thank you for the quick response. Let me check the connection and let you know the outcome ASAP

[Sanoj-kumar-DevOps]

Hello @ndhanushkodi , The problem was with CRI-O sockets. We have upgraded from 1.17.4 to the latest version and it worked :)