Cloud-on-k8s: Kibana pod readiness probe always failing...

[eozcany]

Bug Report

What did you do?

I am trying to implement elastic cloud on kubernetes and following the quickstart guide https://www.elastic.co/guide/en/cloud-on-k8s/master/index.html.
After I deployed the kibana as guided on https://www.elastic.co/guide/en/cloud-on-k8s/master/k8s-deploy-kibana.html, I saw that readiness probe is always failing.

What did you expect to see?

Kibana pod succesfully running.

What did you see instead? Under which circumstances?

Readiness probe failed: Get https://172.16.19.16:5601/login: dial tcp 172.16.19.16:5601: connect: connection refused

Environment

• ECK version: 1.0
• Kubernetes information: EKS - 1.14

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.2", GitCommit:"c97fe5036ef3df2967d086711e6c0c405941e14b", GitTreeState:"clean", BuildDate:"2019-10-15T19:18:23Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.8", GitCommit:"211047e9a1922595eaa3a1127ed365e9299a6c23", GitTreeState:"clean", BuildDate:"2019-10-15T12:02:12Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}

• Kibana pod events:

Events:
  Type     Reason     Age                From                                  Message
  ----     ------     ----               ----                                  -------
  Normal   Pulled     10m                kubelet, ip-10-94-1-161.ec2.internal  Container image "docker.elastic.co/kibana/kibana:7.6.2" already present on machine
  Normal   Created    10m                kubelet, ip-10-94-1-161.ec2.internal  Created container kibana
  Normal   Started    10m                kubelet, ip-10-94-1-161.ec2.internal  Started container kibana
  Normal   Scheduled  10m                default-scheduler                     Successfully assigned elastic-system/quickstart-kb-549b7c7655-rdhfr to ip-10-94-1-161.ec2.internal
  Warning  Unhealthy  10m                kubelet, ip-10-94-1-161.ec2.internal  Readiness probe failed: Get https://172.16.19.16:5601/login: dial tcp 172.16.19.16:5601: connect: connection refused
  Warning  Unhealthy  10m (x3 over 10m)  kubelet, ip-10-94-1-161.ec2.internal  Readiness probe failed: HTTP probe failed with statuscode: 503

• Kibana pod logs:

{
    "type": "response",
    "@timestamp": "2020-04-09T19:39:54Z",
    "tags": [],
    "pid": 8,
    "method": "get",
    "statusCode": 200,
    "req": {
        "url": "/login",
        "method": "get",
        "headers": {
            "host": "172.16.19.16:5601",
            "user-agent": "kube-probe/1.14",
            "accept-encoding": "gzip",
            "connection": "close"
        },
        "remoteAddress": "172.16.19.1",
        "userAgent": "172.16.19.1"
    },
    "res": {
        "statusCode": 200,
        "responseTime": 17,
        "contentLength": 9
    },
    "message": "GET /login 200 17ms - 9.0B"
}

[thbkrkr]

The Kibana events indicate the probe failed with the status code 503 but the Kibana pod logs shows a successful request on /login with the status code 200 from the kube-probe. It is pretty strange.

What is the ouput of kubectl get elastic,pods?
Don't you have more Kibana logs?

[eozcany]

Here is the output of kubectl get elastic,pods

> kubectl get elastic,pods
NAME                                                    HEALTH   NODES   VERSION   PHASE   AGE
elasticsearch.elasticsearch.k8s.elastic.co/quickstart   green    1       7.6.2     Ready   46h

NAME                                      HEALTH   NODES   VERSION   AGE
kibana.kibana.k8s.elastic.co/quickstart   green    1       7.6.2     21h

NAME                                READY   STATUS    RESTARTS   AGE
pod/elastic-operator-0              1/1     Running   0          18h
pod/quickstart-es-default-0         1/1     Running   0          46h
pod/quickstart-kb-c4b644fb6-bffgx   1/1     Running   0          18h

[eozcany]

Where can I find more kibana logs in kibana pod ?

[anyasabo]

pod/quickstart-kb-c4b644fb6-bffgx 1/1 Running 0 18h

It looks like kibana is ready now, correct?

Also note that this should also be resolved in the next release as we have switched from using HTTP readiness probes entirely in https://github.com/elastic/cloud-on-k8s/pull/2528

Where can I find more kibana logs in kibana pod ?

kubectl logs quickstart-kb-c4b644fb6-bffgx will show all the logs, I think we just expected there to be more logs.

[david-kow]

Closing due to inactivity - feel free to reopen if the issue is still there.

[microchip78]

Any update on this ? I am also facing same problem ...

Events:
  Type     Reason     Age                From                  Message
  ----     ------     ----               ----                  -------
  Normal   Scheduled  53m                default-scheduler     Successfully assigned elastic-system/kibana-kb-74789695d9-b8p6c to kk-worker-1
  Normal   Pulling    53m                kubelet, kk-worker-1  Pulling image "docker.elastic.co/kibana/kibana:7.8.0"
  Normal   Pulled     51m                kubelet, kk-worker-1  Successfully pulled image "docker.elastic.co/kibana/kibana:7.8.0"
  Normal   Created    51m                kubelet, kk-worker-1  Created container kibana
  Normal   Started    51m                kubelet, kk-worker-1  Started container kibana
  Warning  Unhealthy  50m (x5 over 50m)  kubelet, kk-worker-1  Readiness probe failed: Get https://10.40.0.7:5601/login: dial tcp 10.40.0.7:5601: connect: connection refused
  Warning  Unhealthy  50m                kubelet, kk-worker-1  Readiness probe failed: HTTP probe failed with statuscode: 503

any suggestion on how to fix this ?

[thbkrkr]

The Kibana readiness probe fails when Kibana is not operational and this may be due from various reasons. Without more information, it's hard to point you in the right direction to fix the problem. I suggest you open a new issue describing your environment, the yaml manifests you used, the state of the pods and the logs you see, etc.

If not, at random, check that the network traffic is not blocked from hosts to containers.