Cloud-on-k8s: Ingress rule for Kibana webUI gives http code 502 (bad gateway)

[ledroide]

Hi,
I am unable to make an ingress rule that matches the Kibana service.

Here is the running kibana service :

$ kubectl get svc kibana-kb-http
NAME             TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
kibana-kb-http   ClusterIP   10.233.28.30   <none>        5601/TCP   23h

The kibana service works fine in port-forwarding, from any port, say 8080, to port 5601 - using kubectl port-forward svc/kibana-kb-http 8080:5601 (Forwarding from 127.0.0.1:8080 -> 5601), and kibana's pod logs look good:

{"type":"response","@timestamp":"2019-11-14T13:56:50Z","tags":[],"pid":6,"method":"post","statusCode":200,"req":{"url":"/api/telemetry/report","method":"post","headers":{"host":"localhost:8080","connection":"keep-alive","content-length":"477","accept":"application/json, text/plain, */*","origin":"https://localhost:8080","kbn-version":"7.4.2","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) snap Chromium/78.0.3904.97 Chrome/78.0.3904.97 Safari/537.36","content-type":"application/json;charset=UTF-8","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","referer":"https://localhost:8080/app/kibana","accept-encoding":"gzip, deflate, br","accept-language":"fr,en-US;q=0.9,en;q=0.8"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"https://localhost:8080/app/kibana"},"res":{"statusCode":200,"responseTime":468,"contentLength":9},"message":"POST /api/telemetry/report 200 468ms - 9.0B"}

So I wanted to create a simple ingress rule, like this :

spec:
  rules:
  - host: kibana.monitoring.gailuron.fr
    http:
      paths:
      - path: /
        backend:
          serviceName: kibana-kb-http
          servicePort: 5601

But I get an http error code 502 (bad gateway), and this appears on the kibana's pod logs :

{"type":"error","@timestamp":"2019-11-14T13:43:26Z","tags":["connection","client","error"],"pid":6,"level":"error","error":{"message":"139712361707392:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:242:\n","name":"Error","stack":"Error: 139712361707392:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:242:\n"},"message":"139712361707392:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:242:\n"}

Searching in issues, I have tried to disable the selfSignedCertificate (no difference) :

apiVersion: kibana.k8s.elastic.co/v1beta1
kind: Kibana
metadata:
  name: kibana
spec:
   http:
     tls:
       selfSignedCertificate:
         disabled: true

Also tried enable it (no difference) :

     tls:
       selfSignedCertificate:
         disabled: false

Also tried to define a SERVER_BASEPATH variable this way (gives http code 404) :

apiVersion: kibana.k8s.elastic.co/v1beta1
kind: Kibana
metadata:
  name: kibana
spec:
  podTemplate:
    spec:
      containers:
      - name: kibana
         env:
          - name: SERVER_BASEPATH
            value: "/kibana"
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: kibana
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    nginx.ingress.kubernetes.io/configuration-snippet: |
      rewrite ^/kibana/(.*)$ /$1 break;
      proxy_set_header Authorization "";
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: kibana.monitoring.gailuron.fr
    http:
      paths:
      - path: /kibana
        backend:
          serviceName: kibana-kb-http
          servicePort: 5601

What is the right rule for this ingress ? Am I the only one to get this issue ?

Additional info :

$ kubectl describe $(kubectl get pod -l common.k8s.elastic.co/type=kibana -o name | tail -1) | grep -B2 "Image:"
  kibana:
    Container ID:   docker://6dd4fca9beab1e397ddc3854d870e5471c1211d7abf0183b984f8ae0593516b5
    Image:          docker.elastic.co/kibana/kibana:7.4.2
$
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.2", GitCommit:"c97fe5036ef3df2967d086711e6c0c405941e14b", GitTreeState:"clean", BuildDate:"2019-10-17T17:16:09Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.2", GitCommit:"c97fe5036ef3df2967d086711e6c0c405941e14b", GitTreeState:"clean", BuildDate:"2019-10-15T19:09:08Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}
$
$ kubectl describe pod/ingress-nginx-controller-28tk4 -n ingress-nginx | grep -B2 "Image:"
  ingress-nginx-controller:
    Container ID:  docker://cb182e7c7f09881a92795d798c6a212a85cb21f54db6042d09edd065b702fbc7
    Image:         quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.26.1

[thbkrkr]

Hello,

Your first attempt with the simple rule and disabling TLS should work.

Here is what works for me (tested on k8s v1.14.7-gke.23).

• Deploy the ingress-controller:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/cloud-generic.yaml

• Deploy Elasticsearch, Kibana and an Ingress:

apiVersion: elasticsearch.k8s.elastic.co/v1beta1
kind: Elasticsearch
metadata:
  name: d3m0
spec:
  version: 7.4.2
  nodeSets:
  - name: default
    count: 3
    config:
      node.store.allow_mmap: false
---
apiVersion: kibana.k8s.elastic.co/v1beta1
kind: Kibana
metadata:
  name: d3m0
spec:
  version: 7.4.2
  count: 1
  elasticsearchRef:
    name: d3m0
  http:
    tls:
      selfSignedCertificate:
        disabled: true       
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: kibana-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
 rules:
 - host: kibana.domain.example
   http:
     paths:
     - path: /
       backend:
         serviceName: d3m0-kb-http
         servicePort: 5601

• Configure your DNS to map the ingress IP with the host declared in the ingress rule:

> kubectl get ing 
NAME             HOSTS                   ADDRESS    PORTS   AGE
kibana-ingress   kibana.domain.example   35.0.1.2   80      13m

> host kibana.domain.example
kibana.domain.example has address 35.0.1.2

Note: with multiple Kibana instances, the ingress must have the nginx.ingress.kubernetes.io/affinity: "cookie" annotation.

[ledroide]

Your first attempt with the simple rule and disabling TLS should work.

You are right : that works the way you recommend it, now.
I suggest this trick to be documented in the cloud-on-k8s page.

Solved. Thank you very much Thibault.

[zhanghba]

@ledroide
how to access kibana with outsite network? thanks

[ledroide]

how to access kibana with outsite network? thanks

@zhanghba : The way @thbkrkr recommends it works for me :

• disable selfSignedCertificate in your Kibana manifest
• set an Ingress rule that redirects a public FQDN to the path / of your kibana service name and port (depending on how you named it).

If you don't succeed, I can give you my whole Kibana configuration in a gist/snippet.

[thbkrkr]

how to access kibana with outsite network? thanks

If it's only about making Kibana accessible from the public Internet, using an ingress is not mandatory, you can also just declare the http.service.spec.type as LoadBalancer.

spec:
  version: 7.4.2
  http:
    service:
      spec:
        type: LoadBalancer

Documentation: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-accessing-elastic-services.html#k8s-allow-public-access.

[zhanghba]

@ledroide @thbkrkr
use ingress successful. on k8s cluster , and through nginx to access the kibana with outside network. Many thanks.

[krixzZ]

• public FQDN

Hi @ledroide

Can u please share your Kibana configuration? Very thanks in advance.

[ledroide]

Hi @ledroide
Can u please share your Kibana configuration? Very thanks in advance.

---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana
  namespace: logging
spec:
  version: 7.7.1
  count: 2
  config:
    elasticsearch.hosts:
      - https://elasticsearch-es-http:9200
    xpack.encryptedSavedObjects.encryptionKey: xxxxxxxxxxxxxxxxxxxx
  podTemplate:
    spec:
      containers:
      - name: kibana
        resources:
          requests:
            memory: 512Mi
            cpu: 500m
          limits:
            memory: 2Gi
            cpu: 1500m
        env:
          - name: ELASTICSEARCH_USERNAME
            value: elastic
          - name: ELASTICSEARCH_PASSWORD
            valueFrom:
              secretKeyRef:
                name: elasticsearch-es-elastic-user
                key: elastic
  http:
    tls:
      selfSignedCertificate:
        disabled: true
    service:
      spec:
        type: LoadBalancer
  elasticsearchRef:
    name: elasticsearch
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: kibana
  namespace: logging
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/affinity: cookie
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
  rules:
  - host: kibana.logging.kube-prd.prd.questel.fr
    http:
      paths:
      - path: /
        backend:
          serviceName: kibana-kb-http
          servicePort: 5601

[harshitmahapatra]

Is there way to expose kibana behind a subpath?
I tried the following, but it does not work as the health check fails:

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana
spec:
  version: 7.9.2
  count: 1
  elasticsearchRef:
    name: elasticsearch
  podTemplate:
    spec:
        containers:
        - name: kibana
          env:
          - name: "SERVER_BASEPATH"
            value: "/kibana"
          - name: "SERVER_REWRITEBASEPATH"
            value: "true"

There is a healthCheckPath option in the helm chart of kibana, does a similar configuration option exist for eck?

[anyasabo]

@harshitmahapatra I think you would need to override the path for the readiness probe in the podtemplate of your kibana resource. Currently it is hardcoded to /login.

[harshitmahapatra]

@anyasabo Thank you!