Kubespray: kibana is not started

[okamototk]

BUG REPORT

• revision: aa0da221e9bd873f778a7595856ba30a9a3a71b0
• OS: CentOS 7.5

When I enabled efk, kibana is not started. After output booting log, kibana process doesn't proceed.

addons.yml

efk_enabled: true

$ kubectl get all --all-namespaces  |grep kibana
kube-system     pod/kibana-logging-57977847fc-fvtxn         1/1       Running   0          40m
kube-system     service/kibana-logging          ClusterIP   10.233.38.35    <none>        5601/TCP        40m
kube-system     deployment.apps/kibana-logging         1         1         1            1           40m
kube-system     replicaset.apps/kibana-logging-57977847fc         1         1         1         40m

$ kubectl get all --all-namespaces  |grep elastic
kube-system     pod/elasticsearch-logging-0                 1/1       Running   0          40m
kube-system     pod/elasticsearch-logging-1                 1/1       Running   0          40m
kube-system     service/elasticsearch-logging   ClusterIP   10.233.37.242   <none>        9200/TCP        40m
kube-system   statefulset.apps/elasticsearch-logging   2         2         40m

$ kubectl get all --all-namespaces  |grep fluent
kube-system     pod/fluentd-es-v2.0.4-ptfh5                 1/1       Running   0          40m
kube-system     daemonset.apps/fluentd-es-v2.0.4          1         1         1         1            1           beta.kubernetes.io/os=linux                                 40m

Kibana's log is here:

$ kubectl logs -f pod/kibana-logging-57977847fc-fvtxn -nkube-system
{"type":"log","@timestamp":"2018-09-17T04:47:03Z","tags":["info","optimize"],"pid":1,"message":"Optimizing and caching bundles for graph, ml, kibana, stateSessionStorageRedirect, timelion and status_page. This may take a few minutes"}

I waited over 1 hour. I can't have any progress.

[okamototk]

After over 1 hours, kibana was working but I met following problem.

URL access by browser for kibana was redirected to following URL and I got error:
http://kibana.example.com/api/v1/namespaces/kube-system/services/kibana-logging/proxy/app/kibana

{"statusCode":404,"error":"Not Found","message":"Not Found"}

Then I got following error in kibana pod:

{"type":"response","@timestamp":"2018-09-17T12:30:50Z","tags":[],"pid":1,"method":"get","statusCode":404,"req":{"url":"/api/v1/namespaces/kube-system/services/kibana-logging/proxy/app/kibana","method":"get","headers":{"host":"kibana.example.com","connection":"close","x-request-id":"52054897a7b9505ce4f1c908db07160f","x-real-ip":"133.218.147.197","x-forwarded-for":"133.218.147.197","x-forwarded-host":"kibana.example.com","x-forwarded-port":"80","x-forwarded-proto":"http","x-original-uri":"/api/v1/namespaces/kube-system/services/kibana-logging/proxy/app/kibana","x-scheme":"http","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8","referer":"http://kibana.example.com/","accept-encoding":"gzip, deflate","accept-language":"ja,en-US;q=0.9,en;q=0.8"},"remoteAddress":"10.233.64.105","userAgent":"10.233.64.105","referer":"http://kibana.example.com/"},"res":{"statusCode":404,"responseTime":9,"contentLength":9},"message":"GET /api/v1/namespaces/kube-system/services/kibana-logging/proxy/app/kibana 404 9ms - 9.0B"}

Workaround

After remove following line in kibana-deployment.yaml, then kibana worked correctly.

        - name: SERVER_BASEPATH
          value: /api/v1/namespaces/kube-system/services/kibana-logging/proxy

[Atoms]

We are removing EFK Stack from kubespray:
https://github.com/kubernetes-incubator/kubespray/issues/3352