Kibana: Error in starting Kibana

[macadminrohit]

Kibana version: 5.4.0

Elasticsearch version: 5.4.0

Server OS version: RHEL 7

Browser version: Chrome 57.0.2987.110

Browser OS version: 57.0.2987.110

Original install method (e.g. download page, yum, from source, etc.):
curl -L -O https://artifacts.elastic.co/downloads/kibana/kibana-5.4.0-x86_64.rpm",
sudo rpm -i kibana-5.4.0-x86_64.rpm

Not able to start Kibana Service

[root@ip-10-206-118-223 uns114]# systemctl status kibana
â kibana.service - Kibana
Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
Active: failed (Result: start-limit) since Thu 2017-07-27 15:14:17 UTC; 59min ago
Process: 6036 ExecStart=/usr/share/kibana/bin/kibana -c /etc/kibana/kibana.yml (code=exited, status=217/USER)
Main PID: 6036 (code=exited, status=217/USER)

Jul 27 15:14:16 ip-10-206-118-223.dqa.capitalone.com systemd[1]: kibana.service: main process exited, code=exited, status=217/USER
Jul 27 15:14:16 ip-10-206-118-223.dqa.capitalone.com systemd[1]: Unit kibana.service entered failed state.
Jul 27 15:14:16 ip-10-206-118-223.dqa.capitalone.com systemd[1]: kibana.service failed.
Jul 27 15:14:17 ip-10-206-118-223.dqa.capitalone.com systemd[1]: kibana.service holdoff time over, scheduling restart.
Jul 27 15:14:17 ip-10-206-118-223.dqa.capitalone.com systemd[1]: start request repeated too quickly for kibana.service
Jul 27 15:14:17 ip-10-206-118-223.dqa.capitalone.com systemd[1]: Failed to start Kibana.
Jul 27 15:14:17 ip-10-206-118-223.dqa.capitalone.com systemd[1]: Unit kibana.service entered failed state.
Jul 27 15:14:17 ip-10-206-118-223.dqa.capitalone.com systemd[1]: kibana.service failed.
[root@ip-10-206-118-223 uns114]# uname -a
Linux ip-10-206-118-223.dqa.capitalone.com 3.10.0-514.26.2.el7.x86_64 #1 SMP Fri Jun 30 05:26:04 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

[jbudz]

Can you share the output of journalctl -fu kibana.service?

[macadminrohit]

-- Logs begin at Thu 2017-07-27 13:10:57 UTC. --
Jul 27 15:14:16 ip-10-206-118-223.dqa.capitalone.com systemd[1]: Started Kibana.
Jul 27 15:14:16 ip-10-206-118-223.dqa.capitalone.com systemd[1]: Starting Kibana...
Jul 27 15:14:16 ip-10-206-118-223.dqa.capitalone.com systemd[1]: kibana.service: main process exited, code=exited, status=217/USER
Jul 27 15:14:16 ip-10-206-118-223.dqa.capitalone.com systemd[1]: Unit kibana.service entered failed state.
Jul 27 15:14:16 ip-10-206-118-223.dqa.capitalone.com systemd[1]: kibana.service failed.
Jul 27 15:14:17 ip-10-206-118-223.dqa.capitalone.com systemd[1]: kibana.service holdoff time over, scheduling restart.
Jul 27 15:14:17 ip-10-206-118-223.dqa.capitalone.com systemd[1]: start request repeated too quickly for kibana.service
Jul 27 15:14:17 ip-10-206-118-223.dqa.capitalone.com systemd[1]: Failed to start Kibana.
Jul 27 15:14:17 ip-10-206-118-223.dqa.capitalone.com systemd[1]: Unit kibana.service entered failed state.
Jul 27 15:14:17 ip-10-206-118-223.dqa.capitalone.com systemd[1]: kibana.service failed.

[jbudz]

Bummer, not much info there. Can you try running kibana directly and see if you get an error message to stdout? Can you also confirm that your kibana.yml has logging.quiet set to false?

cd /usr/share/kibana
sudo -u kibana bin/kibana

[macadminrohit]

[root@ip-10-206-118-223 kibana]# sudo -u kibana bin/kibana
sudo: unknown user: kibana
sudo: unable to initialize policy plugin

Yes logging.quiet is set to false

[macadminrohit]

Added the user and ran this command again, below is the output:
[root@ip-10-206-118-223 kibana]# cd /usr/share/kibana/
[root@ip-10-206-118-223 kibana]# sudo -u kibana bin/kibana
log [18:03:06.661] [warning][config] Settings for "0" were not applied, check for spelling errors and ensure the plugin is loaded.
log [18:03:06.664] [warning][config] Settings for "1" were not applied, check for spelling errors and ensure the plugin is loaded.
log [18:03:06.665] [warning][config] Settings for "2" were not applied, check for spelling errors and ensure the plugin is loaded.
log [18:03:06.665] [warning][config] Settings for "3" were not applied, check for spelling errors and ensure the plugin is loaded.
log [18:03:06.665] [warning][config] Settings for "4" were not applied, check for spelling errors and ensure the plugin is loaded.
log [18:03:06.666] [warning][config] Settings for "5" were not applied, check for spelling errors and ensure the plugin is loaded.
log [18:03:06.666] [warning][config] Settings for "6" were not applied, check for spelling errors and ensure the plugin is loaded.
log [18:03:06.667] [warning][config] Settings for "7" were not applied, check for spelling errors and ensure the plugin is loaded.
log [18:03:06.667] [warning][config] Settings for "8" were not applied, check for spelling errors and ensure the plugin is loaded.
log [18:03:06.667] [warning][config] Settings for "9" were not applied, check for spelling errors and ensure the plugin is loaded.
log [18:03:06.668] [warning][config] Settings for "10" were not applied, check for spelling errors and ensure the plugin is loaded.
log [18:03:06.669] [warning][config] Settings for "11" were not applied, check for spelling errors and ensure the plugin is loaded.
log [18:03:06.669] [warning][config] Settings for "12" were not applied, check for spelling errors and ensure the plugin is loaded.
log [18:03:06.670] [warning][config] Settings for "13" were not applied, check for spelling errors and ensure the plugin is loaded.
log [18:03:06.670] [warning][config] Settings for "14" were not applied, check for spelling errors and ensure the plugin is loaded.
log [18:03:06.671] [warning][config] Settings for "15" were not applied, check for spelling errors and ensure the plugin is loaded.
log [18:03:06.671] [warning][config] Settings for "16" were not applied, check for spelling errors and ensure the plugin is loaded.
log [18:03:06.672] [warning][config] Settings for "17" were not applied, check for spelling errors and ensure the plugin is loaded.
log [18:03:06.672] [warning][config] Settings for "18" were not applied, check for spelling errors and ensure the plugin is loaded.
log [18:03:06.673] [warning][config] Settings for "19" were not applied, check for spelling errors and ensure the plugin is loaded.
log [18:03:06.673] [warning][config] Settings for "20" were not applied, check for spelling errors and ensure the plugin is loaded.
log [18:03:06.674] [warning][config] Settings for "21" were not applied, check for spelling errors and ensure the plugin is loaded.
log [18:03:06.674] [warning][config] Settings for "22" were not applied, check for spelling errors and ensure the plugin is loaded.
log [18:03:06.674] [warning][config] Settings for "23" were not applied, check for spelling errors and ensure the plugin is loaded.
log [18:03:06.675] [warning][config] Settings for "24" were not applied, check for spelling errors and ensure the plugin is loaded.
log [18:03:06.675] [warning][config] Settings for "25" were not applied, check for spelling errors and ensure the plugin is loaded.
log [18:03:06.925] [info][status][plugin:[email protected]] Status changed from uninitialized to green - Ready
log [18:03:07.039] [info][status][plugin:[email protected]] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [18:03:07.079] [info][status][plugin:[email protected]] Status changed from uninitialized to green - Ready
log [18:03:07.109] [info][status][plugin:[email protected]] Status changed from uninitialized to green - Ready
log [18:03:07.285] [info][status][plugin:[email protected]] Status changed from yellow to green - Kibana index ready
log [18:03:07.286] [info][status][plugin:[email protected]] Status changed from uninitialized to green - Ready
log [18:03:07.291] [info][listening] Server running at http://localhost:5601
log [18:03:07.293] [info][status][ui settings] Status changed from uninitialized to green - Ready

[macadminrohit]

[root@ip-10-206-118-223 uns114]# systemctl status kibana -l
â kibana.service - Kibana
Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2017-07-27 18:08:33 UTC; 1s ago
Main PID: 7930 (node)
CGroup: /system.slice/kibana.service
ââ7930 /usr/share/kibana/bin/../node/bin/node --no-warnings /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Jul 27 18:08:33 ip-10-206-118-223.dqa.capitalone.com systemd[1]: kibana.service holdoff time over, scheduling restart.
Jul 27 18:08:33 ip-10-206-118-223.dqa.capitalone.com systemd[1]: Started Kibana.
Jul 27 18:08:33 ip-10-206-118-223.dqa.capitalone.com systemd[1]: Starting Kibana...

[macadminrohit]

[root@ip-10-206-118-223 uns114]# systemctl status kibana -l
â kibana.service - Kibana
Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2017-07-27 18:09:01 UTC; 2s ago
Main PID: 8039 (node)
CGroup: /system.slice/kibana.service
ââ8039 /usr/share/kibana/bin/../node/bin/node --no-warnings /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Jul 27 18:09:03 ip-10-206-118-223.dqa.capitalone.com kibana[8039]: {"type":"log","@timestamp":"2017-07-27T18:09:03Z","tags":["warning","config"],"pid":8039,"message":"Settings for \"22\" were not applied, check for spelling errors and ensure the plugin is loaded."}
Jul 27 18:09:03 ip-10-206-118-223.dqa.capitalone.com kibana[8039]: {"type":"log","@timestamp":"2017-07-27T18:09:03Z","tags":["warning","config"],"pid":8039,"message":"Settings for \"23\" were not applied, check for spelling errors and ensure the plugin is loaded."}
Jul 27 18:09:03 ip-10-206-118-223.dqa.capitalone.com kibana[8039]: {"type":"log","@timestamp":"2017-07-27T18:09:03Z","tags":["warning","config"],"pid":8039,"message":"Settings for \"24\" were not applied, check for spelling errors and ensure the plugin is loaded."}
Jul 27 18:09:03 ip-10-206-118-223.dqa.capitalone.com kibana[8039]: {"type":"log","@timestamp":"2017-07-27T18:09:03Z","tags":["warning","config"],"pid":8039,"message":"Settings for \"25\" were not applied, check for spelling errors and ensure the plugin is loaded."}
Jul 27 18:09:03 ip-10-206-118-223.dqa.capitalone.com kibana[8039]: {"type":"log","@timestamp":"2017-07-27T18:09:03Z","tags":["status","plugin:[email protected]","info"],"pid":8039,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
Jul 27 18:09:03 ip-10-206-118-223.dqa.capitalone.com kibana[8039]: {"type":"log","@timestamp":"2017-07-27T18:09:03Z","tags":["status","plugin:[email protected]","info"],"pid":8039,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
Jul 27 18:09:03 ip-10-206-118-223.dqa.capitalone.com kibana[8039]: {"type":"log","@timestamp":"2017-07-27T18:09:03Z","tags":["status","plugin:[email protected]","info"],"pid":8039,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
Jul 27 18:09:03 ip-10-206-118-223.dqa.capitalone.com kibana[8039]: {"type":"log","@timestamp":"2017-07-27T18:09:03Z","tags":["status","plugin:[email protected]","info"],"pid":8039,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
Jul 27 18:09:03 ip-10-206-118-223.dqa.capitalone.com kibana[8039]: {"type":"log","@timestamp":"2017-07-27T18:09:03Z","tags":["status","plugin:[email protected]","info"],"pid":8039,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
Jul 27 18:09:03 ip-10-206-118-223.dqa.capitalone.com kibana[8039]: {"type":"log","@timestamp":"2017-07-27T18:09:03Z","tags":["status","plugin:[email protected]","info"],"pid":8039,"state":"green","message":"Status changed from yellow to green - Kibana index ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}

[macadminrohit]

Its running finally, I had to change the IP of the server.host as actual IP of the host in kibana.yml file.