Kibana: Kibana service fails to start after upgrading from 4.5.1 to 5.0.0-alpha4

[yuwtennis]

I am reporting a issue arose from the community of elasticsearch.

https://discuss.elastic.co/t/kibana-service-fails-to-start-after-upgrading-from-4-5-1-to-5-0-0-alpha4/55022

Problem
After upgrading kibana from 4.5.1 to 5.0.0-alpha by _yum update_ , kibana fails to start as below.
OS is _RHEL 7.2 64bit_.

Jul 08 15:17:55 ip-172-30-1-196.ec2.internal systemd[1]: kibana.service holdoff time over, scheduling restart.
Jul 08 15:17:55 ip-172-30-1-196.ec2.internal systemd[1]: Started Kibana.
Jul 08 15:17:55 ip-172-30-1-196.ec2.internal systemd[1]: Starting Kibana...
Jul 08 15:17:55 ip-172-30-1-196.ec2.internal systemd[12491]: Failed at step USER spawning /usr/share/kibana/bin/kibana: No such process
Jul 08 15:17:55 ip-172-30-1-196.ec2.internal systemd[1]: kibana.service: main process exited, code=exited, status=217/USER
Jul 08 15:17:55 ip-172-30-1-196.ec2.internal systemd[1]: Unit kibana.service entered failed state.
Jul 08 15:17:55 ip-172-30-1-196.ec2.internal systemd[1]: kibana.service failed.
Jul 08 15:17:55 ip-172-30-1-196.ec2.internal systemd[1]: kibana.service holdoff time over, scheduling restart.
Jul 08 15:17:55 ip-172-30-1-196.ec2.internal systemd[1]: start request repeated too quickly for kibana.service
Jul 08 15:17:55 ip-172-30-1-196.ec2.internal systemd[1]: Failed to start Kibana.
Jul 08 15:17:55 ip-172-30-1-196.ec2.internal systemd[1]: Unit kibana.service entered failed state.
Jul 08 15:17:55 ip-172-30-1-196.ec2.internal systemd[1]: kibana.service failed.

Analysis
After checking the unit file for systemd, kibana now starts up with user _kibana_.

[root@ip-172-30-1-196 logstash]# cat /usr/lib/systemd/system/kibana.service
[Unit]
Description=Kibana

[Service]
Type=simple
User=kibana
Group=kibana
ExecStart=/usr/share/kibana/bin/kibana "-c /etc/kibana/kibana.yml"
Restart=always

[Install]
WantedBy=multi-user.target

After creating user _ #kibana_ , systemd still has errors

[root@ip-172-30-1-196 logstash]# useradd  -M -s /sbin/nologin kibana
[root@ip-172-30-1-196 logstash]# grep kibana /etc/passwd
kibana:x:1001:1001::/home/kibana:/sbin/nologin

Jul 08 15:30:39 ip-172-30-1-196.ec2.internal systemd[1]: kibana.service holdoff time over, scheduling restart.
Jul 08 15:30:39 ip-172-30-1-196.ec2.internal systemd[1]: Started Kibana.
Jul 08 15:30:39 ip-172-30-1-196.ec2.internal systemd[1]: Starting Kibana...
Jul 08 15:30:40 ip-172-30-1-196.ec2.internal kibana[12623]: fs.js:549
Jul 08 15:30:40 ip-172-30-1-196.ec2.internal kibana[12623]: return binding.open(pathModule._makeLong(path), stringToFlags(flags), mode);
Jul 08 15:30:40 ip-172-30-1-196.ec2.internal kibana[12623]: ^
Jul 08 15:30:40 ip-172-30-1-196.ec2.internal kibana[12623]: Error: EACCES: permission denied, open '/usr/share/kibana/optimize/.babelcache.json'
Jul 08 15:30:40 ip-172-30-1-196.ec2.internal kibana[12623]: at Error (native)
Jul 08 15:30:40 ip-172-30-1-196.ec2.internal kibana[12623]: at Object.fs.openSync (fs.js:549:18)
Jul 08 15:30:40 ip-172-30-1-196.ec2.internal kibana[12623]: at Object.fs.writeFileSync (fs.js:1156:15)
Jul 08 15:30:40 ip-172-30-1-196.ec2.internal kibana[12623]: at save (/usr/share/kibana/node_modules/babel-core/lib/api/register/cache.js:35:19)
Jul 08 15:30:40 ip-172-30-1-196.ec2.internal kibana[12623]: at nextTickCallbackWith0Args (node.js:420:9)
Jul 08 15:30:40 ip-172-30-1-196.ec2.internal kibana[12623]: at process._tickDomainCallback (node.js:390:13)
Jul 08 15:30:40 ip-172-30-1-196.ec2.internal kibana[12623]: at Function.Module.runMain (module.js:443:11)
Jul 08 15:30:40 ip-172-30-1-196.ec2.internal kibana[12623]: at startup (node.js:139:18)
Jul 08 15:30:40 ip-172-30-1-196.ec2.internal kibana[12623]: at node.js:968:3

To bypass this error , I now start up kibana with user root and group root.

Question
My question is should I _chown_ the whole kibana env as below after _yum update_?

chown -R kibana:kibana /usr/share/kibana

[Bargs]

@jbudz is this related to https://github.com/elastic/kibana/issues/7475?

[jbudz]

@bargs negative. In 4.5 we don't have complete scripts for managing package installations, on upgrade or removal the kibana user deleted. This is run after the user is created (if neccessary) by the new package, so the user will always end up missing. Going forward, we have this fixed in 4.x and 5.x by https://github.com/elastic/kibana/pull/6674/files.

@yuwtennis to answer your question, /usr/share/kibana/optimize is all you should need to chown.

[Bargs]

Thanks @jbudz

[yuwtennis]

@jbudz thank you.

[jbudz]

Closing this for now, upgrade from 4.5.4+ to 4.x and 5.x should be working.

[ErtanGoenuelal]

Can confirm that the Problem exists with Version 5.6.2-1
Installed: kibana-4.4.2-1.x86_64
Updated via yum after adjusting the yum Repo: kibana-5.6.2-1.x86_64

In /var/log/mesages i see:
Failed at step USER spawning /usr/share/kibana/bin/kibana: No such process
ll /usr/share/kibana/bin/kibana -rwxr-xr-x. 1 root root 612 Sep 23 15:26 /usr/share/kibana/bin/kibana

So no kibana User exists.

If i remove 5.6.2-1 and reinstall again the User will be created but the Folder is still owned by User root:
less /etc/passwd |grep kibana kibana:x:992:990:kibana service user:/home/kibana:/sbin/nologin

OS: CentOS Linux release 7.3.1611 (Core)
Kernel: 3.10.0-514.26.2.el7.x86_64