Logstash: Init.d file and config folder are missing in Logstash 5 Alpha3 Setup

[gauravjavheri]

Hi,

Last day I installed Logstash 5 Alpha3 version using installation guide https://www.elastic.co/guide/en/logstash/5.0/installing-logstash.html1 on RHEL 6.x. After installation found that "/etc/init.d/logstash" script and config folder are missing.

Thanks
Gaurav Javheri

[untergeek]

That's because starting with alpha3, Logstash generates its own startup files based on what the system uses. RHEL 6 uses upstart, so you can start Logstash with initctl start logstash or even just start logstash

[RobinAllen]

Well, I'm confused.
Here am I, on RHEL6, with logstash 5.0.0-alpha4, and I can't run it either???
You say RHEL6 uses upstart, but on my RHEL6 I'm used to using the init.d scripts. I've used RHEL for years, but I've never used "initctl" in my life.

Logstash 2 RPM works fine on my RHEL6, but logstash 5 RPM says no.

$rpm -q logstash
logstash-5.0.0~alpha4-1.noarch

$/usr/bin/sudo service logstash start
logstash: unrecognized service

$/usr/bin/sudo initctl start logstash
initctl: Unknown job: logstash

$/usr/bin/sudo start logstash
start: Unknown job: logstash

$ls -l /etc/init/logstash.conf
ls: cannot access /etc/init/logstash.conf: No such file or directory

$ls -l /etc/init.d/logstash
ls: cannot access /etc/init.d/logstash: No such file or directory

[untergeek]

@RobinAllen
There should have been some output at the end of the RPM installation. If an error occurred creating the startup scripts, some indication should have been made. Please work with us as we try to troubleshoot this with you.

For reference, building and installing the RPM works in clean installs (Vagrant VMs) of CentOS 6.

[untergeek]

Please show the contents of /etc/logstash as well, please.

[RobinAllen]

Why does an rpm post script need to depend on JAVA?

Running Transaction
Installing : 1:logstash-5.0.0~alpha4-1.noarch 1/1
Could not find any executable java binary. Please install java in your PATH or set JAVA_HOME.
warning: %post(logstash-1:5.0.0~alpha4-1.noarch) scriptlet failed, exit status 1
Non-fatal POSTIN scriptlet failure in rpm package 1:logstash-5.0.0~alpha4-1.noarch
Verifying : 1:logstash-5.0.0~alpha4-1.noarch

I have multiple JAVAs installed for different dependencies of different things. So I deliberately do not put any default JAVA on a JAVA_HOME. Otherwise things will pick the wrong one.

[RobinAllen]

And that Java will always be in "/usr/bin/java" is one heck of an assumption to make, too!....

$ source /etc/profile.d/efm-java8-sdk.sh
$ echo $JAVA_HOME
/usr/java/efm-java8-sdk

$ $JAVA_HOME/bin/java -version
java version "1.8.0_60"
Java(TM) SE Runtime Environment (build 1.8.0_60-b27)
Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)

$ /usr/share/logstash/bin/system-install /etc/logstash/startup.options
Using provided startup.options file: /etc/logstash/startup.options
/usr/share/logstash/vendor/jruby/bin/jruby: line 388: /usr/bin/java: No such file or directory
/usr/share/logstash/vendor/jruby/bin/jruby: line 388: exec: /usr/bin/java: cannot execute: No such file or directory

It's pretty rare to find an RPM that depends on a specific Java to be installed in a specific location for the %post script.

[untergeek]

While I understand the RPM convention, Logstash is 100% dependent on Java, so that it is required shouldn't be too big a surprise.

In the event that you need to specify a different Java, go ahead and change the settings in /etc/logstash/startup.options and run /usr/share/logstash/bin/system-install as root. It will regenerate the startup scripts.

[untergeek]

startup.options is described here: https://www.elastic.co/guide/en/logstash/5.0/config-setting-files.html#_settings_files

[RobinAllen]

I agree that Java is obviously a dependency. But compare ElasticSearch experience with the Logstash experience:

To run ElasticSearch 5 alpha4 with a non-standard java in a different location, was simple. A nice clean rpm install with no errors, followed by a simple one-line change in /etc/sysconfig, and run it the usual way with sysv "service elasticsearch start". Automated with ansible in mere minutes.

But then for Logstash, we have an RPM that spits out errors from the %post script.
It demands I set JAVA_HOME. So I set JAVA_HOME, and run the %post again (which any normal user won't have a clue how to find out what's going on inside the %post script)... and it ignores the JAVA_HOME that I've given it and decides to look somewhere else.
So then I configure the SECOND location that it needs the java home setting, and run it a third time, and it tells me it's successfully created the init scripts. Yay!
Except it hasn't. It's generated the Upstart init scripts. So I need to run Logstash using Upstart, even though ElasticSearch insists that I run ElasticSearch with SysV.

Two products. Same company. Completely different methodology and completely different experience. Why would one of them insist on using Upstart, when the other one insists on using SysV? Why would one of them install quick and easy with no issues, but the other one makes rash assumptions about which jdk will be installed, and can only be configured to work correctly AFTER the rpm has already failed to correctly install? (and, as far as I can tell, only failing because it's trying to go OTT with usage of Java and Ruby to install Upstart scripts that weren't needed to begin with)

[untergeek]

I'm sorry that you're frustrated. Before I address your observations about the differences between Logstash and Elasticsearch and their methodologies, please understand that we target an out of the box experience for the majority of our users, who only have a single JVM on their boxes. The environment you've described differs from the majority of our users, however it may be in your setup.

So why doesn't Logstash use SysV? We do, but only when the pleaserun wrapper script can't find upstart or systemd, as a fall-back. Part of the reasoning behind the change is _because_ Logstash isn't Elasticsearch. Elasticsearch is completely different, even if both do require a JVM. We've responded to many user complaints with regards to the previously bundled SysV init scripts (more on this will follow). This is not a problem we have with systemd or upstart. Elasticsearch does not suffer from these problems because it's completely different, and doesn't have plugins that may still be trying to send to external systems. Logstash is sometimes at the mercy of remote systems. It does its best to not lose messages, so it will wait to ensure they get delivered. The delays and pauses have frequently caused issues.

These issues range from conflicted states (up? down? something else?) where pid files remain, or second instances get started because a zombie instance hasn't properly been terminated. Despite many attempts to correct this, each distribution uses its own differing shells (sh, bash, dash, etc.), with their corresponding use quirks. This has caused many headaches. Upstart and systemd have not given us any of these problems. We had hoped that using systemd and upstart—wherever possible—would result in a better user experience.

Additionally, this is an alpha release of a major version, with many changes from previous versions. We made an initial decision to have systemctl start logstash or initctl logstash start be immediately usable, with no extra steps, which obviously requires that java be pre-installed. These design choices have also made it possible for those who deploy the tarball package to install the startup scripts their system defaults to using, including the possibility of easily installing multiple instances on the same machine, each with their own startup scripts.

We appreciate the feedback and will take it under advisement. The final release may be different from what you experienced in alpha 4.

In the meanwhile, since it appears you're using ansible to do deployments, you may find that you have more use-case-friendly results with the tarball package, since you can edit files and set variables before an launching anything.

[RobinAllen]

So back to that comparison of 'experience'....
ElasticSearch, installed and running in <5 minutes.
Logstash, an hour later I'm still debugging through bash scripts to figure out why the Upstart script isn't starting the process with the JAVACMD parameter that I gave it in the startup.options.

Even if I do as instructed, and edit the JAVACMD in startup.options, that only gets it past the init-script installer. So, it successfully installs an init script, which when run, FAILS to start up logstash. It bombs out somewhere in setup_java in logstash.lib.sh, because the script gets JAVACMD from the environment, not from the startup.options file. In fact, the only reference I can find that touches startup.options is just the init-script writer.

[untergeek]

That is correct. The referenced documentation makes clear that startup.options is only used to create the startup files.

If things are not working the way they were designed (particularly with regard to using different environment variables, java versions, etc.), please help us to figure that out with a new issue. Please attach the startup.options file, as well as the jvm.options file, and the resulting upstart file in /etc/init/logstash.conf

[RobinAllen]

I can get around this by adding
JAVA_HOME= ....
into the top of /usr/share/logstash/bin/logstash

But.... It would be really handy if the first thing that /usr/share/logstash/bin/logstash did, was something like
if [ -f /etc/sysconfig/logstash ]; then
. /etc/sysconfig/logstash;
fi
(like elasticsearch does)

Then is would just be a case of dropping the java home var into a separate file, rather than editing the init script or editing the logstash runner script itself.

[untergeek]

I agree that we can improve in this way.

Please submit your findings in a new issue, as addressing alternate JAVA_HOME environments are different from the subject of this issue.

[RobinAllen]

My other observation would be... do you REALLY need to use Java to write an init script? You can figure out sysV or upstart and generate an init script from a template in a few lines of bash. You don't really need that dependency on a whole Java JVM to write out the init scripts. It would be better if the RPM didn't make any assumptions about packages without listing them as "requires: java" in the rpm spec.